README.md

# MWT SDN Plugtests 4 materials

Tools and test suites developed for the 4th ETSI mWT SDN Plugtests

ETSI is organizing the 4th mWT (millimetre Wave Transmission) SDN (Software Defined Network) Plugtests™ event under the mWT Plugtests Programme.

Contact: plugtests@etsi.org.

## How to get Write Access to this repository

Log into this ETSI Forge Gitlab server using your ETSI On Line account (EOL account or ETSI Portal account).

Take a note of your username and send it to Ultan Mulligan or Xavier Piednoir, requesting access.

## lncc.py Tutorial

### Script launch & first steps

The Lightweight NETCONF Controller is build on top of _cmd2_ Python package : it benefits from all [built-in features](https://cmd2.readthedocs.io/en/stable/features/index.html): [help](https://cmd2.readthedocs.io/en/stable/features/help.html), [history](https://cmd2.readthedocs.io/en/stable/features/history.html), [script](https://cmd2.readthedocs.io/en/stable/features/scripting.html) and more...

Start as any script: `python3 lncc.py` or `./lncc.py`. 
* You will get help using the `help` command. More verbose will be available in future, using `help <command>` or `command --help`.
* You will exit using `quit` command or EOF (Ctrl+D).


<details><summary>help & quit commands feedback</summary>

```console
$ python3 lncc.py
Welcome to the Lightweight NETCONF Controller lncc.py
Type 'help' for more information
(Cmd) help

Documented commands (use 'help -v' for verbose/'help <topic>' for details):

Input & output handling
=======================
sleep  table

NETCONF
=======
connect  netconf

Parameter management
====================
mapping  ne

Uncategorized
=============
alias  help     macro  run_pyscript  set    shortcuts
edit   history  quit   run_script    shell

(Cmd) quit
$
```
</details>

<details><summary>"help netconf" commands feedback</summary>

```console
$ python3 lncc.py
Welcome to the Lightweight NETCONF Controller lncc.py
Type 'help' for more information
(Cmd) help netconf
Usage: netconf [-h] [--reply REPLY-NAME]
               {get, get-config, edit-config, copy-config, delete-config, lock,

Run NETCONF operations on NEs

optional arguments:
  -h, --help            show this help message and exit
  --reply REPLY-NAME

NETCONF operations:
  {get, get-config, edit-config, copy-config, delete-config, lock, unlock, clos

(Cmd)
(Cmd) help netconf
Usage: netconf [-h] [--reply REPLY-NAME]
               {get, get-config, edit-config, copy-config, delete-config,
               lock, unlock, close-session, kill-session, commit,
               discard-changes, cancel-commit, validate} ...

Run NETCONF operations on NEs

optional arguments:
  -h, --help            show this help message and exit
  --reply REPLY-NAME

NETCONF operations:
  {get, get-config, edit-config, copy-config, delete-config, lock, unlock, close-session, kill-session, commit, discard-changes, cancel-commit, validate}

(Cmd) quit
```
</details>

### Quick example

- Adding NEs or loading NEs --> `ne add`,
- connect to NEs --> `connect`,
- run NETCONF operations on NEs (get, edit-config, close-session, ...), -> The results is stored with the operation's name.
- transform NETCONF operations replies,
- write tranformed informations as file.

#### Add NEs
```
(Cmd) ne add my_ne1 --host 172.20.183.168 --username netconfuser --password NotSoSecret123*
NE my_ne1 added, current: 1 NEs

(Cmd) ne add my_ne2 --host 172.20.183.169 --username netconfuser --password NotSoSecret123* --port 830
NE my_ne2 added, current: 2 NEs
```
You can save NEs configuration in a YAML file using `ne dump my_nes.yaml`. Then load using `ne load my_nes.yaml`

#### Connect to all NEs

```
(Cmd) connect
2 new connections, 2 total: ['my_ne1', 'my_ne2']
```
#### Run NETCONF operations
``` 
(Cmd) netconf get --filter-file interface_type_microwaveCarrierTermination_filter.xml
NETCONF <get> operation completed: 2 NEs, stored in 'get'
```
#### Transform the results
```
(Cmd) table from_xml --xsl-transform get-interface-transform.xsl
New tables 'get' in each NE, build from XML stored in 'get'

(Cmd) table concat
Table 'get' available for group

(Cmd) table to_excel interfaces.xlsx
Wrote Excel file interfaces.xlsx, with 1 tabs: args.tables
```
### Creating a script

If the commands you can convert
We rely on _cmd2_ build-in commands: [`history --script`](https://cmd2.readthedocs.io/en/stable/features/history.html#for-users) then `run_script`.

Once the set of command make a interesting scenario, you can save it as TXT script file. You can edit this TXT script with you usual editor (type `edit` build-in command to start vim).
```
(Cmd) history --script > get_interface_inventory_script2.txt
(Cmd)
```
Then let's play again the same command sequence, using `run_script` command. You may see errors if you keep adding NEs taht exists.
```
(Cmd) run_script get_interface_inventory_script2.txt
Error: NE my_ne1 already exist
Error: NE my_ne2 already exist
0 new connections, 2 total: ['my_ne1', 'my_ne2']
NETCONF <get> operation completed: 2 NEs, stored in 'get'
New tables 'get' in each NE, build from XML stored in 'get'
Table 'get' available for group
Wrote Excel file interfaces.xlsx, with 1 tabs: args.tables
```

### Creating a transcript
Transcripts enable us to compare the script behavior from different NEs. Transcript records both the commands input and their ouputs. Transcript are created with the ['--transcript' argument of the 'run_script'](https://cmd2.readthedocs.io/en/stable/features/transcripts.html) command.

In order to compare the behavior, we need to redirect the feedback (~stderr) to the ouput (~stdout), using the `set feedback_to_output true`command.

```
(Cmd) set feedback_to_output true
(Cmd) run_script get_interface_inventory_script2.txt --transcript get_interface_inventory_transcript2.txt
7 commands and their outputs saved to transcript file '/home/jean/lncc/get_interface_inventory_transcript2.txt'
```

### Test a transcript

[Transcipt](https://cmd2.readthedocs.io/en/stable/features/transcripts.html#running-a-transcript) have to be run from your shell (not inside the lncc.py!). 
It will play the commands and will compare the output of these commands with the content of the transcript.
```console
$ python3 lncc.py --test get_interface_inventory_transcript.txt
=========================================== cmd2 transcript test ============================================
platform linux -- Python 3.8.10, cmd2-2.4.0, readline-RlType.GNU
cwd: /home/jean/lncc
cmd2 app: lncc.py
collected 1 transcript
.
----------------------------------------------------------------------
Ran 1 test in 3.459s

OK
=================================== 1 transcript passed in 3.459 seconds ====================================
$
```
## Sharing your transcript

The command-line doesn't containt vendor-specific informations: they can be share to check that the behavior is common.

## Already available transcripts

2 transcript are already available in [script directory](https://forge.etsi.org/rep/sdn/mwt/mwt-sdn-plugtests-4-materials/-/tree/main/scripts), as of May 9th 2022. You already have the possibility to test them on your equipements.

- _get_interface_inventory_transcript.txt_ -- Basically the scenario described in this tutorial
- _set_tx_power_transcript.txt_

<details><summary>get_interface_inventory: script & transcript</summary>

```console
$ cat get_interface_inventory_script.txt
# get_interface_inventory_script.txt : script for Lightweight NETCONF controlleur lncc.py
# run me using: lncc.py 'run_script set_tx_power_script.txt' quit

set feedback_to_output true
ne load ne.yaml
# set logging_config logging_config.yaml
connect
netconf get --filter-file interface_type_microwaveCarrierTermination_filter.xml
netconf close-session
table from_xml --xsl-transform get-interface-transform.xsl
table concat
table to_excel interfaces.xlsx
```
```console
$ cat get_interface_inventory_transcript.txt
(Cmd) # get_interface_inventory_script.txt : script for Lightweight NETCONF controlleur lncc.py
(Cmd) # run me using: lncc.py 'run_script set_tx_power_script.txt' quit
(Cmd) set feedback_to_output true
feedback_to_output - was: False
now: True
(Cmd) ne load ne.yaml
Loading complete: 2 new NEs: ['ne1', 'ne2']
(Cmd) # set logging_config logging_config.yaml
(Cmd) connect
2 new connections, 2 total: ['ne1', 'ne2']
(Cmd) netconf get --filter-file interface_type_microwaveCarrierTermination_filter.xml
NETCONF <get> operation completed: 2 NEs, stored in 'get'
(Cmd) netconf close-session
NETCONF <close-session> operation completed: 2 NEs, stored in 'close-session'
(Cmd) table from_xml --xsl-transform get-interface-transform.xsl
New tables 'get' in each NE, build from XML stored in 'get'
(Cmd) table concat
Table 'get' available for group
(Cmd) table to_excel interfaces.xlsx
Wrote Excel file interfaces.xlsx, with 1 tabs: args.tables
```
</details>
<details><summary>set_tx_power: script & transcript</summary>

```console
$ cat set_tx_power_script.txt
# set_tx_power_script.txt for Lightweight NETCONF controlleur lncc.py
# run using: lncc.py 'run_script set_tx_power_script.txt' quit

set feedback_to_output true
ne load ne.yaml
mapping add set-power --ne ne1 '{if-name: RTL-63-255-1, tx-power: -8}'
mapping add set-power --ne ne2 '{if-name: RTL-63-255-1, tx-power: -7}'
connect
netconf lock
netconf edit-config --config-template set_tx_power.xml.pyformat --mappings set-power
netconf commit --confirmed
sleep 10
netconf commit
netconf unlock
netconf close-session
```
```console
$ cat set_tx_power_transcript.txt
(Cmd) # set_tx_power_script.txt for Lightweight NETCONF controlleur lncc.py
(Cmd) # run using: lncc.py 'run_script set_tx_power_script.txt' quit
(Cmd) set feedback_to_output true
feedback_to_output - was: False
now: True
(Cmd) ne load ne.yaml
Loading complete: 2 new NEs: ['ne1', 'ne2']
(Cmd) mapping add set-power --ne ne1 '{if-name: RTL-63-255-1, tx-power: -8}'
(Cmd) mapping add set-power --ne ne2 '{if-name: RTL-63-255-1, tx-power: -7}'
(Cmd) connect
2 new connections, 2 total: ['ne1', 'ne2']
(Cmd) netconf lock
NETCONF <lock> operation completed: 2 NEs, stored in 'lock'
(Cmd) netconf edit-config --config-template set_tx_power.xml.pyformat --mappings set-power
NETCONF <edit-config> operation completed: 2 NEs, stored in 'edit-config'
(Cmd) netconf commit --confirmed
NETCONF <commit> operation completed: 2 NEs, stored in 'commit'
(Cmd) sleep 10
Sleeping for 10 seconds
(Cmd) netconf commit
NETCONF <commit> operation completed: 2 NEs, stored in 'commit'
(Cmd) netconf unlock
NETCONF <unlock> operation completed: 2 NEs, stored in 'unlock'
(Cmd) netconf close-session
NETCONF <close-session> operation completed: 2 NEs, stored in 'close-session'
```
</details>

### Dependencies
* [Python 3](http://python.org)
* [Python 3 packages](https://forge.etsi.org/rep/sdn/mwt/mwt-sdn-plugtests-4-materials/-/tree/main/requirements.txt)

Sometimes the underlying package requirements may change for `lncc.py`.
If you run it from scratch, you can manually update these
requirements by executing something like:
```
    pip3 install --upgrade -r requirements.txt
```

## License

The content of the present repository is released under the BSD-3 Clause license.