From dda30ec43401ec79b286284b93fa3c770c1e02a1 Mon Sep 17 00:00:00 2001 From: Benoit Orihuela Date: Fri, 5 Jan 2024 17:13:51 +0100 Subject: [PATCH 1/6] doc: start documentation on how to add TCs in the Test Suite --- README.md | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 74123594..aea56665 100644 --- a/README.md +++ b/README.md @@ -147,8 +147,35 @@ Robot Framework. Our recommendations are: - Install [Robot Framework Language Server](https://plugins.jetbrains.com/plugin/16086-robot-framework-language-server) - Define as variable the path of the working directory. In Settings > Languages & Frameworks > Robot Framework -(Project), insert the following: `{"EXECDIR": "{path}/auth-test-suite"}`. +(Project), insert the following: `{"EXECDIR": "{path}/ngsi-ld-test-suite"}`. +### Develop a new Test Case + +In order to develop a new Test Case, some rules and conventions have to be followed. + +#### Test Case Structure + +TODO BOR + +#### Generate the documentation + +In some cases, the scripts used to generate the full documentation have to be updated: + +- When a new Test Case is created, it has to be declared in the corresponding `doc/tests/test_{group}.py` Python file. + Complementary to this, updates have to be done: + - In `doc/analysis/initial_setup.py` if it uses a new keyword to set up the initial state of the Test Case + - In `doc/analysis/requests.py` if it updates an endpoint operation, for instance by adding support for a new request + parameter (in `self.op` with the list and position of parameters, in the method that pretty prints the operation) +- When a new global Keyword is created, it has to be declared in the corresponding Python files: + - In `doc/analysis/checks.py` if it is an assertion type keyword (in `self.checks`, `self.args` and add the method that + pretty prints the check) + - In `doc/analysis/requests.py` if it is an endpoint operation (in `self.op` with the list and position of parameters, + in `self.description` to reference the method that pretty print the operation, and add the method that pretty prints + the operation) +- When a new directory containing Test Cases is created, it has to be declared in `doc/generaterobotdata.py` along with + its acronym + +Then a new version of the documentation can be generated by running the `doc/generateDocumentationData.py` Python script. ### Run configurations (PyCharm) -- GitLab From 3b386463e84733b6240d59696f8f33558b4368a0 Mon Sep 17 00:00:00 2001 From: Benoit Orihuela Date: Tue, 9 Jan 2024 21:17:33 +0100 Subject: [PATCH 2/6] doc: integrate feedback from Fernando --- README.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index aea56665..c16716a9 100644 --- a/README.md +++ b/README.md @@ -159,16 +159,28 @@ TODO BOR #### Generate the documentation -In some cases, the scripts used to generate the full documentation have to be updated: - -- When a new Test Case is created, it has to be declared in the corresponding `doc/tests/test_{group}.py` Python file. - Complementary to this, updates have to be done: +The Conformance Tests include an option to automatically generate the documentation of them. If new tests are developed +but include the Check and Request operations already defined, it is not needed to modify the automatic generation system. +Nevertheless, if there are new operations to check and/or new request operations to develop in the Conformance Tests, +it is needed to describe them so that the automatic documentation generated covers these new operations. + +Additionally, there is defined a Unit Test mechanism to identify if new Test Suites are included in the system +(See `doc/tests/test_CheckTests.py`) and if there were some changes in any Test Suite (See the `doc/tests/other test_*.py`). +In these cases, it is needed to provide the corresponding information in the Python code: + +- When a new Test Case is created, it has to be declared in the corresponding `doc/tests/test_{group}.py` Python file + (where {group} represents the group of operations and how they are defined in the ETSI RGS/CIM-0012v211 document). + It includes the creation of the corresponding expected result inside the `doc/files` folders to compare the generation + of the documentation. Complementary to this, updates have to be done: - In `doc/analysis/initial_setup.py` if it uses a new keyword to set up the initial state of the Test Case - In `doc/analysis/requests.py` if it updates an endpoint operation, for instance by adding support for a new request - parameter (in `self.op` with the list and position of parameters, in the method that pretty prints the operation) + parameter (in `self.op` with the list and position of parameters, in the method that pretty prints the operation in + the automated generation of the documentation associated with the Test Cases. If the positions array is empty, it + means that all the parameters will be taken into consideration) - When a new global Keyword is created, it has to be declared in the corresponding Python files: - - In `doc/analysis/checks.py` if it is an assertion type keyword (in `self.checks`, `self.args` and add the method that - pretty prints the check) + - In `doc/analysis/checks.py` if it is an assertion type keyword (in `self.checks` to reference the method that pretty + prints the operation and `self.args` to identify the position of the arguments) and the corresponding method to + provide the description of this new operation - In `doc/analysis/requests.py` if it is an endpoint operation (in `self.op` with the list and position of parameters, in `self.description` to reference the method that pretty print the operation, and add the method that pretty prints the operation) -- GitLab From 2622d76a17b26535feb99adb008ad83988e9142d Mon Sep 17 00:00:00 2001 From: Benoit Orihuela Date: Tue, 9 Jan 2024 21:41:43 +0100 Subject: [PATCH 3/6] doc: introduce Test Case template to be used when creating new Test Cases --- README.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index c16716a9..12b8cd3f 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ test launch command followed by the file name. ## Contribute to the Test Suite -In order to contribute to the ETSI NGSI-LD Test Suite, you recommend the installation of an IDE with the corresponding +In order to contribute to the ETSI NGSI-LD Test Suite, it is recommended to install an IDE with the corresponding Robot Framework. Our recommendations are: - PyCharm @@ -153,9 +153,59 @@ Robot Framework. Our recommendations are: In order to develop a new Test Case, some rules and conventions have to be followed. -#### Test Case Structure +#### Test Case Template -TODO BOR +The following template shows the typical structure of a Test Case. It can be used as a template when creating a new +Test Case. + +``` +*** Settings *** +Documentation {Describe the behavior that is being checked by this Test Case} + +Resource ${EXECDIR}/resources/any/necessary/resource/file + +# For both setup and teardown steps, try as possible to use Keyword names already declared in doc/analysis/initial_setup.py +Suite Setup {An optional setup step to create data necessary for the Test Case} +Suite Teardown {An optional teardown step to delete data created in the setup step} +Test Template {If the Test Case uses permutations, the name of the Keyword that is called for each permutation} + + +*** Variables *** +${my_variable}= my_variable + + +*** Test Cases *** PARAM_1 PARAM_2 +XXX_YY_01 Purpose of the first permutation + [Tags] {resource_request_reference} {section_reference_1} {section_reference_2} + param_1_value param_2_value +XXX_YY_02 Purpose of the second permutation + [Tags] {resource_request_reference} {section_reference_1} {section_reference_2} + param_1_value param_2_value + + +*** Keywords *** +{Keyword name describing what the Test Case is doing} + [Documentation] {Not sure this documentation brings something?} + [Arguments] ${param_1} ${param_2} + + # Call operation that is being tested, passing any needed argument + + # Perform checks on the response that has been received from the operation + + # Optionally call another endpoint to do extra checks (e.g. check an entity has really been created) + +# Add there keywords for setup and teardown steps +# Setup step typically creates data necessary for the Test Case and set some variables that will be used by the Test Case +# using Set Suite Variable or Set Test Variable keywords +# Teardown step must delete any data that has been created in setup step +``` + +Where : + +- The possible values for `{resource_request_reference}` are defined in DGR/CIM-0015v111, section 6.1 +- The meaning of the `{section_reference_x}` tags is defined in DGR/CIM-0015v111, section 6.2. A Test Case can reference + more than one section in the specification (for instance, a Test Case whose goal is to check that scopes are correctly + created when an entity is created could reference sections 4.18 NGSI-LD Scopes and 5.6.1 Create Entity) #### Generate the documentation @@ -169,7 +219,7 @@ Additionally, there is defined a Unit Test mechanism to identify if new Test Sui In these cases, it is needed to provide the corresponding information in the Python code: - When a new Test Case is created, it has to be declared in the corresponding `doc/tests/test_{group}.py` Python file - (where {group} represents the group of operations and how they are defined in the ETSI RGS/CIM-0012v211 document). + (where `{group}` represents the group of operations and how they are defined in the ETSI RGS/CIM-0012v211 document). It includes the creation of the corresponding expected result inside the `doc/files` folders to compare the generation of the documentation. Complementary to this, updates have to be done: - In `doc/analysis/initial_setup.py` if it uses a new keyword to set up the initial state of the Test Case -- GitLab From be8f2eba5c6fb475c0ca9f593cb340b306491c10 Mon Sep 17 00:00:00 2001 From: Benoit Orihuela Date: Tue, 13 Feb 2024 09:06:14 +0100 Subject: [PATCH 4/6] fix: reference last version of DGR/CIM-0015 in the README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 12b8cd3f..6c2a7149 100644 --- a/README.md +++ b/README.md @@ -202,8 +202,8 @@ XXX_YY_02 Purpose of the second permutation Where : -- The possible values for `{resource_request_reference}` are defined in DGR/CIM-0015v111, section 6.1 -- The meaning of the `{section_reference_x}` tags is defined in DGR/CIM-0015v111, section 6.2. A Test Case can reference +- The possible values for `{resource_request_reference}` are defined in DGR/CIM-0015v211, section 6.1 +- The meaning of the `{section_reference_x}` tags is defined in DGR/CIM-0015v211, section 6.2. A Test Case can reference more than one section in the specification (for instance, a Test Case whose goal is to check that scopes are correctly created when an entity is created could reference sections 4.18 NGSI-LD Scopes and 5.6.1 Create Entity) -- GitLab From dc71a749efe06d5e4d2a3fbc39d2441568bcb004 Mon Sep 17 00:00:00 2001 From: Benoit Orihuela Date: Sun, 18 Feb 2024 11:38:21 +0100 Subject: [PATCH 5/6] doc: add reference to RGS/CIM-009v151 --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 6c2a7149..b01ca4ac 100644 --- a/README.md +++ b/README.md @@ -205,7 +205,8 @@ Where : - The possible values for `{resource_request_reference}` are defined in DGR/CIM-0015v211, section 6.1 - The meaning of the `{section_reference_x}` tags is defined in DGR/CIM-0015v211, section 6.2. A Test Case can reference more than one section in the specification (for instance, a Test Case whose goal is to check that scopes are correctly - created when an entity is created could reference sections 4.18 NGSI-LD Scopes and 5.6.1 Create Entity) + created when an entity is created should reference sections 4.18 NGSI-LD Scopes and 5.6.1 Create Entity of the + RGS/CIM-009v151 document) #### Generate the documentation -- GitLab From 1bab0cbb1c6ebb2eac806d06f7db03e82dcfe6ee Mon Sep 17 00:00:00 2001 From: Benoit Orihuela Date: Sun, 18 Feb 2024 11:40:37 +0100 Subject: [PATCH 6/6] fix: reference to RGS/CIM-009v131 for base version --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b01ca4ac..185a169c 100644 --- a/README.md +++ b/README.md @@ -206,7 +206,7 @@ Where : - The meaning of the `{section_reference_x}` tags is defined in DGR/CIM-0015v211, section 6.2. A Test Case can reference more than one section in the specification (for instance, a Test Case whose goal is to check that scopes are correctly created when an entity is created should reference sections 4.18 NGSI-LD Scopes and 5.6.1 Create Entity of the - RGS/CIM-009v151 document) + RGS/CIM-009v131 document) #### Generate the documentation -- GitLab