NGSI-LD Test Suite issueshttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues2021-01-05T16:32:16Zhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/1Add a function that builds up the context link2021-01-05T16:32:16ZHoucem Kacemhoucem.kacem@egm.ioAdd a function that builds up the context linkThe context used as a link header in http requests is built in different keywords:
* Query Temporal Representation Of Entities
* Retrieve Temporal Representation Of Entity
* ...
Better to add a function that builds up the context link.The context used as a link header in http requests is built in different keywords:
* Query Temporal Representation Of Entities
* Retrieve Temporal Representation Of Entity
* ...
Better to add a function that builds up the context link.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/2Review and upade test data files names2023-09-27T14:45:19ZHoucem Kacemhoucem.kacem@egm.ioReview and upade test data files names- Use the `template` marker instead of `sample` in the names
- TBC- Use the `template` marker instead of `sample` in the names
- TBCBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/3Allow to publicly expose an @context from the forge2023-03-09T08:19:08ZBenoit OrihuelaAllow to publicly expose an @context from the forgeCurrently, the project is protected, so we can't reference a JSON-LD @context hosted in the source code in the test cases because the SUT won't be able to resolve it.
We temporarily deployed a context on GitHub (https://raw.githubuserco...Currently, the project is protected, so we can't reference a JSON-LD @context hosted in the source code in the test cases because the SUT won't be able to resolve it.
We temporarily deployed a context on GitHub (https://raw.githubusercontent.com/easy-global-market/ngsild-api-data-models/feature/add-json-ld-context-for-ngsi-ld-test-suite/ngsi-ld-test-suite/ngsi-ld-test-suite-context.jsonld).
When the project is publicly opened, move this context somewhere in the source code.T2 - Improve current Test Suite and testing environmentpoujolpoujolhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/4Migrate the "Load Entity" and "Generate Random Entity Id" keywords to the mor...2023-09-27T14:45:43ZBenoit OrihuelaMigrate the "Load Entity" and "Generate Random Entity Id" keywords to the more generic versionsSee newly created `Load Test Sample` keyword. And create a new keyword to generate an URI (providing it a type), on the `urn:ngsi-ld:<Type>:<Random UUID>` form.See newly created `Load Test Sample` keyword. And create a new keyword to generate an URI (providing it a type), on the `urn:ngsi-ld:<Type>:<Random UUID>` form.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/5Check the response body is empty on queries returning 201 or 2042023-02-22T11:04:27ZBenoit OrihuelaCheck the response body is empty on queries returning 201 or 204https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/6Dynamically inject the endpoint.uri in context source registration subscriptions2021-02-10T09:17:42ZBenoit OrihuelaDynamically inject the endpoint.uri in context source registration subscriptionsBased on notification server information already defined in `variables.py`Based on notification server information already defined in `variables.py`2021-02-10https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/7Add TP and TC for temporal/entityOperations/query endpoint (6.24)2023-03-06T12:33:11ZBenoit OrihuelaAdd TP and TC for temporal/entityOperations/query endpoint (6.24)2021-02-10https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/8Implement missing TC in "query temporal evolution" (021_10)2021-03-19T17:03:02ZBenoit OrihuelaImplement missing TC in "query temporal evolution" (021_10)https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/9Make entity discovery tests agnostic of other existing entities2021-02-16T07:19:32ZBenoit OrihuelaMake entity discovery tests agnostic of other existing entitiesCurrently, the expectation data will only typically work if there are no other entities in the context broker. Of course, this will almost never happen and other tests may have created other entity types and other attributes.
So the tes...Currently, the expectation data will only typically work if there are no other entities in the context broker. Of course, this will almost never happen and other tests may have created other entity types and other attributes.
So the tests should only check for things they have created and are sure to be present. In other words, checking against the whole response body is not bulletproof.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/10Fix the request parameters in the QueryTemporalEvolutionOfEntities TCs2021-03-22T09:53:47ZBenoit OrihuelaFix the request parameters in the QueryTemporalEvolutionOfEntities TCsWhen requesting with a type, an attribute or else an id, a list is used instead the expected comma-separated list.
See for instance in 021_01:
```
@{entity_types_to_be_retrieved}= Create List Vehicle
```
Instead the entity type...When requesting with a type, an attribute or else an id, a list is used instead the expected comma-separated list.
See for instance in 021_01:
```
@{entity_types_to_be_retrieved}= Create List Vehicle
```
Instead the entity types should be prepared like this:
```
${entity_types_to_be_retrieved}= Catenate SEPARATOR=, Vehicle
```2021-03-22https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/11Fix the request parameters in the Query Entities calls made to check the resu...2021-03-22T09:54:03ZBenoit OrihuelaFix the request parameters in the Query Entities calls made to check the results of batch operationsSame problem as in #10Same problem as in #102021-03-22https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/12Review the TCs written for subscriptions2021-03-22T13:46:09ZBenoit OrihuelaReview the TCs written for subscriptionsFor instance:
- 032_02 can't work because you are providing an invalid URI (which is the object of 032_01), you should a non-existing **valid** URI (same for 030_02? and 029_02?)
- 031_01 seems to fail because of the order of the results...For instance:
- 032_02 can't work because you are providing an invalid URI (which is the object of 032_01), you should a non-existing **valid** URI (same for 030_02? and 029_02?)
- 031_01 seems to fail because of the order of the results. Is there an expected order when we query subscriptions?
- I saw a bunch of failed comparisons with `{'dictionary_item_removed': [root['isActive'], root['notification']['timesSent'], root['notification']['attributes']]}` (using Stellio), is it normal??
Review them all and ensure the implementation is correct.2021-03-22https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/13Improve query resources assertion fail message2021-03-24T11:11:52ZHoucem Kacemhoucem.kacem@egm.ioImprove query resources assertion fail messageFor query subscriptions (TC: 031_01) and other query resources TCs, when the assertion fails, the comparison error message is not fully understandable. This should be improved.For query subscriptions (TC: 031_01) and other query resources TCs, when the assertion fails, the comparison error message is not fully understandable. This should be improved.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/14Compliance of payload used in 029_052021-03-24T15:12:15ZBenoit OrihuelaCompliance of payload used in 029_05Payload used in 029_05 is:
```
{
"entities": [
{
"type": "Vehicle"
}
]
}
```
But as said in 5.8.24: "If the data types and restrictions expressed by clause 5.2.12 are not met by the Subscription Frag...Payload used in 029_05 is:
```
{
"entities": [
{
"type": "Vehicle"
}
]
}
```
But as said in 5.8.24: "If the data types and restrictions expressed by clause 5.2.12 are not met by the Subscription Fragment, then an error of type BadRequestData shall be raised".
Which is not what you expect in this TC.2021-03-24https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/15Allow to configure logs to only see requests / responses (expected and actual)2023-07-30T15:47:24ZBenoit OrihuelaAllow to configure logs to only see requests / responses (expected and actual)Currently very verbose, whereas we are often mainly interested in requests / responses and expectations.Currently very verbose, whereas we are often mainly interested in requests / responses and expectations.T2 - Improve current Test Suite and testing environmentpoujolpoujolhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/16Make clear what are the expected / actual payload when comparing data in checks2023-07-30T15:47:11ZBenoit OrihuelaMake clear what are the expected / actual payload when comparing data in checksBy default, it is using a before / after notation which is not very clear. See if it can use the usual "expected / actual" way of expressing such checks.By default, it is using a before / after notation which is not very clear. See if it can use the usual "expected / actual" way of expressing such checks.T2 - Improve current Test Suite and testing environmentpoujolpoujolhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/17Add the tags in the Test Cases2021-04-02T12:28:54ZBenoit OrihuelaAdd the tags in the Test Casesreference to clause in the specification + endpoint name as agreed last Tuesdayreference to clause in the specification + endpoint name as agreed last Tuesday2021-04-06https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/18Implement a specific comparison function for dates2023-09-27T14:44:54ZBenoit OrihuelaImplement a specific comparison function for datesDates with milliseconds precision, but with millisecond value set to 0 (e.g., 2022-04-02T18:52:00.000Z) can be rendered as 2022-04-02T18:52:00.000Z or 2022-04-02T18:52:00Z. Both are correct and should be accepted when testing responses p...Dates with milliseconds precision, but with millisecond value set to 0 (e.g., 2022-04-02T18:52:00.000Z) can be rendered as 2022-04-02T18:52:00.000Z or 2022-04-02T18:52:00Z. Both are correct and should be accepted when testing responses payloads.
As DeepDiff is already used in the test suite, a custom operator as described in https://zepworks.com/deepdiff/current/custom.html may be a good fit.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/19requested for a test issue2023-03-06T12:26:40ZJosé Ignacio Carreterorequested for a test issueAccording to a message received, there's a need to issue.According to a message received, there's a need to issue.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/20Upgrade Robot Framework and other libs to their last version2023-05-04T08:35:58ZBenoit OrihuelaUpgrade Robot Framework and other libs to their last versionBonus: we are using two libs to send HTTP requests (but one does not accept to send invalid payloads...), check if we can get rid of one.Bonus: we are using two libs to send HTTP requests (but one does not accept to send invalid payloads...), check if we can get rid of one.T2 - Improve current Test Suite and testing environmentpoujolpoujolhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/21Duplicate keyword to get the Temporal Representation of an Entity2023-09-05T12:03:05ZpoujolDuplicate keyword to get the Temporal Representation of an EntityCurrently we have two functions to retrieve a temporal entity : _Get Temporal Representation of Entity_ and _Retrieve Temporal Representation of Entity_Currently we have two functions to retrieve a temporal entity : _Get Temporal Representation of Entity_ and _Retrieve Temporal Representation of Entity_T2 - Improve current Test Suite and testing environmentlopezaguilarlopezaguilarhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/22Clean up used libraries2023-06-01T12:36:43ZpoujolClean up used libraries- Remove unused librairies
- Have only one library for HTTP requests (remove RESTinstance Library)- Remove unused librairies
- Have only one library for HTTP requests (remove RESTinstance Library)T2 - Improve current Test Suite and testing environmentpoujolpoujol2023-05-19https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/23Standardizes the use of test and suite2023-06-01T12:38:01ZpoujolStandardizes the use of test and suiteHave a common behaviour. More details on best practices to use variables in [keywords](https://www.linkedin.com/advice/1/what-best-practices-using-variables-keywords-skills-robot-framework) and to write [test case](https://github.com/rob...Have a common behaviour. More details on best practices to use variables in [keywords](https://www.linkedin.com/advice/1/what-best-practices-using-variables-keywords-skills-robot-framework) and to write [test case](https://github.com/robotframework/HowToWriteGoodTestCases/blob/master/HowToWriteGoodTestCases.rst).T2 - Improve current Test Suite and testing environmentpoujolpoujol2023-06-02https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/24Always use arguments when using function2023-06-01T12:37:23ZpoujolAlways use arguments when using functionT2 - Improve current Test Suite and testing environmentpoujolpoujol2023-06-02https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/25Find a way to send a "reset" command when the test suite starts2023-08-08T06:06:35ZBenoit OrihuelaFind a way to send a "reset" command when the test suite startsQuoted from the meeting minutes:
```
Distributed testbed. Everybody runs their own brokers. Everybody needs to implement a “wipeout” service endpoint that takes care of resetting the environment anew for a new regression run.
We also ne...Quoted from the meeting minutes:
```
Distributed testbed. Everybody runs their own brokers. Everybody needs to implement a “wipeout” service endpoint that takes care of resetting the environment anew for a new regression run.
We also need to implement a same client for everybody that sends the wipeout command and notifies when it is done. Is this going to be part of an init() of the run of the tests?
```T2 - Improve current Test Suite and testing environmentBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/26Move the TCs in the Common Behaviors directory in a directory next to each co...2023-09-06T16:24:09ZBenoit OrihuelaMove the TCs in the Common Behaviors directory in a directory next to each concerned endpointTCs in Common Behaviors directory are making use of a lot of different endpoints:
- Harder to manage since a given endpoint is tested in different places in the test suite
- Harder to manage fixture data since a given one can be used fro...TCs in Common Behaviors directory are making use of a lot of different endpoints:
- Harder to manage since a given endpoint is tested in different places in the test suite
- Harder to manage fixture data since a given one can be used from the Common Behaviors and from the directory with TCs for the same endpointT2 - Improve current Test Suite and testing environmenthttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/27Tags are not found when set on keywords2023-09-07T15:52:36ZBenoit OrihuelaTags are not found when set on keywordsTest Cases making use of permutations have such a structure:
```
*** Test Cases *** FILENAME CONTENT_TYPE
001_01_01_MinimalEntity
building-minimal-without-context-sample.jsonld application/ld+json
001_01_02_EntityWithSimple...Test Cases making use of permutations have such a structure:
```
*** Test Cases *** FILENAME CONTENT_TYPE
001_01_01_MinimalEntity
building-minimal-without-context-sample.jsonld application/ld+json
001_01_02_EntityWithSimpleProperties
building-simple-attributes-sample.jsonld application/ld+json
001_01_03_EntityWithRelationshipsProperties
building-relationship-of-property-sample.jsonld application/ld+json
001_01_04_EntityWithLocationAttribute
building-location-attribute-sample.jsonld application/ld+json
*** Keywords ***
Create Entity Scenarios
[Documentation] Check that you can create an entity
[Tags] e-create 5_6_1
[Arguments] ${filename} ${content_type}
...
```
If the Test Suite is launched using the "include tags" option (for instance, `--include 5_6_1`), this Test Case will not be executed.
The solution is to move the declaration of the tags on each Test Case in the Test Cases section.T2 - Improve current Test Suite and testing environmentBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/28Split APIUtils.resource to reflect the NGSI-LD API Structure defined in 4.3.52023-09-05T12:03:39ZBenoit OrihuelaSplit APIUtils.resource to reflect the NGSI-LD API Structure defined in 4.3.5This file is getting big, clearer to split it.This file is getting big, clearer to split it.T2 - Improve current Test Suite and testing environmentlopezaguilarlopezaguilarhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/29Allow any core context in the response when doing a GET request without passi...2023-09-27T14:44:08ZBenoit OrihuelaAllow any core context in the response when doing a GET request without passing a contextcurrently, some tests are failing because, in this case, they are checking for the v1.3 core context and a context broker may respond with another core context, which is still a valid behavior.currently, some tests are failing because, in this case, they are checking for the v1.3 core context and a context broker may respond with another core context, which is still a valid behavior.T2 - Improve current Test Suite and testing environmentBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/30When returning a single context, allow a string and an array of one context2023-09-27T14:43:57ZBenoit OrihuelaWhen returning a single context, allow a string and an array of one contextT2 - Improve current Test Suite and testing environmentBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/31Update the TCs checking the "Id not present" behaviours2023-12-08T06:21:41ZBenoit OrihuelaUpdate the TCs checking the "Id not present" behaviours- Remove the ones that are not testable
- Adapt where it is testable but with other expectations (e.g., 405)- Remove the ones that are not testable
- Adapt where it is testable but with other expectations (e.g., 405)T2 - Improve current Test Suite and testing environmentBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/32Review and fix some incorrect expectations in responses2023-12-04T19:55:09ZBenoit OrihuelaReview and fix some incorrect expectations in responsesFollowing points discussed this morning in NGSI-LD API call (https://docbox.etsi.org/ISG/CIM/05-CONTRIBUTIONS/2023//CIM(23)000143_TTFv2_-_Clarifications_and_final_improvements.pptx):
- slide 4: 207
- slide 5: 204 (silently ignore)
- sli...Following points discussed this morning in NGSI-LD API call (https://docbox.etsi.org/ISG/CIM/05-CONTRIBUTIONS/2023//CIM(23)000143_TTFv2_-_Clarifications_and_final_improvements.pptx):
- slide 4: 207
- slide 5: 204 (silently ignore)
- slide 6: compacted (reason can convey more info, for instance the expanded form, to help users in understanding the problem)T2 - Improve current Test Suite and testing environmentBenoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/33Use of tags in Test Cases2023-10-10T10:27:38ZBenoit OrihuelaUse of tags in Test Cases- Some reference section 5, other reference a sub-section in Common behaviours (6.3.x) when testing a specific behaviour (e.g., JSON-LD @context resolution)
- Decision: reference both
- Decision: also reference appropriate resource d...- Some reference section 5, other reference a sub-section in Common behaviours (6.3.x) when testing a specific behaviour (e.g., JSON-LD @context resolution)
- Decision: reference both
- Decision: also reference appropriate resource definition in section 6
- Keep in mind TCs specific to a binding in the futureT2 - Improve current Test Suite and testing environmenthttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/34Refactor TCs to always use a setup step when some initial data has to be created2023-12-04T06:54:51ZBenoit OrihuelaRefactor TCs to always use a setup step when some initial data has to be created- Cleaner structure, will make TCs easier to read
- Will allow for parsing and generating the documentation- Cleaner structure, will make TCs easier to read
- Will allow for parsing and generating the documentationT2 - Improve current Test Suite and testing environmentlopezaguilarlopezaguilarhttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/35Add TCs for Aggregated Temporal Representation of an Entity (4.5.19)2024-03-23T09:53:52ZBenoit OrihuelaAdd TCs for Aggregated Temporal Representation of an Entity (4.5.19)T3a - New TCs for NGSI-LD API v1.4.1Benoit OrihuelaBenoit Orihuelahttps://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/36Add TCs for LanguageProperty support (4.5.18)2024-03-23T09:53:41ZBenoit OrihuelaAdd TCs for LanguageProperty support (4.5.18)T3a - New TCs for NGSI-LD API v1.4.1https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/37Align checks for ProblemDetails to the newly, more complete, introduced Keywords2024-01-05T16:52:15ZBenoit OrihuelaAlign checks for ProblemDetails to the newly, more complete, introduced KeywordsSee new Keywords definition and a sample use in a TC in https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/commit/a0f70bae8c716d9001dbe6400e0426d9ec95370a.
In particular, the new Keyword do more complete checks with respect to the RFC.See new Keywords definition and a sample use in a TC in https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/commit/a0f70bae8c716d9001dbe6400e0426d9ec95370a.
In particular, the new Keyword do more complete checks with respect to the RFC.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/38Update to RobotFramework 7.02024-02-13T16:57:05ZlopezaguilarUpdate to RobotFramework 7.0There are several important changes in the new version of robotframework that break the current execution of the automatic generation of the test.There are several important changes in the new version of robotframework that break the current execution of the automatic generation of the test.https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/issues/39Error when trying to generate the documentation2024-03-23T09:53:27ZBenoit OrihuelaError when trying to generate the documentationI wanted to update the generated documentation after adding new Test Cases and I got the following error:
```
ngsi-ld-test-suite on feature/tcs-post-1_4 [$!?] via 🐍 v3.11.7 (.venv)
➜ python3 doc/generateDocumentationData.py TP/NGSI-L...I wanted to update the generated documentation after adding new Test Cases and I got the following error:
```
ngsi-ld-test-suite on feature/tcs-post-1_4 [$!?] via 🐍 v3.11.7 (.venv)
➜ python3 doc/generateDocumentationData.py TP/NGSI-LD
Traceback (most recent call last):
File "/Users/bobeal/egm/dev/datahub/ngsi-ld-test-suite/doc/generateDocumentationData.py", line 58, in <module>
resulting_json = create_json_of_robotfile(robot_file_tbp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bobeal/egm/dev/datahub/ngsi-ld-test-suite/doc/generateDocumentationData.py", line 35, in create_json_of_robotfile
data = GenerateRobotData(robot_file=robot_file, execdir=folder_test_suites)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bobeal/egm/dev/datahub/ngsi-ld-test-suite/doc/analysis/generaterobotdata.py", line 14, in __init__
self.suite = TestSuiteBuilder().build(robot_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bobeal/egm/dev/datahub/ngsi-ld-test-suite/.venv/lib/python3.11/site-packages/robot/running/builder/builders.py", line 152, in build
paths = self._normalize_paths(paths)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bobeal/egm/dev/datahub/ngsi-ld-test-suite/.venv/lib/python3.11/site-packages/robot/running/builder/builders.py", line 170, in _normalize_paths
paths = [Path(normpath(p)).absolute() for p in paths]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bobeal/egm/dev/datahub/ngsi-ld-test-suite/.venv/lib/python3.11/site-packages/robot/running/builder/builders.py", line 170, in <listcomp>
paths = [Path(normpath(p)).absolute() for p in paths]
^^^^^^^^^^^
File "<frozen posixpath>", line 391, in normpath
TypeError: expected str, bytes or os.PathLike object, not NoneType
```T2 - Improve current Test Suite and testing environmentlopezaguilarlopezaguilar