MEC Robot Test Suite issueshttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues2020-06-08T13:09:56Zhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/21Schema file name is wrong2020-06-08T13:09:56ZjacobvarghesSchema file name is wrong```
TC_MEC_SRV_SRVSUB_002_OK :: Check that the IUT acknowledges the su...
{
...
...
}
| FAIL |
FileNotFoundError: [Errno 2] No such file or directory: 'schemas//SerAvailabilityNotificationSubscription.schema.json'
```
Script is accessi...```
TC_MEC_SRV_SRVSUB_002_OK :: Check that the IUT acknowledges the su...
{
...
...
}
| FAIL |
FileNotFoundError: [Errno 2] No such file or directory: 'schemas//SerAvailabilityNotificationSubscription.schema.json'
```
Script is accessing schema file named `SerAvailabilityNotificationSubscription.schema.json`, but in schema folder, file doesn't have the `schema` key in it. Please fix this by changing the file name to `SerAvailabilityNotificationSubscription.schema.json`.
![image](/uploads/5797f76a0a3605e5d4eb670f23f6c94b/image.png)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/22Delete trailing comma from json files2020-06-09T07:10:30ZFederico SismondiDelete trailing comma from json filesFound two errors in .json files (tried to push MR but apparently I dont have access to project)
See:
[trailing comma err 1](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/CellChangeSubscriptio...Found two errors in .json files (tried to push MR but apparently I dont have access to project)
See:
[trailing comma err 1](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/CellChangeSubscriptionRequest.json#L9)
[trailing comma err 2](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/UpdateCellChangeSubscriptionRequest.json#L9)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/23keyword missing trying to run `/SRV/RNIS/RnisNotifications.robot`2021-01-19T13:45:26ZFederico Sismondikeyword missing trying to run `/SRV/RNIS/RnisNotifications.robot`Using branch `2.1.1-fix-plu` (commit 5d7b0b754259ac7d2d12b032c670faf5bd73b99b)
I am finding this issue trying to run the robot test:
```
...
[20210105 19:43:46.646] LOG.FAIL - No keyword with name 'Create Mock Request Matcher' found.
....Using branch `2.1.1-fix-plu` (commit 5d7b0b754259ac7d2d12b032c670faf5bd73b99b)
I am finding this issue trying to run the robot test:
```
...
[20210105 19:43:46.646] LOG.FAIL - No keyword with name 'Create Mock Request Matcher' found.
...
```
am I launching the robot incorrectly or is this actually missing ?
I did a quick search on the code for the keyword and couldnt find anything.
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/24Handle optionality for ProblemDetails presence and value check2021-02-03T10:52:32ZMichele CarignaniHandle optionality for ProblemDetails presence and value checkThe `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `...The `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `ProblemDetails` data to always be present.
## Expected behaviour
Allow the response to contain string or empty body, as well as the `ProblemDetails` data.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/25Templates for Notification requests are not resolved2021-02-03T10:53:46ZMichele CarignaniTemplates for Notification requests are not resolvedThe request object for Notification creation contains variables that should be resolved and substituted with the values inserted in variables.txt.
This is currently not supported. As a result, the body of the request contains the variab...The request object for Notification creation contains variables that should be resolved and substituted with the values inserted in variables.txt.
This is currently not supported. As a result, the body of the request contains the variable name, not the variable value.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/26Plugtest 2021: Schemas not defined in few MEC013 test suites2021-02-10T18:21:21ZSimon PastorPlugtest 2021: Schemas not defined in few MEC013 test suitesLot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",...Lot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",
"description": "Json Schema for the given object is not yet defined"
}
},
"required": [
"undefinedJsonSchema"
]
}
This code obviously does not match the received json and tests are failing.
Below are the test suites and schemas impacted
Test Suite
RLOCLOOK -- AccessPointList.schema.json
UELOCSUB -- UserTrackingSubscription.schema.json
UEINFOSUB -- ZonalTrafficSubscription.schema.json
other non tested test suites have the same issue:
UEDISTSUB, UEAREASUB, UEDISTLOOK, UETRACKSUBhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/27Plugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2...2021-02-11T18:16:58ZSimon PastorPlugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2.1.1MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK ...MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK uses endpoint location/v2/zones, and not location/v2/query/zones
All schemas for every test suites seem to use not named object, although the spec seemed to show the use of inline (named) objects. This result in validation to be missing one level of hierarchy (does not match the content of the file LocationAPI describe in the header which uses inline objects in its responses)
Note: Note that UEINFOSUB uses values instead of arrays as specified v2.1.1 in its default editable fields (interestRealm, userCriteria)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/28Plugtest 2021: MEC012 RNISQuery BV RabInfo test error2022-03-03T09:14:09ZSimon PastorPlugtest 2021: MEC012 RNISQuery BV RabInfo test errorThe test case sends the following request with the wrong query parameter:
exampleapi/rni/v2/queries/rab_info?cell_id=0x800000A
but the test case meant to test the app_ins_id=${APP_INS_ID} and not the cell Id since it tries to validate o...The test case sends the following request with the wrong query parameter:
exampleapi/rni/v2/queries/rab_info?cell_id=0x800000A
but the test case meant to test the app_ins_id=${APP_INS_ID} and not the cell Id since it tries to validate on the app_ins_idhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/29Plugtest 2021: MEC012 RNIS RNISQuery BV L2Meas using wrong schema2021-02-11T18:23:08ZSimon PastorPlugtest 2021: MEC012 RNIS RNISQuery BV L2Meas using wrong schemaThe test case uses the schema Layer2MeasInfos.schema.json but the file that is defined in the repo of the test is L2MeasInfos.schema.jsonThe test case uses the schema Layer2MeasInfos.schema.json but the file that is defined in the repo of the test is L2MeasInfos.schema.jsonhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/30Plugtest 2021: MEC012 RnisSubscription BV test suites validation errors2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC012 RnisSubscription BV test suites validation errors1) In test case Request RNIS subscription list
Check with variables seem not to work as expected.
Default value in {LINKS_SELF} doesn't use the proper v2 schema (v2 should use the 'href', not the string directly). It can be changed man...1) In test case Request RNIS subscription list
Check with variables seem not to work as expected.
Default value in {LINKS_SELF} doesn't use the proper v2 schema (v2 should use the 'href', not the string directly). It can be changed manually, but validation fails because the object types are different (dictionary vs string):
{'href': 'http://3.84.255.29/simon/rni/v2/subscriptions'} (dictionary) != {'href': 'http://3.84.255.29/simon/rni/v2/subscriptions'} (string)
2) In test case Create RNIS subscriptions
schemas/cellChangeSubscriptionRequest.schema.json cannot be found so validation cannot occurhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/31Plugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concaten...2021-02-11T18:15:24ZSimon PastorPlugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concatenation with port errorWhen testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and...When testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and host name resulting in tap.plugtests.net8111 and tries to reach the server on port 80. This cannot be resolved and test fails:
ConnectionError:
HTTPConnectionPool(host='tap.plugtests.net8111', port=80): Max retries exceeded with url: /expectation (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1ce504ce80>: Failed to establish a new connection: [Errno -2] Name or service not known',))https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/32Plugtest 2021: MEC013: Schemas have non-valid characters preventing the valid...2022-03-03T09:14:09ZSimon PastorPlugtest 2021: MEC013: Schemas have non-valid characters preventing the validationSome schemas have invalid characters in them preventing the validation to go through.
Example of error:
15:52:11.452 TRACE Arguments: [ 'UserList.schema.json' | {'userList': {'resourceURL': 'http://3.84.255.29/simon/location/v2/queries/...Some schemas have invalid characters in them preventing the validation to go through.
Example of error:
15:52:11.452 TRACE Arguments: [ 'UserList.schema.json' | {'userList': {'resourceURL': 'http://3.84.255.29/simon/location/v2/queries/users'}} ]
**15:52:11.454 FAIL UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 3615: invalid start byte **
15:52:11.457 DEBUG Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/JSONSchemaLibrary/__init__.py", line 31, in validate_json
schema = json.loads(open('{}/{}'.format(self.schema_location, schema_filename)).read())
File "/usr/local/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
This error comes from the schema file position 3615 which has a non valid character:
**Location latitude, expressed in the range -90� to +90�. Cardinality greater than one only if \"shape\" equals 7.**
The error is not visible in the json file, but if looking at the file in the raw format... all there characters can be seen. They are replacing some sort of 'degree' symbol maybe or something else. The same thing has been seen when using apostophes.. so as:
**Structure with attributes relating to the target entity�s velocity, as defined in [14].**
The files below where seen having this problem:
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCLOOK/schemas/UserList.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCLOOK/schemas/UserInfo.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/RLOCLOOK/schemas/AccessPointInfo.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/RLOCLOOK/schemas/AccessPointList.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCSUB/schemas/NotificationSubscriptionList.schema.jsonhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/33Plugtest 2021: MEC013: UELOCLOOK using non valid endpoint2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC013: UELOCLOOK using non valid endpointUELOCLOOK is using endpoint v2/users
This is the only test serie that uses the v1 endpoint version. Others were updated. Needs to be updated for v2/queries/users.UELOCLOOK is using endpoint v2/users
This is the only test serie that uses the v1 endpoint version. Others were updated. Needs to be updated for v2/queries/users.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/34Plugtest 2021: MEC013: ClientCorrelator error in UELOCSUB2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC013: ClientCorrelator error in UELOCSUBTEST UELOCSUB 001 OK
There seem to be a check for an optional parameter, clientCorrelator, but it seems not defined somehow so it cannot look at the response and do the comparison. The parameter is an optional parameter, not sure if thi...TEST UELOCSUB 001 OK
There seem to be a check for an optional parameter, clientCorrelator, but it seems not defined somehow so it cannot look at the response and do the comparison. The parameter is an optional parameter, not sure if this is where the problem is, but cannot understand why this error pops out. It seem to be defined properly in the schema....
```
KEYWORD BuiltIn . Exit For Loop If ${item[${json_field}]} == ${json_value}
Documentation:
Stops executing the enclosing for loop if the condition is true.
Start / End / Elapsed: 20210210 16:49:11.827 / 20210210 16:49:11.831 / 00:00:00.004
16:49:11.828 **FAIL Resolving variable '${item[clientCorrelator]}' failed: NameError: name 'clientCorrelator' is not defined **
16:49:11.826 TRACE Arguments: [ ${expr}={'address': 'acr:10.0.0.1', 'callbackReference': {'notifyURL': 'http://clientApp.example.com/location_notifications/123456'}, 'clientCorrelator': '0125', 'resourceURL': 'http://ourserver/simon/location/v2/subscriptions/userTracking/4', 'userEventCriteria': ['Transferring']} | ${json_field}='clientCorrelator' | ${json_value}='0125' ]
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/35Plugtest 2021: Spec discussion : Validity of returning a list with empty para...2022-03-03T09:14:07ZSimon PastorPlugtest 2021: Spec discussion : Validity of returning a list with empty parameters rather than code 404In many instances, when querying about an endpoint.. for example: query/users, or query/zones
If no users are found, should the error be a NOT FOUND 404, or should it be a userList populating the resourceURL but having no array of users...In many instances, when querying about an endpoint.. for example: query/users, or query/zones
If no users are found, should the error be a NOT FOUND 404, or should it be a userList populating the resourceURL but having no array of users and returning a code 200 (success). Our implementation is of the later because the request was successful on the queried endpoint, but the list is empty. This issue is just to track a discussion on how to handle this situation.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/36Plugtest 2021: MEC012: RNISQuery BV L2Meas info should not be an array2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC012: RNISQuery BV L2Meas info should not be an arrayIn RNISQuery BV
The response to a query/layer2_meas should be a L2Meas schema object, not a L2MeasInfos as per section 7.5a of the standard.In RNISQuery BV
The response to a query/layer2_meas should be a L2Meas schema object, not a L2MeasInfos as per section 7.5a of the standard.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/37Plugtest 2021: MEC012: Rnis Query BI BO RabInfo and PlmnInfo using wrong para...2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC012: Rnis Query BI BO RabInfo and PlmnInfo using wrong parameter errors in the URL pathSpec is not specifying what to expect. When an endpoint does not exist, what should the server return. Test plan seems to expect 400, but implementation is 404. Need a discussion to discuss if 400 or 404 should be selected for an endpoin...Spec is not specifying what to expect. When an endpoint does not exist, what should the server return. Test plan seems to expect 400, but implementation is 404. Need a discussion to discuss if 400 or 404 should be selected for an endpoint that cannot be resolved.
But I think this is a test case error because url is wrong and cannot be resolved (ex: exampleAPI/rni/v2/queries/rab_info)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/38Plugtest 2021: Specification discussion: non existent attribute in a json bod...2021-02-26T20:19:05ZSimon PastorPlugtest 2021: Specification discussion: non existent attribute in a json body should be ignoredUsed as a placeholder for the specific issue below:
When receiving a json body and an attribute is not defined, the current expectation from the plugtest is to receive a code 400. Is a more appropriate action to ignore the unknown param...Used as a placeholder for the specific issue below:
When receiving a json body and an attribute is not defined, the current expectation from the plugtest is to receive a code 400. Is a more appropriate action to ignore the unknown parameter? This allows a less strict approach. It was not found in the spec what should be the behavior as a result of this situation. Need a clearer spec indication in order to standardized the implementations.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/39Plugtest 2021: MEC012: RNIS BI-BO: Error code expected 404 for L2Meas with wr...2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC012: RNIS BI-BO: Error code expected 404 for L2Meas with wrong query paramQuery with wrong parameter is rejected with error code 400, but plugtest is expecting error 404. The test is made in a way that it checks for a mandatory ProblemDetails if error code is not 404.. so this is also an error ... but all comi...Query with wrong parameter is rejected with error code 400, but plugtest is expecting error 404. The test is made in a way that it checks for a mandatory ProblemDetails if error code is not 404.. so this is also an error ... but all coming from the comparison of the initial error code.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/40Plugtest 2021: MEC013: UELOCLOOK_001_OK not findind a defined parameter2021-02-19T18:32:28ZSimon PastorPlugtest 2021: MEC013: UELOCLOOK_001_OK not findind a defined parameterWhen doing this test case... seeing the following:
Resolving variable `'${item[zoneId]}' failed: NameError: name 'zoneId' is not defined`
The named parameter above is defined in the answer received by the test tool.
Not sure how the tes...When doing this test case... seeing the following:
Resolving variable `'${item[zoneId]}' failed: NameError: name 'zoneId' is not defined`
The named parameter above is defined in the answer received by the test tool.
Not sure how the test validation is working but user is an array and that may be what brings an issue, although as i mentioned earlier, i am not aware of the way the validation is working.
For example.. here is the actual response:
```
"userList": {
"resourceURL": "http://myIp/mySandbox/location/v2/queries/users",
---> "user": [
{
"accessPointId": "4g-macro-cell-1",
"address": "10.10.0.3",
"locationInfo": {
"latitude": [
43.72835
],
"longitude": [
7.421479
],
"shape": 2,
"timestamp": {
"nanoSeconds": 0,
"seconds": 1613580011
}
},
"resourceURL": "http://myIp/mySandbox/simon/location/v2/queries/users?address=10.10.0.3",
"timestamp": {
"nanoSeconds": 0,
"seconds": 1613580011
},
---> "zoneId": "zone01"
},
```