MEC Robot Test Suite issueshttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues2022-12-14T14:43:06Zhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/48MEC013_SRV_UELOCSUB_001_OK - ${USERTRACK_NOTIF_CALLBACK_URI} incorrect format2022-12-14T14:43:06ZLaurent VelezMEC013_SRV_UELOCSUB_001_OK - ${USERTRACK_NOTIF_CALLBACK_URI} incorrect formatIn the TC MEC013_SRV_UELOCSUB_001_OK , the The callback reference is compared to the value in the config variable file.
```robot_framework
# Check Result Contains ${response['body']['userTrackingSubscription']} clientCorrelator ...In the TC MEC013_SRV_UELOCSUB_001_OK , the The callback reference is compared to the value in the config variable file.
```robot_framework
# Check Result Contains ${response['body']['userTrackingSubscription']} clientCorrelator ${USERTRACKSUB_CLIENT_ID}
Should Be Equal As Strings ${response['body']['userTrackingSubscription']['callbackReference']} ${USERTRACK_NOTIF_CALLBACK_URI}
```
The robot is expecting value with simple quote but the value of the example in the variables.txt file is with double quote :
`${USERTRACK_NOTIF_CALLBACK_URI} {"notifyURL": "http://clientApp.example.com/location_notifications/123456"}`
The config variables.txt file should be corrected.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/47MEC013_SRV_UEINFOLOOK_001_NF - Incorrect response status code2022-12-14T14:28:42ZLaurent VelezMEC013_SRV_UEINFOLOOK_001_NF - Incorrect response status codeIn the TC MEC013_SRV_UEINFOLOOK_001_NF, the Test System expects to get a 200 OK response to request `for an unknown URI` .
According to the TP and the specification, the IUT must replies with status code "404 Not Found" .In the TC MEC013_SRV_UEINFOLOOK_001_NF, the Test System expects to get a 200 OK response to request `for an unknown URI` .
According to the TP and the specification, the IUT must replies with status code "404 Not Found" .https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/46MEC-013/SRV/UELOCSUB userTrackingSubscription.json file has an incorrect attr...2022-12-15T15:23:42ZLaurent VelezMEC-013/SRV/UELOCSUB userTrackingSubscription.json file has an incorrect attribute formatIn the json file userTrackingSubscription.json, the attribute `userEventCriteria` is an enum and therfore must be between square brackets [], as follows:
{
"userTrackingSubscription": {
"clientCorrelator": "0123",
"callbackReferenc...In the json file userTrackingSubscription.json, the attribute `userEventCriteria` is an enum and therfore must be between square brackets [], as follows:
{
"userTrackingSubscription": {
"clientCorrelator": "0123",
"callbackReference": {
"notifyURL": "http://clientApp.example.com/location_notifications/123456"
},
"address": "acr:10.0.0.1",
"userEventCriteria": ["Transferring"]
}
}https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/45TP_MEC_MEC011_SRV_APPSUB_002_BR: Testcase is expecting wrong response code2021-10-15T15:34:41ZElian KrajaTP_MEC_MEC011_SRV_APPSUB_002_BR: Testcase is expecting wrong response codeElian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/44MEC028 - Measurements endpoint - missing brackets2021-10-08T14:08:11ZElian KrajaMEC028 - Measurements endpoint - missing brackets```Should Be Equal As Strings ${response['body']['measurementId'] ${MEAS_ID}``` is missing closure brackets```Should Be Equal As Strings ${response['body']['measurementId'] ${MEAS_ID}``` is missing closure bracketsElian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/43MEC028 - Measurements endpoint using wrong rest methods2021-10-08T14:08:19ZElian KrajaMEC028 - Measurements endpoint using wrong rest methods"Update a specified measurement configuration" and "Delete a specified measurement configuration" are using GET method instead of, respectively, PUT and DELETE"Update a specified measurement configuration" and "Delete a specified measurement configuration" are using GET method instead of, respectively, PUT and DELETEElian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/42Failure running TC_MEC_MEC013_SRV_UEDISTSUB_001_OK2021-10-07T16:41:44ZElian KrajaFailure running TC_MEC_MEC013_SRV_UEDISTSUB_001_OK**Should Be Present In Json List** keywords fails when executed.
Error: ```Resolving variable '${item['clientCorrelator']}' failed: TypeError: string indices must be integers```**Should Be Present In Json List** keywords fails when executed.
Error: ```Resolving variable '${item['clientCorrelator']}' failed: TypeError: string indices must be integers```Elian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/41Review the TPs and TCs on MEC 0132021-02-18T17:20:02ZMichele CarignaniReview the TPs and TCs on MEC 013https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/40Plugtest 2021: MEC013: UELOCLOOK_001_OK not findind a defined parameter2021-02-19T18:32:28ZSimon PastorPlugtest 2021: MEC013: UELOCLOOK_001_OK not findind a defined parameterWhen doing this test case... seeing the following:
Resolving variable `'${item[zoneId]}' failed: NameError: name 'zoneId' is not defined`
The named parameter above is defined in the answer received by the test tool.
Not sure how the tes...When doing this test case... seeing the following:
Resolving variable `'${item[zoneId]}' failed: NameError: name 'zoneId' is not defined`
The named parameter above is defined in the answer received by the test tool.
Not sure how the test validation is working but user is an array and that may be what brings an issue, although as i mentioned earlier, i am not aware of the way the validation is working.
For example.. here is the actual response:
```
"userList": {
"resourceURL": "http://myIp/mySandbox/location/v2/queries/users",
---> "user": [
{
"accessPointId": "4g-macro-cell-1",
"address": "10.10.0.3",
"locationInfo": {
"latitude": [
43.72835
],
"longitude": [
7.421479
],
"shape": 2,
"timestamp": {
"nanoSeconds": 0,
"seconds": 1613580011
}
},
"resourceURL": "http://myIp/mySandbox/simon/location/v2/queries/users?address=10.10.0.3",
"timestamp": {
"nanoSeconds": 0,
"seconds": 1613580011
},
---> "zoneId": "zone01"
},
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/39Plugtest 2021: MEC012: RNIS BI-BO: Error code expected 404 for L2Meas with wr...2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC012: RNIS BI-BO: Error code expected 404 for L2Meas with wrong query paramQuery with wrong parameter is rejected with error code 400, but plugtest is expecting error 404. The test is made in a way that it checks for a mandatory ProblemDetails if error code is not 404.. so this is also an error ... but all comi...Query with wrong parameter is rejected with error code 400, but plugtest is expecting error 404. The test is made in a way that it checks for a mandatory ProblemDetails if error code is not 404.. so this is also an error ... but all coming from the comparison of the initial error code.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/38Plugtest 2021: Specification discussion: non existent attribute in a json bod...2021-02-26T20:19:05ZSimon PastorPlugtest 2021: Specification discussion: non existent attribute in a json body should be ignoredUsed as a placeholder for the specific issue below:
When receiving a json body and an attribute is not defined, the current expectation from the plugtest is to receive a code 400. Is a more appropriate action to ignore the unknown param...Used as a placeholder for the specific issue below:
When receiving a json body and an attribute is not defined, the current expectation from the plugtest is to receive a code 400. Is a more appropriate action to ignore the unknown parameter? This allows a less strict approach. It was not found in the spec what should be the behavior as a result of this situation. Need a clearer spec indication in order to standardized the implementations.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/37Plugtest 2021: MEC012: Rnis Query BI BO RabInfo and PlmnInfo using wrong para...2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC012: Rnis Query BI BO RabInfo and PlmnInfo using wrong parameter errors in the URL pathSpec is not specifying what to expect. When an endpoint does not exist, what should the server return. Test plan seems to expect 400, but implementation is 404. Need a discussion to discuss if 400 or 404 should be selected for an endpoin...Spec is not specifying what to expect. When an endpoint does not exist, what should the server return. Test plan seems to expect 400, but implementation is 404. Need a discussion to discuss if 400 or 404 should be selected for an endpoint that cannot be resolved.
But I think this is a test case error because url is wrong and cannot be resolved (ex: exampleAPI/rni/v2/queries/rab_info)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/36Plugtest 2021: MEC012: RNISQuery BV L2Meas info should not be an array2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC012: RNISQuery BV L2Meas info should not be an arrayIn RNISQuery BV
The response to a query/layer2_meas should be a L2Meas schema object, not a L2MeasInfos as per section 7.5a of the standard.In RNISQuery BV
The response to a query/layer2_meas should be a L2Meas schema object, not a L2MeasInfos as per section 7.5a of the standard.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/35Plugtest 2021: Spec discussion : Validity of returning a list with empty para...2022-03-03T09:14:07ZSimon PastorPlugtest 2021: Spec discussion : Validity of returning a list with empty parameters rather than code 404In many instances, when querying about an endpoint.. for example: query/users, or query/zones
If no users are found, should the error be a NOT FOUND 404, or should it be a userList populating the resourceURL but having no array of users...In many instances, when querying about an endpoint.. for example: query/users, or query/zones
If no users are found, should the error be a NOT FOUND 404, or should it be a userList populating the resourceURL but having no array of users and returning a code 200 (success). Our implementation is of the later because the request was successful on the queried endpoint, but the list is empty. This issue is just to track a discussion on how to handle this situation.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/34Plugtest 2021: MEC013: ClientCorrelator error in UELOCSUB2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC013: ClientCorrelator error in UELOCSUBTEST UELOCSUB 001 OK
There seem to be a check for an optional parameter, clientCorrelator, but it seems not defined somehow so it cannot look at the response and do the comparison. The parameter is an optional parameter, not sure if thi...TEST UELOCSUB 001 OK
There seem to be a check for an optional parameter, clientCorrelator, but it seems not defined somehow so it cannot look at the response and do the comparison. The parameter is an optional parameter, not sure if this is where the problem is, but cannot understand why this error pops out. It seem to be defined properly in the schema....
```
KEYWORD BuiltIn . Exit For Loop If ${item[${json_field}]} == ${json_value}
Documentation:
Stops executing the enclosing for loop if the condition is true.
Start / End / Elapsed: 20210210 16:49:11.827 / 20210210 16:49:11.831 / 00:00:00.004
16:49:11.828 **FAIL Resolving variable '${item[clientCorrelator]}' failed: NameError: name 'clientCorrelator' is not defined **
16:49:11.826 TRACE Arguments: [ ${expr}={'address': 'acr:10.0.0.1', 'callbackReference': {'notifyURL': 'http://clientApp.example.com/location_notifications/123456'}, 'clientCorrelator': '0125', 'resourceURL': 'http://ourserver/simon/location/v2/subscriptions/userTracking/4', 'userEventCriteria': ['Transferring']} | ${json_field}='clientCorrelator' | ${json_value}='0125' ]
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/33Plugtest 2021: MEC013: UELOCLOOK using non valid endpoint2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC013: UELOCLOOK using non valid endpointUELOCLOOK is using endpoint v2/users
This is the only test serie that uses the v1 endpoint version. Others were updated. Needs to be updated for v2/queries/users.UELOCLOOK is using endpoint v2/users
This is the only test serie that uses the v1 endpoint version. Others were updated. Needs to be updated for v2/queries/users.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/32Plugtest 2021: MEC013: Schemas have non-valid characters preventing the valid...2022-03-03T09:14:09ZSimon PastorPlugtest 2021: MEC013: Schemas have non-valid characters preventing the validationSome schemas have invalid characters in them preventing the validation to go through.
Example of error:
15:52:11.452 TRACE Arguments: [ 'UserList.schema.json' | {'userList': {'resourceURL': 'http://3.84.255.29/simon/location/v2/queries/...Some schemas have invalid characters in them preventing the validation to go through.
Example of error:
15:52:11.452 TRACE Arguments: [ 'UserList.schema.json' | {'userList': {'resourceURL': 'http://3.84.255.29/simon/location/v2/queries/users'}} ]
**15:52:11.454 FAIL UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 3615: invalid start byte **
15:52:11.457 DEBUG Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/JSONSchemaLibrary/__init__.py", line 31, in validate_json
schema = json.loads(open('{}/{}'.format(self.schema_location, schema_filename)).read())
File "/usr/local/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
This error comes from the schema file position 3615 which has a non valid character:
**Location latitude, expressed in the range -90� to +90�. Cardinality greater than one only if \"shape\" equals 7.**
The error is not visible in the json file, but if looking at the file in the raw format... all there characters can be seen. They are replacing some sort of 'degree' symbol maybe or something else. The same thing has been seen when using apostophes.. so as:
**Structure with attributes relating to the target entity�s velocity, as defined in [14].**
The files below where seen having this problem:
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCLOOK/schemas/UserList.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCLOOK/schemas/UserInfo.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/RLOCLOOK/schemas/AccessPointInfo.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/RLOCLOOK/schemas/AccessPointList.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCSUB/schemas/NotificationSubscriptionList.schema.jsonhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/31Plugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concaten...2021-02-11T18:15:24ZSimon PastorPlugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concatenation with port errorWhen testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and...When testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and host name resulting in tap.plugtests.net8111 and tries to reach the server on port 80. This cannot be resolved and test fails:
ConnectionError:
HTTPConnectionPool(host='tap.plugtests.net8111', port=80): Max retries exceeded with url: /expectation (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1ce504ce80>: Failed to establish a new connection: [Errno -2] Name or service not known',))https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/30Plugtest 2021: MEC012 RnisSubscription BV test suites validation errors2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC012 RnisSubscription BV test suites validation errors1) In test case Request RNIS subscription list
Check with variables seem not to work as expected.
Default value in {LINKS_SELF} doesn't use the proper v2 schema (v2 should use the 'href', not the string directly). It can be changed man...1) In test case Request RNIS subscription list
Check with variables seem not to work as expected.
Default value in {LINKS_SELF} doesn't use the proper v2 schema (v2 should use the 'href', not the string directly). It can be changed manually, but validation fails because the object types are different (dictionary vs string):
{'href': 'http://3.84.255.29/simon/rni/v2/subscriptions'} (dictionary) != {'href': 'http://3.84.255.29/simon/rni/v2/subscriptions'} (string)
2) In test case Create RNIS subscriptions
schemas/cellChangeSubscriptionRequest.schema.json cannot be found so validation cannot occurhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/29Plugtest 2021: MEC012 RNIS RNISQuery BV L2Meas using wrong schema2021-02-11T18:23:08ZSimon PastorPlugtest 2021: MEC012 RNIS RNISQuery BV L2Meas using wrong schemaThe test case uses the schema Layer2MeasInfos.schema.json but the file that is defined in the repo of the test is L2MeasInfos.schema.jsonThe test case uses the schema Layer2MeasInfos.schema.json but the file that is defined in the repo of the test is L2MeasInfos.schema.json