MEC Robot Test Suite issueshttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues2021-02-03T10:52:32Zhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/24Handle optionality for ProblemDetails presence and value check2021-02-03T10:52:32ZMichele CarignaniHandle optionality for ProblemDetails presence and value checkThe `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `...The `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `ProblemDetails` data to always be present.
## Expected behaviour
Allow the response to contain string or empty body, as well as the `ProblemDetails` data.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/26Plugtest 2021: Schemas not defined in few MEC013 test suites2021-02-10T18:21:21ZSimon PastorPlugtest 2021: Schemas not defined in few MEC013 test suitesLot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",...Lot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",
"description": "Json Schema for the given object is not yet defined"
}
},
"required": [
"undefinedJsonSchema"
]
}
This code obviously does not match the received json and tests are failing.
Below are the test suites and schemas impacted
Test Suite
RLOCLOOK -- AccessPointList.schema.json
UELOCSUB -- UserTrackingSubscription.schema.json
UEINFOSUB -- ZonalTrafficSubscription.schema.json
other non tested test suites have the same issue:
UEDISTSUB, UEAREASUB, UEDISTLOOK, UETRACKSUBhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/31Plugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concaten...2021-02-11T18:15:24ZSimon PastorPlugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concatenation with port errorWhen testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and...When testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and host name resulting in tap.plugtests.net8111 and tries to reach the server on port 80. This cannot be resolved and test fails:
ConnectionError:
HTTPConnectionPool(host='tap.plugtests.net8111', port=80): Max retries exceeded with url: /expectation (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1ce504ce80>: Failed to establish a new connection: [Errno -2] Name or service not known',))https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/27Plugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2...2021-02-11T18:16:58ZSimon PastorPlugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2.1.1MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK ...MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK uses endpoint location/v2/zones, and not location/v2/query/zones
All schemas for every test suites seem to use not named object, although the spec seemed to show the use of inline (named) objects. This result in validation to be missing one level of hierarchy (does not match the content of the file LocationAPI describe in the header which uses inline objects in its responses)
Note: Note that UEINFOSUB uses values instead of arrays as specified v2.1.1 in its default editable fields (interestRealm, userCriteria)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/29Plugtest 2021: MEC012 RNIS RNISQuery BV L2Meas using wrong schema2021-02-11T18:23:08ZSimon PastorPlugtest 2021: MEC012 RNIS RNISQuery BV L2Meas using wrong schemaThe test case uses the schema Layer2MeasInfos.schema.json but the file that is defined in the repo of the test is L2MeasInfos.schema.jsonThe test case uses the schema Layer2MeasInfos.schema.json but the file that is defined in the repo of the test is L2MeasInfos.schema.jsonhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/40Plugtest 2021: MEC013: UELOCLOOK_001_OK not findind a defined parameter2021-02-19T18:32:28ZSimon PastorPlugtest 2021: MEC013: UELOCLOOK_001_OK not findind a defined parameterWhen doing this test case... seeing the following:
Resolving variable `'${item[zoneId]}' failed: NameError: name 'zoneId' is not defined`
The named parameter above is defined in the answer received by the test tool.
Not sure how the tes...When doing this test case... seeing the following:
Resolving variable `'${item[zoneId]}' failed: NameError: name 'zoneId' is not defined`
The named parameter above is defined in the answer received by the test tool.
Not sure how the test validation is working but user is an array and that may be what brings an issue, although as i mentioned earlier, i am not aware of the way the validation is working.
For example.. here is the actual response:
```
"userList": {
"resourceURL": "http://myIp/mySandbox/location/v2/queries/users",
---> "user": [
{
"accessPointId": "4g-macro-cell-1",
"address": "10.10.0.3",
"locationInfo": {
"latitude": [
43.72835
],
"longitude": [
7.421479
],
"shape": 2,
"timestamp": {
"nanoSeconds": 0,
"seconds": 1613580011
}
},
"resourceURL": "http://myIp/mySandbox/simon/location/v2/queries/users?address=10.10.0.3",
"timestamp": {
"nanoSeconds": 0,
"seconds": 1613580011
},
---> "zoneId": "zone01"
},
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/42Failure running TC_MEC_MEC013_SRV_UEDISTSUB_001_OK2021-10-07T16:41:44ZElian KrajaFailure running TC_MEC_MEC013_SRV_UEDISTSUB_001_OK**Should Be Present In Json List** keywords fails when executed.
Error: ```Resolving variable '${item['clientCorrelator']}' failed: TypeError: string indices must be integers```**Should Be Present In Json List** keywords fails when executed.
Error: ```Resolving variable '${item['clientCorrelator']}' failed: TypeError: string indices must be integers```Elian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/44MEC028 - Measurements endpoint - missing brackets2021-10-08T14:08:11ZElian KrajaMEC028 - Measurements endpoint - missing brackets```Should Be Equal As Strings ${response['body']['measurementId'] ${MEAS_ID}``` is missing closure brackets```Should Be Equal As Strings ${response['body']['measurementId'] ${MEAS_ID}``` is missing closure bracketsElian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/43MEC028 - Measurements endpoint using wrong rest methods2021-10-08T14:08:19ZElian KrajaMEC028 - Measurements endpoint using wrong rest methods"Update a specified measurement configuration" and "Delete a specified measurement configuration" are using GET method instead of, respectively, PUT and DELETE"Update a specified measurement configuration" and "Delete a specified measurement configuration" are using GET method instead of, respectively, PUT and DELETEElian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/45TP_MEC_MEC011_SRV_APPSUB_002_BR: Testcase is expecting wrong response code2021-10-15T15:34:41ZElian KrajaTP_MEC_MEC011_SRV_APPSUB_002_BR: Testcase is expecting wrong response codeElian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/35Plugtest 2021: Spec discussion : Validity of returning a list with empty para...2022-03-03T09:14:07ZSimon PastorPlugtest 2021: Spec discussion : Validity of returning a list with empty parameters rather than code 404In many instances, when querying about an endpoint.. for example: query/users, or query/zones
If no users are found, should the error be a NOT FOUND 404, or should it be a userList populating the resourceURL but having no array of users...In many instances, when querying about an endpoint.. for example: query/users, or query/zones
If no users are found, should the error be a NOT FOUND 404, or should it be a userList populating the resourceURL but having no array of users and returning a code 200 (success). Our implementation is of the later because the request was successful on the queried endpoint, but the list is empty. This issue is just to track a discussion on how to handle this situation.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/37Plugtest 2021: MEC012: Rnis Query BI BO RabInfo and PlmnInfo using wrong para...2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC012: Rnis Query BI BO RabInfo and PlmnInfo using wrong parameter errors in the URL pathSpec is not specifying what to expect. When an endpoint does not exist, what should the server return. Test plan seems to expect 400, but implementation is 404. Need a discussion to discuss if 400 or 404 should be selected for an endpoin...Spec is not specifying what to expect. When an endpoint does not exist, what should the server return. Test plan seems to expect 400, but implementation is 404. Need a discussion to discuss if 400 or 404 should be selected for an endpoint that cannot be resolved.
But I think this is a test case error because url is wrong and cannot be resolved (ex: exampleAPI/rni/v2/queries/rab_info)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/34Plugtest 2021: MEC013: ClientCorrelator error in UELOCSUB2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC013: ClientCorrelator error in UELOCSUBTEST UELOCSUB 001 OK
There seem to be a check for an optional parameter, clientCorrelator, but it seems not defined somehow so it cannot look at the response and do the comparison. The parameter is an optional parameter, not sure if thi...TEST UELOCSUB 001 OK
There seem to be a check for an optional parameter, clientCorrelator, but it seems not defined somehow so it cannot look at the response and do the comparison. The parameter is an optional parameter, not sure if this is where the problem is, but cannot understand why this error pops out. It seem to be defined properly in the schema....
```
KEYWORD BuiltIn . Exit For Loop If ${item[${json_field}]} == ${json_value}
Documentation:
Stops executing the enclosing for loop if the condition is true.
Start / End / Elapsed: 20210210 16:49:11.827 / 20210210 16:49:11.831 / 00:00:00.004
16:49:11.828 **FAIL Resolving variable '${item[clientCorrelator]}' failed: NameError: name 'clientCorrelator' is not defined **
16:49:11.826 TRACE Arguments: [ ${expr}={'address': 'acr:10.0.0.1', 'callbackReference': {'notifyURL': 'http://clientApp.example.com/location_notifications/123456'}, 'clientCorrelator': '0125', 'resourceURL': 'http://ourserver/simon/location/v2/subscriptions/userTracking/4', 'userEventCriteria': ['Transferring']} | ${json_field}='clientCorrelator' | ${json_value}='0125' ]
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/36Plugtest 2021: MEC012: RNISQuery BV L2Meas info should not be an array2022-03-03T09:14:07ZSimon PastorPlugtest 2021: MEC012: RNISQuery BV L2Meas info should not be an arrayIn RNISQuery BV
The response to a query/layer2_meas should be a L2Meas schema object, not a L2MeasInfos as per section 7.5a of the standard.In RNISQuery BV
The response to a query/layer2_meas should be a L2Meas schema object, not a L2MeasInfos as per section 7.5a of the standard.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/33Plugtest 2021: MEC013: UELOCLOOK using non valid endpoint2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC013: UELOCLOOK using non valid endpointUELOCLOOK is using endpoint v2/users
This is the only test serie that uses the v1 endpoint version. Others were updated. Needs to be updated for v2/queries/users.UELOCLOOK is using endpoint v2/users
This is the only test serie that uses the v1 endpoint version. Others were updated. Needs to be updated for v2/queries/users.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/39Plugtest 2021: MEC012: RNIS BI-BO: Error code expected 404 for L2Meas with wr...2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC012: RNIS BI-BO: Error code expected 404 for L2Meas with wrong query paramQuery with wrong parameter is rejected with error code 400, but plugtest is expecting error 404. The test is made in a way that it checks for a mandatory ProblemDetails if error code is not 404.. so this is also an error ... but all comi...Query with wrong parameter is rejected with error code 400, but plugtest is expecting error 404. The test is made in a way that it checks for a mandatory ProblemDetails if error code is not 404.. so this is also an error ... but all coming from the comparison of the initial error code.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/30Plugtest 2021: MEC012 RnisSubscription BV test suites validation errors2022-03-03T09:14:08ZSimon PastorPlugtest 2021: MEC012 RnisSubscription BV test suites validation errors1) In test case Request RNIS subscription list
Check with variables seem not to work as expected.
Default value in {LINKS_SELF} doesn't use the proper v2 schema (v2 should use the 'href', not the string directly). It can be changed man...1) In test case Request RNIS subscription list
Check with variables seem not to work as expected.
Default value in {LINKS_SELF} doesn't use the proper v2 schema (v2 should use the 'href', not the string directly). It can be changed manually, but validation fails because the object types are different (dictionary vs string):
{'href': 'http://3.84.255.29/simon/rni/v2/subscriptions'} (dictionary) != {'href': 'http://3.84.255.29/simon/rni/v2/subscriptions'} (string)
2) In test case Create RNIS subscriptions
schemas/cellChangeSubscriptionRequest.schema.json cannot be found so validation cannot occurhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/32Plugtest 2021: MEC013: Schemas have non-valid characters preventing the valid...2022-03-03T09:14:09ZSimon PastorPlugtest 2021: MEC013: Schemas have non-valid characters preventing the validationSome schemas have invalid characters in them preventing the validation to go through.
Example of error:
15:52:11.452 TRACE Arguments: [ 'UserList.schema.json' | {'userList': {'resourceURL': 'http://3.84.255.29/simon/location/v2/queries/...Some schemas have invalid characters in them preventing the validation to go through.
Example of error:
15:52:11.452 TRACE Arguments: [ 'UserList.schema.json' | {'userList': {'resourceURL': 'http://3.84.255.29/simon/location/v2/queries/users'}} ]
**15:52:11.454 FAIL UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 3615: invalid start byte **
15:52:11.457 DEBUG Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/JSONSchemaLibrary/__init__.py", line 31, in validate_json
schema = json.loads(open('{}/{}'.format(self.schema_location, schema_filename)).read())
File "/usr/local/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
This error comes from the schema file position 3615 which has a non valid character:
**Location latitude, expressed in the range -90� to +90�. Cardinality greater than one only if \"shape\" equals 7.**
The error is not visible in the json file, but if looking at the file in the raw format... all there characters can be seen. They are replacing some sort of 'degree' symbol maybe or something else. The same thing has been seen when using apostophes.. so as:
**Structure with attributes relating to the target entity�s velocity, as defined in [14].**
The files below where seen having this problem:
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCLOOK/schemas/UserList.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCLOOK/schemas/UserInfo.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/RLOCLOOK/schemas/AccessPointInfo.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/RLOCLOOK/schemas/AccessPointList.schema.json
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/raw/2.1.1-fix-plu/SRV/UELOCSUB/schemas/NotificationSubscriptionList.schema.jsonhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/28Plugtest 2021: MEC012 RNISQuery BV RabInfo test error2022-03-03T09:14:09ZSimon PastorPlugtest 2021: MEC012 RNISQuery BV RabInfo test errorThe test case sends the following request with the wrong query parameter:
exampleapi/rni/v2/queries/rab_info?cell_id=0x800000A
but the test case meant to test the app_ins_id=${APP_INS_ID} and not the cell Id since it tries to validate o...The test case sends the following request with the wrong query parameter:
exampleapi/rni/v2/queries/rab_info?cell_id=0x800000A
but the test case meant to test the app_ins_id=${APP_INS_ID} and not the cell Id since it tries to validate on the app_ins_id