MEC Robot Test Suite issueshttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues2021-02-11T18:16:58Zhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/27Plugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2...2021-02-11T18:16:58ZSimon PastorPlugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2.1.1MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK ...MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK uses endpoint location/v2/zones, and not location/v2/query/zones
All schemas for every test suites seem to use not named object, although the spec seemed to show the use of inline (named) objects. This result in validation to be missing one level of hierarchy (does not match the content of the file LocationAPI describe in the header which uses inline objects in its responses)
Note: Note that UEINFOSUB uses values instead of arrays as specified v2.1.1 in its default editable fields (interestRealm, userCriteria)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/31Plugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concaten...2021-02-11T18:15:24ZSimon PastorPlugtest 2021: MEC012 RNISNotifications HTTPConnectionPool host name concatenation with port errorWhen testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and...When testing any notification as part of RNISNotifications test suite, the test does not connect properly to the tap plugtest server.
Rather than using the tap.plugtests.net server on port 8111... it uses a concatenation of the port and host name resulting in tap.plugtests.net8111 and tries to reach the server on port 80. This cannot be resolved and test fails:
ConnectionError:
HTTPConnectionPool(host='tap.plugtests.net8111', port=80): Max retries exceeded with url: /expectation (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1ce504ce80>: Failed to establish a new connection: [Errno -2] Name or service not known',))https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/26Plugtest 2021: Schemas not defined in few MEC013 test suites2021-02-10T18:21:21ZSimon PastorPlugtest 2021: Schemas not defined in few MEC013 test suitesLot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",...Lot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",
"description": "Json Schema for the given object is not yet defined"
}
},
"required": [
"undefinedJsonSchema"
]
}
This code obviously does not match the received json and tests are failing.
Below are the test suites and schemas impacted
Test Suite
RLOCLOOK -- AccessPointList.schema.json
UELOCSUB -- UserTrackingSubscription.schema.json
UEINFOSUB -- ZonalTrafficSubscription.schema.json
other non tested test suites have the same issue:
UEDISTSUB, UEAREASUB, UEDISTLOOK, UETRACKSUBhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/25Templates for Notification requests are not resolved2021-02-03T10:53:46ZMichele CarignaniTemplates for Notification requests are not resolvedThe request object for Notification creation contains variables that should be resolved and substituted with the values inserted in variables.txt.
This is currently not supported. As a result, the body of the request contains the variab...The request object for Notification creation contains variables that should be resolved and substituted with the values inserted in variables.txt.
This is currently not supported. As a result, the body of the request contains the variable name, not the variable value.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/24Handle optionality for ProblemDetails presence and value check2021-02-03T10:52:32ZMichele CarignaniHandle optionality for ProblemDetails presence and value checkThe `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `...The `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `ProblemDetails` data to always be present.
## Expected behaviour
Allow the response to contain string or empty body, as well as the `ProblemDetails` data.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/23keyword missing trying to run `/SRV/RNIS/RnisNotifications.robot`2021-01-19T13:45:26ZFederico Sismondikeyword missing trying to run `/SRV/RNIS/RnisNotifications.robot`Using branch `2.1.1-fix-plu` (commit 5d7b0b754259ac7d2d12b032c670faf5bd73b99b)
I am finding this issue trying to run the robot test:
```
...
[20210105 19:43:46.646] LOG.FAIL - No keyword with name 'Create Mock Request Matcher' found.
....Using branch `2.1.1-fix-plu` (commit 5d7b0b754259ac7d2d12b032c670faf5bd73b99b)
I am finding this issue trying to run the robot test:
```
...
[20210105 19:43:46.646] LOG.FAIL - No keyword with name 'Create Mock Request Matcher' found.
...
```
am I launching the robot incorrectly or is this actually missing ?
I did a quick search on the code for the keyword and couldnt find anything.
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/6Document installation and execution o the tests2020-10-20T10:30:58ZMichele CarignaniDocument installation and execution o the testsIn the README.md file.In the README.md file.Elian KrajaElian Krajahttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/22Delete trailing comma from json files2020-06-09T07:10:30ZFederico SismondiDelete trailing comma from json filesFound two errors in .json files (tried to push MR but apparently I dont have access to project)
See:
[trailing comma err 1](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/CellChangeSubscriptio...Found two errors in .json files (tried to push MR but apparently I dont have access to project)
See:
[trailing comma err 1](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/CellChangeSubscriptionRequest.json#L9)
[trailing comma err 2](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/UpdateCellChangeSubscriptionRequest.json#L9)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/7Response check on DNS testcase for individual query is not proper2020-06-08T13:12:32ZjacobvarghesResponse check on DNS testcase for individual query is not properTest suit: DNS
```
TC_MEC_SRV_DNS_002_OK
[Documentation]
... Check that the IUT responds with the information on a specific DNS rule
... when queried by a MEC Application
...
... Reference ETSI GS MEC 011 ...Test suit: DNS
```
TC_MEC_SRV_DNS_002_OK
[Documentation]
... Check that the IUT responds with the information on a specific DNS rule
... when queried by a MEC Application
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.10.3.1
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.9/MecAppSupportApi.yaml#/definitions/DnsRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Get individual DNS rule ${APP_INSTANCE_ID} ${DNS_RULE_ID}
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is DnsRule
Check Result Contains ${response['body']['DnsRule']} dnsRuleId ${DNS_RULE_ID}
```
The testcase is retrieving an individual dns record and thus the body has to be an object just like below.
```
{
"seconds": 0.002054,
"status": 200,
"body": {
"dnsRuleId": "63684d55-cd66-43a5-975b-ea503c5edb00",
"domainName": "www.example.com",
"ipAddressType": "IP_V4",
"ipAddress": "192.0.2.0",
"ttl": 30,
"state": "ACTIVE"
},
"headers": {
"Content-Type": "application/json; charset=UTF-8",
"Server": "SERVICECENTER/0.0.1",
"Vary": "Accept-Encoding, Origin",
"X-Response-Status": "200",
"Date": "Sat, 23 May 2020 05:47:40 GMT",
"Content-Length": "158"
}
}
```
Here I could see two issues in below line
`Check Result Contains ${response['body']['DnsRule']} dnsRuleId ${DNS_RULE_ID} `
1. Accessing ['DnsRule'] from body, which is different from the API specification.
2. "Check Result Contains" consider the body as a list and check for value inside it, which is wrong. The body is just an object and containes a single dns entry.
Please check and update me if i'm wrong in understanding the script.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/8Message type wrong in DNS update2020-06-08T13:12:28ZjacobvarghesMessage type wrong in DNS updateBelow test case is to update a DNS entry. But looks like the `Update a DNS Rule` statement is sending the POST mesaage instead of PUT.
```
TC_MEC_SRV_DNS_003_OK
[Documentation]
... Check that the IUT updates a specific DNS rul...Below test case is to update a DNS entry. But looks like the `Update a DNS Rule` statement is sending the POST mesaage instead of PUT.
```
TC_MEC_SRV_DNS_003_OK
[Documentation]
... Check that the IUT updates a specific DNS rule
... when commanded by a MEC Application
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.10.3.2
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.9/MecAppSupportApi.yaml#/definitions/DnsRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Update a DNS Rule ${APP_INSTANCE_ID} ${DNS_RULE_ID} DnsRuleUpdate
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is DnsRule
# Check Result Contains ${response['body']['DnsRule']} dnsRuleId ${DNS_RULE_NAME}
# Check Result Contains ${response['body']['DnsRule']} ipAddress ${SOME_IP_ADDRESS}
Update a DNS Rule
[Arguments] ${appInstanceId} ${dnsRuleId} ${content}
Set Headers {"Accept":"application/json"}
Set Headers {"Content-Type":"application/json"}
Set Headers {"Authorization":"${TOKEN}"}
${file}= Catenate SEPARATOR= jsons/ ${content} .json
${body}= Get File ${file}
Post ${apiRoot}/${apiName}/${apiVersion}/applications/${appInstanceId}/dns_rules/${dnsRuleId} ${body}
${output}= Output response
Set Suite Variable ${response} ${output}
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/9Not sending update request on DNS etag error case2020-06-08T13:12:06ZjacobvarghesNot sending update request on DNS etag error caseLooks like the below testcase supposed to send an update with a wrong header and check the response. But the `Update a DNS Rule with invalid etag` just set the headers, not sending the PUT request.
```
TC_MEC_SRV_DNS_003_PF
[Documen...Looks like the below testcase supposed to send an update with a wrong header and check the response. But the `Update a DNS Rule with invalid etag` just set the headers, not sending the PUT request.
```
TC_MEC_SRV_DNS_003_PF
[Documentation]
... Check that the IUT responds with an error when
... a request sent by a MEC Application doesn't comply with a required condition
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.10.3.2
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.9/MecAppSupportApi.yaml#/definitions/DnsRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Update a DNS Rule with invalid etag ${APP_INSTANCE_ID} ${DNS_RULE_ID} DnsRuleUpdate
Check HTTP Response Status Code Is 412
Update a DNS Rule with invalid etag
[Arguments] ${appInstanceId} ${dnsRuleId} ${content}
Set Headers {"If-Match": ${INVALID_ETAG}}
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/10SRV/APPSAQ/PlatAppServices.robot check result is not right2020-06-08T13:11:45ZlimingjiSRV/APPSAQ/PlatAppServices.robot check result is not rightCheck Result Contains ${response['body']['ServiceInfo']} serInstanceId ${SERVICE_ID}
'ServiceInfo' is not included in the results according to v2.1.1 mec api
it should be ${response['body']}Check Result Contains ${response['body']['ServiceInfo']} serInstanceId ${SERVICE_ID}
'ServiceInfo' is not included in the results according to v2.1.1 mec api
it should be ${response['body']}https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/11SRV/APPSAQ/PlatAppServices.robot should be PUT method @line 2142020-06-08T13:11:39ZlimingjiSRV/APPSAQ/PlatAppServices.robot should be PUT method @line 214Post ${apiRoot}/${apiName}/${apiVersion}/applications/${appInstanceId}/services/${serviceId} ${body}
should be PUT method according to:
mec v2.1.1 api
https://forge.etsi.org/swagger/editor/?url=https://forge.etsi.org/gitlab/mec/gs0...Post ${apiRoot}/${apiName}/${apiVersion}/applications/${appInstanceId}/services/${serviceId} ${body}
should be PUT method according to:
mec v2.1.1 api
https://forge.etsi.org/swagger/editor/?url=https://forge.etsi.org/gitlab/mec/gs011-app-enablement-api/raw/master/MecServiceMgmtApi.yamlhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/12SRV/SAQ/PlatServices.robot TC_MEC_SRV_SAQ_002_OK get result keyword wrong2020-06-08T13:11:33ZlimingjiSRV/SAQ/PlatServices.robot TC_MEC_SRV_SAQ_002_OK get result keyword wrongCheck Result Contains ${response['body']['ServiceInfoList']} serInstanceId ${SERVICE_ID}
it should be ${response['body']}, no 'ServiceInfoList' attribute in the resultCheck Result Contains ${response['body']['ServiceInfoList']} serInstanceId ${SERVICE_ID}
it should be ${response['body']}, no 'ServiceInfoList' attribute in the resulthttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/14SRV/SRVSUB/PlatSrvSubscriptions.robot TC_MEC_SRV_SRVSUB_003_OK Check Result ...2020-06-08T13:11:28ZlimingjiSRV/SRVSUB/PlatSrvSubscriptions.robot TC_MEC_SRV_SRVSUB_003_OK Check Result Contains keyword wrongCheck Result Contains ${response['body']['SerAvailabilityNotificationSubscription']} subscriptionType "SerAvailabilityNotificationSubscription"
should be:
${response['body']}Check Result Contains ${response['body']['SerAvailabilityNotificationSubscription']} subscriptionType "SerAvailabilityNotificationSubscription"
should be:
${response['body']}https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/16SRV/APPSUB/PlatAppSubscriptions.robot TC_MEC_SRV_APPSUB_001_OK SubscriptionL...2020-06-08T13:11:22ZlimingjiSRV/APPSUB/PlatAppSubscriptions.robot TC_MEC_SRV_APPSUB_001_OK SubscriptionLinkList name not rightSubscriptionLinkList -> SubscriptionsLinkListSubscriptionLinkList -> SubscriptionsLinkListhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/18SRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_002_OK has multiple issues2020-06-08T13:11:02ZjacobvarghesSRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_002_OK has multiple issuesBelow test case has multiple issues to fix.
1. There won't be any `TrafficRule` field in get response body, as expected in the `Check Result Contains` line. We couldn't find it in any of the MEC documents as well about this.
2. `Check R...Below test case has multiple issues to fix.
1. There won't be any `TrafficRule` field in get response body, as expected in the `Check Result Contains` line. We couldn't find it in any of the MEC documents as well about this.
2. `Check Result Contains` line expects the body as a list of traffic rules, which means the request to get the traffic must be `GET ALL`. Current script just query single rule and expects a list of rules, which is wrong.
```
TC_MEC_SRV_TRAF_002_OK
[Documentation]
... Check that the IUT responds with the information on a specific traffic rule
... when queried by a MEC Application
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.8.3.1
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.8/Mp1.yaml#/definitions/TrafficRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Get individual traffic rule ${APP_INSTANCE_ID} ${TRAFFIC_RULE_ID}
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is TrafficRule
Check Result Contains ${response['body']['TrafficRule']} trafficRuleId ${TRAFFIC_RULE_ID}
```
I guess the script can be fixed like below.
```
TC_MEC_SRV_TRAF_002_OK
[Documentation]
... Check that the IUT responds with the information on a specific traffic rule
... when queried by a MEC Application
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.8.3.1
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.8/Mp1.yaml#/definitions/TrafficRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Get list of traffic rules ${APP_INSTANCE_ID}
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is TrafficRuleList
Check Result Contains ${response['body']} trafficRuleId ${TRAFFIC_RULE_ID}
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/19SRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_003_OK has multiple issues2020-06-08T13:10:56ZjacobvarghesSRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_003_OK has multiple issues![image](/uploads/df5016b97eab70e0dad402fd88377475/image.png)
![image](/uploads/ebef0de7a26c87f3b7325bcae2ab6d59/image.png)
1. At Line 66 the script calls `Update a traffic rule`, and on line 135 inside `Update a traffic rule` function, ...![image](/uploads/df5016b97eab70e0dad402fd88377475/image.png)
![image](/uploads/ebef0de7a26c87f3b7325bcae2ab6d59/image.png)
1. At Line 66 the script calls `Update a traffic rule`, and on line 135 inside `Update a traffic rule` function, it sends a `POST` message insted of `PUT`, which is wrong.
2. Line number 69 and 70 expects a field names `TrafficRule` to be in the response message, which is not correct. The response of update request will have a single traffic rule entry.
3. Line number 69 and 70 again, `Check Result Contains` function expects the `${source}` input as a list, which is wrong in this case as the update response is not a list.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/20TC_MEC_SRV_APPSAQ_002_OK response check is wrong2020-06-08T13:10:43ZjacobvarghesTC_MEC_SRV_APPSAQ_002_OK response check is wrong```
TC_MEC_SRV_APPSAQ_002_OK
[Documentation]
... Check that the IUT notifies the authorised relevant (subscribed) application
... instances when a new service for a given application instance is registered
...
....```
TC_MEC_SRV_APPSAQ_002_OK
[Documentation]
... Check that the IUT notifies the authorised relevant (subscribed) application
... instances when a new service for a given application instance is registered
...
... Reference ETSI GS MEC 011 V2.0.9, clause 8.2.6.3.4
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/master/MecServiceMgmtApi.yaml#/definitions/ServiceInfo
[Tags] PIC_MEC_PLAT PIC_SERVICES
Create new service ServiceInfo ${APP_INSTANCE_ID}
Check HTTP Response Status Code Is 201
Check HTTP Response Body Json Schema Is ServiceInfo
Check HTTP Response Header Contains Location
>> Check Result Contains ${response['body']} serName ${NEW_SERVICE_NAME}
Check Result Contains
[Arguments] ${source} ${parameter} ${value}
Should Be Present In Json List ${source} ${parameter} ${value}
Should Be Present In Json List
[Arguments] ${expr} ${json_field} ${json_value}
Log Check if ${json_field} is present in ${expr} with the value ${jsonvalue}
:FOR ${item} IN @{expr}
\ Exit For Loop If "${item['${json_field}']}" == "${json_value}"
Log Item found ${item}
[return] ${item}
```
`Check Result Contains` expects the response of the service create as a list of json, which is wrong. This is causing the below failure.
```
| FAIL |
Resolving variable '${item['serName']}' failed: TypeError: string indices must be integers
------------------------------------------------------------------------------
PlatAppServices :: A test suite for validating Application Service... | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
```
**This same issue is present in most of the test suite, some of them which we used are listed below. Kindly fix all.**
1. TC_MEC_SRV_APPSAQ_002_OK
2. TC_MEC_SRV_APPSAQ_003_OK
3. TC_MEC_SRV_APPSAQ_004_OK
4. TC_MEC_SRV_APPSUB_002_OK
5. TC_MEC_SRV_APPSUB_003_OK
6. TC_MEC_SRV_DNS_002_OK
7. TC_MEC_SRV_SAQ_002_OK
8. TC_MEC_SRV_SRVSUB_002_OK
9. TC_MEC_SRV_SRVSUB_003_OK
10. TC_MEC_SRV_TRAF_002_OK
11. TC_MEC_SRV_TRAF_003_OKhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/13SRV/APPSAQ/PlatAppServices.robot TC_MEC_SRV_APPSAQ_002_OK better set service_...2020-06-08T13:10:28ZlimingjiSRV/APPSAQ/PlatAppServices.robot TC_MEC_SRV_APPSAQ_002_OK better set service_id as global variable, so the test suite can run together$SERVICE_ID would be called in below test cases, so it should better be set as global variable and pass to below test cases, so that the test suite can run in a row.
a change suggestrion would be:
`
[Tags] PIC_MEC_PLAT PIC_SERVICE...$SERVICE_ID would be called in below test cases, so it should better be set as global variable and pass to below test cases, so that the test suite can run in a row.
a change suggestrion would be:
`
[Tags] PIC_MEC_PLAT PIC_SERVICES
Create new service ServiceInfo ${APP_INSTANCE_ID} Create new service ServiceInfo ${APP_INSTANCE_ID}
Check HTTP Response Status Code Is 201 Check HTTP Response Status Code Is 201
Check HTTP Response Body Json Schema Is ServiceInfo Check HTTP Response Body Json Schema Is ServiceInfo
Check HTTP Response Header Contains Location Check HTTP Response Header Contains Location
Set Global Variable ${SERVICE_ID}
`