MEC Robot Test Suite issueshttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues2022-03-03T09:14:09Zhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/28Plugtest 2021: MEC012 RNISQuery BV RabInfo test error2022-03-03T09:14:09ZSimon PastorPlugtest 2021: MEC012 RNISQuery BV RabInfo test errorThe test case sends the following request with the wrong query parameter:
exampleapi/rni/v2/queries/rab_info?cell_id=0x800000A
but the test case meant to test the app_ins_id=${APP_INS_ID} and not the cell Id since it tries to validate o...The test case sends the following request with the wrong query parameter:
exampleapi/rni/v2/queries/rab_info?cell_id=0x800000A
but the test case meant to test the app_ins_id=${APP_INS_ID} and not the cell Id since it tries to validate on the app_ins_idhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/27Plugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2...2021-02-11T18:16:58ZSimon PastorPlugtest 2021: MEC013 Location API using v1.1.1 schemas and endpoints, not v2.1.1MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK ...MEC013 Location API v2.1.1 is not fully supported by the MEC013 test suites
UELOCLOOK uses endpoint location/v2/users, and not location/v2/query/users
UEINFOLOOK uses endpoint location/v2/users, and not location/v2/query/users
RLOCLOOK uses endpoint location/v2/zones, and not location/v2/query/zones
All schemas for every test suites seem to use not named object, although the spec seemed to show the use of inline (named) objects. This result in validation to be missing one level of hierarchy (does not match the content of the file LocationAPI describe in the header which uses inline objects in its responses)
Note: Note that UEINFOSUB uses values instead of arrays as specified v2.1.1 in its default editable fields (interestRealm, userCriteria)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/26Plugtest 2021: Schemas not defined in few MEC013 test suites2021-02-10T18:21:21ZSimon PastorPlugtest 2021: Schemas not defined in few MEC013 test suitesLot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",...Lot of schemas are undefined which prevent validation tests to pass. The robot testing uses the schema.json that looks like below:
[Content of schema.json example]
{
"properties": {
"undefinedJsonSchema": {
"type": "string",
"description": "Json Schema for the given object is not yet defined"
}
},
"required": [
"undefinedJsonSchema"
]
}
This code obviously does not match the received json and tests are failing.
Below are the test suites and schemas impacted
Test Suite
RLOCLOOK -- AccessPointList.schema.json
UELOCSUB -- UserTrackingSubscription.schema.json
UEINFOSUB -- ZonalTrafficSubscription.schema.json
other non tested test suites have the same issue:
UEDISTSUB, UEAREASUB, UEDISTLOOK, UETRACKSUBhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/25Templates for Notification requests are not resolved2021-02-03T10:53:46ZMichele CarignaniTemplates for Notification requests are not resolvedThe request object for Notification creation contains variables that should be resolved and substituted with the values inserted in variables.txt.
This is currently not supported. As a result, the body of the request contains the variab...The request object for Notification creation contains variables that should be resolved and substituted with the values inserted in variables.txt.
This is currently not supported. As a result, the body of the request contains the variable name, not the variable value.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/24Handle optionality for ProblemDetails presence and value check2021-02-03T10:52:32ZMichele CarignaniHandle optionality for ProblemDetails presence and value checkThe `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `...The `ProblemDetails` data is set to be optional in some cases (see e.g. [ETSI GS MEC 012](https://www.etsi.org/deliver/etsi_gs/MEC/001_099/012/02.01.01_60/gs_mec012v020101p.pdf), Table 7.3.3.1-2).
Currently the test suite requires the `ProblemDetails` data to always be present.
## Expected behaviour
Allow the response to contain string or empty body, as well as the `ProblemDetails` data.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/23keyword missing trying to run `/SRV/RNIS/RnisNotifications.robot`2021-01-19T13:45:26ZFederico Sismondikeyword missing trying to run `/SRV/RNIS/RnisNotifications.robot`Using branch `2.1.1-fix-plu` (commit 5d7b0b754259ac7d2d12b032c670faf5bd73b99b)
I am finding this issue trying to run the robot test:
```
...
[20210105 19:43:46.646] LOG.FAIL - No keyword with name 'Create Mock Request Matcher' found.
....Using branch `2.1.1-fix-plu` (commit 5d7b0b754259ac7d2d12b032c670faf5bd73b99b)
I am finding this issue trying to run the robot test:
```
...
[20210105 19:43:46.646] LOG.FAIL - No keyword with name 'Create Mock Request Matcher' found.
...
```
am I launching the robot incorrectly or is this actually missing ?
I did a quick search on the code for the keyword and couldnt find anything.
https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/22Delete trailing comma from json files2020-06-09T07:10:30ZFederico SismondiDelete trailing comma from json filesFound two errors in .json files (tried to push MR but apparently I dont have access to project)
See:
[trailing comma err 1](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/CellChangeSubscriptio...Found two errors in .json files (tried to push MR but apparently I dont have access to project)
See:
[trailing comma err 1](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/CellChangeSubscriptionRequest.json#L9)
[trailing comma err 2](https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/blob/2.1.1-fix-plu/SRV/RNIS/jsons/UpdateCellChangeSubscriptionRequest.json#L9)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/21Schema file name is wrong2020-06-08T13:09:56ZjacobvarghesSchema file name is wrong```
TC_MEC_SRV_SRVSUB_002_OK :: Check that the IUT acknowledges the su...
{
...
...
}
| FAIL |
FileNotFoundError: [Errno 2] No such file or directory: 'schemas//SerAvailabilityNotificationSubscription.schema.json'
```
Script is accessi...```
TC_MEC_SRV_SRVSUB_002_OK :: Check that the IUT acknowledges the su...
{
...
...
}
| FAIL |
FileNotFoundError: [Errno 2] No such file or directory: 'schemas//SerAvailabilityNotificationSubscription.schema.json'
```
Script is accessing schema file named `SerAvailabilityNotificationSubscription.schema.json`, but in schema folder, file doesn't have the `schema` key in it. Please fix this by changing the file name to `SerAvailabilityNotificationSubscription.schema.json`.
![image](/uploads/5797f76a0a3605e5d4eb670f23f6c94b/image.png)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/20TC_MEC_SRV_APPSAQ_002_OK response check is wrong2020-06-08T13:10:43ZjacobvarghesTC_MEC_SRV_APPSAQ_002_OK response check is wrong```
TC_MEC_SRV_APPSAQ_002_OK
[Documentation]
... Check that the IUT notifies the authorised relevant (subscribed) application
... instances when a new service for a given application instance is registered
...
....```
TC_MEC_SRV_APPSAQ_002_OK
[Documentation]
... Check that the IUT notifies the authorised relevant (subscribed) application
... instances when a new service for a given application instance is registered
...
... Reference ETSI GS MEC 011 V2.0.9, clause 8.2.6.3.4
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/master/MecServiceMgmtApi.yaml#/definitions/ServiceInfo
[Tags] PIC_MEC_PLAT PIC_SERVICES
Create new service ServiceInfo ${APP_INSTANCE_ID}
Check HTTP Response Status Code Is 201
Check HTTP Response Body Json Schema Is ServiceInfo
Check HTTP Response Header Contains Location
>> Check Result Contains ${response['body']} serName ${NEW_SERVICE_NAME}
Check Result Contains
[Arguments] ${source} ${parameter} ${value}
Should Be Present In Json List ${source} ${parameter} ${value}
Should Be Present In Json List
[Arguments] ${expr} ${json_field} ${json_value}
Log Check if ${json_field} is present in ${expr} with the value ${jsonvalue}
:FOR ${item} IN @{expr}
\ Exit For Loop If "${item['${json_field}']}" == "${json_value}"
Log Item found ${item}
[return] ${item}
```
`Check Result Contains` expects the response of the service create as a list of json, which is wrong. This is causing the below failure.
```
| FAIL |
Resolving variable '${item['serName']}' failed: TypeError: string indices must be integers
------------------------------------------------------------------------------
PlatAppServices :: A test suite for validating Application Service... | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
```
**This same issue is present in most of the test suite, some of them which we used are listed below. Kindly fix all.**
1. TC_MEC_SRV_APPSAQ_002_OK
2. TC_MEC_SRV_APPSAQ_003_OK
3. TC_MEC_SRV_APPSAQ_004_OK
4. TC_MEC_SRV_APPSUB_002_OK
5. TC_MEC_SRV_APPSUB_003_OK
6. TC_MEC_SRV_DNS_002_OK
7. TC_MEC_SRV_SAQ_002_OK
8. TC_MEC_SRV_SRVSUB_002_OK
9. TC_MEC_SRV_SRVSUB_003_OK
10. TC_MEC_SRV_TRAF_002_OK
11. TC_MEC_SRV_TRAF_003_OKhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/19SRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_003_OK has multiple issues2020-06-08T13:10:56ZjacobvarghesSRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_003_OK has multiple issues![image](/uploads/df5016b97eab70e0dad402fd88377475/image.png)
![image](/uploads/ebef0de7a26c87f3b7325bcae2ab6d59/image.png)
1. At Line 66 the script calls `Update a traffic rule`, and on line 135 inside `Update a traffic rule` function, ...![image](/uploads/df5016b97eab70e0dad402fd88377475/image.png)
![image](/uploads/ebef0de7a26c87f3b7325bcae2ab6d59/image.png)
1. At Line 66 the script calls `Update a traffic rule`, and on line 135 inside `Update a traffic rule` function, it sends a `POST` message insted of `PUT`, which is wrong.
2. Line number 69 and 70 expects a field names `TrafficRule` to be in the response message, which is not correct. The response of update request will have a single traffic rule entry.
3. Line number 69 and 70 again, `Check Result Contains` function expects the `${source}` input as a list, which is wrong in this case as the update response is not a list.https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/18SRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_002_OK has multiple issues2020-06-08T13:11:02ZjacobvarghesSRV/TRAF/PlatTrafficRules.robot TC_MEC_SRV_TRAF_002_OK has multiple issuesBelow test case has multiple issues to fix.
1. There won't be any `TrafficRule` field in get response body, as expected in the `Check Result Contains` line. We couldn't find it in any of the MEC documents as well about this.
2. `Check R...Below test case has multiple issues to fix.
1. There won't be any `TrafficRule` field in get response body, as expected in the `Check Result Contains` line. We couldn't find it in any of the MEC documents as well about this.
2. `Check Result Contains` line expects the body as a list of traffic rules, which means the request to get the traffic must be `GET ALL`. Current script just query single rule and expects a list of rules, which is wrong.
```
TC_MEC_SRV_TRAF_002_OK
[Documentation]
... Check that the IUT responds with the information on a specific traffic rule
... when queried by a MEC Application
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.8.3.1
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.8/Mp1.yaml#/definitions/TrafficRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Get individual traffic rule ${APP_INSTANCE_ID} ${TRAFFIC_RULE_ID}
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is TrafficRule
Check Result Contains ${response['body']['TrafficRule']} trafficRuleId ${TRAFFIC_RULE_ID}
```
I guess the script can be fixed like below.
```
TC_MEC_SRV_TRAF_002_OK
[Documentation]
... Check that the IUT responds with the information on a specific traffic rule
... when queried by a MEC Application
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.8.3.1
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.8/Mp1.yaml#/definitions/TrafficRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Get list of traffic rules ${APP_INSTANCE_ID}
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is TrafficRuleList
Check Result Contains ${response['body']} trafficRuleId ${TRAFFIC_RULE_ID}
```https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/17SRV/APPSUB/PlatAppSubscriptions.robot TC_MEC_SRV_APPSUB_002_OK better set SUB...2020-06-08T13:10:10ZlimingjiSRV/APPSUB/PlatAppSubscriptions.robot TC_MEC_SRV_APPSUB_002_OK better set SUBSCRIPTION_ID as global variable and keyword error1. TC_MEC_SRV_APPSUB_002_OK better set $SUBSCRIPTION_ID as global variable so it can be passed to below test cases.
2. Check Result Contains ${response['body']['AppTerminationNotificationSubscription']} should be ${response['body']}
...1. TC_MEC_SRV_APPSUB_002_OK better set $SUBSCRIPTION_ID as global variable so it can be passed to below test cases.
2. Check Result Contains ${response['body']['AppTerminationNotificationSubscription']} should be ${response['body']}
suggestion change:
[patch suggest](https://github.com/rexlee8776/mep-robot-test/commit/c62ee36795026cf20b5deddaf03f86c3d1b3f112)https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/16SRV/APPSUB/PlatAppSubscriptions.robot TC_MEC_SRV_APPSUB_001_OK SubscriptionL...2020-06-08T13:11:22ZlimingjiSRV/APPSUB/PlatAppSubscriptions.robot TC_MEC_SRV_APPSUB_001_OK SubscriptionLinkList name not rightSubscriptionLinkList -> SubscriptionsLinkListSubscriptionLinkList -> SubscriptionsLinkListhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/15SRV/SRVSUB/PlatSrvSubscriptions.robot TC_MEC_SRV_SRVSUB_002_OK better set $SU...2020-06-08T13:10:19ZlimingjiSRV/SRVSUB/PlatSrvSubscriptions.robot TC_MEC_SRV_SRVSUB_002_OK better set $SUBSCRIPTION_ID as golbal variablebetter set $SUBSCRIPTION_ID as golbal variable, so it can be passed to below test cases as input, so that the test suite can run in a batch
suggestion:
`
Check HTTP Response Header Contains Location
Dictionary Should Contain ...better set $SUBSCRIPTION_ID as golbal variable, so it can be passed to below test cases as input, so that the test suite can run in a batch
suggestion:
`
Check HTTP Response Header Contains Location
Dictionary Should Contain Item ${response['body']} subscriptionType SerAvailabilityNotificationSubscription Dictionary Should Contain Item ${response['body']} subscriptionType SerAvailabilityNotificationSubscription
Dictionary Should Contain Item ${response['body']} callbackReference ${APP_SRVSUB_NOTIF_CALLBACK_URI} Dictionary Should Contain Item ${response['body']} callbackReference ${APP_SRVSUB_NOTIF_CALLBACK_URI}
${SUBSCRIPTION_URL}= Get From Dictionary ${response['body']['_links']['self']} href
${SUBSCRIPTION_ID}= Fetch From Right ${SUBSCRIPTION_URL} /
Set Global Variable ${SUBSCRIPTION_ID}
`https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/14SRV/SRVSUB/PlatSrvSubscriptions.robot TC_MEC_SRV_SRVSUB_003_OK Check Result ...2020-06-08T13:11:28ZlimingjiSRV/SRVSUB/PlatSrvSubscriptions.robot TC_MEC_SRV_SRVSUB_003_OK Check Result Contains keyword wrongCheck Result Contains ${response['body']['SerAvailabilityNotificationSubscription']} subscriptionType "SerAvailabilityNotificationSubscription"
should be:
${response['body']}Check Result Contains ${response['body']['SerAvailabilityNotificationSubscription']} subscriptionType "SerAvailabilityNotificationSubscription"
should be:
${response['body']}https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/13SRV/APPSAQ/PlatAppServices.robot TC_MEC_SRV_APPSAQ_002_OK better set service_...2020-06-08T13:10:28ZlimingjiSRV/APPSAQ/PlatAppServices.robot TC_MEC_SRV_APPSAQ_002_OK better set service_id as global variable, so the test suite can run together$SERVICE_ID would be called in below test cases, so it should better be set as global variable and pass to below test cases, so that the test suite can run in a row.
a change suggestrion would be:
`
[Tags] PIC_MEC_PLAT PIC_SERVICE...$SERVICE_ID would be called in below test cases, so it should better be set as global variable and pass to below test cases, so that the test suite can run in a row.
a change suggestrion would be:
`
[Tags] PIC_MEC_PLAT PIC_SERVICES
Create new service ServiceInfo ${APP_INSTANCE_ID} Create new service ServiceInfo ${APP_INSTANCE_ID}
Check HTTP Response Status Code Is 201 Check HTTP Response Status Code Is 201
Check HTTP Response Body Json Schema Is ServiceInfo Check HTTP Response Body Json Schema Is ServiceInfo
Check HTTP Response Header Contains Location Check HTTP Response Header Contains Location
Set Global Variable ${SERVICE_ID}
`https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/12SRV/SAQ/PlatServices.robot TC_MEC_SRV_SAQ_002_OK get result keyword wrong2020-06-08T13:11:33ZlimingjiSRV/SAQ/PlatServices.robot TC_MEC_SRV_SAQ_002_OK get result keyword wrongCheck Result Contains ${response['body']['ServiceInfoList']} serInstanceId ${SERVICE_ID}
it should be ${response['body']}, no 'ServiceInfoList' attribute in the resultCheck Result Contains ${response['body']['ServiceInfoList']} serInstanceId ${SERVICE_ID}
it should be ${response['body']}, no 'ServiceInfoList' attribute in the resulthttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/11SRV/APPSAQ/PlatAppServices.robot should be PUT method @line 2142020-06-08T13:11:39ZlimingjiSRV/APPSAQ/PlatAppServices.robot should be PUT method @line 214Post ${apiRoot}/${apiName}/${apiVersion}/applications/${appInstanceId}/services/${serviceId} ${body}
should be PUT method according to:
mec v2.1.1 api
https://forge.etsi.org/swagger/editor/?url=https://forge.etsi.org/gitlab/mec/gs0...Post ${apiRoot}/${apiName}/${apiVersion}/applications/${appInstanceId}/services/${serviceId} ${body}
should be PUT method according to:
mec v2.1.1 api
https://forge.etsi.org/swagger/editor/?url=https://forge.etsi.org/gitlab/mec/gs011-app-enablement-api/raw/master/MecServiceMgmtApi.yamlhttps://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/10SRV/APPSAQ/PlatAppServices.robot check result is not right2020-06-08T13:11:45ZlimingjiSRV/APPSAQ/PlatAppServices.robot check result is not rightCheck Result Contains ${response['body']['ServiceInfo']} serInstanceId ${SERVICE_ID}
'ServiceInfo' is not included in the results according to v2.1.1 mec api
it should be ${response['body']}Check Result Contains ${response['body']['ServiceInfo']} serInstanceId ${SERVICE_ID}
'ServiceInfo' is not included in the results according to v2.1.1 mec api
it should be ${response['body']}https://forge.etsi.org/rep/mec/gs032p3-robot-test-suite/-/issues/9Not sending update request on DNS etag error case2020-06-08T13:12:06ZjacobvarghesNot sending update request on DNS etag error caseLooks like the below testcase supposed to send an update with a wrong header and check the response. But the `Update a DNS Rule with invalid etag` just set the headers, not sending the PUT request.
```
TC_MEC_SRV_DNS_003_PF
[Documen...Looks like the below testcase supposed to send an update with a wrong header and check the response. But the `Update a DNS Rule with invalid etag` just set the headers, not sending the PUT request.
```
TC_MEC_SRV_DNS_003_PF
[Documentation]
... Check that the IUT responds with an error when
... a request sent by a MEC Application doesn't comply with a required condition
...
... Reference ETSI GS MEC 011 V2.0.9, clause 7.2.10.3.2
... OpenAPI https://forge.etsi.org/rep/mec/gs011-app-enablement-api/blob/v2.0.9/MecAppSupportApi.yaml#/definitions/DnsRule
[Tags] PIC_MEC_PLAT PIC_SERVICES
Update a DNS Rule with invalid etag ${APP_INSTANCE_ID} ${DNS_RULE_ID} DnsRuleUpdate
Check HTTP Response Status Code Is 412
Update a DNS Rule with invalid etag
[Arguments] ${appInstanceId} ${dnsRuleId} ${content}
Set Headers {"If-Match": ${INVALID_ETAG}}
```