api-tests issueshttps://forge.etsi.org/rep/nfv/api-tests/-/issues2021-01-14T15:30:16Zhttps://forge.etsi.org/rep/nfv/api-tests/-/issues/105Notification API / NotificationEndpoint test purposes to be re-designed?2021-01-14T15:30:16ZMichele CarignaniNotification API / NotificationEndpoint test purposes to be re-designed?## Background
APIs marked `*Notification-API`specify the behaviour of the endpoints that receive notifications for asynchronous events.
When testing such APIs, the role of the FUT is the to implement the server which is able to receive...## Background
APIs marked `*Notification-API`specify the behaviour of the endpoints that receive notifications for asynchronous events.
When testing such APIs, the role of the FUT is the to implement the server which is able to receive the notification. The role of the Test System should be the one of the producer of notifications, i.e. the client in the notification exchanges.
## Problem
The `Notification-API`s Notification Endpoint tests currently implement the server side of the notification exchange (by means of the MockServer).
In more details:
* according to the SOL specs, the API Consumer implements (e.g. exposes )the NotificationEndpoint ("The API producer can use this resource to send notifications related to VNF alarms or about a rebuilt alarm list to a subscribed API consumer, which has provided the URI of this resource during the subscription process. ", SOL 003, 2.6.1, 7.4.6.1)
* in TST 010, we test the server of the resource
* Therefore the function to be tested in the Notification Endpoints is the API Consumer of a certain API (in the case of VNF Fault Management API this is the NFVO)
* In the tests keywords, you will see that the Test systems expects to receive the notifications and it checks if the notification is correct.
-> I am not saying that this is actually wrong, but it is not in line with the general approach to the test.
=> At least the Test objectives should clarify which IUT is expected to be tested.
### Affected test suites (i.e. Robot files)
* [ ] https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-dev/SOL002/VNFPerformanceManagementNotification-API/PerformanceManagementNotification.robot
* [ ] https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-dev/SOL002/VNFIndicatorNotification-API/VnfIndicatorNotification.robot
* [ ] https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-dev/SOL003/VNFFaultManagementNotification-API/NotificationEndpoint.robot
* [ ] https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-dev/SOL003/VirtualisedResourcesQuotaAvailableNotification-API/NotificationEndpoint.robot
## Proposal
### Option 1
To be fixed in 2.7.1 by changing test purposes and test steps to just executed the following:
* Sending the notification
* verifying the response code is correct
### Option 2
* Change the test purpose to express that we are testing the API Producer capability of sending notifications
2.7.1Giacomo BerniniGiacomo Berninihttps://forge.etsi.org/rep/nfv/api-tests/-/issues/108Disable Individual Network Service Descriptor issue 2.6.1-fix-plu2020-06-17T21:53:28Zraducccradu@luxoft.comDisable Individual Network Service Descriptor issue 2.6.1-fix-pluhttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-dev/SOL005/NSDManagement-API/IndividualNSDescriptor.robot
```
Disable Individual Network Service Descriptor
[Documentation] Test ID: 5.3.1.2.3
... Test title: Disable ...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-dev/SOL005/NSDManagement-API/IndividualNSDescriptor.robot
```
Disable Individual Network Service Descriptor
[Documentation] Test ID: 5.3.1.2.3
... Test title: Disable Individual Network Service Descriptor
... Test objective: The objective is to test the disabling of an individual Network Service Descriptor and perform a JSON schema and content validation of the collected data structure
... Pre-conditions: One or more Network Service Descriptors are onboarded in the NFVO in enabled operational state.
... Reference: Clause 5.4.3.3.4 - ETSI GS NFV-SOL 005 [3] v2.6.1
... Config ID: Config_prod_NFVO
... Applicability: none
... Post-Conditions: The Network Service Descriptor is in operational state DISABLED and usage state is not modified
Send PATCH to disable Individual Network Service Descriptor
Check HTTP Response Status Code Is 200
Check HTTP Response Body Json Schema Is NsdInfoModification
Check Postcondition Network Service Descriptor is in operational state DISABLED
Check Postcondition Network Service Descriptor usage state is unmodified (Implicit)
```
Between these 2 lines:
Check Postcondition Network Service Descriptor is in operational state DISABLED
Check Postcondition Network Service Descriptor usage state is unmodified (Implicit)
There should be an intermediary GET Request querying the ns descriptor with that id, otherwise in the primitive method
"Check Postcondition Network Service Descriptor usage state " evaluation is applied on the previous response object witch is NsdInfoModifications type and not NsdInfo type.
Suggested fix:
Check Postcondition Network Service Descriptor is in operational state DISABLED
**GET Individual Network Service Descriptor Information**
Check Postcondition Network Service Descriptor usage state is unmodified (Implicit)https://forge.etsi.org/rep/nfv/api-tests/-/issues/119Additional parameter to specify arttifact type2020-06-17T21:52:06ZphamvAdditional parameter to specify arttifact type`GET Individual VNF Package Artifact in octet stream format` uses `${artifactPath}` for all contents, request to add additional parameter to indicate content type as text or binary (application/octet-stream).
Current variables are defin...`GET Individual VNF Package Artifact in octet stream format` uses `${artifactPath}` for all contents, request to add additional parameter to indicate content type as text or binary (application/octet-stream).
Current variables are defined in:
api-tests/SOL003/VNFPackageManagement-API/environment/vnfPackageArtifacts.txthttps://forge.etsi.org/rep/nfv/api-tests/-/issues/118Response variable not updated after the GET call2020-06-17T21:51:45ZElian KrajaResponse variable not updated after the GET callhttps://forge.etsi.org/rep/nfv/api-tests/blob/2.4.1-fix-plu/SOL003/VNFPackageManagement-API/VNFPackageManagementKeywords.robot
Check Postcondition VNFD Exist
Log Checking that vnf pacakge still exists
Set Headers {"Acce...https://forge.etsi.org/rep/nfv/api-tests/blob/2.4.1-fix-plu/SOL003/VNFPackageManagement-API/VNFPackageManagementKeywords.robot
Check Postcondition VNFD Exist
Log Checking that vnf pacakge still exists
Set Headers {"Accept": "${ACCEPT_PLAIN}"}
Set Headers {"Accept": "${ACCEPT_ZIP}"}
Run Keyword If ${AUTH_USAGE} == 1 Set Headers {"Authorization": "${AUTHORIZATION}"}
GET ${apiRoot}/${apiName}/${apiVersion}/vnf_packages/${vnfPkgZipVNFD}/vnfd
Check HTTP Response Status Code Is 200
Suggest to add the following lines after GET if possible:
${output}= Output response
Set Suite Variable ${response} ${output}Elian KrajaElian Krajahttps://forge.etsi.org/rep/nfv/api-tests/-/issues/117VnfLcmMntOperationKeywords.robot SOL003 2.6.1 AttributeError headers2020-06-12T13:15:14Zraducccradu@luxoft.comVnfLcmMntOperationKeywords.robot SOL003 2.6.1 AttributeError headers```
GET information about multiple VNF instances to get Paged Response... | FAIL |
Resolving variable '${response.headers}' failed: AttributeError: 'dict' object has no attribute 'headers'
```
https://forge.etsi.org/rep/nfv/api-tests/bl...```
GET information about multiple VNF instances to get Paged Response... | FAIL |
Resolving variable '${response.headers}' failed: AttributeError: 'dict' object has no attribute 'headers'
```
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robothttps://forge.etsi.org/rep/nfv/api-tests/-/issues/116VnfLcmMntOperationKeywords.robot SOL003 2.6.1 Missing separator2020-06-12T13:15:06Zraducccradu@luxoft.comVnfLcmMntOperationKeywords.robot SOL003 2.6.1 Missing separatorhttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robot
`${schema} = Catenate ${input} .schema.json`
Should be:
`${schema} = Catenate SEPARATOR= $...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robot
`${schema} = Catenate ${input} .schema.json`
Should be:
`${schema} = Catenate SEPARATOR= ${input} .schema.json`https://forge.etsi.org/rep/nfv/api-tests/-/issues/115NSLifecycleManagement-API.NSLCMOccurences.robot small bugs 2.6.12020-06-12T13:14:58Zraducccradu@luxoft.comNSLifecycleManagement-API.NSLCMOccurences.robot small bugs 2.6.1https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSLCMOccurences.robot
Attribute of NsLcmOpOcc 'operationParams' has cardinality [0-1], so it shouldn't be a required property.
Proof https://ww...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSLCMOccurences.robot
Attribute of NsLcmOpOcc 'operationParams' has cardinality [0-1], so it shouldn't be a required property.
Proof https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/005/02.06.01_60/gs_nfv-sol005v020601p.pdf
```
GET status information about multiple NS LCM occurrences with "exc... | FAIL |
ValidationError: Validation error for schema NsLcmOpOccs.schema.json: 'operationParams' is a required property
GET status information about multiple NS LCM occurrences with "fie... | FAIL |
ValidationError: Validation error for schema NsLcmOpOccs.schema.json: 'operationParams' is a required property
GET status information about multiple NS LCM occurrences with "exc... | FAIL |
ValidationError: Validation error for schema NsLcmOpOccs.schema.json: 'operationParams' is a required property
GET status information about multiple NS LCM occurrences with "all... | FAIL |
ValidationError: Validation error for schema NsLcmOpOccs.schema.json: 'operationParams' is a required property
```
Schemas that should be modified:
```
schemas/NsLcmOpOcc.schema.json
schemas/NsLcmOpOccs.schema.json
```
```
GET information about multiple NS instances as Paged Response :: T... | FAIL |
Resolving variable '${response.headers}' failed: AttributeError: 'dict' object has no attribute 'headers'
```https://forge.etsi.org/rep/nfv/api-tests/-/issues/114Etag instead of ETag 2.6.1 IndividualNSInstance.robot2020-06-12T13:14:49Zraducccradu@luxoft.comEtag instead of ETag 2.6.1 IndividualNSInstance.robotIndividualNSInstance.robot
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/IndividualNSInstance.robot
```
PUT Individual NSInstance - Method not implemented :: Test ID: 5.3...
Property 'Eta...IndividualNSInstance.robot
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/IndividualNSInstance.robot
```
PUT Individual NSInstance - Method not implemented :: Test ID: 5.3...
Property 'Etag' does not exist in:
```
```
GET Information about an individual NS Instance :: Test ID: 5.3.2.2.2 | FAIL |
Expected property 'response headers Etag' was not found.
```
```
PATCH Individual NSInstance - Method not implemented :: Test ID: 5... | FAIL |
Expected property 'response headers Etag' was not found.
```https://forge.etsi.org/rep/nfv/api-tests/-/issues/113instantiationState instead of nsState in InstantiateNSTask.robot 2.6.12020-06-12T13:14:34Zraducccradu@luxoft.cominstantiationState instead of nsState in InstantiateNSTask.robot 2.6.1Expected property 'response body instantiationState' was not found.
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSLCMOperationKeywords.robot
Response body nsState should be used since in...Expected property 'response body instantiationState' was not found.
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSLCMOperationKeywords.robot
Response body nsState should be used since instantiationState is not a property of NsInstance.
https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/005/02.06.01_60/gs_nfv-sol005v020601p.pdfhttps://forge.etsi.org/rep/nfv/api-tests/-/issues/112NSInstances.robot 2.6.1 Attribute Error2020-06-12T13:14:20Zraducccradu@luxoft.comNSInstances.robot 2.6.1 Attribute Errorhttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSInstances.robot
GET information about multiple NS instances as Paged Response :: T... | FAIL |
Resolving variable '${response.headers}' faile...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSInstances.robot
GET information about multiple NS instances as Paged Response :: T... | FAIL |
Resolving variable '${response.headers}' failed: AttributeError: 'dict' object has no attribute 'headers'https://forge.etsi.org/rep/nfv/api-tests/-/issues/111Unresolvable JSON pointer in NSLifecycleManagement-API .NSInstances.robot2020-06-12T13:13:58Zraducccradu@luxoft.comUnresolvable JSON pointer in NSLifecycleManagement-API .NSInstances.robothttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSInstances.robot
GET information about multiple NS instances :: Test ID: 5.3.2.1.2 | FAIL |
RefResolutionError: Unresolvable JSON pointer:...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSInstances.robot
GET information about multiple NS instances :: Test ID: 5.3.2.1.2 | FAIL |
RefResolutionError: Unresolvable JSON pointer: 'definitions/VnfInstance'
GET NSInstances with "exclude_fields" attribute selector :: Test I... | FAIL |
RefResolutionError: Unresolvable JSON pointer: 'definitions/VnfInstance
Schema definitions missing from schemas/NSInstance.schema.json .https://forge.etsi.org/rep/nfv/api-tests/-/issues/109Seaparator missing 2.6.1 NSLCM2020-06-12T13:13:47Zraducccradu@luxoft.comSeaparator missing 2.6.1 NSLCMhttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSLCMOperationKeywords.robot
```
Check HTTP Response Body Json Schema Is
[Arguments] ${input}
${schema} = Catenate ${input} ...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSLifecycleManagement-API/NSLCMOperationKeywords.robot
```
Check HTTP Response Body Json Schema Is
[Arguments] ${input}
${schema} = Catenate ${input} .schema.json
Validate Json ${schema} ${response[0]['body']}
Log Json Schema Validation OK
```
SEPARATOR argument should be used after Catenate keywordhttps://forge.etsi.org/rep/nfv/api-tests/-/issues/107IndividualNSDescriptor test suite "Etag" instead of "ETag"2020-06-12T13:13:19Zraducccradu@luxoft.comIndividualNSDescriptor test suite "Etag" instead of "ETag"IndividualNSDescriptor . GET Individual Network Service Descriptor Information
does not contain 'Etag'
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/IndividualNSDescriptor.robotIndividualNSDescriptor . GET Individual Network Service Descriptor Information
does not contain 'Etag'
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/IndividualNSDescriptor.robothttps://forge.etsi.org/rep/nfv/api-tests/-/issues/103Typos in NsdInfo schemas SOL005 NSDManagement 2.6.12020-06-12T13:13:02Zraducccradu@luxoft.comTypos in NsdInfo schemas SOL005 NSDManagement 2.6.1"statussss" typo in
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/schemas/NsdInfo.schema.json
and
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/schemas/NsdI..."statussss" typo in
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/schemas/NsdInfo.schema.json
and
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/schemas/NsdInfos.schema.jsonhttps://forge.etsi.org/rep/nfv/api-tests/-/issues/1022.6.1 Check LINK in Header Bug2020-06-12T13:12:52Zraducccradu@luxoft.com2.6.1 Check LINK in Header Bughttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/NSDManagementKeywords.robot
Check LINK in Header
${linkURL}= Get Value From Json ${response.headers} $..Link
Should Not Be Empty ${...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL005/NSDManagement-API/NSDManagementKeywords.robot
Check LINK in Header
${linkURL}= Get Value From Json ${response.headers} $..Link
Should Not Be Empty ${linkURL}
${response['headers']} should be the correct form.https://forge.etsi.org/rep/nfv/api-tests/-/issues/101"Check HTTP Response Status Code Is" method is implemented differently2020-06-12T13:12:42Zraducccradu@luxoft.com"Check HTTP Response Status Code Is" method is implemented differentlyhttps://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robot
```
Check HTTP Response Status Code Is
[Arguments] ${expected_status}
Should Be Equal ${re...https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robot
```
Check HTTP Response Status Code Is
[Arguments] ${expected_status}
Should Be Equal ${response.status_code} ${expected_status}
Log Status code validated
```
Expects response to be an object with status_code attribute.
===========================================================================
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFPackageManagement-API/VNFPackageManagementKeywords.robot
Check HTTP Response Status Code Is
[Arguments] ${expected_status}
Should Be Equal ${response['status']} ${expected_status}
Log Status code validated
Expects response to be an dict with status attribute.https://forge.etsi.org/rep/nfv/api-tests/-/issues/100Assignment of response variable @ vs $ in VnfLcmMntOperationKeywords and NSLC...2020-06-12T13:12:06Zraducccradu@luxoft.comAssignment of response variable @ vs $ in VnfLcmMntOperationKeywords and NSLCMOperationKeywordsPlease check the correct assignment of response variable in keywords robot files.
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robot
Set Global Variable @{r...Please check the correct assignment of response variable in keywords robot files.
https://forge.etsi.org/rep/nfv/api-tests/blob/2.6.1-fix-plu/SOL003/VNFLifecycleManagement-API/VnfLcmMntOperationKeywords.robot
Set Global Variable @{response} ${outputResponse}
vs
Set Global Variable ${response} ${outputResponse}
Both forms are used in keyword files which misleads an NFVO solution.https://forge.etsi.org/rep/nfv/api-tests/-/issues/104Missing keywords2020-06-05T10:42:56ZMichele CarignaniMissing keywordsBy running `robot --dryrun` on the files in the projects a set of issues has been identified.
Attached the stderr and stdout of the execution of the script below.
```bash
str=Keywords.robot;
for i in */*/*.robot ; do
[[ "$i" != *"$str...By running `robot --dryrun` on the files in the projects a set of issues has been identified.
Attached the stderr and stdout of the execution of the script below.
```bash
str=Keywords.robot;
for i in */*/*.robot ; do
[[ "$i" != *"$str"* ]] && \
(echo "++++ Dryrun $i" && \
robot --dryrun --output NONE --report NONE --log NONE $i || \
echo "++++ Issues in file $i");
done
```
[VALIDATION_ERRORS.txt](/uploads/907ed31167a98081c254cb8a0929b763/VALIDATION_ERRORS.txt)
[VALIDATION_LOG.txt](/uploads/66a857ba8a81d64955edbab0d43547d1/VALIDATION_LOG.txt)