fix(021_15, 021_16): temporal pagination of Query Temporal Evolution follows TS 104 176 clause 6.4.7.3

Ken Zangelinrequested to merge
fix/temporal-pagination-query into fix/temporal-pagination-retrieve

TS 104 176 clause 6.4.7.3, temporal pagination, on Query Temporal Evolution of Entities. This is the Query side of !347 (merged), and it is stacked on it: it uses the keywords !347 (merged) adds. Tagged since_ts104_v1.1.1. See #97.

  • 021_15: 7 TCs on GET /temporal/entities. lastN and firstN, with offsetN, the last page, a page that covers every instance, temporalValues, and firstN with lastN (400). Check Temporal Page Size from !347 (merged) replaces the TC's own copy of the per-Attribute page check.
  • 021_16: the POST query (temporalQ.lastN=5 on a 20-instance entity) gives the intervalbefore Link with offsetN=5. It no longer expects a 206 with Content-Range, which TS 104 176 does not have. New query file: entity-operations-paginated-query.jsonld.

8 TCs. All fail on develop and pass with this MR, against coraine. Doc unit tests 13/13, statisticsDocumentationData.py 0 of 607 failed, robocop format --check clean, robot --dryrun clean, no unused test data.

Merge after !347 (merged). Fix MR 7 of the plan: https://famzang.com/ttf/ngsi-ld-suite-plan.html

Merge request reports

TP/NGSI-LD/ContextInformation/Consumption/TemporalEntity/QueryTemporalEvolutionOfEntities/021_15.robot

+76 −123
Changes for TP/NGSI-LD/ContextInformation/Consumption/TemporalEntity/QueryTemporalEvolutionOfEntities/021_15.robot: 76 added lines, 123 removed lines.
Original line number Diff line number Diff line 
*** Settings ***

Documentation       Check temporal pagination is applied when querying the temporal evolution of entities

Documentation       Check that temporal pagination (TS 104-176, clause 6.4.7.3) is applied when querying the temporal evolution of entities



Library             DateTime

Resource            ${EXECDIR}/resources/ApiUtils/Common.resource
@@ -10,7 +10,7 @@ Resource ${EXECDIR}/resources/JsonUtils.resource 


Suite Setup         Setup Initial Entities

Suite Teardown      Delete Initial Entities

Test Template       Retrieve Temporal Entities

Test Template       Query Temporal Entities With Pagination





*** Variables ***
@@ -18,49 +18,70 @@ ${first_vehicle_payload_file}= pagination/2020-01-vehicule-temporal-represe 
${second_vehicle_payload_file}=     2020-09-vehicle-temporal-representation.jsonld

${timeBefore}=                      2019-01-01T01:01:00Z

${timeAfter}=                       2021-01-01T01:01:00Z

${lrt}=                             least recent timestamp

${mrt}=                             most recent timestamp





*** Test Cases ***

021_15_01 Retrieve The Entity With lastN And Timerel Before

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    lastN=${20}    expectedSize=20    timerel=before    timeAt=${timeAfter}    expectedRangeStart=${timeAfter}    expectedRangeEnd=${lrt}

021_15_02 Retrieve The Entity With lastN And Timerel Between

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    lastN=${20}    timerel=between    timeAt=${timeBefore}    endTimeAt=${timeAfter}    expectedRangeStart=${timeAfter}    expectedRangeEnd=${lrt}

021_15_03 Retrieve The Entity With lastN And Timerel After

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    lastN=${20}    expectedSize=20    timerel=after    timeAt=${timeBefore}    expectedRangeStart=${mrt}    expectedRangeEnd=${lrt}

021_15_04 Retrieve The Entity With Timerel Before

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    timerel=before    expectedSize=*    timeAt=${timeAfter}    expectedRangeEnd=${mrt}

021_15_05 Retrieve The Entity With Timerel Between

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    timerel=between    timeAt=${timeBefore}    endTimeAt=${timeAfter}    expectedRangeStart=${timeBefore}    expectedRangeEnd=${mrt}

021_15_06 Retrieve The Entity With Timerel After

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    timerel=after    timeAt=${timeBefore}    expectedRangeStart=${timeBefore}    expectedRangeEnd=${mrt}

021_15_07 Retrieve The Entity With temporalValues And Timerel After

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    representation=temporalValues    timerel=after    timeAt=${timeBefore}    expectedRangeStart=${timeBefore}    expectedRangeEnd=${mrt}

021_15_08 Retrieve The Entity With temporalValues, lastN And Timerel Between

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    since_v1.5.1

    representation=temporalValues    lastN=${20}    timerel=between    timeAt=${timeBefore}    endTimeAt=${timeAfter}    expectedRangeStart=${timeAfter}    expectedRangeEnd=${lrt}

021_15_01 Query With lastN Smaller Than The Available Instances

    [Documentation]    lastN paginates in descending order; the deeper page holds EARLIER timestamps → rel="intervalbefore"

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    lastN=${5}    timerel=between    timeAt=${timeBefore}    endTimeAt=${timeAfter}

    ...    maxInstances=5    expectBeforeOffset=5

021_15_02 Query The Second Page With lastN And offsetN

    [Documentation]    A middle page carries both pointers

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    lastN=${5}    offsetN=${5}    timerel=between    timeAt=${timeBefore}    endTimeAt=${timeAfter}

    ...    maxInstances=5    expectBeforeOffset=10    expectAfterOffset=0

021_15_03 Query With firstN

    [Documentation]    firstN paginates in ascending order; the deeper page holds LATER timestamps → rel="intervalafter"

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    firstN=${5}    timerel=after    timeAt=${timeBefore}

    ...    maxInstances=5    expectAfterOffset=5

021_15_04 Query The Last Page With firstN And offsetN

    [Documentation]    Past the largest attribute's instance count only the backward pointer remains

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    firstN=${5}    offsetN=${18}    timerel=after    timeAt=${timeBefore}

    ...    maxInstances=2    expectBeforeOffset=13

021_15_05 Query With lastN Covering All Instances

    [Documentation]    Nothing remains beyond the page — no pagination pointers

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    lastN=${20}    timerel=after    timeAt=${timeBefore}

    ...    maxInstances=20

021_15_06 Query With temporalValues And lastN

    [Documentation]    Pagination pointers are representation-independent

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    representation=temporalValues    lastN=${5}    timerel=after    timeAt=${timeBefore}

    ...    expectBeforeOffset=5

021_15_07 Query With Both firstN And lastN

    [Documentation]    Ascending and descending pagination at once is contradictory

    [Tags]    te-query    core_11_3_3    http_6_4_7_3    since_ts104_v1.1.1

    firstN=${5}    lastN=${5}    timerel=after    timeAt=${timeBefore}

    ...    expectedStatus=400

021_15_08 Query With limit And firstN

    [Documentation]    Both levels of pagination in one response: one Entity per page, so a pointer to the next page of

    ...    Entities (rel="next", clause 6.4.7.2), and two instances per Attribute, so a pointer to the next page of

    ...    instances (rel="intervalafter", clause 6.4.7.3) - both vehicles have more than two

    [Tags]    te-query    core_11_3_3    http_6_4_7_2    http_6_4_7_3    since_ts104_v1.1.1

    firstN=${2}    limit=${1}    timerel=after    timeAt=${timeBefore}

    ...    maxInstances=2    expectAfterOffset=2    expectNext=${True}





*** Keywords ***

Retrieve Temporal Entities

    [Documentation]    Check temporal pagination behavior on multiple entity endpoint

Query Temporal Entities With Pagination

    [Documentation]    Query the temporal evolution of entities and check the § 6.4.7.3 pagination behaviour

    [Arguments]

    ...    ${representation}=${EMPTY}

    ...    ${timerel}=${EMPTY}

    ...    ${timeAt}=${EMPTY}

    ...    ${endTimeAt}=${EMPTY}

    ...    ${lastN}=${EMPTY}

    ...    ${expectedRangeStart}=${EMPTY}

    ...    ${expectedRangeEnd}=${EMPTY}

    ...    ${expectedSize}=${EMPTY}

    ...    ${firstN}=${EMPTY}

    ...    ${offsetN}=${EMPTY}

    ...    ${limit}=${EMPTY}

    ...    ${maxInstances}=${EMPTY}

    ...    ${expectAfterOffset}=${EMPTY}

    ...    ${expectBeforeOffset}=${EMPTY}

    ...    ${expectNext}=${False}

    ...    ${expectedStatus}=200

    ${entity_types_to_be_retrieved}=    Catenate    SEPARATOR=,    Vehicle



    ${response}=    Query Temporal Representation Of Entities
@@ -71,50 +92,35 @@ Retrieve Temporal Entities 
    ...    timeAt=${timeAt}

    ...    endTimeAt=${endTimeAt}

    ...    lastN=${lastN}

    Check Response Status Code    206    ${response.status_code}



    ${contentRange}=    Get Regexp Matches

    ...    ${response.headers}[Content-Range]

    ...    ([a-zA-Z\-]+) (.*-.*-.*)-(.*-.*-.*)\/(.*)

    ...    1

    ...    2

    ...    3

    ...    4

    ${unit}=    Set Variable    ${contentRange}[0][0]

    ${rangeStart}=    Convert Date    ${contentRange}[0][1]

    ${rangeEnd}=    Convert Date    ${contentRange}[0][2]

    ${size}=    Set Variable    ${contentRange}[0][3]



    Check Content Range Part Equal    ${unit}    date-time



    IF    $expectedRangeStart != ''

        IF    $expectedRangeStart == $lrt

            ${expectedRangeStart}=    Get Least Recent Timestamp From Vehicles    ${response.json()}

        ELSE IF    $expectedRangeStart == $mrt

            ${expectedRangeStart}=    Get Most Recent Timestamp From Vehicles    ${response.json()}

    ...    firstN=${firstN}

    ...    offsetN=${offsetN}

    ...    limit=${limit}



    Check Response Status Code    ${expectedStatus}    ${response.status_code}

    IF    '${expectedStatus}' != '200'    RETURN



    IF    '${limit}' != ''    Length Should Be    ${response.json()}    ${limit}



    IF    ${expectNext}

        Check Pagination Link    ${response.headers}    next

    ELSE

            ${expectedRangeStart}=    Convert Date    ${expectedRangeStart}

        Check Pagination Link Absent    ${response.headers}    next

    END

        Check Content Range Part Equal    ${expectedRangeStart}    ${rangeStart}



    IF    '${maxInstances}' != '' and '${representation}' == ''

        Check Temporal Page Size    ${response.json()}    ${maxInstances}

    END



    IF    $expectedRangeEnd != ''

        IF    $expectedRangeEnd == $lrt

            ${expectedRangeEnd}=    Get Least Recent Timestamp From Vehicles

            ...    ${response.json()}

            ...    ${representation}

        ELSE IF    $expectedRangeEnd == $mrt

            ${expectedRangeEnd}=    Get Most Recent Timestamp From Vehicles

            ...    ${response.json()}

            ...    ${representation}

    IF    '${expectAfterOffset}' != ''

        Check Pagination Link    ${response.headers}    intervalafter    ${expectAfterOffset}

    ELSE

            ${expectedRangeEnd}=    Convert Date    ${expectedRangeEnd}

        END

        Check Content Range Part Equal    ${expectedRangeEnd}    ${rangeEnd}

        Check Pagination Link Absent    ${response.headers}    intervalafter

    END



    IF    $expectedSize != ''

        Check Content Range Part Equal    ${expectedSize}    ${size}

    IF    '${expectBeforeOffset}' != ''

        Check Pagination Link    ${response.headers}    intervalbefore    ${expectBeforeOffset}

    ELSE

        Check Pagination Link Absent    ${response.headers}    intervalbefore

    END



Setup Initial Entities
@@ -136,56 +142,3 @@ Setup Initial Entities 
Delete Initial Entities

    Delete Temporal Representation Of Entity    ${first_temporal_entity_representation_id}

    Delete Temporal Representation Of Entity    ${second_temporal_entity_representation_id}
 


Get Least Recent Timestamp From Vehicles

    [Arguments]    ${body}    ${representation}=${EMPTY}



    ${attributeList}=    Get Attribute List From Vehicle Body    ${body}    ${representation}



    ${leastRecentTimestamp}=    Convert Date    ${timeAfter}

    FOR    ${attribute}    IN    @{attributeList}

        IF    $representation == 'temporalValues'

            ${attributeTime}=    Convert Date    ${attribute}[1]

        ELSE

            ${attributeTime}=    Convert Date    ${attribute}[observedAt]

        END



        IF    $leastRecentTimestamp >= $attributeTime

            ${leastRecentTimestamp}=    Set Variable    ${attributeTime}

        END

    END

    RETURN    ${leastRecentTimestamp}



Get Most Recent Timestamp From Vehicles

    [Arguments]    ${body}    ${representation}=${EMPTY}

    ${attributeList}=    Get Attribute List From Vehicle Body    ${body}    ${representation}



    ${mostRecentTimestamp}=    Convert Date    ${timeBefore}

    FOR    ${attribute}    IN    @{attributeList}

        IF    $representation == 'temporalValues'

            ${attributeTime}=    Convert Date    ${attribute}[1]

        ELSE

            ${attributeTime}=    Convert Date    ${attribute}[observedAt]

        END

        IF    $mostRecentTimestamp <= $attributeTime

            ${mostRecentTimestamp}=    Set Variable    ${attributeTime}

        END

    END

    RETURN    ${mostRecentTimestamp}



Get Attribute List From Vehicle Body

    [Arguments]    ${body}    ${representation}=${EMPTY}

    IF    $representation == 'temporalValues'

        ${attributeList}=    Combine Lists

        ...    ${body}[0][speed][values]

        ...    ${body}[0][fuelLevel][values]

        ...    ${body}[1][speed][values]

        ...    ${body}[1][fuelLevel][values]

    ELSE

        ${attributeList}=    Combine Lists

        ...    ${body}[0][speed]

        ...    ${body}[0][fuelLevel]

        ...    ${body}[1][speed]

        ...    ${body}[1][fuelLevel]

    END

    RETURN    ${attributeList}

TP/NGSI-LD/ContextInformation/Consumption/TemporalEntity/QueryTemporalEvolutionOfEntities/021_16.robot

+8 −15
Changes for TP/NGSI-LD/ContextInformation/Consumption/TemporalEntity/QueryTemporalEvolutionOfEntities/021_16.robot: 8 added lines, 15 removed lines.
Original line number Diff line number Diff line 
*** Settings ***

Documentation       Check temporal pagination is applied when querying the temporal evolution of entities via POST

Documentation       Check that temporal pagination (TS 104-176, clause 6.4.7.3) is applied when querying the temporal evolution of entities via POST



Resource            ${EXECDIR}/resources/ApiUtils/Common.resource

Resource            ${EXECDIR}/resources/ApiUtils/TemporalContextInformationConsumption.resource
@@ -19,31 +19,24 @@ ${second_vehicle_payload_file}= 2020-09-vehicle-temporal-representation.json 


*** Test Cases ***    PAYLOAD_FILE

021_16_01 Retrieve The Entities Via Post

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    http_8_7_3_1    since_v1.5.1

    entity-operations-before-query.jsonld

    [Tags]    te-retrieve    core_11_3_3    http_6_4_7_3    http_8_7_3_1    since_ts104_v1.1.1

    entity-operations-paginated-query.jsonld





*** Keywords ***

Retrieve Temporal Entities Via Post

    [Documentation]    Check that temporal pagination is triggered on the post temporal query

    [Documentation]    Check that temporal pagination (TS 104-176, clause 6.4.7.3) is triggered on the post temporal query

    [Arguments]    ${payload_file}



    ${response}=    Query Temporal Representation Of Entities Via Post

    ...    query_file_name=${payload_file}

    ...    context=${ngsild_test_suite_context}



    Check Response Status Code    206    ${response.status_code}

    Check Response Status Code    200    ${response.status_code}



    ${contentRange}=    Get Regexp Matches

    ...    ${response.headers}[Content-Range]

    ...    ([a-zA-Z\-]+) (.*-.*-.*)-(.*-.*-.*)\/(.*)

    ...    1

    ...    2

    ...    3

    ...    4

    ${unit}=    Set Variable    ${contentRange}[0][0]



    Check Content Range Part Equal    ${unit}    date-time

    # temporalQ.lastN=5 against a 20-instance fixture: instances remain

    # beyond the page in descending order → rel="intervalbefore" pointer.

    Check Pagination Link    ${response.headers}    intervalbefore    5



Setup Initial Entities

    ${first_temporal_entity_representation_id}=    Generate Random Vehicle Entity Id

data/temporalEntities/entity-operations-paginated-query.jsonld

0 → 100644
+11 −0
Changes for data/temporalEntities/entity-operations-paginated-query.jsonld: 11 added lines, 0 removed lines.
Original line number Diff line number Diff line 
{

    "type": "Query",

    "entities": [{

        "type": "Vehicle"

    }],

    "temporalQ": {

       "timerel":"before",

       "timeAt":"2020-08-02T12:05:00Z",

       "lastN": 5

    }

}