feat(scripts): measure the API coverage of a run against TS 104-176

# API coverage of the Test Suite

How much of the NGSI-LD HTTP binding (ETSI TS 104-176) a run of the Test Suite exercises, measured from what the

run sent: every request the Test Cases make goes through RequestsLibrary, which logs the method, the URL, the

headers and the response status in `output.xml`. The script maps each request to its operation in the spec and

counts three things per operation:

- **the operation** - a method on a resource, such as `GET /entities/{entityId}`. A `GET/HEAD` section of the

  spec is two operations, as the two are tested separately.

- **the URL parameters** of the operation's table in the spec that at least one request gave.

- **the status codes** the spec lists for the operation that at least one response had.

It also lists the headers each operation was sent with, the status codes that came back but are not in the

operation's list nor among the general HTTP errors of clause 6.3.2 (405, 406, 411, 413, 415, 501), and the requests that match no operation (Test Cases that send a malformed URL on purpose).

The operations and the URL parameters depend on the Test Cases only. The status codes also depend on the broker

the suite ran against.

## Files

| File                          | What it is                                                                 |

|-------------------------------|----------------------------------------------------------------------------|

| `api_coverage.py`             | The measurement: `output.xml` and the inventory in, the coverage out       |

| `ts-104-176-inventory.json`   | The inventory: resources, methods, operations, URL parameters and status codes of TS 104-176, with the spec commit it was built from |

| `build_inventory.py`          | Builds the inventory from the Markdown source of TS 104-176                |

| `test_api_coverage.py`        | Checks that `api_coverage.py` parses the messages RequestsLibrary logs: `python3 scripts/api_coverage/test_api_coverage.py` |

## Run it

After a run of the suite (see [Execute the Test Suite](../../README.md#execute-the-test-suite)):

```

python3 scripts/api_coverage/api_coverage.py ./results/output.xml

```

The first lines are the totals; a table with one row per operation follows:

```

requests 4533, not an operation of the inventory 22

operations (method x resource, HEAD on its own): 50/90 exercised

URL parameters per operation: 84/275 exercised

status codes per operation, as the spec lists them: 135/286 seen

clause    method resource                                             req tests params codes  missing params | missing codes

7.2.3.1   GET    entities/                                            231   231  18/28   2/5  collation containedBy csf ... | 201 203 501

7.2.3.1   HEAD   entities/                                              0     0   0/28   0/3  attrs collation containedBy ... | 204 400 501

...

```

- `req` is the number of requests to the operation, and `tests` the number of Test Cases that sent them.

- `params` and `codes` give what was exercised out of what the spec lists.

- After the `|` come the status codes no response had.

### What a branch adds

Write the coverage of a run on `develop` to a file, run the branch, and compare the two:

```

python3 scripts/api_coverage/api_coverage.py develop/output.xml --json develop.json

python3 scripts/api_coverage/api_coverage.py branch/output.xml --baseline develop.json

```

```

against the baseline: operations <before> -> <after>, parameters <before> -> <after>, status codes <before> -> <after>

  + <METHOD> <resource>: [the operation] [?<parameter> ...] [<status code> ...]

  - <METHOD> <resource>: ...

```

`+` is what the branch exercises that the baseline does not, and `-` is what it no longer does. A branch that

builds the inventory again also gets a `+` line per operation the baseline's inventory does not have, and a `-`

line per operation it dropped.

### Only part of the suite

The script counts whatever the `output.xml` holds, so a run of one directory measures that directory:

```

robot --outputdir ./results ./TP/NGSI-LD/ContextInformation/Consumption

python3 scripts/api_coverage/api_coverage.py ./results/output.xml

```

## A new version of the spec

Build the inventory again from a clone of the spec repository:

```

git clone https://forge.etsi.org/rep/data/specifications/ts-104-176.git

python3 scripts/api_coverage/build_inventory.py ts-104-176 > scripts/api_coverage/ts-104-176-inventory.json

```

The script reads the `md/clause-*.md` files:

- `## N Resource: <path>` headings give the resources.

- `### N.N.N.n <METHOD>` headings below them give the methods.

- The tables titled "URL parameters" give the parameters.

- `NNN Reason` cells give the status codes.

The commit it read goes into the inventory's `commit` field.

# API coverage of a test-suite run: which operations of the NGSI-LD HTTP binding (ETSI TS 104-176) the

# requests of a run exercised, with which URL parameters and headers, and which of the status codes the

# spec lists for each operation came back. It reads only the inventory (built from the spec by

# build_inventory.py) and the run's output.xml - RequestsLibrary logs every request and response there -

# so it measures the suite, whichever broker it ran against.

#

# Launch it using such a command:

#   python3 scripts/api_coverage/api_coverage.py ./results/output.xml

#   python3 scripts/api_coverage/api_coverage.py ./results/output.xml --json coverage.json

#   python3 scripts/api_coverage/api_coverage.py ./results/output.xml --baseline develop.json

#

# --baseline <json> takes the --json of an earlier run (develop, say) and prints what this run adds and

# loses: operations, parameters and status codes.

import argparse

import json

import os

import re

import sys

from urllib.parse import urlsplit, parse_qsl

from robot.api import ExecutionResult, ResultVisitor

HERE = os.path.dirname(os.path.abspath(__file__))

REQUEST = re.compile(r'^(\w+) Request : url=(\S+) (?:\(redirected\) )?\n path_url=\S+ \n headers=(\{.*?\}) \n body=', re.S)

RESPONSE = re.compile(r'^(\w+) Response : url=\S+ \n status=(\d+)')

HEADER = re.compile(r"'([A-Za-z-]+)': '([^']*)'")

# URL parameters whose values are counted one by one (the others: only that they were given)

VALUED = {'format', 'options', 'geometry', 'georel', 'timerel', 'details', 'kind', 'aggrMethods', 'count', 'local',

          'lang', 'csf', 'timeproperty', 'join', 'entityMapLifetime'}

# Headers of TS 104-176 clause 6 that are counted per operation

HEADERS = ['Link', 'Accept', 'Content-Type', 'NGSILD-Tenant', 'NGSILD-EntityMap', 'NGSILD-Date', 'Prefer',

           'If-Match', 'If-None-Match', 'Accept-Language', 'NGSILD-Path', 'NGSILD-Warning', 'NGSILD-Scope', 'Via']

# The HTTP errors of TS 104-176 clause 6.3.2 that any operation may answer; the operations' own sections do not

# list them, so they are not reported as status codes the spec does not list

GENERAL_CODES = {'405', '406', '411', '413', '415', '501'}

def load_operations(inventory):

    """One operation per (method, resource): a GET/HEAD section of the spec is two, and the HEAD's status codes

    are 204 and the GET's errors"""

    ops = []

    for section in inventory['operations']:

        rx = '^' + re.sub(r'\\\{[^}]+\\\}', '[^/]+', re.escape(section['resource'].strip('/'))) + '/?$'

        for method in section['methods'].split('/'):

            method = method.strip()

            codes = section['codes']

            if method == 'HEAD':

                # TS 104-176 clause 5: a HEAD answers 204 No Content on success, and the errors of the GET

                codes = ['204'] + [c for c in codes if c >= '300']

            ops.append({'clause': section['clause'], 'method': method, 'resource': section['resource'],

                        'operation': section['operation'], 'params': section['params'], 'codes': codes,

                        'rx': re.compile(rx), 'literal': len(re.sub(r'\{[^}]+\}', '', section['resource'])),

                        'requests': 0, 'tests': set(), 'seenParams': {}, 'seenCodes': {}, 'seenHeaders': {}})

    return ops

def count(d, key):

    d[key] = d.get(key, 0) + 1

class RequestVisitor(ResultVisitor):

    def __init__(self, ops):

        self.ops = ops

        self.test = None

        self.pending = None

        self.total = 0

        self.unmatched = {}

    def start_test(self, test):

        self.test = test.full_name

        self.pending = None

    def visit_message(self, msg):

        text = msg.message

        m = REQUEST.match(text)

        if m:

            self.request(m.group(1), m.group(2), m.group(3))

            return

        m = RESPONSE.match(text)

        if m and self.pending is not None:

            count(self.pending['seenCodes'], m.group(2))

            self.pending = None

    def request(self, method, url, headers):

        self.total += 1

        self.pending = None

        parts = urlsplit(url)

        path = re.sub(r'^.*?/ngsi-ld/v1/', '', parts.path).strip('/')

        candidates = [o for o in self.ops if o['method'] == method and o['rx'].match(path)]

        if not candidates:

            count(self.unmatched, f"{method} {re.sub(r'urn:[^/]+', '{id}', path)}")

            return

        op = max(candidates, key=lambda o: o['literal'])  # entities/{id}/attrs over entities/{id}/{x}

        op['requests'] += 1

        op['tests'].add(self.test)

        self.pending = op

        for name, value in parse_qsl(parts.query, keep_blank_values=True):

            seen = op['seenParams'].setdefault(name, {})

            if name in VALUED:

                for v in value.split(','):

                    count(seen, v.split(';')[0])

            else:

                count(seen, '*')

        given = {h.lower(): v for h, v in HEADER.findall(headers)}

        for h in HEADERS:

            if h.lower() in given:

                v = given[h.lower()]

                count(op['seenHeaders'].setdefault(h, {}), v.split(';')[0] if h in ('Accept', 'Content-Type') else '*')

def report(ops, visitor):

    out = {'requests': visitor.total, 'unmatched': visitor.unmatched, 'operations': []}

    for o in ops:

        out['operations'].append({

            'clause': o['clause'], 'method': o['method'], 'resource': o['resource'], 'operation': o['operation'],

            'requests': o['requests'], 'tests': sorted(o['tests']),

            'specParams': o['params'], 'paramsSeen': o['seenParams'],

            'paramsMissing': [p for p in o['params'] if p not in o['seenParams']],

            'specCodes': o['codes'], 'codesSeen': o['seenCodes'],

            'codesMissing': [c for c in o['codes'] if c not in o['seenCodes']],

            'codesNotInSpec': [c for c in o['seenCodes'] if c not in o['codes'] and c not in GENERAL_CODES],

            'headersSeen': o['seenHeaders']})

    return out

def totals(out):

    ops = out['operations']

    return {'operations': (sum(1 for o in ops if o['requests'] > 0), len(ops)),

            'params': (sum(len(o['specParams']) - len(o['paramsMissing']) for o in ops),

                       sum(len(o['specParams']) for o in ops)),

            'codes': (sum(len(o['specCodes']) - len(o['codesMissing']) for o in ops),

                      sum(len(o['specCodes']) for o in ops))}

def key(o):

    return f"{o['method']} {o['resource']}"

def print_text(out):

    t = totals(out)

    print(f"requests {out['requests']}, not an operation of the inventory {sum(out['unmatched'].values())}")

    print(f"operations (method x resource, HEAD on its own): {t['operations'][0]}/{t['operations'][1]} exercised")

    print(f"URL parameters per operation: {t['params'][0]}/{t['params'][1]} exercised")

    print(f"status codes per operation, as the spec lists them: {t['codes'][0]}/{t['codes'][1]} seen")

    print()

    print(f"{'clause':9} {'method':6} {'resource':50} {'req':>5} {'tests':>5} {'params':>6} {'codes':>5}  "

          f"missing params | missing codes")

    for o in out['operations']:

        params = f"{len(o['specParams']) - len(o['paramsMissing'])}/{len(o['specParams'])}"

        codes = f"{len(o['specCodes']) - len(o['codesMissing'])}/{len(o['specCodes'])}"

        print(f"{o['clause']:9} {o['method']:6} {o['resource'][:50]:50} {o['requests']:5} {len(o['tests']):5} "

              f"{params:>6} {codes:>5}  {' '.join(o['paramsMissing'])} | {' '.join(o['codesMissing'])}")

    extra = [(key(o), c) for o in out['operations'] for c in o['codesNotInSpec']]

    if extra:

        print()

        print('status codes seen that the spec does not list for the operation:')

        for k, c in extra:

            print(f'  {c}  {k}')

    if out['unmatched']:

        print()

        print('requests that are no operation of the inventory:')

        for k, n in sorted(out['unmatched'].items()):

            print(f'  {n:4}  {k}')

def print_delta(out, base):

    t, b = totals(out), totals(base)

    print(f"against the baseline: operations {b['operations'][0]} -> {t['operations'][0]}, "

          f"parameters {b['params'][0]} -> {t['params'][0]}, status codes {b['codes'][0]} -> {t['codes'][0]}")

    old = {key(o): o for o in base['operations']}

    for o in out['operations']:

        p = old.get(key(o))

        if p is None:

            print(f"  + {key(o)}: not in the baseline's inventory" + (', exercised' if o['requests'] else ''))

            continue

        gained = []

        lost = []

        if o['requests'] and not p['requests']:

            gained.append('the operation')

        if p['requests'] and not o['requests']:

            lost.append('the operation')

        gained += [f'?{x}' for x in p['paramsMissing'] if x not in o['paramsMissing']]

        lost += [f'?{x}' for x in o['paramsMissing'] if x not in p['paramsMissing']]

        gained += [x for x in p['codesMissing'] if x not in o['codesMissing']]

        lost += [x for x in o['codesMissing'] if x not in p['codesMissing']]

        if gained:

            print(f"  + {key(o)}: {' '.join(gained)}")

        if lost:

            print(f"  - {key(o)}: {' '.join(lost)}")

    new = {key(o) for o in out['operations']}

    for k, p in old.items():

        if k not in new:

            print(f"  - {k}: not in this run's inventory" + (', was exercised' if p['requests'] else ''))

def main():

    parser = argparse.ArgumentParser(description='API coverage of a test-suite run (ETSI TS 104-176)')

    parser.add_argument('output', help="the run's output.xml")

    parser.add_argument('--inventory', default=os.path.join(HERE, 'ts-104-176-inventory.json'))

    parser.add_argument('--json', help='write the coverage as JSON to this file')

    parser.add_argument('--baseline', help="an earlier run's --json: print what this run adds and loses")

    args = parser.parse_args()

    with open(args.inventory, encoding='utf-8') as f:

        ops = load_operations(json.load(f))

    visitor = RequestVisitor(ops)

    ExecutionResult(args.output).visit(visitor)

    out = report(ops, visitor)

    if args.json:

        with open(args.json, 'w', encoding='utf-8') as f:

            json.dump(out, f, indent=1)

    if args.baseline:

        with open(args.baseline, encoding='utf-8') as f:

            print_delta(out, json.load(f))

    else:

        print_text(out)

if __name__ == '__main__':

    sys.exit(main())

# Builds the API inventory from the Markdown source of ETSI GS/TS 104-176 (the NGSI-LD HTTP binding):

# every resource, its methods, the operation, the URL parameters and the status codes of each method's

# section.

#

# Launch it using such a command:

#   python3 scripts/api_coverage/build_inventory.py <ts-104-176 checkout> > scripts/api_coverage/ts-104-176-inventory.json

#

# <ts-104-176 checkout> is a clone of https://forge.etsi.org/rep/data/specifications/ts-104-176 - the

# Markdown is in its md/ directory. The commit it was built from goes into the inventory.

import glob

import json

import os

import re

import subprocess

import sys

RESOURCE = re.compile(r'^##\s+([\d.]+)\s+Resource:\s*(.+)$')

METHOD = re.compile(r'^####?\s+([\d.]+)\s+([A-Z][A-Z/ ]+)$')

TABLE = re.compile(r'^\**Table[^:]{0,20}:\s*(.+)$')

CODE = re.compile(r'(?:^|\|)\s*([1-5]\d\d) [A-Z]')

def natural_key(path):

    return [int(x) if x.isdigit() else x for x in re.split(r'(\d+)', path)]

def first_cells(table):

    """The first-column texts of a grid or pipe table: one per line whose first cell is not empty"""

    out = []

    for line in table:

        if not line.startswith('|'):

            continue

        cell = line.split('|')[1].strip()

        if cell and not re.fullmatch(r'[-:= ]+', cell):

            out.append(cell)

    return out

def operation_names(lines, start):

    """The operation(s) of the paragraph after a method heading: 'associated to the operation(s) "X" [and "Y"]'"""

    text = ''

    for line in lines[start:start + 12]:

        if line.startswith(('#', '<!--', ':::')):

            break

        text += ' ' + line.strip()

    m = (re.search(r'operations?\s+((?:"[^"]+"(?:\s*(?:,|and)\s*)?)+)', text)

         or re.search(r'(?:bound|associated) to the\s+((?:"[^"]+"(?:\s*(?:,|and)\s*)?)+)\s*operations?', text))

    names = re.findall(r'"([^"]+)"', m.group(1)) if m else []

    return ' / '.join(re.sub(r'[_`]', '', n) for n in names)

def add_codes(method, line):

    for code in CODE.findall(line):

        if code not in method['codes']:

            method['codes'].append(code)

def parse_clause(path, inventory):

    with open(path, encoding='utf-8') as f:

        lines = f.read().split('\n')

    resource = None

    method = None

    i = 0

    while i < len(lines):

        line = lines[i]

        m = RESOURCE.match(line)

        if m:

            resource = (m.group(1), m.group(2).strip().replace('\\', ''))

            method = None

        m = METHOD.match(line)

        if m and resource and m.group(1).startswith(resource[0]):

            method = {'clause': m.group(1), 'resource': resource[1], 'methods': m.group(2).strip(),

                      'operation': '', 'params': [], 'codes': []}

            inventory.append(method)

            method['operation'] = operation_names(lines, i + 1)

        if method is None:

            i += 1

            continue

        add_codes(method, line)

        m = TABLE.match(line)

        if m:

            title = m.group(1).lower()

            j = i + 1

            while j < len(lines) and not lines[j].startswith(('+', '|')):

                j += 1

            k = j

            while k < len(lines) and lines[k].startswith(('+', '|')):

                k += 1

            rows = first_cells(lines[j:k])[1:]  # the header row out

            if 'url parameter' in title or 'query parameter' in title:

                method['params'] += [r for r in rows if r not in method['params']]

            for row in lines[j:k]:

                add_codes(method, row)

            i = k

            continue

        i += 1

def main():

    if len(sys.argv) != 2:

        sys.exit('usage: build_inventory.py <ts-104-176 checkout>')

    checkout = sys.argv[1]

    inventory = []

    for path in sorted(glob.glob(os.path.join(checkout, 'md', 'clause-*.md')), key=natural_key):

        parse_clause(path, inventory)

    try:

        commit = subprocess.run(['git', '-C', checkout, 'log', '-1', '--format=%H %cs'],

                                capture_output=True, text=True, check=True).stdout.strip()

    except (OSError, subprocess.CalledProcessError):

        commit = ''

    json.dump({'spec': 'ETSI TS 104-176', 'source': 'https://forge.etsi.org/rep/data/specifications/ts-104-176',

               'commit': commit, 'operations': inventory}, sys.stdout, indent=1)

    sys.stdout.write('\n')

if __name__ == '__main__':

    main()

# Checks that api_coverage.py still parses what RequestsLibrary logs: the messages are made by RequestsLibrary's

# own log functions, so an upgrade that changes their format fails here.

#

# Launch it using such a command:

#   python3 scripts/api_coverage/test_api_coverage.py

import os

import sys

import unittest

from types import SimpleNamespace

from unittest import mock

import RequestsLibrary.log

import requests

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

import api_coverage

INVENTORY = {'operations': [

    {'clause': '7.2.3.1', 'resource': 'entities/', 'methods': 'GET/HEAD', 'operation': 'Query Entities',