feat(scripts): measure the API coverage of a run against TS 104-176

Ken Zangelinrequested to merge
feat/api-coverage into develop

Adds scripts/api_coverage: it reads a run's output.xml and the operation tables of TS 104-176, and reports per operation which operations, URL parameters and status codes the run exercised. It needs no broker, and the operations and parameters it reports do not depend on the broker.

On today's develop (1 112 Test Cases):

  • operations: 50 of 90 (method x resource, HEAD on its own)
  • URL parameters: 84 of 275
  • status codes: 135 of 286

--json keeps a run, and --baseline prints what another run adds or loses. Every later MR of the coverage plan uses this to state what it adds.

  • api_coverage.py: the measurement, about 3 s on a full run
  • ts-104-176-inventory.json: the inventory, with the spec commit it was built from
  • build_inventory.py: builds the inventory again from the Markdown of TS 104-176 when a new version comes out
  • README.md: how to run it, plus a section in the main README

To try it: run the suite, then python3 scripts/api_coverage/api_coverage.py results/output.xml.

No Test Case changes.

The plan: https://famzang.com/ttf/ngsi-ld-suite-plan.html

Merge request reports