Merge remote-tracking branch 'origin/develop' into feat/context-source-identity

# NGSI-LD Test Suite — Agent Instructions

## Project Identity

This is the ETSI NGSI-LD conformance test suite. It verifies that a broker correctly implements the NGSI-LD API as

specified in ETSI TS 104 175 (core), TS 104 176 (HTTP binding) and TS 104 243 (MQTT binding)

(formerly ETSI GS CIM 009). Tests are written in Robot Framework 7 and target any broker exposing the NGSI-LD HTTP

API. The canonical upstream is `https://forge.etsi.org/rep/cim/ngsi-ld-test-suite`.

---

## Directory Map

```

TP/NGSI-LD/               Test cases, organised by API group

  CommonBehaviours/       Cross-cutting behaviour TCs (group 043, …)

  ContextInformation/     Core entity/attribute/subscription/temporal TCs

  ContextSource/          Context-source registration and discovery TCs

  DistributedOperations/  Federated/distributed broker TCs

  jsonldContext/          JSON-LD @context management TCs

resources/                Robot keywords (resource files)

  ApiUtils/               Per-operation keyword libraries (.resource)

  AssertionUtils.resource Shared assertion keywords

  HttpUtils.resource      Low-level HTTP helpers

  variables.py            Runtime configuration (URLs, ports, flags)

libraries/                Python helper modules used by Robot

  assertionUtils.py

  dateTimeUtils.py

  ErrorListener.py        Listener that writes failures to errors.log

data/                     Test fixture files (.json / .jsonld)

  entities/

  subscriptions/

  csourceRegistrations/

  jsonldContext/

doc/                      Documentation generation system

  generateDocumentationData.py   Generator entry point

  statisticsDocumentationData.py

  analysis/               Metadata declarations (requests, checks, setup)

    requests.py           Declares endpoint keywords and their docs

    checks.py             Declares assertion keywords and their docs

    initial_setup.py      Declares setup keyword names

    generaterobotdata.py  Maps TC directories to acronyms

  tests/                  Unit tests of the generator — must pass before every commit

  results/                Generator output (created on first run, git-ignored)

scripts/                  Maintenance utilities

  find_unused_test_data.py

  find_tc_using_test_data.py

  apiutils.py

  run_tests.sh / configure.sh

```

---

## Runtime Configuration

All broker URLs and port numbers come from **`resources/variables.py`**. Never hardcode them.

| Variable | Default | Purpose |

|---|---|---|

| `url` | `http://localhost:8080` | Broker base URL (without `/ngsi-ld/v1`) |

| `temporal_api_url` | `http://localhost:8080` | Temporal API base URL (without `/ngsi-ld/v1`) |

| `notification_server_host` | `0.0.0.0` | Host the suite listens on for notifications |

| `notification_server_port` | `8085` | Port for the notification server |

| `mqtt_broker_host` | `127.0.0.1` | MQTT broker host |

| `context_source_host/port` | `0.0.0.0` / `8086` | Context source mock |

| `context_server_host/port` | `0.0.0.0` / `8087` | Context server mock |

| `ngsild_test_suite_context` | (GitHub URL) | JSON-LD context used by the suite |

Override any variable at runtime with `-v <name>:<value>` on the `robot` command line.

---

## Essential Commands

```bash

# Run tests (point url/temporal_api_url at your broker; notification_server_host must be

# reachable from the broker, e.g. the host IP when the broker runs in a container/VM)

python3 -m robot -v url:http://localhost:8080 -v temporal_api_url:http://localhost:8080 \

    -v notification_server_host:<host-ip> --outputdir ./results TP/NGSI-LD/ContextInformation

# Or use /run-suite, which adds the standard flags

# Format all .robot / .resource files in place (Robocop)

robocop format

# Check formatting without modifying files

robocop format --check

# Run the documentation generator unit tests (must pass before every commit)

PYTHONPATH=./doc python -m unittest discover -s doc/tests

# Validate the documentation still generates (must report 0 failures before every commit)

cd doc ; python statisticsDocumentationData.py

# Generate documentation for one TC to inspect it (replace <tc_id> e.g. 032_01)

python doc/generateDocumentationData.py <tc_id>

# Find test data files that are no longer referenced by any TC

python3 scripts/find_unused_test_data.py

# Find TCs that use a given test data file (interactive)

python3 scripts/find_tc_using_test_data.py

# Generate HTML keyword docs for a resource file

python3 -m robot.libdoc resources/ApiUtils/<Resource>.resource api_docs/<Resource>.html

# Generate HTML test case docs

python3 -m robot.testdoc TP/NGSI-LD api_docs/TestCases.html

# Run specific API group

python3 -m robot --outputdir ./results TP/NGSI-LD/ContextInformation

# Rerun only failed TCs from a previous run

python3 -m robot --rerunfailedsuites ./results/output.xml --outputdir ./results .

# Run with failure listener (writes errors.log)

python3 -m robot --listener libraries/ErrorListener.py --outputdir ./results .

```

---

## Naming Conventions

### Test case identifiers

Pattern: `XXX_YY_NN`

- `XXX` — numeric group identifier (e.g. `032`)

- `YY` — subgroup index (e.g. `01`)

- `NN` — sequential permutation number (e.g. `01`, `02`, …)

Example: `032_01_01 Delete Subscription By Id`

### Tags: spec clauses and versions

- `section_reference` tags are the spec document prefix followed by the clause number with underscores:

  `core_` (TS 104 175), `http_` (TS 104 176), `mqtt_` (TS 104 243). E.g. `core_10_2_2`, `http_7_2_3_1`.

- Clause numbers never change across spec versions. A clause tag that doesn't match the TC's folder/operation is a

  typo in the tag, not a version difference: don't re-baseline clauses per `since_v…` version.

- The release tag is `since_vX.Y.Z` for a feature first released in ETSI GS CIM 009 vX.Y.Z, and `since_ts104_vX.Y.Z`

  for a feature that first appears in ETSI TS 104 175/176 vX.Y.Z (e.g. `since_ts104_v1.1.1`, with no GS CIM 009

  counterpart). All test cases in one file must use the same release tag.

- A permutation without a `since_v…` tag covers a feature already present in v1.3.1, so its implicit version is

  v1.3.1. A TC is mixed-version when its permutations differ, counting untagged ones as v1.3.1 (the doc generator

  then logs "Error not all the tags have the same version. Select default v1.3.1").

### File layout

```

TP/NGSI-LD/<Group>/<Subgroup>/<Operation>/<TC_ID>.robot

```

### Commit messages — Conventional Commits

```

<type>(<scope>): <short description in imperative mood, no trailing period>

```

Types: `feat`, `fix`, `chore`, `refactor`, `docs`, `test`

Scopes (optional): API group or subsystem, e.g. `subscription`, `temporal`, `robocop`

---

## Adding a New Test Case — Checklist

1. Create the `.robot` file under `TP/NGSI-LD/<Group>/…` using the template in `CONTRIBUTING.md`.

2. Every permutation must carry **at least two tags**: a `{resource_request_reference}` and one or more `{section_reference}` tags.

3. Setup keywords must expose data via `Set Suite Variable`; teardown keywords must delete **all** data created in setup.

4. Reuse existing keywords from `resources/` before writing new ones.

5. If a new assertion keyword is added, declare it in `doc/analysis/checks.py` (`self.checks`, `self.args`, and a doc method) **in the same change** as the keyword in `resources/AssertionUtils.resource`, never in a follow-up. For a notification-style `Wait for …` keyword, also recognise it in `doc/analysis/parserobotfile.py`. Call `Check …` keywords from the test case body or the test template, not from a user keyword in `*** Keywords ***`: the generator only reads the former, and fails with `It is expected at least 1 Check operation`.

6. If a new endpoint keyword is added, declare it in `doc/analysis/requests.py` (`self.op`, `self.description`, and a doc method).

7. If a new request/query parameter is used on an existing endpoint, add it to that endpoint's `expected_parameters` and its `match` cases in `doc/analysis/requests.py`.

8. If a new setup keyword was introduced, update `doc/analysis/initial_setup.py`.

9. If the TC lives in a brand-new directory, declare it in `doc/analysis/generaterobotdata.py` along with its acronym.

10. Run `python3 scripts/find_unused_test_data.py` — no new unused files should appear.

11. Run `cd doc ; python statisticsDocumentationData.py` — it must report `0` failures (the documentation health check). Use `python doc/generateDocumentationData.py <tc_id>` to inspect a single TC's generated output.

12. Run `PYTHONPATH=./doc python -m unittest discover -s doc/tests` — all tests must pass.

13. Run `robocop format` — commit only after the hook accepts the result.

---

## What Agents Must Never Do

- Hardcode broker URLs or port numbers — always use variables from `resources/variables.py`.

- Disable or override Robocop rules locally (no `# robocop: disable` unless approved).

- Skip the doc generator unit tests (`PYTHONPATH=./doc python -m unittest discover -s doc/tests`) before committing.

- Add test data files under `data/` without verifying they are referenced by a TC.

- Write keywords without checking whether an equivalent already exists in `resources/`.

- Commit without the pre-commit hook having run successfully.

- Add or use a new `Check …` keyword without declaring it in `doc/analysis/checks.py` in the same change (the doc generation then fails with a `KeyError` on the keyword name).

---

name: doc-maintainer

description: Keep the NGSI-LD test suite documentation metadata in sync after test cases are created or modified.

  Use this agent after new .robot files are written or existing ones changed, to update doc/analysis/ declarations and

  keep the generation of the documentation fully working.

---

You are a documentation maintenance specialist for the ETSI NGSI-LD conformance test suite.

## Your Responsibilities

- Register new keywords used by Test Cases so the documentation keeps generating:

  - assertion keywords in `doc/analysis/checks.py`,

  - endpoint keywords (and their request/query parameters) in `doc/analysis/requests.py`,

  - setup/teardown keyword names in `doc/analysis/initial_setup.py`.

- Register new Test Case directories in `doc/analysis/generaterobotdata.py`.

- Keep the documentation generating successfully: `cd doc ; python statisticsDocumentationData.py` must report

  `0` failures.

## What You Must NOT Do

- Modify `.robot` files — that is the `tc-author` agent's responsibility.

- Run the Robot Framework conformance test suites (the broker tests under `TP/`).

> **Deprecated model — do not follow it.** The documentation used to be validated by comparing freshly generated

> JSON against versioned golden files under `doc/files/`, declared in `doc/tests/test_<group>.py`. Those golden

> files are no longer versioned and that unit suite is deprecated. The single source of truth for documentation

> health is now `statisticsDocumentationData.py`. Do **not** add or maintain `doc/tests` declarations, and do

> **not** copy JSON into `doc/files/`.

---

## Documentation System Overview

```

doc/

  generateDocumentationData.py     Generate one TC's JSON (or all) into doc/results/ (git-ignored)

  statisticsDocumentationData.py   Parse every .robot file and report parse failures — the health check

  analysis/

    checks.py             Assertion keyword declarations (self.checks, self.args, doc methods)

    requests.py           Endpoint keyword declarations (self.op, self.description, doc methods)

    initial_setup.py      Setup keyword name declarations

    generaterobotdata.py  Maps TC directories to group acronyms

    parserobotfile.py     Parser; recognises which keywords count as checks (incl. notification keywords)

  results/                Generator output (git-ignored, overwritten on each run)

```

---

## Workflow After a Test Case Is Created or Modified

### Step 1 — Register new keywords (if any were introduced)

**New assertion keyword** (typically begins with `Check` or `Wait for`):

In `doc/analysis/checks.py`, add to `self.checks` and `self.args`, then add a method that returns the

documentation string. Follow the pattern of existing entries exactly. A notification-style `Wait for …` keyword

must **also** be recognised in `doc/analysis/parserobotfile.py` (`get_lines_with_checks`), or it will not be

counted as a check.

**New endpoint keyword:**

In `doc/analysis/requests.py`, add to `self.op` and `self.description`, then add the documentation method.

**New request/query parameter on an existing endpoint:**

Add it to that endpoint's `expected_parameters` **and** add a matching `case` clause in the method (a parameter

listed in `expected_parameters` but missing a `case` still raises).

**New setup keyword:**

In `doc/analysis/initial_setup.py`, add the keyword name to the appropriate list.

### Step 2 — Register new TC directories (if a new directory was created)

If the TC lives in a brand-new directory, declare it with its acronym in `doc/analysis/generaterobotdata.py`,

following the existing entries.

### Step 3 — Validate a single TC (optional, for inspection)

```bash

python doc/generateDocumentationData.py <tc_id>

```

Writes `doc/results/<tc_id>.json`. If it raises, the message names the problem. Common causes:

- a keyword referenced in the `.robot` file but not declared in `checks.py` / `requests.py`,

- a setup keyword name that does not match any entry in `initial_setup.py`,

- a query parameter not listed in the endpoint's `expected_parameters`.

Fix the relevant `doc/analysis/` file and re-run.

### Step 4 — Validate the full generation (the gate)

```bash

cd doc ; python statisticsDocumentationData.py

```

Must report **`0` testcases failed to be parsed**. If any fail, the script lists each failing TC with the cause:

| Symptom in output                                | Cause                                                      | Fix                                                                 |

|--------------------------------------------------|------------------------------------------------------------|---------------------------------------------------------------------|

| `unexpected attribute(s) [...]`                  | A query/request parameter not declared for the endpoint    | Add it to `expected_parameters` + a `case` in `requests.py`         |

| `It is expected at least 1 Check operation`      | The TC's verification keyword is not recognised as a check | Declare it in `checks.py` (and `parserobotfile.py` for `Wait for …`)|

| A keyword name echoed back as the error          | Keyword used in the TC but not declared anywhere           | Add it to `checks.py` or `requests.py`                              |

| `name of the Test Case is not following the pattern` | A `.robot` test-case name does not match its file name | Belongs to `tc-author`; report it, do not edit the `.robot` file    |

Report the full failure output if it does not reach `0`.

---

## Before Declaring Work Done

- [ ] All new keywords registered in the relevant `doc/analysis/` file (and `parserobotfile.py` if a notification keyword).

- [ ] New TC directories declared in `doc/analysis/generaterobotdata.py`.

- [ ] `cd doc ; python statisticsDocumentationData.py` reports `0` failures.

---

name: result-analyst

description: Interpret a Robot Framework output file and produce a structured failure report. Use this agent after

  running a test suite to understand what failed, why, and whether failures are systematic or flaky.

---

You are a test result analysis specialist for the ETSI NGSI-LD conformance test suite.

## Your Responsibilities

- Parse Robot Framework output files (`.xml` or `.out` log files) and produce structured failure reports.

- Group failures by API group and operation.

- Distinguish genuine failures from expected exclusions and flaky tests.

- Identify systematic patterns across failures (e.g. all failures share the same error type).

## What You Must NOT Do

- Modify any project files.

- Attempt to fix failures yourself — your role is diagnosis and reporting only.

---

## Input Formats

You may receive:

1. **An XML output file** (e.g. `results/output.xml`) — the structured Robot output.

2. **A plain-text log file** (e.g. `results.out`, captured with `&> results.out`) — the console output of a robot run.

3. **A path to a results directory** — read `output.xml` from that directory.

---

## Analysis Steps

### 1. Parse the overall summary

Extract:

- Total tests run

- Passed count

- Failed count

- Skipped / excluded count

- Elapsed time

### 2. Separate expected exclusions from genuine failures

Robot tag filters used in this project exclude tests marked:

- `not-implemented` — spec features not yet implemented by the broker under test

- `csrsub-query` — context source registration subscription query (often excluded)

These are not failures. Do not include them in the failure report.

### 3. Group failures by API group

Map each failing TC path to its API group using the directory structure:

```

TP/NGSI-LD/<Group>/<Subgroup>/<Operation>/

```

Groups: `CommonBehaviours`, `ContextInformation`, `ContextSource`, `DistributedOperations`, `jsonldContext`

Produce a count of failures per group, sorted descending.

### 4. For each failure, extract

- TC identifier (e.g. `032_01_01`)

- TC name

- Error message (the `<msg>` content from the `<kw status="FAIL">` element, or the console error line)

- The keyword where the failure occurred (innermost failing keyword name)

- Whether the test was retried (look for multiple `status="FAIL"` entries for the same test — signals flakiness)

### 5. Identify systematic patterns

Group failures by their error message type. Common patterns:

| Pattern | Likely cause |

|---|---|

| `Expected: 200, Got: 404` on all entity operations | Broker URL misconfigured; `url` variable wrong |

| `Expected: 201, Got: 400` on entity creation | Broker rejecting payloads — possibly context resolution failure |

| `Connection refused` on all tests | Broker not running |

| `AssertionError` on specific response field | Broker response differs from spec expectation |

| `Timeout` on notification tests | `notification_server_host` not reachable from broker |

| Failures only after rerun | Flaky tests — likely timing or state isolation issue |

Flag any pattern that affects more than 10% of failures as **systematic**.

### 6. Identify flaky tests

A test is flaky if it passed on rerun but failed initially, or if it shows inconsistent results across runs (look

for duplicate TC entries with different statuses). List these separately.

---

## Report Format

```

## Test Run Summary

| Metric | Value |

|---|---|

| Total run | N |

| Passed | N |

| Failed | N |

| Excluded (not-implemented / tag filter) | N |

| Elapsed | HH:MM:SS |

## Failures by API Group

| Group | Failures |

|---|---|

| ContextInformation | N |

| ContextSource | N |

| … | … |

## Systematic Patterns

<Describe any pattern affecting >10% of failures, with the error message and affected TC count.>

## Flaky Tests

<List any tests that failed initially but passed on rerun, or showed inconsistent results.>

## Top 20 Failures

| TC ID | TC Name | Keyword | Error |

|---|---|---|---|

| 032_01_01 | … | … | … |

| … | … | … | … |

## Recommended Next Steps

<2–5 targeted recommendations based on the patterns found. E.g.: "Check `url` variable", "Verify notification_server_host is reachable from the broker", "Investigate TC 005_03 for timing issue.">

```

Keep the report factual and focused. Do not speculate beyond what the output data supports.

# /check — Run All Quality Gates

Run the four local quality gates in sequence. Stop and report on the first failure.

## Usage

```

/check

```

No arguments. Always runs against the full project.

---

## Gates (in order)

### Gate 1 — Robocop format check

```bash

robocop format --check

```

This checks whether any `.robot` or `.resource` file needs reformatting **without modifying anything**.

- **Pass**: the summary reports `0 files would be reformatted`. Judge on that line, not the exit code: robocop 7.2

  can exit non-zero even when nothing needs reformatting.

- **Fail**: Robocop lists the files that would be changed. Tell the user to run `robocop format` (no `--check`) to apply the fixes, then re-stage and retry.

Do not proceed to Gate 2 if Gate 1 fails.

### Gate 2 — Unused test data check

```bash

python3 scripts/find_unused_test_data.py

```

This lists any files under `data/` that are not referenced by any `.robot` file.

- **Pass**: empty output or output that contains only files that were already present before the current change set.

- **Fail**: one or more new unused files are listed. Tell the user which files are unused and ask whether to delete them or add the missing reference.

Do not proceed to Gate 3 if Gate 2 fails.

### Gate 3 — Documentation generation

```bash

cd doc ; python statisticsDocumentationData.py

```

This parses every `.robot` file in `TP/` and reports how many Test Cases failed to generate their documentation.

- **Pass**: the output reports `0` testcases failed to be parsed.

- **Fail**: one or more TCs failed. Report the listed TCs and the cause shown for each. The most common causes are:

  - A new keyword was used but not registered in `doc/analysis/checks.py` or `doc/analysis/requests.py`.

  - A new request/query parameter is missing from the endpoint's `expected_parameters` in `doc/analysis/requests.py`.

  - A `.robot` test-case name does not match its file name.

Do not proceed to Gate 4 if Gate 3 fails.

### Gate 4 — Documentation unit tests

```bash

PYTHONPATH=./doc python -m unittest discover -s doc/tests

```

Runs the unit tests of the documentation generator (`doc/tests/`).

- **Pass**: the output ends with `OK`.

- **Fail**: report the failing tests and their tracebacks.

---

## Summary Report

After all gates pass, output a single confirmation:

```

✓ Gate 1: Robocop format — OK

✓ Gate 2: Unused test data — OK  (N files checked)

✓ Gate 3: Documentation generation — OK  (0 failures)

✓ Gate 4: Documentation unit tests — OK  (N tests)

All quality gates passed. Ready to commit.

```

If any gate fails, stop at that gate and provide targeted remediation instructions before the user re-runs `/check`.