Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
# CONTRIBUTING
## TTCN-3 Conformance Tests Overview
These are preliminary instructions on how to use the TTCN-3 conformance test suite.
Unlike ATSs in the usual context, this ATS does _not_ provide any means for test
automation, but the TTCN-3 files provide the test inputs for the TTCN-3 tool - in this case
the TTCN-3 tool is the IUT. This means that test automation has to be somehow scripted, however,
from the provided ATS it should be clear how the output should be interpreted.
### ATS organization
The ATS is organized according to the clauses of the TTCN-3 standard. There are
either positive syntactic tests, negative syntactic tests, positive semantic
tests, or negative semantic tests.
Except for the negative syntactic tests, all TTCN-3 files from the other three
categories are always syntactically correct. In addition, the positive
syntactic tests are designed to be semantically correct as well. The negative
semantic tests are designed to violate the semantics only in the one property
that is subject of the test - at least to the degree that it is possible.
Because all test cases are syntactically correct (except for the negative
syntactic tests), the semantic and negative semantic tests are syntactically
correct as well.
Every TTCN-3 module corresponds to one TTCN-3 conformance test. Where more than
one module is needed, the TTCN-3 file contains multiple modules.
### Document tags
Every module is annotated with document tags. Of relevance for the test automation tooling
is the @verdict tag on top of each module (tag usage adaption in the TTCN-3 maintenance STF
is pending). It is composed of three parts: the tag, the conformance verdict, and keywords
regarding the expected output. For example:
@verdict pass accept, ttcn3verdict:pass
Hence the format of this tag is a three column entry with `@verdict`, `pass`, and
`accept, ttcn3verdict:pass, manual:'Freetext validation'`.
In order to reach the `pass` verdict, the TTCN-3 tool under test, the IUT, must
accept the TTCN-3 module as test input. The result of its execution must be the
TTCN-3 verdict pass. When needed, a manual inspection of the execution results
is required. Therefore, if the TTCN-3 verdict is pass after the execution of
the module, the IUT passes the conformance test. If the tool output is anything
else, or it does not comply with the manual inspection, it fails the
conformance test.
The keywords in use to describe the third column, i.e. the expected output by
the IUT, are (currently) as follows:
reject
accept, noexecution
accept, ttcn3verdict:none
accept, ttcn3verdict:pass
accept, ttcn3verdict:inconc
accept, ttcn3verdict:fail
accept, ttcn3verdict:error
accept, ttcn3verdict:xxx, manual:"Validation inspection, free text" (see Note)
Note: `xxx` for ttcn3verdict is a placeholder for verdict value. The actual test cases must
have one of the allowed values `none`, `pass`, `inconc`, `fail`, or `error` in place of `xxx`.
`reject` implies that the TTCN-3 module is either rejected at compile-time or at execution time.
In the conformance test, we do not differentiate between these two cases as the standard does not
make any statement where semantic checks have to be performed.
`accept, noexecution` implies that the TTCN-3 module should be accepted by the TTCN-3 tool after
the syntactic check. For passing the conformance test, the module simply has to be accepted and
an execution is not necessary.
`accept, ttcn3vercit:xxx` implies that the TTCN-3 module should be accepted by the TTCN-3 tool
after the syntactic check and that an execution should take place. The result of the execution
should be a TTCN-3 verdict and the `xxx` denotes what verdict is the exepcted verdict. If the
verdict differs from the specified `xxx`, the conformance test fails. Otherwise, it passes.
In the usual case, each TTCN-3 file contains only one test case. In these cases the verdict
determination is clear. In a few cases, the TTCN-3 file contains more than one test case.
In that case, the overall conformance verdict is determined according to the TTCN-3 verdict
overwrite rules applied to the results of each test case. Let's say we have two test cases.
The first test case ends with the verdict `fail` and the second one ends with the verdict `pass`.
Then the overall verdict is `fail`.
The manual results inspection is a free text describing how a valid out should
look like. The text is informal since different tools have different logging
formats and facilities. The instruction is surrounded by single or double
quotas on a single line:
@verdict pass, testverdict:pass, manual:'The following elements are logged: charstring "Extra", record { 1, "HELLO"}, integer template "?". Make sure the elements are logged at setverdict, at MTC end, and at test case end.'
All notations are designed to be easily machine readable. Therefore, the test automation using
some sort of scripting will only take a small amount of time.
### Other prerequisites
In order to test communication and matching behavior, we expect that the test cases are executed
using a loopback adapter.