ETSI's Bug Tracker - Part 01: TTCN-3 Core Language
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0007448Part 01: TTCN-3 Core LanguageNew Featurepublic08-07-2016 12:1412-12-2016 18:22
ReporterGyorgy Rethy 
Assigned ToGyorgy Rethy 
PriorityhighSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Versionv4.8.1 (published 2016-07) 
Target Versionv4.9.1 (published 2017-05)Fixed in Versionv4.9.1 (published 2017-05) 
Clause Reference(s)Many
Source (company - Author)L.M.Ericsson
Summary0007448: Allowing multiple encodings for TTCN-3 types
DescriptionToday the type of encoding of the test data is not fixed in the specification in all cases. The simplest example is that IoT common service level data is specified at the abstract level and can be encoded by XML or by JSON, in the discretion of the vendor. The abstract data structure can well be defined in TTCN-3, however the different possible types of encoding is not supported by the language.
Therefore TTCN-3 shall, as well, be able to encode and decode the same abstract data structures in different encodings.

We ask to resolve the CR with urgency, as oneM2M is currently writing the conformance test suite, where this problem has been raised. The project will finish by end of the year, that means they need tool support ASAP.

Proposed solution can be found in the attached file.
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
related to 0007445closed Tomas Urban Part 01: TTCN-3 Core Language usage of encode/variant attributes should be enhanced 
related to 0007353closed Gyorgy Rethy Part 01: TTCN-3 Core Language attributes should be attachable to their definition only 
related to 0007451closed Gyorgy Rethy Part 01: TTCN-3 Core Language Multiple occurrence of an attribute 
related to 0007459closed Gyorgy Rethy Part 01: TTCN-3 Core Language the overwriting rules for attributes and the examples should be written more consistent 
related to 0007524closed Tomas Urban Part 06: TTCN-3 Control Interface TCI extension for multiple attributes 
Attached Filesdocx Allowing multiple encodings of TTCN PA3.docx (30,489) 08-07-2016 12:16
http://oldforge.etsi.org/mantis/file_download.php?file_id=3412&type=bug
docx CR7448-1.docx (167,050) 16-08-2016 16:21
http://oldforge.etsi.org/mantis/file_download.php?file_id=3464&type=bug
docx CR7448-V2.docx (118,202) 17-08-2016 15:15
http://oldforge.etsi.org/mantis/file_download.php?file_id=3476&type=bug
docx CR7448-3.docx (170,626) 18-08-2016 09:43
http://oldforge.etsi.org/mantis/file_download.php?file_id=3486&type=bug
docx CR7448-4.docx (129,174) 18-08-2016 15:28
http://oldforge.etsi.org/mantis/file_download.php?file_id=3492&type=bug
docx CR7448-5.docx (129,427) 19-08-2016 13:24
http://oldforge.etsi.org/mantis/file_download.php?file_id=3499&type=bug
docx CR7448-6.docx (127,120) 19-08-2016 14:35
http://oldforge.etsi.org/mantis/file_download.php?file_id=3503&type=bug
docx CR7448-7.docx (185,850) 16-11-2016 11:21
http://oldforge.etsi.org/mantis/file_download.php?file_id=3532&type=bug
docx CR7448-8.docx (132,164) 16-11-2016 15:09
http://oldforge.etsi.org/mantis/file_download.php?file_id=3541&type=bug
docx CR7448-9.docx (133,532) 17-11-2016 09:50
http://oldforge.etsi.org/mantis/file_download.php?file_id=3547&type=bug
Issue History
Date ModifiedUsernameFieldChange
08-07-2016 12:14Gyorgy RethyNew Issue
08-07-2016 12:16Gyorgy RethyFile Added: Allowing multiple encodings of TTCN PA3.docx
11-07-2016 10:27Jacob Wieland - SpirentRelationship addedrelated to 0007445
11-07-2016 10:28Jacob Wieland - SpirentRelationship addedrelated to 0007353
18-07-2016 09:59Jens GrabowskiAssigned To => Tomas Urban
18-07-2016 09:59Jens GrabowskiStatusnew => assigned
19-07-2016 16:10Tomas UrbanNote Added: 0013993
19-07-2016 16:11Tomas UrbanAssigned ToTomas Urban => Gyorgy Rethy
19-07-2016 16:11Tomas UrbanStatusassigned => feedback
20-07-2016 12:03Gyorgy RethyNote Added: 0014005
20-07-2016 12:03Gyorgy RethyStatusfeedback => assigned
20-07-2016 12:04Gyorgy RethyNote Edited: 0014005bug_revision_view_page.php?bugnote_id=14005#r302
20-07-2016 12:07Gyorgy RethyNote Edited: 0014005bug_revision_view_page.php?bugnote_id=14005#r303
20-07-2016 12:07Gyorgy RethyAssigned ToGyorgy Rethy => Tomas Urban
20-07-2016 12:09Gyorgy RethyNote Edited: 0014005bug_revision_view_page.php?bugnote_id=14005#r304
20-07-2016 16:07Tomas UrbanNote Added: 0014012
20-07-2016 16:08Tomas UrbanAssigned ToTomas Urban => Gyorgy Rethy
20-07-2016 16:08Tomas UrbanStatusassigned => feedback
21-07-2016 08:25Tomas UrbanNote Edited: 0014012bug_revision_view_page.php?bugnote_id=14012#r306
21-07-2016 09:46Gyorgy RethyNote Added: 0014024
21-07-2016 09:46Gyorgy RethyStatusfeedback => assigned
21-07-2016 09:46Gyorgy RethyAssigned ToGyorgy Rethy => Tomas Urban
02-08-2016 10:50Martin HauchNote Added: 0014041
02-08-2016 13:03Gyorgy RethyNote Added: 0014042
02-08-2016 13:46Gyorgy RethyNote Added: 0014043
02-08-2016 13:46Gyorgy RethyNote Edited: 0014042bug_revision_view_page.php?bugnote_id=14042#r312
16-08-2016 16:21Tomas UrbanFile Added: CR7448-1.docx
16-08-2016 16:24Tomas UrbanNote Added: 0014114
16-08-2016 16:24Tomas UrbanAssigned ToTomas Urban => Jens Grabowski
16-08-2016 16:24Tomas UrbanStatusassigned => confirmed
16-08-2016 16:27Tomas UrbanRelationship addedrelated to 0007451
17-08-2016 11:05Jacob Wieland - SpirentTarget Version => v4.9.1 (published 2017-05)
17-08-2016 15:15Jens GrabowskiFile Added: CR7448-V2.docx
17-08-2016 15:17Jens GrabowskiNote Added: 0014134
17-08-2016 15:17Jens GrabowskiAssigned ToJens Grabowski => Tomas Urban
17-08-2016 15:17Jens GrabowskiStatusconfirmed => assigned
18-08-2016 09:43Tomas UrbanFile Added: CR7448-3.docx
18-08-2016 09:48Tomas UrbanNote Added: 0014150
18-08-2016 09:48Tomas UrbanAssigned ToTomas Urban => Kristóf Szabados
18-08-2016 09:48Tomas UrbanStatusassigned => confirmed
18-08-2016 09:49Tomas UrbanRelationship addedrelated to 0007459
18-08-2016 15:28Jacob Wieland - SpirentFile Added: CR7448-4.docx
18-08-2016 15:29Jacob Wieland - SpirentNote Added: 0014158
18-08-2016 16:44Gyorgy RethyNote Added: 0014161
19-08-2016 13:24Kristóf SzabadosFile Added: CR7448-5.docx
19-08-2016 13:26Kristóf SzabadosNote Added: 0014178
19-08-2016 13:26Kristóf SzabadosAssigned ToKristóf Szabados => Tomas Urban
19-08-2016 13:26Kristóf SzabadosStatusconfirmed => assigned
19-08-2016 14:35Kristóf SzabadosFile Added: CR7448-6.docx
19-08-2016 14:36Kristóf SzabadosNote Added: 0014183
14-11-2016 15:38Jens GrabowskiNote Added: 0014230
16-11-2016 11:01Tomas UrbanFile Added: CR7448-7.docx
16-11-2016 11:02Tomas UrbanNote Added: 0014259
16-11-2016 11:02Tomas UrbanAssigned ToTomas Urban => Jacob Wieland - Spirent
16-11-2016 11:02Tomas UrbanStatusassigned => confirmed
16-11-2016 11:15Tomas UrbanRelationship addedrelated to 0007524
16-11-2016 11:21Tomas UrbanFile Deleted: CR7448-7.docx
16-11-2016 11:21Tomas UrbanFile Added: CR7448-7.docx
16-11-2016 15:09Jacob Wieland - SpirentFile Added: CR7448-8.docx
16-11-2016 15:10Jacob Wieland - SpirentNote Added: 0014274
16-11-2016 15:10Jacob Wieland - SpirentStatusconfirmed => resolved
16-11-2016 15:10Jacob Wieland - SpirentFixed in Version => v4.9.1 (published 2017-05)
16-11-2016 15:10Jacob Wieland - SpirentResolutionopen => fixed
16-11-2016 15:10Jacob Wieland - SpirentAssigned ToJacob Wieland - Spirent => Gyorgy Rethy
17-11-2016 09:50Kristóf SzabadosAssigned ToGyorgy Rethy => Kristóf Szabados
17-11-2016 09:50Kristóf SzabadosStatusresolved => feedback
17-11-2016 09:50Kristóf SzabadosResolutionfixed => reopened
17-11-2016 09:50Kristóf SzabadosFile Added: CR7448-9.docx
17-11-2016 09:52Kristóf SzabadosNote Added: 0014288
17-11-2016 09:52Kristóf SzabadosAssigned ToKristóf Szabados => Jens Grabowski
17-11-2016 09:52Kristóf SzabadosStatusfeedback => assigned
17-11-2016 11:53Jens GrabowskiNote Added: 0014300
17-11-2016 11:54Jens GrabowskiStatusassigned => resolved
17-11-2016 11:54Jens GrabowskiResolutionreopened => fixed
17-11-2016 11:54Jens GrabowskiAssigned ToJens Grabowski => Gyorgy Rethy
12-12-2016 18:22Gyorgy RethyNote Added: 0014395
12-12-2016 18:22Gyorgy RethyStatusresolved => closed

Notes
(0013993)
Tomas Urban   
19-07-2016 16:10   
The part of the proposal concerning the way how different encodings and associated variants are specified (multiple encode statements, variants with an encode prefix and a colon) is acceptable without any objections.

In the case of selection the actual encoding the STF proposes to use the static approach. We assume that there's (one or more global PDU) containing potentially complicated structure of elements with multiple encodings and variants. The desired encoding and the associated set of variants will be selected by creating an alias type (or template) with a dedicated "filtering" attribute. Please check the following example:

type Payload {
    XSD.Integer foo,
    XSD.Float bar
} with {
  encode "XML";
  encode "Jason";
  variant "XML":"name as uncapitalized"
  variant "Jason":"name as 'dt'"
}

type record PDU {
  Payload payload
} with {
  encode "XML";
  encode "Jason";
  variant "XML":"element"; // example of an XML variant
  variant "JSON":"name as 'pd'"; // example of a Jason variant
}

type PDU XmlPDU with { encode @only "XML" }
// This definition removes all Jason related attributes and keeps XML (on all
// levels), so that the XmlPDU type is equal to:
// type record XmlPDU {
// Payload payload
// } with {
// encode "XML";
// variant "element";
// variant(payload) "name as uncapitalized";
// }

type PDU JasonPDU { encode @only "JSON" }
// type record JasonPDU {
// Payload payload
// } with {
// encode "Jason";
// variant "name as 'pd'";
// variant(payload) "name as 'dt'";
// }

Could you please comment if the @only filter would work for you? It should be possible to write a universal test suite for XML and Jason with it. You will still need two different receive branches, but handling can be common for both of them as the type structures are equal.

The main advantage of the @only modifier is that it doesn't require complicated changes in the receiving, sending, encoding and decoding statements and the impact on the TCI will be minimal. It is also easily applicable to protocol stacks.
(0014005)
Gyorgy Rethy   
20-07-2016 12:03   
(edited on: 20-07-2016 12:09)
Depending on what the question is:
- if @only is proposed *in addition* what the CR requests (i.e. allow types with multiple encodings, where the actual codec is chosen by the TE based on a runtime configuration parameter), it is acceptable, but is useless, superfluous; it doesn't solve dynamic encoding control either;
- if all the multiple encoding problem is wanted to be solved by aliases, it is not acceptable. It is principally wrong approach. It would not need any language extension, just a clarification of overwriting rules for encode and variant attributes for aliases.

Considering the below type definitions;
The requested solution looks like:

template PDU t_PDU := { payload := { foo := 42, bar := 5.0 }

function f() runs on CT {
  P.send (t_PDU);
  alt {
      [] P.receive (PDU:?){...} //note, that receiving a "wrongly" encoded message, i.e. of encoded by the 'another' encoding causes a runtime error here
  }
  ...
}

----------------------------------------------------------------------
The code with the proposed alias solution would look like:

template XmlPDU t_PDUxml := { payload := { foo := 42, bar := 5.0 }
template JasonPDU t_PDUjson := t_PDUxml;

function f() runs on CT {
  if (modulepar_encoding=="XML") {P.send (t_PDUxml)};
  elseif (modulepar_encoding=="JSON") {P.send (t_PDUjson);
  alt {
      [modulepar_encoding=="XML"] P.receive (XmlPDU :?){...} //note, that if receiving receiving a PDU encoded with "Jason" it will cause a deadlock of the component here
      [modulepar_encoding=="Jason"] P.receive (JasonPDU:?){...}// the same for receiving an "XML" encoded message
//Pls. note, if "XML" encoding is expected from the SUT, but it uses "Jason" instead, this shall be a fail verdict as well!
  }
  ...
}

Drawbacks of the proposed alias solution:
- huge amount of superfluous code, to be written for nothing, with all its readability and other consequences (the TE could choose the needed encoding itself, instead of this, the user shall explicitly control it from the TTCN-3 code);
- horrible maintain-ability: if a new, 3rd (or the 2nd) encoding is introduced, the "whole" TTCN-3 code has to be updated: new type aliases, new template aliases, new if~ and alt branches everywhere... if adding a new if~ or alt branch is forgotten at a single place, or a wrong template alias is used (a typical copy-paste error), the error cannot be discovered @ compile time, only @ runtime, by a wrong test case result (if, at all);
for example, in standard test suites, where the code is typically written by the standard body, but is not executed, this would cause a nightmare.
(0014012)
Tomas Urban   
20-07-2016 16:07   
(edited on: 21-07-2016 08:25)
We propose the following solution to the dynamic problem:

a) Keep the @only modifier for static filtering (for the users that might find some use of it).
b) Introduce a dynamic solution using a new encode statement:
( PortReference | "all" "port" | "self") "." encode "(" (Type | all) "," Expression ")"

The encode statement would activate an encoding filter for a particular port, all port or a whole component. The filter would apply to a particular type, its part (useful for protocol stacks) or to all messages passed to the codec. It would basically do the same thing as the @only modifier, but dynamicly.

Example for our XML/Jason case:
modulepar universal charstring ENCODING; // set to "XML" by TCI-TM
self.encode(PDU, ENCODING); // Disables all encodings that are not "XML"
    // and variants related to them
p.send(PDU:{...}}; // When the message is sent out, only the "XML" encode
    // attribute and related "XML" variants are visible to the codec

// in order to go through all encodings:
type record of universal charstring TEnc;
var TEnc v_enc := {...} // dynamically specify acceptable encodings
var integer i := 0;
p.encode(PDU, v_enc[i]);
alt {
  [] p.receive(PDU:?)
  // [] ... other alternatives
  [else] {
     i := i + 1;
     if (i < lengthof(v_enc)) {
       p.encode(PDU, v_enc[i]);
       repeat;
     }
  }
}

The advantage of this approach is that we don't have to change the syntax of the sending/receiving operations and of the enc_value and dec_value functions. It contains support of protocol stacks as an extended type reference can be used as the first parameter.

Gyorgy, could you please say if it is acceptable for you?
(0014024)
Gyorgy Rethy   
21-07-2016 09:46   
If I understand correctly:
- several encode attributes can be attached to a type (group module ...) and variant attributes can be scoped with the syntax proposed by the CR;
- P.encode(PDU,modPar_encoding); activates a concrete encoding to a given port
- all port.encode(PDU,modPar_encoding); activates a concrete encoding to all ports of the given component, but NOT for encvalue/decvalue-s
- self.encode(PDU,modPar_encoding); activates a concrete encoding to all ports AND encvalue/decvalue-s of the given component
- in addition, a new optional parameter will allow select/change the default encoding for individual encvalue/decvalue/encvalue_unichar/decvalue_unichar calls

This solution, as a whole, is fine with me.
(0014041)
Martin Hauch   
02-08-2016 10:50   
Questions:
Where are the types defined, which should be used for "JSON" and "XML" coding?
Definitions from XSD:
"Part 9: Using XML schema with TTCN-3" does not define variant "XML":"element"; (or other "XML" renamings for elements or attributes). Current "Part 11: Using JSON with TTCN-3" does not define variant "JSON":name as 'pd'"
Should the syntax for variant attributes be changed to bind the variant-atrribute information to an encoding-rule in case of more than 1 encoding-rule is assigned?
How should JSON-variants be added to these from XSD converted types?
How should xsd-defined types be encoded in JSON?

Definitions from TTCN:
I think this should work. The encode- and variant-attributes could be defined for all types (as shown in your example), respecting the possible attribute-values defined in the documents "Part 9: Using XML schema with TTCN-3" and "Part 11: Using JSON with TTCN-3".
(0014042)
Gyorgy Rethy   
02-08-2016 13:03   
(edited on: 02-08-2016 13:46)
Thanks Martin, good points.
I think both Part-9 and draft Part-11 shall be updated to allow generating the variant scopings automatically (for backward compatibility and readibility reasons this should be a user-choice converter configuration parameter).
As JSON doesn't have a schema, no JSON encoding instructions can be generated automatically, they always need to be added manually to TTCN-3 files. Therefore, the output of an XML converter need to be decorated with JSON variant attributes manually.
(0014043)
Gyorgy Rethy   
02-08-2016 13:46   
During the internal discussions we were also discussing a possible compact syntax to assign an encoding instruction to multiple encode-s, something like

variant (resourceName){"XML","JSON"}:"name as 'rn'";

at the end this feature wasn't included into the initial proposal, to keep it simple.

However, now ETSI has raised the same issue, as the main attribute to be used in the IoT conformance tests is name as, which is the same in both XML and JSON. Another ETSI's point is that e.g. if a new, 3rd encoding is added, it is easier to add the name of the new encoding to the list of encodes, than copy-paste-change all the variants for the new encoding, which would make the code long, nasty and hardly readable.

Could you please also consider a compact syntax, to add an encoding instruction to multiple encodings by a single variant?
(0014114)
Tomas Urban   
16-08-2016 16:24   
Proposal uploaded. The document also contains a resolution for all related CRs. Please check.
(0014134)
Jens Grabowski   
17-08-2016 15:17   
Please check again. Ericsson should also proofread before resolving the issue.
(0014150)
Tomas Urban   
18-08-2016 09:48   
I am fine with the corrections. I only made a couple of minor formating changes in the document.

Kristóf, could you please look at the document too and discuss the matter with György?
(0014158)
Jacob Wieland - Spirent   
18-08-2016 15:29   
I have added/changed multiple places according to our discussions.

Please review.
(0014161)
Gyorgy Rethy   
18-08-2016 16:44   
Hi, in general the proposal looks fine, but a few comments/questions:

1) Regarding the @local modifier I’m not sure if it’s really needed: while it complicates the language and the tools, I can see little practical use. Let see the example from the standard:
    module M {
        type record MyRec {
            integer field1,
            integer field1,
        } with { encode @local "CodecB" }
        // the record type MyRec will be encoded with CodecB, but its fields with CodecA,
        // because local attribute CodecB doesn’t affect elements of the MyRec type.
    } with { encode "CodecA" }

In reality, typically protocol messages and their fields have encode attributes, in which case fields are practically always some constrained types, having their own type definitions (and thus inheriting the attribute associated to the group/module). Maybe boolean fields are exceptions, but I haven’t seen any of them being encoded differently than the enclosing type; and if, it is more intuitive to add their encoding to the field directly (than to understand what @local means):
    module M {

type integer (0..255) Intfield;

type record MyRec {
  Intfield field1,
  Intfield field2,
  boolean field3
} with { encode "CodecB";//this does not apply to fields field1 and field2 acc. to the previous rule
         encode (field3) “CodecA”}
} with { encode "CodecA" }

2) I think it would be more secure if variant attribute without encode reference is allowed only, if there is just a single encode applied to the scope (i.e. not to have "global variant"s). Once a new encoding is added to a module, all variants has to be re-checked anyway, during which it is easy to copy-paste the encoding scope in front of the already existing variant. But with the current "global variant" rule, variants forgotten to be cross-checked and scoped, will automatically start effecting on the new encoding as well.

3) If considering the example:
    // Modifying list of allowed encodings
    type Int Int2 with {
        encode "CodecA"; // variants "CodecA"."Rule1" and "GlobalRule" are kept
        encode "CodecC"; variant "CodecC"."Rule6"; // new encoding and related variant
        // "CodecB" encoding and related variant are discarded as "CodecB" is not
        // explicitly referenced
    }

It is unclear why the @only modifier is needed.
In the example
    // Selecting encoding with an @only modifier
    type MyRecord MyRecord2 with {
        encode @only (field1) "CodecB"; // field1 will be encoded with CodecB
        encode @only "CodecA" // the record and field2 will be encoded with CodecA
    }
Applying encode (field1) "CodecB" and encode "CodecA" would have exactly the same effect: whatever other encodings applied to MyRecord and MyRecord.field1 are simply discarded.

4) Has ETSI’s request to allow associating a variant to several encodings with a single statement been considered? Example (syntax below is appropriate, any suitable syntax will do it):
    module M {
type record MyRecX {
  Intfield field1,
  Intfield field2,
  boolean field3
} with { variant(field1) {"CodecA”,”CodecB”}.”name as ‘f1’”;
         variant(field2) {"CodecA”,”CodecB”}.”name as ‘f2’”;
         variant(field3) {"CodecA”,”CodecB”}.”name as ‘f3’”}

} with { encode "CodecA" }
(0014178)
Kristóf Szabados   
19-08-2016 13:26   
Added clarification, that variants without explicit encoding relation are implicitly related to all encodings in effect.
(0014183)
Kristóf Szabados   
19-08-2016 14:36   
Added example to show the difference between encodes with and without @only.
(0014230)
Jens Grabowski   
14-11-2016 15:38   
STF discussion:
1) @local will be kept
2) no global variants for multiple encodings
4) one variant for multiple encodings by explicit naming

3) @only will be skipped (no different scope level, same scope level)

- Precedence rules for 3) will be checked for tools represented in STF.
(0014259)
Tomas Urban   
16-11-2016 11:02   
Specification updated according to STF discussion conclusion. Please review.
(0014274)
Jacob Wieland - Spirent   
16-11-2016 15:10   
added a missing closing parenthesis in BNF, otherwise ok, ready for implementation
(0014288)
Kristóf Szabados   
17-11-2016 09:52   
Corrected some typos.
Added a note before example 4.
Corrected example in section 27.9 (in the testcase ContainerType needed to be changed to PDU as that is the name of the type)
(0014300)
Jens Grabowski   
17-11-2016 11:53   
OK, ready for implementation.
(0014395)
Gyorgy Rethy   
12-12-2016 18:22   
Added to draft V4.8.2