Logo etsi

ETSI's Bug Tracker
Main | My View | View Issues | Change Log | Roadmap | Monitor project
  
Notice: information submitted on the ETSI issue Tracker may be incorporated in ETSI publication(s) and therefore subject to the ETSI IPR policy.

View Issue Details [ Jump to Notes ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007448Part 01: TTCN-3 Core LanguageNew Featurepublic08-07-2016 12:1412-12-2016 18:22
ReporterGyorgy Rethy 
Assigned ToGyorgy Rethy 
PriorityhighSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Versionv4.8.1 (published 2016-07) 
Target Versionv4.9.1 (published 2017-05)Fixed in Versionv4.9.1 (published 2017-05) 
Summary0007448: Allowing multiple encodings for TTCN-3 types
DescriptionToday the type of encoding of the test data is not fixed in the specification in all cases. The simplest example is that IoT common service level data is specified at the abstract level and can be encoded by XML or by JSON, in the discretion of the vendor. The abstract data structure can well be defined in TTCN-3, however the different possible types of encoding is not supported by the language.
Therefore TTCN-3 shall, as well, be able to encode and decode the same abstract data structures in different encodings.

We ask to resolve the CR with urgency, as oneM2M is currently writing the conformance test suite, where this problem has been raised. The project will finish by end of the year, that means they need tool support ASAP.

Proposed solution can be found in the attached file.
TagsNo tags attached.
Clause Reference(s)Many
Source (company - Author)L.M.Ericsson
Attached Filesdocx file icon Allowing multiple encodings of TTCN PA3.docx [^] (30,489 bytes) 08-07-2016 12:16
docx file icon CR7448-1.docx [^] (167,050 bytes) 16-08-2016 16:21
docx file icon CR7448-V2.docx [^] (118,202 bytes) 17-08-2016 15:15
docx file icon CR7448-3.docx [^] (170,626 bytes) 18-08-2016 09:43
docx file icon CR7448-4.docx [^] (129,174 bytes) 18-08-2016 15:28
docx file icon CR7448-5.docx [^] (129,427 bytes) 19-08-2016 13:24
docx file icon CR7448-6.docx [^] (127,120 bytes) 19-08-2016 14:35
docx file icon CR7448-7.docx [^] (185,850 bytes) 16-11-2016 11:21
docx file icon CR7448-8.docx [^] (132,164 bytes) 16-11-2016 15:09
docx file icon CR7448-9.docx [^] (133,532 bytes) 17-11-2016 09:50

- Relationships
related to 0007445closedTomas Urban Part 01: TTCN-3 Core Language usage of encode/variant attributes should be enhanced 
related to 0007353closedGyorgy Rethy Part 01: TTCN-3 Core Language attributes should be attachable to their definition only 
related to 0007451closedGyorgy Rethy Part 01: TTCN-3 Core Language Multiple occurrence of an attribute 
related to 0007459closedGyorgy Rethy Part 01: TTCN-3 Core Language the overwriting rules for attributes and the examples should be written more consistent 
related to 0007524closedTomas Urban Part 06: TTCN-3 Control Interface TCI extension for multiple attributes 

-  Notes
(0013993)
Tomas Urban (developer)
19-07-2016 16:10

 The part of the proposal concerning the way how different encodings and associated variants are specified (multiple encode statements, variants with an encode prefix and a colon) is acceptable without any objections.

In the case of selection the actual encoding the STF proposes to use the static approach. We assume that there's (one or more global PDU) containing potentially complicated structure of elements with multiple encodings and variants. The desired encoding and the associated set of variants will be selected by creating an alias type (or template) with a dedicated "filtering" attribute. Please check the following example:

type Payload {
    XSD.Integer foo,
    XSD.Float bar
} with {
  encode "XML";
  encode "Jason";
  variant "XML":"name as uncapitalized"
  variant "Jason":"name as 'dt'"
}

type record PDU {
  Payload payload
} with {
  encode "XML";
  encode "Jason";
  variant "XML":"element"; // example of an XML variant
  variant "JSON":"name as 'pd'"; // example of a Jason variant
}

type PDU XmlPDU with { encode @only "XML" }
// This definition removes all Jason related attributes and keeps XML (on all
// levels), so that the XmlPDU type is equal to:
// type record XmlPDU {
// Payload payload
// } with {
// encode "XML";
// variant "element";
// variant(payload) "name as uncapitalized";
// }

type PDU JasonPDU { encode @only "JSON" }
// type record JasonPDU {
// Payload payload
// } with {
// encode "Jason";
// variant "name as 'pd'";
// variant(payload) "name as 'dt'";
// }

Could you please comment if the @only filter would work for you? It should be possible to write a universal test suite for XML and Jason with it. You will still need two different receive branches, but handling can be common for both of them as the type structures are equal.

The main advantage of the @only modifier is that it doesn't require complicated changes in the receiving, sending, encoding and decoding statements and the impact on the TCI will be minimal. It is also easily applicable to protocol stacks.
(0014005)
Gyorgy Rethy (reporter)
20-07-2016 12:03
edited on: 20-07-2016 12:09

 Depending on what the question is:
- if @only is proposed *in addition* what the CR requests (i.e. allow types with multiple encodings, where the actual codec is chosen by the TE based on a runtime configuration parameter), it is acceptable, but is useless, superfluous; it doesn't solve dynamic encoding control either;
- if all the multiple encoding problem is wanted to be solved by aliases, it is not acceptable. It is principally wrong approach. It would not need any language extension, just a clarification of overwriting rules for encode and variant attributes for aliases.

Considering the below type definitions;
The requested solution looks like:

template PDU t_PDU := { payload := { foo := 42, bar := 5.0 }

function f() runs on CT {
  P.send (t_PDU);
  alt {
      [] P.receive (PDU:?){...} //note, that receiving a "wrongly" encoded message, i.e. of encoded by the 'another' encoding causes a runtime error here
  }
  ...
}

----------------------------------------------------------------------
The code with the proposed alias solution would look like:

template XmlPDU t_PDUxml := { payload := { foo := 42, bar := 5.0 }
template JasonPDU t_PDUjson := t_PDUxml;

function f() runs on CT {
  if (modulepar_encoding=="XML") {P.send (t_PDUxml)};
  elseif (modulepar_encoding=="JSON") {P.send (t_PDUjson);
  alt {
      [modulepar_encoding=="XML"] P.receive (XmlPDU :?){...} //note, that if receiving receiving a PDU encoded with "Jason" it will cause a deadlock of the component here
      [modulepar_encoding=="Jason"] P.receive (JasonPDU:?){...}// the same for receiving an "XML" encoded message
//Pls. note, if "XML" encoding is expected from the SUT, but it uses "Jason" instead, this shall be a fail verdict as well!
  }
  ...
}

Drawbacks of the proposed alias solution:
- huge amount of superfluous code, to be written for nothing, with all its readability and other consequences (the TE could choose the needed encoding itself, instead of this, the user shall explicitly control it from the TTCN-3 code);
- horrible maintain-ability: if a new, 3rd (or the 2nd) encoding is introduced, the "whole" TTCN-3 code has to be updated: new type aliases, new template aliases, new if~ and alt branches everywhere... if adding a new if~ or alt branch is forgotten at a single place, or a wrong template alias is used (a typical copy-paste error), the error cannot be discovered @ compile time, only @ runtime, by a wrong test case result (if, at all);
for example, in standard test suites, where the code is typically written by the standard body, but is not executed, this would cause a nightmare.
(0014012)
Tomas Urban (developer)
20-07-2016 16:07
edited on: 21-07-2016 08:25

 We propose the following solution to the dynamic problem:

a) Keep the @only modifier for static filtering (for the users that might find some use of it).
b) Introduce a dynamic solution using a new encode statement:
( PortReference | "all" "port" | "self") "." encode "(" (Type | all) "," Expression ")"

The encode statement would activate an encoding filter for a particular port, all port or a whole component. The filter would apply to a particular type, its part (useful for protocol stacks) or to all messages passed to the codec. It would basically do the same thing as the @only modifier, but dynamicly.

Example for our XML/Jason case:
modulepar universal charstring ENCODING; // set to "XML" by TCI-TM
self.encode(PDU, ENCODING); // Disables all encodings that are not "XML"
    // and variants related to them
p.send(PDU:{...}}; // When the message is sent out, only the "XML" encode
    // attribute and related "XML" variants are visible to the codec

// in order to go through all encodings:
type record of universal charstring TEnc;
var TEnc v_enc := {...} // dynamically specify acceptable encodings
var integer i := 0;
p.encode(PDU, v_enc[i]);
alt {
  [] p.receive(PDU:?)
  // [] ... other alternatives
  [else] {
     i := i + 1;
     if (i < lengthof(v_enc)) {
       p.encode(PDU, v_enc[i]);
       repeat;
     }
  }
}

The advantage of this approach is that we don't have to change the syntax of the sending/receiving operations and of the enc_value and dec_value functions. It contains support of protocol stacks as an extended type reference can be used as the first parameter.

Gyorgy, could you please say if it is acceptable for you?
(0014024)
Gyorgy Rethy (reporter)
21-07-2016 09:46

 If I understand correctly:
- several encode attributes can be attached to a type (group module ...) and variant attributes can be scoped with the syntax proposed by the CR;
- P.encode(PDU,modPar_encoding); activates a concrete encoding to a given port
- all port.encode(PDU,modPar_encoding); activates a concrete encoding to all ports of the given component, but NOT for encvalue/decvalue-s
- self.encode(PDU,modPar_encoding); activates a concrete encoding to all ports AND encvalue/decvalue-s of the given component
- in addition, a new optional parameter will allow select/change the default encoding for individual encvalue/decvalue/encvalue_unichar/decvalue_unichar calls

This solution, as a whole, is fine with me.
(0014041)
Martin Hauch (reporter)
02-08-2016 10:50

 Questions:
Where are the types defined, which should be used for "JSON" and "XML" coding?
Definitions from XSD:
"Part 9: Using XML schema with TTCN-3" does not define variant "XML":"element"; (or other "XML" renamings for elements or attributes). Current "Part 11: Using JSON with TTCN-3" does not define variant "JSON":name as 'pd'"
Should the syntax for variant attributes be changed to bind the variant-atrribute information to an encoding-rule in case of more than 1 encoding-rule is assigned?
How should JSON-variants be added to these from XSD converted types?
How should xsd-defined types be encoded in JSON?

Definitions from TTCN:
I think this should work. The encode- and variant-attributes could be defined for all types (as shown in your example), respecting the possible attribute-values defined in the documents "Part 9: Using XML schema with TTCN-3" and "Part 11: Using JSON with TTCN-3".
(0014042)
Gyorgy Rethy (reporter)
02-08-2016 13:03
edited on: 02-08-2016 13:46

 Thanks Martin, good points.
I think both Part-9 and draft Part-11 shall be updated to allow generating the variant scopings automatically (for backward compatibility and readibility reasons this should be a user-choice converter configuration parameter).
As JSON doesn't have a schema, no JSON encoding instructions can be generated automatically, they always need to be added manually to TTCN-3 files. Therefore, the output of an XML converter need to be decorated with JSON variant attributes manually.
(0014043)
Gyorgy Rethy (reporter)
02-08-2016 13:46

 During the internal discussions we were also discussing a possible compact syntax to assign an encoding instruction to multiple encode-s, something like

variant (resourceName){"XML","JSON"}:"name as 'rn'";

at the end this feature wasn't included into the initial proposal, to keep it simple.

However, now ETSI has raised the same issue, as the main attribute to be used in the IoT conformance tests is name as, which is the same in both XML and JSON. Another ETSI's point is that e.g. if a new, 3rd encoding is added, it is easier to add the name of the new encoding to the list of encodes, than copy-paste-change all the variants for the new encoding, which would make the code long, nasty and hardly readable.

Could you please also consider a compact syntax, to add an encoding instruction to multiple encodings by a single variant?
(0014114)
Tomas Urban (developer)
16-08-2016 16:24

 Proposal uploaded. The document also contains a resolution for all related CRs. Please check.
(0014134)
Jens Grabowski (manager)
17-08-2016 15:17

 Please check again. Ericsson should also proofread before resolving the issue.
(0014150)
Tomas Urban (developer)
18-08-2016 09:48

 I am fine with the corrections. I only made a couple of minor formating changes in the document.

Kristóf, could you please look at the document too and discuss the matter with György?
(0014158)
Jacob Wieland - Spirent (reporter)
18-08-2016 15:29

 I have added/changed multiple places according to our discussions.

Please review.
(0014161)
Gyorgy Rethy (reporter)
18-08-2016 16:44

 Hi, in general the proposal looks fine, but a few comments/questions:

1) Regarding the @local modifier I’m not sure if it’s really needed: while it complicates the language and the tools, I can see little practical use. Let see the example from the standard:
    module M {
        type record MyRec {
            integer field1,
            integer field1,
        } with { encode @local "CodecB" }
        // the record type MyRec will be encoded with CodecB, but its fields with CodecA,
        // because local attribute CodecB doesn’t affect elements of the MyRec type.
    } with { encode "CodecA" }

In reality, typically protocol messages and their fields have encode attributes, in which case fields are practically always some constrained types, having their own type definitions (and thus inheriting the attribute associated to the group/module). Maybe boolean fields are exceptions, but I haven’t seen any of them being encoded differently than the enclosing type; and if, it is more intuitive to add their encoding to the field directly (than to understand what @local means):
    module M {

type integer (0..255) Intfield;

type record MyRec {
  Intfield field1,
  Intfield field2,
  boolean field3
} with { encode "CodecB";//this does not apply to fields field1 and field2 acc. to the previous rule
         encode (field3) “CodecA”}
} with { encode "CodecA" }

2) I think it would be more secure if variant attribute without encode reference is allowed only, if there is just a single encode applied to the scope (i.e. not to have "global variant"s). Once a new encoding is added to a module, all variants has to be re-checked anyway, during which it is easy to copy-paste the encoding scope in front of the already existing variant. But with the current "global variant" rule, variants forgotten to be cross-checked and scoped, will automatically start effecting on the new encoding as well.

3) If considering the example:
    // Modifying list of allowed encodings
    type Int Int2 with {
        encode "CodecA"; // variants "CodecA"."Rule1" and "GlobalRule" are kept
        encode "CodecC"; variant "CodecC"."Rule6"; // new encoding and related variant
        // "CodecB" encoding and related variant are discarded as "CodecB" is not
        // explicitly referenced
    }

It is unclear why the @only modifier is needed.
In the example
    // Selecting encoding with an @only modifier
    type MyRecord MyRecord2 with {
        encode @only (field1) "CodecB"; // field1 will be encoded with CodecB
        encode @only "CodecA" // the record and field2 will be encoded with CodecA
    }
Applying encode (field1) "CodecB" and encode "CodecA" would have exactly the same effect: whatever other encodings applied to MyRecord and MyRecord.field1 are simply discarded.

4) Has ETSI’s request to allow associating a variant to several encodings with a single statement been considered? Example (syntax below is appropriate, any suitable syntax will do it):
    module M {
type record MyRecX {
  Intfield field1,
  Intfield field2,
  boolean field3
} with { variant(field1) {"CodecA”,”CodecB”}.”name as ‘f1’”;
         variant(field2) {"CodecA”,”CodecB”}.”name as ‘f2’”;
         variant(field3) {"CodecA”,”CodecB”}.”name as ‘f3’”}

} with { encode "CodecA" }
(0014178)
Kristóf Szabados (reporter)
19-08-2016 13:26

 Added clarification, that variants without explicit encoding relation are implicitly related to all encodings in effect.
(0014183)
Kristóf Szabados (reporter)
19-08-2016 14:36

 Added example to show the difference between encodes with and without @only.
(0014230)
Jens Grabowski (manager)
14-11-2016 15:38

 STF discussion:
1) @local will be kept
2) no global variants for multiple encodings
4) one variant for multiple encodings by explicit naming

3) @only will be skipped (no different scope level, same scope level)

- Precedence rules for 3) will be checked for tools represented in STF.
(0014259)
Tomas Urban (developer)
16-11-2016 11:02

 Specification updated according to STF discussion conclusion. Please review.
(0014274)
Jacob Wieland - Spirent (reporter)
16-11-2016 15:10

 added a missing closing parenthesis in BNF, otherwise ok, ready for implementation
(0014288)
Kristóf Szabados (reporter)
17-11-2016 09:52

 Corrected some typos.
Added a note before example 4.
Corrected example in section 27.9 (in the testcase ContainerType needed to be changed to PDU as that is the name of the type)
(0014300)
Jens Grabowski (manager)
17-11-2016 11:53

 OK, ready for implementation.
(0014395)
Gyorgy Rethy (reporter)
12-12-2016 18:22

 Added to draft V4.8.2

- Issue History
Date Modified Username Field Change
08-07-2016 12:14 Gyorgy Rethy New Issue
08-07-2016 12:16 Gyorgy Rethy File Added: Allowing multiple encodings of TTCN PA3.docx
11-07-2016 10:27 Jacob Wieland - Spirent Relationship added related to 0007445
11-07-2016 10:28 Jacob Wieland - Spirent Relationship added related to 0007353
18-07-2016 09:59 Jens Grabowski Assigned To => Tomas Urban
18-07-2016 09:59 Jens Grabowski Status new => assigned
19-07-2016 16:10 Tomas Urban Note Added: 0013993
19-07-2016 16:11 Tomas Urban Assigned To Tomas Urban => Gyorgy Rethy
19-07-2016 16:11 Tomas Urban Status assigned => feedback
20-07-2016 12:03 Gyorgy Rethy Note Added: 0014005
20-07-2016 12:03 Gyorgy Rethy Status feedback => assigned
20-07-2016 12:04 Gyorgy Rethy Note Edited: 0014005 View Revisions
20-07-2016 12:07 Gyorgy Rethy Note Edited: 0014005 View Revisions
20-07-2016 12:07 Gyorgy Rethy Assigned To Gyorgy Rethy => Tomas Urban
20-07-2016 12:09 Gyorgy Rethy Note Edited: 0014005 View Revisions
20-07-2016 16:07 Tomas Urban Note Added: 0014012
20-07-2016 16:08 Tomas Urban Assigned To Tomas Urban => Gyorgy Rethy
20-07-2016 16:08 Tomas Urban Status assigned => feedback
21-07-2016 08:25 Tomas Urban Note Edited: 0014012 View Revisions
21-07-2016 09:46 Gyorgy Rethy Note Added: 0014024
21-07-2016 09:46 Gyorgy Rethy Status feedback => assigned
21-07-2016 09:46 Gyorgy Rethy Assigned To Gyorgy Rethy => Tomas Urban
02-08-2016 10:50 Martin Hauch Note Added: 0014041
02-08-2016 13:03 Gyorgy Rethy Note Added: 0014042
02-08-2016 13:46 Gyorgy Rethy Note Added: 0014043
02-08-2016 13:46 Gyorgy Rethy Note Edited: 0014042 View Revisions
16-08-2016 16:21 Tomas Urban File Added: CR7448-1.docx
16-08-2016 16:24 Tomas Urban Note Added: 0014114
16-08-2016 16:24 Tomas Urban Assigned To Tomas Urban => Jens Grabowski
16-08-2016 16:24 Tomas Urban Status assigned => confirmed
16-08-2016 16:27 Tomas Urban Relationship added related to 0007451
17-08-2016 11:05 Jacob Wieland - Spirent Target Version => v4.9.1 (published 2017-05)
17-08-2016 15:15 Jens Grabowski File Added: CR7448-V2.docx
17-08-2016 15:17 Jens Grabowski Note Added: 0014134
17-08-2016 15:17 Jens Grabowski Assigned To Jens Grabowski => Tomas Urban
17-08-2016 15:17 Jens Grabowski Status confirmed => assigned
18-08-2016 09:43 Tomas Urban File Added: CR7448-3.docx
18-08-2016 09:48 Tomas Urban Note Added: 0014150
18-08-2016 09:48 Tomas Urban Assigned To Tomas Urban => Kristóf Szabados
18-08-2016 09:48 Tomas Urban Status assigned => confirmed
18-08-2016 09:49 Tomas Urban Relationship added related to 0007459
18-08-2016 15:28 Jacob Wieland - Spirent File Added: CR7448-4.docx
18-08-2016 15:29 Jacob Wieland - Spirent Note Added: 0014158
18-08-2016 16:44 Gyorgy Rethy Note Added: 0014161
19-08-2016 13:24 Kristóf Szabados File Added: CR7448-5.docx
19-08-2016 13:26 Kristóf Szabados Note Added: 0014178
19-08-2016 13:26 Kristóf Szabados Assigned To Kristóf Szabados => Tomas Urban
19-08-2016 13:26 Kristóf Szabados Status confirmed => assigned
19-08-2016 14:35 Kristóf Szabados File Added: CR7448-6.docx
19-08-2016 14:36 Kristóf Szabados Note Added: 0014183
14-11-2016 15:38 Jens Grabowski Note Added: 0014230
16-11-2016 11:01 Tomas Urban File Added: CR7448-7.docx
16-11-2016 11:02 Tomas Urban Note Added: 0014259
16-11-2016 11:02 Tomas Urban Assigned To Tomas Urban => Jacob Wieland - Spirent
16-11-2016 11:02 Tomas Urban Status assigned => confirmed
16-11-2016 11:15 Tomas Urban Relationship added related to 0007524
16-11-2016 11:21 Tomas Urban File Deleted: CR7448-7.docx
16-11-2016 11:21 Tomas Urban File Added: CR7448-7.docx
16-11-2016 15:09 Jacob Wieland - Spirent File Added: CR7448-8.docx
16-11-2016 15:10 Jacob Wieland - Spirent Note Added: 0014274
16-11-2016 15:10 Jacob Wieland - Spirent Status confirmed => resolved
16-11-2016 15:10 Jacob Wieland - Spirent Fixed in Version => v4.9.1 (published 2017-05)
16-11-2016 15:10 Jacob Wieland - Spirent Resolution open => fixed
16-11-2016 15:10 Jacob Wieland - Spirent Assigned To Jacob Wieland - Spirent => Gyorgy Rethy
17-11-2016 09:50 Kristóf Szabados Assigned To Gyorgy Rethy => Kristóf Szabados
17-11-2016 09:50 Kristóf Szabados Status resolved => feedback
17-11-2016 09:50 Kristóf Szabados Resolution fixed => reopened
17-11-2016 09:50 Kristóf Szabados File Added: CR7448-9.docx
17-11-2016 09:52 Kristóf Szabados Note Added: 0014288
17-11-2016 09:52 Kristóf Szabados Assigned To Kristóf Szabados => Jens Grabowski
17-11-2016 09:52 Kristóf Szabados Status feedback => assigned
17-11-2016 11:53 Jens Grabowski Note Added: 0014300
17-11-2016 11:54 Jens Grabowski Status assigned => resolved
17-11-2016 11:54 Jens Grabowski Resolution reopened => fixed
17-11-2016 11:54 Jens Grabowski Assigned To Jens Grabowski => Gyorgy Rethy
12-12-2016 18:22 Gyorgy Rethy Note Added: 0014395
12-12-2016 18:22 Gyorgy Rethy Status resolved => closed

MantisBT 1.2.14 [^]
Copyright © 2000 - 2024 MantisBT Team
Powered by Mantis Bugtracker