Logo etsi

ETSI's Bug Tracker
Main | My View | View Issues | Change Log | Roadmap | Monitor project
  
Notice: information submitted on the ETSI issue Tracker may be incorporated in ETSI publication(s) and therefore subject to the ETSI IPR policy.

View Issue Details [ Jump to Notes ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007805Part 07: Using ASN.1 with TTCN-3Technicalpublic07-10-2018 11:5229-12-2019 14:22
ReporterWolfgang Seka 
Assigned ToGyorgy Rethy 
PriorityhighSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Versionv4.8.1 (published 2020-05) 
Summary0007805: Support of ASN.1 sequence with extension containing mandatory fields
DescriptionThe extension of an ASN.1 sequence may consist of optional and mandatory fields.
Nevertheless even if it has mandatory fields the extension as a whole need not to be present in a message.
But it seems that TTCN-3 has no means to express this properly in templates: There is no way to explicitly address an extension and therefore no way to specify its presence.
In addition at least some of the tools do not allow to omit the mandatory fields of an extension (what could be considered as workaround even though it is not really a solution).


EXAMPLE:
ASN.1 type definition
    My-Field-Type ::= INTEGER (0..255)
    My-Extended-Sequence ::= SEQUENCE {
        field1 My-Field-Type,
        ...
        [[
            field2 My-Field-Type,
            field3 My-Field-Type OPTIONAL
        ]]
    }

In TTCN there is no way to have a template explicitly specifying that the extension with field2, field3 shall not be there in a message to be sent out and there is no way to specify something like "{field2:=*, field3:=?} ifpresent" in a receive template.
TagsNo tags attached.
Clause Reference(s)ES 201 873-1 clause 15
Source (company - Author)     MCC160 (Wolfgang)
Attached Filesdocx file icon CR7805.docx [^] (158,802 bytes) 11-10-2018 15:20
docx file icon CR7805_v2.docx [^] (158,108 bytes) 11-10-2018 16:43

- Relationships

-  Notes
(0015195)
Wolfgang Seka (reporter)
07-10-2018 12:44

 Correction: In the example it shall be "{field2:=?, field3:=*} ifpresent"
(0015196)
Jacob Wieland - Spirent (reporter)
08-10-2018 11:35

 The solution could be to map the type to a (private) type with all extension fields optional and then derive a subtype of that type with a restriction:

({field2 := omit, field3 := omit},{field2 := ?})

or to be general, if your extension fields are named f1 ... fn and of those fields g1 .. gm are mandatory:

({f1 := omit, ..., fn := omit},{g1 := ?, ..., gm := ?})
(0015197)
Jacob Wieland - Spirent (reporter)
08-10-2018 11:36
edited on: 08-10-2018 11:40

 The base type could simply be named like the original type with an underscore added at the end. Since there's no hyphen allowed at the end of ASN.1 identifiers, this can't clash with any other mapped names.

Thus, the whole mapping of your example would be:

    type integer My_Field_Type (0..255);

    private type record My_Extended_Sequence_ {
        My_Field_Type field1,
        My_Field_Type field2 optional,
        My_Field_Type field3 optional
    }
    
    type My_Extended_Sequence_ My_Extended_Sequence
    ({field2:= omit, field3 := omit},
     {field2:=?})
(0015198)
Wolfgang Seka (reporter)
08-10-2018 11:44

 An important requirement which I've forgotten to mention:
Any potential solution needs to be backward compatible as we already have hundreds of ASN.1 type definitions with extensions (but most of them with all fields being optional).
(0015199)
Jacob Wieland - Spirent (reporter)
08-10-2018 12:24
edited on: 08-10-2018 12:25

 Well, I assumed that, which is why I proposed my solution which should be fully backward compatible with former mapping and all values that conform to the ASN.1 restrictions.

The additional private type is just necessary because for now it is not possible to add subtype restrictions directly to structured type definitions.
(0015200)
Wolfgang Seka (reporter)
08-10-2018 13:09

 OK - with the proposed mapping it means that the following templates are supported:

    template (value) My_Extended_Sequence cs_My_Extended_Sequence1 := {
      field1 := 42,
      field2 := omit,
      field3 := omit
    };
    template (value) My_Extended_Sequence cs_My_Extended_Sequence2 := {
      field1 := 42,
      field2 := 49,
      field3 := omit // or any valid value
    };
    template (present) My_Extended_Sequence cr_My_Extended_Sequence1 := {
      field1 := ?,
      field2 := omit,
      field3 := omit
    };
    template (present) My_Extended_Sequence cr_My_Extended_Sequence2 := {
      field1 := ?,
      field2 := *,
      field3 := *
    };
the following template may/should cause a compiler error
    template (present) My_Extended_Sequence cr_My_Extended_Sequence3 := {
      field1 := ?,
      field2 := *,
      field3 := ?
    };

=> there would be no issue for the TTCN-3 core language (part 1) but a requirement for compilers what may need clarification in part 9 (as at least for now there are compilers raising an error if field2 is omitted).

=> even though it is not a perfect solution (as it may cause confusions for TTCN writers and readers), assuming that mandatory fields in ASN.1 extensions are an exceptional case, the proposed solution would serve the purpose.
(0015206)
Martin Hauch (reporter)
08-10-2018 14:23

 The mapping to a (new and different) private type seems to be no solution, because the new defined type is not compatible to the original type (see chapter 6.3.2 core language):

record types are compatible if the number, and optional aspect of the fields in the textual order of definition are identical, the types of each field are compatible and the value of each existing field of the value "b" is compatible with the type of its corresponding field in type "A". The value of each field in the value "b" is assigned to the corresponding field in the value of type "A".

Also the encoding result (e.g. in PER encoding) is not the same for both types.

In my opinion the writing of templates for extended ASN.1 types must be rendered more precisely to represent the ASN.1 semantic correctly. E.g. in templates of types with extensions, the extended fields can be set to omit or left out at all, although they are mandatory. This should be posssible, because the import-statement from ASN.1 modules is like

    import from EUTRA_RRC_ASN1_Definitions language "ASN.1:2002" all with {encode "UNALIGNED_PER_OctetAligned"};

This declares the semantic of types from this module kind and corresponding templates!


Templates of the protocol version1 of an extensible type cannot know names of extension-fields and so fields of extensions cannot
be assigned:

// version1
My-Extended-Sequence ::= SEQUENCE {
    field1 My-Field-Type,
    ...
}

template My_Extended_Sequence tVersion1:= { field1:=0 };


In a later protocol version2 the type might be extended. But the old template must still be valid.

// version2
My-Extended-Sequence ::= SEQUENCE {
    field1 My-Field-Type,
    ...
    [[
        field2 My-Field-Type,
        field3 My-Field-Type OPTIONAL
    ]]
}

// version2
template My_Extended_Sequence tVersion2_2:= {
    field1:=0,
    field2:=1, // this field is mandatory if an extension is present
    field3:=omit
}

And if we receive a message with template tVersion2_2 from a version2 SUT by protocol version1, this should match template tVersion1, because all conditions of protocol version1 are fulfilled!
The received message might have additional extended information (which will be skipped by protocol version1), but this was the reason to define the type extensible!

Also, a template definition like tVersion1 in version1 must be valid in version2:

// version2
template My_Extended_Sequence tVersion2_1:= { field1:=0 };

Because of the semantic for extensible ASN.1-types and the backwards compatibility to previous protocol versions, this is a valid template! The mandatory/optional semantic of TTCN-3 can only be applied
for the root-fields of the sequence-type. Template definitions for extended types have to respect the semantic of the ASN.1 definition.
(0015208)
Jens Grabowski (manager)
08-10-2018 15:23

 STF proposal:
Dedicated syntax for record and set types, some examples.

type record MyRecord {
 integer field1,
 extension {
   integer field2,
   integer field3 optional
 }
}
(0015210)
Jacob Wieland - Spirent (reporter)
09-10-2018 07:57

 In response to Martin Hauch's comment:

First of all, type compatibility is given, have no idea what would violate the rules. There's no new type introduced for protocol version 2, the old type is replaced and if you defined the templates properly future-proof, they will be compatible with the new extended type without any change.

Second of all, if you want to have that kind of backward compatibility for receive templates of extensible types, you need to define them future-proof, for instance by using modifies:

template My_Extended_Sequence tVersion1:= modifies ? := { field1:=0 };
// will set all optional fields (of extensions) to *

This will match both a message of protocol version 1 and will ignore additional fields of newer protocol versions.

For send templates, you need to declare them with the attribute optional "implicit omit" to have that kind of backward compatibility.
(0015211)
Jacob Wieland - Spirent (reporter)
09-10-2018 08:14

 We propose to to simply change the extension mandatory fields to TTCN-3 optional fields in the ASN.1 to TTCN-3 mapping and tools should check for the addtional constraints concerning these fields.

This will allow defining templates in the way described above so that they will work with all protocol versions.

Also, if a tool doesn't check for the additional constraints, at least it won't reject values/templates statically which define the mandatory fields with omit anymore.
(0015212)
Jacob Wieland - Spirent (reporter)
09-10-2018 08:38

 Regarding your question about how to express templates that check for constraints on extensions only if they are there you can use the following coding pattern for each extension:

template My_Extended_Sequence ifpresentExt1(template My_Extended_Sequence t) :=
  (t, modifies ? := {field2 := omit, field3 := omit})

Each template that checks just for some constraints on the extension part should be formulated like this:

template My_Extended_Sequence mwExt1 :=
  modifies ? := { field2 := ..., field3 := ... }

Then template ifpresentExt1(mwExt1) checks the constraints for extension 1 only if at least one of its fields is present.

With advanced matching templates, you can then combine such templates with other templates (say, if you want to check for constraints on multiple extensions simultaneaously)
(0015213)
Martin Hauch (reporter)
09-10-2018 08:47

 RESPONSE to Jakob Wieland (15210,15211)

The type
    private type record My_Extended_Sequence_ {
        My_Field_Type field1,
        My_Field_Type field2 optional,
        My_Field_Type field3 optional
    }
is not compatible to
    My-Extended-Sequence ::= SEQUENCE {
        field1 My-Field-Type,
        ...
        [[
            field2 My-Field-Type,
            field3 My-Field-Type OPTIONAL
        ]]
    }
because field2 must be present if the extension group is used. The TTCN-3 type also allows omit as value!
The TTCN-3 type does not represent the extensions and groups, so encoding and decoding cannot be handled correctly.

template MyExtended_Sequence_ tInvalid:= {field1:=1, field2:=omit, field3:=3};
This is not a valid template for the original type definition My-Extended-Sequence in the ASN.1-module, but for type My_Extended_Sequence_!
(0015214)
Jacob Wieland - Spirent (reporter)
09-10-2018 09:04

 Sorry, the type is compatible with what is described by the ASN.1 type. In general, the extension fields are all optional, they just need to be present under some circumstances.

There is no way in TTCN-3 to express such conditional optionality on the type level (unless using subtype restrictions, as proposed in comment http://oldforge.etsi.org/mantis/view.php?id=7805#c15197 [^]).

Subtype restrictions do not have an effect on type compatibility (other than that empty types are not allowed).

template MyExtended_Sequence_ tInvalid:= {field1:=1, field2:=omit, field3:=3}

is an invalid template as it would violate the subtyping constraints: not both extension fields are omit, therefore, field2 must be matched by ? which is only the case if it is not omit, therefore, any tool that would check for the proposed subtyping constraints would reject the template as invalid.
(0015215)
Martin Hauch (reporter)
09-10-2018 09:21

 RESPONSE to Jens Grabowski (15208)

Mapping the ASN.1 extensible types to a new type-definition does not solve the problem.
1. No backward compatibility, because definitions of templates for existing extended types are no longer valid.
2. The result of encoding a value of the new type is generally not the same as the encoding of the original type.
3. New rules for renaming fields have to be assigned, because "extension" could also be the name of a root field or later defined extensions.

Again the definition of templates for extensible ASN.1 types simply has to be described more precisely, and type mapping as proposed is not necessary.

Templates for extensible ASN.1 types allow to leave out mandatory fields of extensions. This ensures the backward compatibility for
templates that are defined using only root-fields (e.g. templates of the 1st version of the ASN.1 module definition). Semantic-Checker
for TTCN-3 and ASN.1 should be able to recognize extended type-fields and allow to leave out these fields in template definition. And the
semantic check should also be able to recognize incorrect templates for an extended ASN.1 type.

Look the following extended type:

My-Extended-Sequence ::= SEQUENCE {
    field1 My-Field-Type,
    ...
    [[
        field2 My-Field-Type,
        field3 My-Field-Type OPTIONAL
    ]],
    field4 My-Field-Type,
    field5 MyField-Type OPTIONAL,
    field6 My-Field-Type
}


template My_Extended_Sequence t1:={field1:=1};
// is valid, all root-fields are assigned, the value is not extended in case of sending.
// an extended received value is not known in version 1 and the extended fields cannot be checked (skipped).
// an extended received value in a later version can be checked if the template uses the extension-fields.
// This template could be used for all current and future versions of this type.

template My_Extended_Sequence t2:={field1:=1, field2:= 2, field3:=omit};
// is valid,all root-fields are assigned, all fields of the first group-extension are assigned.

template My_Extended_Sequence t3_invalid:= {field1:=1, field4:=4};
// is invalid because extension field4 is assigned and previous defined extensions are missing.

template My_Extended_Sequence t3:= {field1:=1, field2:= 2, field3:=omit, field4:=4};
// is valid, all root-fields are assigned, all fields of the first group-extension are assigned, extension field4 is assigned.

template MyExtended_Sequence t3:= {field1:=1, field2:= 2, field3:=omit, field4:=4, field6:=6};
// Either the template is valid, because field5 is optional (and is handled implicit as omit),
// or it is not valid because the template is not completely assigned ( omit must be assigned explicit to field5).

template MyExtended_Sequence t4:= {field1:=1, field2:= 2, field3:=omit, field4:=4, field5:= omit, field6:=6};
// is valid, all root-fields are assigned, all extension-fields up to field6 are assigned, omit is allowed for optional field5.

template MyExtended_Sequence t5:= {field1:=1, field2:= 2, field3:=omit, field4:=4, field5:= 5};
// is valid, all root-fields are assigned, all extension-fields up to field5 are assigned.

This semantic has to be checked by the TTCN-3 compiler and ASN.1 tools. This should be possible, as the
TTCN-3 parser recognizes, that an imported module is an ASN.1 module (from import statement in TTCN-3 module)!
This is also completely backward compatible, because nothing has to be changed in the syntax for templates.
So simply, only the semantic for ASN.1 extensible type templates must be defined more precisely.

_
(0015219)
Jacob Wieland - Spirent (reporter)
09-10-2018 09:48

 In response to http://oldforge.etsi.org/mantis/view.php?id=7805#c15215 [^]

The templates t1, t2 are only valid, if you add the 'optional' attribute as "implicit omit". Otherwise, they will be invalid. There is no way to express non-root fields in TTCN-3 as you are suggesting. TTCN-3 does not distinguish between root fields and extension fields. This would have to be a completely new feature (with probably wide-ranging consequences for type compatibility etc.).

t3_invalid would be valid if optional "implicit omit" is added (This should be done in general for templates of extensible types, that's what we introduced the feature for).

t3 (both versions) again is only valid if 'implicit omit' is given. Otherwise, it's invalid as TTCN-3 forces you to give all fields explicitly or leaves them uninitialized (which has a different semantic than omit).

t4 is valid

t5 is invalid (see above)

Also, the TTCN-3 abstracts from encoding, so dealing with the encoding of extension fields is outside of the scope of the ASN.1 to TTCN-3 mapping. Codecs have to deal with it properly, of course (they need to know which fields are root, which are non-root and which are extension). This has no bearing on the mapping to TTCN-3 types and their corresponding templates/values.

Giving the proposed subtype constraints models the ASN.1 type semantics on TTCN-3 values precisely (it accepts all valid, fully specified values/templates and rejects all invalid ones).
(0015221)
Wolfgang Seka (reporter)
09-10-2018 10:20

 First of all - as said already - I can live with the approach to treat all fields of ASN.1 extensions as being optional in the first place even though from TTCN-3 user's point of view this may be confusing.
Nevertheless there is another aspect:
Assuming all fields on an extension being optional it may make a difference whether the whole extension is not present or it is present but all its fields are omitted.
According to my knowledge this cannot be distinguished by current means of TTCN.
=> in principle a notation as proposed by Jens would be the real and easy-to-understand solution but I don't know how this approach could be made backward compatible.
(somehow an optional sub-grouping notation for templates would be needed ...)
(0015226)
Martin Hauch (reporter)
09-10-2018 12:52

 RESPONSE to Wolfgang Seka (0015221):
As far as I analyze the ASN.1 syntax, it is not possible to specify an extended type value without defining at least one extension-field with a value. As ASN.1 values could not assign omit to a field (omit is handled by field is not assigned in value-notation) an extended message-value without at least one extension field value, cannot be defined. If not at least one exteded field is assigned, then the value is not extended (e.g.only uses the root-fields of the type).
The sub-grouping solution complicates the template-definitions, because rules for field-renaming must be defined. Also this type-mapping does not represent the ASN.1 semantic.
Handling ASN.1 extensions as being optional is ok, but the name of the type should be the original mapping of ASN.1 type identifier to TTCN-3 type identifier (without appending '_')!
(0015239)
Jacob Wieland - Spirent (reporter)
10-10-2018 10:08

 Response to: http://oldforge.etsi.org/mantis/view.php?id=7805#c15226 [^]

First of all, for the ASN.1 to TTCN-3 mapping, it is completely irrelevant how values are represented in ASN.1 language notation or in encodings. We are only concerned how they are represented in TTCN-3 and that it is possible with the information contained in these values to encode/decode them correctly according to the ASN.1 specification of their type (and of course that the TTCN-3 value universe of the type is isomorphic to the ASN.1 value universe of the corresponding ASN.1 type). Thus, we do not distinguish on the TTCN-3 level between extended and unextended values. All must conform to the same type specification.

The internal (hence, private!, i.e. invisible, inaccessible to the user) name with the underscore is just a proposed workaround around the syntactical TTCN-3 restriction that a subtype restriction cannot be added directly to a record type definition.

Ideally, of course, we would like to map it to a definition like this:

type record My_Extended_Type { ... }
(extRestriction1)
(extRestriction2)
...
(extRestrictionN)

But, since that isn't allowed, a chain of type definitions needs to be used ending with the non-internal type definition with the equivalent semantic of the ASN.1 type.
If there are several extensions, you need one subtyping definition for each additional extension to get the final public type (with the proper original name) to have a conjunction constraint of all the chained subtyping constraints.
Finally, subtypes are always type compatible with their supertypes and the root type is obviously the one where all the extension fields are optional.
All values of these types would have the potential for all the extension fields, it's just a matter of taste whether you define them with implicit or explicit omits and whether you want to use refactoring on old templates when the type system is updated with new extension or want to use future-proof code instead (as has been outlined).

The proposal with the new extension syntax given by Jens does not introduce new fields with the name 'extension'. The proposed extension keyword followed by a field declaration block would have exactly the same syntactic and semantic function as the [[ ]] brackets in ASN.1, i.e. it would introduce all the fields contained as optional fields of the containing type while implicitly adding the proper constraints on those fields to the type (i.e. mandatory fields present if one of the fields contained is present). Therefore, there's no nameclashing problem, as there is no new name introduced and also no new scope in the value. That means, there is no complication added and the type mapping is a 1-to-1 representation of the ASN.1 semantic.
(0015258)
Kristóf Szabados (reporter)
11-10-2018 15:21

 Gyorgy, please check the uploaded proposal.
(0015398)
Kristóf Szabados (reporter)
07-08-2019 12:25

 Please check it.
(0015603)
Gyorgy Rethy (reporter)
29-12-2019 11:34
edited on: 29-12-2019 11:36

 Resolution is OK with me.
(0015605)
Gyorgy Rethy (reporter)
29-12-2019 14:22

 Included in final draft V4.7.2

- Issue History
Date Modified Username Field Change
07-10-2018 11:52 Wolfgang Seka New Issue
07-10-2018 12:44 Wolfgang Seka Note Added: 0015195
08-10-2018 11:35 Jacob Wieland - Spirent Note Added: 0015196
08-10-2018 11:36 Jacob Wieland - Spirent Note Added: 0015197
08-10-2018 11:40 Jacob Wieland - Spirent Note Edited: 0015197 View Revisions
08-10-2018 11:44 Wolfgang Seka Note Added: 0015198
08-10-2018 12:24 Jacob Wieland - Spirent Note Added: 0015199
08-10-2018 12:25 Jacob Wieland - Spirent Note Edited: 0015199 View Revisions
08-10-2018 13:09 Wolfgang Seka Note Added: 0015200
08-10-2018 14:14 Jens Grabowski Assigned To => Kristóf Szabados
08-10-2018 14:14 Jens Grabowski Status new => assigned
08-10-2018 14:23 Martin Hauch Note Added: 0015206
08-10-2018 15:23 Jens Grabowski Note Added: 0015208
09-10-2018 07:57 Jacob Wieland - Spirent Note Added: 0015210
09-10-2018 08:14 Jacob Wieland - Spirent Note Added: 0015211
09-10-2018 08:38 Jacob Wieland - Spirent Note Added: 0015212
09-10-2018 08:47 Martin Hauch Note Added: 0015213
09-10-2018 09:04 Jacob Wieland - Spirent Note Added: 0015214
09-10-2018 09:21 Martin Hauch Note Added: 0015215
09-10-2018 09:48 Jacob Wieland - Spirent Note Added: 0015219
09-10-2018 10:20 Wolfgang Seka Note Added: 0015221
09-10-2018 12:52 Martin Hauch Note Added: 0015226
10-10-2018 10:08 Jacob Wieland - Spirent Note Added: 0015239
11-10-2018 15:20 Kristóf Szabados File Added: CR7805.docx
11-10-2018 15:20 Kristóf Szabados Assigned To Kristóf Szabados => Gyorgy Rethy
11-10-2018 15:21 Kristóf Szabados Note Added: 0015258
11-10-2018 16:43 Kristóf Szabados File Added: CR7805_v2.docx
07-08-2019 12:25 Kristóf Szabados Note Added: 0015398
07-08-2019 12:25 Kristóf Szabados Status assigned => resolved
07-08-2019 12:25 Kristóf Szabados Resolution open => fixed
29-12-2019 11:34 Gyorgy Rethy Note Added: 0015603
29-12-2019 11:36 Gyorgy Rethy Note Edited: 0015603 View Revisions
29-12-2019 14:19 Gyorgy Rethy Project TTCN-3 Change Requests => Part 07: Using ASN.1 with TTCN-3
29-12-2019 14:22 Gyorgy Rethy Note Added: 0015605
29-12-2019 14:22 Gyorgy Rethy Status resolved => closed
29-12-2019 14:22 Gyorgy Rethy Fixed in Version => v4.8.1 (published 2020-05)

MantisBT 1.2.14 [^]
Copyright © 2000 - 2024 MantisBT Team
Powered by Mantis Bugtracker