ETSI's Bug Tracker - Part 01: TTCN-3 Core Language
View Issue Details
0006623Part 01: TTCN-3 Core LanguageNew Featurepublic07-10-2013 11:5107-02-2014 06:38
Jacob Wieland - Spirent 
Ina Schieferdecker 
highmajorN/A
closedfixed 
 
v4.6.1 (published 2014-06)v4.6.1 (published 2014-06) 
5.4.1, 11, 15, 16
Testing Technologies - Jacob Wieland
0006623: Allow lazy evaluation.
Lazy expression/function application evaluation which delays the evaluation of an expression until its actual usage can have any number of applications.

In security testing there is a concept called fuzzing where an operator can either generate one of a list of values randomly (and a different one for every invocation) or mutate a given value randomly, thereby simulating security attacks that the test system must be proof against. Lazy evaluation would allow a simple denotation of fuzz-templates which are only evaluated at the place of their usage, i.e. upon sending, when matching or converting to a value.

Another application or lazy evaluation is optional code which may be active in one context but inactive in another (only determinable at runtime). Having to evaluate the non-active expressions puts unnecessary strain on the testing architecture and can lead to bad performance of the test system. To avoid this, at the moment, the code must be much more complicated than it would have to be using lazy evaluation and would need to replicate the decision-code (which of the different optional code-parts are active) in many places.
No tags attached.
doc CR6623.doc (768,512) 09-10-2013 09:43
http://oldforge.etsi.org/mantis/file_download.php?file_id=2902&type=bug
doc CR6623_v2.doc (651,264) 10-10-2013 13:34
http://oldforge.etsi.org/mantis/file_download.php?file_id=2920&type=bug
doc CR6623_v3.doc (685,568) 11-10-2013 12:30
http://oldforge.etsi.org/mantis/file_download.php?file_id=2927&type=bug
doc CR6623_v4.doc (676,864) 26-11-2013 17:42
http://oldforge.etsi.org/mantis/file_download.php?file_id=2943&type=bug
doc CR6623_v5.doc (707,072) 28-01-2014 16:03
http://oldforge.etsi.org/mantis/file_download.php?file_id=2976&type=bug
Issue History
07-10-2013 11:51Jacob Wieland - SpirentNew Issue
07-10-2013 11:51Jacob Wieland - SpirentClause Reference(s) => 5.4.1, 11, 15, 16
07-10-2013 11:51Jacob Wieland - SpirentSource (company - Author) => Testing Technologies - Jacob Wieland
07-10-2013 11:54Jacob Wieland - SpirentStatusnew => assigned
07-10-2013 11:54Jacob Wieland - SpirentAssigned To => Jacob Wieland - Spirent
07-10-2013 11:58Gyorgy RethyProjectTTCN-3 Change Requests => Part 01: TTCN-3 Core Language
07-10-2013 11:59Gyorgy RethyPrioritynormal => high
07-10-2013 11:59Gyorgy RethyTarget Version => v4.4.1 (published 2012-04)
07-10-2013 12:13Gyorgy RethyTarget Versionv4.4.1 (published 2012-04) => v4.6.1 (published 2014-06)
09-10-2013 09:43Jacob Wieland - SpirentFile Added: CR6623.doc
09-10-2013 09:51Jacob Wieland - SpirentNote Added: 0011719
09-10-2013 09:51Jacob Wieland - SpirentAssigned ToJacob Wieland - Spirent => Jens Grabowski
09-10-2013 09:51Jacob Wieland - SpirentStatusassigned => confirmed
09-10-2013 10:14Jens GrabowskiStatusconfirmed => assigned
09-10-2013 12:21Jens GrabowskiNote Added: 0011726
09-10-2013 12:21Jens GrabowskiAssigned ToJens Grabowski => Jacob Wieland - Spirent
10-10-2013 10:41Gyorgy RethyNote Added: 0011747
10-10-2013 13:34Jacob Wieland - SpirentFile Added: CR6623_v2.doc
10-10-2013 13:40Jacob Wieland - SpirentNote Added: 0011753
10-10-2013 13:40Jacob Wieland - SpirentAssigned ToJacob Wieland - Spirent => Gyorgy Rethy
10-10-2013 13:40Jacob Wieland - SpirentStatusassigned => confirmed
11-10-2013 12:30Gyorgy RethyFile Added: CR6623_v3.doc
11-10-2013 12:33Gyorgy RethyNote Added: 0011775
11-10-2013 12:36Gyorgy RethyAssigned ToGyorgy Rethy => Jacob Wieland - Spirent
11-10-2013 12:36Gyorgy RethyStatusconfirmed => assigned
11-10-2013 13:01Jacob Wieland - SpirentNote Added: 0011776
11-10-2013 13:22Jacob Wieland - SpirentNote Edited: 0011776
11-10-2013 13:23Jacob Wieland - SpirentNote Added: 0011777
11-10-2013 13:23Jacob Wieland - SpirentAssigned ToJacob Wieland - Spirent => Gyorgy Rethy
11-10-2013 13:23Jacob Wieland - SpirentStatusassigned => confirmed
11-10-2013 14:21Gyorgy RethyNote Added: 0011783
11-10-2013 14:21Gyorgy RethyAssigned ToGyorgy Rethy => Jacob Wieland - Spirent
11-10-2013 14:21Gyorgy RethyStatusconfirmed => assigned
11-10-2013 14:24Gyorgy RethyNote Edited: 0011783
11-10-2013 14:59Tomas UrbanNote Added: 0011785
11-10-2013 15:00Tomas UrbanNote Edited: 0011785
11-10-2013 15:26Jacob Wieland - SpirentNote Added: 0011787
11-10-2013 15:29Jacob Wieland - SpirentAssigned ToJacob Wieland - Spirent => Gyorgy Rethy
11-10-2013 15:29Jacob Wieland - SpirentStatusassigned => confirmed
11-10-2013 15:34Tomas UrbanNote Added: 0011788
11-10-2013 16:25Gyorgy RethyNote Added: 0011791
11-10-2013 16:25Gyorgy RethyStatusconfirmed => assigned
11-10-2013 16:25Gyorgy RethyAssigned ToGyorgy Rethy => Jacob Wieland - Spirent
11-10-2013 16:26Tomas UrbanNote Added: 0011792
11-10-2013 16:43Gyorgy RethyNote Added: 0011794
11-10-2013 16:54Jacob Wieland - SpirentNote Added: 0011795
11-10-2013 17:01Jacob Wieland - SpirentNote Added: 0011796
12-10-2013 10:21Gyorgy RethyNote Added: 0011797
12-10-2013 10:27Gyorgy RethyNote Edited: 0011797
12-10-2013 10:40Gyorgy RethyNote Edited: 0011797
12-10-2013 10:43Gyorgy RethyNote Edited: 0011797
12-10-2013 10:47Gyorgy RethyNote Edited: 0011797
12-10-2013 10:48Gyorgy RethyNote Edited: 0011797
12-10-2013 10:49Gyorgy RethyNote Edited: 0011797
12-10-2013 17:00Tomas UrbanNote Added: 0011798
14-10-2013 10:52Gyorgy RethyNote Added: 0011799
14-10-2013 10:54Gyorgy RethyNote Edited: 0011799
14-10-2013 12:26Tomas UrbanNote Added: 0011801
14-10-2013 13:23Jacob Wieland - SpirentNote Added: 0011803
14-10-2013 13:26Jacob Wieland - SpirentNote Added: 0011804
22-11-2013 13:28Gyorgy RethyNote Added: 0011815
22-11-2013 13:29Gyorgy RethyNote Edited: 0011815
26-11-2013 15:28Jacob Wieland - SpirentNote Added: 0011837
26-11-2013 17:42Jacob Wieland - SpirentFile Added: CR6623_v4.doc
26-11-2013 17:43Jacob Wieland - SpirentNote Added: 0011841
26-11-2013 17:43Jacob Wieland - SpirentAssigned ToJacob Wieland - Spirent => Gyorgy Rethy
26-11-2013 17:43Jacob Wieland - SpirentStatusassigned => confirmed
28-11-2013 11:05Gyorgy RethyNote Added: 0011854
28-11-2013 11:05Gyorgy RethyStatusconfirmed => assigned
28-11-2013 11:05Gyorgy RethyAssigned ToGyorgy Rethy => Jacob Wieland - Spirent
28-11-2013 11:50Jacob Wieland - SpirentNote Added: 0011858
28-11-2013 12:06Jacob Wieland - SpirentNote Edited: 0011858
28-11-2013 14:50Gyorgy RethyAssigned ToJacob Wieland - Spirent => Jens Grabowski
28-01-2014 16:02Gyorgy RethyNote Added: 0011917
28-01-2014 16:03Gyorgy RethyFile Added: CR6623_v5.doc
28-01-2014 16:04Gyorgy RethyTarget Versionv4.6.1 (published 2014-06) => v4.7.1 (published 2015-06)
29-01-2014 11:07Jacob Wieland - SpirentNote Added: 0011918
29-01-2014 11:08Jacob Wieland - SpirentAssigned ToJens Grabowski => Ina Schieferdecker
29-01-2014 11:08Jacob Wieland - SpirentStatusassigned => confirmed
04-02-2014 13:05Jacob Wieland - SpirentStatusconfirmed => resolved
04-02-2014 13:05Jacob Wieland - SpirentFixed in Version => v4.6.1 (published 2014-06)
04-02-2014 13:05Jacob Wieland - SpirentResolutionopen => fixed
07-02-2014 06:38Ina SchieferdeckerNote Added: 0011923
07-02-2014 06:38Ina SchieferdeckerStatusresolved => closed
07-02-2014 06:38Ina SchieferdeckerTarget Versionv4.7.1 (published 2015-06) => v4.6.1 (published 2014-06)

Notes
(0011719)
Jacob Wieland - Spirent   
09-10-2013 09:51   
please review
(0011726)
Jens Grabowski   
09-10-2013 12:21   
After personal discussion reassigned to Jacob.
(0011747)
Gyorgy Rethy   
10-10-2013 10:41   
Results of the personal discussion with Jacob yesterday: actually we need two modifiers. In some cases evaluation shall be done at each case when a definition is used (e.g. fuzzy templates for security testing). In other cases evaluation shall be delayed, but done only once, at the first use (e.g. lazy parameter evaluation for performance testing, hoping that no evaluation will be needed at all). This has to be controlled by the user, only he knows its use case.

Consequently we need two modifiers for these two different use cases.

We could think about if the modifiers themselves should be put to the core or to the packages, as it was thought at the beginning?
(0011753)
Jacob Wieland - Spirent   
10-10-2013 13:40   
please review:

fuzzy: lazy evaluation, always re-evaluate
lazy: lazy evaluation, evaluate at most once

- templates now can only be fuzzy
- variables and parameters can be lazy or fuzzy
- functions can no longer be fuzzy (not really necessary)

restrictions are chosen in a way that does not conflict with existing features and that does not allow access to local variables that do not exist anymore when evaluating a fuzzy/lazy variable or parameter.
(0011775)
Gyorgy Rethy   
11-10-2013 12:33   
See my amendments and comments in CR6623_v3.doc.

Btw. once performance and security testing require different behavior and modifiers, we could return to the original idea and add @lazy to the performance package and @fuzzy to a new security package. What do you think? Could you also ask Dirk about it?
(0011776)
Jacob Wieland - Spirent   
11-10-2013 13:01   
(edited on: 11-10-2013 13:22)
The restriction for fuzzy variables only being assigned completely has been introduced because otherwise, the implementation would need to know which parts of the expression need to be re-evaluated on the next usage and which not (otherwise, it can simple re-evaluate upon usage without any additional knowledge whatsoever). This overcomplicates things and so I would rather have this restriction.

Lazy template declaration need not be restricted (in section 15, restriction c)) because they have not been introduced. There's only fuzzy templates and fuzzy/lazy template variables.

Regarding 19.1. semantic description of assignment of lazy variables.

Consider the following:

var @lazy MyRecord x := f_initializeRecord();
x.i := 5;

How should you implement the assignment of field i without first evaluating the x and then changing the value? Actually, if the right-hand-side contains the unchanged symbol, the x also must be evaluated before assignment. Only when the assignment is to x and not a sub-part of x and the right-hand-side does not contain the unchanged symbol would evaluation of x before assignment be unnecessary.

Also, I don't understand your comment "This would cause loss of tool performance and complicates feature for the user without obvoius benefit. For lazy it would just have reverse effect, re-evaluationg all fields all the time, that would not been re-evaluated for an ordinary variable or parameter."

Where is tool performance lost?
What fields are re-evaluated? (None, in my opinion)

For fuzzy variables, if you have the restriction that no subparts of them can be assigned, the problem does not present itself. Otherwise, you would have the same problem, of course.

(0011777)
Jacob Wieland - Spirent   
11-10-2013 13:23   
please review my remarks
(0011783)
Gyorgy Rethy   
11-10-2013 14:21   
(edited on: 11-10-2013 14:24)
"The restriction for fuzzy variables only being assigned completely "

It is too restrictive, this is not they normal way, how variables are used in general. The tool need not keep track which parts has been assigned and which one has not. taking your example (modified):
var @fuzzy MyRecord x := f_initializeRecord();
x.i := 5; //here x need to be evaluated for tool simplicity reasons
x.j := 42;//here x need NOT be re-evaluated before the assignment, just assign 42 to the field j
y := x; //x is evaluated before the assignment
z := x.j //x is evaluated before the assignment as well
P.send (x) //x is evaluated again
==> it is enough to evaluate x if the whole x has been assigned a value in beforehand; but this doesn't mean that field assignments should be forbidden in overall
------------------------------------------------------
"Regarding 19.1. semantic description of assignment of lazy variables"

you wrote: "If only part of a lazy variable is assigned, both the variable and the right-hand-side are evaluated before assignment."

Consider this as this is a very typical situation:
var @lazy MyRecord v_MyRecord;
v_MyRecord.f1 := f_initialize_f1();
//function call is stored but it is useless to evaluate any of the fields
...//v_MyRecord not used here
v_MyRecord.f2 := f_initialize_f2(); //ditto
...//v_MyRecord not used here
v_MyRecord.f3 := f_initialize_f3(); //ditto
if (b){P.send (v_MyRecord)} //this is the first place, where v_MyRecord really needs to be evaluated; all 3 function calls. But acc. to your proposal, up to this point *all fields* of the lazy variable should have been re-evaluated 3 times! resulting a worth performance than without @lazy!
==> it is enough to evaluate v_MyRecord at v_MyRecord.f1 if the whole v_MyRecord has been assigned a value in beforehand, like in your example
x := f_initializeRecord(); x.i := 5;

(0011785)
Tomas Urban   
11-10-2013 14:59   
(edited on: 11-10-2013 15:00)
Have you considered a scope restriction too? In my opinion, expressions that contain variables declared on a lower scope or in a non-related scope shall not be referenced in expressions assigned to fuzzy and lazy variables, because there's a serious danger that at the point of evaluation, the memory allocated for such referenced variables has been already disposed. E.g.

function f1(inout @lazy p)
{
   // y is a local variable that will be disposed when the funcion returns
   var integer y := 5;
   p := y + 4;
}

function f2()
{
   var @lazy integer x;
   f1(x);
   log(x); // the address of y is no longer available
}

Of course, it is possible to save the local memory block together with the entry point for lazy/fuzzy evaluation, but it might make disposing procedures impractically complex.

(0011787)
Jacob Wieland - Spirent   
11-10-2013 15:26   
@Tomas: lazy inout or out are not allowed (specifically because of this problem)

@Gyorgy:

> "The restriction for fuzzy variables only being assigned completely "

> It is too restrictive, this is not they normal way, how variables are used in > general. The tool need not keep track which parts has been assigned and which > one has not. taking your example (modified):
> var @fuzzy MyRecord x := f_initializeRecord();
> x.i := 5; //here x need to be evaluated for tool simplicity reasons

What would be the evaluation of x here?

> x.j := 42;//here x need NOT be re-evaluated before the assignment, just assign 42 to the field j

What would be the evaluation of x here?

> y := x; //x is evaluated before the assignment

> z := x.j //x is evaluated before the assignment as well
> P.send (x) //x is evaluated again
> ==> it is enough to evaluate x if the whole x has been assigned a value in beforehand; but this doesn't mean that field assignments should be forbidden in overall

My problem is, I have no idea what the semantics of x is supposed to be after all these assignments. What happens to the values that WOULD have to be assigned by f_initializeRecord() because of the overwriting of these fields? Do they override the assignments, if so, the assignments have no effect. Do I have to keep the additional assignments extra somewhere and delay them over the evaluation of the original assignment (like a modifies)? This is all far too complicated - no one would understand that semantics either way.

My view of fuzzy variables is simple: they store an unevaluated, unchangeable expression that will be evaluated every time, so that is in essence a bit like a macro variable.

------------------------------------------------------
> "Regarding 19.1. semantic description of assignment of lazy variables"

> you wrote: "If only part of a lazy variable is assigned, both the variable and the right-hand-side are evaluated before assignment."

> Consider this as this is a very typical situation:
> var @lazy MyRecord v_MyRecord;
> v_MyRecord.f1 := f_initialize_f1();
> //function call is stored but it is useless to evaluate any of the fields
> ...//v_MyRecord not used here
> v_MyRecord.f2 := f_initialize_f2(); //ditto
> ...//v_MyRecord not used here
> v_MyRecord.f3 := f_initialize_f3(); //ditto
> if (b){P.send (v_MyRecord)} //this is the first place, where v_MyRecord really needs to be evaluated; all 3 function calls. But acc. to your proposal, > up to this point *all fields* of the lazy variable should have been re-evaluated 3 times! resulting a worth performance than without @lazy!
> ==> it is enough to evaluate v_MyRecord at v_MyRecord.f1 if the whole v_MyRecord has been assigned a value in beforehand, like in your example
> x := f_initializeRecord(); x.i := 5;

This is a total misinterpretation of my intention. Since every evaluation only happens at most ONCE for every lazy variable, it would simply mean that the already assigned fields of the variable are evaluated (if not evaluated already) whenever you assign a new field (that way you do not need to keep track of which fields already have been evaluated and which have not). Of course, if the overall variable is not used, this is worse than if you assigned all the fields in one assignment (because there, nothing would ever be evaluated before usage).

So, assignment of f2 would evaluate rhs of f1 and f2. Assignment of f3 would evaluate rhs of f3. Usage of x would not evaluate anything anymore.This would not be worse than non-lazy, it would be exactly like non-lazy.

But, I can see that this dynamic-lazy-approach that you seem to have in mind can add some value.

It is clear, though, that, if a sub-part of a variable is already assigned something (because it or one of its parent has been assigned), then it needs to be evaluated before further assignment. Can we agree on this part?
(0011788)
Tomas Urban   
11-10-2013 15:34   
> lazy inout or out are not allowed (specifically because of this problem)
Excellent!

But what about this:
var @lazy integer x;
while (...)
{
   var integer y;
   ... // compute y
   x := y + 1;
}
log(x); // x refers to y, but block where y is defined is no
// longer valid at this point
(0011791)
Gyorgy Rethy   
11-10-2013 16:25   
@Tomas
I think it's reasonable to say that all operands of the expression shall be visible at the place, where the variable/parameter is actually evaluated. As the feature is meant for performance testing, I don't think this would cause usability issues (only component variables are used anyway). But this is a reason to add the feature to the real time & perf. package instead of the core.

@Jacob
@fuzzy
You misunderstood. In cases, when x is initialized as a whole but not evaluated yet (two runtime flags), x is evaluated BEFORE x.i.

Hence, for x.i := 5 the semantics is:
x := f_initializeRecord();//delayed evaluation
x.i := 5; //simple field assignment, no re-evaluation
//and then
x.j := 42; //simple field assignment, no re-evaluation

Otherwise you would waste only tool performance for nothing, or even for worth testing results. pls. note, fuzzy doesn't mean random! You may fuzz test by shooting messages with a monotonously increasing value in it (for example, a port scanning attack). If fuzzy values are evaluated all the time, even when it is not necessary, this would result holes in the increasing value.

Also note, that fuzz testing is a form of performance testing! The tool shall try as many possible attacks as it can for as short time period as it can. So, tool performance matter here.

@lazy
in this case the text has to be re-written that it cannot be misinterpreted.
(0011792)
Tomas Urban   
11-10-2013 16:26   
Could you also consider recursive definitions:

var @lazy integer i := 1;
i := i + 1;
log(i); // what will be the result?

Or another example:

var @lazy integer i;
var @lazy integer j;
i := j;
j := i; // definition by a circle
log(i); // infinite loop during resolving

In my opinion, recursive references should not appear in lazy and fuzzy variables.
(0011794)
Gyorgy Rethy   
11-10-2013 16:43   
Hi Tomas, pls. note, only the time of the evaluation is delayed, not its "content". t stick to your example:

var @lazy integer i := 1;
 i := i + 1;

is equal to:
var @lazy integer i := 1;
 i := (i := 1) + 1;

i := j; //it's not possible for normal variables either, because j is uninitialized.

But
var @lazy integer i;
var @lazy integer j := 1;
 i := j;
 j := i;

Would be equivalent to:
var @lazy integer i;
var @lazy integer j := 1;
 i := (j := 1); //i:= 1, 1 is stored in i here
 j := (i := 1)//j:=1
(0011795)
Jacob Wieland - Spirent   
11-10-2013 16:54   
@Gyorgy/fuzzy

I think what you are trying to accomplish can be easily accomplished by
assigning the fuzzy template to a non-lazy/fuzzy variable (that way fixing the current value) and then overwriting the fields that you want to change. That would have a clear semantics in my view. I'm still not clear what your example does. Is the function call evaluated before the field assignment or not? If so, what is the difference to lazy?

@Tomas:

yeah, the inner-local-variable-problem was on my radar as well, but I have not found a solution for that (neither how to formulate the restriction nor how to enforce it).

The recursitivity issue is not a big problem and cannot be avoided in my opinion. It is always possible to write down infinite recursions or endless loops. That cannot be avoided or thoroughly checked statically (semi-decidable problem) so I don't see how another way of doing so harms anyone.
(0011796)
Jacob Wieland - Spirent   
11-10-2013 17:01   
@Gyorgy/recursion

I think Tomas is right in his assessment. Since the time of evaluation is delayed until after the re-assignment of i (or the first assignment of j) is done, the evaluation of the right-hand-side i/j already points to the new initialization of i/j and not the original, overwritten one.

Otherwise, for every re-initialization of a variable, one would have to check the right-hand-side for usages of that variable and then at least evaluate that variable before assigning (as otherwise the original assignment will be lost).

Again, this seems to be very complicated (and pathological, no doubt), which is why I would opt for the easier-to-understand-and-implement version which may cause infinite recursion in such pathological cases (as is always the case for lazy evaluation, I think).
(0011797)
Gyorgy Rethy   
12-10-2013 10:21   
(edited on: 12-10-2013 10:49)
Example extended, corrected:
var @lazy integer i;
 var @lazy integer j := 1;//initialization is a must as the next assignment
                          //(i:=j) would cause runtime error without it
  i := j;
  j := i;
  j := 5;

Is equivalent to:
 var @lazy integer i;
 var @lazy integer j := 1;
  i := (j := 1); //previous j:= 1 is executed here when executing the expression;
                 // 1 is stored in j, nothing is stored in i
  j := (i := j ) //j is assigned to i when evaluating the expression;
                 //as j contains 1 at this point, 1 is assigned to i;
                 //the original assignment (j := i) is not executed at this point,
                 //but delayed to the next use of j on the RHS
  j := 5; //This overrides the previous delayed assignment with a new one;
                 //yes, the stored j := i assignment is replaced here with a new
                 //delayed assignment, but what difference does it make?

I can't see where is the recursion or problem here. Everything is done sequentially, only the time of the previous assignment is delayed.
-----------------------------------------------------
@difference between @lazy and @fuzzy (we have discussed this already)
@lazy evaluation is delayed only until the first use on RHS or FIRST assignment to its part, but NOT re-evaluated at further use on RHS
@fuzzy first evaluation is delayed until the first use on RHS or FIRST assignment to its part AND re-evaluated AT EACH use on RHS
-----------------------------------------------------
@fuzzy semantics
What I say has also a clear semantics, but does not force the user to make "dirty" workarounds, which again,
1) decreasing tool performance;
2) you tend to forget that TTCN-3 is not meant for programming experts, but shall be a language that can also be used correctly by testers without programming experience (in fact, they may be about 90% of the users)! If we do not consider this, we will make a wrong language design;
3) the user may not even be aware that he is using a @fuzzy template/variable; you also tend to forget that majority of the testers are using test frameworks that hides a lots of information from them (e.g. direct access of component variables that can be set/get via library functions only).

(0011798)
Tomas Urban   
12-10-2013 17:00   
György, I am afaid that we have a different understanding how the RHS of lazy/fuzzy variables work. In my opinion, the RHS of assignment to a lazy/fuzzy variable is NOT evaluated at all, i.e. not used. The TE doesn't perform any calculations during the assignment, it simply stores an entry point for the expression and continues with a next command. This is actually the whole point of lazy/fuzzy variables: not to do anything with the RHS in case of assignment. Thus, applied to our sample code:

var @lazy integer i;
var @lazy integer j := 1;
i := j; // value of j is not calculated at this point, because RHS of lazy assignments is not evaluated
j := i; // i is not evaluated as well
log(i); // evaluation occurs at this point, producing endless loop

Also notice that in the example, the RHS is trivial. However, since the RHS might contain function calls with cyclic reference to a fuzzy/lazy variable burried deep inside, there's no way to discover this connection without very complex analysis of the RHS or without executing the function. And I don't believe this is intended.

I also think that unlike standard assignments, assignments of lazy or fuzzy variables might contain uninitialized references on the RHS. These symbols have to be initialized in the moment when the fuzzy/lazy variable is used. Thus, the following code should work without a problem and print 5:
var integer @lazy i;
var integer @lazy j;
i := j;
j := 5;
log(i);

In order to resolve the problem with unresolvable cyclic references, I would propose one of the following rules:
1. If a cyclic reference to a variable (or parameter) being resolved is discovered in the evaluated expression, the TE shall produce a runtime error.

2. The expression on the RHS of lazy/fuzzy variable (or parameter) assignment shall not contain any direct or indirect reference to a lazy/fuzzy variable (or parameter).

Both rules would solve the issue with unresolvable cyclic references. The first proposal deals with the actual problem, but its disadvantage is that it offers no protection agains the problem in compilation time. However, there are other similar issues that compilers do not detect, such as endless loops. With a good debugger, it should not be a big deal to discover the source of the error when it occurs.

The second solution is capable of discovering issues in compilation time (with a good compiler). However, it might be too restrictive. I can imagine situations when chaining lazy/fuzzy variables would be useful.
(0011799)
Gyorgy Rethy   
14-10-2013 10:52   
(edited on: 14-10-2013 10:54)
Tomas, we are talking about two different features here. I agree, if we proposed the feature you are describing, it would cause both technical and usability (for the user) problems. You are distinguishing use in an expression or in an operation/statement, and evaluation is invoked only by the second one; we are talking about a much simpler feature that does not make this distinction. Otherwise is would also be impossible to follow/debug what's happening for the user.

The feature I'm talking about is the one in the CR resolution draft. It specifies: "That means that evaluation is delayed during execution until the first actual usage, other than using it at the left hand side of an assignment or passing it to a fuzzy or lazy formal parameter."

This means that in "var @lazy integer j := 1", j:=1 is not evaluated, but in "i := j", the previous j:=1 is evaluated, because this is the FIRST use "other than using it at the left hand side of an assignment or passing it to a fuzzy or lazy formal parameter". For this reason, your example with j not initialized at declaration will cause an error (at the line of i:=j).

This resolves possible cyclic issues and your proposed restrictions are not needed.

(0011801)
Tomas Urban   
14-10-2013 12:26   
I don't make a distinction where the symbol is used. The whole problem is in understanding of the word "usage" and "usage" in the following rule: "That means that evaluation is delayed during execution until the first actual usage, other than using it at the left hand side of an assignment or passing it to a fuzzy or lazy formal parameter."

We had this discussion last year and the conclusion was that the word "use" doesn't mean syntactical presence, but actual execution of a particular statement. My point is that assignment to a lazy/fuzzy variable doesn't constitute usage, because if the proposed rule is applied to i in i := j (i fullfils the condition of being a lazy variable on the LHS), it triggers delayed execution of the RHS containing j. It means that the expression is not evaluated at the moment of assignment, thus j is not "used" and the rule for evaluation of lazy variables cannot be applied to it.

Your point would be valid, if the words "using" and "usage" were replaced with "occurring" and "occurrence". However, even in this case, there might be a problem with component variables. Consider this:

type component C
{
  var @lazy i := 0;
  var @lazy j := 0;
}

function f1() return integer runs on C
{
  return j;
}

function f2() return integer runs on C
{
  return i;
}


testcase TC() runs on C
{
   i := f1(); // j is referenced indirectly
   j := f2(); // i is referenced indirectly
   log(i);
}

Discovering such indirect references requires complex algorithms (my example is just for demonstration, the dependency might not be that obvious in a real code) and leaving them undetected lead to the cyclic reference problem described in my previous posts.
(0011803)
Jacob Wieland - Spirent   
14-10-2013 13:23   
I have to fully agree with Tomas. My understanding of usage does not include usage on the right-hand-side of assignments to lazy variables (because that is the whole point of lazy variables that their right-hand-sides are only evaluated when the variable is used, not when they are assigned).

Therefore, I think that Tomas is right in his assessment. We would have to agree whether the runtime or the compile time solution is more acceptable.

In my opinion, runtime checking is enough, the second restriction could be used as the basis for warnings of cyclic-lazy-variable-dependence. Of course, if a compiler detects an obvious cyclic dependency of a lazy variable (that is used in a non-lazy context) to itself, it may already produce an error at compile-time.
(0011804)
Jacob Wieland - Spirent   
14-10-2013 13:26   
In regard to the access-to-lower-scope-variables, I think we need a restriction like:

The right-hand-side of an assignment to a lazy/fuzzy variable shall only contain references to variables/constants/templates that are known in the scope of the variable declaration.
(0011815)
Gyorgy Rethy   
22-11-2013 13:28   
(edited on: 22-11-2013 13:29)
Actually, we have two properties of lazy variables
a) they are evaluated when used on the RHS
b) expression on the RHS is not evaluated when there is a lazy variable on the LHS.

Therefore, when there are lazy vars on both sides of an assignment, the two rules collide. Which one of the rules should take precedence? -> is a matter of optimization decision.

Actually, the requirement is to avoid evaluation of vars/parameters that are NEVER used. For tool performance reasons. If a value is assigned to a variable and the variable *is* used, the time of its evaluation is irrelevant from performance point of view -> it requires the same computing resources. Assigning a value to a var and re-assigning it without using the first value is a rare case and a bad programming style that can be corrected in the code. You try to take into account this situation as well, by giving precedence to rule b), but this leads to extra complications and sometimes to a difficult-to-follow code behavior. This can be avoided if rule a) had the precedence above rule b). As a bonus, the code's behavior is much easier to understand, no implicit chains of var value assignments that actually never happen.

As a compromise solution, we could also say that if a lazy var/parameter is used at both sides of an assignment, the lazy var used in the expression on the RHS is evaluated, but the expression itself does not.


I don't fully understand Jacob's proposal regarding lower-scope-variables. For example, it is clear that a component var's value used in an evaluation, shall be its actual value at the time when the RHS is evaluated. This proposal would limit the usability of lazy component vars to zero. In performance testing component variables are used whenever possible, and local variables shall be avoided like a hot iron: the memory for component vars are allocated during tool initialization, when we have more time, while for local vars the memory is allocated during traffic generation, when we are lack of spare time. And memory operations are expensive.

Example; please note, the bahaviour is trivial to follow even in this simple example:
type component MyComp { var integer ci := 0; }

function MyFuncDieIfZero() runs on MyComp return integer {
  if (ci==0) { testcase.stop; } // die if the component variable is zero
  return ci;
}
function MyLazyFunc(in @lazy integer pi) runs on MyComp {
  ci := 1;
  log(pi); // first use of pi -> 3*MyFuncDieIfZero() is evaluated,
           // as ci==1 at this point, 3*1=3 is logged
  ci := 2;
  log(pi); // second use of pi -> not evaluated, the value 3 is used again
}
Calling the function:
MyLazyFunc(3*MyFuncDieIfZero());
  // the MyFuncDieIfZero() function is not evaluated here;
  // as ci==0 at this point, it would die if it was evaluated

(0011837)
Jacob Wieland - Spirent   
26-11-2013 15:28   
Let me see if I understand your concept of lazy variable correctly.

Suppose we have a declaration

var T x := E

where E contains the variables x1, ... xn

Then x is actually assigned the lambda closure

(\\x1,...,xn.E)(eval(x1),...,eval(xn))

so that later on eval(x) is the evaluation of E where the variables x1 to xn are substituted with their values when x was assigned (not when it is evaluated).

If that is the case, then it does not matter whether or not the variables on the right-hand-side are lazy or not, they are evaluated before assignment, i.e. a use of a lazy variable on the right-hand-side is an actual use that triggers evaluation.

This actually means that it does not matter when a lazy variable is evaluated after an assignment (before it is assigned again) - it will always yield the same result.

In that case, we do not need any restriction in regard of which variables can be used on the right hand side of lazy variable assignments.

For field/element assignments of lazy variables, it is still necessary to first evaluate the variable, i.e. field/element assignments are also a "use" of a lazy variable.

I think I like this concept. It has fewer restrictions and it does not deviate from normal variables a lot, i.e. most variables whose right-hand-sides have no side-effectscan simply be turned into a lazy variables.
(0011841)
Jacob Wieland - Spirent   
26-11-2013 17:43   
amended the proposal according to your comments, please review
(0011854)
Gyorgy Rethy   
28-11-2013 11:05   
Using the closure concept sounds interesting, though I didn't really meant this. Yes, you are right, in this case it is almost(or completely?) unimportant if a variable is lazy or not. In another words, this sounds like a tool optimization issue for me. Also, this may raise memory issues for the tools: the use case is the load testing scenarios, where - literally - millions of entities need to be simulated, and their data need to be stored; so tools have memory consumption issues already.

What I meant is to give control to the user, i.e. declaring a var/param lazy or non-lazy should make a difference. In my "half-way" view, when a lazy var/parameter is evaluated - at its first use-, the values of the dependent data at the time of the evaluation, i.e. not at the time of the assignment, is used. Like in my example below with the component variable. Just the evaluation is done at the first use in all cases, independently of what is at the LHS (i.e. is not re-delayed and re-delayed when the LHS is also lazy).
(0011858)
Jacob Wieland - Spirent   
28-11-2013 11:50   
(edited on: 28-11-2013 12:06)
sorry, I either don't udnerstand your concept or it suffers from the problem that occur if the variables used on the right-hand-side are either the variable being assigned or point to a lazy variable that again uses the variable being assigned (circularity) or not accessible/existent anymore in the scope of usage of the lazy variable (because the scope of these variables has already been left).

also, it has kind of weird, unforseeable consequences in case that the variables being used by the lazy variable are changed after the assignment, so the value of the variable when being evaluated depends on the time of evaluation.

(0011917)
Gyorgy Rethy   
28-01-2014 16:02   
please find the updated version in CR6623_v5.doc. I didn't change the sentence related to function return value evaluation in ~_v4 version, so hopefully it will be OK.

Please review.
(0011918)
Jacob Wieland - Spirent   
29-01-2014 11:07   
Section 5.4.2. EXAMPLE 6, the modulepar should be renamed to logComplexMessage (or even logMessage and the comment be amended accordingly. At the moment, the semantics is counter-intuitive to the natural-language semantics.

Section 7.1. Restriction a) has a typo: experssion

Otherwise, I consider this resolved.
(0011923)
Ina Schieferdecker   
07-02-2014 06:38   
Implemented with editorial corrections as proposed in v5 and last comment.