(jjtree) Problem with TRACK_TOKENS - firstToken is wrong

[Carsten Lohmann]

Hi,

I have issues concerning TRACK_TOKENS in jjtree as it sets the wrong
"firstToken" on some nodes in my case.

I'am using the XPath grammar from
http://www.w3.org/2007/01/applets/xpathApplet.html
and added the TRACK_TOKENS option and an UnparseVisitor just as in the
"VTransformer" javacc example.

When I parse the XPath expression "1+2", I get this result as a dump of
the created SimpleNode:

|XPath2 (firstToken '1', lastToken EOF)
|   XPath (firstToken '1', lastToken '2')
|      Expr (firstToken '1', lastToken '2')
|         AdditiveExpr (firstToken EOF, lastToken '2') +
|            IntegerLiteral (firstToken '+', lastToken '1') 1
|            IntegerLiteral (firstToken EOF, lastToken '2') 2

Two things are wrong here:
# firstToken on Literals is wrong
# firstToken on Expr SubNodes is wrong

When parsing an "Expr" (deciding if it's a "AdditiveExpr" or
"MultiplicativeExpr", etc.), the parser creates the corresponding
"AdditiveExpr" node only when it has already reached the end of the
expression. And at that point
jjtn001.jjtSetFirstToken(getToken(1));
is already the token *after the end* of the "AdditiveExpr".


When looking at the code, I see that 2 "AdditiveExpr" nodes are created,
the first has the right firstToken and the second (the one that
survives) has the wrong one.
----------------------------
   final public void AdditiveExpr() throws ParseException {
  /*@bgen(jjtree) #AdditiveExpr(> 1) */
// *1st*
   SimpleNode jjtn000 = new SimpleNode(this, JJTADDITIVEEXPR);
   boolean jjtc000 = true;
   jjtree.openNodeScope(jjtn000);
   jjtn000.jjtSetFirstToken(getToken(1));
     try {
       MultiplicativeExpr();
       label_6:
       while (true) {
         switch ((jj_ntk==-1)?jj_ntk():jj_ntk) {
         case Minus:
[..]
         }
         switch ((jj_ntk==-1)?jj_ntk():jj_ntk) {
         case Plus:
[..]
         }
         MultiplicativeExpr();
// *2nd*
                   SimpleNode jjtn001 = new SimpleNode(this,
JJTADDITIVEEXPR);
                   boolean jjtc001 = true;
                   jjtree.openNodeScope(jjtn001);
                               
// => *wrong token here*, use jjtn000.jjtgetFirstToken() instead???
                   jjtn001.jjtSetFirstToken(getToken(1));
----------------------------

(I changed the Parser code as in the code comment above, using
jjtn000.jjtgetFirstToken(). This fixes the problem on most expressions,
but not all ("ComparisonExpr" nodes still have the wrong first token))

Is this all a problem with the grammar or a bug in JJTree?

TIA
Carsten

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Paul Cager-5]

> Hi,
>
> I have issues concerning TRACK_TOKENS in jjtree as it sets the wrong
> "firstToken" on some nodes in my case.

.....
>
> Is this all a problem with the grammar or a bug in JJTree?
>
> TIA
> Carsten


Hi Carsten,

That sounds like a bug in jjtree (I've not checked the details, but using
getToken(1) doesn't sound right).

Would you mind creating an issue
(https://javacc.dev.java.net/servlets/ProjectIssues) so that we remember
to fix it in the next release?

Cheers.
Paul


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Carsten Lohmann]

Hi Paul,

I created issue 232 for it
https://javacc.dev.java.net/issues/show_bug.cgi?id=232

Cheers,
Carsten

Paul Cager wrote:

>> Hi,
>>
>> I have issues concerning TRACK_TOKENS in jjtree as it sets the wrong
>> "firstToken" on some nodes in my case.
>
> .....
>> Is this all a problem with the grammar or a bug in JJTree?
>>
>> TIA
>> Carsten
>
>
> Hi Carsten,
>
> That sounds like a bug in jjtree (I've not checked the details, but using
> getToken(1) doesn't sound right).
>
> Would you mind creating an issue
> (https://javacc.dev.java.net/servlets/ProjectIssues) so that we remember
> to fix it in the next release?
>
> Cheers.
> Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...