[Issue 232] New - Problem with TRACK_TOKENS - firstToken is wrong

[carstensis]

https://javacc.dev.java.net/issues/show_bug.cgi?id=232
                 Issue #|232
                 Summary|Problem with TRACK_TOKENS - firstToken is wrong
               Component|javacc
                 Version|5.0
                Platform|All
              OS/Version|All
                     URL|
                  Status|NEW
       Status whiteboard|
                Keywords|
              Resolution|
              Issue type|DEFECT
                Priority|P3
            Subcomponent|jjtree
             Assigned to|sreeni
             Reported by|carstensis






------- Additional comments from carstensis@... Sun Nov  8 08:59:58 +0000 2009 -------
I have an issue concerning TRACK_TOKENS in jjtree as it sets the wrong
"firstToken" on some nodes in my case.

I'am using the XPath grammar from
http://www.w3.org/2007/01/applets/xpathApplet.html
and added the TRACK_TOKENS option and an UnparseVisitor just as in the
"VTransformer" javacc example.

When I parse the XPath expression "1+2", I get this result as a dump of the
created SimpleNode:

|XPath2    (firstToken '1', lastToken EOF)
|   XPath    (firstToken '1', lastToken '2')
|      Expr    (firstToken '1', lastToken '2')
|         AdditiveExpr    (firstToken EOF, lastToken '2') +
|            IntegerLiteral    (firstToken '+', lastToken '1') 1
|            IntegerLiteral    (firstToken EOF, lastToken '2') 2

Two things are wrong here:
# firstToken on Literals is wrong (should be the same as lastToken)
# firstToken on "Expr" subnode is wrong (ie. "AdditiveExpr" should have '1' as
firstToken)

When parsing an "Expr" (deciding if it's a "AdditiveExpr" or
"MultiplicativeExpr", etc.), the parser creates the corresponding "AdditiveExpr"
node only when it has already reached the end of the expression. And at that point
jjtn001.jjtSetFirstToken(getToken(1));
is already the token *after the end* of the "AdditiveExpr".


When looking at the code, I see that 2 "AdditiveExpr" nodes are created, the
first has the right firstToken and the second (the one that survives) has the
wrong one.
----------------------------
  final public void AdditiveExpr() throws ParseException {
 /*@bgen(jjtree) #AdditiveExpr(> 1) */
// *1st*
  SimpleNode jjtn000 = new SimpleNode(this, JJTADDITIVEEXPR);
  boolean jjtc000 = true;
  jjtree.openNodeScope(jjtn000);
  jjtn000.jjtSetFirstToken(getToken(1));
    try {
      MultiplicativeExpr();
      label_6:
      while (true) {
        switch ((jj_ntk==-1)?jj_ntk():jj_ntk) {
        case Minus:
[..]
        }
        switch ((jj_ntk==-1)?jj_ntk():jj_ntk) {
        case Plus:
[..]
        }
        MultiplicativeExpr();
// *2nd*
                  SimpleNode jjtn001 = new SimpleNode(this, JJTADDITIVEEXPR);
                  boolean jjtc001 = true;
                  jjtree.openNodeScope(jjtn001);
               
// => *wrong token here*, should use the token "jjtn000.jjtgetFirstToken()"
instead of "getToken(1)"???
                  jjtn001.jjtSetFirstToken(getToken(1));
----------------------------

Here's how I changed the created XPath parser code as a workaround:
- for terminal symbols I replaced
jjtn00x.jjtSetFirstToken(getToken(1));
with
jjtn00x.jjtSetFirstToken(getToken(0));
(the last token was always correct, and for a terminal symbol lastToken ==
firstToken)

- replaced the remaining occurences of jjtn00x.jjtSetFirstToken(getToken(1))
with
jjtn00x.jjtSetFirstToken(jjtn000FirstToken);
and put
Token jjtn000FirstToken = getToken(1);
at the beginning of the method like this:
----
  final public void RelativePathExpr() throws ParseException {
    StepExpr();
    Token jjtn000FirstToken = getToken(1);
    label_11:
    while (true) {
      switch ((jj_ntk==-1)?jj_ntk():jj_ntk) {
----
According to my tests, this fixes the problem for the XPath parser, but of
course I'd be happy to abandon this workaround once the issue has been solved in
JJTree :)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@...
For additional commands, e-mail: issues-help@...