Nabble - java.net - javacc users

Re: JavaCC questions :

2009-11-18T05:56:38Z

Andrew A wrote:

Hey everyone, I am new to javaCC and have looked into it and have some questions if anyone would be so kind to help answer. I couldn't locate the information in question myself.

1. Can the parser that javaCC creates be incorporated into a project as a normal class. That way we can invoke the class without the main method when necessary. I am abit confused as to why javaCC requires a main, unless to demonstrate it working by itself.

Yes it can. I use it without main method like a normal class routinely:

See methods like the following in generated parser main class MyParser.java (assuming parser is called MyParser):

/** Constructor with InputStream. */
public MyParser(java.io.InputStream stream) {
this(stream, null);
}
/** Constructor with InputStream and supplied encoding */
public MyParser(java.io.InputStream stream, String encoding) {
...
}

Create parser using above methods and then call the method corresponding to your start production to do the parsing of the stream.

My only issue is that needed a non-arg constructor and a set method to set the stream. So I had to use some ugly kludges to do that.
If any one know a decent way to address my need please share. Otherwise I hope the dev team would consider this as as RFE.

Thanks.

2. I would like to not only parse text, but either build an object or add additional notes to the parsed text. JavaCC appears to be able to do this, is there any good examples or tutorials that someone could suggest related to this area.

The jjtree program generates an Abstract SYntax Tree (AST) that can be decorated with arbitrary info. I have not used it in a while but it was not to hard to get that working based on available docs as I recall. Perhaps someone more experienced can guide better on this.

Best of luck.

-- 
Regards,
Farrukh

Web: http://www.wellfleetsoftware.com

--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...

JavaCC questions :

2009-11-18T01:55:02Z

Hey everyone, I am new to javaCC and have looked into it and have some questions if anyone would be so kind to help answer. I couldn't locate the information in question myself.

1. Can the parser that javaCC creates be incorporated into a project as a normal class. That way we can invoke the class without the main method when necessary. I am abit confused as to why javaCC requires a main, unless to demonstrate it working by itself.

2. I would like to not only parse text, but either build an object or add additional notes to the parsed text. JavaCC appears to be able to do this, is there any good examples or tutorials that someone could suggest related to this area.

Thank You

New Windows 7: Find the right PC for you. Learn more.

Re: We are trying to create an off-side rule parser, but have problems with multiple UNINDENT.

2009-11-15T09:13:15Z

I don't remember now why I did it, but I had to abandon this. So instead,
we did it using semantic lookahead. Basically, just ignore all white space
as you normally do, setTabSize(1) and then do something like the
following.

Hope that helps,
Sreeni.

try
{
LOOKAHEAD({getToken(0).beginLine == getToken(1).beginLine})
stmt_list() /*<NEWLINE>*/ // multiple statements on the same line
|

(
{ indent = getToken(1).beginColumn; }
(
LOOKAHEAD({ getToken(1).kind != EOF &&
indent == getToken(1).beginColumn })
statement()
) +
)
}
finally
{
indent = oldIndent;
}

> We are trying to create an off-side rule parser that uses indentation
> (whitespace) rather than braces to delimit blocks in the abstract syntax.
> Ultimately, we'll simplify the concrete syntax to be more like
> python/haskell, but right now we are using a concrete syntax similar to
> that
> of C, with the exception of INDENT and UNINDENT tokens to delimit blocks
> rather than brace tokens.
>
> In the token manager declarations, we setup a stack to keep track of
> indentation. The stack stores the length of the whitespace comprising
> indentation as ints, and corresponding peek, pop and push methods are
> defined.
>
> We define a DEFAULT lexical state in which the lexar begins, and we also
> switch back to the DEFAULT state from the IN_LINE state whenever we
> encounter an end of line in the latter state. In the DEFAULT state, we
> skip
> empty lines, and we use MORE to capture all the whitespace comprising the
> indentation.
>
> After matching all the spaces or tabs in the current indentation, we have
> an
> action associated with an empty string match that determines if the
> current
> line is indented, unindented or the same indentation as the previous line.
> We compare the length of the whitespace of the current line to the value
> at
> the top of the stack.
>
> If the length of the whitespace of the current line is greater than the
> value on the top of the stack, then an indent occurred, and we push the
> length on the stack, and we switch to the INDENTING lexar state. In this
> state, an empty string match causes the lexar to send an INDENT token to
> the
> parser, and this works in our tests.
>
> If the length of the whitespace of the current line is less than the value
> on the top of the stack, then an unindent occurred, and we pop the stack
> until the value on the top of the stack is the same as the length of the
> current whitespace. Each time we pop, our goal is to send an UNINDENT
> token
> to the parser, but we are presently unable to send more than one UNINDENT
> token. Strangely, an UNINDENT token is sent only after the last pop,
> whereas it would seem that this would happen after the first pop.
>
> The second problem is that the parser becomes stuck in an infinite loop of
> empty string matches in this state. This is also strange because the
> UNINDENT token is only matched after the last pop, but also because after
> the last pop, the length of the whitespace of the current line is equal to
> the value on the top of the stack, so the lexar should break out of the
> while loop in the action, and the last else clause in the action should
> switch back to the IN_LINE lexar state.
>
> -------------------------------------------------------------------------------
>
> Excerpt from Parser.jj:
>
> TOKEN_MGR_DECLS:
> {
> Integer[] init_array = new Integer[] {new Integer(0)};
> java.util.ArrayList<Integer> stack =
> new
> java.util.ArrayList<Integer>(java.util.Arrays.asList(init_array));
> int peek() { return stack.get(stack.size() - 1); }
> int pop() { return stack.remove(stack.size() - 1); }
> void push(int i) { stack.add(i);}
> }
>
> <DEFAULT>
> SKIP:
> {
> "\r" // { System.out.print("(DEFAULT STATE - SKIP: CR)"); }
> | "\n" // { System.out.print("(DEFAULT STATE - SKIP: NL)"); }
> }
>
> <DEFAULT>
> MORE:
> {
> " " // { System.out.print("(DEFAULT STATE - MORE: S)"); }
> | "\t" // { System.out.print("(DEFAULT STATE - MORE: T)"); }
> | ""
> {
> if (image.length() > peek())
> {
> push(image.length());
> System.out.print("(push " + stack + ")");
> SwitchTo(INDENTING);
> }
> else if (image.length() < peek())
> {
> while (image.length() < peek())
> {
> pop();
> System.out.print("(pop " + stack +")");
> SwitchTo(UNINDENTING);
> }
> if (image.length() != peek())
> {
> //throw new TokenMgrError("Improper unindent.",
> //TokenMgrError.LEXICAL_ERROR);
> System.out.print("Error: Improper unindent.");
> }
> }
> else
> {
> SwitchTo(IN_LINE);
> }
> }
> }
>
> <INDENTING>
> TOKEN:
> {
> <INDENT: ""> { System.out.print("<INDENT>"); } : IN_LINE
> }
>
> <UNINDENTING>
> TOKEN:
> {
> <UNINDENT: ""> { System.out.print("<UNINDENT>"); } : DEFAULT
> }
>
> <IN_LINE>
> SKIP:
> {
> | " "
> | "\t"
> | "\r"
> }
>
> <IN_LINE>
> TOKEN:
> {
>
> /* ALL THE REGULAR TOKENS */
>
> | <EOL: "\n"> { System.out.print("<EOL>\n"); } : DEFAULT
> }
>
> -------------------------------------------------------------------------------
>
> Sample input:
>
> int main ()
> if (true)
> return 1;
> int i;
>
> -------------------------------------------------------------------------------
>
> Sample output:
>
> bash-3.2$ ant && java -cp bin proj.Hmm test1.py
> Buildfile: build.xml
>
> printJavaVersion:
> [echo] YOU ARE USING JAVA 1.6.0_11
> [echo]
> [echo] Java located at: /lusr/opt/jdk1.6.0_11/jre
>
> all:
> [javacc] Java Compiler Compiler Version 4.1 (Parser Generator)
> [javacc] (type "javacc" with no arguments for help)
> [javacc] Reading from file
> /v/filer4b/v38q001/cjh/cs345/project/proj/src/proj/parser/Parser.jj . . .
> [javacc] Warning: Line 43, Column 5: Regular expression can be matched
> by
> the empty string ("") in lexical state DEFAULT. This can result in an
> endless loop of empty string matches.
> [javacc] Warning: Line 86, Column 5: Regular expression for UNINDENT
> can
> be matched by the empty string ("") in lexical state UNINDENTING. This
> regular expression along with the regular expressions at line 43, column 5
> forms the cycle
> [javacc] UNINDENTING-->DEFAULT-->DEFAULT
> [javacc] containing regular expressions with empty matches. This can
> result in an endless loop of empty string matches.
> [javacc] File "TokenMgrError.java" is being rebuilt.
> [javacc] File "ParseException.java" is being rebuilt.
> [javacc] File "Token.java" is being rebuilt.
> [javacc] File "SimpleCharStream.java" is being rebuilt.
> [javacc] Parser generated with 0 errors and 2 warnings.
> [javac] Compiling 7 source files to
> /v/filer4b/v38q001/cjh/cs345/project/proj/bin
> [javac] Note:
> /v/filer4b/v38q001/cjh/cs345/project/proj/src/proj/parser/Parser.java uses
> unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
>
> BUILD SUCCESSFUL
> Total time: 3 seconds
> (program - start)<INT><IDENTIFIER><LPAREN><RPAREN><EOL>
> (block - start)(push [0, 4])<INDENT><IF><LPAREN><RPAREN><EOL>
> (block - start)(push [0, 4, 8])<INDENT><RETURN><INTEGER><SEMI>(return -
> end)<EOL>
> (pop [0, 4])(pop [0])<UNINDENT>(block - end)Exception in thread "main"
> proj.parser.TokenMgrError: Error: Bailing out of infinite loop caused by
> repeated empty string matches at line 4, column 1.
> at proj.parser.ParserTokenManager.MoreLexicalActions(Unknown
> Source)
> at proj.parser.ParserTokenManager.getNextToken(Unknown Source)
> at proj.parser.Parser.jj_ntk(Unknown Source)
> at proj.parser.Parser.ifStatement(Unknown Source)
> at proj.parser.Parser.statement(Unknown Source)
> at proj.parser.Parser.statements(Unknown Source)
> at proj.parser.Parser.block(Unknown Source)
> at proj.parser.Parser.restOfFunction(Unknown Source)
> at proj.parser.Parser.Program(Unknown Source)
> at proj.Hmm.main(Unknown Source)
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

We are trying to create an off-side rule parser, but have problems with multiple UNINDENT.

2009-11-14T15:30:22Z

We are trying to create an off-side rule parser that uses indentation (whitespace) rather than braces to delimit blocks in the abstract syntax. Ultimately, we'll simplify the concrete syntax to be more like python/haskell, but right now we are using a concrete syntax similar to that of C, with the exception of INDENT and UNINDENT tokens to delimit blocks rather than brace tokens.

In the token manager declarations, we setup a stack to keep track of indentation. The stack stores the length of the whitespace comprising indentation as ints, and corresponding peek, pop and push methods are defined.

We define a DEFAULT lexical state in which the lexar begins, and we also switch back to the DEFAULT state from the IN_LINE state whenever we encounter an end of line in the latter state. In the DEFAULT state, we skip empty lines, and we use MORE to capture all the whitespace comprising the indentation.

After matching all the spaces or tabs in the current indentation, we have an action associated with an empty string match that determines if the current line is indented, unindented or the same indentation as the previous line. We compare the length of the whitespace of the current line to the value at the top of the stack.

If the length of the whitespace of the current line is greater than the value on the top of the stack, then an indent occurred, and we push the length on the stack, and we switch to the INDENTING lexar state. In this state, an empty string match causes the lexar to send an INDENT token to the parser, and this works in our tests.

If the length of the whitespace of the current line is less than the value on the top of the stack, then an unindent occurred, and we pop the stack until the value on the top of the stack is the same as the length of the current whitespace. Each time we pop, our goal is to send an UNINDENT token to the parser, but we are presently unable to send more than one UNINDENT token. Strangely, an UNINDENT token is sent only after the last pop, whereas it would seem that this would happen after the first pop.

The second problem is that the parser becomes stuck in an infinite loop of empty string matches in this state. This is also strange because the UNINDENT token is only matched after the last pop, but also because after the last pop, the length of the whitespace of the current line is equal to the value on the top of the stack, so the lexar should break out of the while loop in the action, and the last else clause in the action should switch back to the IN_LINE lexar state.

-------------------------------------------------------------------------------

Excerpt from Parser.jj:

TOKEN_MGR_DECLS:
{
    Integer[] init_array = new Integer[] {new Integer(0)};
    java.util.ArrayList<Integer> stack =
        new java.util.ArrayList<Integer>(java.util.Arrays.asList(init_array));
    int peek() { return stack.get(stack.size() - 1); }
    int pop() { return stack.remove(stack.size() - 1); }
    void push(int i) { stack.add(i);}
}

<DEFAULT>
SKIP:
{
    "\r" // { System.out.print("(DEFAULT STATE - SKIP: CR)"); }
| "\n" // { System.out.print("(DEFAULT STATE - SKIP: NL)"); }
}

<DEFAULT>
MORE:
{
    " " // { System.out.print("(DEFAULT STATE - MORE: S)"); }
| "\t" // { System.out.print("(DEFAULT STATE - MORE: T)"); }
| ""
    {
        if (image.length() > peek())
        {
            push(image.length());
            System.out.print("(push " + stack + ")");
            SwitchTo(INDENTING);
        }
        else if (image.length() < peek())
        {
            while (image.length() < peek())
            {
                pop();
                System.out.print("(pop " + stack +")");
                SwitchTo(UNINDENTING);
            }
            if (image.length() != peek())
            {
                //throw new TokenMgrError("Improper unindent.",
                //TokenMgrError.LEXICAL_ERROR);
                System.out.print("Error: Improper unindent.");
            }
        }
        else
        {
            SwitchTo(IN_LINE);
        }
    }
}

<INDENTING>
TOKEN:
{
    <INDENT: ""> { System.out.print("<INDENT>"); } : IN_LINE
}

<UNINDENTING>
TOKEN:
{
    <UNINDENT: ""> { System.out.print("<UNINDENT>"); } : DEFAULT
}

<IN_LINE>
SKIP:
{
| " "
| "\t"
| "\r"
}

<IN_LINE>
TOKEN:
{

/* ALL THE REGULAR TOKENS */

| <EOL:       "\n"> { System.out.print("<EOL>\n"); } : DEFAULT
}

-------------------------------------------------------------------------------

Sample input:

int main ()
    if (true)
        return 1;
int i;

-------------------------------------------------------------------------------

Sample output:

bash-3.2$ ant && java -cp bin proj.Hmm test1.py
Buildfile: build.xml

printJavaVersion:
     [echo]        YOU ARE USING JAVA 1.6.0_11
     [echo]
     [echo]   Java located at: /lusr/opt/jdk1.6.0_11/jre

all:
   [javacc] Java Compiler Compiler Version 4.1 (Parser Generator)
   [javacc] (type "javacc" with no arguments for help)
   [javacc] Reading from file /v/filer4b/v38q001/cjh/cs345/project/proj/src/proj/parser/Parser.jj . . .
   [javacc] Warning: Line 43, Column 5: Regular expression can be matched by the empty string ("") in lexical state DEFAULT. This can result in an endless loop of empty string matches.
   [javacc] Warning: Line 86, Column 5: Regular expression for UNINDENT can be matched by the empty string ("") in lexical state UNINDENTING. This regular expression along with the regular expressions at line 43, column 5 forms the cycle
   [javacc]    UNINDENTING-->DEFAULT-->DEFAULT
   [javacc] containing regular expressions with empty matches. This can result in an endless loop of empty string matches.
   [javacc] File "TokenMgrError.java" is being rebuilt.
   [javacc] File "ParseException.java" is being rebuilt.
   [javacc] File "Token.java" is being rebuilt.
   [javacc] File "SimpleCharStream.java" is being rebuilt.
   [javacc] Parser generated with 0 errors and 2 warnings.
    [javac] Compiling 7 source files to /v/filer4b/v38q001/cjh/cs345/project/proj/bin
    [javac] Note: /v/filer4b/v38q001/cjh/cs345/project/proj/src/proj/parser/Parser.java uses unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL
Total time: 3 seconds
(program - start)<INT><IDENTIFIER><LPAREN><RPAREN><EOL>
(block - start)(push [0, 4])<INDENT><IF><LPAREN><RPAREN><EOL>
(block - start)(push [0, 4, 8])<INDENT><RETURN><INTEGER><SEMI>(return - end)<EOL>
(pop [0, 4])(pop [0])<UNINDENT>(block - end)Exception in thread "main" proj.parser.TokenMgrError: Error: Bailing out of infinite loop caused by repeated empty string matches at line 4, column 1.
        at proj.parser.ParserTokenManager.MoreLexicalActions(Unknown Source)
        at proj.parser.ParserTokenManager.getNextToken(Unknown Source)
        at proj.parser.Parser.jj_ntk(Unknown Source)
        at proj.parser.Parser.ifStatement(Unknown Source)
        at proj.parser.Parser.statement(Unknown Source)
        at proj.parser.Parser.statements(Unknown Source)
        at proj.parser.Parser.block(Unknown Source)
        at proj.parser.Parser.restOfFunction(Unknown Source)
        at proj.parser.Parser.Program(Unknown Source)
        at proj.Hmm.main(Unknown Source)

Re: Use Longs for Line Numbers

2009-11-10T18:57:07Z

On Nov 9, 2009, at 3:46 PM, sreeni@... wrote:

>
> Yes, we should fix it. But for now, you can simply edit the generated
> files - Token.java and SimpleCharStream.java to make the line numbers
> long. There might be some issues with error reporting, but it should work.
>
> Tom, I'm not sure I understand what you are saying. The line number stuff
> is rather straightforward. It's used purely for reporting. It doesn't
> affect the generated code at all.

Hm, maybe I misunderstood... I was thinking that Kurt was saying that there were more than Integer.MAX_INT tokens... so that all those int[] arrays of tokens and such would need to be long[] instead. But maybe I'm misrepresenting his question...

Yours,

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Use Longs for Line Numbers

2009-11-09T12:46:22Z

Yes, we should fix it. But for now, you can simply edit the generated
files - Token.java and SimpleCharStream.java to make the line numbers
long. There might be some issues with error reporting, but it should work.

Tom, I'm not sure I understand what you are saying. The line number stuff
is rather straightforward. It's used purely for reporting. It doesn't
affect the generated code at all.

>
> On Oct 6, 2009, at 6:57 PM, Kurt Guenther wrote:
>
>>
>> I'm currently parsing an 8 GB file with 241,000,000 lines which is
>> about 12.5% of the 2^31 -1 max value of int. I'm supposed to
>> support files up to 200 GB, so is there a way to generate the token
>> manager with longs instead of ints?
>
> I keep thinking about this one. We can't do just a search and replace
> on the generated code since other stuff uses ints as well - e.g., the
> lexical states are contained in an array of integers. I think it'd
> also affect the parser generating code, since we'd need to make jj_gen
> and jj_ntk and others into longs as well.
>
> At the end of the day, it might be easier to preprocess your input
> data to chunk it into smaller files.
>
> Yours,
>
> Tom
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Dynamic tokenizing

2009-11-09T07:56:47Z

Can't you just use a symbol table? Macro processors often do this kind
of thing. Another similar example is "typedef" in C.

I haven't used COMMON_TOKEN_ACTION since I use USER_TOKEN_MANAGER, but I
would think that you can update a symbol table as the "UserKeywords"
statement is parsed. Then in the common action code you can check
generic word tokens in the symbol table, and if found, change the "kind"
field in the token to a different token type (e.g. USER_KW). The actual
keyword text is in the "image" field. In the grammar you can specify
the <USER_KW> "{" ... "}" sequence anywhere it may occur and process it
as necessary as it is parsed.

Depending on what you are doing with the content within the braces, you
may have to do some additional processing in the common action code. If
you were trying to do text substitution or macro processing (i.e.
replacing the USER_KW {...} sequence with a different string of tokens
to be parsed), that would be a whole different ballgame.

Bill

Bachelier, Georges wrote:

> Hi!
>
>
>
> I have been searching a solution for three days to the following problem:
>
>
>
> I am building a JavaCC parser for a language having an original feature: the language allows you to define user key words which may be found anywhere in the input file after their declaration.
>
>
>
> User key words are declared with the following statement : UserKeywords <my_user_kw1> ... <my_user_kwn> ;
>
>
>
> Once keywords have been declared, they may be used like this :
>
>
>
> <my_user_kw1> {
>
> <any_sequence_of_characters>
>
> }
>
>
>
> <my_user_kwn> {
>
> <any_sequence_of_characters>
>
> }
>
>
>
> The sequences of characters do not matter; we do not use them, but they are surrounded by braces.
>
>
>
> I have written a parser production in order to get the user key words and I have tried to play with the tokenizer CommonTokenAction function, but without success.
>
>
>
> Could someone help me on this issue, please?
>
>
>
> Georges
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Use Longs for Line Numbers

2009-11-09T06:19:53Z

On Oct 6, 2009, at 6:57 PM, Kurt Guenther wrote:

>
> I'm currently parsing an 8 GB file with 241,000,000 lines which is
> about 12.5% of the 2^31 -1 max value of int. I'm supposed to
> support files up to 200 GB, so is there a way to generate the token
> manager with longs instead of ints?

I keep thinking about this one. We can't do just a search and replace
on the generated code since other stuff uses ints as well - e.g., the
lexical states are contained in an array of integers. I think it'd
also affect the parser generating code, since we'd need to make jj_gen
and jj_ntk and others into longs as well.

At the end of the day, it might be easier to preprocess your input
data to chunk it into smaller files.

Yours,

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: JavaCC Version 5.0 released.

2009-11-09T06:09:45Z

That's an interesting idea.... you might want to file an issue here:

https://javacc.dev.java.net/servlets/ProjectIssues

Just to keep it on the radar.

Yours,

tom

On Sep 1, 2009, at 5:02 AM, Artur Rataj wrote:

> Hello.
>
> What about a way to ensure that, at certain points of the grammar,
> backing up a single token is safe and, that a next token exists?
>
> I use the former for error recovery, but I do not know if it won't
> cause weird lexer/parser behavior in some cases.
>
> I can ensure the latter by adding artificial choices to the grammar,
> but I still do not know if it works in all cases.
>
> As of me, such features would add a bit of flexibility that one has
> with hand--made parsers.
>
> But, perhaps there is virtually no demand for the features among
> JavaCC users.
>
> Best regards,
> Artur
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

RE: Dynamic tokenizing

2009-11-09T00:25:59Z

I do not believe you can do dynamic tokenizing directly in JavaCC.

I would think about writing a first parser that recognizes your UserKeywords syntax, and in the corresponding production a) generates a second parser (through println() statements, or a through a template engine) which will contain all the keywords token definitions, b) compile it, and c) execute it on the remaining of the input file.

Marc

De : Bachelier, Georges [mailto:georges.bachelier@...]
Envoyé : vendredi 6 novembre 2009 20:14
À : users@...
Objet : [JavaCC] Dynamic tokenizing

Hi!

I have been searching a solution for three days to the following problem:

I am building a JavaCC parser for a language having an original feature: the language allows you to define user key words which may be found anywhere in the input file after their declaration.

User key words are declared with the following statement : UserKeywords <my_user_kw1> <my_user_kwn> ;

Once keywords have been declared, they may be used like this :

<my_user_kw1> {

<any_sequence_of_characters>

}

<my_user_kwn> {

<any_sequence_of_characters>

}

The sequences of characters do not matter; we do not use them, but they are surrounded by braces.

I have written a parser production in order to get the user key words and I have tried to play with the tokenizer CommonTokenAction function, but without success.

Could someone help me on this issue, please?

Georges

Re: (jjtree) Problem with TRACK_TOKENS - firstToken is wrong

2009-11-08T01:08:06Z

Hi Paul,

I created issue 232 for it
https://javacc.dev.java.net/issues/show_bug.cgi?id=232

Cheers,
Carsten

Paul Cager wrote:

>> Hi,
>>
>> I have issues concerning TRACK_TOKENS in jjtree as it sets the wrong
>> "firstToken" on some nodes in my case.
>
> .....
>> Is this all a problem with the grammar or a bug in JJTree?
>>
>> TIA
>> Carsten
>
>
> Hi Carsten,
>
> That sounds like a bug in jjtree (I've not checked the details, but using
> getToken(1) doesn't sound right).
>
> Would you mind creating an issue
> (https://javacc.dev.java.net/servlets/ProjectIssues) so that we remember
> to fix it in the next release?
>
> Cheers.
> Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Dynamic tokenizing

2009-11-06T12:55:05Z

This sounds doable. You can probably use LOOKAHEAD with a function call
as parameter instead of a constant value.
See an example here:

<http://old.nabble.com/Problem:-semantic-lookahead-calling-boolean-functions-td23087841.html>

Bachelier, Georges wrote:

>
> Hi!
>
> I have been searching a solution for three days to the following problem:
>
> I am building a JavaCC parser for a language having an original
> feature: the language allows you to define user key words which may be
> found anywhere in the input file after their declaration.
>
> User key words are declared with the following statement :
> UserKeywords <my_user_kw1> … <my_user_kwn> ;
>
> Once keywords have been declared, they may be used like this :
>
> <my_user_kw1> {
>
> <any_sequence_of_characters>
>
> }
>
> <my_user_kwn> {
>
> <any_sequence_of_characters>
>
> }
>
> The sequences of characters do not matter; we do not use them, but
> they are surrounded by braces.
>
> I have written a parser production in order to get the user key words
> and I have tried to play with the tokenizer CommonTokenAction
> function, but without success.
>
> Could someone help me on this issue, please?
>
> Georges
>

--
Regards,
Farrukh

Web: http://www.wellfleetsoftware.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Dynamic tokenizing

2009-11-06T11:13:50Z

Hi!

I have been searching a solution for three days to the following problem:

I am building a JavaCC parser for a language having an original feature: the language allows you to define user key words which may be found anywhere in the input file after their declaration.

User key words are declared with the following statement : UserKeywords <my_user_kw1> … <my_user_kwn> ;

Once keywords have been declared, they may be used like this :

<my_user_kw1> {

<any_sequence_of_characters>

}

<my_user_kwn> {

<any_sequence_of_characters>

}

The sequences of characters do not matter; we do not use them, but they are surrounded by braces.

I have written a parser production in order to get the user key words and I have tried to play with the tokenizer CommonTokenAction function, but without success.

Could someone help me on this issue, please?

Georges

LOOKAHEAD question

2009-11-05T00:47:32Z

Hello,

I do not understand how lookahead works exactly.

In a regex, I can use a expression like this: "A+A", which means to get as many As and finally get another A.

(I know that it is equivalent to "AA+" which is simplier, but the previous is an example to help me understand lookahead in javacc)

I've tried to reproduce the "A+A" example in javacc, but it does not work.

From my understanding, a LOOKAHEAD of 2 could process the input "AAA", and a LOOKAHEAD of 4 could process the input "AAAA".

Why this example is not working?

++++++++++++++++++++++++++++++++++

options { STATIC = false; }

PARSER_BEGIN(B)

import java.io.*;

public class B {

static String test = "AAA";

public static void main(String args[]) throws Exception {

B app = new B();

BTokenManager ttm = new BTokenManager(new SimpleCharStream(new StringReader(test)));

app.ReInit(ttm);

app.Entry();

}

public B() { this((Reader) null); }

}

PARSER_END(B)

TOKEN : { <A: "A"> }

void Entry() : { }

{

( LOOKAHEAD(2) <A> )+

<A>

}

++++++++++++++++++++++++++++++++++

Best regards,

DAvid

Re: Update of the JavaCC cobol grammar

2009-10-30T11:40:07Z

On Oct 28, 2009, at 5:42 AM, thierry blind wrote:

> Hello,
> I would like to contribute to this wonderful project by updating the
> JavaCC cobol grammar that was initially made by Bernard Pinon (and
> that can be found in the repository of JavaCC grammars).
> I've heavily modified and improved this grammar, although it is (and
> i think will) never be perfect. The grammar was well tested with
> java 4.2 in a production environment.
> I've attached the new grammar file to this mail. I left some "TODO"
> comments in this file, for some pending improvements, but those
> "ugly" comments can be removed if necessary.
> I don't know if it is the right address to sind this file, hope I'm
> not wrong ;)
>
> Comments and suggestions are welcome :)

Hi Thierry -

I've uploaded your new grammar here:

https://javacc.dev.java.net/servlets/ProjectDocumentList?folderID=110

Thanks!

Yours,

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Matching productions embedded in arbitrary text

2009-10-30T11:33:21Z

On Oct 29, 2009, at 10:36 AM, Farrukh Najmi wrote:

>>
>> To me the tricky bit here is that you don't know the function names
>> until runtime, so an input string like this:
>>
>> texttext fiddle("a", 1)texttext
>>
>> would only be a FunctionCall if the name "fiddle" is in
>> FUNCTION_NAMES, right?
>>
>> With that constraint, I'm not sure how to get JavaCC to tokenize
>> the input data. I wonder if you might want to extract the function
>> declarations using a regex. You could use a regex like ".* (" +
>> someFunctionName + "(.*))" to extract the function declaration from
>> the surrounding text, and then you could pass the match data to
>> JavaCC to create the parse tree. This would only work if it's just
>> a function call, though, it wouldn't work if the function body were
>> there because that could include nested parenthesis which would
>> confuse the regex.
>
> Hi Tom,
>
> My experience suggests that part seems easy. You just define a
> LOOKAHEAD that uses a function seeFunctionCall() where the function
> looks at next few tokens and decides whether it has a function name
> or not. See example below...

Hm, yup, semantic lookahead could be useful there, good idea. But
what I'm wondering about though... how do you tokenize? Do you just
let every character be a separate token?

Yours,

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

javacc better error messaging

2009-10-30T04:34:12Z

I'd like to know about error messaging capabilities.

javacc 5.0

Assumptions:
I have 5 fields with 5 numbers each ... with some separator.
f.e.:
01234*56789*12391*98765

I cannot change the format and have a specification saying that the first 5 digits are variable say A, the next B ...

| <#NUM_CHAR: ["0"-"9"]>
| <N5: <NUM_CHAR><NUM_CHAR><NUM_CHAR><NUM_CHAR><NUM_CHAR> >
| <DELIMITER: *>

Sometimes I get data violating the format. Example: 4 instead of 5 digits. How can I tell which variable A,B,... is meant?

Parse error says only: <N5> expected, got: 1234

My silly workaround is this:

{ currentField = "A"; } (t = <N5>) { obj.setA(t.image); }

and in the parseexception case I attach the "currentField" value to the error msg. How would a pro do this? :-)

---

first i tried this:
| <A: N5>
| <B: N5>
...

Since <N5> was the first public occurrance ... all <A.>, <B.> (dots for non-html markup) always showed up as <N5> which isnt very distinctive.

---

2)
| <N5: <NUM_CHAR><NUM_CHAR><NUM_CHAR><NUM_CHAR><NUM_CHAR> >

According to the FAQ this should work but doesnt:
| <N5: <NUM_CHAR>{5} >

???

Using JavaCC with spring

2009-10-29T13:14:24Z

I am using JavaCC generated parser with spring framework. I define a
singleton instance of my Parser in spring XMl configuration. When I need
to use the parser I need to set it to a new Reader or InputStream. To do
this I needed to add an argument-less constructor and a setInputStream
method as follows:

public FunctionParser() {
}

public void setInputStream(InputStream stream, String encoding) {
//Following is copied from generated constructor with same args
try { jj_input_stream = new SimpleCharStream(stream, encoding,
1, 1); } catch(java.io.UnsupportedEncodingException e) { throw new
RuntimeException(e); }
token_source = new FunctionParserTokenManager(jj_input_stream);
token = new Token();
jj_ntk = -1;
jj_gen = 0;
for (int i = 0; i < 3; i++) jj_la1[i] = -1;
for (int i = 0; i < jj_2_rtns.length; i++) jj_2_rtns[i] = new
JJCalls();
}

This is a hack because the above code copied from generated constructor
with same args changes as grammar changes.
Is there a better way to do this?

--
Regards,
Farrukh

Web: http://www.wellfleetsoftware.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Matching productions embedded in arbitrary text

2009-10-29T10:03:33Z

Farrukh Najmi wrote:

> Tom Copeland wrote:
>> To me the tricky bit here is that you don't know the function names
>> until runtime, so an input string like this:
>>
>> texttext fiddle("a", 1)texttext
>>
>> would only be a FunctionCall if the name "fiddle" is in
>> FUNCTION_NAMES, right?
>>
>> With that constraint, I'm not sure how to get JavaCC to tokenize the
>> input data. I wonder if you might want to extract the function
>> declarations using a regex. You could use a regex like ".* (" +
>> someFunctionName + "(.*))" to extract the function declaration from
>> the surrounding text, and then you could pass the match data to
>> JavaCC to create the parse tree. This would only work if it's just a
>> function call, though, it wouldn't work if the function body were
>> there because that could include nested parenthesis which would
>> confuse the regex.
>
> Hi Tom,
>
> My experience suggests that part seems easy. You just define a
> LOOKAHEAD that uses a function seeFunctionCall() where the function
> looks at next few tokens and decides whether it has a function name or
> not. See example below...
>
> String textOrFunctionCall():
> {
> String rs = new String("");
> String ts;
> }
> {
> (
> LOOKAHEAD({!seeFunctionCall()}) ts = text() {rs = rs + ts;}
> | LOOKAHEAD({seeFunctionCall()}) ts = FunctionCall() {rs = rs + " "
> + ts + " ";}
> )
> {
> return rs;
> }
> }
>

Now that my parser is working with some place holder production for
matching the arbitrary text I now need to make it work for matching
arbitrary text that is not a function call. To me this is the real
challenge (as I said above the user configured function names is a
simple issue that I already have sorted out).

Let me as a more focused question.

Is it possible to define a grammar that can have occurrence of a
specified production within *arbitrary text*? If so does any one have
some sample of what that might look like?

Thanks.

--
Regards,
Farrukh

Web: http://www.wellfleetsoftware.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Matching productions embedded in arbitrary text

2009-10-29T07:36:51Z

Tom Copeland wrote:

>
> On Oct 28, 2009, at 8:56 PM, Farrukh Najmi wrote:
>
>> Hi Chris and Tom,
>>
>> Thank you for your help. Please see inline below...
>>
>> J.Chris Findlay wrote:
>>> I think it might need to be:
>>> texttexttextfoobar("arg1",2.0)texttexttext
>>
>> Above is more closer to what I intended. The only thing to add is that
>> I think the function name would have to be preceded by white space so
>> here a modified version of your example that would be valid:
>>
>> texttexttext foobar("arg1",2.0)texttexttext
>>
>> BTW, to provide some context, I work on ebXML RegRep specificationm.
>> The spec supports multiple query languages (e.g. SQL, EJBQL, SPARQL,
>> XQuery....) and is defining a syntax for user defined functions
>> (UDF). Since the functions could be embedded in any query language
>> expression I cannot assume a specific grammar.
>> I have made some progress on my parser by temporarily changing the
>> text() production to be a choice of various types of tokens for now.
>> So any suggestion on how to handle the arbitrary text requirement
>> would be great.
>
> To me the tricky bit here is that you don't know the function names
> until runtime, so an input string like this:
>
> texttext fiddle("a", 1)texttext
>
> would only be a FunctionCall if the name "fiddle" is in
> FUNCTION_NAMES, right?
>
> With that constraint, I'm not sure how to get JavaCC to tokenize the
> input data. I wonder if you might want to extract the function
> declarations using a regex. You could use a regex like ".* (" +
> someFunctionName + "(.*))" to extract the function declaration from
> the surrounding text, and then you could pass the match data to JavaCC
> to create the parse tree. This would only work if it's just a
> function call, though, it wouldn't work if the function body were
> there because that could include nested parenthesis which would
> confuse the regex.

Hi Tom,

My experience suggests that part seems easy. You just define a LOOKAHEAD
that uses a function seeFunctionCall() where the function looks at next
few tokens and decides whether it has a function name or not. See
example below...

String textOrFunctionCall():
{
String rs = new String("");
String ts;
}
{
(
LOOKAHEAD({!seeFunctionCall()}) ts = text() {rs = rs + ts;}
| LOOKAHEAD({seeFunctionCall()}) ts = FunctionCall() {rs = rs + " "
+ ts + " ";}
)
{
return rs;
}
}

--
Regards,
Farrukh

Web: http://www.wellfleetsoftware.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Matching productions embedded in arbitrary text

2009-10-29T07:26:17Z

On Oct 28, 2009, at 8:56 PM, Farrukh Najmi wrote:

> Hi Chris and Tom,
>
> Thank you for your help. Please see inline below...
>
> J.Chris Findlay wrote:
>> I think it might need to be:
>> texttexttextfoobar("arg1",2.0)texttexttext
>
> Above is more closer to what I intended. The only thing to add is that
> I think the function name would have to be preceded by white space
> so here a modified version of your example that would be valid:
>
> texttexttext foobar("arg1",2.0)texttexttext
>
> BTW, to provide some context, I work on ebXML RegRep specificationm.
> The spec supports multiple query languages (e.g. SQL, EJBQL, SPARQL,
> XQuery....) and is defining a syntax for user defined functions
> (UDF). Since the functions could be embedded in any query language
> expression I cannot assume a specific grammar.
> I have made some progress on my parser by temporarily changing the
> text() production to be a choice of various types of tokens for now.
> So any suggestion on how to handle the arbitrary text requirement
> would be great.

To me the tricky bit here is that you don't know the function names
until runtime, so an input string like this:

texttext fiddle("a", 1)texttext

would only be a FunctionCall if the name "fiddle" is in
FUNCTION_NAMES, right?

With that constraint, I'm not sure how to get JavaCC to tokenize the
input data. I wonder if you might want to extract the function
declarations using a regex. You could use a regex like ".* (" +
someFunctionName + "(.*))" to extract the function declaration from
the surrounding text, and then you could pass the match data to JavaCC
to create the parse tree. This would only work if it's just a
function call, though, it wouldn't work if the function body were
there because that could include nested parenthesis which would
confuse the regex.

Yours,

Tom
http://generatingparserswithjavacc.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Matching productions embedded in arbitrary text

2009-10-28T17:56:39Z

Hi Chris and Tom,

Thank you for your help. Please see inline below...

J.Chris Findlay wrote:
> I think it might need to be:
> texttexttextfoobar("arg1",2.0)texttexttext

Above is more closer to what I intended. The only thing to add is that
I think the function name would have to be preceded by white space so
here a modified version of your example that would be valid:

texttexttext foobar("arg1",2.0)texttexttext

BTW, to provide some context, I work on ebXML RegRep specificationm. The
spec supports multiple query languages (e.g. SQL, EJBQL, SPARQL,
XQuery....) and is defining a syntax for user defined functions (UDF).
Since the functions could be embedded in any query language expression
I cannot assume a specific grammar.

I have made some progress on my parser by temporarily changing the
text() production to be a choice of various types of tokens for now.
So any suggestion on how to handle the arbitrary text requirement would
be great.

Thanks again.

>
> On Thu, Oct 29, 2009 at 11:55 AM, Tom Copeland <tom@...
> <mailto:tom@...>> wrote:
>
>
...

>
> Hi Farrukh -
>
> In this case the error message is indeed accurate; since
> (anything)* is a "zero or more" expansion it can be matched by
> nothing at all. Thus it could just sit there in an infinite loop.
>
> Let's see though... so that I understand it, assuming that
> "foobar" is one of the FUNCTION_NAMES in the HashSet, would this
> be valid input data?
>
> texttexttextfoobar(arg1,arg2)texttexttext
>
> Thanks,
>
> Tom
> http://generatingparserswithjavacc.com/
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> <mailto:users-unsubscribe@...>
> For additional commands, e-mail: users-help@...
> <mailto:users-help@...>
>
>
>
>
> --
> - J.Chris Findlay
> (c:

Re: Matching productions embedded in arbitrary text

2009-10-28T16:04:02Z

I think it might need to be:
texttexttextfoobar("arg1",2.0)texttexttext

On Thu, Oct 29, 2009 at 11:55 AM, Tom Copeland <tom@...> wrote:

On Oct 28, 2009, at 9:50 AM, Farrukh Najmi wrote:

Hi Guys,

I need some help with a grammar (see attached).

The grammar processes text that consists of function calls embedded any where and possibly multiple time within arbitrary text.

The FunctionCall() production is expected to be recognized when a function name (configurable via setFUNCTION_NAMES) is matched in the text.
Arbitrary text is expected to be recognized by the text() production.

When I run javacc I get the following error:

Error: Line 73, Column 5: Expansion within "(...)*" can be matched by empty string.

Can any one help me get over the hump with this problem? If there is a better way to do what I am trying to do please let me know.

Thanks in advance for your help.

Hi Farrukh -

In this case the error message is indeed accurate; since (anything)* is a "zero or more" expansion it can be matched by nothing at all. Thus it could just sit there in an infinite loop.

Let's see though... so that I understand it, assuming that "foobar" is one of the FUNCTION_NAMES in the HashSet, would this be valid input data?

texttexttextfoobar(arg1,arg2)texttexttext

Thanks,

Tom
http://generatingparserswithjavacc.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

--
- J.Chris Findlay
(c:

Re: Matching productions embedded in arbitrary text

2009-10-28T15:55:51Z

On Oct 28, 2009, at 9:50 AM, Farrukh Najmi wrote:

> Hi Guys,
>
> I need some help with a grammar (see attached).
>
> The grammar processes text that consists of function calls embedded
> any where and possibly multiple time within arbitrary text.
>
> The FunctionCall() production is expected to be recognized when a
> function name (configurable via setFUNCTION_NAMES) is matched in the
> text.
> Arbitrary text is expected to be recognized by the text() production.
>
> When I run javacc I get the following error:
>
> Error: Line 73, Column 5: Expansion within "(...)*" can be matched
> by empty string.
>
> Can any one help me get over the hump with this problem? If there is
> a better way to do what I am trying to do please let me know.
>
> Thanks in advance for your help.

Hi Farrukh -

In this case the error message is indeed accurate; since (anything)*
is a "zero or more" expansion it can be matched by nothing at all.
Thus it could just sit there in an infinite loop.

Let's see though... so that I understand it, assuming that "foobar" is
one of the FUNCTION_NAMES in the HashSet, would this be valid input
data?

texttexttextfoobar(arg1,arg2)texttexttext

Thanks,

Tom
http://generatingparserswithjavacc.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Question on examples/Lookahead

2009-10-28T15:14:13Z

On Oct 20, 2009, at 5:41 PM, Riadh Elloumi wrote:

> Hi all,
>
> I am newbie to JavaCC and I started by the tutorials. I tried the
> examples/Lookahead/Example1.jj and I am surprised that the grammar
> does not accept "abc" as it is said on the tutorial. I am using
> JavaCC 5.0 and I compile and run the example with these commands:
>
> riadh@giscard:Lookahead$ ../../bin/javacc Example1.jj
> Java Compiler Compiler Version 5.0 (Parser Generator)
> (type "javacc" with no arguments for help)
> Reading from file Example1.jj . . .
> Warning: Choice conflict in [...] construct at line 60, column 7.
> Expansion nested within construct and expansion following
> construct
> have common prefixes, one of which is: "c"
> Consider using a lookahead of 2 or more for nested expansion.
> File "TokenMgrError.java" is being rebuilt.
> File "ParseException.java" is being rebuilt.
> File "Token.java" is being rebuilt.
> File "SimpleCharStream.java" is being rebuilt.
> Parser generated with 0 errors and 1 warnings.
> riadh@giscard:Lookahead$ javac *.java
> riadh@giscard:Lookahead$ java Example
> abc
> Exception in thread "main" ParseException: Encountered "<EOF>" at
> line 1, column 4.
> Was expecting:
> "c" ...
>
> at Example.generateParseException(Example.java:206)
> at Example.jj_consume_token(Example.java:144)
> at Example.Input(Example.java:12)
> at Example.main(Example.java:6)
> riadh@giscard:Lookahead$
>
> Can you help me to understand why "abc" is not accepted?

Hello Riadh -

I think the idea with examples/Lookahead/Example1.jj is to illustrate
how backtracking affects a grammar. In terms of BNF, Example1.jj can
handle "abc", but since JavaCC parsers don't backtrack, the parser
that JavaCC generates can't. If you look at section 6 of the example/
Lookahead/README file you'll see a discussion of how to solve the
problem in Example1.jj with semantic lookahead.

Hope this helps,

Yours,

Tom
http://generatingparserswithjavacc.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Matching productions embedded in arbitrary text

2009-10-28T10:17:02Z

I would be grateful for *any* general advice on the feasability of my
parser's goals.
Is it even possible to match specific tokens intermixed with arbitrary
text? Has anyone done
a similar parser?

Farrukh Najmi wrote:

> Hi Guys,
>
> I need some help with a grammar (see attached).
>
> The grammar processes text that consists of function calls embedded
> any where and possibly multiple time within arbitrary text.
>
> The FunctionCall() production is expected to be recognized when a
> function name (configurable via setFUNCTION_NAMES) is matched in the
> text.
> Arbitrary text is expected to be recognized by the text() production.
>
> When I run javacc I get the following error:
>
> Error: Line 73, Column 5: Expansion within "(...)*" can be matched by
> empty string.
>
> Can any one help me get over the hump with this problem? If there is a
> better way to do what I am trying to do please let me know.
>
> Thanks in advance for your help.

Re: Matching productions embedded in arbitrary text

2009-10-28T09:53:10Z

The base problem is that in some cases, a tokenizer cannot unambiguously
define the token kind.

This is similar to a parsing problem with PL/I (which has no reserved
keywords). In "IF(...) ... ;" IF can be either a keyword or a variable
name and it cannot be determined until the token after the matching
right paren is examined. There may be any number of tokens (e.g.
expressions) between the parens. One approach that works is to tokenize
IF as a keyword and then use a specialized lookahead routine that alters
the "kind" field in the IF token to "identifier" if necessary (before it
is parsed).

In this case, a similar approach using a specialized lookahead to
identify the token kind (as was pointed out) is reasonably straight
forward.

Farrukh Najmi wrote:

> Tom Copeland wrote:
>>
>> On Oct 28, 2009, at 8:56 PM, Farrukh Najmi wrote:
>>
>>> Hi Chris and Tom,
>>>
>>> Thank you for your help. Please see inline below...
>>>
>>> J.Chris Findlay wrote:
>>>> I think it might need to be:
>>>> texttexttextfoobar("arg1",2.0)texttexttext
>>>
>>> Above is more closer to what I intended. The only thing to add is that
>>> I think the function name would have to be preceded by white space
>>> so here a modified version of your example that would be valid:
>>>
>>> texttexttext foobar("arg1",2.0)texttexttext
>>>
>>> BTW, to provide some context, I work on ebXML RegRep specificationm.
>>> The spec supports multiple query languages (e.g. SQL, EJBQL, SPARQL,
>>> XQuery....) and is defining a syntax for user defined functions
>>> (UDF). Since the functions could be embedded in any query language
>>> expression I cannot assume a specific grammar.
>>> I have made some progress on my parser by temporarily changing the
>>> text() production to be a choice of various types of tokens for now.
>>> So any suggestion on how to handle the arbitrary text requirement
>>> would be great.
>>
>> To me the tricky bit here is that you don't know the function names
>> until runtime, so an input string like this:
>>
>> texttext fiddle("a", 1)texttext
>>
>> would only be a FunctionCall if the name "fiddle" is in
>> FUNCTION_NAMES, right?
>>
>> With that constraint, I'm not sure how to get JavaCC to tokenize the
>> input data. I wonder if you might want to extract the function
>> declarations using a regex. You could use a regex like ".* (" +
>> someFunctionName + "(.*))" to extract the function declaration from
>> the surrounding text, and then you could pass the match data to
>> JavaCC to create the parse tree. This would only work if it's just a
>> function call, though, it wouldn't work if the function body were
>> there because that could include nested parenthesis which would
>> confuse the regex.
>
> Hi Tom,
>
> My experience suggests that part seems easy. You just define a
> LOOKAHEAD that uses a function seeFunctionCall() where the function
> looks at next few tokens and decides whether it has a function name or
> not. See example below...
>
> String textOrFunctionCall():
> {
> String rs = new String("");
> String ts;
> }
> {
> (
> LOOKAHEAD({!seeFunctionCall()}) ts = text() {rs = rs + ts;}
> | LOOKAHEAD({seeFunctionCall()}) ts = FunctionCall() {rs = rs + " "
> + ts + " ";}
> )
> {
> return rs;
> }
> }
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.423 / Virus Database: 270.14.38/2467 - Release Date: 10/29/09 07:38:00
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Matching productions embedded in arbitrary text

2009-10-28T06:50:00Z

Hi Guys,

I need some help with a grammar (see attached).

The grammar processes text that consists of function calls embedded any
where and possibly multiple time within arbitrary text.

The FunctionCall() production is expected to be recognized when a
function name (configurable via setFUNCTION_NAMES) is matched in the text.
Arbitrary text is expected to be recognized by the text() production.

When I run javacc I get the following error:

Error: Line 73, Column 5: Expansion within "(...)*" can be matched by
empty string.

Can any one help me get over the hump with this problem? If there is a
better way to do what I am trying to do please let me know.

Thanks in advance for your help.

--
Regards,
Farrukh

Web: http://www.wellfleetsoftware.com

/**
*
*
*/

options{
STATIC=false ;
IGNORE_CASE=true ;
// DEBUG_LOOKAHEAD= true ;
}

PARSER_BEGIN(FunctionParser)
package com.example.function;

import java.io.Reader;
import java.io.FileInputStream;
import java.util.HashSet;
import java.util.Set;

public class FunctionParser {
protected static Set<String> FUNCTION_NAMES = new HashSet<String>();

public void reInit(Reader input) {
ReInit(input);
}

protected boolean seeFunctionCall() {
return "(".equals(getToken(2).image)
&& FUNCTION_NAMES.contains(getToken(1).image.toUpperCase());
}

public static void setFUNCTION_NAMES(Set<String> functionNames) {
FUNCTION_NAMES = functionNames;
}
}
PARSER_END(FunctionParser)

SKIP:
{
" "
| "\t"
| "\r"
| "\n"
}

TOKEN : /* Numeric Constants */
{
< S_NUMBER: <FLOAT>
| <FLOAT> ( ["e","E"] ([ "-","+"])? <FLOAT> )?
>
| < #FLOAT: <INTEGER>
| <INTEGER> ( "." <INTEGER> )?
| "." <INTEGER>
>
| < #INTEGER: ( <DIGIT> )+ >
| < #DIGIT: ["0" - "9"] >
}

TOKEN:
{
< S_IDENTIFIER: (<LETTER>)+ (<DIGIT> | <LETTER> |<SPECIAL_CHARS>)* >
| < #LETTER: ["a"-"z", "A"-"Z"] >
| < #SPECIAL_CHARS: "_" >
| < S_CHAR_LITERAL: "'" (~["'"])* "'" ("'" (~["'"])* "'")*>
| < S_QUOTED_IDENTIFIER: "\"" (~["\n","\r","\""])* "\"" >
| < S_TEXT: ~[] >
}

void start():
{}
{
(textOrFunctionCall())*

}

void text():
{}
{
( <S_TEXT> )*
}

void textOrFunctionCall():
{}
{
text()
| LOOKAHEAD({seeFunctionCall()}) FunctionCall()
}

/* ---------------- Grammar Productions --------------------- */

void FunctionCall():
{}
{
FunctionReference() "(" (FunctionArgumentList())* ")"
}

void FunctionReference():
{
String name;
}
{
<S_IDENTIFIER> {name = token.image;}

}

void FunctionArgumentList():
{}
{
FunctionArgument() ("," FunctionArgument())*
}

void FunctionArgument():
{}
{
FunctionCall()
| <S_QUOTED_IDENTIFIER>
| <S_NUMBER>
}

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Update of the JavaCC cobol grammar

2009-10-28T02:42:54Z

Hello,
I would like to contribute to this wonderful project by updating the JavaCC cobol grammar that was initially made by Bernard Pinon (and that can be found in the repository of JavaCC grammars).
I've heavily modified and improved this grammar, although it is (and i think will) never be perfect. The grammar was well tested with java 4.2 in a production environment.
I've attached the new grammar file to this mail. I left some "TODO" comments in this file, for some pending improvements, but those "ugly" comments can be removed if necessary.
I don't know if it is the right address to sind this file, hope I'm not wrong ;)

Comments and suggestions are welcome :)

Best regards,
Thierry BLIND.

Nouveau Windows 7 : Trouvez le PC qui vous convient. En savoir plus.
/**
* This is an experimental COBOL-85 grammar for JavaCC.
* <p>References used for this grammar :<ol>
* <li> Ralf Lämmel & Chris Verhoef : VS COBOL II grammar Version 1.0.3
* <li> Fujitsu Cobol compiler reference
* <li> Compaq-HP Non-stop Cobol reference
* </od>
* <p>This grammar is released under the GNU Library Public License (LGPL).
* <p><address>Bernard Pinon, august 2002</address>
* <p><address>Thierry Blind, september 2009</address>
* @see http://www.gnu.org/licenses/lgpl.html
* @see http://www.cwi.nl/
* @see http://www.wins.uva.nl/
* @author Thierry Blind (initial work from Bernard Pinon)
*
* <p>History:<ul>
* <li><date>9-aug-2002</date> [BP] make changes to comply with JTB :<ol>
* <li> Changed Integer() to IntegerConstant() to avoid conflicts with java.lang.Integer
* <li> Removed Java code to make the grammar bare bone
* <li> Fixed Identification Division entries
* </ol>
* <li><date>8-aug-2002</date> [BP] fixed several bugs :<ol>
* <li> INSTALLATION section
* <li> wrong spelling in keywords
* <li> DOT missing in FD/SD entries
* <li> wrong regular expressions for levels
* <li> possibility to have repeated string literals, e.g. "AA" "BB" -> "AABB"
* <li> optimized data division by placing the most common clauses first
* <li> DOT missing in exit statement
* <li> confusion between Identifier and QualifiedDataName (to be checked)
* <li> fixed NumericConstant to use Integer instead of INTEGER
* <li> Correcting AdvancingPhrase so LINE is optional
* <li> Fixed problems with abbreviated conditions
* <li> Fixed Add statement
* </ol>
* <li><date>01-sept-2009</date> :<ol>
* <li> corrected picture values and enhanced linkage section
* <li> corrected relational operators, arithmetic expressions and conditions
* <li> corrected perform, evaluate and call clauses
* <li> corrected cobol word, litteral, identifier, comment declarations
* <li> created Iterables*() variants for some clauses
* <li> added function support from cobol 2000
* <li> added LOOKAHEAD() instructions, with reorderings, to remove parser ambiguities
* <li> ... and many many corrections, new clauses and new tokens relative to cobol 85 and (some from) cobol 2000
* </ol>
* </ul>
*/

/**
* Cobol is not case sensitive. It is not LL(1) neither...
*/

options {
IGNORE_CASE = true;
//FORCE_LA_CHECK = true;
LOOKAHEAD = 3;
//DEBUG_PARSER = true;
//DEBUG_LOOKAHEAD = true;
STATIC = false;
}

/**
* The parser class.
*/

PARSER_BEGIN(CobolParser)
package convex.step2.parser;

public class CobolParser {

}

PARSER_END(CobolParser)

////////////////////////////////////////////////////////////////////////////////
// Lexical structure
////////////////////////////////////////////////////////////////////////////////

SPECIAL_TOKEN :
{
< EOL: "\n" >
| < SPACECHAR: ( " " | "\t" | "\f" | ";" | "\r" )+ >
}

SPECIAL_TOKEN :
{
< COMMENT: ( "*>" | "|" ) (~["\n","\r"])* >
| < PREPROC_COMMENT: "*|" (~["\n","\r"])* >
| < SPACE_SEPARATOR : ( <SPACECHAR> | <EOL> )+ >
| < COMMA_SEPARATOR : "," <SPACE_SEPARATOR> >
}

TOKEN :
{
// TODO : Could be simplified by <DOTCHAR> ?
< DOT : <DOTCHAR> <SPACE_SEPARATOR> >
}

////////////////////////////////////////////////////////////////////////////////
// RESERVED WORDS. Looks like an excerpt of the dictionary of data processing.
// It is very hard to find significative identifiers in COBOL that are not
// reserved...
////////////////////////////////////////////////////////////////////////////////

TOKEN :
{
/* A */
< ACCEPT: "accept" >
| < ACCESS: "access" >
| < ADD: "add" >
| < ADDRESS: "address" > // extension to support pointers
| < ADVANCING: "advancing" >
| < AFTER: "after" >
| < ALL: "all" >
| < ALPHABET: "alphabet" >
| < ALPHABETIC: "alphabetic" >
| < ALPHABETIC_LOWER: "alphabetic-lower" >
| < ALPHABETIC_UPPER: "alphabetic-upper" >
| < ALPHANUMERIC: "alphanumeric" >
| < ALPHANUMERIC_EDITED: "alphanumeric-edited" >
| < ALSO: "also" >
| < ALTER: "alter" >
| < ALTERNATE: "alternate" >
| < AND: "and" >
| < ANY: "any" >
| < APPROXIMATE: "approximate" > // tandem extension
| < ARE: "are" >
| < AREA: "area" >
| < AREAS: "areas" >
| < ASCENDING: "ascending" >
| < ASSIGN: "assign" >
| < AT: "at" >
| < AUTHOR: "author" >
/* B */
| < BEFORE: "before" >
| < BEGINNING: "beginning" > // extension
| < BINARY: "binary" >
| < BLANK: "blank" >
| < BLOCK: "block" >
| < BOTTOM: "bottom" >
| < BY: "by" >
/* C */
| < CALL: "call" >
| < CALL_CONVENTION: "call-convention" >
| < CANCEL: "cancel" >
| < CD: "cd" >
| < CF: "cf" >
| < CH: "ch" >
| < CHARACTER: "character" >
| < CHARACTERS: "characters" >
| < CLASS: "class" >
| < CLOCK_UNITS: "clock-units" >
| < CLOSE: "close" >
//| < COBOL: "cobol" >
| < CODE: "code" >
| < CODE_SET: "code-set" >
| < COLLATING: "collating" >
| < COLUMN: "column" >
| < COMMA: "comma" >
| < COMMON: "common" >
| < COMMUNICATION: "communication" >
| < COMP: "comp" >
| < COMP_1: "comp-1" > // extension in many dialects
| < COMP_2: "comp-2" > // extension in many dialects
| < COMP_3: "comp-3" > // extension in many dialects
| < COMP_4: "comp-4" > // extension in many dialects
| < COMPUTATIONAL: "computational" >
| < COMPUTATIONAL_1: "computational-1" > // extension
| < COMPUTATIONAL_2: "computational-2" > // extension
| < COMPUTATIONAL_3: "computational-3" > // extension
| < COMPUTATIONAL_4: "computational-4" > // extension
| < COMPUTE: "compute" >
| < CONFIGURATION: "configuration" >
| < CONTAINS: "contains" >
| < CONTENT: "content" >
| < CONTINUE: "continue" >
| < CONTROL: "control" >
| < CONTROLS: "controls" >
| < CONVERTING: "converting" >
| < COPY: "copy" >
| < CORR: "corr" >
| < CORRESPONDING: "corresponding" >
| < COUNT: "count" >
| < CURRENCY: "currency" >
/* D */
| < DATA: "data" >
| < DATE: "date" >
| < DATE_COMPILED: "date-compiled" >
| < DATE_WRITTEN: "date-written" >
| < DAY: "day" >
| < DAY_OF_WEEK: "day-of-week" >
| < DBCS: "dbcs" > // extension to support double bytes characters
| < DE: "de" >
| < DEBUG_CONTENTS: "debug-contents" >
| < DEBUG_ITEM: "debug-item" >
| < DEBUG_LINE: "debug-line" >
| < DEBUG_NAME: "debug-name" >
| < DEBUG_SUB_1: "debug-sub-1" >
| < DEBUG_SUB_2: "debug-sub-2" >
| < DEBUG_SUB_3: "debug-sub-3" >
| < DEBUGGING: "debugging" >
| < DECIMAL_POINT: "decimal-point" >
| < DECLARATIVES: "declaratives" >
| < DELETE: "delete" >
| < DELIMITED: "delimited" >
| < DELIMITER: "delimiter" >
| < DEPENDING: "depending" >
| < DESCENDING: "descending" >
| < DESCRIPTOR: "descriptor" >
| < DESTINATION: "destination" >
| < DETAIL: "detail" >
| < DISABLE: "disable" >
| < DISPLAY: "display" >
| < DISPLAY_1: "display-1" > // extension
| < DIVIDE: "divide" >
| < DIVISION: "division" >
| < DOWN: "down" >
| < DUPLICATES: "duplicates" >
| < DYNAMIC: "dynamic" >
/* E */
| < EGCS: "egcs" > // extension
| < EGI: "egi" >
| < ELSE: "else" >
| < EMI: "emi" >
| < ENABLE: "enable" >
| < END: "end" >
| < END_ADD: "end-add" >
| < END_CALL: "end-call" >
| < END_COMPUTE: "end-compute" >
| < END_DELETE: "end-delete" >
| < END_DIVIDE: "end-divide" >
| < END_EVALUATE: "end-evaluate" >
| < END_IF: "end-if" >
| < END_MULTIPLY: "end-multiply" >
| < END_OF_PAGE: "end-of-page" >
| < END_PERFORM: "end-perform" >
| < END_READ: "end-read" >
| < END_RECEIVE: "end-receive" >
| < END_RETURN: "end-return" >
| < END_REWRITE: "end-rewrite" >
| < END_SEARCH: "end-search" >
| < END_START: "end-start" >
| < END_STRING: "end-string" >
| < END_SUBTRACT: "end-subtract" >
| < END_UNSTRING: "end-unstring" >
| < END_WRITE: "end-write" >
| < ENDING: "endinf" >
| < ENTER: "enter" > // not present in all COBOLs, use CALL instead
| < ENTRY: "entry" >
| < ENVIRONMENT: "environment" >
| < EOP: "eop" >
| < EQUAL: "equal" >
| < ERROR: "error" >
| < ESI: "esi" >
| < EVALUATE: "evaluate" >
| < EVERY: "every" >
| < EXCEPTION: "exception" >
/* | < EXCLUSIVE: "exclusive" > tandem extension */
| < EXIT: "exit" >
| < EXTEND: "extend" >
| < EXTERNAL: "external" >
/* F */
| < FALSE: "false" >
| < FD: "fd" >
| < FILE: "file" >
| < FILE_CONTROL: "file-control" >
| < FILLER: "filler" >
| < FINAL: "final" >
| < FIRST: "first" >
| < FOOTING: "footing" >
| < FOR: "for" >
| < FROM: "from" >
| < FUNCTION: "function" >
/* G */
| < GENERATE: "generate" >
| < GOBACK: "goback" > // extension
| < GENERIC: "generic" > // tandem extension
| < GIVING: "giving" >
| < GLOBAL: "global" >
| < GO: "go" >
| < GREATER: "greater" >
| < GROUP: "group" >
/* H */
| < HEADING: "heading" >
| < HIGH_VALUE: "high-value" >
| < HIGH_VALUES: "high-values" >
/* I */
| < I_O: "i-o" >
| < I_O_CONTROL: "i-o-control" >
| < ID: "id" > // extension, synonym for IDENTIFICATION
| < IDENTIFICATION: "identification" >
| < IDENTIFIED: "identified" >
| < IF: "if" >
| < IMPLICIT: "implicit" >
| < IN: "in" >
| < INDEX: "index" >
| < INDEXED: "indexed" >
| < INDICATE: "indicate" >
| < INITIAL: "initial" >
| < INITIALIZE: "initialize" >
| < INITIATE: "initiate" >
| < INPUT: "input" >
| < INPUT_OUTPUT: "input-output" >
| < INSPECT: "inspect" >
| < INSTALLATION: "installation" >
| < INTO: "into" >
| < INVALID: "invalid" >
| < IS: "is" >
/* J */
| < JUST: "just" >
| < JUSTIFIED: "justified" >
| < JUSTIFY: "justify" >
/* K */
| < KANJI: "kanji" > // extension to support Kanji characters (japanese)
| < KEY: "key" >
/* L */
| < LABEL: "label" >
| < LAST: "last" >
| < LEADING: "leading" >
| < LEFT: "left" >
| < LENGTH: "length" >
| < LESS: "less" >
| < LIMIT: "limit" >
| < LIMITS: "limits" >
| < LINAGE: "linage" >
| < LINAGE_COUNTER: "linage_counter" >
| < LINE: "line" >
| < LINES: "lines" >
| < LINE_COUNTER: "line-counter" >
| < LINKAGE: "linkage" >
| < LOCK: "lock" >
| < LOCKFILE: "lockfile" > // tandem extension
| < LOW_VALUE: "low-value" >
| < LOW_VALUES: "low-values" >
/* M */
| < MEMORY: "memory" >
| < MERGE: "merge" >
| < MESSAGE: "message" >
| < MODE: "mode" >
| < MODULES: "modules" >
| < MORE_LABELS: "more-labels" > // IBM extension
| < MOVE: "move" >
| < MULTIPLE: "multiple" >
| < MULTIPLY: "multiply" >
/* N */
| < NATIVE: "native" >
| < NEGATIVE: "negative" >
| < NEXT: "next" >
| < NO: "no" >
| < NOT: "not" >
| < NULL: "null" > // tandem & IBM extension
| < NULLS: "nulls" > // tandem & IBM extension
| < NUMBER: "number" >
| < NUMERIC: "numeric" >
| < NUMERIC_EDITED: "numeric-edited" >
/* O */
| < OBJECT_COMPUTER: "object-computer" >
| < OCCURS: "occurs" >
| < OF: "of" >
| < OFF: "off" >
| < OMITTED: "omitted" >
| < ON: "on" >
| < OPEN: "open" >
| < OPTIONAL: "optional" >
| < OR: "or" >
| < ORDER: "order" >
| < ORGANIZATION: "organization" >
| < OTHER: "other" >
| < OUTPUT: "output" >
| < OVERFLOW: "overflow" >
/* P */
| < PACKED_DECIMAL: "packed-decimal" >
| < PADDING: "padding" >
| < PAGE: "page" >
| < PAGE_COUNTER: "page-counter" >
| < PASSWORD: "password" >
| < PERFORM: "perform" >
| < PF: "pf" >
| < PH: "ph" >
| < PIC: "pic" >
| < PICTURE: "picture" >
| < PLUS: "plus" >
| < POINTER: "pointer" >
| < POSITION: "position" >
| < POSITIVE: "positive" >
| < PRINTING: "printing" >
| < PROCEDURE: "procedure" >
| < PROCEDURES: "procedures" >
| < PROCEDURE_POINTER: "procedure-pointer" >
| < PROCEED: "proceed" >
| < PROGRAM: "program" >
| < PROGRAM_ID: "program-id" >
| < PROGRAM_STATUS: "program-status" > // tandem extension
| < PROMPT: "prompt" > // tandem extension
| < PROTECTED: "protected" > // tandem extension
| < PURGE: "purge" >
/* Q */
| < QUEUE: "queue" >
| < QUOTE: "quote" >
| < QUOTES: "quotes" >
/* R */
| < RANDOM: "random" >
| < RD: "rd" >
| < READ: "read" >
| < RECEIVE: "receive" >
| < RECEIVE_CONTROL: "receive-control" > // tandem extension
| < RECORD: "record" >
| < RECORDING: "recording" > // probably IBM extension
| < RECORDS: "records" >
| < REDEFINES: "redefines" >
| < REEL: "reel" >
| < REFERENCE: "reference" >
| < REFERENCES: "references" >
| < RELATIVE: "relative" >
| < RELEASE: "release" >
| < REMAINDER: "remainder" >
| < REMOVAL: "removal" >
| < RENAMES: "renames" >
| < REPLACE: "replace" >
| < REPLACING: "replacing" >
| < REPLY: "reply" > // tandem extension
| < REPORT: "report" >
| < REPORTING: "reporting" >
| < REPORTS: "reports" >
| < RERUN: "rerun" >
| < RESERVE: "reserve" >
| < RESET: "reset" >
| < RETURN: "return" > // IBM extension - does not do what expected ;-)
| < RETURN_CODE: "return-code" > // special register IBM
| < RETURNED: "returned" >
| < RETURNING: "returning" >
| < REVERSED: "reversed" >
| < REWIND: "rewind" >
| < REWRITE: "rewrite" >
| < RF: "rf" >
| < RH: "rh" >
| < RIGHT: "right" >
| < ROUNDED: "rounded" >
| < RUN: "run" >
/* S */
| < SAME: "same" >
| < SD: "sd" >
| < SEARCH: "search" >
| < SECTION: "section" >
| < SECURITY: "security" >
| < SEGMENT: "segment" >
| < SEGMENT_LIMIT: "segment-limit" >
| < SELECT: "select" >
| < SEND: "send" >
| < SENTENCE: "sentence" >
| < SEPARATE: "separate" >
| < SEQUENCE: "sequence" >
| < SEQUENTIAL: "sequential" >
| < SET: "set" >
| < SHARED: "shared" > // tandem extension, should be in C2000
| < SHIFT_IN: "shift-in" > // IBM special register
| < SHIFT_OUT: "shift-out" > // IBM special register
| < SIGN: "sign" >
| < SIZE: "size" >
| < SORT: "sort" >
| < SORT_CONTROL: "sort-control" > // IBM special register
| < SORT_CORE_SIZE: "sort-core-size" > // IBM special register
| < SORT_FILE_SIZE: "sort-file-size" > // IBM special register
| < SORT_MERGE: "sort-merge" >
| < SORT_MESSAGE: "sort-message" > // IBM special register
| < SORT_MODE_SIZE: "sort-mode-size" > // IBM special register
| < SORT_RETURN: "sort-return" > // IBM special register
| < SOURCE: "source" >
| < SOURCE_COMPUTER: "source-computer" >
| < SPACE: "space" >
| < SPACES: "spaces" >
| < SPECIAL_NAMES: "special-names" >
| < STANDARD: "standard" >
| < STANDARD_1: "standard-1" >
| < STANDARD_2: "standard-2" >
| < START: "start" >
| < STATUS: "status" >
| < STDCALL: "stdcall" >
| < STOP: "stop" >
| < STRING: "string" >
| < SUB_QUEUE_1: "sub-queue-1" >
| < SUB_QUEUE_2: "sub-queue-2" >
| < SUB_QUEUE_3: "sub-queue-3" >
| < SUBTRACT: "subtract" >
| < SUM: "sum" >
| < SUPPRESS: "suppress" >
| < SYMBOLIC: "symbolic" >
| < SYNC: "sync" >
| < SYNCHRONIZED: "synchronized" >
/* T */
| < TABLE: "table" >
| < TALLY: "tally" > // IBM special register
| < TALLYING: "tallying" >
| < TAPE: "tape" >
| < TERMINAL: "terminal" >
| < TERMINATE: "terminate" >
| < TEST: "test" >
| < TEXT: "text" >
| < THAN: "than" >
| < THEN: "then" >
| < THROUGH: "through" >
| < THRU: "thru" >
| < TIME: "time" >
| < TIMES: "times" >
| < TO: "to" >
| < TOP: "top" >
| < TRAILING: "trailing" >
| < TRUE: "true" >
| < TYPE: "type" >
/* U */
| < UNIT: "unit" >
| < UNLOCK: "unlock" > // tandem extension
| < UNLOCKFILE: "unlockfile" > // tandem again
| < UNLOCKRECORD: "unlockrecord" > // guess what
| < UNSTRING: "unstring" >
| < UNTIL: "until" >
| < UP: "up" >
| < UPON: "upon" >
| < USAGE: "usage" >
| < USE: "use" >
| < USING: "using" >
/* V */
| < VALUE: "value" >
| < VALUES: "values" >
| < VARYING: "varying" >
/* W */
| < WHEN: "when" >
| < WHEN_COMPILED: "when-compiled" > // IBM special register
| < WITH: "with" >
| < WORDS: "words" >
| < WORKING_STORAGE: "working-storage" >
| < WRITE: "write" >
/* Z */
| < ZERO: "zero" >
| < ZEROS: "zeros" >
| < ZEROES: "zeroes" >
/* Newly added */
| < ALPHANUMERIC_HASHTABLE: "alphanumeric-hashtable" >
| < BINARY_BYTE: "binary-byte" >
| < BINARY_DOUBLE: "binary-double" >
| < BINARY_LONG: "binary-long" >
| < BINARY_M: "binary-m" >
| < BINARY_REV: "binary-rev" >
| < BINARY_SHORT: "binary-short" >
| < BIT: "bit" >
| < COMPONENT: "component" >
| < COMPUTATIONAL_2_A: "computational-2-a" >
| < COMPUTATIONAL_1_A: "computational-1-a" >
| < COMPUTATIONAL_1_E: "computational-1-e" >
| < COMPUTATIONAL_1_M: "computational-1-m" >
| < COMPUTATIONAL_1_MVS: "computational-1-mvs" >
| < COMPUTATIONAL_1_REV: "computational-1-rev" >
| < COMPUTATIONAL_2_E: "computational-2-e" >
| < COMPUTATIONAL_2_M: "computational-2-m" >
| < COMPUTATIONAL_2_MVS: "computational-2-mvs" >
| < COMPUTATIONAL_2_REV: "computational-2-rev" >
| < COMPUTATIONAL_3_A: "computational-3-a" >
| < COMPUTATIONAL_3_M: "computational-3-m" >
| < COMPUTATIONAL_4_M: "computational-4-m" >
| < COMPUTATIONAL_5: "computational-5" >
| < COMPUTATIONAL_5_M: "computational-5-m" >
| < COMPUTATIONAL_6: "computational-6" >
| < COMPUTATIONAL_A: "computational-a" >
| < COMPUTATIONAL_B: "computational-b" >
| < COMPUTATIONAL_D: "computational-d" >
| < COMPUTATIONAL_M: "computational-m" >
| < COMPUTATIONAL_N: "computational-n" >
| < COMPUTATIONAL_P: "computational-p" >
| < COMPUTATIONAL_S: "computational-s" >
| < COMPUTATIONAL_X: "computational-x" >
| < COMPUTATIONAL_X_REV: "computational-x-rev" >
| < COMP_1_A: "comp-1-a" >
| < COMP_1_E: "comp-1-e" >
| < COMP_1_M: "comp-1-m" >
| < COMP_1_MVS: "comp-1-mvs" >
| < COMP_1_REV: "comp-1-rev" >
| < COMP_2_A: "comp-2-a" >
| < COMP_2_E: "comp-2-e" >
| < COMP_2_M: "comp-2-m" >
| < COMP_2_MVS: "comp-2-mvs" >
| < COMP_2_REV: "comp-2-rev" >
| < COMP_3_A: "comp-3-a" >
| < COMP_3_M: "comp-3-m" >
| < COMP_4_M: "comp-4-m" >
| < COMP_5: "comp-5" >
| < COMP_5_M: "comp-5-m" >
| < COMP_6: "comp-6" >
| < COMP_A: "comp-a" >
| < COMP_B: "comp-b" >
| < COMP_D: "comp-d" >
| < COMP_M: "comp-m" >
| < COMP_N: "comp-n" >
| < COMP_P: "comp-p" >
| < COMP_S: "comp-s" >
| < COMP_X: "comp-x" >
| < COMP_X_REV: "comp-x-rev" >
| < DEFAULT_FONT: "default-font" >
| < DISPLAY_WS: "display-ws" >
| < DOUBLE: "double" >
| < EXTERNAL_FORM: "external-form" >
| < FIXED_FONT: "fixed-font" >
| < FLOAT: "float" >
| < FLOAT_LONG: "float-long" >
| < FLOAT_SHORT: "float-short" >
| < FONT: "font" >
| < HANDLE: "handle" >
| < JBOOLEAN: "jboolean" >
| < JBYTE: "jbyte" >
| < JCHAR: "jchar" >
| < JDOUBLE: "jdouble" >
| < JFLOAT: "jfloat" >
| < JINT: "jint" >
| < JLONG: "jlong" >
| < JPACKED_DECIMAL: "jpacked-decimal" >
| < JSHORT: "jshort" >
| < JSTRING: "jstring" >
| < LARGE_FONT: "large-font" >
| < MEDIUM_FONT: "medium-font" >
| < NATIONAL: "national" >
| < NUMERIC_HASHTABLE: "numeric-hashtable" >
| < OBJECT: "object" >
| < OBJECT_HASHTABLE: "object-hashtable" >
| < PACKED_DECIMAL_A: "packed-decimal-a" >
| < PACKED_DECIMAL_E: "packed-decimal-e" >
| < PACKED_DECIMAL_H: "packed-decimal-h" >
| < PACKED_DECIMAL_I: "packed-decimal-i" >
| < PACKED_DECIMAL_M: "packed-decimal-m" >
| < SIGNED_INT: "signed-int" >
| < SIGNED_LONG: "signed-long" >
| < SIGNED_SHORT: "signed-short" >
| < SMALL_FONT: "small-font" >
| < SQLIND: "sqlind" >
| < THREAD: "thread" >
| < TRADITIONAL_FONT: "traditional-font" >
| < UNSIGNED_INT: "unsigned-int" >
| < UNSIGNED_LONG: "unsigned-long" >
| < UNSIGNED_SHORT: "unsigned-short" >
| < WINDOW: "window" >
}

TOKEN :
{
< LEVEL_66: "66" >
| < LEVEL_77: "77" >
| < LEVEL_88: "88" >
| < LEVEL_NUMBER: ( (("0")? ["1"-"9"]) | (["1"-"4"]["0"-"9"]) | "78" ) >
| < INTEGER: (["0"-"9"])+ >
| < MINUSCHAR: "-" > // a.k.a. dash
| < LPARENCHAR: "(" >
| < RPARENCHAR: ")" >
| < COLONCHAR: ":" >
| < DOTCHAR: "." >
| < COMMACHAR: "," >
| < DOUBLEDQUOTECHAR: "\"\"" >
| < QUOTECHAR: "\"" >
| < DOUBLEDAPOSTROPHE: "''" >
| < APOSTROPHE: "'" >
| < PLUSCHAR: "+" >
| < ASTERISKCHAR: "*" >
| < POWEROF: "**" >
| < SLASHCHAR: "/" >
| < DOLLARCHAR: "$" >
| < LESSTHANOREQUAL: "<=" >
| < LESSTHANCHAR: "<" >
| < MORETHANOREQUAL: ">=" >
| < MORETHANCHAR: ">" >
| < EQUALCHAR: "=" >
| < NOTEQUAL: "<>" >
| < HEXNUMBER: ["h","x"] ( ( <QUOTECHAR> ( ["0"-"9","a"-"f"] )+ <QUOTECHAR> )
| ( <APOSTROPHE> ( ["0"-"9","a"-"f"] )+ <APOSTROPHE> )
)
>
| < QUOTEDSTRING: ( <QUOTECHAR> (~["\""] | <DOUBLEDQUOTECHAR> )* <QUOTECHAR>
| <APOSTROPHE> (~["'"] | <DOUBLEDAPOSTROPHE> )* <APOSTROPHE>
)
>
// NB : COBOL_WORD can be defined as an INTEGER (or a LEVEL_66* or a LEVEL_NUMBER) too !
| < COBOL_WORD: (["a"-"z","0"-"9"])+ ( (<MINUSCHAR> | "_" )+ (["a"-"z","0"-"9"])+ )* >
| < OTHER_CHARS: ~[] >
}

// Contains at least one alphabetic, max. 30 char
void CobolWord() :
{}
{
<COBOL_WORD>
}

void QuotedText() :
{}
{
( IterableQuotedText() )+
}

void IterableQuotedText() :
{}
{
<QUOTEDSTRING> | <DOUBLEDQUOTECHAR> | <DOUBLEDAPOSTROPHE>
}

void IntegerConstant() :
{}
{
<LEVEL_66> | <LEVEL_77> | <LEVEL_88> | <LEVEL_NUMBER> | <INTEGER>
}

void NumericConstant() :
{}
{
[ <PLUSCHAR> | <MINUSCHAR> ] ( ( <DOTCHAR> | <COMMACHAR> ) IntegerConstant() | IntegerConstant() [ ( <DOTCHAR> | <COMMACHAR> ) IntegerConstant() ] )
}

void LevelNumber() :
{}
{
<LEVEL_NUMBER>
}

void FigurativeConstant() :
{}
{
<ZERO> | <ZEROS> | <ZEROES>
| <SPACE> | <SPACES>
| <HIGH_VALUE> | <HIGH_VALUES>
| <LOW_VALUE> | <LOW_VALUES>
| <QUOTE> | <QUOTES>
| <NULL> | <NULLS>
}

void NonNumericConstant() :
{}
{
( QuotedText()
| <HEXNUMBER>
)
}

void IterableNonNumericConstant() :
{}
{
( IterableQuotedText()
| <HEXNUMBER>
)
}

void Literal() :
{}
{
[ <ALL> ] ( NonNumericConstant()
| NumericConstant()
/* | <DBCS> */
| FigurativeConstant()
)
}

void IterableLiteral() :
{}
{
[ <ALL> ] ( IterableNonNumericConstant()
| NumericConstant()
/* | <DBCS> */
| FigurativeConstant()
)
}

////////////////////////////////////////////////////////////////////////////////
// LOGICAL EXPRESSIONS (e.g. in IF statements)
// one of the nice feature of COBOL is to allow for abbeviated conditions
// such as : if X less than 100 and more than 10 then ...
////////////////////////////////////////////////////////////////////////////////

void IterableCondition() :
{}
{
// TODO : This step was created because of the ambiguite of the rule ClassCondition() inside of RelationCondition() :
// CobolWord() CobolWord() === ArithmeticExpression() ClassCondition()
LOOKAHEAD(ArithmeticExpression())
ArithmeticExpression()
| <LPARENCHAR> Condition() <RPARENCHAR>
}

void Condition() :
{}
{
CombinableCondition() ( ( <AND> | <OR> ) ( LOOKAHEAD(CombinableCondition()) CombinableCondition() | AbbreviationRest() ) )*
}

void CombinableCondition() :
{}
{
[ <NOT> ] SimpleCondition()
}

void SimpleCondition() :
{}
{
LOOKAHEAD(RelationCondition())
RelationCondition()
| LOOKAHEAD(ArithmeticExpression())
ArithmeticExpression()
| <LPARENCHAR> Condition() <RPARENCHAR>
}

void ClassCondition() :
{}
{
[ <IS> ] [ <NOT> ] ( <NUMERIC> | <ALPHABETIC> | <ALPHABETIC_LOWER> | <ALPHABETIC_UPPER> | ClassName() | <DBCS> | <KANJI> )
}

void RelationCondition() :
{}
{
ArithmeticExpression() ( LOOKAHEAD(AbbreviationRest())
AbbreviationRest()
| LOOKAHEAD(SignCondition())
SignCondition()
| ClassCondition()
)
}

void SignCondition() :
{}
{
[ <IS> ] [ <NOT> ] ( <POSITIVE> | <NEGATIVE> | <ZERO> | <ZEROS> | <ZEROES> )
}

void RelationalOperator() :
{}
{
[ <IS> ] [ <NOT> ] ( <GREATER> [ <THAN> ] [ <OR> <EQUAL> [ <TO> ] ]
| <MORETHANCHAR>
| <LESS> [ <THAN> ] [ <OR> <EQUAL> [ <TO> ] ]
| <LESSTHANCHAR>
| <EQUAL> [ <TO> ]
| <EQUALCHAR>
| <NOTEQUAL>
| <MORETHANOREQUAL>
| <LESSTHANOREQUAL>
)
}

void AbbreviationLeaf() :
{}
{
LOOKAHEAD(ArithmeticExpression())
ArithmeticExpression()
// TODO : Is there another abbreviation rule ?
| <LPARENCHAR> AbbreviationLeaf() ( <OR> AbbreviationLeaf() )+ <RPARENCHAR>
}

void AbbreviationRest() :
{}
{
RelationalOperator() AbbreviationLeaf()
}

////////////////////////////////////////////////////////////////////////////////
// VARIOUS TYPES OF IDENTIFIERS
////////////////////////////////////////////////////////////////////////////////

void ProcedureName() :
{}
{
( ParagraphName() [ ( <IN> | <OF> ) SectionName() ]
| SectionName() // Useless here ...
)
}

void Subscript() :
{}
{
<LPARENCHAR> ( ArithmeticExpression() )+ <RPARENCHAR>
}

void FunctionClause() :
{}
{
<FUNCTION> FunctionName() [ LOOKAHEAD(Subscript()) Subscript() ] [ <LPARENCHAR> LeftmostCharacterPosition() <COLONCHAR> [ Length() ] <RPARENCHAR> ]
}

void Identifier() :
{}
{
( QualifiedDataName() ( LOOKAHEAD(Subscript()) Subscript() )* [ <LPARENCHAR> LeftmostCharacterPosition() <COLONCHAR> [ Length() ] <RPARENCHAR> ]
| <LINAGE_COUNTER> [ ( <IN> | <OF> ) FileName() ]
| <PROGRAM_STATUS>
)
}

void QualifiedDataName() :
{}
{
( DataName() ( ( <IN> | <OF> ) DataName() )*
[ ( <IN> | <OF> ) FileName() ] // Useless part here ...
| SpecialRegister() // Useless here ...
)
}

void Length() :
{}
{
ArithmeticExpression()
}

void LeftmostCharacterPosition() :
{}
{
ArithmeticExpression()
}

void Mode() :
{}
{
CobolWord()
}

void AlphabetName() :
{}
{
CobolWord()
}

void ClassName() :
{}
{
CobolWord()
}

void ConditionName() :
{}
{
CobolWord()
}

void DataName() :
{}
{
CobolWord()
}

void FileName() :
{}
{
CobolWord()
}

void IndexName() :
{}
{
CobolWord()
}

void MnemonicName() :
{}
{
CobolWord()
}

void RecordName() :
{}
{
Identifier()
}

void RoutineName() :
{}
{
CobolWord()
}

void SymbolicCharacter() :
{}
{
CobolWord()
}

void LibraryName() :
{}
{
CobolWord()
}

void ProgramName() :
{}
{
CobolWord()
}

void SectionName() :
{}
{
// We force IntegerConstant() recognization here ...
IntegerConstant() | CobolWord()
}

void ParagraphName() :
{}
{
// We force IntegerConstant() recognization here ...
IntegerConstant() | CobolWord()
}

void SystemName() :
{}
{
CobolWord()
}

void ComputerName() :
{}
{
SystemName()
}

void LanguageName() :
{}
{
SystemName()
}

void EnvironmentName() :
{}
{
SystemName()
}

void AssignmentName() :
{}
{
SystemName()
}

void BasisName() :
{}
{
ProgramName()
}

void FunctionName() :
{}
{
( CobolWord()
| <LENGTH> // To avoid conflicts with token having same name ...
| <WHEN_COMPILED>
| <SUM>
| <RANDOM>
)
}

void SpecialRegister() :
{}
{
( <ADDRESS> <OF> DataName()
| <DEBUG_ITEM>
| <LENGTH> <OF> Identifier()
| <RETURN_CODE>
| <SHIFT_OUT>
| <SHIFT_IN>
| <SORT_CONTROL>
| <SORT_CORE_SIZE>
| <SORT_FILE_SIZE>
| <SORT_MESSAGE>
| <SORT_MODE_SIZE>
| <SORT_RETURN>
| <TALLY>
| <WHEN_COMPILED>
)
}

void CdName() :
{}
{
CobolWord()
}

void ConventionName() :
{}
{
CobolWord()
}

void HandleComponent() :
{}
{
CobolWord()
}

void ImplicitTitle() :
{}
{
QuotedText()
}

void LevelName66() :
{}
{
DataName()
}

void LevelName77() :
{}
{
DataName()
}

void LevelName88() :
{}
{
ConditionName()
}

void LevelName() :
{}
{
[ DataName() | <FILLER> | <REPLY> | <PROGRAM_STATUS> ]
}

////////////////////////////////////////////////////////////////////////////////
// ARITHMETIC
////////////////////////////////////////////////////////////////////////////////

void ArithmeticExpression() :
{}
{
TimesDiv() ( ( <PLUSCHAR> | <MINUSCHAR> ) TimesDiv() )*
}

void TimesDiv() :
{}
{
Power() ( ( <ASTERISKCHAR> | <SLASHCHAR> ) Power() )*
}

void Power() :
{}
{
[ ( <PLUSCHAR> | <MINUSCHAR> ) ] Basis() ( <POWEROF> Basis() )*
}

void Basis() :
{}
{
// IMPORTANT : From more complex to most specific :
// TODO : Dirty hack : is <ALL> a valid value ?
( FunctionClause() | Identifier() | Literal() | <ALL> | <LPARENCHAR> ArithmeticExpression() <RPARENCHAR> )
}

void CommentLine() :
{}
{
( ( NonDotChars() | Literal() )+ [ <DOT> ] )+
}

////////////////////////////////////////////////////////////////////////////////
// COMPILATION UNIT.
////////////////////////////////////////////////////////////////////////////////

void CompilationUnit() :
{}
{
ProgramUnit()
( NestedProgramUnit() EndProgramStatement() )*
[ EndProgramStatement() ( CompilationUnit() )* ]
<EOF>
}

void ProgramUnit() :
{}
{
IdentificationDivision()
[ EnvironmentDivision() ]
[ DataDivision() ]
[ ProcedureDivision() ]
}

void NestedProgramUnit() :
{}
{
NestedIdentificationDivision()
[ EnvironmentDivision() ]
[ DataDivision() ]
[ ProcedureDivision() ]
}

void EndProgramStatement() :
{}
{
<END> <PROGRAM> ProgramName() <DOT>
}

void CopyBookUnit() :
{}
{
LinkageSectionEntry()
<EOF>
}

////////////////////////////////////////////////////////////////////////////////
// IDENTIFICATION DIVISION.
////////////////////////////////////////////////////////////////////////////////

void IdentificationDivision() :
{}
{
( <IDENTIFICATION> | <ID> ) <DIVISION> <DOT>
ProgramIdParagraph()
( IdentificationDivisionParagraph() )*
}

void NestedIdentificationDivision() :
{}
{
( <IDENTIFICATION> | <ID> ) <DIVISION> <DOT>
NestedProgramIdParagraph()
( IdentificationDivisionParagraph() )*
}

void IdentificationDivisionParagraph() :
{}
{
AuthorParagraph()
| InstallationParagraph()
| DateWrittenParagraph()
| DateCompiledParagraph()
| SecurityParagraph()
}

void ProgramIdParagraph() :
{}
{
<PROGRAM_ID> <DOT> ProgramName() [ [ <IS> ] <INITIAL> [ <PROGRAM> ] ] <DOT>
}

void NestedProgramIdParagraph() :
{}
{
<PROGRAM_ID> <DOT> ProgramName()
[ [ <IS> ] InitialOrCommon() [ <PROGRAM> ] ] <DOT>
}

void InitialOrCommon() :
{}
{
( <INITIAL> [ <COMMON> ]
| <COMMON> [ <INITIAL> ]
)
}

void AuthorParagraph() :
{ /* System.out.println("Deprecated keyword : author - use a comment"); */ }
{
<AUTHOR> <DOT> [ CommentLine() ]
}

void InstallationParagraph() :
{ /* System.out.println("Deprecated keyword : installation - use a comment"); */ }
{
<INSTALLATION> <DOT> [ CommentLine() ]
}

void DateWrittenParagraph() :
{ /* System.out.println("Deprecated keyword : date-written - use a comment"); */ }
{
<DATE_WRITTEN> <DOT> [ CommentLine() ]
}

void DateCompiledParagraph() :
{ /* System.out.println("Deprecated keyword : date-compiled - use a comment"); */ }
{
<DATE_COMPILED> <DOT> [ CommentLine() ]
}

void SecurityParagraph() :
{ /* System.out.println("Deprecated keyword : security - use a comment"); */ }
{
<SECURITY> <DOT> [ CommentLine() ]
}

////////////////////////////////////////////////////////////////////////////////
// ENVIRONMENT DIVISION.
////////////////////////////////////////////////////////////////////////////////

void EnvironmentDivision() :
{}
{
<ENVIRONMENT> <DIVISION> <DOT>
( EnvironmentSection() )*
}

void EnvironmentSection() :
{}
{
ConfigurationSection()
| InputOutputSection()
}

//------------------------------------------------------------------------------
// CONFIGURATION SECTION
//------------------------------------------------------------------------------

void ConfigurationSection() :
{}
{
<CONFIGURATION> <SECTION> <DOT>
( ConfigurationSectionParagraph() )*
}

void ConfigurationSectionParagraph() :
{}
{
SourceComputerParagraph()
| ObjectComputerParagraph()
| SpecialNamesParagraph()
}

void SourceComputerParagraph() :
{}
{
<SOURCE_COMPUTER> <DOT> ComputerName() [ [ <WITH> ] <DEBUGGING> <MODE> ] <DOT>
}

void ObjectComputerParagraph() :
{}
{
<OBJECT_COMPUTER> <DOT>
ComputerName() ( ObjectComputerClause() )* <DOT>
}

void ObjectComputerClause() :
{}
{
MemorySizeClause()
| CollatingSequenceClause()
| SegmentLimitClause()
| CharacterSetClause()
}

void MemorySizeClause() :
{ /* System.out.println("Deprecated keyword : memory - use a comment"); */ }
{
<MEMORY> [ <SIZE> ] IntegerConstant() [ <WORDS> | <CHARACTERS> | <MODULES> ]
}

void CollatingSequenceClause() :
{ /* System.out.println("Deprecated keyword : collating sequence - use a comment"); */ }
{
[ <PROGRAM> ] [ <COLLATING> ] <SEQUENCE> [ <IS> ] AlphabetName()
}

void SegmentLimitClause() :
{ /* System.out.println("Deprecated keyword : segment unit - use a comment"); */ }
{
( <SEGMENT_LIMIT> | <SEGMENT> <LIMIT> ) [ <IS> ] IntegerConstant()
}

// Tandem ???
void CharacterSetClause() :
{}
{
<CHARACTER> <SET> [ CommentLine() ]
}

void SpecialNamesParagraph() :
{}
{
<SPECIAL_NAMES> <DOT>
[ ( SpecialNameClause() )+ <DOT> ]
}

void SpecialNameClause() :
{}
{
AlphabetClause()
| ClassClause()
| CurrencySignClause()
| DecimalPointClause()
| SymbolicCharactersClause()
| CallConventionClause()
| EnvironmentNameIsMnemonicNameClause()
}

void IterableAlphabetPhrase() :
{}
{
( <THROUGH> | <THRU> ) IterableLiteral()
| ( <ALSO> IterableLiteral() )+
}

void AlphabetClause() :
{}
{
<ALPHABET> AlphabetName() [ <IS> ]
( <STANDARD_1>
| <STANDARD_2>
| <NATIVE>
| CobolWord()
| ( IterableLiteral() [ IterableAlphabetPhrase() ] )+
)
}

void ClassClause() :
{}
{
// TODO (SUPPRESS) : Dirty hack : IterableLiteral() --> Identifier() | IterableLiteral()
<CLASS> ClassName() [ <IS> ] ( ( Identifier() | IterableLiteral() ) [ ( <THROUGH> | <THRU> ) ( Identifier() | IterableLiteral() ) ] )+
}

void CurrencySignClause() :
{}
{
<CURRENCY> [ <SIGN> ] [ <IS> ] Literal()
}

void DecimalPointClause() :
{}
{
<DECIMAL_POINT> [ <IS> ] <COMMA>
}

void SymbolicCharactersClause() :
{}
{
<SYMBOLIC> [ <CHARACTERS> ] ( ( SymbolicCharacter() )+ [ ( <ARE> | <IS> ) ] ( NumericConstant() | <HEXNUMBER> )+ )+ [ <IN> AlphabetName() ]
}

void CallConventionClause() :
{}
{
<CALL_CONVENTION> IntegerConstant() [ <IS> ] ConventionName()
}

void EnvironmentNameIsMnemonicNameClause() :
{}
{
EnvironmentName()
( [ <IS> ] MnemonicName() [ SpecialNamesParagraphStatusPhrase() ]
| SpecialNamesParagraphStatusPhrase()
)
}

void SpecialNamesParagraphStatusPhrase() :
{}
{
( <ON> [ <STATUS> ] [ <IS> ] IterableCondition() [ <OFF> [ <STATUS> ] [ <IS> ] IterableCondition() ]
| <OFF> [ <STATUS> ] [ <IS> ] IterableCondition() [ <ON> [ <STATUS> ] [ <IS> ] IterableCondition() ]
)
}

//------------------------------------------------------------------------------
// INPUT-OUTPUT SECTION
//------------------------------------------------------------------------------

void InputOutputSection() :
{}
{
<INPUT_OUTPUT> <SECTION> <DOT>
( InputOutputSectionParagraph() )*
}

void InputOutputSectionParagraph() :
{}
{
FileControlParagraph()
| IOControlParagraph()
}

void FileControlParagraph() :
{}
{
<FILE_CONTROL> ( LOOKAHEAD([ <DOT> ] FileControlEntry()) [ <DOT> ] FileControlEntry() )* <DOT>
}

void FileControlEntry() :
{}
{
SelectClause()
( FileControlClause() )*
}

void FileControlClause() :
{}
{
AssignClause()
| ReserveClause()
| OrganizationClause()
| PaddingCharacterClause()
| RecordDelimiterClause()
| AccessModeClause()
| RecordKeyClause()
| AlternateRecordKeyClause()
| FileStatusClause()
| PasswordClause()
}

void SelectClause() :
{}
{
<SELECT> [ <OPTIONAL> ] FileName()
}

void AssignClause() :
{}
{
<ASSIGN> [ <TO> ] ( AssignmentName() | Literal() ) [ <USING> ( AssignmentName() | Literal() ) ]
}

void ReserveClause() :
{}
{
<RESERVE> IntegerConstant() [ <AREA> | <AREAS> ]
}

void OrganizationClause() :
{}
{
[ <ORGANIZATION> [ <IS> ] ]
( SequentialOrganizationClause()
| LineSequentialOrganizationClause()
| RelativeOrganizationClause()
| IndexedOrganizationClause()
)
}

void SequentialOrganizationClause() :
{}
{
<SEQUENTIAL>
}

void LineSequentialOrganizationClause() :
{}
{
<LINE> <SEQUENTIAL>
}

void RelativeOrganizationClause() :
{}
{
<RELATIVE> [ Identifier() ]
}

void IndexedOrganizationClause() :
{}
{
<INDEXED>
}

void PaddingCharacterClause() :
{}
{
<PADDING> [ <CHARACTER> ] [ <IS> ] ( Identifier() | Literal() )
}

void RecordDelimiterClause() :
{}
{
<RECORD> <DELIMITER> [ <IS> ] ( <STANDARD_1> | <IMPLICIT> | AssignmentName() )
}

void AccessModeClause() :
{}
{
<ACCESS> [ <MODE> ] [ <IS> ]
( SequentialAccessMode()
| RandomAccessMode()
| DynamicAccessMode()
)
}

void SequentialAccessMode() :
{}
{
<SEQUENTIAL> [ RelativeKeyClause() ]
}

void RandomAccessMode() :
{}
{
<RANDOM> [ RelativeKeyClause() ]
}

void DynamicAccessMode() :
{}
{
<DYNAMIC> [ RelativeKeyClause() ]
}

void RelativeKeyClause() :
{}
{
<RELATIVE> [ <KEY> ] [ <IS> ] Identifier()
}

void RecordKeyClause() :
{}
{
<RECORD> [ <KEY> ] [ <IS> ] Identifier() [ PasswordClause() ] [ [ <WITH> ] <DUPLICATES> ]
}

void AlternateRecordKeyClause() :
{}
{
<ALTERNATE> <RECORD> [ <KEY> ] [ <IS> ] Identifier() [ PasswordClause() ] [ [ <WITH> ] <DUPLICATES> ]
}

void PasswordClause() :
{}
{
<PASSWORD> [ <IS> ] DataName()
}

void FileStatusClause() :
{}
{
[ <FILE> ] <STATUS> [ <IS> ] Identifier() [ Identifier() ]
}

void IOControlParagraph() :
{}
{
<I_O_CONTROL>
( LOOKAHEAD([ <DOT> ] IOControlClause())
[ <DOT> ] IOControlClause()
)*
<DOT>
}

void IOControlClause() :
{}
{
RerunClause()
| SameAreaClause()
| MultipleFileClause()
}

void RerunClause() :
{ /* System.out.println("Deprecated keyword : rerun"); */ }
{
<RERUN> [ <ON> ( AssignmentName() | FileName() ) ] <EVERY> ( LOOKAHEAD(Rerun2()) Rerun2() | IntegerConstant() [ <CLOCK_UNITS> ] )
}

void Rerun2() :
{}
{
IntegerConstant() <RECORDS>
| [ <END> ] [ <OF> ] ( <REEL> | <UNIT> ) <OF> FileName()
}

void SameAreaClause() :
{}
{
// TODO : Dirty hack : Are <SAME> <AREA> ... or <SAME> ... valid values ?
<SAME> [ <RECORD> | <SORT> | <SORT_MERGE> ] [ <AREA> ] [ <FOR> ] ( FileName() )+
//| <SAME> <AREA> [ <FOR> ] ( FileName() )+
}

void MultipleFileClause() :
{ /* System.out.println("Deprecated keyword : multiple file"); */ }
{
<MULTIPLE> <FILE> [ <TAPE> ] [ <CONTAINS> ] ( FileName() [ [ <POSITION> ] IntegerConstant() ] )+
}

////////////////////////////////////////////////////////////////////////////////
// DATA DIVISION.
////////////////////////////////////////////////////////////////////////////////

void DataDivision() :
{}
{
<DATA> <DIVISION> <DOT>
( DataDivisionSection() )*
}

void DataDivisionSection() :
{}
{
FileSection()
| WorkingStorageSection()
| LinkageSection()
| CommunicationSection()
}

void FileSection() :
{}
{
<FILE> <SECTION> <DOT>
( FileAndSortDescriptionEntry() ( DataDescriptionEntry() )* )*
}

void FileAndSortDescriptionEntry() :
{}
{
( <FD> | <SD> ) FileName()
(
LOOKAHEAD([ <DOT> ] FileAndSortDescriptionEntryClause())
[ <DOT> ] FileAndSortDescriptionEntryClause()
)* <DOT>
}

void FileAndSortDescriptionEntryClause() :
{}
{
ExternalClause()
| GlobalClause()
| BlockContainsClause()
| RecordContainsClause()
| LabelRecordsClause()
| ValueOfClause()
| DataRecordClause()
| LinageClause()
| CodeSetClause()
| ReportClause()
| RecordingModeClause()
}

void ExternalClause() :
{}
{
[ <IS> ] <EXTERNAL>
}

void GlobalClause() :
{}
{
[ <IS> ] <GLOBAL>
}

void BlockContainsClause() :
{}
{
<BLOCK> [ <CONTAINS> ] IntegerConstant() [ <TO> IntegerConstant() ] [ <RECORDS> | <CHARACTERS> ]
}

void RecordContainsClause() :
{}
{
<RECORD> [ <CONTAINS> ]
( IntegerConstant() [ <TO> IntegerConstant() ] [ <CHARACTERS> ]
| [ <IS> ] <VARYING> [ <IN> ] [ <SIZE> ]
[ [ <FROM> ] IntegerConstant() [ <TO> IntegerConstant() ] [ <CHARACTERS> ] ]
[ <DEPENDING> [ <ON> ] Identifier() ]
)
}

void LabelRecordsClause() :
{}
{
<LABEL> ( <RECORD> [ <IS> ]
| <RECORDS> [ <ARE> ]
) ( <OMITTED> | <STANDARD> | ( DataName() )+ )
}

void ValueOfClause() :
{ /* System.out.println("Deprecated clause : value of"); */ }
{
<VALUE> <OF> ( SystemName() <IS> ( Identifier() | Literal() ) )+
}

void DataRecordClause() :
{ /* System.out.println("Deprecated clause : data record"); */ }
{
<DATA> ( <RECORD> [ <IS> ]
| <RECORDS> [ <ARE> ]
) ( DataName() )+
}

void LinageClause() :
{}
{
<LINAGE> [ <IS> ] ( DataName() | IntegerConstant() ) [ <LINES> ]
( [ <WITH> ] <FOOTING> [ <AT> ] ( DataName() | IntegerConstant() )
| [ <LINES> ] [ <AT> ] ( <TOP> | <BOTTOM> ) ( DataName() | IntegerConstant() )
)*
}

void RecordingModeClause() :
{}
{
<RECORDING> [ <MODE> ] [ <IS> ] Mode()
}

void CodeSetClause() :
{}
{
<CODE_SET> [ <IS> ] AlphabetName()
}

void ReportClause() :
{}
{
( <REPORT> [ <IS> ] | <REPORTS> [ <ARE> ] ) ( Identifier() )+
}

void DataDescriptionEntry() :
{}
{
( <LEVEL_66> LevelName66() RenamesClause() <DOT>
| <LEVEL_77> LevelName77() ( LOOKAHEAD(DataDescriptionEntryClause()) DataDescriptionEntryClause() )* <DOT>
| <LEVEL_88> LevelName88() ConditionValueClause() <DOT>
| LevelNumber() LevelName() ( LOOKAHEAD(DataDescriptionEntryClause()) DataDescriptionEntryClause() )* <DOT>
)
}

void DataDescriptionEntryClause() :
{}
{
DataPictureClause()
| DataValueClause()
| DataUsageClause()
| DataRedefinesClause()
| DataExternalClause()
| DataGlobalClause()
| DataIdentifiedClause()
| DataSignClause()
| DataOccursClause()
| DataSynchronizedClause()
| DataJustifiedClause()
| DataBlankWhenZeroClause()
}

void DataRedefinesClause() :
{}
{
<REDEFINES> DataName() [ DataPictureClause() ]
}

void DataBlankWhenZeroClause() :
{}
{
<BLANK> [ <WHEN> ] ( <ZERO> | <ZEROS> | <ZEROES> )
}

void DataJustifiedClause() :
{}
{
( <JUSTIFIED> | <JUST> ) [ <RIGHT> ]
}

void DataOccursClause() :
{}
{
<OCCURS> IntegerConstant() [ <TO> IntegerConstant() ] [ <TIMES> ]
[ <DEPENDING> [ <ON> ] Identifier() ]
( ( <ASCENDING> | <DESCENDING> ) [ <KEY> ] [ <IS> ] ( Identifier() )+ )*
[ <INDEXED> [ <BY> ] ( IndexName() )+ ]
}

void DataPictureClause() :
{}
{
( <PICTURE> | <PIC> ) [ <IS> ] PictureString()
}

void PictureString() :
{}
{
PictureOccurence() ( LOOKAHEAD(<DOTCHAR> PictureOccurence()) <DOTCHAR> PictureOccurence() | PictureOccurence() )*
// There can be additional dots at end of picture strings ...
( <DOTCHAR> )*
}

void PictureOccurence() :
{}
{
( NonDotChars() )+ [ <LPARENCHAR> IntegerConstant() <RPARENCHAR> ]
| <DOTCHAR> ( <LPARENCHAR> IntegerConstant() <RPARENCHAR> | NonDotChars() )
}

void PicturePunctuation() :
{}
{
<SLASHCHAR> | <COMMACHAR> | <COLONCHAR> | <ASTERISKCHAR> | <MINUSCHAR> | <PLUSCHAR> |
<POWEROF> | <LESSTHANOREQUAL> | <LESSTHANCHAR> | <MORETHANOREQUAL> | <MORETHANCHAR> |
<EQUALCHAR> | <NOTEQUAL>
}

void PictureCurrency() :
{}
{
<DOLLARCHAR> // to be completed
}

void NonDotChars() :
{}
{
IntegerConstant() | CobolWord() | PicturePunctuation() | PictureCurrency()
}

void DataExternalClause() :
{}
{
[ <IS> ] <EXTERNAL> [ <BY> Literal() ]
}

void DataGlobalClause() :
{}
{
[ <IS> ] <GLOBAL>
}

void DataIdentifiedClause() :
{}
{
[ <IS> ] <IDENTIFIED> <BY> Literal()
}

void DataUsageClause() :
{}
{
[ <USAGE> [ <IS> ] ] ( MemoryUsage() | ExternalUsage() | PropertyObjectModifier() )
}

void MemoryUsage() :
{}
{
<DISPLAY> | <DISPLAY_1>
| <COMP> | <COMPUTATIONAL>
| <COMP_N> | <COMPUTATIONAL_N>
| <COMP_1> | <COMPUTATIONAL_1>
| <COMP_1_REV> | <COMPUTATIONAL_1_REV> | <COMP_1_M> | <COMPUTATIONAL_1_M>
| <COMP_1_MVS> | <COMPUTATIONAL_1_MVS> | <COMP_1_E> | <COMPUTATIONAL_1_E>
| <COMP_2> | <COMPUTATIONAL_2>
| <COMP_2_REV> | <COMPUTATIONAL_2_REV> | <COMP_2_M> | <COMPUTATIONAL_2_M>
| <COMP_2_MVS> | <COMPUTATIONAL_2_MVS> | <COMP_2_E> | <COMPUTATIONAL_2_E>
| <COMP_3> | <COMPUTATIONAL_3>
| <COMP_4> | <COMPUTATIONAL_4>
| <COMP_5> | <COMPUTATIONAL_5>
| <COMP_6> | <COMPUTATIONAL_6>
| <COMP_S> | <COMPUTATIONAL_S> | <COMP_1_A> | <COMPUTATIONAL_1_A>
| <COMP_D> | <COMPUTATIONAL_D> | <COMP_2_A> | <COMPUTATIONAL_2_A>
| <PACKED_DECIMAL> | <COMP_P> | <COMPUTATIONAL_P>
| <PACKED_DECIMAL_A> | <COMP_3_A> | <COMPUTATIONAL_3_A>
| <PACKED_DECIMAL_I> | <PACKED_DECIMAL_E> | <PACKED_DECIMAL_H> | <COMP_3_M> | <COMPUTATIONAL_3_M> | <PACKED_DECIMAL_M>
| <BINARY> | <BINARY_M> | <COMP_M> | <COMPUTATIONAL_M> | <COMP_A> | <COMPUTATIONAL_A> | <COMP_4_M> | <COMPUTATIONAL_4_M> | <COMP_B> | <COMPUTATIONAL_B>
| <BINARY_REV>
| <SQLIND>
| <COMP_X> | <COMP_X> | <COMPUTATIONAL_X> | <COMPUTATIONAL_X> | <COMP_5_M> | <COMPUTATIONAL_5_M>
| <COMP_X_REV> | <COMPUTATIONAL_X_REV>
| <DISPLAY_WS>
| <EXTERNAL_FORM>
| <NATIONAL>
| <KANJI>
| <SIGNED_SHORT>
| <UNSIGNED_SHORT>
| <SIGNED_INT>
| <UNSIGNED_INT>
| <SIGNED_LONG>
| <UNSIGNED_LONG>
| <FLOAT>
| <DOUBLE>
| <HANDLE> <OF> <THREAD> // <OF> is not optional here
| <HANDLE> [ [ <OF> ] HandleComponent() [ ImplicitTitle() ] ]
| <HANDLE> [ <OF> ] <FONT> [ <FIXED_FONT> | <TRADITIONAL_FONT> | <DEFAULT_FONT> | <SMALL_FONT> | <MEDIUM_FONT> | <LARGE_FONT> ]
| <HANDLE> [ <OF> ] <WINDOW> [ ImplicitTitle() ]
| <WINDOW>
}

void ExternalUsage() :
{}
{
<INDEX>
| <POINTER>
| <PROCEDURE_POINTER> | <PROCEDURE> <POINTER>
| <STRING>
| <JPACKED_DECIMAL>
| <JSTRING>
| <JBYTE>
| <JSHORT>
| <JINT>
| <JLONG>
| <JBOOLEAN>
| <JFLOAT>
| <JDOUBLE>
| <JCHAR>
| <ALPHANUMERIC_HASHTABLE>
| <NUMERIC_HASHTABLE>
| <OBJECT_HASHTABLE>
| <OBJECT> <REFERENCE> [ [ <OF> ] ( ClassName() | Literal() ) ]
| ( <OBJECT> | <COMPONENT> ) [ [ <OF> ] ClassName() ]
}

void PropertyObjectModifier() :
{}
{
<CHARACTER>
| <BINARY_BYTE>
| <BINARY_SHORT>
| <BINARY>
| <BINARY_LONG>
| <BINARY_DOUBLE>
| <FLOAT_SHORT>
| <FLOAT_LONG>
| <BIT>
}

void DataSignClause() :
{}
{
[ <SIGN> [ <IS> ] ] ( <LEADING> | <TRAILING> ) [ <SEPARATE> [ <CHARACTER> ] ]
}

void DataSynchronizedClause() :
{}
{
( <SYNCHRONIZED> | <SYNC> ) [ ( <LEFT> | <RIGHT> ) ]
}

void DataValueClause() :
{}
{
// TODO (SUPPRESS) : Dirty hack : Literal() --> PictureString() | IterableLiteral()
( <VALUE> [ <IS> ] | <VALUES> [ <ARE> ] ) ( ( LOOKAHEAD(PictureString()) PictureString() | IterableLiteral() ) [ ( <THROUGH> | <THRU> ) IterableLiteral() ] )+
}

void ConditionValueClause() :
{}
{
DataValueClause()
}

void RenamesClause() :
{}
{
<RENAMES> Identifier() [ ( <THROUGH> | <THRU> ) Identifier() ]
}

////////////////////////////////////////////////////////////////////////////////
// DATA DIVISION : COMMUNICATION SECTION.
////////////////////////////////////////////////////////////////////////////////

void CommunicationDescriptionEntry() :
{}
{
(
LOOKAHEAD(CommunicationInputEntry())
CommunicationInputEntry()
| LOOKAHEAD(CommunicationOutputEntry())
CommunicationOutputEntry()
| LOOKAHEAD(CommunicationIOEntry())
CommunicationIOEntry()
) <DOT>
}

void CommunicationInputEntry() :
{}
{
<CD> CdName() [ <FOR> ] [ <INITIAL> ] <INPUT> ( CommunicationInputClause() )* ( DataName() )*
}

void CommunicationOutputEntry() :
{}
{
<CD> CdName() [ <FOR> ] <OUTPUT> ( CommunicationOutputClause() )*
}

void CommunicationIOEntry() :
{}
{
<CD> CdName() [ <FOR> ] [ <INITIAL> ] <I_O> ( CommunicationIOClause() )* ( DataName() )*
}

void CommunicationInputClause() :
{}
{
<MESSAGE> ( <DATE> | <TIME> | <COUNT> ) [ <IS> ] DataName()
| <TEXT> <LENGTH> [ <IS> ] DataName()
| <END> <KEY> [ <IS> ] DataName()
| <STATUS> <KEY> [ <IS> ] DataName()
| <COUNT> [ <IS> ] DataName()
| [ <SYMBOLIC> ] ( <QUEUE> | <SUB_QUEUE_1> | <SUB_QUEUE_2> | <SUB_QUEUE_3> | <SOURCE> ) [ <IS> ] DataName()
}

void CommunicationOutputClause() :
{}
{
<DESTINATION> <COUNT> [ <IS> ] DataName()
| <TEXT> <LENGTH> [ <IS> ] DataName()
| <STATUS> <KEY> [ <IS> ] DataName()
| LOOKAHEAD([ <SYMBOLIC> ] <DESTINATION> [ <IS> ] DataName())
[ <SYMBOLIC> ] <DESTINATION> [ <IS> ] DataName()
| <DESTINATION> <TABLE> <OCCURS> IntegerConstant() [ <TIMES> ] [ <INDEXED> [ <BY> ] ( IndexName() )+ ]
| <ERROR> <KEY> [ <IS> ] DataName()
}

void CommunicationIOClause() :
{}
{
<MESSAGE> ( <DATE> | <TIME> ) [ <IS> ] DataName()
| <TEXT> <LENGTH> [ <IS> ] DataName()
| <END> <KEY> [ <IS> ] DataName()
| <STATUS> <KEY> [ <IS> ] DataName()
| [ <SYMBOLIC> ] <TERMINAL> [ <IS> ] DataName()
}

//------------------------------------------------------------------------------
// WORKING STORAGE SECTION, LINKAGE SECTION.
//------------------------------------------------------------------------------

void WorkingStorageSection() :
{}
{
<WORKING_STORAGE> <SECTION> <DOT>
( DataDescriptionEntry() )*
}

void LinkageSectionEntry() :
{}
{
( DataDescriptionEntry() )*
}

void LinkageSection() :
{}
{
<LINKAGE> <SECTION> <DOT>
LinkageSectionEntry()
}

void CommunicationSection() :
{}
{
<COMMUNICATION> <SECTION> <DOT>
( CommunicationDescriptionEntry() ( DataDescriptionEntry() )* )*
}

////////////////////////////////////////////////////////////////////////////////
// PROCEDURE DIVISION also known as "spaghetti division".
////////////////////////////////////////////////////////////////////////////////

void ProcedureDivision() :
{}
{
<PROCEDURE> <DIVISION> [ ConventionName() ] [ <USING> ( DataName() )+ ] <DOT>
[ Declaratives() ]
ProcedureBody()
}

void Declaratives() :
{}
{
<DECLARATIVES> <DOT>
( SectionHeader() <DOT>
UseStatement() <DOT>
Paragraphs()
)+
<END> <DECLARATIVES> <DOT>
}

void ProcedureBody() :
{}
{
Paragraphs() ( ProcedureSection() )*
}

void ProcedureSection() :
{}
{
SectionHeader() <DOT> Paragraphs()
}

void SectionHeader() :
{}
{
SectionName() <SECTION> [ IntegerConstant() ]
}

void Paragraphs() :
{}
{
( Sentence() )* ( Paragraph() )*
}

void Paragraph() :
{}
{
ParagraphName() <DOT>
( ExitStatement()
| AlteredGoto()
| ( Sentence() )*
)
}

void Sentence() :
{}
{
StatementList() <DOT>
}

void StatementList() :
{}
{
( Statement() )+
}

void Statement() :
{}
{
( AcceptStatement()
| AddStatement()
| AlterStatement()
| CallStatement()
| CancelStatement()
| CloseStatement()
| ComputeStatement()
| ContinueStatement()
| DeleteStatement()
| DisplayStatement()
| DivideStatement()
| EntryStatement()
| EvaluateStatement()
/* | ExitStatement() */
| ExitProgramStatement()
| GobackStatement()
| GotoStatement()
| IfStatement()
| InitializeStatement()
| InspectStatement()
| MergeStatement()
| MoveStatement()
| MultiplyStatement()
| OpenStatement()
| PerformStatement()
| ReadStatement()
| ReleaseStatement()
| ReturnStatement()
| RewriteStatement()
| SearchStatement()
| SetStatement()
| SortStatement()
| StartStatement()
| StopStatement()
| StringStatement()
| SubtractStatement()
| UnstringStatement()
| WriteStatement() )
}

void AcceptStatement() :
{}
{
<ACCEPT> ( <REPLY> | <OMITTED> | Identifier() )
[ <FROM>
( MnemonicName()
| EnvironmentName()
| <DATE>
| <DAY>
| <DAY_OF_WEEK>
| <TIME>
)
]
}

void AddStatement() :
{}
{
<ADD>
( ( <CORRESPONDING> | <CORR> ) Identifier() <TO> Identifier() [ <ROUNDED> ]
| ( Identifier() | Literal() )+ [ <TO> ( Identifier() [ <ROUNDED> ] )+ ]
[ <GIVING> ( Identifier() [ <ROUNDED> ] )+ ]
)
[ [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <NOT> [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <END_ADD> ]
}

void AlteredGoto() :
{ /* System.out.println("***ALTER ALERT***"); */ }
{
<GO> [ <TO> ] <DOT>
}

/**
* ALTER is one of the most dangerous statement ever invented by man.
* ALTER is to GOTO what an atomic bomb is to firecrackers.
*/

void AlterStatement() :
{ /* System.out.println("***ALTER ALERT***"); */ }
{
<ALTER> ( ProcedureName() <TO> [ <PROCEED> <TO> ] ProcedureName() )+
}

void CallByContent() :
{}
{
[ <BY> ] <CONTENT> ( <ADDRESS> <OF> Identifier() | [ <LENGTH> <OF> ] Identifier() | Literal() | <OMITTED> )+
}

void CallByReference() :
{}
{
[ [ <BY> ] <REFERENCE> ] ( <ADDRESS> <OF> Identifier() | Identifier() | FileName() | <OMITTED> )+
}

void CallByDescriptor() :
{}
{
[ <BY> ] <DESCRIPTOR> ( <ADDRESS> <OF> Identifier() | Identifier() | FileName() | <OMITTED> )+
}

void CallByValue() :
{}
{
[ <BY> ] <VALUE> ( <ADDRESS> <OF> Identifier() | [ <LENGTH> <OF> ] Identifier() | Literal() | <OMITTED> )+
}

void CallStatement() :
{}
{
<CALL>
(
LOOKAHEAD(ConventionName() ( Identifier() | Literal() ))
ConventionName() ( Identifier() | Literal() )
| ( Identifier() | Literal() )
)
[ [ <WITH> ] <STDCALL> [ <LINKAGE> ] ] // Fijutsu Cobol
[ <USING>
(
LOOKAHEAD(CallByContent())
CallByContent()
| LOOKAHEAD(CallByDescriptor())
CallByDescriptor()
| LOOKAHEAD(CallByValue())
CallByValue()
| LOOKAHEAD(CallByReference())
CallByReference()
)+
]
[ ( <GIVING> | <RETURNING> ) Identifier() ]
[ LOOKAHEAD([ <ON> ] <EXCEPTION>) [ <ON> ] <EXCEPTION> StatementList() ]
[ LOOKAHEAD(<NOT> [ <ON> ] <EXCEPTION>) <NOT> [ <ON> ] <EXCEPTION> StatementList() ]
[ [ <ON> ] <OVERFLOW> StatementList() ]
[ <NOT> [ <ON> ] <OVERFLOW> StatementList() ]
[ <END_CALL> ]
}

void CancelStatement() :
{}
{
<CANCEL> ( Identifier() | Literal() )+
}

void CloseStatement() :
{}
{
<CLOSE>
( FileName()
[ ( ( <REEL> | <UNIT> ) [ ( [ <FOR> ] <REMOVAL>
| [ <WITH> ] <NO> <REWIND>
)
]
| [ <WITH> ] ( <NO> <REWIND> | <LOCK> )
)
]
)+
}

void ComputeStatement() :
{}
{
<COMPUTE> ( Identifier() [ <ROUNDED> ] )+
( <EQUALCHAR> | <EQUAL> )
ArithmeticExpression()
[ [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <NOT> [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <END_COMPUTE> ]
}

void ContinueStatement() :
{}
{
<CONTINUE>
}

void DeleteStatement() :
{}
{
<DELETE> FileName() [ <RECORD> ]
[ <INVALID> [ <KEY> ] StatementList() ]
[ <NOT> <INVALID> [ <KEY> ] StatementList() ]
[ <END_DELETE> ]
}

void DisplayStatement() :
{}
{
<DISPLAY> ( Identifier() | Literal() )+
[ <UPON> ( MnemonicName() | EnvironmentName() ) ]
[ [ <WITH> ] <NO> <ADVANCING> ]
}

void DivideStatement() :
{}
{
<DIVIDE> ( Identifier() | Literal() )
( <INTO> ( Identifier() | Literal() ) [ <ROUNDED> ] ( <GIVING> ( Identifier() [ <ROUNDED> ] )+ | ( ( Identifier() | Literal() ) [ <ROUNDED> ] )* )
| <BY> ( Identifier() | Literal() ) [ <GIVING> ( Identifier() [ <ROUNDED> ] )+ ]
)
[ <REMAINDER> Identifier() ]
[ [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <NOT> [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <END_DIVIDE> ]
}

void EntryStatement() :
{}
{
<ENTRY> Literal() [ <USING> ( Identifier() )+ ]
}

void EvaluateStatement() :
{}
{
<EVALUATE> EvaluateValue()
( <ALSO> EvaluateValue() )*
( ( EvaluateWhenClause() )+ [ <THEN> ] StatementList() )+
[ <END_EVALUATE> ]

}

void EvaluateValue() :
{}
{
( Condition()
| <TRUE> // Useless here ...
| <FALSE> // Useless here ...
)
}

void EvaluateWhenClause() :
{}
{
( LOOKAHEAD(<WHEN> <OTHER>)
<WHEN> <OTHER>
| <WHEN> EvaluatePhrase() ( <ALSO> EvaluatePhrase() )*
)
}

void EvaluateThruPhrase() :
{}
{
[ <NOT> ] ArithmeticExpression() ( <THROUGH> | <THRU> ) ArithmeticExpression()
}

void EvaluatePhrase() :
{}
{
( LOOKAHEAD(EvaluateThruPhrase())
EvaluateThruPhrase()
| Condition()
| <ANY>
| <TRUE> // Useless here ...
| <FALSE> // Useless here ...
)
}

void ExitStatement() :
{}
{
<EXIT> <DOT>
}

void ExitProgramStatement() :
{}
{
<EXIT> <PROGRAM>
}

void GobackStatement() :
{}
{
<GOBACK>
}

void GotoStatement() :
{}
{
<GO> [ <TO> ]
( ( ProcedureName() )+ [ <DEPENDING> [ <ON> ] Identifier() ]
| <MORE_LABELS> // ??? IBM extension ???
)
}

void IfStatement() :
{}
{
<IF> Condition() [ <THEN> ] ( ( Statement() )+ | <NEXT> <SENTENCE> )
[ <ELSE> ( ( Statement() )+ | <NEXT> <SENTENCE> ) ]
[ <END_IF> ]

}

void InitializeStatement() :
{}
{
<INITIALIZE> ( Identifier() )+
[ <REPLACING> ( ( <ALPHABETIC>
| <ALPHANUMERIC>
| <NUMERIC>
| <ALPHANUMERIC_EDITED>
| <NUMERIC_EDITED>
| <DBCS>
| <EGCS>
) [ <DATA> ] <BY> ( Identifier() | Literal() )
)+
]

}

void InspectStatement() :
{}
{
<INSPECT> Identifier()
( TallyingPhrase()
| ConvertingPhrase()
| ReplacingPhrase()
)
}

void IterableTallyingClause() :
{}
{
<CHARACTERS> ( IterableBeforeAfterPhrase() )*
}

void RecursiveTallyingClause() :
{}
{
( <ALL> | <LEADING> ) IterableBeforeAfterClause()
(
LOOKAHEAD(Identifier() <FOR>)
( Identifier() <FOR> ( RecursiveTallyingClause() | IterableTallyingClause() )+ )+
| IterableBeforeAfterClause()
)*
}

void TallyingPhrase() :
{}
{
<TALLYING>
( Identifier() <FOR> ( RecursiveTallyingClause() | IterableTallyingClause() )+ )+
[ ReplacingPhrase() ]
}

void ConvertingPhrase() :
{}
{
<CONVERTING> ( Identifier() | Literal() )
<TO> IterableBeforeAfterClause()
}

void ReplacingPhrase() :
{}
{
<REPLACING>
( <CHARACTERS> <BY> IterableBeforeAfterClause()
| ( <ALL> | <LEADING> | <FIRST> ) ( LOOKAHEAD(( Identifier() | Literal() ) <BY> IterableBeforeAfterClause())
( Identifier() | Literal() ) <BY> IterableBeforeAfterClause()
)+
)+
}

void IterableBeforeAfterClause() :
{}
{
( Identifier() | IterableLiteral() ) ( IterableBeforeAfterPhrase() )*
}

void IterableBeforeAfterPhrase() :
{}
{
( <BEFORE> | <AFTER> ) [ <INITIAL> ] ( Identifier() | IterableLiteral() )
}

void MergeStatement() :
{}
{
<MERGE> FileName()
( [ <ON> ] ( <ASCENDING> | <DESCENDING> ) [ <KEY> ] ( Identifier() )+ )+
[ [ <COLLATING> ] <SEQUENCE> [ <IS> ] AlphabetName() ] <USING> FileName() ( FileName() )+
( <OUTPUT> <PROCEDURE> [ <IS> ] ProcedureName() [ ( <THROUGH> | <THRU> ) ProcedureName() ]
| <GIVING> ( FileName() )+
)
}

/**
* MOVE is probably the first polymorphic statement in the history of computing.
*/

void MoveStatement() :
{}
{
// <MOVE> FunctionClause() : NON-CONFORMING STANDARD
<MOVE>
( ArithmeticExpression() <TO> ( Identifier() )+
| ( <CORRESPONDING> | <CORR> ) Identifier() <TO> ( Identifier() )+
)
}

void MultiplyStatement() :
{}
{
<MULTIPLY> ( Identifier() | Literal() ) <BY> ( ( Identifier() | Literal() ) [ <ROUNDED> ] )+
[ <GIVING> ( Identifier() [ <ROUNDED> ] )+ ]
[ [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <NOT> [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <END_MULTIPLY> ]
}

void OpenStatement() :
{}
{
<OPEN> ( <INPUT> ( FileName() [ ( <REVERSED> | [ <WITH> ] <NO> <REWIND> ) ] )+
| <OUTPUT> ( FileName() [ [ <WITH> ] <NO> <REWIND> ] )+
| <I_O> ( FileName() )+
| <EXTEND> ( FileName() )+
)+
}

/**
* The "one-size-fits-all" statement of COBOL: it replaces for, while, call...
*/

void PerformStatement() :
{}
{
// TODO : Rule "PERFORM <Identifier Range> ***<With Test>*** <Numeric> TIMES" doesn't exist.
<PERFORM>
( LOOKAHEAD(PerformFlavour() StatementList() <END_PERFORM>)
PerformFlavour() StatementList() <END_PERFORM>
| LOOKAHEAD(PerformProcedureScopeClause() PerformFlavour())
PerformProcedureScopeClause() PerformFlavour()
| LOOKAHEAD(StatementList() <END_PERFORM>)
StatementList() <END_PERFORM>
| PerformProcedureScopeClause()
)
}

void PerformFlavour() :
{}
{
PerformTimeClause()
| [ PerformTestPositionClause() ] ( PerformUntilClause() | PerformVaryingWithAfterClause() )
}

void PerformVaryingWithAfterClause() :
{}
{
<VARYING> PerformVaryingPhrase() ( <AFTER> PerformVaryingPhrase() )*
}

void PerformVaryingPhrase() :
{}
{
Identifier()
<FROM> ( Identifier() | Literal() )
<BY> ( Identifier() | Literal() )
PerformUntilClause()
}

void PerformUntilClause() :
{}
{
<UNTIL> Condition()
}

void PerformTestPositionClause() :
{}
{
[ <WITH> ] <TEST> ( <BEFORE> | <AFTER> )
}

void PerformProcedureScopeClause() :
{}
{
ProcedureName() [ ( <THROUGH> | <THRU> ) ProcedureName() ]
}

void PerformTimeClause() :
{}
{
( Identifier() | IntegerConstant() ) <TIMES>
}

void ReadStatement() :
{}
{
<READ> FileName() [ <NEXT> ] [ <RECORD> ]
[ <INTO> Identifier() ]
[ <KEY> [ <IS> ] Identifier() ]
[ <INVALID> [ <KEY> ] StatementList() ]
[ <NOT> <INVALID> [ <KEY> ] StatementList() ]
[ [ <AT> ] <END> StatementList() ]
[ <NOT> [ <AT> ] <END> StatementList() ]
[ <END_READ> ]
}

void ReleaseStatement() :
{}
{
<RELEASE> RecordName() [ <FROM> Identifier() ]
}

void ReturnStatement() :
{}
{
<RETURN> FileName() [ <RECORD> ] [ <INTO> Identifier() ]
[ <AT> ] <END> StatementList()
[ <NOT> [ <AT> ] <END> StatementList() ]
[ <END_RETURN> ]
}

void RewriteStatement() :
{}
{
<REWRITE> RecordName() [ <FROM> Identifier() ]
[ <INVALID> [ <KEY> ] StatementList() ]
[ <NOT> <INVALID> [ <KEY> ] StatementList() ]
[ <END_REWRITE> ]
}

void SearchStatement() :
{}
{
<SEARCH> [ <ALL> ] Identifier()
[ <VARYING> Identifier() ]
[ [ <AT> ] <END> StatementList() ]
( <WHEN> Condition() ( StatementList() | <NEXT> <SENTENCE> ) )+
[ <END_SEARCH> ]
}

void SetStatement() :
{}
{
<SET> [ <CONFIGURATION> ] ( Identifier() | Literal() )+
( <TO> [ <CONFIGURATION> ] ( Identifier() | <TRUE> | <FALSE> | <ON> | <OFF> | [ <ENTRY> ] Literal() )
| ( <UP> | <DOWN> ) [ <BY> ] ( Identifier() | Literal() )
)
}

void SortStatement() :
{}
{
<SORT> FileName()
( [ <ON> ] ( <ASCENDING> | <DESCENDING> ) [ <KEY> ] ( Identifier() )+ )+
[ [ <WITH> ] <DUPLICATES> [ <IN> ] [ <ORDER> ] ]
[ [ <COLLATING> ] <SEQUENCE> [ <IS> ] AlphabetName() ]
( <USING> ( FileName() )+ | <INPUT> <PROCEDURE> [ <IS> ] ProcedureName() [ ( <THROUGH> | <THRU> ) ProcedureName() ] )
( <GIVING> ( FileName() )+ | <OUTPUT> <PROCEDURE> [ <IS> ] ProcedureName() [ ( <THROUGH> | <THRU> ) ProcedureName() ] )
}

void StartStatement() :
{}
{
<START> FileName()
[ <KEY> RelationalOperator() Identifier() ]
[ <INVALID> [ <KEY> ] StatementList() ]
[ <NOT> <INVALID> [ <KEY> ] StatementList() ]
[ <END_START> ]
}

void StopStatement() :
{}
{
<STOP> ( <RUN> | Literal() )
}

void StringStatement() :
{}
{
<STRING>
( ( Identifier() | IterableLiteral() )+ <DELIMITED> [ <BY> ] ( Identifier() | IterableLiteral() | <SIZE> ) )+
<INTO> Identifier()
[ [ <WITH> ] <POINTER> Identifier() ]
[ [ <ON> ] <OVERFLOW> StatementList() ]
[ <NOT> [ <ON> ] <OVERFLOW> StatementList() ]
[ <END_STRING> ]
}

void SubtractStatement() :
{}
{
<SUBTRACT>
( ( Identifier() | Literal() )+ <FROM> ( ( Identifier() | Literal() ) [ <ROUNDED> ] )+ [ <GIVING> ( Identifier() [ <ROUNDED> ] )+ ]
| ( <CORRESPONDING> | <CORR> ) Identifier() <FROM> Identifier() [ <ROUNDED> ]
)
[ [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <NOT> [ <ON> ] <SIZE> <ERROR> StatementList() ]
[ <END_SUBTRACT> ]
}

void UnstringStatement() :
{}
{
<UNSTRING> Identifier()
[ <DELIMITED> [ <BY> ] [ <ALL> ] ( Identifier() | Literal() ) ( <OR> [ <ALL> ] ( Identifier() | Literal() ) )* ]
<INTO>
( Identifier() [ <DELIMITER> [ <IN> ] Identifier() ] [ <COUNT> [ <IN> ] Identifier() ] )+
[ [ <WITH> ] <POINTER> Identifier() ] [ <TALLYING> [ <IN> ] Identifier() ]
[ [ <ON> ] <OVERFLOW> StatementList() ]
[ <NOT> [ <ON> ] <OVERFLOW> StatementList() ]
[ <END_UNSTRING> ]
}

void UseStatement() :
{}
{
<USE> ( [ <FOR> ] <DEBUGGING> [ <ON> ] ( ( ProcedureName() )+ | <ALL> ( <PROCEDURES> | <REFERENCES> <OF> DataName() ) )
| [ <GLOBAL> ] <AFTER> [ <STANDARD> ]
( ( <EXCEPTION> | <ERROR> )
| [ ( <BEGINNING> | <ENDING> ) ] [ ( <FILE> | <REEL> | <UNIT> ) ] <LABEL>
)
<PROCEDURE> [ <ON> ] ( ( FileName() )+ | <INPUT> | <OUTPUT> | <I_O> | <EXTEND> )
)
}

void WriteStatement() :
{}
{
<WRITE> RecordName() [ <FROM> Identifier() ]
[ AdvancingPhrase() ]
[ [ <AT> ] ( <END_OF_PAGE> | <EOP> ) StatementList() ]
[ <NOT> [ <AT> ] ( <END_OF_PAGE> | <EOP> ) StatementList() ]
[ <INVALID> [ <KEY> ] StatementList() ]
[ <NOT> <INVALID> [ <KEY> ] StatementList() ]
[ <END_WRITE> ]
}

void AdvancingPhrase() :
{}
{
( <BEFORE> | <AFTER> ) [ <ADVANCING> ]
( <PAGE>
| ( IntegerConstant() | Identifier() | Literal() ) [ ( <LINE> | <LINES> ) ]
| MnemonicName() // Useless here ...
)
}

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Generating Parsers with JavaCC, Second Edition

2009-10-27T09:27:29Z

This is a plug... stop reading now if you don't like those!

That said, we're now shipping the second edition of my JavaCC book,
Generating Parsers with JavaCC!

Creating this second edition was a labor of love. I went through every
page of the first edition and found all the places where JavaCC had
moved out from under the book. I've brought those bits up to date,
added new sections all over, fixed numerous typos, added more index
entries, and generally given it a thorough house cleaning. This second
edition also benefits from the intervening two years during which I
worked on the JavaCC code and answered questions from folks on the
javacc-users mailing list. And I've generally had a little more time
to digest some of the information in the first edition, so I think the
second edition is improved because of that.

So, what does the second edition include? Here's a sampling:

Discussion of new JavaCC features like TRACK_TOKENS, TOKEN_EXTENDS,
SUPPORT_CLASS_VISIBILITY_PUBLIC, TOKEN_FACTORY, and more.
Details of new JJTree features like NODE_CLASS, VISITOR_DATA_TYPE,
VISITOR_RETURN_TYPE, and more.
Updated examples to reflect JavaCC's current defaults - e.g.,
JDK_VERSION is now 1.5 by default, the options block is now optional
if empty, and more.
Updated chapter 8 (Case Study: The JavaCC Grammar) to reflect
latest JavaCC grammar.
Update software versions used throughout - the second edition uses
much more recent versions of Java, Maven, JUnit, Jaxen.
Changes since JavaCC 4.0 are flagged in the index for easy lookup.
Many small fixes to the text, to examples, to code samples.

I'm excited to get this second edition out there; I think folks will
find that it's a nice refresh. Enjoy!

http://generatingparserswithjavacc.com/

Yours,

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: (jjtree) Problem with TRACK_TOKENS - firstToken is wrong

2009-10-26T17:31:11Z

> Hi,
>
> I have issues concerning TRACK_TOKENS in jjtree as it sets the wrong
> "firstToken" on some nodes in my case.

.....
>
> Is this all a problem with the grammar or a bug in JJTree?
>
> TIA
> Carsten

Hi Carsten,

That sounds like a bug in jjtree (I've not checked the details, but using
getToken(1) doesn't sound right).

Would you mind creating an issue
(https://javacc.dev.java.net/servlets/ProjectIssues) so that we remember
to fix it in the next release?

Cheers.
Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Token Interface and JFlex

2009-10-26T17:30:16Z

KawaDD is now known as FreeCC I believe.

On Tue, Oct 27, 2009 at 10:45 AM, Paul Cager <paul-debian@...> wrote:

> I also would like to vote for getting rid of the "wired in" access to
> the Token class, and think providing an IToken interface would be better.
>

....

> Some tinkering was necessary to make it work, and I believe the task
> would have been a bit easier if an IToken interface had been used by the
> generated code. Are there currently any plans to move in that direction?
>
> Bill Fenlason

Yes, modernising JavaCC is probably high on the "to do" list. There are
also two forks of JavaCC with a more modern API (but not backwards
compatible):

https://javacc.dev.java.net/servlets/ReadMsg?list=users&msgNo=2923

and

KawaDD.

Cheers,
Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

--
- J.Chris Findlay
(c:

Re: Token Interface and JFlex

2009-10-26T14:45:34Z

> I also would like to vote for getting rid of the "wired in" access to
> the Token class, and think providing an IToken interface would be better.
>
....

> Some tinkering was necessary to make it work, and I believe the task
> would have been a bit easier if an IToken interface had been used by the
> generated code. Are there currently any plans to move in that direction?
>
> Bill Fenlason

Yes, modernising JavaCC is probably high on the "to do" list. There are
also two forks of JavaCC with a more modern API (but not backwards
compatible):

https://javacc.dev.java.net/servlets/ReadMsg?list=users&msgNo=2923

and

KawaDD.

Cheers,
Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Error Recovery

2009-10-26T13:59:33Z

There are examples in the docs for both shallow and deep error recovery.

On Tue, Oct 27, 2009 at 9:57 AM, Kurt Guenther <kurtg@...> wrote:

Are there any patterns for doing error recovery (to report more then one error to the user) ?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

--
- J.Chris Findlay
(c:

Error Recovery

2009-10-26T13:57:51Z

Are there any patterns for doing error recovery (to report more then one
error to the user) ?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...