Antlr v3 and Netbeans Lexer Integration

[Jeff Johnston-3]

I created a short blog about how to integrate a Netbeans Lexer with Antlr.

http://sqlraider.blogspot.com/2009/06/antlr-v3-and-netbeans-lexer-integration.html

-Jeff Johnston

[Andreas Stefik]

Jeff,

Thanks for posting this! I was considering writing one up myself, as I didn't see it online and have been working on something similar in my current project. Now I don't have too.

Perhaps this could be added to the Wiki for new language support, located here:

http://wiki.netbeans.org/New_Language_Support_TutorialDevelopment_Version

Anyway, thanks for posting your solution,

Andy

On Sun, Jun 21, 2009 at 3:30 PM, Jeff Johnston <[e-mail]> wrote:

I created a short blog about how to integrate a Netbeans Lexer with Antlr.

http://sqlraider.blogspot.com/2009/06/antlr-v3-and-netbeans-lexer-integration.html

-Jeff Johnston

[Jeff Johnston-3]

Oh, I did not realize that anyone could update the wiki! I will just move it there, and instead of just having pointers to my code I will put the code right in the wiki. It would be great if we could just keep updating the wiki so that we have a solution that works for Antlr users.

I am also trying to get the parsing API working now with Antlr. Once I figure that I will post the code for that as well.

-Jeff Johnston

On Sun, Jun 21, 2009 at 6:04 PM, Andreas Stefik <stefika@...> wrote:

Jeff,

Thanks for posting this! I was considering writing one up myself, as I didn't see it online and have been working on something similar in my current project. Now I don't have too.

Perhaps this could be added to the Wiki for new language support, located here:

http://wiki.netbeans.org/New_Language_Support_TutorialDevelopment_Version

Anyway, thanks for posting your solution,

Andy

On Sun, Jun 21, 2009 at 3:30 PM, Jeff Johnston <[e-mail]> wrote:

I created a short blog about how to integrate a Netbeans Lexer with Antlr.

http://sqlraider.blogspot.com/2009/06/antlr-v3-and-netbeans-lexer-integration.html

-Jeff Johnston

[Andreas Stefik]

Jeff,

Oh, I did not realize that anyone could update the wiki! I will just move it there, and instead of just having pointers to my code I will put the code right in the wiki. It would be great if we could just keep updating the wiki so that we have a solution that works for Antlr users.

Yaa, absolutely. The current tutorial, as you know, is for JavaCC, which is a perfectly fine Java-only LL parser generator, but .... I've written a number of compilers/VM's in JavaCC, Lex/Yacc, and (as of recently) am working on a new one in ANTLR, which in my opinion is far superior to its competitors (Tree walkers are so much easier to use then the visitor pattern ...) I've also taught classes using JavaCC and ANTLR and students I think, overwhelmingly, find ANTLR easier to use, in part because of ANTLRWorks.

So yaa, I couldn't agree more --- having ANTLR compatability up on the wiki would be nice.
 

I am also trying to get the parsing API working now with Antlr. Once I figure that I will post the code for that as well.

I'm looking forward to seeing that.

Andy
 

-Jeff Johnston

On Sun, Jun 21, 2009 at 6:04 PM, Andreas Stefik <stefika@...> wrote:

Jeff,

Thanks for posting this! I was considering writing one up myself, as I didn't see it online and have been working on something similar in my current project. Now I don't have too.

Perhaps this could be added to the Wiki for new language support, located here:

http://wiki.netbeans.org/New_Language_Support_TutorialDevelopment_Version

Anyway, thanks for posting your solution,

Andy

On Sun, Jun 21, 2009 at 3:30 PM, Jeff Johnston <[e-mail]> wrote:

I created a short blog about how to integrate a Netbeans Lexer with Antlr.

http://sqlraider.blogspot.com/2009/06/antlr-v3-and-netbeans-lexer-integration.html

-Jeff Johnston

[Jeff Johnston-3]

I updated the wiki now to include in more detail everything I have done so far with integrating ANTLR into Netbeans.

http://wiki.netbeans.org/New_Language_Support_Tutorial_Antlr

-Jeff Johnston

On Sun, Jun 21, 2009 at 7:33 PM, Andreas Stefik <stefika@...> wrote:

Jeff,

Oh, I did not realize that anyone could update the wiki! I will just move it there, and instead of just having pointers to my code I will put the code right in the wiki. It would be great if we could just keep updating the wiki so that we have a solution that works for Antlr users.

Yaa, absolutely. The current tutorial, as you know, is for JavaCC, which is a perfectly fine Java-only LL parser generator, but .... I've written a number of compilers/VM's in JavaCC, Lex/Yacc, and (as of recently) am working on a new one in ANTLR, which in my opinion is far superior to its competitors (Tree walkers are so much easier to use then the visitor pattern ...) I've also taught classes using JavaCC and ANTLR and students I think, overwhelmingly, find ANTLR easier to use, in part because of ANTLRWorks.

So yaa, I couldn't agree more --- having ANTLR compatability up on the wiki would be nice.
 

I am also trying to get the parsing API working now with Antlr. Once I figure that I will post the code for that as well.

I'm looking forward to seeing that.

Andy
 

-Jeff Johnston

On Sun, Jun 21, 2009 at 6:04 PM, Andreas Stefik <stefika@...> wrote:

Jeff,

Thanks for posting this! I was considering writing one up myself, as I didn't see it online and have been working on something similar in my current project. Now I don't have too.

Perhaps this could be added to the Wiki for new language support, located here:

http://wiki.netbeans.org/New_Language_Support_TutorialDevelopment_Version

Anyway, thanks for posting your solution,

Andy

On Sun, Jun 21, 2009 at 3:30 PM, Jeff Johnston <[e-mail]> wrote:

I created a short blog about how to integrate a Netbeans Lexer with Antlr.

http://sqlraider.blogspot.com/2009/06/antlr-v3-and-netbeans-lexer-integration.html

-Jeff Johnston

[Andreas Stefik]

This looks great Jeff!

 I'll definitely be using some of the wiki info in my project.  Here's a couple thoughts:

1. I don't totally understand why you have have these methods empty:

public void addChangeListener(ChangeListener changeListener) {}
public void removeChangeListener(ChangeListener changeListener) {}

Does this have to do with ANTLR not being an incremental parser generator, or is this totally unrelated? All the documentation for the API says about these methods (posted here):

http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-parsing-api/org/netbeans/modules/parsing/spi/Parser.html

for this method is "Registers new listener," which doesn't help very much. If you know something more it might be good to post info about it.

2. There's another way of doing error handling of syntax errors in ANTLR. The way Terrence Par talks about it in his book is to define an @rule catch block, which looks like this:

@rulecatch {
    catch(RecognitionException re) {
    }
}

What I usually do for projects like this (for good or ill), is to 1) create some general abstractions (e.g., compiler error manager classes), 2) catch all of the errors ANTLR outputs and send them to the abstractions. This usually makes it a bit easier to manage the errors, especially if you are implementing further stages in a compiler/virtual machine architecture. Anyway, that's all probably too much for your tutorial, but I thought I'd mention @rulecatch as an alternative error handling mechanism in case you'd never heard of it.

Great tutorial and thanks for posting it! If you aren't on the ANTLR mailing list, I can cross post it there. I image there's some folks there that would be interested in this as well.

Andreas

[Jeff Johnston-3]

>>1. I don't totally understand why you have have these methods empty:

I was just trying to adapt the JavaCC tutorial. They left it off and it seems to work fine without it.

http://wiki.netbeans.org/New_Language_Support_Tutorial_Antlr

>>2. There's another way of doing error handling of syntax errors in ANTLR...

My reasoning for not doing the @rulecatch way was I thought that ANTLR would report multiple errors for the same part of the syntax. The reason I thought this is further up the calling chain is this comment and early exit from the reportError() method. I didn't test out this assumption though, and if that is not the case then I like the clarity of the @rulecatch. Plus thinking about it, to really get good error messages for specific parts of the grammer I will probably have to use the @rulecatch on specific grammer methods. Thanks!

public void reportError(RecognitionException e) {
        // if we've already reported an error and have not matched a token
        // yet successfully, don't report any errors.
        if ( state.errorRecovery ) {
            //System.err.print("[SPURIOUS] ");
            return;
        }
       ...
}

I am not on the ANTLR mailing list but I probably should just to engage in the discussions.

This whole learning and integration of Netbeans and ANTLR has been a lot of fun as it is all very new to me! To figure this out I have my ANTLR book, 2 Netbeans books, as well as all the source code checked out for both projects. So then I just keep reading and digging and somehow manage to keep moving forward :). I woud probably feel reluctant to even post what I have done so far but ANTLR for Netbeans doesn't seem to be documented anywhere else so I figure this would at least push someone else in the right direction. As I get better at both and my editor gets better I will keep updating the wiki!

-Jeff Johnston

On Sun, Jul 5, 2009 at 6:11 PM, Andreas Stefik <stefika@...> wrote:

This looks great Jeff!

 I'll definitely be using some of the wiki info in my project.  Here's a couple thoughts:

1. I don't totally understand why you have have these methods empty:

public void addChangeListener(ChangeListener changeListener) {}
public void removeChangeListener(ChangeListener changeListener) {}

Does this have to do with ANTLR not being an incremental parser generator, or is this totally unrelated? All the documentation for the API says about these methods (posted here):

http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-parsing-api/org/netbeans/modules/parsing/spi/Parser.html

for this method is "Registers new listener," which doesn't help very much. If you know something more it might be good to post info about it.

2. There's another way of doing error handling of syntax errors in ANTLR. The way Terrence Par talks about it in his book is to define an @rule catch block, which looks like this:

@rulecatch {
    catch(RecognitionException re) {
    }
}

What I usually do for projects like this (for good or ill), is to 1) create some general abstractions (e.g., compiler error manager classes), 2) catch all of the errors ANTLR outputs and send them to the abstractions. This usually makes it a bit easier to manage the errors, especially if you are implementing further stages in a compiler/virtual machine architecture. Anyway, that's all probably too much for your tutorial, but I thought I'd mention @rulecatch as an alternative error handling mechanism in case you'd never heard of it.

Great tutorial and thanks for posting it! If you aren't on the ANTLR mailing list, I can cross post it there. I image there's some folks there that would be interested in this as well.

Andreas

Great work!

I saw that you implemented TokenId like the orginal guide did.
I think its better to define SqlEditorTokenId as enum, and get rid of
the "int id".


Jonny Heggheim

On Mon, Jul 6, 2009 at 12:11 AM, Jeff
Johnston<jeff.johnston.mn@...> wrote:

> I updated the wiki now to include in more detail everything I have done so
> far with integrating ANTLR into Netbeans.
>
> http://wiki.netbeans.org/New_Language_Support_Tutorial_Antlr
>
> -Jeff Johnston
>
>
> On Sun, Jun 21, 2009 at 7:33 PM, Andreas Stefik <stefika@...> wrote:
>>
>> Jeff,
>>
>>
>>> Oh, I did not realize that anyone could update the wiki! I will just move
>>> it there, and instead of just having pointers to my code I will put the code
>>> right in the wiki. It would be great if we could just keep updating the wiki
>>> so that we have a solution that works for Antlr users.
>>
>> Yaa, absolutely. The current tutorial, as you know, is for JavaCC, which
>> is a perfectly fine Java-only LL parser generator, but .... I've written a
>> number of compilers/VM's in JavaCC, Lex/Yacc, and (as of recently) am
>> working on a new one in ANTLR, which in my opinion is far superior to its
>> competitors (Tree walkers are so much easier to use then the visitor pattern
>> ...) I've also taught classes using JavaCC and ANTLR and students I think,
>> overwhelmingly, find ANTLR easier to use, in part because of ANTLRWorks.
>>
>> So yaa, I couldn't agree more --- having ANTLR compatability up on the
>> wiki would be nice.
>>
>>>
>>> I am also trying to get the parsing API working now with Antlr. Once I
>>> figure that I will post the code for that as well.
>>
>> I'm looking forward to seeing that.
>>
>> Andy
>>
>>>
>>> -Jeff Johnston
>>>
>>>
>>> On Sun, Jun 21, 2009 at 6:04 PM, Andreas Stefik <stefika@...>
>>> wrote:
>>>>
>>>> Jeff,
>>>>
>>>> Thanks for posting this! I was considering writing one up myself, as I
>>>> didn't see it online and have been working on something similar in my
>>>> current project. Now I don't have too.
>>>>
>>>> Perhaps this could be added to the Wiki for new language support,
>>>> located here:
>>>>
>>>>
>>>> http://wiki.netbeans.org/New_Language_Support_TutorialDevelopment_Version
>>>>
>>>> Anyway, thanks for posting your solution,
>>>>
>>>> Andy
>>>>
>>>> On Sun, Jun 21, 2009 at 3:30 PM, Jeff Johnston
>>>> <jeff.johnston.mn@...> wrote:
>>>>>
>>>>> I created a short blog about how to integrate a Netbeans Lexer with
>>>>> Antlr.
>>>>>
>>>>>
>>>>> http://sqlraider.blogspot.com/2009/06/antlr-v3-and-netbeans-lexer-integration.html
>>>>>
>>>>> -Jeff Johnston
>>>>>
>>>>>
>>>>
>>>
>>
>
>

[Andreas Stefik]

Great work!

I saw that you implemented TokenId like the orginal guide did.
I think its better to define SqlEditorTokenId as enum, and get rid of
the "int id".

 

If I recall, in ANTLR, the reason why that int ID is there is because ANTLR generates .token files, which have the ID's embedded. ANTLR uses these id's internally to store the type of each token. For example, mine looks something like this:

FUNCTION=12
EQUALITY=19
MODULO=33
CLASS=9
DOUBLE_QUOTE=39
PERIOD=17

The nice thing, then, about having an ID is that you can then easily write a program that will regenerate your architecture from the .token files, generating stuff that is conceptually like this:

new CompilerTokenId("FUNCTION", "keyword", 12),
new CompilerTokenId("EQUALITY", "operator", 19),
new CompilerTokenId("MODULO", "operator", 33),
new CompilerTokenId("CLASS", "keyword", 9),
new CompilerTokenId("DOUBLE_QUOTE", "whitespace", 39),
new CompilerTokenId("PERIOD", "operator", 17),

I can't speak for everyone, but in my case, having the ID like this makes the architecture map nicely to ANTLR, which had made it easy to debug (I can check ANTLR internals and compare them with mine).

Anyway, just my two cents.

Andy