Dynamic use of JavaCC possible?

[Kevin Jensen]

Hello group. I am still new to JavaCC, but have a basic idea of what it
can do. For my current specific problem I think that using JavaCC can be
of great help, by saving me from writing one or two thousand LOC by hand
and end up with more declerative code instead.

My planned workflow is:
1. lexing an input string into a sequence (for example an ArrayList) of
    Tokens, where Token is my simple class with just the two fields
    "token" and "category".
2. After lexing I want to send this token sequence through 10-20
    preprocessors, which are methods that take a token sequence and
    return one. All the preprocessors search for patterns and replace
    subsequences of tokens with other subsequences.
3. When that is done I then want to partition the token sequence into
    several subsequences on which...
4. ... then my core algorithms continue to work.

Today I thought that maybe JavaCC can do all (steps 1-3) or nearly all
(the 10-20 substeps of step 2) of that work in just one step. What I had
so far in mind for my (not yet) hand written code was to let each
preprocessor walk through the whole sequence again and again,
(possibly/probably) modifiying it each time. That will not run
fast, and in principle they are all doing the same thing, namely replace
some tokens following a pattern with some other pattern.

Provided that JavaCC can do that, now my real challenge comes, and here
I could need your help:
As I said, the preprocessors replace subsequences of tokens with
another subsequence. But that part needs to be dynamic.
Users need to be able to add such substitution rules (over an easy to
use webinterfacer) at runtime. Could that be made possible somehow?

I thought that when a user adds a substitution, my app could then write
out the grammar file and insert the new code at well defined points
into it. Then it could run javacc and javac.
The question is if I then can somehow load the resulting parser into the
live app? Is there an easier way than the one I described?

Also I will need to have many customer specific parsers available at the
same time. Customer A might need different substitutios than customer B
or C.
Also threads currently working on tasks of Customer A should continue to
do so with the parser they started with. Only new jobs for A should use
the new parser.
Does that mean that when writing out the grammar file, that it also
needs to generate a new unique name for the parser class, so that many
versions of it can run at the same time?


kj

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Tom Copeland]

On Aug 4, 2009, at 8:44 PM, Kevin Jensen wrote:

> Hello group. I am still new to JavaCC, but have a basic idea of what  
> it can do. For my current specific problem I think that using JavaCC  
> can be
> of great help, by saving me from writing one or two thousand LOC by  
> hand
> and end up with more declerative code instead.
>
> My planned workflow is:
> 1. lexing an input string into a sequence (for example an ArrayList)  
> of
>   Tokens, where Token is my simple class with just the two fields
>   "token" and "category".

Hi Kevin -

Sounds like a neat project!  And JavaCC can certainly help with that  
part of it - tokenizing a string and getting a List of Token objects.

> I thought that when a user adds a substitution, my app could then  
> write
> out the grammar file and insert the new code at well defined points
> into it. Then it could run javacc and javac.
> The question is if I then can somehow load the resulting parser into  
> the
> live app? Is there an easier way than the one I described?

When the user adds a substitution, you could kick off a background  
process that does all the work (javacc, compile, run the tokenizer  
against the input data) and then have that background process send the  
results over to your web application.  That might be more  
straightforward than sorting through Classloader issues.

> Also I will need to have many customer specific parsers available at  
> the
> same time. Customer A might need different substitutios than  
> customer B
> or C.
> Also threads currently working on tasks of Customer A should  
> continue to
> do so with the parser they started with. Only new jobs for A should  
> use
> the new parser.
> Does that mean that when writing out the grammar file, that it also
> needs to generate a new unique name for the parser class, so that many
> versions of it can run at the same time?

Yeah, I'd have a worker processes that were responsible for doing all  
that stuff in a tmp directory; you could use SHA1 hash of user_id +  
curTimeMillis or something for directory name.  Once the worker  
process is done it sends the results back to the web app.  On the web  
browser side you could have an AJAX gizmo that polls for job  
results... this would also let you queue things and keep the system  
load under control.

Yours,

Tom
http://generatingparserswithjavacc.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...