Check the examples and docs where it talks about extending the
SimpleNode class for jjtree as it does exactly that.
Basically it saves a reference to the first and last token "owned" by
a given node.
Note that saving the token stream for a large input can consume a lot of memory.
Alternatively, just save a reference to the first token as they all
chain off of it.
If you are going to save the entire stream, I'd recommend
post-processing it to add prevNode links.
Also consider if rewriting the format by rewriting the token stream is
more or less work than regenerating it from the parsed tree. If you
are keeping whitespace (even as special tokens rather than skipped
ones), then it's a fair amount of work to decide how to classify
things, as typically the whitespace appears in the token stream with
one token per whitespace character.
On Mon, Apr 27, 2009 at 4:05 PM, Michel Trudeau
<
michel.trudeau@...> wrote:
> My apologies if that is a really basic question but:
>
> How can I save the list of tokens while parsing the input?
>
> I want to write a small grammar for my language and want to invoke the
> parser for the root element but would like to also save all tokens being
> read by the scanner so I can rewrite the file with a possibly new format.
>
> Thanks,
> Michel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
users-unsubscribe@...
> For additional commands, e-mail:
users-help@...
>
>
--
- J.Chris Findlay
(c:
---------------------------------------------------------------------
To unsubscribe, e-mail:
users-unsubscribe@...
For additional commands, e-mail:
users-help@...
