unit testing lexer

[borisd]

Hi,

I wrote a parser for a MiniJava language using ML-Lex and ML-Yacc, and
I'm trying to create unit tests for the lexer part. The tokens that I'm
reading have this form:

val it = TOKEN (T 8,(VOID',17,18)) :
?.MiniJavaLex.UserDeclarations.lexresult

I'm having a hard time breaking this value apart and recreating the
original token I created in the lexer rule, eg:

boolean        => (Tokens.BOOLEAN(yypos, yypos + 7));

Can someone give me a hint?
Thanks.


------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Smlnj-list mailing list
Smlnj-list@...
https://lists.sourceforge.net/lists/listinfo/smlnj-list

[Jon Riehl]

Hi,

I've thought about this as well, and the best I can come up with is
writing a trivial parser (i.e. where each token is a single
production, reducing to the start nonterminal).  Either that, or you
might dig into the code ML Yacc generates and see what it does to get
its token information from the result returned by the lexer, then cut
and paste that into your test driver.

Hope this helps,
-Jon

On Thu, Jan 29, 2009 at 12:48 AM, Boris D <borisd@...> wrote:

> Hi,
>
> I wrote a parser for a MiniJava language using ML-Lex and ML-Yacc, and
> I'm trying to create unit tests for the lexer part. The tokens that I'm
> reading have this form:
>
> val it = TOKEN (T 8,(VOID',17,18)) :
> ?.MiniJavaLex.UserDeclarations.lexresult
>
> I'm having a hard time breaking this value apart and recreating the
> original token I created in the lexer rule, eg:
>
> boolean        => (Tokens.BOOLEAN(yypos, yypos + 7));
>
> Can someone give me a hint?
> Thanks.
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by:
> SourcForge Community
> SourceForge wants to tell your story.
> http://p.sf.net/sfu/sf-spreadtheword
> _______________________________________________
> Smlnj-list mailing list
> Smlnj-list@...
> https://lists.sourceforge.net/lists/listinfo/smlnj-list
>

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Smlnj-list mailing list
Smlnj-list@...
https://lists.sourceforge.net/lists/listinfo/smlnj-list