spirit 2.1: trying to write a conventional lexer/parser using spirit

[Jean-Francois Ostiguy]

Hi -

I recently discovered Spirit 2.1. I decided to write a toy example for
learning purposes and quickly ran into a problem ... most likely I am
missing something ...

Here is my lexer:  
===================================================================================
template <typename Lexer>
struct my_lexer : boost::spirit::lex::lexer<Lexer>
{
   my_lexer() {
       
       identifier = "[a-zA-Z][_\\.a-zA-Z0-9]*";              
       start      = "begin|BEGIN";
       finish     = "end|END";              
       real       = "([0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?)|([-+]?[1-9]+\\.?
([eE][-+]?[0-9]+))";
       integer    = "[0-9]+[0-9]*";  
       comment    = "#.*";              


   this->self.add
            (comment,    ID_COMMENT       )  
         (start,      ID_BEGIN         )  
         (finish,     ID_END           )  
         (integer,    ID_INTEGER       )  
                (real,       ID_REAL          )  
            (identifier, ID_IDENTIFIER    )  
            (".")   <=== ???????????????????????????????
        ;
    }

    lex_ns::token_def<int>          integer;
    lex_ns::token_def<double>       real;
    lex_ns::token_def<std::string>  identifier;
    lex_ns::token_def<std::string>  comment;
    lex_ns::token_def<std::string>  start;    
    lex_ns::token_def<std::string>  finish;
}
=======================================================================================

What I would like to achieve is to have any single character
( e.g. '+','-', '=', '\n'... ) passed directly as 'literal' token.
In standard lex/yacc, the ascii code is used as the token id in that case.
My lexer works, and certainly the characters matched by "." are passed as
tokens, but these 'literal' tokens are simply not recognized
in my grammar. E.g a rule

tok.identifier >> '=' >> tok.real;  

is never matched by  "variable=1.0"

In addition, the blanks resulting from the match with (".") do not seem to
match ascii::space (the skipper I am using) either.  The documentation is
less than clear about exactly what token type generated by the "." regex
above is. While the value is correct, the token id is 256. When I use '=' or
'+' in my  grammar, what token type is generated ? What id does it have ?
When how are 2 tokens declared a match ? Are the ids compared  ? ... or are
the values compared when the ids were left undefined ?

Thanks in advance.

-Francois
 


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> I recently discovered Spirit 2.1. I decided to write a toy example for
> learning purposes and quickly ran into a problem ... most likely I am
> missing something ...
>
> Here is my lexer:
> =======================================================================
> ============
> template <typename Lexer>
> struct my_lexer : boost::spirit::lex::lexer<Lexer>
> {
>    my_lexer() {
>
>        identifier = "[a-zA-Z][_\\.a-zA-Z0-9]*";
>        start      = "begin|BEGIN";
>        finish     = "end|END";
>        real       = "([0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?)|([-+]?[1-
> 9]+\\.?
> ([eE][-+]?[0-9]+))";
>        integer    = "[0-9]+[0-9]*";
>        comment    = "#.*";
>
>
>    this->self.add
>    (comment,    ID_COMMENT       )
>          (start,      ID_BEGIN         )
>          (finish,     ID_END           )
>          (integer,    ID_INTEGER       )
>                 (real,       ID_REAL          )
>    (identifier, ID_IDENTIFIER    )
>    (".")   <=== ???????????????????????????????
>         ;
>     }
>
>     lex_ns::token_def<int>          integer;
>     lex_ns::token_def<double>       real;
>     lex_ns::token_def<std::string>  identifier;
>     lex_ns::token_def<std::string>  comment;
>     lex_ns::token_def<std::string>  start;
>     lex_ns::token_def<std::string>  finish;
> }
> =======================================================================
> ================
>
> What I would like to achieve is to have any single character
> ( e.g. '+','-', '=', '\n'... ) passed directly as 'literal' token.
> In standard lex/yacc, the ascii code is used as the token id in that
> case.
> My lexer works, and certainly the characters matched by "." are passed
> as
> tokens, but these 'literal' tokens are simply not recognized
> in my grammar. E.g a rule
>
> tok.identifier >> '=' >> tok.real;

Even if you want to match characters verbatim, you still need to add them to
your lexer as token definitions:

    this->self += char_('=') | '+' | ...;

then the lexer will generate tokens for those.

> is never matched by  "variable=1.0"

Well, all strings are interpreted as regular expressions, so

    this->self.add (".");

is interpreted as the regular expression "." (any character).

Please note, that single character literals are always interpreted verbatim,
i.e.

    this->self.add ('.');
 
is _not_ a regular expression, but rather equivalent to the regular
expression "\."

> In addition, the blanks resulting from the match with (".") do not seem
> to match ascii::space (the skipper I am using) either.  

Yeah, ascii::space doesn't know anything about token matching. You'll need
to use char_(' '). But it's more efficient to define a token for the things
to skip:

   using boost::spirit::lex::_pass;
   this->self += char_(' ')[_pass = false];

forcing the token to be ignored by the lexer without ever being passed to
the parser. You have to make sure to enable semantic actions in the lexer,
though (by using the lex::lexertl::actor_lexer<> instead of the
lex::lexertl::lexer<> type).

> The documentation is
> less than clear about exactly what token type generated by the "."
> regex above is. While the value is correct, the token id is 256. When
> I use '=' or '+' in my  grammar, what token type is generated ? What
> id does it have?

If you use characters as token definitions they will be represented by a
token with a token id matching their ascii character value.

> When how are 2 tokens declared a match ? Are the ids compared  ? ... or
> are the values compared when the ids were left undefined ?

The token id is being matched (and the lexer state, if you use those). The
token value is never used during matching.

HTH
Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut Kaiser wrote:

>
> Even if you want to match characters verbatim, you still need to add them
> to your lexer as token definitions:
>
>     this->self += char_('=') | '+' | ...;
>
> then the lexer will generate tokens for those.

Ok. I guess I was expecting the equivalent of the flex construct

 .  { return yytext[0]); }


> Yeah, ascii::space doesn't know anything about token matching. You'll need
> to use char_(' ').

I am confused ...

in example "Complex - Our first complex parser"
boost::spirit::ascii::space is used as a skipper
argument to phrase_parse( ... )

Are you implying that space is invalid as a skipper argument ?

> But it's more efficient to define a token for the
> things to skip:
>
>    using boost::spirit::lex::_pass;
>    this->self += char_(' ')[_pass = false];

>
>

Ok - I tried this:

...

  this->self.add
            (comment )  
         (xbegin )  
         (xend )  
         (real)  
         (integer)  
            (identifier)  
            ('=')  
            ('+')  
            ('-')  
            ('*')  
            ('/')  
        ;


    using boost::spirit::lex::_pass;

    this->self +=  lex_ns::char_(' ')[_pass = false];
    this->self +=  lex_ns::char_('\n')[_pass = false];
    this->self +=  lex_ns::char_('\t')[_pass = false];
...

Compilation fails, with lots of template errors followed by:

boost/spirit/home/lex/lexer/action.hpp:47: error: 'const struct
boost::spirit::lex::char_token_def<boost::spirit::char_encoding::standard>'
has no member named 'state'

Compilation succeeds when I remove "[_pass = false]" e.g. just add
this->self +=  lex_ns::char_(' ')

BTW: where is the above construct or for that matter the attribute _pass
mentioned or described in the documentation ?



-Francois


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> > Even if you want to match characters verbatim, you still need to add
> them
> > to your lexer as token definitions:
> >
> >     this->self += char_('=') | '+' | ...;
> >
> > then the lexer will generate tokens for those.
>
> Ok. I guess I was expecting the equivalent of the flex construct
>
>  .  { return yytext[0]); }

Yeah, in this case the token id is not the matched character (but the auto
generated id for the "." token). But you should be able to achieve the same
effect by writing

    namespace lex = boost::spirit::lex;
    lex::token_def<char> any(".");

    this->self += any[lex::_tokenid = lex::_val];

which changes the id of the matched token to the matched value (the
character code).

> > Yeah, ascii::space doesn't know anything about token matching. You'll
> need
> > to use char_(' ').
>
> I am confused ...
>
> in example "Complex - Our first complex parser"
> boost::spirit::ascii::space is used as a skipper
> argument to phrase_parse( ... )
>
> Are you implying that space is invalid as a skipper argument ?

Nope. I just said it's invalid while using a lexer. For lexer-less parsing
it works just fine. The reason is that all qi character class parsers
essentially rely on std::isalpha() et.al. and this is not applicable while
using a lexer.

> > But it's more efficient to define a token for the
> > things to skip:
> >
> >    using boost::spirit::lex::_pass;
> >    this->self += char_(' ')[_pass = false];
>
> Ok - I tried this:
>
> ...
>
>   this->self.add
>    (comment )
>          (xbegin )
>          (xend )
>          (real)
>          (integer)
>    (identifier)
>    ('=')
>    ('+')
>    ('-')
>    ('*')
>    ('/')
>         ;
>
>
>     using boost::spirit::lex::_pass;
>
>     this->self +=  lex_ns::char_(' ')[_pass = false];
>     this->self +=  lex_ns::char_('\n')[_pass = false];
>     this->self +=  lex_ns::char_('\t')[_pass = false];
> ...
>
> Compilation fails, with lots of template errors followed by:
>
> boost/spirit/home/lex/lexer/action.hpp:47: error: 'const struct
> boost::spirit::lex::char_token_def<boost::spirit::char_encoding::standa
> rd>'
> has no member named 'state'

Could you send a small test case reproducing your problem? It actually
should work, so you might miss something minor.

> Compilation succeeds when I remove "[_pass = false]" e.g. just add
> this->self +=  lex_ns::char_(' ')
>
> BTW: where is the above construct or for that matter the attribute
> _pass mentioned or described in the documentation ?

Here: http://tinyurl.com/yf5atmz, and here http://tinyurl.com/yjhayb9
(although very briefly).

Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut Kaiser wrote:

>
> Could you send a small test case reproducing your problem? It actually
> should work, so you might miss something minor.
>
>> Compilation succeeds when I remove "[_pass = false]" e.g. just add
>> this->self +=  lex_ns::char_(' ')
>>

Hartmut -

Many thanks. With your help things are becoming (slowly !) clearer.  

I attach a test case. ... and the error output from the compiler.

-Francois

PS: I am using a recent ( ~2 day old snapshot of the boost trunk code)
    and g++ 4.4.2


[LexerTest.cc]

---

#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/bind.hpp>
#include <boost/ref.hpp>
#include <string>
#include <fstream>

#include <boost/spirit/include/lex_lexertl.hpp>

using namespace std;

using namespace boost::spirit;
using namespace boost::spirit::ascii;

namespace lex_ns = boost::spirit::lex;

template <typename Lexer>
struct my_lexer : boost::spirit::lex::lexer<Lexer>
{
   my_lexer() {

        // define tokens (the regular expression to match and the corresponding
        // token id) and add them to the lexer
       
       identifier = "[a-zA-Z][_\\.a-zA-Z0-9]*";              
       xbegin     = "begin|BEGIN";
       xend       = "end|END";              
       real       = "([0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?)|([-+]?[1-9]+\\.?([eE][-+]?[0-9]+))";
       integer    = "[0-9]+[0-9]*";  
       comment    = "#.*";              


   this->self.add
            (comment )  
         (xbegin )  
         (xend )  
         (real)  
         (integer)  
            (identifier)  
            ('=')  
            ('+')  
            ('-')  
            ('*')  
            ('/')  
        ;


    using boost::spirit::lex::_pass;
   
    this->self += lex_ns::char_(' ')[ _pass = false ];
   
    }

    lex_ns::token_def<int>          integer;
    lex_ns::token_def<double>       real;
    lex_ns::token_def<std::string>  identifier;
    lex_ns::token_def<std::string>  comment;
    lex_ns::token_def<std::string>  xbegin;    
    lex_ns::token_def<std::string>  xend;

};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

typedef lex::lexertl::token<char const*, boost::mpl::vector<std::string> > token_t;
typedef lex::lexertl::lexer<token_t>            lexer_type;
typedef my_lexer<lexer_type>::iterator_type iterator_type;


int main( int argc, char* argv[] )
{
  std::string filename = "lattice.txt";
 
  my_lexer<lexer_type>      lexer;            //  lexer

  ifstream infile(filename.c_str());

  std::string str;
  int lineno = 0;

  while (getline(infile, str)) {
   
     size_t len = str.size();

     ++lineno;
     
     char old_fill = cerr.fill('0');

     std::cerr << setw(5)
                << lineno << " : "
                << str << std::endl;

     cerr.fill(old_fill);
 
     char const* first = str.c_str();
     char const* last = &first[str.size()];
 
     bool r = ( str.empty() ? true : lex::tokenize(first, last, lexer ) );

      if (!r) {
        cout <<  "Line "<< lineno << ": Lexing failed\n";
      }
   

    infile.close();    
  }
}

---

[LexerTest.log]

---

Using built-in specs.
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-4.4.2/work/gcc-4.4.2/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.4.2 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.4.2 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.4.2/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.4.2/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --enable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --enable-cld --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/4.4.2/python --enable-java-awt=gtk --enable-languages=c,c++,java,fortran --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.4.2 p1.0'
Thread model: posix
gcc version 4.4.2 (Gentoo 4.4.2 p1.0)
COLLECT_GCC_OPTIONS='-v' '-I/usr/local/include/boost-trunk' '-c' '-shared-libgcc' '-mtune=generic'
 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.4.2/cc1plus -quiet -v -I/usr/local/include/boost-trunk -D_GNU_SOURCE LexerTest.cc -D_FORTIFY_SOURCE=2 -quiet -dumpbase LexerTest.cc -mtune=generic -auxbase LexerTest -version -o /tmp/ccH2fGKR.s
ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include/boost-trunk
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include/g++-v4
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include/g++-v4/x86_64-pc-linux-gnu
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include/g++-v4/backward
 /usr/local/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include-fixed
 /usr/include
End of search list.
GNU C++ (Gentoo 4.4.2 p1.0) version 4.4.2 (x86_64-pc-linux-gnu)
        compiled by GNU C version 4.4.2, GMP version 4.3.1, MPFR version 2.4.1-p5.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 19bb51645a36c94e506fd8e22f6fbc55
In file included from /usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/partition/../string_token.hpp:11,
                 from /usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/partition/charset.hpp:11,
                 from /usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/generator.hpp:12,
                 from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexertl/lexer.hpp:15,
                 from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer_lexertl.hpp:16,
                 from /usr/local/include/boost-trunk/boost/spirit/include/lex_lexertl.hpp:16,
                 from LexerTest.cc:1:
/usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/partition/../consts.hpp:29: warning: integer overflow in expression
In file included from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer.hpp:18,
                 from /usr/local/include/boost-trunk/boost/spirit/home/lex.hpp:13,
                 from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer_lexertl.hpp:15,
                 from /usr/local/include/boost-trunk/boost/spirit/include/lex_lexertl.hpp:16,
                 from LexerTest.cc:1:
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/action.hpp: In member function 'void boost::spirit::lex::action<Subject, Action>::add_actions(LexerDef&) const [with LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >, Subject = boost::spirit::lex::char_token_def<boost::spirit::char_encoding::standard>, Action = boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<bool>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >]':
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexer.hpp:199:   instantiated from 'void boost::spirit::lex::detail::lexer_def_<LexerDef>::compile2pass(const TokenExpr&) [with TokenExpr = boost::spirit::lex::action<boost::spirit::lex::char_token_def<boost::spirit::char_encoding::standard>, boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<bool>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > > >, LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >]'
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexer.hpp:207:   instantiated from 'void boost::spirit::lex::detail::lexer_def_<LexerDef>::define(const Expr&) [with Expr = boost::proto::exprns_::expr<boost::proto::tag::subscript, boost::proto::argsns_::list2<boost::proto::exprns_::expr<boost::proto::tag::terminal, boost::proto::argsns_::term<boost::spirit::terminal_ex<boost::spirit::tag::char_code<boost::spirit::tag::char_, boost::spirit::char_encoding::standard>, boost::fusion::vector1<char> > >, 0l>&, boost::proto::exprns_::expr<boost::proto::tag::terminal, boost::proto::argsns_::term<const boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<bool>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >&>, 0l> >, 2l>, LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >]'
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexer.hpp:273:   instantiated from 'boost::spirit::lex::detail::lexer_def_<LexerDef>& boost::spirit::lex::detail::operator+=(boost::spirit::lex::detail::lexer_def_<LexerDef>&, const Expr&) [with LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >, Expr = boost::proto::exprns_::expr<boost::proto::tag::subscript, boost::proto::argsns_::list2<boost::proto::exprns_::expr<boost::proto::tag::terminal, boost::proto::argsns_::term<boost::spirit::terminal_ex<boost::spirit::tag::char_code<boost::spirit::tag::char_, boost::spirit::char_encoding::standard>, boost::fusion::vector1<char> > >, 0l>&, boost::proto::exprns_::expr<boost::proto::tag::terminal, boost::proto::argsns_::term<const boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<bool>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >&>, 0l> >, 2l>]'
LexerTest.cc:49:   instantiated from 'my_lexer<Lexer>::my_lexer() [with Lexer = boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > >]'
LexerTest.cc:74:   instantiated from here
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/action.hpp:47: error: 'const struct boost::spirit::lex::char_token_def<boost::spirit::char_encoding::standard>' has no member named 'state'

---

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[CARL BARRON-3]

On Oct 28, 2009, at 5:39 PM, Hartmut Kaiser wrote:
>>
>>    using boost::spirit::lex::_pass;
>>
>>    this->self +=  lex_ns::char_(' ')[_pass = false];
>>    this->self +=  lex_ns::char_('\n')[_pass = false];
>>    this->self +=  lex_ns::char_('\t')[_pass = false];

namespace lex = boost::spirit::lex;

template <class Lexer>
struct calc_input:lex::lexer<Lexer>
{
     calc_input()
     {
        int_ =  "[-+]?[0-9]+;
        ws  =  "[ \t\n]+";
        ds  = '$';
        other =  "[^$]";
        this->self =  int_
                |  ws [lex::_pass = lex::pass_flags::pass_ignore]
                |  ds [lex::_pass = lex::pass_flags::pass_fail] // a '$' is an error
                |  other [lex::_tokenid = *lex::_start] // id = char matched.
                ;
     }
     lex::token_id<int> int_;
     lex::token_id<> ws,other;
};

parser sees  tok.int_ , literaals  '+',-',etc. but not tok.ws.

so no skipper is used with the grammar.

note  lex::pass can have one of three values
pass_flags::pass_normal  -  default value  as in int_ above
pass_flags::pass_ignore  -  'skip this'  as in ws above
pass_flags::pass_fail -  force lexer to fail not consuming, which  
makes it not match
any of the above tokens in the above.



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Carl Barron wrote:
 

> namespace lex = boost::spirit::lex;
>
> template <class Lexer>
> struct calc_input:lex::lexer<Lexer>
> {
>      calc_input()
>      {
> int_ =  "[-+]?[0-9]+;
> ws  =  "[ \t\n]+";
> ds  = '$';
> other =  "[^$]";
> this->self =  int_
> |  ws [lex::_pass = lex::pass_flags::pass_ignore]
> |  ds [lex::_pass = lex::pass_flags::pass_fail] // a '$' is an error
> |  other [lex::_tokenid = *lex::_start] // id = char matched.
> ;
>      }
>      lex::token_id<int> int_;
>      lex::token_id<> ws,other;
> };
>

Carl ( Hartmut )

this is all very nice; I saw the lex::pass_flags::pass_ignore in the
documentation afer Hartmut pointed me to _pass. Unfortunately, while I get
the general idea I still cannot get either my code or the code suggested
above to compile (I am using a 2-day old boost snapshot and g++ 4.4.2).

BTW lex::token_id<> does not exist. I think you meant token_def<> ..

I get an error:

lots of template related noise and then
/usr/local/include/boost/trunk/boost/spirit/home/lex/lexer/lexertl/lexer.hpp:256:
error: 'struct boost::fusion::unused_type' has no member named 'add_action'
which suggests that one cannot assign a semantic action i.e a boost::fusion
expression to a lex::token_def<>  object ...


-Francois



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut Kaiser wrote:


> Could you send a small test case reproducing your problem? It actually
> should work, so you might miss something minor.
>

Hartmut -

I made a trivially simple test case. It compiles only if I remove the
semantic action ( [ pass_= pass_flags::pass_ignore ] ) ...

I attach the code and the compiler output.

-Francois

 
[LexerMinTest.cc]

---

#include <boost/spirit/include/lex_lexertl.hpp>

using namespace std;

using namespace boost::spirit;
using namespace boost::spirit::ascii;

namespace lex_ns = boost::spirit::lex;

template <typename Lexer>
struct my_lexer : boost::spirit::lex::lexer<Lexer>
{
   my_lexer() {

     ws = "[ \t\n]+";

     // THIS DOES NOT WORK  
     this->self += ws[ lex_ns::_pass = lex_ns::pass_flags::pass_ignore];
   
     // THIS WORKS  
     //this->self += ws;

    }

    lex_ns::token_def<> ws;

};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

typedef lex::lexertl::token<>          token_t;
typedef lex::lexertl::lexer<token_t>   lexer_type;
typedef my_lexer<lexer_type>::iterator_type iterator_type;


int main( int argc, char* argv[] )
{
 
  my_lexer<lexer_type>      lexer;            //  lexer

}

---

[LexerMinTest.log]

---

In file included from /usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/partition/../string_token.hpp:11,
                 from /usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/partition/charset.hpp:11,
                 from /usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/generator.hpp:12,
                 from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexertl/lexer.hpp:15,
                 from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer_lexertl.hpp:16,
                 from /usr/local/include/boost-trunk/boost/spirit/include/lex_lexertl.hpp:16,
                 from LexerMinTest.cc:1:
/usr/local/include/boost-trunk/boost/spirit/home/support/detail/lexer/partition/../consts.hpp:29: warning: integer overflow in expression
In file included from /usr/local/include/boost-trunk/boost/spirit/home/lex/lexer_lexertl.hpp:16,
                 from /usr/local/include/boost-trunk/boost/spirit/include/lex_lexertl.hpp:16,
                 from LexerMinTest.cc:1:
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexertl/lexer.hpp: In member function 'void boost::spirit::lex::lexertl::lexer<Token, Iterator, Functor>::add_action(typename Token::id_type, size_t, F) [with F = boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<boost::spirit::lex::pass_flags::enum_type>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >, Token = boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, Iterator = const char*, Functor = boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> >]':
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/action.hpp:47:   instantiated from 'void boost::spirit::lex::action<Subject, Action>::add_actions(LexerDef&) const [with LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >, Subject = boost::spirit::lex::reference<const boost::spirit::lex::token_def<boost::fusion::unused_type, char, long unsigned int>, long unsigned int>, Action = boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<boost::spirit::lex::pass_flags::enum_type>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >]'
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexer.hpp:199:   instantiated from 'void boost::spirit::lex::detail::lexer_def_<LexerDef>::compile2pass(const TokenExpr&) [with TokenExpr = boost::spirit::lex::action<boost::spirit::lex::reference<const boost::spirit::lex::token_def<boost::fusion::unused_type, char, long unsigned int>, long unsigned int>, boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<boost::spirit::lex::pass_flags::enum_type>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > > >, LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >]'
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexer.hpp:207:   instantiated from 'void boost::spirit::lex::detail::lexer_def_<LexerDef>::define(const Expr&) [with Expr = boost::proto::exprns_::expr<boost::proto::tag::subscript, boost::proto::argsns_::list2<boost::spirit::lex::token_def<boost::fusion::unused_type, char, long unsigned int>&, boost::proto::exprns_::expr<boost::proto::tag::terminal, boost::proto::argsns_::term<const boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<boost::spirit::lex::pass_flags::enum_type>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >&>, 0l> >, 2l>, LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >]'
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexer.hpp:273:   instantiated from 'boost::spirit::lex::detail::lexer_def_<LexerDef>& boost::spirit::lex::detail::operator+=(boost::spirit::lex::detail::lexer_def_<LexerDef>&, const Expr&) [with LexerDef = boost::spirit::lex::lexer<boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > > >, Expr = boost::proto::exprns_::expr<boost::proto::tag::subscript, boost::proto::argsns_::list2<boost::spirit::lex::token_def<boost::fusion::unused_type, char, long unsigned int>&, boost::proto::exprns_::expr<boost::proto::tag::terminal, boost::proto::argsns_::term<const boost::phoenix::actor<boost::phoenix::composite<boost::phoenix::assign_eval, boost::fusion::vector<boost::phoenix::argument<2>, boost::phoenix::value<boost::spirit::lex::pass_flags::enum_type>, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> > >&>, 0l> >, 2l>]'
LexerMinTest.cc:18:   instantiated from 'my_lexer<Lexer>::my_lexer() [with Lexer = boost::spirit::lex::lexertl::lexer<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, const char*, boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >, boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<false>, mpl_::bool_<true> > >]'
LexerMinTest.cc:40:   instantiated from here
/usr/local/include/boost-trunk/boost/spirit/home/lex/lexer/lexertl/lexer.hpp:256: error: 'struct boost::fusion::unused_type' has no member named 'add_action'

---

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> > Could you send a small test case reproducing your problem? It
> actually
> > should work, so you might miss something minor.
>
> I made a trivially simple test case. It compiles only if I remove the
> semantic action ( [ pass_= pass_flags::pass_ignore ] ) ...
>
> I attach the code and the compiler output.

The only thing you're missing is that as soon as you need to use semantic
actions you need to replace the

    typedef lex::lexertl::lexer<token_t> lexer_type;

by

    typedef lex::lexertl::actor_lexer<token_t> lexer_type;

then your example compiles just fine.

Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut Kaiser wrote:

> The only thing you're missing is that as soon as you need to use semantic
> actions you need to replace the
>
>     typedef lex::lexertl::lexer<token_t> lexer_type;
>
> by
>
>     typedef lex::lexertl::actor_lexer<token_t> lexer_type;
>
> then your example compiles just fine.
>
>

Hartmut -

Yes ! Thanks!
Just from reading the docs I am not sure I would ever have been able to
figure out that actor_lexer is required, although, after the fact, it is not
entirely surprising.
   
I am getting closer to my objective. Unfortunately when I try to combine
lexer and grammar, my code compile ... but now I get a runtime assertion
error.

I have reduced everything to the simplest possible test case.
The parser attemps to parse a single string of the form

identifier '='  <real> | <int>

The code is attached.

When I try

./ToyParser "var = 1.234"

I get this output :

Parsing string: var = 123
ToyParser: /usr/local/include/boost-
trunk/boost/spirit/home/lex/lexer/token_def.hpp:97: bool
boost::spirit::lex::token_def<Attribute, Char, Idtype>::parse(Iterator&,
const Iterator&, Context&, const Skipper&, Attribute_&) const [with Iterator
=
boost::spirit::lex::lexertl::iterator<boost::spirit::lex::lexertl::functor<boost::spirit::lex::lexertl::token<const
char*, boost::mpl::vector0<mpl_::na>, mpl_::bool_<true> >,
boost::spirit::lex::lexertl::detail::data, const char*, mpl_::bool_<true>,
mpl_::bool_<true> > >, Context =
boost::spirit::context<boost::fusion::cons<boost::fusion::unused_type&,
boost::fusion::nil>, boost::fusion::vector0<void> >, Skipper =
boost::fusion::unused_type, Attribute_ = const boost::fusion::unused_type,
Attribute = std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, Char = char, Idtype = long unsigned int]: Assertion
`std::size_t(~0) != token_state_' failed.
Aborted

Thank you for your patience

-Francois
[ToyParser.cc]

---

//-----------------------------------------------------------------
// ToyParser.cc
// Demonstrates conversional lexing/parsing using boost.spirit 2.1  
// ostiguy@...
//-----------------------------------------------------------------
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_statement.hpp>
#include <boost/spirit/include/phoenix_container.hpp>
#include <string>

namespace lex = boost::spirit::lex;

template <typename Lexer>
struct my_lexer : boost::spirit::lex::lexer<Lexer>
{
  my_lexer() {

     identifier = "[a-zA-Z][_\\.a-zA-Z0-9]*";              
     ws          =    "[ \\t]+";
     real_       = "([0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?)|([-+]?[1-9]+\\.?([eE][-+]?[0-9]+))";
     integer_    = "[1-9][0-9]*";  

     this->self += '=';
     this->self += ws[ lex::_pass = lex::pass_flags::pass_ignore];
   
    }

    lex::token_def<>            ws;
    lex::token_def<std::string> identifier;
    lex::token_def<int>         integer_;
    lex::token_def<double>      real_;

};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

template <typename Iterator>
struct my_grammar : boost::spirit::qi::grammar<Iterator> {
   
  template <typename TokenDef>
  my_grammar( TokenDef const& tok )
      :  my_grammar::base_type(statement) {
       
       using namespace boost::spirit::qi;
   
       statement =  tok.identifier >> '=' >> (tok.integer_ | tok.real_) ;
    }

  boost::spirit::qi::rule<Iterator> statement;
};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

typedef lex::lexertl::token<>                   token_type;
typedef lex::lexertl::actor_lexer<token_type>   lexer_type;
typedef my_lexer<lexer_type>::iterator_type  iterator_type;

#include <iostream>
#include <sstream>

using namespace std;

int main( int argc, char* argv[] )
{
 
  char const* first = argv[1];
  char const* last =  &first[strlen(argv[1])];  

  cout<< "Parsing string: " << argv[1]         << endl;
   
  my_lexer<lexer_type>          lexer;            
  my_grammar<iterator_type>     grammar(lexer);      

   
  bool ok = lex::tokenize_and_parse( first, last, lexer, grammar );

  if ( !ok ) {
    string rest( first,last );
    cout << "Parse failed. \n" << "Stopped at: " << rest << endl;
  }
  else {
    cout << "Success!" << endl;
  }
}

---

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut -

Never mind. I found the error. It works !
I attach the corrected example, for reference.

./ToyParser "var = 123"
Parsing string: var = 123
Success!

Regards

-Francois
[ToyParser.cc]

---

//-----------------------------------------------------------------
// ToyParser.cc
// Demonstrates conversional lexing/parsing using boost.spirit 2.1  
// ostiguy@...
//-----------------------------------------------------------------
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_statement.hpp>
#include <boost/spirit/include/phoenix_container.hpp>
#include <string>

namespace lex = boost::spirit::lex;

template <typename Lexer>
struct my_lexer : boost::spirit::lex::lexer<Lexer>
{
  my_lexer() {

     identifier = "[a-zA-Z][_\\.a-zA-Z0-9]*";              
     ws          =    "[ \\t]+";
     real_       = "([0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?)|([-+]?[1-9]+\\.?([eE][-+]?[0-9]+))";
     integer_    = "[1-9][0-9]*";  

     this->self += identifier;
     this->self += real_;
     this->self += integer_;
     this->self += '=';
     this->self += ws[ lex::_pass = lex::pass_flags::pass_ignore];
   
    }

    lex::token_def<>            ws;
    lex::token_def<std::string> identifier;
    lex::token_def<int>         integer_;
    lex::token_def<double>      real_;

};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

template <typename Iterator>
struct my_grammar : boost::spirit::qi::grammar<Iterator> {
   
  template <typename TokenDef>
  my_grammar( TokenDef const& tok )
      :  my_grammar::base_type(statement) {
       
       using namespace boost::spirit::qi;
   
       statement =  tok.identifier >> '=' >> (tok.integer_ | tok.real_) ;
    }

  boost::spirit::qi::rule<Iterator> statement;
};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

typedef lex::lexertl::token<>                   token_type;
typedef lex::lexertl::actor_lexer<token_type>   lexer_type;
typedef my_lexer<lexer_type>::iterator_type  iterator_type;

#include <iostream>
#include <sstream>

using namespace std;

int main( int argc, char* argv[] )
{
 
  char const* first = argv[1];
  char const* last =  &first[strlen(argv[1])];  

  cout<< "Parsing string: " << argv[1]         << endl;
   
  my_lexer<lexer_type>          lexer;            
  my_grammar<iterator_type>     grammar(lexer);      

   
  bool ok = lex::tokenize_and_parse( first, last, lexer, grammar );

  if ( !ok ) {
    string rest( first,last );
    cout << "Parse failed. \n" << "Stopped at: " << rest << endl;
  }
  else {
    cout << "Success!" << endl;
  }
}

---

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> Never mind. I found the error. It works !
> I attach the corrected example, for reference.
>
> ./ToyParser "var = 123"
> Parsing string: var = 123
> Success!

Cool!

Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general