Lexer Semantic Actions

[Jean-Francois Ostiguy]

In a seperate post, I inquired about how one can set the value attribute of
a token. Here I would like to describe a related problem.

As an exercise, I attempted to assign a semantic action to a token. My lexer
is derived from lexertl::actor_lexer<>. Everything compiles and works if the
action function is not hooked up.

According to the documentation the following signatures are allowed for a
simple c++ semantic action function

 void f (Iterator& start, Iterator& end, pass_flag& matched,
                                         Idtype& id,Context& ctx);
 void f (Iterator& start, Iterator& end, pass_flag& matched, Idtype& id);
 void f (Iterator& start, Iterator& end, pass_flag& matched);
 void f (Iterator& start, Iterator& end);
 void f ();

Ok, so I start by trying to hook up the simplest form i.e. f();

void my_action()
{
  using namespace std;
  cout << "my_action" << endl;
}

as follows:

...
this->self += integer[ &my_action ]
...

This fails. I get a lot of template errors ...the signature is not
recognized and I do not understand why.

Another issue concerns the specific types that should be used.
'pass_flag' and 'Context' are not clearly documented.  As far as I can tell,
'pass_flag' is not a type defined in the lex namespace (or is it ?) .
The doc says that 'Context' is a lexer specific, "unspecified" type ...
Don't I need a specific when writing my semantic access function if want to
access context information ? i.e. which type should be used in the function
signature ? Should/must the semantic action function be a template
function ?

-Francois    



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> In a seperate post, I inquired about how one can set the value
> attribute of
> a token. Here I would like to describe a related problem.
>
> As an exercise, I attempted to assign a semantic action to a token. My
> lexer
> is derived from lexertl::actor_lexer<>. Everything compiles and works
> if the
> action function is not hooked up.
>
> According to the documentation the following signatures are allowed for
> a
> simple c++ semantic action function
>
>  void f (Iterator& start, Iterator& end, pass_flag& matched,
>                                          Idtype& id,Context& ctx);
>  void f (Iterator& start, Iterator& end, pass_flag& matched, Idtype&
> id);
>  void f (Iterator& start, Iterator& end, pass_flag& matched);
>  void f (Iterator& start, Iterator& end);
>  void f ();
>
> Ok, so I start by trying to hook up the simplest form i.e. f();
>
> void my_action()
> {
>   using namespace std;
>   cout << "my_action" << endl;
> }
>
> as follows:
>
> ...
> this->self += integer[ &my_action ]
> ...
>
> This fails. I get a lot of template errors ...the signature is not
> recognized and I do not understand why.
>

That smells like a bug. I'll try to investigate later today.

> Another issue concerns the specific types that should be used.
> 'pass_flag' and 'Context' are not clearly documented.  As far as I can
> tell,
> 'pass_flag' is not a type defined in the lex namespace (or is it ?) .
> The doc says that 'Context' is a lexer specific, "unspecified" type ...
> Don't I need a specific when writing my semantic access function if
> want to
> access context information ? i.e. which type should be used in the
> function
> signature ? Should/must the semantic action function be a template
> function ?

pass_flags is defined as an enumerator (see
boost/spirit/home/lex/lexer/pass_flags.hpp). The ideas was simply not to use
the 5 argument overload for semantic actions as long as you don't need the
context (and the context is deliberately undocumented), or use a template
function/function object:

struct function_obj
{
    template <typename Iterator, typename IdType, typename Context>
    void operator()(Iterator& start, Iterator& end,
        pass_flag& matched, Idtype& id, Context& ctx)
    {
        ctx.set_value(...);
    }
};

this->self = ...[ function_obj() ];

But the preferred way is simply to use a Phoenix expression (like in the
example using the _pass placeholder).

HTH
Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> > Ok, so I start by trying to hook up the simplest form i.e. f();
> >
> > void my_action()
> > {
> >   using namespace std;
> >   cout << "my_action" << endl;
> > }
> >
> > as follows:
> >
> > ...
> > this->self += integer[ &my_action ]
> > ...
> >
> > This fails. I get a lot of template errors ...the signature is not
> > recognized and I do not understand why.
> >
>
> That smells like a bug. I'll try to investigate later today.

Ok, confirmed (and fixed in SVN). I added a new test
lex/semantic_actions.cpp avoiding this kind of problem in the future (and as
an aid of how to use plain functions as lexer semantic actions).

> > Another issue concerns the specific types that should be used.
> > 'pass_flag' and 'Context' are not clearly documented.  As far as I
> can
> > tell,
> > 'pass_flag' is not a type defined in the lex namespace (or is it ?) .
> > The doc says that 'Context' is a lexer specific, "unspecified" type
> ...
> > Don't I need a specific when writing my semantic access function if
> > want to
> > access context information ? i.e. which type should be used in the
> > function
> > signature ? Should/must the semantic action function be a template
> > function ?
>
> pass_flags is defined as an enumerator (see
> boost/spirit/home/lex/lexer/pass_flags.hpp). The ideas was simply not
> to use the 5 argument overload for semantic actions as long as you
> don't need the context (and the context is deliberately undocumented),
> or use a template function/function object:
>
> struct function_obj
> {
>     template <typename Iterator, typename IdType, typename Context>
>     void operator()(Iterator& start, Iterator& end,
>         pass_flag& matched, Idtype& id, Context& ctx)

This actually needs to be:

    template <typename Iterator, typename IdType, typename Context>
    void operator()(Iterator& start, Iterator& end,
        BOOST_SCOPED_ENUM(lex::pass_flags)& pass, Idtype& id,
        Context& ctx)

>     {
>         ctx.set_value(...);
>     }
> };
>
> this->self = ...[ function_obj() ];
>
> But the preferred way is simply to use a Phoenix expression (like in
> the example using the _pass placeholder).

HTH
Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy-2]

Hartmut Kaiser wrote:

> That smells like a bug. I'll try to investigate later today.

Thanks.

> pass_flags is defined as an enumerator (see
> boost/spirit/home/lex/lexer/pass_flags.hpp). The ideas was simply not to
> use the 5 argument overload for semantic actions as long as you don't need
> the context (and the context is deliberately undocumented), or use a
> template function/function object:
>
> struct function_obj
> {
>     template <typename Iterator, typename IdType, typename Context>
>     void operator()(Iterator& start, Iterator& end,
>         pass_flag& matched, Idtype& id, Context& ctx)
>     {
>         ctx.set_value(...);
>     }
> };
>
> this->self = ...[ function_obj() ];
>
> But the preferred way is simply to use a Phoenix expression (like in the
> example using the _pass placeholder).
>

Perhaps I need to invest some time to learn Phoenix. I would really like to
see an example of a semantic action using Phoenix that does just what I
described in my other post: after recognizing the token string
representation "{3,4}" extract 3 and 4 and initialize the corresponding
token value attribute of type Rational using the constructor Rational(3,4).
For now, this is less than obvious to me.  
   
-Francois
 




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> Hartmut Kaiser wrote:
>
> > That smells like a bug. I'll try to investigate later today.
>
> Thanks.
>
>
> > pass_flags is defined as an enumerator (see
> > boost/spirit/home/lex/lexer/pass_flags.hpp). The ideas was simply not
> to
> > use the 5 argument overload for semantic actions as long as you don't
> need
> > the context (and the context is deliberately undocumented), or use a
> > template function/function object:
> >
> > struct function_obj
> > {
> >     template <typename Iterator, typename IdType, typename Context>
> >     void operator()(Iterator& start, Iterator& end,
> >         pass_flag& matched, Idtype& id, Context& ctx)
> >     {
> >         ctx.set_value(...);
> >     }
> > };
> >
> > this->self = ...[ function_obj() ];
> >
> > But the preferred way is simply to use a Phoenix expression (like in
> the
> > example using the _pass placeholder).
> >
>
> Perhaps I need to invest some time to learn Phoenix. I would really
> like to
> see an example of a semantic action using Phoenix that does just what I
> described in my other post: after recognizing the token string
> representation "{3,4}" extract 3 and 4 and initialize the corresponding
> token value attribute of type Rational using the constructor
> Rational(3,4).
> For now, this is less than obvious to me.

I don't think you need Phoenix for just that. As I outlined in the other
response, this can be easily achieve by leveraging the internal conversion
logic in the lexer by specializing the template:

namespace boost { namespace spirit { namespace traits
{
    template <typename Iterator>
    struct assign_to_attribute_from_iterators<rational, Iterator>
    {
        static void
        call(Iterator const& first, Iterator const& last, rational& attr)
        {
            int x, y;
            Iterator b = first;
            qi::parse(b, last,
                '{' >> qi::int_ >> ',' >> qi::int_ >> '}', x, y);
            attr = rational(x, y);
        }
    };
}}}

BTW, if your type rational had a constructor taking two iterators this
wouldn't be needed as the default conversion assumes the type to be
constructible from this pair.

As this template specialization hooks into the internal conversion logic,
all you need to do is to use rational as the type of your token:

    // a lexer recognizing a single token type: rational
    template <typename Lexer>
    struct lex_rational : lex::lexer<Lexer>
    {
        lex_rational()
        {
            this->self.add_pattern("INT", "[1-9][0-9]*");

            rt = "\{{INT},{INT}\}";
            this->self.add(rt);
        }
        lex::token_def<rational> rt;
    };

and, if you later use the token_def 'rt' as a parser it will expose your
rational as its attribute:

    // the token type needs to know the iterator type of the underlying
    // input and the set of used token value types
    typedef lex::lexertl::token<std::string::iterator,
        mpl::vector<rational> > token_type;

    // use actor_lexer<> here if your token definitions have semantic
    // actions
    typedef lex::lexertl::lexer<token_type> lexer_type;

    // this is the iterator exposed by the lexer, we use this for parsing
    typedef lexer_type::iterator_type iterator_type;

    // create a lexer instance
    std::string input("{3,4}");
    lex_rational<lexer_type> lex;
    iterator_type b = lex.begin(input.begin(), input.end());

    // use the embedded token_def as a parser, it exposes its token value
type
    // as its parser attribute type
    rational r;
    qi::parse(b, lex.end(), lex.rt, r);
   
This should succeed and should initialize r from 3 and 4. All of this is
from the top of my head and may contain errors, but the idea is correct.
I'll try to come up with a full example for just that later today.

HTH
Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

 
> This should succeed and should initialize r from 3 and 4. All of this
> is from the top of my head and may contain errors, but the idea is
> correct. I'll try to come up with a full example for just that later
> today.

Done. It's in $BOOST/libs/spirit/example/lex/custom_token_attribute.cpp.

HTH
Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

> I don't think you need Phoenix for just that. As I outlined in the other
> response, this can be easily achieve by leveraging the internal conversion
> logic in the lexer by specializing the template:
>

Harmut. Understood; Phoenix may be an overkill. On the other hand, when I
wrote my post, I did not know about the existence of an interface for
value type conversion ;-)  The example code you give is very illuminating.
I will experiment with it.

Now when you say "you do not need Phoenix for just that", it implies that
one should use Phoenix for more complex tasks. I still think that an example
that illustrates how Pheonix can be used to manipulate the internals
of a token object would have great pedagogical value.

As always, thank you for all your help and patience.
 
-Francois



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut Kaiser wrote:


>> This fails. I get a lot of template errors ...the signature is not
>> recognized and I do not understand why.
>>
>
> That smells like a bug. I'll try to investigate later today.
>

Ok. To simplify you life ...h ere is a simple test case, adapted from one of
you examples. If the action "my_action" is hooked to integer, it does not
compile. If it is removed, it does.
 
-Francois
[LexerActionBug.cc]

---

#include <boost/spirit/include/lex_lexertl.hpp>
#include <iostream>
#include <string>

namespace lex = boost::spirit::lex;

void my_action()
{
  using namespace std;
  cout << " my action" <<endl;
}

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

template <typename Lexer>
struct print_numbers_tokens : lex::lexer<Lexer>
{
    print_numbers_tokens()
      : print_numbers_tokens::base_type()
    {
        using namespace std;
        integer = "[1-9][0-9]*";
        skip    = ".";

        this->self
            =   integer             // OK  
          //    =   integer[&my_action] // THIS FAILS ... WHY ???
            |   skip[lex::_pass = lex::pass_flags::pass_ignore]
            ;
    }
    lex::token_def<int> integer;
    lex::token_def<>    skip;
};

//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
//||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

int main(int argc, char const* argv[] )
{
   typedef lex::lexertl::token<char const*> token_type;
   typedef lex::lexertl::actor_lexer<token_type> lexer_type;
   typedef print_numbers_tokens<lexer_type>::iterator_type iterator_type;

   print_numbers_tokens<lexer_type>  lexer;          // Our lexer
}

---

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Hartmut Kaiser]

> >> This fails. I get a lot of template errors ...the signature is not
> >> recognized and I do not understand why.
> >>
> >
> > That smells like a bug. I'll try to investigate later today.
> >
>
> Ok. To simplify you life ...h ere is a simple test case, adapted from
> one of you examples. If the action "my_action" is hooked to integer, it
> does not compile. If it is removed, it does.

Hmmm, I thought to have fixed this. Did you update from trunk, recently?

Regards Hartmut

-------------------
Meet me at BoostCon
http://boostcon.com




------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Jean-Francois Ostiguy]

Hartmut Kaiser wrote:


>
> Hmmm, I thought to have fixed this. Did you update from trunk, recently?
>

Will try. My current snapshot is from Oct 27.
Thanks
-Francois

 


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Spirit-general mailing list
Spirit-general@...
https://lists.sourceforge.net/lists/listinfo/spirit-general