Pybindgen error ==> Buildbot errors

[Faker Moatamri]

Hi Gustavo and all,
The builds are breaking because of an error generating the pybindgen
version:

error: Could not load the tool 'cflags' in ['/home/buildslave/slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/
wafadmin/Tools', '/home/buildslave/slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin', '/home/buildslave/
slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin/Tools', '/home/buildslave/slave/full-rahan-g++-4.4.0/build',
'/usr/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg', '/usr/lib64/python25.zip', '/usr/lib64/python2.5', '/usr/lib64/python2.5/plat-linux2',
'/usr/lib64/python2.5/lib-tk', '/usr/lib64/python2.5/lib-dynload', '/usr/lib64/python2.5/site-packages', '/usr/lib64/python2.5/site-packages/Numeric',
'/usr/lib64/python2.5/site-packages/PIL', '/usr/lib64/python2.5/site-packages/gst-0.10', '/usr/lib64/python2.5/site-packages/gtk-2.0',
'/usr/lib/python2.5/site-packages']


If I copy the file cflags.py from ns-3-dev/waf-tools to

/home/buildslave/slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/
wafadmin/Tools

this error disappears and another one appears:

error: Could not load the tool 'command' in
['.../.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin',
'.../.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin/Tools',
'/auto/sop-nas2a/u/sop-nas2a/vol/home_planete/fmoatamr/src/ns-3-dev',
'/usr/lib/python2.5/site-packages/ReviewBoard-1.0beta2-py2.5.egg',
'/usr/lib/python2.5/site-packages/flup-1.0.1-py2.5.egg',
'/usr/lib/python2.5/site-packages/Pygments-1.0-py2.5.egg',
'/usr/lib/python2.5/site-packages/Djblets-0.5beta1-py2.5.egg',
'/usr/lib/python2.5/site-packages/django_evolution-0.0.0-py2.5.egg',
'/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg',
'/usr/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg',
'/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg',
'/usr/lib/python2.5/site-packages/zope.interface-3.5.1-py2.5-linux-x86_64.egg',
'/usr/lib64/python25.zip', '/usr/lib64/python2.5',
'/usr/lib64/python2.5/plat-linux2', '/usr/lib64/python2.5/lib-tk',
'/usr/lib64/python2.5/lib-dynload',
'/usr/lib64/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages/Numeric',
'/usr/lib64/python2.5/site-packages/PIL',
'/usr/lib64/python2.5/site-packages/gst-0.10',
'/usr/lib64/python2.5/site-packages/gtk-2.0',
'/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages']

I guess that the file ns-3-dev/waf-tools/command.py should be copied to
some place also but why it doesn't work automatically?
Thanks
Faker Moatamri

[Gustavo Carneiro]

2009/10/26 Faker Moatamri <faker.moatamri@...>

> Hi Gustavo and all,
> The builds are breaking because of an error generating the pybindgen
> version:
>
> error: Could not load the tool 'cflags' in
> ['/home/buildslave/slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/
> wafadmin/Tools',
> '/home/buildslave/slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin',
> '/home/buildslave/
> slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin/Tools',
> '/home/buildslave/slave/full-rahan-g++-4.4.0/build',
> '/usr/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg',
> '/usr/lib64/python25.zip', '/usr/lib64/python2.5',
> '/usr/lib64/python2.5/plat-linux2', '/usr/lib64/python2.5/lib-tk',
> '/usr/lib64/python2.5/lib-dynload', '/usr/lib64/python2.5/site-packages',
> '/usr/lib64/python2.5/site-packages/Numeric',
> '/usr/lib64/python2.5/site-packages/PIL',
> '/usr/lib64/python2.5/site-packages/gst-0.10',
> '/usr/lib64/python2.5/site-packages/gtk-2.0',
> '/usr/lib/python2.5/site-packages']
>
>
> If I copy the file cflags.py from ns-3-dev/waf-tools to
>
>
> /home/buildslave/slave/full-rahan-g++-4.4.0/build/.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/
> wafadmin/Tools
>
> this error disappears and another one appears:
>
> error: Could not load the tool 'command' in
> ['.../.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin',
> '.../.waf-1.5.9-d1e0349fc8937631a656fb8ea7e99063/wafadmin/Tools',
> '/auto/sop-nas2a/u/sop-nas2a/vol/home_planete/fmoatamr/src/ns-3-dev',
> '/usr/lib/python2.5/site-packages/ReviewBoard-1.0beta2-py2.5.egg',
> '/usr/lib/python2.5/site-packages/flup-1.0.1-py2.5.egg',
> '/usr/lib/python2.5/site-packages/Pygments-1.0-py2.5.egg',
> '/usr/lib/python2.5/site-packages/Djblets-0.5beta1-py2.5.egg',
> '/usr/lib/python2.5/site-packages/django_evolution-0.0.0-py2.5.egg',
> '/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg',
> '/usr/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg',
> '/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg',
> '/usr/lib/python2.5/site-packages/zope.interface-3.5.1-py2.5-linux-x86_64.egg',
> '/usr/lib64/python25.zip', '/usr/lib64/python2.5',
> '/usr/lib64/python2.5/plat-linux2', '/usr/lib64/python2.5/lib-tk',
> '/usr/lib64/python2.5/lib-dynload', '/usr/lib64/python2.5/site-packages',
> '/usr/lib64/python2.5/site-packages/Numeric',
> '/usr/lib64/python2.5/site-packages/PIL',
> '/usr/lib64/python2.5/site-packages/gst-0.10',
> '/usr/lib64/python2.5/site-packages/gtk-2.0',
> '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages']
>
> I guess that the file ns-3-dev/waf-tools/command.py should be copied to
> some place also but why it doesn't work automatically?
>

I'm sorry, pybindgen simply was not designed to be used like this.  It does
not come with a waf script because it assumes developers can manage to get
waf (which is not the upstream waf, it's a mini-fork), and in the
distribution tarballs waf is included for end users.

You could try to copy the whole folder, waf-tools, into pybindgen.  Or you
could just grab waf from a pybindgen 0.12 release and copy/use that one.

The way Mathieu tried to work around the lack of waf was to call the ns-3
waf.  But the ns-3 waf is no longer self-contained and requires the
waf-tools now.  Unfortunately these waf tools are discovered by a relative
path which no longer works when calling waf from another directory.


In ns-3-allinone a similar problem could have existed, but I took a
different path:

  1. Peek into $ns3_branch/bindings/python/wscript, see what pybindgen
version is required

  2. Write the required version into $pybindgen/pybindgen/version.py

This way you don't even need to invoke waf in pybindgen, since pybindgen
does not need to be compiled.

I hope this helps.  Regards,

--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

[Faker Moatamri]

>
> I'm sorry, pybindgen simply was not designed to be used like this.  It
> does not come with a waf script because it assumes developers can
> manage to get waf (which is not the upstream waf, it's a mini-fork),
> and in the distribution tarballs waf is included for end users.
>
> You could try to copy the whole folder, waf-tools, into pybindgen.  Or
> you could just grab waf from a pybindgen 0.12 release and copy/use
> that one.
>

This didn't work :-(
> The way Mathieu tried to work around the lack of waf was to call the
> ns-3 waf.  But the ns-3 waf is no longer self-contained and requires
> the waf-tools now.  Unfortunately these waf tools are discovered by a
> relative path which no longer works when calling waf from another
> directory.
>
The third solution would be to use the waf that comes with linux distro.
I tried to do that, it worked on my machine but not on the slave
machine. It generates an error:
Traceback (most recent call last):
  File "/usr/bin/waf", line 141, in <module>
    Scripting.prepare()
  File "/usr/share/waf/wafadmin/Scripting.py", line 236, in prepare
    Utils.set_main_module(os.path.join(candidate, WSCRIPT_FILE))
  File "/usr/share/waf/wafadmin/Utils.py", line 138, in set_main_module
    g_module = load_module(file_path, 'wscript_main')
  File "/usr/share/waf/wafadmin/Utils.py", line 127, in load_module
    exec file in module.__dict__
  File "/home/buildslave/slave/full-rahan-g++-4.2.4/pybindgen/wscript",
line 10, in <module>
    import Logs
ImportError: No module named Logs
I didn't find anything that would solve this error, can you please tell
me what package will contain this module Logs?

>
> In ns-3-allinone a similar problem could have existed, but I took a
> different path:
>
>   1. Peek into $ns3_branch/bindings/python/wscript, see what pybindgen
> version is required
>
>   2. Write the required version into $pybindgen/pybindgen/version.py
>
> This way you don't even need to invoke waf in pybindgen, since
> pybindgen does not need to be compiled.

in that case I would need to change the code in ns-3-dev which I think
is not proper since the version has to be generated by PyBindgen itself.
In fact, when reading the code in ns-3-dev, what we do is to read the
version.py file in pybindgen/pybindgen and compare it to the version
that we require in ns-3-dev. If the file pybindgen/pybindgen/version.py
is itself written from ns-3-dev, than comparing is useless since we
write the version by ourselves. So I prefer to change the generate the
version file from pybindgen itself.
>
> I hope this helps.  Regards,
>
> --
> Gustavo J. A. M. Carneiro
> INESC Porto, Telecommunications and Multimedia Unit
> "The universe is always one step beyond logic." -- Frank Herbert

Thanks
best regards
Faker Moatamri

[Gustavo Carneiro]

2009/10/26 Faker Moatamri <faker.moatamri@...>

>
>
>> I'm sorry, pybindgen simply was not designed to be used like this.  It
>> does not come with a waf script because it assumes developers can manage to
>> get waf (which is not the upstream waf, it's a mini-fork), and in the
>> distribution tarballs waf is included for end users.
>>
>> You could try to copy the whole folder, waf-tools, into pybindgen.  Or you
>> could just grab waf from a pybindgen 0.12 release and copy/use that one.
>>
>>  This didn't work :-(

I'm pretty sure that at least dropping waf copied from pybindgen 0.12 _has_
to work, otherwise how would pybindgen releases be able to work at all for
end users?  You must be doing something wrong.  Please keep in mind there
are many binaries out there called 'waf' and containing very different waf
versions, even incompatible ones.  pybindgen bazaar branches will work with
waf from pybindgen 0.12 or from ns-3.6, but not any other waf.

>
>  The way Mathieu tried to work around the lack of waf was to call the ns-3
>> waf.  But the ns-3 waf is no longer self-contained and requires the
>> waf-tools now.  Unfortunately these waf tools are discovered by a relative
>> path which no longer works when calling waf from another directory.
>>
>>  The third solution would be to use the waf that comes with linux distro.
> I tried to do that, it worked on my machine but not on the slave machine. It
> generates an error:
> Traceback (most recent call last):
>  File "/usr/bin/waf", line 141, in <module>
>   Scripting.prepare()
>  File "/usr/share/waf/wafadmin/Scripting.py", line 236, in prepare
>   Utils.set_main_module(os.path.join(candidate, WSCRIPT_FILE))
>  File "/usr/share/waf/wafadmin/Utils.py", line 138, in set_main_module
>   g_module = load_module(file_path, 'wscript_main')
>  File "/usr/share/waf/wafadmin/Utils.py", line 127, in load_module
>   exec file in module.__dict__
>  File "/home/buildslave/slave/full-rahan-g++-4.2.4/pybindgen/wscript", line
> 10, in <module>
>   import Logs
> ImportError: No module named Logs
> I didn't find anything that would solve this error, can you please tell me
> what package will contain this module Logs?

The system installed version is not compatible.  Just forget about the
system installed version, WAF APIs have been very unstable in the past.

>
>
>> In ns-3-allinone a similar problem could have existed, but I took a
>> different path:
>>
>>  1. Peek into $ns3_branch/bindings/python/wscript, see what pybindgen
>> version is required
>>
>>  2. Write the required version into $pybindgen/pybindgen/version.py
>>
>> This way you don't even need to invoke waf in pybindgen, since pybindgen
>> does not need to be compiled.
>>
> in that case I would need to change the code in ns-3-dev which I think is
> not proper since the version has to be generated by PyBindgen itself. In
> fact, when reading the code in ns-3-dev, what we do is to read the
> version.py file in pybindgen/pybindgen and compare it to the version that we
> require in ns-3-dev. If the file pybindgen/pybindgen/version.py is itself
> written from ns-3-dev, than comparing is useless since we write the version
> by ourselves. So I prefer to change the generate the version file from
> pybindgen itself.

I was talking about the code in ns-3-allinone/download.py, not in ns-3-dev.
I think it is reasonable that we generate the version file ourselves, since
we have already downloading (via bzr) that explicit version.  If we
explicitly download version X.Y.Z.W, it should be safe to manually write
that version into pybindgen/version.py.

--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

[Faker Moatamri]

> I'm pretty sure that at least dropping waf copied from pybindgen 0.12
> _has_
> to work, otherwise how would pybindgen releases be able to work at all for
> end users?  You must be doing something wrong.  Please keep in mind there
> are many binaries out there called 'waf' and containing very different waf
> versions, even incompatible ones.  pybindgen bazaar branches will work
> with
> waf from pybindgen 0.12 or from ns-3.6, but not any other waf.
>
I will verify that
>

>> ImportError: No module named Logs
>> I didn't find anything that would solve this error, can you please tell
>> me
>> what package will contain this module Logs?
>
>
> The system installed version is not compatible.  Just forget about the
> system installed version, WAF APIs have been very unstable in the past.
>
>

ok
>>

>> in that case I would need to change the code in ns-3-dev which I think
>> is
>> not proper since the version has to be generated by PyBindgen itself. In
>> fact, when reading the code in ns-3-dev, what we do is to read the
>> version.py file in pybindgen/pybindgen and compare it to the version
>> that we
>> require in ns-3-dev. If the file pybindgen/pybindgen/version.py is
>> itself
>> written from ns-3-dev, than comparing is useless since we write the
>> version
>> by ourselves. So I prefer to change the generate the version file from
>> pybindgen itself.
>
>
> I was talking about the code in ns-3-allinone/download.py, not in
> ns-3-dev.
> I think it is reasonable that we generate the version file ourselves,
> since
> we have already downloading (via bzr) that explicit version.  If we
> explicitly download version X.Y.Z.W, it should be safe to manually write
> that version into pybindgen/version.py.
>

Actually I need it to work with buildbot which builds from ns-3-dev, yes
we can generate the version ourselves but in that case it is useless to
check the version against the one in pybindgen. I guess at this point that
the easiest solution would be to take a compatible waf from ns-3.6 and
install it on the build slave and call it from buildbot just to generate
the version. Otherwise the solution will need much more work. I should
generate the version.py file from buildbot.
Thanks for help
Best regards
Faker moatamri
> --
> Gustavo J. A. M. Carneiro
> INESC Porto, Telecommunications and Multimedia Unit
> "The universe is always one step beyond logic." -- Frank Herbert
>

[Faker Moatamri]

Faker.Moatamri@... wrote:
>
> Actually I need it to work with buildbot which builds from ns-3-dev, yes
> we can generate the version ourselves but in that case it is useless to
> check the version against the one in pybindgen. I guess at this point that
> the easiest solution would be to take a compatible waf from ns-3.6 and
> install it on the build slave and call it from buildbot just to generate
> the version.
This solution worked very well :-) .
Thanks Gustavo
Best regards
Faker Moatamri

[Mathieu Lacage]

On Mon, 2009-10-26 at 18:02 +0000, Gustavo Carneiro wrote:

> >> I'm sorry, pybindgen simply was not designed to be used like this.  It
> >> does not come with a waf script because it assumes developers can manage to
> >> get waf (which is not the upstream waf, it's a mini-fork), and in the
> >> distribution tarballs waf is included for end users.
> >>
> >> You could try to copy the whole folder, waf-tools, into pybindgen.  Or you
> >> could just grab waf from a pybindgen 0.12 release and copy/use that one.
> >>
> >>  This didn't work :-(
>
>
> I'm pretty sure that at least dropping waf copied from pybindgen 0.12 _has_
> to work, otherwise how would pybindgen releases be able to work at all for
> end users?  You must be doing something wrong.  Please keep in mind there
> are many binaries out there called 'waf' and containing very different waf
> versions, even incompatible ones.  pybindgen bazaar branches will work with
> waf from pybindgen 0.12 or from ns-3.6, but not any other waf.

To make that less painful for users (including me: I just got bitten by
the fact that the waf required by pybindgen is incompatible with the waf
required by ns-3-dev), please, could you store in your bazaar repo a
copy of the _right_ waf binary just like we do in ns-3 ? Please ?

Mathieu

[Mathieu Lacage]

On Fri, 2009-11-06 at 16:46 +0100, Mathieu Lacage wrote:

> On Mon, 2009-10-26 at 18:02 +0000, Gustavo Carneiro wrote:
>
> > >> I'm sorry, pybindgen simply was not designed to be used like this.  It
> > >> does not come with a waf script because it assumes developers can manage to
> > >> get waf (which is not the upstream waf, it's a mini-fork), and in the
> > >> distribution tarballs waf is included for end users.
> > >>
> > >> You could try to copy the whole folder, waf-tools, into pybindgen.  Or you
> > >> could just grab waf from a pybindgen 0.12 release and copy/use that one.
> > >>
> > >>  This didn't work :-(
> >
> >
> > I'm pretty sure that at least dropping waf copied from pybindgen 0.12 _has_
> > to work, otherwise how would pybindgen releases be able to work at all for
> > end users?  You must be doing something wrong.  Please keep in mind there
> > are many binaries out there called 'waf' and containing very different waf
> > versions, even incompatible ones.  pybindgen bazaar branches will work with
> > waf from pybindgen 0.12 or from ns-3.6, but not any other waf.
>
> To make that less painful for users (including me: I just got bitten by
> the fact that the waf required by pybindgen is incompatible with the waf
> required by ns-3-dev), please, could you store in your bazaar repo a
> copy of the _right_ waf binary just like we do in ns-3 ? Please ?

I should have pointed out that I did try your suggestion of taking waf
from pybindgen 0.12 or ns-3.6: none of them worked and ns-3-dev appears
unable to work with pybindgen 0.12 (--with-pybindgen=../pybindgen-0.12
seems to have no effect on pybindgen detection). How are we supposed to
use pybindgen with ns-3-dev ?

Mathieu

[Gustavo Carneiro]

2009/11/6 Mathieu Lacage <mathieu.lacage@...>

> On Fri, 2009-11-06 at 16:46 +0100, Mathieu Lacage wrote:
> > On Mon, 2009-10-26 at 18:02 +0000, Gustavo Carneiro wrote:
> >
> > > >> I'm sorry, pybindgen simply was not designed to be used like this.
>  It
> > > >> does not come with a waf script because it assumes developers can
> manage to
> > > >> get waf (which is not the upstream waf, it's a mini-fork), and in
> the
> > > >> distribution tarballs waf is included for end users.
> > > >>
> > > >> You could try to copy the whole folder, waf-tools, into pybindgen.
>  Or you
> > > >> could just grab waf from a pybindgen 0.12 release and copy/use that
> one.
> > > >>
> > > >>  This didn't work :-(
> > >
> > >
> > > I'm pretty sure that at least dropping waf copied from pybindgen 0.12
> _has_
> > > to work, otherwise how would pybindgen releases be able to work at all
> for
> > > end users?  You must be doing something wrong.  Please keep in mind
> there
> > > are many binaries out there called 'waf' and containing very different
> waf
> > > versions, even incompatible ones.  pybindgen bazaar branches will work
> with
> > > waf from pybindgen 0.12 or from ns-3.6, but not any other waf.
> >
> > To make that less painful for users (including me: I just got bitten by
> > the fact that the waf required by pybindgen is incompatible with the waf
> > required by ns-3-dev), please, could you store in your bazaar repo a
> > copy of the _right_ waf binary just like we do in ns-3 ? Please ?
>
> I should have pointed out that I did try your suggestion of taking waf
> from pybindgen 0.12 or ns-3.6: none of them worked and ns-3-dev appears
> unable to work with pybindgen 0.12 (--with-pybindgen=../pybindgen-0.12
> seems to have no effect on pybindgen detection). How are we supposed to
> use pybindgen with ns-3-dev ?
>

You did something wrong; waf from pybindgen 0.12 works. I just checked.  The
error you are seeing is probably ns-3 requiring not version 0.12.0 but some
later version that wasn't released yet.

OK, if this is causing so much problem, I am not a unreasonable person.  WAF
was added to the pybindgen repository.

--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert