test-suite / run problem with paths containing spaces

[Martin Dyring-Andersen]

Hello everybody,

I have a Boost.Build based project which is currently being built in Windows, Darwin, Linux (Debian) and FreeBSD. Everything builds smoothly with the exception of some tests..

Given the following directory structure as an example:

$HOME/Projects/TFS/MyProject Name Containing Spaces/jamroot
$HOME/Projects/TFS/MyProject Project Name Containing Spaces/somedynamiclib/jamfile
$HOME/Projects/TFS/MyProject Project Name Containing Spaces/test/jamfile

the test/jamfile contains (simplified):

test-suite test
        : [ run test.cpp /somedynamiclib : : : : ]
;

So, during build I should get a test executable that is capable of loading "somedynamiclib". The result is this on the different platforms:

* Windows works like a charm

* Darwin fails with this error:
/bin/sh: line 1: MyProject: command not found
/bin/sh: line 4:  5573 Trace/BPT trap

* Debian works, but gives the following warning:
/bin/sh: line 1: MyProject: command not found
/bin/sh: line 2: export: `LD_LIBRARY_PATH': not a valid identifier

* FreeBSD works, but gives the following warning:
MyProject: not found


Without digging to much, I assume this is simply an issue of not properly escaping / quoting the paths in the cast of test-suite / run.

I can solve this issue myself by simply renaming the project directory (tested and found working). But since all other parts of the build process seems to work the spaces in the path, I assume this is fixable bug?

As a side note: any progress / plans for a release capable of hashed build directories? I have run into the path length issue on Windows a number of times, for now I have avoided them by using --abbreviate-paths but it feels like postponing the inevitable. ;-)

--
Best regards,
Martin Dyring-Andersen
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus]

On Wednesday 24 June 2009 Martin Dyring-Andersen wrote:

> Hello everybody,
>
> I have a Boost.Build based project which is currently being built in Windows, Darwin, Linux (Debian) and FreeBSD. Everything builds smoothly with the exception of some tests..
>
> Given the following directory structure as an example:
>
> $HOME/Projects/TFS/MyProject Name Containing Spaces/jamroot
> $HOME/Projects/TFS/MyProject Project Name Containing Spaces/somedynamiclib/jamfile
> $HOME/Projects/TFS/MyProject Project Name Containing Spaces/test/jamfile
>
> the test/jamfile contains (simplified):
>
> test-suite test
> : [ run test.cpp /somedynamiclib : : : : ]
> ;
>
> So, during build I should get a test executable that is capable of loading "somedynamiclib". The result is this on the different platforms:
>
> * Windows works like a charm
>
> * Darwin fails with this error:
> /bin/sh: line 1: MyProject: command not found
> /bin/sh: line 4:  5573 Trace/BPT trap
>
> * Debian works, but gives the following warning:
> /bin/sh: line 1: MyProject: command not found
> /bin/sh: line 2: export: `LD_LIBRARY_PATH': not a valid identifier

Can you update Boost.Build from SVN and try again? I've just fixed this. If you
were using some system version of Boost.Jam, you'd also need to build Boost.Jam
from SVN. If you were using 1.39 version of Boost.Jam, it's good enough.

> As a side note: any progress / plans for a release capable of hashed build directories?
> I have run into the path length issue on Windows a number of times, for now I have
> avoided them by using --abbreviate-paths but it feels like postponing the inevitable. ;-)

There are no concrete plans at present. If you, or somebody else, find me a suitably
licensed SH1 or MD5 implementation, it would be a big help :-)

- Volodya
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Martin Dyring-Andersen]

Hi Vladimir,

Thank you for the quick reply!

I'm using bjam built from 1.39 sources, bjam --version outputs:
Boost.Build V2 (Milestone 12)
Boost.Jam 03.1.17

> Can you update Boost.Build from SVN and try again? I've just fixed this. If
> you
> were using some system version of Boost.Jam, you'd also need to build
> Boost.Jam
> from SVN. If you were using 1.39 version of Boost.Jam, it's good enough.

Here is an update when latest Boost.Build from SVN:

1) Windows still works

2) Debian still gives warning (but runs test successfully):
/bin/sh: line 1: MyProject: command not found
/bin/sh: line 2: export: `LD_LIBRARY_PATH': not a valid identifier

3) FreeBSD now works without warning

4) Darwin continues to fail. This is dumped (rewritten for clarity):

DYLD_LIBRARY_PATH="$HOME/Projects/TFS/MyProject Name Containing Spaces/trunk/stage/somedynamiclib/darwin-4.0.1/release/address-model-32/architecture-x86/threading-multi:$DYLD_LIBRARY_PATH
export DYLD_LIBRARY_PATH"

It looks to me like something is wrong with the quoting. If my understanding is correct (I'm a Windows guy :-), it should look like this:

DYLD_LIBRARY_PATH="'$HOME/Projects/TFS/MyProject Name Containing Spaces/trunk/stage/somedynamiclib/darwin-4.0.1/release/address-model-32/architecture-x86/threading-multi':$DYLD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH

.. though I must admit that I am a bit confused about how the end quote turned up after export DYLD_LIBRARY_PATH" in the original dump.

> > As a side note: any progress / plans for a release capable of hashed build
> directories?
> > I have run into the path length issue on Windows a number of times, for now
> I have
> > avoided them by using --abbreviate-paths but it feels like postponing the
> inevitable. ;-)
>
> There are no concrete plans at present. If you, or somebody else, find me a suitably licensed SH1 or MD5 implementation, it would be a big help :-)

Actually I was going to suggest either using the Boost CRC library (since I don't think it is actually necessary with a crypto-safe hash anyway?) or, alternatively, use the Windows Wide/Unicode filesystem functions to support the long paths.

As I understand it, this is mainly an issue on Windows - so it maybe it would be better to preserve the existing behaviour on all platforms and simply fix the Windows path limit issue. I would be happy to help with this if you can point me in the right direction.

--
Best regards,
Martin Dyring-Andersen
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus-3]

Martin Dyring-Andersen wrote:

> Hi Vladimir,
>
> Thank you for the quick reply!
>
> I'm using bjam built from 1.39 sources, bjam --version outputs:
> Boost.Build V2 (Milestone 12)
> Boost.Jam 03.1.17
>
>> Can you update Boost.Build from SVN and try again? I've just fixed this. If
>> you
>> were using some system version of Boost.Jam, you'd also need to build
>> Boost.Jam
>> from SVN. If you were using 1.39 version of Boost.Jam, it's good enough.
>
> Here is an update when latest Boost.Build from SVN:
>
> 1) Windows still works
>
> 2) Debian still gives warning (but runs test successfully):
> /bin/sh: line 1: MyProject: command not found
> /bin/sh: line 2: export: `LD_LIBRARY_PATH': not a valid identifier
>
> 3) FreeBSD now works without warning
>
> 4) Darwin continues to fail. This is dumped (rewritten for clarity):
>
> DYLD_LIBRARY_PATH="$HOME/Projects/TFS/MyProject Name Containing
>

Spaces/trunk/stage/somedynamiclib/darwin-4.0.1/release/address-model-32/architecture-x86/threading-multi
$DYLD_LIBRARY_PATH
> export DYLD_LIBRARY_PATH"
>
> It looks to me like something is wrong with the quoting. If my understanding is correct (I'm a
> Windows guy :-), it should look like this:
>
> DYLD_LIBRARY_PATH="'$HOME/Projects/TFS/MyProject Name Containing
>
Spaces/trunk/stage/somedynamiclib/darwin-4.0.1/release/address-model-32/architecture-x86/threading-multi'
$DYLD_LIBRARY_PATH"
> export DYLD_LIBRARY_PATH
>
> .. though I must admit that I am a bit confused about how the end quote turned up after export
> DYLD_LIBRARY_PATH" in the original dump.

Doh! Can you update and try again? I've put quote in a wrong place.

>> There are no concrete plans at present. If you, or somebody else, find me a suitably licensed SH1
>> or MD5 implementation, it would be a big help :-)
>
> Actually I was going to suggest either using the Boost CRC library (since I don't think it is
> actually necessary with a crypto-safe hash anyway?)

Well, Boost CRC is probably not strong enough for this purpose :-)

> or, alternatively, use the Windows
> Wide/Unicode filesystem functions to support the long paths.

Is there some alternative API that does not have path length limits?

> As I understand it, this is mainly an issue on Windows - so it maybe it would be better to
> preserve the existing behaviour on all platforms and simply fix the Windows path limit issue. I
> would be happy to help with this if you can point me in the right direction.

Well, we'd rather have hashed directory names as option everywhere, since long
paths are inconvenient on linux as well.

- Volodya


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Martin Dyring-Andersen]

Hi Vladimir,

> > .. though I must admit that I am a bit confused about how the end quote
> turned up after export
> > DYLD_LIBRARY_PATH" in the original dump.
>
> Doh! Can you update and try again? I've put quote in a wrong place.

Thanks a lot - now tests run perfectly on all platforms!

> > Actually I was going to suggest either using the Boost CRC library (since I
> don't think it is
> > actually necessary with a crypto-safe hash anyway?)
>
> Well, Boost CRC is probably not strong enough for this purpose :-)

Just trying to keep it simple. :-)

I googled a bit for CRC collision rates. This page seems interesting and concludes that chances of a CRC64 collision is extremely low (at least for that class of input data):
http://apollo.backplane.com/matt/crc64.html

So maybe it is not such a bad idea after all.

As for a free md5 implementation, IANAL, but this puppy seems to fit licensing wise:
http://sourceforge.net/project/showfiles.php?group_id=42360

> > or, alternatively, use the Windows
> > Wide/Unicode filesystem functions to support the long paths.
>
> Is there some alternative API that does not have path length limits?

There is a limit, but it is high enough that it should be sufficient: 32,767 characters according to http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx

I couldn't find a definitive answer for all *nixes, but the limit seems to be around 4096 characters.

> > As I understand it, this is mainly an issue on Windows - so it maybe it
> would be better to
> > preserve the existing behaviour on all platforms and simply fix the Windows
> path limit issue. I
> > would be happy to help with this if you can point me in the right direction.
>
> Well, we'd rather have hashed directory names as option everywhere, since long
> paths are inconvenient on linux as well.

It would be great to support both options, since I guess that long paths will continue to cause problems with used in shell scripts etc.

Thanks again for your help on the original issue. :-)

--
Best regards,
Martin Dyring-Andersen
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus]

On Wednesday 24 June 2009 Martin Dyring-Andersen wrote:

Hi Martin,

>
> Thanks a lot - now tests run perfectly on all platforms!

Great.

>
> > > Actually I was going to suggest either using the Boost CRC library (since I
> > don't think it is
> > > actually necessary with a crypto-safe hash anyway?)
> >
> > Well, Boost CRC is probably not strong enough for this purpose :-)
>
> Just trying to keep it simple. :-)
>
> I googled a bit for CRC collision rates. This page seems interesting and concludes that chances of a CRC64 collision is extremely low (at least for that class of input data):
> http://apollo.backplane.com/matt/crc64.html
>
> So maybe it is not such a bad idea after all.

Well, another problem is that using Boost library is too heavy a dependency :-)

> As for a free md5 implementation, IANAL, but this puppy seems to fit licensing wise:
> http://sourceforge.net/project/showfiles.php?group_id=42360

Looks good -- I did not run into this implementation before. Since we're gonna
need MD5 in Boost.Jam anyway, I've integrated this code. If you grab Boost.Jam
from SVN, and also apply the attached Boost.Build patch, all the target paths will
start using MD5 hashes. I did not commit Boost.Build part, because it needs
some work -- in particular a way to explicitly request MD5 paths from command
line, and some mechanism to report which properties a given MD5 path correspond
to -- but the patch I've posted should be enough for experimentation.

> > > or, alternatively, use the Windows
> > > Wide/Unicode filesystem functions to support the long paths.
> >
> > Is there some alternative API that does not have path length limits?
>
> There is a limit, but it is high enough that it should be sufficient:
> 32,767 characters according to http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx

Thanks. I'll keep this in mind.
 
- Volodya

[md5.diff]

---

Index: build/property-set.jam
===================================================================
--- build/property-set.jam (revision 53703)
+++ build/property-set.jam (working copy)
@@ -262,6 +262,7 @@
             else
             {
                 local p = [ as-path ] ;
+                p = [ MD5 $(p) ] ;
                 # A real ugly hack. Boost regression test system requires
                 # specific target paths, and it seems that changing it to handle
                 # other directory layout is really hard. For that reason, we

---

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Martin Dyring-Andersen]

> -----Original Message-----

[using CRC64 for build output folders]
> > So maybe it is not such a bad idea after all.
>
> Well, another problem is that using Boost library is too heavy a dependency :-)

Good point.. ;-)

> > As for a free md5 implementation, IANAL, but this puppy seems to fit
> licensing wise:
> > http://sourceforge.net/project/showfiles.php?group_id=42360
>
> Looks good -- I did not run into this implementation before. Since we're gonna
> need MD5 in Boost.Jam anyway, I've integrated this code. If you grab Boost.Jam
> from SVN, and also apply the attached Boost.Build patch, all the target paths
> will start using MD5 hashes. I did not commit Boost.Build part, because it
> needs some work -- in particular a way to explicitly request MD5 paths from
> command line, and some mechanism to report which properties a given MD5 path
> correspond to -- but the patch I've posted should be enough for
> experimentation.

Just did a quick test on Windows, it works like a charm! Thanks.

--
Best regards,
Martin Dyring-Andersen

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build