specifying libraries on the link line, was "incorrect library names embedded on hp-ux ia64"

[Jim Gallagher]

OK, I have learned more over at the HP IT Resource Center. On hp-ux, a
dependent shared lib supplied to the linker by a path will have the
supplied path embedded as the dependent library name
(<http://docs.hp.com/en/9482/linkertasks.htm#notrecordinglinktime>).
This would appear to be a good thing only if you are linking against
libraries that are in their final installed location and you supply
the full path. Do other unix variants work the same?

So, how can I make boost-build tell the linker about dependent libs
the normal unix way, using -L<search dir> -l<lib name>? This is a
pretty big problem for me. I may have to abandon boost-build for this
project if I cannot get these embedded paths out of my libraries.

Thanks,
Jim
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus]

On Thursday 11 June 2009 Jim Gallagher wrote:

> OK, I have learned more over at the HP IT Resource Center. On hp-ux, a
> dependent shared lib supplied to the linker by a path will have the
> supplied path embedded as the dependent library name
> (<http://docs.hp.com/en/9482/linkertasks.htm#notrecordinglinktime>).
> This would appear to be a good thing only if you are linking against
> libraries that are in their final installed location and you supply
> the full path. Do other unix variants work the same?

Usual unix variants have a special option to make a shared library
contain 'soname'. If a executable is linking to a shared library,
it will reference it via 'soname', not the name specified on the link
line. Can you figure of HPUX has something like that>

>
> So, how can I make boost-build tell the linker about dependent libs
> the normal unix way, using -L<search dir> -l<lib name>?

It's not possible now. I can change to do it this way, but please check
if some soname equivalent is available.

- Volodya
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Jim Gallagher]

>> OK, I have learned more over at the HP IT Resource Center. On hp-ux, a
>> dependent shared lib supplied to the linker by a path will have the
>> supplied path embedded as the dependent library name
>> (<http://docs.hp.com/en/9482/linkertasks.htm#notrecordinglinktime>).
>> This would appear to be a good thing only if you are linking against
>> libraries that are in their final installed location and you supply
>> the full path. Do other unix variants work the same?
>
>Usual unix variants have a special option to make a shared library
>contain 'soname'. If a executable is linking to a shared library,
>it will reference it via 'soname', not the name specified on the link
>line. Can you figure of HPUX has something like that>

Yes, I did run across that. On hp-ux it is +h<internal name>. However,
it doesn't seem to work when multiple libs are involved. For example,
if I have three libs, foo, bar and baz, bar depends on foo and baz depends
on bar, and I add appropriate build requirements using +h:
   <toolset>acc:<linkflags>-Wl,+h,libfoo.so

Libbar links fine, but libbaz does not:

$ for DIR in foo bar baz; do^Jcd $DIR^Jbjam clean^Jbjam -d+2^Jcd ..^Jdone
...found 1 target...
...updating 1 target...
common.Clean clean
...updated 1 target...
acc.compile.c bin/acc/debug/foo.o

    aCC -Ae -c    +d +Z -g   -o "bin/acc/debug/foo.o" "foo.c"

acc.link.dll bin/acc/debug/libfoo.so

    aCC -AA -b -g -Wl,+h,libfoo.so -o "bin/acc/debug/libfoo.so"
"bin/acc/debug/foo.o"

...found 1 target...
...updating 1 target...
common.Clean clean
...updated 1 target...
acc.compile.c bin/acc/debug/bar.o

    aCC -Ae -c    +d +Z -g -I"../foo"  -o "bin/acc/debug/bar.o" "bar.c"

acc.link.dll bin/acc/debug/libbar.so

    aCC -AA -b -g -Wl,+h,libbar.so -o "bin/acc/debug/libbar.so"
"bin/acc/debug/bar.o" "../foo/bin/acc/debug/libfoo.so"

...found 1 target...
...updating 1 target...
common.Clean clean
...updated 1 target...
acc.compile.c bin/acc/debug/baz.o

    aCC -Ae -c    +d +Z -g -I"../bar" -I"../foo"  -o
"bin/acc/debug/baz.o" "baz.c"

acc.link.dll bin/acc/debug/libbaz.so

    aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
"bin/acc/debug/baz.o" "../bar/bin/acc/debug/libbar.so"
"../foo/bin/acc/debug/libfoo.so"

ld: Can't find dependent library "libfoo.so"
Fatal error.
...failed updating 1 target...


Yet, both libfoo.so and libbar.so appear to have the correct
dependencies and embedded names:
$ elfdump -L foo/bin/acc/debug/libfoo.so

foo/bin/acc/debug/libfoo.so:

                *** DYNAMIC Section ***

.dynamic:

Index   Tag      Value

0       Soname   libfoo.so
1       Runpath  /usr/lib/hpux32:/opt/langtools/lib/hpux32
2       Flags    0x0
3       HP Flags 0x0
4       Link tab 0x40010010
5       Hash     0x40002e8
6       Dynstr   0x4000278
7       Dynsym   0x4000218
8       PltRsrv  0x40010000
9       GSTHshVl 0x4000368
10      GSTVersn 0x1
11      HPTime   Thu Jun 11 11:06:53 2009
12      HPChksum 0xf6566078
13      HPSyCksm 0x509b87c5
14      Strsz    0x6f
15      Syment   0x10

$ elfdump -L bar/bin/acc/debug/libbar.so

bar/bin/acc/debug/libbar.so:

                *** DYNAMIC Section ***

.dynamic:

Index   Tag      Value

0       Needed   libfoo.so
1       Soname   libbar.so
2       Runpath  /usr/lib/hpux32:/opt/langtools/lib/hpux32
3       Flags    0x0
4       HP Flags 0x0
5       Link tab 0x40010018
6       Hash     0x4000340
7       Dynstr   0x40002c0
8       Dynsym   0x4000250
9       PltRsrv  0x40010010
10      GSTHshVl 0x40003d8
11      GSTVersn 0x1
12      HPTime   Thu Jun 11 11:06:55 2009
13      HPChksum 0xcea03a5d
14      HPSyCksm 0xa65c4ebe
15      Rela     0x4000368
16      Relasz   0xc
17      Pltrel   0x7
18      Jmprel   0x4000368
19      Pltrelsz 0xc
20      Relaent  0xc
21      Strsz    0x7d
22      Syment   0x10

I think it should work, but it doesn't.

>
>>
>> So, how can I make boost-build tell the linker about dependent libs
>> the normal unix way, using -L<search dir> -l<lib name>?
>
>It's not possible now. I can change to do it this way, but please check
>if some soname equivalent is available.
>
>- Volodya

Thanks,
Jim
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Jim Gallagher]

Volodya,

I have been experimenting, and I have some more info to share. Back to
my example
with libfoo, libbar and libbaz, where libbar depends on libfoo and
libbaz depends
on libbar.

The link line generated by boost-build did not work:
    aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
"bin/acc/debug/baz.o" "../bar/bin/acc/debug/libbar.so"
"../foo/bin/acc/debug/libfoo.so"
ld: Can't find dependent library "libfoo.so"
Fatal error.


Rearranging for a manual test does not work either:
    aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
"bin/acc/debug/baz.o" "../foo/bin/acc/debug/libfoo.so"
"../bar/bin/acc/debug/libbar.so"
ld: Can't find dependent library "libfoo.so"
Fatal error.


However, if I add a search path using -L, it will find libfoo:
    aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
"bin/acc/debug/baz.o" -L../foo/bin/acc/debug
"../bar/bin/acc/debug/libbar.so"

So, it appears that on hp-ux, we must use the typical -L linker search
mechanism. Is this something that I may be able to fix with some
guidance, or do you think it is pretty involved? I am very much a
novice regarding much of what is happening under the hood.

Thanks,
Jim
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Jim Gallagher]

One additional data point: Linux embeds the path in the shared library
the same way as hp-ux does.

I stood up an Ubuntu 8.10 VM to test this. I did not get any errors at
link time, but the libraries whose names included path elements could
not be loaded at runtime, regardless of how I set LD_LIBRARY_PATH.

I am not familiar with the library tools on Linux, but I used xxd to
verify that the path supplied for dependent libraries on the link line
was embedded in the resultant library.

Jim
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus]

On Thursday 11 June 2009 Jim Gallagher wrote:

> Volodya,
>
> I have been experimenting, and I have some more info to share. Back to
> my example
> with libfoo, libbar and libbaz, where libbar depends on libfoo and
> libbaz depends
> on libbar.
>
> The link line generated by boost-build did not work:
>     aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
> "bin/acc/debug/baz.o" "../bar/bin/acc/debug/libbar.so"
> "../foo/bin/acc/debug/libfoo.so"
> ld: Can't find dependent library "libfoo.so"
> Fatal error.
>
>
> Rearranging for a manual test does not work either:
>     aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
> "bin/acc/debug/baz.o" "../foo/bin/acc/debug/libfoo.so"
> "../bar/bin/acc/debug/libbar.so"
> ld: Can't find dependent library "libfoo.so"
> Fatal error.
>
>
> However, if I add a search path using -L, it will find libfoo:
>     aCC -AA -b -g -Wl,+h,libbaz.so -o "bin/acc/debug/libbaz.so"
> "bin/acc/debug/baz.o" -L../foo/bin/acc/debug
> "../bar/bin/acc/debug/libbar.so"
>
> So, it appears that on hp-ux, we must use the typical -L linker search
> mechanism. Is this something that I may be able to fix with some
> guidance, or do you think it is pretty involved? I am very much a
> novice regarding much of what is happening under the hood.

Can you try the attached patch? (If it does not apply for some reason,
both actions are modified by adding:

         -L"$(RPATH_LINK)"

and there's a new flag setting.

At the same time, can you lookup two things in docs:

1. Is there an option to set rpath encoded in the libraries --
that is the path there libraries will be searched at runtime.
I notice your dump of binaries include it, so it must be
possible to specify it as well.

2. Is there an option to separately control where linker looks
for dependency shared libraries. It's called "rpath-link" in
Linux, maybe same for ACC.

Thanks,
Volodya

 

[rpath-link.diff]

---

Index: acc.jam
===================================================================
--- acc.jam (revision 53703)
+++ acc.jam (working copy)
@@ -66,6 +66,8 @@
 flags acc CFLAGS <address-model>64 : +DD64 ;
 flags acc LINKFLAGS <address-model>64 : +DD64 ;
 
+flags acc RPATH_LINK : <xdll-path> ;
+
 flags acc CFLAGS <cflags> ;
 flags acc C++FLAGS <cxxflags> ;
 flags acc DEFINES <define> ;
@@ -89,13 +91,13 @@
 
 actions acc.link bind NEEDLIBS
 {
-    $(CONFIG_COMMAND) -AA $(LINKFLAGS) -o "$(<[1])" -L$(LIBPATH) -L$(STDLIBPATH) "$(>)" "$(NEEDLIBS)" "$(NEEDLIBS)" -l$(FINDLIBS) $(OPTIONS)
+    $(CONFIG_COMMAND) -AA $(LINKFLAGS) -o "$(<[1])" -L"$(RPATH_LINK)" -L$(LIBPATH) -L$(STDLIBPATH) "$(>)" "$(NEEDLIBS)" "$(NEEDLIBS)" -l$(FINDLIBS) $(OPTIONS)
 }
 
 SPACE = " " ;
 actions acc.link.dll bind NEEDLIBS
 {
-    $(CONFIG_COMMAND) -AA -b $(LINKFLAGS) -o "$(<[1])" -Wl,+h$(SPACE)-Wl,$(<[-1]:D=)  -L$(LIBPATH) -L$(STDLIBPATH) "$(>)" "$(NEEDLIBS)" "$(NEEDLIBS)" -l$(FINDLIBS) $(OPTIONS)
+    $(CONFIG_COMMAND) -AA -b $(LINKFLAGS) -o "$(<[1])" -L"$(RPATH_LINK)" -Wl,+h$(SPACE)-Wl,$(<[-1]:D=)  -L$(LIBPATH) -L$(STDLIBPATH) "$(>)" "$(NEEDLIBS)" "$(NEEDLIBS)" -l$(FINDLIBS) $(OPTIONS)
 }
 
 actions acc.compile.c

---

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Jim Gallagher]

Volodya,

>Can you try the attached patch? (If it does not apply for some reason,
>both actions are modified by adding:
>
>         -L"$(RPATH_LINK)"
>
>and there's a new flag setting.
>
>At the same time, can you lookup two things in docs:
>
>1. Is there an option to set rpath encoded in the libraries --
>that is the path there libraries will be searched at runtime.
>I notice your dump of binaries include it, so it must be
>possible to specify it as well.
>
>2. Is there an option to separately control where linker looks
>for dependency shared libraries. It's called "rpath-link" in
>Linux, maybe same for ACC.
>
>Thanks,
>Volodya

The patch works well, thank very much, it's a life saver.

I changed
-Wl,+h$(SPACE)-Wl,$(<[-1]:D=) to
-Wl,+h$(<[-1]:D=).
Looks a little cleaner and works the same.

Here is the output from the build:

...found 1 target...
...updating 1 target...
common.Clean clean
...updated 1 target...
acc.compile.c bin/acc/debug/foo.o

    aCC -Ae -c    +d -g   -o "bin/acc/debug/foo.o" "foo.c"

acc.link.dll bin/acc/debug/libfoo.so

        aCC -AA -b -g -o "bin/acc/debug/libfoo.so"  -Wl,+hlibfoo.so
"bin/acc/debug/foo.o"


...found 1 target...
...updating 1 target...
common.Clean clean
...updated 1 target...
acc.compile.c bin/acc/debug/bar.o

    aCC -Ae -c    +d -g -I"../foo"  -o "bin/acc/debug/bar.o" "bar.c"

acc.link.dll bin/acc/debug/libbar.so

        aCC -AA -b -g -o "bin/acc/debug/libbar.so"
-L"/home/tce/ul081b/foo/bin/acc/debug" -Wl,+hlibbar.so
"bin/acc/debug/bar.o" "../foo/bin/acc/debug/libfoo.so"


...found 1 target...
...updating 1 target...
common.Clean clean
...updated 1 target...
acc.compile.c bin/acc/debug/baz.o

    aCC -Ae -c    +d -g -I"../bar" -I"../foo"  -o "bin/acc/debug/baz.o" "baz.c"

acc.link.dll bin/acc/debug/libbaz.so

        aCC -AA -b -g -o "bin/acc/debug/libbaz.so"
-L"/home/tce/ul081b/bar/bin/acc/debug"
-L"/home/tce/ul081b/foo/bin/acc/debug" -Wl,+hlibbaz.so
"bin/acc/debug/baz.o" "../bar/bin/acc/debug/libbar.so"
"../foo/bin/acc/debug/libfoo.so"



And the internals of libbaz.so:
$ elfdump -L baz/bin/acc/debug/libbaz.so

baz/bin/acc/debug/libbaz.so:

                *** DYNAMIC Section ***

.dynamic:

Index   Tag      Value

0       Needed   libbar.so
1       Needed   libfoo.so
2       Soname   libbaz.so
3       Runpath
/home/tce/ul081b/bar/bin/acc/debug:/home/tce/ul081b/foo/bin/acc/debug:/usr/lib/hpux32:/opt/langtools/lib/hpux32
4       Flags    0x0
5       HP Flags 0x0
6       Link tab 0x40010018
7       Hash     0x4000398
8       Dynstr   0x40002c8
9       Dynsym   0x4000258
10      PltRsrv  0x40010010
11      GSTHshVl 0x4000430
12      GSTVersn 0x1
13      HPTime   Fri Jun 12 09:36:08 2009
14      HPChksum 0x78065ff5
15      HPSyCksm 0x2014100c
16      Rela     0x40003c0
17      Relasz   0xc
18      Pltrel   0x7
19      Jmprel   0x40003c0
20      Pltrelsz 0xc
21      Relaent  0xc
22      Strsz    0xcd
23      Syment   0x10


So, everything looks good. For my case, I will probably add
+nodefaultrpath, which will
prevent the embedding of the paths supplied with -L. Those paths are
needed at link time,
but they are not appropriate for our runtime configuration.

To shift gears a bit, I have been following the mailing list for a
while, and I agree
with your (the maintainers) philosophy, that boost-build should not
surprise users
and should do what is usually the right thing. I think this is a case
that, at least
for HP-UX and Linux systems, and maybe for Unix in general, that
boost-build is doing
something that is surprising and not what most developers would want.
I suggest that
the default boost-build link lines for Unix systems should use the -L
and -l path
and lib specifiers.

Thanks again,

Jim
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus]

On Friday 12 June 2009 Jim Gallagher wrote:


> The patch works well, thank very much, it's a life saver.
>
> I changed
> -Wl,+h$(SPACE)-Wl,$(<[-1]:D=) to
> -Wl,+h$(<[-1]:D=).
> Looks a little cleaner and works the same.

I have finally checked this in in SVN.

> So, everything looks good. For my case, I will probably add
> +nodefaultrpath, which will
> prevent the embedding of the paths supplied with -L. Those paths are
> needed at link time,
> but they are not appropriate for our runtime configuration.

Aha, I see.

> To shift gears a bit, I have been following the mailing list for a
> while, and I agree
> with your (the maintainers) philosophy, that boost-build should not
> surprise users
> and should do what is usually the right thing. I think this is a case
> that, at least
> for HP-UX and Linux systems, and maybe for Unix in general, that
> boost-build is doing
> something that is surprising and not what most developers would want.
> I suggest that
> the default boost-build link lines for Unix systems should use the -L
> and -l path
> and lib specifiers.

Well, -L/-l is less precise, while specifying full name guaranteed that linker
will use the specified file.

- Volodya
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build