[Boost][build] Bjam ordering source $(>) list

[K. Noel Belcourt]

Hi,

In jam/src/make1.c, make1list(), I find this comment

* make1list() - turn a list of targets into a LIST, for $(<) and $(>).

which seems like the place where I could ensure that sources for, say  
a link action $(>) are ordered so that objects occur before archives  
or shared libraries.  Some background.

We're bringing up support for linking Fortran programs and have  
discovered that when linking C programs, the objects in $(>) always  
precede the libraries, which is good since the main is often in an  
object file.  This is an issue if, for example, your linker doesn't  
support start / end group and the library precedes the object  
containing main (you get linker errors).

In our case, it seems that our objects always precede any libraries  
when linking with C, but when linking with Fortran, we find the  
libraries precede the objects and, hence, we get linker errors.

Back to make1list().  It seems like this is the place where the list  
of sources and targets for an action are turned into a list for $(<)  
and $(>).  Does anyone know (1) if bjam intentionally sorts objects  
before libraries for $(>) and, if it does as it seems to, (2) where  
is to code in bjam that does this?

Any ideas?

Thanks.

-- Noel Belcourt


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Steven Watanabe-4]

AMDG

K. Noel Belcourt wrote:

> In jam/src/make1.c, make1list(), I find this comment
>
> * make1list() - turn a list of targets into a LIST, for $(<) and $(>).
>
> which seems like the place where I could ensure that sources for, say
> a link action $(>) are ordered so that objects occur before archives
> or shared libraries.  Some background.
>
> We're bringing up support for linking Fortran programs and have
> discovered that when linking C programs, the objects in $(>) always
> precede the libraries, which is good since the main is often in an
> object file.  This is an issue if, for example, your linker doesn't
> support start / end group and the library precedes the object
> containing main (you get linker errors).

I'm not quite sure that I follow.  Libraries ought to be in a separate
variable from sources.

> In our case, it seems that our objects always precede any libraries
> when linking with C, but when linking with Fortran, we find the
> libraries precede the objects and, hence, we get linker errors.
>
> Back to make1list().  It seems like this is the place where the list
> of sources and targets for an action are turned into a list for $(<)
> and $(>).  Does anyone know (1) if bjam intentionally sorts objects
> before libraries for $(>) and, if it does as it seems to, (2) where is
> to code in bjam that does this?

The order of the targets ought to be controlled by Boost.Build in
gcc.link for example.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[K. Noel Belcourt]

Hi Steven,

On Jun 25, 2009, at 4:49 PM, Steven Watanabe wrote:

> K. Noel Belcourt wrote:
>> In jam/src/make1.c, make1list(), I find this comment
>>
>> * make1list() - turn a list of targets into a LIST, for $(<) and $
>> (>).
>>
>> which seems like the place where I could ensure that sources for, say
>> a link action $(>) are ordered so that objects occur before archives
>> or shared libraries.  Some background.
>>
>> We're bringing up support for linking Fortran programs and have
>> discovered that when linking C programs, the objects in $(>) always
>> precede the libraries, which is good since the main is often in an
>> object file.  This is an issue if, for example, your linker doesn't
>> support start / end group and the library precedes the object
>> containing main (you get linker errors).
>
> I'm not quite sure that I follow.  Libraries ought to be in a separate
> variable from sources.

Ignore for a second that start group / end group is available to us.  
Consider this gcc link action from gcc.jam:

actions link bind LIBRARIES
{
     "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$
(SPACE)-Wl,"$(RPATH)" -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)" -o "$
(<)" $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$
(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $
(OPTIONS) $(USER_OPTIONS)
}

with an C executable target like this:

exe Utst_element
   :
     $(framework-root)/src/utest/Utst_element.C
     apub
     elem
     elements
     fmwk
     parser
     /sierra/user_subroutines//usersublib
     /sierra/utility//utility
   : <tag>@runtest-tag
   ;

the expanded sources passed into this $(>) link action variable is  
this (severely truncated):

Utst_element.o libparser.a libelem.a libutility.a libfmwk.a  
libelements.a libusersublib.a libapub.a libsolver-control.a libmath.a  
libgeometry.a libcholla.a libgeom.a libcubit.a libfrio.a  
libtransfer.a libacme.a libmmod.a libsearch.a libelemint.a libewed.a  
libetet.a libetri.a libehex.a libequa.a libepub.a libebar.a libepyr.a  
librbal.a libzoltan.a libparmetis.a libmetis.a libzoltan_dd.a  
libzoltan_comm.a libzoltan_mem.a libfmwk.a libioinit.a libiotr.a  
libiohb.a libioex.a libnemesis.a libexodus.a libnetcdf.a libioel.a  
libparser.a libxml.a libfrii.a libioss.a libelem.a  
libgeometry_toolkit.a libshards.a libutility.a

with the object always preceding the libraries.  Note that all this  
is in the $(>) link action variable.

>> In our case, it seems that our objects always precede any libraries
>> when linking with C, but when linking with Fortran, we find the
>> libraries precede the objects and, hence, we get linker errors.
>>
>> Back to make1list().  It seems like this is the place where the list
>> of sources and targets for an action are turned into a list for $(<)
>> and $(>).  Does anyone know (1) if bjam intentionally sorts objects
>> before libraries for $(>) and, if it does as it seems to, (2)  
>> where is
>> to code in bjam that does this?
>
> The order of the targets ought to be controlled by Boost.Build in
> gcc.link for example.

I'm sure there's more going on than I know, but it seems to me that  
any sources in the exe rule are expanded (as above) by including the  
sources in those dependencies.

These are the sources from the exe rule above.

     $(framework-root)/src/utest/Utst_element.C
     apub
     elem
     elements
     fmwk
     parser
     /sierra/user_subroutines//usersublib
     /sierra/utility//utility

All these sources (and the sources in these dependencies) get lumped  
into the $(>) variable that is passed into the link action.

I guess what I'm wondering is why are all the exe (and dependent)  
sources (object and libraries) lumped into $(>)?  Perhaps the follow-
on question is: given that the objects and libraries in the exe rule  
sources are all in $(>), can we apply a stable sort to $(>) so that  
the objects precede the libraries while retaining the original  
relative ordering.  This will help us on systems where we don't have  
a multi-pass linker nor options like start / end group.

Does that make any sense?

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Juergen Hunold-2]

Hi !

On Friday 26 June 2009 01:13:26 K. Noel Belcourt wrote:
> All these sources (and the sources in these dependencies) get lumped  
> into the $(>) variable that is passed into the link action.

Yes, that is what I found out, too.

> I guess what I'm wondering is why are all the exe (and dependent)  
> sources (object and libraries) lumped into $(>)?  Perhaps the follow-
> on question is: given that the objects and libraries in the exe rule  
> sources are all in $(>), can we apply a stable sort to $(>) so that  
> the objects precede the libraries while retaining the original  
> relative ordering.  This will help us on systems where we don't have  
> a multi-pass linker nor options like start / end group.
>
> Does that make any sense?

At least for me, yes.

I've encountered a similar problem while implementing response file support
for gcc on windows. I ended up extrating the libs from $(>) and passing them
to the build rule. Unfortunately, I have historical circular dependencies.
I attach the relevant part of my gcc.jam patch for completeness.

I would like to know more about building the sources of $(>), too.


Yours,

Jürgen
--
* Dipl.-Math. Jürgen Hunold       ! Ingenieurgesellschaft für
* voice: ++49 511 262926 57       ! Verkehrs- und Eisenbahnwesen mbH  
* fax  : ++49 511 262926 99       ! Lister Straße 15
* juergen.hunold@...        ! www.ivembh.de
*
* Geschäftsführer:                ! Sitz des Unternehmens: Hannover
* Prof. Dr.-Ing. Thomas Siefer    ! Amtsgericht Hannover, HRB 56965
* PD Dr.-Ing. Alfons Radtke       !

[short.diff]

---

Index: tools/build/v2/tools/gcc.jam
===================================================================
--- tools/build/v2/tools/gcc.jam (revision 54371)
+++ tools/build/v2/tools/gcc.jam (working copy)
@@ -789,12 +790,17 @@
     # parallel is just slower. For now, serialize only gcc links, it might be a
     # good idea to serialize all links.
     JAM_SEMAPHORE on $(targets) = <s>gcc-link-semaphore ;
+
+    LIBS = [ MATCH "<p(.*\\.lib)" : $(sources) ] ;
+    EXTRA_LIBS on $(targets) = [ regex.replace-list $(LIBS) : ">" : "\\" ] ;
 }
 
 actions link bind LIBRARIES
 {
-    "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,"$(RPATH)" -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)" -o "$(<)" $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS)
-
+    "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,"$(RPATH)" -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)" -o "$(<)" @($(<[1]:T).link:E=INPUT( $(>) );) $(START-GROUP) "$(EXTRA_LIBS)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS)
 }
 
 # Default value. Mostly for the sake of intel-linux that inherits from gcc, but
@@ -841,7 +847,7 @@
 # That warning is produced only on some platforms, for whatever reasons.
 actions piecemeal archive
 {
-    "$(.AR)" $(AROPTIONS) rc "$(<)" "$(>)"
+    "$(.AR)" $(AROPTIONS) rc "$(<)" "@"@($(<[1]:T).rsp:E=$(>:T))
 }
 
 
@@ -856,7 +866,7 @@
 # Differs from 'link' above only by -shared.
 actions link.dll bind LIBRARIES
 {
-    "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,"$(RPATH)" "$(.IMPLIB-COMMAND)$(<[1])" -o "$(<[-1])" $(HAVE_SONAME)-Wl,$(SONAME_OPTION)$(SPACE)-Wl,$(<[-1]:D=) -shared $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS)
+    "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,"$(RPATH)" "$(.IMPLIB-COMMAND)$(<[1])" -o "$(<[-1])" $(HAVE_SONAME)-Wl,$(SONAME_OPTION)$(SPACE)-Wl,$(<[-1]:D=) -shared @($(<[1]:T).link:E=INPUT( $(>) );) $(START-GROUP) "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS)
 }
 
 rule setup-threading ( targets * : sources * : properties * )

---

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[K. Noel Belcourt]

Hi Jurgen,

On Jun 26, 2009, at 3:53 AM, Juergen Hunold wrote:

> On Friday 26 June 2009 01:13:26 K. Noel Belcourt wrote:
>
>>
>> I guess what I'm wondering is why are all the exe (and dependent)
>> sources (object and libraries) lumped into $(>)?  Perhaps the follow-
>> on question is: given that the objects and libraries in the exe rule
>> sources are all in $(>), can we apply a stable sort to $(>) so that
>> the objects precede the libraries while retaining the original
>> relative ordering.  This will help us on systems where we don't have
>> a multi-pass linker nor options like start / end group.
>>
>> Does that make any sense?
>
> At least for me, yes.
>
> I've encountered a similar problem while implementing response file  
> support
> for gcc on windows. I ended up extrating the libs from $(>) and  
> passing them
> to the build rule. Unfortunately, I have historical circular  
> dependencies.
> I attach the relevant part of my gcc.jam patch for completeness.
>
> I would like to know more about building the sources of $(>), too.

Thanks for the push, I dug in a bit more to see how sources $(>) are  
populated for the link action.

In builtin.jam, linking-generator.generated-targets() is the place to  
alter what ends up in $(>), and in what order.  For my problem, I  
separated the OBJ sources from all other sources so I could place the  
OBJ before STATIC_LIB, etc...  The fix turned out to work beautifully  
as you can see here.

My Fortran executable link action was being passed two libraries  
before it sees the object containing the main, leading to my link  
errors.

STATIC_LIB
STATIC_LIB
OBJ
STATIC_LIB
STATIC_LIB
STATIC_LIB
STATIC_LIB

object(file-target)@15922 object(file-target)@16343 object(file-
target)@23952 object(file-target)@2015 object(file-target)@2286 object
(file-target)@8052 object(file-target)@8568

The actual OBJ file that I need to appear first in $(>) is actually  
the third entity in the list ( object(file-target)@23952 ).   With  
the attached patch, the OBJ is now the first source in $(>), as you  
can see here.

object(file-target)@23952 object(file-target)@15922 object(file-
target)@16343 object(file-target)@2015 object(file-target)@2286 object
(file-target)@8052 object(file-target)@8568

This patch moves all objects before non-objects and retains their  
respective relative ordering, so as not to break the dependency  
ordering.

Anyone think this is this worth committing to Boost.Build?

I appreciate your insight Jurgen!

-- Noel







_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Steven Watanabe-4]

AMDG

K. Noel Belcourt wrote:

> Thanks for the push, I dug in a bit more to see how sources $(>) are
> populated for the link action.
>
> In builtin.jam, linking-generator.generated-targets() is the place to
> alter what ends up in $(>), and in what order.  For my problem, I
> separated the OBJ sources from all other sources so I could place the
> OBJ before STATIC_LIB, etc...  The fix turned out to work beautifully
> as you can see here.
> <snip>
>
> This patch moves all objects before non-objects and retains their
> respective relative ordering, so as not to break the dependency ordering.
>
> Anyone think this is this worth committing to Boost.Build?

Maybe you can use unix-linking-generator which also
makes sure that dependent libraries are placed in the correct order?

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[K. Noel Belcourt]

On Jun 26, 2009, at 2:08 PM, Steven Watanabe wrote:

> K. Noel Belcourt wrote:
>> Thanks for the push, I dug in a bit more to see how sources $(>) are
>> populated for the link action.
>>
>> In builtin.jam, linking-generator.generated-targets() is the place to
>> alter what ends up in $(>), and in what order.  For my problem, I
>> separated the OBJ sources from all other sources so I could place the
>> OBJ before STATIC_LIB, etc...  The fix turned out to work beautifully
>> as you can see here.
>> <snip>
>>
>> This patch moves all objects before non-objects and retains their
>> respective relative ordering, so as not to break the dependency  
>> ordering.
>>
>> Anyone think this is this worth committing to Boost.Build?
>
> Maybe you can use unix-linking-generator which also
> makes sure that dependent libraries are placed in the correct order?

That would mean that generators that don't inherit from unix-linking-
generator wouldn't benefit from this change (e.g. msvc-linking-
generator), but I guess that doesn't really matter.

Sure, I can move it into unix.jam.

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Steven Watanabe-4]

AMDG

K. Noel Belcourt wrote:

> On Jun 26, 2009, at 2:08 PM, Steven Watanabe wrote:
>
>> K. Noel Belcourt wrote:
>>> Thanks for the push, I dug in a bit more to see how sources $(>) are
>>> populated for the link action.
>>>
>>> In builtin.jam, linking-generator.generated-targets() is the place to
>>> alter what ends up in $(>), and in what order.  For my problem, I
>>> separated the OBJ sources from all other sources so I could place the
>>> OBJ before STATIC_LIB, etc...  The fix turned out to work beautifully
>>> as you can see here.
>>> <snip>
>>>
>>> This patch moves all objects before non-objects and retains their
>>> respective relative ordering, so as not to break the dependency
>>> ordering.
>>>
>>> Anyone think this is this worth committing to Boost.Build?
>>
>> Maybe you can use unix-linking-generator which also
>> makes sure that dependent libraries are placed in the correct order?
>
>
> That would mean that generators that don't inherit from
> unix-linking-generator wouldn't benefit from this change (e.g.
> msvc-linking-generator), but I guess that doesn't really matter.
>
> Sure, I can move it into unix.jam.

Any toolset that cares about library ordering ought to use
unix-linking-generator  AFAIK, the only reason that unix.jam
exists at all is to make sure that libraries are placed in the correct
location on the command line.  The logic that's already there is
opposite your patch, it finds all LIB targets and puts them after
the other sources.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

[Vladimir Prus]

On Saturday 27 June 2009 Steven Watanabe wrote:

> AMDG
>
> K. Noel Belcourt wrote:
> > On Jun 26, 2009, at 2:08 PM, Steven Watanabe wrote:
> >
> >> K. Noel Belcourt wrote:
> >>> Thanks for the push, I dug in a bit more to see how sources $(>) are
> >>> populated for the link action.
> >>>
> >>> In builtin.jam, linking-generator.generated-targets() is the place to
> >>> alter what ends up in $(>), and in what order.  For my problem, I
> >>> separated the OBJ sources from all other sources so I could place the
> >>> OBJ before STATIC_LIB, etc...  The fix turned out to work beautifully
> >>> as you can see here.
> >>> <snip>
> >>>
> >>> This patch moves all objects before non-objects and retains their
> >>> respective relative ordering, so as not to break the dependency
> >>> ordering.
> >>>
> >>> Anyone think this is this worth committing to Boost.Build?
> >>
> >> Maybe you can use unix-linking-generator which also
> >> makes sure that dependent libraries are placed in the correct order?
> >
> >
> > That would mean that generators that don't inherit from
> > unix-linking-generator wouldn't benefit from this change (e.g.
> > msvc-linking-generator), but I guess that doesn't really matter.
> >
> > Sure, I can move it into unix.jam.
>
> Any toolset that cares about library ordering ought to use
> unix-linking-generator  AFAIK, the only reason that unix.jam
> exists at all is to make sure that libraries are placed in the correct
> location on the command line.  The logic that's already there is
> opposite your patch, it finds all LIB targets and puts them after
> the other sources.

Right, that's where the magic is. I'd imagine that 'fortran linker'
should extend this unix-linking-generator, if possible.

- Volodya
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build