Re: Bug#553936: FTBFS with binutils-gold

[Lucas Nussbaum]

Hi,

I thought people were supposed to discuss it on -devel@ before starting
a MBF?

Anyway, ways you could have made it better:
- provide a step by step guide to reproduce the problem
- use a usertag to follow all the bugs
- provide a link to a wiki page where you would have put more info about
  solving the common problems.

> Tried to build your package and it fails to build with GNU binutils-gold. The
> important difference is that --no-add-needed is the default behavior of of GNU
> binutils-gold. Please provide all needed libraries to the linker when building
> your executables.

Since this obviously breaks lots of packages, what about changing the
default in binutils-gold instead?
--
| Lucas Nussbaum
| lucas@...   http://www.lucas-nussbaum.net/ |
| jabber: lucas@...             GPG: 1024D/023B3F4F |


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Sune Vuorela-2]

On 2009-11-02, Lucas Nussbaum <lucas@...> wrote:
>
>> Tried to build your package and it fails to build with GNU binutils-gold. The
>> important difference is that --no-add-needed is the default behavior of of GNU
>> binutils-gold. Please provide all needed libraries to the linker when building
>> your executables.
>
> Since this obviously breaks lots of packages, what about changing the
> default in binutils-gold instead?

Binutils-gold is not the default. It is a bit 'stricter' in how it
works, but it is notably faster.

/Sune


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Lucas Nussbaum]

On 02/11/09 at 12:12 +0000, Sune Vuorela wrote:

> On 2009-11-02, Lucas Nussbaum <lucas@...> wrote:
> >
> >> Tried to build your package and it fails to build with GNU binutils-gold. The
> >> important difference is that --no-add-needed is the default behavior of of GNU
> >> binutils-gold. Please provide all needed libraries to the linker when building
> >> your executables.
> >
> > Since this obviously breaks lots of packages, what about changing the
> > default in binutils-gold instead?
>
> Binutils-gold is not the default. It is a bit 'stricter' in how it
> works, but it is notably faster.

I was talking about not making --no-add-needed the default in
binutils-gold.
--
| Lucas Nussbaum
| lucas@...   http://www.lucas-nussbaum.net/ |
| jabber: lucas@...             GPG: 1024D/023B3F4F |


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Peter Fritzsche]

Lucas Nussbaum wrote:
> I thought people were supposed to discuss it on -devel@ before starting
> a MBF?
What is a MBF?

> Anyway, ways you could have made it better:
> - provide a step by step guide to reproduce the problem
> - use a usertag to follow all the bugs
> - provide a link to a wiki page where you would have put more info about
>   solving the common problems.
>
> > Tried to build your package and it fails to build with GNU binutils-gold.
> > The important difference is that --no-add-needed is the default behavior
> > of of GNU binutils-gold. Please provide all needed libraries to the
> > linker when building your executables.
>
> Since this obviously breaks lots of packages, what about changing the
> default in binutils-gold instead?

I am not sure but do you think that it is a good way to link against a library
without specify that you link against it?

What I am currently testing is if there are crashes/segfaults possible when
linking with binutils-gold. But it seems that many packages doesn't create
problems for binutils-gold, but fail to build because they rely on the fact
that other libraries link against the libraries. So when they stop to link
against them the build of the "unrelated" executable would break (as it  
breaks right now with --no-add-needed or binutils-gold).

So things I could do is: ignore the fact that they don't specify the libraries
which must be linked to work and wait until binutils-gold replaces old ld/old
ld switches to more sane default/third party library stops to link against the
needed library - or report the problem and let the maintainer decide what to
do.

I choose the latter one because I think that most maintainers don't know about
the problems.

Best regards,
        Peter


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Sune Vuorela-2]

On 2009-11-02, Lucas Nussbaum <lucas@...> wrote:
> On 02/11/09 at 12:12 +0000, Sune Vuorela wrote:
>>
>> Binutils-gold is not the default. It is a bit 'stricter' in how it
>> works, but it is notably faster.

> I was talking about not making --no-add-needed the default in
> binutils-gold.

then you are effectively neutering gold.
thes is what I consider one of the features of gold, and it is also one
ef the ways it is faster (by not traversing the depndencies of the
libraries to see if they are needed)

the major difference is:
your app uses both libfoo and libbar. libbar uses libfoo.  with ld, it
is sufficient to do -lbar, and with gold you need -lfoo -lbar.
in both cases, your app will have NEEDED entries for both libfoo and
libbar.

/Sune


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Lucas Nussbaum]

On 02/11/09 at 13:50 +0100, Peter Fritzsche wrote:
> Lucas Nussbaum wrote:
> > I thought people were supposed to discuss it on -devel@ before starting
> > a MBF?
> What is a MBF?

mass bug filing.

> > Anyway, ways you could have made it better:
> > - provide a step by step guide to reproduce the problem
> > - use a usertag to follow all the bugs
> > - provide a link to a wiki page where you would have put more info about
> >   solving the common problems.
> >
> > > Tried to build your package and it fails to build with GNU binutils-gold.
> > > The important difference is that --no-add-needed is the default behavior
> > > of of GNU binutils-gold. Please provide all needed libraries to the
> > > linker when building your executables.
> >
> > Since this obviously breaks lots of packages, what about changing the
> > default in binutils-gold instead?
> I am not sure but do you think that it is a good way to link against a library
> without specify that you link against it?

The question is not about what I think. The question is whether it's
reasonable to expect A LOT of packages to be modified to accomodate
this.

> What I am currently testing is if there are crashes/segfaults possible when
> linking with binutils-gold. But it seems that many packages doesn't create
> problems for binutils-gold, but fail to build because they rely on the fact
> that other libraries link against the libraries. So when they stop to link
> against them the build of the "unrelated" executable would break (as it  
> breaks right now with --no-add-needed or binutils-gold).
>
> So things I could do is: ignore the fact that they don't specify the libraries
> which must be linked to work and wait until binutils-gold replaces old ld/old
> ld switches to more sane default/third party library stops to link against the
> needed library - or report the problem and let the maintainer decide what to
> do.
>
> I choose the latter one because I think that most maintainers don't know about
> the problems.

Could you provide some numbers on the:
- packages that FTBFS with binutils-gold
- packages that FTBFS with binutils-gold because of --no-add-needed by
  default
  ?
--
| Lucas Nussbaum
| lucas@...   http://www.lucas-nussbaum.net/ |
| jabber: lucas@...             GPG: 1024D/023B3F4F |


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Peter Fritzsche]

Lucas Nussbaum wrote:
> > What is a MBF?
>
> mass bug filing.
thanks

> > I am not sure but do you think that it is a good way to link against a
> > library without specify that you link against it?
>
> The question is not about what I think. The question is whether it's
> reasonable to expect A LOT of packages to be modified to accomodate
> this.

This is way it is currently not marked as serious. It doesn't affect the
release of the next debian version, but all these things could break in the
future. I am willing to gather more informations in the wiki but currently
don't know how.


> Could you provide some numbers on the:
> - packages that FTBFS with binutils-gold
> - packages that FTBFS with binutils-gold because of --no-add-needed by
>   default
>   ?

I have only build some packages. I have currently following numbers:

50 Packages break
555 Packages work (without arch all)

As you can see I have only data for about 7% of all packages so I cannot make
a real forecast how many packages are real affected. If you want to
extrapolate the data (real bad extrapolation) about 700 packages of around
14000 source packages in Debian would break. I am currently hoping that they
are less than that. Some packages in the statistics seems to be also broken
with the current ld - avifile seems to be one of them. The author is currently
waiting for a sponsor to fix that problem.

To illustrate the problem:
 exe -> liba -> libb

exe gets linked against liba, but actual need symbols from libb. It works in
the case of the old GNU ld because it loads all libraries in that dependency
graph until it finds the symbols. This seems to be ok unless liba gets
modified and doesn't link anymore against libb. In this case the compilation
stops during the linker run.

Keep in mind that these bugs aren't release critical, but the maintainer and
upstream should know about these problems. Maybe I could reduce the problem of
all bugs to minor.

About the usertags? Is there a special format in which they should be? And a
maintainer asked for a wiki entry at
http://wiki.debian.org/qa.debian.org/FTBFS but I am not allowed to change that
patch - and I am sure that I shouldn't be allowed to edit it.

Best regards,
        Peter


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Lucas Nussbaum]

On 02/11/09 at 15:04 +0100, Peter Fritzsche wrote:
> About the usertags? Is there a special format in which they should be? And a
> maintainer asked for a wiki entry at
> http://wiki.debian.org/qa.debian.org/FTBFS but I am not allowed to change that
> patch - and I am sure that I shouldn't be allowed to edit it.

You probably just need a wiki account.
--
| Lucas Nussbaum
| lucas@...   http://www.lucas-nussbaum.net/ |
| jabber: lucas@...             GPG: 1024D/023B3F4F |


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Peter Samuelson]

[Sune Vuorela]
> the major difference is:
> your app uses both libfoo and libbar. libbar uses libfoo.  with ld, it
> is sufficient to do -lbar, and with gold you need -lfoo -lbar.
> in both cases, your app will have NEEDED entries for both libfoo and
> libbar.

If you want symbol versioning to work properly (assuming libfoo has
versioned symbols), you need to specify -lfoo -lbar regardless.  Been
there, didn't do that, got the segfaults.
--
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Peter Fritzsche]

Lucas Nussbaum wrote:
> On 02/11/09 at 15:04 +0100, Peter Fritzsche wrote:
> > About the usertags? Is there a special format in which they should be?
> > And a maintainer asked for a wiki entry at
> > http://wiki.debian.org/qa.debian.org/FTBFS but I am not allowed to change
> > that patch - and I am sure that I shouldn't be allowed to edit it.
>
> You probably just need a wiki account.
Thanks for the info. A draft entry can be found at
http://wiki.debian.org/qa.debian.org/FTBFS#A2009-11-02Packagesfailingbecausebinutils-gold.2BAC8-indirectlinking


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Biebl-3]

As I received a couple of bug reports today about packages FTBFS, I installed
binutils-gold and tried to compile a few of my packages with it.

What I noticed is, that every package with symbols file, produced a lintian
error, as binutils-gold added new symbols, the most common one was
end@base

What's the reason for those additional symbol(s) with binutils-gold?

Michael


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Mike Hommey]

On Fri, Nov 06, 2009 at 12:44:31AM +0100, Michael Biebl wrote:
> As I received a couple of bug reports today about packages FTBFS, I installed
> binutils-gold and tried to compile a few of my packages with it.
>
> What I noticed is, that every package with symbols file, produced a lintian
> error, as binutils-gold added new symbols, the most common one was
> end@base
>
> What's the reason for those additional symbol(s) with binutils-gold?

Corollary: can't they be hidden ?

Mike


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Raphael Hertzog-3]

On Fri, 06 Nov 2009, Mike Hommey wrote:

> On Fri, Nov 06, 2009 at 12:44:31AM +0100, Michael Biebl wrote:
> > As I received a couple of bug reports today about packages FTBFS, I installed
> > binutils-gold and tried to compile a few of my packages with it.
> >
> > What I noticed is, that every package with symbols file, produced a lintian
> > error, as binutils-gold added new symbols, the most common one was
> > end@base
> >
> > What's the reason for those additional symbol(s) with binutils-gold?
>
> Corollary: can't they be hidden ?

If necessary, they can be blacklisted by dpkg-gensymbols, it already
does that for a couple of internal symbols that the toolchain does add
automatically. Please file a wishlist bug against dpkg-dev with the list
of symbols in that case.

Cheers,
--
Raphaël Hertzog


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Peter Fritzsche]

Michael Biebl wrote:
> As I received a couple of bug reports today about packages FTBFS, I
>  installed binutils-gold and tried to compile a few of my packages with it.
>
> What I noticed is, that every package with symbols file, produced a lintian
> error, as binutils-gold added new symbols, the most common one was
> end@base
>
> What's the reason for those additional symbol(s) with binutils-gold?
This is an open problem reported as #553435 (see also forwarded bug). I
haven't received any updates to this problem(?) yet. I will post information
about that inside the bug report if I receive anything useful from the author.

Best regards,
        Peter


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Biebl-3]

Peter Fritzsche wrote:

> Michael Biebl wrote:
>> As I received a couple of bug reports today about packages FTBFS, I
>>  installed binutils-gold and tried to compile a few of my packages with it.
>>
>> What I noticed is, that every package with symbols file, produced a lintian
>> error, as binutils-gold added new symbols, the most common one was
>> end@base
>>
>> What's the reason for those additional symbol(s) with binutils-gold?
> This is an open problem reported as #553435 (see also forwarded bug). I
> haven't received any updates to this problem(?) yet. I will post information
> about that inside the bug report if I receive anything useful from the author.

Apart from the addition end@base symbol, binutils-gold also doesn't seem to
honor the -export-symbols-regex linker flag, which is most likely the cause for
the other additional symbols I see.

Is this a known issue? If not, could you please forward that to upstream.

Thanks,
Michael

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Peter Fritzsche]

Michael Biebl wrote:

> Peter Fritzsche wrote:
> > Michael Biebl wrote:
> >> What I noticed is, that every package with symbols file, produced a
> >> lintian error, as binutils-gold added new symbols, the most common one
> >> was end@base
> >>
> >> What's the reason for those additional symbol(s) with binutils-gold?
> >
> > This is an open problem reported as #553435 (see also forwarded bug). I
> > haven't received any updates to this problem(?) yet. I will post
> > information about that inside the bug report if I receive anything useful
> > from the author.
>
> Apart from the addition end@base symbol, binutils-gold also doesn't seem to
> honor the -export-symbols-regex linker flag, which is most likely the cause
>  for the other additional symbols I see.

Thanks for the info. I will  forward that.

Best regards,
        Peter


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Peter Fritzsche]

Michael Biebl wrote:
> Apart from the addition end@base symbol, binutils-gold also doesn't seem to
> honor the -export-symbols-regex linker flag, which is most likely the cause
>  for the other additional symbols I see.
>
> Is this a known issue? If not, could you please forward that to upstream.
I checked it and it seems to work with:

echo 'void testme(unsigned int t){}int main(){return 0;}' > test.c && \
echo "{ global:\nmain;\nlocal: *; };" > test.ver && \
gcc test.c -o test -Wl,-version-script,test.ver && \
readelf -s test|grep testme |grep GLOBAL

This should make testme local.... and it works.

-version-script is what does the work and -export-symbols-regex is just an
option of libtool. I would guess that libtool-stuff is the problem here..... I
looked through the code in dbus-glib and found the -version-script stuff in
the configure script. It is enabled and disabled using
supports_anon_versioning. The script to test for that support is:

    supports_anon_versioning=no
    case `$LD -v 2>&1` in
      *\ [01].* | *\ 2.[0-9].* | *\ 2.10.*) ;; # catch versions < 2.11
      *\ 2.11.93.0.2\ *) supports_anon_versioning=yes ;; # RH7.3 ...
      *\ 2.11.92.0.12\ *) supports_anon_versioning=yes ;; # Mandrake 8.2 ...
      *\ 2.11.*) ;; # other 2.11 versions
      *) supports_anon_versioning=yes ;;
    esac

The output of `ld -v 2` is "GNU gold (GNU Binutils for Debian 2.20) 1.9". So
it will catch the 1.9 here and just say "hey, i am sure that you are evil"....
which is of course wrong. So auto* stuff must be updated here. I will create a
bug for libtool.

Best regards,
        Peter


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Biebl-3]

Peter Fritzsche wrote:
> Michael Biebl wrote:
>> Apart from the addition end@base symbol, binutils-gold also doesn't seem to
>> honor the -export-symbols-regex linker flag, which is most likely the cause
>>  for the other additional symbols I see.
..
>
> The output of `ld -v 2` is "GNU gold (GNU Binutils for Debian 2.20) 1.9". So
> it will catch the 1.9 here and just say "hey, i am sure that you are evil"....
> which is of course wrong. So auto* stuff must be updated here. I will create a
> bug for libtool.
>

Thanks for the insightful reply.

Nice work.

Michael


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Peter Fritzsche]

Michael Biebl wrote:
> As I received a couple of bug reports today about packages FTBFS, I
>  installed binutils-gold and tried to compile a few of my packages with it.
>
> What I noticed is, that every package with symbols file, produced a lintian
> error, as binutils-gold added new symbols, the most common one was
> end@base
>
> What's the reason for those additional symbol(s) with binutils-gold?
Got response by the author and was a bug
http://sourceware.org/bugzilla/show_bug.cgi?id=10876#c4

I had opened a bug #554852 in dpkg-dev but it is now closed again.

Best regards,
        Peter


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Henrique de Moraes Holschuh]

On Fri, 06 Nov 2009, Peter Fritzsche wrote:
> The output of `ld -v 2` is "GNU gold (GNU Binutils for Debian 2.20) 1.9". So
> it will catch the 1.9 here and just say "hey, i am sure that you are evil"....
> which is of course wrong. So auto* stuff must be updated here. I will create a
> bug for libtool.

One is supposed to use Debian libtool, instead of whatever crap
(non-libtool) upstream added to the tarball, and this has been true for many
years, now.

Still, policy doesn't mandate that Debian libtool be used, and it will take
about 15 years or so to get all packages updated if you go the "let it
filter upstream" way ;-)

If that's a major problem for a complete switch to -gold, you may want to
keep that in mind.  Maybe we could mandate that all packages be
re-libtoolized before build, using Debian libtool... but I fear that will be
quite a lot of trouble.

--
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...