[LTO merge][0/15] Description of the final 15 patches

[Diego Novillo-3]

I will be sending the final 15 patches to bring all the
functionality from the LTO branch.  I tried to divide the patches
along maintainer lines, but there are some overlaps

- C FE
        Adds processing of -flto and -fwhopr.

- C++ FE
        Adds a langhook used by need_assembler_name_p to handle
        templates.  This implements Jason's suggestion from
        http://gcc.gnu.org/ml/gcc-patches/2009-09/msg00534.html

- Fortran FE
        Adds processing of -flto and -fwhopr.  Note that the
        Fortran testsuite still shows ~140 failures with LTO
        enabled.  IIUC, most of these have corresponding bugs
        filed and they are not LTO-related (in that LTO exposes
        some bugs in the FE).  We should discuss whether we want
        to include Fortran support for LTO in 4.5.

- Java FE
        There is no LTO support for Java, but since LTO uses
        zlib, Java does not need to include it.

- LTO FE
        This is the new front end for GIMPLE.

- Configury
        Detection of libelf, LTO support, etc.

- Contrib
        A minor patch to add lto-plugin to the timestamps.

- Documentation
        Documentation for the different LTO options.

- Driver
        The somewhat ugly changes to collect2 and the main gcc
        driver to support spawning lto1 when we detect a .o file
        with LTO sections in it.

- Streamer
        The code needed to support the on-disk representation of
        GIMPLE, CFG and cgraph.

- Langhooks
        New langhooks to support manipulating ELF sections and
        the new langhook for asking the FE if a DECL needs to
        have an assembler name set.

- libiberty
        I need help with this one.  When the linker plugin is
        enabled (if GCC is configured to use gold), LTO can
        detect LTO objects inside archives via the callbacks it
        gets from the linker.  Since the linker plugin is a
        shared object, and it uses libiberty functions, it needs
        to use a shared libiberty.

        Currently, we just force --enable-shared on libiberty,
        but I would only want to do that if gold and lto are
        enabled.  We detect gold and lto support in the top
        configure script, but how do I send that down to
        libiberty's configure?

- Linker plugin
        This is the set of callbacks that gold uses to talk with
        lto1 and resolve symbols out of object files in the link.

- Middle end
        All sorts of changes, mostly in the callgraph, IPA and
        pass manager.

- Testsuite
        New tests for LTO and new dejagnu support for writing LTO
        tests.  Most of this has already been reviewed by Janis.


Major outstanding work:

- Support for debugging information.  Currently, -flto -g does
  not work (no debug information is generated).  This is because
  free_lang_data does not preserve enough information for the
  debug hooks to emit it.  I will fix this during stage 3.

- Outstanding bugzilla (http://tinyurl.com/yarz5q9).

- Some 'FIXME lto' markers.  These are cleanup markers, several
  related to WHOPR.  We could probably debate whether to include
  the WHOPR functionality in 4.5.  It certainly is not very
  different from LTO, but it does need some TLC.

Once I get all the approvals I need, I will request trunk to be
frozen for a day or so to give me time to commit the whole thing.
I already have a merged local tree and this exact same code is in
the LTO branch.


Thanks.  Diego.

[Joseph S. Myers]

On Mon, 28 Sep 2009, Diego Novillo wrote:

> - libiberty
> I need help with this one.  When the linker plugin is
> enabled (if GCC is configured to use gold), LTO can
> detect LTO objects inside archives via the callbacks it
> gets from the linker.  Since the linker plugin is a
> shared object, and it uses libiberty functions, it needs
> to use a shared libiberty.
>
> Currently, we just force --enable-shared on libiberty,
> but I would only want to do that if gold and lto are
> enabled.  We detect gold and lto support in the top
> configure script, but how do I send that down to
> libiberty's configure?

Shared libiberty seems like a bad idea as you then need to deal with
soname allocation and changing the soname whenever an ABI-incompatible
change is made.  What you actually need is a PIC libiberty to link into
the plugin (and avoiding using any global data in libiberty that needs a
single copy in any program; hopefully it doesn't have any), not a shared
one.

(You also need to make sure this doesn't affect what is built for the
target, only for the host.)

--
Joseph S. Myers
joseph@...

[Joseph S. Myers]

On Mon, 28 Sep 2009, Diego Novillo wrote:

> I will be sending the final 15 patches to bring all the
> functionality from the LTO branch.  I tried to divide the patches
> along maintainer lines, but there are some overlaps

I'll go through the individual patches later, but a general comment:

You say "tested on x86_64".  More detail is needed.  Have you tested both
LTO-enabled and LTO-disabled configurations, making sure there are no
regressions in each case?  I think such testing of both configurations is
needed.  In addition, an LTO-enabled cross from a non-ELF host to an ELF
target should be tested (there are five non-ELF hosts in the secondary
platforms list for 4.5; any one of them could be used for this, or a
non-ELF host not on that list; I realise gold's plugin support may not yet
be portable to such hosts so that part of the patch may not be testable
for this case).

--
Joseph S. Myers
joseph@...

[Diego Novillo-3]

On Mon, Sep 28, 2009 at 22:00, Joseph S. Myers <joseph@...> wrote:

> You say "tested on x86_64".  More detail is needed.  Have you tested both

Sorry, I completely spaced out and forgot to mention this.  Various
folks are testing the branch.  I sent a request for testing the branch
(http://gcc.gnu.org/ml/gcc/2009-09/msg00566.html).  I'm using that
thread to keep track of breakage that should be fixed before the final
merge.

> LTO-enabled and LTO-disabled configurations, making sure there are no
> regressions in each case?

Yes.  Targets with no ELF support refuse to enable LTO and error out
at configure time if --enable-lto is forced.  There was a bug in some
of the testsuite patches that Richi fixed.


Diego.

[Ian Lance Taylor-3]

"Joseph S. Myers" <joseph@...> writes:

> On Mon, 28 Sep 2009, Diego Novillo wrote:
>
>> - libiberty
>> I need help with this one.  When the linker plugin is
>> enabled (if GCC is configured to use gold), LTO can
>> detect LTO objects inside archives via the callbacks it
>> gets from the linker.  Since the linker plugin is a
>> shared object, and it uses libiberty functions, it needs
>> to use a shared libiberty.
>>
>> Currently, we just force --enable-shared on libiberty,
>> but I would only want to do that if gold and lto are
>> enabled.  We detect gold and lto support in the top
>> configure script, but how do I send that down to
>> libiberty's configure?
>
> Shared libiberty seems like a bad idea as you then need to deal with
> soname allocation and changing the soname whenever an ABI-incompatible
> change is made.  What you actually need is a PIC libiberty to link into
> the plugin (and avoiding using any global data in libiberty that needs a
> single copy in any program; hopefully it doesn't have any), not a shared
> one.

Yes.  Fortunately there is no such thing as a shared libiberty.  If
you configure libiberty with --enable-shared, what you get is a PIC
libiberty (libiberty/pic/libiberty.a) alongside the non-PIC one
(libiberty/libiberty.a).

So all Diego needs to do is pass --enable-shared down to libiberty
when --enable-lto/--enable-gold.  The way to do that is something like
the appended.

Ian


Index: configure.ac
===================================================================
--- configure.ac (revision 152253)
+++ configure.ac (working copy)
@@ -2495,6 +2495,11 @@ case $enable_bootstrap in
     ;;
 esac
 
+extra_host_libiberty_configure_flags=
+if test "$enable_gold" = "yes" -a "$enable_lto" = "yes"; then
+  extra_host_libiberty_configure_flags=--enable-shared
+fi
+
 AC_MSG_CHECKING(for default BUILD_CONFIG)
 
 AC_ARG_WITH([build-config],
Index: Makefile.def
===================================================================
--- Makefile.def (revision 152253)
+++ Makefile.def (working copy)
@@ -97,7 +97,8 @@ host_modules= { module= ld; bootstrap=tr
 host_modules= { module= libcpp; bootstrap=true; };
 host_modules= { module= libdecnumber; bootstrap=true; };
 host_modules= { module= libgui; };
-host_modules= { module= libiberty; bootstrap=true; };
+host_modules= { module= libiberty; bootstrap=true;
+        extra_configure_flags='@extra_host_libiberty_configure_flags@'};
 // We abuse missing to avoid installing anything for libiconv.
 host_modules= { module= libiconv;
  extra_configure_flags='--disable-shared';

[Diego Novillo-3]

On Tue, Sep 29, 2009 at 10:47, Ian Lance Taylor <iant@...> wrote:

> So all Diego needs to do is pass --enable-shared down to libiberty
> when --enable-lto/--enable-gold.  The way to do that is something like
> the appended.

Thanks.  Testing.


Diego.

[Paolo Bonzini-2]

> So all Diego needs to do is pass --enable-shared down to libiberty
> when --enable-lto/--enable-gold.  The way to do that is something like
> the appended.

What about just always adding --enable-shared to the host libiberty?

Paolo

[Ian Lance Taylor-3]

Paolo Bonzini <bonzini@...> writes:

>> So all Diego needs to do is pass --enable-shared down to libiberty
>> when --enable-lto/--enable-gold.  The way to do that is something like
>> the appended.
>
> What about just always adding --enable-shared to the host libiberty?

That will just cause everybody to always compile libiberty twice.  Why
do that if we don't have to?

Ian

[Paolo Bonzini-2]

On 09/29/2009 06:52 PM, Ian Lance Taylor wrote:

> Paolo Bonzini<bonzini@...>  writes:
>
>>> So all Diego needs to do is pass --enable-shared down to libiberty
>>> when --enable-lto/--enable-gold.  The way to do that is something like
>>> the appended.
>>
>> What about just always adding --enable-shared to the host libiberty?
>
> That will just cause everybody to always compile libiberty twice.  Why
> do that if we don't have to?

It's just a matter of time before we need to do that, anyway (say, when
we have plugins that we distribute by default, for example for mudflap).
  Conditionalizing on GCC could be a middle ground, or maybe the worst
possible choice? :-)

Paolo

[Ian Lance Taylor-3]

Paolo Bonzini <bonzini@...> writes:

> On 09/29/2009 06:52 PM, Ian Lance Taylor wrote:
>> Paolo Bonzini<bonzini@...>  writes:
>>
>>>> So all Diego needs to do is pass --enable-shared down to libiberty
>>>> when --enable-lto/--enable-gold.  The way to do that is something like
>>>> the appended.
>>>
>>> What about just always adding --enable-shared to the host libiberty?
>>
>> That will just cause everybody to always compile libiberty twice.  Why
>> do that if we don't have to?
>
> It's just a matter of time before we need to do that, anyway (say,
> when we have plugins that we distribute by default, for example for
> mudflap). Conditionalizing on GCC could be a middle ground, or maybe
> the worst possible choice? :-)

The shared libiberty is only required for the gold plugin.  It's
possible to get benefits from LTO without the gold plugin.  Which is a
good thing, since gold only works for a few targets.

Ian