LTO branch merged into trunk - trunk remains CLOSED

[Diego Novillo-3]

The LTO branch has been merged into trunk at revision 152434.

Please note that trunk remains CLOSED.  I would like to give
automatic testers a chance to pick up the merge and test it in
isolation.  I plan to open trunk in 48 hours (Mon 5 Oct).

Following this message, I will post the final 13 patches, roughly
split by directories.

Instructions on how to enable LTO support are described in the
manual.  The following is a summary:

- Install libelf 0.8.12+ (http://www.mr511.de/software/libelf-0.8.12.tar.gz)
  Other versions of libelf are commonly installed in Linux
  distros.  Unfortunately, libelf implementations vary in
  annoying ways and they are not all compatible.  The libelf
  version in Fedora 11 works fine, however.

- Configure and build with

        $ mkdir bld && cd bld
        $ ../trunk/configure --with-libelf=<path-to-libelf>
        $ make

If libelf is installed in a system path, you don't need to
specify --with-libelf, of course.

To enable LTO, simply add the flag '-flto' to both compile and
link commands.  It doesn't really matter whether you compile and
link in separate invocations or a single one.  All that matters
is that -flto should be in both places:

        $ gcc -flto -O2 -o f f1.c f2.c

                        or

        $ gcc -flto -c f1.c
        $ gcc -flto -c f2.c
        $ gcc -o f -flto -O2 f1.o f2.o

That's it.  The result should, in principle, execute faster but
our IPA cost models are still not tweaked for LTO.  We've seen
speedups as well as slowdowns in benchmarks (see the LTO testers
at http://gcc.opensuse.org/).

Please report performance and correctness issues in Bugzilla.

I bootstrapped and tested the merge on x86_64-unknown-linux-gnu,
i686-pc-linux-gnu and powerpc64-unknown-linux-gnu.

Additionally the merge has been bootstrapped and tested on these
targets (http://gcc.gnu.org/ml/gcc/2009-09/msg00566.html):

alphaev68-unknown-linux-gnu
i386-pc-solaris2.10
i586-suse-linux-gnu
i686-pc-linux-gnu
ia64-suse-linux-gnu
mipsisa64-unknown-elf
powerpc64-suse-linux-gnu
powerpc64-unknown-linux-gnu
s390-ibm-linux-gnu
s390x-ibm-linux-gnu
sh4-unknown-linux-gnu
x86_64-suse-linux-gnu
x86_64-unknown-linux-gnu
sparc-sun-solaris2.11

Richi bootstrapped and tested on
{i586,x86_64,ppc,ppc64,ia64,s390,s390x}-linux-gnu with LTO
disabled.

Thanks to all the folks who contributed cycles for this!

You should expect the following LTO-related failures, which will
be fixed during stage 3.  From what I've seen, I believe that
most of these failures are already in bugzilla.  I've summarized
the LTO-related failures below, if there is anything missing
please report it in bugzilla.  Particularly those that seem
target-specific.

---------------------------------------------------------------------------

Common to most targets:

FAIL: g++.dg/lto/20090106 cp_lto_20090106_0.o-cp_lto_20090106_0.o link
FAIL: g++.dg/lto/20090311 cp_lto_20090311_0.o-cp_lto_20090311_1.o link
FAIL: gcc.c-torture/execute/builtins/sprintf-chk.c execution
FAIL: gcc.c-torture/execute/builtins/stpcpy-chk.c execution
FAIL: gcc.c-torture/execute/builtins/strcpy-chk.c execution
FAIL: gfortran.dg/lto/pr40725 f_lto_pr40725_0.o-f_lto_pr40725_1.o execute -O2 -flto
FAIL: gfortran.dg/lto/pr40725 f_lto_pr40725_0.o-f_lto_pr40725_1.o execute -O2 -fwhopr

---------------------------------------------------------------------------

Common to some targets:

i386-pc-solaris2.10
i686-pc-linux-gnu
i586-suse-linux-gnu
mipsisa64-elf
powerpc64-suse-linux-gnu
sh4-unknown-linux-gnu
FAIL: g++.dg/lto/20081109 cp_lto_20081109_0.o-cp_lto_20081109_1.o execute -O0 -fwhopr
FAIL: g++.dg/lto/20081109 cp_lto_20081109_0.o-cp_lto_20081109_1.o execute -O2 -fwhopr


i386-pc-solaris2.10
mipsisa64-elf
sparc-sun-solaris2.11
FAIL: g++.dg/lto/20081120-1 cp_lto_20081120-1_0.o-cp_lto_20081120-1_1.o link
FAIL: g++.dg/lto/20081120-2 cp_lto_20081120-2_0.o-cp_lto_20081120-2_1.o link


i586-suse-linux-gnu
ia64-suse-linux-gnu
s390-ibm-linux-gnu
sh4-unknown-linux-gnu
FAIL: gcc.dg/torture/builtin-math-7.c  -O2 -flto  (internal compiler error)
FAIL: gcc.dg/torture/builtin-math-7.c  -O2 -flto  (test for excess errors)
FAIL: gcc.dg/torture/builtin-math-7.c  -O2 -fwhopr  (internal compiler error)
FAIL: gcc.dg/torture/builtin-math-7.c  -O2 -fwhopr  (test for excess errors)

mipsisa64-elf
sh4-unknown-linux-gnu
FAIL: gcc.dg/lto/20081112 c_lto_20081112_0.o-c_lto_20081112_1.o link
FAIL: gcc.dg/lto/20081118 c_lto_20081118_0.o-c_lto_20081118_2.o link

---------------------------------------------------------------------------

Individual target failures:

i386-pc-solaris2.10
FAIL: g++.dg/lto/20080908-3 cp_lto_20080908-3_0.o-cp_lto_20080908-3_0.o execute -O0 -flto
FAIL: g++.dg/lto/20080908-3 cp_lto_20080908-3_0.o-cp_lto_20080908-3_0.o execute -O0 -fwhopr
FAIL: gcc.dg/lto/20081126 c_lto_20081126_0.o-c_lto_20081126_0.o link
FAIL: gcc.dg/lto/20090126-1 c_lto_20090126-1_0.o-c_lto_20090126-1_0.o link
FAIL: gcc.dg/lto/20090126-2 c_lto_20090126-2_0.o-c_lto_20090126-2_0.o link
FAIL: gcc.dg/lto/20090729 c_lto_20090729_0.o-c_lto_20090729_1.o link

ia64-suse-linux-gnu
FAIL: g++.dg/lto/20081217-1 cp_lto_20081217-1_0.o-cp_lto_20081217-1_0.o link
FAIL: g++.dg/lto/20081217-2 cp_lto_20081217-2_0.o-cp_lto_20081217-2_0.o link
FAIL: gcc.c-torture/execute/vector-2.c execution,  -O2 -flto
FAIL: gcc.c-torture/execute/vector-2.c execution,  -O2 -fwhopr

mipsisa64-elf
FAIL: g++.dg/lto/20081119-1 cp_lto_20081119-1_0.o-cp_lto_20081119-1_1.o link
FAIL: g++.dg/lto/20081123 cp_lto_20081123_0.o-cp_lto_20081123_1.o link
FAIL: g++.dg/lto/20081204-1 cp_lto_20081204-1_0.o-cp_lto_20081204-1_1.o link
FAIL: g++.dg/lto/20081219 cp_lto_20081219_0.o-cp_lto_20081219_1.o link
FAIL: g++.dg/lto/20090302 cp_lto_20090302_0.o-cp_lto_20090302_1.o link
FAIL: g++.dg/lto/20090313 cp_lto_20090313_0.o-cp_lto_20090313_1.o link
FAIL: gcc.dg/lto/20081204-1 c_lto_20081204-1_0.o-c_lto_20081204-1_1.o link
FAIL: gcc.dg/lto/20081212-1 c_lto_20081212-1_0.o-c_lto_20081212-1_0.o link

powerpc64-suse-linux-gnu
FAIL: gcc.c-torture/execute/20071030-1.c execution,  -O2 -flto
FAIL: gcc.c-torture/execute/20071030-1.c execution,  -O2 -fwhopr
FAIL: gcc.dg/vmx/* -O2 {-fwhopr,-flto}

sparc-sun-solaris2.11
FAIL: gcc.dg/lto/20081120-1 c_lto_20081120-1_0.o-c_lto_20081120-1_1.o link
FAIL: gcc.dg/lto/20081201-2 c_lto_20081201-2_0.o-c_lto_20081201-2_1.o
FAIL: gcc.dg/lto/20090729 c_lto_20090729_0.o-c_lto_20090729_1.o link
FAIL: tmpdir-g++.dg-struct-layout-1/t003 cp_compat_x_tst.o-cp_compat_y_tst.o execute "-O2","-flto"
FAIL: tmpdir-g++.dg-struct-layout-1/t003 cp_compat_x_tst.o-cp_compat_y_tst.o execute "-O2","-fwhopr"
FAIL: tmpdir-g++.dg-struct-layout-1/t019 cp_compat_x_tst.o-cp_compat_y_tst.o execute "-O2","-flto"
FAIL: tmpdir-g++.dg-struct-layout-1/t019 cp_compat_x_tst.o-cp_compat_y_tst.o execute "-O2","-fwhopr"
FAIL: tmpdir-g++.dg-struct-layout-1/t030 cp_compat_x_tst.o-cp_compat_y_tst.o execute "-O2","-flto"
FAIL: tmpdir-g++.dg-struct-layout-1/t030 cp_compat_x_tst.o-cp_compat_y_tst.o execute "-O2","-fwhopr"

sh4-unknown-linux-gnu
FAIL: gcc.dg/lto/20090218 c_lto_20090218_0.o-c_lto_20090218_3.o link
FAIL: gcc.dg/lto/20090218-2 c_lto_20090218-2_0.o-c_lto_20090218-2_1.o link
FAIL: gfortran.dg/lto/pr40724 f_lto_pr40724_0.o-f_lto_pr40724_1.o link

[Richard Guenther-2]

On Sat, Oct 3, 2009 at 11:12 PM, Diego Novillo <dnovillo@...> wrote:
> The LTO branch has been merged into trunk at revision 152434.
...

> To enable LTO, simply add the flag '-flto' to both compile and
> link commands.  It doesn't really matter whether you compile and
> link in separate invocations or a single one.  All that matters
> is that -flto should be in both places:
>
>        $ gcc -flto -O2 -o f f1.c f2.c
>
>                        or
>
>        $ gcc -flto -c f1.c
>        $ gcc -flto -c f2.c
>        $ gcc -o f -flto -O2 f1.o f2.o

Remember though that options specified at compile time are _not_ automatically
enabled at link time!  Thus the 2nd example above would perform no initial
optimization (effectively skipping all early optimizations until the IPA passes)
while enabling optimization (starting with IPA passes and then remaining
optimization passes) at link time.  Thus the better canonical example would
have been

  $ gcc -O2 -flto -c f1.c
  $ gcc -O2 -flto -c f2.c
  $ gcc -o f -flto -O2 f1.o f2.o

where in particular enabling -fwhole-program is what should enable extra
optimization (well, if you do actually link a final binary).

Note that one missing feature is picking up entries from static library
archives at link time.  To not end up with unused LTO sections from
static libraries in the final binary you need to have recent enough binutils
or strip them manually.

Richard.

[Diego Novillo-3]

On Sat, Oct 3, 2009 at 17:32, Richard Guenther
<richard.guenther@...> wrote:

> Note that one missing feature is picking up entries from static library
> archives at link time.

This is true when using GNU ld.  But it works just fine if you
configure gcc with --enable-gold and compile with -use-linker-plugin.


Diego.

[Richard Guenther-2]

On Sat, Oct 3, 2009 at 11:12 PM, Diego Novillo <dnovillo@...> wrote:
> The LTO branch has been merged into trunk at revision 152434.

Btw, I think this deserves a news entry on the main page as well as (obviously)
an entry for gcc-4.5/changes.html.

Richard.

[Diego Novillo-3]

On Sat, Oct 3, 2009 at 18:24, Richard Guenther
<richard.guenther@...> wrote:
> On Sat, Oct 3, 2009 at 11:12 PM, Diego Novillo <dnovillo@...> wrote:
>> The LTO branch has been merged into trunk at revision 152434.
>
> Btw, I think this deserves a news entry on the main page as well as (obviously)
> an entry for gcc-4.5/changes.html.

Yes.  I will take care of both.


Diego.

[Diego Novillo-3]

On Sat, Oct 3, 2009 at 19:40, Diego Novillo <dnovillo@...> wrote:
> On Sat, Oct 3, 2009 at 18:24, Richard Guenther
> <richard.guenther@...> wrote:
>> On Sat, Oct 3, 2009 at 11:12 PM, Diego Novillo <dnovillo@...> wrote:
>>> The LTO branch has been merged into trunk at revision 152434.
>>
>> Btw, I think this deserves a news entry on the main page as well as (obviously)
>> an entry for gcc-4.5/changes.html.
>
> Yes.  I will take care of both.

Done with this patch.  Committed to wwwdocs.

Diego.


Index: htdocs/index.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/index.html,v
retrieving revision 1.718
diff -d -u -p -r1.718 index.html
--- htdocs/index.html 19 Sep 2009 21:00:47 -0000 1.718
+++ htdocs/index.html 4 Oct 2009 01:01:32 -0000
@@ -45,6 +45,13 @@ mission statement</a>.</p>

 <dl class="news">

+<dt>October 3, 2009</dt>
+<dd>The <a href="http://gcc.gnu.org/wiki/LinkTimeOptimization">LTO</a>
+branch has been merged into trunk.  The next release of GCC will
+feature a new whole-program optimizer, able to perform interprocedural
+optimizations across different files, even if they are written in
+different languages.</dd>
+
 <dt>August 4, 2009</dt>
 <dd><a href="gcc-4.3/">GCC 4.3.4</a> has been released.</dd>

Index: htdocs/svn.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/svn.html,v
retrieving revision 1.138
diff -d -u -p -r1.138 svn.html
--- htdocs/svn.html 30 Sep 2009 05:10:15 -0000 1.138
+++ htdocs/svn.html 4 Oct 2009 01:01:32 -0000
@@ -175,11 +175,6 @@ the command <code>svn log --stop-on-copy
   <dd>The purpose of this branch is to develop an infrastructure for loop
   transforms using the polyhedral model.</dd>

-  <dt><a href="http://gcc.gnu.org/wiki/LinkTimeOptimization">lto</a></dt>
-  <dd>This branch aims to implement link-time optimization.  Patches
-  and discussion on this branch should be marked with the tag
-  <code>[lto]</code> in the subject line.</dd>
-
   <dt><a href="http://gcc.gnu.org/wiki/LightweightIpo">lw-ipo</a></dt>
   <dd>This branch aims to implement lightweight IPO.  Patches
   and discussion on this branch should be marked with the tag
@@ -950,6 +945,9 @@ be prefixed with the initials of the dis
   Michael Meissner,
   <a href="mailto:meissner@...">meissner@...</a>.</dd>

+  <dt><a href="http://gcc.gnu.org/wiki/LinkTimeOptimization">lto</a></dt>
+  <dd>This branch implemented link-time optimization.</dd>
+
 </dl>

 </body>
Index: htdocs/gcc-4.5/changes.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/gcc-4.5/changes.html,v
retrieving revision 1.33
diff -d -u -p -r1.33 changes.html
--- htdocs/gcc-4.5/changes.html 30 Sep 2009 18:25:32 -0000 1.33
+++ htdocs/gcc-4.5/changes.html 4 Oct 2009 01:01:32 -0000
@@ -48,6 +48,16 @@
     supported before; for those a fall back is used if MPC is not
     available.</li>

+    <li>A new link-time optimizer has been added (<code>-flto</code>).
+    When this flag is used, GCC generates a bytecode representation of
+    each input file and writes it to special ELF sections in each
+    object file.  When the object files are linked together, all the
+    function bodies are read from these ELF sections and instantiated
+    as if they had been part of the same translation unit.  This
+    enables interprocedural optimizations to work across different
+    files (and even different languages), potentially improving the
+    performance of the generated code.</li>
+
   </ul>

 <h2>New Languages and Language specific improvements</h2>

[Frank Ch. Eigler]

Diego Novillo <dnovillo@...> writes:

> The LTO branch has been merged into trunk at revision 152434.
> [...]

Congrats.

> [...]  That's it.  The result should, in principle, execute faster
> but our IPA cost models are still not tweaked for LTO.  We've seen
> speedups as well as slowdowns in benchmarks (see the LTO testers at
> http://gcc.opensuse.org/).  [...]

Would it make sense to keep -flto unemphasized in NEWS etc., until
some consistently positive performance results of some sort have been
identified?

- FChE