Newbie questions (where's the FAQ)

[boggler]

I'm kinda new to this whole GNU thing, so I need to figure out how to do some basic things.
I'm very interested in using the gold linker to link some large executables.

I downloaded the binutils-2.19.1 package last week (Feb 24), unpacked it, ran configure and make and make install, and now I can indeed run gold; but there are issues.

Questions:

1.  Is there a FAQ?  Where is it?

Hopefully someone will point me to  a FAQ that answers at least some of  my remaining questions, but I will list them here anyway, in case they aren't in the FAQ.  

2.  I need a patch that is described in the forum, but it was not included in the download (that is, the gold I built behaves in a way that indicates the patch is not there).  In fact, the file that the patch modifies was not included in the download.  So how do I apply the patch, or get a download with the patch in it?  Is there a .bleeding download, or some such, with all patches applied?

3.  I believe I will eventually want to make changes to the linker, at least for my own use (and to offer back to the community, if they want them).  I was able to do the anonymous CVS to download the full binutils source; but I don't know how to build gold from that.  If I could do that, I could probably apply patches.

4.  The README file for gold is a reasonable 50,000-foot overview of what it does, but it would also be very helpful to have a 10,000, or even 5000-foot view, describing the passes the linker performs and what it does in each pass, plus an overview of the data structures used, and ELF layout of the output files produced.  Does that exist?  If so, where?

5.  I am especially interested in incremental linking, both as described in some messages on the forum (existing executable plus new/replacement .o files, to create a new executable quickly), AND something else that I think of as incremental linking, but which may need a different name to avoid confusion - namely, pre-linking together significant chunks of object code, so that the final link only deals with a few large chunks, with minimal cross-connections, rather than 50,000+ separate object files with hundreds of thousands (if not millions) of references to resolve.  I would be interested in hearing from people who are working on, or have a special interest in, either of these concepts.

[Cary Coutant-2]

> I'm kinda new to this whole GNU thing, so I need to figure out how to do some
> basic things.
> I'm very interested in using the gold linker to link some large executables.

Welcome! I hope we can help.

> 1.  Is there a FAQ?  Where is it?

For gold, no, there's no FAQ at the moment. There's a bit of advice to
be found by searching the archives of this mailing list. Ian also has
a very helpful series of blog entries on the linker, starting here:

http://www.airs.com/blog/archives/date/2007/08/

> 2.  I need a patch that is described in the forum, but it was not included
> in the download (that is, the gold I built behaves in a way that indicates
> the patch is not there).  In fact, the file that the patch modifies was not
> included in the download.  So how do I apply the patch, or get a download
> with the patch in it?  Is there a .bleeding download, or some such, with all
> patches applied?

Based on your next question, it sounds like you discovered how to
download the latest top-of-trunk binutils. That should already include
any patches you've seen on this list.

> 3.  I believe I will eventually want to make changes to the linker, at least
> for my own use (and to offer back to the community, if they want them).  I
> was able to do the anonymous CVS to download the full binutils source; but I
> don't know how to build gold from that.  If I could do that, I could
> probably apply patches.

It should be the same as how you built after downloading a source
tarball. You need to use the --enable-gold option when you configure
at the top level, and you need to make all-binutils before you can
make all-gold.

> 4.  The README file for gold is a reasonable 50,000-foot overview of what it
> does, but it would also be very helpful to have a 10,000, or even 5000-foot
> view, describing the passes the linker performs and what it does in each
> pass, plus an overview of the data structures used, and ELF layout of the
> output files produced.  Does that exist?  If so, where?

Well, here's a 49,000-foot view:

It's actually a fairly standard "2-pass" linker where the first pass
scans over the input files collecting symbols and sections, and the
second pass goes over the input files again to apply relocations and
write the output file. In between the two passes is a fairly
substantial layout phase. You can get a good idea of the overall
control flow by looking at queue_initial_tasks, queue_middle_tasks,
and queue_final_tasks, in the source file gold.cc. The control flow is
handled by the workqueue -- everything is broken into individual tasks
that can be run in parallel threads.

The central concepts include: the Layout class (layout.h), which
manages the layout of the output file; the Object class and the Relobj
class derived from it (object.h), which represent the input objects;
the Output_data class (output.h), which represents the sections of the
output file (mostly Output_section objects that collect Input_section
objects); the Symbol_table class (symbol.h); and the Target class
(target.h), which abstracts all the target specifics.

Feel free to post questions here -- that's the best way to get started on a FAQ.

-cary

[Ian Lance Taylor-3]

boggler <fklotz@...> writes:

> 5.  I am especially interested in incremental linking, both as described in
> some messages on the forum (existing executable plus new/replacement .o
> files, to create a new executable quickly), AND something else that I think
> of as incremental linking, but which may need a different name to avoid
> confusion - namely, pre-linking together significant chunks of object code,
> so that the final link only deals with a few large chunks, with minimal
> cross-connections, rather than 50,000+ separate object files with hundreds
> of thousands (if not millions) of references to resolve.  I would be
> interested in hearing from people who are working on, or have a special
> interest in, either of these concepts.

Cary addressed your other questions.

On this one, I think it's an interesting idea, although I'm not sure how
much it will speed things up in practice.  Right now a relocatable link
(with the -r option) does not eliminate any of the relocations.

In practice, ld -r could eliminate all PC-relative relocations against
symbols which can not be overridden.  That would be a clear speedup.
There is no reason not to do that for cases where it is possible.  The
trick is knowing when it is permitted.  By default, it is normally not
permitted.  The output of ld -r might be linked into a shared library.
When linking into a shared library, by default relocations against
external symbols must remain, because the dynamic linker might need to
override them at runtime.  So the only way to do this with ld -r is to
add some option specifying that the output will not be linked into a
shared library.

Converting non-PC-relative relocations is never a clear speedup.  The
linker still has to compute the final symbol values.  Once you do that,
getting the value to use during a relocation computation is fast.

Ian

[boggler]

Lance,

Thanks for the explanation.  I am using the linker to build for embedded systems where we do not use dynamic libraries, so for me a sensible default for -r relocatable builds would be to resolve all the references we can, and discard those relocations, as we will definitely not need them again.  But I do see from your explanation that this might not be widely useful to others, just to me.

If I were adding an option for this, it would be a single command-line option for the entire build, not a per-symbol or per-file option addressing different symbols/relocations differently.

Thanks,
Frank

Ian Lance Taylor-3 wrote:

boggler <[e-mail]> writes:

> 5.  I am especially interested in incremental linking, both as described in
> some messages on the forum (existing executable plus new/replacement .o
> files, to create a new executable quickly), AND something else that I think
> of as incremental linking, but which may need a different name to avoid
> confusion - namely, pre-linking together significant chunks of object code,
> so that the final link only deals with a few large chunks, with minimal
> cross-connections, rather than 50,000+ separate object files with hundreds
> of thousands (if not millions) of references to resolve.  I would be
> interested in hearing from people who are working on, or have a special
> interest in, either of these concepts.

Cary addressed your other questions.

On this one, I think it's an interesting idea, although I'm not sure how
much it will speed things up in practice.  Right now a relocatable link
(with the -r option) does not eliminate any of the relocations.

In practice, ld -r could eliminate all PC-relative relocations against
symbols which can not be overridden.  That would be a clear speedup.
There is no reason not to do that for cases where it is possible.  The
trick is knowing when it is permitted.  By default, it is normally not
permitted.  The output of ld -r might be linked into a shared library.
When linking into a shared library, by default relocations against
external symbols must remain, because the dynamic linker might need to
override them at runtime.  So the only way to do this with ld -r is to
add some option specifying that the output will not be linked into a
shared library.

Converting non-PC-relative relocations is never a clear speedup.  The
linker still has to compute the final symbol values.  Once you do that,
getting the value to use during a relocation computation is fast.

Ian

[Jim Wilson-4]

boggler wrote:
> 3.  I believe I will eventually want to make changes to the linker, at least
> for my own use (and to offer back to the community, if they want them).

Contributing patches to the FSF requires a copyright assignment, and
unfortunately it is currently impossible for Cisco employees to get one.
  Maybe this will change in the future, but for now, you should assume
that you can not contribute any changes back.

Jim