[bug #22923] option to prevent "interspersed" output in parallel builds

[Paolo Bonzini-2]

URL:
  <http://savannah.gnu.org/bugs/?22923>

                 Summary: option to prevent "interspersed" output in parallel
builds
                 Project: make
            Submitted by: bkustel
            Submitted on: Wednesday 04/16/2008 at 03:26
                Severity: 3 - Normal
              Item Group: Enhancement
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 3.81
        Operating System: Any
           Fixed Release: None

    _______________________________________________________

Details:

In the Parallel Execution section of the manual, it accurately states:

   "One unpleasant consequence of running several commands
    simultaneously is that output generated by the commands
    appears whenever each command sends it, so messages
    from different commands may be interspersed."

But it should be possible have the output of each parallel command/process
batch up its output and only write it to the primary stdout & stderr streams
when the process completes in a synchronized fashion. This seems like it
should be relatively simple to accomplish. The only downside I can see is that
for LONG processes with lots of output, the display would not be in real time
and there could potentially be some storage issues with holding onto that data
until the process completes. But I imagine one could spool the stdout and
stderr off to a temp file marking which blocks are stdout and which are stderr
and then this could be replayed back later to reproduce the proper ordering of
the stdout/stderr output.

I suspect most users of the Parallel execution feature would prefer to have
the output of each individual command be contiguous, though perhaps some would
not.

Thus, I propose that an option be added to force the output of each parallel
process to be made continguous in the primary stdout & stderr streams.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Paolo Bonzini-2]

Follow-up Comment #1, bug #22923 (project make):

Some commands are long-running.  And while it would be useful for post-build
analysis to have the entire output of each command grouped together, it's much
less fun to watch.  :)

Perhaps something like MSFT did with parallel builds in Visual Studio?  That
is:

1) Line-buffer (i.e., don't intermix output from multiple processes on the
same line); and
2) Prefix each line with a number to indicate which process generated the
message.  (Visual Studio uses "1>", "2>", etc.)



    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Paolo Bonzini-2]

Follow-up Comment #2, bug #22923 (project make):

It's not actually quite that simple, although it's do-able.  Make is not a
multi-threaded application, and so it cannot read from multiple streams at the
same time (such as stdin and stderr, either for a single job or for multiple
jobs) without a lot of complexity and potential loss of performance.

The only realistic way this could be done would be, as bkustel mentions,
redirecting the output of each job to a temporary file then printing the
contents of that file after the job is complete.  It would be necessary to
redirect both stdout and stderr for each job to one file and there's no way to
"mark" them, since make is not actually processing the output of the jobs.

Similarly, ijprest's idea of having make read lines and print them, prefixed
with a job number, cannot really be implemented by make itself for the reasons
mentioned in the first paragraph.

However, one idea would be a new special variable that defined a program that
job output would be piped to.  If that variable was set, then every time make
runs a job it would run this program and arrange for all stdout/stderr from
the job to be piped to the command line that was the value of that variable.
It could have various format options that could optionally be provided, to
give a job number, the command being invoked, or whatever.  Then people could
provide their own script or program that did whatever they liked: stored the
output and printed it all at the end, or printed it line-by-line with a job
prefix, etc.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Paolo Bonzini-2]

Follow-up Comment #3, bug #22923 (project make):

Thanks ipjrest and psmith for both replies. They are very appreciated.

Responding to ipjrest's comments:
=================================
I agree some commands are long-running and that if "watching" progress is
important, the proposed behavior may not be desireable. This is one reason I
suggested the proposed new behavior be optional.

1) Line-buffering: While it would help in some cases, not intermixing lines
from different processes is only partially helpful. The fact that a "single
line" from process A might be interrupted by a line from process B is my
concern here. For example, instead of seeing
  Process A Fragment 1 was emitted and then fragment 2 was emitted.
  Process B Message
one might see
  A: Process A Fragment 1 was emitted
  B: Process B Message
  A: and then fragment 2 was emitted.
if the line for process A is emitted in 2 separate write operations.
2) The VS line prefixing would be OK. In fact I like this idea. But it still
has the problem that single lines can be broken if the tool being run
sometimes emits lines "in pieces". I have noticed that gcc has this problem in
some cases (at least the gcc version I am currently using, 4.1.0).

Responding to psmith's comments:
================================
Thanks for pointing out that make is not multithreaded. I guess I was
assuming it was. I agree this complicates matters somewhat. Also, I see your
point about my original stdout/stderr "marking" idea. In both cases I assumed
make would have either an auxilliary thread or its main thread doing these
tasks but I now realize that this is not how make currently works.

But I think something along the lines of your "special variable" idea could
be employed in a generic fashion. Perhaps this is what you intended your
second paragraph below to imply? Specifically, the proposed option could cause
all "commands" to have their output piped to a standard tool which stores
(its) stdin to a temp file and then writes it synchronously to stdout after
stdin is closed. Also, I think it could be reasonable to allow for an option
to either combine stdout/stderr here or to pipe them to separate processes (so
that stdout/stderr could be atomically emitted to stdout/stderr
respectively).

While I have nothing against the "special variable" idea, I'd very much like
to see a way to accomplish this via a make command-line option and the
variable idea sounded as if it might require a makefile modification.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Howard Chu]

Brian Kustel wrote:

>
> Follow-up Comment #3, bug #22923 (project make):
>
> Thanks ipjrest and psmith for both replies. They are very appreciated.
>
> Responding to ipjrest's comments:
> =================================
> I agree some commands are long-running and that if "watching" progress is
> important, the proposed behavior may not be desireable. This is one reason I
> suggested the proposed new behavior be optional.
>
> 1) Line-buffering: While it would help in some cases, not intermixing lines
> from different processes is only partially helpful. The fact that a "single
> line" from process A might be interrupted by a line from process B is my
> concern here. For example, instead of seeing
>    Process A Fragment 1 was emitted and then fragment 2 was emitted.
>    Process B Message
> one might see
>    A: Process A Fragment 1 was emitted
>    B: Process B Message
>    A: and then fragment 2 was emitted.
> if the line for process A is emitted in 2 separate write operations.
> 2) The VS line prefixing would be OK. In fact I like this idea. But it still
> has the problem that single lines can be broken if the tool being run
> sometimes emits lines "in pieces". I have noticed that gcc has this problem in
> some cases (at least the gcc version I am currently using, 4.1.0).
>
> Responding to psmith's comments:
> ================================
> Thanks for pointing out that make is not multithreaded. I guess I was
> assuming it was. I agree this complicates matters somewhat. Also, I see your
> point about my original stdout/stderr "marking" idea. In both cases I assumed
> make would have either an auxilliary thread or its main thread doing these
> tasks but I now realize that this is not how make currently works.
>
> But I think something along the lines of your "special variable" idea could
> be employed in a generic fashion. Perhaps this is what you intended your
> second paragraph below to imply? Specifically, the proposed option could cause
> all "commands" to have their output piped to a standard tool which stores
> (its) stdin to a temp file and then writes it synchronously to stdout after
> stdin is closed. Also, I think it could be reasonable to allow for an option
> to either combine stdout/stderr here or to pipe them to separate processes (so
> that stdout/stderr could be atomically emitted to stdout/stderr
> respectively).
>
> While I have nothing against the "special variable" idea, I'd very much like
> to see a way to accomplish this via a make command-line option and the
> variable idea sounded as if it might require a makefile modification.

Just my 2 cents, the Alliant parallel make (which inspired my design of the
make -j jobserver) also redirected all of the spawned processes' stdio to a
set of pipes, so that all of their output was prefixed with |jobnum|. It was
quite nice, but I never felt it was important enough to duplicate that
behavior for gmake.

--
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/


_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Paolo Bonzini-2]

Follow-up Comment #4, bug #22923 (project make):

We have implemented output merging with a shell wrapper script. We ask make
to use this wrapper script via the SHELL variable.

The script itself is pretty simple, although it may be bash-centric:

----------------------------------------
#!/bin/sh

# Simple shell wrapper for stdout and stderr buffering
# Created 2002, Erik Cumps
#
# Use with parallel makes to prevent confusing output.
# Run make SHELL=makesh ... or edit makefile.
#
# This version merges stdout/stderr.

cleanup() {
        if [ -f /tmp/makesh-$$ ]; then
                cat /tmp/makesh-$$
                rm -f /tmp/makesh-$$
        fi
        if [ "${RC}" = "beach" ]; then
                # interrupted before reaping child status or weird error
                echo "*** makesh: interrupted by signal before reaping child
status" 1>&2
                exit 42
        else
                exit ${RC}
        fi
}

# Run command, redirect stdout/stderr to temp file
# Remember command's exit status
RC="beach"
trap cleanup SIGHUP SIGINT SIGQUIT SIGTERM
/bin/bash -c "$@" 1>/tmp/makesh-$$ 2>&1
RC=$?

cleanup
----------------------------------------


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Paolo Bonzini-2]

Additional Item Attachment, bug #22923 (project make):

File name: makesh                         Size:0 KB
File name: makesh-split                   Size:0 KB


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make