bad awk output on vkernel

[YONETANI Tomokazu-4]

Hello.
Am I only one, or is anyone else seeing this problem too?
I don't remember when, but running `make buildworld' in vkernel stopped
working a few months ago.  It always fails while it's building gcc4.4
in _gcc44_tools.  I managed to narrow down to an awk script in gcc producing
inconsistent output, here's the script which can reproduce the behavior:

  #!/bin/sh
  prefix=/usr/src/gnu/usr.bin/cc41/cc_prep
  cc41contrib=$prefix/../../../../contrib/gcc-4.1

  list=`seq 0 20`
  first=
  while :; do
          last=$(env - PATH=$PATH \
          /usr/bin/awk -f $cc41contrib/gcc/opt-gather.awk \
          $cc41contrib/gcc/c.opt \
          $cc41contrib/gcc/common.opt \
          $cc41contrib/gcc/config/i386/i386.opt)
          md5=$(echo "$last" | /sbin/md5)
          echo $md5
          if [ -z "$first" ]; then
                  first="$last"
                  first_md5=$md5
          elif [ "$md5" != "$first_md5" ]; then
                  echo "$first" > first
                  echo "$last" > last
                  break
          fi
  done

Things I've already tried include:
- running the same script on native DragonFly box (always constistent md5)
- running on an SMP and non-SMP VKERNEL (both reproduced the problem)
- md5 on the source tree is always consistent
- running this script for source tree in the local filesystem on vkernel,
  and on an NFS-exported on from vkernel host (both reproduced the problem)
- using awk program statically-linked on DragonFly host machine (NG)

I made sure that the vkernel and the world are in sync.  If you have any
other idea to try, please let me know.

Thanks.

[Matthew Dillon]

:Hello.
:Am I only one, or is anyone else seeing this problem too?
:I don't remember when, but running `make buildworld' in vkernel stopped
:working a few months ago.  It always fails while it's building gcc4.4
:in _gcc44_tools.  I managed to narrow down to an awk script in gcc producing
:inconsistent output, here's the script which can reproduce the behavior:

    gcc4.4 was only added recently, but your script seems to messing
    around with gcc4.1 so its a bit confusing.

    In anycase, try building with NO_GCC44=1

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Simon 'corecode' Schubert]

Matthew Dillon wrote:
> :Hello.
> :Am I only one, or is anyone else seeing this problem too?
> :I don't remember when, but running `make buildworld' in vkernel stopped
> :working a few months ago.  It always fails while it's building gcc4.4
> :in _gcc44_tools.  I managed to narrow down to an awk script in gcc producing
> :inconsistent output, here's the script which can reproduce the behavior:
>
>     gcc4.4 was only added recently, but your script seems to messing
>     around with gcc4.1 so its a bit confusing.

I think he only mistyped 44.  The shell code only references 41, and it seems to imply that awk is producing a different output sometimes.

cheers
  simon

[YONETANI Tomokazu-4]

On Thu, Aug 13, 2009 at 07:37:54AM -0700, Matthew Dillon wrote:

>
> :Hello.
> :Am I only one, or is anyone else seeing this problem too?
> :I don't remember when, but running `make buildworld' in vkernel stopped
> :working a few months ago.  It always fails while it's building gcc4.4
> :in _gcc44_tools.  I managed to narrow down to an awk script in gcc producing
> :inconsistent output, here's the script which can reproduce the behavior:
>
>     gcc4.4 was only added recently, but your script seems to messing
>     around with gcc4.1 so its a bit confusing.

Ah, yes, I originally wrote the script before gcc4.4 was imported, and
at that time the buildworld failed while trying to build gcc4.1.

>     In anycase, try building with NO_GCC44=1

I wasn't clear abaut the `inconsistency'; by `inconsistent' I meant that
the output of awk script differs between runs, and that's why I'm using
md5 command.  The script terminates after running 180 iterations, or
when the output of awk script differs from that of the first iteration.

[Matthew Dillon]

    Is this with the latest master ?   I did fix some pipe bugs last week
    which could very well cause this sort of behavior (where the data gets
    corrupted through the pipe).  I'm pretty sure I got all the bugs, there
    shouldn't be any left.

    If it isn't that I'll have to try to reproduce it on a test bed and
    then track the issue down.

                                                -Matt

[YONETANI Tomokazu-4]

On Thu, Aug 13, 2009 at 11:48:18PM -0700, Matthew Dillon wrote:
>     Is this with the latest master ?   I did fix some pipe bugs last week
>     which could very well cause this sort of behavior (where the data gets
>     corrupted through the pipe).  I'm pretty sure I got all the bugs, there
>     shouldn't be any left.
>
>     If it isn't that I'll have to try to reproduce it on a test bed and
>     then track the issue down.

I'd say it's not a pipe issue, but specific to running awk on vkernel,
because this perl version never produces bad output on vkernel:

  my $SUBSEP = "\034";
  my $flag = 0;
  my @record;
  while (<>) {
      chomp;
      if (/^[ \t]*(;|$)/) {
  $flag = 0;
  next;
      }
      if (/^[^ \t]/) {
  if ($flag == 0) {
     push(@record, $_);
     $flag = 1;
  }
  else {
     my $l = @record - 1;
     $record[$l] .= $SUBSEP;
     $record[$l] .= $_;
  }
      }
  }
 
  foreach (sort @record) {
      print $_, "\n";
  }

My guess is that awk on vkernel has problem handling an array of long
strings, especially its elements are assigned to in a non-sequential way;
if I commented the line calling sort() function and it greatly reduced
the probability of inconsistent output :)

I also tried gawk from pkgsrc, but it seems to have similar issue.
Probably I need to take a closer look at the source code.