Re: 3.1.6 system() fails when at the ulimit of open files

[Aharon Robbins]

Hi. I just tried this with gawk 3.1.7 and 3.1.6 on my Linux system
and I cannot reproduce it.

$ mkdir z
$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 32621
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32621
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
$ yes|head -2000|gawk-3.1.6 '{print "hi" > ("z/"z++)}END{system("echo hiho;date")}'
$ ls z | wc -l
2000

So, the usual questions:

1. What operating system?
2. What compiler?
3. Did you try recompiling from source if you used a pre-supplied binary?

Thanks,

Arnold

> Date: Wed, 14 Oct 2009 08:45:35 -0500
> Subject: 3.1.6 system() fails when at the ulimit of open files
> From: Chris Pearson <kermit4@...>
> To: bug-gawk@...
>
> With an open file limit of 1024:
>
> mkdir z
>
> yes|head -200|gawk '{print "hi" > ("z/"z++)}END{system("echo
> hiho;date")}'   # the system() call works
> yes|head -2000|gawk '{print "hi" > ("z/"z++)}END{system("echo
> hiho;date")}'   # the system() call doesnt work, even though all 2000
> files are created.
>
> So, I guess whatever decides to make room to open a new file getting
> called for system(), which is inconsistant with how open file juggling
> is handled by gawk in other cases, and probably easily fixed.   Or at
> the very least should print an error if not.

[Dave B-6]

On Wednesday 14 October 2009, Aharon Robbins wrote:

> Hi. I just tried this with gawk 3.1.7 and 3.1.6 on my Linux system
> and I cannot reproduce it.
>[snip]
> $ yes|head -2000|gawk-3.1.6 '{print "hi" > ("z/"z++)}END{system("echo
>  hiho;date")}'
>  $ ls z | wc -l
> 2000

Where's the output of the system() call there?

--
D.

[Aharon Robbins]

OK, ok, ok. This is what I get for trying to debug late at night.

It's a real bug. Patch is below. Thanks to Chris for reporting it.

Arnold

> From: Dave B <dave_br@...>
> To: bug-gawk@...
> Subject: Re: 3.1.6 system() fails when at the ulimit of open files
> Date: Wed, 14 Oct 2009 21:35:41 +0100
>
> On Wednesday 14 October 2009, Aharon Robbins wrote:
>
> > Hi. I just tried this with gawk 3.1.7 and 3.1.6 on my Linux system
> > and I cannot reproduce it.
> >[snip]
> > $ yes|head -2000|gawk-3.1.6 '{print "hi" > ("z/"z++)}END{system("echo
> >  hiho;date")}'
> >  $ ls z | wc -l
> > 2000
>
> Where's the output of the system() call there?
>
> --
> D.

Wed Oct 14 23:25:47 2009  Arnold D. Robbins  <arnold@...>

        * io.c (redirect): When opening a file, set the close-on-exec flag.
        Thanks to Chris Pearson <kermit4@...> for the bug report.

Index: io.c
===================================================================
RCS file: /d/mongo/cvsrep/gawk-stable/io.c,v
retrieving revision 1.23
diff -u -r1.23 io.c
--- io.c 9 Jul 2009 19:54:38 -0000 1.23
+++ io.c 14 Oct 2009 21:24:12 -0000
@@ -740,6 +740,7 @@
  else
 #endif
  rp->fp = fdopen(fd, (const char *) mode);
+ os_close_on_exec(fileno(rp->fp), str, "file", "");
  rp->mode = (const char *) mode;
  /* don't leak file descriptors */
  if (rp->fp == NULL)

[Andreas Schwab-2]

Aharon Robbins <arnold@...> writes:

> Index: io.c
> ===================================================================
> RCS file: /d/mongo/cvsrep/gawk-stable/io.c,v
> retrieving revision 1.23
> diff -u -r1.23 io.c
> --- io.c 9 Jul 2009 19:54:38 -0000 1.23
> +++ io.c 14 Oct 2009 21:24:12 -0000
> @@ -740,6 +740,7 @@
>   else
>  #endif
>   rp->fp = fdopen(fd, (const char *) mode);
> + os_close_on_exec(fileno(rp->fp), str, "file", "");
>   rp->mode = (const char *) mode;
>   /* don't leak file descriptors */
>   if (rp->fp == NULL)

If this is true then you crash above.

Andreas.

--
Andreas Schwab, schwab@...
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

[Aharon Robbins]

Andreas,

You're right. That's what I get for coding late at night. Attached is
a follow-on diff.

Thanks,

Arnold
----------------------------------------------------------
Fri Oct 16 08:41:29 2009  Arnold D. Robbins  <arnold@...>

        * io.c (redirect): Do the close-on-exec setting before
        attempt to get a FILE *. Thanks to Andreas Schwab.

Index: io.c
===================================================================
RCS file: /d/mongo/cvsrep/gawk-stable/io.c,v
retrieving revision 1.24
diff -u -r1.24 io.c
--- io.c 14 Oct 2009 21:29:57 -0000 1.24
+++ io.c 16 Oct 2009 06:48:33 -0000
@@ -731,16 +731,16 @@
  else if (fd == fileno(stderr))
  rp->fp = stderr;
  else {
+ const char *omode = mode;
 #if defined(F_GETFL) && defined(O_APPEND)
  int fd_flags;
 
  fd_flags = fcntl(fd, F_GETFL);
  if (fd_flags != -1 && (fd_flags & O_APPEND) == O_APPEND)
- rp->fp = fdopen(fd, binmode("a"));
- else
+ omode = binmode("a");
 #endif
- rp->fp = fdopen(fd, (const char *) mode);
- os_close_on_exec(fileno(rp->fp), str, "file", "");
+ os_close_on_exec(fd, str, "file", "");
+ rp->fp = fdopen(fd, (const char *) omode);
  rp->mode = (const char *) mode;
  /* don't leak file descriptors */
  if (rp->fp == NULL)

> From: Andreas Schwab <schwab@...>
> To: Aharon Robbins <arnold@...>
> Cc: bug-gawk@..., dave_br@..., kermit4@...
> Subject: Re: 3.1.6 system() fails when at the ulimit of open files
> Date: Thu, 15 Oct 2009 01:25:45 +0200
>
> Aharon Robbins <arnold@...> writes:
>
> > Index: io.c
> > ===================================================================
> > RCS file: /d/mongo/cvsrep/gawk-stable/io.c,v
> > retrieving revision 1.23
> > diff -u -r1.23 io.c
> > --- io.c 9 Jul 2009 19:54:38 -0000 1.23
> > +++ io.c 14 Oct 2009 21:24:12 -0000
> > @@ -740,6 +740,7 @@
> >   else
> >  #endif
> >   rp->fp = fdopen(fd, (const char *) mode);
> > + os_close_on_exec(fileno(rp->fp), str, "file", "");
> >   rp->mode = (const char *) mode;
> >   /* don't leak file descriptors */
> >   if (rp->fp == NULL)
>
> If this is true then you crash above.
>
> Andreas.
>
> --
> Andreas Schwab, schwab@...
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>