Parcellfun leaving defunct processes

[Travis Collier]

Parcellfun (from the General octave-forge package) is really
promising, but is not quite behaving.

Using octave 3.1.55 and the current download of parcellfun (no version
number on that sadly) under Ubuntu 9.04...

Something as simple as:
a = parcellfun(4,@(x)horzcat(1,x),num2cell([1:10]),'UniformOutput',false);
leaves a bunch of defunct octave processes laying around.

These get cleared when octave exits, but that is the only way I find
to remove them.  Not a problem for small jobs, but I'm literally
running into the wall of maximum number of processes.

It has been a while since I did any parallel programming outside of
mpi, so I'm not sure what the problem really is.
IIRC though, doesn't the parent process have to catch SIG_CHILD?  I
don't see that happening in the script.

BTW: If this problem gets sorted, I have a very minor bugfix and
improvement to submit.  They are pointless if the major bug doesn't
get fixed first though.

Travis Collier
UCLA biology grad student

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Travis Collier]

Answering my own email... bad form I suppose but I got it working
pretty easily...

parcellfun needs to call waitpid to clear those child processes.  Simple fix.

There was a minor bug with the options parsing... it failed when no
options were set.  Fixed with a simple protecting if.

I also included a 'VerboseLevel' option which allows for quiet (0),
normal (1), or more (2+) levels of verbosity to the output.
I'd humbly encourage including this in any reasonably complex
function... especially ones where debugging output is frequently
desirable.

Here is a diff against the version of parcellfun.m contained in
general-1.1.2 (the current one on octave-forge)... please drop me a
note if I should instead post the full file.

3a4
> ## Modified by: Travis Collier 31-5-2009
31a33,34
> ## A VerboseLevel option is supported controlling level of output.  0 implies quiet and 1 is default.
> ##
60a64
>   verbose_level = 1;
66,78c70,89
<   do
<     if (strcmp (args{nargs-1}, "UniformOutput"))
<       uniform_output = args{nargs};
<       nargs -= 2;
<       continue;
<     endif
<     if (strcmp (args{nargs-1}, "ErrorHandler"))
<       error_handler = args{nargs};
<       nargs -= 2;
<       continue;
<     endif
<     break;
<   until (nargs < 2);
---

>   if( nargs > 1 )
>     do
>       if (strcmp (args{nargs-1}, "UniformOutput"))
>         uniform_output = args{nargs};
>         nargs -= 2;
>         continue;
>       endif
>       if (strcmp (args{nargs-1}, "ErrorHandler"))
>         error_handler = args{nargs};
>         nargs -= 2;
>         continue;
>       endif
>       if (strcmp (args{nargs-1}, "VerboseLevel"))
>         verbose_level = args{nargs};
>         nargs -= 2;
>         continue;
>       endif
>       break;
>     until (nargs < 2);
>   end

112a124
>   child_pids = zeros(nproc,1); # have to keep track of child process ids
121a134,137
>       child_pids(nsuc) = pid;
>       if( verbose_level > 1 )
>         fprintf(stderr,'parcellfun: child processes %d created\n', child_pids(i));
>       end
285,286c301,304
<         fprintf (stderr, "\rparcellfun: %d/%d jobs done", pjobs -
sum (pending != 0), njobs);
<         fflush (stderr);
---
>         if( verbose_level > 0 )
>           fprintf (stderr, "\rparcellfun: %d/%d jobs done", pjobs - sum (pending != 0), njobs);
>           fflush (stderr);
>         endif
288,289c306,309
<       fputs (stderr, "\n");
<       fflush (stderr);
---
>       if( verbose_level > 0 )
>         fputs (stderr, "\n");
>         fflush (stderr);
>       endif
306a327,334
>       ## wait for all child processes to exit
>       for i = 1:nproc
>         if( verbose_level > 1 )
>           fprintf(stderr,'parcellfun: waiting for child processes %d to close\n', child_pids(i));
>         end
>       waitpid(child_pids(i));
>     end

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Jaroslav Hajek-2]

On Mon, Jun 1, 2009 at 6:52 AM, Travis Collier <travcollier@...> wrote:

> Parcellfun (from the General octave-forge package) is really
> promising, but is not quite behaving.
>
> Using octave 3.1.55 and the current download of parcellfun (no version
> number on that sadly) under Ubuntu 9.04...
>
> Something as simple as:
> a = parcellfun(4,@(x)horzcat(1,x),num2cell([1:10]),'UniformOutput',false);
> leaves a bunch of defunct octave processes laying around.
>
> These get cleared when octave exits, but that is the only way I find
> to remove them.  Not a problem for small jobs, but I'm literally
> running into the wall of maximum number of processes.
>
> It has been a while since I did any parallel programming outside of
> mpi, so I'm not sure what the problem really is.
> IIRC though, doesn't the parent process have to catch SIG_CHILD?  I
> don't see that happening in the script.
>
> BTW: If this problem gets sorted, I have a very minor bugfix and
> improvement to submit.  They are pointless if the major bug doesn't
> get fixed first though.
>
> Travis Collier
> UCLA biology grad student
>

Hi Travis,

thanks for your suggestion. I have just checked in a patch that stores
the PIDs and calls waitpid after everything is done; this should
ensure that all children are reaped. Your example also discovered a
bug in the pipe handling, which I hopefully fixed. More in the next
mail.

regards

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Jaroslav Hajek-2]

On Mon, Jun 1, 2009 at 8:37 AM, Travis Collier <travcollier@...> wrote:

> Answering my own email... bad form I suppose but I got it working
> pretty easily...
>
> parcellfun needs to call waitpid to clear those child processes.  Simple fix.
>
> There was a minor bug with the options parsing... it failed when no
> options were set.  Fixed with a simple protecting if.
>
> I also included a 'VerboseLevel' option which allows for quiet (0),
> normal (1), or more (2+) levels of verbosity to the output.
> I'd humbly encourage including this in any reasonably complex
> function... especially ones where debugging output is frequently
> desirable.
>
> Here is a diff against the version of parcellfun.m contained in
> general-1.1.2 (the current one on octave-forge)... please drop me a
> note if I should instead post the full file.
>

Hi again,

I'm glad to see someone else contributing to parcellfun (or the
"general" package). The suggested improvement as well as bug fix are
OK, but I have a couple of suggestions/advices:

1. Never send context-free patches (such as the one you posted) - they
are too risky to apply. Use the -c or -u switch to diff (or see 2.
below). For more info, see `man diff'.

2. If possible, use the SVN sources to work with. Under unixy OSes,
this should be as easy as doing
svn co https://octave.svn.sourceforge.net/viewvc/octave/trunk/octave-forge/main/general
in a suitable directory (this will check out just the general package,
strip the path if you want the whole tree).
After you edit some files in the working copy, you can just use `svn
diff' to make a proper diff and send that over the mail, preferably as
an attachment with MIME type text/x-patch, text/x-diff or text/plain
(I think Gmail will do this automatically if you use the .diff or
.patch extension). "svn diff" will always generate a proper patch, so
you can ignore advice 1. if you do so.
Pretty convenient SVN clients also exist for Windows and Mac.

3. Don't mix logically independent changes. Yes, "logically
independent" is vague - you should use your common sense, but some
basic rules of thumb exist. For instance, you shouldn't mix an
extension with a bug fix (which apparently you just attempted to),
even if the bug fix is trivial. Make it two separate patches.

4. If you find yourself making more contributions, consider getting a
SourceForge account and join the OctaveForge project.

Please re-post your patch taking these under consideration (and of
course ask if you have doubts).

best regards

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Travis Collier]

Here is a diff adding a "VerboseLevel" option to parcellfun.
It is against the most current svn snapshot, as of about 15 minutes
ago at least ;)

Anyway, it may seem like a dumb thing, but I really need to be able to
suppress the output to keep track of what is going on at higher levels
(some using the backspace trick for 'counter' output).
It also allows cranking up the verbosity level, which for a function
like this (and many others) can be very helpful debugging.

If anyone has a suggestion of a favored --diff-cmd option (less
verbose or at least treating whitespace a bit more intelligently), it
would be appreciated for future reference.

PS: parcellfun from general 1.1.3 tar.gz posted on octave-forge breaks
in an unhappy core-dump way for me (fresh octave 3.2.0 install from
source on ubuntu jaunty).

--- parcellfun.m (revision 5947)
+++ parcellfun.m (working copy)
@@ -1,17 +1,17 @@
 ## Copyright (C) 2009 VZLU Prague, a.s., Czech Republic
 ##
 ## Author: Jaroslav Hajek
-##
+##
 ## This program is free software; you can redistribute it and/or modify
 ## it under the terms of the GNU General Public License as published by
 ## the Free Software Foundation; either version 3 of the License, or
 ## (at your option) any later version.
-##
+##
 ## This program is distributed in the hope that it will be useful,
 ## but WITHOUT ANY WARRANTY; without even the implied warranty of
 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 ## GNU General Public License for more details.
-##
+##
 ## You should have received a copy of the GNU General Public License
 ## along with this program; see the file COPYING.  If not, see
 ## <http://www.gnu.org/licenses/>.
@@ -22,16 +22,17 @@
 ## @deftypefnx{Function File} parcellfun (nproc, fun, @dots{},
"ErrorHandler", @var{errfunc})
 ## Evaluates a function for multiple argument sets using multiple processes.
 ## @var{nproc} should specify the number of processes. A maximum
recommended value is
-## equal to number of CPUs on your machine or one less.
+## equal to number of CPUs on your machine or one less.
 ## @var{fun} is a function handle pointing to the requested
evaluating function.
 ## @var{a1}, @var{a2} etc. should be cell arrays of equal size.
 ## @var{o1}, @var{o2} etc. will be set to corresponding output arguments.
 ##
 ## The UniformOutput and ErrorHandler options are supported with
meaning identical
 ## to @dfn{cellfun}.
+## A VerboseLevel option is supported controlling level of output.  0
implies quiet and 1 is default.
 ##
 ## NOTE: this function is implemented using "fork" and a number of
pipes for IPC.
-## Suitable for systems with an efficient "fork" implementation (such
as GNU/Linux),
+## Suitable for systems with an efficient "fork" implementation (such
as GNU/Linux),
 ## on other systems (Windows) it should be used with caution.
 ## Also, if you use a multithreaded BLAS, it may be wise to turn off
multi-threading
 ## when using this function.
@@ -41,11 +42,11 @@
 ## especially if unhandled errors occur. Under GNU and compatible
systems, the following
 ## shell command may be used to display orphaned Octave processes:
 ## ps --ppid 1 | grep octave
-##
+##
 ## @end deftypefn

 function varargout = parcellfun (nproc, fun, varargin)
-
+
   if (nargin < 3 || ! isscalar (nproc) || nproc <= 0)
     print_usage ();
   endif
@@ -58,6 +59,7 @@

   uniform_output = true;
   error_handler = [];
+  verbose_level = 1; # controls verbosity of terminal output

   args = varargin;
   nargs = length (varargin);
@@ -75,12 +77,17 @@
         nargs -= 2;
         continue;
       endif
+      if (strcmp (args{nargs-1}, "VerboseLevel"))
+        verbose_level = args{nargs};
+        nargs -= 2;
+        continue;
+      endif
       break;
     until (nargs < 2);
   endif

   args = args(1:nargs);
-
+
   if (length (args) == 0)
     print_usage ();
   elseif (length (args) > 1 && ! size_equal (args{:}))
@@ -124,6 +131,10 @@
       ## parent process. fork succeded.
       nsuc ++;
       pids(i) = pid;
+      if( verbose_level > 1 )
+        fprintf(stderr,'parcellfun: child processes %d created\n', pids(i));
+        fflush (stderr);
+      endif
     elseif (pid == 0)
       ## child process.
       iproc = i;
@@ -250,7 +261,7 @@
     end_unwind_protect

   else
-    ## parent process.
+    ## parent process.
     njobs = numel (varargin{1});
     res = cell (nargout, njobs);

@@ -296,11 +307,15 @@
           ## clear pending state
           pending(isubp) = 0;
         endif
-        fprintf (stderr, "\rparcellfun: %d/%d jobs done", pjobs - sum
(pending != 0), njobs);
+        if( verbose_level > 0 )
+          fprintf (stderr, "\rparcellfun: %d/%d jobs done", pjobs -
sum (pending != 0), njobs);
+          fflush (stderr);
+        endif
+      endwhile
+      if( verbose_level > 0 )
+        fputs (stderr, "\n");
         fflush (stderr);
-      endwhile
-      fputs (stderr, "\n");
-      fflush (stderr);
+      endif

     unwind_protect_cleanup

@@ -314,6 +329,10 @@
       ## close all pipe ends
       fclose (statr);
       for i = 1:nproc
+        if( verbose_level > 1 )
+          fprintf(stderr,'parcellfun: waiting for child processes %d
to close\n', pids(i));
+          fflush (stderr);
+        end
         fclose (resr(i));
       endfor

@@ -333,7 +352,7 @@
         varargout{i} = cell2mat (varargout{i});
       endif
     endfor
-
+
   endif

 endfunction

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Jaroslav Hajek-2]

On Mon, Jun 15, 2009 at 12:51 PM, Travis Collier<travcollier@...> wrote:

> Here is a diff adding a "VerboseLevel" option to parcellfun.
> It is against the most current svn snapshot, as of about 15 minutes
> ago at least ;)
>
> Anyway, it may seem like a dumb thing, but I really need to be able to
> suppress the output to keep track of what is going on at higher levels
> (some using the backspace trick for 'counter' output).
> It also allows cranking up the verbosity level, which for a function
> like this (and many others) can be very helpful debugging.
>
> If anyone has a suggestion of a favored --diff-cmd option (less
> verbose or at least treating whitespace a bit more intelligently), it
> would be appreciated for future reference.
>
> PS: parcellfun from general 1.1.3 tar.gz posted on octave-forge breaks
> in an unhappy core-dump way for me (fresh octave 3.2.0 install from
> source on ubuntu jaunty).
>

It seems your mailer garbled the patch (wrapped long lines), so that
it does not apply. Could you please repost this as an attachement?
Preferably as text/plain, text/x-diff or text/x-patch (if you'll use
.diff or .patch extension, Gmail will do it correctly).

thanks

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Jaroslav Hajek-2]

On Mon, Jun 15, 2009 at 12:51 PM, Travis Collier<travcollier@...> wrote:

> Here is a diff adding a "VerboseLevel" option to parcellfun.
> It is against the most current svn snapshot, as of about 15 minutes
> ago at least ;)
>
> Anyway, it may seem like a dumb thing, but I really need to be able to
> suppress the output to keep track of what is going on at higher levels
> (some using the backspace trick for 'counter' output).
> It also allows cranking up the verbosity level, which for a function
> like this (and many others) can be very helpful debugging.
>
> If anyone has a suggestion of a favored --diff-cmd option (less
> verbose or at least treating whitespace a bit more intelligently), it
> would be appreciated for future reference.
>
> PS: parcellfun from general 1.1.3 tar.gz posted on octave-forge breaks
> in an unhappy core-dump way for me (fresh octave 3.2.0 install from
> source on ubuntu jaunty).

Do you have an example that I can test? Anyway, there have been a few
fixes since the release, I think, so you can try updating from SVN.


--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Travis Collier]

Sorry, I've blasted that installed version so I can't test it again.
However, very simple commands were hanging and/or core dumping with a
broken pipe error:
a = parcellfun( 2, @(x) (x+1), num2cell([1 2 3]) )

What was a bit odd, and took me a minute to figure out, was that if
nproc was set at 1, everything worked fine.  I don't see an obvious
"if nproc == 1 don't bother to fork" (which would actually make a lot
of sense to include), so I don't know what was really causing the
difference.  Sorry, no time to investigate properly.

Basically, don't just test with nproc = 1.

Attached is that VerboseLevel patch one more time.  It really is
trivial, but I guess it is a good opportunity to learn the proper way
of contributing to a distributed development project like this.
The other way of doing it is to have the output as warnings which can
be switched off, but that can get complicated and annoying really
fast.

PS: Any reason the output is to stderr instead of stdout?  Seems like
a bad practice... especially using the '\r' trick on stderr.  I'm
surprised that actually works, since that is more of a bug-in-waiting
than a feature.
I personally like setting the nominal output stream as a variable up
front (just set to stdout 99.9% of the time).

2009/6/15 Jaroslav Hajek <highegg@...>:

> On Mon, Jun 15, 2009 at 12:51 PM, Travis Collier<travcollier@...> wrote:
>> Here is a diff adding a "VerboseLevel" option to parcellfun.
>> It is against the most current svn snapshot, as of about 15 minutes
>> ago at least ;)
>>
>> Anyway, it may seem like a dumb thing, but I really need to be able to
>> suppress the output to keep track of what is going on at higher levels
>> (some using the backspace trick for 'counter' output).
>> It also allows cranking up the verbosity level, which for a function
>> like this (and many others) can be very helpful debugging.
>>
>> If anyone has a suggestion of a favored --diff-cmd option (less
>> verbose or at least treating whitespace a bit more intelligently), it
>> would be appreciated for future reference.
>>
>> PS: parcellfun from general 1.1.3 tar.gz posted on octave-forge breaks
>> in an unhappy core-dump way for me (fresh octave 3.2.0 install from
>> source on ubuntu jaunty).
>
> Do you have an example that I can test? Anyway, there have been a few
> fixes since the release, I think, so you can try updating from SVN.
>
>
> --
> RNDr. Jaroslav Hajek
> computing expert & GNU Octave developer
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz
>

[parcellfun.m.diff]

---

Index: parcellfun.m
===================================================================
--- parcellfun.m (revision 5958)
+++ parcellfun.m (working copy)
@@ -1,17 +1,17 @@
 ## Copyright (C) 2009 VZLU Prague, a.s., Czech Republic
 ##
 ## Author: Jaroslav Hajek
-##
+##
 ## This program is free software; you can redistribute it and/or modify
 ## it under the terms of the GNU General Public License as published by
 ## the Free Software Foundation; either version 3 of the License, or
 ## (at your option) any later version.
-##
+##
 ## This program is distributed in the hope that it will be useful,
 ## but WITHOUT ANY WARRANTY; without even the implied warranty of
 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 ## GNU General Public License for more details.
-##
+##
 ## You should have received a copy of the GNU General Public License
 ## along with this program; see the file COPYING.  If not, see
 ## <http://www.gnu.org/licenses/>.
@@ -22,16 +22,19 @@
 ## @deftypefnx{Function File} parcellfun (nproc, fun, @dots{}, "ErrorHandler", @var{errfunc})
 ## Evaluates a function for multiple argument sets using multiple processes.
 ## @var{nproc} should specify the number of processes. A maximum recommended value is
-## equal to number of CPUs on your machine or one less.
+## equal to number of CPUs on your machine or one less.
 ## @var{fun} is a function handle pointing to the requested evaluating function.
 ## @var{a1}, @var{a2} etc. should be cell arrays of equal size.
 ## @var{o1}, @var{o2} etc. will be set to corresponding output arguments.
 ##
 ## The UniformOutput and ErrorHandler options are supported with meaning identical
 ## to @dfn{cellfun}.
+## A VerboseLevel option controlling the level output is supported.  
+## A value of 0 is quiet, 1 is normal, and 2 or more enables
+## debugging output.
 ##
 ## NOTE: this function is implemented using "fork" and a number of pipes for IPC.
-## Suitable for systems with an efficient "fork" implementation (such as GNU/Linux),
+## Suitable for systems with an efficient "fork" implementation (such as GNU/Linux),
 ## on other systems (Windows) it should be used with caution.
 ## Also, if you use a multithreaded BLAS, it may be wise to turn off multi-threading
 ## when using this function.
@@ -41,11 +44,11 @@
 ## especially if unhandled errors occur. Under GNU and compatible systems, the following
 ## shell command may be used to display orphaned Octave processes:
 ## ps --ppid 1 | grep octave
-##
+##
 ## @end deftypefn
 
 function varargout = parcellfun (nproc, fun, varargin)
-  
+
   if (nargin < 3 || ! isscalar (nproc) || nproc <= 0)
     print_usage ();
   endif
@@ -58,6 +61,7 @@
 
   uniform_output = true;
   error_handler = [];
+  verbose_level = 1; # default to normal output level
 
   args = varargin;
   nargs = length (varargin);
@@ -75,12 +79,17 @@
         nargs -= 2;
         continue;
       endif
+      if (strcmp (args{nargs-1}, "VerboseLevel"))
+        error_handler = args{nargs};
+        nargs -= 2;
+        continue;
+      endif
       break;
     until (nargs < 2);
   endif
 
   args = args(1:nargs);
-  
+
   if (length (args) == 0)
     print_usage ();
   elseif (length (args) > 1 && ! size_equal (args{:}))
@@ -124,6 +133,10 @@
       ## parent process. fork succeded.
       nsuc ++;
       pids(i) = pid;
+      if( verbose_level > 1 )
+        fprintf(stderr,'parcellfun: child processes %d created\n', pids(i));
+        fflush (stderr);
+      endif
     elseif (pid == 0)
       ## child process.
       iproc = i;
@@ -250,7 +263,7 @@
     end_unwind_protect
 
   else
-    ## parent process.
+    ## parent process.
     njobs = numel (varargin{1});
     res = cell (nargout, njobs);
 
@@ -296,11 +309,15 @@
           ## clear pending state
           pending(isubp) = 0;
         endif
-        fprintf (stderr, "\rparcellfun: %d/%d jobs done", pjobs - sum (pending != 0), njobs);
+        if( verbose_level > 0 )
+          fprintf (stderr, "\rparcellfun: %d/%d jobs done", pjobs - sum (pending != 0), njobs);
+          fflush (stderr);
+        endif
+      endwhile
+      if( verbose_level > 0 )
+        fputs (stderr, "\n");
         fflush (stderr);
-      endwhile
-      fputs (stderr, "\n");
-      fflush (stderr);
+      endif
 
     unwind_protect_cleanup
 
@@ -319,6 +336,10 @@
 
       ## explicitly recognize all terminated processes.
       for i = 1:nproc
+        if( verbose_level > 1 )
+          fprintf(stderr,'parcellfun: waiting for child processes %d to close\n', pids(i));
+          fflush (stderr);
+        end
         [pid, status] = waitpid (pids(i));
       endfor
 
@@ -333,7 +354,7 @@
         varargout{i} = cell2mat (varargout{i});
       endif
     endfor
-    
+
   endif
 
 endfunction

---

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Travis Collier]

One more thing for parcellfun.  If nproc > the number of jobs, it
breaks in a not nice way.

a = parcellfun(4, @(x) (x+1), num2cell([1:2]))

A bit more complete test:
a = parcellfun(4, @(x,y) (x+y), num2cell([1:2]), num2cell([1:2]))

Easy one line fix as the last thing in the argument checking (just
after the "arguments size much match" block):
if (nproc > numel(args{1}) ), nproc = numel(args{1}); endif

patch attached anyway

[parcellfun.m.2.diff]

---

Index: parcellfun.m
===================================================================
--- parcellfun.m (revision 5958)
+++ parcellfun.m (working copy)
@@ -87,6 +87,8 @@
     error ("arguments size must match");
   endif
 
+  if (nproc > numel(args{1}) ), nproc = numel(args{1}); endif
+
   ## create communication pipes.
   cmdr = cmdw = resr = resw = zeros (nproc, 1);
   err = 0;

---

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Jaroslav Hajek-2]

On Wed, Jun 17, 2009 at 5:30 AM, Travis Collier<travcollier@...> wrote:

> Sorry, I've blasted that installed version so I can't test it again.
> However, very simple commands were hanging and/or core dumping with a
> broken pipe error:
> a = parcellfun( 2, @(x) (x+1), num2cell([1 2 3]) )
>
> What was a bit odd, and took me a minute to figure out, was that if
> nproc was set at 1, everything worked fine.  I don't see an obvious
> "if nproc == 1 don't bother to fork" (which would actually make a lot
> of sense to include), so I don't know what was really causing the
> difference.  Sorry, no time to investigate properly.
>

Works normally for me. However, if by any chance you compile Octave
from sources (like me), updating the sources from time to time, be
aware that a package often needs to be recompiled when the binary
interface changes, and that happens (and happened recently, in fact I
needed to recompile the package just now). When using releases, you
should actually get an error message of incompatible ABI, but of
course in development sources we don't bother to increase the ABI
counter after every change.


> Basically, don't just test with nproc = 1.
>
> Attached is that VerboseLevel patch one more time.  It really is
> trivial, but I guess it is a good opportunity to learn the proper way
> of contributing to a distributed development project like this.
> The other way of doing it is to have the output as warnings which can
> be switched off, but that can get complicated and annoying really
> fast.

OK, thanks. I applied this but corrected a few typos (like "child
processes %d forked" -> "child process %d forked")
and actually you pleased all who believe in Murphy's law by managing
to make a bug here:

@@ -75,12 +79,17 @@
         nargs -= 2;
         continue;
       endif
+      if (strcmp (args{nargs-1}, "VerboseLevel"))
+        error_handler = args{nargs};    <--- BUG
+        nargs -= 2;
+        continue;
+      endif
       break;
     until (nargs < 2);
   endif

   args = args(1:nargs);

But in any case, thanks for the extension.

> PS: Any reason the output is to stderr instead of stdout?  Seems like
> a bad practice... especially using the '\r' trick on stderr.  I'm
> surprised that actually works, since that is more of a bug-in-waiting
> than a feature.
> I personally like setting the nominal output stream as a variable up
> front (just set to stdout 99.9% of the time).

There's a very good reason actually, and it's the fact that normally,
what's written to stdout in Octave is piped through the pager (`less'
or `more'), which is totally useless for an "animated" progress track
(hence the '\r' trick). Stderr is output directly. Admittedly, it is
still a kind of abuse, but I think it's better than temporarily
disabling the pager (which is another solution), because then you need
to be careful to restore the state. Of course, even better would be if
Octave supported a default stream like stdoutnp bypassing the pager.
'\r' works well on UNIXes since the dawn of time, and I think Windows
terminal also supports it.

regards

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Jaroslav Hajek-2]

On Wed, Jun 17, 2009 at 5:58 AM, Travis Collier<travcollier@...> wrote:
> One more thing for parcellfun.  If nproc > the number of jobs, it
> breaks in a not nice way.
>
> a = parcellfun(4, @(x) (x+1), num2cell([1:2]))
>
> A bit more complete test:
> a = parcellfun(4, @(x,y) (x+y), num2cell([1:2]), num2cell([1:2]))


Ah, I see, there was a bug in handling processes that never recieve
jobs. Got that fixed now. Thanks!

> Easy one line fix as the last thing in the argument checking (just
> after the "arguments size much match" block):
> if (nproc > numel(args{1}) ), nproc = numel(args{1}); endif
>
> patch attached anyway
>

In fact the bug was deeper; it can happen (though it's extremely
unlikely) than when you have 3 jobs and 3 processes, for whatever
reason the distribution will be 2-1-0 in which case the bug would have
still occured.
But cutting the excessive processes is surely a good idea anyway, so I
applied the patch, then after some thought
I simplified that a little to

nproc = min (nproc, numel (args{1}));

so, parcellfun is a bit smarter again. Thanks.
regards

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev