Error handling question

[Ciprian Dorin, Craciun]

    Shouldn't any of the following scripts print `error`? (Bash
4.0.35(2)-release on ArchLinux.)

    Or I've miss-interpreted the documentation...

    Thanks,
    Ciprian.


~~~~
set -e -o pipefail
( false ; echo ok ; ) || echo error
~~~~

~~~~
set -e -o pipefail
( false ; echo ok ; ) | true || echo error
~~~~

~~~~
set -e -o pipefail
{ false ; echo ok ; } || echo error
~~~~

~~~~
set -e -o pipefail
{ false ; echo ok ; } | true || echo error
~~~~

[Chet Ramey]

Ciprian Dorin, Craciun wrote:

>     Shouldn't any of the following scripts print `error`? (Bash
> 4.0.35(2)-release on ArchLinux.)
>
>     Or I've miss-interpreted the documentation...
>
>     Thanks,
>     Ciprian.
>
>
> ~~~~
> set -e -o pipefail
> ( false ; echo ok ; ) || echo error
> ~~~~
>
> ~~~~
> set -e -o pipefail
> ( false ; echo ok ; ) | true || echo error
> ~~~~
>
> ~~~~
> set -e -o pipefail
> { false ; echo ok ; } || echo error
> ~~~~
>
> ~~~~
> set -e -o pipefail
> { false ; echo ok ; } | true || echo error
> ~~~~

No.  Since `set -e' has no effect on the left side of the || or &&
operators, all of the commands preceding the || exit with status 0.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

[Ciprian Dorin, Craciun]

On Mon, Nov 9, 2009 at 4:49 AM, Chet Ramey <chet.ramey@...> wrote:

> Ciprian Dorin, Craciun wrote:
>>     Shouldn't any of the following scripts print `error`? (Bash
>> 4.0.35(2)-release on ArchLinux.)
>>
>>     Or I've miss-interpreted the documentation...
>>
>>     Thanks,
>>     Ciprian.
>>
>>
>> ~~~~
>> set -e -o pipefail
>> ( false ; echo ok ; ) || echo error
>> ~~~~
>>
>> ~~~~
>> set -e -o pipefail
>> ( false ; echo ok ; ) | true || echo error
>> ~~~~
>>
>> ~~~~
>> set -e -o pipefail
>> { false ; echo ok ; } || echo error
>> ~~~~
>>
>> ~~~~
>> set -e -o pipefail
>> { false ; echo ok ; } | true || echo error
>> ~~~~
>
> No.  Since `set -e' has no effect on the left side of the || or &&
> operators, all of the commands preceding the || exit with status 0.
>
> Chet
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>                 ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

    Sorry, but I don't understand at all... So please bare with me and
make me understand.

    So I've interpreted `set -e` as a way to tell bash to treat any
process exiting with non-zero (and not succeeded by a || ), as an
error and end the current shell / sub-shell.

    Thus if I say: `set -e ; { false ; true ; }` it works, but when I
put the `||`, it doesn't...

    So my question is how can I solve this problem? (And obtain the
needed behaviour.) (I see `()` and `{}` as blocks in normal
programming languages (of course with some particularities), and
non-zero exit codes as exceptions. And this is very helpful to write
robust bash scripts.)

    Thanks,
    Ciprian.

[Jan Schampera]

Ciprian Dorin, Craciun schrieb:

>     Thus if I say: `set -e ; { false ; true ; }` it works, but when I
> put the `||`, it doesn't...

I think it's because { ...; } isn't a simple command (however, its
components are).

J.

[Ciprian Dorin, Craciun]

On Mon, Nov 9, 2009 at 8:23 AM, Ciprian Dorin, Craciun
<ciprian.craciun@...> wrote:

> On Mon, Nov 9, 2009 at 4:49 AM, Chet Ramey <chet.ramey@...> wrote:
>> Ciprian Dorin, Craciun wrote:
>>>     Shouldn't any of the following scripts print `error`? (Bash
>>> 4.0.35(2)-release on ArchLinux.)
>>>
>>>     Or I've miss-interpreted the documentation...
>>>
>>>     Thanks,
>>>     Ciprian.
>>>
>>>
>>> ~~~~
>>> set -e -o pipefail
>>> ( false ; echo ok ; ) || echo error
>>> ~~~~
>>>
>>> ~~~~
>>> set -e -o pipefail
>>> ( false ; echo ok ; ) | true || echo error
>>> ~~~~
>>>
>>> ~~~~
>>> set -e -o pipefail
>>> { false ; echo ok ; } || echo error
>>> ~~~~
>>>
>>> ~~~~
>>> set -e -o pipefail
>>> { false ; echo ok ; } | true || echo error
>>> ~~~~
>>
>> No.  Since `set -e' has no effect on the left side of the || or &&
>> operators, all of the commands preceding the || exit with status 0.
>>
>> Chet
>> --
>> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>>                 ``Ars longa, vita brevis'' - Hippocrates
>> Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/
>
>
>    Sorry, but I don't understand at all... So please bare with me and
> make me understand.
>
>    So I've interpreted `set -e` as a way to tell bash to treat any
> process exiting with non-zero (and not succeeded by a || ), as an
> error and end the current shell / sub-shell.
>
>    Thus if I say: `set -e ; { false ; true ; }` it works, but when I
> put the `||`, it doesn't...
>
>    So my question is how can I solve this problem? (And obtain the
> needed behaviour.) (I see `()` and `{}` as blocks in normal
> programming languages (of course with some particularities), and
> non-zero exit codes as exceptions. And this is very helpful to write
> robust bash scripts.)
>
>    Thanks,
>    Ciprian.

    I'm also pasting from the documentation.

    The interesting part is :`The shell does not exit if the command
that fails is part of the command list immediately following [...]
part  of  any command  executed  in  a && or |⎪ list, [...], The
option applies to [...] each sub-shell environment [...] and may cause
subshells to exit before executing all the commands in the subshell`.

    So from this I understand that the (parent) shell doesn't exit if
the (child) sub-shell fails and is at left of `||`. But that the
subshell inherits the `-e` option, and should exit.

    Ciprian.

~~~~
              -e      Exit immediately if a pipeline (which may
consist of a single simple command),
                      a subshell command enclosed in parentheses, or
one of  the  commands  exe‐
                      cuted  as  part  of a command list enclosed by
braces (see SHELL GRAMMAR above)
                      exits with a non-zero status.  The shell does
not exit if the command that
                      fails is part of the command list immediately
following a while or until keyword, part
                      of the test following the if or elif reserved
words,  part  of  any
                      command  executed  in  a && or ⎪⎪ list except
the command following the final
                      && or ⎪⎪, any command in a pipeline but the
last, or if the command's return
                      value is being inverted with !.  A trap on ERR,
if set, is executed before the shell exits.
                      This option applies to the shell environment
and  each  sub‐
                      shell environment separately (see COMMAND
EXECUTION ENVIRONMENT above),
                      and may cause subshells to exit before executing
all the commands in the subshell.
~~~~

[Ciprian Dorin, Craciun]

On Mon, Nov 9, 2009 at 8:35 AM, Jan Schampera <jan.schampera@...> wrote:
> Ciprian Dorin, Craciun schrieb:
>
>>     Thus if I say: `set -e ; { false ; true ; }` it works, but when I
>> put the `||`, it doesn't...
>
> I think it's because { ...; } isn't a simple command (however, its
> components are).
>
> J.


    But then how can I solve the problem? (How about `()` which
clearly is a new shell instance.)

[Greg Wooledge]

On Mon, Nov 09, 2009 at 08:39:57AM +0200, Ciprian Dorin, Craciun wrote:
>     But then how can I solve the problem? (How about `()` which
> clearly is a new shell instance.)

The problem being "how to use set -e in a consistent manner across all
shells"?  You can't.  set -e is unpredictable, unreliable, and should be
shunned.  As you can see by the last dozen or so message on this mailing
list, not even bash gurus (other than Chet) can figure out its semantics.

Check for errors yourself on the commands that matter.  That way you'll
know what will cause termination and what won't.  Yes, I know, it means
your script is longer.  But longer is better than unpredictable.

[Ciprian Dorin, Craciun]

On Mon, Nov 9, 2009 at 3:37 PM, Greg Wooledge <wooledg@...> wrote:

> On Mon, Nov 09, 2009 at 08:39:57AM +0200, Ciprian Dorin, Craciun wrote:
>>     But then how can I solve the problem? (How about `()` which
>> clearly is a new shell instance.)
>
> The problem being "how to use set -e in a consistent manner across all
> shells"?  You can't.  set -e is unpredictable, unreliable, and should be
> shunned.  As you can see by the last dozen or so message on this mailing
> list, not even bash gurus (other than Chet) can figure out its semantics.
>
> Check for errors yourself on the commands that matter.  That way you'll
> know what will cause termination and what won't.  Yes, I know, it means
> your script is longer.  But longer is better than unpredictable.

    Unfortunately I'm not subscribed to this mailing list. Could you
point me to the right thread?

    Thanks,
    Ciprian.

    P.S.: The fact that some features of Bash, mainly `set -e`, which
should be a safety-net for scripts, is "unpredictable" is not so very
reassuring... More-over the entire Linux / Unix software
infrastructure is based on Bash, from init scripts, to complex
applications (like makepkg in ArchLinux, or the installers that come
with enterprise software (Oracle, VMWare, etc.)...)

    P.P.S.: Really, I'm scared now... :(

[Chet Ramey]

>     Sorry, but I don't understand at all... So please bare with me and
> make me understand.
>
>     So I've interpreted `set -e` as a way to tell bash to treat any
> process exiting with non-zero (and not succeeded by a || ), as an
> error and end the current shell / sub-shell.

Close.  There are a few conditions under which set -e has no effect.  The
&& and || operators are a two places, but there are more.

>     So my question is how can I solve this problem? (And obtain the
> needed behaviour.) (I see `()` and `{}` as blocks in normal
> programming languages (of course with some particularities), and
> non-zero exit codes as exceptions. And this is very helpful to write
> robust bash scripts.)

Don't rely on set -e if its behavior is not what you want.  Check exit
statuses yourself and stop execution when appropriate.  You could use
set -e and check exit statuses in the exception cases to reduce the
amount of extra work.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

[Greg Wooledge]

On Mon, Nov 09, 2009 at 03:49:09PM +0200, Ciprian Dorin, Craciun wrote:
>     Unfortunately I'm not subscribed to this mailing list. Could you
> point me to the right thread?

http://lists.gnu.org/archive/html/bug-bash/2009-11/threads.html

And... pretty much every month, especially since bash 4 changed the
behavior of set -e.

>     P.S.: The fact that some features of Bash, mainly `set -e`, which
> should be a safety-net for scripts, is "unpredictable" is not so very
> reassuring... More-over the entire Linux / Unix software
> infrastructure is based on Bash, from init scripts, to complex
> applications [...]

(Actually, you might be shocked to see that many newer Linux distributions
use dash, not bash, for their init scripts.)

What you say might be true for some versions of Linux, but the entire
_Unix_ shell infrastructure is based on the the Bourne shell, and the
practice of using the absolute lowest common denominator feature set in
order to make your script run everywhere.  With Solaris still shipping
a Bourne shell (rather than a POSIX shell) in /bin/sh that means you're
stuck with 7th Edition semantics unless you're willing to put in tricks
to try to re-invoke your script under /usr/xpg4/bin/sh or /bin/sh5 or
ksh or bash or whatever.

When you have to worry about whether you can use functions or $() or
[[:alpha:]] or arithmetic or other POSIX features in your scripts,
set -e is just another thing you *don't* want to have to mess with.
It's so much easier to check for errors yourself than to have to worry
about whether your complex nested conditional loop switch structure
thing will break set -e on Dynix or SCO.

[Chet Ramey]

Ciprian Dorin, Craciun wrote:

>>    Sorry, but I don't understand at all... So please bare with me and
>> make me understand.
>>
>>    So I've interpreted `set -e` as a way to tell bash to treat any
>> process exiting with non-zero (and not succeeded by a || ), as an
>> error and end the current shell / sub-shell.
>>
>>    Thus if I say: `set -e ; { false ; true ; }` it works, but when I
>> put the `||`, it doesn't...
>>
>>    So my question is how can I solve this problem? (And obtain the
>> needed behaviour.) (I see `()` and `{}` as blocks in normal
>> programming languages (of course with some particularities), and
>> non-zero exit codes as exceptions. And this is very helpful to write
>> robust bash scripts.)
>>
>>    Thanks,
>>    Ciprian.
>
>     I'm also pasting from the documentation.
>
>     The interesting part is :`The shell does not exit if the command
> that fails is part of the command list immediately following [...]
> part  of  any command  executed  in  a && or |⎪ list, [...], The
> option applies to [...] each sub-shell environment [...] and may cause
> subshells to exit before executing all the commands in the subshell`.
>
>     So from this I understand that the (parent) shell doesn't exit if
> the (child) sub-shell fails and is at left of `||`. But that the
> subshell inherits the `-e` option, and should exit.

The subshell does inherit the -e option.  However, it also inherits the
knowledge that it's the lhs of ||, which cancels the effect of set -e.
The setting is ignored and enabling it has no effect.  (This is actually
historical sh behavior that the Posix working group attempted to codify
last spring.)

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

[Marc Herbert-5]

Greg Wooledge a écrit :
 
> The problem being "how to use set -e in a consistent manner across all
> shells"?  You can't.  set -e is unpredictable, unreliable, and should be
> shunned.  As you can see by the last dozen or so message on this mailing
> list, not even bash gurus (other than Chet) can figure out its semantics.
>

The fact that "set -e" is not the default looks like a big design
mistake to me.  I cannot understand how ignoring errors in a program and
going on like nothing happened is a desirable thing. When I write a
succession of commands in a script, a later command depends on the
success of earlier commands more often than not and there is no point
going on with the rest of script. Ignoring errors should be the
exception and not the rule. The same can probably be said for C.


> Check for errors yourself on the commands that matter.  That way you'll
> know what will cause termination and what won't.  Yes, I know, it means
> your script is longer.  But longer is better than unpredictable.

ALL my commands matter so I practically cannot check the return status
of everyone of them.  Instead I use "set -e" systematically. It
works. It is unpredictable but I do not to care: a safety net with a few
holes is way better than none at all. The very few times I actually want
to ignore an error I just append " || true" to the corresponding
command.

[Chris F.A. Johnson-6]

On Mon, 9 Nov 2009, Marc Herbert wrote:

> Greg Wooledge a ?crit :
>  
> > The problem being "how to use set -e in a consistent manner across all
> > shells"?  You can't.  set -e is unpredictable, unreliable, and should be
> > shunned.  As you can see by the last dozen or so message on this mailing
> > list, not even bash gurus (other than Chet) can figure out its semantics.
> >
>
> The fact that "set -e" is not the default looks like a big design
> mistake to me.

    Making it the default would be a huge mistake.

> I cannot understand how ignoring errors in a program and
> going on like nothing happened is a desirable thing.

    It isn't desirable. However, commands will sometimes fail; they
    are intended to fail. Their failure provides necessary information
    to the script.

    For example, if you want to know whether user 'john' is in the
    password file:

grep -q ^john: /etc/passwd

    Why would you want the script to exit if that fails? You want to
    get the return code and execute code depending on the result
    (which can be 0, 1 or >1; 'if grep ...' is not adequate).

> When I write a succession of commands in a script, a later command
> depends on the success of earlier commands more often than not and
> there is no point going on with the rest of script.

    Then you are not taking advantage of the power of the shell. Using
    a return code provides more than a two-way branch.

> Ignoring errors should be the exception and not the rule.

    Not using set -e does not imply ignoring errors; one *uses*
    errors.

> The same can probably be said for C.
>
>
> > Check for errors yourself on the commands that matter.  That way you'll
> > know what will cause termination and what won't.  Yes, I know, it means
> > your script is longer.  But longer is better than unpredictable.
>
> ALL my commands matter so I practically cannot check the return status
> of everyone of them.  Instead I use "set -e" systematically. It
> works. It is unpredictable but I do not to care: a safety net with a few
> holes is way better than none at all. The very few times I actually want
> to ignore an error I just append " || true" to the corresponding
> command.

--
   Chris F.A. Johnson, webmaster         <http://woodbine-gerrard.com>
   ===================================================================
   Author:
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
   Pro Bash Programming: Scripting the GNU/Linux Shell (2009, Apress)

[Chet Ramey]

> The fact that "set -e" is not the default looks like a big design
> mistake to me.

Wow.  That ship sailed 30 years ago.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

[Sven Mascheck-3]

On Mon, Nov 09, 2009 at 09:10:02AM -0500, Greg Wooledge wrote:
> On Mon, Nov 09, 2009 at 03:49:09PM +0200, Ciprian Dorin, Craciun wrote:

> >     P.S.: The fact that some features of Bash, mainly `set -e`, which
> > should be a safety-net for scripts, is "unpredictable" is not so very
> > reassuring...

Demand for this safety-net might be obvious, but I suppose too few are
using it regularly?  There are places where you have to ignore a failing
command (simple example: in test conditions) and the more subtle cases
apparently never were really recognized (let alone settled on)?  See a
comparison among various shells: http://www.in-ulm.de/~mascheck/various/set-e/



And discussing a more offtopic-lke part:

> [...] practice of using the absolute lowest common denominator
> feature set in order to make your script run everywhere.

Huh, that's fundamental decisions in a casual sentence.
Yes, the lowest common denominator is quite traditional, but I believe
the real decision is to find out what amount of portability you actually
need.


> With Solaris still shipping a Bourne shell (rather than a POSIX shell)
> in /bin/sh that means you're stuck with 7th Edition semantics unless
> you're willing to put in tricks to try to re-invoke your script under
> /usr/xpg4/bin/sh or /bin/sh5 or ksh or bash or whatever.

Such dicussions are a mine field.  But I believe here you are really
confusing Solaris (quite recent SVR4-like shell) with Ultrix (sh: V7-like
shell without functions ./. sh5: SVR2-like shell).

[Chet Ramey]

> > With Solaris still shipping a Bourne shell (rather than a POSIX shell)
> > in /bin/sh that means you're stuck with 7th Edition semantics unless
> > you're willing to put in tricks to try to re-invoke your script under
> > /usr/xpg4/bin/sh or /bin/sh5 or ksh or bash or whatever.
>
> Such dicussions are a mine field.  But I believe here you are really
> confusing Solaris (quite recent SVR4-like shell) with Ultrix (sh: V7-like
> shell without functions ./. sh5: SVR2-like shell).

Solaris's default shell, though more modern than the old Ultrix one, is not
Posix-conformant.  It is more capable than the 7th edition shell,
nevertheless. :-)

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

[Sven Mascheck-3]

On Mon, Nov 09, 2009 at 02:43:43PM -0500, Chet Ramey wrote:
> > [...] confusing Solaris (quite recent SVR4-like shell)
> > with Ultrix (sh: V7-like shell without functions

> Solaris's default shell, though more modern than the old Ultrix one,
> is not Posix-conformant.

Gosh, my "recent" ('88) is certainly not related to POSIX.
If my remark was ambiguous, then thanks for avoiding even
more confusion! :-)

[Marc Herbert-5]

Chris F.A. Johnson a écrit :

>     It isn't desirable. However, commands will sometimes fail; they
>     are intended to fail. Their failure provides necessary information
>     to the script.

Because it lacks proper exceptions, the language is indeed creating a
confusion between exceptional errors and normal return values. But I
found it surprisingly easy to deal with this confusion in practice.


>     For example, if you want to know whether user 'john' is in the
>     password file:
>
> grep -q ^john: /etc/passwd
>
> You want to get the return code and execute code depending on the result

I do.

> (which can be 0, 1 or >1; 'if grep ...' is not adequate).

I find "if grep ..." adequate. Ignoring >1 here is just another small
hole in the "set -e" safety net (and it will log an error message).

In the few cases where I am really afraid of exceptional errors in a test
I can check the pre-conditions by myself. For instance:

 set -e
  ...
 [ -r /etc/passwd ] || fatal "could not read /etc/passwd"
 if grep -q ^john: /etc/passwd; then ...


But honestly, how many times in real life have you seen code actually
checking for errors of commands as simple as "grep /etc/passwd" ?


>     Then you are not taking advantage of the power of the shell. Using
>     a return code provides more than a two-way branch.

I do not consider exceptions as an actual branch, which makes most
commands "two-way branch" or less. Whenever I want to return something
more complex than 0 / 1 then I do not use "return" at all but "printf"
for instance.  An integer is too poor anyway; so I use exit statuses
only for simple boolean tests xor exceptions. The only minor issue is
that it is not possible for boolean tests to "throw exceptions": each
boolean test creates a small hole in the safety net. I can live with
that.

Do you have an other example than "grep" maybe?


>> Ignoring errors should be the exception and not the rule.
>
>     Not using set -e does not imply ignoring errors; one *uses*
>     errors.

For the clarity of this discussion could you please refrain to confuse
exceptional errors with normal return values of tests? I know that the
design of the language creates this confusion in the first place, but I
think the documentation does not and sticks to the English meaning. See
for instance "man grep":

EXIT STATUS
The following exit values are returned:
0
  One or more lines were selected.
1
  No lines were selected.
>1
  An ERROR occurred.


Cheers,

Marc

[Marc Herbert-5]

Chet Ramey a écrit :
>> The fact that "set -e" is not the default looks like a big design
>> mistake to me.
>
> Wow.  That ship sailed 30 years ago.

but forgot "set -e" on the quay?


See also this ship:
<http://www.reddit.com/r/programming/comments/7l89a/null_references_the_billion_dollar_mistake/>

[Marc Herbert-5]

Marc Herbert a écrit :

> Chris F.A. Johnson a écrit :
>>     For example, if you want to know whether user 'john' is in the
>>     password file:
>>
>> grep -q ^john: /etc/passwd
>>
>> You want to get the return code and execute code depending on the result
>>
>> (which can be 0, 1 or >1; 'if grep ...' is not adequate).
>
> I find "if grep ..." adequate. Ignoring >1 here is just another small
> hole in the "set -e" safety net (and it will log an error message).

Just found this, even better: not even any extra hole in the "set -e"
safety net.


  set -e
  ...
  if grep -q ^john: /etc/passwd; then
      echo "FOUND"
  else grepstatus=$?
      # inspect $grepstatus for errors
  fi