(Ab)using rcng's features to keep rc.d-style services running should they fail.

[Alex Trull]

Hi all,

I realised that because portupgrade/portmaster don't always
cleanly restart processes that have died due to being
upgraded (mysqld, often!) that this was something I wanted
to fix.

However, I'd seen the daemontools and wasn't a fan - too
much to configure with weird directories and so forth and
while monit is very powerful it also takes too much effort
to do what could be so much simpler.

So why not just (ab)use the rcng system with a script ? the
functionality is all there already to do almost everything
needed. To check whether something is running and (if not!)
start it.

So this is my dirty hack so far - runs out of cron with
"2>/dev/null" every few minutes and mails me about attempted
startups as they happen :

#!/bin/sh
# start things that should be running
find /usr/local/etc/rc.d/ /etc/rc.d/ -type f | egrep -v '(newsyslog|devd|sendmail)' | awk '{print $0" status| grep \"is not running\" && "$0" start"}'  | sh

Performance is not stunning, thankfuly my cpus are quite
idle.

real    0m1.198s
user    0m0.610s
sys     0m0.877s

(devd, newsyslog and sendmail are left out because their
scripts don't behave quite right.)

Initialy I used it purely for the /usr/local/etc/rc.d but I
had a base ntpd die on me one evening so decided to throw in
/etc/rc.d/ too. This script has also caught a few other
failures in port-installed daemons in addition to the
ever-common mysqld-upgraded one.

Of course it is relatively inefficient executing all those
scripts on a regular basis - but it works - has anyone
thought of cleaner/more efficient ways of doing this and
getting more out of the rcng framework ? Or simpler for that
matter.

--
Alex

[Doug Barton]

Alex Trull wrote:
> Hi all,
>
> I realised that because portupgrade/portmaster don't always
> cleanly restart processes that have died due to being
> upgraded (mysqld, often!) that this was something I wanted
> to fix.

I can't speak to portupgrade, however for portmaster there is no such
facility whatsoever. The admin is expected to disable things prior to
an upgrade and re-enable them when the upgrade is done. I don't feel
that this is an overwhelming burden. :)

That said I have it in mind to add a facility to handle this feature.
Stay tuned for more news about this.

> So why not just (ab)use the rcng system with a script ?

First, it's rc.d now if you please. Second, I don't think that there
is anything wrong with your concept that would classify it as abuse,
although I'm not sure I would have implemented it in quite the same way.

> the
> functionality is all there already to do almost everything
> needed. To check whether something is running and (if not!)
> start it.
>
> So this is my dirty hack so far - runs out of cron with
> "2>/dev/null" every few minutes and mails me about attempted
> startups as they happen :
>
> #!/bin/sh
> # start things that should be running
> find /usr/local/etc/rc.d/ /etc/rc.d/ -type f | egrep -v '(newsyslog|devd|sendmail)' | awk '{print $0" status| grep \"is not running\" && "$0" start"}'  | sh

There are a couple of "problems" with this, although please understand
I'm not criticizing, I'm just offering what I hope are constructive
suggestions.

First, (and I consider this to be a bug) there are several scripts in
/etc/rc.d that are not actually 'startup' scripts in the true sense.
Therefore I would not attempt to run them all. Personally if I were
going to do what you're doing I would make an explicit list of scripts
I wanted to test for.

If you are going to continue to use awk you might want to learn how to
avoid piping it to grep, that's an extra subshell that you don't
really need.

Finally I would do something like this (untested):

for service in ntpd mysqld foo bar; do
        if [ -x /usr/local/etc/rc.d/$service ]; then
                service=/usr/local/etc/rc.d/$service
        elif [ -x /etc/rc.d/$service ]; then
                service=/etc/rc.d/$service
        else
                echo "Cannot find $service in /etc/rc.d or /usr/local/etc/rc.d"
                exit 1
        fi

        $service start | grep -v 'already running'
done

> (devd, newsyslog and sendmail are left out because their
> scripts don't behave quite right.)

I don't see anything wrong with devd's output from the status command,
sendmail's is a little hard to parse because it's doing a lot of
things in one script. newsyslog is spitting out 'not running' which
arguably it should not do since that script is not for starting a
persistent service, it's just a 'run at boot' thing. In any case, if
you find what you think are bugs in rc.d related stuff feel free to
report them to freebsd-rc@....


hth,

Doug

--

    This .signature sanitized for your protection

_______________________________________________
freebsd-rc@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-rc
To unsubscribe, send any mail to "freebsd-rc-unsubscribe@..."

[wxs-2]

On Sun, Oct 04, 2009 at 12:30:55PM -0700, Doug Barton wrote:

> Alex Trull wrote:
> > Hi all,
> >
> > I realised that because portupgrade/portmaster don't always
> > cleanly restart processes that have died due to being
> > upgraded (mysqld, often!) that this was something I wanted
> > to fix.
>
> I can't speak to portupgrade, however for portmaster there is no such
> facility whatsoever. The admin is expected to disable things prior to
> an upgrade and re-enable them when the upgrade is done. I don't feel
> that this is an overwhelming burden. :)

There is the @stopdaemon directive in plists (which gets translated into
@unexec to forcestop the script). Some ports use it and some do not.
Personally I think ports doing this automatically are quite annoying,
and would love to rip them all out from the ports. Something like
portmaster growing support for it would be welcome provided it does not
happen by default.

I've always found it funny that there is no @startdaemon directive
(rightfully so, as we want people to explicitly turn things on) but it's
acceptable if things get turned off via @stopdaemon without explicit
permission. If a particular upgrade requires that the thing be not
running we should check for that and abort, not go shutting things down.

-- WXS
_______________________________________________
freebsd-rc@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-rc
To unsubscribe, send any mail to "freebsd-rc-unsubscribe@..."

[Xin LI-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wesley Shields wrote:

> On Sun, Oct 04, 2009 at 12:30:55PM -0700, Doug Barton wrote:
>> Alex Trull wrote:
>>> Hi all,
>>>
>>> I realised that because portupgrade/portmaster don't always
>>> cleanly restart processes that have died due to being
>>> upgraded (mysqld, often!) that this was something I wanted
>>> to fix.
>> I can't speak to portupgrade, however for portmaster there is no such
>> facility whatsoever. The admin is expected to disable things prior to
>> an upgrade and re-enable them when the upgrade is done. I don't feel
>> that this is an overwhelming burden. :)
>
> There is the @stopdaemon directive in plists (which gets translated into
> @unexec to forcestop the script). Some ports use it and some do not.
> Personally I think ports doing this automatically are quite annoying,
> and would love to rip them all out from the ports. Something like
> portmaster growing support for it would be welcome provided it does not
> happen by default.

+1

I think this feature should be user-controllable (or, the 'make install'
should be 'restart'ing the rc.d script at very least).

Cheers,
- --
Xin LI <delphij@...> http://www.delphij.net/
FreeBSD - The Power to Serve!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.13 (FreeBSD)

iEYEARECAAYFAkrLyFcACgkQi+vbBBjt66DZ5QCfU3LSI+RiZwJv3huFx4wd3QNe
UUsAn37vdhs30y+2eE/HLaw424CS7dMh
=1FW0
-----END PGP SIGNATURE-----
_______________________________________________
freebsd-rc@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-rc
To unsubscribe, send any mail to "freebsd-rc-unsubscribe@..."

[Daniel O'Connor-3]

On Wed, 7 Oct 2009, Xin LI wrote:

> Wesley Shields wrote:
> > On Sun, Oct 04, 2009 at 12:30:55PM -0700, Doug Barton wrote:
> >> Alex Trull wrote:
> >>> Hi all,
> >>>
> >>> I realised that because portupgrade/portmaster don't always
> >>> cleanly restart processes that have died due to being
> >>> upgraded (mysqld, often!) that this was something I wanted
> >>> to fix.
> >>
> >> I can't speak to portupgrade, however for portmaster there is no
> >> such facility whatsoever. The admin is expected to disable things
> >> prior to an upgrade and re-enable them when the upgrade is done. I
> >> don't feel that this is an overwhelming burden. :)
> >
> > There is the @stopdaemon directive in plists (which gets translated
> > into @unexec to forcestop the script). Some ports use it and some
> > do not. Personally I think ports doing this automatically are quite
> > annoying, and would love to rip them all out from the ports.
> > Something like portmaster growing support for it would be welcome
> > provided it does not happen by default.
>
> +1
>
> I think this feature should be user-controllable (or, the 'make
> install' should be 'restart'ing the rc.d script at very least).

It won't actually start anything you haven't enabled in rc.conf though
since all ports install rc.d scripts which require FOO_enable to be
YES.

That said a knob like RESTART_SERVICES or similar would be nice.



--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C