synchronizing multiple services rotation

[Joan Picanyol i Puig-9]

Hi,

I'm setting up a system in which several consumers do some batch
processing of data logged from several producers, all of them services.
Producers are continuously running services, and on log rotation they
start once the consumer services (which detect what producer are being
rotated from reading $PWD). They are fed the rotated file, retrieve
saved state information including last execution on the filesystem, and
open the other consumer's current log. This way, the consumers are
invoked at least as often as the fastest rotating producer and no data
is lost.

However, there's a race here if more than one producer rotates
simultaniously: the consumer might not see the logs from the producer
which didn't cause it's invocation.

producer 1's log rotates
                            consumer is started once
producer 2's log rotates
                            consumer is fed producer 1's log through
                            stdin and goes reading producer 2's current
                            which is already empty
                   
The only way to block log rotation seems to be blocking the /previous
rotation/ processor, but I can't come up with a way to use this to
close the race across diferent services.

Ideas? Suggestions? I need to solve this for the general case (more than one
consumer), forward looking insights in that direction most welcome.

tks
--
pica

[Daryl Tester-3]

(* Reply to dev null'd *)

Joan Picanyol i Puig wrote:

> I'm setting up a system in which several consumers do some batch
> processing of data logged from several producers, all of them services.
> Producers are continuously running services, and on log rotation they
> start once the consumer services (which detect what producer are being
> rotated from reading $PWD). They are fed the rotated file, retrieve
> saved state information including last execution on the filesystem, and
> open the other consumer's current log.

This sounds ... confusing (which to me is usually a warning).

> This way, the consumers are invoked at least as often as the fastest
> rotating producer and no data is lost.

So any of the participating process' multilog processor can process
*all* of the "producer" log files?  It's essentially the same script?
I guess it would have to be if it's processing another producer's
current file (but then what does the processing script do with another's
current file?  How is it stopped from being processed again?  Or aren't
you looking at any of the previous data (is this where the saved state
is coming in?).

> However, there's a race here if more than one producer rotates
> simultaniously: the consumer might not see the logs from the producer
> which didn't cause it's invocation.

Without really knowing enough of your problem, would:

*)  Having a common script for processing (possibly parameterised to
know which producer invoked it)
*)  Using setlock (or the singleton locking mechanism of your choice)
within each multilogs's processor to fire off the above script

do what you want?


--
Regards,
  Daryl Tester

"Scheme is an exotic sports car. Fast. Manual transmission. No radio.
 Common Lisp is Howl's Moving Castle."
  -- Steve Yegge, comparing Lisp families to cars.

[Joan Picanyol i Puig-9]

[FWIW I'm using runit, not daemontools]

* Daryl Tester <dt-djb@...> [20091109 03:37]:

> Joan Picanyol i Puig wrote:
>
> >I'm setting up a system in which several consumers do some batch
> >processing of data logged from several producers, all of them services.
> >Producers are continuously running services, and on log rotation they
> >start once the consumer services (which detect what producer are being
> >rotated from reading $PWD). They are fed the rotated file, retrieve
> >saved state information including last execution on the filesystem, and
> >open the other consumer's current log.
>
> This sounds ... confusing (which to me is usually a warning).

OK, let me try to state the requirements in a simpler way:

1. N > 1 "producer"s (data loggers), for which rotated logs must be
   kept
2. M > 1 "consumer"s (data processors), each of which need data from any
   subset "producer"s

And let me try to explain again the system's architecture:

All "producer"s share a common processor script that reads it's
"consumer"s from the environment (set up in each "producer"'s log
configuration), signals them ('/command/sv once /service/consumer') and
feeds them 'current' through stdin upon rotation. "consumer"s use $PWD
to find out what they are being fed through stdin and go fetch the rest
of the data they need at the other's (not being currently rotated)
"producer"s log dir.

> >This way, the consumers are invoked at least as often as the fastest
> >rotating producer and no data is lost.
>
> So any of the participating process' multilog processor can process
> *all* of the "producer" log files?

Yes (assuming *all* refers to the set of "producer"s), but the data must
still be available for other "processor"s.

> It's essentially the same script?

"consumer"s are actually programs which process data, not just simple
scripts. That's one of the reasons for setting them up as services, so
processing can proceed asynchronously from log rotation (note that it's
not this that causes the race). The other is the fact that every
"producer"'s log might be needed any subset of "consumer"s.

> I guess it would have to be if it's processing another producer's
> current file (but then what does the processing script do with another's
> current file?  How is it stopped from being processed again?  Or aren't
> you looking at any of the previous data (is this where the saved state
> is coming in?).

"consumer"s use the saved state to avoid processing duplicate data and
to ensure correctness, since log rotation is not coupled with
"consumer"s logical intervals.

> >However, there's a race here if more than one producer rotates
> >simultaniously: the consumer might not see the logs from the producer
> >which didn't cause it's invocation.
>
> Without really knowing enough of your problem, would:
>
> *)  Having a common script for processing (possibly parameterised to
> know which producer invoked it)
> *)  Using setlock (or the singleton locking mechanism of your choice)
> within each multilogs's processor to fire off the above script

My reading of svlogd.c (and multilog.c for that matter) tells me that
rotation (i.e.: 'current' renaming) takes place before 'processor''s
invocation. Thus invoking setlock within the processor does not close
the race.

I hope the issue is clearer now. My only idea so far is to not feed
current through stdin but rather make "consumer"s scan all "producer"s
log directory for data.

Ideas? Suggestions?
--
pica

[Daryl Tester-3]

(* Reply to dev null'd *)

Joan Picanyol i Puig wrote:

> [FWIW I'm using runit, not daemontools]

I've just noticed with the mention of sv and svlogd.c (and prior to
reading this bit).  I don't know anything about runit so my comments
are likely to be way off.

> All "producer"s share a common processor script that reads it's
> "consumer"s from the environment (set up in each "producer"'s log
> configuration), signals them ('/command/sv once /service/consumer') and
> feeds them 'current' through stdin upon rotation. "consumer"s use $PWD
> to find out what they are being fed through stdin and go fetch the rest
> of the data they need at the other's (not being currently rotated)
> "producer"s log dir.

I'm not sure that it's relevant, but I'm confused at how you are tying
the producer's multilog processor to the consumer's stdin when consumer
is also a service?  Named pipe?

> My reading of svlogd.c (and multilog.c for that matter) tells me that
> rotation (i.e.: 'current' renaming) takes place before 'processor''s
> invocation. Thus invoking setlock within the processor does not close
> the race.

Hmmm.  My understanding is this - under daemontools, prior to the
processor script being run the current file is renamed to previous.
If your consumer script has obtained a lock that other producers
and consumers must abide by, then if any current file isn't named
current, it must be named previous.  If an open(2) of "current"
fails with ENOENT, then an open(2) of "previous" must succeed
provided all the other multilog processors are still holding
waiting for the lock (once multilog completes, the file should be
renamed to its tai64n format).  If the file is renamed after the
open() (from current to previous), then that's not an issue - you
already have the file open (of course, substitute open() for
whatever language you're using).

I'm unsure if this is the race you're thinking of.

> I hope the issue is clearer now. My only idea so far is to not feed
> current through stdin but rather make "consumer"s scan all "producer"s
> log directory for data.

I was presuming your were doing the latter - in fact, I'm not sure
how you would be doing the former (even after your explanation above)
without scanning for the other producer's files and opening them
yourself anyway?

Bear in mind that this lock acquiring mechanism will block the
producers if their multilog rolls and the intervening pipe fills
up.  This may be an issue if your consumer process is a long
running one.


--
Regards,
  Daryl Tester

"Scheme is an exotic sports car. Fast. Manual transmission. No radio.
 Common Lisp is Howl's Moving Castle."
  -- Steve Yegge, comparing Lisp families to cars.