|
Make process title - % complete
|
|
Re: Make process title - % complete
Hi Ivan,
On Mon, Oct 19, 2009 at 03:52:30PM +0200, Ivan Voras wrote:
> if nobody objects, I'll commit it :)
I seem to recall that setproctitle() is quite expensive to call; perhaps
it would make sense offer a flag to prevent make(1) from calling it? [1]
Anyway, the feature looks nice! I'd like to have it...
[1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
expect it's negligable but who knows...
--
Rink P.W. Springer - http://rink.nu"Beauty often seduces us on the road to truth."
- Dr. Wilson
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
2009/10/19 Rink Springer < rink@...>:
> Hi Ivan,
>
> On Mon, Oct 19, 2009 at 03:52:30PM +0200, Ivan Voras wrote:
>> if nobody objects, I'll commit it :)
>
> I seem to recall that setproctitle() is quite expensive to call; perhaps
> it would make sense offer a flag to prevent make(1) from calling it? [1]
>
> Anyway, the feature looks nice! I'd like to have it...
>
> [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
> expect it's negligable but who knows...
The loop it's called in is not processed bazillion times per second
(though it *is* called surprisingly often; small, fast jobs can result
in somewhere in the order of magnitude of 100 iterations per second on
a fast CPU). As you said - I expect it's negligable compared to fork()
and the work jobs themselves do.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
Ivan Voras wrote:
> I have a small patch that makes "make" display percentage complete in
> process title, which can be retrieved in "top" in the form of:
>
> 71466 root 1 76 0 7008K 5696K select 0 0:00 0.00%
> make: 95% (55 more targets out of 1360) (make)
Also: is there someone here more familiar with "make" who can tell me if
the "current" top level target (i.e. the one taken from the command
line) is kept track of somewhere? For example "clean" in "make clean
install".
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
 Re: Make process title - % complete
On Mon, Oct 19, 2009 at 04:35:08PM +0200, Ivan Voras wrote:
> >> if nobody objects, I'll commit it :)
> >
> > I seem to recall that setproctitle() is quite expensive to call; perhaps
> > it would make sense offer a flag to prevent make(1) from calling it? [1]
> >
> > Anyway, the feature looks nice! I'd like to have it...
> >
> > [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
> > expect it's negligable but who knows...
>
> The loop it's called in is not processed bazillion times per second
> (though it *is* called surprisingly often; small, fast jobs can result
> in somewhere in the order of magnitude of 100 iterations per second on
> a fast CPU). As you said - I expect it's negligable compared to fork()
> and the work jobs themselves do.
How about add this statistic to make info handler?
--
Adios
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
2009/10/19 Alex Kozlov < spam@...>:
> On Mon, Oct 19, 2009 at 04:35:08PM +0200, Ivan Voras wrote:
>> >> if nobody objects, I'll commit it :)
>> >
>> > I seem to recall that setproctitle() is quite expensive to call; perhaps
>> > it would make sense offer a flag to prevent make(1) from calling it? [1]
>> >
>> > Anyway, the feature looks nice! I'd like to have it...
>> >
>> > [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
>> > expect it's negligable but who knows...
>>
>> The loop it's called in is not processed bazillion times per second
>> (though it *is* called surprisingly often; small, fast jobs can result
>> in somewhere in the order of magnitude of 100 iterations per second on
>> a fast CPU). As you said - I expect it's negligable compared to fork()
>> and the work jobs themselves do.
> How about add this statistic to make info handler?
You mean SIGINFO?
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Mon, Oct 19, 2009 at 05:51:42PM +0200, Ivan Voras wrote:
> 2009/10/19 Alex Kozlov < spam@...>:
> > On Mon, Oct 19, 2009 at 04:35:08PM +0200, Ivan Voras wrote:
> >> >> if nobody objects, I'll commit it :)
> >> >
> >> > I seem to recall that setproctitle() is quite expensive to call; perhaps
> >> > it would make sense offer a flag to prevent make(1) from calling it? [1]
> >> >
> >> > Anyway, the feature looks nice! I'd like to have it...
> >> >
> >> > [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
> >> > expect it's negligable but who knows...
> >>
> >> The loop it's called in is not processed bazillion times per second
> >> (though it *is* called surprisingly often; small, fast jobs can result
> >> in somewhere in the order of magnitude of 100 iterations per second on
> >> a fast CPU). As you said - I expect it's negligable compared to fork()
> >> and the work jobs themselves do.
> > How about add this statistic to make info handler?
> You mean SIGINFO?
Yes
--
Adios
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
Quoting Alex Kozlov < spam@...>:
> On Mon, Oct 19, 2009 at 05:51:42PM +0200, Ivan Voras wrote:
>> 2009/10/19 Alex Kozlov < spam@...>:
>> > On Mon, Oct 19, 2009 at 04:35:08PM +0200, Ivan Voras wrote:
>> >> >> if nobody objects, I'll commit it :)
>> >> >
>> >> > I seem to recall that setproctitle() is quite expensive to
>> call; perhaps
>> >> > it would make sense offer a flag to prevent make(1) from
>> calling it? [1]
>> >> >
>> >> > Anyway, the feature looks nice! I'd like to have it...
>> >> >
>> >> > [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
>> >> > expect it's negligable but who knows...
>> >>
>> >> The loop it's called in is not processed bazillion times per second
>> >> (though it *is* called surprisingly often; small, fast jobs can result
>> >> in somewhere in the order of magnitude of 100 iterations per second on
>> >> a fast CPU). As you said - I expect it's negligable compared to fork()
>> >> and the work jobs themselves do.
>> > How about add this statistic to make info handler?
>> You mean SIGINFO?
> Yes
Using SIGINFO sounds nice, but make produces so much output that
normally you won't see the result because it is scrolled up just after
sending the signal.
|
|
Re: Make process title - % complete
On Tue, Oct 20, 2009 at 10:07:07AM +0200, Lars Engels wrote:
> Quoting Alex Kozlov < spam@...>:
>
> > On Mon, Oct 19, 2009 at 05:51:42PM +0200, Ivan Voras wrote:
> >> 2009/10/19 Alex Kozlov < spam@...>:
> >> > On Mon, Oct 19, 2009 at 04:35:08PM +0200, Ivan Voras wrote:
> >> >> >> if nobody objects, I'll commit it :)
> >> >> >
> >> >> > I seem to recall that setproctitle() is quite expensive to
> >> call; perhaps
> >> >> > it would make sense offer a flag to prevent make(1) from
> >> calling it? [1]
> >> >> >
> >> >> > Anyway, the feature looks nice! I'd like to have it...
> >> >> >
> >> >> > [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
> >> >> > expect it's negligable but who knows...
> >> >>
> >> >> The loop it's called in is not processed bazillion times per second
> >> >> (though it *is* called surprisingly often; small, fast jobs can result
> >> >> in somewhere in the order of magnitude of 100 iterations per second on
> >> >> a fast CPU). As you said - I expect it's negligable compared to fork()
> >> >> and the work jobs themselves do.
> >> > How about add this statistic to make info handler?
> >> You mean SIGINFO?
> > Yes
>
> Using SIGINFO sounds nice, but make produces so much output that
> normally you won't see the result because it is scrolled up just after
> sending the signal.
Of course ps or top output much more convenient, but if setproctitle so
expencive and will be called so often, then SIGINFO may be good
compromise.
--
Adios
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
Alex Kozlov wrote:
> Of course ps or top output much more convenient, but if setproctitle so
> expencive and will be called so often, then SIGINFO may be good
> compromise.
Regarding speed of setproctitle(), here are some microbenchmark results
from the attached test source:
getpid: 3661124.75 iterations/s
setproctitle: 591357.56 iterations/s
Meaning, setprocitle() is around 6 times more expensive than getpid(),
meaning it can only be pulled off nearly 600,000 calls/s on a 2.3 GHz
Core 2 CPU.
I really want to be enlightened about how it could affect wallclock time
in make(1).
----
#include <sys/time.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#define NITER 1e7
double now() {
struct timeval tp;
gettimeofday(&tp, NULL);
return tp.tv_sec + (double)tp.tv_usec / 1e6f;
}
int main() {
double t1, t2, t3;
int i;
t1 = now();
for (i = 0; i < NITER; i++)
getpid();
t2 = now() - t1;
printf("getpid: %0.2f iterations/s\n", (float)(NITER/t2));
t1 = now();
for (i = 0; i < NITER; i++)
setproctitle("t%d", i);
t3 = now() - t1;
printf("setproctitle: %0.2f iterations/s\n", (float)(NITER/t3));
}
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
Hi,
On Mon, Oct 19, 2009 at 4:40 PM, Ivan Voras < ivoras@...> wrote:
> Ivan Voras wrote:
>>
>> I have a small patch that makes "make" display percentage complete in
>> process title, which can be retrieved in "top" in the form of:
>>
>> 71466 root 1 76 0 7008K 5696K select 0 0:00 0.00%
>> make: 95% (55 more targets out of 1360) (make)
>
> Also: is there someone here more familiar with "make" who can tell me if the
> "current" top level target (i.e. the one taken from the command line) is
> kept track of somewhere? For example "clean" in "make clean install".
>
gmake does show the nesting level in its output and indeed it's a
valuable information if setproctitle is to be used...
gmake appears to use a MAKELEVEL environment variable to keep track in
between parent/child runs. I see a similar mechanism in our make
(using the __MKLVL__ environment variable) but it's restricted only to
the check_make_level() function that is checking the nesting level,
thus no global variable is available to use.
Regards,
Adrian Penisoara
EnterpriseBSD.com
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Tue, 2009-10-20 at 14:42 +0200, Ivan Voras wrote:
> Alex Kozlov wrote:
>
> > Of course ps or top output much more convenient, but if setproctitle so
> > expencive and will be called so often, then SIGINFO may be good
> > compromise.
>
> Regarding speed of setproctitle(), here are some microbenchmark results
> from the attached test source:
>
> getpid: 3661124.75 iterations/s
> setproctitle: 591357.56 iterations/s
>
> Meaning, setprocitle() is around 6 times more expensive than getpid(),
> meaning it can only be pulled off nearly 600,000 calls/s on a 2.3 GHz
> Core 2 CPU.
>
> I really want to be enlightened about how it could affect wallclock time
> in make(1).
What is the relative difference in buildworld time with and without?
robert.
> ----
>
> #include <sys/time.h>
> #include <stdlib.h>
> #include <stdio.h>
> #include <string.h>
> #include <unistd.h>
>
> #define NITER 1e7
>
> double now() {
> struct timeval tp;
> gettimeofday(&tp, NULL);
> return tp.tv_sec + (double)tp.tv_usec / 1e6f;
> }
>
> int main() {
> double t1, t2, t3;
> int i;
>
> t1 = now();
> for (i = 0; i < NITER; i++)
> getpid();
> t2 = now() - t1;
>
> printf("getpid: %0.2f iterations/s\n", (float)(NITER/t2));
>
> t1 = now();
> for (i = 0; i < NITER; i++)
> setproctitle("t%d", i);
> t3 = now() - t1;
>
> printf("setproctitle: %0.2f iterations/s\n", (float)(NITER/t3));
> }
>
> _______________________________________________
> freebsd-hackers@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers> To unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
--
Robert Noland < rnoland@...>
FreeBSD
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Tue, Oct 20, 2009 at 02:42:17PM +0200, Ivan Voras wrote:
> Alex Kozlov wrote:
>
> > Of course ps or top output much more convenient, but if setproctitle so
> > expencive and will be called so often, then SIGINFO may be good
> > compromise.
>
> Regarding speed of setproctitle(), here are some microbenchmark results
> from the attached test source:
>
> getpid: 3661124.75 iterations/s
> setproctitle: 591357.56 iterations/s
>
> Meaning, setprocitle() is around 6 times more expensive than getpid(),
> meaning it can only be pulled off nearly 600,000 calls/s on a 2.3 GHz
> Core 2 CPU.
>
> I really want to be enlightened about how it could affect wallclock time
> in make(1).
make universe few times with and without patch?
--
Adios
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Tue, Oct 20, 2009 at 02:42:17PM +0200, Ivan Voras wrote:
> Alex Kozlov wrote:
>
> >Of course ps or top output much more convenient, but if setproctitle so
> >expencive and will be called so often, then SIGINFO may be good
> >compromise.
>
> Regarding speed of setproctitle(), here are some microbenchmark results
> from the attached test source:
>
> getpid: 3661124.75 iterations/s
> setproctitle: 591357.56 iterations/s
>
> Meaning, setprocitle() is around 6 times more expensive than getpid(),
> meaning it can only be pulled off nearly 600,000 calls/s on a 2.3 GHz
> Core 2 CPU.
what about contention? setproctitle() is an sysctl so it will prevent
other sysctl's from working when being executed..
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
2009/10/20 Roman Divacky < rdivacky@...>:
> On Tue, Oct 20, 2009 at 02:42:17PM +0200, Ivan Voras wrote:
>> Alex Kozlov wrote:
>>
>> >Of course ps or top output much more convenient, but if setproctitle so
>> >expencive and will be called so often, then SIGINFO may be good
>> >compromise.
>>
>> Regarding speed of setproctitle(), here are some microbenchmark results
>> from the attached test source:
>>
>> getpid: 3661124.75 iterations/s
>> setproctitle: 591357.56 iterations/s
>>
>> Meaning, setprocitle() is around 6 times more expensive than getpid(),
>> meaning it can only be pulled off nearly 600,000 calls/s on a 2.3 GHz
>> Core 2 CPU.
>
> what about contention? setproctitle() is an sysctl so it will prevent
> other sysctl's from working when being executed..
Others sysctls... for that particular process (since it modifies
process-global data) which happens to be single-threaded :P
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Mon, 19 Oct 2009 17:51:42 +0200, Ivan Voras < ivoras@...> wrote:
>2009/10/19 Alex Kozlov < spam@...>:
>> How about add this statistic to make info handler?
>
> You mean SIGINFO?
Yes, that's the ``info handler''.
While printing something on SINGINFO arrival is a nice idea, it may not
be extremely useful for make(1). With dd(1) this is very useful to see,
but with long-running make jobs that write tons of output to stderr any
information from SIGINFO may be lost in the noise.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
Giorgos Keramidas wrote:
> On Mon, 19 Oct 2009 17:51:42 +0200, Ivan Voras < ivoras@...> wrote:
>
>> 2009/10/19 Alex Kozlov < spam@...>:
>>
>>> How about add this statistic to make info handler?
>>>
>> You mean SIGINFO?
>>
>
> Yes, that's the ``info handler''.
>
> While printing something on SINGINFO arrival is a nice idea, it may not
> be extremely useful for make(1). With dd(1) this is very useful to see,
> but with long-running make jobs that write tons of output to stderr any
> information from SIGINFO may be lost in the noise.
>
> _______________________________________________
> freebsd-hackers@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers> To unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
>
The SIGINFO handler could be made to put the make process to sleep for,
say, a second to give the user some time to read the statistics, but I'm
sure there are lots of objections to be made to that, too.
-Boris
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Monday 19 October 2009 16:08:06 Rink Springer wrote:
> Hi Ivan,
>
> On Mon, Oct 19, 2009 at 03:52:30PM +0200, Ivan Voras wrote:
> > if nobody objects, I'll commit it :)
>
> I seem to recall that setproctitle() is quite expensive to call; perhaps
> it would make sense offer a flag to prevent make(1) from calling it? [1]
Just rate-limit the setproctitle() call to once/sec or once/percentage-step
and be done with it.
I must say that trying it out on a kernel build didn't proof too useful as the
targets have vastly different runtimes, but I think it's a good addition
nonetheless. So please, go for it Ivan.
> Anyway, the feature looks nice! I'd like to have it...
>
> [1] I'm unsure how expensive it is compared to fork(1)-ing etc; I'd
> expect it's negligable but who knows...
>
--
/"\ Best regards, | mlaier@...
\ / Max Laier | ICQ #67774661
X http://pf4freebsd.love2party.net/ | mlaier@EFnet
/ \ ASCII Ribbon Campaign | Against HTML Mail and News
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
|
Re: Make process title - % complete
On Tue, 20 Oct 2009 20:28:40 -0400, Boris Kochergin < spawk@...> wrote:
>Giorgos Keramidas wrote:
>>On Mon, 19 Oct 2009 17:51:42 +0200, Ivan Voras < ivoras@...> wrote:
>>>2009/10/19 Alex Kozlov < spam@...>:
>>>> How about add this statistic to make info handler?
>>>
>>> You mean SIGINFO?
>>
>> Yes, that's the ``info handler''.
>>
>> While printing something on SINGINFO arrival is a nice idea, it may
>> not be extremely useful for make(1). With dd(1) this is very useful
>> to see, but with long-running make jobs that write tons of output to
>> stderr any information from SIGINFO may be lost in the noise.
>
> The SIGINFO handler could be made to put the make process to sleep
> for, say, a second to give the user some time to read the statistics,
> but I'm sure there are lots of objections to be made to that, too.
That would be bad, indeed. David Wolfskill has emailed me, in the
meantime, that it's probably ok to `lose' SIGINFO output in the noise of
build output, as long as it is easy to grep for it. So I still like the
idea, but without a delay please :)
|
|
Re: Make process title - % complete
Hi,
On Wed, Oct 21, 2009 at 3:11 AM, Max Laier < max@...> wrote:
> On Monday 19 October 2009 16:08:06 Rink Springer wrote:
>> Hi Ivan,
>>
>> On Mon, Oct 19, 2009 at 03:52:30PM +0200, Ivan Voras wrote:
>> > if nobody objects, I'll commit it :)
>>
>> I seem to recall that setproctitle() is quite expensive to call; perhaps
>> it would make sense offer a flag to prevent make(1) from calling it? [1]
>
> Just rate-limit the setproctitle() call to once/sec or once/percentage-step
> and be done with it.
Rather try to setproctitle() in the same make process every second
with a one second initial delay (so that short lived make processes
won't be bogged down by this expensive call). And preferentially do
the timing check after returning from exec() of a child make.
This way the stats won't be "perturbed" by the short lived make's and
only one make process will call setproctitle() at any time (except
when running "make -j" ?).
>
> I must say that trying it out on a kernel build didn't proof too useful as the
> targets have vastly different runtimes, but I think it's a good addition
> nonetheless. So please, go for it Ivan.
>
If you implement it, please use a control mechanism like, say, an
environment variable MAKE_TRACK_PROGRESS, which, for performance and
POLA sake, might default to disabled (including when environment is
not defined).
My 5cents,
Adrian Penisoara
EnterpriseBSD.com
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackersTo unsubscribe, send any mail to " freebsd-hackers-unsubscribe@..."
|
attachment0 (202 bytes) 