Nabble - RRDTool - Dev

Hi~! would you mind if i ask you to use RRDtool?

2009-11-23T00:52:11Z

Hi~!!

I'm korean who study network in Sungkyunkwan university.

Sorry, i don't speak English very well.

Anyway, i hope to make a simple network monitoring program using RRdtool and Libpcap

I tried to search document that is RRD menual using C API, but got a little information

Please, help me.

i'dont understand how to use a function of RRDtool in real network?

read the example..

char * updateparams[]={

"rrdupdate",

"myfile.rrd",

"0:42",

NULL

};

rrd_update(3,updateparams);

rrd_crate and rrd_graph are almost same..

But it's not enough.. T_T

I hope to know a code...

Now, there are a many example, but i don't know a real code with libpcap.. T_T

I already completed coding to capture packet using libpcap

Please, help me.. T_T

Thank you, best regards

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: [PATCH 1/2] Add utility functions to allocate pointers in variable size chunks.

2009-11-09T07:19:33Z

On Mon, Nov 09, 2009 at 09:57:43AM +0100, Tobias Oetiker wrote:
> Hi Kevin,
>
> since 1.4.x is now out, I would like to hold off on adding new
> public functions to the 1.4 branche ... these would go into the new
> trunk. or is there also a bug-fix component to this change ?

Tobi,

This improves performance for implementations where realloc() may be
expensive.. It causes the realloc() for cache_item_t.values to be in
larger chunks, and less frequent.

This is in response to Thorsten's performance testing, where he sees CPU
spikes at specific times after start-up, when his cache_item_t.values
arrays all simultaneously cross some size threshold.

Up to you.

--
kevin brintnall =~ /kbrint@.../

> cheers
> tobi
>
> Oct 31 kevin brintnall wrote:
>
> > ---
> > doc/librrd.pod | 18 ++++++++++++++++++
> > src/rrd.h | 4 ++++
> > src/rrd_utils.c | 53 ++++++++++++++++++++++++++++++++++++++++++-----------
> > 3 files changed, 64 insertions(+), 11 deletions(-)
> >
> > diff --git a/doc/librrd.pod b/doc/librrd.pod
> > index 038746c..396e50c 100644
> > --- a/doc/librrd.pod
> > +++ b/doc/librrd.pod
> > @@ -81,6 +81,16 @@ end of the new C<dest>. Returns 1 on success, 0 on failure.
> > if (!rrd_add_ptr(&arr, &arr_size, elem))
> > handle_failure();
> >
> > +=item B<rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src, size_t *alloc, size_t chunk)>
> > +
> > +Like C<rrd_add_ptr>, except the destination is allocated in chunks of
> > +C<chunk>. C<alloc> points to the number of entries allocated, whereas
> > +C<dest_size> points to the number of valid pointers. If more pointers are
> > +needed, C<chunk> pointers are allocated and C<alloc> is increased
> > +accordingly. C<alloc> must be E<gt>= C<dest_size>.
> > +
> > +This method improves performance on hosts with expensive C<realloc()>.
> > +
> > =item B<rrd_add_strdup(char ***dest, size_t *dest_size, char *src)>
> >
> > Like C<rrd_add_ptr>, except adds a C<strdup> of the source string.
> > @@ -91,6 +101,14 @@ Like C<rrd_add_ptr>, except adds a C<strdup> of the source string.
> > if (!rrd_add_strdup(&arr, &arr_size, str))
> > handle_failure();
> >
> > +=item B<rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src, size_t *alloc, size_t chunk)>
> > +
> > +Like C<rrd_add_strdup>, except the destination is allocated in chunks of
> > +C<chunk>. C<alloc> points to the number of entries allocated, whereas
> > +C<dest_size> points to the number of valid pointers. If more pointers are
> > +needed, C<chunk> pointers are allocated and C<alloc> is increased
> > +accordingly. C<alloc> must be E<gt>= C<dest_size>.
> > +
> > =item B<rrd_free_ptrs(void ***src, size_t *cnt)>
> >
> > Free an array of pointers allocated by C<rrd_add_ptr> or
> > diff --git a/src/rrd.h b/src/rrd.h
> > index ce7e749..6008794 100644
> > --- a/src/rrd.h
> > +++ b/src/rrd.h
> > @@ -333,8 +333,12 @@ int rrd_proc_start_end(
> >
> > long rrd_random(void);
> >
> > + int rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src,
> > + size_t *alloc, size_t chunk);
> > int rrd_add_ptr(void ***dest, size_t *dest_size, void *src);
> > int rrd_add_strdup(char ***dest, size_t *dest_size, char *src);
> > + int rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src,
> > + size_t *alloc, size_t chunk);
> > void rrd_free_ptrs(void ***src, size_t *cnt);
> >
> > int rrd_mkdir_p(const char *pathname, mode_t mode);
> > diff --git a/src/rrd_utils.c b/src/rrd_utils.c
> > index 3936cff..6853c66 100644
> > --- a/src/rrd_utils.c
> > +++ b/src/rrd_utils.c
> > @@ -50,29 +50,53 @@ long rrd_random(void)
> > return random();
> > }
> >
> > -/* rrd_add_ptr: add a pointer to a dynamically sized array of pointers,
> > - * realloc as necessary. returns 1 on success, 0 on failure.
> > +/* rrd_add_ptr_chunk: add a pointer to a dynamically sized array of
> > + * pointers, realloc as necessary in multiples of "chunk".
> > + *
> > + * "alloc" is the number of pointers allocated
> > + * "dest_size" is the number of valid pointers
> > + *
> > + * returns 1 on success, 0 on failure.
> > */
> >
> > -int rrd_add_ptr(void ***dest, size_t *dest_size, void *src)
> > +int rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src,
> > + size_t *alloc, size_t chunk)
> > {
> > void **temp;
> >
> > assert(dest != NULL);
> > + assert(alloc != NULL);
> > + assert(*alloc >= *dest_size);
> >
> > - temp = (void **) rrd_realloc(*dest, (*dest_size+1) * sizeof(*dest));
> > - if (!temp)
> > - return 0;
> > + if (*alloc == *dest_size)
> > + {
> > + temp = (void **) rrd_realloc(*dest, (*alloc+chunk) * sizeof(*dest));
> > + if (!temp)
> > + return 0;
> >
> > - *dest = temp;
> > - temp[*dest_size] = src;
> > + *dest = temp;
> > + *alloc += chunk;
> > + }
> > +
> > + (*dest)[*dest_size] = src;
> > (*dest_size)++;
> >
> > return 1;
> > }
> >
> > -/* like rrd_add_ptr, but calls strdup() on a string first. */
> > -int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
> > +/* rrd_add_ptr: add a pointer to a dynamically sized array of pointers,
> > + * realloc as necessary. returns 1 on success, 0 on failure.
> > + */
> > +int rrd_add_ptr(void ***dest, size_t *dest_size, void *src)
> > +{
> > + size_t alloc = *dest_size;
> > +
> > + return rrd_add_ptr_chunk(dest, dest_size, src, &alloc, 1);
> > +}
> > +
> > +/* like rrd_add_ptr_chunk, but calls strdup() on a string first. */
> > +int rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src,
> > + size_t *alloc, size_t chunk)
> > {
> > char *dup_src;
> > int add_ok;
> > @@ -84,13 +108,20 @@ int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
> > if (!dup_src)
> > return 0;
> >
> > - add_ok = rrd_add_ptr((void ***)dest, dest_size, (void *)dup_src);
> > + add_ok = rrd_add_ptr_chunk((void ***)dest, dest_size, (void *)dup_src, alloc, chunk);
> > if (!add_ok)
> > free(dup_src);
> >
> > return add_ok;
> > }
> >
> > +int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
> > +{
> > + size_t alloc = *dest_size;
> > +
> > + return rrd_add_strdup_chunk(dest, dest_size, src, &alloc, 1);
> > +}
> > +
> > void rrd_free_ptrs(void ***src, size_t *cnt)
> > {
> > void **sp;
> >
>
> --
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: [PATCH 1/2] Add utility functions to allocate pointers in variable size chunks.

2009-11-09T00:57:43Z

Hi Kevin,

since 1.4.x is now out, I would like to hold off on adding new
public functions to the 1.4 branche ... these would go into the new
trunk. or is there also a bug-fix component to this change ?

cheers
tobi

Oct 31 kevin brintnall wrote:

> ---
> doc/librrd.pod | 18 ++++++++++++++++++
> src/rrd.h | 4 ++++
> src/rrd_utils.c | 53 ++++++++++++++++++++++++++++++++++++++++++-----------
> 3 files changed, 64 insertions(+), 11 deletions(-)
>
> diff --git a/doc/librrd.pod b/doc/librrd.pod
> index 038746c..396e50c 100644
> --- a/doc/librrd.pod
> +++ b/doc/librrd.pod
> @@ -81,6 +81,16 @@ end of the new C<dest>. Returns 1 on success, 0 on failure.
> if (!rrd_add_ptr(&arr, &arr_size, elem))
> handle_failure();
>
> +=item B<rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src, size_t *alloc, size_t chunk)>
> +
> +Like C<rrd_add_ptr>, except the destination is allocated in chunks of
> +C<chunk>. C<alloc> points to the number of entries allocated, whereas
> +C<dest_size> points to the number of valid pointers. If more pointers are
> +needed, C<chunk> pointers are allocated and C<alloc> is increased
> +accordingly. C<alloc> must be E<gt>= C<dest_size>.
> +
> +This method improves performance on hosts with expensive C<realloc()>.
> +
> =item B<rrd_add_strdup(char ***dest, size_t *dest_size, char *src)>
>
> Like C<rrd_add_ptr>, except adds a C<strdup> of the source string.
> @@ -91,6 +101,14 @@ Like C<rrd_add_ptr>, except adds a C<strdup> of the source string.
> if (!rrd_add_strdup(&arr, &arr_size, str))
> handle_failure();
>
> +=item B<rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src, size_t *alloc, size_t chunk)>
> +
> +Like C<rrd_add_strdup>, except the destination is allocated in chunks of
> +C<chunk>. C<alloc> points to the number of entries allocated, whereas
> +C<dest_size> points to the number of valid pointers. If more pointers are
> +needed, C<chunk> pointers are allocated and C<alloc> is increased
> +accordingly. C<alloc> must be E<gt>= C<dest_size>.
> +
> =item B<rrd_free_ptrs(void ***src, size_t *cnt)>
>
> Free an array of pointers allocated by C<rrd_add_ptr> or
> diff --git a/src/rrd.h b/src/rrd.h
> index ce7e749..6008794 100644
> --- a/src/rrd.h
> +++ b/src/rrd.h
> @@ -333,8 +333,12 @@ int rrd_proc_start_end(
>
> long rrd_random(void);
>
> + int rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src,
> + size_t *alloc, size_t chunk);
> int rrd_add_ptr(void ***dest, size_t *dest_size, void *src);
> int rrd_add_strdup(char ***dest, size_t *dest_size, char *src);
> + int rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src,
> + size_t *alloc, size_t chunk);
> void rrd_free_ptrs(void ***src, size_t *cnt);
>
> int rrd_mkdir_p(const char *pathname, mode_t mode);
> diff --git a/src/rrd_utils.c b/src/rrd_utils.c
> index 3936cff..6853c66 100644
> --- a/src/rrd_utils.c
> +++ b/src/rrd_utils.c
> @@ -50,29 +50,53 @@ long rrd_random(void)
> return random();
> }
>
> -/* rrd_add_ptr: add a pointer to a dynamically sized array of pointers,
> - * realloc as necessary. returns 1 on success, 0 on failure.
> +/* rrd_add_ptr_chunk: add a pointer to a dynamically sized array of
> + * pointers, realloc as necessary in multiples of "chunk".
> + *
> + * "alloc" is the number of pointers allocated
> + * "dest_size" is the number of valid pointers
> + *
> + * returns 1 on success, 0 on failure.
> */
>
> -int rrd_add_ptr(void ***dest, size_t *dest_size, void *src)
> +int rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src,
> + size_t *alloc, size_t chunk)
> {
> void **temp;
>
> assert(dest != NULL);
> + assert(alloc != NULL);
> + assert(*alloc >= *dest_size);
>
> - temp = (void **) rrd_realloc(*dest, (*dest_size+1) * sizeof(*dest));
> - if (!temp)
> - return 0;
> + if (*alloc == *dest_size)
> + {
> + temp = (void **) rrd_realloc(*dest, (*alloc+chunk) * sizeof(*dest));
> + if (!temp)
> + return 0;
>
> - *dest = temp;
> - temp[*dest_size] = src;
> + *dest = temp;
> + *alloc += chunk;
> + }
> +
> + (*dest)[*dest_size] = src;
> (*dest_size)++;
>
> return 1;
> }
>
> -/* like rrd_add_ptr, but calls strdup() on a string first. */
> -int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
> +/* rrd_add_ptr: add a pointer to a dynamically sized array of pointers,
> + * realloc as necessary. returns 1 on success, 0 on failure.
> + */
> +int rrd_add_ptr(void ***dest, size_t *dest_size, void *src)
> +{
> + size_t alloc = *dest_size;
> +
> + return rrd_add_ptr_chunk(dest, dest_size, src, &alloc, 1);
> +}
> +
> +/* like rrd_add_ptr_chunk, but calls strdup() on a string first. */
> +int rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src,
> + size_t *alloc, size_t chunk)
> {
> char *dup_src;
> int add_ok;
> @@ -84,13 +108,20 @@ int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
> if (!dup_src)
> return 0;
>
> - add_ok = rrd_add_ptr((void ***)dest, dest_size, (void *)dup_src);
> + add_ok = rrd_add_ptr_chunk((void ***)dest, dest_size, (void *)dup_src, alloc, chunk);
> if (!add_ok)
> free(dup_src);
>
> return add_ok;
> }
>
> +int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
> +{
> + size_t alloc = *dest_size;
> +
> + return rrd_add_strdup_chunk(dest, dest_size, src, &alloc, 1);
> +}
> +
> void rrd_free_ptrs(void ***src, size_t *cnt)
> {
> void **sp;
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: patch to do vertical gradients

2009-11-06T06:49:28Z

On Fri, Nov 6, 2009 at 7:20 AM, Mark Mills <extremely@...> wrote:
> If it was used to carry information, one that might help would be scale
> differences. I have had issues in the past where my boss at the time
> freaked out when a graph spiked and we had to point out that the one he
> was looking at had spiked from bits to kbits... not from mbits to gbits.
> ;-) A way to color the scale in the area (logarithmically?) would help
> when scanning graphs quickly. It wouldn't give the same cool fire effect
> but "ripples" in the area paint could effectively replace the need for
> grid lines.

Thats one of the reasons I wrote it. A negative height would be
aligned to the x axis, so that increasing values on the line change to
the next color, ie a gradient from green to red. Of course, to be
really useful, I would need to change it so that the height was
relative to the data set, and not given in pixels. I'll have to see
how to do that. gfx_area_fit maybe?

> I'd also recommend that the default second color be set to the graph
> background color with the transparency set to 0x20 and not be forced to
> #000000. Might as well make it do something cool by default no matter
> what color scheme they use.

Maybe. I figured it didn't matter much what the color was if the alpha
was zero. That way it always appeared to fade into whatever is behind
it, be it the background or another AREA. I'll have to play with it
and see what it looks like.

> Also, I'm jealous of Rian because I've been wanting to write this patch
> for like 5-6 years and never had the time. ;-) It looks even better than
> I hoped. I'm looking forward to this being in the main build.

All the heavy lifting is done by cairo :D. All I did was allow the
source pattern to be a gradient as well as a solid color. I think the
other option is to make the source pattern another image.

--
rian

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: patch to do vertical gradients

2009-11-06T06:20:54Z

Tobias Oetiker wrote:
>>> thanks for this ... it looks cool ... quersion though, is there
>>> some information carying capability in this feature ?
>>>
If it was used to carry information, one that might help would be scale
differences. I have had issues in the past where my boss at the time
freaked out when a graph spiked and we had to point out that the one he
was looking at had spiked from bits to kbits... not from mbits to gbits.
;-) A way to color the scale in the area (logarithmically?) would help
when scanning graphs quickly. It wouldn't give the same cool fire effect
but "ripples" in the area paint could effectively replace the need for
grid lines.

I'd also recommend that the default second color be set to the graph
background color with the transparency set to 0x20 and not be forced to
#000000. Might as well make it do something cool by default no matter
what color scheme they use.

Also, I'm jealous of Rian because I've been wanting to write this patch
for like 5-6 years and never had the time. ;-) It looks even better than
I hoped. I'm looking forward to this being in the main build.

--mark

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: patch to do vertical gradients

2009-11-05T12:10:35Z

Hi Rian,

Today Rian Shelley wrote:

> This time i remembered to attach the patch :D
>
>
> > thanks for this ... it looks cool ... quersion though, is there
> > some information carying capability in this feature ?
>
> Not really. For the most part, I wanted to add some control over how
> the AREA's were drawn, especially when you have two of them on the
> same graph that keep covering one another, it makes it easier for the
> eye to pick out the two graphs, as opposed to simply using an alpha,
> which gives a solid color that can be mis-interpreted as another
> graph.
>
> I also wanted the ability to make it so that peak values could be
> highlighted in a different color if desired, as in a cpu graph going
> from green through yellow to red.
>
> I haven't thought about making it carry information, although it might
> be neat to show temporal distribution of an average value (ie, turn
> the dataset on its side, and make it represent color instead of
> displacement). I don't really know how to get into that, since all
> I've played with so far is the drawing code.
>
> >
> > can you please also add a patch for the documentation ?
>
> Hopefully the attached patch will suffice.

great ... thanks very much
tobi

>
>
> --
> rian
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: patch to do vertical gradients

2009-11-05T11:33:34Z

This time i remembered to attach the patch :D

> thanks for this ... it looks cool ... quersion though, is there
> some information carying capability in this feature ?

Not really. For the most part, I wanted to add some control over how
the AREA's were drawn, especially when you have two of them on the
same graph that keep covering one another, it makes it easier for the
eye to pick out the two graphs, as opposed to simply using an alpha,
which gives a solid color that can be mis-interpreted as another
graph.

I also wanted the ability to make it so that peak values could be
highlighted in a different color if desired, as in a cpu graph going
from green through yellow to red.

I haven't thought about making it carry information, although it might
be neat to show temporal distribution of an average value (ie, turn
the dataset on its side, and make it represent color instead of
displacement). I don't really know how to get into that, since all
I've played with so far is the drawing code.

>
> can you please also add a patch for the documentation ?

Hopefully the attached patch will suffice.

--
rian

[grad_doc.patch]

diff -crB rrdtool-1.3.8/doc/rrdgraph_graph.pod ../rrdtool-1.3.8/doc/rrdgraph_graph.pod *** rrdtool-1.3.8/doc/rrdgraph_graph.pod Sat Feb 21 02:43:58 2009 --- ../rrdtool-1.3.8/doc/rrdgraph_graph.pod Thu Nov 5 12:23:08 2009 *************** *** 262,267 **** --- 262,275 ---- See B<LINE>, however the area between the x-axis and the line will be filled. + =head3 B<GRAD>B<:>I<value>[B<#>I<color1>[B<#>I<color2>B<:>I<height>][B<:>[I<legend>][B<:STACK>]] + + Similar to B<AREA>, except the area between the line and the x-axis will contain a gradient from color1 to color2. + + The I<height> parameter can create three different behaviors. If I<height> > 0, then the gradient is a fixed height, starting at the line going down. If I<height> < 0, then the gradient starts at fixed height above the x-axis, going down to the x-axis. If I<height> == 0, then the gradient goes from the line to x-axis. + + If not present, I<color2> defaults to #00000000 and I<height> defaults to 50. + =head3 B<TICK>B<:>I<vname>B<#>I<rrggbb>[I<aa>][B<:>I<fraction>[B<:>I<legend>]] Plot a tick mark (a vertical line) for each value of I<vname> that is

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: patch to do vertical gradients

2009-11-05T11:32:20Z

> thanks for this ... it looks cool ... quersion though, is there
> some information carying capability in this feature ?

Not really. For the most part, I wanted to add some control over how
the AREA's were drawn, especially when you have two of them on the
same graph that keep covering one another, it makes it easier for the
eye to pick out the two graphs, as opposed to simply using an alpha,
which gives a solid color that can be mis-interpreted as another
graph.

I also wanted the ability to make it so that peak values could be
highlighted in a different color if desired, as in a cpu graph going
from green through yellow to red.

I haven't thought about making it carry information, although it might
be neat to show temporal distribution of an average value (ie, turn
the dataset on its side, and make it represent color instead of
displacement). I don't really know how to get into that, since all
I've played with so far is the drawing code.

>
> can you please also add a patch for the documentation ?

Hopefully the attached patch will suffice.

--
rian

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

patch to do vertical gradients

2009-11-05T10:08:39Z

Somebody on the irc channel suggested that I post this patch here. The
patch is against rrdtool 1.3.8.

This adds the GRAD element, which acts more or less like an AREA,
except you can specify a second color and a height which is used to
create a gradient from one color to the next.

I also attached an example graph that uses this patch. The gradients
in it are generated with these commands:

GRAD:ips#4444ff:half\\ firewall\\ states
GRAD:Bin#00cc00#0099441f:0:incoming\\ bandwidth

The syntax is
vname-or-value[#color[#color:gradientheight][:legend]][:STACK]

if the second color is missing, it is assumed to be #00000000 and the
gradientheight defaults to 50

gradientheight can actually produce 3 kinds of results depending on its value:
gradientheight > 0
generates a fixed-height gradient from the data point down. (fire style)
gradientheight < 0
generates a fixed-height gradient from the 0 line to the data
point. (fixed style)
gradientheight == 0
generates a gradient that is stretched from the data point to the 0 line.
this is what the example image is using.

I would appreciate any comments or suggestions.

--
rian

[grad.patch]

diff -crB rrdtool-1.3.8/src/rrd_gfx.c ../rrdtool-1.3.8/src/rrd_gfx.c *** rrdtool-1.3.8/src/rrd_gfx.c Tue May 19 07:45:05 2009 --- ../rrdtool-1.3.8/src/rrd_gfx.c Thu Nov 5 10:17:01 2009 *************** *** 105,110 **** --- 106,147 ---- cairo_line_to(cr, x, y); } + /* add a point to a line or to an area */ + void gfx_add_rect_fadey( + image_desc_t *im, + double x1,double y1, + double x2,double y2, + double py, + gfx_color_t color1, + gfx_color_t color2, + double height) + { + cairo_t *cr = im->cr; + + cairo_new_path(cr); + gfx_area_fit(im, &x1, &y1); + gfx_area_fit(im, &x2, &y2); + cairo_line_to(cr, x1, y1); + cairo_line_to(cr, x1, y2); + cairo_line_to(cr, x2, y2); + cairo_line_to(cr, x2, y1); + cairo_close_path(cr); + cairo_pattern_t* p; + if (height < 0) { + p = cairo_pattern_create_linear(x1,y1,x2,y1+height); + } else if (height > 0) { + p = cairo_pattern_create_linear(x1,(y2+py)/2+height,x2,(y2+py)/2); + } else { + p = cairo_pattern_create_linear(x1,y1,x2,(y2+py)/2); + } + //cairo_pattern_t* p = cairo_pattern_create_linear(x1,py+50,x2,py); + cairo_pattern_add_color_stop_rgba(p, 1, color1.red,color1.green,color1.blue,color1.alpha); + cairo_pattern_add_color_stop_rgba(p, 0, color2.red,color2.green,color2.blue,color2.alpha); + cairo_set_source(cr, p); + cairo_pattern_destroy(p); + cairo_fill(cr); + } + void gfx_close_path( image_desc_t *im) { diff -crB rrdtool-1.3.8/src/rrd_graph.c ../rrdtool-1.3.8/src/rrd_graph.c *** rrdtool-1.3.8/src/rrd_graph.c Tue May 19 07:45:05 2009 --- ../rrdtool-1.3.8/src/rrd_graph.c Thu Nov 5 10:19:45 2009 *************** *** 227,232 **** --- 227,233 ---- conv_if(VRULE, GF_VRULE); conv_if(LINE, GF_LINE); conv_if(AREA, GF_AREA); + conv_if(GRAD, GF_GRAD); conv_if(STACK, GF_STACK); conv_if(TICK, GF_TICK); conv_if(TEXTALIGN, GF_TEXTALIGN); *************** *** 1176,1182 **** /* memory for the processed data */ for (i = 0; i < im->gdes_c; i++) { if ((im->gdes[i].gf == GF_LINE) || ! (im->gdes[i].gf == GF_AREA) || (im->gdes[i].gf == GF_TICK)) { if ((im->gdes[i].p_data = (rrd_value_t*)(malloc((im->xsize + 1) * sizeof(rrd_value_t)))) == NULL) { rrd_set_error("malloc data_proc"); --- 1177,1185 ---- /* memory for the processed data */ for (i = 0; i < im->gdes_c; i++) { if ((im->gdes[i].gf == GF_LINE) || ! (im->gdes[i].gf == GF_AREA) || ! (im->gdes[i].gf == GF_GRAD) || ! (im->gdes[i].gf == GF_TICK)) { if ((im->gdes[i].p_data = (rrd_value_t*)(malloc((im->xsize + 1) * sizeof(rrd_value_t)))) == NULL) { rrd_set_error("malloc data_proc"); *************** *** 1197,1202 **** --- 1200,1206 ---- switch (im->gdes[ii].gf) { case GF_LINE: case GF_AREA: + case GF_GRAD: case GF_TICK: if (!im->gdes[ii].stack) paintval = 0.0; *************** *** 1576,1581 **** --- 1580,1586 ---- break; case GF_LINE: case GF_AREA: + case GF_GRAD: case GF_TICK: graphelement = 1; break; *************** *** 3208,3213 **** --- 3213,3219 ---- break; case GF_LINE: case GF_AREA: + case GF_GRAD: /* fix data points at oo and -oo */ for (ii = 0; ii < im->xsize; ii++) { if (isinf(im->gdes[i].p_data[ii])) { *************** *** 3306,3311 **** --- 3312,3319 ---- cairo_stroke(im->cr); cairo_restore(im->cr); } else { + double lastx=0; + double lasty=0; int idxI = -1; double *foreY = (double *) malloc(sizeof(double) * im->xsize * 2); *************** *** 3336,3347 **** [cntI + 1], 4)) { cntI++; } ! gfx_new_area(im, ! backX[0], backY[0], ! foreX[0], foreY[0], ! foreX[cntI], ! foreY[cntI], im->gdes[i].col); ! while (cntI < idxI) { lastI = cntI; cntI++; while (cntI < idxI --- 3344,3360 ---- [cntI + 1], 4)) { cntI++; } ! if (im->gdes[i].gf != GF_GRAD) { ! gfx_new_area(im, ! backX[0], backY[0], ! foreX[0], foreY[0], ! foreX[cntI], ! foreY[cntI], im->gdes[i].col); ! } else { ! lastx = foreX[cntI]; ! lasty = foreY[cntI]; ! } ! while (cntI < idxI) { lastI = cntI; cntI++; while (cntI < idxI *************** *** 3357,3365 **** + 1], 4)) { cntI++; } ! gfx_add_point(im, foreX[cntI], foreY[cntI]); } ! gfx_add_point(im, backX[idxI], backY[idxI]); while (idxI > 1) { lastI = idxI; idxI--; --- 3370,3401 ---- + 1], 4)) { cntI++; } ! if (im->gdes[i].gf != GF_GRAD) { ! gfx_add_point(im, foreX[cntI], foreY[cntI]); ! } else { ! gfx_add_rect_fadey(im, ! lastx, foreY[0], ! foreX[cntI], foreY[cntI], lasty, ! im->gdes[i].col, ! im->gdes[i].col2, ! im->gdes[i].gradheight ! ); ! lastx = foreX[cntI]; ! lasty = foreY[cntI]; ! } } ! if (im->gdes[i].gf != GF_GRAD) { ! gfx_add_point(im, backX[idxI], backY[idxI]); ! } else { ! gfx_add_rect_fadey(im, ! lastx, foreY[0], ! backX[idxI], backY[idxI], lasty, ! im->gdes[i].col, ! im->gdes[i].col2, ! im->gdes[i].gradheight); ! lastx = backX[idxI]; ! lasty = backY[idxI]; ! } while (idxI > 1) { lastI = idxI; idxI--; *************** *** 3376,3386 **** - 1], 4)) { idxI--; } ! gfx_add_point(im, backX[idxI], backY[idxI]); } idxI = -1; drawem = 0; ! gfx_close_path(im); } if (drawem != 0) { drawem = 0; --- 3412,3434 ---- - 1], 4)) { idxI--; } ! if (im->gdes[i].gf != GF_GRAD) { ! gfx_add_point(im, backX[idxI], backY[idxI]); ! } else { ! gfx_add_rect_fadey(im, ! lastx, foreY[0], ! backX[idxI], backY[idxI], lasty, ! im->gdes[i].col, ! im->gdes[i].col2, ! im->gdes[i].gradheight); ! lastx = backX[idxI]; ! lasty = backY[idxI]; ! } } idxI = -1; drawem = 0; ! if (im->gdes[i].gf != GF_GRAD) ! gfx_close_path(im); } if (drawem != 0) { drawem = 0; *************** *** 3574,3579 **** --- 3622,3632 ---- im->gdes[im->gdes_c - 1].col.green = 0.0; im->gdes[im->gdes_c - 1].col.blue = 0.0; im->gdes[im->gdes_c - 1].col.alpha = 0.0; + im->gdes[im->gdes_c - 1].col2.red = 0.0; + im->gdes[im->gdes_c - 1].col2.green = 0.0; + im->gdes[im->gdes_c - 1].col2.blue = 0.0; + im->gdes[im->gdes_c - 1].col2.alpha = 0.0; + im->gdes[im->gdes_c - 1].gradheight = 50.0; im->gdes[im->gdes_c - 1].legend[0] = '\0'; im->gdes[im->gdes_c - 1].format[0] = '\0'; im->gdes[im->gdes_c - 1].strftm = 0; diff -crB rrdtool-1.3.8/src/rrd_graph.h ../rrdtool-1.3.8/src/rrd_graph.h *** rrdtool-1.3.8/src/rrd_graph.h Fri Dec 26 01:05:03 2008 --- ../rrdtool-1.3.8/src/rrd_graph.h Thu Nov 5 10:36:52 2009 *************** *** 51,57 **** #define GRIDWIDTH 0.4 enum gf_en { GF_PRINT = 0, GF_GPRINT, GF_COMMENT, GF_HRULE, GF_VRULE, GF_LINE, ! GF_AREA, GF_STACK, GF_TICK, GF_TEXTALIGN, GF_DEF, GF_CDEF, GF_VDEF, GF_SHIFT, GF_XPORT }; --- 51,57 ---- #define GRIDWIDTH 0.4 enum gf_en { GF_PRINT = 0, GF_GPRINT, GF_COMMENT, GF_HRULE, GF_VRULE, GF_LINE, ! GF_AREA,GF_GRAD, GF_STACK, GF_TICK, GF_TEXTALIGN, GF_DEF, GF_CDEF, GF_VDEF, GF_SHIFT, GF_XPORT }; *************** *** 156,162 **** long ds; /* data source number */ enum cf_en cf; /* consolidation function */ enum cf_en cf_reduce; /* consolidation function for reduce_data() */ ! struct gfx_color_t col; /* graph color */ char format[FMT_LEG_LEN + 5]; /* format for PRINT AND GPRINT */ char legend[FMT_LEG_LEN + 5]; /* legend */ int strftm; /* should the VDEF legend be formated with strftime */ --- 156,163 ---- long ds; /* data source number */ enum cf_en cf; /* consolidation function */ enum cf_en cf_reduce; /* consolidation function for reduce_data() */ ! struct gfx_color_t col, col2; /* graph color */ ! double gradheight; char format[FMT_LEG_LEN + 5]; /* format for PRINT AND GPRINT */ char legend[FMT_LEG_LEN + 5]; /* legend */ int strftm; /* should the VDEF legend be formated with strftime */ *************** *** 424,429 **** --- 425,449 ---- double x, double y); + /* create a rect that has a gradient from color1 to color2 in height pixels + * height > 0: + * gradient starts at top and goes down a fixed number of pixels (fire style) + * height < 0: + * gradient starts at bottom and goes up a fixed number of pixels (constant style) + * height == 0: + * gradient is stretched between two points + */ + void gfx_add_rect_fadey( + image_desc_t *im, + double x1,double y1, + double x2,double y2, + double py, + gfx_color_t color1, + gfx_color_t color2, + double height); + + + /* close current path so it ends at the same point as it started */ void gfx_close_path( image_desc_t *im); diff -crB rrdtool-1.3.8/src/rrd_graph_helper.c ../rrdtool-1.3.8/src/rrd_graph_helper.c *** rrdtool-1.3.8/src/rrd_graph_helper.c Tue May 19 07:45:05 2009 --- ../rrdtool-1.3.8/src/rrd_graph_helper.c Thu Nov 5 10:38:27 2009 *************** *** 532,538 **** graph_desc_t *const gdp, image_desc_t *const im) { ! int i, j, k; int colorfound = 0; char tmpstr[MAX_VNAME_LEN + 10]; /* vname#RRGGBBAA\0 */ static int spacecnt = 0; --- 532,538 ---- graph_desc_t *const gdp, image_desc_t *const im) { ! int i, j, k, j2; int colorfound = 0; char tmpstr[MAX_VNAME_LEN + 10]; /* vname#RRGGBBAA\0 */ static int spacecnt = 0; *************** *** 567,576 **** rrd_set_error("Cannot parse line '%s'", line); return 1; } ! ! j = i; while (j > 0 && tmpstr[j] != '#') j--; if (j) { tmpstr[j] = '\0'; --- 567,588 ---- rrd_set_error("Cannot parse line '%s'", line); return 1; } ! ! j = i; while (j > 0 && tmpstr[j] != '#') j--; + //see if there is a second color + j2 = j-1; + while (j2 > 0 && tmpstr[j2] != '#') + j2--; + if (j && j2) { //yes, swap j and j2, so that j is first color, j2 is second + int tmp = j; + j = j2; + j2 = tmp; + tmpstr[j2] = '\0'; + } else { + j2 = 0; + } if (j) { tmpstr[j] = '\0'; *************** *** 578,585 **** /* We now have: * tmpstr[0] containing vname * tmpstr[j] if j!=0 then containing color ! * i size of vname + color * j if j!=0 then size of vname */ /* Number or vname ? --- 590,599 ---- /* We now have: * tmpstr[0] containing vname * tmpstr[j] if j!=0 then containing color ! * tmpstr[j2] if j2!=0 then containing second color ! * i size of vname * j if j!=0 then size of vname + * j2 if j2!=0 then size of vname + first color */ /* Number or vname ? *************** *** 644,649 **** --- 658,690 ---- dprintf("- parsed color %0.0f,%0.0f,%0.0f,%0.0f\n", gdp->col.red, gdp->col.green, gdp->col.blue, gdp->col.alpha); colorfound = 1; + if (j2) { //second color? + j2++; + dprintf("- examining second color '%s'\n", &tmpstr[j2]); + //TODO: maybe rrd_parse_color should take a pointer to gdp->col instead of gdp + struct gfx_color_t firstcol = gdp->col; + if (rrd_parse_color(&tmpstr[j2], gdp)) { + rrd_set_error("Could not parse color in '%s'", &tmpstr[j2]); + return 1; + } + dprintf("- parsed color %0.0f,%0.0f,%0.0f,%0.0f\n", gdp->col.red, + gdp->col.green, gdp->col.blue, gdp->col.alpha); + gdp->col2 = gdp->col; + gdp->col = firstcol; + //we now have a mandatory grid height + (*eaten) += i; + if (line[*eaten] != '\0') { + (*eaten)++; + } + dprintf("- examining gradient height\n"); + i = scan_for_col(&line[*eaten], MAX_VNAME_LEN + 9, tmpstr); + sscanf(tmpstr, "%lf%n", &gdp->gradheight, &j); + if (i != j) { + rrd_set_error("Could not parse gradient height in '%s'", tmpstr); + return 1; + } + dprintf("- parsed gradientheight %0.0f\n", gdp->gradheight); + } } else { dprintf("- no color present in '%s'\n", tmpstr); } *************** *** 1113,1118 **** --- 1154,1160 ---- case GF_HRULE: /* value#color[:legend] */ case GF_LINE: /* vname-or-value[#color[:legend]][:STACK] */ case GF_AREA: /* vname-or-value[#color[:legend]][:STACK] */ + case GF_GRAD: /* vname-or-value[#color[:legend][#color[:gradientheight]]][:STACK] */ case GF_TICK: /* vname#color[:num[:legend]] */ if (rrd_parse_PVHLAST(argv[i], &eaten, gdp, im)) return; *************** *** 1122,1128 **** case GF_STACK: /* vname-or-value[#color[:legend]] */ if (rrd_parse_PVHLAST(argv[i], &eaten, gdp, im)) return; ! if (last_gf == GF_LINE || last_gf == GF_AREA) { gdp->gf = last_gf; gdp->linewidth = last_linewidth; } else { --- 1164,1170 ---- case GF_STACK: /* vname-or-value[#color[:legend]] */ if (rrd_parse_PVHLAST(argv[i], &eaten, gdp, im)) return; ! if (last_gf == GF_LINE || last_gf == GF_AREA || last_gf == GF_GRAD) { gdp->gf = last_gf; gdp->linewidth = last_linewidth; } else {

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

samplegradientgraph.png (98K) Download Attachment

Re: Auth in rrdtools 1.5 - library suggestion

2009-11-02T18:35:45Z

Hi Thomas,

I just had a look at that page and it does indeed sound interesting
...

thanks for the hint

tobi

Yesterday Thomas Guyot-Sionnest wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
>
> I've seen in a wiki page that rrdtool may use authentication or
> authorization in the future. I'd like to suggest a possible library for
> this task: NaCl. This is a fairly new library mare with simplicity and
> high speed in mind. The project was started by Daniel J. Berstein (aka.
> djb), a cryptologist very well known for his very secure software
> (djbdns, Qmail...).
>
> The NaCL library project page:
> http://nacl.cace-project.eu/
>
> djb's home page:
> http://cr.yp.to/djb.html
>
> I believe this library should seriously be considered as an alternative
> to TLS (openssl, gnutls) or other libraries for both its speed and security.
>
> Thank you
>
> - --
> Thomas
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkrvEHcACgkQ6dZ+Kt5BchZS2wCgjqaLOSpwMNBJZ4qtq8V9/m0a
> Az4An0445M1oIbJNQvl1ktg74JmCQ4HT
> =v8Qf
> -----END PGP SIGNATURE-----
>
>

Re: rrdcached performance problem

2009-10-31T13:14:19Z

> It's pretty clear that the performance when operating in memory is very
> nice and smooth, but when memory is exceeded things become bumpy. It
> degrades nicely and recovers well: I saw 9k queue items at one point and
> that got worked off over close to an hour but did work itself out as
> opposed to spinning out of control.

9k/hour seems awfully slow. Even my old development box can flush
55k/hour. Maybe more entries are being enqueued at the same time...

> What value of -m do you suggest? -w divided by step size?

That's the maximum; any more than that is just waste. IIRC you had around
~400 cached values per RRD file. So, if you use '-m 400' you'll only have
call realloc() one time. However, the 401st value will cause a large
realloc()...

The ideal value is highly dependent on your malloc() implementation.. I
would start with a power of two large enough to get you relatively few
realloc() but small enough that you don't waste too much... In your case,
maybe '-m 64'.

--
kevin brintnall =~ /kbrint@.../

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-31T12:12:31Z

kevin brintnall wrote:

> On Sat, Oct 31, 2009 at 09:52:15AM -0700, Thorsten von Eicken wrote:
>
>> Quick follow-up. I decided to add another 3k updates per second (extra
>> 30k tree nodes) to my test run. See results in
>> http://www.voneicken.com/dl/rrd/rrdcached-7.png
>> What's interesting is that the server got somewhat overloaded sitting a
>> lot in I/O wait. By and large the flush queue length remained under
>> control, except when doing backups (10pm, 8:30am). Memory usage by
>> rrdcached and collectd remained under control, but there is a long term
>> upward-trending slope to rrdcached's memory usage which is not good.
>> Possibly related to the power-of-two allocator patch that Florian
>> provided. The graph I find the most interesting one is the disk sdk disk
>> ops (3rd from the end). Before adding the last chunk of traffic the disk
>> load was write-dominated, which means that rrds were mostly cached in
>> memory (5-6 GB left after the processes). After adding the extra load
>> the disk load became read-dominated indicating that the rrd working set
>> exceeded memory.
>>
>
> Thorsten,
>
> If you're becoming read dominated, you should consider lowering your file
> update/sec rate by increasing your -w/-f timers. This just trades one
> kind of cache memory (f/s blocks) for another (update strings).
>

Yeah, but given that I'm already using 1 hour of caching it starts
getting a bit uncomfortable. I haven't re-tested the journal reading
yet, but at some point restarting rrdcached becomes really difficult.
But regardless, I suspect there are 3 performance regimes:
- working set much smaller than memory - cpu scales linearly/smoothly
with update rate
- working set much larger than memory - it's all about random small
disk I/O throughput
- working set similar to memory - relatively sharp (?) transition from
cpu/memory speed to disk speed
It's pretty clear that the performance when operating in memory is very
nice and smooth, but when memory is exceeded things become bumpy. It
degrades nicely and recovers well: I saw 9k queue items at one point and
that got worked off over close to an hour but did work itself out as
opposed to spinning out of control. But it doesn't look like a safe
operating regime. So basically gotta keep the rrd working set in memory.

> I'm sending a linear chunk allocator along for allocating
> cache_item_t.values in operator-defined block sizes.. I'd appreciate if
> you'd test it with your load to see if it reduces your CPU usage related
> to frequent realloc().
>
>
What value of -m do you suggest? -w divided by step size?

Thanks,
Thorsten

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

[PATCH 2/2] Introduce "-m" argument to reduce calls to realloc().

2009-10-31T11:46:19Z

---
doc/rrdcached.pod | 9 +++++++++
src/rrd_daemon.c | 23 ++++++++++++++++++++---
2 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/doc/rrdcached.pod b/doc/rrdcached.pod
index 043e020..5e8728f 100644
--- a/doc/rrdcached.pod
+++ b/doc/rrdcached.pod
@@ -18,6 +18,7 @@ B<rrdcached>
[-F]
[-g]
[B<-b>E<nbsp>I<base_dir>E<nbsp>[B<-B>]]
+[B<-m>E<nbsp>I<alloc_size>]

=head1 DESCRIPTION

@@ -197,6 +198,14 @@ Only permit writes into the base directory specified in B<-b> (and any
sub-directories). This does B<NOT> detect symbolic links. Paths
containing C<../> will also be blocked.

+=item B<-m> I<alloc_size>
+
+Allocate value pointers in chunks of I<alloc_size>. This may improve CPU
+utilization on machines with slow C<realloc()> implementations, in
+exchange for slightly higher memory utilization. The default isE<nbsp>1.
+Do not set this more than the B<-w> value divided by your average RRD step
+size.
+
=back

=head1 AFFECTED RRDTOOL COMMANDS
diff --git a/src/rrd_daemon.c b/src/rrd_daemon.c
index 4c84f19..79ea17b 100644
--- a/src/rrd_daemon.c
+++ b/src/rrd_daemon.c
@@ -174,7 +174,8 @@ struct cache_item_s
{
char *file;
char **values;
- size_t values_num;
+ size_t values_num; /* number of valid pointers */
+ size_t values_alloc; /* number of allocated pointers */
time_t last_flush_time;
time_t last_update_stamp;
#define CI_FLAGS_IN_TREE (1<<0)
@@ -251,6 +252,7 @@ static char *config_pid_file = NULL;
static char *config_base_dir = NULL;
static size_t _config_base_dir_len = 0;
static int config_write_base_only = 0;
+static size_t config_alloc_chunk = 1;

static listen_socket_t **config_listen_address_list = NULL;
static size_t config_listen_address_list_len = 0;
@@ -638,6 +640,7 @@ static void wipe_ci_values(cache_item_t *ci, time_t when)
{
ci->values = NULL;
ci->values_num = 0;
+ ci->values_alloc = 0;

ci->last_flush_time = when;
if (config_write_jitter > 0)
@@ -1434,7 +1437,8 @@ static int handle_request_update (HANDLER_PROTO) /* {{{ */
else
ci->last_update_stamp = stamp;

- if (!rrd_add_strdup(&ci->values, &ci->values_num, value))
+ if (!rrd_add_strdup_chunk(&ci->values, &ci->values_num, value,
+ &ci->values_alloc, config_alloc_chunk))
{
RRDD_LOG (LOG_ERR, "handle_request_update: rrd_add_strdup failed.");
continue;
@@ -2746,7 +2750,7 @@ static int read_options (int argc, char **argv) /* {{{ */
char **permissions = NULL;
size_t permissions_len = 0;

- while ((option = getopt(argc, argv, "gl:P:f:w:z:t:Bb:p:Fj:h?")) != -1)
+ while ((option = getopt(argc, argv, "gl:P:f:w:z:t:Bb:p:Fj:m:h?")) != -1)
{
switch (option)
{
@@ -3003,6 +3007,19 @@ static int read_options (int argc, char **argv) /* {{{ */
}
break;

+ case 'm':
+ {
+ int temp = atoi(optarg);
+ if (temp > 0)
+ config_alloc_chunk = temp;
+ else
+ {
+ fprintf(stderr, "Invalid allocation size: %s\n", optarg);
+ status = 10;
+ }
+ }
+ break;
+
case 'h':
case '?':
printf ("RRDCacheD %s\n"
--
1.6.4

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

[PATCH 1/2] Add utility functions to allocate pointers in variable size chunks.

2009-10-31T11:46:18Z

---
doc/librrd.pod | 18 ++++++++++++++++++
src/rrd.h | 4 ++++
src/rrd_utils.c | 53 ++++++++++++++++++++++++++++++++++++++++++-----------
3 files changed, 64 insertions(+), 11 deletions(-)

diff --git a/doc/librrd.pod b/doc/librrd.pod
index 038746c..396e50c 100644
--- a/doc/librrd.pod
+++ b/doc/librrd.pod
@@ -81,6 +81,16 @@ end of the new C<dest>. Returns 1 on success, 0 on failure.
if (!rrd_add_ptr(&arr, &arr_size, elem))
handle_failure();

+=item B<rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src, size_t *alloc, size_t chunk)>
+
+Like C<rrd_add_ptr>, except the destination is allocated in chunks of
+C<chunk>. C<alloc> points to the number of entries allocated, whereas
+C<dest_size> points to the number of valid pointers. If more pointers are
+needed, C<chunk> pointers are allocated and C<alloc> is increased
+accordingly. C<alloc> must be E<gt>= C<dest_size>.
+
+This method improves performance on hosts with expensive C<realloc()>.
+
=item B<rrd_add_strdup(char ***dest, size_t *dest_size, char *src)>

Like C<rrd_add_ptr>, except adds a C<strdup> of the source string.
@@ -91,6 +101,14 @@ Like C<rrd_add_ptr>, except adds a C<strdup> of the source string.
if (!rrd_add_strdup(&arr, &arr_size, str))
handle_failure();

+=item B<rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src, size_t *alloc, size_t chunk)>
+
+Like C<rrd_add_strdup>, except the destination is allocated in chunks of
+C<chunk>. C<alloc> points to the number of entries allocated, whereas
+C<dest_size> points to the number of valid pointers. If more pointers are
+needed, C<chunk> pointers are allocated and C<alloc> is increased
+accordingly. C<alloc> must be E<gt>= C<dest_size>.
+
=item B<rrd_free_ptrs(void ***src, size_t *cnt)>

Free an array of pointers allocated by C<rrd_add_ptr> or
diff --git a/src/rrd.h b/src/rrd.h
index ce7e749..6008794 100644
--- a/src/rrd.h
+++ b/src/rrd.h
@@ -333,8 +333,12 @@ int rrd_proc_start_end(

long rrd_random(void);

+ int rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src,
+ size_t *alloc, size_t chunk);
int rrd_add_ptr(void ***dest, size_t *dest_size, void *src);
int rrd_add_strdup(char ***dest, size_t *dest_size, char *src);
+ int rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src,
+ size_t *alloc, size_t chunk);
void rrd_free_ptrs(void ***src, size_t *cnt);

int rrd_mkdir_p(const char *pathname, mode_t mode);
diff --git a/src/rrd_utils.c b/src/rrd_utils.c
index 3936cff..6853c66 100644
--- a/src/rrd_utils.c
+++ b/src/rrd_utils.c
@@ -50,29 +50,53 @@ long rrd_random(void)
return random();
}

-/* rrd_add_ptr: add a pointer to a dynamically sized array of pointers,
- * realloc as necessary. returns 1 on success, 0 on failure.
+/* rrd_add_ptr_chunk: add a pointer to a dynamically sized array of
+ * pointers, realloc as necessary in multiples of "chunk".
+ *
+ * "alloc" is the number of pointers allocated
+ * "dest_size" is the number of valid pointers
+ *
+ * returns 1 on success, 0 on failure.
*/

-int rrd_add_ptr(void ***dest, size_t *dest_size, void *src)
+int rrd_add_ptr_chunk(void ***dest, size_t *dest_size, void *src,
+ size_t *alloc, size_t chunk)
{
void **temp;

assert(dest != NULL);
+ assert(alloc != NULL);
+ assert(*alloc >= *dest_size);

- temp = (void **) rrd_realloc(*dest, (*dest_size+1) * sizeof(*dest));
- if (!temp)
- return 0;
+ if (*alloc == *dest_size)
+ {
+ temp = (void **) rrd_realloc(*dest, (*alloc+chunk) * sizeof(*dest));
+ if (!temp)
+ return 0;

- *dest = temp;
- temp[*dest_size] = src;
+ *dest = temp;
+ *alloc += chunk;
+ }
+
+ (*dest)[*dest_size] = src;
(*dest_size)++;

return 1;
}

-/* like rrd_add_ptr, but calls strdup() on a string first. */
-int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
+/* rrd_add_ptr: add a pointer to a dynamically sized array of pointers,
+ * realloc as necessary. returns 1 on success, 0 on failure.
+ */
+int rrd_add_ptr(void ***dest, size_t *dest_size, void *src)
+{
+ size_t alloc = *dest_size;
+
+ return rrd_add_ptr_chunk(dest, dest_size, src, &alloc, 1);
+}
+
+/* like rrd_add_ptr_chunk, but calls strdup() on a string first. */
+int rrd_add_strdup_chunk(char ***dest, size_t *dest_size, char *src,
+ size_t *alloc, size_t chunk)
{
char *dup_src;
int add_ok;
@@ -84,13 +108,20 @@ int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
if (!dup_src)
return 0;

- add_ok = rrd_add_ptr((void ***)dest, dest_size, (void *)dup_src);
+ add_ok = rrd_add_ptr_chunk((void ***)dest, dest_size, (void *)dup_src, alloc, chunk);
if (!add_ok)
free(dup_src);

return add_ok;
}

+int rrd_add_strdup(char ***dest, size_t *dest_size, char *src)
+{
+ size_t alloc = *dest_size;
+
+ return rrd_add_strdup_chunk(dest, dest_size, src, &alloc, 1);
+}
+
void rrd_free_ptrs(void ***src, size_t *cnt)
{
void **sp;
--
1.6.4

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-31T11:42:53Z

On Sat, Oct 31, 2009 at 09:52:15AM -0700, Thorsten von Eicken wrote:

> Quick follow-up. I decided to add another 3k updates per second (extra
> 30k tree nodes) to my test run. See results in
> http://www.voneicken.com/dl/rrd/rrdcached-7.png
> What's interesting is that the server got somewhat overloaded sitting a
> lot in I/O wait. By and large the flush queue length remained under
> control, except when doing backups (10pm, 8:30am). Memory usage by
> rrdcached and collectd remained under control, but there is a long term
> upward-trending slope to rrdcached's memory usage which is not good.
> Possibly related to the power-of-two allocator patch that Florian
> provided. The graph I find the most interesting one is the disk sdk disk
> ops (3rd from the end). Before adding the last chunk of traffic the disk
> load was write-dominated, which means that rrds were mostly cached in
> memory (5-6 GB left after the processes). After adding the extra load
> the disk load became read-dominated indicating that the rrd working set
> exceeded memory.

Thorsten,

If you're becoming read dominated, you should consider lowering your file
update/sec rate by increasing your -w/-f timers. This just trades one
kind of cache memory (f/s blocks) for another (update strings).

I'm sending a linear chunk allocator along for allocating
cache_item_t.values in operator-defined block sizes.. I'd appreciate if
you'd test it with your load to see if it reduces your CPU usage related
to frequent realloc().

--
kevin brintnall =~ /kbrint@.../

> Thorsten
>
>
> Thorsten von Eicken wrote:
> > Thorsten von Eicken wrote:
> >
> >>> 37.1 % of the time it spent in ?handle_request_update? the daemon is
> >>> actually waiting for ?realloc?. This is (to me) very unexpected and a
> >>> schoolbook example of ?measure before you optimize?.
> >>>
> >>> I think we can get rid of this bottleneck by writing a specialized
> >>> version of ?rrd_add_strdup? which reallocates powers of ten. Something
> >>> like:
> >>>
> >>> [...]
> >>>
> >>> It'd be great if you could give the attached patch a try
> >>>
> >> spends all of its time (more than 99 %) in ?realloc?. So in consequence
> >> Test is running, including Kevin's simplification... Thanks for the
> >> help!!!
> >>
> > Things are again looking much better, almost great I should say! The one
> > thing that still makes me a bit uncomfortable is that at the end of the
> > second hour of run-time there was a cpu spike which caused collectd to
> > grow rapidly. (Still using -w 3600 -z 3600 -f 7200, I put a load of ~50k
> > tree nodes right from the start.) You can see the graphs at
> > http://www.voneicken.com/dl/rrd/ look for the rrdcached-6* series. It
> > flattens out nicely after the spike, but it's one of those things that
> > tend to bite sooner or later. I'm not sure what to do about it.
> > Thorsten
> >
> > _______________________________________________
> > rrd-developers mailing list
> > rrd-developers@...
> > https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
> >
> >
>
> _______________________________________________
> rrd-developers mailing list
> rrd-developers@...
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-31T09:52:15Z

Quick follow-up. I decided to add another 3k updates per second (extra
30k tree nodes) to my test run. See results in
http://www.voneicken.com/dl/rrd/rrdcached-7.png
What's interesting is that the server got somewhat overloaded sitting a
lot in I/O wait. By and large the flush queue length remained under
control, except when doing backups (10pm, 8:30am). Memory usage by
rrdcached and collectd remained under control, but there is a long term
upward-trending slope to rrdcached's memory usage which is not good.
Possibly related to the power-of-two allocator patch that Florian
provided. The graph I find the most interesting one is the disk sdk disk
ops (3rd from the end). Before adding the last chunk of traffic the disk
load was write-dominated, which means that rrds were mostly cached in
memory (5-6 GB left after the processes). After adding the extra load
the disk load became read-dominated indicating that the rrd working set
exceeded memory.

Thorsten

Thorsten von Eicken wrote:

> Thorsten von Eicken wrote:
>
>>> 37.1 % of the time it spent in “handle_request_update” the daemon is
>>> actually waiting for “realloc”. This is (to me) very unexpected and a
>>> schoolbook example of “measure before you optimize”.
>>>
>>> I think we can get rid of this bottleneck by writing a specialized
>>> version of “rrd_add_strdup” which reallocates powers of ten. Something
>>> like:
>>>
>>> [...]
>>>
>>> It'd be great if you could give the attached patch a try
>>>
>> spends all of its time (more than 99 %) in “realloc”. So in consequence
>> Test is running, including Kevin's simplification... Thanks for the
>> help!!!
>>
> Things are again looking much better, almost great I should say! The one
> thing that still makes me a bit uncomfortable is that at the end of the
> second hour of run-time there was a cpu spike which caused collectd to
> grow rapidly. (Still using -w 3600 -z 3600 -f 7200, I put a load of ~50k
> tree nodes right from the start.) You can see the graphs at
> http://www.voneicken.com/dl/rrd/ look for the rrdcached-6* series. It
> flattens out nicely after the spike, but it's one of those things that
> tend to bite sooner or later. I'm not sure what to do about it.
> Thorsten
>
> _______________________________________________
> rrd-developers mailing list
> rrd-developers@...
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
>
>

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: [PATCH] JOURNAL_REPLAY macro clarifies our intentions

2009-10-30T15:50:26Z

Hi Kevin,
Today kevin brintnall wrote:

> ---
> src/rrd_daemon.c | 18 +++++++++---------
> 1 files changed, 9 insertions(+), 9 deletions(-)

it's getting better all the time ...

cheers
tobi

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

[PATCH] JOURNAL_REPLAY macro clarifies our intentions

2009-10-30T12:17:33Z

---
src/rrd_daemon.c | 18 +++++++++---------
1 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/src/rrd_daemon.c b/src/rrd_daemon.c
index 1946003..4c84f19 100644
--- a/src/rrd_daemon.c
+++ b/src/rrd_daemon.c
@@ -265,6 +265,7 @@ static uint64_t stats_journal_rotate = 0;
static pthread_mutex_t stats_lock = PTHREAD_MUTEX_INITIALIZER;

/* Journaled updates */
+#define JOURNAL_REPLAY(s) ((s) == NULL)
#define JOURNAL_BASE "rrd.journal"
static journal_set *journal_cur = NULL;
static journal_set *journal_old = NULL;
@@ -528,7 +529,7 @@ static int add_response_info(listen_socket_t *sock, char *fmt, ...) /* {{{ */
char buffer[CMD_MAX];
int len;

- if (sock == NULL) return 0; /* journal replay mode */
+ if (JOURNAL_REPLAY(sock)) return 0;
if (sock->batch_start) return 0; /* no extra info returned when in BATCH */

va_start(argp, fmt);
@@ -575,7 +576,7 @@ static int send_response (listen_socket_t *sock, response_code rc,
ssize_t wrote;
int rclen, len;

- if (sock == NULL) return rc; /* journal replay mode */
+ if (JOURNAL_REPLAY(sock)) return rc;

if (sock->batch_start)
{
@@ -1043,7 +1044,7 @@ static int check_file_access (const char *file, listen_socket_t *sock) /* {{{ */
assert(file != NULL);

if (!config_write_base_only
- || sock == NULL /* journal replay */
+ || JOURNAL_REPLAY(sock)
|| config_base_dir == NULL)
return 1;

@@ -1272,7 +1273,7 @@ static int handle_request_forget(HANDLER_PROTO) /* {{{ */

if (found == TRUE)
{
- if (sock != NULL)
+ if (!JOURNAL_REPLAY(sock))
journal_write("forget", file);

return send_response(sock, RESP_OK, "Gone!\n");
@@ -1312,7 +1313,7 @@ static int handle_request_update (HANDLER_PROTO) /* {{{ */
cache_item_t *ci;

/* save it for the journal later */
- if (sock != NULL)
+ if (!JOURNAL_REPLAY(sock))
strncpy(orig_buf, buffer, buffer_size);

status = buffer_get_field (&buffer, &buffer_size, &file);
@@ -1398,7 +1399,7 @@ static int handle_request_update (HANDLER_PROTO) /* {{{ */
assert (ci != NULL);

/* don't re-write updates in replay mode */
- if (sock != NULL)
+ if (!JOURNAL_REPLAY(sock))
journal_write("update", orig_buf);

while (buffer_size > 0)
@@ -1674,7 +1675,7 @@ static int socket_permission_check (listen_socket_t *sock, /* {{{ */
{
ssize_t i;

- if (sock == NULL) /* journal replay */
+ if (JOURNAL_REPLAY(sock))
return (1);

if (cmd == NULL)
@@ -1713,7 +1714,7 @@ static int socket_permission_add (listen_socket_t *sock, /* {{{ */
/* check whether commands are received in the expected context */
static int command_check_context(listen_socket_t *sock, command_t *cmd)
{
- if (sock == NULL)
+ if (JOURNAL_REPLAY(sock))
return (cmd->context & CMD_CONTEXT_JOURNAL);
else if (sock->batch_start)
return (cmd->context & CMD_CONTEXT_BATCH);
@@ -1765,7 +1766,6 @@ static int handle_request_help (HANDLER_PROTO) /* {{{ */
return send_response(sock, RESP_OK, resp_txt);
} /* }}} int handle_request_help */

-/* if sock==NULL, we are in journal replay mode */
static int handle_request (DISPATCH_PROTO) /* {{{ */
{
char *buffer_ptr = buffer;
--
1.6.4

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-30T09:57:44Z

On Fri, Oct 30, 2009 at 09:47:43AM -0700, Thorsten von Eicken wrote:

> > What's the start time on that daemon? If it was started slightly before
> > the graphs (~21:30-21:35), then it's possible that the CPU increase is
> > associated with the first flush (-f).
> >
> It was right at the start of the graphs. If you look at the
> "write-data_sets" graph, you can see very clearly how the flushing
> starts at 22:38-22:39 and the first hour of flushing ends 23:46-23:47.
> The cpu spike starts to build at 23:26-23:28 so it's not on a clean
> boundary at all.If you look at the if-packets graph you can see how
> inbound traffic is 100% stable throughout the whole run.

OK.. That's what I thought :|

> > Something is queueing a lot of files; I would only expect that to
> > result from the flush process if writing had stopped for a set of your
> > RRDs. For example, if you have a large number of RRD files in the
> > journal that aren't being re-written afterwards, then the first flush
> > would contain all those.

When the queue length jumps up, try issuing the "QUEUE" command to the
daemon. Maybe the list of queued files (or their order) will be
instructive.

--
kevin brintnall =~ /kbrint@.../

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-30T09:47:43Z

kevin brintnall wrote:

> On Fri, Oct 30, 2009 at 09:19:30AM -0700, Thorsten von Eicken wrote:
>
>> Thorsten von Eicken wrote:
>>
>>>> 37.1 % of the time it spent in ?handle_request_update? the daemon is
>>>> actually waiting for ?realloc?. This is (to me) very unexpected and a
>>>> schoolbook example of ?measure before you optimize?.
>>>>
>>>> I think we can get rid of this bottleneck by writing a specialized
>>>> version of ?rrd_add_strdup? which reallocates powers of ten. Something
>>>> like:
>>>>
>>>> [...]
>>>>
>>>> It'd be great if you could give the attached patch a try
>>>>
>>> spends all of its time (more than 99 %) in ?realloc?. So in consequence
>>> Test is running, including Kevin's simplification... Thanks for the
>>> help!!!
>>>
>> Things are again looking much better, almost great I should say! The one
>> thing that still makes me a bit uncomfortable is that at the end of the
>> second hour of run-time there was a cpu spike which caused collectd to
>> grow rapidly. (Still using -w 3600 -z 3600 -f 7200, I put a load of ~50k
>> tree nodes right from the start.) You can see the graphs at
>> http://www.voneicken.com/dl/rrd/ look for the rrdcached-6* series. It
>> flattens out nicely after the spike, but it's one of those things that
>> tend to bite sooner or later. I'm not sure what to do about it.
>>
>
> Thorsten,
>
> What's the start time on that daemon? If it was started slightly before
> the graphs (~21:30-21:35), then it's possible that the CPU increase is
> associated with the first flush (-f).
>

It was right at the start of the graphs. If you look at the
"write-data_sets" graph, you can see very clearly how the flushing
starts at 22:38-22:39 and the first hour of flushing ends 23:46-23:47.
The cpu spike starts to build at 23:26-23:28 so it's not on a clean
boundary at all.If you look at the if-packets graph you can see how
inbound traffic is 100% stable throughout the whole run.
> Something is queueing a lot of files; I would only expect that to result
> from the flush process if writing had stopped for a set of your RRDs. For
> example, if you have a large number of RRD files in the journal that
> aren't being re-written afterwards, then the first flush would contain all
> those.
>
> Did you use journal on startup for this test?
>
I cleared the journals before starting rrdcached.
> If you could profile during that "hump" it would be very instructive...
>
Yeah, no kidding. The question is how to reproduce this given the
callgrind slow-down. I may try later next week when I hope to have some
more time.

Thorsten

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-30T09:31:44Z

On Fri, Oct 30, 2009 at 09:19:30AM -0700, Thorsten von Eicken wrote:

> Thorsten von Eicken wrote:
> >> 37.1 % of the time it spent in ?handle_request_update? the daemon is
> >> actually waiting for ?realloc?. This is (to me) very unexpected and a
> >> schoolbook example of ?measure before you optimize?.
> >>
> >> I think we can get rid of this bottleneck by writing a specialized
> >> version of ?rrd_add_strdup? which reallocates powers of ten. Something
> >> like:
> >>
> >> [...]
> >>
> >> It'd be great if you could give the attached patch a try
> > spends all of its time (more than 99 %) in ?realloc?. So in consequence
> > Test is running, including Kevin's simplification... Thanks for the
> > help!!!
> Things are again looking much better, almost great I should say! The one
> thing that still makes me a bit uncomfortable is that at the end of the
> second hour of run-time there was a cpu spike which caused collectd to
> grow rapidly. (Still using -w 3600 -z 3600 -f 7200, I put a load of ~50k
> tree nodes right from the start.) You can see the graphs at
> http://www.voneicken.com/dl/rrd/ look for the rrdcached-6* series. It
> flattens out nicely after the spike, but it's one of those things that
> tend to bite sooner or later. I'm not sure what to do about it.

Thorsten,

What's the start time on that daemon? If it was started slightly before
the graphs (~21:30-21:35), then it's possible that the CPU increase is
associated with the first flush (-f).

Something is queueing a lot of files; I would only expect that to result
from the flush process if writing had stopped for a set of your RRDs. For
example, if you have a large number of RRD files in the journal that
aren't being re-written afterwards, then the first flush would contain all
those.

Did you use journal on startup for this test?

If you could profile during that "hump" it would be very instructive...

--
kevin brintnall =~ /kbrint@.../

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-30T09:19:30Z

Thorsten von Eicken wrote:

>> 37.1 % of the time it spent in “handle_request_update” the daemon is
>> actually waiting for “realloc”. This is (to me) very unexpected and a
>> schoolbook example of “measure before you optimize”.
>>
>> I think we can get rid of this bottleneck by writing a specialized
>> version of “rrd_add_strdup” which reallocates powers of ten. Something
>> like:
>>
>> [...]
>>
>> It'd be great if you could give the attached patch a try
> spends all of its time (more than 99 %) in “realloc”. So in consequence
> Test is running, including Kevin's simplification... Thanks for the
> help!!!

Things are again looking much better, almost great I should say! The one
thing that still makes me a bit uncomfortable is that at the end of the
second hour of run-time there was a cpu spike which caused collectd to
grow rapidly. (Still using -w 3600 -z 3600 -f 7200, I put a load of ~50k
tree nodes right from the start.) You can see the graphs at
http://www.voneicken.com/dl/rrd/ look for the rrdcached-6* series. It
flattens out nicely after the spike, but it's one of those things that
tend to bite sooner or later. I'm not sure what to do about it.
Thorsten

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

rrdcached protocol issue: clients don't reconnect

2009-10-29T22:20:53Z

I'm having problems with clients not reconnecting if I restart the
rrdcached daemon. For example, collectd prints out the following type of
error forever if I do a "service rrdcached restart" (using unix socket,
but I believe same happens with TCP):

Oct 30 05:03:52 sketchy1-1b collectd[3551]: rrdcached plugin:
rrdc_update (/rrds/.../memcached_command-set.rrd,
[1256878677:2904863359], 1) failed with status -3.

I believe this comes from rrd_client.c in response_read:
buffer_ptr = fgets (buffer, sizeof (buffer), sh);
if (buffer_ptr == NULL)
return (-3);
chomp (buffer);

Seems to me that this is to be expected:
"gets() and fgets() return s on success, and NULL on error or
when end of file occurs
while no characters have been read."

The proper error handling to close the connection seems to be missing?

Thorsten

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-29T21:55:07Z

Florian Forster wrote:
>> 547,131,570 < rrd_daemon.c:queue_thread_main'2 (263x) [/usr/bin/rrdcached]
>> 3,186,524 < rrd_daemon.c:queue_thread_main (3x) [/usr/bin/rrdcached]
>> 550,318,094 * rrd_update.c:rrd_update_r [/usr/lib64/librrd_th.so.4.1.0]
>>
>
> I wonder where this second update thread is coming from. Did you
> configure one or two? If you configured two, why isn't the second one
> doing any real work?
>
Yes, there are two configured. Your question is a good one. If I get to
run callgrind again I'll run it a bit longer and we'll see whether it
was due to the short observation window. I don't know how the
locking/dequeueing primitives work, is there perhaps something that
prefers one thread and the second doesn't run if the first one is fast
enough at dequeueing?

> spends all of its time (more than 99 %) in “realloc”. So in consequence
> 37.1 % of the time it spent in “handle_request_update” the daemon is
> actually waiting for “realloc”. This is (to me) very unexpected and a
> schoolbook example of “measure before you optimize”.
>
> I think we can get rid of this bottleneck by writing a specialized
> version of “rrd_add_strdup” which reallocates powers of ten. Something
> like:
>
> [...]
>
> It'd be great if you could give the attached patch a try

Test is running, including Kevin's simplification... Thanks for the help!!!

By the way, we were calculating some stats the other day and noticed
that we're collecting 680k RRDs (most have 1 DS) every 20 seconds.
That's 35K RRD updates per second. No, this is not on one server...

Cheers,
Thorsten

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-29T06:44:57Z

> spends all of its time (more than 99??%) in ???realloc???. So in consequence
> 37.1??% of the time it spent in ???handle_request_update??? the daemon is
> actually waiting for ???realloc???. This is (to me) very unexpected and a
> schoolbook example of ???measure before you optimize???.

Florian et al,

I'd rather see the realloc in linear chunks vs. exponential.. i.e. when we
run out, allocate another X pointers. We'll still reduce calls to
realloc(), but without risking exponential jumps in memory utilization.

i.e. we could add cache_item_t->values_alloc to keep track of how many
values have been allocated. Then, we can realloc() a new chunk on demand.
This chunk size could even be configurable.. a clever admin could
pre-allocate all the pointers (i.e. "-w"/rrd_step).

P.S.: is_power_of_two = old_size && !(old_size & (old_size-1));

--
kevin brintnall =~ /kbrint@.../

> + if (old_size > 0)
> + {
> + is_power_of_two = old_size;
> + while ((is_power_of_two & 0x01) == 0)
> + is_power_of_two >>= 1;
> + }
> +
> + /* If the current size is a power of two (or zero), increase the array size.
> + * Otherwise enough space has been allocated previously. */
> + if (old_size == 0)
> + new_size = 1;
> + else if (is_power_of_two == 1)
> + new_size = 2 * old_size;
> + else
> + new_size = old_size;
> +
> + /* We need to allocate more space. */
> + if (new_size != old_size)
> + {
> + char **tmp;
> +
> + tmp = realloc (*dest, new_size * sizeof (char *));
> + if (tmp == NULL)
> + return (ENOMEM);
> + *dest = tmp;
> + }
> +
> + /* Duplicate the string. On error, the array size is not decreased. This
> + * means the callee may have to free `dest' even if this function failed. */
> + (*dest)[old_size] = strdup (src);
> + if ((*dest)[old_size] == NULL)
> + return (ENOMEM);
> +
> + *dest_size = old_size + 1;
> + return (0);
> +} /* }}} int add_strdup_exp */
> +
> static int count_lines(char *str) /* {{{ */
> {
> int lines = 0;
> @@ -1433,7 +1486,7 @@ static int handle_request_update (HANDLER_PROTO) /* {{{ */
> else
> ci->last_update_stamp = stamp;
>
> - if (!rrd_add_strdup(&ci->values, &ci->values_num, value))
> + if (add_strdup_exp (&ci->values, &ci->values_num, value) != 0)
> {
> RRDD_LOG (LOG_ERR, "handle_request_update: rrd_add_strdup failed.");
> continue;
> --
> 1.6.3.3
>

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdcached performance problem

2009-10-29T01:40:05Z

Hi,

On Wed, Oct 28, 2009 at 11:25:03PM -0700, Thorsten von Eicken wrote:
> it's clearly active both receiving and flushing, and realloc is the
> top primitive that I see. Mhhh, I haven't dealt with these things in
> eons so I'll have to see whether there's a different distro I can use
> to get a better realloc implementation.

Disclaimer: I have not looked into the whole issue in detail yet, this
is just a more or less wild guess … ;-)

You're on Linux, right? So, I presume, you're using the GNU libc. IIrc,
the glibc memory management implementation (*alloc, free) is optimized
for memory usage rather than speed, so the speed is (supposedly - did
not check that myself) rather slow.

This might also further explain why Kevin did not stumble across that
issue, since (iIrc) he's on FreeBSD which might have a "better" (speed-
wise) implementation of those functions.

Since we're using the glib (as in GNOME lib - note the missing 'c' ;-))
anyway, we might give it a try to use their memory management functions
(g_*malloc*, g_free) - those are supposed to be quite a bit faster (yet,
I did not check that myself neither).

Cheers,
Sebastian

--
Sebastian "tokkee" Harl +++ GnuPG-ID: 0x8501C7FC +++ http://tokkee.org/

Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety. -- Benjamin Franklin

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

signature.asc (204 bytes) Download Attachment

RRDtool 1.4 - higher Performance and cool Features

2009-10-28T12:59:33Z

Just in time for LISA'09 I am releasing the all new RRDtool 1.4.
RRDtool 1.4 comes with a much anticipated RRDcache Daemon, elevating the
system to new performance levels it also contains a host of new and
improved features. See the List below. Get your copy from:

http://oss.oetiker.ch/rrdtool/

cheers tobi

Brought to you by the 2009 RRDtool sponsors:
-----------------------------------------------------------------------
GOLD: http://www.zenoss.com
http://www.groundworkopensource.com

SILVER: http://www.digicomp.ch http://www.atc-onlane.com
http://www.terreactive.ch http://hosteurope.de
http://www.hitmeister.de http://www.contego.com
http://www.sidarion.ch http://chinavasion.com
-----------------------------------------------------------------------
ps. if you would like to become a sponsor for 2010, be in touch!

#####################################
Major Changes between 1.3.x and 1.4.x
-------------------------------------
$Id: NEWS 1958 2009-10-27 21:40:09Z oetiker $

RRD Caching Daemon (rrdcached)
------------------------------
by Florian Forster and Kevin Brintnall

The RRD Caching Daemon can dramatically improve the 'update' performance
of your system. Due to file handling overheads, the time it takes todo one
update is virtually the same as to doing two updates in a row.

The Cache Daemon intercepts rrdtool update calls, assembling multiple
updates before writing them to the actual rrd file. When calling rrdtool
graph in such a setup, the command will tell the daemon to flush out all
pending updates for the rrd files, required to draw the graph.

See rrdcached documentation.

RRD Dumping and Restoring (rrdtool dump/restore)
------------------------------------------------
by Tobi Oetiker

The output of rrdtool dump has been adjusted to be simpler to parse by
existing xml parsers.

The restore core has been completely re-written yet again and is now relying
on an incremental xml parser. This has the advantage that the memory
consumption while restoring xml files is only slightly larger than the
resulting rrd file. Which is much less than the requirements of the 1.3 and
even 1.2

RRD Graphing functions (rrdtool graph)
--------------------------------------
by Martin Sperl

* VDEF PERCENTNAN (a PRECENT that ignores NAN)

* CDEF PREDICT and PREDICTSIGMA functions for on-the-fly
data prediction without the need to modify existing rrd files as it is
required for HoltWinters.

* LibDBI integration provides a path to read data directly of a supported
SQL database into rrdtool graph. See rrdgraph_libdbi documentation.

Miscellaneous Changes
---------------------
* graph legends can now be placed left, right or above the graph with the
new --legend-direction and --legend-positon placement options.
by Melchior Rabe

* switched to using automake 1.11 which provides a 'silent' build process,
causing errors and warnings to stand out much more than before.
by Tobi Oetiker

* switched from intltoolize to autopoint for the i18n configuration.
by Tobi Oetiker

* new graph option --grid-dash on:off to configure the dash length
in the grid painted over the graph by Tobi Oetiker

* lua bindings for rrdtool
by Fidelis Assis

* various improvements to rrd_open functions and mmap handling
by Daniel Pocock

* allow the HW smoothing window size to be set to 0 with rrdtool tune
by sylvain luiset

* new graph option --border to set the 3d border width
by Bernhard Reutner-Fischer

* draw different color swats depending on the type of line drawn in the
graph by Loïc Tortay

for more detail see the CHANGES file.

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900
_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: [PATCH] Avoid unnecessary string handling for UPDATE commands.

2009-10-27T22:18:19Z

Hi Kevin,

Today kevin brintnall wrote:

> - only copy as many bytes as necessary
> - during journal replay, avoid unnecessary copy (string is discarded)
>
> Thank you Thorsten von Eicken <tve@...>

you di have perfect timing :-) so it will be an 1.4.1 announement today. But I
guess the performance impact is well worth it. Thank you!

cheers
tobi

> ---
> src/rrd_daemon.c | 3 ++-
> 1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/src/rrd_daemon.c b/src/rrd_daemon.c
> index 0ca1818..1946003 100644
> --- a/src/rrd_daemon.c
> +++ b/src/rrd_daemon.c
> @@ -1312,7 +1312,8 @@ static int handle_request_update (HANDLER_PROTO) /* {{{ */
> cache_item_t *ci;
>
> /* save it for the journal later */
> - strncpy(orig_buf, buffer, sizeof(orig_buf)-1);
> + if (sock != NULL)
> + strncpy(orig_buf, buffer, buffer_size);
>
> status = buffer_get_field (&buffer, &buffer_size, &file);
> if (status != 0)
>

[PATCH] Avoid unnecessary string handling for UPDATE commands.

2009-10-27T20:52:44Z

- only copy as many bytes as necessary
- during journal replay, avoid unnecessary copy (string is discarded)

Thank you Thorsten von Eicken <tve@...>
---
src/rrd_daemon.c | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/src/rrd_daemon.c b/src/rrd_daemon.c
index 0ca1818..1946003 100644
--- a/src/rrd_daemon.c
+++ b/src/rrd_daemon.c
@@ -1312,7 +1312,8 @@ static int handle_request_update (HANDLER_PROTO) /* {{{ */
cache_item_t *ci;

/* save it for the journal later */
- strncpy(orig_buf, buffer, sizeof(orig_buf)-1);
+ if (sock != NULL)
+ strncpy(orig_buf, buffer, buffer_size);

status = buffer_get_field (&buffer, &buffer_size, &file);
if (status != 0)
--
1.6.4

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: No ordering information in info hash

2009-10-27T16:33:34Z

Hi Hermann,

try rrdtool-1.4.0.tar.gz ... which I just uploaded ... will
announce tomorrow.

cheers
tobi

Yesterday Hermann Lauer wrote:

> Hello Tobi,
>
> On Tue, Oct 27, 2009 at 09:43:40PM +0100, Tobias Oetiker wrote:
> > Hi Hermann,
> >
> > there was an extension recently added a ds[x].index entry which
> > will help you resolve this problem ...
>
> Great. Sounds exactly like what I missed.
>
> > make sure you use a current snapshot of the archive ...
>
> Just downloaded rrdtool-trunk-svn-snap.tar.gz, which unpacked to
> rrdtool-1.3.99909102700. "configure" and "make" ends with:
>
> CC rrd_restore.lo
> rrd_restore.c: In function ??rrd_restore??:
> rrd_restore.c:1260: warning: implicit declaration of function ??setlocale??
> rrd_restore.c:1260: warning: nested extern declaration of ??setlocale??
> rrd_restore.c:1260: error: ??LC_NUMERIC?? undeclared (first use in this function)
> rrd_restore.c:1260: error: (Each undeclared identifier is reported only once
> rrd_restore.c:1260: error: for each function it appears in.)
> rrd_restore.c:1260: warning: assignment makes pointer from integer without a cast
> make[2]: *** [rrd_restore.lo] Error 1
> make[2]: Leaving directory `/tmp/rrd/rrdtool-1.3.99909102700/src'
>
> This is on a current debian lenny. Probably I should wait for the next
> snapshot.
>
> But anyway, thanks for your help.
> Greetings
> Hermann
>
>

Re: No ordering information in info hash

2009-10-27T13:43:40Z

Hi Hermann,

there was an extension recently added a ds[x].index entry which
will help you resolve this problem ...

make sure you use a current snapshot of the archive ...

cheers
tobi

Today Hermann Lauer wrote:

> Hello all,
>
> I noticed that in the python bindings of rrdtool-1.3.99909060808
> the rrdtool.info() function the datasources in the flat hash dict returned
> are partly indexed with the datasource name and partly indexed with numbers
> (see example below). Besides beeing somehow inconsistent a problem arises:
>
> How to find out the ordering of datasources to
> find the datasource name for the rra[].cdp_prep[] values (or for
> using rrdtool.update('N:...:...:...')) without knowing the order
> of the datasources.
>
> How is this handled in other bindings (e.g. perl) ?
>
> Please tell me if I missed something obvious,
> greetings
> Hermann
>
>
> >>> import rrdtool
> >>> rrdtool.__version__
> '$Revision: 1.14 $'
> >>> rrdtool.create('/tmp/test.rrd','--step','20','DS:ds2:GAUGE:300:0:100','DS:x1:GAUGE:300:0:100','DS:ds0:GAUGE:300:-5:100','RRA:AVERAGE:0.5:3:2')
> >>> rrdtool.info('/tmp/test.rrd')
> {'rra[0].cdp_prep[0].value': None, 'ds[ds2].min': 0.0, 'ds[ds0].min': -5.0, 'rra[0].cdp_prep[1].unknown_datapoints': 1L, 'ds[ds2].value': 0.0, 'ds[ds2].last_ds': 'U', 'ds[x1].max': 100.0, 'ds[ds2].unknown_sec': 5L, 'ds[ds0].value': 0.0, 'ds[ds0].unknown_sec': 5L, 'ds[ds0].max': 100.0, 'ds[ds2].max': 100.0, 'rrd_version': '0003', 'filename': '/tmp/test.rrd', 'last_update': 1256646805L, 'rra[0].cf': 'AVERAGE', 'ds[x1].type': 'GAUGE', 'rra[0].pdp_per_row': 3L, 'rra[0].cur_row': 1L, 'header_size': 1172L, 'ds[ds0].last_ds': 'U', 'step': 20L, 'rra[0].cdp_prep[0].unknown_datapoints': 1L, 'ds[ds0].type': 'GAUGE', 'rra[0].cdp_prep[2].value': None, 'ds[x1].last_ds': 'U', 'ds[ds2].minimal_heartbeat': 300L, 'rra[0].rows': 2L, 'ds[x1].min': 0.0, 'ds[ds0].minimal_heartbeat': 300L, 'ds[x1].unknown_sec': 5L, 'rra[0].xff': 0.5, 'rra[0].cdp_prep[2].unknown_datapoints': 1L, 'ds[ds2].type': 'GAUGE', 'rra[0].cdp_prep[1].value': None, 'ds[x1].value': 0.0, 'ds[x1].minimal_heartbeat': 300L}
>
>

Re: No ordering information in info hash

2009-10-27T08:47:08Z

Hermann Lauer wrote:

> Hello all,
>
> I noticed that in the python bindings of rrdtool-1.3.99909060808
> the rrdtool.info() function the datasources in the flat hash dict returned
> are partly indexed with the datasource name and partly indexed with numbers
> (see example below). Besides beeing somehow inconsistent a problem arises:
>
> How to find out the ordering of datasources to
> find the datasource name for the rra[].cdp_prep[] values (or for
> using rrdtool.update('N:...:...:...')) without knowing the order
> of the datasources.
>
> How is this handled in other bindings (e.g. perl) ?
>
> Please tell me if I missed something obvious,
> greetings
> Hermann
>
>
>>>> import rrdtool
>>>> rrdtool.__version__
> '$Revision: 1.14 $'
>>>> rrdtool.create('/tmp/test.rrd','--step','20','DS:ds2:GAUGE:300:0:100','DS:x1:GAUGE:300:0:100','DS:ds0:GAUGE:300:-5:100','RRA:AVERAGE:0.5:3:2')
>>>> rrdtool.info('/tmp/test.rrd')
> {'rra[0].cdp_prep[0].value': None, 'ds[ds2].min': 0.0, 'ds[ds0].min': -5.0, 'rra[0].cdp_prep[1].unknown_datapoints': 1L, 'ds[ds2].value': 0.0, 'ds[ds2].last_ds': 'U', 'ds[x1].max': 100.0, 'ds[ds2].unknown_sec': 5L, 'ds[ds0].value': 0.0, 'ds[ds0].unknown_sec': 5L, 'ds[ds0].max': 100.0, 'ds[ds2].max': 100.0, 'rrd_version': '0003', 'filename': '/tmp/test.rrd', 'last_update': 1256646805L, 'rra[0].cf': 'AVERAGE', 'ds[x1].type': 'GAUGE', 'rra[0].pdp_per_row': 3L, 'rra[0].cur_row': 1L, 'header_size': 1172L, 'ds[ds0].last_ds': 'U', 'step': 20L, 'rra[0].cdp_prep[0].unknown_datapoints': 1L, 'ds[ds0].type': 'GAUGE', 'rra[0].cdp_prep[2].value': None, 'ds[x1].last_ds': 'U', 'ds[ds2].minimal_heartbeat': 300L, 'rra[0].rows': 2L, 'ds[x1].min': 0.0, 'ds[ds0].minimal_heartbeat': 300L, 'ds[x1].unknown_sec': 5L, 'rra[0].xff': 0.5, 'rra[0].cdp_prep[2].unknown_datapoints': 1L, 'ds[ds2].type': 'GAUGE', 'rra[0].cdp_prep[1].value': None, 'ds[x1].value': 0.0, 'ds[x1].minimal_heartbeat': 300L}
>

Hey Herman,

It's important to note that both the keys and the values in that data
tructure are strings -- don't let the form trick you :-) As such, you
can simply sort them like this:
sorted(d.items())

And that will give you a list of (key, value) tuples that you can
iterate through and check. Since these are string values, you have to do
stringy things...

PyRRD does provide non-string keys in its data structures, so you have
more flexibility. If you want to email the PyRRD list, we can talk about
the type of data access you'd like to have... adding features like this
to PyRRD is probably much easier than doing so to the RRDTool Python
bindings.

Thanks,

d

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

No ordering information in info hash

2009-10-27T06:14:09Z

Hello all,

I noticed that in the python bindings of rrdtool-1.3.99909060808
the rrdtool.info() function the datasources in the flat hash dict returned
are partly indexed with the datasource name and partly indexed with numbers
(see example below). Besides beeing somehow inconsistent a problem arises:

How to find out the ordering of datasources to
find the datasource name for the rra[].cdp_prep[] values (or for
using rrdtool.update('N:...:...:...')) without knowing the order
of the datasources.

How is this handled in other bindings (e.g. perl) ?

Please tell me if I missed something obvious,
greetings
Hermann

>>> import rrdtool
>>> rrdtool.__version__
'$Revision: 1.14 $'
>>> rrdtool.create('/tmp/test.rrd','--step','20','DS:ds2:GAUGE:300:0:100','DS:x1:GAUGE:300:0:100','DS:ds0:GAUGE:300:-5:100','RRA:AVERAGE:0.5:3:2')
>>> rrdtool.info('/tmp/test.rrd')
{'rra[0].cdp_prep[0].value': None, 'ds[ds2].min': 0.0, 'ds[ds0].min': -5.0, 'rra[0].cdp_prep[1].unknown_datapoints': 1L, 'ds[ds2].value': 0.0, 'ds[ds2].last_ds': 'U', 'ds[x1].max': 100.0, 'ds[ds2].unknown_sec': 5L, 'ds[ds0].value': 0.0, 'ds[ds0].unknown_sec': 5L, 'ds[ds0].max': 100.0, 'ds[ds2].max': 100.0, 'rrd_version': '0003', 'filename': '/tmp/test.rrd', 'last_update': 1256646805L, 'rra[0].cf': 'AVERAGE', 'ds[x1].type': 'GAUGE', 'rra[0].pdp_per_row': 3L, 'rra[0].cur_row': 1L, 'header_size': 1172L, 'ds[ds0].last_ds': 'U', 'step': 20L, 'rra[0].cdp_prep[0].unknown_datapoints': 1L, 'ds[ds0].type': 'GAUGE', 'rra[0].cdp_prep[2].value': None, 'ds[x1].last_ds': 'U', 'ds[ds2].minimal_heartbeat': 300L, 'rra[0].rows': 2L, 'ds[x1].min': 0.0, 'ds[ds0].minimal_heartbeat': 300L, 'ds[x1].unknown_sec': 5L, 'rra[0].xff': 0.5, 'rra[0].cdp_prep[2].unknown_datapoints': 1L, 'ds[ds2].type': 'GAUGE', 'rra[0].cdp_prep[1].value': None, 'ds[x1].value': 0.0, 'ds[x1].minimal_heartbeat': 300L}

--
Netzwerkadministration/Zentrale Dienste, Interdiziplinaeres
Zentrum fuer wissenschaftliches Rechnen der Universitaet Heidelberg
IWR; INF 368; 69120 Heidelberg; Tel: (06221)54-8236 Fax: -5224
Email: Hermann.Lauer@...

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: rrdtool 1.3.8: RRDs::update do no update mtime RHEL 5.1 (Tikanga) ??

2009-10-27T04:20:01Z

On Tue, Oct 27, 2009 at 12:04:29PM +0100, Ole Bj�rn Hessen wrote:
>
>Hello,
>
>I've compiled rrdtool 1.3.8 on an Linux RHEL 5.1 (Tikanga) kernel
>version 2.6.18-92.1.18.el5 out of the box using only the --prefix
>option.

>There are no fancy mount options - just a simple ext3 rw.
>
> batch-rrd4(root) ~ 724# mount | grep batch-rrd4
> /dev/mapper/vg00-rrd on /export/batch-rrd4 type ext3 (rw)
> /export/batch-rrd4 on /local/net/mnt/batch-rrd4 type none (rw,bind)
>
>Is this a simple linux bug on an RHEL 5.1 "old kernel" ?

This is a known bug on (old) redhat kernels, but configure should have
detected this bug and enabled a workaround. See (just a guess):
$ grep -A5 -B5 mmap config.log

It should show a probe for buggy mmap behaviour not updating mtime..

PS: IIRC 1.3.9 was released recently and contains some bugfixes, so i
recommend you use that if you compile it anyway.

HTH,

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

rrdtool 1.3.8: RRDs::update do no update mtime RHEL 5.1 (Tikanga) ??

2009-10-27T04:04:29Z

Hello,

I've compiled rrdtool 1.3.8 on an Linux RHEL 5.1 (Tikanga) kernel
version 2.6.18-92.1.18.el5 out of the box using only the --prefix
option.

|rrdtool config.status 1.3.8
|configured by ./configure, generated by GNU Autoconf 2.63,
| with options \"'--prefix=/local/net/arch/linux_x86_64/encap/rrdtool-1.3.8'\"

Unfortunately, it seems like mtime does not get updated.

ls(1) reports that files was last updated on "Oct 16 09:30" but strace
shows that the files is beeing updated. Also rrdtool fetch shows that
the content has been updated. The problem seems to be that mtime is not
updated.

An example file /export/batch-rrd4/interface/54/14654.rrd

ls -slt /export/batch-rrd4/interface/54/14654.rrd
6376 -rw-rw-r-- 1 tiadm tiadm 6515984 Oct 16 09:30 /export/batch-rrd4/interface/54/14654.rrd
strace -tt:

11:45:09.732496 stat("/export/batch-rrd4/interface/54/14654.rrd", {st_mode=S_IFREG|0664, st_size=6515984, ...}) = 0
11:45:09.732586 open("/export/batch-rrd4/interface/54/14654.rrd", O_RDWR) = 7
11:45:09.732637 fstat(7, {st_mode=S_IFREG|0664, st_size=6515984, ...}) = 0
11:45:09.732697 fadvise64(7, 0, 0, POSIX_FADV_RANDOM) = 0
11:45:09.732737 mmap(NULL, 6515984, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0) = 0x2aaab05a4000
11:45:09.732780 madvise(0x2aaab05a4000, 6515984, 0x1 /* MADV_??? */) = 0
11:45:09.732819 madvise(0x2aaab05a4000, 128, MADV_SEQUENTIAL|0x1) = 0
11:45:09.732862 madvise(0x2aaab05a4000, 7680, MADV_SEQUENTIAL|0x1) = 0
11:45:09.732900 madvise(0x2aaab05a5000, 600, MADV_SEQUENTIAL|0x1) = 0
11:45:09.732938 madvise(0x2aaab05a6000, 16, MADV_SEQUENTIAL|0x1) = 0
11:45:09.732982 fcntl(7, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
11:45:09.733136 msync(0x2aaab05a4000, 6515984, MS_ASYNC) = 0
11:45:09.733174 munmap(0x2aaab05a4000, 6515984) = 0
11:45:09.733219 close(7) = 0

There are no fancy mount options - just a simple ext3 rw.

batch-rrd4(root) ~ 724# mount | grep batch-rrd4
/dev/mapper/vg00-rrd on /export/batch-rrd4 type ext3 (rw)
/export/batch-rrd4 on /local/net/mnt/batch-rrd4 type none (rw,bind)

Is this a simple linux bug on an RHEL 5.1 "old kernel" ?

Ole Bjørn Hessen,
NMS-IP
Telenor

_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers