convert rrd into new rrd with different RRAs

[Karl Fischer-3]

Hi,

I'm re-posting this since I didn't get a single reply - any ideas?

I'm trying to convert many existing rrds into new rrds with different RRAs.

so for example I might have rrd's created with:
(RRAs: 1s, 10s, 60s, 600s)

  --step=1
  DS:somevalue:GAUGE:120:0:U
  RRA:AVERAGE:0.5:1:3600    # 1s for 1h
  RRA:AVERAGE:0.5:10:8640   # 10s for 1d
  RRA:AVERAGE:0.5:60:10080  # 60s for 1w
  RRA:AVERAGE:0.5:600:4320  # 10m for 1m

and I'd like to convert this into a new rdd that looks like this:
(RRAs: 2s, 10s, 60s, 900s)

  --step=2
  DS:somevalue:GAUGE:60:0:U
  RRA:AVERAGE:0.5:1:5400    # 2s for 3h
  RRA:AVERAGE:0.5:10:8640   # 10s for 1d
  RRA:AVERAGE:0.5:30:20160  # 60s for 2w
  RRA:AVERAGE:0.5:300:12960 # 10m for 3m

Of course that means that some data needs to be calculated from higher
RRAs, for example:

the first half of the 2s RRA in the new database can be taken 1:1 from the
1s RRA of the exiting database by using the CF on two datapoints each time.
The second half is more tricky since there is no data in 1s resolution in
the first RRD, so the values need to be interpolated from the 10s RRA ...

I wonder if I have to re-invent the wheel and to write the required code
completely myself or if someone has been there already ... ?

Any hints welcome ...

- Karl

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Alex van den Bogaerdt-2]

> I wonder if I have to re-invent the wheel and to write the required code
> completely myself or if someone has been there already ... ?
>
> Any hints welcome ...


Do you expect everybody to answer "no, I have not"  ?

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Karl Fischer-3]

Alex van den Bogaerdt wrote:
>> I wonder if I have to re-invent the wheel and to write the required code
>> completely myself or if someone has been there already ... ?
>>
>> Any hints welcome ...
>
> Do you expect everybody to answer "no, I have not"  ?

;-) well, hope dies last ...
I was hoping that /the one person who might have done it before/ didn't
see the post.
Sorry for bothering ...

- Karl


_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Alex van den Bogaerdt-2]

----- Original Message -----
From: "Karl Fischer" <rrd-users@...>
To: "Alex van den Bogaerdt" <alex@...>
Cc: "rrd-users" <rrd-users@...>
Sent: Monday, March 23, 2009 8:57 PM
Subject: Re: [rrd-users] convert rrd into new rrd with different RRAs

> Alex van den Bogaerdt wrote:
>>> I wonder if I have to re-invent the wheel and to write the required code
>>> completely myself or if someone has been there already ... ?
>>>
>>> Any hints welcome ...
>>
>> Do you expect everybody to answer "no, I have not"  ?
>
> ;-) well, hope dies last ...
> I was hoping that /the one person who might have done it before/ didn't
> see the post.
> Sorry for bothering ...

I'm positive that you can find out how to do it.

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Stefan M. Brandl]

On Mon, Mar 23, 2009 at 07:32:07PM +0100, Karl Fischer wrote:
> Hi,
>

Hello Karl,

> I'm re-posting this since I didn't get a single reply - any ideas?
>
> I'm trying to convert many existing rrds into new rrds with different RRAs.
>
> so for example I might have rrd's created with:
> (RRAs: 1s, 10s, 60s, 600s)
>
>   --step=1
>   DS:somevalue:GAUGE:120:0:U
>   RRA:AVERAGE:0.5:1:3600    # 1s for 1h
>   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
>   RRA:AVERAGE:0.5:60:10080  # 60s for 1w
>   RRA:AVERAGE:0.5:600:4320  # 10m for 1m
>
> and I'd like to convert this into a new rdd that looks like this:
> (RRAs: 2s, 10s, 60s, 900s)
>
>   --step=2
>   DS:somevalue:GAUGE:60:0:U
>   RRA:AVERAGE:0.5:1:5400    # 2s for 3h
>   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
>   RRA:AVERAGE:0.5:30:20160  # 60s for 2w
>   RRA:AVERAGE:0.5:300:12960 # 10m for 3m
>
> Of course that means that some data needs to be calculated from higher
> RRAs, for example:
>
> the first half of the 2s RRA in the new database can be taken 1:1 from the
> 1s RRA of the exiting database by using the CF on two datapoints each time.
> The second half is more tricky since there is no data in 1s resolution in
> the first RRD, so the values need to be interpolated from the 10s RRA ...
>
> I wonder if I have to re-invent the wheel and to write the required code
> completely myself or if someone has been there already ... ?
>
> Any hints welcome ...
>

As I did not see any hints to the mailing list, I wanted to
ask if there are any news on this topic?


Stefan

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Tobias Oetiker-3]

Hi Stefan,

there is no such program to that does this as fahr as I can tell
... the simples way to achieve this is to write an app that pulls
the data from the original rrd, sorting the updates by time picking
the higest resolution available for each instant and then feeding
it back to a newly setup rrd ... it will get trickier if you want
to support MAX RRAs as well, but you get the idea ?

cheers
tobi

Today Stefan M. Brandl wrote:

> On Mon, Mar 23, 2009 at 07:32:07PM +0100, Karl Fischer wrote:
> > Hi,
> >
>
> Hello Karl,
>
> > I'm re-posting this since I didn't get a single reply - any ideas?
> >
> > I'm trying to convert many existing rrds into new rrds with different RRAs.
> >
> > so for example I might have rrd's created with:
> > (RRAs: 1s, 10s, 60s, 600s)
> >
> >   --step=1
> >   DS:somevalue:GAUGE:120:0:U
> >   RRA:AVERAGE:0.5:1:3600    # 1s for 1h
> >   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
> >   RRA:AVERAGE:0.5:60:10080  # 60s for 1w
> >   RRA:AVERAGE:0.5:600:4320  # 10m for 1m
> >
> > and I'd like to convert this into a new rdd that looks like this:
> > (RRAs: 2s, 10s, 60s, 900s)
> >
> >   --step=2
> >   DS:somevalue:GAUGE:60:0:U
> >   RRA:AVERAGE:0.5:1:5400    # 2s for 3h
> >   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
> >   RRA:AVERAGE:0.5:30:20160  # 60s for 2w
> >   RRA:AVERAGE:0.5:300:12960 # 10m for 3m
> >
> > Of course that means that some data needs to be calculated from higher
> > RRAs, for example:
> >
> > the first half of the 2s RRA in the new database can be taken 1:1 from the
> > 1s RRA of the exiting database by using the CF on two datapoints each time.
> > The second half is more tricky since there is no data in 1s resolution in
> > the first RRD, so the values need to be interpolated from the 10s RRA ...
> >
> > I wonder if I have to re-invent the wheel and to write the required code
> > completely myself or if someone has been there already ... ?
> >
> > Any hints welcome ...
> >
>
> As I did not see any hints to the mailing list, I wanted to
> ask if there are any news on this topic?
>
>
> Stefan
>
> _______________________________________________
> rrd-users mailing list
> rrd-users@...
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
>
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Karl Fischer-3]

Tobias Oetiker schrieb:

> Hi Stefan,
>
> there is no such program to that does this as fahr as I can tell
> ... the simples way to achieve this is to write an app that pulls
> the data from the original rrd, sorting the updates by time picking
> the higest resolution available for each instant and then feeding
> it back to a newly setup rrd ... it will get trickier if you want
> to support MAX RRAs as well, but you get the idea ?
>
> cheers
> tobi

I've tried this aproach before, but with large rrds this takes forever,
because it always feeds the finest resolution first and then runs through
all aggregation steps. I have rrds with values for years, so they're really
large ...

Now lets say I'm sorting the updates by time picking the higest resolution
available, that highest resolution might be 10m (after 2 weeks in that given
example), but I'm feeding those 10m values into the new rrd which has a 2s
step size, so it has to go through all the aggregation steps for all DPs.

A smarter solution might be: create multiple new rrds, one for each wanted
rra-size, and then feed each of the new rrds with the resolution suitable for
it's rra-size, eg. not the highest, but the lowest that is still higher than
the new wanted.
After having done that, combine the multiple rrds into one rrd with multiple
rras. That is much faster, but much more complicated :-(

It get's even more trickier if you need to keep track on 'gaps' in the data,
because the system was offline for a while. The gap might be large enough to
trigger the heartbeat/xff for the first rra, but not for the others ...
That makes it even more difficult ...

I have postponed this project to some long dark cold winter nights ... :-/

- Karl






 

> Today Stefan M. Brandl wrote:
>
>> On Mon, Mar 23, 2009 at 07:32:07PM +0100, Karl Fischer wrote:
>>> Hi,
>>>
>> Hello Karl,
>>
>>> I'm re-posting this since I didn't get a single reply - any ideas?
>>>
>>> I'm trying to convert many existing rrds into new rrds with different RRAs.
>>>
>>> so for example I might have rrd's created with:
>>> (RRAs: 1s, 10s, 60s, 600s)
>>>
>>>   --step=1
>>>   DS:somevalue:GAUGE:120:0:U
>>>   RRA:AVERAGE:0.5:1:3600    # 1s for 1h
>>>   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
>>>   RRA:AVERAGE:0.5:60:10080  # 60s for 1w
>>>   RRA:AVERAGE:0.5:600:4320  # 10m for 1m
>>>
>>> and I'd like to convert this into a new rdd that looks like this:
>>> (RRAs: 2s, 10s, 60s, 900s)
>>>
>>>   --step=2
>>>   DS:somevalue:GAUGE:60:0:U
>>>   RRA:AVERAGE:0.5:1:5400    # 2s for 3h
>>>   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
>>>   RRA:AVERAGE:0.5:30:20160  # 60s for 2w
>>>   RRA:AVERAGE:0.5:300:12960 # 10m for 3m
>>>
>>> Of course that means that some data needs to be calculated from higher
>>> RRAs, for example:
>>>
>>> the first half of the 2s RRA in the new database can be taken 1:1 from the
>>> 1s RRA of the exiting database by using the CF on two datapoints each time.
>>> The second half is more tricky since there is no data in 1s resolution in
>>> the first RRD, so the values need to be interpolated from the 10s RRA ...
>>>
>>> I wonder if I have to re-invent the wheel and to write the required code
>>> completely myself or if someone has been there already ... ?
>>>
>>> Any hints welcome ...
>>>
>> As I did not see any hints to the mailing list, I wanted to
>> ask if there are any news on this topic?
>>
>>
>> Stefan

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Tobias Oetiker-3]

Today rrd-users@... wrote:

> Tobias Oetiker schrieb:
> > Hi Stefan,
> >
> > there is no such program to that does this as fahr as I can tell
> > ... the simples way to achieve this is to write an app that pulls
> > the data from the original rrd, sorting the updates by time picking
> > the higest resolution available for each instant and then feeding
> > it back to a newly setup rrd ... it will get trickier if you want
> > to support MAX RRAs as well, but you get the idea ?
> >
> > cheers
> > tobi
>
> I've tried this aproach before, but with large rrds this takes forever,
> because it always feeds the finest resolution first and then runs through
> all aggregation steps. I have rrds with values for years, so they're really
> large ...
>
> Now lets say I'm sorting the updates by time picking the higest resolution
> available, that highest resolution might be 10m (after 2 weeks in that given
> example), but I'm feeding those 10m values into the new rrd which has a 2s
> step size, so it has to go through all the aggregation steps for all DPs.

have you actually tried it ? it should not take all that long since
it will all happen in memory ...

> A smarter solution might be: create multiple new rrds, one for each wanted
> rra-size, and then feed each of the new rrds with the resolution suitable for
> it's rra-size, eg. not the highest, but the lowest that is still higher than
> the new wanted.
> After having done that, combine the multiple rrds into one rrd with multiple
> rras. That is much faster, but much more complicated :-(
>
> It get's even more trickier if you need to keep track on 'gaps' in the data,
> because the system was offline for a while. The gap might be large enough to
> trigger the heartbeat/xff for the first rra, but not for the others ...
> That makes it even more difficult ...
>
> I have postponed this project to some long dark cold winter nights ... :-/

yes ... in that case it might be best to resample from the
available data directly into the target RRAs maybe into the xml
format and then restore this ... I have written such code for mrtg
when it converts the old mrtg.log files to rrdtool.

cheers
tobi

>
> - Karl
>
>
>
>
>
>
>
> > Today Stefan M. Brandl wrote:
> >
> > > On Mon, Mar 23, 2009 at 07:32:07PM +0100, Karl Fischer wrote:
> > > > Hi,
> > > >
> > > Hello Karl,
> > >
> > > > I'm re-posting this since I didn't get a single reply - any ideas?
> > > >
> > > > I'm trying to convert many existing rrds into new rrds with different
> > > > RRAs.
> > > >
> > > > so for example I might have rrd's created with:
> > > > (RRAs: 1s, 10s, 60s, 600s)
> > > >
> > > >   --step=1
> > > >   DS:somevalue:GAUGE:120:0:U
> > > >   RRA:AVERAGE:0.5:1:3600    # 1s for 1h
> > > >   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
> > > >   RRA:AVERAGE:0.5:60:10080  # 60s for 1w
> > > >   RRA:AVERAGE:0.5:600:4320  # 10m for 1m
> > > >
> > > > and I'd like to convert this into a new rdd that looks like this:
> > > > (RRAs: 2s, 10s, 60s, 900s)
> > > >
> > > >   --step=2
> > > >   DS:somevalue:GAUGE:60:0:U
> > > >   RRA:AVERAGE:0.5:1:5400    # 2s for 3h
> > > >   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
> > > >   RRA:AVERAGE:0.5:30:20160  # 60s for 2w
> > > >   RRA:AVERAGE:0.5:300:12960 # 10m for 3m
> > > >
> > > > Of course that means that some data needs to be calculated from higher
> > > > RRAs, for example:
> > > >
> > > > the first half of the 2s RRA in the new database can be taken 1:1 from
> > > > the
> > > > 1s RRA of the exiting database by using the CF on two datapoints each
> > > > time.
> > > > The second half is more tricky since there is no data in 1s resolution
> > > > in
> > > > the first RRD, so the values need to be interpolated from the 10s RRA
> > > > ...
> > > >
> > > > I wonder if I have to re-invent the wheel and to write the required code
> > > > completely myself or if someone has been there already ... ?
> > > >
> > > > Any hints welcome ...
> > > >
> > > As I did not see any hints to the mailing list, I wanted to
> > > ask if there are any news on this topic?
> > >
> > >
> > > Stefan
>
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@... ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-users mailing list
rrd-users@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users