Planet file space requirements

---

The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

Hi,

Linden Varley wrote:
> I was just wondering what the space requirements were for loading the whole planet file. I have about ~50GB free and am running out of space when running osm2pgsql in slim mode.

Yes, you will need about 70 GB.

> Also does slim mode create extra tables which don't get created in normal mode? i.e planet_osm_nodes , planet_osm_ways, planet_osm_rels

Yes. You can save a lot of disk space and import time if you work
without --slim, however this requires at least 12 (?) GB of memory and
makes it impossible to do differential updates.

Bye
Frederik

--
Frederik Ramm  ##  eMail frederik@...  ##  N49°00'09" E008°23'33"

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

Is there potential to be able to do differential updates on a non-slim imported db?

Reason I ask is I have plenty of memory, but little DB space (SSD) and I would like to do weekly diffs, import with osm2pgsql and get a tile expiry list to re-cache changed tiles.

If not, I'll import with slim on a larger disk and do a filesystem copy across to the SSD.

- Linden

-----Original Message-----
From: Frederik Ramm [mailto:frederik@...]
Sent: Tuesday, 27 October 2009 6:48 PM
To: Linden Varley
Cc: dev@...
Subject: Re: [OSM-dev] Planet file space requirements

Hi,

Linden Varley wrote:
> I was just wondering what the space requirements were for loading the whole planet file. I have about ~50GB free and am running out of space when running osm2pgsql in slim mode.

Yes, you will need about 70 GB.

> Also does slim mode create extra tables which don't get created in normal mode? i.e planet_osm_nodes , planet_osm_ways, planet_osm_rels

Yes. You can save a lot of disk space and import time if you work
without --slim, however this requires at least 12 (?) GB of memory and
makes it impossible to do differential updates.

Bye
Frederik

--
Frederik Ramm  ##  eMail frederik@...  ##  N49°00'09" E008°23'33"

The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On Fri, 30 Oct 2009, Linden Varley wrote:

> Is there potential to be able to do differential updates on a non-slim imported db?
>
> Reason I ask is I have plenty of memory, but little DB space (SSD) and I would like to do weekly diffs, import with osm2pgsql and get a tile expiry list to re-cache changed tiles.
>
> If not, I'll import with slim on a larger disk and do a filesystem copy across to the SSD.

I rather suggest to ditch the SSD, the chances that it will be killed (in
performance) are rather high with non-static datasets as OSM.


Stefan


_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On 29 Oct 2009, at 22:45, Linden Varley wrote:

> Is there potential to be able to do differential updates on a non-
> slim imported db?

No you need slim mode to do diff updates.


Shaun

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On Thu, 2009-10-29 at 23:40 +0000, Shaun McDonald wrote:
> On 29 Oct 2009, at 22:45, Linden Varley wrote:
>
> > Is there potential to be able to do differential updates on a non-
> > slim imported db?
>
> No you need slim mode to do diff updates.
>

There is an alternative - you could use osmosis to regenerate the
complete planet.osm file each week from the diffs. Then this could be
imported without slim mode into osm2pgsql. I don't know how well this
would work in practice.

        Jon



_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

I disagree. Weekly writes to the SSD will not be that bad and the performance benefits far outweighs the disadvantages. We can do a full planet file load in 4 hours and database reads are outstanding to the point where it's not even necessary to pre-cache tiles.

- Linden

-----Original Message-----
From: Stefan de Konink [mailto:stefan@...]
Sent: Friday, 30 October 2009 9:56 AM
To: Linden Varley
Cc: Frederik Ramm; dev@...
Subject: Re: [OSM-dev] Planet file space requirements

On Fri, 30 Oct 2009, Linden Varley wrote:

> Is there potential to be able to do differential updates on a non-slim imported db?
>
> Reason I ask is I have plenty of memory, but little DB space (SSD) and I would like to do weekly diffs, import with osm2pgsql and get a tile expiry list to re-cache changed tiles.
>
> If not, I'll import with slim on a larger disk and do a filesystem copy across to the SSD.

I rather suggest to ditch the SSD, the chances that it will be killed (in
performance) are rather high with non-static datasets as OSM.


Stefan


The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

Thanks Jon that sounds like a good idea although I don't think I would be able to generate a tile expiry list using that method? Might have to manually do it by parsing the diff to get tiles.

-Linden

-----Original Message-----
From: Jon Burgess [mailto:jburgess777@...]
Sent: Friday, 30 October 2009 11:09 AM
To: Linden Varley
Cc: Shaun McDonald; dev@...
Subject: Re: [OSM-dev] Planet file space requirements

On Thu, 2009-10-29 at 23:40 +0000, Shaun McDonald wrote:
> On 29 Oct 2009, at 22:45, Linden Varley wrote:
>
> > Is there potential to be able to do differential updates on a non-
> > slim imported db?
>
> No you need slim mode to do diff updates.
>

There is an alternative - you could use osmosis to regenerate the
complete planet.osm file each week from the diffs. Then this could be
imported without slim mode into osm2pgsql. I don't know how well this
would work in practice.

        Jon



The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On Fri, 30 Oct 2009, Linden Varley wrote:

> I disagree. Weekly writes to the SSD will not be that bad and the
> performance benefits far outweighs the disadvantages. We can do a full
> planet file load in 4 hours and database reads are outstanding to the
> point where it's not even necessary to pre-cache tiles.

Weekly -> Minutely

The performance degrades will be visible :) but if you want to spend your
hardware and software money in a bad way please help yourself ;)


Stefan


_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

Hi,

Linden Varley wrote:
> Is there potential to be able to do differential updates on a
> non-slim imported db? Reason I ask is I have plenty of memory, but
> little DB space (SSD) and I would like to do weekly diffs, import
> with osm2pgsql and get a tile expiry list to re-cache changed tiles.

If I were you then I would not bother with diffs - you say you can do a
full non-slim import in four hours, and my guess is that a weekly diff
in slim mode will not be much faster.

Simply download the full planet file each week and do a full import. You
say that your rendering is quite fast, so I would not bother doing
traditional tile expiry either - just regularly run a "find" on the tile
cache that deletes everything older than X days. If you do want to
retain cached images then write a script to go over the planet file,
check the last-modified date of every node, and expire affected tiles only.

I would be interested to hear more performance data of your setup
(make/model of SSD, time to import planet, time to apply a daily diff,
some rendering figures).

Bye
Frederik

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On Fri, Oct 30, 2009 at 12:08 AM, Jon Burgess
<jburgess777@...> wrote:
It works fine, that's what we do with the cycle map. Once a week we
build a diff from the daily diffs and apply it to a planet to make a
new planet, then import that.

Cheers,
Andy

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On Fri, Oct 30, 2009 at 8:44 AM, Frederik Ramm <frederik@...> wrote:

> Simply download the full planet file each week and do a full import.

I'd advise against that - download the diffs, and use them to "patch"
your locally held planet file, then do a full import. The (daily)
diffs come out earlier than the planet, and save a significant amount
of time (and bandwidth) over downloading the whole thing each week.

Cheers,
Andy

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

Hi,

Andy Allan wrote:
> I'd advise against that - download the diffs, and use them to "patch"
> your locally held planet file, then do a full import. The (daily)
> diffs come out earlier than the planet, and save a significant amount
> of time (and bandwidth) over downloading the whole thing each week.

But the daily diffs suffer from potentially missing data (which has only
been solved by the minute-replication diffs). Only downloading the full
planet will make sure that you're not accumulating "holes" in your data
set.

And if you are content with weekly updates, then waiting one more day
for the planet file to be ready doesn't really make a difference.

Bye
Frederik

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

On 30 Oct 2009, at 7:43 , Frederik Ramm wrote:

> Hi,
>
> Andy Allan wrote:
>> I'd advise against that - download the diffs, and use them to "patch"
>> your locally held planet file, then do a full import. The (daily)
>> diffs come out earlier than the planet, and save a significant amount
>> of time (and bandwidth) over downloading the whole thing each week.
>

for a short time there was a torrent, adding one on the main server  
could save most of the bandwith and improve download speed for all



_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

Stefan de Konink schrieb:
Just go on an test is with the SSDs. If it works, tell us. If it
doesn't, tell us too. Anyway, we'll learn sth.

Peter

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev