Live Database Copy of Planet File

[Peter Childs-5]

Does anyone know where I can find some "simple" instructions on how to create a live copy of the planet file.

Ideally I want to feed the diff and the plate file in and have a database that is admittedly read only but is as up to date as possible.

I'm happy if its just a flow chart as I will probably want to pre-process the data for my self as it goes in.

I've this feeling this should be doable with a mix of osmosis and cron and I'm sure I could work it out for my self, but no point in repeating the task if there are already instructions to do this.

Ideally I want to use this for Geocoding, so I would probably need to update my indexes as I add the diff files, and use a customized parser.

May also need some way so I can have a live copy of just part of the planet say just Uk or Europe to start with....

Peter.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Timo Juhani Lindfors]

Peter Childs <pchilds@...> writes:
> Does anyone know where I can find some "simple" instructions on how to
> create a live copy of the planet file.

I would like to do this too to provide a local mirror.  Last time I
tried to do this I understood that this can't be done since the 0.6
style diffs do not have enough information to create 100% identical
copy. Is this still correct? I think the problem had something to do
with changesets not being atomic.

-Timo

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Peter Childs-5]

2009/11/5 Timo Juhani Lindfors <timo.lindfors@...>

Peter Childs <pchilds@...> writes:
> Does anyone know where I can find some "simple" instructions on how to
> create a live copy of the planet file.

I would like to do this too to provide a local mirror.  Last time I
tried to do this I understood that this can't be done since the 0.6
style diffs do not have enough information to create 100% identical
copy. Is this still correct? I think the problem had something to do
with changesets not being atomic.

-Timo

Hmm I'm guess that would be caused by the use of timestamps and missing half started transactions when the diffs is created...

There should be some way of converting the Postgres WAL into diffs but when ever I suggested it on the postgres boards (and this was a few years ago now) it was always condemned as a silly pointless idea..... My view is that WAL could be used for a whole bunch of forensics and replication to other database formats as well as PITR.... Currently all you can do in postgres is roll your database back and then step forward and play spot the difference. you should be able to turn the WAL into a human readable format....(Enough Postgres Theory)

Peter.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Timo Juhani Lindfors]

Peter Childs <pchilds@...> writes:
> There should be some way of converting the Postgres WAL into diffs but when
> ever I suggested it on the postgres boards (and this was a few years ago

Maybe a silly idea but: How about just logging all HTTP requests that
modify the DB and let mirrors replay those? ;-)

-Timo

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Frederik Ramm]

Hi,

Timo Juhani Lindfors wrote:
> I would like to do this too to provide a local mirror.  Last time I
> tried to do this I understood that this can't be done since the 0.6
> style diffs do not have enough information to create 100% identical
> copy. Is this still correct? I think the problem had something to do
> with changesets not being atomic.

There are now "replication" changesets (along with a special Osmosis
task, --rri, to read and assemble them) which are supposed to capture
any change to the data base.

Bye
Frederik


_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Emilie Laffray]

2009/11/5 Frederik Ramm <frederik@...>

Hi,

Timo Juhani Lindfors wrote:
> I would like to do this too to provide a local mirror.  Last time I
> tried to do this I understood that this can't be done since the 0.6
> style diffs do not have enough information to create 100% identical
> copy. Is this still correct? I think the problem had something to do
> with changesets not being atomic.

There are now "replication" changesets (along with a special Osmosis
task, --rri, to read and assemble them) which are supposed to capture
any change to the data base.

Interesting. Do you have more documentation on this? I looked at the reference page for Osmosis and I couldn't find it.
With those changesets, how do you proceed from a planet dump? Is there any documentation on how to use this.

Emilie Laffray

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Matt Amos-3]

On Thu, Nov 5, 2009 at 12:00 PM, Frederik Ramm <frederik@...> wrote:

> Timo Juhani Lindfors wrote:
>> I would like to do this too to provide a local mirror.  Last time I
>> tried to do this I understood that this can't be done since the 0.6
>> style diffs do not have enough information to create 100% identical
>> copy. Is this still correct? I think the problem had something to do
>> with changesets not being atomic.
>
> There are now "replication" changesets (along with a special Osmosis
> task, --rri, to read and assemble them) which are supposed to capture
> any change to the data base.

this doesn't create a 100% identical copy, as the changeset objects
themselves aren't included in the replication diffs. the problem isn't
that changesets aren't atomic - that just makes it harder - but that
someone needs to write the code in osmosis to track changes to the
changesets table.

if you don't need a 100% accurate mirror, and having an accurate
mirror of nodes, ways and relations (which changeset references, but
not changeset tags) is enough, then use osmosis - it does everything
you want.

cheers,

matt

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Matt Amos-3]

On Thu, Nov 5, 2009 at 12:07 PM, Emilie Laffray
<emilie.laffray@...> wrote:

> 2009/11/5 Frederik Ramm <frederik@...>
>> Timo Juhani Lindfors wrote:
>> > I would like to do this too to provide a local mirror.  Last time I
>> > tried to do this I understood that this can't be done since the 0.6
>> > style diffs do not have enough information to create 100% identical
>> > copy. Is this still correct? I think the problem had something to do
>> > with changesets not being atomic.
>>
>> There are now "replication" changesets (along with a special Osmosis
>> task, --rri, to read and assemble them) which are supposed to capture
>> any change to the data base.
>
> Interesting. Do you have more documentation on this? I looked at the
> reference page for Osmosis and I couldn't find it.
> With those changesets, how do you proceed from a planet dump? Is there any
> documentation on how to use this.

i don't think there is any documentation yet, and i'm sure brett would
be happy if someone could help out with that. :-)

it's been discussed a couple of times on this ML, though:
http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html

cheers,

matt

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Emilie Laffray]

2009/11/5 Matt Amos <zerebubuth@...>

i don't think there is any documentation yet, and i'm sure brett would
be happy if someone could help out with that. :-)

it's been discussed a couple of times on this ML, though:
http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html

Thanks for the link.
I will see if I have the time to do the documentation.

Emilie Laffray

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Peter Childs-5]

2009/11/5 Emilie Laffray <emilie.laffray@...>

2009/11/5 Matt Amos <zerebubuth@...>

i don't think there is any documentation yet, and i'm sure brett would
be happy if someone could help out with that. :-)

it's been discussed a couple of times on this ML, though:
http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html

Thanks for the link.
I will see if I have the time to do the documentation.

Emilie Laffray

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

I don't really care for the changesets, just the data... (and only certain parts of that) I had this theory it could be done with a mix of Cron and Osmosis and was really looking at what to feed what it the right order. 

Once I've got it collecting the right files and putting it into postgres using Osmosis I can then replace Osmosis to do the extra preprocessing I want. All I was looking for was a simple step by step guide or an example. I don't want to reinvent the wheel.

I think we need somthing on the wiki

Peter.

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[sly (sylvain letuffe)]

On jeudi 5 novembre 2009, Peter Childs wrote:
> I think we need somthing on the wiki

http://wiki.openstreetmap.org/wiki/Minutely_Mapnik

You just need to convert osm2pgsql lines to osmosis lines


--
sly
Sylvain Letuffe liste@...
qui suis-je : http://slyserv.dyndns.org




_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Timo Juhani Lindfors]

Peter Childs <pchilds@...> writes:
> want. All I was looking for was a simple step by step guide or an example. I
> don't want to reinvent the wheel.
>
> I think we need somthing on the wiki

Indeed. I'm hoping that with a local mirror I should be able to fetch
a much larger area to JOSM than with the primary db server.

-Timo


_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Timo Juhani Lindfors]

Matt Amos <zerebubuth@...> writes:
> it's been discussed a couple of times on this ML, though:
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html

Any idea what version of osmosis is required? osmosis 0.31.2 in debian
unstable tried to retrieve

http://planet.openstreetmap.org/minute-replicate/000078846

but now the files seem have a different layout completely?

-Timo

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Emilie Laffray]

2009/11/5 Timo Juhani Lindfors <timo.lindfors@...>

Matt Amos <zerebubuth@...> writes:
> it's been discussed a couple of times on this ML, though:
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html

Any idea what version of osmosis is required? osmosis 0.31.2 in debian
unstable tried to retrieve

http://planet.openstreetmap.org/minute-replicate/000078846

but now the files seem have a different layout completely?

Matt mentioned that those files need to be run with the new function rri. They are not your traditional change file from OSM.

Emilie Laffray

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Lars Francke-2]

> I don't really care for the changesets, just the data... (and only certain
> parts of that) I had this theory it could be done with a mix of Cron and
> Osmosis and was really looking at what to feed what it the right order.
> Once I've got it collecting the right files and putting it into postgres
> using Osmosis I can then replace Osmosis to do the extra preprocessing I
> want. All I was looking for was a simple step by step guide or an example. I
> don't want to reinvent the wheel.
> I think we need somthing on the wiki

1) Download a planet.osm-dump and a current Osmosis version (you may
have to build it yourself or I could upload a version somewhere)
2) osmosis --write-pgsql (with all the options you need)
3) osmosis --read-xml <your planet.osm> --write-pgsql-dump <your
database information>
4) osmosis --read-replication-interval-init workingDirectory=<your
working directory here>
5) - Now go to http://planet.openstreetmap.org/minute-replicate/ and
find the state.txt file that has a timestamp a hour or so before the
planet dump started (to be on the safe side) and copy it to your
working directory (and its "data" subdirectory, I can't remember the
correct location but copying it to both locations won't do any harm)
- Change the configuration.txt to point to /minute-replicate (the
default is wrong)
6) Set up a cron job following the example in
http://wiki.openstreetmap.org/wiki/Minutely_Mapnik but change a few
things:
- Replace read-change-interval with read-replication-interval
- Replace write-xml-change with write-pgsql-change
- Delete the osm2pgsql stuff
- As there currently are only minutely diffs you'd have to run this
cron job every minute (there is the "merge-replication-files" task in
Osmosis but I don't know if it is stable yet)
7) Enjoy your local osm database (in case I didn't forget anything)

But if you want to use it with JOSM you'd have to set up the Rails port, too.

Hope that helps.

Cheers,
Lars

Look up the correct options up on
http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Matt Amos-3]

On Thu, Nov 5, 2009 at 1:50 PM, Emilie Laffray <emilie.laffray@...> wrote:

> 2009/11/5 Timo Juhani Lindfors <timo.lindfors@...>
>>
>> Matt Amos <zerebubuth@...> writes:
>> > it's been discussed a couple of times on this ML, though:
>> > http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
>> > http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html
>>
>> Any idea what version of osmosis is required? osmosis 0.31.2 in debian
>> unstable tried to retrieve
>>
>> http://planet.openstreetmap.org/minute-replicate/000078846
>>
>> but now the files seem have a different layout completely?
>>
>
> Matt mentioned that those files need to be run with the new function rri.
> They are not your traditional change file from OSM.

yeah. i'd use the latest trunk version of osmosis. i've been using it
and it's pretty stable.

cheers,

matt

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Timo Juhani Lindfors]

Emilie Laffray <emilie.laffray@...> writes:
> Matt mentioned that those files need to be run with the new function rri.
> They are not your traditional change file from OSM.

Hmm, what should I set as the url in the configuration file?

http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html

just says that "It has a wrong URL for the minute-replicate diffs, so
correct that." but does not exactly tell me what it is.

-Timo


_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Peter Körner-4]

> Indeed. I'm hoping that with a local mirror I should be able to fetch
> a much larger area to JOSM than with the primary db server.
If this is all you want to do you can use osmosis to cut put a piece out
from a planet or one of the geofabrik cuts and open it with josm directly..

Peter

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Lennard-2]

Timo Juhani Lindfors wrote:

> just says that "It has a wrong URL for the minute-replicate diffs, so
> correct that." but does not exactly tell me what it is.

http://planet.openstreetmap.org/minute-replicate/

--
Lennard

_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev

[Brett Henderson]

Matt Amos wrote:

> On Thu, Nov 5, 2009 at 12:07 PM, Emilie Laffray
> <emilie.laffray@...> wrote:
>  
>> 2009/11/5 Frederik Ramm <frederik@...>
>>    
>>> Timo Juhani Lindfors wrote:
>>>      
>>>> I would like to do this too to provide a local mirror.  Last time I
>>>> tried to do this I understood that this can't be done since the 0.6
>>>> style diffs do not have enough information to create 100% identical
>>>> copy. Is this still correct? I think the problem had something to do
>>>> with changesets not being atomic.
>>>>        
>>> There are now "replication" changesets (along with a special Osmosis
>>> task, --rri, to read and assemble them) which are supposed to capture
>>> any change to the data base.
>>>      
>> Interesting. Do you have more documentation on this? I looked at the
>> reference page for Osmosis and I couldn't find it.
>> With those changesets, how do you proceed from a planet dump? Is there any
>> documentation on how to use this.
>>    
>
> i don't think there is any documentation yet, and i'm sure brett would
> be happy if someone could help out with that. :-)
>  

This is true :-)

The minute replication tasks are working and I believe they're
reliable.  There's some additional tasks used to automate their
downloads, and some tasks in the works to merge them into hour/day
changesets.  They still don't contain full changeset information.  They
do contain full history.

I've been away on holiday for five weeks, starting a new project at
work, and now I'm moving to a new house which won't have Internet
connectivity for at least a month.  Suffice to say I won't have much
time to improve on the current state of affairs in the very near future.

Some of the stuff to be done includes:
* The new tasks need to be documented.
* Existing tasks need to be examined to see if the full history diffs
(ie. changesets with potentially multiple changes to a single entity in
the same file) work correctly.
* A new task may need to be created to condense full history diffs into
simpler diffs to work with tasks where full history doesn't make sense
(eg. I suspect merging tasks will have issues).
* All changeset extraction needs to be migrated onto the new servers.
* New jobs and osmosis tasks need to be created to merge minute
replication files into larger files (ie. hour and day).
* The existing "history" extraction process needs to be continued to
provide full history since day 1.

I'll probably get there eventually but any assistance would be appreciated.
> it's been discussed a couple of times on this ML, though:
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html
>  
Brett


_______________________________________________
dev mailing list
dev@...
http://lists.openstreetmap.org/listinfo/dev