Processing time to convert large gpx files to gdb.

[Gary Turner-6]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi all,

 

Firstly a bit of background:

I'm a mapper with an open source gps mapping project here in New Zealand. To update our maps, we download the official government address data as a shapefile. We import that into a postgis database, which, among other things we use to split the addresses into tiles corresponding to our maps. We output the data as a gpx file of POI, which we then convert to gdb using gpsbabel.

 

Since these files are pretty large, processing time is a bit more of an issue than it is for more 'normal' sized POI files. Using my 2.8GHz Core 2 duo machine running XP SP3, the postgis database generated one of the largest gpx files in about 26 minutes. The file has 454,000 points in it, and is about 28MB. The file has simple coordinates, a name (the address) and a comment (the numeric id of the road).

 

Using GPS Mapedit, the gpx file loads up in a bit under a minute, however it's quite a bit faster to load a gdb file, which is also a bit smaller. As we're editing our maps pretty regularly, minimising the load time makes life more pleasant, so we like to use gdb files.

 

However - my problem at the moment is the conversion time using gpsbabel. The conversion for that one file has so far been been running for 3 hours, and still isn't finished. Last time I ran the entire dataset, the database extraction and gpsbabel conversion took more than overnight. I'm just using a simple [paths simplified for clarity] gpsbabel -i gpx -o gdb -f file.gpx -F file.gdb commandline.

 

So - is there anything that I can do to speed up the conversion?

Is it possible that there may be optimisations that can be done in gpsbabel to speed up the conversion?

 

I can supply the gpx file if that would help. It's about 8MB zipped.

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

[Robert Lipe-4]

On Mon, Jan 30, 2012 at 8:30 PM, Gary Turner <g@...> wrote:

Hi all,

 

Firstly a bit of background:

I'm a mapper with an open source gps mapping project here in New Zealand. To update our maps, we download the official government address data as a shapefile. We import that into a postgis database, which, among other things we use to split the addresses into tiles corresponding to our maps. We output the data as a gpx file of POI, which we then convert to gdb using gpsbabel.

 

Since these files are pretty large, processing time is a bit more of an issue than it is for more 'normal' sized POI files. Using my 2.8GHz Core 2 duo machine running XP SP3, the postgis database generated one of the largest gpx files in about 26 minutes. The file has 454,000 points in it, and is about 28MB. The file has simple coordinates, a name (the address) and a comment (the numeric id of the road). 

Reading GPX isn't fast; it's just plain a bulky format.   But without an actual test case that we can use to reproduce to study whether it's the reader, the writer, or our infrastructure between them, we're just left to guess.   But what you're describing should not be an overnight task for any reasonable computer, so we have something that's not scaling well.  (Or we're using more memory than we should be or your computer doesn't have enough memory available or...)   Off the cuff, I'd expect the units to be "minutes" and not "hours".

Please put your actual test data on a server somewhere so that we may study it in a profiler.

As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this workflow?

RJL

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

[Bence Sprok]

Hi Everyone,

A bit OFF:

"As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this workflow?"

Reading huge gpx files (and this is case as I read) from MapSource is nearly impossible. It freezes constantly above a certain size, which is way below the mentioned 28MB. No matter what build we're talking about, the past few years I haven't seen any MS version to do so. Even a 3-4MB gpx could take a few minutes to parse in my expirence.

Cheers,
Bence

ON.

On Tue, Jan 31, 2012 at 6:42 PM, Robert Lipe <robertlipe@...> wrote:

On Mon, Jan 30, 2012 at 8:30 PM, Gary Turner <g@...> wrote:

Hi all,

 

Firstly a bit of background:

I'm a mapper with an open source gps mapping project here in New Zealand. To update our maps, we download the official government address data as a shapefile. We import that into a postgis database, which, among other things we use to split the addresses into tiles corresponding to our maps. We output the data as a gpx file of POI, which we then convert to gdb using gpsbabel.

 

Since these files are pretty large, processing time is a bit more of an issue than it is for more 'normal' sized POI files. Using my 2.8GHz Core 2 duo machine running XP SP3, the postgis database generated one of the largest gpx files in about 26 minutes. The file has 454,000 points in it, and is about 28MB. The file has simple coordinates, a name (the address) and a comment (the numeric id of the road). 

Reading GPX isn't fast; it's just plain a bulky format.   But without an actual test case that we can use to reproduce to study whether it's the reader, the writer, or our infrastructure between them, we're just left to guess.   But what you're describing should not be an overnight task for any reasonable computer, so we have something that's not scaling well.  (Or we're using more memory than we should be or your computer doesn't have enough memory available or...)   Off the cuff, I'd expect the units to be "minutes" and not "hours".

Please put your actual test data on a server somewhere so that we may study it in a profiler.

As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this workflow?

RJL

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

[Robert Lipe-4]

On Tue, Jan 31, 2012 at 12:07 PM, Bence Sprok <sprok.bence@...> wrote:

Hi Everyone,

A bit OFF:

I'm OK with that as it's definitely related to the conversation.

 

"As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this workflow?"

Reading huge gpx files (and this is case as I read) from MapSource is nearly impossible. It freezes constantly above a certain size, which is way below the mentioned 28MB. No matter what build we're talking about, the past few years I haven't seen any MS version to do so. Even a 3-4MB gpx could take a few minutes to parse in my expirence.

Wow, that's pretty bad.   Sure enough, I just slurped a big GPX file into Bascamp, MapSources successor, and it beachballed for several minutes before crashing.

I love it when GPSBabel is used to work around the shortcomings of a $3Bn/year company...in a commercial environment.   It's particularly irritating that GDB is one of our most painful formats.

Please do work with Garmin to get that fixed.  Let us know if the creator/lead developer gets back to you. :-)

RJL

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

[Bence Sprok]

Please do work with Garmin to get that fixed.  Let us know if the creator/lead developer gets back to you. :-)

I've decided years ago not write any more comments/suggestions to Garmin. They don't even reply... But I really don't mind that, I have my own opinion as well. :)

On Tue, Jan 31, 2012 at 7:31 PM, Robert Lipe <robertlipe@...> wrote:

On Tue, Jan 31, 2012 at 12:07 PM, Bence Sprok <sprok.bence@...> wrote:

Hi Everyone,

A bit OFF:

I'm OK with that as it's definitely related to the conversation.

 

"As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this workflow?"

Reading huge gpx files (and this is case as I read) from MapSource is nearly impossible. It freezes constantly above a certain size, which is way below the mentioned 28MB. No matter what build we're talking about, the past few years I haven't seen any MS version to do so. Even a 3-4MB gpx could take a few minutes to parse in my expirence.

Wow, that's pretty bad.   Sure enough, I just slurped a big GPX file into Bascamp, MapSources successor, and it beachballed for several minutes before crashing.

I love it when GPSBabel is used to work around the shortcomings of a $3Bn/year company...in a commercial environment.   It's particularly irritating that GDB is one of our most painful formats.

Please do work with Garmin to get that fixed.  Let us know if the creator/lead developer gets back to you. :-)

RJL

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

[Gary Turner-6]

----- Original Message -----
From: "Robert Lipe" <robertlipe@...>
To: "Gary Turner" <g@...>
Cc: <gpsbabel-misc@...>
Sent: Wednesday, February 01, 2012 6:42 AM
Subject: Re: [Gpsbabel-misc] Processing time to convert large gpx files to
gdb.


> Please put your actual test data on a server somewhere so that we may
> study it in a profiler.

http://gtmaps.co.nz/samples/Auckland-numbers.zip

> As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this
> workflow?

Apart from the fact that Mapsource does choke on even slightly big files?
Scriptability mainly. We have 8 files to process. (Apart from the fact that
the developers here DO listen and care ;^)

Thanks.



------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

[Robert Lipe-4]

On Tue, Jan 31, 2012 at 4:40 PM, Gary Turner <g@...> wrote:

----- Original Message -----
From: "Robert Lipe" <robertlipe@...>
To: "Gary Turner" <g@...>
Cc: <gpsbabel-misc@...>
Sent: Wednesday, February 01, 2012 6:42 AM
Subject: Re: [Gpsbabel-misc] Processing time to convert large gpx files to
gdb.


> Please put your actual test data on a server somewhere so that we may
> study it in a profiler.

http://gtmaps.co.nz/samples/Auckland-numbers.zip

Ugh.  It's the GDB writer.  We can write that file to other formats in ~6 seconds.

There's some O(n^2) behavior   in the GDB writer.  For each of your 450,000 waypoints, we compare it to the other 449,999 in an unfortunate way to avoid name collisions (and yes, you do have some) which apparently drives MapSource batty.

It's not that we're really even doing that much work inside the QUEUE_FOR_EACH inside gdb_find_wayptq(); it's the blasted explosion of iterations.  It's not trivial to fix.

Is there any other format you can write that Mapsource/Basecamp can read?   Our Mapsource writer doesn't have this code, but I don't know if that means that mps doesn't have this limitation or if people quit using mps before we noticed the issue that we're coding around.

 

> As an aside, Mapsource reads GPX.  Why are you involving GPSBabel in this
> workflow?

Apart from the fact that Mapsource does choke on even slightly big files?
Scriptability mainly. We have 8 files to process. (Apart from the fact that
the developers here DO listen and care ;^)

I may be talking through my cold medicine but it's awesome that Mapsource falls over at a few hundred.  We fall over at a few hundred THOUSAND, ironically because we're trying to work around bugs in Mapsource's GDB reader. 

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@...
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc