Wrong or unsupported GPI file

View: New views
6 Messages — Rating Filter:   Alert me  

Wrong or unsupported GPI file

by Sven Geggus-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi there,

looks like all Garmin POI-Files from http://www.garminonline.de/extras/poi/
are unreadable by gpsbabel.

The Format seems be quite interesting, because ist seems to contain complete
Address Data. The other way round, is ist possible to generate POI data
containing complete addresses using gpsbabel?

Sven

--
"We just typed make"
(Stephen Lambrigh, Director of Server Product Marketing at Informix
                                      about porting their Database to Linux)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code

Re: Wrong or unsupported GPI file

by Robert Lipe-4 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



On Tue, Jul 28, 2009 at 3:40 AM, Sven Geggus <lists@...> wrote:
Hi there,

looks like all Garmin POI-Files from http://www.garminonline.de/extras/poi/
are unreadable by gpsbabel.

These files have four record types we've not seen before.  I've modified our reader to not flip out when seeing these, but if you can apply some local knowledge of this data and help us understand how they relate to the nearby data, perhaps we can decode them more effectively.   The new records are 0x11, 0x0b, 0x0c, and 0x80007.  I don't know what the first two are.  0xc I described in the code as:
                 /* appears to be web links.  If the first 16 bit
                     value  is 0x10, the remainder is a length/URL pair.
                     if it's 0x19, there is a length/phone # (?) pair followed
                     by a length/URL pair.*/
and
               case 0x80007:
                /* Looks like some kind of calendar information. */


Here are samples of each:
Tag: 11

04 00 00 00 00 00 00 00 1f 00 00 00 44 45 1b 00 55 6d 73 6f 6e 73 74 20 2b 20 44 72 61 75 ffffffdf 65 6e 20 46 65 73 74 69 76 61 6c 73 43 00 00 00 44 45 3f 00 20 4c 49 43 5f 31 34 34 34 2c 20 31 34 34 34 2d 33 32 20 55 6d 73 6f 6e 73 74 44 72 61 75 73 73 65 6e 31 47 61 72 6d 69 6e 20 44 65 75 74 73 63 68 6c 61 6e 64 20 47 6d 62 48 20 32 30 30 39
............DE..Umsonst...Drau.en.FestivalsC...DE...LIC.1444..1444.32.UmsonstDraussen1Garmin.Deutschland.GmbH.2009

Tag: b

39 00 0c 00 00 00 44 45 08 00 46 72 65 69 73 69 6e 67 05 00 38 35 33 35 36 13 00 00 00 44 45 0f 00 4c 75 69 74 70 6f 6c 64 73 74 72 61 73 73 65 01 00 31
9.....DE..Freising..85356....DE..Luitpoldstrasse..1

Tag: c

19 00 15 00 2b 34 39 20 20 36 31 20 35 31 20 39 20 37 31 20 31 32 20 32 37 19 00 69 6e 66 6f 40 73 63 68 6c 6f 73 73 67 72 61 62 65 6e 66 65 73 74 2e 64 65 18 00 77 77 77 2e 73 63 68 6c 6f 73 73 67 72 61 62 65 6e 66 65 73 74 2e 64 65
.....49..61.51.9.71.12.27..info.schlossgrabenfest.de..www.schlossgrabenfest.de

Tag: 80007

0d 00 00 00 00 00 07 00 00 00 44 45 03 00 4d 61 69 04 00 00 00 02 00 00 00 01 00
..........DE..Mai..........

If you can help us understand what that data is, perhaps we can do better.   Picking apart the bits and bytes would be best, but even understanding the German, how it's shown on  your device,  and how it relates to the surrounding data (e.g. "This is the mailing address" or "this is a link shown on screen and this is the link text") would be helpful.

Two  tips to anyone reading bits and bytes in this format:
  1) This format is fond of (length,string) pairs.   Sometimes the length is two bytes and sometimes it's four.
  2) Turning on GPI_DBG in garmin_gpi.c and reading reference/umsonstdraussen.gpi is educational.

So in the "tag b" example above, I'm reasonably sure that the first two bytes are a thing.   The next six aren't clear, but the presence of "DE" in a German file is probably not random.  The next two bytes are a count of "8" and  "Freising" is a thing.  The next two bytes are a count of 5 and "85356" is a thing.     Freising is a city, but I don't understand what a Luitpoldstrasse is or how the 85356 and 1 fare related to these other things.

 
The Format seems be quite interesting, because ist seems to contain complete
Address Data. The other way round, is ist possible to generate POI data
containing complete addresses using gpsbabel?

If the question is, "given only a lat/long, can GPSBabel reverse geocode it to get the nearest street address and insert that into an address field", the answer is "no".

RJL

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code

Re: Wrong or unsupported GPI file

by Robert Lipe-4 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thanx to a hint from Alexander Stapff, record type B is now in hand.  They're just like a type 0x8000b without the unknown 32 bit value in front.   (I had it in my mind that German postal codes had letters in them...)

Tag C is some kind of "more information" field.  The first two bytes are a bitmask of the following fields.
if bit zero is set, the first field contains a phone number.
If bit three is set, there is something that looks like a web address, but isn't.
If bit four is set, there is a web address, though it may not always be valid.

This file never has bits one and two set...

Tag: c

18 00 14 00 69 6e 66 6f 40 66 65 73 74 69 76 61 6c 6b 75 6c 74 2e 64 65 13 00 77 77 77 2e 66 65 73 74 69 76 61 6c 6b 75 6c 74 2e 64 65
....info.festivalkult.de..www.festivalkult.de

Tag: c

19 00 13 00 2b 34 39 20 38 31 32 32 20 20 38 38 30 20 39 38 39 20 30 11 00 69 6e 66 6f 40 73 69 6e 6e 66 6c 75 74 2e 62 69 7a 10 00 77 77 7
7 2e 73 69 6e 6e 66 6c 75 74 2e 62 69 7a
.....49.8122..880.989.0..info.sinnflut.biz..www.sinnflut.biz

Tag: c

19 00 0f 00 2b 34 39 20 36 31 35 38 20 39 38 35 30 32 35 18 00 69 6e 66 6f 40 77 75 74 7a 64 6f 67 2d 66 65 73 74 69 76 61 6c 2e 64 65 17 0
0 77 77 77 2e 77 75 74 7a 64 6f 67 2d 66 65 73 74 69 76 61 6c 2e 64 65
.....49.6158.985025..info.wutzdog.festival.de..www.wutzdog.festival.de


Any ideas what the "info." stuff above is?

RJL


On Thu, Aug 6, 2009 at 8:27 AM, Robert Lipe <robertlipe@...> wrote:


On Tue, Jul 28, 2009 at 3:40 AM, Sven Geggus <lists@...> wrote:
Hi there,

looks like all Garmin POI-Files from http://www.garminonline.de/extras/poi/
are unreadable by gpsbabel.

These files have four record types we've not seen before.  I've modified our reader to not flip out when seeing these, but if you can apply some local knowledge of this data and help us understand how they relate to the nearby data, perhaps we can decode them more effectively.   The new records are 0x11, 0x0b, 0x0c, and 0x80007.  I don't know what the first two are.  0xc I described in the code as:
                 /* appears to be web links.  If the first 16 bit
                     value  is 0x10, the remainder is a length/URL pair.
                     if it's 0x19, there is a length/phone # (?) pair followed
                     by a length/URL pair.*/
and
               case 0x80007:
                /* Looks like some kind of calendar information. */


Here are samples of each:
Tag: 11

04 00 00 00 00 00 00 00 1f 00 00 00 44 45 1b 00 55 6d 73 6f 6e 73 74 20 2b 20 44 72 61 75 ffffffdf 65 6e 20 46 65 73 74 69 76 61 6c 73 43 00 00 00 44 45 3f 00 20 4c 49 43 5f 31 34 34 34 2c 20 31 34 34 34 2d 33 32 20 55 6d 73 6f 6e 73 74 44 72 61 75 73 73 65 6e 31 47 61 72 6d 69 6e 20 44 65 75 74 73 63 68 6c 61 6e 64 20 47 6d 62 48 20 32 30 30 39
............DE..Umsonst...Drau.en.FestivalsC...DE...LIC.1444..1444.32.UmsonstDraussen1Garmin.Deutschland.GmbH.2009

Tag: b

39 00 0c 00 00 00 44 45 08 00 46 72 65 69 73 69 6e 67 05 00 38 35 33 35 36 13 00 00 00 44 45 0f 00 4c 75 69 74 70 6f 6c 64 73 74 72 61 73 73 65 01 00 31
9.....DE..Freising..85356....DE..Luitpoldstrasse..1

Tag: c

19 00 15 00 2b 34 39 20 20 36 31 20 35 31 20 39 20 37 31 20 31 32 20 32 37 19 00 69 6e 66 6f 40 73 63 68 6c 6f 73 73 67 72 61 62 65 6e 66 65 73 74 2e 64 65 18 00 77 77 77 2e 73 63 68 6c 6f 73 73 67 72 61 62 65 6e 66 65 73 74 2e 64 65
.....49..61.51.9.71.12.27..info.schlossgrabenfest.de..www.schlossgrabenfest.de

Tag: 80007

0d 00 00 00 00 00 07 00 00 00 44 45 03 00 4d 61 69 04 00 00 00 02 00 00 00 01 00
..........DE..Mai..........

If you can help us understand what that data is, perhaps we can do better.   Picking apart the bits and bytes would be best, but even understanding the German, how it's shown on  your device,  and how it relates to the surrounding data (e.g. "This is the mailing address" or "this is a link shown on screen and this is the link text") would be helpful.

Two  tips to anyone reading bits and bytes in this format:
  1) This format is fond of (length,string) pairs.   Sometimes the length is two bytes and sometimes it's four.
  2) Turning on GPI_DBG in garmin_gpi.c and reading reference/umsonstdraussen.gpi is educational.

So in the "tag b" example above, I'm reasonably sure that the first two bytes are a thing.   The next six aren't clear, but the presence of "DE" in a German file is probably not random.  The next two bytes are a count of "8" and  "Freising" is a thing.  The next two bytes are a count of 5 and "85356" is a thing.     Freising is a city, but I don't understand what a Luitpoldstrasse is or how the 85356 and 1 fare related to these other things.

 
The Format seems be quite interesting, because ist seems to contain complete
Address Data. The other way round, is ist possible to generate POI data
containing complete addresses using gpsbabel?

If the question is, "given only a lat/long, can GPSBabel reverse geocode it to get the nearest street address and insert that into an address field", the answer is "no".

RJL


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code

Parent Message unknown Re: Wrong or unsupported GPI file

by Øyvind Jonsson :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Some parts of this message have been removed. Learn more about Nabble's security policy.
Hi!
I have been experimenting with GPX files to determine how the different fields end up in the GPI file, and have some more info related to the 0x0b and 0x0c records:
 
Record 0x0b holds address details
- First word is a bit mask stating which of the following fields are present. All fields except PostalCode contain a 4-byte length, a 2-byte language code, and a string (with a 2-byte length field).
- Field 1: City
- Field 2: Country
- Field 3: PostalCode (note that this field only contains the 2-byte length + string part).
- Field 4: StreetAddress
 
Record 0x0c holds contact details
- First word is a bit mask stating which of the following fields are present. All fields contain a 2-byte length and a string
- Field 1: PhoneNumber (with Category set to "Phone"). This is the number that can be dialed directly on BT-enabled Garmins.
- Field 2: PhoneNumber (with Category set to "Phone2")
- Field 3: PhoneNumber (with Category set to "Fax")
- Field 4: PhoneNumber (with Category set to "Email")
 
Also, from trying to decipher the GPI format, my understanding is that the record format may be slightly different from the assumptions in garmin_gpi.c:
- 2 byte record type
- 2 byte extension flag (0 for "normal" records and 8 for "extended" records)
- for extended records only, 4-byte extended record length (total number of bytes for all record fields and all nested records, starting after the length field)
- 4-byte length field (total number of bytes for all record fields excluding any nested records, starting after the length field)
 
Hope this is useful!
 
peroy

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code

Re: Wrong or unsupported GPI file

by Robert Lipe-4 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thanx.  I agree with your interpretations and have tweaked our readers to match.

2009/8/11 Øyvind Jonsson <peroy@...>
Hi!
I have been experimenting with GPX files to determine how the different fields end up in the GPI file, and have some more info related to the 0x0b and 0x0c records:
 
Record 0x0b holds address details
- First word is a bit mask stating which of the following fields are present. All fields except PostalCode contain a 4-byte length, a 2-byte language code, and a string (with a 2-byte length field).
- Field 1: City
- Field 2: Country
- Field 3: PostalCode (note that this field only contains the 2-byte length + string part).
- Field 4: StreetAddress
 
Record 0x0c holds contact details
- First word is a bit mask stating which of the following fields are present. All fields contain a 2-byte length and a string
- Field 1: PhoneNumber (with Category set to "Phone"). This is the number that can be dialed directly on BT-enabled Garmins.
- Field 2: PhoneNumber (with Category set to "Phone2")
- Field 3: PhoneNumber (with Category set to "Fax")
- Field 4: PhoneNumber (with Category set to "Email")
 
Also, from trying to decipher the GPI format, my understanding is that the record format may be slightly different from the assumptions in garmin_gpi.c:
- 2 byte record type
- 2 byte extension flag (0 for "normal" records and 8 for "extended" records)
- for extended records only, 4-byte extended record length (total number of bytes for all record fields and all nested records, starting after the length field)
- 4-byte length field (total number of bytes for all record fields excluding any nested records, starting after the length field)
 
Hope this is useful!
 
peroy

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code

Re: Wrong or unsupported GPI file

by Øyvind Jonsson :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Some parts of this message have been removed. Learn more about Nabble's security policy.
Hi,
 
Some more info from my efforts to decode the Garmin GPI format.
 
<LegalStuff>
In case any Garmin lawyers are seeing this: All below info has been gathered by the following procedures, all of which are believed to be within what I am legally allowed to do by the licensing terms to which I have agreed:
- Create GPX and CSV files according to the publicly available definitions of the contents of these files, using fictious or publicly available content, then converting the files by POI Loader.
- Load the resulting GPI to my Garmin device, and observe the behaviour (the device display, as well as the actions when driving to/past a POI).
- Load the resulting GPI to my computer, and decode the generated GPI files using hex editors and other tools, matching the contents to the source GPX/CSV files.
- Load publicly available GPI files to my Garmin device, and observe the behaviour.
- Decode those same GPI files, and match the device behaviour to the file contents.
</LegalStuff>
 
In any case, I only recently came across the GPSBabel project, and since I really like this type of project, I thought it would be a good idea to support it by providing my findings. At a future time I may be able to submit code, but for now I feel a lot more comfortable letting the experts handle that part!
 
 
All line numbers refer to garmin_gpi.c rev 1.18. Also, I have grown into the habit of using bytes (8 bit), words (16 bit) and dwords (32 bit). Hopefully I have avoided mixing them up below.
 
 
read_poi comment (record 0x02/0x80002):
 
The byte being ignored in line 350 seems to be a flag indicating the type of alert info being present. If set to 0, there is no alert (i.e. no record 0x03), if 1 there is a record 0x03. I have seen other values as well (at least 2, 3 and 7), but have not yet managed to correlate those with any specific functionality.
 
 
Some comments to the processing in read_tag:
 
Record 0x03 (alert info, only present if the POI has an alert):
Following the proximity and speed, 2 32-bit fields are read (lines 486, 487). The last one (line 487) seems to have the following layout:
- First an unknown byte which is "always" 1.
- Next a byte with the type of alert (0 for proximity only, 1 for speed, 2 for TourGuide)
- Next a byte giving the source of the alert sound for this POI (0x10 built-in, 0x20 MP3 file)
- Finally, the specific sound. If built-in, I have seen 4 (speed=0) and 5 (speed!=0). If from an MP3 file, this byte holds the MP3 "index" (or whatever), which matches the first byte in the record 0x12 that holds the actual MP3 content.
 
Record 0x04, line 490 (only present if a BMP is associated with the POI or the group of POIs):
Word, identifies the symbol used for a POI (i.e. the BMP file). Holds the "index" of the BMP, matching the first word of the record 0x05 that holds the actual BMP.
 
Record 0x06 (only present if the gpx/csv source file is in a sub-directory):
Word, identifies the group or category name within which this POI resides. Holds the "index" of the file name, matching the first word of the record 0x07 that holds the actual filename.
 
Record 0x05 (BMP file):
This one has a rather complex structure, and there are several fields that are unclear. However, in the interest of being able to build or extract the BMP, here are the details I have discovered so far (in sequence from start of record):
Word, "index" that matches the one in record 0x04.
Word, BMP height in pixels.
Word, BMP width in pixels.
Word, unknown use, but related to color depth (0x0c for mono/4-bit, 0x18 for 8-bit, 0x60 for 24-bit).
Word, number of bits per pixel.
Word, unknown and "always" 0.
Dword, size of BMP image in bytes.
Dword, offset of BMP image in bytes from start of record content.
Dword, unclear, set to 0 for 24-bit, set to 16 for mono.
Dword, unknown and usually 0xff-0x00-0xff-0x00.
Dword, unknown and usually 1.
Dword, offset of BMP color table in bytes from start of record content (set to 0 for 24-bit).
Byte array for the BMP image, 1 byte each pixel, left to right, top to bottom (which is the reverse vertical order from the BMP file).
Dword array for the BMP color table, 256 entries. Within each dword, the first is red, second Green, third Blue, and last unknown (always 0). The first 3 bytes are in reverse order from the BMP file.
 
Record 0x07 matches my notes.
 
Record 0x0a is the Comment field (GPX "cmt", CSV col 4), rather than the description.
 
Record 0x0e contains the Description (GPX "desc", not present in CSV):
 
Record 0x80002: I have also seen POI records that are not extended (i.e. record type 0x02). This would need a flag to read_poi to avoid reading the extended length (or perhaps better, do that before calling read_poi.
 
Record 0x0b has still another field discovered only yesterday (line 562):
Mask bit 5, when set, shows the presence of something that can only be the house number (a string without language indicator). Found by looking up the address of one of the German POIs via the web. I have not found any way of setting this from a GPX file, however. I guess the usual would be to have this as part of the street address (Germans and Scandinavians place the house number after the street name, which may explain why this field is there in the first place).
 
Record 0x0c has a very minor issue on line 570:
The GPX "Category" tag for this field has a space between "Phone" and "2".
 
Record 0x8000c (line 587) most likely can use the code for record 0x0c.
 
Record 0x12 (line 601):
This record has the MP3 sound for a POI. Layout:
Byte, sound number matching the field in record 0x03.
Word, unknown and "always" 0x0120.
2-byte language(?) indicator (always "EN" on my files).
Next follows the actual MP3 file (a direct copy).
 
Record 0x11 (line 604) is definitely separate from record 0x07, although the meaning of this record is rather unclear. Also, record 0x11 belongs in the GPI heading part (I have only seen this one nested inside a record 0x01).
 
Record 0x80007 is the same as record 0x07, but being "extended", it contains a nested record 0x04.
 
 
A couple of comments to wdata_write:
 
The comment to record 0x02 also affects line 934: This should probably be "if (dt->alerts) gbputc(1, fout);", as is also implied by the comment.
 
The constant in line 957 should probably be 0x1000100 (an extra 0 between the 1's).
 
The constant in line 959 should possibly be 0 if speed is zero (i.e. a proximity alert). I'm not sure how to detect that the POI is for a TourGuide - if so, the value should probably be 2.
 
The "flag" that gets written in line 960 matches my notes exactly, but does not allow the option of adding MP3 files. Maybe this is for later, but the MP3 "index" goes here, and line 961 should then write 0x20.
 
The section in lines 964 through 968 writes record type 0x04. In all files I have analyzed, this record is placed in front of record 0x03 (i.e. just before line 938). I don't know if this is significant. In line 966 the length is written (always 2). The next line writes the BMP "index", starting with 0 for the first BMP.
 
Line 971 writes record 0x0a - this is the comment rather than the description. May need to change lines 914/915.
 
Line 991: The 2 is the length field if the record is extended (having room for the bit mask only). I have also seen this record non-extended, in which case the length field would have to include all subsequent field. "Phone 2", "Fax", "Email" and "Link" may need to be added here (ref the decoder for record 0x0c).
 
If the POI has associated JPGs, these get written following record 0x0c. One JPG per record.
 
And finally the description (record 0x0e) gets written. All these records appear as nested records to the 0x02 POI record, and thus would have to be included in the extended length field of this record (line 927).
 
 
peroy
 

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gpsbabel-code mailing list  http://www.gpsbabel.org
Gpsbabel-code@...
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code