amarok doesn't see all the audio files on my Sansa E260 (includes a patch)

[Aran Cox-2]

I've been using my Sansa E260 with Amarok and I noticed that when I
"Read Device" not all the songs show up in the collection.  It seems
that oggs are identified as "audio/x-vorbis+ogg" so I added that to
the list and now all but a handful of songs show up in the
collection.  I haven't pinnned down exactly which ones are not getting
added but when I do I'll try to figure out why.



 

diff --git a/src/collection/umscollection/handler/UmsHandler.cpp b/src/collection/umscollection/handler/UmsHandler.cpp
index a6e0ed2..827e23b 100644
--- a/src/collection/umscollection/handler/UmsHandler.cpp
+++ b/src/collection/umscollection/handler/UmsHandler.cpp
@@ -141,7 +141,7 @@ UmsHandler::init()
 */
     m_mimetypes << "audio/mpeg" << "audio/x-wav" << "video/x-ms-asf" << "audio/flac"
             << "audio/x-ms-wma" << "application/x-ogg" << "audio/x-aac" << "audio/mp4a-latm"
-            << "video/mp4" << "audio/ac3";
+            << "video/mp4" << "audio/ac3" << "audio/x-vorbis+ogg";
 
 
     m_timer.setSingleShot( true );

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:
> I've been using my Sansa E260 with Amarok and I noticed that when I
> "Read Device" not all the songs show up in the collection.  It seems
> that oggs are identified as "audio/x-vorbis+ogg" so I added that to
> the list and now all but a handful of songs show up in the
> collection.  I haven't pinnned down exactly which ones are not getting
> added but when I do I'll try to figure out why.

Aran,

Do you know what is providing that mimetype? audio/vorbis and audio/ogg
are defined mime types by IANA, and according to Xiph.org*, audio/ogg
should be used for Ogg/Vorbis files -- at least, before it got redefined
to application/ogg.

I can't find anything that thinks x-vorbis+ogg is correct, so getting
this fixed at the source would be good.

*http://wiki.xiph.org/index.php/MIME_Types_and_File_Extensions#.ogg_-_audio.2Fogg

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-3]

On Fri, Oct 09, 2009 at 04:58:23PM -0400, Jeff Mitchell wrote:

> Aran Cox wrote:
> > I've been using my Sansa E260 with Amarok and I noticed that when I
> > "Read Device" not all the songs show up in the collection.  It seems
> > that oggs are identified as "audio/x-vorbis+ogg" so I added that to
> > the list and now all but a handful of songs show up in the
> > collection.  I haven't pinnned down exactly which ones are not getting
> > added but when I do I'll try to figure out why.
>
> Aran,
>
> Do you know what is providing that mimetype? audio/vorbis and audio/ogg
> are defined mime types by IANA, and according to Xiph.org*, audio/ogg
> should be used for Ogg/Vorbis files -- at least, before it got redefined
> to application/ogg.
>
> I can't find anything that thinks x-vorbis+ogg is correct, so getting
> this fixed at the source would be good.
>
> *http://wiki.xiph.org/index.php/MIME_Types_and_File_Extensions#.ogg_-_audio.2Fogg
>
> --Jeff
>

It looks like it's returned by KMimeType::findByFileContent() which is
a part of kdelibs.  I'm on FC11 and the kdelibs version I have
installed is kdelibs-4.3.1-3.fc11.i586.  It looks like x-vorbis+ogg is
defined in a file /usr/share/mime/packages/freedesktop.org.xml which
is provided by the shared-mime-info package on my system.

I have no idea what's right or not of course!
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Colin Guthrie-6]

'Twas brillig, and Aran Cox at 10/10/09 04:14 did gyre and gimble:

> On Fri, Oct 09, 2009 at 04:58:23PM -0400, Jeff Mitchell wrote:
>> Aran Cox wrote:
>>> I've been using my Sansa E260 with Amarok and I noticed that when I
>>> "Read Device" not all the songs show up in the collection.  It seems
>>> that oggs are identified as "audio/x-vorbis+ogg" so I added that to
>>> the list and now all but a handful of songs show up in the
>>> collection.  I haven't pinnned down exactly which ones are not getting
>>> added but when I do I'll try to figure out why.
>> Aran,
>>
>> Do you know what is providing that mimetype? audio/vorbis and audio/ogg
>> are defined mime types by IANA, and according to Xiph.org*, audio/ogg
>> should be used for Ogg/Vorbis files -- at least, before it got redefined
>> to application/ogg.
>>
>> I can't find anything that thinks x-vorbis+ogg is correct, so getting
>> this fixed at the source would be good.
>>
>> *http://wiki.xiph.org/index.php/MIME_Types_and_File_Extensions#.ogg_-_audio.2Fogg
>>
>> --Jeff
>>
>
> It looks like it's returned by KMimeType::findByFileContent() which is
> a part of kdelibs.  I'm on FC11 and the kdelibs version I have
> installed is kdelibs-4.3.1-3.fc11.i586.  It looks like x-vorbis+ogg is
> defined in a file /usr/share/mime/packages/freedesktop.org.xml which
> is provided by the shared-mime-info package on my system.
>
> I have no idea what's right or not of course!

This sounds like an issue i came across a couple years ago with the
gstreamer backend for phonon.

http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/phonon/current/SOURCES/phonon-4.2.0-ogg-mime-type.patch?view=markup

I don't think it's exactly the same issue, but the solution could
probably be worked out with the same approach...

If I get some time I'll clean up the patch and commit it upstream. Let
me know if you can fix it using this approach.

Col


--

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited [http://www.tribalogic.net/]
Open Source:
   Mandriva Linux Contributor [http://www.mandriva.com/]
   PulseAudio Hacker [http://www.pulseaudio.org/]
   Trac Hacker [http://trac.edgewall.org/]

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Colin Guthrie-6]

'Twas brillig, and Colin Guthrie at 10/10/09 11:02 did gyre and gimble:
> This sounds like an issue i came across a couple years ago

s/a couple years/about a year/

:p

--

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited [http://www.tribalogic.net/]
Open Source:
   Mandriva Linux Contributor [http://www.mandriva.com/]
   PulseAudio Hacker [http://www.pulseaudio.org/]
   Trac Hacker [http://trac.edgewall.org/]

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Colin Guthrie wrote:
> 'Twas brillig, and Colin Guthrie at 10/10/09 11:02 did gyre and gimble:
>> This sounds like an issue i came across a couple years ago
>
> s/a couple years/about a year/

Yeah, another fricking shared-mime-info issue.

The problem is that the shared-mime-info database returns very specific
(and AFAICT not actually defined in any official specs) mime strings,
which it then defines as sub-types of more legitimate mime types.

So in this case audio/x-vorbis+ogg is deifed as a subtype of audio/ogg,
which itself is defined as a subtype of application/ogg -- which is
arguably what it should actually be returning in the first place.

If you look at the KMimeType docs, under ::is(), you see:

"Do not use name()=="somename" anymore, to check for a given mimetype"

Unfortunately, the code is doing just that.

I should mention that some of the types in that list should probably
still be updated. For instance, it uses application/x-ogg, which is
defined as an alias for application/ogg, so might as well just use
application/ogg. And the parent levels should always be what are in that
list, so that checking using ::is(), which takes into account
inheritance, should work.

Anyways, I've committed some changes in fd111a2. Please let me know if
this helps.

Thanks,
Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-2]

On Sat, Oct 10, 2009 at 10:27:38AM -0400, Jeff Mitchell wrote:

> Colin Guthrie wrote:
> > 'Twas brillig, and Colin Guthrie at 10/10/09 11:02 did gyre and gimble:
> >> This sounds like an issue i came across a couple years ago
> >
> > s/a couple years/about a year/
>
> Yeah, another fricking shared-mime-info issue.
>
> The problem is that the shared-mime-info database returns very specific
> (and AFAICT not actually defined in any official specs) mime strings,
> which it then defines as sub-types of more legitimate mime types.
>
> So in this case audio/x-vorbis+ogg is deifed as a subtype of audio/ogg,
> which itself is defined as a subtype of application/ogg -- which is
> arguably what it should actually be returning in the first place.
>
> If you look at the KMimeType docs, under ::is(), you see:
>
> "Do not use name()=="somename" anymore, to check for a given mimetype"
>
> Unfortunately, the code is doing just that.
>
> I should mention that some of the types in that list should probably
> still be updated. For instance, it uses application/x-ogg, which is
> defined as an alias for application/ogg, so might as well just use
> application/ogg. And the parent levels should always be what are in that
> list, so that checking using ::is(), which takes into account
> inheritance, should work.
>
> Anyways, I've committed some changes in fd111a2. Please let me know if
> this helps.
>

I tried the latest changes and my ogg files are recognized (without my
patch) but my .m4a files are not.  They show up as audio/mp4.
The way I read http://www.rfc-editor.org/rfc/rfc4337.txt mp4 should be
a valid mime type for MPEG4 files with audio and no video.  Should it
be added to that list?

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-2]

On Sat, Oct 10, 2009 at 9:27 AM, Jeff Mitchell <mitchell@...> wrote:

> Colin Guthrie wrote:
>> 'Twas brillig, and Colin Guthrie at 10/10/09 11:02 did gyre and gimble:
>>> This sounds like an issue i came across a couple years ago
>>
>> s/a couple years/about a year/
>
> Yeah, another fricking shared-mime-info issue.
>
> The problem is that the shared-mime-info database returns very specific
> (and AFAICT not actually defined in any official specs) mime strings,
> which it then defines as sub-types of more legitimate mime types.
>
> So in this case audio/x-vorbis+ogg is deifed as a subtype of audio/ogg,
> which itself is defined as a subtype of application/ogg -- which is
> arguably what it should actually be returning in the first place.
>
> If you look at the KMimeType docs, under ::is(), you see:
>
> "Do not use name()=="somename" anymore, to check for a given mimetype"
>
> Unfortunately, the code is doing just that.
>
> I should mention that some of the types in that list should probably
> still be updated. For instance, it uses application/x-ogg, which is
> defined as an alias for application/ogg, so might as well just use
> application/ogg. And the parent levels should always be what are in that
> list, so that checking using ::is(), which takes into account
> inheritance, should work.
>
> Anyways, I've committed some changes in fd111a2. Please let me know if
> this helps.

It did help.  I thought that it was still missing my m4a (audio/mp4)
files but I was mistaken.  Actually what is happening is that some of
my files are showing up with a mime-type of application/octet-stream.
These files include every flac I've attempted to copy to the device
and a couple of random mp3 files.  Amarok and my Rockbox'ed Sansa E260
can play these files, but the UmsHandler won't scan them because they
don't look like audio files.  I don't usually copy flac's to my Sansa
of course but the Sansa will play them.

Also, if you attempt to use amarok to copy a file with a format not
listed in formats it will say the file is already on the device, not
that the file format isn't supported which is the real reason the code
refused to copy them.
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-2]

On Sat, Oct 10, 2009 at 1:52 PM, Aran Cox <arancox@...> wrote:

> On Sat, Oct 10, 2009 at 10:27:38AM -0400, Jeff Mitchell wrote:
>> Colin Guthrie wrote:
>> > 'Twas brillig, and Colin Guthrie at 10/10/09 11:02 did gyre and gimble:
>> >> This sounds like an issue i came across a couple years ago
>> >
>> > s/a couple years/about a year/
>>
>> Yeah, another fricking shared-mime-info issue.
>>
>> The problem is that the shared-mime-info database returns very specific
>> (and AFAICT not actually defined in any official specs) mime strings,
>> which it then defines as sub-types of more legitimate mime types.
>>
>> So in this case audio/x-vorbis+ogg is deifed as a subtype of audio/ogg,
>> which itself is defined as a subtype of application/ogg -- which is
>> arguably what it should actually be returning in the first place.
>>
>> If you look at the KMimeType docs, under ::is(), you see:
>>
>> "Do not use name()=="somename" anymore, to check for a given mimetype"
>>
>> Unfortunately, the code is doing just that.
>>
>> I should mention that some of the types in that list should probably
>> still be updated. For instance, it uses application/x-ogg, which is
>> defined as an alias for application/ogg, so might as well just use
>> application/ogg. And the parent levels should always be what are in that
>> list, so that checking using ::is(), which takes into account
>> inheritance, should work.
>>
>> Anyways, I've committed some changes in fd111a2. Please let me know if
>> this helps.
>>
>
> I tried the latest changes and my ogg files are recognized (without my
> patch) but my .m4a files are not.  They show up as audio/mp4.
> The way I read http://www.rfc-editor.org/rfc/rfc4337.txt mp4 should be
> a valid mime type for MPEG4 files with audio and no video.  Should it
> be added to that list?
>

Ignore this and see my other email... I confused myself with which
version I was testing but I see now that audio/mp4 was added to the
mimetype list.
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:
> It did help.  I thought that it was still missing my m4a (audio/mp4)
> files but I was mistaken.  Actually what is happening is that some of
> my files are showing up with a mime-type of application/octet-stream.
> These files include every flac I've attempted to copy to the device
> and a couple of random mp3 files.  Amarok and my Rockbox'ed Sansa E260
> can play these files, but the UmsHandler won't scan them because they
> don't look like audio files.  I don't usually copy flac's to my Sansa
> of course but the Sansa will play them.

application/octet-stream is a default mime type for when it can't be
determined. This could signal a problem with the flac files...not sure.
It would be good if there was a way to check a file against the
shared-mime-info directory directly -- I don't know if there is, but if
you can find one, see what is returned. Also compare that to the output
of "file" run on those files. It could be a bug in the s-m-i database.

> Also, if you attempt to use amarok to copy a file with a format not
> listed in formats it will say the file is already on the device, not
> that the file format isn't supported which is the real reason the code
> refused to copy them.

xevix: ^



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-2]

On Sun, Oct 11, 2009 at 10:29 AM, Jeff Mitchell <mitchell@...> wrote:

> Aran Cox wrote:
>> It did help.  I thought that it was still missing my m4a (audio/mp4)
>> files but I was mistaken.  Actually what is happening is that some of
>> my files are showing up with a mime-type of application/octet-stream.
>> These files include every flac I've attempted to copy to the device
>> and a couple of random mp3 files.  Amarok and my Rockbox'ed Sansa E260
>> can play these files, but the UmsHandler won't scan them because they
>> don't look like audio files.  I don't usually copy flac's to my Sansa
>> of course but the Sansa will play them.
>
> application/octet-stream is a default mime type for when it can't be
> determined. This could signal a problem with the flac files...not sure.
> It would be good if there was a way to check a file against the
> shared-mime-info directory directly -- I don't know if there is, but if
> you can find one, see what is returned. Also compare that to the output
> of "file" run on those files. It could be a bug in the s-m-i database.

There is a gst-typefind program that will print mime types.  It thinks
my mp3 files are application/x-id3 and my flac files are audio/x-flac!
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:
>> application/octet-stream is a default mime type for when it can't be
>> determined. This could signal a problem with the flac files...not sure.
>> It would be good if there was a way to check a file against the
>> shared-mime-info directory directly -- I don't know if there is, but if
>> you can find one, see what is returned. Also compare that to the output
>> of "file" run on those files. It could be a bug in the s-m-i database.
>
> There is a gst-typefind program that will print mime types.  It thinks
> my mp3 files are application/x-id3

That's crazy. And it's certainly not from shared-mime-info -- at least,
not from the default file included with it.

> and my flac files are audio/x-flac!

audio/x-flac should be getting detected now with my updated code...are
you really still having problems with them?

If you can find one file of each type that you can send to me or make
available to me, I could examine them more closely.

Thanks!
--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-2]

On Sun, Oct 11, 2009 at 10:34 PM, Jeff Mitchell <mitchell@...> wrote:

> Aran Cox wrote:
>>> application/octet-stream is a default mime type for when it can't be
>>> determined. This could signal a problem with the flac files...not sure.
>>> It would be good if there was a way to check a file against the
>>> shared-mime-info directory directly -- I don't know if there is, but if
>>> you can find one, see what is returned. Also compare that to the output
>>> of "file" run on those files. It could be a bug in the s-m-i database.
>>
>> There is a gst-typefind program that will print mime types.  It thinks
>> my mp3 files are application/x-id3
>
> That's crazy. And it's certainly not from shared-mime-info -- at least,
> not from the default file included with it.
>
>> and my flac files are audio/x-flac!
>
> audio/x-flac should be getting detected now with my updated code...are
> you really still having problems with them?
>
> If you can find one file of each type that you can send to me or make
> available to me, I could examine them more closely.
>

I used strace to see what files gst-typefind is opening and it doesn't
look like it opens any of the files in /usr/share/mime/ . x-id3
doesn't seem to be defined in any of those files anyway, so it's
possible gst-typefind is using it's own database of mimetypes and
isn't a good test at all.

audio/x-flac is defined in the freedesktop.org.xml file, but only as a
wildcard.  I looked at a flac in a hex-editor and it starts with a
magic string, just like an ogg so I wonder why it isn't defined like
x-ogg is.  I added this to the freedesktop.org.xml definition for
audio/x-flac:

   <magic priority="50">
      <match value="fLaC" type="string" offset="0"/>
    </magic>

and then used update-mime-database /usr/share/mime.

Now, amarok recoginizes my flac files on my Sansa!

There are still the two mp3s that show up as application/octet-stream
and I'll look at those files later.  There must be something odd about
the files or the mime definition for mp3, I would guess.

I'd like to make a little wrapper program around the KMimeType calls
at some point, too, but we'll see.
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:
> I used strace to see what files gst-typefind is opening and it doesn't
> look like it opens any of the files in /usr/share/mime/ . x-id3
> doesn't seem to be defined in any of those files anyway, so it's
> possible gst-typefind is using it's own database of mimetypes and
> isn't a good test at all.

yep  :-(

> audio/x-flac is defined in the freedesktop.org.xml file, but only as a
> wildcard.

Err...

  <mime-type type="audio/x-flac">
    <comment>FLAC audio</comment>
    <comment xml:lang="en_GB">FLAC audio</comment>
    <glob pattern="*.flac"/>
  </mime-type>

(translations chopped out)

That's top-level, not a wildcard. What version of shared-mime-info?

I looked at a flac in a hex-editor and it starts with a
> magic string, just like an ogg so I wonder why it isn't defined like
> x-ogg is.

?/x-ogg is only an alias, with varying ?s.

I still think it's either something with your files, or with your
shared-mime-info version.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Jeff Mitchell wrote:

> Aran Cox wrote:
>> I used strace to see what files gst-typefind is opening and it doesn't
>> look like it opens any of the files in /usr/share/mime/ . x-id3
>> doesn't seem to be defined in any of those files anyway, so it's
>> possible gst-typefind is using it's own database of mimetypes and
>> isn't a good test at all.
>
> yep  :-(
>
>> audio/x-flac is defined in the freedesktop.org.xml file, but only as a
>> wildcard.

Ah, wait, I understand what you mean.

Do your FLACs not end in .flac? What do they end in?

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:
>    <magic priority="50">
>       <match value="fLaC" type="string" offset="0"/>
>     </magic>

I've filed a bug requesting this be added with the shared-mime-info
bugzilla at FDO.

In the meantime, answer the other email :-)

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Aran Cox-2]

On Mon, Oct 12, 2009 at 12:14 PM, Jeff Mitchell <mitchell@...> wrote:
> Jeff Mitchell wrote:
>> Aran Cox wrote:

>> yep  :-(
>>
>>> audio/x-flac is defined in the freedesktop.org.xml file, but only as a
>>> wildcard.
>
> Ah, wait, I understand what you mean.
>
> Do your FLACs not end in .flac? What do they end in?
>

They end in .flac, all lower case!  But adding the string match fixed
it so... I have no idea. 4 letter extension on a vfat filesystem
maybe?
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:

> On Mon, Oct 12, 2009 at 12:14 PM, Jeff Mitchell <mitchell@...> wrote:
>> Jeff Mitchell wrote:
>>> Aran Cox wrote:
>
>>> yep  :-(
>>>
>>>> audio/x-flac is defined in the freedesktop.org.xml file, but only as a
>>>> wildcard.
>> Ah, wait, I understand what you mean.
>>
>> Do your FLACs not end in .flac? What do they end in?
>>
>
> They end in .flac, all lower case!  But adding the string match fixed
> it so... I have no idea. 4 letter extension on a vfat filesystem
> maybe?

Hrm. FAT or FAT-32?

The former I could definitely see it being an issue (it might only be
presenting ".fla"); the latter I *thought* should handle four.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Aran Cox wrote:
>    <magic priority="50">
>       <match value="fLaC" type="string" offset="0"/>
>     </magic>

Good news!

The shared-mime-info maintainer just committed this patch. The current
version is 0.7.0, so it will probably be in 0.8 (or 0.7.x if there are any).

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok