Amarok 1.4 collection and artwork importing

Hi,

All the issues I'm describing here are related to Amarok 2.2 truck,
pulled from git repository. I'm running OpenSuse 11.1, KDE 4.3.2 (From
OpenSuse 4.3.x repository).

The first issue is about the import of my collection from Amarok 1.4.
It imports only part of my collection (1604 out of 3400 musics). I've
checked the source code
(src/databaseimporter/amarok14/FastForwardWorker.cpp), and I see that
the SQL query trusts the information presented at the "statistics"
table. The thing is that "statistics" table contains information of
music that has been played at least once. So, if a song has never been
played it would not be imported. Also, the table contains information
of songs from removable devices (i.e iPod) that probably aren't
present when importing. Below are two example lines from my
"statistics" table dump: the first one shows a bad entry about a song
from my iPod. The second is an ordinary and valid entry:

/iPod_Control/Music/F01/USLG.mp3|2|1208408141|1209562831|9.5|0|3||0
./dados/mp3/B/Barão Vermelho/Malandragem Da Um
Tempo.mp3|-1|1208452330|1214098269|79.0|0|2|amarok-sqltrackuid://e6ecb3fc5326ff474dbcdd21620beb68|0

The fact is that ALL song from my collection are listed on the "tags"
label. So I consider that it would be correct if the SQL query would
do a left outer join from "tags" and "statistics" first. It would
eliminate the files from removable devices, keeping all my local
songs, but preserving statistics from songs already played. I've
proposed a new SQL query and attached the diff file to this message. I
am not totally familiar with the Amarok 1.4 DB schema (I've dumped it
and tried to figure it out by myself), so I would appreciate a any
help from experienced users on this.

The second issue is related to the importation of my artwork from
Amarok 1.4. The line below from FastForwardWorker.cpp shows that all
Amarok does is to copy each file from
~/.kde/share/apps/amarok/albumcovers/large to
~/.kde4/share/apps/amarok/albumcovers/large.

Line 332: KIO::FileCopyJob *job = KIO::file_copy( src, dst, -1 /*no
special perms*/ , KIO::HideProgressInfo );

But it takes too long. If I copy the image files (about 192MiB) using
the 'cp' command from shell, it would take less than one minute. But
if I import the same files using the import wizard, it takes almost 2
hours.  I've started amarok with debug on. So whenever I try to import
artwork, I get the message below for each of the image files I try to
import:

amarok:    image copy: "e173e4437e8ff6b14e85ddfa6da8f42b"  :
"/home/pcgomes/.kde/share/apps/amarok/albumcovers/large/e173e4437e8ff6b14e85ddfa6da8f42b"
QObject::connect: Cannot queue arguments of type 'KIO::filesize_t'
(Make sure 'KIO::filesize_t' is registered using qRegisterMetaType().)
QObject::connect: Cannot queue arguments of type 'KIO::filesize_t'
(Make sure 'KIO::filesize_t' is registered using qRegisterMetaType().)

Does this message means anything? Does anyone have a clue why is it so
slow? This would be very helpful for me to solve this problem.

Thanks in advance,

Pedro Gomes


_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Pedro de Carvalho Gomes wrote:
> Hi,
>
> All the issues I'm describing here are related to Amarok 2.2 truck,

We have a truck!?!?!

> The first issue is about the import of my collection from Amarok 1.4.
> It imports only part of my collection (1604 out of 3400 musics). I've
> checked the source code
> (src/databaseimporter/amarok14/FastForwardWorker.cpp), and I see that
> the SQL query trusts the information presented at the "statistics"
> table. The thing is that "statistics" table contains information of
> music that has been played at least once. So, if a song has never been
> played it would not be imported.

I don't follow why this is a problem. I guess you could have ratings for
songs that have never been played, but that seems kinda unlikely.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Wed, Oct 28, 2009 at 4:58 PM, Jeff Mitchell <mitchell@...> wrote:

Pedro de Carvalho Gomes wrote:
> Hi,
>
> All the issues I'm describing here are related to Amarok 2.2 truck,

We have a truck!?!?!

> The first issue is about the import of my collection from Amarok 1.4.
> It imports only part of my collection (1604 out of 3400 musics). I've
> checked the source code
> (src/databaseimporter/amarok14/FastForwardWorker.cpp), and I see that
> the SQL query trusts the information presented at the "statistics"
> table. The thing is that "statistics" table contains information of
> music that has been played at least once. So, if a song has never been
> played it would not be imported.

I don't follow why this is a problem. I guess you could have ratings for
songs that have never been played, but that seems kinda unlikely.

I think he misunderstood the use of the import functionality. It is not to import your collection from amarok 1.x, but rather to import your /statistics/ from 1.x. So if you have an empty collection, and you import from 1.x, you'll only see tracks that had statistics, as that's what's been added to your database. You need to have amarok scan your music files to actually see them all--the import will just add your stats.

iirc...

leo 

--
______________________________________________________
lfranchi@...                       Tufts  University 2010
leonardo.franchi@...         The KDE Project

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Dan Leinir Turthra Jensen-2]

On Wednesday 28 October 2009 21:58:21 Jeff Mitchell wrote:

> Pedro de Carvalho Gomes wrote:
> > Hi,
> >
> > All the issues I'm describing here are related to Amarok 2.2 truck,
>
> We have a truck!?!?!
>
> > The first issue is about the import of my collection from Amarok 1.4.
> > It imports only part of my collection (1604 out of 3400 musics). I've
> > checked the source code
> > (src/databaseimporter/amarok14/FastForwardWorker.cpp), and I see that
> > the SQL query trusts the information presented at the "statistics"
> > table. The thing is that "statistics" table contains information of
> > music that has been played at least once. So, if a song has never been
> > played it would not be imported.
>
> I don't follow why this is a problem. I guess you could have ratings for
> songs that have never been played, but that seems kinda unlikely.

  On the contrary, that's a perfectly valid use case. Look at it this way: You
just got a hold of a track that you know you like (say Brittney Spears' Oops,
I Did It Again ;) ), and you happen to know that you would rate that song at 5
stars. Now, of course, that song hasn't been played, but... :)

--
..Dan // Leinir..
http://leinir.dk/

                          Co-
                            existence
                          or no
                            existence

                          - Piet Hein
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Dan Leinir Turthra Jensen wrote:

> On Wednesday 28 October 2009 21:58:21 Jeff Mitchell wrote:
>> Pedro de Carvalho Gomes wrote:
>>> Hi,
>>>
>>> All the issues I'm describing here are related to Amarok 2.2 truck,
>> We have a truck!?!?!
>>
>>> The first issue is about the import of my collection from Amarok 1.4.
>>> It imports only part of my collection (1604 out of 3400 musics). I've
>>> checked the source code
>>> (src/databaseimporter/amarok14/FastForwardWorker.cpp), and I see that
>>> the SQL query trusts the information presented at the "statistics"
>>> table. The thing is that "statistics" table contains information of
>>> music that has been played at least once. So, if a song has never been
>>> played it would not be imported.
>> I don't follow why this is a problem. I guess you could have ratings for
>> songs that have never been played, but that seems kinda unlikely.
>
>   On the contrary, that's a perfectly valid use case. Look at it this way: You
> just got a hold of a track that you know you like (say Brittney Spears' Oops,
> I Did It Again ;) ), and you happen to know that you would rate that song at 5
> stars. Now, of course, that song hasn't been played, but... :)

I didn't say it wasn't a valid use case, I said it wasn't likely  :-)

But I think Leo is right -- I think he's simply not understanding what
the converter is supposed to do.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

I don't really agree with the use of the collection import for
statistics only. I can enumerate some reasons:

- First, the button "Import Collection" is side by side with "Fully
Rescan Collection", indicating that both are distinct methods to
populate your Local Collection Database. Also, when you mention
"collection", it implies the filenames, statistics, ratings, artwork,
playlists (I'm planning to implement this). Users are leaded to
believe that all information would be present at the new collection.

- Today the consequence of using "Import Collection" is that only a
part of songs are imported. It misleads users and they could think
that they did something wrong, or that they've lost data.

- I would prefer to import all files from Amarok 1.4 than to fully
rescan my disc because search for files at disc may consume too much
time and my machine start trashing. Also, why would I want to rescan
if I believe I already have a reliable source of information (Amarok
1.4 database).

I'd like experienced developers to consider this points, because I
believe that importation should be able to make the transition from
Amarok 1.4 (or even iTunes) to Amarok 2 as smooth as possible.

Cheers,

Pedro

On Wed, Oct 28, 2009 at 7:19 PM, Jeff Mitchell <mitchell@...> wrote:

> Dan Leinir Turthra Jensen wrote:
>> On Wednesday 28 October 2009 21:58:21 Jeff Mitchell wrote:
>>> Pedro de Carvalho Gomes wrote:
>>>> Hi,
>>>>
>>>> All the issues I'm describing here are related to Amarok 2.2 truck,
>>> We have a truck!?!?!
>>>
>>>> The first issue is about the import of my collection from Amarok 1.4.
>>>> It imports only part of my collection (1604 out of 3400 musics). I've
>>>> checked the source code
>>>> (src/databaseimporter/amarok14/FastForwardWorker.cpp), and I see that
>>>> the SQL query trusts the information presented at the "statistics"
>>>> table. The thing is that "statistics" table contains information of
>>>> music that has been played at least once. So, if a song has never been
>>>> played it would not be imported.
>>> I don't follow why this is a problem. I guess you could have ratings for
>>> songs that have never been played, but that seems kinda unlikely.
>>
>>   On the contrary, that's a perfectly valid use case. Look at it this way: You
>> just got a hold of a track that you know you like (say Brittney Spears' Oops,
>> I Did It Again ;) ), and you happen to know that you would rate that song at 5
>> stars. Now, of course, that song hasn't been played, but... :)
>
> I didn't say it wasn't a valid use case, I said it wasn't likely  :-)
>
> But I think Leo is right -- I think he's simply not understanding what
> the converter is supposed to do.
>
> --Jeff
>
>
> _______________________________________________
> Amarok mailing list
> Amarok@...
> https://mail.kde.org/mailman/listinfo/amarok
>
>

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Pedro de Carvalho Gomes wrote:
> I don't really agree with the use of the collection import for
> statistics only. I can enumerate some reasons:

It doesn't matter if you agree with it or not. The schema has changed in
many fundamental ways and there is not a good way to map everything 1:1.

> - I would prefer to import all files from Amarok 1.4 than to fully
> rescan my disc because search for files at disc may consume too much
> time and my machine start trashing.

If scanning makes your machine thrash, fix your machine.

> Also, why would I want to rescan
> if I believe I already have a reliable source of information (Amarok
> 1.4 database).

Because it's not as reliable as you think it is -- at least in the sense
that it's not really usable by A2.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Jeff Mitchell wrote:
> Pedro de Carvalho Gomes wrote:
>> I don't really agree with the use of the collection import for
>> statistics only. I can enumerate some reasons:
>
> It doesn't matter if you agree with it or not. The schema has changed in
> many fundamental ways and there is not a good way to map everything 1:1.

By the by. One of the ways it has changed is in the unique identifiers
used to, err, uniquely identify each track.

To figure out the appropriate A2 identifier each track would need to be
scanned, since it's based upon contents of the file and the algorithm
has changed since A1. There is no way to upconvert.

Therefore every file *must* be scanned with the A2 collection scanner
regardless of whether you've imported every detail of every track or not
from the A1 database, at which time the scanner can (and should) also
read fresh details of all the other tags. Therefore there is no point in
converting all these throwaway details over, no effort will be made to
convert these, and any patch that implements such behavior will be
discarded as useless.

QED.

P.S.: You could make a case that whatever dialog is shown when Import
Collection is selected should detail what will actually be imported, so
that the user is aware of this.




_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Guðlaugur Jóhannesson-3]

On Wednesday 28 October 2009 16:49:57 Jeff Mitchell wrote:
> Jeff Mitchell wrote:
> P.S.: You could make a case that whatever dialog is shown when Import
> Collection is selected should detail what will actually be imported, so
> that the user is aware of this.

The fact that you call it import collection is the first sign of confusion.  
Why not call it import statistics like it actually does and fewer will be
confused?

Gudlaugur
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Jonas Norlander]

2009/10/29 Jeff Mitchell <mitchell@...>:

> Pedro de Carvalho Gomes wrote:
>> I don't really agree with the use of the collection import for
>> statistics only. I can enumerate some reasons:
>
> It doesn't matter if you agree with it or not. The schema has changed in
> many fundamental ways and there is not a good way to map everything 1:1.
>
>> - I would prefer to import all files from Amarok 1.4 than to fully
>> rescan my disc because search for files at disc may consume too much
>> time and my machine start trashing.
>
> If scanning makes your machine thrash, fix your machine.

How stupid comment is this?
He could have a perfect ordinary machine that works for most things,
it doesn't mean it's something wrong with it. I have a Quad core, 4Gb
Ram and fast sata2 disks and it still i a pain and take several hours
to do a full scan of my collection.

>> Also, why would I want to rescan
>> if I believe I already have a reliable source of information (Amarok
>> 1.4 database).
>
> Because it's not as reliable as you think it is -- at least in the sense
> that it's not really usable by A2.
>

In that case it should be mention somewhere that the import of 1.4
collection is not as usable in Amarok 2 as one would think and I don't
think that hes alone to think that.

/ Jonas
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Guðlaugur Jóhannesson wrote:
> On Wednesday 28 October 2009 16:49:57 Jeff Mitchell wrote:
>> Jeff Mitchell wrote:
>> P.S.: You could make a case that whatever dialog is shown when Import
>> Collection is selected should detail what will actually be imported, so
>> that the user is aware of this.
>
> The fact that you call it import collection is the first sign of confusion.  
> Why not call it import statistics like it actually does and fewer will be
> confused?

Yes, that's one good option. (Not for 2.2.1 as we're in string freeze.)

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Jonas Norlander wrote:

> 2009/10/29 Jeff Mitchell <mitchell@...>:
>> Pedro de Carvalho Gomes wrote:
>>> I don't really agree with the use of the collection import for
>>> statistics only. I can enumerate some reasons:
>> It doesn't matter if you agree with it or not. The schema has changed in
>> many fundamental ways and there is not a good way to map everything 1:1.
>>
>>> - I would prefer to import all files from Amarok 1.4 than to fully
>>> rescan my disc because search for files at disc may consume too much
>>> time and my machine start trashing.
>> If scanning makes your machine thrash, fix your machine.
>
> How stupid comment is this?
> He could have a perfect ordinary machine that works for most things,
> it doesn't mean it's something wrong with it. I have a Quad core, 4Gb
> Ram and fast sata2 disks and it still i a pain and take several hours
> to do a full scan of my collection.

The fact that you fail to know what thrashing is doesn't mean the
comment is stupid. There's a large difference between taking time and
thrashing your machine.

http://en.wikipedia.org/wiki/Paging#Thrashing
/proc/sys/vm/swappiness

>>> Also, why would I want to rescan
>>> if I believe I already have a reliable source of information (Amarok
>>> 1.4 database).
>> Because it's not as reliable as you think it is -- at least in the sense
>> that it's not really usable by A2.
>>
>
> In that case it should be mention somewhere that the import of 1.4
> collection is not as usable in Amarok 2 as one would think and I don't
> think that hes alone to think that.

We've already all agreed that that could be made clearer, but thanks
anyways.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

I've understood that metadata from A1 isn't useful to import my
collection to A2. But that's not my point. The fact is that, if I use
the importation tool and my collection is empty (i.e. using A2 for the
first time), it still populates my A2 collection with files from the
SQL query that have a valid URL, which are just a subset from my A1
collection. For example, check this log when then importation tool
tries to import a song from A1 collection.

amarok:    [CUEFILE]:  "/dados/mp3/B/Badfinger/Day After Day.cue"  -
Shoot blindly and missed, searching for other cue files.
amarok:    [CUEFILE]: - Didn't find any matching cue file.
amarok:    possible encoding:  "UTF-8"
amarok:    encoding decoded as UTF-8
amarok:    "Read metadata from file for: Day After Day"
amarok:    185  found track by URL:  "/dados/mp3/B/Badfinger/Day After Day.mp3"
amarok:    185  inserting track:
KUrl("file:///dados/mp3/B/Badfinger/Day After Day.mp3")
amarok:    not a track no match

The line below finds a file in the collection that wasn't there
(remember that A2 db was empty):

Line 182: Meta::TrackPtr track =
CollectionManager::instance()->trackForUrl( KUrl( url ) );

This file will be added to the A2 Local Collection when the method
reaches the line below:

Line 318: location->insertTracks( tracksForInsert );

The consequence is that only a subset of the files from my A1
collection would be imported to the A2 collection. Together with the
"Import Collection" mistake, that's why I first though it would be a
bug and have suggested to change the SQL query, so that I could import
my entire A1 collection.

Pedro

On Wed, Oct 28, 2009 at 9:49 PM, Jeff Mitchell <mitchell@...> wrote:

> Jeff Mitchell wrote:
>> Pedro de Carvalho Gomes wrote:
>>> I don't really agree with the use of the collection import for
>>> statistics only. I can enumerate some reasons:
>>
>> It doesn't matter if you agree with it or not. The schema has changed in
>> many fundamental ways and there is not a good way to map everything 1:1.
>
> By the by. One of the ways it has changed is in the unique identifiers
> used to, err, uniquely identify each track.
>
> To figure out the appropriate A2 identifier each track would need to be
> scanned, since it's based upon contents of the file and the algorithm
> has changed since A1. There is no way to upconvert.
>
> Therefore every file *must* be scanned with the A2 collection scanner
> regardless of whether you've imported every detail of every track or not
> from the A1 database, at which time the scanner can (and should) also
> read fresh details of all the other tags. Therefore there is no point in
> converting all these throwaway details over, no effort will be made to
> convert these, and any patch that implements such behavior will be
> discarded as useless.
>
> QED.
>
> P.S.: You could make a case that whatever dialog is shown when Import
> Collection is selected should detail what will actually be imported, so
> that the user is aware of this.
>
>
>
> _______________________________________________
> Amarok mailing list
> Amarok@...
> https://mail.kde.org/mailman/listinfo/amarok
>
>

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Thanks Guðlaugur.

This "Import Collection" name is just one of my points. The name
mislead me to think that it should import my entire collection. And
the second point is that it indeed imports my A1 collection partially
. I've shown how this happens on my last e-mail.

Anyway, I still believe that maybe "Import Collection" can be used to
import the entire collection. I've understood that all files should be
rescaned, but I think that the valid Urls returned from the SQL query
could be used as the input of the rescan, as an alternative to the
directories chosen at the "Collection Folders" treeview. This way the
import Tool would mark the collection folders. Again, If someone has a
constructive opinion about this, I would really appreciate.

Pedro

2009/10/28 Guðlaugur Jóhannesson <gudlaugu@...>:

> On Wednesday 28 October 2009 16:49:57 Jeff Mitchell wrote:
>> Jeff Mitchell wrote:
>> P.S.: You could make a case that whatever dialog is shown when Import
>> Collection is selected should detail what will actually be imported, so
>> that the user is aware of this.
>
> The fact that you call it import collection is the first sign of confusion.
> Why not call it import statistics like it actually does and fewer will be
> confused?
>
> Gudlaugur
> _______________________________________________
> Amarok mailing list
> Amarok@...
> https://mail.kde.org/mailman/listinfo/amarok
>

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Pedro de Carvalho Gomes wrote:
> Thanks Guðlaugur.
>
> This "Import Collection" name is just one of my points. The name
> mislead me to think that it should import my entire collection. And
> the second point is that it indeed imports my A1 collection partially
> . I've shown how this happens on my last e-mail.
>
> Anyway, I still believe that maybe "Import Collection" can be used to
> import the entire collection.


It's nice that you continue to believe that. But despite your totally
unfounded belief, you're factually wrong. The "partial" imports are
likely due to the importer doing things it shouldn't, which is
populating data with only partial consistency, and the importer should
be fixed to ignore those things. The reason you see partial collections
is because the collection browser is getting bad and inconsistent data
back from queries, not because you're only getting part of the
collection imported and it's magically skipping some tracks.

I honestly don't understand why you won't listen to the actual facts
being told to you by the main Amarok developer working on the SQL
collection -- specifically the scanning and database schema and
insertion parts of it.

> I've understood that all files should be
> rescaned, but I think that the valid Urls returned from the SQL query
> could be used as the input of the rescan, as an alternative to the
> directories chosen at the "Collection Folders" treeview. This way the
> import Tool would mark the collection folders. Again, If someone has a
> constructive opinion about this, I would really appreciate.

You want to be constructive? Fix the importer to not import partial and
unusable fragments of data, i.e. fix it to only import statistics, and
update the dialog that shows when you press Import Collection to not
suggest that it will import track information. Then send a merge request.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Thu, Oct 29, 2009 at 12:20 PM, Jeff Mitchell <mitchell@...> wrote:

> Pedro de Carvalho Gomes wrote:
>> Thanks Guðlaugur.
>>
>> This "Import Collection" name is just one of my points. The name
>> mislead me to think that it should import my entire collection. And
>> the second point is that it indeed imports my A1 collection partially
>> . I've shown how this happens on my last e-mail.
>>
>> Anyway, I still believe that maybe "Import Collection" can be used to
>> import the entire collection.
>
>
> It's nice that you continue to believe that. But despite your totally
> unfounded belief, you're factually wrong. The "partial" imports are
> likely due to the importer doing things it shouldn't, which is
> populating data with only partial consistency, and the importer should
> be fixed to ignore those things. The reason you see partial collections
> is because the collection browser is getting bad and inconsistent data
> back from queries, not because you're only getting part of the
> collection imported and it's magically skipping some tracks.

Man, don't mix my suggestion with the bug. If you can't see, they are
two different things. If you don't agree with my suggestion, ignores
it and try to focus on the bug. I've already admitted that I'm new to
Amarok code, that I've made some confusion about the collection
imported, have explained the reason, and have moved forward. If it
stills bothers you, I'm sorry.

> I honestly don't understand why you won't listen to the actual facts
> being told to you by the main Amarok developer working on the SQL
> collection -- specifically the scanning and database schema and
> insertion parts of it.

What you've told me was that files SHOULD be rescaned because the
unique ID schema depends on data from the files, that are not
available at A1 database. Just it. If you have bothered to read the
message without stones on your hand, you would have seen that I  ask
the feasibility to use the URLs from a query to the the A1 database as
input to the collection scanner, as an alternative to the directories
selected at the treeview. I've stated to rescan each file, NOT to
REUSE any A1 data at all. Those are distinct things.

>> I've understood that all files should be
>> rescaned, but I think that the valid Urls returned from the SQL query
>> could be used as the input of the rescan, as an alternative to the
>> directories chosen at the "Collection Folders" treeview. This way the
>> import Tool would mark the collection folders. Again, If someone has a
>> constructive opinion about this, I would really appreciate.
>
> You want to be constructive? Fix the importer to not import partial and
> unusable fragments of data, i.e. fix it to only import statistics, and
> update the dialog that shows when you press Import Collection to not
> suggest that it will import track information. Then send a merge request.

I really do want to be constructive. I've been very clear about this
since my first to my last e-mails. And all my messages have been quiet
polite and well-toned. I've presented at least three different
situations at them (the partial import bug, my "idiot" suggestion and
the misleading button name). But you've chosen to mock my first
e-mail, and then to attack me because of the suggestion on the other
ones, instead of looking at the big picture.

Anyway, let's try to start over again: I'm new to Amarok code, I've
found a case that causes a bug, and I'd like to help to solve it.
Would you, or anyone one else, help me to fix this?

Cheers,

Pedro

>
> --Jeff
>
>
> _______________________________________________
> Amarok mailing list
> Amarok@...
> https://mail.kde.org/mailman/listinfo/amarok
>
>

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Pedro de Carvalho Gomes wrote:
>> I honestly don't understand why you won't listen to the actual facts
>> being told to you by the main Amarok developer working on the SQL
>> collection -- specifically the scanning and database schema and
>> insertion parts of it.
>
> What you've told me was that files SHOULD be rescanned

No, I said MUST.

> I  ask
> the feasibility to use the URLs from a query to the the A1 database as
> input to the collection scanner, as an alternative to the directories
> selected at the treeview. I've stated to rescan each file, NOT to
> REUSE any A1 data at all. Those are distinct things.

That was very confusingly put the first time:

"I think that the valid Urls returned from the SQL query
could be used as the input of the rescan, as an alternative to the
directories chosen at the "Collection Folders" treeview. This way the
import Tool would mark the collection folders."

Mainly because of the last sentence there. Regardless, the
collectionscanner does not currently allow this. Theoretically it could
be changed to do so, but I prefer to leave it alone -- right now it
works quite stably and rearchitecting both it and the scan result
processor to accept files instead of directories isn't something that
seems worth it for the very few numbers of users it would help,
especially when a simple full rescan fixes the problem anyways. I'm much
more interested in improving scanning performance for all users, which
has been a focus of mine recently.

> I really do want to be constructive. I've been very clear about this
> since my first to my last e-mails.

No, they have not been very clear.

> And all my messages have been quiet
> polite and well-toned.

No, they haven't.

> I've presented at least three different
> situations at them (the partial import bug

It's not a partial import bug, because there's no real import happening.

> , my "idiot" suggestion and

I like that you put the quotes around "idiot", as if to imply that I
called you one. Which I haven't.

> the misleading button name).

And multiple times I've suggested or agreed with ways that this
misleading could be remedied.

> But you've chosen to mock my first
> e-mail,

No, I didn't mock your first e-mail.

> and then to attack me because of the suggestion on the other
> ones

You say "attack", I say "present the facts in an increasingly frustrated
manner".

> , instead of looking at the big picture.

Lulz.

> Anyway, let's try to start over again: I'm new to Amarok code, I've
> found a case that causes a bug, and I'd like to help to solve it.
> Would you, or anyone one else, help me to fix this?

I have already detailed in a very specific fashion exactly how you could
go about fixing this. Instead of following up on this, you ignored it
entirely.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

>>> I honestly don't understand why you won't listen to the actual facts
>>> being told to you by the main Amarok developer working on the SQL
>>> collection -- specifically the scanning and database schema and
>>> insertion parts of it.
>>
>> What you've told me was that files SHOULD be rescanned
>
> No, I said MUST.

I've said "should" because the "collection importer" wasn't supposed
to import files from A1 (as you have mentioned) and it in fact DOES
(see the log I've put at my e-mail  1:00pm UTC), And I can't really
tell for sure if files were rescanned or not. But whatever...

>> I  ask
>> the feasibility to use the URLs from a query to the the A1 database as
>> input to the collection scanner, as an alternative to the directories
>> selected at the treeview. I've stated to rescan each file, NOT to
>> REUSE any A1 data at all. Those are distinct things.
>
> That was very confusingly put the first time:

I've read it again, and I don't thing it was confusing. Anyway,
English is not my primary language, and sorry if I haven't made myself
clear.

> "I think that the valid Urls returned from the SQL query
> could be used as the input of the rescan, as an alternative to the
> directories chosen at the "Collection Folders" treeview. This way the
> import Tool would mark the collection folders."

> Mainly because of the last sentence there. Regardless, the
> collectionscanner does not currently allow this. Theoretically it could
> be changed to do so, but I prefer to leave it alone -- right now it
> works quite stably and rearchitecting both it and the scan result
> processor to accept files instead of directories isn't something that
> seems worth it for the very few numbers of users it would help,
> especially when a simple full rescan fixes the problem anyways. I'm much
> more interested in improving scanning performance for all users, which
> has been a focus of mine recently.

One way to do this it to import the [Collection Folders] section from
A1's ~/.kde/share/config/amarokrc. But I agree that it would take some
work. I wasn't suggesting you to implement it. All I've asked was what
would the be the consequences. Thank you for the opinion, I'll sure
take into account when I decide if I am going to implement it or not.

>> I really do want to be constructive. I've been very clear about this
>> since my first to my last e-mails.
>
> No, they have not been very clear.

I won't argue with that. It is as clear as it can be that I wanted to
help Amarok. I've just asked some feedback about the situation, and
have also sent useful information. (if you didn't receive the mail
with the logs i've sent to the list, I may re-send).

>> And all my messages have been quiet
>> polite and well-toned.
>
> No, they haven't.

Again, it's up to you (and anyone else) to decide it. If you think
I've failed to be polite, I apologize. I wan't my intention. Anyway,
you haven't been polite at all.

>> I've presented at least three different
>> situations at them (the partial import bug
>
> It's not a partial import bug, because there's no real import happening.

Man, I'm telling you there is import: partial, erroneous, real, etc...
whatever. The files are listed on my Local Collection browser, I can
add the to playlist and even play them! Do you want a VNC/SSH account
on my machine, so you can check it with your eyes?

>> , my "idiot" suggestion and
>
> I like that you put the quotes around "idiot", as if to imply that I
> called you one. Which I haven't.

Now, when it seems that finally you've understood what I've proposed,
it doesn't seem idiot anymore. But before, you've been very ironic
about it, although you haven't used the word "idiot".

>> the misleading button name).
>
> And multiple times I've suggested or agreed with ways that this
> misleading could be remedied.

I know you've agreed  with that. It's clear when you replied yesterday
at 11:39 pm (UTC). But the fact that you seemed to be unable to
separate my points led me to exemplify all of them all over again,
just to try to undo the mess.

>> But you've chosen to mock my first
>> e-mail,
>
> No, I didn't mock your first e-mail.

Try to look for those phrases on your  e-mails at this thread. They
are just samples. If there is a need, I can get some more (and there
are the personal mails also):
- We have a truck!?!?!
- It doesn't matter if you agree with it or not.
- If scanning makes your machine thrash, fix your machine.

>> and then to attack me because of the suggestion on the other
>> ones
>
> You say "attack", I say "present the facts in an increasingly frustrated
> manner".

I've understood all your points from the first time they were
presented. But you've failed to understand mine (which I have agreed
that may have been my fault). But it doesn't give you any right to be
rude.

>> , instead of looking at the big picture.
>
> Lulz.

Whatever. I've already shown my points.

>> Anyway, let's try to start over again: I'm new to Amarok code, I've
>> found a case that causes a bug, and I'd like to help to solve it.
>> Would you, or anyone one else, help me to fix this?
>
> I have already detailed in a very specific fashion exactly how you could
> go about fixing this. Instead of following up on this, you ignored it
> entirely.

No, I didn't ignore them. I'll really take them into account when I'll
try to fix it. Specially you bid that the problem is caused by
corrupted data.

Thank you for the useful information

Pedro

> --Jeff
>
>
> _______________________________________________
> Amarok mailing list
> Amarok@...
> https://mail.kde.org/mailman/listinfo/amarok
>
>
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Pedro de Carvalho Gomes wrote:
> One way to do this it to import the [Collection Folders] section from
> A1's ~/.kde/share/config/amarokrc.

It would be a nice thing to add the ability to the importer to import
the collection folders from a given 1.4 amarokrc. Note that you can't
rely on ~/.kde vs. ~/.kde4 as it's distribution-dependent, so you'd need
to have the user specify the location of their 1.4 amarokrc. (Although
this won't achieve your original objective, it will at least make it
nicer for the 1.4->2.2 user to get started.)

If you do this, take a look at what the code from the settings dialog
does when you select directories, and imitate that behavior.

> But I agree that it would take some
> work. I wasn't suggesting you to implement it. All I've asked was what
> would the be the consequences. Thank you for the opinion, I'll sure
> take into account when I decide if I am going to implement it or not.

Please keep in mind what I've said before: that if you attempt to modify
the import behavior to remove the need for scanning the files, your
patch is likely to be rejected. There are other things that go on during
scanning besides just populating the database, and most of these have
changed their behavior in A2, and need a rescan to get things sane
again. Things like compilation/various artists detection, file tracking
updates, and so on.

It's not really a worthwhile goal, especially since rescanning is
something that will have to be done periodically anyways (for instance,
the upcoming 2.2.1 will require a full rescan upon first startup). It's
not that we want to force the user to do full rescans over and over, but
sometimes it's necessary -- for instance, when the db schema changes in
a major way. If we sometimes find it necessary to force full rescans on
even minor updates, I don't see why anyone should have a serious problem
with it when there is a major update.

Better to just change the text to not claim to import tracks, fix the
importer to not try to (incorrectly) import tracks, and be done with it.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Here goes my fix for the "partial population" issue. I've tested it
with my Amarok 1.4 db and it has worked fine. I've made the same
changes for the iTunes importation, but I didn't have a valid database
to test it.

Basically I've fixed the erroneous test to check if
CollectionManager::instance()->trackForUrl( KUrl( url ) ) return zero.
This means that the URL is invalid, not that the URL is about a song
that is not at the collection. Now, to import statistics for a song
the URL must both be valid and the song should be at collection. Also
I've removed the store and insertion of valid URLs with the
CollectionLocation::insertStatistics and
CollectionLocation::insertTracks methods. Such call were present at
both Amarok and iTunes importation code.

Finally, I've seen this code at collection/CollectionLocation.h:
191:        /**
192:         * Inserts a set of TrackPtrs directly into the database
without needing to actuall move any files
193:         * This is a hack required by the DatabaseImporter
194:         */
195:        virtual void insertTracks( const QMap<Meta::TrackPtr,
QString> &trackMap ) { Q_UNUSED( trackMap ); }
196:       virtual void insertStatistics( const QMap<Meta::TrackPtr,
QString> &trackMap ) { Q_UNUSED( trackMap ); }

I guess it would be that case to remove both methods. They are
implemented only at
collection/sqlcollection/SqlCollectionLocation.cpp. But I'm not 100%
confident, because both methods are called only once, at
SqlCollectionLocation::slotJobFinished( KJob *job ).

I'd appreciate any comments on this.

Pedro
On Thu, Oct 29, 2009 at 7:26 PM, Jeff Mitchell <mitchell@...> wrote:

> Pedro de Carvalho Gomes wrote:
>> One way to do this it to import the [Collection Folders] section from
>> A1's ~/.kde/share/config/amarokrc.
>
> It would be a nice thing to add the ability to the importer to import
> the collection folders from a given 1.4 amarokrc. Note that you can't
> rely on ~/.kde vs. ~/.kde4 as it's distribution-dependent, so you'd need
> to have the user specify the location of their 1.4 amarokrc. (Although
> this won't achieve your original objective, it will at least make it
> nicer for the 1.4->2.2 user to get started.)
>
> If you do this, take a look at what the code from the settings dialog
> does when you select directories, and imitate that behavior.
>
>> But I agree that it would take some
>> work. I wasn't suggesting you to implement it. All I've asked was what
>> would the be the consequences. Thank you for the opinion, I'll sure
>> take into account when I decide if I am going to implement it or not.
>
> Please keep in mind what I've said before: that if you attempt to modify
> the import behavior to remove the need for scanning the files, your
> patch is likely to be rejected. There are other things that go on during
> scanning besides just populating the database, and most of these have
> changed their behavior in A2, and need a rescan to get things sane
> again. Things like compilation/various artists detection, file tracking
> updates, and so on.
>
> It's not really a worthwhile goal, especially since rescanning is
> something that will have to be done periodically anyways (for instance,
> the upcoming 2.2.1 will require a full rescan upon first startup). It's
> not that we want to force the user to do full rescans over and over, but
> sometimes it's necessary -- for instance, when the db schema changes in
> a major way. If we sometimes find it necessary to force full rescans on
> even minor updates, I don't see why anyone should have a serious problem
> with it when there is a major update.
>
> Better to just change the text to not claim to import tracks, fix the
> importer to not try to (incorrectly) import tracks, and be done with it.
>
> --Jeff
>
>
> _______________________________________________
> Amarok mailing list
> Amarok@...
> https://mail.kde.org/mailman/listinfo/amarok
>
>

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok