I need *your* benchmarking help!

No apologies for the cross-post  :-)

Over the last few days I've been working extremely hard -- just ask my
poor neglected wife -- on solving one of the most longstanding issues in
A2 -- and, for that matter, A1 -- scanning performance.

I've done all kinds of tweaks in the amarokcollectionscanner binary
itself (
http://blog.jefferai.org/2009/10/14/speed-never-gets-old-at-least-in-software-1129
) but the problem remained that, when it came down to it, we were still
just accessing the database too damn much.

So, I finally bit the bullet and did what me and Leo had figured a while
back was the only way to solve this problem -- replace all SQL queries
in the middle of the scan with batch queries at the front and back and a
series of hashes with types like
QHash<int, QLinkedList<QStringList*> *>. In other words, the
ScanResultProcessor now writes a bunch of code to populate the hashes
with SQL at the start, then uses *only* the hashes during all of the
"inserts" and queries, and then writes out all the hashes to SQL at the
end. While maintaining cache coherency the whole way. If I didn't mess
up. Which I didn't. I think. Pretty sure. Possibly.

If you didn't understand that, don't worry. All you have to know is that
it was an absolute fuckload of work, and now I need some help
determining whether or not it was all worth it, which means I need
people benchmarking.

If you'd like to help out -- and please, do help out -- you'll need a a
large enough collection that a normal full scan takes a noticeable
amount of time, and you'll need to be running Amarok built from Git.
Here's what you do:

1) Update master and build. Open Amarok and run two full rescans
(Settings->Collection->Fully Rescan Collection). So click, let it run
until the progress bar reaches 100%, then click again. The second time,
time it with a stopwatch or some such thing. (The reason it's done twice
is so that the effects of disk caching can be reasonably ignored between
this version and the new one.)

2) Add my clone as a remote (use Google if you need help). My clone is
at git://gitorious.org/~jefferai/amarok/jefferai-work.git and the branch
you want is called "uidhash". Build it.

3) Run two full rescans again, timing the second one.

4) Close Amarok and re-open it.*

5) If you see any oddities (that aren't fixed by switching back to
master and rebuilding, then running a full rescan, then closing and
opening Amarok -- yes, all those steps) please be sure to report them
along with your benchmark results.

Many thanks in advance to those that help out.
--Jeff

* The reason you need to close and reopen Amarok is that there are
longstanding bugs in the collection browser that cause it to not be
updated properly when new data is scanned. So it can *look* like your
collection is messed up when what's really happening is that the browser
is using bad cached data. Since the browser reloads from SQL every time
you open Amarok, closing and opening Amarok ensures that you're seeing
the browser without these bugs interfering, so you can actually see
whether or not something is truly not working right.



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Thu, 22 Oct 2009 22:37:57 -0400, Jeff Mitchell <mitchell@...>
wrote:
> <snip>
> Many thanks in advance to those that help out.
> --Jeff

Hi Jeff,

I'd be happy to help testing. I'm eager to test your changes to see if
they improve the collection scanner, although I can't give it a shot
earlier then tomorrow morning (Saturday 24 oct). I hope that's not a
problem?

I think I can divide your test into 2 sub-cases;

1. I can do the test with my new, current and mostly correctly tagged
collection. I know what to expect and what I want to see (which is pretty
much what I am seeing now) :). That means that I can give good feedback on
the fact how the new scanner works. This collection is relatively small,
about 3000 songs. If I have to make a guess it currently takes about a
minute or two to process the lot.

2. A test with all the songs I can find. This one would be just one to
test the benchmarking since the tags will me most likely screwed up. This
would end up in a collection of around 10.000 songs (if not, more).

Please let me know if I can skip one of the two (especially #2 since it
requires me to dig up old backup CD's/DVD's).

The PC I'll be using is quite new but not high end. Please let me know if
you need some specs of it.

--
Met vriendelijke groet / With kind regards,
Eelko Berkenpies
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Eelko Berkenpies wrote:

> On Thu, 22 Oct 2009 22:37:57 -0400, Jeff Mitchell <mitchell@...>
> wrote:
>> <snip>
>> Many thanks in advance to those that help out.
>> --Jeff
>
> Hi Jeff,
>
> I'd be happy to help testing. I'm eager to test your changes to see if
> they improve the collection scanner, although I can't give it a shot
> earlier then tomorrow morning (Saturday 24 oct). I hope that's not a
> problem?
>
> I think I can divide your test into 2 sub-cases;
>
> 1. I can do the test with my new, current and mostly correctly tagged
> collection. I know what to expect and what I want to see (which is pretty
> much what I am seeing now) :). That means that I can give good feedback on
> the fact how the new scanner works. This collection is relatively small,
> about 3000 songs. If I have to make a guess it currently takes about a
> minute or two to process the lot.
>
> 2. A test with all the songs I can find. This one would be just one to
> test the benchmarking since the tags will me most likely screwed up. This
> would end up in a collection of around 10.000 songs (if not, more).
>
> Please let me know if I can skip one of the two (especially #2 since it
> requires me to dig up old backup CD's/DVD's).

The first one is fine  :-)

Any scan that is long enough for you to actually time it should be good.
Larger collections are better since they're more likely to spend more
time (in master) sitting doing SQL queries, but one that takes a minute
or two should be just fine. Plus, being able to spot if anything is
wrong (it really shouldn't be, as I did extensive testing for
correctness of this result vs. the SQL method during development) is a
bonus.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

> 1) Update master and build. Open Amarok and run two full rescans
> (Settings->Collection->Fully Rescan Collection). So click, let it run
> until the progress bar reaches 100%, then click again. The second time,
> time it with a stopwatch or some such thing. (The reason it's done twice
> is so that the effects of disk caching can be reasonably ignored between
> this version and the new one.)

Shouldn't we be able to do this without the GUI? By running the
collectionscanner ourselves?

Casey
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Casey Link wrote:
>> 1) Update master and build. Open Amarok and run two full rescans
>> (Settings->Collection->Fully Rescan Collection). So click, let it run
>> until the progress bar reaches 100%, then click again. The second time,
>> time it with a stopwatch or some such thing. (The reason it's done twice
>> is so that the effects of disk caching can be reasonably ignored between
>> this version and the new one.)
>
> Shouldn't we be able to do this without the GUI? By running the
> collectionscanner ourselves?

No, because the improvements aren't in the scanner. They're in the scan
results processor.

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Simon Bühler]

hi,

i'd like to test but it seem i'm unable to handle the git checkout stuff:

my buildscript is: (part of it)

---
PRODUCT="$1"
cd $HOME/kde/src/$PRODUCT                                            
git pull                                                              
mkdir -p $HOME/kde/build/$PRODUCT                                    
cd $HOME/kde/build/$PRODUCT                                          
cmake -DCMAKE_INSTALL_PREFIX=$HOME/kde -DCMAKE_BUILD_TYPE=debugfull
$HOME/kde/src/$PRODUCT/
make install
---

now i tried

clone git://gitorious.org/~jefferai/amarok/jefferai-work.git uidhash

Initialized empty Git repository in /home/user/kde/src/uidhash/.git/
remote: Counting objects: 195929, done.
remote: Compressing objects: 100% (40144/40144), done.
remote: Total 195929 (delta 156915), reused 194091 (delta 155247)
Receiving objects: 100% (195929/195929), 59.26 MiB | 701 KiB/s, done.
Resolving deltas: 100% (156915/156915), done.

warning: remote HEAD refers to nonexistent ref, unable to checkout

also a

git clone git://gitorious.org/~jefferai/amarok/jefferai-work.git  jeffs_stuff

resulted in the same error

any hints please?
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Simon Bühler wrote:

> hi,
>
> i'd like to test but it seem i'm unable to handle the git checkout stuff:
>
> my buildscript is: (part of it)
>
> ---
> PRODUCT="$1"
> cd $HOME/kde/src/$PRODUCT                                            
> git pull                                                              
> mkdir -p $HOME/kde/build/$PRODUCT                                    
> cd $HOME/kde/build/$PRODUCT                                          
> cmake -DCMAKE_INSTALL_PREFIX=$HOME/kde -DCMAKE_BUILD_TYPE=debugfull
> $HOME/kde/src/$PRODUCT/
> make install
> ---
>
> now i tried
>
> clone git://gitorious.org/~jefferai/amarok/jefferai-work.git uidhash
>
> Initialized empty Git repository in /home/user/kde/src/uidhash/.git/
> remote: Counting objects: 195929, done.
> remote: Compressing objects: 100% (40144/40144), done.
> remote: Total 195929 (delta 156915), reused 194091 (delta 155247)
> Receiving objects: 100% (195929/195929), 59.26 MiB | 701 KiB/s, done.
> Resolving deltas: 100% (156915/156915), done.
>
> warning: remote HEAD refers to nonexistent ref, unable to checkout
>
> also a
>
> git clone git://gitorious.org/~jefferai/amarok/jefferai-work.git  jeffs_stuff
>
> resulted in the same error
>
> any hints please?

I'm not sure I understand what the problem is. Does git branch -a not
show the uidhash branch?

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Sat, 24 Oct 2009 11:04:49 -0400, Jeff Mitchell <mitchell@...>
wrote:
> Simon Bühler wrote:
>> hi,
>>
>> i'd like to test but it seem i'm unable to handle the git checkout
stuff:

>>
>> my buildscript is: (part of it)
>>
>> ---
>> PRODUCT="$1"
>> cd $HOME/kde/src/$PRODUCT                                            
>> git pull                                                              
>> mkdir -p $HOME/kde/build/$PRODUCT                                    
>> cd $HOME/kde/build/$PRODUCT                                          
>> cmake -DCMAKE_INSTALL_PREFIX=$HOME/kde -DCMAKE_BUILD_TYPE=debugfull
>> $HOME/kde/src/$PRODUCT/
>> make install
>> ---
>>
>> now i tried
>>
>> clone git://gitorious.org/~jefferai/amarok/jefferai-work.git uidhash
>>
>> Initialized empty Git repository in /home/user/kde/src/uidhash/.git/
>> remote: Counting objects: 195929, done.
>> remote: Compressing objects: 100% (40144/40144), done.
>> remote: Total 195929 (delta 156915), reused 194091 (delta 155247)
>> Receiving objects: 100% (195929/195929), 59.26 MiB | 701 KiB/s, done.
>> Resolving deltas: 100% (156915/156915), done.
>>
>> warning: remote HEAD refers to nonexistent ref, unable to checkout
>>
>> also a
>>
>> git clone git://gitorious.org/~jefferai/amarok/jefferai-work.git
>> jeffs_stuff
>>
>> resulted in the same error
>>
>> any hints please?
>
> I'm not sure I understand what the problem is. Does git branch -a not
> show the uidhash branch?
>
> --Jeff

I'm having the exact same problem and markey confirmed it for me (branch
-a does show all a list of branches within jefferai-work btw). But there's
simply no content at all (except for a .git folder).

--
Met vriendelijke groet / With kind regards,
Eelko Berkenpies
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Sun, Oct 25, 2009 at 11:27 AM, Eelko Berkenpies <fedora@...> wrote:
> I'm having the exact same problem and markey confirmed it for me (branch
> -a does show all a list of branches within jefferai-work btw). But there's
> simply no content at all (except for a .git folder).

Actually it works fine, never mind. The whole thing was a misunderstanding.

--
Mark Kretschmann
Amarok Developer
www.kde.org - amarok.kde.org
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Sat, 24 Oct 2009 13:21:34 +0200, Simon Bühler
<simon@...> wrote:
> hi,
>
> i'd like to test but it seem i'm unable to handle the git checkout
stuff:

>
> my buildscript is: (part of it)
>
> ---
> PRODUCT="$1"
> cd $HOME/kde/src/$PRODUCT                                            
> git pull                                                              
> mkdir -p $HOME/kde/build/$PRODUCT                                    
> cd $HOME/kde/build/$PRODUCT                                          
> cmake -DCMAKE_INSTALL_PREFIX=$HOME/kde -DCMAKE_BUILD_TYPE=debugfull
> $HOME/kde/src/$PRODUCT/
> make install
> ---
>
> now i tried
>
> clone git://gitorious.org/~jefferai/amarok/jefferai-work.git uidhash
>
> Initialized empty Git repository in /home/user/kde/src/uidhash/.git/
> remote: Counting objects: 195929, done.
> remote: Compressing objects: 100% (40144/40144), done.
> remote: Total 195929 (delta 156915), reused 194091 (delta 155247)
> Receiving objects: 100% (195929/195929), 59.26 MiB | 701 KiB/s, done.
> Resolving deltas: 100% (156915/156915), done.
>
> warning: remote HEAD refers to nonexistent ref, unable to checkout
>
> also a
>
> git clone git://gitorious.org/~jefferai/amarok/jefferai-work.git
> jeffs_stuff
>
> resulted in the same error
>
> any hints please?

By courtesy of markey;

- git clone git://gitorious.org/~jefferai/amarok/jefferai-work.git
- git checkout -b uidhash --track remotes/origin/uidhash

So, thanks again markey!

As already stated in an other mail, it was all a misunderstanding and
everything works fine. :)

--
Met vriendelijke groet / With kind regards,
Eelko Berkenpies
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

On Thu, 22 Oct 2009 22:37:57 -0400, Jeff Mitchell <mitchell@...>
wrote:
> 1) Update master and build. Open Amarok and run two full rescans
> (Settings->Collection->Fully Rescan Collection). So click, let it run
> until the progress bar reaches 100%, then click again. The second time,
> time it with a stopwatch or some such thing. (The reason it's done twice
> is so that the effects of disk caching can be reasonably ignored between
> this version and the new one.)

The scan took somewhere around 12 seconds on the second scan. This is less
than the initial 1 or 2 minutes I thought it would take. It takes that long
when I wipe my database (which makes it an initial scan).

> 2) Add my clone as a remote (use Google if you need help). My clone is
> at git://gitorious.org/~jefferai/amarok/jefferai-work.git and the branch
> you want is called "uidhash". Build it.
>
> 3) Run two full rescans again, timing the second one.

The scan literally took half the time and finished in 6-7 seconds. I was
not able to time it any closer than this because I couldn't find a
stopwatch of some sort (I used my watch). On a side note, processing an
initial scan seems to be a little bit faster with this build. I don't if
that's possible and / or if that's interesting?
 
> 4) Close Amarok and re-open it.*
>
> 5) If you see any oddities (that aren't fixed by switching back to
> master and rebuilding, then running a full rescan, then closing and
> opening Amarok -- yes, all those steps) please be sure to report them
> along with your benchmark results.

A problem I have discovered is that some leafs don't want to fold out in
the collection browser using your build. Does that make sense? What I mean
is when I try to unfold an album to see it's content it will display a "-"
in front of the album but it's content will not appear. Works fine in GIT
master on the same album(s).

I hope this helps you out. :)

--
Met vriendelijke groet / With kind regards,
Eelko Berkenpies
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Tycho Andersen]

On Thu, 22 Oct 2009, Jeff Mitchell wrote:

> 2) Add my clone as a remote (use Google if you need help). My clone is
> at git://gitorious.org/~jefferai/amarok/jefferai-work.git and the branch
> you want is called "uidhash". Build it.

When I run the code in the uidhash branch, I get the dreaded:

[ERROR!] MySQL library initialization failed.

However, with git master, everything works fine. My mysql is a self
built 5.1.37. Any ideas as to why this might be? Full log available
here: http://files.tycho.ws/tmp/amarok.log

\t
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Jeff,

here are my benchmarking results (3904 tracks, 418 folders):

* amarok2 master

- scan time: 04:16.4

* amarok2 jefferai-work.git->uidhash

- scan time: 04:01.0

hope this helps.
--

Regards,

Andreas

---

www - andreas-stangl@...

on Friday 23 October 2009 at 02:37:57, Jeff Mitchell <mitchell@...> wrote:

> No apologies for the cross-post :-)

>

> Over the last few days I've been working extremely hard -- just ask my

> poor neglected wife -- on solving one of the most longstanding issues in

> A2 -- and, for that matter, A1 -- scanning performance.

>

> I've done all kinds of tweaks in the amarokcollectionscanner binary

> itself (

> http://blog.jefferai.org/2009/10/14/speed-never-gets-old-at-least-in-softwa

> re-1129 ) but the problem remained that, when it came down to it, we were

> still just accessing the database too damn much.

>

> So, I finally bit the bullet and did what me and Leo had figured a while

> back was the only way to solve this problem -- replace all SQL queries

> in the middle of the scan with batch queries at the front and back and a

> series of hashes with types like

> QHash<int, QLinkedList<QStringList*> *>. In other words, the

> ScanResultProcessor now writes a bunch of code to populate the hashes

> with SQL at the start, then uses *only* the hashes during all of the

> "inserts" and queries, and then writes out all the hashes to SQL at the

> end. While maintaining cache coherency the whole way. If I didn't mess

> up. Which I didn't. I think. Pretty sure. Possibly.

>

> If you didn't understand that, don't worry. All you have to know is that

> it was an absolute fuckload of work, and now I need some help

> determining whether or not it was all worth it, which means I need

> people benchmarking.

>

> If you'd like to help out -- and please, do help out -- you'll need a a

> large enough collection that a normal full scan takes a noticeable

> amount of time, and you'll need to be running Amarok built from Git.

> Here's what you do:

>

> 1) Update master and build. Open Amarok and run two full rescans

> (Settings->Collection->Fully Rescan Collection). So click, let it run

> until the progress bar reaches 100%, then click again. The second time,

> time it with a stopwatch or some such thing. (The reason it's done twice

> is so that the effects of disk caching can be reasonably ignored between

> this version and the new one.)

>

> 2) Add my clone as a remote (use Google if you need help). My clone is

> at git://gitorious.org/~jefferai/amarok/jefferai-work.git and the branch

> you want is called "uidhash". Build it.

>

> 3) Run two full rescans again, timing the second one.

>

> 4) Close Amarok and re-open it.*

>

> 5) If you see any oddities (that aren't fixed by switching back to

> master and rebuilding, then running a full rescan, then closing and

> opening Amarok -- yes, all those steps) please be sure to report them

> along with your benchmark results.

>

> Many thanks in advance to those that help out.

> --Jeff

>

> * The reason you need to close and reopen Amarok is that there are

> longstanding bugs in the collection browser that cause it to not be

> updated properly when new data is scanned. So it can *look* like your

> collection is messed up when what's really happening is that the browser

> is using bad cached data. Since the browser reloads from SQL every time

> you open Amarok, closing and opening Amarok ensures that you're seeing

> the browser without these bugs interfering, so you can actually see

> whether or not something is truly not working right.

>

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

> When I run the code in the uidhash branch, I get the dreaded:
>
> [ERROR!] MySQL library initialization failed.
>
> However, with git master, everything works fine.

Ditto. I'm having the same problem. Checked out the branch, compiled..
b0rked mysql.
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Simon Bühler]

ok with the instructions i was able to build jeffs branch, here are my
findings:

i timed the time from button pressed until the progressbar starts at 0% (where
in master the sql output begins) and the time needed to complete to 100%.

master:
Collection has 20896 tracks

phase 1: 2:34 minutes
phase2: 13:03 minutes

ther were a lot of

 [ERROR!] GREPME MySQL query failed! Duplicate entry 'Wayne wonder' for key 2  
on  "INSERT INTO artists_temp( name ) VALUES ('Wayne wonder');"
[ERROR!] GREPME MySQL query failed! Duplicate entry 'A Little Deeper-94' for
key 2  on  "INSERT INTO albums_temp( artist, name ) VALUES ( 94, 'A Little
Deeper');"          

errors.

jeffs branch:

i got this at startup:

amarok:                 [PluginManager] Trying to load:  "amarok_collection-
mysqlservercollection"                                                                      
amarok:                 [PluginManager]                                                                                                                                  
"                " PluginManager Service Info:                                                                                                                          
"                " ---------------------------                                                                                                                          
"                " name                          : "MySQLServer-Sammlung"                                                                                                
"                " library                       : "amarok_collection-
mysqlservercollection"                                                                            
"                " desktopEntryPath              : "amarok_collection-
mysqlservercollection.desktop"                                                                    
"                " X-KDE-Amarok-plugintype       : "collection"                                                                                                          
"                " X-KDE-Amarok-name             : "mysqlserver-collection"                                                                                              
"                " X-KDE-Amarok-authors          : ("Jeff Mitchell")                                                                                                    
"                " X-KDE-Amarok-rank             : "100"                                                                                                                
"                " X-KDE-Amarok-version          : "1"                                                                                                                  
"                " X-KDE-Amarok-framework-version: "49"                                                                                                                  

amarok:               END__: static Amarok::Plugin*
PluginManager::createFromService(KSharedPtr<KService>) - Took 0.035s
amarok:               [CollectionManager] Initialising sqlcollection                                                    
amarok:               BEGIN: ScanManager::ScanManager(SqlCollection*)                                                    
amarok:               END__: ScanManager::ScanManager(SqlCollection*) - Took
0.00013s                                    
amarok:               BEGIN:
MySqlServerCollection::MySqlServerCollection(const QString&, const QString&)                
amarok:                  Automatic reconnect successfully activated                                                      
amarok:                  Automatic reconnect successfully activated                                                      
amarok:                  Connected to MySQL server 5.0.70-log                                                            
amarok:                  Connected to MySQL server 5.0.70-log                                                            
amarok:                  Initialized thread, count== 1                                                                  
amarok:                 BEGIN: void DatabaseUpdater::update()                                                            
amarok:                    Database version:  9                                                                          
amarok:                    Database out of date: database version is 9 ,
current version is 10                          
amarok:                   BEGIN: void DatabaseUpdater::upgradeVersion9to10()                                            
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'jamendo_albums'  on  "DROP TABLE jamendo_albums"
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'jamendo_artists'  on  "DROP TABLE jamendo_artists"
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'jamendo_genre'  on  "DROP TABLE jamendo_genre"    
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'jamendo_tracks'  on  "DROP TABLE jamendo_tracks"  
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'magnatune_albums'  on  "DROP TABLE magnatune_albums"
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'magnatune_artists'  on  "DROP TABLE magnatune_artists"
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'magnatune_genre'  on  "DROP TABLE magnatune_genre"    
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'magnatune_moods'  on  "DROP TABLE magnatune_moods"    
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'magnatune_tracks'  on  "DROP TABLE magnatune_tracks"  
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'opmldirectory_albums'  on  "DROP TABLE opmldirectory_albums"
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'opmldirectory_artists'  on  "DROP TABLE opmldirectory_artists"
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'opmldirectory_genre'  on  "DROP TABLE opmldirectory_genre"    
amarok:                      [ERROR!] GREPME MySQL query failed! Unknown table
'opmldirectory_tracks'  on  "DROP TABLE opmldirectory_tracks"  
amarok:                      [ERROR!] GREPME MySQL query failed! Duplicate key
name 'devices_rshare'  on  "CREATE INDEX devices_rshare ON devices(
servername, sharename );"                                                                                                                                                                      
amarok:                      [ERROR!] GREPME MySQL query failed! Duplicate key
name 'lyrics_url'  on  "CREATE UNIQUE INDEX lyrics_url ON lyrics(url);"                  
amarok:                      [ERROR!] GREPME MySQL query failed! Duplicate key
name 'urls_id_rpath'  on  "CREATE UNIQUE INDEX urls_id_rpath ON urls(deviceid,
rpath);"  
amarok:                      [ERROR!] GREPME MySQL query failed! Duplicate key
name 'stats_tag_name_artist_album'  on  "CREATE UNIQUE INDEX
stats_tag_name_artist_album ON statistics_tag(name,artist,album)"                                                                                                                                    
amarok:                   END__: void DatabaseUpdater::upgradeVersion9to10() -
DELAY Took (quite long) 14s                                                              
amarok:                 END__: void DatabaseUpdater::update() - DELAY Took
(quite long) 14s          


the times:
Collection has 20891 tracks
phase 1: 25 sec
phase 2: 13:36 min


so jeffs version finds 5 tracks less and needs ~14mins, master 15:30mins

maybe this is caused by the longstanding "slow disk access under heavy load on
amd64 systems" bug i'm still experencing to some extends

_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Eelko Berkenpies wrote:
> A problem I have discovered is that some leafs don't want to fold out in
> the collection browser using your build. Does that make sense? What I mean
> is when I try to unfold an album to see it's content it will display a "-"
> in front of the album but it's content will not appear. Works fine in GIT
> master on the same album(s).
>
> I hope this helps you out. :)

Yeah, there's a problem with Various Artists I have to look into, but it
seems to manifest itself that way.

Thanks for the results!

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Simon Bühler wrote:

> ok with the instructions i was able to build jeffs branch, here are my
> findings:
>
> i timed the time from button pressed until the progressbar starts at 0% (where
> in master the sql output begins) and the time needed to complete to 100%.
>
> master:
> Collection has 20896 tracks
>
> phase 1: 2:34 minutes
> phase2: 13:03 minutes
>
> ther were a lot of
>
>  [ERROR!] GREPME MySQL query failed! Duplicate entry 'Wayne wonder' for key 2  
> on  "INSERT INTO artists_temp( name ) VALUES ('Wayne wonder');"
> [ERROR!] GREPME MySQL query failed! Duplicate entry 'A Little Deeper-94' for
> key 2  on  "INSERT INTO albums_temp( artist, name ) VALUES ( 94, 'A Little
> Deeper');"          

Yep -- this is in fact something that the new branch should fix since it
updates the database to be case-sensitive on the backend. You're
probably seeing these errors because you have the same artist or album
names with different capitalization.

> errors.
>
> jeffs branch:
>
> i got this at startup:

<snip>

You can ignore those errors, it was just trying to drop tables that you
didn't have, which is okay.

> the times:
> Collection has 20891 tracks
> phase 1: 25 sec
> phase 2: 13:36 min
>
>
> so jeffs version finds 5 tracks less and needs ~14mins, master 15:30mins

Interesting. Can you figure out which one is correct, i.e. were you
getting extraneous tracks before or missing five now?

> maybe this is caused by the longstanding "slow disk access under heavy load on
> amd64 systems" bug i'm still experencing to some extends

What bug is that?

Thanks for the help,
Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Casey Link wrote:
>> When I run the code in the uidhash branch, I get the dreaded:
>>
>> [ERROR!] MySQL library initialization failed.
>>
>> However, with git master, everything works fine.
>
> Ditto. I'm having the same problem. Checked out the branch, compiled..
> b0rked mysql.

Please try again -- I've reverted one of the two changes made to the
library being initialized. It seems like they might have removed an
option in 5.1.

If that still doesn't work, open up mysql CLI and type "show collation;"
and check that utf8_bin is present. (If it isn't, bug your packager.)

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

Jeff Mitchell wrote:

> Eelko Berkenpies wrote:
>> A problem I have discovered is that some leafs don't want to fold out in
>> the collection browser using your build. Does that make sense? What I mean
>> is when I try to unfold an album to see it's content it will display a "-"
>> in front of the album but it's content will not appear. Works fine in GIT
>> master on the same album(s).
>>
>> I hope this helps you out. :)
>
> Yeah, there's a problem with Various Artists I have to look into, but it
> seems to manifest itself that way.
>
> Thanks for the results!

The problem with Various Artists should be solved, if you don't mind
trying again (although it will be a bit slower than it was due to some
other code I have in there while debugging).

--Jeff



_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok

[Tycho Andersen]

On Sun, 25 Oct 2009, Jeff Mitchell wrote:

> Please try again -- I've reverted one of the two changes made to the
> library being initialized. It seems like they might have removed an
> option in 5.1.

That fixes it here, thanks. Here's my data:

In a collection with 11,883 tracks ~160 GB in size:

git master:

Behavior: I press the full rescan button and Amarok pegs the cpu and
says 100% completion. Then after about 3 minutes, the percent bar goes
from 0-100%. Total times are below (I did 5 runs just to make sure
nothing was wacky).

Scan # |  time
1      | 4:49
2      | 4:13
3      | 4:12
4      | 4:11
5      | 4:09

jeffrai-work/uidhash:

Here, the first phase takes about 20 seconds or so (percent bar still
says
100%).

Scan # |  time
1      | 3:05
2      | 1:35
3      | 1:25
4      | 1:25
5      | 1:20

So far, everything looks good in the collections (the numbers scanned
match, etc.). One minor issue: the "Search collection" box is now case
sensitive.

\t
_______________________________________________
Amarok mailing list
Amarok@...
https://mail.kde.org/mailman/listinfo/amarok