FF3.1beta3 in Peacekeeper performance test

[Mike Cramer]

Hi,

I follow the impressive performance improvements of FF very closely, but
today I saw a new performance test which I could not believe.
http://www.futuremark.com/pressroom/pressreleases/55300/

"..Peacekeeper measures your browser's performance by testing its
JavaScript functionality. ..."

The following features are tested:
http://service.futuremark.com/peacekeeper/faq.action

As result they show this
http://www.heise.de/mobil/Neuer-Benchmark-fuer-Internetbrowser--/zoom/134567/0

which means that FF 3.1b3 is only ~1/2 as fast as safari 4.0

I tried this test on my mac pro (2,4ghz, 4gb) and I got the following
results:
Safari 3.2.1   791points
FF3.1beta3     592points

Has anyone an explaination why FF3.1 performs so poor in this test,
because sofar I saw only tests where FF3.1 was only faster or fast as
chrome.

Regards,
Mike
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Boris Zbarsky]

Mike Cramer wrote:
> "..Peacekeeper measures your browser's performance by testing its
> JavaScript functionality. ..."

That's not what it tests.

> The following features are tested:
> http://service.futuremark.com/peacekeeper/faq.action

Right.

> As result they show this
> http://www.heise.de/mobil/Neuer-Benchmark-fuer-Internetbrowser--/zoom/134567/0 

Which is semi-meaningless, since they don't ever explain how they
compute those numbers.

 > which means that FF 3.1b3 is only ~1/2 as fast as safari 4.0

No, it means their scoring function, which they never explain, assigned
it half as high a score.  What that means in practice depends on what
that function is.  If you can get them to tell you that, I'd love to
know the results.

> I tried this test on my mac pro (2,4ghz, 4gb) and I got the following
> results:
> Safari 3.2.1   791points
> FF3.1beta3     592points

Which means what?

> Has anyone an explaination why FF3.1 performs so poor in this test

It's pretty easy to write tests on which FF3.1 performs worse than
Safari 4 (e.g. dromaeo).  If you want me to explain numbers that are
being computed by some hidden formula, I think you're out of luck. ;)

For what it's worth, I spent a few hours on Friday trying to get this
test suite into a shape where I could run one test at a time (and hence
profile the tests to see where we spend time).  I gave up when I
couldn't even get any of the tests but the array tests from their web
server: either they send different tests for the same URI based on
cookies, or their code is cleverly obfuscated beyond my ability to
comprehend.  Or both.

Given that they don't publish sub-scores for the parts of the benchmark,
don't provide their benchmark's tests in a way that can be read (much
less run individually), and don't explain how their final score is
computed from the sub-scores, the credibility of these folks is about 0
for me right now.  They could just be making all the numbers up using a
random number generator; I have no way to tell.

-Boris
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Boris Zbarsky]

Mike Cramer wrote:
> Has anyone an explaination why FF3.1 performs so poor in this test,
> because sofar I saw only tests where FF3.1 was only faster or fast as
> chrome.

Oh, and the key here is that you must have been looking at JavaScript
performance tests, whereas this thing is mostly not testing performance
of the JavaScript language itself.

-Boris
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance