JRuby vs MRI - Petstore shootout

[Alexey Verkhovsky-2]

Hi, all,

I've been doing some quick and dirty shootouts running the Petstore
app from TW-commons RubyForge project.

The purpose of this exercise was not to draw any conclusions, but to
get a gut feel for the various factors that have significant impact on
performance of a Rails application running under JRuby.

Test procedure, in a nutshell was just repeatedly hitting a particular
page and measuring response rate. More details below.

I see a huge difference in favor of MRI so far. I am also a bit
surprised that AR-JDBC is slower than a pure-Ruby MySQL driver from
Rails.

Thoughts?

MRI / Mongrel / native mysql driver: 71 requests/sec
MRI / Mongrel / Rails mysql driver: 57 requests/sec
MRI / Webrick / native mysql driver: 52 requests/sec
MRI / Webrick / Rails mysql driver: 43 requests/sec

JRuby trunk / Mongrel / Rails mysql driver: 17 requests/sec
JRuby trunk / Mongrel / AR:JDBC:  14 requests/sec
JRuby trunk / Webrick / Rails mysql driver: 13 requests/sec
JRuby trunk / Webrick / AR:JDBC: 11 requests/sec

JRuby 1.0 / Mongrel / Rails mysql driver: 16 requests/sec
JRuby 1.0 / Mongrel / AR:JDBC: 14 requests/sec
JRuby 1.0 / Webrick / Rails mysql driver: 13 requests/sec
JRuby 1.0 / Webrick / AR:JDBC: 12 requests/sec

In all cases, even with MRI, performance is completely CPU-bound, with
the app using 97-99% of CPU, and MySQL only 1-2%.

System under test:

* Petstore: SVN rev 22 from http://tw-commons.rubyforge.org/svn/trunk/
* Mongrel-jruby: 1.0.1
* Mongrel: 1.0.1
* Rails MySQL driver: pure Ruby driver included in Rails 1.2.3
* native MySQL driver: 2.7
* ActiveRecord-JDBC: 0.4
* JDBC MySQL driver: MySQL Connector/J, packaged in Ubuntu as
libmysql-java 5.0.4+dfsg-2
* Ruby: ruby 1.8.5 (2007-03-13 patchlevel 35) [i686-linux]
* JRuby trunk: SVN rev 4183
* Java: build 1.6.0-b105
* MySQL: 5.0.38
* OS: Ubuntu 7.0 desktop
* Hardware: Dell Latitude D620 laptop

Test procedure:

* rake RAILS_ENV=production db:dataload (loads some small amount of test data)
* Start the app
* Warm up the JIT compiler:
  ab -c 1 -n 1000 http://localhost:3002/shop/viewProduct.shtml?product=K9-DL-01
* Test:
  ab -c 1 -n 100 http://localhost:3002/shop/viewProduct.shtml?product=K9-DL-01

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Charles Oliver Nutter-2]

Alexey Verkhovsky wrote:

> Hi, all,
>
> I've been doing some quick and dirty shootouts running the Petstore
> app from TW-commons RubyForge project.
>
> The purpose of this exercise was not to draw any conclusions, but to
> get a gut feel for the various factors that have significant impact on
> performance of a Rails application running under JRuby.
>
> Test procedure, in a nutshell was just repeatedly hitting a particular
> page and measuring response rate. More details below.
>
> I see a huge difference in favor of MRI so far. I am also a bit
> surprised that AR-JDBC is slower than a pure-Ruby MySQL driver from
> Rails.

A couple things to consider...

1. File IO is still rather slow in JRuby, and petstore is configured for
normal file-based sessions. Switching to memory-based sessions would
help quite a bit. Of course it would help MRI too, but it should help us
more.
2. ab doesn't maintain sessions between requests, so you're spinning up
a new session for each request. That combined with the slow file IO is
probably a big part of our slowness. Try using httperf instead.
3. Again, slow file IO means turning off logging helps too.
4. User Java 6 server VM for  best results
5. Turn off ObjectSpace

Here's my unscientific numbers with petstore, httperf, and the changes
above:

All on JRuby trunk, on my MacBook Pro 2.16ghz, one core disabled

[requesting /]
java 6 flat, webrick, rails mysql driver: 23.6 req/s (42.5 ms/req)
java 6 server, -O, webrick, rails mysql driver: 23.9 req/s (41.8 ms/req)

The fact that these numbers are so similar with such a drastic change in
runtime parameters makes me think there's a non-CPU bottleneck
somewhere, like IO. Here's mongrel numbers:

java 6 server, -O, mongrel, rails mysql driver: 88.9 req/s (11.3 ms/req)

I don't have current AR-JDBC handy, so I'll leave it to others to do a
full retest. Here's Ruby numbers on my machine:

MRI, webrick, rails mysql driver: 39.7 req/s (25.2 ms/req)
MRI, mongrel, rails mysql driver: 171.5 req/s (5.8 ms/req)

Here's my httperf line:

httperf --wsess 1,1000,0 --server localhost --port 3000 --rate=0

So, with a few tweaks JRuby's something less than twice as slow for this
test. Here's a page with data on it, presumably coming from the database:

java 6 server, -O, mongrel: 32.3 req/s (30.9 ms/req)
MRI, mongrel: 76.5 req/s (13.1 ms/req)

Still about 50%. httperf line:

httperf --wsess 1,1000,0 --server localhost --uri
/shop/viewCategory.shtml?category=fish --port 3000 --rate=0

Running these tests with the app deployed as a WAR would be interesting
to see too. Still, we obviously have more work to do to get
mongrel/webrick execution up to par. As with most things, I'd guess it
comes down to two things:

- IO performance
- regexp performance (used for request processing)

If others want to duplicate these results and try a few other tweaks, it
would be nice to see.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 8/18/07, Charles Oliver Nutter <charles.nutter@...> wrote:
> A couple things to consider...

Thanks Charles. Silly me, forgetting about file-based sessions...

So, round 2 is coming:

* Database-driven sessions
* With/without ObjectSpace
* With/without logging - I will be quite surprised if writing out 5
lines of text makes any difference in a 75 msec operation, but hey,
being surprised is the whole point of this exercise
* server VM

I want to create a new session per request, and keep it all nicely
single-threaded and synchronous for the time being. Things are easier
to interpret and profile this way. That's why I'm using ab. When get
that 8-cores box, we'll start doing something more interesting.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

Round 2.

Updated JRuby to rev 4185.
Updated Petstore to rev 25.

Created 10000 sessions both in database and file systems (so that
timings are not affected too much because of growing number of
sessions).

OBSERVATIONS
* A huge difference is made by turning off logging.
* Somewhat smaller, but still quite noticeable difference is made by
disabling ObjectSpace.
* server VM makes very little difference to performance (when we
measure, JIT is warmed up)
* file vs database sessions make no visible difference to JRuby, but
only because there aren't many enough sessions. With 6-digit number of
sessions, file-based sessions are a well-known performance killer.

BASELINE (MRI / Mongrel / native MySQL driver)
With file-based sessions : 69 req/sec
With database sessions : 88 req/sec
With disabled logging: 95 requests/sec

JRUBY (trunk / Mongrel / Rails MySQL driver)
Run like yesterday: 17 requests/sec
With database sessions: 17 requests/sec
Turned off ObjectSpace (jruby -O): 21 requests/sec
Server VM (jruby -O -J-server): 22 requests/sec
With disabled logging: 35 requests/sec

JRUBY with AR:JDBC (trunk / Mongrel / AR:JDBC MySQL driver)
Run like yesterday: 14 requests/sec
With database sessions: 14 requests/sec
Turned off ObjectSpace (jruby -O): 16 requests/sec
Server VM (jruby -O -J-server): 16 requests/sec
With disabled logging: 33 requests/sec

* To turn off logging, add RAILS_DEFAULT_LOGGER.level = Logger::FATAL
to config/environments/production.rb

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

Talked with Jon Tirsen about this, and we agreed that the next thing
to try would be running the same test under Jetty instead of Mongrel.
Which may or may not make a non-trivial difference, we'll see. We just
need to make rails-integration / JRubyWorks compatible with the trunk
JRuby first.

Another thing I am going to try in the next round is generating a much
larger dataset, so that Petstore has to display more data on the page.

Does anybody have any more suggestions? To remind, I'm looking for
factors that have a large impact on JRuby Rails performance.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Ola Bini-3]

Alexey Verkhovsky wrote:
> Talked with Jon Tirsen about this, and we agreed that the next thing
> to try would be running the same test under Jetty instead of Mongrel.
> Which may or may not make a non-trivial difference, we'll see. We just
> need to make rails-integration / JRubyWorks compatible with the trunk
> JRuby first.
>  
Rails-integration IS compatible with trunk. You just need to use the
goldspike-snapshot plugin, instead of goldspike.

> Another thing I am going to try in the next round is generating a much
> larger dataset, so that Petstore has to display more data on the page.
>
> Does anybody have any more suggestions? To remind, I'm looking for
> factors that have a large impact on JRuby Rails performance.
>
>  

Cheers

--
 Ola Bini (http://ola-bini.blogspot.com)
 JRuby Core Developer
 Developer, ThoughtWorks Studios (http://studios.thoughtworks.com)

 "Yields falsehood when quined" yields falsehood when quined.



---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 8/21/07, Ola Bini <ola.bini@...> wrote:
> > We just need to make rails-integration / JRubyWorks compatible with the trunk
> > JRuby first.
> Rails-integration IS compatible with trunk. You just need to use the
> goldspike-snapshot plugin, instead of goldspike.

OK, change of wording. I have to make it all play together on my rig. :)

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

I've hacked together enough of JRubyWorks to run Petstore under Jetty with it.

JRubyWorks is yet another way to run a Rails app under servlet
container, so far completely experimental... we are talking "don't
touch it with 10 feet pole" kind of experimental, so there, you've
been warned :)

Jetty proved to be much faster than Mongrel-jruby (to the tune of 10
msec CPU time saved per hit), but still way slower than Mongrel + MRI.
The numbers are below.

JRuby rev 4246

BASELINE (MRI / Mongrel / native MySQL driver / DB sessions)
logging enabled: 89 req/sec
logging disabled: 92 req/sec

BASELINE JRUBY (JRuby / Mongrel / Ruby MySQL driver / DB sessions)
logging enabled: 21 req/sec
logging disabled: 32 req/sec

JETTY (ObjectSpace enabled)
logging enabled: 23 req/sec
logging disabled  30 req/sec

JETTY (ObjectSpace disabled)
logging enabled: 29 req/sec
logging disabled: 45 req/sec (!)

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Charles Oliver Nutter-2]

Alexey Verkhovsky wrote:

> BASELINE (MRI / Mongrel / native MySQL driver / DB sessions)
> logging enabled: 89 req/sec
> logging disabled: 92 req/sec
>
> BASELINE JRUBY (JRuby / Mongrel / Ruby MySQL driver / DB sessions)
> logging enabled: 21 req/sec
> logging disabled: 32 req/sec
>
> JETTY (ObjectSpace enabled)
> logging enabled: 23 req/sec
> logging disabled  30 req/sec
>
> JETTY (ObjectSpace disabled)
> logging enabled: 29 req/sec
> logging disabled: 45 req/sec (!)

Hey, not bad...certainly getting there. I agree with (!)...45s is
comfortable at 50% perf, and stunningly faster than with logging
enabled. Pretty miserable IO performance there. It's likely that similar
IO bottlenecks exist for sockets, which would certainly explain to some
extent overall poor performance.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 8/30/07, Charles Oliver Nutter <charles.nutter@...> wrote:
> > JETTY (ObjectSpace disabled)
> > logging enabled: 29 req/sec
> > logging disabled: 45 req/sec (!)

> It's likely that similar
> IO bottlenecks exist for sockets, which would certainly explain to some
> extent overall poor performance.

Jetty scenario with logging turned off, obviously, involves
practically no I/O at JRuby level at all. I've tried to profile it,
and saw no obvious bottlenecks in core classes. Most of the CPU time
seems to consumed within the JRuby runtime stuff (ThreadContext,
RubyObject and such).

By the way, Petstore is a very simple Rails application. I'm going to
do some comparisons with a couple of much bigger apps TW is building,
we'll see how that stacks up. I won't be able to publish it, like
Petstore, but I'll try to at least write up the general observations.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Thomas E Enebo]

On 8/31/07, Alexey Verkhovsky <alexey.verkhovsky@...> wrote:

> On 8/30/07, Charles Oliver Nutter <charles.nutter@...> wrote:
> > > JETTY (ObjectSpace disabled)
> > > logging enabled: 29 req/sec
> > > logging disabled: 45 req/sec (!)
>
> > It's likely that similar
> > IO bottlenecks exist for sockets, which would certainly explain to some
> > extent overall poor performance.
>
> Jetty scenario with logging turned off, obviously, involves
> practically no I/O at JRuby level at all. I've tried to profile it,
> and saw no obvious bottlenecks in core classes. Most of the CPU time
> seems to consumed within the JRuby runtime stuff (ThreadContext,
> RubyObject and such).
>
> By the way, Petstore is a very simple Rails application. I'm going to
> do some comparisons with a couple of much bigger apps TW is building,
> we'll see how that stacks up. I won't be able to publish it, like
> Petstore, but I'll try to at least write up the general observations.

Alexy, how many times are you hitting these urls again?  You said 1000
at the top of this thread.  Can you try hitting like 100,000 to get an
idea if hotspot starts warming more.    Perhaps a start before you
leave work sort of test?

-Tom

--
Blog: http://www.bloglines.com/blog/ThomasEEnebo
Email: enebo@... , tom.enebo@...

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Charles Oliver Nutter-2]

Alexey Verkhovsky wrote:
> * rake RAILS_ENV=production db:dataload (loads some small amount of test data)
> * Start the app
> * Warm up the JIT compiler:
>   ab -c 1 -n 1000 http://localhost:3002/shop/viewProduct.shtml?product=K9-DL-01
> * Test:
>   ab -c 1 -n 100 http://localhost:3002/shop/viewProduct.shtml?product=K9-DL-01

Tom mentioned this, but I just noticed it too...1000 requests isn't
enough to warm up hotspot with the server VM. I've typically done a
minimum of 10k requests to get it really cranking before I record any
numbers. Tom's 100k is sure to do it.

Basically the HotSpot server VM doesn't optimize methods until they're
hit something like 10k times. That's why it's really best for server
applications; it takes a long time to warm up. The client VM will warm
up faster, but it will never get as fast.

Java 7 promises to merge the two.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 8/31/07, Thomas E Enebo <tom.enebo@...> wrote:
> Can you try hitting like 100,000 to get an
> idea if hotspot starts warming more.

Sure.

Actually, most time-consuming methods (from profiler point of view)
get 5- or 6-digit number of hits during a thousand page hits, anyway.
I'll test it though.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 9/4/07, Alexey Verkhovsky <alexey.verkhovsky@...> wrote:
> > Can you try hitting like 100,000 to get an
> > idea if hotspot starts warming more.
> Sure.
>
> Actually, most time-consuming methods (from profiler point of view)
> get 5- or 6-digit number of hits during a thousand page hits, anyway.
> I'll test it though.

So, I did. After the first thousand hits, all performance metrics
improve by about 4% in the next 700 hits, after which they didn't
change at all.

Conclusions:

1. The right number number of page hits to warm up the JIT compiler in
Rails benchmarks is 2000.

2. This information doesn't really change the big picture portrayed
earlier in this thread.

As a sidenote, I've done a similar quick-n-dirty Petstore shootout
with MRI+Rails vs lean and mean Java (Clinton Begin's JPetstore 5.0).
The latter is ~3 times faster.

I don't have a burning desire right now to install a full blown J2EE
server and deploy Sun Blueprints classic on it :), but from what I
remember, this makes MRI+Rails approximately as fast as J2EE. So (as I
always said) if J2EE was fast/scalable enough for The Enterprise 8
years ago (despite the much slower hardware), surely Rails is
fast/scalable enough for it now.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

Some new numbers.

SUMMARY

In Petstore script/benchmark_action:
  * trunk runs two times faster than 1.0.1
  * there seems to be a significant bottleneck somewhere in executing
the Rails framework
  * when framework processing is excluded, JRuby is about 1.5 times
slower than MRI
  * Unlike earlier test runs, AR-JDBC in the current trunk has a
noticeable advantage over pure-Ruby MySQL driver

DETAILS

Ola and yours truly are on the next iteration of JRuby performance
tuning for Rails. I wrote a couple of scripts to drill down into
various parts of Rails, to decide where to focus.

The scripts, essentially, measure the average roundtrip time of
app.get('url'), controller.send(action) and controller.render(), by
1000 iterations of each block, 10 runs.

JRuby JIT compiler is warmed up by 10000 iterations of each test prior
to measurements. The numbers reported below are from the 10th run.
Variation between last 5 runs in all of these tests is under 5%, so
it's doesn't really matter that it's not an average.

Note that there is no web server, and no session handling involved in
these tests.
Also, note that the relative amount of database / model work involved
in CategoriesController#show is rather small.

Here are the numbers:

Runtime MRI Jruby Jruby 1.0.1
DB driver native pure Ruby AR-JDBC pure Ruby AR-JDBC
                                       
Controller 1.1 2.0 1.7 2.5 7.19
View 4.5 4.4 6.7 6.6 17.08
Framework 2.7 3.5 9.3 9.5 11.8
Full action 8.3 9.9 17.7 18.6 36.03

JRuby 1.0.1 / AR-JDBC:
controller :   1.867000   0.000000   1.867000 (  1.867000)


TEST SETUP AND PROCEDURE

Hardware: Dell D620
OS: Ubuntu 7.04
Kernel: 2.6.20-16-generic #2 SMP Fri Aug 31 00:55:27 UTC 2007 i686
GNU/Linux, default from the distro.
Ruby: ruby 1.8.5 (2007-03-13 patchlevel 35) [i686-linux]
JRuby: trunk rev 4384
Database: MySQL 5.0.38 (binary from the distro)
Application: Petstore from
http://tw-commons.rubyforge.org/svn/petstore/trunk/ (rev 177).
Measuring:
  Ruby: echo 'load "script/benchmark_action"' | ruby script/console production
  JRuby: echo 'load "script/benchmark_action"' | jruby
-J-Djruby.objectspace.enabled=false script/console production
      # replace production with ar_jdbc for running with AR:JDBC driver

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 9/24/07, Alexey Verkhovsky <alexey.verkhovsky@...> wrote:
> Some new numbers.

Attaching an OpenOffice spreadsheet.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Charles Oliver Nutter-2]

Alexey Verkhovsky wrote:
> In Petstore script/benchmark_action:
>   * trunk runs two times faster than 1.0.1

That fits with most of my measurements of compiled versus interpreted
code. There's more gains to be had in the compiler, but it seems like a
rough doubling of performance at the moment.

>   * there seems to be a significant bottleneck somewhere in executing
> the Rails framework

That's the feeling I have too...something's broken somewhere. If simpler
scripts across a variety of libraries and syntax are in general much
faster than MRI, there's likely a fixable bottleneck somewhere slowing
down Rails.

>   * when framework processing is excluded, JRuby is about 1.5 times
> slower than MRI

Hmm interesting. I'm guessing this is going to turn up as missing
optimizations (or broken implementations) in core classes. That's the
next area we need to focus on for performance after the compiler is
complete.

>   * Unlike earlier test runs, AR-JDBC in the current trunk has a
> noticeable advantage over pure-Ruby MySQL driver

That's good to hear. Perhaps it's because the remaining Ruby code in
AR-JDBC is now compiling, eliminating that as a primary bottleneck.

I want to have the compiler done by the end of the week, and then I'll
start launching into core-class-related benchmarking and bottlenecks. If
you guys work from the top down and I work from the bottom up, we can
meet in the middle.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Nick Sieger-2]

On 9/25/07, Alexey Verkhovsky <alexey.verkhovsky@...> wrote:
>   * Unlike earlier test runs, AR-JDBC in the current trunk has a
> noticeable advantage over pure-Ruby MySQL driver

Can you drill down into this further? I'm having trouble seeing the
difference from your table below.  Could be the formatting, but the
way I'm reading it it appears that JRuby trunk + mysql.rb is 9.9 for
the full action while JRuby trunk + ar-jdbc is 17.7? Is that correct?

>
>
> Here are the numbers:
>
> Runtime MRI             Jruby           Jruby 1.0.1
> DB driver       native  pure Ruby       AR-JDBC pure Ruby       AR-JDBC
>
> Controller      1.1     2.0     1.7     2.5     7.19
> View    4.5     4.4     6.7     6.6     17.08
> Framework       2.7     3.5     9.3     9.5     11.8
> Full action     8.3     9.9     17.7    18.6    36.03
>
> JRuby 1.0.1 / AR-JDBC:
> controller :   1.867000   0.000000   1.867000 (  1.867000)

This is good stuff Alexey, thanks for sharing it!

/Nick

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alexey Verkhovsky-2]

On 9/25/07, Nick Sieger <nicksieger@...> wrote:
> On 9/25/07, Alexey Verkhovsky <alexey.verkhovsky@...> wrote:
> I'm having trouble seeing the difference from your table below.

Can you open the spreadsheet I sent in a follow up message?

If not, the first two columns are for MRI - with native and pure Ruby
MySQL driver, respectively. Next two columns are for JRuby trunk -
with AR-JDBC and pure Ruby drivers. Last column is JRuby 1.0.1 with
AR-JDBC driver.

So, pure Ruby vs AR-JDBC is 18.6 vs 17.7 msec/hit. Controller line is
almost entirely database work, and there we look at 2.5 vs 1.7 msec
difference.

I am planning to drill further down once I get a JProfiler license
(should be today or tomorrow).

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Christian Seiler]

I also ran the petstore benchmark a couple of times today with
different setup options. The petstore app as it is in SVN does some
heavy logging, so I decided not only to test JRuby 1.0.1 but also
the latest fixes-1_0 branch (with slow IO being fixed).

The setup procedure was like Alexey described in one of his previous
postings. I tested the ar-jdbc version only.

My results:

1.0.x
controller :   5.729000   0.000000   5.729000 (  5.729000)
view       :  21.572000   0.000000  21.572000 ( 21.572000)
full action:  42.043000   0.000000  42.043000 ( 42.043000)


1.0.x -J-Djruby.jit.enabled=false
controller :   5.337000   0.000000   5.337000 (  5.337000)
view       :  19.633000   0.000000  19.633000 ( 19.633000)
full action:  38.506000   0.000000  38.506000 ( 38.506000)

TRUNK
controller :   5.427000   0.000000   5.427000 (  5.428000)
view       :  18.653000   0.000000  18.653000 ( 18.653000)
full action:  50.560000   0.000000  50.560000 ( 50.560000)

TRUNK -J-Djruby.jit.enabled=false
controller :   6.671000   0.000000   6.671000 (  6.671000)
view       :  18.512000   0.000000  18.512000 ( 18.512000)
full action:  49.191000   0.000000  49.191000 ( 49.192000)

1.0.1
controller :   8.909000   0.000000   8.909000 (  8.909000)
view       :  25.409000   0.000000  25.409000 ( 25.409000)
full action:  52.643000   0.000000  52.643000 ( 52.643000)

1.0.1 -J-Djruby.jit.enabled=false
controller :   8.618000   0.000000   8.618000 (  8.618000)
view       :  23.495000   0.000000  23.495000 ( 23.495000)
full action:  50.521000   0.000000  50.521000 ( 50.521000)


and some benchmarks using server Hotspot:

TRUNK -J-server
controller :   5.428000   0.000000   5.428000 (  5.428000)
view       :  17.659000   0.000000  17.659000 ( 17.659000)
full action:  43.340000   0.000000  43.340000 ( 43.340000)

TRUNK -J-server -J-Djruby.jit.enabled=false
controller :   5.636000   0.000000   5.636000 (  5.636000)
view       :  13.643000   0.000000  13.643000 ( 13.643000)
full action:  36.770000   0.000000  36.770000 ( 36.770000)

1.0.x -J-server
controller :   5.415000   0.000000   5.415000 (  5.415000)
view       :  18.243000   0.000000  18.243000 ( 18.243000)
full action:  38.209000   0.000000  38.209000 ( 38.209000)

1.0.x -J-server -J-Djruby.jit.enabled=false
controller :   4.578000   0.000000   4.578000 (  4.578000)
view       :  15.037000   0.000000  15.037000 ( 15.037000)
full action:  32.002000   0.000000  32.002000 ( 32.002000)

MRI:
controller :   2.010000   0.100000   2.110000 (  2.248452)
view       :   9.420000   0.730000  10.150000 ( 10.183790)
full action:  17.250000   1.120000  18.370000 ( 19.341900)


=> my (partially weird?) results:
1. 1.0.x is faster than trunk
2. 1.0.1 is slower than trunk (probably due to slow file IO)
3. most tests run faster without JIT (only controller part seems
to be faster with JIT)!!
4. fastest JRuby setup (1.0.x / nojit / -server): about 50% of MRI performance

Any idea? Trunk faster than 1.0.x and JIT causing a slowdown??
Is there a special issue with my hardware (maybe Hotspot / CPU related)?

My setup (Ubuntu this time):
Hardware: IBM Thinkpad R40, an older one (Intel(R) Pentium(R) M processor 1300MHz, 1024MB cache, 1199.90 bogomips)
OS: Ubuntu 7.04
Kernel: 2.6.20-16-generic #2 SMP Sun Sep 23 19:50:39 UTC 2007 i686 GNU/Linux
Ruby: ruby 1.8.5 (2006-08-25) [i486-linux]
JRuby: trunk rev 4384, fixes-1_0 rev 4385
Database: MySQL 5.0.38
Application: Petstore rev 177

Alexey Verkhovsky-2 wrote:

Some new numbers.

SUMMARY

In Petstore script/benchmark_action:
  * trunk runs two times faster than 1.0.1
  * there seems to be a significant bottleneck somewhere in executing
the Rails framework
  * when framework processing is excluded, JRuby is about 1.5 times
slower than MRI
  * Unlike earlier test runs, AR-JDBC in the current trunk has a
noticeable advantage over pure-Ruby MySQL driver

DETAILS

Ola and yours truly are on the next iteration of JRuby performance
tuning for Rails. I wrote a couple of scripts to drill down into
various parts of Rails, to decide where to focus.

The scripts, essentially, measure the average roundtrip time of
app.get('url'), controller.send(action) and controller.render(), by
1000 iterations of each block, 10 runs.

JRuby JIT compiler is warmed up by 10000 iterations of each test prior
to measurements. The numbers reported below are from the 10th run.
Variation between last 5 runs in all of these tests is under 5%, so
it's doesn't really matter that it's not an average.

Note that there is no web server, and no session handling involved in
these tests.
Also, note that the relative amount of database / model work involved
in CategoriesController#show is rather small.

Here are the numbers:

Runtime MRI Jruby Jruby 1.0.1
DB driver native pure Ruby AR-JDBC pure Ruby AR-JDBC
                                       
Controller 1.1 2.0 1.7 2.5 7.19
View 4.5 4.4 6.7 6.6 17.08
Framework 2.7 3.5 9.3 9.5 11.8
Full action 8.3 9.9 17.7 18.6 36.03

JRuby 1.0.1 / AR-JDBC:
controller :   1.867000   0.000000   1.867000 (  1.867000)


TEST SETUP AND PROCEDURE

Hardware: Dell D620
OS: Ubuntu 7.04
Kernel: 2.6.20-16-generic #2 SMP Fri Aug 31 00:55:27 UTC 2007 i686
GNU/Linux, default from the distro.
Ruby: ruby 1.8.5 (2007-03-13 patchlevel 35) [i686-linux]
JRuby: trunk rev 4384
Database: MySQL 5.0.38 (binary from the distro)
Application: Petstore from
http://tw-commons.rubyforge.org/svn/petstore/trunk/ (rev 177).
Measuring:
  Ruby: echo 'load "script/benchmark_action"' | ruby script/console production
  JRuby: echo 'load "script/benchmark_action"' | jruby
-J-Djruby.objectspace.enabled=false script/console production
      # replace production with ar_jdbc for running with AR:JDBC driver

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email