|
View:
New views
16 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 | Next > |
|
|
Re: JRuby vs MRI - Petstore shootout-O in those tests as well? ObjectSpace hugely skews these numbers....
-Tom On 9/25/07, Christian Seiler <chr.seiler@...> wrote: > > I also ran the petstore benchmark a couple of times today with > different setup options. The petstore app as it is in SVN does some > heavy logging, so I decided not only to test JRuby 1.0.1 but also > the latest fixes-1_0 branch (with slow IO being fixed). > > The setup procedure was like Alexey described in one of his previous > postings. I tested the ar-jdbc version only. > > My results: > > 1.0.x > controller : 5.729000 0.000000 5.729000 ( 5.729000) > view : 21.572000 0.000000 21.572000 ( 21.572000) > full action: 42.043000 0.000000 42.043000 ( 42.043000) > > > 1.0.x -J-Djruby.jit.enabled=false > controller : 5.337000 0.000000 5.337000 ( 5.337000) > view : 19.633000 0.000000 19.633000 ( 19.633000) > full action: 38.506000 0.000000 38.506000 ( 38.506000) > > TRUNK > controller : 5.427000 0.000000 5.427000 ( 5.428000) > view : 18.653000 0.000000 18.653000 ( 18.653000) > full action: 50.560000 0.000000 50.560000 ( 50.560000) > > TRUNK -J-Djruby.jit.enabled=false > controller : 6.671000 0.000000 6.671000 ( 6.671000) > view : 18.512000 0.000000 18.512000 ( 18.512000) > full action: 49.191000 0.000000 49.191000 ( 49.192000) > > 1.0.1 > controller : 8.909000 0.000000 8.909000 ( 8.909000) > view : 25.409000 0.000000 25.409000 ( 25.409000) > full action: 52.643000 0.000000 52.643000 ( 52.643000) > > 1.0.1 -J-Djruby.jit.enabled=false > controller : 8.618000 0.000000 8.618000 ( 8.618000) > view : 23.495000 0.000000 23.495000 ( 23.495000) > full action: 50.521000 0.000000 50.521000 ( 50.521000) > > > and some benchmarks using server Hotspot: > > TRUNK -J-server > controller : 5.428000 0.000000 5.428000 ( 5.428000) > view : 17.659000 0.000000 17.659000 ( 17.659000) > full action: 43.340000 0.000000 43.340000 ( 43.340000) > > TRUNK -J-server -J-Djruby.jit.enabled=false > controller : 5.636000 0.000000 5.636000 ( 5.636000) > view : 13.643000 0.000000 13.643000 ( 13.643000) > full action: 36.770000 0.000000 36.770000 ( 36.770000) > > 1.0.x -J-server > controller : 5.415000 0.000000 5.415000 ( 5.415000) > view : 18.243000 0.000000 18.243000 ( 18.243000) > full action: 38.209000 0.000000 38.209000 ( 38.209000) > > 1.0.x -J-server -J-Djruby.jit.enabled=false > controller : 4.578000 0.000000 4.578000 ( 4.578000) > view : 15.037000 0.000000 15.037000 ( 15.037000) > full action: 32.002000 0.000000 32.002000 ( 32.002000) > > MRI: > controller : 2.010000 0.100000 2.110000 ( 2.248452) > view : 9.420000 0.730000 10.150000 ( 10.183790) > full action: 17.250000 1.120000 18.370000 ( 19.341900) > > > => my (partially weird?) results: > 1. 1.0.x is faster than trunk > 2. 1.0.1 is slower than trunk (probably due to slow file IO) > 3. most tests run faster without JIT (only controller part seems > to be faster with JIT)!! > 4. fastest JRuby setup (1.0.x / nojit / -server): about 50% of MRI > performance > > Any idea? Trunk faster than 1.0.x and JIT causing a slowdown?? > Is there a special issue with my hardware (maybe Hotspot / CPU related)? > > My setup (Ubuntu this time): > Hardware: IBM Thinkpad R40, an older one (Intel(R) Pentium(R) M processor > 1300MHz, 1024MB cache, 1199.90 bogomips) > OS: Ubuntu 7.04 > Kernel: 2.6.20-16-generic #2 SMP Sun Sep 23 19:50:39 UTC 2007 i686 GNU/Linux > Ruby: ruby 1.8.5 (2006-08-25) [i486-linux] > JRuby: trunk rev 4384, fixes-1_0 rev 4385 > Database: MySQL 5.0.38 > Application: Petstore rev 177 > > > > Alexey Verkhovsky-2 wrote: > > > > Some new numbers. > > > > SUMMARY > > > > In Petstore script/benchmark_action: > > * trunk runs two times faster than 1.0.1 > > * there seems to be a significant bottleneck somewhere in executing > > the Rails framework > > * when framework processing is excluded, JRuby is about 1.5 times > > slower than MRI > > * Unlike earlier test runs, AR-JDBC in the current trunk has a > > noticeable advantage over pure-Ruby MySQL driver > > > > DETAILS > > > > Ola and yours truly are on the next iteration of JRuby performance > > tuning for Rails. I wrote a couple of scripts to drill down into > > various parts of Rails, to decide where to focus. > > > > The scripts, essentially, measure the average roundtrip time of > > app.get('url'), controller.send(action) and controller.render(), by > > 1000 iterations of each block, 10 runs. > > > > JRuby JIT compiler is warmed up by 10000 iterations of each test prior > > to measurements. The numbers reported below are from the 10th run. > > Variation between last 5 runs in all of these tests is under 5%, so > > it's doesn't really matter that it's not an average. > > > > Note that there is no web server, and no session handling involved in > > these tests. > > Also, note that the relative amount of database / model work involved > > in CategoriesController#show is rather small. > > > > Here are the numbers: > > > > Runtime MRI Jruby Jruby 1.0.1 > > DB driver native pure Ruby AR-JDBC pure Ruby AR-JDBC > > > > Controller 1.1 2.0 1.7 2.5 7.19 > > View 4.5 4.4 6.7 6.6 17.08 > > Framework 2.7 3.5 9.3 9.5 11.8 > > Full action 8.3 9.9 17.7 18.6 36.03 > > > > JRuby 1.0.1 / AR-JDBC: > > controller : 1.867000 0.000000 1.867000 ( 1.867000) > > > > > > TEST SETUP AND PROCEDURE > > > > Hardware: Dell D620 > > OS: Ubuntu 7.04 > > Kernel: 2.6.20-16-generic #2 SMP Fri Aug 31 00:55:27 UTC 2007 i686 > > GNU/Linux, default from the distro. > > Ruby: ruby 1.8.5 (2007-03-13 patchlevel 35) [i686-linux] > > JRuby: trunk rev 4384 > > Database: MySQL 5.0.38 (binary from the distro) > > Application: Petstore from > > http://tw-commons.rubyforge.org/svn/petstore/trunk/ (rev 177). > > Measuring: > > Ruby: echo 'load "script/benchmark_action"' | ruby script/console > > production > > JRuby: echo 'load "script/benchmark_action"' | jruby > > -J-Djruby.objectspace.enabled=false script/console production > > # replace production with ar_jdbc for running with AR:JDBC driver > > > > -- > > Alexey Verkhovsky > > CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com] > > RubyWorks [http://rubyworks.thoughtworks.com] > > > > --------------------------------------------------------------------- > > To unsubscribe from this list please visit: > > > > http://xircles.codehaus.org/manage_email > > > > > > > > -- > View this message in context: http://www.nabble.com/JRuby-vs-MRI---Petstore-shootout-tf4289470.html#a12884516 > Sent from the JRuby - Dev mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > -- Blog: http://www.bloglines.com/blog/ThomasEEnebo Email: enebo@... , tom.enebo@... --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutChristian Seiler wrote:
> I also ran the petstore benchmark a couple of times today with > different setup options. The petstore app as it is in SVN does some > heavy logging, so I decided not only to test JRuby 1.0.1 but also > the latest fixes-1_0 branch (with slow IO being fixed). > > The setup procedure was like Alexey described in one of his previous > postings. I tested the ar-jdbc version only. Your results are truly puzzling! I presume Alexey's numbers left JIT at default settings, and he showed a rough doubling of performance. Your numbers not only show trunk being slower, but compiled mode being slower than JIT mode as well. Something's not right. Can you try running the various benchmarks in test/bench to see what sort of results you get with trunk versus 1.0.x? I don't have any good explanation for what you're seeing right now. - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutCharles Oliver Nutter wrote:
> Christian Seiler wrote: >> I also ran the petstore benchmark a couple of times today with >> different setup options. The petstore app as it is in SVN does some >> heavy logging, so I decided not only to test JRuby 1.0.1 but also >> the latest fixes-1_0 branch (with slow IO being fixed). >> >> The setup procedure was like Alexey described in one of his previous >> postings. I tested the ar-jdbc version only. > > Your results are truly puzzling! I presume Alexey's numbers left JIT at > default settings, and he showed a rough doubling of performance. Your > numbers not only show trunk being slower, but compiled mode being slower > than JIT mode as well. Something's not right. Oops, of course I meant compiled slower than interpreted...anyway, weird. - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutTwo questions:
1. Did you pass -J-Djruby.objectspace.enabled=false ? 2. Do you see the benchmarking script do JIT warmup when running under JRuby? Charles: yep, I didn't change any settings other than jruby.objectspace.enabled. -- Alexey Verkhovsky CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com] RubyWorks [http://rubyworks.thoughtworks.com] --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutSorry if you told it somewhere already,
but are you sure to be in production mode? I suspect Rails to get inefficient in development mode that the compiler completes as it's re-compiling things over and over. Raphaël Valyi On 9/25/07, Alexey Verkhovsky <alexey.verkhovsky@...> wrote: Two questions: |
|
|
Re: JRuby vs MRI - Petstore shootoutI did use the "ar_jdbc" mode which should have settings like production has.
For instance: echo 'load "script/benchmark_action"' | ~/jruby-1_0/bin/jruby -J-server -J-Djruby.jit.enabled=false -J-Djruby.objectspace.enabled=false script/console ar_jdbc And yes, I saw the "JIT warmup" output. Alexey: Have you also tried the latest 1.0.x (with IO sync removed)?
|
|
|
Re: JRuby vs MRI - Petstore shootoutI tried a few of them, bench_block_invocation.rb, bench_quicksort.rb (the latter needs ages to load the data with JRuby while MRI loads it in a fraction of a second. The test numbers itself are pretty close to MRI).
trunk is always slightly faster than 1.0.x. When turning off the JIT the performance drops significantly. Probably these results are as exepected, although the peformance gain of trunk over 1.0.x is pretty small (like 10%).
|
|
|
Re: JRuby vs MRI - Petstore shootoutChristian Seiler wrote:
> I tried a few of them, bench_block_invocation.rb, bench_quicksort.rb (the > latter needs ages to load the data with JRuby while MRI loads it in a > fraction of a second. The test numbers itself are pretty close to MRI). > > trunk is always slightly faster than 1.0.x. When turning off the JIT the > performance drops significantly. > Probably these results are as exepected, although the peformance gain of > trunk over 1.0.x is pretty small (like 10%). Many of those benchmarks won't show a huge increase over 1.0 because most of them were compiling then too. But at least we're seeing the opposite results here from what you're seeing on Rails. Hmm...I'm stumped for the moment. - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutOn 9/26/07, Charles Oliver Nutter <charles.nutter@...> wrote:
> Many of those benchmarks won't show a huge increase over 1.0 because > most of them were compiling then too. Much (most?) of the difference between trunk and 1.0.1 must be the logging bottleneck. It's removed in 1.0 branch, but still present in 1.0.1 release. -- Alexey Verkhovsky CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com] RubyWorks [http://rubyworks.thoughtworks.com] --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutCould you give the 1.0 branch a try? Because if your results aren't inline with mine
(i.e. trunk slower than latest 1.0), I actually won't know what to try next. I even tried the IBM JDK and JRockit (which is faster than Sun JDK but crashes often), the same picture.
|
|
|
Re: JRuby vs MRI - Petstore shootoutAlexey Verkhovsky wrote:
> Some new numbers. I'm getting this when I try to run: Loading production environment. >> load "script/benchmark_action.rb" Using pure-Ruby MySQL driver from Rails Warming up JIT Done nil >> => [] Using petstore trunk. - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutCharles Oliver Nutter wrote:
> Alexey Verkhovsky wrote: >> Some new numbers. > > I'm getting this when I try to run: Got it to run, but it's only outputting the "full action" times en masse. Adding to that, I don't know how to interpret these numbers. They don't look particularly higher or lower for either Ruby or JRuby, though they both range over a very wide time span. MRI: full action: 2.400000 0.200000 2.600000 ( 5.796986) full action: 2.380000 0.190000 2.570000 ( 5.783790) full action: 2.410000 0.200000 2.610000 ( 5.836291) full action: 2.470000 0.190000 2.660000 ( 6.058336) full action: 2.380000 0.190000 2.570000 ( 5.763391) full action: 2.410000 0.200000 2.610000 ( 5.797751) full action: 2.410000 0.190000 2.600000 ( 5.812139) full action: 2.390000 0.190000 2.580000 ( 5.793960) full action: 2.360000 0.190000 2.550000 ( 5.704807) full action: 2.330000 0.180000 2.510000 ( 5.681507) full action: 12.197000 0.000000 12.197000 ( 12.197000) full action: 12.289000 0.000000 12.289000 ( 12.289000) full action: 12.504000 0.000000 12.504000 ( 12.504000) full action: 11.914000 0.000000 11.914000 ( 11.914000) full action: 12.904000 0.000000 12.904000 ( 12.904000) full action: 12.355000 0.000000 12.355000 ( 12.355000) full action: 12.336000 0.000000 12.336000 ( 12.336000) full action: 11.591000 0.000000 11.591000 ( 11.591000) full action: 11.256000 0.000000 11.256000 ( 11.257000) full action: 11.652000 0.000000 11.652000 ( 11.653000) full action: 2.510000 0.230000 2.740000 ( 6.931387) full action: 2.490000 0.220000 2.710000 ( 6.357651) full action: 2.520000 0.230000 2.750000 ( 6.315906) full action: 2.480000 0.210000 2.690000 ( 5.997861) full action: 2.480000 0.220000 2.700000 ( 6.364616) full action: 2.520000 0.230000 2.750000 ( 6.266817) full action: 2.460000 0.210000 2.670000 ( 6.052257) full action: 2.460000 0.210000 2.670000 ( 6.103867) full action: 2.420000 0.210000 2.630000 ( 5.902875) full action: 2.450000 0.220000 2.670000 ( 6.007488) JRuby: full action: 12.897000 0.000000 12.897000 ( 12.897000) full action: 12.325000 0.000000 12.325000 ( 12.324000) full action: 12.233000 0.000000 12.233000 ( 12.233000) full action: 11.821000 0.000000 11.821000 ( 11.822000) full action: 11.582000 0.000000 11.582000 ( 11.582000) full action: 11.806000 0.000000 11.806000 ( 11.806000) full action: 16.221000 0.000000 16.221000 ( 16.221000) full action: 12.348000 0.000000 12.348000 ( 12.348000) full action: 11.633000 0.000000 11.633000 ( 11.633000) full action: 2.400000 0.200000 2.600000 ( 5.796986) full action: 2.380000 0.190000 2.570000 ( 5.783790) full action: 2.410000 0.200000 2.610000 ( 5.836291) full action: 2.470000 0.190000 2.660000 ( 6.058336) full action: 2.380000 0.190000 2.570000 ( 5.763391) full action: 2.410000 0.200000 2.610000 ( 5.797751) full action: 2.410000 0.190000 2.600000 ( 5.812139) full action: 2.390000 0.190000 2.580000 ( 5.793960) full action: 2.360000 0.190000 2.550000 ( 5.704807) full action: 2.330000 0.180000 2.510000 ( 5.681507) full action: 12.197000 0.000000 12.197000 ( 12.197000) full action: 12.289000 0.000000 12.289000 ( 12.289000) full action: 12.504000 0.000000 12.504000 ( 12.504000) full action: 11.914000 0.000000 11.914000 ( 11.914000) full action: 12.904000 0.000000 12.904000 ( 12.904000) full action: 12.355000 0.000000 12.355000 ( 12.355000) full action: 12.336000 0.000000 12.336000 ( 12.336000) full action: 11.591000 0.000000 11.591000 ( 11.591000) full action: 11.256000 0.000000 11.256000 ( 11.257000) It's rather unclear to me how to make the test print out something interesting, or whether this data is interesting itself... - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: JRuby vs MRI - Petstore shootoutChristian Seiler wrote:
> I tried a few of them, bench_block_invocation.rb, bench_quicksort.rb (the > latter needs ages to load the data with JRuby while MRI loads it in a > fraction of a second. The test numbers itself are pretty close to MRI). > > trunk is always slightly faster than 1.0.x. When turning off the JIT the > performance drops significantly. > Probably these results are as exepected, although the peformance gain of > trunk over 1.0.x is pretty small (like 10%). FYI, I corrected the quicksort thing so it builds up the large array rather than trying to compile a gigantic literal array (which was what took all that time). It should be a more useful benchmark now. - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
|
|
|
Re: Re: JRuby vs MRI - Petstore shootoutJust out of curiousity, has anyone tried rails benchmarks after applied Charles' caching compiled evals patch? Or is that in trunk already?
--Chris On 10/4/07,
Christian Seiler <chr.seiler@...
> wrote: Hi, |
|
|
Re: Re: JRuby vs MRI - Petstore shootoutChris Nelson wrote:
> Just out of curiousity, has anyone tried rails benchmarks after applied > Charles' caching compiled evals patch? Or is that in trunk already? It's not rock solid so it hasn't been committed yet. Perhaps today... - Charlie --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
| < Prev | 1 - 2 | Next > |
| Free embeddable forum powered by Nabble | Forum Help |