Grinder vs. JMeter "speed"

[Tuvell_Walter]

First the disclaimer: I'm not interested in "Grinder vs. JMeter"
religious issues, or abstract/generalized comparisons.  Instead, I've
run across a legitimate, narrow issue I'd like to understand.

I have been using Grinder extensively for more than a year now, to drive
a massively-distributed ("cloud"-class) web-services system.  Since the
target system is highly functional, my Grinder script is very
sophisticated at this point (and big: 16,000 lines [I can't imagine
doing this in JMeter!]).  Now, in addition to web-services, the target
system also supports file-services (NFS, CIFS, and proprietary
FUSE-based installable file system).  I hadn't addressed file-services
before, because elsewhere in the company (China) some folks have
independently written some JMeter scripts (called "test-plans" in
JMeter-speak) to test file-services.  But supporting disparate load/perf
tools is inconvenient, so we're exploring possibilities for
consolidation.

To that end, this week I've been looking at JMeter for the first time.
One thing I immediately wanted to know was the relative "speed"
(efficiency) of the 2 tools (and, their accuracy in measuring that
speed).

I decided to run a "bake-off" test, specifically targeted at comparing
the speed of Grinder vs. JMeter themselves (without bringing target
server idiosyncrasies into the picture).  To eliminate as many variables
as possible, I chose the simplest possible test I could think of: single
Grinder or JMeter instance (which means Console/Agent/Worker for
Grinder, but JMeter runs them all in a single process), doing HTTP GET
to a single page on a single Apache server.  Here's the script, in
Grinder-speak:

    from net.grinder.script import Test
    from net.grinder.plugin.http import HTTPRequest

    def get():
        HTTPRequest().GET('http://10.5.115.74:80/')

    class TestRunner:
        def __call__(self):
            Test(101, 'Get').wrap(get)()

The translation to JMeter test-plan:

    Start jmeter.sh.
    Add Thread Group, 50 threads, loop count forever.
    Add Sampler, HTTP Request, IP=10.5.115.74, path=<empty>
    Add Listener, Summary Report.

(This experiment is easily reproducible, if anyone would care to try.
I've attempted to do everything I can to make the test even-handed, such
as using the same JVM and JVM parameters, but of course I could have
missed something ...)

What I'm finding is that JMeter consistently outperforms Grinder by a
few percentage points.  By that I mean when running head-to-head (e.g.,
simultaneously on the same desktop), JMeter completes more
tests/transactions and throughput per second, with better mean/average
time per operation.

For example, as I'm sitting here watching such a simultaneous test in
progress, I'm observing Grinder doing 1.29M operations at 57.3 ms/op,
869 TPS, 251 KB/sec; while JMeter is doing 1.34M ops at 55ms/op, 899
TPS, 253.76 KB/sec.  (This test uses 50 threads each, and an Apache
server running on ESX VM, which is OK since we only care about comparing
Grinder and JMeter, not testing Apache).

This seems to indicate more "internal friction" in Grinder than in
JMeter (perhaps in Grinder HTTPClient vs. JMeter HttpClient?).  Not the
least consequence of this experiment is that it seems JMeter is driving
the target system harder (good characteristics for "load testing"), and
getting better results out of it (good for "perf testing").  The latter
("better numbers", however small) can have practical ramifications,
because we're using Grinder with customers, and they tend focus on TPS.
Where you're getting near to customer reqts, a few percentage point
improvement in TPS has a noticeable impact.  And it doesn't cut it to
say, "Oh, you can just chalk that last 3% you wanted up to experimental
error".

Thoughts?


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use

[Travis Bear]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Jython 2.5.1 was released a few days back.  Phil any news on updating The Grinder for compatibility w/ current jython versions?

Thanks!


-Travis

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use

[Philip Aston-3]

Travis Bear wrote:
> Jython 2.5.1 was released a few days back.  Phil any news on updating
> The Grinder for compatibility w/ current jython versions?

Working on it.  For various reasons, it requires a new, more general
instrumentation engine which is a major task.

Estimate is "before Christmas".

- Phil

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use

[Philip Aston-3]

Hi Walt,

Clearly there's some Jython overhead that may well make a pure Java
process more efficient (more TPS on same kit). Scripting productivity
costs you a little hardware! I'm more concerned by the difference in
response times.

The 57ms vs 55ms could be down to the use of System.currentTimeMillis().
Try running with grinder.useNanoTime (this will be default in the next
version of The Grinder).

I presume you have also checked that JMeter is establishing a brand new
HTTP connection for each cycle?

Finally, I'm not overly worried about this. I appreciate 3% is a lot to
your customers, but its negligible compared to the differences between
the simulation and the real world introduced by network overhead,
differences between The Grinder and a browser, etc.

- Phil

Tuvell_Walter@... wrote:

> First the disclaimer: I'm not interested in "Grinder vs. JMeter"
> religious issues, or abstract/generalized comparisons.  Instead, I've
> run across a legitimate, narrow issue I'd like to understand.
>
> I have been using Grinder extensively for more than a year now, to drive
> a massively-distributed ("cloud"-class) web-services system.  Since the
> target system is highly functional, my Grinder script is very
> sophisticated at this point (and big: 16,000 lines [I can't imagine
> doing this in JMeter!]).  Now, in addition to web-services, the target
> system also supports file-services (NFS, CIFS, and proprietary
> FUSE-based installable file system).  I hadn't addressed file-services
> before, because elsewhere in the company (China) some folks have
> independently written some JMeter scripts (called "test-plans" in
> JMeter-speak) to test file-services.  But supporting disparate load/perf
> tools is inconvenient, so we're exploring possibilities for
> consolidation.
>
> To that end, this week I've been looking at JMeter for the first time.
> One thing I immediately wanted to know was the relative "speed"
> (efficiency) of the 2 tools (and, their accuracy in measuring that
> speed).
>
> I decided to run a "bake-off" test, specifically targeted at comparing
> the speed of Grinder vs. JMeter themselves (without bringing target
> server idiosyncrasies into the picture).  To eliminate as many variables
> as possible, I chose the simplest possible test I could think of: single
> Grinder or JMeter instance (which means Console/Agent/Worker for
> Grinder, but JMeter runs them all in a single process), doing HTTP GET
> to a single page on a single Apache server.  Here's the script, in
> Grinder-speak:
>
>     from net.grinder.script import Test
>     from net.grinder.plugin.http import HTTPRequest
>
>     def get():
>         HTTPRequest().GET('http://10.5.115.74:80/')
>
>     class TestRunner:
>         def __call__(self):
>             Test(101, 'Get').wrap(get)()
>
> The translation to JMeter test-plan:
>
>     Start jmeter.sh.
>     Add Thread Group, 50 threads, loop count forever.
>     Add Sampler, HTTP Request, IP=10.5.115.74, path=<empty>
>     Add Listener, Summary Report.
>
> (This experiment is easily reproducible, if anyone would care to try.
> I've attempted to do everything I can to make the test even-handed, such
> as using the same JVM and JVM parameters, but of course I could have
> missed something ...)
>
> What I'm finding is that JMeter consistently outperforms Grinder by a
> few percentage points.  By that I mean when running head-to-head (e.g.,
> simultaneously on the same desktop), JMeter completes more
> tests/transactions and throughput per second, with better mean/average
> time per operation.
>
> For example, as I'm sitting here watching such a simultaneous test in
> progress, I'm observing Grinder doing 1.29M operations at 57.3 ms/op,
> 869 TPS, 251 KB/sec; while JMeter is doing 1.34M ops at 55ms/op, 899
> TPS, 253.76 KB/sec.  (This test uses 50 threads each, and an Apache
> server running on ESX VM, which is OK since we only care about comparing
> Grinder and JMeter, not testing Apache).
>
> This seems to indicate more "internal friction" in Grinder than in
> JMeter (perhaps in Grinder HTTPClient vs. JMeter HttpClient?).  Not the
> least consequence of this experiment is that it seems JMeter is driving
> the target system harder (good characteristics for "load testing"), and
> getting better results out of it (good for "perf testing").  The latter
> ("better numbers", however small) can have practical ramifications,
> because we're using Grinder with customers, and they tend focus on TPS.
> Where you're getting near to customer reqts, a few percentage point
> improvement in TPS has a noticeable impact.  And it doesn't cut it to
> say, "Oh, you can just chalk that last 3% you wanted up to experimental
> error".
>
> Thoughts?
>
>  

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use

[Tuvell_Walter]

Thinking about it a little more this morning,  I've found a "workaround"
(if you want to call it that) that I'm comfortable with.  That is, I can
comfortably use it to "equalize" this aspect of the Grinder vs. JMeter
argument, thereby moving the discussion to other aspects of the
comparison.

The answer turns out to be quite simple and straightforward: Just run
Grinder with a few more threads than JMeter.  Using the example I gave
yesterday, giving Grinder 50 threads and JMeter 45 more than equalizes
the differences with respect to TPS & throughput bytes/sec.  Which is,
as I mentioned, the most important metric for Grinder vs. JMeter
"speed".  (Not to mention throwing machines into the mix as required
...)

It's a bit irksome that JMeter seems to achieve a slightly faster
mean/average for ms/operation time (for equal numbers of threads).  My
seat-of-the-pants guess is that may be due to the underlying HTTP client
library, or perhaps some threads or time-capture tweaking.  Which may be
of some mild academic interest for Grinder/JMeter developers, but takes
a distinct backseat to TPS/throughput in my view.  And anyway it becomes
less important, proportionately speaking, as one moves further away from
the trivial/toy HTTP GET test example towards the much more complicated
(distributed) operations we're really interested in.

That "speed" aspect of the Grinder/JMeter comparison having been
equalized, in a way that I can defend in the context of both my
consolidation project as well as customer interactions, it's now
perfectly clear to me that Grinder is the platform we'll go forward
with.  Which had been my inclination all long, but after all one must
always do the right thing regardless of personal preferences, and
proceed in the direction due-diligence investigation leads.

Cheers!

> _____________________________________________
> From: Tuvell, Walter  
> Sent: Thursday, October 01, 2009 5:00 PM
> To: 'grinder-use'
> Subject: Grinder vs. JMeter "speed"
>
> First the disclaimer: I'm not interested in "Grinder vs. JMeter"
> religious issues, or abstract/generalized comparisons.  Instead, I've
> run across a legitimate, narrow issue I'd like to understand.
>
> I have been using Grinder extensively for more than a year now, to
> drive a massively-distributed ("cloud"-class) web-services system.
> Since the target system is highly functional, my Grinder script is
> very sophisticated at this point (and big: 16,000 lines [I can't
> imagine doing this in JMeter!]).  Now, in addition to web-services,
> the target system also supports file-services (NFS, CIFS, and
> proprietary FUSE-based installable file system).  I hadn't addressed
> file-services before, because elsewhere in the company (China) some
> folks have independently written some JMeter scripts (called
> "test-plans" in JMeter-speak) to test file-services.  But supporting
> disparate load/perf tools is inconvenient, so we're exploring
> possibilities for consolidation.
>
> To that end, this week I've been looking at JMeter for the first time.
> One thing I immediately wanted to know was the relative "speed"
> (efficiency) of the 2 tools (and, their accuracy in measuring that
> speed).
>
> I decided to run a "bake-off" test, specifically targeted at comparing
> the speed of Grinder vs. JMeter themselves (without bringing target
> server idiosyncrasies into the picture).  To eliminate as many
> variables as possible, I chose the simplest possible test I could
> think of: single Grinder or JMeter instance (which means
> Console/Agent/Worker for Grinder, but JMeter runs them all in a single
> process), doing HTTP GET to a single page on a single Apache server.
> Here's the script, in Grinder-speak:
>
>     from net.grinder.script import Test
>     from net.grinder.plugin.http import HTTPRequest
>
>     def get():
>         HTTPRequest().GET('http://10.5.115.74:80/')
>
>     class TestRunner:
>         def __call__(self):
>             Test(101, 'Get').wrap(get)()
>
> The translation to JMeter test-plan:
>
>     Start jmeter.sh.
>     Add Thread Group, 50 threads, loop count forever.
>     Add Sampler, HTTP Request, IP=10.5.115.74, path=<empty>
>     Add Listener, Summary Report.
>
> (This experiment is easily reproducible, if anyone would care to try.
> I've attempted to do everything I can to make the test even-handed,
> such as using the same JVM and JVM parameters, but of course I could
> have missed something ...)
>
> What I'm finding is that JMeter consistently outperforms Grinder by a
> few percentage points.  By that I mean when running head-to-head
> (e.g., simultaneously on the same desktop), JMeter completes more
> tests/transactions and throughput per second, with better mean/average
> time per operation.
>
> For example, as I'm sitting here watching such a simultaneous test in
> progress, I'm observing Grinder doing 1.29M operations at 57.3 ms/op,
> 869 TPS, 251 KB/sec; while JMeter is doing 1.34M ops at 55ms/op, 899
> TPS, 253.76 KB/sec.  (This test uses 50 threads each, and an Apache
> server running on ESX VM, which is OK since we only care about
> comparing Grinder and JMeter, not testing Apache).
>
> This seems to indicate more "internal friction" in Grinder than in
> JMeter (perhaps in Grinder HTTPClient vs. JMeter HttpClient?).  Not
> the least consequence of this experiment is that it seems JMeter is
> driving the target system harder (good characteristics for "load
> testing"), and getting better results out of it (good for "perf
> testing").  The latter ("better numbers", however small) can have
> practical ramifications, because we're using Grinder with customers,
> and they tend focus on TPS.  Where you're getting near to customer
> reqts, a few percentage point improvement in TPS has a noticeable
> impact.  And it doesn't cut it to say, "Oh, you can just chalk that
> last 3% you wanted up to experimental error".
>
> Thoughts?
>

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use

[Tuvell_Walter]

Hi, Phil.

Concerning Jython: I regard that as the HUGEST plus, but needed to
divorce it from the "speed" aspect for analysis reasons.

Concerning latency/response times: I too speculated about "time-capture
tweaks" (see my follow-up note) which you hint with the nanotime
comment.  Glad to hear you're looking at that.

Concerning HTTP connections: Given my newness to Jython, I couldn't
figure out how to determine that easily.  So I did the opposite: I made
Grinder reuse the connection, by putting a loop (not an infinite one!)
directly inside the TestRunner.__call__(), waiting for the answer, and
dividing.  Grinder sped up a bit (less than I'd expected), but JMeter
still beat it for latency/response time.

In any case, since you responded to my first note in this thread, you
may not have yet seen my follow-up to it, wherein I restore Grinder's
"dignity".  :-)  No worries, Grinder still beats JMeter by miles when
everything is taken into consideration.

Can't wait for the new release ...

- Walt


 

-----Original Message-----
From: Philip Aston [mailto:philip.aston@...]
Sent: Friday, October 02, 2009 8:21 AM
To: grinder-use
Subject: Re: [Grinder-use] Grinder vs. JMeter "speed"

Hi Walt,

Clearly there's some Jython overhead that may well make a pure Java
process more efficient (more TPS on same kit). Scripting productivity
costs you a little hardware! I'm more concerned by the difference in
response times.

The 57ms vs 55ms could be down to the use of System.currentTimeMillis().
Try running with grinder.useNanoTime (this will be default in the next
version of The Grinder).

I presume you have also checked that JMeter is establishing a brand new
HTTP connection for each cycle?

Finally, I'm not overly worried about this. I appreciate 3% is a lot to
your customers, but its negligible compared to the differences between
the simulation and the real world introduced by network overhead,
differences between The Grinder and a browser, etc.

- Phil

Tuvell_Walter@... wrote:
> First the disclaimer: I'm not interested in "Grinder vs. JMeter"
> religious issues, or abstract/generalized comparisons.  Instead, I've
> run across a legitimate, narrow issue I'd like to understand.
>
> I have been using Grinder extensively for more than a year now, to
drive
> a massively-distributed ("cloud"-class) web-services system.  Since
the
> target system is highly functional, my Grinder script is very
> sophisticated at this point (and big: 16,000 lines [I can't imagine
> doing this in JMeter!]).  Now, in addition to web-services, the target
> system also supports file-services (NFS, CIFS, and proprietary
> FUSE-based installable file system).  I hadn't addressed file-services
> before, because elsewhere in the company (China) some folks have
> independently written some JMeter scripts (called "test-plans" in
> JMeter-speak) to test file-services.  But supporting disparate
load/perf

> tools is inconvenient, so we're exploring possibilities for
> consolidation.
>
> To that end, this week I've been looking at JMeter for the first time.
> One thing I immediately wanted to know was the relative "speed"
> (efficiency) of the 2 tools (and, their accuracy in measuring that
> speed).
>
> I decided to run a "bake-off" test, specifically targeted at comparing
> the speed of Grinder vs. JMeter themselves (without bringing target
> server idiosyncrasies into the picture).  To eliminate as many

variables
> as possible, I chose the simplest possible test I could think of:
single

> Grinder or JMeter instance (which means Console/Agent/Worker for
> Grinder, but JMeter runs them all in a single process), doing HTTP GET
> to a single page on a single Apache server.  Here's the script, in
> Grinder-speak:
>
>     from net.grinder.script import Test
>     from net.grinder.plugin.http import HTTPRequest
>
>     def get():
>         HTTPRequest().GET('http://10.5.115.74:80/')
>
>     class TestRunner:
>         def __call__(self):
>             Test(101, 'Get').wrap(get)()
>
> The translation to JMeter test-plan:
>
>     Start jmeter.sh.
>     Add Thread Group, 50 threads, loop count forever.
>     Add Sampler, HTTP Request, IP=10.5.115.74, path=<empty>
>     Add Listener, Summary Report.
>
> (This experiment is easily reproducible, if anyone would care to try.
> I've attempted to do everything I can to make the test even-handed,

such
> as using the same JVM and JVM parameters, but of course I could have
> missed something ...)
>
> What I'm finding is that JMeter consistently outperforms Grinder by a
> few percentage points.  By that I mean when running head-to-head
(e.g.,
> simultaneously on the same desktop), JMeter completes more
> tests/transactions and throughput per second, with better mean/average
> time per operation.
>
> For example, as I'm sitting here watching such a simultaneous test in
> progress, I'm observing Grinder doing 1.29M operations at 57.3 ms/op,
> 869 TPS, 251 KB/sec; while JMeter is doing 1.34M ops at 55ms/op, 899
> TPS, 253.76 KB/sec.  (This test uses 50 threads each, and an Apache
> server running on ESX VM, which is OK since we only care about
comparing
> Grinder and JMeter, not testing Apache).
>
> This seems to indicate more "internal friction" in Grinder than in
> JMeter (perhaps in Grinder HTTPClient vs. JMeter HttpClient?).  Not
the
> least consequence of this experiment is that it seems JMeter is
driving
> the target system harder (good characteristics for "load testing"),
and
> getting better results out of it (good for "perf testing").  The
latter
> ("better numbers", however small) can have practical ramifications,
> because we're using Grinder with customers, and they tend focus on
TPS.
> Where you're getting near to customer reqts, a few percentage point
> improvement in TPS has a noticeable impact.  And it doesn't cut it to
> say, "Oh, you can just chalk that last 3% you wanted up to
experimental
> error".
>
> Thoughts?
>
>  


------------------------------------------------------------------------
------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and
stay
ahead of the curve. Join us from November 9-12, 2009. Register
now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use

[Chad Bellan]

I'm finding that when I set grinder.threads to any value greater than 2,
that the threads are being dropped.  
Can anyone tell me why this might occur?


Chad

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use