Re: Benchmark results -- follow-up

Raul / Ross

Here are my comments..

Don't forget you are getting your information for a platform that wants to compete with Mule :) We were never able to run the benchmarks with the information provided since it's unclear how they ran each platform.  If fact they never originally published the benchmark code, but I see now it's available since Feb 11th.

I can confirm that we shared necessary information with Eugene Ciurana on this benchmark after the public comment here.. and offered to re-run the tests with any suggestions from Mule. I specifically stated "We would be glad to incorporate your suggestions and to retest and publish new sets of results" .. However, we never heard back from Eugene or anyone else! Except for this whining from Dave :-) (to which Sanjiva replied)

This  comment from Ruwan confirms that the Mule configuration was available publicly since 30th of August 2007 (I don't know why you stated Feb 11th date.. ?? whats significant about that date??)

The issue they had with keep alive is a configuration error as far as I can see, and Mule has supported web service proxying for years.

Well then, why didn't any body from Mule reply to this email on 8th July 2007?... actually no one has replied to this even upto now.. I don't think silence in the mailing lists is a good way to support your many 'production deployments' and 'clients'?..

My suggestion is to take the two projects for a test drive if you are unsure.  I think you'll like Mule.  Mule has been running in production environments (including high-performance messaging) for over 4 years now.  It's been battle-tested more that any other open and closed-source ESB, and provides more functionality out of the box than the others.

Let me quote Eugene again, Ref (http://wso2.org/blog/asankha/2263)

Recently on TSS, Eugene Ciurana was claiming that Mule was able to do 200 million transactions a day, and that no other ESB they evaluated, commercial or open source, could do that on the same hardware. This means that no ESB was able to perform 2315 TPS? I wish they would describe the scenario tested and the hardware used, instead of claiming such numbers out loud. For example a single instance of the WSO2 ESB / Synapse could do well over 3200 TPS with 40 concurrent users for some scenarios we've tested, on standard hardware (see 'Notes' here). I am almost sure that Eugene is yet to see the WSO2 ESB /Apache Synapse in action, cause they are not like the good ol' Mules that supposedly "Just works"... they work much faster and smarter! and they are extremely lightweight.

And Eugene replied to this publicly stating"I began looking at your software on Sunday 15.July, on a recommendation from Joe Ottinger... so your comment was accurate as of 12.July!" and we haven't heard from him since then..:)

I am also very interested to see that Mule actually contains code to recover even from a Java OOM Exception? I guess this validates your ability to run in production environments for over 4 years now :-)?

As for defending Mule, it does a very good job of defending itself; just take it for a test drive.

btw, I'll ask our QA guys to see if there has been enough detail about this benchmark published so that we could run it. If so we will happily publish real results for Mule.

Well, lets look at your own comments on Dain Hansen (BEA)'s blog from November 2006

"Dain, its very misleading to quote your benchmark figures in a comparison with figures for Mule that were quoted originally by a user without any context and certainly are not benchmark figures. We are working on publishing some figures by the end of the year, but in the meantime maybe you should edit your posting. Cheers, Ross

Posted by: rossmason on November 10, 2006 at 7:37 AM"

Its been over two (2) years! since this comment Ross.. but why don't we still see *any* performance benchmarks from Mule yet? Are you hiding something now ;-) ?



Raul, I agree with Ross on his comment that you should test drive both Mule and the WSO2 ESB (and/or Apache Synapse). Feel free to contact me for any help

thanks
asankha


References:
http://wso2.org/library/2259
http://wso2.org/esb