Re: Benchmark results -- follow-up

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.



  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"