Re: Benchmark results -- follow-up

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha