Re: Benchmark results -- follow-up

Ross Mason-3 wrote:

Hi Raul,

We currently have a lot going on and running these benchmarks (that  
only test a small subset of Mule's functionality) is not going to jump  
the queue. We will get to it.  However, this is an open source project  
so you have everything available to you to make your own assessment.

Cheers,

Ross Mason
CTO, Co-Founder
MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com





On 21 Apr 2008, at 02:34, raulvk wrote:

> Hi Mule people,
>
> Any news on this? Have you run the benchmarks?
>
> We, users, are getting impacient to see results. Can you please give  
> us an estimate as to when the results will be published, in case you  
> don't have the results already?
>
> Thank you,
>
> Raul.
>
>
> On 14/04/2008, Ruwan Linton <ruwan.linton@gmail.com> wrote:
> Hi Ross,
>
> Nice to see that you are going to run the performance tests.
>
> On Mon, Apr 14, 2008 at 9:20 AM, Ross Mason <themuleman@gmail.com>  
> wrote:
> Hi Ruwan,
>
> Thanks for reaching out.  We will run these benchmarks along side  
> our own and publish the results. I do agree with David Dossot here  
> though in that vendors are not the best people to run benchmarks  
> since they are not impartial.  I am surprised there is not an  
> independent body for this yet.  I don't think anyone regards users  
> as dumb (not sure why you said this)
>
> I told this, because you all tried to defend using statements on the  
> mailing lists and using blogs rather than running the performance  
> tests and publishing the performance figures.
>
> and while users like empirical data they also will not accept  
> benchmark results from a single vendor without doing their own  
> research. That was what I was pushing for in my original response to  
> this thread.
>
> Well, we did a quite a lot of research on these products (especially  
> ServiceMix and Mule). We had one engineer per product working for  
> weeks to get these configurations optimized and we asked for help  
> over the public lists. For example, I was the one who was looking at  
> the Service Mix and I read the full JBI spec before developing the  
> SM configuration apart from there documents and the questions over  
> the lists to get the SM configurations optimized.
>
> Chathura is the one who did the Mule configuration and here are some  
> questions that he posted to your lists;
>
> http://markmail.org/message/t2as5sco5gct7gna
> http://www.nabble.com/http-transport-problem-td11116197.html
> http://osdir.com/ml/java.mule.devel/2007-07/msg00046.html
> http://markmail.org/message/qzkpe36yocgpq6nr#query:+page:1+mid:4hnlmywwv4va62he+state:results
> http://www.mail-archive.com/user@mule.codehaus.org/msg04358.html
> http://markmail.org/message/mkrtjk3k2yj4mb4f#query:chathura%20mule 
> %20XFire+page:1+mid:vz2hr5mpety7rm5c+state:results
>
> So I think he did a good set of researches on getting the Mule  
> configurations tuned for performance.
>
> Thanks,
> Ruwan
>
>
> Cheers,
>
> Ross Mason
> CTO, Co-Founder
> MuleSource Inc.
>
> http://mulesource.com | http://blog.rossmason.com
>
>
>
>
>
> On 13 Apr 2008, at 18:40, Ruwan Linton wrote:
>
>> Hi Mule devs,
>>
>> I think Raul is exactly correct on this. You should have a look at  
>> the configurations (I am sure you already had a deep look in to  
>> those :-)) that we used in doing the performance tests and improve  
>> the Mule once to optimize the configurations for performance and re-
>> run the performance tests. This will enable the users to understand  
>> the performance benchmarks of the two products very easily.
>>
>> If you need any help in getting Synapse configurations and the  
>> product optimized for performance or any information on setting up  
>> the environment we would like to incorporate with you. So don't  
>> waist any more time and get started with the performance tests. You  
>> should bring the counter arguments over Synapse / WSO2 ESB with  
>> facts in your hand, otherwise I don't think users will accept those  
>> (You should keep in mind that users are not dumb). If you can  
>> perform faster than Synapse you don't have to worry at all just  
>> show the figures.
>>
>> Thanks,
>> Ruwan
>>
>> On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@gmail.com> wrote:
>> Hi everyone,
>>
>> First of all, I'd like to state that I am researching the use of an  
>> open source ESB with the uttermost neutrality: our client had  
>> determined that they want an open source ESB to serve as their  
>> integration backbone, but they have left it up to us to pick the  
>> most appropriate one.
>>
>> I seems that this discussion has deviated and ended up turning into  
>> a war between vendors, which doesn't serve us prospective end users  
>> at all. My aim when I wrote the first message was to throw some  
>> light on why Mule hadn't apparently pronounced themselves on the  
>> results.
>>
>> To the Mule people, even though the benchmark was run by another  
>> vendor, the code was released publicly and help was requested on  
>> the forums. So there has been a high degree of visibility into the  
>> benchmark process. Of course, the WSO2 team are experts in Apache  
>> Synapse, which enables them to configure Synapse optimally. That is  
>> the only "unfairness" factor that I perceive, but once again, they  
>> asked for help on the forums when it came to Mule's config.
>>
>> Anyway, bottom line is that the only method to address this  
>> discussion *productively* is for the Mule team to look at the  
>> published benchmark code and point out *where* the configuration  
>> mistakes that Ross mentions are, and how they can be fixed. We  
>> could then run a new round of benchmarks with an optimised Mule  
>> config.
>>
>> Please consider that the constant goal of both vendors is to gain a  
>> larger user base, and *we users* are moved by *facts and figures*,  
>> so let's bring them to the table instead of criticising each other.
>>
>> Raul.
>>
>>
>>
>> On 13/04/2008, Asankha C. Perera <asankha@wso2.com> wrote:
>> Gary
>>> Asankha your comments fairly look baised.
>> I must say that I am a lead committer of Apache Synapse and lead  
>> the development of the WSO2 ESB; And I ran the benchmarks against  
>> Mule, ServiceMix and a leading commercial ESB and published the  
>> results.
>>>  While we could not test close to a million transactions in a day  
>>> ( did not have full day available) but We have
>>> tested Mule with close to a 100000 transactions in 4-5 hours and  
>>> it performed fanstatically.
>> Interesting.. this translates to just over 6 TPS. Can you explain  
>> the scenarios in more detail?
>>>  Our configurations of course may not be optimal as
>>> we were still fine tuning our system. The WSO2 results have been  
>>> published
>>>
>>>
>>> by they themselves and they are simply another competitor in the  
>>> same space.
>>>
>>>
>>>
>>> To get a true analysis, it would have to be done by a third party  
>>> nuetral
>>> company.
>> The reality is you can wait till this "neutral third party" comes  
>> up with an independent benchmark for ESB's, or you can run your  
>> own. What I believe is that as long as you clearly explain the  
>> scenarios, make any code/configuration and tools you use public  
>> (and free), and also ask each vendor for assistance when you find  
>> issues and give them an opportunity to reply, you are good.
>>> The fact that WSO2 is basically trying to bench mark against mule  
>>> suggests in iteslf that WSO2 acknowledges that Mule may be "The  
>>> Standard" to beat. Better to go with the "The Standard" I guess.
>>>
>>>
>>>
>>>
>>>
>>>
>> Oh no! The first ESB we benchmarked against was a commercial closed  
>> source ESB (See http://wso2.org/library/1721). It was only after we  
>> published those results that we ran the benchmark a second time.  
>> And we included Apache ServiceMix, and ofcourse Mule. We considered  
>> OpenESB and JBoss ESB as well, but due to many issues we faced, we  
>> compared only Apache SM and Mule. You may note that Apache  
>> ServiceMix had very interesting results and did better in  
>> transformations initially! And we did not hide this fact (http://wso2.org/library/2259 
>> ), We also reported the issue we faced against SM and they looked  
>> into it unlike Mule.
>>
>> asankha
>>
>>
>>
>>
>> --
>> Ruwan Linton
>> http://www.wso2.org - "Oxygenating the Web Services Platform"
>
>
>
>
> --
> Ruwan Linton
> http://www.wso2.org - "Oxygenating the Web Services Platform"
>