Benchmark results -- follow-up

[raulvk]

Hi,

We are evaluating different options for Open Source ESBs and have come across the WSO2 benchmark results, which puts Mule in a terrible position. However, we haven't been able to find any strong counter-comments from the people who develop Mule.

Therefore, we have a few questions remaining:
  1) have the bugs found by the WSO2 team been repaired? How come they hadn't been encountered before running the benchmarks? They are pretty serious problems, it seems.
  2) has performance been improved since the results were published?

Anyway, I would appreciate some insight as to why the Mule community has made no move to defend their product, except for the following on the benchmark page itself:

Mule configuration help?
Submitted by ciurana on July 19, 2007 - 06:23.

Hi Asankha!

Interesting results -- thanks for performing the tests.

May I ask how well do you know how to configure Mule? Some of the testing issues that you're having sound like configuration and setup issues, rather than Mule problems. Can you please post the Mule XML configuration file so that we can take a look? Please annotate the file with "objective was X, we did Y, got Z instead" comments. It may be that the configuration you have set up could be optimized.

Thanks!

E

Thanks in advance for your insights.

[Ross Mason-3]

Hi,

Don't forget you are getting your information for a platform that  
wants to compete with Mule :) We were never able to run the benchmarks  
with the information provided since it's unclear how they ran each  
platform.  If fact they never originally published the benchmark code,  
but I see now it's available since Feb 11th.
The issue they had with keep alive is a configuration error as far as  
I can see, and Mule has supported web service proxying for years.

My suggestion is to take the two projects for a test drive if you are  
unsure.  I think you'll like Mule.  Mule has been running in  
production environments (including high-performance messaging) for  
over 4 years now.  It's been battle-tested more that any other open  
and closed-source ESB, and provides more functionality out of the box  
than the others.

As for defending Mule, it does a very good job of defending itself;  
just take it for a test drive.

btw, I'll ask our QA guys to see if there has been enough detail about  
this benchmark published so that we could run it. If so we will  
happily publish real results for Mule.

Cheers,

Ross Mason
CTO, Co-Founder
MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com





On 9 Apr 2008, at 09:25, raulvk wrote:

>
> Hi,
>
> We are evaluating different options for Open Source ESBs and have come
> across the WSO2 benchmark results, which puts Mule in a terrible  
> position.
> However, we haven't been able to find any strong counter-comments  
> from the
> people who develop Mule.
>
> Therefore, we have a few questions remaining:
>  1) have the bugs found by the WSO2 team been repaired? How come they
> hadn't been encountered before running the benchmarks? They are pretty
> serious problems, it seems.
>  2) has performance been improved since the results were published?
>
> Anyway, I would appreciate some insight as to why the Mule community  
> has
> made no move to defend their product, except for the following on the
> benchmark page itself:
>
>
>
>> Mule configuration help?
>> Submitted by ciurana on July 19, 2007 - 06:23.
>>
>> Hi Asankha!
>>
>> Interesting results -- thanks for performing the tests.
>>
>> May I ask how well do you know how to configure Mule? Some of the  
>> testing
>> issues that you're having sound like configuration and setup issues,
>> rather than Mule problems. Can you please post the Mule XML  
>> configuration
>> file so that we can take a look? Please annotate the file with  
>> "objective
>> was X, we did Y, got Z instead" comments. It may be that the  
>> configuration
>> you have set up could be optimized.
>>
>> Thanks!
>>
>> E
>>
>
> Thanks in advance for your insights.
>
> --
> View this message in context: http://www.nabble.com/Benchmark-results----follow-up-tp16590538p16590538.html
> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[raulvk]

Hi Ross,

Thank you for your answer. I think that the Mule configuration that the WSO2 people used during the benchmark is available here: http://wso2.org/repos/wso2/trunk/commons/performance/esb/mule-1.4.1/.

Could you have someone from your team look at it and let us know your findings? I am very interested in learning your views. To be honest, I think it is important that you do pronounce yourself as soon as possible if the configuration that was used was not optimal, because I'm sure there are many people who've been discouraged to have a further look at Mule due to the inferior results that it achieved on the benchmark (in our case, we were a bit reluctant, since transformation and keepalive connections are imperative in our project).

I'll be looking forward to hearing from you guys. You may contact me at raulvk [ at ] gmail [ dot ] com, if you prefer to speak in private. In the meantime, we will give Mule a test-drive for ourselves.

Thank you very much,

Raul.

Ross Mason-3 wrote:

Hi,

Don't forget you are getting your information for a platform that  
wants to compete with Mule :) We were never able to run the benchmarks  
with the information provided since it's unclear how they ran each  
platform.  If fact they never originally published the benchmark code,  
but I see now it's available since Feb 11th.
The issue they had with keep alive is a configuration error as far as  
I can see, and Mule has supported web service proxying for years.

My suggestion is to take the two projects for a test drive if you are  
unsure.  I think you'll like Mule.  Mule has been running in  
production environments (including high-performance messaging) for  
over 4 years now.  It's been battle-tested more that any other open  
and closed-source ESB, and provides more functionality out of the box  
than the others.

As for defending Mule, it does a very good job of defending itself;  
just take it for a test drive.

btw, I'll ask our QA guys to see if there has been enough detail about  
this benchmark published so that we could run it. If so we will  
happily publish real results for Mule.

Cheers,

Ross Mason
CTO, Co-Founder
MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com





On 9 Apr 2008, at 09:25, raulvk wrote:

>
> Hi,
>
> We are evaluating different options for Open Source ESBs and have come
> across the WSO2 benchmark results, which puts Mule in a terrible  
> position.
> However, we haven't been able to find any strong counter-comments  
> from the
> people who develop Mule.
>
> Therefore, we have a few questions remaining:
>  1) have the bugs found by the WSO2 team been repaired? How come they
> hadn't been encountered before running the benchmarks? They are pretty
> serious problems, it seems.
>  2) has performance been improved since the results were published?
>
> Anyway, I would appreciate some insight as to why the Mule community  
> has
> made no move to defend their product, except for the following on the
> benchmark page itself:
>
>
>
>> Mule configuration help?
>> Submitted by ciurana on July 19, 2007 - 06:23.
>>
>> Hi Asankha!
>>
>> Interesting results -- thanks for performing the tests.
>>
>> May I ask how well do you know how to configure Mule? Some of the  
>> testing
>> issues that you're having sound like configuration and setup issues,
>> rather than Mule problems. Can you please post the Mule XML  
>> configuration
>> file so that we can take a look? Please annotate the file with  
>> "objective
>> was X, we did Y, got Z instead" comments. It may be that the  
>> configuration
>> you have set up could be optimized.
>>
>> Thanks!
>>
>> E
>>
>
> Thanks in advance for your insights.
>
> --
> View this message in context: http://www.nabble.com/Benchmark-results----follow-up-tp16590538p16590538.html
> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[asankha]

Raul / Ross

Here are my comments..

Don't forget you are getting your information for a platform that wants to compete with Mule :) We were never able to run the benchmarks with the information provided since it's unclear how they ran each platform.  If fact they never originally published the benchmark code, but I see now it's available since Feb 11th.

I can confirm that we shared necessary information with Eugene Ciurana on this benchmark after the public comment here.. and offered to re-run the tests with any suggestions from Mule. I specifically stated "We would be glad to incorporate your suggestions and to retest and publish new sets of results" .. However, we never heard back from Eugene or anyone else! Except for this whining from Dave :-) (to which Sanjiva replied)

This  comment from Ruwan confirms that the Mule configuration was available publicly since 30th of August 2007 (I don't know why you stated Feb 11th date.. ?? whats significant about that date??)

The issue they had with keep alive is a configuration error as far as I can see, and Mule has supported web service proxying for years.

Well then, why didn't any body from Mule reply to this email on 8th July 2007?... actually no one has replied to this even upto now.. I don't think silence in the mailing lists is a good way to support your many 'production deployments' and 'clients'?..

My suggestion is to take the two projects for a test drive if you are unsure.  I think you'll like Mule.  Mule has been running in production environments (including high-performance messaging) for over 4 years now.  It's been battle-tested more that any other open and closed-source ESB, and provides more functionality out of the box than the others.

Let me quote Eugene again, Ref (http://wso2.org/blog/asankha/2263)

Recently on TSS, Eugene Ciurana was claiming that Mule was able to do 200 million transactions a day, and that no other ESB they evaluated, commercial or open source, could do that on the same hardware. This means that no ESB was able to perform 2315 TPS? I wish they would describe the scenario tested and the hardware used, instead of claiming such numbers out loud. For example a single instance of the WSO2 ESB / Synapse could do well over 3200 TPS with 40 concurrent users for some scenarios we've tested, on standard hardware (see 'Notes' here). I am almost sure that Eugene is yet to see the WSO2 ESB /Apache Synapse in action, cause they are not like the good ol' Mules that supposedly "Just works"... they work much faster and smarter! and they are extremely lightweight.

And Eugene replied to this publicly stating"I began looking at your software on Sunday 15.July, on a recommendation from Joe Ottinger... so your comment was accurate as of 12.July!" and we haven't heard from him since then..:)

I am also very interested to see that Mule actually contains code to recover even from a Java OOM Exception? I guess this validates your ability to run in production environments for over 4 years now :-)?

As for defending Mule, it does a very good job of defending itself; just take it for a test drive.

btw, I'll ask our QA guys to see if there has been enough detail about this benchmark published so that we could run it. If so we will happily publish real results for Mule.

Well, lets look at your own comments on Dain Hansen (BEA)'s blog from November 2006

"Dain, its very misleading to quote your benchmark figures in a comparison with figures for Mule that were quoted originally by a user without any context and certainly are not benchmark figures. We are working on publishing some figures by the end of the year, but in the meantime maybe you should edit your posting. Cheers, Ross

Posted by: rossmason on November 10, 2006 at 7:37 AM"

Its been over two (2) years! since this comment Ross.. but why don't we still see *any* performance benchmarks from Mule yet? Are you hiding something now ;-) ?



Raul, I agree with Ross on his comment that you should test drive both Mule and the WSO2 ESB (and/or Apache Synapse). Feel free to contact me for any help

thanks
asankha


References:
http://wso2.org/library/2259
http://wso2.org/esb

[ddossot]

Asankha,

I am also very interested to see that Mule actually contains code to recover even from a Java OOM Exception? I guess this validates your ability to run in production environments for over 4 years now :-)?

You realize that this kind of statement does not serve the purpose of this discussion. A smiley at the end of it does not make it acceptable either.

You should also realize that what is at stake here is not Mule's credibility but the credibility of a benchmark run by a particular vendor that makes it pass with flying colors. There is a long history of such benchmarks and nothing satisfying never came out of them, at least as far as the end users are concerned.

As an end user, I would rather see you guys, and other ESB vendors, engage a neutral and trusted 3rd party to run such a benchmark. This would help making this discussion less sterile and hopefully more professional.

Thank you,
David

[Gary S]

Asankha your comments fairly look baised. While we could not test close to a million transactions in a day ( did not have full day available) but We have tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically. Our configurations of course may not be optimal as we were still fine tuning our system. The WSO2 results have been published by they themselves and they are simply another competitor in the same space. To get a true analysis, it would have to be done by a third party nuetral company.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.

ddossot wrote:

Asankha,

I am also very interested to see that Mule actually contains
code<http://svn.codehaus.org/mule/trunk/mule/transports/file/src/main/java/org/mule/providers/file/transformers/FileToByteArray.java>to
recover even from a Java OOM
> Exception<http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/gdaog.html#gbzrr>?
> I guess this validates your ability to run in production environments for
> over 4 years now :-)?
>

You realize that this kind of statement does not serve the purpose of this
discussion. A smiley at the end of it does not make it acceptable either.

You should also realize that what is at stake here is not Mule's credibility
but the credibility of a benchmark run by a particular vendor that makes it
pass with flying colors. There is a long history of such benchmarks and
nothing satisfying never came out of them, at least as far as the end users
are concerned.

As an end user, I would rather see you guys, and other ESB vendors, engage a
neutral and trusted 3rd party to run such a benchmark. This would help
making this discussion less sterile and hopefully more professional.

Thank you,
David

[asankha]

David
> You should also realize that what is at stake here is not Mule's
> credibility but the credibility of a benchmark run by a particular
> vendor that makes it pass with flying colors. There is a long history
> of such benchmarks and nothing satisfying never came out of them, at
> least as far as the end users are concerned.
The benchmark we ran, and the configurations used were all made public
and the scenarios explained clearly, and even the tools (such as the
Java ApacheBench clone) shared. We requested anyone to help us fine tune
them, and offered to re-run them. We reported each problem we
encountered during the tests to the Mule team before we published any
results - but we did not get any responses to the issues. You can verify
these statements from http://wso2.org/blog/asankha/2263

> As an end user, I would rather see you guys, and other ESB vendors,
> engage a neutral and trusted 3rd party to run such a benchmark. This
> would help making this discussion less sterile and hopefully more
> professional.
See my note about Rosss' own comments on Dain Hansens' blog about Mule
benchmarking itself.. its been over two years since then, but Mule has
never published any performance results, and this only makes one wonder
as to why?

cheers
asankha

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[asankha]

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

[raulvk.soa]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.


To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.


  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

[Ruwan Linton]

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.



  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[Ross Mason-3]

Hi Ruwan,

Thanks for reaching out.  We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial.  I am surprised there is not an independent body for this yet.  I don't think anyone regards users as dumb (not sure why you said this) and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 13 Apr 2008, at 18:40, Ruwan Linton wrote:

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.



  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[raulvk.soa]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Ross,

Let me speak from the user's point of view. You are very right when you assert that vendors are not typically the best people to run benchmarks. However, I do think that this performance test has not been a *typical* one because usually vendors do not give their competitors a public chance to:
   a) correct possible errors/mistakes in the config, prior to publishing the results
   b) analyse and possibly fix the code used for the benchmarks
   c) re-run the benchmarks and re-publish a new, fresh set of results

On the other hand, to the WSO2 guys: I would appreciate it if you could provide us with code you used for the implementation of the SimpleStockQuoteService, unless it's the same you've package with the binary distribution of WSO2 ESB, in which case do let us know.

Thank you,

Raul.

On 14/04/2008, Ross Mason <themuleman@...> wrote:

Hi Ruwan,

Thanks for reaching out.  We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial.  I am surprised there is not an independent body for this yet.  I don't think anyone regards users as dumb (not sure why you said this) and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 13 Apr 2008, at 18:40, Ruwan Linton wrote:

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published

by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.




  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[Ruwan Linton]

Hi Ross,

Nice to see that you are going to run the performance tests.

On Mon, Apr 14, 2008 at 9:20 AM, Ross Mason <themuleman@...> wrote:

Hi Ruwan,

Thanks for reaching out.  We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial.  I am surprised there is not an independent body for this yet.  I don't think anyone regards users as dumb (not sure why you said this)

I told this, because you all tried to defend using statements on the mailing lists and using blogs rather than running the performance tests and publishing the performance figures.
 

and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.

Well, we did a quite a lot of research on these products (especially ServiceMix and Mule). We had one engineer per product working for weeks to get these configurations optimized and we asked for help over the public lists. For example, I was the one who was looking at the Service Mix and I read the full JBI spec before developing the SM configuration apart from there documents and the questions over the lists to get the SM configurations optimized.

Chathura is the one who did the Mule configuration and here are some questions that he posted to your lists;

http://markmail.org/message/t2as5sco5gct7gna
http://www.nabble.com/http-transport-problem-td11116197.html
http://osdir.com/ml/java.mule.devel/2007-07/msg00046.html
http://markmail.org/message/qzkpe36yocgpq6nr#query:+page:1+mid:4hnlmywwv4va62he+state:results
http://www.mail-archive.com/user@.../msg04358.html
http://markmail.org/message/mkrtjk3k2yj4mb4f#query:chathura%20mule%20XFire+page:1+mid:vz2hr5mpety7rm5c+state:results

So I think he did a good set of researches on getting the Mule configurations tuned for performance.

Thanks,
Ruwan
 

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 13 Apr 2008, at 18:40, Ruwan Linton wrote:

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published

by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.




  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[Ruwan Linton]

Hi Raul,

Yes. It is the same SimpleStockQuoteService distributed with the product under samples/axis2Server/src/SimpleStockQuoteService, except the System.out.println lines removed for better response time.

Or the code is available in the svn under;
http://svn.apache.org/repos/asf/synapse/trunk/java/modules/samples/services/SimpleStockQuoteService/

Thanks,
Ruwan

On Mon, Apr 14, 2008 at 2:41 PM, raulvk <raulvk.soa@...> wrote:

Hi Ross,

Let me speak from the user's point of view. You are very right when you assert that vendors are not typically the best people to run benchmarks. However, I do think that this performance test has not been a *typical* one because usually vendors do not give their competitors a public chance to:
   a) correct possible errors/mistakes in the config, prior to publishing the results
   b) analyse and possibly fix the code used for the benchmarks
   c) re-run the benchmarks and re-publish a new, fresh set of results

On the other hand, to the WSO2 guys: I would appreciate it if you could provide us with code you used for the implementation of the SimpleStockQuoteService, unless it's the same you've package with the binary distribution of WSO2 ESB, in which case do let us know.

Thank you,

Raul.

On 14/04/2008, Ross Mason <themuleman@...> wrote:

Hi Ruwan,

Thanks for reaching out.  We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial.  I am surprised there is not an independent body for this yet.  I don't think anyone regards users as dumb (not sure why you said this) and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 13 Apr 2008, at 18:40, Ruwan Linton wrote:

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published


by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.





  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[raulvk.soa]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Mule people,

Any news on this? Have you run the benchmarks?

We, users, are getting impacient to see results. Can you please give us an estimate as to when the results will be published, in case you don't have the results already?

Thank you,

Raul.

On 14/04/2008, Ruwan Linton <ruwan.linton@...> wrote:

Hi Ross,

Nice to see that you are going to run the performance tests.

On Mon, Apr 14, 2008 at 9:20 AM, Ross Mason <themuleman@...> wrote:

Hi Ruwan,

Thanks for reaching out.  We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial.  I am surprised there is not an independent body for this yet.  I don't think anyone regards users as dumb (not sure why you said this)

I told this, because you all tried to defend using statements on the mailing lists and using blogs rather than running the performance tests and publishing the performance figures.
 

and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.

Well, we did a quite a lot of research on these products (especially ServiceMix and Mule). We had one engineer per product working for weeks to get these configurations optimized and we asked for help over the public lists. For example, I was the one who was looking at the Service Mix and I read the full JBI spec before developing the SM configuration apart from there documents and the questions over the lists to get the SM configurations optimized.

Chathura is the one who did the Mule configuration and here are some questions that he posted to your lists;

http://markmail.org/message/t2as5sco5gct7gna
http://www.nabble.com/http-transport-problem-td11116197.html
http://osdir.com/ml/java.mule.devel/2007-07/msg00046.html
http://markmail.org/message/qzkpe36yocgpq6nr#query:+page:1+mid:4hnlmywwv4va62he+state:results
http://www.mail-archive.com/user@.../msg04358.html
http://markmail.org/message/mkrtjk3k2yj4mb4f#query:chathura%20mule%20XFire+page:1+mid:vz2hr5mpety7rm5c+state:results

So I think he did a good set of researches on getting the Mule configurations tuned for performance.

Thanks,
Ruwan
 

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 13 Apr 2008, at 18:40, Ruwan Linton wrote:

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published


by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.





  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[Ross Mason-3]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Raul,

We currently have a lot going on and running these benchmarks (that only test a small subset of Mule's functionality) is not going to jump the queue. We will get to it.  However, this is an open source project so you have everything available to you to make your own assessment.

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 21 Apr 2008, at 02:34, raulvk wrote:

Hi Mule people,

Any news on this? Have you run the benchmarks?

We, users, are getting impacient to see results. Can you please give us an estimate as to when the results will be published, in case you don't have the results already?

Thank you,

Raul.

On 14/04/2008, Ruwan Linton <ruwan.linton@...> wrote:

Hi Ross,

Nice to see that you are going to run the performance tests.

On Mon, Apr 14, 2008 at 9:20 AM, Ross Mason <themuleman@...> wrote:

Hi Ruwan,

Thanks for reaching out.  We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial.  I am surprised there is not an independent body for this yet.  I don't think anyone regards users as dumb (not sure why you said this)

I told this, because you all tried to defend using statements on the mailing lists and using blogs rather than running the performance tests and publishing the performance figures.
 

and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.

Well, we did a quite a lot of research on these products (especially ServiceMix and Mule). We had one engineer per product working for weeks to get these configurations optimized and we asked for help over the public lists. For example, I was the one who was looking at the Service Mix and I read the full JBI spec before developing the SM configuration apart from there documents and the questions over the lists to get the SM configurations optimized.

Chathura is the one who did the Mule configuration and here are some questions that he posted to your lists;

http://markmail.org/message/t2as5sco5gct7gna
http://www.nabble.com/http-transport-problem-td11116197.html
http://osdir.com/ml/java.mule.devel/2007-07/msg00046.html
http://markmail.org/message/qzkpe36yocgpq6nr#query:+page:1+mid:4hnlmywwv4va62he+state:results
http://www.mail-archive.com/user@.../msg04358.html
http://markmail.org/message/mkrtjk3k2yj4mb4f#query:chathura%20mule%20XFire+page:1+mid:vz2hr5mpety7rm5c+state:results

So I think he did a good set of researches on getting the Mule configurations tuned for performance.

Thanks,
Ruwan
 

Cheers,

Ross Mason

CTO, Co-Founder

MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com

On 13 Apr 2008, at 18:40, Ruwan Linton wrote:

Hi Mule devs,

I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.

If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.

Thanks,
Ruwan

On Sun, Apr 13, 2008 at 6:15 PM, raulvk <raulvk.soa@...> wrote:

Hi everyone,

First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.

I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.

To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.

Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.

Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.

Raul.

On 13/04/2008, Asankha C. Perera <asankha@...> wrote:

Gary

Asankha your comments fairly look baised.

I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.

 While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.

Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?

 Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published


by they themselves and they are simply another competitor in the same space.



To get a true analysis, it would have to be done by a third party nuetral
company.

The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.

The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.





  

Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact (http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.

asankha

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"

[VE]

Hi,

I was wondering if you guys (mulesource) had got anywhere with this?  If so, any ideas when you will be publishing your results?  Frankly i'm surprised it's taking this long for you to counter WSO2 tests.

I have recently performed perfermonce tests and wanted to cross verify.

Thanks

Ross Mason-3 wrote:

Hi Raul,

We currently have a lot going on and running these benchmarks (that  
only test a small subset of Mule's functionality) is not going to jump  
the queue. We will get to it.  However, this is an open source project  
so you have everything available to you to make your own assessment.

Cheers,

Ross Mason
CTO, Co-Founder
MuleSource Inc.

http://mulesource.com | http://blog.rossmason.com





On 21 Apr 2008, at 02:34, raulvk wrote:

> Hi Mule people,
>
> Any news on this? Have you run the benchmarks?
>
> We, users, are getting impacient to see results. Can you please give  
> us an estimate as to when the results will be published, in case you  
> don't have the results already?
>
> Thank you,
>
> Raul.
>
>
> On 14/04/2008, Ruwan Linton <[e-mail]> wrote:
> Hi Ross,
>
> Nice to see that you are going to run the performance tests.
>
> On Mon, Apr 14, 2008 at 9:20 AM, Ross Mason <[e-mail]>  
> wrote:
> Hi Ruwan,
>
> Thanks for reaching out.  We will run these benchmarks along side  
> our own and publish the results. I do agree with David Dossot here  
> though in that vendors are not the best people to run benchmarks  
> since they are not impartial.  I am surprised there is not an  
> independent body for this yet.  I don't think anyone regards users  
> as dumb (not sure why you said this)
>
> I told this, because you all tried to defend using statements on the  
> mailing lists and using blogs rather than running the performance  
> tests and publishing the performance figures.
>
> and while users like empirical data they also will not accept  
> benchmark results from a single vendor without doing their own  
> research. That was what I was pushing for in my original response to  
> this thread.
>
> Well, we did a quite a lot of research on these products (especially  
> ServiceMix and Mule). We had one engineer per product working for  
> weeks to get these configurations optimized and we asked for help  
> over the public lists. For example, I was the one who was looking at  
> the Service Mix and I read the full JBI spec before developing the  
> SM configuration apart from there documents and the questions over  
> the lists to get the SM configurations optimized.
>
> Chathura is the one who did the Mule configuration and here are some  
> questions that he posted to your lists;
>
> http://markmail.org/message/t2as5sco5gct7gna
> http://www.nabble.com/http-transport-problem-td11116197.html
> http://osdir.com/ml/java.mule.devel/2007-07/msg00046.html
> http://markmail.org/message/qzkpe36yocgpq6nr#query:+page:1+mid:4hnlmywwv4va62he+state:results
> http://www.mail-archive.com/[e-mail]/msg04358.html
> http://markmail.org/message/mkrtjk3k2yj4mb4f#query:chathura%20mule 
> %20XFire+page:1+mid:vz2hr5mpety7rm5c+state:results
>
> So I think he did a good set of researches on getting the Mule  
> configurations tuned for performance.
>
> Thanks,
> Ruwan
>
>
> Cheers,
>
> Ross Mason
> CTO, Co-Founder
> MuleSource Inc.
>
> http://mulesource.com | http://blog.rossmason.com
>
>
>
>
>
> On 13 Apr 2008, at 18:40, Ruwan Linton wrote:
>
>> Hi Mule devs,
>>
>> I think Raul is exactly correct on this. You should have a look at  
>> the configurations (I am sure you already had a deep look in to  
>> those :-)) that we used in doing the performance tests and improve  
>> the Mule once to optimize the configurations for performance and re-
>> run the performance tests. This will enable the users to understand  
>> the performance benchmarks of the two products very easily.
>>
>> If you need any help in getting Synapse configurations and the  
>> product optimized for performance or any information on setting up  
>> the environment we would like to incorporate with you. So don't  
>> waist any more time and get started with the performance tests. You  
>> should bring the counter arguments over Synapse / WSO2 ESB with  
>> facts in your hand, otherwise I don't think users will accept those  
>> (You should keep in mind that users are not dumb). If you can  
>> perform faster than Synapse you don't have to worry at all just  
>> show the figures.
>>
>> Thanks,
>> Ruwan
>>
>> On Sun, Apr 13, 2008 at 6:15 PM, raulvk <[e-mail]> wrote:
>> Hi everyone,
>>
>> First of all, I'd like to state that I am researching the use of an  
>> open source ESB with the uttermost neutrality: our client had  
>> determined that they want an open source ESB to serve as their  
>> integration backbone, but they have left it up to us to pick the  
>> most appropriate one.
>>
>> I seems that this discussion has deviated and ended up turning into  
>> a war between vendors, which doesn't serve us prospective end users  
>> at all. My aim when I wrote the first message was to throw some  
>> light on why Mule hadn't apparently pronounced themselves on the  
>> results.
>>
>> To the Mule people, even though the benchmark was run by another  
>> vendor, the code was released publicly and help was requested on  
>> the forums. So there has been a high degree of visibility into the  
>> benchmark process. Of course, the WSO2 team are experts in Apache  
>> Synapse, which enables them to configure Synapse optimally. That is  
>> the only "unfairness" factor that I perceive, but once again, they  
>> asked for help on the forums when it came to Mule's config.
>>
>> Anyway, bottom line is that the only method to address this  
>> discussion *productively* is for the Mule team to look at the  
>> published benchmark code and point out *where* the configuration  
>> mistakes that Ross mentions are, and how they can be fixed. We  
>> could then run a new round of benchmarks with an optimised Mule  
>> config.
>>
>> Please consider that the constant goal of both vendors is to gain a  
>> larger user base, and *we users* are moved by *facts and figures*,  
>> so let's bring them to the table instead of criticising each other.
>>
>> Raul.
>>
>>
>>
>> On 13/04/2008, Asankha C. Perera <[e-mail]> wrote:
>> Gary
>>> Asankha your comments fairly look baised.
>> I must say that I am a lead committer of Apache Synapse and lead  
>> the development of the WSO2 ESB; And I ran the benchmarks against  
>> Mule, ServiceMix and a leading commercial ESB and published the  
>> results.
>>>  While we could not test close to a million transactions in a day  
>>> ( did not have full day available) but We have
>>> tested Mule with close to a 100000 transactions in 4-5 hours and  
>>> it performed fanstatically.
>> Interesting.. this translates to just over 6 TPS. Can you explain  
>> the scenarios in more detail?
>>>  Our configurations of course may not be optimal as
>>> we were still fine tuning our system. The WSO2 results have been  
>>> published
>>>
>>>
>>> by they themselves and they are simply another competitor in the  
>>> same space.
>>>
>>>
>>>
>>> To get a true analysis, it would have to be done by a third party  
>>> nuetral
>>> company.
>> The reality is you can wait till this "neutral third party" comes  
>> up with an independent benchmark for ESB's, or you can run your  
>> own. What I believe is that as long as you clearly explain the  
>> scenarios, make any code/configuration and tools you use public  
>> (and free), and also ask each vendor for assistance when you find  
>> issues and give them an opportunity to reply, you are good.
>>> The fact that WSO2 is basically trying to bench mark against mule  
>>> suggests in iteslf that WSO2 acknowledges that Mule may be "The  
>>> Standard" to beat. Better to go with the "The Standard" I guess.
>>>
>>>
>>>
>>>
>>>
>>>
>> Oh no! The first ESB we benchmarked against was a commercial closed  
>> source ESB (See http://wso2.org/library/1721). It was only after we  
>> published those results that we ran the benchmark a second time.  
>> And we included Apache ServiceMix, and ofcourse Mule. We considered  
>> OpenESB and JBoss ESB as well, but due to many issues we faced, we  
>> compared only Apache SM and Mule. You may note that Apache  
>> ServiceMix had very interesting results and did better in  
>> transformations initially! And we did not hide this fact (http://wso2.org/library/2259 
>> ), We also reported the issue we faced against SM and they looked  
>> into it unlike Mule.
>>
>> asankha
>>
>>
>>
>>
>> --
>> Ruwan Linton
>> http://www.wso2.org - "Oxygenating the Web Services Platform"
>
>
>
>
> --
> Ruwan Linton
> http://www.wso2.org - "Oxygenating the Web Services Platform"
>

[chuckanut]

Can you share your results?

On Thu, May 22, 2008 at 2:07 AM, VE <lahrasub@...> wrote:
>

> I have recently performed perfermonce tests
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Ruwan Linton]

Hi Ross and all,

In case if you want to have a look at the performance comparisons of latest versions of the ESBs including

• a proprietary ESB
• Apache Service Mix 3.2.1
• Mule Community Edition 2.0.1 (Mule CE)
• a proprietary version of an open source ESB
  
• WSO2 ESB 1.7 / Apache Synapse 1.2

here is the third round of the performance test results [1].

BTW: Ross, were you able to run any of the performance tests?

Thanks,
Ruwan

[1] - http://wso2.org/library/3740

--
Ruwan Linton
http://wso2.org - "Oxygenating the Web Services Platform"
http://ruwansblog.blogspot.com/

[pangea]

We are also doing POC for quite a while and are interested to see the benchmark results...Also do you guys have any plans of updaing your documentation with more content? For an average developer the current documentation is not sufficient!