Re: Unable to customize threading in synchronized mode

• The only way I can make the model scale to more than 4 simultaneous instances is to set synchronous to false. Setting synchronous to true overrides every configuration made regarding threading behaviour and limits the execution to 4 simultaneous instances. I need to get the response back so I must have a synchronous model definition, but in addition I must be able to scale the application to more than 4 simultaneous instances.
  

Hi,

Sorry for the late reply.

I have done some more testing and succesfully configured a model using response-router and reply-to. Here follows the configuration:
<mule-configuration id="testApp" version="1.0">

    <mule-environment-properties serverUrl="">
       <threading-profile id="default" maxThreadsActive="100"
           maxThreadsIdle="100" threadTTL="60000" poolExhaustedAction="RUN"
           threadWaitTimeout="60000" maxBufferSize="1" doThreading="true" />
   </mule-environment-properties>

    <connector name="HttpConnector"
       className="org.mule.providers.http.HttpConnector">
       <properties>
           <map name="serviceOverrides">
               <property name="inbound.transformer"
                   value="no.fast.pf.pluto.transformers.EmptyTransformer" />
           </map>
       </properties>
   </connector>
   
    <transformers>
       <transformer name="EmptyTransformer" className="no.fast.pf.pluto.transformers.EmptyTransformer"/>
   </transformers>

    <model name="PlutoServer">
       <mule-descriptor name="QRServer"
           implementation="org.mule.components.simple.PassThroughComponent">
           <inbound-router>
               <endpoint address="http://localhost:11100" synchronous="false" connector="HttpConnector" />
           </inbound-router>

            <outbound-router>
               <router
                   className="org.mule.routing.outbound.OutboundPassThroughRouter">
                   <endpoint address="vm://responserouter" synchronous="true"></endpoint>
               </router>
           </outbound-router>

            <response-router>
               <endpoint address="vm://response"/>
               <router className="org.mule.routing.response.SingleResponseRouter">
               </router>
           </response-router>

        </mule-descriptor>
       
        <mule-descriptor name="responseRouter" implementation="org.mule.components.simple.EchoComponent">
           <inbound-router>
               <endpoint address="vm://responserouter"/>
           </inbound-router>
           
            <outbound-router>
                <router className="org.mule.routing.outbound.OutboundPassThroughRouter">
                   <endpoint address="vm://nullendpoint" />
                   <reply-to address="vm://response"/>
               </router>
           </outbound-router>
           
        </mule-descriptor>
       
        <mule-descriptor name="nullD" implementation="org.mule.components.simple.EchoComponent">
           <inbound-router>
               <endpoint address="vm://nullendpoint">
               </endpoint>
           </inbound-router>
       </mule-descriptor>
   </model>

</mule-configuration>

As you can see the only custom code is still the “EmptyTransformer”. Which does a sleep for 10 seconds (see earlier mail in thread). To summarize a lot of troubleshooting the behaviour is exactly the same: having the synchronous to “false” makes to model scale to the configured theading-profile, while having the synchronous to “true” limits the model to only 4 simultaneous instances. To be more precise it is the number of simultaneous http instances (which has the EmtpyTransformer as serviceoverride) that are limited to 4 simultaneous instances. I must use synchronous=”true” or else the response is NOT posted back to the client. From the log I can see that the response is always posted to the response-router (vm://response) regardless of whether I use synchronous=true or false, however to connection is immediately dropped after posting the request when the inbound http endpoint is configured as synchronous=”false”. And when the connection is dropped it does not matter whether the model posts the response back to the response-router as there isn’t any connection to the client anymore. As I have understood after this session of troubleshooting is that it’s crucial for the inbound endpoint to call doSend() to be able to “give” something back to the client, and for the inbound endpoint to call doSend() the synchronous bit must somehow be set to “true” which then limits the number of simultaneous connector instances to 4 (a limitation I don’t want to happen).

Whether I am using a response-router with a matching reply-to (as in the configuration posted above) or whether I use the simple synchronous model posted in earlier mail, it doesn’t really matter. This because the number of instances is already limited by the inbound http endpoint when it’s set to synchronous=”true”. And how the model looks like beyond this point doesn’t really matter as the limitation is already done at the inbound level.

I also did another test to assure that the limitation is done on the inbound http endpoint with the following ultra-simple model definition:
<mule-configuration id="testApp" version="1.0">

    <mule-environment-properties serverUrl="">
       <threading-profile id="default" maxThreadsActive="100"
           maxThreadsIdle="100" threadTTL="60000" poolExhaustedAction="RUN"
           threadWaitTimeout="60000" maxBufferSize="1" doThreading="true" />
   </mule-environment-properties>

    <connector name="HttpConnector"
       className="org.mule.providers.http.HttpConnector">
       <properties>
           <map name="serviceOverrides">
               <property name="inbound.transformer"
                   value="no.fast.pf.pluto.transformers.EmptyTransformer" />
           </map>
       </properties>
   </connector>
   
    <transformers>
       <transformer name="EmptyTransformer" className="no.fast.pf.pluto.transformers.EmptyTransformer"/>
   </transformers>

    <model name="PlutoServer">
       <mule-descriptor name="QRServer"
           implementation="org.mule.components.simple.PassThroughComponent">
           <inbound-router>
               <endpoint address="http://localhost:11100" synchronous="false" connector="HttpConnector" />
           </inbound-router>

            <pooling-profile exhaustedAction="FAIL" maxActive="4" maxWait="4"/>
       </mule-descriptor>
    </model>

</mule-configuration>

Here you can see that I don’t even have any outbound endpoint to the only mule-descriptor defined in the model. But I have configured a pooling-profile for the component. The “EmptyTransformer” still exists wich only performs a sleep. When configuring the inbound endpoint as synchronous=true I get no errors and 4 “WAITING START” written due to the “EmptyTransformer” (only 4 instances of the inbound endpoint). I have set the component pool to FAIL if the pool is asked for more than 4 instances at a time. I get no errors and the code performs as expected with responses coming back to the client, but no more than 4 simultaneous instances. When I set synchronous=false, I get the following exception (which is expected and supports the fact that the limitation to 4 instances is done on the inbound http endpoint):
Exception stack is:
1. Pool exhausted (java.util.NoSuchElementException)  org.apache.commons.pool.impl.GenericObjectPool:938 (http://java.sun.com/j2se/1.5.0/docs/api/java/util/NoSuchElementException.html) 2. Proxy pool timed out. Component that caused exception is: QRServer. Message payload is of type: String (org.mule.umo.ComponentException)  org.mule.impl.model.seda.SedaComponent:499 (http://mule.mulesource.org/docs/apidocs/org/mule/umo/ComponentException.html) ******************************************************************************** Root Exception stack trace: java.util.NoSuchElementException: Pool exhausted    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:938)    at org.mule.config.pool.CommonsPoolProxyPool.borrowObject(CommonsPoolProxyPool.java:111)    at org.mule.impl.model.seda.SedaComponent.getProxy(SedaComponent.java:544)    at org.mule.impl.model.seda.SedaComponent.run(SedaComponent.java:480)    at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)    at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)    at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)    at java.lang.Thread.run(Thread.java:613)

The component pool is exhausted because according to the threading profile there should be up to 100 simultaneous threads. And this behaviour is expected because I have limited the component pool to 4 and set it to “FAIL” if exhausted. The fact that this error only occurs when the inbound endpoint is set to synchronous=false is not consistent. The error should also happen if the synchronous but is set to true because we are still configuring 100 threads and component-pool to 4. This means the component pool is never asked for more than 4 instances when the inbound endpoint is set to synchronous=true.

To me this looks like a bug and I can’t find any bugreports or tickets associated to this scenario even if it’s a quite basic http proxy use-case. I will test the same type of model in mule 2 and see if this reproduces in the recent version of mule.

Anybody with thoughts or suggestions, please comment.

Thanks in advance,
Vikram

On 23/12/2008 09:48, "Antoine Borg" <antoine.borg@...> wrote:

> Hi,
>  
> Where is your outbound-endpoint?
>  
> A
>  
> Antoine Borg, Senior Consultant | Tel: +32 28 504 696
> ricston Ltd., BP 2, 1180 Uccle, Brussels, BELGIUM
> email: antoine.borg@... <antoine.borg@...>  | blog:
> blog.ricston.com <http://blog.ricston.com/>  | web: ricston.com
> <http://www.ricston.com/>  
>  
>
>
> From: Vikramjeet Singh Jassal [vikramsj@...]
> Sent: Tuesday, December 09, 2008 11:19 AM
> To: user@...
> Subject: Re: [mule-user] Unable to customize threading in synchronized mode
>
> Hi,
>
> I have not tried using a response router and my code does not require it
> either. But since you suggested it I give it a shot. I did a test using the
> following configuration:
>
> <mule-configuration id="testApp" version="1.0">    
> <mule-environment-properties serverUrl="">        <threading-profile
> id="default" maxThreadsActive="100"            maxThreadsIdle="100"
> threadTTL="60000" poolExhaustedAction="RUN"            
> threadWaitTimeout="60000" maxBufferSize="0" doThreading="true" />    
> </mule-environment-properties>    <connector name="HttpConnector"        
> className="org.mule.providers.http.HttpConnector">        <properties>            
> <map name="serviceOverrides">                <property
> name="inbound.transformer"                    
> value="no.fast.pf.pluto.transformers.EmptyTransformer" />            </map>        
> </properties>    </connector>    <model name="PlutoServer">        
> <mule-descriptor name="QRServer"            
> implementation="org.mule.components.simple.EchoComponent">            
> <inbound-router>                <endpoint address="http://localhost:11100"
> synchronous="false" remoteSync="false"                    
> connector="HttpConnector">                </endpoint>            
> </inbound-router>            <outbound-router>                <router                    
> className="org.mule.routing.outbound.OutboundPassThroughRouter">                    
> <reply-to address="vm://response" />                </router>            
> </outbound-router>            <response-router>                <endpoint
> address="vm://response" />            </response-router>        
> </mule-descriptor>    </model> </mule-configuration>
>
> This works, but is still asynchronous. The reply is not posted back to the
> client.
> Any other suggestions? Thanks for your time.
>
> Vikram
>
> On 09/12/2008 08:32, "Antoine Borg" <antoine.borg@...> wrote:
>
>> Hi,
>>
>> So case 2 (everything is asynchronous) works  best but does not return a
>> response.  Have you tried using a response  router?  I know that the config
>> you posted below may not need/use a  response router but you do mention that
>> your live code is a little more  complex.
>>
>> HTH
>>
>> Antoine Borg, Senior Consultant | Tel: +32 28 504 696  
>> ricston  Ltd., BP 2, 1180 Uccle, Brussels, BELGIUM
>> email:  antoine.borg@... <antoine.borg@...>  | blog:  
>> blog.ricston.com <http://blog.ricston.com/>  | web: ricston.com
>> <http://www.ricston.com/>   
>>
>>
>>  
>>
>>  From: Vikramjeet Singh Jassal  [vikramsj@...]  
>> Sent: Monday, December 08, 2008 1:01 PM
>> To: user@...
>> Subject:  Re: [mule-user] Unable to customize threading in synchronized  mode
>>
>> Off  course,
>> Maybe I was a little unclear there.
>> I have tried all the  following:
>>  
>> 1. remoteSync="false" synchronous="true"  
>> 2. remoteSync="false" synchronous="false"  
>> 3. remoteSync="true" synchronous="false"  
>> 4. remoteSync="true" synchronous="true"
>>
>> And case 1, 3 and 4 gives the same result: only 4  simultaneous instances,
>> while case 2 makes number of instances customizable  through
>> pooling/threading-profiles (scaling correctly) but doesn’t give me the  
>> response back which is crucial for my  use-case.
>>
>> Thanks,
>> Vikram
>>
>> On 03/12/2008 11:42, "Antoine Borg"  <antoine.borg@...>  wrote:
>>
>>  
>>> I don't mean to be annoying but in the  config  you pasted below, remoteSync
>>> is set to false .. have you tried  setting it to  true?
>>>
>>> A
>>>
>>> Antoine Borg, Senior Consultant | Tel: +32 28 504 696   
>>> ricston  Ltd., BP 2, 1180 Uccle, Brussels,  BELGIUM
>>> email:  antoine.borg@... <antoine.borg@...>   | blog:   
>>> blog.ricston.com  <http://blog.ricston.com/>   | web:  ricston.com
>>> <http://www.ricston.com/>    
>>>
>>>
>>>  
>>>  
>>>
>>>  From: Vikramjeet Singh  Jassal  [vikramsj@...]   
>>> Sent: Wednesday, November 26, 2008 12:40 PM
>>> To:  user@...
>>> Subject:   Re: [mule-user] Unable to customize threading in synchronized   
>>> mode
>>>
>>> Hi   again,
>>>
>>> Yes I have tried the remoteSync attribute too. It gives   identical
>>> behaviour as the synchronized attribute. I get the responses  back,  but no
>>> more than 4 simultaneous instances.
>>>
>>> Any other   suggestions?
>>>
>>> Thanks,
>>> Vikram
>>>
>>>
>>> On 25/11/2008 09:19,  "Antoine  Borg" <antoine.borg@...>   wrote:
>>>
>>>  
>>>  
>>>> Hi,
>>>>
>>>> Have you tried using the remoteSync   attribute  on the endpoint?
>>>>
>>>> http://ricston.com/blog/?p=115    <http://ricston.com/blog/?p=115>    
>>>>
>>>>  
>>>> A
>>>>
>>>> Antoine Borg, Senior Consultant | Tel: +32 28 504  696   
>>>> ricston  Ltd., BP 2, 1180  Uccle, Brussels,  BELGIUM
>>>> email:   antoine.borg@... <antoine.borg@...>    |
>>>> blog:    blog.ricston.com  <http://blog.ricston.com/>    | web:   
>>>> ricston.com <http://www.ricston.com/>     
>>>>
>>>>
>>>>  
>>>>  
>>>>  
>>>>
>>>>  From: Vikramjeet Singh   Jassal  [vikramsj@...]    
>>>> Sent: Thursday, November 20, 2008 1:48  PM
>>>> To: user@...
>>>> Subject:    [mule-user] Unable to customize threading in synchronized   
>>>> mode
>>>>
>>>> Hi,
>>>>
>>>> I have  implemented  a  simple http proxy which does some custom  
>>>> modifications to the  querystring  before submitting it to a  backend. My
>>>> problem is that I am  not able to make  mule execute  more than 4 instances
>>>> of this model  simultaneously. It’s crucial   to get the response back so I
>>>> have  configured the http inbound  endpoint to be  synchronized.After
>>>> trying  out both 1.4.1 and  1.4.4 releases without luck, I  have
>>>> implemented a  really  simple mule-model to debug this scenario. Here is my    
>>>> configuration:
>>>>
>>>> <mule-configuration id="testApp" version="1.0">
>>>>
>>>>     <mule-environment-properties
>>>>      serverUrl="">
>>>>     <threading-profile  id="default"  maxThreadsActive="100"
>>>>          maxThreadsIdle="100" threadTTL="60000" poolExhaustedAction="RUN"
>>>>          threadWaitTimeout="60000"    maxBufferSize="0"  doThreading="true"   
>>>> />
>>>> </mule-environment-properties>
>>>>
>>>>      <connector name="HttpConnector"
>>>> className="org.mule.providers.http.HttpConnector" >
>>>> <properties>
>>>>     <map name="serviceOverrides">
>>>>         <property name="inbound.transformer"
>>>>              value="no.fast.pf.pluto.transformers.EmptyTransformer"  />
>>>>     </map>
>>>> </properties>
>>>> </connector>
>>>>
>>>>     <model name="PlutoServer">
>>>>      <mule-descriptor name="QRServer"
>>>> implementation="org.mule.components.simple.EchoComponent">
>>>>         <inbound-router>
>>>>             <endpoint address="${server.qrserver}"    
>>>> connector="HttpConnector"  
>>>>                              remoteSync="false" synchronous="true">
>>>>             </endpoint>
>>>>         </inbound-router>                   
>>>>         </mule-descriptor>
>>>>     </model>
>>>>
>>>> </mule-configuration>    
>>>>
>>>> As you can see the only custom code is the    
>>>> no.fast.pf.pluto.transformers.EmptyTransformer.   Here  is the code:
>>>> public class EmptyTransformer extends   AbstractTransformer{      @Override      
>>>> protected Object  doTransform(Object arg0,  String  arg1) throws
>>>> TransformerException {           System.err.println("WAITING   START");         
>>>> try {               Thread.sleep(10000);           } catch   
>>>> (InterruptedException e) {               e.printStackTrace();           }           
>>>> System.err.println("WAITING   DONE");         return arg0;      }  }
>>>>
>>>> I have tried to find out where  the  limitation to 4 instances is  actually
>>>> done, so I  configured this  transformer to debug this as early as  
>>>> possible  in the pipe. The  transformer simply writes out a message and
>>>> sleeps   for 10 seconds.  This makes it possible to monitor how many  
>>>> instances that  prints out  “WAITING START” before “WAITING  DONE” starts
>>>> popping up. No matter  how  i configure the  thread-model and other pools,
>>>> I am not able to have more   than  4 “WAITING START” in a row. I have tried
>>>> several parameters on  the  http  connector as it seems the limitation to 4
>>>> instances is done   already on the  inbound side before reaching the
>>>> component.  These are  the switches I have  tried out on the connector with  
>>>> no  luck:
>>>> <!--            <property    name="keepSendSocketOpen" value="true"/>           
>>>> <property    name="numberOfConcurrentTransactedReceivers" value="40"/>           
>>>> <property    name="numberOfConcurrentTransactedDispatchers" value="40"/>           
>>>> <property    name="maxDispatchersActive" value="40"/>           <property    
>>>> name="maxReceiversActive" value="40"/>
>>>> ->
>>>>
>>>> I  have also   configured pooling-profile and threading-profile  (with
>>>> id=component |  receiver  | dispatcher | default) directly  inside the
>>>> mule-descriptor  without  luck.
>>>> HOWEVER, most  important observation is that only way  I can make the  
>>>> model  follow the configurations I make in the  threading-profile is to  
>>>> change  from synchronous=true to  synchronous=false. Then I get  as many
>>>> “WAITING START”  in a row as I  have configured in the  threading-profile
>>>> or as many as  http-requests I  issue. This I  have succesfully scaled up
>>>> to 40 and 50 in a row  without  any  problem. BUT it’s crucial for me to
>>>> get the response back to the    client calling the model, so I have to set
>>>> synchronous to  true. I hope  someone  who have tried out similar stuff
>>>> could  guide me in the right  direction here  for scaling the model  above
>>>> 4 simultaneous  instances.
>>>>
>>>> Thanks in   advance,
>>>> Vikram  
>>>
>>
>