TCP protocol acknowledgement delay(looks like piggybacking with the message response)

[Erinc Arikan]

Hi everybody;

I implemented a byte based TCP protocol, To accomplish that I implemented my
own protocolcodecfilter to decode byte stream into POJO's. My Protocol
requires me to send an ACK/NAK whenever I receive a complete, meaningful
message. I thought that I could write ACK directly to session in my decoder
right after I am done with decoding but I couldn't make it work.

So instead of doing that, I pass my POJO's to my custom message handler
where all my business logic lies. And before processing POJO, first thing I
do is writing the ACK back to session using session.write method. And I
start processing the message after that. But it looks like my ACKs never get
to the client right away, most of the time they wait for the response of
actual message , and will be delivered to client together(they only have 200
ms difference).

This is a huge problem for me because there's no point in having the ACK
when you experience this issue. By the time they have the ACK, they will get
the response anyway.

I am wondering if there's a way to guarantee transmission of the ACK in a
non blocking manner, without waiting for the actual message response. Mostly
I am processing credit card authorizations which is a synchronous process
and can take up to 4-5 seconds. I add them to ACTIVEMQ, payment processor
picks them up.  Definitely I want to be able to send the ack right away
without waiting for 5 seconds.

Any thoughts on how to do that?

[Alex Shneyderman]

config.getSessionConfig().setTcpNoDelay(true);

this is what I have for the connector but I am sure acceptor will have
the same sort of setting.

On Tue, Nov 3, 2009 at 9:37 PM, Erinc Arikan <erincarikan@...> wrote:

> Hi everybody;
>
> I implemented a byte based TCP protocol, To accomplish that I implemented my
> own protocolcodecfilter to decode byte stream into POJO's. My Protocol
> requires me to send an ACK/NAK whenever I receive a complete, meaningful
> message. I thought that I could write ACK directly to session in my decoder
> right after I am done with decoding but I couldn't make it work.
>
> So instead of doing that, I pass my POJO's to my custom message handler
> where all my business logic lies. And before processing POJO, first thing I
> do is writing the ACK back to session using session.write method. And I
> start processing the message after that. But it looks like my ACKs never get
> to the client right away, most of the time they wait for the response of
> actual message , and will be delivered to client together(they only have 200
> ms difference).
>
> This is a huge problem for me because there's no point in having the ACK
> when you experience this issue. By the time they have the ACK, they will get
> the response anyway.
>
> I am wondering if there's a way to guarantee transmission of the ACK in a
> non blocking manner, without waiting for the actual message response. Mostly
> I am processing credit card authorizations which is a synchronous process
> and can take up to 4-5 seconds. I add them to ACTIVEMQ, payment processor
> picks them up.  Definitely I want to be able to send the ack right away
> without waiting for 5 seconds.
>
> Any thoughts on how to do that?
>

[Fredrik Jonson-2]

Erinc Arikan wrote:
 
>  I implemented a byte based TCP protocol, To accomplish that I implemented my
>  own protocolcodecfilter to decode byte stream into POJO's. But it looks like
>  my ACKs never get to the client right away, most of the time they wait for
>  the response of actual message, and will be delivered to client together.

Sounds like you're protocol could have an issue with the tcp nagle algorithm?
Try SocketSessionConfig.setTcpNoDelay(true) on the connector.

http://mina.apache.org/faq.html

--
Fredrik Jonson

[Erinc Arikan]

When I set tcpNodelay to true of socketSessionConfig of the server's
acceptor. I see the exact same timestamp for ACK and Response message in
client logs. But when I set it to false I see 150-200 ms between ACK and
message response although message processing on average takes around 800 ms

On Tue, Nov 3, 2009 at 3:48 PM, Fredrik Jonson <fredrik@...> wrote:

> Erinc Arikan wrote:
>
> >  I implemented a byte based TCP protocol, To accomplish that I
> implemented my
> >  own protocolcodecfilter to decode byte stream into POJO's. But it looks
> like
> >  my ACKs never get to the client right away, most of the time they wait
> for
> >  the response of actual message, and will be delivered to client
> together.
>
> Sounds like you're protocol could have an issue with the tcp nagle
> algorithm?
> Try SocketSessionConfig.setTcpNoDelay(true) on the connector.
>
> http://mina.apache.org/faq.html
>
> --
> Fredrik Jonson
>
>

[elecharny-2]

Erinc Arikan wrote:
> When I set tcpNodelay to true of socketSessionConfig of the server's
> acceptor. I see the exact same timestamp for ACK and Response message in
> client logs. But when I set it to false I see 150-200 ms between ACK and
> message response although message processing on average takes around 800 ms
>  

You may have to add an executor at the end of your chain so that your
written ACK is sent even if the message is being processed, otherwise
everything will be processed sequentially (ie, when the handler will
have finished to process your messgae, then the thread will be freed,
and the IoPorcessor will be able to send the data, which mean that all
the written bytes are tored and sent at the same time  at the end of the
processing).

Here is what I used in Apache DS to fix the exact same problem :

            IoFilterChainBuilder chain = new DefaultIoFilterChainBuilder();
         
            // Inject the codec into the chain
            ((DefaultIoFilterChainBuilder)chain).addLast( "codec",
                    new ProtocolCodecFilter(
this.getProtocolCodecFactory() ) );
           
            // Now inject an ExecutorFilter for the write operations
            // We use the same number of thread than the number of
IoProcessor
            ((DefaultIoFilterChainBuilder)chain).addLast( "executor",
                    new ExecutorFilter(
                        new UnorderedThreadPoolExecutor(
transport.getNbThreads() ),
                        IoEventType.MESSAGE_RECEIVED ) );

            /*
             * The server is now initialized, we can
             * install the default requests handlers, which need
             * access to the DirectoryServer instance.
             */
            installDefaultHandlers();    

            startNetwork( transport, chain );
...


Just give it a try.

PS : *please*, when you post a mail on this ML, just consider providing
information about the MINA version you are using. The way MINA 1.0, MINA
1.1 and MINA 2.0 behave is really different. Thanks !

--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

[Erinc Arikan]

Thanks so much  Emmanuel;

I totally forgot to mention that I am using MINA 2.0 RC1.

I gave it a try and It worked great, I see 2-3 seconds between ack and
response now.

One thing that I wonder is optimum number of threads for an executor.

For now I set it to 10, But I don't think that I would need more than 2-3
threads at a time for a connection. Do you think that it would affect the
performance drastically when I have 10000 devices connected to my gateway.

Thanks

Erinc

On Tue, Nov 3, 2009 at 4:29 PM, Emmanuel Lecharny <elecharny@...>wrote:

> Erinc Arikan wrote:
>
>> When I set tcpNodelay to true of socketSessionConfig of the server's
>> acceptor. I see the exact same timestamp for ACK and Response message in
>> client logs. But when I set it to false I see 150-200 ms between ACK and
>> message response although message processing on average takes around 800
>> ms
>>
>>
>
> You may have to add an executor at the end of your chain so that your
> written ACK is sent even if the message is being processed, otherwise
> everything will be processed sequentially (ie, when the handler will have
> finished to process your messgae, then the thread will be freed, and the
> IoPorcessor will be able to send the data, which mean that all the written
> bytes are tored and sent at the same time  at the end of the processing).
>
> Here is what I used in Apache DS to fix the exact same problem :
>
>           IoFilterChainBuilder chain = new DefaultIoFilterChainBuilder();
>                   // Inject the codec into the chain
>           ((DefaultIoFilterChainBuilder)chain).addLast( "codec",
>                   new ProtocolCodecFilter( this.getProtocolCodecFactory() )
> );
>                     // Now inject an ExecutorFilter for the write
> operations
>           // We use the same number of thread than the number of
> IoProcessor
>           ((DefaultIoFilterChainBuilder)chain).addLast( "executor",
>                   new ExecutorFilter(
>                       new UnorderedThreadPoolExecutor(
> transport.getNbThreads() ),
>                       IoEventType.MESSAGE_RECEIVED ) );
>
>           /*
>            * The server is now initialized, we can
>            * install the default requests handlers, which need
>            * access to the DirectoryServer instance.
>            */
>           installDefaultHandlers();
>           startNetwork( transport, chain );
> ...
>
>
> Just give it a try.
>
> PS : *please*, when you post a mail on this ML, just consider providing
> information about the MINA version you are using. The way MINA 1.0, MINA 1.1
> and MINA 2.0 behave is really different. Thanks !
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

[elecharny-2]

Erinc Arikan wrote:
> Thanks so much  Emmanuel;
>
> I totally forgot to mention that I am using MINA 2.0 RC1.
>
> I gave it a try and It worked great, I see 2-3 seconds between ack and
> response now.
>  
Great !
> One thing that I wonder is optimum number of threads for an executor.
>
> For now I set it to 10, But I don't think that I would need more than 2-3
> threads at a time for a connection. Do you think that it would affect the
> performance drastically when I have 10000 devices connected to my gateway.
>  
I think it does not matter too much, as soon as if your processing is
fast, then the threads will be made available fast enough for other tasks.

Now, if it takes, say, 100 ms to process a request, and if you have 100
simultaneous tasks running, then you definitively need 10 threads
available in the pool (probably a few more in order to avoid
bottlenecks). So if you have 10000 devices doing one request every 5
seconds, and if it costs 100 ms to process a request, the the math is
simple : you will have an average of 200 requests being processed
simultaneously, so you'll need 200 threads. But a 100 ms processing time
is HUGE.

Note : when I say *simultaneous*, it means requests being processed *at
the same time*. (people are often confusing requests per second and
number of requests that can be processed simultaneous at a specific moment).

--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org