Processing order of filters

[pinus]

Well, I use version 2.0M6 and I have trouble with the order of filters.
I created a protocol encoder/decoder and some filters. The filters
extend IoFilterAdapter and overwrite messageReceived/Sent.
This works well for for inbound messages, the protocol is used first
and than the filters are used on their order.
The trouble starts outbound. I would expect that the filter chain is
processed in the reverse order but id did not! The filters are called
in the same order as the for the inbound messages. Is this the designed
behaviour? Any ideas?

[Ashish-24]

On Tue, Oct 27, 2009 at 3:14 PM,  <pinus@...> wrote:
> Well, I use version 2.0M6 and I have trouble with the order of filters.
> I created a protocol encoder/decoder and some filters. The filters
> extend IoFilterAdapter and overwrite messageReceived/Sent.
> This works well for for inbound messages, the protocol is used first
> and than the filters are used on their order.
> The trouble starts outbound. I would expect that the filter chain is
> processed in the reverse order but id did not! The filters are called
> in the same order as the for the inbound messages. Is this the designed
> behaviour? Any ideas?


This is how it works today :-(

This is going to change in 3.0, where we plan to have different chains
for incoming and outgoing messages.
It shall be a while before you get to use 3.0

thanks
ashish

[pinus]

Zitat von Ashish <paliwalashish@...>:

> On Tue, Oct 27, 2009 at 3:14 PM,  <pinus@...> wrote:
>> Well, I use version 2.0M6 and I have trouble with the order of filters.
>> I created a protocol encoder/decoder and some filters. The filters
>> extend IoFilterAdapter and overwrite messageReceived/Sent.
>> This works well for for inbound messages, the protocol is used first
>> and than the filters are used on their order.
>> The trouble starts outbound. I would expect that the filter chain is
>> processed in the reverse order but id did not! The filters are called
>> in the same order as the for the inbound messages. Is this the designed
>> behaviour? Any ideas?
>
>
> This is how it works today :-(
>
> This is going to change in 3.0, where we plan to have different chains
> for incoming and outgoing messages.
> It shall be a while before you get to use 3.0

I do not understand why you need two chains? The filters are bidirectional,
messageReceived and messageSent. The only issue is the order, because
it does not work as described in the wiki. The session.write method
should only call the filters in the reverse order.

I want to use a protocol, filter 1-5 and a handler. The handler should
be able to send back a message. The message back should be processed by
filter 5,4,3,2,1 and the protocol. That's how I interpret the architecture
diagram on the wiki and how it makes sense for me. For me the current
behaviour is a bug, it simply doesn't make sense. I dare the claim that
nobody implemented sent and receive in one filter.
I created an issue (DIRMINA-744) just before I read your answer.

The only solution I see is that each filter has to find the next filter by
itself. It could use the session to get the filter chain, find myself and
call messageSent of the predecessor. No comment about quality of code.

Since I think the changes in MINA are minimal to reverse the order
for sending. How about an option/configuration that changes the order?
The filter chain already has a method getAllReversed(). This would allow
to match the architecture diagram on the wiki on one hand and allow
compatibility where it is needed.

[elecharny-2]

pinus@... wrote:

> Zitat von Ashish <paliwalashish@...>:
>
>> On Tue, Oct 27, 2009 at 3:14 PM,  <pinus@...> wrote:
>>> Well, I use version 2.0M6 and I have trouble with the order of filters.
>>> I created a protocol encoder/decoder and some filters. The filters
>>> extend IoFilterAdapter and overwrite messageReceived/Sent.
>>> This works well for for inbound messages, the protocol is used first
>>> and than the filters are used on their order.
>>> The trouble starts outbound. I would expect that the filter chain is
>>> processed in the reverse order but id did not! The filters are called
>>> in the same order as the for the inbound messages. Is this the designed
>>> behaviour? Any ideas?
>>
>>
>> This is how it works today :-(
>>
>> This is going to change in 3.0, where we plan to have different chains
>> for incoming and outgoing messages.
>> It shall be a while before you get to use 3.0
>
> I do not understand why you need two chains?

Just because you don't need to do the same work in both ways. Typically,
you may want to split the encoder and the decoder in two different
filters. You may also avoid havng to pass through an executor when
writing data back to the client.

In any case, having two chains is most certainly simpler to deal with
(that does not mean we expose those two chains to the user if not needed).

> The filters are bidirectional,
> messageReceived and messageSent. The only issue is the order, because
> it does not work as described in the wiki. The session.write method
> should only call the filters in the reverse order.
No. The wiki is representing a flow which is not correct, but that does
not mean the code is broken. We have tens of applications based on MINA
which works perfectly well, I don't see any issue considering that you
won't have tens of filters in the chain anyway (the most you can pile up
is probably four : a codec, a SslFilter, an executor and a logging
filter). Generally speaking, two of them are probably enough : SSL and
codec.
>
> I want to use a protocol, filter 1-5 and a handler. The handler should
> be able to send back a message. The message back should be processed by
> filter 5,4,3,2,1 and the protocol. That's how I interpret the
> architecture
> diagram on the wiki and how it makes sense for me.
But the diagram is just to express that messages are going through the
filters and back to the client. Probably need an extra explaination to
eliminate possible confusion about the order.
> For me the current
> behaviour is a bug, it simply doesn't make sense.
I would rather claim that it's confusing. It's certainly not a bug.

> I dare the claim that
> nobody implemented sent and receive in one filter.
Wrong ! If you don't need SSL, then a codec is enough. Apache Directory
Server implements NTP protocol with one single filter.
> I created an issue (DIRMINA-744) just before I read your answer.
>
> The only solution I see is that each filter has to find the next
> filter by
> itself.
If you read the code, this is *exactly* what the code does. But the way
it find the next filter is just by reading the chain :)

We are thinking about another approach, as Ashish axplained, which is
certainly easier to understand. Anyway, both approach will work...

Keep in mind that you may not implement an action (ie, messageSent) in a
filter if not needed. The filter will then be bypassed.

> It could use the session to get the filter chain, find myself and
> call messageSent of the predecessor. No comment about quality of code.
>
> Since I think the changes in MINA are minimal to reverse the order
> for sending. How about an option/configuration that changes the order?
> The filter chain already has a method getAllReversed(). This would allow
> to match the architecture diagram on the wiki on one hand and allow
> compatibility where it is needed.
Wed'd rather think about defining the two chains and let the user
configure it. In any case, that will end with the same result.

What's is puzzling is that the current implementation, no matter how
weird it sounds to you, and no matter how irrelevant is the doco (!) is
doing its job fine :)

In any case, can you just tell us about your implementation problem, so
that we can give you a hand ?

Thanks !

--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

[pinus]

Emmanuel Lecharny schrieb:

> pinus@... wrote:
>> Zitat von Ashish <paliwalashish@...>:
>>
>>> On Tue, Oct 27, 2009 at 3:14 PM,  <pinus@...> wrote:
>>>> Well, I use version 2.0M6 and I have trouble with the order of
>>>> filters.
>>>> I created a protocol encoder/decoder and some filters. The filters
>>>> extend IoFilterAdapter and overwrite messageReceived/Sent.
>>>> This works well for for inbound messages, the protocol is used first
>>>> and than the filters are used on their order.
>>>> The trouble starts outbound. I would expect that the filter chain is
>>>> processed in the reverse order but id did not! The filters are called
>>>> in the same order as the for the inbound messages. Is this the
>>>> designed
>>>> behaviour? Any ideas?
>>>
>>>
>>> This is how it works today :-(
>>>
>>> This is going to change in 3.0, where we plan to have different chains
>>> for incoming and outgoing messages.
>>> It shall be a while before you get to use 3.0
>>
>> I do not understand why you need two chains?
> Just because you don't need to do the same work in both ways.
> Typically, you may want to split the encoder and the decoder in two
> different filters. You may also avoid havng to pass through an
> executor when writing data back to the client.
>
> In any case, having two chains is most certainly simpler to deal with
> (that does not mean we expose those two chains to the user if not
> needed).
>
>> The filters are bidirectional,
>> messageReceived and messageSent. The only issue is the order, because
>> it does not work as described in the wiki. The session.write method
>> should only call the filters in the reverse order.
> No. The wiki is representing a flow which is not correct, but that
> does not mean the code is broken. We have tens of applications based
> on MINA which works perfectly well, I don't see any issue considering
> that you won't have tens of filters in the chain anyway (the most you
> can pile up is probably four : a codec, a SslFilter, an executor and a
> logging filter). Generally speaking, two of them are probably enough :
> SSL and codec.

I wonder what happens if you chain the SSL and the logging filter. How
do you configure your system to log the data not crypted. Both
directions, inbound and outbound? If SSL is the first filter and logging
is the second, inbound you log the data not crypted. Outbound you log
crypted data. Is that right?

>> I want to use a protocol, filter 1-5 and a handler. The handler should
>> be able to send back a message. The message back should be processed by
>> filter 5,4,3,2,1 and the protocol. That's how I interpret the
>> architecture
>> diagram on the wiki and how it makes sense for me.
> But the diagram is just to express that messages are going through the
> filters and back to the client. Probably need an extra explaination to
> eliminate possible confusion about the order.
Yes this very confusing and misleading. And what makes it most confusing
is that you have a messageSent and messageReceived method. From what I
understand the direction is not important. If the IoService wants to
push a message to the handler it uses the IoFilterChain filter 1,2,3 and
the output is passed to the handler. In the other direction the
IoHandler also uses the IoFilterChain filter 1,2,3 and the output is
passed to the IoService.

And that you mention the servlet filter is even more confusing because
it works different. It also doesn't match the filter definition of Unix
or DOS command line filters. In these cases filters are something different.

What you do is executing a list of functions on the data. You can argue
that this is filtering and that splitting it to send and receive avoids
a if inbound then else. For me your filtering is using a function on the
data in different contexts, inbound and outbound.

>> For me the current
>> behaviour is a bug, it simply doesn't make sense.
> I would rather claim that it's confusing. It's certainly not a bug.
>
>> I dare the claim that
>> nobody implemented sent and receive in one filter.
> Wrong ! If you don't need SSL, then a codec is enough. Apache
> Directory Server implements NTP protocol with one single filter.
>> I created an issue (DIRMINA-744) just before I read your answer.
>>
>> The only solution I see is that each filter has to find the next
>> filter by
>> itself.
> If you read the code, this is *exactly* what the code does. But the
> way it find the next filter is just by reading the chain :)
>
> We are thinking about another approach, as Ashish axplained, which is
> certainly easier to understand. Anyway, both approach will work...
>
> Keep in mind that you may not implement an action (ie, messageSent) in
> a filter if not needed. The filter will then be bypassed.
>
>> It could use the session to get the filter chain, find myself and
>> call messageSent of the predecessor. No comment about quality of code.
>>
>> Since I think the changes in MINA are minimal to reverse the order
>> for sending. How about an option/configuration that changes the order?
>> The filter chain already has a method getAllReversed(). This would allow
>> to match the architecture diagram on the wiki on one hand and allow
>> compatibility where it is needed.
> Wed'd rather think about defining the two chains and let the user
> configure it. In any case, that will end with the same result.
>
> What's is puzzling is that the current implementation, no matter how
> weird it sounds to you, and no matter how irrelevant is the doco (!)
> is doing its job fine :)
>
> In any case, can you just tell us about your implementation problem,
> so that we can give you a hand ?

I write a middle-ware application that communicates with different
partners and has to adapt the different protocols. The protocols are
usually binary protocols. The communication uses client only, server
only or server and client sockets for communication. The communication
direction is bidirectional, both sides can cause messages. One protocol
uses a client and a server socket, one direction of the socket is the
main direction the other is used for acknowledgements only. We currently
have a back-end that is able to parse the message and normalise it for
further processing. The back-end requires a unique message identifier
for each message. I thought that I can use a filter that adds an
identifier to the message inbound and strips the identifier outbound.
The same is true for other information, checksums, message type, sender
and receiver information and so on. That's the place where I thought the
filters are useful, having piped Unix commands in mind. The idea is to
adapt to new protocols simply by configuring new filters, e.g. adding a
filter that converts a XML message into the normalised message format
and vice versa. We could change the normalised format to e.g. XML simply
by adding a filter. I think I have to create a handler that implements
my own filter chain.

I messed around with the filter chain to fool MINA by adding the "right"
next filter at the end of each messageSent method. But that doesn't seem
to work easily because yo can not access the previous filter because it
is not part of the interface. What really bugs me is that it is part of
the implementation (EntryImpl).
Another thing I tried is the implementation of
filterChain.getAllReverse() gives you the reverse ordered filters. But
if you ask the retrieved list with getNextFilter() on  you get the old
order. I would assume that you get the reverse order too, but you will
probably answer that it works as designed. I can't argue about this.
All of this is quite frustrating, I think I have to create my own filter
chain as handler. That's the only solution I see that matches my time frame.

[elecharny-2]

Haug Bürger wrote:
>>    
> I wonder what happens if you chain the SSL and the logging filter.
Surprisingly, it works ! At least, if the SSL filter is the first in the
chain :)

Ok, here, we have a potential situation here : this SSL filter *must* be
the first in the chain.

As this is nothing but logical, we think that the SSL filter must be
removed from the chain and be put before the chain is called (this is
discussed here : http://mina.apache.org/mina-30-design.html)
> How
> do you configure your system to log the data not crypted. Both
> directions, inbound and outbound? If SSL is the first filter and logging
> is the second, inbound you log the data not crypted. Outbound you log
> crypted data. Is that right?
>  
Well, the SSL filter is not responsible for encryption or description.
It's just used to handle the handshake between the client and the
server, as the SSL layer is handled by the socket itself.

>  
>>> I want to use a protocol, filter 1-5 and a handler. The handler should
>>> be able to send back a message. The message back should be processed by
>>> filter 5,4,3,2,1 and the protocol. That's how I interpret the
>>> architecture
>>> diagram on the wiki and how it makes sense for me.
>>>      
>> But the diagram is just to express that messages are going through the
>> filters and back to the client. Probably need an extra explaination to
>> eliminate possible confusion about the order.
>>    
> Yes this very confusing and misleading. And what makes it most confusing
> is that you have a messageSent and messageReceived method. From what I
> understand the direction is not important.

true.
> If the IoService wants to
> push a message to the handler it uses the IoFilterChain filter 1,2,3 and
> the output is passed to the handler. In the other direction the
> IoHandler also uses the IoFilterChain filter 1,2,3 and the output is
> passed to the IoService.
>  
right.
> And that you mention the servlet filter is even more confusing because
> it works different.
right.
> It also doesn't match the filter definition of Unix
> or DOS command line filters. In these cases filters are something different.
>  
Most certainly. However, 'Filter' can have more than one semantic, and
it's really difficult to define it by comparing the other semantics in
use. How does it compare to a filter used in a car for instance?

So we stick to the basic definition for filter : it takes some input, do
some transformation, and produce an output. Whatever it is inbound or
outbound.
> What you do is executing a list of functions on the data. You can argue
> that this is filtering and that splitting it to send and receive avoids
> a if inbound then else. For me your filtering is using a function on the
> data in different contexts, inbound and outbound.
>  
I don't think it does matter a lot what you or me are thinking about
what a filter is doing :) What is important is that the doco does not
spread confusion ...

>>> It could use the session to get the filter chain, find myself and
>>> call messageSent of the predecessor. No comment about quality of code.
>>>
>>> Since I think the changes in MINA are minimal to reverse the order
>>> for sending. How about an option/configuration that changes the order?
>>> The filter chain already has a method getAllReversed(). This would allow
>>> to match the architecture diagram on the wiki on one hand and allow
>>> compatibility where it is needed.
>>>      
>> Wed'd rather think about defining the two chains and let the user
>> configure it. In any case, that will end with the same result.
>>
>> What's is puzzling is that the current implementation, no matter how
>> weird it sounds to you, and no matter how irrelevant is the doco (!)
>> is doing its job fine :)
>>
>> In any case, can you just tell us about your implementation problem,
>> so that we can give you a hand ?
>>    
> I write a middle-ware application that communicates with different
> partners and has to adapt the different protocols. The protocols are
> usually binary protocols. The communication uses client only, server
> only or server and client sockets for communication. The communication
> direction is bidirectional, both sides can cause messages. One protocol
> uses a client and a server socket, one direction of the socket is the
> main direction the other is used for acknowledgements only. We currently
> have a back-end that is able to parse the message and normalise it for
> further processing. The back-end requires a unique message identifier
> for each message. I thought that I can use a filter that adds an
> identifier to the message inbound and strips the identifier outbound.
> The same is true for other information, checksums, message type, sender
> and receiver information and so on. That's the place where I thought the
> filters are useful, having piped Unix commands in mind. The idea is to
> adapt to new protocols simply by configuring new filters, e.g. adding a
> filter that converts a XML message into the normalised message format
> and vice versa. We could change the normalised format to e.g. XML simply
> by adding a filter. I think I have to create a handler that implements
> my own filter chain.
>  

Sadly, yes. The main issue we have (and you can read about it in the
page I mentioned above) is that we are transfering IoBuffer from filter
to filter. This is a major design flaw we want to fix in 3.0, as it
forbids you to chain codecs. This makes it a bit difficult to use
filters, and most certainly you hit this wall here.

We have had the same issue in Apache Directory project, and we have had
to implement the codec in a certain way in order to make it work, as it
was a two stage decoder. Not really easy...
> I messed around with the filter chain to fool MINA by adding the "right"
> next filter at the end of each messageSent method. But that doesn't seem
> to work easily because yo can not access the previous filter because it
> is not part of the interface. What really bugs me is that it is part of
> the implementation (EntryImpl).
>  
You can still add a filter somewhere in the chain dynamically as soon as
you know the name of each filter.

I must admit that the current impl is also a perfect maze... Debugging
MINA is just requiring a very high IQ :/
> Another thing I tried is the implementation of
> filterChain.getAllReverse() gives you the reverse ordered filters. But
> if you ask the retrieved list with getNextFilter() on  you get the old
> order. I would assume that you get the reverse order too, but you will
> probably answer that it works as designed. I can't argue about this.
> All of this is quite frustrating, I think I have to create my own filter
> chain as handler. That's the only solution I see that matches my time frame.
>  
Probably... Sorry for that, the current code is not perfect, and this is
the reason we are really impatient to start the 3.0 version.

--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

[Julien Vermillard]

Le Tue, 27 Oct 2009 10:44:30 +0100,
pinus@... a écrit :

> Well, I use version 2.0M6 and I have trouble with the order of
> filters. I created a protocol encoder/decoder and some filters. The
> filters extend IoFilterAdapter and overwrite messageReceived/Sent.
> This works well for for inbound messages, the protocol is used first
> and than the filters are used on their order.
> The trouble starts outbound. I would expect that the filter chain is
> processed in the reverse order but id did not! The filters are called
> in the same order as the for the inbound messages. Is this the
> designed behaviour? Any ideas?
>
>
>

The messageSent is a PITA, the filterWrite is in reverse order.

MessageSent is a "utility" designed to inform you you PDU  was
completely sent.

IMO it should be removed for 3.0 because less I use it, easier MINA
usage is.

Julien

[elecharny-2]

Julien Vermillard wrote:

> Le Tue, 27 Oct 2009 10:44:30 +0100,
> pinus@... a écrit :
>
>  
>> Well, I use version 2.0M6 and I have trouble with the order of
>> filters. I created a protocol encoder/decoder and some filters. The
>> filters extend IoFilterAdapter and overwrite messageReceived/Sent.
>> This works well for for inbound messages, the protocol is used first
>> and than the filters are used on their order.
>> The trouble starts outbound. I would expect that the filter chain is
>> processed in the reverse order but id did not! The filters are called
>> in the same order as the for the inbound messages. Is this the
>> designed behaviour? Any ideas?
>>
>>
>>
>>    
> The messageSent is a PITA, the filterWrite is in reverse order.
>  

Right, I forgot to mention that the write is done in reverse order. this
is where it's even more confusing...

--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

[pinus]

Zitat von Emmanuel Lecharny <elecharny@...>:

>> I write a middle-ware application that communicates with different
>> partners and has to adapt the different protocols. The protocols are
>> usually binary protocols. The communication uses client only, server
>> only or server and client sockets for communication. The communication
>> direction is bidirectional, both sides can cause messages. One protocol
>> uses a client and a server socket, one direction of the socket is the
>> main direction the other is used for acknowledgements only. We currently
>> have a back-end that is able to parse the message and normalise it for
>> further processing. The back-end requires a unique message identifier
>> for each message. I thought that I can use a filter that adds an
>> identifier to the message inbound and strips the identifier outbound.
>> The same is true for other information, checksums, message type, sender
>> and receiver information and so on. That's the place where I thought the
>> filters are useful, having piped Unix commands in mind. The idea is to
>> adapt to new protocols simply by configuring new filters, e.g. adding a
>> filter that converts a XML message into the normalised message format
>> and vice versa. We could change the normalised format to e.g. XML simply
>> by adding a filter. I think I have to create a handler that implements
>> my own filter chain.
>>
> Sadly, yes. The main issue we have (and you can read about it in the
> page I mentioned above) is that we are transfering IoBuffer from filter
> to filter. This is a major design flaw we want to fix in 3.0, as it
> forbids you to chain codecs. This makes it a bit difficult to use
> filters, and most certainly you hit this wall here.
>
> We have had the same issue in Apache Directory project, and we have had
> to implement the codec in a certain way in order to make it work, as it
> was a two stage decoder. Not really easy...

Could you please explain?

> I must admit that the current impl is also a perfect maze... Debugging
> MINA is just requiring a very high IQ :/

Doesn't that make you think?

>> Another thing I tried is the implementation of
>> filterChain.getAllReverse() gives you the reverse ordered filters. But
>> if you ask the retrieved list with getNextFilter() on  you get the old
>> order. I would assume that you get the reverse order too, but you will
>> probably answer that it works as designed. I can't argue about this.
>> All of this is quite frustrating, I think I have to create my own filter
>> chain as handler. That's the only solution I see that matches my time frame.
>>
> Probably... Sorry for that, the current code is not perfect, and this
> is the reason we are really impatient to start the 3.0 version.

And again. I can not use a handler to implement my own filter chain.
The handler (messageSent) is to late on the outbound side. What to filter
if it's already sent?

If I want to create my own filter chain I need to implement it as a filter,
not a handler. I already implemented a ProtocolEncoder, ProtocolDecoder using
a ProtocolCodecFilter. How can I reuse this component?

[elecharny-2]

>>
>> We have had the same issue in Apache Directory project, and we have had
>> to implement the codec in a certain way in order to make it work, as it
>> was a two stage decoder. Not really easy...
>
> Could you please explain?
We have a two stages decoder. It's all done in one single codec, which
is stateful. We store the current state in the IoSession, and when we
receive more bytes, we resume the decoding exactly where it stopped.
>
>> I must admit that the current impl is also a perfect maze... Debugging
>> MINA is just requiring a very high IQ :/
>
> Doesn't that make you think?
Eh... Why do you think we started a 3.0 ???

>
>>> Another thing I tried is the implementation of
>>> filterChain.getAllReverse() gives you the reverse ordered filters. But
>>> if you ask the retrieved list with getNextFilter() on  you get the old
>>> order. I would assume that you get the reverse order too, but you will
>>> probably answer that it works as designed. I can't argue about this.
>>> All of this is quite frustrating, I think I have to create my own
>>> filter
>>> chain as handler. That's the only solution I see that matches my
>>> time frame.
>>>
>> Probably... Sorry for that, the current code is not perfect, and this
>> is the reason we are really impatient to start the 3.0 version.
>
> And again. I can not use a handler to implement my own filter chain.
> The handler (messageSent) is to late on the outbound side. What to filter
> if it's already sent?

The MessageSent event is just produced when the message was actually
*sent*. You can do a lot of thing with this information, like gathering
stats, etc. Now, it may not have to be processed by a codec. In this
case, you don't have to implement the method in your filters.

Overall, I suggest you have a look at the many examples we provide, to
see how MINA works in detail. There is probably something you have
missed, due to the documentation which is not explicit enough. I
understand that's a bit frustrating, but there is little I can do here
to help you with your frustration except telling you that if you feel
you can help building a better framework, then please, join us !

--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org