|
View:
New views
17 Messages
—
Rating Filter:
Alert me
|
|
|
Asynchronous transport for permanent connectionsHi,
I've made a proof of concept of an asynchronous transport for permanent socket connections as a final thesis for my bachelor degree. The main objective was to manage a large number of simultaneous connections using SEDA, especially with protocols that need to keep the connections open between requests. For this reason I've used non-blocking sockets (through Mina library): so while a request is processed there is no thread waiting for a response, greatly lowering the number of threads required. Looking for a SEDA implementation the choice fell on the excellent Mule ESB. The code is composed of some general classes for implementing transports with nio and two sample implementations: a sender/receiver tcp transport and a receiver-only http transport. This can be useful not only for long lasting/permanent connections, but also to send large quantities of data, streaming and push protocols. Since it is based on Mina it can also be useful to integrate Mina based protocols with Mule. I've arranged a trac repository with the code and a better description of how it works: http://minamule.madarco.net/trac I'd be glad to hear some feedback from the Mule developer community :) Marco D'Alia --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsMarco,
Let me kindly forward you to http://mule.mulesource.org/jira/browse/MULE-64 Some legal work - I see this disclaimer on the page:
This program is distrubuted as-is bla bla bla ecc.. and you can do whatever you want with it (Public Domain) Are you willing to give up the copyright in order for the code to eventually be incorporated into the Mule codebase? IMNAL, but keep in mind your reply may mean a legal agreement. Sorry for inconvenience, but some housekeeping is a must. And thanks a lot for being such a great community member :) Cheers! Andrew On 3/6/07,
Marco D'Alia <mule-dev@...> wrote: Hi, |
|
|
Re: Asynchronous transport for permanent connectionsMarco,
I can't find a direct SVN link to checkout everything, Trac never displays it in the UI. Could you please post one (read-only is ok)? Andrew |
|
|
Re: Asynchronous transport for permanent connectionsIf there's gonna be a follow up, I'll start. Minor things, but this one should be addressed first-hand, IMO:
MinaConnector uses URIs as keys for the ConcurrentHashMap. I try to avoid URIs as plague, it's just unfixable, period. Have a look at http://www.intertwingly.net/blog/2004/07/31/URI-Equivalence , especially the first comment about RFC 2616. Andrew
On 3/6/07, Andrew Perepelytsya <aperepel@...> wrote: Marco, |
|
|
Re: Asynchronous transport for permanent connectionsSure, I would be happy if my code will become part of Mule, I wrote that
copyright notice for this reason. I've found that issue in my preliminary work for the thesis, and I know that this transport can be a pain since some changes from the Tcp transport are deep, but I hope that someone could find a good use for it. I'm sorry, the svn repository is: http://minamule.madarco.net/svn Andrew Perepelytsya ha scritto: > Marco, > > Let me kindly forward you to > http://mule.mulesource.org/jira/browse/MULE-64 > > Some legal work - I see this disclaimer on the page: > > This program is distrubuted as-is bla bla bla ecc.. and you can do > whatever you want with it (Public Domain) > > Are you willing to give up the copyright in order for the code to > eventually be incorporated into the Mule codebase? IMNAL, but keep in > mind your reply may mean a legal agreement. > > Sorry for inconvenience, but some housekeeping is a must. And thanks a > lot for being such a great community member :) > > Cheers! > Andrew > > On 3/6/07, * Marco D'Alia* <mule-dev@... > <mailto:mule-dev@...>> wrote: > > Hi, > I've made a proof of concept of an asynchronous transport for > permanent > socket connections as a final thesis for my bachelor degree. > > The main objective was to manage a large number of simultaneous > connections using SEDA, especially with protocols that need to > keep the > connections open between requests. > For this reason I've used non-blocking sockets (through Mina library): > so while a request is processed there is no thread waiting for a > response, greatly lowering the number of threads required. > Looking for a SEDA implementation the choice fell on the excellent > Mule ESB. > > The code is composed of some general classes for implementing > transports > with nio and two sample implementations: a sender/receiver tcp > transport > and a receiver-only http transport. > > This can be useful not only for long lasting/permanent > connections, but > also to send large quantities of data, streaming and push protocols. > > Since it is based on Mina it can also be useful to integrate Mina > based > protocols with Mule. > > I've arranged a trac repository with the code and a better description > of how it works: http://minamule.madarco.net/trac > > I'd be glad to hear some feedback from the Mule developer community :) > > Marco D'Alia > > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsI totally agree,
In that piece of code there is also the problem that the endpoint uri in the reply-to shoud match exactly the one in the inbound endpoint. In addSession, getSession ecc the URI can be removed, and use instead a global hashmap, but it is required in hasServer(..) Andrew Perepelytsya ha scritto: > If there's gonna be a follow up, I'll start. Minor things, but this > one should be addressed first-hand, IMO: > > MinaConnector uses URIs as keys for the ConcurrentHashMap. I try to > avoid URIs as plague, it's just unfixable, period. Have a look at > http://www.intertwingly.net/blog/2004/07/31/URI-Equivalence , > especially the first comment about RFC 2616. > > Andrew > > On 3/6/07, *Andrew Perepelytsya* <aperepel@... > <mailto:aperepel@...>> wrote: > > Marco, > > I can't find a direct SVN link to checkout everything, Trac never > displays it in the UI. Could you please post one (read-only is ok)? > > Andrew > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
|
|
|
Re: Asynchronous transport for permanent connectionsGood work Marco!
We'd be interested in incorporating this module into Mule after a code review. The steps to becoming a committer can be found here: http://muledocs.org/Becoming+a+Committer Cheers, Ross On 6 Mar 2007, at 19:42, Marco D'Alia wrote: > Hi, > I've made a proof of concept of an asynchronous transport for > permanent socket connections as a final thesis for my bachelor degree. > > The main objective was to manage a large number of simultaneous > connections using SEDA, especially with protocols that need to > keep the > connections open between requests. > For this reason I've used non-blocking sockets (through Mina library): > so while a request is processed there is no thread waiting for a > response, greatly lowering the number of threads required. > Looking for a SEDA implementation the choice fell on the excellent > Mule ESB. > > The code is composed of some general classes for implementing > transports > with nio and two sample implementations: a sender/receiver tcp > transport > and a receiver-only http transport. > > This can be useful not only for long lasting/permanent connections, > but > also to send large quantities of data, streaming and push protocols. > > Since it is based on Mina it can also be useful to integrate Mina > based > protocols with Mule. > > I've arranged a trac repository with the code and a better description > of how it works: http://minamule.madarco.net/trac > > I'd be glad to hear some feedback from the Mule developer community :) > > Marco D'Alia > > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsHi Ben,
I'm sorry but that feature isn't working, in the first revision in the svn repository you can find an utility class that fix some problem in the Mina's StreamIoHandler and some tests transport for streaming. I've not continued that feature for my lack of understanding of the Mule stream support ( getStreamMessageAdapter and the meaning of the OutputStream parameter of routeMessage). Beware that the StreamIoHandler allocate a bytebuffer with ALL the data in the stream, I think that the only way to implement streaming is to use messageSent method in the ServerHandler to keep the bytebuffer full. Ben Hood ha scritto: > Hi Marco, > > Interesting work! > > >> This can be useful not only for long lasting/permanent >> connections, but also to send large quantities of data, >> streaming and push protocols. >> > > How have you found streaming to work with the MINA API? > > I was looking at the class HttpServerConnection from your transport to > find out how you are passing off a stream to a Mule component and if I'm > not wrong, this is using the HttpResponse class from the mule http > transport. > > Doesn't this materialize the stream before passing it on to the actual > processing component? > > Cheers, > > Ben > > > > ***************************************************** > > You can find us at www.voca.com > > ***************************************************** > This communication is confidential and intended for > the exclusive use of the addressee only. You should > not disclose its contents to any other person. > If you are not the intended recipient please notify > the sender named above immediately. > > Registered in England, No 1023742, > Registered Office: Voca Limited > Drake House, Three Rivers Court, > Homestead Road, Rickmansworth, > Hertfordshire, WD3 1FX > > > This message has been checked for all email viruses by MessageLabs. > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsThanks Ross :)
In the current version, the transport lack all configuration support, good exception handling, a complete "outbound" support and support for stream message payload. Ross Mason ha scritto: > Good work Marco! > > We'd be interested in incorporating this module into Mule after a code > review. The steps to becoming a committer can be found here: > http://muledocs.org/Becoming+a+Committer > > Cheers, > > Ross > > On 6 Mar 2007, at 19:42, Marco D'Alia wrote: > >> Hi, >> I've made a proof of concept of an asynchronous transport for >> permanent socket connections as a final thesis for my bachelor degree. >> >> The main objective was to manage a large number of simultaneous >> connections using SEDA, especially with protocols that need to keep the >> connections open between requests. >> For this reason I've used non-blocking sockets (through Mina library): >> so while a request is processed there is no thread waiting for a >> response, greatly lowering the number of threads required. >> Looking for a SEDA implementation the choice fell on the excellent >> Mule ESB. >> >> The code is composed of some general classes for implementing transports >> with nio and two sample implementations: a sender/receiver tcp transport >> and a receiver-only http transport. >> >> This can be useful not only for long lasting/permanent connections, but >> also to send large quantities of data, streaming and push protocols. >> >> Since it is based on Mina it can also be useful to integrate Mina based >> protocols with Mule. >> >> I've arranged a trac repository with the code and a better description >> of how it works: http://minamule.madarco.net/trac >> >> I'd be glad to hear some feedback from the Mule developer community :) >> >> Marco D'Alia >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list please visit: >> >> http://xircles.codehaus.org/manage_email >> > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
|
|
|
|
|
|
Re: Asynchronous transport for permanent connectionsBen,
I was thinking it was experimental because of many "todo" comments in the code. In some cases, such when there are large chunks of data to send, it can be convenient to send a stream reference payload, even if there are some limitations. In the dispatcher then, instead of send the entire stream, it is possible to allocate only a part of it and try to check when this buffer is empty and refill it. I think this can be done in the messageSent() method of the IoHandler. Marco Ben Hood ha scritto: > Marco, > > >> I'm sorry but that feature isn't working, in the first >> revision in the svn repository you can find an utility class >> that fix some problem in the Mina's StreamIoHandler and some >> tests transport for streaming. >> I've not continued that feature for my lack of understanding >> of the Mule stream support ( getStreamMessageAdapter and the >> meaning of the OutputStream parameter of routeMessage). >> > > The issue of streaming is being tackled in the 1.4 version of Mule, but > is still at an early stage, because I don't if it has been fully > conceptualized (please correct me if I'm wrong). > > In 1.4 you can declare a model type to be streaming, but then you can > only use streaming based components in that model because it passes in a > reference to the raw stream to the component. > > There are also some issues about passing around references to streams > between different components so this is why the approach has been > conservative. > > This is why in my transport I decided to handle streaming natively (i.e. > within the connector) until such a stage that the mule based methodology > stabilizes. I've tried to keep the code as abstract and decoupled as > possible in order to be able to migrate in the future. > > HTH, > > Ben > > > ***************************************************** > > You can find us at www.voca.com > > ***************************************************** > This communication is confidential and intended for > the exclusive use of the addressee only. You should > not disclose its contents to any other person. > If you are not the intended recipient please notify > the sender named above immediately. > > Registered in England, No 1023742, > Registered Office: Voca Limited > Drake House, Three Rivers Court, > Homestead Road, Rickmansworth, > Hertfordshire, WD3 1FX > > > This message has been checked for all email viruses by MessageLabs. > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsHi,
I've committed to my repository an example on streaming and some small bugfixes. I've used org.apache.mina.filter.StreamWriteFilter, after adding it as a IoFilter, writing an InputStream to a session will gradually send the data while it is readed from the remote host (will consume the stream in small chunks). hope it may help. Ben Hood ha scritto: > Marco, > > >> Beware that the StreamIoHandler allocate a bytebuffer with ALL >> the data in the stream, I think that the only way to implement >> streaming is to use messageSent method in the ServerHandler to >> keep the bytebuffer full. >> > > I've been thinking about that too. > > Basically I was wondering whether the InputStream as passed into the > StreamIoHandler is actually backed by a ByteBuffer, thus consuming the > whole stream. I got this suspicion when I saw the asInputStream() and > asOutputStream() methods in the ByteBuffer class. Checking this out is > on my list of TODOs. > > What did you mean about keeping the buffer full? > > Ben > > > ***************************************************** > > You can find us at www.voca.com > > ***************************************************** > This communication is confidential and intended for > the exclusive use of the addressee only. You should > not disclose its contents to any other person. > If you are not the intended recipient please notify > the sender named above immediately. > > Registered in England, No 1023742, > Registered Office: Voca Limited > Drake House, Three Rivers Court, > Homestead Road, Rickmansworth, > Hertfordshire, WD3 1FX > > > This message has been checked for all email viruses by MessageLabs. > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsHi Ross,
Thanks for the interest in my transport, I'll be glad if it will be included into Mule and I'd like to help you and the others to review the code. Since I don't understand some Mule internals and I know there are some "weak points" in the transport, I'd like if someone can take a look at the code, and help me refining it. I'll continue using this mailing list, or I have to open an issue in Jira? Thanks, Marco D'Alia Ross Mason ha scritto: > Good work Marco! > > We'd be interested in incorporating this module into Mule after a code > review. The steps to becoming a committer can be found here: > http://muledocs.org/Becoming+a+Committer > > Cheers, > > Ross > > On 6 Mar 2007, at 19:42, Marco D'Alia wrote: > >> Hi, >> I've made a proof of concept of an asynchronous transport for >> permanent socket connections as a final thesis for my bachelor degree. >> >> The main objective was to manage a large number of simultaneous >> connections using SEDA, especially with protocols that need to keep the >> connections open between requests. >> For this reason I've used non-blocking sockets (through Mina library): >> so while a request is processed there is no thread waiting for a >> response, greatly lowering the number of threads required. >> Looking for a SEDA implementation the choice fell on the excellent >> Mule ESB. >> >> The code is composed of some general classes for implementing transports >> with nio and two sample implementations: a sender/receiver tcp transport >> and a receiver-only http transport. >> >> This can be useful not only for long lasting/permanent connections, but >> also to send large quantities of data, streaming and push protocols. >> >> Since it is based on Mina it can also be useful to integrate Mina based >> protocols with Mule. >> >> I've arranged a trac repository with the code and a better description >> of how it works: http://minamule.madarco.net/trac >> >> I'd be glad to hear some feedback from the Mule developer community :) >> >> Marco D'Alia >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list please visit: >> >> http://xircles.codehaus.org/manage_email >> > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsHi Marco,
I would start a Jira issue if were you, things get lost really easily in the mailing list. I haven't had a chance yet to look at your new code, will do so shortly. I had a quick look at the ByteBuffer issue (without diving into the Mina code just yet) and it seems to have a smooth memory consumption for a very large throughput size (>500000 messages whilst limiting the heap to a max of 20M). I've still got some more testing to go before I'm convinced that this is really working in a streaming fashion, but my initial assumption is that it is chunking off the stream somehow and avoiding excessive memory consumption. You can see my test code at http://0x6e6562.zone.mod3.co.uk/hg/mule-transport-hessian/file/813e4ea3d879/src/test/java/org/mule/providers/hessian/HessianStreamingTest.java HTH, Ben On 3/7/07, Marco D'Alia <mule-dev@...> wrote: > Hi Ross, > Thanks for the interest in my transport, I'll be glad if it will be > included into Mule and I'd like to help you and the others to review the > code. > Since I don't understand some Mule internals and I know there are some > "weak points" in the transport, I'd like if someone can take a look at > the code, and help me refining it. > > I'll continue using this mailing list, or I have to open an issue in Jira? > > Thanks, > > Marco D'Alia > > Ross Mason ha scritto: > > Good work Marco! > > > > We'd be interested in incorporating this module into Mule after a code > > review. The steps to becoming a committer can be found here: > > http://muledocs.org/Becoming+a+Committer > > > > Cheers, > > > > Ross > > > > On 6 Mar 2007, at 19:42, Marco D'Alia wrote: > > > >> Hi, > >> I've made a proof of concept of an asynchronous transport for > >> permanent socket connections as a final thesis for my bachelor degree. > >> > >> The main objective was to manage a large number of simultaneous > >> connections using SEDA, especially with protocols that need to keep the > >> connections open between requests. > >> For this reason I've used non-blocking sockets (through Mina library): > >> so while a request is processed there is no thread waiting for a > >> response, greatly lowering the number of threads required. > >> Looking for a SEDA implementation the choice fell on the excellent > >> Mule ESB. > >> > >> The code is composed of some general classes for implementing transports > >> with nio and two sample implementations: a sender/receiver tcp transport > >> and a receiver-only http transport. > >> > >> This can be useful not only for long lasting/permanent connections, but > >> also to send large quantities of data, streaming and push protocols. > >> > >> Since it is based on Mina it can also be useful to integrate Mina based > >> protocols with Mule. > >> > >> I've arranged a trac repository with the code and a better description > >> of how it works: http://minamule.madarco.net/trac > >> > >> I'd be glad to hear some feedback from the Mule developer community :) > >> > >> Marco D'Alia > >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe from this list please visit: > >> > >> http://xircles.codehaus.org/manage_email > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe from this list please visit: > > > > http://xircles.codehaus.org/manage_email > > > > > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > > -- int main(){long u=0x6e6562;printf("%.4s",(char*)&u);} --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Asynchronous transport for permanent connectionsOn 7 Mar 2007, at 22:06, Marco D'Alia wrote: > Hi Ross, > Thanks for the interest in my transport, I'll be glad if it will be > included into Mule and I'd like to help you and the others to > review the code. > Since I don't understand some Mule internals and I know there are > some "weak points" in the transport, I'd like if someone can take a > look at the code, and help me refining it. No problem, we can help out there :) > > I'll continue using this mailing list, or I have to open an issue > in Jira? Andrew has already linked you to a Jira: http://mule.mulesource.org/ jira/browse/MULE-64. It would be best to do all further communication on this Jira To get Access to the mule-contrib SVN, would you mind going through the committer process (its painless :) ) here: http://muledocs.org/ Becoming+a+Committer Cheers, Ross > > Thanks, > > Marco D'Alia > > Ross Mason ha scritto: >> Good work Marco! >> >> We'd be interested in incorporating this module into Mule after a >> code review. The steps to becoming a committer can be found here: >> http://muledocs.org/Becoming+a+Committer >> >> Cheers, >> >> Ross >> >> On 6 Mar 2007, at 19:42, Marco D'Alia wrote: >> >>> Hi, >>> I've made a proof of concept of an asynchronous transport for >>> permanent socket connections as a final thesis for my bachelor >>> degree. >>> >>> The main objective was to manage a large number of simultaneous >>> connections using SEDA, especially with protocols that need to >>> keep the >>> connections open between requests. >>> For this reason I've used non-blocking sockets (through Mina >>> library): >>> so while a request is processed there is no thread waiting for a >>> response, greatly lowering the number of threads required. >>> Looking for a SEDA implementation the choice fell on the >>> excellent Mule ESB. >>> >>> The code is composed of some general classes for implementing >>> transports >>> with nio and two sample implementations: a sender/receiver tcp >>> transport >>> and a receiver-only http transport. >>> >>> This can be useful not only for long lasting/permanent >>> connections, but >>> also to send large quantities of data, streaming and push protocols. >>> >>> Since it is based on Mina it can also be useful to integrate Mina >>> based >>> protocols with Mule. >>> >>> I've arranged a trac repository with the code and a better >>> description >>> of how it works: http://minamule.madarco.net/trac >>> >>> I'd be glad to hear some feedback from the Mule developer >>> community :) >>> >>> Marco D'Alia >>> >>> >>> >>> -------------------------------------------------------------------- >>> - >>> To unsubscribe from this list please visit: >>> >>> http://xircles.codehaus.org/manage_email >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email |
| Free embeddable forum powered by Nabble | Forum Help |