HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null

[Moise, Daniel]

Hi,

I've found the same problem as the one listed below - every time a
malformed request is given to Synapse, an infinite loop will be
triggered making the Synapse server sluggish, and consuming the entire
log file (in case the output is redirected). I've noticed that the
problem might get fixed. Could you please let me know if the problem was
fixes, and if yes in which release.

Thanks,
Daniel


                Hi All

                I guess this boils down to how we deal with broken,
partial, and even
                possibly malicious requests in the NIO transport.. I
will fix the issue
                pointed to by Andreas and Murali ASAP as I consider this
critical.

                Murali - I believe you should plan on updating to
Synapse 1.3 in the
                near future. There are a few issues we have fixed over
the past year
                that should be very important for any production
deployment. As I
                believe your environment / configuration is
straightforward, so the
                migration should be quite trivial too - Could you also
raise an issue on
                the JIRA?

                This runtime exception is caught in
BaseIOReactor#writable and delegated to the
                registered exception handler (in our case an anonymous
inner class of
                HttpCoreNIOListener with the following implementation:

                ioReactor.setExceptionHandler(new
IOReactorExceptionHandler() {
                                public boolean handle(IOException
ioException) {
                                    log.warn("System may be unstable:
IOReactor encountered a
                checked exception : "
                                            + ioException.getMessage(),
ioException);
                                    return true;
                                }

                                public boolean handle(RuntimeException
runtimeException) {
                                    log.warn("System may be unstable:
IOReactor encountered a
                runtime exception : "
                                            +
runtimeException.getMessage(), runtimeException);
                                    return true;
                                }
                            });

                I wondering a bit about the return value. Taken from the
JavaDoc of the
                interface IOReactorExceptionHandler#handle

                True if it is safe to ignore the exception and continue
execution of the I/O
                reactor; if the I/O reactor must throw {@link
RuntimeException} and terminate

                Hmm, is it really safe to proceed here? We do not test
any detail of the
                RuntimeException. To me this looks wrong.

                Yes, we return true since the later versions of HttpCore
dump some
                information and try to continue in the face of RT
exceptions. What I
                feel is that we need to enhance the NIO transport to be
stable in the
                face of broken, partial and malicious requests so that
RT exceptions
                will not be thrown in such cases, and we will drop the
connection with a
                WARN

                cheers
                asankha

                --
                Asankha C. Perera
                AdroitLogic, http://adroitlogic.org
               
                http://esbmagic.blogspot.com

       
---------------------------------------------------------------------
                To unsubscribe, e-mail: dev-...@...
                For additional commands, e-mail:
dev-...@...
               

[Ruwan Linton]

Hi Daniel,

This should be fixed now on the trunk, were you trying this on the trunk?
Can you please verify this on a SNAPSHOT build?

This will be available with the 1.3 release.

Thanks,
Ruwan

On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel <Daniel_Moise@...>wrote:

> Hi,
>
> I've found the same problem as the one listed below - every time a
> malformed request is given to Synapse, an infinite loop will be
> triggered making the Synapse server sluggish, and consuming the entire
> log file (in case the output is redirected). I've noticed that the
> problem might get fixed. Could you please let me know if the problem was
> fixes, and if yes in which release.
>
> Thanks,
> Daniel
>
>
>                Hi All
>
>                I guess this boils down to how we deal with broken,
> partial, and even
>                possibly malicious requests in the NIO transport.. I
> will fix the issue
>                pointed to by Andreas and Murali ASAP as I consider this
> critical.
>
>                Murali - I believe you should plan on updating to
> Synapse 1.3 in the
>                near future. There are a few issues we have fixed over
> the past year
>                that should be very important for any production
> deployment. As I
>                believe your environment / configuration is
> straightforward, so the
>                migration should be quite trivial too - Could you also
> raise an issue on
>                the JIRA?
>
>                This runtime exception is caught in
> BaseIOReactor#writable and delegated to the
>                registered exception handler (in our case an anonymous
> inner class of
>                HttpCoreNIOListener with the following implementation:
>
>                ioReactor.setExceptionHandler(new
> IOReactorExceptionHandler() {
>                                public boolean handle(IOException
> ioException) {
>                                    log.warn("System may be unstable:
> IOReactor encountered a
>                checked exception : "
>                                            + ioException.getMessage(),
> ioException);
>                                    return true;
>                                }
>
>                                public boolean handle(RuntimeException
> runtimeException) {
>                                    log.warn("System may be unstable:
> IOReactor encountered a
>                runtime exception : "
>                                            +
> runtimeException.getMessage(), runtimeException);
>                                    return true;
>                                }
>                            });
>
>                I wondering a bit about the return value. Taken from the
> JavaDoc of the
>                interface IOReactorExceptionHandler#handle
>
>                True if it is safe to ignore the exception and continue
> execution of the I/O
>                reactor; if the I/O reactor must throw {@link
> RuntimeException} and terminate
>
>                Hmm, is it really safe to proceed here? We do not test
> any detail of the
>                RuntimeException. To me this looks wrong.
>
>                Yes, we return true since the later versions of HttpCore
> dump some
>                information and try to continue in the face of RT
> exceptions. What I
>                feel is that we need to enhance the NIO transport to be
> stable in the
>                face of broken, partial and malicious requests so that
> RT exceptions
>                will not be thrown in such cases, and we will drop the
> connection with a
>                WARN
>
>                cheers
>                asankha
>
>                --
>                Asankha C. Perera
>                AdroitLogic, http://adroitlogic.org
>
>                http://esbmagic.blogspot.com
>
>
> ---------------------------------------------------------------------
>                To unsubscribe, e-mail: dev-...@...
>                For additional commands, e-mail:
> dev-...@...
>
>

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Moise, Daniel]

Hi Ruwan,

I've tried the last changes in the 1.3 branch, and the problem is still
there. To test it, we've just run a 'telnet localhost 8280'.
Based on your answer, it seems that you guys have a solution for this.
Could you send me a patch or the changes for that. You don't have to
necessarily check them in the branch for now.

Thanks,
Daniel

-----Original Message-----
From: Ruwan Linton [mailto:ruwan.linton@...]
Sent: Wednesday, September 23, 2009 6:04 PM
To: user@...
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null

Hi Daniel,

This should be fixed now on the trunk, were you trying this on the
trunk?
Can you please verify this on a SNAPSHOT build?

This will be available with the 1.3 release.

Thanks,
Ruwan

On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
<Daniel_Moise@...>wrote:

> Hi,
>
> I've found the same problem as the one listed below - every time a
> malformed request is given to Synapse, an infinite loop will be
> triggered making the Synapse server sluggish, and consuming the entire
> log file (in case the output is redirected). I've noticed that the
> problem might get fixed. Could you please let me know if the problem
was

> fixes, and if yes in which release.
>
> Thanks,
> Daniel
>
>
>                Hi All
>
>                I guess this boils down to how we deal with broken,
> partial, and even
>                possibly malicious requests in the NIO transport.. I
> will fix the issue
>                pointed to by Andreas and Murali ASAP as I consider

this

> critical.
>
>                Murali - I believe you should plan on updating to
> Synapse 1.3 in the
>                near future. There are a few issues we have fixed over
> the past year
>                that should be very important for any production
> deployment. As I
>                believe your environment / configuration is
> straightforward, so the
>                migration should be quite trivial too - Could you also
> raise an issue on
>                the JIRA?
>
>                This runtime exception is caught in
> BaseIOReactor#writable and delegated to the
>                registered exception handler (in our case an anonymous
> inner class of
>                HttpCoreNIOListener with the following implementation:
>
>                ioReactor.setExceptionHandler(new
> IOReactorExceptionHandler() {
>                                public boolean handle(IOException
> ioException) {
>                                    log.warn("System may be unstable:
> IOReactor encountered a
>                checked exception : "
>                                            + ioException.getMessage(),
> ioException);
>                                    return true;
>                                }
>
>                                public boolean handle(RuntimeException
> runtimeException) {
>                                    log.warn("System may be unstable:
> IOReactor encountered a
>                runtime exception : "
>                                            +
> runtimeException.getMessage(), runtimeException);
>                                    return true;
>                                }
>                            });
>
>                I wondering a bit about the return value. Taken from

the

> JavaDoc of the
>                interface IOReactorExceptionHandler#handle
>
>                True if it is safe to ignore the exception and continue
> execution of the I/O
>                reactor; if the I/O reactor must throw {@link
> RuntimeException} and terminate
>
>                Hmm, is it really safe to proceed here? We do not test
> any detail of the
>                RuntimeException. To me this looks wrong.
>
>                Yes, we return true since the later versions of

HttpCore

> dump some
>                information and try to continue in the face of RT
> exceptions. What I
>                feel is that we need to enhance the NIO transport to be
> stable in the
>                face of broken, partial and malicious requests so that
> RT exceptions
>                will not be thrown in such cases, and we will drop the
> connection with a
>                WARN
>
>                cheers
>                asankha
>
>                --
>                Asankha C. Perera
>                AdroitLogic, http://adroitlogic.org
>
>                http://esbmagic.blogspot.com
>
>
> ---------------------------------------------------------------------
>                To unsubscribe, e-mail: dev-...@...
>                For additional commands, e-mail:
> dev-...@...
>
>

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Ruwan Linton]

Let me get back to you after testing this scenario. Thanks for the input
Daniel. You may also raise a JIRA issue for this.

Thanks,
Ruwan

On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel <Daniel_Moise@...>wrote:

> Hi Ruwan,
>
> I've tried the last changes in the 1.3 branch, and the problem is still
> there. To test it, we've just run a 'telnet localhost 8280'.
> Based on your answer, it seems that you guys have a solution for this.
> Could you send me a patch or the changes for that. You don't have to
> necessarily check them in the branch for now.
>
> Thanks,
> Daniel
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@...]
> Sent: Wednesday, September 23, 2009 6:04 PM
> To: user@...
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Hi Daniel,
>
> This should be fixed now on the trunk, were you trying this on the
> trunk?
> Can you please verify this on a SNAPSHOT build?
>
> This will be available with the 1.3 release.
>
> Thanks,
> Ruwan
>
> On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> <Daniel_Moise@...>wrote:
>
> > Hi,
> >
> > I've found the same problem as the one listed below - every time a
> > malformed request is given to Synapse, an infinite loop will be
> > triggered making the Synapse server sluggish, and consuming the entire
> > log file (in case the output is redirected). I've noticed that the
> > problem might get fixed. Could you please let me know if the problem
> was
> > fixes, and if yes in which release.
> >
> > Thanks,
> > Daniel
> >
> >
> >                Hi All
> >
> >                I guess this boils down to how we deal with broken,
> > partial, and even
> >                possibly malicious requests in the NIO transport.. I
> > will fix the issue
> >                pointed to by Andreas and Murali ASAP as I consider
> this
> > critical.
> >
> >                Murali - I believe you should plan on updating to
> > Synapse 1.3 in the
> >                near future. There are a few issues we have fixed over
> > the past year
> >                that should be very important for any production
> > deployment. As I
> >                believe your environment / configuration is
> > straightforward, so the
> >                migration should be quite trivial too - Could you also
> > raise an issue on
> >                the JIRA?
> >
> >                This runtime exception is caught in
> > BaseIOReactor#writable and delegated to the
> >                registered exception handler (in our case an anonymous
> > inner class of
> >                HttpCoreNIOListener with the following implementation:
> >
> >                ioReactor.setExceptionHandler(new
> > IOReactorExceptionHandler() {
> >                                public boolean handle(IOException
> > ioException) {
> >                                    log.warn("System may be unstable:
> > IOReactor encountered a
> >                checked exception : "
> >                                            + ioException.getMessage(),
> > ioException);
> >                                    return true;
> >                                }
> >
> >                                public boolean handle(RuntimeException
> > runtimeException) {
> >                                    log.warn("System may be unstable:
> > IOReactor encountered a
> >                runtime exception : "
> >                                            +
> > runtimeException.getMessage(), runtimeException);
> >                                    return true;
> >                                }
> >                            });
> >
> >                I wondering a bit about the return value. Taken from
> the
> > JavaDoc of the
> >                interface IOReactorExceptionHandler#handle
> >
> >                True if it is safe to ignore the exception and continue
> > execution of the I/O
> >                reactor; if the I/O reactor must throw {@link
> > RuntimeException} and terminate
> >
> >                Hmm, is it really safe to proceed here? We do not test
> > any detail of the
> >                RuntimeException. To me this looks wrong.
> >
> >                Yes, we return true since the later versions of
> HttpCore
> > dump some
> >                information and try to continue in the face of RT
> > exceptions. What I
> >                feel is that we need to enhance the NIO transport to be
> > stable in the
> >                face of broken, partial and malicious requests so that
> > RT exceptions
> >                will not be thrown in such cases, and we will drop the
> > connection with a
> >                WARN
> >
> >                cheers
> >                asankha
> >
> >                --
> >                Asankha C. Perera
> >                AdroitLogic, http://adroitlogic.org
> >
> >                http://esbmagic.blogspot.com
> >
> >
> > ---------------------------------------------------------------------
> >                To unsubscribe, e-mail: dev-...@...
> >                For additional commands, e-mail:
> > dev-...@...
> >
> >
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@...; cell: [phone]
> blog: http://ruwansblog.blogspot.com
>

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Moise, Daniel]

The file that contains the error is
synapse-nhttp-transport/ServerHandler.java:209
int bytesWritten = outBuf.produceContent(encoder);
where the outBuf is
ContentOutputBuffer outBuf = (ContentOutputBuffer)
context.getAttribute(RESPONSE_SOURCE_BUFFER);

In this case when the outBuf is null, I think a possible solution to fix
this is to just close the connection.
What do you think?

Thanks,
Daniel


-----Original Message-----
From: Ruwan Linton [mailto:ruwan.linton@...]
Sent: Thursday, September 24, 2009 11:53 AM
To: user@...
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null

Let me get back to you after testing this scenario. Thanks for the input
Daniel. You may also raise a JIRA issue for this.

Thanks,
Ruwan

On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel
<Daniel_Moise@...>wrote:

> Hi Ruwan,
>
> I've tried the last changes in the 1.3 branch, and the problem is
still

> there. To test it, we've just run a 'telnet localhost 8280'.
> Based on your answer, it seems that you guys have a solution for this.
> Could you send me a patch or the changes for that. You don't have to
> necessarily check them in the branch for now.
>
> Thanks,
> Daniel
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@...]
> Sent: Wednesday, September 23, 2009 6:04 PM
> To: user@...
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Hi Daniel,
>
> This should be fixed now on the trunk, were you trying this on the
> trunk?
> Can you please verify this on a SNAPSHOT build?
>
> This will be available with the 1.3 release.
>
> Thanks,
> Ruwan
>
> On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> <Daniel_Moise@...>wrote:
>
> > Hi,
> >
> > I've found the same problem as the one listed below - every time a
> > malformed request is given to Synapse, an infinite loop will be
> > triggered making the Synapse server sluggish, and consuming the

entire

> > log file (in case the output is redirected). I've noticed that the
> > problem might get fixed. Could you please let me know if the problem
> was
> > fixes, and if yes in which release.
> >
> > Thanks,
> > Daniel
> >
> >
> >                Hi All
> >
> >                I guess this boils down to how we deal with broken,
> > partial, and even
> >                possibly malicious requests in the NIO transport.. I
> > will fix the issue
> >                pointed to by Andreas and Murali ASAP as I consider
> this
> > critical.
> >
> >                Murali - I believe you should plan on updating to
> > Synapse 1.3 in the
> >                near future. There are a few issues we have fixed

over
> > the past year
> >                that should be very important for any production
> > deployment. As I
> >                believe your environment / configuration is
> > straightforward, so the
> >                migration should be quite trivial too - Could you
also
> > raise an issue on
> >                the JIRA?
> >
> >                This runtime exception is caught in
> > BaseIOReactor#writable and delegated to the
> >                registered exception handler (in our case an
anonymous
> > inner class of
> >                HttpCoreNIOListener with the following
implementation:
> >
> >                ioReactor.setExceptionHandler(new
> > IOReactorExceptionHandler() {
> >                                public boolean handle(IOException
> > ioException) {
> >                                    log.warn("System may be unstable:
> > IOReactor encountered a
> >                checked exception : "
> >                                            +
ioException.getMessage(),
> > ioException);
> >                                    return true;
> >                                }
> >
> >                                public boolean
handle(RuntimeException

> > runtimeException) {
> >                                    log.warn("System may be unstable:
> > IOReactor encountered a
> >                runtime exception : "
> >                                            +
> > runtimeException.getMessage(), runtimeException);
> >                                    return true;
> >                                }
> >                            });
> >
> >                I wondering a bit about the return value. Taken from
> the
> > JavaDoc of the
> >                interface IOReactorExceptionHandler#handle
> >
> >                True if it is safe to ignore the exception and

continue
> > execution of the I/O
> >                reactor; if the I/O reactor must throw {@link
> > RuntimeException} and terminate
> >
> >                Hmm, is it really safe to proceed here? We do not
test
> > any detail of the
> >                RuntimeException. To me this looks wrong.
> >
> >                Yes, we return true since the later versions of
> HttpCore
> > dump some
> >                information and try to continue in the face of RT
> > exceptions. What I
> >                feel is that we need to enhance the NIO transport to
be
> > stable in the
> >                face of broken, partial and malicious requests so
that
> > RT exceptions
> >                will not be thrown in such cases, and we will drop
the

> > connection with a
> >                WARN
> >
> >                cheers
> >                asankha
> >
> >                --
> >                Asankha C. Perera
> >                AdroitLogic, http://adroitlogic.org
> >
> >                http://esbmagic.blogspot.com
> >
> >
> >

---------------------------------------------------------------------

> >                To unsubscribe, e-mail: dev-...@...
> >                For additional commands, e-mail:
> > dev-...@...
> >
> >
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@...; cell: [phone]
> blog: http://ruwansblog.blogspot.com
>

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Obendorf, Mike]

Daniel -

Did you get a fix from Ruwan for this? We're now seeing the same error in one of our deployments and would like to patch that build until 1.3 is officially released. BTW, I just downloaded the latest 1.3 source and get the same error as well...

Thanks,
Mike

-----Original Message-----
From: Moise, Daniel [mailto:Daniel_Moise@...]
Sent: Thursday, September 24, 2009 11:02 AM
To: user@...
Subject: RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null

The file that contains the error is
synapse-nhttp-transport/ServerHandler.java:209
int bytesWritten = outBuf.produceContent(encoder);
where the outBuf is
ContentOutputBuffer outBuf = (ContentOutputBuffer)
context.getAttribute(RESPONSE_SOURCE_BUFFER);

In this case when the outBuf is null, I think a possible solution to fix
this is to just close the connection.
What do you think?

Thanks,
Daniel


-----Original Message-----
From: Ruwan Linton [mailto:ruwan.linton@...]
Sent: Thursday, September 24, 2009 11:53 AM
To: user@...
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null

Let me get back to you after testing this scenario. Thanks for the input
Daniel. You may also raise a JIRA issue for this.

Thanks,
Ruwan

On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel
<Daniel_Moise@...>wrote:

> Hi Ruwan,
>
> I've tried the last changes in the 1.3 branch, and the problem is
still

> there. To test it, we've just run a 'telnet localhost 8280'.
> Based on your answer, it seems that you guys have a solution for this.
> Could you send me a patch or the changes for that. You don't have to
> necessarily check them in the branch for now.
>
> Thanks,
> Daniel
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@...]
> Sent: Wednesday, September 23, 2009 6:04 PM
> To: user@...
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Hi Daniel,
>
> This should be fixed now on the trunk, were you trying this on the
> trunk?
> Can you please verify this on a SNAPSHOT build?
>
> This will be available with the 1.3 release.
>
> Thanks,
> Ruwan
>
> On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> <Daniel_Moise@...>wrote:
>
> > Hi,
> >
> > I've found the same problem as the one listed below - every time a
> > malformed request is given to Synapse, an infinite loop will be
> > triggered making the Synapse server sluggish, and consuming the

entire

> > log file (in case the output is redirected). I've noticed that the
> > problem might get fixed. Could you please let me know if the problem
> was
> > fixes, and if yes in which release.
> >
> > Thanks,
> > Daniel
> >
> >
> >                Hi All
> >
> >                I guess this boils down to how we deal with broken,
> > partial, and even
> >                possibly malicious requests in the NIO transport.. I
> > will fix the issue
> >                pointed to by Andreas and Murali ASAP as I consider
> this
> > critical.
> >
> >                Murali - I believe you should plan on updating to
> > Synapse 1.3 in the
> >                near future. There are a few issues we have fixed

over
> > the past year
> >                that should be very important for any production
> > deployment. As I
> >                believe your environment / configuration is
> > straightforward, so the
> >                migration should be quite trivial too - Could you
also
> > raise an issue on
> >                the JIRA?
> >
> >                This runtime exception is caught in
> > BaseIOReactor#writable and delegated to the
> >                registered exception handler (in our case an
anonymous
> > inner class of
> >                HttpCoreNIOListener with the following
implementation:
> >
> >                ioReactor.setExceptionHandler(new
> > IOReactorExceptionHandler() {
> >                                public boolean handle(IOException
> > ioException) {
> >                                    log.warn("System may be unstable:
> > IOReactor encountered a
> >                checked exception : "
> >                                            +
ioException.getMessage(),
> > ioException);
> >                                    return true;
> >                                }
> >
> >                                public boolean
handle(RuntimeException

> > runtimeException) {
> >                                    log.warn("System may be unstable:
> > IOReactor encountered a
> >                runtime exception : "
> >                                            +
> > runtimeException.getMessage(), runtimeException);
> >                                    return true;
> >                                }
> >                            });
> >
> >                I wondering a bit about the return value. Taken from
> the
> > JavaDoc of the
> >                interface IOReactorExceptionHandler#handle
> >
> >                True if it is safe to ignore the exception and

continue
> > execution of the I/O
> >                reactor; if the I/O reactor must throw {@link
> > RuntimeException} and terminate
> >
> >                Hmm, is it really safe to proceed here? We do not
test
> > any detail of the
> >                RuntimeException. To me this looks wrong.
> >
> >                Yes, we return true since the later versions of
> HttpCore
> > dump some
> >                information and try to continue in the face of RT
> > exceptions. What I
> >                feel is that we need to enhance the NIO transport to
be
> > stable in the
> >                face of broken, partial and malicious requests so
that
> > RT exceptions
> >                will not be thrown in such cases, and we will drop
the

> > connection with a
> >                WARN
> >
> >                cheers
> >                asankha
> >
> >                --
> >                Asankha C. Perera
> >                AdroitLogic, http://adroitlogic.org
> >
> >                http://esbmagic.blogspot.com
> >
> >
> >

---------------------------------------------------------------------

> >                To unsubscribe, e-mail: dev-...@...
> >                For additional commands, e-mail:
> > dev-...@...
> >
> >
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@...; cell: [phone]
> blog: http://ruwansblog.blogspot.com
>

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Ruwan Linton]

Hi Danile/Mike,

I had a look at the issue and I think Daniels suggestion will work. I will
fix this soon on the 1.3 branch.

Thanks,
Ruwan

On Sat, Sep 26, 2009 at 4:38 AM, Obendorf, Mike <Michael_Obendorf@...
> wrote:

> Daniel -
>
> Did you get a fix from Ruwan for this? We're now seeing the same error in
> one of our deployments and would like to patch that build until 1.3 is
> officially released. BTW, I just downloaded the latest 1.3 source and get
> the same error as well...
>
> Thanks,
> Mike
>
> -----Original Message-----
> From: Moise, Daniel [mailto:Daniel_Moise@...]
> Sent: Thursday, September 24, 2009 11:02 AM
> To: user@...
> Subject: RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> The file that contains the error is
> synapse-nhttp-transport/ServerHandler.java:209
> int bytesWritten = outBuf.produceContent(encoder);
> where the outBuf is
> ContentOutputBuffer outBuf = (ContentOutputBuffer)
> context.getAttribute(RESPONSE_SOURCE_BUFFER);
>
> In this case when the outBuf is null, I think a possible solution to fix
> this is to just close the connection.
> What do you think?
>
> Thanks,
> Daniel
>
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@...]
> Sent: Thursday, September 24, 2009 11:53 AM
> To: user@...
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Let me get back to you after testing this scenario. Thanks for the input
> Daniel. You may also raise a JIRA issue for this.
>
> Thanks,
> Ruwan
>
> On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel
> <Daniel_Moise@...>wrote:
>
> > Hi Ruwan,
> >
> > I've tried the last changes in the 1.3 branch, and the problem is
> still
> > there. To test it, we've just run a 'telnet localhost 8280'.
> > Based on your answer, it seems that you guys have a solution for this.
> > Could you send me a patch or the changes for that. You don't have to
> > necessarily check them in the branch for now.
> >
> > Thanks,
> > Daniel
> >
> > -----Original Message-----
> > From: Ruwan Linton [mailto:ruwan.linton@...]
> > Sent: Wednesday, September 23, 2009 6:04 PM
> > To: user@...
> > Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> > encountered a runtime exception : null
> >
> > Hi Daniel,
> >
> > This should be fixed now on the trunk, were you trying this on the
> > trunk?
> > Can you please verify this on a SNAPSHOT build?
> >
> > This will be available with the 1.3 release.
> >
> > Thanks,
> > Ruwan
> >
> > On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> > <Daniel_Moise@...>wrote:
> >
> > > Hi,
> > >
> > > I've found the same problem as the one listed below - every time a
> > > malformed request is given to Synapse, an infinite loop will be
> > > triggered making the Synapse server sluggish, and consuming the
> entire
> > > log file (in case the output is redirected). I've noticed that the
> > > problem might get fixed. Could you please let me know if the problem
> > was
> > > fixes, and if yes in which release.
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > >                Hi All
> > >
> > >                I guess this boils down to how we deal with broken,
> > > partial, and even
> > >                possibly malicious requests in the NIO transport.. I
> > > will fix the issue
> > >                pointed to by Andreas and Murali ASAP as I consider
> > this
> > > critical.
> > >
> > >                Murali - I believe you should plan on updating to
> > > Synapse 1.3 in the
> > >                near future. There are a few issues we have fixed
> over
> > > the past year
> > >                that should be very important for any production
> > > deployment. As I
> > >                believe your environment / configuration is
> > > straightforward, so the
> > >                migration should be quite trivial too - Could you
> also
> > > raise an issue on
> > >                the JIRA?
> > >
> > >                This runtime exception is caught in
> > > BaseIOReactor#writable and delegated to the
> > >                registered exception handler (in our case an
> anonymous
> > > inner class of
> > >                HttpCoreNIOListener with the following
> implementation:
> > >
> > >                ioReactor.setExceptionHandler(new
> > > IOReactorExceptionHandler() {
> > >                                public boolean handle(IOException
> > > ioException) {
> > >                                    log.warn("System may be unstable:
> > > IOReactor encountered a
> > >                checked exception : "
> > >                                            +
> ioException.getMessage(),
> > > ioException);
> > >                                    return true;
> > >                                }
> > >
> > >                                public boolean
> handle(RuntimeException
> > > runtimeException) {
> > >                                    log.warn("System may be unstable:
> > > IOReactor encountered a
> > >                runtime exception : "
> > >                                            +
> > > runtimeException.getMessage(), runtimeException);
> > >                                    return true;
> > >                                }
> > >                            });
> > >
> > >                I wondering a bit about the return value. Taken from
> > the
> > > JavaDoc of the
> > >                interface IOReactorExceptionHandler#handle
> > >
> > >                True if it is safe to ignore the exception and
> continue
> > > execution of the I/O
> > >                reactor; if the I/O reactor must throw {@link
> > > RuntimeException} and terminate
> > >
> > >                Hmm, is it really safe to proceed here? We do not
> test
> > > any detail of the
> > >                RuntimeException. To me this looks wrong.
> > >
> > >                Yes, we return true since the later versions of
> > HttpCore
> > > dump some
> > >                information and try to continue in the face of RT
> > > exceptions. What I
> > >                feel is that we need to enhance the NIO transport to
> be
> > > stable in the
> > >                face of broken, partial and malicious requests so
> that
> > > RT exceptions
> > >                will not be thrown in such cases, and we will drop
> the
> > > connection with a
> > >                WARN
> > >
> > >                cheers
> > >                asankha
> > >
> > >                --
> > >                Asankha C. Perera
> > >                AdroitLogic, http://adroitlogic.org
> > >
> > >                http://esbmagic.blogspot.com
> > >
> > >
> > >
> ---------------------------------------------------------------------
> > >                To unsubscribe, e-mail: dev-...@...
> > >                For additional commands, e-mail:
> > > dev-...@...
> > >
> > >
> >
> >
> > --
> > Ruwan Linton
> > Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> > WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> > email: ruwan@...; cell: [phone]
> > blog: http://ruwansblog.blogspot.com
> >
>
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@...; cell: [phone]
> blog: http://ruwansblog.blogspot.com
>

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Asankha C. Perera]

Hi Ruwan / Daniel

> I had a look at the issue and I think Daniels suggestion will work. I will
> fix this soon on the 1.3 branch.
>
>  
>> The file that contains the error is
>> synapse-nhttp-transport/ServerHandler.java:209
>> int bytesWritten = outBuf.produceContent(encoder);
>> where the outBuf is
>> ContentOutputBuffer outBuf = (ContentOutputBuffer)
>> context.getAttribute(RESPONSE_SOURCE_BUFFER);
>>
>> In this case when the outBuf is null, I think a possible solution to fix
>> this is to just close the connection.
>> What do you think?
>>    

Could you create a JIRA and attach a log you see? Ensure that it
captures the first time the runtime exception occurs. Is it also
possible to get a tcpdump of the communication that causes this during
normal operation - unless someone is trying to intentionally break
Synapse. Simply closing the connection here would not be the proper
solution - but something that will just cover up something else that
happened. We should use this opportunity to understand why this happens
during normal/abnormal operations and ensure that other possibilities
which may lead to this is also analyzed

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

[Obendorf, Mike]

Looking at the differences between 1.2 and 1.3 it appears that keepalive
connection optimizations were put in place that perhaps caused this
regression. I say regression, because I don't see the infinite loop when
running 1.2 synapse with the simple telnet 8280 test. Also, I did not
see this error in any of my testing until I deployed to a box which was
subject to keepalive processing. I'm definitely no expert in this area,
so this is all just based on anecdotal evidence - I'm just throwing
these observations out there for you to chew on...  

Thanks,
Mike

-----Original Message-----
From: Asankha Perera [mailto:asankha.apache@...] On Behalf Of
Asankha C. Perera
Sent: Friday, September 25, 2009 8:28 PM
To: user@...
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null

Hi Ruwan / Daniel
> I had a look at the issue and I think Daniels suggestion will work. I
will

> fix this soon on the 1.3 branch.
>
>  
>> The file that contains the error is
>> synapse-nhttp-transport/ServerHandler.java:209
>> int bytesWritten = outBuf.produceContent(encoder);
>> where the outBuf is
>> ContentOutputBuffer outBuf = (ContentOutputBuffer)
>> context.getAttribute(RESPONSE_SOURCE_BUFFER);
>>
>> In this case when the outBuf is null, I think a possible solution to

fix
>> this is to just close the connection.
>> What do you think?
>>    
Could you create a JIRA and attach a log you see? Ensure that it
captures the first time the runtime exception occurs. Is it also
possible to get a tcpdump of the communication that causes this during
normal operation - unless someone is trying to intentionally break
Synapse. Simply closing the connection here would not be the proper
solution - but something that will just cover up something else that
happened. We should use this opportunity to understand why this happens
during normal/abnormal operations and ensure that other possibilities
which may lead to this is also analyzed

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

[Asankha C. Perera]

Hi Mike
> Looking at the differences between 1.2 and 1.3 it appears that keepalive
> connection optimizations were put in place that perhaps caused this
> regression. I say regression, because I don't see the infinite loop when
> running 1.2 synapse with the simple telnet 8280 test. Also, I did not
> see this error in any of my testing until I deployed to a box which was
> subject to keepalive processing. I'm definitely no expert in this area,
> so this is all just based on anecdotal evidence - I'm just throwing
> these observations out there for you to chew on...  
>  
There was a proposal from one of the users that we should recover from
fatal exceptions by auto-restarting the IOReactors. I think thats a good
thing as certain severe network level failures etc could otherwise put
Synapse into a non-recoverable state, in which case shutting down and
restarting automatically is better than trying to log the error and
continue in an infinite loop.

However, I cannot think why this could happen in 1.3, instead of 1.2 in
"normal" processing.. if you could provide a tcp dump (taken on Linux as
"sudo tcpdump -i eth0 -X -n -s 4096 port 8280 > dump.txt" would be very
valuable to find out the cause of this. Will it be possible to get this log?

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

[Obendorf, Mike]

Thanks for the tcpdump reminder... Here is log output (nohup) and tcdump
(dump) from 1.2 and 1.3. You can see the infinite loop in the 1.3 log (I
truncated for brevity), whereas the 1.2 log just shows the error once.

Thanks,
Mike


-----Original Message-----
From: Asankha Perera [mailto:asankha.apache@...] On Behalf Of
Asankha C. Perera
Sent: Saturday, September 26, 2009 7:47 PM
To: user@...
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null

Hi Mike
> Looking at the differences between 1.2 and 1.3 it appears that
keepalive
> connection optimizations were put in place that perhaps caused this
> regression. I say regression, because I don't see the infinite loop
when
> running 1.2 synapse with the simple telnet 8280 test. Also, I did not
> see this error in any of my testing until I deployed to a box which
was
> subject to keepalive processing. I'm definitely no expert in this
area,
> so this is all just based on anecdotal evidence - I'm just throwing
> these observations out there for you to chew on...  
>  
There was a proposal from one of the users that we should recover from
fatal exceptions by auto-restarting the IOReactors. I think thats a good
thing as certain severe network level failures etc could otherwise put
Synapse into a non-recoverable state, in which case shutting down and
restarting automatically is better than trying to log the error and
continue in an infinite loop.

However, I cannot think why this could happen in 1.3, instead of 1.2 in
"normal" processing.. if you could provide a tcp dump (taken on Linux as
"sudo tcpdump -i eth0 -X -n -s 4096 port 8280 > dump.txt" would be very
valuable to find out the cause of this. Will it be possible to get this
log?

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com





13:03:54.840459 IP 172.19.52.21.19472 > 172.16.204.44.http: S 462311085:462311085(0) win 64240 <mss 1370,nop,wscale 0,nop,nop,sackOK>
        0x0000:  4500 0034 b84d 4000 7c06 ee10 ac13 3415  E..4.M@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4ead 0000 0000  ...,L..P..N.....
        0x0020:  8002 faf0 6580 0000 0204 055a 0103 0300  ....e......Z....
        0x0030:  0101 0402                                ....
13:03:54.840468 IP 172.16.204.44.http > 172.19.52.21.19472: S 468148089:468148089(0) ack 462311086 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
        0x0000:  4500 0034 0000 4000 4006 e25e ac10 cc2c  E..4..@.@..^...,
        0x0010:  ac13 3415 0050 4c10 1be7 5f79 1b8e 4eae  ..4..PL..._y..N.
        0x0020:  8012 16d0 cdce 0000 0204 05b4 0101 0402  ................
        0x0030:  0103 0307                                ....
13:03:54.858528 IP 172.19.52.21.19472 > 172.16.204.44.http: . ack 1 win 64390
        0x0000:  4500 0028 b84f 4000 7c06 ee1a ac13 3415  E..(.O@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eae 1be7 5f7a  ...,L..P..N..._z
        0x0020:  5010 fb86 29ea 0000                      P...)...
13:03:55.966792 IP 172.19.52.21.19472 > 172.16.204.44.http: P 1:2(1) ack 1 win 64390
        0x0000:  4500 0029 b85f 4000 7c06 ee09 ac13 3415  E..)._@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eae 1be7 5f7a  ...,L..P..N..._z
        0x0020:  5018 fb86 c2e0 0000 67                   P.......g
13:03:55.966811 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 2 win 46
        0x0000:  4500 0028 cd2f 4000 4006 153b ac10 cc2c  E..(./@.@..;...,
        0x0010:  ac13 3415 0050 4c10 1be7 5f7a 1b8e 4eaf  ..4..PL..._z..N.
        0x0020:  5010 002e 2542 0000                      P...%B..
13:03:56.044420 IP 172.19.52.21.19472 > 172.16.204.44.http: P 2:3(1) ack 1 win 64390
        0x0000:  4500 0029 b863 4000 7c06 ee05 ac13 3415  E..).c@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eaf 1be7 5f7a  ...,L..P..N..._z
        0x0020:  5018 fb86 c4df 0000 65                   P.......e
13:03:56.044434 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 3 win 46
        0x0000:  4500 0028 cd30 4000 4006 153a ac10 cc2c  E..(.0@.@..:...,
        0x0010:  ac13 3415 0050 4c10 1be7 5f7a 1b8e 4eb0  ..4..PL..._z..N.
        0x0020:  5010 002e 2541 0000                      P...%A..
13:03:56.154721 IP 172.19.52.21.19472 > 172.16.204.44.http: P 3:4(1) ack 1 win 64390
        0x0000:  4500 0029 b865 4000 7c06 ee03 ac13 3415  E..).e@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb0 1be7 5f7a  ...,L..P..N..._z
        0x0020:  5018 fb86 b5de 0000 74                   P.......t
13:03:56.154732 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 4 win 46
        0x0000:  4500 0028 cd31 4000 4006 1539 ac10 cc2c  E..(.1@.@..9...,
        0x0010:  ac13 3415 0050 4c10 1be7 5f7a 1b8e 4eb1  ..4..PL..._z..N.
        0x0020:  5010 002e 2540 0000                      P...%@..
13:03:56.256981 IP 172.19.52.21.19472 > 172.16.204.44.http: P 4:6(2) ack 1 win 64390
        0x0000:  4500 002a b869 4000 7c06 edfe ac13 3415  E..*.i@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb1 1be7 5f7a  ...,L..P..N..._z
        0x0020:  5018 fb86 1cd3 0000 0d0a                 P.........
13:03:56.257004 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 6 win 46
        0x0000:  4500 0028 cd32 4000 4006 1538 ac10 cc2c  E..(.2@.@..8...,
        0x0010:  ac13 3415 0050 4c10 1be7 5f7a 1b8e 4eb3  ..4..PL..._z..N.
        0x0020:  5010 002e 253e 0000                      P...%>..
13:03:56.272411 IP 172.16.204.44.http > 172.19.52.21.19472: P 1:149(148) ack 6 win 46
        0x0000:  4500 00bc cd33 4000 4006 14a3 ac10 cc2c  E....3@.@......,
        0x0010:  ac13 3415 0050 4c10 1be7 5f7a 1b8e 4eb3  ..4..PL..._z..N.
        0x0020:  5018 002e 67a8 0000 4854 5450 2f31 2e30  P...g...HTTP/1.0
        0x0030:  2034 3030 2042 6164 2052 6571 7565 7374  .400.Bad.Request
        0x0040:  0d0a 4461 7465 3a20 5375 6e2c 2032 3720  ..Date:.Sun,.27.
        0x0050:  5365 7020 3230 3039 2032 303a 3033 3a35  Sep.2009.20:03:5
        0x0060:  3620 474d 540d 0a43 6f6e 7465 6e74 2d4c  6.GMT..Content-L
        0x0070:  656e 6774 683a 2034 390d 0a43 6f6e 7465  ength:.49..Conte
        0x0080:  6e74 2d54 7970 653a 2074 6578 742f 706c  nt-Type:.text/pl
        0x0090:  6169 6e3b 2063 6861 7273 6574 3d55 532d  ain;.charset=US-
        0x00a0:  4153 4349 490d 0a43 6f6e 6e65 6374 696f  ASCII..Connectio
        0x00b0:  6e3a 2043 6c6f 7365 0d0a 0d0a            n:.Close....
13:03:56.464548 IP 172.19.52.21.19472 > 172.16.204.44.http: . ack 149 win 64242
        0x0000:  4500 0028 b86d 4000 7c06 edfc ac13 3415  E..(.m@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb3 1be7 600e  ...,L..P..N...`.
        0x0020:  5010 faf2 29e5 0000                      P...)...
13:04:00.300434 IP 172.19.52.21.19472 > 172.16.204.44.http: P 6:7(1) ack 149 win 64242
        0x0000:  4500 0029 b89d 4000 7c06 edcb ac13 3415  E..)..@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb3 1be7 600e  ...,L..P..N...`.
        0x0020:  5018 faf2 c2db 0000 67                   P.......g
13:04:00.340475 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 7 win 46
        0x0000:  4500 0028 cd34 4000 4006 1536 ac10 cc2c  E..(.4@.@..6...,
        0x0010:  ac13 3415 0050 4c10 1be7 600e 1b8e 4eb4  ..4..PL...`...N.
        0x0020:  5010 002e 24a9 0000                      P...$...
13:04:00.395461 IP 172.19.52.21.19472 > 172.16.204.44.http: P 7:8(1) ack 149 win 64242
        0x0000:  4500 0029 b8a1 4000 7c06 edc7 ac13 3415  E..)..@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb4 1be7 600e  ...,L..P..N...`.
        0x0020:  5018 faf2 c4da 0000 65                   P.......e
13:04:00.395475 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 8 win 46
        0x0000:  4500 0028 cd35 4000 4006 1535 ac10 cc2c  E..(.5@.@..5...,
        0x0010:  ac13 3415 0050 4c10 1be7 600e 1b8e 4eb5  ..4..PL...`...N.
        0x0020:  5010 002e 24a8 0000                      P...$...
13:04:00.503398 IP 172.19.52.21.19472 > 172.16.204.44.http: P 8:9(1) ack 149 win 64242
        0x0000:  4500 0029 b8a3 4000 7c06 edc5 ac13 3415  E..)..@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb5 1be7 600e  ...,L..P..N...`.
        0x0020:  5018 faf2 b5d9 0000 74                   P.......t
13:04:00.503419 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 9 win 46
        0x0000:  4500 0028 cd36 4000 4006 1534 ac10 cc2c  E..(.6@.@..4...,
        0x0010:  ac13 3415 0050 4c10 1be7 600e 1b8e 4eb6  ..4..PL...`...N.
        0x0020:  5010 002e 24a7 0000                      P...$...
13:04:00.607526 IP 172.19.52.21.19472 > 172.16.204.44.http: P 9:11(2) ack 149 win 64242
        0x0000:  4500 002a b8a7 4000 7c06 edc0 ac13 3415  E..*..@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb6 1be7 600e  ...,L..P..N...`.
        0x0020:  5018 faf2 1cce 0000 0d0a                 P.........
13:04:00.607544 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 11 win 46
        0x0000:  4500 0028 cd37 4000 4006 1533 ac10 cc2c  E..(.7@.@..3...,
        0x0010:  ac13 3415 0050 4c10 1be7 600e 1b8e 4eb8  ..4..PL...`...N.
        0x0020:  5010 002e 24a5 0000                      P...$...
13:04:02.286520 IP 172.19.52.21.19472 > 172.16.204.44.http: P 11:12(1) ack 149 win 64242
        0x0000:  4500 0029 b8bf 4000 7c06 eda9 ac13 3415  E..)..@.|.....4.
        0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb8 1be7 600e  ...,L..P..N...`.
        0x0020:  5018 faf2 26d7 0000 03                   P...&....
13:04:02.286542 IP 172.16.204.44.http > 172.19.52.21.19472: . ack 12 win 46
        0x0000:  4500 0028 cd38 4000 4006 1532 ac10 cc2c  E..(.8@.@..2...,
        0x0010:  ac13 3415 0050 4c10 1be7 600e 1b8e 4eb9  ..4..PL...`...N.
        0x0020:  5010 002e 24a4 0000                      P...$...


11:04:53.558474 IP 172.18.33.181.3821 > 172.16.204.44.8280: S 4091873927:4091873927(0) win 65535 <mss 1460,nop,nop,sackOK>
        0x0000:  4500 0030 ce15 4000 7c06 eaad ac12 21b5  E..0..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a87 0000 0000  ...,...X........
        0x0020:  7002 ffff 0f69 0000 0204 05b4 0101 0402  p....i..........
11:04:53.558484 IP 172.16.204.44.8280 > 172.18.33.181.3821: S 1509937167:1509937167(0) ack 4091873928 win 5840 <mss 1460,nop,nop,sackOK>
        0x0000:  4500 0030 0000 4000 4006 f4c3 ac10 cc2c  E..0..@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d00f f3e5 0a88  ..!..X..Y.......
        0x0020:  7012 16d0 ce78 0000 0204 05b4 0101 0402  p....x..........
11:04:53.558779 IP 172.18.33.181.3821 > 172.16.204.44.8280: . ack 1 win 65535
        0x0000:  4500 0028 ce16 4000 7c06 eab4 ac12 21b5  E..(..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a88 59ff d010  ...,...X....Y...
        0x0020:  5010 ffff 120d 0000                      P.......
11:04:55.770722 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 1:2(1) ack 1 win 65535
        0x0000:  4500 0029 ce39 4000 7c06 ea90 ac12 21b5  E..).9@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a88 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff ab03 0000 67                   P.......g
11:04:55.770741 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 2 win 5840
        0x0000:  4500 0028 43e6 4000 4006 b0e5 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a89  ..!..X..Y.......
        0x0020:  5010 16d0 fb3b 0000                      P....;..
11:04:55.872081 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 2:3(1) ack 1 win 65535
        0x0000:  4500 0029 ce3c 4000 7c06 ea8d ac12 21b5  E..).<@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a89 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff ad02 0000 65                   P.......e
11:04:55.872111 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 3 win 5840
        0x0000:  4500 0028 43e7 4000 4006 b0e4 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a8a  ..!..X..Y.......
        0x0020:  5010 16d0 fb3a 0000                      P....:..
11:04:55.952952 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 3:4(1) ack 1 win 65535
        0x0000:  4500 0029 ce3e 4000 7c06 ea8b ac12 21b5  E..).>@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a8a 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 9e01 0000 74                   P.......t
11:04:55.952964 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 4 win 5840
        0x0000:  4500 0028 43e8 4000 4006 b0e3 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a8b  ..!..X..Y.......
        0x0020:  5010 16d0 fb39 0000                      P....9..
11:04:56.126595 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 4:6(2) ack 1 win 65535
        0x0000:  4500 002a ce42 4000 7c06 ea86 ac12 21b5  E..*.B@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a8b 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 04f6 0000 0d0a                 P.........
11:04:56.126606 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 6 win 5840
        0x0000:  4500 0028 43e9 4000 4006 b0e2 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a8d  ..!..X..Y.......
        0x0020:  5010 16d0 fb37 0000                      P....7..
11:04:59.132791 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 6:7(1) ack 1 win 65535
        0x0000:  4500 0029 ce65 4000 7c06 ea64 ac12 21b5  E..).e@.|..d..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a8d 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 0eff 0000 03                   P........
11:04:59.132810 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 7 win 5840
        0x0000:  4500 0028 43ea 4000 4006 b0e1 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a8e  ..!..X..Y.......
        0x0020:  5010 16d0 fb36 0000                      P....6..
11:04:59.680453 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 7:9(2) ack 1 win 65535
        0x0000:  4500 002a ce6c 4000 7c06 ea5c ac12 21b5  E..*.l@.|..\..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a8e 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 04f3 0000 0d0a                 P.........
11:04:59.680472 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 9 win 5840
        0x0000:  4500 0028 43eb 4000 4006 b0e0 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a90  ..!..X..Y.......
        0x0020:  5010 16d0 fb34 0000                      P....4..
11:04:59.998298 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 9:10(1) ack 1 win 65535
        0x0000:  4500 0029 ce71 4000 7c06 ea58 ac12 21b5  E..).q@.|..X..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a90 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff acfb 0000 65                   P.......e
11:04:59.998311 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 10 win 5840
        0x0000:  4500 0028 43ec 4000 4006 b0df ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a91  ..!..X..Y.......
        0x0020:  5010 16d0 fb33 0000                      P....3..
11:05:00.210518 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 10:11(1) ack 1 win 65535
        0x0000:  4500 0029 ce80 4000 7c06 ea49 ac12 21b5  E..)..@.|..I..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a91 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 99fa 0000 78                   P.......x
11:05:00.210535 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 11 win 5840
        0x0000:  4500 0028 43ed 4000 4006 b0de ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a92  ..!..X..Y.......
        0x0020:  5010 16d0 fb32 0000                      P....2..
11:05:00.304898 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 11:12(1) ack 1 win 65535
        0x0000:  4500 0029 ce84 4000 7c06 ea45 ac12 21b5  E..)..@.|..E..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a92 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff a8f9 0000 69                   P.......i
11:05:00.304931 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 12 win 5840
        0x0000:  4500 0028 43ee 4000 4006 b0dd ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a93  ..!..X..Y.......
        0x0020:  5010 16d0 fb31 0000                      P....1..
11:05:00.431722 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 12:13(1) ack 1 win 65535
        0x0000:  4500 0029 ce89 4000 7c06 ea40 ac12 21b5  E..)..@.|..@..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a93 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 9df8 0000 74                   P.......t
11:05:00.431801 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 13 win 5840
        0x0000:  4500 0028 43ef 4000 4006 b0dc ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a94  ..!..X..Y.......
        0x0020:  5010 16d0 fb30 0000                      P....0..
11:05:00.502522 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 13:15(2) ack 1 win 65535
        0x0000:  4500 002a ce8b 4000 7c06 ea3d ac12 21b5  E..*..@.|..=..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a94 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 04ed 0000 0d0a                 P.........
11:05:00.502590 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 15 win 5840
        0x0000:  4500 0028 43f0 4000 4006 b0db ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a96  ..!..X..Y.......
        0x0020:  5010 16d0 fb2e 0000                      P.......
11:05:01.192547 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 15:16(1) ack 1 win 65535
        0x0000:  4500 0029 ce97 4000 7c06 ea32 ac12 21b5  E..)..@.|..2..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a96 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff a0f5 0000 71                   P.......q
11:05:01.192738 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 16 win 5840
        0x0000:  4500 0028 43f1 4000 4006 b0da ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a97  ..!..X..Y.......
        0x0020:  5010 16d0 fb2d 0000                      P....-..
11:05:01.256701 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 16:17(1) ack 1 win 65535
        0x0000:  4500 0029 ce99 4000 7c06 ea30 ac12 21b5  E..)..@.|..0..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a97 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 9cf4 0000 75                   P.......u
11:05:01.256917 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 17 win 5840
        0x0000:  4500 0028 43f2 4000 4006 b0d9 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a98  ..!..X..Y.......
        0x0020:  5010 16d0 fb2c 0000                      P....,..
11:05:01.332463 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 17:18(1) ack 1 win 65535
        0x0000:  4500 0029 ce9b 4000 7c06 ea2e ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a98 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff a8f3 0000 69                   P.......i
11:05:01.332638 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 18 win 5840
        0x0000:  4500 0028 43f3 4000 4006 b0d8 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a99  ..!..X..Y.......
        0x0020:  5010 16d0 fb2b 0000                      P....+..
11:05:01.406443 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 18:20(2) ack 1 win 65535
        0x0000:  4500 002a cea0 4000 7c06 ea28 ac12 21b5  E..*..@.|..(..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a99 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 04e8 0000 0d0a                 P.........
11:05:01.406494 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 20 win 5840
        0x0000:  4500 0028 43f4 4000 4006 b0d7 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a9b  ..!..X..Y.......
        0x0020:  5010 16d0 fb29 0000                      P....)..
11:05:01.475288 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 20:21(1) ack 1 win 65535
        0x0000:  4500 0029 cea3 4000 7c06 ea26 ac12 21b5  E..)..@.|..&..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a9b 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 9df0 0000 74                   P.......t
11:05:01.475416 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 21 win 5840
        0x0000:  4500 0028 43f5 4000 4006 b0d6 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a9c  ..!..X..Y.......
        0x0020:  5010 16d0 fb28 0000                      P....(..
11:05:05.772536 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 21:22(1) ack 1 win 65535
        0x0000:  4500 0029 ced6 4000 7c06 e9f3 ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a9c 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 0ef0 0000 03                   P........
11:05:05.772558 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 22 win 5840
        0x0000:  4500 0028 43f6 4000 4006 b0d5 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a9d  ..!..X..Y.......
        0x0020:  5010 16d0 fb27 0000                      P....'..
11:05:06.812477 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 22:24(2) ack 1 win 65535
        0x0000:  4500 002a ceed 4000 7c06 e9db ac12 21b5  E..*..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a9d 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 04e4 0000 0d0a                 P.........
11:05:06.812490 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 24 win 5840
        0x0000:  4500 0028 43f7 4000 4006 b0d4 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0a9f  ..!..X..Y.......
        0x0020:  5010 16d0 fb25 0000                      P....%..
11:05:07.853725 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 24:25(1) ack 1 win 65535
        0x0000:  4500 0029 cefc 4000 7c06 e9cd ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0a9f 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff feec 0000 13                   P........
11:05:07.853748 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 25 win 5840
        0x0000:  4500 0028 43f8 4000 4006 b0d3 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0aa0  ..!..X..Y.......
        0x0020:  5010 16d0 fb24 0000                      P....$..
11:05:08.058392 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 25:26(1) ack 1 win 65535
        0x0000:  4500 0029 cf01 4000 7c06 e9c8 ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0aa0 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff feeb 0000 13                   P........
11:05:08.058408 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 26 win 5840
        0x0000:  4500 0028 43f9 4000 4006 b0d2 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0aa1  ..!..X..Y.......
        0x0020:  5010 16d0 fb23 0000                      P....#..
11:05:08.274458 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 26:27(1) ack 1 win 65535
        0x0000:  4500 0029 cf06 4000 7c06 e9c3 ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0aa1 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff feea 0000 13                   P........
11:05:08.274477 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 27 win 5840
        0x0000:  4500 0028 43fa 4000 4006 b0d1 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0aa2  ..!..X..Y.......
        0x0020:  5010 16d0 fb22 0000                      P...."..
11:05:08.494150 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 27:28(1) ack 1 win 65535
        0x0000:  4500 0029 cf0c 4000 7c06 e9bd ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0aa2 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 10ea 0000 01                   P........
11:05:08.494274 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 28 win 5840
        0x0000:  4500 0028 43fb 4000 4006 b0d0 ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0aa3  ..!..X..Y.......
        0x0020:  5010 16d0 fb21 0000                      P....!..
11:05:08.790210 IP 172.18.33.181.3821 > 172.16.204.44.8280: P 28:29(1) ack 1 win 65535
        0x0000:  4500 0029 cf13 4000 7c06 e9b6 ac12 21b5  E..)..@.|.....!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0aa3 59ff d010  ...,...X....Y...
        0x0020:  5018 ffff 0de9 0000 04                   P........
11:05:08.790329 IP 172.16.204.44.8280 > 172.18.33.181.3821: . ack 29 win 5840
        0x0000:  4500 0028 43fc 4000 4006 b0cf ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0aa4  ..!..X..Y.......
        0x0020:  5010 16d0 fb20 0000                      P.......
11:05:18.176182 IP 172.18.33.181.3821 > 172.16.204.44.8280: F 29:29(0) ack 1 win 65535
        0x0000:  4500 0028 cf97 4000 7c06 e933 ac12 21b5  E..(..@.|..3..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0aa4 59ff d010  ...,...X....Y...
        0x0020:  5011 ffff 11f0 0000                      P.......
11:05:18.176911 IP 172.16.204.44.8280 > 172.18.33.181.3821: F 1:1(0) ack 30 win 5840
        0x0000:  4500 0028 43fd 4000 4006 b0ce ac10 cc2c  E..(C.@.@......,
        0x0010:  ac12 21b5 2058 0eed 59ff d010 f3e5 0aa5  ..!..X..Y.......
        0x0020:  5011 16d0 fb1e 0000                      P.......
11:05:18.177186 IP 172.18.33.181.3821 > 172.16.204.44.8280: . ack 2 win 65535
        0x0000:  4500 0028 cf98 4000 7c06 e932 ac12 21b5  E..(..@.|..2..!.
        0x0010:  ac10 cc2c 0eed 2058 f3e5 0aa5 59ff d011  ...,...X....Y...
        0x0020:  5010 ffff 11ef 0000                      P.......


 Using Bouncy castle JAR for Java 1.5
Starting Synapse/Java ...
Using SYNAPSE_HOME:    /pharos/env/env_1743/synapse-1.2
Using JAVA_HOME:       /pharos/lang_app/jdk1.5.0_12
Using SYNAPSE_XML:     /pharos/env/env_1743/synapse-1.2/repository/conf/synapse.xml
2009-09-27 17:55:04,715 [-] [main]  INFO ServerManager Using Synapse home as : /pharos/env/env_1743/synapse-1.2
2009-09-27 17:55:04,716 [-] [main]  INFO ServerManager Using the Axis2 Repository : /pharos/env/env_1743/synapse-1.2/repository
2009-09-27 17:55:04,717 [-] [main]  INFO ServerManager Using the axis2.xml : /pharos/env/env_1743/synapse-1.2/repository/conf/axis2.xml
2009-09-27 17:55:04,717 [-] [main]  INFO ServerManager Using server name : localhost
2009-09-27 17:55:04,717 [-] [main]  INFO ServerManager The timeout handler will run every : 15s
2009-09-27 17:55:05,188 [-] [main]  INFO SynapseInitializationModule Initializing Synapse at : Sun Sep 27 17:55:05 GMT 2009
2009-09-27 17:55:05,189 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseInitializationModule Loading mediator extensions...
2009-09-27 17:55:05,189 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseInitializationModule Initializing the Synapse configuration ...
2009-09-27 17:55:05,197 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration
2009-09-27 17:55:05,276 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseConfigurationBuilder Loaded Synapse configuration from : /pharos/env/env_1743/synapse-1.2/repository/conf/synapse.xml
2009-09-27 17:55:05,282 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseInitializationModule Deploying the Synapse service..
2009-09-27 17:55:05,300 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseInitializationModule Synapse server name : localhost
2009-09-27 17:55:05,300 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseInitializationModule Deploying Proxy services...
2009-09-27 17:55:05,300 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO SynapseInitializationModule Synapse initialized successfully...!
2009-09-27 17:55:05,325 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOSSLSender Loading Identity Keystore from : lib/identity.jks
2009-09-27 17:55:05,340 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOSSLSender Loading Trust Keystore from : lib/trust.jks
2009-09-27 17:55:05,344 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOSender HTTPS Sender starting
2009-09-27 17:55:05,373 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOSender HTTP Sender starting
2009-09-27 17:55:05,376 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO VFSTransportSender VFS Sender started
2009-09-27 17:55:05,470 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO JMSSender JMS Sender started
2009-09-27 17:55:05,478 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO ServerManager Starting transport https on port 8243
2009-09-27 17:55:05,479 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOSSLListener Loading Identity Keystore from : lib/identity.jks
2009-09-27 17:55:05,481 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOSSLListener Loading Trust Keystore from : lib/trust.jks
2009-09-27 17:55:05,483 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOListener HTTPS Listener starting on port : 8243
2009-09-27 17:55:05,483 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO ServerManager Starting transport http on port 8280
2009-09-27 17:55:05,485 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO HttpCoreNIOListener HTTP Listener starting on port : 8280
2009-09-27 17:55:05,485 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO ServerManager Starting transport vfs
2009-09-27 17:55:05,494 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO VFSTransportListener VFS Listener started
2009-09-27 17:55:05,497 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO ServerManager Starting transport mailto
2009-09-27 17:55:05,498 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO MailTransportListener MAILTO Listener started
2009-09-27 17:55:05,498 [127.0.0.1-host2053.pharos.intuit.com] [main]  INFO ServerManager Ready for processing
2009-09-27 17:55:26,898 [127.0.0.1-host2053.pharos.intuit.com] [I/O dispatcher 5]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.exception(ServerHandler.java:274)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput(DefaultNHttpServerConnection.java:146)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.inputReady(DefaultServerIOEventDispatch.java:98)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:98)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:195)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:180)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:142)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:70)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:318)
        at java.lang.Thread.run(Thread.java:595)

 [Warn] Synapse is tested only with Java 5
Starting Synapse/Java ...
Using SYNAPSE_HOME:    /pharos/env/env_1743/Gateway-1.0.0/synapse
Using JAVA_HOME:       /pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux
Using SYNAPSE_XML:     /pharos/env/env_1743/Gateway-1.0.0/synapse/repository/conf/synapse.xml
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] com.wily.introscope.agentProfile=wily/IntroscopeAgent.profile
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] com.wily.log4j.defaultInitOverride=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] file.encoding=UTF-8
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] file.encoding.pkg=sun.io
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] file.separator=/
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.agentAutoNamingEnabled=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds=120
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.agentAutoRenamingIntervalInMinutes=10
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.agentName=SynapseMetrics
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.agentNameSystemPropertyKey=
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.blame.type=boundary
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.canAutoNameAgent=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.clonedAgent=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.customProcessName=Synapse
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.defaultProcessName=UnknownProcess
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.disableLogFileAutoNaming=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.enterprisemanager.connectionorder=DEFAULT
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=host2104.pharos.intuit.com
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT=5001
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT=com.wily.isengard.postofficehub.link.net.DefaultSocketFactory
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.errorsnapshots.enable=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.errorsnapshots.throttle=10
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.extensions.directory=ext
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.jmx.enable=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.jmx.excludeStringMetrics=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.jmx.name.filter=
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.collectAllocationStackTraces=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.enable=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.0=org.apache.taglibs.standard.lang.jstl.*
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.1=com.bea.medrec.entities.RecordEJB_xwcp6o__WebLogic_CMP_RDBMS
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.2=net.sf.hibernate.collection.*
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.3=org.jnp.interfaces.FastNamingProperties
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.4=java.util.SubList
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.5=com.sun.faces.context.BaseContextMap$EntrySet
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.6=com.sun.faces.context.BaseContextMap$KeySet
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.7=com.sun.faces.context.SessionMap
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.8=java.util.Collections$UnmodifiableMap
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.ignore.9=org.hibernate.collection.PersistentSet
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.leakSensitivity=5
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.logfile.append=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.logfile.location=logs/LeakHunter.log
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.leakhunter.timeoutInMinutes=120
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.metricAging.dataChunk=500
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.metricAging.heartbeatInterval=1800
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.metricAging.metricExclude.ignore.0=Threads*
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.metricAging.numberTimeslices=3000
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.metricAging.turnOn=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.alarmManager=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.bean=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.cache=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.connectionPool=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.hamanager=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.j2c=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.jvmRuntime=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.objectPool=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.orbPerf=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.scheduler=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.servletSessions=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.system=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.threadPool=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.transaction=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.webApp=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.webServices=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.wlm=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.pmi.enable.wsgw=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.remoteagentconfiguration.allowedFiles=domainconfig.xml
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.remoteagentconfiguration.enabled=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.sockets.reportRateMetrics=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.sqlagent.useblame=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.stalls.resolutionseconds=10
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.stalls.thresholdseconds=30
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.urlgroup.group.default.format=Default
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.urlgroup.group.default.pathprefix=*
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.agent.urlgroup.keys=default
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.directivesFile=default-full.pbl,hotdeploy
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo=10
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs=1
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.dynamicinstrument.enabled=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.dynamicinstrument.pollIntervalMinutes=1
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.enable=true
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.autoprobe.logfile=logs/AutoProbe.log
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] introscope.enterprisemanager.serversockets.reuseaddr=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.awt.printerjob=sun.print.PSPrinterJob
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.class.path=/pharos/env/env_1743/Gateway-1.0.0/synapse/repository/conf:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/tools.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/patches:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/patches/httpcore-nio-4.0-patch-httpcore-193.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/activation-1.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axiom-api-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axiom-dom-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axiom-impl-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-adb-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-clustering-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-codegen-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-kernel-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-mtompolicy-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-saaj-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-transport-base-1.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-transport-http-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-transport-jms-1.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/axis2-transport-mail-1.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/backport-util-concurrent-2.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/bcel-5.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/bcprov-jdk15-140.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/bsf-all-3.0-beta2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-cli-1.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-codec-1.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-collections-3.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-dbcp-1.2.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-fileupload-1.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-httpclient-3.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-io-1.4.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-lang-1.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-logging-1.1.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-net-1.4.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-pool-1.3.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/commons-vfs-1.1-587797.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/dom4j-1.4.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/Gateway-1.0.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-activation_1.1_spec-1.0.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-javamail_1.4_spec-1.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-jms_1.1_spec-1.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-jta_1.1_spec-1.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-saaj_1.3_spec-1.0.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-stax-api_1.0_spec-1.0.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/geronimo-ws-metadata_2.0_spec-1.1.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/groovy-all-1.1-rc-1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/httpcore-4.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/httpcore-nio-4.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/jakarta-regexp-1.4.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/jargs-1.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/java-cup-0.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/jaxen-1.1.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/JLex-0.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/jline-0.9.94.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/jruby-complete-0.9.9.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/js-1.6R5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/jsch-0.1.31.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/juli-6.0.16.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/log4j-1.2.14.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/mail-1.4.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/mex-1.5-impl.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/neethi-2.0.4.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/opensaml-2.2.3.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/openws-1.2.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/oro-2.0.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/quartz-1.6.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/rampart-core-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/rampart-policy-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/rampart-trust-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/resolver-2.9.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/sandesha2-core-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/saxon-8.9.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/saxon-dom-8.9.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/saxon-xqj-8.9.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/serializer-2.9.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/slf4j-jdk14-1.5.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/spring-aop-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/spring-beans-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/spring-context-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/spring-core-1.2.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/stax-api-1.0.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-commons-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-core-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-experimental-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-extensions-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-nhttp-transport-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-samples-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-tasks-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/synapse-vfs-transport-1.3.0-SNAPSHOT.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/tribes-6.0.16.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/velocity-1.5.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/woden-api-1.0M8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/woden-impl-dom-1.0M8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wrapper-3.2.3.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wsdl4j-1.6.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wso2caching-core-3.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wso2eventing-api-2.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wso2throttle-core-3.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wss4j-1.5.8.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/wstx-asl-3.2.4.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xalan-2.7.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xbean-2.2.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xercesImpl-2.9.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xml-apis-2.9.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xmlParserAPIs-2.6.0.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/XmlSchema-1.4.3.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xmlsec-1.4.2.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/xmlunit-1.1.jar::wily/Agent.jar
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.class.version=50.0
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.endorsed.dirs=/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/endorsed
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.ext.dirs=/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/ext:/usr/java/packages/lib/ext
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.home=/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.io.tmpdir=/pharos/env/env_1743/Gateway-1.0.0/synapse/work/temp/synapse
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.library.path=/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/i386/server:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/i386:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.runtime.name=Java(TM) SE Runtime Environment
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.runtime.version=1.6.0_14-b08
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.specification.name=Java Platform API Specification
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.specification.vendor=Sun Microsystems Inc.
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.specification.version=1.6
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vendor=Sun Microsystems Inc.
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vendor.url=http://java.sun.com/
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vendor.url.bug=http://java.sun.com/cgi-bin/bugreport.cgi
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.version=1.6.0_14
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.info=mixed mode
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.name=Java HotSpot(TM) Server VM
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.specification.name=Java Virtual Machine Specification
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.specification.vendor=Sun Microsystems Inc.
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.specification.version=1.0
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.vendor=Sun Microsystems Inc.
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] java.vm.version=14.0-b16
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] line.separator=

9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.additivity.IntroscopeAgent=false
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.console=com.wily.org.apache.log4j.ConsoleAppender
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.console.layout=com.wily.org.apache.log4j.PatternLayout
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.console.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z} [%-3p] [%c] %m%n
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.console.target=System.err
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.logfile=com.wily.introscope.agent.AutoNamingRollingFileAppender
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.logfile.File=logs/IntroscopeAgent.log
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.logfile.MaxBackupIndex=4
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.logfile.MaxFileSize=2MB
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.logfile.layout=com.wily.org.apache.log4j.PatternLayout
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.appender.logfile.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z} [%-3p] [%c] %m%n
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] log4j.logger.IntroscopeAgent=INFO, console, logfile
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] org.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XMLGrammarCachingConfiguration
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] os.arch=i386
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] os.name=Linux
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] os.version=2.6.18-028stab060.15
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] path.separator=:
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.arch.data.model=32
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.boot.class.path=/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/endorsed/xercesImpl-2.8.1.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/endorsed/xml-apis-1.3.04.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/resources.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/rt.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/sunrsasign.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/jsse.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/jce.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/charsets.jar:/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/classes:/pharos/env/env_1743/Gateway-1.0.0/synapse/wily/Agent.jar
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.boot.library.path=/pharos/env/env_1743/Gateway-1.0.0/synapse/jre1.6.0_14-Linux/lib/i386
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.cpu.endian=little
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.cpu.isalist=
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.io.unicode.encoding=UnicodeLittle
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.java.launcher=SUN_STANDARD
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.jnu.encoding=UTF-8
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.management.compiler=HotSpot Tiered Compilers
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] sun.os.patch.level=unknown
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] user.country=US
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] user.dir=/pharos/env/env_1743/Gateway-1.0.0/synapse
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] user.home=/root
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] user.language=en
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] user.name=root
9/27/09 08:02:32 PM GMT [INFO] [IntroscopeAgent.Properties] user.timezone=GMT
2009-09-27 20:02:33,118 [-] [main]  INFO SynapseServer Starting Apache Synapse...
2009-09-27 20:02:33,141 [-] [main]  INFO SynapseControllerFactory Using Synapse home : /pharos/env/env_1743/Gateway-1.0.0/synapse
2009-09-27 20:02:33,141 [-] [main]  INFO SynapseControllerFactory Using Axis2 repository : /pharos/env/env_1743/Gateway-1.0.0/synapse/repository
2009-09-27 20:02:33,142 [-] [main]  INFO SynapseControllerFactory Using axis2.xml location : /pharos/env/env_1743/Gateway-1.0.0/synapse/repository/conf/axis2.xml
2009-09-27 20:02:33,142 [-] [main]  INFO SynapseControllerFactory Using synapse.xml location : /pharos/env/env_1743/Gateway-1.0.0/synapse/repository/conf/synapse.xml
2009-09-27 20:02:33,142 [-] [main]  INFO SynapseControllerFactory Using server name : localhost
2009-09-27 20:02:33,163 [-] [main]  INFO SynapseControllerFactory The timeout handler will run every : 15s
2009-09-27 20:02:33,252 [-] [main]  INFO Axis2SynapseController Initializing Synapse at : Sun Sep 27 20:02:33 GMT 2009
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Introscope Agent Release 8.1.0.0 (Build 472023)
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Using Java VM version "Java HotSpot(TM) Server VM 1.6.0_14" from Sun Microsystems Inc.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Trying to load agent profile based on system property "com.wily.introscope.agentProfile"
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Trying to load file from wily/IntroscopeAgent.profile
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Loaded file from wily/IntroscopeAgent.profile
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Agent Metric Aging is turned on
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Agent Metric Aging: property introscope.agent.metricAging.heartbeatInterval is set to 1800
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Agent Metric Aging: property introscope.agent.metricAging.dataChunk is set to 500
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Agent Metric Aging: property introscope.agent.metricAging.numberTimeslices is set to 3000
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Looking for agent profile property "introscope.agent.extensions.directory" to locate the extensions directory.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] The agent extensions directory /pharos/env/env_1743/Gateway-1.0.0/synapse/wily/ext was successfully located
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Introscope AutoProbe Release 8.1.0.0 (Build 472023)
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] The Introscope AutoProbe log has been created at /pharos/env/env_1743/Gateway-1.0.0/synapse/wily/logs/AutoProbe.log
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] AutoProbe has reloaded all directive configuration.  This is because a change to a directive configuration file was detected.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Introscope Agent startup complete.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] The Agent will attempt to determine its name.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Unable to determine the Agent name from the System Property "com.wily.introscope.agent.agentName" because that property was either not found or had no value.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Unable to determine the Agent name from a user specified System Property because the property "introscope.agent.agentNameSystemPropertyKey" in the Introscope Agent Profile was either not found or had no value.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] The Agent will not attempt to automatically determine its name from the Application Server because that feature has been disabled in the Introscope Agent Profile.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Using the Agent name "SynapseMetrics" based on the "introscope.agent.agentName" property in the Introscope Agent Profile.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Initial agent name set to SynapseMetrics
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent.Isengard] Initiating connection attempts to the Introscope Enterprise Manager.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent.Isengard] The Agent reconnection delay is set to 15 second(s).
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Linux IntelAmd32 Platform Monitor started successfully.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Dynamic Instrumentation is enabled.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Started transaction sampling service
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] The supportability metric -> Metric Count will not be reported since introscope.ext.agent.metric.count is set to false
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] No Agent profile value found for introscope.agent.platform.monitor.system, using os.name system property to load Platform Monitor.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Transaction Tracer sampling interval has been reset to 120 milliseconds.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Transaction Tracer samples per interval has been set to 1.
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent.IsengardServerConnectionManager] Connected Agent to the Introscope Enterprise Manager at host2104.pharos.intuit.com:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory. Host = "host2053", Process = "Synapse", Agent Name = "SynapseMetrics".
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Configured error reporting throttle at 10
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Error snapshot collection has been enabled
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Started Error Reporting service
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Stall event service: stall threshold of 30 seconds
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Stall event service: resolution of 10 seconds
9/27/09 08:02:33 PM GMT [INFO] [IntroscopeAgent] Started Stall Service
2009-09-27 20:02:33,733 [-] [main]  INFO ClusterBuilder Clustering has been disabled
2009-09-27 20:02:33,807 [-] [main]  INFO DeploymentEngine No services directory was found under /pharos/env/env_1743/Gateway-1.0.0/synapse/repository.
2009-09-27 20:02:33,941 [-] [main]  INFO ModuleDeployer Deploying module: addressing - file:/pharos/env/env_1743/Gateway-1.0.0/synapse/repository/modules/addressing.mar
2009-09-27 20:02:34,001 [-] [main]  INFO ModuleDeployer Deploying module: rampart - file:/pharos/env/env_1743/Gateway-1.0.0/synapse/repository/modules/rampart.mar
2009-09-27 20:02:34,190 [-] [main]  INFO ModuleDeployer Deploying module: sandesha2 - file:/pharos/env/env_1743/Gateway-1.0.0/synapse/repository/modules/sandesha2.mar
2009-09-27 20:02:34,225 [-] [main]  INFO ModuleDeployer Deploying module: metadataExchange - file:/pharos/env/env_1743/Gateway-1.0.0/synapse/lib/mex-1.5-impl.jar
2009-09-27 20:02:34,302 [-] [main] ERROR SandeshaModule Could not load module policies. Using default values.
2009-09-27 20:02:34,352 [-] [main]  INFO HttpCoreNIOSSLSender Loading Identity Keystore from : lib/identity.jks
2009-09-27 20:02:34,379 [-] [main]  INFO HttpCoreNIOSSLSender Loading Trust Keystore from : lib/trust.jks
2009-09-27 20:02:34,547 [-] [main] DEBUG NativeWorkerPool Using native util.concurrent package..
2009-09-27 20:02:34,567 [-] [main]  INFO HttpCoreNIOSender HTTPS Sender starting
2009-09-27 20:02:34,577 [-] [main] DEBUG NativeWorkerPool Using native util.concurrent package..
2009-09-27 20:02:34,580 [-] [main]  INFO HttpCoreNIOSender HTTP Sender starting
2009-09-27 20:02:34,580 [-] [main]  INFO JMSSender JMS Sender started
2009-09-27 20:02:34,585 [-] [main]  INFO JMSSender JMS Transport Sender initialized...
2009-09-27 20:02:34,586 [-] [main]  INFO VFSTransportSender VFS Sender started
2009-09-27 20:02:34,620 [-] [main]  INFO DefaultFileReplicator Using "/pharos/env/env_1743/Gateway-1.0.0/synapse/work/temp/synapse/vfs_cache" as temporary files store.
2009-09-27 20:02:34,861 [-] [main]  WARN JmxAdapter Using unsecured JMX remote access!
2009-09-27 20:02:35,014 [-] [main] DEBUG NativeWorkerPool Using native util.concurrent package..
2009-09-27 20:02:35,025 [-] [main]  INFO HttpCoreNIOSSLListener Loading Identity Keystore from : lib/identity.jks
2009-09-27 20:02:35,027 [-] [main]  INFO HttpCoreNIOSSLListener Loading Trust Keystore from : lib/trust.jks
2009-09-27 20:02:35,068 [-] [main] DEBUG NativeWorkerPool Using native util.concurrent package..
2009-09-27 20:02:35,069 [-] [main]  INFO DefaultFileReplicator Using "/pharos/env/env_1743/Gateway-1.0.0/synapse/work/temp/synapse/vfs_cache" as temporary files store.
2009-09-27 20:02:35,082 [-] [main]  INFO Axis2SynapseController Loading mediator extensions...
2009-09-27 20:02:35,132 [-] [main]  INFO XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration
2009-09-27 20:02:35,295 [-] [main]  INFO SynapseConfigurationBuilder Loaded Synapse configuration from : /pharos/env/env_1743/Gateway-1.0.0/synapse/repository/conf/synapse.xml
2009-09-27 20:02:35,296 [-] [main]  WARN SynapseConfigUtils IO Error reading from URL : /pharos/env/env_1743/Gateway-1.0.0/synapse/./repository/conf/resources/faultjava.io.FileNotFoundException: /pharos/env/env_1743/Gateway-1.0.0/synapse/./repository/conf/resources/fault (No such file or directory)
2009-09-27 20:02:35,373 [-] [main]  INFO Axis2SynapseController Deploying the Synapse service...
2009-09-27 20:02:35,405 [-] [main]  INFO Axis2SynapseController Deploying Proxy services...
2009-09-27 20:02:35,405 [-] [main]  INFO ProxyService Building Axis service for Proxy service : PersonaProfileManagerSOAP
2009-09-27 20:02:36,644 [-] [main]  INFO ProxyService Adding service PersonaProfileManagerSOAP to the Axis2 configuration
2009-09-27 20:02:36,649 [-] [main]  INFO ProxyService Successfully created the Axis2 service for Proxy service : PersonaProfileManagerSOAP
2009-09-27 20:02:36,649 [-] [main]  INFO Axis2SynapseController Deployed Proxy service : PersonaProfileManagerSOAP
2009-09-27 20:02:36,649 [-] [main]  INFO ProxyService Building Axis service for Proxy service : RealmProfileManagerSOAP
2009-09-27 20:02:36,976 [-] [main]  INFO ProxyService Adding service RealmProfileManagerSOAP to the Axis2 configuration
2009-09-27 20:02:36,980 [-] [main]  INFO ProxyService Successfully created the Axis2 service for Proxy service : RealmProfileManagerSOAP
2009-09-27 20:02:36,980 [-] [main]  INFO Axis2SynapseController Deployed Proxy service : RealmProfileManagerSOAP
2009-09-27 20:02:36,981 [-] [main]  INFO Axis2SynapseController Deploying EventSources...
2009-09-27 20:02:37,042 [-] [main]  INFO SimpleThreadPool Job execution threads will use class loader of thread: main
2009-09-27 20:02:37,075 [-] [main]  INFO QuartzScheduler Quartz Scheduler v.1.6.0 created.
2009-09-27 20:02:37,079 [-] [main]  INFO RAMJobStore RAMJobStore initialized.
2009-09-27 20:02:37,079 [-] [main]  INFO StdSchedulerFactory Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2009-09-27 20:02:37,079 [-] [main]  INFO StdSchedulerFactory Quartz scheduler version: 1.6.0
2009-09-27 20:02:37,080 [-] [main]  INFO QuartzScheduler Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2009-09-27 20:02:37,098 [-] [main]  INFO MailTransportListener MAILTO Listener started
2009-09-27 20:02:37,127 [-] [DefaultQuartzScheduler_Worker-1]  INFO GatewayInit Gateway Environment from synapse.xml is: DEV
2009-09-27 20:02:37,127 [-] [main] DEBUG NativeWorkerPool Using native util.concurrent package..
2009-09-27 20:02:37,133 [-] [DefaultQuartzScheduler_Worker-1]  INFO GatewayInit Creating endpoint definition file: ./repository/conf/resources/endpoint/IDProfile-PersonaProfileManager-Autogen.xml
2009-09-27 20:02:37,135 [-] [DefaultQuartzScheduler_Worker-1]  INFO GatewayInit Endpoint URI: https://idprofile.e2e.ieptc.intuit.com/idprofile/services/PersonaProfileManagerSOAP
2009-09-27 20:02:37,136 [-] [DefaultQuartzScheduler_Worker-1]  INFO GatewayInit Creating endpoint definition file: ./repository/conf/resources/endpoint/IDProfile-RealmProfileManager-Autogen.xml
2009-09-27 20:02:37,137 [-] [DefaultQuartzScheduler_Worker-1]  INFO GatewayInit Endpoint URI: https://idprofile.e2e.ieptc.intuit.com/idprofile/services/RealmProfileManagerSOAP
2009-09-27 20:02:37,140 [-] [main]  INFO HttpCoreNIOListener HTTPS Listener started on port : 443
2009-09-27 20:02:37,141 [-] [main] DEBUG NativeWorkerPool Using native util.concurrent package..
2009-09-27 20:02:37,146 [-] [main]  INFO HttpCoreNIOListener HTTP Listener started on port : 80
2009-09-27 20:02:37,146 [-] [main]  INFO VFSTransportListener VFS Listener started
2009-09-27 20:02:37,146 [-] [main]  INFO Axis2SynapseController Management using JMX available via: service:jmx:rmi:///jndi/rmi://host2053.pharos.intuit.com:1099/synapse
2009-09-27 20:02:37,146 [-] [main]  INFO ServerManager Server ready for processing...
2009-09-27 20:02:37,146 [-] [main]  INFO SynapseServer Apache Synapse started successfully
2009-09-27 20:03:56,274 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,276 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,276 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,276 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,277 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,277 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,277 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,277 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,278 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,278 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,278 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,278 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,278 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,279 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,279 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,279 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,279 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,280 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,280 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,280 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,280 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,280 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,281 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,281 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,281 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,281 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,282 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,282 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:317)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
        at java.lang.Thread.run(Unknown Source)
2009-09-27 20:03:56,282 [-] [http-Listener I/O dispatcher-1]  WARN HttpCoreNIOListener System may be unstable: IOReactor encountered a runtime exception : null
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.ServerHandler.outputReady(ServerHandler.java:209)
        at org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:213)
        at org.apache.http.impl.nio.DefaultServerIOEventDispatch.outputReady(DefaultServerIOEventDispatch.java:151)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:177)

[Asankha C. Perera]

Hi Mike/Daniel
> Thanks for the tcpdump reminder... Here is log output (nohup) and tcdump
> (dump) from 1.2 and 1.3. You can see the infinite loop in the 1.3 log (I
> truncated for brevity), whereas the 1.2 log just shows the error once.
>  
Thanks for the log file. However, it shows that you probably telnet into
the system:

13:03:56.256981 IP 172.19.52.21.19472 > 172.16.204.44.http: P 4:6(2) ack
1 win 64390
    0x0000:  4500 002a b869 4000 7c06 edfe ac13 3415  E..*.i@.|.....4.
    0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb1 1be7 5f7a  ...,L..P..N..._z
    0x0020:  5018 fb86 1cd3 0000 *0d0a                * P.........

I was wondering if you see this by any "other" means, during supposedly
normal interactions with a client/backend system, in which case the
issue may still need more investigation. Since the suggested fix is
trivial, and serves the purpose for now, I have commited it, but I would
still like to know if this occurs under any normal circumstances.

I've filed an enhancement for HttpCore-208 after which we could improve
Synapse to better recover from generally any unexpected error. Eric has
also suggested that its better to re-start the IOReactors on fatal
exceptions, which we will incoporate with Synapse-584

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

[Obendorf, Mike]

Asankha - Thanks for committing the fix. The 'other' means by which this
error comes up, is when I deploy to a server that has active keepalive
requests hitting synapse. Unfortunately I don't know enough about these
requests to link the problem conclusively to that activity, and also
unfortunately, I was unable to run the tests in that particular
environment this weekend. I'll see if I can get on the system tomorrow
night and run tcpdump from there.

Mike

 -----Original Message-----
From: Asankha Perera [mailto:asankha.apache@...] On Behalf Of
Asankha C. Perera
Sent: Sunday, September 27, 2009 10:21 PM
To: user@...
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null

Hi Mike/Daniel
> Thanks for the tcpdump reminder... Here is log output (nohup) and
tcdump
> (dump) from 1.2 and 1.3. You can see the infinite loop in the 1.3 log
(I
> truncated for brevity), whereas the 1.2 log just shows the error once.

>  
Thanks for the log file. However, it shows that you probably telnet into
the system:

13:03:56.256981 IP 172.19.52.21.19472 > 172.16.204.44.http: P 4:6(2) ack
1 win 64390
    0x0000:  4500 002a b869 4000 7c06 edfe ac13 3415  E..*.i@.|.....4.
    0x0010:  ac10 cc2c 4c10 0050 1b8e 4eb1 1be7 5f7a  ...,L..P..N..._z
    0x0020:  5018 fb86 1cd3 0000 *0d0a                * P.........

I was wondering if you see this by any "other" means, during supposedly
normal interactions with a client/backend system, in which case the
issue may still need more investigation. Since the suggested fix is
trivial, and serves the purpose for now, I have commited it, but I would
still like to know if this occurs under any normal circumstances.

I've filed an enhancement for HttpCore-208 after which we could improve
Synapse to better recover from generally any unexpected error. Eric has
also suggested that its better to re-start the IOReactors on fatal
exceptions, which we will incoporate with Synapse-584

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

[Asankha C. Perera]

Hi Mike
> Asankha - Thanks for committing the fix. The 'other' means by which this
> error comes up, is when I deploy to a server that has active keepalive
> requests hitting synapse. Unfortunately I don't know enough about these
> requests to link the problem conclusively to that activity, and also
> unfortunately, I was unable to run the tests in that particular
> environment this weekend. I'll see if I can get on the system tomorrow
> night and run tcpdump from there.
>  
The possibility I could think of is a new and valid request coming over
a keepalive connection, which is just being closed by Synapse.. the
probability of which is low, unless the system is under load. The
current fix should work even in that case, but the client will get an
error, which may not be perfect, but I think possible options we have is
limited.

If possible, leave the tcpdump running and gzip the file once this
happens - else you may end up with a huge log file.. You may send the
file directly to me as the tcp dump may reveal message data if posted
publicly.

cheers
asankha

--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com