« Return to Thread: Difficult Performance Problem

Re: Difficult Performance Problem

by jsexton0 :: Rate this Message:

Hello -
I changed the HTTP log level to "fine" and ran the entire process in question. The result is about 60 MBs of log files. If that is of value to you, I can zip it up and place it somewhere where you can ftp it, and I can tell you where the key parts are. Otherwise, I'm not sure quite how to trim it down right at this point. As is usually the case, I'm under some time constraints to work around this issue, but I can try.

Increased payload size is definitely the issue, yes. The entire message in about 5 MBs. And yes, the delay occurs in something like this: EJB->(X)->WS where "WS" is the entry to a BPEL. The BPEL has an assign and then a call out to a remote service to pass on the data using a dynamic address. The BPEL and the calling EJB are in separate SAs, so as I understand how this works (?), the HTTP-BC is involved, rather than a direct call. The EJB calls the BPEL using jaxb style code, generated by Netbeans.

The whole process works, but I am going to have to find a way to make it faster, and avoid the time outs.

What do you think?
Thank you

Murali Pottlapelli wrote:

Increased logging may help to understand the issue, change logging
level on BPEL-SE and HTTP-BC to FINEST.

I do not need actual project, is it possible to build straw man version
of it? let us skip C# services in first version. You can start with
what you suspect most.

If I understand right call sequence is

SOAP-UI->EJB->(X)->BPEL-SE

is this right? X could be Java EE SE or http-bc? which one you are
using?

Do you think payload size is causing the issue? I did test with 10 MB
pay load.

You could use visualvm to do monitor memory and threads, will you
please share thread dump?

Regards
Murali

jsexton0 wrote:
I don't think I can easily post enough material to duplicate the issue at
this time. The process in question involves quite a lot of code, including
a service on an IIS server, in C#, and SAs on remote Glassfish servers.
I've got quite a large application here...

I did realize that I should have added a few points to my original post. 1)
The EJB driving the process is in a different SA from the BPEL it is calling
(but on the same server). And 2) the amount of data is relatively large.
The BPEL is being called with an image of about one MB, that has been
marshaled into a string field (a string, for reasons having to do with the
other servers it goes on to from the BPEL). The large data block seems to
be key. I have not seen this in other places in the application.

3) Only restarting Glassfish seems to clear it up, reloading the SAs does
not help.

What I'm wondering about is perhaps some resource being consumed in the HTTP
BC maybe? I'm not sure how to go about collecting more information, or on
the impacts of increasing any of the default cache and pool settings for
HTTP or EJBs.

Murali Pottlapelli wrote:

I am not aware of this issue. Will you please an issue at
https://open-esb.dev.java.net/servlets/ProjectIssues and attach the
project to reproduce the issue?

Regards
Murali

jsexton0 wrote:

I have a system of BPELs and EJBs that perform a task. Most of the task
is
driven by an EJB, which invokes a few BPEL processes. One of the BPELs
takes about 2 seconds to run.

The problem is that if for any reason the client (I'm using SoapUI and
Netbeans unit tests) goes away, such as a timeout, then the next time the
process is run, the same process that took 2 seconds to run takes more
like
seven seconds.

Let the client time out again, and that becomes more like 2-3 minutes.
Do
this 3-5 times and the system becomes unusable, the delay becomes several
hours.

I have used logging to isolate the time being consumed to be just between
the Java/EJB's call to the BPEL's web service, and the start of the
initial
receive action in the BPEL.

How can I avoid this problem?
Thanks

« Return to Thread: Difficult Performance Problem