Profiling results for Continuum
|
View:
New views
10 Messages
—
Rating Filter:
Alert me
|
 |
Profiling results for Continuum
by Marc Lustig
::
Rate this Message:
Hi there,
as requested I repost this here. our teams have been suffering from the bad performance of the Continuum-UI for a long time . I found now the time to profile one of our servers running 3 Continuum-1.2.2-instances (webapps) on a single Tomcat6, JDK6, RHEL4 and Oracle-RAC as database-backend. The tomcat-JVM is consuming between 150 and 200 % of our dual-core Xeon-server. This is just the tomcat-JVM, not any maven-process running aside. And I am not talking about peaks, the load is as high like that for hours. This is a snapshot taken by Jprofiler at a time without any user-activity on one of the UI's. the 3 threads causing the highest load are: 37,3% - 54.415 ms - 5 inv. org.jpox.AbstractPersistenceManager.detachCopyAll 13,6% - 19.842 ms - 14 inv. org.codehaus.plexus.jdo.PlexusJdoUtils.getObjectById 34,5% - 50.362 ms - 4.480 inv. edu.emory.mathcs.backport.java.util.concurrent.BlockingQueue.poll (100% = total load caused by the JVM) The first 2 threads are caused by the JDO-impl. Is it possible that Continuum has an extremely "bad" fetching stragety, e. g. doing all kinds of unnecessary loading cascades? This is my first impression. About the 3rd thread, I have no idea why this BlockingQueue is causing such a tremendous load. What is the purpose of this thread? Can it be optimized. The slow performance is not only while adding new project groups. There is a high CPU-load almost constantly between 100 and 200 % (on dual core): 12795 tomcat 16 0 171 7238:01 40.9 12.0g 6.3g 10m S java thanks Marc |
|
|
Re: Profiling results for Continuum
by Emmanuel Venisse-2
::
Rate this Message:
It it possible to have a stack trace of this threads to know where to look
in the code? Emmanuel On Mon, Jun 22, 2009 at 9:28 AM, Marc Lustig <ml@...> wrote: > > Hi there, > as requested I repost this here. > > our teams have been suffering from the bad performance of the Continuum-UI > for a long time . I found now the time to profile one of our servers > running > 3 Continuum-1.2.2-instances (webapps) on a single Tomcat6, JDK6, RHEL4 and > Oracle-RAC as database-backend. > > The tomcat-JVM is consuming between 150 and 200 % of our dual-core > Xeon-server. > This is just the tomcat-JVM, not any maven-process running aside. > And I am not talking about peaks, the load is as high like that for hours. > > This is a snapshot taken by Jprofiler at a time without any user-activity > on > one of the UI's. > > the 3 threads causing the highest load are: > 37,3% - 54.415 ms - 5 inv. > org.jpox.AbstractPersistenceManager.detachCopyAll > 13,6% - 19.842 ms - 14 inv. > org.codehaus.plexus.jdo.PlexusJdoUtils.getObjectById > 34,5% - 50.362 ms - 4.480 inv. > edu.emory.mathcs.backport.java.util.concurrent.BlockingQueue.poll > (100% = total load caused by the JVM) > > The first 2 threads are caused by the JDO-impl. Is it possible that > Continuum has an extremely "bad" fetching stragety, e. g. doing all kinds > of > unnecessary loading cascades? This is my first impression. > > About the 3rd thread, I have no idea why this BlockingQueue is causing such > a tremendous load. > What is the purpose of this thread? Can it be optimized. > > The slow performance is not only while adding new project groups. There is > a > high CPU-load almost constantly between 100 and 200 % (on dual core): > > 12795 tomcat 16 0 171 7238:01 40.9 12.0g 6.3g 10m S java > > thanks > > Marc > -- > View this message in context: > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24142787.html > Sent from the Continuum - Dev mailing list archive at Nabble.com. > > |
|
|
Re: Profiling results for Continuum
by Marc Lustig
::
Rate this Message:
Yeah, sure. I have attached an html-file. I hope it will be accessible thru nabble.
Call_Tree_2.html
|
|
|
Re: Profiling results for Continuum
by Emmanuel Venisse-2
::
Rate this Message:
I don't see your previous results in this file.
The output you attached seems to be ok for our startup process because we have lot of beans to load with spring and lot of datas to load with JPOX. Emmanuel On Tue, Jun 23, 2009 at 2:16 PM, Marc Lustig <ml@...> wrote: > > Yeah, sure. I have attached an html-file. I hope it will be accessible thru > nabble. > http://www.nabble.com/file/p24164853/Call_Tree_2.html Call_Tree_2.html > > > > Emmanuel Venisse-2 wrote: > > > > It it possible to have a stack trace of this threads to know where to > look > > in the code? > > > > Emmanuel > > > > On Mon, Jun 22, 2009 at 9:28 AM, Marc Lustig <ml@...> wrote: > > > >> > >> Hi there, > >> as requested I repost this here. > >> > >> our teams have been suffering from the bad performance of the > >> Continuum-UI > >> for a long time . I found now the time to profile one of our servers > >> running > >> 3 Continuum-1.2.2-instances (webapps) on a single Tomcat6, JDK6, RHEL4 > >> and > >> Oracle-RAC as database-backend. > >> > >> The tomcat-JVM is consuming between 150 and 200 % of our dual-core > >> Xeon-server. > >> This is just the tomcat-JVM, not any maven-process running aside. > >> And I am not talking about peaks, the load is as high like that for > >> hours. > >> > >> This is a snapshot taken by Jprofiler at a time without any > user-activity > >> on > >> one of the UI's. > >> > >> the 3 threads causing the highest load are: > >> 37,3% - 54.415 ms - 5 inv. > >> org.jpox.AbstractPersistenceManager.detachCopyAll > >> 13,6% - 19.842 ms - 14 inv. > >> org.codehaus.plexus.jdo.PlexusJdoUtils.getObjectById > >> 34,5% - 50.362 ms - 4.480 inv. > >> edu.emory.mathcs.backport.java.util.concurrent.BlockingQueue.poll > >> (100% = total load caused by the JVM) > >> > >> The first 2 threads are caused by the JDO-impl. Is it possible that > >> Continuum has an extremely "bad" fetching stragety, e. g. doing all > kinds > >> of > >> unnecessary loading cascades? This is my first impression. > >> > >> About the 3rd thread, I have no idea why this BlockingQueue is causing > >> such > >> a tremendous load. > >> What is the purpose of this thread? Can it be optimized. > >> > >> The slow performance is not only while adding new project groups. There > >> is > >> a > >> high CPU-load almost constantly between 100 and 200 % (on dual core): > >> > >> 12795 tomcat 16 0 171 7238:01 40.9 12.0g 6.3g 10m S java > >> > >> thanks > >> > >> Marc > >> -- > >> View this message in context: > >> > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24142787.html > >> Sent from the Continuum - Dev mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24164853.html > Sent from the Continuum - Dev mailing list archive at Nabble.com. > > |
|
|
Re: Profiling results for Continuum
by Marc Lustig
::
Rate this Message:
Yes, you were right.
Now I have the results for a "normal" situation, about 1 hour after deployment. The tomcat-process consuming between 150 and 200 % of the machine. Call_Tree_3.html
|
|
|
Re: Profiling results for Continuum
by Emmanuel Venisse-2
::
Rate this Message:
Do you have lot of projectgroups/projects/buildresults?
I started to review our JPOX calls and to fix some of them to get less datas. Emmanuel On Tue, Jun 23, 2009 at 3:54 PM, Marc Lustig <ml@...> wrote: > > Yes, you were right. > Now I have the results for a "normal" situation, about 1 hour after > deployment. > The tomcat-process consuming between 150 and 200 % of the machine. > > http://www.nabble.com/file/p24166543/Call_Tree_3.html Call_Tree_3.html > > > Emmanuel Venisse-2 wrote: > > > > I don't see your previous results in this file. > > The output you attached seems to be ok for our startup process because we > > have lot of beans to load with spring and lot of datas to load with JPOX. > > > > Emmanuel > > > > On Tue, Jun 23, 2009 at 2:16 PM, Marc Lustig <ml@...> wrote: > > > >> > >> Yeah, sure. I have attached an html-file. I hope it will be accessible > >> thru > >> nabble. > >> http://www.nabble.com/file/p24164853/Call_Tree_2.html Call_Tree_2.html > >> > >> > >> > >> Emmanuel Venisse-2 wrote: > >> > > >> > It it possible to have a stack trace of this threads to know where to > >> look > >> > in the code? > >> > > >> > Emmanuel > >> > > >> > On Mon, Jun 22, 2009 at 9:28 AM, Marc Lustig <ml@...> > wrote: > >> > > >> >> > >> >> Hi there, > >> >> as requested I repost this here. > >> >> > >> >> our teams have been suffering from the bad performance of the > >> >> Continuum-UI > >> >> for a long time . I found now the time to profile one of our servers > >> >> running > >> >> 3 Continuum-1.2.2-instances (webapps) on a single Tomcat6, JDK6, > RHEL4 > >> >> and > >> >> Oracle-RAC as database-backend. > >> >> > >> >> The tomcat-JVM is consuming between 150 and 200 % of our dual-core > >> >> Xeon-server. > >> >> This is just the tomcat-JVM, not any maven-process running aside. > >> >> And I am not talking about peaks, the load is as high like that for > >> >> hours. > >> >> > >> >> This is a snapshot taken by Jprofiler at a time without any > >> user-activity > >> >> on > >> >> one of the UI's. > >> >> > >> >> the 3 threads causing the highest load are: > >> >> 37,3% - 54.415 ms - 5 inv. > >> >> org.jpox.AbstractPersistenceManager.detachCopyAll > >> >> 13,6% - 19.842 ms - 14 inv. > >> >> org.codehaus.plexus.jdo.PlexusJdoUtils.getObjectById > >> >> 34,5% - 50.362 ms - 4.480 inv. > >> >> edu.emory.mathcs.backport.java.util.concurrent.BlockingQueue.poll > >> >> (100% = total load caused by the JVM) > >> >> > >> >> The first 2 threads are caused by the JDO-impl. Is it possible that > >> >> Continuum has an extremely "bad" fetching stragety, e. g. doing all > >> kinds > >> >> of > >> >> unnecessary loading cascades? This is my first impression. > >> >> > >> >> About the 3rd thread, I have no idea why this BlockingQueue is > causing > >> >> such > >> >> a tremendous load. > >> >> What is the purpose of this thread? Can it be optimized. > >> >> > >> >> The slow performance is not only while adding new project groups. > >> There > >> >> is > >> >> a > >> >> high CPU-load almost constantly between 100 and 200 % (on dual core): > >> >> > >> >> 12795 tomcat 16 0 171 7238:01 40.9 12.0g 6.3g 10m S java > >> >> > >> >> thanks > >> >> > >> >> Marc > >> >> -- > >> >> View this message in context: > >> >> > >> > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24142787.html > >> >> Sent from the Continuum - Dev mailing list archive at Nabble.com. > >> >> > >> >> > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24164853.html > >> Sent from the Continuum - Dev mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24166543.html > Sent from the Continuum - Dev mailing list archive at Nabble.com. > > |
|
|
Re: Profiling results for Continuum
by Marc Lustig
::
Rate this Message:
Yes, we do. Here are the stats:
3 instances running in this tomcat-container. instance 1, 2 3 projects 2, 148, 96 buildresults 9881, 93875, 26076 total projects: ~ 250 total build-results: ~130.000 Let me know if you need any more stats.
|
|
|
Re: Profiling results for Continuum
by Emmanuel Venisse-2
::
Rate this Message:
I'll continue to review the code and when I'll think it is ok, I'll ping
you. Will you be able to test a Continuum snapshot on your big DB? Emmanuel On Fri, Jun 26, 2009 at 1:56 PM, Marc Lustig <ml@...> wrote: > > Yes, we do. Here are the stats: > > 3 instances running in this tomcat-container. > > instance 1, 2 3 > projects 2, 148, 96 > buildresults 9881, 93875, 26076 > > total projects: ~ 250 > total build-results: ~130.000 > > Let me know if you need any more stats. > > > Emmanuel Venisse-2 wrote: > > > > Do you have lot of projectgroups/projects/buildresults? > > > > I started to review our JPOX calls and to fix some of them to get less > > datas. > > > > Emmanuel > > > > On Tue, Jun 23, 2009 at 3:54 PM, Marc Lustig <ml@...> wrote: > > > >> > >> Yes, you were right. > >> Now I have the results for a "normal" situation, about 1 hour after > >> deployment. > >> The tomcat-process consuming between 150 and 200 % of the machine. > >> > >> http://www.nabble.com/file/p24166543/Call_Tree_3.html Call_Tree_3.html > >> > >> > >> Emmanuel Venisse-2 wrote: > >> > > >> > I don't see your previous results in this file. > >> > The output you attached seems to be ok for our startup process because > >> we > >> > have lot of beans to load with spring and lot of datas to load with > >> JPOX. > >> > > >> > Emmanuel > >> > > >> > On Tue, Jun 23, 2009 at 2:16 PM, Marc Lustig <ml@...> > wrote: > >> > > >> >> > >> >> Yeah, sure. I have attached an html-file. I hope it will be > accessible > >> >> thru > >> >> nabble. > >> >> http://www.nabble.com/file/p24164853/Call_Tree_2.htmlCall_Tree_2.html > >> >> > >> >> > >> >> > >> >> Emmanuel Venisse-2 wrote: > >> >> > > >> >> > It it possible to have a stack trace of this threads to know where > >> to > >> >> look > >> >> > in the code? > >> >> > > >> >> > Emmanuel > >> >> > > >> >> > On Mon, Jun 22, 2009 at 9:28 AM, Marc Lustig <ml@...> > >> wrote: > >> >> > > >> >> >> > >> >> >> Hi there, > >> >> >> as requested I repost this here. > >> >> >> > >> >> >> our teams have been suffering from the bad performance of the > >> >> >> Continuum-UI > >> >> >> for a long time . I found now the time to profile one of our > >> servers > >> >> >> running > >> >> >> 3 Continuum-1.2.2-instances (webapps) on a single Tomcat6, JDK6, > >> RHEL4 > >> >> >> and > >> >> >> Oracle-RAC as database-backend. > >> >> >> > >> >> >> The tomcat-JVM is consuming between 150 and 200 % of our dual-core > >> >> >> Xeon-server. > >> >> >> This is just the tomcat-JVM, not any maven-process running aside. > >> >> >> And I am not talking about peaks, the load is as high like that > for > >> >> >> hours. > >> >> >> > >> >> >> This is a snapshot taken by Jprofiler at a time without any > >> >> user-activity > >> >> >> on > >> >> >> one of the UI's. > >> >> >> > >> >> >> the 3 threads causing the highest load are: > >> >> >> 37,3% - 54.415 ms - 5 inv. > >> >> >> org.jpox.AbstractPersistenceManager.detachCopyAll > >> >> >> 13,6% - 19.842 ms - 14 inv. > >> >> >> org.codehaus.plexus.jdo.PlexusJdoUtils.getObjectById > >> >> >> 34,5% - 50.362 ms - 4.480 inv. > >> >> >> edu.emory.mathcs.backport.java.util.concurrent.BlockingQueue.poll > >> >> >> (100% = total load caused by the JVM) > >> >> >> > >> >> >> The first 2 threads are caused by the JDO-impl. Is it possible > >> that > >> >> >> Continuum has an extremely "bad" fetching stragety, e. g. doing > all > >> >> kinds > >> >> >> of > >> >> >> unnecessary loading cascades? This is my first impression. > >> >> >> > >> >> >> About the 3rd thread, I have no idea why this BlockingQueue is > >> causing > >> >> >> such > >> >> >> a tremendous load. > >> >> >> What is the purpose of this thread? Can it be optimized. > >> >> >> > >> >> >> The slow performance is not only while adding new project groups. > >> >> There > >> >> >> is > >> >> >> a > >> >> >> high CPU-load almost constantly between 100 and 200 % (on dual > >> core): > >> >> >> > >> >> >> 12795 tomcat 16 0 171 7238:01 40.9 12.0g 6.3g 10m S java > >> >> >> > >> >> >> thanks > >> >> >> > >> >> >> Marc > >> >> >> -- > >> >> >> View this message in context: > >> >> >> > >> >> > >> > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24142787.html > >> >> >> Sent from the Continuum - Dev mailing list archive at Nabble.com. > >> >> >> > >> >> >> > >> >> > > >> >> > > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24164853.html > >> >> Sent from the Continuum - Dev mailing list archive at Nabble.com. > >> >> > >> >> > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24166543.html > >> Sent from the Continuum - Dev mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24218977.html > Sent from the Continuum - Dev mailing list archive at Nabble.com. > > |
|
|
Re: Profiling results for Continuum
by Marc Lustig
::
Rate this Message:
yes, no problem. Let me know where to download the snapshot and I will test it. |
|
|
Re: Profiling results for Continuum
by Emmanuel Venisse-2
::
Rate this Message:
Marc,
The patch is done in continuum-1.3.x branch. Can you build it and run it? If you can't build it, I'll can provide to you a snapshot. Emmanuel On Fri, Jun 26, 2009 at 2:20 PM, Marc Lustig <ml@...> wrote: > > > > Emmanuel Venisse-2 wrote: > > > > I'll continue to review the code and when I'll think it is ok, I'll ping > > you. > > Will you be able to test a Continuum snapshot on your big DB? > > > > yes, no problem. Let me know where to download the snapshot and I will test > it. > -- > View this message in context: > http://www.nabble.com/Profiling-results-for-Continuum-tp24142787p24219393.html > Sent from the Continuum - Dev mailing list archive at Nabble.com. > > |
| Free embeddable forum powered by Nabble | Forum Help |
