Magnolia under the stress

[Crank]

Hello All,

I've been playing with Magnolia for a while trying to asses its stability. Eventually I put it under the stress on previous week - 800 concurrent users on a single box.

Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon 2 Cores, 4Gb RAM
JDK 1.6.0_16 64-bit version
Magnolia 4.1 Community Edition (STK Installed, Debug log is disabled) + MySQL 5.1.39
Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
Tomcat startup params:
JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
Client - J-Meter, 800 Concurrent threads, 5 static links + 1 search link, Gaussian timer set for 1000ms+-1500ms
Response time 954ms, Errors 17%
Avg. CPU utilization ~85%, RAM Utilization ~2Gb

Now the strange behavior I noticed is following: Magnolia is responding to incoming requests normally for about 2-5 minutes (more or less) and then just freezes. CPU usage = 0%, RAM is the same (GC is obviously not running). No responses at all.
After removing the stress from Magnolia it starts responding in 30-40 seconds.
500 Concurrent users will produce the same effect with the only exception it might take 50-100 minutes (maybe more) to reproduce.
Tried to look at MySQL connection stack using remote admin - 8 connections all of them are idle. Seems like all the content was cached by JCR previously.

Question: What exactly might cause this strange freezing? It looks like tomcat has enough maximum threads, connection pool to MySQL is not that busy... Could it be some internal cache of jackrabbit? Not sure but it looks like JVM is not the rootcause in this case.

Would appreciate your thoughts on this point.
Thank you in advance!

Regards,
Denis

[caste]

Ciao Marco sul forum di Magnlia ho letto questo post che sembra interessante:

Noi conosciamo questi i valori di questi parametri di Tomcat nella configurazione attuale?

Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
Tomcat startup params:
JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis

Andrea

2009/10/5 Denis Demichev <demichev@...>

Hello All,

I've been playing with Magnolia for a while trying to asses its stability. Eventually I put it under the stress on previous week - 800 concurrent users on a single box.

Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon 2 Cores, 4Gb RAM
JDK 1.6.0_16 64-bit version
Magnolia 4.1 Community Edition (STK Installed, Debug log is disabled) + MySQL 5.1.39
Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
Tomcat startup params:
JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
Client - J-Meter, 800 Concurrent threads, 5 static links + 1 search link, Gaussian timer set for 1000ms+-1500ms
Response time 954ms, Errors 17%
Avg. CPU utilization ~85%, RAM Utilization ~2Gb

Now the strange behavior I noticed is following: Magnolia is responding to incoming requests normally for about 2-5 minutes (more or less) and then just freezes. CPU usage = 0%, RAM is the same (GC is obviously not running). No responses at all.
After removing the stress from Magnolia it starts responding in 30-40 seconds.
500 Concurrent users will produce the same effect with the only exception it might take 50-100 minutes (maybe more) to reproduce.
Tried to look at MySQL connection stack using remote admin - 8 connections all of them are idle. Seems like all the content was cached by JCR previously.

Question: What exactly might cause this strange freezing? It looks like tomcat has enough maximum threads, connection pool to MySQL is not that busy... Could it be some internal cache of jackrabbit? Not sure but it looks like JVM is not the rootcause in this case.

Would appreciate your thoughts on this point.
Thank you in advance!

Regards,
Denis

[Jan Haderka-2]

Interesting tests. Would like to get more details (like detailed timings
for those concurrent users or at least summary stats) once you are done
with it if you don't mind sharing.

Now to explain your results:
- the DB connections are normal. JR maintains one connection per
workspace no matter what the load, so the amount of connections will not
change. What might change is how busy those connections are (i.e. how
many requests they are executing against the DB)
- the connections being idle and CPU usage being 0% suggests waiting for
some timeout ... it might be a deadlock, but if that is somewhere in
Magnolia itself I would expect it to be rather busy re-trying to either
save or read the content rather then simply sleeping. I might be able to
tell more if I knew the details of the tests - was it on public or on
the author box? Was it just reading the content? Was the content cached
by Magnolia? Were you hitting just single piece of content or
distributing the requests all over the instance (i.e. randomly
retrieving/editing different pieces of content).
- Just because you have 800 threads, doesn't mean you are able to serve
800 concurrent users. If the processing of the request takes more time
then the average time between requests ... Might be that you simply
filled the queue to the limit and Tomcat stopped queuing further
requests until the queue was emptied. If you have Tomcat management
console installed it would help to see the info from there as well. Also
you can try to have some minimal webapp deployed next to Magnolia and
see if that one will be still responding even when Magnolia is not ...
that should rule out the tomcat involvement.

JR uses internal in-memory cache for the items, but it is somewhat
limited to about 30MB of content. It would help to know what kind of
content were you retrieving in your test - binary or text?

Another thing related to threads ... for some ops Magnolia spawn
additional threads and so does the JackRabbit. Not sure if those counts
into the 800 threads limit or not. If so and if you were updating the
content as part of your test, it might be that JR has simply stopped at
some point as it was not able to create indexer threads for re-indexing
the updated content.

It would also help to see what happened just after the silence period
was over ... what was written in FS, what was the first few DB requests
sent  ...

Also during the test, do you see memory consumption growing or do you
see response times getting longer before the silence?
Have you (or can you) try with different GC collectors (e.g. the default
one)?

Cheers & good luck with more testing.
Jan



On Mon, 2009-10-05 at 09:15 -0400, Denis Demichev wrote:

> Hello All,
>
> I've been playing with Magnolia for a while trying to asses its
> stability. Eventually I put it under the stress on previous week - 800
> concurrent users on a single box.
>
> Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon 2 Cores, 4Gb
> RAM
> JDK 1.6.0_16 64-bit version
> Magnolia 4.1 Community Edition (STK Installed, Debug log is disabled)
> + MySQL 5.1.39
> Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
> Tomcat startup params:
> JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:
> +UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
> Client - J-Meter, 800 Concurrent threads, 5 static links + 1 search
> link, Gaussian timer set for 1000ms+-1500ms
> Response time 954ms, Errors 17%
> Avg. CPU utilization ~85%, RAM Utilization ~2Gb
>
> Now the strange behavior I noticed is following: Magnolia is
> responding to incoming requests normally for about 2-5 minutes (more
> or less) and then just freezes. CPU usage = 0%, RAM is the same (GC is
> obviously not running). No responses at all.
> After removing the stress from Magnolia it starts responding in 30-40
> seconds.
> 500 Concurrent users will produce the same effect with the only
> exception it might take 50-100 minutes (maybe more) to reproduce.
> Tried to look at MySQL connection stack using remote admin - 8
> connections all of them are idle. Seems like all the content was
> cached by JCR previously.
>
> Question: What exactly might cause this strange freezing? It looks
> like tomcat has enough maximum threads, connection pool to MySQL is
> not that busy... Could it be some internal cache of jackrabbit? Not
> sure but it looks like JVM is not the rootcause in this case.
>
> Would appreciate your thoughts on this point.
> Thank you in advance!
>
> Regards,
> Denis

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Crank]

Hello

I'm not a big expert in Italian - sorry. Is that a question for me?
If I interpreted it right, that means do I know what exactly these tomcat parameters mean?

If so, I can say that these parameters stand for how much threads will be created by tomcat to handle incoming user requests. As soon as I'm planning 800-1000 concurrent users I thought it makes sense to increase default thread number.


Regards,
Denis

On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli <andrea.castelli.1@...> wrote:

Ciao Marco sul forum di Magnlia ho letto questo post che sembra interessante:

Noi conosciamo questi i valori di questi parametri di Tomcat nella configurazione attuale?

Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
Tomcat startup params:
JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis

Andrea

2009/10/5 Denis Demichev <demichev@...>

Hello All,

I've been playing with Magnolia for a while trying to asses its stability. Eventually I put it under the stress on previous week - 800 concurrent users on a single box.

Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon 2 Cores, 4Gb RAM
JDK 1.6.0_16 64-bit version
Magnolia 4.1 Community Edition (STK Installed, Debug log is disabled) + MySQL 5.1.39
Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
Tomcat startup params:
JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
Client - J-Meter, 800 Concurrent threads, 5 static links + 1 search link, Gaussian timer set for 1000ms+-1500ms
Response time 954ms, Errors 17%
Avg. CPU utilization ~85%, RAM Utilization ~2Gb

Now the strange behavior I noticed is following: Magnolia is responding to incoming requests normally for about 2-5 minutes (more or less) and then just freezes. CPU usage = 0%, RAM is the same (GC is obviously not running). No responses at all.
After removing the stress from Magnolia it starts responding in 30-40 seconds.
500 Concurrent users will produce the same effect with the only exception it might take 50-100 minutes (maybe more) to reproduce.
Tried to look at MySQL connection stack using remote admin - 8 connections all of them are idle. Seems like all the content was cached by JCR previously.

Question: What exactly might cause this strange freezing? It looks like tomcat has enough maximum threads, connection pool to MySQL is not that busy... Could it be some internal cache of jackrabbit? Not sure but it looks like JVM is not the rootcause in this case.

Would appreciate your thoughts on this point.
Thank you in advance!

Regards,
Denis

[Jan Haderka-2]

I believe Andrea meant to forward your mail to someone else (Marco) and
just hit the "send" button too early :D


On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev wrote:

> Hello
>
> I'm not a big expert in Italian - sorry. Is that a question for me?
> If I interpreted it right, that means do I know what exactly these
> tomcat parameters mean?
>
> If so, I can say that these parameters stand for how much threads will
> be created by tomcat to handle incoming user requests. As soon as I'm
> planning 800-1000 concurrent users I thought it makes sense to
> increase default thread number.
>
>
> Regards,
> Denis
>
>
> On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
> <andrea.castelli.1@...> wrote:
>         Ciao Marco sul forum di Magnlia ho letto questo post che
>         sembra interessante:
>        
>         Noi conosciamo questi i valori di questi parametri di Tomcat
>         nella configurazione attuale?
>        
>        
>         Tomcat: maxThreads="800" minSpareThreads="25"
>         maxSpareThreads="75"
>         Tomcat startup params:
>         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:
>         +UseConcMarkSweepGC -XX:+CMSIncrementalMode
>         -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
>        
>        
>         Andrea
>        
>         2009/10/5 Denis Demichev <demichev@...>
>        
>        
>                 Hello All,
>                
>                 I've been playing with Magnolia for a while trying to
>                 asses its stability. Eventually I put it under the
>                 stress on previous week - 800 concurrent users on a
>                 single box.
>                
>                 Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon
>                 2 Cores, 4Gb RAM
>                 JDK 1.6.0_16 64-bit version
>                 Magnolia 4.1 Community Edition (STK Installed, Debug
>                 log is disabled) + MySQL 5.1.39
>                 Tomcat: maxThreads="800" minSpareThreads="25"
>                 maxSpareThreads="75"
>                 Tomcat startup params:
>                 JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>                 -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:
>                 +CMSIncrementalMode -Djava.net.preferIPv4Stack=true
>                 -XX:+DoEscapeAnalysis
>                 Client - J-Meter, 800 Concurrent threads, 5 static
>                 links + 1 search link, Gaussian timer set for 1000ms
>                 +-1500ms
>                 Response time 954ms, Errors 17%
>                 Avg. CPU utilization ~85%, RAM Utilization ~2Gb
>                
>                 Now the strange behavior I noticed is following:
>                 Magnolia is responding to incoming requests normally
>                 for about 2-5 minutes (more or less) and then just
>                 freezes. CPU usage = 0%, RAM is the same (GC is
>                 obviously not running). No responses at all.
>                 After removing the stress from Magnolia it starts
>                 responding in 30-40 seconds.
>                 500 Concurrent users will produce the same effect with
>                 the only exception it might take 50-100 minutes (maybe
>                 more) to reproduce.
>                 Tried to look at MySQL connection stack using remote
>                 admin - 8 connections all of them are idle. Seems like
>                 all the content was cached by JCR previously.
>                
>                 Question: What exactly might cause this strange
>                 freezing? It looks like tomcat has enough maximum
>                 threads, connection pool to MySQL is not that busy...
>                 Could it be some internal cache of jackrabbit? Not
>                 sure but it looks like JVM is not the rootcause in
>                 this case.
>                
>                 Would appreciate your thoughts on this point.
>                 Thank you in advance!
>                
>                 Regards,
>                 Denis
>        
>

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[caste]

Yes, I made a mistake.
I'm reading the debate because I'm very interested.
I noticed low performance when I use the command "activate this page", and the page has other subpages. My question is similar: what does JR do in this operation? Maybe does it made lock on the subpages so performance goes down?

Thank you, sorry for my mistake.
Best regards.
Andrea


 

2009/10/5 Jan Haderka <jan.haderka@...>

I believe Andrea meant to forward your mail to someone else (Marco) and
just hit the "send" button too early :D

On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev wrote:
> Hello
>
> I'm not a big expert in Italian - sorry. Is that a question for me?
> If I interpreted it right, that means do I know what exactly these
> tomcat parameters mean?
>
> If so, I can say that these parameters stand for how much threads will
> be created by tomcat to handle incoming user requests. As soon as I'm
> planning 800-1000 concurrent users I thought it makes sense to
> increase default thread number.
>
>
> Regards,
> Denis
>
>
> On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
> <andrea.castelli.1@...> wrote:
>         Ciao Marco sul forum di Magnlia ho letto questo post che
>         sembra interessante:
>
>         Noi conosciamo questi i valori di questi parametri di Tomcat
>         nella configurazione attuale?
>
>
>         Tomcat: maxThreads="800" minSpareThreads="25"
>         maxSpareThreads="75"
>         Tomcat startup params:
>         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:
>         +UseConcMarkSweepGC -XX:+CMSIncrementalMode
>         -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
>
>
>         Andrea
>
>         2009/10/5 Denis Demichev <demichev@...>
>
>
>                 Hello All,
>
>                 I've been playing with Magnolia for a while trying to
>                 asses its stability. Eventually I put it under the
>                 stress on previous week - 800 concurrent users on a
>                 single box.
>
>                 Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon
>                 2 Cores, 4Gb RAM
>                 JDK 1.6.0_16 64-bit version
>                 Magnolia 4.1 Community Edition (STK Installed, Debug
>                 log is disabled) + MySQL 5.1.39
>                 Tomcat: maxThreads="800" minSpareThreads="25"
>                 maxSpareThreads="75"
>                 Tomcat startup params:
>                 JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>                 -XX:MaxPermSize=1024M -XX:+UseConcMarkSweepGC -XX:
>                 +CMSIncrementalMode -Djava.net.preferIPv4Stack=true
>                 -XX:+DoEscapeAnalysis
>                 Client - J-Meter, 800 Concurrent threads, 5 static
>                 links + 1 search link, Gaussian timer set for 1000ms
>                 +-1500ms
>                 Response time 954ms, Errors 17%
>                 Avg. CPU utilization ~85%, RAM Utilization ~2Gb
>
>                 Now the strange behavior I noticed is following:
>                 Magnolia is responding to incoming requests normally
>                 for about 2-5 minutes (more or less) and then just
>                 freezes. CPU usage = 0%, RAM is the same (GC is
>                 obviously not running). No responses at all.
>                 After removing the stress from Magnolia it starts
>                 responding in 30-40 seconds.
>                 500 Concurrent users will produce the same effect with
>                 the only exception it might take 50-100 minutes (maybe
>                 more) to reproduce.
>                 Tried to look at MySQL connection stack using remote
>                 admin - 8 connections all of them are idle. Seems like
>                 all the content was cached by JCR previously.
>
>                 Question: What exactly might cause this strange
>                 freezing? It looks like tomcat has enough maximum
>                 threads, connection pool to MySQL is not that busy...
>                 Could it be some internal cache of jackrabbit? Not
>                 sure but it looks like JVM is not the rootcause in
>                 this case.
>
>                 Would appreciate your thoughts on this point.
>                 Thank you in advance!
>
>                 Regards,
>                 Denis
>
>

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Crank]

Hello Jan,

I really appreciate your prompt reply. let me answer some of the questions below.
- Deadlock: I was thinking about this (race condition?) as well. Not sure how exactly I would verify if that theory is correct
Stress test was performed on a public instance against STK project pages in a loop:
/magnoliaPublic/demo-project.html
/magnoliaPublic/demo-project/about.html
/magnoliaPublic/demo-project/news-and-events.html
/magnoliaPublic/demo-project/multimedia.html
/magnoliaPublic/demo-project/service.html
/magnoliaPublic/demo-project/service/search-result.html?queryStr=preceded
All of them have Gaussian Timeout set to 1000ms offset and 1500ms deviation.
90% responses are in 954ms, that's why I chose 800 threads to handle requests.
- Was content cached? I assume yes, because SQL Admin showed no activity during the test - all MySQL connections were idle for a while at the moment.
- I had another application deployed and its not responding during that "stall" period.
But if I deploy that application to tomcat separately (or I can use tomcat's JSP samples) the Tomcat instance will handle up to 1000 threads with 80% of CPU load.
- I have other applications deployed on the same box - they won't be accessible as well - not just magnolia.
- I tried with regular GC and other versions (_13, 32-bit) of JDK with no success.

Thank you.




Regards,
Denis

On Mon, Oct 5, 2009 at 9:37 AM, Jan Haderka <jan.haderka@...> wrote:

Interesting tests. Would like to get more details (like detailed timings
for those concurrent users or at least summary stats) once you are done
with it if you don't mind sharing.

Now to explain your results:
- the DB connections are normal. JR maintains one connection per
workspace no matter what the load, so the amount of connections will not
change. What might change is how busy those connections are (i.e. how
many requests they are executing against the DB)
- the connections being idle and CPU usage being 0% suggests waiting for
some timeout ... it might be a deadlock, but if that is somewhere in
Magnolia itself I would expect it to be rather busy re-trying to either
save or read the content rather then simply sleeping. I might be able to
tell more if I knew the details of the tests - was it on public or on
the author box? Was it just reading the content? Was the content cached
by Magnolia? Were you hitting just single piece of content or
distributing the requests all over the instance (i.e. randomly
retrieving/editing different pieces of content).
- Just because you have 800 threads, doesn't mean you are able to serve
800 concurrent users. If the processing of the request takes more time
then the average time between requests ... Might be that you simply
filled the queue to the limit and Tomcat stopped queuing further
requests until the queue was emptied. If you have Tomcat management
console installed it would help to see the info from there as well. Also
you can try to have some minimal webapp deployed next to Magnolia and
see if that one will be still responding even when Magnolia is not ...
that should rule out the tomcat involvement.

JR uses internal in-memory cache for the items, but it is somewhat
limited to about 30MB of content. It would help to know what kind of
content were you retrieving in your test - binary or text?

Another thing related to threads ... for some ops Magnolia spawn
additional threads and so does the JackRabbit. Not sure if those counts
into the 800 threads limit or not. If so and if you were updating the
content as part of your test, it might be that JR has simply stopped at
some point as it was not able to create indexer threads for re-indexing
the updated content.

It would also help to see what happened just after the silence period
was over ... what was written in FS, what was the first few DB requests
sent  ...

Also during the test, do you see memory consumption growing or do you
see response times getting longer before the silence?
Have you (or can you) try with different GC collectors (e.g. the default
one)?

Cheers & good luck with more testing.
Jan

On Mon, 2009-10-05 at 09:15 -0400, Denis Demichev wrote:
> Hello All,
>
> I've been playing with Magnolia for a while trying to asses its
> stability. Eventually I put it under the stress on previous week - 800
> concurrent users on a single box.
>
> Server: Windows Server 2007(?) SP1 64 bit,  Intel Xeon 2 Cores, 4Gb
> RAM
> JDK 1.6.0_16 64-bit version
> Magnolia 4.1 Community Edition (STK Installed, Debug log is disabled)
> + MySQL 5.1.39
> Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75"
> Tomcat startup params:
> JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX:
> +UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis
> Client - J-Meter, 800 Concurrent threads, 5 static links + 1 search
> link, Gaussian timer set for 1000ms+-1500ms
> Response time 954ms, Errors 17%
> Avg. CPU utilization ~85%, RAM Utilization ~2Gb
>
> Now the strange behavior I noticed is following: Magnolia is
> responding to incoming requests normally for about 2-5 minutes (more
> or less) and then just freezes. CPU usage = 0%, RAM is the same (GC is
> obviously not running). No responses at all.
> After removing the stress from Magnolia it starts responding in 30-40
> seconds.
> 500 Concurrent users will produce the same effect with the only
> exception it might take 50-100 minutes (maybe more) to reproduce.
> Tried to look at MySQL connection stack using remote admin - 8
> connections all of them are idle. Seems like all the content was
> cached by JCR previously.
>
> Question: What exactly might cause this strange freezing? It looks
> like tomcat has enough maximum threads, connection pool to MySQL is
> not that busy... Could it be some internal cache of jackrabbit? Not
> sure but it looks like JVM is not the rootcause in this case.
>
> Would appreciate your thoughts on this point.
> Thank you in advance!
>
> Regards,
> Denis

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Jan Haderka-2]

> Yes, I made a mistake.
> I'm reading the debate because I'm very interested.
> I noticed low performance when I use the command "activate this page",
> and the page has other subpages. My question is similar: what does JR
> do in this operation? Maybe does it made lock on the subpages so
> performance goes down?

If you have enabled the versioning, what happens is that the page (and
all the sub-pages, if selected) has to be versioned at that point. The
activation then happens on the versioned content, to make sure that the
what gets activated is what you actually selected and not any possible
modifications made by others later. The versioning (among other things)
means that value of each node has to be copied so it involves intensive
reading and writing, hence the impact on the performance.
Whether you are using workflow or now, the copying of the node from the
author instance to the public instance happens later. Again, this means
that value for each property in the activated content has to be read
from the repository, the content is converted in the xml and sent over
to the public instance (together with all other relevant info, such as
ordering of that content, etc.). The public instance has to process such
incoming content and import it in the repository. In case of
transactional information, it needs also backup the previous version of
the content in case there is a rollback request issued later during the
transaction. Again it involves both reading and writing from/to repo for
each property of the activated content.
Each of those things are expensive and have negative impact on
performance.

HTH,
Jan

>
> Thank you, sorry for my mistake.
> Best regards.
> Andrea
>
>
>  
>
>
>
> 2009/10/5 Jan Haderka <jan.haderka@...>
>        
>         I believe Andrea meant to forward your mail to someone else
>         (Marco) and
>         just hit the "send" button too early :D
>        
>        
>        
>         On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev wrote:
>         > Hello
>         >
>         > I'm not a big expert in Italian - sorry. Is that a question
>         for me?
>         > If I interpreted it right, that means do I know what exactly
>         these
>         > tomcat parameters mean?
>         >
>         > If so, I can say that these parameters stand for how much
>         threads will
>         > be created by tomcat to handle incoming user requests. As
>         soon as I'm
>         > planning 800-1000 concurrent users I thought it makes sense
>         to
>         > increase default thread number.
>         >
>         >
>         > Regards,
>         > Denis
>         >
>         >
>         > On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
>         > <andrea.castelli.1@...> wrote:
>         >         Ciao Marco sul forum di Magnlia ho letto questo post
>         che
>         >         sembra interessante:
>         >
>         >         Noi conosciamo questi i valori di questi parametri
>         di Tomcat
>         >         nella configurazione attuale?
>         >
>         >
>         >         Tomcat: maxThreads="800" minSpareThreads="25"
>         >         maxSpareThreads="75"
>         >         Tomcat startup params:
>         >         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         -XX:MaxPermSize=1024M -XX:
>         >         +UseConcMarkSweepGC -XX:+CMSIncrementalMode
>         >         -Djava.net.preferIPv4Stack=true -XX:
>         +DoEscapeAnalysis
>         >
>         >
>         >         Andrea
>         >
>         >         2009/10/5 Denis Demichev <demichev@...>
>         >
>         >
>         >                 Hello All,
>         >
>         >                 I've been playing with Magnolia for a while
>         trying to
>         >                 asses its stability. Eventually I put it
>         under the
>         >                 stress on previous week - 800 concurrent
>         users on a
>         >                 single box.
>         >
>         >                 Server: Windows Server 2007(?) SP1 64 bit,
>          Intel Xeon
>         >                 2 Cores, 4Gb RAM
>         >                 JDK 1.6.0_16 64-bit version
>         >                 Magnolia 4.1 Community Edition (STK
>         Installed, Debug
>         >                 log is disabled) + MySQL 5.1.39
>         >                 Tomcat: maxThreads="800"
>         minSpareThreads="25"
>         >                 maxSpareThreads="75"
>         >                 Tomcat startup params:
>         >                 JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         >                 -XX:MaxPermSize=1024M -XX:
>         +UseConcMarkSweepGC -XX:
>         >                 +CMSIncrementalMode
>         -Djava.net.preferIPv4Stack=true
>         >                 -XX:+DoEscapeAnalysis
>         >                 Client - J-Meter, 800 Concurrent threads, 5
>         static
>         >                 links + 1 search link, Gaussian timer set
>         for 1000ms
>         >                 +-1500ms
>         >                 Response time 954ms, Errors 17%
>         >                 Avg. CPU utilization ~85%, RAM Utilization
>         ~2Gb
>         >
>         >                 Now the strange behavior I noticed is
>         following:
>         >                 Magnolia is responding to incoming requests
>         normally
>         >                 for about 2-5 minutes (more or less) and
>         then just
>         >                 freezes. CPU usage = 0%, RAM is the same (GC
>         is
>         >                 obviously not running). No responses at all.
>         >                 After removing the stress from Magnolia it
>         starts
>         >                 responding in 30-40 seconds.
>         >                 500 Concurrent users will produce the same
>         effect with
>         >                 the only exception it might take 50-100
>         minutes (maybe
>         >                 more) to reproduce.
>         >                 Tried to look at MySQL connection stack
>         using remote
>         >                 admin - 8 connections all of them are idle.
>         Seems like
>         >                 all the content was cached by JCR
>         previously.
>         >
>         >                 Question: What exactly might cause this
>         strange
>         >                 freezing? It looks like tomcat has enough
>         maximum
>         >                 threads, connection pool to MySQL is not
>         that busy...
>         >                 Could it be some internal cache of
>         jackrabbit? Not
>         >                 sure but it looks like JVM is not the
>         rootcause in
>         >                 this case.
>         >
>         >                 Would appreciate your thoughts on this
>         point.
>         >                 Thank you in advance!
>         >
>         >                 Regards,
>         >                 Denis
>         >
>         >
>        
>        
>        
>        
>         ----------------------------------------------------------------
>         For list details see
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         To unsubscribe, E-mail to:
>         <user-list-unsubscribe@...>
>         ----------------------------------------------------------------
>        
>        
>

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Jan Haderka-2]

Cool. Thanks for info.

On Mon, 2009-10-05 at 10:15 -0400, Denis Demichev wrote:
> Hello Jan,
>
> I really appreciate your prompt reply. let me answer some of the
> questions below.
> - Deadlock: I was thinking about this (race condition?) as well. Not
> sure how exactly I would verify if that theory is correct

If it was a deadlock it would occur rather randomly and with not
necessary correlation to the amount of threads, apart from growing
likelihood for it to occur, but not changing the timings. So if you can
run 10 times with 800 threads and each time it blocks in approx same
time, it is flooding of the queue rather then classic deadlock which
would be more random.

> Stress test was performed on a public instance against STK project
> pages in a loop:
> /magnoliaPublic/demo-project.html
> /magnoliaPublic/demo-project/about.html
> /magnoliaPublic/demo-project/news-and-events.html
> /magnoliaPublic/demo-project/multimedia.html
> /magnoliaPublic/demo-project/service.html

all of the above will be cached and being relatively small, they will be
served from the in-memory cache. If you run the test with those only,
only very tiny bit of Magnolia will be still busy (all filters up to
cache filter) but there will be minimal access to the repository.

> /magnoliaPublic/demo-project/service/search-result.html?queryStr=preceded

This one will be never cached so that is the only page that will hit the
repo each time, even with cache enabled (default settings).

> All of them have Gaussian Timeout set to 1000ms offset and 1500ms
> deviation.
> 90% responses are in 954ms, that's why I chose 800 threads to handle
> requests.

Hmm, it might be still fine ... or not depending on the remaining
10% ... I'd say this setting is pretty much on the edge. Can you try
what happens with 1000 tomcat threads without changing any other
parameters of the test?

> - Was content cached? I assume yes, because SQL Admin showed no
> activity during the test - all MySQL connections were idle for a while
> at the moment.
answered above.

> - I had another application deployed and its not responding during
> that "stall" period.

That suggests Tomcat queue being flooded rather then anything else. Are
you in position to repeat the same on JBoss and/or Glassfish?

> But if I deploy that application to tomcat separately (or I can use
> tomcat's JSP samples) the Tomcat instance will handle up to 1000
> threads with 80% of CPU load.

This really depends on underlying load ... it involves some processing
to have content stored and served from JCR repository so that doesn't
really mean anything.

> - I tried with regular GC and other versions (_13, 32-bit) of JDK with
> no success.

Good to know. Thanks.

Jan


----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Crank]

Hello Jan,


Hmm, it might be still fine ... or not depending on the remaining
10% ... I'd say this setting is pretty much on the edge. Can you try
what happens with 1000 tomcat threads without changing any other
parameters of the test?
---------
Even if I put 1600 tomcat threads it doesn't make any difference.
Actually I'm able to hang tomcat with 500 concurrent threads having 800 threads set in tomcat.
Seems like 450 concurrent users is a sort of magic figure: magnolia is able to handle less than 450, but hangs when its 500.
Will try to download JBoss and deploy Magnolia there.

Regards,
Denis

On Mon, Oct 5, 2009 at 10:42 AM, Jan Haderka <jan.haderka@...> wrote:

Cool. Thanks for info.

On Mon, 2009-10-05 at 10:15 -0400, Denis Demichev wrote:
> Hello Jan,
>
> I really appreciate your prompt reply. let me answer some of the
> questions below.
> - Deadlock: I was thinking about this (race condition?) as well. Not
> sure how exactly I would verify if that theory is correct

If it was a deadlock it would occur rather randomly and with not
necessary correlation to the amount of threads, apart from growing
likelihood for it to occur, but not changing the timings. So if you can
run 10 times with 800 threads and each time it blocks in approx same
time, it is flooding of the queue rather then classic deadlock which
would be more random.

> Stress test was performed on a public instance against STK project
> pages in a loop:
> /magnoliaPublic/demo-project.html
> /magnoliaPublic/demo-project/about.html
> /magnoliaPublic/demo-project/news-and-events.html
> /magnoliaPublic/demo-project/multimedia.html
> /magnoliaPublic/demo-project/service.html

all of the above will be cached and being relatively small, they will be
served from the in-memory cache. If you run the test with those only,
only very tiny bit of Magnolia will be still busy (all filters up to
cache filter) but there will be minimal access to the repository.

> /magnoliaPublic/demo-project/service/search-result.html?queryStr=preceded

This one will be never cached so that is the only page that will hit the
repo each time, even with cache enabled (default settings).

> All of them have Gaussian Timeout set to 1000ms offset and 1500ms
> deviation.
> 90% responses are in 954ms, that's why I chose 800 threads to handle
> requests.

Hmm, it might be still fine ... or not depending on the remaining
10% ... I'd say this setting is pretty much on the edge. Can you try
what happens with 1000 tomcat threads without changing any other
parameters of the test?

> - Was content cached? I assume yes, because SQL Admin showed no
> activity during the test - all MySQL connections were idle for a while
> at the moment.

answered above.

> - I had another application deployed and its not responding during
> that "stall" period.

That suggests Tomcat queue being flooded rather then anything else. Are
you in position to repeat the same on JBoss and/or Glassfish?

> But if I deploy that application to tomcat separately (or I can use
> tomcat's JSP samples) the Tomcat instance will handle up to 1000
> threads with 80% of CPU load.

This really depends on underlying load ... it involves some processing
to have content stored and served from JCR repository so that doesn't
really mean anything.

> - I tried with regular GC and other versions (_13, 32-bit) of JDK with
> no success.

Good to know. Thanks.

Jan

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Jan Haderka-2]

> ---------
> Even if I put 1600 tomcat threads it doesn't make any difference.
> Actually I'm able to hang tomcat with 500 concurrent threads having
> 800 threads set in tomcat.
> Seems like 450 concurrent users is a sort of magic figure: magnolia is
> able to handle less than 450, but hangs when its 500.
> Will try to download JBoss and deploy Magnolia there.
>
Good to know ... do you have the info for remaining 10% of the requests
that do not finish in 950ms? Are there some that do not return at all?
Do we know on what urls those happened? Are the affected urls same all
the time or is the pattern for those 10% of longer timed requests
random?

Jan


----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Rakesh Vidyadharan]

On 05/10/2009 9:56:08 AM, "Denis Demichev" <demichev@...> wrote:

Even if I put 1600 tomcat threads it doesn't make any difference.
Actually I'm able to hang tomcat with 500 concurrent threads having 800 threads set in tomcat.
Seems like 450 concurrent users is a sort of magic figure: magnolia is able to handle less than 450, but hangs when its 500.
Will try to download JBoss and deploy Magnolia there.

Regards,
Denis

It may be better to try GlassFish instead of JBoss.  JBoss also uses Tomcat as the servlet container, so if the issue is related in some way to Tomcat, you should see the same behaviour.

Rakesh

---

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[caste]

Thank you very much. I disabled versioning and the gain factor is about 100 and Magnolia now is "Supersonic".
But who should use these feature (versioning)? In a context where there are  20-50 person in the same moment connected to Magnolia is very hard to work with the admin interface, also browse the tree.

My question is: what are your recommendations or the best practices to mantain the versioning enabled? Or is versioning  a feature that in my environment should stay turned off and regular export of the only important data is better?

Thank you.
Andrea

You can reply in another thread to do not overlap the Denis' question.

2009/10/5 Jan Haderka <jan.haderka@...>

> Yes, I made a mistake.
> I'm reading the debate because I'm very interested.
> I noticed low performance when I use the command "activate this page",
> and the page has other subpages. My question is similar: what does JR
> do in this operation? Maybe does it made lock on the subpages so
> performance goes down?

If you have enabled the versioning, what happens is that the page (and
all the sub-pages, if selected) has to be versioned at that point. The
activation then happens on the versioned content, to make sure that the
what gets activated is what you actually selected and not any possible
modifications made by others later. The versioning (among other things)
means that value of each node has to be copied so it involves intensive
reading and writing, hence the impact on the performance.
Whether you are using workflow or now, the copying of the node from the
author instance to the public instance happens later. Again, this means
that value for each property in the activated content has to be read
from the repository, the content is converted in the xml and sent over
to the public instance (together with all other relevant info, such as
ordering of that content, etc.). The public instance has to process such
incoming content and import it in the repository. In case of
transactional information, it needs also backup the previous version of
the content in case there is a rollback request issued later during the
transaction. Again it involves both reading and writing from/to repo for
each property of the activated content.
Each of those things are expensive and have negative impact on
performance.

HTH,
Jan

>
> Thank you, sorry for my mistake.
> Best regards.
> Andrea
>
>
>
>
>
>
> 2009/10/5 Jan Haderka <jan.haderka@...>
>
>         I believe Andrea meant to forward your mail to someone else
>         (Marco) and
>         just hit the "send" button too early :D
>
>
>
>         On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev wrote:
>         > Hello
>         >
>         > I'm not a big expert in Italian - sorry. Is that a question
>         for me?
>         > If I interpreted it right, that means do I know what exactly
>         these
>         > tomcat parameters mean?
>         >
>         > If so, I can say that these parameters stand for how much
>         threads will
>         > be created by tomcat to handle incoming user requests. As
>         soon as I'm
>         > planning 800-1000 concurrent users I thought it makes sense
>         to
>         > increase default thread number.
>         >
>         >
>         > Regards,
>         > Denis
>         >
>         >
>         > On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
>         > <andrea.castelli.1@...> wrote:
>         >         Ciao Marco sul forum di Magnlia ho letto questo post
>         che
>         >         sembra interessante:
>         >
>         >         Noi conosciamo questi i valori di questi parametri
>         di Tomcat
>         >         nella configurazione attuale?
>         >
>         >
>         >         Tomcat: maxThreads="800" minSpareThreads="25"
>         >         maxSpareThreads="75"
>         >         Tomcat startup params:
>         >         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         -XX:MaxPermSize=1024M -XX:
>         >         +UseConcMarkSweepGC -XX:+CMSIncrementalMode
>         >         -Djava.net.preferIPv4Stack=true -XX:
>         +DoEscapeAnalysis
>         >
>         >
>         >         Andrea
>         >
>         >         2009/10/5 Denis Demichev <demichev@...>
>         >
>         >
>         >                 Hello All,
>         >
>         >                 I've been playing with Magnolia for a while
>         trying to
>         >                 asses its stability. Eventually I put it
>         under the
>         >                 stress on previous week - 800 concurrent
>         users on a
>         >                 single box.
>         >
>         >                 Server: Windows Server 2007(?) SP1 64 bit,
>          Intel Xeon
>         >                 2 Cores, 4Gb RAM
>         >                 JDK 1.6.0_16 64-bit version
>         >                 Magnolia 4.1 Community Edition (STK
>         Installed, Debug
>         >                 log is disabled) + MySQL 5.1.39
>         >                 Tomcat: maxThreads="800"
>         minSpareThreads="25"
>         >                 maxSpareThreads="75"
>         >                 Tomcat startup params:
>         >                 JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         >                 -XX:MaxPermSize=1024M -XX:
>         +UseConcMarkSweepGC -XX:
>         >                 +CMSIncrementalMode
>         -Djava.net.preferIPv4Stack=true
>         >                 -XX:+DoEscapeAnalysis
>         >                 Client - J-Meter, 800 Concurrent threads, 5
>         static
>         >                 links + 1 search link, Gaussian timer set
>         for 1000ms
>         >                 +-1500ms
>         >                 Response time 954ms, Errors 17%
>         >                 Avg. CPU utilization ~85%, RAM Utilization
>         ~2Gb
>         >
>         >                 Now the strange behavior I noticed is
>         following:
>         >                 Magnolia is responding to incoming requests
>         normally
>         >                 for about 2-5 minutes (more or less) and
>         then just
>         >                 freezes. CPU usage = 0%, RAM is the same (GC
>         is
>         >                 obviously not running). No responses at all.
>         >                 After removing the stress from Magnolia it
>         starts
>         >                 responding in 30-40 seconds.
>         >                 500 Concurrent users will produce the same
>         effect with
>         >                 the only exception it might take 50-100
>         minutes (maybe
>         >                 more) to reproduce.
>         >                 Tried to look at MySQL connection stack
>         using remote
>         >                 admin - 8 connections all of them are idle.
>         Seems like
>         >                 all the content was cached by JCR
>         previously.
>         >
>         >                 Question: What exactly might cause this
>         strange
>         >                 freezing? It looks like tomcat has enough
>         maximum
>         >                 threads, connection pool to MySQL is not
>         that busy...
>         >                 Could it be some internal cache of
>         jackrabbit? Not
>         >                 sure but it looks like JVM is not the
>         rootcause in
>         >                 this case.
>         >
>         >                 Would appreciate your thoughts on this
>         point.
>         >                 Thank you in advance!
>         >
>         >                 Regards,
>         >                 Denis
>         >
>         >
>
>
>
>
>         ----------------------------------------------------------------
>         For list details see
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         To unsubscribe, E-mail to:
>         <user-list-unsubscribe@...>
>         ----------------------------------------------------------------
>
>
>


----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Jan Haderka-2]

This is a question which is nearly impossible to answer generally. There
is no "silver bullet" settings. It depends on your exact configuration,
performance of the system you are running Magnolia on, amount of
content, editors, updates, size of the content, and so on.
The versioning feature is not meant for the backup of the content, but
for a possibility to re-trace the changes and rollback of individual
content. If your business requirements do not mandate versioning of the
content and if you have straight publishing process you are nearly
always better of without versioning. Not only does it impact the
performance, but also the total size of the backend storage.
On the other hand even if you need versioning you have some options by
structuring the content properly.
The versioning of the website content happens automatically on
activation, which is what you have noticed, but for example for DMS you
can set versioning upon saving of the content rather then during
activation so you will version only and only when you actually upload
new version of the content.
The time necessary for versioning is a function of the size of the
content. Think about it when deciding on your content structure. Try to
separate as much of the long living rarely changing values from those
changing frequently (for example by making use of inheritance and
storing rarely changing common values in parent pages, even when
rendering then only in child pages).
Another strategy you can put in the mix, if the content is not changing
rapidly in your environment and if you use workflow is to move
versioning into the activation command chain. This way versioning will
not happen interactively, so the AdminCentral will respond much faster
to the activation request, but there will be tiny window (less then
second) in which content could be modified after you click "activate"
and before the versioning of the content actually happens. If that is
acceptable risk (which should be fine if the content is not changing
rapidly or if your editorial process ensures that each editor is
responsible for different part of the site and they in general do not
meddle with each others content) you can use this to improve perceived
activation speed.

... you see there are many ifs and maybes in the above because it really
depends on exact setting and environment and other imposed
limitations ... still I hope it gives you at least some insights in what
and how can be improved.

Cheers,
Jan

On Tue, 2009-10-06 at 08:27 +0200, Andrea Castelli wrote:

> Thank you very much. I disabled versioning and the gain factor is
> about 100 and Magnolia now is "Supersonic".
> But who should use these feature (versioning)? In a context where
> there are  20-50 person in the same moment connected to Magnolia is
> very hard to work with the admin interface, also browse the tree.
>
> My question is: what are your recommendations or the best practices to
> mantain the versioning enabled? Or is versioning  a feature that in my
> environment should stay turned off and regular export of the only
> important data is better?
>
> Thank you.
> Andrea
>
> You can reply in another thread to do not overlap the Denis'
> question.
>
> 2009/10/5 Jan Haderka <jan.haderka@...>
>        
>        
>         > Yes, I made a mistake.
>         > I'm reading the debate because I'm very interested.
>         > I noticed low performance when I use the command "activate
>         this page",
>         > and the page has other subpages. My question is similar:
>         what does JR
>         > do in this operation? Maybe does it made lock on the
>         subpages so
>         > performance goes down?
>        
>        
>         If you have enabled the versioning, what happens is that the
>         page (and
>         all the sub-pages, if selected) has to be versioned at that
>         point. The
>         activation then happens on the versioned content, to make sure
>         that the
>         what gets activated is what you actually selected and not any
>         possible
>         modifications made by others later. The versioning (among
>         other things)
>         means that value of each node has to be copied so it involves
>         intensive
>         reading and writing, hence the impact on the performance.
>         Whether you are using workflow or now, the copying of the node
>         from the
>         author instance to the public instance happens later. Again,
>         this means
>         that value for each property in the activated content has to
>         be read
>         from the repository, the content is converted in the xml and
>         sent over
>         to the public instance (together with all other relevant info,
>         such as
>         ordering of that content, etc.). The public instance has to
>         process such
>         incoming content and import it in the repository. In case of
>         transactional information, it needs also backup the previous
>         version of
>         the content in case there is a rollback request issued later
>         during the
>         transaction. Again it involves both reading and writing
>         from/to repo for
>         each property of the activated content.
>         Each of those things are expensive and have negative impact on
>         performance.
>        
>         HTH,
>         Jan
>        
>        
>         >
>         > Thank you, sorry for my mistake.
>         > Best regards.
>         > Andrea
>         >
>         >
>         >
>         >
>         >
>         >
>         > 2009/10/5 Jan Haderka <jan.haderka@...>
>         >
>         >         I believe Andrea meant to forward your mail to
>         someone else
>         >         (Marco) and
>         >         just hit the "send" button too early :D
>         >
>         >
>         >
>         >         On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev
>         wrote:
>         >         > Hello
>         >         >
>         >         > I'm not a big expert in Italian - sorry. Is that a
>         question
>         >         for me?
>         >         > If I interpreted it right, that means do I know
>         what exactly
>         >         these
>         >         > tomcat parameters mean?
>         >         >
>         >         > If so, I can say that these parameters stand for
>         how much
>         >         threads will
>         >         > be created by tomcat to handle incoming user
>         requests. As
>         >         soon as I'm
>         >         > planning 800-1000 concurrent users I thought it
>         makes sense
>         >         to
>         >         > increase default thread number.
>         >         >
>         >         >
>         >         > Regards,
>         >         > Denis
>         >         >
>         >         >
>         >         > On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
>         >         > <andrea.castelli.1@...> wrote:
>         >         >         Ciao Marco sul forum di Magnlia ho letto
>         questo post
>         >         che
>         >         >         sembra interessante:
>         >         >
>         >         >         Noi conosciamo questi i valori di questi
>         parametri
>         >         di Tomcat
>         >         >         nella configurazione attuale?
>         >         >
>         >         >
>         >         >         Tomcat: maxThreads="800"
>         minSpareThreads="25"
>         >         >         maxSpareThreads="75"
>         >         >         Tomcat startup params:
>         >         >         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         >         -XX:MaxPermSize=1024M -XX:
>         >         >         +UseConcMarkSweepGC -XX:
>         +CMSIncrementalMode
>         >         >         -Djava.net.preferIPv4Stack=true -XX:
>         >         +DoEscapeAnalysis
>         >         >
>         >         >
>         >         >         Andrea
>         >         >
>         >         >         2009/10/5 Denis Demichev
>         <demichev@...>
>         >         >
>         >         >
>         >         >                 Hello All,
>         >         >
>         >         >                 I've been playing with Magnolia
>         for a while
>         >         trying to
>         >         >                 asses its stability. Eventually I
>         put it
>         >         under the
>         >         >                 stress on previous week - 800
>         concurrent
>         >         users on a
>         >         >                 single box.
>         >         >
>         >         >                 Server: Windows Server 2007(?) SP1
>         64 bit,
>         >          Intel Xeon
>         >         >                 2 Cores, 4Gb RAM
>         >         >                 JDK 1.6.0_16 64-bit version
>         >         >                 Magnolia 4.1 Community Edition
>         (STK
>         >         Installed, Debug
>         >         >                 log is disabled) + MySQL 5.1.39
>         >         >                 Tomcat: maxThreads="800"
>         >         minSpareThreads="25"
>         >         >                 maxSpareThreads="75"
>         >         >                 Tomcat startup params:
>         >         >                 JAVA_OPTS=%JAVA_OPTS% -Xms2G
>         -Xmx2G
>         >         >                 -XX:MaxPermSize=1024M -XX:
>         >         +UseConcMarkSweepGC -XX:
>         >         >                 +CMSIncrementalMode
>         >         -Djava.net.preferIPv4Stack=true
>         >         >                 -XX:+DoEscapeAnalysis
>         >         >                 Client - J-Meter, 800 Concurrent
>         threads, 5
>         >         static
>         >         >                 links + 1 search link, Gaussian
>         timer set
>         >         for 1000ms
>         >         >                 +-1500ms
>         >         >                 Response time 954ms, Errors 17%
>         >         >                 Avg. CPU utilization ~85%, RAM
>         Utilization
>         >         ~2Gb
>         >         >
>         >         >                 Now the strange behavior I noticed
>         is
>         >         following:
>         >         >                 Magnolia is responding to incoming
>         requests
>         >         normally
>         >         >                 for about 2-5 minutes (more or
>         less) and
>         >         then just
>         >         >                 freezes. CPU usage = 0%, RAM is
>         the same (GC
>         >         is
>         >         >                 obviously not running). No
>         responses at all.
>         >         >                 After removing the stress from
>         Magnolia it
>         >         starts
>         >         >                 responding in 30-40 seconds.
>         >         >                 500 Concurrent users will produce
>         the same
>         >         effect with
>         >         >                 the only exception it might take
>         50-100
>         >         minutes (maybe
>         >         >                 more) to reproduce.
>         >         >                 Tried to look at MySQL connection
>         stack
>         >         using remote
>         >         >                 admin - 8 connections all of them
>         are idle.
>         >         Seems like
>         >         >                 all the content was cached by JCR
>         >         previously.
>         >         >
>         >         >                 Question: What exactly might cause
>         this
>         >         strange
>         >         >                 freezing? It looks like tomcat has
>         enough
>         >         maximum
>         >         >                 threads, connection pool to MySQL
>         is not
>         >         that busy...
>         >         >                 Could it be some internal cache of
>         >         jackrabbit? Not
>         >         >                 sure but it looks like JVM is not
>         the
>         >         rootcause in
>         >         >                 this case.
>         >         >
>         >         >                 Would appreciate your thoughts on
>         this
>         >         point.
>         >         >                 Thank you in advance!
>         >         >
>         >         >                 Regards,
>         >         >                 Denis
>         >         >
>         >         >
>         >
>         >
>         >
>         >
>         >
>         ----------------------------------------------------------------
>         >         For list details see
>         >
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         >         To unsubscribe, E-mail to:
>         >         <user-list-unsubscribe@...>
>         >
>         ----------------------------------------------------------------
>         >
>         >
>         >
>        
>        
>         ----------------------------------------------------------------
>         For list details see
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         To unsubscribe, E-mail to:
>         <user-list-unsubscribe@...>
>         ----------------------------------------------------------------
>        
>        
>

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[caste]

Thank you, of course a better knowledge of alternatives and processes lead everyone to  make a better choice.

Cheers,
Andrea

2009/10/6 Jan Haderka <jan.haderka@...>

This is a question which is nearly impossible to answer generally. There
is no "silver bullet" settings. It depends on your exact configuration,
performance of the system you are running Magnolia on, amount of
content, editors, updates, size of the content, and so on.
The versioning feature is not meant for the backup of the content, but
for a possibility to re-trace the changes and rollback of individual
content. If your business requirements do not mandate versioning of the
content and if you have straight publishing process you are nearly
always better of without versioning. Not only does it impact the
performance, but also the total size of the backend storage.
On the other hand even if you need versioning you have some options by
structuring the content properly.
The versioning of the website content happens automatically on
activation, which is what you have noticed, but for example for DMS you
can set versioning upon saving of the content rather then during
activation so you will version only and only when you actually upload
new version of the content.
The time necessary for versioning is a function of the size of the
content. Think about it when deciding on your content structure. Try to
separate as much of the long living rarely changing values from those
changing frequently (for example by making use of inheritance and
storing rarely changing common values in parent pages, even when
rendering then only in child pages).
Another strategy you can put in the mix, if the content is not changing
rapidly in your environment and if you use workflow is to move
versioning into the activation command chain. This way versioning will
not happen interactively, so the AdminCentral will respond much faster
to the activation request, but there will be tiny window (less then
second) in which content could be modified after you click "activate"
and before the versioning of the content actually happens. If that is
acceptable risk (which should be fine if the content is not changing
rapidly or if your editorial process ensures that each editor is
responsible for different part of the site and they in general do not
meddle with each others content) you can use this to improve perceived
activation speed.

... you see there are many ifs and maybes in the above because it really
depends on exact setting and environment and other imposed
limitations ... still I hope it gives you at least some insights in what
and how can be improved.

Cheers,
Jan

On Tue, 2009-10-06 at 08:27 +0200, Andrea Castelli wrote:
> Thank you very much. I disabled versioning and the gain factor is
> about 100 and Magnolia now is "Supersonic".
> But who should use these feature (versioning)? In a context where
> there are  20-50 person in the same moment connected to Magnolia is
> very hard to work with the admin interface, also browse the tree.
>
> My question is: what are your recommendations or the best practices to
> mantain the versioning enabled? Or is versioning  a feature that in my
> environment should stay turned off and regular export of the only
> important data is better?
>
> Thank you.
> Andrea
>
> You can reply in another thread to do not overlap the Denis'
> question.
>
> 2009/10/5 Jan Haderka <jan.haderka@...>
>
>
>         > Yes, I made a mistake.
>         > I'm reading the debate because I'm very interested.
>         > I noticed low performance when I use the command "activate
>         this page",
>         > and the page has other subpages. My question is similar:
>         what does JR
>         > do in this operation? Maybe does it made lock on the
>         subpages so
>         > performance goes down?
>
>
>         If you have enabled the versioning, what happens is that the
>         page (and
>         all the sub-pages, if selected) has to be versioned at that
>         point. The
>         activation then happens on the versioned content, to make sure
>         that the
>         what gets activated is what you actually selected and not any
>         possible
>         modifications made by others later. The versioning (among
>         other things)
>         means that value of each node has to be copied so it involves
>         intensive
>         reading and writing, hence the impact on the performance.
>         Whether you are using workflow or now, the copying of the node
>         from the
>         author instance to the public instance happens later. Again,
>         this means
>         that value for each property in the activated content has to
>         be read
>         from the repository, the content is converted in the xml and
>         sent over
>         to the public instance (together with all other relevant info,
>         such as
>         ordering of that content, etc.). The public instance has to
>         process such
>         incoming content and import it in the repository. In case of
>         transactional information, it needs also backup the previous
>         version of
>         the content in case there is a rollback request issued later
>         during the
>         transaction. Again it involves both reading and writing
>         from/to repo for
>         each property of the activated content.
>         Each of those things are expensive and have negative impact on
>         performance.
>
>         HTH,
>         Jan
>
>
>         >
>         > Thank you, sorry for my mistake.
>         > Best regards.
>         > Andrea
>         >
>         >
>         >
>         >
>         >
>         >
>         > 2009/10/5 Jan Haderka <jan.haderka@...>
>         >
>         >         I believe Andrea meant to forward your mail to
>         someone else
>         >         (Marco) and
>         >         just hit the "send" button too early :D
>         >
>         >
>         >
>         >         On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev
>         wrote:
>         >         > Hello
>         >         >
>         >         > I'm not a big expert in Italian - sorry. Is that a
>         question
>         >         for me?
>         >         > If I interpreted it right, that means do I know
>         what exactly
>         >         these
>         >         > tomcat parameters mean?
>         >         >
>         >         > If so, I can say that these parameters stand for
>         how much
>         >         threads will
>         >         > be created by tomcat to handle incoming user
>         requests. As
>         >         soon as I'm
>         >         > planning 800-1000 concurrent users I thought it
>         makes sense
>         >         to
>         >         > increase default thread number.
>         >         >
>         >         >
>         >         > Regards,
>         >         > Denis
>         >         >
>         >         >
>         >         > On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
>         >         > <andrea.castelli.1@...> wrote:
>         >         >         Ciao Marco sul forum di Magnlia ho letto
>         questo post
>         >         che
>         >         >         sembra interessante:
>         >         >
>         >         >         Noi conosciamo questi i valori di questi
>         parametri
>         >         di Tomcat
>         >         >         nella configurazione attuale?
>         >         >
>         >         >
>         >         >         Tomcat: maxThreads="800"
>         minSpareThreads="25"
>         >         >         maxSpareThreads="75"
>         >         >         Tomcat startup params:
>         >         >         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         >         -XX:MaxPermSize=1024M -XX:
>         >         >         +UseConcMarkSweepGC -XX:
>         +CMSIncrementalMode
>         >         >         -Djava.net.preferIPv4Stack=true -XX:
>         >         +DoEscapeAnalysis
>         >         >
>         >         >
>         >         >         Andrea
>         >         >
>         >         >         2009/10/5 Denis Demichev
>         <demichev@...>
>         >         >
>         >         >
>         >         >                 Hello All,
>         >         >
>         >         >                 I've been playing with Magnolia
>         for a while
>         >         trying to
>         >         >                 asses its stability. Eventually I
>         put it
>         >         under the
>         >         >                 stress on previous week - 800
>         concurrent
>         >         users on a
>         >         >                 single box.
>         >         >
>         >         >                 Server: Windows Server 2007(?) SP1
>         64 bit,
>         >          Intel Xeon
>         >         >                 2 Cores, 4Gb RAM
>         >         >                 JDK 1.6.0_16 64-bit version
>         >         >                 Magnolia 4.1 Community Edition
>         (STK
>         >         Installed, Debug
>         >         >                 log is disabled) + MySQL 5.1.39
>         >         >                 Tomcat: maxThreads="800"
>         >         minSpareThreads="25"
>         >         >                 maxSpareThreads="75"
>         >         >                 Tomcat startup params:
>         >         >                 JAVA_OPTS=%JAVA_OPTS% -Xms2G
>         -Xmx2G
>         >         >                 -XX:MaxPermSize=1024M -XX:
>         >         +UseConcMarkSweepGC -XX:
>         >         >                 +CMSIncrementalMode
>         >         -Djava.net.preferIPv4Stack=true
>         >         >                 -XX:+DoEscapeAnalysis
>         >         >                 Client - J-Meter, 800 Concurrent
>         threads, 5
>         >         static
>         >         >                 links + 1 search link, Gaussian
>         timer set
>         >         for 1000ms
>         >         >                 +-1500ms
>         >         >                 Response time 954ms, Errors 17%
>         >         >                 Avg. CPU utilization ~85%, RAM
>         Utilization
>         >         ~2Gb
>         >         >
>         >         >                 Now the strange behavior I noticed
>         is
>         >         following:
>         >         >                 Magnolia is responding to incoming
>         requests
>         >         normally
>         >         >                 for about 2-5 minutes (more or
>         less) and
>         >         then just
>         >         >                 freezes. CPU usage = 0%, RAM is
>         the same (GC
>         >         is
>         >         >                 obviously not running). No
>         responses at all.
>         >         >                 After removing the stress from
>         Magnolia it
>         >         starts
>         >         >                 responding in 30-40 seconds.
>         >         >                 500 Concurrent users will produce
>         the same
>         >         effect with
>         >         >                 the only exception it might take
>         50-100
>         >         minutes (maybe
>         >         >                 more) to reproduce.
>         >         >                 Tried to look at MySQL connection
>         stack
>         >         using remote
>         >         >                 admin - 8 connections all of them
>         are idle.
>         >         Seems like
>         >         >                 all the content was cached by JCR
>         >         previously.
>         >         >
>         >         >                 Question: What exactly might cause
>         this
>         >         strange
>         >         >                 freezing? It looks like tomcat has
>         enough
>         >         maximum
>         >         >                 threads, connection pool to MySQL
>         is not
>         >         that busy...
>         >         >                 Could it be some internal cache of
>         >         jackrabbit? Not
>         >         >                 sure but it looks like JVM is not
>         the
>         >         rootcause in
>         >         >                 this case.
>         >         >
>         >         >                 Would appreciate your thoughts on
>         this
>         >         point.
>         >         >                 Thank you in advance!
>         >         >
>         >         >                 Regards,
>         >         >                 Denis
>         >         >
>         >         >
>         >
>         >
>         >
>         >
>         >
>         ----------------------------------------------------------------
>         >         For list details see
>         >
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         >         To unsubscribe, E-mail to:
>         >         <user-list-unsubscribe@...>
>         >
>         ----------------------------------------------------------------
>         >
>         >
>         >
>
>
>         ----------------------------------------------------------------
>         For list details see
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         To unsubscribe, E-mail to:
>         <user-list-unsubscribe@...>
>         ----------------------------------------------------------------
>
>
>


----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Garima Parakh]

 

 Thanks for the insight into versioning issues. I have a related questions. Does limiting the number of versions increase the performance of activation?

 

Thanks

Garima

2009/10/6 Jan Haderka <jan.haderka@...>

This is a question which is nearly impossible to answer generally. There
is no "silver bullet" settings. It depends on your exact configuration,
performance of the system you are running Magnolia on, amount of
content, editors, updates, size of the content, and so on.
The versioning feature is not meant for the backup of the content, but
for a possibility to re-trace the changes and rollback of individual
content. If your business requirements do not mandate versioning of the
content and if you have straight publishing process you are nearly
always better of without versioning. Not only does it impact the
performance, but also the total size of the backend storage.
On the other hand even if you need versioning you have some options by
structuring the content properly.
The versioning of the website content happens automatically on
activation, which is what you have noticed, but for example for DMS you
can set versioning upon saving of the content rather then during
activation so you will version only and only when you actually upload
new version of the content.
The time necessary for versioning is a function of the size of the
content. Think about it when deciding on your content structure. Try to
separate as much of the long living rarely changing values from those
changing frequently (for example by making use of inheritance and
storing rarely changing common values in parent pages, even when
rendering then only in child pages).
Another strategy you can put in the mix, if the content is not changing
rapidly in your environment and if you use workflow is to move
versioning into the activation command chain. This way versioning will
not happen interactively, so the AdminCentral will respond much faster
to the activation request, but there will be tiny window (less then
second) in which content could be modified after you click "activate"
and before the versioning of the content actually happens. If that is
acceptable risk (which should be fine if the content is not changing
rapidly or if your editorial process ensures that each editor is
responsible for different part of the site and they in general do not
meddle with each others content) you can use this to improve perceived
activation speed.

... you see there are many ifs and maybes in the above because it really
depends on exact setting and environment and other imposed
limitations ... still I hope it gives you at least some insights in what
and how can be improved.

Cheers,
Jan

On Tue, 2009-10-06 at 08:27 +0200, Andrea Castelli wrote:

> Thank you very much. I disabled versioning and the gain factor is
> about 100 and Magnolia now is "Supersonic".
> But who should use these feature (versioning)? In a context where
> there are  20-50 person in the same moment connected to Magnolia is
> very hard to work with the admin interface, also browse the tree.
>
> My question is: what are your recommendations or the best practices to
> mantain the versioning enabled? Or is versioning  a feature that in my
> environment should stay turned off and regular export of the only
> important data is better?
>
> Thank you.
> Andrea
>
> You can reply in another thread to do not overlap the Denis'
> question.
>
> 2009/10/5 Jan Haderka <jan.haderka@...>
>
>
>         > Yes, I made a mistake.
>         > I'm reading the debate because I'm very interested.
>         > I noticed low performance when I use the command "activate
>         this page",
>         > and the page has other subpages. My question is similar:
>         what does JR
>         > do in this operation? Maybe does it made lock on the
>         subpages so
>         > performance goes down?
>
>
>         If you have enabled the versioning, what happens is that the
>         page (and
>         all the sub-pages, if selected) has to be versioned at that
>         point. The
>         activation then happens on the versioned content, to make sure
>         that the
>         what gets activated is what you actually selected and not any
>         possible
>         modifications made by others later. The versioning (among
>         other things)
>         means that value of each node has to be copied so it involves
>         intensive
>         reading and writing, hence the impact on the performance.
>         Whether you are using workflow or now, the copying of the node
>         from the
>         author instance to the public instance happens later. Again,
>         this means
>         that value for each property in the activated content has to
>         be read
>         from the repository, the content is converted in the xml and
>         sent over
>         to the public instance (together with all other relevant info,
>         such as
>         ordering of that content, etc.). The public instance has to
>         process such
>         incoming content and import it in the repository. In case of
>         transactional information, it needs also backup the previous
>         version of
>         the content in case there is a rollback request issued later
>         during the
>         transaction. Again it involves both reading and writing
>         from/to repo for
>         each property of the activated content.
>         Each of those things are expensive and have negative impact on
>         performance.
>
>         HTH,
>         Jan
>
>
>         >
>         > Thank you, sorry for my mistake.
>         > Best regards.
>         > Andrea
>         >
>         >
>         >
>         >
>         >
>         >
>         > 2009/10/5 Jan Haderka <jan.haderka@...>
>         >
>         >         I believe Andrea meant to forward your mail to
>         someone else
>         >         (Marco) and
>         >         just hit the "send" button too early :D
>         >
>         >
>         >
>         >         On Mon, 2009-10-05 at 09:48 -0400, Denis Demichev
>         wrote:
>         >         > Hello
>         >         >
>         >         > I'm not a big expert in Italian - sorry. Is that a
>         question
>         >         for me?
>         >         > If I interpreted it right, that means do I know
>         what exactly
>         >         these
>         >         > tomcat parameters mean?
>         >         >
>         >         > If so, I can say that these parameters stand for
>         how much
>         >         threads will
>         >         > be created by tomcat to handle incoming user
>         requests. As
>         >         soon as I'm
>         >         > planning 800-1000 concurrent users I thought it
>         makes sense
>         >         to
>         >         > increase default thread number.
>         >         >
>         >         >
>         >         > Regards,
>         >         > Denis
>         >         >
>         >         >
>         >         > On Mon, Oct 5, 2009 at 9:29 AM, Andrea Castelli
>         >         > <andrea.castelli.1@...> wrote:
>         >         >         Ciao Marco sul forum di Magnlia ho letto
>         questo post
>         >         che
>         >         >         sembra interessante:
>         >         >
>         >         >         Noi conosciamo questi i valori di questi
>         parametri
>         >         di Tomcat
>         >         >         nella configurazione attuale?
>         >         >
>         >         >
>         >         >         Tomcat: maxThreads="800"
>         minSpareThreads="25"
>         >         >         maxSpareThreads="75"
>         >         >         Tomcat startup params:
>         >         >         JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G
>         >         -XX:MaxPermSize=1024M -XX:
>         >         >         +UseConcMarkSweepGC -XX:
>         +CMSIncrementalMode
>         >         >         -Djava.net.preferIPv4Stack=true -XX:
>         >         +DoEscapeAnalysis
>         >         >
>         >         >
>         >         >         Andrea
>         >         >
>         >         >         2009/10/5 Denis Demichev
>         <demichev@...>
>         >         >
>         >         >
>         >         >                 Hello All,
>         >         >
>         >         >                 I've been playing with Magnolia
>         for a while
>         >         trying to
>         >         >                 asses its stability. Eventually I
>         put it
>         >         under the
>         >         >                 stress on previous week - 800
>         concurrent
>         >         users on a
>         >         >                 single box.
>         >         >
>         >         >                 Server: Windows Server 2007(?) SP1
>         64 bit,
>         >          Intel Xeon
>         >         >                 2 Cores, 4Gb RAM
>         >         >                 JDK 1.6.0_16 64-bit version
>         >         >                 Magnolia 4.1 Community Edition
>         (STK
>         >         Installed, Debug
>         >         >                 log is disabled) + MySQL 5.1.39
>         >         >                 Tomcat: maxThreads="800"
>         >         minSpareThreads="25"
>         >         >                 maxSpareThreads="75"
>         >         >                 Tomcat startup params:
>         >         >                 JAVA_OPTS=%JAVA_OPTS% -Xms2G
>         -Xmx2G
>         >         >                 -XX:MaxPermSize=1024M -XX:
>         >         +UseConcMarkSweepGC -XX:
>         >         >                 +CMSIncrementalMode
>         >         -Djava.net.preferIPv4Stack=true
>         >         >                 -XX:+DoEscapeAnalysis
>         >         >                 Client - J-Meter, 800 Concurrent
>         threads, 5
>         >         static
>         >         >                 links + 1 search link, Gaussian
>         timer set
>         >         for 1000ms
>         >         >                 +-1500ms
>         >         >                 Response time 954ms, Errors 17%
>         >         >                 Avg. CPU utilization ~85%, RAM
>         Utilization
>         >         ~2Gb
>         >         >
>         >         >                 Now the strange behavior I noticed
>         is
>         >         following:
>         >         >                 Magnolia is responding to incoming
>         requests
>         >         normally
>         >         >                 for about 2-5 minutes (more or
>         less) and
>         >         then just
>         >         >                 freezes. CPU usage = 0%, RAM is
>         the same (GC
>         >         is
>         >         >                 obviously not running). No
>         responses at all.
>         >         >                 After removing the stress from
>         Magnolia it
>         >         starts
>         >         >                 responding in 30-40 seconds.
>         >         >                 500 Concurrent users will produce
>         the same
>         >         effect with
>         >         >                 the only exception it might take
>         50-100
>         >         minutes (maybe
>         >         >                 more) to reproduce.
>         >         >                 Tried to look at MySQL connection
>         stack
>         >         using remote
>         >         >                 admin - 8 connections all of them
>         are idle.
>         >         Seems like
>         >         >                 all the content was cached by JCR
>         >         previously.
>         >         >
>         >         >                 Question: What exactly might cause
>         this
>         >         strange
>         >         >                 freezing? It looks like tomcat has
>         enough
>         >         maximum
>         >         >                 threads, connection pool to MySQL
>         is not
>         >         that busy...
>         >         >                 Could it be some internal cache of
>         >         jackrabbit? Not
>         >         >                 sure but it looks like JVM is not
>         the
>         >         rootcause in
>         >         >                 this case.
>         >         >
>         >         >                 Would appreciate your thoughts on
>         this
>         >         point.
>         >         >                 Thank you in advance!
>         >         >
>         >         >                 Regards,
>         >         >                 Denis
>         >         >
>         >         >
>         >
>         >
>         >
>         >
>         >
>         ----------------------------------------------------------------
>         >         For list details see
>         >
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         >         To unsubscribe, E-mail to:
>         >         <user-list-unsubscribe@...>
>         >
>         ----------------------------------------------------------------
>         >
>         >
>         >
>
>
>         ----------------------------------------------------------------
>         For list details see
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         To unsubscribe, E-mail to:
>         <user-list-unsubscribe@...>
>         ----------------------------------------------------------------
>
>
>

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

---

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------

[Jan Haderka-2]

Just changed the topic to better describe what is really discussed in
this thread now.

And, no, limiting the number of versions has no impact on performance of
the activation per se. It has the impact on the total size of the
database and therefore it might indirectly impact the performance of the
Magnolia instance as a whole, but there is no direct correlation between
number of versions and speed of activation.

If the versioning is slowing you down and you are more interested in
versions as a snapshots of the content at given time, rather then to
trace what exactly have been done, you can disable versioning on
activation all together and run the scheduled task that would version
the whole site each time it is executed (say once a month).

HTH,
Jan

On Wed, 2009-10-07 at 12:01 -0400, Garima Parakh wrote:

>  
>
>  Thanks for the insight into versioning issues. I have a related
> questions. Does limiting the number of versions increase the
> performance of activation?
>  
> Thanks
> Garima
>
>
>
> 2009/10/6 Jan Haderka <jan.haderka@...>
>        
>         This is a question which is nearly impossible to answer
>         generally. There
>         is no "silver bullet" settings. It depends on your exact
>         configuration,
>         performance of the system you are running Magnolia on, amount
>         of
>         content, editors, updates, size of the content, and so on.
>         The versioning feature is not meant for the backup of the
>         content, but
>         for a possibility to re-trace the changes and rollback of
>         individual
>         content. If your business requirements do not mandate
>         versioning of the
>         content and if you have straight publishing process you are
>         nearly
>         always better of without versioning. Not only does it impact
>         the
>         performance, but also the total size of the backend storage.
>         On the other hand even if you need versioning you have some
>         options by
>         structuring the content properly.
>         The versioning of the website content happens automatically on
>         activation, which is what you have noticed, but for example
>         for DMS you
>         can set versioning upon saving of the content rather then
>         during
>         activation so you will version only and only when you actually
>         upload
>         new version of the content.
>         The time necessary for versioning is a function of the size of
>         the
>         content. Think about it when deciding on your content
>         structure. Try to
>         separate as much of the long living rarely changing values
>         from those
>         changing frequently (for example by making use of inheritance
>         and
>         storing rarely changing common values in parent pages, even
>         when
>         rendering then only in child pages).
>         Another strategy you can put in the mix, if the content is not
>         changing
>         rapidly in your environment and if you use workflow is to move
>         versioning into the activation command chain. This way
>         versioning will
>         not happen interactively, so the AdminCentral will respond
>         much faster
>         to the activation request, but there will be tiny window (less
>         then
>         second) in which content could be modified after you click
>         "activate"
>         and before the versioning of the content actually happens. If
>         that is
>         acceptable risk (which should be fine if the content is not
>         changing
>         rapidly or if your editorial process ensures that each editor
>         is
>         responsible for different part of the site and they in general
>         do not
>         meddle with each others content) you can use this to improve
>         perceived
>         activation speed.
>        
>         ... you see there are many ifs and maybes in the above because
>         it really
>         depends on exact setting and environment and other imposed
>         limitations ... still I hope it gives you at least some
>         insights in what
>         and how can be improved.
>        
>         Cheers,
>         Jan
>        
>        
>         On Tue, 2009-10-06 at 08:27 +0200, Andrea Castelli wrote:
>         > Thank you very much. I disabled versioning and the gain
>         factor is
>         > about 100 and Magnolia now is "Supersonic".
>         > But who should use these feature (versioning)? In a context
>         where
>         > there are  20-50 person in the same moment connected to
>         Magnolia is
>         > very hard to work with the admin interface, also browse the
>         tree.
>         >
>         > My question is: what are your recommendations or the best
>         practices to
>         > mantain the versioning enabled? Or is versioning  a feature
>         that in my
>         > environment should stay turned off and regular export of the
>         only
>         > important data is better?
>         >
>         > Thank you.
>         > Andrea
>         >
>         > You can reply in another thread to do not overlap the Denis'
>         > question.
>         >
>         > 2009/10/5 Jan Haderka <jan.haderka@...>
>         >
>         >
>         >         > Yes, I made a mistake.
>         >         > I'm reading the debate because I'm very
>         interested.
>         >         > I noticed low performance when I use the command
>         "activate
>         >         this page",
>         >         > and the page has other subpages. My question is
>         similar:
>         >         what does JR
>         >         > do in this operation? Maybe does it made lock on
>         the
>         >         subpages so
>         >         > performance goes down?
>         >
>         >
>         >         If you have enabled the versioning, what happens is
>         that the
>         >         page (and
>         >         all the sub-pages, if selected) has to be versioned
>         at that
>         >         point. The
>         >         activation then happens on the versioned content, to
>         make sure
>         >         that the
>         >         what gets activated is what you actually selected
>         and not any
>         >         possible
>         >         modifications made by others later. The versioning
>         (among
>         >         other things)
>         >         means that value of each node has to be copied so it
>         involves
>         >         intensive
>         >         reading and writing, hence the impact on the
>         performance.
>         >         Whether you are using workflow or now, the copying
>         of the node
>         >         from the
>         >         author instance to the public instance happens
>         later. Again,
>         >         this means
>         >         that value for each property in the activated
>         content has to
>         >         be read
>         >         from the repository, the content is converted in the
>         xml and
>         >         sent over
>         >         to the public instance (together with all other
>         relevant info,
>         >         such as
>         >         ordering of that content, etc.). The public instance
>         has to
>         >         process such
>         >         incoming content and import it in the repository. In
>         case of
>         >         transactional information, it needs also backup the
>         previous
>         >         version of
>         >         the content in case there is a rollback request
>         issued later
>         >         during the
>         >         transaction. Again it involves both reading and
>         writing
>         >         from/to repo for
>         >         each property of the activated content.
>         >         Each of those things are expensive and have negative
>         impact on
>         >         performance.
>         >
>         >         HTH,
>         >         Jan
>         >
>         >
>         >         >
>         >         > Thank you, sorry for my mistake.
>         >         > Best regards.
>         >         > Andrea
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >         > 2009/10/5 Jan Haderka
>         <jan.haderka@...>
>         >         >
>         >         >         I believe Andrea meant to forward your
>         mail to
>         >         someone else
>         >         >         (Marco) and
>         >         >         just hit the "send" button too early :D
>         >         >
>         >         >
>         >         >
>         >         >         On Mon, 2009-10-05 at 09:48 -0400, Denis
>         Demichev
>         >         wrote:
>         >         >         > Hello
>         >         >         >
>         >         >         > I'm not a big expert in Italian - sorry.
>         Is that a
>         >         question
>         >         >         for me?
>         >         >         > If I interpreted it right, that means do
>         I know
>         >         what exactly
>         >         >         these
>         >         >         > tomcat parameters mean?
>         >         >         >
>         >         >         > If so, I can say that these parameters
>         stand for
>         >         how much
>         >         >         threads will
>         >         >         > be created by tomcat to handle incoming
>         user
>         >         requests. As
>         >         >         soon as I'm
>         >         >         > planning 800-1000 concurrent users I
>         thought it
>         >         makes sense
>         >         >         to
>         >         >         > increase default thread number.
>         >         >         >
>         >         >         >
>         >         >         > Regards,
>         >         >         > Denis
>         >         >         >
>         >         >         >
>         >         >         > On Mon, Oct 5, 2009 at 9:29 AM, Andrea
>         Castelli
>         >         >         > <andrea.castelli.1@...> wrote:
>         >         >         >         Ciao Marco sul forum di Magnlia
>         ho letto
>         >         questo post
>         >         >         che
>         >         >         >         sembra interessante:
>         >         >         >
>         >         >         >         Noi conosciamo questi i valori
>         di questi
>         >         parametri
>         >         >         di Tomcat
>         >         >         >         nella configurazione attuale?
>         >         >         >
>         >         >         >
>         >         >         >         Tomcat: maxThreads="800"
>         >         minSpareThreads="25"
>         >         >         >         maxSpareThreads="75"
>         >         >         >         Tomcat startup params:
>         >         >         >         JAVA_OPTS=%JAVA_OPTS% -Xms2G
>         -Xmx2G
>         >         >         -XX:MaxPermSize=1024M -XX:
>         >         >         >         +UseConcMarkSweepGC -XX:
>         >         +CMSIncrementalMode
>         >         >         >         -Djava.net.preferIPv4Stack=true
>         -XX:
>         >         >         +DoEscapeAnalysis
>         >         >         >
>         >         >         >
>         >         >         >         Andrea
>         >         >         >
>         >         >         >         2009/10/5 Denis Demichev
>         >         <demichev@...>
>         >         >         >
>         >         >         >
>         >         >         >                 Hello All,
>         >         >         >
>         >         >         >                 I've been playing with
>         Magnolia
>         >         for a while
>         >         >         trying to
>         >         >         >                 asses its stability.
>         Eventually I
>         >         put it
>         >         >         under the
>         >         >         >                 stress on previous week
>         - 800
>         >         concurrent
>         >         >         users on a
>         >         >         >                 single box.
>         >         >         >
>         >         >         >                 Server: Windows Server
>         2007(?) SP1
>         >         64 bit,
>         >         >          Intel Xeon
>         >         >         >                 2 Cores, 4Gb RAM
>         >         >         >                 JDK 1.6.0_16 64-bit
>         version
>         >         >         >                 Magnolia 4.1 Community
>         Edition
>         >         (STK
>         >         >         Installed, Debug
>         >         >         >                 log is disabled) + MySQL
>         5.1.39
>         >         >         >                 Tomcat: maxThreads="800"
>         >         >         minSpareThreads="25"
>         >         >         >                 maxSpareThreads="75"
>         >         >         >                 Tomcat startup params:
>         >         >         >                 JAVA_OPTS=%JAVA_OPTS%
>         -Xms2G
>         >         -Xmx2G
>         >         >         >                 -XX:MaxPermSize=1024M
>         -XX:
>         >         >         +UseConcMarkSweepGC -XX:
>         >         >         >                 +CMSIncrementalMode
>         >         >         -Djava.net.preferIPv4Stack=true
>         >         >         >                 -XX:+DoEscapeAnalysis
>         >         >         >                 Client - J-Meter, 800
>         Concurrent
>         >         threads, 5
>         >         >         static
>         >         >         >                 links + 1 search link,
>         Gaussian
>         >         timer set
>         >         >         for 1000ms
>         >         >         >                 +-1500ms
>         >         >         >                 Response time 954ms,
>         Errors 17%
>         >         >         >                 Avg. CPU utilization
>         ~85%, RAM
>         >         Utilization
>         >         >         ~2Gb
>         >         >         >
>         >         >         >                 Now the strange behavior
>         I noticed
>         >         is
>         >         >         following:
>         >         >         >                 Magnolia is responding
>         to incoming
>         >         requests
>         >         >         normally
>         >         >         >                 for about 2-5 minutes
>         (more or
>         >         less) and
>         >         >         then just
>         >         >         >                 freezes. CPU usage = 0%,
>         RAM is
>         >         the same (GC
>         >         >         is
>         >         >         >                 obviously not running).
>         No
>         >         responses at all.
>         >         >         >                 After removing the
>         stress from
>         >         Magnolia it
>         >         >         starts
>         >         >         >                 responding in 30-40
>         seconds.
>         >         >         >                 500 Concurrent users
>         will produce
>         >         the same
>         >         >         effect with
>         >         >         >                 the only exception it
>         might take
>         >         50-100
>         >         >         minutes (maybe
>         >         >         >                 more) to reproduce.
>         >         >         >                 Tried to look at MySQL
>         connection
>         >         stack
>         >         >         using remote
>         >         >         >                 admin - 8 connections
>         all of them
>         >         are idle.
>         >         >         Seems like
>         >         >         >                 all the content was
>         cached by JCR
>         >         >         previously.
>         >         >         >
>         >         >         >                 Question: What exactly
>         might cause
>         >         this
>         >         >         strange
>         >         >         >                 freezing? It looks like
>         tomcat has
>         >         enough
>         >         >         maximum
>         >         >         >                 threads, connection pool
>         to MySQL
>         >         is not
>         >         >         that busy...
>         >         >         >                 Could it be some
>         internal cache of
>         >         >         jackrabbit? Not
>         >         >         >                 sure but it looks like
>         JVM is not
>         >         the
>         >         >         rootcause in
>         >         >         >                 this case.
>         >         >         >
>         >         >         >                 Would appreciate your
>         thoughts on
>         >         this
>         >         >         point.
>         >         >         >                 Thank you in advance!
>         >         >         >
>         >         >         >                 Regards,
>         >         >         >                 Denis
>         >         >         >
>         >         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >
>         ----------------------------------------------------------------
>         >         >         For list details see
>         >         >
>         >
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         >         >         To unsubscribe, E-mail to:
>         >         >         <user-list-unsubscribe@...>
>         >         >
>         >
>         ----------------------------------------------------------------
>         >         >
>         >         >
>         >         >
>         >
>         >
>         >
>         ----------------------------------------------------------------
>         >         For list details see
>         >
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         >         To unsubscribe, E-mail to:
>         >         <user-list-unsubscribe@...>
>         >
>         ----------------------------------------------------------------
>         >
>         >
>         >
>        
>        
>         ----------------------------------------------------------------
>         For list details see
>         http://www.magnolia-cms.com/home/community/mailing-lists.html
>         To unsubscribe, E-mail to:
>         <user-list-unsubscribe@...>
>         ----------------------------------------------------------------
>        
>        
>
>
>
>
> ______________________________________________________________________
> ----------------------------------------------------------------
> For list details see
> http://www.magnolia-cms.com/home/community/mailing-lists.html
> To unsubscribe, E-mail to: <user-list-unsubscribe@...>
> ----------------------------------------------------------------

----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <user-list-unsubscribe@...>
----------------------------------------------------------------