Starting Geronimo with More PermGen

[Q Beukes]

Hey,

How do I start geronimo with more permgen.

I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
only gives it to the bootstrap loader, and not the server itself, as
can be seen from these processes:

19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
-XX:MaxPermSize=256m -jar
/opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
geronimo/start-server --logfile '/opt/kms/server/geron
19637 pts/12   Sl     0:36  \_
/opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
-javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
-Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200


--
Quintin Beukes

[kevan]

On Sep 11, 2009, at 9:52 AM, Quintin Beukes wrote:

> Hey,
>
> How do I start geronimo with more permgen.
>
> I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
> only gives it to the bootstrap loader, and not the server itself, as
> can be seen from these processes:
>
> 19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
> -XX:MaxPermSize=256m -jar
> /opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
> geronimo/start-server --logfile '/opt/kms/server/geron
> 19637 pts/12   Sl     0:36  \_
> /opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
> -javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
> -Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200

See http://cwiki.apache.org/confluence/display/GMOxDOC22/Runtime+issues#Runtimeissues-JVMarguments

You need to update /etc/rc.d/start-server,default.groovy or add a -J  
parameter.

We could probably do with a bin/README / comments in the .bat/.sh  
files to assist with these issues.

--kevan

[Quintin Beukes-2]

Had I had a README I would have read them... well I have the .sh files
but they don't mention it either.

ls bin/
client.bat  cxf-tools.bat  deploy.bat    geronimo.bat    gsh
 jaxws-tools.bat  jpa.jar         setjavaenv.sh  shutdown.sh
startup.bat  stop-server.bat
client.jar  cxf-tools.jar  deployer.jar  geronimo.sh     gsh.bat
 jaxws-tools.jar  server.jar      shutdown.bat   start-server
startup.sh
client.sh   cxf-tools.sh   deploy.sh     gserviceReg.sh  isdeployed.sh
 jaxws-tools.sh   setjavaenv.bat  shutdown.jar   start-server.bat
stop-server

Beyond that I have another question. Inside the .groovy file, how can
I reference environment variables? I'm hoping to only update
setjavaenv.sh to make upgrading easier.

Q

On Fri, Sep 11, 2009 at 5:50 PM, Kevan Miller <kevan.miller@...> wrote:

>
> On Sep 11, 2009, at 9:52 AM, Quintin Beukes wrote:
>
>> Hey,
>>
>> How do I start geronimo with more permgen.
>>
>> I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
>> only gives it to the bootstrap loader, and not the server itself, as
>> can be seen from these processes:
>>
>> 19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
>> -XX:MaxPermSize=256m -jar
>> /opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
>> geronimo/start-server --logfile '/opt/kms/server/geron
>> 19637 pts/12   Sl     0:36  \_
>> /opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
>> -javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
>> -Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200
>
> See
> http://cwiki.apache.org/confluence/display/GMOxDOC22/Runtime+issues#Runtimeissues-JVMarguments
>
> You need to update /etc/rc.d/start-server,default.groovy or add a -J
> parameter.
>
> We could probably do with a bin/README / comments in the .bat/.sh files to
> assist with these issues.
>
> --kevan
>

--
Quintin Beukes

[Russell Collins]

Unless you are tied into Hotspot (Sun JVM), BEA/Oracle's JRockit does not have a perm size issue and it will work with Geronimo as a service.


Russell Collins
Sr. Software Engineer
McLane Advanced Technology

"Do or do not, there is no try." - Yoda

-----Original Message-----
From: Quintin Beukes [mailto:quintin@...]
Sent: Friday, September 11, 2009 10:57 AM
To: user@...
Subject: Re: Starting Geronimo with More PermGen

Had I had a README I would have read them... well I have the .sh files
but they don't mention it either.

ls bin/
client.bat  cxf-tools.bat  deploy.bat    geronimo.bat    gsh
 jaxws-tools.bat  jpa.jar         setjavaenv.sh  shutdown.sh
startup.bat  stop-server.bat
client.jar  cxf-tools.jar  deployer.jar  geronimo.sh     gsh.bat
 jaxws-tools.jar  server.jar      shutdown.bat   start-server
startup.sh
client.sh   cxf-tools.sh   deploy.sh     gserviceReg.sh  isdeployed.sh
 jaxws-tools.sh   setjavaenv.bat  shutdown.jar   start-server.bat
stop-server

Beyond that I have another question. Inside the .groovy file, how can
I reference environment variables? I'm hoping to only update
setjavaenv.sh to make upgrading easier.

Q

On Fri, Sep 11, 2009 at 5:50 PM, Kevan Miller <kevan.miller@...> wrote:

>
> On Sep 11, 2009, at 9:52 AM, Quintin Beukes wrote:
>
>> Hey,
>>
>> How do I start geronimo with more permgen.
>>
>> I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
>> only gives it to the bootstrap loader, and not the server itself, as
>> can be seen from these processes:
>>
>> 19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
>> -XX:MaxPermSize=256m -jar
>> /opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
>> geronimo/start-server --logfile '/opt/kms/server/geron
>> 19637 pts/12   Sl     0:36  \_
>> /opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
>> -javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
>> -Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200
>
> See
> http://cwiki.apache.org/confluence/display/GMOxDOC22/Runtime+issues#Runtimeissues-JVMarguments
>
> You need to update /etc/rc.d/start-server,default.groovy or add a -J
> parameter.
>
> We could probably do with a bin/README / comments in the .bat/.sh files to
> assist with these issues.
>
> --kevan
>

--
Quintin Beukes

[Quintin Beukes-2]

It doesn't have a permgen issue?

Do you have anything to read on this, as I never really thought it was
an issue. I just thought it had to do with the rate at which classes
are created?

I know there is something about embedded anonymous classes and a
permgen problem relating to these, but more than what I said now I
don't know.

You have links to any articles/blog posts on this?

Q

On Sat, Sep 12, 2009 at 11:26 PM, Russell Collins
<Russell.Collins@...> wrote:

> Unless you are tied into Hotspot (Sun JVM), BEA/Oracle's JRockit does not have a perm size issue and it will work with Geronimo as a service.
>
>
> Russell Collins
> Sr. Software Engineer
> McLane Advanced Technology
>
> "Do or do not, there is no try." - Yoda
>
> -----Original Message-----
> From: Quintin Beukes [mailto:quintin@...]
> Sent: Friday, September 11, 2009 10:57 AM
> To: user@...
> Subject: Re: Starting Geronimo with More PermGen
>
> Had I had a README I would have read them... well I have the .sh files
> but they don't mention it either.
>
> ls bin/
> client.bat  cxf-tools.bat  deploy.bat    geronimo.bat    gsh
>  jaxws-tools.bat  jpa.jar         setjavaenv.sh  shutdown.sh
> startup.bat  stop-server.bat
> client.jar  cxf-tools.jar  deployer.jar  geronimo.sh     gsh.bat
>  jaxws-tools.jar  server.jar      shutdown.bat   start-server
> startup.sh
> client.sh   cxf-tools.sh   deploy.sh     gserviceReg.sh  isdeployed.sh
>  jaxws-tools.sh   setjavaenv.bat  shutdown.jar   start-server.bat
> stop-server
>
> Beyond that I have another question. Inside the .groovy file, how can
> I reference environment variables? I'm hoping to only update
> setjavaenv.sh to make upgrading easier.
>
> Q
>
> On Fri, Sep 11, 2009 at 5:50 PM, Kevan Miller <kevan.miller@...> wrote:
>>
>> On Sep 11, 2009, at 9:52 AM, Quintin Beukes wrote:
>>
>>> Hey,
>>>
>>> How do I start geronimo with more permgen.
>>>
>>> I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
>>> only gives it to the bootstrap loader, and not the server itself, as
>>> can be seen from these processes:
>>>
>>> 19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
>>> -XX:MaxPermSize=256m -jar
>>> /opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
>>> geronimo/start-server --logfile '/opt/kms/server/geron
>>> 19637 pts/12   Sl     0:36  \_
>>> /opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
>>> -javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
>>> -Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200
>>
>> See
>> http://cwiki.apache.org/confluence/display/GMOxDOC22/Runtime+issues#Runtimeissues-JVMarguments
>>
>> You need to update /etc/rc.d/start-server,default.groovy or add a -J
>> parameter.
>>
>> We could probably do with a bin/README / comments in the .bat/.sh files to
>> assist with these issues.
>>
>> --kevan
>>
>
>
>
> --
> Quintin Beukes
>

--
Quintin Beukes

[kevan]

On Sep 13, 2009, at 8:15 AM, Quintin Beukes wrote:

> It doesn't have a permgen issue?
>
> Do you have anything to read on this, as I never really thought it was
> an issue. I just thought it had to do with the rate at which classes
> are created?
>
> I know there is something about embedded anonymous classes and a
> permgen problem relating to these, but more than what I said now I
> don't know.
>
> You have links to any articles/blog posts on this?

I don't have any experience with JRockit. However, IBM's JDK doesn't  
have a notion of PermGen, either. PermGen is an implementation feature  
of Sun's JVM and separates Class objects from normal java objects.  
Presumably, PermGen permits more efficient garbage collection in the  
Sun JVM -- I'm sure non-Sun implementations might argue with that...  
IBM and I assume JRocket allocate Class objects from the Heap.

Either way you your Class metadata is stored in memory. Just a  
question of what pool of storage (normal Heap or specialized PermGen).

--kevan

[Russell Collins]

The main issue with permgen that I have had is with continuous build processes.  At the end of my continuous build process, the artifacts are deployed to the server.  After about 10 or so deployments, the permgen issue arises without fail.  What ends up happening is that the Geronimo server needs to be restarted.  This memory leak/hogging/etc does not happen when Geronimo uses JRockit.


Russell Collins
Sr. Software Engineer
McLane Advanced Technology

"Do or do not, there is no try." - Yoda

-----Original Message-----
From: Quintin Beukes [mailto:quintin@...]
Sent: Sunday, September 13, 2009 7:15 AM
To: user@...
Subject: Re: Starting Geronimo with More PermGen

It doesn't have a permgen issue?

Do you have anything to read on this, as I never really thought it was
an issue. I just thought it had to do with the rate at which classes
are created?

I know there is something about embedded anonymous classes and a
permgen problem relating to these, but more than what I said now I
don't know.

You have links to any articles/blog posts on this?

Q

On Sat, Sep 12, 2009 at 11:26 PM, Russell Collins
<Russell.Collins@...> wrote:

> Unless you are tied into Hotspot (Sun JVM), BEA/Oracle's JRockit does not have a perm size issue and it will work with Geronimo as a service.
>
>
> Russell Collins
> Sr. Software Engineer
> McLane Advanced Technology
>
> "Do or do not, there is no try." - Yoda
>
> -----Original Message-----
> From: Quintin Beukes [mailto:quintin@...]
> Sent: Friday, September 11, 2009 10:57 AM
> To: user@...
> Subject: Re: Starting Geronimo with More PermGen
>
> Had I had a README I would have read them... well I have the .sh files
> but they don't mention it either.
>
> ls bin/
> client.bat  cxf-tools.bat  deploy.bat    geronimo.bat    gsh
>  jaxws-tools.bat  jpa.jar         setjavaenv.sh  shutdown.sh
> startup.bat  stop-server.bat
> client.jar  cxf-tools.jar  deployer.jar  geronimo.sh     gsh.bat
>  jaxws-tools.jar  server.jar      shutdown.bat   start-server
> startup.sh
> client.sh   cxf-tools.sh   deploy.sh     gserviceReg.sh  isdeployed.sh
>  jaxws-tools.sh   setjavaenv.bat  shutdown.jar   start-server.bat
> stop-server
>
> Beyond that I have another question. Inside the .groovy file, how can
> I reference environment variables? I'm hoping to only update
> setjavaenv.sh to make upgrading easier.
>
> Q
>
> On Fri, Sep 11, 2009 at 5:50 PM, Kevan Miller <kevan.miller@...> wrote:
>>
>> On Sep 11, 2009, at 9:52 AM, Quintin Beukes wrote:
>>
>>> Hey,
>>>
>>> How do I start geronimo with more permgen.
>>>
>>> I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
>>> only gives it to the bootstrap loader, and not the server itself, as
>>> can be seen from these processes:
>>>
>>> 19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
>>> -XX:MaxPermSize=256m -jar
>>> /opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
>>> geronimo/start-server --logfile '/opt/kms/server/geron
>>> 19637 pts/12   Sl     0:36  \_
>>> /opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
>>> -javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
>>> -Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200
>>
>> See
>> http://cwiki.apache.org/confluence/display/GMOxDOC22/Runtime+issues#Runtimeissues-JVMarguments
>>
>> You need to update /etc/rc.d/start-server,default.groovy or add a -J
>> parameter.
>>
>> We could probably do with a bin/README / comments in the .bat/.sh files to
>> assist with these issues.
>>
>> --kevan
>>
>
>
>
> --
> Quintin Beukes
>

--
Quintin Beukes

[Quintin Beukes-2]

Doesn't Sun have a GC on the PermGen? Or is it just low priority,
meaning this won't happen on production servers. Even in a the common
production host you deploy a lot. They might be further apart, but
they are many none the less. And these servers are usually solid,
meaning they don't fall over. I will easily reach 10 deployments
without a server restart.

Though on my dev machine I have gone to about 20 deployments (around
there - I just counted the log entries from server start while
pressing page down the whole time and counted 18) before it gave a
permgen. This was in a 2 hour period.

If this is going to be a problem, is JRocket expensive?

Q

On Mon, Sep 14, 2009 at 3:59 PM, Russell Collins
<Russell.Collins@...> wrote:

> The main issue with permgen that I have had is with continuous build processes.  At the end of my continuous build process, the artifacts are deployed to the server.  After about 10 or so deployments, the permgen issue arises without fail.  What ends up happening is that the Geronimo server needs to be restarted.  This memory leak/hogging/etc does not happen when Geronimo uses JRockit.
>
>
> Russell Collins
> Sr. Software Engineer
> McLane Advanced Technology
>
> "Do or do not, there is no try." - Yoda
>
> -----Original Message-----
> From: Quintin Beukes [mailto:quintin@...]
> Sent: Sunday, September 13, 2009 7:15 AM
> To: user@...
> Subject: Re: Starting Geronimo with More PermGen
>
> It doesn't have a permgen issue?
>
> Do you have anything to read on this, as I never really thought it was
> an issue. I just thought it had to do with the rate at which classes
> are created?
>
> I know there is something about embedded anonymous classes and a
> permgen problem relating to these, but more than what I said now I
> don't know.
>
> You have links to any articles/blog posts on this?
>
> Q
>
> On Sat, Sep 12, 2009 at 11:26 PM, Russell Collins
> <Russell.Collins@...> wrote:
>> Unless you are tied into Hotspot (Sun JVM), BEA/Oracle's JRockit does not have a perm size issue and it will work with Geronimo as a service.
>>
>>
>> Russell Collins
>> Sr. Software Engineer
>> McLane Advanced Technology
>>
>> "Do or do not, there is no try." - Yoda
>>
>> -----Original Message-----
>> From: Quintin Beukes [mailto:quintin@...]
>> Sent: Friday, September 11, 2009 10:57 AM
>> To: user@...
>> Subject: Re: Starting Geronimo with More PermGen
>>
>> Had I had a README I would have read them... well I have the .sh files
>> but they don't mention it either.
>>
>> ls bin/
>> client.bat  cxf-tools.bat  deploy.bat    geronimo.bat    gsh
>>  jaxws-tools.bat  jpa.jar         setjavaenv.sh  shutdown.sh
>> startup.bat  stop-server.bat
>> client.jar  cxf-tools.jar  deployer.jar  geronimo.sh     gsh.bat
>>  jaxws-tools.jar  server.jar      shutdown.bat   start-server
>> startup.sh
>> client.sh   cxf-tools.sh   deploy.sh     gserviceReg.sh  isdeployed.sh
>>  jaxws-tools.sh   setjavaenv.bat  shutdown.jar   start-server.bat
>> stop-server
>>
>> Beyond that I have another question. Inside the .groovy file, how can
>> I reference environment variables? I'm hoping to only update
>> setjavaenv.sh to make upgrading easier.
>>
>> Q
>>
>> On Fri, Sep 11, 2009 at 5:50 PM, Kevan Miller <kevan.miller@...> wrote:
>>>
>>> On Sep 11, 2009, at 9:52 AM, Quintin Beukes wrote:
>>>
>>>> Hey,
>>>>
>>>> How do I start geronimo with more permgen.
>>>>
>>>> I tried editing setjavaenv.sh and adding a JAVA_OPTS="...", but this
>>>> only gives it to the bootstrap loader, and not the server itself, as
>>>> can be seen from these processes:
>>>>
>>>> 19571 pts/12   Sl     0:02 /opt/kms/java/jdk5/jre/bin/java -Xmx1024m
>>>> -XX:MaxPermSize=256m -jar
>>>> /opt/kms/server/geronimo/lib/boot/gshell-bootstrap.jar -c
>>>> geronimo/start-server --logfile '/opt/kms/server/geron
>>>> 19637 pts/12   Sl     0:36  \_
>>>> /opt/kms/java/sun-jdk1.5.0_17/jre/bin/java -Xmx512m
>>>> -javaagent:/opt/kms/server/geronimo-2.2-20090908/bin/jpa.jar
>>>> -Dorg.apache.geronimo.home.dir=/opt/kms/server/geronimo-2.2-200
>>>
>>> See
>>> http://cwiki.apache.org/confluence/display/GMOxDOC22/Runtime+issues#Runtimeissues-JVMarguments
>>>
>>> You need to update /etc/rc.d/start-server,default.groovy or add a -J
>>> parameter.
>>>
>>> We could probably do with a bin/README / comments in the .bat/.sh files to
>>> assist with these issues.
>>>
>>> --kevan
>>>
>>
>>
>>
>> --
>> Quintin Beukes
>>
>
>
>
> --
> Quintin Beukes
>

--
Quintin Beukes

[kevan]

On Sep 14, 2009, at 9:59 AM, Russell Collins wrote:

> The main issue with permgen that I have had is with continuous build  
> processes.  At the end of my continuous build process, the artifacts  
> are deployed to the server.  After about 10 or so deployments, the  
> permgen issue arises without fail.  What ends up happening is that  
> the Geronimo server needs to be restarted.  This memory leak/hogging/
> etc does not happen when Geronimo uses JRockit.

Russell,
Is your build process undeploying the artifacts? If your artifacts are  
being undeployed and over time you run out of PermGen, then there's a  
ClassLoader memory leak. Could be in Geronimo or in application/
application library code. Using JRockit, you're moving the memory leak  
from a specialized PermGen space, to a generalized Heap space (which  
is larger and thus you're less likely to run out of memory). Either  
way, you still have a memory leak...

Using Sun JDK, if you run with -XX:+HeapDumpOnOutOfMemoryError you'll  
generate a java_pid<process-id>.hprof file. This file can be used to  
diagnose your problem. We can help, if you collect this data and post  
it to a Jira.

--kevan

[kevan]

On Sep 14, 2009, at 10:07 AM, Quintin Beukes wrote:

> Doesn't Sun have a GC on the PermGen? Or is it just low priority,
> meaning this won't happen on production servers. Even in a the common
> production host you deploy a lot. They might be further apart, but
> they are many none the less. And these servers are usually solid,
> meaning they don't fall over. I will easily reach 10 deployments
> without a server restart.

Yes, PermGen is GC'ed. However, if there is a ClassLoader memory leak,  
then a ClassLoader (and associated classes) can't be GC'ed, even  
though the application has been undeployed.

>
> Though on my dev machine I have gone to about 20 deployments (around
> there - I just counted the log entries from server start while
> pressing page down the whole time and counted 18) before it gave a
> permgen. This was in a 2 hour period.
>
> If this is going to be a problem, is JRocket expensive?

I assume that JRockit stores class meta data in heap space. So, just  
means you have more storage for your class meta data, rather than  
specialized PermGen.

If you recreate your OOME PermGen with -XX:
+HeapDumpOnOutOfMemoryError, we can help diagnose your problem. One of  
these days, I'll write a blog on this...

--kevan

[Quintin Beukes-2]

I'm running with that now, but since the last time i haven't deployed
much without restarting the server. So whenever it does happen I'll
put it on a server and send you the link. Luckily they compress quite
well.

Also, since the bandwidth is going to be used anyway, I can upload it
to a local public FTP, meaning you'll probably get faster download
next time.

Q

On Mon, Sep 14, 2009 at 5:06 PM, Kevan Miller <kevan.miller@...> wrote:

>
> On Sep 14, 2009, at 10:07 AM, Quintin Beukes wrote:
>
>> Doesn't Sun have a GC on the PermGen? Or is it just low priority,
>> meaning this won't happen on production servers. Even in a the common
>> production host you deploy a lot. They might be further apart, but
>> they are many none the less. And these servers are usually solid,
>> meaning they don't fall over. I will easily reach 10 deployments
>> without a server restart.
>
> Yes, PermGen is GC'ed. However, if there is a ClassLoader memory leak, then
> a ClassLoader (and associated classes) can't be GC'ed, even though the
> application has been undeployed.
>
>>
>> Though on my dev machine I have gone to about 20 deployments (around
>> there - I just counted the log entries from server start while
>> pressing page down the whole time and counted 18) before it gave a
>> permgen. This was in a 2 hour period.
>>
>> If this is going to be a problem, is JRocket expensive?
>
> I assume that JRockit stores class meta data in heap space. So, just means
> you have more storage for your class meta data, rather than specialized
> PermGen.
>
> If you recreate your OOME PermGen with -XX:+HeapDumpOnOutOfMemoryError, we
> can help diagnose your problem. One of these days, I'll write a blog on
> this...
>
> --kevan
>
>

--
Quintin Beukes

[Quintin Beukes-2]

Though I have to mention that these dumps will probably be huge, since
I would be running the server for much longer, doing many things in
between (compared to the other one that was just 2 deploys, and about
3 minutes of uptime).

I hope this is fine by you?

Sorry.. bandwidth might not be an issue for you guys. Over here it's
very expensive, so you tend to start treating it like our ancestors
treated food during a dry season... especially when you use a lot of
it already.

Q

On Mon, Sep 14, 2009 at 6:31 PM, Quintin Beukes <quintin@...> wrote:

> I'm running with that now, but since the last time i haven't deployed
> much without restarting the server. So whenever it does happen I'll
> put it on a server and send you the link. Luckily they compress quite
> well.
>
> Also, since the bandwidth is going to be used anyway, I can upload it
> to a local public FTP, meaning you'll probably get faster download
> next time.
>
> Q
>
> On Mon, Sep 14, 2009 at 5:06 PM, Kevan Miller <kevan.miller@...> wrote:
>>
>> On Sep 14, 2009, at 10:07 AM, Quintin Beukes wrote:
>>
>>> Doesn't Sun have a GC on the PermGen? Or is it just low priority,
>>> meaning this won't happen on production servers. Even in a the common
>>> production host you deploy a lot. They might be further apart, but
>>> they are many none the less. And these servers are usually solid,
>>> meaning they don't fall over. I will easily reach 10 deployments
>>> without a server restart.
>>
>> Yes, PermGen is GC'ed. However, if there is a ClassLoader memory leak, then
>> a ClassLoader (and associated classes) can't be GC'ed, even though the
>> application has been undeployed.
>>
>>>
>>> Though on my dev machine I have gone to about 20 deployments (around
>>> there - I just counted the log entries from server start while
>>> pressing page down the whole time and counted 18) before it gave a
>>> permgen. This was in a 2 hour period.
>>>
>>> If this is going to be a problem, is JRocket expensive?
>>
>> I assume that JRockit stores class meta data in heap space. So, just means
>> you have more storage for your class meta data, rather than specialized
>> PermGen.
>>
>> If you recreate your OOME PermGen with -XX:+HeapDumpOnOutOfMemoryError, we
>> can help diagnose your problem. One of these days, I'll write a blog on
>> this...
>>
>> --kevan
>>
>>
>
>
>
> --
> Quintin Beukes
>

--
Quintin Beukes

[Quintin Beukes-2]

> Using Sun JDK, if you run with -XX:+HeapDumpOnOutOfMemoryError you'll
> generate a java_pid<process-id>.hprof file. This file can be used to
> diagnose your problem. We can help, if you collect this data and post it to
> a Jira.

Sorry, I didn't read this well. Should I post it to JIRA even if its >
20MB ? > 50MB? What would you say is your preferential limit, and then
the physical JIRA limit?

Q

[Quintin Beukes-2]

I have read about this problem many times, and can never really
understand that if the PermGen is GC'ed how a memory leak can occur
that would cause a crash?

After all, if all references to a Class is cleared it should be GC'ed, right?

My understanding of how these memory leaks occur in these dynamically
loading containers, like app servers (deployment), servlet containers
(deployment), OSGi (bundles), etc. That means the same class is loaded
many times, and sometimes anonymous/embedded classes in them, and
dynamically generated classes (like proxies in Geronimo). Now, if you
undeploy, won't this clear all references to these classes?

Can you give me an example scenario of a real life memory leak which
causes a leak and eventually PermGen OOME.

Thanks,
Q

On Mon, Sep 14, 2009 at 5:06 PM, Kevan Miller <kevan.miller@...> wrote:

>
> On Sep 14, 2009, at 10:07 AM, Quintin Beukes wrote:
>
>> Doesn't Sun have a GC on the PermGen? Or is it just low priority,
>> meaning this won't happen on production servers. Even in a the common
>> production host you deploy a lot. They might be further apart, but
>> they are many none the less. And these servers are usually solid,
>> meaning they don't fall over. I will easily reach 10 deployments
>> without a server restart.
>
> Yes, PermGen is GC'ed. However, if there is a ClassLoader memory leak, then
> a ClassLoader (and associated classes) can't be GC'ed, even though the
> application has been undeployed.
>
>>
>> Though on my dev machine I have gone to about 20 deployments (around
>> there - I just counted the log entries from server start while
>> pressing page down the whole time and counted 18) before it gave a
>> permgen. This was in a 2 hour period.
>>
>> If this is going to be a problem, is JRocket expensive?
>
> I assume that JRockit stores class meta data in heap space. So, just means
> you have more storage for your class meta data, rather than specialized
> PermGen.
>
> If you recreate your OOME PermGen with -XX:+HeapDumpOnOutOfMemoryError, we
> can help diagnose your problem. One of these days, I'll write a blog on
> this...
>
> --kevan
>
>

--
Quintin Beukes

[Quintin Beukes-2]

I assume that if you have a reference to one of these classes from
outside the application then it can prevent some classes to be GC'ed,
like a static reference you created in some global class? Is this
mostly the case, or is it more specialized?

Q

On Mon, Sep 14, 2009 at 6:58 PM, Quintin Beukes <quintin@...> wrote:

> I have read about this problem many times, and can never really
> understand that if the PermGen is GC'ed how a memory leak can occur
> that would cause a crash?
>
> After all, if all references to a Class is cleared it should be GC'ed, right?
>
> My understanding of how these memory leaks occur in these dynamically
> loading containers, like app servers (deployment), servlet containers
> (deployment), OSGi (bundles), etc. That means the same class is loaded
> many times, and sometimes anonymous/embedded classes in them, and
> dynamically generated classes (like proxies in Geronimo). Now, if you
> undeploy, won't this clear all references to these classes?
>
> Can you give me an example scenario of a real life memory leak which
> causes a leak and eventually PermGen OOME.
>
> Thanks,
> Q
>
> On Mon, Sep 14, 2009 at 5:06 PM, Kevan Miller <kevan.miller@...> wrote:
>>
>> On Sep 14, 2009, at 10:07 AM, Quintin Beukes wrote:
>>
>>> Doesn't Sun have a GC on the PermGen? Or is it just low priority,
>>> meaning this won't happen on production servers. Even in a the common
>>> production host you deploy a lot. They might be further apart, but
>>> they are many none the less. And these servers are usually solid,
>>> meaning they don't fall over. I will easily reach 10 deployments
>>> without a server restart.
>>
>> Yes, PermGen is GC'ed. However, if there is a ClassLoader memory leak, then
>> a ClassLoader (and associated classes) can't be GC'ed, even though the
>> application has been undeployed.
>>
>>>
>>> Though on my dev machine I have gone to about 20 deployments (around
>>> there - I just counted the log entries from server start while
>>> pressing page down the whole time and counted 18) before it gave a
>>> permgen. This was in a 2 hour period.
>>>
>>> If this is going to be a problem, is JRocket expensive?
>>
>> I assume that JRockit stores class meta data in heap space. So, just means
>> you have more storage for your class meta data, rather than specialized
>> PermGen.
>>
>> If you recreate your OOME PermGen with -XX:+HeapDumpOnOutOfMemoryError, we
>> can help diagnose your problem. One of these days, I'll write a blog on
>> this...
>>
>> --kevan
>>
>>
>
>
>
> --
> Quintin Beukes
>

--
Quintin Beukes

[kevan]

On Sep 14, 2009, at 1:29 PM, Quintin Beukes wrote:

> I assume that if you have a reference to one of these classes from
> outside the application then it can prevent some classes to be GC'ed,
> like a static reference you created in some global class? Is this
> mostly the case, or is it more specialized?

That's pretty much it. All it takes is a strong reference to any  
application object (object instance, class, etc). This reference will  
prevent the ClassLoader from being GC'ed. And this in turn will  
prevent all of the Classes loaded by the ClassLoader from being GC'ed.  
These strong references can come from ThreadLocals, statics, non-
static references, or even stack references.

Once you've generated a .hprof file, using post-mortem analysis, you  
can identify the causes of the memory leaks...

--kevan

[kevan]

On Sep 14, 2009, at 12:48 PM, Quintin Beukes wrote:

>> Using Sun JDK, if you run with -XX:+HeapDumpOnOutOfMemoryError you'll
>> generate a java_pid<process-id>.hprof file. This file can be used to
>> diagnose your problem. We can help, if you collect this data and  
>> post it to
>> a Jira.
>
> Sorry, I didn't read this well. Should I post it to JIRA even if its >
> 20MB ? > 50MB? What would you say is your preferential limit, and then
> the physical JIRA limit?

I don't know if there is a file size limit on Jira. If you can get the  
hprof file to a public site, I'll download it. As I think you noted  
earlier, it compresses pretty well...

--kevan

[Quintin Beukes-2]

Phew. I can actually imagine ThreadLocals being a hassle in a
multithreaded system regarding these leaks, as you can't really "check
the value" of the "variable". Especially when the same thread type
might behave differently at times.

Q

On Mon, Sep 14, 2009 at 9:07 PM, Kevan Miller <kevan.miller@...> wrote:

>
> On Sep 14, 2009, at 1:29 PM, Quintin Beukes wrote:
>
>> I assume that if you have a reference to one of these classes from
>> outside the application then it can prevent some classes to be GC'ed,
>> like a static reference you created in some global class? Is this
>> mostly the case, or is it more specialized?
>
> That's pretty much it. All it takes is a strong reference to any application
> object (object instance, class, etc). This reference will prevent the
> ClassLoader from being GC'ed. And this in turn will prevent all of the
> Classes loaded by the ClassLoader from being GC'ed. These strong references
> can come from ThreadLocals, statics, non-static references, or even stack
> references.
>
> Once you've generated a .hprof file, using post-mortem analysis, you can
> identify the causes of the memory leaks...
>
> --kevan
>
>

--
Quintin Beukes

[Russell Collins]

This permgen issue is not just a problem with Geronimo.  Apache Tomcat (6 and 5) has the same issues with the permgen and continuous build integration.  

When deploying to Geronimo, we first undeploy the application then redeploy.  When we deploy to Apache Tomcat, we do not do an undeploy.  Same results.


Russell Collins
Sr. Software Engineer
McLane Advanced Technology

"Do or do not, there is no try." - Yoda

-----Original Message-----
From: Quintin Beukes [mailto:quintin@...]
Sent: Monday, September 14, 2009 3:43 PM
To: user@...
Subject: Re: Starting Geronimo with More PermGen

Phew. I can actually imagine ThreadLocals being a hassle in a
multithreaded system regarding these leaks, as you can't really "check
the value" of the "variable". Especially when the same thread type
might behave differently at times.

Q

On Mon, Sep 14, 2009 at 9:07 PM, Kevan Miller <kevan.miller@...> wrote:

>
> On Sep 14, 2009, at 1:29 PM, Quintin Beukes wrote:
>
>> I assume that if you have a reference to one of these classes from
>> outside the application then it can prevent some classes to be GC'ed,
>> like a static reference you created in some global class? Is this
>> mostly the case, or is it more specialized?
>
> That's pretty much it. All it takes is a strong reference to any application
> object (object instance, class, etc). This reference will prevent the
> ClassLoader from being GC'ed. And this in turn will prevent all of the
> Classes loaded by the ClassLoader from being GC'ed. These strong references
> can come from ThreadLocals, statics, non-static references, or even stack
> references.
>
> Once you've generated a .hprof file, using post-mortem analysis, you can
> identify the causes of the memory leaks...
>
> --kevan
>
>

--
Quintin Beukes