Persistent outrigger doesn't shrink logstore

-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...

Asaf Lahav wrote: Can you expand on "doesn't shrink"? Which files? After how many operations?

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

The logStore files are constantly growing in size.
I would expect the logStore to reduce the size of the repository as entries
are being taken.
On a test I did I write about 5000 entries and then took all the existing
5000 entries.
Yet, two hours later, the logStore files are still a few megabytes in size.

On another test I did, I defined 2 minutes in lease time for 5000 entries
and after 3 minutes tried to take those entries. Non were returned (as
expected) yet, the logStore files were still as big as if the entries are
still stored in there.

I would expect the logStore files to be reduced as entries either expire or
taken.
Currently it seems as if the logStore files constantly increase in size and
may consume all the disk space if given enough time.

Thanks in advance,
Asaf

-----Original Message-----
From: John.McClain@... [mailto:John.McClain@...]
Sent: Tuesday, November 18, 2008 4:22 PM
To: Asaf Lahav
Cc: JINI-USERS@...
Subject: Re: Persistent outrigger doesn't shrink logstore

Asaf Lahav wrote: Can you expand on "doesn't shrink"? Which files? After how many operations?

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

Asaf Lahav wrote:
> The logStore files are constantly growing in size.
>  
There are a bunch of different files logstore uses, which one are you
seeing a problem with? An "ls -l" would be helpful.
The "store reduction" process isn't continuous with logStore (or
snapLogStore) it happen in chunks. By default (in v2.0.2 - BTW, what
version are you using?) individual log files grow until they contain
1000 operations (takes, writes, lease expirations, or event
registrations - don't remember if event deliveries get logged). Once a
log is full the operations are replayed into an object db (PSEPro), and
once the update to the db is committed the log file is removed. We
periodically need to GC the object DB, too, that we do every 5 logs
(again by default).  It isn't clear to me which files you are seeing a
problem with. If you have 10 log files, that is a problem, if you only
have one then the log file part of the process is working. If files
associated with the object database appear to be growing without bound,
it may just be that you haven't done enough operations to trigger the
necessary GC.

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

Hi John,

If "store reduction" is performed every 1000 actions, then 5000 actions were
supposed to trigger the reduction process.
Another problem is that the process holding the java space is constantly
accumulating more and more RAM.

The version I'm using is jini2.1...
Here is the files list where the persistent space logs the entries/actions:

Volume in drive C has no label.
 Volume Serial Number is 28C5-1A79

 Directory of C:\Program Files\jini2_1\installverify\persistenceDir

11/23/2008  10:40 AM    <DIR>          .
11/23/2008  10:40 AM    <DIR>          ..
11/23/2008  10:00 AM        25,006,180 LogStore.10
11/23/2008  10:04 AM        24,938,420 LogStore.11
11/23/2008  10:08 AM        25,057,032 LogStore.12
11/23/2008  10:12 AM        24,799,524 LogStore.13
11/23/2008  10:16 AM        24,542,008 LogStore.14
11/23/2008  10:20 AM        25,108,532 LogStore.15
11/23/2008  10:24 AM        24,851,184 LogStore.16
11/23/2008  10:28 AM        24,747,860 LogStore.17
11/23/2008  10:32 AM        25,161,200 LogStore.18
11/23/2008  10:36 AM        24,799,520 LogStore.19
11/23/2008  09:28 AM        25,045,356 LogStore.2
11/23/2008  10:39 AM        14,640,429 LogStore.20
11/23/2008  09:32 AM        25,165,292 LogStore.3
11/23/2008  09:36 AM        24,700,372 LogStore.4
11/23/2008  09:40 AM        24,700,280 LogStore.5
11/23/2008  09:44 AM        25,371,304 LogStore.6
11/23/2008  09:48 AM        24,592,520 LogStore.7
11/23/2008  09:52 AM        25,152,096 LogStore.8
11/23/2008  09:56 AM        24,609,056 LogStore.9
11/23/2008  10:40 AM                 0 output.txt
11/23/2008  10:36 AM           362,413 Snapshot.175
              21 File(s)    463,350,578 bytes
               2 Dir(s)   1,089,073,152 bytes free


Are there any definition I can perform in order to control the reduction/GC
process?

Any help on this would be very appreciated.

Thanks,
Asaf



-----Original Message-----
From: John.McClain@... [mailto:John.McClain@...]
Sent: Thursday, November 20, 2008 4:01 PM
To: Asaf Lahav
Cc: JINI-USERS@...
Subject: Re: Persistent outrigger doesn't shrink logstore

Asaf Lahav wrote:
> The logStore files are constantly growing in size.
>  
There are a bunch of different files logstore uses, which one are you
seeing a problem with? An "ls -l" would be helpful.

> I would expect the logStore to reduce the size of the repository as
entries
> are being taken.
> On a test I did I write about 5000 entries and then took all the existing
> 5000 entries.
> Yet, two hours later, the logStore files are still a few megabytes in
size.
>
> On another test I did, I defined 2 minutes in lease time for 5000 entries
> and after 3 minutes tried to take those entries. Non were returned (as
> expected) yet, the logStore files were still as big as if the entries are
> still stored in there.
>
> I would expect the logStore files to be reduced as entries either expire
or
> taken.
> Currently it seems as if the logStore files constantly increase in size
and
> may consume all the disk space if given enough time.
>  
The "store reduction" process isn't continuous with logStore (or
snapLogStore) it happen in chunks. By default (in v2.0.2 - BTW, what
version are you using?) individual log files grow until they contain
1000 operations (takes, writes, lease expirations, or event
registrations - don't remember if event deliveries get logged). Once a
log is full the operations are replayed into an object db (PSEPro), and
once the update to the db is committed the log file is removed. We
periodically need to GC the object DB, too, that we do every 5 logs
(again by default).  It isn't clear to me which files you are seeing a
problem with. If you have 10 log files, that is a problem, if you only
have one then the log file part of the process is working. If files
associated with the object database appear to be growing without bound,
it may just be that you haven't done enough operations to trigger the
necessary GC.

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

Asaf Lahav wrote:
> Hi John,
>
> If "store reduction" is performed every 1000 actions, then 5000 actions were
> supposed to trigger the reduction process.

It wasn't clear from your messages if the logs were growing or the
object database, 5000 should be enough to see the logs GCed, but it
might not be quite enough to see the object database be GCed.

[as an aside, it turns out you aren't using logstore, but snaplogstore,
which doesn't use PSEPro but just std. Java object serialization - so
the object DB GC evey 5 logs isn't an issue....]

Looking at the number of log files, I would have guessed that the log
processing thread died, is deadlocked, or fallen behind, but that
doesn't seem consistent with the Snapshot file having an update time
later than the update time of the last log file (suppose that could
happen if the processing thread was really bogged down, but I would
expect the log process thread to catch up and that doesn't seem to be
happening in your case). I am also confused by the store being on the
175th snapshot - I don't have the source handy, but that seems
inconsistent with the number of logs.

I assume there no messages in stdout or stderr?

I think the next step would be to turn on logging, be a while, so I will
need to check the code tomorrow what the right logger would be.

> Are there any definition I can perform in order to control the reduction/GC
> process?

There are a few configuration options that control the snaplogstore.
They are listed in the top level java doc for the com.sun.jini.outrigger
package - but my guess is none of them are relevant here. Can you post
your complete Outrigger config file?

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

Hi John,
Thank you for helping.
Here are the configuration files:
startAll.config:
---------------------------------------------------------------
import com.sun.jini.start.NonActivatableServiceDescriptor;
import com.sun.jini.start.ServiceDescriptor;
import com.sun.jini.config.ConfigUtil;

com.sun.jini.start {
    private static policy = "C:\\Program
Files\\jini2_1\\installverify\\support\\jsk-all.policy";
    private static host = ConfigUtil.getHostName();
    private static port = "8081";
    private static jskdl = " http://" + host + ":" + port + "/jsk-dl.jar";

    serviceDescriptors = new ServiceDescriptor[]{
        new NonActivatableServiceDescriptor(
            "",
            policy,
            "C:\\Program Files\\jini2_1\\lib\\classserver.jar",
            "com.sun.jini.tool.ClassServer",
            new String[]{"-port", port, "-dir",
                "C:\\Program Files\\jini2_1\\lib-dl",
                "-verbose"}),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/browser-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\browser.jar",
            "com.sun.jini.example.browser.Browser",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\browser.config" }),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/reggie-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\reggie.jar",
            "com.sun.jini.reggie.TransientRegistrarImpl",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\reggie.config" }),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/fiddler-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\fiddler.jar",
            "com.sun.jini.fiddler.TransientFiddlerImpl",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\fiddler.config" }),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/mahalo-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\mahalo.jar",
            "com.sun.jini.mahalo.TransientMahaloImpl",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\mahalo.config" }),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/mercury-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\mercury.jar",
            "com.sun.jini.mercury.TransientMercuryImpl",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\mercury.config" }),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/norm-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\norm.jar",
            "com.sun.jini.norm.TransientNormServerImpl",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\norm.config" }),

        new NonActivatableServiceDescriptor(
            "http://" + host + ":" + port + "/outrigger-dl.jar" + jskdl,
            policy,
            "C:\\Program Files\\jini2_1\\lib\\outrigger.jar",
            "com.sun.jini.outrigger.PersistentOutriggerImpl",
            new String[] { "C:\\Program
Files\\jini2_1\\installverify\\support\\outrigger.config" })

    };
}




Outrigger.config:
--------------------------------------------------------
com.sun.jini.outrigger {
    initialLookupGroups = new String[] { System.getProperty("user.name") +
"InstallVerifyGroup" };
        persistenceDirectory = "C:/Program
Files/jini2_1/installverify/persistenceDir";
        store = new com.sun.jini.outrigger.snaplogstore.LogStore(this);
}




-----Original Message-----
From: John.McClain@... [mailto:John.McClain@...]
Sent: Tuesday, November 25, 2008 2:47 AM
To: Asaf Lahav
Cc: JINI-USERS@...
Subject: Re: Persistent outrigger doesn't shrink logstore

Asaf Lahav wrote:
> Hi John,
>
> If "store reduction" is performed every 1000 actions, then 5000 actions
were
> supposed to trigger the reduction process.

It wasn't clear from your messages if the logs were growing or the
object database, 5000 should be enough to see the logs GCed, but it
might not be quite enough to see the object database be GCed.

[as an aside, it turns out you aren't using logstore, but snaplogstore,
which doesn't use PSEPro but just std. Java object serialization - so
the object DB GC evey 5 logs isn't an issue....]

> Another problem is that the process holding the java space is constantly
> accumulating more and more RAM.
>
> The version I'm using is jini2.1...
> Here is the files list where the persistent space logs the
entries/actions:
Looking at the number of log files, I would have guessed that the log
processing thread died, is deadlocked, or fallen behind, but that
doesn't seem consistent with the Snapshot file having an update time
later than the update time of the last log file (suppose that could
happen if the processing thread was really bogged down, but I would
expect the log process thread to catch up and that doesn't seem to be
happening in your case). I am also confused by the store being on the
175th snapshot - I don't have the source handy, but that seems
inconsistent with the number of logs.

I assume there no messages in stdout or stderr?

I think the next step would be to turn on logging, be a while, so I will
need to check the code tomorrow what the right logger would be.

> Are there any definition I can perform in order to control the
reduction/GC
> process?

There are a few configuration options that control the snaplogstore.
They are listed in the top level java doc for the com.sun.jini.outrigger
package - but my guess is none of them are relevant here. Can you post
your complete Outrigger config file?

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

John McClain wrote:
> I assume there no messages in stdout or stderr?
>
> I think the next step would be to turn on logging, be a while, so I
> will need to check the code tomorrow what the right logger would be.
The relevant logger is "com.sun.jini.outrigger.store", need at least the
FINE level to get anything useful here, not sure it will actually have
the right info (and looks like it will generate a lot more detail than
what we really need here), but is probably the right place to start.  Do
you have some test case you can re-run on a fresh space?

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...