[jira] Created: (FOR-1031) Switch to EHCache for Locationmap Cache

Switch to EHCache for Locationmap Cache
---------------------------------------

                 Key: FOR-1031
                 URL: https://issues.apache.org/jira/browse/FOR-1031
             Project: Forrest
          Issue Type: Improvement
          Components: Locationmap
            Reporter: Tim Williams


Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Williams updated FOR-1031:
------------------------------

    Attachment: lm_cache.diff

Remove the simple all-or-nothing map cache for ehache.

> Switch to EHCache for Locationmap Cache
> ---------------------------------------
>
>                 Key: FOR-1031
>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>             Project: Forrest
>          Issue Type: Improvement
>          Components: Locationmap
>            Reporter: Tim Williams
>         Attachments: lm_cache.diff
>
>
> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Tim Williams]

Anyone had a chance to check this out?  I'll go ahead and apply it but
since some of the earlier problems only occurred under stress
conditions, I wanted to give an opportunity for others to test too.
Thanks,
--tim

On Fri, Jan 16, 2009 at 10:51 PM, Tim Williams (JIRA) <jira@...> wrote:

>
>     [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Tim Williams updated FOR-1031:
> ------------------------------
>
>    Attachment: lm_cache.diff
>
> Remove the simple all-or-nothing map cache for ehache.
>
>> Switch to EHCache for Locationmap Cache
>> ---------------------------------------
>>
>>                 Key: FOR-1031
>>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>>             Project: Forrest
>>          Issue Type: Improvement
>>          Components: Locationmap
>>            Reporter: Tim Williams
>>         Attachments: lm_cache.diff
>>
>>
>> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

[David Crossley]

Tim Williams wrote:
> Anyone had a chance to check this out?  I'll go ahead and apply it but
> since some of the earlier problems only occurred under stress
> conditions, I wanted to give an opportunity for others to test too.

I made some time to try it today. However i don't know what
is a good test nor what to look for. Anyway i tried building
the Forrest docs at "site-author" with the level for locationmap
logging set to DEBUG.

Before it took about 4 minutes to build.
After adding your LM patch it takes about 2.5 minutes.

Do you have a reference to the "stress conditions" that
you refer to above?

Do we need a default ehcache configuration file?
Where does it get placed?

Does that configuration file affect the main Cocoon
cache which has some configuartion via main/webapp/WEB-INF/cocoon.xconf
and main/webapp/WEB-INF/properties/core.properties ?

Sorry for more questions rather than much help.

-David

> >     [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> >
> > Tim Williams updated FOR-1031:
> > ------------------------------
> >
> >    Attachment: lm_cache.diff
> >
> > Remove the simple all-or-nothing map cache for ehache.
> >
> >> Switch to EHCache for Locationmap Cache
> >> ---------------------------------------
> >>
> >>                 Key: FOR-1031
> >>                 URL: https://issues.apache.org/jira/browse/FOR-1031
> >>             Project: Forrest
> >>          Issue Type: Improvement
> >>          Components: Locationmap
> >>            Reporter: Tim Williams
> >>         Attachments: lm_cache.diff
> >>
> >>
> >> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
>

[Tim Williams]

On Fri, Feb 20, 2009 at 1:13 AM, David Crossley <crossley@...> wrote:

> Tim Williams wrote:
>> Anyone had a chance to check this out?  I'll go ahead and apply it but
>> since some of the earlier problems only occurred under stress
>> conditions, I wanted to give an opportunity for others to test too.
>
> I made some time to try it today. However i don't know what
> is a good test nor what to look for. Anyway i tried building
> the Forrest docs at "site-author" with the level for locationmap
> logging set to DEBUG.
>
> Before it took about 4 minutes to build.
> After adding your LM patch it takes about 2.5 minutes.

I'm surprised the locationmap is being used on enough duplicate
requests to make that much of a difference in static mode, that's
pretty good.

> Do you have a reference to the "stress conditions" that
> you refer to above?

Yeah, I was specifically referring to Thorsten's synchronization
problems before.  Our old cache was a bit of a hack and Thorsten
patched it by synchronizing certain accesses.  We knew then that it
was a short-term solution. He's got tests that do run numerous threads
and cause the problem he was seeing, so I thought it'd be helpful to
use those tests for this.  It should be much more capable in a
multi-threaded environment.

> Do we need a default ehcache configuration file?
> Where does it get placed?

Yeah, we should.  I'll try to get to that too, likely in the source
directory but as long as it makes it on the classpath all is well.

> Does that configuration file affect the main Cocoon
> cache which has some configuartion via main/webapp/WEB-INF/cocoon.xconf
> and main/webapp/WEB-INF/properties/core.properties ?

No, it shouldn't. Cocoon, AFAIK has created a 'store' abstraction to
their cache, which happens to be backed by ehcache.  Those properties,
cocoon.xconf in particular, are used for programmatically creating
their store.  In other words, I *think* even if an ehcache file were
on the classpath, they'd effectively ignore it for anything but
default values - which effectively means it'd be totally ignored.  A
long way of saying, I think we're safe - but I'm going from memory on
much of this so maybe someone with deeper Cocoon knowledge could
confirm this though?

> Sorry for more questions rather than much help.

No, this was good, Thanks David
--tim

> -David
>
>> >     [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>> >
>> > Tim Williams updated FOR-1031:
>> > ------------------------------
>> >
>> >    Attachment: lm_cache.diff
>> >
>> > Remove the simple all-or-nothing map cache for ehache.
>> >
>> >> Switch to EHCache for Locationmap Cache
>> >> ---------------------------------------
>> >>
>> >>                 Key: FOR-1031
>> >>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>> >>             Project: Forrest
>> >>          Issue Type: Improvement
>> >>          Components: Locationmap
>> >>            Reporter: Tim Williams
>> >>         Attachments: lm_cache.diff
>> >>
>> >>
>> >> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.
>> >
>> > --
>> > This message is automatically generated by JIRA.
>> > -
>> > You can reply to this email to add a comment to the issue online.
>> >
>> >
>>
>

[David Crossley]

Tim Williams wrote:

> David Crossley wrote:
> > Tim Williams wrote:
>
> >> Anyone had a chance to check this out?  I'll go ahead and apply it but
> >> since some of the earlier problems only occurred under stress
> >> conditions, I wanted to give an opportunity for others to test too.
> >
> > I made some time to try it today. However i don't know what
> > is a good test nor what to look for. Anyway i tried building
> > the Forrest docs at "site-author" with the level for locationmap
> > logging set to DEBUG.
> >
> > Before it took about 4 minutes to build.
> > After adding your LM patch it takes about 2.5 minutes.
>
> I'm surprised the locationmap is being used on enough duplicate
> requests to make that much of a difference in static mode, that's
> pretty good.

I tested again, and confirmed the speed improvement.

The locationmap is very central to Forrest's operations now.

Perhaps this is an indicator that we should look deeper
at the locationmap requests. There is something in our
mail archives describing the last time when we used
the Cocoon Sitemap Profiler:
http://forrest.apache.org/howto-dev.html#debug

-David

    [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682928#action_12682928 ]

David Crossley commented on FOR-1031:
-------------------------------------

See some discussion on the dev list in response to this issue:
http://thread.gmane.org/gmane.text.xml.forrest.devel/24814/focus=26924

> Switch to EHCache for Locationmap Cache
> ---------------------------------------
>
>                 Key: FOR-1031
>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>             Project: Forrest
>          Issue Type: Improvement
>          Components: Locationmap
>            Reporter: Tim Williams
>         Attachments: lm_cache.diff
>
>
> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682930#action_12682930 ]

David Crossley commented on FOR-1031:
-------------------------------------

I am seeing some issues now.

In the PDF output plugin, building its own little website.
(Also saw this with site-author.)

Did 'forrest clean; forrest run' all okay.

Then did 'forrest' to build the "site".
Got one error:
--------
ERROR - cocoon-ehcache-1Cache: Could not read disk store element for key PK_R-resource-file:///svn/asf/forrest/main/webapp/skins/pelt/images/instruction_arrow.png. Error was invalid stream header
java.io.StreamCorruptedException: invalid stream header
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:764)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:277)
        at net.sf.ehcache.store.DiskStore$1.<init>(DiskStore.java:272)
        at net.sf.ehcache.store.DiskStore.loadElementFromDiskElement(DiskStore.java:267)
...
--------

Then did 'forrest run' again and request index.html
Got two errors:
--------
ERROR - cocoon-ehcache-1Cache: Could not read disk store element for key PK_G-file-file:///svn/asf/forrest/plugins/org.apache.forrest.plugin.output.pdf/src/documentation/content/xdocs/tabs.xml_T-xinclude-XInclude. Error was invalid stream header
...
ERROR - cocoon-ehcache-1Cache: Could not read disk store element for key PK_G-file-cocoon://abs-linkmap?pipelinehash=-4182989005625245977_T-xslt-file:///svn/asf/forrest/main/webapp/resources/stylesheets/relativize-linkmap.xsl;site-root=;path=index.html. Error was invalid stream header
--------

Subsequent requests for index.html yielded no errors.
Requests for other pages also yielded no errors.

Request index.pdf
Got one error:
--------
ERROR - cocoon-ehcache-1Cache: Could not read disk store element for key PK_G-module-properties. Error was invalid stream header
--------

Then did 'forrest' again.
No errors.

Then did 'forrest run' again.
Request index.html
Got one error:
--------
ERROR - cocoon-ehcache-1Cache: Could not read disk store element for key PK_G-file-file:///svn/asf/forrest/plugins/org.apache.forrest.plugin.output.pdf/src/documentation/content/xdocs/tabs.xml_T-xinclude-XInclude. Error was invalid stream header
--------

Subsequent requests for various docs are all okay.

Then did 'forrest' again.
No errors.

Then did 'forrest' again, yes did 'forrest' again.
Got one error:
--------
ERROR - cocoon-ehcache-1Cache: Could not read disk store element for key PK_G-file-file:///svn/asf/forrest/plugins/org.apache.forrest.plugin.output.pdf/src/documentation/content/xdocs/tabs.xml_T-xinclude-XInclude. Error was invalid stream header
--------

Same on every subsequest 'forrest'.

Can repeat all that behaviour after another 'forrest clean'.

For each session after the first, there is only one broken cache request.
Always tabs.xml_T-xinclude-XInclude.
The first 'forrest' after a 'forrest run' is okay. Subsequent invocations have that single cache error.

> Switch to EHCache for Locationmap Cache
> ---------------------------------------
>
>                 Key: FOR-1031
>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>             Project: Forrest
>          Issue Type: Improvement
>          Components: Locationmap
>            Reporter: Tim Williams
>         Attachments: lm_cache.diff
>
>
> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[David Crossley]

I am seeing some issues now.

Described it in a comment to FOR-1031.

-David

    [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688196#action_12688196 ]

David Crossley commented on FOR-1031:
-------------------------------------

Fixed those unrelated issues. See Tim's comments at:
 Re: Cache error invalid stream header
 http://thread.gmane.org/gmane.text.xml.forrest.devel/26992

i.e. Needed to remove Cocoon's cache:
 cd $java.io.tmpdir (e.g. /tmp)
 rm cocoon-ehcache*

> Switch to EHCache for Locationmap Cache
> ---------------------------------------
>
>                 Key: FOR-1031
>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>             Project: Forrest
>          Issue Type: Improvement
>          Components: Locationmap
>            Reporter: Tim Williams
>         Attachments: lm_cache.diff
>
>
> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/FOR-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Williams resolved FOR-1031.
-------------------------------

       Resolution: Fixed
    Fix Version/s: 0.9-dev
         Assignee: Tim Williams

r834295 applies the patch that adds ehcache and the config file to get rid of the warning.  We'll consider this done and address any side effects in their own issue.

> Switch to EHCache for Locationmap Cache
> ---------------------------------------
>
>                 Key: FOR-1031
>                 URL: https://issues.apache.org/jira/browse/FOR-1031
>             Project: Forrest
>          Issue Type: Improvement
>          Components: Locationmap
>            Reporter: Tim Williams
>            Assignee: Tim Williams
>             Fix For: 0.9-dev
>
>         Attachments: lm_cache.diff
>
>
> Should be an easy switch and get much better cache support/configuration options.  Expiry of cache can be per-element vs. the current implementation where it expires in its entirey.  It will make the LM stuff much cleaner too.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.