EclipseLink cache configuration for Struts plus Spring application

[sonavor]

I have looked through a number of the JPA caching posts on the forum but I haven't found a solution to what my problem is.

My application is a web application running on Tomcat (5.5) and using MySql as the database.  The web application framework is Struts and the business layer is managed by Spring.  In this web application most users will be running queries for data so no transactions will be going on and I would like speed for these tasks.  Data entry will be going on from time to time by some administrator screens where new data is entered and/or existing data updated.

The application is performing pretty well so far (half-way through development).  The problem I am having is I am seeing a caching problem(?) when new data is entered through the admin screen followed by a user finding the data in a query.

So an admin user enters several new records.  Then an application user executes a search where the new data is included in the result set.  When the result page attempts to display attributes of the search result data I am seeing some errors due to the newly entered records having nulls for some their attributes.  The missing data is in the database but is just not being returned with the search results.  

Is this more of a fetch strategy problem?  The missing attribute data will show up if I use a screen that specifically finds and returns the individual record.   After that the searches then also return all of the attributes.  Or is this a problem with my cache settings?  I have tried setting the caching in the persistence.xml with and without the entry -
<property name="eclipselink.cache.shared.default" value="false"/>
and I don't see any difference in performance or my error.

One other thing about the search not immediately seeing all of the entity attributes - If I reboot my server or redeploy my application the search will find all of the data it needs.  It is only immediately after the new entity is created in the database that the JPA search query is not finding the data.

My search queries performing the search are both queries like -

Query q = em.createQuery(qSB.toString());
List<MyEntity> eList = q.getResultList();

and

ReadAllQuery readAllQuery = new ReadAllQuery(MyEntity.class);
....set variables...
javax.persistence.Query jpaquery = ((JpaEntityManager) em.getDelegate()).createQuery(readAllQuery);
...set variables....
List<MyEntity> results = jpaquery.getResultList();

Thanks for any help with a good solution to provide fast searches yet be able to sync with new entity additions.

-sonavor

[James Sutherland]

If you get the same behavior with caching off, then it does not seem to be a caching issue, but is very odd, I cannot think of what could cause this.

Try turning on logging, and see what SQL is being generated.  Also try setting refreshIdentityMapResult() in the query.  Also include the full code for your querym what is "....set variables..." doing?

sonavor wrote:

I have looked through a number of the JPA caching posts on the forum but I haven't found a solution to what my problem is.

My application is a web application running on Tomcat (5.5) and using MySql as the database.  The web application framework is Struts and the business layer is managed by Spring.  In this web application most users will be running queries for data so no transactions will be going on and I would like speed for these tasks.  Data entry will be going on from time to time by some administrator screens where new data is entered and/or existing data updated.

The application is performing pretty well so far (half-way through development).  The problem I am having is I am seeing a caching problem(?) when new data is entered through the admin screen followed by a user finding the data in a query.

So an admin user enters several new records.  Then an application user executes a search where the new data is included in the result set.  When the result page attempts to display attributes of the search result data I am seeing some errors due to the newly entered records having nulls for some their attributes.  The missing data is in the database but is just not being returned with the search results.  

Is this more of a fetch strategy problem?  The missing attribute data will show up if I use a screen that specifically finds and returns the individual record.   After that the searches then also return all of the attributes.  Or is this a problem with my cache settings?  I have tried setting the caching in the persistence.xml with and without the entry -
<property name="eclipselink.cache.shared.default" value="false"/>
and I don't see any difference in performance or my error.

One other thing about the search not immediately seeing all of the entity attributes - If I reboot my server or redeploy my application the search will find all of the data it needs.  It is only immediately after the new entity is created in the database that the JPA search query is not finding the data.

My search queries performing the search are both queries like -

Query q = em.createQuery(qSB.toString());
List<MyEntity> eList = q.getResultList();

and

ReadAllQuery readAllQuery = new ReadAllQuery(MyEntity.class);
....set variables...
javax.persistence.Query jpaquery = ((JpaEntityManager) em.getDelegate()).createQuery(readAllQuery);
...set variables....
List<MyEntity> results = jpaquery.getResultList();

Thanks for any help with a good solution to provide fast searches yet be able to sync with new entity additions.

-sonavor

[sonavor]

It is odd.
I turned more logging on but there is too much information to put in here.

Here is some more information about what is going on.  I use and web application admin screen to add new media definition records to a MySql table.  A new media definition includes several child, media item, records.  Two of those child, media item records represent a thumbnail image and a preview image.  In my latest test I added three new media definitions.  Those records go into the database as usual without a hitch.  For each of those three definitions I placed the necessary image (JPG) files where the web app can find them.  Now here is where the odd behavior occurs.  After those inserts I run a query from my web application search screen that finds the new records.  Of those three inserted media definition records, the first one does not return the thumbnail.  If I drill into the media definition where the application performs an EntityManager find(MediaListing.class, mediaId) then the thumbnail appears...and from then on, appears on a search.  When I say "drill into" - what I mean is that the search finds the media definition record but it does not seem to have all of the necessary information such as the filename of the thumbnail image.  It isn't able to retrieve that child attribute.

To double check I ran another test where I entered three new media definitions.  Same thing happens - the first record I entered cannot find the thumbnail on the search until I drill into the record.  

The JPA query I am running is this -
            Integer subjectId = searchDto.getMediaSubject().getSubjectId();
            StringBuffer qSB = new StringBuffer();
            qSB.append("SELECT object(m) FROM MediaListing as m JOIN m.mediaSubject s WHERE s.subjectId = :argSubjectId");
            Query q = em.createQuery(qSB.toString());
            q.setParameter("argSubjectId", subjectId);
            q.setFirstResult(startRow);
            q.setMaxResults(rowCount);
            List<MediaListing> mList = q.getResultList();
            if (mList != null) {
                log("findMediaSearchByPaged...found " + mList.size() + "MediaListing records ...");
                for (Iterator i = mList.iterator(); i.hasNext();) {
                    MediaListing ml = (MediaListing) i.next();
                    ml.getMediaType();
                    ml.getMediaRepositoryItemCollection();
                }
            }

I added the ml.getMediaType() and ml.getMediaRepositoryItemCollection() lines to force the entity manager to retrieve the MediaListing (ml) record child records because of the lazy fetch.  I know I can set the fetch strategy with an annotation but I put those lines in there just to see if it would force the child attributes to be received.  In this case it doesn't seem to help.

The EclipseLink output for the query where I retrieve the media items is like this (I run a count first because I paginate the searches) -
[EL Fine]: 2009-09-29 10:52:02.109--ServerSession(4729773)--Connection(14151320)--Thread(Thread[TP-Processor3,5,main])--SELECT COUNT(t0.MEDIA_ID) FROM MEDIA_LISTING t0, MEDIA_SUBJECT t1 WHERE ((t1.SUBJECT_ID = ?) AND (t1.SUBJECT_ID = t0.MEDIA_SUBJECT_ID))
        bind => [518]

followed by -

[EL Fine]: 2009-09-29 10:52:02.202--ServerSession(4729773)--Connection(26375417)--Thread(Thread[TP-Processor3,5,main])--SELECT t1.MEDIA_ID AS MEDIA_ID1, t1.LAST_UPDATE AS LAST_UPDATE2, t1.MEDIA_CODE AS MEDIA_CODE3, t1.MEDIA_TITLE AS MEDIA_TITLE6, t1.MEDIA_DESCRIPTION AS MEDIA_DESCRIPTION7, t1.MEDIA_TYPE_ID AS MEDIA_TYPE_ID11, t1.AUTHOR_ID AS AUTHOR_ID12, t1.MEDIA_SUBJECT_ID AS MEDIA_SUBJECT_ID13, t1.MEDIA_CATEGORY_ID AS MEDIA_CATEGORY_ID14, t1.MEDIA_FORMAT_ID AS MEDIA_FORMAT_ID15 FROM MEDIA_SUBJECT t0, MEDIA_LISTING t1 WHERE ((t0.SUBJECT_ID = ?) AND (t0.SUBJECT_ID = t1.MEDIA_SUBJECT_ID)) LIMIT ?, ?
        bind => [518, 0, 16]

I have added logic to my web page build for the search results to detect any null attribute values for the returned MediaListing objects and that is preventing any page blow-ups.  In this case where one of the media records is not able to retrieve the thumbnail file image I detect that and substitute an "image not found" JPG.  

I can't use the refreshIdentityMapResult() method on the query I am running now because it is just a basic Query type object, not a ReadAllQuery.

-sonavor









If you get the same behavior with caching off, then it does not seem to be a caching issue, but is very odd, I cannot think of what could cause this.

Try turning on logging, and see what SQL is being generated.  Also try setting refreshIdentityMapResult() in the query.  Also include the full code for your querym what is "....set variables..." doing?

sonavor wrote:

I have looked through a number of the JPA caching posts on the forum but I haven't found a solution to what my problem is.

My application is a web application running on Tomcat (5.5) and using MySql as the database.  The web application framework is Struts and the business layer is managed by Spring.  In this web application most users will be running queries for data so no transactions will be going on and I would like speed for these tasks.  Data entry will be going on from time to time by some administrator screens where new data is entered and/or existing data updated.

The application is performing pretty well so far (half-way through development).  The problem I am having is I am seeing a caching problem(?) when new data is entered through the admin screen followed by a user finding the data in a query.

So an admin user enters several new records.  Then an application user executes a search where the new data is included in the result set.  When the result page attempts to display attributes of the search result data I am seeing some errors due to the newly entered records having nulls for some their attributes.  The missing data is in the database but is just not being returned with the search results.  

Is this more of a fetch strategy problem?  The missing attribute data will show up if I use a screen that specifically finds and returns the individual record.   After that the searches then also return all of the attributes.  Or is this a problem with my cache settings?  I have tried setting the caching in the persistence.xml with and without the entry -
<property name="eclipselink.cache.shared.default" value="false"/>
and I don't see any difference in performance or my error.

One other thing about the search not immediately seeing all of the entity attributes - If I reboot my server or redeploy my application the search will find all of the data it needs.  It is only immediately after the new entity is created in the database that the JPA search query is not finding the data.

My search queries performing the search are both queries like -

Query q = em.createQuery(qSB.toString());
List<MyEntity> eList = q.getResultList();

and

ReadAllQuery readAllQuery = new ReadAllQuery(MyEntity.class);
....set variables...
javax.persistence.Query jpaquery = ((JpaEntityManager) em.getDelegate()).createQuery(readAllQuery);
...set variables....
List<MyEntity> results = jpaquery.getResultList();

Thanks for any help with a good solution to provide fast searches yet be able to sync with new entity additions.

-sonavor

[James Sutherland]

Is it just the thumbnail that is null, or other mapped attributes?  Are you mapping the thumbnail to the database, or not storing it in the database?  If it is transient, then how do you expect it to come back when reading from the database?

Otherwise, it may have something to do with the first/maxResults, try removing this to confirm if this is causing the issue.  Also you can set the query to refresh using the query hint, "eclipselink.refresh"="true".

sonavor wrote:

It is odd.
I turned more logging on but there is too much information to put in here.

Here is some more information about what is going on.  I use and web application admin screen to add new media definition records to a MySql table.  A new media definition includes several child, media item, records.  Two of those child, media item records represent a thumbnail image and a preview image.  In my latest test I added three new media definitions.  Those records go into the database as usual without a hitch.  For each of those three definitions I placed the necessary image (JPG) files where the web app can find them.  Now here is where the odd behavior occurs.  After those inserts I run a query from my web application search screen that finds the new records.  Of those three inserted media definition records, the first one does not return the thumbnail.  If I drill into the media definition where the application performs an EntityManager find(MediaListing.class, mediaId) then the thumbnail appears...and from then on, appears on a search.  When I say "drill into" - what I mean is that the search finds the media definition record but it does not seem to have all of the necessary information such as the filename of the thumbnail image.  It isn't able to retrieve that child attribute.

To double check I ran another test where I entered three new media definitions.  Same thing happens - the first record I entered cannot find the thumbnail on the search until I drill into the record.  

The JPA query I am running is this -
            Integer subjectId = searchDto.getMediaSubject().getSubjectId();
            StringBuffer qSB = new StringBuffer();
            qSB.append("SELECT object(m) FROM MediaListing as m JOIN m.mediaSubject s WHERE s.subjectId = :argSubjectId");
            Query q = em.createQuery(qSB.toString());
            q.setParameter("argSubjectId", subjectId);
            q.setFirstResult(startRow);
            q.setMaxResults(rowCount);
            List<MediaListing> mList = q.getResultList();
            if (mList != null) {
                log("findMediaSearchByPaged...found " + mList.size() + "MediaListing records ...");
                for (Iterator i = mList.iterator(); i.hasNext();) {
                    MediaListing ml = (MediaListing) i.next();
                    ml.getMediaType();
                    ml.getMediaRepositoryItemCollection();
                }
            }

I added the ml.getMediaType() and ml.getMediaRepositoryItemCollection() lines to force the entity manager to retrieve the MediaListing (ml) record child records because of the lazy fetch.  I know I can set the fetch strategy with an annotation but I put those lines in there just to see if it would force the child attributes to be received.  In this case it doesn't seem to help.

The EclipseLink output for the query where I retrieve the media items is like this (I run a count first because I paginate the searches) -
[EL Fine]: 2009-09-29 10:52:02.109--ServerSession(4729773)--Connection(14151320)--Thread(Thread[TP-Processor3,5,main])--SELECT COUNT(t0.MEDIA_ID) FROM MEDIA_LISTING t0, MEDIA_SUBJECT t1 WHERE ((t1.SUBJECT_ID = ?) AND (t1.SUBJECT_ID = t0.MEDIA_SUBJECT_ID))
        bind => [518]

followed by -

[EL Fine]: 2009-09-29 10:52:02.202--ServerSession(4729773)--Connection(26375417)--Thread(Thread[TP-Processor3,5,main])--SELECT t1.MEDIA_ID AS MEDIA_ID1, t1.LAST_UPDATE AS LAST_UPDATE2, t1.MEDIA_CODE AS MEDIA_CODE3, t1.MEDIA_TITLE AS MEDIA_TITLE6, t1.MEDIA_DESCRIPTION AS MEDIA_DESCRIPTION7, t1.MEDIA_TYPE_ID AS MEDIA_TYPE_ID11, t1.AUTHOR_ID AS AUTHOR_ID12, t1.MEDIA_SUBJECT_ID AS MEDIA_SUBJECT_ID13, t1.MEDIA_CATEGORY_ID AS MEDIA_CATEGORY_ID14, t1.MEDIA_FORMAT_ID AS MEDIA_FORMAT_ID15 FROM MEDIA_SUBJECT t0, MEDIA_LISTING t1 WHERE ((t0.SUBJECT_ID = ?) AND (t0.SUBJECT_ID = t1.MEDIA_SUBJECT_ID)) LIMIT ?, ?
        bind => [518, 0, 16]

I have added logic to my web page build for the search results to detect any null attribute values for the returned MediaListing objects and that is preventing any page blow-ups.  In this case where one of the media records is not able to retrieve the thumbnail file image I detect that and substitute an "image not found" JPG.  

I can't use the refreshIdentityMapResult() method on the query I am running now because it is just a basic Query type object, not a ReadAllQuery.

-sonavor

[sonavor]

I decided to try the query hint first.

That appears to work.  It is a little difficult to setup the test scenario to test the problem as I have to create some test images (at least three entries) so I ran two test scenarios - two tests with three new media entries for each scenario.  In both tests the page populated by the query I have had the problem with worked after using the query hint - q.setHint("eclipselink.refresh", "true");

So that is great!

The thumbnail attribute is just a file name String.  It isn't a transient property but an actual database column.  There are probably some other attributes that are (were) not refreshing but the thumbnail attribute is the one that is noticeable on the resulting web page.

Thanks for the solution.  I really appreciate it.

-sonavor


Is it just the thumbnail that is null, or other mapped attributes?  Are you mapping the thumbnail to the database, or not storing it in the database?  If it is transient, then how do you expect it to come back when reading from the database?

Otherwise, it may have something to do with the first/maxResults, try removing this to confirm if this is causing the issue.  Also you can set the query to refresh using the query hint, "eclipselink.refresh"="true".

sonavor wrote:

It is odd.
I turned more logging on but there is too much information to put in here.

Here is some more information about what is going on.  I use and web application admin screen to add new media definition records to a MySql table.  A new media definition includes several child, media item, records.  Two of those child, media item records represent a thumbnail image and a preview image.  In my latest test I added three new media definitions.  Those records go into the database as usual without a hitch.  For each of those three definitions I placed the necessary image (JPG) files where the web app can find them.  Now here is where the odd behavior occurs.  After those inserts I run a query from my web application search screen that finds the new records.  Of those three inserted media definition records, the first one does not return the thumbnail.  If I drill into the media definition where the application performs an EntityManager find(MediaListing.class, mediaId) then the thumbnail appears...and from then on, appears on a search.  When I say "drill into" - what I mean is that the search finds the media definition record but it does not seem to have all of the necessary information such as the filename of the thumbnail image.  It isn't able to retrieve that child attribute.

To double check I ran another test where I entered three new media definitions.  Same thing happens - the first record I entered cannot find the thumbnail on the search until I drill into the record.  

The JPA query I am running is this -
            Integer subjectId = searchDto.getMediaSubject().getSubjectId();
            StringBuffer qSB = new StringBuffer();
            qSB.append("SELECT object(m) FROM MediaListing as m JOIN m.mediaSubject s WHERE s.subjectId = :argSubjectId");
            Query q = em.createQuery(qSB.toString());
            q.setParameter("argSubjectId", subjectId);
            q.setFirstResult(startRow);
            q.setMaxResults(rowCount);
            List<MediaListing> mList = q.getResultList();
            if (mList != null) {
                log("findMediaSearchByPaged...found " + mList.size() + "MediaListing records ...");
                for (Iterator i = mList.iterator(); i.hasNext();) {
                    MediaListing ml = (MediaListing) i.next();
                    ml.getMediaType();
                    ml.getMediaRepositoryItemCollection();
                }
            }

I added the ml.getMediaType() and ml.getMediaRepositoryItemCollection() lines to force the entity manager to retrieve the MediaListing (ml) record child records because of the lazy fetch.  I know I can set the fetch strategy with an annotation but I put those lines in there just to see if it would force the child attributes to be received.  In this case it doesn't seem to help.

The EclipseLink output for the query where I retrieve the media items is like this (I run a count first because I paginate the searches) -
[EL Fine]: 2009-09-29 10:52:02.109--ServerSession(4729773)--Connection(14151320)--Thread(Thread[TP-Processor3,5,main])--SELECT COUNT(t0.MEDIA_ID) FROM MEDIA_LISTING t0, MEDIA_SUBJECT t1 WHERE ((t1.SUBJECT_ID = ?) AND (t1.SUBJECT_ID = t0.MEDIA_SUBJECT_ID))
        bind => [518]

followed by -

[EL Fine]: 2009-09-29 10:52:02.202--ServerSession(4729773)--Connection(26375417)--Thread(Thread[TP-Processor3,5,main])--SELECT t1.MEDIA_ID AS MEDIA_ID1, t1.LAST_UPDATE AS LAST_UPDATE2, t1.MEDIA_CODE AS MEDIA_CODE3, t1.MEDIA_TITLE AS MEDIA_TITLE6, t1.MEDIA_DESCRIPTION AS MEDIA_DESCRIPTION7, t1.MEDIA_TYPE_ID AS MEDIA_TYPE_ID11, t1.AUTHOR_ID AS AUTHOR_ID12, t1.MEDIA_SUBJECT_ID AS MEDIA_SUBJECT_ID13, t1.MEDIA_CATEGORY_ID AS MEDIA_CATEGORY_ID14, t1.MEDIA_FORMAT_ID AS MEDIA_FORMAT_ID15 FROM MEDIA_SUBJECT t0, MEDIA_LISTING t1 WHERE ((t0.SUBJECT_ID = ?) AND (t0.SUBJECT_ID = t1.MEDIA_SUBJECT_ID)) LIMIT ?, ?
        bind => [518, 0, 16]

I have added logic to my web page build for the search results to detect any null attribute values for the returned MediaListing objects and that is preventing any page blow-ups.  In this case where one of the media records is not able to retrieve the thumbnail file image I detect that and substitute an "image not found" JPG.  

I can't use the refreshIdentityMapResult() method on the query I am running now because it is just a basic Query type object, not a ReadAllQuery.

-sonavor