|
View:
New views
12 Messages
—
Rating Filter:
Alert me
|
|
|
The ScrollableCursor's issuesHi
I am trying to use the ScrollableCursor for retrieving about 10.000 records. I have to get them divided into small portions which include 100 records each. All works well except two issues. Firstly I see that the size of memory which is used by JVM is being increased while the sequence of invocations of the cursor's next(100) method is happening although I want memory which was allocated for previous portions to be released. The total amount of used memory is about 1Gb for 10.000 records. Secondly the transaction which is started to scroll through the cursor switches to rollback state after 5 minutes although a lot of invocations of the cursor's next(100) method are happening. My questions are: 1. How can I decrease the size of memory which is used by Eclipselink to walk through the cursor? 2. Can I set the period of transaction's timeout? Here is my code: The servlet: @EJB ejb3.OebsInterface oebs; ... oebs.findOebsByTime(time); int from = 10000; int quantity = 100; Vector<RootType> rootList = null; while((rootList = oebs.getOebsByTime(from, quantity)) != null){ int i = 0; HeaderBaseType header; while(i < rootList.size()){ header = rootList.elementAt(i).getHeader(); ... i++; } from = from - quantity; } oebs.releaseOebsByTime(); The session bean: @Stateful @TransactionManagement(TransactionManagementType.BEAN) public class OebsInterfaceImpl implements OebsInterface { @PersistenceContext(name="oebsEJB") EntityManager em; @Resource javax.transaction.UserTransaction ut; ... protected JpaEntityManager jpaEM = null; protected UnitOfWork uow = null; protected ScrollableCursor cursorByTime = null; public void findOebsByTime(XMLGregorianCalendar time) throws NotSupportedException, SystemException{ ExpressionBuilder oebs = new ExpressionBuilder(); Vector arguments = new Vector(); arguments.addElement("/o:root/o:header[o:timeOfCreation=\"" + time.toString() + "000+00:00\"]"); arguments.addElement("xmlns:o=\"http://www.rosbank.ru/oebs\""); Expression where = oebs.getField("OBJECT_VALUE").getFunction(Oracle10Platform_Customizer.ExistsNodeWithNamespaces, arguments).greaterThan(0); ReadAllQuery queryByTime = new ReadAllQuery(OebsView.class, where); queryByTime.useScrollableCursor(); ut.begin(); jpaEM = org.eclipse.persistence.jpa.JpaHelper.getEntityManager(em); uow = jpaEM.getActiveSession().getActiveUnitOfWork(); if(uow == null){ uow = jpaEM.getActiveSession().acquireUnitOfWork(); } cursorByTime = (ScrollableCursor) uow.executeQuery(queryByTime); } public Vector<RootType> getOebsByTime(int from, int quantity) throws JAXBException{ Vector<RootType> rootList = null; if(from <= cursorByTime.size()){ rootList = new Vector<RootType>(); cursorByTime.absolute(from); int q; if((cursorByTime.size() - from) + 1 >= quantity){ q = quantity; } else{ q = cursorByTime.size() - from + 1; } Vector<ejb3.OebsView> oebsList = new Vector<ejb3.OebsView>(); oebsList = cursorByTime.next(q); System.out.println("oebsList.size is " + oebsList.size()); JAXBContext jaxbContext = JAXBContext.newInstance("ru.rosbank.oebs:ru.rosbank.oebs.header:ru.rosbank.oebs.content.assignment:ru.rosbank.oebs.content.organization:ru.rosbank.oebs.content.qualifications:com.ibm.xmlns.prod.websphere.j2ca.jdbc.appsxx_assignment_v:com.ibm.xmlns.prod.websphere.j2ca.jdbc.appsxx_organization_v:com.ibm.xmlns.prod.websphere.j2ca.jdbc.appsxx_qualifications_v"); Unmarshaller unmarshaller = jaxbContext.createUnmarshaller(); JAXBElement<RootType> jaxbElement; RootType root; int i = 0; while(i < oebsList.size()){ jaxbElement = (JAXBElement<RootType>) unmarshaller.unmarshal(new InputSource(new StringReader(oebsList.elementAt(i).getObjectValue()))); root = (RootType) jaxbElement.getValue(); rootList.add(root); i++; } } return rootList; } public void releaseOebsByTime() throws SecurityException, IllegalStateException, RollbackException, HeuristicMixedException, HeuristicRollbackException, SystemException{ uow.release(); ut.commit(); } This code workes in the Websphere container. Regards Dmitry |
|
|
Re: The ScrollableCursor's issuesYou are executing the query in the UnitOfWork so all of the returned objects will be managed (and not allowed to garbage collect). If you execute it in the Session (uow.getParent()), then the objects will not be managed and free to gc, (but must be used as read-only). You could also call clear() in between each page, or set the query to be read-only (setIsReadOnly(true) same as exec in session).
The rollback seems like it is cause because you have a transaction timeout set somewhere in WAS or your DB. You will need to find and increase this timeout. You could also potentially use JPA firstResult/maxResult to page the results.
James Sutherland EclipseLink, TopLink Wiki: EclipseLink, TopLink Forums: TopLink, EclipseLink Book: Java Persistence |
|
|
Survey: Anybody using EclipseLink in Production?My organization is evaluating EclipseLink. One of the question raise
during risk evaluation is if anybody already uses it and what was their experience including how fast it is compared to previous version: TopLink 9/10, TopLink Essential. To start the survey, I'm providing our setup: Number of application: 1 Existing application: yes Migrated from: TopLink 10.x Stage: early evaluation, only smoke tested Application Server: WebSphere 6.1 Hardware: Sun Unix, 4 boxes, 1 JVM each Database: Oracle 10 Use ORM: Yes Use XML-Java support: Yes Number of mappings: 150 Use SessionBroker: yes, with two children session, pointing to 2 different DB schemas Number of ORM EclipseLink projects (XML files): 3 Number of XML-Java projects: 3 Change propagation between servers: No Invalidation propagation between servers: Yes Propagation using RMI or JMS: JMS Use OSGI: No Use JPA Mapping: No Use proprietary EclipseLink API: Yes Use JPA API: No Call stored procedure: Yes Use flush during UOW: No, but will Use conform in unit of work: Yes Use DB Sequence: Yes Evaluation so far: - Easy to migrate session, mapping, and Java code - WebSphere integration is not yet coded, so extraction of Oracle JDBC connection that is needed by different EclipseLink mechanisms is not working - JMS invalidation support is still not compatible with J2EE specification - SessionBroker is still broken - Some bugs not existing with TopLink 10.x have been introduced - Support on usage via users' forum is good - Support on bug via developers' forum is minimal - Realistic documentation of supported platform/setup is missing - We still not have numbers about performance _______________________________________________ eclipselink-users mailing list eclipselink-users@... https://dev.eclipse.org/mailman/listinfo/eclipselink-users |
|
|
Re: Survey: Anybody using EclipseLink in Production?Database: PostgreSQL
Exisiting Appliction: Yes Migrated from: custom JDBC based DAO. Stage: Production for ~6 mo. Environment: J2SE RCP osgi application. (Thick Client) Number of Entities: ~100 OSGi: yes JPA Annotations: Yes, Yes, Yes! Use Dali: Yes Sequences: DB Driven Use Expression framework: no Use Unit of Work For Change Tracking: Yes Caveats: Being a thick client we basically disable caching, so we are unable to report in that area. However we can report it is possible to bypass shared caches :) Support from users: good via mailing list Support from devs: good via mailing list. We've had tons of help from developers on the list here, even though we don't use Oracle and are doing somewhat unorthodox things. ./tch On Wed, Sep 10, 2008 at 12:21 PM, Sebastien Tardif <stardif@...> wrote: > My organization is evaluating EclipseLink. One of the question raise > during risk evaluation is if anybody already uses it and what was their > experience including how fast it is compared to previous version: > TopLink 9/10, TopLink Essential. > > To start the survey, I'm providing our setup: > > Number of application: 1 > Existing application: yes > Migrated from: TopLink 10.x > Stage: early evaluation, only smoke tested > Application Server: WebSphere 6.1 > Hardware: Sun Unix, 4 boxes, 1 JVM each > Database: Oracle 10 > Use ORM: Yes > Use XML-Java support: Yes > Number of mappings: 150 > Use SessionBroker: yes, with two children session, pointing to 2 > different DB schemas > Number of ORM EclipseLink projects (XML files): 3 > Number of XML-Java projects: 3 > Change propagation between servers: No > Invalidation propagation between servers: Yes > Propagation using RMI or JMS: JMS > Use OSGI: No > Use JPA Mapping: No > Use proprietary EclipseLink API: Yes > Use JPA API: No > Call stored procedure: Yes > Use flush during UOW: No, but will > Use conform in unit of work: Yes > Use DB Sequence: Yes > > Evaluation so far: > - Easy to migrate session, mapping, and Java code > - WebSphere integration is not yet coded, so extraction of Oracle JDBC > connection that is needed by different EclipseLink mechanisms is not > working > - JMS invalidation support is still not compatible with J2EE > specification > - SessionBroker is still broken > - Some bugs not existing with TopLink 10.x have been introduced > - Support on usage via users' forum is good > - Support on bug via developers' forum is minimal > - Realistic documentation of supported platform/setup is missing > - We still not have numbers about performance > > _______________________________________________ > eclipselink-users mailing list > eclipselink-users@... > https://dev.eclipse.org/mailman/listinfo/eclipselink-users > eclipselink-users mailing list eclipselink-users@... https://dev.eclipse.org/mailman/listinfo/eclipselink-users |
|
|
Re: The ScrollableCursor's issuesThanks a lot for your answer, James Sutherland.
It is very useful for me. Regards Dmitry
|
|
|
Re: The ScrollableCursor's issuesHi, James Sutherland!
Thanks a lot for your help. One your tip helped me but another did not help. I found settings for the transaction service and changed the value of timeout. It works. But a memory for the previous block is not released after the next block is getted although I changed my code: ReadAllQuery queryByTime = new ReadAllQuery(OebsView.class, where); queryByTime.useScrollableCursor(); queryByTime.setIsReadOnly(true); ut.setTransactionTimeout(60*60*24); ut.begin(); jpaEM = org.eclipse.persistence.jpa.JpaHelper.getEntityManager(em); uow = jpaEM.getActiveSession().getActiveUnitOfWork(); if(uow == null){ System.out.println("uow is null"); uow = jpaEM.getActiveSession().acquireUnitOfWork(); } cursorByTimeAndOrganizationId = (ScrollableCursor) uow.getParent().executeQuery(queryByTime); I could not try the third way which You advised because I could not find the class where the clear() method is declared. Regards, Dmitry
|
|
|
Re: The ScrollableCursor's issuesThe objects will no longer be held by the UnitOfWork/EntityManager, but will still be in the cache. Depending on what caching you are using, they will eventually garbage collect (weak references). SoftWeak is the default cache type, which will hold 100 objects using soft reference and the rest using weak references, weak will garbage collect in the next gc, and soft will garbage collect when memory is low. You can change your cache type to weak to allow better garbage collection. You could potentially also set dontMaintainCache() on the query to avoid the cache entirely.
If you set read-only on the query, you don't need to execute it with the Session, using the EntityManager is fine.
James Sutherland EclipseLink, TopLink Wiki: EclipseLink, TopLink Forums: TopLink, EclipseLink Book: Java Persistence |
|
|
Re: The ScrollableCursor's issuesHi, James Sutherland
Thanks for your help and patience The level of the cache which is configured for the descriptor of the OebsView.class is None: // ClassDescriptor Properties. descriptor.useNoIdentityMap(); descriptor.setIdentityMapSize(0); descriptor.useRemoteNoIdentityMap(); descriptor.setRemoteIdentityMapSize(0); descriptor.setIsIsolated(true); descriptor.setAlias("OebsView"); descriptor.setCacheSynchronizationType(ClassDescriptor.DO_NOT_SEND_CHANGES); I added the dontMaintainCache() but it did not change anything in the allocation of the jvm's memory: ReadAllQuery queryByTime = new ReadAllQuery(OebsView.class, where); queryByTime.useScrollableCursor(); queryByTime.setIsReadOnly(true); queryByTime.dontMaintainCache(); ut.setTransactionTimeout(60*60*24); ut.begin(); jpaEM = org.eclipse.persistence.jpa.JpaHelper.getEntityManager(em); uow = jpaEM.getActiveSession().getActiveUnitOfWork(); if(uow == null){ System.out.println("uow is null"); uow = jpaEM.getActiveSession().acquireUnitOfWork(); } cursorByTimeAndOrganizationId = (ScrollableCursor) uow.executeQuery(queryByTime); The memory is still allocated after the transaction is committed. Is this a jvm's feature? Can I use a jvm's interface to release memory? Regards, Dmitry
|
|
|
Re: The ScrollableCursor's issuesIn Java the VM will garbage collect whenever it feels it needs to. There is an API on java.lang.System.gc(), but this is only a suggestion, the VM may still not perform a gc.
Ensure that nothing in your application is holding onto the objects. If anything references an object, it cannot gc. There are memory profilers such as JProfiler that can pinpoint memory leaks. The easiest way to find out if an object has gc'd is to create a WeakReference to it, and check if the reference is null. In general I would not recommend using NoIdentityMap, weak should be fine. Are you running out of memory? Did you try increasing your JVM's memory?
|
|
|
Re: The ScrollableCursor's issuesHi, James Sutherland
Thank you again! I implemented the finalize method for the class whose instances are retrieved by the cursor and I can see the result of the finalizing in the system log so I am sure that instances are collected. @Override protected void finalize() throws Throwable { // TODO Auto-generated method stub super.finalize(); System.out.println("OebsView.finalize()"); System.out.println("id=" + this.getId()); } And I use this code to tell the JVM to collect the garbage: Runtime rt = Runtime.getRuntime(); rt.gc(); But I have the java.lang.OutOfMemoryError in the end. Regards, Dmitry
|
|
|
Re: The ScrollableCursor's issuesI investigated reasons for leaks and I can say that about 50% of the heap is free during the total size of the memory which is allocated is being increased by 1Gb so the heap is not a reason for the java.lang.OutOfMemoryError.
Regards, Dmitry
|
|
|
Re: The ScrollableCursor's issuesHi
I found the workaround which is really useful in case you are working with XMLTYPE's data and using the scrollable cursor. In this case you need to implement the custom class which extends the Oracle11Platform class and override the getObjectFromResultSet method in this way: public Object getObjectFromResultSet(ResultSet resultSet, int columnNumber, int type, AbstractSession session) throws java.sql.SQLException { if (type == OracleTypes.OPAQUE) { Object result = resultSet.getObject(columnNumber); if(!(result instanceof OPAQUE)) { // Report Queries can cause result to not be an instance of OPAQUE. return result; } XMLType xml; xml = (XMLType) result; String xmlstr = xml.getStringVal(); xml.close(); return xmlstr; } else{ return super.getObjectFromResultSet(resultSet, columnNumber, type, session); } } By doing this you are able to escape from very big memory leaks. Regards, Dmitry
|
| Free embeddable forum powered by Nabble | Forum Help |