Mysql4 -> Oracle 10g: loader fails on 4M rows containing blob

[Thorn Roby]

I'm trying to load a table from MySql 4.1.9 containing 4 million rows, average length around 600 bytes, containing a few integer and char fields followed by a BLOB. I'm using the latest Mysql driver. I've disabled index creation, set defaultFetchSize to 10000 in LoaderJob.olj (which seems to be ignored) and set Commit to 10000 in the Loader screen (which seems to have no effect as no rows are ever inserted). I increased the Java heap to 1Gb (on a Windows XP system with 1GB ram). The load process runs for about 45 minutes, Task Manager shows the Java heap grows to about 360MB, then drops to about 80MB (this cycle happens several times). Records are delivered from the source database for about 30 minutes or so, then the load operation terminates with "Loader exception:" but no message. There is never any transmission to the destination Oracle database after the initial connection.

 

Thorn Roby

Senior Systems Architect - Cavion

Harland Financial Solutions

 

--
You receive this message as a subscriber of the octopus@... mailing list.
To unsubscribe: mailto:octopus-unsubscribe@...
For general help: mailto:sympa@...?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

[Thorn Roby]

Correction: I realize now that my addition of the defaultFetchSize
property to LaoderJob.olj was overwritten by the Generator. Does anyone
know how I might set this? The mysql connector documentation suggests
adding it at the end of the connect string after the database name, but
I tried that initially and the Generator wouldn't accept it. If I add it
into LoaderJob.olj manually before running Loader, the XML is rejected.
 
Thorn Roby
Senior Systems Architect - Cavion
Harland Financial Solutions




--
You receive this message as a subscriber of the octopus@... mailing list.
To unsubscribe: mailto:octopus-unsubscribe@...
For general help: mailto:sympa@...?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

[Thorn Roby]

I re-ran the transfer with full logging and got this exception:

org.webdocwf.util.loader.LoaderException: Exception:
 at org.webdocwf.util.loader.Loader.load(Unknown Source)
 at
org.webdocwf.util.loader.wizard.OctopusLoaderThread$OctopusThread.run(Un
known Source)
Caused by: org.webdocwf.util.loader.LoaderException: SQLException:
 ... 2 more
Caused by: com.mysql.jdbc.CommunicationsException: Communications link
failure due to underlying exception:
 
** BEGIN NESTED EXCEPTION **
 
java.io.EOFException
 
STACKTRACE:
 
java.io.EOFException
 at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1913)
 at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2357)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812)
 at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1295)
 at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2262)
 at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:439)
 at
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:1970)
 at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1387)
 at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1727)
 at com.mysql.jdbc.Connection.execSQL(Connection.java:3170)
 at com.mysql.jdbc.Connection.execSQL(Connection.java:3099)
 at com.mysql.jdbc.Statement.executeQuery(Statement.java:1169)
 at org.webdocwf.util.loader.Loader.loadSourceCT(Unknown Source)
 at org.webdocwf.util.loader.Loader.load(Unknown Source)
 at
org.webdocwf.util.loader.wizard.OctopusLoaderThread$OctopusThread.run(Un
known Source)
 

** END NESTED EXCEPTION **
 
 
 
Last packet sent to the server was 895576 ms ago.
 at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2515)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812)
 at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1295)
 at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2262)
 at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:439)
 at
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:1970)
 at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1387)
 at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1727)
 at com.mysql.jdbc.Connection.execSQL(Connection.java:3170)
 at com.mysql.jdbc.Connection.execSQL(Connection.java:3099)
 at com.mysql.jdbc.Statement.executeQuery(Statement.java:1169)
 at org.webdocwf.util.loader.Loader.loadSourceCT(Unknown Source)
 ... 2 more
java.lang.Throwable
 at org.webdocwf.util.loader.LoaderException.printStackTrace(Unknown
Source)
 at
org.webdocwf.util.loader.LoaderException.getStackTraceAsString(Unknown
Source)
 at
org.webdocwf.util.loader.wizard.OctopusLoaderThread$OctopusThread.run(Un
known Source)
 

 
 
Thorn Roby
Senior Systems Architect - Cavion
Harland Financial Solutions




--
You receive this message as a subscriber of the octopus@... mailing list.
To unsubscribe: mailto:octopus-unsubscribe@...
For general help: mailto:sympa@...?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws