|
»
Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
|
View:
New views
8 Messages
—
Rating Filter:
Alert me
|
 |
Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by Dronamraju, Santosh B (GE Indust, ConsInd)
::
Rate this Message:
Hi everyone,
For large excel exports, I get out of memory error. Is there any way to write directly to physical file record by record instead of using heap and then writing the whole data to file system at the end? Because increasing heap size is not working even if I use 1024 MB. As in our application we have large and number of export to excel. Thanks, Santosh |
|
|
Re: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by MSB
::
Rate this Message:
May I ask which version of the Excel file format you are targetting? Are you trying to create binary .xls files or xml based .xlsx files? If the latter then you could take a look at the http://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/poi/xssf/usermodel/examples/BigGridDemo.java proof that Yegor created. It is only intended to demonstrate a technique and must not be thought of as a complete application however.
If, however, you are targetting the binary file format then I am afraid that the news is not too good. Owing to the file format - it consists of a series of linked lists - it is not possible to write the file out in chunks. Yours Mark B
|
|
|
RE: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by Dronamraju, Santosh B (GE Indust, ConsInd)
::
Rate this Message:
I am using POI V3.5, and was creating binary .xls files.
Now that I understand large export are possible only using xml based .xlsx files approach, thanks for you help. I have another question, for an export more than 65,535 records, how to instruct xml to open up in a separate sheet instead of not showing the rest of the records? Thanks, Santosh -----Original Message----- From: MSB [mailto:markbrdsly@...] Sent: Friday, October 23, 2009 7:57 PM To: user@... Subject: Re: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size. May I ask which version of the Excel file format you are targetting? Are you trying to create binary .xls files or xml based .xlsx files? If the latter then you could take a look at the http://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/po i/xssf/usermodel/examples/BigGridDemo.java proof that Yegor created. It is only intended to demonstrate a technique and must not be thought of as a complete application however. If, however, you are targetting the binary file format then I am afraid that the news is not too good. Owing to the file format - it consists of a series of linked lists - it is not possible to write the file out in chunks. Yours Mark B Dronamraju, Santosh B (GE Indust, ConsInd) wrote: > > Hi everyone, > > > > For large excel exports, I get out of memory error. Is there any way to > write directly to physical file record by record instead of using heap > and then writing the whole data to file system at the end? > > > > Because increasing heap size is not working even if I use 1024 MB. As in > our application we have large and number of export to excel. > > > > Thanks, > > Santosh > > > > > -- View this message in context: http://www.nabble.com/Any-solution-to-eliminate-out-of-memory-error-for- large-export-to-excel-apart-from-increasing-the-heap-size.-tp26022706p26 027366.html Sent from the POI - User mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... |
|
|
RE: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by MSB
::
Rate this Message:
I do not remember saying that Yegor's BigGridDemo demonstrated the ONLY way to proceed with larger data-sets merely that it demonstrated a possible approach if you working with the OpenXML based file format.
Having said that, can I just be a little more clear about what you are asking please? Are you saying that you have sufficient data to populate more than sixty five thousand rows and that rather than see this data converted into a single Excel worksheet, you want to see it split across several worksheets? If so, how should the data be split or divided; what criteria should determine which sheet each row is written to? Yours Mark B
|
|
|
RE: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by Dronamraju, Santosh B (GE Indust, ConsInd)
::
Rate this Message:
Let's say I have 100000 records to be exported,
Now in one sheet of a workbook it can contain only 65535 records, Then the other set of 34465 records should get created in another sheet of the same workbook. Is it possible in Yegor's BigGridDemo? Using the xml approach? Thanks, Santosh -----Original Message----- From: MSB [mailto:markbrdsly@...] Sent: Tuesday, October 27, 2009 12:58 PM To: user@... Subject: RE: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size. I do not remember saying that Yegor's BigGridDemo demonstrated the ONLY way to proceed with larger data-sets merely that it demonstrated a possible approach if you working with the OpenXML based file format. Having said that, can I just be a little more clear about what you are asking please? Are you saying that you have sufficient data to populate more than sixty five thousand rows and that rather than see this data converted into a single Excel worksheet, you want to see it split across several worksheets? If so, how should the data be split or divided; what criteria should determine which sheet each row is written to? Yours Mark B Dronamraju, Santosh B (GE Indust, ConsInd) wrote: > > I am using POI V3.5, and was creating binary .xls files. > > Now that I understand large export are possible only using xml based > .xlsx files approach, thanks for you help. > > I have another question, for an export more than 65,535 records, how to > instruct xml to open up in a separate sheet instead of not showing the > rest of the records? > > Thanks, > Santosh > -----Original Message----- > From: MSB [mailto:markbrdsly@...] > Sent: Friday, October 23, 2009 7:57 PM > To: user@... > Subject: Re: Any solution to eliminate out of memory error for large > export to excel apart from increasing the heap size. > > > May I ask which version of the Excel file format you are targetting? > you > trying to create binary .xls files or xml based .xlsx files? If the > latter > then you could take a look at the > http://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/po > i/xssf/usermodel/examples/BigGridDemo.java > proof that Yegor created. It is only intended to demonstrate a technique > and > must not be thought of as a complete application however. > > If, however, you are targetting the binary file format then I am afraid > that > the news is not too good. Owing to the file format - it consists of a > series > of linked lists - it is not possible to write the file out in chunks. > > Yours > > Mark B > > > Dronamraju, Santosh B (GE Indust, ConsInd) wrote: >> >> Hi everyone, >> >> >> >> For large excel exports, I get out of memory error. Is there any way > to >> write directly to physical file record by record instead of using >> and then writing the whole data to file system at the end? >> >> >> >> Because increasing heap size is not working even if I use 1024 MB. As > in >> our application we have large and number of export to excel. >> >> >> >> Thanks, >> >> Santosh >> >> >> >> >> > > -- > View this message in context: > > large-export-to-excel-apart-from-increasing-the-heap-size.-tp26022706p26 > 027366.html > Sent from the POI - User mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@... > For additional commands, e-mail: user-help@... > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@... > For additional commands, e-mail: user-help@... > > > -- View this message in context: http://www.nabble.com/Any-solution-to-eliminate-out-of-memory-error-for- large-export-to-excel-apart-from-increasing-the-heap-size.-tp26022706p26 073325.html Sent from the POI - User mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... |
|
|
Re: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by Yegor Kozlov
::
Rate this Message:
> Let's say I have 100000 records to be exported,
> Now in one sheet of a workbook it can contain only 65535 records, > Then the other set of 34465 records should get created in another sheet > of the same workbook. > > Is it possible in Yegor's BigGridDemo? Using the xml approach? > Yes, it is possible. The BigGridDemo application streams the generated data into a temporary file and then injects it in the target OOXML package. This approach allows generating really large sets of data, up to the grid limits. Splitting the data across multiple sheets is possible, actually it requires only a few lines of code to add. Yegor > Thanks, > Santosh > > -----Original Message----- > From: MSB [mailto:markbrdsly@...] > Sent: Tuesday, October 27, 2009 12:58 PM > To: user@... > Subject: RE: Any solution to eliminate out of memory error for large > export to excel apart from increasing the heap size. > > > I do not remember saying that Yegor's BigGridDemo demonstrated the ONLY > way > to proceed with larger data-sets merely that it demonstrated a possible > approach if you working with the OpenXML based file format. > > Having said that, can I just be a little more clear about what you are > asking please? Are you saying that you have sufficient data to populate > more > than sixty five thousand rows and that rather than see this data > converted > into a single Excel worksheet, you want to see it split across several > worksheets? If so, how should the data be split or divided; what > criteria > should determine which sheet each row is written to? > > Yours > > Mark B > > > Dronamraju, Santosh B (GE Indust, ConsInd) wrote: >> I am using POI V3.5, and was creating binary .xls files. >> >> Now that I understand large export are possible only using xml based >> .xlsx files approach, thanks for you help. >> >> I have another question, for an export more than 65,535 records, how > to >> instruct xml to open up in a separate sheet instead of not showing the >> rest of the records? >> >> Thanks, >> Santosh >> -----Original Message----- >> From: MSB [mailto:markbrdsly@...] >> Sent: Friday, October 23, 2009 7:57 PM >> To: user@... >> Subject: Re: Any solution to eliminate out of memory error for large >> export to excel apart from increasing the heap size. >> >> >> May I ask which version of the Excel file format you are targetting? > Are >> you >> trying to create binary .xls files or xml based .xlsx files? If the >> latter >> then you could take a look at the >> > http://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/po >> i/xssf/usermodel/examples/BigGridDemo.java >> proof that Yegor created. It is only intended to demonstrate a > technique >> and >> must not be thought of as a complete application however. >> >> If, however, you are targetting the binary file format then I am > afraid >> that >> the news is not too good. Owing to the file format - it consists of a >> series >> of linked lists - it is not possible to write the file out in chunks. >> >> Yours >> >> Mark B >> >> >> Dronamraju, Santosh B (GE Indust, ConsInd) wrote: >>> Hi everyone, >>> >>> >>> >>> For large excel exports, I get out of memory error. Is there any way >> to >>> write directly to physical file record by record instead of using > heap >>> and then writing the whole data to file system at the end? >>> >>> >>> >>> Because increasing heap size is not working even if I use 1024 MB. As >> in >>> our application we have large and number of export to excel. >>> >>> >>> >>> Thanks, >>> >>> Santosh >>> >>> >>> >>> >>> >> -- >> View this message in context: >> > http://www.nabble.com/Any-solution-to-eliminate-out-of-memory-error-for- > large-export-to-excel-apart-from-increasing-the-heap-size.-tp26022706p26 >> 027366.html >> Sent from the POI - User mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@... >> For additional commands, e-mail: user-help@... >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@... >> For additional commands, e-mail: user-help@... >> >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... |
|
|
RE: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by Dronamraju, Santosh B (GE Indust, ConsInd)
::
Rate this Message:
Hi Yegor,
What are those few lines of code that I need to add and where? It would be very nice of you if you could help me out. Thanks, Santosh -----Original Message----- From: Yegor Kozlov [mailto:yegor@...] Sent: Tuesday, October 27, 2009 2:30 PM To: POI Users List Subject: Re: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size. > Let's say I have 100000 records to be exported, > Now in one sheet of a workbook it can contain only 65535 records, > Then the other set of 34465 records should get created in another sheet > of the same workbook. > > Is it possible in Yegor's BigGridDemo? Using the xml approach? > Yes, it is possible. The BigGridDemo application streams the generated data into a temporary file and then injects it in the target OOXML package. This approach allows generating really large sets of data, up to the grid limits. Splitting the data across multiple sheets is possible, actually it requires only a few lines of code to add. Yegor > Thanks, > Santosh > > -----Original Message----- > From: MSB [mailto:markbrdsly@...] > Sent: Tuesday, October 27, 2009 12:58 PM > To: user@... > Subject: RE: Any solution to eliminate out of memory error for large > export to excel apart from increasing the heap size. > > > I do not remember saying that Yegor's BigGridDemo demonstrated the > way > to proceed with larger data-sets merely that it demonstrated a possible > approach if you working with the OpenXML based file format. > > Having said that, can I just be a little more clear about what you are > asking please? Are you saying that you have sufficient data to populate > more > than sixty five thousand rows and that rather than see this data > converted > into a single Excel worksheet, you want to see it split across several > worksheets? If so, how should the data be split or divided; what > criteria > should determine which sheet each row is written to? > > Yours > > Mark B > > > Dronamraju, Santosh B (GE Indust, ConsInd) wrote: >> I am using POI V3.5, and was creating binary .xls files. >> >> Now that I understand large export are possible only using xml based >> .xlsx files approach, thanks for you help. >> >> I have another question, for an export more than 65,535 records, how > to >> instruct xml to open up in a separate sheet instead of not showing >> rest of the records? >> >> Thanks, >> Santosh >> -----Original Message----- >> From: MSB [mailto:markbrdsly@...] >> Sent: Friday, October 23, 2009 7:57 PM >> To: user@... >> Subject: Re: Any solution to eliminate out of memory error for large >> export to excel apart from increasing the heap size. >> >> >> May I ask which version of the Excel file format you are targetting? > Are >> you >> trying to create binary .xls files or xml based .xlsx files? If the >> latter >> then you could take a look at the >> > >> i/xssf/usermodel/examples/BigGridDemo.java >> proof that Yegor created. It is only intended to demonstrate a > technique >> and >> must not be thought of as a complete application however. >> >> If, however, you are targetting the binary file format then I am > afraid >> that >> the news is not too good. Owing to the file format - it consists of a >> series >> of linked lists - it is not possible to write the file out in chunks. >> >> Yours >> >> Mark B >> >> >> Dronamraju, Santosh B (GE Indust, ConsInd) wrote: >>> Hi everyone, >>> >>> >>> >>> For large excel exports, I get out of memory error. Is there any way >> to >>> write directly to physical file record by record instead of using > heap >>> and then writing the whole data to file system at the end? >>> >>> >>> >>> Because increasing heap size is not working even if I use 1024 MB. >> in >>> our application we have large and number of export to excel. >>> >>> >>> >>> Thanks, >>> >>> Santosh >>> >>> >>> >>> >>> >> -- >> View this message in context: >> > > large-export-to-excel-apart-from-increasing-the-heap-size.-tp26022706p26 >> 027366.html >> Sent from the POI - User mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@... >> For additional commands, e-mail: user-help@... >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@... >> For additional commands, e-mail: user-help@... >> >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@... For additional commands, e-mail: user-help@... |
|
|
Re: Any solution to eliminate out of memory error for large export to excel apart from increasing the heap size.
by kishore bandari
::
Rate this Message:
Hi Santosh,
Even I am also facing out of memory issue for large data. I am using binary xls format. I have one possible solution you may try this. Instead of writing new code for xlsx format, there is one option to convert existing binary xls to xlsx code. This option is help ful when existing code is implemented in xls format. You can get more inforamtion at http://poi.apache.org/spreadsheet/converting.html Regards Kishore Kumar
|
| Free embeddable forum powered by Nabble | Forum Help |
