Limit to the number of files loadable in LibTiff
|
View:
New views
4 Messages
—
Rating Filter:
Alert me
|
 |
Limit to the number of files loadable in LibTiff
by Simon R. Proud
::
Rate this Message:
Hi, I am using libtiff on a linux system (OpenSuSE 11) to open a very
large number of files. I notice that if I open more than about 210 files I receive some unexpected errors from LibTiff: "TIFFWriteBufferSetup: [filename]: No space for output buffer I suspect it is some kind of limit within LibTiff, so is there any way to overcome this? I really would like to load more than 210 files at once (I need about 250-300), so if there's a method of changing the limit it would be superb to know! Thanks, Simon. PhD Student Department of Geography Øster Voldgade 10 University of Copenhagen DK-1350, Copenhagen Denmark Email: srp@... _______________________________________________ Tiff mailing list: Tiff@... http://lists.maptools.org/mailman/listinfo/tiff http://www.remotesensing.org/libtiff/ |
|
|
Re: Limit to the number of files loadable in LibTiff
by jcupitt
::
Rate this Message:
2009/10/19 Simon R. Proud <Srp@...>:
> Hi, I am using libtiff on a linux system (OpenSuSE 11) to open a very > large number of files. > I notice that if I open more than about 210 files I receive some > unexpected errors from LibTiff: > "TIFFWriteBufferSetup: [filename]: No space for output buffer > > I suspect it is some kind of limit within LibTiff, so is there any way > to overcome this? I really would like to load more than 210 files at > once (I need about 250-300), so if there's a method of changing the > limit it would be superb to know! Looking at the source, the message is generated if TIFFSetupStrips(tif) fails. This in turn fails if _TIFFmalloc() fails. So it sounds like you are running out of memory. I would try watching your program with "top" as it runs and see how large the RSIZE column gets. Perhaps you can alter your program to use less RAM? John _______________________________________________ Tiff mailing list: Tiff@... http://lists.maptools.org/mailman/listinfo/tiff http://www.remotesensing.org/libtiff/ |
|
|
Re: Limit to the number of files loadable in LibTiff
by Bob Friesenhahn
::
Rate this Message:
On Mon, 19 Oct 2009, jcupitt@... wrote:
> > Looking at the source, the message is generated if > TIFFSetupStrips(tif) fails. This in turn fails if _TIFFmalloc() fails. > So it sounds like you are running out of memory. I would try watching > your program with "top" as it runs and see how large the RSIZE column > gets. It is certainly a resource issue. Either due to a leak or using too much of something. Since you are using Linux you mean be able to use the 'memusage' facility to help understand memory usage: LD_PRELOAD=/lib/libmemusage.so yourprog And for detailed analysis, use valgrind --tool=memcheck --leak-check=full yourprog to check for leaks, and use valgrind --tool=massif yourprog to find areas of excess consumption. Bob -- Bob Friesenhahn bfriesen@..., http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ _______________________________________________ Tiff mailing list: Tiff@... http://lists.maptools.org/mailman/listinfo/tiff http://www.remotesensing.org/libtiff/ |
|
|
Re: Limit to the number of files loadable in LibTiff
by Simon R. Proud
::
Rate this Message:
>>
>> Looking at the source, the message is generated if >> TIFFSetupStrips(tif) fails. This in turn fails if _TIFFmalloc() fails. >> So it sounds like you are running out of memory. I would try watching >> your program with "top" as it runs and see how large the RSIZE column >> gets. > >It is certainly a resource issue. Either due to a leak or using too >much of something. Since you are using Linux you mean be able to use >the 'memusage' facility to help understand memory usage: Yep, it definitely isn't a libtiff issue! I have discovered that as I was running 32bit linux I can't access the whole 8Gb of memory, but now that I have switched to 64bit the problem appears to be gone. I'm no longer getting the buffer message. >And for detailed analysis, use > > valgrind --tool=memcheck --leak-check=full yourprog > >to check for leaks, and use > > valgrind --tool=massif yourprog > >to find areas of excess consumption. Thanks. They show no leaks in the code, I think the problem is simply that I'm crunching a LOT of data concurrently and I simply exceeded the memory limitations. Anyway, as I say it works now. Thanks for the help, guys! Much appreciated. PhD Student Department of Geography Øster Voldgade 10 University of Copenhagen DK-1350, Copenhagen Denmark Email: srp@... _______________________________________________ Tiff mailing list: Tiff@... http://lists.maptools.org/mailman/listinfo/tiff http://www.remotesensing.org/libtiff/ |
| Free embeddable forum powered by Nabble | Forum Help |
