possible memory leak in enblend & enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)

[Harry van der Wolf-3]

2009/10/7 Lukáš Jirkovský <l.jirkovsky@...>

I'm not Mac user (although I find it really cool but very expensive)
but I may found solution for out of memory problem. I had a discussion
about memory and I mentioned these fragmentation problems on OS. And
get interesting advice – use TLSF [1]. It is said that it doesn't
suffer from fragmentation much.

Maybe some of you want to take a look at it.


[1] http://rtportal.upv.es/rtmalloc/

Lukas

Hi Lukáš,

(You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take looooooooong).

Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend,  we regularly see the error message of enblend crashing due to a memory allocation error.  We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well.

Untill now we blamed it on memory fragmentation, but maybe it's something else.

George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my "big disk" backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1.

My question to you now is: You recently did some "memory leak" patching on the hugin trunk, using cppcheck, thereby finding some "things" in celeste. You reported this via <http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80>.
Can you do the same for the enblend trunk?
If you want to do this and you find the time for it "in the near furure", be so kind to publish this on the hugin-ptx list.
But please: don't feel obliged. If you don't have time or just do not want to do this: just say so.

= If you don't want to do this, you can now stop reading.  =
If you want to do this or are at least interested: please continue reading.

Below you will find my tests on OSX. It's done on the 2.5GB bracked "village hotel" project of George Row.

Below the "tail" of the enblend error for a very recent 32bit "Christoph Spiel" build:
enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1
enblend: info: creating blend mask: 1/3enblend(11221) malloc: ***
vm_allocate(size=580620288) failed (error code=3)
enblend(11221) malloc: *** error: can't allocate region
enblend(11221) malloc: *** set a breakpoint in szone_error to debug

enblend: out of memory
enblend: St9bad_alloc
gnumake: *** [FoyleDays_M2_04.tif] Error 1


Below the "tail" of the error for the stable 32bit enblend 3.2 build (This to prove it's not a recent problem. It's already there in the 3.2 stable build).
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

enblend: out of memory
St9bad_alloc
gnumake: *** [mamaloe_exposure_00.tif] Error 1

Test report for a 64bit enblend: After 6 hours and much further in the process my system crashed. I will rerun tonight.

(I can start and monitor remote. I can't start a crashed mac from remote.)

Note: I build 32bit binaries by default as they run on every mac. A 64 bit version only runs on Leopard and Snow Leopard on 64bit hardware. 64bit brings hardly performance gain or other benefits, only when making gigapixel pano's.)

To me this does NOT mean that the 64 bit behaves better. It only proves IMO that due to the huge 64bit address space, enblend can (might) just continue leaving it 's "memory junk" without filling the address space that fast and that fragmentation is less an issue within the huge 64bit address space. In other words: it will only crash at a later stage when trying to stitch (even bigger) projects. But that's an assumption which I can't prove right now.
 

 

Hoi,

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Stefan Peter-4]

Hi

I'd be interested in doing some tests, too. I remember having had memory
issues as well, but I was never able to reproduce them reliably here on
Linux / Linux_64 and Windows. Is there someplace one could get the
project in question?

Cheers

Stefan Peter

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

To answer my own mail. :-)

I just compiled cppcheck on OSX and did a standard run on the enblend trunk.
It displays the following
[./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor
[./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor

Then I did a "full check". That resulted in exactly the same result as mentioned above.

This is like giving a monkey an encyclopedia. He has no idea what to do with it. I don't even know if this is usefull.
I hope some clever progammers (super monkeys?) know what to do with this.

Harry

2009/10/7 Harry van der Wolf <hvdwolf@...>

2009/10/7 Lukáš Jirkovský <l.jirkovsky@...>

I'm not Mac user (although I find it really cool but very expensive)
but I may found solution for out of memory problem. I had a discussion
about memory and I mentioned these fragmentation problems on OS. And
get interesting advice – use TLSF [1]. It is said that it doesn't
suffer from fragmentation much.

Maybe some of you want to take a look at it.


[1] http://rtportal.upv.es/rtmalloc/

Lukas

Hi Lukáš,

(You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take looooooooong).

Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend,  we regularly see the error message of enblend crashing due to a memory allocation error.  We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well.

Untill now we blamed it on memory fragmentation, but maybe it's something else.

George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my "big disk" backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1.

My question to you now is: You recently did some "memory leak" patching on the hugin trunk, using cppcheck, thereby finding some "things" in celeste. You reported this via <http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80>.
Can you do the same for the enblend trunk?
If you want to do this and you find the time for it "in the near furure", be so kind to publish this on the hugin-ptx list.
But please: don't feel obliged. If you don't have time or just do not want to do this: just say so.

= If you don't want to do this, you can now stop reading.  =
If you want to do this or are at least interested: please continue reading.

Below you will find my tests on OSX. It's done on the 2.5GB bracked "village hotel" project of George Row.

Below the "tail" of the enblend error for a very recent 32bit "Christoph Spiel" build:
enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1
enblend: info: creating blend mask: 1/3enblend(11221) malloc: ***
vm_allocate(size=580620288) failed (error code=3)
enblend(11221) malloc: *** error: can't allocate region
enblend(11221) malloc: *** set a breakpoint in szone_error to debug

enblend: out of memory
enblend: St9bad_alloc
gnumake: *** [FoyleDays_M2_04.tif] Error 1


Below the "tail" of the error for the stable 32bit enblend 3.2 build (This to prove it's not a recent problem. It's already there in the 3.2 stable build).
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

enblend: out of memory
St9bad_alloc
gnumake: *** [mamaloe_exposure_00.tif] Error 1

Test report for a 64bit enblend: After 6 hours and much further in the process my system crashed. I will rerun tonight.

(I can start and monitor remote. I can't start a crashed mac from remote.)

Note: I build 32bit binaries by default as they run on every mac. A 64 bit version only runs on Leopard and Snow Leopard on 64bit hardware. 64bit brings hardly performance gain or other benefits, only when making gigapixel pano's.)

To me this does NOT mean that the 64 bit behaves better. It only proves IMO that due to the huge 64bit address space, enblend can (might) just continue leaving it 's "memory junk" without filling the address space that fast and that fragmentation is less an issue within the huge 64bit address space. In other words: it will only crash at a later stage when trying to stitch (even bigger) projects. But that's an assumption which I can't prove right now.
 

 

Hoi,

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Lukáš Jirkovský]

Hi Harry,

2009/10/7 Harry van der Wolf <hvdwolf@...>:

>
> 2009/10/7 Lukáš Jirkovský <l.jirkovsky@...>
>>
>>
>>
>> I'm not Mac user (although I find it really cool but very expensive)
>> but I may found solution for out of memory problem. I had a discussion
>> about memory and I mentioned these fragmentation problems on OS. And
>> get interesting advice – use TLSF [1]. It is said that it doesn't
>> suffer from fragmentation much.
>>
>> Maybe some of you want to take a look at it.
>>
>>
>> [1] http://rtportal.upv.es/rtmalloc/
>>
>> Lukas
>>
>>
>
> Hi Lukáš,
>
> (You are the one person who I had hoped to react. I started to do tests
> myself yesterday evening but they take looooooooong).

Thanks for being confident in me.

>
> Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions
> of enblend,  we regularly see the error message of enblend crashing due to a
> memory allocation error.  We see this both at hugin-ptx as well as in the
> bugtracker for large projects. We sometimes see it for enfuse in case of
> large projects which need to be fused as well.
> Untill now we blamed it on memory fragmentation, but maybe it's something
> else.

It is possible that it's something else. The question is: how to find it?

> George Row is one of the persons who encounters these errors on OSX. I have
> received a large set (2.5GB) from George in June, some time before my mac
> crashed. At that time I did some tests. These results are gone (no backup of
> test results).Yesterday I took Georges set from my "big disk" backup server
> and did some tests myself trying to stich a 12000X6000 (slightly bigger)
> panorama in hugin-2009.4.0-beta1.
>
> My question to you now is: You recently did some "memory leak" patching on
> the hugin trunk, using cppcheck, thereby finding some "things" in celeste.
> You reported this via
> <http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80>.
> Can you do the same for the enblend trunk?
> If you want to do this and you find the time for it "in the near furure", be
> so kind to publish this on the hugin-ptx list.
> But please: don't feel obliged. If you don't have time or just do not want
> to do this: just say so.

The cppcheck is still running but enblend.cc has already been checked.
It has not detected any (interesting) problems yet. The major leak of
these static analysis tools is that they are not perfect. Even the
ones like Coverity uses. It may be interesting to try valgrind.

Note: I've just read your post that it doesn't detect anything interesting.

>
> = If you don't want to do this, you can now stop reading.  =
> If you want to do this or are at least interested: please continue reading.
>
> Below you will find my tests on OSX. It's done on the 2.5GB bracked "village
> hotel" project of George Row.
>
> Below the "tail" of the enblend error for a very recent 32bit "Christoph
> Spiel" build:
> enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1
> enblend: info: creating blend mask: 1/3enblend(11221) malloc: ***
> vm_allocate(size=580620288) failed (error code=3)
> enblend(11221) malloc: *** error: can't allocate region
> enblend(11221) malloc: *** set a breakpoint in szone_error to debug
>
> enblend: out of memory
> enblend: St9bad_alloc
> gnumake: *** [FoyleDays_M2_04.tif] Error 1
>
>
> Below the "tail" of the error for the stable 32bit enblend 3.2 build (This
> to prove it's not a recent problem. It's already there in the 3.2 stable
> build).
> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> enblend: out of memory
> St9bad_alloc
> gnumake: *** [mamaloe_exposure_00.tif] Error 1

It would be nice to find out which malloc() fails. According to the
older discussion it seems to be a problem in CachedFileImage. IMO the
best thing how to try it would be do build enblend without image cache
and create HUGE swap space so it won't run out of memory. If stitch
works then we should look for the problem in image cache.

I'd try it here but it would take ages since I don't have very
powerful PC. I'll try to limit RAM on my PC (there is some switch to
Linux kernel but IIRC it's almost undocumented) If it doesn't work I
can try to replace all my RAM modules with an old 256MB RAM module and
disable/reduce swap space. Then it may occur earlier and with smaller
projects.

I'm a bit afraid that it doesn't depend on how much RAM (or better
virtual memory) is there but on the stitch size. ie. that when the
stitch output is big enough it exposes some weird bug when it
allocates memory even if it may not be necessary.

>
> Test report for a 64bit enblend: After 6 hours and much further in the
> process my system crashed. I will rerun tonight.
> (I can start and monitor remote. I can't start a crashed mac from remote.)
> Note: I build 32bit binaries by default as they run on every mac. A 64 bit
> version only runs on Leopard and Snow Leopard on 64bit hardware. 64bit
> brings hardly performance gain or other benefits, only when making gigapixel
> pano's.)
> To me this does NOT mean that the 64 bit behaves better. It only proves IMO
> that due to the huge 64bit address space, enblend can (might) just continue
> leaving it 's "memory junk" without filling the address space that fast and
> that fragmentation is less an issue within the huge 64bit address space. In
> other words: it will only crash at a later stage when trying to stitch (even
> bigger) projects. But that's an assumption which I can't prove right now.
>
>
> Hoi,
> Harry
>
> >
>

I'll try changing RAM and valgrind tomorrow. I hope the smaller RAM
allows me to run into the problem quite early.

Lukáš

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

I did another run from the Gui (took a few minutes to find how to compile that one). That gave more results:

[assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is preferred to Post-Incrementing
[assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is preferred to Post-Incrementing
[vigra_impex/exr.cxx:310]: (style) Redundant condition. It is safe to deallocate a NULL pointer
[vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the constructor 'HDRCodecImpl::height'
[vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the constructor 'HDRCodecImpl::width'
[vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor
[vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor
[vigra_impex/rgbe.c:90]: (style) The scope of the variable f can be limited
[vigra_impex/tiff.cxx:199]: (style) Member variable not initialized in the constructor 'TIFFCodecImpl::layers'
[vigra_impex/viff.cxx:245]: (style) Function parameter 'index' is passed by value. It could be passed by reference instead.


This doesn't seem to be memory leaks of some kind. Still I report it here.

Harry

2009/10/7 Harry van der Wolf <hvdwolf@...>

To answer my own mail. :-)

I just compiled cppcheck on OSX and did a standard run on the enblend trunk.
It displays the following
[./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor
[./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor

Then I did a "full check". That resulted in exactly the same result as mentioned above.

This is like giving a monkey an encyclopedia. He has no idea what to do with it. I don't even know if this is usefull.
I hope some clever progammers (super monkeys?) know what to do with this.

Harry

2009/10/7 Harry van der Wolf <hvdwolf@...>

2009/10/7 Lukáš Jirkovský <l.jirkovsky@...>

I'm not Mac user (although I find it really cool but very expensive)
but I may found solution for out of memory problem. I had a discussion
about memory and I mentioned these fragmentation problems on OS. And
get interesting advice – use TLSF [1]. It is said that it doesn't
suffer from fragmentation much.

Maybe some of you want to take a look at it.


[1] http://rtportal.upv.es/rtmalloc/

Lukas

Hi Lukáš,

(You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take looooooooong).

Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend,  we regularly see the error message of enblend crashing due to a memory allocation error.  We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well.

Untill now we blamed it on memory fragmentation, but maybe it's something else.

George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my "big disk" backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1.

My question to you now is: You recently did some "memory leak" patching on the hugin trunk, using cppcheck, thereby finding some "things" in celeste. You reported this via <http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80>.
Can you do the same for the enblend trunk?
If you want to do this and you find the time for it "in the near furure", be so kind to publish this on the hugin-ptx list.
But please: don't feel obliged. If you don't have time or just do not want to do this: just say so.

= If you don't want to do this, you can now stop reading.  =
If you want to do this or are at least interested: please continue reading.

Below you will find my tests on OSX. It's done on the 2.5GB bracked "village hotel" project of George Row.

Below the "tail" of the enblend error for a very recent 32bit "Christoph Spiel" build:
enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1
enblend: info: creating blend mask: 1/3enblend(11221) malloc: ***
vm_allocate(size=580620288) failed (error code=3)
enblend(11221) malloc: *** error: can't allocate region
enblend(11221) malloc: *** set a breakpoint in szone_error to debug

enblend: out of memory
enblend: St9bad_alloc
gnumake: *** [FoyleDays_M2_04.tif] Error 1


Below the "tail" of the error for the stable 32bit enblend 3.2 build (This to prove it's not a recent problem. It's already there in the 3.2 stable build).
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

enblend: out of memory
St9bad_alloc
gnumake: *** [mamaloe_exposure_00.tif] Error 1

Test report for a 64bit enblend: After 6 hours and much further in the process my system crashed. I will rerun tonight.

(I can start and monitor remote. I can't start a crashed mac from remote.)

Note: I build 32bit binaries by default as they run on every mac. A 64 bit version only runs on Leopard and Snow Leopard on 64bit hardware. 64bit brings hardly performance gain or other benefits, only when making gigapixel pano's.)

To me this does NOT mean that the 64 bit behaves better. It only proves IMO that due to the huge 64bit address space, enblend can (might) just continue leaving it 's "memory junk" without filling the address space that fast and that fragmentation is less an issue within the huge 64bit address space. In other words: it will only crash at a later stage when trying to stitch (even bigger) projects. But that's an assumption which I can't prove right now.
 

 

Hoi,

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Stefan Peter-4]

Hi Lukáš

instead of changing the RAM on your PC, you could use a virtual machine
like vmware or virtualbox. There, you can limit the resources at your will.

Regards

Stefan Peter


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Lukáš Jirkovský]

2009/10/7 Harry van der Wolf <hvdwolf@...>:
> I did another run from the Gui (took a few minutes to find how to compile
> that one). That gave more results:

I didn't know that there is a GUI for it.

>
> [assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is
> preferred to Post-Incrementing
> [assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is
> preferred to Post-Incrementing
> [vigra_impex/exr.cxx:310]: (style) Redundant condition. It is safe to
> deallocate a NULL pointer
> [vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the
> constructor 'HDRCodecImpl::height'
> [vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the
> constructor 'HDRCodecImpl::width'
> [vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited
> by class JPEGDecoderImplBase does not have a virtual destructor
> [vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited
> by class JPEGEncoderImplBase does not have a virtual destructor
> [vigra_impex/rgbe.c:90]: (style) The scope of the variable f can be limited
> [vigra_impex/tiff.cxx:199]: (style) Member variable not initialized in the
> constructor 'TIFFCodecImpl::layers'
> [vigra_impex/viff.cxx:245]: (style) Function parameter 'index' is passed by
> value. It could be passed by reference instead.

I've got these:
[enblend/include/vigra/multi_array.hxx:318]: (error) Class
MultiArrayView which is inherited by class MultiArray does not have a
virtual destructor
[enblend/src/vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl
which is inherited by class JPEGDecoderImplBase does not have a
virtual destructor
[enblend/src/vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl
which is inherited by class JPEGEncoderImplBase does not have a
virtual destructor

>
>
> This doesn't seem to be memory leaks of some kind. Still I report it here.

Thanks

>
> Harry
>
>
>
> 2009/10/7 Harry van der Wolf <hvdwolf@...>
>>
>> To answer my own mail. :-)
>>
>> I just compiled cppcheck on OSX and did a standard run on the enblend
>> trunk.
>> It displays the following
>> [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is
>> inherited by class JPEGDecoderImplBase does not have a virtual destructor
>> [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is
>> inherited by class JPEGEncoderImplBase does not have a virtual destructor
>>
>> Then I did a "full check". That resulted in exactly the same result as
>> mentioned above.
>>
>> This is like giving a monkey an encyclopedia. He has no idea what to do
>> with it. I don't even know if this is usefull.
>> I hope some clever progammers (super monkeys?) know what to do with this.
>>
>> Harry
>>
>>
>>
>> 2009/10/7 Harry van der Wolf <hvdwolf@...>
>>>
>>> 2009/10/7 Lukáš Jirkovský <l.jirkovsky@...>
>>>>
>>>>
>>>>
>>>> I'm not Mac user (although I find it really cool but very expensive)
>>>> but I may found solution for out of memory problem. I had a discussion
>>>> about memory and I mentioned these fragmentation problems on OS. And
>>>> get interesting advice – use TLSF [1]. It is said that it doesn't
>>>> suffer from fragmentation much.
>>>>
>>>> Maybe some of you want to take a look at it.
>>>>
>>>>
>>>> [1] http://rtportal.upv.es/rtmalloc/
>>>>
>>>> Lukas
>>>>
>>>>
>>>
>>> Hi Lukáš,
>>>
>>> (You are the one person who I had hoped to react. I started to do tests
>>> myself yesterday evening but they take looooooooong).
>>>
>>> Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit
>>> versions of enblend,  we regularly see the error message of enblend crashing
>>> due to a memory allocation error.  We see this both at hugin-ptx as well as
>>> in the bugtracker for large projects. We sometimes see it for enfuse in case
>>> of large projects which need to be fused as well.
>>> Untill now we blamed it on memory fragmentation, but maybe it's something
>>> else.
>>> George Row is one of the persons who encounters these errors on OSX. I
>>> have received a large set (2.5GB) from George in June, some time before my
>>> mac crashed. At that time I did some tests. These results are gone (no
>>> backup of test results).Yesterday I took Georges set from my "big disk"
>>> backup server and did some tests myself trying to stich a 12000X6000
>>> (slightly bigger) panorama in hugin-2009.4.0-beta1.
>>>
>>> My question to you now is: You recently did some "memory leak" patching
>>> on the hugin trunk, using cppcheck, thereby finding some "things" in
>>> celeste. You reported this via
>>> <http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80>.
>>> Can you do the same for the enblend trunk?
>>> If you want to do this and you find the time for it "in the near furure",
>>> be so kind to publish this on the hugin-ptx list.
>>> But please: don't feel obliged. If you don't have time or just do not
>>> want to do this: just say so.
>>>
>>> = If you don't want to do this, you can now stop reading.  =
>>> If you want to do this or are at least interested: please continue
>>> reading.
>>>
>>> Below you will find my tests on OSX. It's done on the 2.5GB bracked
>>> "village hotel" project of George Row.
>>>
>>> Below the "tail" of the enblend error for a very recent 32bit "Christoph
>>> Spiel" build:
>>> enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1
>>> enblend: info: creating blend mask: 1/3enblend(11221) malloc: ***
>>> vm_allocate(size=580620288) failed (error code=3)
>>> enblend(11221) malloc: *** error: can't allocate region
>>> enblend(11221) malloc: *** set a breakpoint in szone_error to debug
>>>
>>> enblend: out of memory
>>> enblend: St9bad_alloc
>>> gnumake: *** [FoyleDays_M2_04.tif] Error 1
>>>
>>>
>>> Below the "tail" of the error for the stable 32bit enblend 3.2 build
>>> (This to prove it's not a recent problem. It's already there in the 3.2
>>> stable build).
>>> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>> enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>>
>>> enblend: out of memory
>>> St9bad_alloc
>>> gnumake: *** [mamaloe_exposure_00.tif] Error 1
>>>
>>> Test report for a 64bit enblend: After 6 hours and much further in the
>>> process my system crashed. I will rerun tonight.
>>> (I can start and monitor remote. I can't start a crashed mac from
>>> remote.)
>>> Note: I build 32bit binaries by default as they run on every mac. A 64
>>> bit version only runs on Leopard and Snow Leopard on 64bit hardware. 64bit
>>> brings hardly performance gain or other benefits, only when making gigapixel
>>> pano's.)
>>> To me this does NOT mean that the 64 bit behaves better. It only proves
>>> IMO that due to the huge 64bit address space, enblend can (might) just
>>> continue leaving it 's "memory junk" without filling the address space that
>>> fast and that fragmentation is less an issue within the huge 64bit address
>>> space. In other words: it will only crash at a later stage when trying to
>>> stitch (even bigger) projects. But that's an assumption which I can't prove
>>> right now.
>>>
>>>
>>> Hoi,
>>> Harry
>>
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Lukáš Jirkovský]

2009/10/7 Stefan Peter <s_peter@...>:

>
> Hi Lukáš
>
> instead of changing the RAM on your PC, you could use a virtual machine
> like vmware or virtualbox. There, you can limit the resources at your will.
>
> Regards
>
> Stefan Peter
>
>
> >
>

Hi Stefan,
I know about this option but I thing changing RAM is faster. I'm
already using vmware but I don't have necessary development tools
installed on any hosted OS.

Lukas

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

2009/10/8 Lukáš Jirkovský <l.jirkovsky@...>

2009/10/7 Harry van der Wolf <hvdwolf@...>:

> I did another run from the Gui (took a few minutes to find how to compile
> that one). That gave more results:

I didn't know that there is a GUI for it.

 

 

cd into the gui directory and run "qmake; make; make install".

This off course implies that the QT development stuff and (runtime) libraries are available on your system.

 

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Lukáš Jirkovský]

2009/10/8 Harry van der Wolf <hvdwolf@...>:

>
>
> 2009/10/8 Lukáš Jirkovský <l.jirkovsky@...>
>>
>> 2009/10/7 Harry van der Wolf <hvdwolf@...>:
>> > I did another run from the Gui (took a few minutes to find how to
>> > compile
>> > that one). That gave more results:
>>
>> I didn't know that there is a GUI for it.
>>
>>
>
>
> cd into the gui directory and run "qmake; make; make install".
> This off course implies that the QT development stuff and (runtime)
> libraries are available on your system.
>
> Harry
> >
>

It seems that the package I'm using doesn't contain GUI. Maybe I'll
try it some day but the command line is quite sufficient for me ;-)

Lukas

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

I downloaded the git from danmar_cppcheck

2009/10/8 Lukáš Jirkovský <l.jirkovsky@...>

2009/10/8 Harry van der Wolf <hvdwolf@...>:

>
>
> 2009/10/8 Lukáš Jirkovský <l.jirkovsky@...>
>>
>> 2009/10/7 Harry van der Wolf <hvdwolf@...>:
>> > I did another run from the Gui (took a few minutes to find how to
>> > compile
>> > that one). That gave more results:
>>
>> I didn't know that there is a GUI for it.
>>
>>
>
>
> cd into the gui directory and run "qmake; make; make install".
> This off course implies that the QT development stuff and (runtime)
> libraries are available on your system.
>
> Harry
> >
>

It seems that the package I'm using doesn't contain GUI. Maybe I'll
try it some day but the command line is quite sufficient for me ;-)

Lukas

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Lukáš Jirkovský]

I've tried to reproduce the bug but without success.

I've remapped images. Enblend usually takes about 2-3 GB of memory
with this pano. I lowered RAM to 256MB and swap from 512MB to 195MB.
(KDE takes about 200MB itself). I've run enblend with: -m 200 -b 8196
to expose the bug even earlier. After almost 4 hours when the free
memory was always about 50MB and my PC almost didn't respond (after
key press in terminal it took about 2 minutes to draw anything) I've
not exposed the bug.

Lukas

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Kornel Benko]

Am Thursday 08 October 2009 schrieb Lukáš Jirkovský:

>
> I've tried to reproduce the bug but without success.
>
> I've remapped images. Enblend usually takes about 2-3 GB of memory
> with this pano. I lowered RAM to 256MB and swap from 512MB to 195MB.
> (KDE takes about 200MB itself). I've run enblend with: -m 200 -b 8196
> to expose the bug even earlier. After almost 4 hours when the free
> memory was always about 50MB and my PC almost didn't respond (after
> key press in terminal it took about 2 minutes to draw anything) I've
> not exposed the bug.
>
> Lukas
>

Maybe the '-m' option _hides_ the bug. At least I observed this behaviour earlier.

Now I use enblend without image cache, here the '-m' option does not make sense.

        Kornel
--
Kornel Benko
Kornel.Benko@...

[Stefan Peter-4]

Hi List,

Sorry if the following is "all known by the old hands". Please feel free
to correct me at your leisure, otherwise I may die dumb ;)

I have downloaded the project in question, and here are some results.
I used Hugin 2009.2.0.4461 on linux X86_64 with 4GB Memory and 8 GB swap
for the first test. After creating a makefile from the pto, all tests
have been conducted on the command line.

o The original project worked fine in relatively short time (~ 30
minutes, IIRC).
o After adding an additional target (blended and fused equirect), the
system started to thrash after about 1 hour and I had to cancel the job
after a runtime of 9 hours without result.
These tests where done using enblend 4.0-da9bc1a1ed87

Afterwards, I switched to Linux X86_32 with 3GB of memory and 1 GB of swap:
o The original project did not finish. An strace of the final enblend
call (the one that creates the final equirect) showed that an mmap
operation on one of the source files (400 MBytes!) failed.
o Subsequent tests with various differently compiled enblend versions
had some influence, but did not provide a remedy. I was not able to
stitch the final pano, no matter what options i used for compiling
enblend. But the point of failure shifted from Image 2 to Image 17 when
disabling openmp and enabling image cache.

As a preliminary statement, I'd say that
o Enblending/enfusing large panos with 16bit tiffs is very memory
intensive. If you don't have the RAM required, the final enblend step
will fail with an "out of memory" message".
o Having plenty of swap space is no cure: the system starts thrashing
and will get unresponsive up to the point where you cancel the job
voluntarily.
o Although I did some valgrind and dmalloc tests, I could not find any
leeks or problems beside the fact that all tiff files where accessed
using mmap. This results in a memory footprint that mostly depends on
the size of the input files.

As a temporary fix, I would recommend to use "--disable--openmp" and
"--enable-image-cache"  in order to limit the memory consumption.
However, this has a negative influence on the runtime of enblend / enfuse.
Another option is to reduce the size of the input images. It may be
sufficient to use a compression for tiffs (I have not tested this), but
reducing the color depth from 16 to 8 bit will have, most probably, the
largest influence.


The final solution for the memory problems in enblend / enfuse can only
be an access method that does not rely on being able to access the
_whole_ source files over memory maps. Without having looked at the code
responsible for this (and not being a C++ programmer at all), I think
that it should be possible to use a "window" (let's say 128 MBytes) for
accessing at least the input files.

With kind regards

Stefan Peter

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Lukáš Jirkovský]

Hi Stefan,

2009/10/8 Stefan Peter <s_peter@...>:

>
> Hi List,
>
> Sorry if the following is "all known by the old hands". Please feel free
> to correct me at your leisure, otherwise I may die dumb ;)
>
> I have downloaded the project in question, and here are some results.
> I used Hugin 2009.2.0.4461 on linux X86_64 with 4GB Memory and 8 GB swap
> for the first test. After creating a makefile from the pto, all tests
> have been conducted on the command line.
>
> o The original project worked fine in relatively short time (~ 30
> minutes, IIRC).
> o After adding an additional target (blended and fused equirect), the
> system started to thrash after about 1 hour and I had to cancel the job
> after a runtime of 9 hours without result.
> These tests where done using enblend 4.0-da9bc1a1ed87
>
> Afterwards, I switched to Linux X86_32 with 3GB of memory and 1 GB of swap:
> o The original project did not finish. An strace of the final enblend
> call (the one that creates the final equirect) showed that an mmap
> operation on one of the source files (400 MBytes!) failed.
> o Subsequent tests with various differently compiled enblend versions
> had some influence, but did not provide a remedy. I was not able to
> stitch the final pano, no matter what options i used for compiling
> enblend. But the point of failure shifted from Image 2 to Image 17 when
> disabling openmp and enabling image cache.
>
> As a preliminary statement, I'd say that
> o Enblending/enfusing large panos with 16bit tiffs is very memory
> intensive. If you don't have the RAM required, the final enblend step
> will fail with an "out of memory" message".
> o Having plenty of swap space is no cure: the system starts thrashing
> and will get unresponsive up to the point where you cancel the job
> voluntarily.
> o Although I did some valgrind and dmalloc tests, I could not find any
> leeks or problems beside the fact that all tiff files where accessed
> using mmap. This results in a memory footprint that mostly depends on
> the size of the input files.
>
> As a temporary fix, I would recommend to use "--disable--openmp" and
> "--enable-image-cache"  in order to limit the memory consumption.
> However, this has a negative influence on the runtime of enblend / enfuse.
> Another option is to reduce the size of the input images. It may be
> sufficient to use a compression for tiffs (I have not tested this), but
> reducing the color depth from 16 to 8 bit will have, most probably, the
> largest influence.
>
>
> The final solution for the memory problems in enblend / enfuse can only
> be an access method that does not rely on being able to access the
> _whole_ source files over memory maps. Without having looked at the code
> responsible for this (and not being a C++ programmer at all), I think
> that it should be possible to use a "window" (let's say 128 MBytes) for
> accessing at least the input files.
>
> With kind regards
>
> Stefan Peter
>
> >
>

Thank you very much for you in depth analysis.

Here are my thoughts. Could anyone verify them?
1. the problem occurs when image cache is disabled or the limit
exceeds (or almost fits almost exatly) memory size
2. the problem occurs when image cache is enabled but there is not
enough memory to write temporary to disk.

Both of them makes me think that it can be avoided by carefully
setting image cache to let's say available memory minus 100MB. So the
simplest solution may be setting apropriate image cache value at
runtime. But it would have it's drawbacks on systems with hot
pluggable memory.

Lukas

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Stefan Peter-4]

Hi Lukáš

Lukáš Jirkovský schrieb:
> Hi Stefan,
>
>
> Here are my thoughts. Could anyone verify them?
> 1. the problem occurs when image cache is disabled or the limit
> exceeds (or almost fits almost exatly) memory size
> 2. the problem occurs when image cache is enabled but there is not
> enough memory to write temporary to disk.

I'm still investigating, but I don't think it has something to do with
the image cache. Form what I have seen yet, the way the files are read
and written seems to be the main culprit: In my tests, a 1.2 GByte Tiff
for the resulting output file is created and accessed in its full size
using mmap. This requires 1.2 GBytes of memory allocated to the application.
Additional memory is required for every of the source files. And, of
course, the application itself will need some memory, too.
If physical memory gets tight, the system will start to swap. This leads
to the unresponsiveness you mentioned. And in some cases, this swapping
leads to thrashing: If the OS permanently is occupied with swapping in /
out of pages, not much CPU performance is left over for the application.

The effect I noticed is worsened by using MP: There, the memory
requirements occur in parallel whereas a nonMP application allocates
memory resources sequentially. Therefore, nonMP compiled enblend
versions will allow larger output files without breaking.

I have enabled the image cache only because I read that it has to be
disabled when using OpenMP.

I'd like to have a closer look at the image handling functions (which I
think are implemented somewhere in the vigra library. If there es a way
to switch this from "full size access" (you have the full file
accessible at the memory region used for the mmap) to "floating window
access" (you only have a certain amount of the file mapped into the
memory map of the application), the possible final size of the image
would almost be infinite, but requiring only the amount of memory for
the window size. Of course, performance would most probably suffer due
to the fact that the window has to be shifted into the position required
for the next write.

It actually may be that I'm on a totally wrong trace. Most information I
try to explain here rely on some strace runs of
enblend --compression NONE -v --fine-mask --fine-mask \
  -w -f12018x6009 -o Village_Hotel.tif \
  Village_Hotel0000.tif Village_Hotel0001.tif Village_Hotel0002.tif\
  Village_Hotel0003.tif Village_Hotel0004.tif Village_Hotel0005.tif\
  Village_Hotel0006.tif Village_Hotel0007.tif Village_Hotel0008.tif\
  Village_Hotel0009.tif Village_Hotel0010.tif Village_Hotel0011.tif\
  Village_Hotel0012.tif Village_Hotel0013.tif Village_Hotel0014.tif\
  Village_Hotel0015.tif Village_Hotel0016.tif Village_Hotel0017.tif\
  Village_Hotel0018.tif Village_Hotel0019.tif Village_Hotel0020.tif\
  Village_Hotel0024.tif Village_Hotel0025.tif Village_Hotel0026.tif

One notable issue may be that the mmap for Village_Hotel.tif is roughly
1.26 GBytes in size on my notebook according to strace, on the desktop
with 4GB of memory, where the above command does finish without an
error, the size of the final Village_Hotel.tif is only 551MBytes. You
see, I will have to keep investigating.


If someone else has some insight into this area, I'd be delighted to be
corrected or informed.

BTW, George, what amount of memory and swap space does your OSX
installation have? Did you encounter unresponsiveness, too?

Cheers

Stefan Peter


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[grow]

Stefan,
Thanks for your analysis of this.

You asked about memory and responsiveness.

My Mac has 5.5Gb of RAM ... when I have a large stitch to do I
sometime restart the machine and re-open the Hugin project with only
the Stitch tab open and no preview or anything else taking up
memory ... that way I have about 4Gb of free memory at the start of
the stitch.

I have a box on the shelf with 2GB more RAM that I will install
"soon"  - probably this weekend - but I said that last weekend also.
That will take the RAM to 7Gb (as I will have to take out 2x256Mb in
order to fit the the 2x1Gb  new cards).  I will of course try my
problematic Hugin projects after the upgrade to check whether it makes
a difference.

On responsiveness ... I haven't been systematic in measuring and
plotting the times.  These are all ROUGH NUMBERS FROM MEMORY - this is
NOT the result of a carefully controlled and systematically logged
analysis.

But with that disclaimer out of the way it goes something like
this ...
If I launch a Hugin-stitch of one of the problematic projects at say
2,000 x 1,000 it will take a few minutes (typically I will set a tiny-
test-stitch running ... switch windows to check email and the like ...
and by the time I check back on Hugin it has finished)
at 5,000 x 2,500 it might take perhaps 30 minutes  ... (I go
downstairs and watch an episode of The SImpsons ... when I get back to
my desk the stitch is finishing up)
at 8,000 x 4,000 it will run for about 2 hours (I go and cook and
serve dinner)
when it gets to 12,000 x 6,000 (I set it running before I go to bed
and check before breakfast in the morning  - either we have a good
final image or an error report) when I check the file creation times
it will have run for 6 or 7 hours!

So my intuition is that there is a square or possibly cubic
progression in the processing time as the final image size increases.
The processing time is proportional to something like  n^3 where n is
width of the image.  It is NOT proportional to n^4  - so it is NOT the
square of the area.

The crashing generally happens when complex masks are involved so when
I need a large image my way of getting the stitch to complete has been
to remove the masks and do Photoshop patching by hand afterwards
rather than masking before hand.

My guess is that there is something that happens in the process for
every pixel in the image - comparing the pixel to some other aspect of
the image - the mask perhaps?  If there is a bit-wize mask it might
be  that the determining factor in the processing time is the multiple
of the area of the image and the area of the mask. ... Oh that doesn't
make any sense because of course there isn't a single mask ...

When I get a chance I will try your suggestions from the other
message,

all the best

George

On Oct 9, 7:54 pm, Stefan Peter <s_pe...@...> wrote:

>...
> BTW, George, what amount of memory and swap space does your OSX
> installation have? Did you encounter unresponsiveness, too?
>
> Cheers
>
> Stefan Peter


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Robert Krawitz-2]

   Date: Fri, 9 Oct 2009 17:33:53 -0700 (PDT)
   From: grow <georgerow@...>

   Stefan,
   Thanks for your analysis of this.

   You asked about memory and responsiveness.

   My Mac has 5.5Gb of RAM ... when I have a large stitch to do I
   sometime restart the machine and re-open the Hugin project with only
   the Stitch tab open and no preview or anything else taking up
   memory ... that way I have about 4Gb of free memory at the start of
   the stitch.

   I have a box on the shelf with 2GB more RAM that I will install
   "soon"  - probably this weekend - but I said that last weekend also.
   That will take the RAM to 7Gb (as I will have to take out 2x256Mb in
   order to fit the the 2x1Gb  new cards).  I will of course try my
   problematic Hugin projects after the upgrade to check whether it makes
   a difference.

   On responsiveness ... I haven't been systematic in measuring and
   plotting the times.  These are all ROUGH NUMBERS FROM MEMORY - this is
   NOT the result of a carefully controlled and systematically logged
   analysis.

   But with that disclaimer out of the way it goes something like
   this ...
   If I launch a Hugin-stitch of one of the problematic projects at say
   2,000 x 1,000 it will take a few minutes (typically I will set a tiny-
   test-stitch running ... switch windows to check email and the like ...
   and by the time I check back on Hugin it has finished)
   at 5,000 x 2,500 it might take perhaps 30 minutes  ... (I go
   downstairs and watch an episode of The SImpsons ... when I get back to
   my desk the stitch is finishing up)
   at 8,000 x 4,000 it will run for about 2 hours (I go and cook and
   serve dinner)
   when it gets to 12,000 x 6,000 (I set it running before I go to bed
   and check before breakfast in the morning  - either we have a good
   final image or an error report) when I check the file creation times
   it will have run for 6 or 7 hours!

That's more than linear, but not by much -- N*sqrt(N).

 5000x2500   =   12.5 MP, 0.5 hr  = 25 MP/hr
 8000x4000   =   32.0 MP, 2.0 hr  = 16 MP/hr
12000x6000   =   72.0 MP, 6.0 hr  =  8 MP/hr

Or, based on the time for 12.5 MP, you'd expect the 8Kx4K to take 1.3
hours and the 72 MP to take about 3 hours if it were linear.  If it
were N^2, you'd expect 32 MP to take about 3~3.5 hours and 72 MP to
take about 15 hours.  If it were N*sqrt(N), the 32 MP should take
about 2.0 hours and the 72 MP about 6.5~7 hours -- exactly what you're
seeing.  The 2 MP image should take about 0.064 hours, or 4 minutes.
You're seeing consistent behavior all the way up.  That's not usually
a signature of memory thrashing -- usually when that happens you hit a
wall, where above that point the time may well increase by orders of
magnitude.

   So my intuition is that there is a square or possibly cubic
   progression in the processing time as the final image size
   increases.  The processing time is proportional to something like
   n^3 where n is width of the image.  It is NOT proportional to n^4 -
   so it is NOT the square of the area.

Proportional to the cube of the width (linear dimension) appears to be
correct based on these 3 or 4 data points, but since each pixel has to
be processed I'd expect to calculate complexity based on the pixel
area.  Thus, N * sqrt(N).  Perhaps the developers can explain why it's
more than linear, but it's certainly not quadratic.

--
Robert Krawitz                                     <rlk@...>

Tall Clubs International  --  http://www.tall.org/ or 1-888-IM-TALL-2
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Stefan Peter-4]

Hi George

Thank you for the info. I' sure that adding 2 GB of RAM will help you
with your current pano. But it will only shift the pano size limit up a
notch, it will not solve the problem per se. I think that it should be
possible to enblend or enfuse images of any size with a given amount of
RAM, although you may have to pay the price of increased computation
time in return.

Instead of adding RAM to your system, another solution may be to
increase the SWAP size (sometimes dubbed virtual memory or page cache)
of your system.
I don't know how this is handled under OSX; Under Linux, you use special
harddisk partitions for this purpose (but you can use files, too). They
are created during the installation and it is recommended to use a size
of about 50% of your RAM size for swap. The background of this is, if
the swap area is to large, it may happen that a system starts being
occupied mainly with moving pages from RAM to swap and back. This is
called thrashing and results in a system that does not or only very
slowly to your commands.

In this respect, I think it may be a good idea to check out your swap
settings. From the fact that you will have to remove 2x256MB, I suppose
you started wit 0.5 GB of main memory. If the size of the swap space was
fixed upon installation, you would have a meager 256MB if the 50% of RAM
size rule is used under OSX, too. With the new RAM size of 7 GBytes, you
should be able to extend that to 3.5 GBytes. In this case, enblend would
throw you the dreaded memory error only when the memory requirements of
all running applications and the OS exceed 10 GB.

Regards

Stefan Peter


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

2009/10/10 Stefan Peter <s_peter@...>

 

In this respect, I think it may be a good idea to check out your swap
settings. From the fact that you will have to remove 2x256MB, I suppose
you started wit 0.5 GB of main memory. If the size of the swap space was
fixed upon installation, you would have a meager 256MB if the 50% of RAM
size rule is used under OSX, too. With the new RAM size of 7 GBytes, you
should be able to extend that to 3.5 GBytes. In this case, enblend would
throw you the dreaded memory error only when the memory requirements of
all running applications and the OS exceed 10 GB.

Regards

Stefan Peter

OSX doesn't use a swapfile or a swap partition. It uses a swap directory (/private/var/vm). The swap directory can grow until you run out of diskspace.
It is off course possible to create separate partitions and mount and use one of those partitions as a "folder". This folder is  still not a swap partition in the linux kind of way. It's an OSX partition which you can use as swap partition.
This default configuration makes it extremely easy for the user. If you have 90GB free disk space, your used "swap space" can grow to 90GB (apart from some root "administration" space).
So it depends on how much free diskspace George has.

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---