possible memory leak in enblend & enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)

[grow]

Stefan,

Thanks for these comments ... I installed the extra RAM this
afternoon ... I'll investigate separately how MacOSX handles overall
virtual memory size ~ I THINK it is set at start-up rather than
installation ... but I have never tried to override the OS
defaults ...  so I will investigate.

Anyway ... as I said I installed the extra RAM and the G5 now has
7Gb.  Immediately on restart I launched a stitch of one of the
troublesome projects  - not the Village_Hotel project that has been
mentioned here but a much simpler one that is just 9 images (no
exposure bracketing) and Alpha Channel Masks on three of the images -
the photographer's feet in the two down shots and moving person on one
one of the horizontal image overlaps.

I set the stitch going at Hugin's recommended maximum size which was
roughly 12,080x604.  The extra RAM made a huge difference in
processing time ... I would have previously  expected this size of
stitch to take several hours but instead it took only 20 minutes or so
to reach the crash point.

So it is sort of,  good-news/bad-news. ...  adding extra RAM makes the
problematic project crash faster!

I just ran it with the usual defaults - tomorrow  I will try the
parameters that you suggested yesterday.

all the best

George


On Oct 10, 4:27 pm, Stefan Peter <s_pe...@...> wrote:

> Hi George
>
> Thank you for the info. I' sure that adding 2 GB of RAM will help you
> with your current pano. But it will only shift the pano size limit up a
> notch, it will not solve the problem per se. I think that it should be
> possible to enblend or enfuse images of any size with a given amount of
> RAM, although you may have to pay the price of increased computation
> time in return.
>
> Instead of adding RAM to your system, another solution may be to
> increase the SWAP size (sometimes dubbed virtual memory or page cache)
> of your system.
> I don't know how this is handled under OSX; Under Linux, you use special
> harddisk partitions for this purpose (but you can use files, too). They
> are created during the installation and it is recommended to use a size
> of about 50% of your RAM size for swap. The background of this is, if
> the swap area is to large, it may happen that a system starts being
> occupied mainly with moving pages from RAM to swap and back. This is
> called thrashing and results in a system that does not or only very
> slowly to your commands.
>
> In this respect, I think it may be a good idea to check out your swap
> settings. From the fact that you will have to remove 2x256MB, I suppose
> you started wit 0.5 GB of main memory. If the size of the swap space was
> fixed upon installation, you would have a meager 256MB if the 50% of RAM
> size rule is used under OSX, too. With the new RAM size of 7 GBytes, you
> should be able to extend that to 3.5 GBytes. In this case, enblend would
> throw you the dreaded memory error only when the memory requirements of
> all running applications and the OS exceed 10 GB.
>
> Regards
>
> Stefan Peter

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

--==**I'm doing my best not to start shouting**==--

George is NOT having a swap space, or RAM limitation problems.
His project has some 24 images.

Hugin calls his enblend step with:

  enblend --compression NONE -v --fine-mask --fine-mask -w \
  -f12000x6000 -o t3_exposure_00.tif t3_exposure_layers_0000.tif \
  t3_exposure_layers_0001.tif t3_exposure_layers_0002.tif \
  t3_exposure_layers_0003.tif t3_exposure_layers_0004.tif \
  t3_exposure_layers_0005.tif t3_exposure_layers_0006.tif \
  t3_exposure_layers_0007.tif t3_exposure_layers_0008.tif \
  t3_exposure_layers_0009.tif t3_exposure_layers_0010.tif \
  t3_exposure_layers_0011.tif t3_exposure_layers_0012.tif \
  t3_exposure_layers_0013.tif t3_exposure_layers_0014.tif \
  t3_exposure_layers_0015.tif t3_exposure_layers_0016.tif \
  t3_exposure_layers_0017.tif t3_exposure_layers_0018.tif \
  t3_exposure_layers_0019.tif t3_exposure_layers_0020.tif \
  t3_exposure_layers_0024.tif t3_exposure_layers_0025.tif \
  t3_exposure_layers_0026.tif


and then, after a few hours of crunching (strongly depending on the
amount of RAM you have) enblend crashes while processing *_0025.tif
....

However, if you skip loading and blending of EVERYTHING before
*_0024.tif and try to blend only '24 and '25, the exact same crash
happens.

/this/ is the commandline I'm currently using to reproduce george's
crash:

  enblend --compression NONE -v --fine-mask --fine-mask -w \
  -f12000x6000 -o t3_exposure_00.tif t3_exposure_layers_0024.tif \
  t3_exposure_layers_0025.tif


In a few moments you should be able to download the two required
images to reproduce this from:

        http://prive.bitwizard.nl/george_pics.tgz 

the size of the file is 244Mb. (255307820 bytes)

(the upload will take another hour from now... ) (on the other hand,
this is much longer than other updates to my private site, so the
automatic upload (that runs every hour) might intervene, causing the
whole procedure to take a whopping three hours. So be it. )


I have downloaded the most recent version of enblend, and compiled it
myself. I have, before I had to do things outside my house today come
to the point where I have reduced the enblend commandline to only
those two images, and verifying that it also crashes in the same way
with my home-compiled version.

I will next enable debugging and recompile my "home built" version.
Then I'll test if it still crashes with debugging enabled. Next I will
try to trace this problem and find the bug.

What I know so far is that it is really /really/ not that enblend runs
out of memory or swap space. I really /really/ have enough of that.
So does george. Enblend tries to limit itself to using only 1G of
memory. Or something like that. When the bug hits, enblend allocates
memory until the system decides that enblend is running wild. So the
system refuses more allocation requests, and enblend then errors with:
"cannot allocate memory". But the reason it ran out of memory is not
that it needs more memory but because it ran wild.

I hope to be able to put some more time into this tomorrow.

        Roger.


On Sat, Oct 10, 2009 at 06:40:58PM +0200, Harry van der Wolf wrote:

> 2009/10/10 Stefan Peter <s_peter@...>
>
>
> > In this respect, I think it may be a good idea to check out your swap
> > settings. From the fact that you will have to remove 2x256MB, I suppose
> > you started wit 0.5 GB of main memory. If the size of the swap space was
> > fixed upon installation, you would have a meager 256MB if the 50% of RAM
> > size rule is used under OSX, too. With the new RAM size of 7 GBytes, you
> > should be able to extend that to 3.5 GBytes. In this case, enblend would
> > throw you the dreaded memory error only when the memory requirements of
> > all running applications and the OS exceed 10 GB.
> >
> > Regards
> >
> > Stefan Peter
> >
> >
> OSX doesn't use a swapfile or a swap partition. It uses a swap directory
> (/private/var/vm). The swap directory can grow until you run out of
> diskspace.
> It is off course possible to create separate partitions and mount and use
> one of those partitions as a "folder". This folder is  still not a swap
> partition in the linux kind of way. It's an OSX partition which you can use
> as swap partition.
> This default configuration makes it extremely easy for the user. If you have
> 90GB free disk space, your used "swap space" can grow to 90GB (apart from
> some root "administration" space).
> So it depends on how much free diskspace George has.
>
> Harry
>
> >

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

On Sat, Oct 10, 2009 at 09:46:37AM -0700, grow wrote:
> So it is sort of,  good-news/bad-news. ...  adding extra RAM makes the
> problematic project crash faster!

On my system the "village hotel" project crashes in just over one
minute of wall clock time:


assurancetourix:~/grow> time enblend --compression NONE -v --fine-mask --fine-mask -w -f12000x6000 -o t3_exposure_00.tif t3_exposure_layers_0024.tif t3_exposure_layers_0025.tif
Input image "t3_exposure_layers_0024.tif" RGB UINT16 position=0x3133 size=12000x2867
Input image "t3_exposure_layers_0025.tif" RGB UINT16 position=0x3133 size=12000x2867
Output image size: [(0, 0) to (12000, 6000) = (12000x6000)]
Loading next image: t3_exposure_layers_0024.tif
assembled images bounding box: [(0, 3133) to (12000, 6000) = (12000x2867)]
Loading next image: t3_exposure_layers_0025.tif
assembled images bounding box: [(0, 3133) to (12000, 6000) = (12000x2867)]
image union bounding box: [(0, 3133) to (12000, 6000) = (12000x2867)]
image intersection bounding box: [(0, 3133) to (12000, 6000) = (12000x2867)]
Estimated space required for mask generation: 826MB
Creating blend mask: 1/6 2/6 3/6 4/6 5/6 6/6

enblend: out of memory
std::bad_alloc
41.286u 3.876s 1:06.40 67.9%    0+0k 9736+382568io 55pf+0w
assurancetourix:~/grow>


        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

On Sat, Oct 10, 2009 at 09:46:37AM -0700, grow wrote:
> I set the stitch going at Hugin's recommended maximum size which was
> roughly 12,080x604.  The extra RAM made a huge difference in

Ok, guys. I'm a bit further and it's bedtime for me.  If anyone wants
to continue where I left off, feel free, keep me posted.


In enblend-3.2/src/mask.h we have:

(the cout's are my instrumentation: gdb is worthless for this
problem. If anyone disagrees, feel free to teach me what I'm doing
wrong...)


        cout << "R11c";fflush (stdout);
        if (snake->front().first) {
            cout << "R16";fflush (stdout);
        // First vertex is moveable. Rotate list so that first vertex is nonmoveable.
            Segment::iterator firstNonmoveableVertex = snake->begin();
            while (firstNonmoveableVertex->first) ++firstNonmoveableVertex;

            cout << "R17";fflush (stdout);
            // Copy initial run on moveable vertices and first nonmoveable vertex to end of list.
            Segment::iterator firstNonmoveablePlusOne = firstNonmoveableVertex;
            ++firstNonmoveablePlusOne;
            cout << "R18";fflush (stdout);
            snake->insert(snake->end(), snake->begin(), firstNonmoveablePlusOne);

            cout << "R19";fflush (stdout);
            // Erase initial run of moveable vertices.
            snake->erase(snake->begin(), firstNonmoveableVertex);
            cout << "R20";fflush (stdout);
        }


Things go pearshaped between the R18 and the R19. in other words: It's
the snake->insert (and/or its arguments)...

Up to that point in time enblend is using about 1036Mb of
memory. inside that single line of code, it allocates another 2Gb of
memory and runs out when the OS refuses more memory.

Snake is "Segment". Segment is derived from "slist". I haven't gotten
around to figuring out where that "insert" code lives, and wether
execution gets there.... (if not, it's the snake->end() or one of the
others)


        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[grow]

Roger,

Thank you  ... I remember you starting on this line of analysis
earlier in the year ... it certainly looks like it is worth
pursuing ...  it is consistent with the fact that the problem only
arises when I include images with an Alpha Channel Mask and there is
something (I am not sure what) about the complexity of the mask ... if
I make a few masks whose outlines consist of a few straight lines ...
it will usually work - especially if I keep to a less-than-maximal
output file.  The crashes tend to happen as the number of mask and/or
the complexity of their shape increases,  especially if I request a
full-size output file.

all the best

George

On 10 Oct, 23:50, Rogier Wolff <rew-googlegro...@...> wrote:

> On Sat, Oct 10, 2009 at 09:46:37AM -0700, grow wrote:
> > I set the stitch going at Hugin's recommended maximum size which was
> > roughly 12,080x604.  The extra RAM made a huge difference in
>
> Ok, guys. I'm a bit further and it's bedtime for me.  If anyone wants
> to continue where I left off, feel free, keep me posted.
>
> In enblend-3.2/src/mask.h we have:
>
> (the cout's are my instrumentation: gdb is worthless for this
> problem. If anyone disagrees, feel free to teach me what I'm doing
> wrong...)
>
>         cout << "R11c";fflush (stdout);
>         if (snake->front().first) {
>             cout << "R16";fflush (stdout);
>         // First vertex is moveable. Rotate list so that first vertex is nonmoveable.
>             Segment::iterator firstNonmoveableVertex = snake->begin();
>             while (firstNonmoveableVertex->first) ++firstNonmoveableVertex;
>
>             cout << "R17";fflush (stdout);
>             // Copy initial run on moveable vertices and first nonmoveable vertex to end of list.
>             Segment::iterator firstNonmoveablePlusOne = firstNonmoveableVertex;
>             ++firstNonmoveablePlusOne;
>             cout << "R18";fflush (stdout);
>             snake->insert(snake->end(), snake->begin(), firstNonmoveablePlusOne);
>
>             cout << "R19";fflush (stdout);
>             // Erase initial run of moveable vertices.
>             snake->erase(snake->begin(), firstNonmoveableVertex);
>             cout << "R20";fflush (stdout);
>         }
>
> Things go pearshaped between the R18 and the R19. in other words: It's
> the snake->insert (and/or its arguments)...
>
> Up to that point in time enblend is using about 1036Mb of
> memory. inside that single line of code, it allocates another 2Gb of
> memory and runs out when the OS refuses more memory.
>
> Snake is "Segment". Segment is derived from "slist". I haven't gotten
> around to figuring out where that "insert" code lives, and wether
> execution gets there.... (if not, it's the snake->end() or one of the
> others)
>
>         Roger.
>
> --
> ** R.E.Wo...@... **http://www.BitWizard.nl/** [phone] **
> **    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
> *-- BitWizard writes Linux device drivers for any device you may have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
> Does it sit on the couch all day? Is it unemployed? Please be specific!
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

Let me mention that I'm good at C but I don't have much if any C++
experience. I just know the principles.

On Sat, Oct 10, 2009 at 05:14:08PM -0700, grow wrote:

> >
> >         cout << "R11c";fflush (stdout);
> >         if (snake->front().first) {
> >             cout << "R16";fflush (stdout);
> >         // First vertex is moveable. Rotate list so that first vertex is nonmoveable.
> >             Segment::iterator firstNonmoveableVertex = snake->begin();
> >             while (firstNonmoveableVertex->first) ++firstNonmoveableVertex;
> >
> >             cout << "R17";fflush (stdout);
> >             // Copy initial run on moveable vertices and first nonmoveable vertex to end of list.
> >             Segment::iterator firstNonmoveablePlusOne = firstNonmoveableVertex;
> >             ++firstNonmoveablePlusOne;
> >             cout << "R18";fflush (stdout);
> >             snake->insert(snake->end(), snake->begin(), firstNonmoveablePlusOne);
> >
> >             cout << "R19";fflush (stdout);
> >             // Erase initial run of moveable vertices.
> >             snake->erase(snake->begin(), firstNonmoveableVertex);
> >             cout << "R20";fflush (stdout);
> >         }

Segment is an slist, which seems to be standard C++.

OK. I was fearing a thousand-entry list as 'snake', and having to
step through all that. Not so... When things go wrong:

the snake is:
   1<10322,2103> 0<10320,2087>

So first nonmovable is the second entry.

So firstNonmoveablePlusOne is snake.end()

and then when trying to insert the part of the list upto-end just
before end().... you end up in an infinite loop. This code doesn't work
if the list is just two entries long.

My C experience tells me that when you code things like this just
right, you shouldn't need, any special cases. However my C++
experience is not good enough to know how to code this so that special
casing is not required.

Anyway, I'll try to create some special-case code for the two element
list that needs to be swapped.

On the other hand, maybe a snake of only two elements is
nonsensical. I have not studied the code enough to know what it's
doing here. I just traced the code to find where and why it's
crashing. So maybe after adding the code to just swap the two
elements, which it is trying to do, things will crash just a little
bit further on because two element lists is never expected. We'll see.

        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

On Sat, Oct 10, 2009 at 05:14:08PM -0700, grow wrote:
> output file.  The crashes tend to happen as the number of mask and/or
> the complexity of their shape increases,  especially if I request a
> full-size output file.

It seems you have so many (157) seams that some of them are quite
short (only two entries). Apparently two entries is not a problem,
except for the situation where they are the wrong way around: The
first should not be moveable(*). So having corrected the code for
making sure that the first one is (non)movable, enblend runs just
fine.

The patch for enblend is attached.

George's village hotel now stitches just fine at 12000x6000.


        Roger.


(*) Or the other way around, I don't care. Too many negatives makes my
head spin.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

---

diff --exclude config.log -ur enblend-enfuse-3.2/src/mask.h enblend-enfuse-3.2-rew/src/mask.h
--- enblend-enfuse-3.2/src/mask.h 2008-03-02 22:19:47.000000000 +0100
+++ enblend-enfuse-3.2-rew/src/mask.h 2009-10-11 12:04:28.000000000 +0200
@@ -512,9 +512,15 @@
 
             // Copy initial run on moveable vertices and first nonmoveable vertex to end of list.
             Segment::iterator firstNonmoveablePlusOne = firstNonmoveableVertex;
-            ++firstNonmoveablePlusOne;
-            snake->insert(snake->end(), snake->begin(), firstNonmoveablePlusOne);
 
+    // This (the else clause of this if) used to crash on
+    // two-element lists. I'd rather not special case this
+    // situation, but I personally don't know any better. Sorry. --REW
+    if (++firstNonmoveablePlusOne == snake->end()) {
+      snake->insert_after (firstNonmoveableVertex, snake->begin(), firstNonmoveableVertex);
+    } else {
+      snake->insert(snake->end(), snake->begin(), firstNonmoveablePlusOne);
+    }
             // Erase initial run of moveable vertices.
             snake->erase(snake->begin(), firstNonmoveableVertex);
         }

---

[grow]

Roger,

WOW!

all the best

George

On 11 Oct, 12:10, Rogier Wolff <rew-googlegro...@...> wrote:

> On Sat, Oct 10, 2009 at 05:14:08PM -0700, grow wrote:
> > output file.  The crashes tend to happen as the number of mask and/or
> > the complexity of their shape increases,  especially if I request a
> > full-size output file.
>
> It seems you have so many (157) seams that some of them are quite
> short (only two entries). Apparently two entries is not a problem,
> except for the situation where they are the wrong way around: The
> first should not be moveable(*). So having corrected the code for
> making sure that the first one is (non)movable, enblend runs just
> fine.
>
> The patch for enblend is attached.
>
> George's village hotel now stitches just fine at 12000x6000.
>
>         Roger.
>
> (*) Or the other way around, I don't care. Too many negatives makes my
> head spin.
>
> --
> ** R.E.Wo...@... **http://www.BitWizard.nl/** [phone] **
> **    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
> *-- BitWizard writes Linux device drivers for any device you may have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
> Does it sit on the couch all day? Is it unemployed? Please be specific!
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
>
>  infinite-fix.diff
> 1KViewDownload

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[grow]

Roger,
(I will now display my naivety about the development process.)

This seemed like you have got to the underlying bug.
Do we need to do something else to get this fix into the next Hugin
release?  Does it need to be formally entered in the bug tracker ... I
vaguely remember Harry - I think it was - logging a bug when this
first came up  ... or will someone appropriate have noticed and just
incorporated it?

all the best

George

On 11 Oct, 12:10, Rogier Wolff <rew-googlegro...@...> wrote:

> On Sat, Oct 10, 2009 at 05:14:08PM -0700, grow wrote:
> > output file.  The crashes tend to happen as the number of mask and/or
> > the complexity of their shape increases,  especially if I request a
> > full-size output file.
>
> It seems you have so many (157) seams that some of them are quite
> short (only two entries). Apparently two entries is not a problem,
> except for the situation where they are the wrong way around: The
> first should not be moveable(*). So having corrected the code for
> making sure that the first one is (non)movable, enblend runs just
> fine.
>
> The patch for enblend is attached.
>
> George's village hotel now stitches just fine at 12000x6000.
>
>         Roger.
>
> (*) Or the other way around, I don't care. Too many negatives makes my
> head spin.
>
> --
> ** R.E.Wo...@... **http://www.BitWizard.nl/** [phone] **
> **    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
> *-- BitWizard writes Linux device drivers for any device you may have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
> Does it sit on the couch all day? Is it unemployed? Please be specific!
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
>
>  infinite-fix.diff
> 1KViewDownload

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

On Mon, Oct 12, 2009 at 04:01:13PM -0700, grow wrote:

>
> Roger,
> (I will now display my naivety about the development process.)
>
> This seemed like you have got to the underlying bug.
> Do we need to do something else to get this fix into the next Hugin
> release?  Does it need to be formally entered in the bug tracker ... I
> vaguely remember Harry - I think it was - logging a bug when this
> first came up  ... or will someone appropriate have noticed and just
> incorporated it?

I have the impression that emblend is just another open source tool
that belongs in the hugin package. Or should I say just a tool that
hugin depends on.  I googled for enblend home page, and got sent
somewhere that had a bug tracker link. I submitted the bug and fix
there.

I got a reply from a guy named Chris who CC'ed Harry and Yuv. He said
he included the patch, and that he ported it to the source of the next
major release which is going to be 4.0. (I submitted the patch against
3.22)

Anyway, Chris said that he couldn't reproduce the bug even with the
two village hotel images that I published. So I replied with some
possible causes for that (- not the right options - his slist
implementation not having the same behaviour - his enblend not
generating the two-item "snake"). I haven't had a reply (yet).

So I think I've got it nailed.

        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[cspiel]

Roger -

On Oct 13, 8:47 am, Rogier Wolff <rew-googlegro...@...>
wrote:
> On Mon, Oct 12, 2009 at 04:01:13PM -0700, grow wrote:
> I have the impression that emblend is just another open source tool
> that belongs in the hugin package. Or should I say just a tool that
> hugin depends on.

1. Enblend and Enfuse are Open Source and are
   even GPLed.
2. They DO NOT belong to the Hugin package.
3. Strictly speaking Hugin does not depend on
   them, but currently they are the most popular
   "backends" for Hugin.


> I got a reply from a guy named Chris who CC'ed Harry and Yuv.

        This guy is me.  I'm something like the
Release Manager of Enblend/Enfuse version 4.0.

> He said he included the patch, and that he ported it to the source of the next
> major release which is going to be 4.0. (I submitted the patch against
> 3.22)

1. I said that I included a modified version of
   your patch in my _local_ stack of patches.
   It has not been put into the hg repository,
   yet.
2. An Enblend version called "3.22" never has
   existed.


> Anyway, Chris said that he couldn't reproduce the bug even with the
> two village hotel images that I published.

        This is correct.  Neither did I run into
problems with SMP/OpenMP nor without it.
Furthermore, the examples you gave, work for me
even with "--fine-mask".  However, the seam
lines there raise other questions.  8-O


> So I replied with some
> possible causes for that (- not the right options - his slist
> implementation not having the same behaviour - his enblend not
> generating the two-item "snake"). I haven't had a reply (yet).

        Sorry, but your e-mail never reached me.
Could you please be so kind and re-send it?

Thanks,
        Chris

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

On Tue, Oct 13, 2009 at 01:53:51AM -0700, cspiel wrote:

>
> Roger -
>
> On Oct 13, 8:47 am, Rogier Wolff <rew-googlegro...@...>
> wrote:
> > On Mon, Oct 12, 2009 at 04:01:13PM -0700, grow wrote:
> > I have the impression that emblend is just another open source tool
> > that belongs in the hugin package. Or should I say just a tool that
> > hugin depends on.
>
> 1. Enblend and Enfuse are Open Source and are
>    even GPLed.
> 2. They DO NOT belong to the Hugin package.

On the other hand, they do go "hand-in-hand" from an enduser point-of
view.

> 3. Strictly speaking Hugin does not depend on
>    them, but currently they are the most popular
>    "backends" for Hugin.

Agreed. This is the nice thing with open source and alternatives.

Strictly speaking hugin depends on "a blender" and "enblend" happens
to be one. So in apt-language, enblend provides "ablender", and hugin
depends on "ablender".

> > I got a reply from a guy named Chris who CC'ed Harry and Yuv.
>
>         This guy is me.  I'm something like the
> Release Manager of Enblend/Enfuse version 4.0.

:-)

> > He said he included the patch, and that he ported it to the source of
> > the next
> > major release which is going to be 4.0. (I submitted the patch against
> > 3.22)
 
> 1. I said that I included a modified version of
>    your patch in my _local_ stack of patches.
>    It has not been put into the hg repository,
>    yet.

Oh. ok.

> 2. An Enblend version called "3.22" never has
>    existed.

Oh! My bad. I apparently had too much to drink. It's 3.2 I was talking
about.

> > Anyway, Chris said that he couldn't reproduce the bug even with the
> > two village hotel images that I published.
>
>         This is correct.  Neither did I run into
> problems with SMP/OpenMP nor without it.
> Furthermore, the examples you gave, work for me
> even with "--fine-mask".  However, the seam
> lines there raise other questions.  8-O

OK.
 
> > So I replied with some
> > possible causes for that (- not the right options - his slist
> > implementation not having the same behaviour - his enblend not
> > generating the two-item "snake"). I haven't had a reply (yet).
>
>         Sorry, but your e-mail never reached me.
> Could you please be so kind and re-send it?

OK. Will do.

        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

2009/10/13 grow <georgerow@...>

Roger,
(I will now display my naivety about the development process.)

This seemed like you have got to the underlying bug.
Do we need to do something else to get this fix into the next Hugin
release?  Does it need to be formally entered in the bug tracker ... I
vaguely remember Harry - I think it was - logging a bug when this
first came up  ... or will someone appropriate have noticed and just
incorporated it?

all the best

George

Hi George,

I had a look at the patch and as such it's easily to implement and build enblend with the patch. For the time being I will await Christoph Spiels actions as he also modified the patch for the current enblend version that is being used but he did npt yet add it to the enblend trunk.
I assume he has a good reason for this modification.

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[cspiel]

On Oct 13, 10:52 pm, Harry van der Wolf <hvdw...@...> wrote:
> For the time being I will await Christoph Spiel's actions as he also
> modified the patch for the current enblend version that is being used
> but he did not yet add it to the Enblend trunk.

        The change has just been committed.
See, e.g.,
        http://enblend.hg.sourceforge.net/hgweb/enblend/enblend/diff/75de5e1ffce6/src/mask.h


> I assume he has a good reason for this modification.

        Oh, I just wanted to follow the DRY
principle and to heed Roger's concerns about the
runtime penalty of an if-clause inside the loop
of all snake segments.  Nothing fancy, really.


/Chris

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

Hi Chris,


On Wed, Oct 14, 2009 at 12:55:23AM -0700, cspiel wrote:
> Oh, I just wanted to follow the DRY principle and to heed Roger's
> concerns about the runtime penalty of an if-clause inside the loop
> of all snake segments.  Nothing fancy, really.

I wasn't concerned about the runtime impications of the if. I simply
dislike a special case for a situation that should be possible to
handle in the main loop.

so strcmp is defined to be implemented as:

        while (*a && (*a == *b)) {
                a++; b++
        }
        return *a-*b;

No speclial cases, quickly handle end-of-string etc etc. Nice and
neat.

Similarly, the case of rotating a list that has only one element on
one of the ends should not need special casing.

I was more concerned with the "neatness" of the code, and the size of
the code base. If the program grows and grows, it will too quickly
become unmaintainable.

        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Andrew Mihal-2]

Hi,
    A correct fix will require determining the cause of the two-point
snake. A two-point polygon has zero area, so it is unclear what region
of the mask this is outlining. Perhaps the mask has isolated
single-pixel spots of black and white? E.g. if the user set the input
alpha masks with spots like this. Maybe the contour iterator does not
handle this case gracefully.

Andrew

On Wed, Oct 14, 2009 at 6:51 AM, Rogier Wolff
<rew-googlegroups@...> wrote:

>
>
> Hi Chris,
>
>
> On Wed, Oct 14, 2009 at 12:55:23AM -0700, cspiel wrote:
>> Oh, I just wanted to follow the DRY principle and to heed Roger's
>> concerns about the runtime penalty of an if-clause inside the loop
>> of all snake segments.  Nothing fancy, really.
>
> I wasn't concerned about the runtime impications of the if. I simply
> dislike a special case for a situation that should be possible to
> handle in the main loop.
>
> so strcmp is defined to be implemented as:
>
>        while (*a && (*a == *b)) {
>                a++; b++
>        }
>        return *a-*b;
>
> No speclial cases, quickly handle end-of-string etc etc. Nice and
> neat.
>
> Similarly, the case of rotating a list that has only one element on
> one of the ends should not need special casing.
>
> I was more concerned with the "neatness" of the code, and the size of
> the code base. If the program grows and grows, it will too quickly
> become unmaintainable.
>
>        Roger.
>
> --
> ** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
> **    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
> *-- BitWizard writes Linux device drivers for any device you may have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
> Does it sit on the couch all day? Is it unemployed? Please be specific!
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[cspiel]

Andrew -

On Oct 14, 5:49 pm, Andrew Mihal <andrewcmi...@...> wrote:
> A two-point polygon has zero area, so it is unclear what region
> of the mask this is outlining.

        This was the very reason, why I
hesitated to immediately apply the patch.  I
pondered on a kind of snake cleanup, too.

However, it is conceivable that the endpoints of
a 2-point polygon, i.e., a line join the
borders of the overlap region.  Admittedly, this
is an extreme but valid case, and I could not
convince myself why it should be impossible.
Thus, I decided to apply Roger's patch.


/Chris

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Harry van der Wolf-3]

2009/10/10 Rogier Wolff <rew-googlegroups@...>

However, if you skip loading and blending of EVERYTHING before
*_0024.tif and try to blend only '24 and '25, the exact same crash
happens.

/this/ is the commandline I'm currently using to reproduce george's
crash:

 enblend --compression NONE -v --fine-mask --fine-mask -w \
 -f12000x6000 -o t3_exposure_00.tif t3_exposure_layers_0024.tif \
 t3_exposure_layers_0025.tif

 

Edit: Monday 13 October
I tried to reproduce the crash on MacOSX 10.5.8 on an Intel MacBook Pro (where George has a MacOSX 10.4.11 PPC based G5).

In my bundles I already use the Christoph Spiel builds for enblend/enfuse for quite some time, where you (Rogier) use the standard final release 3.2 version.

I did several tests on the complete "village hotel" set with both the standard 3.2 versions (32bit and 64bit) as with several "Christoph Spiel" versions. In all cases the total set crashes.

(As a side note: the 32 bit versions crash when trying to allocate more than 4 GB real+virtual memory. The 64bit version made my system crash when using 3.3GB real memory and 41+GB virtual swap area.)

 

Thanks to Rogier's tests we have pinpointed the problem more accurately.

In my case I used the exact same command (slightly different filenames) with unpatcjed versions of the standard 3.2 build and the latest "Christoph Spiel" build.

 

The latest Christoph Spiel build for me is (using enblend -v --version):

 

I did the following tests:

• When using the 3.2 version from the command line the blending crashes.
• The mentioned two images with the latest unpatched enblend from the command line doesn't crash and blends fine. Note: this is a 32bit Universal version.
• I built the hugin-2009.4.0-RC1 (to be released tonight or so) with the latest enblend BEFORE patching. Hugin crashes again.
  

The weird phenomenon now is that the latest build before patching blends the 2 images correctly but still breaks on the entire set.

Edit: Wednesday 15 October
Christoph released his patch yesterday morning so I built that enblend/enfuse and then built the hugin-2009.4.0-RC1 with the patched version.

• The 2 images test with patched enblend from the command line: it blends fine and delivers a blended image.
• The complete set from Hugin2009.4.0-RC1 with the patched version: it still crashes enblend on the 24th image
• The complete set from the command line with the patched version: It still crashes on the 24th image (using 247! distinct seams)
  

This also contributes to the fact that Christoph could not reproduce the error as the later enblend builds do blend the 2 separate images correctly. The 3 versions ( 2 unpatched, 1 patched) I tested do that for me too.

 

 

I'll try to release the hugin2009.4.0-RC1 build tonight.

As I will go on holiday this saturday I don't have time time to build a 64bit version to see how much memory enblend is trying to use. If the new patched 64bit version uses, for example, 5GB and than finishes fine, it means that we need to switch to 64bit builds for larger images and the user needs to be at least on Leopard. If it still runs "into infinity" (and make my system crash) than the issue is still not solved.

 

Harry

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Rogier Wolff]

On Thu, Oct 15, 2009 at 03:27:07PM +0200, Harry van der Wolf wrote:

> As I will go on holiday this saturday I don't have time time to
> build a 64bit version to see how much memory enblend is trying to
> use. If the new patched 64bit version uses, for example, 5GB and
> than finishes fine, it means that we need to switch to 64bit builds
> for larger images and the user needs to be at least on Leopard. If
> it still runs "into infinity" (and make my system crash) than the
> issue is still not solved.

I don't think enblend needs much memory. On my system I use the
default -m 1024, which tells enblend to limit memory usage to 1Gb of
memory. In that case, it uses 1036 Mb of memory on my
system. Apparently It limits "image data" to 1Gb, and uses a few
megabytes for storage of other stuff (program seam lists, etc).

When it crashes with out of memory, it's been doing something with
slists that the slist library can't handle.

What do I need to do to reproduce? I'll track this one down as well.
(I still have george's images).

        Roger.

--
** R.E.Wolff@... ** http://www.BitWizard.nl/ ** [phone] **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---

[Stefan Peter-4]

Hi Roger

Rogier Wolff wrote:

>
> I don't think enblend needs much memory. On my system I use the
> default -m 1024, which tells enblend to limit memory usage to 1Gb of
> memory. In that case, it uses 1036 Mb of memory on my
> system. Apparently It limits "image data" to 1Gb, and uses a few
> megabytes for storage of other stuff (program seam lists, etc).

I did come checks with strace to this end. On my 32bit Linux, enblend
consumes at max 751.45 MB for data space. However, it uses mmap2 for
accessing the files to blend, and this consumes another 3927.75 MBytes
of RAM.
The values for the patched version differ in data space consumption
only: it uses 704.77 MBytes.

Therefore, I conclude that the -m switch does only limit the use of
memory for data space. Depending on the size of the files you try to
blend, the total memory footprint can be much higher.

Cheers

Stefan Peter


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
To post to this group, send email to hugin-ptx@...
To unsubscribe from this group, send email to hugin-ptx+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/hugin-ptx
-~----------~----~----~----~------~----~------~--~---