cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0

[Justin P. Mattock]

Hello,
I'm not sure how to handle this,
while compiling firefox-3.6b1.source
I get this with the default compiling options,
as well as custom:


[  532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
[  532.942330] Pid: 16002, comm: cc1plus Tainted: P
2.6.32-rc5-00083-g04ea458 #2
[  532.942333] Call Trace:
[  532.942342]  [<ffffffff810bce11>] T.417+0x7c/0x245
[  532.942347]  [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
[  532.942352]  [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
[  532.942357]  [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
[  532.942363]  [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
[  532.942369]  [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
[  532.942373]  [<ffffffff813ddb25>] page_fault+0x25/0x30
[  532.942376] Mem-Info:
[  532.942378] DMA per-cpu:
[  532.942380] CPU    0: hi:    0, btch:   1 usd:   0
[  532.942383] CPU    1: hi:    0, btch:   1 usd:   0
[  532.942385] DMA32 per-cpu:
[  532.942388] CPU    0: hi:  186, btch:  31 usd:  94
[  532.942391] CPU    1: hi:  186, btch:  31 usd:  23
[  532.942393] Normal per-cpu:
[  532.942395] CPU    0: hi:  186, btch:  31 usd: 150
[  532.942398] CPU    1: hi:  186, btch:  31 usd: 155
[  532.942404] active_anon:707575 inactive_anon:264673 isolated_anon:0
[  532.942406]  active_file:58 inactive_file:33 isolated_file:0
[  532.942407]  unevictable:0 dirty:0 writeback:0 unstable:0 buffer:71
[  532.942408]  free:6998 slab_reclaimable:2697 slab_unreclaimable:16267
[  532.942409]  mapped:136 shmem:64 pagetables:2761 bounce:0
[  532.942418] DMA free:15944kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15360kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[  532.942425] lowmem_reserve[]: 0 2976 3986 3986
[  532.942436] DMA32 free:10020kB min:6020kB low:7524kB high:9028kB
active_anon:2360492kB inactive_anon:590196kB active_file:84kB
inactive_file:68kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:3047792kB mlocked:0kB dirty:0kB
writeback:0kB mapped:136kB shmem:80kB slab_reclaimable:0kB
slab_unreclaimable:88kB kernel_stack:0kB pagetables:6100kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448
all_unreclaimable? yes
[  532.942443] lowmem_reserve[]: 0 0 1010 1010
[  532.942454] Normal free:2028kB min:2040kB low:2548kB high:3060kB
active_anon:469808kB inactive_anon:468496kB active_file:148kB
inactive_file:64kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB
writeback:0kB mapped:408kB shmem:176kB slab_reclaimable:10788kB
slab_unreclaimable:64972kB kernel_stack:800kB pagetables:4944kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:413
all_unreclaimable? yes
[  532.942461] lowmem_reserve[]: 0 0 0 0
[  532.942465] DMA: 2*4kB 2*8kB 3*16kB 2*32kB 3*64kB 2*128kB 2*256kB
1*512kB 2*1024kB 2*2048kB 2*4096kB = 15944kB
[  532.942478] DMA32: 7*4kB 7*8kB 5*16kB 4*32kB 2*64kB 1*128kB 1*256kB
4*512kB 3*1024kB 0*2048kB 1*4096kB = 10020kB
[  532.942490] Normal: 507*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028kB
[  532.942501] 7974 total pagecache pages
[  532.942503] 7778 pages in swap cache
[  532.942506] Swap cache stats: add 112290, delete 104512, find 3595/4051
[  532.942508] Free swap  = 0kB
[  532.942510] Total swap = 431632kB
[  532.957941] 1048576 pages RAM
[  532.957943] 40518 pages reserved
[  532.957945] 312 pages shared
[  532.957947] 1000176 pages non-shared
[  532.957951] Out of memory: kill process 16001 (c++) score 543727 or a child
[  532.957955] Killed process 16002 (cc1plus)

I just compiled the latest gcc snapshot a few days
ago.


--
Justin P. Mattock

[Jiri Slaby]

On 11/02/2009 10:29 PM, Justin Mattock wrote:

> [  532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
> [  532.942330] Pid: 16002, comm: cc1plus Tainted: P
> 2.6.32-rc5-00083-g04ea458 #2
> [  532.942333] Call Trace:
> [  532.942342]  [<ffffffff810bce11>] T.417+0x7c/0x245
> [  532.942347]  [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
> [  532.942352]  [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
> [  532.942357]  [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
> [  532.942363]  [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
> [  532.942369]  [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
> [  532.942373]  [<ffffffff813ddb25>] page_fault+0x25/0x30

...
> [  532.957951] Out of memory: kill process 16001 (c++) score 543727 or a child
> [  532.957955] Killed process 16002 (cc1plus)
>
> I just compiled the latest gcc snapshot a few days
> ago.

How many jobs did you run in parallel? Was there anything else memory
consuming running on that machine? Do you run the same jobs count every
time you compile such big c++ projects?

[Justin P. Mattock]

On Mon, Nov 2, 2009 at 1:49 PM, Jiri Slaby <jirislaby@...> wrote:

> On 11/02/2009 10:29 PM, Justin Mattock wrote:
>> [  532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
>> [  532.942330] Pid: 16002, comm: cc1plus Tainted: P
>> 2.6.32-rc5-00083-g04ea458 #2
>> [  532.942333] Call Trace:
>> [  532.942342]  [<ffffffff810bce11>] T.417+0x7c/0x245
>> [  532.942347]  [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
>> [  532.942352]  [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
>> [  532.942357]  [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
>> [  532.942363]  [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
>> [  532.942369]  [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
>> [  532.942373]  [<ffffffff813ddb25>] page_fault+0x25/0x30
> ...
>> [  532.957951] Out of memory: kill process 16001 (c++) score 543727 or a child
>> [  532.957955] Killed process 16002 (cc1plus)
>>
>> I just compiled the latest gcc snapshot a few days
>> ago.
>
> How many jobs did you run in parallel? Was there anything else memory
> consuming running on that machine? Do you run the same jobs count every
> time you compile such big c++ projects?
>

This would be the only job running.

as for other types of jobs under ps aux there's only
udev,dbus,acpid,sshd,wicd running.

Now with this oom-killer I'm
hitting this on an imac9,1 with
gcc (GCC) 4.5.0 20091029 (experimental)
2.6.32-rc5-00083-g04ea458

On my other machine macbook pro ati chipset
(same system pure64)firefox seems to be going a lot farther
than on the imac(doing this in the past I think this takes a good
45min to compile).

the macbook has:
gcc (GCC) Cross-LFS 4.4.1.20090722) 4.4.1
2.6.32-rc4-00039-ga3f6f2e



--
Justin P. Mattock

[Jiri Slaby]

On 11/02/2009 11:02 PM, Justin Mattock wrote:
> Now with this oom-killer I'm
> hitting this on an imac9,1 with
> gcc (GCC) 4.5.0 20091029 (experimental)

So there is probably a leak in the gcc chain. Does this happen with a
stable gcc version?

[Justin P. Mattock]

Jiri Slaby wrote:

> On 11/02/2009 11:02 PM, Justin Mattock wrote:
>    
>> Now with this oom-killer I'm
>> hitting this on an imac9,1 with
>> gcc (GCC) 4.5.0 20091029 (experimental)
>>      
>
> So there is probably a leak in the gcc chain. Does this happen with a
> stable gcc version?
>
>    

well right now I'(on the macbook), seems to be going for a good ten
minutes before
I hit an error with a header file(nss)

I'll see if I can get this thing to compile all the way through.
(knock on wood) then go from there.

Justin P. Mattock

[Justin P. Mattock]

Jiri Slaby wrote:

> On 11/02/2009 11:02 PM, Justin Mattock wrote:
>    
>> Now with this oom-killer I'm
>> hitting this on an imac9,1 with
>> gcc (GCC) 4.5.0 20091029 (experimental)
>>      
>
> So there is probably a leak in the gcc chain. Does this happen with a
> stable gcc version?
>
>    

o.k. I think it's something with the latest gcc(snapshot)
right now she's been compiling firefox for 45min
(then crapped out because I compiled nss without sqlite)
without no omm-killer.

Justin P. Mattock

[KOSAKI Motohiro]

> Hello,
> I'm not sure how to handle this,
> while compiling firefox-3.6b1.source
> I get this with the default compiling options,
> as well as custom:
>
>
> [  532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
> [  532.942330] Pid: 16002, comm: cc1plus Tainted: P
> 2.6.32-rc5-00083-g04ea458 #2
> [  532.942333] Call Trace:
> [  532.942342]  [<ffffffff810bce11>] T.417+0x7c/0x245
> [  532.942347]  [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
> [  532.942352]  [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
> [  532.942357]  [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
> [  532.942363]  [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
> [  532.942369]  [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
> [  532.942373]  [<ffffffff813ddb25>] page_fault+0x25/0x30
> [  532.942376] Mem-Info:
> [  532.942378] DMA per-cpu:
> [  532.942380] CPU    0: hi:    0, btch:   1 usd:   0
> [  532.942383] CPU    1: hi:    0, btch:   1 usd:   0
> [  532.942385] DMA32 per-cpu:
> [  532.942388] CPU    0: hi:  186, btch:  31 usd:  94
> [  532.942391] CPU    1: hi:  186, btch:  31 usd:  23
> [  532.942393] Normal per-cpu:
> [  532.942395] CPU    0: hi:  186, btch:  31 usd: 150
> [  532.942398] CPU    1: hi:  186, btch:  31 usd: 155
> [  532.942404] active_anon:707575 inactive_anon:264673 isolated_anon:0
> [  532.942406]  active_file:58 inactive_file:33 isolated_file:0

file cache (active_file + inactive_file) was very little. It indicate anyone waste too much memory.
I doubt you use buggy compiler.

> [  532.942407]  unevictable:0 dirty:0 writeback:0 unstable:0 buffer:71
> [  532.942408]  free:6998 slab_reclaimable:2697 slab_unreclaimable:16267
> [  532.942409]  mapped:136 shmem:64 pagetables:2761 bounce:0
> [  532.942418] DMA free:15944kB min:28kB low:32kB high:40kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15360kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? yes
> [  532.942425] lowmem_reserve[]: 0 2976 3986 3986
> [  532.942436] DMA32 free:10020kB min:6020kB low:7524kB high:9028kB
> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
> inactive_file:68kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:3047792kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:136kB shmem:80kB slab_reclaimable:0kB
> slab_unreclaimable:88kB kernel_stack:0kB pagetables:6100kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448
> all_unreclaimable? yes
> [  532.942443] lowmem_reserve[]: 0 0 1010 1010
> [  532.942454] Normal free:2028kB min:2040kB low:2548kB high:3060kB
> active_anon:469808kB inactive_anon:468496kB active_file:148kB
> inactive_file:64kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:408kB shmem:176kB slab_reclaimable:10788kB
> slab_unreclaimable:64972kB kernel_stack:800kB pagetables:4944kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:413
> all_unreclaimable? yes
> [  532.942461] lowmem_reserve[]: 0 0 0 0
> [  532.942465] DMA: 2*4kB 2*8kB 3*16kB 2*32kB 3*64kB 2*128kB 2*256kB
> 1*512kB 2*1024kB 2*2048kB 2*4096kB = 15944kB
> [  532.942478] DMA32: 7*4kB 7*8kB 5*16kB 4*32kB 2*64kB 1*128kB 1*256kB
> 4*512kB 3*1024kB 0*2048kB 1*4096kB = 10020kB
> [  532.942490] Normal: 507*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
> 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028kB
> [  532.942501] 7974 total pagecache pages
> [  532.942503] 7778 pages in swap cache
> [  532.942506] Swap cache stats: add 112290, delete 104512, find 3595/4051
> [  532.942508] Free swap  = 0kB
> [  532.942510] Total swap = 431632kB
> [  532.957941] 1048576 pages RAM
> [  532.957943] 40518 pages reserved
> [  532.957945] 312 pages shared
> [  532.957947] 1000176 pages non-shared
> [  532.957951] Out of memory: kill process 16001 (c++) score 543727 or a child
> [  532.957955] Killed process 16002 (cc1plus)
>
> I just compiled the latest gcc snapshot a few days
> ago.

[Justin P. Mattock]

KOSAKI Motohiro wrote:

>> Hello,
>> I'm not sure how to handle this,
>> while compiling firefox-3.6b1.source
>> I get this with the default compiling options,
>> as well as custom:
>>
>>
>> [  532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
>> [  532.942330] Pid: 16002, comm: cc1plus Tainted: P
>> 2.6.32-rc5-00083-g04ea458 #2
>> [  532.942333] Call Trace:
>> [  532.942342]  [<ffffffff810bce11>] T.417+0x7c/0x245
>> [  532.942347]  [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
>> [  532.942352]  [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
>> [  532.942357]  [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
>> [  532.942363]  [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
>> [  532.942369]  [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
>> [  532.942373]  [<ffffffff813ddb25>] page_fault+0x25/0x30
>> [  532.942376] Mem-Info:
>> [  532.942378] DMA per-cpu:
>> [  532.942380] CPU    0: hi:    0, btch:   1 usd:   0
>> [  532.942383] CPU    1: hi:    0, btch:   1 usd:   0
>> [  532.942385] DMA32 per-cpu:
>> [  532.942388] CPU    0: hi:  186, btch:  31 usd:  94
>> [  532.942391] CPU    1: hi:  186, btch:  31 usd:  23
>> [  532.942393] Normal per-cpu:
>> [  532.942395] CPU    0: hi:  186, btch:  31 usd: 150
>> [  532.942398] CPU    1: hi:  186, btch:  31 usd: 155
>> [  532.942404] active_anon:707575 inactive_anon:264673 isolated_anon:0
>> [  532.942406]  active_file:58 inactive_file:33 isolated_file:0
>>      
>
> file cache (active_file + inactive_file) was very little. It indicate anyone waste too much memory.
> I doubt you use buggy compiler.
>
>
>
>    

hmm... then this is something with firefox then..
In that case I'll continue to build my system with
this compiler.
Although a bit concerned building everything
for the system with a compiler that shows some
memory issue,but if you say its not the compiler,
then I'll carry on with what I'm doing.
(and use an older compiler for firefox).

Justin P. Mattock

[Andrew Morton-9]

On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock <justinmattock@...> wrote:

> Hello,
> I'm not sure how to handle this,
> while compiling firefox-3.6b1.source
> I get this with the default compiling options,
> as well as custom:
>
> ...
>
> active_anon:2360492kB inactive_anon:590196kB active_file:84kB

2.8GB of anonymous memory

> [  532.942508] Free swap  = 0kB
> [  532.942510] Total swap = 431632kB

430MB of swap, all used up.

That's a genuine OOM.  Something (presumably cc1plus) has consumed
waaaay too much memory, quite possibly leaked it.

It would help if the oom-killer were to print some information about
the oom-killed process's memory footprint.

[Justin P. Mattock]

Andrew Morton wrote:

> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock<justinmattock@...>  wrote:
>
>    
>> Hello,
>> I'm not sure how to handle this,
>> while compiling firefox-3.6b1.source
>> I get this with the default compiling options,
>> as well as custom:
>>
>> ...
>>
>> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>>      
>
> 2.8GB of anonymous memory
>
>    

figured it would be good enough
(I think I have 4gig's total)
>> [  532.942508] Free swap  = 0kB
>> [  532.942510] Total swap = 431632kB
>>      
>
> 430MB of swap, all used up.
>
>    
yep, narrow down to the smallest amount.
> That's a genuine OOM.  Something (presumably cc1plus) has consumed
> waaaay too much memory, quite possibly leaked it.
>
> It would help if the oom-killer were to print some information about
> the oom-killed process's memory footprint.
>
>
>    
I still have everything setup(if you need me to add a debug patch
let me know)
as for compiling: libc compiled fine, kernel fine,
and every package on the clfs list up to boot up the fresh system.
(was figuring out how to compiling/install firefox before
  I threw the old system away).

stable gcc(4.4*) on the macbook(same os/kernel) compiled fine
firefox, xulrunner, and in the process thunderbird...

Justin P. Mattock

[Jiri Slaby]

On 11/04/2009 07:44 AM, Justin P. Mattock wrote:
> as for compiling: libc compiled fine, kernel fine,
> and every package on the clfs list up to boot up the fresh system.

It might be pretty c++ only, I think.

[KOSAKI Motohiro]

> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock <justinmattock@...> wrote:
>
> > Hello,
> > I'm not sure how to handle this,
> > while compiling firefox-3.6b1.source
> > I get this with the default compiling options,
> > as well as custom:
> >
> > ...
> >
> > active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>
> 2.8GB of anonymous memory
>
> > [  532.942508] Free swap  = 0kB
> > [  532.942510] Total swap = 431632kB
>
> 430MB of swap, all used up.
>
> That's a genuine OOM.  Something (presumably cc1plus) has consumed
> waaaay too much memory, quite possibly leaked it.
>
> It would help if the oom-killer were to print some information about
> the oom-killed process's memory footprint.
>

How about this?

========
Subject: [PATCH] oom: show vsz and rss information of the killed process

In typical oom anylysis scenario, we frequently want to know the killed
process has memory leak or not at first step.
This patch add vsz and rss information to oom log for helping its
analysis. It save much times of debugging guys.

example:
===================================================================
rsyslogd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Pid: 1308, comm: rsyslogd Not tainted 2.6.32-rc6 #24
Call Trace:
[<ffffffff8132e35b>] ?_spin_unlock+0x2b/0x40
[<ffffffff810f186e>] oom_kill_process+0xbe/0x2b0

(snip)

492283 pages non-shared
Out of memory: kill process 2341 (memhog) score 527276 or a child
Killed process 2341 (memhog) vsz:1054552kB, anon-rss:970588kB, file-rss:4kB
===========================================================================
                             ^
                             |
                            here

Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@...>
---
 mm/oom_kill.c |   15 ++++++++++++---
 1 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index ea2147d..498e6f6 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -337,6 +337,8 @@ static void dump_tasks(const struct mem_cgroup *mem)
  } while_each_thread(g, p);
 }
 
+#define K(x) ((x) << (PAGE_SHIFT-10))
+
 /*
  * Send SIGKILL to the selected  process irrespective of  CAP_SYS_RAW_IO
  * flag though it's unlikely that  we select a process with CAP_SYS_RAW_IO
@@ -356,9 +358,16 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
  return;
  }
 
- if (verbose)
- printk(KERN_ERR "Killed process %d (%s)\n",
- task_pid_nr(p), p->comm);
+ if (verbose) {
+ task_lock(p);
+ printk(KERN_ERR "Killed process %d (%s) "
+       "vsz:%lukB, anon-rss:%lukB, file-rss:%lukB\n",
+       task_pid_nr(p), p->comm,
+       K(p->mm->total_vm),
+       K(get_mm_counter(p->mm, anon_rss)),
+       K(get_mm_counter(p->mm, file_rss)));
+ task_unlock(p);
+ }
 
  /*
  * We give our sacrificial lamb high priority and access to
--
1.6.2.5

[Dave Korn-6]

Andrew Morton wrote:

> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock <justinmattock@...> wrote:
>
>> Hello,
>> I'm not sure how to handle this,
>> while compiling firefox-3.6b1.source
>> I get this with the default compiling options,
>> as well as custom:
>>
>> ...
>>
>> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>
> 2.8GB of anonymous memory
>
>> [  532.942508] Free swap  = 0kB
>> [  532.942510] Total swap = 431632kB
>
> 430MB of swap, all used up.
>
> That's a genuine OOM.  Something (presumably cc1plus) has consumed
> waaaay too much memory, quite possibly leaked it.
>
> It would help if the oom-killer were to print some information about
> the oom-killed process's memory footprint.

  I would think that the quickest way to proceed would be to re-run the
failing compile command under gdb at the command-line and see what it's doing
when the oom killer signals it, wouldn't it?  Or turn up the swap until it
doesn't get killed and see what info can be gleaned from the cc1(plus?)
-fmem-report output.

    cheers,
      DaveK

[Justin P. Mattock]

Dave Korn wrote:

> Andrew Morton wrote:
>    
>> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock<justinmattock@...>  wrote:
>>
>>      
>>> Hello,
>>> I'm not sure how to handle this,
>>> while compiling firefox-3.6b1.source
>>> I get this with the default compiling options,
>>> as well as custom:
>>>
>>> ...
>>>
>>> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>>>        
>> 2.8GB of anonymous memory
>>
>>      
>>> [  532.942508] Free swap  = 0kB
>>> [  532.942510] Total swap = 431632kB
>>>        
>> 430MB of swap, all used up.
>>
>> That's a genuine OOM.  Something (presumably cc1plus) has consumed
>> waaaay too much memory, quite possibly leaked it.
>>
>> It would help if the oom-killer were to print some information about
>> the oom-killed process's memory footprint.
>>      
>
>    I would think that the quickest way to proceed would be to re-run the
> failing compile command under gdb at the command-line and see what it's doing
> when the oom killer signals it, wouldn't it?  Or turn up the swap until it
> doesn't get killed and see what info can be gleaned from the cc1(plus?)
> -fmem-report output.
>
>      cheers,
>        DaveK
>
>
>    

I can try, only issue I have is I don't
use a distro, so building anything requires me
to hand compile it(hopefully not difficult for gdb).

So give me some time on this and I'll see if I can get this up
and running, and add that patch to kernel then go from there.

Justin P. Mattock

[Andrew Morton-9]

On Wed,  4 Nov 2009 18:32:16 +0900 (JST) KOSAKI Motohiro <kosaki.motohiro@...> wrote:

> > It would help if the oom-killer were to print some information about
> > the oom-killed process's memory footprint.
> >
>
>
> How about this?

looks good, thanks.

> ========
> Subject: [PATCH] oom: show vsz and rss information of the killed process
>
> In typical oom anylysis scenario, we frequently want to know the killed
> process has memory leak or not at first step.
> This patch add vsz and rss information to oom log for helping its
> analysis. It save much times of debugging guys.
>
> example:
> ===================================================================
> rsyslogd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
> Pid: 1308, comm: rsyslogd Not tainted 2.6.32-rc6 #24
> Call Trace:
> [<ffffffff8132e35b>] ?_spin_unlock+0x2b/0x40
> [<ffffffff810f186e>] oom_kill_process+0xbe/0x2b0
>
> (snip)
>
> 492283 pages non-shared
> Out of memory: kill process 2341 (memhog) score 527276 or a child
> Killed process 2341 (memhog) vsz:1054552kB, anon-rss:970588kB, file-rss:4kB
> ===========================================================================
>                              ^
>                              |
>                             here
> ...
>
> + if (verbose) {
> + task_lock(p);

We need to be careful with which locks we take on the oom-killer path,
because it can be called by code which already holds locks.  But I
expect task_lock() will be OK.

> + printk(KERN_ERR "Killed process %d (%s) "
> +       "vsz:%lukB, anon-rss:%lukB, file-rss:%lukB\n",
> +       task_pid_nr(p), p->comm,
> +       K(p->mm->total_vm),
> +       K(get_mm_counter(p->mm, anon_rss)),
> +       K(get_mm_counter(p->mm, file_rss)));
> + task_unlock(p);
> + }

[Dave Korn-6]

Justin P. Mattock wrote:

> I can try, only issue I have is I don't
> use a distro, so building anything requires me
> to hand compile it

  Oh, ouch!

> (hopefully not difficult for gdb).

  Indeed, hopefully not.

> So give me some time on this and I'll see if I can get this up
> and running, and add that patch to kernel then go from there.

  The one thing you can still try straight away for minimal effort is the
-fmem-report option, but it's also the least informative...

    cheers,
      DaveK

[Justin P. Mattock]

On Wed, Nov 4, 2009 at 7:45 AM, Dave Korn
<dave.korn.cygwin@...> wrote:
> Justin P. Mattock wrote:
>
>> I can try, only issue I have is I don't
>> use a distro, so building anything requires me
>> to hand compile it
>
>  Oh, ouch!
>

I know.. I'm a horror for optimization

>> (hopefully not difficult for gdb).
>
>  Indeed, hopefully not.
>

you never know, some packages big/small turn into brain surgery
just to get going.(I'll try after I do some morning exercises)

>> So give me some time on this and I'll see if I can get this up
>> and running, and add that patch to kernel then go from there.
>
>  The one thing you can still try straight away for minimal effort is the
> -fmem-report option, but it's also the least informative...
>
>    cheers,
>      DaveK
>
>

O.k. here is the info from dmesg(with the patch added)
and what -fmem-report:


[  205.931940] kjournald starting.  Commit interval 5 seconds
[  205.931957] EXT3-fs warning: maximal mount count reached, running
e2fsck is recommended
[  205.935509] EXT3 FS on sdb1, internal journal
[  205.935513] EXT3-fs: mounted filesystem with writeback data mode.
[  205.956396] SELinux: initialized (dev sdb1, type ext3), uses xattr
[  434.205304] __ratelimit: 75 callbacks suppressed
[  434.205308] wicd-monitor invoked oom-killer: gfp_mask=0x201da,
order=0, oom_adj=0
[  434.205313] Pid: 1563, comm: wicd-monitor Tainted: P
2.6.32-rc5-00081-g964fe08-dirty #36
[  434.205316] Call Trace:
[  434.205325]  [<ffffffff810bc1af>] oom_kill_process+0x7c/0x243
[  434.205330]  [<ffffffff810bc6e0>] __out_of_memory+0x146/0x15d
[  434.205335]  [<ffffffff810bc909>] out_of_memory+0x6e/0x9d
[  434.205339]  [<ffffffff810bf7c0>] __alloc_pages_nodemask+0x498/0x5ce
[  434.205345]  [<ffffffff810c10e8>] __do_page_cache_readahead+0xa0/0x1a1
[  434.205350]  [<ffffffff810c1436>] ra_submit+0x1c/0x20
[  434.205353]  [<ffffffff810ba620>] filemap_fault+0x1a6/0x346
[  434.205359]  [<ffffffff810cf388>] __do_fault+0x4f/0x3d9
[  434.205363]  [<ffffffff810eec2e>] ? do_sync_read+0xe3/0x120
[  434.205369]  [<ffffffff811a2571>] ? file_has_perm+0x90/0x9e
[  434.205373]  [<ffffffff810d1cf7>] handle_mm_fault+0x3ab/0x6a7
[  434.205379]  [<ffffffff813d44a3>] do_page_fault+0x2bb/0x2d3
[  434.205383]  [<ffffffff813d23a5>] page_fault+0x25/0x30
[  434.205386] Mem-Info:
[  434.205388] DMA per-cpu:
[  434.205391] CPU    0: hi:    0, btch:   1 usd:   0
[  434.205394] CPU    1: hi:    0, btch:   1 usd:   0
[  434.205396] DMA32 per-cpu:
[  434.205399] CPU    0: hi:  186, btch:  31 usd: 125
[  434.205401] CPU    1: hi:  186, btch:  31 usd: 105
[  434.205404] Normal per-cpu:
[  434.205406] CPU    0: hi:  186, btch:  31 usd: 172
[  434.205409] CPU    1: hi:  186, btch:  31 usd: 154
[  434.205416] active_anon:708764 inactive_anon:266208 isolated_anon:0
[  434.205417]  active_file:71 inactive_file:11 isolated_file:0
[  434.205419]  unevictable:0 dirty:0 writeback:0 unstable:0 buffer:74
[  434.205420]  free:6961 slab_reclaimable:2782 slab_unreclaimable:16224
[  434.205421]  mapped:65 shmem:35 pagetables:2861 bounce:0
[  434.205430] DMA free:15944kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15360kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[  434.205438] lowmem_reserve[]: 0 2976 3986 3986
[  434.205449] DMA32 free:9976kB min:6020kB low:7524kB high:9028kB
active_anon:2360156kB inactive_anon:589924kB active_file:60kB
inactive_file:44kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:3047792kB mlocked:0kB dirty:0kB
writeback:0kB mapped:88kB shmem:4kB slab_reclaimable:148kB
slab_unreclaimable:316kB kernel_stack:40kB pagetables:5952kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:225
all_unreclaimable? yes
[  434.205457] lowmem_reserve[]: 0 0 1010 1010
[  434.205468] Normal free:1924kB min:2040kB low:2548kB high:3060kB
active_anon:474900kB inactive_anon:474908kB active_file:224kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB
writeback:0kB mapped:172kB shmem:136kB slab_reclaimable:10980kB
slab_unreclaimable:64572kB kernel_stack:824kB pagetables:5492kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:677
all_unreclaimable? yes
[  434.205476] lowmem_reserve[]: 0 0 0 0
[  434.205481] DMA: 2*4kB 2*8kB 3*16kB 2*32kB 3*64kB 2*128kB 2*256kB
1*512kB 2*1024kB 2*2048kB 2*4096kB = 15944kB
[  434.205493] DMA32: 2*4kB 14*8kB 16*16kB 10*32kB 3*64kB 1*128kB
1*256kB 1*512kB 2*1024kB 1*2048kB 1*4096kB = 9976kB
[  434.205505] Normal: 481*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1924kB
[  434.205516] 8029 total pagecache pages
[  434.205519] 7893 pages in swap cache
[  434.205521] Swap cache stats: add 112490, delete 104597, find 5058/5479
[  434.205524] Free swap  = 0kB
[  434.205526] Total swap = 431632kB
[  434.220125] 1048576 pages RAM
[  434.220127] 40493 pages reserved
[  434.220129] 170 pages shared
[  434.220131] 1000179 pages non-shared
[  434.220135] Out of memory: kill process 7925 (c++) score 539395 or a child
[  434.220141] Killed process 7926 (cc1plus) vsz:4280180kB,
anon-rss:3831924kB, file-rss:4kB
[  434.259045] cc1plus: page allocation failure. order:0, mode:0x280da
[  434.259051] Pid: 7926, comm: cc1plus Tainted: P
2.6.32-rc5-00081-g964fe08-dirty #36
[  434.259054] Call Trace:
[  434.259063]  [<ffffffff810bf874>] __alloc_pages_nodemask+0x54c/0x5ce
[  434.259070]  [<ffffffff810d1bc3>] handle_mm_fault+0x277/0x6a7
[  434.259076]  [<ffffffff813d44a3>] do_page_fault+0x2bb/0x2d3
[  434.259080]  [<ffffffff813d23a5>] page_fault+0x25/0x30
[  434.259083] Mem-Info:
[  434.259085] DMA per-cpu:
[  434.259088] CPU    0: hi:    0, btch:   1 usd:   0
[  434.259090] CPU    1: hi:    0, btch:   1 usd:   0
[  434.259092] DMA32 per-cpu:
[  434.259095] CPU    0: hi:  186, btch:  31 usd: 125
[  434.259098] CPU    1: hi:  186, btch:  31 usd: 105
[  434.259100] Normal per-cpu:
[  434.259103] CPU    0: hi:  186, btch:  31 usd: 172
[  434.259106] CPU    1: hi:  186, btch:  31 usd: 154
[  434.259113] active_anon:708764 inactive_anon:266208 isolated_anon:0
[  434.259115]  active_file:71 inactive_file:11 isolated_file:0
[  434.259116]  unevictable:0 dirty:0 writeback:0 unstable:0 buffer:74
[  434.259117]  free:6961 slab_reclaimable:2782 slab_unreclaimable:16224
[  434.259119]  mapped:65 shmem:35 pagetables:2861 bounce:0
[  434.259128] DMA free:15944kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15360kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[  434.259135] lowmem_reserve[]: 0 2976 3986 3986
[  434.259147] DMA32 free:9976kB min:6020kB low:7524kB high:9028kB
active_anon:2360156kB inactive_anon:589924kB active_file:60kB
inactive_file:44kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:3047792kB mlocked:0kB dirty:0kB
writeback:0kB mapped:88kB shmem:4kB slab_reclaimable:148kB
slab_unreclaimable:316kB kernel_stack:40kB pagetables:5952kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:225
all_unreclaimable? yes
[  434.259154] lowmem_reserve[]: 0 0 1010 1010
[  434.259166] Normal free:1924kB min:2040kB low:2548kB high:3060kB
active_anon:474900kB inactive_anon:474908kB active_file:224kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB
writeback:0kB mapped:172kB shmem:136kB slab_reclaimable:10980kB
slab_unreclaimable:64572kB kernel_stack:824kB pagetables:5492kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:677
all_unreclaimable? yes
[  434.259173] lowmem_reserve[]: 0 0 0 0
[  434.259178] DMA: 2*4kB 2*8kB 3*16kB 2*32kB 3*64kB 2*128kB 2*256kB
1*512kB 2*1024kB 2*2048kB 2*4096kB = 15944kB
[  434.259190] DMA32: 2*4kB 14*8kB 16*16kB 10*32kB 3*64kB 1*128kB
1*256kB 1*512kB 2*1024kB 1*2048kB 1*4096kB = 9976kB
[  434.259202] Normal: 481*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1924kB
[  434.259214] 8029 total pagecache pages
[  434.259216] 7893 pages in swap cache
[  434.259219] Swap cache stats: add 112490, delete 104597, find 5058/5480
[  434.259221] Free swap  = 0kB
[  434.259223] Total swap = 431632kB
[  434.273830] 1048576 pages RAM
[  434.273832] 40493 pages reserved
[  434.273834] 170 pages shared
[  434.273836] 1000179 pages non-shared




/********** and -fmem-report **************/




Memory still allocated at the end of the compilation process
Size   Allocated        Used    Overhead
8             56k         43k       1680
16           196k         81k       4312
32           188k         40k       3384
64           336k        232k       5376
128          132k        128k       1848
512           28k         13k        392
1024          20k       9216         280
2048          16k       8192         224
4096         348k        348k       4872
8192          56k         56k        392
16384         16k         16k         56
32768         64k         64k        112
24           256k         53k       4608
40           256k        199k       4096
48           800k        579k         12k
56            92k       6608        1472
72            72k       5040        1008
80          8192        1440         112
88          8192         704         112
96           940k        580k         12k
112           80k         39k       1120
120           16k        840         224
192          192k        157k       2688
136          600k        580k       8400
160          200k        172k       2800
176          976k        795k         13k
152           84k         33k       1176
104          120k         28k       1680
256         1040k       1019k         14k
144         4096         144          56
Total       7200k       5293k        104k

String pool
entries         9928
identifiers     6178 (62.23%)
slots           16384
deleted         3726
bytes           86k (17592186044415M overhead)
table size      128k
coll/search     0.3177
ins/search      0.1518
avg. entry      8.92 bytes (+/- 9.51)
longest entry   112

??? tree nodes created

(No per-node statistics)
Type hash: size 4093, 2550 elements, 1.102267 collisions
DECL_DEBUG_EXPR  hash: size 1021, 0 elements, 0.015692 collisions
DECL_VALUE_EXPR  hash: size 1021, 0 elements, 0.000000 collisions
no search statistics
No gimple statistics

Alias oracle query stats:
  refs_may_alias_p: 11 disambiguations, 29 queries
  ref_maybe_used_by_call_p: 0 disambiguations, 62 queries
  call_may_clobber_ref_p: 0 disambiguations, 0 queries

PTA query stats:
  pt_solution_includes: 7 disambiguations, 166 queries
  pt_solutions_intersect: 0 disambiguations, 507 queries
jsxml.cpp
c++ -o jsxml.o -c  -DOSTYPE=\"Linux2.6\" -DOSARCH=Linux
-DEXPORT_JS_API  -DJS_USE_SAFE_ARENA
-I/home/name/LFS/firefox/mozilla-1.9.2/js/src -I.
-I./../../dist/include -I./../../dist/include/nsprpub
-I/usr/include/nspr
-I/home/name/LFS/firefox/mozilla-1.9.2/js/src    -fPIC   -fno-rtti
-fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual
-Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align
-Wno-invalid-offsetof -Wno-variadic-macros -Wno-long-long
-pedantic -fno-strict-aliasing -pthread -pipe  -DNDEBUG -DTRIMMED -m64
-mtune=core2 -march=core2 -O2 -pipe -fmem-report
-fomit-frame-pointer   -DMOZILLA_CLIENT -include ./js-confdefs.h
-Wp,-MD,.deps/jsxml.pp
/home/name/LFS/firefox/mozilla-1.9.2/js/src/jsxml.cpp
{standard input}: Assembler messages:
{standard input}:271839: Warning: end of file not at end of a line;
newline inserted
{standard input}:271896: Error: suffix or operands invalid for `movq'
{standard input}:271896: Error: open CFI at the end of file; missing
.cfi_endproc directive
c++: Internal error: Killed (program cc1plus)
Please submit a full bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[4]: *** [jsxml.o] Error 1
make[4]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu/js/src'
make[3]: *** [libs_tier_js] Error 2
make[3]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu'
make[2]: *** [tier_js] Error 2
make[2]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu'
make[1]: *** [default] Error 2
make[1]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu'
make: *** [build] Error 2




--
Justin P. Mattock

[Dave Korn-6]

Justin Mattock wrote:

> O.k. here is the info from dmesg(with the patch added)
> and what -fmem-report:

  I don't know how to read the oom dmesg, but as to the -fmem-report:

> Memory still allocated at the end of the compilation process
> Size   Allocated        Used    Overhead
> Total       7200k       5293k        104k

... what that's telling us is that there isn't a substantial leak in GCC, as
there's only 7 meg left unreclaimed by GC at the end.  I think we'll have to
wait and see what the debugger tells us; either GCC really is using that much
memory in processing the file, or there's some kind of system or kernel bug
you're running into that is causing a leak in the VMM rather than the application.

> String pool

> bytes           86k (17592186044415M overhead)

  0xFFFFFFFFFFF00000, lol, wut?  It's possible that indicates some sort of
memory corruption going on.  Maybe valgrind can help, do you have that?

    cheers,
      DaveK

[Justin P. Mattock]

Dave Korn wrote:

> Justin Mattock wrote:
>
>    
>> O.k. here is the info from dmesg(with the patch added)
>> and what -fmem-report:
>>      
>
>    I don't know how to read the oom dmesg, but as to the -fmem-report:
>
>    
>> Memory still allocated at the end of the compilation process
>> Size   Allocated        Used    Overhead
>> Total       7200k       5293k        104k
>>      
>
> ... what that's telling us is that there isn't a substantial leak in GCC, as
> there's only 7 meg left unreclaimed by GC at the end.  I think we'll have to
> wait and see what the debugger tells us; either GCC really is using that much
> memory in processing the file, or there's some kind of system or kernel bug
> you're running into that is causing a leak in the VMM rather than the application.
>
>    

just finished compiling and installing gdb/valgrind

>> String pool
>>      
>
>    
>> bytes           86k (17592186044415M overhead)
>>      
>
>    0xFFFFFFFFFFF00000, lol, wut?  It's possible that indicates some sort of
> memory corruption going on.  Maybe valgrind can help, do you have that?
>
>      cheers,
>        DaveK
>
>
>    

Not sure how to use these.(need to read)
Any quick commands I can do to get the info
to you?

Justin P. Mattock

[Justin P. Mattock]

here's what I did:
 valgrind --tool=memcheck --leak-check=full -v make -f client.mk build


e2 -march=core2 -O2 -pipe -fomit-frame-pointer   -DMOZILLA_CLIENT
-include ./js-confdefs.h -Wp,-MD,.deps/jsxml.pp
/home/justin/LFS/firefox/mozilla-1.9.2/js/src/jsxml.cpp
{standard input}: Assembler messages:
{standard input}:271839: Warning: end of file not at end of a line;
newline inserted
{standard input}:271896: Error: suffix or operands invalid for `movq'
{standard input}:271896: Error: open CFI at the end of file; missing
.cfi_endproc directive
c++: Internal error: Killed (program cc1plus)
Please submit a full bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[4]: *** [jsxml.o] Error 1
make[4]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu/js/src'
make[3]: *** [libs_tier_js] Error 2
make[3]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu'
make[2]: *** [tier_js] Error 2
make[2]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu'
make[1]: *** [default] Error 2
make[1]: Leaving directory
`/home/name/LFS/firefox/mozilla-1.9.2/obj-x86_64-unknown-linux-gnu'
make: *** [build] Error 2
==4072==
==4072== HEAP SUMMARY:
==4072==     in use at exit: 201,183 bytes in 4,237 blocks
==4072==   total heap usage: 28,879 allocs, 24,642 frees, 2,947,434
bytes allocated
==4072==
==4072== Searching for pointers to 4,237 not-freed blocks
==4072== Checked 268,808 bytes
==4072==
==4072== 6 bytes in 1 blocks are possibly lost in loss record 41 of 295
==4072==    at 0x4C2488A: malloc (vg_replace_malloc.c:195)
==4072==    by 0x50ABEE1: strdup (in /lib/libc-2.10.90.so)
==4072==    by 0x4118D8: xstrdup (in /usr/bin/make)
==4072==    by 0x41BAA4: define_variable_in_set (in /usr/bin/make)
==4072==    by 0x4160B4: eval (in /usr/bin/make)
==4072==    by 0x416766: eval_makefile (in /usr/bin/make)
==4072==    by 0x416A3F: read_all_makefiles (in /usr/bin/make)
==4072==    by 0x410853: main (in /usr/bin/make)
==4072==
==4072== 14 bytes in 1 blocks are possibly lost in loss record 69 of 295
==4072==    at 0x4C2488A: malloc (vg_replace_malloc.c:195)
==4072==    by 0x411977: xmalloc (in /usr/bin/make)
==4072==    by 0x411AA8: savestring (in /usr/bin/make)
==4072==    by 0x41BB7A: define_variable_in_set (in /usr/bin/make)
==4072==    by 0x410832: main (in /usr/bin/make)
==4072==
==4072== LEAK SUMMARY:
==4072==    definitely lost: 0 bytes in 0 blocks
==4072==    indirectly lost: 0 bytes in 0 blocks
==4072==      possibly lost: 20 bytes in 2 blocks
==4072==    still reachable: 201,163 bytes in 4,235 blocks
==4072==         suppressed: 0 bytes in 0 blocks
==4072== Reachable blocks (those to which a pointer was found) are not shown.
==4072== To see them, rerun with: --leak-check=full --show-reachable=yes
==4072==
==4072== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 5 from 5)
--4072--
--4072-- used_suppression:      2 dl-hack3-cond-1
--4072-- used_suppression:      3 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==4072==
==4072== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 5 from 5)

I'll try out gdb, and more of valgrind.

--
Justin P. Mattock