extra large process size

[Milos Negovanovic]

Hi,

Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
process size. Freshly started Io looks like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14348 milosn    20   0   99m  86m 1064 S  0.0 33.7   0:00.24 io

Host is:
Linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/Linux

Any idea what might be causing this extra large process size?

Regards
--
Milos Negovanovic
milos.negovanovic@...

[Steve Dekorte]

On 2009-07-16, at 11:43 AM, Milos Negovanovic wrote:

> Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
> process size. Freshly started Io looks like this:
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 14348 milosn    20   0   99m  86m 1064 S  0.0 33.7   0:00.24 io
>
> Host is:
> Linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/
> Linux
>
> Any idea what might be causing this extra large process size?

Here's what it looks like for me on OSX:

  2588 io           0.0%  0:00.08   1    14     30   2284K   188K  
2936K    21

Are you using the latest git repo?

Steve

[Chris Double]

> Any idea what might be causing this extra large process size?

This is what it looks like for me freshly started on Linux (using a
git repos build):

NI  VIRT   RES   SHR S   %CPU %MEM    TIME+  COMMAND
20  81732 4164  484   S    0        0.2           0:00.44 io

Chris.
--
http://www.bluishcoder.co.nz

[Steve Dekorte]

On 2009-07-16, at 5:20 PM, Chris Double wrote:
> NI  VIRT   RES   SHR S   %CPU %MEM    TIME+  COMMAND
> 20  81732 4164  484   S    0        0.2           0:00.44 io

I'm never sure of how to read these things. Does this mean it's using  
81M? Are you on a 64bit machine with the compiler set to actually use  
64bits? Perhaps a 64bit issue with hashtables or a memory leak  
specific to 64bit systems? Can someone on one of these machines see  
what is eating memory?

[Chris Double]

On Fri, Jul 17, 2009 at 12:27 PM, Steve Dekorte<steve@...> wrote:
>
>
> On 2009-07-16, at 5:20 PM, Chris Double wrote:
>> NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 20 81732 4164 484 S 0 0.2 0:00.44 io
>
> I'm never sure of how to read these things. Does this mean it's using
> 81M?

No, all the numbers are kilobytes. I had to look that up too because I
assumed it was 81M at first :) This is on a 32 bit machine.

Chris.
--
http://www.bluishcoder.co.nz

[Steve Dekorte]

On 2009-07-16, at 5:31 PM, Chris Double wrote:

>> On 2009-07-16, at 5:20 PM, Chris Double wrote:
>>> NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 20 81732 4164 484 S 0 0.2 0:00.44 io
>>
>> I'm never sure of how to read these things. Does this mean it's using
>> 81M?
>
> No, all the numbers are kilobytes. I had to look that up too because I
> assumed it was 81M at first :) This is on a 32 bit machine.

So 81732kb != 81M? Why is the virtual memory usage so large?

[Chris Double]

On Fri, Jul 17, 2009 at 1:09 PM, Steve Dekorte<steve@...> wrote:
>
> So 81732kb != 81M? Why is the virtual memory usage so large?

Sorry, you're right, that is 81M. I made a mistake and that was
another io process running on the system. Starting a new one from
fresh, without any addons shows 8MB. Interestingly just doing a single
'Object clone' makes it rise by about 3MB. Just hitting 'enter' at the
io prompt makes it rise by about 2MB every time I hit enter.

Chris.
--
http://www.bluishcoder.co.nz

[Steve Dekorte]

On 2009-07-16, at 6:56 PM, Chris Double wrote:
> Sorry, you're right, that is 81M. I made a mistake and that was
> another io process running on the system. Starting a new one from
> fresh, without any addons shows 8MB. Interestingly just doing a single
> 'Object clone' makes it rise by about 3MB. Just hitting 'enter' at the
> io prompt makes it rise by about 2MB every time I hit enter.

I don't see this on OSX. Could it be a readline issue?
What happens if you turn off readline in the CLI?

[Chris Double]

On Fri, Jul 17, 2009 at 2:00 PM, Steve Dekorte<steve@...> wrote:
> I don't see this on OSX. Could it be a readline issue?
> What happens if you turn off readline in the CLI?

This is without the readline addon installed. I'll try and dig further.

Chris.
--
http://www.bluishcoder.co.nz

[simonhaines]

Hmm. This is what I got from my linux build of the latest git repository:
$ ps -C io -F
 PID PPID  C   SZ  RSS PSR STIME   TTY     TIME CMD
4050 2346 62 2170 3780   0 11:58 pts/0 00:00:08  io

Where 'SZ' (virtual size) is in Kb: 2170 = 2.12Mb, and 'RSS' (real memory, resident set size) is in Kb: 3.7Mb. Note I'm running under qemu, hence long startup time.

On a recent debug build on Win32, the mem usage is 5,096K which is large, but not excessive.

Why do my results differ so dramatically from yours?

[Chris Double]

On Fri, Jul 17, 2009 at 2:12 PM, simonhaines<simonhaines@...> wrote:
> Hmm. This is what I got from my linux build of the latest git repository:
> $ ps -C io -F
> PID PPID C SZ RSS PSR STIME TTY TIME CMD
> 4050 2346 62 2170 3780 0 11:58 pts/0 00:00:08 io

I was using 'top' to get my VSS figure. If I use ps as you do I get
similar values to you. If I do a 'Collector collect' the numbers drop
back down to low values so I'm assuming that what I'm seeing is just
because the GC hasn't run.

Chris.
--
http://www.bluishcoder.co.nz

[bblochl2-2]

--- In iolanguage@..., Milos Negovanovic <milos.negovanovic@...> wrote:

>
> Hi,
>
> Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
> process size. Freshly started Io looks like this:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 14348 milosn    20   0   99m  86m 1064 S  0.0 33.7   0:00.24 io
>
> Host is:
> Linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/Linux
>
> Any idea what might be causing this extra large process size?
>
> Regards
> --
> Milos Negovanovic
> milos.negovanovic@...
>

On 64 bit debian I get between 2.7 and 2.9 MB. In detail one can get the memory usage with the process number of io ( pmap 6534
 ps -A). and ~$ cat /proc/6534/status (6534 is the process id found with ps -A). My data:
Name: io
State: S (sleeping)
Tgid: 6534
Pid: 6534
PPid: 6518
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 256
Groups: 4 20 24 46 104 108 123 124 1000
VmPeak:   90500 kB
VmSize:   10672 kB
VmLck:       0 kB
VmHWM:    4492 kB
VmRSS:    4316 kB
VmData:    6496 kB
VmStk:      84 kB
VmExe:       4 kB
VmLib:    3688 kB
VmPTE:      24 kB
Threads: 1
SigQ: 0/16382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000008386006
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: 00000000,00000003
Cpus_allowed_list: 0-1
Mems_allowed: 1
Mems_allowed_list: 0
voluntary_ctxt_switches: 1
nonvoluntary_ctxt_switches: 7

Certainly you only get a snapshot, not an average or some thing of that. As you see io is sleeping.

~$ pmap 6534 is useful as well.

Regards BB

[Milos Negovanovic]

On Thu, Jul 16, 2009 at 03:43:16PM -0700, Steve Dekorte wrote:

>
>
>    On 2009-07-16, at 11:43 AM, Milos Negovanovic wrote:
>    > Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
>    > process size. Freshly started Io looks like this:
>    >
>    > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>    > 14348 milosn 20 0 99m 86m 1064 S 0.0 33.7 0:00.24 io
>    >
>    > Host is:
>    > Linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/
>    > Linux
>    >
>    > Any idea what might be causing this extra large process size?
>
>    Here's what it looks like for me on OSX:
>
>    2588 io 0.0% 0:00.08 1 14 30 2284K 188K
>    2936K 21
>
>    Are you using the latest git repo?
>
>    Steve

It was 2 week old checkout. With latest tar.gz I get:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15973 milosn    20   0  103m  91m 1060 S  0.0 35.6   0:00.28 io

Even worse now 91M!

Regards
--
Milos Negovanovic
milos.negovanovic@...

[Milos Negovanovic]

On Fri, Jul 17, 2009 at 07:47:27AM -0000, bblochl2 wrote:

>    On 64 bit debian I get between 2.7 and 2.9 MB. In detail one can get the
>    memory usage with the process number of io ( pmap 6534
>    ps -A). and ~$ cat /proc/6534/status (6534 is the process id found with ps
>    -A). My data:
>    Name: io
>    State: S (sleeping)
>    Tgid: 6534
>    Pid: 6534
>    PPid: 6518
>    TracerPid: 0
>    Uid: 1000 1000 1000 1000
>    Gid: 1000 1000 1000 1000
>    FDSize: 256
>    Groups: 4 20 24 46 104 108 123 124 1000
>    VmPeak: 90500 kB
>    VmSize: 10672 kB
>    VmLck: 0 kB
>    VmHWM: 4492 kB
>    VmRSS: 4316 kB
>    VmData: 6496 kB
>    VmStk: 84 kB
>    VmExe: 4 kB
>    VmLib: 3688 kB
>    VmPTE: 24 kB
>    Threads: 1
>    SigQ: 0/16382
>    SigPnd: 0000000000000000
>    ShdPnd: 0000000000000000
>    SigBlk: 0000000000000000
>    SigIgn: 0000000000000000
>    SigCgt: 0000000008386006
>    CapInh: 0000000000000000
>    CapPrm: 0000000000000000
>    CapEff: 0000000000000000
>    CapBnd: ffffffffffffffff
>    Cpus_allowed: 00000000,00000003
>    Cpus_allowed_list: 0-1
>    Mems_allowed: 1
>    Mems_allowed_list: 0
>    voluntary_ctxt_switches: 1
>    nonvoluntary_ctxt_switches: 7
>

cat /proc/15973/status
Name:   io
State:  S (sleeping)
Tgid:   15973
Pid:    15973
PPid:   4265
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    100     100     100     100
FDSize: 256
Groups: 4 100
VmPeak:   106268 kB
VmSize:   106268 kB
VmLck:         0 kB
VmHWM:     93464 kB
VmRSS:     93464 kB
VmData:    95040 kB
VmStk:        84 kB
VmExe:         4 kB
VmLib:      2884 kB
VmPTE:       224 kB
Threads:        1
SigQ:   0/2112
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000000000002
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
voluntary_ctxt_switches:  4
nonvoluntary_ctxt_switches:     59


Something is definitely wrong on this host. Just to figure out what.

Regards
--
Milos Negovanovic
milos.negovanovic@...

[Milos Negovanovic]

On Fri, Jul 17, 2009 at 09:24:18AM +0100, Milos Negovanovic wrote:

> On Thu, Jul 16, 2009 at 03:43:16PM -0700, Steve Dekorte wrote:
> >
> >
> >    On 2009-07-16, at 11:43 AM, Milos Negovanovic wrote:
> >    > Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
> >    > process size. Freshly started Io looks like this:
> >    >
> >    > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >    > 14348 milosn 20 0 99m 86m 1064 S 0.0 33.7 0:00.24 io
> >    >
> >    > Host is:
> >    > Linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/
> >    > Linux
> >    >
> >    > Any idea what might be causing this extra large process size?
> >
> >    Here's what it looks like for me on OSX:
> >
> >    2588 io 0.0% 0:00.08 1 14 30 2284K 188K
> >    2936K 21
> >
> >    Are you using the latest git repo?
> >
> >    Steve
>
> It was 2 week old checkout. With latest tar.gz I get:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 15973 milosn    20   0  103m  91m 1060 S  0.0 35.6   0:00.28 io
>
> Even worse now 91M!
>

If I run 'Collector collect' few times at Io prompt RES size drops right
down to 10M:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15976 milosn    20   0 23684  10m 1068 S  0.0  4.2   0:00.30 io

Now this is better, but still don't understand why would initial memory
usage be so high.

Regards
--
Milos Negovanovic
milos.negovanovic@...

[bblochl2-2]

--- In iolanguage@..., Milos Negovanovic <milos.negovanovic@...> wrote:
>
> Hi,
>
> Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
> process size. Freshly started Io looks like this:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 14348 milosn    20   0   99m  86m 1064 S  0.0 33.7   0:00.24 io
>
usually an output from 'ps aux' for Xen VPS server looks like that:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
You have just something different? Do you use Xen or OpenVZ? With the same usage 300 MB use of ram in a VPS with openvz, but about 820 MB of ram VPS with xen is normal. I have heard from CS freaks - not a personal experience - that hosters try to balance memory usage and try to stay that about 30%. So the naked number means nothing. How is the memory usage with running no extra application? You will see there is a remarkable amount used. Most of the memory usually is from Linux caching.

Try free -m to get more details.

I forgot to give the Lua memory usage in my last posting for comparison, so here are extended data of memory usage again:

Io about 2.8 MB
Lua about 208 kB

The kB is correct in the Lua case! An embedded Io SOC with about 3 MB?

Regards BB

[Milos Negovanovic]

On Fri, Jul 17, 2009 at 08:47:55AM -0000, bblochl2 wrote:

>
>
>    --- In [1]iolanguage@..., Milos Negovanovic
>    <milos.negovanovic@...> wrote:
>    >
>    > Hi,
>    >
>    > Ive compiled Io on one of my Xen VPS-s and Ive noticed really unusual
>    > process size. Freshly started Io looks like this:
>    >
>    > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>    > 14348 milosn 20 0 99m 86m 1064 S 0.0 33.7 0:00.24 io
>    >
>    usually an output from 'ps aux' for Xen VPS server looks like that:
>    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>    You have just something different? Do you use Xen or OpenVZ?

Its a Xen host. Output is from top.

>    With the same
>    usage 300 MB use of ram in a VPS with openvz, but about 820 MB of ram VPS
>    with xen is normal. I have heard from CS freaks - not a personal
>    experience - that hosters try to balance memory usage and try to stay that
>    about 30%. So the naked number means nothing. How is the memory usage with
>    running no extra application? You will see there is a remarkable amount
>    used. Most of the memory usually is from Linux caching.

Line describes memory usage of a single process (freshly started Io in
this case). I know linux caches lots of stuff but don't think its related
to my problem.

Regards
--
Milos Negovanovic
milos.negovanovic@...

[Milos Negovanovic]

On Fri, Jul 17, 2009 at 09:44:12AM +0100, Milos Negovanovic wrote:
>
> If I run 'Collector collect' few times at Io prompt RES size drops right
> down to 10M:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 15976 milosn    20   0 23684  10m 1068 S  0.0  4.2   0:00.30 io
>

I was speaking to early, after initializing single addon (Socket) RES
size jumps up to 71M.

Regards
--
Milos Negovanovic
milos.negovanovic@...

[bblochl2-2]

--- In iolanguage@..., Milos Negovanovic <milos.negovanovic@...> wrote:

>
> On Fri, Jul 17, 2009 at 09:44:12AM +0100, Milos Negovanovic wrote:
> >
> > If I run 'Collector collect' few times at Io prompt RES size drops right
> > down to 10M:
> >
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> > 15976 milosn    20   0 23684  10m 1068 S  0.0  4.2   0:00.30 io
> >
>
> I was speaking to early, after initializing single addon (Socket) RES
> size jumps up to 71M.
>
> Regards
> --
> Milos Negovanovic
> milos.negovanovic@...
>

If you have a total of about 1690.5M available, then the 4.2% memory usage I see here are equivalent to that 71M that you report here.

Calculating with percentages is really a hard task! I have actually to dry my sweat after that calculation with my pocket calculator.

If I use pmap on a "normal" machine debian lenny 64 bit I get a total of memory used for RAM and needed files of about 10 MB for Io. In a virtual machine you need that memory as well. Virtual machine does not mean only to virtualize the aqctive used RAM. OOP never was a memory saver and never will be!

For python you need about 6,3 MB,
For Lua you need about 2,8 MB.
SpiderMonkey (an interpreter Version of JavaScript) needs 4,8 MB.
and "retro", a "minimalistic FORTH" needs opulent 21.6 MB!

Computer usage without memory need is like the wish to drive a car without gasoline need - an unaccomplishable dream. So where is your real problem - Could not really be the memory use? Must sit a bit deeper.  

BB

[Milos Negovanovic]

On Fri, Jul 17, 2009 at 02:13:32PM -0000, bblochl2 wrote:

>    --- In [1]iolanguage@..., Milos Negovanovic
>    <milos.negovanovic@...> wrote:
>    >
>    > On Fri, Jul 17, 2009 at 09:44:12AM +0100, Milos Negovanovic wrote:
>    > >
>    > > If I run 'Collector collect' few times at Io prompt RES size drops
>    right
>    > > down to 10M:
>    > >
>    > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>    > > 15976 milosn 20 0 23684 10m 1068 S 0.0 4.2 0:00.30 io
>    > >
>    >
>    > I was speaking to early, after initializing single addon (Socket) RES
>    > size jumps up to 71M.
>    >
>    > Regards
>    > --
>    > Milos Negovanovic
>    > milos.negovanovic@...
>    >
>    If you have a total of about 1690.5M available, then the 4.2% memory usage
>    I see here are equivalent to that 71M that you report here.

VPS has 256M of RAM.

>
>    Calculating with percentages is really a hard task! I have actually to dry
>    my sweat after that calculation with my pocket calculator.
>
>    If I use pmap on a "normal" machine debian lenny 64 bit I get a total of
>    memory used for RAM and needed files of about 10 MB for Io. In a virtual
>    machine you need that memory as well. Virtual machine does not mean only
>    to virtualize the aqctive used RAM. OOP never was a memory saver and never
>    will be!
>
>    For python you need about 6,3 MB,
>    For Lua you need about 2,8 MB.
>    SpiderMonkey (an interpreter Version of JavaScript) needs 4,8 MB.
>    and "retro", a "minimalistic FORTH" needs opulent 21.6 MB!
>
>    Computer usage without memory need is like the wish to drive a car without
>    gasoline need - an unaccomplishable dream. So where is your real problem -
>    Could not really be the memory use? Must sit a bit deeper.
>

I think that you misunderstood reasons for my post. I was just asking
if anyone knows why this might be happening. I don't think its normal
behavior. As an example look at this 2 top snippets from 2 different
systems running same Io script (basic web sever which I am writing):

1) NetBSD/amd64

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
14631 milosn    85    0    25M 8560K kqueue/2   0:00  0.00%  0.00% io


2) x86_64 GNU/Linux (as Xen domU)

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16002 milosn    20   0  123m  91m 1480 S  0.0 35.7   0:08.18 io


As you can see on Linux box memory usage seems to be 10 times higher!

Regards
--
Milos Negovanovic
milos.negovanovic@...