No end to swap zone exhausted problems?

[Anders Nordby]

Hi,

Better post about my swap zone problems here than tear all my hair out.
This concerns me:

1) How can SWAPMETA usage continue increasing, while swap space used is
not? Example: http://anders.fupp.net/test/swapmeta.png. In the swap meta
graph, used space is USED and free space is LIMIT-USED (FREE numbers are
a bit odd, it starts with 0) from the SWAPMETA line when running vmstat
-z. In the week graph, you can see that after 3 days swap space usage
settles at around 90 GB, while swap meta keeps increasing.

Is there a leak?

2) By default, kern.maxswzone is set to 32 MB in FreeBSD:

kern.maxswzone: 33554432

This leads to vmstat -z showing:

ITEM      SIZE     LIMIT      USED      FREE  REQUESTS FAILURES
SWAPMETA: 288,   116519,        0,        0,        0, 0

According to vmstat man page, memory numbers are printed in kilobytes.
Where is the relation between maxswzone and SWAPMETA?

By booting with different settings for maxswzone and checking what
SWAPMETA LIMIT turns out to be, I get this table:

kern.maxswzone SWAPMETA LIMIT
============== ==============

32 MB 116519

64 MB 233025

128 MB 466037

256 MB 932074

512 MB 995410

1 GB 995410

2 GB 995410

How come increasing maxswzone beyond 256 MB does not yield a larger
SWAPMETA than 995410? Does it stop around 260 MB maxswzone? How can I
increase SWAPMETA further to be able to swap more?

When maxswzone is set to 2 GB, the number goes negative:

kern.maxswzone: -2147483648

I guess my luck is out?

Some additional info:

- When LIMIT-USED from SWAPMETA gets to 0, the system freezes with the
dreaded "swap zone exhausted, increase kern.maxswzone" message. I
monitor this. It's nice to know when your system will freeze.

- KMEM settings are:

vm.kmem_size_max: 3865468109
vm.kmem_size_min: 0
vm.kmem_size: 2718302208

Had no luck by tuning and increasing kmem_size and kmem_size_max, it
mostly leads to crashes.

- I am running FreeBSD/amd64 7.2-RELEASE, and I have 8 GB physical RAM.

- By lowering kern.maxusers from 386 to 16 and trimming my kernel config
down to a bare minimum I got SWAPMETA LIMIT increased to a whopping
995475, an increase of ~64 KB. I need more than this.

- Why do I use so much swap space? I am running Varnish with malloc for
storage, and I have a rather large dataset (6 million objects for this
set of servers) with mostly 1 week expire time. Performance wise it
works well, better than mmapping a large file which is/was the most
common method for Varnish (this gives occasional ~2 minute freezes).

Regards,

--
Anders.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[David Schultz]

On Fri, Oct 02, 2009, Anders Nordby wrote:
> Better post about my swap zone problems here than tear all my hair out.
> This concerns me:
>
> 1) How can SWAPMETA usage continue increasing, while swap space used is
> not? Example: http://anders.fupp.net/test/swapmeta.png. In the swap meta
> graph, used space is USED and free space is LIMIT-USED (FREE numbers are
> a bit odd, it starts with 0) from the SWAPMETA line when running vmstat
> -z. In the week graph, you can see that after 3 days swap space usage
> settles at around 90 GB, while swap meta keeps increasing.

IIRC, the swap pager tracks virtual memory regions that include
swapped pages on a granularity of 16 pages.  An increase in
swapmeta usage could simply mean that swap has become fragmented.
It should eventually level out (unless there is actually a leak,
but I think we'd know by now).

> How come increasing maxswzone beyond 256 MB does not yield a larger
> SWAPMETA than 995410? Does it stop around 260 MB maxswzone? How can I
> increase SWAPMETA further to be able to swap more?

The code appears to limit the number of swapmeta entries (each
entry being about 100 bytes) to half the number of pages of
physical memory you have (sys/vm/swap_pager.c):

        n = cnt.v_page_count / 2;
        if (maxswzone && n > maxswzone / sizeof(struct swblock))
                n = maxswzone / sizeof(struct swblock);

You could try commenting out the first two of these lines, leaving
the third, and see if that lets you raise the limit to something
that works better for you.  Note that raising it too high is also
going to cause problems, as all of your physical memory will be
filled with swap metadata.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."