rsync failures - possible ulimit issues?

[Jim MacKenzie]

Hi all,

I'm a hobbyist that happens to own an interesting VAXstation 4000/60.  I've
had OpenBSD running on it for some time, updated to 4.5 now.

Recently I've been setting up rsync to make backups of each of my hobby
machines.  All of my other machines have gone off without a hitch but I keep
getting errors on the VAX:

[sender] make_file(usr/obj/games/fortune/datfiles/limerick.dat,*,2)
[sender] make_file(usr/obj/games/fortune/fortune/.depend,*,2)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util.c(117)
[sender=3.0.5]
_exit_cleanup(code=22, file=util.c, line=117): about to call exit(22)

This is on -vvv.  The failure occurs near this point (but not always at this
exact point) consistently.

My initial instinct after ruling out that the system is not exhausting RAM
(it has 104 MB, maximum for the machine, and is not running X nor doing much
so there is quite a lot of free RAM) was that it was a data space problem so
I checked ulimit:

# ulimit -a
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         32768
stack(kbytes)        8192
lockedmem(kbytes)    32718
memory(kbytes)       97852
nofiles(descriptors) 128
processes            158

Changing the limit from 32 MB does not work.  Editing /etc/login.conf to
permit a login group with unlimited credentials results in the same 32 MB
limit.

So, I suppose I have two questions:

1.  Is this likely my problem, as I surmise?
2.  If it is, is there some way this limit can be increased on the VAX?  Am
I missing something, or does it require a change to the code?

In case it matters, here is my rsync batch file:

# cat backup.sh
rsync -vvv -a -e ssh --progress --delete --exclude /usr/src --exclude
/dev --exclude /public /
adelaide.photojim.prv:/home/jim/backup/maroochydore/

(/usr/src and /public are excluded because they are NFS mounts so I back
them up by backing up my NFS server.)

I know this is an ancient machine, but this problem is irritating me.  :)

I have to warn you that I'm not a coder, just a serious hobbyist.

Thanks much,

Jim

[Tobias Ulmer]

On Fri, Aug 21, 2009 at 02:56:20PM -0600, Jim MacKenzie wrote:

> Hi all,
>
> I'm a hobbyist that happens to own an interesting VAXstation 4000/60.  
> I've had OpenBSD running on it for some time, updated to 4.5 now.
>
> Recently I've been setting up rsync to make backups of each of my hobby  
> machines.  All of my other machines have gone off without a hitch but I
> keep getting errors on the VAX:
>
> [sender] make_file(usr/obj/games/fortune/datfiles/limerick.dat,*,2)
> [sender] make_file(usr/obj/games/fortune/fortune/.depend,*,2)
> ERROR: out of memory in flist_expand [sender]
> rsync error: error allocating core memory buffers (code 22) at
> util.c(117) [sender=3.0.5]
> _exit_cleanup(code=22, file=util.c, line=117): about to call exit(22)
>
> This is on -vvv.  The failure occurs near this point (but not always at
> this exact point) consistently.
>
> My initial instinct after ruling out that the system is not exhausting
> RAM (it has 104 MB, maximum for the machine, and is not running X nor
> doing much so there is quite a lot of free RAM) was that it was a data
> space problem so I checked ulimit:
>
> # ulimit -a
> time(cpu-seconds)    unlimited
> file(blocks)         unlimited
> coredump(blocks)     unlimited
> data(kbytes)         32768
> stack(kbytes)        8192
> lockedmem(kbytes)    32718
> memory(kbytes)       97852
> nofiles(descriptors) 128
> processes            158
>
> Changing the limit from 32 MB does not work.  Editing /etc/login.conf to  
> permit a login group with unlimited credentials results in the same 32 MB
> limit.
>
> So, I suppose I have two questions:
>
> 1.  Is this likely my problem, as I surmise?

Yes

> 2.  If it is, is there some way this limit can be increased on the VAX?  
> Am I missing something, or does it require a change to the code?

MAXDSIZ in /usr/src/sys/arch/vax/include/vmparam.h is limited to 32MB.
You could try to increase that limit, either by editing the header or with
a new kernel configuration.
Of course all kinds of bad things may happen...

>
> In case it matters, here is my rsync batch file:
>
> # cat backup.sh
> rsync -vvv -a -e ssh --progress --delete --exclude /usr/src --exclude  
> /dev --exclude /public /  
> adelaide.photojim.prv:/home/jim/backup/maroochydore/
>
> (/usr/src and /public are excluded because they are NFS mounts so I back  
> them up by backing up my NFS server.)
>
> I know this is an ancient machine, but this problem is irritating me.  :)
>
> I have to warn you that I'm not a coder, just a serious hobbyist.
>
> Thanks much,
>
> Jim

[Jim MacKenzie]

----- Original Message -----
From: "Tobias Ulmer" <tobiasu@...>
To: "Jim MacKenzie" <jim@...>
Cc: <vax@...>
Sent: Saturday, August 22, 2009 3:28 AM
Subject: Re: rsync failures - possible ulimit issues?


> MAXDSIZ in /usr/src/sys/arch/vax/include/vmparam.h is limited to 32MB.
> You could try to increase that limit, either by editing the header or with
> a new kernel configuration.
> Of course all kinds of bad things may happen...

I tried exactly that (by editing the header and recompiling).

I invoked my rsync script and it worked... for a long time (much further
than it did previously).  But I did eventually get a kernel panic.

My instinct at this point is to file a bug report for rsync, because it
clearly does not work reliably with 32 MB of data space and obviously we
have at least one platform that is limited to that amount.

My other instinct is to try a different kernel header setting (perhaps 48MB)
and see if that is more reliable.

I was going to do a kernel build if rsync succeeded, to test the kernel
reliability, but that is obviously of no benefit with my existing machine
state.  :)

If you (or anyone else on the list) have any other suggestions of things I
can try (both to help my situation and to provide useful information for the
developers), please let me know.

Jim