Discussion:
ZFS memory out of control
(too old to reply)
Kartweel
2012-04-05 14:03:15 UTC
Permalink
Hi,

I find all sorts of references to ZFS-FUSE memory growing too big, but have
yet encountered it (within reason). Until now... I have a system where
zfs-fuse is unusable. Memory usage keeps increasing until it freezes the
system. It is a backup server which pulls in VM images and file systems
over the network and keeps snapshots of them. The server has 4GB ram. Intel
Core 2 duo. Linux 3.1.8 64 bit. zfs-fuse 7.0.0.

I initially had dedup and compress on the zpool, but I disabled those. I
have also destroyed the zpool and re-created it without dedup and
compression with the same results. Currently has no snapshots. I can't
image there are any open file handles because the process that was writing
to it has crashed (out of memory).

Currently has 40GB of data. General filesystem data, documents, images, etc.

With the original setup I tried to copy off the data before destroying the
zpool, but I couldn't.... I could only copy off 4GB of data before the
memory usage got too high and it crashed. (Curious as the system has 4GB of
data!).

Current zfs setup is a single disk with no cache or log drives.

Funny thing is I have the exact same OS setup on another system (It is
using the same system image, except it isn't a backup server, it is just
running virtual machines), intel xeon 16GB ram. The zfs-fuse process is
only using 413mb. It does have an SSD for cache and log and 2 drives in a
mirror config.

Any pointers to where I should be looking? I don't think there is a zfsrc
file, so it would be using defaults?

Only messages in syslog and dmesg is an out of memory notice from the
kernel and it killing (sacrificing in it's words) other processes to free
memory. I have no swap configured, so where ram is all used up there is
nowhere to go.

I can reproduce this problem on this system (it seems just by running the
backup for a few hours)
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Emmanuel Anne
2012-04-05 14:27:53 UTC
Permalink
/etc/zfs/zfsrc ?
Anyway if you have 2 systems using the same image, with 1 working and the
other not, then you are the only one who can find the difference between
these 2, we'll just be able to guess...
Post by Kartweel
Hi,
I find all sorts of references to ZFS-FUSE memory growing too big, but
have yet encountered it (within reason). Until now... I have a system where
zfs-fuse is unusable. Memory usage keeps increasing until it freezes the
system. It is a backup server which pulls in VM images and file systems
over the network and keeps snapshots of them. The server has 4GB ram. Intel
Core 2 duo. Linux 3.1.8 64 bit. zfs-fuse 7.0.0.
I initially had dedup and compress on the zpool, but I disabled those. I
have also destroyed the zpool and re-created it without dedup and
compression with the same results. Currently has no snapshots. I can't
image there are any open file handles because the process that was writing
to it has crashed (out of memory).
Currently has 40GB of data. General filesystem data, documents, images, etc.
With the original setup I tried to copy off the data before destroying the
zpool, but I couldn't.... I could only copy off 4GB of data before the
memory usage got too high and it crashed. (Curious as the system has 4GB of
data!).
Current zfs setup is a single disk with no cache or log drives.
Funny thing is I have the exact same OS setup on another system (It is
using the same system image, except it isn't a backup server, it is just
running virtual machines), intel xeon 16GB ram. The zfs-fuse process is
only using 413mb. It does have an SSD for cache and log and 2 drives in a
mirror config.
Any pointers to where I should be looking? I don't think there is a zfsrc
file, so it would be using defaults?
Only messages in syslog and dmesg is an out of memory notice from the
kernel and it killing (sacrificing in it's words) other processes to free
memory. I have no swap configured, so where ram is all used up there is
nowhere to go.
I can reproduce this problem on this system (it seems just by running the
backup for a few hours)
--
To visit our Web site, click on http://zfs-fuse.net/
--
my zfs-fuse git repository :
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-05 14:52:35 UTC
Permalink
There is only an /etc/zfsrc.sample file. The init script looks for a
/etc/sysconfig/zfs-fuse but it doesn't exist. So I assume it uses
defaults of 128mb arc ?

It is the exact same base system image except for SSH keys and system
name. There is no differences in that. It only differs in content and
structure of the zpool (which is just holding backup data, documents,
pictures and some vm images).

So I have no idea where to look :). I've temporarily added 10GB of swap
space on a separate HDD partition to try and stop it running out of
memory. I've also destroyed and re-created the zpool again...

Maybe I'll try it on different hardware and see if that makes a
difference? Or try it on a different system build and see if it
works?... Any pointers would be appreciated... Or should I try and play
with zfs settings to get it to work?

I could run the same backup scripts on the other working machine to see
if it has the same effect... but that machine is running nicely and I
don't really feel like rebuilding it if the filesystem dies... :)
Post by Emmanuel Anne
/etc/zfs/zfsrc ?
Anyway if you have 2 systems using the same image, with 1 working and
the other not, then you are the only one who can find the difference
between these 2, we'll just be able to guess...
Hi,
I find all sorts of references to ZFS-FUSE memory growing too big,
but have yet encountered it (within reason). Until now... I have a
system where zfs-fuse is unusable. Memory usage keeps increasing
until it freezes the system. It is a backup server which pulls in
VM images and file systems over the network and keeps snapshots of
them. The server has 4GB ram. Intel Core 2 duo. Linux 3.1.8 64
bit. zfs-fuse 7.0.0.
I initially had dedup and compress on the zpool, but I disabled
those. I have also destroyed the zpool and re-created it without
dedup and compression with the same results. Currently has no
snapshots. I can't image there are any open file handles because
the process that was writing to it has crashed (out of memory).
Currently has 40GB of data. General filesystem data, documents, images, etc.
With the original setup I tried to copy off the data before
destroying the zpool, but I couldn't.... I could only copy off 4GB
of data before the memory usage got too high and it crashed.
(Curious as the system has 4GB of data!).
Current zfs setup is a single disk with no cache or log drives.
Funny thing is I have the exact same OS setup on another system
(It is using the same system image, except it isn't a backup
server, it is just running virtual machines), intel xeon 16GB ram.
The zfs-fuse process is only using 413mb. It does have an SSD for
cache and log and 2 drives in a mirror config.
Any pointers to where I should be looking? I don't think there is
a zfsrc file, so it would be using defaults?
Only messages in syslog and dmesg is an out of memory notice from
the kernel and it killing (sacrificing in it's words) other
processes to free memory. I have no swap configured, so where ram
is all used up there is nowhere to go.
I can reproduce this problem on this system (it seems just by
running the backup for a few hours)
--
To visit our Web site, click on http://zfs-fuse.net/
--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Andras Korn
2012-04-05 16:03:00 UTC
Permalink
Maybe I'll try it on different hardware and see if that makes a difference? Or
try it on a different system build and see if it works?... Any pointers would
be appreciated... Or should I try and play with zfs settings to get it to work?
Maybe try running zfs-fuse with valgrind.
--
Andras Korn <korn at elan.rulez.org>
Purranoia: The fear that your cat is up to something.
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-06 01:43:15 UTC
Permalink
Doh. I spoke too soon. Now other machine has just done the same thing.
After 2 weeks of running flawlessly, and no changes in load or anything,
it decided to go kaplut :(

But it seems to be after it has been restarted, because I rebooted it
not long before it did it.

So quite possibly something wrong with my system build? Maybe kernel
version or something? I'll give valgrind a go. Maybe I should rebuild
the kernel without preempt and try zfsonlinux for a comparison? This is
not gonna be fun :/
Post by Andras Korn
Maybe I'll try it on different hardware and see if that makes a difference? Or
try it on a different system build and see if it works?... Any pointers would
be appreciated... Or should I try and play with zfs settings to get it to work?
Maybe try running zfs-fuse with valgrind.
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-06 01:47:12 UTC
Permalink
Plus on the original system that was playing up, memory usage is now ok,
but CPU usage is always around 100% for zfs-fuse and zpool iostat
indicates it is pretty much idle. No dedup or compression enabled, so I
don't know what it is doing...
Post by Ryan How
Doh. I spoke too soon. Now other machine has just done the same thing.
After 2 weeks of running flawlessly, and no changes in load or
anything, it decided to go kaplut :(
But it seems to be after it has been restarted, because I rebooted it
not long before it did it.
So quite possibly something wrong with my system build? Maybe kernel
version or something? I'll give valgrind a go. Maybe I should rebuild
the kernel without preempt and try zfsonlinux for a comparison? This
is not gonna be fun :/
Post by Andras Korn
Maybe I'll try it on different hardware and see if that makes a difference? Or
try it on a different system build and see if it works?... Any pointers would
be appreciated... Or should I try and play with zfs settings to get it to work?
Maybe try running zfs-fuse with valgrind.
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-06 05:12:00 UTC
Permalink
So giving extra 10GB of swap didn't work :(. It just keeps going till it
kills the system.... It was running well until I rebooted it today...

2424 root 1 0 8119m 5.5g 0 S 24 35.3 1:06.11 zfs-fuse

Problem seems to be on both my systems now and both are unusable :(

I'm going to try rebuilding the systems, different kernel version, etc
and see if that makes any difference. I'll also compile zfsonlinux and
see if that goes any better (Is there any issue with having zfs-fuse and
zfsonlinx on the same system??. I imagine they would conflict?). Also
are there any issues with kernel 3.3.1 if I go for something that recent?

Thanks!
Post by Ryan How
Plus on the original system that was playing up, memory usage is now
ok, but CPU usage is always around 100% for zfs-fuse and zpool iostat
indicates it is pretty much idle. No dedup or compression enabled, so
I don't know what it is doing...
Post by Ryan How
Doh. I spoke too soon. Now other machine has just done the same
thing. After 2 weeks of running flawlessly, and no changes in load or
anything, it decided to go kaplut :(
But it seems to be after it has been restarted, because I rebooted it
not long before it did it.
So quite possibly something wrong with my system build? Maybe kernel
version or something? I'll give valgrind a go. Maybe I should rebuild
the kernel without preempt and try zfsonlinux for a comparison? This
is not gonna be fun :/
Post by Andras Korn
Maybe I'll try it on different hardware and see if that makes a difference? Or
try it on a different system build and see if it works?... Any pointers would
be appreciated... Or should I try and play with zfs settings to get it to work?
Maybe try running zfs-fuse with valgrind.
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Andras Korn
2012-04-06 05:39:57 UTC
Permalink
Post by Ryan How
I'm going to try rebuilding the systems, different kernel version,
etc and see if that makes any difference.
My guess is it won't. I also had a system where zfs-fuse's memory usage just
kept increasing (albeit at a slower rate then what you're reporting). I
think it's a bug that only manifests under certain load conditions, but what
those are, I don't know.
Post by Ryan How
I'll also compile zfsonlinux and see if that goes any better (Is there any
issue with having zfs-fuse and zfsonlinx on the same system??. I imagine
they would conflict?).
Yes, they do conflict, in several ways:

1. they both come with identically named, but different userspace tools.

2. they support different pool versions (NB I haven't looked at zfs-fuse
recently, so maybe this difference is gone). Zfsonlinux will create version
28 pools which zfs-fuse won't be able to read. No problem vice-versa.

3. I imagine things would break in quite interesting ways if you were to
start zfs-fuse on a system with the zfs kernel module loaded (or vice
versa).

For me, zfsonlinux works pretty well; it's still not rock solid, but crashes
less often than zfs-fuse. Paradoxically, it feels somewhat slower, but Brian
Behlendorf said they hadn't really optimised for speed yet. I expect it'll
improve.
Post by Ryan How
Also are there any issues with kernel 3.3.1 if I
go for something that recent?
I run 3.3.0 with zfsonlinux and have no issues I didn't have with earlier
kernels.
--
Andras Korn <korn at elan.rulez.org>
Some people confuse their imagination with their memory.
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Emmanuel Anne
2012-04-06 07:00:27 UTC
Permalink
valgrind would make it to run very slowly, making it unusable here.
Well it seems to be something which changed after the reboot of your
system, but good luck to guess what.
Maybe something related to /proc although I don't see what in /proc could
do that.

It's not preempt, I have been using zfs-fuse with a preempt kernel almost
for ever.
No idea for now.
Post by Andras Korn
Post by Ryan How
I'm going to try rebuilding the systems, different kernel version,
etc and see if that makes any difference.
My guess is it won't. I also had a system where zfs-fuse's memory usage just
kept increasing (albeit at a slower rate then what you're reporting). I
think it's a bug that only manifests under certain load conditions, but what
those are, I don't know.
Post by Ryan How
I'll also compile zfsonlinux and see if that goes any better (Is there
any
Post by Ryan How
issue with having zfs-fuse and zfsonlinx on the same system??. I imagine
they would conflict?).
1. they both come with identically named, but different userspace tools.
2. they support different pool versions (NB I haven't looked at zfs-fuse
recently, so maybe this difference is gone). Zfsonlinux will create version
28 pools which zfs-fuse won't be able to read. No problem vice-versa.
3. I imagine things would break in quite interesting ways if you were to
start zfs-fuse on a system with the zfs kernel module loaded (or vice
versa).
For me, zfsonlinux works pretty well; it's still not rock solid, but crashes
less often than zfs-fuse. Paradoxically, it feels somewhat slower, but Brian
Behlendorf said they hadn't really optimised for speed yet. I expect it'll
improve.
Post by Ryan How
Also are there any issues with kernel 3.3.1 if I
go for something that recent?
I run 3.3.0 with zfsonlinux and have no issues I didn't have with earlier
kernels.
--
Andras Korn <korn at elan.rulez.org>
Some people confuse their imagination with their memory.
--
To visit our Web site, click on http://zfs-fuse.net/
--
my zfs-fuse git repository :
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-06 13:27:59 UTC
Permalink
Thanks everyone for the help to far. Emmanuel, I've followed your
advice, the only thing I changed on the system that I can remember in
the past few days is the timezone, and I think it may have caused the
problem. This is my theory...

There was a problem with the localtime and utc config, on shutdown it
synced the hardware clock to localtime, on startup it thought the clock
was UTC. So when ntp synced the time there was a huge difference (It set
the clock back 8 hours actually). It then synced the hardware clock,
which I don't know if further complicates it or not. When the timezone
was UTC it didn't matter. But the zfs-fuse process started before the
time sync happened, so I wonder if the huge clock jump while zfs-fuse
was running messed up the cache?. I do java programming, and I always
expect System.getCurrentTimeMillis() to increase, never to decrease
(There's a new one I need to remember!). So I can imagine it causing
errors in anything that relies on comparing time.

I've corrected the clock sync issue, and it has only been running 10
minutes, but memory usage is looking a lot more promising. 162mb on
system A, 284 on system B. Oh I just dropped to 283... a very good sign! :)

Wow... that was hard to track down! :). But a lot better than rebuilding
the entire system!

Thanks!
Post by Emmanuel Anne
valgrind would make it to run very slowly, making it unusable here.
Well it seems to be something which changed after the reboot of your
system, but good luck to guess what.
Maybe something related to /proc although I don't see what in /proc
could do that.
It's not preempt, I have been using zfs-fuse with a preempt kernel
almost for ever.
No idea for now.
Post by Ryan How
I'm going to try rebuilding the systems, different kernel version,
etc and see if that makes any difference.
My guess is it won't. I also had a system where zfs-fuse's memory usage just
kept increasing (albeit at a slower rate then what you're
reporting). I
think it's a bug that only manifests under certain load
conditions, but what
those are, I don't know.
Post by Ryan How
I'll also compile zfsonlinux and see if that goes any better (Is
there any
Post by Ryan How
issue with having zfs-fuse and zfsonlinx on the same system??. I
imagine
Post by Ryan How
they would conflict?).
1. they both come with identically named, but different userspace tools.
2. they support different pool versions (NB I haven't looked at zfs-fuse
recently, so maybe this difference is gone). Zfsonlinux will create version
28 pools which zfs-fuse won't be able to read. No problem vice-versa.
3. I imagine things would break in quite interesting ways if you were to
start zfs-fuse on a system with the zfs kernel module loaded (or vice
versa).
For me, zfsonlinux works pretty well; it's still not rock solid, but crashes
less often than zfs-fuse. Paradoxically, it feels somewhat slower, but Brian
Behlendorf said they hadn't really optimised for speed yet. I expect it'll
improve.
Post by Ryan How
Also are there any issues with kernel 3.3.1 if I
go for something that recent?
I run 3.3.0 with zfsonlinux and have no issues I didn't have with earlier
kernels.
--
Andras Korn <korn at elan.rulez.org
<http://elan.rulez.org>>
Some people confuse their imagination with their memory.
--
To visit our Web site, click on http://zfs-fuse.net/
--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Seth Heeren
2012-04-06 15:10:52 UTC
Permalink
If that is the cause, for sure that is an interesting bug.

Normally, programmers use a monotonic system clock precisely to
prevent issues like this, IIRC (e.g.
clock_gettime(CLOCK_MONOTONIC, ...)
From the manpage
CLOCK_MONOTONIC
Clock that cannot be set and represents monotonic time since some
unspecified starting point.

Have you looked at code, or did you just figure this out by looking at
it from the outside?
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Emmanuel Anne
2012-04-06 16:27:24 UTC
Permalink
Well I tried to reproduce it by simply adjusting the time with the date
command (8 hours earlier), and well everything continued to work well.
There was a commit related to that :

commit 287db9929f696f3e80f12f937faac871d4cf7105
Author: Emmanuel Anne <emmanuel.anne-***@public.gmane.org>
Date: Fri May 20 19:09:56 2011 +0200

change cv_timedwait to use the hardware clock

instead of the system time. It allows it to work even if time is
changed by ntp
to some time in the past.
It still fails if you use ctrl-z to suspend zfs-fuse for a few seconds
though.

Very old commit, but I don't know if it made it to 0.7.0, I have got a lot
of changes here which never got to 0.7.0.

Anyway, glad to know it's back to normal !
Post by Seth Heeren
If that is the cause, for sure that is an interesting bug.
Normally, programmers use a monotonic system clock precisely to
prevent issues like this, IIRC (e.g.
clock_gettime(CLOCK_MONOTONIC, ...)
From the manpage
CLOCK_MONOTONIC
Clock that cannot be set and represents monotonic time since some
unspecified starting point.
Have you looked at code, or did you just figure this out by looking at
it from the outside?
--
To visit our Web site, click on http://zfs-fuse.net/
--
my zfs-fuse git repository :
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-07 03:17:05 UTC
Permalink
In my case it was changing the hardware clock. It had a hwclock
--systohc straight after the ntpdate to sync the time... I don't know if
that would have made the difference?
Post by Emmanuel Anne
change cv_timedwait to use the hardware clock
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-07 03:21:10 UTC
Permalink
I haven't looked at the code. But I could reproduce the issue every
time, and fixing my clock issue appears to have fixed the problem. As I
said it was updating the hardware clock also, not just the system time,
but it sounds like the monotonic clock shouldn't be affected by that. I
have no idea what is used in the source.
Post by Seth Heeren
If that is the cause, for sure that is an interesting bug.
Normally, programmers use a monotonic system clock precisely to
prevent issues like this, IIRC (e.g.
clock_gettime(CLOCK_MONOTONIC, ...)
From the manpage
CLOCK_MONOTONIC
Clock that cannot be set and represents monotonic time since some
unspecified starting point.
Have you looked at code, or did you just figure this out by looking at
it from the outside?
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Loading...