Discussion:
Write Performance / Buffering
(too old to reply)
Ryan How
2012-04-15 02:01:24 UTC
Permalink
Hi,

I've noticed when using ZFS-Fuse, writes to the disk burst and then
pause, burst and then pause, as accoring to zfs iostat. It seems to
burst to about 60MB/s, then pause for 2 seconds, then a burst again. I'm
guessing it is some kind of write buffer that fills up, then flushes to
disk.

If there any way to tune it to be smoother?. I've come across a few
threads in the BSD ZFS about limiting write throughput, which seemed to
actually give a significant increase in write speed!. But they aren't
really applicable here from what I can tell?


NAME
STATE READ WRITE CKSUM
glhzpool1
ONLINE 0 0 0
mirror-0
ONLINE 0 0 0
disk/by-id/ata-ST1000DM003-9YN162_W1D08X7Y
ONLINE 0 0 0
disk/by-id/ata-ST1000DM003-9YN162_W1D092GT
ONLINE 0 0 0
logs
disk/by-id/ata-INTEL_SSDSC2CW060A3_CVCV153202UA060AGN-part3
ONLINE 0 0 0
cache
sdc5
ONLINE 0 0 0


Thanks!

Ryan
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
sgheeren
2012-04-15 11:27:20 UTC
Permalink
ZFS on Solaris used to have tunables that zfs-fuse never directly
supported. See the ZFS Evil Tuning Guide
<http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#The_Tunables>

You don't mention the BSD settings that you refer to, but I wouldn't be
surprised when you actually mean these.

Most of these tunables are still available as static variables. E.g. you
can find

src/lib/libzpool/dmu_zfetch.c:int zfs_prefetch_disable = 0;


or

src/lib/libzpool/zil.c:int zil_disable = 0; /* disable intent
logging */ // DON'T

Be sure you know what you're doing when you touch these internal
settings. Also, have a look at contrib/zfsrc (it should usually go in
/etc/zfs/)

Cheers,
Seth
Post by Ryan How
Hi,
I've noticed when using ZFS-Fuse, writes to the disk burst and then
pause, burst and then pause, as accoring to zfs iostat. It seems to
burst to about 60MB/s, then pause for 2 seconds, then a burst again.
I'm guessing it is some kind of write buffer that fills up, then
flushes to disk.
If there any way to tune it to be smoother?. I've come across a few
threads in the BSD ZFS about limiting write throughput, which seemed
to actually give a significant increase in write speed!. But they
aren't really applicable here from what I can tell?
NAME
STATE READ WRITE CKSUM
glhzpool1
ONLINE 0 0 0
mirror-0
ONLINE 0 0 0
disk/by-id/ata-ST1000DM003-9YN162_W1D08X7Y
ONLINE 0 0 0
disk/by-id/ata-ST1000DM003-9YN162_W1D092GT
ONLINE 0 0 0
logs
disk/by-id/ata-INTEL_SSDSC2CW060A3_CVCV153202UA060AGN-part3
ONLINE 0 0 0
cache
sdc5
ONLINE 0 0 0
Thanks!
Ryan
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-15 15:41:42 UTC
Permalink
This is one of the BSD threads I'm referring to

http://forums.freebsd.org/archive/index.php/t-15647.html

The variable appears to be zfs.txg.write_limit_override . No idea if
that is BSD specific or not. But it seems to be a similar problem to
what I am getting.

I'm getting writes of average about 20MB/s. Just a simple Mirror, 2 x
1TB Sata 7200rpm and SSD with 50GB l2arc and 3GB ZIL. I know ZFS does
fancy stuff compared to your traditional file system, but this seems a
bit low?.... I'm trying to pause a VM and save it's state (about a 1GB
file), then take a snapshot, then start the VM up again. The whole
process takes about 3-4 minutes.

I guess I can always add a few more HDDs to speed it up :).
Post by sgheeren
ZFS on Solaris used to have tunables that zfs-fuse never directly
supported. See the ZFS Evil Tuning Guide
<http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#The_Tunables>
You don't mention the BSD settings that you refer to, but I wouldn't
be surprised when you actually mean these.
Most of these tunables are still available as static variables. E.g.
you can find
src/lib/libzpool/dmu_zfetch.c:int zfs_prefetch_disable = 0;
or
src/lib/libzpool/zil.c:int zil_disable = 0; /* disable intent
logging */ // DON'T
Be sure you know what you're doing when you touch these internal
settings. Also, have a look at contrib/zfsrc (it should usually go in
/etc/zfs/)
Cheers,
Seth
Post by Ryan How
Hi,
I've noticed when using ZFS-Fuse, writes to the disk burst and then
pause, burst and then pause, as accoring to zfs iostat. It seems to
burst to about 60MB/s, then pause for 2 seconds, then a burst again.
I'm guessing it is some kind of write buffer that fills up, then
flushes to disk.
If there any way to tune it to be smoother?. I've come across a few
threads in the BSD ZFS about limiting write throughput, which seemed
to actually give a significant increase in write speed!. But they
aren't really applicable here from what I can tell?
NAME
STATE READ WRITE CKSUM
glhzpool1
ONLINE 0 0 0
mirror-0
ONLINE 0 0 0
disk/by-id/ata-ST1000DM003-9YN162_W1D08X7Y
ONLINE 0 0 0
disk/by-id/ata-ST1000DM003-9YN162_W1D092GT
ONLINE 0 0 0
logs
disk/by-id/ata-INTEL_SSDSC2CW060A3_CVCV153202UA060AGN-part3
ONLINE 0 0 0
cache
sdc5
ONLINE 0 0 0
Thanks!
Ryan
--
To visit our Web site, click on http://zfs-fuse.net/
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
sgheeren
2012-04-15 16:22:45 UTC
Permalink
Post by Ryan How
The variable appears to be zfs.txg.write_limit_override . No idea if
that is BSD specific or not. But it seems to be a similar problem to
what I am getting.
If you want to try it:

***@mint12:~/custom/_ZFS$ git grep write_limit_override
src/lib/libzpool/dsl_pool.c:uint64_t zfs_write_limit_override = 0;

Needless to say you need to build your own zfs-fuse from source

Cheers,
Seth
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-17 14:23:08 UTC
Permalink
No worries. I already built it from source.

Do you think about 100 of these [flush-0:nn] showing up when I do a ps
ax may be related?. It is like the buffer fills and it all stalls while
the buffer is flushed.

I think I'm getting a bit over ZFS. It looks really great, but trying to
get it to work is a nightmare :). Although my other system which has
been running for months on end has no trouble... but that doesn't use
snapshots at all.
Post by sgheeren
Post by Ryan How
The variable appears to be zfs.txg.write_limit_override . No idea if
that is BSD specific or not. But it seems to be a similar problem to
what I am getting.
src/lib/libzpool/dsl_pool.c:uint64_t zfs_write_limit_override = 0;
Needless to say you need to build your own zfs-fuse from source
Cheers,
Seth
--
To visit our Web site, click on http://zfs-fuse.net/
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
sgheeren
2012-04-17 15:54:38 UTC
Permalink
Post by Ryan How
I think I'm getting a bit over ZFS. It looks really great, but trying
to get it to work is a nightmare :). Although my other system which
has been running for months on end has no trouble... but that doesn't
use snapshots at all.
I've been running zfs-fuse for several years with avg. 26 filesystems
and using snapshots. I have reduced the amount of snapshots, since I ran
into severe performance issues (mainly send/receive but also pool import
if a dataset deletion action was incomplete).

However, these issues were known upstream issues, and might have been
fixed with the new pool versions (26+ if I remember well).

Then again, I'm ogling btrfs and zfs-on-linux. I'm quite content with
the functionality of zfs-on-linux (although the performance benefit is
not quite as disastrous as I might have expected and it has a few
stability snags with nfs for me, it has a few more features).

Btrfs has the 'pre' of becoming the active on linux (well supported for
boot, e.g.) but is vastly (I really mean vastly) less admin-friendly.

My $0.02
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Aneurin Price
2012-04-17 16:19:10 UTC
Permalink
Then again, I'm ogling btrfs and zfs-on-linux. I'm quite content with the
functionality of zfs-on-linux (although the performance benefit is not quite
as disastrous as I might have expected and it has a few stability snags with
nfs for me, it has a few more features).
Could you elaborate on that? I've been considering switching to nfs
from samba since it more than doubles file transfer throughput. So far
the only reason I haven't done so is because I haven't had time to
think through any possible security implications, but if there are
stability problems as well then it would be good to know.
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
sgheeren
2012-04-17 17:11:05 UTC
Permalink
Post by Aneurin Price
Could you elaborate on that? I've been considering switching to nfs
from samba since it more than doubles file transfer throughput. So far
the only reason I haven't done so is because I haven't had time to
think through any possible security implications, but if there are
stability problems as well then it would be good to know.
I run into quite predictable (but seemingly harmless?) null reference
problems. I think it is a known bug and I've been too lazy to find out
whether it is a match.

The other (maybe related?) issue is after a while (apparently, after
some time of inactivity) the nfs mount may become non-responsive
blocking the client system. This might be a consequence of the first
issue, in case it exhaust a pool of workers of some sort and the next
request blocks since all workers died.

This is the only gripe I had, and I might find that by upgrading to tip
of master, it is already fixed

Seth
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Christ Schlacta
2012-04-17 17:15:31 UTC
Permalink
I've encountered absolutely no stability issues with nfs in recent months.
I use nfs to export data to Linux nodes and samba for windows started and
domain members.
Post by Aneurin Price
Then again, I'm ogling btrfs and zfs-on-linux. I'm quite content with the
functionality of zfs-on-linux (although the performance benefit is not
quite
as disastrous as I might have expected and it has a few stability snags
with
nfs for me, it has a few more features).
Could you elaborate on that? I've been considering switching to nfs
from samba since it more than doubles file transfer throughput. So far
the only reason I haven't done so is because I haven't had time to
think through any possible security implications, but if there are
stability problems as well then it would be good to know.
--
To visit our Web site, click on http://zfs-fuse.net/
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Aneurin Price
2012-04-18 10:43:44 UTC
Permalink
I've encountered absolutely no stability issues with nfs in recent months. I
use nfs to export data to Linux nodes and samba for windows started and
domain members.
Thanks for the response, both. (And sorry for hijacking the thread!)
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Ryan How
2012-04-18 03:59:16 UTC
Permalink
Now I'm getting the OS reporting timeout when flushing the buffers. I
think something is quite wrong with my setup :/. It isn't even under any
load 95% of the time. I think I'm gonna go back to EXT4 / LVM / RAID
(which is no where near as "nice", but at least it works :). Just for a
while until I can sort out these issues.

Do you generally find performance of ZOL better? I might try looking
down that path...
Post by sgheeren
Post by Ryan How
I think I'm getting a bit over ZFS. It looks really great, but trying
to get it to work is a nightmare :). Although my other system which
has been running for months on end has no trouble... but that doesn't
use snapshots at all.
I've been running zfs-fuse for several years with avg. 26 filesystems
and using snapshots. I have reduced the amount of snapshots, since I
ran into severe performance issues (mainly send/receive but also pool
import if a dataset deletion action was incomplete).
However, these issues were known upstream issues, and might have been
fixed with the new pool versions (26+ if I remember well).
Then again, I'm ogling btrfs and zfs-on-linux. I'm quite content with
the functionality of zfs-on-linux (although the performance benefit is
not quite as disastrous as I might have expected and it has a few
stability snags with nfs for me, it has a few more features).
Btrfs has the 'pre' of becoming the active on linux (well supported
for boot, e.g.) but is vastly (I really mean vastly) less admin-friendly.
My $0.02
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
Loading...