Discussion:
Another (different) out-of-order metadata issue with ZFS?
(too old to reply)
Durval Menezes
2013-07-19 13:06:31 UTC
Permalink
Hello folks,

During some extended testing with ZFS, I've encountered a weird issue,
which seems to me to be an "out-of-order metadata" issue (possibly
related, albeit much different, to the one posted here on this list by
Specus, with subject "ZFS timestamps out of order"):

0) I overwrote a file in ZFS by moving it from another filesystem
(ext4 in my case):
mv screenlog.2 /ZFSFS/SOMEPATH/somefile
(PS: /ZFSFS/somefile existed previous to the moving);

1) Then I edited the overwritten file with vim:
vim /ZFSFS/SOMEPATH/somefile
(PS: vim showed me the *correct*, new data for the file);

2) Then I started modifying the file, after a few seconds, vim showed
me the following message at the bottom of the screen:
W11: Warning: File "/ZFSFS/SOMEPATH/somefile" has changed since
editing started
See ":help W11" for more info.
[O]K, (L)oad File:

Of course I checked "ps" and "lsof" and there were no other processes
with that file open at the time.

I then tried reproducing (moving the file back from ZFS to ext4, then
recovering the original ZFS file using co as it was under RCS, then
repeating the above) but it did not happen again, at least not on my
first attempt.

So, is that a known issue? Is there anything I can do to help debug this?

Additional data:

a) I'm running ZoL 0.6.1 directly compiled from sources and installed
using the DKMS method, on a Springdale (very similar to CentOS) 6.4
system with the latest updates including kernel (which happens to be
kernel-2.6.32-358.11.1.el6.x86_64);

b) The ZFS layout at the moment of the test was:
sudo zpool status
pool: ZFS
state: DEGRADED
status: One or more devices could not be opened. Sufficient
replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://zfsonlinux.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 1h22m with 0 errors on Tue Jul 16
00:51:41 2013
config:
NAME
STATE READ WRITE CKSUM
ZFS
DEGRADED 0 0 0
mirror-0
DEGRADED 0 0 0
12595782546360574761
UNAVAIL 0 0 0 was /tmp/fake-sda1.img
scsi-SATA_WDC_WD7500BPKT-_WD-XXXXXXXX-part1
ONLINE 0 0 0
logs
ata-M4-CT256M4SSD3_XXXXXXXXXXXXXXXXXX-part5
ONLINE 0 0 0
cache
ata-M4-CT256M4SSD3_XXXXXXXXXXXXXXXXXX-part6
ONLINE 0 0 0
errors: No known data errors

(The "degraded" part above is because I created the mirror
with only one of the two VDEVs plus a fake "file VDEV" so as to be
able to add the real disk later).

c) The old overwritten file had ~7.8KB, the file I overwrote it with
had about the same size; the old file resided in a standard ext4
partition on my SSD.

Please advise.

Cheers,
--
Durval.
--
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-fuse+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Emmanuel Anne
2013-07-19 18:43:10 UTC
Permalink
Why do you send this to zfs-fuse if you are using ZoL ?
Anyway I have seen this issue once in zfs-fuse too, but didn't bother to
investigate, sorry...
Post by Durval Menezes
Hello folks,
During some extended testing with ZFS, I've encountered a weird issue,
which seems to me to be an "out-of-order metadata" issue (possibly
related, albeit much different, to the one posted here on this list by
0) I overwrote a file in ZFS by moving it from another filesystem
mv screenlog.2 /ZFSFS/SOMEPATH/somefile
(PS: /ZFSFS/somefile existed previous to the moving);
vim /ZFSFS/SOMEPATH/somefile
(PS: vim showed me the *correct*, new data for the file);
2) Then I started modifying the file, after a few seconds, vim showed
W11: Warning: File "/ZFSFS/SOMEPATH/somefile" has changed since
editing started
See ":help W11" for more info.
Of course I checked "ps" and "lsof" and there were no other processes
with that file open at the time.
I then tried reproducing (moving the file back from ZFS to ext4, then
recovering the original ZFS file using co as it was under RCS, then
repeating the above) but it did not happen again, at least not on my
first attempt.
So, is that a known issue? Is there anything I can do to help debug this?
a) I'm running ZoL 0.6.1 directly compiled from sources and installed
using the DKMS method, on a Springdale (very similar to CentOS) 6.4
system with the latest updates including kernel (which happens to be
kernel-2.6.32-358.11.1.el6.x86_64);
sudo zpool status
pool: ZFS
state: DEGRADED
status: One or more devices could not be opened. Sufficient
replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://zfsonlinux.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 1h22m with 0 errors on Tue Jul 16
00:51:41 2013
NAME
STATE READ WRITE CKSUM
ZFS
DEGRADED 0 0 0
mirror-0
DEGRADED 0 0 0
12595782546360574761
UNAVAIL 0 0 0 was /tmp/fake-sda1.img
scsi-SATA_WDC_WD7500BPKT-_WD-XXXXXXXX-part1
ONLINE 0 0 0
logs
ata-M4-CT256M4SSD3_XXXXXXXXXXXXXXXXXX-part5
ONLINE 0 0 0
cache
ata-M4-CT256M4SSD3_XXXXXXXXXXXXXXXXXX-part6
ONLINE 0 0 0
errors: No known data errors
(The "degraded" part above is because I created the mirror
with only one of the two VDEVs plus a fake "file VDEV" so as to be
able to add the real disk later).
c) The old overwritten file had ~7.8KB, the file I overwrote it with
had about the same size; the old file resided in a standard ext4
partition on my SSD.
Please advise.
Cheers,
--
Durval.
--
--
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.
--
my zfs-fuse git repository :
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-fuse+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Durval Menezes
2013-07-19 19:18:08 UTC
Permalink
Hi Emmanuel,
Post by Emmanuel Anne
Why do you send this to zfs-fuse if you are using ZoL ?
Wanted to hear your highly regarded opinion first... :-)
Post by Emmanuel Anne
Anyway I have seen this issue once in zfs-fuse too, but didn't bother to
investigate, sorry...
No prob. Thanks anyway.

Cheers,
--
Durval.
Post by Emmanuel Anne
Post by Durval Menezes
Hello folks,
During some extended testing with ZFS, I've encountered a weird issue,
which seems to me to be an "out-of-order metadata" issue (possibly
related, albeit much different, to the one posted here on this list by
0) I overwrote a file in ZFS by moving it from another filesystem
mv screenlog.2 /ZFSFS/SOMEPATH/somefile
(PS: /ZFSFS/somefile existed previous to the moving);
vim /ZFSFS/SOMEPATH/somefile
(PS: vim showed me the *correct*, new data for the file);
2) Then I started modifying the file, after a few seconds, vim showed
W11: Warning: File "/ZFSFS/SOMEPATH/somefile" has changed since
editing started
See ":help W11" for more info.
Of course I checked "ps" and "lsof" and there were no other processes
with that file open at the time.
I then tried reproducing (moving the file back from ZFS to ext4, then
recovering the original ZFS file using co as it was under RCS, then
repeating the above) but it did not happen again, at least not on my
first attempt.
So, is that a known issue? Is there anything I can do to help debug this?
a) I'm running ZoL 0.6.1 directly compiled from sources and installed
using the DKMS method, on a Springdale (very similar to CentOS) 6.4
system with the latest updates including kernel (which happens to be
kernel-2.6.32-358.11.1.el6.x86_64);
sudo zpool status
pool: ZFS
state: DEGRADED
status: One or more devices could not be opened. Sufficient
replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://zfsonlinux.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 1h22m with 0 errors on Tue Jul 16
00:51:41 2013
NAME
STATE READ WRITE CKSUM
ZFS
DEGRADED 0 0 0
mirror-0
DEGRADED 0 0 0
12595782546360574761
UNAVAIL 0 0 0 was /tmp/fake-sda1.img
scsi-SATA_WDC_WD7500BPKT-_WD-XXXXXXXX-part1
ONLINE 0 0 0
logs
ata-M4-CT256M4SSD3_XXXXXXXXXXXXXXXXXX-part5
ONLINE 0 0 0
cache
ata-M4-CT256M4SSD3_XXXXXXXXXXXXXXXXXX-part6
ONLINE 0 0 0
errors: No known data errors
(The "degraded" part above is because I created the mirror
with only one of the two VDEVs plus a fake "file VDEV" so as to be
able to add the real disk later).
c) The old overwritten file had ~7.8KB, the file I overwrote it with
had about the same size; the old file resided in a standard ext4
partition on my SSD.
Please advise.
Cheers,
--
Durval.
--
--
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups
"zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.
--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
--
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.
--
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-fuse+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Loading...