invalid vdev, corrupted label 2 and label 3, patches to ignore label anyone?

Discussion:

2010-10-06 02:36:20 UTC

Hi,

I had problem with zpool import after a two disk drive failure,
my system is a raidz with 3 disk arrays,
I managed to recover the content of the disks and move it to another
healthy
disk, and put the disk back online, but zfs import doesnt work.
from zfs import, the disk order is correct, and showing ONLINE,
but the import process said about invalid vdev configuration,
I build the array on FREENAS (FreeBSD 7.1), zfs v6,
and hoping that zfs-fuse with linux can recover the data.

I read the post from someone a while back here
http://comments.gmane.org/gmane.os.solaris.opensolaris.zfs/24743

he try to modify the zfs-fuse source code to only read LABEL 0 and
LABEL 1,
because zdb -l shows that LABEL 2 and LABEL 3 is corrupted,
just like my situation right now.

is it possible or anyone willing to help me how to slightly modify
the zfs-fuse source code to only read LABEL 0 and LABEL 1, ignoring
other LABEL that can not be read?

appreciate for the help.

regards,

diyanc

--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/

Emmanuel Anne

2010-10-07 07:21:11 UTC

Permalink

err,output of zdb -l please ?

Post by DC
Hi,
I had problem with zpool import after a two disk drive failure,
my system is a raidz with 3 disk arrays,
I managed to recover the content of the disks and move it to another
healthy
disk, and put the disk back online, but zfs import doesnt work.
from zfs import, the disk order is correct, and showing ONLINE,
but the import process said about invalid vdev configuration,
I build the array on FREENAS (FreeBSD 7.1), zfs v6,
and hoping that zfs-fuse with linux can recover the data.
I read the post from someone a while back here
http://comments.gmane.org/gmane.os.solaris.opensolaris.zfs/24743
he try to modify the zfs-fuse source code to only read LABEL 0 and
LABEL 1,
because zdb -l shows that LABEL 2 and LABEL 3 is corrupted,
just like my situation right now.
is it possible or anyone willing to help me how to slightly modify
the zfs-fuse source code to only read LABEL 0 and LABEL 1, ignoring
other LABEL that can not be read?
appreciate for the help.
regards,
diyanc
--
To visit our Web site, click on http://zfs-fuse.net/

--
zfs-fuse git repository :
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/

Emmanuel Anne

2010-10-07 07:28:13 UTC

Permalink

I'd say the "dd" solution is easier.
Modifying the zfs-fuse source to do that implies to spend time looking for
what to modify exactly in the source.
I didn't find any post from this kyle in my archives, so he probably never
posted his modifications, and so I have absolutely no idea how he did that.
The dd method implies to look into the zfs on disk format documentation to
see where the labels are located,and the copy the correct labels from drive
1 to drive 2. You have to be extremely careful when doing this, but normally
it should work.

Anyway if you still want to try the zfs-fuse modification, start by sending
the output of zdb -l for the 2 drives, but no promise here...

Post by Emmanuel Anne
err,output of zdb -l please ?
Hi,

Post by DC
I had problem with zpool import after a two disk drive failure,
my system is a raidz with 3 disk arrays,
I managed to recover the content of the disks and move it to another
healthy
disk, and put the disk back online, but zfs import doesnt work.
from zfs import, the disk order is correct, and showing ONLINE,
but the import process said about invalid vdev configuration,
I build the array on FREENAS (FreeBSD 7.1), zfs v6,
and hoping that zfs-fuse with linux can recover the data.
I read the post from someone a while back here
http://comments.gmane.org/gmane.os.solaris.opensolaris.zfs/24743
he try to modify the zfs-fuse source code to only read LABEL 0 and
LABEL 1,
because zdb -l shows that LABEL 2 and LABEL 3 is corrupted,
just like my situation right now.
is it possible or anyone willing to help me how to slightly modify
the zfs-fuse source code to only read LABEL 0 and LABEL 1, ignoring
other LABEL that can not be read?
appreciate for the help.
regards,
diyanc
--
To visit our Web site, click on http://zfs-fuse.net/

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

2010-10-08 07:33:34 UTC

Permalink

Hi,

would you mind giving me an example of copying the label with dd?
searching reference on using dd to copy the label I'm a bit loss with
the calculation of the skip= parameter
(according to this forum
http://forums.freebsd.org/showpost.php?p=34667&postcount=3)

I'll post the zdb -l output soon.

thank you for your help.

regards,

diyanc

Post by Emmanuel Anne
I'd say the "dd" solution is easier.
Modifying the zfs-fuse source to do that implies to spend time looking for
what to modify exactly in the source.
I didn't find any post from this kyle in my archives, so he probably never
posted his modifications, and so I have absolutely no idea how he did that.
The dd method implies to look into the zfs on disk format documentation to
see where the labels are located,and the copy the correct labels from drive
1 to drive 2. You have to be extremely careful when doing this, but normally
it should work.
Anyway if you still want to try the zfs-fuse modification, start by sending
the output of zdb -l for the 2 drives, but no promise here...

Post by Emmanuel Anne
err,output of zdb -l please ?

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/

Emmanuel Anne

2010-10-08 08:28:46 UTC

Permalink

That's a simple question.
The easiest answer is by a quick patch I just committed so that zdb -l now
also displays labels offsets and sizes.
You'll find the patch there :
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

it's the master branch of my repository.

Notice that the 2 1st labels are at the beginning of the device, and the 2
last ones are at the end.
If you miss 2 of them, then I fear it's probably because something went
wrong in your restoration.
But since you still have the original disk + a backup, you should be able to
experiment with this...

Post by DC
Hi,
would you mind giving me an example of copying the label with dd?
searching reference on using dd to copy the label I'm a bit loss with
the calculation of the skip= parameter
(according to this forum
http://forums.freebsd.org/showpost.php?p=34667&postcount=3)
I'll post the zdb -l output soon.
thank you for your help.
regards,
diyanc

Post by Emmanuel Anne
I'd say the "dd" solution is easier.
Modifying the zfs-fuse source to do that implies to spend time looking

for

Post by Emmanuel Anne
what to modify exactly in the source.
I didn't find any post from this kyle in my archives, so he probably

never

Post by Emmanuel Anne
posted his modifications, and so I have absolutely no idea how he did

that.

Post by Emmanuel Anne
The dd method implies to look into the zfs on disk format documentation

Post by Emmanuel Anne
see where the labels are located,and the copy the correct labels from

drive

Post by Emmanuel Anne
1 to drive 2. You have to be extremely careful when doing this, but

normally

Post by Emmanuel Anne
it should work.
Anyway if you still want to try the zfs-fuse modification, start by

sending

Post by Emmanuel Anne
the output of zdb -l for the 2 drives, but no promise here...

Post by Emmanuel Anne
err,output of zdb -l please ?

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To visit our Web site, click on http://zfs-fuse.net/

2010-10-11 00:10:51 UTC

Permalink

I'm sorry for the late reply,

I had'nt the chance to run your modification yet,
since the disk is somewhere else (at the recovery services),

about the calculation, please correct me if I'm wrong, for example
if from the zdb -l output my disk asize is 300417024, then
what I need to copy the LABEL from the start of the disk to the end
of the disk is

8192+(asize/disk unit/512)

8192+(300417024/3/512) = 203776 -> converted to hexadecimal 0x31c00
(I don't know where did 8192 come from, care for explanation?)

then with dd,

dd if=/dev/ad4 of=/ad4_head_label count=0x200
dd if=/ad4_head_label of=/ad4 skip=0x31c00 count=0x200

then repeat with the rest of the disk.

maybe it will be much more easier to use your patch to find the correct offset.

thank you,

regards,

diyanc

Post by Emmanuel Anne
That's a simple question.
The easiest answer is by a quick patch I just committed so that zdb -l now
also displays labels offsets and sizes.
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
it's the master branch of my repository.
Notice that the 2 1st labels are at the beginning of the device, and the 2
last ones are at the end.
If you miss 2 of them, then I fear it's probably because something went
wrong in your restoration.
But since you still have the original disk + a backup, you should be able to
experiment with this...

Post by Emmanuel Anne
err,output of zdb -l please ?

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To visit our Web site, click on http://zfs-fuse.net/

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/

Emmanuel Anne

2010-10-11 08:00:10 UTC

Permalink

don't bother to caculate, just use the patch for zdb, it will display the
offset and the size of each label,then just copy and paste the offsets to a
dd command.
But I said I doubt it will work, it's very likely your restoration skipped a
few sectors and it would explain why you miss the labels at the end.

No problem for the delay, take your time.

Post by DC
I'm sorry for the late reply,
I had'nt the chance to run your modification yet,
since the disk is somewhere else (at the recovery services),
about the calculation, please correct me if I'm wrong, for example
if from the zdb -l output my disk asize is 300417024, then
what I need to copy the LABEL from the start of the disk to the end
of the disk is
8192+(asize/disk unit/512)
8192+(300417024/3/512) = 203776 -> converted to hexadecimal 0x31c00
(I don't know where did 8192 come from, care for explanation?)
then with dd,
dd if=/dev/ad4 of=/ad4_head_label count=0x200
dd if=/ad4_head_label of=/ad4 skip=0x31c00 count=0x200
then repeat with the rest of the disk.
maybe it will be much more easier to use your patch to find the correct offset.
thank you,
regards,
diyanc

Post by Emmanuel Anne
That's a simple question.
The easiest answer is by a quick patch I just committed so that zdb -l

now

Post by Emmanuel Anne
also displays labels offsets and sizes.
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
it's the master branch of my repository.
Notice that the 2 1st labels are at the beginning of the device, and the

Post by Emmanuel Anne
last ones are at the end.
If you miss 2 of them, then I fear it's probably because something went
wrong in your restoration.
But since you still have the original disk + a backup, you should be able

Post by Emmanuel Anne
experiment with this...

documentation

Post by Emmanuel Anne

Post by DC

Post by Emmanuel Anne
to
see where the labels are located,and the copy the correct labels from drive
1 to drive 2. You have to be extremely careful when doing this, but normally
it should work.
Anyway if you still want to try the zfs-fuse modification, start by sending
the output of zdb -l for the 2 drives, but no promise here...

Post by Emmanuel Anne
err,output of zdb -l please ?

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To visit our Web site, click on http://zfs-fuse.net/

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To visit our Web site, click on http://zfs-fuse.net/

diyan christian

2010-10-14 03:38:32 UTC

Permalink

Hi,

this is the output from zdb -l,

***@ubuntu-ocs:~# zdb -l /dev/sdb

--------------------------------------------

LABEL 0 offset 0 size 262144

--------------------------------------------

version: 6

name: 'zpool'

state: 0

txg: 4453331

pool_guid: 10872466618542081051

hostid: 0

hostname: ''

top_guid: 10798917829253125038

guid: 10863830116566972496

vdev_tree:

type: 'raidz'

id: 0

guid: 10798917829253125038

nparity: 1

metaslab_array: 14

metaslab_shift: 35

ashift: 9

asize: 6001182375936

children[0]:

type: 'disk'

id: 0

guid: 10863830116566972496

path: '/dev/ad6'

devid: 'ad:5XW14ALE'

whole_disk: 0

DTL: 26

children[1]:

type: 'disk'

id: 1

guid: 16447223924793968503

path: '/dev/ad7'

devid: 'ad:6XW0GXZ0'

whole_disk: 0

DTL: 25

children[2]:

type: 'disk'

id: 2

guid: 6223582794630230370

path: '/dev/ad10'

devid: 'ad:9XW0B3A3'

whole_disk: 0

DTL: 8193

--------------------------------------------

LABEL 1 offset 262144 size 262144

--------------------------------------------

version: 6

name: 'zpool'

state: 0

txg: 4453331

pool_guid: 10872466618542081051

hostid: 0

hostname: ''

top_guid: 10798917829253125038

guid: 10863830116566972496

vdev_tree:

type: 'raidz'

id: 0

guid: 10798917829253125038

nparity: 1

metaslab_array: 14

metaslab_shift: 35

ashift: 9

asize: 6001182375936

children[0]:

type: 'disk'

id: 0

guid: 10863830116566972496

path: '/dev/ad6'

devid: 'ad:5XW14ALE'

whole_disk: 0

DTL: 26

children[1]:

type: 'disk'

id: 1

guid: 16447223924793968503

path: '/dev/ad7'

devid: 'ad:6XW0GXZ0'

whole_disk: 0

DTL: 25

children[2]:

type: 'disk'

id: 2

guid: 6223582794630230370

path: '/dev/ad10'

devid: 'ad:9XW0B3A3'

whole_disk: 0

DTL: 8193

--------------------------------------------

LABEL 2 offset 2000398319616 size 262144

--------------------------------------------

failed to unpack label 2

--------------------------------------------

LABEL 3 offset 2000398581760 size 262144

--------------------------------------------

failed to unpack label 3

I did try and sucessfully copy the label at the start offset to the end
at the specified offset from the zdb -l output.

now the zdb -l output

***@ubuntu-ocs:~# zdb -l /dev/sdb

--------------------------------------------

LABEL 0 offset 0 size 262144

--------------------------------------------

version: 6

name: 'zpool'

state: 0

txg: 4453331

pool_guid: 10872466618542081051

hostid: 0

hostname: ''

top_guid: 10798917829253125038

guid: 10863830116566972496

vdev_tree:

type: 'raidz'

id: 0

guid: 10798917829253125038

nparity: 1

metaslab_array: 14

metaslab_shift: 35

ashift: 9

asize: 6001182375936

children[0]:

type: 'disk'

id: 0

guid: 10863830116566972496

path: '/dev/ad6'

devid: 'ad:5XW14ALE'

whole_disk: 0

DTL: 26

children[1]:

type: 'disk'

id: 1

guid: 16447223924793968503

path: '/dev/ad7'

devid: 'ad:6XW0GXZ0'

whole_disk: 0

DTL: 25

children[2]:

type: 'disk'

id: 2

guid: 6223582794630230370

path: '/dev/ad10'

devid: 'ad:9XW0B3A3'

whole_disk: 0

DTL: 8193

--------------------------------------------

LABEL 1 offset 262144 size 262144

--------------------------------------------

version: 6

name: 'zpool'

state: 0

txg: 4453331

pool_guid: 10872466618542081051

hostid: 0

hostname: ''

top_guid: 10798917829253125038

guid: 10863830116566972496

vdev_tree:

type: 'raidz'

id: 0

guid: 10798917829253125038

nparity: 1

metaslab_array: 14

metaslab_shift: 35

ashift: 9

asize: 6001182375936

children[0]:

type: 'disk'

id: 0

guid: 10863830116566972496

path: '/dev/ad6'

devid: 'ad:5XW14ALE'

whole_disk: 0

DTL: 26

children[1]:

type: 'disk'

id: 1

guid: 16447223924793968503

path: '/dev/ad7'

devid: 'ad:6XW0GXZ0'

whole_disk: 0

DTL: 25

children[2]:

type: 'disk'

id: 2

guid: 6223582794630230370

path: '/dev/ad10'

devid: 'ad:9XW0B3A3'

whole_disk: 0

DTL: 8193

--------------------------------------------

LABEL 2 offset 2000398319616 size 262144

--------------------------------------------

version: 6

name: 'zpool'

state: 0

txg: 4453331

pool_guid: 10872466618542081051

hostid: 0

hostname: ''

top_guid: 10798917829253125038

guid: 10863830116566972496

vdev_tree:

type: 'raidz'

id: 0

guid: 10798917829253125038

nparity: 1

metaslab_array: 14

metaslab_shift: 35

ashift: 9

asize: 6001182375936

children[0]:

type: 'disk'

id: 0

guid: 10863830116566972496

path: '/dev/ad6'

devid: 'ad:5XW14ALE'

whole_disk: 0

DTL: 26

children[1]:

type: 'disk'

id: 1

guid: 16447223924793968503

path: '/dev/ad7'

devid: 'ad:6XW0GXZ0'

whole_disk: 0

DTL: 25

children[2]:

type: 'disk'

id: 2

guid: 6223582794630230370

path: '/dev/ad10'

devid: 'ad:9XW0B3A3'

whole_disk: 0

DTL: 8193

--------------------------------------------

LABEL 3 offset 2000398581760 size 262144

--------------------------------------------

version: 6

name: 'zpool'

state: 0

txg: 4453331

pool_guid: 10872466618542081051

hostid: 0

hostname: ''

top_guid: 10798917829253125038

guid: 10863830116566972496

vdev_tree:

type: 'raidz'

id: 0

guid: 10798917829253125038

nparity: 1

metaslab_array: 14

metaslab_shift: 35

ashift: 9

asize: 6001182375936

children[0]:

type: 'disk'

id: 0

guid: 10863830116566972496

path: '/dev/ad6'

devid: 'ad:5XW14ALE'

whole_disk: 0

DTL: 26

children[1]:

type: 'disk'

id: 1

guid: 16447223924793968503

path: '/dev/ad7'

devid: 'ad:6XW0GXZ0'

whole_disk: 0

DTL: 25

children[2]:

type: 'disk'

id: 2

guid: 6223582794630230370

path: '/dev/ad10'

devid: 'ad:9XW0B3A3'

whole_disk: 0

DTL: 8193

but still unable to import,

***@ubuntu-ocs:~# zpool import

pool: zpool

id: 10872466618542081051

state: FAULTED

status: One or more devices contains corrupted data.

action: The pool cannot be imported due to damaged devices or data.

The pool may be active on another system, but can be imported using

the '-f' flag.

see: http://www.sun.com/msg/ZFS-8000-5E

config:

zpool FAULTED corrupted data

raidz1-0 ONLINE

disk/by-id/ata-ST32000542AS_5XW14ALE UNAVAIL corrupted data

disk/by-id/ata-ST32000542AS_6XW0GXZ0 ONLINE

disk/by-id/ata-ST32000542AS_9XW0B3A3 UNAVAIL corrupted data

***@ubuntu-ocs:~# zpool import zpool

cannot import 'zpool': pool may be in use from other system

use '-f' to import anyway

***@ubuntu-ocs:~# zpool import -f zpool

cannot import 'zpool': I/O error

Destroy and re-create the pool from

a backup source.

I try to boot with FreeBSD liveCD 8.1, from zdb -l output,
label 2 and label 3 is failed to unpack, I think this has something to do
with the HPA, from linux dmesg snippet:

[ 1.942444] ata2.00: HPA unlocked: 3907027055 -> 3907029168, native
3907029168

while in FreeBSD reported as

3876018 = 3907026144 sectors = 1907727MB

starting to feel desperate, since I had no backup.
I learned the hard way.

is there still hope? is there any recovery services that can help at this stage?

thank you,

regards,

diyanc

Post by Emmanuel Anne
don't bother to caculate, just use the patch for zdb, it will display the
offset and the size of each label,then just copy and paste the offsets to a
dd command.
But I said I doubt it will work, it's very likely your restoration skipped a
few sectors and it would explain why you miss the labels at the end.
No problem for the delay, take your time.

Post by Emmanuel Anne
I'd say the "dd" solution is easier.
Modifying the zfs-fuse source to do that implies to spend time looking
for
what to modify exactly in the source.
I didn't find any post from this kyle in my archives, so he probably never
posted his modifications, and so I have absolutely no idea how he did that.
The dd method implies to look into the zfs on disk format documentation
to
see where the labels are located,and the copy the correct labels from drive
1 to drive 2. You have to be extremely careful when doing this, but normally
it should work.
Anyway if you still want to try the zfs-fuse modification, start by sending
the output of zdb -l for the 2 drives, but no promise here...

Post by Emmanuel Anne
err,output of zdb -l please ?

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To visit our Web site, click on http://zfs-fuse.net/

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To visit our Web site, click on http://zfs-fuse.net/

--
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
--
To visit our Web site, click on http://zfs-fuse.net/

--
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To visit our Web site, click on http://zfs-fuse.net/

Emmanuel Anne

2010-10-14 06:25:30 UTC

Permalink

This post might be inappropriate. Click to display it.