"zfs list -t snapshot" missing entries

[Aneurin Price]

Hello all,

I've been testing Darik Horn's Linux version of zfs-auto-snapshot
today. This is the first time I've tried using snapshots with zfs-fuse
(I'm using 0.7.0), and I'm confused about why 'zfs list' doesn't seem
to show all the snapshots made, unless they're listed explicitly.

For example:

$zfs list 'srv/***@zfs-auto-snap_frequent-2011-12-22-1400'
NAME USED AVAIL REFER
MOUNTPOINT
srv/***@zfs-auto-snap_frequent-2011-12-22-1400 0 - 3.83T -

So there's a snapshot named
'srv/***@zfs-auto-snap_frequent-2011-12-22-1400', but:

$zfs list -t snapshot|grep video
srv/***@zfs-auto-snap_hourly-2011-12-22-1117 0
- 3.79T -
srv/***@zfs-auto-snap_frequent-2011-12-22-1300 58K
- 3.82T -
srv/***@zfs-auto-snap_frequent-2011-12-22-1345 26K
- 3.82T -
srv/***@zfs-auto-snap_frequent-2011-12-22-1415 0
- 3.83T -

That snapshot isn't listed. What am I missing here?
The 14:00 snapshot should already have been automatically removed by
now, but zfs-auto-snapshot doesn't even know it's there since 'zfs
list' isn't listing it; I'm worried that I'm going to end up with
hundreds of snapshots hanging around that I can't remove without
guessing that they're there so I can name them explicitly.

Can anyone tell me what's happening?

Thanks,
Nye

[Darik Horn]

Post by Aneurin Price
I've been testing Darik Horn's Linux version of zfs-auto-snapshot
today. This is the first time I've tried using snapshots with zfs-fuse
(I'm using 0.7.0), and I'm confused about why 'zfs list' doesn't seem
to show all the snapshots made, unless they're listed explicitly.

Use `zfs list -t all` or `zfs list -t snapshot` and you should get the
desired result.

By default, snapshots are no longer displayed in the `zfs list` output.

[Aneurin Price]

Post by Darik Horn

Post by Aneurin Price
I've been testing Darik Horn's Linux version of zfs-auto-snapshot
today. This is the first time I've tried using snapshots with zfs-fuse
(I'm using 0.7.0), and I'm confused about why 'zfs list' doesn't seem
to show all the snapshots made, unless they're listed explicitly.

Use `zfs list -t all` or `zfs list -t snapshot` and you should get the
desired result.

Um, I already am - look at the examples I pasted in :P
(And apologies for the wrapping BTW; damn Gmail)

Thanks,
Nye

[Darik Horn]

Post by Aneurin Price
Um, I already am - look at the examples I pasted in :P
(And apologies for the wrapping BTW; damn Gmail)

The zfs-auto-snapshot program for Linux never invokes a bare `zfs
list` command, so you need to explain why the default behavior of `zfs
list` is causing a problem. Both instances of it are qualified with
the `-t` parameter.

Snapshots are not deleted by zfs-auto-snapshot unless the `--keep`
parameter is given. Look at the bundled cron parts for an example of
how to invoke zfs-auto-snapshot properly.

[Aneurin Price]

Post by Darik Horn

Post by Aneurin Price
Um, I already am - look at the examples I pasted in :P
(And apologies for the wrapping BTW; damn Gmail)

The zfs-auto-snapshot program for Linux never invokes a bare `zfs
list` command, so you need to explain why the default behavior of `zfs
list` is causing a problem.  Both instances of it are qualified with
the `-t` parameter.

I think you've misunderstood me; this isn't a problem with
zfs-auto-snapshot, so far as I can tell - I just mentioned it for
completeness in case it turns out to be relevant. I'll restate the
most relevant parts which might help to clarify:

There exists a snapshot named
'srv/***@zfs-auto-snap_frequent-2011-12-22-1400', which can be
demonstrated by giving it explicitly as an argument to 'zfs list'.
(This is just an example; there are other snapshots with the same
issue.)

Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
creation -o name' as zfs-auto-snapshot does) shows a list of several
snapshots, *but not all of them*. In particular,
'srv/***@zfs-auto-snap_frequent-2011-12-22-1400' is not listed even
though it's verified to exist.

Does that make it clearer?

[Darik Horn]

Post by Aneurin Price
I think you've misunderstood me; this isn't a problem with
zfs-auto-snapshot, so far as I can tell - I just mentioned it for
completeness in case it turns out to be relevant. I'll restate the
There exists a snapshot named
demonstrated by giving it explicitly as an argument to 'zfs list'.
(This is just an example; there are other snapshots with the same
issue.)
Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
creation -o name' as zfs-auto-snapshot does) shows a list of several
snapshots, *but not all of them*. In particular,
though it's verified to exist.
Does that make it clearer?

Yes, my bad.

Please create a recursive snapshot on the pool like this:

# zfs snapshot -r ***@test-recursive

And a regular snapshot like this:

# zfs snapshot ***@test-nonrecursive

Do they both appear in the snapshot list?

# zfs list -t snapshot | grep @test

[Aneurin Price]

Post by Darik Horn

Post by Aneurin Price
I think you've misunderstood me; this isn't a problem with
zfs-auto-snapshot, so far as I can tell - I just mentioned it for
completeness in case it turns out to be relevant. I'll restate the
There exists a snapshot named
demonstrated by giving it explicitly as an argument to 'zfs list'.
(This is just an example; there are other snapshots with the same
issue.)
Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
creation -o name' as zfs-auto-snapshot does) shows a list of several
snapshots, *but not all of them*. In particular,
though it's verified to exist.
Does that make it clearer?

Yes, my bad.

Hmm, now I have a sinking feeling:

cannot create snapshot 'srv/***@test-recursive': dataset is busy
no snapshots were created

That dataset (srv/broken) has been sitting around for so long I'd
forgotten about it. It can't be destroyed without crashing zfs :(.

Everything scrubs just fine, but now I'm wondering if the problem is
simply that my pool is borked.

Post by Darik Horn
Do they both appear in the snapshot list?

The non-recursive version works, and shows up.
I guess it may not be relevant in light of that fact that the pool
seems to be damaged, but I did try the following:
# zfs snapshot -r srv/***@test-recursive
# zfs snapshot srv/***@test-nonrecursive
# zfs list -t snapshot|grep @test
srv/***@test-recursive 0
- 3.83T -

All the snapshots that aren't showing up were made with '-r' though,
so I don't think there's any correlation between it being a recursive
snapshot or not. So far it seems frustratingly random.

I take it you've not seen anything like this before - any ideas short
of destroying and recreating the pool?

Thanks for your time,
Nye

[Emmanuel Anne]

Well eventually the estimation of how many snapshots were created with
their names could help to see if the problem is because there were too many
snapshots (?) or because of something else.
(and subvolumes also maybe).

The ideal would be to be able to reproduce the creation of all these
snapshots on a disk image to see what happens (with a script ?).

Post by Aneurin Price

Post by Darik Horn

Post by Aneurin Price
I think you've misunderstood me; this isn't a problem with
zfs-auto-snapshot, so far as I can tell - I just mentioned it for
completeness in case it turns out to be relevant. I'll restate the
There exists a snapshot named
demonstrated by giving it explicitly as an argument to 'zfs list'.
(This is just an example; there are other snapshots with the same
issue.)
Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
creation -o name' as zfs-auto-snapshot does) shows a list of several
snapshots, *but not all of them*. In particular,
though it's verified to exist.
Does that make it clearer?

Yes, my bad.

no snapshots were created
That dataset (srv/broken) has been sitting around for so long I'd
forgotten about it. It can't be destroyed without crashing zfs :(.
Everything scrubs just fine, but now I'm wondering if the problem is
simply that my pool is borked.

Post by Darik Horn
Do they both appear in the snapshot list?

The non-recursive version works, and shows up.
I guess it may not be relevant in light of that fact that the pool
- 3.83T -
All the snapshots that aren't showing up were made with '-r' though,
so I don't think there's any correlation between it being a recursive
snapshot or not. So far it seems frustratingly random.
I take it you've not seen anything like this before - any ideas short
of destroying and recreating the pool?
Thanks for your time,
Nye
--
To visit our Web site, click on http://zfs-fuse.net/

[Aneurin Price]

Well eventually the estimation of how many snapshots were created with their
names could help to see if the problem is because there were too many
snapshots (?) or because of something else.
(and subvolumes also maybe).

There are only a few dozen, so that shouldn't be a problem. Since it's
not causing me any immediate trouble, I think I'll leave it until
after Christmas to investigate further.

(Actually, I might see what zfsonlinux does, since that should be
fairly quick and painless. I'll report back if it does anything
different.)

Nye

[Emmanuel Anne]

Just be careful to test zfs on linux first on a separate pool, just to be
on the safe side.

Post by Aneurin Price

Post by Emmanuel Anne
Well eventually the estimation of how many snapshots were created with

their

Post by Emmanuel Anne
names could help to see if the problem is because there were too many
snapshots (?) or because of something else.
(and subvolumes also maybe).

There are only a few dozen, so that shouldn't be a problem. Since it's
not causing me any immediate trouble, I think I'll leave it until
after Christmas to investigate further.
(Actually, I might see what zfsonlinux does, since that should be
fairly quick and painless. I'll report back if it does anything
different.)
Nye
--
To visit our Web site, click on http://zfs-fuse.net/

[Aneurin Price]

Post by Aneurin Price
(Actually, I might see what zfsonlinux does, since that should be
fairly quick and painless. I'll report back if it does anything
different.)

And it works! All the snapshots are listed by ZoL. I wonder if it's
just because it uses a newer version of ZFS (I haven't upgraded the
pool since I want to be able to go back to zfs-fuse).

And with this puzzle piece now I really am going to forget about it
until after Christmas.

[sgheeren]

Post by Aneurin Price

Post by Aneurin Price
(Actually, I might see what zfsonlinux does, since that should be
fairly quick and painless. I'll report back if it does anything
different.)

And it works! All the snapshots are listed by ZoL. I wonder if it's
just because it uses a newer version of ZFS (I haven't upgraded the
pool since I want to be able to go back to zfs-fuse).
And with this puzzle piece now I really am going to forget about it
until after Christmas.

Thanks for the other angle on this issue. I might have a a look at that
some of these days :)

[hornowitz]

Hello guys,

first of all thanks to Darik for his auto-snapshot work! It works quite
nice on my home zol servers.
But I also have some old boxes around, that still run zfs-fuse (never touch
a running... etc)...
So I thougt I will try and use dariks zfs-auto-snapshot script there as
well and noticed the same problem as decribed above.
After some debugging I've found out, that it works perfectly if you removethe com.sun:auto-snapshot-desc='-' at the snapshot taking:

Original:
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r
'***@zfs-auto-snap_daily-2012-10-29-0947'

Modified:
zfs snapshot -r '***@zfs-auto-snap_daily-2012-10-29-0947'

Sounds like a bug to me, that zfs-fuse doesn't know how to handle the
com.sun:auto-snapshot-desc='-' when the snapshot is taken.

I modified darik's script now and it looks like it is working. I am still
testing.

Thanks all for your great work. I love having zfs on linux! (fuse or
native...)

Bye,
Dominic

Post by sgheeren

Post by Aneurin Price

Post by Aneurin Price
(Actually, I might see what zfsonlinux does, since that should be
fairly quick and painless. I'll report back if it does anything
different.)

And it works! All the snapshots are listed by ZoL. I wonder if it's
just because it uses a newer version of ZFS (I haven't upgraded the
pool since I want to be able to go back to zfs-fuse).
And with this puzzle piece now I really am going to forget about it
until after Christmas.

Thanks for the other angle on this issue. I might have a a look at that
some of these days :)

[Darik Horn]

Post by Aneurin Price
Um, I already am - look at the examples I pasted in :P
(And apologies for the wrapping BTW; damn Gmail)

BTW, if the stuff that you're pasting here is being stripped or
mangled, then please post it as an attachment, or put it into a github
issue here:

https://github.com/dajhorn/zfs-auto-snapshot/issues/new

[Seth Heeren]

Post by Aneurin Price
Can anyone tell me what's happening?
Thanks,
Nye

No this behaviour isn't known to me; I have used snapshots very
intensively (with my own 5-minute auto-snapshot service on linux) for
a long time, though admittedly not since upgrading to 0.7.0. Never had
any trouble with that (though, unrelated: the reason I stopped doing
the many many snapshots was that sending/receiving incremental
datasets was getting extremely slow on the receiving side; creating
many - empty - snapshots on the receiving end takes several seconds
_per dataset_ _per snapshot_ regardless of whether it contains any
data. Annoying when you are replaying them. Luckily, the performance
is as pathetic on Solaris).

You can see what really happens in terms of snapshots and possibly
temp snapshot holds (just a suspicion there) by doing

zpool history tank

to see the holds being acquired/released:

zpool history -i tank

This should also clearly tell you whether you are piling up those
snapshots that should have been expired. Can you analyse the history
to see whether you can get any more information from that?

I must admit, that it does remind me of the infamous put_nvlist bug we
had in 0.6.0 (which failed to grow the client buffer large enough when
receiving large server replies from zfs-fuse). Something like that
could be happening here too. I'll have a look if you can't find
anything helpful from the zpool history.

Regards
Seth