ZFS, Proxmox, and the Art of Not Losing Data

I found this one during a long audit of the homelab, before anything had visibly broken. Every drive passed SMART. It was ZFS that said something was wrong, and on a filesystem without checksums I think I'd have found out much later, from a movie that wouldn't play.

What the pool looked like

The media pool was three drives striped together with no redundancy: a 16TB Exos, a 20TB Exos and an 8TB Barracuda. A stripe gives you all the space and none of the safety, so one bad drive is everyone's problem.

zpool status showed the Barracuda with over 11,000 checksum errors, and ZFS could tell me which files were affected. There were 59 of them. That list is the thing I'd point to if someone asks why I bother with ZFS at home. Another filesystem would have handed those bytes over without a word.

The cause turned out to be the drive type rather than a dying disk. That Barracuda is SMR, shingled recording, which handles the random writes ZFS does very badly. I tried re-downloading the broken files, and the new copies came out corrupt as well, because they were written to the same drive. That settled it.

Rebuilding it

You can't just pull a drive out of a ZFS stripe. The data is spread across all of them, so the way out was to destroy the pool and create a new one. It came back as just the two Exos drives, empty, and the media was downloaded again by the same automation that got it the first time. That backfill ran for days with the download clients going flat out, and it caused a mess of its own (written up here).

There was fallout I hadn't planned for. Jellyfin's database still pointed at the old files, and the library scans started failing on foreign key errors. I tried deleting about 10,000 orphaned rows to fix it in place, which didn't help, so in the end I wiped the database and let it rebuild the library from scratch.

The 16TB had its own issue on the way. It showed 939 CRC errors, which are errors on the link between the controller and the disk, not on the disk itself. No data was damaged. On this HP box the likely culprit is the connection to the backplane, and the fix is reseating the drive.

What changed afterwards

The 8TB Barracuda didn't go in the bin. It's formatted XFS now and does the one job SMR is fine at, taking big sequential writes. Every night at 02:00 Proxmox snapshots all the VMs onto it, and restic puts app configs and database dumps there too, with a second copy going to Backblaze B2.

The media itself isn't backed up. All of it can be downloaded again, and the rebuild proved that the hard way. The configs and databases are what took real work to get right, so they're the part with two copies.

I also stopped trusting myself to look at zpool status. A timer on my agent VM checks every six hours and stays quiet while things are fine:

bash
STATUS=$(run_ssh "$PROXMOX_HOST" "zpool status -x")

# All healthy, clear state and exit
if echo "$STATUS" | grep -q "all pools are healthy"; then
  rm -f "$STATE_FILE"
  exit 0
fi

# Problem: only alert when the details change, so one bad
# disk doesn't post the same message every six hours
DETAILS=$(run_ssh "$PROXMOX_HOST" \
  "zpool status | grep -E 'pool:|state:|errors:|DEGRADED|FAULTED|CKSUM'")

Where it is now

The rebuilt pool is 32.7T as ZFS counts it, and it's 96% full. The last scrub took 1 day and 19 hours and found nothing, which is the result I want and also a reminder that two big disks are slow to read end to end. It's still a stripe. If one of those drives dies outright the media goes with it, and the plan is the same as last time, a new pool and a long backfill.

What I'd tell anyone building a pool at home is to check whether your drives are SMR before they go in. The model number is enough to look it up.