📖 Chapter 05 — Disaster Recovery

What to do when the worst happens. The playbooks, the practice, the recovery.

Learning Objectives

• Define the disaster scenarios for a home NAS
• Build the playbooks for each scenario
• Test the playbooks with practice runs
• Recover from a real disaster with confidence

Introduction

The 3-2-1 rule (Chapter 1) is the defense. Disaster recovery is the offense: when the worst happens, what do you do? The difference between "I have backups" and "I can recover" is the playbook: the documented steps for restoring the system, tested and ready.

This chapter covers the disaster scenarios, the playbooks, the practice. The goal: when the worst happens, you know what to do. You're not figuring it out in the moment; you're executing the playbook.

The disaster scenarios

For a home NAS, the realistic disasters:

• Single drive failure. The most common disaster. RAIDZ2 handles it (with 2-drive fault tolerance). The recovery: replace the drive, let the pool resilver.
• Two drive failures in the same vdev. The pool is degraded; data may be lost. The recovery: replace the drives, hope the data is recoverable, restore from backup if not.
• Pool loss (all drives fail, NAS hardware dies). The data is gone from the NAS. The recovery: restore from backup (the local external drive or the offsite backup).
• Ransomware attack. The data is encrypted on the NAS. The snapshots are also encrypted (if the attack had admin access). The recovery: restore from the offline backup (which wasn't accessible to the attacker).
• Theft. The NAS is stolen. The data is gone. The recovery: restore from the offsite backup.
• Fire / flood / natural disaster. The house is damaged. The NAS and local backup are both gone. The recovery: restore from the offsite backup (at a friend's house, in the cloud).

For each, the playbook is different. The common thread: the data is recoverable if the 3-2-1 backups are in place. The playbooks are how to recover.

The single-drive failure playbook

SMART alert: drive 3 is failing. The pool is degraded but functional.

1. Identify the failing drive by serial number (SMART test result + TrueNAS Storage → Disks).
2. Order a replacement drive (same model, or larger).
3. When the drive arrives, shut down the NAS.
4. Replace the drive (the failing one's serial is on the label).
5. Power on. The NAS detects the new drive.
6. Go to Storage → Manage Devices on the pool. Click Replace on the failed drive. Select the new drive.
7. ZFS starts a resilver. This reads every block from the surviving drives, computes parity, writes the new data. For a 6×12 TB pool, expect 12-24 hours.
8. While the resilver runs, the pool is degraded but functional. Don't stress it (avoid heavy reads/writes).
9. When the resilver completes, the pool is back to healthy.

This is the most common disaster. The playbook is the same every time. Practice it once (on a test pool or a scheduled drive swap) so you know the steps.

The two-drive failure playbook

Two drives fail at the same time (or close enough that the second failure happens before the first is replaced). The pool is degraded with no fault tolerance remaining.

1. Assess: are both failures in the same vdev? If yes, the pool is at risk. If they're in different vdevs (in a multi-vdev pool), the pool is degraded but functional.
2. Don't write to the pool. The remaining fault tolerance is zero; another failure could lose data.
3. Order replacement drives for both failed drives.
4. When the drives arrive, replace one at a time. Each replacement triggers a resilver.
5. If the data is corrupt after the resilvers (because both failures were in the same vdev and ZFS couldn't recover), restore from backup.

The two-drive failure is the case that tests the 3-2-1 rule. If the 3-2-1 backups are in place, the worst case is "we lost a few hours of recent changes."

The pool loss playbook

The NAS is dead (hardware failure, fire, theft, ransomware that encrypted everything). The data on the NAS is gone. The recovery: rebuild from backup.

1. Acquire a new NAS (or temporarily use any machine with sufficient storage).
2. Install TrueNAS (or another ZFS-based system) on the new machine.
3. Connect the external backup drive. Import the pool on the backup drive (or restore the files directly).
4. Recreate the dataset structure on the new pool.
5. Copy the data from the backup to the new pool. zfs send | zfs receive is the fast way; rsync or a simple copy works too.
6. Recreate the user accounts, the SMB shares, the snapshot policies, the apps.
7. Verify the data: spot-check files, check the family can access the photos, etc.

For TK's setup, the TrueNAS configuration is backed up as a small file (System → General → Save Config). With the config file, the rebuild is faster: import the config, recreate the pool, restore the data.

The ransomware playbook

The data on the NAS is encrypted by ransomware. The snapshots might also be encrypted (if the attacker had admin access). The recovery: restore from the offline backup.

1. Disconnect the NAS from the network. Stop the spread.
2. Assess: what was encrypted? What was the attack vector? How did the attacker get in?
3. Do NOT pay the ransom. It doesn't guarantee recovery, and it funds the attackers.
4. Wipe the NAS. Start with a clean install.
5. Restore from the offline backup. The offline backup was never accessible to the attacker (assuming you followed the 3-2-1 rule and the backup is truly offline).
6. Restore the most recent good state. You might lose a few hours of recent changes.
7. Patch the vulnerability that allowed the attack. Update the apps. Change the passwords.
8. Verify the system is clean. Run malware scans on the restored data.

The 3-2-1 rule is the defense against ransomware. The offline backup is the recovery. The discipline: keep the backup truly offline (not mounted, not accessible to the network).

The fire/flood playbook

The house is damaged. The NAS and the local backup are both gone. The recovery: the offsite backup.

1. Once safe, assess what you have. The offsite backup (at a friend's house, in the cloud) is the source of recovery.
2. Acquire a new NAS. Same as the pool loss playbook from this point.
3. Retrieve the offsite backup (or download from the cloud).
4. Restore.

The 3-2-1 rule is the defense. The offsite backup is the recovery. The discipline: the offsite backup is far enough away that the same disaster doesn't reach it.

Practice the playbooks

The playbooks are worthless if you haven't practiced them. The practice:

1. Once a year, simulate a disaster recovery. Pick a dataset. Wipe it from the NAS. Restore from backup. Verify the data is intact.
2. Once a year, simulate a drive failure. Pick a drive. Mark it as failed in the TrueNAS UI. Replace it with a spare. Let it resilver. Verify the pool is healthy.
3. Once a year, retrieve the offsite backup. Verify it can be connected to a different machine and the data is readable.

The practice is the difference between "I think this would work" and "I know this works." The 30 minutes of practice once a year is the price of recovery confidence.

The "TrueNAS config backup" trick

TrueNAS can save its configuration as a small file (typically 100-500 KB). The file contains: user accounts, share definitions, snapshot policies, app configurations, network settings. It does NOT contain the data (the data is on the pool).

Save the config file in two places: on the NAS (in tank/Lab/backup/) and on the offsite backup. With the config file, rebuilding the NAS is much faster: import the config, recreate the pool, restore the data.

The discipline: save the config whenever you make a significant change. Schedule a monthly export to the backup location.

The "what if I forgot the password" playbook

Forgotten passwords happen. The recovery:

• TrueNAS admin: if you have the root password (the one set during install), you can use it. If you've lost it, the only recovery is a full reinstall (the data is still there, but the users and shares need to be recreated). Store the root password in Vaultwarden.
• App passwords: reset through the app's UI, or via the database if you have direct access.
• Tailscale: recover via the Tailscale admin console. The auth key can be re-generated.

The discipline: store critical passwords in Vaultwarden. Don't rely on memory.

What NOT to do in a disaster

• Don't panic-write to the pool. If the pool is degraded, more writes make the situation worse. Let the resilver happen; don't add load.
• Don't pay the ransom. It funds the attackers and doesn't guarantee recovery.
• Don't skip the playbook. The playbook is what you follow in a crisis. Skipping steps because they "seem unnecessary" is how you make the situation worse.
• Don't restore over the existing pool. If the data is encrypted, restoring to the same pool (without wiping it first) might not work, or might re-encrypt the restored data. Wipe first, then restore.

Engineering Note

Summary

Disaster scenarios: single drive failure (most common, RAIDZ2 handles), two drive failure (rare, restores from backup), pool loss (the whole NAS is gone, restores from offsite backup), ransomware (restores from offline backup), fire/flood (restores from offsite backup). For each, a playbook. Practice once a year. Save the TrueNAS config to the backup. The discipline: rehearse the worst so the real thing is calm.

Checklist

• ⬜ Document each playbook in the runbook (Chapter 7)
• ⬜ Save the TrueNAS config to the backup monthly
• ⬜ Once a year: simulate a drive failure, verify the replacement works
• ⬜ Once a year: simulate a pool loss, verify the backup can be restored
• ⬜ Once a year: retrieve the offsite backup, verify it's readable on a different machine
• ⬜ Store critical passwords in Vaultwarden; have a backup of the Vaultwarden data

Looking Ahead

Chapter 06 is update strategy. The discipline of keeping software current without breaking things. Update cycles, the testing approach, the rollback plan. The chapter that turns "I should update" into "I have a system for updating."

Ch 05 · v0.1 · drafted from the original ChatGPT conversation, July 2026