Designing Resilient, Layered Backups

Introduction

When I first built my homelab, I focused on virtualization and network segmentation,but my backup story was an afterthought. In this post, I’ll share the end-to-end architecture I now trust: a mirrored ZFS NAS, Linux containers, and continuous cloud syncs, each layer treated like an adversary probing for weakness.

---

1. ZFS Mirroring for Local Redundancy

At the heart of my NAS sits a 1 TB mirrored ZFS pool (for the mean-time). Here’s why ZFS is my go-to:

• Copy-On-Write & Checksums
  Every block gets a checksum. If corruption ever creeps in, ZFS detects and repairs it automatically from the other mirror.

• Efficient Snapshots
  Instant, space-efficient snapshots let you roll back to any point in time without cloning entire datasets.

• Simple Expansion
  You can add new mirror vdevs or replace drives one at a time—as long as you stay in sync.

    # Partition new drives as GPT, type ‘bf’ (Solaris root)
    sgdisk -a1 -n1:0:0 -t1:bf /dev/sd{b,c}
  
    # Create mirror
    zpool create homelab mirror /dev/sdb1 /dev/sdc1 \
      -o ashift=12 -O compression=lz4 -O atime=off
  

---

2. Containers + Cloud Backup Syncing

Linux CT (on Proxmox) that:

1. Mounts the ZFS pool.

2. Runs rclone to push new snapshots off-site.

3. Uses systemd timers to schedule exports, keeping me GUI-free and minimizing attack surface.

This separation lets me treat the orchestrator CT as a “jump box,” hardened with only SSH, ZFS tools, and my sync scripts.

1. Snapshot Export

    # Export latest snapshot to a tar
    zfs snapshot homelab/data@auto-$(date +%F)
    zfs send homelab/data@auto-$(date +%F) | gzip > /backups/data-$(date +%F).zst
   

2. Sync with rclone

    rclone sync /backups remote:homelab-backups \
      --transfers 4 --checkers 8 --quiet
   

3. Immutable Retention
   On my cloud provider, I enforce a 30-day immutable retention policy, protecting archived snapshots from accidental or malicious deletion.

---

3. Integrity Checks & Alerting

• Local Verification
  After every send/receive cycle, compare checksum of source and destination datasets.

    zfs diff-checksum homelab/data@auto-2025-06-18 remote/data@auto-2025-06-18
  

---

4. Principle of Least Privilege

Treat every service as a potential foothold:

• SSH only (key-based), no password logins

• nftables firewall restricted to internal network

• Runs as non-root user

• Mounts only /backups read-only

• Resource-limits applied (CPU, memory)

By segmenting roles, I ensure that even if the sync container is compromised somehow, the attacker would have a harder time moving laterally.

---

Conclusion

By applying an adversary mindset to each layer, from ZFS mirroring and local snapshots to headless automation and immutable cloud archives, one can sleep… better.