I have a load-bearing raspberry pi on my network - it runs a DNS server, zigbee2mqtt, unifi controller, and a restic rest server. This raspberry pi, as is tradition, boots from a microSD card. As we all know, microSD cards suck a little bit and die pretty often; I’ve personally had this happen not all that long ago.

I’d like to keep a reasonably up-to-date hot spare ready, so when it does give up the ghost I can just swap them out and move on with my life. I can think of a few ways to accomplish this, but I’m not really sure what’s the best:

  • The simplest is probably cron + dd, but I’m worried about filesystem corruption from imaging a running system and could this also wear out the spare card?
  • recreate partition structure, create an fstab with new UUIDs, rsync everything else. Backups are incremental and we won’t get filesystem corruption, but we still aren’t taking a point-in-time backup which means data files could be inconsistent with each other. (honestly unlikely with the services I’m running.)
  • Migrate to BTRFS or ZFS, send/receive snapshots. This would be annoying to set up because I’d need to switch the rpi’s filesystem, but once done I think this might be the best option? We get incremental updates, point-in-time backups, and even rollback on the original card if I want it.

I’m thinking out loud a little bit here, but do y’all have any thoughts? I think I’m leaning towards ZFS or BTRFS.

  • Avid AmoebaEnglish
    2•3 months ago

    Perhaps the best answer by far is ZFS but I don’t know how much pain it is to set it up to boot from on a Pi. The easiest to setup is probably LVM.

    With ZFS you can trivially keep a hot spare even over the network. Just tell syncoid where to replicate.

      • Avid AmoebaEnglish
        3•3 months ago

        I used it on a Pi 4 in 2019 for an USB-connected mirror and it worked well. Unencrypted throughput was upwards from 200MB/s. Encrypted throughput dropped down to under 100MB/s due to insufficient compute. The Pi 4 is a powerful computer and the Pi 5 even more so. Pi 3 and older, not so much.

          • Possibly linuxEnglish
            4•3 months ago

            You could boot from from an SD and then store data on a external drive. I would go Btrfs over ZFS unless you have at least 3 or more disks

          • Avid AmoebaEnglish
            1•3 months ago

            I would try it. My only issue is I have no idea how to set it up on root on a Pi. Perhaps there’s docs somewhere. If had to setup a new Pi with Pi OS/Debian/Ubuntu today I’d definitely try it. Most of my Pis are running OpenWrt though.

        • Possibly linuxEnglish
          1•3 months ago

          You say that but when the system starts to lock up you might change your tune

          • Avid AmoebaEnglish
            1•3 months ago

            Why would it lock up? ZFS will use as much RAM as you give it and it doesn’t seem CPU-bound unless you turn on encryption. It’s not a cluster FS like Ceph. Why do you expect ZFS to lock up and Btrfs not to?