It’s fairly obvious why stopping a service while backing it up makes sense. Imagine backing up Immich while it’s running. You start the backup, db is backed up, now image assets are being copied. That could take an hour. While the assets are being backed up, a new image is uploaded. The live database knows about it but the one you’ve backed up doesn’t. Then your backup process reaches the new image asset and it copies it. If you restore this backup, Immich will contain an asset that isn’t known by the database. In order to avoid scenarios like this, you’d stop Immich while the backup is running.

Now consider a system that can do instant snapshots like ZFS or LVM. Immich is running, you stop it, take a snapshot, then restart it. Then you backup Immich from the snapshot while Immich is running. This should reduce the downtime needed to the time it takes to do the snapshot. The state of Immich data in the snapshot should be equivalent to backing up a stopped Immich instance.

Now consider a case like above without stopping Immich while taking the snapshot. In theory the data you’re backing up should represent the complete state of Immich at a point in time eliminating the possibility of divergent data between databases and assets. It would however represent the state of a live Immich instance. E.g. lock files, etc. Wouldn’t restoring from such a backup be equivalent to kill -9 or pulling the cable and restarting the service? If a service can recover from a cable pull, is it reasonable to consider it should recover from restoring from a snapshot taken while live? If so, is there much point to stopping services during snapshots?

  • Avid AmoebaOP
    link
    fedilink
    English
    2
    edit-2
    4 months ago

    Thanks for validating my reasoning. And yeah, this isn’t Immich-specific, it would be valid for any process and its data.

    • @[email protected]
      link
      fedilink
      English
      24 months ago

      What i have seen for corporate server is when backup is started the database goes into a different mode, and a temp writable partition is used while readonly database is backed up, at end of backup that blob created is also stored.

      • Avid AmoebaOP
        link
        fedilink
        English
        1
        edit-2
        4 months ago

        Yeah if you’re making a backup using the database system itself, then it would make sense for it do something like that if it stays live while backing up. If you think about it, it’s kinda similar to taking a snapshot of the volume where an app’s data files are while it still runs. It keeps writing as normally while you copy the data from the snapshot, which is read-only. Of course there’s no built-in way to get the newly written data without stopping the process. But you could get the downtime to a small number. 😄

        • @gedhrel
          link
          English
          24 months ago

          The other thing to watch out for is if you’re splitting state between volumes, but i think you’ve already ruled that out.

          • Avid AmoebaOP
            link
            fedilink
            English
            1
            edit-2
            4 months ago

            Oh yeah, that would be a disaster. If not handled correctly.