Immich Disaster Recover Guide for TrueNas Scale – “Common Mistakes and How to Avoid Them”

So while i was working on the article for the Immich Storage Layout migration, i accidently deleted the wrong instance of Immich …. nice .. NOT !

Prerequisites

As per Community Guidelines Backup and Restore | Immich you should have backed up the following items:

  1. Database Dump
  2. Backup of
    • UPLOAD_LOCATION/library
    • UPLOAD_LOCATION/upload
    • UPLOAD_LOCATION/profile
  3. Database Password

How to dump the database?

By default, Immich will keep the last 14 database dumps and create a new dump every day at 2:00 AM.

You are able to trigger a database dump in the admin job status page. Visit the page, open the “Create job” modal from the top right, select “Create Database Dump” and click “Confirm”. A job will run and trigger a dump, you can verify this worked correctly by checking the logs or the backups/ folder. This dumps will count towards the last X dumps that will be kept based on your settings.

Backup your Assets

Immich stores two types of content in the filesystem: (a) original, unmodified assets (photos and videos), and (b) generated content. Its recommend backing up the entire contents of UPLOAD_LOCATION, but only the original content is critical, which is stored in the following folders:

  1. UPLOAD_LOCATION/library
  2. UPLOAD_LOCATION/upload
  3. UPLOAD_LOCATION/profile

If you choose to back up only those folders, you will need to rerun the transcoding and thumbnail generation jobs for all assets after you restore from a backup.

I myself rsync to another vdev + another nas + aws cloud

The Rsync Task on the TrueNas Gui wont allow to configure local2local rsync so i’ve opened a Feature Request here: Rsync Task should allow local 2 local synchronisation ( Local Destination Target ) – Feature Requests – TrueNAS Community Forums

Instead i’ve created a Cron Job like this for each asset:

As an alternative you can also setup a cloud sync task Cloud Sync Tasks | TrueNAS Documentation Hub

The Disaster Recovery

1.) Create the Datasets / Folder Structure for your new Immich Instance

Note: If you want to use your old / original Datasets,
just wipe of all the data. You may peform a snapshot before.

2.) Copy the backed-up assets into the corresponding folders

3.) Set up a new Immich Instance of the App

Note: Do not log into the App prior the restore!

4.) Postgres Database Restore

a.) Figure out the Container ID for Postgres and copy the Databasedump into a location the postgres container has access to

b.) Figure out the external path ( Source ) and internal mount path within the container ( Destination )

c.) Copy the SQL Dump to the external Path, gunzip and adjust the permissons

d.) Restore SQL Dump via psql

6.) Re-Start the Immich App

Potential Issues during restore and how to avoid them

1.) PSQL Error: File not found

The namespace /mnt/ssd-pool/immich/pgData/ is correct for the initial data transfer, but the mount path inside the container is needed here. See here

2.) Server Exited due to failed authentication

microservices worker error: PostgresError: password authentication failed for user "immich", stack: PostgresError: password authentication failed for user "immich

On the TrueNas Apps Page select your Immich Instance, then under Workloads click on the Shell prompt icon. You can then use immich-admin as needed:

3.) SQL Restore Error

NOTE: Note that for the database restore to proceed properly, it requires a completely fresh install (i.e. the Immich server has never run since creating the Docker containers). If the Immich app has run, Postgres conflicts may be encountered upon database restoration (relation already exists, violated foreign key constraints, multiple primary keys, etc.), in which case you need to delete the DB_DATA_LOCATION folder to reset the database.

4.) ENOENT: no such file or directory, open ‘upload/encoded-video/.immich’

File does not exist - user restored from a backup, but did not restore each folder (user should restore all folders or manually create .immich in any missing folders) See: Immich System Integrity | Folder checks