Hobby projects → When the NAS boots second

Guide · Proxmox · Docker · systemd · September 2026

When the NAS boots second: a mount guard for Docker

A power cut, a reboot, and the storage box comes up a minute after the services that depend on it. The NFS mounts fail once and stay failed; Docker helpfully creates empty directories where the shares should be; the photo server refuses to start, the file server exits, and the media server scans an empty library and starts forgetting things. This is the four-part fix we run on our AI appliance, generalised so it drops onto any Docker host that mounts from a NAS.

The pattern, before the code

The root problem is timing, and timing is never fixed by one change. Four ideas together make the host indifferent to when the NAS shows up:

Give the NAS a head start. If the storage server is a VM on the same hypervisor, tell the hypervisor to start it first and to wait before starting anything else. A NAS has no guest agent to say “ready”, so the wait is a fixed number of seconds — long enough for the array to come up.

Retry instead of failing once. A network mount in fstab is attempted at boot and never again. Mark it nofail so a slow NAS cannot degrade the boot, then let a minutely timer keep trying until it succeeds.

Never start on a bare mountpoint. The dangerous moment is not the failed mount, it is the container that starts anyway. A marker file that exists only on the NAS, bound into the container as a file Compose is forbidden to create, turns “the share is missing” into “the container will not start” — which is exactly right.

Say so when it happens. When the guard does remount, it restarts the consumers and posts one line to your notifier. A silent recovery is a bug you never learn about.

Respect the application’s own guard. Immich writes a hidden .immich marker into each of its folders and refuses to start if one is missing — that refusal is the best thing that happened in our incident, because it stopped the server from writing uploads onto the wrong disk. Never set IMMICH_IGNORE_MOUNT_CHECK_ERRORS to make the message go away. Fix the mount.

What you need

A Docker host that mounts shares from a NAS. Ours is a CachyOS VM; anything with systemd and Docker Compose v2 works. The examples use three shares: photos, media and files, mounted at /mnt/<share> from a NAS at 192.168.1.10.
Optionally, Proxmox with both machines as guests. Step 1 only applies if the NAS is a VM on the same host. Skip it for a physical NAS — steps 2 to 4 carry the whole load on their own.
A notifier. We use a tiny Telegram script; swap the one line for whatever you have.

Step 1 — a head start in the hypervisor

Proxmox starts guests in startup order and, with up=, waits that many seconds before moving to the next one. Give the NAS VM a low order and a real delay; the Docker VM comes later.

On the Proxmox host (NAS is VM 109, Docker host is VM 104)

qm set 109 --onboot 1 --startup order=2,up=180
qm set 104 --onboot 1 --startup order=5

# see the whole boot sequence at a glance
for c in /etc/pve/qemu-server/*.conf; do
  printf "%-10s %s\n" "$(basename $c)" "$(grep -E '^(name|onboot|startup):' $c | tr '\n' ' ')"
done

The delay is a guess about how long the array takes to come up, not a promise. That is why it is only step one. Also make sure the NAS itself starts its array automatically — on Unraid that is Settings → Disk Settings → Enable auto start, and it needs a paid licence: a trial key waits for a human.

Step 2 — mounts that do not give up

nofail stops a missing share from degrading the boot; x-systemd.mount-timeout stops it from delaying the boot either. The retry lives in step 3, not here — fstab has no retry.

/etc/fstab on the Docker host

192.168.1.10:/mnt/user/photos /mnt/photos nfs rw,soft,timeo=150,_netdev,nofail,x-systemd.mount-timeout=30 0 0
192.168.1.10:/mnt/user/media  /mnt/media  nfs rw,soft,timeo=150,_netdev,nofail,x-systemd.mount-timeout=30 0 0
192.168.1.10:/mnt/user/files  /mnt/files  nfs rw,soft,timeo=150,_netdev,nofail,x-systemd.mount-timeout=30 0 0

soft or hard? These are demonstration shares behind applications that verify their own writes, so soft with a bounded timeo is the honest choice — a stalled request errors instead of hanging the container forever. For a share that receives irreplaceable uploads with no application-level check, prefer hard and accept that a dead NAS will freeze the writer.

Step 3 — the minutely guard

A root timer that remounts anything missing, clears the empty directories Docker created while the NAS was away, restarts the consumers only if it actually remounted something, and says so once.

/usr/local/sbin/nfs-guard.sh

#!/bin/bash
# Remount lost NAS shares; restart their consumers only when a remount actually
# happened; say so once. Runs as root, every minute.
changed=""
for m in photos media files; do
  if ! mountpoint -q /mnt/$m; then
    # An unmounted mountpoint may hold directories Docker auto-created while the
    # NAS was away. They would be hidden under the mount later — clear them so the
    # state stays honest and nothing ever lands on the local disk by mistake.
    find /mnt/$m -mindepth 1 -maxdepth 1 -exec rm -rf {} + 2>/dev/null
    mount /mnt/$m 2>/dev/null && changed="$changed $m"
  fi
done
if [ -n "$changed" ]; then
  docker restart immich_server jellyfin >/dev/null 2>&1
  docker start opencloud >/dev/null 2>&1
  # swap for your notifier of choice
  sudo -u duncan -H /home/duncan/scripts/tg-send.sh \
    "🔁 nfs-guard remounted:$changed — consumers restarted (the NAS had been away)." \
    >/dev/null 2>&1 || true
  logger -t nfs-guard "remounted:$changed; consumers restarted"
fi
exit 0

/etc/systemd/system/nfs-guard.service and nfs-guard.timer

# nfs-guard.service
[Unit]
Description=Remount NAS shares and restart their consumers if they were lost

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/nfs-guard.sh

# nfs-guard.timer
[Unit]
Description=Minutely NFS mount guard

[Timer]
OnBootSec=30s
OnUnitActiveSec=1min
AccuracySec=10s

[Install]
WantedBy=timers.target

Install and arm

sudo chmod 755 /usr/local/sbin/nfs-guard.sh
sudo systemctl daemon-reload
sudo systemctl enable --now nfs-guard.timer
systemctl list-timers nfs-guard.timer --no-legend

Why the restart list is explicit. A bind mount is private to the container that holds it: once the share is remounted on the host, a running container still sees the empty directory it started with. Restarting is the only way to hand it the real share. Listing the consumers by name keeps the guard from touching anything that does not depend on the NAS — in our case the whole voice pipeline runs on the same box and must never be restarted by a storage hiccup.

Step 4 — the marker file no container can start without

Create an empty marker on each share (it lives on the NAS, so it vanishes with the mount), then bind it into every consumer as a file with create_host_path: false. Compose refuses to invent the missing file, so the container cannot start until the share is back — and the guard’s restart brings it up the moment it is.

Once, with the shares mounted

for m in photos media files; do sudo touch /mnt/$m/.on-nas; done

compose.yaml — each service that reads from a share (Jellyfin shown)

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    volumes:
      - ./config:/config
      - ./cache:/cache
      - /mnt/media:/media:ro
      # the gate: a file that exists only on the NAS, and that Compose may not create
      - type: bind
        source: /mnt/media/.on-nas
        target: /on-nas
        read_only: true
        bind:
          create_host_path: false
    restart: unless-stopped

Immich ships its own docker-compose.yml that you should not edit; put the same block in a docker-compose.override.yml under immich-server: — Compose merges the volume lists.

Verify

Prove each layer separately, then pull the plug on purpose.

On the Docker host

# exports appear only once the NAS array is up
showmount -e 192.168.1.10

# mounts, not directory listings — an empty local directory lists just fine
for m in photos media files; do mountpoint -q /mnt/$m && echo "/mnt/$m mounted" || echo "/mnt/$m MISSING"; done

# the gate is in place when every consumer shows the marker bind
for c in jellyfin opencloud immich_server; do
  docker inspect -f '{{.Name}}: {{range .Mounts}}{{if eq .Destination "/on-nas"}}{{.Source}}{{end}}{{end}}' $c
done

# the real test: stop the NAS array, watch the consumers refuse, start it, watch the guard
journalctl -t nfs-guard -f

Notes from the incident that paid for this

“ls” is not proof of a mount. An empty local directory lists perfectly well. mountpoint -q, or a marker file that only the NAS can supply, is the test.
A failed mount is quiet; a container on the wrong disk is loud, later. The media server does not complain about an empty library — it scans it and starts removing items. The gate in step 4 is what keeps a storage blip from becoming a library rebuild.
The guard restarts only what it remounted. Housemates that share the box but not the NAS — here, an entire voice assistant — never notice.
All four steps, not the best one. The head start covers the normal reboot, the retry covers a slow array, the gate covers everything else, and the message is how you find out the first three were needed.