Proxmox Backup and Node Health Alerts on Your Phone

Proxmox does tell you when a backup fails. It sends an email, through a local mail transport that was never configured, to root@pam, an address that does not receive mail. The notification exists. It just goes nowhere.

Most people discover this the way everyone discovers backup problems: they need a backup, and the most recent one is from March. This takes about fifteen minutes to fix and covers the three things actually worth knowing about a node.

Step 1: A notify helper on the node

Create an application in TheNotificationApp, call it Proxmox, and copy the app_key. On the node, save this as /usr/local/bin/tna-notify:

#!/usr/bin/env bash
# tna-notify "Title" "Body"
APP_KEY="your_app_key_here"
HOST="$(hostname -s)"

curl -fsS --max-time 10 -X POST https://thenotification.app/api/sendNotification \
  -H "app_key: $APP_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg t "$1" --arg b "[$HOST] $2" '{title: $t, body: $b}')" \
  >/dev/null || logger -t tna-notify "notification failed"

chmod +x /usr/local/bin/tna-notify
apt install -y jq          # not installed on a stock Proxmox node
tna-notify "Proxmox test" "If this arrives, the helper works."

Building the JSON with jq rather than string interpolation matters here, because Proxmox error messages contain quotes and colons and will happily break a hand-built payload on exactly the day you need the alert.

The hostname in the body is not decoration. The moment you have two nodes, a notification that does not say which one is a notification that makes you go and look.

Step 2: Hook the backups

vzdump supports a hook script that runs at defined phases of a backup job. Save this as /usr/local/bin/vzdump-hook.sh:

#!/usr/bin/env bash
# vzdump hook: $1 = phase, environment carries the rest
PHASE="$1"

case "$PHASE" in
  backup-end)
    logger -t vzdump-hook "backup ok for VMID $VMID"
    ;;
  backup-abort)
    tna-notify "Backup FAILED: VMID $VMID" "${LOGFILE:-no log}. Job aborted."
    ;;
  job-end)
    tna-notify "Proxmox backup job finished" "Store: ${DUMPDIR:-unknown}"
    ;;
  job-abort)
    tna-notify "Proxmox backup job ABORTED" "Check the task log on this node."
    ;;
esac

exit 0

chmod +x /usr/local/bin/vzdump-hook.sh

Then point vzdump at it, in /etc/vzdump.conf:

script: /usr/local/bin/vzdump-hook.sh

Two things worth knowing. The hook must exit 0: a non-zero exit from the hook can abort the backup, which is a spectacular way to make your monitoring cause the outage. And notice there is no notification on backup-end, only a log line. A nightly job over twelve VMs would otherwise send twelve success notifications every morning, and you would mute the whole thing by Thursday.

Step 3: Catch the failure before it happens

Most backup failures are the same failure: the storage filled up. That is knowable hours in advance, and a warning at 80% is worth more than an alert at 100%.

#!/usr/bin/env bash
# /usr/local/bin/check-pve-storage.sh
THRESHOLD=80
STATE=/var/tmp/pve-storage-warned

pvesm status --noborder --noheader | while read -r name type status total used free pct; do
  [ "$status" = "active" ] || continue
  used_pct="${pct%\%}"
  used_pct="${used_pct%.*}"
  [ -z "$used_pct" ] && continue

  if [ "$used_pct" -ge "$THRESHOLD" ]; then
    grep -q "^$name$" "$STATE" 2>/dev/null && continue
    echo "$name" >> "$STATE"
    tna-notify "Storage $name at ${used_pct}%" "Backups will start failing soon."
  else
    sed -i "/^$name$/d" "$STATE" 2>/dev/null
  fi
done

The state file is what makes this usable. Without it, a storage sitting at 85% notifies you every single time cron runs, which at hourly is twenty-four notifications a day about a thing you already know. The file records that you have been told, and clears itself when the storage drops back under the line, so you get told again next time it matters.

chmod +x /usr/local/bin/check-pve-storage.sh
# hourly, via /etc/cron.d/pve-storage
0 * * * * root /usr/local/bin/check-pve-storage.sh

Step 4: Know when the node reboots

An unexplained reboot is the single most useful signal a hypervisor produces, because it means either the hardware is unhappy or something rebooted it without asking you.

# /etc/systemd/system/tna-boot.service
[Unit]
Description=Notify on boot
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/tna-notify "Proxmox node booted" "Uptime reset. If you did not do this, look at the logs."

[Install]
WantedBy=multi-user.target

systemctl enable tna-boot.service

The After=network-online.target is load-bearing. Without it the service runs before networking is up, the curl fails, and you get silence at precisely the moment you wanted a notification.

The honest part

These four together are a low-traffic setup by design: a failed backup, a storage warning, a reboot. In a healthy month that is zero notifications, which is exactly right and means the free tier of 100 for the lifetime of the account lasts a long time.

The thing that will burn it is the storage check without the state file, or notifying on every successful backup. Both are tempting and both are wrong. Pro is $2.99 a month for 1,000 if you run several nodes.

Worth saying plainly: this tells you a backup failed, it does not tell you a backup is restorable. Those are different problems and only a test restore solves the second one.

Where this fits

The same helper works for anything else on the node. If you run containers on top of Proxmox, the crash-alert version is in Docker container alerts, and the request shape is in the API reference.

Grab a key at thenotification.app and stop trusting an email that goes nowhere.

New to this? Start with the server-down alert this is built on.