How it works
The internal breakdown of a ProxMenux backup — filesystem, manifest and application inventory — and how the restore consumes all three to reproduce the source host on a target that may not share the same kernel.
One archive, three payloads
.tar.zst, PBS backup or Borg archive) contains this exact layout. The restore reads the three payloads independently, in a specific order that guarantees correctness: rootfs is copied first to lay down configuration, the manifest is consulted to detect drift and decide what to skip, and the application inventory drives the post-boot reinstall pass. No cross-payload dependencies — each one can be inspected or extracted independently.Archive layout
Every archive follows the same tree layout regardless of destination. The metadata/ subdirectory holds the structured payloads; the rootfs/ subdirectory holds the filesystem copy.
backup-[timestamp]/
├── manifest.json # structured host state
├── metadata/
│ ├── packages.manual.list # apt-mark showmanual
│ ├── run_info.env # backup run identity + kernel version
│ ├── paths_archived.txt # exact list of paths that reached rootfs/
│ └── missing_paths.txt # paths from the profile absent on source
└── rootfs/
├── etc/ # /etc/pve, /etc/network, /etc/ssh, …
├── root/ # /root (with volatile subdirs excluded)
├── usr/local/ # /usr/local/bin, /usr/local/share/proxmenux, …
└── var/ # /var/lib/pve-cluster, /var/spool/cron/…The rootfs payload
The rootfs/ tree is a plain filesystem copy produced by rsync from the source host. It contains a curated default profile of paths that matter for a Proxmox restore, plus any custom paths added by the operator to the backup job or interactive session. The set is deliberately narrow: only paths that either hold configuration or hold state that Proxmox cannot regenerate on its own.
The default profile
The default profile is defined by hb_default_profile_paths in lib_host_backup_common.sh. It covers eight categories that together describe a working Proxmox host:
| Path categories | Paths | Why |
|---|---|---|
| PVE core | /etc/pve, /var/lib/pve-cluster, /etc/vzdump.conf | Cluster filesystem contents, cluster live data, vzdump defaults. |
| Host identity & network | /etc/hostname, /etc/hosts, /etc/timezone, /etc/resolv.conf, /etc/network | Everything the host needs to come up on the network with the same identity. |
| Access & auth | /etc/ssh, /etc/sudoers, /etc/sudoers.d, /etc/pam.d, /etc/security | SSH keys, sudo rules and PAM configuration. Losing these locks the operator out of the restored host. |
| Kernel & boot | /etc/default/grub, /etc/kernel, /etc/modules, /etc/modules-load.d, /etc/modprobe.d, /etc/sysctl.conf, /etc/sysctl.d, /etc/udev/rules.d, /etc/fstab, /etc/iscsi, /etc/multipath | IOMMU tokens, module blacklists, VFIO device IDs, mount table, storage stack config. |
| Shell & locale | /etc/environment, /etc/bash.bashrc, /etc/inputrc, /etc/profile, /etc/profile.d, /etc/locale.gen, /etc/locale.conf | System-wide shell setup, environment variables, locale generation configuration. |
| Packaging & cron | /etc/apt, /etc/cron.d, /etc/cron.{daily,hourly,weekly,monthly}, /etc/cron.allow, /etc/cron.deny, /var/spool/cron/crontabs | APT sources for consistent package resolution, scheduled tasks defined by the operator. |
| ProxMenux state & tools | /etc/proxmenux, /etc/systemd/system, /etc/log2ram.conf, /etc/logrotate.conf, /etc/logrotate.d, /etc/lm-sensors, /etc/sensors3.conf, /etc/fail2ban, /etc/snmp, /etc/postfix, /etc/wireguard, /etc/openvpn, /etc/grafana, /etc/influxdb, /etc/prometheus, /etc/telegraf, /etc/zabbix | Optional but common Proxmox tooling. Missing paths are noted in metadata/missing_paths.txt without stopping the backup. |
| ProxMenux binaries & root | /usr/local/bin, /usr/local/sbin, /usr/local/share/proxmenux, /root (volatile subdirs excluded) | ProxMenux-installed binaries and per-user configuration under /root. Volatile paths (.bash_history, .cache/, tmp/, .local/share/Trash/) are excluded from the copy. |
| ZFS state (conditional) | /etc/zfs | Only included when the source host runs ZFS. Contains zpool.cache and hostid. |
Extending the profile with custom paths
Beyond the default profile, ProxMenux offers two ways to include additional paths in a backup. They compose without conflict and both apply to interactive backups and scheduled jobs.
1. Persistent extras (per-host file)
A text file at /usr/local/share/proxmenux/backup-extra-paths.txt holds a list of absolute paths the operator has marked as "always include" on this host. When a backup runs in Default mode, ProxMenux appends these paths to the default profile automatically without asking. The file is edited from the interface — no manual editing needed — and persists across reboots and updates. One absolute path per line; # comments are allowed.
2. Custom mode (per-run)
Launching a backup in Custom mode replaces the automatic application of the default profile with a checklist showing every path: the default-profile entries and the persistent extras (prefixed with [+] and pre-checked). The operator ticks or unticks entries for that specific run, and can also press Add custom path to enter a new path — which is then persisted to the extras file for future backups.
Paths absent from the source
Any path from the profile (default or added) that does not exist on the source host is recorded in metadata/missing_paths.txt inside the archive. The backup neither fails nor stops — the operator sees a summary of archived paths and missing paths at the end. In practice this happens for optional tooling paths like /etc/wireguard or /etc/prometheus when those tools are not installed.
The manifest payload
manifest.json is a structured JSON document that describes the source host at the moment of the backup. It is produced by six independent collectors orchestrated by build_manifest.sh. Each collector is read-only, produces a well-defined JSON fragment, and falls back to a safe empty default if it fails — the manifest is still usable if one section is incomplete.
| Collector | Produces | Content |
|---|---|---|
| collect_source_host.sh | source_host | Hostname, PVE version (pveversion), PBS version if the host runs the backup-server role, kernel (uname -r), boot mode (efi/bios), root filesystem type, CPU model + architecture, memory in KB. |
| collect_hardware.sh | hardware_inventory | GPUs (with vendor and ProxMenux installer mapping), TPUs (Coral USB + M.2 with lsusb/lspci detection), NICs (with MAC, PCI slot and bridge membership), wireless devices. GPU entries carry a passthrough_eligible heuristic. |
| collect_storage.sh | storage_inventory | ZFS pools (with pool type + member disks resolved to /dev/disk/by-id/* for portability), LVM volume groups + thin pools, physical disks with SMART capability, PVE storage.cfg entries, external mounts. |
| collect_kernel.sh | kernel_params | Operator-authored tokens from /proc/cmdline (stripped of boilerplate like BOOT_IMAGE=, root=, ro/rw, quiet, splash), modules loaded at boot from /etc/modules, and paths of /etc/modprobe.d/*.conf files with actual directives (options, blacklist, install, alias, softdep). |
| collect_proxmenux_state.sh | proxmenux_installed_components | Reads /usr/local/share/proxmenux/managed_installs.json (registry of everything ProxMenux has installed) and installed_tools.json. Each entry keeps the installer path (menu_script) so the restore can trigger the exact same install flow. |
| collect_guests.sh | vms_lxcs_at_backup | Enumerates VMs (qm list) and LXCs (pct list) present at backup time — VMID, name, current status. Only the inventory: the actual guest data is the responsibility of vzdump / PBS. |
The six collectors compose the manifest. Each collector runs in its own subprocess; a failure in one falls back to a documented safe default and warns but does not abort the backup.
Schema validation
The manifest validates against scripts/backup_restore/schema/manifest.schema.json. Running build_manifest.sh --validate triggers a Python-side JSON Schema validation (requires python3 + jsonschema). If the module is not present, the check is skipped silently — validation is primarily a developer aid, not a runtime dependency.
The application inventory
Two files under metadata/ catalogue everything installed on the source that is not part of the base Proxmox VE package set. The restore uses them to reproduce the exact set of user-installed software on the target, using either APT or ProxMenux's own installers depending on how the software was originally installed.
packages.manual.list
A plain-text list produced by apt-mark showmanual: every APT package that was explicitly installed on the source host, sorted alphabetically. This excludes packages installed as dependencies of the base Proxmox VE ISO (they are re-pulled automatically by the target's own APT). Read by _rs_run_complete_extras during restore, filtered through a three-pass cascade-safe filter (dpkg -s for already-installed, sibling-major detection for library packages, apt-get install --simulate for cascade-remove risk), then installed with apt-get install -y.
components_status.json (part of the rootfs)
A JSON registry under /usr/local/share/proxmenux/ that records every component ProxMenux has installed with its exact state: version, ProxMenux-specific flags (for NVIDIA: patched boolean; for Coral: DKMS version). This file lives inside the rootfs — not in metadata/ — because it is read after the rootfs has been copied to the target. The post-boot dispatcher (apply_cluster_postboot.sh) iterates over its entries and runs each component's --auto-reinstall hook, which reads the recorded state and reproduces the install against the target's current kernel.
Component installers
Four ProxMenux installers currently expose a --auto-reinstall entry point:
| Component | Installer | Action on --auto-reinstall |
|---|---|---|
| nvidia_driver | gpu_tpu/nvidia_installer.sh | Reads version + patched. Downloads the exact same NVIDIA runfile, builds DKMS modules against the target's kernel, re-applies the ProxMenux patch if the source had it. |
| coral_driver | gpu_tpu/install_coral.sh | Reads the Coral driver version. Compiles the DKMS module against the target's kernel. |
| amdgpu_top | gpu_tpu/amd_gpu_tools.sh | Reads the recorded version. Re-downloads the exact .deb from the GitHub release. |
| intel_gpu_tools | gpu_tpu/intel_gpu_tools.sh | APT-installs the package. Idempotent if already present from packages.manual.list. |
How the restore consumes the three payloads
The restore is a five-stage pipeline. Each stage reads a specific subset of the archive and updates the target host. No stage requires the source host to be reachable — the archive is fully self-contained.
| # | Stage | Reads | Action |
|---|---|---|---|
| 1 | Compatibility check | manifest.json | Runs hb_compat_check. Compares source vs. target hardware (NICs, storage IDs), PVE version, and major kernel version. Sets HB_COMPAT_KERNEL_DIRECTION (same, bk_newer or bk_older) and populates RS_SKIP_PATHS with hardware-drift and cross-kernel exclusions. |
| 2 | Hot apply | rootfs/ (safe paths only) | _rs_apply … hot copies hb_classify_path=hot entries directly to the live target. Anything under /etc/pve, /etc/network or paths classified as reboot/dangerous is deferred. |
| 3 | Pending prepare | rootfs/ (reboot + dangerous paths) | _rs_prepare_pending_restore stages risky paths under /var/lib/proxmenux/pending-restore/, writes plan.env, apply-on-boot.list and rs-skip-paths.txt, and enables proxmenux-restore-onboot.service to fire on the next boot. |
| 4 | Package install | metadata/packages.manual.list | _rs_run_complete_extras runs the cascade-safe filter and calls apt-get install -y with the surviving package list. Full output goes to /var/log/proxmenux/restore-apt-*.log. |
| 5 | Post-boot | rootfs (already applied) + components_status.json | After reboot, apply_pending_restore.sh plays back the deferred paths and apply_cluster_postboot.sh runs update-initramfs, update-grub (or proxmox-boot-tool refresh) and iterates over components_status.json firing each component's --auto-reinstall hook. |
Stage 1 uses the manifest to decide what to touch. Stage 2 copies rootfs paths that are safe to apply on a running system. Stage 3 stages the risky paths for the next boot. Stage 4 handles packages. Stage 5 runs after the reboot and reinstalls components against the target's kernel.
Why the three-payload split is the one used
The split is not a filesystem decision — it is a lifecycle decision. Filesystem content moves with rsync: fast, transparent, atomic per file. Configuration state that a restore has to interpret before touching the target moves as structured JSON: readable independently, versionable through a schema, machine-diffable against the target's own state. Software that has to be re-installed against the target's environment moves as an inventory: names and versions only, letting the target's package manager and ProxMenux's own installers decide the actual binaries. Each payload optimises for what it needs to do, and the three combine into a restore that is atomic in intent but fault-tolerant in practice: a corrupt manifest still leaves the rootfs restorable, a missing package still leaves the components installable, a component installer failing on one entry does not stop the next.