Cross-kernel restore
How ProxMenux handles a restore when the target host runs a different kernel from the one recorded in the backup — especially when the target runs a newer kernel. Documents how the difference is detected, how boot-critical paths that could break the target are filtered, and how the user's own configuration is reapplied without copying kernel-tied files verbatim.
Every kernel jump is handled differently
6.17.13-2-pve. The first two numbers (6.17 in this example) define the major version: when they change, some configuration files written for the previous kernel may not be valid on the new one. When the restore detects that the backup and the target have different major kernel versions, the flow branches based on the direction of the jump — whether the backup is older or newer than the target — because the two cases have opposite failure modes. Backups newer than the target restore cleanly as-is (bench-verified across multiple kernel jumps). Backups older than the target need a filter to avoid breaking the target's boot with configuration written for a kernel that has since changed.The direction check
During the compatibility check, hb_compat_check compares the kernel recorded in the backup's manifest against the target's current kernel (uname -r) and classifies the restore into one of three cases, stored in the internal variable HB_COMPAT_KERNEL_DIRECTION:
| Direction | Condition | Behaviour |
|---|---|---|
| same | The major kernel version matches between backup and target. | The full restore flow runs unchanged. No extra filter, no hydration, no special notice in the UI. |
| bk_newer | Backup kernel is NEWER than the target's (for example: the backup was made on kernel 7.0 and the target runs kernel 6.17). | The full restore flow runs unchanged, exactly like same. No filter is applied. Verified empirically: drivers auto-reinstall against the target's kernel, IOMMU config applies cleanly, VMs with GPU passthrough come up without issue. |
| bk_older | Backup kernel is OLDER than the target's (for example: the backup was made on kernel 6.17 and the target runs kernel 7.0). | The safe-subset filter and the four-phase hydration below are activated. The restore proceeds — the target reproduces the source, but boot-critical files are not copied verbatim. |
Why a backup with a newer kernel than the target restores unchanged
When the restore runs against a target with an older kernel than the one recorded in the backup, every mechanism the restore depends on is independent of the kernel version. GPU driver installers and other PCI-device installers detect the running kernel with uname -r and compile DKMS against whatever the target has. The APT package install pulls binaries built for the target's distribution. IOMMU cmdline tokens like intel_iommu=on are stable across major kernel versions. The backup carries paths written under a newer kernel, but those paths (module blacklists, GRUB defaults, initramfs config) are still valid syntax on the older one — kernels ignore tokens they do not recognise instead of failing. This case is equivalent in practice to a same-kernel restore, and ProxMenux treats it as such.
The safe-subset filter (only when the target kernel is newer)
When the target kernel is newer than the backup's, the compatibility check adds 16 boot-critical paths from hb_unsafe_paths_cross_version to RS_SKIP_PATHS. These paths are excluded from the restore because writing a version of them from an older kernel over a target running a newer one has caused kernel panics in bench tests. The paths cover four categories:
| Category | Paths | Why they are skipped |
|---|---|---|
| Bootloader | /etc/default/grub, /etc/kernel | GRUB defaults tied to the prior kernel order, and proxmox-boot-tool state (cmdline, ESP UUIDs, hooks) that references paths and identifiers from the older install. |
| Kernel modules and boot artifacts | /etc/modules-load.d, /etc/modprobe.d, /etc/initramfs-tools | Autoload lists that may reference modules renamed between kernel majors, module options that may not apply, initramfs hooks written for the older kernel. |
| Storage stack and filesystem identity | /etc/fstab, /etc/multipath, /etc/iscsi, /etc/udev/rules.d, /etc/zfs | UUIDs that may not exist on this install, multipath drivers that change between kernels, iSCSI parameters that evolve, udev rules that may bind to nonexistent subsystems, ZFS state (zpool.cache + hostid) that can lock the pool as foreign. |
| APT sources | /etc/apt | APT source suites may trigger a downgrade of critical packages on the next upgrade. |
| systemd | /etc/systemd/system, /etc/systemd/journald.conf, /etc/systemd/logind.conf, /etc/systemd/system.conf, /etc/systemd/user.conf | Unit overrides and .wants tied to the older systemd major; configuration keys that may not parse on a newer systemd. |
The filter runs BEFORE the confirmation dialog so the user sees the exact list of paths that will be skipped, categorised by reason. The restore still proceeds — everything else (VMs, LXCs, network, /etc/pve, users, cron, ProxMenux state, packages, drivers, /root) is restored normally.
Kernel-agnostic hydration
The safe-subset filter alone would leave the target without the tuning the user had inside those boot-critical files: IOMMU cmdline for GPU passthrough, VFIO device IDs, custom GRUB_TIMEOUT, nvidia blacklists. The hydration pass re-applies those bits kernel-agnostically. Four phases run when the target kernel is newer than the backup's, each additive (never overwrites a value the target already carries) and idempotent (running twice is a no-op).
| Phase | Detail |
|---|---|
| 1a — GRUB path | For hosts using GRUB (ext4/lvm installs). _rs_hyd_grub merges every token from the backup's manifest.kernel_params.cmdline_extra into the target's live GRUB_CMDLINE_LINUX_DEFAULT, skipping tokens whose key the target already carries. Then merges whitelisted GRUB_* keys (GRUB_TIMEOUT, GRUB_TIMEOUT_STYLE, GRUB_DEFAULT, GRUB_TERMINAL, GRUB_DISABLE_OS_PROBER, GRUB_SERIAL_COMMAND, GRUB_GFXMODE, GRUB_GFXPAYLOAD_LINUX) from the backup's /etc/default/grub if they differ from the target's. |
| 1b — systemd-boot / ZFS path | For hosts using systemd-boot (typically ZFS-on-root). _rs_hyd_kernel_cmdline merges operator tokens from cmdline_extra into the target's /etc/kernel/cmdline, keeping the target's own root=, boot= and rootflags= boilerplate intact. |
| 2 — /etc/modules merge | _rs_hyd_modules appends modules from manifest.kernel_params.modules_loaded_at_boot that are in the whitelist (vfio, vfio_pci, vfio_iommu_type1, vfio_virqfd, kvm, kvm_intel, kvm_amd, nvidia, nvidia_drm, nvidia_modeset, nvidia_uvm, i915, xe) AND not already present in the target's /etc/modules. |
| 3 — Whitelisted files copy | _rs_hyd_files copies operator-authored files from the staging rootfs to the live target when the content differs. Whitelist covers VFIO/nvidia/blacklist files under /etc/modprobe.d, /etc/modules-load.d, and the ProxMenux VFIO bind rule + nvidia udev rules under /etc/udev/rules.d. Distro-owned files (pve-blacklist.conf, mdadm.conf, nvme.conf) are intentionally excluded — their contents evolve between releases. |
| 4 — Force post-boot reflows | The four phases write directly to the live target OUTSIDE the normal restore pipeline. To make the merged tokens/modules/files take effect on the next boot, HB_HYDRATION_APPLIED=1 propagates through plan.env to apply_pending_restore.sh, which forces NEEDS_INITRAMFS=1 and NEEDS_GRUB=1 regardless of what was in the apply list. The post-boot dispatcher then regenerates initramfs and refreshes the bootloader. |
Plan vs commit — the operator sees a preview first
The hydration runs in two modes. Before the confirmation dialog, ProxMenux runs _rs_apply_bk_older_hydration in plan mode: it computes exactly what would be merged, populates RS_HYDRATION_SUMMARY with a green block listing each action, and returns without writing. The confirmation dialog shows that green block alongside the amber safe-subset skip list, so the user sees UPFRONT what will be re-applied automatically. After the user confirms, ProxMenux re-runs the same helper in commit mode — same phases, same logic, but this time each phase writes to the live target. Cancelling the confirmation dialog leaves the target untouched.
The flow when the target kernel is newer than the backup's
The steps added when the target kernel is newer sit inside the normal restore flow. Everything else — hot apply, pending prepare, package install, post-boot dispatcher — runs identically to a same-kernel restore.
┌────────────────────────────────────────────────────────────┐
│ hb_compat_check │
│ HB_COMPAT_KERNEL_DIRECTION = bk_older │
└───────────────────────────┬────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Add 16 boot-critical paths to RS_SKIP_PATHS │
│ /etc/default/grub, /etc/kernel, /etc/modules-load.d, ... │
└───────────────────────────┬────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ _rs_apply_bk_older_hydration "plan" │
│ Compute what would be merged │
│ Populate RS_HYDRATION_SUMMARY (green block) │
│ No writes │
└───────────────────────────┬────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Confirmation dialog │
│ Amber: paths skipped by safe-subset filter │
│ Green: tokens/files reapplied by hydration │
│ User accepts or cancels │
└───────────────────────────┬────────────────────────────────┘
│ (accepted)
▼
┌────────────────────────────────────────────────────────────┐
│ _rs_apply_bk_older_hydration "commit" │
│ Phase 1a/1b: merge cmdline tokens + GRUB keys │
│ Phase 2: append modules to /etc/modules │
│ Phase 3: copy whitelisted vfio/nvidia files │
│ Set HB_HYDRATION_APPLIED=1 │
└───────────────────────────┬────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Rest of the restore runs normally │
│ _rs_apply hot (skips RS_SKIP_PATHS) │
│ _rs_prepare_pending_restore (writes plan.env with │
│ HB_HYDRATION_APPLIED=1) │
│ packages.manual.list install │
│ Reboot │
│ apply_pending_restore.sh (forces NEEDS_INITRAMFS=1, │
│ NEEDS_GRUB=1 because of the hydration flag) │
│ apply_cluster_postboot.sh │
│ update-initramfs -u -k all │
│ update-grub / proxmox-boot-tool refresh │
│ component --auto-reinstall │
└────────────────────────────────────────────────────────────┘Concrete examples
The hydration pass is not abstract — it produces observable, correct outcomes for common scenarios. Two examples that hydration handles automatically:
| Scenario | What hydration does |
|---|---|
| GPU passthrough (VFIO) | Source had intel_iommu=on iommu=pt in the cmdline, vfio/vfio_pci/vfio_iommu_type1 in /etc/modules, an /etc/modprobe.d/vfio.conf with options vfio-pci ids=10de:2216, and a /etc/modprobe.d/blacklist-nvidia.conf. Hydration merges the cmdline tokens into the target's GRUB or kernel cmdline, appends the vfio modules to /etc/modules, and copies the two user-authored modprobe files. On the next boot, IOMMU is active, VFIO modules load, the GPU is bound to vfio-pci and the VM comes up with passthrough working. |
| Custom GRUB defaults | Source had GRUB_TIMEOUT=1 and GRUB_DISABLE_OS_PROBER=true. Hydration reads both keys from the backup's /etc/default/grub, sees they differ from the fresh install's defaults, and rewrites those two lines in the target's file (leaving everything else, including GRUB_DISTRIBUTOR, untouched). |
What hydration does not reapply when the target kernel is newer
Hydration reapplies only the configuration ProxMenux knows to be safe across kernel versions: IOMMU tokens, VFIO IDs, whitelisted GRUB keys, and modules from a fixed list (vfio*, nvidia*, i915, xe, kvm*). Anything outside that set — custom initramfs hooks under /etc/initramfs-tools/hooks/, non-whitelisted files under /etc/modprobe.d/, user-authored systemd unit overrides — remains excluded. The reason is concrete: those files can call kernel-internal interfaces (module APIs, /sys layout, udev hooks) that change between major versions, and applying them verbatim over the newer kernel can prevent the target from booting. When exact reproduction of the boot chain is required, the restore must run on a host with the same major kernel version as the backup.
Where the mechanisms live
For developers looking to trace or extend the cross-kernel behaviour:
| Component | Location |
|---|---|
| Direction detection | hb_compat_check in lib_host_backup_common.sh. Sets HB_COMPAT_KERNEL_DIRECTION. |
| Safe-subset path list | hb_unsafe_paths_cross_version in lib_host_backup_common.sh. Emits path\treason lines used by both the CLI filter and the Web endpoint /api/host-backups/restore/prepare. |
| Hydration phases | _rs_hyd_grub, _rs_hyd_kernel_cmdline, _rs_hyd_modules, _rs_hyd_files in backup_host.sh. Orchestrated by _rs_apply_bk_older_hydration. |
| Post-boot reflow forcing | apply_pending_restore.sh reads HB_HYDRATION_APPLIED from plan.env and forces NEEDS_INITRAMFS=1/NEEDS_GRUB=1. |
| Web preview | /api/host-backups/restore/prepare in flask_server.py. Sources the helper library, runs _rs_apply_bk_older_hydration in plan mode, returns the actions to the Web modal. |
Where to go next
- Restoring — the full restore pipeline this page's mechanisms plug into.
- How it works — the manifest and collectors that produce the <code>kernel_params</code> block hydration reads from.