Multi-container applications

1. The plan, before any container exists

No LXC is created while the Compose definition is interpreted. A complete plan comes first, with every member, image, VMID, network, path, secret, order and health check. If the plan is not consistent, the installation does not start.

2. Creation, member by member

1. 1

   Reserve every VMID

   The Proxmox VE inventory and the instance registry are checked. An existing CT is not adopted and a contract that still belongs to another instance is not reused.

2. 2

   Prepare every image

   All images are resolved, downloaded and verified before the first container is created.

3. 3

   Create each rootfs

   The official OCI metadata is imported, and the ProxMenux instance identity and the member role are added.

4. 4

   Private network

   A bridge and subnet are created and each member gets its fixed address; only the main member also gets the LAN interface.

5. 5

   Persistence

   Each database and configuration gets its own container disk; only data meant to be shared uses host directories.

6. 6

   Environment

   Internal endpoints, shared secrets and service variables are written. The application reaches its dependencies at their reserved private addresses.

7. 7

   Record

   Each member records its native configuration, the rootfs changes that updates have to reproduce and its relation to the stack.

3. Start and check each container

A created LXC does not mean a ready service. Dependencies start in order and each one passes a check specific to its service.

4. The dependency hook

The hook is set only on the main container of a dependent stack. Proxmox VE keeps the script as a snippet and runs it as the hookscript of that CT. The stack recipe is not written in the script: it lives in a separate private contract.

hookscript: local:snippets/proxmenux-stack-dependencies.sh

/etc/pve/priv/proxmenux-stack-VMID.json
{
  "schema": 1,
  "stack": "immich",
  "dependencies": [
    {"vmid": 107, "label": "PostgreSQL", "healthcheck": {...}},
    {"vmid": 108, "label": "Valkey", "healthcheck": {...}},
    {"vmid": 106, "label": "Machine Learning", "healthcheck": {...}}
  ]
}

#!/usr/bin/env bash
set -Eeuo pipefail

PATH=/usr/sbin:/usr/bin:/sbin:/bin

die() {
  printf 'ERROR: %s\n' "$*" >&2
  exit 1
}

find_snippet_storage() {
  local storage
  if pvesm status --content snippets 2>/dev/null \
    | awk 'NR > 1 && $1 == "local" && $3 == "active" {found=1} END {exit !found}'; then
    printf 'local'
    return
  fi
  storage=$(pvesm status --content snippets 2>/dev/null \
    | awk 'NR > 1 && $3 == "active" {print $1; exit}')
  [[ -n $storage ]] || die "No active storage supports snippets"
  printf '%s' "$storage"
}

install_hook() {
  local main_id=${1:?missing main VMID} source_config=${2:?missing lifecycle JSON}
  local storage hook_volume hook_path target_config
  [[ $main_id =~ ^[0-9]+$ ]] || die "Invalid main VMID"
  jq -e '
    .schema == 1 and
    (.dependencies | type == "array") and
    (.dependencies | length > 0) and
    (all(.dependencies[];
      (.vmid | type == "number") and
      (.label | type == "string") and
      (.healthcheck.type | IN("exec", "http", "running")) and
      (.healthcheck.timeout_seconds | type == "number")
    ))
  ' "$source_config" >/dev/null || die "Invalid dependency contract"

  storage=$(find_snippet_storage)
  hook_volume="${storage}:snippets/proxmenux-stack-dependencies.sh"
  hook_path=$(pvesm path "$hook_volume")
  install -D -m 0755 "$0" "$hook_path"

  target_config="/etc/pve/priv/proxmenux-stack-${main_id}.json"
  umask 077
  cat "$source_config" >"$target_config"
  pct set "$main_id" --hookscript "$hook_volume" >/dev/null
  printf 'Proxmox hookscript installed: CT %s starts its dependencies through %s\n' \
    "$main_id" "$hook_volume"
}

dependency_is_healthy() {
  local id=$1 healthcheck=$2 type url
  type=$(jq -r '.type' <<<"$healthcheck")
  case "$type" in
    running)
      [[ $(pct status "$id" 2>/dev/null || true) == "status: running" ]]
      ;;
    exec)
      local -a command=()
      mapfile -t command < <(jq -r '.argv[]' <<<"$healthcheck")
      ((${#command[@]} > 0)) || return 1
      pct exec "$id" -- "${command[@]}" >/dev/null 2>&1
      ;;
    http)
      url=$(jq -r '.url' <<<"$healthcheck")
      curl -fsS --max-time 3 "$url" >/dev/null 2>&1
      ;;
    *) return 1 ;;
  esac
}

start_dependencies() {
  local main_id=$1 config="/etc/pve/priv/proxmenux-stack-${1}.json"
  local encoded dependency id label healthcheck timeout elapsed
  [[ -r $config ]] || die "Missing dependency contract for main CT $main_id"

  exec 9>"/run/lock/proxmenux-stack-${main_id}.lock"
  flock 9
  while IFS= read -r encoded; do
    [[ -n $encoded ]] || continue
    dependency=$(base64 -d <<<"$encoded")
    id=$(jq -r '.vmid' <<<"$dependency")
    label=$(jq -r '.label' <<<"$dependency")
    healthcheck=$(jq -c '.healthcheck' <<<"$dependency")
    timeout=$(jq -r '.healthcheck.timeout_seconds' <<<"$dependency")
    [[ $id =~ ^[0-9]+$ && $timeout =~ ^[0-9]+$ && $timeout -gt 0 ]] \
      || die "Invalid dependency in $config"
    pct config "$id" >/dev/null 2>&1 \
      || die "Dependency $label (CT $id) does not exist"

    if [[ $(pct status "$id" 2>/dev/null || true) != "status: running" ]]; then
      printf 'Starting dependency %s (CT %s)...\n' "$label" "$id"
      pct start "$id" || die "Could not start $label (CT $id)"
    else
      printf 'Dependency %s (CT %s) was already running.\n' "$label" "$id"
    fi

    elapsed=0
    while (( elapsed < timeout )); do
      if dependency_is_healthy "$id" "$healthcheck"; then
        printf 'Dependency %s (CT %s): ready.\n' "$label" "$id"
        break
      fi
      [[ $(pct status "$id" 2>/dev/null || true) == "status: running" ]] \
        || die "$label (CT $id) stopped while starting"
      sleep 2
      elapsed=$((elapsed + 2))
    done
    (( elapsed < timeout )) \
      || die "$label (CT $id) did not pass its health check within ${timeout}s"
  done < <(jq -r '.dependencies[] | @base64' "$config")
}

if [[ ${1:-} == "--install" ]]; then
  shift
  install_hook "$@"
  exit 0
fi

vmid=${1:?missing VMID}
phase=${2:?missing lifecycle phase}
case "$phase" in
  pre-start) start_dependencies "$vmid" ;;
  post-start|pre-stop|post-stop) ;;
  *) die "Unknown lifecycle phase: $phase" ;;
esac

The script is the same for every stack. VMIDs, names, check types and timeouts come from the private contract /etc/pve/priv/proxmenux-stack-VMID.json of each stack.

1. 1

   Proxmox VE calls pre-start

   Before the main CT starts, the hook runs with its VMID and the lifecycle phase.

2. 2

   A lock per stack

   flock on /run/lock/proxmenux-stack-VMID.lock prevents two start sequences at the same time.

3. 3

   The contract is read and validated

   A known schema, numeric dependencies, labels, an exec, http or running check and a positive timeout are required.

4. 4

   Dependencies in order

   Each CT must exist; a stopped one is started and one already running is not restarted.

5. 5

   Wait for health

   The check runs every two seconds, and the CT is also confirmed to be still running.

6. 6

   The main CT starts

   When every dependency is ready, pre-start ends and Proxmox VE starts the application.

5. Checks on the stack as a whole

• Every VMID exists, is unique and keeps the expected instance identity.
• Every contract belongs to the same stack, keeps its role and its recorded native configuration.
• The main member is last in start_order and first in stop_order.
• No member, volume or rootfs adaptation is missing.
• The hook points to the official snippet, its content is unchanged and its contract matches the stack recipe.
• The observed images and digests match the prepared OCI archives.
• Devices, GPU profiles, mounts, secrets and endpoints still match the contracts.
• After every dependency passes, the endpoint of the main application checks the integration between members.

Managing an installed stack

In Manage installed OCI applications, any member leads to the whole stack. The menu of a stack offers Update every container of the application, Modify extra paths and devices, Recreate every container with its saved configuration and Remove: delete the application and its containers. Recreate rebuilds every container from the image it was installed with, by its saved digest and without looking for a newer one, with the same steps as an update; it is offered for the stacks that can be updated.

1. 1

   Any member

   The contract of the member names the main VMID and the complete list of members.

2. 2

   The stack is reproducible

   Identities, contracts, hook, adaptations and a coordinated replay for that recipe are validated.

3. 3

   Images first

   The application is not stopped until every digest has been resolved, downloaded and verified.

4. 4

   Stop and back up the set

   Verified native backups are taken with the stack stopped, so application and databases belong to the same moment.

5. 5

   Update and check each member

   Adaptation, network, mounts, secrets and devices are applied again before the health check of each member.

6. 6

   Publish or recover everything

   The new contracts are published only when every member passes. If one fails, every member is restored.

6. When something fails

During an installation, an error stops and removes the incomplete containers that operation created, and the private bridge if the operation created it. A stack is not published as valid until the whole sequence finishes.

An update is transactional: all images are prepared first, then the stack stops, each member is backed up and the backup verified, and only then is each rootfs replaced. Members start one by one with their health check; if one fails, every member is restored from the same set of backups, so the database and the application never belong to different moments.

What a stack does not include

• A suite installed in one flow, such as the Arr suite, is not a dependent stack: its containers have no start order between them.
• The private network does not replace authentication, TLS or the configuration of each application.
• The backup of one LXC does not contain the bridge, the hook or the contracts of the rest of the stack.
• Prowlarr, Sonarr or Radarr receive no indexers, profiles or providers from ProxMenux.