Health Monitor

How it works

The Health Monitor runs on two parallel lanes inside the Monitor process. A lightweight vital signs sampler reads CPU, memory and temperature every few seconds so that sustained-threshold conditions are detected fast; in parallel, the full health cycle runs every five minutes and exercises every category from end to end. Both lanes converge into the same SQLite tables — and from there, three consumers read the state independently.

From sample to stored finding

From stored finding to user

The ten categories

Every cycle exercises ten independent checkers. Each produces one of four statuses (OK, INFO, WARNING, CRITICAL) plus a structured payload — device names, sample log lines, exact thresholds — that surface in the dashboard and travel through to the notification body.

Severity model

A category that is OK but has dismissed events still inside their suppression window is rendered as INFO — to remind you that something is being silenced rather than that nothing was ever wrong.

The dashboard view

The Health Monitor lives inside the Overview tab. The header status pill (Healthy / Warning / Critical) opens a modal that splits findings into two lists:

• Active — every category with an unresolved finding. Each row expands to show the individual checks that produced the status, the raw reason string, the device or VM ID involved, and (for categories that link to a tab) a click-through into Storage / Network / VMs / Logs / Hardware to investigate.
• Dismissed — items previously acknowledged by the user that are still inside their suppression window. Each row shows how much of the suppression remains and the configured duration. When the window expires, the item disappears from this list; if the underlying condition is still present and the category supports re-firing, it re-appears in Active.

Dismissing alerts and the Suppression Duration

Some events are noisy by nature — a System Updates: pending updates available stays true until you patch the host, and you don't want a notification every five minutes for a week. The Health Monitor solves this with two coupled mechanisms:

1. Per-event Dismiss action in the modal. The Dismiss button opens a small dropdown with three options — 24 hours, 7 days or Permanently — letting you choose how long this specific alert stays silenced regardless of the category's default. Picking one calls POST /api/health/acknowledge with the error_key and the chosen suppression_hours (-1 for permanent). The event moves to the Dismissed list with a timestamped acknowledged_at.
2. Per-category Suppression Duration setting. From the Settings → Health Monitor card (or POST /api/health/settings), each of the ten categories has its own default window applied when a Dismiss is fired without a per-event choice:

   24 hours       (default)
   72 hours
   168 hours      (one week)
   720 hours      (one month)
   8760 hours     (one year)
   -1             (permanent — never re-fires)
   <custom>       (any positive integer of hours)

While an event is suppressed, the scanner still runs and updates the row's last_seen timestamp, but no new notification is dispatched and the dashboard stays calm. When the window expires, the next cycle re-evaluates the condition and either re-fires fresh or, if the condition has cleared on its own, drops the row from the lists.

Reviewing and reverting dismisses — the Active Suppressions panel

Every currently-silenced alert (time-limited and permanent) is listed under Settings → Health Monitor → Active Suppressions. Each row shows the alert identifier, category, severity, when it was dismissed and how much time is left, plus a Re-enable button that clears the acknowledgment so the alert can fire again on the next scan. Permanent dismisses can only be reverted from here; time-limited ones can also be force-revived without waiting for the countdown. The Re-enable action is gated by the Health Monitor Edit mode at the top of that card — toggle Edit, click Re-enable on each row you want to revive (queued rows show a green border and a strike-through), then click Save to commit. Cancel discards the queue.

Auto-suppression when you change the Duration

Setting a category's Suppression Duration to anything other than the default 24 h has a second effect beyond user-initiated dismissals: future findings in that category enter the table already acknowledged with that duration. This is by design — if you've told the Monitor that you want disk-related events silenced for a week, brand-new disk findings honour that intent without you having to dismiss each one by hand. They appear directly in the Dismissed list with the configured remaining time. Categories left at 24 h are unaffected and behave the classic way (new findings land in Active until you act).

Findings that cannot be dismissed

A handful of findings are flagged non-dismissable on purpose — they signal a condition where silencing the alert could cost data, hardware or connectivity. The Dismiss button is hidden for these rows; the alert clears only when the underlying condition recovers and the auto-resolve cleanup picks it up. Other findings (transient I/O events on a healthy disk, recovered states) are also marked non-dismissable but for the opposite reason: there's nothing to silence because the row is already informational and self-clearing.

Everything else can be dismissed. The principle is: alerts that indicate "real damage in progress" or that have already self-resolved are kept off the dismiss path; alerts about sustained conditions you may want to acknowledge and re-check later (high CPU usage, pending updates, certificate near expiry, log warnings, VM startup hiccups, etc.) all expose the Dismiss button.

Auto-resolution and cleanup

Many alerts should clear themselves when the condition goes away — a VM that was failing to start and is now running, a disk that's no longer in the system, a temperature that dropped back to normal. A cleanup routine runs at the end of each five-minute cycle and applies these rules:

Disk observations — the permanent history

Disk events are special. A SMART warning on /dev/sdh at 02:14 AM is something you want to remember even after the I/O storm subsided and the error auto-resolved — the disk has a track record now. For that purpose, the Health Monitor keeps a separate permanent table: disk_observations.

Practical consequence: an alert can clear from the dashboard while the same incident is still recorded in the disk's history. When you click into a disk under Storage, the card shows the count of outstanding observations and a list with timestamps, severity and the original raw message — useful when you're deciding whether a drive needs replacement.

From a finding to a notification

Every active error is also a candidate for the notification engine. The flow:

1. The scanner records the finding with category + severity + structured details.
2. If the event type is enabled in the global notification settings, and the channel hasn't silenced this category, an event is queued.
3. The template engine renders a (title, body) pair from the structured details. If the AI rewriter is enabled, the same pair is also passed through the configured provider for a plain-language version.
4. The channel implementation ships it: Telegram message, Discord embed, Gotify push or email. The dispatch outcome is stored in notification_history.
5. If a dismiss arrives later, the suppression window kicks in and any further re-fires of the same error_key stay queue-side until the window closes.

Channel configuration (Telegram bot token, webhook URLs, AI provider keys, per-event toggles, channel overrides) is documented in Notifications and AI Assistant.

REST endpoints

Everything the modal does is callable from the API — handy for scripts, custom dashboards or your own chat-bot integration.

# Snapshot the current health for a script
curl -H "Authorization: Bearer <api-token>" \
  http://<host>:8008/api/health/full | jq '.health.overall'

# Dismiss a specific error
curl -X POST http://<host>:8008/api/health/acknowledge \
  -H "Authorization: Bearer <api-token>" \
  -H "Content-Type: application/json" \
  -d '{"error_key":"smart_sdh"}'

# Set the disks-category suppression to a week
curl -X POST http://<host>:8008/api/health/settings \
  -H "Authorization: Bearer <api-token>" \
  -H "Content-Type: application/json" \
  -d '{"suppress_disks":"168"}'

Where to next

• Dashboard — where the Health Monitor modal is opened from in the UI.
• Notifications — channels, per-event toggles, the AI rewrite hook, history.
• AI Assistant — provider configuration (OpenAI / Anthropic / Gemini / Groq / Ollama / OpenRouter), prompt mode, per-channel detail level, language.
• Architecture — the SQLite schema (errors, disk_observations, events) and the background-thread cadence.