Alerting
Alerting is built from three distinct concepts so it can grow without becoming a tangle of special cases.
1. Alerters (channels)
An Alerter is where notifications go. Pick a channel type:
- Email — via the SMTP transport (
Smtp__*env; the demo uses Mailpit at http://localhost:8025). - Webhook — POSTs a structured JSON payload of the alert to each configured URL.
- Slack — posts a formatted message to each Slack incoming-webhook URL.
- Discord — posts a formatted message to each Discord webhook URL (2000-character cap).
- Custom HTTP request — sends an operator-defined request (method, URL, custom headers, body) to a
single endpoint. The URL, header values and body support
{{placeholder}}tokens (see Custom HTTP templating) so you can integrate any HTTP API — PagerDuty, Opsgenie, a home-grown endpoint, etc.
Recipients accept plain email addresses/URLs, or a user:{id} token that resolves dynamically to a
user's current notification email — so a rotating on-call user stays correct without editing the
alerter. Webhook/Slack/Discord/Custom-HTTP URLs are re-validated against the
SSRF guard at send time and redacted from logs.
Alerters can also carry labels (free-form tags) for grouping and filtering.
Custom HTTP templating
A Custom HTTP alerter's URL, each header value, and the body are templates: any {{token}} is
substituted from the firing alert's context at send time. Tokens that don't apply to the firing alert
type are left unchanged. Nothing is auto-escaped — you author the body and its content type (default
application/json when a body is present and no Content-Type header is set), so quote/escape values
as your target API needs.
Which tokens are available depends on the alert type of the rule that fires the alerter, so the Alerter form shows a per-type reference list. The common set is always present:
| Token | Value |
|---|---|
{{alert.type}} | The alert type that fired (e.g. ResourceDown). |
{{alert.name}} | The rule's display name (may be empty). |
{{event}} | triggered or resolved. |
{{firedAt}} | UTC timestamp the notification fired. |
Resource alerts add {{target.name}}, {{target.status}}, {{target.since}},
{{target.consecutiveFailures}}; cert-expiry adds {{cert.expiresAt}} and {{cert.thresholdDays}};
Scout alerts add {{scout.name}} and {{scout.status}}.
2. Alert types
The kinds of events you can alert on:
| Type | Fires when | Threshold |
|---|---|---|
| Resource Down | A target transitions to Down. | Consecutive failures OR a duration — either alone is enough. |
| Resource Recovered | A target transitions Down → Up. | — |
| Resource Degraded | A target transitions to Degraded. | Duration. |
| Scout Down / Up | A probe agent disconnects or reconnects. | Scheduled grace delay. |
| Cert Expiring | A monitored TLS certificate is within N days of expiry. | List of day thresholds (e.g. 30/14/7/3/2/1) — each fires once and re-arms on renewal. |
| Clock Skew Warning | An Scout's clock drifts ≥5s from the server. | — |
| Key Rotation Failed | An Scout's key rotation stays stuck past the retry window while it's online. | — |
| Scout Untrusted | An Scout's key age exceeds the max without a successful rotation. | — |
| Incident Event | An incident is created, reaches a chosen status, or is resolved. | Configurable trigger set. |
KeyRotationFailed and ScoutUntrusted are system-level — they're always unscoped, not tied to a
resource.
3. Alert rules
A Rule ties an alert type to an alerter and decides when to notify:
- Name (optional) — a display name so a rule is identifiable at a glance. The Alert Rules list has a search box that filters by name, type, alerter, target and labels.
- Labels (optional) — free-form tags for grouping/filtering rules.
- Scope — a specific resource (for resource/cert types), or leave it unscoped to match all (required for the system-level types above).
- Thresholds — for resource-down rules, only alert after N consecutive failures or after a duration of continuous downtime, to avoid noise from single blips.
- Cert-expiry thresholds — a list of day counts (e.g. 30, 14, 7, 3, 1); each fires once and re-arms when the certificate is renewed.
- Also notify when this resolves (Resource Down / Scout Down rules, default on) — sends a follow-up "resolved" notification through the same alerter once the target recovers or the Scout reconnects, so you don't need a separate Recovered/Up rule to close the loop.
- Each rule can be enabled/disabled independently and remembers its last-fired timestamp.
Notifications are deduplicated per-outage and per-cert-threshold (an "already notified" flag clears on recovery/renewal), and maintenance windows silently suppress notifications for resources they cover — checks keep running and logging normally, only the paging is muted (see below).
Every fired, suppressed, or failed alert is recorded as an event, visible on the dashboard's "last alerts" feed with a Sent / Failed / Suppressed status.
Creating a rule
- Create an Alerter with at least one recipient.
- Go to Alerting → Alert Rules → New.
- Pick an alert type, choose the alerter, set the scope, and (optionally) thresholds.
Sent alerts land in Mailpit in the demo stack. Point Smtp__Host/Smtp__Port at a real relay for
production delivery.
Maintenance windows
Planned downtime shouldn't page anyone. A maintenance window (Maintenance section) suppresses alert notifications for the resources it covers — checks still run and are logged; only the paging is muted.
Schedule:
- One-off — an absolute UTC start/end.
- Recurring — Daily, Weekly (pick one or more days of the week), or Monthly (a specific day-of-month, or "last day of the month"). Recurring windows run on wall-clock time in an IANA timezone with DST-safe occurrence math, and a duration (in minutes) that may cross midnight.
Scope: specific resources, or leave it empty to cover everything (global).
Lifecycle: a window is derived as Scheduled → In Progress → Completed, or Cancelled if you cancel it early.
Effects: while a window is in progress, alert notifications for its covered resources are suppressed (checks keep running/logging normally), and — optionally — any status page containing an affected component shows a public "Under Maintenance" badge (see Status pages).
Each window has a sanitized public message and a per-window enable/disable toggle.