Alerts & Maintenance
Monitors alert through KubeSense Alerts, so synthetic alerts reach the same notification channels, and get the same acknowledgement, muting and routing, as your other alerts.
How a Monitor Goes Down
Each run passes or fails. A monitor's state follows its alert settings:
| State | When |
|---|---|
| Up | Runs pass |
| Degraded | Runs pass, but slower than Degraded above |
| Down | Failure threshold consecutive runs failed (and for at least Alert only after failing for, when set) in at least Location threshold (N-of-M) locations |
| No data | No result arrived within the No-data timeout |
| Paused | The monitor is paused |
A down monitor is up again after Recovery threshold consecutive passing runs. Each down period is recorded as an outage on the monitor, whether or not anyone was notified.
Fast retries re-run a failed attempt before the result is recorded, so a single dropped packet does not count as a failure. Flap detection turns a monitor that keeps changing state into one alert that stays open until it settles.
Alert Rules
KubeSense keeps one alert rule per monitor, named [Synthetics] <monitor name>. It is created, changed, paused and deleted with the monitor, and fires while the monitor is down or has no data.
| Rule | Fires when |
|---|---|
| [Synthetics] <monitor> | The monitor is down or reports no data |
| [Synthetics] <monitor> — SSL certificate expires soon / expiry critical | An SSL monitor's certificate is within its warning or critical days |
| [Synthetics] Location has no online agent | A location that has agents has had none online for 5 minutes |
These rules are managed by Synthetics: change them through the monitor, not in the alert rule editor. Their alerts carry the monitor's Priority as severity, and its labels.
Notification channels
Pick channels in a monitor's Notify channels field. They are the KubeSense notification channels — Slack, Microsoft Teams, email, PagerDuty, webhooks and the others configured there. Defaults for new monitors are set in Settings → Defaults & usage.
The Notification message is a Markdown template. It can use the monitor's details, and sections that show only for some events:
{{#is_alert}}{{monitor.name}} is down. Runbook: https://wiki.example.com/{{monitor.service}}{{/is_alert}}
{{#is_recovery}}{{monitor.name}} recovered after {{incident.duration}}.{{/is_recovery}}
{{monitor.link}}| Variable | Value |
|---|---|
monitor.name, monitor.id, monitor.type | The monitor |
monitor.service, monitor.env, monitor.team | The monitor's labels of those names |
monitor.link | The monitor's page in KubeSense |
alert.status, alert.startsAt, alert.duration | The alert |
incident.duration | How long the monitor has been (or was) down |
labels.<key> | Any label of the alert |
Sections: {{#is_alert}}, {{#is_recovery}}, {{#is_renotify}}, {{#is_warning}}, {{#is_no_data}}, {{#is_flapping}}, {{#is_priority "P1"}}, {{#is_match "labels.env" "prod"}} and {{#is_exact_match …}}, each closed with {{/…}}; {{^is_alert}} is the inverse.
Re-notify while DOWN repeats the notification while the monitor stays down, up to Stop after … re-notifications. Notification content can hide the monitor's type and target, or @-mentions, from the message.
The Alerts page
Synthetics → Alerts lists the synthetic alerts that are firing now, from KubeSense alerting. Open in Alert history opens the alert events page, where you can acknowledge, assign and follow each alert. Monitor state changes also appear in the Events explorer.
Maintenance Windows
A maintenance window silences or pauses monitors for a planned period, such as a deployment or database patching. Manage them under Settings → Synthetics → Maintenance windows, or select monitors in the list and use Add to maintenance window. A suite's Add to downtime creates one for the suite.
Click New window:
| Field | Description |
|---|---|
| Name | e.g. Weekly DB patching |
| Mode | Mute alerts: checks keep running, their alerts are silenced. Pause checks: checks do not run, and the window is left out of uptime |
| Monitors / Suites / Match labels | What the window applies to: picked monitors, every monitor of picked suites, or every monitor whose labels match |
| Mute states | Which alerts to mute: DOWN, DEGRADED, NO_DATA. Empty mutes all |
| Schedule | One-shot with Starts at, or Recurring (cron) with a cron expression (presets: daily at 02:00, every 6 hours, first of the month, weekdays at 22:00, weekly on Sunday) |
| Duration (minutes) | How long each occurrence lasts |
| Timezone | Timezone of the start time and cron expression |
| Enabled | Switch the window off without deleting it |
The list shows each window's mode, scope, schedule, next start, duration and status (active now, scheduled or disabled), as a list or a calendar. Active now only shows the windows in effect.
A muting window also silences the monitors' KubeSense alerts while it is active. A monitor's Properties tab lists the windows that apply to it.
note: To stop a monitor outside fixed hours every day or week, use its Active hours instead of a recurring maintenance window: outside them the monitor does not run, alert or count toward uptime.