Alerts API

The same alert rules you build in the UI can be driven from your own tooling — a GitOps pipeline that reconciles rules from a repo, a migration script, or a service that provisions a standard alert pack for every new cluster.

This page covers the full alerting surface: reading rules and their firing history, creating them, updating, enabling/disabling and deleting them, and the same for notification channels. For reading logs and traces see the Logs & Traces API; for metrics see the Metrics API; for dashboards see the Dashboards API.

Authentication

An API key in the X-API-Key header:

X-API-Key: <your-api-key>

Create one under Settings → API Key Management. See Authentication for details.

API keys are scoped per key: you choose which modules a key may reach when you create it. For the endpoints on this page the key needs the alerts scope (for rules) and alerts_notification_channels (for channels) — a key without the scope is rejected for that module regardless of who created it.

Request conventions

Endpoints

Alert rules

Alert activity

Notification channels

Rule {id} is the rule's UUID; channel {id} is an integer.

List alert rules

GET /api/alerts/rules

curl -s "https://<your-kubesense-host>/api/alerts/rules?limit=50&search=cpu" \
  -H "X-API-Key: $KUBESENSE_API_KEY"

Rules come back newest first (created_at descending). The envelope carries a count alongside data — the total matching your filters, not the size of the page, so it is what you paginate against.

Fetch one alert rule

GET /api/alerts/rules/{id} returns the stored rule.

The response also carries evaluation state the write side has no say over — current_state, last_evaluation_time, last_state_change_time, firing_since, consecutive_breaches — plus created_at / created_by.

Create an alert rule

POST /api/alerts/rules

curl -s -X POST "https://<your-kubesense-host>/api/alerts/rules" \
  -H "X-API-Key: $KUBESENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High pod CPU {{pod}}",
    "description": "Pod CPU above 80% for 5 minutes",
    "severity": "warning",
    "status": "active",
    "query": [
      {
        "label": "A",
        "selectedMode": "metrics",
        "queryMode": "code",
        "promql": "max by (pod)(rate(container_cpu_usage_seconds_total{namespace=\"prod\"}[5m]))",
        "visible": true,
        "fields": []
      }
    ],
    "metric_query_label": "A",
    "threshold": { "operator": "greater_than", "value": 0.8, "frequency": "at_least_once" },
    "threshold_frequency": "at_least_once",
    "time_window_prometheus_format": "5m",
    "evaluation_interval_prometheus_format": "1m",
    "notificationChannels": [3],
    "routeByLabels": false,
    "labels": { "team": "platform" },
    "unit": "percent_unit",
    "no_data_state": "normal"
  }'

Top-level fields

Query objects

Each entry in query describes one data source. selectedMode is metrics, logs, traces, or formula.

Metrics — supply PromQL and put grouping in by (...):

{ "label": "A", "selectedMode": "metrics", "queryMode": "code",
  "promql": "sum by (pod)(rate(http_requests_total[5m]))",
  "visible": true, "fields": [] }

Logs / traces — aggregate with value_operation, group with groupBy, filter with rawFilters:

{ "label": "A", "selectedMode": "logs",
  "value_operation": "row_count",
  "groupBy": [{ "field": "workload", "type": "string", "is_attribute": false }],
  "rawFilters": { "level": ["ERROR"] },
  "fields": [] }

value_operation accepts row_count, unique_count, avg, sum, max, min, p99, p95, p90, p75, p50. For a numeric aggregation, name the column in fields — e.g. trace latency:

{ "label": "A", "selectedMode": "traces", "value_operation": "p95",
  "fields": [{ "field": "duration", "type": "float", "is_attribute": false }],
  "groupBy": [{ "field": "workload", "type": "string", "is_attribute": false }] }

Formula — combine other queries by label:

{ "label": "C", "selectedMode": "formula", "expression": "(B/A)*100", "visible": true }

Set metric_query_label to the formula's label, and visible: false on the helper queries so only the computed line is charted.

Update an alert rule

PUT /api/alerts/rules/{id} takes the same body as create — every field listed above.

curl -s -X PUT "https://<your-kubesense-host>/api/alerts/rules/$RULE_ID" \
  -H "X-API-Key: $KUBESENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d @rule.json

An unparseable UUID is 400; a UUID that matches no rule is 404.

Changing the query or the threshold takes effect on the rule's next evaluation — there is no separate reload call. A rule that is currently firing keeps its instances until the new condition resolves them.

Enable or disable a rule

curl -s -X POST "https://<your-kubesense-host>/api/alerts/rules/$RULE_ID/disable" \
  -H "X-API-Key: $KUBESENSE_API_KEY"

POST /{id}/enable and POST /{id}/disable flip evaluation without touching the rule's definition. Neither takes a body.

Disabling is the right way to silence a rule you intend to bring back — a maintenance window, a noisy rule pending a fix. It stops evaluation, so no new instances fire. Deleting is not reversible.

Delete an alert rule

DELETE /api/alerts/rules/{id}

curl -s -X DELETE "https://<your-kubesense-host>/api/alerts/rules/$RULE_ID" \
  -H "X-API-Key: $KUBESENSE_API_KEY"

Returns 200 with data: null. The deletion is recorded in the audit log against the key's user.

Bulk create

POST /api/alerts/rules/bulk — up to 500 rules per request, each element in the same shape as a single create.

curl -s -X POST "https://<your-kubesense-host>/api/alerts/rules/bulk" \
  -H "X-API-Key: $KUBESENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "dry_run": true, "rules": [ { "name": "…", "…": "…" } ] }'

The response reports each rule individually (index, name, valid, created, plus any errors), so a partially-invalid batch tells you exactly which entries failed rather than rejecting the lot silently.

Reading alert activity

Four read endpoints cover "what has this rule been doing" and "what is on fire right now". All of them are plain GETs needing only the alerts scope.

GET /api/alerts/rules/{id}/events — fire/resolve events for one rule, newest first. limit defaults to 50 and is capped at 1000.

GET /api/alerts/rules/{id}/alerts — the instances of that rule currently firing.

GET /api/alerts/rules/stats — counts by state across every rule, for a dashboard tile or a health check.

GET /api/alerts/events — global history across all rules:

GET /api/alerts/active — alert instances rather than events:

# Everything firing right now, with its assignee and ack state.
curl -s "https://<your-kubesense-host>/api/alerts/active?limit=50" \
  -H "X-API-Key: $KUBESENSE_API_KEY"

Both list endpoints return count — the total matching the filters — next to data.

List notification channels

GET /api/alerts/notification-channels returns every channel; GET /api/alerts/notification-channels/{id} returns one.

Create a notification channel

POST /api/alerts/notification-channels

curl -s -X POST "https://<your-kubesense-host>/api/alerts/notification-channels" \
  -H "X-API-Key: $KUBESENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "platform-slack",
    "type": "slack",
    "enabled": true,
    "config": {
      "webhook_url": "https://hooks.slack.com/services/T0/B0/xxx",
      "route_match": []
    }
  }'

config.route_match holds label matchers for label-based routing ([{"label":"severity","operator":"=","value":"critical"}]); leave it [] for direct assignment.

Update a notification channel

PUT /api/alerts/notification-channels/{id}

Unlike a rule update, this one is partial — send only the fields you want to change:

curl -s -X PUT "https://<your-kubesense-host>/api/alerts/notification-channels/3" \
  -H "X-API-Key: $KUBESENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "enabled": false }'

Changing type is allowed — converting a slack webhook channel to slack_app, say. The config is validated against the new type, so send a config that suits it (or make sure the stored one already does).

Delete a notification channel

DELETE /api/alerts/notification-channels/{id} — {id} is the integer id.

Check what still points at a channel before removing it: rules that referenced it by id in notificationChannels lose that destination.

Test a notification channel

POST /api/alerts/notification-channels/{id}/test sends a sample alert through a saved channel. POST /api/alerts/notification-channels/test does the same for a config you have not saved yet — useful in CI to prove a webhook or token works before committing it.

Both are write-gated, and both surface the provider's own failure reason (an invalid Slack channel, a rejected token) in the message field, so log the body on a 500 rather than just the status.

Errors

Errors use the same envelope as successes, with the reason in message:

{ "error": true, "message": "Alert rule not found", "data": null }