Alert Rules

The Alert Config page lets you create, manage, and monitor alert rules that evaluate conditions on your metrics, logs, and traces data.

Alert Rules Table

All configured alerts are listed with their current status:

---

Creating an Alert Rule

Click Add Alert to open the alert configuration page. There are four sections to configure.

Step 1 — Define Alert Query

Build the data query that the alert evaluates. This uses the same query builder available in the Data Explorer.

Click Run to preview the query result as a time-series chart. The red dashed threshold line shows where your alert condition will trigger.

Multiple Queries and Formulas

You can add multiple queries using Add Query (each labeled A, B, C, etc.) and combine them with Add Formula to create computed conditions. For example:

• Query A = total request count (logs)
• Query B = error request count (logs)
• Formula C = B / A * 100 (error rate percentage)

The alert condition can then evaluate the formula result.

Step 2 — Alert Conditions

Choose the condition type and configure when the alert should fire.

Threshold Alert

Fires when the query value crosses a fixed threshold.

Firing frequency controls how many breaches are required before the alert fires:

When using more than once, you specify both the number of required breaches and the time window over which to count them. For example, "for 2 times within 5 Minutes" means the condition must be breached more than 2 times in the last 5-minute window before the alert fires.

When using always, the alert fires only if every single evaluation within the breach counting window resulted in a breach — useful for avoiding false positives from transient spikes.

Change Alert

Fires when the query value changes by a specified amount relative to a previous time period.

Use change alerts to detect sudden spikes or drops relative to recent behavior. For example, "fire when error count changes by more than 50% compared to 5 minutes ago."

New Value Alert

Fires when a value appears that was not seen in the comparison window.

Use new value alerts to detect previously unseen patterns — new error types, new endpoints, new log levels, or any other unexpected values appearing in your data.

Step 3 — Alert Configuration

Step 4 — Alert Routing

Choose how notifications are delivered when the alert fires.

• Notification Channel — Select a specific notification channel (Slack, Teams, Email, etc.) to receive this alert
• Notification Channel Policy — Route alerts dynamically based on label matching. Alerts are sent to channels whose matching labels correspond to the alert's labels.

---

Alert Lifecycle

States

State Transitions

• Normal → Firing — The condition is met and the firing frequency requirement is satisfied. A notification is sent and an alert event is recorded.
• Firing → Normal — The condition is no longer met (for at_least_once: no breaches in the breach counting window; for more_than_once: breach count drops below the threshold; for always: at least one evaluation did not breach). A resolved notification is sent.

Resolved Notifications

When an alert transitions from firing back to normal, a resolved notification is automatically sent to all configured channels, so your team knows the issue has been addressed.

---

Multi-Series Alerts

When a query uses Group By, the alert evaluates each group independently. For example, if you group by workload and three workloads breach the threshold, you receive three separate alert notifications — each with the specific workload name and value in the labels.

This is useful for scenarios like:

• Error count per workload exceeding a threshold
• Latency per namespace crossing an SLO boundary
• Request rate per service dropping below expected levels

---

Example Alert Rules