Terraform

The KubeSense Terraform provider lets you keep your monitoring setup in code: reviewed in pull requests, versioned in git, and the same in every environment. Anything you already built in the UI can be exported to Terraform and brought under management without recreating it.

What you can manage

Two data sources, kubesense_dashboard and kubesense_integration, read items Terraform does not manage, such as system dashboards and GitHub.

Step 1: Install Terraform

You need Terraform 1.5 or later. Exported code uses import blocks, which older versions do not support.

macOS (Homebrew):

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Linux (Ubuntu or Debian):

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

Windows (Chocolatey):

choco install terraform

For other systems, download the binary from developer.hashicorp.com/terraform/install and put it on your PATH.

Check that it works:

terraform -version

Step 2: Create a service account key

Terraform signs in with a service account key, not a personal login.

1. Go to Settings → Service Accounts and create a service account. Give it a role with write access to what you will manage. Only administrators can do this.
2. Open the account's ⋮ menu → Manage keys → Create key.
3. Turn on the scopes you need at Editor: Dashboard for dashboards and lists, Alerts for alert rules, Integrations for integrations and cloud accounts.
4. Click Generate key and copy it. It is shown only once.

The Use with Terraform section under the new key shows the provider block for your deployment.

Step 3: Configure the provider

Create a folder for your configuration with a main.tf:

terraform {
  required_providers {
    kubesense = {
      source = "kubesense-ai/kubesense"
    }
  }
}

provider "kubesense" {
  api_url = "https://<your-kubesense-host>"
  # The API key comes from KUBESENSE_API_KEY.
}

Keep the key out of the file. Set it in the shell, or in your CI as a secret:

export KUBESENSE_API_KEY=<your-key>

api_url can also come from KUBESENSE_API_URL. Then install the provider:

terraform init

Step 4: Export what you already have

Every dashboard, alert rule, integration and cloud account can be exported as ready-to-use Terraform.

For dashboards and alert rules, the Export window has two tabs, JSON and Terraform. Open Terraform, then Copy or Download (.tf), and add it to your folder.

Secrets such as API keys and client secrets are never exported. They appear as Terraform variables for you to fill in, for example in a terraform.tfvars file or with TF_VAR_<name> in the environment.

Step 5: Apply to take it over

Exported code starts with an import block that carries the item's ID:

# Manages the existing one: the first apply adopts it instead of creating a copy.
# Delete this block to create a copy instead.
import {
  to = kubesense_alert_rule.cpu_high
  id = "7c9e6679-7425-40de-944b-e07fc1f90ae7"
}

resource "kubesense_alert_rule" "cpu_high" {
  # ...
}

Run a plan. It should only import:

terraform plan

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.

Apply it, and the existing item is now managed by Terraform:

terraform apply

A plan straight after reports No changes. You can keep the import block or delete it once applied. To create a copy instead (in another environment, say), delete the import block before the first apply.

Step 6: Change it in code

Edit an attribute in the .tf file, for example an alert's threshold:

threshold_value = 95

Then review and apply:

terraform plan    # shows 1 to change
terraform apply

The item is updated in place, and the change shows up in KubeSense.

Step 7: Catch changes made in the UI

If someone edits a Terraform-managed item in the UI, the next terraform plan shows the difference. terraform apply puts it back to what the code says. Settle on one place to make changes so the two do not fight.

Writing resources by hand

You do not have to start from an export. Leave out what you do not need; KubeSense fills its defaults.

Alert rule

resource "kubesense_alert_rule" "cpu_high" {
  name                       = "cpu high"
  severity                   = "critical"
  metric_query_label         = "A"
  threshold_operator         = "greater_than"
  threshold_value            = 80
  time_window                = "5m"
  evaluation_interval        = "1m"
  notification_channel_names = ["slack-oncall"]

  query_config {
    label         = "A"
    query_mode    = "code"
    selected_mode = "metrics"
    promql        = "avg(rate(container_cpu_usage_seconds_total[5m]))"
  }
}

A rule needs a notification channel, or route_by_labels = true, unless it is disabled (enabled = false).

Dashboard

resource "kubesense_dashboard" "checkout" {
  name        = "Checkout"
  description = "Golden signals for checkout"

  grid_layout {
    i = "p1"
    x = 0
    y = 0
    w = 12
    h = 8
  }

  panel {
    name       = "CPU by pod"
    panel_type = "timeSeries"

    query {
      label         = "A"
      query_mode    = "code"
      selected_mode = "metrics"
      promql        = "sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)"
    }
  }
}

Settings that have no block of their own are *_json attributes. Set them with jsonencode(), for example aggregation_json = jsonencode({ function = "row_count" }).

Integration

resource "kubesense_integration" "jira" {
  integration_provider = "jira"
  name                 = "jira-prod"
  api_key              = var.jira_api_token
  config = {
    deployment = "cloud"
    base_url   = "https://acme.atlassian.net"
    username   = "kubesense-bot@acme.com"
  }
}

variable "jira_api_token" {
  type      = string
  sensitive = true
}

Cloud account

resource "kubesense_aws_account" "prod" {
  account_id = "123456789012"
  mode       = "role_access"
  role_arn   = "arn:aws:iam::123456789012:role/KubeSenseCollectorRole"

  resource_types = {
    ec2 = { collect = true, metric_enabled = true }
    rds = { collect = true, metric_enabled = true, metric_tier = "expanded" }
  }
}

Many alike items from one block

Use for_each to create several rules (or dashboards) from one definition:

locals {
  cpu_limits = { api = 80, worker = 90 }
}

resource "kubesense_alert_rule" "cpu" {
  for_each = local.cpu_limits

  name                = "cpu high: ${each.key}"
  severity            = "warning"
  route_by_labels     = true
  labels              = { component = each.key }
  metric_query_label  = "A"
  threshold_operator  = "greater_than"
  threshold_value     = each.value
  time_window         = "5m"
  evaluation_interval = "1m"

  query_config {
    label         = "A"
    query_mode    = "code"
    selected_mode = "metrics"
    promql        = "avg(rate(container_cpu_usage_seconds_total{container=\"${each.key}\"}[5m]))"
  }
}

Good to know

• Changes are checked before they are saved. Dashboards and alert rules from Terraform are held to the same rules as the editor, strictly. An invalid one is refused with a message saying what is wrong, and dashboard problems show up already at terraform plan.

• Secrets are write-only. API keys and cloud secrets are never read back, so a change to them made outside Terraform is not detected, and they are not set on import.

• Integrations test the connection. Creating an integration, or changing its credentials, makes KubeSense connect to it first. Apply from somewhere KubeSense can reach the target, with working credentials.

• resource_types on cloud accounts. When set, Terraform owns collection: types switched on in the UI but not listed are switched off on the next apply. When left out, collection is left alone.

• dashboard_ids on dashboard lists. When set, Terraform owns membership: dashboards added in the UI are removed from the list on the next apply. When left out, membership is left alone.

• System dashboards are read-only. Their export is a data source you can read and build your own dashboard from:

  data "kubesense_dashboard" "ec2" {
    name = "AWS EC2 Overview"
  }

• GitHub connects only through the UI. Read it with the kubesense_integration data source.

Stop managing without deleting

terraform destroy, or removing a resource from the code, deletes the item in KubeSense. To stop managing an item but keep it, remove it from state instead:

terraform state rm kubesense_alert_rule.cpu_high

Troubleshooting