Deployment

Synthetic Monitoring is added to a running KubeSense installation, on Kubernetes with Helm or on a VM with Docker Compose. It is not installed on its own.

Before You Start

• A KubeSense server installation — on Kubernetes, the kubesense or kubesense-server Helm chart with global.deploymentType set to incluster or server (a sensor-only installation has no place for it); on a VM, the KubeSense Docker Compose stack.
• Platform alerting turned on — global.alertconfig.enabled and global.metrics.enabled on Kubernetes; Alertmanager and the rule engine running on a VM. Monitor alerts use it.
• Pull access to the KubeSense registry you already pull KubeSense images from. Synthetic Monitoring adds these images (all three share one version, currently 1.0.0):

• Capacity for the defaults below. Browser agents are the heavy part: plan about 1 vCPU and 1 GiB of memory per browser test that runs at the same time.

• Network — agents must reach the endpoints you monitor, and agents outside the cluster or VM must reach synthetics-api over HTTPS (see Locations & Agents).

Install on Kubernetes (Helm)

One values file turns Synthetic Monitoring on in your existing release:

# values-synthetics.yaml
synthetics:
  enabled: true
  edition: basic
  browserAgent:
    enabled: false        # true to run browser tests (see the capacity table)

helm repo update
helm upgrade -i kubesense kubesense/kubesense -n kubesense \
  -f values-custom.yaml -f values-synthetics.yaml

Use the chart and values file you installed KubeSense with (kubesense-server for a server-only installation); values-synthetics.yaml only adds to them.

The upgrade deploys:

It also turns on the Synthetics pages in the web app, and the synthetics routes and alert rules in the KubeSense API.

Nothing else needs configuring. The release's own stores are used:

• a synthetics database on the bundled MySQL (or global.externalMySQL), created and migrated by synthetics-api on start;
• the kube_synthetics database on the bundled ClickHouse (or your external ClickHouse) for run results, uptime, failure snapshots and browser screenshots, whose tables the KubeSense database migrations create;
• the metrics store, which receives the kubesense_synthetics_* series that alerts and dashboards read.

An external MySQL user needs CREATE on the synthetics database, or create the database first.

Secrets — the token the KubeSense API uses for synthetics-api, the agent registration token, and the key that encrypts stored secrets such as secure variables — are generated on first install and kept in kubesense-app-secrets across upgrades. To manage them yourself, create a Secret with SYNTHETICS_API_TOKEN, SYNTHETICS_AGENT_TOKEN and SYNTHETICS_SECRET_KEYS, and set synthetics.existingSecret to its name.

Every other setting is described in Configuration.

Install on a VM (Docker Compose)

On a VM, Synthetic Monitoring is three more services in the KubeSense Compose stack, on the same Docker network as kubecoldb (MySQL), datastore (ClickHouse), victoriametrics and kubesense-api. Run the steps from the stack's directory.

Step 1 — Secrets

Create common/config/synthetics/synthetics.env. It holds secrets, so keep it out of version control:

# MySQL root password (kubecoldb) and ClickHouse default user password (datastore)
SYNTHETICS_MYSQL_PASSWORD=<kubecoldb password>
SYNTHETICS_CLICKHOUSE_PASSWORD=<datastore password>
# Token the KubeSense API presents to synthetics-api: openssl rand -hex 32
SYNTHETICS_API_TOKEN=<random>
# Token agents register with: openssl rand -hex 32
SYNTHETICS_AGENT_TOKEN=<random>
# Key that encrypts stored secrets: "k1:" + $(openssl rand -base64 32). Back it up.
SYNTHETICS_SECRET_KEYS=k1:<base64 key>

Both tokens are required: without either, synthetics-api does not start.

Then create common/config/synthetics/synthetics-agent.env for the agents:

SYNTHETICS_AGENT_TOKEN=<the same value as in synthetics.env>

Step 2 — Configuration

Create common/config/synthetics/synthetics-api.yaml:

listen_port: 8080
mysql:
  host: kubecoldb
  port: 3306
  database: synthetics   # created and migrated on start
  username: root
clickhouse:
  host: datastore
  port: 9000
  database: synthetics
  username: default
migrate: true
remote_write:
  url: http://victoriametrics:8428/api/v1/write

Step 3 — Services

Add these to docker-compose.yaml, with <registry> the KubeSense registry you pull from and <version> the Synthetic Monitoring release:

  synthetics-api:
    image: <registry>/synthetics/api:<version>
    container_name: synthetics-api
    restart: always
    command: ["-config", "/etc/synthetics/synthetics-api.yaml"]
    env_file:
      - ./common/config/synthetics/synthetics.env
    environment:
      SYNTHETICS_EDITION: basic
      SYNTHETICS_ALERTING_MODE: platform
      # Monitor state changes also appear in the KubeSense Events explorer.
      SYNTHETICS_EVENTS_URL: http://kubesense-api:3000/internal/synthetics/events
    volumes:
      - ./common/config/synthetics/synthetics-api.yaml:/etc/synthetics/synthetics-api.yaml:ro
    depends_on: [kubecoldb, datastore, victoriametrics]
    networks: [kubesense]

  synthetics-agent:
    image: <registry>/synthetics/agent:<version>
    container_name: synthetics-agent
    restart: always
    command:
      - -api=http://synthetics-api:8080
      - -id=vm-agent-1
      - -name=vm-agent-1
      - -location=vm
      - -token-file=/var/lib/synthetics-agent/probe-token
    env_file:
      - ./common/config/synthetics/synthetics-agent.env
    volumes:
      - ./data/synthetics-agents/agent:/var/lib/synthetics-agent
    depends_on: [synthetics-api]
    networks: [kubesense]

For browser tests, add a browser agent as well (about 1 vCPU and 1 GiB per concurrent run):

  synthetics-agent-browser:
    image: <registry>/synthetics/browser-agent:<version>
    container_name: synthetics-agent-browser
    restart: always
    command:
      - -api=http://synthetics-api:8080
      - -id=vm-browser-1
      - -name=vm-browser-1
      - -location=vm
      - -browser
      - -browser-concurrency=2
      - -chrome-no-sandbox
      - -token-file=/var/lib/synthetics-agent/probe-token
    env_file:
      - ./common/config/synthetics/synthetics-agent.env
    volumes:
      - ./data/synthetics-agents/browser:/var/lib/synthetics-agent
    tmpfs:
      - /tmp:size=1g,mode=1777
    shm_size: 1gb
    mem_limit: 3g
    depends_on: [synthetics-api]
    networks: [kubesense]

synthetics-api needs no published port: the KubeSense API and the agents reach it on the Docker network. Publish 8080 only to put a reverse proxy in front of it for agents on other machines (see Locations & Agents).

Step 4 — Connect KubeSense

Add to the kubesense-api service, with SYNTHETICS_TOKEN set to the SYNTHETICS_API_TOKEN value from step 1:

    environment:
      SYNTHETICS_ENABLED: "true"
      SYNTHETICS_BASE_URL: http://synthetics-api:8080
      SYNTHETICS_ALERTS_PLATFORM_REFS: passthrough
      SYNTHETICS_TOKEN: <SYNTHETICS_API_TOKEN>

and ENABLE_SYNTHETICS: "true" to the web app's environment, so the Synthetics pages appear.

Step 5 — Start

docker compose up -d synthetics-api
# then recreate the KubeSense API and web app services so they pick up the new environment
docker compose up -d kubesense-api <web app service>

Then, in KubeSense under Settings → Synthetics → Locations & agents, add a location named vm. Its id is made from the name, and the agents join it with -location=vm. Start the agents:

docker compose up -d synthetics-agent      # and synthetics-agent-browser

They register under the vm location within a minute. The token file keeps their registration across restarts, so a restarted agent is back at once.

Check the Installation

A healthy installation has every synthetics pod or container running, an API log that shows its database schema up to date, and agents online under their location.

kubectl -n kubesense get pods -l app.kubernetes.io/part-of=synthetics
kubectl -n kubesense logs deploy/kubesense-synthetics-api | grep -E 'mysql schema|telemetry: clickhouse|tables missing|ERROR'
kubectl -n kubesense logs kubesense-synthetics-agent-0 -c ensure-location

docker compose ps synthetics-api synthetics-agent
docker logs synthetics-api 2>&1 | grep -E 'mysql schema|telemetry: clickhouse|tables missing|ERROR'

The MySQL schema line reads "mode":"upgrade" on the first start and "mode":"up-to-date" after that, and telemetry: clickhouse shows the results store is connected. A line kube_synthetics tables missing means the KubeSense database migrations have not run yet: on Kubernetes they run right after the pods start, so it can show briefly during an install or upgrade. Until they have run, runs are not stored; agents keep them and send them again.

In KubeSense:

1. Settings → Synthetics → Locations & agents — the in-cluster (or vm) location lists its agents as online.
2. Synthetics → Monitors → New monitor — create an HTTP check of a URL you know answers, pick the location, and use Test before saving. The test result shows the status code and timings.
3. After its first scheduled runs, the monitor shows uptime and response times, and Synthetics → Overview counts it.

Upgrades and Backups

Upgrades. On Kubernetes, a helm upgrade to a newer KubeSense chart brings the Synthetic Monitoring images of that release. On a VM, change the image tags of the synthetics services and run docker compose up -d. In both cases synthetics-api migrates its MySQL database on start, and monitors, results and secrets are kept. Upgrade the API before agents that run elsewhere: agents older than the API keep working, and the Locations page shows each agent's version.

Back up the secrets key. SYNTHETICS_SECRET_KEYS encrypts secure variables and monitor credentials. Without it they cannot be decrypted, so store a copy somewhere safe. On Kubernetes it is in the kubesense-app-secrets Secret, or in your own Secret when you use synthetics.existingSecret. On a VM it is in synthetics.env.

Monitors, locations and settings live in the synthetics MySQL database, and run history in the kube_synthetics ClickHouse database. Both are covered by your usual KubeSense database backups.

Upgrading from 1.0. Earlier releases kept run history in a ClickHouse database of their own (synthetics). After the upgrade, synthetics-api copies it into kube_synthetics in the background: the last 30 days of runs, a year of daily uptime, and the failure snapshots and screenshots still kept. Monitors keep their history; the copy resumes by itself after a restart. An hour after the copy finishes, the old database is removed.