Troubleshooting
Installation
| Symptom | Cause | Fix |
|---|---|---|
| No Synthetics menu in KubeSense | The web app or the KubeSense API does not have Synthetic Monitoring enabled, or your role has no Synthetics access | Kubernetes: check synthetics.enabled: true and that the upgrade finished. VM: set ENABLE_SYNTHETICS on the web app and SYNTHETICS_ENABLED on the API, then recreate both. Check the role's Synthetics permissions |
| Synthetics pages show an error or service unavailable | The KubeSense API cannot reach synthetics-api, or the tokens differ | Check SYNTHETICS_BASE_URL. SYNTHETICS_TOKEN on the API must equal SYNTHETICS_API_TOKEN on synthetics-api |
No new runs; the synthetics-api log shows kube_synthetics tables missing | The KubeSense database migrations have not run on this ClickHouse | Kubernetes: check the db-migrations job of the release and rerun the upgrade if it failed. VM: run the KubeSense migrations again |
synthetics-api restarts; its log shows store init failed | Wrong database credentials, or the MySQL user cannot create the synthetics database | Check the MySQL and ClickHouse passwords; give the user CREATE, or create the database first |
The log shows Error 1129 ... is blocked because of many connection errors | MySQL blocked synthetics-api's address after repeated failed connections | Fix the credentials, then run mysqladmin flush-hosts on the MySQL server |
synthetics-api exits at startup; its log says -api-token and -agent-token are required | SYNTHETICS_API_TOKEN or SYNTHETICS_AGENT_TOKEN is empty (the log names which) | VM: set both in synthetics.env and recreate synthetics-api. Kubernetes: with synthetics.existingSecret, add both keys to that Secret |
| synthetics-api exits at startup naming a config key | An unknown key or a wrong value in configFile / synthetics-api.yaml | Fix the key named in the log. Durations take s, m or h |
Agents and Locations
| Symptom | Cause | Fix |
|---|---|---|
| An agent never appears under its location | Wrong or revoked token, the location does not exist, or the agent cannot reach the API | Check the agent's log; generate a new token for the location; check the agent endpoint (ingress or proxy) from the agent's machine |
| The install commands say they work only inside the cluster | The agent ingress is off | Enable synthetics.agentIngress (Kubernetes) or a reverse proxy (VM) — see Private Locations |
| A restarted agent takes about a minute to come back | Its registration is not kept, so it waits for the old one to expire | Keep the token file on a volume (-token-file on a mounted directory; synthetics.agent.persistence.enabled in the KubeSense chart) |
A location shows Offline or Degraded | Agents stopped or cannot reach synthetics-api | Check the agents' pods or containers and their logs |
An agent is marked outdated | It is older than the installation's minimum agent version | Upgrade the agent (helm upgrade, or a new image tag) |
| Ping checks fail with a permission error | The agent lacks NET_RAW | enableICMP=true (Helm) or --cap-add NET_RAW (Docker) |
Internal targets fail with SSRF_BLOCKED | The agent's egress rules refuse the address | Check allowedDomains, blockedDomains and the IP range settings of the agent |
| Kubernetes checks never run | No agent with k8sChecks=true in the monitor's locations | Install one with the synthetics-agent chart |
Browser Tests
| Symptom | Cause | Fix |
|---|---|---|
| Browser tests never run | No browser agent in the monitor's locations | Enable browserAgent (Kubernetes) or add the browser agent (VM), or pick a location that has one |
| The browser agent restarts or runs out of memory | Too many concurrent browser runs for its memory | Lower the concurrency, or give it more memory and shared memory (shm_size on a VM) |
| Steps fail with element not found although the page looks right | The selector changed, or the page loads slowly | Open the run's Journey: compare the screenshot and the page's HTML; add alternative selectors or raise the step timeout |
Monitors and Alerts
| Symptom | Cause | Fix |
|---|---|---|
A monitor stays at No data | No online agent in its locations runs it (for a browser test, no browser agent) | Check the locations' agents under Locations & agents |
| Monitors fail but nobody is notified | No notification channel on the monitor and no default channels, or a maintenance window mutes it | Pick channels in the monitor's alert settings or set defaults under Settings → Synthetics → Defaults & usage; check the monitor's Properties → Maintenance windows in effect |
| Too many alerts for short blips | Thresholds too low for the target | Raise Failure threshold, set Alert only after failing for, use Fast retries or Flap detection, or raise the Location threshold |
| Saving a monitor fails with a tag error | The tag policy is enforced | Add the required labels shown in the banner of the first step |
| Saving fails with a quota error | A limit in Defaults & usage or the installation is reached | Raise the interval or remove locations, or ask an administrator to change the limit |
| Secure variables or monitor credentials fail after a reinstall | SYNTHETICS_SECRET_KEYS changed, so stored secrets cannot be decrypted | Restore the previous key from your backup |
| Run now or Test answers too many requests | Too many on-demand runs at once | Wait a moment; the limit is Max concurrent on-demand runs |