External Service Logs & Metrics

Overview

Managed SaaS platforms, third-party APIs, and applications running on VMs sit outside your Kubernetes cluster, so no KubeSense agent can be installed next to them. These services instead push their telemetry to an endpoint you provide, or expose it for you to pull and forward.

KubeSense exposes two ingestion surfaces for this. Both are ordinary Kubernetes Services, so you publish them by turning them into internal load balancers (an NLB on AWS):

Part 1 — Expose the endpoints

Aggregator (logs)

Set the aggregator service to LoadBalancer and add your cloud's annotations:

aggregator:
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-aaa, subnet-bbb, subnet-ccc
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled="true"
  sources:
    otel:
      enabled: true       # OTLP logs  — off by default
    file_logs:
      enabled: true       # JSON HTTP  — off by default

Metrics-scraper (metrics)

The metrics-scraper Service is disabled by default and must be switched on as well as exposed:

metrics-scraper:
  service:
    enabled: true         # false by default
    type: LoadBalancer
    servicePort: 30060
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-aaa, subnet-bbb, subnet-ccc

Apply with:

helm upgrade kubesense kubesense/kubesense \
  -f values.yaml --namespace kubesense

Then collect the two hostnames:

kubectl -n kubesense get svc kubesense-aggregator kubesense-metrics-scraper \
  -o custom-columns=NAME:.metadata.name,LB:.status.loadBalancer.ingress[0].hostname

Part 2 — Pushing logs

Option A — OTLP logs (recommended)

Enable aggregator.sources.otel and send OTLP to the NLB:

For a service outside Kubernetes, the aggregator reads these resource attributes to attribute the log:

Severity comes from the OTLP severity_text field; when it is empty, KubeSense falls back to regex-matching the message body for a level.

A collector forwarding logs from an external service would be configured like this:

processors:
  resource/kubesense:
    attributes:
      - { key: kubesense.env_type, value: external,           action: upsert }
      - { key: kubesense.cluster,  value: prod-observability,  action: upsert }
      - { key: service.name,       value: payments-api,        action: upsert }
      - { key: service.namespace,  value: billing,             action: upsert }

exporters:
  otlphttp/kubesense:
    logs_endpoint: http://<aggregator-nlb>:30055/v1/logs
    tls: { insecure: true }

service:
  pipelines:
    logs:
      processors: [resource/kubesense]
      exporters: [otlphttp/kubesense]

Option B — File log (plain JSON POST)

Enable aggregator.sources.file_logs and POST JSON to http://<aggregator-nlb>:30054. The body may be a single JSON object or an array of objects, in which case each element becomes one log record.

The aggregator maps a fixed set of keys:

Records ingested this way are always tagged source = "file" and env_type = "legacy".

curl -X POST http://<aggregator-nlb>:30054 \
  -H 'Content-Type: application/json' \
  -d '[
        {
          "timestamp": "2026-07-23T10:15:00.000Z",
          "service": "payments-api",
          "application": "billing",
          "host": "vm-payments-01",
          "message": "WARN upstream connection reset, retrying in 2s"
        }
      ]'

Part 3 — Pushing Prometheus metrics

The metrics-scraper is a vmagent instance. Besides scraping, it accepts pushed metrics on two paths:

Remote write

Point any Prometheus-compatible sender at the metrics-scraper NLB:

remote_write:
  - url: http://<metrics-scraper-nlb>:30060/api/v1/write

From an OpenTelemetry Collector:

exporters:
  prometheusremotewrite/kubesense:
    endpoint: http://<metrics-scraper-nlb>:30060/api/v1/write
    tls: { insecure: true }
    external_labels:
      cluster: prod-observability

Pushing exposition-format text

If you can already fetch a Prometheus-format payload, forward it unchanged:

curl -s https://<source>/metrics \
  | curl -X POST --data-binary @- \
      'http://<metrics-scraper-nlb>:30060/api/v1/import/prometheus?extra_label=cluster=prod-observability'

The extra_label parameter attaches a label to every series in the request — use it to tag the origin, since pushed metrics carry no cluster attribution of their own.

Alternative — let the metrics-scraper pull instead

Many providers do not push at all. They expose a metrics export endpoint that already returns OpenMetrics/Prometheus format and expects you to poll it, usually behind an API key. When the endpoint is reachable from the cluster, scraping it is simpler than building a push relay — and needs no load balancer at all:

metrics-scraper:
  extraScrapeConfigs:
    - job_name: external-provider
      scrape_interval: 60s
      scheme: https
      metrics_path: /v1/metrics/export
      params:
        resource_id: ["<RESOURCE_ID>"]
      basic_auth:
        username: <API_KEY>
        password: <API_SECRET>
      static_configs:
        - targets: ["metrics.provider.example.com"]

Verification

• kubectl -n kubesense get svc kubesense-aggregator shows an external address
• The sender can reach the port: nc -vz <aggregator-nlb> 30055
• Logs appear in the Log Explorer, filtered by the service.name or service value you set
• The cluster and env_type values are populated, not blank
• Timestamps reflect event time, not ingest time — a wall of identical timestamps means the timestamp field was not parsed
• Metrics appear in the Data Explorer; confirm the series name and that your cluster label is attached
• Check for records the pipeline rejected: kubectl -n kubesense logs -l app=aggregator | grep -i dropped

Troubleshooting

Best practices

• Keep both load balancers internal and source-range restricted; neither endpoint authenticates callers.
• Set kubesense.cluster and kubesense.env_type on every external source so the data is filterable and does not blend into in-cluster telemetry.
• Give each external source a distinct service.name — it becomes the workload in the UI and is the field you will filter on.
• External sources often need a different retention policy from in-cluster logs — some are low-volume but must be kept for a long time. Set this deliberately in Settings → Data Retention.
• Watch cardinality after adding a push source. Vendor metrics often carry high-cardinality identifiers; review Settings → Metrics afterwards.