Kubernetes (OpenTelemetry Operator)

Zero-code instrumentation for Kubernetes workloads with the OpenTelemetry Operator.

The OpenTelemetry Operator instruments Kubernetes workloads without any code or image changes. You add one annotation to a workload. When its pods start, the operator's admission webhook injects the OpenTelemetry agent for that language and configures it to send traces.

This page sets up the shared components once per cluster. After that, see the language pages for the annotation to use.

Pod (annotated) ── agent injected by the Operator
      │  OTLP/HTTP :4318
      ▼
OpenTelemetry Collector (kubesense namespace)
  └─ adds kubesense.cluster, kubesense.env_type and Kubernetes metadata
      │  OTLP/HTTP
      ▼
kubesense-kubecol:33443 → KubeSense APM

Prerequisites

Trace ingestion enabled on KubeSense (kubecol.external_agent_traces.enabled: true). See Instrumentation.
Helm 3 and kubectl access to the cluster.
Optional: cert-manager installed in the cluster. Without it, the operator generates its own webhook certificate.

Step 1: Install the OpenTelemetry Operator

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Create operator-values.yaml:

manager:
  collectorImage:
    repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s
  extraArgs:
    # Required only if you want to instrument Go services
    - "--enable-go-instrumentation"

admissionWebhooks:
  certManager:
    enabled: false        # set to true if cert-manager is installed
  autoGenerateCert:
    enabled: true

Install it:

helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace kubesense \
  -f operator-values.yaml

Check that the operator is running:

kubectl get pods -n kubesense -l app.kubernetes.io/name=opentelemetry-operator

Step 2: Install the OpenTelemetry Collector

The collector receives traces from the injected agents, adds the KubeSense resource attributes and Kubernetes metadata, and forwards everything to KubeSense.

Create collector-values.yaml. Replace <CLUSTER_NAME> with the cluster name shown in KubeSense.

mode: deployment

image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib

command:
  name: otelcol-contrib

presets:
  kubernetesAttributes:
    enabled: true      # adds the RBAC the k8sattributes processor needs

resources:
  requests:
    cpu: 200m
    memory: 500Mi
  limits:
    cpu: "2"
    memory: 4Gi

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 15
    k8sattributes:
      passthrough: false
      extract:
        metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
      pod_association:
        - sources:
            - from: resource_attribute
              name: k8s.pod.ip
        - sources:
            - from: resource_attribute
              name: k8s.pod.uid
        - sources:
            - from: connection
    resource:
      attributes:
        - action: insert
          key: kubesense.cluster
          value: <CLUSTER_NAME>
        - action: insert
          key: kubesense.env_type
          value: k8s
    batch:
      send_batch_size: 10000
      timeout: 10s

  exporters:
    otlphttp/kubesense:
      endpoint: http://kubesense-kubecol.kubesense.svc.cluster.local:33443
      tls:
        insecure: true

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, resource, batch]
        exporters: [otlphttp/kubesense]

Install it:

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace kubesense \
  -f collector-values.yaml

The chart creates the Service otel-collector-opentelemetry-collector in the kubesense namespace. Confirm the name, because Step 3 uses it:

kubectl get svc -n kubesense | grep otel-collector

Step 3: Create the Instrumentation resource

The Instrumentation resource tells the operator how to configure the agents it injects. Create one in the kubesense namespace. Workloads in any namespace can reference it as kubesense/otel-instrumentation.

Create instrumentation.yaml:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-instrumentation
  namespace: kubesense
spec:
  exporter:
    endpoint: http://otel-collector-opentelemetry-collector.kubesense.svc.cluster.local:4318
  env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: http/protobuf
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"          # 1 = keep every trace; for example, 0.25 keeps 25%

kubectl apply -f instrumentation.yaml
kubectl get instrumentation -n kubesense

note: You can also create a separate Instrumentation resource in each application namespace, for example to use a different sampling rate. Workloads then reference it by name only, for example "otel-instrumentation" instead of "kubesense/otel-instrumentation".

Step 4: Annotate your workloads

Add the annotation for the language to the pod template (spec.template.metadata.annotations), not to the Deployment's own metadata:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-nodejs: "kubesense/otel-instrumentation"
    spec:
      containers:
        - name: app
          image: <YOUR_REGISTRY>/checkout-api:1.4.2

Language	Annotation	Additional annotations
Java	`instrumentation.opentelemetry.io/inject-java`	—
Node.js / NestJS	`instrumentation.opentelemetry.io/inject-nodejs`	—
.NET	`instrumentation.opentelemetry.io/inject-dotnet`	`instrumentation.opentelemetry.io/otel-dotnet-auto-runtime`: `linux-x64` (glibc, default) or `linux-musl-x64` (Alpine). See .NET on Kubernetes
Python	`instrumentation.opentelemetry.io/inject-python`	—
Go	`instrumentation.opentelemetry.io/inject-go`	`instrumentation.opentelemetry.io/otel-go-auto-target-exe`: full path of the binary in the container. Requires `--enable-go-instrumentation` (Step 1)

For pods with several containers, choose which containers to instrument:

instrumentation.opentelemetry.io/container-names: "app"

The webhook only modifies pods when they are created. Restart the workload after adding the annotation:

kubectl rollout restart deployment/checkout-api -n <namespace>

The service name defaults to the workload name, for example the Deployment name. To override it, set OTEL_SERVICE_NAME in the container's env.

Step 5: Verify

The agent was injected. The pod has an init container named opentelemetry-auto-instrumentation-<language> and OTEL_* environment variables:
```
kubectl describe pod <pod-name> -n <namespace> | grep -E 'opentelemetry-auto-instrumentation|OTEL_'
```

The collector receives and exports spans. No exporter errors appear:

kubectl logs -n kubesense deploy/otel-collector-opentelemetry-collector --tail=50

Traces appear in KubeSense. Generate some traffic, then open APM → Traces, select your cluster and filter by the service name.

Troubleshooting

Symptom	Check
No init container injected	The annotation must be on the pod template, and the pod must have been created after the annotation was added. Check the operator logs: `kubectl logs -n kubesense deploy/opentelemetry-operator`
`Instrumentation not found` in the operator logs	The annotation value must match `<namespace>/<name>` of an existing `Instrumentation` resource
Pod starts but no spans arrive	Check the collector logs. Confirm the exporter endpoint in the `Instrumentation` resource matches the collector Service name
Collector logs `connection refused` to `kubesense-kubecol:33443`	Trace ingestion is not enabled on KubeSense (`kubecol.external_agent_traces.enabled`)
Webhook errors when creating pods	Certificate issue. Use either `certManager.enabled: true` with cert-manager installed, or `autoGenerateCert.enabled: true`
Spans missing Kubernetes metadata	The `kubernetesAttributes` preset must be enabled, and `k8sattributes` must be in the traces pipeline

Removing instrumentation

Remove the annotation from the pod template and restart the workload. To remove everything:

kubectl delete instrumentation otel-instrumentation -n kubesense
helm uninstall otel-collector -n kubesense
helm uninstall opentelemetry-operator -n kubesense

Kubernetes (OpenTelemetry Operator)

ON THIS PAGE