Python

Send OpenTelemetry traces and metrics from Python applications (Flask, Django, FastAPI, Celery) to KubeSense.

Python applications are instrumented with OpenTelemetry Python auto-instrumentation. The opentelemetry-instrument launcher wraps your start command and creates spans for Flask, Django, FastAPI, requests, urllib3, database drivers and many other libraries. You don't change application code.

Option 1: Kubernetes with the OpenTelemetry Operator (zero code)

If you have set up the OpenTelemetry Operator, add this annotation to the pod template and restart the workload:

spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "kubesense/otel-instrumentation"

For images based on Alpine (musl), also add:

        instrumentation.opentelemetry.io/otel-python-platform: "musl"

Option 2: Auto-instrumentation on VMs (no code changes)

Use this on VMs, or on Kubernetes clusters without the operator. You only install packages and change the start command; the application code is not changed.

Step 1: Install the packages

On a VM, run these commands in the application's Python environment (without RUN). For a container image, add them to your Dockerfile after your application dependencies are installed:

RUN pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http

# Detects the libraries your app uses (Flask, requests, ...) and installs their instrumentations
RUN opentelemetry-bootstrap -a install

Package	Purpose
`opentelemetry-distro`	OpenTelemetry SDK, the `opentelemetry-instrument` launcher and default configuration
`opentelemetry-exporter-otlp-proto-http`	Exports over OTLP/HTTP, which KubeSense port `33443` uses

Run opentelemetry-bootstrap after installing your own requirements, so it can see which libraries you use.

Step 2: Start the application with `opentelemetry-instrument`

Prefix your start command:

# Before
CMD ["python", "app.py"]

# After
CMD ["opentelemetry-instrument", "python", "app.py"]

Other servers work the same way:

opentelemetry-instrument gunicorn -b 0.0.0.0:8000 app:app
opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000
opentelemetry-instrument python manage.py runserver --noreload

The launcher is required: If the application starts with plain python app.py, nothing is instrumented. Auto-reloaders also break instrumentation: Flask debug mode, Django runserver without --noreload, and uvicorn --reload. Disable them where you want traces.

Step 3: Configure the exporter

Traces only, directly to KubeSense:

OTEL_SERVICE_NAME=orders-api
OTEL_EXPORTER_OTLP_ENDPOINT=http://<KUBESENSE_INGEST_HOST>:33443
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=kubesense.cluster=<CLUSTER_NAME>,kubesense.env_type=legacy
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none

Inside the KubeSense cluster, use http://kubesense-kubecol.kubesense.svc.cluster.local:33443 and kubesense.env_type=k8s.

Traces and metrics, through an OpenTelemetry Collector:

env:
  - name: OTEL_SERVICE_NAME
    value: orders-api
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector-opentelemetry-collector.kubesense.svc.cluster.local:4318
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: http/protobuf
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: kubesense.cluster=<CLUSTER_NAME>,kubesense.env_type=k8s
  - name: OTEL_TRACES_EXPORTER
    value: otlp
  - name: OTEL_METRICS_EXPORTER
    value: otlp
  - name: OTEL_LOGS_EXPORTER
    value: none          # logs are collected by the KubeSense log sensor

The collector must have a metrics pipeline, not only traces. If you use the collector from the Operator guide, add this exporter and pipeline to its config:

  exporters:
    prometheusremotewrite/kubesense:
      endpoint: http://kubesense-metrics-scraper.kubesense.svc.cluster.local:30060/api/v1/write
      resource_to_telemetry_conversion:
        enabled: true

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, resource, batch]
        exporters: [prometheusremotewrite/kubesense]

To also export host CPU, memory and process metrics, install opentelemetry-instrumentation-system-metrics. It is picked up automatically.

Step 4: Verify

Generate some traffic, then open APM → Traces in KubeSense and filter by your service name.

What is captured

Signal	Captured automatically
Traces	Incoming HTTP requests (Flask, Django, FastAPI), Celery task publishing and execution, outgoing HTTP calls (`requests`, `urllib3`, `httpx`), database calls (for supported drivers), exceptions
Metrics	HTTP server metrics; with system metrics installed, process and host CPU and memory

Celery (background tasks)

Celery workers are instrumented like any other Python process. When both the application that sends tasks (for example your Django or Flask API) and the worker are instrumented, the trace context travels with the task message. A request and the tasks it triggers then appear as one trace: the API request, the apply_async (publish) span and the worker's run span.

The Celery instrumentation creates spans for:

publishing a task (apply_async / delay) in the sending application
running a task in the worker, with the task name, ID and state
retries and failures, with the exception recorded on the span

Kubernetes with the OpenTelemetry Operator (zero code)

Annotate the worker Deployment in the same way as the web application, and give the worker its own service name:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-worker
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "kubesense/otel-instrumentation"
    spec:
      containers:
        - name: worker
          image: <YOUR_REGISTRY>/orders:1.0.0
          command: ["celery", "-A", "orders", "worker", "--loglevel=info"]
          env:
            - name: OTEL_SERVICE_NAME
              value: orders-worker

Annotate the sending application too, so the trace continues from the request into the task. If you run Celery beat, annotate it as well; it then creates a span for every scheduled task it publishes.

VM: auto-instrumentation (no code changes)

Install the packages in the worker's environment. opentelemetry-bootstrap adds opentelemetry-instrumentation-celery when Celery is installed:

pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http
opentelemetry-bootstrap -a install

Start the worker through opentelemetry-instrument, with the same environment variables as in Step 3 and a worker-specific service name:

export OTEL_SERVICE_NAME=orders-worker
opentelemetry-instrument celery -A orders worker --loglevel=info

Start Celery beat the same way, for example with OTEL_SERVICE_NAME=orders-beat.

For a systemd service, prefix ExecStart in the same way:

ExecStart=/opt/orders/venv/bin/opentelemetry-instrument /opt/orders/venv/bin/celery -A orders worker --loglevel=info

Manual SDK (Celery workers)

Celery's default worker pool (prefork) runs tasks in child processes that it starts by forking the main worker process. Set up the SDK once in each worker process, with Celery's worker_process_init signal.

Install the Celery instrumentation in addition to the packages in Manual instrumentation (SDK):

pip install opentelemetry-instrumentation-celery

In your Celery app module, for example orders/celery.py, use the setup_telemetry() function from Step 2:

from celery import Celery
from celery.signals import worker_process_init
from opentelemetry.instrumentation.celery import CeleryInstrumentor

from telemetry import setup_telemetry  # import at the top of the module

app = Celery("orders")

@worker_process_init.connect(weak=False)
def init_telemetry(*args, **kwargs):
    setup_telemetry()                 # runs once in every worker process
    CeleryInstrumentor().instrument()

Import at the top of the module: Import setup_telemetry at the top of the module, as shown, not inside the signal handler. Celery adds your project directory to Python's import path only while it loads the app, so the module may not be importable later in the worker processes.

In the application that sends tasks, call both once at startup, so the trace context is added to each task message:

from opentelemetry.instrumentation.celery import CeleryInstrumentor
from telemetry import setup_telemetry

setup_telemetry()
CeleryInstrumentor().instrument()

Troubleshooting Celery

Symptom	Check
API traces appear, but no task spans	The worker isn't instrumented. Check the annotation on the worker Deployment, or that the worker is started with `opentelemetry-instrument`
Task spans appear as separate traces, not linked to the request	The sending application isn't instrumented, so no trace context is added to the task message. Instrument both sides
Spans missing after upgrading Celery	Upgrade `opentelemetry-instrumentation-celery` to a version that supports your Celery version, then run `opentelemetry-bootstrap -a install` again

Optional: libraries without auto-instrumentation (example: ScyllaDB / Cassandra)

If a library has no instrumentation, its calls don't appear as spans. For ScyllaDB and Cassandra, first try the contrib package, which supports cassandra-driver and scylla-driver:

pip install opentelemetry-instrumentation-cassandra

If that doesn't fit your driver version, wrap the session's execute method so that every query creates a span:

from opentelemetry import trace

_tracer = trace.get_tracer("scylla.db")

def instrument_session(session, *, keyspace: str) -> None:
    """Create a span for every Session.execute call."""
    if getattr(session, "_otel_execute_wrapped", False):
        return  # already instrumented

    original_execute = session.execute

    def execute_with_span(query, parameters=None, *args, **kwargs):
        statement = str(query)
        with _tracer.start_as_current_span("scylla.execute") as span:
            span.set_attribute("db.system", "scylladb")
            span.set_attribute("db.name", keyspace)
            span.set_attribute("db.statement", statement)
            span.set_attribute("db.operation", statement.split(" ", 1)[0].upper() if statement else "EXECUTE")
            return original_execute(query, parameters, *args, **kwargs)

    session.execute = execute_with_span
    session._otel_execute_wrapped = True

Apply it right after the connection is created:

session = cluster.connect()
instrument_session(session, keyspace=keyspace)

Every SELECT, INSERT, UPDATE and DELETE now creates a scylla.execute span under the API request that issued it.

note: db.statement records the query text. Use parameterised queries, so that values such as user data are passed as parameters and not embedded in the statement.

Manual instrumentation (SDK)

With manual instrumentation, you install the OpenTelemetry SDK packages yourself and add the trace setup code to the application. Use this instead of opentelemetry-instrument.

Step 1: Install the packages

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation-flask

Package	Purpose
`opentelemetry-sdk`	OpenTelemetry SDK
`opentelemetry-exporter-otlp-proto-http`	Sends traces over OTLP/HTTP
`opentelemetry-instrumentation-flask`	Spans for Flask requests. Use `-django`, `-fastapi`, `-requests`, `-psycopg2` and so on for other libraries

Step 2: Add the setup code

Create telemetry.py:

# telemetry.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


def setup_telemetry():
    """Set up tracing from the OTEL_* environment variables."""
    # Service name and kubesense.* attributes from OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES
    resource = Resource.create()

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(tracer_provider)

Call it at startup, before the app handles requests, then instrument your framework:

# app.py
from telemetry import setup_telemetry

setup_telemetry()

from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

Step 3: Configure the endpoints

Set the environment variables from Manual SDK: where to send traces. Don't also start the app with opentelemetry-instrument; use one or the other.

Troubleshooting

Set OTEL_LOG_LEVEL=debug temporarily to see the SDK's export activity.

Symptom	Check
No spans at all	The app isn't started through `opentelemetry-instrument`, or an auto-reloader is active
Some libraries missing	Run `opentelemetry-bootstrap -a install` again after installing them, and rebuild the image
Connection errors to `:33443`	Trace ingestion is not enabled on KubeSense, or the port is not reachable
Exporter errors for metrics or logs when sending directly to `:33443`	Port `33443` is the trace endpoint. Set `OTEL_METRICS_EXPORTER=none` and `OTEL_LOGS_EXPORTER=none`

Python

ON THIS PAGE