Kubesense

Valkey (and Redis) client tracing

Trace every application call to Valkey with OpenTelemetry, by client library (redis-py, valkey-py, Valkey GLIDE).

Scope: Sending traces of every application → Valkey call to KubeSense. Which method to use depends on the client library your application uses to talk to Valkey: some are covered by OpenTelemetry auto-instrumentation, some need a small manual module, and one has tracing built in.

Applies to: Valkey (validated on 9.0; the same methods apply to Redis, which uses the same protocol). Applications on VMs, Docker or Kubernetes.

1. How Valkey tracing works

Valkey is a key-value datastore. It does not emit traces itself. A "Valkey trace" is a client span: a span created inside your application each time it sends a command to Valkey.

 ┌──────────────── your application ────────────────┐
 │                                                   │
 │  POST /checkout  (HTTP span)                      │
 │    ├─ HGETALL cart:?   (Valkey span, 0.2 ms)  ────┼──▶ Valkey
 │    └─ LPUSH orders ?   (Valkey span, 0.1 ms)  ────┼──▶ Valkey
 │                                                   │
 └───────────────┬───────────────────────────────────┘
                 │ OTLP
                 ▼
          KubeSense trace endpoint

Each Valkey span records:

  • the command (GET, SET, HGETALL, MULTI, …). Argument values are masked as ?; whether the key name is kept depends on the library (§4.3, §5.1, §6.2).
  • the latency as the application sees it (network time included)
  • errors (WRONGTYPE, timeouts, connection failures)
  • which service and request made the call

Three pieces are always involved:

#PieceWhere it livesWhat it does
①SDK + exporter setupStartup command or entrypoint, plus environment variablesNames the service and says where to send spans
②InstrumentationA library package, or one module in your codeWraps every Valkey command in a span
③ExportEnvironment variablesSends spans to KubeSense over OTLP

① and ③ are the same for every application. Only ② depends on your client library, so that's where this guide branches.


2. Step 1 — Identify your Valkey client library

Search the application code:

# Python
grep -rn -E "^\s*(import|from) (redis|valkey|glide)\b" --include=*.py .
grep -i -E "^(redis|valkey|valkey-glide)\b" requirements*.txt pyproject.toml Pipfile 2>/dev/null

Then pick the matching path:

Client library you findPackageInstrumentation methodCode changesSection
import redisredis (redis-py)Auto-instrumentationNone§4
import valkeyvalkey (valkey-py)Manual (one module + one call)2 lines + 1 file§5
from glide import …valkey-glideBuilt-in to the client1 init call§6

valkey-py is not covered by auto-instrumentation: The valkey-py documentation suggests RedisInstrumentor().instrument(). In our testing this produces no spans for valkey.Valkey() clients: that instrumentor only patches the redis package. Use §5 instead.

Other languages: the same model applies. Java (Jedis, Lettuce), Node.js (ioredis, redis), Go (go-redis via redisotel) and .NET (StackExchange.Redis) all have official OpenTelemetry Redis instrumentation, which works against Valkey because the protocol is the same. Valkey GLIDE's built-in tracing (§6) is available in Python, Java, Node.js, Go, C#, PHP and Ruby. These were not part of our validation; contact KubeSense support before rolling them out.


3. Common setup (all paths)

These environment variables are needed whichever path you follow. Set them where the application's environment is defined:

  • Kubernetes Deployment env:
  • docker-compose.yaml environment:
  • systemd unit Environment=
  • a .env file
OTEL_SERVICE_NAME=<your-service-name>                      # shown as the workload in KubeSense
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<kubesense-trace-endpoint>:33443/v1/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none
OTEL_RESOURCE_ATTRIBUTES=kubesense.cluster=<cluster-name>,kubesense.env_type=<k8s|legacy>
  • kubesense.cluster must match the cluster name configured in KubeSense, so the traces show under that cluster.
  • kubesense.env_type is k8s for Kubernetes workloads and legacy for VMs and plain Docker hosts.
  • If you run an OpenTelemetry Collector, point OTEL_EXPORTER_OTLP_ENDPOINT at the collector (e.g. http://<collector>:4318) and have the collector export to the KubeSense endpoint. Set the two kubesense.* attributes on the collector instead, with a resource processor.

4. Path A — redis-py: auto-instrumentation (no code changes)

opentelemetry-instrumentation-redis patches redis-py at startup. Every command, pipeline and transaction then produces a span, nested under the incoming request.

4.1 Dependencies (requirements.txt or equivalent)

opentelemetry-distro
opentelemetry-exporter-otlp-proto-http

Then run this once during the build. It detects the installed libraries (Flask, Django, FastAPI, redis-py, …) and installs the matching instrumentations:

opentelemetry-bootstrap -a install

4.2 Startup command

Prefix the existing start command with opentelemetry-instrument:

Where it's definedBeforeAfter
DockerfileCMD ["gunicorn", "-b", "0.0.0.0:8080", "app:app"]CMD ["opentelemetry-instrument", "gunicorn", "-b", "0.0.0.0:8080", "app:app"]
Kubernetes command:["python", "main.py"]["opentelemetry-instrument", "python", "main.py"]
systemd ExecStart=/opt/app/venv/bin/uvicorn main:app/opt/app/venv/bin/opentelemetry-instrument /opt/app/venv/bin/uvicorn main:app

4.3 Environment

Use the variables from §3. No extra setting is needed for privacy: the redis-py instrumentation always masks every argument after the command name (for example SET ? ?), so neither key names nor values are recorded.

4.4 Result

Every Valkey call appears as a Redis span under the HTTP request that made it. No application code changes.


5. Path B — valkey-py: manual instrumentation

No OpenTelemetry package exists for valkey-py. This path adds one file to your codebase and one call at startup. The module patches valkey-py the same way the official instrumentor patches redis-py. It covers:

  • sync and asyncio clients
  • pipelines and transactions (MULTI/EXEC)
  • error recording

Your existing valkey.Valkey(...) calls stay as they are.

Known gap: commands sent on a pipeline while it is in WATCH mode (before MULTI) run through Pipeline.immediate_execute_command and are not traced individually. The EXEC that follows is traced as the MULTI span.

5.1 Add valkey_otel.py to your application

Place it next to your entrypoint (or anywhere importable):

"""OpenTelemetry tracing for valkey-py (`import valkey`).

opentelemetry-instrumentation-redis only patches redis-py, so valkey-py
clients produce no spans under auto-instrumentation. Call `instrument()`
once at startup, before or after creating clients; every command and
pipeline then emits a CLIENT span using the standard database attributes.

    import valkey_otel
    valkey_otel.instrument()
"""
import functools

import valkey
import valkey.asyncio
import valkey.asyncio.client
import valkey.client
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

_tracer = trace.get_tracer("valkey_otel", "1.0.0")
_instrumented = False


def _statement(args):
    # Command name plus the key; the remaining arguments (values) are masked.
    parts = [str(a) for a in args[:2]]
    return " ".join(parts + ["?"] * (len(args) - 2))


def _attributes(client, operation, statement):
    kw = client.connection_pool.connection_kwargs
    return {
        "db.system": "redis",          # older semconv, read by most backends
        "db.system.name": "redis",     # current semconv
        "db.operation": operation,
        "db.operation.name": operation,
        "db.statement": statement,
        "db.query.text": statement,
        "db.redis.database_index": kw.get("db", 0),
        "server.address": kw.get("host", ""),
        "server.port": kw.get("port", 6379),
        "net.peer.name": kw.get("host", ""),
        "net.peer.port": kw.get("port", 6379),
    }


def _fail(span, exc):
    span.record_exception(exc)
    span.set_status(Status(StatusCode.ERROR, str(exc)))


def _pipeline_parts(pipe):
    names = [str(args[0]).upper() for args, _ in pipe.command_stack]
    op = "MULTI" if pipe.transaction else "PIPELINE"
    return op, "\n".join(names)


def _wrap_command(original):
    @functools.wraps(original)
    def wrapper(self, *args, **options):
        op = str(args[0]).upper()
        with _tracer.start_as_current_span(
            op, kind=SpanKind.CLIENT, attributes=_attributes(self, op, _statement(args)),
            record_exception=False, set_status_on_exception=False,
        ) as span:
            try:
                return original(self, *args, **options)
            except Exception as exc:
                _fail(span, exc)
                raise
    return wrapper


def _wrap_command_async(original):
    @functools.wraps(original)
    async def wrapper(self, *args, **options):
        op = str(args[0]).upper()
        with _tracer.start_as_current_span(
            op, kind=SpanKind.CLIENT, attributes=_attributes(self, op, _statement(args)),
            record_exception=False, set_status_on_exception=False,
        ) as span:
            try:
                return await original(self, *args, **options)
            except Exception as exc:
                _fail(span, exc)
                raise
    return wrapper


def _wrap_pipeline(original):
    @functools.wraps(original)
    def wrapper(self, *args, **kwargs):
        if not self.command_stack:
            return original(self, *args, **kwargs)
        op, stmt = _pipeline_parts(self)
        attrs = _attributes(self, op, stmt)
        attrs["db.operation.batch.size"] = len(self.command_stack)
        with _tracer.start_as_current_span(
            op, kind=SpanKind.CLIENT, attributes=attrs,
            record_exception=False, set_status_on_exception=False,
        ) as span:
            try:
                return original(self, *args, **kwargs)
            except Exception as exc:
                _fail(span, exc)
                raise
    return wrapper


def _wrap_pipeline_async(original):
    @functools.wraps(original)
    async def wrapper(self, *args, **kwargs):
        if not self.command_stack:
            return await original(self, *args, **kwargs)
        op, stmt = _pipeline_parts(self)
        attrs = _attributes(self, op, stmt)
        attrs["db.operation.batch.size"] = len(self.command_stack)
        with _tracer.start_as_current_span(
            op, kind=SpanKind.CLIENT, attributes=attrs,
            record_exception=False, set_status_on_exception=False,
        ) as span:
            try:
                return await original(self, *args, **kwargs)
            except Exception as exc:
                _fail(span, exc)
                raise
    return wrapper


def instrument():
    """Patch valkey-py once per process. Safe to call more than once."""
    global _instrumented
    if _instrumented:
        return
    # Pipeline subclasses Valkey but overrides execute_command to only queue
    # commands, so patching Valkey.execute_command does not double-count them.
    valkey.Valkey.execute_command = _wrap_command(valkey.Valkey.execute_command)
    valkey.client.Pipeline.execute = _wrap_pipeline(valkey.client.Pipeline.execute)
    valkey.asyncio.Valkey.execute_command = _wrap_command_async(valkey.asyncio.Valkey.execute_command)
    valkey.asyncio.client.Pipeline.execute = _wrap_pipeline_async(valkey.asyncio.client.Pipeline.execute)
    _instrumented = True

5.2 Call it once at startup

Put the call in the module that the web server imports. Typical locations:

FrameworkFile
Flaskthe module with app = Flask(...)
FastAPImain.py
Djangowsgi.py / asgi.py, or the AppConfig.ready() of one app
Celerythe Celery app module

Put it near the top, before the first Valkey call:

import valkey_otel
valkey_otel.instrument()

import valkey
client = valkey.Valkey(host="<valkey-host>", port=6379)   # unchanged

With pre-forking servers (gunicorn, uWSGI), each worker process imports the module itself, so each worker is instrumented. Nothing extra is needed.

5.3 SDK, exporter and request spans

The module creates the Valkey spans. The SDK still has to be started so those spans get exported, and so incoming HTTP requests get their own spans for the Valkey calls to nest under. Use the same steps as Path A:

  • dependencies, §4.1
  • opentelemetry-instrument startup prefix, §4.2
  • environment variables, §3

5.4 Result

Every Valkey command appears as a Redis span named after the command, nested under its request:

  • pipelines appear as one PIPELINE span, and transactions as one MULTI span
  • failed commands are marked as errors, with the exception recorded

6. Path C — Valkey GLIDE: built-in OpenTelemetry

GLIDE creates spans itself, inside its core. There is no separate instrumentation package. You enable it with one call.

6.1 Enable it in the entrypoint

Call this once per process, before any client is created:

import os
from glide import OpenTelemetry, OpenTelemetryConfig, OpenTelemetryTracesConfig

OpenTelemetry.init(OpenTelemetryConfig(
    traces=OpenTelemetryTracesConfig(
        endpoint=os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"],   # http://, https://, grpc://
        sample_percentage=100,                                        # default is 1 — see below
    ),
    flush_interval_ms=1000,
))

This is for the async client (package valkey-glide, from glide import …). For the sync client (package valkey-glide-sync), make the same call with the classes imported from glide_sync:

from glide_sync import OpenTelemetry, OpenTelemetryConfig, OpenTelemetryTracesConfig

Also set OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES from §3. GLIDE reads both, so its spans carry the right workload name and kubesense.* attributes even when exporting straight to KubeSense without a collector.

With pre-forking servers (gunicorn, uWSGI), put the call in the module the workers import, as in §5.2. Each worker process then runs it once.

6.2 Things to know about GLIDE tracing

BehaviourImpactWhat to do
Sampling defaults to 1%99 of every 100 commands are not tracedSet sample_percentage=100, or an agreed lower rate for very high-volume services
Spans are not linked to the application's request tracesEach command appears as its own trace, not under the HTTP requestAccept it, or use Path A/B for linked traces
Failed commands are not marked as errorsError rate for Valkey calls reads 0Use Path A/B if error tracking matters
Batches (pipelines and transactions) appear as one Batch spanIndividual commands inside a batch don't get their own spansNothing needed; the span covers the whole round trip
The key appears in db.query.textKey names are visible (values are masked)Avoid putting sensitive data in key names
init() only works once per processLater calls are ignoredCall it in the entrypoint only

7. Validation results

Test scope:

  • The same Python web application (Flask + gunicorn) was run once per client library: redis-py, valkey-py, and valkey-glide (sync client). All three were run against Valkey 9.0 under steady traffic: cache reads/writes, cart hashes, a pipelined transaction, a slow Lua script, and a deliberate WRONGTYPE error.
  • All three were tested in two environments, with the same results in both:
    • a Linux VM (application and Valkey in Docker, exporting through an OpenTelemetry Collector)
    • Kubernetes (application and Valkey as Deployments, exporting directly to KubeSense)

Auto-instrumentation only (opentelemetry-instrument, no code changes), 100 commands per library:

Client libraryInstrumentation detected by opentelemetry-bootstrapSpans received
redis-pyopentelemetry-instrumentation-redis100 / 100
valkey-pynone0
valkey-glidenone (needs §6 init)0

With the method from this guide for each library:

redis-py (§4, auto)valkey-py (§5, manual)valkey-glide (§6, built-in)
Spans received (VM and Kubernetes)✅✅✅ (at 100% sampling)
Command as the span's operation name✅ GET, HGETALL…✅ GET, HINCRBY, EVAL…✅ Get, HGetAll, Incr…
Command in the span's method field— empty✅✅
Nested under the HTTP request✅ 100%✅ 100%❌ 0%, separate traces
Errors flagged✅✅❌
Pipelines / transactions✅ one span listing the commands (HGETALL DEL LPUSH)✅ one MULTI span✅ one Batch span
Code changesnone1 file + 2 lines1 init call

8. Verify in KubeSense

  1. Generate some traffic to the application.
  2. Open Traces and pick the cluster from kubesense.cluster.
  3. Filter Workload = <your-service-name> (the OTEL_SERVICE_NAME). Valkey calls are the spans with Protocol = Redis, and the Valkey host is shown as the partner/destination.
  4. Open a request trace: the Valkey spans appear as children of the HTTP span (Paths A and B).

Why there is no "valkey" workload with SDK tracing: With source OTel, every span belongs to the application that made the call. Valkey only appears as the destination.


9. Troubleshooting

SymptomLikely causeCheck / fix
No spans at all from the serviceSDK not startedThe process must run under opentelemetry-instrument (or have the SDK started in code). Check the startup command.
HTTP spans arrive, but no Valkey spansClient library not instrumentedvalkey-py needs §5. For GLIDE, check OpenTelemetry.init() ran before the client was created.
Only a few Valkey spans (GLIDE)1% default samplingSet sample_percentage=100.
Spans under the wrong or no clusterMissing resource attributesCheck OTEL_RESOURCE_ATTRIBUTES contains kubesense.cluster and kubesense.env_type.
Export errors in the app logsEndpoint unreachableTest curl -s -o /dev/null -w '%{http_code}' -X POST <traces-endpoint> from the app host. Any HTTP code means it's reachable; a timeout means a firewall or routing problem.