Trace Filters

Trace filters decide which traces KubeSense keeps. Each rule either drops the traces it matches or collects only the traces it matches. Filtering happens as traces arrive, before anything is stored. A dropped trace is never written, so it does not appear in trace search, on the Services or Endpoints pages, or in any view built from traces.

Manage them under Settings → Trace Filters.

Filters apply to traces from the KubeSense sensor (eBPF), OpenTelemetry and Datadog. They run in KubeSense, after the traces have been sent to it; see Where filtering happens.

The rule list

Each row in Trace Filters is one rule. The toggle on a row enables or disables it. A disabled rule has no effect at all, which makes it the safe way to pause a rule without losing it.

Changes take effect within about two minutes of saving, once the collectors pick up the new rules.

Anatomy of a rule

Click Create New to open the rule form. It reads top to bottom as one sentence: <Action> traces in <domains> that match <conditions>.

Fields you can match on

Values are matched exactly and are case-sensitive. There are no wildcards or prefixes: payments does not match payments-v2. The value picker suggests values seen in your data, which is the reliable way to get them right.

How rules combine

Three rules cover every combination:

Drop rules are independent. A trace is dropped if any Drop rule matches it. Two Drop rules drop traces matching either one; one Drop rule with two conditions drops only traces matching both.

Collect only rules form one allowlist. All Collect only rules that apply to a domain merge, field by field:

• Values for the same field add up. A rule collecting namespace prod and another collecting namespace staging together collect both.
• Different fields must all pass. A rule collecting namespace prod and another collecting workload api together collect only traces that are in prod and from api.

This means that for Collect only, putting conditions in one rule or splitting them across rules gives the same result. For Drop it does not.

Drop wins. A trace that passes the Collect only allowlist is still dropped if a Drop rule matches it. That is how "collect only production, except one noisy job" is written — see the examples below.

When a trace has no value for a field

Not every trace carries every field. External services, legacy VMs and Docker hosts often have no namespace, and a span may have no workload.

• Collect only: a condition on a field the trace has no value for is skipped, and the trace is kept. Without this, one rule collecting namespace prod would silently drop every span that has no namespace at all.
• Drop: a condition on a field the trace has no value for does not match, so the trace is not dropped by it. In a Drop rule with several conditions, one missing field means the whole rule does not match.

Two Collect only rules are exceptions and drop a trace that lacks the value: a rule listing domains and nothing else drops traces whose domain cannot be determined (see Rules without conditions), and a bare-key tag such as team drops traces without that attribute (see Tag conditions). Drop rules have no exceptions: a missing value never causes a drop, and a bare-key tag in a Drop rule drops every trace that has the attribute, whatever its value.

Domains and scope

A rule with Select domain(s) only affects traces from the domains it lists. Traces from other domains never consult it.

A rule with All domains applies to every domain, and combines with each domain's own rules as if it had been written for that domain too. A domain can widen an All domains rule, for example by adding more values to drop, but it cannot switch one off.

Rules may overlap. If two rules cover the same field in the same domain, the form marks the field with "Another rule already covers this field for one of the selected domains — the two will combine", and the rules combine as described above.

How a trace's domain is determined

A trace's domain is the KubeSense domain it was collected from. For OpenTelemetry it is read from the resource attribute kubesense.cluster, then k8s.cluster.name, cluster_name, kube_cluster_name. Datadog traces use kubesense.cluster, then the environment (kubesense.env, then env, which carries DD_ENV), then cluster_name, each looked for in the Datadog Agent's container tags, then the batch, then its spans (where a tracer's DD_TAGS arrive); a Datadog trace naming none of them goes to the default domain configured for pushed traces, if one is set. Because the environment ranks above cluster_name, a service that sets DD_ENV should name its domain with kubesense.cluster in DD_TAGS. Sensor (eBPF) traces take the domain of the sensor that captured them.

Rules without conditions

Removing every condition from a rule is allowed, and the result depends on the action and scope:

The Collect only + selected domains row is the one rule whose reach goes beyond the domains it names: it drops traffic from every domain it does not list.

Tag conditions

A tag condition matches a trace's resource attributes (OpenTelemetry) — the attributes that describe the process or service, not individual spans — or, for Datadog, the tags stored on each span. Each entry is either:

• key:value — the attribute exists and has exactly this value, for example deployment.environment:staging;
• key — a bare key, meaning the attribute exists, with any value.

For OpenTelemetry, tags are checked once per incoming batch, before its spans are processed, so a whole batch is kept or dropped together. For Datadog, each span is checked on its own, before it is processed. Sensor (eBPF) traces have no resource attributes, so tag conditions never apply to them.

Which Datadog tags a condition sees

A Datadog trace has no resource attributes, so a tag condition checks each span against the tags it is stored with. It looks in three places, in this order, and uses the first value it finds:

1. Container tags added by the Datadog Agent, such as kube_namespace, pod_name, kube_deployment and image_name.
2. Tags that describe the whole batch: telemetry.sdk.language, telemetry.sdk.version, process.runtime.version and service.version (from DD_VERSION). Through the Datadog Agent these also include deployment.environment (from DD_ENV), host.name and container.id.
3. The span's own tags. This is where DD_TAGS, and DD_ENV as env, arrive when a tracer sends straight to KubeSense. They also include request tags such as http.method: a rule on http.method:GET drops just the spans for GET requests.

Write the key exactly as it appears on the trace in KubeSense, for example team:payments for DD_TAGS=team:payments.

A tag condition cannot share a rule with any other field, because tags are read at an earlier stage than namespace, workload, container and endpoint. To combine them, write two rules.

Tags in a Drop rule

A Drop rule takes one tag condition. A batch is dropped if any entry matches:

• env:ci drops batches whose env attribute is ci;
• a bare debug drops every batch that carries a debug attribute, whatever its value.

Tags in a Collect only rule

A Collect only rule can take several tag conditions, each shown as its own box:

• Inside one box, any entry is enough, even when entries name different keys. env:prod and team:payments in one box collect batches that have either.
• Every box must be satisfied. With a box env:prod and a second box team:payments, a batch that carries env must have env:prod, and a batch that carries team must have team:payments.
• A box whose keys a batch does not carry is skipped, like any other missing field, unless the box contains a bare key. In the example above, a batch with env:prod and no team attribute is kept.
• A bare key requires the attribute. A box containing env drops batches that have no env attribute and no other entry of the box matching. This is the way to say "collect only services that report env". To require a specific value and the attribute's presence, use two boxes: env:prod and a bare env.

Collect only rules over the same attribute keys combine: one rule collecting env:prod and another collecting env:staging together collect both.

Examples

Each example is written as the rule you would enter. AND separates conditions in one rule; commas separate values in one condition.

The order rules are checked in

For each trace, KubeSense decides in this order and stops at the first drop:

1. Domain. If a Collect only rule lists domains and this trace's domain is not among them, it is dropped. If a Drop rule with no conditions covers the domain, it is dropped.
2. Tags. For OpenTelemetry, tag conditions are checked once per incoming batch, and a batch that fails is dropped whole. For Datadog they are checked per span.
3. Collect only. Each field with a Collect only rule must match, unless the trace has no value for that field.
4. Drop. Any matching Drop rule, or any Drop rule whose conditions all match, drops the trace.
5. Otherwise the trace is kept.

Where filtering happens

Filters run in KubeSense as traces arrive, not in the sensor or in your tracers. Filtered traces are still sent to KubeSense and discarded on arrival, so a rule saves storage but not the network traffic from where the traces came from. To stop a domain sending traffic at all, stop or uninstall its sensor and tracers rather than filtering it.

How early a trace is dropped depends on where it came from:

Trace filters apply only to traces. The network flow data the sensor captures is not affected by them.

Restricting who can change a rule

Every trace filter rule is open by default: anyone whose role has Trace Filters at Editor can change it. Restricting a rule narrows that to its owner and the people the owner names. The rule stays visible to everyone with Trace Filters access. Only changing it is restricted.

Opening the dialog

In the rule list, open the rule's ⋯ menu and choose Access. Anyone who can see the rule can open the dialog and read who may change it. Only the rule's owner or a Trace Filter Admin can change it.

Restricting a rule

1. Click Restrict Editing.
2. Under People with access, add people by name or email. Each person gets a level, Viewer or Editor. The owner is pinned at the top and cannot be removed.
3. Under Everyone else, choose what anyone with Trace Filters access who is not named gets. Viewer is the default.
4. Click Save. Nothing is saved until you do.

Restore Full Access removes the restriction and the list of people with it.

A restricted rule shows a lock beside its name. For someone who may not change it, the toggle and Delete rule are disabled, and the rule opens with Save off and a note saying it is restricted.

Owners and Trace Filter Admins

Whoever creates a rule owns it. Rules that existed before this feature are owned by their creator where that user still exists, and stay unrestricted until someone restricts them.

Trace Filter Admin is a role permission (see Role Access). Someone who holds it, along with Trace Filters at Editor, can change any restricted rule and who has access to it. Each time they change access on a rule they do not own, the audit log records it as an admin override. A rule with no owner can only be restricted by a Trace Filter Admin, who becomes its owner.

Things to know

• A grant never gives more than the person's role allows. Someone whose role has Trace Filters at Viewer cannot change a rule, whatever level they are given on it.
• The address you add must belong to an existing user. Saving is refused otherwise, and the message names the addresses that did not match.
• If someone else changes the rule's access while you have the dialog open, your save is refused and the dialog shows their version.
• Exclude on the Endpoints and trace pages adds to an existing rule only when you may change that rule. Otherwise it creates a new rule, which has the same effect.
• The API enforces the restriction, so it applies to direct API calls and API keys as well as the page.
• Every access change is recorded in Settings → Audit Logs.

Limits and validation

The form and the API refuse a rule that could not behave as written:

• At most 8 conditions per rule.
• Each field appears once per rule. List all of its values in that one condition instead.
• Every condition needs at least one value.
• A tag condition cannot be combined with namespace, workload, container or endpoint in the same rule.
• A Drop rule takes one tag condition, because its entries already match on any of them.
• Tag entries must name an attribute key. :prod is refused; env:prod and env are accepted. Spaces around the key and value are trimmed when you save.
• A Collect only rule with no conditions must name the domains to collect from.

Dropping an endpoint from the Endpoints page

Where enabled, each row on the Endpoints page has an Exclude button. It creates a Drop rule on that endpoint, scoped to the row's domain, which then appears in Settings → Trace Filters like any other rule.

Checking what your rules drop

Every trace a rule drops is counted, so a rule that drops more than intended shows up without waiting for someone to notice missing traces:

• Every 60 seconds the collector writes one kubesense_trace_drop line per reason, tenant, rule, source and domain to its log, for example kubesense_trace_drop reason=field_rule tenant=1 rule=drop-health-checks source=otel domain=prod-us detail=namespace count=1520 unit=spans window_s=60. rule is the rule's name, or kuid: and the start of its ID for an unnamed rule; when several rules caused the drop they are joined with +.
• Where KubeSense pushes its own metrics to VictoriaMetrics, kubecol_collector_trace_filter_drop_count breaks drops down by tenant_id, source, domain, service, field, reason and rule, counted in spans.