Creating & Managing Pipelines

This guide walks you through creating a log pipeline from scratch and covers best practices for combining rules effectively.

Creating a Pipeline

1. Navigate to Logs > Log Pipelines and click Add Pipeline in the top right.

2. Basic Details — Enter a Rule Group Name that describes the pipeline's purpose (e.g., redact-pii-api-gateway, parse-nginx-logs).

3. Rule Matcher — Define which logs this pipeline applies to:

   • Enter a workload or namespace pattern (e.g., us-2/log-printer/*, prod/api-gateway/*)
   • Use * wildcards to match multiple workloads within a namespace
   • Skip the selection to apply the pipeline to all namespaces

4. Add Rules — Click + Add Rule and choose a rule type. Configure the rule fields, then repeat to add more rules as needed.

5. Reorder Rules — Drag rules to change their execution order. Rules run top-to-bottom, and each rule operates on the output of the previous one.

6. Preview — The right panel shows how the rules affect your logs. Use this to validate behavior before saving.

7. Click Save to activate the pipeline.

Editing a Pipeline

Click on any pipeline in the list to open it for editing. You can:

• Modify existing rules or their configuration
• Add new rules or remove existing ones
• Reorder rules by dragging
• Update the Rule Matcher to target different workloads

Click Save to apply changes. Updated pipelines take effect on newly ingested logs.

Rule Ordering Best Practices

Since rules execute sequentially, the order you place them in matters:

1. Parse first — If your logs are unstructured, place Parse rules at the top so subsequent rules can operate on the extracted fields.
2. Extract and enrich next — Extract, JSON Extract, Add Field, and Timestamp Extract rules work best after the log has been structured.
3. Replace before Block — Redact sensitive data before deciding whether to drop a log, so even blocked logs have their PII removed during the processing window.
4. Block last — Place Block rules toward the end so they evaluate against the fully processed log.
5. Remove Fields at the end — Strip unwanted fields as the final step, after all other transformations are complete.

Recommended order:

Parse → Extract / JSON Extract → Timestamp Extract → Add Field → Replace → Block → Remove Fields

Common Pipeline Patterns

Structuring and Enriching Application Logs

Goal: Parse unstructured Node.js logs and tag with team ownership.

PII Redaction Pipeline

Goal: Ensure no personally identifiable information is stored in logs. The example below shows a "Redact-PII" pipeline scoped to cluster-1/node-plum/* with a Replace rule that masks phone numbers using a regex pattern.

Noise Reduction Pipeline

Goal: Cut log volume and costs by filtering out low-value logs.

JSON Log Enrichment Pipeline

Goal: Extract key fields from structured JSON logs and add routing metadata.