Rehydrate Datadog Archives

Rehydration loads Datadog log archives from Amazon S3 or Azure Blob Storage into KubeSense Logs. Once loaded, archived logs can be searched, filtered and queried like live logs in the Log Explorer, SPL and SQL. Use it to bring history across when you move from Datadog, or to look into an incident older than your live retention.

Each request loads one archive path under a domain name of your choice, and keeps the logs for the number of days you pick. Requests run one at a time in a queue, in the background: you can close the browser while they run.

Where to Find It

Open Logs → Archived Logs in the left sidebar and click Rehydrate Datadog Archive at the top right. The page lists every request with its domain, archive path, hours, retention, status, progress, and who made it.

Making or stopping requests needs write access to the Archived Logs module. Read access is enough to see the list.

Before You Start

Rehydration needs two things on the install:

1. KubeSense with kubecol migration 2.0.162 or later. Until then the page shows Rehydration isn't set up on this install.
2. The rehydrator, a small service the Helm chart installs. It reads the archive with its own cloud identity, so no access keys or secrets are stored in KubeSense. Until it runs, the page shows The rehydrator isn't running, and requests wait in the queue.

Installing the Rehydrator

Enable it in your values file and give it the identity it reads storage with: an IAM role for S3, or an Azure managed identity for Blob Storage. One install can have both.

rehydrator:
  enabled: true
  serviceAccount:
    # S3 archives: an IAM role (IRSA on EKS)
    roleArn: "arn:aws:iam::123456789012:role/kubesense-rehydrator"
    # Azure Blob archives: a user-assigned managed identity (AKS Workload Identity)
    azureClientId: "00000000-0000-0000-0000-000000000000"

The rehydrator runs as the service account kubesense-rehydrator in the KubeSense namespace. Its token for talking to the API is generated on first install and kept across upgrades.

Amazon S3 (IAM role)

Create an IAM role that trusts the service account system:serviceaccount:<namespace>:kubesense-rehydrator through your EKS cluster's OIDC provider, and set its ARN as rehydrator.serviceAccount.roleArn.

• EKS Pod Identity: leave roleArn empty and associate the role with the service account in EKS.
• Outside EKS (AKS, GKE, on-premises): set rehydrator.serviceAccount.webIdentity: true as well. The pod then assumes the role with a projected service account token, so the role must trust your cluster's OIDC issuer as an IAM identity provider.

Buckets in any region work: the rehydrator looks up each bucket's region itself.

Azure Blob Storage (managed identity)

On an AKS cluster with the OIDC issuer and Workload Identity enabled:

# 1. A user-assigned managed identity for the rehydrator
az identity create --name kubesense-rehydrator \
  --resource-group <resource-group> --location <region>

# 2. Trust it from the rehydrator's service account
az identity federated-credential create --name kubesense-rehydrator \
  --identity-name kubesense-rehydrator --resource-group <resource-group> \
  --issuer "$(az aks show -n <aks-cluster> -g <aks-resource-group> \
      --query oidcIssuerProfile.issuerUrl -o tsv)" \
  --subject system:serviceaccount:<namespace>:kubesense-rehydrator \
  --audiences api://AzureADTokenExchange

# 3. Its client id goes in rehydrator.serviceAccount.azureClientId
az identity show --name kubesense-rehydrator \
  --resource-group <resource-group> --query clientId -o tsv

Storage accounts with shared key access turned off work, because the rehydrator signs in with Microsoft Entra ID.

Granting Access to the Archive

Whoever owns the bucket or storage account grants the rehydrator read access. The New Rehydration form shows the rehydrator's role ARN or client id, with a copy button and an example grant for the path you type.

An example bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "KubeSenseRehydratorList",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::123456789012:role/kubesense-rehydrator" },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-datadog-archive"
    },
    {
      "Sid": "KubeSenseRehydratorRead",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::123456789012:role/kubesense-rehydrator" },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-datadog-archive/logs/*"
    }
  ]
}

An example Azure role assignment:

az role assignment create \
  --assignee <rehydrator client id> \
  --role "Storage Blob Data Reader" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>"

A new Azure role assignment can take a few minutes to apply.

Making a Request

Click New Rehydration and fill in:

• Domain name — the name the archived logs are stored under. It becomes the value of the cluster field on every loaded log. Letters, digits, -, _ and ., up to 128 characters. Pick a new name such as datadog-2024.
• Days to keep — how long the logs stay, counted from the day they are loaded, not from the logs' own times. 1 to 3650 days; 7 by default.
• Archive path — the folder that holds the archive's dt= folders:
  • S3: s3://my-datadog-archive/logs/
  • Azure Blob: https://<account>.blob.core.windows.net/<container>/logs/
• Hours (UTC) — optional. To load only part of the archive, enter the first and last hour, both included, like 2024-01-05T00 and 2024-01-05T23. Leave both empty to load everything under the path.

Then click one of:

• Check files — lists the archive files and counts the ones already loaded, without loading anything. It also proves the rehydrator can read the storage, so run it first: an access problem shows up here rather than partway through a load. A finished check has a Load these files action in its row menu.
• Load — queues the load.

The archive must be in Datadog's archive layout, dt=YYYYMMDD/hour=HH/archive_*.json.gz, as Datadog writes it.

Loading into an Existing Domain

If a domain with that name already exists in Settings → Domain, KubeSense asks before going on: the archive and that domain's own logs would share one retention rule, so the domain's live logs would also be kept for the chosen number of days after they arrive. Pick a new domain name unless that is what you want.

Retention

Datadog archives hold old logs. A retention rule that counted from each log's own time would delete them as soon as they arrived, so before any load KubeSense adds a rule for the domain to the Logs table in Settings → Data Retention:

• It is named Rehydrated: <domain>, matches cluster in the domain, and deletes rows the chosen number of days after they were loaded.
• It keeps your default rule's storage tiers that take effect before the delete.
• Your other rules are not changed. Loading the same domain again with a different number of days updates its rule.

A request is refused, with the reason shown in the form, when the rule can't be added safely:

• The Logs retention is a custom expression that doesn't mention the domain. Add a rule for the domain counting from ingestion time in Data Retention, then try again.
• Another rule already covers the domain but counts from each log's time, keeps it for fewer days than you chose, or hasn't reached every ClickHouse shard yet.

To remove rehydrated logs early, delete or shorten the domain's rule in Data Retention.

Statuses and Progress

The list refreshes on its own while a request is queued or running.

• Cancel / Stop — in the row menu. A queued request is cancelled at once; a running one stops after the file it is on, showing Stopping until then.
• Running again is safe. Every loaded row records the archive file it came from. A rerun skips files that are fully loaded and reloads any that were only partly loaded, so retrying a failed or stopped request never duplicates logs.
• If the rehydrator restarts mid-load, the request carries on when it is back. A request whose rehydrator stays silent for 3 minutes is handed to the next one that asks.

Searching Rehydrated Logs

Filter on cluster = "<domain>" and set the time range to the logs' original dates: logs keep their original timestamps. A few fields are worth knowing:

To load only part of an archive, set Hours (UTC) or point the path at a single day's folder, such as s3://my-datadog-archive/logs/dt=20240105/. A path to the archive's top folder with no hours loads every file under it.

Charts and Filter Counts

The Log Explorer's trend graph and the counts under Filters come from log count rollups, not from the logs themselves. Each rollup keeps its own retention, counted from each log's original time. The defaults:

For dates older than these, the trend graph and the filter counts stay empty, while the log table still lists every rehydrated log. A daily step shows logs up to 450 days old. The same applies to charts built on log metrics.