Fastly CDN Logs

Fastly records every request its edge serves for your site: the URL, the response status, whether it was a cache hit, which POP served it, the client and how long it took. With a Fastly HTTPS logging endpoint, Fastly pushes these request logs to KubeSense within seconds. They are stored with your other logs, so you can search them on the Logs page, alert on 5xx spikes and put CDN traffic next to your application logs and traces.

Fastly posts straight to your KubeSense endpoint. You don't need to run any relay function or extra service. If you already send Fastly logs to another tool, such as Datadog, add KubeSense as a second endpoint. Both keep receiving the same logs.

How it flows

Fastly service (Logging > HTTPS endpoint)
   │  POST https://<KUBESENSE_INGEST_HOST>/fastlylogs   (JSON array, gzip, batched)
   │  Authorization: Basic <token>
   ▼
KubeSense
   │  checks the token, turns each request into a log
   ▼
Logs page

Fastly sends logs in batches, every few seconds or when a batch fills up. Requests usually show up in KubeSense within a minute.

Before you start

You need:

• Access to the Fastly service that serves your domain, with permission to edit and activate its configuration.
• Two values from KubeSense. Ask your KubeSense contact for them. KubeSense sets up the endpoint on its side.

1. Create the HTTPS endpoint in Fastly

Fastly control panel

1. Open your service and click Edit configuration → Clone version to create a new draft version. An active version can't be edited.
2. Go to Logging and, under HTTPS, click Create endpoint.
3. Fill in the settings:

4. Click Create, then click Activate to activate the new version. The endpoint only starts sending once the version that contains it is active.

Fastly CLI (alternative)

fastly service logging https create \
  --service-id <SERVICE_ID> --version active --autoclone \
  --name "KubeSense" \
  --url "https://<KUBESENSE_INGEST_HOST>/fastlylogs" \
  --content-type "application/json" \
  --header-name "Authorization" \
  --header-value "Basic <token>" \
  --method POST \
  --json-format 1 \
  --message-type blank \
  --compression-codec gzip \
  --request-max-entries 10000 \
  --request-max-bytes 10485760 \
  --format log-format.json

fastly service version activate --service-id <SERVICE_ID> --version latest

--autoclone creates a new draft version from the active one. --json-format 1 is Array of JSON. --format takes the path to a file that holds your log format.

Log format

Use any JSON log format. If you already send Fastly logs to Datadog, reuse that exact format: KubeSense reads the Datadog-style fields (http.*, network.*, dd.trace_id) directly, and trace IDs link the CDN request to its trace.

If you don't have a format yet, this one covers the fields KubeSense uses:

{
  "date": "%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
  "service": "%{req.service_id}V",
  "host": "%{Fastly-Orig-Host}i",
  "server_datacenter": "%{server.datacenter}V",
  "fastly_info_state": "%{fastly_info.state}V",
  "http": {
    "method": "%m",
    "url": "%{cstr_escape(req.url)}V",
    "status_code": %s,
    "request_time_ms": %D,
    "useragent": "%{User-Agent}i",
    "referer": "%{Referer}i",
    "protocol": "%H"
  },
  "network": {
    "client": { "ip": "%h" },
    "geoip": {
      "geo_country_code": "%{client.geo.country_code}V",
      "geo_city": "%{client.geo.city.utf8}V"
    },
    "bytes_written": %B
  }
}

Fastly's own default JSON format works too. Fields KubeSense doesn't recognise are still kept as attributes. Text formats, such as Apache combined, are not supported.

Check that logs arrive

Open a few pages on your site through the Fastly domain. Within a minute, the requests show up on the KubeSense Logs page with Source = fastly.

2. Name the cluster and add tags (optional)

On the Logs page, every log belongs to a cluster. To set the cluster, the environment type or tags such as platform and env, add them to the endpoint URL as query parameters. Each one is optional. Each endpoint can use its own values, so several Fastly services or environments can send to the same KubeSense endpoint and still be told apart.

https://<KUBESENSE_INGEST_HOST>/fastlylogs?kubesense.tags=platform:web,env:prod&kubesense.cluster=cdn-prod

Tags are also attributes, so you can filter on them like any other field. A tag replaces a log field with the same name: env:prod wins over an env field in your log format.

URL-encode values that contain spaces or &, for example %20 for a space. A change takes effect once the new version is activated.

3. Where the logs show up

Fastly logs appear on the Logs page next to your other logs. Filter on Source = fastly to see only them.

Attributes

Every field of your log format becomes an attribute, so you can filter, group and alert on it, using the field name from your log format. Nested fields are flattened with dots, for example http.status_code or network.geoip.geo_country_code. Number fields, such as status codes, timings and byte counts, can be used in numeric filters and aggregations. Empty values ("", -, (null)) are left out.

With the format from Log format, useful attributes include:

With Fastly's default format, the same values are called request_method, url, response_status, response_state and client_ip. Whatever your format calls them, KubeSense still fills in the Body, Level, Workload and Instance from them.

Levels

Things to try

• All CDN errors: filter Source = fastly and Level = ERROR.
• Cache hit ratio: group by fastly_info_state.
• Slowest URLs: sort or aggregate on http.request_time_ms.
• Traffic by POP or country: group by Instance or network.geoip.geo_country_code.
• 5xx spike alerts: alert when the count of Source = fastly and Level = ERROR exceeds a threshold. See Alerts.
• From a slow request to its trace: with dd.trace_id in your log format, open the linked trace from the log.

4. Troubleshooting

Before sending logs, Fastly checks that the endpoint accepts logs from your service by requesting /.well-known/fastly/logging/challenge on the endpoint's host (Fastly's HTTPS logging guide). KubeSense answers this check for you, so there's nothing to set up on your side. If it fails, Fastly reports a challenge check error and sends nothing until the URL is fixed.

Changing the token

If the token has to be changed, for example after it was shared by mistake:

1. Ask your KubeSense contact for a new token.
2. Clone the active version, paste the new token into the endpoint's Custom header value, and activate the new version.

Fastly retries batches that fail, so requests logged while the token is being changed are delivered once the new token is in place.