Kubesense
Integrations/AWS Integrations/EKS Fargate Logs

EKS Fargate Logs

Ingesting EKS Fargate Logs with KubeSense

KubeSense supports ingesting logs from Amazon EKS Fargate pods. Since Fargate pods don't have node-level access, logs are typically delivered via CloudWatch Logs and then forwarded to KubeSense aggregator through Amazon Firehose.

Prerequisites

Before you begin, ensure you have:

  1. EKS Fargate cluster running
  2. Pods configured to send logs to CloudWatch Logs
  3. KubeSense aggregator deployed with public ingress enabled
  4. AWS permissions to create Firehose streams and subscription filters

Architecture

EKS Fargate logs flow through the following path:

Fargate Pods → CloudWatch Logs → Firehose → KubeSense Aggregator

Step 1: Configure Fargate Pod Logging

EKS Fargate automatically sends container logs to CloudWatch Logs. Ensure your pods are configured correctly:

Verify Log Group

Fargate creates log groups in the format:

/aws/eks/<cluster-name>/cluster
/aws/containerinsights/<cluster-name>/application

Check Pod Log Configuration

Ensure your pods have proper logging configuration. Fargate automatically captures stdout/stderr from containers.

Step 2: Create Firehose Stream

Create a Firehose stream to forward CloudWatch Logs to KubeSense:

  1. Go to Amazon Data Firehose
  2. Click Create Firehose stream
  3. Configure:
    • Source: Direct PUT
    • Destination: HTTP Endpoint
    • Delivery stream name: kubesense-eks-fargate-logs
    • HTTP endpoint URL: https://<KUBESENSE_AGGREGATOR_HOST>:<PORT>/firehose
    • Access key: Your KubeSense API token
    • Content encoding: GZIP
    • Parameters: Add env_name parameter (e.g., eks-fargate-prod)

Step 3: Create IAM Role for CloudWatch

Create an IAM role that allows CloudWatch to write to Firehose:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Attach a policy allowing Firehose write access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "arn:aws:firehose:region:account:deliverystream/kubesense-eks-fargate-logs"
    }
  ]
}

Step 4: Create Subscription Filters

Create subscription filters for EKS Fargate log groups:

Using AWS CLI

# For cluster logs
aws logs put-subscription-filter \
  --log-group-name "/aws/eks/<cluster-name>/cluster" \
  --filter-name "kubesense-fargate-cluster" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:region:account:deliverystream/kubesense-eks-fargate-logs" \
  --role-arn "arn:aws:iam::account:role/KubeSenseCloudWatchFirehoseRole"

# For application logs
aws logs put-subscription-filter \
  --log-group-name "/aws/containerinsights/<cluster-name>/application" \
  --filter-name "kubesense-fargate-app" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:region:account:deliverystream/kubesense-eks-fargate-logs" \
  --role-arn "arn:aws:iam::account:role/KubeSenseCloudWatchFirehoseRole"

Using AWS Console

  1. Go to CloudWatch Logs
  2. Navigate to the EKS Fargate log groups
  3. Click Subscription filters tab
  4. Click Create
  5. Select Create Amazon Data Firehose subscription filter
  6. Select your Firehose stream
  7. Select the IAM role
  8. Click Start streaming

Step 5: Configure KubeSense Aggregator

Configure the aggregator to handle EKS Fargate logs:

aggregator:
  customSources:
    enabled: true
    sources:
      eks_fargate_logs:
        type: aws_kinesis_firehose
        address: 0.0.0.0:443
        access_keys:
          - "<KUBESENSE_API_TOKEN>"

Alternative: LogSensor Sidecar (Experimental)

For more direct control, you can deploy LogSensor as a sidecar container in Fargate pods:

Pod Configuration Example

apiVersion: v1
kind: Pod
metadata:
  name: fargate-app-with-logsensor
spec:
  containers:
    - name: app
      image: your-app:latest
    - name: logsensor
      image: kubesense/logsensor:latest
      env:
        - name: AGGREGATOR_HOST
          value: "<KUBESENSE_AGGREGATOR_HOST>"
        - name: AGGREGATOR_PORT
          value: "30052"
        - name: CLUSTER_NAME
          value: "<cluster-name>"
        - name: LOG_PATHS
          value: "/var/log/containers"
      volumeMounts:
        - name: varlog
          mountPath: /var/log
      resources:
        requests:
          memory: "128Mi"
          cpu: "100m"
        limits:
          memory: "256Mi"
          cpu: "200m"
  volumes:
    - name: varlog
      emptyDir: {}

Note: LogSensor sidecar in Fargate is experimental and may have limitations. The CloudWatch → Firehose method is recommended for production use.

Log Enrichment

KubeSense aggregator automatically enriches EKS Fargate logs with:

  • Pod metadata: Pod name, namespace, labels
  • Container information: Container name, image
  • Cluster context: Cluster name, region
  • Fargate metadata: Task ID, platform version

Monitoring and Verification

  1. Check CloudWatch Logs: Verify logs are being written to CloudWatch
  2. Monitor Firehose: Check Firehose delivery metrics in AWS Console
  3. Verify subscription filters: Ensure filters are active
  4. Check KubeSense dashboard: Verify logs appear in KubeSense with proper metadata
  5. Review aggregator logs: Check for any ingestion errors

Troubleshooting

Logs Not Appearing

  1. Verify Fargate logging: Check that pods are generating logs
  2. Check CloudWatch log groups: Ensure logs are being written
  3. Verify Firehose stream: Check stream status and metrics
  4. Review subscription filters: Ensure filters are active and correct
  5. Check IAM permissions: Verify all roles have correct permissions
  6. Review aggregator logs: Check for connection or parsing errors

Missing Metadata

  1. Verify log format: Ensure logs include Kubernetes metadata
  2. Check enrichment config: Verify aggregator enrichment is enabled
  3. Review log structure: Check that logs match expected format

Best Practices

  • Use descriptive env_name: Set meaningful environment names in Firehose parameters
  • Filter logs: Use subscription filter patterns to only send relevant logs
  • Monitor costs: CloudWatch and Firehose can generate significant costs
  • Set retention: Configure appropriate CloudWatch Logs retention
  • Use log groups: Organize logs by namespace or application using log group prefixes
  • Monitor Firehose: Set up CloudWatch alarms for Firehose delivery failures

Cost Considerations

  • CloudWatch Logs: Charged per GB ingested and stored
  • Firehose: Charged per GB processed
  • Data transfer: Consider data transfer costs between AWS and KubeSense
  • Optimize with filters: Use filter patterns to reduce log volume

Conclusion

EKS Fargate logs integration enables comprehensive observability for serverless Kubernetes workloads, providing the same rich log analysis capabilities as traditional EKS deployments.

back

VPC Flow Logs

next

ECS Logs