CloudWatch Logs

Ingesting CloudWatch Logs with KubeSense

KubeSense allows you to ingest logs from Amazon CloudWatch by setting up an Amazon Firehose stream. This integration enables you to centralize your AWS logs alongside your Kubernetes and application observability data.

Prerequisites

Before you begin, ensure you have:

1. Access to AWS Console with permissions to create Firehose streams, IAM roles, and CloudWatch subscription filters
2. KubeSense aggregator deployed with public ingress enabled
3. The KubeSense aggregator Firehose endpoint URL and API token

Setup Process

The integration involves three main steps:

1. Setup a Firehose stream - Configure Amazon Data Firehose to deliver logs to KubeSense
2. Create an IAM role and policy - Set up permissions for CloudWatch to send logs to Firehose
3. Create a subscription filter - Configure CloudWatch log groups to stream to Firehose

Step 1: Setup a Firehose Stream

1. Go to Amazon Data Firehose
2. Click on Create Firehose stream
3. Configure the stream:
   • Source: Select Direct PUT
   • Destination: Select HTTP Endpoint
   • Delivery stream name: Create a name for your stream, for example kubesense-cloudwatch-logs
4. Configure destination settings:
   • HTTP endpoint URL: Enter the KubeSense aggregator Firehose endpoint URL
     • This can be obtained from your KubeSense deployment configuration
     • Format: https://<KUBESENSE_AGGREGATOR_HOST>:<PORT>/firehose
   • Access key: Enter your KubeSense API token
     • This can be retrieved from your KubeSense dashboard or deployment configuration
   • Content encoding: Select GZIP
   • Parameters: Add the following parameter:
     • env_name - Specify your environment name. This will show up in the KubeSense application under this environment
5. Configure backup settings:
   • Choose a backup S3 bucket, or create a new one
   • This ensures logs are backed up in case of delivery failures
6. Click Create Firehose stream

Step 2: Create an IAM Role and Policy

1. Go to Amazon IAM
2. Click on Roles in the sidebar
3. Click on Create Role
4. Select Custom trust policy
5. Paste the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

6. Click Next twice (we'll attach permissions later)
7. Provide a name for the role, for example KubeSenseCloudWatchFirehoseRole
8. Click Create Role

Attach Permissions Policy

1. Go to your newly created role
2. In the Permissions section, click on Add permissions and then Create inline policy
3. Click on JSON and paste the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "<YOUR_FIREHOSE_STREAM_ARN>"
    }
  ]
}

4. Click Next
5. Give the policy a name, for example KubeSenseFirehosePutPolicy
6. Click Create Policy

Step 3: Create a Subscription Filter

Now that the Firehose stream and IAM role are configured, you can add a subscription filter to the desired CloudWatch log group.

Using AWS CLI

The following is an example of how to create a subscription filter through the AWS CLI:

aws logs put-subscription-filter \
  --log-group-name "<LOG_GROUP_NAME>" \
  --filter-name "<FILTER_NAME>" \
  --filter-pattern "" \
  --destination-arn "<FIREHOSE_STREAM_ARN>" \
  --role-arn "<IAM_ROLE_ARN>"

Replace the placeholders:

• <LOG_GROUP_NAME> - The name of your CloudWatch log group
• <FILTER_NAME> - A name for your subscription filter
• <FIREHOSE_STREAM_ARN> - The ARN of your Firehose stream
• <IAM_ROLE_ARN> - The ARN of the IAM role you created in Step 2

Using AWS Console

1. Go to Amazon CloudWatch Logs
2. Navigate to the specific log group you want to stream
3. Click on the Subscription filters tab
4. Click on Create
5. Select Create Amazon Data Firehose subscription filter
6. Select the Firehose delivery stream created in Step 1
7. Select the IAM role created in Step 2
8. Configure log format and filters as needed:
   • Filter pattern: Leave empty to stream all logs, or specify a pattern to filter specific log entries
   • Log format: Select the appropriate format for your logs
9. Choose a name for the subscription filter
10. Click Start streaming

Step 4: Configure KubeSense Aggregator Custom Sources

After setting up the Firehose stream, you need to configure the KubeSense aggregator to accept logs from the Firehose endpoint.

Update KubeSense Aggregator Configuration

Update your KubeSense aggregator configuration to include the Firehose endpoint as a custom source. This can be done through Helm values or configuration files:

Using Helm Values:

aggregator:
  customSources:
    enabled: true
    sources:
      firehose_logs:
        type: aws_kinesis_firehose
        address: 0.0.0.0:443
        access_keys:
          - "<KUBESENSE_API_TOKEN>"

Or update the aggregator deployment directly:

1. Locate your KubeSense aggregator deployment configuration
2. Ensure the aggregator service is exposed with the appropriate port (typically 30052 for logs)
3. Verify the Firehose endpoint is accessible from AWS

Verifying the Integration

After completing all steps, you can verify the integration:

1. Check Firehose metrics: In the AWS Console, navigate to your Firehose stream and check the delivery metrics
2. Check CloudWatch subscription filter: Verify the subscription filter is active and processing logs
3. Check KubeSense logs: In your KubeSense dashboard, navigate to the logs section and verify logs are being ingested from CloudWatch
4. Monitor aggregator logs: Check the KubeSense aggregator logs to ensure it's receiving data from Firehose

Troubleshooting

If logs are not appearing in KubeSense:

1. Verify Firehose stream status: Check that the Firehose stream is active and not throttled
2. Check IAM permissions: Verify the IAM role has the correct permissions to write to Firehose
3. Verify subscription filter: Ensure the subscription filter is active and attached to the correct log group
4. Check network connectivity: Verify that AWS can reach your KubeSense aggregator endpoint
5. Review aggregator logs: Check the KubeSense aggregator logs for any errors or connection issues
6. Verify API token: Ensure the API token configured in Firehose is valid and has the necessary permissions

Best Practices

• Use specific filter patterns: Instead of streaming all logs, use filter patterns to only send relevant logs to reduce costs and improve performance
• Monitor Firehose delivery: Set up CloudWatch alarms to monitor Firehose delivery failures
• Backup configuration: Always configure an S3 backup bucket for Firehose to prevent log loss
• Environment naming: Use descriptive environment names in the env_name parameter to easily identify logs in KubeSense
• Cost optimization: Consider using log group filters to only stream logs that require analysis in KubeSense

Conclusion

With CloudWatch logs integration configured, KubeSense provides a unified platform for analyzing AWS logs alongside your Kubernetes infrastructure and application data. This enables comprehensive observability across your entire technology stack.