Kubesense
Integrations/AWS Integrations/Load Balancer Access Logs

Load Balancer Access Logs

Ingesting Load Balancer Access Logs with KubeSense

KubeSense supports ingesting access logs from AWS Application Load Balancers (ALB), Network Load Balancers (NLB), and Classic Load Balancers (ELB). These logs provide valuable insights into traffic patterns, request/response details, and security events.

Prerequisites

Before you begin, ensure you have:

  1. AWS Load Balancer (ALB, NLB, or ELB) with access logging enabled
  2. S3 bucket configured for load balancer access logs
  3. KubeSense aggregator deployed and accessible
  4. AWS IAM credentials with read access to the S3 bucket

Supported Load Balancers

  • Application Load Balancer (ALB)
  • Network Load Balancer (NLB)
  • Classic Load Balancer (ELB)

Step 1: Enable Access Logging

For Application Load Balancer (ALB)

  1. Go to EC2 Console → Load Balancers
  2. Select your ALB
  3. Click Attributes tab
  4. Click Edit under Access logs
  5. Enable access logs and configure:
    • S3 location: Your S3 bucket (e.g., s3://alb-access-logs/my-alb/)
    • Prefix: Optional prefix for log organization
  6. Click Save

For Network Load Balancer (NLB)

  1. Go to EC2 Console → Load Balancers
  2. Select your NLB
  3. Click Attributes tab
  4. Click Edit under Access logs
  5. Enable access logs and configure S3 bucket
  6. Click Save

For Classic Load Balancer (ELB)

  1. Go to EC2 Console → Load Balancers
  2. Select your ELB
  3. Click Attributes tab
  4. Enable access logs and configure S3 bucket
  5. Click Save

Step 2: Configure S3 Bucket Permissions

Ensure the load balancer can write to S3. The bucket policy should allow:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logdelivery.elasticloadbalancing.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::alb-access-logs/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logdelivery.elasticloadbalancing.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::alb-access-logs/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

Step 3: Configure KubeSense Aggregator for S3

Configure the aggregator to read load balancer logs from S3:

aggregator:
  customSources:
    enabled: true
    sources:
      alb_access_logs:
        type: aws_s3
        region: us-east-1
        bucket: alb-access-logs
        key_prefix: AWSLogs/123456789012/elasticloadbalancing/
        poll_interval_secs: 300
        auth:
          access_key_id: "<AWS_ACCESS_KEY_ID>"
          secret_access_key: "<AWS_SECRET_ACCESS_KEY>"

Log Format Parsing

KubeSense aggregator automatically parses load balancer log formats:

ALB Access Log Format

Fields include:

  • type - Log type (always "http")
  • time - Request timestamp
  • elb - Load balancer ID
  • client:port - Client IP and port
  • target:port - Target IP and port
  • request_processing_time - Time to process request
  • target_processing_time - Time target took to respond
  • response_processing_time - Time to process response
  • elb_status_code - ELB response code
  • target_status_code - Target response code
  • received_bytes - Bytes received
  • sent_bytes - Bytes sent
  • request - HTTP request line
  • user_agent - User agent string
  • ssl_cipher - SSL cipher
  • ssl_protocol - SSL protocol
  • target_group_arn - Target group ARN
  • trace_id - Request trace ID
  • domain_name - Request domain
  • chosen_cert_arn - Certificate ARN
  • matched_rule_priority - Matched rule priority
  • request_creation_time - Request creation time
  • actions_executed - Actions executed
  • redirect_url - Redirect URL
  • error_reason - Error reason

NLB Access Log Format

Similar to ALB but with NLB-specific fields.

ELB Access Log Format

Classic ELB format with different field structure.

Use Cases

Traffic Analysis

  • Request patterns: Analyze request volume, patterns, and trends
  • Geographic distribution: Understand traffic sources
  • Peak usage: Identify peak traffic times
  • Protocol distribution: Monitor HTTP vs HTTPS usage

Security Monitoring

  • Failed requests: Identify potential attacks or misconfigurations
  • Suspicious patterns: Detect unusual request patterns
  • **SSL issues: Identify SSL/TLS issues
  • Error analysis: Monitor error rates and types

Performance Monitoring

  • Response times: Track target processing times
  • Latency analysis: Identify slow targets
  • Throughput: Monitor bytes sent/received
  • Target health: Correlate with target health checks

Cost Optimization

  • Data transfer: Monitor data transfer volumes
  • Request distribution: Optimize target distribution
  • Idle connections: Identify and optimize idle connections

Configuration Examples

ALB Logs Only

aggregator:
  customSources:
    enabled: true
    sources:
      alb_logs:
        type: aws_s3
        region: us-east-1
        bucket: alb-access-logs
        key_prefix: AWSLogs/123456789012/elasticloadbalancing/us-east-1/
        poll_interval_secs: 300
        auth:
          access_key_id: "<AWS_ACCESS_KEY_ID>"
          secret_access_key: "<AWS_SECRET_ACCESS_KEY>"

Multiple Load Balancers

aggregator:
  customSources:
    enabled: true
    sources:
      alb_logs:
        type: aws_s3
        region: us-east-1
        bucket: alb-access-logs
        key_prefix: AWSLogs/123456789012/elasticloadbalancing/us-east-1/
        auth:
          access_key_id: "<AWS_ACCESS_KEY_ID>"
          secret_access_key: "<AWS_SECRET_ACCESS_KEY>"
      nlb_logs:
        type: aws_s3
        region: us-east-1
        bucket: nlb-access-logs
        key_prefix: AWSLogs/123456789012/elasticloadbalancing/us-east-1/
        auth:
          access_key_id: "<AWS_ACCESS_KEY_ID>"
          secret_access_key: "<AWS_SECRET_ACCESS_KEY>"

Real-time Processing

For near-real-time processing, use CloudWatch Logs subscription (if available) or reduce poll interval:

aggregator:
  customSources:
    enabled: true
    sources:
      alb_logs_realtime:
        type: aws_s3
        region: us-east-1
        bucket: alb-access-logs
        key_prefix: AWSLogs/123456789012/elasticloadbalancing/
        poll_interval_secs: 60
        auth:
          access_key_id: "<AWS_ACCESS_KEY_ID>"
          secret_access_key: "<AWS_SECRET_ACCESS_KEY>"

Monitoring and Verification

  1. Check S3 bucket: Verify logs are being written to S3
  2. Monitor aggregator: Check aggregator logs for S3 polling activity
  3. Verify parsing: Check that log fields are correctly parsed in KubeSense
  4. Check dashboards: View load balancer metrics and logs in KubeSense
  5. Review errors: Check for any parsing or ingestion errors

Troubleshooting

Logs Not Appearing

  1. Verify access logging: Ensure access logging is enabled on the load balancer
  2. Check S3 permissions: Verify load balancer can write to S3
  3. Verify S3 path: Check that the S3 prefix matches actual log location
  4. Check IAM credentials: Ensure aggregator has read access to S3
  5. Review aggregator logs: Check for S3 access errors

Parsing Issues

  1. Verify log format: Ensure logs match expected format for load balancer type
  2. Check aggregator version: Update aggregator if using newer log formats
  3. Review sample logs: Verify log structure matches expected format
  4. Check encoding: Ensure logs are in expected encoding (usually UTF-8)

Performance Issues

  1. Adjust poll interval: Increase interval for large log volumes
  2. Use prefixes: Narrow down to specific load balancer prefixes
  3. Enable compression: Use compressed logs if available
  4. Batch processing: Process logs in batches

Best Practices

  • Organize by prefix: Use S3 prefixes to organize logs by load balancer, date, or environment
  • Set retention: Configure S3 lifecycle policies for log retention
  • Monitor costs: Track S3 storage and API call costs
  • Use compression: Enable S3 compression to reduce storage costs
  • Filter logs: Process only relevant logs to reduce processing overhead
  • Regular monitoring: Set up alerts for log ingestion failures
  • Archive old logs: Move old logs to Glacier for cost optimization

Cost Considerations

  • S3 storage: Charged per GB stored
  • S3 requests: Charged per API call (ListBucket, GetObject)
  • Data transfer: Consider data transfer costs
  • Processing: Aggregator processing resources

Conclusion

Load balancer access logs integration provides comprehensive visibility into traffic patterns, security events, and performance metrics, enabling better decision-making and troubleshooting for your AWS infrastructure.

back

ECS Logs

next

AWS Lambda Logs (Go)