VPC Flow Logs

Ingesting VPC Flow Logs with KubeSense

KubeSense supports ingesting AWS VPC Flow Logs to provide network observability, security analysis, and traffic monitoring. VPC Flow Logs capture information about IP traffic going to and from network interfaces in your VPC.

Prerequisites

Before you begin, ensure you have:

VPC Flow Logs enabled in your AWS account
Flow logs configured to deliver to CloudWatch Logs or S3
KubeSense aggregator deployed and accessible
Appropriate AWS permissions to configure flow logs

Supported Delivery Methods

VPC Flow Logs can be ingested via two methods:

CloudWatch Logs → Firehose → KubeSense Aggregator (Real-time)
S3 → KubeSense Aggregator (Batch/Historical)

Method 1: CloudWatch Logs via Firehose (Recommended for Real-time)

This method provides real-time ingestion of VPC Flow Logs.

Step 1: Enable VPC Flow Logs to CloudWatch

Go to Amazon VPC Console
Select your VPC
Click on Flow logs tab
Click Create flow log
Configure:
- Filter: All (or specific traffic types)
- Destination: Send to CloudWatch Logs
- Destination log group: Create or select a log group (e.g., /aws/vpc/flowlogs)
- IAM role: Create or select an IAM role with permissions to write to CloudWatch Logs

Step 2: Create Firehose Stream

Follow the same process as CloudWatch Logs integration, but configure it for the VPC Flow Logs log group:

Create a Firehose stream with destination as HTTP Endpoint
Set the endpoint URL to your KubeSense aggregator Firehose endpoint
Configure the subscription filter for the VPC Flow Logs log group

Step 3: Configure KubeSense Aggregator

The aggregator will automatically parse VPC Flow Log format. Ensure the Firehose endpoint is configured:

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs:
        type: aws_kinesis_firehose
        address: 0.0.0.0:443
        access_keys:
          - "<KUBESENSE_API_TOKEN>"

Method 2: S3 Batch Ingestion

For historical analysis or batch processing of VPC Flow Logs stored in S3:

Step 1: Enable VPC Flow Logs to S3

Go to Amazon VPC Console
Select your VPC
Click on Flow logs tab
Click Create flow log
Configure:
- Filter: All (or specific traffic types)
- Destination: Send to S3
- S3 bucket ARN: Your S3 bucket for flow logs
- Log record format: Default or custom format

Step 2: Configure S3 Ingestion

Configure the KubeSense aggregator to read from the S3 bucket:

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs_s3:
        type: aws_s3
        region: us-east-1
        bucket: vpc-flow-logs-bucket
        key_prefix: AWSLogs/123456789012/vpcflowlogs/
        poll_interval_secs: 300
        auth:
          access_key_id: "<AWS_ACCESS_KEY_ID>"
          secret_access_key: "<AWS_SECRET_ACCESS_KEY>"

See S3 Log Archives for detailed S3 configuration.

VPC Flow Log Format

VPC Flow Logs are automatically parsed by KubeSense aggregator. The standard format includes:

version - Flow log version
account-id - AWS account ID
interface-id - Network interface ID
srcaddr - Source IP address
dstaddr - Destination IP address
srcport - Source port
dstport - Destination port
protocol - IP protocol number
packets - Number of packets
bytes - Number of bytes
start - Start time (Unix timestamp)
end - End time (Unix timestamp)
action - ACCEPT or REJECT
log-status - Status of the flow log

Use Cases

Network Security Analysis

Monitor and analyze network traffic for security threats:

Detect unusual traffic patterns
Identify potential security breaches
Track denied connections
Monitor port scanning attempts

Traffic Monitoring

Understand network traffic patterns:

Top talkers (highest bandwidth consumers)
Protocol distribution
Geographic traffic patterns
Peak usage times

Cost Optimization

Analyze network costs:

Identify high-bandwidth consumers
Optimize data transfer costs
Monitor cross-AZ traffic
Track internet egress

Configuration Example

Complete Helm configuration for VPC Flow Logs via Firehose:

global:
  cluster_name: "production-cluster"

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs:
        type: aws_kinesis_firehose
        address: 0.0.0.0:443
        access_keys:
          - "<KUBESENSE_API_TOKEN>"

Monitoring and Verification

Check Firehose metrics: Verify logs are being delivered to KubeSense
Verify log parsing: Check that VPC Flow Log fields are correctly parsed
Monitor network dashboards: View network traffic visualizations in KubeSense
Check aggregator logs: Ensure no parsing errors

Troubleshooting

Flow Logs Not Appearing

Verify VPC Flow Logs enabled: Check that flow logs are active in the VPC console
Check CloudWatch log group: Verify logs are being written to CloudWatch
Verify Firehose subscription: Ensure subscription filter is active
Check IAM permissions: Verify all IAM roles have correct permissions
Review aggregator logs: Check for parsing or ingestion errors

Parsing Issues

Verify log format: Ensure flow logs are in standard format
Check aggregator version: Update aggregator if using custom flow log formats
Review sample logs: Verify log structure matches expected format

Best Practices

Use appropriate filters: Filter flow logs to reduce volume and costs (e.g., only REJECT, or specific subnets)
Enable for all VPCs: Enable flow logs for all VPCs for comprehensive visibility
Monitor costs: VPC Flow Logs can generate significant CloudWatch costs
Set retention: Configure appropriate retention periods for CloudWatch Logs
Use S3 for long-term: Archive old flow logs to S3 for cost optimization
Tag resources: Use tags to identify and filter flow logs by environment or application

Conclusion

VPC Flow Logs integration provides comprehensive network observability, enabling security analysis, traffic monitoring, and cost optimization across your AWS infrastructure.

VPC Flow Logs

ON THIS PAGE