VPC Flow Logs

Ingesting GCP VPC Flow Logs with KubeSense

KubeSense supports ingesting GCP VPC Flow Logs to provide network observability, security analysis, and traffic monitoring. VPC Flow Logs capture information about IP traffic going to and from network interfaces in your VPC.

Prerequisites

Before you begin, ensure you have:

VPC Flow Logs enabled in your GCP project
Flow logs configured to deliver to Cloud Logging or Cloud Storage
KubeSense aggregator deployed and accessible
Appropriate GCP permissions to configure flow logs and access logs

Supported Delivery Methods

VPC Flow Logs can be ingested via two methods:

Cloud Logging → Pub/Sub → KubeSense Aggregator (Real-time)
Cloud Storage (GCS) → KubeSense Aggregator (Batch/Historical)

Method 1: Cloud Logging via Pub/Sub (Recommended for Real-time)

This method provides real-time ingestion of VPC Flow Logs.

Step 1: Enable VPC Flow Logs to Cloud Logging

Enable VPC Flow Logs for your VPC:

# Enable flow logs for a VPC
gcloud compute networks update VPC_NAME \
  --enable-flow-logs \
  --flow-log-sampling=1.0 \
  --flow-log-metadata=INCLUDE_ALL_METADATA \
  --project=YOUR_PROJECT_ID

Or using Terraform:

resource "google_compute_network" "vpc" {
  name                    = "my-vpc"
  auto_create_subnetworks = false
  enable_flow_logs       = true
}

resource "google_compute_subnetwork" "subnet" {
  name          = "my-subnet"
  ip_cidr_range = "10.0.0.0/24"
  network       = google_compute_network.vpc.id
  enable_flow_logs = true
}

Step 2: Create Log Sink to Pub/Sub

Create a log sink to export VPC Flow Logs to Pub/Sub:

# Create Pub/Sub topic
gcloud pubsub topics create vpc-flow-logs \
  --project=YOUR_PROJECT_ID

# Create log sink
gcloud logging sinks create vpc-flow-logs-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/vpc-flow-logs \
  --log-filter='resource.type="gce_subnetwork" AND jsonPayload.flow_logs="true"' \
  --project=YOUR_PROJECT_ID

# Grant permissions
SERVICE_ACCOUNT=$(gcloud logging sinks describe vpc-flow-logs-sink \
  --project=YOUR_PROJECT_ID \
  --format="value(writerIdentity)")

gcloud pubsub topics add-iam-policy-binding vpc-flow-logs \
  --member="$SERVICE_ACCOUNT" \
  --role="roles/pubsub.publisher" \
  --project=YOUR_PROJECT_ID

Step 3: Create Pub/Sub Subscription

Create a push subscription to forward logs to KubeSense:

gcloud pubsub subscriptions create vpc-flow-logs-subscription \
  --topic=vpc-flow-logs \
  --push-endpoint=https://<KUBESENSE_AGGREGATOR_HOST>:<PORT>/pubsub \
  --push-auth-token=<KUBESENSE_API_TOKEN> \
  --project=YOUR_PROJECT_ID

Step 4: Configure KubeSense Aggregator

Configure the aggregator to handle VPC Flow Logs:

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/vpc-flow-logs-subscription
        credentials_path: /etc/kubesense/gcs-key.json

Method 2: GCS Batch Ingestion

For historical analysis or batch processing of VPC Flow Logs stored in GCS:

Step 1: Enable VPC Flow Logs to GCS

Configure VPC Flow Logs to export to Cloud Storage:

# Create GCS bucket
gsutil mb -p YOUR_PROJECT_ID gs://vpc-flow-logs-bucket

# Enable flow logs with GCS export (requires API or Terraform)
# Note: GCP VPC Flow Logs primarily use Cloud Logging, but you can export to GCS

Step 2: Export Cloud Logging to GCS

Export VPC Flow Logs from Cloud Logging to GCS:

# Create log sink to GCS
gcloud logging sinks create vpc-flow-logs-gcs-sink \
  storage.googleapis.com/vpc-flow-logs-bucket \
  --log-filter='resource.type="gce_subnetwork" AND jsonPayload.flow_logs="true"' \
  --project=YOUR_PROJECT_ID

Step 3: Configure GCS Ingestion

Configure the KubeSense aggregator to read from the GCS bucket via Pub/Sub:

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs_gcs:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/vpc-flow-logs-gcs-subscription
        credentials_path: /etc/kubesense/gcs-key.json

Set up GCS bucket notification:

gsutil notification create -t vpc-flow-logs-notifications \
  -f json \
  -e OBJECT_FINALIZE \
  -p vpc-flow-logs/ \
  gs://vpc-flow-logs-bucket

See GCS Log Archives for detailed GCS configuration.

VPC Flow Log Format

GCP VPC Flow Logs are automatically parsed by KubeSense aggregator. The format includes:

connection - Connection information
src_instance - Source instance details
src_ip - Source IP address
src_port - Source port
dest_instance - Destination instance details
dest_ip - Destination IP address
dest_port - Destination port
protocol - IP protocol
bytes_sent - Bytes sent
packets_sent - Packets sent
rtt_msec - Round-trip time in milliseconds
reporter - Reporter (SRC or DST)
start_time - Flow start time
end_time - Flow end time

Use Cases

Network Security Analysis

Monitor and analyze network traffic for security threats:

Detect unusual traffic patterns
Identify potential security breaches
Track denied connections
Monitor port scanning attempts
Analyze cross-VPC traffic

Traffic Monitoring

Understand network traffic patterns:

Top talkers (highest bandwidth consumers)
Protocol distribution
Geographic traffic patterns
Peak usage times
Inter-zone traffic analysis

Cost Optimization

Analyze network costs:

Identify high-bandwidth consumers
Optimize data transfer costs
Monitor cross-region traffic
Track internet egress
Analyze inter-VPC traffic

Configuration Example

Complete Helm configuration for VPC Flow Logs via Pub/Sub:

global:
  cluster_name: "gcp-cluster"

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/vpc-flow-logs-subscription
        credentials_path: /etc/kubesense/gcs-key.json

Monitoring and Verification

Check VPC Flow Logs: Verify flow logs are enabled and generating data
Monitor Pub/Sub: Verify logs are being published to Pub/Sub topic
Verify subscription: Ensure subscription is active and delivering messages
Check KubeSense dashboard: View network traffic visualizations
Review aggregator logs: Ensure no parsing errors

Troubleshooting

Flow Logs Not Appearing

Verify VPC Flow Logs enabled: Check that flow logs are enabled on VPC/subnet
Check Cloud Logging: Verify logs are being written to Cloud Logging
Verify log sink: Ensure log sink is active and exporting to Pub/Sub
Check IAM permissions: Verify all service accounts have correct permissions
Review aggregator logs: Check for parsing or ingestion errors
Check sampling rate: Verify flow log sampling is configured correctly

Parsing Issues

Verify log format: Ensure flow logs are in expected format
Check aggregator version: Update aggregator if using newer flow log formats
Review sample logs: Verify log structure matches expected format
Check metadata: Ensure flow log metadata is included

Best Practices

Enable for all subnets: Enable flow logs for all subnets for comprehensive visibility
Use appropriate sampling: Adjust sampling rate based on traffic volume and cost
Monitor costs: VPC Flow Logs can generate significant Cloud Logging costs
Set retention: Configure appropriate retention periods for Cloud Logging
Use GCS for long-term: Export old flow logs to GCS for cost optimization
Tag resources: Use GCP labels to identify and filter flow logs by environment or application
Filter at sink level: Use log sink filters to reduce volume and costs

Cost Considerations

Cloud Logging: Charged per GB ingested and stored
Pub/Sub: Charged per million messages
GCS storage: Charged per GB stored (if using GCS export)
Data transfer: Consider data transfer costs between GCP and KubeSense

Advanced Configuration

Filter Flow Logs

Use log sink filters to reduce volume:

# Only export specific subnets
gcloud logging sinks create vpc-flow-logs-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/vpc-flow-logs \
  --log-filter='resource.type="gce_subnetwork" AND resource.labels.subnetwork_name="production-subnet"' \
  --project=YOUR_PROJECT_ID

Multiple VPCs

Configure flow logs for multiple VPCs:

aggregator:
  customSources:
    enabled: true
    sources:
      vpc_flow_logs_prod:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/vpc-flow-logs-prod
        credentials_path: /etc/kubesense/gcs-key.json
      vpc_flow_logs_dev:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/vpc-flow-logs-dev
        credentials_path: /etc/kubesense/gcs-key.json

Conclusion

GCP VPC Flow Logs integration provides comprehensive network observability, enabling security analysis, traffic monitoring, and cost optimization across your GCP infrastructure.

VPC Flow Logs

ON THIS PAGE