GKE Control Plane Logs

Ingesting GKE Control Plane Logs with KubeSense

KubeSense supports ingesting GKE (Google Kubernetes Engine) control plane logs to provide comprehensive observability into cluster management, API server activities, and control plane operations.

Prerequisites

Before you begin, ensure you have:

GKE cluster running
Cloud Logging API enabled
KubeSense aggregator deployed with public ingress enabled
Appropriate GCP IAM permissions to create log sinks and Pub/Sub resources

Architecture

GKE control plane logs flow through the following path:

GKE Control Plane → Cloud Logging → Log Sink → Pub/Sub Topic → Pub/Sub Subscription → KubeSense Aggregator

Step 1: Enable Control Plane Logging

GKE control plane logs are automatically sent to Cloud Logging. Ensure logging is enabled:

Verify Logging Components

GKE control plane logs include:

API server logs - Kubernetes API server activities
Scheduler logs - Pod scheduling decisions
Controller manager logs - Controller operations
Audit logs - API audit events

Check Log Categories

Verify which log categories are enabled:

gcloud container clusters describe CLUSTER_NAME \
  --zone=ZONE \
  --format="value(loggingService,monitoringService)" \
  --project=YOUR_PROJECT_ID

Step 2: Create Pub/Sub Topic

Create a Pub/Sub topic for GKE control plane logs:

gcloud pubsub topics create gke-control-plane-logs \
  --project=YOUR_PROJECT_ID

Step 3: Create Log Sink

Create a log sink to export GKE control plane logs to Pub/Sub:

Using gcloud CLI

# Create log sink for API server logs
gcloud logging sinks create gke-api-server-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/gke-control-plane-logs \
  --log-filter='resource.type="k8s_cluster" AND resource.labels.cluster_name="CLUSTER_NAME" AND jsonPayload.kind="Event" OR jsonPayload.verb' \
  --project=YOUR_PROJECT_ID

# Grant permissions
SERVICE_ACCOUNT=$(gcloud logging sinks describe gke-api-server-sink \
  --project=YOUR_PROJECT_ID \
  --format="value(writerIdentity)")

gcloud pubsub topics add-iam-policy-binding gke-control-plane-logs \
  --member="$SERVICE_ACCOUNT" \
  --role="roles/pubsub.publisher" \
  --project=YOUR_PROJECT_ID

Log Filter Examples

Filter by specific control plane components:

# API server logs only
resource.type="k8s_cluster" AND 
resource.labels.cluster_name="CLUSTER_NAME" AND 
jsonPayload.verb

# Scheduler logs
resource.type="k8s_cluster" AND 
resource.labels.cluster_name="CLUSTER_NAME" AND 
jsonPayload.component="kube-scheduler"

# Controller manager logs
resource.type="k8s_cluster" AND 
resource.labels.cluster_name="CLUSTER_NAME" AND 
jsonPayload.component="kube-controller-manager"

# Audit logs
resource.type="k8s_cluster" AND 
resource.labels.cluster_name="CLUSTER_NAME" AND 
logName=~"k8s-audit"

Step 4: Create Pub/Sub Subscription

Create a push subscription to forward logs to KubeSense:

gcloud pubsub subscriptions create gke-control-plane-subscription \
  --topic=gke-control-plane-logs \
  --push-endpoint=https://<KUBESENSE_AGGREGATOR_HOST>:<PORT>/pubsub \
  --push-auth-token=<KUBESENSE_API_TOKEN> \
  --project=YOUR_PROJECT_ID

Step 5: Configure KubeSense Aggregator

Configure the aggregator to handle GKE control plane logs:

aggregator:
  customSources:
    enabled: true
    sources:
      gke_control_plane_logs:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/gke-control-plane-subscription
        credentials_path: /etc/kubesense/gcs-key.json

Update Helm Values

If deploying via Helm:

global:
  cluster_name: "gke-cluster"

aggregator:
  customSources:
    enabled: true
    sources:
      gke_control_plane_logs:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/gke-control-plane-subscription
        credentials_path: /etc/kubesense/gcs-key.json

Log Types and Use Cases

API Server Logs

Monitor Kubernetes API activities:

API requests: Track all API requests to the cluster
Authentication: Monitor authentication events
Authorization: Track authorization decisions
Resource operations: Monitor create, update, delete operations

Audit Logs

Comprehensive audit trail:

User actions: Track all user-initiated actions
Resource changes: Monitor all resource modifications
Access patterns: Analyze access patterns and anomalies
Compliance: Maintain compliance audit trails

Scheduler Logs

Pod scheduling insights:

Scheduling decisions: Understand why pods are scheduled to specific nodes
Resource constraints: Identify resource constraints affecting scheduling
Performance: Monitor scheduling performance

Controller Manager Logs

Controller operations:

Replica set management: Monitor replica set scaling
Deployment updates: Track deployment rollouts
State management: Monitor desired vs actual state reconciliation

Configuration Examples

Multiple Clusters

Configure control plane logs for multiple GKE clusters:

aggregator:
  customSources:
    enabled: true
    sources:
      gke_control_plane_prod:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/gke-control-plane-prod
        credentials_path: /etc/kubesense/gcs-key.json
      gke_control_plane_dev:
        type: gcp_pubsub
        project: YOUR_PROJECT_ID
        subscription: projects/YOUR_PROJECT_ID/subscriptions/gke-control-plane-dev
        credentials_path: /etc/kubesense/gcs-key.json

Filtered Logs

Create separate sinks for different log types:

# API server logs
gcloud logging sinks create gke-api-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/gke-api-logs \
  --log-filter='resource.type="k8s_cluster" AND jsonPayload.verb' \
  --project=YOUR_PROJECT_ID

# Audit logs
gcloud logging sinks create gke-audit-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/gke-audit-logs \
  --log-filter='resource.type="k8s_cluster" AND logName=~"k8s-audit"' \
  --project=YOUR_PROJECT_ID

Monitoring and Verification

Check Cloud Logging: Verify control plane logs are being written
Monitor Pub/Sub: Check message count and delivery metrics
Verify subscription: Ensure subscription is active and delivering messages
Check KubeSense dashboard: Verify logs appear with GKE metadata
Review aggregator logs: Check for any ingestion errors

Troubleshooting

Logs Not Appearing

Verify logging enabled: Check that logging is enabled on the GKE cluster
Check log sink: Verify the sink is active and exporting logs
Verify Pub/Sub topic: Ensure messages are being published
Check IAM permissions: Verify service account has Pub/Sub publisher role
Review network connectivity: Ensure GCP can reach KubeSense aggregator
Check aggregator logs: Review aggregator logs for connection or parsing errors

Missing Log Types

Verify log filters: Ensure log filters match the log types you want
Check GKE version: Some log types may require specific GKE versions
Review log categories: Verify which log categories are enabled on the cluster

Performance Issues

Filter logs: Use more specific log filters to reduce volume
Adjust batch size: Configure Pub/Sub batch settings
Scale aggregator: Increase aggregator resources if needed

Best Practices

Use specific filters: Filter logs at the sink level to reduce volume and costs
Separate by cluster: Create separate topics/sinks for different clusters
Monitor costs: Control plane logs can generate significant Cloud Logging costs
Set retention: Configure appropriate retention periods
Use structured queries: Leverage Cloud Logging query syntax for better filtering
Tag resources: Use GCP labels for better log organization
Monitor quotas: Be aware of Pub/Sub quotas and limits

Cost Considerations

Cloud Logging: Charged per GB ingested and stored
Pub/Sub messages: Charged per million messages
Data transfer: Consider data transfer costs between GCP and KubeSense
Control plane logs: Can be high volume, especially audit logs

Security Considerations

Sensitive data: Control plane logs may contain sensitive information
Access control: Ensure proper access control for log sinks and subscriptions
Encryption: Use encrypted Pub/Sub topics for sensitive logs
Audit compliance: Maintain audit logs for compliance requirements

Conclusion

GKE control plane logs integration provides comprehensive observability into cluster management, enabling better troubleshooting, security analysis, and compliance monitoring for your GKE clusters.

GKE Control Plane Logs

ON THIS PAGE