Kubesense
Integrations/Azure Integrations/AKS Control Plane Logs

AKS Control Plane Logs

Ingesting AKS Control Plane Logs with KubeSense

KubeSense supports ingesting AKS (Azure Kubernetes Service) control plane logs to provide comprehensive observability into cluster management, API server activities, and control plane operations.

Prerequisites

Before you begin, ensure you have:

  1. AKS cluster running
  2. Azure Monitor Diagnostic Settings enabled
  3. Event Hub or Log Analytics Workspace configured
  4. KubeSense aggregator deployed and accessible
  5. Appropriate Azure permissions to configure diagnostic settings

Architecture

AKS control plane logs flow through the following path:

AKS Control Plane → Diagnostic Settings → Event Hub/Log Analytics → KubeSense Aggregator

Step 1: Enable AKS Control Plane Logging

Enable diagnostic settings on your AKS cluster to send control plane logs to Event Hub or Log Analytics.

Using Azure CLI

# Get AKS cluster resource ID
AKS_CLUSTER_ID=$(az aks show \
  --resource-group kubesense-rg \
  --name my-aks-cluster \
  --query id -o tsv)

# Get Event Hub authorization rule ID
EVENT_HUB_ID=$(az eventhubs eventhub authorization-rule show \
  --resource-group kubesense-rg \
  --namespace-name kubesense-logs-namespace \
  --eventhub-name aks-control-plane-logs \
  --name RootManageSharedAccessKey \
  --query id -o tsv)

# Create diagnostic setting for control plane logs
az monitor diagnostic-settings create \
  --name aks-control-plane-logs \
  --resource $AKS_CLUSTER_ID \
  --event-hub aks-control-plane-logs \
  --event-hub-rule $EVENT_HUB_ID \
  --logs '[{"category":"kube-apiserver","enabled":true},{"category":"kube-audit","enabled":true},{"category":"kube-controller-manager","enabled":true},{"category":"kube-scheduler","enabled":true},{"category":"cluster-autoscaler","enabled":true}]'

Using Azure Portal

  1. Go to Azure Portal
  2. Navigate to your AKS cluster
  3. Click Diagnostic settings in the left menu
  4. Click + Add diagnostic setting
  5. Configure:
    • Name: aks-control-plane-logs
    • Destination: Select Stream to an event hub or Send to Log Analytics workspace
    • Logs: Enable the following categories:
      • kube-apiserver
      • kube-audit
      • kube-controller-manager
      • kube-scheduler
      • cluster-autoscaler
  6. Click Save

Step 2: Create Event Hub with Kafka Protocol (if using Event Hub)

If using Event Hub as destination, ensure Kafka protocol is enabled:

# Create Event Hub namespace with Kafka enabled
az eventhubs namespace create \
  --resource-group kubesense-rg \
  --name kubesense-logs-namespace \
  --location eastus \
  --sku Standard \
  --enable-kafka true

# Create Event Hub for AKS control plane logs
az eventhubs eventhub create \
  --resource-group kubesense-rg \
  --namespace-name kubesense-logs-namespace \
  --name aks-control-plane-logs \
  --message-retention 7 \
  --partition-count 4

Step 3: Configure KubeSense Aggregator

For Event Hub Destination

aggregator:
  customSources:
    enabled: true
    sources:
      aks_control_plane_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - aks-control-plane-logs
        group_id: aks-cp-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true
          verify_certificate: true
          verify_hostname: true

Note: Ensure the Event Hub namespace has Kafka protocol enabled (--enable-kafka true) when creating it.

For Log Analytics Destination

If using Log Analytics, export to Event Hub or Blob Storage first, then configure as described in Log Analytics or Blob Storage integrations.

Log Types and Use Cases

API Server Logs (kube-apiserver)

Monitor Kubernetes API activities:

  • API requests: Track all API requests to the cluster
  • Authentication: Monitor authentication events
  • Authorization: Track authorization decisions
  • Resource operations: Monitor create, update, delete operations
  • Rate limiting: Identify rate limiting events

Audit Logs (kube-audit)

Comprehensive audit trail:

  • User actions: Track all user-initiated actions
  • Resource changes: Monitor all resource modifications
  • Access patterns: Analyze access patterns and anomalies
  • Compliance: Maintain compliance audit trails
  • Security events: Identify security-related events

Controller Manager Logs (kube-controller-manager)

Controller operations:

  • Replica set management: Monitor replica set scaling
  • Deployment updates: Track deployment rollouts
  • State management: Monitor desired vs actual state reconciliation
  • Node management: Track node lifecycle events

Scheduler Logs (kube-scheduler)

Pod scheduling insights:

  • Scheduling decisions: Understand why pods are scheduled to specific nodes
  • Resource constraints: Identify resource constraints affecting scheduling
  • Performance: Monitor scheduling performance
  • Predicate failures: Track failed scheduling attempts

Cluster Autoscaler Logs (cluster-autoscaler)

Autoscaling operations:

  • Scale-up events: Monitor when nodes are added
  • Scale-down events: Monitor when nodes are removed
  • Scaling decisions: Understand scaling decision logic

Configuration Examples

Multiple AKS Clusters

Configure control plane logs for multiple AKS clusters:

aggregator:
  customSources:
    enabled: true
    sources:
      aks_prod_control_plane:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - aks-prod-control-plane
        group_id: aks-prod-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true
      aks_dev_control_plane:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - aks-dev-control-plane
        group_id: aks-dev-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true

Filtered Log Categories

Enable only specific log categories:

# Enable only API server and audit logs
az monitor diagnostic-settings create \
  --name aks-api-audit-logs \
  --resource $AKS_CLUSTER_ID \
  --event-hub aks-control-plane-logs \
  --event-hub-rule $EVENT_HUB_ID \
  --logs '[{"category":"kube-apiserver","enabled":true},{"category":"kube-audit","enabled":true}]'

Monitoring and Verification

  1. Check diagnostic settings: Verify diagnostic settings are enabled and active
  2. Monitor Event Hub: Verify logs are being published to Event Hub
  3. Verify aggregator connection: Check aggregator logs for Event Hub connection
  4. Check KubeSense dashboard: Verify logs appear with AKS metadata
  5. Review aggregator logs: Check for any ingestion errors

Troubleshooting

Logs Not Appearing

  1. Verify diagnostic settings: Check that diagnostic settings are enabled on the AKS cluster
  2. Check Event Hub: Verify events are being published to Event Hub
  3. Verify Kafka protocol: Ensure Event Hub namespace has Kafka protocol enabled (--enable-kafka true)
  4. Check bootstrap servers: Verify the bootstrap server address is correct (format: <namespace>.servicebus.windows.net:9093)
  5. Verify connection string: Ensure connection string is correct and has Event Hub data receiver permission
  6. Check topic name: Verify the topic name matches the Event Hub name
  7. Check log categories: Verify the log categories you need are enabled
  8. Review network connectivity: Ensure aggregator can reach Event Hub on port 9093
  9. Check aggregator logs: Review aggregator logs for Kafka connection or parsing errors

Missing Log Categories

  1. Verify AKS version: Some log categories may require specific AKS versions
  2. Check diagnostic settings: Ensure all needed categories are enabled
  3. Review AKS configuration: Some log categories may require specific AKS configurations

Performance Issues

  1. Increase partitions: Add more partitions to Event Hub for better throughput
  2. Scale aggregator: Increase aggregator resources if needed
  3. Filter logs: Enable only needed log categories to reduce volume
  4. Monitor consumer lag: Check for consumer lag in Event Hub metrics

Best Practices

  • Enable all categories: Enable all control plane log categories for comprehensive observability
  • Enable Kafka protocol: Ensure Event Hub namespace has Kafka protocol enabled during creation
  • Use Event Hub for real-time: Use Event Hub with Kafka protocol for real-time log streaming
  • Monitor costs: Control plane logs can generate significant costs
  • Set retention: Configure appropriate retention periods
  • Separate by cluster: Create separate Event Hubs (topics) for different clusters
  • Use connection strings: Connection strings are required for SASL PLAIN authentication with Kafka
  • Monitor quotas: Be aware of Event Hub throughput units and quotas
  • Consumer groups: Use different consumer group IDs for different aggregator instances

Cost Considerations

  • Diagnostic settings: No additional charge for diagnostic settings
  • Event Hub: Charged per million events and storage
  • Log Analytics: Charged per GB ingested (if using Log Analytics)
  • Data transfer: Consider data transfer costs between Azure and KubeSense
  • Control plane logs: Can be high volume, especially audit logs

Security Considerations

  • Sensitive data: Control plane logs may contain sensitive information
  • Access control: Ensure proper access control for diagnostic settings and Event Hub
  • Encryption: Use encrypted Event Hub for sensitive logs
  • Audit compliance: Maintain audit logs for compliance requirements
  • Managed Identity: Use Managed Identity for secure authentication

Advanced Configuration

Custom Log Processing

Configure aggregator for custom log processing using transforms (configured separately):

aggregator:
  customSources:
    enabled: true
    sources:
      aks_control_plane_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - aks-control-plane-logs
        group_id: aks-cp-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true

Conclusion

AKS control plane logs integration provides comprehensive observability into cluster management, enabling better troubleshooting, security analysis, and compliance monitoring for your AKS clusters.

back

Blob Storage Logs

next

SSO