Event Hub Logs

Ingesting Azure Logs via Event Hub with KubeSense

KubeSense supports ingesting logs from Azure services by configuring Azure Diagnostic settings to send logs to Event Hub. Since Azure Event Hub supports the Kafka protocol, the KubeSense aggregator uses Vector's Kafka source to consume logs from Event Hub. This integration enables real-time log streaming from various Azure resources.

Note: Azure Event Hub must have Kafka protocol enabled. The KubeSense aggregator uses Vector's Kafka source to connect to Event Hub.

Prerequisites

Before you begin, ensure you have:

Azure subscription with appropriate permissions
Event Hub namespace with Kafka protocol enabled
Event Hub created within the namespace
KubeSense aggregator deployed and accessible
Event Hub connection string with read permissions
Azure CLI or Azure Portal access

Architecture

Azure logs flow through the following path:

Azure Resources → Diagnostic Settings → Event Hub (Kafka Protocol) → KubeSense Aggregator (Kafka Source)

Step 1: Create Event Hub Namespace and Event Hub

Using Azure CLI

# Create resource group
az group create --name kubesense-rg --location eastus

# Create Event Hub namespace with Kafka enabled
az eventhubs namespace create \
  --resource-group kubesense-rg \
  --name kubesense-logs-namespace \
  --location eastus \
  --sku Standard \
  --enable-kafka true

# Create Event Hub
az eventhubs eventhub create \
  --resource-group kubesense-rg \
  --namespace-name kubesense-logs-namespace \
  --name kubesense-logs \
  --message-retention 7 \
  --partition-count 4

Using Azure Portal

Go to Azure Portal
Navigate to Event Hubs
Click + Create
Configure:
- Namespace name: kubesense-logs-namespace
- Pricing tier: Standard
- Location: Select your region
Click Review + Create, then Create
Once created, navigate to the namespace
Click + Event Hub to create an event hub named kubesense-logs

Step 2: Configure Diagnostic Settings

Configure diagnostic settings on Azure resources to send logs to Event Hub.

For AKS Cluster

# Get AKS cluster resource ID
AKS_CLUSTER_ID=$(az aks show \
  --resource-group kubesense-rg \
  --name my-aks-cluster \
  --query id -o tsv)

# Get Event Hub authorization rule
EVENT_HUB_ID=$(az eventhubs eventhub authorization-rule show \
  --resource-group kubesense-rg \
  --namespace-name kubesense-logs-namespace \
  --eventhub-name kubesense-logs \
  --name RootManageSharedAccessKey \
  --query id -o tsv)

# Create diagnostic setting
az monitor diagnostic-settings create \
  --name kubesense-aks-logs \
  --resource $AKS_CLUSTER_ID \
  --event-hub kubesense-logs \
  --event-hub-rule $EVENT_HUB_ID \
  --logs '[{"category":"kube-apiserver","enabled":true},{"category":"kube-audit","enabled":true},{"category":"kube-controller-manager","enabled":true},{"category":"kube-scheduler","enabled":true}]'

For Virtual Machines

# Get VM resource ID
VM_ID=$(az vm show \
  --resource-group kubesense-rg \
  --name my-vm \
  --query id -o tsv)

# Create diagnostic setting
az monitor diagnostic-settings create \
  --name kubesense-vm-logs \
  --resource $VM_ID \
  --event-hub kubesense-logs \
  --event-hub-rule $EVENT_HUB_ID \
  --logs '[{"category":"WindowsEventLog","enabled":true},{"category":"Application","enabled":true}]'

For App Services

# Get App Service resource ID
APP_SERVICE_ID=$(az webapp show \
  --resource-group kubesense-rg \
  --name my-app-service \
  --query id -o tsv)

# Create diagnostic setting
az monitor diagnostic-settings create \
  --name kubesense-app-logs \
  --resource $APP_SERVICE_ID \
  --event-hub kubesense-logs \
  --event-hub-rule $EVENT_HUB_ID \
  --logs '[{"category":"AppServiceHTTPLogs","enabled":true},{"category":"AppServiceConsoleLogs","enabled":true}]'

Step 3: Get Event Hub Connection Details

Get the Event Hub connection string and Kafka endpoint:

# Get Event Hub connection string
az eventhubs eventhub authorization-rule keys list \
  --resource-group kubesense-rg \
  --namespace-name kubesense-logs-namespace \
  --eventhub-name kubesense-logs \
  --name RootManageSharedAccessKey \
  --query primaryConnectionString -o tsv

# Get Kafka bootstrap servers (format: <namespace>.servicebus.windows.net:9093)
# Example: kubesense-logs-namespace.servicebus.windows.net:9093

Step 4: Configure KubeSense Aggregator

The KubeSense aggregator uses Vector's Kafka source to consume from Event Hub. Configure the aggregator:

aggregator:
  customSources:
    enabled: true
    sources:
      eventhub_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - kubesense-logs
        group_id: kubesense-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true
          verify_certificate: true
          verify_hostname: true

Using Helm Values

global:
  cluster_name: "azure-cluster"

aggregator:
  customSources:
    enabled: true
    sources:
      eventhub_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - kubesense-logs
        group_id: kubesense-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true
          verify_certificate: true
          verify_hostname: true

Using Managed Identity (Alternative)

For Managed Identity, you'll need to use connection strings with managed identity authentication. The Kafka source still requires SASL PLAIN with connection string format:

aggregator:
  customSources:
    enabled: true
    sources:
      eventhub_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - kubesense-logs
        group_id: kubesense-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING_WITH_MANAGED_IDENTITY>"
        tls:
          enabled: true

Log Enrichment

KubeSense aggregator automatically enriches Azure logs with:

Resource metadata: Resource type, resource group, subscription
Azure context: Subscription ID, tenant ID, location
Log metadata: Category, level, timestamp
Source information: Resource name, resource ID

Monitoring and Verification

Check Event Hub metrics: Verify events are being sent to Event Hub
Monitor diagnostic settings: Ensure diagnostic settings are active
Verify aggregator connection: Check aggregator logs for Event Hub connection
Check KubeSense dashboard: Verify logs appear with Azure metadata
Review aggregator logs: Check for any ingestion errors

Troubleshooting

Logs Not Appearing

Verify diagnostic settings: Check that diagnostic settings are enabled on the Azure resources
Check Event Hub: Verify events are being published to Event Hub
Verify Kafka protocol: Ensure Event Hub namespace has Kafka protocol enabled (--enable-kafka true)
Check bootstrap servers: Verify the bootstrap server address is correct (format: <namespace>.servicebus.windows.net:9093)
Verify connection string: Ensure connection string is correct and has Event Hub data receiver permission
Check topic name: Verify the topic name matches the Event Hub name
Review network connectivity: Ensure aggregator can reach Event Hub on port 9093
Check aggregator logs: Review aggregator logs for Kafka connection or parsing errors

Authentication Issues

Verify connection string: Ensure connection string is valid and not expired
Check SASL configuration: Verify username is $ConnectionString and password is the full connection string
Check permissions: Verify the connection string has Event Hub data receiver permission
Verify TLS: Ensure TLS is enabled (required for Event Hub Kafka)
Review aggregator auth config: Ensure SASL PLAIN authentication is properly configured

Performance Issues

Increase partitions: Add more partitions to Event Hub for better throughput
Scale aggregator: Increase aggregator resources if needed
Adjust batch size: Configure Event Hub batch settings
Monitor consumer lag: Check for consumer lag in Event Hub metrics

Best Practices

Enable Kafka protocol: Ensure Event Hub namespace has Kafka protocol enabled during creation
Use connection strings: Connection strings are required for SASL PLAIN authentication with Kafka
Organize by topic: Create separate Event Hubs (topics) for different log types or environments
Monitor Event Hub quotas: Be aware of Event Hub throughput units and quotas
Set up alerts: Configure alerts for Event Hub consumer lag and Kafka consumer group lag
Use structured logging: Ensure applications use structured logging for better parsing
Tag resources: Use Azure tags for better log organization
Monitor costs: Track Event Hub ingestion and storage costs
Consumer groups: Use different consumer group IDs for different aggregator instances

Cost Considerations

Event Hub: Charged per million events and storage
Throughput units: Consider throughput unit costs for high-volume scenarios
Kafka protocol: No additional charge for Kafka protocol support
Data transfer: Consider data transfer costs between Azure and KubeSense
Diagnostic settings: No additional charge for diagnostic settings

Advanced Configuration

Multiple Event Hubs

Configure multiple event hubs for different log types:

aggregator:
  customSources:
    enabled: true
    sources:
      aks_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - aks-logs
        group_id: aks-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true
      vm_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - vm-logs
        group_id: vm-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true

Custom Log Processing

Configure aggregator for custom log processing using transforms (configured separately):

aggregator:
  customSources:
    enabled: true
    sources:
      eventhub_logs:
        type: kafka
        bootstrap_servers: "<NAMESPACE>.servicebus.windows.net:9093"
        topics:
          - kubesense-logs
        group_id: kubesense-consumer
        auth:
          sasl:
            mechanism: PLAIN
            username: "$ConnectionString"
            password: "<EVENT_HUB_CONNECTION_STRING>"
        tls:
          enabled: true

Conclusion

Event Hub logs integration provides real-time log streaming from Azure services, enabling comprehensive observability across your entire Azure infrastructure alongside your Kubernetes and application data.

Event Hub Logs

ON THIS PAGE