Azure Blob Storage Integration

Overview

KubeSense supports configuring cold storage integration with Azure Blob Storage for long-term storage of historical observability data. This enables you to move data to Azure after a specified duration while keeping your active storage optimized.

Prerequisites

Before setting up Azure Blob Storage integration, ensure you have:

• KubeSense deployed in an Azure or Kubernetes environment
• An Azure Storage Account created
• Access to Azure storage account name and keys
• Access to modify the KubeSense Helm values

Step 1: Create Azure Storage Account

1. Go to Azure Portal → Storage accounts → Create
2. Enter a storage account name (e.g., kubesensestorage)
3. Select your subscription and resource group
4. Choose performance tier (Standard recommended for cold storage)
5. Select replication type
6. Create the storage account

Step 2: Create Blob Container

1. Navigate to your storage account in Azure Portal
2. Go to Containers under Data storage
3. Click + Container
4. Enter container name (e.g., kubesense)
5. Set public access level to Private
6. Click Create

Step 3: Get Storage Account Keys

1. In your storage account, go to Access keys under Security + networking
2. Click Show next to key1 or key2 to reveal the key
3. Copy the storage account name and one of the access keys

Step 4: Update KubeSense Configuration

Add the cold storage configuration to your KubeSense Helm values:

coldStorageConfig:
  enabled: true
  endpoint: "https://kubesensestorage.blob.core.windows.net"
  # Required for Azure
  account_name: "kubesensestorage"
  account_key: "YOUR_STORAGE_ACCOUNT_KEY"
  container_name: "kubesense"
  cloudProvider: "azure"

Replace:

• kubesensestorage with your actual storage account name
• YOUR_STORAGE_ACCOUNT_KEY with your actual storage account key
• kubesense with your container name
• Update the endpoint URL with your storage account name and Azure region if needed

Step 5: Upgrade KubeSense

Apply the configuration by upgrading your KubeSense Helm deployment:

helm upgrade kubesense ./kubesense-chart \
  -f values.yaml \
  --namespace kubesense

Configuration Parameters

Here's a detailed breakdown of the Azure cold storage configuration:

Verifying Integration

After configuring Azure Blob Storage integration:

1. Check KubeSense logs to ensure Azure connection is successful:

   kubectl logs -n kubesense sts/kubesense-datastore-shard-0 -f

2. Verify data in Azure Portal:

   • Navigate to your storage account in Azure Portal
   • Check the container for backup files
   • Monitor data being written over time

3. Test cold storage functionality:

   • Query historical data that should be in cold storage
   • Verify data is accessible through KubeSense UI
   • Check data retrieval performance

Best Practices

Security

• Use Azure Key Vault for managing storage account keys in production (optional)
• Enable Azure Storage encryption at rest
• Use Managed Identity when running on AKS
• Set up network rules to restrict access to storage account
• Consider geo-redundancy: For disaster recovery requirements

Performance

• Choose appropriate tier: Use Hot if frequently queried. Use Cool or Archive tier for long term storage
• Consider same region buckets: For lower latency

Troubleshooting

Issue: Authentication failures

Symptoms: 403 Forbidden or 401 Unauthorized errors

Solution:

• Verify the storage account name is correct
• Check that the account key matches what's configured
• Ensure the storage account exists and is accessible

Issue: Container not found errors

Symptoms: Container does not exist errors

Solution:

• Verify the container name matches the configuration
• Ensure the container was created successfully in Azure Portal
• Check that the storage account name is correct

Issue: Network connectivity issues

Symptoms: Cannot connect to Azure Blob Storage endpoint

Solution:

• Verify network rules allow access from your cluster
• Check firewall rules if using Azure Private Endpoints
• Ensure the endpoint URL is correctly formatted

Issue: Cold storage not working

Symptoms: Data not being written to Azure

Solution:

• Check Datastore logs for errors
• Verify the configuration parameters are correct
• Ensure sufficient permissions on the storage account
• Check if the container has proper access settings

Advanced Configuration

Using Private Endpoints

For enhanced security, you can use Azure Private Endpoints:

1. Create a Private Endpoint for your storage account
2. Configure DNS resolution for the private endpoint
3. Update the endpoint URL in your configuration if needed
4. Ensure network policies allow traffic through the private endpoint

Configuring Lifecycle Management

Set up Azure Blob lifecycle policies to automatically move data to Cool or Archive tiers:

1. Go to your storage account → Lifecycle management
2. Add a policy rule
3. Configure tier transitions based on access patterns
4. Apply the policy to optimize costs

Conclusion

Azure Blob Storage integration provides a reliable, cost-effective solution for storing historical observability data in Azure environments. By properly configuring authentication, monitoring, and lifecycle policies, you can optimize both performance and costs for long-term data storage.