Backup & Restore Metrics

KubeSense leverages VictoriaMetrics as its core metrics storage engine. This integration provides seamless compatibility with VictoriaMetrics vmbackup and vmrestore utilities for comprehensive data protection.

Creating Incremental Backups

Prerequisites

Download and install vmutils from the official VictoriaMetrics releases
Ensure you have appropriate permissions to access the kubesense namespace

Step 1: Access Metrics Store Service

Establish a port-forward connection to kubesense's metrics-store service:

# List metrics-store services in the kubesense namespace
kubectl get svc -n kubesense | grep "metrics-store"

# Create port-forward tunnel (replace with actual service name)
kubectl port-forward svc/YOUR_METRICS_STORE_SERVICE_NAME \
  -n kubesense 8428:8428

Step 2: Execute Backup

Run the vmbackup utility to create an incremental backup. This example uses AWS S3, but multiple storage providers are supported:

./vmbackup \
  -credsFilePath=YOUR_AWS_CREDENTIALS_PATH \
  -storageDataPath=YOUR_METRICS_STORE_DATA_PATH \
  -snapshot.createURL=http://localhost:8428/snapshot/create \
  -dst=s3://YOUR_BUCKET_NAME/YOUR_BACKUP_PATH

Info: vmbackup automatically detects existing backups and performs incremental updates, ensuring efficient storage utilization

Restoring from Backup

Step 1: Prepare for Restoration

Scale down the Metrics store StatefulSet to ensure data integrity during restoration:

kubectl scale sts YOUR_RELEASE_NAME-metrics-store --replicas=0

Step 2: Identify Storage Resources

Locate the Metrics store PersistentVolumeClaim:

kubectl get pvc -n kubesense | grep metrics-store

Step 3: Create Restoration Job

Create a Kubernetes Job manifest named vm-restore.yaml:

Info: Replace YOUR_METRICS_STORE_PVC_NAME with the actual PVC name from the previous step

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vm-restore
  annotations:
    eks.amazonaws.com/role-arn: YOUR_IAM_ROLE_ARN # Role with S3 bucket access permissions
---
apiVersion: batch/v1
kind: Job
metadata:
  name: vm-restore
spec:
  ttlSecondsAfterFinished: 600
  template:
    spec:
      serviceAccountName: vm-restore
      restartPolicy: OnFailure
      volumes:
      - name: vmstorage-volume
        persistentVolumeClaim:
          claimName: "YOUR_METRICS_STORE_PVC_NAME"
      containers:
      - name: vm-restore
        image: victoriametrics/vmrestore
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /storage
          name: vmstorage-volume
        command:
        - /bin/sh
        - -c
        - /vmrestore-prod -src=s3://YOUR_BUCKET_NAME/YOUR_BACKUP_PATH -storageDataPath=/storage

Step 4: Execute Restoration

Deploy the restoration job and monitor its progress:

kubectl apply -f vm-restore.yaml -n kubesense

# Monitor job status
kubectl get jobs -n kubesense
kubectl logs job/vm-restore -n kubesense

Step 5: Restart Metrics store

Once restoration is complete, scale up the Metrics store instance:

kubectl scale sts YOUR_RELEASE_NAME-metrics-store --replicas=1

Best Practices

Regular Backups: Schedule automated backups during low-activity periods
Test Restorations: Periodically verify backup integrity by performing test restorations
Storage Monitoring: Monitor backup storage usage and implement retention policies
Security: Ensure proper IAM roles and access controls for backup storage

Backup & Restore Metrics

ON THIS PAGE