Migrating Persistent Storage in Kubernetes from One subscription to another
How to migrate Prometheus with Persistent Volume and Persistent Volume Claim (PVC and PV) in a Kubernetes cluster, step-by-step.
Here I will use Prometheus as an example, and the most interesting thing is that I migrate the PVC from one subscription to another and cross region (central India to eastus) also.
🤔Why We Migrated Prometheus with Persistent Storage?🤔
Prometheus is a critical monitoring tool that stores metrics and data important for observing system health. By default, Prometheus uses ephemeral storage, meaning all data is lost if the pod restarts or the deployment is upgraded. To ensure **data persistence and reliability**, we migrated Prometheus to a new Kubernetes cluster with the following objectives:
- Preserve Historical Data:
Preserve all metrics from monitoring in case of restart, upgrade, or migration.
2. Cross-Cluster Migration
— Move all Prometheus and associated data without loss across clusters with ease (from one region to another).
3. Utilizing Resilient Storage from Azure
— Leverage persistent storage from Azure Disk for keeping data alive as well as recoverable.
4. Easier Upgrade and Scaling:
- Decouple data from the Prometheus pod lifecycle, making it easier to scale, roll up, and recover from disasters.
This configuration ensures that Prometheus is highly available and reliable while supporting long-term data retention and operational continuity.
Introduction
When working with Prometheus in Kubernetes, using persistent storage is critical to ensure data resilience across deployments and upgrades. This blog demonstrates how to:
- Create a Persistent Volume (PV) and Persistent Volume Claim (PVC) on Azure.
- Configure Prometheus to use the PV.
- Migrate Prometheus data and resolve common issues, such as
VolumeBinding filter plugin
errors due to CSI migration.
Step 1: Create an Azure Disk for Persistent Storage
- Log in to Azure CLI:
az login
2. Create a Resource Group:
az group create --name monitoring-rg --location eastus
3. Create a Managed Disk:
az disk create --resource-group monitoring-rg --name prometheus-pvc-disk --size-gb 20 --sku Standard_LRS
4. Note the Disk URI: After creating the disk, note the diskURI
from the output, which you'll use in your PV manifest.
Step 2: Define Persistent Volume (PV) and Persistent Volume Claim (PVC)
- Create the PV manifest: Save the following to
prometheus-pv.yaml
:
apiVersion: v1
kind: PersistentVolume
metadata:
name: prometheus-pv <name of your pv>
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
csi:
driver: disk.csi.azure.com
volumeHandle: prometheus-pvc-disk
volumeAttributes:
storageAccount: <storage-account-name>
containerName: <snapshot-container-name>
diskURI: <YOUR_DISK_URI>
claimRef:
namespace: <namespace>
name: prometheus-pvc <name of your pvc>
2. Create the PVC manifest: Save the following to prometheus-pvc.yaml
:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
namespace: <namespace>
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
3. Apply the PV and PVC:
kubectl apply -f prometheus-pv.yaml
kubectl apply -f prometheus-pvc.yaml
4. Verify PV and PVC Binding:
kubectl get pv
kubectl get pvc -n <namespace>
Step 3: Deploy Prometheus with Helm
- Add the Prometheus Helm Repository:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
2. Get values.yaml
from helm chart and edit the following to values.yaml
:
server:
persistentVolume:
existingClaim: prometheus-pvc <name of your pvc in my case prometheus-pvc>
3. Install or Upgrade Prometheus:
helm upgrade --install prometheus prometheus-community/prometheus -f values.yaml -n <namespace>
4. Check Deployment Status:
kubectl get pods -n <namespace>
Step 4: Troubleshooting Common Issues
NOTE: Issue i faced while setting up this task
- Error:
VolumeBinding filter plugin
:
- Ensure the PV is using the
disk.csi.azure.com
driver. - Verify that the disk URI in the PV manifest matches the Azure disk created earlier.
2. Debugging Commands:
- Check PVC events:
kubectl describe pvc <pvc-name> -n <namespace>
- Check PC events:
kubectl describe pv <pv name> -n <namespace>
By following this step-by-step guide, you can successfully configure Prometheus with persistent storage in Kubernetes.
I tried to explain this process as clearly as possible. I hope you learned something valuable from this guide. Feel free to connect with me on LinkedIn to discuss more or share your thoughts!
I’d love to hear your feedback, so don’t hesitate to leave a comment or share suggestions for improvement. Happy learning and deploying!