Migrating Persistent Storage in Kubernetes from One subscription to another

Jatin Lodhi
3 min readJan 17, 2025

--

How to migrate Prometheus with Persistent Volume and Persistent Volume Claim (PVC and PV) in a Kubernetes cluster, step-by-step.

Here I will use Prometheus as an example, and the most interesting thing is that I migrate the PVC from one subscription to another and cross region (central India to eastus) also.

🤔Why We Migrated Prometheus with Persistent Storage?🤔

Prometheus is a critical monitoring tool that stores metrics and data important for observing system health. By default, Prometheus uses ephemeral storage, meaning all data is lost if the pod restarts or the deployment is upgraded. To ensure **data persistence and reliability**, we migrated Prometheus to a new Kubernetes cluster with the following objectives:

  1. Preserve Historical Data:
    Preserve all metrics from monitoring in case of restart, upgrade, or migration.

2. Cross-Cluster Migration
— Move all Prometheus and associated data without loss across clusters with ease (from one region to another).

3. Utilizing Resilient Storage from Azure
— Leverage persistent storage from Azure Disk for keeping data alive as well as recoverable.

4. Easier Upgrade and Scaling:
- Decouple data from the Prometheus pod lifecycle, making it easier to scale, roll up, and recover from disasters.

This configuration ensures that Prometheus is highly available and reliable while supporting long-term data retention and operational continuity.

Introduction

When working with Prometheus in Kubernetes, using persistent storage is critical to ensure data resilience across deployments and upgrades. This blog demonstrates how to:

  • Create a Persistent Volume (PV) and Persistent Volume Claim (PVC) on Azure.
  • Configure Prometheus to use the PV.
  • Migrate Prometheus data and resolve common issues, such as VolumeBinding filter plugin errors due to CSI migration.

Step 1: Create an Azure Disk for Persistent Storage

  1. Log in to Azure CLI:
az login

2. Create a Resource Group:

az group create --name monitoring-rg --location eastus

3. Create a Managed Disk:

az disk create --resource-group monitoring-rg --name prometheus-pvc-disk --size-gb 20 --sku Standard_LRS

4. Note the Disk URI: After creating the disk, note the diskURI from the output, which you'll use in your PV manifest.

Step 2: Define Persistent Volume (PV) and Persistent Volume Claim (PVC)

  1. Create the PV manifest: Save the following to prometheus-pv.yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
name: prometheus-pv <name of your pv>
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
csi:
driver: disk.csi.azure.com
volumeHandle: prometheus-pvc-disk
volumeAttributes:
storageAccount: <storage-account-name>
containerName: <snapshot-container-name>
diskURI: <YOUR_DISK_URI>
claimRef:
namespace: <namespace>
name: prometheus-pvc <name of your pvc>

2. Create the PVC manifest: Save the following to prometheus-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
namespace: <namespace>
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi

3. Apply the PV and PVC:

kubectl apply -f prometheus-pv.yaml
kubectl apply -f prometheus-pvc.yaml

4. Verify PV and PVC Binding:

kubectl get pv
kubectl get pvc -n <namespace>

Step 3: Deploy Prometheus with Helm

  1. Add the Prometheus Helm Repository:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2. Get values.yaml from helm chart and edit the following to values.yaml:

server:
persistentVolume:
existingClaim: prometheus-pvc <name of your pvc in my case prometheus-pvc>

3. Install or Upgrade Prometheus:

helm upgrade --install prometheus prometheus-community/prometheus -f values.yaml -n <namespace>

4. Check Deployment Status:

kubectl get pods -n <namespace>

Step 4: Troubleshooting Common Issues

NOTE: Issue i faced while setting up this task

  1. Error: VolumeBinding filter plugin:
  • Ensure the PV is using the disk.csi.azure.com driver.
  • Verify that the disk URI in the PV manifest matches the Azure disk created earlier.

2. Debugging Commands:

  • Check PVC events:
kubectl describe pvc <pvc-name> -n <namespace>
  • Check PC events:
kubectl describe pv <pv name> -n <namespace>

By following this step-by-step guide, you can successfully configure Prometheus with persistent storage in Kubernetes.

I tried to explain this process as clearly as possible. I hope you learned something valuable from this guide. Feel free to connect with me on LinkedIn to discuss more or share your thoughts!

I’d love to hear your feedback, so don’t hesitate to leave a comment or share suggestions for improvement. Happy learning and deploying!

THANKS FOR EVERYONE TO READ THIS ARTICLE !!🤗

--

--

Jatin Lodhi
Jatin Lodhi

Written by Jatin Lodhi

I am an IT Enthusiast, who is passionate about exploring/learn all the latest technologies from research perspective.

No responses yet