Kubernetes PV and PVC

Kubernetes PV and PVC: Real-World DevOps Guide Let's approach this from the perspective of an experienced DevOps engineer managing Kubernetes in production. We'll focus on what actually happens when pods restart, worker nodes fail, persistent volumes get deleted, storage fills up, and applications lose access to their data.

The most important principle to remember is:

A PVC requests storage, a PV represents storage, and the underlying storage system holds the actual data. These are related, but they are not the same thing.

For example, in an AWS EKS cluster, your application might use a PVC that binds to a PV backed by an Amazon EBS volume. If a pod crashes, Kubernetes can recreate the pod and reconnect it to the existing storage. If the worker node disappears, Kubernetes may be able to attach the same EBS volume to another suitable node.

But if somebody deletes the underlying EBS volume, the data may be lost. Persistent storage is not automatically a backup.

Let's explore the complete lifecycle, architecture, failure scenarios, commands, troubleshooting procedures, and production safeguards.

1. Understand the architecture first Imagine you deploy a payment application on EKS. The application stores payment-related data on persistent storage.

The architecture looks like this: EKS CLUSTER | +---------+---------+ | | v v Worker Node 1 Worker Node 2 | | v | Payment Pod | | | v | PVC: payment-data | | | v | PV: payment-pv | | | v | Amazon EBS Volume | | | v | Persistent Data |

The PVC and PV are Kubernetes resources. The EBS volume is the actual storage device that holds the data.

In a typical dynamically provisioned EKS environment, you don't manually create the PV or EBS volume. You create a PVC, and the Kubernetes storage components provision the necessary resources. What happens when something fails? Event Expected result Application process crashes The container may restart; data on persistent storage remains. Pod gets deleted A replacement pod can mount the same storage, subject to access mode and scheduling constraints. Worker node fails The pod may move to another suitable node, and storage may be reattached. Worker node is deleted Data on independent EBS storage normally survives, provided the volume itself is retained. PVC is deleted The PV and underlying storage may be retained or deleted, depending on the reclaim policy and storage configuration. PV is deleted The Kubernetes storage object disappears; the underlying data may or may not survive. EBS volume is deleted The data on that volume is lost unless a usable backup or snapshot exists. Entire EKS cluster is deleted Independently retained EBS volumes can survive, but their Kubernetes references and configuration may need to be recreated.

These are expected behaviors, not unconditional guarantees. Storage drivers, reclaim policies, volume attachment state, and the exact deletion operation matter. 2. What exactly are PV, PVC, and StorageClass? Before troubleshooting failures, you need to understand who is responsible for each part. 2.1 PersistentVolume (PV) A PersistentVolume is a Kubernetes resource representing a piece of storage available to a cluster.

It can represent storage backed by:

• Amazon EBS.
• Amazon EFS.
• Azure Managed Disks.
• Google Persistent Disk.
• NFS.
• Other storage systems supported by a CSI driver. For example, a PV might represent a 20 GiB EBS volume. apiVersion: v1 kind: PersistentVolume metadata: name: payment-pv spec: capacity: storage: 20Gi accessModes:
  • ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: manual hostPath: path: /data/payment

This is an illustrative, manually created PV using hostPath. It is useful for understanding the resource structure, but it is not an appropriate production EBS configuration.

Important: A PV does not necessarily contain the data itself. It represents storage provided by an underlying storage system. 2.2 PersistentVolumeClaim (PVC) A PVC is a request for storage made by an application or its administrator.

For example: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: payment-data spec: accessModes: - ReadWriteOnce storageClassName: gp3 resources: requests: storage: 20Gi

This tells Kubernetes:

"I need 20 GiB of storage with the requested access mode, provisioned using the gp3 StorageClass."

Notice that the PVC does not directly specify an EBS volume ID.

Kubernetes determines which suitable existing PV can satisfy the claim or provisions new storage if dynamic provisioning is configured. 2.3 StorageClass A StorageClass tells Kubernetes how to provision storage.

In EKS, a StorageClass might specify the Amazon EBS CSI driver and an EBS volume type.

For example: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 provisioner: ebs.csi.aws.com parameters: type: gp3 reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

Here is what each important field means: Field Purpose provisioner Identifies the storage driver responsible for provisioning volumes. type: gp3 Requests the EBS gp3 volume type. reclaimPolicy: Retain Requests that the PV and underlying volume be retained after the PVC is released, subject to the driver's behavior. volumeBindingMode Controls when the volume is provisioned or bound relative to pod scheduling. allowVolumeExpansion Allows supported volume expansion through the PVC.

The EBS CSI driver must be installed and configured correctly for this example to work.

Also, Retain does not protect data against every possible deletion. Someone can still delete the EBS volume directly through AWS, and a retained volume is not a backup. 2.4 How the three components work together Developer creates PVC | v Kubernetes finds StorageClass | v Storage provisioner creates volume | v PV is created | v PVC binds to PV | v Pod references PVC | v Storage driver mounts volume | v Application reads and writes data

The storage lifecycle involves several different components:

• Kubernetes API server: Stores and serves the PVC, PV, pod, and StorageClass objects.
• PersistentVolume controller: Handles PV/PVC binding and related lifecycle operations.
• External provisioner: Coordinates dynamic volume provisioning through the storage driver.
• CSI controller components: Handle storage operations such as provisioning, attachment, and expansion.
• CSI node plugin: Performs node-side storage operations, including mounting supported volumes.
• Kubelet: Coordinates the pod's volume setup on the worker node.
• Storage backend: Actually stores the data. The exact division of responsibilities depends on the storage driver.

3. What happens when a pod is deleted? This is one of the most important production scenarios.

Suppose you have: Pod: payment-7f8c9 PVC: payment-data PV: payment-pv EBS Volume: vol-0123456789

Your application writes customer transaction data to the mounted volume.

Now someone executes: kubectl delete pod payment-7f8c9 -n production

What happens internally? Step 1: Kubernetes deletes the pod object.

If the pod belongs to a Deployment or StatefulSet, its controller normally creates a replacement pod to maintain the desired replica count.

Step 2: The replacement pod is scheduled onto a suitable worker node.

The scheduler considers resource availability, volume topology, access constraints, affinity rules, and other scheduling requirements.

Step 3: Kubernetes processes the PVC reference.

The new pod references the same PVC, so Kubernetes uses the existing claim rather than automatically creating a new claim.

Step 4: The storage driver prepares the volume.

If the volume needs to be detached from the previous node and attached to the new node, the storage components coordinate those operations.

Step 5: The volume is mounted and the application starts.

The application can access the existing data. Original Pod | v Writes data to EBS | v Pod gets deleted | v Deployment creates replacement | v Replacement mounts same PVC | v Existing data is available

Production reality Deleting a pod does not normally delete its PVC or EBS volume.

However, there are important exceptions:

• A controller might also manage or delete associated claims.
• A StatefulSet PVC can be subject to its configured retention policy.
• An operator or cleanup process might delete the PVC.
• The storage backend might have been deleted separately. Interview answer: "When a pod is deleted, its controller can recreate it. If the replacement uses the same PVC and the underlying storage is healthy, the application can access its existing data. I would check volume attachment, mount status, and application consistency if recovery fails."

4. What happens when a worker node is deleted? Suppose your EKS cluster has two worker nodes: Worker Node A Worker Node B | | v | Payment Pod | | | v | PVC: payment-data | | | v | EBS Volume |

Now Worker Node A fails or is terminated. Step-by-step recovery Step 1: The node becomes unavailable.

The Kubernetes control plane detects the node's unhealthy state through its node monitoring mechanisms.

Detection is not instantaneous. Kubernetes waits for relevant health checks and failure-detection thresholds.

Step 2: The pod becomes unavailable.

The original pod might remain visible in the API for a while. Kubernetes does not necessarily delete and replace it immediately.

Step 3: Kubernetes attempts recovery.

If the workload controller determines that a replacement pod is needed, the replacement must be scheduled onto an eligible node.

Step 4: The storage system must release and reattach the volume.

If the EBS volume is still attached to the failed node, Kubernetes and the storage infrastructure must resolve the old attachment before a safe new attachment can occur.

A failed node can delay this process.

Step 5: The replacement pod mounts the existing volume.

Once the volume is safely available and all scheduling constraints are satisfied, the application can start using its existing data. Critical EBS limitation: Availability Zones An EBS volume is tied to a specific Availability Zone.

For example: ap-south-1a | +-- Worker Node A | +-- EBS Volume A

ap-south-1b | +-- Worker Node B

If the EBS volume belongs to ap-south-1a, it cannot simply be attached to a worker node in ap-south-1b.

Kubernetes must find a suitable node in the volume's Availability Zone.

If no eligible node is available there, the pod might remain in Pending. What if the node is permanently lost? The data on an independently stored EBS volume normally survives the loss of its worker node.

However, recovery can be delayed by:

• A stale volume attachment.
• An unavailable Availability Zone.
• Missing permissions for the CSI driver.
• An unhealthy EBS CSI controller or node plugin.
• A volume attachment limit.
• Filesystem or application recovery requirements. For important applications, design the node groups and storage placement so that a replacement node can be launched in the volume's Availability Zone.

Interview answer: "If a worker node fails, EBS-backed data normally survives because it is stored independently of the node. Kubernetes can recover the workload on another eligible node after the volume is safely detached and reattached. I would verify the volume's Availability Zone, attachment state, CSI driver health, and pod events." 5. What happens when a PVC is deleted? This scenario is more dangerous because the outcome depends on the PV's reclaim policy.

Imagine: PVC: payment-data | v PV: payment-pv | v EBS Volume | v Customer transaction data

Now execute: kubectl delete pvc payment-data -n production

The PVC object is deleted, but what happens to the PV and the EBS volume?

There are two main reclaim policies to understand. 5.1 Reclaim policy: Delete With Delete, when the claim is released, the storage provisioner normally deletes the PV and its associated storage asset, if supported and successfully executed.

For EBS, this can result in deletion of the underlying volume. PVC deleted | v PV released | v Reclaim policy: Delete | v Storage cleanup requested | v EBS volume may be deleted | v Data may be lost

This is a common default for dynamically provisioned storage, although the actual policy depends on the StorageClass and PV configuration.

Production risk: Deleting a PVC can ultimately delete valuable data. 5.2 Reclaim policy: Retain With Retain, Kubernetes retains the PV and underlying storage after the claim is released. PVC deleted | v PV becomes Released | v EBS volume remains | v Data remains on the volume

The PV generally becomes Released, not immediately Available.

That distinction matters.

A Released PV may still contain the old claim's metadata and is not automatically ready for a new PVC to bind to it.

An administrator may need to recover the existing volume and configure an appropriate PV or claim before reusing the data. 5.3 How do you check the reclaim policy? Run: kubectl get pv

Example: NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS payment-pv 20Gi RWO Retain Bound

For detailed information: kubectl describe pv payment-pv

Look for: PersistentVolumeReclaimPolicy: Retain Status: Bound Claim: production/payment-data

To check the StorageClass: kubectl get storageclass

Then: kubectl describe storageclass gp3

Remember that an existing PV has its own reclaim policy. Changing a StorageClass does not automatically change the policy on every previously created PV. 5.4 Can you change the reclaim policy? Yes, where permitted, you can update the existing PV's policy.

For example: kubectl patch pv payment-pv
-p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

This changes the PV's reclaim policy. It does not create a backup, and it does not restore data that has already been deleted.

For a new dynamically provisioned PV, you can also configure the desired policy in the StorageClass.

Production recommendation: For important databases and other irreplaceable data, consider Retain, appropriate backup policies, and strict controls on who can delete PVCs and volumes. 6. What happens when a PV is deleted? A PV is a Kubernetes resource, not the physical storage itself.

Suppose an administrator runs: kubectl delete pv payment-pv

The consequences depend on the volume's reclaim policy, its current state, the storage driver's behavior, and whether deletion protection or finalizers are involved. If the reclaim policy is Delete The storage controller may delete the underlying volume as part of cleanup. If the reclaim policy is Retain The intended behavior is to retain the underlying storage. However, deleting a PV object is not the same as safely preserving all the information needed to recover it.

You may need to reconstruct the PV configuration and reference the existing storage asset to recover the data. If the PV is Bound to a PVC Deleting a bound PV is not a normal application cleanup procedure.

Kubernetes protection mechanisms may delay deletion while the volume is in use. Removing the PV or PVC can also disrupt the application.

Never treat deletion of a PV object as a safe way to clean up storage. Check the PVC, PV reclaim policy, underlying volume ID, attachment state, and backup status first. 7. What happens if the underlying EBS volume is deleted? This is the point where Kubernetes cannot magically save your data.

Suppose: PVC: payment-data | v PV: payment-pv | v EBS Volume: vol-0123456789

An administrator goes to the AWS console and deletes vol-0123456789.

The Kubernetes PVC might still exist. The PV might still exist. But the storage asset they reference is gone.

Consequently:

• The application may fail to mount its volume.
• Pods may remain in ContainerCreating or fail with volume-related errors.
• The PVC might still appear Bound temporarily, even though the underlying storage is unavailable.
• Kubernetes cannot recreate the deleted data merely by recreating a pod. Recovery depends on an available snapshot, backup, replica, or other copy of the data.

Remember: Retain protects against certain Kubernetes storage cleanup operations. It does not prevent an administrator or automation from deleting the underlying EBS volume directly. 8. What if the entire EKS cluster is deleted? Suppose Argo CD or Terraform deletes the EKS cluster.

Whether your data survives depends on how the cluster and storage were managed.

For example: EKS Cluster | +-- Pods | +-- PVC objects | +-- PV objects | +-- EBS volumes

The Kubernetes objects belong to the cluster's API state, whereas EBS volumes are AWS resources.

If the cluster is deleted, the Kubernetes objects disappear with its control plane. Independently retained EBS volumes can survive, but their survival depends on the cleanup configuration and deletion behavior.

For recovery, you may need to:

1. Recreate the EKS cluster.
2. Install and configure the EBS CSI driver.
3. Identify the retained EBS volumes.
4. Recreate PVs referencing those volumes.
5. Recreate the necessary PVCs.
6. Recreate the application workloads.
7. Verify filesystem and application consistency. This is why infrastructure deletion procedures must account for storage resources separately from compute resources.
8. How to protect PV and PVC data in production Now let's move from understanding failures to preventing them. 9.1 Use an appropriate reclaim policy For critical storage, evaluate whether the PV should use: persistentVolumeReclaimPolicy: Retain

This reduces the risk of the provisioner deleting the storage as part of PVC cleanup.

However, retaining the volume means you must also manage orphaned volumes and their costs. 9.2 Use snapshots and backups For EBS-backed storage, you can use Amazon EBS snapshots or a suitable backup solution.

A snapshot can help recover data after accidental deletion or corruption, depending on when the snapshot was taken and what it captured.

For a database, storage snapshots alone might not always provide application-consistent recovery. Follow the database's backup and recovery requirements.

A production strategy should specify:

• Backup frequency.
• Retention period.
• Encryption.
• Cross-account or cross-region protection where required.
• Recovery-point objective (RPO).
• Recovery-time objective (RTO).
• Periodic restore testing. A backup that has never been restored successfully is an untested assumption, not a proven recovery plan. 9.3 Restrict destructive permissions Use Kubernetes RBAC to restrict who can delete PVCs and PVs.

Use AWS IAM policies to restrict who can delete EBS volumes and snapshots.

Also review:

• Terraform destroy permissions.
• GitOps pruning and deletion settings.
• Namespace deletion permissions.
• Automated cleanup jobs.
• StatefulSet PVC retention policies. 9.4 Use appropriate storage for the workload Storage Typical use Important consideration EBS Databases and block-storage workloads Usually tied to one Availability Zone; access mode and attachment constraints matter. EFS Shared files across multiple nodes Network filesystem; performance and throughput must be designed appropriately. S3 Objects, backups, and application files Object storage, not a conventional filesystem volume. hostPath Specialized node-local workloads and limited development scenarios Data is tied to the node's filesystem.

For a typical EKS database using a single EBS-backed PVC, you should not assume that two pods on different nodes can mount the same volume for simultaneous read-write access. 10. Real-world troubleshooting commands When a PVC-backed application fails, investigate in a fixed order rather than guessing. Step 1: Check the pod kubectl get pods -n production -o wide

Look for:

• Pending
• ContainerCreating
• CrashLoopBackOff
• The node on which the pod is scheduled. Step 2: Inspect pod events kubectl describe pod -n production

Look for events such as:

• FailedMount.
• FailedAttachVolume.
• FailedScheduling.
• Multi-Attach errors.
• Missing PVC references. These events help identify whether the failure is related to scheduling, attachment, mounting, or something else. Step 3: Check the PVC kubectl get pvc -n production

A healthy claim often shows: NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS payment-data Bound payment-pv 20Gi RWO gp3

If the PVC is Pending, inspect it: kubectl describe pvc payment-data -n production

Possible causes include:

• No suitable PV.
• A missing or invalid StorageClass.
• Failed dynamic provisioning.
• Availability Zone or topology constraints.
• Storage driver problems. Step 4: Check the PV kubectl get pv

Then: kubectl describe pv payment-pv

Verify:

• Status.
• Capacity.
• Reclaim policy.
• StorageClass.
• Claim reference.
• CSI driver and volume handle.
• Node affinity or topology constraints. Step 5: Check the StorageClass kubectl get storageclass

kubectl describe storageclass gp3

Confirm that the provisioner, reclaim policy, volume binding mode, and expansion configuration match your

All right, let's do it from an eight year DevOps brain.

So imagine you're running a payment app on EKS. You create a PVC, and under the hood, EKS with the storage class, provisions, an EBS volume. So the PVC is the Kafka request, and the PV is the actual storage mechanism, often backed by AWS. If the pod crashes, Kubernetes can just restart the pod and remount the same storage. If the worker node dies, the pod can be rescheduled onto another node, and the EBS volume can reattach, but depends on on the reclaim policy policy. If you delete the pod, the deployment controller, like a deployment, comes back up, using the same PVC. and it remounts the same EBS volume. But if you delete the PVC and your reclaim policy is set to delete, the underlying EBS volume may also get deleted, which means data loss if there's no snapshot, backup. If the underlying EBS volume is deleted, then your pods

and PVC objects disappear with the control plane. But the EBS volumes could still exist in AWS if they weren't cleaned up. You'd have to rebuild the cluster, reinstall the storage driver, and reattach the volume, if it's still there. That's why snapshots and backups are so important. For prevention, you'd typically use the retain policy, strict RBAC, so only certain people can delete PVCs or volumes, and regular snapshots. And practice restores, because a backup you never tested is just a theory, not a recovery plan. And for troubleshooting right now, you'd start simple, check the pod events for mount errors, then describe the PVC, then the PV, and confirm the volume ID. If the node died, check the AZ of the EBS volume, because it can only attach within the same AZ. And if the volume was deleted, then you're in restore land. No magic, just backups.