How to Troubleshoot AWS EC2 Disk Space and EBS Volume Issues

Running out of disk space on an AWS EC2 instance can cause serious production issues: system services freeze, web servers drop incoming traffic, and you may even get locked out of SSH because the operating system cannot write temporary authorization files.

Many administrators increase the volume size in the AWS EC2 Management Console, only to discover that the Linux server still reports the disk as 100% full. In Linux, increasing an EBS volume requires two separate actions: modifying the cloud volume, and then expanding the OS partition and filesystem.

---

Quick Answer

After increasing your EBS volume in the AWS Console (or via AWS CLI), connect to your EC2 instance and inspect your block devices:

lsblk
df -hT

If lsblk shows the new disk size but df -hT shows the old smaller filesystem, expand the partition and filesystem online with zero downtime:

# Expand partition 1 on nvme0n1 (or xvda)
sudo growpart /dev/nvme0n1 1

# If ext4 filesystem:
sudo resize2fs /dev/nvme0n1p1

# If XFS filesystem:
sudo xfs_growfs -d /

No reboot is required. The additional space is immediately available.

---

Symptoms

• CloudWatch alarms for DiskSpaceUtilization exceed 95% or 100%.
• SSH logins fail with System is booting up. Unprivileged users are not permitted to log in yet or Cannot create temporary file.
• Databases like MySQL or MongoDB shut down with storage write errors.
• AWS Systems Manager (SSM) agent stops reporting status.
• CloudWatch Agent fails to push metrics or logs.

---

Common Causes

1. Unextended OS filesystem: The EBS volume was modified in AWS from 30GB to 100GB, but the Linux kernel partition and filesystem were never informed of the new geometry.
2. Accumulating application or systemd logs: Unrotated logs in /var/log or /var/log/journal consuming all available space on /dev/root.
3. EBS burst balance exhaustion (gp2 volumes): Small gp2 volumes (under 100GB) running out of burst IOPS credits, causing extreme I/O wait that mimics disk failure.
4. Dangling Docker container images and build caches: /var/lib/docker filling up the root EBS volume.
5. Core dumps: Unconfigured application crashes dumping multi-gigabyte memory dumps into /var/crash or root home.

---

Before You Start

• Take an EBS Snapshot: Always take an automated snapshot of your EBS volume through the AWS Management Console or AWS CLI before performing partition changes:

  aws ec2 create-snapshot --volume-id vol-0a1b2c3d4e5f6g7h8 --description "Pre-disk-resize safety snapshot"

• Verify filesystem type: Linux uses different commands to expand ext4 (resize2fs) and XFS (xfs_growfs). Checking your filesystem type first prevents syntax errors.
• For enterprise AWS management, consult our AWS management services for automated EBS volume scaling.

---

Step 1 — Check Block Devices and Filesystem Types

Log into your EC2 instance over SSH or AWS SSM Session Manager.

Run lsblk to view physical disks and their partition layouts:

lsblk

Sample Output (Modern Nitro Instances - NVMe)

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme0n1       259:0    0  100G  0 disk 
├─nvme0n1p1   259:1    0   30G  0 part /
└─nvme0n1p128 259:2    0    1M  0 part 

What This Tells You

• The physical EBS disk (nvme0n1) is 100GB.
• The partition (nvme0n1p1) mounted on the root directory / is still only 30GB.
• The remaining 70GB is unpartitioned free space waiting to be extended.

Now run df -hT to check the filesystem type:

df -hT /

Expected Output

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 ext4   30G   30G     0 100% /

Note the Type column: it will typically be either ext4 (common on Ubuntu and Debian) or xfs (default on Amazon Linux 2023, Amazon Linux 2, RHEL, and Rocky Linux).

---

Step 2 — Increase the EBS Volume in AWS (If Not Already Done)

If lsblk shows the disk is still the old size, you must first modify the volume in AWS.

You can modify the volume via the AWS Console or using the AWS CLI:

aws ec2 modify-volume --volume-id vol-0a1b2c3d4e5f6g7h8 --size 100 --volume-type gp3

Optimization Tip: If the volume is currently gp2, upgrade it to gp3. Volume type gp3 is up to 20% cheaper than gp2 and provides a baseline 3,000 IOPS and 125 MB/s throughput regardless of storage size.

Wait 1 to 2 minutes until the volume status transitions from modifying to optimizing or completed.

---

Step 3 — Extend the Partition Using growpart

Now tell the Linux kernel to extend the partition to fill the newly available EBS disk capacity.

Install cloud-guest-utils if the growpart utility is not present:

# On Ubuntu / Debian:
sudo apt-get update && sudo apt-get install -y cloud-guest-utils

# On Amazon Linux / RHEL / AlmaLinux:
sudo dnf install -y cloud-utils-growpart

Run growpart to expand partition 1 on the disk:

# Notice the space between the device name and the partition number:
sudo growpart /dev/nvme0n1 1

• For older Xen-based instances using /dev/xvda:

  sudo growpart /dev/xvda 1

Expected Output

CHANGED: partition=1 start=2099200 old: size=60815327 end=62914527 new: size=207615967 end=209715167

Verify that the partition size has grown:

lsblk

nvme0n1p1 will now show the full 100G.

---

Step 4 — Expand the Filesystem Online

Now expand the filesystem so applications can use the extra space. This operation is 100% online and does not interrupt running services.

If Filesystem is ext4:

Use resize2fs on the partition:

sudo resize2fs /dev/nvme0n1p1

(On older instances, substitute /dev/xvda1)

If Filesystem is XFS:

XFS requires specifying the mount point directory, not the device file:

sudo xfs_growfs -d /

The -d parameter tells XFS to expand to the maximum size supported by the underlying partition.

---

Step 5 — Verify Free Space

Run df -hT to verify that the new space is recognized by the operating system:

df -hT /

Healthy Target State

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 ext4   99G   30G   69G  31% /

The filesystem now has 69GB of available free space.

---

Step 6 — What to Do If You Cannot SSH Into the EC2 Instance

If the disk was 100% full before you could expand it and SSH refuses your connection, use this rescue procedure:

1. Stop the broken EC2 instance from the AWS Management Console.
2. Detach the root EBS volume (vol-xxxx).
3. Launch or use a temporary helper EC2 instance in the same Availability Zone.
4. Attach the volume to the helper instance as a secondary disk (e.g., /dev/sdf).
5. SSH into the helper instance and mount the disk:

   sudo mkdir -p /mnt/rescue
   sudo mount /dev/nvme1n1p1 /mnt/rescue

6. Safely clean up large log files in /mnt/rescue/var/log/ or truncate oversized files.
7. Unmount the disk: sudo umount /mnt/rescue
8. Detach the volume from the helper, re-attach it to the original instance as /dev/xvda or /dev/sda1, and start the original instance.

---

Common Mistakes

1. Adding space inside growpart with partition syntax wrong: Running growpart /dev/nvme0n1p1 1 instead of growpart /dev/nvme0n1 1. The first argument is the disk; the second argument is the partition number.
2. Running resize2fs on an XFS partition: Will result in resize2fs: Bad magic number in super-block. Always check df -hT first.
3. Assuming EBS expansion is instant: AWS allows modifying EBS volume size once every 6 hours per volume. Plan your capacity sizing carefully so you do not have to wait 6 hours for another expansion.

---

Prevention Checklist

• Migrate legacy gp2 EBS volumes to gp3 for consistent baseline performance.
• Configure CloudWatch Agent to emit disk_used_percent metrics every 60 seconds.
• Create AWS CloudWatch Alarms that notify your DevOps team via SNS/Slack when disk usage reaches 80%.
• Enable automated AWS Backup or Data Lifecycle Manager (DLM) snapshots for all production EBS volumes.
• Review our AWS management and server backup management plans for automated 24/7 cloud oversight.

---

Quick Reference Commands

Task	NVMe (Modern Instances)	Xen (Legacy Instances)
View disk topology	lsblk	lsblk
View filesystem & type	df -hT /	df -hT /
Expand partition	sudo growpart /dev/nvme0n1 1	sudo growpart /dev/xvda 1
Expand ext4 filesystem	sudo resize2fs /dev/nvme0n1p1	sudo resize2fs /dev/xvda1
Expand XFS filesystem	sudo xfs_growfs -d /	sudo xfs_growfs -d /

---

Frequently Asked Questions

Do I need to reboot my EC2 instance after resizing an EBS volume?

No. AWS Elastic Block Store and the Linux kernel support live online resizing. With growpart and resize2fs or xfs_growfs, the disk, partition, and filesystem can all be expanded while your web server and database are running live.

Why is my EBS volume modification taking a long time?

While the storage capacity is available almost immediately after modification begins, AWS enters an “optimizing” phase in the background where blocks are recalculated. This can take several hours depending on volume size, but the instance can be used normally during optimization.

Can I decrease an EBS volume size if I allocated too much?

No. AWS does not support shrinking EBS volumes directly. To reduce volume size, you must create a new smaller EBS volume, format it, copy your data over (using rsync), and switch the root attachment.

Why does my NVMe device name in Linux differ from the AWS Console?

On modern AWS Nitro-based instances (such as c5, m5, t3, c6g, t4g), EBS volumes are exposed as standard NVMe storage controllers (/dev/nvme0n1, /dev/nvme1n1) rather than legacy Xen block devices (/dev/xvda).

How does ServerCare360 support AWS cloud environments?

Our certified AWS management team provides 24/7 CloudWatch monitoring, automated snapshot lifecycle policies, automated disk expansion scripting, cost-saving gp2 to gp3 migrations, and emergency outage recovery for cloud workloads.