How to Fix “No Space Left on Device” on a Linux Server

Seeing bash: cannot create temp file: No space left on device is one of the most disruptive errors a systems administrator encounters. Services like MySQL immediately stop accepting new writes, web servers fail to create temporary session files, and SSH shell logins may fail completely.

Fixing this error requires verifying whether you have exhausted disk blocks (actual storage megabytes), exhausted inodes (file counter metadata), or if unlinked deleted files are still held open by running processes.

Quick Answer

Check both disk blocks and inodes immediately:

df -h
df -i

If df -h shows 100% on a partition, locate large files using du -ahx /var | sort -rh | head -n 20.

If df -h shows free space but you still get the error, check df -i. An inode usage of 100% means millions of tiny files have consumed all metadata entries.

If neither shows 100%, check for deleted files held open by processes with lsof +L1.

Symptoms

Applications fail with write error: No space left on device or ENOSPC.
MySQL/MariaDB shuts down with errors like Disk is full writing './binlog/...'.
Crontab jobs fail to write output or create locks.
Tab completion in Bash stops functioning.
Web applications return HTTP 500 errors due to session creation failures in /var/lib/php/sessions or /tmp.

Common Causes

Unrotated log files: Stale logs in /var/log or container directories growing to tens of gigabytes.
Inode exhaustion (df -i 100%): Millions of tiny session files, email spools, or cache files filling up the inode table even while gigabytes of disk space remain free.
Unlinked open files: An administrator deletes a huge log file with rm, but the application writing to it never restarted. Linux keeps the disk blocks allocated until the process closes the file descriptor.
Package manager caches: Old APT (/var/cache/apt/archives) or DNF cache archives remaining after system updates.
Dangling Docker artifacts: Unpruned container layers, anonymous volumes, and build caches accumulating in /var/lib/docker.

Before You Start

Never delete active database files inside /var/lib/mysql or /var/lib/postgresql. Removing .ibd or transaction log files will cause permanent data loss.
Truncate log files rather than deleting them if a live process is actively logging. Truncating (> /path/to/logfile) zeroes out the file immediately without breaking the open file descriptor.
If your partition is managed on AWS or a cloud hypervisor, you may also need to expand your storage volume. Follow our guide on AWS EC2 EBS disk space troubleshooting.

Step 1 — Distinguish Block Full vs Inode Full

Log into the server and run these two commands.

Check Storage Block Usage

df -h -x tmpfs -x devtmpfs

Look at the Use% and Mounted on columns:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        50G   50G     0 100% /
/dev/vda15      105M  6.1M   99M   6% /boot/efi

If the root filesystem / is at 100%, you have a standard storage block shortage.

Check Inode Usage

If df -h shows plenty of free space (for example, 40% used), run:

df -i -x tmpfs -x devtmpfs

Look at the IUse% column:

Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/root       3276800 3276800       0  100% /

If IUse% is 100%, the disk has space, but the filesystem cannot create a single new file because every allocated inode record is taken.

Step 2 — Find Large Files and Directories

If storage blocks are 100% full, locate the directories consuming the most disk space.

Run this command starting at the root directory:

du -ahx / 2>/dev/null | sort -rh | head -n 25

The -x flag ensures du stays on one filesystem and avoids scanning mounted network drives or virtual filesystems like /proc.
The -h flag prints human-readable sizes (G, M).
sort -rh | head -n 25 sorts the largest consumers to the top.

Common Hotspots to Check First

System logs: /var/log
Systemd journal: /var/log/journal
Web server logs: /var/log/nginx/ or /var/log/httpd/
Docker storage: /var/lib/docker
Temporary files: /tmp and /var/tmp
Application backups: /home/*/backups or /root/backup

Step 3 — Detect Deleted Files Held Open by Processes

If df -h shows 100% full, but du -sh / reports only 20GB used on a 50GB disk, a deleted file is still held open by an active process.

Run lsof to find unlinked files with active references:

lsof +L1

Or run:

lsof | grep '(deleted)' | sort -nr -k 7 | head -n 15

Expected Output

COMMAND    PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK     NODE NAME
nginx    14012 root    2w   REG  253,0 28410294120     0  2819124 /var/log/nginx/access.log (deleted)

In this example, Nginx is holding open a 28GB deleted log file (access.log). Even though the file was deleted with rm, Linux cannot free the blocks until Nginx reloads or restarts.

To free the space immediately without causing downtime, reload the process:

systemctl reload nginx

If the process cannot be reloaded, truncate the file descriptor directly via /proc:

# Replace 14012 with the PID and 2 with the FD number from lsof output:
: > /proc/14012/fd/2

Step 4 — Fix Inode Exhaustion (100% Inodes)

If df -i showed 100% inode usage, pinpoint the directory containing millions of tiny files:

find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -k 1 -n | tail -n 15

This counts the number of files in each folder across the filesystem.

Typical Culprits for Inode Exhaustion

PHP Sessions: Millions of abandoned sessions in /var/lib/php/sessions:
```
find /var/lib/php/sessions -type f -cmin +1440 -delete
```
Mail Spools: Bounced messages or undelivered mail in /var/spool/postfix/maildrop or /var/spool/clientmqueue:
```
find /var/spool/postfix/maildrop -type f -delete
```
Application cache directories: Millions of thumbnail or template cache files in /var/www/html/cache/.

Step 5 — Safe Cleanup Commands

Apply these safe cleanup commands to reclaim gigabytes of disk space immediately:

1. Vacuum Systemd Journals

Restrict systemd logs to the last 3 days or a maximum size:

journalctl --vacuum-time=3d
journalctl --vacuum-size=500M

2. Clean Package Manager Caches

On Ubuntu/Debian:

apt-get clean
apt-get autoremove -y

On AlmaLinux/Rocky Linux/RHEL:

dnf clean all

3. Safely Truncate Oversized Active Logs

Never rm active logs. Zero them out using truncation:

truncate -s 0 /var/log/nginx/access.log
truncate -s 0 /var/log/apache2/other_vhosts_access.log

4. Prune Unused Docker Storage

If you run Docker, prune unused stopped containers, dangling images, and build caches:

docker system prune -f

Step 6 — Verify Available Space and Inodes

Run df again to confirm that disk blocks and inodes are back below safe thresholds:

df -h /
df -i /

Healthy Target State

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        50G   18G   30G  38% /

Ensure Use% and IUse% are below 80%.

Common Mistakes

Deleting active MySQL binlogs with rm: This desynchronizes the MySQL binary log index (binlog.index), preventing the database from starting. Always purge binlogs through the MySQL shell using PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;.
Forgetting to configure logrotate: Clearing a log file manually solves today’s emergency, but without log rotation (/etc/logrotate.conf), the file will fill the disk again next week.
Deleting /tmp directory instead of its contents: Running rm -rf /tmp removes critical sticky bit permissions (chmod 1777 /tmp), breaking software logins and temporary sockets.

Prevention Checklist

Set up infrastructure monitoring alerts at 80% and 90% disk utilization.
Verify logrotate is running daily: systemctl status logrotate.timer.
Implement automated cron cleanup for PHP session directories.
Configure Docker log limits (max-size: "50m", max-file: "3") in /etc/docker/daemon.json.
Maintain regular, verified backups with our server backup management solutions.

Quick Reference Commands

Operation	Command
Check disk space	`df -h`
Check inode usage	`df -i`
Find 20 largest files/folders	`du -ahx / \| sort -rh \| head -20`
Detect deleted open files	`lsof +L1`
Truncate active log file	`truncate -s 0 /path/to/log.log`
Vacuum journal logs	`journalctl --vacuum-size=200M`
Clean APT package cache	`apt-get clean`
Clean DNF/YUM cache	`dnf clean all`

Frequently Asked Questions

Why does df -h say 100% full, but du shows only a fraction used?

The most common reason is deleted files held open by active processes. When you run rm /path/to/huge.log, the filename is unlinked from the filesystem directory, but the operating system kernel cannot free the disk blocks until the process writing to that file closes its file handle. Run lsof +L1 to find these processes and reload them.

Can running out of disk space corrupt my database?

Yes. When MySQL or PostgreSQL attempts to commit an active transaction or write to the undo/redo log and the write operation fails with ENOSPC, the database daemon may crash into recovery mode. In worst-case scenarios with unfinished page writes, table corruption can occur.

What is an inode, and why does it cause disk full errors?

An inode is a data structure on Linux filesystems that stores metadata about a file (permissions, owner, size, physical block locations). Every file and directory consumes exactly one inode. If a partition runs out of inodes, no new files can be created, even if there are hundreds of gigabytes of raw storage free.

How do I prevent PHP session files from filling all inodes?

Ensure your distribution’s automated session cleanup cron job or systemd timer is enabled (systemctl status phpsessionclean.timer on Debian/Ubuntu). Alternatively, configure PHP to store sessions in Redis or Memcached instead of local disk storage.

How does ServerCare360 assist with recurring storage emergencies?

Our Linux server support and emergency server support engineers configure automated log rotation, deploy proactive multi-threshold disk and inode alerts, tune database retention policies, and execute safe zero-downtime storage volume expansions.