No space left on device – running out of Inodes

One of our development servers went down today. Problems started with deployment script that claimed that claimed “No space left on device”, although partition was not nearly full. If you ever run into such trouble – most likely you have too many small or 0-sized files on your disk, and while you have enough disk space, you have exhausted all available Inodes. Below is the solution for this problem.

1. check available disk space to ensure that you still have some

$ df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda             33030016  10407780  22622236  32% /
tmpfs                   368748         0    368748   0% /lib/init/rw
varrun                  368748        56    368692   1% /var/run
varlock                 368748         0    368748   0% /var/lock
udev                    368748       108    368640   1% /dev
tmpfs                   368748         0    368748   0% /dev/shm

2. check available Inodes

$ df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda            2080768 2080768       0  100% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm
If you have IUse% at 100 or near, then huge number of small files is the reason for “No space left on device” errors.

3. find those little bastards

$ for i in /*; do echo $i; find $i |wc -l; done

This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.

$ for i in /home/*; do echo $i; find $i |wc -l; done

4. once you found the suspect – just delete the files

$ sudo rm -rf /home/bad_user/directory_with_lots_of_empty_files

You’re done. Check the results with df -i command again. You should see something like this:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/xvda            2080768  284431 1796337   14% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm

114 thoughts on “No space left on device – running out of Inodes”

  1. `for i in /home/*; do echo $i; find $i |wc -l; done`
    you need to quote the $i

  2. My man, OH MY MAN! This little girl ows you days of her life! Thank you so mcho for this comprehensive and clear guide.

    If you happen to visit Panama, contact me and lets chat about tech and drink some coffee (tea for me) or beer! Thank you!

  3. I see you don’t monetize,
    don’t waste your traffic, you can earn extra cash every month with new monetization method.
    This is the best adsense alternative for any type of website (they approve all sites), for more info simply
    search in gooogle: murgrabia’s tools

  4. Can someone please clarify this part with a real example? /home/bad_user/directory_with_lots_of_empty_files

  5. I tried this way.
    I went to
    then ran this command
    I’ve found that at one folder it is taking too much time.
    i pressed
    Ctrl+C (to break the process)
    then I change the command with the directory where it was stopped and run it again.
    i tried the same thing 3-4 times.
    and i found one folder where there were 4096 folders
    inside each folder there were more than 3000+ files / folder
    so i’ve removed that and everything went well… the inode was showing 50% usage now what it was showing 100% earlier 🙂

    Thanks again

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.