No space left on device – running out of Inodes

One of our development servers went down today. Problems started with deployment script that claimed that claimed “No space left on device”, although partition was not nearly full. If you ever run into such trouble – most likely you have too many small or 0-sized files on your disk, and while you have enough disk space, you have exhausted all available Inodes. Below is the solution for this problem.

1. check available disk space to ensure that you still have some

$ df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda             33030016  10407780  22622236  32% /
tmpfs                   368748         0    368748   0% /lib/init/rw
varrun                  368748        56    368692   1% /var/run
varlock                 368748         0    368748   0% /var/lock
udev                    368748       108    368640   1% /dev
tmpfs                   368748         0    368748   0% /dev/shm

2. check available Inodes

$ df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda            2080768 2080768       0  100% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm
If you have IUse% at 100 or near, then huge number of small files is the reason for “No space left on device” errors.

3. find those little bastards

$ for i in /*; do echo $i; find $i |wc -l; done

This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.

$ for i in /home/*; do echo $i; find $i |wc -l; done

4. once you found the suspect – just delete the files

$ sudo rm -rf /home/bad_user/directory_with_lots_of_empty_files

You’re done. Check the results with df -i command again. You should see something like this:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/xvda            2080768  284431 1796337   14% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm
Share

39 thoughts on “No space left on device – running out of Inodes”

  1. Many thanks for this; ran into the mysterious no space left problem and was baffled, but these instructions worked like a charm.

  2. Great information – This may have resolved several months of scratching our heads trying to figure out why /var showed 52% usage and the disk space was filling up.

    Great info.

    Rgds,

    Chad

  3. Ivan, you’re my hero of the day !

    I thought I have to make a new installation of my Ubuntu-Lucid Server, but you helped my out.
    My Twonky MediaServer Program creates billons of useless files… 🙁

    Thx
    Guido

  4. Very helpful. We were out of imagination to solve a no disk space problem when the disk actually had enough free space 🙂

  5. Just the cure!

    My /tmp ran of out inodes… mainly in /tmp/orbit-gdm.

    I nuked that folder and went for 100%I to just 1%I usage.

    Thanks!

  6. Hi, does it cover RH linux as well? I tried this syntax for RH linux bu it doesnt work…

  7. Some times is difficult to erase the files with the command rm -f -r /tmp/*

    In such situation you could you use the next command:

    sudo find / -type f -print | xargs — sudo rm -f —

    This tip is for all the people that stay in the fire line jaja lol ; )

  8. Thanks a lot for sharing!
    This help us, and prevent from helping so frequently. You’re so kind!!

  9. Great help! my webserver hung the same way. find /opt showed Zend application kept millions of sessions so rm -R zendapp/var/sessions & mkdir & chmod

  10. Ivan, is there a simple way to increase the number of available inodes? Because due to the nature of my backups I use them all up after about 3 months. I’d happily sacrifice a few gigs to quadruple the available inodes… if it’s possible, and if it doesn’t cause other issues.

  11. Thanks, this put me into right track.
    For all to know: /var/log/* {daily} is a bad, bad thing to have in your logrotate.conf… 🙂

  12. Pingback: Knuckle Depressed
  13. I never write comments, but this inode article probably best on web! Thanks this helped me to diagnose inode filled folders and clear them out! Other solutions were not helpful because system reached 100% inodes and any complex command line was not possible because out of inode space!

  14. I was very happy to uncover this site. I wanted to thank you for your time just for this fantastic read!!
    I definitely enjoyed every little bit of it and I have you saved as a favorite to
    see new stuff in your site.
       hashmi. . . . . .  .

  15. Much obliged for this; kept running into the secretive no space left issue and was astounded, however these directions had exactly the intended effect.

  16. Why not use the find command with size option to find all the files and then delete them

    find -type f -size 0 | xargs rm

  17. Pingback: Am Feeling Bad

Leave a Reply

Your email address will not be published. Required fields are marked *