Skip to content

No space left on device – running out of Inodes

One of our development servers went down today. Problems started with deployment script that claimed that claimed “No space left on device”, although partition was not nearly full. If you ever run into such trouble – most likely you have too many small or 0-sized files on your disk, and while you have enough disk space, you have exhausted all available Inodes. Below is the solution for this problem.

1. check available disk space to ensure that you still have some

$ df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda             33030016  10407780  22622236  32% /
tmpfs                   368748         0    368748   0% /lib/init/rw
varrun                  368748        56    368692   1% /var/run
varlock                 368748         0    368748   0% /var/lock
udev                    368748       108    368640   1% /dev
tmpfs                   368748         0    368748   0% /dev/shm

2. check available Inodes

$ df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda            2080768 2080768       0  100% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm
If you have IUse% at 100 or near, then huge number of small files is the reason for “No space left on device” errors.

3. find those little bastards

$ for i in /*; do echo $i; find $i |wc -l; done

This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.

$ for i in /home/*; do echo $i; find $i |wc -l; done

4. once you found the suspect – just delete the files

$ sudo rm -rf /home/bad_user/directory_with_lots_of_empty_files

You’re done. Check the results with df -i command again. You should see something like this:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/xvda            2080768  284431 1796337   14% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm
Share

42 Comments

  1. Mads wrote:

    Thanks for sharing, helped me out :)

    Thursday, August 19, 2010 at 22:33 | Permalink
  2. Jack wrote:

    Many thanks for this; ran into the mysterious no space left problem and was baffled, but these instructions worked like a charm.

    Monday, September 27, 2010 at 20:02 | Permalink
  3. philippe wrote:

    Thx :) So helpfull

    Monday, December 13, 2010 at 13:48 | Permalink
  4. chad wrote:

    Great information – This may have resolved several months of scratching our heads trying to figure out why /var showed 52% usage and the disk space was filling up.

    Great info.

    Rgds,

    Chad

    Wednesday, January 5, 2011 at 19:38 | Permalink
  5. Guido wrote:

    Ivan, you’re my hero of the day !

    I thought I have to make a new installation of my Ubuntu-Lucid Server, but you helped my out.
    My Twonky MediaServer Program creates billons of useless files… :-(

    Thx
    Guido

    Wednesday, February 9, 2011 at 21:41 | Permalink
  6. Tikki wrote:

    Very helpful. We were out of imagination to solve a no disk space problem when the disk actually had enough free space :-)

    Thursday, February 17, 2011 at 16:21 | Permalink
  7. Manny wrote:

    Just the cure!

    My /tmp ran of out inodes… mainly in /tmp/orbit-gdm.

    I nuked that folder and went for 100%I to just 1%I usage.

    Thanks!

    Wednesday, March 9, 2011 at 19:11 | Permalink
  8. Chetan wrote:

    You saved lots of time , its happens to my amazon ec2 frequently .

    Saturday, June 11, 2011 at 11:00 | Permalink
  9. Aleksandar wrote:

    Hi, does it cover RH linux as well? I tried this syntax for RH linux bu it doesnt work…

    Wednesday, August 10, 2011 at 9:19 | Permalink
  10. Aleksandar wrote:

    Hi agains ,

    It is working!!
    I’ve just made mistake ..thanks a lot!

    Wednesday, August 10, 2011 at 10:50 | Permalink
  11. Alberto wrote:

    Thanks for the info !
    very useful !

    Greetings

    Monday, October 31, 2011 at 17:21 | Permalink
  12. Thanks man…… Thank u very much…..

    Sunday, November 20, 2011 at 20:58 | Permalink
  13. Some times is difficult to erase the files with the command rm -f -r /tmp/*

    In such situation you could you use the next command:

    sudo find / -type f -print | xargs — sudo rm -f –

    This tip is for all the people that stay in the fire line jaja lol ; )

    Thursday, December 22, 2011 at 22:17 | Permalink
  14. Abser wrote:

    Just Rocking ! Gr8.

    Monday, January 16, 2012 at 11:17 | Permalink
  15. vetras wrote:

    Thanks for sharing, helped me out. Its exactly that problem… :D

    Monday, April 9, 2012 at 18:04 | Permalink
  16. We also had this problem, thanks :)

    Saturday, April 28, 2012 at 0:19 | Permalink
  17. merc wrote:

    Thanks a lot for sharing!
    This help us, and prevent from helping so frequently. You’re so kind!!

    Thursday, July 26, 2012 at 11:33 | Permalink
  18. KroonWeb wrote:

    Great help! my webserver hung the same way. find /opt showed Zend application kept millions of sessions so rm -R zendapp/var/sessions & mkdir & chmod

    Wednesday, September 12, 2012 at 0:01 | Permalink
  19. Marlin wrote:

    It’s Very HelpFul…
    Thnaks

    Tuesday, September 18, 2012 at 4:14 | Permalink
  20. JLau wrote:

    I also had this problem and it helped me out. Thanks a lot for sharing!

    Friday, November 2, 2012 at 10:59 | Permalink
  21. Dan W. wrote:

    Ivan, is there a simple way to increase the number of available inodes? Because due to the nature of my backups I use them all up after about 3 months. I’d happily sacrifice a few gigs to quadruple the available inodes… if it’s possible, and if it doesn’t cause other issues.

    Friday, December 21, 2012 at 0:03 | Permalink
  22. Kicune wrote:

    Thanks, this put me into right track.
    For all to know: /var/log/* {daily} is a bad, bad thing to have in your logrotate.conf… :)

    Friday, February 15, 2013 at 18:53 | Permalink
  23. makooota wrote:

    Thank you! It worked :)

    Thursday, February 28, 2013 at 4:05 | Permalink
  24. Alexander Mednikov wrote:

    Спасибо, Ваня, помогло.

    Saturday, March 9, 2013 at 23:14 | Permalink
  25. Ryan wrote:

    Saved my ass. Thanks

    Tuesday, March 12, 2013 at 3:22 | Permalink
  26. Robert Patag wrote:

    Ivan,

    Thanks for guiding me in the right direction. We couldn’t delete the files, but ended up extending the file system to increase the # of inodes.

    Saturday, March 23, 2013 at 6:44 | Permalink
  27. Matthew Gladman wrote:

    OMG THANK YOU!

    We were having this issue for the past few days, I was like “there is 3GB free, it is not out of space, what is going on!?”.

    But one of our apps create a bunch of tiny cache files and apparently that filled up the inodes! :/

    Wednesday, March 27, 2013 at 1:13 | Permalink
  28. TwoStarII wrote:

    Great, Thanks This Solved My Problem!

    Wednesday, June 26, 2013 at 10:51 | Permalink
  29. SB wrote:

    Thanks mate for this article, save my ass too :)

    Tuesday, July 23, 2013 at 18:10 | Permalink
  30. Mauricio wrote:

    Thanks!!!!

    Wednesday, August 14, 2013 at 23:53 | Permalink
  31. Grezer wrote:

    Thank You very much!!!!

    You saved my day! :) ))))

    Friday, October 11, 2013 at 20:56 | Permalink
  32. Nikhilesh Sharma wrote:

    Thank you very much. Helped me out.

    Monday, November 25, 2013 at 9:21 | Permalink
  33. ibpark wrote:

    Win7/cygwin

    $ cd /q
    -bash: cd: /q: No space left on device

    and access denied by shell, any apps

    How can I fix that?

    Saturday, January 18, 2014 at 20:14 | Permalink
  34. Hari wrote:

    Thanks dude! Really helpful!! :)

    Monday, March 31, 2014 at 13:34 | Permalink
  35. Jehan wrote:

    Thnk u bro. it helped me out

    Monday, March 31, 2014 at 14:09 | Permalink
  36. Sweety wrote:

    I get the same error every time I try to copy some files. But when I execute the command it says
    for :command nit found
    i: undefined variable

    What should be done in this case. Please help me

    Thursday, April 3, 2014 at 18:54 | Permalink
  37. dsdwn wrote:

    NNIICCEE !

    Thursday, April 10, 2014 at 13:36 | Permalink
  38. done!
    Many thanks!!! :D

    Friday, April 25, 2014 at 8:46 | Permalink
  39. Jomon wrote:

    Luckily I came across this page before trying all those blind shots! You saved my day… Thanks a lot!

    Friday, April 25, 2014 at 11:15 | Permalink
  40. Maverick wrote:

    Can you explain the shell script , you have used : STEP BY STEP

    Friday, May 9, 2014 at 15:13 | Permalink
  41. Maverick wrote:

    what ‘s the use of >>>>>find $i |wc -l<<<<
    line , please somebody explain , Thanks in advance

    Friday, May 9, 2014 at 15:15 | Permalink
  42. David wrote:

    Thanks, it helped us out a lot !

    Tuesday, June 10, 2014 at 13:31 | Permalink

4 Trackbacks/Pingbacks

  1. problem mit vserver nichts geht mehr on Sunday, October 23, 2011 at 18:42

    [...] [...]

  2. [...] are running out of inodes. If you are able to identify a value near 100% follow the instructions on Ivan Kuznetsov’s blog. This entry was posted in Linux, Linux Mint, Oracle ADF and tagged debian, JDeveloper, no space [...]

  3. No space left on device « *nix blog on Thursday, July 26, 2012 at 19:22

    [...] эти файлы. Спасибо Ивану Кузнецову за метод for i in /*; do echo $i; find $i |wc -l; [...]

  4. [...] After some google, found an article where OP suggested to check for inodes: http://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *
*
*