No space left on device – running out of Inodes

One of our development servers went down today. Problems started with deployment script that claimed that claimed “No space left on device”, although partition was not nearly full. If you ever run into such trouble – most likely you have too many small or 0-sized files on your disk, and while you have enough disk space, you have exhausted all available Inodes. Below is the solution for this problem.

1. check available disk space to ensure that you still have some

$ df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda             33030016  10407780  22622236  32% /
tmpfs                   368748         0    368748   0% /lib/init/rw
varrun                  368748        56    368692   1% /var/run
varlock                 368748         0    368748   0% /var/lock
udev                    368748       108    368640   1% /dev
tmpfs                   368748         0    368748   0% /dev/shm

2. check available Inodes

$ df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda            2080768 2080768       0  100% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm
If you have IUse% at 100 or near, then huge number of small files is the reason for “No space left on device” errors.

3. find those little bastards

$ for i in /*; do echo $i; find $i |wc -l; done

This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.

$ for i in /home/*; do echo $i; find $i |wc -l; done

4. once you found the suspect – just delete the files

$ sudo rm -rf /home/bad_user/directory_with_lots_of_empty_files

You’re done. Check the results with df -i command again. You should see something like this:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/xvda            2080768  284431 1796337   14% /
tmpfs                  92187       3   92184    1% /lib/init/rw
varrun                 92187      38   92149    1% /var/run
varlock                92187       4   92183    1% /var/lock
udev                   92187    4404   87783    5% /dev
tmpfs                  92187       1   92186    1% /dev/shm

Comments

114 responses to “No space left on device – running out of Inodes”

  1. Thanks for sharing, helped me out 🙂

  2. Many thanks for this; ran into the mysterious no space left problem and was baffled, but these instructions worked like a charm.

  3. Thx 🙂 So helpfull

  4. Great information – This may have resolved several months of scratching our heads trying to figure out why /var showed 52% usage and the disk space was filling up.

    Great info.

    Rgds,

    Chad

  5. Ivan, you’re my hero of the day !

    I thought I have to make a new installation of my Ubuntu-Lucid Server, but you helped my out.
    My Twonky MediaServer Program creates billons of useless files… 🙁

    Thx
    Guido

  6. Very helpful. We were out of imagination to solve a no disk space problem when the disk actually had enough free space 🙂

  7. Just the cure!

    My /tmp ran of out inodes… mainly in /tmp/orbit-gdm.

    I nuked that folder and went for 100%I to just 1%I usage.

    Thanks!

  8. You saved lots of time , its happens to my amazon ec2 frequently .

  9. Aleksandar

    Hi, does it cover RH linux as well? I tried this syntax for RH linux bu it doesnt work…

  10. Aleksandar

    Hi agains ,

    It is working!!
    I’ve just made mistake ..thanks a lot!

  11. Thanks for the info !
    very useful !

    Greetings

  12. Thanks man…… Thank u very much…..

  13. Some times is difficult to erase the files with the command rm -f -r /tmp/*

    In such situation you could you use the next command:

    sudo find / -type f -print | xargs — sudo rm -f —

    This tip is for all the people that stay in the fire line jaja lol ; )

  14. Just Rocking ! Gr8.

  15. vetras

    Thanks for sharing, helped me out. Its exactly that problem… 😀

  16. We also had this problem, thanks 🙂

  17. […] are running out of inodes. If you are able to identify a value near 100% follow the instructions on Ivan Kuznetsov’s blog. This entry was posted in Linux, Linux Mint, Oracle ADF and tagged debian, JDeveloper, no space […]

  18. Thanks a lot for sharing!
    This help us, and prevent from helping so frequently. You’re so kind!!

  19. […] эти файлы. Спасибо Ивану Кузнецову за метод for i in /*; do echo $i; find $i |wc -l; […]

  20. Great help! my webserver hung the same way. find /opt showed Zend application kept millions of sessions so rm -R zendapp/var/sessions & mkdir & chmod

  21. It’s Very HelpFul…
    Thnaks

  22. I also had this problem and it helped me out. Thanks a lot for sharing!

  23. Ivan, is there a simple way to increase the number of available inodes? Because due to the nature of my backups I use them all up after about 3 months. I’d happily sacrifice a few gigs to quadruple the available inodes… if it’s possible, and if it doesn’t cause other issues.

  24. Thanks, this put me into right track.
    For all to know: /var/log/* {daily} is a bad, bad thing to have in your logrotate.conf… 🙂

  25. Thank you! It worked 🙂

  26. Alexander Mednikov

    Спасибо, Ваня, помогло.

  27. Saved my ass. Thanks

  28. TwoStarII

    Great, Thanks This Solved My Problem!

  29. Knuckle Depressed

    … – So give yourself the question, thus becomes, what is it likely to present reality. In other words, it is used in the health and energy symptoms like anxiety and depression. … Ivan Kuznetsov › No space left on device – running out of Inodes …

  30. Tonny

    I never write comments, but this inode article probably best on web! Thanks this helped me to diagnose inode filled folders and clear them out! Other solutions were not helpful because system reached 100% inodes and any complex command line was not possible because out of inode space!

  31. I was very happy to uncover this site. I wanted to thank you for your time just for this fantastic read!!
    I definitely enjoyed every little bit of it and I have you saved as a favorite to
    see new stuff in your site.
       hashmi. . . . . .  .

  32. Much obliged for this; kept running into the secretive no space left issue and was astounded, however these directions had exactly the intended effect.

  33. Jay Rajput

    Why not use the find command with size option to find all the files and then delete them

    find -type f -size 0 | xargs rm

  34. Am Feeling Bad

    … – None of the time that their heart as I mentioned at the start, you will have a purpose. That onslaught of the most common group of australian scientists did not know how to deal… Ivan Kuznetsov › No space left on device – running out of Inodes…

  35. Maswood Alam

    Really usefull information ..Thanks

  36. Javed Shaikh

    Thanks alot for the help..This was very helpful

  37. […] /home/* — это путь, который можно менять для поиска в конкретном месте. Подсмотрено здесь. […]

  38. Thank you for sharing but it will not be always the best solution to search for the directory having the most files, sometime you have a directory that have one file and this file is the biggest file in your server.
    In my case I had a file that took alone 77% of my device space.
    I recommand searching for the biggest directories instead of directories having the most files:
    This command will list the top 10 largest file/directories under the directory var for example.
    du -a /var | sort -n -r | head -n 10

  39. You are batman of LiNuX city 🙂 made my day!! thanx…

  40. Thx. Very helpful.
    Now is the MySQL/Writing file error (Errcode 28) problem fixed 🙂

  41. Thanks! I was looking for suspiciously large files but the key was in the small ones filling the inodes 😉

  42. Genius! Thanks for sharing, helped me out massively.

  43. Sys Admin

    Thanks a ton! You saved my day!

  44. […] No space left on device – running out of Inodes […]

  45. saved my day. thanks a lot man.

    just want to be confirm about 1 thing. that command only delete bad empty files not important files.
    because i have just delete one folder with that command and freed up 5% inodes. but i have couple of more folders with large number of files but im a bit scared to apply this with those.

    because i used this command
    sudo rm -rf /usr/src

    and now i can see the whole src folder is now deleted.

  46. you really help me.

    Thanks

  47. All those files are to important to be deleted, i am using wordpress. Please Help me

  48. I can’t thank you enough. You just saved my weekend. This was the exact solution I needed today to resolve an emergency on a failing server. 1k Kudos to you!

  49. Lovely guide to resolving, thanks.

  50. QMan-Junkie-Files

    This is the best guide on this topic. Gazillion thanks!

  51. Awesome guide, to bad it took me that long to actually find this…

  52. Physicist turned software engineer turned entrepreneur turned product manager TURNED saviour! you’re my hero 🙂 thanks heaps for saving me!

  53. Alexandre Pawlowski

    Thank you guy so much for this guide!

    I’ve been woken up at 04:00 in the morning to fix a critical Solaris server, this helped me to solve this issue in less than 30 minutes.

    Way to go!

  54. Thank You so Much..:)

  55. may be somewhat faster to attack the suspected fodler

    for i in /*; do count=`find $i | wc -l`; if [ $count -gt 1000 ]; then echo $i $count; fi; done

  56. You are awesome mate! thanks for the wonderful tutorial! Greetings from Greece. You just mentioned the absolutely necessary stuff. No hassle!

  57. Thank you alot, your article really saved me, Simon from South Africa

  58. Thank you for sharing this was really helpful. Gonna recommend to my friends also 🙂

  59. […] No space left on device – running out of Inodes – Ivan … – One of our development servers went down today. Problems started with deployment script that claimed that claimed “No space left on device”, although partition … […]

  60. Thank you! (/var/glpi _sessions folder on Debian…)
    Have a nice day

  61. […] damit hatte ich bis anhin noch nicht zu kämpfen, was macht man da also ? In diesem Blog konnte ich dann was dazu […]

  62. Impossible

    Here is another bump.

    You helped me find the little bastards and cheer me up at the same time

  63. Thanks for this!! Great indeed 🙂

  64. I have never heard a word “inodes” prior to this day. My server has crashed and was down for over half of the day, I spend 4 hours trying to find out how to fix it until I found this tutorial. Seems that deleting old backup files + all the postfix files after using all these scripts resolved the problem with 100% usage of inodes. From the time I read this article till my server fix there was just about 40 min difference. Thanks man, its really saved my day (or night, since it is 1 am now).

  65. Prabhat

    Thank you for the nice troubleshooting. If this method did not help us then is there any other method to troubleshoot.

  66. Perfect cheers

  67. Fernando

    Thank you so much! I have been using Linux for 4 years and, except for the first half year which was tough (coming from Windows I guess it is understandable), my work is much faster since then and I cannot be happier. But today I had this problem with the “inodes” (I had never heard of them until today) and started to sweat. I was thinking of formatting my laptop but you saved me a lot of time.
    Thank you very much again!

  68. thx a lot!

  69. Great. Worked well. Thanks.

  70. Hi,
    Could you please provide the script for disk space/old logs delete.
    The script should show the difference between disk space (in %) before and after cleared the logs.
    Thanks,
    Eswar

  71. Excellent guide – thanks!

  72. Great! Solved my problem!

  73. Thank you so much for this wonderful article. This saved my day. I followed the steps and found out that the /usr/src was taking up huge amount of space. I ran the for loop in there to find out that it were all the linux packages. It struck to me that it might be the apt-get cache and the apt-get packages that might be unused.

    I ran the following commands to clear the system.

    apt-get autoclean
    apt-get autoremove

    This brought down the inodes usage from 100% to 26%.

  74. Concise and good. Thank you very much!

  75. Well, I came here to say you just saved another man’s life

  76. Thanks a ton mate, Helped a lot for my P1 critical issues.

  77. Just want to say THANK YOU for sharing these special things… free download corel draw 12 full version with serial number

  78. This fixed my problem perfectly! Awesome.

  79. great article.

  80. Be careful, this method doesn’t check special directories, with a dot before (/root/.cache as example).

  81. Deepak Sahni

    Thank you so much. You are a champ !!

  82. Thanks for good guide and simple. I go to the folder (/var/lib/oracle/grid/rdbms/audit) which contain 3762154 files of type +ASM_ora_…., removing all these files there’l be no impact to the database. I use oracle linux.

    Cheers!

  83. […] No space left on device – running out of Inodes […]

  84. This fixed my issue. Thanks

  85. hai
    this article was good.
    but i dnt get d result.
    when i check: df -i

    tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
    /dev/loop2 82M 82M 0 100% /snap/core/4206
    /dev/loop3 82M 82M 0 100% /snap/core/4110
    cgmfs 100K 0 100K 0% /run/cgmanager/fs
    tmpfs 362M 96K 362M 1% /run/user/1000

    am not able ti remove d snap/core

    plz any one help me

  86. Many thanks to you Ivan!!!

  87. Darshak Shah

    This solution worked like a charm for me. I faced this issue on AWS EC2 Linux instance. Using the technique above by Ivor, I realized that the culprit was /usr/src directory, where kernel header files are copied during instance update.
    Deleting all files from /usr/src directory solved the inode count issue .. Just too good Ivor. Thanks a ton.

  88. Thank you so much ! There’s so much bullshit everywhere on the subject and you just solve it in a few lines with some nice explanations.

  89. I had deleted the /usr/src/ content where linux headers were present. This led to breaking the key based authentication with AWS EC2.
    Be careful, while performing this activity. Instead, I went for deleting some user sessions logged on server and it brought the site back.

  90. Alberto

    Lo máximo .. muchas gracias. Saludos desde Perú

  91. THANKS SO MUCH! Solved an emergency situation!

  92. […] directories and figure out where they’re coming from, and then manage them. Here’s a great article describing how to do that, and you should try this option if the second option doesn’t […]

  93. This is very good explanation. Thanks, Ivan!

  94. David Price

    Thank you!

    As an ex Windows user this one was really baffling until I worked through your post. Cheers!

  95. Hi,
    I have a question. Is it safe to delete all files & folders from these directories?
    I found most of the file that have high inodes is in:
    – /usr/share/perl/5.18.2 (mostly .pm file)
    – /usr/share/doc
    – /usr/lib/python2.7
    – /usr/lib/python3
    – /usr/lib/python3.4
    – /usr/lib/perl
    – /sys/module
    – /sys/kernel/debug

    Just want to know if it is safe to delete all of these files.
    Thank you in advance!

  96. thank you so much very useful

  97. Ian B

    `for i in /home/*; do echo $i; find $i |wc -l; done`
    you need to quote the $i

  98. My man, OH MY MAN! This little girl ows you days of her life! Thank you so mcho for this comprehensive and clear guide.

    If you happen to visit Panama, contact me and lets chat about tech and drink some coffee (tea for me) or beer! Thank you!

  99. I see you don’t monetize ivankuznetsov.com,
    don’t waste your traffic, you can earn extra cash every month with new monetization method.
    This is the best adsense alternative for any type of website (they approve all sites), for more info simply
    search in gooogle: murgrabia’s tools

  100. This Article is very useful, saved my day.

  101. Love you yaar

  102. Thanks a lot. You saved my day.

  103. Thanks you very muchy

  104. […] directories and figure out where they’re coming from, and then manage them. Here’s a great article describing how to do that, and you should try this option if the second option doesn’t […]

  105. anh tuan

    it’s not working for me

  106. Can someone please clarify this part with a real example? /home/bad_user/directory_with_lots_of_empty_files

  107. AWESOME!!! THANK!!!

  108. GREG, its a dir where u see the most inodes and u want to clean it up

  109. I tried this way.
    I went to
    cd/
    then ran this command
    I’ve found that at one folder it is taking too much time.
    i pressed
    Ctrl+C (to break the process)
    then I change the command with the directory where it was stopped and run it again.
    i tried the same thing 3-4 times.
    and i found one folder where there were 4096 folders
    inside each folder there were more than 3000+ files / folder
    so i’ve removed that and everything went well… the inode was showing 50% usage now what it was showing 100% earlier 🙂

    Thanks again

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.