One of our development servers went down today. Problems started with deployment script that claimed that claimed “No space left on device”, although partition was not nearly full. If you ever run into such trouble – most likely you have too many small or 0-sized files on your disk, and while you have enough disk space, you have exhausted all available Inodes. Below is the solution for this problem.
1. check available disk space to ensure that you still have some
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 33030016 10407780 22622236 32% /
tmpfs 368748 0 368748 0% /lib/init/rw
varrun 368748 56 368692 1% /var/run
varlock 368748 0 368748 0% /var/lock
udev 368748 108 368640 1% /dev
tmpfs 368748 0 368748 0% /dev/shm
2. check available Inodes
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda 2080768 2080768 0 100% /
tmpfs 92187 3 92184 1% /lib/init/rw
varrun 92187 38 92149 1% /var/run
varlock 92187 4 92183 1% /var/lock
udev 92187 4404 87783 5% /dev
tmpfs 92187 1 92186 1% /dev/shm
3. find those little bastards
$ for i in /*; do echo $i; find $i |wc -l; done
This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.
$ for i in /home/*; do echo $i; find $i |wc -l; done
4. once you found the suspect – just delete the files
$ sudo rm -rf /home/bad_user/directory_with_lots_of_empty_files
You’re done. Check the results with df -i command again. You should see something like this:
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda 2080768 284431 1796337 14% / tmpfs 92187 3 92184 1% /lib/init/rw varrun 92187 38 92149 1% /var/run varlock 92187 4 92183 1% /var/lock udev 92187 4404 87783 5% /dev tmpfs 92187 1 92186 1% /dev/shm
27 Comments
Thanks for sharing, helped me out
Many thanks for this; ran into the mysterious no space left problem and was baffled, but these instructions worked like a charm.
Thx
So helpfull
Great information – This may have resolved several months of scratching our heads trying to figure out why /var showed 52% usage and the disk space was filling up.
Great info.
Rgds,
Chad
Ivan, you’re my hero of the day !
I thought I have to make a new installation of my Ubuntu-Lucid Server, but you helped my out.
My Twonky MediaServer Program creates billons of useless files…
Thx
Guido
Very helpful. We were out of imagination to solve a no disk space problem when the disk actually had enough free space
Just the cure!
My /tmp ran of out inodes… mainly in /tmp/orbit-gdm.
I nuked that folder and went for 100%I to just 1%I usage.
Thanks!
You saved lots of time , its happens to my amazon ec2 frequently .
Hi, does it cover RH linux as well? I tried this syntax for RH linux bu it doesnt work…
Hi agains ,
It is working!!
I’ve just made mistake ..thanks a lot!
Thanks for the info !
very useful !
Greetings
Thanks man…… Thank u very much…..
Some times is difficult to erase the files with the command rm -f -r /tmp/*
In such situation you could you use the next command:
sudo find / -type f -print | xargs — sudo rm -f –
This tip is for all the people that stay in the fire line jaja lol ; )
Just Rocking ! Gr8.
Thanks for sharing, helped me out. Its exactly that problem…
We also had this problem, thanks
Thanks a lot for sharing!
This help us, and prevent from helping so frequently. You’re so kind!!
Great help! my webserver hung the same way. find /opt showed Zend application kept millions of sessions so rm -R zendapp/var/sessions & mkdir & chmod
It’s Very HelpFul…
Thnaks
I also had this problem and it helped me out. Thanks a lot for sharing!
Ivan, is there a simple way to increase the number of available inodes? Because due to the nature of my backups I use them all up after about 3 months. I’d happily sacrifice a few gigs to quadruple the available inodes… if it’s possible, and if it doesn’t cause other issues.
Thanks, this put me into right track.
For all to know: /var/log/* {daily} is a bad, bad thing to have in your logrotate.conf…
Thank you! It worked
Спасибо, Ваня, помогло.
Saved my ass. Thanks
Ivan,
Thanks for guiding me in the right direction. We couldn’t delete the files, but ended up extending the file system to increase the # of inodes.
OMG THANK YOU!
We were having this issue for the past few days, I was like “there is 3GB free, it is not out of space, what is going on!?”.
But one of our apps create a bunch of tiny cache files and apparently that filled up the inodes! :/
4 Trackbacks/Pingbacks
[...] [...]
[...] are running out of inodes. If you are able to identify a value near 100% follow the instructions on Ivan Kuznetsov’s blog. This entry was posted in Linux, Linux Mint, Oracle ADF and tagged debian, JDeveloper, no space [...]
[...] эти файлы. Спасибо Ивану Кузнецову за метод for i in /*; do echo $i; find $i |wc -l; [...]
[...] After some google, found an article where OP suggested to check for inodes: http://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html [...]
Post a Comment