unix sysadmin archives
Donation will make us pay more time on the project:
          

Thursday 9 June 2011

Your filesystem is full but you cant find anything that is consuming space

Scenario: Your filesystem is full, but du -k /filesystem says it is not eating space that much.

Most of the time when we do some house keeping on our filesystem, we neglect to check whether the file we are deleting is still in use. This leaves some of our resources occupied by the system. In actual these files no longer exist on the real filesystem.
But the space will not be released until the process closes it, or in many cases until the process dies.

So, here is a one liner that finds files that have been deleted but some applications is still holding them.

find /proc/*/fd -type f -links 0 -exec ls -la {} \;

It will give you a list similar to this:

/proc/8/fd/5

The "8" is the process ID.

The "5" is the file descriptor.

Btw, a file descriptor is a handle created by a process when a file is opened. A new descriptor is created each time the file is opened. It is associated with a file object which includes information such as the mode in which the file was opened and the offset pointer where the next operation will begin.  File descriptors are retired when the file is closed or the process terminates. If you want to dig in deeper why this file is still open the file descriptor is a point to start.

Getting back to out topic, Knowing the process ID, you can get a full listing of every process running which has PID equals 8.
Then of course you can trace the process owner and politely ask to them restart their applications.

Now you are on the hunt! Good luck!

No comments:

Post a Comment