Why is my root filesystem full, I can't find anything that taking up the filesystem space
Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
And when i do 'du / | sort -rn | head', nothing is taking the space that big
Code:
[root@prodbdaast1dr usr]# du / | sort -rn | head
du: cannot access `/proc/49888/task/49888/fd/4': No such file or directory
du: cannot access `/proc/49888/task/49888/fdinfo/4': No such file or directory
du: cannot access `/proc/49888/fd/4': No such file or directory
du: cannot access `/proc/49888/fdinfo/4': No such file or directory
11426196 /
4117232 /usr
3064312 /home
3064052 /home/beehive
2504112 /opt
2140156 /usr/share
2005040 /opt/cloudera
2004880 /opt/cloudera/parcels
2004872 /opt/cloudera/parcels/CDH-5.7.5-1.cdh5.7.5.p0.3
1396272 /home/beehive/toolchain
How to know what is taking up my root filesystem space
You have probably data hidden by mounting a different filesystem at the directory (/home, /usr, /var). Mount your / again with the "--bind"-option somewhere else and analyze it from there:
Code:
mkdir /tmp/rootbind
mount --bind / /tmp/rootbind
du -d 1 -h /tmp/rootbind
And then expand that directory tree down the rabbit hole until you find unexpected hogs of space.
That will only help if the hogs happen to be visible on the file system. As earlier mentioned, that may not be the case. There are several things that can result in hidden hogs (aside from camo paint spilled into the hog pen ) such as mounting over the hog folder. Also, the issue may not be space as such, there could be inode issues (although I do not suspect those in this case).
We need more data to determine which cases to pursue, so we await the feedback from the OP.
I'm not an admin, but I found something online that can cause a filesystem to fill up too.
Sometimes a process will continue to write to an unlinked file and fill up disk space quickly. Below is an excerpt from the website explaining it better. Click on the link below to learn how to find them. Again, this is just a suggestion for you to try and/or investigate.
Quote:
a. Finding an Unlinked Open File
=================================
A pesky variant of a file that is filling a file system is an
unlinked file to which some process is still writing. When a
process opens a file and then unlinks it, the file's resources
remain in use by the process, but the file's directory entries
are removed. Hence, even when you know the directory where the
file once resided, you can't detect it with ls.
This can be an administrative problem when the unlinked file is
large, and the process that holds it open continues to write to
it. Only when the process closes the file will its resources,
particularly disk space, be released.
Lsof can help you find unlinked files on local disks. It has an
option, +L, that will list the link counts of open files. That
helps because an unlinked file on a local disk has a zero link
count. Note: this is NOT true for NFS files, accessed from a
remote server.
You could use the option to list all files and look for a zero
link count in the NLINK column -- e.g.,
There is a limited number of possible entries in the directory of root.
It is often the directory that fills up not the disk space.
Just create folders and move the entries for a lot of files.
There is a limited number of possible entries in the directory of root.
That is not true. It was true back in the days of FAT16, FAT12, and earlier. In FAT32 (and later) and all Linux filesystems, the root directory is an ordinary directory and can grow to a size limited only by the available disk space.
[root@prodbdaast1dr ~]# sudo du -smx /* | sort -n
du: cannot access `/proc/49335/task/49335/fd/4': No such file or directory
du: cannot access `/proc/49335/task/49335/fdinfo/4': No such file or directory
du: cannot access `/proc/49335/fd/4': No such file or directory
du: cannot access `/proc/49335/fdinfo/4': No such file or directory
0 /misc
0 /net
0 /primary
0 /proc
0 /sys
1 /cgroup
1 /data
1 /dev
1 /lost+found
1 /media
1 /mnt
1 /selinux
1 /srv
3 /tmp
8 /bin
15 /sbin
27 /lib64
32 /boot
40 /etc
161 /lib
395 /var
457 /root
2446 /opt
2993 /home
4021 /usr
sudo mount
Code:
[root@prodbdaast1dr ~]# sudo mount
/dev/mapper/vg_prodbdaast1dr-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1 on /boot type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_home on /home type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_opt on /opt type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_usr on /usr type ext4 (rw)
/dev/mapper/vg_prodbdaast1dr-lv_var on /var type ext4 (rw)
/dev/sda1 on /data/01 type ext4 (rw,noatime)
/dev/sdb1 on /data/02 type ext4 (rw,noatime)
/dev/sdc1 on /data/03 type ext4 (rw,noatime)
/dev/sdd1 on /data/04 type ext4 (rw,noatime)
/dev/sde1 on /data/05 type ext4 (rw,noatime)
/dev/sdf1 on /data/06 type ext4 (rw,noatime)
/dev/sdg1 on /data/07 type ext4 (rw,noatime)
/dev/sdh1 on /data/08 type ext4 (rw,noatime)
/dev/sdi1 on /data/09 type ext4 (rw,noatime)
/dev/sdj1 on /data/10 type ext4 (rw,noatime)
/dev/sdk1 on /data/11 type ext4 (rw,noatime)
/dev/sdl1 on /data/12 type ext4 (rw,noatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
cm_processes on /var/run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/blkio type cgroup (rw,blkio)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpuacct type cgroup (rw,cpuacct)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpu type cgroup (rw,cpu)
cm_cgroups on /var/run/cloudera-scm-agent/cgroups/memory type cgroup (rw,memory)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
[root@prodbdaast1dr ~]# du -h --max-depth=1 /
7.7M /bin
15M /sbin
0 /net
12K /.dbus
2.4G /opt
4.0K /selinux
4.0G /usr
4.0K /srv
392K /dev
569M /data
27M /lib64
4.0K /mnt
8.0K /cgroup
161M /lib
3.0G /home
du: cannot access `/proc/5775/task/6338/fd/251': No such file or directory
du: cannot access `/proc/50951/task/50951/fd/4': No such file or directory
du: cannot access `/proc/50951/task/50951/fdinfo/4': No such file or directory
du: cannot access `/proc/50951/fd/4': No such file or directory
du: cannot access `/proc/50951/fdinfo/4': No such file or directory
0 /proc
4.0K /media
2.9M /tmp
0 /misc
32M /boot
404M /var
457M /root
40M /etc
16K /lost+found
0 /sys
11G /
I'm not sure if this is the cause, but it doesn't hurt to investigate. Commands like ls, du, and find willl not list unlinked files because they've been deleted, but some process is still writing to it or holding it open. Once you find and kill the process, it will release the file and restore the disk space and other sources.
Look for a zero in the NLINK column and deleted at the end of line. Also look under the SIZE/OFF column. It will be in bytes.
You appear to have 104GB hiding under the /data/12 mount point directory. You can look in /tmp/rootbind/data/12 and its subdirectories and see if there is anything there you need to keep, and either move it elsewhere or delete it.
It's a bit odd that the filesystem you have mounted on /data/12 only appears to be using about 69MB. How 104GB got stored in that directory while that filesystem was not mounted is a bit of a mystery.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.