No more free space! df 99.9%, du 29% usage

Topper_BG · 02-07-2012, 11:29 AM

ОС Suse Enterprise 10.0
Filesystem ReiserFS
SCSI hardrives 36Gb in HW RAID1
root partition 27Gb
Acting like MTA/MySQL/LDAP etc. server

One morning stopped, not enough space. Missing 19GB (according previous report from logwatch)

Quote:

/ # df
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 28G 27G 1.3G 96% /
tmpfs 1013M 0 1013M 0% /dev/shm
/dev/sda3 70G 47G 24G 67% /DATA
/dev/sdb1 151G 1.6G 149G 2% /backup

Quote:

du -shx /
8.0G /

Quote:

du -sxh /var/
3.7G /var/

Quote:

lsof -a +L1 /
COMMAND PID USER FD TYPE DEVICE SIZE NLINK NODE NAME
mysqld 13712 mysql 6u REG 8,1 0 0 15016 /tmp/ibKpdj7k (deleted)
mysqld 13712 mysql 7u REG 8,1 0 0 15018 /tmp/ibXBeg2u (deleted)
mysqld 13712 mysql 11u REG 8,1 0 0 14979 /tmp/ib6ZuzwG (deleted)

Going trough all subdirs of root with df . && du -shk - nothing so big !
Looking for big files - no, looking for bigest files - nothing.

Umounted all other partitions and checked:

Quote:

du -sh /backup
0 /backup
du -sh /DATA
0 /DATA
du -sh /tmp/
0 /tmp/

Restarted, login in single user mode, fsck - no errors, still missing space.

Please advice guys ?

unSpawn · 02-07-2012, 11:59 AM

Reserved block count? As in 'tune2fs -l /dev/sda1|grep -i block;' standard five per cent of the partitions size. Can be lowered with 'tune2fs -m n' but read 'man tune2fs' first.

Topper_BG · 02-07-2012, 01:04 PM

The filesystem is ReiserFS as described - not a ext2/ext3!

SecretCode · 02-07-2012, 01:09 PM

I wondered about that but does tune2fs work on ReiserFS? [crossposted with OP]

Just to check - you are running du as root, aren't you?

Topper_BG · 02-07-2012, 01:40 PM

Sure like root.

SecretCode · 02-07-2012, 01:51 PM

You've fsck'd I see

Have you tried booting from a live CD so /dev/sda1 is not actually in use? Possibly even mounting it ro?

Topper_BG · 02-07-2012, 01:55 PM

Running in single user, also mounted as RO.
Tomorrow will try with RescueCD, but if I reboot it... will extend it.
But the problem is problem, right ?

unSpawn · 02-07-2012, 11:47 PM

Quote:

Originally Posted by Topper_BG

The filesystem is ReiserFS as described - not a ext2/ext3!

Well call up fs specs with your 'tune2fs' equivalent (if any). I'm sure most Linux file systems reserve inodes.

Topper_BG · 02-08-2012, 12:33 AM

Quote:

tune2fs -l /dev/sda
tune2fs 1.36 (05-Feb-2005)
tune2fs: Bad magic number in super-block while trying to open /dev/sda
Couldn't find valid filesystem superblock.

Already post, that tune2fs is not compatible with ReiserFS

Wikipedia:

Quote:

Some Unix-style file systems such as ReiserFS omit an inode table, but must store equivalent data in order to provide equivalent capabilities. The data may be called stat data, in reference to the stat system call that provides the data to programs.

Topper_BG · 02-08-2012, 12:50 AM

Also, there isn't inodes:

Quote:

df -T -i -x tmpfs
Filesystem Type Inodes IUsed IFree IUse% Mounted on
/dev/sda1 reiserfs 0 0 0 - /
/dev/sda3 reiserfs 0 0 0 - /DATA
/dev/sdb1 reiserfs 0 0 0 - /backup
/dev/sdb2 ext3 19M 151K 19M 1% /mnt/usb2

Topper_BG · 02-08-2012, 10:09 AM

Bump ?

unSpawn · 02-08-2012, 12:28 PM

Quote:

Originally Posted by Topper_BG

Already post, that tune2fs is not compatible with ReiserFS

I said tune2fs equivalent so go use debugreiserfs.

Topper_BG · 02-08-2012, 12:51 PM

This is the results:

Quote:

Filesystem state: consistency is not checked after last mounting

Reiserfs super block in block 16 on 0x801 of format 3.6 with standard journal
Count of blocks on the device: 7323616
Number of bitmaps: 224
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 425459
Root block: 11270
Filesystem is NOT clean
Tree height: 5
Hash function used to sort names: "r5"
Objectid map size 594, max 972
Journal parameters:
Device [0x0]
Magic [0x292b00bd]
Size 8193 blocks (including 1 for journal header) (first block 18)
Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
Blocks reserved by journal: 0
Fs state field: 0x0:
sb_version: 2
inode generation number: 15651833
UUID: 0a9b77a6-ed65-47be-b032-08078232fc12
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN

What can I do, or what to look after ?

SecretCode · 02-08-2012, 01:35 PM

I'd still like to see if anything changes if you examine the disk from a live CD - without even any files open for read.

"Filesystem is NOT clean" presumably just means it's mounted rw. But fsck it from a live CD anyway.

If you have a spare HDD (and plenty of spare time), doing a deep copy of the entire volume to another identically sized and formatted volume might be instructive.

Topper_BG · 02-08-2012, 02:40 PM

Plenty of spare time - good point.
This is a production system, but lucky it is a VM.
So bad hard disk is possible in other case, not in mine.