Short hard disk accesses like a beating heart

JZL240I-U · 12-04-2016, 11:01 AM

The system has no errors (I know of) and runs as it should.

But I can hear that one hard disk is briefly accessed about 88 times / minute -- sounds like a quietly beating heart. "top" and "ps ax" don't show anything suspicious (for me, that is). I stopped "akonadi" and its helpers. "iostat" gives

Quote:

me@PC:~> iostat
Linux 4.8.10-1-default (PC) 04.12.2016 _x86_64_ (6 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3,39 0,08 1,17 0,83 0,00 94,52

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 11,16 94,89 97,80 377823 389424
sdb 17,70 69,57 162,74 277011 647964
sdc 8,28 350,03 94,74 1393684 377228
sdd 0,03 1,07 0,00 4276 0

me@PC:~>

sdd is not mounted, sdc is "/" on a SSD, sda and sdb are magnetic disks (for details see my signature).

How can I find out what is causing this and how to make it stop or finish faster if it is really needed?

business_kid · 12-05-2016, 04:55 AM

What are your mount options in /etc/fstab? If you mount with something like 'relatime' on all hard disks does that change things? Can you find which disk? Let's say it's sdX; What does

Code:

lsof |grep sdX

show?

Doug G · 12-05-2016, 07:49 PM

iotop?

Jjanel · 12-06-2016, 03:03 AM

Interesting... Let us know! Yup, iotop is #1, from web-search: linux monitor disk activity

JZL240I-U · 12-07-2016, 10:22 AM

Sorry to be back so late.

I had to power down after my first post. Symptoms are gone for the moment, but they'll come back infrequently in my experience. I'll keep all of you advised

.

Quote:

Originally Posted by business_kid

What are your mount options in /etc/fstab? If you mount with something like 'relatime' on all hard disks does that change things? Can you find which disk? Let's say it's sdX; What does

Code:

lsof |grep sdX

show?

This is my fstab:

Code:

me@PC:~> cat /etc/fstab
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 / btrfs defaults 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /boot/grub2/i386-pc btrfs subvol=@/boot/grub2/i386-pc 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /boot/grub2/x86_64-efi btrfs subvol=@/boot/grub2/x86_64-efi 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /opt btrfs subvol=@/opt 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /srv btrfs subvol=@/srv 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /tmp btrfs subvol=@/tmp 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /usr/local btrfs subvol=@/usr/local 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/cache btrfs subvol=@/var/cache 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/crash btrfs subvol=@/var/crash 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/libvirt/images btrfs subvol=@/var/lib/libvirt/images 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/machines btrfs subvol=@/var/lib/machines 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/mailman btrfs subvol=@/var/lib/mailman 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/named btrfs subvol=@/var/lib/named 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/lib/pgsql btrfs subvol=@/var/lib/pgsql 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/log btrfs subvol=@/var/log 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/opt btrfs subvol=@/var/opt 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/spool btrfs subvol=@/var/spool 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /var/tmp btrfs subvol=@/var/tmp 0 0
UUID=57edc8bb-1fdb-487b-9f33-cef529a46f75 swap swap defaults 0 0
UUID=b7f0fdef-e97f-48fa-9a73-3e74373df4c5 swap swap defaults 0 0
UUID=1041c945-1c37-489d-be5f-e7a18dc02d91 /.snapshots btrfs subvol=@/.snapshots 0 0
UUID=72719c81-1f89-4d11-a669-cf6305c9a1ed /home                ext4       defaults              1 2
UUID=2618c23e-dba9-4e58-a69f-75e2fee34286 /backup   ext4    defaults    1   2

Haven't tried relatime yet, since the disks are quiet for now, same with lsof.

JZL240I-U · 12-07-2016, 10:25 AM

@Doug_G & @Jjanel I'll try "iotop" and keep you advised as soon as it is taking off again

.

syg00 · 12-07-2016, 05:12 PM

Whoa !!! - I'm a long-time btrfs user, but that is a seriously ummmm ... interesting config ...

In the past, things like desktop search tools running in the background have been a source of things like this. I also found firefox (and chrome) hitting .cache continually by running some kernel function tracing on ext4 for a while. Even moved .cache into tmpfs for a while to try and isolate the load.
Never noticed anything on btrfs, so never attempted any tracing there. Will add it to my "to-do" list.

Soadyheid · 12-07-2016, 07:51 PM

Quote:

I can hear that one hard disk is briefly accessed about 88 times / minute -- sounds like a quietly beating heart.

Is this a new disk? I mean, does it do this just after formatting and having filesystem(s)generated on it or a new partition?

I recently built a replacement system (Mint 18) and noticed that after the installation you could hear a distinct "tick", "tick" from the disk for quite a while. It continued for a couple of days (My system isn't on permanently so maybe a couple of hours per session?), it finally stopped and hasn't done it since.

My thoughts at the time were that it was due to having to create some sort of index file in the background, maybe to support journaling?

Anyway, that's my

Play Bonny!

JZL240I-U · 12-08-2016, 01:49 AM

Quote:

Originally Posted by syg00

Whoa !!! - I'm a long-time btrfs user, but that is a seriously ummmm ... interesting config ...

.

Well, this is openSUSE's original configuration written during installation. I didn't change anything. If you know of improvements I'd be delighted if you have a mind to share

.

Quote:

Originally Posted by syg00

In the past, things like desktop search tools running in the background have been a source of things like this. ...
Never noticed anything on btrfs, so never attempted any tracing there. Will add it to my "to-do" list.

Yes, that's why I killed akonadi. But I am quite sure the noise not emanating from the SSD where "/" on btrfs resides. It is definitely the movement of the heads of a spinning magnetic hard disk, most probably /home with ext4, possibly /backup - (which I doubt) also ext4. There should have been no swapping, I did nothing memory intensive and there is 12 GB RAM.

JZL240I-U · 12-08-2016, 01:58 AM

Quote:

Originally Posted by Soadyheid

Is this a new disk? I mean, does it do this just after formatting and having filesystem(s)generated on it or a new partition?...

No, this ("/") is my tumbleweed installation of quite some "age" -- about half a year possibly an entire year but frequently updated. /home is ancient -- 4 years? I really dunno.

business_kid · 12-08-2016, 04:24 AM

If you're still hunting for the disk, try this (as root)on each mechanical disk

Code:

lsof |grep sda

That gives me an output like this

Code:

bash-4.3$ sudo lsof |grep sda
jbd2/sda3    61       root  cwd       DIR                8,3      4096          2 /
jbd2/sda3    61       root  rtd       DIR                8,3      4096          2 /
jbd2/sda3    61       root  txt   unknown                                         /proc/61/exe
jbd2/sda5   384       root  cwd       DIR                8,3      4096          2 /
jbd2/sda5   384       root  rtd       DIR                8,3      4096          2 /
jbd2/sda5   384       root  txt   unknown                                         /proc/384/exe
jbd2/sda7   386       root  cwd       DIR                8,3      4096          2 /
jbd2/sda7   386       root  rtd       DIR                8,3      4096          2 /
jbd2/sda7   386       root  txt   unknown                                         /proc/386/exe

Then, for each of the /proc/ sub directories it mentions, run this command

Code:

bash-4.3$ sudo cat /proc/61/io |grep write
write_bytes: 17813504
cancelled_write_bytes: 0
bash-4.3$ sudo cat /proc/384/io |grep write
write_bytes: 240218112
cancelled_write_bytes: 0
bash-4.3$ sudo cat /proc/386/io |grep write
write_bytes: 0
cancelled_write_bytes: 0

In my case,
sda3=/proc/61=/,
sda5=/proc/384=/home, &
sda7=/proc/386=/mnt/virtual, where I have a few virtual machines to run odd bits of software. You will at least find what has the traffic.

JZL240I-U · 12-08-2016, 04:35 AM

Thanks, business_kid I'll do that when it is misbehaving again (right now I'm not home). I will report then.

syg00 · 12-09-2016, 09:47 PM

Quote:

Originally Posted by JZL240I-U

If you know of improvements I'd be delighted if you have a mind to share

.

Nah, if it ships like that, you might as well leave it, Been many years since I looked at openSUSE. Just not how I would build it.

Quote:

... most probably /home with ext4, possibly /backup - (which I doubt) also ext4.

This is (was) a really difficult issue to resolve - the backward mapping of a bunch of I/O sectors to file name(s). And the frequency distribution. As it happens there are some really useful additions to the kernel (f)trace coming in 4.9 that really help with this.
Much more easy to set up trace from userspace - hopefully some userland tools to expose the results will be in the pipeline soon as well.

Doug G · 12-09-2016, 10:28 PM

It's possible you just had a temp file that gets updated regularly, and it happened to get allocated to a spot on the disk that caused a long head seek each time it was accessed, then returning to the idle location. The actuator seeking a long distance would cause the kind of sound you described.

Then later, the temp file got reallocated to a location that doesn't need a lengthy seek, and the noise went away.

If this were the case, you probably wouldn't see anything significant in iotop or any other disk activity reporting.

kilgoretrout · 12-09-2016, 10:52 PM

Just to eliminate the possibility of a hardware problem, I recommend running the hard drive manufacturer's diagnostic utility in "thorough" mode. Download the Seatools for DOS here:

http://www.seagate.com/support/downloads/seatools/

It's a bootable iso that you burn to a cd-r just like any linux installation cd. Boot with your cd-r and run the hard drive diagnostics on both of your Seagate hard drives from there(see Seagate SeaTools documentation).