Slow HDD write performance when decompressing files

Hammett · 06-04-2022, 05:27 AM

Hi all,

I have a very strange problem I am unable to identify what it the cause: HDD is very slow at writing when decompressing large files. I noticed this problem when updating chromium, that it would take an extreme amount of time to decompress the tar file into the working directory.

Manually decompressing the tar file results in the same performance, and I also checked with another HDD drive I have installed and the issue is the same.

Here's the output of iostat when decompressing the file:

Code:

Average:          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
Average:       162.88    303.3k      2.5M      0.0k     17.8k     23.26    141.58     75.2% sdc

There is a quite significant await time. In fact, when I cancel the decompression, the disk is at 100% util for the next 10-15min until it finishes all the writes it needs to make. The await time while decompressing the image is around 200!

I checked performance with hdparm and it looks OK

Code:

/dev/sdc1:
 Timing cached reads:   29606 MB in  2.00 seconds = 14825.09 MB/sec
 Timing buffered disk reads: 584 MB in  3.01 seconds = 194.29 MB/sec

I also checked with dd:

Code:

$ dd if=/dev/zero of=/mnt/Multimedia/src/dddump bs=1M count=3000
3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB, 2.9 GiB) copied, 19.9332 s, 158 MB/s

The weirdest thing is that I have an old kernel image around and when I boot that image, this problem with the write speeds is not happening.

For comparison, here's the iostat output of decompressing the chromium tar file

Code:

Average:          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
Average:       513.58      9.6k     80.2M      0.0k    159.9k     20.19     39.17     68.2% sdc

The await time and the write throughput is much higher and inline with what one could expect of and HDD performance.

This leads me to believe it might be a kernel config that is messing with the HDDs. I do not have the sources around and the kernel is complied without the IKCONFIG, so I am unable to retrieve the config for this image.

Kernels I have tried with the issue:
5.15.41
5.17.1
5.17.5
5.18.1-r2

Kernel without the issue
5.17.1-zen1 (from zen-sources)

I am completely lost at what could cause this issue with the disks. All the kernels have the bfq IO scheduler and I can't think of any change I have made to newer kernels that could explain this poor perfomance. Any tips on where to look in the kernel would be greatly appretiated.

jailbait · 06-04-2022, 02:59 PM

Quote:

Originally Posted by Hammett

This leads me to believe it might be a kernel config that is messing with the HDDs. I do not have the sources around and the kernel is complied without the IKCONFIG, so I am unable to retrieve the config for this image.

Kernels I have tried with the issue:
5.15.41
5.17.1
5.17.5
5.18.1-r2

Kernel without the issue
5.17.1-zen1 (from zen-sources)

Another suspect would be the gzip program. Are you using different versions of gzip in your various test runs?

Hammett · 06-05-2022, 03:14 AM

Quote:

Originally Posted by jailbait

Another suspect would be the gzip program. Are you using different versions of gzip in your various test runs?

Hi jailbait. No, just switching kernels, the utils are the same.

I have found a bugreport on the kernel bugzilla about the same issue as mine, but for XFS filesystem. https://bugzilla.kernel.org/show_bug.cgi?id=216007

In the bugreport they have posted a patch for kernel 5.18.1, but it is not working in my case (EXT4), but seems to be solving the issue for XFS. I have added myself to that report to see if they can help.