Hi friends, I encountered a disk io speed problem. First of all, the basic system information is:

1) Dell Precision T3500, 8G MEM, 2x3T Seagate SATA HDD.
2) OS: Centos 5.7 (also tried RHEL 5.7/6, Centos 5.3/5.4/6, all the same problem).

Problem:
===================

Each time after reboot, I begin to test the disk speed. At the beginning all is fine. However after about 30 minutes, the disk IO speed drops:

[root]# hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 25924 MB in 2.00 seconds = 12988.70 MB/sec
Timing buffered disk reads: 464 MB in 3.02 seconds = 153.86 MB/sec
[root]# hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 27048 MB in 2.00 seconds = 13553.33 MB/sec
Timing buffered disk reads: 408 MB in 3.01 seconds = 135.65 MB/sec
[root]# hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 27224 MB in 2.00 seconds = 13642.57 MB/sec
Timing buffered disk reads: 318 MB in 3.00 seconds = 105.90 MB/sec

[root]# hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 3468 MB in 2.00 seconds = 1733.12 MB/sec
Timing buffered disk reads: 286 MB in 3.01 seconds = 94.99 MB/sec

============
Anyone knows what happens? Thanks a lot.

Do you have anything in the logs? Look into /var/log/messages. Also, are you sure you do not have any other process that is heavily using the disk?

Thanks.

I did not run any heavy IO, but the hdparm command. After about 20 times, the io speed suddenly drops and never returns to normal.

Also I did not find any special message from /var/log/messages.

personally I don't think you can do this by simply timing a command. what's the system doing while the command is actually running? even more importantly what's happening as it transitions from fast to slow?

the only way to get the bottom of this sort of thing is by continuous monitoring and watching everything else going on in the system with a tool like collectl. it samples everything every 10 seconds and if you know about the time the speed drops play back the collectl data around that time period and see what's different.

I've never user hdparm so can't comment. It doesn't sound like a tool that stresses the disks over long periods of time. personally I do long reads/writes on the order of minutes, if I want accurate measurements. Since this behavior occurs in ~1/2 hour, it sounds relatively easy to reproduce. I use 'dt' to generate a continuous load by continuously writing a large file in a loop and running collectl both as a service as well as in a separate window to watch what the system is doing in real time.

-mark

What kind of hard disk do you have? Is it 4k sectors one?! Have you this problem after heavy IO or just 30 minutes after reboot or after wake up from suspend?
Do you use download managers like dta (Downthemall)?

Can you reproduce the situation when using
You should always use the --direct option when accessing devices, because it avoids the page cache.

You can also see if
affects your timings. (It should not affect --direct timings though, because that should completely bypass the caches anyway.) This command is safe to run at any time; it just flushes all caches and unwritten changes to disk, then clears the caches. It will never discard unwritten changes.

You should make sure you have smartmontools installed, and check the SMART data:
Running a self-test ( -t short or -t long ), waiting (without rebooting -- otherwise the disk can be used normally) usually quite some time (you can use smartctl -a device to see the progress), and then an offline test ( -t offline ) to update all offline attributes, and rerunning smartctl -a device will tell you more about the physical state of your storage device. Uh, assuming it does have SMART support.

Finally, you could install and run iotop before testing, to really see if you have background processes (like indexing or other housekeeping stuff, that most Linux distributions do use) accessing storage, just before running hdparm --direct -t. (Personally, I find hdparm -T to be quite worthless in practice.)

Hi all, I've been searching for some days and found this post. My problem is similar to this: very low speed on sata disk. Tha problem is intermitent, some times hdparm reports around 600 MB/s and then drops to as low a 100 kB/s. Tried forcing some parameters of libata (yet unable to set 1.5G link speed...).
Previously I was using a CF card with IDE adapter instead of the HD and worked quite fast, now moved to sata HD and got this problem.
My setup: Intel D945GCLF mini ITX board, Atom N270, 1GB DDR2 667 DRAM, WD Scorpio Blue 500GB SATA disk, Slackware 13.37 fresh install. Disabled hd read cache with hdparm -W0 and set boottime kernel parameters: "libate.force=1:1.5g,noncq,norst noapic acpi=off". This seems to be te "less worse" situation.
Finally sorry for my bad english, my primary language is spanish.

Thanks in advance!