[SOLVED] Copy to USB stick slows down the more I copy

GPGAgent · 05-09-2022, 06:17 AM

If I copy 500Mb of files, all mp3, files of about 4-10Mb in size it goes at a madrate pace.

If I copy 2Gb of files of a similar nature it starts quick enough but slows down to a snails pace

Okat, it is USB 2.0 so I'm not expecting lightning speed but it is extremely slow.
I've used the USB Stick Formatter and tried FAT32 and NTFS - no difference. To be fair NTFS looked good at first, really fast but after a 1 or 2Gb it slowed down.

I've tried usong Thunar, Right Click, Copy and using cp -r -v in a terminal. No difference in speed.

Doing free shows this

Code:

              total        used        free      shared  buff/cache   available
Mem:       12282524     2339168     1217300       71400     8726056     9486072
Swap:       2097148           0     2097148

And I've included a few screenshots of Thunar progress.

So I was wondering if there might be a cunning way of doing this copy, for example a loop through A to Z (All the directories begin with an Uppercase letter) in a terminal might be quicker.

Or is there any buffer setting I could tweak?

Soadyheid · 05-09-2022, 07:50 AM

I can't think of any technical reason off the top of my head but my first thought is:

"Where are you copying the files FROM?"

If it's the Internet, depending on where in the World the files are, Internet speeds and whether the files are stored in a contiguous format somewhere could impact the copy speed.

If it's local storage... Does copying to a different stick behave differently?

Play Bonny!

GPGAgent · 05-09-2022, 08:54 AM

Quote:

Originally Posted by Soadyheid

I can't think of any technical reason off the top of my head but my first thought is:

"Where are you copying the files FROM?"

If it's the Internet, depending on where in the World the files are, Internet speeds and whether the files are stored in a contiguous format somewhere could impact the copy speed.

If it's local storage... Does copying to a different stick behave differently?

Play Bonny!

It's from a local HDD to a USB 2.0 32GB stick, and using a different stick doesn't make much difference

Jan K. · 05-09-2022, 09:09 AM

And just to rule it out: copying from one local hdd to another runs smooth and fast?

GPGAgent · 05-09-2022, 09:36 AM

Quote:

Originally Posted by Jan K.

And just to rule it out: copying from one local hdd to another runs smooth and fast?

Oh yes, that works just fine.

GPGAgent · 05-09-2022, 10:30 AM

I've just done an iostat on the usb stick, here are the results

Code:

Using cp in a terminal -  iostat every second
=============================================
jo@charlie:~$ sudo iostat -d 1 sdc
Linux 5.4.0-91-generic (charlie) 	09/05/22 	_x86_64_	(16 CPU)

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sdc               4.14         0.59       418.97         0.00       9709    6868187          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc              25.00         0.00      2920.00         0.00          0       2920          0
sdc               3.00         0.00       303.00         0.00          0        303          0
sdc               5.00         0.00         4.50         0.00          0          4          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc              64.00         0.00      7416.50         0.00          0       7416          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               3.00         0.00         4.50         0.00          0          4          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc              31.00         0.00      3462.50         0.00          0       3462          0

and using copy in Thunar

Code:

Using copy in Thunar -  iostat every second
=============================================

jo@charlie:~$ sudo iostat -d 1 sdc
Linux 5.4.0-91-generic (charlie) 	09/05/22 	_x86_64_	(16 CPU)

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sdc               4.15         0.59       420.91         0.00       9709    6971749          0
sdc              12.00         0.00      1331.50         0.00          0       1331          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               2.00         0.00         3.00         0.00          0          3          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               1.00         0.00         0.50         0.00          0          0          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc              15.00         0.00      1800.00         0.00          0       1800          0
sdc              19.00         0.00      2181.00         0.00          0       2181          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               3.00         0.00         3.50         0.00          0          3          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc              33.00         0.00      3926.50         0.00          0       3926          0
sdc               4.17         0.58       421.94         0.00       9709    7063919          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               4.00         0.00       476.50         0.00          0        476          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               3.00         0.00         2.00         0.00          0          2          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               1.00         0.00         0.50         0.00          0          0          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc              13.00         0.00       530.00         0.00          0        530          0
sdc              46.00         0.00      2881.00         0.00          0       2881          0
sdc              46.00         0.00      2908.00         0.00          0       2908          0
sdc              51.00         0.00      3357.00         0.00          0       3357          0
sdc               4.00         0.00       360.50         0.00          0        360          0
sdc              12.00         0.00       619.00         0.00          0        619          0
sdc              44.00         0.00      2899.50         0.00          0       2899          0
sdc              55.00         0.00      4466.50         0.00          0       4466          0
sdc              79.00         0.00      7892.50         0.00          0       7892          0
sdc              11.00         0.00      1320.00         0.00          0       1320          0
sdc              21.00         0.00      2274.00         0.00          0       2274          0
sdc               0.00         0.00         0.00         0.00          0          0          0
sdc               1.00         0.00         0.50         0.00          0          0          0

It looks like Thunar is quicker by these readings, but in elapsed time it isn't, I need a bigger sample for the terminal cp method.

Jan K. · 05-09-2022, 11:17 AM

Tried another usb port?

Could it simply be card buffer being filled up combined with slow-ish usb2 speed?

Your numbers says nothing for me, but a pci card with usb3 ports for your front panel cost next to nothing... could be an option if you fail to gain speed.

Has it always been slow? New kernel? I've noticed numerous times that kernel had bug/quirk fixes re. usb ports and cards...

rclark · 05-09-2022, 11:44 AM

I've seen this. It is simply a buffer gets filled and then must wait for writes to complete. Writes to the USB Flash slows down the copy process. This is normal. I've since moved on to using USB SSD drives (Samsung T5s and T7s) which seem to have a 'bigger' buffer and are normally quite quick. Also everything I use now is USB 3.x compatible if possible.

jailbait · 05-09-2022, 11:53 AM

There is an area of memory set aside as a cache. When you are copying from one device to another each input record is read into the cache and then written from cache to the output device. If the input device is faster than the output device the copy runs at the input device speed but writes from cache to the output device at the output device speed until the cache fills up. Then the copy cannot read an input record until cache space becomes available for another input record. So at that point the reported copy speed drops to the speed of the output device.

Some copy programs report the end of the copy when the last input record is read into the cache. In fact the copy does not end until the cache is completely written to the output device. So is important that you issue a umount command before removing any devices. The umount command makes sure that there are no outstanding records still waiting to be written from the cache before unmounting the device.

As to device speed there are many cheap USB devices that run at less than the rated speed so two supposedly equal devices may run at different speeds.

rclark · 05-09-2022, 12:02 PM

Quote:

So is important that you issue a umount command before removing any devices.

Just to highlight! Important. If the USB drive has a light, you need to wait until it stops flashing (or wait a few seconds if no light) after the umount command is issued (whether from command line or from GUI) before manually removing drive.

GPGAgent · 05-09-2022, 12:13 PM

Thanks folks, I think it's just a rubbish cheap usb 2.0 stick, whether it's a command line cp or using copy with Thunar I'm not sure if it makes any difference.

Here is an iostat every 60 seconds where I used cp

Code:

jon@charlie:~$ sudo iostat -d 60 sdc
Linux 5.4.0-91-generic (charlie) 	09/05/22 	_x86_64_	(16 CPU)

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sdc               5.20         0.50       534.78         0.00       9709   10459080          0
sdc               6.60         0.00       691.04         0.00          0      41462          0
sdc               7.87         0.00       841.96         0.00          0      50517          0
sdc              13.73         0.00      1553.96         0.00          0      93237          0
sdc              13.50         0.00      1518.60         0.00          0      91116          0
sdc              13.15         0.00      1456.36         0.00          0      87381          0
sdc              15.25         0.00      1707.76         0.00          0     102465          0
sdc              16.88         0.00      1912.41         0.00          0     114744          0
sdc               5.62         0.00       559.78         0.00          0      33587          0
sdc              21.00         0.00      1895.37         0.00          0     113722          0
sdc              39.27         0.00      4001.70         0.00          0     240102          0
sdc              15.73         0.00      1646.35         0.00          0      98781          0
sdc              29.48         0.00      3067.38         0.00          0     184042          0
sdc              12.10         0.00      1293.17         0.00          0      77590          0
sdc              16.55         0.00      1792.58         0.00          0     107554          0
sdc               7.70         0.00       797.24         0.00          0      47834          0
sdc              25.88         0.00      2694.83         0.00          0     161689          0
sdc               7.02         0.00       736.12         0.00          0      44167          0
sdc               6.45         0.00       679.97         0.00          0      40798          0
sdc               6.08         0.00       630.57         0.00          0      37834          0
sdc               5.92         0.00       603.92         0.00          0      36235          0
sdc               6.45         0.00       674.84         0.00          0      40490          0
sdc               5.82         0.00       598.87         0.00          0      35932          0
sdc               7.98         0.00       814.88         0.00          0      48893          0
sdc              14.22         0.00      1489.63         0.00          0      89378          0
sdc               5.78         0.00       589.88         0.00          0      35393          0
sdc               6.45         0.00       599.45         0.00          0      35967          0
sdc               9.53         0.00       915.08         0.00          0      54905          0
sdc               2.20         0.00       218.99         0.00          0      13139          0
sdc               0.00         0.00         0.00         0.00          0          0          0

This is all the machine was doing

I'll do a bit more investigation and maybe report back here.

Actually if you DDG linux slow usb copy you'll find this isn't uncommon, but no one seems to have an answer, not sure if Windows behaves the same?

GPGAgent · 05-09-2022, 12:14 PM

Quote:

Originally Posted by rclark

Just to highlight! Important. If the USB drive has a light, you need to wait until it stops flashing (or wait a few seconds if no light) after the umount command is issued (whether from command line or from GUI) before manually removing drive.

Yep I know that, and yes I always unmount usb devices before removing them.

Thanks

teckk · 05-09-2022, 12:37 PM

I think that some of the kernels are responsible for this. I've got 3 little 2GB usb3 canvio basic external hard drives. Around kernel 5.10.something(arch). That little hard drive had bad write speeds that fluctuated all over, took forever to write. I thought maybe the drive was going bad, or the board on the drive. I updated to kernel 5.12.something(arch). Problem went away. Had no problem since, until kernel 5.16.14(arch). Same problem appeared, so I downgraded the kernel to 5.14.7. which I had in the cache, problem went away again. And that is on multiple drives.

When you change kernels and the usb write speed and stability changes, I can't help but think that is the problem.

With a good kernel the write speed is 24-26MBps on USB2, and 110-120MBps on USB3. According to gkrellm.

With the problem kernels the speeds on USB2 fluctuate from 24MBps down to 1.5Mbps.

And that is from SATA hard drive to usb port.

I don' have a fix. I've read about how that works like others have. Something for you to consider.

I was looking into this:
(Dont run this, just a direction of thought from my notes.)

Code:

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

https://www.kernel.org/doc/html/late...sysctl/vm.html
https://sysctl-explorer.net/vm/dirty_background_bytes/
https://stackoverflow.com/questions/...ckground-ratio

ondoho · 05-11-2022, 12:20 AM

I see this is marked SOLVED already - but I have experienced similar things in the past.

Have you tried to use cp/mv instead of GUI to copy - any faster?

Also read what it says in the README here: https://github.com/Feh/nocache

GPGAgent · 05-11-2022, 12:54 PM

Quote:

Originally Posted by ondoho

I see this is marked SOLVED already - but I have experienced similar things in the past.

Have you tried to use cp/mv instead of GUI to copy - any faster?

Also read what it says in the README here: https://github.com/Feh/nocache

I have tried cp and it seems not to slow down as it copies, so I'm sticking with that for now.

The link looks interesting - I may try it out Thanks