high memory and cpu usage when copying large amounts, KSWAPD0

ouigkmy · 07-28-2011, 09:23 AM

Hi!

I am copying large amounts of data from one hard drive to the other.

After some time, "free -m" tells me all my memory (2GB) gets filled up. A process named "KSWAPD0" appears that uses a lot of cpu power.

I've searched for this process; apparently it is related to swap space use; but free does not say that the swap is used (I have 4GB swap space).

Even long after the copying is done, the memory remains in use, KSWAPD0 is running, and the whole system is slow.

Could anyone please tell me what is going on? Thanks!

(I use the JFS filesystem on a disk with LVM partitions)

mesiol · 07-28-2011, 11:39 AM

memory usage is done to buffer files into memory, so that a write to the target disk is not done for each single file.

kswapd is kernel swap daemon, a part of the virtual memory usage. This has only partially to do with swap space. kswapd organizes which memory pages to swap in/out or to disk. High usage of kswapd was in older kernel version a bug, but should be fixed.

Could you provide some more information?
Did you use

Code:

iostat
vmstat
sar

to check what your system is really doing on I/O and memory? What disks are you using? PATA/SATA/SCSI? Are you really copying between two disks or between two partitions on the same disk?

sundialsvcs · 07-29-2011, 08:13 AM

It sounds like the program that you're using is slurping the entire file into a memory-buffer, then writing it all out again.

The problem with this very-common (and, very-misguided...) strategy is of course that "memory" is virtual, that is to say, "a disk file."

Most operating systems, no doubt including Linux, provide a standard "file copy" primitive which is designed to be very efficient. Clearly, this is not how the program-in-question is now doing things. Clearly also, the correct solution is to rewrite or to replace it, your choice.

ouigkmy · 07-29-2011, 04:37 PM

top:

Code:

top op - 00:41:10 up  1:58,  4 users,  load average: 1.00, 0.98, 0.72
Tasks: 111 total,   3 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.7%us, 28.0%sy,  0.0%ni, 71.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1900844k total,  1622024k used,   278820k free,     4236k buffers
Swap:  4194300k total,     3752k used,  4190548k free,   935284k cached

 28 root      20   0     0    0    0 R  100  0.0  31:30.90 kswapd0

free -m:

Code:

             total       used       free     shared    buffers     cached
Mem:          1856       1717        138          0          3        831
-/+ buffers/cache:        882        973
Swap:         4095          3       4092

vmstat:

Code:

pprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0   3760 223136   4040 855424    0    0    78    79  321 1418  3  6 90  1

iostat:

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.58    0.06    6.56    0.93    0.00   89.87

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              28.05       310.36       313.13    2052386    2070673
dm-0              0.37         1.47         0.02       9753        101
dm-1              3.67        61.23        21.05     404928     139180
dm-2              0.18         0.14         0.57        896       3796
dm-3              1.39         5.09         2.46      33648      16236
dm-4              7.27         7.58        24.03      50136     158936
dm-5              2.00         4.07         3.94      26900      26036
dm-6              2.98         6.05         5.88      39984      38868
dm-7             39.34       224.59       255.19    1485200    1687520

Sar gives me an error; Cannot open /var/log/sa/sa29: No such file or directory.

Yes, I am copying from one hard drive to the other. I have SATA drives. I have however seen that the problem also persists if I do large operations on the target drive only.

As to the strange program I'm using..... it is just "cp" !

ps im on arch linux 86_64 using GPT partition tables and LVM on a Samsung HD204UI

ouigkmy · 07-31-2011, 06:08 AM

This seems to be a bug in the Linux kernel. I found these threads:

https://bugzilla.kernel.org/show_bug.cgi?id=12309

https://bbs.archlinux.org/viewtopic.php?id=93960

http://forums.gentoo.org/viewtopic-t-793263.html

Gonna try some other kernel builds. Wish me luck!

sundialsvcs · 08-02-2011, 06:32 AM

cp is an easy program, but not always the most efficient one...

http://www.cs.umbc.edu/~tsimo1/page1/page0/page0.html

for example.

Having said that, you also need to be wary of "lies, damned lies, and statistics." It is possible that what you are seeing in the stats for kswapd is, simply, the I/O that is taking place.