HI all,

I have ported a recent kernel (4.4.x branch) to one of our custom ARM ( imx53-based ) boards.
This board has been running kernel 2.6.35.x for some years, and at this time we decided we needed newer kernel ( for wifi operation). so far so good

When running this a NEW ( recent ) kernel 4.4.61 ( and/or 4.4.73) I see a HIGH CPU load compared to the older 2.6.35 kernel ( when running the same applications)

Comparing systems is difficult I know.. but... What has been changed ? only the KERNEL ( same hardware, same gcc-compiler, same C-lib, same user-space applications) only the KERNEL has been upgraded (nothing else) , both boards I compare are identical .. running at the same clock-speed... same amount of ram / storage. ! 100% sure on this ! .


When comparing 2.6.35.x to 4.4.x kernels I see that a LARGE fraction of the CPU power is consumed into system ( looking at TOP)


A) TOP compared :

NEW-KERNEL 4.4.61

Mem: 190652K used, 830200K free, 0K shrd, 44K buff, 44K cached
CPU: 4% usr 35% sys 0% nic 60% idle 0% io 0% irq 0% sirq
Load average: 2.12 1.91 1.23 2/112 4019


OLD-KERNEL 2.6.35.xx

Mem: 107156K used, 894284K free, 0K shrd, 0K buff, 42268K cached
CPU: 1% usr 9% sys 0% nic 89% idle 0% io 0% irq 0% sirq
Load average: 7.39 6.60 3.89 1/93 3865


Means : 9% system vs 35% system ( 100% sure running same userspace app's )


When comparing the number of interrupts on the old versus the new kernels.. I get approx.the same 'number' of interrupts on both kernels.


cool smiley The New kernel seems to be very slow for ‘kernel’-calls … For example 10 million clock_gettime(CLOCK_MONOTONIC, &tp) calls on the
2.6.35 kernel takes 98 seconds
4.4.61 kernel takes 280 seconds.

When for example doing 10 million calculations ( add or multiply) I do have the same ‘calculation time’ on both kernels.


C) For the new kernel there are a LOT of ‘alignment’ fixes… ( may this is a reason of being slow to)


NEW-KERNEL 4.4.61

# cat /proc/cpu/alignment
User: 9387159
System: 4026787 (inet_gro_receive+0x1b4/0x424)
Skipped: 0
Half: 0
Word: 0
DWord: 6
Multi: 13413940
User faults: 2 (fixup)


OLD-KERNEL 2.6.35.xx
# cat /proc/cpu/alignment
User: 882081
System: 0
Skipped: 0
Half: 0
Word: 0
DWord: 0
Multi: 882081
User faults: 2 (fixup)


Any one able to pinpoint me in a direction ? why the new 4.4.61 kernel consumes so much system time ??????

Eve been searching into the kernel code .. for weeks now.. hoping to find any possible explanation why the NEW 4.4.x kernels take so much CPU power. ( configuration of both kernel is matching as good as possible, same scheduler-config, same timer config.. etc etc.)

But I cant not find / explain why so much cpu power is consumed by the new 4.4.x kernel itself.

Any hints ?

Is this of any interest ?. Particularly the end of the thread - and the date.
4.4 was January 2016-ish.

Hi Syg00,


Thanks for the Hint.. I did patch ( as described in your link) ... which leads to less alignment error ... ( alignment errors occasionally count... 2 a 3 times second, which is quite reasonable , before it was counting 200 alignment errors a sec.)

BUT .. sill no improvement in CPU load ...
at this time I have compared... 2 empty systems... ( running NOTHING) except HTOP / top.



HTOP: on the older 2.6.35 kernel ( 1% cpu load )

HTOP: on the new 4.4.x kernel ( 31 % cpu load)



TOP: OLD-2.6.35 - KERNEL
Mem: 96320K used, 905120K free, 0K shrd, 0K buff, 46156K cached
CPU: 0% usr 1% sys 0% nic 98% idle 0% io 0% irq 0% sirq
Load average: 1.48 1.36 1.20 1/69 23003


TOP NEW 4.4.x kernel
Mem: 177244K used, 843632K free, 0K shrd, 44K buff, 44K cached
CPU: 0% usr 28% sys 0% nic 71% idle 0% io 0% irq 0% sirq
Load average: 1.36 1.23 1.20 1/86 21964




I'm searching for this for some ( +5) weeks .. but I can not get the Clue why this behavior is changed that dramatically.


Best Regards
Noel

You moved two major-versions ahead in one swoop. The CPU-handling, and the reporting of the same, can be expected to be very substantially different between the two. You are, in this case, comparing apples to oranges. "Yes, it's still Linux. No, it's not."

Hi sundialsvcs,


I know that I moved couple of versions ( 2.6.35 -> 4.4.x is quite a big step there is no discussion about that, we all know that, and want to understand that )

BUT I did not expect the same 'user application' (for example TOP / HTOP , or my custom application) to use that MUCH CPU ( running on 100% the same hardware configuration ). { did not recompile any application, the only thing that has been upgraded is the kernel nothing else , same compiler, same Clib, etc..}


Reaction speed/time of my custom app is very bad.. when I compare .. 2.6.35 to 4.4.x ( even with the same (aprox the same) kernel configuration)


I do not compare apples with oranges, I do compare the reaction speed of both kernels on the same user application, and I can not understand that an 'empty/clean' Linux system (custom app being removed) just running TOP (or HTOP) on a ARM CPU (running at 1GHZ) is causing 31% of cpu load on a 4.4 l kernel and 1% on 2.6.35-kernel.
{ I show you here the results on top/htop because everyone know this, but the 'real' issue I'm fighting with is the fact that all is responding very badly on those 4.4.x kernels ! }

I don't know if you did take a good look at the numbers I showed you in the reporting before but, keeping a 1GHZ CPU busy for more then 31%, will require some 'heavy' looping in the kernel ( I did not recompile TOP / HTOP ) which leads me to the fact that the recent kernels are eating .. LOTS of CPU power... ( FYI: the system was nearly empty , just init, shell, and TOP/HTOP, and sshd)


whatever you seem to do on a 4.4.x kernel .. it seems to be using that much cpu very quickly, and due to this high cpu load all applications ( even bare basic system application like top/htop / watch .. etc ... ) are NOT running smooth !




Thanks and best regards
Noel

( sorry hit post 2 times , deleted here)

There could be, for example, more daemons running.

More Daemons ? No there are not.

Both system I compare have been stripped down to a 'bare' minimum required ( When I compare both.)

( I could send you a screenshot .. of both if you want.. 100% the same processes are running... same builds, same versions, build with same compiler, running on the same HARDWARE , the SINGLE change is the kernel-version 2.6.35 vs 4.4.x ! )

I have exactly the same number of TASKS/processes (in htop) and I have exactly the same process layout (when comparing both).

On the 4.4.x kernel I see HIGH CPU load, 'system load' ( what means kernel load) on the 2.6.35 kernel.. I have way less cpu load. ( Running exactly the same number of applications)

On the 4.4.x system I see a load up to +31% (system) on HTOP/TOP while I see for the 2.6.35 kernel... a load of 1% ... with the SAME processes running ! ( you can count on that )

Spending 31% load on a 1GHZ CPU ... with a system that is nearly 'empty' while on the same hardware and a 2.6.35. kernel ( 100% same setup , same process tree ) I nearly have 1% CPU load.


And the load is that different on both kernels you can 'feel' this ( you can see this in almost every action you take, for example on display drawing, reaction on key-input.. etc etc...)

(since I can not include pictures here I've include some 'live' textual info)


LOOK at the results below ....


====================================4.4.x=========================

Kernel 4.4.x

uname -a
Linux DU11 4.4.73 #3 PREEMPT Tue Jul 4 09:06:46

#top
Mem: 176076K used, 844800K free, 0K shrd, 44K buff, 44K cached
CPU: 0% usr 31% sys 0% nic 68% idle 0% io 0% irq 0% sirq
Load average: 1.31 1.41 1.43 1/85 12125
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
12125 21246 root R 1184 0% 23% top
12142 1 root S 3728 0% 0% /usr/sbin/mosquitto -c /etc/mosquitto/

# htop
CPU[##********** 40.0%] Tasks: 32, 0 thr; 1 running
Mem[|||||** 142/978MB] Load average: 7.20 7.18 7.12
Swp[ 0/0MB] Uptime: 16:22:48

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
30886 root 20 0 1064 576 432 R 33.0 0.1 0:00.06 htop

====================================2.6.35=========================


Kernel 2.6.35
uname -a
Linux OLD 2.6.35.3 #1 PREEMPT Tue Jun 14 13:45:24

#top
Mem: 97064K used, 904376K free, 0K shrd, 0K buff, 46160K cached
CPU: 0% usr 0% sys 0% nic 99% idle 0% io 0% irq 0% sirq
Load average: 1.41 1.27 1.26 2/70 18808
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
18696 30503 root R 1176 0% 0% top
842 1 root S 3928 0% 0% /usr/sbin/mosquitto -c /etc/mosquitto/



#htop
CPU[#* 1.5%] Tasks: 32, 0 thr; 1 running
Mem[||||** 126/996MB] Load average: 1.21 1.23 1.30
Swp[ 0/0MB] Uptime: 2 days, 00:05:53

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
13431 root 20 0 1076 772 628 R 1.0 0.1 0:00.66 htop

It's probably a cpu kernel config issue. One wrong move and you can really slow down a system. Did you generate a new kernel config based on the new kernel?

Yes, I did generate new config .

( i've been busy for MULTIPLE weeks on this, i've spend a huge amount of time on this, impossible to explain all the things I did )

But yes also did generate new configs ( even used default configs imx6 based , to get started... )



Regards
Noel