LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 03-08-2020, 09:42 PM   #1
Jack Ou
LQ Newbie
 
Registered: Mar 2020
Posts: 3

Rep: Reputation: Disabled
kernel scheduler latency


Hi,
I am doing an embedded project. Kernel is 4.9.
A simple app runs when whole system is almost idle (no other apps running)

int main(int argc, char **argv)
{
int i = 0;
system("date");
for(i=0; i<6000; i++)
usleep(400);
system("date");
}

You would expect it can finish in 2.4 seconds with a bit overhead. But this one takes 60 seconds for the loop, I measure each usleep(400) takes around 10ms.

Tried below:
1, patch 4.9 from 4.9.1 to 4.9.214, No improvement.
2, with rt patch (choose "Fully Preemptible Kernel RT") or without rt patch (choose "Preemptible Kernel Low-latency Desktop"). No difference.
3, change this small app's real time priority from 0 to 90, No difference.
4, try replacing usleep with nanosleep/select with timeout, No improvement.
5, Change HZ from 100 to 1000, each usleep(400) takes around 1 ms.

Nothing else is competing for CPU. It seems 4.9 kernel adds extra scheduler delay, and it is related to HZ value (100HZ is around 10ms, 1000HZ is around 1ms, it seems it always consumes a full jiffy before being rescheduled). I have run similar kernel options with 4.1 kernel, it takes <3 seconds to finish, that is correct.

If it is not scheduler issue, it may be timer issue: the app is not waken up at 400us, but 10ms or 1ms later depending on HZ value. The hrtimer is enabled in test.

This issue only happens in 4.9, but not in 4.1.
Any thoughts on this ? very much appreciated !

Regards,
Jack.
 
Old 03-10-2020, 06:47 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 22,041

Rep: Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348
I wanted to suggest to try different values to see if that counts (usleep(2), usleep(20),...).
But if kernel 4.1 works you need to check the relevant part of the changelog...
 
Old 03-18-2020, 05:18 PM   #3
Jack Ou
LQ Newbie
 
Registered: Mar 2020
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks pan64 ! It is definitely a kernel issue, other guys from the embedded BSP provider can also re-produce this problem. We are still working on it. Will update once it is solved.
 
Old 04-15-2020, 07:05 PM   #4
Jack Ou
LQ Newbie
 
Registered: Mar 2020
Posts: 3

Original Poster
Rep: Reputation: Disabled
I got a patch from BSP provider. And it solves my problem.

---
arch/arm/kernel/time.c | 2 ++
1 file changed, 2 insertions(+)

diff --git a/arch/arm/kernel/time.c b/arch/arm/kernel/time.c
index 97b22fa..253de76 100644
--- a/arch/arm/kernel/time.c
+++ b/arch/arm/kernel/time.c
@@ -26,6 +26,7 @@
#include <linux/time.h>
#include <linux/timex.h>
#include <linux/timer.h>
+#include <linux/clockchips.h>

#include <asm/mach/arch.h>
#include <asm/mach/time.h>
@@ -121,5 +122,6 @@ void __init time_init(void)
of_clk_init(NULL);
#endif
clocksource_probe();
+ tick_setup_hrtimer_broadcast();
}
}
--
2.9.5
 
1 members found this post helpful.
Old 04-15-2020, 07:36 PM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,153

Rep: Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125
Tx for the update.
 
Old 04-16-2020, 01:26 AM   #6
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 22,041

Rep: Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348Reputation: 7348
yes, thanks. It was - at least - interesting for me. A very special kind of problem which requires very special knowledge.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Kernel build with PREEMPT_RT patch (Ubuntu 16.04, 4.4.1 kernel) : Inputs related to reducing jitter/scheduler latency barz_83_LQ Linux - Kernel 4 12-04-2017 07:37 PM
How to improve scheduler and interrupt latency guy_sarav Linux - Kernel 6 12-15-2016 08:26 PM
Kernel panic, Preemptible Kernel (Low-Latency Desktop) Smartnow Linux - Kernel 3 04-13-2008 02:46 AM
ho to find I/O Scheduler Latency Kotresh Linux - Kernel 1 04-25-2007 08:02 AM
Mozilla 1.6 latency resolving DNS lookup w/ Fedora and Kernel 2.6.1 rtt2 Linux - Software 2 04-30-2006 03:03 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 03:59 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration