Thanks a lot, Petr and Greg.
I saw the patches and just started to prepare the test on devices using the 4.14.y tree. will update the test result later.
@Greg Kroah-Hartman Sorry for not providing the details in the public thread. I can sync w/ you offline.
Thanks, Yi Fan
On Tue, Nov 9, 2021 at 7:28 AM Petr Mladek pmladek@suse.com wrote:
On Tue 2021-11-09 07:27:35, Greg KH wrote:
On Mon, Nov 08, 2021 at 11:17:07AM -0800, Yi Fan wrote:
On Mon, Nov 8, 2021 at 12:00 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Nov 04, 2021 at 12:40:32PM -0700, Yi Fan wrote:
Reply inline.
On Thu, Nov 4, 2021 at 11:56 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Nov 04, 2021 at 11:14:55AM -0700, Yi Fan wrote: > Resend the email using plain text. > > I found some kernel performance regression issues that might be > related w/ 4.14.y LTS commit. > > 4.14.y commit: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v... > > The issue is observed when "console=" is used as a kernel parameter to > disable the kernel console.
I think that I see the problem. linux-4.14.y stable branch currently ignores "console=" parameter. As a result, a console (ttyX) is enabled by default.
What exact "performance issue" are you seeing?
[YF] one kernel thread was randomly blocked for more than ~40 milliseconds, causing a certain task to fail to process in time. [YF] the issue is highly random on a single device. But it might happen a few times per 24 hours on a certain percentage of devices. The overall percentage of devices that show the issue seems quite stable over a long period of time (somehow the magic number is ~40%.). [YF] local test on a pool of devices does not show any correlation w/ any particular devices.
This might happen when there is a flood of messages to be printed to the console. It does not happen when there is no console.
It has been fixed by the upstream commit 3cffa06aeef7ece30f6b5ac0 ("printk/console: Allow to disable console output by using console="" or console=null")
The fix needs some tweaking for the stable branches because __add_preferred_console() has gained more parameters over time.
It seems that all longterm stable branches are affected. I am going to prepare the backports.
Best Regards, Petr