[ Adding Greg to CC ]
On 6/12/19 6:04 AM, Jan Kara wrote:
On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
On 5/30/19 3:45 AM, Paolo Valente wrote:
[...]
At any rate, since you pointed out that you are interested in out-of-the-box performance, let me complete the context: in case low_latency is left set, one gets, in return for this 12% loss, a) at least 1000% higher responsiveness, e.g., 1000% lower start-up times of applications under load [1]; b) 500-1000% higher throughput in multi-client server workloads, as I already pointed out [2].
I'm very happy that you could solve the problem without having to compromise on any of the performance characteristics/features of BFQ!
I'm going to prepare complete patches. In addition, if ok for you, I'll report these results on the bug you created. Then I guess we can close it.
Sounds great!
Hi Paolo,
Hope you are doing great!
I was wondering if you got a chance to post these patches to LKML for review and inclusion... (No hurry, of course!)
Also, since your fixes address the performance issues in BFQ, do you have any thoughts on whether they can be adapted to CFQ as well, to benefit the older stable kernels that still support CFQ?
Since CFQ doesn't exist in current upstream kernel anymore, I seriously doubt you'll be able to get any performance improvements for it in the stable kernels...
I suspected as much, but that seems unfortunate though. The latest LTS kernel is based on 4.19, which still supports CFQ. It would have been great to have a process to address significant issues on older kernels too.
Greg, do you have any thoughts on this? The context is that both CFQ and BFQ I/O schedulers have issues that cause I/O throughput to suffer upto 10x - 30x on certain workloads and system configurations, as reported in [1].
In this thread, Paolo posted patches to fix BFQ performance on mainline. However CFQ suffers from the same performance collapse, but CFQ was removed from the kernel in v5.0. So obviously the usual stable backporting path won't work here for several reasons:
1. There won't be a mainline commit to backport from, as CFQ no longer exists in mainline.
2. This is not a security/stability fix, and is likely to involve invasive changes.
I was wondering if there was a way to address the performance issues in CFQ in the older stable kernels (including the latest LTS 4.19), despite the above constraints, since the performance drop is much too significant. I guess not, but thought I'd ask :-)
[1]. https://lore.kernel.org/lkml/8d72fcf7-bbb4-2965-1a06-e9fc177a8938@csail.mit....
Regards, Srivatsa VMware Photon OS
On Wed, Jun 12, 2019 at 12:36:53PM -0700, Srivatsa S. Bhat wrote:
[ Adding Greg to CC ]
On 6/12/19 6:04 AM, Jan Kara wrote:
On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
On 5/30/19 3:45 AM, Paolo Valente wrote:
[...]
At any rate, since you pointed out that you are interested in out-of-the-box performance, let me complete the context: in case low_latency is left set, one gets, in return for this 12% loss, a) at least 1000% higher responsiveness, e.g., 1000% lower start-up times of applications under load [1]; b) 500-1000% higher throughput in multi-client server workloads, as I already pointed out [2].
I'm very happy that you could solve the problem without having to compromise on any of the performance characteristics/features of BFQ!
I'm going to prepare complete patches. In addition, if ok for you, I'll report these results on the bug you created. Then I guess we can close it.
Sounds great!
Hi Paolo,
Hope you are doing great!
I was wondering if you got a chance to post these patches to LKML for review and inclusion... (No hurry, of course!)
Also, since your fixes address the performance issues in BFQ, do you have any thoughts on whether they can be adapted to CFQ as well, to benefit the older stable kernels that still support CFQ?
Since CFQ doesn't exist in current upstream kernel anymore, I seriously doubt you'll be able to get any performance improvements for it in the stable kernels...
I suspected as much, but that seems unfortunate though. The latest LTS kernel is based on 4.19, which still supports CFQ. It would have been great to have a process to address significant issues on older kernels too.
Greg, do you have any thoughts on this? The context is that both CFQ and BFQ I/O schedulers have issues that cause I/O throughput to suffer upto 10x - 30x on certain workloads and system configurations, as reported in [1].
In this thread, Paolo posted patches to fix BFQ performance on mainline. However CFQ suffers from the same performance collapse, but CFQ was removed from the kernel in v5.0. So obviously the usual stable backporting path won't work here for several reasons:
There won't be a mainline commit to backport from, as CFQ no longer exists in mainline.
This is not a security/stability fix, and is likely to involve invasive changes.
I was wondering if there was a way to address the performance issues in CFQ in the older stable kernels (including the latest LTS 4.19), despite the above constraints, since the performance drop is much too significant. I guess not, but thought I'd ask :-)
If someone cares about something like this, then I strongly just recommend they move to the latest kernel version. There should not be anything stoping them from doing that, right? Nothing "forces" anyone to be on the 4.19.y release, especially when it really starts to show its age.
Don't ever treat the LTS releases as "the only thing someone can run, so we must backport huge things to it!" Just use 5.1, and then move to 5.2 when it is out and so on. That's always the preferred way, you always get better support, faster kernels, newer features, better hardware support, and most importantly, more bugfixes.
I wrote a whole essay on this thing, but no one ever seems to read it...
thanks,
greg k-h
On 6/12/19 11:02 PM, Greg Kroah-Hartman wrote:
On Wed, Jun 12, 2019 at 12:36:53PM -0700, Srivatsa S. Bhat wrote:
[ Adding Greg to CC ]
On 6/12/19 6:04 AM, Jan Kara wrote:
On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
On 5/30/19 3:45 AM, Paolo Valente wrote:
[...]
At any rate, since you pointed out that you are interested in out-of-the-box performance, let me complete the context: in case low_latency is left set, one gets, in return for this 12% loss, a) at least 1000% higher responsiveness, e.g., 1000% lower start-up times of applications under load [1]; b) 500-1000% higher throughput in multi-client server workloads, as I already pointed out [2].
I'm very happy that you could solve the problem without having to compromise on any of the performance characteristics/features of BFQ!
I'm going to prepare complete patches. In addition, if ok for you, I'll report these results on the bug you created. Then I guess we can close it.
Sounds great!
Hi Paolo,
Hope you are doing great!
I was wondering if you got a chance to post these patches to LKML for review and inclusion... (No hurry, of course!)
Also, since your fixes address the performance issues in BFQ, do you have any thoughts on whether they can be adapted to CFQ as well, to benefit the older stable kernels that still support CFQ?
Since CFQ doesn't exist in current upstream kernel anymore, I seriously doubt you'll be able to get any performance improvements for it in the stable kernels...
I suspected as much, but that seems unfortunate though. The latest LTS kernel is based on 4.19, which still supports CFQ. It would have been great to have a process to address significant issues on older kernels too.
Greg, do you have any thoughts on this? The context is that both CFQ and BFQ I/O schedulers have issues that cause I/O throughput to suffer upto 10x - 30x on certain workloads and system configurations, as reported in [1].
In this thread, Paolo posted patches to fix BFQ performance on mainline. However CFQ suffers from the same performance collapse, but CFQ was removed from the kernel in v5.0. So obviously the usual stable backporting path won't work here for several reasons:
There won't be a mainline commit to backport from, as CFQ no longer exists in mainline.
This is not a security/stability fix, and is likely to involve invasive changes.
I was wondering if there was a way to address the performance issues in CFQ in the older stable kernels (including the latest LTS 4.19), despite the above constraints, since the performance drop is much too significant. I guess not, but thought I'd ask :-)
If someone cares about something like this, then I strongly just recommend they move to the latest kernel version. There should not be anything stoping them from doing that, right? Nothing "forces" anyone to be on the 4.19.y release, especially when it really starts to show its age.
Don't ever treat the LTS releases as "the only thing someone can run, so we must backport huge things to it!" Just use 5.1, and then move to 5.2 when it is out and so on. That's always the preferred way, you always get better support, faster kernels, newer features, better hardware support, and most importantly, more bugfixes.
Thank you for the clarification!
Regards, Srivatsa VMware Photon OS
On Wed 12-06-19 12:36:53, Srivatsa S. Bhat wrote:
[ Adding Greg to CC ]
On 6/12/19 6:04 AM, Jan Kara wrote:
On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
On 5/30/19 3:45 AM, Paolo Valente wrote:
[...]
At any rate, since you pointed out that you are interested in out-of-the-box performance, let me complete the context: in case low_latency is left set, one gets, in return for this 12% loss, a) at least 1000% higher responsiveness, e.g., 1000% lower start-up times of applications under load [1]; b) 500-1000% higher throughput in multi-client server workloads, as I already pointed out [2].
I'm very happy that you could solve the problem without having to compromise on any of the performance characteristics/features of BFQ!
I'm going to prepare complete patches. In addition, if ok for you, I'll report these results on the bug you created. Then I guess we can close it.
Sounds great!
Hi Paolo,
Hope you are doing great!
I was wondering if you got a chance to post these patches to LKML for review and inclusion... (No hurry, of course!)
Also, since your fixes address the performance issues in BFQ, do you have any thoughts on whether they can be adapted to CFQ as well, to benefit the older stable kernels that still support CFQ?
Since CFQ doesn't exist in current upstream kernel anymore, I seriously doubt you'll be able to get any performance improvements for it in the stable kernels...
I suspected as much, but that seems unfortunate though. The latest LTS kernel is based on 4.19, which still supports CFQ. It would have been great to have a process to address significant issues on older kernels too.
Well, you could still tune the performance difference by changing slice_idle and group_idle tunables for CFQ (in /sys/block/<device>/queue/iosched/). Changing these to lower values will reduce the throughput loss when switching between cgroups at the cost of lower accuracy of enforcing configured IO proportions among cgroups.
Honza
On 6/13/19 1:20 AM, Jan Kara wrote:
On Wed 12-06-19 12:36:53, Srivatsa S. Bhat wrote:
[ Adding Greg to CC ]
On 6/12/19 6:04 AM, Jan Kara wrote:
On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
On 5/30/19 3:45 AM, Paolo Valente wrote:
[...]
At any rate, since you pointed out that you are interested in out-of-the-box performance, let me complete the context: in case low_latency is left set, one gets, in return for this 12% loss, a) at least 1000% higher responsiveness, e.g., 1000% lower start-up times of applications under load [1]; b) 500-1000% higher throughput in multi-client server workloads, as I already pointed out [2].
I'm very happy that you could solve the problem without having to compromise on any of the performance characteristics/features of BFQ!
I'm going to prepare complete patches. In addition, if ok for you, I'll report these results on the bug you created. Then I guess we can close it.
Sounds great!
Hi Paolo,
Hope you are doing great!
I was wondering if you got a chance to post these patches to LKML for review and inclusion... (No hurry, of course!)
Also, since your fixes address the performance issues in BFQ, do you have any thoughts on whether they can be adapted to CFQ as well, to benefit the older stable kernels that still support CFQ?
Since CFQ doesn't exist in current upstream kernel anymore, I seriously doubt you'll be able to get any performance improvements for it in the stable kernels...
I suspected as much, but that seems unfortunate though. The latest LTS kernel is based on 4.19, which still supports CFQ. It would have been great to have a process to address significant issues on older kernels too.
Well, you could still tune the performance difference by changing slice_idle and group_idle tunables for CFQ (in /sys/block/<device>/queue/iosched/). Changing these to lower values will reduce the throughput loss when switching between cgroups at the cost of lower accuracy of enforcing configured IO proportions among cgroups.
Good point, and seems fair enough, thank you!
Regards, Srivatsa VMware Photon OS
On 6/12/19 1:36 PM, Srivatsa S. Bhat wrote:
[ Adding Greg to CC ]
On 6/12/19 6:04 AM, Jan Kara wrote:
On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
On 5/30/19 3:45 AM, Paolo Valente wrote:
[...]
At any rate, since you pointed out that you are interested in out-of-the-box performance, let me complete the context: in case low_latency is left set, one gets, in return for this 12% loss, a) at least 1000% higher responsiveness, e.g., 1000% lower start-up times of applications under load [1]; b) 500-1000% higher throughput in multi-client server workloads, as I already pointed out [2].
I'm very happy that you could solve the problem without having to compromise on any of the performance characteristics/features of BFQ!
I'm going to prepare complete patches. In addition, if ok for you, I'll report these results on the bug you created. Then I guess we can close it.
Sounds great!
Hi Paolo,
Hope you are doing great!
I was wondering if you got a chance to post these patches to LKML for review and inclusion... (No hurry, of course!)
Also, since your fixes address the performance issues in BFQ, do you have any thoughts on whether they can be adapted to CFQ as well, to benefit the older stable kernels that still support CFQ?
Since CFQ doesn't exist in current upstream kernel anymore, I seriously doubt you'll be able to get any performance improvements for it in the stable kernels...
I suspected as much, but that seems unfortunate though. The latest LTS kernel is based on 4.19, which still supports CFQ. It would have been great to have a process to address significant issues on older kernels too.
Greg, do you have any thoughts on this? The context is that both CFQ and BFQ I/O schedulers have issues that cause I/O throughput to suffer upto 10x - 30x on certain workloads and system configurations, as reported in [1].
In this thread, Paolo posted patches to fix BFQ performance on mainline. However CFQ suffers from the same performance collapse, but CFQ was removed from the kernel in v5.0. So obviously the usual stable backporting path won't work here for several reasons:
There won't be a mainline commit to backport from, as CFQ no longer exists in mainline.
This is not a security/stability fix, and is likely to involve invasive changes.
I was wondering if there was a way to address the performance issues in CFQ in the older stable kernels (including the latest LTS 4.19), despite the above constraints, since the performance drop is much too significant. I guess not, but thought I'd ask :-)
[1]. https://lore.kernel.org/lkml/8d72fcf7-bbb4-2965-1a06-e9fc177a8938@csail.mit....
This issue has always been there. There will be no specific patches made for stable for something that doesn't even exist in the newer kernels.
linux-stable-mirror@lists.linaro.org
-
Greg Kroah-Hartman -
Jan Kara -
Jens Axboe -
Srivatsa S. Bhat