(+ more scheduler folks)

tl;dr

JB has a workload that hates aggressive migration on the 2nd Generation EPYC platform that has a small LLC domain (4C/8T) and very noticeable C2C latency.

Based on JB's observation so far, reverting commit 16b0a7a1a0af ("sched/fair: Ensure tasks spreading in LLC during LB") and commit c5b0a7eefc70 ("sched/fair: Remove sysctl_sched_migration_cost condition") helps the workload. Both those commits allow aggressive migrations for work conservation except it also increased cache misses which slows the workload quite a bit.

"relax_domain_level" helps but cannot be set at runtime and I couldn't think of any stable / debug interfaces that JB hasn't tried out already that can help this workload.

There is a patch towards the end to set "relax_domain_level" at runtime but given cpusets got away with this when transitioning to cgroup-v2, I don't know what the sentiments are around its usage. Any input / feedback is greatly appreciated.

On 4/28/2025 1:13 PM, Jean-Baptiste Roquefere wrote:

Hello Prateek,

thank's for your reponse.

...

Looking at the commit logs, it looks like these commits do solve other problems around load balancing and might not be trivial to revert without evaluating the damages.

it's definitely not a productizable workaround !

...

The processor you are running on, the AME EPYC 7702P based on the Zen2 architecture contains 4 cores / 8 threads per CCX (LLC domain) which is perhaps why reducing the thread count to below this limit is helping your workload.

What we suspect is that when running the workload, the threads that regularly sleep trigger a newidle balancing which causes them to move to another CCX leading to higher number of L3 misses.

To confirm this, would it be possible to run the workload with the not-yet-upstream perf sched stats [1] tool and share the result from perf sched stats diff for the data from v6.12.17 and v6.12.17 + patch to rule out any other second order effect.

[1] https://lore.kernel.org/all/20250311120230.61774-1-swapnil.sapkal@amd.com/

I had to patch tools/perf/util/session.c : static int open_file_read(struct perf_data *data) due to "failed to open perf.data: File exists" (looked more like a compiler issue than a tool/perf issue)

$ ./perf sched stats diff perf.data.6.12.17 perf.data.6.12.17patched > perf.diff (see perf.diff attached)

Thank you for all the information Jean. I'll highlight the interesting bits (at least the bits that stood out to me)

(left is mainline, right is mainline with the two commits mentioned by JB reverted)

total runtime by tasks on this processor (in jiffies) : 123927676874,108531911002 | -12.42% | total waittime by tasks on this processor (in jiffies) : 34729211241, 27076295778 | -22.04% | ( 28.02%, 24.95% ) total timeslices run on this cpu : 501606, 489799 | -2.35% |

Since "total runtime" is lower on the right, it means that the CPUs were not as well utilized with the commits reverted however the reduction in the "total waittime" suggests things are running faster and on overage there are 0.28 waiting tasks on mainline compared to 0.24 with the commits reverted.

---------------------------------------- <Category newidle - SMT> ---------------------------------------- load_balance() count on cpu newly idle : 331664, 31153 | -90.61% | $ 0.15, 1.55 $ load_balance() failed to find busier group on cpu newly idle : 300234, 28470 | -90.52% | $ 0.16, 1.70 $ *load_balance() success count on cpu newly idle : 28386, 1544 | -94.56% | *avg task pulled per successful lb attempt (cpu newly idle) : 1.00, 1.01 | 0.46% | ---------------------------------------- <Category newidle - MC > ---------------------------------------- load_balance() count on cpu newly idle : 258017, 29345 | -88.63% | $ 0.19, 1.65 $ load_balance() failed to find busier group on cpu newly idle : 131096, 16081 | -87.73% | $ 0.37, 3.01 $ *load_balance() success count on cpu newly idle : 23286, 2181 | -90.63% | *avg task pulled per successful lb attempt (cpu newly idle) : 1.03, 1.01 | -1.23% | ---------------------------------------- <Category newidle - PKG> ---------------------------------------- load_balance() count on cpu newly idle : 124013, 27086 | -78.16% | $ 0.39, 1.78 $ load_balance() failed to find busier group on cpu newly idle : 11812, 3063 | -74.07% | $ 4.09, 15.78 $ *load_balance() success count on cpu newly idle : 13892, 4739 | -65.89% | *avg task pulled per successful lb attempt (cpu newly idle) : 1.07, 1.10 | 3.32% | ----------------------------------------------------------------------------------------------------------

Most migrations are from newidle balancing which seems to move task across cores ( > 50% of time) and the LLC too (~8% of the times).

...

Assuming you control these deployments, would it possible to run the workload on a kernel running with "relax_domain_level=2" kernel cmdline that restricts newidle balance to only within the CCX. As a side effect, it also limits  task wakeups to the same LLC domain but I would still like to know if this makes a difference to the workload you are running.

On vanilla 6.12.17 it gives the IPC we expected:

Thank you JB for trying out this experiment. I'm not very sure what the views are on "relax_domain_level" and I'm hoping the other scheduler folks will chime in here - Is it a debug knob? Can it be used in production?

I know it had additional uses with cpuset in cgroup-v1 but was not adopted in v2 - are there any nasty historic reasons for this?

+--------------------+--------------------------+-----------------------+ |                    | relax_domain_level unset | relax_domain_level=2  | +--------------------+--------------------------+-----------------------+ | Threads            |  210                     | 210                  | | Utilization (%)    |  65,86                   | 52,01                | | CPU effective freq |  1 622,93                |  1 294,12             | | IPC                |  1,14                    | 1,42                 | | L2 access (pti)    |  34,36                   | 38,18                | | L2 miss   (pti)    |  7,34                    | 7,78                 | | L3 miss   (abs)    |  39 711 971 741          |  33 929 609 924       | | Mem (GB/s)         |  70,68                   | 49,10                | | Context switches   |  109 281 524             |  107 896 729          | +--------------------+--------------------------+-----------------------+

Kind regards,

JB

JB asked if there is any way to toggle "relax_domain_level" at runtime on mainline and I couldn't find any easy way other than using cpusets with cgroup-v1 which is probably harder to deploy at scale than the pinning strategy that JB mentioned originally.

I currently cannot think of any stable interface that exists currently to allow sticky behavior and mitigate aggressive migration for work conservation - JB did try almost everything available that he summarized in his original report.

Could something like below be a stop-gap band-aid to remedy such the case of workloads that don't mind temporary imbalance in favor of cache hotness?

--- From: K Prateek Nayak kprateek.nayak@amd.com Subject: [RFC PATCH] sched/debug: Allow overriding "relax_domain_level" at runtime

Jean-Baptiste noted that Ateme's workload experiences poor IPC on a 2nd Generation EPYC system and narrowed down the major culprits to commit 16b0a7a1a0af ("sched/fair: Ensure tasks spreading in LLC during LB") and commit c5b0a7eefc70 ("sched/fair: Remove sysctl_sched_migration_cost condition") both of which enable more aggressive migrations in favor of work conservation.

The larger C2C latency on the platform coupled with a smaller L3 size of 4C/8T makes downside of aggressive balance very prominent. Looking at the perf sched stats report from JB [1], when the two commits are reverted, despite the "total runtime" seeing a dip of 11% showing a better load distribution on mainline, the "total waittime" dips by 22% showing despite the imbalance, the workload runs faster and this improvement can be co-related to the higher IPC and the reduced L3 misses in data shared by JB. Most of the migration during load balancing can be attributed to newidle balance.

JB confirmed that using "relax_domain_level=2" in kernel cmdline helps this particular workload by restricting the scope of wakeups and migrations during newidle balancing however "relax_domain_level" works on topology levels before degeneration and setting the level before inspecting the topology might not be trivial at boot time.

Furthermore, a runtime knob that can help quickly narrow down any changes in workload behavior to aggressive migrations during load balancing can be helpful during debugs.

Introduce "relax_domain_level" in sched debugfs and allow overriding the knob at runtime.

# cat /sys/kernel/debug/sched/relax_domain_level -1

# echo Y > /sys/kernel/debug/sched/verbose # cat /sys/kernel/debug/sched/domains/cpu0/domain*/flags SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_LLC SD_PREFER_SIBLING SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_PREFER_SIBLING SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SERIALIZE SD_OVERLAP SD_NUMA

To restrict newidle balance to only within the LLC, "relax_domain_level" can be set to level 3 (SMT, CLUSTER, *MC* , PKG, NUMA)

# echo 3 > /sys/kernel/debug/sched/relax_domain_level # cat /sys/kernel/debug/sched/domains/cpu0/domain*/flags SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_LLC SD_PREFER_SIBLING SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_PREFER_SIBLING SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SERIALIZE SD_OVERLAP SD_NUMA

"relax_domain_level" forgives short term imbalances. Longer term imbalances will be eventually caught by the periodic load balancer and the system will reach a state of balance, only slightly later.

Link: https://lore.kernel.org/all/996ca8cb-3ac8-4f1b-93f1-415f43922d7a@ateme.com/ [1] Signed-off-by: K Prateek Nayak kprateek.nayak@amd.com --- include/linux/sched/topology.h | 6 ++-- kernel/sched/debug.c | 52 ++++++++++++++++++++++++++++++++++ kernel/sched/topology.c | 2 +- 3 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 198bb5cc1774..5f59bdc1d5b1 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -65,8 +65,10 @@ struct sched_domain_attr { int relax_domain_level; };

-#define SD_ATTR_INIT (struct sched_domain_attr) { \ - .relax_domain_level = -1, \ +extern int default_relax_domain_level; + +#define SD_ATTR_INIT (struct sched_domain_attr) { \ + .relax_domain_level = default_relax_domain_level, \ }

extern int sched_domain_level_max; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 557246880a7e..cc6944b35535 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -214,6 +214,57 @@ static const struct file_operations sched_scaling_fops = { .release = single_release, };

+DEFINE_MUTEX(relax_domain_mutex); + +static ssize_t sched_relax_domain_write(struct file *filp, + const char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + int relax_domain_level; + char buf[16]; + + if (cnt > 15) + cnt = 15; + + if (copy_from_user(&buf, ubuf, cnt)) + return -EFAULT; + buf[cnt] = '\0'; + + if (kstrtoint(buf, 10, &relax_domain_level)) + return -EINVAL; + + if (relax_domain_level < -1 || relax_domain_level > sched_domain_level_max + 1) + return -EINVAL; + + guard(mutex)(&relax_domain_mutex); + + if (relax_domain_level != default_relax_domain_level) { + default_relax_domain_level = relax_domain_level; + rebuild_sched_domains(); + } + + *ppos += cnt; + return cnt; +} +static int sched_relax_domain_show(struct seq_file *m, void *v) +{ + seq_printf(m, "%d\n", default_relax_domain_level); + return 0; +} + +static int sched_relax_domain_open(struct inode *inode, struct file *filp) +{ + return single_open(filp, sched_relax_domain_show, NULL); +} + +static const struct file_operations sched_relax_domain_fops = { + .open = sched_relax_domain_open, + .write = sched_relax_domain_write, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + #endif /* SMP */

#ifdef CONFIG_PREEMPT_DYNAMIC @@ -516,6 +567,7 @@ static __init int sched_init_debug(void) debugfs_create_file("tunable_scaling", 0644, debugfs_sched, NULL, &sched_scaling_fops); debugfs_create_u32("migration_cost_ns", 0644, debugfs_sched, &sysctl_sched_migration_cost); debugfs_create_u32("nr_migrate", 0644, debugfs_sched, &sysctl_sched_nr_migrate); + debugfs_create_file("relax_domain_level", 0644, debugfs_sched, NULL, &sched_relax_domain_fops);

sched_domains_mutex_lock(); update_sched_domain_debugfs(); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index a2a38e1b6f18..eb5c8a9cd904 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1513,7 +1513,7 @@ static void asym_cpu_capacity_scan(void) * Non-inlined to reduce accumulated stack pressure in build_sched_domains() */

-static int default_relax_domain_level = -1; +int default_relax_domain_level = -1; int sched_domain_level_max;

static int __init setup_relax_domain_level(char *str)