On Wed, Nov 6, 2019 at 5:43 PM Roman Gushchin guro@fb.com wrote:
On Wed, Nov 06, 2019 at 05:25:26PM -0800, Shakeel Butt wrote:
On Wed, Nov 6, 2019 at 4:22 PM Johannes Weiner hannes@cmpxchg.org wrote:
On Wed, Nov 06, 2019 at 02:51:30PM -0800, Roman Gushchin wrote:
We've encountered a rcu stall in get_mem_cgroup_from_mm():
rcu: INFO: rcu_sched self-detected stall on CPU rcu: 33-....: (21000 ticks this GP) idle=6c6/1/0x4000000000000002 softirq=35441/35441 fqs=5017 (t=21031 jiffies g=324821 q=95837) NMI backtrace for cpu 33 <...> RIP: 0010:get_mem_cgroup_from_mm+0x2f/0x90 <...> __memcg_kmem_charge+0x55/0x140 __alloc_pages_nodemask+0x267/0x320 pipe_write+0x1ad/0x400 new_sync_write+0x127/0x1c0 __kernel_write+0x4f/0xf0 dump_emit+0x91/0xc0 writenote+0xa0/0xc0 elf_core_dump+0x11af/0x1430 do_coredump+0xc65/0xee0 ? unix_stream_sendmsg+0x37d/0x3b0 get_signal+0x132/0x7c0 do_signal+0x36/0x640 ? recalc_sigpending+0x17/0x50 exit_to_usermode_loop+0x61/0xd0 do_syscall_64+0xd4/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The problem is caused by an exiting task which is associated with an offline memcg. We're iterating over and over in the do {} while (!css_tryget_online()) loop, but obviously the memcg won't become online and the exiting task won't be migrated to a live memcg.
Let's fix it by switching from css_tryget_online() to css_tryget().
As css_tryget_online() cannot guarantee that the memcg won't go offline, the check is usually useless, except some rare cases when for example it determines if something should be presented to a user.
A similar problem is described by commit 18fa84a2db0e ("cgroup: Use css_tryget() instead of css_tryget_online() in task_get_css()").
Signed-off-by: Roman Gushchin guro@fb.com Cc: stable@vger.kernel.org Cc: Tejun Heo tj@kernel.org
Acked-by: Johannes Weiner hannes@cmpxchg.org
The bug aside, it doesn't matter whether the cgroup is online for the callers. It used to matter when offlining needed to evacuate all charges from the memcg, and so needed to prevent new ones from showing up, but we don't care now.
Should get_mem_cgroup_from_current() and get_mem_cgroup_from_page() be switched to css_tryget() as well then?
In those case it can't cause a rcu stall, so it's not a so urgent. But you are right, we should probably do the same here. I'll look at all remaining callers and prepare the patchset.
I'll also probably rename it to css_tryget_if_online() to make obvious that it doesn't hold the cgroup from being onlined.
SGTM