After upgrading to the 4.19 series I've started getting problems with early OOM.
I run a Gentoo system and do large compiles like the chromium browser in a v1 memory cgroup. When I build chromium in the memory cgroup the OOM killer runs and kills programs outside of the cgroup. This happens even when there is plenty of free memory both in and outside of the cgroup.
The memory cgroup is named "12G" and is setup like this: /sys/fs/cgroup/memory/12G/memory.kmem.limit_in_bytes:1073741824 /sys/fs/cgroup/memory/12G/memory.kmem.tcp.limit_in_bytes:1073741824 /sys/fs/cgroup/memory/12G/memory.limit_in_bytes:12884901888 /sys/fs/cgroup/memory/12G/memory.memsw.limit_in_bytes:12884901888 /sys/fs/cgroup/memory/12G/memory.soft_limit_in_bytes:9223372036854771712
The system has MemTotal: 16244996 kB
I run the chromium compile job using Gentoo's package manager like this: cgexec -g memory:12G emerge -1 www-client/chromium
That compile job usually fails and dmesg looks like this:
[ 1084.634827] SLUB: Unable to allocate memory on node -1, gfp=0x6000c0(GFP_KERNEL) [ 1084.634836] cache: dentry(100:12G), object size: 192, buffer size: 192, default order: 0, min order: 0 [ 1084.634838] node 0: slabs: 26888, objs: 564648, free: 0 [ 1084.634857] SLUB: Unable to allocate memory on node -1, gfp=0x6000c0(GFP_KERNEL) [ 1084.634860] cache: dentry(100:12G), object size: 192, buffer size: 192, default order: 0, min order: 0 [ 1084.634861] node 0: slabs: 26888, objs: 564648, free: 0 [ 1084.648583] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.648593] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.648595] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.648695] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.648700] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.648702] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.648794] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.648799] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.648801] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.648898] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.648900] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.648908] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.649000] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.649004] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.649006] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.649103] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.649107] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.649109] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.649198] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.649199] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.649199] node 0: slabs: 19132, objs: 592952, free: 0 [ 1084.649293] SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) [ 1084.649299] cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 [ 1084.649299] node 0: slabs: 19132, objs: 592952, free: 0 [ 1146.798499] Purging GPU memory, 41040 pages freed, 6933 pages still pinned. [ 1146.798512] 4 and 0 pages still available in the bound and unbound GPU page lists. [ 1146.798649] Purging GPU memory, 0 pages freed, 6933 pages still pinned. [ 1146.798653] 4 and 0 pages still available in the bound and unbound GPU page lists. [ 1146.798696] emerge invoked oom-killer: gfp_mask=0x0(), nodemask=(null), order=0, oom_score_adj=0 [ 1146.798699] emerge cpuset= [ 1146.798701] / [ 1146.798703] mems_allowed=0 [ 1146.798705] CPU: 4 PID: 16719 Comm: emerge Not tainted 4.19.69 #43 [ 1146.798707] Hardware name: Gigabyte Technology Co., Ltd. Z97X-Gaming G1/Z97X-Gaming G1, BIOS F9 07/31/2015 [ 1146.798708] Call Trace: [ 1146.798713] dump_stack+0x46/0x60 [ 1146.798718] dump_header+0x67/0x28d [ 1146.798721] oom_kill_process.cold.31+0xb/0x1f3 [ 1146.798723] out_of_memory+0x129/0x250 [ 1146.798728] pagefault_out_of_memory+0x64/0x77 [ 1146.798732] __do_page_fault+0x3c1/0x3d0 [ 1146.798735] do_page_fault+0x2c/0x123 [ 1146.798738] ? page_fault+0x8/0x30 [ 1146.798740] page_fault+0x1e/0x30 [ 1146.798743] RIP: 0033:0x7f3745ccf628 [ 1146.798744] Code: 43 68 48 8b 40 08 48 89 44 24 08 48 8b 83 00 03 00 00 48 85 c0 0f 84 ff 00 00 00 48 8b 74 24 20 8b 54 24 30 23 93 f8 02 00 00 <48> 8b 14 d0 8b 83 fc 02 00 00 89 f7 c4 e2 fb f7 c6 c4 e2 fb f7 c2 [ 1146.798746] RSP: 002b:00007ffd34c11460 EFLAGS: 00010202 [ 1146.798748] RAX: 000056439d3e5288 RBX: 00007f3745cf0130 RCX: 0000000000000013 [ 1146.798750] RDX: 0000000000000001 RSI: 00000000777f2f73 RDI: 00007f37458c7303 [ 1146.798752] RBP: 0000000000000000 R08: 00007ffd34c11590 R09: 00007f3745cf03f0 [ 1146.798754] R10: 00007f3745c91000 R11: 00007f374557b5c0 R12: 0000000000000000 [ 1146.798756] R13: 0000000000000001 R14: 0000000000000000 R15: 00007f3745c92f58 [ 1146.798758] Mem-Info: [ 1146.798761] active_anon:438779 inactive_anon:41657 isolated_anon:0 active_file:591199 inactive_file:2307684 isolated_file:0 unevictable:1 dirty:499 writeback:0 unstable:0 slab_reclaimable:299846 slab_unreclaimable:22656 mapped:134859 shmem:51982 pagetables:5460 bounce:0 free:311407 free_pcp:4594 free_cma:0 [ 1146.798765] Node 0 active_anon:1755116kB inactive_anon:166628kB active_file:2364796kB inactive_file:9230736kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:539436kB dirty:1996kB writeback:0kB shmem:207928kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 598016kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1146.798770] DMA free:15900kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1146.798772] lowmem_reserve[]: [ 1146.798773] 0 [ 1146.798774] 3030 [ 1146.798775] 15777 [ 1146.798776] 15777 [ 1146.798780] DMA32 free:1111400kB min:12968kB low:16208kB high:19448kB active_anon:253352kB inactive_anon:1160kB active_file:48104kB inactive_file:1489456kB unevictable:0kB writepending:0kB present:3172224kB managed:3172224kB mlocked:0kB kernel_stack:16kB pagetables:248kB bounce:0kB free_pcp:10952kB local_pcp:1356kB free_cma:0kB [ 1146.798782] lowmem_reserve[]: [ 1146.798783] 0 [ 1146.798784] 0 [ 1146.798786] 12746 [ 1146.798787] 12746 [ 1146.798791] Normal free:118328kB min:54544kB low:68180kB high:81816kB active_anon:1501764kB inactive_anon:165596kB active_file:2316692kB inactive_file:7741280kB unevictable:4kB writepending:1996kB present:13367296kB managed:13056872kB mlocked:4kB kernel_stack:9216kB pagetables:21592kB bounce:0kB free_pcp:7412kB local_pcp:1320kB free_cma:0kB [ 1146.798793] lowmem_reserve[]: [ 1146.798794] 0 [ 1146.798795] 0 [ 1146.798796] 0 [ 1146.798797] 0 [ 1146.798799] DMA: [ 1146.798800] 1*4kB [ 1146.798802] (U) [ 1146.798803] 1*8kB [ 1146.798805] (U) [ 1146.798806] 1*16kB [ 1146.798807] (U) [ 1146.798808] 0*32kB [ 1146.798810] 2*64kB [ 1146.798811] (U) [ 1146.798812] 1*128kB [ 1146.798813] (U) [ 1146.798815] 1*256kB [ 1146.798816] (U) [ 1146.798817] 0*512kB [ 1146.798818] 1*1024kB [ 1146.798819] (U) [ 1146.798821] 1*2048kB [ 1146.798822] (M) [ 1146.798823] 3*4096kB [ 1146.798824] (M) [ 1146.798826] = 15900kB [ 1146.798828] DMA32: [ 1146.798829] 6346*4kB [ 1146.798831] (UME) [ 1146.798977] 2650*8kB [ 1146.798980] (UME) [ 1146.798981] 1118*16kB [ 1146.799118] (UME) [ 1146.799119] 774*32kB [ 1146.799121] (UME) [ 1146.799122] 367*64kB [ 1146.799123] (UME) [ 1146.799125] 191*128kB [ 1146.799126] (UME) [ 1146.799128] 64*256kB [ 1146.799290] (UME) [ 1146.799291] 37*512kB [ 1146.799291] (UM) [ 1146.799291] 21*1024kB [ 1146.799292] (UME) [ 1146.799292] 194*2048kB [ 1146.799292] (UM) [ 1146.799292] 127*4096kB [ 1146.799293] (M) [ 1146.799293] = 1111512kB [ 1146.799293] Normal: [ 1146.799294] 2360*4kB [ 1146.799294] (UME) [ 1146.799294] 1483*8kB [ 1146.799295] (UME) [ 1146.799295] 506*16kB [ 1146.799295] (UME) [ 1146.799295] 377*32kB [ 1146.799296] (UME) [ 1146.799296] 171*64kB [ 1146.799296] (UME) [ 1146.799296] 59*128kB [ 1146.799297] (UME) [ 1146.799297] 46*256kB [ 1146.799297] (UME) [ 1146.799297] 7*512kB [ 1146.799298] (UME) [ 1146.799298] 4*1024kB [ 1146.799298] (UE) [ 1146.799299] 5*2048kB [ 1146.799299] (ME) [ 1146.799299] 7*4096kB [ 1146.799299] (M) [ 1146.799300] = 118328kB [ 1146.799301] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1146.799301] 2950857 total pagecache pages [ 1146.799303] 0 pages in swap cache [ 1146.799576] Swap cache stats: add 0, delete 0, find 0/0 [ 1146.799578] Free swap = 4194300kB [ 1146.799579] Total swap = 4194300kB [ 1146.799581] 4138876 pages RAM [ 1146.799582] 0 pages HighMem/MovableOnly [ 1146.799584] 77627 pages reserved [ 1146.799585] Tasks state (memory values in pages): [ 1146.799586] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 1146.799601] [ 971] 0 971 3161 967 57344 0 0 systemd-udevd [ 1146.799605] [ 2152] 103 2152 902 692 40960 0 0 dbus-daemon [ 1146.799607] [ 2185] 0 2185 3040 597 57344 0 0 syslog-ng [ 1146.799609] [ 2186] 0 2186 77434 1839 94208 0 0 syslog-ng [ 1146.799611] [ 2187] 0 2187 3299 1286 69632 0 0 syslog_redirect [ 1146.799798] [ 2222] 0 2222 96102 1318 110592 0 0 console-kit-dae [ 1146.799800] [ 2239] 119 2239 486941 5736 237568 0 0 polkitd [ 1146.799801] [ 2259] 0 2259 612 23 45056 0 0 gpm [ 1146.799802] [ 2735] 0 2735 691 66 45056 0 0 dhcpcd [ 1146.799803] [ 2806] 123 2806 983 412 45056 0 0 ntpd [ 1146.799805] [ 2807] 123 2807 999 483 49152 0 0 ntpd [ 1146.799814] [ 2817] 0 2817 945 54 45056 0 0 ntpd [ 1146.799815] [ 2847] 0 2847 18576 902 118784 0 0 virtlogd [ 1146.799817] [ 2878] 0 2878 18577 885 110592 0 0 virtlockd [ 1146.799822] [ 2912] 0 2912 352359 5586 299008 0 0 libvirtd [ 1146.799823] [ 3055] 0 3055 1917 374 53248 0 0 smartd [ 1146.799824] [ 3084] 0 3084 2010 454 53248 0 0 cron [ 1146.799977] [ 3159] 0 3159 78172 1652 106496 0 0 lightdm [ 1146.799978] [ 3178] 0 3178 223859 45920 868352 0 0 X [ 1146.799984] [ 3218] 0 3218 42557 1645 98304 0 0 lightdm [ 1146.799985] [ 3230] 1000 3230 2302 795 61440 0 0 sh [ 1146.799986] [ 3238] 1000 3238 2160 559 57344 0 0 dbus-launch [ 1146.799992] [ 3239] 1000 3239 1023 705 49152 0 0 dbus-daemon [ 1146.799994] [ 3245] 1000 3245 47306 4073 274432 0 0 xfce4-session [ 1146.799998] [ 3247] 1000 3247 3564 1246 73728 0 0 xfconfd [ 1146.799999] [ 3250] 1000 3250 1776 82 49152 0 0 ssh-agent [ 1146.800000] [ 3252] 1000 3252 40853 214 73728 0 0 gpg-agent [ 1146.800001] [ 3255] 1000 3255 37070 6711 294912 0 0 xfwm4 [ 1146.800002] [ 3257] 1000 3257 29454 4047 266240 0 0 Thunar [ 1146.800004] [ 3259] 1000 3259 56521 6429 315392 0 0 xfce4-panel [ 1146.800005] [ 3260] 1000 3260 76198 5296 307200 0 0 xfsettingsd [ 1146.800006] [ 3264] 1000 3264 164200 15514 430080 0 0 xfdesktop [ 1146.800007] [ 3269] 0 3269 61046 1602 102400 0 0 upowerd [ 1146.800008] [ 3285] 1000 3285 66179 4569 290816 0 0 panel-2-actions [ 1146.800009] [ 3287] 1000 3287 36585 5112 278528 0 0 panel-8-quickla [ 1146.800010] [ 3288] 1000 3288 633361 62058 1359872 0 0 akregator [ 1146.800011] [ 3290] 1000 3290 71292 7355 294912 0 0 gkrellm [ 1146.800012] [ 3292] 1000 3292 60506 1728 102400 0 0 gvfsd [ 1146.800013] [ 3294] 1000 3294 28673 3873 262144 0 0 panel-6-systray [ 1146.800014] [ 3295] 1000 3295 155110 18850 507904 0 0 konversation [ 1146.800015] [ 3296] 1000 3296 35017 5801 274432 0 0 panel-1-genmon [ 1146.800016] [ 3297] 1000 3297 95199 7413 335872 0 0 panel-4-xfce4-t [ 1146.800018] [ 3298] 1000 3298 118574 21860 516096 0 0 kate [ 1146.800045] [ 3300] 1000 3300 35016 5745 278528 0 0 panel-7-datetim [ 1146.800046] [ 3301] 1000 3301 116227 18944 458752 0 0 konsole [ 1146.800047] [ 3303] 1000 3303 389058 54055 917504 0 0 clementine [ 1146.800048] [ 3372] 1000 3372 77563 1551 102400 0 0 at-spi-bus-laun [ 1146.800201] [ 3381] 1000 3381 865 687 49152 0 0 dbus-daemon [ 1146.800203] [ 3383] 1000 3383 41371 1516 90112 0 0 at-spi2-registr [ 1146.800205] [ 3399] 1000 3399 150830 9120 372736 0 0 kactivitymanage [ 1146.800212] [ 3418] 1000 3418 34971 4175 126976 0 0 clementine-tagr [ 1146.800214] [ 3419] 1000 3419 34971 4210 131072 0 0 clementine-tagr [ 1146.800218] [ 3420] 1000 3420 34971 4198 126976 0 0 clementine-tagr [ 1146.800219] [ 3421] 1000 3421 34971 4193 126976 0 0 clementine-tagr [ 1146.800220] [ 3422] 1000 3422 34971 4166 135168 0 0 clementine-tagr [ 1146.800222] [ 3427] 1000 3427 72028 6718 278528 0 0 kglobalaccel5 [ 1146.800228] [ 3428] 1000 3428 34971 4195 126976 0 0 clementine-tagr [ 1146.800229] [ 3429] 1000 3429 34971 4215 131072 0 0 clementine-tagr [ 1146.800231] [ 3431] 1000 3431 34971 4219 126976 0 0 clementine-tagr [ 1146.800236] [ 3444] 1000 3444 6474 5049 98304 0 0 bash [ 1146.800237] [ 3447] 1000 3447 90666 12645 548864 0 0 QtWebEngineProc [ 1146.800242] [ 3452] 1000 3452 149454 569 122880 0 0 mergerfs [ 1146.800243] [ 3487] 0 3487 100615 3753 151552 0 0 udisksd [ 1146.800244] [ 3512] 1000 3512 8823 1162 102400 0 0 sd_cicero [ 1146.800246] [ 3516] 1000 3516 8823 1123 98304 0 0 sd_dummy [ 1146.800247] [ 3519] 1000 3519 8829 1121 94208 0 0 sd_generic [ 1146.800251] [ 3521] 1000 3521 31475 2098 131072 0 0 sd_espeak [ 1146.800252] [ 3523] 1000 3523 74986 7943 311296 0 0 kiod5 [ 1146.800253] [ 3546] 0 3546 1992 411 53248 0 0 agetty [ 1146.800258] [ 3547] 0 3547 2025 601 53248 0 0 agetty [ 1146.800259] [ 3548] 0 3548 2025 588 57344 0 0 agetty [ 1146.800264] [ 3549] 0 3549 2025 649 49152 0 0 agetty [ 1146.800265] [ 3550] 0 3550 2025 608 49152 0 0 agetty [ 1146.800269] [ 3551] 0 3551 2025 604 49152 0 0 agetty [ 1146.800270] [ 3559] 1000 3559 39834 554 81920 0 0 speech-dispatch [ 1146.800274] [ 3569] 1000 3569 60094 2084 94208 0 0 gvfs-udisks2-vo [ 1146.800276] [ 3574] 1000 3574 24587 2321 172032 0 0 kdeinit5 [ 1146.800279] [ 3575] 1000 3575 76563 9081 331776 0 0 klauncher [ 1146.800280] [ 3596] 1000 3596 74713 9413 311296 0 0 kded5 [ 1146.800281] [ 3623] 1000 3623 46085 4836 212992 0 0 kio_http_cache_ [ 1146.800282] [ 3653] 1000 3653 138832 11871 372736 0 0 easystroke [ 1146.800289] [ 3654] 1000 3654 83783 12054 348160 0 0 redshift-gtk [ 1146.800290] [ 3655] 1000 3655 41365 7457 294912 0 0 orage [ 1146.800291] [ 3661] 1000 3661 60416 5023 249856 0 0 polkit-gnome-au [ 1146.800292] [ 3664] 1000 3664 60900 4510 245760 0 0 parcellite [ 1146.800293] [ 3680] 1000 3680 21790 2021 122880 0 0 xbindkeys [ 1146.800297] [ 3695] 1000 3695 3987 390 69632 0 0 redshift [ 1146.800298] [ 3696] 1000 3696 2198 691 57344 0 0 su [ 1146.800299] [ 3701] 1000 3701 57610 9559 380928 0 0 fcitx [ 1146.800305] [ 3706] 1000 3706 934 551 53248 0 0 dbus-daemon [ 1146.800306] [ 3711] 1000 3711 1224 28 49152 0 0 fcitx-dbus-watc [ 1146.800307] [ 3713] 1000 3713 89866 7908 303104 0 0 xfce4-notifyd [ 1146.800433] [ 3719] 1000 3719 27953 3652 196608 0 0 file.so [ 1146.800435] [ 3722] 0 3722 6339 4921 90112 0 0 bash [ 1146.800436] [ 3754] 0 3754 2899 865 65536 0 0 tmux: client [ 1146.800437] [ 3756] 0 3756 3593 1512 65536 0 0 tmux: server [ 1146.800438] [ 3760] 0 3760 1256 492 49152 0 0 tomoyo-queryd [ 1146.800439] [ 3764] 0 3764 6369 4973 90112 0 0 bash [ 1146.800440] [ 3771] 1000 3771 6473 5063 94208 0 0 bash [ 1146.800441] [ 3831] 1000 3831 6474 5057 90112 0 0 bash [ 1146.800442] [ 3847] 1000 3847 2084 226 57344 0 0 dmesg [ 1146.800444] [ 4187] 1000 4187 726438 221563 3211264 0 0 firefox [ 1146.800459] [ 23437] 1000 23437 416243 18201 1122304 0 300 QtWebEngineProc [ 1146.800460] [ 16704] 0 16704 69439 48429 610304 0 0 emerge [ 1146.800462] [ 16719] 0 16719 69439 46469 585728 0 0 emerge [ 1146.800467] [ 16724] 0 16724 1764 316 53248 0 0 rssnotify.pl [ 1146.800468] Out of memory: Kill process 23437 (QtWebEngineProc) score 303 or sacrifice child [ 1146.800491] Killed process 23437 (QtWebEngineProc) total-vm:1664972kB, anon-rss:16924kB, file-rss:55892kB, shmem-rss:0kB [ 1146.803320] oom_reaper: reaped process 23437 (QtWebEngineProc), now anon-rss:0kB, file-rss:0kB, shmem-rss:8kB [ 1146.837984] emerge: vmalloc: allocation failure, allocated 8192 of 20480 bytes, mode:0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null) [ 1146.837993] emerge cpuset= [ 1146.837995] / [ 1146.837997] mems_allowed=0 [ 1146.837999] CPU: 4 PID: 16742 Comm: emerge Not tainted 4.19.69 #43 [ 1146.838001] Hardware name: Gigabyte Technology Co., Ltd. Z97X-Gaming G1/Z97X-Gaming G1, BIOS F9 07/31/2015 [ 1146.838002] Call Trace: [ 1146.838008] dump_stack+0x46/0x60 [ 1146.838013] warn_alloc.cold.133+0x68/0xe8 [ 1146.838016] ? __alloc_pages_nodemask+0x1f7/0x290 [ 1146.838019] __vmalloc_node_range+0x148/0x220 [ 1146.838023] copy_process+0xaab/0x27b0 [ 1146.838025] ? _do_fork+0xb2/0x390 [ 1146.838028] _do_fork+0xb2/0x390 [ 1146.838030] do_syscall_64+0x59/0x180 [ 1146.838033] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1146.838036] RIP: 0033:0x7f3745793aae [ 1146.838038] Code: db 0f 85 25 01 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 a6 00 00 00 41 89 c4 85 c0 0f 85 b3 00 00 [ 1146.838040] RSP: 002b:00007ffd34c12ed0 EFLAGS: 00000246 [ 1146.838041] ORIG_RAX: 0000000000000038 [ 1146.838045] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3745793aae [ 1146.838046] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 [ 1146.838048] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f374557b5c0 [ 1146.838050] R10: 00007f374557b890 R11: 0000000000000246 R12: 00007f3744cb17b0 [ 1146.838051] R13: 00007f3745545ad4 R14: 00007f373a32d168 R15: 00007f3745538050 [ 1146.838053] Mem-Info: [ 1146.838056] active_anon:439660 inactive_anon:41529 isolated_anon:0 active_file:591639 inactive_file:2307377 isolated_file:0 unevictable:1 dirty:500 writeback:0 unstable:0 slab_reclaimable:299846 slab_unreclaimable:22643 mapped:133960 shmem:51818 pagetables:5302 bounce:0 free:310992 free_pcp:4130 free_cma:0 [ 1146.838060] Node 0 active_anon:1758640kB inactive_anon:166116kB active_file:2366556kB inactive_file:9229508kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:535840kB dirty:2000kB writeback:0kB shmem:207272kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 598016kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1146.838063] DMA free:15900kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1146.838064] lowmem_reserve[]: [ 1146.838065] 0 [ 1146.838067] 3030 [ 1146.838068] 15777 [ 1146.838069] 15777 [ 1146.838071] DMA32 free:1115072kB min:12968kB low:16208kB high:19448kB active_anon:250368kB inactive_anon:416kB active_file:48104kB inactive_file:1489456kB unevictable:0kB writepending:0kB present:3172224kB managed:3172224kB mlocked:0kB kernel_stack:16kB pagetables:228kB bounce:0kB free_pcp:11000kB local_pcp:1348kB free_cma:0kB [ 1146.838073] lowmem_reserve[]: [ 1146.838074] 0 [ 1146.838075] 0 [ 1146.838076] 12746 [ 1146.838077] 12746 [ 1146.838080] Normal free:112996kB min:54544kB low:68180kB high:81816kB active_anon:1508272kB inactive_anon:165700kB active_file:2318452kB inactive_file:7740052kB unevictable:4kB writepending:2000kB present:13367296kB managed:13056872kB mlocked:4kB kernel_stack:9220kB pagetables:20980kB bounce:0kB free_pcp:5472kB local_pcp:508kB free_cma:0kB [ 1146.838081] lowmem_reserve[]: [ 1146.838082] 0 [ 1146.838083] 0 [ 1146.838085] 0 [ 1146.838086] 0 [ 1146.838088] DMA: [ 1146.838089] 1*4kB [ 1146.838090] (U) [ 1146.838091] 1*8kB [ 1146.838092] (U) [ 1146.838093] 1*16kB [ 1146.838094] (U) [ 1146.838095] 0*32kB [ 1146.838097] 2*64kB [ 1146.838098] (U) [ 1146.838099] 1*128kB [ 1146.838100] (U) [ 1146.838101] 1*256kB [ 1146.838102] (U) [ 1146.838103] 0*512kB [ 1146.838104] 1*1024kB [ 1146.838106] (U) [ 1146.838107] 1*2048kB [ 1146.838279] (M) [ 1146.838282] 3*4096kB [ 1146.838284] (M) [ 1146.838287] = 15900kB [ 1146.838290] DMA32: [ 1146.838292] 6524*4kB [ 1146.838294] (UME) [ 1146.838295] 2692*8kB [ 1146.838297] (UME) [ 1146.838298] 1125*16kB [ 1146.838300] (UME) [ 1146.838301] 775*32kB [ 1146.838303] (UME) [ 1146.838304] 368*64kB [ 1146.838306] (UME) [ 1146.838307] 193*128kB [ 1146.838308] (UME) [ 1146.838310] 64*256kB [ 1146.838311] (UME) [ 1146.838312] 37*512kB [ 1146.838313] (UM) [ 1146.838315] 21*1024kB [ 1146.838316] (UME) [ 1146.838318] 195*2048kB [ 1146.838319] (UM) [ 1146.838321] 127*4096kB [ 1146.838322] (M) [ 1146.838323] = 1115072kB [ 1146.838324] Normal: [ 1146.838325] 1083*4kB [ 1146.838326] (UME) [ 1146.838327] 1426*8kB [ 1146.838328] (UM) [ 1146.838329] 493*16kB [ 1146.838330] (UE) [ 1146.838331] 359*32kB [ 1146.838333] (U) [ 1146.838334] 161*64kB [ 1146.838335] (UME) [ 1146.838336] 54*128kB [ 1146.838337] (UM) [ 1146.838339] 43*256kB [ 1146.838340] (UM) [ 1146.838341] 9*512kB [ 1146.838342] (UME) [ 1146.838343] 5*1024kB [ 1146.838345] (UME) [ 1146.838347] 5*2048kB [ 1146.838348] (ME) [ 1146.838349] 7*4096kB [ 1146.838350] (M) [ 1146.838351] = 111980kB [ 1146.838353] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1146.838355] 2950826 total pagecache pages [ 1146.838357] 0 pages in swap cache [ 1146.838358] Swap cache stats: add 0, delete 0, find 0/0 [ 1146.838360] Free swap = 4194300kB [ 1146.838362] Total swap = 4194300kB [ 1146.838363] 4138876 pages RAM [ 1146.838364] 0 pages HighMem/MovableOnly [ 1146.838365] 77627 pages reserved
From the output it looks like there was about 9G inactive file memory and no swap used at all when the OOM killer triggered. The output is messy because I use netconsole.
Swap is using a zram dev setup like this: echo 8 > /sys/block/zram0/max_comp_streams echo lz4 > /sys/block/zram0/comp_algorithm echo 4G > /sys/block/zram0/disksize mkswap /dev/zram0 swapon -v --discard /dev/zram0
Those kernel memory allocation failures can also cause kernel NULL pointer dereference. Here is a dmesg captured over netconsole when that happens:
4,1210,922134743,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1211,922134751,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1212,922134753,-; node 0: slabs: 18346, objs: 568698, free: 0 4,1213,922134762,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1214,922134764,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1215,922134765,-; node 0: slabs: 18346, objs: 568698, free: 0 4,1216,922134770,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1217,922134771,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1218,922134773,-; node 0: slabs: 18346, objs: 568698, free: 0 4,1219,922134776,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1220,922134778,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1221,922134779,-; node 0: slabs: 18346, objs: 568698, free: 0 4,1222,922134784,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1223,922134872,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1224,922134874,-; node 0: slabs: 18346, objs: 568698, free: 0 4,1225,922135143,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1226,922135147,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1227,922135148,-; node 0: slabs: 18351, objs: 568713, free: 0 4,1228,922135152,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1229,922135154,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1230,922135162,-; node 0: slabs: 18351, objs: 568713, free: 0 4,1231,922135165,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1232,922135166,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1233,922135166,-; node 0: slabs: 18351, objs: 568713, free: 0 4,1234,922135168,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1235,922135175,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1236,922135176,-; node 0: slabs: 18351, objs: 568713, free: 0 4,1237,922135181,-;SLUB: Unable to allocate memory on node -1, gfp=0x600040(GFP_NOFS) 4,1238,922135183,-; cache: ext4_inode_cache(100:12G), object size: 1024, buffer size: 1032, default order: 3, min order: 0 4,1239,922135183,-; node 0: slabs: 18351, objs: 568713, free: 0 1,1240,922137835,-;BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 6,1241,922137839,-;PGD 0 4,1242,922137840,c;P4D 0 4,1243,922137842,-;Oops: 0000 [#1] PREEMPT SMP PTI 4,1244,922137844,-;CPU: 3 PID: 16923 Comm: gtk-update-icon Not tainted 4.19.51 #42 4,1245,922137846,-;Hardware name: Gigabyte Technology Co., Ltd. Z97X-Gaming G1/Z97X-Gaming G1, BIOS F9 07/31/2015 4,1246,922137850,-;RIP: 0010:create_empty_buffers+0x24/0x100 4,1247,922137852,-;Code: cd 0f 1f 44 00 00 0f 1f 44 00 00 41 54 49 89 d4 ba 01 00 00 00 55 53 48 89 fb e8 97 fe ff ff 48 89 c5 48 89 c2 eb 03 48 89 ca <48> 8b 4a 08 4c 09 22 48 85 c9 75 f1 48 89 6a 08 48 8b 43 18 48 8d 4,1248,922137853,-;RSP: 0018:ffff927ac1b37bf8 EFLAGS: 00010286 4,1249,922137855,-;RAX: 0000000000000000 RBX: fffff2d4429fd740 RCX: 0000000100097149 4,1250,922137856,-;RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff9075a99fbe00 4,1251,922137857,-;RBP: 0000000000000000 R08: fffff2d440949cc8 R09: 00000000000960c0 4,1252,922137859,-;R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000000 4,1253,922137860,-;R13: ffff907601f18360 R14: 0000000000002000 R15: 0000000000001000 4,1254,922137861,-;FS: 00007fb55b288bc0(0000) GS:ffff90761f8c0000(0000) knlGS:0000000000000000 4,1255,922137863,-;CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 4,1256,922137865,-;CR2: 0000000000000008 CR3: 000000007aebc002 CR4: 00000000001606e0 4,1257,922137866,-;Call Trace: 4,1258,922137967,-; create_page_buffers+0x4d/0x60 4,1259,922137969,-; __block_write_begin_int+0x8e/0x5a0 4,1260,922137972,-; ? ext4_inode_attach_jinode.part.82+0xb0/0xb0 4,1261,922137975,-; ? jbd2__journal_start+0xd7/0x1f0 4,1262,922137977,-; ext4_da_write_begin+0x112/0x3d0 4,1263,922137980,-; generic_perform_write+0xf1/0x1b0 4,1264,922137983,-; ? file_update_time+0x70/0x140 4,1265,922137985,-; __generic_file_write_iter+0x141/0x1a0 4,1266,922137988,-; ext4_file_write_iter+0xef/0x3b0 4,1267,922137990,-; __vfs_write+0x17e/0x1e0 4,1268,922137992,-; vfs_write+0xa5/0x1a0 4,1269,922137994,-; ksys_write+0x57/0xd0 4,1270,922137997,-; do_syscall_64+0x55/0x160 4,1271,922138000,-; entry_SYSCALL_64_after_hwframe+0x44/0xa9 4,1272,922138003,-;RIP: 0033:0x7fb55ba9c0d8 4,1273,922138004,-;Code: 00 90 48 83 ec 38 64 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 48 8d 05 05 96 0d 00 8b 00 85 c0 75 27 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 60 48 8b 4c 24 28 64 48 33 0c 25 28 00 00 00 4,1274,922138006,-;RSP: 002b:00007fff718c1260 EFLAGS: 00000246 4,1275,922138007,c; ORIG_RAX: 0000000000000001 4,1276,922138124,-;RAX: ffffffffffffffda RBX: 0000000000001000 RCX: 00007fb55ba9c0d8 4,1277,922138125,-;RDX: 0000000000001000 RSI: 000056395ab78710 RDI: 0000000000000003 4,1278,922138126,-;RBP: 000056395ab78710 R08: 00007fb55b288bc0 R09: 0000000000000000 4,1279,922138128,-;R10: 0000000000000002 R11: 0000000000000246 R12: 000056395ab69900 4,1280,922138129,-;R13: 0000000000001000 R14: 00007fb55bb6c760 R15: 0000000000001000 4,1281,922138131,-;Modules linked in: 4,1282,922138133,c; 8021q 4,1283,922138134,c; iptable_mangle 4,1284,922138135,c; xt_limit 4,1285,922138136,c; xt_conntrack 4,1286,922138137,c; iptable_filter 4,1287,922138138,c; iptable_nat 4,1288,922138139,c; nf_nat_ipv4 4,1289,922138141,c; nf_nat 4,1290,922138142,c; ip_tables 4,1291,922138143,c; arc4 4,1292,922138144,c; ath9k_htc 4,1293,922138145,c; ath9k_common 4,1294,922138146,c; ath9k_hw 4,1295,922138147,c; ath 4,1296,922138148,c; mac80211 4,1297,922138150,c; kvm_intel 4,1298,922138151,c; cfg80211 4,1299,922138152,c; kvm 4,1300,922138153,c; uas 4,1301,922138154,c; crc32_pclmul 4,1302,922138156,c; usb_storage 4,1303,922138157,c; joydev 4,1304,922138201,c; cdc_acm 4,1305,922138203,-;CR2: 0000000000000008 4,1306,922138270,-;---[ end trace ee8624c121072f8e ]--- 4,1307,922138273,-;RIP: 0010:create_empty_buffers+0x24/0x100 4,1308,922138275,-;Code: cd 0f 1f 44 00 00 0f 1f 44 00 00 41 54 49 89 d4 ba 01 00 00 00 55 53 48 89 fb e8 97 fe ff ff 48 89 c5 48 89 c2 eb 03 48 89 ca <48> 8b 4a 08 4c 09 22 48 85 c9 75 f1 48 89 6a 08 48 8b 43 18 48 8d 4,1309,922138278,-;RSP: 0018:ffff927ac1b37bf8 EFLAGS: 00010286 4,1310,922138280,-;RAX: 0000000000000000 RBX: fffff2d4429fd740 RCX: 0000000100097149 4,1311,922138281,-;RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff9075a99fbe00 4,1312,922138282,-;RBP: 0000000000000000 R08: fffff2d440949cc8 R09: 00000000000960c0 4,1313,922138284,-;R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000000 4,1314,922138285,-;R13: ffff907601f18360 R14: 0000000000002000 R15: 0000000000001000 4,1315,922138286,-;FS: 00007fb55b288bc0(0000) GS:ffff90761f8c0000(0000) knlGS:0000000000000000 4,1316,922138288,-;CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 4,1317,922138289,-;CR2: 0000000000000008 CR3: 000000007aebc002 CR4: 00000000001606e0 0,1318,922138290,-;Kernel panic - not syncing: Fatal exception 0,1319,922138296,-;Kernel Offset: 0x6000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) 0,1320,922138299,-;---[ end Kernel panic - not syncing: Fatal exception ]---
Since I never had this problem with 4.14 I tested the old 4.14.115 that I used before updating to 4.19 and I couldn't reproduce the problem with it. I can easily reproduce the problem at least back to 4.19.42 in the 4.19 series. I could bisect the problem but that's going to take forever so I'm hoping I can avoid that.
The problem only occurs when that "12G" memory cgroups is used to build chromium and nothing else. Chromium is the largest package I regularly build and I selectively enable ccache for the chromium build. My gut feeling tells the that the massive number of file operations needed for the build is what is triggering the problem. Perhaps when the memory.kmem.limit_in_bytes limit is reached?
Here is some more info I don't think fit in a single mail.
full dmesg https://pastebin.com/raw/tKgTCTJ2 .config https://pastebin.com/raw/jKhSqqCX
Some relevant parts of /etc/sysctl.conf: vm.dirty_writeback_centisecs = 3000 vm.dirty_background_bytes = 52428800 vm.dirty_bytes = 262144000 vm.swappiness = 99 vm.vfs_cache_pressure = 25