On Sat, 7 Jan 2023 at 05:33, Jakub Kicinski kuba@kernel.org wrote:
On Thu, 5 Jan 2023 20:19:27 +0530 Naresh Kamboju wrote:
Following kernel warnings noticed on arm beagleboard X15 device while running selftests: tc-testing: tdc.sh with stable-rc 6.1.
This is always reproducible with kselftest merge configs. The build, config, vmlinux and test details links provided [1].
[ 228.686798] WARNING: inconsistent lock state [ 228.193450] WARNING: CPU: 1 PID: 2386 at include/linux/u64_stats_sync.h:145 __u64_stats_update_begin+0x180/0x1a4 [sch_gred] [ 228.439208] WARNING: CPU: 1 PID: 2386 at include/linux/seqlock.h:269 __u64_stats_update_begin+0x1a0/0x1a4 [sch_gred
commit 44b0c2957adc ("u64_stats: Streamline the implementation") looks like the likely culprit. Would you be able to test a revert?
As per your suggestion I have reverted 44b0c2957adc ("u64_stats: Streamline the implementation")
And test shows below warning (seqlock.h:269 gred_dump+0x60c/0x880 [sch_gred])
test log: ----------- # selftests: tc-testing: tdc.sh # considering category actions # -- buildebpf/SubPlugin.__init__ # -- ns/SubPlugin.__init__ # Unable to import the scapy python module. # # If not already installed, you may do so with: # pip3 install scapy==2.4.2 [ 72.334625] IPv6: ADDRCONF(NETDEV_CHANGE): v0p1: link becomes ready [ 72.342132] IPv6: ADDRCONF(NETDEV_CHANGE): v0p0: link becomes ready [ 92.637207] ------------[ cut here ]------------ [ 92.641876] WARNING: CPU: 0 PID: 687 at include/linux/seqlock.h:269 gred_dump+0x60c/0x880 [sch_gred] [ 92.651153] Modules linked in: sch_gred sch_multiq sch_cake veth netdevsim psample cfg80211 bluetooth snd_soc_simple_card etnaviv snd_soc_simple_card_utils gpu_sched onboard_usb_hub snd_soc_davinci_mcasp snd_soc_ti_udma snd_soc_ti_edma snd_soc_ti_sdma snd_soc_core ac97_bus snd_pcm_dmaengine snd_pcm snd_timer snd soundcore display_connector sch_fq_codel fuse [ 92.683319] CPU: 0 PID: 687 Comm: tc Not tainted 6.1.5-rc1 #1 [ 92.689117] Hardware name: Generic DRA74X (Flattened Device Tree) [ 92.695251] unwind_backtrace from show_stack+0x18/0x1c [ 92.700500] show_stack from dump_stack_lvl+0x58/0x70 [ 92.705596] dump_stack_lvl from __warn+0xd0/0x1f0 [ 92.710449] __warn from warn_slowpath_fmt+0x64/0xc8 [ 92.715454] warn_slowpath_fmt from gred_dump+0x60c/0x880 [sch_gred] [ 92.721862] gred_dump [sch_gred] from tc_fill_qdisc+0x154/0x44c [ 92.727905] tc_fill_qdisc from qdisc_notify+0x11c/0x130 [ 92.733245] qdisc_notify from qdisc_graft+0x440/0x624 [ 92.738433] qdisc_graft from tc_modify_qdisc+0x558/0x850 [ 92.743865] tc_modify_qdisc from rtnetlink_rcv_msg+0x180/0x56c [ 92.749816] rtnetlink_rcv_msg from netlink_rcv_skb+0xc0/0x118 [ 92.755706] netlink_rcv_skb from netlink_unicast+0x19c/0x268 [ 92.761474] netlink_unicast from netlink_sendmsg+0x1f8/0x484 [ 92.767272] netlink_sendmsg from ____sys_sendmsg+0x224/0x2bc [ 92.773040] ____sys_sendmsg from ___sys_sendmsg+0x70/0x9c [ 92.778594] ___sys_sendmsg from sys_sendmsg+0x54/0x90 [ 92.783752] sys_sendmsg from ret_fast_syscall+0x0/0x1c [ 92.789031] Exception stack(0xf10f5fa8 to 0xf10f5ff0) [ 92.794128] 5fa0: 00000000 00000001 00000003 be92cbdc 00000000 00000000 [ 92.802337] 5fc0: 00000000 00000001 b6f44800 00000128 63bfdcd4 00000000 00000000 00076000 [ 92.810577] 5fe0: 00000128 be92cb78 b6dbf253 b6d31ae6 [ 92.815734] irq event stamp: 25413 [ 92.819213] hardirqs last enabled at (25423): [<c03da980>] __up_console_sem+0x58/0x68 [ 92.827239] hardirqs last disabled at (25432): [<c03da96c>] __up_console_sem+0x44/0x68 [ 92.835235] softirqs last enabled at (25412): [<c0301fc8>] __do_softirq+0x300/0x538 [ 92.843078] softirqs last disabled at (25393): [<c035a224>] __irq_exit_rcu+0x14c/0x170 [ 92.851196] ---[ end trace 0000000000000000 ]--- [ 92.855865] [ 92.857360] ================================ [ 92.861663] WARNING: inconsistent lock state [ 92.865936] 6.1.5-rc1 #1 Tainted: G W [ 92.871032] -------------------------------- [ 92.875305] inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage. [ 92.881347] tc/687 [HC0[0]:SC0[0]:HE1:SE1] takes: [ 92.886077] c9d31cb4 (&(&b->syncp)->seq){+.?.}-{0:0}, at: tc_fill_qdisc+0x154/0x44c [ 92.893798] {IN-SOFTIRQ-W} state was registered at: [ 92.898681] __dev_queue_xmit+0xf68/0x1514 [ 92.902893] __neigh_update+0x2b0/0xe78 [ 92.906860] neigh_update+0x24/0x2c [ 92.910430] arp_process+0x264/0x95c [ 92.914123] __netif_receive_skb_one_core+0x58/0x74 [ 92.919128] netif_receive_skb+0xe4/0x474 [ 92.923248] cpsw_rx_handler+0x1a0/0x42c [ 92.927276] __cpdma_chan_process+0xf4/0x188 [ 92.931671] cpdma_chan_process+0x44/0x5c [ 92.935791] cpsw_rx_mq_poll+0x4c/0x9c [ 92.939636] __napi_poll+0x3c/0x28c [ 92.943237] net_rx_action+0x160/0x350 [ 92.947082] __do_softirq+0x130/0x538 [ 92.950866] __irq_exit_rcu+0x14c/0x170 [ 92.954803] irq_exit+0x10/0x30 [ 92.958038] call_with_stack+0x18/0x20 [ 92.961914] __irq_svc+0x9c/0xb8 [ 92.965240] lock_acquire+0x12c/0x364 [ 92.969024] __might_fault+0x5c/0x7c [ 92.972717] __rseq_handle_notify_resume+0x60/0x6e0 [ 92.977691] do_work_pending+0x10c/0x57c [ 92.981719] slow_work_pending+0xc/0x20 [ 92.985687] irq event stamp: 25499 [ 92.989074] hardirqs last enabled at (25499): [<c03da980>] __up_console_sem+0x58/0x68 [ 92.997039] hardirqs last disabled at (25498): [<c03da96c>] __up_console_sem+0x44/0x68 [ 93.005004] softirqs last enabled at (25488): [<c0301fc8>] __do_softirq+0x300/0x538 [ 93.012786] softirqs last disabled at (25461): [<c035a224>] __irq_exit_rcu+0x14c/0x170 [ 93.020751] [ 93.020751] other info that might help us debug this: [ 93.027313] Possible unsafe locking scenario: [ 93.027313] [ 93.033264] CPU0 [ 93.035705] ---- [ 93.038177] lock(&(&b->syncp)->seq); [ 93.041931] <Interrupt> [ 93.044586] lock(&(&b->syncp)->seq); [ 93.048522] [ 93.048522] *** DEADLOCK *** [ 93.048522] [ 93.054473] 1 lock held by tc/687: [ 93.057891] #0: c25b0720 (rtnl_mutex){+.+.}-{3:3}, at: qdisc_create+0x3cc/0x5bc [ 93.065338] [ 93.065338] stack backtrace: [ 93.069732] CPU: 0 PID: 687 Comm: tc Tainted: G W 6.1.5-rc1 #1 [ 93.076995] Hardware name: Generic DRA74X (Flattened Device Tree) [ 93.083099] unwind_backtrace from show_stack+0x18/0x1c [ 93.088378] show_stack from dump_stack_lvl+0x58/0x70 [ 93.093444] dump_stack_lvl from mark_lock.part.0+0xb74/0x128c [ 93.099334] mark_lock.part.0 from __lock_acquire+0x984/0x2a8c [ 93.105194] __lock_acquire from lock_acquire+0x110/0x364 [ 93.110626] lock_acquire from gred_dump+0x20c/0x880 [sch_gred] [ 93.116577] gred_dump [sch_gred] from tc_fill_qdisc+0x154/0x44c [ 93.122619] tc_fill_qdisc from qdisc_notify+0x11c/0x130 [ 93.127960] qdisc_notify from qdisc_graft+0x440/0x624 [ 93.133117] qdisc_graft from tc_modify_qdisc+0x558/0x850 [ 93.138549] tc_modify_qdisc from rtnetlink_rcv_msg+0x180/0x56c [ 93.144500] rtnetlink_rcv_msg from netlink_rcv_skb+0xc0/0x118 [ 93.150390] netlink_rcv_skb from netlink_unicast+0x19c/0x268 [ 93.156158] netlink_unicast from netlink_sendmsg+0x1f8/0x484 [ 93.161926] netlink_sendmsg from ____sys_sendmsg+0x224/0x2bc [ 93.167724] ____sys_sendmsg from ___sys_sendmsg+0x70/0x9c [ 93.173217] ___sys_sendmsg from sys_sendmsg+0x54/0x90 [ 93.178405] sys_sendmsg from ret_fast_syscall+0x0/0x1c [ 93.183654] Exception stack(0xf10f5fa8 to 0xf10f5ff0) [ 93.188751] 5fa0: 00000000 00000001 00000003 be92cbdc 00000000 00000000 [ 93.196960] 5fc0: 00000000 00000001 b6f44800 00000128 63bfdcd4 00000000 00000000 00076000 [ 93.205169] 5fe0: 00000128 be92cb78 b6dbf253 b6d31ae6 [ 96.129364] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 96.508056] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 96.892181] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 97.278503] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 97.660583] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 98.038024] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 98.428039] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 98.495635] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 98.901245] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) ! [ 98.989288] sch_tbf: burst 1500 is lower than device dummy1 mtu (1514) !
https://lkft.validation.linaro.org/scheduler/job/6047355#L2951
- Naresh