[ 768.621699] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 768.627188] nipa-hw-worker: [5/14] Retrying drivers/net:napi_threaded.py
[ 768.755448] kselftest: Running tests in drivers/net
[ 989.765588] INFO: task python3:5512 blocked for more than 122 seconds.
[ 989.773791] Not tainted 7.1.0-rc7-clnv-g75e7aaa3b088 #1
[ 989.781361] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.790953] task:python3 state:D stack:25648 pid:5512 tgid:5512 ppid:5511 task_flags:0x400100 flags:0x00080000
[ 989.804162] Call Trace:
[ 989.807703]
[ 989.810854] __schedule+0xcc3/0x19c0
[ 989.815671] ? io_schedule_timeout+0x140/0x140
[ 989.821459] ? find_held_lock+0x2b/0x80
[ 989.826562] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.832441] schedule+0xd8/0x260
[ 989.836860] schedule_preempt_disabled+0x18/0x30
[ 989.842839] __mutex_lock+0x6af/0x2010
[ 989.847843] ? threaded_store+0x132/0x220
[ 989.853142] ? ww_mutex_lock+0x160/0x160
[ 989.858343] ? selinux_inode_free_security+0x330/0x330
[ 989.864913] ? cap_capable+0x1d6/0x3e0
[ 989.869923] ? threaded_store+0x132/0x220
[ 989.875218] threaded_store+0x132/0x220
[ 989.880319] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.885621] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.890922] ? sysfs_kf_write+0x65/0x160
[ 989.896124] ? sysfs_file_ops+0x140/0x140
[ 989.901427] kernfs_fop_write_iter+0x335/0x510
[ 989.907211] ? kernfs_ops+0xf0/0xf0
[ 989.911923] new_sync_write+0x333/0x750
[ 989.917026] ? __lock_acquire+0x508/0xc10
[ 989.922322] ? new_sync_read+0x740/0x740
[ 989.927527] ? lock_acquire.part.0+0xbc/0x260
[ 989.933215] ? ksys_write+0x116/0x250
[ 989.938124] vfs_write+0x657/0x9b0
[ 989.942742] ksys_write+0x116/0x250
[ 989.947456] ? __ia32_sys_read+0xc0/0xc0
[ 989.952654] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.958242] ? rcu_is_watching+0x15/0xd0
[ 989.963436] do_syscall_64+0xf3/0x650
[ 989.968342] ? irq_exit_rcu+0x1a/0x30
[ 989.973252] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.979722] RIP: 0033:0x7f639109164e
[ 989.984530] RSP: 002b:00007ffc08edc0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.993830] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f639109164e
[ 990.002638] RDX: 0000000000000001 RSI: 000055798794abf0 RDI: 0000000000000007
[ 990.011445] RBP: 00007ffc08edc0e0 R08: 0000000000000000 R09: 0000000000000000
[ 990.020254] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6390f0a390
[ 990.029062] R13: 000055798794abf0 R14: 0000000000000007 R15: 00007f6390e9d120
[ 990.037878]
[ 990.041159] INFO: task python3:5512 is blocked on a mutex likely owned by task python3:3481.
[ 990.051435] task:python3 state:D stack:25672 pid:3481 tgid:3481 ppid:3480 task_flags:0x400100 flags:0x00080802
[ 990.064637] Call Trace:
[ 990.068176]
[ 990.071328] __schedule+0xcc3/0x19c0
[ 990.076139] ? io_schedule_timeout+0x140/0x140
[ 990.081925] ? find_held_lock+0x2b/0x80
[ 990.087030] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.093208] schedule+0xd8/0x260
[ 990.097628] schedule_timeout+0x13f/0x290
[ 990.102916] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.109287] ? trace_contention_end+0xb3/0x1a0
[ 990.115075] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.120767] ? ww_mutex_lock+0x160/0x160
[ 990.125969] msleep+0x1b/0x30
[ 990.130094] napi_set_threaded+0x3a5/0x440
[ 990.135485] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.140980] netif_set_threaded+0x257/0x400
[ 990.146475] threaded_store+0x1b8/0x220
[ 990.151576] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.156876] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.162175] ? sysfs_kf_write+0x65/0x160
[ 990.167376] ? sysfs_file_ops+0x140/0x140
[ 990.172673] kernfs_fop_write_iter+0x335/0x510
[ 990.178458] ? kernfs_ops+0xf0/0xf0
[ 990.183168] new_sync_write+0x333/0x750
[ 990.188272] ? __lock_acquire+0x508/0xc10
[ 990.193567] ? new_sync_read+0x740/0x740
[ 990.198770] ? lock_acquire.part.0+0xbc/0x260
[ 990.204456] ? ksys_write+0x116/0x250
[ 990.209366] vfs_write+0x657/0x9b0
[ 990.213982] ksys_write+0x116/0x250
[ 990.218695] ? __ia32_sys_read+0xc0/0xc0
[ 990.223884] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.229476] ? rcu_is_watching+0x15/0xd0
[ 990.234676] do_syscall_64+0xf3/0x650
[ 990.239581] ? irq_exit_rcu+0x1a/0x30
[ 990.244488] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.250953] RIP: 0033:0x7f389444b64e
[ 990.255761] RSP: 002b:00007ffe74968b50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.265060] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f389444b64e
[ 990.273869] RDX: 0000000000000001 RSI: 000056140978abf0 RDI: 0000000000000007
[ 990.282680] RBP: 00007ffe74968b60 R08: 0000000000000000 R09: 0000000000000000
[ 990.291492] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f38942c4390
[ 990.300301] R13: 000056140978abf0 R14: 0000000000000007 R15: 00007f3894259120
[ 990.309116]
[ 990.312377] \x0aShowing all locks held in the system:
[ 990.320917] 1 lock held by khungtaskd/185:
[ 990.326313] #0: ffffffffaccfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.337787] 2 locks held by systemd-journal/808:
[ 990.343767] #0: ff1100180dc374e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.353655] #1: ffffffffacf02ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.366483] 4 locks held by python3/3481:
[ 990.371777] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.381667] #1: ff1100014c79c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.392629] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.403788] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.413883] 4 locks held by python3/5512:
[ 990.419179] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.429066] #1: ff110001ab2e1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.440030] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.451188] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.461275]
[ 990.463740] =============================================\x0a
[ 1112.646897] INFO: task python3:5512 blocked for more than 245 seconds.
[ 1112.655049] Not tainted 7.1.0-rc7-clnv-g75e7aaa3b088 #1
[ 1112.662300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.671893] task:python3 state:D stack:25648 pid:5512 tgid:5512 ppid:5511 task_flags:0x400100 flags:0x00080000
[ 1112.685103] Call Trace:
[ 1112.688643]
[ 1112.691797] __schedule+0xcc3/0x19c0
[ 1112.696612] ? io_schedule_timeout+0x140/0x140
[ 1112.702398] ? find_held_lock+0x2b/0x80
[ 1112.707505] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.713389] schedule+0xd8/0x260
[ 1112.717810] schedule_preempt_disabled+0x18/0x30
[ 1112.723788] __mutex_lock+0x6af/0x2010
[ 1112.728793] ? threaded_store+0x132/0x220
[ 1112.734093] ? ww_mutex_lock+0x160/0x160
[ 1112.739285] ? selinux_inode_free_security+0x330/0x330
[ 1112.745854] ? cap_capable+0x1d6/0x3e0
[ 1112.750864] ? threaded_store+0x132/0x220
[ 1112.756163] threaded_store+0x132/0x220
[ 1112.761267] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.766566] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.771867] ? sysfs_kf_write+0x65/0x160
[ 1112.777071] ? sysfs_file_ops+0x140/0x140
[ 1112.782365] kernfs_fop_write_iter+0x335/0x510
[ 1112.788152] ? kernfs_ops+0xf0/0xf0
[ 1112.792865] new_sync_write+0x333/0x750
[ 1112.797971] ? __lock_acquire+0x508/0xc10
[ 1112.803267] ? new_sync_read+0x740/0x740
[ 1112.808470] ? lock_acquire.part.0+0xbc/0x260
[ 1112.814156] ? ksys_write+0x116/0x250
[ 1112.819069] vfs_write+0x657/0x9b0
[ 1112.823682] ksys_write+0x116/0x250
[ 1112.828395] ? __ia32_sys_read+0xc0/0xc0
[ 1112.833596] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.839188] ? rcu_is_watching+0x15/0xd0
[ 1112.844389] do_syscall_64+0xf3/0x650
[ 1112.849294] ? irq_exit_rcu+0x1a/0x30
[ 1112.854202] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.860672] RIP: 0033:0x7f639109164e
[ 1112.865482] RSP: 002b:00007ffc08edc0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.874781] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f639109164e
[ 1112.883591] RDX: 0000000000000001 RSI: 000055798794abf0 RDI: 0000000000000007
[ 1112.892400] RBP: 00007ffc08edc0e0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.901209] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6390f0a390
[ 1112.910019] R13: 000055798794abf0 R14: 0000000000000007 R15: 00007f6390e9d120
[ 1112.918833]
[ 1112.922115] INFO: task python3:5512 is blocked on a mutex likely owned by task python3:3481.
[ 1112.932386] task:python3 state:D stack:25672 pid:3481 tgid:3481 ppid:3480 task_flags:0x400100 flags:0x00080802
[ 1112.945588] Call Trace:
[ 1112.949126]
[ 1112.952279] __schedule+0xcc3/0x19c0
[ 1112.957091] ? io_schedule_timeout+0x140/0x140
[ 1112.962872] ? find_held_lock+0x2b/0x80
[ 1112.967976] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.974154] schedule+0xd8/0x260
[ 1112.978572] schedule_timeout+0x13f/0x290
[ 1112.983867] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.990237] ? trace_contention_end+0xb3/0x1a0
[ 1112.996024] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1113.001707] ? ww_mutex_lock+0x160/0x160
[ 1113.006908] msleep+0x1b/0x30
[ 1113.011025] napi_set_threaded+0x3a5/0x440
[ 1113.016420] ? napi_resume_irqs+0x1a0/0x1a0
[ 1113.021918] netif_set_threaded+0x257/0x400
[ 1113.027412] threaded_store+0x1b8/0x220
[ 1113.032512] ? xps_rxqs_store+0x2e0/0x2e0
[ 1113.037809] ? sysfs_file_kobj+0xbd/0x1e0
[ 1113.043100] ? sysfs_kf_write+0x65/0x160
[ 1113.048300] ? sysfs_file_ops+0x140/0x140
[ 1113.053595] kernfs_fop_write_iter+0x335/0x510
[ 1113.059379] ? kernfs_ops+0xf0/0xf0
[ 1113.064091] new_sync_write+0x333/0x750
[ 1113.069193] ? __lock_acquire+0x508/0xc10
[ 1113.074489] ? new_sync_read+0x740/0x740
[ 1113.079690] ? lock_acquire.part.0+0xbc/0x260
[ 1113.085377] ? ksys_write+0x116/0x250
[ 1113.090286] vfs_write+0x657/0x9b0
[ 1113.094903] ksys_write+0x116/0x250
[ 1113.099615] ? __ia32_sys_read+0xc0/0xc0
[ 1113.104814] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.110406] ? rcu_is_watching+0x15/0xd0
[ 1113.115605] do_syscall_64+0xf3/0x650
[ 1113.120511] ? irq_exit_rcu+0x1a/0x30
[ 1113.125417] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.131885] RIP: 0033:0x7f389444b64e
[ 1113.136684] RSP: 002b:00007ffe74968b50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.145982] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f389444b64e
[ 1113.154793] RDX: 0000000000000001 RSI: 000056140978abf0 RDI: 0000000000000007
[ 1113.163603] RBP: 00007ffe74968b60 R08: 0000000000000000 R09: 0000000000000000
[ 1113.172414] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f38942c4390
[ 1113.181221] R13: 000056140978abf0 R14: 0000000000000007 R15: 00007f3894259120
[ 1113.190036]
[ 1113.193300] \x0aShowing all locks held in the system:
[ 1113.201844] 1 lock held by khungtaskd/185:
[ 1113.207240] #0: ffffffffaccfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.218725] 4 locks held by python3/3481:
[ 1113.224021] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.233912] #1: ff1100014c79c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.244875] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.256032] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.266121] 4 locks held by python3/5512:
[ 1113.271418] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.281306] #1: ff110001ab2e1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.292268] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.303425] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.313517]
[ 1113.315984] =============================================\x0a
[ 1210.951929] kworker/u130:2 (347) used greatest stack depth: 21456 bytes left
[ 1235.528184] INFO: task python3:5512 blocked for more than 368 seconds.
[ 1235.536330] Not tainted 7.1.0-rc7-clnv-g75e7aaa3b088 #1
[ 1235.543580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.553167] task:python3 state:D stack:25648 pid:5512 tgid:5512 ppid:5511 task_flags:0x400100 flags:0x00080002
[ 1235.566372] Call Trace:
[ 1235.569913]
[ 1235.573065] __schedule+0xcc3/0x19c0
[ 1235.577879] ? io_schedule_timeout+0x140/0x140
[ 1235.583664] ? find_held_lock+0x2b/0x80
[ 1235.588765] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.594649] schedule+0xd8/0x260
[ 1235.599065] schedule_preempt_disabled+0x18/0x30
[ 1235.605043] __mutex_lock+0x6af/0x2010
[ 1235.610046] ? threaded_store+0x132/0x220
[ 1235.615345] ? ww_mutex_lock+0x160/0x160
[ 1235.620545] ? selinux_inode_free_security+0x330/0x330
[ 1235.627114] ? cap_capable+0x1d6/0x3e0
[ 1235.632123] ? threaded_store+0x132/0x220
[ 1235.637417] threaded_store+0x132/0x220
[ 1235.642516] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.647811] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.653111] ? sysfs_kf_write+0x65/0x160
[ 1235.658305] ? sysfs_file_ops+0x140/0x140
[ 1235.663602] kernfs_fop_write_iter+0x335/0x510
[ 1235.669386] ? kernfs_ops+0xf0/0xf0
[ 1235.674099] new_sync_write+0x333/0x750
[ 1235.679200] ? __lock_acquire+0x508/0xc10
[ 1235.684496] ? new_sync_read+0x740/0x740
[ 1235.689694] ? lock_acquire.part.0+0xbc/0x260
[ 1235.695381] ? ksys_write+0x116/0x250
[ 1235.700291] vfs_write+0x657/0x9b0
[ 1235.704903] ksys_write+0x116/0x250
[ 1235.709611] ? __ia32_sys_read+0xc0/0xc0
[ 1235.714809] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.720395] ? rcu_is_watching+0x15/0xd0
[ 1235.725597] do_syscall_64+0xf3/0x650
[ 1235.730503] ? irq_exit_rcu+0x1a/0x30
[ 1235.735412] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.741870] RIP: 0033:0x7f639109164e
[ 1235.746671] RSP: 002b:00007ffc08edc0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.755961] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f639109164e
[ 1235.764767] RDX: 0000000000000001 RSI: 000055798794abf0 RDI: 0000000000000007
[ 1235.773574] RBP: 00007ffc08edc0e0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.782379] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6390f0a390
[ 1235.791186] R13: 000055798794abf0 R14: 0000000000000007 R15: 00007f6390e9d120
[ 1235.799998]
[ 1235.803283] INFO: task python3:5512 is blocked on a mutex likely owned by task python3:3481.
[ 1235.813554] task:python3 state:D stack:25672 pid:3481 tgid:3481 ppid:3480 task_flags:0x400100 flags:0x00080802
[ 1235.826755] Call Trace:
[ 1235.830293]
[ 1235.833442] __schedule+0xcc3/0x19c0
[ 1235.838249] ? io_schedule_timeout+0x140/0x140
[ 1235.844033] ? find_held_lock+0x2b/0x80
[ 1235.849134] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.855301] schedule+0xd8/0x260
[ 1235.859716] schedule_timeout+0x13f/0x290
[ 1235.865013] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.871381] ? trace_contention_end+0xb3/0x1a0
[ 1235.877164] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.882853] ? ww_mutex_lock+0x160/0x160
[ 1235.888050] msleep+0x1b/0x30
[ 1235.892173] napi_set_threaded+0x3a5/0x440
[ 1235.897569] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.903064] netif_set_threaded+0x257/0x400
[ 1235.908556] threaded_store+0x1b8/0x220
[ 1235.913658] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.918944] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.924240] ? sysfs_kf_write+0x65/0x160
[ 1235.929439] ? sysfs_file_ops+0x140/0x140
[ 1235.934732] kernfs_fop_write_iter+0x335/0x510
[ 1235.940518] ? kernfs_ops+0xf0/0xf0
[ 1235.945226] new_sync_write+0x333/0x750
[ 1235.950326] ? __lock_acquire+0x508/0xc10
[ 1235.955620] ? new_sync_read+0x740/0x740
[ 1235.960818] ? lock_acquire.part.0+0xbc/0x260
[ 1235.966503] ? ksys_write+0x116/0x250
[ 1235.971413] vfs_write+0x657/0x9b0
[ 1235.976030] ksys_write+0x116/0x250
[ 1235.980741] ? __ia32_sys_read+0xc0/0xc0
[ 1235.985941] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.991520] ? rcu_is_watching+0x15/0xd0
[ 1235.996717] do_syscall_64+0xf3/0x650
[ 1236.001622] ? irq_exit_rcu+0x1a/0x30
[ 1236.006526] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1236.012993] RIP: 0033:0x7f389444b64e
[ 1236.017801] RSP: 002b:00007ffe74968b50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1236.027096] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f389444b64e
[ 1236.035904] RDX: 0000000000000001 RSI: 000056140978abf0 RDI: 0000000000000007
[ 1236.044702] RBP: 00007ffe74968b60 R08: 0000000000000000 R09: 0000000000000000
[ 1236.053509] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f38942c4390
[ 1236.062315] R13: 000056140978abf0 R14: 0000000000000007 R15: 00007f3894259120
[ 1236.071125]
[ 1236.074390] \x0aShowing all locks held in the system:
[ 1236.082930] 1 lock held by khungtaskd/185:
[ 1236.088323] #0: ffffffffaccfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.099804] 4 locks held by python3/3481:
[ 1236.105097] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.114986] #1: ff1100014c79c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.125947] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.137102] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.147197] 4 locks held by python3/5512:
[ 1236.152492] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.162377] #1: ff110001ab2e1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.173335] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.184492] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.194586]
[ 1236.197051] =============================================\x0a
[ 1358.409438] INFO: task python3:5512 blocked for more than 491 seconds.
[ 1358.417585] Not tainted 7.1.0-rc7-clnv-g75e7aaa3b088 #1
[ 1358.424837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.434426] task:python3 state:D stack:25648 pid:5512 tgid:5512 ppid:5511 task_flags:0x400100 flags:0x00080002
[ 1358.447624] Call Trace:
[ 1358.451165]
[ 1358.454316] __schedule+0xcc3/0x19c0
[ 1358.459132] ? io_schedule_timeout+0x140/0x140
[ 1358.464918] ? find_held_lock+0x2b/0x80
[ 1358.470021] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.475907] schedule+0xd8/0x260
[ 1358.480325] schedule_preempt_disabled+0x18/0x30
[ 1358.486303] __mutex_lock+0x6af/0x2010
[ 1358.491308] ? threaded_store+0x132/0x220
[ 1358.496607] ? ww_mutex_lock+0x160/0x160
[ 1358.501807] ? selinux_inode_free_security+0x330/0x330
[ 1358.508378] ? cap_capable+0x1d6/0x3e0
[ 1358.513384] ? threaded_store+0x132/0x220
[ 1358.518679] threaded_store+0x132/0x220
[ 1358.523782] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.529082] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.534383] ? sysfs_kf_write+0x65/0x160
[ 1358.539584] ? sysfs_file_ops+0x140/0x140
[ 1358.544878] kernfs_fop_write_iter+0x335/0x510
[ 1358.550662] ? kernfs_ops+0xf0/0xf0
[ 1358.555375] new_sync_write+0x333/0x750
[ 1358.560478] ? __lock_acquire+0x508/0xc10
[ 1358.565776] ? new_sync_read+0x740/0x740
[ 1358.570980] ? lock_acquire.part.0+0xbc/0x260
[ 1358.576667] ? ksys_write+0x116/0x250
[ 1358.581577] vfs_write+0x657/0x9b0
[ 1358.586190] ksys_write+0x116/0x250
[ 1358.590904] ? __ia32_sys_read+0xc0/0xc0
[ 1358.596102] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.601694] ? rcu_is_watching+0x15/0xd0
[ 1358.606898] do_syscall_64+0xf3/0x650
[ 1358.611803] ? irq_exit_rcu+0x1a/0x30
[ 1358.616712] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.623179] RIP: 0033:0x7f639109164e
[ 1358.627990] RSP: 002b:00007ffc08edc0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.637294] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f639109164e
[ 1358.646105] RDX: 0000000000000001 RSI: 000055798794abf0 RDI: 0000000000000007
[ 1358.654914] RBP: 00007ffc08edc0e0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.663724] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6390f0a390
[ 1358.672530] R13: 000055798794abf0 R14: 0000000000000007 R15: 00007f6390e9d120
[ 1358.681343]
[ 1358.684618] INFO: task python3:5512 is blocked on a mutex likely owned by task python3:3481.
[ 1358.694889] task:python3 state:D stack:25672 pid:3481 tgid:3481 ppid:3480 task_flags:0x400100 flags:0x00080802
[ 1358.708094] Call Trace:
[ 1358.711633]
[ 1358.714785] __schedule+0xcc3/0x19c0
[ 1358.719596] ? io_schedule_timeout+0x140/0x140
[ 1358.725381] ? find_held_lock+0x2b/0x80
[ 1358.730477] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.736659] schedule+0xd8/0x260
[ 1358.741078] schedule_timeout+0x13f/0x290
[ 1358.746365] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.752733] ? trace_contention_end+0xb3/0x1a0
[ 1358.758517] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.764205] ? ww_mutex_lock+0x160/0x160
[ 1358.769405] msleep+0x1b/0x30
[ 1358.773532] napi_set_threaded+0x3a5/0x440
[ 1358.778928] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.784426] netif_set_threaded+0x257/0x400
[ 1358.789920] threaded_store+0x1b8/0x220
[ 1358.795024] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.800322] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.805619] ? sysfs_kf_write+0x65/0x160
[ 1358.810820] ? sysfs_file_ops+0x140/0x140
[ 1358.816117] kernfs_fop_write_iter+0x335/0x510
[ 1358.821901] ? kernfs_ops+0xf0/0xf0
[ 1358.826612] new_sync_write+0x333/0x750
[ 1358.831716] ? __lock_acquire+0x508/0xc10
[ 1358.837012] ? new_sync_read+0x740/0x740
[ 1358.842215] ? lock_acquire.part.0+0xbc/0x260
[ 1358.847902] ? ksys_write+0x116/0x250
[ 1358.852806] vfs_write+0x657/0x9b0
[ 1358.857422] ksys_write+0x116/0x250
[ 1358.862132] ? __ia32_sys_read+0xc0/0xc0
[ 1358.867330] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.872918] ? rcu_is_watching+0x15/0xd0
[ 1358.878121] do_syscall_64+0xf3/0x650
[ 1358.883025] ? irq_exit_rcu+0x1a/0x30
[ 1358.887934] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.894403] RIP: 0033:0x7f389444b64e
[ 1358.899213] RSP: 002b:00007ffe74968b50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.908511] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f389444b64e
[ 1358.917316] RDX: 0000000000000001 RSI: 000056140978abf0 RDI: 0000000000000007
[ 1358.926126] RBP: 00007ffe74968b60 R08: 0000000000000000 R09: 0000000000000000
[ 1358.934933] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f38942c4390
[ 1358.943733] R13: 000056140978abf0 R14: 0000000000000007 R15: 00007f3894259120
[ 1358.952549]
[ 1358.955822] \x0aShowing all locks held in the system:
[ 1358.964363] 1 lock held by khungtaskd/185:
[ 1358.969755] #0: ffffffffaccfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.981236] 4 locks held by python3/3481:
[ 1358.986550] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.996441] #1: ff1100014c79c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.007405] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.018563] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.028658] 4 locks held by python3/5512:
[ 1359.033953] #0: ff1100012f7b13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.043842] #1: ff110001ab2e1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.054795] #2: ff110001121deb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.065947] #3: ff110001220e9e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.076043]
[ 1359.078510] =============================================\x0a