[ 763.307337] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 763.309927] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 763.487795] kselftest: Running tests in drivers/net
[ 989.698623] INFO: task python3:5465 blocked for more than 122 seconds.
[ 989.706851] Not tainted 7.1.0-rc7-ckzo-gcfdb3b4a9410 #1
[ 989.714106] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.724350] task:python3 state:D stack:25672 pid:5465 tgid:5465 ppid:5464 task_flags:0x400100 flags:0x00080000
[ 989.737567] Call Trace:
[ 989.741111]
[ 989.744259] __schedule+0xcc3/0x19c0
[ 989.749164] ? io_schedule_timeout+0x140/0x140
[ 989.754951] ? find_held_lock+0x2b/0x80
[ 989.760059] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.765951] schedule+0xd8/0x260
[ 989.770440] schedule_preempt_disabled+0x18/0x30
[ 989.776422] __mutex_lock+0x6af/0x2010
[ 989.781432] ? threaded_store+0x132/0x220
[ 989.786737] ? ww_mutex_lock+0x160/0x160
[ 989.791942] ? selinux_inode_free_security+0x330/0x330
[ 989.798522] ? cap_capable+0x1d6/0x3e0
[ 989.803534] ? threaded_store+0x132/0x220
[ 989.808836] threaded_store+0x132/0x220
[ 989.813946] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.819251] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.824552] ? sysfs_kf_write+0x65/0x160
[ 989.829755] ? sysfs_file_ops+0x140/0x140
[ 989.835057] kernfs_fop_write_iter+0x335/0x510
[ 989.840849] ? kernfs_ops+0xf0/0xf0
[ 989.845563] new_sync_write+0x333/0x750
[ 989.850674] ? __lock_acquire+0x508/0xc10
[ 989.855977] ? new_sync_read+0x740/0x740
[ 989.861294] ? lock_acquire.part.0+0xbc/0x260
[ 989.866986] ? ksys_write+0x116/0x250
[ 989.871903] vfs_write+0x657/0x9b0
[ 989.876520] ksys_write+0x116/0x250
[ 989.881236] ? __ia32_sys_read+0xc0/0xc0
[ 989.886439] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.892033] ? rcu_is_watching+0x15/0xd0
[ 989.897237] do_syscall_64+0xf3/0x650
[ 989.902146] ? trace_hardirqs_off+0xd/0x30
[ 989.907546] ? exc_page_fault+0xda/0xf0
[ 989.912655] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.919118] RIP: 0033:0x7fa5624b264e
[ 989.923946] RSP: 002b:00007ffe0ffa36a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.933251] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa5624b264e
[ 989.942064] RDX: 0000000000000001 RSI: 00005577400f9bf0 RDI: 0000000000000007
[ 989.950873] RBP: 00007ffe0ffa36b0 R08: 0000000000000000 R09: 0000000000000000
[ 989.959690] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa56232b390
[ 989.968503] R13: 00005577400f9bf0 R14: 0000000000000007 R15: 00007fa5622c1120
[ 989.977327]
[ 989.980615] INFO: task python3:5465 is blocked on a mutex likely owned by task python3:3479.
[ 989.990894] task:python3 state:D stack:25672 pid:3479 tgid:3479 ppid:3478 task_flags:0x400100 flags:0x00080802
[ 990.004100] Call Trace:
[ 990.007641]
[ 990.010794] __schedule+0xcc3/0x19c0
[ 990.015607] ? io_schedule_timeout+0x140/0x140
[ 990.021394] ? find_held_lock+0x2b/0x80
[ 990.026499] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.032677] schedule+0xd8/0x260
[ 990.037096] schedule_timeout+0x13f/0x290
[ 990.042391] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.048760] ? trace_contention_end+0xb3/0x1a0
[ 990.054544] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.060233] ? ww_mutex_lock+0x160/0x160
[ 990.065437] msleep+0x1b/0x30
[ 990.069560] napi_set_threaded+0x3a5/0x440
[ 990.074957] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.080459] netif_set_threaded+0x257/0x400
[ 990.085952] threaded_store+0x1b8/0x220
[ 990.091057] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.096353] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.101649] ? sysfs_kf_write+0x65/0x160
[ 990.106848] ? sysfs_file_ops+0x140/0x140
[ 990.112143] kernfs_fop_write_iter+0x335/0x510
[ 990.117931] ? kernfs_ops+0xf0/0xf0
[ 990.122644] new_sync_write+0x333/0x750
[ 990.127746] ? __lock_acquire+0x508/0xc10
[ 990.133044] ? new_sync_read+0x740/0x740
[ 990.138244] ? lock_acquire.part.0+0xbc/0x260
[ 990.143923] ? ksys_write+0x116/0x250
[ 990.148824] vfs_write+0x657/0x9b0
[ 990.153439] ksys_write+0x116/0x250
[ 990.158153] ? __ia32_sys_read+0xc0/0xc0
[ 990.163353] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.168944] ? rcu_is_watching+0x15/0xd0
[ 990.174143] do_syscall_64+0xf3/0x650
[ 990.179052] ? trace_hardirqs_off+0xd/0x30
[ 990.184444] ? exc_page_fault+0xda/0xf0
[ 990.189545] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.196012] RIP: 0033:0x7f1238da864e
[ 990.200819] RSP: 002b:00007ffcd5bef040 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.210114] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1238da864e
[ 990.218921] RDX: 0000000000000001 RSI: 000056518cbadbf0 RDI: 0000000000000007
[ 990.227729] RBP: 00007ffcd5bef050 R08: 0000000000000000 R09: 0000000000000000
[ 990.236535] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1238c21390
[ 990.245341] R13: 000056518cbadbf0 R14: 0000000000000007 R15: 00007f1238bb5120
[ 990.254153]
[ 990.257413] \x0aShowing all locks held in the system:
[ 990.265950] 1 lock held by khungtaskd/186:
[ 990.271342] #0: ffffffffb1afdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.282825] 4 locks held by python3/3479:
[ 990.288113] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.298005] #1: ff110001c1538880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.308970] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.320128] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.330220] 4 locks held by python3/5465:
[ 990.335518] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.345407] #1: ff110001832e8280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.356367] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.367528] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.377616] 5 locks held by systemd/6190:
[ 990.382910] #0: ff110001310f13e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.392799] #1: ff1100019f342c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.403753] #2: ffffffffb1b69c20 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x640
[ 990.415397] #3: ffffffffb19687b0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 990.426945] #4: ffffffffb1b69990 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 990.439273]
[ 990.441738] =============================================\x0a
[ 1112.579575] INFO: task python3:5465 blocked for more than 245 seconds.
[ 1112.587722] Not tainted 7.1.0-rc7-ckzo-gcfdb3b4a9410 #1
[ 1112.594972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.604563] task:python3 state:D stack:25672 pid:5465 tgid:5465 ppid:5464 task_flags:0x400100 flags:0x00080000
[ 1112.617773] Call Trace:
[ 1112.621313]
[ 1112.624462] __schedule+0xcc3/0x19c0
[ 1112.629281] ? io_schedule_timeout+0x140/0x140
[ 1112.635067] ? find_held_lock+0x2b/0x80
[ 1112.640172] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.646061] schedule+0xd8/0x260
[ 1112.650481] schedule_preempt_disabled+0x18/0x30
[ 1112.656461] __mutex_lock+0x6af/0x2010
[ 1112.661468] ? threaded_store+0x132/0x220
[ 1112.666766] ? ww_mutex_lock+0x160/0x160
[ 1112.671966] ? selinux_inode_free_security+0x330/0x330
[ 1112.678537] ? cap_capable+0x1d6/0x3e0
[ 1112.683546] ? threaded_store+0x132/0x220
[ 1112.688841] threaded_store+0x132/0x220
[ 1112.693942] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.699241] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.704541] ? sysfs_kf_write+0x65/0x160
[ 1112.709743] ? sysfs_file_ops+0x140/0x140
[ 1112.715037] kernfs_fop_write_iter+0x335/0x510
[ 1112.720823] ? kernfs_ops+0xf0/0xf0
[ 1112.725537] new_sync_write+0x333/0x750
[ 1112.730642] ? __lock_acquire+0x508/0xc10
[ 1112.735937] ? new_sync_read+0x740/0x740
[ 1112.741139] ? lock_acquire.part.0+0xbc/0x260
[ 1112.746824] ? ksys_write+0x116/0x250
[ 1112.751736] vfs_write+0x657/0x9b0
[ 1112.756352] ksys_write+0x116/0x250
[ 1112.761063] ? __ia32_sys_read+0xc0/0xc0
[ 1112.766264] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.771852] ? rcu_is_watching+0x15/0xd0
[ 1112.777057] do_syscall_64+0xf3/0x650
[ 1112.781963] ? trace_hardirqs_off+0xd/0x30
[ 1112.787357] ? exc_page_fault+0xda/0xf0
[ 1112.792461] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.798928] RIP: 0033:0x7fa5624b264e
[ 1112.803737] RSP: 002b:00007ffe0ffa36a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.813034] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa5624b264e
[ 1112.821842] RDX: 0000000000000001 RSI: 00005577400f9bf0 RDI: 0000000000000007
[ 1112.830652] RBP: 00007ffe0ffa36b0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.839461] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa56232b390
[ 1112.848270] R13: 00005577400f9bf0 R14: 0000000000000007 R15: 00007fa5622c1120
[ 1112.857082]
[ 1112.860347] INFO: task python3:5465 is blocked on a mutex likely owned by task python3:3479.
[ 1112.870621] task:python3 state:D stack:25672 pid:3479 tgid:3479 ppid:3478 task_flags:0x400100 flags:0x00080802
[ 1112.883825] Call Trace:
[ 1112.887363]
[ 1112.890513] __schedule+0xcc3/0x19c0
[ 1112.895324] ? io_schedule_timeout+0x140/0x140
[ 1112.901111] ? find_held_lock+0x2b/0x80
[ 1112.906215] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.912393] schedule+0xd8/0x260
[ 1112.916814] schedule_timeout+0x13f/0x290
[ 1112.922102] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.928470] ? trace_contention_end+0xb3/0x1a0
[ 1112.934255] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.939937] ? ww_mutex_lock+0x160/0x160
[ 1112.945138] msleep+0x1b/0x30
[ 1112.949253] napi_set_threaded+0x3a5/0x440
[ 1112.954649] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.960134] netif_set_threaded+0x257/0x400
[ 1112.965627] threaded_store+0x1b8/0x220
[ 1112.970729] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.976025] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.981320] ? sysfs_kf_write+0x65/0x160
[ 1112.986519] ? sysfs_file_ops+0x140/0x140
[ 1112.991814] kernfs_fop_write_iter+0x335/0x510
[ 1112.997602] ? kernfs_ops+0xf0/0xf0
[ 1113.002314] new_sync_write+0x333/0x750
[ 1113.007408] ? __lock_acquire+0x508/0xc10
[ 1113.012710] ? new_sync_read+0x740/0x740
[ 1113.017909] ? lock_acquire.part.0+0xbc/0x260
[ 1113.023596] ? ksys_write+0x116/0x250
[ 1113.028506] vfs_write+0x657/0x9b0
[ 1113.033123] ksys_write+0x116/0x250
[ 1113.037836] ? __ia32_sys_read+0xc0/0xc0
[ 1113.043025] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.048614] ? rcu_is_watching+0x15/0xd0
[ 1113.053815] do_syscall_64+0xf3/0x650
[ 1113.058723] ? trace_hardirqs_off+0xd/0x30
[ 1113.064116] ? exc_page_fault+0xda/0xf0
[ 1113.069219] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.075688] RIP: 0033:0x7f1238da864e
[ 1113.080496] RSP: 002b:00007ffcd5bef040 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.089793] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1238da864e
[ 1113.098591] RDX: 0000000000000001 RSI: 000056518cbadbf0 RDI: 0000000000000007
[ 1113.107400] RBP: 00007ffcd5bef050 R08: 0000000000000000 R09: 0000000000000000
[ 1113.116208] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1238c21390
[ 1113.125016] R13: 000056518cbadbf0 R14: 0000000000000007 R15: 00007f1238bb5120
[ 1113.133829]
[ 1113.137093] \x0aShowing all locks held in the system:
[ 1113.145630] 1 lock held by khungtaskd/186:
[ 1113.151022] #0: ffffffffb1afdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.162495] 3 locks held by systemd-journal/805:
[ 1113.168476] #0: ff1100180e5374e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.178365] #1: ffffffffb1d02ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.191183] #2: ff11000187a3fdb0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.202742] 4 locks held by python3/3479:
[ 1113.208038] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.217926] #1: ff110001c1538880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.228883] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.240042] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.250135] 4 locks held by python3/5465:
[ 1113.255432] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.265320] #1: ff110001832e8280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.276283] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.287443] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.297532]
[ 1113.300000] =============================================\x0a
[ 1133.380921] block nvme0n1: No UUID available providing old NGUID
SUBSYSTEM=block
DEVICE=b259:0
[ 1235.460576] INFO: task python3:5465 blocked for more than 368 seconds.
[ 1235.468724] Not tainted 7.1.0-rc7-ckzo-gcfdb3b4a9410 #1
[ 1235.475975] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.485566] task:python3 state:D stack:25672 pid:5465 tgid:5465 ppid:5464 task_flags:0x400100 flags:0x00080002
[ 1235.498773] Call Trace:
[ 1235.502314]
[ 1235.505467] __schedule+0xcc3/0x19c0
[ 1235.510286] ? io_schedule_timeout+0x140/0x140
[ 1235.516070] ? find_held_lock+0x2b/0x80
[ 1235.521169] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.527056] schedule+0xd8/0x260
[ 1235.531474] schedule_preempt_disabled+0x18/0x30
[ 1235.537454] __mutex_lock+0x6af/0x2010
[ 1235.542460] ? threaded_store+0x132/0x220
[ 1235.547757] ? ww_mutex_lock+0x160/0x160
[ 1235.552960] ? selinux_inode_free_security+0x330/0x330
[ 1235.559532] ? cap_capable+0x1d6/0x3e0
[ 1235.564542] ? threaded_store+0x132/0x220
[ 1235.569836] threaded_store+0x132/0x220
[ 1235.574938] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.580237] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.585541] ? sysfs_kf_write+0x65/0x160
[ 1235.590740] ? sysfs_file_ops+0x140/0x140
[ 1235.596036] kernfs_fop_write_iter+0x335/0x510
[ 1235.601824] ? kernfs_ops+0xf0/0xf0
[ 1235.606535] new_sync_write+0x333/0x750
[ 1235.611639] ? __lock_acquire+0x508/0xc10
[ 1235.616938] ? new_sync_read+0x740/0x740
[ 1235.622137] ? lock_acquire.part.0+0xbc/0x260
[ 1235.627825] ? ksys_write+0x116/0x250
[ 1235.632725] vfs_write+0x657/0x9b0
[ 1235.637339] ksys_write+0x116/0x250
[ 1235.642052] ? __ia32_sys_read+0xc0/0xc0
[ 1235.647249] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.652841] ? rcu_is_watching+0x15/0xd0
[ 1235.658043] do_syscall_64+0xf3/0x650
[ 1235.662951] ? trace_hardirqs_off+0xd/0x30
[ 1235.668346] ? exc_page_fault+0xda/0xf0
[ 1235.673448] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.679915] RIP: 0033:0x7fa5624b264e
[ 1235.684715] RSP: 002b:00007ffe0ffa36a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.694011] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa5624b264e
[ 1235.702812] RDX: 0000000000000001 RSI: 00005577400f9bf0 RDI: 0000000000000007
[ 1235.711621] RBP: 00007ffe0ffa36b0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.720432] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa56232b390
[ 1235.729241] R13: 00005577400f9bf0 R14: 0000000000000007 R15: 00007fa5622c1120
[ 1235.738053]
[ 1235.741320] INFO: task python3:5465 is blocked on a mutex likely owned by task python3:3479.
[ 1235.751594] task:python3 state:D stack:25672 pid:3479 tgid:3479 ppid:3478 task_flags:0x400100 flags:0x00080802
[ 1235.764799] Call Trace:
[ 1235.768337]
[ 1235.771487] __schedule+0xcc3/0x19c0
[ 1235.776299] ? io_schedule_timeout+0x140/0x140
[ 1235.782084] ? find_held_lock+0x2b/0x80
[ 1235.787185] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.793365] schedule+0xd8/0x260
[ 1235.797786] schedule_timeout+0x13f/0x290
[ 1235.803083] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.809445] ? trace_contention_end+0xb3/0x1a0
[ 1235.815234] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.820924] ? ww_mutex_lock+0x160/0x160
[ 1235.826125] msleep+0x1b/0x30
[ 1235.830248] napi_set_threaded+0x3a5/0x440
[ 1235.835643] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.841137] netif_set_threaded+0x257/0x400
[ 1235.846630] threaded_store+0x1b8/0x220
[ 1235.851725] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.857023] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.862320] ? sysfs_kf_write+0x65/0x160
[ 1235.867520] ? sysfs_file_ops+0x140/0x140
[ 1235.872814] kernfs_fop_write_iter+0x335/0x510
[ 1235.878597] ? kernfs_ops+0xf0/0xf0
[ 1235.883311] new_sync_write+0x333/0x750
[ 1235.888414] ? __lock_acquire+0x508/0xc10
[ 1235.893718] ? new_sync_read+0x740/0x740
[ 1235.898911] ? lock_acquire.part.0+0xbc/0x260
[ 1235.904598] ? ksys_write+0x116/0x250
[ 1235.909510] vfs_write+0x657/0x9b0
[ 1235.914126] ksys_write+0x116/0x250
[ 1235.918840] ? __ia32_sys_read+0xc0/0xc0
[ 1235.924037] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.929627] ? rcu_is_watching+0x15/0xd0
[ 1235.934820] do_syscall_64+0xf3/0x650
[ 1235.939727] ? trace_hardirqs_off+0xd/0x30
[ 1235.945120] ? exc_page_fault+0xda/0xf0
[ 1235.950221] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.956687] RIP: 0033:0x7f1238da864e
[ 1235.961495] RSP: 002b:00007ffcd5bef040 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.970794] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1238da864e
[ 1235.979601] RDX: 0000000000000001 RSI: 000056518cbadbf0 RDI: 0000000000000007
[ 1235.988400] RBP: 00007ffcd5bef050 R08: 0000000000000000 R09: 0000000000000000
[ 1235.997210] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1238c21390
[ 1236.006019] R13: 000056518cbadbf0 R14: 0000000000000007 R15: 00007f1238bb5120
[ 1236.014834]
[ 1236.018101] \x0aShowing all locks held in the system:
[ 1236.026638] 1 lock held by khungtaskd/186:
[ 1236.032030] #0: ffffffffb1afdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.043502] 3 locks held by systemd-journal/805:
[ 1236.049480] #0: ff1100180e5b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.059367] #1: ffffffffb1d02ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.072176] #2: ff11000187a3ff10 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1236.083733] 4 locks held by python3/3479:
[ 1236.089029] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.098917] #1: ff110001c1538880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.109878] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.121033] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.131128] 4 locks held by python3/5465:
[ 1236.136430] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.146319] #1: ff110001832e8280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.157274] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.168434] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.178526]
[ 1236.180995] =============================================\x0a
[ 1358.341610] INFO: task python3:5465 blocked for more than 491 seconds.
[ 1358.349770] Not tainted 7.1.0-rc7-ckzo-gcfdb3b4a9410 #1
[ 1358.357027] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.366620] task:python3 state:D stack:25672 pid:5465 tgid:5465 ppid:5464 task_flags:0x400100 flags:0x00080002
[ 1358.379838] Call Trace:
[ 1358.383382]
[ 1358.386530] __schedule+0xcc3/0x19c0
[ 1358.391350] ? io_schedule_timeout+0x140/0x140
[ 1358.397140] ? find_held_lock+0x2b/0x80
[ 1358.402245] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.408139] schedule+0xd8/0x260
[ 1358.412562] schedule_preempt_disabled+0x18/0x30
[ 1358.418546] __mutex_lock+0x6af/0x2010
[ 1358.423555] ? threaded_store+0x132/0x220
[ 1358.428859] ? ww_mutex_lock+0x160/0x160
[ 1358.434064] ? selinux_inode_free_security+0x330/0x330
[ 1358.440641] ? cap_capable+0x1d6/0x3e0
[ 1358.445654] ? threaded_store+0x132/0x220
[ 1358.450953] threaded_store+0x132/0x220
[ 1358.456056] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.461356] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.466660] ? sysfs_kf_write+0x65/0x160
[ 1358.471870] ? sysfs_file_ops+0x140/0x140
[ 1358.477170] kernfs_fop_write_iter+0x335/0x510
[ 1358.482961] ? kernfs_ops+0xf0/0xf0
[ 1358.487674] new_sync_write+0x333/0x750
[ 1358.492780] ? __lock_acquire+0x508/0xc10
[ 1358.498079] ? new_sync_read+0x740/0x740
[ 1358.503289] ? lock_acquire.part.0+0xbc/0x260
[ 1358.508979] ? ksys_write+0x116/0x250
[ 1358.513896] vfs_write+0x657/0x9b0
[ 1358.518515] ksys_write+0x116/0x250
[ 1358.523232] ? __ia32_sys_read+0xc0/0xc0
[ 1358.528434] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.534026] ? rcu_is_watching+0x15/0xd0
[ 1358.539230] do_syscall_64+0xf3/0x650
[ 1358.544138] ? trace_hardirqs_off+0xd/0x30
[ 1358.549535] ? exc_page_fault+0xda/0xf0
[ 1358.554633] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.561105] RIP: 0033:0x7fa5624b264e
[ 1358.565921] RSP: 002b:00007ffe0ffa36a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.575223] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa5624b264e
[ 1358.584039] RDX: 0000000000000001 RSI: 00005577400f9bf0 RDI: 0000000000000007
[ 1358.592851] RBP: 00007ffe0ffa36b0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.601666] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa56232b390
[ 1358.610478] R13: 00005577400f9bf0 R14: 0000000000000007 R15: 00007fa5622c1120
[ 1358.619303]
[ 1358.622581] INFO: task python3:5465 is blocked on a mutex likely owned by task python3:3479.
[ 1358.632862] task:python3 state:D stack:25672 pid:3479 tgid:3479 ppid:3478 task_flags:0x400100 flags:0x00080802
[ 1358.646073] Call Trace:
[ 1358.649614]
[ 1358.652768] __schedule+0xcc3/0x19c0
[ 1358.657586] ? io_schedule_timeout+0x140/0x140
[ 1358.663365] ? find_held_lock+0x2b/0x80
[ 1358.668472] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.674654] schedule+0xd8/0x260
[ 1358.679078] schedule_timeout+0x13f/0x290
[ 1358.684380] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.690755] ? trace_contention_end+0xb3/0x1a0
[ 1358.696543] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.702242] ? ww_mutex_lock+0x160/0x160
[ 1358.707451] msleep+0x1b/0x30
[ 1358.711579] napi_set_threaded+0x3a5/0x440
[ 1358.716978] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.722482] netif_set_threaded+0x257/0x400
[ 1358.727976] threaded_store+0x1b8/0x220
[ 1358.733083] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.738385] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.743685] ? sysfs_kf_write+0x65/0x160
[ 1358.748888] ? sysfs_file_ops+0x140/0x140
[ 1358.754190] kernfs_fop_write_iter+0x335/0x510
[ 1358.759975] ? kernfs_ops+0xf0/0xf0
[ 1358.764690] new_sync_write+0x333/0x750
[ 1358.769795] ? __lock_acquire+0x508/0xc10
[ 1358.775089] ? new_sync_read+0x740/0x740
[ 1358.780294] ? lock_acquire.part.0+0xbc/0x260
[ 1358.785983] ? ksys_write+0x116/0x250
[ 1358.790896] vfs_write+0x657/0x9b0
[ 1358.795517] ksys_write+0x116/0x250
[ 1358.800234] ? __ia32_sys_read+0xc0/0xc0
[ 1358.805436] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.811028] ? rcu_is_watching+0x15/0xd0
[ 1358.816234] do_syscall_64+0xf3/0x650
[ 1358.821143] ? trace_hardirqs_off+0xd/0x30
[ 1358.826540] ? exc_page_fault+0xda/0xf0
[ 1358.831644] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.838109] RIP: 0033:0x7f1238da864e
[ 1358.842921] RSP: 002b:00007ffcd5bef040 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.852226] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1238da864e
[ 1358.861038] RDX: 0000000000000001 RSI: 000056518cbadbf0 RDI: 0000000000000007
[ 1358.869852] RBP: 00007ffcd5bef050 R08: 0000000000000000 R09: 0000000000000000
[ 1358.878665] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1238c21390
[ 1358.887479] R13: 000056518cbadbf0 R14: 0000000000000007 R15: 00007f1238bb5120
[ 1358.896295]
[ 1358.899569] \x0aShowing all locks held in the system:
[ 1358.908102] 1 lock held by khungtaskd/186:
[ 1358.913500] #0: ffffffffb1afdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.924983] 4 locks held by python3/3479:
[ 1358.930284] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.940181] #1: ff110001c1538880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.951149] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.962316] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.972418] 4 locks held by python3/5465:
[ 1358.977714] #0: ff11000130ff13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.987611] #1: ff110001832e8280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.998583] #2: ff110017d93eab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.009749] #3: ff11000124051e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.019850]
[ 1359.022319] =============================================\x0a