[ 779.576957] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 779.579396] nipa-hw-worker: [4/40] Retrying drivers/net:napi_threaded.py
[ 779.732903] kselftest: Running tests in drivers/net
[ 989.661645] INFO: task python3:5623 blocked for more than 122 seconds.
[ 989.669873] Not tainted 7.0.0-rc6-yiqv-g629b28ff78c6 #1
[ 989.677125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.686717] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080000
[ 989.699924] Call Trace:
[ 989.703469]
[ 989.706621] __schedule+0xae3/0x14d0
[ 989.711436] ? io_schedule_timeout+0x140/0x140
[ 989.717224] ? find_held_lock+0x2b/0x80
[ 989.722328] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.728215] schedule+0xd4/0x260
[ 989.732633] schedule_preempt_disabled+0x18/0x30
[ 989.738612] __mutex_lock+0x6b2/0x1ee0
[ 989.743617] ? rcu_is_watching+0x15/0xd0
[ 989.748816] ? threaded_store+0x132/0x220
[ 989.754116] ? ww_mutex_lock+0x160/0x160
[ 989.759309] ? selinux_inode_free_security+0x330/0x330
[ 989.765882] ? cap_capable+0x1ed/0x3f0
[ 989.770892] ? threaded_store+0x132/0x220
[ 989.776187] threaded_store+0x132/0x220
[ 989.781292] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.786594] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.791893] ? sysfs_kf_write+0x65/0x160
[ 989.797095] ? sysfs_file_ops+0x140/0x140
[ 989.802389] kernfs_fop_write_iter+0x335/0x510
[ 989.808174] ? kernfs_ops+0xf0/0xf0
[ 989.812886] new_sync_write+0x333/0x750
[ 989.817992] ? __lock_acquire+0x508/0xc10
[ 989.823289] ? new_sync_read+0x740/0x740
[ 989.828488] ? lock_acquire.part.0+0xbc/0x260
[ 989.834175] ? ksys_write+0x116/0x250
[ 989.839078] vfs_write+0x657/0x9b0
[ 989.843691] ksys_write+0x116/0x250
[ 989.848402] ? __ia32_sys_read+0xc0/0xc0
[ 989.853601] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.859182] ? rcu_is_watching+0x15/0xd0
[ 989.864382] do_syscall_64+0xf3/0x5e0
[ 989.869289] ? trace_hardirqs_off+0xd/0x30
[ 989.874678] ? exc_page_fault+0xda/0xf0
[ 989.879785] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.886252] RIP: 0033:0x7f23aadf422e
[ 989.891056] RSP: 002b:00007ffcf537f980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.900356] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f23aadf422e
[ 989.909164] RDX: 0000000000000001 RSI: 0000556f0d201f40 RDI: 0000000000000007
[ 989.917971] RBP: 00007ffcf537f990 R08: 0000000000000000 R09: 0000000000000000
[ 989.926779] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f23aac6c390
[ 989.935586] R13: 0000556f0d201f40 R14: 0000000000000007 R15: 0000000000000003
[ 989.944401]
[ 989.947681] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3565.
[ 989.957954] task:python3 state:D stack:25336 pid:3565 tgid:3565 ppid:3564 task_flags:0x400100 flags:0x00080802
[ 989.971157] Call Trace:
[ 989.974697]
[ 989.977847] __schedule+0xae3/0x14d0
[ 989.982660] ? io_schedule_timeout+0x140/0x140
[ 989.988443] ? find_held_lock+0x2b/0x80
[ 989.993548] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.999728] schedule+0xd4/0x260
[ 990.004149] schedule_timeout+0x13f/0x290
[ 990.009448] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 990.015816] ? trace_contention_end+0xb3/0x1a0
[ 990.021600] ? hrtimers_cpu_dying+0x260/0x260
[ 990.027289] ? ww_mutex_lock+0x160/0x160
[ 990.032488] msleep+0x1b/0x30
[ 990.036613] napi_set_threaded+0x3a5/0x440
[ 990.042008] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.047505] netif_set_threaded+0x257/0x400
[ 990.053000] threaded_store+0x1b8/0x220
[ 990.058105] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.063403] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.068699] ? sysfs_kf_write+0x65/0x160
[ 990.073897] ? sysfs_file_ops+0x140/0x140
[ 990.079191] kernfs_fop_write_iter+0x335/0x510
[ 990.084976] ? kernfs_ops+0xf0/0xf0
[ 990.089690] new_sync_write+0x333/0x750
[ 990.094790] ? __lock_acquire+0x508/0xc10
[ 990.100087] ? new_sync_read+0x740/0x740
[ 990.105289] ? lock_acquire.part.0+0xbc/0x260
[ 990.110978] ? ksys_write+0x116/0x250
[ 990.115887] vfs_write+0x657/0x9b0
[ 990.120503] ksys_write+0x116/0x250
[ 990.125213] ? __ia32_sys_read+0xc0/0xc0
[ 990.130410] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.135999] ? rcu_is_watching+0x15/0xd0
[ 990.141200] do_syscall_64+0xf3/0x5e0
[ 990.146107] ? trace_hardirqs_off+0xd/0x30
[ 990.151495] ? exc_page_fault+0xda/0xf0
[ 990.156599] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.163064] RIP: 0033:0x7fa93317522e
[ 990.167871] RSP: 002b:00007ffd38a083a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.177169] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa93317522e
[ 990.185977] RDX: 0000000000000001 RSI: 00005559215f4f40 RDI: 0000000000000007
[ 990.194784] RBP: 00007ffd38a083b0 R08: 0000000000000000 R09: 0000000000000000
[ 990.203590] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa932fed390
[ 990.212400] R13: 00005559215f4f40 R14: 0000000000000007 R15: 0000000000000003
[ 990.221212]
[ 990.224477] \x0aShowing all locks held in the system:
[ 990.233014] 1 lock held by khungtaskd/185:
[ 990.238407] #0: ffffffffba8cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.249891] 4 locks held by python3/3565:
[ 990.255175] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.265067] #1: ff1100015863d688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.276030] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.287188] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.297290] 4 locks held by python3/5623:
[ 990.302584] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.312475] #1: ff110001c601ca88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.323439] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.334597] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.344685]
[ 990.347150] =============================================\x0a
[ 1112.542771] INFO: task python3:5623 blocked for more than 245 seconds.
[ 1112.550916] Not tainted 7.0.0-rc6-yiqv-g629b28ff78c6 #1
[ 1112.558169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.567764] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080000
[ 1112.580979] Call Trace:
[ 1112.584596]
[ 1112.587750] __schedule+0xae3/0x14d0
[ 1112.592573] ? io_schedule_timeout+0x140/0x140
[ 1112.598500] ? find_held_lock+0x2b/0x80
[ 1112.603610] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.609500] schedule+0xd4/0x260
[ 1112.613922] schedule_preempt_disabled+0x18/0x30
[ 1112.619905] __mutex_lock+0x6b2/0x1ee0
[ 1112.624915] ? rcu_is_watching+0x15/0xd0
[ 1112.630119] ? threaded_store+0x132/0x220
[ 1112.635535] ? ww_mutex_lock+0x160/0x160
[ 1112.640738] ? selinux_inode_free_security+0x330/0x330
[ 1112.647319] ? cap_capable+0x1ed/0x3f0
[ 1112.652331] ? threaded_store+0x132/0x220
[ 1112.657634] threaded_store+0x132/0x220
[ 1112.662743] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.668157] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.673457] ? sysfs_kf_write+0x65/0x160
[ 1112.678653] ? sysfs_file_ops+0x140/0x140
[ 1112.683954] kernfs_fop_write_iter+0x335/0x510
[ 1112.689855] ? kernfs_ops+0xf0/0xf0
[ 1112.694572] new_sync_write+0x333/0x750
[ 1112.699678] ? __lock_acquire+0x508/0xc10
[ 1112.704979] ? new_sync_read+0x740/0x740
[ 1112.710184] ? lock_acquire.part.0+0xbc/0x260
[ 1112.715872] ? ksys_write+0x116/0x250
[ 1112.720791] vfs_write+0x657/0x9b0
[ 1112.725410] ksys_write+0x116/0x250
[ 1112.730126] ? __ia32_sys_read+0xc0/0xc0
[ 1112.735331] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.740926] ? rcu_is_watching+0x15/0xd0
[ 1112.746130] do_syscall_64+0xf3/0x5e0
[ 1112.751043] ? trace_hardirqs_off+0xd/0x30
[ 1112.756441] ? exc_page_fault+0xda/0xf0
[ 1112.761538] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.768010] RIP: 0033:0x7f23aadf422e
[ 1112.772822] RSP: 002b:00007ffcf537f980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.782126] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f23aadf422e
[ 1112.790942] RDX: 0000000000000001 RSI: 0000556f0d201f40 RDI: 0000000000000007
[ 1112.799824] RBP: 00007ffcf537f990 R08: 0000000000000000 R09: 0000000000000000
[ 1112.808640] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f23aac6c390
[ 1112.817451] R13: 0000556f0d201f40 R14: 0000000000000007 R15: 0000000000000003
[ 1112.826278]
[ 1112.829603] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3565.
[ 1112.839876] task:python3 state:D stack:25336 pid:3565 tgid:3565 ppid:3564 task_flags:0x400100 flags:0x00080802
[ 1112.853085] Call Trace:
[ 1112.856628]
[ 1112.859775] __schedule+0xae3/0x14d0
[ 1112.864588] ? io_schedule_timeout+0x140/0x140
[ 1112.870377] ? find_held_lock+0x2b/0x80
[ 1112.875485] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.881665] schedule+0xd4/0x260
[ 1112.886088] schedule_timeout+0x13f/0x290
[ 1112.891388] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1112.897761] ? trace_contention_end+0xb3/0x1a0
[ 1112.903551] ? hrtimers_cpu_dying+0x260/0x260
[ 1112.909249] ? ww_mutex_lock+0x160/0x160
[ 1112.914455] msleep+0x1b/0x30
[ 1112.918583] napi_set_threaded+0x3a5/0x440
[ 1112.923984] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.929487] netif_set_threaded+0x257/0x400
[ 1112.934987] threaded_store+0x1b8/0x220
[ 1112.940210] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.945505] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.950803] ? sysfs_kf_write+0x65/0x160
[ 1112.955992] ? sysfs_file_ops+0x140/0x140
[ 1112.961286] kernfs_fop_write_iter+0x335/0x510
[ 1112.967073] ? kernfs_ops+0xf0/0xf0
[ 1112.971786] new_sync_write+0x333/0x750
[ 1112.976888] ? __lock_acquire+0x508/0xc10
[ 1112.982184] ? new_sync_read+0x740/0x740
[ 1112.987387] ? lock_acquire.part.0+0xbc/0x260
[ 1112.993075] ? ksys_write+0x116/0x250
[ 1112.997985] vfs_write+0x657/0x9b0
[ 1113.002603] ksys_write+0x116/0x250
[ 1113.007316] ? __ia32_sys_read+0xc0/0xc0
[ 1113.012505] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.018094] ? rcu_is_watching+0x15/0xd0
[ 1113.023295] do_syscall_64+0xf3/0x5e0
[ 1113.028202] ? trace_hardirqs_off+0xd/0x30
[ 1113.033599] ? exc_page_fault+0xda/0xf0
[ 1113.038699] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.045165] RIP: 0033:0x7fa93317522e
[ 1113.049974] RSP: 002b:00007ffd38a083a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.059274] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa93317522e
[ 1113.068084] RDX: 0000000000000001 RSI: 00005559215f4f40 RDI: 0000000000000007
[ 1113.076892] RBP: 00007ffd38a083b0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.085699] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa932fed390
[ 1113.094506] R13: 00005559215f4f40 R14: 0000000000000007 R15: 0000000000000003
[ 1113.103312]
[ 1113.106574] \x0aShowing all locks held in the system:
[ 1113.115113] 1 lock held by khungtaskd/185:
[ 1113.120507] #0: ffffffffba8cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.131986] 4 locks held by systemd-journal/798:
[ 1113.137964] #0: ff1100180dc372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1113.147853] #1: ffffffffbaac8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.160671] #2: ff110001a45c6ff0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.172220] #3: ff1100019b364268 (&xa->xa_lock#5){-.-.}-{3:3}, at: get_from_partial_node+0x7a/0x550
[ 1113.183296] 4 locks held by python3/3565:
[ 1113.188583] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.198474] #1: ff1100015863d688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.209434] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.220601] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.230704] 4 locks held by python3/5623:
[ 1113.236000] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.245887] #1: ff110001c601ca88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.256851] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.268006] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.278093] 5 locks held by systemd/6666:
[ 1113.283390] #0: ff110017dbae13f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.293279] #1: ff11000236a3b888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.304234] #2: ffffffffba937108 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x560
[ 1113.315878] #3: ffffffffba768b50 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 1113.327425] #4: ffffffffba936e10 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 1113.339747]
[ 1113.342213] =============================================\x0a
[ 1125.468482] block nvme0n1: No UUID available providing old NGUID
SUBSYSTEM=block
DEVICE=b259:0
[ 1235.423882] INFO: task python3:5623 blocked for more than 368 seconds.
[ 1235.432027] Not tainted 7.0.0-rc6-yiqv-g629b28ff78c6 #1
[ 1235.439277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.448864] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080002
[ 1235.462069] Call Trace:
[ 1235.465608]
[ 1235.468759] __schedule+0xae3/0x14d0
[ 1235.473579] ? io_schedule_timeout+0x140/0x140
[ 1235.479362] ? find_held_lock+0x2b/0x80
[ 1235.484465] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.490348] schedule+0xd4/0x260
[ 1235.494763] schedule_preempt_disabled+0x18/0x30
[ 1235.500739] __mutex_lock+0x6b2/0x1ee0
[ 1235.505747] ? rcu_is_watching+0x15/0xd0
[ 1235.510948] ? threaded_store+0x132/0x220
[ 1235.516245] ? ww_mutex_lock+0x160/0x160
[ 1235.521442] ? selinux_inode_free_security+0x330/0x330
[ 1235.528010] ? cap_capable+0x1ed/0x3f0
[ 1235.533017] ? threaded_store+0x132/0x220
[ 1235.538313] threaded_store+0x132/0x220
[ 1235.543416] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.548715] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.554011] ? sysfs_kf_write+0x65/0x160
[ 1235.559209] ? sysfs_file_ops+0x140/0x140
[ 1235.564504] kernfs_fop_write_iter+0x335/0x510
[ 1235.570291] ? kernfs_ops+0xf0/0xf0
[ 1235.575001] new_sync_write+0x333/0x750
[ 1235.580105] ? __lock_acquire+0x508/0xc10
[ 1235.585398] ? new_sync_read+0x740/0x740
[ 1235.590599] ? lock_acquire.part.0+0xbc/0x260
[ 1235.596285] ? ksys_write+0x116/0x250
[ 1235.601192] vfs_write+0x657/0x9b0
[ 1235.605808] ksys_write+0x116/0x250
[ 1235.610520] ? __ia32_sys_read+0xc0/0xc0
[ 1235.615709] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.621295] ? rcu_is_watching+0x15/0xd0
[ 1235.626493] do_syscall_64+0xf3/0x5e0
[ 1235.631397] ? trace_hardirqs_off+0xd/0x30
[ 1235.636792] ? exc_page_fault+0xda/0xf0
[ 1235.641893] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.648362] RIP: 0033:0x7f23aadf422e
[ 1235.653173] RSP: 002b:00007ffcf537f980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.662470] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f23aadf422e
[ 1235.671270] RDX: 0000000000000001 RSI: 0000556f0d201f40 RDI: 0000000000000007
[ 1235.680075] RBP: 00007ffcf537f990 R08: 0000000000000000 R09: 0000000000000000
[ 1235.688883] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f23aac6c390
[ 1235.697682] R13: 0000556f0d201f40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.706497]
[ 1235.709777] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3565.
[ 1235.720048] task:python3 state:D stack:25336 pid:3565 tgid:3565 ppid:3564 task_flags:0x400100 flags:0x00080802
[ 1235.733250] Call Trace:
[ 1235.736788]
[ 1235.739935] __schedule+0xae3/0x14d0
[ 1235.744749] ? io_schedule_timeout+0x140/0x140
[ 1235.750533] ? find_held_lock+0x2b/0x80
[ 1235.755637] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.761814] schedule+0xd4/0x260
[ 1235.766233] schedule_timeout+0x13f/0x290
[ 1235.771529] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1235.777899] ? trace_contention_end+0xb3/0x1a0
[ 1235.783680] ? hrtimers_cpu_dying+0x260/0x260
[ 1235.789368] ? ww_mutex_lock+0x160/0x160
[ 1235.794568] msleep+0x1b/0x30
[ 1235.798685] napi_set_threaded+0x3a5/0x440
[ 1235.804080] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.809574] netif_set_threaded+0x257/0x400
[ 1235.815064] threaded_store+0x1b8/0x220
[ 1235.820165] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.825464] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.830750] ? sysfs_kf_write+0x65/0x160
[ 1235.835950] ? sysfs_file_ops+0x140/0x140
[ 1235.841238] kernfs_fop_write_iter+0x335/0x510
[ 1235.847021] ? kernfs_ops+0xf0/0xf0
[ 1235.851736] new_sync_write+0x333/0x750
[ 1235.856834] ? __lock_acquire+0x508/0xc10
[ 1235.862131] ? new_sync_read+0x740/0x740
[ 1235.867329] ? lock_acquire.part.0+0xbc/0x260
[ 1235.873016] ? ksys_write+0x116/0x250
[ 1235.877917] vfs_write+0x657/0x9b0
[ 1235.882530] ksys_write+0x116/0x250
[ 1235.887239] ? __ia32_sys_read+0xc0/0xc0
[ 1235.892438] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.898025] ? rcu_is_watching+0x15/0xd0
[ 1235.903225] do_syscall_64+0xf3/0x5e0
[ 1235.908122] ? trace_hardirqs_off+0xd/0x30
[ 1235.913514] ? exc_page_fault+0xda/0xf0
[ 1235.918612] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.925077] RIP: 0033:0x7fa93317522e
[ 1235.929884] RSP: 002b:00007ffd38a083a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.939178] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa93317522e
[ 1235.947985] RDX: 0000000000000001 RSI: 00005559215f4f40 RDI: 0000000000000007
[ 1235.956791] RBP: 00007ffd38a083b0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.965597] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa932fed390
[ 1235.974402] R13: 00005559215f4f40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.983214]
[ 1235.986481] \x0aShowing all locks held in the system:
[ 1235.995017] 1 lock held by khungtaskd/185:
[ 1236.000410] #0: ffffffffba8cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.011885] 2 locks held by systemd-journal/798:
[ 1236.017852] #0: ff1100180e0b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1236.027746] #1: ff110001a24bf558 (&ep->lock){-.-.}-{3:3}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.038032] 4 locks held by python3/3565:
[ 1236.043317] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.053204] #1: ff1100015863d688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.064166] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.075316] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.085411] 4 locks held by python3/5623:
[ 1236.090707] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.100597] #1: ff110001c601ca88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.111556] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.122710] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.132798]
[ 1236.135265] =============================================\x0a
[ 1358.304031] INFO: task python3:5623 blocked for more than 491 seconds.
[ 1358.312178] Not tainted 7.0.0-rc6-yiqv-g629b28ff78c6 #1
[ 1358.319428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.329016] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080002
[ 1358.342216] Call Trace:
[ 1358.345754]
[ 1358.348902] __schedule+0xae3/0x14d0
[ 1358.353718] ? io_schedule_timeout+0x140/0x140
[ 1358.359501] ? find_held_lock+0x2b/0x80
[ 1358.364606] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.370499] schedule+0xd4/0x260
[ 1358.374955] schedule_preempt_disabled+0x18/0x30
[ 1358.380935] __mutex_lock+0x6b2/0x1ee0
[ 1358.385939] ? rcu_is_watching+0x15/0xd0
[ 1358.391139] ? threaded_store+0x132/0x220
[ 1358.396446] ? ww_mutex_lock+0x160/0x160
[ 1358.401642] ? selinux_inode_free_security+0x330/0x330
[ 1358.408213] ? cap_capable+0x1ed/0x3f0
[ 1358.413219] ? threaded_store+0x132/0x220
[ 1358.418503] threaded_store+0x132/0x220
[ 1358.423606] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.428901] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.434201] ? sysfs_kf_write+0x65/0x160
[ 1358.439399] ? sysfs_file_ops+0x140/0x140
[ 1358.444692] kernfs_fop_write_iter+0x335/0x510
[ 1358.450478] ? kernfs_ops+0xf0/0xf0
[ 1358.455178] new_sync_write+0x333/0x750
[ 1358.460280] ? __lock_acquire+0x508/0xc10
[ 1358.465577] ? new_sync_read+0x740/0x740
[ 1358.470777] ? lock_acquire.part.0+0xbc/0x260
[ 1358.476452] ? ksys_write+0x116/0x250
[ 1358.481359] vfs_write+0x657/0x9b0
[ 1358.485974] ksys_write+0x116/0x250
[ 1358.490686] ? __ia32_sys_read+0xc0/0xc0
[ 1358.495878] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.501470] ? rcu_is_watching+0x15/0xd0
[ 1358.506667] do_syscall_64+0xf3/0x5e0
[ 1358.511574] ? trace_hardirqs_off+0xd/0x30
[ 1358.516968] ? exc_page_fault+0xda/0xf0
[ 1358.522068] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.528536] RIP: 0033:0x7f23aadf422e
[ 1358.533345] RSP: 002b:00007ffcf537f980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.542641] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f23aadf422e
[ 1358.551449] RDX: 0000000000000001 RSI: 0000556f0d201f40 RDI: 0000000000000007
[ 1358.560248] RBP: 00007ffcf537f990 R08: 0000000000000000 R09: 0000000000000000
[ 1358.569048] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f23aac6c390
[ 1358.577855] R13: 0000556f0d201f40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.586668]
[ 1358.589959] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3565.
[ 1358.600232] task:python3 state:D stack:25336 pid:3565 tgid:3565 ppid:3564 task_flags:0x400100 flags:0x00080802
[ 1358.613432] Call Trace:
[ 1358.616968]
[ 1358.620119] __schedule+0xae3/0x14d0
[ 1358.624930] ? io_schedule_timeout+0x140/0x140
[ 1358.630714] ? find_held_lock+0x2b/0x80
[ 1358.635817] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.641994] schedule+0xd4/0x260
[ 1358.646402] schedule_timeout+0x13f/0x290
[ 1358.651697] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1358.658065] ? trace_contention_end+0xb3/0x1a0
[ 1358.663850] ? hrtimers_cpu_dying+0x260/0x260
[ 1358.669528] ? ww_mutex_lock+0x160/0x160
[ 1358.674728] msleep+0x1b/0x30
[ 1358.678844] napi_set_threaded+0x3a5/0x440
[ 1358.684237] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.689730] netif_set_threaded+0x257/0x400
[ 1358.695214] threaded_store+0x1b8/0x220
[ 1358.700315] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.705613] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.710898] ? sysfs_kf_write+0x65/0x160
[ 1358.716098] ? sysfs_file_ops+0x140/0x140
[ 1358.721385] kernfs_fop_write_iter+0x335/0x510
[ 1358.727168] ? kernfs_ops+0xf0/0xf0
[ 1358.731881] new_sync_write+0x333/0x750
[ 1358.736971] ? __lock_acquire+0x508/0xc10
[ 1358.742265] ? new_sync_read+0x740/0x740
[ 1358.747466] ? lock_acquire.part.0+0xbc/0x260
[ 1358.753143] ? ksys_write+0x116/0x250
[ 1358.758053] vfs_write+0x657/0x9b0
[ 1358.762665] ksys_write+0x116/0x250
[ 1358.767376] ? __ia32_sys_read+0xc0/0xc0
[ 1358.772573] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.778160] ? rcu_is_watching+0x15/0xd0
[ 1358.783360] do_syscall_64+0xf3/0x5e0
[ 1358.788266] ? trace_hardirqs_off+0xd/0x30
[ 1358.793656] ? exc_page_fault+0xda/0xf0
[ 1358.798754] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.805218] RIP: 0033:0x7fa93317522e
[ 1358.810023] RSP: 002b:00007ffd38a083a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.819333] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa93317522e
[ 1358.828139] RDX: 0000000000000001 RSI: 00005559215f4f40 RDI: 0000000000000007
[ 1358.836947] RBP: 00007ffd38a083b0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.845756] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa932fed390
[ 1358.854560] R13: 00005559215f4f40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.863371]
[ 1358.866641] \x0aShowing all locks held in the system:
[ 1358.875182] 1 lock held by khungtaskd/185:
[ 1358.880567] #0: ffffffffba8cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.892043] 2 locks held by systemd-journal/798:
[ 1358.898018] #0: ff1100180e2372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1358.907907] #1: ffffffffbaac8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.920726] 4 locks held by python3/3565:
[ 1358.926021] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.935910] #1: ff1100015863d688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.946870] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.958029] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.968128] 4 locks held by python3/5623:
[ 1358.973422] #0: ff110017db9e93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.983312] #1: ff110001c601ca88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.994270] #2: ff1100012274b0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.005424] #3: ff110001227c9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.015516]
[ 1359.017980] =============================================\x0a