[ 777.431650] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 777.434010] nipa-hw-worker: [4/39] Retrying drivers/net:napi_threaded.py
[ 777.586217] kselftest: Running tests in drivers/net
[ 850.410176] kworker/u130:4 (529) used greatest stack depth: 21192 bytes left
[ 989.675694] INFO: task python3:5555 blocked for more than 122 seconds.
[ 989.683910] Not tainted 7.0.0-rc6-fowg-gae9854baecae #1
[ 989.691159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.700746] task:python3 state:D stack:25336 pid:5555 tgid:5555 ppid:5554 task_flags:0x400100 flags:0x00080000
[ 989.713952] Call Trace:
[ 989.717490]
[ 989.720642] __schedule+0xae3/0x14d0
[ 989.725458] ? io_schedule_timeout+0x140/0x140
[ 989.731242] ? find_held_lock+0x2b/0x80
[ 989.736347] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.742230] schedule+0xd4/0x260
[ 989.746645] schedule_preempt_disabled+0x18/0x30
[ 989.752624] __mutex_lock+0x6b2/0x1ee0
[ 989.757630] ? rcu_is_watching+0x15/0xd0
[ 989.762827] ? threaded_store+0x132/0x220
[ 989.768123] ? ww_mutex_lock+0x160/0x160
[ 989.773321] ? selinux_inode_free_security+0x330/0x330
[ 989.779888] ? cap_capable+0x1ed/0x3f0
[ 989.784895] ? threaded_store+0x132/0x220
[ 989.790191] threaded_store+0x132/0x220
[ 989.795296] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.800594] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.805890] ? sysfs_kf_write+0x65/0x160
[ 989.811088] ? sysfs_file_ops+0x140/0x140
[ 989.816382] kernfs_fop_write_iter+0x335/0x510
[ 989.822164] ? kernfs_ops+0xf0/0xf0
[ 989.826873] new_sync_write+0x333/0x750
[ 989.831977] ? __lock_acquire+0x508/0xc10
[ 989.837271] ? new_sync_read+0x740/0x740
[ 989.842470] ? lock_acquire.part.0+0xbc/0x260
[ 989.848155] ? ksys_write+0x116/0x250
[ 989.853063] vfs_write+0x657/0x9b0
[ 989.857670] ksys_write+0x116/0x250
[ 989.862378] ? __ia32_sys_read+0xc0/0xc0
[ 989.867574] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.873162] ? rcu_is_watching+0x15/0xd0
[ 989.878359] do_syscall_64+0xf3/0x5e0
[ 989.883262] ? irq_exit_rcu+0x39/0xf0
[ 989.888170] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.894626] RIP: 0033:0x7f9288a6122e
[ 989.899436] RSP: 002b:00007ffd7e792df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.908733] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9288a6122e
[ 989.917545] RDX: 0000000000000001 RSI: 00005645c66ae340 RDI: 0000000000000007
[ 989.926350] RBP: 00007ffd7e792e00 R08: 0000000000000000 R09: 0000000000000000
[ 989.935157] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f92888d9390
[ 989.943972] R13: 00005645c66ae340 R14: 0000000000000007 R15: 0000000000000003
[ 989.952783]
[ 989.956061] INFO: task python3:5555 is blocked on a mutex likely owned by task python3:3512.
[ 989.966335] task:python3 state:D stack:25336 pid:3512 tgid:3512 ppid:3511 task_flags:0x400100 flags:0x00080802
[ 989.979541] Call Trace:
[ 989.983079]
[ 989.986226] __schedule+0xae3/0x14d0
[ 989.991036] ? io_schedule_timeout+0x140/0x140
[ 989.996813] ? find_held_lock+0x2b/0x80
[ 990.001916] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.008095] schedule+0xd4/0x260
[ 990.012512] schedule_timeout+0x13f/0x290
[ 990.017810] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 990.024171] ? trace_contention_end+0xb3/0x1a0
[ 990.029955] ? hrtimers_cpu_dying+0x260/0x260
[ 990.035642] ? ww_mutex_lock+0x160/0x160
[ 990.040838] msleep+0x1b/0x30
[ 990.044964] napi_set_threaded+0x3a5/0x440
[ 990.050356] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.055848] netif_set_threaded+0x257/0x400
[ 990.061342] threaded_store+0x1b8/0x220
[ 990.066440] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.071739] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.077034] ? sysfs_kf_write+0x65/0x160
[ 990.082234] ? sysfs_file_ops+0x140/0x140
[ 990.087521] kernfs_fop_write_iter+0x335/0x510
[ 990.093305] ? kernfs_ops+0xf0/0xf0
[ 990.098017] new_sync_write+0x333/0x750
[ 990.103120] ? __lock_acquire+0x508/0xc10
[ 990.108416] ? new_sync_read+0x740/0x740
[ 990.113617] ? lock_acquire.part.0+0xbc/0x260
[ 990.119303] ? ksys_write+0x116/0x250
[ 990.124212] vfs_write+0x657/0x9b0
[ 990.128823] ksys_write+0x116/0x250
[ 990.133537] ? __ia32_sys_read+0xc0/0xc0
[ 990.138736] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.144322] ? rcu_is_watching+0x15/0xd0
[ 990.149520] do_syscall_64+0xf3/0x5e0
[ 990.154424] ? irq_exit_rcu+0x39/0xf0
[ 990.159330] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.165795] RIP: 0033:0x7f2b27b6622e
[ 990.170599] RSP: 002b:00007ffc60b4ec10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.179896] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2b27b6622e
[ 990.188703] RDX: 0000000000000001 RSI: 0000561746f29340 RDI: 0000000000000007
[ 990.197508] RBP: 00007ffc60b4ec20 R08: 0000000000000000 R09: 0000000000000000
[ 990.206315] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2b279de390
[ 990.215119] R13: 0000561746f29340 R14: 0000000000000007 R15: 0000000000000003
[ 990.223931]
[ 990.227188] \x0aShowing all locks held in the system:
[ 990.235726] 1 lock held by khungtaskd/185:
[ 990.241119] #0: ffffffff970cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.252588] 3 locks held by systemd-journal/798:
[ 990.258564] #0: ff1100180ea372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 990.268452] #1: ffffffff972c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: nd_jump_root+0x353/0x550
[ 990.280770] #2: ff110001a3019e60 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.292328] 4 locks held by python3/3512:
[ 990.297621] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.307509] #1: ff11000151592c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.318461] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.329610] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.339702] 4 locks held by python3/5555:
[ 990.345000] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.354888] #1: ff110001d2075688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.365839] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.376994] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.387083]
[ 990.389550] =============================================\x0a
[ 1112.556991] INFO: task python3:5555 blocked for more than 245 seconds.
[ 1112.565139] Not tainted 7.0.0-rc6-fowg-gae9854baecae #1
[ 1112.572396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.581976] task:python3 state:D stack:25336 pid:5555 tgid:5555 ppid:5554 task_flags:0x400100 flags:0x00080000
[ 1112.595179] Call Trace:
[ 1112.598719]
[ 1112.601867] __schedule+0xae3/0x14d0
[ 1112.606680] ? io_schedule_timeout+0x140/0x140
[ 1112.612466] ? find_held_lock+0x2b/0x80
[ 1112.617572] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.623456] schedule+0xd4/0x260
[ 1112.627872] schedule_preempt_disabled+0x18/0x30
[ 1112.633849] __mutex_lock+0x6b2/0x1ee0
[ 1112.638854] ? rcu_is_watching+0x15/0xd0
[ 1112.644055] ? threaded_store+0x132/0x220
[ 1112.649356] ? ww_mutex_lock+0x160/0x160
[ 1112.654556] ? selinux_inode_free_security+0x330/0x330
[ 1112.661127] ? cap_capable+0x1ed/0x3f0
[ 1112.666132] ? threaded_store+0x132/0x220
[ 1112.671421] threaded_store+0x132/0x220
[ 1112.676517] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.681814] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.687102] ? sysfs_kf_write+0x65/0x160
[ 1112.692298] ? sysfs_file_ops+0x140/0x140
[ 1112.697594] kernfs_fop_write_iter+0x335/0x510
[ 1112.703379] ? kernfs_ops+0xf0/0xf0
[ 1112.708084] new_sync_write+0x333/0x750
[ 1112.713188] ? __lock_acquire+0x508/0xc10
[ 1112.718474] ? new_sync_read+0x740/0x740
[ 1112.723673] ? lock_acquire.part.0+0xbc/0x260
[ 1112.729351] ? ksys_write+0x116/0x250
[ 1112.734260] vfs_write+0x657/0x9b0
[ 1112.738873] ksys_write+0x116/0x250
[ 1112.743583] ? __ia32_sys_read+0xc0/0xc0
[ 1112.748780] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.754367] ? rcu_is_watching+0x15/0xd0
[ 1112.759566] do_syscall_64+0xf3/0x5e0
[ 1112.764474] ? irq_exit_rcu+0x39/0xf0
[ 1112.769382] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.775850] RIP: 0033:0x7f9288a6122e
[ 1112.780658] RSP: 002b:00007ffd7e792df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.789955] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9288a6122e
[ 1112.798753] RDX: 0000000000000001 RSI: 00005645c66ae340 RDI: 0000000000000007
[ 1112.807562] RBP: 00007ffd7e792e00 R08: 0000000000000000 R09: 0000000000000000
[ 1112.816370] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f92888d9390
[ 1112.825260] R13: 00005645c66ae340 R14: 0000000000000007 R15: 0000000000000003
[ 1112.834141]
[ 1112.837431] INFO: task python3:5555 is blocked on a mutex likely owned by task python3:3512.
[ 1112.847701] task:python3 state:D stack:25336 pid:3512 tgid:3512 ppid:3511 task_flags:0x400100 flags:0x00080802
[ 1112.860904] Call Trace:
[ 1112.864445]
[ 1112.867596] __schedule+0xae3/0x14d0
[ 1112.872407] ? io_schedule_timeout+0x140/0x140
[ 1112.878192] ? find_held_lock+0x2b/0x80
[ 1112.883293] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.889471] schedule+0xd4/0x260
[ 1112.893888] schedule_timeout+0x13f/0x290
[ 1112.899186] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1112.905544] ? trace_contention_end+0xb3/0x1a0
[ 1112.911326] ? hrtimers_cpu_dying+0x260/0x260
[ 1112.917015] ? ww_mutex_lock+0x160/0x160
[ 1112.922216] msleep+0x1b/0x30
[ 1112.926342] napi_set_threaded+0x3a5/0x440
[ 1112.931740] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.937237] netif_set_threaded+0x257/0x400
[ 1112.942730] threaded_store+0x1b8/0x220
[ 1112.947841] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.953139] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.958438] ? sysfs_kf_write+0x65/0x160
[ 1112.963636] ? sysfs_file_ops+0x140/0x140
[ 1112.968931] kernfs_fop_write_iter+0x335/0x510
[ 1112.974719] ? kernfs_ops+0xf0/0xf0
[ 1112.979433] new_sync_write+0x333/0x750
[ 1112.984535] ? __lock_acquire+0x508/0xc10
[ 1112.989839] ? new_sync_read+0x740/0x740
[ 1112.995038] ? lock_acquire.part.0+0xbc/0x260
[ 1113.000724] ? ksys_write+0x116/0x250
[ 1113.005636] vfs_write+0x657/0x9b0
[ 1113.010251] ksys_write+0x116/0x250
[ 1113.014964] ? __ia32_sys_read+0xc0/0xc0
[ 1113.020162] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.025752] ? rcu_is_watching+0x15/0xd0
[ 1113.030953] do_syscall_64+0xf3/0x5e0
[ 1113.035862] ? irq_exit_rcu+0x39/0xf0
[ 1113.040768] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.047236] RIP: 0033:0x7f2b27b6622e
[ 1113.052035] RSP: 002b:00007ffc60b4ec10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.061331] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2b27b6622e
[ 1113.070139] RDX: 0000000000000001 RSI: 0000561746f29340 RDI: 0000000000000007
[ 1113.078948] RBP: 00007ffc60b4ec20 R08: 0000000000000000 R09: 0000000000000000
[ 1113.087757] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2b279de390
[ 1113.096563] R13: 0000561746f29340 R14: 0000000000000007 R15: 0000000000000003
[ 1113.105376]
[ 1113.108642] \x0aShowing all locks held in the system:
[ 1113.117171] 1 lock held by khungtaskd/185:
[ 1113.122566] #0: ffffffff970cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.134047] 4 locks held by python3/3512:
[ 1113.139343] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.149234] #1: ff11000151592c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.160194] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.171353] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.181452] 4 locks held by python3/5555:
[ 1113.186746] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.196633] #1: ff110001d2075688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.207600] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.218757] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.228841] 5 locks held by systemd/6613:
[ 1113.234137] #0: ff1100012f1dc3f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.244026] #1: ff11000111fae288 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.254981] #2: ffffffff97137108 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x560
[ 1113.266626] #3: ffffffff96f68b50 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 1113.278175] #4: ffffffff97136e10 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 1113.290504]
[ 1113.292973] =============================================\x0a
[ 1235.437284] INFO: task python3:5555 blocked for more than 368 seconds.
[ 1235.445435] Not tainted 7.0.0-rc6-fowg-gae9854baecae #1
[ 1235.452686] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.462276] task:python3 state:D stack:25336 pid:5555 tgid:5555 ppid:5554 task_flags:0x400100 flags:0x00080002
[ 1235.475482] Call Trace:
[ 1235.479023]
[ 1235.482177] __schedule+0xae3/0x14d0
[ 1235.486994] ? io_schedule_timeout+0x140/0x140
[ 1235.492778] ? find_held_lock+0x2b/0x80
[ 1235.497883] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.503769] schedule+0xd4/0x260
[ 1235.508187] schedule_preempt_disabled+0x18/0x30
[ 1235.514165] __mutex_lock+0x6b2/0x1ee0
[ 1235.519171] ? rcu_is_watching+0x15/0xd0
[ 1235.524370] ? threaded_store+0x132/0x220
[ 1235.529660] ? ww_mutex_lock+0x160/0x160
[ 1235.534861] ? selinux_inode_free_security+0x330/0x330
[ 1235.541433] ? cap_capable+0x1ed/0x3f0
[ 1235.546444] ? threaded_store+0x132/0x220
[ 1235.551743] threaded_store+0x132/0x220
[ 1235.556848] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.562149] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.567448] ? sysfs_kf_write+0x65/0x160
[ 1235.572650] ? sysfs_file_ops+0x140/0x140
[ 1235.577947] kernfs_fop_write_iter+0x335/0x510
[ 1235.583731] ? kernfs_ops+0xf0/0xf0
[ 1235.588443] new_sync_write+0x333/0x750
[ 1235.593539] ? __lock_acquire+0x508/0xc10
[ 1235.598826] ? new_sync_read+0x740/0x740
[ 1235.604029] ? lock_acquire.part.0+0xbc/0x260
[ 1235.609716] ? ksys_write+0x116/0x250
[ 1235.614618] vfs_write+0x657/0x9b0
[ 1235.619236] ksys_write+0x116/0x250
[ 1235.623946] ? __ia32_sys_read+0xc0/0xc0
[ 1235.629143] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.634733] ? rcu_is_watching+0x15/0xd0
[ 1235.639931] do_syscall_64+0xf3/0x5e0
[ 1235.644838] ? irq_exit_rcu+0x39/0xf0
[ 1235.649748] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.656218] RIP: 0033:0x7f9288a6122e
[ 1235.661030] RSP: 002b:00007ffd7e792df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.670329] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9288a6122e
[ 1235.679137] RDX: 0000000000000001 RSI: 00005645c66ae340 RDI: 0000000000000007
[ 1235.687939] RBP: 00007ffd7e792e00 R08: 0000000000000000 R09: 0000000000000000
[ 1235.696749] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f92888d9390
[ 1235.705556] R13: 00005645c66ae340 R14: 0000000000000007 R15: 0000000000000003
[ 1235.714372]
[ 1235.717638] INFO: task python3:5555 is blocked on a mutex likely owned by task python3:3512.
[ 1235.727914] task:python3 state:D stack:25336 pid:3512 tgid:3512 ppid:3511 task_flags:0x400100 flags:0x00080802
[ 1235.741109] Call Trace:
[ 1235.744650]
[ 1235.747798] __schedule+0xae3/0x14d0
[ 1235.752611] ? io_schedule_timeout+0x140/0x140
[ 1235.758398] ? find_held_lock+0x2b/0x80
[ 1235.763500] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.769681] schedule+0xd4/0x260
[ 1235.774101] schedule_timeout+0x13f/0x290
[ 1235.779399] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1235.785771] ? trace_contention_end+0xb3/0x1a0
[ 1235.791558] ? hrtimers_cpu_dying+0x260/0x260
[ 1235.797247] ? ww_mutex_lock+0x160/0x160
[ 1235.802449] msleep+0x1b/0x30
[ 1235.806576] napi_set_threaded+0x3a5/0x440
[ 1235.811974] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.817471] netif_set_threaded+0x257/0x400
[ 1235.822967] threaded_store+0x1b8/0x220
[ 1235.828066] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.833365] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.838661] ? sysfs_kf_write+0x65/0x160
[ 1235.843861] ? sysfs_file_ops+0x140/0x140
[ 1235.849149] kernfs_fop_write_iter+0x335/0x510
[ 1235.854925] ? kernfs_ops+0xf0/0xf0
[ 1235.859636] new_sync_write+0x333/0x750
[ 1235.864740] ? __lock_acquire+0x508/0xc10
[ 1235.870035] ? new_sync_read+0x740/0x740
[ 1235.875237] ? lock_acquire.part.0+0xbc/0x260
[ 1235.880924] ? ksys_write+0x116/0x250
[ 1235.885833] vfs_write+0x657/0x9b0
[ 1235.890449] ksys_write+0x116/0x250
[ 1235.895161] ? __ia32_sys_read+0xc0/0xc0
[ 1235.900360] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.905948] ? rcu_is_watching+0x15/0xd0
[ 1235.911146] do_syscall_64+0xf3/0x5e0
[ 1235.916052] ? irq_exit_rcu+0x39/0xf0
[ 1235.920959] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.927428] RIP: 0033:0x7f2b27b6622e
[ 1235.932239] RSP: 002b:00007ffc60b4ec10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.941534] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2b27b6622e
[ 1235.950341] RDX: 0000000000000001 RSI: 0000561746f29340 RDI: 0000000000000007
[ 1235.959147] RBP: 00007ffc60b4ec20 R08: 0000000000000000 R09: 0000000000000000
[ 1235.967954] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2b279de390
[ 1235.976755] R13: 0000561746f29340 R14: 0000000000000007 R15: 0000000000000003
[ 1235.985570]
[ 1235.988837] \x0aShowing all locks held in the system:
[ 1235.997367] 1 lock held by khungtaskd/185:
[ 1236.002761] #0: ffffffff970cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.014234] 2 locks held by systemd-journal/798:
[ 1236.020212] #0: ff1100180dcb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1236.030101] #1: ffffffff972c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.042927] 4 locks held by python3/3512:
[ 1236.048223] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.058112] #1: ff11000151592c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.069078] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.080234] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.090329] 4 locks held by python3/5555:
[ 1236.095625] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.105517] #1: ff110001d2075688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.116477] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.127633] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.137724]
[ 1236.140192] =============================================\x0a
[ 1358.319569] INFO: task python3:5555 blocked for more than 491 seconds.
[ 1358.328240] Not tainted 7.0.0-rc6-fowg-gae9854baecae #1
[ 1358.335493] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.345084] task:python3 state:D stack:25336 pid:5555 tgid:5555 ppid:5554 task_flags:0x400100 flags:0x00080002
[ 1358.358290] Call Trace:
[ 1358.361834]
[ 1358.364986] __schedule+0xae3/0x14d0
[ 1358.369803] ? io_schedule_timeout+0x140/0x140
[ 1358.375588] ? find_held_lock+0x2b/0x80
[ 1358.380694] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.386581] schedule+0xd4/0x260
[ 1358.391001] schedule_preempt_disabled+0x18/0x30
[ 1358.396980] __mutex_lock+0x6b2/0x1ee0
[ 1358.401986] ? rcu_is_watching+0x15/0xd0
[ 1358.407185] ? threaded_store+0x132/0x220
[ 1358.412477] ? ww_mutex_lock+0x160/0x160
[ 1358.417675] ? selinux_inode_free_security+0x330/0x330
[ 1358.424249] ? cap_capable+0x1ed/0x3f0
[ 1358.429259] ? threaded_store+0x132/0x220
[ 1358.434554] threaded_store+0x132/0x220
[ 1358.439658] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.444955] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.450255] ? sysfs_kf_write+0x65/0x160
[ 1358.455453] ? sysfs_file_ops+0x140/0x140
[ 1358.460748] kernfs_fop_write_iter+0x335/0x510
[ 1358.466536] ? kernfs_ops+0xf0/0xf0
[ 1358.471250] new_sync_write+0x333/0x750
[ 1358.476357] ? __lock_acquire+0x508/0xc10
[ 1358.481654] ? new_sync_read+0x740/0x740
[ 1358.486854] ? lock_acquire.part.0+0xbc/0x260
[ 1358.492544] ? ksys_write+0x116/0x250
[ 1358.497456] vfs_write+0x657/0x9b0
[ 1358.502070] ksys_write+0x116/0x250
[ 1358.506781] ? __ia32_sys_read+0xc0/0xc0
[ 1358.511979] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.517567] ? rcu_is_watching+0x15/0xd0
[ 1358.522770] do_syscall_64+0xf3/0x5e0
[ 1358.527678] ? irq_exit_rcu+0x39/0xf0
[ 1358.532585] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.539052] RIP: 0033:0x7f9288a6122e
[ 1358.543861] RSP: 002b:00007ffd7e792df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.553161] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9288a6122e
[ 1358.561970] RDX: 0000000000000001 RSI: 00005645c66ae340 RDI: 0000000000000007
[ 1358.570778] RBP: 00007ffd7e792e00 R08: 0000000000000000 R09: 0000000000000000
[ 1358.579575] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f92888d9390
[ 1358.588382] R13: 00005645c66ae340 R14: 0000000000000007 R15: 0000000000000003
[ 1358.597196]
[ 1358.600461] INFO: task python3:5555 is blocked on a mutex likely owned by task python3:3512.
[ 1358.610734] task:python3 state:D stack:25336 pid:3512 tgid:3512 ppid:3511 task_flags:0x400100 flags:0x00080802
[ 1358.623939] Call Trace:
[ 1358.627482]
[ 1358.630624] __schedule+0xae3/0x14d0
[ 1358.635433] ? io_schedule_timeout+0x140/0x140
[ 1358.641211] ? find_held_lock+0x2b/0x80
[ 1358.646312] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.652485] schedule+0xd4/0x260
[ 1358.656902] schedule_timeout+0x13f/0x290
[ 1358.662202] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1358.668570] ? trace_contention_end+0xb3/0x1a0
[ 1358.674357] ? hrtimers_cpu_dying+0x260/0x260
[ 1358.680047] ? ww_mutex_lock+0x160/0x160
[ 1358.685247] msleep+0x1b/0x30
[ 1358.689366] napi_set_threaded+0x3a5/0x440
[ 1358.694766] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.700260] netif_set_threaded+0x257/0x400
[ 1358.705752] threaded_store+0x1b8/0x220
[ 1358.710854] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.716151] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.721448] ? sysfs_kf_write+0x65/0x160
[ 1358.726647] ? sysfs_file_ops+0x140/0x140
[ 1358.731941] kernfs_fop_write_iter+0x335/0x510
[ 1358.737726] ? kernfs_ops+0xf0/0xf0
[ 1358.742440] new_sync_write+0x333/0x750
[ 1358.747541] ? __lock_acquire+0x508/0xc10
[ 1358.752835] ? new_sync_read+0x740/0x740
[ 1358.758037] ? lock_acquire.part.0+0xbc/0x260
[ 1358.763722] ? ksys_write+0x116/0x250
[ 1358.768633] vfs_write+0x657/0x9b0
[ 1358.773248] ksys_write+0x116/0x250
[ 1358.777951] ? __ia32_sys_read+0xc0/0xc0
[ 1358.783149] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.788739] ? rcu_is_watching+0x15/0xd0
[ 1358.793941] do_syscall_64+0xf3/0x5e0
[ 1358.798848] ? irq_exit_rcu+0x39/0xf0
[ 1358.803753] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.810220] RIP: 0033:0x7f2b27b6622e
[ 1358.815030] RSP: 002b:00007ffc60b4ec10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.824330] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2b27b6622e
[ 1358.833136] RDX: 0000000000000001 RSI: 0000561746f29340 RDI: 0000000000000007
[ 1358.841945] RBP: 00007ffc60b4ec20 R08: 0000000000000000 R09: 0000000000000000
[ 1358.850752] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2b279de390
[ 1358.859551] R13: 0000561746f29340 R14: 0000000000000007 R15: 0000000000000003
[ 1358.868355]
[ 1358.871621] \x0aShowing all locks held in the system:
[ 1358.880158] 1 lock held by khungtaskd/185:
[ 1358.885553] #0: ffffffff970cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.897016] 2 locks held by systemd-journal/798:
[ 1358.903004] 4 locks held by python3/3512:
[ 1358.908299] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.918190] #1: ff11000151592c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.929152] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.940302] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.950395] 4 locks held by python3/5555:
[ 1358.955691] #0: ff1100012f09c3f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.965582] #1: ff110001d2075688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.976546] #2: ff11000111c210f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.987703] #3: ff11000111c01d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.997787]
[ 1359.000254] =============================================\x0a