[ 779.976722] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 779.979518] nipa-hw-worker: [4/40] Retrying drivers/net:napi_threaded.py
[ 780.111114] kselftest: Running tests in drivers/net
[ 989.659622] INFO: task python3:5572 blocked for more than 122 seconds.
[ 989.667855] Not tainted 7.0.0-rc6-ocqi-g88b19b5f0e96 #1
[ 989.675106] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.684693] task:python3 state:D stack:25336 pid:5572 tgid:5572 ppid:5571 task_flags:0x400100 flags:0x00080000
[ 989.697924] Call Trace:
[ 989.701473]
[ 989.704624] __schedule+0xae3/0x14d0
[ 989.709438] ? io_schedule_timeout+0x140/0x140
[ 989.715214] ? find_held_lock+0x2b/0x80
[ 989.720317] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.726203] schedule+0xd4/0x260
[ 989.730620] schedule_preempt_disabled+0x18/0x30
[ 989.736598] __mutex_lock+0x6b2/0x1ee0
[ 989.741603] ? rcu_is_watching+0x15/0xd0
[ 989.746801] ? threaded_store+0x132/0x220
[ 989.752099] ? ww_mutex_lock+0x160/0x160
[ 989.757300] ? selinux_inode_free_security+0x330/0x330
[ 989.763872] ? cap_capable+0x1ed/0x3f0
[ 989.763880] ? threaded_store+0x132/0x220
[ 989.763884] threaded_store+0x132/0x220
[ 989.763888] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.763891] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.789939] ? sysfs_kf_write+0x65/0x160
[ 989.795140] ? sysfs_file_ops+0x140/0x140
[ 989.800434] kernfs_fop_write_iter+0x335/0x510
[ 989.806218] ? kernfs_ops+0xf0/0xf0
[ 989.810928] new_sync_write+0x333/0x750
[ 989.816031] ? __lock_acquire+0x508/0xc10
[ 989.821325] ? new_sync_read+0x740/0x740
[ 989.826526] ? lock_acquire.part.0+0xbc/0x260
[ 989.832212] ? ksys_write+0x116/0x250
[ 989.837119] vfs_write+0x657/0x9b0
[ 989.841731] ksys_write+0x116/0x250
[ 989.846442] ? __ia32_sys_read+0xc0/0xc0
[ 989.851639] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.857226] ? rcu_is_watching+0x15/0xd0
[ 989.862424] do_syscall_64+0xf3/0x5e0
[ 989.867328] ? trace_hardirqs_off+0xd/0x30
[ 989.872723] ? exc_page_fault+0xda/0xf0
[ 989.877825] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.884304] RIP: 0033:0x7f518f65b22e
[ 989.889113] RSP: 002b:00007ffc39b3f7e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.898412] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f518f65b22e
[ 989.907217] RDX: 0000000000000001 RSI: 000055d6f6eaf340 RDI: 0000000000000007
[ 989.916025] RBP: 00007ffc39b3f7f0 R08: 0000000000000000 R09: 0000000000000000
[ 989.924836] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f518f4d3390
[ 989.933643] R13: 000055d6f6eaf340 R14: 0000000000000007 R15: 0000000000000003
[ 989.942460]
[ 989.945797] INFO: task python3:5572 is blocked on a mutex likely owned by task python3:3493.
[ 989.956069] task:python3 state:D stack:25336 pid:3493 tgid:3493 ppid:3492 task_flags:0x400100 flags:0x00080802
[ 989.969270] Call Trace:
[ 989.972806]
[ 989.975956] __schedule+0xae3/0x14d0
[ 989.980766] ? io_schedule_timeout+0x140/0x140
[ 989.986548] ? find_held_lock+0x2b/0x80
[ 989.991651] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.997827] schedule+0xd4/0x260
[ 990.002234] schedule_timeout+0x13f/0x290
[ 990.007528] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 990.013898] ? trace_contention_end+0xb3/0x1a0
[ 990.019682] ? hrtimers_cpu_dying+0x260/0x260
[ 990.025373] ? ww_mutex_lock+0x160/0x160
[ 990.030573] msleep+0x1b/0x30
[ 990.034697] napi_set_threaded+0x3a5/0x440
[ 990.040092] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.045587] netif_set_threaded+0x257/0x400
[ 990.051079] threaded_store+0x1b8/0x220
[ 990.056184] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.061479] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.066773] ? sysfs_kf_write+0x65/0x160
[ 990.071972] ? sysfs_file_ops+0x140/0x140
[ 990.077266] kernfs_fop_write_iter+0x335/0x510
[ 990.083051] ? kernfs_ops+0xf0/0xf0
[ 990.087753] new_sync_write+0x333/0x750
[ 990.092854] ? __lock_acquire+0x508/0xc10
[ 990.098148] ? new_sync_read+0x740/0x740
[ 990.103346] ? lock_acquire.part.0+0xbc/0x260
[ 990.109030] ? ksys_write+0x116/0x250
[ 990.113938] vfs_write+0x657/0x9b0
[ 990.118546] ksys_write+0x116/0x250
[ 990.123257] ? __ia32_sys_read+0xc0/0xc0
[ 990.128456] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.134037] ? rcu_is_watching+0x15/0xd0
[ 990.139234] do_syscall_64+0xf3/0x5e0
[ 990.144141] ? irq_exit_rcu+0x39/0xf0
[ 990.149049] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.155512] RIP: 0033:0x7fa1b32de22e
[ 990.160318] RSP: 002b:00007ffdd946c2b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.169615] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa1b32de22e
[ 990.178423] RDX: 0000000000000001 RSI: 000055a403dd1340 RDI: 0000000000000007
[ 990.187233] RBP: 00007ffdd946c2c0 R08: 0000000000000000 R09: 0000000000000000
[ 990.196040] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa1b3156390
[ 990.204846] R13: 000055a403dd1340 R14: 0000000000000007 R15: 0000000000000003
[ 990.213659]
[ 990.216919] \x0aShowing all locks held in the system:
[ 990.225463] 1 lock held by khungtaskd/185:
[ 990.230852] #0: ffffffffb98cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.242326] 4 locks held by systemd-journal/801:
[ 990.248304] #0: ff1100180e8372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 990.258191] #1: ffffffffb9ac8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: link_path_walk+0x9a7/0x1830
[ 990.270810] #2: ff1100019bf0a4f0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.282357] #3: ff1100017e00c268 (&xa->xa_lock#4){-.-.}-{3:3}, at: __folio_mark_dirty+0x9d/0x2c0
[ 990.293135] 4 locks held by python3/3493:
[ 990.298430] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.308320] #1: ff1100016548e888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.319284] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.330444] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.340549] 4 locks held by python3/5572:
[ 990.345845] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.355733] #1: ff110001b0ec8888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.366683] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.377837] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.387925] 5 locks held by (systemd-user-)/6267:
[ 990.394001] #0: ff1100012fad13f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.403890] #1: ff110001b88ff488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.414849] #2: ffffffffb9937108 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0
[ 990.425517] #3: ffffffffb9768b50 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x6b0
[ 990.437158] #4: ffffffffb9936e10 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x6b0
[ 990.449586]
[ 990.452051] =============================================\x0a
[ 1112.539871] INFO: task python3:5572 blocked for more than 245 seconds.
[ 1112.548018] Not tainted 7.0.0-rc6-ocqi-g88b19b5f0e96 #1
[ 1112.555268] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.564859] task:python3 state:D stack:25336 pid:5572 tgid:5572 ppid:5571 task_flags:0x400100 flags:0x00080000
[ 1112.578065] Call Trace:
[ 1112.581605]
[ 1112.584759] __schedule+0xae3/0x14d0
[ 1112.589578] ? io_schedule_timeout+0x140/0x140
[ 1112.595363] ? find_held_lock+0x2b/0x80
[ 1112.600472] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.606355] schedule+0xd4/0x260
[ 1112.610768] schedule_preempt_disabled+0x18/0x30
[ 1112.616748] __mutex_lock+0x6b2/0x1ee0
[ 1112.621757] ? rcu_is_watching+0x15/0xd0
[ 1112.626958] ? threaded_store+0x132/0x220
[ 1112.632260] ? ww_mutex_lock+0x160/0x160
[ 1112.637460] ? selinux_inode_free_security+0x330/0x330
[ 1112.644029] ? cap_capable+0x1ed/0x3f0
[ 1112.649037] ? threaded_store+0x132/0x220
[ 1112.654334] threaded_store+0x132/0x220
[ 1112.659430] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.664731] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.670028] ? sysfs_kf_write+0x65/0x160
[ 1112.675229] ? sysfs_file_ops+0x140/0x140
[ 1112.680526] kernfs_fop_write_iter+0x335/0x510
[ 1112.686305] ? kernfs_ops+0xf0/0xf0
[ 1112.691016] new_sync_write+0x333/0x750
[ 1112.696120] ? __lock_acquire+0x508/0xc10
[ 1112.701410] ? new_sync_read+0x740/0x740
[ 1112.706612] ? lock_acquire.part.0+0xbc/0x260
[ 1112.712298] ? ksys_write+0x116/0x250
[ 1112.717208] vfs_write+0x657/0x9b0
[ 1112.721824] ksys_write+0x116/0x250
[ 1112.726537] ? __ia32_sys_read+0xc0/0xc0
[ 1112.731730] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.737322] ? rcu_is_watching+0x15/0xd0
[ 1112.742520] do_syscall_64+0xf3/0x5e0
[ 1112.747428] ? trace_hardirqs_off+0xd/0x30
[ 1112.752823] ? exc_page_fault+0xda/0xf0
[ 1112.757925] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.764385] RIP: 0033:0x7f518f65b22e
[ 1112.769196] RSP: 002b:00007ffc39b3f7e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.778494] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f518f65b22e
[ 1112.787304] RDX: 0000000000000001 RSI: 000055d6f6eaf340 RDI: 0000000000000007
[ 1112.796114] RBP: 00007ffc39b3f7f0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.804923] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f518f4d3390
[ 1112.813731] R13: 000055d6f6eaf340 R14: 0000000000000007 R15: 0000000000000003
[ 1112.822544]
[ 1112.825815] INFO: task python3:5572 is blocked on a mutex likely owned by task python3:3493.
[ 1112.836090] task:python3 state:D stack:25336 pid:3493 tgid:3493 ppid:3492 task_flags:0x400100 flags:0x00080802
[ 1112.849284] Call Trace:
[ 1112.852822]
[ 1112.855974] __schedule+0xae3/0x14d0
[ 1112.860787] ? io_schedule_timeout+0x140/0x140
[ 1112.866572] ? find_held_lock+0x2b/0x80
[ 1112.871668] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.877845] schedule+0xd4/0x260
[ 1112.882263] schedule_timeout+0x13f/0x290
[ 1112.887561] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1112.893922] ? trace_contention_end+0xb3/0x1a0
[ 1112.899712] ? hrtimers_cpu_dying+0x260/0x260
[ 1112.905404] ? ww_mutex_lock+0x160/0x160
[ 1112.910605] msleep+0x1b/0x30
[ 1112.914733] napi_set_threaded+0x3a5/0x440
[ 1112.920132] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.925628] netif_set_threaded+0x257/0x400
[ 1112.931122] threaded_store+0x1b8/0x220
[ 1112.936227] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.941525] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.946811] ? sysfs_kf_write+0x65/0x160
[ 1112.952011] ? sysfs_file_ops+0x140/0x140
[ 1112.957308] kernfs_fop_write_iter+0x335/0x510
[ 1112.963092] ? kernfs_ops+0xf0/0xf0
[ 1112.967803] new_sync_write+0x333/0x750
[ 1112.972907] ? __lock_acquire+0x508/0xc10
[ 1112.978203] ? new_sync_read+0x740/0x740
[ 1112.983404] ? lock_acquire.part.0+0xbc/0x260
[ 1112.989095] ? ksys_write+0x116/0x250
[ 1112.994008] vfs_write+0x657/0x9b0
[ 1112.998624] ksys_write+0x116/0x250
[ 1113.003337] ? __ia32_sys_read+0xc0/0xc0
[ 1113.008534] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.014125] ? rcu_is_watching+0x15/0xd0
[ 1113.019328] do_syscall_64+0xf3/0x5e0
[ 1113.024233] ? irq_exit_rcu+0x39/0xf0
[ 1113.029142] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.035610] RIP: 0033:0x7fa1b32de22e
[ 1113.040409] RSP: 002b:00007ffdd946c2b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.049704] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa1b32de22e
[ 1113.058515] RDX: 0000000000000001 RSI: 000055a403dd1340 RDI: 0000000000000007
[ 1113.067323] RBP: 00007ffdd946c2c0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.076131] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa1b3156390
[ 1113.084939] R13: 000055a403dd1340 R14: 0000000000000007 R15: 0000000000000003
[ 1113.093745]
[ 1113.097009] \x0aShowing all locks held in the system:
[ 1113.105547] 1 lock held by khungtaskd/185:
[ 1113.110938] #0: ffffffffb98cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.122419] 4 locks held by python3/3493:
[ 1113.127720] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.137611] #1: ff1100016548e888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.148572] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.159728] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.169822] 4 locks held by python3/5572:
[ 1113.175119] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.185010] #1: ff110001b0ec8888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.195971] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.207127] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.217221]
[ 1113.219688] =============================================\x0a
[ 1212.057485] block nvme0n1: No UUID available providing old NGUID
SUBSYSTEM=block
DEVICE=b259:0
[ 1235.421137] INFO: task python3:5572 blocked for more than 368 seconds.
[ 1235.429286] Not tainted 7.0.0-rc6-ocqi-g88b19b5f0e96 #1
[ 1235.436536] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.446126] task:python3 state:D stack:25336 pid:5572 tgid:5572 ppid:5571 task_flags:0x400100 flags:0x00080002
[ 1235.459336] Call Trace:
[ 1235.462876]
[ 1235.466024] __schedule+0xae3/0x14d0
[ 1235.470843] ? io_schedule_timeout+0x140/0x140
[ 1235.476628] ? find_held_lock+0x2b/0x80
[ 1235.481736] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.487622] schedule+0xd4/0x260
[ 1235.492039] schedule_preempt_disabled+0x18/0x30
[ 1235.498022] __mutex_lock+0x6b2/0x1ee0
[ 1235.503029] ? rcu_is_watching+0x15/0xd0
[ 1235.508228] ? threaded_store+0x132/0x220
[ 1235.513530] ? ww_mutex_lock+0x160/0x160
[ 1235.518728] ? selinux_inode_free_security+0x330/0x330
[ 1235.525291] ? cap_capable+0x1ed/0x3f0
[ 1235.530369] ? threaded_store+0x132/0x220
[ 1235.535667] threaded_store+0x132/0x220
[ 1235.540769] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.546066] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.551364] ? sysfs_kf_write+0x65/0x160
[ 1235.556565] ? sysfs_file_ops+0x140/0x140
[ 1235.561860] kernfs_fop_write_iter+0x335/0x510
[ 1235.567668] ? kernfs_ops+0xf0/0xf0
[ 1235.572411] new_sync_write+0x333/0x750
[ 1235.577517] ? __lock_acquire+0x508/0xc10
[ 1235.582884] ? new_sync_read+0x740/0x740
[ 1235.588086] ? lock_acquire.part.0+0xbc/0x260
[ 1235.593772] ? ksys_write+0x116/0x250
[ 1235.598682] vfs_write+0x657/0x9b0
[ 1235.603296] ksys_write+0x116/0x250
[ 1235.608006] ? __ia32_sys_read+0xc0/0xc0
[ 1235.613205] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.618793] ? rcu_is_watching+0x15/0xd0
[ 1235.623995] do_syscall_64+0xf3/0x5e0
[ 1235.628902] ? trace_hardirqs_off+0xd/0x30
[ 1235.634297] ? exc_page_fault+0xda/0xf0
[ 1235.639399] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.645867] RIP: 0033:0x7f518f65b22e
[ 1235.650676] RSP: 002b:00007ffc39b3f7e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.659971] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f518f65b22e
[ 1235.668780] RDX: 0000000000000001 RSI: 000055d6f6eaf340 RDI: 0000000000000007
[ 1235.677588] RBP: 00007ffc39b3f7f0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.686397] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f518f4d3390
[ 1235.695205] R13: 000055d6f6eaf340 R14: 0000000000000007 R15: 0000000000000003
[ 1235.704021]
[ 1235.707372] INFO: task python3:5572 is blocked on a mutex likely owned by task python3:3493.
[ 1235.717645] task:python3 state:D stack:25336 pid:3493 tgid:3493 ppid:3492 task_flags:0x400100 flags:0x00080802
[ 1235.730850] Call Trace:
[ 1235.734390]
[ 1235.737540] __schedule+0xae3/0x14d0
[ 1235.742352] ? io_schedule_timeout+0x140/0x140
[ 1235.748136] ? find_held_lock+0x2b/0x80
[ 1235.753240] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.759417] schedule+0xd4/0x260
[ 1235.763834] schedule_timeout+0x13f/0x290
[ 1235.769132] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1235.775503] ? trace_contention_end+0xb3/0x1a0
[ 1235.781286] ? hrtimers_cpu_dying+0x260/0x260
[ 1235.786980] ? ww_mutex_lock+0x160/0x160
[ 1235.792182] msleep+0x1b/0x30
[ 1235.796306] napi_set_threaded+0x3a5/0x440
[ 1235.801700] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.807196] netif_set_threaded+0x257/0x400
[ 1235.812687] threaded_store+0x1b8/0x220
[ 1235.817790] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.823091] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.828386] ? sysfs_kf_write+0x65/0x160
[ 1235.833584] ? sysfs_file_ops+0x140/0x140
[ 1235.838881] kernfs_fop_write_iter+0x335/0x510
[ 1235.844664] ? kernfs_ops+0xf0/0xf0
[ 1235.849375] new_sync_write+0x333/0x750
[ 1235.854477] ? __lock_acquire+0x508/0xc10
[ 1235.859831] ? new_sync_read+0x740/0x740
[ 1235.865033] ? lock_acquire.part.0+0xbc/0x260
[ 1235.870720] ? ksys_write+0x116/0x250
[ 1235.875630] vfs_write+0x657/0x9b0
[ 1235.880247] ksys_write+0x116/0x250
[ 1235.884959] ? __ia32_sys_read+0xc0/0xc0
[ 1235.890149] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.895739] ? rcu_is_watching+0x15/0xd0
[ 1235.900941] do_syscall_64+0xf3/0x5e0
[ 1235.905911] ? irq_exit_rcu+0x39/0xf0
[ 1235.910819] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.917288] RIP: 0033:0x7fa1b32de22e
[ 1235.922095] RSP: 002b:00007ffdd946c2b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.931394] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa1b32de22e
[ 1235.940204] RDX: 0000000000000001 RSI: 000055a403dd1340 RDI: 0000000000000007
[ 1235.949013] RBP: 00007ffdd946c2c0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.957824] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa1b3156390
[ 1235.966833] R13: 000055a403dd1340 R14: 0000000000000007 R15: 0000000000000003
[ 1235.975647]
[ 1235.978916] \x0aShowing all locks held in the system:
[ 1235.987458] 1 lock held by khungtaskd/185:
[ 1235.992844] #0: ffffffffb98cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.004329] 4 locks held by python3/3493:
[ 1236.009623] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.019513] #1: ff1100016548e888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.030473] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.041630] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.051724] 4 locks held by python3/5572:
[ 1236.057020] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.066907] #1: ff110001b0ec8888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.077870] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.089029] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.099121] 1 lock held by systemd/7081:
[ 1236.104320]
[ 1236.106786] =============================================\x0a
[ 1358.302377] INFO: task python3:5572 blocked for more than 491 seconds.
[ 1358.310526] Not tainted 7.0.0-rc6-ocqi-g88b19b5f0e96 #1
[ 1358.317778] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.327370] task:python3 state:D stack:25336 pid:5572 tgid:5572 ppid:5571 task_flags:0x400100 flags:0x00080002
[ 1358.340580] Call Trace:
[ 1358.344122]
[ 1358.347276] __schedule+0xae3/0x14d0
[ 1358.352097] ? io_schedule_timeout+0x140/0x140
[ 1358.357884] ? find_held_lock+0x2b/0x80
[ 1358.362990] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.368915] schedule+0xd4/0x260
[ 1358.373551] schedule_preempt_disabled+0x18/0x30
[ 1358.379533] __mutex_lock+0x6b2/0x1ee0
[ 1358.384539] ? rcu_is_watching+0x15/0xd0
[ 1358.389740] ? threaded_store+0x132/0x220
[ 1358.395041] ? ww_mutex_lock+0x160/0x160
[ 1358.400239] ? selinux_inode_free_security+0x330/0x330
[ 1358.406811] ? cap_capable+0x1ed/0x3f0
[ 1358.411822] ? threaded_store+0x132/0x220
[ 1358.417117] threaded_store+0x132/0x220
[ 1358.422219] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.427520] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.432820] ? sysfs_kf_write+0x65/0x160
[ 1358.438022] ? sysfs_file_ops+0x140/0x140
[ 1358.443320] kernfs_fop_write_iter+0x335/0x510
[ 1358.449107] ? kernfs_ops+0xf0/0xf0
[ 1358.453821] new_sync_write+0x333/0x750
[ 1358.458926] ? __lock_acquire+0x508/0xc10
[ 1358.464229] ? new_sync_read+0x740/0x740
[ 1358.469429] ? lock_acquire.part.0+0xbc/0x260
[ 1358.475108] ? ksys_write+0x116/0x250
[ 1358.480019] vfs_write+0x657/0x9b0
[ 1358.484632] ksys_write+0x116/0x250
[ 1358.489344] ? __ia32_sys_read+0xc0/0xc0
[ 1358.494544] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.500125] ? rcu_is_watching+0x15/0xd0
[ 1358.505319] do_syscall_64+0xf3/0x5e0
[ 1358.510232] ? trace_hardirqs_off+0xd/0x30
[ 1358.515628] ? exc_page_fault+0xda/0xf0
[ 1358.520730] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.527187] RIP: 0033:0x7f518f65b22e
[ 1358.531988] RSP: 002b:00007ffc39b3f7e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.541289] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f518f65b22e
[ 1358.550098] RDX: 0000000000000001 RSI: 000055d6f6eaf340 RDI: 0000000000000007
[ 1358.558906] RBP: 00007ffc39b3f7f0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.567715] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f518f4d3390
[ 1358.576526] R13: 000055d6f6eaf340 R14: 0000000000000007 R15: 0000000000000003
[ 1358.585342]
[ 1358.588624] INFO: task python3:5572 is blocked on a mutex likely owned by task python3:3493.
[ 1358.598898] task:python3 state:D stack:25336 pid:3493 tgid:3493 ppid:3492 task_flags:0x400100 flags:0x00080802
[ 1358.612102] Call Trace:
[ 1358.615643]
[ 1358.618794] __schedule+0xae3/0x14d0
[ 1358.623597] ? io_schedule_timeout+0x140/0x140
[ 1358.629379] ? find_held_lock+0x2b/0x80
[ 1358.634483] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.640664] schedule+0xd4/0x260
[ 1358.645086] schedule_timeout+0x13f/0x290
[ 1358.650386] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1358.656759] ? trace_contention_end+0xb3/0x1a0
[ 1358.662543] ? hrtimers_cpu_dying+0x260/0x260
[ 1358.668238] ? ww_mutex_lock+0x160/0x160
[ 1358.673438] msleep+0x1b/0x30
[ 1358.677563] napi_set_threaded+0x3a5/0x440
[ 1358.682959] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.688455] netif_set_threaded+0x257/0x400
[ 1358.693946] threaded_store+0x1b8/0x220
[ 1358.699049] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.704349] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.709647] ? sysfs_kf_write+0x65/0x160
[ 1358.714849] ? sysfs_file_ops+0x140/0x140
[ 1358.720148] kernfs_fop_write_iter+0x335/0x510
[ 1358.725934] ? kernfs_ops+0xf0/0xf0
[ 1358.730638] new_sync_write+0x333/0x750
[ 1358.735739] ? __lock_acquire+0x508/0xc10
[ 1358.741036] ? new_sync_read+0x740/0x740
[ 1358.746240] ? lock_acquire.part.0+0xbc/0x260
[ 1358.751929] ? ksys_write+0x116/0x250
[ 1358.756839] vfs_write+0x657/0x9b0
[ 1358.761455] ksys_write+0x116/0x250
[ 1358.766169] ? __ia32_sys_read+0xc0/0xc0
[ 1358.771369] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.776958] ? rcu_is_watching+0x15/0xd0
[ 1358.782159] do_syscall_64+0xf3/0x5e0
[ 1358.787063] ? irq_exit_rcu+0x39/0xf0
[ 1358.791974] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.798444] RIP: 0033:0x7fa1b32de22e
[ 1358.803256] RSP: 002b:00007ffdd946c2b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.812555] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa1b32de22e
[ 1358.821367] RDX: 0000000000000001 RSI: 000055a403dd1340 RDI: 0000000000000007
[ 1358.830178] RBP: 00007ffdd946c2c0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.838985] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa1b3156390
[ 1358.847787] R13: 000055a403dd1340 R14: 0000000000000007 R15: 0000000000000003
[ 1358.856603]
[ 1358.859876] \x0aShowing all locks held in the system:
[ 1358.868413] 1 lock held by khungtaskd/185:
[ 1358.873808] #0: ffffffffb98cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.885291] 4 locks held by python3/3493:
[ 1358.890588] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.900472] #1: ff1100016548e888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.911435] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.922593] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.932680] 4 locks held by python3/5572:
[ 1358.937975] #0: ff1100012f9d13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.947865] #1: ff110001b0ec8888 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.958827] #2: ff11000122d230f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.969977] #3: ff11000122d91d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.980075]
[ 1358.982555] =============================================\x0a