[ 769.487346] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 769.515329] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 769.641873] kselftest: Running tests in drivers/net
[ 989.708270] INFO: task python3:5593 blocked for more than 122 seconds.
[ 989.716508] Not tainted 7.1.0-rc7-ffhv-ged76b27e5ef1 #1
[ 989.724431] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.734020] task:python3 state:D stack:25672 pid:5593 tgid:5593 ppid:5592 task_flags:0x400100 flags:0x00080000
[ 989.747225] Call Trace:
[ 989.750765]
[ 989.753915] __schedule+0xcc3/0x19c0
[ 989.758732] ? io_schedule_timeout+0x140/0x140
[ 989.764514] ? find_held_lock+0x2b/0x80
[ 989.769619] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.775503] schedule+0xd8/0x260
[ 989.779920] schedule_preempt_disabled+0x18/0x30
[ 989.785897] __mutex_lock+0x6af/0x2010
[ 989.790903] ? threaded_store+0x132/0x220
[ 989.796199] ? ww_mutex_lock+0x160/0x160
[ 989.801397] ? selinux_inode_free_security+0x330/0x330
[ 989.807957] ? cap_capable+0x1d6/0x3e0
[ 989.813046] ? threaded_store+0x132/0x220
[ 989.818344] threaded_store+0x132/0x220
[ 989.823436] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.828733] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.834030] ? sysfs_kf_write+0x65/0x160
[ 989.839229] ? sysfs_file_ops+0x140/0x140
[ 989.844526] kernfs_fop_write_iter+0x335/0x510
[ 989.850308] ? kernfs_ops+0xf0/0xf0
[ 989.855019] new_sync_write+0x333/0x750
[ 989.860114] ? __lock_acquire+0x508/0xc10
[ 989.865409] ? new_sync_read+0x740/0x740
[ 989.870610] ? lock_acquire.part.0+0xbc/0x260
[ 989.876295] ? ksys_write+0x116/0x250
[ 989.881203] vfs_write+0x657/0x9b0
[ 989.885818] ksys_write+0x116/0x250
[ 989.890520] ? __ia32_sys_read+0xc0/0xc0
[ 989.895717] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.901307] ? rcu_is_watching+0x15/0xd0
[ 989.906508] do_syscall_64+0xf3/0x650
[ 989.911417] ? irq_exit_rcu+0x1a/0x30
[ 989.916327] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.922795] RIP: 0033:0x7f3448d3a64e
[ 989.927605] RSP: 002b:00007fff7278a270 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.936903] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3448d3a64e
[ 989.945712] RDX: 0000000000000001 RSI: 0000560ce561bbf0 RDI: 0000000000000007
[ 989.954522] RBP: 00007fff7278a280 R08: 0000000000000000 R09: 0000000000000000
[ 989.963327] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3448bb3390
[ 989.972134] R13: 0000560ce561bbf0 R14: 0000000000000007 R15: 00007f3448b49120
[ 989.980944]
[ 989.984271] INFO: task python3:5593 is blocked on a mutex likely owned by task python3:3562.
[ 989.994604] task:python3 state:D stack:25672 pid:3562 tgid:3562 ppid:3561 task_flags:0x400100 flags:0x00080802
[ 990.007804] Call Trace:
[ 990.011344]
[ 990.014492] __schedule+0xcc3/0x19c0
[ 990.019303] ? io_schedule_timeout+0x140/0x140
[ 990.025084] ? find_held_lock+0x2b/0x80
[ 990.030184] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.036361] schedule+0xd8/0x260
[ 990.040781] schedule_timeout+0x13f/0x290
[ 990.046077] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.052443] ? trace_contention_end+0xb3/0x1a0
[ 990.058228] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.063916] ? ww_mutex_lock+0x160/0x160
[ 990.069117] msleep+0x1b/0x30
[ 990.073233] napi_set_threaded+0x3a5/0x440
[ 990.078631] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.084125] netif_set_threaded+0x257/0x400
[ 990.089616] threaded_store+0x1b8/0x220
[ 990.094715] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.100010] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.105308] ? sysfs_kf_write+0x65/0x160
[ 990.110500] ? sysfs_file_ops+0x140/0x140
[ 990.115858] kernfs_fop_write_iter+0x335/0x510
[ 990.121633] ? kernfs_ops+0xf0/0xf0
[ 990.126343] new_sync_write+0x333/0x750
[ 990.131445] ? __lock_acquire+0x508/0xc10
[ 990.136739] ? new_sync_read+0x740/0x740
[ 990.141937] ? lock_acquire.part.0+0xbc/0x260
[ 990.147623] ? ksys_write+0x116/0x250
[ 990.152533] vfs_write+0x657/0x9b0
[ 990.157148] ksys_write+0x116/0x250
[ 990.161981] ? __ia32_sys_read+0xc0/0xc0
[ 990.167177] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.172765] ? rcu_is_watching+0x15/0xd0
[ 990.177963] do_syscall_64+0xf3/0x650
[ 990.182872] ? trace_hardirqs_off+0xd/0x30
[ 990.188266] ? exc_page_fault+0xda/0xf0
[ 990.193365] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.199832] RIP: 0033:0x7fdb2d66164e
[ 990.204639] RSP: 002b:00007fffd8bca580 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.213934] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb2d66164e
[ 990.222743] RDX: 0000000000000001 RSI: 000055650bbc3bf0 RDI: 0000000000000007
[ 990.231549] RBP: 00007fffd8bca590 R08: 0000000000000000 R09: 0000000000000000
[ 990.240357] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb2d4da390
[ 990.249164] R13: 000055650bbc3bf0 R14: 0000000000000007 R15: 00007fdb2d46d120
[ 990.257975]
[ 990.261234] \x0aShowing all locks held in the system:
[ 990.269775] 1 lock held by khungtaskd/185:
[ 990.275168] #0: ffffffffa32fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.286658] 4 locks held by python3/3562:
[ 990.292028] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.301977] #1: ff1100019ff2d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.312938] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.324095] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.334200] 4 locks held by python3/5593:
[ 990.339493] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.349390] #1: ff110001ae423280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.360523] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.371687] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.381832] 1 lock held by systemd/6263:
[ 990.387091]
[ 990.389555] =============================================\x0a
[ 1092.110538] kworker/u129:9 (290) used greatest stack depth: 21936 bytes left
[ 1112.590628] INFO: task python3:5593 blocked for more than 245 seconds.
[ 1112.598777] Not tainted 7.1.0-rc7-ffhv-ged76b27e5ef1 #1
[ 1112.606028] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.615618] task:python3 state:D stack:25672 pid:5593 tgid:5593 ppid:5592 task_flags:0x400100 flags:0x00080000
[ 1112.628824] Call Trace:
[ 1112.632367]
[ 1112.635517] __schedule+0xcc3/0x19c0
[ 1112.640337] ? io_schedule_timeout+0x140/0x140
[ 1112.646122] ? find_held_lock+0x2b/0x80
[ 1112.651225] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.657111] schedule+0xd8/0x260
[ 1112.661530] schedule_preempt_disabled+0x18/0x30
[ 1112.667508] __mutex_lock+0x6af/0x2010
[ 1112.672513] ? threaded_store+0x132/0x220
[ 1112.677813] ? ww_mutex_lock+0x160/0x160
[ 1112.683014] ? selinux_inode_free_security+0x330/0x330
[ 1112.689585] ? cap_capable+0x1d6/0x3e0
[ 1112.694595] ? threaded_store+0x132/0x220
[ 1112.699890] threaded_store+0x132/0x220
[ 1112.704992] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.710289] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.715591] ? sysfs_kf_write+0x65/0x160
[ 1112.720792] ? sysfs_file_ops+0x140/0x140
[ 1112.726090] kernfs_fop_write_iter+0x335/0x510
[ 1112.731875] ? kernfs_ops+0xf0/0xf0
[ 1112.736577] new_sync_write+0x333/0x750
[ 1112.741674] ? __lock_acquire+0x508/0xc10
[ 1112.746971] ? new_sync_read+0x740/0x740
[ 1112.752166] ? lock_acquire.part.0+0xbc/0x260
[ 1112.757854] ? ksys_write+0x116/0x250
[ 1112.762755] vfs_write+0x657/0x9b0
[ 1112.767369] ksys_write+0x116/0x250
[ 1112.772078] ? __ia32_sys_read+0xc0/0xc0
[ 1112.777277] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.782865] ? rcu_is_watching+0x15/0xd0
[ 1112.788068] do_syscall_64+0xf3/0x650
[ 1112.792974] ? irq_exit_rcu+0x1a/0x30
[ 1112.797883] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.804350] RIP: 0033:0x7f3448d3a64e
[ 1112.809152] RSP: 002b:00007fff7278a270 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.818451] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3448d3a64e
[ 1112.827263] RDX: 0000000000000001 RSI: 0000560ce561bbf0 RDI: 0000000000000007
[ 1112.836070] RBP: 00007fff7278a280 R08: 0000000000000000 R09: 0000000000000000
[ 1112.844881] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3448bb3390
[ 1112.853688] R13: 0000560ce561bbf0 R14: 0000000000000007 R15: 00007f3448b49120
[ 1112.862501]
[ 1112.865774] INFO: task python3:5593 is blocked on a mutex likely owned by task python3:3562.
[ 1112.876050] task:python3 state:D stack:25672 pid:3562 tgid:3562 ppid:3561 task_flags:0x400100 flags:0x00080802
[ 1112.889251] Call Trace:
[ 1112.892791]
[ 1112.895944] __schedule+0xcc3/0x19c0
[ 1112.900754] ? io_schedule_timeout+0x140/0x140
[ 1112.906540] ? find_held_lock+0x2b/0x80
[ 1112.911642] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.917820] schedule+0xd8/0x260
[ 1112.922239] schedule_timeout+0x13f/0x290
[ 1112.927537] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.933900] ? trace_contention_end+0xb3/0x1a0
[ 1112.939683] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.945371] ? ww_mutex_lock+0x160/0x160
[ 1112.950573] msleep+0x1b/0x30
[ 1112.954696] napi_set_threaded+0x3a5/0x440
[ 1112.960092] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.965587] netif_set_threaded+0x257/0x400
[ 1112.971080] threaded_store+0x1b8/0x220
[ 1112.976172] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.981474] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.986771] ? sysfs_kf_write+0x65/0x160
[ 1112.991973] ? sysfs_file_ops+0x140/0x140
[ 1112.997267] kernfs_fop_write_iter+0x335/0x510
[ 1113.003050] ? kernfs_ops+0xf0/0xf0
[ 1113.007763] new_sync_write+0x333/0x750
[ 1113.012863] ? __lock_acquire+0x508/0xc10
[ 1113.018160] ? new_sync_read+0x740/0x740
[ 1113.023354] ? lock_acquire.part.0+0xbc/0x260
[ 1113.029042] ? ksys_write+0x116/0x250
[ 1113.033943] vfs_write+0x657/0x9b0
[ 1113.038558] ksys_write+0x116/0x250
[ 1113.043269] ? __ia32_sys_read+0xc0/0xc0
[ 1113.048473] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.054062] ? rcu_is_watching+0x15/0xd0
[ 1113.059263] do_syscall_64+0xf3/0x650
[ 1113.064168] ? trace_hardirqs_off+0xd/0x30
[ 1113.069565] ? exc_page_fault+0xda/0xf0
[ 1113.074670] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.081137] RIP: 0033:0x7fdb2d66164e
[ 1113.085946] RSP: 002b:00007fffd8bca580 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.095236] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb2d66164e
[ 1113.104042] RDX: 0000000000000001 RSI: 000055650bbc3bf0 RDI: 0000000000000007
[ 1113.112851] RBP: 00007fffd8bca590 R08: 0000000000000000 R09: 0000000000000000
[ 1113.121658] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb2d4da390
[ 1113.130470] R13: 000055650bbc3bf0 R14: 0000000000000007 R15: 00007fdb2d46d120
[ 1113.139284]
[ 1113.142546] \x0aShowing all locks held in the system:
[ 1113.151082] 1 lock held by khungtaskd/185:
[ 1113.156477] #0: ffffffffa32fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.167949] 3 locks held by systemd-journal/804:
[ 1113.173930] #0: ff1100180ddb74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.183822] #1: ffffffffa3502ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.196640] #2: ff110001de6d44f0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.208198] 4 locks held by python3/3562:
[ 1113.213494] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.223385] #1: ff1100019ff2d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.234347] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.245495] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.255585] 4 locks held by python3/5593:
[ 1113.260875] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.270762] #1: ff110001ae423280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.281725] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.292883] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.302974]
[ 1113.305439] =============================================\x0a
[ 1235.471634] INFO: task python3:5593 blocked for more than 368 seconds.
[ 1235.479783] Not tainted 7.1.0-rc7-ffhv-ged76b27e5ef1 #1
[ 1235.487034] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.496627] task:python3 state:D stack:25672 pid:5593 tgid:5593 ppid:5592 task_flags:0x400100 flags:0x00080002
[ 1235.509833] Call Trace:
[ 1235.513373]
[ 1235.516524] __schedule+0xcc3/0x19c0
[ 1235.521337] ? io_schedule_timeout+0x140/0x140
[ 1235.527123] ? find_held_lock+0x2b/0x80
[ 1235.532224] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.538111] schedule+0xd8/0x260
[ 1235.542529] schedule_preempt_disabled+0x18/0x30
[ 1235.548508] __mutex_lock+0x6af/0x2010
[ 1235.553504] ? threaded_store+0x132/0x220
[ 1235.558804] ? ww_mutex_lock+0x160/0x160
[ 1235.564005] ? selinux_inode_free_security+0x330/0x330
[ 1235.570578] ? cap_capable+0x1d6/0x3e0
[ 1235.575589] ? threaded_store+0x132/0x220
[ 1235.580885] threaded_store+0x132/0x220
[ 1235.585988] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.591292] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.596592] ? sysfs_kf_write+0x65/0x160
[ 1235.601791] ? sysfs_file_ops+0x140/0x140
[ 1235.607089] kernfs_fop_write_iter+0x335/0x510
[ 1235.612876] ? kernfs_ops+0xf0/0xf0
[ 1235.617588] new_sync_write+0x333/0x750
[ 1235.622694] ? __lock_acquire+0x508/0xc10
[ 1235.627991] ? new_sync_read+0x740/0x740
[ 1235.633186] ? lock_acquire.part.0+0xbc/0x260
[ 1235.638874] ? ksys_write+0x116/0x250
[ 1235.643784] vfs_write+0x657/0x9b0
[ 1235.648398] ksys_write+0x116/0x250
[ 1235.653108] ? __ia32_sys_read+0xc0/0xc0
[ 1235.658310] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.663897] ? rcu_is_watching+0x15/0xd0
[ 1235.669099] do_syscall_64+0xf3/0x650
[ 1235.674005] ? irq_exit_rcu+0x1a/0x30
[ 1235.678911] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.685378] RIP: 0033:0x7f3448d3a64e
[ 1235.690188] RSP: 002b:00007fff7278a270 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.699489] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3448d3a64e
[ 1235.708296] RDX: 0000000000000001 RSI: 0000560ce561bbf0 RDI: 0000000000000007
[ 1235.717105] RBP: 00007fff7278a280 R08: 0000000000000000 R09: 0000000000000000
[ 1235.725914] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3448bb3390
[ 1235.734723] R13: 0000560ce561bbf0 R14: 0000000000000007 R15: 00007f3448b49120
[ 1235.743539]
[ 1235.746865] INFO: task python3:5593 is blocked on a mutex likely owned by task python3:3562.
[ 1235.757142] task:python3 state:D stack:25672 pid:3562 tgid:3562 ppid:3561 task_flags:0x400100 flags:0x00080802
[ 1235.770344] Call Trace:
[ 1235.773886]
[ 1235.777038] __schedule+0xcc3/0x19c0
[ 1235.781850] ? io_schedule_timeout+0x140/0x140
[ 1235.787634] ? find_held_lock+0x2b/0x80
[ 1235.792739] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.798918] schedule+0xd8/0x260
[ 1235.803336] schedule_timeout+0x13f/0x290
[ 1235.808633] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.815001] ? trace_contention_end+0xb3/0x1a0
[ 1235.820789] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.826486] ? ww_mutex_lock+0x160/0x160
[ 1235.831684] msleep+0x1b/0x30
[ 1235.835808] napi_set_threaded+0x3a5/0x440
[ 1235.841205] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.846702] netif_set_threaded+0x257/0x400
[ 1235.852196] threaded_store+0x1b8/0x220
[ 1235.857296] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.862594] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.867893] ? sysfs_kf_write+0x65/0x160
[ 1235.873093] ? sysfs_file_ops+0x140/0x140
[ 1235.878389] kernfs_fop_write_iter+0x335/0x510
[ 1235.884175] ? kernfs_ops+0xf0/0xf0
[ 1235.888885] new_sync_write+0x333/0x750
[ 1235.893988] ? __lock_acquire+0x508/0xc10
[ 1235.899286] ? new_sync_read+0x740/0x740
[ 1235.904492] ? lock_acquire.part.0+0xbc/0x260
[ 1235.910179] ? ksys_write+0x116/0x250
[ 1235.915090] vfs_write+0x657/0x9b0
[ 1235.919707] ksys_write+0x116/0x250
[ 1235.924418] ? __ia32_sys_read+0xc0/0xc0
[ 1235.929619] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.935210] ? rcu_is_watching+0x15/0xd0
[ 1235.940411] do_syscall_64+0xf3/0x650
[ 1235.945319] ? trace_hardirqs_off+0xd/0x30
[ 1235.950714] ? exc_page_fault+0xda/0xf0
[ 1235.955817] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.962276] RIP: 0033:0x7fdb2d66164e
[ 1235.967084] RSP: 002b:00007fffd8bca580 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.976380] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb2d66164e
[ 1235.985188] RDX: 0000000000000001 RSI: 000055650bbc3bf0 RDI: 0000000000000007
[ 1235.993988] RBP: 00007fffd8bca590 R08: 0000000000000000 R09: 0000000000000000
[ 1236.002795] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb2d4da390
[ 1236.011594] R13: 000055650bbc3bf0 R14: 0000000000000007 R15: 00007fdb2d46d120
[ 1236.020407]
[ 1236.023673] \x0aShowing all locks held in the system:
[ 1236.032216] 1 lock held by khungtaskd/185:
[ 1236.037611] #0: ffffffffa32fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.049107] 4 locks held by python3/3562:
[ 1236.054402] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.064293] #1: ff1100019ff2d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.075252] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.086411] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.096516] 4 locks held by python3/5593:
[ 1236.101811] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.111699] #1: ff110001ae423280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.122661] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.133809] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.143900]
[ 1236.146365] =============================================\x0a
[ 1358.352795] INFO: task python3:5593 blocked for more than 491 seconds.
[ 1358.360937] Not tainted 7.1.0-rc7-ffhv-ged76b27e5ef1 #1
[ 1358.368185] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.377774] task:python3 state:D stack:25672 pid:5593 tgid:5593 ppid:5592 task_flags:0x400100 flags:0x00080002
[ 1358.390981] Call Trace:
[ 1358.394523]
[ 1358.397673] __schedule+0xcc3/0x19c0
[ 1358.402489] ? io_schedule_timeout+0x140/0x140
[ 1358.408277] ? find_held_lock+0x2b/0x80
[ 1358.413383] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.419282] schedule+0xd8/0x260
[ 1358.423705] schedule_preempt_disabled+0x18/0x30
[ 1358.429684] __mutex_lock+0x6af/0x2010
[ 1358.434681] ? threaded_store+0x132/0x220
[ 1358.439970] ? ww_mutex_lock+0x160/0x160
[ 1358.445171] ? selinux_inode_free_security+0x330/0x330
[ 1358.451743] ? cap_capable+0x1d6/0x3e0
[ 1358.456753] ? threaded_store+0x132/0x220
[ 1358.462048] threaded_store+0x132/0x220
[ 1358.467150] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.472449] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.477748] ? sysfs_kf_write+0x65/0x160
[ 1358.482951] ? sysfs_file_ops+0x140/0x140
[ 1358.488245] kernfs_fop_write_iter+0x335/0x510
[ 1358.494028] ? kernfs_ops+0xf0/0xf0
[ 1358.498742] new_sync_write+0x333/0x750
[ 1358.503845] ? __lock_acquire+0x508/0xc10
[ 1358.509142] ? new_sync_read+0x740/0x740
[ 1358.514347] ? lock_acquire.part.0+0xbc/0x260
[ 1358.520034] ? ksys_write+0x116/0x250
[ 1358.524943] vfs_write+0x657/0x9b0
[ 1358.529556] ksys_write+0x116/0x250
[ 1358.534270] ? __ia32_sys_read+0xc0/0xc0
[ 1358.539467] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.545055] ? rcu_is_watching+0x15/0xd0
[ 1358.550257] do_syscall_64+0xf3/0x650
[ 1358.555166] ? irq_exit_rcu+0x1a/0x30
[ 1358.560072] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.566543] RIP: 0033:0x7f3448d3a64e
[ 1358.571353] RSP: 002b:00007fff7278a270 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.580648] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3448d3a64e
[ 1358.589456] RDX: 0000000000000001 RSI: 0000560ce561bbf0 RDI: 0000000000000007
[ 1358.598253] RBP: 00007fff7278a280 R08: 0000000000000000 R09: 0000000000000000
[ 1358.607059] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3448bb3390
[ 1358.615866] R13: 0000560ce561bbf0 R14: 0000000000000007 R15: 00007f3448b49120
[ 1358.624683]
[ 1358.627965] INFO: task python3:5593 is blocked on a mutex likely owned by task python3:3562.
[ 1358.638239] task:python3 state:D stack:25672 pid:3562 tgid:3562 ppid:3561 task_flags:0x400100 flags:0x00080802
[ 1358.651441] Call Trace:
[ 1358.654980]
[ 1358.658128] __schedule+0xcc3/0x19c0
[ 1358.662942] ? io_schedule_timeout+0x140/0x140
[ 1358.668725] ? find_held_lock+0x2b/0x80
[ 1358.673826] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.679999] schedule+0xd8/0x260
[ 1358.684420] schedule_timeout+0x13f/0x290
[ 1358.689718] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.696079] ? trace_contention_end+0xb3/0x1a0
[ 1358.701865] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.707554] ? ww_mutex_lock+0x160/0x160
[ 1358.712756] msleep+0x1b/0x30
[ 1358.716883] napi_set_threaded+0x3a5/0x440
[ 1358.722278] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.727772] netif_set_threaded+0x257/0x400
[ 1358.733269] threaded_store+0x1b8/0x220
[ 1358.738370] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.743665] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.748962] ? sysfs_kf_write+0x65/0x160
[ 1358.754160] ? sysfs_file_ops+0x140/0x140
[ 1358.759457] kernfs_fop_write_iter+0x335/0x510
[ 1358.765232] ? kernfs_ops+0xf0/0xf0
[ 1358.769945] new_sync_write+0x333/0x750
[ 1358.775049] ? __lock_acquire+0x508/0xc10
[ 1358.780347] ? new_sync_read+0x740/0x740
[ 1358.785550] ? lock_acquire.part.0+0xbc/0x260
[ 1358.791237] ? ksys_write+0x116/0x250
[ 1358.796146] vfs_write+0x657/0x9b0
[ 1358.800764] ksys_write+0x116/0x250
[ 1358.805474] ? __ia32_sys_read+0xc0/0xc0
[ 1358.810672] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.816260] ? rcu_is_watching+0x15/0xd0
[ 1358.821463] do_syscall_64+0xf3/0x650
[ 1358.826367] ? trace_hardirqs_off+0xd/0x30
[ 1358.831764] ? exc_page_fault+0xda/0xf0
[ 1358.836870] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.843336] RIP: 0033:0x7fdb2d66164e
[ 1358.848145] RSP: 002b:00007fffd8bca580 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.857434] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb2d66164e
[ 1358.866240] RDX: 0000000000000001 RSI: 000055650bbc3bf0 RDI: 0000000000000007
[ 1358.875051] RBP: 00007fffd8bca590 R08: 0000000000000000 R09: 0000000000000000
[ 1358.883856] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb2d4da390
[ 1358.892667] R13: 000055650bbc3bf0 R14: 0000000000000007 R15: 00007fdb2d46d120
[ 1358.901480]
[ 1358.904747] \x0aShowing all locks held in the system:
[ 1358.913290] 1 lock held by khungtaskd/185:
[ 1358.918683] #0: ffffffffa32fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.930160] 2 locks held by systemd-journal/804:
[ 1358.936140] #0: ff1100180e4b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1358.946031] #1: ffffffffa3502ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.958856] 4 locks held by python3/3562:
[ 1358.964150] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.974039] #1: ff1100019ff2d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.985001] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.996161] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.006258] 4 locks held by python3/5593:
[ 1359.011555] #0: ff1100011fa7c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.021445] #1: ff110001ae423280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.032408] #2: ff1100011ba3ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.043567] #3: ff11000112581e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.053666]
[ 1359.056124] =============================================\x0a