[ 788.962882] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 788.965336] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py
[ 789.121921] kselftest: Running tests in drivers/net
[ 989.689652] INFO: task python3:5917 blocked for more than 122 seconds.
[ 989.697866] Not tainted 7.1.0-rc1-xxpp-g2382e43ef545 #1
[ 989.705653] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.715243] task:python3 state:D stack:25512 pid:5917 tgid:5917 ppid:5916 task_flags:0x400100 flags:0x00080000
[ 989.728451] Call Trace:
[ 989.731993]
[ 989.735144] __schedule+0xcba/0x1940
[ 989.739963] ? io_schedule_timeout+0x140/0x140
[ 989.745751] ? find_held_lock+0x2b/0x80
[ 989.750857] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.756743] schedule+0xd4/0x260
[ 989.761160] schedule_preempt_disabled+0x18/0x30
[ 989.767143] __mutex_lock+0x6a3/0x2000
[ 989.772149] ? threaded_store+0x132/0x220
[ 989.777452] ? ww_mutex_lock+0x160/0x160
[ 989.782652] ? selinux_inode_free_security+0x330/0x330
[ 989.789222] ? cap_capable+0x1d6/0x3e0
[ 989.794230] ? threaded_store+0x132/0x220
[ 989.799527] threaded_store+0x132/0x220
[ 989.804629] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.809928] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.815229] ? sysfs_kf_write+0x65/0x160
[ 989.820432] ? sysfs_file_ops+0x140/0x140
[ 989.825730] kernfs_fop_write_iter+0x335/0x510
[ 989.831504] ? kernfs_ops+0xf0/0xf0
[ 989.836217] new_sync_write+0x333/0x750
[ 989.841321] ? __lock_acquire+0x508/0xc10
[ 989.846607] ? new_sync_read+0x740/0x740
[ 989.851810] ? lock_acquire.part.0+0xbc/0x260
[ 989.857498] ? ksys_write+0x116/0x250
[ 989.862408] vfs_write+0x657/0x9b0
[ 989.867025] ksys_write+0x116/0x250
[ 989.871739] ? __ia32_sys_read+0xc0/0xc0
[ 989.876937] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.882529] ? rcu_is_watching+0x15/0xd0
[ 989.887730] do_syscall_64+0xf3/0x650
[ 989.892637] ? trace_hardirqs_off+0xd/0x30
[ 989.898032] ? exc_page_fault+0xda/0xf0
[ 989.903135] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.909596] RIP: 0033:0x7f7dddc5822e
[ 989.914407] RSP: 002b:00007ffdb0988380 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.923705] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f7dddc5822e
[ 989.932516] RDX: 0000000000000001 RSI: 000056126e9efa10 RDI: 0000000000000007
[ 989.941322] RBP: 00007ffdb0988390 R08: 0000000000000000 R09: 0000000000000000
[ 989.950133] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f7dddad0390
[ 989.958942] R13: 000056126e9efa10 R14: 0000000000000007 R15: 0000000000000003
[ 989.967756]
[ 989.971041] INFO: task python3:5917 is blocked on a mutex likely owned by task python3:3831.
[ 989.981313] task:python3 state:D stack:25672 pid:3831 tgid:3831 ppid:3830 task_flags:0x400100 flags:0x00080802
[ 989.994516] Call Trace:
[ 989.998055]
[ 990.001206] __schedule+0xcba/0x1940
[ 990.006009] ? io_schedule_timeout+0x140/0x140
[ 990.011794] ? find_held_lock+0x2b/0x80
[ 990.016891] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.023069] schedule+0xd4/0x260
[ 990.027491] schedule_timeout+0x13f/0x290
[ 990.032789] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.039160] ? trace_contention_end+0xb3/0x1a0
[ 990.044937] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.050636] ? ww_mutex_lock+0x160/0x160
[ 990.055838] msleep+0x1b/0x30
[ 990.059965] napi_set_threaded+0x3a5/0x440
[ 990.065362] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.070859] netif_set_threaded+0x257/0x400
[ 990.076351] threaded_store+0x1b8/0x220
[ 990.081447] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.086747] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.092042] ? sysfs_kf_write+0x65/0x160
[ 990.097243] ? sysfs_file_ops+0x140/0x140
[ 990.102530] kernfs_fop_write_iter+0x335/0x510
[ 990.108313] ? kernfs_ops+0xf0/0xf0
[ 990.113024] new_sync_write+0x333/0x750
[ 990.118127] ? __lock_acquire+0x508/0xc10
[ 990.123423] ? new_sync_read+0x740/0x740
[ 990.128624] ? lock_acquire.part.0+0xbc/0x260
[ 990.134312] ? ksys_write+0x116/0x250
[ 990.139221] vfs_write+0x657/0x9b0
[ 990.143839] ksys_write+0x116/0x250
[ 990.148552] ? __ia32_sys_read+0xc0/0xc0
[ 990.153751] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.159338] ? rcu_is_watching+0x15/0xd0
[ 990.164541] do_syscall_64+0xf3/0x650
[ 990.169449] ? trace_hardirqs_off+0xd/0x30
[ 990.174845] ? exc_page_fault+0xda/0xf0
[ 990.179948] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.186417] RIP: 0033:0x7f090cfdc22e
[ 990.191225] RSP: 002b:00007ffcb2ebee60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.200522] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f090cfdc22e
[ 990.209324] RDX: 0000000000000001 RSI: 0000560031b45a10 RDI: 0000000000000007
[ 990.218131] RBP: 00007ffcb2ebee70 R08: 0000000000000000 R09: 0000000000000000
[ 990.226931] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f090ce54390
[ 990.235739] R13: 0000560031b45a10 R14: 0000000000000007 R15: 0000000000000003
[ 990.244554]
[ 990.247813] \x0aShowing all locks held in the system:
[ 990.256353] 1 lock held by khungtaskd/185:
[ 990.261746] #0: ffffffff888fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.273222] 3 locks held by systemd-journal/808:
[ 990.279202] #0: ff1100180e3372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.289090] #1: ffffffff88b03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.301906] #2: ff110001a4879150 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.313466] 4 locks held by python3/3831:
[ 990.318762] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.328651] #1: ff11000184150880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.339620] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.350791] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.360895] 4 locks held by python3/5917:
[ 990.366189] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.376077] #1: ff110001e7004a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.387041] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.398199] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.408289]
[ 990.410755] =============================================\x0a
[ 1112.571773] INFO: task python3:5917 blocked for more than 245 seconds.
[ 1112.579921] Not tainted 7.1.0-rc1-xxpp-g2382e43ef545 #1
[ 1112.587174] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.596760] task:python3 state:D stack:25512 pid:5917 tgid:5917 ppid:5916 task_flags:0x400100 flags:0x00080000
[ 1112.609963] Call Trace:
[ 1112.613502]
[ 1112.616650] __schedule+0xcba/0x1940
[ 1112.621468] ? io_schedule_timeout+0x140/0x140
[ 1112.627253] ? find_held_lock+0x2b/0x80
[ 1112.632355] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.638242] schedule+0xd4/0x260
[ 1112.642657] schedule_preempt_disabled+0x18/0x30
[ 1112.648633] __mutex_lock+0x6a3/0x2000
[ 1112.653635] ? threaded_store+0x132/0x220
[ 1112.658933] ? ww_mutex_lock+0x160/0x160
[ 1112.664129] ? selinux_inode_free_security+0x330/0x330
[ 1112.670697] ? cap_capable+0x1d6/0x3e0
[ 1112.675705] ? threaded_store+0x132/0x220
[ 1112.681004] threaded_store+0x132/0x220
[ 1112.686104] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.691400] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.696696] ? sysfs_kf_write+0x65/0x160
[ 1112.701894] ? sysfs_file_ops+0x140/0x140
[ 1112.707188] kernfs_fop_write_iter+0x335/0x510
[ 1112.713039] ? kernfs_ops+0xf0/0xf0
[ 1112.717812] new_sync_write+0x333/0x750
[ 1112.722915] ? __lock_acquire+0x508/0xc10
[ 1112.728208] ? new_sync_read+0x740/0x740
[ 1112.733407] ? lock_acquire.part.0+0xbc/0x260
[ 1112.739089] ? ksys_write+0x116/0x250
[ 1112.744001] vfs_write+0x657/0x9b0
[ 1112.748618] ksys_write+0x116/0x250
[ 1112.753329] ? __ia32_sys_read+0xc0/0xc0
[ 1112.758528] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.764115] ? rcu_is_watching+0x15/0xd0
[ 1112.769318] do_syscall_64+0xf3/0x650
[ 1112.774225] ? trace_hardirqs_off+0xd/0x30
[ 1112.779619] ? exc_page_fault+0xda/0xf0
[ 1112.784721] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.791188] RIP: 0033:0x7f7dddc5822e
[ 1112.795997] RSP: 002b:00007ffdb0988380 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.805296] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f7dddc5822e
[ 1112.814102] RDX: 0000000000000001 RSI: 000056126e9efa10 RDI: 0000000000000007
[ 1112.822910] RBP: 00007ffdb0988390 R08: 0000000000000000 R09: 0000000000000000
[ 1112.831717] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f7dddad0390
[ 1112.840524] R13: 000056126e9efa10 R14: 0000000000000007 R15: 0000000000000003
[ 1112.849336]
[ 1112.852659] INFO: task python3:5917 is blocked on a mutex likely owned by task python3:3831.
[ 1112.862930] task:python3 state:D stack:25672 pid:3831 tgid:3831 ppid:3830 task_flags:0x400100 flags:0x00080802
[ 1112.876132] Call Trace:
[ 1112.879667]
[ 1112.882816] __schedule+0xcba/0x1940
[ 1112.887626] ? io_schedule_timeout+0x140/0x140
[ 1112.893407] ? find_held_lock+0x2b/0x80
[ 1112.898509] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.904688] schedule+0xd4/0x260
[ 1112.909103] schedule_timeout+0x13f/0x290
[ 1112.914398] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.920764] ? trace_contention_end+0xb3/0x1a0
[ 1112.926547] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.932236] ? ww_mutex_lock+0x160/0x160
[ 1112.937434] msleep+0x1b/0x30
[ 1112.941557] napi_set_threaded+0x3a5/0x440
[ 1112.946951] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.952445] netif_set_threaded+0x257/0x400
[ 1112.957936] threaded_store+0x1b8/0x220
[ 1112.963039] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.968335] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.973630] ? sysfs_kf_write+0x65/0x160
[ 1112.978828] ? sysfs_file_ops+0x140/0x140
[ 1112.984123] kernfs_fop_write_iter+0x335/0x510
[ 1112.989906] ? kernfs_ops+0xf0/0xf0
[ 1112.994616] new_sync_write+0x333/0x750
[ 1112.999718] ? __lock_acquire+0x508/0xc10
[ 1113.005011] ? new_sync_read+0x740/0x740
[ 1113.010211] ? lock_acquire.part.0+0xbc/0x260
[ 1113.015895] ? ksys_write+0x116/0x250
[ 1113.020806] vfs_write+0x657/0x9b0
[ 1113.025419] ksys_write+0x116/0x250
[ 1113.030133] ? __ia32_sys_read+0xc0/0xc0
[ 1113.035332] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.040919] ? rcu_is_watching+0x15/0xd0
[ 1113.046120] do_syscall_64+0xf3/0x650
[ 1113.051026] ? trace_hardirqs_off+0xd/0x30
[ 1113.056419] ? exc_page_fault+0xda/0xf0
[ 1113.061523] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.067991] RIP: 0033:0x7f090cfdc22e
[ 1113.072870] RSP: 002b:00007ffcb2ebee60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.082156] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f090cfdc22e
[ 1113.090960] RDX: 0000000000000001 RSI: 0000560031b45a10 RDI: 0000000000000007
[ 1113.099766] RBP: 00007ffcb2ebee70 R08: 0000000000000000 R09: 0000000000000000
[ 1113.108571] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f090ce54390
[ 1113.117377] R13: 0000560031b45a10 R14: 0000000000000007 R15: 0000000000000003
[ 1113.126189]
[ 1113.129454] \x0aShowing all locks held in the system:
[ 1113.138055] 1 lock held by khungtaskd/185:
[ 1113.143446] #0: ffffffff888fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.154928] 4 locks held by python3/3831:
[ 1113.160223] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.170114] #1: ff11000184150880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.181075] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.192229] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.202323] 4 locks held by python3/5917:
[ 1113.207619] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.217507] #1: ff110001e7004a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.228471] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.239631] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.249722]
[ 1113.252506] =============================================\x0a
[ 1235.452947] INFO: task python3:5917 blocked for more than 368 seconds.
[ 1235.461094] Not tainted 7.1.0-rc1-xxpp-g2382e43ef545 #1
[ 1235.468345] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.477934] task:python3 state:D stack:25512 pid:5917 tgid:5917 ppid:5916 task_flags:0x400100 flags:0x00080002
[ 1235.491134] Call Trace:
[ 1235.494676]
[ 1235.497817] __schedule+0xcba/0x1940
[ 1235.502637] ? io_schedule_timeout+0x140/0x140
[ 1235.508425] ? find_held_lock+0x2b/0x80
[ 1235.513529] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.519414] schedule+0xd4/0x260
[ 1235.523833] schedule_preempt_disabled+0x18/0x30
[ 1235.529811] __mutex_lock+0x6a3/0x2000
[ 1235.534815] ? threaded_store+0x132/0x220
[ 1235.540115] ? ww_mutex_lock+0x160/0x160
[ 1235.545307] ? selinux_inode_free_security+0x330/0x330
[ 1235.551878] ? cap_capable+0x1d6/0x3e0
[ 1235.556884] ? threaded_store+0x132/0x220
[ 1235.562179] threaded_store+0x132/0x220
[ 1235.567281] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.572572] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.577873] ? sysfs_kf_write+0x65/0x160
[ 1235.583074] ? sysfs_file_ops+0x140/0x140
[ 1235.588372] kernfs_fop_write_iter+0x335/0x510
[ 1235.594156] ? kernfs_ops+0xf0/0xf0
[ 1235.598868] new_sync_write+0x333/0x750
[ 1235.603971] ? __lock_acquire+0x508/0xc10
[ 1235.609267] ? new_sync_read+0x740/0x740
[ 1235.614471] ? lock_acquire.part.0+0xbc/0x260
[ 1235.620158] ? ksys_write+0x116/0x250
[ 1235.625067] vfs_write+0x657/0x9b0
[ 1235.629686] ksys_write+0x116/0x250
[ 1235.634399] ? __ia32_sys_read+0xc0/0xc0
[ 1235.639599] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.645189] ? rcu_is_watching+0x15/0xd0
[ 1235.650392] do_syscall_64+0xf3/0x650
[ 1235.655298] ? trace_hardirqs_off+0xd/0x30
[ 1235.660695] ? exc_page_fault+0xda/0xf0
[ 1235.665800] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.672267] RIP: 0033:0x7f7dddc5822e
[ 1235.677069] RSP: 002b:00007ffdb0988380 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.686367] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f7dddc5822e
[ 1235.695177] RDX: 0000000000000001 RSI: 000056126e9efa10 RDI: 0000000000000007
[ 1235.703987] RBP: 00007ffdb0988390 R08: 0000000000000000 R09: 0000000000000000
[ 1235.712795] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f7dddad0390
[ 1235.721597] R13: 000056126e9efa10 R14: 0000000000000007 R15: 0000000000000003
[ 1235.730412]
[ 1235.733680] INFO: task python3:5917 is blocked on a mutex likely owned by task python3:3831.
[ 1235.743953] task:python3 state:D stack:25672 pid:3831 tgid:3831 ppid:3830 task_flags:0x400100 flags:0x00080802
[ 1235.757155] Call Trace:
[ 1235.760694]
[ 1235.763848] __schedule+0xcba/0x1940
[ 1235.768657] ? io_schedule_timeout+0x140/0x140
[ 1235.774444] ? find_held_lock+0x2b/0x80
[ 1235.779546] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.785724] schedule+0xd4/0x260
[ 1235.790143] schedule_timeout+0x13f/0x290
[ 1235.795442] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.801813] ? trace_contention_end+0xb3/0x1a0
[ 1235.807597] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.813290] ? ww_mutex_lock+0x160/0x160
[ 1235.818493] msleep+0x1b/0x30
[ 1235.822619] napi_set_threaded+0x3a5/0x440
[ 1235.828015] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.833500] netif_set_threaded+0x257/0x400
[ 1235.838995] threaded_store+0x1b8/0x220
[ 1235.844099] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.849398] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.854694] ? sysfs_kf_write+0x65/0x160
[ 1235.859896] ? sysfs_file_ops+0x140/0x140
[ 1235.865193] kernfs_fop_write_iter+0x335/0x510
[ 1235.870976] ? kernfs_ops+0xf0/0xf0
[ 1235.875689] new_sync_write+0x333/0x750
[ 1235.880793] ? __lock_acquire+0x508/0xc10
[ 1235.886089] ? new_sync_read+0x740/0x740
[ 1235.891288] ? lock_acquire.part.0+0xbc/0x260
[ 1235.896975] ? ksys_write+0x116/0x250
[ 1235.901877] vfs_write+0x657/0x9b0
[ 1235.906490] ksys_write+0x116/0x250
[ 1235.911206] ? __ia32_sys_read+0xc0/0xc0
[ 1235.916404] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.921994] ? rcu_is_watching+0x15/0xd0
[ 1235.927194] do_syscall_64+0xf3/0x650
[ 1235.932100] ? trace_hardirqs_off+0xd/0x30
[ 1235.937495] ? exc_page_fault+0xda/0xf0
[ 1235.942590] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.949059] RIP: 0033:0x7f090cfdc22e
[ 1235.953869] RSP: 002b:00007ffcb2ebee60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.963167] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f090cfdc22e
[ 1235.971973] RDX: 0000000000000001 RSI: 0000560031b45a10 RDI: 0000000000000007
[ 1235.980786] RBP: 00007ffcb2ebee70 R08: 0000000000000000 R09: 0000000000000000
[ 1235.989594] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f090ce54390
[ 1235.998403] R13: 0000560031b45a10 R14: 0000000000000007 R15: 0000000000000003
[ 1236.007218]
[ 1236.010481] \x0aShowing all locks held in the system:
[ 1236.019019] 1 lock held by khungtaskd/185:
[ 1236.024412] #0: ffffffff888fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.035893] 4 locks held by python3/3831:
[ 1236.041190] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.051075] #1: ff11000184150880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.062039] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.073196] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.083288] 4 locks held by python3/5917:
[ 1236.088585] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.098476] #1: ff110001e7004a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.109437] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.120595] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.130690]
[ 1236.133157] =============================================\x0a
[ 1358.334163] INFO: task python3:5917 blocked for more than 491 seconds.
[ 1358.342320] Not tainted 7.1.0-rc1-xxpp-g2382e43ef545 #1
[ 1358.349570] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.359160] task:python3 state:D stack:25512 pid:5917 tgid:5917 ppid:5916 task_flags:0x400100 flags:0x00080002
[ 1358.372358] Call Trace:
[ 1358.375897]
[ 1358.379048] __schedule+0xcba/0x1940
[ 1358.383864] ? io_schedule_timeout+0x140/0x140
[ 1358.389651] ? find_held_lock+0x2b/0x80
[ 1358.394753] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.400714] schedule+0xd4/0x260
[ 1358.405132] schedule_preempt_disabled+0x18/0x30
[ 1358.411111] __mutex_lock+0x6a3/0x2000
[ 1358.416113] ? threaded_store+0x132/0x220
[ 1358.421412] ? ww_mutex_lock+0x160/0x160
[ 1358.426608] ? selinux_inode_free_security+0x330/0x330
[ 1358.433179] ? cap_capable+0x1d6/0x3e0
[ 1358.438184] ? threaded_store+0x132/0x220
[ 1358.443470] threaded_store+0x132/0x220
[ 1358.448576] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.453873] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.459162] ? sysfs_kf_write+0x65/0x160
[ 1358.464361] ? sysfs_file_ops+0x140/0x140
[ 1358.469656] kernfs_fop_write_iter+0x335/0x510
[ 1358.475442] ? kernfs_ops+0xf0/0xf0
[ 1358.480151] new_sync_write+0x333/0x750
[ 1358.485253] ? __lock_acquire+0x508/0xc10
[ 1358.490550] ? new_sync_read+0x740/0x740
[ 1358.495753] ? lock_acquire.part.0+0xbc/0x260
[ 1358.501437] ? ksys_write+0x116/0x250
[ 1358.506348] vfs_write+0x657/0x9b0
[ 1358.510964] ksys_write+0x116/0x250
[ 1358.515677] ? __ia32_sys_read+0xc0/0xc0
[ 1358.520873] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.526459] ? rcu_is_watching+0x15/0xd0
[ 1358.531660] do_syscall_64+0xf3/0x650
[ 1358.536568] ? trace_hardirqs_off+0xd/0x30
[ 1358.541964] ? exc_page_fault+0xda/0xf0
[ 1358.547066] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.553534] RIP: 0033:0x7f7dddc5822e
[ 1358.558342] RSP: 002b:00007ffdb0988380 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.567638] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f7dddc5822e
[ 1358.576443] RDX: 0000000000000001 RSI: 000056126e9efa10 RDI: 0000000000000007
[ 1358.585249] RBP: 00007ffdb0988390 R08: 0000000000000000 R09: 0000000000000000
[ 1358.594056] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f7dddad0390
[ 1358.602862] R13: 000056126e9efa10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.611672]
[ 1358.614974] INFO: task python3:5917 is blocked on a mutex likely owned by task python3:3831.
[ 1358.625247] task:python3 state:D stack:25672 pid:3831 tgid:3831 ppid:3830 task_flags:0x400100 flags:0x00080802
[ 1358.638451] Call Trace:
[ 1358.642055]
[ 1358.645204] __schedule+0xcba/0x1940
[ 1358.650016] ? io_schedule_timeout+0x140/0x140
[ 1358.655797] ? find_held_lock+0x2b/0x80
[ 1358.660899] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.667077] schedule+0xd4/0x260
[ 1358.671492] schedule_timeout+0x13f/0x290
[ 1358.676789] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.683150] ? trace_contention_end+0xb3/0x1a0
[ 1358.688934] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.694625] ? ww_mutex_lock+0x160/0x160
[ 1358.699825] msleep+0x1b/0x30
[ 1358.703949] napi_set_threaded+0x3a5/0x440
[ 1358.709345] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.714829] netif_set_threaded+0x257/0x400
[ 1358.720314] threaded_store+0x1b8/0x220
[ 1358.725418] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.730715] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.736012] ? sysfs_kf_write+0x65/0x160
[ 1358.741276] ? sysfs_file_ops+0x140/0x140
[ 1358.746569] kernfs_fop_write_iter+0x335/0x510
[ 1358.752352] ? kernfs_ops+0xf0/0xf0
[ 1358.757063] new_sync_write+0x333/0x750
[ 1358.762166] ? __lock_acquire+0x508/0xc10
[ 1358.767452] ? new_sync_read+0x740/0x740
[ 1358.772653] ? lock_acquire.part.0+0xbc/0x260
[ 1358.778337] ? ksys_write+0x116/0x250
[ 1358.783245] vfs_write+0x657/0x9b0
[ 1358.787861] ksys_write+0x116/0x250
[ 1358.792575] ? __ia32_sys_read+0xc0/0xc0
[ 1358.797775] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.803364] ? rcu_is_watching+0x15/0xd0
[ 1358.808555] do_syscall_64+0xf3/0x650
[ 1358.813460] ? trace_hardirqs_off+0xd/0x30
[ 1358.818854] ? exc_page_fault+0xda/0xf0
[ 1358.823956] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.830422] RIP: 0033:0x7f090cfdc22e
[ 1358.835228] RSP: 002b:00007ffcb2ebee60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.844524] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f090cfdc22e
[ 1358.853331] RDX: 0000000000000001 RSI: 0000560031b45a10 RDI: 0000000000000007
[ 1358.862140] RBP: 00007ffcb2ebee70 R08: 0000000000000000 R09: 0000000000000000
[ 1358.870948] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f090ce54390
[ 1358.879745] R13: 0000560031b45a10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.888559]
[ 1358.891823] \x0aShowing all locks held in the system:
[ 1358.900365] 1 lock held by khungtaskd/185:
[ 1358.905758] #0: ffffffff888fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.917232] 5 locks held by systemd-journal/808:
[ 1358.923211] #0: ff1100180deb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1358.933099] #1: ffffffff88b03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.945914] #2: ff1100011b867a40 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.957460] #3: ff1100019f1c4f58 (&xa->xa_lock#5){-.-.}-{3:3}, at: __folio_mark_dirty+0x9d/0x2c0
[ 1358.968225] #4: ffffffff888fdee0 (rcu_read_lock){....}-{1:3}, at: mem_cgroup_track_foreign_dirty+0x59/0x3a0
[ 1358.980075] 4 locks held by python3/3831:
[ 1358.985370] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.995256] #1: ff11000184150880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.006216] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.017371] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.027469] 4 locks held by python3/5917:
[ 1359.032765] #0: ff1100012f9693e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.042653] #1: ff110001e7004a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.053613] #2: ff110017d94f6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.064769] #3: ff110017d94d1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.074857] 2 locks held by systemd/7756:
[ 1359.080155]
[ 1359.082622] =============================================\x0a