[ 789.792975] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 789.795599] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py
[ 789.926248] kselftest: Running tests in drivers/net
[ 989.685905] INFO: task python3:5912 blocked for more than 122 seconds.
[ 989.694138] Not tainted 7.1.0-rc1-kvpk-g8d59040b039c #1
[ 989.702073] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.711659] task:python3 state:D stack:25672 pid:5912 tgid:5912 ppid:5911 task_flags:0x400100 flags:0x00080000
[ 989.724864] Call Trace:
[ 989.728404]
[ 989.731552] __schedule+0xcba/0x1940
[ 989.736369] ? io_schedule_timeout+0x140/0x140
[ 989.742153] ? find_held_lock+0x2b/0x80
[ 989.747247] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.753132] schedule+0xd4/0x260
[ 989.757551] schedule_preempt_disabled+0x18/0x30
[ 989.763531] __mutex_lock+0x6a3/0x2000
[ 989.768536] ? threaded_store+0x132/0x220
[ 989.773833] ? ww_mutex_lock+0x160/0x160
[ 989.779033] ? selinux_inode_free_security+0x330/0x330
[ 989.785600] ? cap_capable+0x1d6/0x3e0
[ 989.790608] ? threaded_store+0x132/0x220
[ 989.795904] threaded_store+0x132/0x220
[ 989.801003] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.806305] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.811601] ? sysfs_kf_write+0x65/0x160
[ 989.816802] ? sysfs_file_ops+0x140/0x140
[ 989.822098] kernfs_fop_write_iter+0x335/0x510
[ 989.827881] ? kernfs_ops+0xf0/0xf0
[ 989.832591] new_sync_write+0x333/0x750
[ 989.837694] ? __lock_acquire+0x508/0xc10
[ 989.842990] ? new_sync_read+0x740/0x740
[ 989.848188] ? lock_acquire.part.0+0xbc/0x260
[ 989.853874] ? ksys_write+0x116/0x250
[ 989.858783] vfs_write+0x657/0x9b0
[ 989.863396] ksys_write+0x116/0x250
[ 989.868104] ? __ia32_sys_read+0xc0/0xc0
[ 989.873300] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.878886] ? rcu_is_watching+0x15/0xd0
[ 989.884088] do_syscall_64+0xf3/0x650
[ 989.888993] ? trace_hardirqs_off+0xd/0x30
[ 989.894388] ? exc_page_fault+0xda/0xf0
[ 989.899487] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.905951] RIP: 0033:0x7f2fb23be22e
[ 989.910758] RSP: 002b:00007ffc633ae2e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.920057] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2fb23be22e
[ 989.928864] RDX: 0000000000000001 RSI: 0000559bcde0fa10 RDI: 0000000000000007
[ 989.937669] RBP: 00007ffc633ae2f0 R08: 0000000000000000 R09: 0000000000000000
[ 989.946475] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2fb2236390
[ 989.955283] R13: 0000559bcde0fa10 R14: 0000000000000007 R15: 0000000000000003
[ 989.964087]
[ 989.967362] INFO: task python3:5912 is blocked on a mutex likely owned by task python3:3870.
[ 989.977625] task:python3 state:D stack:25672 pid:3870 tgid:3870 ppid:3869 task_flags:0x400100 flags:0x00080802
[ 989.990828] Call Trace:
[ 989.994367]
[ 989.997516] __schedule+0xcba/0x1940
[ 990.002327] ? io_schedule_timeout+0x140/0x140
[ 990.008113] ? find_held_lock+0x2b/0x80
[ 990.013205] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.019384] schedule+0xd4/0x260
[ 990.023804] schedule_timeout+0x13f/0x290
[ 990.029092] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.035463] ? trace_contention_end+0xb3/0x1a0
[ 990.041237] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.046926] ? ww_mutex_lock+0x160/0x160
[ 990.052134] msleep+0x1b/0x30
[ 990.056257] napi_set_threaded+0x3a5/0x440
[ 990.061652] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.067145] netif_set_threaded+0x257/0x400
[ 990.072635] threaded_store+0x1b8/0x220
[ 990.077743] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.083037] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.088334] ? sysfs_kf_write+0x65/0x160
[ 990.093530] ? sysfs_file_ops+0x140/0x140
[ 990.098825] kernfs_fop_write_iter+0x335/0x510
[ 990.104613] ? kernfs_ops+0xf0/0xf0
[ 990.109323] new_sync_write+0x333/0x750
[ 990.114429] ? __lock_acquire+0x508/0xc10
[ 990.119726] ? new_sync_read+0x740/0x740
[ 990.124927] ? lock_acquire.part.0+0xbc/0x260
[ 990.130611] ? ksys_write+0x116/0x250
[ 990.135520] vfs_write+0x657/0x9b0
[ 990.140136] ksys_write+0x116/0x250
[ 990.144848] ? __ia32_sys_read+0xc0/0xc0
[ 990.150048] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.155638] ? rcu_is_watching+0x15/0xd0
[ 990.160841] do_syscall_64+0xf3/0x650
[ 990.165750] ? trace_hardirqs_off+0xd/0x30
[ 990.171145] ? exc_page_fault+0xda/0xf0
[ 990.176245] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.182715] RIP: 0033:0x7f6693b4d22e
[ 990.187524] RSP: 002b:00007ffe9792f0b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.196820] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6693b4d22e
[ 990.205630] RDX: 0000000000000001 RSI: 000055f2fffdea10 RDI: 0000000000000007
[ 990.214439] RBP: 00007ffe9792f0c0 R08: 0000000000000000 R09: 0000000000000000
[ 990.223249] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f66939c5390
[ 990.232057] R13: 000055f2fffdea10 R14: 0000000000000007 R15: 0000000000000003
[ 990.240872]
[ 990.244130] \x0aShowing all locks held in the system:
[ 990.252649] 2 locks held by systemd/1:
[ 990.257650] #0: ff110001309913e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.267543] #1: ff110001c7f22080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.278529] 1 lock held by khungtaskd/184:
[ 990.283925] #0: ffffffff9fefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.295397] 3 locks held by systemd-journal/805:
[ 990.301377] #0: ff1100180e4b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.311265] #1: ff110001a2998378 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.322032] #2: ff110001a246ce90 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.333588] 4 locks held by python3/3870:
[ 990.338886] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.348776] #1: ff1100013e9fa680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.359745] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.370902] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.381005] 4 locks held by python3/5912:
[ 990.386298] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.396188] #1: ff110017d9753e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.407148] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.418306] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.428396]
[ 990.430865] =============================================\x0a
[ 1112.567262] INFO: task python3:5912 blocked for more than 245 seconds.
[ 1112.575408] Not tainted 7.1.0-rc1-kvpk-g8d59040b039c #1
[ 1112.582658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.592249] task:python3 state:D stack:25672 pid:5912 tgid:5912 ppid:5911 task_flags:0x400100 flags:0x00080000
[ 1112.605458] Call Trace:
[ 1112.609000]
[ 1112.612151] __schedule+0xcba/0x1940
[ 1112.616969] ? io_schedule_timeout+0x140/0x140
[ 1112.622755] ? find_held_lock+0x2b/0x80
[ 1112.627861] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.633749] schedule+0xd4/0x260
[ 1112.638171] schedule_preempt_disabled+0x18/0x30
[ 1112.644153] __mutex_lock+0x6a3/0x2000
[ 1112.649160] ? threaded_store+0x132/0x220
[ 1112.654459] ? ww_mutex_lock+0x160/0x160
[ 1112.659659] ? selinux_inode_free_security+0x330/0x330
[ 1112.666227] ? cap_capable+0x1d6/0x3e0
[ 1112.671240] ? threaded_store+0x132/0x220
[ 1112.676548] threaded_store+0x132/0x220
[ 1112.681656] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.686956] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.692255] ? sysfs_kf_write+0x65/0x160
[ 1112.697454] ? sysfs_file_ops+0x140/0x140
[ 1112.702751] kernfs_fop_write_iter+0x335/0x510
[ 1112.708528] ? kernfs_ops+0xf0/0xf0
[ 1112.713238] new_sync_write+0x333/0x750
[ 1112.718334] ? __lock_acquire+0x508/0xc10
[ 1112.723633] ? new_sync_read+0x740/0x740
[ 1112.728834] ? lock_acquire.part.0+0xbc/0x260
[ 1112.734519] ? ksys_write+0x116/0x250
[ 1112.739430] vfs_write+0x657/0x9b0
[ 1112.744048] ksys_write+0x116/0x250
[ 1112.748761] ? __ia32_sys_read+0xc0/0xc0
[ 1112.753953] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.759533] ? rcu_is_watching+0x15/0xd0
[ 1112.764729] do_syscall_64+0xf3/0x650
[ 1112.769636] ? trace_hardirqs_off+0xd/0x30
[ 1112.775033] ? exc_page_fault+0xda/0xf0
[ 1112.780135] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.786605] RIP: 0033:0x7f2fb23be22e
[ 1112.791415] RSP: 002b:00007ffc633ae2e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.800714] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2fb23be22e
[ 1112.809524] RDX: 0000000000000001 RSI: 0000559bcde0fa10 RDI: 0000000000000007
[ 1112.818333] RBP: 00007ffc633ae2f0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.827144] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2fb2236390
[ 1112.835954] R13: 0000559bcde0fa10 R14: 0000000000000007 R15: 0000000000000003
[ 1112.844769]
[ 1112.848046] INFO: task python3:5912 is blocked on a mutex likely owned by task python3:3870.
[ 1112.858320] task:python3 state:D stack:25672 pid:3870 tgid:3870 ppid:3869 task_flags:0x400100 flags:0x00080802
[ 1112.871523] Call Trace:
[ 1112.875063]
[ 1112.878212] __schedule+0xcba/0x1940
[ 1112.883024] ? io_schedule_timeout+0x140/0x140
[ 1112.888807] ? find_held_lock+0x2b/0x80
[ 1112.893911] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.900098] schedule+0xd4/0x260
[ 1112.904516] schedule_timeout+0x13f/0x290
[ 1112.909816] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.916184] ? trace_contention_end+0xb3/0x1a0
[ 1112.921971] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.927662] ? ww_mutex_lock+0x160/0x160
[ 1112.932861] msleep+0x1b/0x30
[ 1112.936978] napi_set_threaded+0x3a5/0x440
[ 1112.942376] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.947873] netif_set_threaded+0x257/0x400
[ 1112.953364] threaded_store+0x1b8/0x220
[ 1112.958469] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.963766] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.969063] ? sysfs_kf_write+0x65/0x160
[ 1112.974260] ? sysfs_file_ops+0x140/0x140
[ 1112.979557] kernfs_fop_write_iter+0x335/0x510
[ 1112.985342] ? kernfs_ops+0xf0/0xf0
[ 1112.990055] new_sync_write+0x333/0x750
[ 1112.995160] ? __lock_acquire+0x508/0xc10
[ 1113.000458] ? new_sync_read+0x740/0x740
[ 1113.005658] ? lock_acquire.part.0+0xbc/0x260
[ 1113.011343] ? ksys_write+0x116/0x250
[ 1113.016254] vfs_write+0x657/0x9b0
[ 1113.020871] ksys_write+0x116/0x250
[ 1113.025586] ? __ia32_sys_read+0xc0/0xc0
[ 1113.030785] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.036375] ? rcu_is_watching+0x15/0xd0
[ 1113.041575] do_syscall_64+0xf3/0x650
[ 1113.046481] ? trace_hardirqs_off+0xd/0x30
[ 1113.051875] ? exc_page_fault+0xda/0xf0
[ 1113.056978] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.063445] RIP: 0033:0x7f6693b4d22e
[ 1113.068255] RSP: 002b:00007ffe9792f0b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.077552] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6693b4d22e
[ 1113.086359] RDX: 0000000000000001 RSI: 000055f2fffdea10 RDI: 0000000000000007
[ 1113.095168] RBP: 00007ffe9792f0c0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.103977] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f66939c5390
[ 1113.112785] R13: 000055f2fffdea10 R14: 0000000000000007 R15: 0000000000000003
[ 1113.121598]
[ 1113.124862] \x0aShowing all locks held in the system:
[ 1113.133404] 1 lock held by khungtaskd/184:
[ 1113.138790] #0: ffffffff9fefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.150254] 3 locks held by systemd-journal/805:
[ 1113.156233] #0: ff1100180dcb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1113.166120] #1: ffffffffa0103120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.178939] #2: ff110001a246dc50 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.190500] 4 locks held by python3/3870:
[ 1113.195795] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.205684] #1: ff1100013e9fa680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.216641] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.227797] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.237890] 4 locks held by python3/5912:
[ 1113.243187] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.253074] #1: ff110017d9753e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.264037] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.275196] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.285284]
[ 1113.287752] =============================================\x0a
[ 1235.448644] INFO: task python3:5912 blocked for more than 368 seconds.
[ 1235.456790] Not tainted 7.1.0-rc1-kvpk-g8d59040b039c #1
[ 1235.464040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.473629] task:python3 state:D stack:25672 pid:5912 tgid:5912 ppid:5911 task_flags:0x400100 flags:0x00080002
[ 1235.486828] Call Trace:
[ 1235.490370]
[ 1235.493521] __schedule+0xcba/0x1940
[ 1235.498339] ? io_schedule_timeout+0x140/0x140
[ 1235.504126] ? find_held_lock+0x2b/0x80
[ 1235.509233] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.515118] schedule+0xd4/0x260
[ 1235.519539] schedule_preempt_disabled+0x18/0x30
[ 1235.525520] __mutex_lock+0x6a3/0x2000
[ 1235.530524] ? threaded_store+0x132/0x220
[ 1235.535826] ? ww_mutex_lock+0x160/0x160
[ 1235.541026] ? selinux_inode_free_security+0x330/0x330
[ 1235.547596] ? cap_capable+0x1d6/0x3e0
[ 1235.552605] ? threaded_store+0x132/0x220
[ 1235.557906] threaded_store+0x132/0x220
[ 1235.563011] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.568311] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.573614] ? sysfs_kf_write+0x65/0x160
[ 1235.578816] ? sysfs_file_ops+0x140/0x140
[ 1235.584112] kernfs_fop_write_iter+0x335/0x510
[ 1235.589899] ? kernfs_ops+0xf0/0xf0
[ 1235.594610] new_sync_write+0x333/0x750
[ 1235.599704] ? __lock_acquire+0x508/0xc10
[ 1235.605003] ? new_sync_read+0x740/0x740
[ 1235.610205] ? lock_acquire.part.0+0xbc/0x260
[ 1235.615891] ? ksys_write+0x116/0x250
[ 1235.620791] vfs_write+0x657/0x9b0
[ 1235.625407] ksys_write+0x116/0x250
[ 1235.630109] ? __ia32_sys_read+0xc0/0xc0
[ 1235.635307] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.640896] ? rcu_is_watching+0x15/0xd0
[ 1235.646096] do_syscall_64+0xf3/0x650
[ 1235.651001] ? trace_hardirqs_off+0xd/0x30
[ 1235.656394] ? exc_page_fault+0xda/0xf0
[ 1235.661493] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.667963] RIP: 0033:0x7f2fb23be22e
[ 1235.672773] RSP: 002b:00007ffc633ae2e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.682073] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2fb23be22e
[ 1235.690883] RDX: 0000000000000001 RSI: 0000559bcde0fa10 RDI: 0000000000000007
[ 1235.699692] RBP: 00007ffc633ae2f0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.708502] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2fb2236390
[ 1235.717313] R13: 0000559bcde0fa10 R14: 0000000000000007 R15: 0000000000000003
[ 1235.726126]
[ 1235.729403] INFO: task python3:5912 is blocked on a mutex likely owned by task python3:3870.
[ 1235.739678] task:python3 state:D stack:25672 pid:3870 tgid:3870 ppid:3869 task_flags:0x400100 flags:0x00080802
[ 1235.752881] Call Trace:
[ 1235.756419]
[ 1235.759570] __schedule+0xcba/0x1940
[ 1235.764381] ? io_schedule_timeout+0x140/0x140
[ 1235.770165] ? find_held_lock+0x2b/0x80
[ 1235.775269] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.781449] schedule+0xd4/0x260
[ 1235.785868] schedule_timeout+0x13f/0x290
[ 1235.791165] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.797535] ? trace_contention_end+0xb3/0x1a0
[ 1235.803323] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.809014] ? ww_mutex_lock+0x160/0x160
[ 1235.814214] msleep+0x1b/0x30
[ 1235.818330] napi_set_threaded+0x3a5/0x440
[ 1235.823728] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.829222] netif_set_threaded+0x257/0x400
[ 1235.834716] threaded_store+0x1b8/0x220
[ 1235.839821] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.845118] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.850416] ? sysfs_kf_write+0x65/0x160
[ 1235.855616] ? sysfs_file_ops+0x140/0x140
[ 1235.860911] kernfs_fop_write_iter+0x335/0x510
[ 1235.866693] ? kernfs_ops+0xf0/0xf0
[ 1235.871404] new_sync_write+0x333/0x750
[ 1235.876506] ? __lock_acquire+0x508/0xc10
[ 1235.881801] ? new_sync_read+0x740/0x740
[ 1235.887002] ? lock_acquire.part.0+0xbc/0x260
[ 1235.892678] ? ksys_write+0x116/0x250
[ 1235.897592] vfs_write+0x657/0x9b0
[ 1235.902210] ksys_write+0x116/0x250
[ 1235.906923] ? __ia32_sys_read+0xc0/0xc0
[ 1235.912121] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.917711] ? rcu_is_watching+0x15/0xd0
[ 1235.922914] do_syscall_64+0xf3/0x650
[ 1235.927819] ? trace_hardirqs_off+0xd/0x30
[ 1235.933212] ? exc_page_fault+0xda/0xf0
[ 1235.938314] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.944773] RIP: 0033:0x7f6693b4d22e
[ 1235.949572] RSP: 002b:00007ffe9792f0b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.958870] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6693b4d22e
[ 1235.967680] RDX: 0000000000000001 RSI: 000055f2fffdea10 RDI: 0000000000000007
[ 1235.976495] RBP: 00007ffe9792f0c0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.985305] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f66939c5390
[ 1235.994114] R13: 000055f2fffdea10 R14: 0000000000000007 R15: 0000000000000003
[ 1236.002927]
[ 1236.006194] \x0aShowing all locks held in the system:
[ 1236.014732] 1 lock held by khungtaskd/184:
[ 1236.020128] #0: ffffffff9fefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.031611] 4 locks held by python3/3870:
[ 1236.036907] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.046799] #1: ff1100013e9fa680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.057764] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.068922] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.079016] 4 locks held by python3/5912:
[ 1236.084312] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.094200] #1: ff110017d9753e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.105162] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.116323] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.126417]
[ 1236.128886] =============================================\x0a
[ 1358.330052] INFO: task python3:5912 blocked for more than 491 seconds.
[ 1358.338202] Not tainted 7.1.0-rc1-kvpk-g8d59040b039c #1
[ 1358.345457] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.355067] task:python3 state:D stack:25672 pid:5912 tgid:5912 ppid:5911 task_flags:0x400100 flags:0x00080002
[ 1358.368290] Call Trace:
[ 1358.371840]
[ 1358.374999] __schedule+0xcba/0x1940
[ 1358.379823] ? io_schedule_timeout+0x140/0x140
[ 1358.385613] ? find_held_lock+0x2b/0x80
[ 1358.390724] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.396614] schedule+0xd4/0x260
[ 1358.401037] schedule_preempt_disabled+0x18/0x30
[ 1358.407019] __mutex_lock+0x6a3/0x2000
[ 1358.412020] ? threaded_store+0x132/0x220
[ 1358.417325] ? ww_mutex_lock+0x160/0x160
[ 1358.422530] ? selinux_inode_free_security+0x330/0x330
[ 1358.429106] ? cap_capable+0x1d6/0x3e0
[ 1358.434121] ? threaded_store+0x132/0x220
[ 1358.439422] threaded_store+0x132/0x220
[ 1358.444527] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.449828] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.455128] ? sysfs_kf_write+0x65/0x160
[ 1358.460333] ? sysfs_file_ops+0x140/0x140
[ 1358.465631] kernfs_fop_write_iter+0x335/0x510
[ 1358.471421] ? kernfs_ops+0xf0/0xf0
[ 1358.476138] new_sync_write+0x333/0x750
[ 1358.481248] ? __lock_acquire+0x508/0xc10
[ 1358.486549] ? new_sync_read+0x740/0x740
[ 1358.491755] ? lock_acquire.part.0+0xbc/0x260
[ 1358.497445] ? ksys_write+0x116/0x250
[ 1358.502360] vfs_write+0x657/0x9b0
[ 1358.506979] ksys_write+0x116/0x250
[ 1358.511695] ? __ia32_sys_read+0xc0/0xc0
[ 1358.516902] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.522496] ? rcu_is_watching+0x15/0xd0
[ 1358.527707] do_syscall_64+0xf3/0x650
[ 1358.532618] ? trace_hardirqs_off+0xd/0x30
[ 1358.538017] ? exc_page_fault+0xda/0xf0
[ 1358.543123] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.549598] RIP: 0033:0x7f2fb23be22e
[ 1358.554410] RSP: 002b:00007ffc633ae2e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.563715] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2fb23be22e
[ 1358.572519] RDX: 0000000000000001 RSI: 0000559bcde0fa10 RDI: 0000000000000007
[ 1358.581331] RBP: 00007ffc633ae2f0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.590141] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f2fb2236390
[ 1358.598957] R13: 0000559bcde0fa10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.607766]
[ 1358.611101] INFO: task python3:5912 is blocked on a mutex likely owned by task python3:3870.
[ 1358.621377] task:python3 state:D stack:25672 pid:3870 tgid:3870 ppid:3869 task_flags:0x400100 flags:0x00080802
[ 1358.634583] Call Trace:
[ 1358.638123]
[ 1358.641278] __schedule+0xcba/0x1940
[ 1358.646095] ? io_schedule_timeout+0x140/0x140
[ 1358.651883] ? find_held_lock+0x2b/0x80
[ 1358.656990] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.663177] schedule+0xd4/0x260
[ 1358.667601] schedule_timeout+0x13f/0x290
[ 1358.672905] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.679279] ? trace_contention_end+0xb3/0x1a0
[ 1358.685069] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.690765] ? ww_mutex_lock+0x160/0x160
[ 1358.695971] msleep+0x1b/0x30
[ 1358.700101] napi_set_threaded+0x3a5/0x440
[ 1358.705503] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.711002] netif_set_threaded+0x257/0x400
[ 1358.716503] threaded_store+0x1b8/0x220
[ 1358.721612] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.726915] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.732217] ? sysfs_kf_write+0x65/0x160
[ 1358.737421] ? sysfs_file_ops+0x140/0x140
[ 1358.742722] kernfs_fop_write_iter+0x335/0x510
[ 1358.748514] ? kernfs_ops+0xf0/0xf0
[ 1358.753227] new_sync_write+0x333/0x750
[ 1358.758338] ? __lock_acquire+0x508/0xc10
[ 1358.763639] ? new_sync_read+0x740/0x740
[ 1358.768844] ? lock_acquire.part.0+0xbc/0x260
[ 1358.774533] ? ksys_write+0x116/0x250
[ 1358.779451] vfs_write+0x657/0x9b0
[ 1358.784069] ksys_write+0x116/0x250
[ 1358.788784] ? __ia32_sys_read+0xc0/0xc0
[ 1358.793987] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.799582] ? rcu_is_watching+0x15/0xd0
[ 1358.804786] do_syscall_64+0xf3/0x650
[ 1358.809697] ? trace_hardirqs_off+0xd/0x30
[ 1358.815097] ? exc_page_fault+0xda/0xf0
[ 1358.820205] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.826675] RIP: 0033:0x7f6693b4d22e
[ 1358.831488] RSP: 002b:00007ffe9792f0b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.840793] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6693b4d22e
[ 1358.849606] RDX: 0000000000000001 RSI: 000055f2fffdea10 RDI: 0000000000000007
[ 1358.858420] RBP: 00007ffe9792f0c0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.867233] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f66939c5390
[ 1358.876043] R13: 000055f2fffdea10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.884868]
[ 1358.888133] \x0aShowing all locks held in the system:
[ 1358.896688] 1 lock held by khungtaskd/184:
[ 1358.902084] #0: ffffffff9fefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.913577] 3 locks held by systemd-journal/805:
[ 1358.919558] #0: ff1100180e0372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1358.929453] #1: ffffffffa0103120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: dput.part.0+0x2fd/0x410
[ 1358.941691] #2: ff110017d9e484f0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.953261] 4 locks held by python3/3870:
[ 1358.958559] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.968456] #1: ff1100013e9fa680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.979429] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.990593] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.000701] 4 locks held by python3/5912:
[ 1359.006004] #0: ff110001308a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.015906] #1: ff110017d9753e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.026876] #2: ff11000124562b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.038040] #3: ff11000124599d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.048145]
[ 1359.050614] =============================================\x0a