[ 779.955597] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 779.958030] nipa-hw-worker: [4/40] Retrying drivers/net:napi_threaded.py
[ 780.109611] kselftest: Running tests in drivers/net
[ 989.683953] INFO: task python3:5623 blocked for more than 122 seconds.
[ 989.692195] Not tainted 7.0.0-rc6-yesk-ga6e6a7dcb853 #1
[ 989.699446] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.709034] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080000
[ 989.722237] Call Trace:
[ 989.725778]
[ 989.728930] __schedule+0xae3/0x14d0
[ 989.733745] ? io_schedule_timeout+0x140/0x140
[ 989.739521] ? find_held_lock+0x2b/0x80
[ 989.744625] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.750507] schedule+0xd4/0x260
[ 989.754925] schedule_preempt_disabled+0x18/0x30
[ 989.760904] __mutex_lock+0x6b2/0x1ee0
[ 989.765908] ? rcu_is_watching+0x15/0xd0
[ 989.771104] ? threaded_store+0x132/0x220
[ 989.776403] ? ww_mutex_lock+0x160/0x160
[ 989.781603] ? selinux_inode_free_security+0x330/0x330
[ 989.788175] ? cap_capable+0x1ed/0x3f0
[ 989.793181] ? threaded_store+0x132/0x220
[ 989.798478] threaded_store+0x132/0x220
[ 989.803582] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.808878] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.814173] ? sysfs_kf_write+0x65/0x160
[ 989.819373] ? sysfs_file_ops+0x140/0x140
[ 989.825140] kernfs_fop_write_iter+0x335/0x510
[ 989.830960] ? kernfs_ops+0xf0/0xf0
[ 989.835675] new_sync_write+0x333/0x750
[ 989.840786] ? __lock_acquire+0x508/0xc10
[ 989.846084] ? new_sync_read+0x740/0x740
[ 989.851294] ? lock_acquire.part.0+0xbc/0x260
[ 989.856984] ? ksys_write+0x116/0x250
[ 989.861895] vfs_write+0x657/0x9b0
[ 989.866512] ksys_write+0x116/0x250
[ 989.871227] ? __ia32_sys_read+0xc0/0xc0
[ 989.876429] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.882019] ? rcu_is_watching+0x15/0xd0
[ 989.887283] do_syscall_64+0xf3/0x5e0
[ 989.892194] ? trace_hardirqs_off+0xd/0x30
[ 989.897588] ? exc_page_fault+0xda/0xf0
[ 989.902692] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.909164] RIP: 0033:0x7f4225baf22e
[ 989.913975] RSP: 002b:00007ffc74f835c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.923278] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4225baf22e
[ 989.932090] RDX: 0000000000000001 RSI: 0000557332870f40 RDI: 0000000000000007
[ 989.940898] RBP: 00007ffc74f835d0 R08: 0000000000000000 R09: 0000000000000000
[ 989.949697] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4225a27390
[ 989.958510] R13: 0000557332870f40 R14: 0000000000000007 R15: 0000000000000003
[ 989.967326]
[ 989.970679] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3572.
[ 989.980944] task:python3 state:D stack:25336 pid:3572 tgid:3572 ppid:3571 task_flags:0x400100 flags:0x00080802
[ 989.994150] Call Trace:
[ 989.997689]
[ 990.000842] __schedule+0xae3/0x14d0
[ 990.005656] ? io_schedule_timeout+0x140/0x140
[ 990.011442] ? find_held_lock+0x2b/0x80
[ 990.016546] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.022728] schedule+0xd4/0x260
[ 990.027151] schedule_timeout+0x13f/0x290
[ 990.032451] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 990.038824] ? trace_contention_end+0xb3/0x1a0
[ 990.044613] ? hrtimers_cpu_dying+0x260/0x260
[ 990.050306] ? ww_mutex_lock+0x160/0x160
[ 990.055506] msleep+0x1b/0x30
[ 990.059632] napi_set_threaded+0x3a5/0x440
[ 990.065030] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.070524] netif_set_threaded+0x257/0x400
[ 990.076021] threaded_store+0x1b8/0x220
[ 990.081127] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.086430] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.091728] ? sysfs_kf_write+0x65/0x160
[ 990.096929] ? sysfs_file_ops+0x140/0x140
[ 990.102224] kernfs_fop_write_iter+0x335/0x510
[ 990.108010] ? kernfs_ops+0xf0/0xf0
[ 990.112721] new_sync_write+0x333/0x750
[ 990.117826] ? __lock_acquire+0x508/0xc10
[ 990.123124] ? new_sync_read+0x740/0x740
[ 990.128325] ? lock_acquire.part.0+0xbc/0x260
[ 990.134012] ? ksys_write+0x116/0x250
[ 990.138923] vfs_write+0x657/0x9b0
[ 990.143538] ksys_write+0x116/0x250
[ 990.148243] ? __ia32_sys_read+0xc0/0xc0
[ 990.153444] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.159035] ? rcu_is_watching+0x15/0xd0
[ 990.164236] do_syscall_64+0xf3/0x5e0
[ 990.169145] ? trace_hardirqs_off+0xd/0x30
[ 990.174541] ? exc_page_fault+0xda/0xf0
[ 990.179644] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.186113] RIP: 0033:0x7f9218cf122e
[ 990.190921] RSP: 002b:00007ffca309c320 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.200221] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9218cf122e
[ 990.209031] RDX: 0000000000000001 RSI: 00005577240bef40 RDI: 0000000000000007
[ 990.217843] RBP: 00007ffca309c330 R08: 0000000000000000 R09: 0000000000000000
[ 990.226654] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f9218b69390
[ 990.235463] R13: 00005577240bef40 R14: 0000000000000007 R15: 0000000000000003
[ 990.244279]
[ 990.247530] \x0aShowing all locks held in the system:
[ 990.256069] 1 lock held by khungtaskd/186:
[ 990.261464] #0: ffffffff8facdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.272948] 4 locks held by python3/3572:
[ 990.278244] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.288136] #1: ff1100016dea4a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.299099] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.310261] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.320374] 4 locks held by python3/5623:
[ 990.325672] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.335555] #1: ff110001e9762688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.346516] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.357669] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.367759] 1 lock held by systemd/6309:
[ 990.372958]
[ 990.375489] =============================================\x0a
[ 1112.564202] INFO: task python3:5623 blocked for more than 245 seconds.
[ 1112.572349] Not tainted 7.0.0-rc6-yesk-ga6e6a7dcb853 #1
[ 1112.579598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.589190] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080000
[ 1112.602399] Call Trace:
[ 1112.605940]
[ 1112.609090] __schedule+0xae3/0x14d0
[ 1112.613911] ? io_schedule_timeout+0x140/0x140
[ 1112.619696] ? find_held_lock+0x2b/0x80
[ 1112.624803] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.630693] schedule+0xd4/0x260
[ 1112.635115] schedule_preempt_disabled+0x18/0x30
[ 1112.641095] __mutex_lock+0x6b2/0x1ee0
[ 1112.646103] ? rcu_is_watching+0x15/0xd0
[ 1112.651301] ? threaded_store+0x132/0x220
[ 1112.656603] ? ww_mutex_lock+0x160/0x160
[ 1112.661802] ? selinux_inode_free_security+0x330/0x330
[ 1112.668368] ? cap_capable+0x1ed/0x3f0
[ 1112.673368] ? threaded_store+0x132/0x220
[ 1112.678666] threaded_store+0x132/0x220
[ 1112.683763] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.689063] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.694361] ? sysfs_kf_write+0x65/0x160
[ 1112.699560] ? sysfs_file_ops+0x140/0x140
[ 1112.704855] kernfs_fop_write_iter+0x335/0x510
[ 1112.710645] ? kernfs_ops+0xf0/0xf0
[ 1112.715357] new_sync_write+0x333/0x750
[ 1112.720453] ? __lock_acquire+0x508/0xc10
[ 1112.725749] ? new_sync_read+0x740/0x740
[ 1112.730952] ? lock_acquire.part.0+0xbc/0x260
[ 1112.736637] ? ksys_write+0x116/0x250
[ 1112.741550] vfs_write+0x657/0x9b0
[ 1112.746167] ksys_write+0x116/0x250
[ 1112.750882] ? __ia32_sys_read+0xc0/0xc0
[ 1112.756081] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.761664] ? rcu_is_watching+0x15/0xd0
[ 1112.766862] do_syscall_64+0xf3/0x5e0
[ 1112.771771] ? trace_hardirqs_off+0xd/0x30
[ 1112.777169] ? exc_page_fault+0xda/0xf0
[ 1112.782272] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.788741] RIP: 0033:0x7f4225baf22e
[ 1112.793551] RSP: 002b:00007ffc74f835c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.802840] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4225baf22e
[ 1112.811650] RDX: 0000000000000001 RSI: 0000557332870f40 RDI: 0000000000000007
[ 1112.820459] RBP: 00007ffc74f835d0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.829268] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4225a27390
[ 1112.838077] R13: 0000557332870f40 R14: 0000000000000007 R15: 0000000000000003
[ 1112.846891]
[ 1112.850167] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3572.
[ 1112.860443] task:python3 state:D stack:25336 pid:3572 tgid:3572 ppid:3571 task_flags:0x400100 flags:0x00080802
[ 1112.873646] Call Trace:
[ 1112.877185]
[ 1112.880338] __schedule+0xae3/0x14d0
[ 1112.885151] ? io_schedule_timeout+0x140/0x140
[ 1112.890935] ? find_held_lock+0x2b/0x80
[ 1112.896044] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.902223] schedule+0xd4/0x260
[ 1112.906642] schedule_timeout+0x13f/0x290
[ 1112.911934] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1112.918305] ? trace_contention_end+0xb3/0x1a0
[ 1112.924091] ? hrtimers_cpu_dying+0x260/0x260
[ 1112.929783] ? ww_mutex_lock+0x160/0x160
[ 1112.934982] msleep+0x1b/0x30
[ 1112.939099] napi_set_threaded+0x3a5/0x440
[ 1112.944492] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.949990] netif_set_threaded+0x257/0x400
[ 1112.955485] threaded_store+0x1b8/0x220
[ 1112.960591] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.965889] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.971187] ? sysfs_kf_write+0x65/0x160
[ 1112.976386] ? sysfs_file_ops+0x140/0x140
[ 1112.981682] kernfs_fop_write_iter+0x335/0x510
[ 1112.987470] ? kernfs_ops+0xf0/0xf0
[ 1112.992182] new_sync_write+0x333/0x750
[ 1112.997276] ? __lock_acquire+0x508/0xc10
[ 1113.002577] ? new_sync_read+0x740/0x740
[ 1113.007777] ? lock_acquire.part.0+0xbc/0x260
[ 1113.013463] ? ksys_write+0x116/0x250
[ 1113.018371] vfs_write+0x657/0x9b0
[ 1113.022989] ksys_write+0x116/0x250
[ 1113.027703] ? __ia32_sys_read+0xc0/0xc0
[ 1113.032905] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.038496] ? rcu_is_watching+0x15/0xd0
[ 1113.043689] do_syscall_64+0xf3/0x5e0
[ 1113.048596] ? trace_hardirqs_off+0xd/0x30
[ 1113.053979] ? exc_page_fault+0xda/0xf0
[ 1113.059083] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.065550] RIP: 0033:0x7f9218cf122e
[ 1113.070359] RSP: 002b:00007ffca309c320 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.079656] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9218cf122e
[ 1113.088465] RDX: 0000000000000001 RSI: 00005577240bef40 RDI: 0000000000000007
[ 1113.097275] RBP: 00007ffca309c330 R08: 0000000000000000 R09: 0000000000000000
[ 1113.106083] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f9218b69390
[ 1113.114892] R13: 00005577240bef40 R14: 0000000000000007 R15: 0000000000000003
[ 1113.123706]
[ 1113.126973] \x0aShowing all locks held in the system:
[ 1113.135514] 1 lock held by khungtaskd/186:
[ 1113.140909] #0: ffffffff8facdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.152385] 2 locks held by systemd-journal/798:
[ 1113.158363] #0: ff1100180e0b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1113.168250] #1: ffffffff8fcc8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.181075] 4 locks held by python3/3572:
[ 1113.186368] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.196258] #1: ff1100016dea4a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.207221] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.218380] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.228479] 4 locks held by python3/5623:
[ 1113.233779] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.243672] #1: ff110001e9762688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.254633] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.265791] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.275884]
[ 1113.278349] =============================================\x0a
[ 1235.445461] INFO: task python3:5623 blocked for more than 368 seconds.
[ 1235.453614] Not tainted 7.0.0-rc6-yesk-ga6e6a7dcb853 #1
[ 1235.460864] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.470559] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080002
[ 1235.483769] Call Trace:
[ 1235.487381]
[ 1235.490523] __schedule+0xae3/0x14d0
[ 1235.495338] ? io_schedule_timeout+0x140/0x140
[ 1235.501121] ? find_held_lock+0x2b/0x80
[ 1235.506228] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.512111] schedule+0xd4/0x260
[ 1235.516530] schedule_preempt_disabled+0x18/0x30
[ 1235.522510] __mutex_lock+0x6b2/0x1ee0
[ 1235.527517] ? rcu_is_watching+0x15/0xd0
[ 1235.532714] ? threaded_store+0x132/0x220
[ 1235.538017] ? ww_mutex_lock+0x160/0x160
[ 1235.543218] ? selinux_inode_free_security+0x330/0x330
[ 1235.549789] ? cap_capable+0x1ed/0x3f0
[ 1235.554800] ? threaded_store+0x132/0x220
[ 1235.560098] threaded_store+0x132/0x220
[ 1235.565202] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.570498] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.575799] ? sysfs_kf_write+0x65/0x160
[ 1235.580999] ? sysfs_file_ops+0x140/0x140
[ 1235.586301] kernfs_fop_write_iter+0x335/0x510
[ 1235.592090] ? kernfs_ops+0xf0/0xf0
[ 1235.596802] new_sync_write+0x333/0x750
[ 1235.601897] ? __lock_acquire+0x508/0xc10
[ 1235.607195] ? new_sync_read+0x740/0x740
[ 1235.612397] ? lock_acquire.part.0+0xbc/0x260
[ 1235.618084] ? ksys_write+0x116/0x250
[ 1235.622991] vfs_write+0x657/0x9b0
[ 1235.627609] ksys_write+0x116/0x250
[ 1235.632325] ? __ia32_sys_read+0xc0/0xc0
[ 1235.637524] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.643115] ? rcu_is_watching+0x15/0xd0
[ 1235.648317] do_syscall_64+0xf3/0x5e0
[ 1235.653228] ? trace_hardirqs_off+0xd/0x30
[ 1235.658624] ? exc_page_fault+0xda/0xf0
[ 1235.663719] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.670186] RIP: 0033:0x7f4225baf22e
[ 1235.674999] RSP: 002b:00007ffc74f835c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.684303] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4225baf22e
[ 1235.693132] RDX: 0000000000000001 RSI: 0000557332870f40 RDI: 0000000000000007
[ 1235.701959] RBP: 00007ffc74f835d0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.710767] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4225a27390
[ 1235.719563] R13: 0000557332870f40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.728375]
[ 1235.731729] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3572.
[ 1235.742001] task:python3 state:D stack:25336 pid:3572 tgid:3572 ppid:3571 task_flags:0x400100 flags:0x00080802
[ 1235.755207] Call Trace:
[ 1235.758745]
[ 1235.761894] __schedule+0xae3/0x14d0
[ 1235.766706] ? io_schedule_timeout+0x140/0x140
[ 1235.772491] ? find_held_lock+0x2b/0x80
[ 1235.777593] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.783772] schedule+0xd4/0x260
[ 1235.788190] schedule_timeout+0x13f/0x290
[ 1235.793484] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1235.799852] ? trace_contention_end+0xb3/0x1a0
[ 1235.805636] ? hrtimers_cpu_dying+0x260/0x260
[ 1235.811323] ? ww_mutex_lock+0x160/0x160
[ 1235.816520] msleep+0x1b/0x30
[ 1235.820645] napi_set_threaded+0x3a5/0x440
[ 1235.826041] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.831532] netif_set_threaded+0x257/0x400
[ 1235.837026] threaded_store+0x1b8/0x220
[ 1235.842129] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.847426] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.852712] ? sysfs_kf_write+0x65/0x160
[ 1235.857909] ? sysfs_file_ops+0x140/0x140
[ 1235.863201] kernfs_fop_write_iter+0x335/0x510
[ 1235.868983] ? kernfs_ops+0xf0/0xf0
[ 1235.873695] new_sync_write+0x333/0x750
[ 1235.878799] ? __lock_acquire+0x508/0xc10
[ 1235.884097] ? new_sync_read+0x740/0x740
[ 1235.889300] ? lock_acquire.part.0+0xbc/0x260
[ 1235.894986] ? ksys_write+0x116/0x250
[ 1235.899896] vfs_write+0x657/0x9b0
[ 1235.904509] ksys_write+0x116/0x250
[ 1235.909219] ? __ia32_sys_read+0xc0/0xc0
[ 1235.914418] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.920005] ? rcu_is_watching+0x15/0xd0
[ 1235.925206] do_syscall_64+0xf3/0x5e0
[ 1235.930110] ? trace_hardirqs_off+0xd/0x30
[ 1235.935501] ? exc_page_fault+0xda/0xf0
[ 1235.940601] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.947068] RIP: 0033:0x7f9218cf122e
[ 1235.951877] RSP: 002b:00007ffca309c320 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.961176] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9218cf122e
[ 1235.969984] RDX: 0000000000000001 RSI: 00005577240bef40 RDI: 0000000000000007
[ 1235.978792] RBP: 00007ffca309c330 R08: 0000000000000000 R09: 0000000000000000
[ 1235.987591] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f9218b69390
[ 1235.996390] R13: 00005577240bef40 R14: 0000000000000007 R15: 0000000000000003
[ 1236.005203]
[ 1236.008470] \x0aShowing all locks held in the system:
[ 1236.017016] 1 lock held by khungtaskd/186:
[ 1236.022410] #0: ffffffff8facdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.033884] 3 locks held by systemd-journal/798:
[ 1236.039862] #0: ff1100180e9b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1236.049747] #1: ff1100019f4ff1c0 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.060513] #2: ff110001b49358e0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1236.072069] 4 locks held by python3/3572:
[ 1236.077362] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.087248] #1: ff1100016dea4a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.098208] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.109362] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.119470] 4 locks held by python3/5623:
[ 1236.124760] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.134644] #1: ff110001e9762688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.145604] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.156766] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.166854] 5 locks held by systemd/7089:
[ 1236.172146] #0: ff1100012ee093f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.182036] #1: ff110001b855fa88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.192995] #2: ffffffff8fb37108 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x560
[ 1236.204638] #3: ffffffff8f968b50 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 1236.216184] #4: ffffffff8fb36e10 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 1236.228514]
[ 1236.230977] =============================================\x0a
[ 1358.326745] INFO: task python3:5623 blocked for more than 491 seconds.
[ 1358.334895] Not tainted 7.0.0-rc6-yesk-ga6e6a7dcb853 #1
[ 1358.342144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.351735] task:python3 state:D stack:25336 pid:5623 tgid:5623 ppid:5622 task_flags:0x400100 flags:0x00080002
[ 1358.364942] Call Trace:
[ 1358.368482]
[ 1358.371635] __schedule+0xae3/0x14d0
[ 1358.376452] ? io_schedule_timeout+0x140/0x140
[ 1358.382238] ? find_held_lock+0x2b/0x80
[ 1358.387343] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.393219] schedule+0xd4/0x260
[ 1358.397640] schedule_preempt_disabled+0x18/0x30
[ 1358.403620] __mutex_lock+0x6b2/0x1ee0
[ 1358.408628] ? rcu_is_watching+0x15/0xd0
[ 1358.413829] ? threaded_store+0x132/0x220
[ 1358.419129] ? ww_mutex_lock+0x160/0x160
[ 1358.424329] ? selinux_inode_free_security+0x330/0x330
[ 1358.430899] ? cap_capable+0x1ed/0x3f0
[ 1358.435908] ? threaded_store+0x132/0x220
[ 1358.441207] threaded_store+0x132/0x220
[ 1358.446308] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.451611] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.456910] ? sysfs_kf_write+0x65/0x160
[ 1358.462111] ? sysfs_file_ops+0x140/0x140
[ 1358.467407] kernfs_fop_write_iter+0x335/0x510
[ 1358.473191] ? kernfs_ops+0xf0/0xf0
[ 1358.477905] new_sync_write+0x333/0x750
[ 1358.483009] ? __lock_acquire+0x508/0xc10
[ 1358.488297] ? new_sync_read+0x740/0x740
[ 1358.493497] ? lock_acquire.part.0+0xbc/0x260
[ 1358.499175] ? ksys_write+0x116/0x250
[ 1358.504079] vfs_write+0x657/0x9b0
[ 1358.508697] ksys_write+0x116/0x250
[ 1358.513408] ? __ia32_sys_read+0xc0/0xc0
[ 1358.518608] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.524201] ? rcu_is_watching+0x15/0xd0
[ 1358.529403] do_syscall_64+0xf3/0x5e0
[ 1358.534311] ? trace_hardirqs_off+0xd/0x30
[ 1358.539706] ? exc_page_fault+0xda/0xf0
[ 1358.544807] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.551274] RIP: 0033:0x7f4225baf22e
[ 1358.556084] RSP: 002b:00007ffc74f835c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.565372] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4225baf22e
[ 1358.574172] RDX: 0000000000000001 RSI: 0000557332870f40 RDI: 0000000000000007
[ 1358.582969] RBP: 00007ffc74f835d0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.591779] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4225a27390
[ 1358.600589] R13: 0000557332870f40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.609402]
[ 1358.612667] INFO: task python3:5623 is blocked on a mutex likely owned by task python3:3572.
[ 1358.622941] task:python3 state:D stack:25336 pid:3572 tgid:3572 ppid:3571 task_flags:0x400100 flags:0x00080802
[ 1358.636142] Call Trace:
[ 1358.639682]
[ 1358.642832] __schedule+0xae3/0x14d0
[ 1358.647642] ? io_schedule_timeout+0x140/0x140
[ 1358.653426] ? find_held_lock+0x2b/0x80
[ 1358.658522] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.664701] schedule+0xd4/0x260
[ 1358.669120] schedule_timeout+0x13f/0x290
[ 1358.674408] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1358.680779] ? trace_contention_end+0xb3/0x1a0
[ 1358.686565] ? hrtimers_cpu_dying+0x260/0x260
[ 1358.692256] ? ww_mutex_lock+0x160/0x160
[ 1358.697458] msleep+0x1b/0x30
[ 1358.701585] napi_set_threaded+0x3a5/0x440
[ 1358.706980] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.712473] netif_set_threaded+0x257/0x400
[ 1358.717967] threaded_store+0x1b8/0x220
[ 1358.723071] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.728369] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.733657] ? sysfs_kf_write+0x65/0x160
[ 1358.738855] ? sysfs_file_ops+0x140/0x140
[ 1358.744152] kernfs_fop_write_iter+0x335/0x510
[ 1358.749938] ? kernfs_ops+0xf0/0xf0
[ 1358.754651] new_sync_write+0x333/0x750
[ 1358.759752] ? __lock_acquire+0x508/0xc10
[ 1358.765049] ? new_sync_read+0x740/0x740
[ 1358.770251] ? lock_acquire.part.0+0xbc/0x260
[ 1358.775938] ? ksys_write+0x116/0x250
[ 1358.780849] vfs_write+0x657/0x9b0
[ 1358.785466] ksys_write+0x116/0x250
[ 1358.790178] ? __ia32_sys_read+0xc0/0xc0
[ 1358.795379] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.800970] ? rcu_is_watching+0x15/0xd0
[ 1358.806171] do_syscall_64+0xf3/0x5e0
[ 1358.811077] ? trace_hardirqs_off+0xd/0x30
[ 1358.816472] ? exc_page_fault+0xda/0xf0
[ 1358.821573] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.828042] RIP: 0033:0x7f9218cf122e
[ 1358.832852] RSP: 002b:00007ffca309c320 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.842150] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f9218cf122e
[ 1358.850959] RDX: 0000000000000001 RSI: 00005577240bef40 RDI: 0000000000000007
[ 1358.859769] RBP: 00007ffca309c330 R08: 0000000000000000 R09: 0000000000000000
[ 1358.868580] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f9218b69390
[ 1358.877388] R13: 00005577240bef40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.886203]
[ 1358.889476] \x0aShowing all locks held in the system:
[ 1358.898012] 1 lock held by khungtaskd/186:
[ 1358.903406] #0: ffffffff8facdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.914880] 3 locks held by systemd-journal/798:
[ 1358.920860] #0: ff1100180ddb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1358.930750] #1: ffffffff8fcc8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.943565] #2: ff11000188580a70 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.955128] 4 locks held by python3/3572:
[ 1358.960422] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.970311] #1: ff1100016dea4a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.981271] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.992429] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.002525] 4 locks held by python3/5623:
[ 1359.007822] #0: ff1100012eac13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.017711] #1: ff110001e9762688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.028667] #2: ff1100012199f0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.039822] #3: ff11000121959d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.049914]
[ 1359.052382] =============================================\x0a