[ 761.132595] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 761.137887] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py [ 761.299472] kselftest: Running tests in drivers/net [ 989.718483] INFO: task python3:5318 blocked for more than 122 seconds. [ 989.726719] Not tainted 7.1.0-szyj-g301dae6d4adb #1 [ 989.734270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.743859] task:python3 state:D stack:25704 pid:5318 tgid:5318 ppid:5317 task_flags:0x400100 flags:0x00080000 [ 989.757062] Call Trace: [ 989.760601] [ 989.763750] __schedule+0xeeb/0x2010 [ 989.768565] ? io_schedule_timeout+0x140/0x140 [ 989.774349] ? find_held_lock+0x2b/0x80 [ 989.779445] schedule+0xe7/0x2f0 [ 989.783865] schedule_preempt_disabled+0x18/0x30 [ 989.789836] __mutex_lock+0x6a1/0x1f10 [ 989.794840] ? threaded_store+0x132/0x220 [ 989.800140] ? ww_mutex_lock+0x160/0x160 [ 989.805336] ? selinux_inode_free_security+0x330/0x330 [ 989.811904] ? cap_capable+0x1d7/0x3d0 [ 989.816911] ? sysfs_file_ops+0x140/0x140 [ 989.822207] ? threaded_store+0x132/0x220 [ 989.827503] threaded_store+0x132/0x220 [ 989.832604] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.837901] ? sysfs_file_ops+0x140/0x140 [ 989.843202] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.848500] ? sysfs_kf_write+0x65/0x160 [ 989.853690] kernfs_fop_write_iter+0x335/0x510 [ 989.859465] ? kernfs_ops+0xf0/0xf0 [ 989.864177] new_sync_write+0x33e/0x760 [ 989.869280] ? __lock_acquire+0x518/0xc20 [ 989.874577] ? new_sync_read+0x750/0x750 [ 989.879774] ? lock_acquire.part.0+0xbc/0x260 [ 989.885461] ? ksys_write+0x116/0x250 [ 989.890371] vfs_write+0x657/0x970 [ 989.894984] ksys_write+0x116/0x250 [ 989.899691] ? __ia32_sys_read+0xc0/0xc0 [ 989.904889] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.910477] ? rcu_is_watching+0x15/0xd0 [ 989.915678] do_syscall_64+0xf3/0x650 [ 989.920582] ? trace_hardirqs_off+0xd/0x30 [ 989.925975] ? exc_page_fault+0xda/0xf0 [ 989.931077] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.937534] RIP: 0033:0x7fe584de964e [ 989.942341] RSP: 002b:00007ffd2d1e5940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.951636] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe584de964e [ 989.960441] RDX: 0000000000000001 RSI: 000056196505cbf0 RDI: 0000000000000007 [ 989.969248] RBP: 00007ffd2d1e5950 R08: 0000000000000000 R09: 0000000000000000 [ 989.978055] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe584c62390 [ 989.986867] R13: 000056196505cbf0 R14: 0000000000000007 R15: 00007fe584bf5120 [ 989.995678] [ 989.998956] INFO: task python3:5318 is blocked on a mutex likely owned by task python3:3358. [ 990.009229] task:python3 state:D stack:25704 pid:3358 tgid:3358 ppid:3357 task_flags:0x400100 flags:0x00080803 [ 990.022427] Call Trace: [ 990.025967] [ 990.029115] __schedule+0xeeb/0x2010 [ 990.033923] ? io_schedule_timeout+0x140/0x140 [ 990.039705] ? find_held_lock+0x2b/0x80 [ 990.044810] schedule+0xe7/0x2f0 [ 990.049228] schedule_timeout+0x13f/0x290 [ 990.054524] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.060890] ? trace_contention_end+0xb3/0x180 [ 990.066674] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.072355] ? ww_mutex_lock+0x160/0x160 [ 990.077543] msleep+0x1b/0x30 [ 990.081667] napi_set_threaded+0x3a5/0x440 [ 990.087062] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.092556] netif_set_threaded+0x257/0x400 [ 990.098047] ? sysfs_file_ops+0x140/0x140 [ 990.103344] threaded_store+0x1b8/0x220 [ 990.108446] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.113733] ? sysfs_file_ops+0x140/0x140 [ 990.119028] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.124328] ? sysfs_kf_write+0x65/0x160 [ 990.129526] kernfs_fop_write_iter+0x335/0x510 [ 990.135310] ? kernfs_ops+0xf0/0xf0 [ 990.140021] new_sync_write+0x33e/0x760 [ 990.145122] ? __lock_acquire+0x518/0xc20 [ 990.150417] ? new_sync_read+0x750/0x750 [ 990.155617] ? lock_acquire.part.0+0xbc/0x260 [ 990.161305] ? ksys_write+0x116/0x250 [ 990.166211] vfs_write+0x657/0x970 [ 990.170827] ksys_write+0x116/0x250 [ 990.175527] ? __ia32_sys_read+0xc0/0xc0 [ 990.180723] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.186314] ? rcu_is_watching+0x15/0xd0 [ 990.191505] do_syscall_64+0xf3/0x650 [ 990.196408] ? irq_exit_rcu+0x1a/0x30 [ 990.201316] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.207781] RIP: 0033:0x7f3dd34a764e [ 990.212589] RSP: 002b:00007ffc42264720 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.221882] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3dd34a764e [ 990.230691] RDX: 0000000000000001 RSI: 000056334aa16bf0 RDI: 0000000000000007 [ 990.239502] RBP: 00007ffc42264730 R08: 0000000000000000 R09: 0000000000000000 [ 990.248307] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3dd3320390 [ 990.257112] R13: 000056334aa16bf0 R14: 0000000000000007 R15: 00007f3dd32b5120 [ 990.265926] [ 990.269185] \x0aShowing all locks held in the system: [ 990.277720] 1 lock held by khungtaskd/184: [ 990.283111] #0: ffffffffb8302b40 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.294596] 4 locks held by python3/3358: [ 990.299892] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.309780] #1: ff110001ed04f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 990.320742] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.331888] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.341979] 4 locks held by python3/5318: [ 990.347271] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.357157] #1: ff11000141e25c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 990.368119] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.379277] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.389366] [ 990.391830] =============================================\x0a [ 1112.599736] INFO: task python3:5318 blocked for more than 245 seconds. [ 1112.607873] Not tainted 7.1.0-szyj-g301dae6d4adb #1 [ 1112.614731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.624312] task:python3 state:D stack:25704 pid:5318 tgid:5318 ppid:5317 task_flags:0x400100 flags:0x00080000 [ 1112.637517] Call Trace: [ 1112.641058] [ 1112.644208] __schedule+0xeeb/0x2010 [ 1112.649024] ? io_schedule_timeout+0x140/0x140 [ 1112.654808] ? find_held_lock+0x2b/0x80 [ 1112.659909] schedule+0xe7/0x2f0 [ 1112.664331] schedule_preempt_disabled+0x18/0x30 [ 1112.670310] __mutex_lock+0x6a1/0x1f10 [ 1112.675314] ? threaded_store+0x132/0x220 [ 1112.680616] ? ww_mutex_lock+0x160/0x160 [ 1112.685813] ? selinux_inode_free_security+0x330/0x330 [ 1112.692382] ? cap_capable+0x1d7/0x3d0 [ 1112.697388] ? sysfs_file_ops+0x140/0x140 [ 1112.702685] ? threaded_store+0x132/0x220 [ 1112.707980] threaded_store+0x132/0x220 [ 1112.713085] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.718379] ? sysfs_file_ops+0x140/0x140 [ 1112.723677] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.728969] ? sysfs_kf_write+0x65/0x160 [ 1112.734171] kernfs_fop_write_iter+0x335/0x510 [ 1112.739959] ? kernfs_ops+0xf0/0xf0 [ 1112.744671] new_sync_write+0x33e/0x760 [ 1112.749774] ? __lock_acquire+0x518/0xc20 [ 1112.755072] ? new_sync_read+0x750/0x750 [ 1112.760271] ? lock_acquire.part.0+0xbc/0x260 [ 1112.765950] ? ksys_write+0x116/0x250 [ 1112.770861] vfs_write+0x657/0x970 [ 1112.775476] ksys_write+0x116/0x250 [ 1112.780190] ? __ia32_sys_read+0xc0/0xc0 [ 1112.785387] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.790975] ? rcu_is_watching+0x15/0xd0 [ 1112.796177] do_syscall_64+0xf3/0x650 [ 1112.801083] ? trace_hardirqs_off+0xd/0x30 [ 1112.806476] ? exc_page_fault+0xda/0xf0 [ 1112.811583] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.818053] RIP: 0033:0x7fe584de964e [ 1112.822864] RSP: 002b:00007ffd2d1e5940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.832165] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe584de964e [ 1112.840974] RDX: 0000000000000001 RSI: 000056196505cbf0 RDI: 0000000000000007 [ 1112.849785] RBP: 00007ffd2d1e5950 R08: 0000000000000000 R09: 0000000000000000 [ 1112.858596] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe584c62390 [ 1112.867403] R13: 000056196505cbf0 R14: 0000000000000007 R15: 00007fe584bf5120 [ 1112.876218] [ 1112.879493] INFO: task python3:5318 is blocked on a mutex likely owned by task python3:3358. [ 1112.889765] task:python3 state:D stack:25704 pid:3358 tgid:3358 ppid:3357 task_flags:0x400100 flags:0x00080803 [ 1112.902967] Call Trace: [ 1112.906505] [ 1112.909657] __schedule+0xeeb/0x2010 [ 1112.914471] ? io_schedule_timeout+0x140/0x140 [ 1112.920256] ? find_held_lock+0x2b/0x80 [ 1112.925354] schedule+0xe7/0x2f0 [ 1112.929773] schedule_timeout+0x13f/0x290 [ 1112.935070] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.941440] ? trace_contention_end+0xb3/0x180 [ 1112.947227] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.952919] ? ww_mutex_lock+0x160/0x160 [ 1112.958120] msleep+0x1b/0x30 [ 1112.962246] napi_set_threaded+0x3a5/0x440 [ 1112.967640] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.973136] netif_set_threaded+0x257/0x400 [ 1112.978630] ? sysfs_file_ops+0x140/0x140 [ 1112.983929] threaded_store+0x1b8/0x220 [ 1112.989033] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.994328] ? sysfs_file_ops+0x140/0x140 [ 1112.999626] ? sysfs_file_kobj+0xbd/0x1e0 [ 1113.004913] ? sysfs_kf_write+0x65/0x160 [ 1113.010113] kernfs_fop_write_iter+0x335/0x510 [ 1113.015898] ? kernfs_ops+0xf0/0xf0 [ 1113.020608] new_sync_write+0x33e/0x760 [ 1113.025713] ? __lock_acquire+0x518/0xc20 [ 1113.031012] ? new_sync_read+0x750/0x750 [ 1113.036211] ? lock_acquire.part.0+0xbc/0x260 [ 1113.041888] ? ksys_write+0x116/0x250 [ 1113.046799] vfs_write+0x657/0x970 [ 1113.051414] ksys_write+0x116/0x250 [ 1113.056126] ? __ia32_sys_read+0xc0/0xc0 [ 1113.061326] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.066915] ? rcu_is_watching+0x15/0xd0 [ 1113.072116] do_syscall_64+0xf3/0x650 [ 1113.077023] ? irq_exit_rcu+0x1a/0x30 [ 1113.081930] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.088399] RIP: 0033:0x7f3dd34a764e [ 1113.093209] RSP: 002b:00007ffc42264720 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.102499] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3dd34a764e [ 1113.111312] RDX: 0000000000000001 RSI: 000056334aa16bf0 RDI: 0000000000000007 [ 1113.120121] RBP: 00007ffc42264730 R08: 0000000000000000 R09: 0000000000000000 [ 1113.128932] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3dd3320390 [ 1113.137738] R13: 000056334aa16bf0 R14: 0000000000000007 R15: 00007f3dd32b5120 [ 1113.146552] [ 1113.149816] \x0aShowing all locks held in the system: [ 1113.158355] 1 lock held by khungtaskd/184: [ 1113.163749] #0: ffffffffb8302b40 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.175223] 2 locks held by systemd-journal/808: [ 1113.181201] #0: ff110014e1bb73a0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18c/0x2010 [ 1113.191091] #1: ffffffffb850f000 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.203920] 4 locks held by python3/3358: [ 1113.209213] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.219103] #1: ff110001ed04f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1113.230065] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.241222] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.251318] 4 locks held by python3/5318: [ 1113.256617] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.266508] #1: ff11000141e25c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1113.277472] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.288627] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.298720] [ 1113.301189] =============================================\x0a [ 1235.480982] INFO: task python3:5318 blocked for more than 368 seconds. [ 1235.489128] Not tainted 7.1.0-szyj-g301dae6d4adb #1 [ 1235.495991] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.505579] task:python3 state:D stack:25704 pid:5318 tgid:5318 ppid:5317 task_flags:0x400100 flags:0x00080002 [ 1235.518785] Call Trace: [ 1235.522324] [ 1235.525472] __schedule+0xeeb/0x2010 [ 1235.530289] ? io_schedule_timeout+0x140/0x140 [ 1235.536071] ? find_held_lock+0x2b/0x80 [ 1235.541176] schedule+0xe7/0x2f0 [ 1235.545587] schedule_preempt_disabled+0x18/0x30 [ 1235.551555] __mutex_lock+0x6a1/0x1f10 [ 1235.556559] ? threaded_store+0x132/0x220 [ 1235.561858] ? ww_mutex_lock+0x160/0x160 [ 1235.567054] ? selinux_inode_free_security+0x330/0x330 [ 1235.573621] ? cap_capable+0x1d7/0x3d0 [ 1235.578629] ? sysfs_file_ops+0x140/0x140 [ 1235.583927] ? threaded_store+0x132/0x220 [ 1235.589226] threaded_store+0x132/0x220 [ 1235.594327] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.599621] ? sysfs_file_ops+0x140/0x140 [ 1235.604914] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.610209] ? sysfs_kf_write+0x65/0x160 [ 1235.615408] kernfs_fop_write_iter+0x335/0x510 [ 1235.621181] ? kernfs_ops+0xf0/0xf0 [ 1235.625888] new_sync_write+0x33e/0x760 [ 1235.630992] ? __lock_acquire+0x518/0xc20 [ 1235.636288] ? new_sync_read+0x750/0x750 [ 1235.641487] ? lock_acquire.part.0+0xbc/0x260 [ 1235.647165] ? ksys_write+0x116/0x250 [ 1235.652075] vfs_write+0x657/0x970 [ 1235.656689] ksys_write+0x116/0x250 [ 1235.661397] ? __ia32_sys_read+0xc0/0xc0 [ 1235.666594] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.672182] ? rcu_is_watching+0x15/0xd0 [ 1235.677383] do_syscall_64+0xf3/0x650 [ 1235.682288] ? trace_hardirqs_off+0xd/0x30 [ 1235.687683] ? exc_page_fault+0xda/0xf0 [ 1235.692786] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.699253] RIP: 0033:0x7fe584de964e [ 1235.704064] RSP: 002b:00007ffd2d1e5940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.713362] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe584de964e [ 1235.722169] RDX: 0000000000000001 RSI: 000056196505cbf0 RDI: 0000000000000007 [ 1235.730975] RBP: 00007ffd2d1e5950 R08: 0000000000000000 R09: 0000000000000000 [ 1235.739781] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe584c62390 [ 1235.748591] R13: 000056196505cbf0 R14: 0000000000000007 R15: 00007fe584bf5120 [ 1235.757405] [ 1235.760691] INFO: task python3:5318 is blocked on a mutex likely owned by task python3:3358. [ 1235.770963] task:python3 state:D stack:25704 pid:3358 tgid:3358 ppid:3357 task_flags:0x400100 flags:0x00080803 [ 1235.784165] Call Trace: [ 1235.787704] [ 1235.790850] __schedule+0xeeb/0x2010 [ 1235.795660] ? io_schedule_timeout+0x140/0x140 [ 1235.801440] ? find_held_lock+0x2b/0x80 [ 1235.806543] schedule+0xe7/0x2f0 [ 1235.810963] schedule_timeout+0x13f/0x290 [ 1235.816261] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.822618] ? trace_contention_end+0xb3/0x180 [ 1235.828401] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.834090] ? ww_mutex_lock+0x160/0x160 [ 1235.839279] msleep+0x1b/0x30 [ 1235.843402] napi_set_threaded+0x3a5/0x440 [ 1235.848796] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.854292] netif_set_threaded+0x257/0x400 [ 1235.859782] ? sysfs_file_ops+0x140/0x140 [ 1235.865079] threaded_store+0x1b8/0x220 [ 1235.870177] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.875474] ? sysfs_file_ops+0x140/0x140 [ 1235.880767] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.886064] ? sysfs_kf_write+0x65/0x160 [ 1235.891261] kernfs_fop_write_iter+0x335/0x510 [ 1235.897043] ? kernfs_ops+0xf0/0xf0 [ 1235.901753] new_sync_write+0x33e/0x760 [ 1235.906852] ? __lock_acquire+0x518/0xc20 [ 1235.912146] ? new_sync_read+0x750/0x750 [ 1235.917346] ? lock_acquire.part.0+0xbc/0x260 [ 1235.923022] ? ksys_write+0x116/0x250 [ 1235.927930] vfs_write+0x657/0x970 [ 1235.932536] ksys_write+0x116/0x250 [ 1235.937240] ? __ia32_sys_read+0xc0/0xc0 [ 1235.942428] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.948015] ? rcu_is_watching+0x15/0xd0 [ 1235.953213] do_syscall_64+0xf3/0x650 [ 1235.958119] ? irq_exit_rcu+0x1a/0x30 [ 1235.963018] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.969481] RIP: 0033:0x7f3dd34a764e [ 1235.974289] RSP: 002b:00007ffc42264720 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.983586] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3dd34a764e [ 1235.992393] RDX: 0000000000000001 RSI: 000056334aa16bf0 RDI: 0000000000000007 [ 1236.001196] RBP: 00007ffc42264730 R08: 0000000000000000 R09: 0000000000000000 [ 1236.010003] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3dd3320390 [ 1236.018808] R13: 000056334aa16bf0 R14: 0000000000000007 R15: 00007f3dd32b5120 [ 1236.027621] [ 1236.030884] \x0aShowing all locks held in the system: [ 1236.039419] 1 lock held by khungtaskd/184: [ 1236.044811] #0: ffffffffb8302b40 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.056281] 2 locks held by systemd-journal/808: [ 1236.062258] #0: ff110014e1c373a0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18c/0x2010 [ 1236.072145] #1: ffffffffb850f000 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: link_path_walk+0x9a7/0x1830 [ 1236.084775] 4 locks held by python3/3358: [ 1236.090071] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.099958] #1: ff110001ed04f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1236.110920] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.122078] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.132170] 4 locks held by python3/5318: [ 1236.137465] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.147356] #1: ff11000141e25c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1236.158315] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.169473] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.179563] [ 1236.182019] =============================================\x0a [ 1358.362243] INFO: task python3:5318 blocked for more than 491 seconds. [ 1358.370396] Not tainted 7.1.0-szyj-g301dae6d4adb #1 [ 1358.377259] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.386849] task:python3 state:D stack:25704 pid:5318 tgid:5318 ppid:5317 task_flags:0x400100 flags:0x00080002 [ 1358.400055] Call Trace: [ 1358.403596] [ 1358.406747] __schedule+0xeeb/0x2010 [ 1358.411567] ? io_schedule_timeout+0x140/0x140 [ 1358.417350] ? find_held_lock+0x2b/0x80 [ 1358.422460] schedule+0xe7/0x2f0 [ 1358.426881] schedule_preempt_disabled+0x18/0x30 [ 1358.432859] __mutex_lock+0x6a1/0x1f10 [ 1358.437865] ? threaded_store+0x132/0x220 [ 1358.443165] ? ww_mutex_lock+0x160/0x160 [ 1358.448363] ? selinux_inode_free_security+0x330/0x330 [ 1358.454935] ? cap_capable+0x1d7/0x3d0 [ 1358.459944] ? sysfs_file_ops+0x140/0x140 [ 1358.465242] ? threaded_store+0x132/0x220 [ 1358.470541] threaded_store+0x132/0x220 [ 1358.475644] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.480933] ? sysfs_file_ops+0x140/0x140 [ 1358.486219] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.491515] ? sysfs_kf_write+0x65/0x160 [ 1358.496714] kernfs_fop_write_iter+0x335/0x510 [ 1358.502501] ? kernfs_ops+0xf0/0xf0 [ 1358.507214] new_sync_write+0x33e/0x760 [ 1358.512317] ? __lock_acquire+0x518/0xc20 [ 1358.517614] ? new_sync_read+0x750/0x750 [ 1358.522816] ? lock_acquire.part.0+0xbc/0x260 [ 1358.528503] ? ksys_write+0x116/0x250 [ 1358.533413] vfs_write+0x657/0x970 [ 1358.538029] ksys_write+0x116/0x250 [ 1358.542743] ? __ia32_sys_read+0xc0/0xc0 [ 1358.547944] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.553536] ? rcu_is_watching+0x15/0xd0 [ 1358.558736] do_syscall_64+0xf3/0x650 [ 1358.563636] ? trace_hardirqs_off+0xd/0x30 [ 1358.569032] ? exc_page_fault+0xda/0xf0 [ 1358.574127] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.580594] RIP: 0033:0x7fe584de964e [ 1358.585395] RSP: 002b:00007ffd2d1e5940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.594695] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe584de964e [ 1358.603505] RDX: 0000000000000001 RSI: 000056196505cbf0 RDI: 0000000000000007 [ 1358.612315] RBP: 00007ffd2d1e5950 R08: 0000000000000000 R09: 0000000000000000 [ 1358.621123] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe584c62390 [ 1358.629934] R13: 000056196505cbf0 R14: 0000000000000007 R15: 00007fe584bf5120 [ 1358.638749] [ 1358.642070] INFO: task python3:5318 is blocked on a mutex likely owned by task python3:3358. [ 1358.652347] task:python3 state:D stack:25704 pid:3358 tgid:3358 ppid:3357 task_flags:0x400100 flags:0x00080803 [ 1358.665551] Call Trace: [ 1358.669089] [ 1358.672239] __schedule+0xeeb/0x2010 [ 1358.677049] ? io_schedule_timeout+0x140/0x140 [ 1358.682835] ? find_held_lock+0x2b/0x80 [ 1358.687943] schedule+0xe7/0x2f0 [ 1358.692360] schedule_timeout+0x13f/0x290 [ 1358.697659] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.704028] ? trace_contention_end+0xb3/0x180 [ 1358.709813] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.715505] ? ww_mutex_lock+0x160/0x160 [ 1358.720704] msleep+0x1b/0x30 [ 1358.724828] napi_set_threaded+0x3a5/0x440 [ 1358.730225] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.735722] netif_set_threaded+0x257/0x400 [ 1358.741214] ? sysfs_file_ops+0x140/0x140 [ 1358.746505] threaded_store+0x1b8/0x220 [ 1358.751607] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.756897] ? sysfs_file_ops+0x140/0x140 [ 1358.762192] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.767491] ? sysfs_kf_write+0x65/0x160 [ 1358.772691] kernfs_fop_write_iter+0x335/0x510 [ 1358.778474] ? kernfs_ops+0xf0/0xf0 [ 1358.783186] new_sync_write+0x33e/0x760 [ 1358.788287] ? __lock_acquire+0x518/0xc20 [ 1358.793584] ? new_sync_read+0x750/0x750 [ 1358.798786] ? lock_acquire.part.0+0xbc/0x260 [ 1358.804474] ? ksys_write+0x116/0x250 [ 1358.809383] vfs_write+0x657/0x970 [ 1358.813997] ksys_write+0x116/0x250 [ 1358.818712] ? __ia32_sys_read+0xc0/0xc0 [ 1358.823912] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.829500] ? rcu_is_watching+0x15/0xd0 [ 1358.834700] do_syscall_64+0xf3/0x650 [ 1358.839604] ? irq_exit_rcu+0x1a/0x30 [ 1358.844514] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.850980] RIP: 0033:0x7f3dd34a764e [ 1358.855789] RSP: 002b:00007ffc42264720 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.865089] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3dd34a764e [ 1358.873895] RDX: 0000000000000001 RSI: 000056334aa16bf0 RDI: 0000000000000007 [ 1358.882704] RBP: 00007ffc42264730 R08: 0000000000000000 R09: 0000000000000000 [ 1358.891513] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3dd3320390 [ 1358.900322] R13: 000056334aa16bf0 R14: 0000000000000007 R15: 00007f3dd32b5120 [ 1358.909137] [ 1358.912407] \x0aShowing all locks held in the system: [ 1358.920953] 1 lock held by khungtaskd/184: [ 1358.926345] #0: ffffffffb8302b40 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.937836] 4 locks held by python3/3358: [ 1358.943133] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.953026] #1: ff110001ed04f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1358.963988] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.975145] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.985234] 4 locks held by python3/5318: [ 1358.990530] #0: ff1100012fcd1428 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.000422] #1: ff11000141e25c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1359.011385] #2: ff11000121fe0910 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.022547] #3: ff11000121f81e10 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.032642] [ 1359.035109] =============================================\x0a