[ 763.962415] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 763.964882] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py [ 764.095207] kselftest: Running tests in drivers/net [ 846.349832] kworker/u130:7 (762) used greatest stack depth: 21936 bytes left [ 989.711116] INFO: task python3:5434 blocked for more than 122 seconds. [ 989.719300] Not tainted 7.1.0-rc6-wvse-g05452c154d25 #1 [ 989.726790] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.736381] task:python3 state:D stack:25672 pid:5434 tgid:5434 ppid:5433 task_flags:0x400100 flags:0x00080000 [ 989.749589] Call Trace: [ 989.753127] [ 989.756278] __schedule+0xcc3/0x19c0 [ 989.761096] ? io_schedule_timeout+0x140/0x140 [ 989.766881] ? find_held_lock+0x2b/0x80 [ 989.771986] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.777867] schedule+0xd8/0x260 [ 989.782285] schedule_preempt_disabled+0x18/0x30 [ 989.788265] __mutex_lock+0x6af/0x2010 [ 989.793261] ? threaded_store+0x132/0x220 [ 989.798562] ? ww_mutex_lock+0x160/0x160 [ 989.803758] ? selinux_inode_free_security+0x330/0x330 [ 989.810326] ? cap_capable+0x1d6/0x3e0 [ 989.815333] ? threaded_store+0x132/0x220 [ 989.820629] threaded_store+0x132/0x220 [ 989.825731] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.831027] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.836318] ? sysfs_kf_write+0x65/0x160 [ 989.841508] ? sysfs_file_ops+0x140/0x140 [ 989.846803] kernfs_fop_write_iter+0x335/0x510 [ 989.852585] ? kernfs_ops+0xf0/0xf0 [ 989.857294] new_sync_write+0x333/0x750 [ 989.862394] ? __lock_acquire+0x508/0xc10 [ 989.867689] ? new_sync_read+0x740/0x740 [ 989.872889] ? lock_acquire.part.0+0xbc/0x260 [ 989.878574] ? ksys_write+0x116/0x250 [ 989.883485] vfs_write+0x657/0x9b0 [ 989.888101] ksys_write+0x116/0x250 [ 989.892809] ? __ia32_sys_read+0xc0/0xc0 [ 989.898005] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.903592] ? rcu_is_watching+0x15/0xd0 [ 989.908791] do_syscall_64+0xf3/0x650 [ 989.913694] ? irq_exit_rcu+0x1a/0x30 [ 989.918602] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.925066] RIP: 0033:0x7fccec67064e [ 989.929875] RSP: 002b:00007ffeb2343870 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.939172] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fccec67064e [ 989.947979] RDX: 0000000000000001 RSI: 000055fda857d420 RDI: 0000000000000007 [ 989.956786] RBP: 00007ffeb2343880 R08: 0000000000000000 R09: 0000000000000000 [ 989.965585] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fccec4e9390 [ 989.974383] R13: 000055fda857d420 R14: 0000000000000007 R15: 00007fccec47d120 [ 989.983193] [ 989.986498] INFO: task python3:5434 is blocked on a mutex likely owned by task python3:3453. [ 989.996775] task:python3 state:D stack:25672 pid:3453 tgid:3453 ppid:3452 task_flags:0x400100 flags:0x00080802 [ 990.009978] Call Trace: [ 990.013516] [ 990.016666] __schedule+0xcc3/0x19c0 [ 990.021468] ? io_schedule_timeout+0x140/0x140 [ 990.027252] ? find_held_lock+0x2b/0x80 [ 990.032351] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.038528] schedule+0xd8/0x260 [ 990.042945] schedule_timeout+0x13f/0x290 [ 990.048243] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.054602] ? trace_contention_end+0xb3/0x1a0 [ 990.060384] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.066074] ? ww_mutex_lock+0x160/0x160 [ 990.071274] msleep+0x1b/0x30 [ 990.075397] napi_set_threaded+0x3a5/0x440 [ 990.080790] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.086282] netif_set_threaded+0x257/0x400 [ 990.091775] threaded_store+0x1b8/0x220 [ 990.096875] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.102170] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.107464] ? sysfs_kf_write+0x65/0x160 [ 990.112664] ? sysfs_file_ops+0x140/0x140 [ 990.117957] kernfs_fop_write_iter+0x335/0x510 [ 990.123739] ? kernfs_ops+0xf0/0xf0 [ 990.128449] new_sync_write+0x333/0x750 [ 990.133552] ? __lock_acquire+0x508/0xc10 [ 990.138837] ? new_sync_read+0x740/0x740 [ 990.144037] ? lock_acquire.part.0+0xbc/0x260 [ 990.149714] ? ksys_write+0x116/0x250 [ 990.154623] vfs_write+0x657/0x9b0 [ 990.159237] ksys_write+0x116/0x250 [ 990.163946] ? __ia32_sys_read+0xc0/0xc0 [ 990.169144] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.174733] ? rcu_is_watching+0x15/0xd0 [ 990.179931] do_syscall_64+0xf3/0x650 [ 990.184836] ? trace_hardirqs_off+0xd/0x30 [ 990.190231] ? exc_page_fault+0xda/0xf0 [ 990.195334] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.201799] RIP: 0033:0x7fa552fb264e [ 990.206604] RSP: 002b:00007fffb256eef0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.215901] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa552fb264e [ 990.224705] RDX: 0000000000000001 RSI: 000055aa82f90420 RDI: 0000000000000007 [ 990.233511] RBP: 00007fffb256ef00 R08: 0000000000000000 R09: 0000000000000000 [ 990.242319] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa552e2b390 [ 990.251125] R13: 000055aa82f90420 R14: 0000000000000007 R15: 00007fa552dc1120 [ 990.259935] [ 990.263191] \x0aShowing all locks held in the system: [ 990.271730] 1 lock held by khungtaskd/185: [ 990.277124] #0: ffffffffbaefdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.288605] 4 locks held by python3/3453: [ 990.293897] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.303786] #1: ff110001400a2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.314745] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.325901] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.335991] 4 locks held by python3/5434: [ 990.341285] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.351174] #1: ff1100016cfd5c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.362132] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.373288] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.383376] [ 990.385841] =============================================\x0a [ 1112.592386] INFO: task python3:5434 blocked for more than 245 seconds. [ 1112.600533] Not tainted 7.1.0-rc6-wvse-g05452c154d25 #1 [ 1112.607781] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.617372] task:python3 state:D stack:25672 pid:5434 tgid:5434 ppid:5433 task_flags:0x400100 flags:0x00080000 [ 1112.630577] Call Trace: [ 1112.634117] [ 1112.637265] __schedule+0xcc3/0x19c0 [ 1112.642079] ? io_schedule_timeout+0x140/0x140 [ 1112.647860] ? find_held_lock+0x2b/0x80 [ 1112.652968] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.658851] schedule+0xd8/0x260 [ 1112.663268] schedule_preempt_disabled+0x18/0x30 [ 1112.669244] __mutex_lock+0x6af/0x2010 [ 1112.674248] ? threaded_store+0x132/0x220 [ 1112.679546] ? ww_mutex_lock+0x160/0x160 [ 1112.684745] ? selinux_inode_free_security+0x330/0x330 [ 1112.691306] ? cap_capable+0x1d6/0x3e0 [ 1112.696314] ? threaded_store+0x132/0x220 [ 1112.701609] threaded_store+0x132/0x220 [ 1112.706709] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.712007] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.717310] ? sysfs_kf_write+0x65/0x160 [ 1112.722512] ? sysfs_file_ops+0x140/0x140 [ 1112.727805] kernfs_fop_write_iter+0x335/0x510 [ 1112.733589] ? kernfs_ops+0xf0/0xf0 [ 1112.738299] new_sync_write+0x333/0x750 [ 1112.743400] ? __lock_acquire+0x508/0xc10 [ 1112.748695] ? new_sync_read+0x740/0x740 [ 1112.753893] ? lock_acquire.part.0+0xbc/0x260 [ 1112.759576] ? ksys_write+0x116/0x250 [ 1112.764485] vfs_write+0x657/0x9b0 [ 1112.769098] ksys_write+0x116/0x250 [ 1112.773807] ? __ia32_sys_read+0xc0/0xc0 [ 1112.779004] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.784592] ? rcu_is_watching+0x15/0xd0 [ 1112.789791] do_syscall_64+0xf3/0x650 [ 1112.794697] ? irq_exit_rcu+0x1a/0x30 [ 1112.799602] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.806068] RIP: 0033:0x7fccec67064e [ 1112.810878] RSP: 002b:00007ffeb2343870 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.820177] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fccec67064e [ 1112.828982] RDX: 0000000000000001 RSI: 000055fda857d420 RDI: 0000000000000007 [ 1112.837787] RBP: 00007ffeb2343880 R08: 0000000000000000 R09: 0000000000000000 [ 1112.846592] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fccec4e9390 [ 1112.855399] R13: 000055fda857d420 R14: 0000000000000007 R15: 00007fccec47d120 [ 1112.864214] [ 1112.867509] INFO: task python3:5434 is blocked on a mutex likely owned by task python3:3453. [ 1112.877780] task:python3 state:D stack:25672 pid:3453 tgid:3453 ppid:3452 task_flags:0x400100 flags:0x00080802 [ 1112.890982] Call Trace: [ 1112.894522] [ 1112.897669] __schedule+0xcc3/0x19c0 [ 1112.902481] ? io_schedule_timeout+0x140/0x140 [ 1112.908262] ? find_held_lock+0x2b/0x80 [ 1112.913363] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.919533] schedule+0xd8/0x260 [ 1112.923949] schedule_timeout+0x13f/0x290 [ 1112.929246] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.935613] ? trace_contention_end+0xb3/0x1a0 [ 1112.941397] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.947084] ? ww_mutex_lock+0x160/0x160 [ 1112.952281] msleep+0x1b/0x30 [ 1112.956408] napi_set_threaded+0x3a5/0x440 [ 1112.961793] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.967287] netif_set_threaded+0x257/0x400 [ 1112.972778] threaded_store+0x1b8/0x220 [ 1112.977879] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.983175] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.988471] ? sysfs_kf_write+0x65/0x160 [ 1112.993669] ? sysfs_file_ops+0x140/0x140 [ 1112.998962] kernfs_fop_write_iter+0x335/0x510 [ 1113.004744] ? kernfs_ops+0xf0/0xf0 [ 1113.009453] new_sync_write+0x333/0x750 [ 1113.014552] ? __lock_acquire+0x508/0xc10 [ 1113.019849] ? new_sync_read+0x740/0x740 [ 1113.025047] ? lock_acquire.part.0+0xbc/0x260 [ 1113.030730] ? ksys_write+0x116/0x250 [ 1113.035642] vfs_write+0x657/0x9b0 [ 1113.040256] ksys_write+0x116/0x250 [ 1113.044966] ? __ia32_sys_read+0xc0/0xc0 [ 1113.050166] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.055752] ? rcu_is_watching+0x15/0xd0 [ 1113.060952] do_syscall_64+0xf3/0x650 [ 1113.065848] ? trace_hardirqs_off+0xd/0x30 [ 1113.071241] ? exc_page_fault+0xda/0xf0 [ 1113.076343] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.082809] RIP: 0033:0x7fa552fb264e [ 1113.087614] RSP: 002b:00007fffb256eef0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.096909] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa552fb264e [ 1113.105716] RDX: 0000000000000001 RSI: 000055aa82f90420 RDI: 0000000000000007 [ 1113.114522] RBP: 00007fffb256ef00 R08: 0000000000000000 R09: 0000000000000000 [ 1113.123328] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa552e2b390 [ 1113.132136] R13: 000055aa82f90420 R14: 0000000000000007 R15: 00007fa552dc1120 [ 1113.140947] [ 1113.144213] \x0aShowing all locks held in the system: [ 1113.152753] 1 lock held by khungtaskd/185: [ 1113.158147] #0: ffffffffbaefdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.169625] 4 locks held by python3/3453: [ 1113.174919] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.184809] #1: ff110001400a2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.195769] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.206922] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.217018] 4 locks held by python3/5434: [ 1113.222312] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.232202] #1: ff1100016cfd5c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.243153] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.254308] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.264401] [ 1113.266866] =============================================\x0a [ 1235.473703] INFO: task python3:5434 blocked for more than 368 seconds. [ 1235.481850] Not tainted 7.1.0-rc6-wvse-g05452c154d25 #1 [ 1235.489101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.498688] task:python3 state:D stack:25672 pid:5434 tgid:5434 ppid:5433 task_flags:0x400100 flags:0x00080002 [ 1235.511885] Call Trace: [ 1235.515425] [ 1235.518572] __schedule+0xcc3/0x19c0 [ 1235.523387] ? io_schedule_timeout+0x140/0x140 [ 1235.529170] ? find_held_lock+0x2b/0x80 [ 1235.534272] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.540158] schedule+0xd8/0x260 [ 1235.544575] schedule_preempt_disabled+0x18/0x30 [ 1235.550554] __mutex_lock+0x6af/0x2010 [ 1235.555558] ? threaded_store+0x132/0x220 [ 1235.560857] ? ww_mutex_lock+0x160/0x160 [ 1235.566053] ? selinux_inode_free_security+0x330/0x330 [ 1235.572624] ? cap_capable+0x1d6/0x3e0 [ 1235.577629] ? threaded_store+0x132/0x220 [ 1235.582924] threaded_store+0x132/0x220 [ 1235.588025] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.593322] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.598620] ? sysfs_kf_write+0x65/0x160 [ 1235.603822] ? sysfs_file_ops+0x140/0x140 [ 1235.609116] kernfs_fop_write_iter+0x335/0x510 [ 1235.614899] ? kernfs_ops+0xf0/0xf0 [ 1235.619610] new_sync_write+0x333/0x750 [ 1235.624711] ? __lock_acquire+0x508/0xc10 [ 1235.630007] ? new_sync_read+0x740/0x740 [ 1235.635205] ? lock_acquire.part.0+0xbc/0x260 [ 1235.640891] ? ksys_write+0x116/0x250 [ 1235.645800] vfs_write+0x657/0x9b0 [ 1235.650414] ksys_write+0x116/0x250 [ 1235.655126] ? __ia32_sys_read+0xc0/0xc0 [ 1235.660323] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.665910] ? rcu_is_watching+0x15/0xd0 [ 1235.671113] do_syscall_64+0xf3/0x650 [ 1235.676017] ? irq_exit_rcu+0x1a/0x30 [ 1235.680922] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.687388] RIP: 0033:0x7fccec67064e [ 1235.692198] RSP: 002b:00007ffeb2343870 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.701496] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fccec67064e [ 1235.710303] RDX: 0000000000000001 RSI: 000055fda857d420 RDI: 0000000000000007 [ 1235.719110] RBP: 00007ffeb2343880 R08: 0000000000000000 R09: 0000000000000000 [ 1235.727914] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fccec4e9390 [ 1235.736722] R13: 000055fda857d420 R14: 0000000000000007 R15: 00007fccec47d120 [ 1235.745536] [ 1235.748826] INFO: task python3:5434 is blocked on a mutex likely owned by task python3:3453. [ 1235.759095] task:python3 state:D stack:25672 pid:3453 tgid:3453 ppid:3452 task_flags:0x400100 flags:0x00080802 [ 1235.772295] Call Trace: [ 1235.775833] [ 1235.778981] __schedule+0xcc3/0x19c0 [ 1235.783792] ? io_schedule_timeout+0x140/0x140 [ 1235.789573] ? find_held_lock+0x2b/0x80 [ 1235.794673] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.800854] schedule+0xd8/0x260 [ 1235.805272] schedule_timeout+0x13f/0x290 [ 1235.810569] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.816936] ? trace_contention_end+0xb3/0x1a0 [ 1235.822718] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.828406] ? ww_mutex_lock+0x160/0x160 [ 1235.833602] msleep+0x1b/0x30 [ 1235.837726] napi_set_threaded+0x3a5/0x440 [ 1235.843120] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.848615] netif_set_threaded+0x257/0x400 [ 1235.854105] threaded_store+0x1b8/0x220 [ 1235.859207] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.864505] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.869800] ? sysfs_kf_write+0x65/0x160 [ 1235.874998] ? sysfs_file_ops+0x140/0x140 [ 1235.880291] kernfs_fop_write_iter+0x335/0x510 [ 1235.886073] ? kernfs_ops+0xf0/0xf0 [ 1235.890781] new_sync_write+0x333/0x750 [ 1235.895880] ? __lock_acquire+0x508/0xc10 [ 1235.901173] ? new_sync_read+0x740/0x740 [ 1235.906371] ? lock_acquire.part.0+0xbc/0x260 [ 1235.912054] ? ksys_write+0x116/0x250 [ 1235.916965] vfs_write+0x657/0x9b0 [ 1235.921579] ksys_write+0x116/0x250 [ 1235.926289] ? __ia32_sys_read+0xc0/0xc0 [ 1235.931490] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.937068] ? rcu_is_watching+0x15/0xd0 [ 1235.942268] do_syscall_64+0xf3/0x650 [ 1235.947174] ? trace_hardirqs_off+0xd/0x30 [ 1235.952566] ? exc_page_fault+0xda/0xf0 [ 1235.957668] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.964133] RIP: 0033:0x7fa552fb264e [ 1235.968939] RSP: 002b:00007fffb256eef0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.978234] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa552fb264e [ 1235.987042] RDX: 0000000000000001 RSI: 000055aa82f90420 RDI: 0000000000000007 [ 1235.995840] RBP: 00007fffb256ef00 R08: 0000000000000000 R09: 0000000000000000 [ 1236.004647] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa552e2b390 [ 1236.013452] R13: 000055aa82f90420 R14: 0000000000000007 R15: 00007fa552dc1120 [ 1236.022262] [ 1236.025531] \x0aShowing all locks held in the system: [ 1236.034070] 1 lock held by khungtaskd/185: [ 1236.039464] #0: ffffffffbaefdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.050934] 2 locks held by systemd-journal/805: [ 1236.056913] #0: ff1100180df374e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1236.066800] #1: ffffffffbb102ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: dput.part.0+0x2fd/0x410 [ 1236.079038] 4 locks held by python3/3453: [ 1236.084330] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.094219] #1: ff110001400a2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.105170] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.116326] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.126421] 4 locks held by python3/5434: [ 1236.131718] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.141606] #1: ff1100016cfd5c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.152568] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.163712] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.173803] 4 locks held by sshd-session/6911: [ 1236.179578] [ 1236.182043] =============================================\x0a [ 1358.355087] INFO: task python3:5434 blocked for more than 491 seconds. [ 1358.363237] Not tainted 7.1.0-rc6-wvse-g05452c154d25 #1 [ 1358.370488] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.380078] task:python3 state:D stack:25672 pid:5434 tgid:5434 ppid:5433 task_flags:0x400100 flags:0x00080002 [ 1358.393286] Call Trace: [ 1358.396828] [ 1358.399982] __schedule+0xcc3/0x19c0 [ 1358.404798] ? io_schedule_timeout+0x140/0x140 [ 1358.410585] ? find_held_lock+0x2b/0x80 [ 1358.415689] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.421574] schedule+0xd8/0x260 [ 1358.425993] schedule_preempt_disabled+0x18/0x30 [ 1358.431972] __mutex_lock+0x6af/0x2010 [ 1358.436976] ? threaded_store+0x132/0x220 [ 1358.442275] ? ww_mutex_lock+0x160/0x160 [ 1358.447466] ? selinux_inode_free_security+0x330/0x330 [ 1358.454036] ? cap_capable+0x1d6/0x3e0 [ 1358.459042] ? threaded_store+0x132/0x220 [ 1358.464341] threaded_store+0x132/0x220 [ 1358.469443] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.474742] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.480043] ? sysfs_kf_write+0x65/0x160 [ 1358.485247] ? sysfs_file_ops+0x140/0x140 [ 1358.490542] kernfs_fop_write_iter+0x335/0x510 [ 1358.496325] ? kernfs_ops+0xf0/0xf0 [ 1358.501038] new_sync_write+0x333/0x750 [ 1358.506144] ? __lock_acquire+0x508/0xc10 [ 1358.511440] ? new_sync_read+0x740/0x740 [ 1358.516644] ? lock_acquire.part.0+0xbc/0x260 [ 1358.522331] ? ksys_write+0x116/0x250 [ 1358.527231] vfs_write+0x657/0x9b0 [ 1358.531846] ksys_write+0x116/0x250 [ 1358.536556] ? __ia32_sys_read+0xc0/0xc0 [ 1358.541755] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.547342] ? rcu_is_watching+0x15/0xd0 [ 1358.552543] do_syscall_64+0xf3/0x650 [ 1358.557449] ? irq_exit_rcu+0x1a/0x30 [ 1358.562360] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.568829] RIP: 0033:0x7fccec67064e [ 1358.573641] RSP: 002b:00007ffeb2343870 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.582940] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fccec67064e [ 1358.591748] RDX: 0000000000000001 RSI: 000055fda857d420 RDI: 0000000000000007 [ 1358.600557] RBP: 00007ffeb2343880 R08: 0000000000000000 R09: 0000000000000000 [ 1358.609366] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fccec4e9390 [ 1358.618176] R13: 000055fda857d420 R14: 0000000000000007 R15: 00007fccec47d120 [ 1358.626989] [ 1358.630261] INFO: task python3:5434 is blocked on a mutex likely owned by task python3:3453. [ 1358.640535] task:python3 state:D stack:25672 pid:3453 tgid:3453 ppid:3452 task_flags:0x400100 flags:0x00080802 [ 1358.653740] Call Trace: [ 1358.657279] [ 1358.660430] __schedule+0xcc3/0x19c0 [ 1358.665235] ? io_schedule_timeout+0x140/0x140 [ 1358.671021] ? find_held_lock+0x2b/0x80 [ 1358.676122] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.682303] schedule+0xd8/0x260 [ 1358.686722] schedule_timeout+0x13f/0x290 [ 1358.692010] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.698378] ? trace_contention_end+0xb3/0x1a0 [ 1358.704160] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.709853] ? ww_mutex_lock+0x160/0x160 [ 1358.715054] msleep+0x1b/0x30 [ 1358.719182] napi_set_threaded+0x3a5/0x440 [ 1358.724577] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.730071] netif_set_threaded+0x257/0x400 [ 1358.735566] threaded_store+0x1b8/0x220 [ 1358.740669] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.745958] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.751245] ? sysfs_kf_write+0x65/0x160 [ 1358.756443] ? sysfs_file_ops+0x140/0x140 [ 1358.761740] kernfs_fop_write_iter+0x335/0x510 [ 1358.767514] ? kernfs_ops+0xf0/0xf0 [ 1358.772226] new_sync_write+0x333/0x750 [ 1358.777330] ? __lock_acquire+0x508/0xc10 [ 1358.782619] ? new_sync_read+0x740/0x740 [ 1358.787819] ? lock_acquire.part.0+0xbc/0x260 [ 1358.793507] ? ksys_write+0x116/0x250 [ 1358.798418] vfs_write+0x657/0x9b0 [ 1358.803036] ksys_write+0x116/0x250 [ 1358.807746] ? __ia32_sys_read+0xc0/0xc0 [ 1358.812946] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.818525] ? rcu_is_watching+0x15/0xd0 [ 1358.823725] do_syscall_64+0xf3/0x650 [ 1358.828621] ? trace_hardirqs_off+0xd/0x30 [ 1358.834014] ? exc_page_fault+0xda/0xf0 [ 1358.839117] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.845583] RIP: 0033:0x7fa552fb264e [ 1358.850389] RSP: 002b:00007fffb256eef0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.859687] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa552fb264e [ 1358.868494] RDX: 0000000000000001 RSI: 000055aa82f90420 RDI: 0000000000000007 [ 1358.877305] RBP: 00007fffb256ef00 R08: 0000000000000000 R09: 0000000000000000 [ 1358.886112] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa552e2b390 [ 1358.894921] R13: 000055aa82f90420 R14: 0000000000000007 R15: 00007fa552dc1120 [ 1358.903736] [ 1358.907008] \x0aShowing all locks held in the system: [ 1358.915546] 1 lock held by khungtaskd/185: [ 1358.920942] #0: ffffffffbaefdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.932407] 2 locks held by systemd-journal/805: [ 1358.938384] #0: ff1100180e2b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1358.948276] #1: ffffffffbb102ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1358.961100] 4 locks held by python3/3453: [ 1358.966393] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.976283] #1: ff110001400a2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.987244] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.998402] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.008499] 4 locks held by python3/5434: [ 1359.013797] #0: ff1100012f7f93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.023688] #1: ff1100016cfd5c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.034649] #2: ff11000124692b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.045807] #3: ff110001246d1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.055901] [ 1359.058368] =============================================\x0a