[ 780.431645] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 780.434123] nipa-hw-worker: [5/43] Retrying drivers/net:napi_threaded.py [ 780.586593] kselftest: Running tests in drivers/net [ 788.277202] block nvme0n1: No UUID available providing old NGUID SUBSYSTEM=block DEVICE=b259:0 [ 989.715246] INFO: task python3:5752 blocked for more than 122 seconds. [ 989.723448] Not tainted 7.1.0-rc1-rvdw-gd4ba57e96f24 #1 [ 989.730988] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.740579] task:python3 state:D stack:25672 pid:5752 tgid:5752 ppid:5751 task_flags:0x400100 flags:0x00080000 [ 989.753785] Call Trace: [ 989.757323] [ 989.760475] __schedule+0xcba/0x1940 [ 989.765293] ? io_schedule_timeout+0x140/0x140 [ 989.771083] ? find_held_lock+0x2b/0x80 [ 989.776188] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.782082] schedule+0xd4/0x260 [ 989.786502] schedule_preempt_disabled+0x18/0x30 [ 989.792482] __mutex_lock+0x6a3/0x2000 [ 989.797487] ? threaded_store+0x132/0x220 [ 989.802787] ? ww_mutex_lock+0x160/0x160 [ 989.807988] ? selinux_inode_free_security+0x330/0x330 [ 989.814558] ? cap_capable+0x1d6/0x3e0 [ 989.819565] ? threaded_store+0x132/0x220 [ 989.824860] threaded_store+0x132/0x220 [ 989.829963] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.835264] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.840565] ? sysfs_kf_write+0x65/0x160 [ 989.845763] ? sysfs_file_ops+0x140/0x140 [ 989.851058] kernfs_fop_write_iter+0x335/0x510 [ 989.856843] ? kernfs_ops+0xf0/0xf0 [ 989.861555] new_sync_write+0x333/0x750 [ 989.866661] ? __lock_acquire+0x508/0xc10 [ 989.871961] ? new_sync_read+0x740/0x740 [ 989.877164] ? lock_acquire.part.0+0xbc/0x260 [ 989.882853] ? ksys_write+0x116/0x250 [ 989.887763] vfs_write+0x657/0x9b0 [ 989.892377] ksys_write+0x116/0x250 [ 989.897089] ? __ia32_sys_read+0xc0/0xc0 [ 989.902288] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.907875] ? rcu_is_watching+0x15/0xd0 [ 989.913082] do_syscall_64+0xf3/0x650 [ 989.917988] ? trace_hardirqs_off+0xd/0x30 [ 989.923385] ? exc_page_fault+0xda/0xf0 [ 989.928486] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.934954] RIP: 0033:0x7f807fb1022e [ 989.939764] RSP: 002b:00007ffe51a143d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.949065] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f807fb1022e [ 989.957873] RDX: 0000000000000001 RSI: 00005573765b9a10 RDI: 0000000000000007 [ 989.966674] RBP: 00007ffe51a143e0 R08: 0000000000000000 R09: 0000000000000000 [ 989.975482] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f807f988390 [ 989.984289] R13: 00005573765b9a10 R14: 0000000000000007 R15: 0000000000000003 [ 989.993095] [ 989.996386] INFO: task python3:5752 is blocked on a mutex likely owned by task python3:3660. [ 990.006659] task:python3 state:D stack:25672 pid:3660 tgid:3660 ppid:3659 task_flags:0x400100 flags:0x00080802 [ 990.019864] Call Trace: [ 990.023402] [ 990.026551] __schedule+0xcba/0x1940 [ 990.031361] ? io_schedule_timeout+0x140/0x140 [ 990.037143] ? find_held_lock+0x2b/0x80 [ 990.042245] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.048414] schedule+0xd4/0x260 [ 990.052835] schedule_timeout+0x13f/0x290 [ 990.058123] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.064490] ? trace_contention_end+0xb3/0x1a0 [ 990.070275] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.075955] ? ww_mutex_lock+0x160/0x160 [ 990.081166] msleep+0x1b/0x30 [ 990.085290] napi_set_threaded+0x3a5/0x440 [ 990.090687] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.096174] netif_set_threaded+0x257/0x400 [ 990.101666] threaded_store+0x1b8/0x220 [ 990.106760] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.112050] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.117346] ? sysfs_kf_write+0x65/0x160 [ 990.122544] ? sysfs_file_ops+0x140/0x140 [ 990.127838] kernfs_fop_write_iter+0x335/0x510 [ 990.133623] ? kernfs_ops+0xf0/0xf0 [ 990.138327] new_sync_write+0x333/0x750 [ 990.143420] ? __lock_acquire+0x508/0xc10 [ 990.148718] ? new_sync_read+0x740/0x740 [ 990.153916] ? lock_acquire.part.0+0xbc/0x260 [ 990.159602] ? ksys_write+0x116/0x250 [ 990.164513] vfs_write+0x657/0x9b0 [ 990.169128] ksys_write+0x116/0x250 [ 990.173840] ? __ia32_sys_read+0xc0/0xc0 [ 990.179040] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.184629] ? rcu_is_watching+0x15/0xd0 [ 990.189830] do_syscall_64+0xf3/0x650 [ 990.194737] ? trace_hardirqs_off+0xd/0x30 [ 990.200129] ? exc_page_fault+0xda/0xf0 [ 990.205230] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.211698] RIP: 0033:0x7fd31616d22e [ 990.216505] RSP: 002b:00007ffd0e210c30 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.225800] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd31616d22e [ 990.234610] RDX: 0000000000000001 RSI: 000055c1b0ccaa10 RDI: 0000000000000007 [ 990.243417] RBP: 00007ffd0e210c40 R08: 0000000000000000 R09: 0000000000000000 [ 990.252225] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd315fe5390 [ 990.261033] R13: 000055c1b0ccaa10 R14: 0000000000000007 R15: 0000000000000003 [ 990.269837] [ 990.273100] \x0aShowing all locks held in the system: [ 990.281640] 1 lock held by khungtaskd/184: [ 990.287034] #0: ffffffffa9afdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.298521] 4 locks held by python3/3660: [ 990.303816] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.313705] #1: ff1100015ec93e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.324666] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.335822] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.345914] 4 locks held by python3/5752: [ 990.351208] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.361097] #1: ff11000157b59a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.372049] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.383207] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.393298] [ 990.395765] =============================================\x0a [ 1112.596403] INFO: task python3:5752 blocked for more than 245 seconds. [ 1112.604557] Not tainted 7.1.0-rc1-rvdw-gd4ba57e96f24 #1 [ 1112.611809] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.621399] task:python3 state:D stack:25672 pid:5752 tgid:5752 ppid:5751 task_flags:0x400100 flags:0x00080000 [ 1112.634607] Call Trace: [ 1112.638148] [ 1112.641301] __schedule+0xcba/0x1940 [ 1112.646121] ? io_schedule_timeout+0x140/0x140 [ 1112.651908] ? find_held_lock+0x2b/0x80 [ 1112.657011] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.662898] schedule+0xd4/0x260 [ 1112.667317] schedule_preempt_disabled+0x18/0x30 [ 1112.673298] __mutex_lock+0x6a3/0x2000 [ 1112.678304] ? threaded_store+0x132/0x220 [ 1112.683605] ? ww_mutex_lock+0x160/0x160 [ 1112.688806] ? selinux_inode_free_security+0x330/0x330 [ 1112.695375] ? cap_capable+0x1d6/0x3e0 [ 1112.700386] ? threaded_store+0x132/0x220 [ 1112.705683] threaded_store+0x132/0x220 [ 1112.710780] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.716075] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.721373] ? sysfs_kf_write+0x65/0x160 [ 1112.726576] ? sysfs_file_ops+0x140/0x140 [ 1112.731873] kernfs_fop_write_iter+0x335/0x510 [ 1112.737648] ? kernfs_ops+0xf0/0xf0 [ 1112.742359] new_sync_write+0x333/0x750 [ 1112.747462] ? __lock_acquire+0x508/0xc10 [ 1112.752760] ? new_sync_read+0x740/0x740 [ 1112.757954] ? lock_acquire.part.0+0xbc/0x260 [ 1112.763642] ? ksys_write+0x116/0x250 [ 1112.768552] vfs_write+0x657/0x9b0 [ 1112.773167] ksys_write+0x116/0x250 [ 1112.777880] ? __ia32_sys_read+0xc0/0xc0 [ 1112.783149] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.788739] ? rcu_is_watching+0x15/0xd0 [ 1112.793941] do_syscall_64+0xf3/0x650 [ 1112.798849] ? trace_hardirqs_off+0xd/0x30 [ 1112.804244] ? exc_page_fault+0xda/0xf0 [ 1112.809348] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.815817] RIP: 0033:0x7f807fb1022e [ 1112.820627] RSP: 002b:00007ffe51a143d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.829916] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f807fb1022e [ 1112.838716] RDX: 0000000000000001 RSI: 00005573765b9a10 RDI: 0000000000000007 [ 1112.847524] RBP: 00007ffe51a143e0 R08: 0000000000000000 R09: 0000000000000000 [ 1112.856327] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f807f988390 [ 1112.865137] R13: 00005573765b9a10 R14: 0000000000000007 R15: 0000000000000003 [ 1112.873951] [ 1112.877271] INFO: task python3:5752 is blocked on a mutex likely owned by task python3:3660. [ 1112.887540] task:python3 state:D stack:25672 pid:3660 tgid:3660 ppid:3659 task_flags:0x400100 flags:0x00080802 [ 1112.900744] Call Trace: [ 1112.904281] [ 1112.907432] __schedule+0xcba/0x1940 [ 1112.912238] ? io_schedule_timeout+0x140/0x140 [ 1112.918024] ? find_held_lock+0x2b/0x80 [ 1112.923121] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.929299] schedule+0xd4/0x260 [ 1112.933721] schedule_timeout+0x13f/0x290 [ 1112.939019] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.945390] ? trace_contention_end+0xb3/0x1a0 [ 1112.951174] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.956864] ? ww_mutex_lock+0x160/0x160 [ 1112.962065] msleep+0x1b/0x30 [ 1112.966191] napi_set_threaded+0x3a5/0x440 [ 1112.971589] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.977084] netif_set_threaded+0x257/0x400 [ 1112.982578] threaded_store+0x1b8/0x220 [ 1112.987670] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.992966] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.998261] ? sysfs_kf_write+0x65/0x160 [ 1113.003460] ? sysfs_file_ops+0x140/0x140 [ 1113.008759] kernfs_fop_write_iter+0x335/0x510 [ 1113.014545] ? kernfs_ops+0xf0/0xf0 [ 1113.019256] new_sync_write+0x333/0x750 [ 1113.024358] ? __lock_acquire+0x508/0xc10 [ 1113.029657] ? new_sync_read+0x740/0x740 [ 1113.034858] ? lock_acquire.part.0+0xbc/0x260 [ 1113.040544] ? ksys_write+0x116/0x250 [ 1113.045455] vfs_write+0x657/0x9b0 [ 1113.050069] ksys_write+0x116/0x250 [ 1113.054780] ? __ia32_sys_read+0xc0/0xc0 [ 1113.059982] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.065569] ? rcu_is_watching+0x15/0xd0 [ 1113.070771] do_syscall_64+0xf3/0x650 [ 1113.075680] ? trace_hardirqs_off+0xd/0x30 [ 1113.081073] ? exc_page_fault+0xda/0xf0 [ 1113.086177] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.092645] RIP: 0033:0x7fd31616d22e [ 1113.097455] RSP: 002b:00007ffd0e210c30 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.106751] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd31616d22e [ 1113.115552] RDX: 0000000000000001 RSI: 000055c1b0ccaa10 RDI: 0000000000000007 [ 1113.124362] RBP: 00007ffd0e210c40 R08: 0000000000000000 R09: 0000000000000000 [ 1113.133175] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd315fe5390 [ 1113.141982] R13: 000055c1b0ccaa10 R14: 0000000000000007 R15: 0000000000000003 [ 1113.150795] [ 1113.154063] \x0aShowing all locks held in the system: [ 1113.162606] 1 lock held by khungtaskd/184: [ 1113.168000] #0: ffffffffa9afdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.179475] 2 locks held by systemd-journal/805: [ 1113.185535] #0: ff1100180dcb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1113.195418] #1: ffffffffa9d030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.208250] 4 locks held by python3/3660: [ 1113.213544] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.223495] #1: ff1100015ec93e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.234458] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.245615] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.255704] 4 locks held by python3/5752: [ 1113.261004] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.270891] #1: ff11000157b59a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.281853] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.293072] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.303175] [ 1113.305651] =============================================\x0a [ 1235.478535] INFO: task python3:5752 blocked for more than 368 seconds. [ 1235.486685] Not tainted 7.1.0-rc1-rvdw-gd4ba57e96f24 #1 [ 1235.493934] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.503523] task:python3 state:D stack:25672 pid:5752 tgid:5752 ppid:5751 task_flags:0x400100 flags:0x00080002 [ 1235.516727] Call Trace: [ 1235.520266] [ 1235.523414] __schedule+0xcba/0x1940 [ 1235.528230] ? io_schedule_timeout+0x140/0x140 [ 1235.534013] ? find_held_lock+0x2b/0x80 [ 1235.539115] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.545000] schedule+0xd4/0x260 [ 1235.549416] schedule_preempt_disabled+0x18/0x30 [ 1235.555394] __mutex_lock+0x6a3/0x2000 [ 1235.560396] ? threaded_store+0x132/0x220 [ 1235.565695] ? ww_mutex_lock+0x160/0x160 [ 1235.570886] ? selinux_inode_free_security+0x330/0x330 [ 1235.577456] ? cap_capable+0x1d6/0x3e0 [ 1235.582462] ? threaded_store+0x132/0x220 [ 1235.587757] threaded_store+0x132/0x220 [ 1235.592858] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.598155] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.603456] ? sysfs_kf_write+0x65/0x160 [ 1235.608656] ? sysfs_file_ops+0x140/0x140 [ 1235.613949] kernfs_fop_write_iter+0x335/0x510 [ 1235.619732] ? kernfs_ops+0xf0/0xf0 [ 1235.624444] new_sync_write+0x333/0x750 [ 1235.629545] ? __lock_acquire+0x508/0xc10 [ 1235.634842] ? new_sync_read+0x740/0x740 [ 1235.640044] ? lock_acquire.part.0+0xbc/0x260 [ 1235.645731] ? ksys_write+0x116/0x250 [ 1235.650641] vfs_write+0x657/0x9b0 [ 1235.655254] ksys_write+0x116/0x250 [ 1235.659962] ? __ia32_sys_read+0xc0/0xc0 [ 1235.665160] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.670746] ? rcu_is_watching+0x15/0xd0 [ 1235.675948] do_syscall_64+0xf3/0x650 [ 1235.680853] ? trace_hardirqs_off+0xd/0x30 [ 1235.686244] ? exc_page_fault+0xda/0xf0 [ 1235.691344] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.697810] RIP: 0033:0x7f807fb1022e [ 1235.702617] RSP: 002b:00007ffe51a143d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.711912] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f807fb1022e [ 1235.720717] RDX: 0000000000000001 RSI: 00005573765b9a10 RDI: 0000000000000007 [ 1235.729524] RBP: 00007ffe51a143e0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.738330] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f807f988390 [ 1235.747138] R13: 00005573765b9a10 R14: 0000000000000007 R15: 0000000000000003 [ 1235.755940] [ 1235.759212] INFO: task python3:5752 is blocked on a mutex likely owned by task python3:3660. [ 1235.769481] task:python3 state:D stack:25672 pid:3660 tgid:3660 ppid:3659 task_flags:0x400100 flags:0x00080802 [ 1235.782685] Call Trace: [ 1235.786224] [ 1235.789372] __schedule+0xcba/0x1940 [ 1235.794187] ? io_schedule_timeout+0x140/0x140 [ 1235.799970] ? find_held_lock+0x2b/0x80 [ 1235.805072] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.811252] schedule+0xd4/0x260 [ 1235.815667] schedule_timeout+0x13f/0x290 [ 1235.820961] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.827326] ? trace_contention_end+0xb3/0x1a0 [ 1235.833109] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.838799] ? ww_mutex_lock+0x160/0x160 [ 1235.843996] msleep+0x1b/0x30 [ 1235.848119] napi_set_threaded+0x3a5/0x440 [ 1235.853512] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.859007] netif_set_threaded+0x257/0x400 [ 1235.864499] threaded_store+0x1b8/0x220 [ 1235.869596] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.874893] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.880189] ? sysfs_kf_write+0x65/0x160 [ 1235.885387] ? sysfs_file_ops+0x140/0x140 [ 1235.890681] kernfs_fop_write_iter+0x335/0x510 [ 1235.896467] ? kernfs_ops+0xf0/0xf0 [ 1235.901175] new_sync_write+0x333/0x750 [ 1235.906274] ? __lock_acquire+0x508/0xc10 [ 1235.911571] ? new_sync_read+0x740/0x740 [ 1235.916771] ? lock_acquire.part.0+0xbc/0x260 [ 1235.922446] ? ksys_write+0x116/0x250 [ 1235.927353] vfs_write+0x657/0x9b0 [ 1235.931968] ksys_write+0x116/0x250 [ 1235.936670] ? __ia32_sys_read+0xc0/0xc0 [ 1235.941871] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.947451] ? rcu_is_watching+0x15/0xd0 [ 1235.952648] do_syscall_64+0xf3/0x650 [ 1235.957551] ? trace_hardirqs_off+0xd/0x30 [ 1235.962945] ? exc_page_fault+0xda/0xf0 [ 1235.968046] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.974509] RIP: 0033:0x7fd31616d22e [ 1235.979316] RSP: 002b:00007ffd0e210c30 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.988612] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd31616d22e [ 1235.997417] RDX: 0000000000000001 RSI: 000055c1b0ccaa10 RDI: 0000000000000007 [ 1236.006222] RBP: 00007ffd0e210c40 R08: 0000000000000000 R09: 0000000000000000 [ 1236.015027] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd315fe5390 [ 1236.023835] R13: 000055c1b0ccaa10 R14: 0000000000000007 R15: 0000000000000003 [ 1236.032649] [ 1236.035915] \x0aShowing all locks held in the system: [ 1236.044446] 1 lock held by khungtaskd/184: [ 1236.049837] #0: ffffffffa9afdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.061309] 3 locks held by systemd-journal/805: [ 1236.067287] #0: ff1100180e7372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1236.077173] #1: ffffffffa9d030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: lockref_get_not_dead+0xe/0x60 [ 1236.089988] #2: ff110001a2d82860 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1236.101543] 4 locks held by python3/3660: [ 1236.106837] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.116725] #1: ff1100015ec93e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.127687] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.138842] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.148938] 4 locks held by python3/5752: [ 1236.154232] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.164120] #1: ff11000157b59a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.175079] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.186234] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.196323] [ 1236.198789] =============================================\x0a [ 1358.358693] INFO: task python3:5752 blocked for more than 491 seconds. [ 1358.366846] Not tainted 7.1.0-rc1-rvdw-gd4ba57e96f24 #1 [ 1358.374098] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.383687] task:python3 state:D stack:25672 pid:5752 tgid:5752 ppid:5751 task_flags:0x400100 flags:0x00080002 [ 1358.396894] Call Trace: [ 1358.400436] [ 1358.403588] __schedule+0xcba/0x1940 [ 1358.408408] ? io_schedule_timeout+0x140/0x140 [ 1358.414194] ? find_held_lock+0x2b/0x80 [ 1358.419303] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.425190] schedule+0xd4/0x260 [ 1358.429612] schedule_preempt_disabled+0x18/0x30 [ 1358.435593] __mutex_lock+0x6a3/0x2000 [ 1358.440597] ? threaded_store+0x132/0x220 [ 1358.445898] ? ww_mutex_lock+0x160/0x160 [ 1358.451100] ? selinux_inode_free_security+0x330/0x330 [ 1358.457671] ? cap_capable+0x1d6/0x3e0 [ 1358.462681] ? threaded_store+0x132/0x220 [ 1358.467980] threaded_store+0x132/0x220 [ 1358.473087] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.478386] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.483683] ? sysfs_kf_write+0x65/0x160 [ 1358.488886] ? sysfs_file_ops+0x140/0x140 [ 1358.494182] kernfs_fop_write_iter+0x335/0x510 [ 1358.499969] ? kernfs_ops+0xf0/0xf0 [ 1358.504679] new_sync_write+0x333/0x750 [ 1358.509775] ? __lock_acquire+0x508/0xc10 [ 1358.515073] ? new_sync_read+0x740/0x740 [ 1358.520274] ? lock_acquire.part.0+0xbc/0x260 [ 1358.525963] ? ksys_write+0x116/0x250 [ 1358.530874] vfs_write+0x657/0x9b0 [ 1358.535488] ksys_write+0x116/0x250 [ 1358.540200] ? __ia32_sys_read+0xc0/0xc0 [ 1358.545403] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.550994] ? rcu_is_watching+0x15/0xd0 [ 1358.556196] do_syscall_64+0xf3/0x650 [ 1358.561104] ? trace_hardirqs_off+0xd/0x30 [ 1358.566501] ? exc_page_fault+0xda/0xf0 [ 1358.571605] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.578071] RIP: 0033:0x7f807fb1022e [ 1358.582881] RSP: 002b:00007ffe51a143d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.592172] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f807fb1022e [ 1358.600980] RDX: 0000000000000001 RSI: 00005573765b9a10 RDI: 0000000000000007 [ 1358.609791] RBP: 00007ffe51a143e0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.618598] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f807f988390 [ 1358.627402] R13: 00005573765b9a10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.636218] [ 1358.639490] INFO: task python3:5752 is blocked on a mutex likely owned by task python3:3660. [ 1358.649765] task:python3 state:D stack:25672 pid:3660 tgid:3660 ppid:3659 task_flags:0x400100 flags:0x00080802 [ 1358.662968] Call Trace: [ 1358.666580] [ 1358.669723] __schedule+0xcba/0x1940 [ 1358.674539] ? io_schedule_timeout+0x140/0x140 [ 1358.680323] ? find_held_lock+0x2b/0x80 [ 1358.685427] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.691607] schedule+0xd4/0x260 [ 1358.696025] schedule_timeout+0x13f/0x290 [ 1358.701324] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.707691] ? trace_contention_end+0xb3/0x1a0 [ 1358.713477] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.719229] ? ww_mutex_lock+0x160/0x160 [ 1358.724430] msleep+0x1b/0x30 [ 1358.728558] napi_set_threaded+0x3a5/0x440 [ 1358.733957] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.739454] netif_set_threaded+0x257/0x400 [ 1358.744946] threaded_store+0x1b8/0x220 [ 1358.750047] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.755410] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.760708] ? sysfs_kf_write+0x65/0x160 [ 1358.765909] ? sysfs_file_ops+0x140/0x140 [ 1358.771205] kernfs_fop_write_iter+0x335/0x510 [ 1358.776991] ? kernfs_ops+0xf0/0xf0 [ 1358.781704] new_sync_write+0x333/0x750 [ 1358.786808] ? __lock_acquire+0x508/0xc10 [ 1358.792104] ? new_sync_read+0x740/0x740 [ 1358.797305] ? lock_acquire.part.0+0xbc/0x260 [ 1358.802993] ? ksys_write+0x116/0x250 [ 1358.807903] vfs_write+0x657/0x9b0 [ 1358.812518] ksys_write+0x116/0x250 [ 1358.817234] ? __ia32_sys_read+0xc0/0xc0 [ 1358.822436] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.828025] ? rcu_is_watching+0x15/0xd0 [ 1358.833225] do_syscall_64+0xf3/0x650 [ 1358.838122] ? trace_hardirqs_off+0xd/0x30 [ 1358.843515] ? exc_page_fault+0xda/0xf0 [ 1358.848617] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.855084] RIP: 0033:0x7fd31616d22e [ 1358.859893] RSP: 002b:00007ffd0e210c30 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.869190] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd31616d22e [ 1358.878001] RDX: 0000000000000001 RSI: 000055c1b0ccaa10 RDI: 0000000000000007 [ 1358.886811] RBP: 00007ffd0e210c40 R08: 0000000000000000 R09: 0000000000000000 [ 1358.895620] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd315fe5390 [ 1358.904426] R13: 000055c1b0ccaa10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.913241] [ 1358.916514] \x0aShowing all locks held in the system: [ 1358.925056] 1 lock held by khungtaskd/184: [ 1358.930440] #0: ffffffffa9afdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.941922] 4 locks held by python3/3660: [ 1358.947222] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.957114] #1: ff1100015ec93e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.968076] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.979226] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.989322] 4 locks held by python3/5752: [ 1358.994616] #0: ff1100012e7e93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.004506] #1: ff11000157b59a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.015471] #2: ff11000121084b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.026627] #3: ff11000121071d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.036719] 1 lock held by systemd/7642: [ 1359.041919] [ 1359.044381] =============================================\x0a