[ 761.354907] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 761.357478] nipa-hw-worker: [4/37] Retrying drivers/net:napi_threaded.py
[ 761.520290] kselftest: Running tests in drivers/net
[ 973.365793] kworker/u129:14 (313) used greatest stack depth: 21336 bytes left
[ 989.749952] INFO: task python3:5071 blocked for more than 122 seconds.
[ 989.758176] Not tainted 7.0.0-rc5-rtof-gf6a13172a0ea #1
[ 989.765428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.775019] task:python3 state:D stack:25312 pid:5071 tgid:5071 ppid:5070 task_flags:0x400100 flags:0x00080000
[ 989.788228] Call Trace:
[ 989.791767]
[ 989.794921] __schedule+0xad9/0x1490
[ 989.799738] ? io_schedule_timeout+0x130/0x130
[ 989.805524] ? find_held_lock+0x2b/0x80
[ 989.810628] ? __lock_release.isra.0+0x59/0x170
[ 989.816515] schedule+0xd4/0x260
[ 989.820936] schedule_preempt_disabled+0x18/0x30
[ 989.826915] __mutex_lock+0xd76/0x1f20
[ 989.831921] ? threaded_store+0x132/0x220
[ 989.837222] ? ww_mutex_lock+0x160/0x160
[ 989.842424] ? selinux_inode_free_security+0x310/0x310
[ 989.848995] ? cap_capable+0x1ed/0x3f0
[ 989.854002] ? sysfs_file_ops+0x140/0x140
[ 989.859300] ? threaded_store+0x132/0x220
[ 989.864596] threaded_store+0x132/0x220
[ 989.869700] ? tx_maxrate_store+0x4e0/0x4e0
[ 989.875195] ? sysfs_file_ops+0x140/0x140
[ 989.880491] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.885792] ? sysfs_kf_write+0x65/0x160
[ 989.890995] kernfs_fop_write_iter+0x335/0x510
[ 989.896785] ? kernfs_ops+0xf0/0xf0
[ 989.901498] new_sync_write+0x308/0x6f0
[ 989.906604] ? new_sync_read+0x6f0/0x6f0
[ 989.911805] ? lock_acquire.part.0+0xbc/0x260
[ 989.917493] ? ksys_write+0xff/0x200
[ 989.922307] vfs_write+0x62a/0x980
[ 989.926925] ksys_write+0xff/0x200
[ 989.931539] ? __ia32_sys_read+0xc0/0xc0
[ 989.936728] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.942317] ? rcu_is_watching+0x15/0xd0
[ 989.947520] do_syscall_64+0xf3/0x5e0
[ 989.952426] ? exc_page_fault+0x9e/0xb0
[ 989.957526] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.963997] RIP: 0033:0x7f50e00e9c5e
[ 989.968805] RSP: 002b:00007ffec4fb6770 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.978112] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f50e00e9c5e
[ 989.986915] RDX: 0000000000000001 RSI: 000055d02cf04110 RDI: 0000000000000007
[ 989.995725] RBP: 00007ffec4fb6780 R08: 0000000000000000 R09: 0000000000000000
[ 990.004534] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f50dff84e80
[ 990.013344] R13: 000055d02cf04110 R14: 0000000000000007 R15: 00007f50e07eab98
[ 990.022159]
[ 990.025426] INFO: task python3:5071 is blocked on a mutex likely owned by task python3:3121.
[ 990.035700] task:python3 state:D stack:25312 pid:3121 tgid:3121 ppid:3120 task_flags:0x400100 flags:0x00080802
[ 990.048904] Call Trace:
[ 990.052446]
[ 990.055594] __schedule+0xad9/0x1490
[ 990.060407] ? io_schedule_timeout+0x130/0x130
[ 990.066192] ? find_held_lock+0x2b/0x80
[ 990.071285] ? __try_to_del_timer_sync+0x154/0x1f0
[ 990.077456] schedule+0xd4/0x260
[ 990.081876] schedule_timeout+0x103/0x260
[ 990.087174] ? rcu_is_watching+0x15/0xd0
[ 990.092373] ? hrtimer_nanosleep_restart+0x180/0x180
[ 990.098736] ? hrtimers_cpu_dying+0x250/0x250
[ 990.104427] ? ww_mutex_lock+0x160/0x160
[ 990.109629] msleep+0x1b/0x30
[ 990.113758] napi_set_threaded+0x376/0x410
[ 990.119152] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.124647] netif_set_threaded+0x254/0x400
[ 990.130141] ? sysfs_file_ops+0x140/0x140
[ 990.135439] threaded_store+0x1bb/0x220
[ 990.140543] ? tx_maxrate_store+0x4e0/0x4e0
[ 990.146036] ? sysfs_file_ops+0x140/0x140
[ 990.151334] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.156631] ? sysfs_kf_write+0x65/0x160
[ 990.161830] kernfs_fop_write_iter+0x335/0x510
[ 990.167616] ? kernfs_ops+0xf0/0xf0
[ 990.172330] new_sync_write+0x308/0x6f0
[ 990.177433] ? new_sync_read+0x6f0/0x6f0
[ 990.182637] ? lock_acquire.part.0+0xbc/0x260
[ 990.188323] ? ksys_write+0xff/0x200
[ 990.193135] vfs_write+0x62a/0x980
[ 990.197751] ksys_write+0xff/0x200
[ 990.202366] ? __ia32_sys_read+0xc0/0xc0
[ 990.207556] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.213148] ? rcu_is_watching+0x15/0xd0
[ 990.218349] do_syscall_64+0xf3/0x5e0
[ 990.223254] ? exc_page_fault+0x9e/0xb0
[ 990.228358] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.234824] RIP: 0033:0x7fc46a0e3c5e
[ 990.239633] RSP: 002b:00007ffec8495ab0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.248930] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc46a0e3c5e
[ 990.257731] RDX: 0000000000000001 RSI: 0000559397594110 RDI: 0000000000000007
[ 990.266537] RBP: 00007ffec8495ac0 R08: 0000000000000000 R09: 0000000000000000
[ 990.275347] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc469f7ee80
[ 990.284155] R13: 0000559397594110 R14: 0000000000000007 R15: 00007fc469f0e480
[ 990.292971]
[ 990.296231] \x0aShowing all locks held in the system:
[ 990.304769] 1 lock held by khungtaskd/185:
[ 990.310162] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.321644] 4 locks held by python3/3121:
[ 990.326941] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 990.336734] #1: ff1100013aeca688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 990.347699] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.358857] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.368953] 4 locks held by python3/5071:
[ 990.374247] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 990.384038] #1: ff110001aa7ddc88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 990.394999] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.406160] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.416250]
[ 990.418709] =============================================\x0a
[ 1112.631297] INFO: task python3:5071 blocked for more than 245 seconds.
[ 1112.639451] Not tainted 7.0.0-rc5-rtof-gf6a13172a0ea #1
[ 1112.646701] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.656289] task:python3 state:D stack:25312 pid:5071 tgid:5071 ppid:5070 task_flags:0x400100 flags:0x00080000
[ 1112.669496] Call Trace:
[ 1112.673037]
[ 1112.676187] __schedule+0xad9/0x1490
[ 1112.681004] ? io_schedule_timeout+0x130/0x130
[ 1112.686788] ? find_held_lock+0x2b/0x80
[ 1112.691892] ? __lock_release.isra.0+0x59/0x170
[ 1112.697766] schedule+0xd4/0x260
[ 1112.702182] schedule_preempt_disabled+0x18/0x30
[ 1112.708162] __mutex_lock+0xd76/0x1f20
[ 1112.713166] ? threaded_store+0x132/0x220
[ 1112.718468] ? ww_mutex_lock+0x160/0x160
[ 1112.723665] ? selinux_inode_free_security+0x310/0x310
[ 1112.730232] ? cap_capable+0x1ed/0x3f0
[ 1112.735229] ? sysfs_file_ops+0x140/0x140
[ 1112.740528] ? threaded_store+0x132/0x220
[ 1112.745823] threaded_store+0x132/0x220
[ 1112.750924] ? tx_maxrate_store+0x4e0/0x4e0
[ 1112.756416] ? sysfs_file_ops+0x140/0x140
[ 1112.761709] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.767006] ? sysfs_kf_write+0x65/0x160
[ 1112.772198] kernfs_fop_write_iter+0x335/0x510
[ 1112.777984] ? kernfs_ops+0xf0/0xf0
[ 1112.782696] new_sync_write+0x308/0x6f0
[ 1112.787790] ? new_sync_read+0x6f0/0x6f0
[ 1112.793011] ? lock_acquire.part.0+0xbc/0x260
[ 1112.798727] ? ksys_write+0xff/0x200
[ 1112.803543] vfs_write+0x62a/0x980
[ 1112.808161] ksys_write+0xff/0x200
[ 1112.812778] ? __ia32_sys_read+0xc0/0xc0
[ 1112.817978] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.823569] ? rcu_is_watching+0x15/0xd0
[ 1112.828770] do_syscall_64+0xf3/0x5e0
[ 1112.833679] ? exc_page_fault+0x9e/0xb0
[ 1112.838781] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.845250] RIP: 0033:0x7f50e00e9c5e
[ 1112.850065] RSP: 002b:00007ffec4fb6770 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.859365] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f50e00e9c5e
[ 1112.868176] RDX: 0000000000000001 RSI: 000055d02cf04110 RDI: 0000000000000007
[ 1112.876984] RBP: 00007ffec4fb6780 R08: 0000000000000000 R09: 0000000000000000
[ 1112.885786] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f50dff84e80
[ 1112.894598] R13: 000055d02cf04110 R14: 0000000000000007 R15: 00007f50e07eab98
[ 1112.903413]
[ 1112.906767] INFO: task python3:5071 is blocked on a mutex likely owned by task python3:3121.
[ 1112.917045] task:python3 state:D stack:25312 pid:3121 tgid:3121 ppid:3120 task_flags:0x400100 flags:0x00080802
[ 1112.930251] Call Trace:
[ 1112.933793]
[ 1112.936942] __schedule+0xad9/0x1490
[ 1112.941757] ? io_schedule_timeout+0x130/0x130
[ 1112.947542] ? find_held_lock+0x2b/0x80
[ 1112.952645] ? __try_to_del_timer_sync+0x154/0x1f0
[ 1112.958825] schedule+0xd4/0x260
[ 1112.963244] schedule_timeout+0x103/0x260
[ 1112.968531] ? rcu_is_watching+0x15/0xd0
[ 1112.973735] ? hrtimer_nanosleep_restart+0x180/0x180
[ 1112.980111] ? hrtimers_cpu_dying+0x250/0x250
[ 1112.985805] ? ww_mutex_lock+0x160/0x160
[ 1112.991009] msleep+0x1b/0x30
[ 1112.995137] napi_set_threaded+0x376/0x410
[ 1113.000534] ? napi_resume_irqs+0x1a0/0x1a0
[ 1113.006029] netif_set_threaded+0x254/0x400
[ 1113.011525] ? sysfs_file_ops+0x140/0x140
[ 1113.016826] threaded_store+0x1bb/0x220
[ 1113.021932] ? tx_maxrate_store+0x4e0/0x4e0
[ 1113.027427] ? sysfs_file_ops+0x140/0x140
[ 1113.032723] ? sysfs_file_kobj+0xbd/0x1e0
[ 1113.038022] ? sysfs_kf_write+0x65/0x160
[ 1113.043222] kernfs_fop_write_iter+0x335/0x510
[ 1113.049008] ? kernfs_ops+0xf0/0xf0
[ 1113.053720] new_sync_write+0x308/0x6f0
[ 1113.058826] ? new_sync_read+0x6f0/0x6f0
[ 1113.064121] ? lock_acquire.part.0+0xbc/0x260
[ 1113.069806] ? ksys_write+0xff/0x200
[ 1113.074621] vfs_write+0x62a/0x980
[ 1113.079239] ksys_write+0xff/0x200
[ 1113.083855] ? __ia32_sys_read+0xc0/0xc0
[ 1113.089048] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.094637] ? rcu_is_watching+0x15/0xd0
[ 1113.099838] do_syscall_64+0xf3/0x5e0
[ 1113.104746] ? exc_page_fault+0x9e/0xb0
[ 1113.109850] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.116319] RIP: 0033:0x7fc46a0e3c5e
[ 1113.121132] RSP: 002b:00007ffec8495ab0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.130432] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc46a0e3c5e
[ 1113.139242] RDX: 0000000000000001 RSI: 0000559397594110 RDI: 0000000000000007
[ 1113.148051] RBP: 00007ffec8495ac0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.156862] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc469f7ee80
[ 1113.165671] R13: 0000559397594110 R14: 0000000000000007 R15: 00007fc469f0e480
[ 1113.174503]
[ 1113.177768] \x0aShowing all locks held in the system:
[ 1113.186310] 1 lock held by khungtaskd/185:
[ 1113.191705] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.203251] 4 locks held by python3/3121:
[ 1113.208547] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1113.218341] #1: ff1100013aeca688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1113.229303] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.240454] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.250563] 4 locks held by python3/5071:
[ 1113.255849] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1113.265640] #1: ff110001aa7ddc88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1113.276602] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.287764] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.297853] 1 lock held by systemd/6147:
[ 1113.303057]
[ 1113.305527] =============================================\x0a
[ 1235.511572] INFO: task python3:5071 blocked for more than 368 seconds.
[ 1235.519721] Not tainted 7.0.0-rc5-rtof-gf6a13172a0ea #1
[ 1235.526973] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.536565] task:python3 state:D stack:25312 pid:5071 tgid:5071 ppid:5070 task_flags:0x400100 flags:0x00080002
[ 1235.549773] Call Trace:
[ 1235.553314]
[ 1235.556468] __schedule+0xad9/0x1490
[ 1235.561287] ? io_schedule_timeout+0x130/0x130
[ 1235.567073] ? find_held_lock+0x2b/0x80
[ 1235.572179] ? __lock_release.isra.0+0x59/0x170
[ 1235.578069] schedule+0xd4/0x260
[ 1235.582491] schedule_preempt_disabled+0x18/0x30
[ 1235.588486] __mutex_lock+0xd76/0x1f20
[ 1235.593502] ? threaded_store+0x132/0x220
[ 1235.598806] ? ww_mutex_lock+0x160/0x160
[ 1235.604004] ? selinux_inode_free_security+0x310/0x310
[ 1235.610567] ? cap_capable+0x1ed/0x3f0
[ 1235.615573] ? sysfs_file_ops+0x140/0x140
[ 1235.620871] ? threaded_store+0x132/0x220
[ 1235.626168] threaded_store+0x132/0x220
[ 1235.631260] ? tx_maxrate_store+0x4e0/0x4e0
[ 1235.636755] ? sysfs_file_ops+0x140/0x140
[ 1235.642052] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.647347] ? sysfs_kf_write+0x65/0x160
[ 1235.652549] kernfs_fop_write_iter+0x335/0x510
[ 1235.658334] ? kernfs_ops+0xf0/0xf0
[ 1235.663036] new_sync_write+0x308/0x6f0
[ 1235.668143] ? new_sync_read+0x6f0/0x6f0
[ 1235.673342] ? lock_acquire.part.0+0xbc/0x260
[ 1235.679019] ? ksys_write+0xff/0x200
[ 1235.683833] vfs_write+0x62a/0x980
[ 1235.688448] ksys_write+0xff/0x200
[ 1235.693062] ? __ia32_sys_read+0xc0/0xc0
[ 1235.698261] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.703842] ? rcu_is_watching+0x15/0xd0
[ 1235.709046] do_syscall_64+0xf3/0x5e0
[ 1235.713955] ? exc_page_fault+0x9e/0xb0
[ 1235.719048] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.725508] RIP: 0033:0x7f50e00e9c5e
[ 1235.730319] RSP: 002b:00007ffec4fb6770 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.739617] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f50e00e9c5e
[ 1235.748422] RDX: 0000000000000001 RSI: 000055d02cf04110 RDI: 0000000000000007
[ 1235.757230] RBP: 00007ffec4fb6780 R08: 0000000000000000 R09: 0000000000000000
[ 1235.766037] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f50dff84e80
[ 1235.774835] R13: 000055d02cf04110 R14: 0000000000000007 R15: 00007f50e07eab98
[ 1235.783651]
[ 1235.786923] INFO: task python3:5071 is blocked on a mutex likely owned by task python3:3121.
[ 1235.797197] task:python3 state:D stack:25312 pid:3121 tgid:3121 ppid:3120 task_flags:0x400100 flags:0x00080802
[ 1235.810392] Call Trace:
[ 1235.813931]
[ 1235.817082] __schedule+0xad9/0x1490
[ 1235.821893] ? io_schedule_timeout+0x130/0x130
[ 1235.827678] ? find_held_lock+0x2b/0x80
[ 1235.832782] ? __try_to_del_timer_sync+0x154/0x1f0
[ 1235.838959] schedule+0xd4/0x260
[ 1235.843377] schedule_timeout+0x103/0x260
[ 1235.848675] ? rcu_is_watching+0x15/0xd0
[ 1235.853867] ? hrtimer_nanosleep_restart+0x180/0x180
[ 1235.860239] ? hrtimers_cpu_dying+0x250/0x250
[ 1235.865932] ? ww_mutex_lock+0x160/0x160
[ 1235.871132] msleep+0x1b/0x30
[ 1235.875257] napi_set_threaded+0x376/0x410
[ 1235.880654] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.886150] netif_set_threaded+0x254/0x400
[ 1235.891643] ? sysfs_file_ops+0x140/0x140
[ 1235.896937] threaded_store+0x1bb/0x220
[ 1235.902039] ? tx_maxrate_store+0x4e0/0x4e0
[ 1235.907533] ? sysfs_file_ops+0x140/0x140
[ 1235.912830] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.918126] ? sysfs_kf_write+0x65/0x160
[ 1235.923326] kernfs_fop_write_iter+0x335/0x510
[ 1235.929102] ? kernfs_ops+0xf0/0xf0
[ 1235.933814] new_sync_write+0x308/0x6f0
[ 1235.938916] ? new_sync_read+0x6f0/0x6f0
[ 1235.944117] ? lock_acquire.part.0+0xbc/0x260
[ 1235.949803] ? ksys_write+0xff/0x200
[ 1235.954615] vfs_write+0x62a/0x980
[ 1235.959229] ksys_write+0xff/0x200
[ 1235.963841] ? __ia32_sys_read+0xc0/0xc0
[ 1235.969039] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.974629] ? rcu_is_watching+0x15/0xd0
[ 1235.979820] do_syscall_64+0xf3/0x5e0
[ 1235.984719] ? exc_page_fault+0x9e/0xb0
[ 1235.989822] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.996290] RIP: 0033:0x7fc46a0e3c5e
[ 1236.001098] RSP: 002b:00007ffec8495ab0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1236.010393] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc46a0e3c5e
[ 1236.019202] RDX: 0000000000000001 RSI: 0000559397594110 RDI: 0000000000000007
[ 1236.028009] RBP: 00007ffec8495ac0 R08: 0000000000000000 R09: 0000000000000000
[ 1236.036818] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc469f7ee80
[ 1236.045626] R13: 0000559397594110 R14: 0000000000000007 R15: 00007fc469f0e480
[ 1236.054439]
[ 1236.057703] \x0aShowing all locks held in the system:
[ 1236.066240] 1 lock held by khungtaskd/185:
[ 1236.071636] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.083117] 4 locks held by python3/3121:
[ 1236.088412] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1236.098209] #1: ff1100013aeca688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1236.109173] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.120329] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.130431] 4 locks held by python3/5071:
[ 1236.135724] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1236.145517] #1: ff110001aa7ddc88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1236.156482] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.167642] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.177734]
[ 1236.180198] =============================================\x0a
[ 1358.393897] INFO: task python3:5071 blocked for more than 491 seconds.
[ 1358.402044] Not tainted 7.0.0-rc5-rtof-gf6a13172a0ea #1
[ 1358.409293] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.418874] task:python3 state:D stack:25312 pid:5071 tgid:5071 ppid:5070 task_flags:0x400100 flags:0x00080002
[ 1358.432079] Call Trace:
[ 1358.435617]
[ 1358.438766] __schedule+0xad9/0x1490
[ 1358.443581] ? io_schedule_timeout+0x130/0x130
[ 1358.449367] ? find_held_lock+0x2b/0x80
[ 1358.454472] ? __lock_release.isra.0+0x59/0x170
[ 1358.460347] schedule+0xd4/0x260
[ 1358.464767] schedule_preempt_disabled+0x18/0x30
[ 1358.470742] __mutex_lock+0xd76/0x1f20
[ 1358.475749] ? threaded_store+0x132/0x220
[ 1358.481050] ? ww_mutex_lock+0x160/0x160
[ 1358.486249] ? selinux_inode_free_security+0x310/0x310
[ 1358.492808] ? cap_capable+0x1ed/0x3f0
[ 1358.497814] ? sysfs_file_ops+0x140/0x140
[ 1358.503112] ? threaded_store+0x132/0x220
[ 1358.508406] threaded_store+0x132/0x220
[ 1358.513508] ? tx_maxrate_store+0x4e0/0x4e0
[ 1358.518992] ? sysfs_file_ops+0x140/0x140
[ 1358.524292] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.529586] ? sysfs_kf_write+0x65/0x160
[ 1358.534786] kernfs_fop_write_iter+0x335/0x510
[ 1358.540570] ? kernfs_ops+0xf0/0xf0
[ 1358.545274] new_sync_write+0x308/0x6f0
[ 1358.550376] ? new_sync_read+0x6f0/0x6f0
[ 1358.555577] ? lock_acquire.part.0+0xbc/0x260
[ 1358.561262] ? ksys_write+0xff/0x200
[ 1358.566073] vfs_write+0x62a/0x980
[ 1358.570688] ksys_write+0xff/0x200
[ 1358.575301] ? __ia32_sys_read+0xc0/0xc0
[ 1358.580496] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.586082] ? rcu_is_watching+0x15/0xd0
[ 1358.591283] do_syscall_64+0xf3/0x5e0
[ 1358.596188] ? exc_page_fault+0x9e/0xb0
[ 1358.601290] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.607756] RIP: 0033:0x7f50e00e9c5e
[ 1358.612566] RSP: 002b:00007ffec4fb6770 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.621863] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f50e00e9c5e
[ 1358.630668] RDX: 0000000000000001 RSI: 000055d02cf04110 RDI: 0000000000000007
[ 1358.639476] RBP: 00007ffec4fb6780 R08: 0000000000000000 R09: 0000000000000000
[ 1358.648274] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f50dff84e80
[ 1358.657074] R13: 000055d02cf04110 R14: 0000000000000007 R15: 00007f50e07eab98
[ 1358.665886]
[ 1358.669153] INFO: task python3:5071 is blocked on a mutex likely owned by task python3:3121.
[ 1358.679425] task:python3 state:D stack:25312 pid:3121 tgid:3121 ppid:3120 task_flags:0x400100 flags:0x00080802
[ 1358.692626] Call Trace:
[ 1358.696164]
[ 1358.699311] __schedule+0xad9/0x1490
[ 1358.704118] ? io_schedule_timeout+0x130/0x130
[ 1358.709899] ? find_held_lock+0x2b/0x80
[ 1358.714999] ? __try_to_del_timer_sync+0x154/0x1f0
[ 1358.721179] schedule+0xd4/0x260
[ 1358.725598] schedule_timeout+0x103/0x260
[ 1358.730886] ? rcu_is_watching+0x15/0xd0
[ 1358.736082] ? hrtimer_nanosleep_restart+0x180/0x180
[ 1358.742457] ? hrtimers_cpu_dying+0x250/0x250
[ 1358.748147] ? ww_mutex_lock+0x160/0x160
[ 1358.753348] msleep+0x1b/0x30
[ 1358.757474] napi_set_threaded+0x376/0x410
[ 1358.762869] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.768363] netif_set_threaded+0x254/0x400
[ 1358.773853] ? sysfs_file_ops+0x140/0x140
[ 1358.779148] threaded_store+0x1bb/0x220
[ 1358.784253] ? tx_maxrate_store+0x4e0/0x4e0
[ 1358.789749] ? sysfs_file_ops+0x140/0x140
[ 1358.795042] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.800338] ? sysfs_kf_write+0x65/0x160
[ 1358.805535] kernfs_fop_write_iter+0x335/0x510
[ 1358.811321] ? kernfs_ops+0xf0/0xf0
[ 1358.816030] new_sync_write+0x308/0x6f0
[ 1358.821131] ? new_sync_read+0x6f0/0x6f0
[ 1358.826329] ? lock_acquire.part.0+0xbc/0x260
[ 1358.832012] ? ksys_write+0xff/0x200
[ 1358.836825] vfs_write+0x62a/0x980
[ 1358.841441] ksys_write+0xff/0x200
[ 1358.846056] ? __ia32_sys_read+0xc0/0xc0
[ 1358.851256] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.856846] ? rcu_is_watching+0x15/0xd0
[ 1358.862039] do_syscall_64+0xf3/0x5e0
[ 1358.866946] ? exc_page_fault+0x9e/0xb0
[ 1358.872047] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.878514] RIP: 0033:0x7fc46a0e3c5e
[ 1358.883321] RSP: 002b:00007ffec8495ab0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.892619] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc46a0e3c5e
[ 1358.901426] RDX: 0000000000000001 RSI: 0000559397594110 RDI: 0000000000000007
[ 1358.910232] RBP: 00007ffec8495ac0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.919039] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc469f7ee80
[ 1358.927845] R13: 0000559397594110 R14: 0000000000000007 R15: 00007fc469f0e480
[ 1358.936659]
[ 1358.939928] \x0aShowing all locks held in the system:
[ 1358.948463] 1 lock held by khungtaskd/185:
[ 1358.953857] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.965337] 4 locks held by python3/3121:
[ 1358.970631] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1358.980422] #1: ff1100013aeca688 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1358.991386] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.002546] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.012641] 4 locks held by python3/5071:
[ 1359.017938] #0: ff1100012e7513f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1359.027731] #1: ff110001aa7ddc88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1359.038694] #2: ff11000111fab0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.049853] #3: ff11000122061d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.059946] 2 locks held by 9/6914:
[ 1359.059951]
[ 1359.059953] =============================================\x0a