[ 784.126694] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 784.129299] nipa-hw-worker: [5/45] Retrying drivers/net:napi_threaded.py
[ 784.281451] kselftest: Running tests in drivers/net
[ 989.640842] INFO: task python3:5781 blocked for more than 122 seconds.
[ 989.649073] Not tainted 7.1.0-rc1-cjuf-g3748b1e7f236 #1
[ 989.656946] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.666536] task:python3 state:D stack:25672 pid:5781 tgid:5781 ppid:5780 task_flags:0x400100 flags:0x00080000
[ 989.679738] Call Trace:
[ 989.683275]
[ 989.686425] __schedule+0xcba/0x1940
[ 989.691241] ? io_schedule_timeout+0x140/0x140
[ 989.697024] ? find_held_lock+0x2b/0x80
[ 989.702129] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.708014] schedule+0xd4/0x260
[ 989.712432] schedule_preempt_disabled+0x18/0x30
[ 989.718412] __mutex_lock+0x6a3/0x2000
[ 989.723414] ? threaded_store+0x132/0x220
[ 989.728713] ? ww_mutex_lock+0x160/0x160
[ 989.733909] ? selinux_inode_free_security+0x330/0x330
[ 989.740479] ? cap_capable+0x1d6/0x3e0
[ 989.745489] ? threaded_store+0x132/0x220
[ 989.750783] threaded_store+0x132/0x220
[ 989.755884] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.761182] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.766483] ? sysfs_kf_write+0x65/0x160
[ 989.771681] ? sysfs_file_ops+0x140/0x140
[ 989.776976] kernfs_fop_write_iter+0x335/0x510
[ 989.782760] ? kernfs_ops+0xf0/0xf0
[ 989.787469] new_sync_write+0x333/0x750
[ 989.792570] ? __lock_acquire+0x508/0xc10
[ 989.797867] ? new_sync_read+0x740/0x740
[ 989.803066] ? lock_acquire.part.0+0xbc/0x260
[ 989.808750] ? ksys_write+0x116/0x250
[ 989.813662] vfs_write+0x657/0x9b0
[ 989.818277] ksys_write+0x116/0x250
[ 989.822988] ? __ia32_sys_read+0xc0/0xc0
[ 989.828189] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.833768] ? rcu_is_watching+0x15/0xd0
[ 989.838962] do_syscall_64+0xf3/0x650
[ 989.843864] ? trace_hardirqs_off+0xd/0x30
[ 989.849258] ? exc_page_fault+0xda/0xf0
[ 989.854357] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.860823] RIP: 0033:0x7fab1a67322e
[ 989.865631] RSP: 002b:00007ffce3ccd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.874928] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab1a67322e
[ 989.883736] RDX: 0000000000000001 RSI: 0000555bf6f4b2f0 RDI: 0000000000000007
[ 989.892534] RBP: 00007ffce3ccd440 R08: 0000000000000000 R09: 0000000000000000
[ 989.901342] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab1a4eb390
[ 989.910149] R13: 0000555bf6f4b2f0 R14: 0000000000000007 R15: 0000000000000003
[ 989.918960]
[ 989.922277] INFO: task python3:5781 is blocked on a mutex likely owned by task python3:3682.
[ 989.932549] task:python3 state:D stack:25672 pid:3682 tgid:3682 ppid:3681 task_flags:0x400100 flags:0x00080802
[ 989.945751] Call Trace:
[ 989.949286]
[ 989.952435] __schedule+0xcba/0x1940
[ 989.957245] ? io_schedule_timeout+0x140/0x140
[ 989.963028] ? find_held_lock+0x2b/0x80
[ 989.968129] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.974308] schedule+0xd4/0x260
[ 989.978724] schedule_timeout+0x13f/0x290
[ 989.984017] ? hrtimer_nanosleep_restart+0x270/0x270
[ 989.990383] ? trace_contention_end+0xb3/0x1a0
[ 989.996166] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.001855] ? ww_mutex_lock+0x160/0x160
[ 990.007064] msleep+0x1b/0x30
[ 990.011188] napi_set_threaded+0x3a5/0x440
[ 990.016584] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.022079] netif_set_threaded+0x257/0x400
[ 990.027570] threaded_store+0x1b8/0x220
[ 990.032671] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.037972] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.043268] ? sysfs_kf_write+0x65/0x160
[ 990.048465] ? sysfs_file_ops+0x140/0x140
[ 990.053758] kernfs_fop_write_iter+0x335/0x510
[ 990.059544] ? kernfs_ops+0xf0/0xf0
[ 990.064252] new_sync_write+0x333/0x750
[ 990.069352] ? __lock_acquire+0x508/0xc10
[ 990.074649] ? new_sync_read+0x740/0x740
[ 990.079848] ? lock_acquire.part.0+0xbc/0x260
[ 990.085532] ? ksys_write+0x116/0x250
[ 990.090442] vfs_write+0x657/0x9b0
[ 990.095057] ksys_write+0x116/0x250
[ 990.099768] ? __ia32_sys_read+0xc0/0xc0
[ 990.104968] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.110556] ? rcu_is_watching+0x15/0xd0
[ 990.115756] do_syscall_64+0xf3/0x650
[ 990.120661] ? trace_hardirqs_off+0xd/0x30
[ 990.126053] ? exc_page_fault+0xda/0xf0
[ 990.131152] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.137618] RIP: 0033:0x7fc5ee45322e
[ 990.142424] RSP: 002b:00007fff9690a9d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.151721] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc5ee45322e
[ 990.160521] RDX: 0000000000000001 RSI: 00005623d49bf2f0 RDI: 0000000000000007
[ 990.169328] RBP: 00007fff9690a9e0 R08: 0000000000000000 R09: 0000000000000000
[ 990.178134] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc5ee2cb390
[ 990.186939] R13: 00005623d49bf2f0 R14: 0000000000000007 R15: 0000000000000003
[ 990.195750]
[ 990.199007] \x0aShowing all locks held in the system:
[ 990.207547] 1 lock held by khungtaskd/185:
[ 990.212941] #0: ffffffff8c6fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.224426] 4 locks held by python3/3682:
[ 990.229719] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.239609] #1: ff11000163d5d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.250568] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.261727] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.271826] 4 locks held by python3/5781:
[ 990.277112] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.287006] #1: ff110001e0a6d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.297977] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.309139] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.319225]
[ 990.321690] =============================================\x0a
[ 1112.523272] INFO: task python3:5781 blocked for more than 245 seconds.
[ 1112.531420] Not tainted 7.1.0-rc1-cjuf-g3748b1e7f236 #1
[ 1112.538671] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.548260] task:python3 state:D stack:25672 pid:5781 tgid:5781 ppid:5780 task_flags:0x400100 flags:0x00080000
[ 1112.561464] Call Trace:
[ 1112.565003]
[ 1112.568150] __schedule+0xcba/0x1940
[ 1112.572967] ? io_schedule_timeout+0x140/0x140
[ 1112.578753] ? find_held_lock+0x2b/0x80
[ 1112.583858] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.589740] schedule+0xd4/0x260
[ 1112.594155] schedule_preempt_disabled+0x18/0x30
[ 1112.600136] __mutex_lock+0x6a3/0x2000
[ 1112.605141] ? threaded_store+0x132/0x220
[ 1112.610442] ? ww_mutex_lock+0x160/0x160
[ 1112.615639] ? selinux_inode_free_security+0x330/0x330
[ 1112.622198] ? cap_capable+0x1d6/0x3e0
[ 1112.627205] ? threaded_store+0x132/0x220
[ 1112.632501] threaded_store+0x132/0x220
[ 1112.637600] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.642899] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.648197] ? sysfs_kf_write+0x65/0x160
[ 1112.653398] ? sysfs_file_ops+0x140/0x140
[ 1112.658693] kernfs_fop_write_iter+0x335/0x510
[ 1112.664479] ? kernfs_ops+0xf0/0xf0
[ 1112.669180] new_sync_write+0x333/0x750
[ 1112.674279] ? __lock_acquire+0x508/0xc10
[ 1112.679574] ? new_sync_read+0x740/0x740
[ 1112.684775] ? lock_acquire.part.0+0xbc/0x260
[ 1112.690461] ? ksys_write+0x116/0x250
[ 1112.695370] vfs_write+0x657/0x9b0
[ 1112.699982] ksys_write+0x116/0x250
[ 1112.704690] ? __ia32_sys_read+0xc0/0xc0
[ 1112.709886] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.715471] ? rcu_is_watching+0x15/0xd0
[ 1112.720671] do_syscall_64+0xf3/0x650
[ 1112.725574] ? trace_hardirqs_off+0xd/0x30
[ 1112.730969] ? exc_page_fault+0xda/0xf0
[ 1112.736072] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.742537] RIP: 0033:0x7fab1a67322e
[ 1112.747347] RSP: 002b:00007ffce3ccd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.756636] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab1a67322e
[ 1112.765445] RDX: 0000000000000001 RSI: 0000555bf6f4b2f0 RDI: 0000000000000007
[ 1112.774253] RBP: 00007ffce3ccd440 R08: 0000000000000000 R09: 0000000000000000
[ 1112.783062] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab1a4eb390
[ 1112.791871] R13: 0000555bf6f4b2f0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.800685]
[ 1112.804007] INFO: task python3:5781 is blocked on a mutex likely owned by task python3:3682.
[ 1112.814284] task:python3 state:D stack:25672 pid:3682 tgid:3682 ppid:3681 task_flags:0x400100 flags:0x00080802
[ 1112.827489] Call Trace:
[ 1112.831026]
[ 1112.834177] __schedule+0xcba/0x1940
[ 1112.838990] ? io_schedule_timeout+0x140/0x140
[ 1112.844774] ? find_held_lock+0x2b/0x80
[ 1112.849877] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.856044] schedule+0xd4/0x260
[ 1112.860462] schedule_timeout+0x13f/0x290
[ 1112.865757] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.872126] ? trace_contention_end+0xb3/0x1a0
[ 1112.877911] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.883602] ? ww_mutex_lock+0x160/0x160
[ 1112.888801] msleep+0x1b/0x30
[ 1112.892926] napi_set_threaded+0x3a5/0x440
[ 1112.898320] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.903812] netif_set_threaded+0x257/0x400
[ 1112.909308] threaded_store+0x1b8/0x220
[ 1112.914408] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.919704] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.924998] ? sysfs_kf_write+0x65/0x160
[ 1112.930197] ? sysfs_file_ops+0x140/0x140
[ 1112.935493] kernfs_fop_write_iter+0x335/0x510
[ 1112.941274] ? kernfs_ops+0xf0/0xf0
[ 1112.945984] new_sync_write+0x333/0x750
[ 1112.951087] ? __lock_acquire+0x508/0xc10
[ 1112.956382] ? new_sync_read+0x740/0x740
[ 1112.961583] ? lock_acquire.part.0+0xbc/0x260
[ 1112.967269] ? ksys_write+0x116/0x250
[ 1112.972177] vfs_write+0x657/0x9b0
[ 1112.976793] ksys_write+0x116/0x250
[ 1112.981505] ? __ia32_sys_read+0xc0/0xc0
[ 1112.986702] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.992287] ? rcu_is_watching+0x15/0xd0
[ 1112.997485] do_syscall_64+0xf3/0x650
[ 1113.002390] ? trace_hardirqs_off+0xd/0x30
[ 1113.007785] ? exc_page_fault+0xda/0xf0
[ 1113.012887] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.019350] RIP: 0033:0x7fc5ee45322e
[ 1113.024156] RSP: 002b:00007fff9690a9d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.033453] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc5ee45322e
[ 1113.042258] RDX: 0000000000000001 RSI: 00005623d49bf2f0 RDI: 0000000000000007
[ 1113.051062] RBP: 00007fff9690a9e0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.059876] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc5ee2cb390
[ 1113.068678] R13: 00005623d49bf2f0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.077491]
[ 1113.080753] \x0aShowing all locks held in the system:
[ 1113.089295] 1 lock held by khungtaskd/185:
[ 1113.094687] #0: ffffffff8c6fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.106175] 4 locks held by python3/3682:
[ 1113.111473] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.121362] #1: ff11000163d5d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.132314] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.143469] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.153568] 4 locks held by python3/5781:
[ 1113.158855] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.168745] #1: ff110001e0a6d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.179705] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.190861] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.200952]
[ 1113.203417] =============================================\x0a
[ 1235.404717] INFO: task python3:5781 blocked for more than 368 seconds.
[ 1235.412864] Not tainted 7.1.0-rc1-cjuf-g3748b1e7f236 #1
[ 1235.420119] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.429723] task:python3 state:D stack:25672 pid:5781 tgid:5781 ppid:5780 task_flags:0x400100 flags:0x00080002
[ 1235.442924] Call Trace:
[ 1235.446464]
[ 1235.449615] __schedule+0xcba/0x1940
[ 1235.454433] ? io_schedule_timeout+0x140/0x140
[ 1235.460217] ? find_held_lock+0x2b/0x80
[ 1235.465321] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.471202] schedule+0xd4/0x260
[ 1235.475618] schedule_preempt_disabled+0x18/0x30
[ 1235.481599] __mutex_lock+0x6a3/0x2000
[ 1235.486601] ? threaded_store+0x132/0x220
[ 1235.491899] ? ww_mutex_lock+0x160/0x160
[ 1235.497096] ? selinux_inode_free_security+0x330/0x330
[ 1235.503665] ? cap_capable+0x1d6/0x3e0
[ 1235.508673] ? threaded_store+0x132/0x220
[ 1235.513968] threaded_store+0x132/0x220
[ 1235.519067] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.524364] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.529659] ? sysfs_kf_write+0x65/0x160
[ 1235.534859] ? sysfs_file_ops+0x140/0x140
[ 1235.540146] kernfs_fop_write_iter+0x335/0x510
[ 1235.545932] ? kernfs_ops+0xf0/0xf0
[ 1235.550633] new_sync_write+0x333/0x750
[ 1235.555733] ? __lock_acquire+0x508/0xc10
[ 1235.561028] ? new_sync_read+0x740/0x740
[ 1235.566229] ? lock_acquire.part.0+0xbc/0x260
[ 1235.571915] ? ksys_write+0x116/0x250
[ 1235.576824] vfs_write+0x657/0x9b0
[ 1235.581436] ksys_write+0x116/0x250
[ 1235.586149] ? __ia32_sys_read+0xc0/0xc0
[ 1235.591346] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.596935] ? rcu_is_watching+0x15/0xd0
[ 1235.602134] do_syscall_64+0xf3/0x650
[ 1235.607038] ? trace_hardirqs_off+0xd/0x30
[ 1235.612435] ? exc_page_fault+0xda/0xf0
[ 1235.617536] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.624003] RIP: 0033:0x7fab1a67322e
[ 1235.628805] RSP: 002b:00007ffce3ccd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.638095] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab1a67322e
[ 1235.646900] RDX: 0000000000000001 RSI: 0000555bf6f4b2f0 RDI: 0000000000000007
[ 1235.655706] RBP: 00007ffce3ccd440 R08: 0000000000000000 R09: 0000000000000000
[ 1235.664510] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab1a4eb390
[ 1235.673318] R13: 0000555bf6f4b2f0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.682133]
[ 1235.685427] INFO: task python3:5781 is blocked on a mutex likely owned by task python3:3682.
[ 1235.695698] task:python3 state:D stack:25672 pid:3682 tgid:3682 ppid:3681 task_flags:0x400100 flags:0x00080802
[ 1235.708900] Call Trace:
[ 1235.712437]
[ 1235.715586] __schedule+0xcba/0x1940
[ 1235.720399] ? io_schedule_timeout+0x140/0x140
[ 1235.726183] ? find_held_lock+0x2b/0x80
[ 1235.731283] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.737459] schedule+0xd4/0x260
[ 1235.741878] schedule_timeout+0x13f/0x290
[ 1235.747175] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.753543] ? trace_contention_end+0xb3/0x1a0
[ 1235.759327] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.765015] ? ww_mutex_lock+0x160/0x160
[ 1235.770214] msleep+0x1b/0x30
[ 1235.774340] napi_set_threaded+0x3a5/0x440
[ 1235.779734] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.785228] netif_set_threaded+0x257/0x400
[ 1235.790723] threaded_store+0x1b8/0x220
[ 1235.795824] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.801122] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.806417] ? sysfs_kf_write+0x65/0x160
[ 1235.811616] ? sysfs_file_ops+0x140/0x140
[ 1235.816909] kernfs_fop_write_iter+0x335/0x510
[ 1235.822692] ? kernfs_ops+0xf0/0xf0
[ 1235.827403] new_sync_write+0x333/0x750
[ 1235.832506] ? __lock_acquire+0x508/0xc10
[ 1235.837801] ? new_sync_read+0x740/0x740
[ 1235.842998] ? lock_acquire.part.0+0xbc/0x260
[ 1235.848686] ? ksys_write+0x116/0x250
[ 1235.853596] vfs_write+0x657/0x9b0
[ 1235.858209] ksys_write+0x116/0x250
[ 1235.862917] ? __ia32_sys_read+0xc0/0xc0
[ 1235.868115] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.873703] ? rcu_is_watching+0x15/0xd0
[ 1235.878903] do_syscall_64+0xf3/0x650
[ 1235.883806] ? trace_hardirqs_off+0xd/0x30
[ 1235.889196] ? exc_page_fault+0xda/0xf0
[ 1235.894295] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.900759] RIP: 0033:0x7fc5ee45322e
[ 1235.905565] RSP: 002b:00007fff9690a9d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.914864] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc5ee45322e
[ 1235.923675] RDX: 0000000000000001 RSI: 00005623d49bf2f0 RDI: 0000000000000007
[ 1235.932481] RBP: 00007fff9690a9e0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.941285] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc5ee2cb390
[ 1235.950090] R13: 00005623d49bf2f0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.958901]
[ 1235.962165] \x0aShowing all locks held in the system:
[ 1235.970705] 1 lock held by khungtaskd/185:
[ 1235.976098] #0: ffffffff8c6fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1235.987583] 4 locks held by python3/3682:
[ 1235.992876] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.002764] #1: ff11000163d5d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.013722] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.024878] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.034974] 4 locks held by python3/5781:
[ 1236.040266] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.050153] #1: ff110001e0a6d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.061115] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.072264] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.082353]
[ 1236.084817] =============================================\x0a
[ 1358.285194] INFO: task python3:5781 blocked for more than 491 seconds.
[ 1358.293348] Not tainted 7.1.0-rc1-cjuf-g3748b1e7f236 #1
[ 1358.300600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.310192] task:python3 state:D stack:25672 pid:5781 tgid:5781 ppid:5780 task_flags:0x400100 flags:0x00080002
[ 1358.323400] Call Trace:
[ 1358.326941]
[ 1358.330094] __schedule+0xcba/0x1940
[ 1358.334912] ? io_schedule_timeout+0x140/0x140
[ 1358.340698] ? find_held_lock+0x2b/0x80
[ 1358.345809] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.351693] schedule+0xd4/0x260
[ 1358.356115] schedule_preempt_disabled+0x18/0x30
[ 1358.362096] __mutex_lock+0x6a3/0x2000
[ 1358.367102] ? threaded_store+0x132/0x220
[ 1358.372401] ? ww_mutex_lock+0x160/0x160
[ 1358.377600] ? selinux_inode_free_security+0x330/0x330
[ 1358.384169] ? cap_capable+0x1d6/0x3e0
[ 1358.389178] ? threaded_store+0x132/0x220
[ 1358.394474] threaded_store+0x132/0x220
[ 1358.399575] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.404872] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.410172] ? sysfs_kf_write+0x65/0x160
[ 1358.415372] ? sysfs_file_ops+0x140/0x140
[ 1358.420659] kernfs_fop_write_iter+0x335/0x510
[ 1358.426445] ? kernfs_ops+0xf0/0xf0
[ 1358.431156] new_sync_write+0x333/0x750
[ 1358.436252] ? __lock_acquire+0x508/0xc10
[ 1358.441549] ? new_sync_read+0x740/0x740
[ 1358.446751] ? lock_acquire.part.0+0xbc/0x260
[ 1358.452439] ? ksys_write+0x116/0x250
[ 1358.457350] vfs_write+0x657/0x9b0
[ 1358.461968] ksys_write+0x116/0x250
[ 1358.466680] ? __ia32_sys_read+0xc0/0xc0
[ 1358.471881] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.477472] ? rcu_is_watching+0x15/0xd0
[ 1358.482673] do_syscall_64+0xf3/0x650
[ 1358.487572] ? trace_hardirqs_off+0xd/0x30
[ 1358.492968] ? exc_page_fault+0xda/0xf0
[ 1358.498071] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.504541] RIP: 0033:0x7fab1a67322e
[ 1358.509351] RSP: 002b:00007ffce3ccd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.518649] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab1a67322e
[ 1358.527458] RDX: 0000000000000001 RSI: 0000555bf6f4b2f0 RDI: 0000000000000007
[ 1358.536268] RBP: 00007ffce3ccd440 R08: 0000000000000000 R09: 0000000000000000
[ 1358.545077] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab1a4eb390
[ 1358.553887] R13: 0000555bf6f4b2f0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.562701]
[ 1358.565984] INFO: task python3:5781 is blocked on a mutex likely owned by task python3:3682.
[ 1358.576250] task:python3 state:D stack:25672 pid:3682 tgid:3682 ppid:3681 task_flags:0x400100 flags:0x00080802
[ 1358.589453] Call Trace:
[ 1358.592989]
[ 1358.596142] __schedule+0xcba/0x1940
[ 1358.600953] ? io_schedule_timeout+0x140/0x140
[ 1358.606738] ? find_held_lock+0x2b/0x80
[ 1358.611833] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.618009] schedule+0xd4/0x260
[ 1358.622429] schedule_timeout+0x13f/0x290
[ 1358.627724] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.634093] ? trace_contention_end+0xb3/0x1a0
[ 1358.639872] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.645564] ? ww_mutex_lock+0x160/0x160
[ 1358.650763] msleep+0x1b/0x30
[ 1358.654880] napi_set_threaded+0x3a5/0x440
[ 1358.660279] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.665773] netif_set_threaded+0x257/0x400
[ 1358.671271] threaded_store+0x1b8/0x220
[ 1358.676372] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.681671] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.686969] ? sysfs_kf_write+0x65/0x160
[ 1358.692170] ? sysfs_file_ops+0x140/0x140
[ 1358.697468] kernfs_fop_write_iter+0x335/0x510
[ 1358.703246] ? kernfs_ops+0xf0/0xf0
[ 1358.707956] new_sync_write+0x333/0x750
[ 1358.713059] ? __lock_acquire+0x508/0xc10
[ 1358.718355] ? new_sync_read+0x740/0x740
[ 1358.723556] ? lock_acquire.part.0+0xbc/0x260
[ 1358.729243] ? ksys_write+0x116/0x250
[ 1358.734153] vfs_write+0x657/0x9b0
[ 1358.738770] ksys_write+0x116/0x250
[ 1358.743485] ? __ia32_sys_read+0xc0/0xc0
[ 1358.748686] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.754277] ? rcu_is_watching+0x15/0xd0
[ 1358.759476] do_syscall_64+0xf3/0x650
[ 1358.764384] ? trace_hardirqs_off+0xd/0x30
[ 1358.769778] ? exc_page_fault+0xda/0xf0
[ 1358.774881] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.781348] RIP: 0033:0x7fc5ee45322e
[ 1358.786157] RSP: 002b:00007fff9690a9d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.795455] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc5ee45322e
[ 1358.804263] RDX: 0000000000000001 RSI: 00005623d49bf2f0 RDI: 0000000000000007
[ 1358.813074] RBP: 00007fff9690a9e0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.821883] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc5ee2cb390
[ 1358.830689] R13: 00005623d49bf2f0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.839504]
[ 1358.842775] \x0aShowing all locks held in the system:
[ 1358.851316] 1 lock held by khungtaskd/185:
[ 1358.856711] #0: ffffffff8c6fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.868196] 4 locks held by python3/3682:
[ 1358.873493] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.883385] #1: ff11000163d5d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.894348] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.905502] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.915596] 4 locks held by python3/5781:
[ 1358.920891] #0: ff11000130c443e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.930779] #1: ff110001e0a6d080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.941743] #2: ff110017d91b2b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.952902] #3: ff110017d9211d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.962995]
[ 1358.965462] =============================================\x0a