[ 762.584073] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 762.586822] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 762.737402] kselftest: Running tests in drivers/net
[ 989.735903] INFO: task python3:5421 blocked for more than 122 seconds.
[ 989.744127] Not tainted 7.1.0-rc7-tgje-g535aecdaa85c #1
[ 989.752013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.761604] task:python3 state:D stack:25672 pid:5421 tgid:5421 ppid:5420 task_flags:0x400100 flags:0x00080000
[ 989.774808] Call Trace:
[ 989.778346]
[ 989.781496] __schedule+0xcc3/0x19c0
[ 989.786311] ? io_schedule_timeout+0x140/0x140
[ 989.792095] ? find_held_lock+0x2b/0x80
[ 989.797191] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.803073] schedule+0xd8/0x260
[ 989.807491] schedule_preempt_disabled+0x18/0x30
[ 989.813468] __mutex_lock+0x6af/0x2010
[ 989.818472] ? threaded_store+0x132/0x220
[ 989.823773] ? ww_mutex_lock+0x160/0x160
[ 989.828971] ? selinux_inode_free_security+0x330/0x330
[ 989.835541] ? cap_capable+0x1d6/0x3e0
[ 989.840547] ? threaded_store+0x132/0x220
[ 989.845840] threaded_store+0x132/0x220
[ 989.850943] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.856239] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.861541] ? sysfs_kf_write+0x65/0x160
[ 989.866744] ? sysfs_file_ops+0x140/0x140
[ 989.872039] kernfs_fop_write_iter+0x335/0x510
[ 989.877825] ? kernfs_ops+0xf0/0xf0
[ 989.882524] new_sync_write+0x333/0x750
[ 989.887627] ? __lock_acquire+0x508/0xc10
[ 989.892925] ? new_sync_read+0x740/0x740
[ 989.898124] ? lock_acquire.part.0+0xbc/0x260
[ 989.903800] ? ksys_write+0x116/0x250
[ 989.908710] vfs_write+0x657/0x9b0
[ 989.913326] ksys_write+0x116/0x250
[ 989.918035] ? __ia32_sys_read+0xc0/0xc0
[ 989.923235] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.928813] ? rcu_is_watching+0x15/0xd0
[ 989.934013] do_syscall_64+0xf3/0x650
[ 989.938916] ? trace_hardirqs_off+0xd/0x30
[ 989.944312] ? exc_page_fault+0xda/0xf0
[ 989.949413] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.955881] RIP: 0033:0x7f3cbef1264e
[ 989.960691] RSP: 002b:00007ffd12a84660 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.969987] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3cbef1264e
[ 989.978787] RDX: 0000000000000001 RSI: 0000560488afdbf0 RDI: 0000000000000007
[ 989.987586] RBP: 00007ffd12a84670 R08: 0000000000000000 R09: 0000000000000000
[ 989.996386] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3cbed8b390
[ 990.005193] R13: 0000560488afdbf0 R14: 0000000000000007 R15: 00007f3cbed21120
[ 990.014014]
[ 990.017307] INFO: task python3:5421 is blocked on a mutex likely owned by task python3:3422.
[ 990.027577] task:python3 state:D stack:25672 pid:3422 tgid:3422 ppid:3421 task_flags:0x400100 flags:0x00080802
[ 990.040779] Call Trace:
[ 990.044316]
[ 990.047467] __schedule+0xcc3/0x19c0
[ 990.052277] ? io_schedule_timeout+0x140/0x140
[ 990.058061] ? find_held_lock+0x2b/0x80
[ 990.063164] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.069341] schedule+0xd8/0x260
[ 990.073748] schedule_timeout+0x13f/0x290
[ 990.079044] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.085414] ? trace_contention_end+0xb3/0x1a0
[ 990.091199] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.096886] ? ww_mutex_lock+0x160/0x160
[ 990.102086] msleep+0x1b/0x30
[ 990.106201] napi_set_threaded+0x3a5/0x440
[ 990.111594] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.117087] netif_set_threaded+0x257/0x400
[ 990.122580] threaded_store+0x1b8/0x220
[ 990.127671] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.132968] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.138264] ? sysfs_kf_write+0x65/0x160
[ 990.143463] ? sysfs_file_ops+0x140/0x140
[ 990.148760] kernfs_fop_write_iter+0x335/0x510
[ 990.154544] ? kernfs_ops+0xf0/0xf0
[ 990.159254] new_sync_write+0x333/0x750
[ 990.164356] ? __lock_acquire+0x508/0xc10
[ 990.169651] ? new_sync_read+0x740/0x740
[ 990.174851] ? lock_acquire.part.0+0xbc/0x260
[ 990.180537] ? ksys_write+0x116/0x250
[ 990.185444] vfs_write+0x657/0x9b0
[ 990.190057] ksys_write+0x116/0x250
[ 990.194766] ? __ia32_sys_read+0xc0/0xc0
[ 990.199963] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.205552] ? rcu_is_watching+0x15/0xd0
[ 990.210755] do_syscall_64+0xf3/0x650
[ 990.215660] ? trace_hardirqs_off+0xd/0x30
[ 990.221055] ? exc_page_fault+0xda/0xf0
[ 990.226157] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.232621] RIP: 0033:0x7ffae976e64e
[ 990.237429] RSP: 002b:00007ffc63eafb90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.246725] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ffae976e64e
[ 990.255523] RDX: 0000000000000001 RSI: 000055c27bf86bf0 RDI: 0000000000000007
[ 990.264329] RBP: 00007ffc63eafba0 R08: 0000000000000000 R09: 0000000000000000
[ 990.273137] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffae95e7390
[ 990.281942] R13: 000055c27bf86bf0 R14: 0000000000000007 R15: 00007ffae957d120
[ 990.290761]
[ 990.294019] \x0aShowing all locks held in the system:
[ 990.302559] 1 lock held by khungtaskd/185:
[ 990.307952] #0: ffffffffb14fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.319435] 4 locks held by python3/3422:
[ 990.324721] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.334608] #1: ff11000135ffbe80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.345569] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.356716] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.366813] 4 locks held by python3/5421:
[ 990.372105] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.381993] #1: ff11000185161480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.392955] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.404113] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.414199]
[ 990.416664] =============================================\x0a
[ 1112.617056] INFO: task python3:5421 blocked for more than 245 seconds.
[ 1112.625204] Not tainted 7.1.0-rc7-tgje-g535aecdaa85c #1
[ 1112.632456] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.642048] task:python3 state:D stack:25672 pid:5421 tgid:5421 ppid:5420 task_flags:0x400100 flags:0x00080000
[ 1112.655254] Call Trace:
[ 1112.658795]
[ 1112.661949] __schedule+0xcc3/0x19c0
[ 1112.666765] ? io_schedule_timeout+0x140/0x140
[ 1112.672553] ? find_held_lock+0x2b/0x80
[ 1112.677656] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.683543] schedule+0xd8/0x260
[ 1112.687964] schedule_preempt_disabled+0x18/0x30
[ 1112.693943] __mutex_lock+0x6af/0x2010
[ 1112.698949] ? threaded_store+0x132/0x220
[ 1112.704248] ? ww_mutex_lock+0x160/0x160
[ 1112.709441] ? selinux_inode_free_security+0x330/0x330
[ 1112.716011] ? cap_capable+0x1d6/0x3e0
[ 1112.721022] ? threaded_store+0x132/0x220
[ 1112.726317] threaded_store+0x132/0x220
[ 1112.731419] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.736716] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.742018] ? sysfs_kf_write+0x65/0x160
[ 1112.747222] ? sysfs_file_ops+0x140/0x140
[ 1112.752517] kernfs_fop_write_iter+0x335/0x510
[ 1112.758301] ? kernfs_ops+0xf0/0xf0
[ 1112.763014] new_sync_write+0x333/0x750
[ 1112.768117] ? __lock_acquire+0x508/0xc10
[ 1112.773414] ? new_sync_read+0x740/0x740
[ 1112.778617] ? lock_acquire.part.0+0xbc/0x260
[ 1112.784306] ? ksys_write+0x116/0x250
[ 1112.789214] vfs_write+0x657/0x9b0
[ 1112.793828] ksys_write+0x116/0x250
[ 1112.798537] ? __ia32_sys_read+0xc0/0xc0
[ 1112.803735] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.809323] ? rcu_is_watching+0x15/0xd0
[ 1112.814526] do_syscall_64+0xf3/0x650
[ 1112.819431] ? trace_hardirqs_off+0xd/0x30
[ 1112.824829] ? exc_page_fault+0xda/0xf0
[ 1112.829934] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.836402] RIP: 0033:0x7f3cbef1264e
[ 1112.841214] RSP: 002b:00007ffd12a84660 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.850514] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3cbef1264e
[ 1112.859320] RDX: 0000000000000001 RSI: 0000560488afdbf0 RDI: 0000000000000007
[ 1112.868128] RBP: 00007ffd12a84670 R08: 0000000000000000 R09: 0000000000000000
[ 1112.876936] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3cbed8b390
[ 1112.885743] R13: 0000560488afdbf0 R14: 0000000000000007 R15: 00007f3cbed21120
[ 1112.894559]
[ 1112.897856] INFO: task python3:5421 is blocked on a mutex likely owned by task python3:3422.
[ 1112.908129] task:python3 state:D stack:25672 pid:3422 tgid:3422 ppid:3421 task_flags:0x400100 flags:0x00080802
[ 1112.921331] Call Trace:
[ 1112.924869]
[ 1112.928022] __schedule+0xcc3/0x19c0
[ 1112.932831] ? io_schedule_timeout+0x140/0x140
[ 1112.938617] ? find_held_lock+0x2b/0x80
[ 1112.943713] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.949896] schedule+0xd8/0x260
[ 1112.954315] schedule_timeout+0x13f/0x290
[ 1112.959603] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.965973] ? trace_contention_end+0xb3/0x1a0
[ 1112.971761] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.977454] ? ww_mutex_lock+0x160/0x160
[ 1112.982654] msleep+0x1b/0x30
[ 1112.986778] napi_set_threaded+0x3a5/0x440
[ 1112.992178] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.997671] netif_set_threaded+0x257/0x400
[ 1113.003167] threaded_store+0x1b8/0x220
[ 1113.008269] ? xps_rxqs_store+0x2e0/0x2e0
[ 1113.013567] ? sysfs_file_kobj+0xbd/0x1e0
[ 1113.018854] ? sysfs_kf_write+0x65/0x160
[ 1113.024055] ? sysfs_file_ops+0x140/0x140
[ 1113.029353] kernfs_fop_write_iter+0x335/0x510
[ 1113.035137] ? kernfs_ops+0xf0/0xf0
[ 1113.039848] new_sync_write+0x333/0x750
[ 1113.044953] ? __lock_acquire+0x508/0xc10
[ 1113.050249] ? new_sync_read+0x740/0x740
[ 1113.055451] ? lock_acquire.part.0+0xbc/0x260
[ 1113.061137] ? ksys_write+0x116/0x250
[ 1113.066047] vfs_write+0x657/0x9b0
[ 1113.070663] ksys_write+0x116/0x250
[ 1113.075377] ? __ia32_sys_read+0xc0/0xc0
[ 1113.080577] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.086165] ? rcu_is_watching+0x15/0xd0
[ 1113.091364] do_syscall_64+0xf3/0x650
[ 1113.096273] ? trace_hardirqs_off+0xd/0x30
[ 1113.101668] ? exc_page_fault+0xda/0xf0
[ 1113.106763] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.113231] RIP: 0033:0x7ffae976e64e
[ 1113.118032] RSP: 002b:00007ffc63eafb90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.127330] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ffae976e64e
[ 1113.136143] RDX: 0000000000000001 RSI: 000055c27bf86bf0 RDI: 0000000000000007
[ 1113.144951] RBP: 00007ffc63eafba0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.153748] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffae95e7390
[ 1113.162551] R13: 000055c27bf86bf0 R14: 0000000000000007 R15: 00007ffae957d120
[ 1113.171363]
[ 1113.174626] \x0aShowing all locks held in the system:
[ 1113.183170] 1 lock held by khungtaskd/185:
[ 1113.188563] #0: ffffffffb14fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.200048] 4 locks held by python3/3422:
[ 1113.205345] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.215237] #1: ff11000135ffbe80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.226199] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.237355] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.247451] 4 locks held by python3/5421:
[ 1113.252747] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.262634] #1: ff11000185161480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.273596] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.284755] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.294848]
[ 1113.297311] =============================================\x0a
[ 1169.961549] kworker/u130:5 (532) used greatest stack depth: 21936 bytes left
[ 1235.498192] INFO: task python3:5421 blocked for more than 368 seconds.
[ 1235.506337] Not tainted 7.1.0-rc7-tgje-g535aecdaa85c #1
[ 1235.513588] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.523174] task:python3 state:D stack:25672 pid:5421 tgid:5421 ppid:5420 task_flags:0x400100 flags:0x00080002
[ 1235.536379] Call Trace:
[ 1235.539919]
[ 1235.543067] __schedule+0xcc3/0x19c0
[ 1235.547880] ? io_schedule_timeout+0x140/0x140
[ 1235.553666] ? find_held_lock+0x2b/0x80
[ 1235.558767] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.564653] schedule+0xd8/0x260
[ 1235.569069] schedule_preempt_disabled+0x18/0x30
[ 1235.575045] __mutex_lock+0x6af/0x2010
[ 1235.580049] ? threaded_store+0x132/0x220
[ 1235.585338] ? ww_mutex_lock+0x160/0x160
[ 1235.590533] ? selinux_inode_free_security+0x330/0x330
[ 1235.597103] ? cap_capable+0x1d6/0x3e0
[ 1235.602108] ? threaded_store+0x132/0x220
[ 1235.607402] threaded_store+0x132/0x220
[ 1235.612504] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.617798] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.623096] ? sysfs_kf_write+0x65/0x160
[ 1235.628295] ? sysfs_file_ops+0x140/0x140
[ 1235.633588] kernfs_fop_write_iter+0x335/0x510
[ 1235.639375] ? kernfs_ops+0xf0/0xf0
[ 1235.644084] new_sync_write+0x333/0x750
[ 1235.649186] ? __lock_acquire+0x508/0xc10
[ 1235.654483] ? new_sync_read+0x740/0x740
[ 1235.659682] ? lock_acquire.part.0+0xbc/0x260
[ 1235.665365] ? ksys_write+0x116/0x250
[ 1235.670275] vfs_write+0x657/0x9b0
[ 1235.674888] ksys_write+0x116/0x250
[ 1235.679597] ? __ia32_sys_read+0xc0/0xc0
[ 1235.684795] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.690383] ? rcu_is_watching+0x15/0xd0
[ 1235.695585] do_syscall_64+0xf3/0x650
[ 1235.700490] ? trace_hardirqs_off+0xd/0x30
[ 1235.705882] ? exc_page_fault+0xda/0xf0
[ 1235.710985] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.717452] RIP: 0033:0x7f3cbef1264e
[ 1235.722250] RSP: 002b:00007ffd12a84660 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.731546] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3cbef1264e
[ 1235.740353] RDX: 0000000000000001 RSI: 0000560488afdbf0 RDI: 0000000000000007
[ 1235.749160] RBP: 00007ffd12a84670 R08: 0000000000000000 R09: 0000000000000000
[ 1235.757966] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3cbed8b390
[ 1235.766773] R13: 0000560488afdbf0 R14: 0000000000000007 R15: 00007f3cbed21120
[ 1235.775586]
[ 1235.778901] INFO: task python3:5421 is blocked on a mutex likely owned by task python3:3422.
[ 1235.789172] task:python3 state:D stack:25672 pid:3422 tgid:3422 ppid:3421 task_flags:0x400100 flags:0x00080802
[ 1235.802374] Call Trace:
[ 1235.805914]
[ 1235.809061] __schedule+0xcc3/0x19c0
[ 1235.813870] ? io_schedule_timeout+0x140/0x140
[ 1235.819654] ? find_held_lock+0x2b/0x80
[ 1235.824753] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.830931] schedule+0xd8/0x260
[ 1235.835350] schedule_timeout+0x13f/0x290
[ 1235.840648] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.847005] ? trace_contention_end+0xb3/0x1a0
[ 1235.852789] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.858478] ? ww_mutex_lock+0x160/0x160
[ 1235.863674] msleep+0x1b/0x30
[ 1235.867798] napi_set_threaded+0x3a5/0x440
[ 1235.873193] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.878687] netif_set_threaded+0x257/0x400
[ 1235.884180] threaded_store+0x1b8/0x220
[ 1235.889291] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.894581] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.899876] ? sysfs_kf_write+0x65/0x160
[ 1235.905077] ? sysfs_file_ops+0x140/0x140
[ 1235.910371] kernfs_fop_write_iter+0x335/0x510
[ 1235.916155] ? kernfs_ops+0xf0/0xf0
[ 1235.920862] new_sync_write+0x333/0x750
[ 1235.925963] ? __lock_acquire+0x508/0xc10
[ 1235.931260] ? new_sync_read+0x740/0x740
[ 1235.936450] ? lock_acquire.part.0+0xbc/0x260
[ 1235.942133] ? ksys_write+0x116/0x250
[ 1235.947045] vfs_write+0x657/0x9b0
[ 1235.951659] ksys_write+0x116/0x250
[ 1235.956368] ? __ia32_sys_read+0xc0/0xc0
[ 1235.961567] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.967156] ? rcu_is_watching+0x15/0xd0
[ 1235.972356] do_syscall_64+0xf3/0x650
[ 1235.977253] ? trace_hardirqs_off+0xd/0x30
[ 1235.982646] ? exc_page_fault+0xda/0xf0
[ 1235.987746] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.994212] RIP: 0033:0x7ffae976e64e
[ 1235.999019] RSP: 002b:00007ffc63eafb90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1236.008314] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ffae976e64e
[ 1236.017120] RDX: 0000000000000001 RSI: 000055c27bf86bf0 RDI: 0000000000000007
[ 1236.025928] RBP: 00007ffc63eafba0 R08: 0000000000000000 R09: 0000000000000000
[ 1236.034734] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffae95e7390
[ 1236.043541] R13: 000055c27bf86bf0 R14: 0000000000000007 R15: 00007ffae957d120
[ 1236.052354]
[ 1236.055618] \x0aShowing all locks held in the system:
[ 1236.064159] 1 lock held by khungtaskd/185:
[ 1236.069540] #0: ffffffffb14fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.081023] 4 locks held by python3/3422:
[ 1236.086319] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.096205] #1: ff11000135ffbe80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.107166] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.118320] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.128414] 4 locks held by python3/5421:
[ 1236.133709] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.143598] #1: ff11000185161480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.154549] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.165707] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.175787]
[ 1236.178252] =============================================\x0a
[ 1358.379326] INFO: task python3:5421 blocked for more than 491 seconds.
[ 1358.387479] Not tainted 7.1.0-rc7-tgje-g535aecdaa85c #1
[ 1358.394732] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.404322] task:python3 state:D stack:25672 pid:5421 tgid:5421 ppid:5420 task_flags:0x400100 flags:0x00080002
[ 1358.417528] Call Trace:
[ 1358.421069]
[ 1358.424219] __schedule+0xcc3/0x19c0
[ 1358.429035] ? io_schedule_timeout+0x140/0x140
[ 1358.434823] ? find_held_lock+0x2b/0x80
[ 1358.439928] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.445813] schedule+0xd8/0x260
[ 1358.450232] schedule_preempt_disabled+0x18/0x30
[ 1358.456211] __mutex_lock+0x6af/0x2010
[ 1358.461215] ? threaded_store+0x132/0x220
[ 1358.466514] ? ww_mutex_lock+0x160/0x160
[ 1358.471711] ? selinux_inode_free_security+0x330/0x330
[ 1358.478282] ? cap_capable+0x1d6/0x3e0
[ 1358.483291] ? threaded_store+0x132/0x220
[ 1358.488588] threaded_store+0x132/0x220
[ 1358.493690] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.498981] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.504279] ? sysfs_kf_write+0x65/0x160
[ 1358.509478] ? sysfs_file_ops+0x140/0x140
[ 1358.514773] kernfs_fop_write_iter+0x335/0x510
[ 1358.520557] ? kernfs_ops+0xf0/0xf0
[ 1358.525270] new_sync_write+0x333/0x750
[ 1358.530376] ? __lock_acquire+0x508/0xc10
[ 1358.535674] ? new_sync_read+0x740/0x740
[ 1358.540866] ? lock_acquire.part.0+0xbc/0x260
[ 1358.546554] ? ksys_write+0x116/0x250
[ 1358.551465] vfs_write+0x657/0x9b0
[ 1358.556082] ksys_write+0x116/0x250
[ 1358.560792] ? __ia32_sys_read+0xc0/0xc0
[ 1358.565991] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.571580] ? rcu_is_watching+0x15/0xd0
[ 1358.576774] do_syscall_64+0xf3/0x650
[ 1358.581681] ? trace_hardirqs_off+0xd/0x30
[ 1358.587069] ? exc_page_fault+0xda/0xf0
[ 1358.592176] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.598645] RIP: 0033:0x7f3cbef1264e
[ 1358.603458] RSP: 002b:00007ffd12a84660 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.612755] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3cbef1264e
[ 1358.621563] RDX: 0000000000000001 RSI: 0000560488afdbf0 RDI: 0000000000000007
[ 1358.630361] RBP: 00007ffd12a84670 R08: 0000000000000000 R09: 0000000000000000
[ 1358.639174] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3cbed8b390
[ 1358.647983] R13: 0000560488afdbf0 R14: 0000000000000007 R15: 00007f3cbed21120
[ 1358.656796]
[ 1358.660080] INFO: task python3:5421 is blocked on a mutex likely owned by task python3:3422.
[ 1358.670353] task:python3 state:D stack:25672 pid:3422 tgid:3422 ppid:3421 task_flags:0x400100 flags:0x00080802
[ 1358.683557] Call Trace:
[ 1358.687097]
[ 1358.690250] __schedule+0xcc3/0x19c0
[ 1358.695060] ? io_schedule_timeout+0x140/0x140
[ 1358.700846] ? find_held_lock+0x2b/0x80
[ 1358.705949] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.712128] schedule+0xd8/0x260
[ 1358.716547] schedule_timeout+0x13f/0x290
[ 1358.721838] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.728209] ? trace_contention_end+0xb3/0x1a0
[ 1358.733994] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.739683] ? ww_mutex_lock+0x160/0x160
[ 1358.744880] msleep+0x1b/0x30
[ 1358.749008] napi_set_threaded+0x3a5/0x440
[ 1358.754403] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.759900] netif_set_threaded+0x257/0x400
[ 1358.765396] threaded_store+0x1b8/0x220
[ 1358.770498] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.775794] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.781090] ? sysfs_kf_write+0x65/0x160
[ 1358.786291] ? sysfs_file_ops+0x140/0x140
[ 1358.791585] kernfs_fop_write_iter+0x335/0x510
[ 1358.797370] ? kernfs_ops+0xf0/0xf0
[ 1358.802083] new_sync_write+0x333/0x750
[ 1358.807178] ? __lock_acquire+0x508/0xc10
[ 1358.812476] ? new_sync_read+0x740/0x740
[ 1358.817667] ? lock_acquire.part.0+0xbc/0x260
[ 1358.823351] ? ksys_write+0x116/0x250
[ 1358.828261] vfs_write+0x657/0x9b0
[ 1358.832876] ksys_write+0x116/0x250
[ 1358.837583] ? __ia32_sys_read+0xc0/0xc0
[ 1358.842780] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.848370] ? rcu_is_watching+0x15/0xd0
[ 1358.853561] do_syscall_64+0xf3/0x650
[ 1358.858459] ? trace_hardirqs_off+0xd/0x30
[ 1358.863853] ? exc_page_fault+0xda/0xf0
[ 1358.868955] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.875421] RIP: 0033:0x7ffae976e64e
[ 1358.880229] RSP: 002b:00007ffc63eafb90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.889526] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ffae976e64e
[ 1358.898328] RDX: 0000000000000001 RSI: 000055c27bf86bf0 RDI: 0000000000000007
[ 1358.907138] RBP: 00007ffc63eafba0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.915949] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffae95e7390
[ 1358.924756] R13: 000055c27bf86bf0 R14: 0000000000000007 R15: 00007ffae957d120
[ 1358.933570]
[ 1358.936840] \x0aShowing all locks held in the system:
[ 1358.945380] 1 lock held by khungtaskd/185:
[ 1358.950775] #0: ffffffffb14fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.962257] 4 locks held by python3/3422:
[ 1358.967552] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.977444] #1: ff11000135ffbe80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.988405] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.999562] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.009657] 4 locks held by python3/5421:
[ 1359.014952] #0: ff1100013047c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.024839] #1: ff11000185161480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.035800] #2: ff11000123cdcb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.046953] #3: ff11000123c91e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.057047]
[ 1359.059512] =============================================\x0a