[ 787.363050] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 787.366041] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 787.530576] kselftest: Running tests in drivers/net
[ 989.684909] INFO: task python3:5822 blocked for more than 122 seconds.
[ 989.693124] Not tainted 7.1.0-rc6-vred-g5155d9e2d4ac #1
[ 989.700379] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.709972] task:python3 state:D stack:25672 pid:5822 tgid:5822 ppid:5821 task_flags:0x400100 flags:0x00080000
[ 989.723617] Call Trace:
[ 989.727157]
[ 989.730308] __schedule+0xcc3/0x19c0
[ 989.735122] ? io_schedule_timeout+0x140/0x140
[ 989.740898] ? find_held_lock+0x2b/0x80
[ 989.745995] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.751878] schedule+0xd8/0x260
[ 989.756288] schedule_preempt_disabled+0x18/0x30
[ 989.762267] __mutex_lock+0x6af/0x2010
[ 989.767271] ? threaded_store+0x132/0x220
[ 989.772573] ? ww_mutex_lock+0x160/0x160
[ 989.777772] ? selinux_inode_free_security+0x330/0x330
[ 989.784345] ? cap_capable+0x1d6/0x3e0
[ 989.789353] ? threaded_store+0x132/0x220
[ 989.794651] threaded_store+0x132/0x220
[ 989.799754] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.805047] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.810349] ? sysfs_kf_write+0x65/0x160
[ 989.815548] ? sysfs_file_ops+0x140/0x140
[ 989.820845] kernfs_fop_write_iter+0x335/0x510
[ 989.826631] ? kernfs_ops+0xf0/0xf0
[ 989.831345] new_sync_write+0x333/0x750
[ 989.836449] ? __lock_acquire+0x508/0xc10
[ 989.841751] ? new_sync_read+0x740/0x740
[ 989.846954] ? lock_acquire.part.0+0xbc/0x260
[ 989.852640] ? ksys_write+0x116/0x250
[ 989.857550] vfs_write+0x657/0x9b0
[ 989.862164] ksys_write+0x116/0x250
[ 989.866877] ? __ia32_sys_read+0xc0/0xc0
[ 989.872066] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.877659] ? rcu_is_watching+0x15/0xd0
[ 989.882861] do_syscall_64+0xf3/0x650
[ 989.887766] ? trace_hardirqs_off+0xd/0x30
[ 989.893163] ? exc_page_fault+0xda/0xf0
[ 989.898258] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.904733] RIP: 0033:0x7fb637b8b22e
[ 989.909543] RSP: 002b:00007ffc31dac050 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.918842] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb637b8b22e
[ 989.927649] RDX: 0000000000000001 RSI: 00005591dd5a1e90 RDI: 0000000000000007
[ 989.936460] RBP: 00007ffc31dac060 R08: 0000000000000000 R09: 0000000000000000
[ 989.945270] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb637a03390
[ 989.954079] R13: 00005591dd5a1e90 R14: 0000000000000007 R15: 0000000000000003
[ 989.962891]
[ 989.966183] INFO: task python3:5822 is blocked on a mutex likely owned by task python3:3836.
[ 989.976456] task:python3 state:D stack:25672 pid:3836 tgid:3836 ppid:3835 task_flags:0x400100 flags:0x00080802
[ 989.989660] Call Trace:
[ 989.993199]
[ 989.996348] __schedule+0xcc3/0x19c0
[ 990.001162] ? io_schedule_timeout+0x140/0x140
[ 990.006945] ? find_held_lock+0x2b/0x80
[ 990.012047] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.018224] schedule+0xd8/0x260
[ 990.022635] schedule_timeout+0x13f/0x290
[ 990.027931] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.034299] ? trace_contention_end+0xb3/0x1a0
[ 990.040087] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.045778] ? ww_mutex_lock+0x160/0x160
[ 990.050979] msleep+0x1b/0x30
[ 990.055106] napi_set_threaded+0x3a5/0x440
[ 990.060502] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.065998] netif_set_threaded+0x257/0x400
[ 990.071495] threaded_store+0x1b8/0x220
[ 990.076597] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.081885] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.087183] ? sysfs_kf_write+0x65/0x160
[ 990.092376] ? sysfs_file_ops+0x140/0x140
[ 990.097671] kernfs_fop_write_iter+0x335/0x510
[ 990.103455] ? kernfs_ops+0xf0/0xf0
[ 990.108166] new_sync_write+0x333/0x750
[ 990.113267] ? __lock_acquire+0x508/0xc10
[ 990.118564] ? new_sync_read+0x740/0x740
[ 990.123764] ? lock_acquire.part.0+0xbc/0x260
[ 990.129450] ? ksys_write+0x116/0x250
[ 990.134363] vfs_write+0x657/0x9b0
[ 990.138981] ksys_write+0x116/0x250
[ 990.143694] ? __ia32_sys_read+0xc0/0xc0
[ 990.148895] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.154483] ? rcu_is_watching+0x15/0xd0
[ 990.159686] do_syscall_64+0xf3/0x650
[ 990.164591] ? trace_hardirqs_off+0xd/0x30
[ 990.169986] ? exc_page_fault+0xda/0xf0
[ 990.175089] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.181556] RIP: 0033:0x7fb96019e22e
[ 990.186354] RSP: 002b:00007ffcf4228f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.195651] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb96019e22e
[ 990.204458] RDX: 0000000000000001 RSI: 00005618fbd5ee90 RDI: 0000000000000007
[ 990.213268] RBP: 00007ffcf4228f90 R08: 0000000000000000 R09: 0000000000000000
[ 990.222077] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb960016390
[ 990.230887] R13: 00005618fbd5ee90 R14: 0000000000000007 R15: 0000000000000003
[ 990.239701]
[ 990.242961] \x0aShowing all locks held in the system:
[ 990.251501] 1 lock held by khungtaskd/185:
[ 990.256897] #0: ffffffffba2fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.268372] 2 locks held by systemd-journal/803:
[ 990.274353] #0: ff1100180ddb74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.284240] #1: ffffffffba502ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.297067] 4 locks held by python3/3836:
[ 990.302364] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.312251] #1: ff11000142e8ca80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.323214] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.334374] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.344469] 4 locks held by python3/5822:
[ 990.349766] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.359656] #1: ff11000211f76880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.370620] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.381777] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.391858]
[ 990.394326] =============================================\x0a
[ 1112.566061] INFO: task python3:5822 blocked for more than 245 seconds.
[ 1112.574211] Not tainted 7.1.0-rc6-vred-g5155d9e2d4ac #1
[ 1112.581454] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.591045] task:python3 state:D stack:25672 pid:5822 tgid:5822 ppid:5821 task_flags:0x400100 flags:0x00080000
[ 1112.604252] Call Trace:
[ 1112.607793]
[ 1112.610942] __schedule+0xcc3/0x19c0
[ 1112.615761] ? io_schedule_timeout+0x140/0x140
[ 1112.621548] ? find_held_lock+0x2b/0x80
[ 1112.626652] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.632537] schedule+0xd8/0x260
[ 1112.636958] schedule_preempt_disabled+0x18/0x30
[ 1112.642939] __mutex_lock+0x6af/0x2010
[ 1112.647945] ? threaded_store+0x132/0x220
[ 1112.653245] ? ww_mutex_lock+0x160/0x160
[ 1112.658442] ? selinux_inode_free_security+0x330/0x330
[ 1112.665004] ? cap_capable+0x1d6/0x3e0
[ 1112.670014] ? threaded_store+0x132/0x220
[ 1112.675309] threaded_store+0x132/0x220
[ 1112.680415] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.685716] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.691017] ? sysfs_kf_write+0x65/0x160
[ 1112.696219] ? sysfs_file_ops+0x140/0x140
[ 1112.701513] kernfs_fop_write_iter+0x335/0x510
[ 1112.707299] ? kernfs_ops+0xf0/0xf0
[ 1112.712013] new_sync_write+0x333/0x750
[ 1112.717116] ? __lock_acquire+0x508/0xc10
[ 1112.722411] ? new_sync_read+0x740/0x740
[ 1112.727612] ? lock_acquire.part.0+0xbc/0x260
[ 1112.733292] ? ksys_write+0x116/0x250
[ 1112.738202] vfs_write+0x657/0x9b0
[ 1112.742819] ksys_write+0x116/0x250
[ 1112.747532] ? __ia32_sys_read+0xc0/0xc0
[ 1112.752724] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.758315] ? rcu_is_watching+0x15/0xd0
[ 1112.763517] do_syscall_64+0xf3/0x650
[ 1112.768423] ? trace_hardirqs_off+0xd/0x30
[ 1112.773820] ? exc_page_fault+0xda/0xf0
[ 1112.778916] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.785384] RIP: 0033:0x7fb637b8b22e
[ 1112.790193] RSP: 002b:00007ffc31dac050 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.799491] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb637b8b22e
[ 1112.808300] RDX: 0000000000000001 RSI: 00005591dd5a1e90 RDI: 0000000000000007
[ 1112.817109] RBP: 00007ffc31dac060 R08: 0000000000000000 R09: 0000000000000000
[ 1112.825919] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb637a03390
[ 1112.834728] R13: 00005591dd5a1e90 R14: 0000000000000007 R15: 0000000000000003
[ 1112.843541]
[ 1112.846815] INFO: task python3:5822 is blocked on a mutex likely owned by task python3:3836.
[ 1112.857089] task:python3 state:D stack:25672 pid:3836 tgid:3836 ppid:3835 task_flags:0x400100 flags:0x00080802
[ 1112.870291] Call Trace:
[ 1112.873832]
[ 1112.876981] __schedule+0xcc3/0x19c0
[ 1112.881794] ? io_schedule_timeout+0x140/0x140
[ 1112.887578] ? find_held_lock+0x2b/0x80
[ 1112.892682] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.898858] schedule+0xd8/0x260
[ 1112.903267] schedule_timeout+0x13f/0x290
[ 1112.908565] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.914932] ? trace_contention_end+0xb3/0x1a0
[ 1112.920719] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.926410] ? ww_mutex_lock+0x160/0x160
[ 1112.931609] msleep+0x1b/0x30
[ 1112.935734] napi_set_threaded+0x3a5/0x440
[ 1112.941131] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.946625] netif_set_threaded+0x257/0x400
[ 1112.952119] threaded_store+0x1b8/0x220
[ 1112.957219] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.962516] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.967807] ? sysfs_kf_write+0x65/0x160
[ 1112.973010] ? sysfs_file_ops+0x140/0x140
[ 1112.978307] kernfs_fop_write_iter+0x335/0x510
[ 1112.984094] ? kernfs_ops+0xf0/0xf0
[ 1112.988806] new_sync_write+0x333/0x750
[ 1112.993911] ? __lock_acquire+0x508/0xc10
[ 1112.999208] ? new_sync_read+0x740/0x740
[ 1113.004411] ? lock_acquire.part.0+0xbc/0x260
[ 1113.010100] ? ksys_write+0x116/0x250
[ 1113.015008] vfs_write+0x657/0x9b0
[ 1113.019625] ksys_write+0x116/0x250
[ 1113.024338] ? __ia32_sys_read+0xc0/0xc0
[ 1113.029526] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.035116] ? rcu_is_watching+0x15/0xd0
[ 1113.040317] do_syscall_64+0xf3/0x650
[ 1113.045223] ? trace_hardirqs_off+0xd/0x30
[ 1113.050620] ? exc_page_fault+0xda/0xf0
[ 1113.055725] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.062192] RIP: 0033:0x7fb96019e22e
[ 1113.066994] RSP: 002b:00007ffcf4228f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.076289] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb96019e22e
[ 1113.085098] RDX: 0000000000000001 RSI: 00005618fbd5ee90 RDI: 0000000000000007
[ 1113.093910] RBP: 00007ffcf4228f90 R08: 0000000000000000 R09: 0000000000000000
[ 1113.102719] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb960016390
[ 1113.111529] R13: 00005618fbd5ee90 R14: 0000000000000007 R15: 0000000000000003
[ 1113.120341]
[ 1113.123603] \x0aShowing all locks held in the system:
[ 1113.132141] 1 lock held by khungtaskd/185:
[ 1113.137536] #0: ffffffffba2fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.149020] 4 locks held by python3/3836:
[ 1113.154315] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.164204] #1: ff11000142e8ca80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.175168] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.186328] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.196420] 4 locks held by python3/5822:
[ 1113.201716] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.211605] #1: ff11000211f76880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.222557] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.233717] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.243810]
[ 1113.246277] =============================================\x0a
[ 1235.447164] INFO: task python3:5822 blocked for more than 368 seconds.
[ 1235.455311] Not tainted 7.1.0-rc6-vred-g5155d9e2d4ac #1
[ 1235.462559] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.472141] task:python3 state:D stack:25672 pid:5822 tgid:5822 ppid:5821 task_flags:0x400100 flags:0x00080002
[ 1235.485350] Call Trace:
[ 1235.488893]
[ 1235.492042] __schedule+0xcc3/0x19c0
[ 1235.496860] ? io_schedule_timeout+0x140/0x140
[ 1235.502644] ? find_held_lock+0x2b/0x80
[ 1235.507748] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.513633] schedule+0xd8/0x260
[ 1235.518055] schedule_preempt_disabled+0x18/0x30
[ 1235.524034] __mutex_lock+0x6af/0x2010
[ 1235.529031] ? threaded_store+0x132/0x220
[ 1235.534334] ? ww_mutex_lock+0x160/0x160
[ 1235.539533] ? selinux_inode_free_security+0x330/0x330
[ 1235.546106] ? cap_capable+0x1d6/0x3e0
[ 1235.551115] ? threaded_store+0x132/0x220
[ 1235.556412] threaded_store+0x132/0x220
[ 1235.561513] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.566811] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.572113] ? sysfs_kf_write+0x65/0x160
[ 1235.577314] ? sysfs_file_ops+0x140/0x140
[ 1235.582600] kernfs_fop_write_iter+0x335/0x510
[ 1235.588383] ? kernfs_ops+0xf0/0xf0
[ 1235.593097] new_sync_write+0x333/0x750
[ 1235.598202] ? __lock_acquire+0x508/0xc10
[ 1235.603488] ? new_sync_read+0x740/0x740
[ 1235.608692] ? lock_acquire.part.0+0xbc/0x260
[ 1235.614381] ? ksys_write+0x116/0x250
[ 1235.619290] vfs_write+0x657/0x9b0
[ 1235.623904] ksys_write+0x116/0x250
[ 1235.628614] ? __ia32_sys_read+0xc0/0xc0
[ 1235.633812] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.639400] ? rcu_is_watching+0x15/0xd0
[ 1235.644604] do_syscall_64+0xf3/0x650
[ 1235.649511] ? trace_hardirqs_off+0xd/0x30
[ 1235.654909] ? exc_page_fault+0xda/0xf0
[ 1235.660015] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.666483] RIP: 0033:0x7fb637b8b22e
[ 1235.671292] RSP: 002b:00007ffc31dac050 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.680590] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb637b8b22e
[ 1235.689399] RDX: 0000000000000001 RSI: 00005591dd5a1e90 RDI: 0000000000000007
[ 1235.698208] RBP: 00007ffc31dac060 R08: 0000000000000000 R09: 0000000000000000
[ 1235.707018] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb637a03390
[ 1235.715828] R13: 00005591dd5a1e90 R14: 0000000000000007 R15: 0000000000000003
[ 1235.724639]
[ 1235.727911] INFO: task python3:5822 is blocked on a mutex likely owned by task python3:3836.
[ 1235.738184] task:python3 state:D stack:25672 pid:3836 tgid:3836 ppid:3835 task_flags:0x400100 flags:0x00080802
[ 1235.751388] Call Trace:
[ 1235.754926]
[ 1235.758078] __schedule+0xcc3/0x19c0
[ 1235.762878] ? io_schedule_timeout+0x140/0x140
[ 1235.768664] ? find_held_lock+0x2b/0x80
[ 1235.773758] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.779937] schedule+0xd8/0x260
[ 1235.784353] schedule_timeout+0x13f/0x290
[ 1235.789651] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.796022] ? trace_contention_end+0xb3/0x1a0
[ 1235.801811] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.807502] ? ww_mutex_lock+0x160/0x160
[ 1235.812702] msleep+0x1b/0x30
[ 1235.816821] napi_set_threaded+0x3a5/0x440
[ 1235.822217] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.827713] netif_set_threaded+0x257/0x400
[ 1235.833209] threaded_store+0x1b8/0x220
[ 1235.838312] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.843612] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.848911] ? sysfs_kf_write+0x65/0x160
[ 1235.854114] ? sysfs_file_ops+0x140/0x140
[ 1235.859408] kernfs_fop_write_iter+0x335/0x510
[ 1235.865195] ? kernfs_ops+0xf0/0xf0
[ 1235.869908] new_sync_write+0x333/0x750
[ 1235.875016] ? __lock_acquire+0x508/0xc10
[ 1235.880313] ? new_sync_read+0x740/0x740
[ 1235.885504] ? lock_acquire.part.0+0xbc/0x260
[ 1235.891188] ? ksys_write+0x116/0x250
[ 1235.896097] vfs_write+0x657/0x9b0
[ 1235.900714] ksys_write+0x116/0x250
[ 1235.905426] ? __ia32_sys_read+0xc0/0xc0
[ 1235.910625] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.916213] ? rcu_is_watching+0x15/0xd0
[ 1235.921415] do_syscall_64+0xf3/0x650
[ 1235.926320] ? trace_hardirqs_off+0xd/0x30
[ 1235.931714] ? exc_page_fault+0xda/0xf0
[ 1235.936816] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.943284] RIP: 0033:0x7fb96019e22e
[ 1235.948094] RSP: 002b:00007ffcf4228f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.957381] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb96019e22e
[ 1235.966180] RDX: 0000000000000001 RSI: 00005618fbd5ee90 RDI: 0000000000000007
[ 1235.974989] RBP: 00007ffcf4228f90 R08: 0000000000000000 R09: 0000000000000000
[ 1235.983800] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb960016390
[ 1235.992608] R13: 00005618fbd5ee90 R14: 0000000000000007 R15: 0000000000000003
[ 1236.001412]
[ 1236.004676] \x0aShowing all locks held in the system:
[ 1236.013214] 1 lock held by khungtaskd/185:
[ 1236.018606] #0: ffffffffba2fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.030086] 4 locks held by python3/3836:
[ 1236.035380] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.045272] #1: ff11000142e8ca80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.056233] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.067381] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.077476] 4 locks held by python3/5822:
[ 1236.082772] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.092663] #1: ff11000211f76880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.103624] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.114781] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.124874]
[ 1236.127338] =============================================\x0a
[ 1358.329358] INFO: task python3:5822 blocked for more than 491 seconds.
[ 1358.337507] Not tainted 7.1.0-rc6-vred-g5155d9e2d4ac #1
[ 1358.344757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.354347] task:python3 state:D stack:25672 pid:5822 tgid:5822 ppid:5821 task_flags:0x400100 flags:0x00080002
[ 1358.367552] Call Trace:
[ 1358.371092]
[ 1358.374245] __schedule+0xcc3/0x19c0
[ 1358.379061] ? io_schedule_timeout+0x140/0x140
[ 1358.384850] ? find_held_lock+0x2b/0x80
[ 1358.389956] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.395843] schedule+0xd8/0x260
[ 1358.400263] schedule_preempt_disabled+0x18/0x30
[ 1358.406244] __mutex_lock+0x6af/0x2010
[ 1358.411238] ? threaded_store+0x132/0x220
[ 1358.416532] ? ww_mutex_lock+0x160/0x160
[ 1358.421731] ? selinux_inode_free_security+0x330/0x330
[ 1358.428301] ? cap_capable+0x1d6/0x3e0
[ 1358.433313] ? threaded_store+0x132/0x220
[ 1358.438612] threaded_store+0x132/0x220
[ 1358.443707] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.449004] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.454302] ? sysfs_kf_write+0x65/0x160
[ 1358.459502] ? sysfs_file_ops+0x140/0x140
[ 1358.464801] kernfs_fop_write_iter+0x335/0x510
[ 1358.470586] ? kernfs_ops+0xf0/0xf0
[ 1358.475291] new_sync_write+0x333/0x750
[ 1358.480396] ? __lock_acquire+0x508/0xc10
[ 1358.485692] ? new_sync_read+0x740/0x740
[ 1358.490896] ? lock_acquire.part.0+0xbc/0x260
[ 1358.496582] ? ksys_write+0x116/0x250
[ 1358.501492] vfs_write+0x657/0x9b0
[ 1358.506109] ksys_write+0x116/0x250
[ 1358.510819] ? __ia32_sys_read+0xc0/0xc0
[ 1358.516017] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.521609] ? rcu_is_watching+0x15/0xd0
[ 1358.526812] do_syscall_64+0xf3/0x650
[ 1358.531720] ? trace_hardirqs_off+0xd/0x30
[ 1358.537115] ? exc_page_fault+0xda/0xf0
[ 1358.542216] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.548686] RIP: 0033:0x7fb637b8b22e
[ 1358.553497] RSP: 002b:00007ffc31dac050 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.562797] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb637b8b22e
[ 1358.571606] RDX: 0000000000000001 RSI: 00005591dd5a1e90 RDI: 0000000000000007
[ 1358.580413] RBP: 00007ffc31dac060 R08: 0000000000000000 R09: 0000000000000000
[ 1358.589222] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb637a03390
[ 1358.598032] R13: 00005591dd5a1e90 R14: 0000000000000007 R15: 0000000000000003
[ 1358.606845]
[ 1358.610129] INFO: task python3:5822 is blocked on a mutex likely owned by task python3:3836.
[ 1358.620403] task:python3 state:D stack:25672 pid:3836 tgid:3836 ppid:3835 task_flags:0x400100 flags:0x00080802
[ 1358.633604] Call Trace:
[ 1358.637142]
[ 1358.640292] __schedule+0xcc3/0x19c0
[ 1358.645104] ? io_schedule_timeout+0x140/0x140
[ 1358.650888] ? find_held_lock+0x2b/0x80
[ 1358.655993] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.662172] schedule+0xd8/0x260
[ 1358.666591] schedule_timeout+0x13f/0x290
[ 1358.671892] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.678262] ? trace_contention_end+0xb3/0x1a0
[ 1358.684047] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.689737] ? ww_mutex_lock+0x160/0x160
[ 1358.694936] msleep+0x1b/0x30
[ 1358.699063] napi_set_threaded+0x3a5/0x440
[ 1358.704462] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.709957] netif_set_threaded+0x257/0x400
[ 1358.715453] threaded_store+0x1b8/0x220
[ 1358.720557] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.725856] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.731154] ? sysfs_kf_write+0x65/0x160
[ 1358.736356] ? sysfs_file_ops+0x140/0x140
[ 1358.741653] kernfs_fop_write_iter+0x335/0x510
[ 1358.747437] ? kernfs_ops+0xf0/0xf0
[ 1358.752147] new_sync_write+0x333/0x750
[ 1358.757250] ? __lock_acquire+0x508/0xc10
[ 1358.762537] ? new_sync_read+0x740/0x740
[ 1358.767737] ? lock_acquire.part.0+0xbc/0x260
[ 1358.773415] ? ksys_write+0x116/0x250
[ 1358.778325] vfs_write+0x657/0x9b0
[ 1358.782939] ksys_write+0x116/0x250
[ 1358.787653] ? __ia32_sys_read+0xc0/0xc0
[ 1358.792852] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.798442] ? rcu_is_watching+0x15/0xd0
[ 1358.803636] do_syscall_64+0xf3/0x650
[ 1358.808544] ? trace_hardirqs_off+0xd/0x30
[ 1358.813941] ? exc_page_fault+0xda/0xf0
[ 1358.819042] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.825510] RIP: 0033:0x7fb96019e22e
[ 1358.830311] RSP: 002b:00007ffcf4228f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.839607] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb96019e22e
[ 1358.848416] RDX: 0000000000000001 RSI: 00005618fbd5ee90 RDI: 0000000000000007
[ 1358.857224] RBP: 00007ffcf4228f90 R08: 0000000000000000 R09: 0000000000000000
[ 1358.866031] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb960016390
[ 1358.874832] R13: 00005618fbd5ee90 R14: 0000000000000007 R15: 0000000000000003
[ 1358.883645]
[ 1358.886914] \x0aShowing all locks held in the system:
[ 1358.895452] 1 lock held by khungtaskd/185:
[ 1358.900845] #0: ffffffffba2fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.912328] 4 locks held by python3/3836:
[ 1358.917625] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.927514] #1: ff11000142e8ca80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.938474] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.949631] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.959716] 4 locks held by python3/5822:
[ 1358.965011] #0: ff1100013080c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.974901] #1: ff11000211f76880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.985865] #2: ff11000124110b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.997024] #3: ff11000124189d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.007117]
[ 1359.009583] =============================================\x0a