[ 788.528168] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 788.530624] nipa-hw-worker: [5/43] Retrying drivers/net:napi_threaded.py
[ 788.689238] kselftest: Running tests in drivers/net
[ 989.718402] INFO: task python3:5889 blocked for more than 122 seconds.
[ 989.726622] Not tainted 7.1.0-rc1-kfng-gabaddbf07c5d #1
[ 989.734344] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.743932] task:python3 state:D stack:25640 pid:5889 tgid:5889 ppid:5888 task_flags:0x400100 flags:0x00080000
[ 989.757135] Call Trace:
[ 989.760674]
[ 989.763825] __schedule+0xcba/0x1940
[ 989.768640] ? io_schedule_timeout+0x140/0x140
[ 989.774426] ? find_held_lock+0x2b/0x80
[ 989.779528] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.785414] schedule+0xd4/0x260
[ 989.789831] schedule_preempt_disabled+0x18/0x30
[ 989.795807] __mutex_lock+0x6a3/0x2000
[ 989.800812] ? threaded_store+0x132/0x220
[ 989.806110] ? ww_mutex_lock+0x160/0x160
[ 989.811308] ? selinux_inode_free_security+0x330/0x330
[ 989.817869] ? cap_capable+0x1d6/0x3e0
[ 989.822874] ? threaded_store+0x132/0x220
[ 989.828169] threaded_store+0x132/0x220
[ 989.833269] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.838566] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.843865] ? sysfs_kf_write+0x65/0x160
[ 989.849064] ? sysfs_file_ops+0x140/0x140
[ 989.854360] kernfs_fop_write_iter+0x335/0x510
[ 989.860143] ? kernfs_ops+0xf0/0xf0
[ 989.864854] new_sync_write+0x333/0x750
[ 989.869955] ? __lock_acquire+0x508/0xc10
[ 989.875251] ? new_sync_read+0x740/0x740
[ 989.880450] ? lock_acquire.part.0+0xbc/0x260
[ 989.886137] ? ksys_write+0x116/0x250
[ 989.891047] vfs_write+0x657/0x9b0
[ 989.895660] ksys_write+0x116/0x250
[ 989.900369] ? __ia32_sys_read+0xc0/0xc0
[ 989.905569] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.911157] ? rcu_is_watching+0x15/0xd0
[ 989.916356] do_syscall_64+0xf3/0x650
[ 989.921264] ? irq_exit_rcu+0x1a/0x30
[ 989.926172] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.932637] RIP: 0033:0x7f832597b22e
[ 989.937445] RSP: 002b:00007ffcb9313f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.946740] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f832597b22e
[ 989.955547] RDX: 0000000000000001 RSI: 0000559f91faba10 RDI: 0000000000000007
[ 989.964354] RBP: 00007ffcb9313f90 R08: 0000000000000000 R09: 0000000000000000
[ 989.973159] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f83257f3390
[ 989.981966] R13: 0000559f91faba10 R14: 0000000000000007 R15: 0000000000000003
[ 989.990777]
[ 989.994091] INFO: task python3:5889 is blocked on a mutex likely owned by task python3:3799.
[ 990.004362] task:python3 state:D stack:25672 pid:3799 tgid:3799 ppid:3798 task_flags:0x400100 flags:0x00080802
[ 990.017562] Call Trace:
[ 990.021101]
[ 990.024250] __schedule+0xcba/0x1940
[ 990.029058] ? io_schedule_timeout+0x140/0x140
[ 990.034843] ? find_held_lock+0x2b/0x80
[ 990.039934] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.046112] schedule+0xd4/0x260
[ 990.050527] schedule_timeout+0x13f/0x290
[ 990.055823] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.062189] ? trace_contention_end+0xb3/0x1a0
[ 990.067973] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.073662] ? ww_mutex_lock+0x160/0x160
[ 990.078854] msleep+0x1b/0x30
[ 990.082984] napi_set_threaded+0x3a5/0x440
[ 990.088377] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.093872] netif_set_threaded+0x257/0x400
[ 990.099365] threaded_store+0x1b8/0x220
[ 990.104465] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.109760] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.115054] ? sysfs_kf_write+0x65/0x160
[ 990.120253] ? sysfs_file_ops+0x140/0x140
[ 990.125546] kernfs_fop_write_iter+0x335/0x510
[ 990.131329] ? kernfs_ops+0xf0/0xf0
[ 990.136038] new_sync_write+0x333/0x750
[ 990.141138] ? __lock_acquire+0x508/0xc10
[ 990.146434] ? new_sync_read+0x740/0x740
[ 990.151632] ? lock_acquire.part.0+0xbc/0x260
[ 990.157315] ? ksys_write+0x116/0x250
[ 990.162225] vfs_write+0x657/0x9b0
[ 990.166838] ksys_write+0x116/0x250
[ 990.171546] ? __ia32_sys_read+0xc0/0xc0
[ 990.176746] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.182334] ? rcu_is_watching+0x15/0xd0
[ 990.187530] do_syscall_64+0xf3/0x650
[ 990.192436] ? irq_exit_rcu+0x1a/0x30
[ 990.197341] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.203808] RIP: 0033:0x7f810554f22e
[ 990.208614] RSP: 002b:00007ffc60aa3ac0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.217908] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f810554f22e
[ 990.226716] RDX: 0000000000000001 RSI: 00005597fa403a10 RDI: 0000000000000007
[ 990.235514] RBP: 00007ffc60aa3ad0 R08: 0000000000000000 R09: 0000000000000000
[ 990.244323] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f81053c7390
[ 990.253133] R13: 00005597fa403a10 R14: 0000000000000007 R15: 0000000000000003
[ 990.261946]
[ 990.265204] \x0aShowing all locks held in the system:
[ 990.273744] 1 lock held by khungtaskd/184:
[ 990.279137] #0: ffffffff8f2fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.290611] 2 locks held by systemd-journal/800:
[ 990.296589] #0: ff1100180de372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.306474] #1: ffffffff8f5030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.319301] 4 locks held by python3/3799:
[ 990.324593] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.334479] #1: ff1100016af6ee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.345440] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.356586] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.366696] 4 locks held by python3/5889:
[ 990.372001] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.381889] #1: ff110001e60c9a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.392850] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.403994] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.414081]
[ 990.416544] =============================================\x0a
[ 1112.600602] INFO: task python3:5889 blocked for more than 245 seconds.
[ 1112.608749] Not tainted 7.1.0-rc1-kfng-gabaddbf07c5d #1
[ 1112.616000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.625591] task:python3 state:D stack:25640 pid:5889 tgid:5889 ppid:5888 task_flags:0x400100 flags:0x00080000
[ 1112.638798] Call Trace:
[ 1112.642338]
[ 1112.645490] __schedule+0xcba/0x1940
[ 1112.650307] ? io_schedule_timeout+0x140/0x140
[ 1112.656093] ? find_held_lock+0x2b/0x80
[ 1112.661199] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.667087] schedule+0xd4/0x260
[ 1112.671503] schedule_preempt_disabled+0x18/0x30
[ 1112.677482] __mutex_lock+0x6a3/0x2000
[ 1112.682484] ? threaded_store+0x132/0x220
[ 1112.687785] ? ww_mutex_lock+0x160/0x160
[ 1112.692985] ? selinux_inode_free_security+0x330/0x330
[ 1112.699552] ? cap_capable+0x1d6/0x3e0
[ 1112.704560] ? threaded_store+0x132/0x220
[ 1112.709858] threaded_store+0x132/0x220
[ 1112.714957] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.720257] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.725556] ? sysfs_kf_write+0x65/0x160
[ 1112.730757] ? sysfs_file_ops+0x140/0x140
[ 1112.736042] kernfs_fop_write_iter+0x335/0x510
[ 1112.741824] ? kernfs_ops+0xf0/0xf0
[ 1112.746533] new_sync_write+0x333/0x750
[ 1112.751636] ? __lock_acquire+0x508/0xc10
[ 1112.756930] ? new_sync_read+0x740/0x740
[ 1112.762132] ? lock_acquire.part.0+0xbc/0x260
[ 1112.767817] ? ksys_write+0x116/0x250
[ 1112.772725] vfs_write+0x657/0x9b0
[ 1112.777328] ksys_write+0x116/0x250
[ 1112.782039] ? __ia32_sys_read+0xc0/0xc0
[ 1112.787239] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.792828] ? rcu_is_watching+0x15/0xd0
[ 1112.798027] do_syscall_64+0xf3/0x650
[ 1112.802932] ? irq_exit_rcu+0x1a/0x30
[ 1112.807837] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.814302] RIP: 0033:0x7f832597b22e
[ 1112.819112] RSP: 002b:00007ffcb9313f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.828409] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f832597b22e
[ 1112.837215] RDX: 0000000000000001 RSI: 0000559f91faba10 RDI: 0000000000000007
[ 1112.846024] RBP: 00007ffcb9313f90 R08: 0000000000000000 R09: 0000000000000000
[ 1112.854828] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f83257f3390
[ 1112.863633] R13: 0000559f91faba10 R14: 0000000000000007 R15: 0000000000000003
[ 1112.872452]
[ 1112.875732] INFO: task python3:5889 is blocked on a mutex likely owned by task python3:3799.
[ 1112.886002] task:python3 state:D stack:25672 pid:3799 tgid:3799 ppid:3798 task_flags:0x400100 flags:0x00080802
[ 1112.899203] Call Trace:
[ 1112.902743]
[ 1112.905891] __schedule+0xcba/0x1940
[ 1112.910704] ? io_schedule_timeout+0x140/0x140
[ 1112.916487] ? find_held_lock+0x2b/0x80
[ 1112.921587] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.927766] schedule+0xd4/0x260
[ 1112.932183] schedule_timeout+0x13f/0x290
[ 1112.937478] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.943846] ? trace_contention_end+0xb3/0x1a0
[ 1112.949632] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.955319] ? ww_mutex_lock+0x160/0x160
[ 1112.960518] msleep+0x1b/0x30
[ 1112.964641] napi_set_threaded+0x3a5/0x440
[ 1112.970034] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.975527] netif_set_threaded+0x257/0x400
[ 1112.981021] threaded_store+0x1b8/0x220
[ 1112.986124] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.991419] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.996713] ? sysfs_kf_write+0x65/0x160
[ 1113.001914] ? sysfs_file_ops+0x140/0x140
[ 1113.007210] kernfs_fop_write_iter+0x335/0x510
[ 1113.012991] ? kernfs_ops+0xf0/0xf0
[ 1113.017700] new_sync_write+0x333/0x750
[ 1113.022802] ? __lock_acquire+0x508/0xc10
[ 1113.028096] ? new_sync_read+0x740/0x740
[ 1113.033298] ? lock_acquire.part.0+0xbc/0x260
[ 1113.038984] ? ksys_write+0x116/0x250
[ 1113.043893] vfs_write+0x657/0x9b0
[ 1113.048505] ksys_write+0x116/0x250
[ 1113.053214] ? __ia32_sys_read+0xc0/0xc0
[ 1113.058415] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.064001] ? rcu_is_watching+0x15/0xd0
[ 1113.069199] do_syscall_64+0xf3/0x650
[ 1113.074102] ? irq_exit_rcu+0x1a/0x30
[ 1113.079009] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.085473] RIP: 0033:0x7f810554f22e
[ 1113.090279] RSP: 002b:00007ffc60aa3ac0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.099575] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f810554f22e
[ 1113.108382] RDX: 0000000000000001 RSI: 00005597fa403a10 RDI: 0000000000000007
[ 1113.117187] RBP: 00007ffc60aa3ad0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.125996] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f81053c7390
[ 1113.134801] R13: 00005597fa403a10 R14: 0000000000000007 R15: 0000000000000003
[ 1113.143612]
[ 1113.146870] \x0aShowing all locks held in the system:
[ 1113.155409] 1 lock held by khungtaskd/184:
[ 1113.160802] #0: ffffffff8f2fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.172285] 4 locks held by python3/3799:
[ 1113.177577] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.187466] #1: ff1100016af6ee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.198429] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.209590] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.219686] 4 locks held by python3/5889:
[ 1113.224981] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.234866] #1: ff110001e60c9a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.245825] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.256981] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.267070]
[ 1113.269536] =============================================\x0a
[ 1235.481857] INFO: task python3:5889 blocked for more than 368 seconds.
[ 1235.490005] Not tainted 7.1.0-rc1-kfng-gabaddbf07c5d #1
[ 1235.497248] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.506836] task:python3 state:D stack:25640 pid:5889 tgid:5889 ppid:5888 task_flags:0x400100 flags:0x00080002
[ 1235.520043] Call Trace:
[ 1235.523581]
[ 1235.526731] __schedule+0xcba/0x1940
[ 1235.531547] ? io_schedule_timeout+0x140/0x140
[ 1235.537334] ? find_held_lock+0x2b/0x80
[ 1235.542438] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.548322] schedule+0xd4/0x260
[ 1235.552739] schedule_preempt_disabled+0x18/0x30
[ 1235.558719] __mutex_lock+0x6a3/0x2000
[ 1235.563721] ? threaded_store+0x132/0x220
[ 1235.569018] ? ww_mutex_lock+0x160/0x160
[ 1235.574216] ? selinux_inode_free_security+0x330/0x330
[ 1235.580785] ? cap_capable+0x1d6/0x3e0
[ 1235.585794] ? threaded_store+0x132/0x220
[ 1235.591092] threaded_store+0x132/0x220
[ 1235.596182] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.601479] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.606774] ? sysfs_kf_write+0x65/0x160
[ 1235.611976] ? sysfs_file_ops+0x140/0x140
[ 1235.617271] kernfs_fop_write_iter+0x335/0x510
[ 1235.623053] ? kernfs_ops+0xf0/0xf0
[ 1235.627764] new_sync_write+0x333/0x750
[ 1235.632858] ? __lock_acquire+0x508/0xc10
[ 1235.638153] ? new_sync_read+0x740/0x740
[ 1235.643355] ? lock_acquire.part.0+0xbc/0x260
[ 1235.649040] ? ksys_write+0x116/0x250
[ 1235.653950] vfs_write+0x657/0x9b0
[ 1235.658565] ksys_write+0x116/0x250
[ 1235.663277] ? __ia32_sys_read+0xc0/0xc0
[ 1235.668473] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.674062] ? rcu_is_watching+0x15/0xd0
[ 1235.679254] do_syscall_64+0xf3/0x650
[ 1235.684158] ? irq_exit_rcu+0x1a/0x30
[ 1235.689066] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.695526] RIP: 0033:0x7f832597b22e
[ 1235.700337] RSP: 002b:00007ffcb9313f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.709635] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f832597b22e
[ 1235.718441] RDX: 0000000000000001 RSI: 0000559f91faba10 RDI: 0000000000000007
[ 1235.727245] RBP: 00007ffcb9313f90 R08: 0000000000000000 R09: 0000000000000000
[ 1235.736052] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f83257f3390
[ 1235.744857] R13: 0000559f91faba10 R14: 0000000000000007 R15: 0000000000000003
[ 1235.753669]
[ 1235.756936] INFO: task python3:5889 is blocked on a mutex likely owned by task python3:3799.
[ 1235.767205] task:python3 state:D stack:25672 pid:3799 tgid:3799 ppid:3798 task_flags:0x400100 flags:0x00080802
[ 1235.780405] Call Trace:
[ 1235.783935]
[ 1235.787084] __schedule+0xcba/0x1940
[ 1235.791892] ? io_schedule_timeout+0x140/0x140
[ 1235.797674] ? find_held_lock+0x2b/0x80
[ 1235.802774] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.808951] schedule+0xd4/0x260
[ 1235.813369] schedule_timeout+0x13f/0x290
[ 1235.818667] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.825033] ? trace_contention_end+0xb3/0x1a0
[ 1235.830817] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.836506] ? ww_mutex_lock+0x160/0x160
[ 1235.841704] msleep+0x1b/0x30
[ 1235.845831] napi_set_threaded+0x3a5/0x440
[ 1235.851225] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.856720] netif_set_threaded+0x257/0x400
[ 1235.862211] threaded_store+0x1b8/0x220
[ 1235.867313] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.872607] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.877903] ? sysfs_kf_write+0x65/0x160
[ 1235.883099] ? sysfs_file_ops+0x140/0x140
[ 1235.888395] kernfs_fop_write_iter+0x335/0x510
[ 1235.894179] ? kernfs_ops+0xf0/0xf0
[ 1235.898887] new_sync_write+0x333/0x750
[ 1235.903986] ? __lock_acquire+0x508/0xc10
[ 1235.909282] ? new_sync_read+0x740/0x740
[ 1235.914480] ? lock_acquire.part.0+0xbc/0x260
[ 1235.920163] ? ksys_write+0x116/0x250
[ 1235.925075] vfs_write+0x657/0x9b0
[ 1235.929689] ksys_write+0x116/0x250
[ 1235.934399] ? __ia32_sys_read+0xc0/0xc0
[ 1235.939599] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.945186] ? rcu_is_watching+0x15/0xd0
[ 1235.950385] do_syscall_64+0xf3/0x650
[ 1235.955291] ? irq_exit_rcu+0x1a/0x30
[ 1235.960197] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.966663] RIP: 0033:0x7f810554f22e
[ 1235.971467] RSP: 002b:00007ffc60aa3ac0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.980761] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f810554f22e
[ 1235.989566] RDX: 0000000000000001 RSI: 00005597fa403a10 RDI: 0000000000000007
[ 1235.998371] RBP: 00007ffc60aa3ad0 R08: 0000000000000000 R09: 0000000000000000
[ 1236.007179] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f81053c7390
[ 1236.015976] R13: 00005597fa403a10 R14: 0000000000000007 R15: 0000000000000003
[ 1236.024788]
[ 1236.028050] \x0aShowing all locks held in the system:
[ 1236.036585] 1 lock held by khungtaskd/184:
[ 1236.041979] #0: ffffffff8f2fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.053459] 4 locks held by python3/3799:
[ 1236.058752] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.068638] #1: ff1100016af6ee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.079601] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.090754] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.100844] 4 locks held by python3/5889:
[ 1236.106139] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.116028] #1: ff110001e60c9a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.126988] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.138145] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.148229]
[ 1236.150692] =============================================\x0a
[ 1268.249169] kworker/u130:2 (327) used greatest stack depth: 21456 bytes left
[ 1358.362157] INFO: task python3:5889 blocked for more than 491 seconds.
[ 1358.370305] Not tainted 7.1.0-rc1-kfng-gabaddbf07c5d #1
[ 1358.377553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.387142] task:python3 state:D stack:25640 pid:5889 tgid:5889 ppid:5888 task_flags:0x400100 flags:0x00080002
[ 1358.400343] Call Trace:
[ 1358.403884]
[ 1358.407034] __schedule+0xcba/0x1940
[ 1358.411854] ? io_schedule_timeout+0x140/0x140
[ 1358.417640] ? find_held_lock+0x2b/0x80
[ 1358.422744] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.428629] schedule+0xd4/0x260
[ 1358.433051] schedule_preempt_disabled+0x18/0x30
[ 1358.439028] __mutex_lock+0x6a3/0x2000
[ 1358.444033] ? threaded_store+0x132/0x220
[ 1358.449333] ? ww_mutex_lock+0x160/0x160
[ 1358.454524] ? selinux_inode_free_security+0x330/0x330
[ 1358.461094] ? cap_capable+0x1d6/0x3e0
[ 1358.466102] ? threaded_store+0x132/0x220
[ 1358.471400] threaded_store+0x132/0x220
[ 1358.476492] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.481790] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.487089] ? sysfs_kf_write+0x65/0x160
[ 1358.492289] ? sysfs_file_ops+0x140/0x140
[ 1358.497586] kernfs_fop_write_iter+0x335/0x510
[ 1358.503372] ? kernfs_ops+0xf0/0xf0
[ 1358.508085] new_sync_write+0x333/0x750
[ 1358.513187] ? __lock_acquire+0x508/0xc10
[ 1358.518482] ? new_sync_read+0x740/0x740
[ 1358.523686] ? lock_acquire.part.0+0xbc/0x260
[ 1358.529374] ? ksys_write+0x116/0x250
[ 1358.534286] vfs_write+0x657/0x9b0
[ 1358.538903] ksys_write+0x116/0x250
[ 1358.543616] ? __ia32_sys_read+0xc0/0xc0
[ 1358.548808] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.554389] ? rcu_is_watching+0x15/0xd0
[ 1358.559594] do_syscall_64+0xf3/0x650
[ 1358.564502] ? irq_exit_rcu+0x1a/0x30
[ 1358.569401] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.575871] RIP: 0033:0x7f832597b22e
[ 1358.580680] RSP: 002b:00007ffcb9313f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.589979] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f832597b22e
[ 1358.598790] RDX: 0000000000000001 RSI: 0000559f91faba10 RDI: 0000000000000007
[ 1358.607599] RBP: 00007ffcb9313f90 R08: 0000000000000000 R09: 0000000000000000
[ 1358.616409] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f83257f3390
[ 1358.625220] R13: 0000559f91faba10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.634033]
[ 1358.637301] INFO: task python3:5889 is blocked on a mutex likely owned by task python3:3799.
[ 1358.647575] task:python3 state:D stack:25672 pid:3799 tgid:3799 ppid:3798 task_flags:0x400100 flags:0x00080802
[ 1358.660777] Call Trace:
[ 1358.664318]
[ 1358.667467] __schedule+0xcba/0x1940
[ 1358.672278] ? io_schedule_timeout+0x140/0x140
[ 1358.678063] ? find_held_lock+0x2b/0x80
[ 1358.683159] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.689339] schedule+0xd4/0x260
[ 1358.693759] schedule_timeout+0x13f/0x290
[ 1358.699056] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.705419] ? trace_contention_end+0xb3/0x1a0
[ 1358.711203] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.716895] ? ww_mutex_lock+0x160/0x160
[ 1358.722095] msleep+0x1b/0x30
[ 1358.726211] napi_set_threaded+0x3a5/0x440
[ 1358.731605] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.737099] netif_set_threaded+0x257/0x400
[ 1358.742594] threaded_store+0x1b8/0x220
[ 1358.747696] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.752999] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.758295] ? sysfs_kf_write+0x65/0x160
[ 1358.763495] ? sysfs_file_ops+0x140/0x140
[ 1358.768794] kernfs_fop_write_iter+0x335/0x510
[ 1358.774576] ? kernfs_ops+0xf0/0xf0
[ 1358.779287] new_sync_write+0x333/0x750
[ 1358.784388] ? __lock_acquire+0x508/0xc10
[ 1358.789683] ? new_sync_read+0x740/0x740
[ 1358.794885] ? lock_acquire.part.0+0xbc/0x260
[ 1358.800571] ? ksys_write+0x116/0x250
[ 1358.805483] vfs_write+0x657/0x9b0
[ 1358.810101] ksys_write+0x116/0x250
[ 1358.814815] ? __ia32_sys_read+0xc0/0xc0
[ 1358.820014] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.825605] ? rcu_is_watching+0x15/0xd0
[ 1358.830804] do_syscall_64+0xf3/0x650
[ 1358.835709] ? irq_exit_rcu+0x1a/0x30
[ 1358.840615] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.847085] RIP: 0033:0x7f810554f22e
[ 1358.851895] RSP: 002b:00007ffc60aa3ac0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.861192] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f810554f22e
[ 1358.869989] RDX: 0000000000000001 RSI: 00005597fa403a10 RDI: 0000000000000007
[ 1358.878797] RBP: 00007ffc60aa3ad0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.887608] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f81053c7390
[ 1358.896416] R13: 00005597fa403a10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.905228]
[ 1358.908495] \x0aShowing all locks held in the system:
[ 1358.917033] 1 lock held by khungtaskd/184:
[ 1358.922428] #0: ffffffff8f2fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.933909] 4 locks held by python3/3799:
[ 1358.939204] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.949094] #1: ff1100016af6ee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.960047] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.971197] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.981292] 4 locks held by python3/5889:
[ 1358.986588] #0: ff1100011e7643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.996479] #1: ff110001e60c9a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.007443] #2: ff11000121e8cb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.018599] #3: ff11000121f01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.028686]
[ 1359.031152] =============================================\x0a