[ 785.847278] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 785.849703] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 785.977641] kselftest: Running tests in drivers/net
[ 829.931128] kworker/u130:0 (178) used greatest stack depth: 21456 bytes left
[ 989.677783] INFO: task python3:5776 blocked for more than 122 seconds.
[ 989.685977] Not tainted 7.1.0-rc6-fxsj-gd1384a325c14 #1
[ 989.693455] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.703042] task:python3 state:D stack:25672 pid:5776 tgid:5776 ppid:5775 task_flags:0x400100 flags:0x00080000
[ 989.716247] Call Trace:
[ 989.719784]
[ 989.722934] __schedule+0xcc3/0x19c0
[ 989.727750] ? io_schedule_timeout+0x140/0x140
[ 989.733533] ? find_held_lock+0x2b/0x80
[ 989.738634] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.744521] schedule+0xd8/0x260
[ 989.748938] schedule_preempt_disabled+0x18/0x30
[ 989.754915] __mutex_lock+0x6af/0x2010
[ 989.759920] ? threaded_store+0x132/0x220
[ 989.765219] ? ww_mutex_lock+0x160/0x160
[ 989.770414] ? selinux_inode_free_security+0x330/0x330
[ 989.776985] ? cap_capable+0x1d6/0x3e0
[ 989.781991] ? threaded_store+0x132/0x220
[ 989.787278] threaded_store+0x132/0x220
[ 989.792381] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.797667] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.802962] ? sysfs_kf_write+0x65/0x160
[ 989.808160] ? sysfs_file_ops+0x140/0x140
[ 989.813454] kernfs_fop_write_iter+0x335/0x510
[ 989.819239] ? kernfs_ops+0xf0/0xf0
[ 989.823947] new_sync_write+0x333/0x750
[ 989.829049] ? __lock_acquire+0x508/0xc10
[ 989.834346] ? new_sync_read+0x740/0x740
[ 989.839537] ? lock_acquire.part.0+0xbc/0x260
[ 989.845221] ? ksys_write+0x116/0x250
[ 989.850132] vfs_write+0x657/0x9b0
[ 989.854747] ksys_write+0x116/0x250
[ 989.859448] ? __ia32_sys_read+0xc0/0xc0
[ 989.864647] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.870237] ? rcu_is_watching+0x15/0xd0
[ 989.875440] do_syscall_64+0xf3/0x650
[ 989.880346] ? trace_hardirqs_off+0xd/0x30
[ 989.885732] ? exc_page_fault+0xda/0xf0
[ 989.890832] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.897300] RIP: 0033:0x7f03f09ae22e
[ 989.902109] RSP: 002b:00007fff43769ea0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.911405] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03f09ae22e
[ 989.920212] RDX: 0000000000000001 RSI: 0000556d9e586e90 RDI: 0000000000000007
[ 989.929020] RBP: 00007fff43769eb0 R08: 0000000000000000 R09: 0000000000000000
[ 989.937829] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f03f0826390
[ 989.946635] R13: 0000556d9e586e90 R14: 0000000000000007 R15: 0000000000000003
[ 989.955455]
[ 989.958758] INFO: task python3:5776 is blocked on a mutex likely owned by task python3:3721.
[ 989.969031] task:python3 state:D stack:25568 pid:3721 tgid:3721 ppid:3720 task_flags:0x400100 flags:0x00080802
[ 989.982231] Call Trace:
[ 989.985758]
[ 989.988909] __schedule+0xcc3/0x19c0
[ 989.993710] ? io_schedule_timeout+0x140/0x140
[ 989.999493] ? find_held_lock+0x2b/0x80
[ 990.004595] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.010771] schedule+0xd8/0x260
[ 990.015188] schedule_timeout+0x13f/0x290
[ 990.020483] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.026851] ? trace_contention_end+0xb3/0x1a0
[ 990.032636] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.038327] ? ww_mutex_lock+0x160/0x160
[ 990.043526] msleep+0x1b/0x30
[ 990.047650] napi_set_threaded+0x3a5/0x440
[ 990.053043] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.058538] netif_set_threaded+0x257/0x400
[ 990.064030] threaded_store+0x1b8/0x220
[ 990.069125] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.074421] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.079718] ? sysfs_kf_write+0x65/0x160
[ 990.084917] ? sysfs_file_ops+0x140/0x140
[ 990.090213] kernfs_fop_write_iter+0x335/0x510
[ 990.095999] ? kernfs_ops+0xf0/0xf0
[ 990.100708] new_sync_write+0x333/0x750
[ 990.105810] ? __lock_acquire+0x508/0xc10
[ 990.111104] ? new_sync_read+0x740/0x740
[ 990.116304] ? lock_acquire.part.0+0xbc/0x260
[ 990.121988] ? ksys_write+0x116/0x250
[ 990.126896] vfs_write+0x657/0x9b0
[ 990.131511] ksys_write+0x116/0x250
[ 990.136214] ? __ia32_sys_read+0xc0/0xc0
[ 990.141409] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.146999] ? rcu_is_watching+0x15/0xd0
[ 990.152191] do_syscall_64+0xf3/0x650
[ 990.157098] ? trace_hardirqs_off+0xd/0x30
[ 990.162491] ? exc_page_fault+0xda/0xf0
[ 990.167593] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.174057] RIP: 0033:0x7fc21112d22e
[ 990.178866] RSP: 002b:00007fffc26ea640 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.188162] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc21112d22e
[ 990.196970] RDX: 0000000000000001 RSI: 00005614bd8dce90 RDI: 0000000000000007
[ 990.205771] RBP: 00007fffc26ea650 R08: 0000000000000000 R09: 0000000000000000
[ 990.214578] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc210fa5390
[ 990.223381] R13: 00005614bd8dce90 R14: 0000000000000007 R15: 0000000000000003
[ 990.232194]
[ 990.235451] \x0aShowing all locks held in the system:
[ 990.243989] 1 lock held by khungtaskd/185:
[ 990.249380] #0: ffffffff96cfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.260861] 4 locks held by python3/3721:
[ 990.266157] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.276044] #1: ff1100016022e280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.287005] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.298161] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.308260] 4 locks held by python3/5776:
[ 990.313556] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.323445] #1: ff110001e7bc3880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.334395] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.345551] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.355641]
[ 990.358096] =============================================\x0a
[ 1112.559069] INFO: task python3:5776 blocked for more than 245 seconds.
[ 1112.567216] Not tainted 7.1.0-rc6-fxsj-gd1384a325c14 #1
[ 1112.574466] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.584057] task:python3 state:D stack:25672 pid:5776 tgid:5776 ppid:5775 task_flags:0x400100 flags:0x00080000
[ 1112.597265] Call Trace:
[ 1112.600804]
[ 1112.603957] __schedule+0xcc3/0x19c0
[ 1112.608776] ? io_schedule_timeout+0x140/0x140
[ 1112.614563] ? find_held_lock+0x2b/0x80
[ 1112.619669] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.625557] schedule+0xd8/0x260
[ 1112.629978] schedule_preempt_disabled+0x18/0x30
[ 1112.635960] __mutex_lock+0x6af/0x2010
[ 1112.640967] ? threaded_store+0x132/0x220
[ 1112.646269] ? ww_mutex_lock+0x160/0x160
[ 1112.651467] ? selinux_inode_free_security+0x330/0x330
[ 1112.658036] ? cap_capable+0x1d6/0x3e0
[ 1112.663044] ? threaded_store+0x132/0x220
[ 1112.668341] threaded_store+0x132/0x220
[ 1112.673445] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.678745] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.684047] ? sysfs_kf_write+0x65/0x160
[ 1112.689248] ? sysfs_file_ops+0x140/0x140
[ 1112.694545] kernfs_fop_write_iter+0x335/0x510
[ 1112.700331] ? kernfs_ops+0xf0/0xf0
[ 1112.705042] new_sync_write+0x333/0x750
[ 1112.710146] ? __lock_acquire+0x508/0xc10
[ 1112.715437] ? new_sync_read+0x740/0x740
[ 1112.720639] ? lock_acquire.part.0+0xbc/0x260
[ 1112.726324] ? ksys_write+0x116/0x250
[ 1112.731236] vfs_write+0x657/0x9b0
[ 1112.735852] ksys_write+0x116/0x250
[ 1112.740563] ? __ia32_sys_read+0xc0/0xc0
[ 1112.745764] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.751354] ? rcu_is_watching+0x15/0xd0
[ 1112.756559] do_syscall_64+0xf3/0x650
[ 1112.761466] ? trace_hardirqs_off+0xd/0x30
[ 1112.766861] ? exc_page_fault+0xda/0xf0
[ 1112.771964] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.778434] RIP: 0033:0x7f03f09ae22e
[ 1112.783244] RSP: 002b:00007fff43769ea0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.792532] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03f09ae22e
[ 1112.801340] RDX: 0000000000000001 RSI: 0000556d9e586e90 RDI: 0000000000000007
[ 1112.810149] RBP: 00007fff43769eb0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.818958] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f03f0826390
[ 1112.827766] R13: 0000556d9e586e90 R14: 0000000000000007 R15: 0000000000000003
[ 1112.836578]
[ 1112.839848] INFO: task python3:5776 is blocked on a mutex likely owned by task python3:3721.
[ 1112.850122] task:python3 state:D stack:25568 pid:3721 tgid:3721 ppid:3720 task_flags:0x400100 flags:0x00080802
[ 1112.863324] Call Trace:
[ 1112.866862]
[ 1112.870015] __schedule+0xcc3/0x19c0
[ 1112.874826] ? io_schedule_timeout+0x140/0x140
[ 1112.880611] ? find_held_lock+0x2b/0x80
[ 1112.885715] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.891896] schedule+0xd8/0x260
[ 1112.896318] schedule_timeout+0x13f/0x290
[ 1112.901613] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.907982] ? trace_contention_end+0xb3/0x1a0
[ 1112.913769] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.919458] ? ww_mutex_lock+0x160/0x160
[ 1112.924657] msleep+0x1b/0x30
[ 1112.928783] napi_set_threaded+0x3a5/0x440
[ 1112.934172] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.939669] netif_set_threaded+0x257/0x400
[ 1112.945162] threaded_store+0x1b8/0x220
[ 1112.950268] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.955565] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.960855] ? sysfs_kf_write+0x65/0x160
[ 1112.966055] ? sysfs_file_ops+0x140/0x140
[ 1112.971342] kernfs_fop_write_iter+0x335/0x510
[ 1112.977129] ? kernfs_ops+0xf0/0xf0
[ 1112.981839] new_sync_write+0x333/0x750
[ 1112.986942] ? __lock_acquire+0x508/0xc10
[ 1112.992230] ? new_sync_read+0x740/0x740
[ 1112.997430] ? lock_acquire.part.0+0xbc/0x260
[ 1113.003107] ? ksys_write+0x116/0x250
[ 1113.008016] vfs_write+0x657/0x9b0
[ 1113.012641] ksys_write+0x116/0x250
[ 1113.017359] ? __ia32_sys_read+0xc0/0xc0
[ 1113.022562] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.028156] ? rcu_is_watching+0x15/0xd0
[ 1113.033362] do_syscall_64+0xf3/0x650
[ 1113.038275] ? trace_hardirqs_off+0xd/0x30
[ 1113.043671] ? exc_page_fault+0xda/0xf0
[ 1113.048776] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.055246] RIP: 0033:0x7fc21112d22e
[ 1113.060057] RSP: 002b:00007fffc26ea640 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.069361] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc21112d22e
[ 1113.078175] RDX: 0000000000000001 RSI: 00005614bd8dce90 RDI: 0000000000000007
[ 1113.086985] RBP: 00007fffc26ea650 R08: 0000000000000000 R09: 0000000000000000
[ 1113.095799] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc210fa5390
[ 1113.104612] R13: 00005614bd8dce90 R14: 0000000000000007 R15: 0000000000000003
[ 1113.113435]
[ 1113.116703] \x0aShowing all locks held in the system:
[ 1113.125253] 1 lock held by khungtaskd/185:
[ 1113.130650] #0: ffffffff96cfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.142137] 3 locks held by systemd-journal/804:
[ 1113.148121] #0: ff1100180dfb74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.158020] #1: ffffffff96f02ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: dput.part.0+0x2fd/0x410
[ 1113.170253] #2: ffffffff96cfdea0 (rcu_read_lock){....}-{1:3}, at: get_mem_cgroup_from_objcg+0x1a3/0x360
[ 1113.181719] 4 locks held by python3/3721:
[ 1113.187009] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.196910] #1: ff1100016022e280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.208007] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.219174] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.229347] 4 locks held by python3/5776:
[ 1113.234761] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.244649] #1: ff110001e7bc3880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.255604] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.266766] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.266782] 3 locks held by 9/6830:
[ 1113.266787]
[ 1113.284074] =============================================\x0a
[ 1235.439357] INFO: task python3:5776 blocked for more than 368 seconds.
[ 1235.447508] Not tainted 7.1.0-rc6-fxsj-gd1384a325c14 #1
[ 1235.454748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.464338] task:python3 state:D stack:25672 pid:5776 tgid:5776 ppid:5775 task_flags:0x400100 flags:0x00080002
[ 1235.477547] Call Trace:
[ 1235.481087]
[ 1235.484242] __schedule+0xcc3/0x19c0
[ 1235.489060] ? io_schedule_timeout+0x140/0x140
[ 1235.494846] ? find_held_lock+0x2b/0x80
[ 1235.499952] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.505837] schedule+0xd8/0x260
[ 1235.510258] schedule_preempt_disabled+0x18/0x30
[ 1235.516239] __mutex_lock+0x6af/0x2010
[ 1235.521242] ? threaded_store+0x132/0x220
[ 1235.526543] ? ww_mutex_lock+0x160/0x160
[ 1235.531742] ? selinux_inode_free_security+0x330/0x330
[ 1235.538310] ? cap_capable+0x1d6/0x3e0
[ 1235.543320] ? threaded_store+0x132/0x220
[ 1235.548615] threaded_store+0x132/0x220
[ 1235.553716] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.559013] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.564316] ? sysfs_kf_write+0x65/0x160
[ 1235.569519] ? sysfs_file_ops+0x140/0x140
[ 1235.574815] kernfs_fop_write_iter+0x335/0x510
[ 1235.580603] ? kernfs_ops+0xf0/0xf0
[ 1235.585314] new_sync_write+0x333/0x750
[ 1235.590419] ? __lock_acquire+0x508/0xc10
[ 1235.595709] ? new_sync_read+0x740/0x740
[ 1235.600912] ? lock_acquire.part.0+0xbc/0x260
[ 1235.606598] ? ksys_write+0x116/0x250
[ 1235.611506] vfs_write+0x657/0x9b0
[ 1235.616114] ksys_write+0x116/0x250
[ 1235.620825] ? __ia32_sys_read+0xc0/0xc0
[ 1235.626022] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.631613] ? rcu_is_watching+0x15/0xd0
[ 1235.636819] do_syscall_64+0xf3/0x650
[ 1235.641727] ? trace_hardirqs_off+0xd/0x30
[ 1235.647122] ? exc_page_fault+0xda/0xf0
[ 1235.652225] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.658695] RIP: 0033:0x7f03f09ae22e
[ 1235.663505] RSP: 002b:00007fff43769ea0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.672793] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03f09ae22e
[ 1235.681594] RDX: 0000000000000001 RSI: 0000556d9e586e90 RDI: 0000000000000007
[ 1235.690403] RBP: 00007fff43769eb0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.699212] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f03f0826390
[ 1235.708022] R13: 0000556d9e586e90 R14: 0000000000000007 R15: 0000000000000003
[ 1235.716836]
[ 1235.720134] INFO: task python3:5776 is blocked on a mutex likely owned by task python3:3721.
[ 1235.730408] task:python3 state:D stack:25568 pid:3721 tgid:3721 ppid:3720 task_flags:0x400100 flags:0x00080802
[ 1235.743610] Call Trace:
[ 1235.747152]
[ 1235.750304] __schedule+0xcc3/0x19c0
[ 1235.755116] ? io_schedule_timeout+0x140/0x140
[ 1235.760902] ? find_held_lock+0x2b/0x80
[ 1235.766006] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.772186] schedule+0xd8/0x260
[ 1235.776608] schedule_timeout+0x13f/0x290
[ 1235.781905] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.788273] ? trace_contention_end+0xb3/0x1a0
[ 1235.794060] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.799750] ? ww_mutex_lock+0x160/0x160
[ 1235.804948] msleep+0x1b/0x30
[ 1235.809073] napi_set_threaded+0x3a5/0x440
[ 1235.814471] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.819968] netif_set_threaded+0x257/0x400
[ 1235.825461] threaded_store+0x1b8/0x220
[ 1235.830566] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.835863] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.841161] ? sysfs_kf_write+0x65/0x160
[ 1235.846362] ? sysfs_file_ops+0x140/0x140
[ 1235.851658] kernfs_fop_write_iter+0x335/0x510
[ 1235.857444] ? kernfs_ops+0xf0/0xf0
[ 1235.862158] new_sync_write+0x333/0x750
[ 1235.867261] ? __lock_acquire+0x508/0xc10
[ 1235.872551] ? new_sync_read+0x740/0x740
[ 1235.877751] ? lock_acquire.part.0+0xbc/0x260
[ 1235.883436] ? ksys_write+0x116/0x250
[ 1235.888353] vfs_write+0x657/0x9b0
[ 1235.892969] ksys_write+0x116/0x250
[ 1235.897681] ? __ia32_sys_read+0xc0/0xc0
[ 1235.902881] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.908460] ? rcu_is_watching+0x15/0xd0
[ 1235.913664] do_syscall_64+0xf3/0x650
[ 1235.918572] ? trace_hardirqs_off+0xd/0x30
[ 1235.923966] ? exc_page_fault+0xda/0xf0
[ 1235.929070] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.935538] RIP: 0033:0x7fc21112d22e
[ 1235.940336] RSP: 002b:00007fffc26ea640 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.949633] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc21112d22e
[ 1235.958432] RDX: 0000000000000001 RSI: 00005614bd8dce90 RDI: 0000000000000007
[ 1235.967242] RBP: 00007fffc26ea650 R08: 0000000000000000 R09: 0000000000000000
[ 1235.976052] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc210fa5390
[ 1235.984861] R13: 00005614bd8dce90 R14: 0000000000000007 R15: 0000000000000003
[ 1235.993674]
[ 1235.996941] \x0aShowing all locks held in the system:
[ 1236.005484] 1 lock held by khungtaskd/185:
[ 1236.010879] #0: ffffffff96cfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.022349] 2 locks held by systemd-journal/804:
[ 1236.028329] #0: ff1100180da374e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.038217] #1: ffffffff96f02ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.051042] 4 locks held by python3/3721:
[ 1236.056340] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.066231] #1: ff1100016022e280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.077198] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.088357] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.098458] 4 locks held by python3/5776:
[ 1236.103758] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.113649] #1: ff110001e7bc3880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.124610] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.135765] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.145859]
[ 1236.148324] =============================================\x0a
[ 1358.321644] INFO: task python3:5776 blocked for more than 491 seconds.
[ 1358.329799] Not tainted 7.1.0-rc6-fxsj-gd1384a325c14 #1
[ 1358.337050] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.346642] task:python3 state:D stack:25672 pid:5776 tgid:5776 ppid:5775 task_flags:0x400100 flags:0x00080002
[ 1358.359916] Call Trace:
[ 1358.363454]
[ 1358.366607] __schedule+0xcc3/0x19c0
[ 1358.371489] ? io_schedule_timeout+0x140/0x140
[ 1358.377275] ? find_held_lock+0x2b/0x80
[ 1358.382384] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.388286] schedule+0xd8/0x260
[ 1358.392710] schedule_preempt_disabled+0x18/0x30
[ 1358.398691] __mutex_lock+0x6af/0x2010
[ 1358.403694] ? threaded_store+0x132/0x220
[ 1358.408988] ? ww_mutex_lock+0x160/0x160
[ 1358.414249] ? selinux_inode_free_security+0x330/0x330
[ 1358.420817] ? cap_capable+0x1d6/0x3e0
[ 1358.425828] ? threaded_store+0x132/0x220
[ 1358.431124] threaded_store+0x132/0x220
[ 1358.436229] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.441526] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.446829] ? sysfs_kf_write+0x65/0x160
[ 1358.452029] ? sysfs_file_ops+0x140/0x140
[ 1358.457377] kernfs_fop_write_iter+0x335/0x510
[ 1358.463164] ? kernfs_ops+0xf0/0xf0
[ 1358.467877] new_sync_write+0x333/0x750
[ 1358.472978] ? __lock_acquire+0x508/0xc10
[ 1358.478274] ? new_sync_read+0x740/0x740
[ 1358.483472] ? lock_acquire.part.0+0xbc/0x260
[ 1358.489846] ? ksys_write+0x116/0x250
[ 1358.494756] vfs_write+0x657/0x9b0
[ 1358.499436] ksys_write+0x116/0x250
[ 1358.504148] ? __ia32_sys_read+0xc0/0xc0
[ 1358.509347] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.514936] ? rcu_is_watching+0x15/0xd0
[ 1358.520129] do_syscall_64+0xf3/0x650
[ 1358.525035] ? trace_hardirqs_off+0xd/0x30
[ 1358.530488] ? exc_page_fault+0xda/0xf0
[ 1358.535587] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.542054] RIP: 0033:0x7f03f09ae22e
[ 1358.546856] RSP: 002b:00007fff43769ea0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.556156] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03f09ae22e
[ 1358.564963] RDX: 0000000000000001 RSI: 0000556d9e586e90 RDI: 0000000000000007
[ 1358.573768] RBP: 00007fff43769eb0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.582575] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f03f0826390
[ 1358.591381] R13: 0000556d9e586e90 R14: 0000000000000007 R15: 0000000000000003
[ 1358.600195]
[ 1358.603526] INFO: task python3:5776 is blocked on a mutex likely owned by task python3:3721.
[ 1358.613798] task:python3 state:D stack:25568 pid:3721 tgid:3721 ppid:3720 task_flags:0x400100 flags:0x00080802
[ 1358.627001] Call Trace:
[ 1358.630539]
[ 1358.633752] __schedule+0xcc3/0x19c0
[ 1358.638562] ? io_schedule_timeout+0x140/0x140
[ 1358.644336] ? find_held_lock+0x2b/0x80
[ 1358.649440] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.655615] schedule+0xd8/0x260
[ 1358.660030] schedule_timeout+0x13f/0x290
[ 1358.665328] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.671694] ? trace_contention_end+0xb3/0x1a0
[ 1358.677486] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.683174] ? ww_mutex_lock+0x160/0x160
[ 1358.688374] msleep+0x1b/0x30
[ 1358.692499] napi_set_threaded+0x3a5/0x440
[ 1358.697892] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.703384] netif_set_threaded+0x257/0x400
[ 1358.708876] threaded_store+0x1b8/0x220
[ 1358.713981] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.719278] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.724566] ? sysfs_kf_write+0x65/0x160
[ 1358.729766] ? sysfs_file_ops+0x140/0x140
[ 1358.735058] kernfs_fop_write_iter+0x335/0x510
[ 1358.740908] ? kernfs_ops+0xf0/0xf0
[ 1358.745621] new_sync_write+0x333/0x750
[ 1358.750722] ? __lock_acquire+0x508/0xc10
[ 1358.756015] ? new_sync_read+0x740/0x740
[ 1358.761215] ? lock_acquire.part.0+0xbc/0x260
[ 1358.766891] ? ksys_write+0x116/0x250
[ 1358.771799] vfs_write+0x657/0x9b0
[ 1358.776413] ksys_write+0x116/0x250
[ 1358.781124] ? __ia32_sys_read+0xc0/0xc0
[ 1358.786321] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.791972] ? rcu_is_watching+0x15/0xd0
[ 1358.797173] do_syscall_64+0xf3/0x650
[ 1358.802077] ? trace_hardirqs_off+0xd/0x30
[ 1358.807469] ? exc_page_fault+0xda/0xf0
[ 1358.812569] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.819036] RIP: 0033:0x7fc21112d22e
[ 1358.823843] RSP: 002b:00007fffc26ea640 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.833137] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc21112d22e
[ 1358.841941] RDX: 0000000000000001 RSI: 00005614bd8dce90 RDI: 0000000000000007
[ 1358.850813] RBP: 00007fffc26ea650 R08: 0000000000000000 R09: 0000000000000000
[ 1358.859619] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc210fa5390
[ 1358.868424] R13: 00005614bd8dce90 R14: 0000000000000007 R15: 0000000000000003
[ 1358.877237]
[ 1358.880512] \x0aShowing all locks held in the system:
[ 1358.889056] 1 lock held by khungtaskd/185:
[ 1358.894448] #0: ffffffff96cfdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.905930] 4 locks held by python3/3721:
[ 1358.911225] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.921112] #1: ff1100016022e280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.932074] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.943232] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.953332] 4 locks held by python3/5776:
[ 1358.958626] #0: ff1100012d8343e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.968516] #1: ff110001e7bc3880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.979479] #2: ff11000111ad4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.990643] #3: ff11000111a81d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.000736] 2 locks held by systemd/7625:
[ 1359.006033]
[ 1359.008497] =============================================\x0a