[ 778.643272] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 778.645820] nipa-hw-worker: [5/45] Retrying drivers/net:napi_threaded.py
[ 778.798725] kselftest: Running tests in drivers/net
[ 989.715460] INFO: task python3:5676 blocked for more than 122 seconds.
[ 989.723693] Not tainted 7.1.0-rc1-fhcd-gccc70ebf7dd8 #1
[ 989.731576] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.741165] task:python3 state:D stack:25672 pid:5676 tgid:5676 ppid:5675 task_flags:0x400100 flags:0x00080000
[ 989.754370] Call Trace:
[ 989.757911]
[ 989.761064] __schedule+0xcba/0x1940
[ 989.765885] ? io_schedule_timeout+0x140/0x140
[ 989.771672] ? find_held_lock+0x2b/0x80
[ 989.776779] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.782665] schedule+0xd4/0x260
[ 989.787084] schedule_preempt_disabled+0x18/0x30
[ 989.793062] __mutex_lock+0x6a3/0x2000
[ 989.798069] ? threaded_store+0x132/0x220
[ 989.803368] ? ww_mutex_lock+0x160/0x160
[ 989.808565] ? selinux_inode_free_security+0x330/0x330
[ 989.815137] ? cap_capable+0x1d6/0x3e0
[ 989.820147] ? threaded_store+0x132/0x220
[ 989.825446] threaded_store+0x132/0x220
[ 989.830547] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.835843] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.841144] ? sysfs_kf_write+0x65/0x160
[ 989.846347] ? sysfs_file_ops+0x140/0x140
[ 989.851645] kernfs_fop_write_iter+0x335/0x510
[ 989.857429] ? kernfs_ops+0xf0/0xf0
[ 989.862142] new_sync_write+0x333/0x750
[ 989.867246] ? __lock_acquire+0x508/0xc10
[ 989.872542] ? new_sync_read+0x740/0x740
[ 989.877741] ? lock_acquire.part.0+0xbc/0x260
[ 989.883430] ? ksys_write+0x116/0x250
[ 989.888340] vfs_write+0x657/0x9b0
[ 989.892957] ksys_write+0x116/0x250
[ 989.897668] ? __ia32_sys_read+0xc0/0xc0
[ 989.902867] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.908460] ? rcu_is_watching+0x15/0xd0
[ 989.913661] do_syscall_64+0xf3/0x650
[ 989.918558] ? trace_hardirqs_off+0xd/0x30
[ 989.923953] ? exc_page_fault+0xda/0xf0
[ 989.929057] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.935527] RIP: 0033:0x7fc7bf1ad22e
[ 989.940338] RSP: 002b:00007fff65f574c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.949635] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc7bf1ad22e
[ 989.958444] RDX: 0000000000000001 RSI: 000055e6f27d6e10 RDI: 0000000000000007
[ 989.967250] RBP: 00007fff65f574d0 R08: 0000000000000000 R09: 0000000000000000
[ 989.976059] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc7bf025390
[ 989.984869] R13: 000055e6f27d6e10 R14: 0000000000000007 R15: 0000000000000003
[ 989.993681]
[ 989.996993] INFO: task python3:5676 is blocked on a mutex likely owned by task python3:3631.
[ 990.007264] task:python3 state:D stack:25672 pid:3631 tgid:3631 ppid:3630 task_flags:0x400100 flags:0x00080802
[ 990.020469] Call Trace:
[ 990.024010]
[ 990.027160] __schedule+0xcba/0x1940
[ 990.031972] ? io_schedule_timeout+0x140/0x140
[ 990.037757] ? find_held_lock+0x2b/0x80
[ 990.042859] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.049037] schedule+0xd4/0x260
[ 990.053455] schedule_timeout+0x13f/0x290
[ 990.058754] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.065125] ? trace_contention_end+0xb3/0x1a0
[ 990.070910] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.076611] ? ww_mutex_lock+0x160/0x160
[ 990.081813] msleep+0x1b/0x30
[ 990.085940] napi_set_threaded+0x3a5/0x440
[ 990.091338] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.096833] netif_set_threaded+0x257/0x400
[ 990.102327] threaded_store+0x1b8/0x220
[ 990.107430] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.112728] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.118026] ? sysfs_kf_write+0x65/0x160
[ 990.123226] ? sysfs_file_ops+0x140/0x140
[ 990.128516] kernfs_fop_write_iter+0x335/0x510
[ 990.134299] ? kernfs_ops+0xf0/0xf0
[ 990.139009] new_sync_write+0x333/0x750
[ 990.144112] ? __lock_acquire+0x508/0xc10
[ 990.149399] ? new_sync_read+0x740/0x740
[ 990.154601] ? lock_acquire.part.0+0xbc/0x260
[ 990.160286] ? ksys_write+0x116/0x250
[ 990.165201] vfs_write+0x657/0x9b0
[ 990.169809] ksys_write+0x116/0x250
[ 990.174522] ? __ia32_sys_read+0xc0/0xc0
[ 990.179722] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.185310] ? rcu_is_watching+0x15/0xd0
[ 990.190510] do_syscall_64+0xf3/0x650
[ 990.195407] ? irq_exit_rcu+0x1a/0x30
[ 990.200315] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.206784] RIP: 0033:0x7fbbc1a1a22e
[ 990.211593] RSP: 002b:00007fff0c4f7070 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.220890] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbbc1a1a22e
[ 990.229696] RDX: 0000000000000001 RSI: 000055f956940e10 RDI: 0000000000000007
[ 990.238504] RBP: 00007fff0c4f7080 R08: 0000000000000000 R09: 0000000000000000
[ 990.247311] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbbc1892390
[ 990.256120] R13: 000055f956940e10 R14: 0000000000000007 R15: 0000000000000003
[ 990.264935]
[ 990.268196] \x0aShowing all locks held in the system:
[ 990.276737] 1 lock held by khungtaskd/185:
[ 990.282128] #0: ffffffffb14fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.293606] 3 locks held by systemd-journal/806:
[ 990.299586] #0: ff1100180e5b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.309477] #1: ffffffffb1703120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.322298] #2: ff110001a246bc50 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.333857] 4 locks held by python3/3631:
[ 990.339152] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.349042] #1: ff1100015c97ac80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.360009] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.371179] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.381279] 4 locks held by python3/5676:
[ 990.386573] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.396461] #1: ff11000189783e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.407422] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.418577] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.428666]
[ 990.431134] =============================================\x0a
[ 1067.541297] kworker/u129:15 (310) used greatest stack depth: 21936 bytes left
[ 1112.597790] INFO: task python3:5676 blocked for more than 245 seconds.
[ 1112.605934] Not tainted 7.1.0-rc1-fhcd-gccc70ebf7dd8 #1
[ 1112.613184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.622776] task:python3 state:D stack:25672 pid:5676 tgid:5676 ppid:5675 task_flags:0x400100 flags:0x00080000
[ 1112.635983] Call Trace:
[ 1112.639523]
[ 1112.642674] __schedule+0xcba/0x1940
[ 1112.647491] ? io_schedule_timeout+0x140/0x140
[ 1112.653277] ? find_held_lock+0x2b/0x80
[ 1112.658383] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.664267] schedule+0xd4/0x260
[ 1112.668686] schedule_preempt_disabled+0x18/0x30
[ 1112.674657] __mutex_lock+0x6a3/0x2000
[ 1112.679664] ? threaded_store+0x132/0x220
[ 1112.684966] ? ww_mutex_lock+0x160/0x160
[ 1112.690167] ? selinux_inode_free_security+0x330/0x330
[ 1112.696735] ? cap_capable+0x1d6/0x3e0
[ 1112.701743] ? threaded_store+0x132/0x220
[ 1112.707041] threaded_store+0x132/0x220
[ 1112.712136] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.717435] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.722738] ? sysfs_kf_write+0x65/0x160
[ 1112.727939] ? sysfs_file_ops+0x140/0x140
[ 1112.733235] kernfs_fop_write_iter+0x335/0x510
[ 1112.739021] ? kernfs_ops+0xf0/0xf0
[ 1112.743731] new_sync_write+0x333/0x750
[ 1112.748836] ? __lock_acquire+0x508/0xc10
[ 1112.754122] ? new_sync_read+0x740/0x740
[ 1112.759325] ? lock_acquire.part.0+0xbc/0x260
[ 1112.765010] ? ksys_write+0x116/0x250
[ 1112.769922] vfs_write+0x657/0x9b0
[ 1112.774538] ksys_write+0x116/0x250
[ 1112.779249] ? __ia32_sys_read+0xc0/0xc0
[ 1112.784450] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.790039] ? rcu_is_watching+0x15/0xd0
[ 1112.795239] do_syscall_64+0xf3/0x650
[ 1112.800147] ? trace_hardirqs_off+0xd/0x30
[ 1112.805542] ? exc_page_fault+0xda/0xf0
[ 1112.810645] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.817115] RIP: 0033:0x7fc7bf1ad22e
[ 1112.821925] RSP: 002b:00007fff65f574c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.831223] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc7bf1ad22e
[ 1112.840033] RDX: 0000000000000001 RSI: 000055e6f27d6e10 RDI: 0000000000000007
[ 1112.849433] RBP: 00007fff65f574d0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.858245] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc7bf025390
[ 1112.867054] R13: 000055e6f27d6e10 R14: 0000000000000007 R15: 0000000000000003
[ 1112.875869]
[ 1112.879172] INFO: task python3:5676 is blocked on a mutex likely owned by task python3:3631.
[ 1112.889448] task:python3 state:D stack:25672 pid:3631 tgid:3631 ppid:3630 task_flags:0x400100 flags:0x00080802
[ 1112.902650] Call Trace:
[ 1112.906190]
[ 1112.909343] __schedule+0xcba/0x1940
[ 1112.914155] ? io_schedule_timeout+0x140/0x140
[ 1112.919931] ? find_held_lock+0x2b/0x80
[ 1112.925033] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.931202] schedule+0xd4/0x260
[ 1112.935622] schedule_timeout+0x13f/0x290
[ 1112.940925] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.947285] ? trace_contention_end+0xb3/0x1a0
[ 1112.953071] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.958762] ? ww_mutex_lock+0x160/0x160
[ 1112.963962] msleep+0x1b/0x30
[ 1112.968088] napi_set_threaded+0x3a5/0x440
[ 1112.973478] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.978973] netif_set_threaded+0x257/0x400
[ 1112.984467] threaded_store+0x1b8/0x220
[ 1112.989573] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.994870] ? sysfs_file_kobj+0xbd/0x1e0
[ 1113.000156] ? sysfs_kf_write+0x65/0x160
[ 1113.005355] ? sysfs_file_ops+0x140/0x140
[ 1113.010651] kernfs_fop_write_iter+0x335/0x510
[ 1113.016437] ? kernfs_ops+0xf0/0xf0
[ 1113.021150] new_sync_write+0x333/0x750
[ 1113.026254] ? __lock_acquire+0x508/0xc10
[ 1113.031551] ? new_sync_read+0x740/0x740
[ 1113.036744] ? lock_acquire.part.0+0xbc/0x260
[ 1113.042429] ? ksys_write+0x116/0x250
[ 1113.047345] vfs_write+0x657/0x9b0
[ 1113.051951] ksys_write+0x116/0x250
[ 1113.056664] ? __ia32_sys_read+0xc0/0xc0
[ 1113.061862] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.067453] ? rcu_is_watching+0x15/0xd0
[ 1113.072656] do_syscall_64+0xf3/0x650
[ 1113.077566] ? irq_exit_rcu+0x1a/0x30
[ 1113.082476] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.088944] RIP: 0033:0x7fbbc1a1a22e
[ 1113.093751] RSP: 002b:00007fff0c4f7070 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.103051] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbbc1a1a22e
[ 1113.111969] RDX: 0000000000000001 RSI: 000055f956940e10 RDI: 0000000000000007
[ 1113.120778] RBP: 00007fff0c4f7080 R08: 0000000000000000 R09: 0000000000000000
[ 1113.129586] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbbc1892390
[ 1113.138396] R13: 000055f956940e10 R14: 0000000000000007 R15: 0000000000000003
[ 1113.147208]
[ 1113.150471] \x0aShowing all locks held in the system:
[ 1113.159015] 1 lock held by khungtaskd/185:
[ 1113.164407] #0: ffffffffb14fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.175892] 4 locks held by python3/3631:
[ 1113.181188] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.191080] #1: ff1100015c97ac80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.202044] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.213203] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.223302] 4 locks held by python3/5676:
[ 1113.228590] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.238478] #1: ff11000189783e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.249440] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.260599] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.270689]
[ 1113.273154] =============================================\x0a
[ 1235.478150] INFO: task python3:5676 blocked for more than 368 seconds.
[ 1235.486297] Not tainted 7.1.0-rc1-fhcd-gccc70ebf7dd8 #1
[ 1235.493546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.503133] task:python3 state:D stack:25672 pid:5676 tgid:5676 ppid:5675 task_flags:0x400100 flags:0x00080002
[ 1235.516336] Call Trace:
[ 1235.519876]
[ 1235.523024] __schedule+0xcba/0x1940
[ 1235.527839] ? io_schedule_timeout+0x140/0x140
[ 1235.533623] ? find_held_lock+0x2b/0x80
[ 1235.538727] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.544609] schedule+0xd4/0x260
[ 1235.549024] schedule_preempt_disabled+0x18/0x30
[ 1235.555005] __mutex_lock+0x6a3/0x2000
[ 1235.560009] ? threaded_store+0x132/0x220
[ 1235.565309] ? ww_mutex_lock+0x160/0x160
[ 1235.570506] ? selinux_inode_free_security+0x330/0x330
[ 1235.577073] ? cap_capable+0x1d6/0x3e0
[ 1235.582080] ? threaded_store+0x132/0x220
[ 1235.587377] threaded_store+0x132/0x220
[ 1235.592481] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.597776] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.603064] ? sysfs_kf_write+0x65/0x160
[ 1235.608264] ? sysfs_file_ops+0x140/0x140
[ 1235.613560] kernfs_fop_write_iter+0x335/0x510
[ 1235.619345] ? kernfs_ops+0xf0/0xf0
[ 1235.624046] new_sync_write+0x333/0x750
[ 1235.629149] ? __lock_acquire+0x508/0xc10
[ 1235.634442] ? new_sync_read+0x740/0x740
[ 1235.639642] ? lock_acquire.part.0+0xbc/0x260
[ 1235.645327] ? ksys_write+0x116/0x250
[ 1235.650236] vfs_write+0x657/0x9b0
[ 1235.654851] ksys_write+0x116/0x250
[ 1235.659564] ? __ia32_sys_read+0xc0/0xc0
[ 1235.664763] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.670351] ? rcu_is_watching+0x15/0xd0
[ 1235.675550] do_syscall_64+0xf3/0x650
[ 1235.680456] ? trace_hardirqs_off+0xd/0x30
[ 1235.685841] ? exc_page_fault+0xda/0xf0
[ 1235.690940] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.697408] RIP: 0033:0x7fc7bf1ad22e
[ 1235.702216] RSP: 002b:00007fff65f574c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.711515] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc7bf1ad22e
[ 1235.720320] RDX: 0000000000000001 RSI: 000055e6f27d6e10 RDI: 0000000000000007
[ 1235.729124] RBP: 00007fff65f574d0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.737931] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc7bf025390
[ 1235.746735] R13: 000055e6f27d6e10 R14: 0000000000000007 R15: 0000000000000003
[ 1235.755549]
[ 1235.758856] INFO: task python3:5676 is blocked on a mutex likely owned by task python3:3631.
[ 1235.769125] task:python3 state:D stack:25672 pid:3631 tgid:3631 ppid:3630 task_flags:0x400100 flags:0x00080802
[ 1235.782326] Call Trace:
[ 1235.785863]
[ 1235.789014] __schedule+0xcba/0x1940
[ 1235.793816] ? io_schedule_timeout+0x140/0x140
[ 1235.799589] ? find_held_lock+0x2b/0x80
[ 1235.804689] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.810865] schedule+0xd4/0x260
[ 1235.815274] schedule_timeout+0x13f/0x290
[ 1235.820570] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.826935] ? trace_contention_end+0xb3/0x1a0
[ 1235.832717] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.838404] ? ww_mutex_lock+0x160/0x160
[ 1235.843604] msleep+0x1b/0x30
[ 1235.847730] napi_set_threaded+0x3a5/0x440
[ 1235.853125] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.858619] netif_set_threaded+0x257/0x400
[ 1235.864114] threaded_store+0x1b8/0x220
[ 1235.869216] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.874505] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.879800] ? sysfs_kf_write+0x65/0x160
[ 1235.884996] ? sysfs_file_ops+0x140/0x140
[ 1235.890293] kernfs_fop_write_iter+0x335/0x510
[ 1235.896077] ? kernfs_ops+0xf0/0xf0
[ 1235.900785] new_sync_write+0x333/0x750
[ 1235.905886] ? __lock_acquire+0x508/0xc10
[ 1235.911180] ? new_sync_read+0x740/0x740
[ 1235.916378] ? lock_acquire.part.0+0xbc/0x260
[ 1235.922064] ? ksys_write+0x116/0x250
[ 1235.926978] vfs_write+0x657/0x9b0
[ 1235.931590] ksys_write+0x116/0x250
[ 1235.936299] ? __ia32_sys_read+0xc0/0xc0
[ 1235.941497] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.947083] ? rcu_is_watching+0x15/0xd0
[ 1235.952282] do_syscall_64+0xf3/0x650
[ 1235.957187] ? irq_exit_rcu+0x1a/0x30
[ 1235.962095] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.968559] RIP: 0033:0x7fbbc1a1a22e
[ 1235.973366] RSP: 002b:00007fff0c4f7070 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.982661] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbbc1a1a22e
[ 1235.991466] RDX: 0000000000000001 RSI: 000055f956940e10 RDI: 0000000000000007
[ 1236.000271] RBP: 00007fff0c4f7080 R08: 0000000000000000 R09: 0000000000000000
[ 1236.009078] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbbc1892390
[ 1236.017883] R13: 000055f956940e10 R14: 0000000000000007 R15: 0000000000000003
[ 1236.026693]
[ 1236.029958] \x0aShowing all locks held in the system:
[ 1236.038488] 1 lock held by khungtaskd/185:
[ 1236.043881] #0: ffffffffb14fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.055360] 4 locks held by python3/3631:
[ 1236.060654] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.070539] #1: ff1100015c97ac80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.081500] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.092654] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.102747] 4 locks held by python3/5676:
[ 1236.108040] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.117925] #1: ff11000189783e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.128888] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.140042] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.150132]
[ 1236.152590] =============================================\x0a
[ 1358.360490] INFO: task python3:5676 blocked for more than 491 seconds.
[ 1358.368635] Not tainted 7.1.0-rc1-fhcd-gccc70ebf7dd8 #1
[ 1358.375887] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.385478] task:python3 state:D stack:25672 pid:5676 tgid:5676 ppid:5675 task_flags:0x400100 flags:0x00080002
[ 1358.398683] Call Trace:
[ 1358.402225]
[ 1358.405374] __schedule+0xcba/0x1940
[ 1358.410192] ? io_schedule_timeout+0x140/0x140
[ 1358.415979] ? find_held_lock+0x2b/0x80
[ 1358.421084] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.426967] schedule+0xd4/0x260
[ 1358.431388] schedule_preempt_disabled+0x18/0x30
[ 1358.437369] __mutex_lock+0x6a3/0x2000
[ 1358.442375] ? threaded_store+0x132/0x220
[ 1358.447674] ? ww_mutex_lock+0x160/0x160
[ 1358.452872] ? selinux_inode_free_security+0x330/0x330
[ 1358.459445] ? cap_capable+0x1d6/0x3e0
[ 1358.464455] ? threaded_store+0x132/0x220
[ 1358.469753] threaded_store+0x132/0x220
[ 1358.474857] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.480153] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.485447] ? sysfs_kf_write+0x65/0x160
[ 1358.490649] ? sysfs_file_ops+0x140/0x140
[ 1358.495944] kernfs_fop_write_iter+0x335/0x510
[ 1358.501732] ? kernfs_ops+0xf0/0xf0
[ 1358.506445] new_sync_write+0x333/0x750
[ 1358.511550] ? __lock_acquire+0x508/0xc10
[ 1358.516840] ? new_sync_read+0x740/0x740
[ 1358.522042] ? lock_acquire.part.0+0xbc/0x260
[ 1358.527727] ? ksys_write+0x116/0x250
[ 1358.532640] vfs_write+0x657/0x9b0
[ 1358.537255] ksys_write+0x116/0x250
[ 1358.541958] ? __ia32_sys_read+0xc0/0xc0
[ 1358.547159] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.552749] ? rcu_is_watching+0x15/0xd0
[ 1358.557949] do_syscall_64+0xf3/0x650
[ 1358.562857] ? trace_hardirqs_off+0xd/0x30
[ 1358.568253] ? exc_page_fault+0xda/0xf0
[ 1358.573346] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.579815] RIP: 0033:0x7fc7bf1ad22e
[ 1358.584625] RSP: 002b:00007fff65f574c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.593924] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc7bf1ad22e
[ 1358.602732] RDX: 0000000000000001 RSI: 000055e6f27d6e10 RDI: 0000000000000007
[ 1358.611544] RBP: 00007fff65f574d0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.620353] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc7bf025390
[ 1358.629163] R13: 000055e6f27d6e10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.637977]
[ 1358.641254] INFO: task python3:5676 is blocked on a mutex likely owned by task python3:3631.
[ 1358.651529] task:python3 state:D stack:25672 pid:3631 tgid:3631 ppid:3630 task_flags:0x400100 flags:0x00080802
[ 1358.664723] Call Trace:
[ 1358.668264]
[ 1358.671416] __schedule+0xcba/0x1940
[ 1358.676227] ? io_schedule_timeout+0x140/0x140
[ 1358.682013] ? find_held_lock+0x2b/0x80
[ 1358.687116] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.693297] schedule+0xd4/0x260
[ 1358.697714] schedule_timeout+0x13f/0x290
[ 1358.703010] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.709380] ? trace_contention_end+0xb3/0x1a0
[ 1358.715165] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.720856] ? ww_mutex_lock+0x160/0x160
[ 1358.726057] msleep+0x1b/0x30
[ 1358.730185] napi_set_threaded+0x3a5/0x440
[ 1358.735582] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.741075] netif_set_threaded+0x257/0x400
[ 1358.746569] threaded_store+0x1b8/0x220
[ 1358.751673] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.756968] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.762266] ? sysfs_kf_write+0x65/0x160
[ 1358.767469] ? sysfs_file_ops+0x140/0x140
[ 1358.772764] kernfs_fop_write_iter+0x335/0x510
[ 1358.778550] ? kernfs_ops+0xf0/0xf0
[ 1358.783262] new_sync_write+0x333/0x750
[ 1358.788356] ? __lock_acquire+0x508/0xc10
[ 1358.793651] ? new_sync_read+0x740/0x740
[ 1358.798850] ? lock_acquire.part.0+0xbc/0x260
[ 1358.804535] ? ksys_write+0x116/0x250
[ 1358.809448] vfs_write+0x657/0x9b0
[ 1358.814064] ksys_write+0x116/0x250
[ 1358.818766] ? __ia32_sys_read+0xc0/0xc0
[ 1358.823966] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.829555] ? rcu_is_watching+0x15/0xd0
[ 1358.834754] do_syscall_64+0xf3/0x650
[ 1358.839659] ? irq_exit_rcu+0x1a/0x30
[ 1358.844566] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.851033] RIP: 0033:0x7fbbc1a1a22e
[ 1358.855831] RSP: 002b:00007fff0c4f7070 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.865127] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbbc1a1a22e
[ 1358.873928] RDX: 0000000000000001 RSI: 000055f956940e10 RDI: 0000000000000007
[ 1358.882737] RBP: 00007fff0c4f7080 R08: 0000000000000000 R09: 0000000000000000
[ 1358.891546] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbbc1892390
[ 1358.900356] R13: 000055f956940e10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.909169]
[ 1358.912437] \x0aShowing all locks held in the system:
[ 1358.920976] 1 lock held by khungtaskd/185:
[ 1358.926371] #0: ffffffffb14fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.937852] 4 locks held by python3/3631:
[ 1358.943148] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.953039] #1: ff1100015c97ac80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.963999] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.975155] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.985247] 4 locks held by python3/5676:
[ 1358.990543] #0: ff1100012e00c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.000434] #1: ff11000189783e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.011394] #2: ff11000120be8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.022553] #3: ff11000120bc9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.032646]
[ 1359.035113] =============================================\x0a