[ 761.933019] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 761.935473] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 762.063616] kselftest: Running tests in drivers/net
[ 989.705584] INFO: task python3:5380 blocked for more than 122 seconds.
[ 989.713779] Not tainted 7.1.0-rc6-obzi-g16b35835e365 #1
[ 989.721378] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.730970] task:python3 state:D stack:25672 pid:5380 tgid:5380 ppid:5379 task_flags:0x400100 flags:0x00080000
[ 989.744179] Call Trace:
[ 989.747720]
[ 989.750870] __schedule+0xcc3/0x19c0
[ 989.755687] ? io_schedule_timeout+0x140/0x140
[ 989.761470] ? find_held_lock+0x2b/0x80
[ 989.766566] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.772446] schedule+0xd8/0x260
[ 989.776865] schedule_preempt_disabled+0x18/0x30
[ 989.782843] __mutex_lock+0x6af/0x2010
[ 989.787848] ? threaded_store+0x132/0x220
[ 989.793148] ? ww_mutex_lock+0x160/0x160
[ 989.798347] ? selinux_inode_free_security+0x330/0x330
[ 989.804919] ? cap_capable+0x1d6/0x3e0
[ 989.809928] ? threaded_store+0x132/0x220
[ 989.815224] threaded_store+0x132/0x220
[ 989.820329] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.825628] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.830931] ? sysfs_kf_write+0x65/0x160
[ 989.836135] ? sysfs_file_ops+0x140/0x140
[ 989.841433] kernfs_fop_write_iter+0x335/0x510
[ 989.847218] ? kernfs_ops+0xf0/0xf0
[ 989.851921] new_sync_write+0x333/0x750
[ 989.857028] ? __lock_acquire+0x508/0xc10
[ 989.862324] ? new_sync_read+0x740/0x740
[ 989.867528] ? lock_acquire.part.0+0xbc/0x260
[ 989.873212] ? ksys_write+0x116/0x250
[ 989.878122] vfs_write+0x657/0x9b0
[ 989.882740] ksys_write+0x116/0x250
[ 989.887451] ? __ia32_sys_read+0xc0/0xc0
[ 989.892651] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.898239] ? rcu_is_watching+0x15/0xd0
[ 989.903441] do_syscall_64+0xf3/0x650
[ 989.908347] ? trace_hardirqs_off+0xd/0x30
[ 989.913741] ? exc_page_fault+0xda/0xf0
[ 989.918845] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.925312] RIP: 0033:0x7f90af58564e
[ 989.930124] RSP: 002b:00007ffd7e1f7f50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.939428] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90af58564e
[ 989.948235] RDX: 0000000000000001 RSI: 0000561d5526a420 RDI: 0000000000000007
[ 989.957037] RBP: 00007ffd7e1f7f60 R08: 0000000000000000 R09: 0000000000000000
[ 989.965847] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90af3fe390
[ 989.974655] R13: 0000561d5526a420 R14: 0000000000000007 R15: 00007f90af391120
[ 989.983466]
[ 989.986751] INFO: task python3:5380 is blocked on a mutex likely owned by task python3:3385.
[ 989.997023] task:python3 state:D stack:25672 pid:3385 tgid:3385 ppid:3384 task_flags:0x400100 flags:0x00080802
[ 990.010227] Call Trace:
[ 990.013766]
[ 990.016919] __schedule+0xcc3/0x19c0
[ 990.021733] ? io_schedule_timeout+0x140/0x140
[ 990.027518] ? find_held_lock+0x2b/0x80
[ 990.032623] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.038805] schedule+0xd8/0x260
[ 990.043225] schedule_timeout+0x13f/0x290
[ 990.048513] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.054877] ? trace_contention_end+0xb3/0x1a0
[ 990.060660] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.066354] ? ww_mutex_lock+0x160/0x160
[ 990.071555] msleep+0x1b/0x30
[ 990.075681] napi_set_threaded+0x3a5/0x440
[ 990.081075] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.086569] netif_set_threaded+0x257/0x400
[ 990.092064] threaded_store+0x1b8/0x220
[ 990.097166] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.102465] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.107761] ? sysfs_kf_write+0x65/0x160
[ 990.112963] ? sysfs_file_ops+0x140/0x140
[ 990.118261] kernfs_fop_write_iter+0x335/0x510
[ 990.124044] ? kernfs_ops+0xf0/0xf0
[ 990.128757] new_sync_write+0x333/0x750
[ 990.133851] ? __lock_acquire+0x508/0xc10
[ 990.139139] ? new_sync_read+0x740/0x740
[ 990.144341] ? lock_acquire.part.0+0xbc/0x260
[ 990.150027] ? ksys_write+0x116/0x250
[ 990.154938] vfs_write+0x657/0x9b0
[ 990.159555] ksys_write+0x116/0x250
[ 990.164269] ? __ia32_sys_read+0xc0/0xc0
[ 990.169469] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.175057] ? rcu_is_watching+0x15/0xd0
[ 990.180258] do_syscall_64+0xf3/0x650
[ 990.185162] ? irq_exit_rcu+0x1a/0x30
[ 990.190071] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.196537] RIP: 0033:0x7f548ac1564e
[ 990.201346] RSP: 002b:00007ffcbdba1250 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.210644] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f548ac1564e
[ 990.219453] RDX: 0000000000000001 RSI: 00005585d1f2b420 RDI: 0000000000000007
[ 990.228260] RBP: 00007ffcbdba1260 R08: 0000000000000000 R09: 0000000000000000
[ 990.237068] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f548aa8e390
[ 990.245877] R13: 00005585d1f2b420 R14: 0000000000000007 R15: 00007f548aa21120
[ 990.254693]
[ 990.257953] \x0aShowing all locks held in the system:
[ 990.266492] 1 lock held by khungtaskd/185:
[ 990.271884] #0: ffffffffab6fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.283368] 4 locks held by python3/3385:
[ 990.288665] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.298554] #1: ff110001413cd680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.309518] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.320676] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.330772] 4 locks held by python3/5380:
[ 990.336070] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.345951] #1: ff11000187b23880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.356903] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.368060] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.378144]
[ 990.380611] =============================================\x0a
[ 1112.586984] INFO: task python3:5380 blocked for more than 245 seconds.
[ 1112.595134] Not tainted 7.1.0-rc6-obzi-g16b35835e365 #1
[ 1112.602581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.612725] task:python3 state:D stack:25672 pid:5380 tgid:5380 ppid:5379 task_flags:0x400100 flags:0x00080000
[ 1112.625941] Call Trace:
[ 1112.629483]
[ 1112.632626] __schedule+0xcc3/0x19c0
[ 1112.637442] ? io_schedule_timeout+0x140/0x140
[ 1112.643227] ? find_held_lock+0x2b/0x80
[ 1112.648334] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.654221] schedule+0xd8/0x260
[ 1112.658642] schedule_preempt_disabled+0x18/0x30
[ 1112.664625] __mutex_lock+0x6af/0x2010
[ 1112.669631] ? threaded_store+0x132/0x220
[ 1112.674999] ? ww_mutex_lock+0x160/0x160
[ 1112.680200] ? selinux_inode_free_security+0x330/0x330
[ 1112.686773] ? cap_capable+0x1d6/0x3e0
[ 1112.691782] ? threaded_store+0x132/0x220
[ 1112.697080] threaded_store+0x132/0x220
[ 1112.702174] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.707464] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.712763] ? sysfs_kf_write+0x65/0x160
[ 1112.717963] ? sysfs_file_ops+0x140/0x140
[ 1112.723249] kernfs_fop_write_iter+0x335/0x510
[ 1112.729035] ? kernfs_ops+0xf0/0xf0
[ 1112.733747] new_sync_write+0x333/0x750
[ 1112.738853] ? __lock_acquire+0x508/0xc10
[ 1112.744149] ? new_sync_read+0x740/0x740
[ 1112.749351] ? lock_acquire.part.0+0xbc/0x260
[ 1112.755040] ? ksys_write+0x116/0x250
[ 1112.759952] vfs_write+0x657/0x9b0
[ 1112.764568] ksys_write+0x116/0x250
[ 1112.769284] ? __ia32_sys_read+0xc0/0xc0
[ 1112.774483] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.780071] ? rcu_is_watching+0x15/0xd0
[ 1112.785274] do_syscall_64+0xf3/0x650
[ 1112.790181] ? trace_hardirqs_off+0xd/0x30
[ 1112.795577] ? exc_page_fault+0xda/0xf0
[ 1112.800681] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.807148] RIP: 0033:0x7f90af58564e
[ 1112.811957] RSP: 002b:00007ffd7e1f7f50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.821259] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90af58564e
[ 1112.830067] RDX: 0000000000000001 RSI: 0000561d5526a420 RDI: 0000000000000007
[ 1112.838868] RBP: 00007ffd7e1f7f60 R08: 0000000000000000 R09: 0000000000000000
[ 1112.847676] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90af3fe390
[ 1112.856478] R13: 0000561d5526a420 R14: 0000000000000007 R15: 00007f90af391120
[ 1112.865293]
[ 1112.868639] INFO: task python3:5380 is blocked on a mutex likely owned by task python3:3385.
[ 1112.878914] task:python3 state:D stack:25672 pid:3385 tgid:3385 ppid:3384 task_flags:0x400100 flags:0x00080802
[ 1112.892116] Call Trace:
[ 1112.895656]
[ 1112.898808] __schedule+0xcc3/0x19c0
[ 1112.903620] ? io_schedule_timeout+0x140/0x140
[ 1112.909405] ? find_held_lock+0x2b/0x80
[ 1112.914509] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.920688] schedule+0xd8/0x260
[ 1112.925108] schedule_timeout+0x13f/0x290
[ 1112.930407] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.936775] ? trace_contention_end+0xb3/0x1a0
[ 1112.942628] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.948317] ? ww_mutex_lock+0x160/0x160
[ 1112.953517] msleep+0x1b/0x30
[ 1112.957642] napi_set_threaded+0x3a5/0x440
[ 1112.963037] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.968533] netif_set_threaded+0x257/0x400
[ 1112.974027] threaded_store+0x1b8/0x220
[ 1112.979129] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.984506] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.989802] ? sysfs_kf_write+0x65/0x160
[ 1112.995005] ? sysfs_file_ops+0x140/0x140
[ 1113.000302] kernfs_fop_write_iter+0x335/0x510
[ 1113.006088] ? kernfs_ops+0xf0/0xf0
[ 1113.010870] new_sync_write+0x333/0x750
[ 1113.015975] ? __lock_acquire+0x508/0xc10
[ 1113.021270] ? new_sync_read+0x740/0x740
[ 1113.026469] ? lock_acquire.part.0+0xbc/0x260
[ 1113.032157] ? ksys_write+0x116/0x250
[ 1113.037069] vfs_write+0x657/0x9b0
[ 1113.041683] ksys_write+0x116/0x250
[ 1113.046394] ? __ia32_sys_read+0xc0/0xc0
[ 1113.051595] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.057186] ? rcu_is_watching+0x15/0xd0
[ 1113.062387] do_syscall_64+0xf3/0x650
[ 1113.067296] ? irq_exit_rcu+0x1a/0x30
[ 1113.072204] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.078670] RIP: 0033:0x7f548ac1564e
[ 1113.083480] RSP: 002b:00007ffcbdba1250 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.092779] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f548ac1564e
[ 1113.101590] RDX: 0000000000000001 RSI: 00005585d1f2b420 RDI: 0000000000000007
[ 1113.110400] RBP: 00007ffcbdba1260 R08: 0000000000000000 R09: 0000000000000000
[ 1113.119206] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f548aa8e390
[ 1113.128016] R13: 00005585d1f2b420 R14: 0000000000000007 R15: 00007f548aa21120
[ 1113.136836]
[ 1113.140103] \x0aShowing all locks held in the system:
[ 1113.148647] 1 lock held by khungtaskd/185:
[ 1113.154039] #0: ffffffffab6fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.165516] 3 locks held by systemd-journal/803:
[ 1113.171496] #0: ff1100180dd374e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.181384] #1: ffffffffab902ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.194200] #2: ff11000143cc16d0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.205759] 4 locks held by python3/3385:
[ 1113.211055] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.220945] #1: ff110001413cd680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.231911] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.243072] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.253179] 4 locks held by python3/5380:
[ 1113.258473] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.268362] #1: ff11000187b23880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.279324] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.290483] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.300574] 5 locks held by systemd/6457:
[ 1113.305868] #0: ff110001308b93e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.315758] #1: ff1100012d905680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.326720] #2: ffffffffab769c20 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x640
[ 1113.338364] #3: ffffffffab5687b0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 1113.349911] #4: ffffffffab769990 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 1113.362251]
[ 1113.364727] =============================================\x0a
[ 1235.468386] INFO: task python3:5380 blocked for more than 368 seconds.
[ 1235.476534] Not tainted 7.1.0-rc6-obzi-g16b35835e365 #1
[ 1235.483784] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.493376] task:python3 state:D stack:25672 pid:5380 tgid:5380 ppid:5379 task_flags:0x400100 flags:0x00080002
[ 1235.506584] Call Trace:
[ 1235.510125]
[ 1235.513279] __schedule+0xcc3/0x19c0
[ 1235.518098] ? io_schedule_timeout+0x140/0x140
[ 1235.523885] ? find_held_lock+0x2b/0x80
[ 1235.528992] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.534881] schedule+0xd8/0x260
[ 1235.539302] schedule_preempt_disabled+0x18/0x30
[ 1235.545283] __mutex_lock+0x6af/0x2010
[ 1235.550291] ? threaded_store+0x132/0x220
[ 1235.555593] ? ww_mutex_lock+0x160/0x160
[ 1235.560792] ? selinux_inode_free_security+0x330/0x330
[ 1235.567362] ? cap_capable+0x1d6/0x3e0
[ 1235.572371] ? threaded_store+0x132/0x220
[ 1235.577669] threaded_store+0x132/0x220
[ 1235.582772] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.588071] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.593373] ? sysfs_kf_write+0x65/0x160
[ 1235.598575] ? sysfs_file_ops+0x140/0x140
[ 1235.603873] kernfs_fop_write_iter+0x335/0x510
[ 1235.609659] ? kernfs_ops+0xf0/0xf0
[ 1235.614373] new_sync_write+0x333/0x750
[ 1235.619478] ? __lock_acquire+0x508/0xc10
[ 1235.624776] ? new_sync_read+0x740/0x740
[ 1235.629979] ? lock_acquire.part.0+0xbc/0x260
[ 1235.635667] ? ksys_write+0x116/0x250
[ 1235.640579] vfs_write+0x657/0x9b0
[ 1235.645197] ksys_write+0x116/0x250
[ 1235.649910] ? __ia32_sys_read+0xc0/0xc0
[ 1235.655101] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.660691] ? rcu_is_watching+0x15/0xd0
[ 1235.665895] do_syscall_64+0xf3/0x650
[ 1235.670802] ? trace_hardirqs_off+0xd/0x30
[ 1235.676197] ? exc_page_fault+0xda/0xf0
[ 1235.681299] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.687758] RIP: 0033:0x7f90af58564e
[ 1235.692569] RSP: 002b:00007ffd7e1f7f50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.701868] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90af58564e
[ 1235.710677] RDX: 0000000000000001 RSI: 0000561d5526a420 RDI: 0000000000000007
[ 1235.719486] RBP: 00007ffd7e1f7f60 R08: 0000000000000000 R09: 0000000000000000
[ 1235.728295] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90af3fe390
[ 1235.737103] R13: 0000561d5526a420 R14: 0000000000000007 R15: 00007f90af391120
[ 1235.745917]
[ 1235.749186] INFO: task python3:5380 is blocked on a mutex likely owned by task python3:3385.
[ 1235.759458] task:python3 state:D stack:25672 pid:3385 tgid:3385 ppid:3384 task_flags:0x400100 flags:0x00080802
[ 1235.772661] Call Trace:
[ 1235.776199]
[ 1235.779351] __schedule+0xcc3/0x19c0
[ 1235.784162] ? io_schedule_timeout+0x140/0x140
[ 1235.789946] ? find_held_lock+0x2b/0x80
[ 1235.795050] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.801232] schedule+0xd8/0x260
[ 1235.805651] schedule_timeout+0x13f/0x290
[ 1235.810947] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.817316] ? trace_contention_end+0xb3/0x1a0
[ 1235.823101] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.828793] ? ww_mutex_lock+0x160/0x160
[ 1235.833993] msleep+0x1b/0x30
[ 1235.838120] napi_set_threaded+0x3a5/0x440
[ 1235.843508] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.849005] netif_set_threaded+0x257/0x400
[ 1235.854499] threaded_store+0x1b8/0x220
[ 1235.859604] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.864902] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.870192] ? sysfs_kf_write+0x65/0x160
[ 1235.875392] ? sysfs_file_ops+0x140/0x140
[ 1235.880686] kernfs_fop_write_iter+0x335/0x510
[ 1235.886473] ? kernfs_ops+0xf0/0xf0
[ 1235.891186] new_sync_write+0x333/0x750
[ 1235.896290] ? __lock_acquire+0x508/0xc10
[ 1235.901588] ? new_sync_read+0x740/0x740
[ 1235.906788] ? lock_acquire.part.0+0xbc/0x260
[ 1235.912477] ? ksys_write+0x116/0x250
[ 1235.917390] vfs_write+0x657/0x9b0
[ 1235.922007] ksys_write+0x116/0x250
[ 1235.926720] ? __ia32_sys_read+0xc0/0xc0
[ 1235.931919] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.937510] ? rcu_is_watching+0x15/0xd0
[ 1235.942712] do_syscall_64+0xf3/0x650
[ 1235.947621] ? irq_exit_rcu+0x1a/0x30
[ 1235.952528] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.958994] RIP: 0033:0x7f548ac1564e
[ 1235.963803] RSP: 002b:00007ffcbdba1250 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.973098] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f548ac1564e
[ 1235.981906] RDX: 0000000000000001 RSI: 00005585d1f2b420 RDI: 0000000000000007
[ 1235.990714] RBP: 00007ffcbdba1260 R08: 0000000000000000 R09: 0000000000000000
[ 1235.999524] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f548aa8e390
[ 1236.008332] R13: 00005585d1f2b420 R14: 0000000000000007 R15: 00007f548aa21120
[ 1236.017146]
[ 1236.020413] \x0aShowing all locks held in the system:
[ 1236.028951] 1 lock held by khungtaskd/185:
[ 1236.034344] #0: ffffffffab6fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.045819] 3 locks held by systemd-journal/803:
[ 1236.051790] #0: ff1100180e3b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.061680] #1: ffffffffab902ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.074495] #2: ff1100019432ef40 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1236.086053] 4 locks held by python3/3385:
[ 1236.091348] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.101240] #1: ff110001413cd680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.112206] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.123364] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.133461] 4 locks held by python3/5380:
[ 1236.138760] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.148641] #1: ff11000187b23880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.159605] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.170765] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.180855]
[ 1236.183320] =============================================\x0a
[ 1358.349804] INFO: task python3:5380 blocked for more than 491 seconds.
[ 1358.357964] Not tainted 7.1.0-rc6-obzi-g16b35835e365 #1
[ 1358.365216] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.374807] task:python3 state:D stack:25672 pid:5380 tgid:5380 ppid:5379 task_flags:0x400100 flags:0x00080002
[ 1358.388015] Call Trace:
[ 1358.391553]
[ 1358.394702] __schedule+0xcc3/0x19c0
[ 1358.399518] ? io_schedule_timeout+0x140/0x140
[ 1358.405302] ? find_held_lock+0x2b/0x80
[ 1358.410406] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.416289] schedule+0xd8/0x260
[ 1358.420704] schedule_preempt_disabled+0x18/0x30
[ 1358.426681] __mutex_lock+0x6af/0x2010
[ 1358.431682] ? threaded_store+0x132/0x220
[ 1358.436981] ? ww_mutex_lock+0x160/0x160
[ 1358.442179] ? selinux_inode_free_security+0x330/0x330
[ 1358.448748] ? cap_capable+0x1d6/0x3e0
[ 1358.453754] ? threaded_store+0x132/0x220
[ 1358.459050] threaded_store+0x132/0x220
[ 1358.464150] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.469444] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.474744] ? sysfs_kf_write+0x65/0x160
[ 1358.479943] ? sysfs_file_ops+0x140/0x140
[ 1358.485239] kernfs_fop_write_iter+0x335/0x510
[ 1358.491020] ? kernfs_ops+0xf0/0xf0
[ 1358.495731] new_sync_write+0x333/0x750
[ 1358.500835] ? __lock_acquire+0x508/0xc10
[ 1358.506128] ? new_sync_read+0x740/0x740
[ 1358.511326] ? lock_acquire.part.0+0xbc/0x260
[ 1358.517012] ? ksys_write+0x116/0x250
[ 1358.521922] vfs_write+0x657/0x9b0
[ 1358.526535] ksys_write+0x116/0x250
[ 1358.531246] ? __ia32_sys_read+0xc0/0xc0
[ 1358.536445] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.542031] ? rcu_is_watching+0x15/0xd0
[ 1358.547233] do_syscall_64+0xf3/0x650
[ 1358.552136] ? trace_hardirqs_off+0xd/0x30
[ 1358.557531] ? exc_page_fault+0xda/0xf0
[ 1358.562632] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.569097] RIP: 0033:0x7f90af58564e
[ 1358.573907] RSP: 002b:00007ffd7e1f7f50 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.583197] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90af58564e
[ 1358.592003] RDX: 0000000000000001 RSI: 0000561d5526a420 RDI: 0000000000000007
[ 1358.600807] RBP: 00007ffd7e1f7f60 R08: 0000000000000000 R09: 0000000000000000
[ 1358.609613] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90af3fe390
[ 1358.618418] R13: 0000561d5526a420 R14: 0000000000000007 R15: 00007f90af391120
[ 1358.627229]
[ 1358.630523] INFO: task python3:5380 is blocked on a mutex likely owned by task python3:3385.
[ 1358.640793] task:python3 state:D stack:25672 pid:3385 tgid:3385 ppid:3384 task_flags:0x400100 flags:0x00080802
[ 1358.653994] Call Trace:
[ 1358.657530]
[ 1358.660680] __schedule+0xcc3/0x19c0
[ 1358.665491] ? io_schedule_timeout+0x140/0x140
[ 1358.671275] ? find_held_lock+0x2b/0x80
[ 1358.676378] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.682555] schedule+0xd8/0x260
[ 1358.686974] schedule_timeout+0x13f/0x290
[ 1358.692269] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.698636] ? trace_contention_end+0xb3/0x1a0
[ 1358.704419] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.710106] ? ww_mutex_lock+0x160/0x160
[ 1358.715306] msleep+0x1b/0x30
[ 1358.719432] napi_set_threaded+0x3a5/0x440
[ 1358.724825] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.730317] netif_set_threaded+0x257/0x400
[ 1358.735809] threaded_store+0x1b8/0x220
[ 1358.740909] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.746205] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.751501] ? sysfs_kf_write+0x65/0x160
[ 1358.756700] ? sysfs_file_ops+0x140/0x140
[ 1358.761994] kernfs_fop_write_iter+0x335/0x510
[ 1358.767776] ? kernfs_ops+0xf0/0xf0
[ 1358.772486] new_sync_write+0x333/0x750
[ 1358.777589] ? __lock_acquire+0x508/0xc10
[ 1358.782884] ? new_sync_read+0x740/0x740
[ 1358.788084] ? lock_acquire.part.0+0xbc/0x260
[ 1358.793761] ? ksys_write+0x116/0x250
[ 1358.798670] vfs_write+0x657/0x9b0
[ 1358.803283] ksys_write+0x116/0x250
[ 1358.807996] ? __ia32_sys_read+0xc0/0xc0
[ 1358.813184] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.818772] ? rcu_is_watching+0x15/0xd0
[ 1358.823971] do_syscall_64+0xf3/0x650
[ 1358.828876] ? irq_exit_rcu+0x1a/0x30
[ 1358.833784] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.840249] RIP: 0033:0x7f548ac1564e
[ 1358.845055] RSP: 002b:00007ffcbdba1250 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.854352] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f548ac1564e
[ 1358.863170] RDX: 0000000000000001 RSI: 00005585d1f2b420 RDI: 0000000000000007
[ 1358.871976] RBP: 00007ffcbdba1260 R08: 0000000000000000 R09: 0000000000000000
[ 1358.880785] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f548aa8e390
[ 1358.889589] R13: 00005585d1f2b420 R14: 0000000000000007 R15: 00007f548aa21120
[ 1358.898400]
[ 1358.901675] \x0aShowing all locks held in the system:
[ 1358.910216] 1 lock held by khungtaskd/185:
[ 1358.915610] #0: ffffffffab6fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.927090] 4 locks held by python3/3385:
[ 1358.932383] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.942270] #1: ff110001413cd680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.953231] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.964386] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.974481] 4 locks held by python3/5380:
[ 1358.979773] #0: ff110001306213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.989662] #1: ff11000187b23880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.000621] #2: ff110017d9a84b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.011776] #3: ff110017d9ad1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.021868]
[ 1359.024333] =============================================\x0a