[ 791.657221] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 791.659638] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py
[ 791.819639] kselftest: Running tests in drivers/net
[ 989.705121] INFO: task python3:5945 blocked for more than 122 seconds.
[ 989.713328] Not tainted 7.1.0-rc1-gzdh-g8a2dfb0b7e27 #1
[ 989.720956] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.730547] task:python3 state:D stack:25672 pid:5945 tgid:5945 ppid:5944 task_flags:0x400100 flags:0x00080000
[ 989.743751] Call Trace:
[ 989.747289]
[ 989.750440] __schedule+0xcba/0x1940
[ 989.755259] ? io_schedule_timeout+0x140/0x140
[ 989.761042] ? find_held_lock+0x2b/0x80
[ 989.766147] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.772021] schedule+0xd4/0x260
[ 989.776436] schedule_preempt_disabled+0x18/0x30
[ 989.782416] __mutex_lock+0x6a3/0x2000
[ 989.787411] ? threaded_store+0x132/0x220
[ 989.792712] ? ww_mutex_lock+0x160/0x160
[ 989.797910] ? selinux_inode_free_security+0x330/0x330
[ 989.804477] ? cap_capable+0x1d6/0x3e0
[ 989.809484] ? threaded_store+0x132/0x220
[ 989.814779] threaded_store+0x132/0x220
[ 989.819881] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.825171] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.830466] ? sysfs_kf_write+0x65/0x160
[ 989.835666] ? sysfs_file_ops+0x140/0x140
[ 989.840962] kernfs_fop_write_iter+0x335/0x510
[ 989.846743] ? kernfs_ops+0xf0/0xf0
[ 989.851452] new_sync_write+0x333/0x750
[ 989.856555] ? __lock_acquire+0x508/0xc10
[ 989.861848] ? new_sync_read+0x740/0x740
[ 989.867048] ? lock_acquire.part.0+0xbc/0x260
[ 989.872725] ? ksys_write+0x116/0x250
[ 989.877637] vfs_write+0x657/0x9b0
[ 989.882253] ksys_write+0x116/0x250
[ 989.886966] ? __ia32_sys_read+0xc0/0xc0
[ 989.892162] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.897751] ? rcu_is_watching+0x15/0xd0
[ 989.902955] do_syscall_64+0xf3/0x650
[ 989.907859] ? irq_exit_rcu+0x1a/0x30
[ 989.912766] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.919233] RIP: 0033:0x7f96ad6a322e
[ 989.924043] RSP: 002b:00007ffd07dbae70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.933339] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f96ad6a322e
[ 989.942147] RDX: 0000000000000001 RSI: 000055f81ac3fa10 RDI: 0000000000000007
[ 989.950956] RBP: 00007ffd07dbae80 R08: 0000000000000000 R09: 0000000000000000
[ 989.959763] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f96ad51b390
[ 989.968583] R13: 000055f81ac3fa10 R14: 0000000000000007 R15: 0000000000000003
[ 989.977395]
[ 989.980714] INFO: task python3:5945 is blocked on a mutex likely owned by task python3:3858.
[ 989.990987] task:python3 state:D stack:25672 pid:3858 tgid:3858 ppid:3857 task_flags:0x400100 flags:0x00080802
[ 990.004187] Call Trace:
[ 990.007723]
[ 990.010872] __schedule+0xcba/0x1940
[ 990.015684] ? io_schedule_timeout+0x140/0x140
[ 990.021458] ? find_held_lock+0x2b/0x80
[ 990.026560] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.032742] schedule+0xd4/0x260
[ 990.037158] schedule_timeout+0x13f/0x290
[ 990.042454] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.048822] ? trace_contention_end+0xb3/0x1a0
[ 990.054606] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.060297] ? ww_mutex_lock+0x160/0x160
[ 990.065495] msleep+0x1b/0x30
[ 990.069621] napi_set_threaded+0x3a5/0x440
[ 990.075016] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.080508] netif_set_threaded+0x257/0x400
[ 990.086000] threaded_store+0x1b8/0x220
[ 990.091103] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.096392] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.101687] ? sysfs_kf_write+0x65/0x160
[ 990.106886] ? sysfs_file_ops+0x140/0x140
[ 990.112181] kernfs_fop_write_iter+0x335/0x510
[ 990.117966] ? kernfs_ops+0xf0/0xf0
[ 990.122667] new_sync_write+0x333/0x750
[ 990.127769] ? __lock_acquire+0x508/0xc10
[ 990.133062] ? new_sync_read+0x740/0x740
[ 990.138262] ? lock_acquire.part.0+0xbc/0x260
[ 990.143938] ? ksys_write+0x116/0x250
[ 990.148846] vfs_write+0x657/0x9b0
[ 990.153462] ksys_write+0x116/0x250
[ 990.158165] ? __ia32_sys_read+0xc0/0xc0
[ 990.163354] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.168948] ? rcu_is_watching+0x15/0xd0
[ 990.174148] do_syscall_64+0xf3/0x650
[ 990.179053] ? trace_hardirqs_off+0xd/0x30
[ 990.184449] ? exc_page_fault+0xda/0xf0
[ 990.189552] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.196015] RIP: 0033:0x7f3f1216622e
[ 990.200824] RSP: 002b:00007ffcd6855280 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.210119] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3f1216622e
[ 990.218924] RDX: 0000000000000001 RSI: 00005637fe556a10 RDI: 0000000000000007
[ 990.227729] RBP: 00007ffcd6855290 R08: 0000000000000000 R09: 0000000000000000
[ 990.236538] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3f11fde390
[ 990.245345] R13: 00005637fe556a10 R14: 0000000000000007 R15: 0000000000000003
[ 990.254158]
[ 990.257415] \x0aShowing all locks held in the system:
[ 990.265963] 1 lock held by khungtaskd/186:
[ 990.271354] #0: ffffffffabafdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.282832] 4 locks held by systemd-journal/807:
[ 990.288812] #0: ff1100180e7372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.298699] #1: ffffffffabd03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.311515] #2: ff110001a27c1db0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.323061] #3: ff1100019e374f58 (&xa->xa_lock#5){-.-.}-{3:3}, at: __folio_mark_dirty+0x9d/0x2c0
[ 990.333827] 4 locks held by python3/3858:
[ 990.339115] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.349000] #1: ff11000170050880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.359960] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.371119] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.381225] 4 locks held by python3/5945:
[ 990.386516] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.396406] #1: ff110001c53b3280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.407366] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.418521] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.428610]
[ 990.431074] =============================================\x0a
[ 1112.587213] INFO: task python3:5945 blocked for more than 245 seconds.
[ 1112.595357] Not tainted 7.1.0-rc1-gzdh-g8a2dfb0b7e27 #1
[ 1112.602607] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.612199] task:python3 state:D stack:25672 pid:5945 tgid:5945 ppid:5944 task_flags:0x400100 flags:0x00080000
[ 1112.625404] Call Trace:
[ 1112.628945]
[ 1112.632094] __schedule+0xcba/0x1940
[ 1112.636913] ? io_schedule_timeout+0x140/0x140
[ 1112.642700] ? find_held_lock+0x2b/0x80
[ 1112.647807] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.653693] schedule+0xd4/0x260
[ 1112.658111] schedule_preempt_disabled+0x18/0x30
[ 1112.664092] __mutex_lock+0x6a3/0x2000
[ 1112.669096] ? threaded_store+0x132/0x220
[ 1112.674395] ? ww_mutex_lock+0x160/0x160
[ 1112.679594] ? selinux_inode_free_security+0x330/0x330
[ 1112.686164] ? cap_capable+0x1d6/0x3e0
[ 1112.691173] ? threaded_store+0x132/0x220
[ 1112.696472] threaded_store+0x132/0x220
[ 1112.701576] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.706875] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.712175] ? sysfs_kf_write+0x65/0x160
[ 1112.717379] ? sysfs_file_ops+0x140/0x140
[ 1112.722677] kernfs_fop_write_iter+0x335/0x510
[ 1112.728464] ? kernfs_ops+0xf0/0xf0
[ 1112.733174] new_sync_write+0x333/0x750
[ 1112.738269] ? __lock_acquire+0x508/0xc10
[ 1112.743565] ? new_sync_read+0x740/0x740
[ 1112.748768] ? lock_acquire.part.0+0xbc/0x260
[ 1112.754455] ? ksys_write+0x116/0x250
[ 1112.759355] vfs_write+0x657/0x9b0
[ 1112.763972] ksys_write+0x116/0x250
[ 1112.768685] ? __ia32_sys_read+0xc0/0xc0
[ 1112.773878] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.779470] ? rcu_is_watching+0x15/0xd0
[ 1112.784675] do_syscall_64+0xf3/0x650
[ 1112.789585] ? irq_exit_rcu+0x1a/0x30
[ 1112.794493] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.800963] RIP: 0033:0x7f96ad6a322e
[ 1112.805773] RSP: 002b:00007ffd07dbae70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.815061] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f96ad6a322e
[ 1112.823870] RDX: 0000000000000001 RSI: 000055f81ac3fa10 RDI: 0000000000000007
[ 1112.832678] RBP: 00007ffd07dbae80 R08: 0000000000000000 R09: 0000000000000000
[ 1112.841487] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f96ad51b390
[ 1112.850297] R13: 000055f81ac3fa10 R14: 0000000000000007 R15: 0000000000000003
[ 1112.859112]
[ 1112.862382] INFO: task python3:5945 is blocked on a mutex likely owned by task python3:3858.
[ 1112.872671] task:python3 state:D stack:25672 pid:3858 tgid:3858 ppid:3857 task_flags:0x400100 flags:0x00080802
[ 1112.885880] Call Trace:
[ 1112.889418]
[ 1112.892570] __schedule+0xcba/0x1940
[ 1112.897384] ? io_schedule_timeout+0x140/0x140
[ 1112.903168] ? find_held_lock+0x2b/0x80
[ 1112.908264] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.914444] schedule+0xd4/0x260
[ 1112.918863] schedule_timeout+0x13f/0x290
[ 1112.924158] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.930528] ? trace_contention_end+0xb3/0x1a0
[ 1112.936305] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.941998] ? ww_mutex_lock+0x160/0x160
[ 1112.947198] msleep+0x1b/0x30
[ 1112.951325] napi_set_threaded+0x3a5/0x440
[ 1112.956722] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.962220] netif_set_threaded+0x257/0x400
[ 1112.967715] threaded_store+0x1b8/0x220
[ 1112.972820] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.978117] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.983408] ? sysfs_kf_write+0x65/0x160
[ 1112.988608] ? sysfs_file_ops+0x140/0x140
[ 1112.993895] kernfs_fop_write_iter+0x335/0x510
[ 1112.999681] ? kernfs_ops+0xf0/0xf0
[ 1113.004392] new_sync_write+0x333/0x750
[ 1113.009495] ? __lock_acquire+0x508/0xc10
[ 1113.014786] ? new_sync_read+0x740/0x740
[ 1113.019987] ? lock_acquire.part.0+0xbc/0x260
[ 1113.025663] ? ksys_write+0x116/0x250
[ 1113.030572] vfs_write+0x657/0x9b0
[ 1113.035189] ksys_write+0x116/0x250
[ 1113.039902] ? __ia32_sys_read+0xc0/0xc0
[ 1113.045105] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.050695] ? rcu_is_watching+0x15/0xd0
[ 1113.055898] do_syscall_64+0xf3/0x650
[ 1113.060806] ? trace_hardirqs_off+0xd/0x30
[ 1113.066202] ? exc_page_fault+0xda/0xf0
[ 1113.071305] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.077765] RIP: 0033:0x7f3f1216622e
[ 1113.082574] RSP: 002b:00007ffcd6855280 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.091872] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3f1216622e
[ 1113.100680] RDX: 0000000000000001 RSI: 00005637fe556a10 RDI: 0000000000000007
[ 1113.109489] RBP: 00007ffcd6855290 R08: 0000000000000000 R09: 0000000000000000
[ 1113.118310] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3f11fde390
[ 1113.127124] R13: 00005637fe556a10 R14: 0000000000000007 R15: 0000000000000003
[ 1113.135938]
[ 1113.139203] \x0aShowing all locks held in the system:
[ 1113.147744] 1 lock held by khungtaskd/186:
[ 1113.153140] #0: ffffffffabafdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.164616] 3 locks held by systemd-journal/807:
[ 1113.170595] #0: ff1100180e5b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1113.180484] #1: ffffffffabd03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: lockref_get_not_dead+0xe/0x60
[ 1113.193299] #2: ff110001a5f38650 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.204860] 4 locks held by python3/3858:
[ 1113.210154] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.220047] #1: ff11000170050880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.231012] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.242173] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.252275] 4 locks held by python3/5945:
[ 1113.257568] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.267456] #1: ff110001c53b3280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.278420] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.289578] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.299668]
[ 1113.302134] =============================================\x0a
[ 1210.891215] kworker/u130:7 (855) used greatest stack depth: 21456 bytes left
[ 1235.467535] INFO: task python3:5945 blocked for more than 368 seconds.
[ 1235.475682] Not tainted 7.1.0-rc1-gzdh-g8a2dfb0b7e27 #1
[ 1235.482923] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.492511] task:python3 state:D stack:25672 pid:5945 tgid:5945 ppid:5944 task_flags:0x400100 flags:0x00080002
[ 1235.505714] Call Trace:
[ 1235.509256]
[ 1235.512406] __schedule+0xcba/0x1940
[ 1235.517222] ? io_schedule_timeout+0x140/0x140
[ 1235.523009] ? find_held_lock+0x2b/0x80
[ 1235.528117] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.534002] schedule+0xd4/0x260
[ 1235.538420] schedule_preempt_disabled+0x18/0x30
[ 1235.544402] __mutex_lock+0x6a3/0x2000
[ 1235.549405] ? threaded_store+0x132/0x220
[ 1235.554704] ? ww_mutex_lock+0x160/0x160
[ 1235.559905] ? selinux_inode_free_security+0x330/0x330
[ 1235.566475] ? cap_capable+0x1d6/0x3e0
[ 1235.571485] ? threaded_store+0x132/0x220
[ 1235.576785] threaded_store+0x132/0x220
[ 1235.581886] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.587184] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.592481] ? sysfs_kf_write+0x65/0x160
[ 1235.597683] ? sysfs_file_ops+0x140/0x140
[ 1235.602981] kernfs_fop_write_iter+0x335/0x510
[ 1235.608755] ? kernfs_ops+0xf0/0xf0
[ 1235.613466] new_sync_write+0x333/0x750
[ 1235.618572] ? __lock_acquire+0x508/0xc10
[ 1235.623868] ? new_sync_read+0x740/0x740
[ 1235.629071] ? lock_acquire.part.0+0xbc/0x260
[ 1235.634758] ? ksys_write+0x116/0x250
[ 1235.639662] vfs_write+0x657/0x9b0
[ 1235.644280] ksys_write+0x116/0x250
[ 1235.648992] ? __ia32_sys_read+0xc0/0xc0
[ 1235.654190] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.659781] ? rcu_is_watching+0x15/0xd0
[ 1235.664982] do_syscall_64+0xf3/0x650
[ 1235.669888] ? irq_exit_rcu+0x1a/0x30
[ 1235.674797] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.681267] RIP: 0033:0x7f96ad6a322e
[ 1235.686078] RSP: 002b:00007ffd07dbae70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.695379] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f96ad6a322e
[ 1235.704190] RDX: 0000000000000001 RSI: 000055f81ac3fa10 RDI: 0000000000000007
[ 1235.712999] RBP: 00007ffd07dbae80 R08: 0000000000000000 R09: 0000000000000000
[ 1235.721808] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f96ad51b390
[ 1235.730617] R13: 000055f81ac3fa10 R14: 0000000000000007 R15: 0000000000000003
[ 1235.739432]
[ 1235.742730] INFO: task python3:5945 is blocked on a mutex likely owned by task python3:3858.
[ 1235.753004] task:python3 state:D stack:25672 pid:3858 tgid:3858 ppid:3857 task_flags:0x400100 flags:0x00080802
[ 1235.766198] Call Trace:
[ 1235.769736]
[ 1235.772887] __schedule+0xcba/0x1940
[ 1235.777700] ? io_schedule_timeout+0x140/0x140
[ 1235.783484] ? find_held_lock+0x2b/0x80
[ 1235.788589] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.794767] schedule+0xd4/0x260
[ 1235.799186] schedule_timeout+0x13f/0x290
[ 1235.804483] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.810853] ? trace_contention_end+0xb3/0x1a0
[ 1235.816640] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.822333] ? ww_mutex_lock+0x160/0x160
[ 1235.827533] msleep+0x1b/0x30
[ 1235.831659] napi_set_threaded+0x3a5/0x440
[ 1235.837047] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.842545] netif_set_threaded+0x257/0x400
[ 1235.848039] threaded_store+0x1b8/0x220
[ 1235.853144] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.858442] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.863738] ? sysfs_kf_write+0x65/0x160
[ 1235.868938] ? sysfs_file_ops+0x140/0x140
[ 1235.874234] kernfs_fop_write_iter+0x335/0x510
[ 1235.880020] ? kernfs_ops+0xf0/0xf0
[ 1235.884731] new_sync_write+0x333/0x750
[ 1235.889834] ? __lock_acquire+0x508/0xc10
[ 1235.895133] ? new_sync_read+0x740/0x740
[ 1235.900333] ? lock_acquire.part.0+0xbc/0x260
[ 1235.906010] ? ksys_write+0x116/0x250
[ 1235.910923] vfs_write+0x657/0x9b0
[ 1235.915540] ksys_write+0x116/0x250
[ 1235.920252] ? __ia32_sys_read+0xc0/0xc0
[ 1235.925446] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.931036] ? rcu_is_watching+0x15/0xd0
[ 1235.936230] do_syscall_64+0xf3/0x650
[ 1235.941138] ? trace_hardirqs_off+0xd/0x30
[ 1235.946533] ? exc_page_fault+0xda/0xf0
[ 1235.951637] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.958097] RIP: 0033:0x7f3f1216622e
[ 1235.962906] RSP: 002b:00007ffcd6855280 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.972202] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3f1216622e
[ 1235.981001] RDX: 0000000000000001 RSI: 00005637fe556a10 RDI: 0000000000000007
[ 1235.989810] RBP: 00007ffcd6855290 R08: 0000000000000000 R09: 0000000000000000
[ 1235.998619] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3f11fde390
[ 1236.007425] R13: 00005637fe556a10 R14: 0000000000000007 R15: 0000000000000003
[ 1236.016239]
[ 1236.019508] \x0aShowing all locks held in the system:
[ 1236.028050] 1 lock held by khungtaskd/186:
[ 1236.033446] #0: ffffffffabafdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.044917] 2 locks held by systemd-journal/807:
[ 1236.050895] #0: ff1100180dab72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1236.060783] #1: ffffffffabd03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.073608] 4 locks held by python3/3858:
[ 1236.078902] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.088791] #1: ff11000170050880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.099754] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.110912] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.121009] 4 locks held by python3/5945:
[ 1236.126305] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.136197] #1: ff110001c53b3280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.147158] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.158305] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.168401]
[ 1236.170868] =============================================\x0a
[ 1358.349798] INFO: task python3:5945 blocked for more than 491 seconds.
[ 1358.357947] Not tainted 7.1.0-rc1-gzdh-g8a2dfb0b7e27 #1
[ 1358.365199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.374790] task:python3 state:D stack:25672 pid:5945 tgid:5945 ppid:5944 task_flags:0x400100 flags:0x00080002
[ 1358.387997] Call Trace:
[ 1358.391537]
[ 1358.394688] __schedule+0xcba/0x1940
[ 1358.399505] ? io_schedule_timeout+0x140/0x140
[ 1358.405291] ? find_held_lock+0x2b/0x80
[ 1358.410395] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.416277] schedule+0xd4/0x260
[ 1358.420692] schedule_preempt_disabled+0x18/0x30
[ 1358.426673] __mutex_lock+0x6a3/0x2000
[ 1358.431669] ? threaded_store+0x132/0x220
[ 1358.436967] ? ww_mutex_lock+0x160/0x160
[ 1358.442164] ? selinux_inode_free_security+0x330/0x330
[ 1358.448733] ? cap_capable+0x1d6/0x3e0
[ 1358.453742] ? threaded_store+0x132/0x220
[ 1358.459038] threaded_store+0x132/0x220
[ 1358.464137] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.469435] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.474733] ? sysfs_kf_write+0x65/0x160
[ 1358.479932] ? sysfs_file_ops+0x140/0x140
[ 1358.485223] kernfs_fop_write_iter+0x335/0x510
[ 1358.491005] ? kernfs_ops+0xf0/0xf0
[ 1358.495716] new_sync_write+0x333/0x750
[ 1358.500816] ? __lock_acquire+0x508/0xc10
[ 1358.506111] ? new_sync_read+0x740/0x740
[ 1358.511313] ? lock_acquire.part.0+0xbc/0x260
[ 1358.516999] ? ksys_write+0x116/0x250
[ 1358.521907] vfs_write+0x657/0x9b0
[ 1358.526524] ksys_write+0x116/0x250
[ 1358.531236] ? __ia32_sys_read+0xc0/0xc0
[ 1358.536423] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.542013] ? rcu_is_watching+0x15/0xd0
[ 1358.547213] do_syscall_64+0xf3/0x650
[ 1358.552113] ? irq_exit_rcu+0x1a/0x30
[ 1358.557012] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.563469] RIP: 0033:0x7f96ad6a322e
[ 1358.568274] RSP: 002b:00007ffd07dbae70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.577572] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f96ad6a322e
[ 1358.586382] RDX: 0000000000000001 RSI: 000055f81ac3fa10 RDI: 0000000000000007
[ 1358.595186] RBP: 00007ffd07dbae80 R08: 0000000000000000 R09: 0000000000000000
[ 1358.603994] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f96ad51b390
[ 1358.612792] R13: 000055f81ac3fa10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.621602]
[ 1358.624895] INFO: task python3:5945 is blocked on a mutex likely owned by task python3:3858.
[ 1358.635165] task:python3 state:D stack:25672 pid:3858 tgid:3858 ppid:3857 task_flags:0x400100 flags:0x00080802
[ 1358.648364] Call Trace:
[ 1358.651904]
[ 1358.655051] __schedule+0xcba/0x1940
[ 1358.659859] ? io_schedule_timeout+0x140/0x140
[ 1358.665639] ? find_held_lock+0x2b/0x80
[ 1358.670740] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.676919] schedule+0xd4/0x260
[ 1358.681338] schedule_timeout+0x13f/0x290
[ 1358.686635] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.693001] ? trace_contention_end+0xb3/0x1a0
[ 1358.698784] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.704473] ? ww_mutex_lock+0x160/0x160
[ 1358.709660] msleep+0x1b/0x30
[ 1358.713783] napi_set_threaded+0x3a5/0x440
[ 1358.719179] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.724673] netif_set_threaded+0x257/0x400
[ 1358.730163] threaded_store+0x1b8/0x220
[ 1358.735264] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.740559] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.745855] ? sysfs_kf_write+0x65/0x160
[ 1358.751052] ? sysfs_file_ops+0x140/0x140
[ 1358.756345] kernfs_fop_write_iter+0x335/0x510
[ 1358.762127] ? kernfs_ops+0xf0/0xf0
[ 1358.766835] new_sync_write+0x333/0x750
[ 1358.771935] ? __lock_acquire+0x508/0xc10
[ 1358.777230] ? new_sync_read+0x740/0x740
[ 1358.782429] ? lock_acquire.part.0+0xbc/0x260
[ 1358.788111] ? ksys_write+0x116/0x250
[ 1358.793022] vfs_write+0x657/0x9b0
[ 1358.797636] ksys_write+0x116/0x250
[ 1358.802347] ? __ia32_sys_read+0xc0/0xc0
[ 1358.807545] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.813132] ? rcu_is_watching+0x15/0xd0
[ 1358.818332] do_syscall_64+0xf3/0x650
[ 1358.823237] ? trace_hardirqs_off+0xd/0x30
[ 1358.828633] ? exc_page_fault+0xda/0xf0
[ 1358.833734] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.840199] RIP: 0033:0x7f3f1216622e
[ 1358.845006] RSP: 002b:00007ffcd6855280 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.854300] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3f1216622e
[ 1358.863107] RDX: 0000000000000001 RSI: 00005637fe556a10 RDI: 0000000000000007
[ 1358.871913] RBP: 00007ffcd6855290 R08: 0000000000000000 R09: 0000000000000000
[ 1358.880718] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3f11fde390
[ 1358.889526] R13: 00005637fe556a10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.898327]
[ 1358.901592] \x0aShowing all locks held in the system:
[ 1358.910130] 1 lock held by khungtaskd/186:
[ 1358.915524] #0: ffffffffabafdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.926996] 3 locks held by systemd-journal/807:
[ 1358.932973] #0: ff1100180e9b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1358.942858] #1: ffffffffabd03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.955671] #2: ff110001ee0c8d30 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.967228] 4 locks held by python3/3858:
[ 1358.972520] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.982407] #1: ff11000170050880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.993370] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.004526] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.014619] 4 locks held by python3/5945:
[ 1359.019913] #0: ff110001301b93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.029799] #1: ff110001c53b3280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.040761] #2: ff110017d9b70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.051920] #3: ff110017d9bf1d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.062009]
[ 1359.064474] =============================================\x0a