[ 790.905807] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 790.908215] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py [ 791.059094] kselftest: Running tests in drivers/net [ 989.719551] INFO: task python3:5918 blocked for more than 122 seconds. [ 989.727777] Not tainted 7.1.0-rc1-rwpk-gdc8800b5d2c4 #1 [ 989.735680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.745269] task:python3 state:D stack:25672 pid:5918 tgid:5918 ppid:5917 task_flags:0x400100 flags:0x00080000 [ 989.758464] Call Trace: [ 989.762001] [ 989.765152] __schedule+0xcba/0x1940 [ 989.769967] ? io_schedule_timeout+0x140/0x140 [ 989.775750] ? find_held_lock+0x2b/0x80 [ 989.780855] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.786740] schedule+0xd4/0x260 [ 989.791159] schedule_preempt_disabled+0x18/0x30 [ 989.797136] __mutex_lock+0x6a3/0x2000 [ 989.802139] ? threaded_store+0x132/0x220 [ 989.807429] ? ww_mutex_lock+0x160/0x160 [ 989.812625] ? selinux_inode_free_security+0x330/0x330 [ 989.819193] ? cap_capable+0x1d6/0x3e0 [ 989.824197] ? threaded_store+0x132/0x220 [ 989.829491] threaded_store+0x132/0x220 [ 989.834593] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.839889] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.845191] ? sysfs_kf_write+0x65/0x160 [ 989.850398] ? sysfs_file_ops+0x140/0x140 [ 989.855693] kernfs_fop_write_iter+0x335/0x510 [ 989.861478] ? kernfs_ops+0xf0/0xf0 [ 989.866187] new_sync_write+0x333/0x750 [ 989.871288] ? __lock_acquire+0x508/0xc10 [ 989.876585] ? new_sync_read+0x740/0x740 [ 989.881774] ? lock_acquire.part.0+0xbc/0x260 [ 989.887458] ? ksys_write+0x116/0x250 [ 989.892365] vfs_write+0x657/0x9b0 [ 989.896979] ksys_write+0x116/0x250 [ 989.901690] ? __ia32_sys_read+0xc0/0xc0 [ 989.906889] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.912479] ? rcu_is_watching+0x15/0xd0 [ 989.917677] do_syscall_64+0xf3/0x650 [ 989.922581] ? trace_hardirqs_off+0xd/0x30 [ 989.927977] ? exc_page_fault+0xda/0xf0 [ 989.933078] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.939544] RIP: 0033:0x7f90e5a0522e [ 989.944353] RSP: 002b:00007ffe623ac190 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.953651] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90e5a0522e [ 989.962460] RDX: 0000000000000001 RSI: 0000555ddff3ba10 RDI: 0000000000000007 [ 989.971267] RBP: 00007ffe623ac1a0 R08: 0000000000000000 R09: 0000000000000000 [ 989.980075] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90e587d390 [ 989.988881] R13: 0000555ddff3ba10 R14: 0000000000000007 R15: 0000000000000003 [ 989.997692] [ 990.001000] INFO: task python3:5918 is blocked on a mutex likely owned by task python3:3830. [ 990.011273] task:python3 state:D stack:25672 pid:3830 tgid:3830 ppid:3829 task_flags:0x400100 flags:0x00080802 [ 990.024474] Call Trace: [ 990.028010] [ 990.031160] __schedule+0xcba/0x1940 [ 990.035969] ? io_schedule_timeout+0x140/0x140 [ 990.041751] ? find_held_lock+0x2b/0x80 [ 990.046854] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.053029] schedule+0xd4/0x260 [ 990.057445] schedule_timeout+0x13f/0x290 [ 990.062741] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.069110] ? trace_contention_end+0xb3/0x1a0 [ 990.074894] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.080592] ? ww_mutex_lock+0x160/0x160 [ 990.085792] msleep+0x1b/0x30 [ 990.089907] napi_set_threaded+0x3a5/0x440 [ 990.095299] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.100794] netif_set_threaded+0x257/0x400 [ 990.106286] threaded_store+0x1b8/0x220 [ 990.111380] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.116680] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.121977] ? sysfs_kf_write+0x65/0x160 [ 990.127174] ? sysfs_file_ops+0x140/0x140 [ 990.132469] kernfs_fop_write_iter+0x335/0x510 [ 990.138251] ? kernfs_ops+0xf0/0xf0 [ 990.142963] new_sync_write+0x333/0x750 [ 990.148065] ? __lock_acquire+0x508/0xc10 [ 990.153358] ? new_sync_read+0x740/0x740 [ 990.158557] ? lock_acquire.part.0+0xbc/0x260 [ 990.164242] ? ksys_write+0x116/0x250 [ 990.169153] vfs_write+0x657/0x9b0 [ 990.173768] ksys_write+0x116/0x250 [ 990.178479] ? __ia32_sys_read+0xc0/0xc0 [ 990.183675] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.189265] ? rcu_is_watching+0x15/0xd0 [ 990.194456] do_syscall_64+0xf3/0x650 [ 990.199351] ? irq_exit_rcu+0x1a/0x30 [ 990.204256] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.210722] RIP: 0033:0x7fde79c1222e [ 990.215531] RSP: 002b:00007ffec124f170 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.224828] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fde79c1222e [ 990.233629] RDX: 0000000000000001 RSI: 000055ca50733a10 RDI: 0000000000000007 [ 990.242435] RBP: 00007ffec124f180 R08: 0000000000000000 R09: 0000000000000000 [ 990.251243] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fde79a8a390 [ 990.260042] R13: 000055ca50733a10 R14: 0000000000000007 R15: 0000000000000003 [ 990.268856] [ 990.272111] \x0aShowing all locks held in the system: [ 990.280649] 1 lock held by khungtaskd/184: [ 990.286039] #0: ffffffff948fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.297512] 3 locks held by systemd-journal/807: [ 990.303491] #0: ff1100180e9372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 990.313379] #1: ffffffff94b03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: clockevents_program_event+0x1ff/0x7e0 [ 990.326976] #2: ff110001a3680230 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 990.338530] 4 locks held by python3/3830: [ 990.343825] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.353711] #1: ff110001727c6e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.364671] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.375827] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.385929] 4 locks held by python3/5918: [ 990.391224] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.401111] #1: ff1100020846d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.412071] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.423226] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.433310] [ 990.435777] =============================================\x0a [ 1112.599677] INFO: task python3:5918 blocked for more than 245 seconds. [ 1112.607825] Not tainted 7.1.0-rc1-rwpk-gdc8800b5d2c4 #1 [ 1112.615076] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.624664] task:python3 state:D stack:25672 pid:5918 tgid:5918 ppid:5917 task_flags:0x400100 flags:0x00080000 [ 1112.637870] Call Trace: [ 1112.641411] [ 1112.644560] __schedule+0xcba/0x1940 [ 1112.649375] ? io_schedule_timeout+0x140/0x140 [ 1112.655162] ? find_held_lock+0x2b/0x80 [ 1112.660269] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.666156] schedule+0xd4/0x260 [ 1112.670574] schedule_preempt_disabled+0x18/0x30 [ 1112.676552] __mutex_lock+0x6a3/0x2000 [ 1112.681555] ? threaded_store+0x132/0x220 [ 1112.686846] ? ww_mutex_lock+0x160/0x160 [ 1112.692046] ? selinux_inode_free_security+0x330/0x330 [ 1112.698617] ? cap_capable+0x1d6/0x3e0 [ 1112.703626] ? threaded_store+0x132/0x220 [ 1112.708925] threaded_store+0x132/0x220 [ 1112.714027] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.719329] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.724628] ? sysfs_kf_write+0x65/0x160 [ 1112.729845] ? sysfs_file_ops+0x140/0x140 [ 1112.735134] kernfs_fop_write_iter+0x335/0x510 [ 1112.740917] ? kernfs_ops+0xf0/0xf0 [ 1112.745629] new_sync_write+0x333/0x750 [ 1112.750735] ? __lock_acquire+0x508/0xc10 [ 1112.756030] ? new_sync_read+0x740/0x740 [ 1112.761233] ? lock_acquire.part.0+0xbc/0x260 [ 1112.766920] ? ksys_write+0x116/0x250 [ 1112.771829] vfs_write+0x657/0x9b0 [ 1112.776443] ksys_write+0x116/0x250 [ 1112.781155] ? __ia32_sys_read+0xc0/0xc0 [ 1112.786348] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.791939] ? rcu_is_watching+0x15/0xd0 [ 1112.797142] do_syscall_64+0xf3/0x650 [ 1112.802047] ? trace_hardirqs_off+0xd/0x30 [ 1112.807442] ? exc_page_fault+0xda/0xf0 [ 1112.812546] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.819016] RIP: 0033:0x7f90e5a0522e [ 1112.823827] RSP: 002b:00007ffe623ac190 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.833125] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90e5a0522e [ 1112.841934] RDX: 0000000000000001 RSI: 0000555ddff3ba10 RDI: 0000000000000007 [ 1112.850744] RBP: 00007ffe623ac1a0 R08: 0000000000000000 R09: 0000000000000000 [ 1112.859555] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90e587d390 [ 1112.868362] R13: 0000555ddff3ba10 R14: 0000000000000007 R15: 0000000000000003 [ 1112.877175] [ 1112.880442] INFO: task python3:5918 is blocked on a mutex likely owned by task python3:3830. [ 1112.890713] task:python3 state:D stack:25672 pid:3830 tgid:3830 ppid:3829 task_flags:0x400100 flags:0x00080802 [ 1112.903914] Call Trace: [ 1112.907452] [ 1112.910605] __schedule+0xcba/0x1940 [ 1112.915419] ? io_schedule_timeout+0x140/0x140 [ 1112.921205] ? find_held_lock+0x2b/0x80 [ 1112.926310] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.932488] schedule+0xd4/0x260 [ 1112.936907] schedule_timeout+0x13f/0x290 [ 1112.942205] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.948576] ? trace_contention_end+0xb3/0x1a0 [ 1112.954364] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.960056] ? ww_mutex_lock+0x160/0x160 [ 1112.965257] msleep+0x1b/0x30 [ 1112.969383] napi_set_threaded+0x3a5/0x440 [ 1112.974781] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.980274] netif_set_threaded+0x257/0x400 [ 1112.985760] threaded_store+0x1b8/0x220 [ 1112.990863] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.996161] ? sysfs_file_kobj+0xbd/0x1e0 [ 1113.001456] ? sysfs_kf_write+0x65/0x160 [ 1113.006657] ? sysfs_file_ops+0x140/0x140 [ 1113.011953] kernfs_fop_write_iter+0x335/0x510 [ 1113.017741] ? kernfs_ops+0xf0/0xf0 [ 1113.022452] new_sync_write+0x333/0x750 [ 1113.027556] ? __lock_acquire+0x508/0xc10 [ 1113.032852] ? new_sync_read+0x740/0x740 [ 1113.038054] ? lock_acquire.part.0+0xbc/0x260 [ 1113.043741] ? ksys_write+0x116/0x250 [ 1113.048650] vfs_write+0x657/0x9b0 [ 1113.053264] ksys_write+0x116/0x250 [ 1113.057978] ? __ia32_sys_read+0xc0/0xc0 [ 1113.063175] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.068766] ? rcu_is_watching+0x15/0xd0 [ 1113.073965] do_syscall_64+0xf3/0x650 [ 1113.078871] ? irq_exit_rcu+0x1a/0x30 [ 1113.083778] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.090247] RIP: 0033:0x7fde79c1222e [ 1113.095055] RSP: 002b:00007ffec124f170 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.104352] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fde79c1222e [ 1113.113161] RDX: 0000000000000001 RSI: 000055ca50733a10 RDI: 0000000000000007 [ 1113.121971] RBP: 00007ffec124f180 R08: 0000000000000000 R09: 0000000000000000 [ 1113.130781] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fde79a8a390 [ 1113.139590] R13: 000055ca50733a10 R14: 0000000000000007 R15: 0000000000000003 [ 1113.148404] [ 1113.151666] \x0aShowing all locks held in the system: [ 1113.160205] 1 lock held by khungtaskd/184: [ 1113.165597] #0: ffffffff948fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.177069] 2 locks held by systemd-journal/807: [ 1113.183049] #0: ff1100180ddb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1113.192937] #1: ffffffff94b03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.205753] 4 locks held by python3/3830: [ 1113.211047] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.220936] #1: ff110001727c6e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.231900] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.243059] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.253152] 4 locks held by python3/5918: [ 1113.258448] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.268336] #1: ff1100020846d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.279297] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.290453] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.300543] [ 1113.303009] =============================================\x0a [ 1235.481792] INFO: task python3:5918 blocked for more than 368 seconds. [ 1235.489938] Not tainted 7.1.0-rc1-rwpk-gdc8800b5d2c4 #1 [ 1235.497188] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.506774] task:python3 state:D stack:25672 pid:5918 tgid:5918 ppid:5917 task_flags:0x400100 flags:0x00080002 [ 1235.519982] Call Trace: [ 1235.523521] [ 1235.526668] __schedule+0xcba/0x1940 [ 1235.531482] ? io_schedule_timeout+0x140/0x140 [ 1235.537267] ? find_held_lock+0x2b/0x80 [ 1235.542369] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.548254] schedule+0xd4/0x260 [ 1235.552670] schedule_preempt_disabled+0x18/0x30 [ 1235.558646] __mutex_lock+0x6a3/0x2000 [ 1235.563649] ? threaded_store+0x132/0x220 [ 1235.568947] ? ww_mutex_lock+0x160/0x160 [ 1235.574146] ? selinux_inode_free_security+0x330/0x330 [ 1235.580715] ? cap_capable+0x1d6/0x3e0 [ 1235.585719] ? threaded_store+0x132/0x220 [ 1235.591016] threaded_store+0x132/0x220 [ 1235.596107] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.601405] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.606704] ? sysfs_kf_write+0x65/0x160 [ 1235.611898] ? sysfs_file_ops+0x140/0x140 [ 1235.617195] kernfs_fop_write_iter+0x335/0x510 [ 1235.622977] ? kernfs_ops+0xf0/0xf0 [ 1235.627688] new_sync_write+0x333/0x750 [ 1235.632789] ? __lock_acquire+0x508/0xc10 [ 1235.638085] ? new_sync_read+0x740/0x740 [ 1235.643283] ? lock_acquire.part.0+0xbc/0x260 [ 1235.648969] ? ksys_write+0x116/0x250 [ 1235.653878] vfs_write+0x657/0x9b0 [ 1235.658491] ksys_write+0x116/0x250 [ 1235.663204] ? __ia32_sys_read+0xc0/0xc0 [ 1235.668392] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.673978] ? rcu_is_watching+0x15/0xd0 [ 1235.679180] do_syscall_64+0xf3/0x650 [ 1235.684086] ? trace_hardirqs_off+0xd/0x30 [ 1235.689478] ? exc_page_fault+0xda/0xf0 [ 1235.694581] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.701047] RIP: 0033:0x7f90e5a0522e [ 1235.705853] RSP: 002b:00007ffe623ac190 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.715152] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90e5a0522e [ 1235.723959] RDX: 0000000000000001 RSI: 0000555ddff3ba10 RDI: 0000000000000007 [ 1235.732763] RBP: 00007ffe623ac1a0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.741570] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90e587d390 [ 1235.750377] R13: 0000555ddff3ba10 R14: 0000000000000007 R15: 0000000000000003 [ 1235.759178] [ 1235.762441] INFO: task python3:5918 is blocked on a mutex likely owned by task python3:3830. [ 1235.772714] task:python3 state:D stack:25672 pid:3830 tgid:3830 ppid:3829 task_flags:0x400100 flags:0x00080802 [ 1235.785915] Call Trace: [ 1235.789449] [ 1235.792598] __schedule+0xcba/0x1940 [ 1235.797406] ? io_schedule_timeout+0x140/0x140 [ 1235.803187] ? find_held_lock+0x2b/0x80 [ 1235.808290] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.814465] schedule+0xd4/0x260 [ 1235.818880] schedule_timeout+0x13f/0x290 [ 1235.824175] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.830542] ? trace_contention_end+0xb3/0x1a0 [ 1235.836325] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.842015] ? ww_mutex_lock+0x160/0x160 [ 1235.847212] msleep+0x1b/0x30 [ 1235.851337] napi_set_threaded+0x3a5/0x440 [ 1235.856732] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.862223] netif_set_threaded+0x257/0x400 [ 1235.867715] threaded_store+0x1b8/0x220 [ 1235.872817] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.878114] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.883411] ? sysfs_kf_write+0x65/0x160 [ 1235.888608] ? sysfs_file_ops+0x140/0x140 [ 1235.893903] kernfs_fop_write_iter+0x335/0x510 [ 1235.899688] ? kernfs_ops+0xf0/0xf0 [ 1235.904387] new_sync_write+0x333/0x750 [ 1235.909487] ? __lock_acquire+0x508/0xc10 [ 1235.914783] ? new_sync_read+0x740/0x740 [ 1235.919983] ? lock_acquire.part.0+0xbc/0x260 [ 1235.925668] ? ksys_write+0x116/0x250 [ 1235.930576] vfs_write+0x657/0x9b0 [ 1235.935190] ksys_write+0x116/0x250 [ 1235.939902] ? __ia32_sys_read+0xc0/0xc0 [ 1235.945102] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.950682] ? rcu_is_watching+0x15/0xd0 [ 1235.955879] do_syscall_64+0xf3/0x650 [ 1235.960786] ? irq_exit_rcu+0x1a/0x30 [ 1235.965691] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.972158] RIP: 0033:0x7fde79c1222e [ 1235.976965] RSP: 002b:00007ffec124f170 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.986261] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fde79c1222e [ 1235.995062] RDX: 0000000000000001 RSI: 000055ca50733a10 RDI: 0000000000000007 [ 1236.003869] RBP: 00007ffec124f180 R08: 0000000000000000 R09: 0000000000000000 [ 1236.012678] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fde79a8a390 [ 1236.021487] R13: 000055ca50733a10 R14: 0000000000000007 R15: 0000000000000003 [ 1236.030299] [ 1236.033562] \x0aShowing all locks held in the system: [ 1236.042099] 1 lock held by khungtaskd/184: [ 1236.047490] #0: ffffffff948fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.058961] 2 locks held by systemd-journal/807: [ 1236.064939] #0: ff1100180e1372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1236.074824] #1: ffffffff94b03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1236.087654] 4 locks held by python3/3830: [ 1236.092937] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.102825] #1: ff110001727c6e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.113783] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.124941] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.135032] 4 locks held by python3/5918: [ 1236.140327] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.150214] #1: ff1100020846d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.161172] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.172328] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.182416] [ 1236.184879] =============================================\x0a [ 1358.362926] INFO: task python3:5918 blocked for more than 491 seconds. [ 1358.371073] Not tainted 7.1.0-rc1-rwpk-gdc8800b5d2c4 #1 [ 1358.378316] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.387906] task:python3 state:D stack:25672 pid:5918 tgid:5918 ppid:5917 task_flags:0x400100 flags:0x00080002 [ 1358.401113] Call Trace: [ 1358.404653] [ 1358.407801] __schedule+0xcba/0x1940 [ 1358.412614] ? io_schedule_timeout+0x140/0x140 [ 1358.418401] ? find_held_lock+0x2b/0x80 [ 1358.423507] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.429392] schedule+0xd4/0x260 [ 1358.433809] schedule_preempt_disabled+0x18/0x30 [ 1358.439786] __mutex_lock+0x6a3/0x2000 [ 1358.444789] ? threaded_store+0x132/0x220 [ 1358.450088] ? ww_mutex_lock+0x160/0x160 [ 1358.455287] ? selinux_inode_free_security+0x330/0x330 [ 1358.461856] ? cap_capable+0x1d6/0x3e0 [ 1358.466860] ? threaded_store+0x132/0x220 [ 1358.472158] threaded_store+0x132/0x220 [ 1358.477250] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.482549] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.487847] ? sysfs_kf_write+0x65/0x160 [ 1358.493038] ? sysfs_file_ops+0x140/0x140 [ 1358.498331] kernfs_fop_write_iter+0x335/0x510 [ 1358.504115] ? kernfs_ops+0xf0/0xf0 [ 1358.508826] new_sync_write+0x333/0x750 [ 1358.513931] ? __lock_acquire+0x508/0xc10 [ 1358.519227] ? new_sync_read+0x740/0x740 [ 1358.524429] ? lock_acquire.part.0+0xbc/0x260 [ 1358.530115] ? ksys_write+0x116/0x250 [ 1358.535023] vfs_write+0x657/0x9b0 [ 1358.539627] ksys_write+0x116/0x250 [ 1358.544336] ? __ia32_sys_read+0xc0/0xc0 [ 1358.549534] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.555121] ? rcu_is_watching+0x15/0xd0 [ 1358.560322] do_syscall_64+0xf3/0x650 [ 1358.565228] ? trace_hardirqs_off+0xd/0x30 [ 1358.570624] ? exc_page_fault+0xda/0xf0 [ 1358.575728] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.582194] RIP: 0033:0x7f90e5a0522e [ 1358.587002] RSP: 002b:00007ffe623ac190 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.596297] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f90e5a0522e [ 1358.605104] RDX: 0000000000000001 RSI: 0000555ddff3ba10 RDI: 0000000000000007 [ 1358.613910] RBP: 00007ffe623ac1a0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.622715] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f90e587d390 [ 1358.631522] R13: 0000555ddff3ba10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.640333] [ 1358.643605] INFO: task python3:5918 is blocked on a mutex likely owned by task python3:3830. [ 1358.653876] task:python3 state:D stack:25672 pid:3830 tgid:3830 ppid:3829 task_flags:0x400100 flags:0x00080802 [ 1358.667076] Call Trace: [ 1358.670607] [ 1358.673755] __schedule+0xcba/0x1940 [ 1358.678565] ? io_schedule_timeout+0x140/0x140 [ 1358.684338] ? find_held_lock+0x2b/0x80 [ 1358.689439] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.695618] schedule+0xd4/0x260 [ 1358.700034] schedule_timeout+0x13f/0x290 [ 1358.705331] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.711688] ? trace_contention_end+0xb3/0x1a0 [ 1358.717471] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.723161] ? ww_mutex_lock+0x160/0x160 [ 1358.728359] msleep+0x1b/0x30 [ 1358.732482] napi_set_threaded+0x3a5/0x440 [ 1358.737879] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.743370] netif_set_threaded+0x257/0x400 [ 1358.748862] threaded_store+0x1b8/0x220 [ 1358.753956] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.759243] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.764539] ? sysfs_kf_write+0x65/0x160 [ 1358.769736] ? sysfs_file_ops+0x140/0x140 [ 1358.775031] kernfs_fop_write_iter+0x335/0x510 [ 1358.780816] ? kernfs_ops+0xf0/0xf0 [ 1358.785525] new_sync_write+0x333/0x750 [ 1358.790626] ? __lock_acquire+0x508/0xc10 [ 1358.795923] ? new_sync_read+0x740/0x740 [ 1358.801123] ? lock_acquire.part.0+0xbc/0x260 [ 1358.806798] ? ksys_write+0x116/0x250 [ 1358.811705] vfs_write+0x657/0x9b0 [ 1358.816320] ksys_write+0x116/0x250 [ 1358.821032] ? __ia32_sys_read+0xc0/0xc0 [ 1358.826221] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.831811] ? rcu_is_watching+0x15/0xd0 [ 1358.836999] do_syscall_64+0xf3/0x650 [ 1358.841907] ? irq_exit_rcu+0x1a/0x30 [ 1358.846814] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.853278] RIP: 0033:0x7fde79c1222e [ 1358.858086] RSP: 002b:00007ffec124f170 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.867383] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fde79c1222e [ 1358.876186] RDX: 0000000000000001 RSI: 000055ca50733a10 RDI: 0000000000000007 [ 1358.884994] RBP: 00007ffec124f180 R08: 0000000000000000 R09: 0000000000000000 [ 1358.893803] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fde79a8a390 [ 1358.902611] R13: 000055ca50733a10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.911425] [ 1358.914693] \x0aShowing all locks held in the system: [ 1358.923230] 1 lock held by khungtaskd/184: [ 1358.928625] #0: ffffffff948fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.940096] 2 locks held by systemd-journal/807: [ 1358.946074] #0: ff1100180e8372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1358.955959] #1: ffffffff94b03120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1358.968789] 4 locks held by python3/3830: [ 1358.974081] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.983970] #1: ff110001727c6e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.994932] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.006090] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.016182] 4 locks held by python3/5918: [ 1359.021475] #0: ff1100012fda13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.031361] #1: ff1100020846d680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.042319] #2: ff1100012337ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.053471] #3: ff11000123459d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.063561] [ 1359.066017] =============================================\x0a