[ 763.167988] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 763.170661] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py [ 763.299218] kselftest: Running tests in drivers/net [ 797.157001] kworker/u129:11 (306) used greatest stack depth: 23024 bytes left [ 989.671012] INFO: task python3:5342 blocked for more than 122 seconds. [ 989.679236] Not tainted 7.1.0-rc7-tvub-g38826fe51e02 #1 [ 989.687078] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.696667] task:python3 state:D stack:24960 pid:5342 tgid:5342 ppid:5341 task_flags:0x400100 flags:0x00080000 [ 989.709874] Call Trace: [ 989.713413] [ 989.716563] __schedule+0xcc3/0x19c0 [ 989.721380] ? io_schedule_timeout+0x140/0x140 [ 989.727165] ? find_held_lock+0x2b/0x80 [ 989.732266] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.738151] schedule+0xd8/0x260 [ 989.742571] schedule_preempt_disabled+0x18/0x30 [ 989.748541] __mutex_lock+0x6af/0x2010 [ 989.753547] ? threaded_store+0x132/0x220 [ 989.758854] ? ww_mutex_lock+0x160/0x160 [ 989.764053] ? selinux_inode_free_security+0x330/0x330 [ 989.770625] ? cap_capable+0x1d6/0x3e0 [ 989.775634] ? threaded_store+0x132/0x220 [ 989.780933] threaded_store+0x132/0x220 [ 989.786037] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.791338] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.796637] ? sysfs_kf_write+0x65/0x160 [ 989.801831] ? sysfs_file_ops+0x140/0x140 [ 989.807129] kernfs_fop_write_iter+0x335/0x510 [ 989.812922] ? kernfs_ops+0xf0/0xf0 [ 989.817624] new_sync_write+0x333/0x750 [ 989.822727] ? __lock_acquire+0x508/0xc10 [ 989.828024] ? new_sync_read+0x740/0x740 [ 989.833225] ? lock_acquire.part.0+0xbc/0x260 [ 989.838911] ? ksys_write+0x116/0x250 [ 989.843823] vfs_write+0x657/0x9b0 [ 989.848438] ksys_write+0x116/0x250 [ 989.853148] ? __ia32_sys_read+0xc0/0xc0 [ 989.858346] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.863936] ? rcu_is_watching+0x15/0xd0 [ 989.869138] do_syscall_64+0xf3/0x650 [ 989.874114] ? trace_hardirqs_off+0xd/0x30 [ 989.879512] ? exc_page_fault+0xda/0xf0 [ 989.884613] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.891095] RIP: 0033:0x7f4bc78de64e [ 989.895908] RSP: 002b:00007ffe97c7d910 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.905207] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4bc78de64e [ 989.914016] RDX: 0000000000000001 RSI: 000056290c369bf0 RDI: 0000000000000007 [ 989.922822] RBP: 00007ffe97c7d920 R08: 0000000000000000 R09: 0000000000000000 [ 989.931634] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4bc7757390 [ 989.940440] R13: 000056290c369bf0 R14: 0000000000000007 R15: 00007f4bc76ed120 [ 989.949253] [ 989.952560] INFO: task python3:5342 is blocked on a mutex likely owned by task python3:3382. [ 989.962834] task:python3 state:D stack:25672 pid:3382 tgid:3382 ppid:3381 task_flags:0x400100 flags:0x00080802 [ 989.976100] Call Trace: [ 989.979641] [ 989.982790] __schedule+0xcc3/0x19c0 [ 989.987604] ? io_schedule_timeout+0x140/0x140 [ 989.993388] ? find_held_lock+0x2b/0x80 [ 989.998493] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.004673] schedule+0xd8/0x260 [ 990.009158] schedule_timeout+0x13f/0x290 [ 990.014461] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.020832] ? trace_contention_end+0xb3/0x1a0 [ 990.026618] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.032309] ? ww_mutex_lock+0x160/0x160 [ 990.037511] msleep+0x1b/0x30 [ 990.041638] napi_set_threaded+0x3a5/0x440 [ 990.047035] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.052530] netif_set_threaded+0x257/0x400 [ 990.058023] threaded_store+0x1b8/0x220 [ 990.063128] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.068426] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.073722] ? sysfs_kf_write+0x65/0x160 [ 990.078922] ? sysfs_file_ops+0x140/0x140 [ 990.084219] kernfs_fop_write_iter+0x335/0x510 [ 990.090007] ? kernfs_ops+0xf0/0xf0 [ 990.094721] new_sync_write+0x333/0x750 [ 990.099955] ? __lock_acquire+0x508/0xc10 [ 990.105251] ? new_sync_read+0x740/0x740 [ 990.110452] ? lock_acquire.part.0+0xbc/0x260 [ 990.116140] ? ksys_write+0x116/0x250 [ 990.121116] vfs_write+0x657/0x9b0 [ 990.125729] ksys_write+0x116/0x250 [ 990.130444] ? __ia32_sys_read+0xc0/0xc0 [ 990.135643] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.141311] ? rcu_is_watching+0x15/0xd0 [ 990.146506] do_syscall_64+0xf3/0x650 [ 990.151415] ? irq_exit_rcu+0x1a/0x30 [ 990.156323] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.162789] RIP: 0033:0x7ff5f749864e [ 990.167598] RSP: 002b:00007fff859a80c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.176896] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff5f749864e [ 990.185704] RDX: 0000000000000001 RSI: 00005589e9c51bf0 RDI: 0000000000000007 [ 990.194515] RBP: 00007fff859a80d0 R08: 0000000000000000 R09: 0000000000000000 [ 990.203321] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff5f7311390 [ 990.212130] R13: 00005589e9c51bf0 R14: 0000000000000007 R15: 00007ff5f72a5120 [ 990.220942] [ 990.224204] \x0aShowing all locks held in the system: [ 990.232745] 1 lock held by khungtaskd/185: [ 990.238138] #0: ffffffff910fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.249611] 2 locks held by systemd-journal/803: [ 990.255589] #0: ff1100180e4b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 990.265479] #1: ffffffff91302ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 990.278302] 4 locks held by python3/3382: [ 990.283597] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.293485] #1: ff1100013eb82c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.304448] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.315606] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.325697] 4 locks held by python3/5342: [ 990.330996] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.340888] #1: ff110001a5a16e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.351855] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.363012] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.373101] 2 locks held by systemd/6070: [ 990.378406] [ 990.380879] =============================================\x0a [ 1112.552286] INFO: task python3:5342 blocked for more than 245 seconds. [ 1112.560436] Not tainted 7.1.0-rc7-tvub-g38826fe51e02 #1 [ 1112.567687] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.577277] task:python3 state:D stack:24960 pid:5342 tgid:5342 ppid:5341 task_flags:0x400100 flags:0x00080000 [ 1112.590483] Call Trace: [ 1112.594021] [ 1112.597169] __schedule+0xcc3/0x19c0 [ 1112.601986] ? io_schedule_timeout+0x140/0x140 [ 1112.607769] ? find_held_lock+0x2b/0x80 [ 1112.612873] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.618748] schedule+0xd8/0x260 [ 1112.623164] schedule_preempt_disabled+0x18/0x30 [ 1112.629143] __mutex_lock+0x6af/0x2010 [ 1112.634146] ? threaded_store+0x132/0x220 [ 1112.639442] ? ww_mutex_lock+0x160/0x160 [ 1112.644639] ? selinux_inode_free_security+0x330/0x330 [ 1112.651207] ? cap_capable+0x1d6/0x3e0 [ 1112.656212] ? threaded_store+0x132/0x220 [ 1112.661506] threaded_store+0x132/0x220 [ 1112.666604] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.671898] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.677199] ? sysfs_kf_write+0x65/0x160 [ 1112.682398] ? sysfs_file_ops+0x140/0x140 [ 1112.687695] kernfs_fop_write_iter+0x335/0x510 [ 1112.693479] ? kernfs_ops+0xf0/0xf0 [ 1112.698190] new_sync_write+0x333/0x750 [ 1112.703286] ? __lock_acquire+0x508/0xc10 [ 1112.708574] ? new_sync_read+0x740/0x740 [ 1112.713772] ? lock_acquire.part.0+0xbc/0x260 [ 1112.719457] ? ksys_write+0x116/0x250 [ 1112.724367] vfs_write+0x657/0x9b0 [ 1112.728982] ksys_write+0x116/0x250 [ 1112.733693] ? __ia32_sys_read+0xc0/0xc0 [ 1112.738891] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.744477] ? rcu_is_watching+0x15/0xd0 [ 1112.749680] do_syscall_64+0xf3/0x650 [ 1112.754584] ? trace_hardirqs_off+0xd/0x30 [ 1112.759979] ? exc_page_fault+0xda/0xf0 [ 1112.765079] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.771547] RIP: 0033:0x7f4bc78de64e [ 1112.776357] RSP: 002b:00007ffe97c7d910 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.785646] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4bc78de64e [ 1112.794455] RDX: 0000000000000001 RSI: 000056290c369bf0 RDI: 0000000000000007 [ 1112.803264] RBP: 00007ffe97c7d920 R08: 0000000000000000 R09: 0000000000000000 [ 1112.812072] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4bc7757390 [ 1112.820872] R13: 000056290c369bf0 R14: 0000000000000007 R15: 00007f4bc76ed120 [ 1112.829682] [ 1112.832970] INFO: task python3:5342 is blocked on a mutex likely owned by task python3:3382. [ 1112.843240] task:python3 state:D stack:25672 pid:3382 tgid:3382 ppid:3381 task_flags:0x400100 flags:0x00080802 [ 1112.856444] Call Trace: [ 1112.859983] [ 1112.863134] __schedule+0xcc3/0x19c0 [ 1112.867934] ? io_schedule_timeout+0x140/0x140 [ 1112.873714] ? find_held_lock+0x2b/0x80 [ 1112.878813] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.884989] schedule+0xd8/0x260 [ 1112.889405] schedule_timeout+0x13f/0x290 [ 1112.894700] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.901066] ? trace_contention_end+0xb3/0x1a0 [ 1112.906850] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.912542] ? ww_mutex_lock+0x160/0x160 [ 1112.917742] msleep+0x1b/0x30 [ 1112.921857] napi_set_threaded+0x3a5/0x440 [ 1112.927249] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.932742] netif_set_threaded+0x257/0x400 [ 1112.938225] threaded_store+0x1b8/0x220 [ 1112.943327] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.948622] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.953915] ? sysfs_kf_write+0x65/0x160 [ 1112.959113] ? sysfs_file_ops+0x140/0x140 [ 1112.964406] kernfs_fop_write_iter+0x335/0x510 [ 1112.970190] ? kernfs_ops+0xf0/0xf0 [ 1112.974902] new_sync_write+0x333/0x750 [ 1112.980002] ? __lock_acquire+0x508/0xc10 [ 1112.985296] ? new_sync_read+0x740/0x740 [ 1112.990496] ? lock_acquire.part.0+0xbc/0x260 [ 1112.996185] ? ksys_write+0x116/0x250 [ 1113.001095] vfs_write+0x657/0x9b0 [ 1113.005708] ksys_write+0x116/0x250 [ 1113.010419] ? __ia32_sys_read+0xc0/0xc0 [ 1113.015616] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.021205] ? rcu_is_watching+0x15/0xd0 [ 1113.026403] do_syscall_64+0xf3/0x650 [ 1113.031309] ? irq_exit_rcu+0x1a/0x30 [ 1113.036216] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.042680] RIP: 0033:0x7ff5f749864e [ 1113.047485] RSP: 002b:00007fff859a80c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.056780] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff5f749864e [ 1113.065586] RDX: 0000000000000001 RSI: 00005589e9c51bf0 RDI: 0000000000000007 [ 1113.074390] RBP: 00007fff859a80d0 R08: 0000000000000000 R09: 0000000000000000 [ 1113.083197] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff5f7311390 [ 1113.092001] R13: 00005589e9c51bf0 R14: 0000000000000007 R15: 00007ff5f72a5120 [ 1113.100811] [ 1113.104075] \x0aShowing all locks held in the system: [ 1113.112611] 1 lock held by khungtaskd/185: [ 1113.118002] #0: ffffffff910fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.129470] 2 locks held by systemd-journal/803: [ 1113.135446] #0: ff1100180e1b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1113.145330] #1: ffffffff91302ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.158155] 4 locks held by python3/3382: [ 1113.163448] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.173337] #1: ff1100013eb82c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.184298] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.195445] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.205538] 4 locks held by python3/5342: [ 1113.210833] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.220720] #1: ff110001a5a16e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.231682] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.242836] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.252924] [ 1113.255388] =============================================\x0a [ 1235.433568] INFO: task python3:5342 blocked for more than 368 seconds. [ 1235.441714] Not tainted 7.1.0-rc7-tvub-g38826fe51e02 #1 [ 1235.448966] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.458553] task:python3 state:D stack:24960 pid:5342 tgid:5342 ppid:5341 task_flags:0x400100 flags:0x00080002 [ 1235.471757] Call Trace: [ 1235.475295] [ 1235.478443] __schedule+0xcc3/0x19c0 [ 1235.483261] ? io_schedule_timeout+0x140/0x140 [ 1235.489045] ? find_held_lock+0x2b/0x80 [ 1235.494143] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.500027] schedule+0xd8/0x260 [ 1235.504443] schedule_preempt_disabled+0x18/0x30 [ 1235.510422] __mutex_lock+0x6af/0x2010 [ 1235.515425] ? threaded_store+0x132/0x220 [ 1235.520722] ? ww_mutex_lock+0x160/0x160 [ 1235.525920] ? selinux_inode_free_security+0x330/0x330 [ 1235.532480] ? cap_capable+0x1d6/0x3e0 [ 1235.537487] ? threaded_store+0x132/0x220 [ 1235.542782] threaded_store+0x132/0x220 [ 1235.547886] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.553182] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.558481] ? sysfs_kf_write+0x65/0x160 [ 1235.563681] ? sysfs_file_ops+0x140/0x140 [ 1235.568978] kernfs_fop_write_iter+0x335/0x510 [ 1235.574765] ? kernfs_ops+0xf0/0xf0 [ 1235.579474] new_sync_write+0x333/0x750 [ 1235.584578] ? __lock_acquire+0x508/0xc10 [ 1235.589866] ? new_sync_read+0x740/0x740 [ 1235.595066] ? lock_acquire.part.0+0xbc/0x260 [ 1235.600746] ? ksys_write+0x116/0x250 [ 1235.605660] vfs_write+0x657/0x9b0 [ 1235.610275] ksys_write+0x116/0x250 [ 1235.614989] ? __ia32_sys_read+0xc0/0xc0 [ 1235.620188] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.625769] ? rcu_is_watching+0x15/0xd0 [ 1235.630970] do_syscall_64+0xf3/0x650 [ 1235.635867] ? trace_hardirqs_off+0xd/0x30 [ 1235.641264] ? exc_page_fault+0xda/0xf0 [ 1235.646367] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.652836] RIP: 0033:0x7f4bc78de64e [ 1235.657648] RSP: 002b:00007ffe97c7d910 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.666947] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4bc78de64e [ 1235.675757] RDX: 0000000000000001 RSI: 000056290c369bf0 RDI: 0000000000000007 [ 1235.684566] RBP: 00007ffe97c7d920 R08: 0000000000000000 R09: 0000000000000000 [ 1235.693374] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4bc7757390 [ 1235.702183] R13: 000056290c369bf0 R14: 0000000000000007 R15: 00007f4bc76ed120 [ 1235.710996] [ 1235.714311] INFO: task python3:5342 is blocked on a mutex likely owned by task python3:3382. [ 1235.724586] task:python3 state:D stack:25672 pid:3382 tgid:3382 ppid:3381 task_flags:0x400100 flags:0x00080802 [ 1235.737790] Call Trace: [ 1235.741330] [ 1235.744483] __schedule+0xcc3/0x19c0 [ 1235.749293] ? io_schedule_timeout+0x140/0x140 [ 1235.755077] ? find_held_lock+0x2b/0x80 [ 1235.760183] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.766361] schedule+0xd8/0x260 [ 1235.770781] schedule_timeout+0x13f/0x290 [ 1235.776080] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.782448] ? trace_contention_end+0xb3/0x1a0 [ 1235.788226] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.793918] ? ww_mutex_lock+0x160/0x160 [ 1235.799118] msleep+0x1b/0x30 [ 1235.803243] napi_set_threaded+0x3a5/0x440 [ 1235.808638] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.814132] netif_set_threaded+0x257/0x400 [ 1235.819625] threaded_store+0x1b8/0x220 [ 1235.824729] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.830026] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.835322] ? sysfs_kf_write+0x65/0x160 [ 1235.840522] ? sysfs_file_ops+0x140/0x140 [ 1235.845808] kernfs_fop_write_iter+0x335/0x510 [ 1235.851592] ? kernfs_ops+0xf0/0xf0 [ 1235.856302] new_sync_write+0x333/0x750 [ 1235.861405] ? __lock_acquire+0x508/0xc10 [ 1235.866704] ? new_sync_read+0x740/0x740 [ 1235.871903] ? lock_acquire.part.0+0xbc/0x260 [ 1235.877589] ? ksys_write+0x116/0x250 [ 1235.882500] vfs_write+0x657/0x9b0 [ 1235.887115] ksys_write+0x116/0x250 [ 1235.891830] ? __ia32_sys_read+0xc0/0xc0 [ 1235.897029] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.902620] ? rcu_is_watching+0x15/0xd0 [ 1235.907819] do_syscall_64+0xf3/0x650 [ 1235.912723] ? irq_exit_rcu+0x1a/0x30 [ 1235.917633] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.924101] RIP: 0033:0x7ff5f749864e [ 1235.928911] RSP: 002b:00007fff859a80c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.938198] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff5f749864e [ 1235.947008] RDX: 0000000000000001 RSI: 00005589e9c51bf0 RDI: 0000000000000007 [ 1235.955817] RBP: 00007fff859a80d0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.964627] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff5f7311390 [ 1235.973434] R13: 00005589e9c51bf0 R14: 0000000000000007 R15: 00007ff5f72a5120 [ 1235.982240] [ 1235.985509] \x0aShowing all locks held in the system: [ 1235.994050] 1 lock held by khungtaskd/185: [ 1235.999445] #0: ffffffff910fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.010926] 4 locks held by python3/3382: [ 1236.016222] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.026111] #1: ff1100013eb82c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.037063] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.048223] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.058321] 4 locks held by python3/5342: [ 1236.063619] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.073508] #1: ff110001a5a16e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.084469] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.095629] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.105723] [ 1236.108191] =============================================\x0a [ 1358.314846] INFO: task python3:5342 blocked for more than 491 seconds. [ 1358.322990] Not tainted 7.1.0-rc7-tvub-g38826fe51e02 #1 [ 1358.330239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.339827] task:python3 state:D stack:24960 pid:5342 tgid:5342 ppid:5341 task_flags:0x400100 flags:0x00080002 [ 1358.353025] Call Trace: [ 1358.356562] [ 1358.359709] __schedule+0xcc3/0x19c0 [ 1358.364524] ? io_schedule_timeout+0x140/0x140 [ 1358.370308] ? find_held_lock+0x2b/0x80 [ 1358.375409] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.381296] schedule+0xd8/0x260 [ 1358.385711] schedule_preempt_disabled+0x18/0x30 [ 1358.391690] __mutex_lock+0x6af/0x2010 [ 1358.396704] ? threaded_store+0x132/0x220 [ 1358.402001] ? ww_mutex_lock+0x160/0x160 [ 1358.407201] ? selinux_inode_free_security+0x330/0x330 [ 1358.413771] ? cap_capable+0x1d6/0x3e0 [ 1358.418778] ? threaded_store+0x132/0x220 [ 1358.424063] threaded_store+0x132/0x220 [ 1358.429165] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.434454] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.439752] ? sysfs_kf_write+0x65/0x160 [ 1358.444952] ? sysfs_file_ops+0x140/0x140 [ 1358.450247] kernfs_fop_write_iter+0x335/0x510 [ 1358.456032] ? kernfs_ops+0xf0/0xf0 [ 1358.460740] new_sync_write+0x333/0x750 [ 1358.465844] ? __lock_acquire+0x508/0xc10 [ 1358.471137] ? new_sync_read+0x740/0x740 [ 1358.476338] ? lock_acquire.part.0+0xbc/0x260 [ 1358.482022] ? ksys_write+0x116/0x250 [ 1358.486930] vfs_write+0x657/0x9b0 [ 1358.491544] ksys_write+0x116/0x250 [ 1358.496257] ? __ia32_sys_read+0xc0/0xc0 [ 1358.501453] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.507042] ? rcu_is_watching+0x15/0xd0 [ 1358.512234] do_syscall_64+0xf3/0x650 [ 1358.517141] ? trace_hardirqs_off+0xd/0x30 [ 1358.522534] ? exc_page_fault+0xda/0xf0 [ 1358.527635] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.534101] RIP: 0033:0x7f4bc78de64e [ 1358.538909] RSP: 002b:00007ffe97c7d910 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.548206] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4bc78de64e [ 1358.557014] RDX: 0000000000000001 RSI: 000056290c369bf0 RDI: 0000000000000007 [ 1358.565819] RBP: 00007ffe97c7d920 R08: 0000000000000000 R09: 0000000000000000 [ 1358.574623] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4bc7757390 [ 1358.583432] R13: 000056290c369bf0 R14: 0000000000000007 R15: 00007f4bc76ed120 [ 1358.592236] [ 1358.595513] INFO: task python3:5342 is blocked on a mutex likely owned by task python3:3382. [ 1358.605783] task:python3 state:D stack:25672 pid:3382 tgid:3382 ppid:3381 task_flags:0x400100 flags:0x00080802 [ 1358.618984] Call Trace: [ 1358.622519] [ 1358.625667] __schedule+0xcc3/0x19c0 [ 1358.630478] ? io_schedule_timeout+0x140/0x140 [ 1358.636263] ? find_held_lock+0x2b/0x80 [ 1358.641363] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.647539] schedule+0xd8/0x260 [ 1358.651958] schedule_timeout+0x13f/0x290 [ 1358.657251] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.663617] ? trace_contention_end+0xb3/0x1a0 [ 1358.669402] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.675091] ? ww_mutex_lock+0x160/0x160 [ 1358.680287] msleep+0x1b/0x30 [ 1358.684410] napi_set_threaded+0x3a5/0x440 [ 1358.689804] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.695297] netif_set_threaded+0x257/0x400 [ 1358.700782] threaded_store+0x1b8/0x220 [ 1358.705881] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.711176] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.716469] ? sysfs_kf_write+0x65/0x160 [ 1358.721665] ? sysfs_file_ops+0x140/0x140 [ 1358.726961] kernfs_fop_write_iter+0x335/0x510 [ 1358.732745] ? kernfs_ops+0xf0/0xf0 [ 1358.737456] new_sync_write+0x333/0x750 [ 1358.742559] ? __lock_acquire+0x508/0xc10 [ 1358.747844] ? new_sync_read+0x740/0x740 [ 1358.753041] ? lock_acquire.part.0+0xbc/0x260 [ 1358.758725] ? ksys_write+0x116/0x250 [ 1358.763635] vfs_write+0x657/0x9b0 [ 1358.768249] ksys_write+0x116/0x250 [ 1358.772960] ? __ia32_sys_read+0xc0/0xc0 [ 1358.778160] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.783749] ? rcu_is_watching+0x15/0xd0 [ 1358.788947] do_syscall_64+0xf3/0x650 [ 1358.793853] ? irq_exit_rcu+0x1a/0x30 [ 1358.798762] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.805226] RIP: 0033:0x7ff5f749864e [ 1358.810034] RSP: 002b:00007fff859a80c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.819330] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff5f749864e [ 1358.828127] RDX: 0000000000000001 RSI: 00005589e9c51bf0 RDI: 0000000000000007 [ 1358.836933] RBP: 00007fff859a80d0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.845741] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff5f7311390 [ 1358.854546] R13: 00005589e9c51bf0 R14: 0000000000000007 R15: 00007ff5f72a5120 [ 1358.863356] [ 1358.866619] \x0aShowing all locks held in the system: [ 1358.875154] 1 lock held by khungtaskd/185: [ 1358.880548] #0: ffffffff910fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.892026] 4 locks held by python3/3382: [ 1358.897321] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.907209] #1: ff1100013eb82c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.918168] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.929325] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.939415] 4 locks held by python3/5342: [ 1358.944710] #0: ff110001309ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.954598] #1: ff110001a5a16e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.965552] #2: ff11000112baab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.976713] #3: ff11000112b99e00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.986804] [ 1358.989270] =============================================\x0a