[ 788.670888] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 788.673452] nipa-hw-worker: [5/43] Retrying drivers/net:napi_threaded.py [ 788.821306] kselftest: Running tests in drivers/net [ 989.692762] INFO: task python3:5840 blocked for more than 122 seconds. [ 989.700975] Not tainted 7.1.0-rc1-kkmz-g8b203681ba40 #1 [ 989.708727] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.718317] task:python3 state:D stack:25032 pid:5840 tgid:5840 ppid:5839 task_flags:0x400100 flags:0x00080000 [ 989.731514] Call Trace: [ 989.735055] [ 989.738205] __schedule+0xcba/0x1940 [ 989.743026] ? io_schedule_timeout+0x140/0x140 [ 989.748811] ? find_held_lock+0x2b/0x80 [ 989.753916] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.759795] schedule+0xd4/0x260 [ 989.764213] schedule_preempt_disabled+0x18/0x30 [ 989.770192] __mutex_lock+0x6a3/0x2000 [ 989.775196] ? threaded_store+0x132/0x220 [ 989.780487] ? ww_mutex_lock+0x160/0x160 [ 989.785687] ? selinux_inode_free_security+0x330/0x330 [ 989.792260] ? cap_capable+0x1d6/0x3e0 [ 989.797270] ? threaded_store+0x132/0x220 [ 989.802571] threaded_store+0x132/0x220 [ 989.807675] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.812974] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.818273] ? sysfs_kf_write+0x65/0x160 [ 989.823474] ? sysfs_file_ops+0x140/0x140 [ 989.828770] kernfs_fop_write_iter+0x335/0x510 [ 989.834556] ? kernfs_ops+0xf0/0xf0 [ 989.839259] new_sync_write+0x333/0x750 [ 989.844361] ? __lock_acquire+0x508/0xc10 [ 989.849658] ? new_sync_read+0x740/0x740 [ 989.854859] ? lock_acquire.part.0+0xbc/0x260 [ 989.860548] ? ksys_write+0x116/0x250 [ 989.865458] vfs_write+0x657/0x9b0 [ 989.870072] ksys_write+0x116/0x250 [ 989.874783] ? __ia32_sys_read+0xc0/0xc0 [ 989.879982] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.885570] ? rcu_is_watching+0x15/0xd0 [ 989.890772] do_syscall_64+0xf3/0x650 [ 989.895679] ? trace_hardirqs_off+0xd/0x30 [ 989.901068] ? exc_page_fault+0xda/0xf0 [ 989.906172] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.912638] RIP: 0033:0x7fef9095a22e [ 989.917447] RSP: 002b:00007fffa56e7fe0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.926738] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fef9095a22e [ 989.935546] RDX: 0000000000000001 RSI: 000055a415b2ea10 RDI: 0000000000000007 [ 989.944345] RBP: 00007fffa56e7ff0 R08: 0000000000000000 R09: 0000000000000000 [ 989.953144] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fef907d2390 [ 989.961941] R13: 000055a415b2ea10 R14: 0000000000000007 R15: 0000000000000003 [ 989.970754] [ 989.974036] INFO: task python3:5840 is blocked on a mutex likely owned by task python3:3754. [ 989.984310] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802 [ 989.997510] Call Trace: [ 990.001041] [ 990.004191] __schedule+0xcba/0x1940 [ 990.009002] ? io_schedule_timeout+0x140/0x140 [ 990.014785] ? find_held_lock+0x2b/0x80 [ 990.019887] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.026067] schedule+0xd4/0x260 [ 990.030485] schedule_timeout+0x13f/0x290 [ 990.035783] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.042142] ? trace_contention_end+0xb3/0x1a0 [ 990.047926] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.053608] ? ww_mutex_lock+0x160/0x160 [ 990.058815] msleep+0x1b/0x30 [ 990.062940] napi_set_threaded+0x3a5/0x440 [ 990.068337] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.073824] netif_set_threaded+0x257/0x400 [ 990.079317] threaded_store+0x1b8/0x220 [ 990.084410] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.089706] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.095004] ? sysfs_kf_write+0x65/0x160 [ 990.100202] ? sysfs_file_ops+0x140/0x140 [ 990.105498] kernfs_fop_write_iter+0x335/0x510 [ 990.111284] ? kernfs_ops+0xf0/0xf0 [ 990.115993] new_sync_write+0x333/0x750 [ 990.121094] ? __lock_acquire+0x508/0xc10 [ 990.126392] ? new_sync_read+0x740/0x740 [ 990.131593] ? lock_acquire.part.0+0xbc/0x260 [ 990.137280] ? ksys_write+0x116/0x250 [ 990.142192] vfs_write+0x657/0x9b0 [ 990.146807] ksys_write+0x116/0x250 [ 990.151510] ? __ia32_sys_read+0xc0/0xc0 [ 990.156711] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.162301] ? rcu_is_watching+0x15/0xd0 [ 990.167502] do_syscall_64+0xf3/0x650 [ 990.172408] ? trace_hardirqs_off+0xd/0x30 [ 990.177802] ? exc_page_fault+0xda/0xf0 [ 990.182903] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.189370] RIP: 0033:0x7f285b30522e [ 990.194177] RSP: 002b:00007fff8a332f00 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.203474] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f285b30522e [ 990.212281] RDX: 0000000000000001 RSI: 00005577f1e0aa10 RDI: 0000000000000007 [ 990.221088] RBP: 00007fff8a332f10 R08: 0000000000000000 R09: 0000000000000000 [ 990.229896] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f285b17d390 [ 990.238695] R13: 00005577f1e0aa10 R14: 0000000000000007 R15: 0000000000000003 [ 990.247499] [ 990.250757] \x0aShowing all locks held in the system: [ 990.259297] 1 lock held by khungtaskd/185: [ 990.264689] #0: ffffffff9cefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.276165] 2 locks held by systemd-journal/809: [ 990.282143] #0: ff1100180dc372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 990.292029] #1: ff110001a4338378 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120 [ 990.302808] 4 locks held by python3/3754: [ 990.308102] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.317993] #1: ff11000141fb7a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.328945] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.340100] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.350193] 4 locks held by python3/5840: [ 990.355489] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.365379] #1: ff1100020bf66e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.376332] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.387491] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.397578] [ 990.400046] =============================================\x0a [ 1112.574978] INFO: task python3:5840 blocked for more than 245 seconds. [ 1112.583125] Not tainted 7.1.0-rc1-kkmz-g8b203681ba40 #1 [ 1112.590368] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.599959] task:python3 state:D stack:25032 pid:5840 tgid:5840 ppid:5839 task_flags:0x400100 flags:0x00080000 [ 1112.613162] Call Trace: [ 1112.616702] [ 1112.619850] __schedule+0xcba/0x1940 [ 1112.624664] ? io_schedule_timeout+0x140/0x140 [ 1112.630450] ? find_held_lock+0x2b/0x80 [ 1112.635557] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.641441] schedule+0xd4/0x260 [ 1112.645848] schedule_preempt_disabled+0x18/0x30 [ 1112.651826] __mutex_lock+0x6a3/0x2000 [ 1112.656828] ? threaded_store+0x132/0x220 [ 1112.662126] ? ww_mutex_lock+0x160/0x160 [ 1112.667325] ? selinux_inode_free_security+0x330/0x330 [ 1112.673895] ? cap_capable+0x1d6/0x3e0 [ 1112.678904] ? threaded_store+0x132/0x220 [ 1112.684198] threaded_store+0x132/0x220 [ 1112.689298] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.694595] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.699894] ? sysfs_kf_write+0x65/0x160 [ 1112.705092] ? sysfs_file_ops+0x140/0x140 [ 1112.710384] kernfs_fop_write_iter+0x335/0x510 [ 1112.716167] ? kernfs_ops+0xf0/0xf0 [ 1112.720875] new_sync_write+0x333/0x750 [ 1112.725977] ? __lock_acquire+0x508/0xc10 [ 1112.731274] ? new_sync_read+0x740/0x740 [ 1112.736472] ? lock_acquire.part.0+0xbc/0x260 [ 1112.742159] ? ksys_write+0x116/0x250 [ 1112.747068] vfs_write+0x657/0x9b0 [ 1112.751682] ksys_write+0x116/0x250 [ 1112.756391] ? __ia32_sys_read+0xc0/0xc0 [ 1112.761589] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.767176] ? rcu_is_watching+0x15/0xd0 [ 1112.772377] do_syscall_64+0xf3/0x650 [ 1112.777284] ? trace_hardirqs_off+0xd/0x30 [ 1112.782676] ? exc_page_fault+0xda/0xf0 [ 1112.787780] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.794247] RIP: 0033:0x7fef9095a22e [ 1112.799054] RSP: 002b:00007fffa56e7fe0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.808349] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fef9095a22e [ 1112.817156] RDX: 0000000000000001 RSI: 000055a415b2ea10 RDI: 0000000000000007 [ 1112.825962] RBP: 00007fffa56e7ff0 R08: 0000000000000000 R09: 0000000000000000 [ 1112.834767] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fef907d2390 [ 1112.843573] R13: 000055a415b2ea10 R14: 0000000000000007 R15: 0000000000000003 [ 1112.852384] [ 1112.855660] INFO: task python3:5840 is blocked on a mutex likely owned by task python3:3754. [ 1112.865930] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802 [ 1112.879130] Call Trace: [ 1112.882656] [ 1112.885805] __schedule+0xcba/0x1940 [ 1112.890615] ? io_schedule_timeout+0x140/0x140 [ 1112.896388] ? find_held_lock+0x2b/0x80 [ 1112.901490] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.907665] schedule+0xd4/0x260 [ 1112.912081] schedule_timeout+0x13f/0x290 [ 1112.917445] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.923811] ? trace_contention_end+0xb3/0x1a0 [ 1112.929597] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.935287] ? ww_mutex_lock+0x160/0x160 [ 1112.940487] msleep+0x1b/0x30 [ 1112.944611] napi_set_threaded+0x3a5/0x440 [ 1112.950007] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.955493] netif_set_threaded+0x257/0x400 [ 1112.960985] threaded_store+0x1b8/0x220 [ 1112.966089] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.971384] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.976680] ? sysfs_kf_write+0x65/0x160 [ 1112.981879] ? sysfs_file_ops+0x140/0x140 [ 1112.987174] kernfs_fop_write_iter+0x335/0x510 [ 1112.992959] ? kernfs_ops+0xf0/0xf0 [ 1112.997662] new_sync_write+0x333/0x750 [ 1113.002763] ? __lock_acquire+0x508/0xc10 [ 1113.008060] ? new_sync_read+0x740/0x740 [ 1113.013260] ? lock_acquire.part.0+0xbc/0x260 [ 1113.018945] ? ksys_write+0x116/0x250 [ 1113.023854] vfs_write+0x657/0x9b0 [ 1113.028470] ksys_write+0x116/0x250 [ 1113.033182] ? __ia32_sys_read+0xc0/0xc0 [ 1113.038373] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.043962] ? rcu_is_watching+0x15/0xd0 [ 1113.049163] do_syscall_64+0xf3/0x650 [ 1113.054071] ? trace_hardirqs_off+0xd/0x30 [ 1113.059463] ? exc_page_fault+0xda/0xf0 [ 1113.064567] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.071035] RIP: 0033:0x7f285b30522e [ 1113.075843] RSP: 002b:00007fff8a332f00 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.085138] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f285b30522e [ 1113.094014] RDX: 0000000000000001 RSI: 00005577f1e0aa10 RDI: 0000000000000007 [ 1113.102830] RBP: 00007fff8a332f10 R08: 0000000000000000 R09: 0000000000000000 [ 1113.111639] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f285b17d390 [ 1113.120448] R13: 00005577f1e0aa10 R14: 0000000000000007 R15: 0000000000000003 [ 1113.129266] [ 1113.132526] \x0aShowing all locks held in the system: [ 1113.141067] 1 lock held by khungtaskd/185: [ 1113.146460] #0: ffffffff9cefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.157948] 4 locks held by python3/3754: [ 1113.163246] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.173138] #1: ff11000141fb7a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.184100] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.195246] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.205341] 4 locks held by python3/5840: [ 1113.210637] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.220528] #1: ff1100020bf66e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.231489] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.242647] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.252727] 5 locks held by (systemd-user-)/6882: [ 1113.258803] #0: ff1100012ff113e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.268694] #1: ff110001404a0e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.279656] #2: ffffffff9cf69c20 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0 [ 1113.290327] #3: ffffffff9cd68970 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x6b0 [ 1113.301970] #4: ffffffff9cf69990 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x6b0 [ 1113.314396] [ 1113.316862] =============================================\x0a [ 1235.455215] INFO: task python3:5840 blocked for more than 368 seconds. [ 1235.463363] Not tainted 7.1.0-rc1-kkmz-g8b203681ba40 #1 [ 1235.470615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.480202] task:python3 state:D stack:25032 pid:5840 tgid:5840 ppid:5839 task_flags:0x400100 flags:0x00080002 [ 1235.493406] Call Trace: [ 1235.496946] [ 1235.500094] __schedule+0xcba/0x1940 [ 1235.504908] ? io_schedule_timeout+0x140/0x140 [ 1235.510694] ? find_held_lock+0x2b/0x80 [ 1235.515797] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.521680] schedule+0xd4/0x260 [ 1235.526096] schedule_preempt_disabled+0x18/0x30 [ 1235.532074] __mutex_lock+0x6a3/0x2000 [ 1235.537077] ? threaded_store+0x132/0x220 [ 1235.542366] ? ww_mutex_lock+0x160/0x160 [ 1235.547566] ? selinux_inode_free_security+0x330/0x330 [ 1235.554134] ? cap_capable+0x1d6/0x3e0 [ 1235.559143] ? threaded_store+0x132/0x220 [ 1235.564441] threaded_store+0x132/0x220 [ 1235.569541] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.574841] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.580140] ? sysfs_kf_write+0x65/0x160 [ 1235.585340] ? sysfs_file_ops+0x140/0x140 [ 1235.590636] kernfs_fop_write_iter+0x335/0x510 [ 1235.596419] ? kernfs_ops+0xf0/0xf0 [ 1235.601130] new_sync_write+0x333/0x750 [ 1235.606234] ? __lock_acquire+0x508/0xc10 [ 1235.611530] ? new_sync_read+0x740/0x740 [ 1235.616732] ? lock_acquire.part.0+0xbc/0x260 [ 1235.622417] ? ksys_write+0x116/0x250 [ 1235.627327] vfs_write+0x657/0x9b0 [ 1235.631938] ksys_write+0x116/0x250 [ 1235.636651] ? __ia32_sys_read+0xc0/0xc0 [ 1235.641842] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.647429] ? rcu_is_watching+0x15/0xd0 [ 1235.652628] do_syscall_64+0xf3/0x650 [ 1235.657524] ? trace_hardirqs_off+0xd/0x30 [ 1235.662920] ? exc_page_fault+0xda/0xf0 [ 1235.668023] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.674489] RIP: 0033:0x7fef9095a22e [ 1235.679296] RSP: 002b:00007fffa56e7fe0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.688591] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fef9095a22e [ 1235.697397] RDX: 0000000000000001 RSI: 000055a415b2ea10 RDI: 0000000000000007 [ 1235.706201] RBP: 00007fffa56e7ff0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.715007] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fef907d2390 [ 1235.723816] R13: 000055a415b2ea10 R14: 0000000000000007 R15: 0000000000000003 [ 1235.732629] [ 1235.735916] INFO: task python3:5840 is blocked on a mutex likely owned by task python3:3754. [ 1235.746187] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802 [ 1235.759388] Call Trace: [ 1235.762927] [ 1235.766074] __schedule+0xcba/0x1940 [ 1235.770884] ? io_schedule_timeout+0x140/0x140 [ 1235.776669] ? find_held_lock+0x2b/0x80 [ 1235.781770] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.787948] schedule+0xd4/0x260 [ 1235.792365] schedule_timeout+0x13f/0x290 [ 1235.797663] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.804023] ? trace_contention_end+0xb3/0x1a0 [ 1235.809806] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.815494] ? ww_mutex_lock+0x160/0x160 [ 1235.820686] msleep+0x1b/0x30 [ 1235.824812] napi_set_threaded+0x3a5/0x440 [ 1235.830206] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.835700] netif_set_threaded+0x257/0x400 [ 1235.841192] threaded_store+0x1b8/0x220 [ 1235.846294] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.851583] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.856870] ? sysfs_kf_write+0x65/0x160 [ 1235.862067] ? sysfs_file_ops+0x140/0x140 [ 1235.867361] kernfs_fop_write_iter+0x335/0x510 [ 1235.873144] ? kernfs_ops+0xf0/0xf0 [ 1235.877852] new_sync_write+0x333/0x750 [ 1235.882953] ? __lock_acquire+0x508/0xc10 [ 1235.888250] ? new_sync_read+0x740/0x740 [ 1235.893448] ? lock_acquire.part.0+0xbc/0x260 [ 1235.899132] ? ksys_write+0x116/0x250 [ 1235.904048] vfs_write+0x657/0x9b0 [ 1235.908662] ksys_write+0x116/0x250 [ 1235.913374] ? __ia32_sys_read+0xc0/0xc0 [ 1235.918573] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.924160] ? rcu_is_watching+0x15/0xd0 [ 1235.929357] do_syscall_64+0xf3/0x650 [ 1235.934264] ? trace_hardirqs_off+0xd/0x30 [ 1235.939655] ? exc_page_fault+0xda/0xf0 [ 1235.944756] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.951221] RIP: 0033:0x7f285b30522e [ 1235.956028] RSP: 002b:00007fff8a332f00 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.965324] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f285b30522e [ 1235.974131] RDX: 0000000000000001 RSI: 00005577f1e0aa10 RDI: 0000000000000007 [ 1235.982938] RBP: 00007fff8a332f10 R08: 0000000000000000 R09: 0000000000000000 [ 1235.991745] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f285b17d390 [ 1236.000541] R13: 00005577f1e0aa10 R14: 0000000000000007 R15: 0000000000000003 [ 1236.009354] [ 1236.012621] \x0aShowing all locks held in the system: [ 1236.021159] 1 lock held by khungtaskd/185: [ 1236.026553] #0: ffffffff9cefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.038026] 2 locks held by systemd-journal/809: [ 1236.044003] #0: ff1100180df372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1236.053888] #1: ffffffff9d1030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1236.066713] 4 locks held by python3/3754: [ 1236.072009] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.081896] #1: ff11000141fb7a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.092855] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.104011] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.114104] 4 locks held by python3/5840: [ 1236.119399] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.129289] #1: ff1100020bf66e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.140251] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.151407] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.161497] [ 1236.163952] =============================================\x0a [ 1358.336456] INFO: task python3:5840 blocked for more than 491 seconds. [ 1358.344611] Not tainted 7.1.0-rc1-kkmz-g8b203681ba40 #1 [ 1358.351861] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.361452] task:python3 state:D stack:25032 pid:5840 tgid:5840 ppid:5839 task_flags:0x400100 flags:0x00080002 [ 1358.374657] Call Trace: [ 1358.378200] [ 1358.381350] __schedule+0xcba/0x1940 [ 1358.386169] ? io_schedule_timeout+0x140/0x140 [ 1358.391954] ? find_held_lock+0x2b/0x80 [ 1358.397061] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.402948] schedule+0xd4/0x260 [ 1358.407370] schedule_preempt_disabled+0x18/0x30 [ 1358.413351] __mutex_lock+0x6a3/0x2000 [ 1358.418358] ? threaded_store+0x132/0x220 [ 1358.423657] ? ww_mutex_lock+0x160/0x160 [ 1358.428858] ? selinux_inode_free_security+0x330/0x330 [ 1358.435431] ? cap_capable+0x1d6/0x3e0 [ 1358.440504] ? threaded_store+0x132/0x220 [ 1358.445802] threaded_store+0x132/0x220 [ 1358.450907] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.456208] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.461506] ? sysfs_kf_write+0x65/0x160 [ 1358.466704] ? sysfs_file_ops+0x140/0x140 [ 1358.471999] kernfs_fop_write_iter+0x335/0x510 [ 1358.477784] ? kernfs_ops+0xf0/0xf0 [ 1358.482494] new_sync_write+0x333/0x750 [ 1358.487597] ? __lock_acquire+0x508/0xc10 [ 1358.492894] ? new_sync_read+0x740/0x740 [ 1358.498092] ? lock_acquire.part.0+0xbc/0x260 [ 1358.503779] ? ksys_write+0x116/0x250 [ 1358.508690] vfs_write+0x657/0x9b0 [ 1358.513314] ksys_write+0x116/0x250 [ 1358.518027] ? __ia32_sys_read+0xc0/0xc0 [ 1358.523226] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.528814] ? rcu_is_watching+0x15/0xd0 [ 1358.534020] do_syscall_64+0xf3/0x650 [ 1358.538993] ? trace_hardirqs_off+0xd/0x30 [ 1358.544389] ? exc_page_fault+0xda/0xf0 [ 1358.549494] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.555961] RIP: 0033:0x7fef9095a22e [ 1358.560762] RSP: 002b:00007fffa56e7fe0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.570063] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fef9095a22e [ 1358.578871] RDX: 0000000000000001 RSI: 000055a415b2ea10 RDI: 0000000000000007 [ 1358.587672] RBP: 00007fffa56e7ff0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.596479] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fef907d2390 [ 1358.605286] R13: 000055a415b2ea10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.614090] [ 1358.617418] INFO: task python3:5840 is blocked on a mutex likely owned by task python3:3754. [ 1358.627691] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802 [ 1358.640892] Call Trace: [ 1358.644429] [ 1358.647580] __schedule+0xcba/0x1940 [ 1358.652394] ? io_schedule_timeout+0x140/0x140 [ 1358.658181] ? find_held_lock+0x2b/0x80 [ 1358.663286] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.669466] schedule+0xd4/0x260 [ 1358.673886] schedule_timeout+0x13f/0x290 [ 1358.679185] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.685553] ? trace_contention_end+0xb3/0x1a0 [ 1358.691340] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.697032] ? ww_mutex_lock+0x160/0x160 [ 1358.702233] msleep+0x1b/0x30 [ 1358.706351] napi_set_threaded+0x3a5/0x440 [ 1358.711749] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.717246] netif_set_threaded+0x257/0x400 [ 1358.722742] threaded_store+0x1b8/0x220 [ 1358.727844] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.733133] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.738429] ? sysfs_kf_write+0x65/0x160 [ 1358.743628] ? sysfs_file_ops+0x140/0x140 [ 1358.748925] kernfs_fop_write_iter+0x335/0x510 [ 1358.754778] ? kernfs_ops+0xf0/0xf0 [ 1358.759488] new_sync_write+0x333/0x750 [ 1358.764592] ? __lock_acquire+0x508/0xc10 [ 1358.769889] ? new_sync_read+0x740/0x740 [ 1358.775090] ? lock_acquire.part.0+0xbc/0x260 [ 1358.780778] ? ksys_write+0x116/0x250 [ 1358.785680] vfs_write+0x657/0x9b0 [ 1358.790294] ksys_write+0x116/0x250 [ 1358.795007] ? __ia32_sys_read+0xc0/0xc0 [ 1358.800196] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.805784] ? rcu_is_watching+0x15/0xd0 [ 1358.810985] do_syscall_64+0xf3/0x650 [ 1358.815895] ? trace_hardirqs_off+0xd/0x30 [ 1358.821289] ? exc_page_fault+0xda/0xf0 [ 1358.826456] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.832924] RIP: 0033:0x7f285b30522e [ 1358.837724] RSP: 002b:00007fff8a332f00 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.847015] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f285b30522e [ 1358.855827] RDX: 0000000000000001 RSI: 00005577f1e0aa10 RDI: 0000000000000007 [ 1358.864636] RBP: 00007fff8a332f10 R08: 0000000000000000 R09: 0000000000000000 [ 1358.873446] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f285b17d390 [ 1358.882253] R13: 00005577f1e0aa10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.891066] [ 1358.894337] \x0aShowing all locks held in the system: [ 1358.902879] 1 lock held by khungtaskd/185: [ 1358.908277] #0: ffffffff9cefdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.919763] 4 locks held by python3/3754: [ 1358.925060] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.934950] #1: ff11000141fb7a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.945977] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.957136] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.967231] 4 locks held by python3/5840: [ 1358.972528] #0: ff1100012fd213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.982417] #1: ff1100020bf66e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.993370] #2: ff11000123374b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.004527] #3: ff110001233b9d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.014622] [ 1359.017090] =============================================\x0a