[ 780.074412] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 780.076938] nipa-hw-worker: [4/40] Retrying drivers/net:napi_threaded.py
[ 780.229197] kselftest: Running tests in drivers/net
[ 989.655268] INFO: task python3:5603 blocked for more than 122 seconds.
[ 989.663508] Not tainted 7.0.0-rc6-twpr-ga220ed2c96ca #1
[ 989.670760] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.680352] task:python3 state:D stack:25336 pid:5603 tgid:5603 ppid:5602 task_flags:0x400100 flags:0x00080000
[ 989.693559] Call Trace:
[ 989.697098]
[ 989.700250] __schedule+0xae3/0x14d0
[ 989.705067] ? io_schedule_timeout+0x140/0x140
[ 989.710858] ? find_held_lock+0x2b/0x80
[ 989.715966] schedule+0xd4/0x260
[ 989.720387] schedule_preempt_disabled+0x18/0x30
[ 989.726366] __mutex_lock+0x6b2/0x1ee0
[ 989.731372] ? rcu_is_watching+0x15/0xd0
[ 989.736569] ? threaded_store+0x132/0x220
[ 989.741868] ? ww_mutex_lock+0x160/0x160
[ 989.747067] ? selinux_inode_free_security+0x330/0x330
[ 989.753642] ? cap_capable+0x1ed/0x3f0
[ 989.758651] ? threaded_store+0x132/0x220
[ 989.763947] threaded_store+0x132/0x220
[ 989.769050] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.774355] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.779652] ? sysfs_kf_write+0x65/0x160
[ 989.784851] ? sysfs_file_ops+0x140/0x140
[ 989.790145] kernfs_fop_write_iter+0x335/0x510
[ 989.795928] ? kernfs_ops+0xf0/0xf0
[ 989.800636] new_sync_write+0x333/0x750
[ 989.805739] ? __lock_acquire+0x508/0xc10
[ 989.811032] ? new_sync_read+0x740/0x740
[ 989.816233] ? lock_acquire.part.0+0xbc/0x260
[ 989.821919] ? ksys_write+0x116/0x250
[ 989.826831] vfs_write+0x657/0x9b0
[ 989.831446] ksys_write+0x116/0x250
[ 989.836149] ? __ia32_sys_read+0xc0/0xc0
[ 989.841346] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.847003] ? rcu_is_watching+0x15/0xd0
[ 989.852200] do_syscall_64+0xf3/0x5e0
[ 989.857106] ? trace_hardirqs_off+0xd/0x30
[ 989.862500] ? exc_page_fault+0xda/0xf0
[ 989.867599] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.874065] RIP: 0033:0x7fd8ceee222e
[ 989.878877] RSP: 002b:00007fffe5baae90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.888173] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd8ceee222e
[ 989.896981] RDX: 0000000000000001 RSI: 000055a6005b7f40 RDI: 0000000000000007
[ 989.905788] RBP: 00007fffe5baaea0 R08: 0000000000000000 R09: 0000000000000000
[ 989.914595] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd8ced5a390
[ 989.923394] R13: 000055a6005b7f40 R14: 0000000000000007 R15: 0000000000000003
[ 989.932221]
[ 989.935562] INFO: task python3:5603 is blocked on a mutex likely owned by task python3:3544.
[ 989.945834] task:python3 state:D stack:25336 pid:3544 tgid:3544 ppid:3543 task_flags:0x400100 flags:0x00080802
[ 989.959036] Call Trace:
[ 989.962575]
[ 989.965722] __schedule+0xae3/0x14d0
[ 989.970532] ? io_schedule_timeout+0x140/0x140
[ 989.976314] ? find_held_lock+0x2b/0x80
[ 989.981416] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.987582] schedule+0xd4/0x260
[ 989.991998] schedule_timeout+0x13f/0x290
[ 989.997292] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 990.003660] ? trace_contention_end+0xb3/0x1a0
[ 990.009444] ? hrtimers_cpu_dying+0x260/0x260
[ 990.015135] ? ww_mutex_lock+0x160/0x160
[ 990.020332] msleep+0x1b/0x30
[ 990.024455] napi_set_threaded+0x3a5/0x440
[ 990.029851] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.035345] netif_set_threaded+0x257/0x400
[ 990.040903] threaded_store+0x1b8/0x220
[ 990.046069] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.051370] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.056665] ? sysfs_kf_write+0x65/0x160
[ 990.061863] ? sysfs_file_ops+0x140/0x140
[ 990.067158] kernfs_fop_write_iter+0x335/0x510
[ 990.072940] ? kernfs_ops+0xf0/0xf0
[ 990.077650] new_sync_write+0x333/0x750
[ 990.082752] ? __lock_acquire+0x508/0xc10
[ 990.088045] ? new_sync_read+0x740/0x740
[ 990.093245] ? lock_acquire.part.0+0xbc/0x260
[ 990.098922] ? ksys_write+0x116/0x250
[ 990.103828] vfs_write+0x657/0x9b0
[ 990.108444] ksys_write+0x116/0x250
[ 990.113147] ? __ia32_sys_read+0xc0/0xc0
[ 990.118334] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.123924] ? rcu_is_watching+0x15/0xd0
[ 990.129114] do_syscall_64+0xf3/0x5e0
[ 990.134019] ? irq_exit_rcu+0x39/0xf0
[ 990.138927] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.145390] RIP: 0033:0x7f67cef5122e
[ 990.150199] RSP: 002b:00007ffcdb6690d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.159497] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f67cef5122e
[ 990.168302] RDX: 0000000000000001 RSI: 00005563b6399f40 RDI: 0000000000000007
[ 990.177109] RBP: 00007ffcdb6690e0 R08: 0000000000000000 R09: 0000000000000000
[ 990.185916] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f67cedc9390
[ 990.194715] R13: 00005563b6399f40 R14: 0000000000000007 R15: 0000000000000003
[ 990.203526]
[ 990.206782] \x0aShowing all locks held in the system:
[ 990.215323] 1 lock held by khungtaskd/184:
[ 990.220715] #0: ffffffff99ecdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.232199] 2 locks held by systemd-journal/803:
[ 990.238177] #0: ff1100180de372e0 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2c/0x40
[ 990.249044] #1: ffffffff9a0c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.261874] 4 locks held by python3/3544:
[ 990.267159] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.277046] #1: ff1100016adb7488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.288009] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.299165] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.309275] 4 locks held by python3/5603:
[ 990.314572] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.324458] #1: ff110001c10d2c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.335419] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.346574] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.356666]
[ 990.359131] =============================================\x0a
[ 1112.537414] INFO: task python3:5603 blocked for more than 245 seconds.
[ 1112.545560] Not tainted 7.0.0-rc6-twpr-ga220ed2c96ca #1
[ 1112.552812] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.562400] task:python3 state:D stack:25336 pid:5603 tgid:5603 ppid:5602 task_flags:0x400100 flags:0x00080000
[ 1112.575605] Call Trace:
[ 1112.579146]
[ 1112.582300] __schedule+0xae3/0x14d0
[ 1112.587115] ? io_schedule_timeout+0x140/0x140
[ 1112.592903] ? find_held_lock+0x2b/0x80
[ 1112.598015] schedule+0xd4/0x260
[ 1112.602434] schedule_preempt_disabled+0x18/0x30
[ 1112.608405] __mutex_lock+0x6b2/0x1ee0
[ 1112.613412] ? rcu_is_watching+0x15/0xd0
[ 1112.618614] ? threaded_store+0x132/0x220
[ 1112.623915] ? ww_mutex_lock+0x160/0x160
[ 1112.629112] ? selinux_inode_free_security+0x330/0x330
[ 1112.635676] ? cap_capable+0x1ed/0x3f0
[ 1112.640682] ? threaded_store+0x132/0x220
[ 1112.645979] threaded_store+0x132/0x220
[ 1112.651082] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.656382] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.661680] ? sysfs_kf_write+0x65/0x160
[ 1112.666878] ? sysfs_file_ops+0x140/0x140
[ 1112.672173] kernfs_fop_write_iter+0x335/0x510
[ 1112.677959] ? kernfs_ops+0xf0/0xf0
[ 1112.682664] new_sync_write+0x333/0x750
[ 1112.687766] ? __lock_acquire+0x508/0xc10
[ 1112.693065] ? new_sync_read+0x740/0x740
[ 1112.698264] ? lock_acquire.part.0+0xbc/0x260
[ 1112.703950] ? ksys_write+0x116/0x250
[ 1112.708861] vfs_write+0x657/0x9b0
[ 1112.713477] ksys_write+0x116/0x250
[ 1112.718187] ? __ia32_sys_read+0xc0/0xc0
[ 1112.723387] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.728977] ? rcu_is_watching+0x15/0xd0
[ 1112.734179] do_syscall_64+0xf3/0x5e0
[ 1112.739086] ? trace_hardirqs_off+0xd/0x30
[ 1112.744484] ? exc_page_fault+0xda/0xf0
[ 1112.749584] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.756053] RIP: 0033:0x7fd8ceee222e
[ 1112.760864] RSP: 002b:00007fffe5baae90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.770165] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd8ceee222e
[ 1112.778972] RDX: 0000000000000001 RSI: 000055a6005b7f40 RDI: 0000000000000007
[ 1112.787781] RBP: 00007fffe5baaea0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.796589] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd8ced5a390
[ 1112.805399] R13: 000055a6005b7f40 R14: 0000000000000007 R15: 0000000000000003
[ 1112.814212]
[ 1112.817484] INFO: task python3:5603 is blocked on a mutex likely owned by task python3:3544.
[ 1112.827758] task:python3 state:D stack:25336 pid:3544 tgid:3544 ppid:3543 task_flags:0x400100 flags:0x00080802
[ 1112.840959] Call Trace:
[ 1112.844499]
[ 1112.847650] __schedule+0xae3/0x14d0
[ 1112.852461] ? io_schedule_timeout+0x140/0x140
[ 1112.858247] ? find_held_lock+0x2b/0x80
[ 1112.863351] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.869531] schedule+0xd4/0x260
[ 1112.873952] schedule_timeout+0x13f/0x290
[ 1112.879253] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1112.885622] ? trace_contention_end+0xb3/0x1a0
[ 1112.891408] ? hrtimers_cpu_dying+0x260/0x260
[ 1112.897098] ? ww_mutex_lock+0x160/0x160
[ 1112.902297] msleep+0x1b/0x30
[ 1112.906421] napi_set_threaded+0x3a5/0x440
[ 1112.911818] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.917312] netif_set_threaded+0x257/0x400
[ 1112.922804] threaded_store+0x1b8/0x220
[ 1112.927899] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.933195] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.938491] ? sysfs_kf_write+0x65/0x160
[ 1112.943689] ? sysfs_file_ops+0x140/0x140
[ 1112.948984] kernfs_fop_write_iter+0x335/0x510
[ 1112.954770] ? kernfs_ops+0xf0/0xf0
[ 1112.959479] new_sync_write+0x333/0x750
[ 1112.964582] ? __lock_acquire+0x508/0xc10
[ 1112.969879] ? new_sync_read+0x740/0x740
[ 1112.975073] ? lock_acquire.part.0+0xbc/0x260
[ 1112.980761] ? ksys_write+0x116/0x250
[ 1112.985674] vfs_write+0x657/0x9b0
[ 1112.990290] ksys_write+0x116/0x250
[ 1112.995002] ? __ia32_sys_read+0xc0/0xc0
[ 1113.000201] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.005790] ? rcu_is_watching+0x15/0xd0
[ 1113.010991] do_syscall_64+0xf3/0x5e0
[ 1113.015898] ? irq_exit_rcu+0x39/0xf0
[ 1113.020808] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.027274] RIP: 0033:0x7f67cef5122e
[ 1113.032082] RSP: 002b:00007ffcdb6690d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.041378] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f67cef5122e
[ 1113.050186] RDX: 0000000000000001 RSI: 00005563b6399f40 RDI: 0000000000000007
[ 1113.058984] RBP: 00007ffcdb6690e0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.067783] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f67cedc9390
[ 1113.076582] R13: 00005563b6399f40 R14: 0000000000000007 R15: 0000000000000003
[ 1113.085395]
[ 1113.088662] \x0aShowing all locks held in the system:
[ 1113.097201] 1 lock held by khungtaskd/184:
[ 1113.102595] #0: ffffffff99ecdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.114068] 2 locks held by systemd-journal/803:
[ 1113.120046] #0: ff1100180de372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1113.129933] #1: ffffffff9a0c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.142749] 4 locks held by python3/3544:
[ 1113.148046] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.157938] #1: ff1100016adb7488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.168899] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.180057] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.190156] 4 locks held by python3/5603:
[ 1113.195455] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.205343] #1: ff110001c10d2c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.216298] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.227456] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.237547]
[ 1113.240015] =============================================\x0a
[ 1235.418576] INFO: task python3:5603 blocked for more than 368 seconds.
[ 1235.426722] Not tainted 7.0.0-rc6-twpr-ga220ed2c96ca #1
[ 1235.433962] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.443548] task:python3 state:D stack:25336 pid:5603 tgid:5603 ppid:5602 task_flags:0x400100 flags:0x00080002
[ 1235.456753] Call Trace:
[ 1235.460292]
[ 1235.463440] __schedule+0xae3/0x14d0
[ 1235.468252] ? io_schedule_timeout+0x140/0x140
[ 1235.474038] ? find_held_lock+0x2b/0x80
[ 1235.479144] schedule+0xd4/0x260
[ 1235.483563] schedule_preempt_disabled+0x18/0x30
[ 1235.489542] __mutex_lock+0x6b2/0x1ee0
[ 1235.494548] ? rcu_is_watching+0x15/0xd0
[ 1235.499746] ? threaded_store+0x132/0x220
[ 1235.505037] ? ww_mutex_lock+0x160/0x160
[ 1235.510233] ? selinux_inode_free_security+0x330/0x330
[ 1235.516800] ? cap_capable+0x1ed/0x3f0
[ 1235.521797] ? threaded_store+0x132/0x220
[ 1235.527093] threaded_store+0x132/0x220
[ 1235.532194] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.537490] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.542785] ? sysfs_kf_write+0x65/0x160
[ 1235.547985] ? sysfs_file_ops+0x140/0x140
[ 1235.553279] kernfs_fop_write_iter+0x335/0x510
[ 1235.559061] ? kernfs_ops+0xf0/0xf0
[ 1235.563770] new_sync_write+0x333/0x750
[ 1235.568874] ? __lock_acquire+0x508/0xc10
[ 1235.574167] ? new_sync_read+0x740/0x740
[ 1235.579366] ? lock_acquire.part.0+0xbc/0x260
[ 1235.585043] ? ksys_write+0x116/0x250
[ 1235.589950] vfs_write+0x657/0x9b0
[ 1235.594562] ksys_write+0x116/0x250
[ 1235.599270] ? __ia32_sys_read+0xc0/0xc0
[ 1235.604469] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.610056] ? rcu_is_watching+0x15/0xd0
[ 1235.615254] do_syscall_64+0xf3/0x5e0
[ 1235.620160] ? trace_hardirqs_off+0xd/0x30
[ 1235.625556] ? exc_page_fault+0xda/0xf0
[ 1235.630660] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.637125] RIP: 0033:0x7fd8ceee222e
[ 1235.641934] RSP: 002b:00007fffe5baae90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.651230] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd8ceee222e
[ 1235.660036] RDX: 0000000000000001 RSI: 000055a6005b7f40 RDI: 0000000000000007
[ 1235.668844] RBP: 00007fffe5baaea0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.677653] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd8ced5a390
[ 1235.686458] R13: 000055a6005b7f40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.695272]
[ 1235.698570] INFO: task python3:5603 is blocked on a mutex likely owned by task python3:3544.
[ 1235.708840] task:python3 state:D stack:25336 pid:3544 tgid:3544 ppid:3543 task_flags:0x400100 flags:0x00080802
[ 1235.722041] Call Trace:
[ 1235.725577]
[ 1235.728724] __schedule+0xae3/0x14d0
[ 1235.733535] ? io_schedule_timeout+0x140/0x140
[ 1235.739317] ? find_held_lock+0x2b/0x80
[ 1235.744425] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.750602] schedule+0xd4/0x260
[ 1235.755019] schedule_timeout+0x13f/0x290
[ 1235.760314] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1235.766682] ? trace_contention_end+0xb3/0x1a0
[ 1235.772468] ? hrtimers_cpu_dying+0x260/0x260
[ 1235.778156] ? ww_mutex_lock+0x160/0x160
[ 1235.783345] msleep+0x1b/0x30
[ 1235.787469] napi_set_threaded+0x3a5/0x440
[ 1235.792863] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.798355] netif_set_threaded+0x257/0x400
[ 1235.803848] threaded_store+0x1b8/0x220
[ 1235.808949] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.814246] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.819538] ? sysfs_kf_write+0x65/0x160
[ 1235.824738] ? sysfs_file_ops+0x140/0x140
[ 1235.830033] kernfs_fop_write_iter+0x335/0x510
[ 1235.835815] ? kernfs_ops+0xf0/0xf0
[ 1235.840527] new_sync_write+0x333/0x750
[ 1235.845625] ? __lock_acquire+0x508/0xc10
[ 1235.850920] ? new_sync_read+0x740/0x740
[ 1235.856120] ? lock_acquire.part.0+0xbc/0x260
[ 1235.861805] ? ksys_write+0x116/0x250
[ 1235.866713] vfs_write+0x657/0x9b0
[ 1235.871325] ksys_write+0x116/0x250
[ 1235.876038] ? __ia32_sys_read+0xc0/0xc0
[ 1235.881225] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.886812] ? rcu_is_watching+0x15/0xd0
[ 1235.892010] do_syscall_64+0xf3/0x5e0
[ 1235.896914] ? irq_exit_rcu+0x39/0xf0
[ 1235.901822] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.908287] RIP: 0033:0x7f67cef5122e
[ 1235.913095] RSP: 002b:00007ffcdb6690d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.922390] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f67cef5122e
[ 1235.931196] RDX: 0000000000000001 RSI: 00005563b6399f40 RDI: 0000000000000007
[ 1235.940001] RBP: 00007ffcdb6690e0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.948807] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f67cedc9390
[ 1235.957615] R13: 00005563b6399f40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.966429]
[ 1235.969696] \x0aShowing all locks held in the system:
[ 1235.978234] 1 lock held by khungtaskd/184:
[ 1235.983625] #0: ffffffff99ecdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1235.995094] 2 locks held by systemd-journal/803:
[ 1236.001073] #0: ff1100180e9b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1236.010962] #1: ffffffff9a0c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: pcpu_alloc+0x2c6/0x470
[ 1236.023092] 4 locks held by python3/3544:
[ 1236.028384] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.038270] #1: ff1100016adb7488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.049230] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.060387] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.070479] 2 locks held by kworker/30:0/5367:
[ 1236.076265] #0: ff110001000da748 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xd52/0x1410
[ 1236.087704] #1: ffa00000280efd20 (key_gc_work){+.+.}-{0:0}, at: process_one_work+0xdb4/0x1410
[ 1236.098181] 4 locks held by python3/5603:
[ 1236.103475] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.113363] #1: ff110001c10d2c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.124312] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.135466] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.145560]
[ 1236.148026] =============================================\x0a
[ 1333.722584] kworker/u129:8 (285) used greatest stack depth: 21192 bytes left
[ 1358.299794] INFO: task python3:5603 blocked for more than 491 seconds.
[ 1358.307941] Not tainted 7.0.0-rc6-twpr-ga220ed2c96ca #1
[ 1358.315194] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.324784] task:python3 state:D stack:25336 pid:5603 tgid:5603 ppid:5602 task_flags:0x400100 flags:0x00080002
[ 1358.337989] Call Trace:
[ 1358.341525]
[ 1358.344674] __schedule+0xae3/0x14d0
[ 1358.349487] ? io_schedule_timeout+0x140/0x140
[ 1358.355273] ? find_held_lock+0x2b/0x80
[ 1358.360380] schedule+0xd4/0x260
[ 1358.364815] schedule_preempt_disabled+0x18/0x30
[ 1358.370797] __mutex_lock+0x6b2/0x1ee0
[ 1358.375803] ? rcu_is_watching+0x15/0xd0
[ 1358.381002] ? threaded_store+0x132/0x220
[ 1358.386293] ? ww_mutex_lock+0x160/0x160
[ 1358.391481] ? selinux_inode_free_security+0x330/0x330
[ 1358.398051] ? cap_capable+0x1ed/0x3f0
[ 1358.403056] ? threaded_store+0x132/0x220
[ 1358.408351] threaded_store+0x132/0x220
[ 1358.413454] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.418740] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.424038] ? sysfs_kf_write+0x65/0x160
[ 1358.429237] ? sysfs_file_ops+0x140/0x140
[ 1358.434531] kernfs_fop_write_iter+0x335/0x510
[ 1358.440316] ? kernfs_ops+0xf0/0xf0
[ 1358.445025] new_sync_write+0x333/0x750
[ 1358.450126] ? __lock_acquire+0x508/0xc10
[ 1358.455419] ? new_sync_read+0x740/0x740
[ 1358.460618] ? lock_acquire.part.0+0xbc/0x260
[ 1358.466303] ? ksys_write+0x116/0x250
[ 1358.471213] vfs_write+0x657/0x9b0
[ 1358.475828] ksys_write+0x116/0x250
[ 1358.480531] ? __ia32_sys_read+0xc0/0xc0
[ 1358.485730] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.491311] ? rcu_is_watching+0x15/0xd0
[ 1358.496509] do_syscall_64+0xf3/0x5e0
[ 1358.501413] ? trace_hardirqs_off+0xd/0x30
[ 1358.506807] ? exc_page_fault+0xda/0xf0
[ 1358.511908] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.518376] RIP: 0033:0x7fd8ceee222e
[ 1358.523185] RSP: 002b:00007fffe5baae90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.532483] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd8ceee222e
[ 1358.541292] RDX: 0000000000000001 RSI: 000055a6005b7f40 RDI: 0000000000000007
[ 1358.550100] RBP: 00007fffe5baaea0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.558910] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd8ced5a390
[ 1358.567715] R13: 000055a6005b7f40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.576529]
[ 1358.579824] INFO: task python3:5603 is blocked on a mutex likely owned by task python3:3544.
[ 1358.590096] task:python3 state:D stack:25336 pid:3544 tgid:3544 ppid:3543 task_flags:0x400100 flags:0x00080802
[ 1358.603297] Call Trace:
[ 1358.606824]
[ 1358.609974] __schedule+0xae3/0x14d0
[ 1358.614777] ? io_schedule_timeout+0x140/0x140
[ 1358.620561] ? find_held_lock+0x2b/0x80
[ 1358.625664] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.631841] schedule+0xd4/0x260
[ 1358.636259] schedule_timeout+0x13f/0x290
[ 1358.641555] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1358.647922] ? trace_contention_end+0xb3/0x1a0
[ 1358.653707] ? hrtimers_cpu_dying+0x260/0x260
[ 1358.659389] ? ww_mutex_lock+0x160/0x160
[ 1358.664587] msleep+0x1b/0x30
[ 1358.668712] napi_set_threaded+0x3a5/0x440
[ 1358.674106] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.679598] netif_set_threaded+0x257/0x400
[ 1358.685090] threaded_store+0x1b8/0x220
[ 1358.690191] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.695489] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.700777] ? sysfs_kf_write+0x65/0x160
[ 1358.705973] ? sysfs_file_ops+0x140/0x140
[ 1358.711268] kernfs_fop_write_iter+0x335/0x510
[ 1358.717052] ? kernfs_ops+0xf0/0xf0
[ 1358.721763] new_sync_write+0x333/0x750
[ 1358.726866] ? __lock_acquire+0x508/0xc10
[ 1358.732161] ? new_sync_read+0x740/0x740
[ 1358.737361] ? lock_acquire.part.0+0xbc/0x260
[ 1358.743038] ? ksys_write+0x116/0x250
[ 1358.747946] vfs_write+0x657/0x9b0
[ 1358.752553] ksys_write+0x116/0x250
[ 1358.757266] ? __ia32_sys_read+0xc0/0xc0
[ 1358.762453] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.768040] ? rcu_is_watching+0x15/0xd0
[ 1358.773239] do_syscall_64+0xf3/0x5e0
[ 1358.778144] ? irq_exit_rcu+0x39/0xf0
[ 1358.783052] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.789516] RIP: 0033:0x7f67cef5122e
[ 1358.794326] RSP: 002b:00007ffcdb6690d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.803623] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f67cef5122e
[ 1358.812427] RDX: 0000000000000001 RSI: 00005563b6399f40 RDI: 0000000000000007
[ 1358.821236] RBP: 00007ffcdb6690e0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.830041] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f67cedc9390
[ 1358.838846] R13: 00005563b6399f40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.847662]
[ 1358.850932] \x0aShowing all locks held in the system:
[ 1358.859469] 1 lock held by khungtaskd/184:
[ 1358.864861] #0: ffffffff99ecdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.876331] 2 locks held by systemd-journal/803:
[ 1358.882306] #0: ff1100180e0372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 1358.892196] #1: ffffffff9a0c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: dput.part.0+0x2fd/0x410
[ 1358.904433] 4 locks held by python3/3544:
[ 1358.909718] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.919607] #1: ff1100016adb7488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.930568] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.941714] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.951811] 4 locks held by python3/5603:
[ 1358.957109] #0: ff1100012eaa93f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.966999] #1: ff110001c10d2c88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.977959] #2: ff110001221dd0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.989112] #3: ff11000122241d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.999203]
[ 1359.001668] =============================================\x0a