[ 772.417388] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 772.421804] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 772.573090] kselftest: Running tests in drivers/net
[ 989.714900] INFO: task python3:5453 blocked for more than 122 seconds.
[ 989.723086] Not tainted 7.1.0-rc6-tgcp-gc3e973d955ca #1
[ 989.730587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.740180] task:python3 state:D stack:25672 pid:5453 tgid:5453 ppid:5452 task_flags:0x400100 flags:0x00080000
[ 989.753387] Call Trace:
[ 989.756927]
[ 989.760080] __schedule+0xcc3/0x19c0
[ 989.764897] ? io_schedule_timeout+0x140/0x140
[ 989.770682] ? find_held_lock+0x2b/0x80
[ 989.775790] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.781675] schedule+0xd8/0x260
[ 989.786092] schedule_preempt_disabled+0x18/0x30
[ 989.792071] __mutex_lock+0x6af/0x2010
[ 989.797076] ? threaded_store+0x132/0x220
[ 989.802376] ? ww_mutex_lock+0x160/0x160
[ 989.807578] ? selinux_inode_free_security+0x330/0x330
[ 989.814150] ? cap_capable+0x1d6/0x3e0
[ 989.819162] ? threaded_store+0x132/0x220
[ 989.824457] threaded_store+0x132/0x220
[ 989.829559] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.834854] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.840157] ? sysfs_kf_write+0x65/0x160
[ 989.845358] ? sysfs_file_ops+0x140/0x140
[ 989.850656] kernfs_fop_write_iter+0x335/0x510
[ 989.856442] ? kernfs_ops+0xf0/0xf0
[ 989.861147] new_sync_write+0x333/0x750
[ 989.866251] ? __lock_acquire+0x508/0xc10
[ 989.871548] ? new_sync_read+0x740/0x740
[ 989.876745] ? lock_acquire.part.0+0xbc/0x260
[ 989.882430] ? ksys_write+0x116/0x250
[ 989.887342] vfs_write+0x657/0x9b0
[ 989.891960] ksys_write+0x116/0x250
[ 989.896672] ? __ia32_sys_read+0xc0/0xc0
[ 989.901871] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.907459] ? rcu_is_watching+0x15/0xd0
[ 989.912653] do_syscall_64+0xf3/0x650
[ 989.917561] ? trace_hardirqs_off+0xd/0x30
[ 989.922956] ? exc_page_fault+0xda/0xf0
[ 989.928056] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.934525] RIP: 0033:0x7f371775864e
[ 989.939335] RSP: 002b:00007ffea6a259e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.948627] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f371775864e
[ 989.957435] RDX: 0000000000000001 RSI: 000055956fe00bf0 RDI: 0000000000000007
[ 989.966243] RBP: 00007ffea6a259f0 R08: 0000000000000000 R09: 0000000000000000
[ 989.975052] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f37175d1390
[ 989.983861] R13: 000055956fe00bf0 R14: 0000000000000007 R15: 00007f3717565120
[ 989.992681]
[ 989.995985] INFO: task python3:5453 is blocked on a mutex likely owned by task python3:3484.
[ 990.006258] task:python3 state:D stack:25608 pid:3484 tgid:3484 ppid:3483 task_flags:0x400100 flags:0x00080802
[ 990.019461] Call Trace:
[ 990.023001]
[ 990.026152] __schedule+0xcc3/0x19c0
[ 990.030967] ? io_schedule_timeout+0x140/0x140
[ 990.036749] ? find_held_lock+0x2b/0x80
[ 990.041853] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.048034] schedule+0xd8/0x260
[ 990.052455] schedule_timeout+0x13f/0x290
[ 990.057754] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.064122] ? trace_contention_end+0xb3/0x1a0
[ 990.069907] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.075598] ? ww_mutex_lock+0x160/0x160
[ 990.080800] msleep+0x1b/0x30
[ 990.084928] napi_set_threaded+0x3a5/0x440
[ 990.090324] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.095819] netif_set_threaded+0x257/0x400
[ 990.101315] threaded_store+0x1b8/0x220
[ 990.106417] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.111716] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.117011] ? sysfs_kf_write+0x65/0x160
[ 990.122209] ? sysfs_file_ops+0x140/0x140
[ 990.127507] kernfs_fop_write_iter+0x335/0x510
[ 990.133293] ? kernfs_ops+0xf0/0xf0
[ 990.138004] new_sync_write+0x333/0x750
[ 990.143109] ? __lock_acquire+0x508/0xc10
[ 990.148406] ? new_sync_read+0x740/0x740
[ 990.153607] ? lock_acquire.part.0+0xbc/0x260
[ 990.159294] ? ksys_write+0x116/0x250
[ 990.164205] vfs_write+0x657/0x9b0
[ 990.168822] ksys_write+0x116/0x250
[ 990.173534] ? __ia32_sys_read+0xc0/0xc0
[ 990.178739] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.184331] ? rcu_is_watching+0x15/0xd0
[ 990.189530] do_syscall_64+0xf3/0x650
[ 990.194435] ? trace_hardirqs_off+0xd/0x30
[ 990.199830] ? exc_page_fault+0xda/0xf0
[ 990.204930] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.211396] RIP: 0033:0x7f336ad9564e
[ 990.216207] RSP: 002b:00007ffdcde6f8a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.225507] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f336ad9564e
[ 990.234315] RDX: 0000000000000001 RSI: 000055cfe86d8bf0 RDI: 0000000000000007
[ 990.243115] RBP: 00007ffdcde6f8b0 R08: 0000000000000000 R09: 0000000000000000
[ 990.251924] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f336ac0e390
[ 990.260732] R13: 000055cfe86d8bf0 R14: 0000000000000007 R15: 00007f336aba1120
[ 990.269547]
[ 990.272807] \x0aShowing all locks held in the system:
[ 990.281347] 1 lock held by khungtaskd/185:
[ 990.286744] #0: ffffffffa96fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.298234] 4 locks held by python3/3484:
[ 990.303530] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.313421] #1: ff110001418a9480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.324385] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.335542] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.345632] 4 locks held by python3/5453:
[ 990.350929] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.360821] #1: ff110001a5766280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.371783] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.382940] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.393021]
[ 990.395487] =============================================\x0a
[ 1112.594991] INFO: task python3:5453 blocked for more than 245 seconds.
[ 1112.603142] Not tainted 7.1.0-rc6-tgcp-gc3e973d955ca #1
[ 1112.610394] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.619981] task:python3 state:D stack:25672 pid:5453 tgid:5453 ppid:5452 task_flags:0x400100 flags:0x00080000
[ 1112.633186] Call Trace:
[ 1112.636726]
[ 1112.639875] __schedule+0xcc3/0x19c0
[ 1112.644690] ? io_schedule_timeout+0x140/0x140
[ 1112.650476] ? find_held_lock+0x2b/0x80
[ 1112.655578] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.661464] schedule+0xd8/0x260
[ 1112.665881] schedule_preempt_disabled+0x18/0x30
[ 1112.671858] __mutex_lock+0x6af/0x2010
[ 1112.676861] ? threaded_store+0x132/0x220
[ 1112.682161] ? ww_mutex_lock+0x160/0x160
[ 1112.687351] ? selinux_inode_free_security+0x330/0x330
[ 1112.693923] ? cap_capable+0x1d6/0x3e0
[ 1112.698932] ? threaded_store+0x132/0x220
[ 1112.704226] threaded_store+0x132/0x220
[ 1112.709328] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.714697] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.719992] ? sysfs_kf_write+0x65/0x160
[ 1112.725193] ? sysfs_file_ops+0x140/0x140
[ 1112.730489] kernfs_fop_write_iter+0x335/0x510
[ 1112.736272] ? kernfs_ops+0xf0/0xf0
[ 1112.740985] new_sync_write+0x333/0x750
[ 1112.746085] ? __lock_acquire+0x508/0xc10
[ 1112.751379] ? new_sync_read+0x740/0x740
[ 1112.756581] ? lock_acquire.part.0+0xbc/0x260
[ 1112.762266] ? ksys_write+0x116/0x250
[ 1112.767174] vfs_write+0x657/0x9b0
[ 1112.771789] ksys_write+0x116/0x250
[ 1112.776501] ? __ia32_sys_read+0xc0/0xc0
[ 1112.781700] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.787289] ? rcu_is_watching+0x15/0xd0
[ 1112.792482] do_syscall_64+0xf3/0x650
[ 1112.797389] ? trace_hardirqs_off+0xd/0x30
[ 1112.802774] ? exc_page_fault+0xda/0xf0
[ 1112.807877] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.814342] RIP: 0033:0x7f371775864e
[ 1112.819153] RSP: 002b:00007ffea6a259e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.828450] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f371775864e
[ 1112.837250] RDX: 0000000000000001 RSI: 000055956fe00bf0 RDI: 0000000000000007
[ 1112.846047] RBP: 00007ffea6a259f0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.854852] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f37175d1390
[ 1112.863661] R13: 000055956fe00bf0 R14: 0000000000000007 R15: 00007f3717565120
[ 1112.872473]
[ 1112.875808] INFO: task python3:5453 is blocked on a mutex likely owned by task python3:3484.
[ 1112.886132] task:python3 state:D stack:25608 pid:3484 tgid:3484 ppid:3483 task_flags:0x400100 flags:0x00080802
[ 1112.899337] Call Trace:
[ 1112.902875]
[ 1112.906026] __schedule+0xcc3/0x19c0
[ 1112.910841] ? io_schedule_timeout+0x140/0x140
[ 1112.916626] ? find_held_lock+0x2b/0x80
[ 1112.921719] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.927895] schedule+0xd8/0x260
[ 1112.932304] schedule_timeout+0x13f/0x290
[ 1112.937602] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.943969] ? trace_contention_end+0xb3/0x1a0
[ 1112.949753] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.955444] ? ww_mutex_lock+0x160/0x160
[ 1112.960642] msleep+0x1b/0x30
[ 1112.964766] napi_set_threaded+0x3a5/0x440
[ 1112.970161] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.975654] netif_set_threaded+0x257/0x400
[ 1112.981139] threaded_store+0x1b8/0x220
[ 1112.986244] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.991539] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.996836] ? sysfs_kf_write+0x65/0x160
[ 1113.002033] ? sysfs_file_ops+0x140/0x140
[ 1113.007329] kernfs_fop_write_iter+0x335/0x510
[ 1113.013111] ? kernfs_ops+0xf0/0xf0
[ 1113.017822] new_sync_write+0x333/0x750
[ 1113.022929] ? __lock_acquire+0x508/0xc10
[ 1113.028223] ? new_sync_read+0x740/0x740
[ 1113.033423] ? lock_acquire.part.0+0xbc/0x260
[ 1113.039101] ? ksys_write+0x116/0x250
[ 1113.044002] vfs_write+0x657/0x9b0
[ 1113.048618] ksys_write+0x116/0x250
[ 1113.053329] ? __ia32_sys_read+0xc0/0xc0
[ 1113.058591] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.064168] ? rcu_is_watching+0x15/0xd0
[ 1113.069369] do_syscall_64+0xf3/0x650
[ 1113.074277] ? trace_hardirqs_off+0xd/0x30
[ 1113.079673] ? exc_page_fault+0xda/0xf0
[ 1113.084774] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.091234] RIP: 0033:0x7f336ad9564e
[ 1113.096039] RSP: 002b:00007ffdcde6f8a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.105336] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f336ad9564e
[ 1113.114136] RDX: 0000000000000001 RSI: 000055cfe86d8bf0 RDI: 0000000000000007
[ 1113.122945] RBP: 00007ffdcde6f8b0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.131753] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f336ac0e390
[ 1113.140559] R13: 000055cfe86d8bf0 R14: 0000000000000007 R15: 00007f336aba1120
[ 1113.149371]
[ 1113.152635] \x0aShowing all locks held in the system:
[ 1113.161180] 1 lock held by khungtaskd/185:
[ 1113.166577] #0: ffffffffa96fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.178068] 4 locks held by python3/3484:
[ 1113.183365] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.193247] #1: ff110001418a9480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.204199] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.215355] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.225458] 4 locks held by python3/5453:
[ 1113.230756] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.240646] #1: ff110001a5766280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.251597] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.262756] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.272849] 5 locks held by systemd/6526:
[ 1113.278145] #0: ff11000131c4c3e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.288035] #1: ff1100014d581a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.299004] #2: ffffffffa9769c20 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x640
[ 1113.310666] #3: ffffffffa95687b0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 1113.322220] #4: ffffffffa9769990 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 1113.334552]
[ 1113.337187] =============================================\x0a
[ 1235.477112] INFO: task python3:5453 blocked for more than 368 seconds.
[ 1235.485263] Not tainted 7.1.0-rc6-tgcp-gc3e973d955ca #1
[ 1235.492514] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.502104] task:python3 state:D stack:25672 pid:5453 tgid:5453 ppid:5452 task_flags:0x400100 flags:0x00080002
[ 1235.515310] Call Trace:
[ 1235.518851]
[ 1235.522006] __schedule+0xcc3/0x19c0
[ 1235.526827] ? io_schedule_timeout+0x140/0x140
[ 1235.532613] ? find_held_lock+0x2b/0x80
[ 1235.537717] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.543602] schedule+0xd8/0x260
[ 1235.548019] schedule_preempt_disabled+0x18/0x30
[ 1235.553998] __mutex_lock+0x6af/0x2010
[ 1235.559005] ? threaded_store+0x132/0x220
[ 1235.564309] ? ww_mutex_lock+0x160/0x160
[ 1235.569509] ? selinux_inode_free_security+0x330/0x330
[ 1235.576080] ? cap_capable+0x1d6/0x3e0
[ 1235.581088] ? threaded_store+0x132/0x220
[ 1235.586387] threaded_store+0x132/0x220
[ 1235.591491] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.596791] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.602091] ? sysfs_kf_write+0x65/0x160
[ 1235.607294] ? sysfs_file_ops+0x140/0x140
[ 1235.612592] kernfs_fop_write_iter+0x335/0x510
[ 1235.618375] ? kernfs_ops+0xf0/0xf0
[ 1235.623086] new_sync_write+0x333/0x750
[ 1235.628192] ? __lock_acquire+0x508/0xc10
[ 1235.633489] ? new_sync_read+0x740/0x740
[ 1235.638692] ? lock_acquire.part.0+0xbc/0x260
[ 1235.644377] ? ksys_write+0x116/0x250
[ 1235.649287] vfs_write+0x657/0x9b0
[ 1235.653901] ksys_write+0x116/0x250
[ 1235.658615] ? __ia32_sys_read+0xc0/0xc0
[ 1235.663812] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.669403] ? rcu_is_watching+0x15/0xd0
[ 1235.674604] do_syscall_64+0xf3/0x650
[ 1235.679513] ? trace_hardirqs_off+0xd/0x30
[ 1235.684910] ? exc_page_fault+0xda/0xf0
[ 1235.690014] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.696484] RIP: 0033:0x7f371775864e
[ 1235.701296] RSP: 002b:00007ffea6a259e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.710596] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f371775864e
[ 1235.719404] RDX: 0000000000000001 RSI: 000055956fe00bf0 RDI: 0000000000000007
[ 1235.728201] RBP: 00007ffea6a259f0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.737011] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f37175d1390
[ 1235.745820] R13: 000055956fe00bf0 R14: 0000000000000007 R15: 00007f3717565120
[ 1235.754634]
[ 1235.757901] INFO: task python3:5453 is blocked on a mutex likely owned by task python3:3484.
[ 1235.768176] task:python3 state:D stack:25608 pid:3484 tgid:3484 ppid:3483 task_flags:0x400100 flags:0x00080802
[ 1235.781377] Call Trace:
[ 1235.784916]
[ 1235.788069] __schedule+0xcc3/0x19c0
[ 1235.792882] ? io_schedule_timeout+0x140/0x140
[ 1235.798668] ? find_held_lock+0x2b/0x80
[ 1235.803773] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.809957] schedule+0xd8/0x260
[ 1235.814373] schedule_timeout+0x13f/0x290
[ 1235.819671] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.826032] ? trace_contention_end+0xb3/0x1a0
[ 1235.831819] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.837511] ? ww_mutex_lock+0x160/0x160
[ 1235.842712] msleep+0x1b/0x30
[ 1235.846838] napi_set_threaded+0x3a5/0x440
[ 1235.852232] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.857729] netif_set_threaded+0x257/0x400
[ 1235.863222] threaded_store+0x1b8/0x220
[ 1235.868324] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.873622] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.878918] ? sysfs_kf_write+0x65/0x160
[ 1235.884119] ? sysfs_file_ops+0x140/0x140
[ 1235.889407] kernfs_fop_write_iter+0x335/0x510
[ 1235.895189] ? kernfs_ops+0xf0/0xf0
[ 1235.899901] new_sync_write+0x333/0x750
[ 1235.905005] ? __lock_acquire+0x508/0xc10
[ 1235.910301] ? new_sync_read+0x740/0x740
[ 1235.915502] ? lock_acquire.part.0+0xbc/0x260
[ 1235.921189] ? ksys_write+0x116/0x250
[ 1235.926099] vfs_write+0x657/0x9b0
[ 1235.930716] ksys_write+0x116/0x250
[ 1235.935430] ? __ia32_sys_read+0xc0/0xc0
[ 1235.940631] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.946219] ? rcu_is_watching+0x15/0xd0
[ 1235.951421] do_syscall_64+0xf3/0x650
[ 1235.956330] ? trace_hardirqs_off+0xd/0x30
[ 1235.961725] ? exc_page_fault+0xda/0xf0
[ 1235.966816] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.973284] RIP: 0033:0x7f336ad9564e
[ 1235.978094] RSP: 002b:00007ffdcde6f8a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.987382] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f336ad9564e
[ 1235.996182] RDX: 0000000000000001 RSI: 000055cfe86d8bf0 RDI: 0000000000000007
[ 1236.004989] RBP: 00007ffdcde6f8b0 R08: 0000000000000000 R09: 0000000000000000
[ 1236.013799] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f336ac0e390
[ 1236.022608] R13: 000055cfe86d8bf0 R14: 0000000000000007 R15: 00007f336aba1120
[ 1236.031423]
[ 1236.034692] \x0aShowing all locks held in the system:
[ 1236.043228] 1 lock held by khungtaskd/185:
[ 1236.048621] #0: ffffffffa96fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.060095] 2 locks held by systemd-journal/809:
[ 1236.066075] #0: ff1100180e3b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.075966] #1: ffffffffa9902ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.088793] 4 locks held by python3/3484:
[ 1236.094091] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.103980] #1: ff110001418a9480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.114943] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.126106] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.136199] 4 locks held by python3/5453:
[ 1236.141495] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.151384] #1: ff110001a5766280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.162339] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.173497] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.183591]
[ 1236.186058] =============================================\x0a
[ 1358.358253] INFO: task python3:5453 blocked for more than 491 seconds.
[ 1358.366404] Not tainted 7.1.0-rc6-tgcp-gc3e973d955ca #1
[ 1358.373654] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.383242] task:python3 state:D stack:25672 pid:5453 tgid:5453 ppid:5452 task_flags:0x400100 flags:0x00080002
[ 1358.396446] Call Trace:
[ 1358.399985]
[ 1358.403137] __schedule+0xcc3/0x19c0
[ 1358.407952] ? io_schedule_timeout+0x140/0x140
[ 1358.413728] ? find_held_lock+0x2b/0x80
[ 1358.418834] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.424719] schedule+0xd8/0x260
[ 1358.429135] schedule_preempt_disabled+0x18/0x30
[ 1358.435113] __mutex_lock+0x6af/0x2010
[ 1358.440115] ? threaded_store+0x132/0x220
[ 1358.445413] ? ww_mutex_lock+0x160/0x160
[ 1358.450613] ? selinux_inode_free_security+0x330/0x330
[ 1358.457183] ? cap_capable+0x1d6/0x3e0
[ 1358.462188] ? threaded_store+0x132/0x220
[ 1358.467486] threaded_store+0x132/0x220
[ 1358.472578] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.477879] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.483176] ? sysfs_kf_write+0x65/0x160
[ 1358.488374] ? sysfs_file_ops+0x140/0x140
[ 1358.493667] kernfs_fop_write_iter+0x335/0x510
[ 1358.499449] ? kernfs_ops+0xf0/0xf0
[ 1358.504161] new_sync_write+0x333/0x750
[ 1358.509266] ? __lock_acquire+0x508/0xc10
[ 1358.514561] ? new_sync_read+0x740/0x740
[ 1358.519764] ? lock_acquire.part.0+0xbc/0x260
[ 1358.525451] ? ksys_write+0x116/0x250
[ 1358.530351] vfs_write+0x657/0x9b0
[ 1358.534965] ksys_write+0x116/0x250
[ 1358.539678] ? __ia32_sys_read+0xc0/0xc0
[ 1358.544876] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.550466] ? rcu_is_watching+0x15/0xd0
[ 1358.555667] do_syscall_64+0xf3/0x650
[ 1358.560572] ? trace_hardirqs_off+0xd/0x30
[ 1358.565964] ? exc_page_fault+0xda/0xf0
[ 1358.571065] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.577523] RIP: 0033:0x7f371775864e
[ 1358.582321] RSP: 002b:00007ffea6a259e0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.591615] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f371775864e
[ 1358.600423] RDX: 0000000000000001 RSI: 000055956fe00bf0 RDI: 0000000000000007
[ 1358.609230] RBP: 00007ffea6a259f0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.618037] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f37175d1390
[ 1358.626842] R13: 000055956fe00bf0 R14: 0000000000000007 R15: 00007f3717565120
[ 1358.635653]
[ 1358.638931] INFO: task python3:5453 is blocked on a mutex likely owned by task python3:3484.
[ 1358.649202] task:python3 state:D stack:25608 pid:3484 tgid:3484 ppid:3483 task_flags:0x400100 flags:0x00080802
[ 1358.662403] Call Trace:
[ 1358.665942]
[ 1358.669089] __schedule+0xcc3/0x19c0
[ 1358.673906] ? io_schedule_timeout+0x140/0x140
[ 1358.679687] ? find_held_lock+0x2b/0x80
[ 1358.684787] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.690966] schedule+0xd8/0x260
[ 1358.695385] schedule_timeout+0x13f/0x290
[ 1358.700683] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.707053] ? trace_contention_end+0xb3/0x1a0
[ 1358.712838] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.718527] ? ww_mutex_lock+0x160/0x160
[ 1358.723724] msleep+0x1b/0x30
[ 1358.727848] napi_set_threaded+0x3a5/0x440
[ 1358.733243] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.738737] netif_set_threaded+0x257/0x400
[ 1358.744229] threaded_store+0x1b8/0x220
[ 1358.749322] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.754618] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.759914] ? sysfs_kf_write+0x65/0x160
[ 1358.765111] ? sysfs_file_ops+0x140/0x140
[ 1358.770405] kernfs_fop_write_iter+0x335/0x510
[ 1358.776188] ? kernfs_ops+0xf0/0xf0
[ 1358.780901] new_sync_write+0x333/0x750
[ 1358.786001] ? __lock_acquire+0x508/0xc10
[ 1358.791297] ? new_sync_read+0x740/0x740
[ 1358.796496] ? lock_acquire.part.0+0xbc/0x260
[ 1358.802180] ? ksys_write+0x116/0x250
[ 1358.807097] vfs_write+0x657/0x9b0
[ 1358.811711] ksys_write+0x116/0x250
[ 1358.816421] ? __ia32_sys_read+0xc0/0xc0
[ 1358.821621] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.827209] ? rcu_is_watching+0x15/0xd0
[ 1358.832409] do_syscall_64+0xf3/0x650
[ 1358.837315] ? trace_hardirqs_off+0xd/0x30
[ 1358.842707] ? exc_page_fault+0xda/0xf0
[ 1358.847806] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.854272] RIP: 0033:0x7f336ad9564e
[ 1358.859079] RSP: 002b:00007ffdcde6f8a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.868374] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f336ad9564e
[ 1358.877183] RDX: 0000000000000001 RSI: 000055cfe86d8bf0 RDI: 0000000000000007
[ 1358.885989] RBP: 00007ffdcde6f8b0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.894795] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f336ac0e390
[ 1358.903602] R13: 000055cfe86d8bf0 R14: 0000000000000007 R15: 00007f336aba1120
[ 1358.912406]
[ 1358.915675] \x0aShowing all locks held in the system:
[ 1358.924213] 1 lock held by khungtaskd/185:
[ 1358.929606] #0: ffffffffa96fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.941077] 2 locks held by systemd-journal/809:
[ 1358.947054] #0: ff1100180e7b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1358.956940] #1: ffffffffa9902ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: link_path_walk+0x9a7/0x1830
[ 1358.969568] 4 locks held by python3/3484:
[ 1358.974864] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.984753] #1: ff110001418a9480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.995706] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.006864] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.016954] 4 locks held by python3/5453:
[ 1359.022252] #0: ff11000131b4c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.032142] #1: ff110001a5766280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.043109] #2: ff1100012438ab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.054265] #3: ff11000124409d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.064355]
[ 1359.066819] =============================================\x0a