[ 780.068701] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 780.071156] nipa-hw-worker: [4/40] Retrying drivers/net:napi_threaded.py
[ 780.220993] kselftest: Running tests in drivers/net
[ 989.673343] INFO: task python3:5590 blocked for more than 122 seconds.
[ 989.681571] Not tainted 7.0.0-rc6-fuct-g3f97ca4f8289 #1
[ 989.688823] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.698411] task:python3 state:D stack:25000 pid:5590 tgid:5590 ppid:5589 task_flags:0x400100 flags:0x00080000
[ 989.711615] Call Trace:
[ 989.715155]
[ 989.718306] __schedule+0xae3/0x14d0
[ 989.723120] ? io_schedule_timeout+0x140/0x140
[ 989.728903] ? find_held_lock+0x2b/0x80
[ 989.734000] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.739876] schedule+0xd4/0x260
[ 989.744293] schedule_preempt_disabled+0x18/0x30
[ 989.750271] __mutex_lock+0x6b2/0x1ee0
[ 989.755276] ? rcu_is_watching+0x15/0xd0
[ 989.760474] ? threaded_store+0x132/0x220
[ 989.765774] ? ww_mutex_lock+0x160/0x160
[ 989.770964] ? selinux_inode_free_security+0x330/0x330
[ 989.777535] ? cap_capable+0x1ed/0x3f0
[ 989.782544] ? threaded_store+0x132/0x220
[ 989.787839] threaded_store+0x132/0x220
[ 989.792944] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.798241] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.803536] ? sysfs_kf_write+0x65/0x160
[ 989.808732] ? sysfs_file_ops+0x140/0x140
[ 989.814025] kernfs_fop_write_iter+0x335/0x510
[ 989.819809] ? kernfs_ops+0xf0/0xf0
[ 989.824518] new_sync_write+0x333/0x750
[ 989.829620] ? __lock_acquire+0x508/0xc10
[ 989.834916] ? new_sync_read+0x740/0x740
[ 989.840118] ? lock_acquire.part.0+0xbc/0x260
[ 989.845802] ? ksys_write+0x116/0x250
[ 989.850712] vfs_write+0x657/0x9b0
[ 989.855324] ksys_write+0x116/0x250
[ 989.860035] ? __ia32_sys_read+0xc0/0xc0
[ 989.865235] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.870823] ? rcu_is_watching+0x15/0xd0
[ 989.876025] do_syscall_64+0xf3/0x5e0
[ 989.880931] ? trace_hardirqs_off+0xd/0x30
[ 989.886316] ? exc_page_fault+0xda/0xf0
[ 989.891417] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.897885] RIP: 0033:0x7fdd5e00b22e
[ 989.902694] RSP: 002b:00007fff8d5f2400 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.911991] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdd5e00b22e
[ 989.920799] RDX: 0000000000000001 RSI: 000055a1167aef40 RDI: 0000000000000007
[ 989.929608] RBP: 00007fff8d5f2410 R08: 0000000000000000 R09: 0000000000000000
[ 989.938416] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdd5de83390
[ 989.947229] R13: 000055a1167aef40 R14: 0000000000000007 R15: 0000000000000003
[ 989.956042]
[ 989.959343] INFO: task python3:5590 is blocked on a mutex likely owned by task python3:3511.
[ 989.969604] task:python3 state:D stack:25336 pid:3511 tgid:3511 ppid:3510 task_flags:0x400100 flags:0x00080802
[ 989.982807] Call Trace:
[ 989.986342]
[ 989.989494] __schedule+0xae3/0x14d0
[ 989.994294] ? io_schedule_timeout+0x140/0x140
[ 990.000076] ? find_held_lock+0x2b/0x80
[ 990.005178] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.011345] schedule+0xd4/0x260
[ 990.015763] schedule_timeout+0x13f/0x290
[ 990.021057] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 990.027424] ? trace_contention_end+0xb3/0x1a0
[ 990.033208] ? hrtimers_cpu_dying+0x260/0x260
[ 990.038898] ? ww_mutex_lock+0x160/0x160
[ 990.044098] msleep+0x1b/0x30
[ 990.048215] napi_set_threaded+0x3a5/0x440
[ 990.053608] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.059102] netif_set_threaded+0x257/0x400
[ 990.064592] threaded_store+0x1b8/0x220
[ 990.069691] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.074986] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.080282] ? sysfs_kf_write+0x65/0x160
[ 990.085480] ? sysfs_file_ops+0x140/0x140
[ 990.090774] kernfs_fop_write_iter+0x335/0x510
[ 990.096560] ? kernfs_ops+0xf0/0xf0
[ 990.101269] new_sync_write+0x333/0x750
[ 990.106371] ? __lock_acquire+0x508/0xc10
[ 990.111665] ? new_sync_read+0x740/0x740
[ 990.116863] ? lock_acquire.part.0+0xbc/0x260
[ 990.122549] ? ksys_write+0x116/0x250
[ 990.127459] vfs_write+0x657/0x9b0
[ 990.132074] ksys_write+0x116/0x250
[ 990.136777] ? __ia32_sys_read+0xc0/0xc0
[ 990.141975] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.147566] ? rcu_is_watching+0x15/0xd0
[ 990.152764] do_syscall_64+0xf3/0x5e0
[ 990.157667] ? trace_hardirqs_off+0xd/0x30
[ 990.163061] ? exc_page_fault+0xda/0xf0
[ 990.168161] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.174629] RIP: 0033:0x7fab236c622e
[ 990.179427] RSP: 002b:00007fff237657b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.188724] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab236c622e
[ 990.197529] RDX: 0000000000000001 RSI: 00005588e6742f40 RDI: 0000000000000007
[ 990.206335] RBP: 00007fff237657c0 R08: 0000000000000000 R09: 0000000000000000
[ 990.215139] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab2353e390
[ 990.223944] R13: 00005588e6742f40 R14: 0000000000000007 R15: 0000000000000003
[ 990.232758]
[ 990.236017] \x0aShowing all locks held in the system:
[ 990.244555] 1 lock held by khungtaskd/185:
[ 990.249946] #0: ffffffff974cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.261418] 2 locks held by systemd-journal/800:
[ 990.267397] #0: ff1100180e8b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x194/0x14d0
[ 990.277285] #1: ffffffff976c8fe0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.290102] 4 locks held by python3/3511:
[ 990.295396] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.305285] #1: ff110001620f8288 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.316244] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.327399] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.337496] 4 locks held by python3/5590:
[ 990.342790] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.352678] #1: ff110001c1a5f488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.363628] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.374782] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.384868]
[ 990.387332] =============================================\x0a
[ 1112.554506] INFO: task python3:5590 blocked for more than 245 seconds.
[ 1112.562656] Not tainted 7.0.0-rc6-fuct-g3f97ca4f8289 #1
[ 1112.569908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.579495] task:python3 state:D stack:25000 pid:5590 tgid:5590 ppid:5589 task_flags:0x400100 flags:0x00080000
[ 1112.592704] Call Trace:
[ 1112.596243]
[ 1112.599392] __schedule+0xae3/0x14d0
[ 1112.604211] ? io_schedule_timeout+0x140/0x140
[ 1112.609997] ? find_held_lock+0x2b/0x80
[ 1112.615101] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.620989] schedule+0xd4/0x260
[ 1112.625411] schedule_preempt_disabled+0x18/0x30
[ 1112.631390] __mutex_lock+0x6b2/0x1ee0
[ 1112.636391] ? rcu_is_watching+0x15/0xd0
[ 1112.641589] ? threaded_store+0x132/0x220
[ 1112.646891] ? ww_mutex_lock+0x160/0x160
[ 1112.652093] ? selinux_inode_free_security+0x330/0x330
[ 1112.658666] ? cap_capable+0x1ed/0x3f0
[ 1112.663674] ? threaded_store+0x132/0x220
[ 1112.668975] threaded_store+0x132/0x220
[ 1112.674081] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.679378] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.684675] ? sysfs_kf_write+0x65/0x160
[ 1112.689875] ? sysfs_file_ops+0x140/0x140
[ 1112.695162] kernfs_fop_write_iter+0x335/0x510
[ 1112.700950] ? kernfs_ops+0xf0/0xf0
[ 1112.705659] new_sync_write+0x333/0x750
[ 1112.710764] ? __lock_acquire+0x508/0xc10
[ 1112.716059] ? new_sync_read+0x740/0x740
[ 1112.721262] ? lock_acquire.part.0+0xbc/0x260
[ 1112.726951] ? ksys_write+0x116/0x250
[ 1112.731863] vfs_write+0x657/0x9b0
[ 1112.736476] ksys_write+0x116/0x250
[ 1112.741187] ? __ia32_sys_read+0xc0/0xc0
[ 1112.746386] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.751971] ? rcu_is_watching+0x15/0xd0
[ 1112.757172] do_syscall_64+0xf3/0x5e0
[ 1112.762081] ? trace_hardirqs_off+0xd/0x30
[ 1112.767477] ? exc_page_fault+0xda/0xf0
[ 1112.772579] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.779048] RIP: 0033:0x7fdd5e00b22e
[ 1112.783859] RSP: 002b:00007fff8d5f2400 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.793158] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdd5e00b22e
[ 1112.801966] RDX: 0000000000000001 RSI: 000055a1167aef40 RDI: 0000000000000007
[ 1112.810776] RBP: 00007fff8d5f2410 R08: 0000000000000000 R09: 0000000000000000
[ 1112.819588] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdd5de83390
[ 1112.828398] R13: 000055a1167aef40 R14: 0000000000000007 R15: 0000000000000003
[ 1112.837211]
[ 1112.840483] INFO: task python3:5590 is blocked on a mutex likely owned by task python3:3511.
[ 1112.850758] task:python3 state:D stack:25336 pid:3511 tgid:3511 ppid:3510 task_flags:0x400100 flags:0x00080802
[ 1112.863959] Call Trace:
[ 1112.867497]
[ 1112.870649] __schedule+0xae3/0x14d0
[ 1112.875451] ? io_schedule_timeout+0x140/0x140
[ 1112.881238] ? find_held_lock+0x2b/0x80
[ 1112.886348] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.892524] schedule+0xd4/0x260
[ 1112.896943] schedule_timeout+0x13f/0x290
[ 1112.902239] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1112.908609] ? trace_contention_end+0xb3/0x1a0
[ 1112.914396] ? hrtimers_cpu_dying+0x260/0x260
[ 1112.920089] ? ww_mutex_lock+0x160/0x160
[ 1112.925292] msleep+0x1b/0x30
[ 1112.929418] napi_set_threaded+0x3a5/0x440
[ 1112.934812] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.940308] netif_set_threaded+0x257/0x400
[ 1112.945803] threaded_store+0x1b8/0x220
[ 1112.950908] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.956204] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.961499] ? sysfs_kf_write+0x65/0x160
[ 1112.966698] ? sysfs_file_ops+0x140/0x140
[ 1112.971996] kernfs_fop_write_iter+0x335/0x510
[ 1112.977771] ? kernfs_ops+0xf0/0xf0
[ 1112.982485] new_sync_write+0x333/0x750
[ 1112.987588] ? __lock_acquire+0x508/0xc10
[ 1112.992884] ? new_sync_read+0x740/0x740
[ 1112.998082] ? lock_acquire.part.0+0xbc/0x260
[ 1113.003769] ? ksys_write+0x116/0x250
[ 1113.008678] vfs_write+0x657/0x9b0
[ 1113.013293] ksys_write+0x116/0x250
[ 1113.018006] ? __ia32_sys_read+0xc0/0xc0
[ 1113.023206] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.028874] ? rcu_is_watching+0x15/0xd0
[ 1113.034065] do_syscall_64+0xf3/0x5e0
[ 1113.038973] ? trace_hardirqs_off+0xd/0x30
[ 1113.044366] ? exc_page_fault+0xda/0xf0
[ 1113.049469] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.055935] RIP: 0033:0x7fab236c622e
[ 1113.060745] RSP: 002b:00007fff237657b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.070102] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab236c622e
[ 1113.078912] RDX: 0000000000000001 RSI: 00005588e6742f40 RDI: 0000000000000007
[ 1113.087723] RBP: 00007fff237657c0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.096532] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab2353e390
[ 1113.105344] R13: 00005588e6742f40 R14: 0000000000000007 R15: 0000000000000003
[ 1113.114160]
[ 1113.117426] \x0aShowing all locks held in the system:
[ 1113.125956] 1 lock held by khungtaskd/185:
[ 1113.131353] #0: ffffffff974cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.142900] 4 locks held by python3/3511:
[ 1113.148196] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.158087] #1: ff110001620f8288 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.169050] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.180208] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.190304] 4 locks held by python3/5590:
[ 1113.195599] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.205491] #1: ff110001c1a5f488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.216454] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.227611] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.237701] 2 locks held by systemd/6646:
[ 1113.243000]
[ 1113.245468] =============================================\x0a
[ 1235.436683] INFO: task python3:5590 blocked for more than 368 seconds.
[ 1235.444831] Not tainted 7.0.0-rc6-fuct-g3f97ca4f8289 #1
[ 1235.452074] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.461668] task:python3 state:D stack:25000 pid:5590 tgid:5590 ppid:5589 task_flags:0x400100 flags:0x00080002
[ 1235.474876] Call Trace:
[ 1235.478417]
[ 1235.481567] __schedule+0xae3/0x14d0
[ 1235.486387] ? io_schedule_timeout+0x140/0x140
[ 1235.492172] ? find_held_lock+0x2b/0x80
[ 1235.497280] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.503164] schedule+0xd4/0x260
[ 1235.507586] schedule_preempt_disabled+0x18/0x30
[ 1235.513566] __mutex_lock+0x6b2/0x1ee0
[ 1235.518575] ? rcu_is_watching+0x15/0xd0
[ 1235.523773] ? threaded_store+0x132/0x220
[ 1235.529075] ? ww_mutex_lock+0x160/0x160
[ 1235.534272] ? selinux_inode_free_security+0x330/0x330
[ 1235.540842] ? cap_capable+0x1ed/0x3f0
[ 1235.545851] ? threaded_store+0x132/0x220
[ 1235.551149] threaded_store+0x132/0x220
[ 1235.556253] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.561553] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.566855] ? sysfs_kf_write+0x65/0x160
[ 1235.572055] ? sysfs_file_ops+0x140/0x140
[ 1235.577342] kernfs_fop_write_iter+0x335/0x510
[ 1235.583128] ? kernfs_ops+0xf0/0xf0
[ 1235.587833] new_sync_write+0x333/0x750
[ 1235.592936] ? __lock_acquire+0x508/0xc10
[ 1235.598235] ? new_sync_read+0x740/0x740
[ 1235.603434] ? lock_acquire.part.0+0xbc/0x260
[ 1235.609119] ? ksys_write+0x116/0x250
[ 1235.614031] vfs_write+0x657/0x9b0
[ 1235.618648] ksys_write+0x116/0x250
[ 1235.623362] ? __ia32_sys_read+0xc0/0xc0
[ 1235.628562] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.634154] ? rcu_is_watching+0x15/0xd0
[ 1235.639354] do_syscall_64+0xf3/0x5e0
[ 1235.644263] ? trace_hardirqs_off+0xd/0x30
[ 1235.649657] ? exc_page_fault+0xda/0xf0
[ 1235.654759] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.661228] RIP: 0033:0x7fdd5e00b22e
[ 1235.666038] RSP: 002b:00007fff8d5f2400 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.675335] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdd5e00b22e
[ 1235.684145] RDX: 0000000000000001 RSI: 000055a1167aef40 RDI: 0000000000000007
[ 1235.692953] RBP: 00007fff8d5f2410 R08: 0000000000000000 R09: 0000000000000000
[ 1235.701762] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdd5de83390
[ 1235.710571] R13: 000055a1167aef40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.719386]
[ 1235.722654] INFO: task python3:5590 is blocked on a mutex likely owned by task python3:3511.
[ 1235.732925] task:python3 state:D stack:25336 pid:3511 tgid:3511 ppid:3510 task_flags:0x400100 flags:0x00080802
[ 1235.746127] Call Trace:
[ 1235.749667]
[ 1235.752817] __schedule+0xae3/0x14d0
[ 1235.757627] ? io_schedule_timeout+0x140/0x140
[ 1235.763411] ? find_held_lock+0x2b/0x80
[ 1235.768519] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.774696] schedule+0xd4/0x260
[ 1235.779105] schedule_timeout+0x13f/0x290
[ 1235.784404] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1235.790772] ? trace_contention_end+0xb3/0x1a0
[ 1235.796558] ? hrtimers_cpu_dying+0x260/0x260
[ 1235.802250] ? ww_mutex_lock+0x160/0x160
[ 1235.807451] msleep+0x1b/0x30
[ 1235.811579] napi_set_threaded+0x3a5/0x440
[ 1235.816976] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.822472] netif_set_threaded+0x257/0x400
[ 1235.827966] threaded_store+0x1b8/0x220
[ 1235.833070] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.838366] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.843663] ? sysfs_kf_write+0x65/0x160
[ 1235.848862] ? sysfs_file_ops+0x140/0x140
[ 1235.854158] kernfs_fop_write_iter+0x335/0x510
[ 1235.859945] ? kernfs_ops+0xf0/0xf0
[ 1235.864659] new_sync_write+0x333/0x750
[ 1235.869762] ? __lock_acquire+0x508/0xc10
[ 1235.875057] ? new_sync_read+0x740/0x740
[ 1235.880256] ? lock_acquire.part.0+0xbc/0x260
[ 1235.885941] ? ksys_write+0x116/0x250
[ 1235.890850] vfs_write+0x657/0x9b0
[ 1235.895466] ksys_write+0x116/0x250
[ 1235.900177] ? __ia32_sys_read+0xc0/0xc0
[ 1235.905376] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.910967] ? rcu_is_watching+0x15/0xd0
[ 1235.916169] do_syscall_64+0xf3/0x5e0
[ 1235.921076] ? trace_hardirqs_off+0xd/0x30
[ 1235.926460] ? exc_page_fault+0xda/0xf0
[ 1235.931562] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.938018] RIP: 0033:0x7fab236c622e
[ 1235.942825] RSP: 002b:00007fff237657b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.952123] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab236c622e
[ 1235.960932] RDX: 0000000000000001 RSI: 00005588e6742f40 RDI: 0000000000000007
[ 1235.969744] RBP: 00007fff237657c0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.978552] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab2353e390
[ 1235.987362] R13: 00005588e6742f40 R14: 0000000000000007 R15: 0000000000000003
[ 1235.996177]
[ 1235.999444] \x0aShowing all locks held in the system:
[ 1236.007981] 1 lock held by khungtaskd/185:
[ 1236.013376] #0: ffffffff974cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.024862] 4 locks held by python3/3511:
[ 1236.030158] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.040048] #1: ff110001620f8288 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.051009] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.062165] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.072261] 4 locks held by python3/5590:
[ 1236.077555] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.087444] #1: ff110001c1a5f488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.098396] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.109555] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.119648]
[ 1236.122116] =============================================\x0a
[ 1358.317863] INFO: task python3:5590 blocked for more than 491 seconds.
[ 1358.326011] Not tainted 7.0.0-rc6-fuct-g3f97ca4f8289 #1
[ 1358.333265] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.342853] task:python3 state:D stack:25000 pid:5590 tgid:5590 ppid:5589 task_flags:0x400100 flags:0x00080002
[ 1358.356060] Call Trace:
[ 1358.359672]
[ 1358.362819] __schedule+0xae3/0x14d0
[ 1358.367635] ? io_schedule_timeout+0x140/0x140
[ 1358.373418] ? find_held_lock+0x2b/0x80
[ 1358.378520] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.384414] schedule+0xd4/0x260
[ 1358.388836] schedule_preempt_disabled+0x18/0x30
[ 1358.394814] __mutex_lock+0x6b2/0x1ee0
[ 1358.399818] ? rcu_is_watching+0x15/0xd0
[ 1358.405018] ? threaded_store+0x132/0x220
[ 1358.410317] ? ww_mutex_lock+0x160/0x160
[ 1358.415514] ? selinux_inode_free_security+0x330/0x330
[ 1358.422083] ? cap_capable+0x1ed/0x3f0
[ 1358.427090] ? threaded_store+0x132/0x220
[ 1358.432384] threaded_store+0x132/0x220
[ 1358.437485] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.442779] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.448079] ? sysfs_kf_write+0x65/0x160
[ 1358.453268] ? sysfs_file_ops+0x140/0x140
[ 1358.458560] kernfs_fop_write_iter+0x335/0x510
[ 1358.464343] ? kernfs_ops+0xf0/0xf0
[ 1358.469056] new_sync_write+0x333/0x750
[ 1358.474160] ? __lock_acquire+0x508/0xc10
[ 1358.479450] ? new_sync_read+0x740/0x740
[ 1358.484652] ? lock_acquire.part.0+0xbc/0x260
[ 1358.490338] ? ksys_write+0x116/0x250
[ 1358.495250] vfs_write+0x657/0x9b0
[ 1358.499863] ksys_write+0x116/0x250
[ 1358.504576] ? __ia32_sys_read+0xc0/0xc0
[ 1358.509787] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.515385] ? rcu_is_watching+0x15/0xd0
[ 1358.520586] do_syscall_64+0xf3/0x5e0
[ 1358.525491] ? trace_hardirqs_off+0xd/0x30
[ 1358.530885] ? exc_page_fault+0xda/0xf0
[ 1358.535983] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.542450] RIP: 0033:0x7fdd5e00b22e
[ 1358.547260] RSP: 002b:00007fff8d5f2400 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.556558] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdd5e00b22e
[ 1358.565364] RDX: 0000000000000001 RSI: 000055a1167aef40 RDI: 0000000000000007
[ 1358.574170] RBP: 00007fff8d5f2410 R08: 0000000000000000 R09: 0000000000000000
[ 1358.582979] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdd5de83390
[ 1358.591785] R13: 000055a1167aef40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.600598]
[ 1358.603926] INFO: task python3:5590 is blocked on a mutex likely owned by task python3:3511.
[ 1358.614196] task:python3 state:D stack:25336 pid:3511 tgid:3511 ppid:3510 task_flags:0x400100 flags:0x00080802
[ 1358.627396] Call Trace:
[ 1358.630925]
[ 1358.634076] __schedule+0xae3/0x14d0
[ 1358.638885] ? io_schedule_timeout+0x140/0x140
[ 1358.644665] ? find_held_lock+0x2b/0x80
[ 1358.649767] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.655933] schedule+0xd4/0x260
[ 1358.660351] schedule_timeout+0x13f/0x290
[ 1358.665645] ? hrtimer_nanosleep_restart+0x1c0/0x1c0
[ 1358.672011] ? trace_contention_end+0xb3/0x1a0
[ 1358.677795] ? hrtimers_cpu_dying+0x260/0x260
[ 1358.683484] ? ww_mutex_lock+0x160/0x160
[ 1358.688685] msleep+0x1b/0x30
[ 1358.692807] napi_set_threaded+0x3a5/0x440
[ 1358.698202] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.703701] netif_set_threaded+0x257/0x400
[ 1358.709190] threaded_store+0x1b8/0x220
[ 1358.714293] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.719588] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.724884] ? sysfs_kf_write+0x65/0x160
[ 1358.730079] ? sysfs_file_ops+0x140/0x140
[ 1358.735373] kernfs_fop_write_iter+0x335/0x510
[ 1358.741159] ? kernfs_ops+0xf0/0xf0
[ 1358.745863] new_sync_write+0x333/0x750
[ 1358.750954] ? __lock_acquire+0x508/0xc10
[ 1358.756249] ? new_sync_read+0x740/0x740
[ 1358.761448] ? lock_acquire.part.0+0xbc/0x260
[ 1358.767131] ? ksys_write+0x116/0x250
[ 1358.772040] vfs_write+0x657/0x9b0
[ 1358.776653] ksys_write+0x116/0x250
[ 1358.781365] ? __ia32_sys_read+0xc0/0xc0
[ 1358.786564] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.792152] ? rcu_is_watching+0x15/0xd0
[ 1358.797353] do_syscall_64+0xf3/0x5e0
[ 1358.802261] ? trace_hardirqs_off+0xd/0x30
[ 1358.807644] ? exc_page_fault+0xda/0xf0
[ 1358.812744] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.819207] RIP: 0033:0x7fab236c622e
[ 1358.824013] RSP: 002b:00007fff237657b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.833307] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fab236c622e
[ 1358.842111] RDX: 0000000000000001 RSI: 00005588e6742f40 RDI: 0000000000000007
[ 1358.850921] RBP: 00007fff237657c0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.859727] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fab2353e390
[ 1358.868535] R13: 00005588e6742f40 R14: 0000000000000007 R15: 0000000000000003
[ 1358.877338]
[ 1358.880606] \x0aShowing all locks held in the system:
[ 1358.889146] 1 lock held by khungtaskd/185:
[ 1358.894539] #0: ffffffff974cdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.906022] 4 locks held by python3/3511:
[ 1358.911307] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.921198] #1: ff110001620f8288 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.932161] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.943318] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.953419] 4 locks held by python3/5590:
[ 1358.958714] #0: ff110001302f13f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.968603] #1: ff110001c1a5f488 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.979554] #2: ff110017da7ad0f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.990713] #3: ff110017da7e1d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.000805]
[ 1359.003269] =============================================\x0a