[ 761.446458] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 761.450978] nipa-hw-worker: [4/38] Retrying drivers/net:napi_threaded.py
[ 761.613611] kselftest: Running tests in drivers/net
[ 989.741629] INFO: task python3:5083 blocked for more than 122 seconds.
[ 989.749861] Not tainted 7.0.0-rc5-mrfa-g36bf160680ba #1
[ 989.757110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.766701] task:python3 state:D stack:25312 pid:5083 tgid:5083 ppid:5082 task_flags:0x400100 flags:0x00080000
[ 989.779906] Call Trace:
[ 989.783448]
[ 989.786593] __schedule+0xad9/0x1490
[ 989.791409] ? io_schedule_timeout+0x130/0x130
[ 989.797196] ? find_held_lock+0x2b/0x80
[ 989.802302] ? __lock_release.isra.0+0x59/0x170
[ 989.808189] schedule+0xd4/0x260
[ 989.812610] schedule_preempt_disabled+0x18/0x30
[ 989.818589] __mutex_lock+0xd76/0x1f20
[ 989.823595] ? threaded_store+0x132/0x220
[ 989.828896] ? ww_mutex_lock+0x160/0x160
[ 989.834098] ? selinux_inode_free_security+0x310/0x310
[ 989.840670] ? cap_capable+0x1ed/0x3f0
[ 989.845677] ? sysfs_file_ops+0x140/0x140
[ 989.850975] ? threaded_store+0x132/0x220
[ 989.856272] threaded_store+0x132/0x220
[ 989.861375] ? tx_maxrate_store+0x4e0/0x4e0
[ 989.866871] ? sysfs_file_ops+0x140/0x140
[ 989.872168] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.877467] ? sysfs_kf_write+0x65/0x160
[ 989.882666] kernfs_fop_write_iter+0x335/0x510
[ 989.888451] ? kernfs_ops+0xf0/0xf0
[ 989.893167] new_sync_write+0x308/0x6f0
[ 989.898272] ? new_sync_read+0x6f0/0x6f0
[ 989.903466] ? lockdep_hardirqs_off+0x8f/0xe0
[ 989.909162] vfs_write+0x62a/0x980
[ 989.913779] ksys_write+0xff/0x200
[ 989.918394] ? __ia32_sys_read+0xc0/0xc0
[ 989.923593] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.929174] ? rcu_is_watching+0x15/0xd0
[ 989.934367] do_syscall_64+0xf3/0x5e0
[ 989.939274] ? exc_page_fault+0x9e/0xb0
[ 989.944378] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.950846] RIP: 0033:0x7fac923dbc5e
[ 989.955656] RSP: 002b:00007ffc162045d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.964956] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fac923dbc5e
[ 989.973772] RDX: 0000000000000001 RSI: 00005589a35f8110 RDI: 0000000000000007
[ 989.982579] RBP: 00007ffc162045e0 R08: 0000000000000000 R09: 0000000000000000
[ 989.991389] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fac92276e80
[ 990.000199] R13: 00005589a35f8110 R14: 0000000000000007 R15: 00007fac92adcb98
[ 990.009013]
[ 990.012284] INFO: task python3:5083 is blocked on a mutex likely owned by task python3:3102.
[ 990.022557] task:python3 state:D stack:25312 pid:3102 tgid:3102 ppid:3101 task_flags:0x400100 flags:0x00080802
[ 990.035758] Call Trace:
[ 990.039300]
[ 990.042449] __schedule+0xad9/0x1490
[ 990.047261] ? io_schedule_timeout+0x130/0x130
[ 990.053043] ? find_held_lock+0x2b/0x80
[ 990.058145] ? __try_to_del_timer_sync+0x154/0x1f0
[ 990.064326] schedule+0xd4/0x260
[ 990.068743] schedule_timeout+0x103/0x260
[ 990.074040] ? rcu_is_watching+0x15/0xd0
[ 990.079239] ? hrtimer_nanosleep_restart+0x180/0x180
[ 990.085608] ? hrtimers_cpu_dying+0x250/0x250
[ 990.091298] ? ww_mutex_lock+0x160/0x160
[ 990.096496] msleep+0x1b/0x30
[ 990.100621] napi_set_threaded+0x376/0x410
[ 990.106017] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.111504] netif_set_threaded+0x254/0x400
[ 990.116997] ? sysfs_file_ops+0x140/0x140
[ 990.122293] threaded_store+0x1bb/0x220
[ 990.127394] ? tx_maxrate_store+0x4e0/0x4e0
[ 990.132888] ? sysfs_file_ops+0x140/0x140
[ 990.138184] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.143482] ? sysfs_kf_write+0x65/0x160
[ 990.148680] kernfs_fop_write_iter+0x335/0x510
[ 990.154468] ? kernfs_ops+0xf0/0xf0
[ 990.159179] new_sync_write+0x308/0x6f0
[ 990.164283] ? new_sync_read+0x6f0/0x6f0
[ 990.169482] ? lock_acquire.part.0+0xbc/0x260
[ 990.175171] ? ksys_write+0xff/0x200
[ 990.179984] vfs_write+0x62a/0x980
[ 990.184598] ksys_write+0xff/0x200
[ 990.189213] ? __ia32_sys_read+0xc0/0xc0
[ 990.194412] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.200001] ? rcu_is_watching+0x15/0xd0
[ 990.205203] do_syscall_64+0xf3/0x5e0
[ 990.210109] ? exc_page_fault+0x9e/0xb0
[ 990.215213] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.221679] RIP: 0033:0x7f0228ffec5e
[ 990.226486] RSP: 002b:00007fff119bb000 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.235782] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0228ffec5e
[ 990.244589] RDX: 0000000000000001 RSI: 0000555f27a86110 RDI: 0000000000000007
[ 990.253390] RBP: 00007fff119bb010 R08: 0000000000000000 R09: 0000000000000000
[ 990.262200] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0228e99e80
[ 990.271007] R13: 0000555f27a86110 R14: 0000000000000007 R15: 00007f0228e2a480
[ 990.279823]
[ 990.283084] \x0aShowing all locks held in the system:
[ 990.291624] 1 lock held by khungtaskd/185:
[ 990.297018] #0: ffffffff948ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.308506] 4 locks held by python3/3102:
[ 990.313804] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 990.323594] #1: ff1100014d884a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 990.334558] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.345714] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.355809] 4 locks held by python3/5083:
[ 990.361106] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 990.370897] #1: ff110001bb3ba088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 990.381858] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.393018] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.403112]
[ 990.405581] =============================================\x0a
[ 1112.621978] INFO: task python3:5083 blocked for more than 245 seconds.
[ 1112.630122] Not tainted 7.0.0-rc5-mrfa-g36bf160680ba #1
[ 1112.637362] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.646954] task:python3 state:D stack:25312 pid:5083 tgid:5083 ppid:5082 task_flags:0x400100 flags:0x00080000
[ 1112.660158] Call Trace:
[ 1112.663699]
[ 1112.666848] __schedule+0xad9/0x1490
[ 1112.671666] ? io_schedule_timeout+0x130/0x130
[ 1112.677453] ? find_held_lock+0x2b/0x80
[ 1112.682560] ? __lock_release.isra.0+0x59/0x170
[ 1112.688445] schedule+0xd4/0x260
[ 1112.692862] schedule_preempt_disabled+0x18/0x30
[ 1112.698844] __mutex_lock+0xd76/0x1f20
[ 1112.703848] ? threaded_store+0x132/0x220
[ 1112.709147] ? ww_mutex_lock+0x160/0x160
[ 1112.714346] ? selinux_inode_free_security+0x310/0x310
[ 1112.720915] ? cap_capable+0x1ed/0x3f0
[ 1112.725928] ? sysfs_file_ops+0x140/0x140
[ 1112.731236] ? threaded_store+0x132/0x220
[ 1112.736527] threaded_store+0x132/0x220
[ 1112.741629] ? tx_maxrate_store+0x4e0/0x4e0
[ 1112.747112] ? sysfs_file_ops+0x140/0x140
[ 1112.752410] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.757709] ? sysfs_kf_write+0x65/0x160
[ 1112.762911] kernfs_fop_write_iter+0x335/0x510
[ 1112.768696] ? kernfs_ops+0xf0/0xf0
[ 1112.773400] new_sync_write+0x308/0x6f0
[ 1112.778494] ? new_sync_read+0x6f0/0x6f0
[ 1112.783694] ? lockdep_hardirqs_off+0x8f/0xe0
[ 1112.789379] vfs_write+0x62a/0x980
[ 1112.793995] ksys_write+0xff/0x200
[ 1112.798608] ? __ia32_sys_read+0xc0/0xc0
[ 1112.803811] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.809403] ? rcu_is_watching+0x15/0xd0
[ 1112.814606] do_syscall_64+0xf3/0x5e0
[ 1112.819517] ? exc_page_fault+0x9e/0xb0
[ 1112.824619] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.831089] RIP: 0033:0x7fac923dbc5e
[ 1112.835901] RSP: 002b:00007ffc162045d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.845199] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fac923dbc5e
[ 1112.854009] RDX: 0000000000000001 RSI: 00005589a35f8110 RDI: 0000000000000007
[ 1112.862818] RBP: 00007ffc162045e0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.871626] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fac92276e80
[ 1112.880434] R13: 00005589a35f8110 R14: 0000000000000007 R15: 00007fac92adcb98
[ 1112.889249]
[ 1112.892520] INFO: task python3:5083 is blocked on a mutex likely owned by task python3:3102.
[ 1112.902794] task:python3 state:D stack:25312 pid:3102 tgid:3102 ppid:3101 task_flags:0x400100 flags:0x00080802
[ 1112.916000] Call Trace:
[ 1112.919539]
[ 1112.922691] __schedule+0xad9/0x1490
[ 1112.927502] ? io_schedule_timeout+0x130/0x130
[ 1112.933287] ? find_held_lock+0x2b/0x80
[ 1112.938391] ? __try_to_del_timer_sync+0x154/0x1f0
[ 1112.944570] schedule+0xd4/0x260
[ 1112.948988] schedule_timeout+0x103/0x260
[ 1112.954286] ? rcu_is_watching+0x15/0xd0
[ 1112.959486] ? hrtimer_nanosleep_restart+0x180/0x180
[ 1112.965848] ? hrtimers_cpu_dying+0x250/0x250
[ 1112.971540] ? ww_mutex_lock+0x160/0x160
[ 1112.976741] msleep+0x1b/0x30
[ 1112.980867] napi_set_threaded+0x376/0x410
[ 1112.986261] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.991755] netif_set_threaded+0x254/0x400
[ 1112.997249] ? sysfs_file_ops+0x140/0x140
[ 1113.002544] threaded_store+0x1bb/0x220
[ 1113.007645] ? tx_maxrate_store+0x4e0/0x4e0
[ 1113.013135] ? sysfs_file_ops+0x140/0x140
[ 1113.018434] ? sysfs_file_kobj+0xbd/0x1e0
[ 1113.023729] ? sysfs_kf_write+0x65/0x160
[ 1113.028929] kernfs_fop_write_iter+0x335/0x510
[ 1113.034706] ? kernfs_ops+0xf0/0xf0
[ 1113.039419] new_sync_write+0x308/0x6f0
[ 1113.044520] ? new_sync_read+0x6f0/0x6f0
[ 1113.049722] ? lock_acquire.part.0+0xbc/0x260
[ 1113.055407] ? ksys_write+0xff/0x200
[ 1113.060222] vfs_write+0x62a/0x980
[ 1113.064836] ksys_write+0xff/0x200
[ 1113.069450] ? __ia32_sys_read+0xc0/0xc0
[ 1113.074650] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.080242] ? rcu_is_watching+0x15/0xd0
[ 1113.085441] do_syscall_64+0xf3/0x5e0
[ 1113.090350] ? exc_page_fault+0x9e/0xb0
[ 1113.095452] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.101913] RIP: 0033:0x7f0228ffec5e
[ 1113.106720] RSP: 002b:00007fff119bb000 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.116017] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0228ffec5e
[ 1113.124824] RDX: 0000000000000001 RSI: 0000555f27a86110 RDI: 0000000000000007
[ 1113.133634] RBP: 00007fff119bb010 R08: 0000000000000000 R09: 0000000000000000
[ 1113.142442] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0228e99e80
[ 1113.151252] R13: 0000555f27a86110 R14: 0000000000000007 R15: 00007f0228e2a480
[ 1113.160067]
[ 1113.163333] \x0aShowing all locks held in the system:
[ 1113.171851] 1 lock held by systemd/1:
[ 1113.176780] 1 lock held by khungtaskd/185:
[ 1113.182174] #0: ffffffff948ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.193659] 4 locks held by python3/3102:
[ 1113.198958] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1113.208753] #1: ff1100014d884a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1113.219721] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.230876] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.240973] 4 locks held by python3/5083:
[ 1113.246270] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1113.256061] #1: ff110001bb3ba088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1113.267024] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.278183] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.288273] 5 locks held by (time-dir)/6159:
[ 1113.293862] #0: ff1100012eb493f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1113.303653] #1: ff110001f398b288 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1113.314615] #2: ffffffff94937248 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xd2/0x3b0
[ 1113.325284] #3: ffffffff94768f30 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e7/0x640
[ 1113.336927] #4: ffffffff94936f50 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f3/0x640
[ 1113.349348]
[ 1113.351817] =============================================\x0a
[ 1235.504313] INFO: task python3:5083 blocked for more than 368 seconds.
[ 1235.512461] Not tainted 7.0.0-rc5-mrfa-g36bf160680ba #1
[ 1235.519710] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.529302] task:python3 state:D stack:25312 pid:5083 tgid:5083 ppid:5082 task_flags:0x400100 flags:0x00080002
[ 1235.542509] Call Trace:
[ 1235.546049]
[ 1235.549202] __schedule+0xad9/0x1490
[ 1235.554021] ? io_schedule_timeout+0x130/0x130
[ 1235.559806] ? find_held_lock+0x2b/0x80
[ 1235.564913] ? __lock_release.isra.0+0x59/0x170
[ 1235.570798] schedule+0xd4/0x260
[ 1235.575219] schedule_preempt_disabled+0x18/0x30
[ 1235.581200] __mutex_lock+0xd76/0x1f20
[ 1235.586208] ? threaded_store+0x132/0x220
[ 1235.591510] ? ww_mutex_lock+0x160/0x160
[ 1235.596709] ? selinux_inode_free_security+0x310/0x310
[ 1235.603278] ? cap_capable+0x1ed/0x3f0
[ 1235.608286] ? sysfs_file_ops+0x140/0x140
[ 1235.613586] ? threaded_store+0x132/0x220
[ 1235.618885] threaded_store+0x132/0x220
[ 1235.623988] ? tx_maxrate_store+0x4e0/0x4e0
[ 1235.629485] ? sysfs_file_ops+0x140/0x140
[ 1235.634783] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.640083] ? sysfs_kf_write+0x65/0x160
[ 1235.645284] kernfs_fop_write_iter+0x335/0x510
[ 1235.651069] ? kernfs_ops+0xf0/0xf0
[ 1235.655781] new_sync_write+0x308/0x6f0
[ 1235.660878] ? new_sync_read+0x6f0/0x6f0
[ 1235.666077] ? lockdep_hardirqs_off+0x8f/0xe0
[ 1235.671770] vfs_write+0x62a/0x980
[ 1235.676385] ksys_write+0xff/0x200
[ 1235.680990] ? __ia32_sys_read+0xc0/0xc0
[ 1235.686191] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.691782] ? rcu_is_watching+0x15/0xd0
[ 1235.696986] do_syscall_64+0xf3/0x5e0
[ 1235.701896] ? exc_page_fault+0x9e/0xb0
[ 1235.706997] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.713466] RIP: 0033:0x7fac923dbc5e
[ 1235.718277] RSP: 002b:00007ffc162045d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.727575] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fac923dbc5e
[ 1235.736384] RDX: 0000000000000001 RSI: 00005589a35f8110 RDI: 0000000000000007
[ 1235.745192] RBP: 00007ffc162045e0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.754000] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fac92276e80
[ 1235.762798] R13: 00005589a35f8110 R14: 0000000000000007 R15: 00007fac92adcb98
[ 1235.771612]
[ 1235.774888] INFO: task python3:5083 is blocked on a mutex likely owned by task python3:3102.
[ 1235.785162] task:python3 state:D stack:25312 pid:3102 tgid:3102 ppid:3101 task_flags:0x400100 flags:0x00080802
[ 1235.798364] Call Trace:
[ 1235.801903]
[ 1235.805054] __schedule+0xad9/0x1490
[ 1235.809865] ? io_schedule_timeout+0x130/0x130
[ 1235.815650] ? find_held_lock+0x2b/0x80
[ 1235.820745] ? __try_to_del_timer_sync+0x154/0x1f0
[ 1235.826923] schedule+0xd4/0x260
[ 1235.831340] schedule_timeout+0x103/0x260
[ 1235.836638] ? rcu_is_watching+0x15/0xd0
[ 1235.841838] ? hrtimer_nanosleep_restart+0x180/0x180
[ 1235.848209] ? hrtimers_cpu_dying+0x250/0x250
[ 1235.853891] ? ww_mutex_lock+0x160/0x160
[ 1235.859091] msleep+0x1b/0x30
[ 1235.863209] napi_set_threaded+0x376/0x410
[ 1235.868607] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.874101] netif_set_threaded+0x254/0x400
[ 1235.879586] ? sysfs_file_ops+0x140/0x140
[ 1235.884880] threaded_store+0x1bb/0x220
[ 1235.889984] ? tx_maxrate_store+0x4e0/0x4e0
[ 1235.895478] ? sysfs_file_ops+0x140/0x140
[ 1235.900775] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.906061] ? sysfs_kf_write+0x65/0x160
[ 1235.911263] kernfs_fop_write_iter+0x335/0x510
[ 1235.917048] ? kernfs_ops+0xf0/0xf0
[ 1235.921752] new_sync_write+0x308/0x6f0
[ 1235.926853] ? new_sync_read+0x6f0/0x6f0
[ 1235.932055] ? lock_acquire.part.0+0xbc/0x260
[ 1235.937740] ? ksys_write+0xff/0x200
[ 1235.942551] vfs_write+0x62a/0x980
[ 1235.947165] ksys_write+0xff/0x200
[ 1235.951779] ? __ia32_sys_read+0xc0/0xc0
[ 1235.956979] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.962558] ? rcu_is_watching+0x15/0xd0
[ 1235.967758] do_syscall_64+0xf3/0x5e0
[ 1235.972667] ? exc_page_fault+0x9e/0xb0
[ 1235.977768] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.984237] RIP: 0033:0x7f0228ffec5e
[ 1235.989046] RSP: 002b:00007fff119bb000 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.998343] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0228ffec5e
[ 1236.007156] RDX: 0000000000000001 RSI: 0000555f27a86110 RDI: 0000000000000007
[ 1236.015967] RBP: 00007fff119bb010 R08: 0000000000000000 R09: 0000000000000000
[ 1236.024776] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0228e99e80
[ 1236.033585] R13: 0000555f27a86110 R14: 0000000000000007 R15: 00007f0228e2a480
[ 1236.042399]
[ 1236.045668] \x0aShowing all locks held in the system:
[ 1236.054206] 1 lock held by khungtaskd/185:
[ 1236.059602] #0: ffffffff948ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.071074] 3 locks held by systemd-journal/806:
[ 1236.077051] #0: ff1100180e7b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18f/0x1490
[ 1236.086937] #1: ffffffff94ac91e0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.099756] #2: ff1100019ff86d30 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0xf4/0x280
[ 1236.111218] 4 locks held by python3/3102:
[ 1236.116516] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1236.126305] #1: ff1100014d884a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1236.137266] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.148422] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.158520] 4 locks held by python3/5083:
[ 1236.163815] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1236.173605] #1: ff110001bb3ba088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1236.184566] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.195714] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.205799]
[ 1236.208264] =============================================\x0a
[ 1358.384663] INFO: task python3:5083 blocked for more than 491 seconds.
[ 1358.392822] Not tainted 7.0.0-rc5-mrfa-g36bf160680ba #1
[ 1358.400073] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.409662] task:python3 state:D stack:25312 pid:5083 tgid:5083 ppid:5082 task_flags:0x400100 flags:0x00080002
[ 1358.422861] Call Trace:
[ 1358.426400]
[ 1358.429550] __schedule+0xad9/0x1490
[ 1358.434366] ? io_schedule_timeout+0x130/0x130
[ 1358.440150] ? find_held_lock+0x2b/0x80
[ 1358.445258] ? __lock_release.isra.0+0x59/0x170
[ 1358.451139] schedule+0xd4/0x260
[ 1358.455555] schedule_preempt_disabled+0x18/0x30
[ 1358.461535] __mutex_lock+0xd76/0x1f20
[ 1358.466542] ? threaded_store+0x132/0x220
[ 1358.471838] ? ww_mutex_lock+0x160/0x160
[ 1358.477037] ? selinux_inode_free_security+0x310/0x310
[ 1358.483596] ? cap_capable+0x1ed/0x3f0
[ 1358.488604] ? sysfs_file_ops+0x140/0x140
[ 1358.493902] ? threaded_store+0x132/0x220
[ 1358.499196] threaded_store+0x132/0x220
[ 1358.504298] ? tx_maxrate_store+0x4e0/0x4e0
[ 1358.509781] ? sysfs_file_ops+0x140/0x140
[ 1358.515077] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.520372] ? sysfs_kf_write+0x65/0x160
[ 1358.525573] kernfs_fop_write_iter+0x335/0x510
[ 1358.531358] ? kernfs_ops+0xf0/0xf0
[ 1358.536061] new_sync_write+0x308/0x6f0
[ 1358.541154] ? new_sync_read+0x6f0/0x6f0
[ 1358.546352] ? lockdep_hardirqs_off+0x8f/0xe0
[ 1358.552048] vfs_write+0x62a/0x980
[ 1358.556662] ksys_write+0xff/0x200
[ 1358.561275] ? __ia32_sys_read+0xc0/0xc0
[ 1358.566471] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.572059] ? rcu_is_watching+0x15/0xd0
[ 1358.577259] do_syscall_64+0xf3/0x5e0
[ 1358.582158] ? exc_page_fault+0x9e/0xb0
[ 1358.587261] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.593730] RIP: 0033:0x7fac923dbc5e
[ 1358.598537] RSP: 002b:00007ffc162045d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.607837] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fac923dbc5e
[ 1358.616644] RDX: 0000000000000001 RSI: 00005589a35f8110 RDI: 0000000000000007
[ 1358.625449] RBP: 00007ffc162045e0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.634257] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fac92276e80
[ 1358.643064] R13: 00005589a35f8110 R14: 0000000000000007 R15: 00007fac92adcb98
[ 1358.651877]
[ 1358.655189] INFO: task python3:5083 is blocked on a mutex likely owned by task python3:3102.
[ 1358.665472] task:python3 state:D stack:25312 pid:3102 tgid:3102 ppid:3101 task_flags:0x400100 flags:0x00080802
[ 1358.678679] Call Trace:
[ 1358.682219]
[ 1358.685369] __schedule+0xad9/0x1490
[ 1358.690181] ? io_schedule_timeout+0x130/0x130
[ 1358.695964] ? find_held_lock+0x2b/0x80
[ 1358.701068] ? __try_to_del_timer_sync+0x154/0x1f0
[ 1358.707245] schedule+0xd4/0x260
[ 1358.711662] schedule_timeout+0x103/0x260
[ 1358.716959] ? rcu_is_watching+0x15/0xd0
[ 1358.722158] ? hrtimer_nanosleep_restart+0x180/0x180
[ 1358.728519] ? hrtimers_cpu_dying+0x250/0x250
[ 1358.734207] ? ww_mutex_lock+0x160/0x160
[ 1358.739397] msleep+0x1b/0x30
[ 1358.743522] napi_set_threaded+0x376/0x410
[ 1358.748916] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.754409] netif_set_threaded+0x254/0x400
[ 1358.759901] ? sysfs_file_ops+0x140/0x140
[ 1358.765185] threaded_store+0x1bb/0x220
[ 1358.770288] ? tx_maxrate_store+0x4e0/0x4e0
[ 1358.775778] ? sysfs_file_ops+0x140/0x140
[ 1358.781074] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.786368] ? sysfs_kf_write+0x65/0x160
[ 1358.791568] kernfs_fop_write_iter+0x335/0x510
[ 1358.797354] ? kernfs_ops+0xf0/0xf0
[ 1358.802066] new_sync_write+0x308/0x6f0
[ 1358.807166] ? new_sync_read+0x6f0/0x6f0
[ 1358.812367] ? lock_acquire.part.0+0xbc/0x260
[ 1358.818052] ? ksys_write+0xff/0x200
[ 1358.822862] vfs_write+0x62a/0x980
[ 1358.827466] ksys_write+0xff/0x200
[ 1358.832079] ? __ia32_sys_read+0xc0/0xc0
[ 1358.837274] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.842861] ? rcu_is_watching+0x15/0xd0
[ 1358.848059] do_syscall_64+0xf3/0x5e0
[ 1358.852963] ? exc_page_fault+0x9e/0xb0
[ 1358.858064] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.864529] RIP: 0033:0x7f0228ffec5e
[ 1358.869336] RSP: 002b:00007fff119bb000 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.878635] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0228ffec5e
[ 1358.887440] RDX: 0000000000000001 RSI: 0000555f27a86110 RDI: 0000000000000007
[ 1358.896247] RBP: 00007fff119bb010 R08: 0000000000000000 R09: 0000000000000000
[ 1358.905055] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0228e99e80
[ 1358.913866] R13: 0000555f27a86110 R14: 0000000000000007 R15: 00007f0228e2a480
[ 1358.922679]
[ 1358.925948] \x0aShowing all locks held in the system:
[ 1358.934495] 1 lock held by khungtaskd/185:
[ 1358.939886] #0: ffffffff948ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.951361] 3 locks held by systemd-journal/806:
[ 1358.957339] #0: ff1100180e1b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18f/0x1490
[ 1358.967226] #1: ffffffff94ac91e0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: nd_jump_root+0x353/0x550
[ 1358.979554] #2: ff110001a0c2e440 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0xf4/0x280
[ 1358.991016] 4 locks held by python3/3102:
[ 1358.996310] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1359.006100] #1: ff1100014d884a88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1359.017064] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.028220] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.038318] 4 locks held by python3/5083:
[ 1359.043612] #0: ff1100012ea613f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200
[ 1359.053400] #1: ff110001bb3ba088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510
[ 1359.064362] #2: ff11000121a750f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.075515] #3: ff11000121ab9d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.085608]
[ 1359.088076] =============================================\x0a