[ 761.641621] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 761.645582] nipa-hw-worker: [5/41] Retrying drivers/net:napi_threaded.py [ 761.808836] kselftest: Running tests in drivers/net [ 989.741734] INFO: task python3:5243 blocked for more than 122 seconds. [ 989.749958] Not tainted 7.0.0-rc5-lqbr-gd8ddcf9dcc28 #1 [ 989.757207] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.766788] task:python3 state:D stack:25312 pid:5243 tgid:5243 ppid:5242 task_flags:0x400100 flags:0x00080000 [ 989.779992] Call Trace: [ 989.783531] [ 989.786680] __schedule+0xad9/0x1490 [ 989.791495] ? io_schedule_timeout+0x130/0x130 [ 989.797278] ? find_held_lock+0x2b/0x80 [ 989.802380] ? __lock_release.isra.0+0x59/0x170 [ 989.808265] schedule+0xd4/0x260 [ 989.812681] schedule_preempt_disabled+0x18/0x30 [ 989.818659] __mutex_lock+0xd76/0x1f20 [ 989.823663] ? threaded_store+0x132/0x220 [ 989.828962] ? ww_mutex_lock+0x160/0x160 [ 989.834158] ? selinux_inode_free_security+0x310/0x310 [ 989.840729] ? cap_capable+0x1ed/0x3f0 [ 989.845734] ? sysfs_file_ops+0x140/0x140 [ 989.851030] ? threaded_store+0x132/0x220 [ 989.856327] threaded_store+0x132/0x220 [ 989.861433] ? tx_maxrate_store+0x4e0/0x4e0 [ 989.866927] ? sysfs_file_ops+0x140/0x140 [ 989.872221] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.877517] ? sysfs_kf_write+0x65/0x160 [ 989.882714] kernfs_fop_write_iter+0x335/0x510 [ 989.888497] ? kernfs_ops+0xf0/0xf0 [ 989.893208] new_sync_write+0x308/0x6f0 [ 989.898311] ? new_sync_read+0x6f0/0x6f0 [ 989.903509] ? lock_acquire.part.0+0xbc/0x260 [ 989.909194] ? ksys_write+0xff/0x200 [ 989.914003] vfs_write+0x62a/0x980 [ 989.918619] ksys_write+0xff/0x200 [ 989.923224] ? __ia32_sys_read+0xc0/0xc0 [ 989.928423] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.934011] ? rcu_is_watching+0x15/0xd0 [ 989.939213] do_syscall_64+0xf3/0x5e0 [ 989.944118] ? exc_page_fault+0x9e/0xb0 [ 989.949218] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.955685] RIP: 0033:0x7fb6b3facc5e [ 989.960494] RSP: 002b:00007fff64225610 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.969794] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb6b3facc5e [ 989.978602] RDX: 0000000000000001 RSI: 000056207e514760 RDI: 0000000000000007 [ 989.987398] RBP: 00007fff64225620 R08: 0000000000000000 R09: 0000000000000000 [ 989.996204] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb6b3e47e80 [ 990.005013] R13: 000056207e514760 R14: 0000000000000007 R15: 00007fb6b46adb98 [ 990.013827] [ 990.017115] INFO: task python3:5243 is blocked on a mutex likely owned by task python3:3268. [ 990.027390] task:python3 state:D stack:25312 pid:3268 tgid:3268 ppid:3267 task_flags:0x400100 flags:0x00080802 [ 990.040590] Call Trace: [ 990.044117] [ 990.047266] __schedule+0xad9/0x1490 [ 990.052075] ? io_schedule_timeout+0x130/0x130 [ 990.057856] ? find_held_lock+0x2b/0x80 [ 990.062955] ? __try_to_del_timer_sync+0x154/0x1f0 [ 990.069134] schedule+0xd4/0x260 [ 990.073553] schedule_timeout+0x103/0x260 [ 990.078839] ? rcu_is_watching+0x15/0xd0 [ 990.084038] ? hrtimer_nanosleep_restart+0x180/0x180 [ 990.090407] ? hrtimers_cpu_dying+0x250/0x250 [ 990.096098] ? ww_mutex_lock+0x160/0x160 [ 990.101296] msleep+0x1b/0x30 [ 990.105420] napi_set_threaded+0x376/0x410 [ 990.110815] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.116310] netif_set_threaded+0x254/0x400 [ 990.121801] ? sysfs_file_ops+0x140/0x140 [ 990.127094] threaded_store+0x1bb/0x220 [ 990.132196] ? tx_maxrate_store+0x4e0/0x4e0 [ 990.137685] ? sysfs_file_ops+0x140/0x140 [ 990.142981] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.148277] ? sysfs_kf_write+0x65/0x160 [ 990.153475] kernfs_fop_write_iter+0x335/0x510 [ 990.159258] ? kernfs_ops+0xf0/0xf0 [ 990.163968] new_sync_write+0x308/0x6f0 [ 990.169071] ? new_sync_read+0x6f0/0x6f0 [ 990.174271] ? lock_acquire.part.0+0xbc/0x260 [ 990.179946] ? ksys_write+0xff/0x200 [ 990.184755] vfs_write+0x62a/0x980 [ 990.189369] ksys_write+0xff/0x200 [ 990.193981] ? __ia32_sys_read+0xc0/0xc0 [ 990.199181] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.204762] ? rcu_is_watching+0x15/0xd0 [ 990.209955] do_syscall_64+0xf3/0x5e0 [ 990.214858] ? exc_page_fault+0x9e/0xb0 [ 990.219960] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.226428] RIP: 0033:0x7f3de85f7c5e [ 990.231224] RSP: 002b:00007fff46626810 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.240522] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3de85f7c5e [ 990.249329] RDX: 0000000000000001 RSI: 000055b3c07af760 RDI: 0000000000000007 [ 990.258141] RBP: 00007fff46626820 R08: 0000000000000000 R09: 0000000000000000 [ 990.266950] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3de8492e80 [ 990.275758] R13: 000055b3c07af760 R14: 0000000000000007 R15: 00007f3de8422480 [ 990.284561] [ 990.287822] \x0aShowing all locks held in the system: [ 990.296361] 1 lock held by khungtaskd/185: [ 990.301754] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.313235] 4 locks held by python3/3268: [ 990.318527] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 990.328316] #1: ff1100013ed63e88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 990.339277] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.350432] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.360526] 4 locks held by python3/5243: [ 990.365817] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 990.375606] #1: ff110001c5e15088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 990.386566] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.397730] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.407818] [ 990.410283] =============================================\x0a [ 1112.622061] INFO: task python3:5243 blocked for more than 245 seconds. [ 1112.630208] Not tainted 7.0.0-rc5-lqbr-gd8ddcf9dcc28 #1 [ 1112.637459] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.647051] task:python3 state:D stack:25312 pid:5243 tgid:5243 ppid:5242 task_flags:0x400100 flags:0x00080000 [ 1112.660259] Call Trace: [ 1112.663802] [ 1112.666943] __schedule+0xad9/0x1490 [ 1112.671762] ? io_schedule_timeout+0x130/0x130 [ 1112.671767] ? find_held_lock+0x2b/0x80 [ 1112.671774] ? __lock_release.isra.0+0x59/0x170 [ 1112.671780] schedule+0xd4/0x260 [ 1112.671783] schedule_preempt_disabled+0x18/0x30 [ 1112.698996] __mutex_lock+0xd76/0x1f20 [ 1112.704000] ? threaded_store+0x132/0x220 [ 1112.709302] ? ww_mutex_lock+0x160/0x160 [ 1112.714502] ? selinux_inode_free_security+0x310/0x310 [ 1112.721072] ? cap_capable+0x1ed/0x3f0 [ 1112.726080] ? sysfs_file_ops+0x140/0x140 [ 1112.731379] ? threaded_store+0x132/0x220 [ 1112.736676] threaded_store+0x132/0x220 [ 1112.741781] ? tx_maxrate_store+0x4e0/0x4e0 [ 1112.747274] ? sysfs_file_ops+0x140/0x140 [ 1112.752571] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.757861] ? sysfs_kf_write+0x65/0x160 [ 1112.763060] kernfs_fop_write_iter+0x335/0x510 [ 1112.768845] ? kernfs_ops+0xf0/0xf0 [ 1112.773559] new_sync_write+0x308/0x6f0 [ 1112.778661] ? new_sync_read+0x6f0/0x6f0 [ 1112.783862] ? lock_acquire.part.0+0xbc/0x260 [ 1112.789550] ? ksys_write+0xff/0x200 [ 1112.794362] vfs_write+0x62a/0x980 [ 1112.798976] ksys_write+0xff/0x200 [ 1112.803591] ? __ia32_sys_read+0xc0/0xc0 [ 1112.808792] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.814382] ? rcu_is_watching+0x15/0xd0 [ 1112.819584] do_syscall_64+0xf3/0x5e0 [ 1112.824490] ? exc_page_fault+0x9e/0xb0 [ 1112.829592] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.836059] RIP: 0033:0x7fb6b3facc5e [ 1112.840869] RSP: 002b:00007fff64225610 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.850166] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb6b3facc5e [ 1112.858965] RDX: 0000000000000001 RSI: 000056207e514760 RDI: 0000000000000007 [ 1112.867772] RBP: 00007fff64225620 R08: 0000000000000000 R09: 0000000000000000 [ 1112.876581] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb6b3e47e80 [ 1112.885389] R13: 000056207e514760 R14: 0000000000000007 R15: 00007fb6b46adb98 [ 1112.894195] [ 1112.897483] INFO: task python3:5243 is blocked on a mutex likely owned by task python3:3268. [ 1112.907759] task:python3 state:D stack:25312 pid:3268 tgid:3268 ppid:3267 task_flags:0x400100 flags:0x00080802 [ 1112.920959] Call Trace: [ 1112.924498] [ 1112.927651] __schedule+0xad9/0x1490 [ 1112.932463] ? io_schedule_timeout+0x130/0x130 [ 1112.938249] ? find_held_lock+0x2b/0x80 [ 1112.943353] ? __try_to_del_timer_sync+0x154/0x1f0 [ 1112.949533] schedule+0xd4/0x260 [ 1112.953949] schedule_timeout+0x103/0x260 [ 1112.959248] ? rcu_is_watching+0x15/0xd0 [ 1112.964448] ? hrtimer_nanosleep_restart+0x180/0x180 [ 1112.970818] ? hrtimers_cpu_dying+0x250/0x250 [ 1112.976512] ? ww_mutex_lock+0x160/0x160 [ 1112.981712] msleep+0x1b/0x30 [ 1112.985838] napi_set_threaded+0x376/0x410 [ 1112.991233] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.996725] netif_set_threaded+0x254/0x400 [ 1113.002208] ? sysfs_file_ops+0x140/0x140 [ 1113.007504] threaded_store+0x1bb/0x220 [ 1113.012610] ? tx_maxrate_store+0x4e0/0x4e0 [ 1113.018101] ? sysfs_file_ops+0x140/0x140 [ 1113.023398] ? sysfs_file_kobj+0xbd/0x1e0 [ 1113.028693] ? sysfs_kf_write+0x65/0x160 [ 1113.033900] kernfs_fop_write_iter+0x335/0x510 [ 1113.039685] ? kernfs_ops+0xf0/0xf0 [ 1113.044397] new_sync_write+0x308/0x6f0 [ 1113.049494] ? new_sync_read+0x6f0/0x6f0 [ 1113.054695] ? lock_acquire.part.0+0xbc/0x260 [ 1113.060381] ? ksys_write+0xff/0x200 [ 1113.065195] vfs_write+0x62a/0x980 [ 1113.069812] ksys_write+0xff/0x200 [ 1113.074425] ? __ia32_sys_read+0xc0/0xc0 [ 1113.079623] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.085215] ? rcu_is_watching+0x15/0xd0 [ 1113.090418] do_syscall_64+0xf3/0x5e0 [ 1113.095326] ? exc_page_fault+0x9e/0xb0 [ 1113.100420] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.106890] RIP: 0033:0x7f3de85f7c5e [ 1113.111700] RSP: 002b:00007fff46626810 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.120999] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3de85f7c5e [ 1113.129808] RDX: 0000000000000001 RSI: 000055b3c07af760 RDI: 0000000000000007 [ 1113.138617] RBP: 00007fff46626820 R08: 0000000000000000 R09: 0000000000000000 [ 1113.147427] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3de8492e80 [ 1113.156233] R13: 000055b3c07af760 R14: 0000000000000007 R15: 00007f3de8422480 [ 1113.165047] [ 1113.168310] \x0aShowing all locks held in the system: [ 1113.176849] 1 lock held by khungtaskd/185: [ 1113.182244] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.193721] 4 locks held by python3/3268: [ 1113.199016] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1113.208806] #1: ff1100013ed63e88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1113.219768] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.230928] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.241028] 4 locks held by python3/5243: [ 1113.246316] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1113.256108] #1: ff110001c5e15088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1113.267067] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.278224] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.288314] 5 locks held by (time-dir)/6324: [ 1113.293905] #0: ff1100012e2713f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1113.303694] #1: ff110001e80bd088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1113.314655] #2: ffffffffb7137248 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xd2/0x3b0 [ 1113.325325] #3: ffffffffb6f68f30 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e7/0x640 [ 1113.336959] #4: ffffffffb7136f50 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f3/0x640 [ 1113.349386] [ 1113.351851] =============================================\x0a [ 1235.504392] INFO: task python3:5243 blocked for more than 368 seconds. [ 1235.512540] Not tainted 7.0.0-rc5-lqbr-gd8ddcf9dcc28 #1 [ 1235.519780] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.529370] task:python3 state:D stack:25312 pid:5243 tgid:5243 ppid:5242 task_flags:0x400100 flags:0x00080002 [ 1235.542573] Call Trace: [ 1235.546110] [ 1235.549260] __schedule+0xad9/0x1490 [ 1235.554076] ? io_schedule_timeout+0x130/0x130 [ 1235.559861] ? find_held_lock+0x2b/0x80 [ 1235.564966] ? __lock_release.isra.0+0x59/0x170 [ 1235.570852] schedule+0xd4/0x260 [ 1235.575268] schedule_preempt_disabled+0x18/0x30 [ 1235.581247] __mutex_lock+0xd76/0x1f20 [ 1235.586253] ? threaded_store+0x132/0x220 [ 1235.591553] ? ww_mutex_lock+0x160/0x160 [ 1235.596751] ? selinux_inode_free_security+0x310/0x310 [ 1235.603319] ? cap_capable+0x1ed/0x3f0 [ 1235.608331] ? sysfs_file_ops+0x140/0x140 [ 1235.613637] ? threaded_store+0x132/0x220 [ 1235.618934] threaded_store+0x132/0x220 [ 1235.624026] ? tx_maxrate_store+0x4e0/0x4e0 [ 1235.629516] ? sysfs_file_ops+0x140/0x140 [ 1235.634819] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.640113] ? sysfs_kf_write+0x65/0x160 [ 1235.645312] kernfs_fop_write_iter+0x335/0x510 [ 1235.651097] ? kernfs_ops+0xf0/0xf0 [ 1235.655807] new_sync_write+0x308/0x6f0 [ 1235.660909] ? new_sync_read+0x6f0/0x6f0 [ 1235.666109] ? lock_acquire.part.0+0xbc/0x260 [ 1235.671794] ? ksys_write+0xff/0x200 [ 1235.676607] vfs_write+0x62a/0x980 [ 1235.681219] ksys_write+0xff/0x200 [ 1235.685834] ? __ia32_sys_read+0xc0/0xc0 [ 1235.691033] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.696613] ? rcu_is_watching+0x15/0xd0 [ 1235.701811] do_syscall_64+0xf3/0x5e0 [ 1235.706719] ? exc_page_fault+0x9e/0xb0 [ 1235.711820] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.718289] RIP: 0033:0x7fb6b3facc5e [ 1235.723099] RSP: 002b:00007fff64225610 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.732388] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb6b3facc5e [ 1235.741193] RDX: 0000000000000001 RSI: 000056207e514760 RDI: 0000000000000007 [ 1235.750002] RBP: 00007fff64225620 R08: 0000000000000000 R09: 0000000000000000 [ 1235.758808] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb6b3e47e80 [ 1235.767615] R13: 000056207e514760 R14: 0000000000000007 R15: 00007fb6b46adb98 [ 1235.776421] [ 1235.779705] INFO: task python3:5243 is blocked on a mutex likely owned by task python3:3268. [ 1235.789977] task:python3 state:D stack:25312 pid:3268 tgid:3268 ppid:3267 task_flags:0x400100 flags:0x00080802 [ 1235.803177] Call Trace: [ 1235.806716] [ 1235.809863] __schedule+0xad9/0x1490 [ 1235.814674] ? io_schedule_timeout+0x130/0x130 [ 1235.820450] ? find_held_lock+0x2b/0x80 [ 1235.825544] ? __try_to_del_timer_sync+0x154/0x1f0 [ 1235.831723] schedule+0xd4/0x260 [ 1235.836141] schedule_timeout+0x103/0x260 [ 1235.841429] ? rcu_is_watching+0x15/0xd0 [ 1235.846625] ? hrtimer_nanosleep_restart+0x180/0x180 [ 1235.852997] ? hrtimers_cpu_dying+0x250/0x250 [ 1235.858684] ? ww_mutex_lock+0x160/0x160 [ 1235.863883] msleep+0x1b/0x30 [ 1235.868010] napi_set_threaded+0x376/0x410 [ 1235.873403] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.878895] netif_set_threaded+0x254/0x400 [ 1235.884380] ? sysfs_file_ops+0x140/0x140 [ 1235.889676] threaded_store+0x1bb/0x220 [ 1235.894777] ? tx_maxrate_store+0x4e0/0x4e0 [ 1235.900268] ? sysfs_file_ops+0x140/0x140 [ 1235.905565] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.910859] ? sysfs_kf_write+0x65/0x160 [ 1235.916060] kernfs_fop_write_iter+0x335/0x510 [ 1235.921845] ? kernfs_ops+0xf0/0xf0 [ 1235.926549] new_sync_write+0x308/0x6f0 [ 1235.931651] ? new_sync_read+0x6f0/0x6f0 [ 1235.936840] ? lock_acquire.part.0+0xbc/0x260 [ 1235.942524] ? ksys_write+0xff/0x200 [ 1235.947335] vfs_write+0x62a/0x980 [ 1235.951950] ksys_write+0xff/0x200 [ 1235.956561] ? __ia32_sys_read+0xc0/0xc0 [ 1235.961758] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.967345] ? rcu_is_watching+0x15/0xd0 [ 1235.972546] do_syscall_64+0xf3/0x5e0 [ 1235.977456] ? exc_page_fault+0x9e/0xb0 [ 1235.982557] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.989024] RIP: 0033:0x7f3de85f7c5e [ 1235.993833] RSP: 002b:00007fff46626810 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1236.003131] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3de85f7c5e [ 1236.011936] RDX: 0000000000000001 RSI: 000055b3c07af760 RDI: 0000000000000007 [ 1236.020742] RBP: 00007fff46626820 R08: 0000000000000000 R09: 0000000000000000 [ 1236.029547] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3de8492e80 [ 1236.038352] R13: 000055b3c07af760 R14: 0000000000000007 R15: 00007f3de8422480 [ 1236.047163] [ 1236.050431] \x0aShowing all locks held in the system: [ 1236.058968] 1 lock held by khungtaskd/185: [ 1236.064360] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.075830] 3 locks held by systemd-journal/804: [ 1236.081809] #0: ff1100180ea372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18f/0x1490 [ 1236.091697] #1: ffffffffb72c91e0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1236.104504] #2: ff1100019f6c2bd0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0xf4/0x280 [ 1236.115962] 4 locks held by python3/3268: [ 1236.121248] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1236.131035] #1: ff1100013ed63e88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1236.141997] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.153155] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.163253] 4 locks held by python3/5243: [ 1236.168546] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1236.178336] #1: ff110001c5e15088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1236.189298] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.200452] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.210540] [ 1236.213004] =============================================\x0a [ 1358.385720] INFO: task python3:5243 blocked for more than 491 seconds. [ 1358.393876] Not tainted 7.0.0-rc5-lqbr-gd8ddcf9dcc28 #1 [ 1358.401126] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.410715] task:python3 state:D stack:25312 pid:5243 tgid:5243 ppid:5242 task_flags:0x400100 flags:0x00080002 [ 1358.423913] Call Trace: [ 1358.427451] [ 1358.430600] __schedule+0xad9/0x1490 [ 1358.435417] ? io_schedule_timeout+0x130/0x130 [ 1358.441201] ? find_held_lock+0x2b/0x80 [ 1358.446305] ? __lock_release.isra.0+0x59/0x170 [ 1358.452189] schedule+0xd4/0x260 [ 1358.456607] schedule_preempt_disabled+0x18/0x30 [ 1358.462583] __mutex_lock+0xd76/0x1f20 [ 1358.467585] ? threaded_store+0x132/0x220 [ 1358.472883] ? ww_mutex_lock+0x160/0x160 [ 1358.478080] ? selinux_inode_free_security+0x310/0x310 [ 1358.484648] ? cap_capable+0x1ed/0x3f0 [ 1358.489654] ? sysfs_file_ops+0x140/0x140 [ 1358.494954] ? threaded_store+0x132/0x220 [ 1358.500239] threaded_store+0x132/0x220 [ 1358.505340] ? tx_maxrate_store+0x4e0/0x4e0 [ 1358.510833] ? sysfs_file_ops+0x140/0x140 [ 1358.516121] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.521407] ? sysfs_kf_write+0x65/0x160 [ 1358.526607] kernfs_fop_write_iter+0x335/0x510 [ 1358.532391] ? kernfs_ops+0xf0/0xf0 [ 1358.537100] new_sync_write+0x308/0x6f0 [ 1358.542202] ? new_sync_read+0x6f0/0x6f0 [ 1358.547401] ? lock_acquire.part.0+0xbc/0x260 [ 1358.553087] ? ksys_write+0xff/0x200 [ 1358.557898] vfs_write+0x62a/0x980 [ 1358.562512] ksys_write+0xff/0x200 [ 1358.567123] ? __ia32_sys_read+0xc0/0xc0 [ 1358.572318] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.577909] ? rcu_is_watching+0x15/0xd0 [ 1358.583108] do_syscall_64+0xf3/0x5e0 [ 1358.588012] ? exc_page_fault+0x9e/0xb0 [ 1358.593113] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.599574] RIP: 0033:0x7fb6b3facc5e [ 1358.604384] RSP: 002b:00007fff64225610 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.613674] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb6b3facc5e [ 1358.622480] RDX: 0000000000000001 RSI: 000056207e514760 RDI: 0000000000000007 [ 1358.631286] RBP: 00007fff64225620 R08: 0000000000000000 R09: 0000000000000000 [ 1358.640090] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb6b3e47e80 [ 1358.648896] R13: 000056207e514760 R14: 0000000000000007 R15: 00007fb6b46adb98 [ 1358.657708] [ 1358.660984] INFO: task python3:5243 is blocked on a mutex likely owned by task python3:3268. [ 1358.671257] task:python3 state:D stack:25312 pid:3268 tgid:3268 ppid:3267 task_flags:0x400100 flags:0x00080802 [ 1358.684460] Call Trace: [ 1358.687998] [ 1358.691148] __schedule+0xad9/0x1490 [ 1358.695958] ? io_schedule_timeout+0x130/0x130 [ 1358.701734] ? find_held_lock+0x2b/0x80 [ 1358.706825] ? __try_to_del_timer_sync+0x154/0x1f0 [ 1358.713005] schedule+0xd4/0x260 [ 1358.717423] schedule_timeout+0x103/0x260 [ 1358.722719] ? rcu_is_watching+0x15/0xd0 [ 1358.727917] ? hrtimer_nanosleep_restart+0x180/0x180 [ 1358.734285] ? hrtimers_cpu_dying+0x250/0x250 [ 1358.739973] ? ww_mutex_lock+0x160/0x160 [ 1358.745173] msleep+0x1b/0x30 [ 1358.749286] napi_set_threaded+0x376/0x410 [ 1358.754680] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.760175] netif_set_threaded+0x254/0x400 [ 1358.765666] ? sysfs_file_ops+0x140/0x140 [ 1358.770962] threaded_store+0x1bb/0x220 [ 1358.776063] ? tx_maxrate_store+0x4e0/0x4e0 [ 1358.781559] ? sysfs_file_ops+0x140/0x140 [ 1358.786853] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.792149] ? sysfs_kf_write+0x65/0x160 [ 1358.797347] kernfs_fop_write_iter+0x335/0x510 [ 1358.803134] ? kernfs_ops+0xf0/0xf0 [ 1358.807843] new_sync_write+0x308/0x6f0 [ 1358.812945] ? new_sync_read+0x6f0/0x6f0 [ 1358.818144] ? lock_acquire.part.0+0xbc/0x260 [ 1358.823827] ? ksys_write+0xff/0x200 [ 1358.828639] vfs_write+0x62a/0x980 [ 1358.833254] ksys_write+0xff/0x200 [ 1358.837869] ? __ia32_sys_read+0xc0/0xc0 [ 1358.843066] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.848654] ? rcu_is_watching+0x15/0xd0 [ 1358.853853] do_syscall_64+0xf3/0x5e0 [ 1358.858750] ? exc_page_fault+0x9e/0xb0 [ 1358.863849] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.870315] RIP: 0033:0x7f3de85f7c5e [ 1358.875124] RSP: 002b:00007fff46626810 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.884421] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3de85f7c5e [ 1358.893226] RDX: 0000000000000001 RSI: 000055b3c07af760 RDI: 0000000000000007 [ 1358.902037] RBP: 00007fff46626820 R08: 0000000000000000 R09: 0000000000000000 [ 1358.910845] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3de8492e80 [ 1358.919642] R13: 000055b3c07af760 R14: 0000000000000007 R15: 00007f3de8422480 [ 1358.928454] [ 1358.931720] \x0aShowing all locks held in the system: [ 1358.940256] 1 lock held by khungtaskd/185: [ 1358.945648] #0: ffffffffb70ce120 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.957129] 4 locks held by python3/3268: [ 1358.962423] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1358.972212] #1: ff1100013ed63e88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1358.983174] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.994332] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.004426] 4 locks held by python3/5243: [ 1359.009722] #0: ff1100012e1793f8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0xff/0x200 [ 1359.019509] #1: ff110001c5e15088 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x247/0x510 [ 1359.030469] #2: ff11000120aa90f0 (kn->active#127){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.041627] #3: ff11000120b29d00 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.051720] [ 1359.054184] =============================================\x0a