[ 763.394942] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 763.397315] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py [ 763.603743] kselftest: Running tests in drivers/net [ 989.708178] INFO: task python3:5496 blocked for more than 122 seconds. [ 989.716406] Not tainted 7.1.0-rc6-igqq-geedd24e6ca78 #1 [ 989.724278] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.733870] task:python3 state:D stack:25672 pid:5496 tgid:5496 ppid:5495 task_flags:0x400100 flags:0x00080000 [ 989.747076] Call Trace: [ 989.750615] [ 989.753767] __schedule+0xcc3/0x19c0 [ 989.758587] ? io_schedule_timeout+0x140/0x140 [ 989.764371] ? find_held_lock+0x2b/0x80 [ 989.769478] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.775363] schedule+0xd8/0x260 [ 989.779783] schedule_preempt_disabled+0x18/0x30 [ 989.785763] __mutex_lock+0x6af/0x2010 [ 989.790760] ? threaded_store+0x132/0x220 [ 989.796060] ? ww_mutex_lock+0x160/0x160 [ 989.801259] ? selinux_inode_free_security+0x330/0x330 [ 989.807823] ? cap_capable+0x1d6/0x3e0 [ 989.812831] ? threaded_store+0x132/0x220 [ 989.818129] threaded_store+0x132/0x220 [ 989.823235] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.828533] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.833832] ? sysfs_kf_write+0x65/0x160 [ 989.839027] ? sysfs_file_ops+0x140/0x140 [ 989.844324] kernfs_fop_write_iter+0x335/0x510 [ 989.850102] ? kernfs_ops+0xf0/0xf0 [ 989.854813] new_sync_write+0x333/0x750 [ 989.859915] ? __lock_acquire+0x508/0xc10 [ 989.865213] ? new_sync_read+0x740/0x740 [ 989.870416] ? lock_acquire.part.0+0xbc/0x260 [ 989.876101] ? ksys_write+0x116/0x250 [ 989.881018] vfs_write+0x657/0x9b0 [ 989.885635] ksys_write+0x116/0x250 [ 989.890348] ? __ia32_sys_read+0xc0/0xc0 [ 989.895538] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.901129] ? rcu_is_watching+0x15/0xd0 [ 989.906334] do_syscall_64+0xf3/0x650 [ 989.911244] ? trace_hardirqs_off+0xd/0x30 [ 989.916638] ? exc_page_fault+0xda/0xf0 [ 989.921740] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.928199] RIP: 0033:0x7faecd24a64e [ 989.933014] RSP: 002b:00007ffdc61ad3c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.942312] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faecd24a64e [ 989.951120] RDX: 0000000000000001 RSI: 000055c4f5773bf0 RDI: 0000000000000007 [ 989.959922] RBP: 00007ffdc61ad3d0 R08: 0000000000000000 R09: 0000000000000000 [ 989.968730] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faecd0c3390 [ 989.977537] R13: 000055c4f5773bf0 R14: 0000000000000007 R15: 00007faecd059120 [ 989.986360] [ 989.989658] INFO: task python3:5496 is blocked on a mutex likely owned by task python3:3522. [ 989.999932] task:python3 state:D stack:25504 pid:3522 tgid:3522 ppid:3521 task_flags:0x400100 flags:0x00080802 [ 990.013133] Call Trace: [ 990.016674] [ 990.019824] __schedule+0xcc3/0x19c0 [ 990.024637] ? io_schedule_timeout+0x140/0x140 [ 990.030423] ? find_held_lock+0x2b/0x80 [ 990.035526] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.041707] schedule+0xd8/0x260 [ 990.046125] schedule_timeout+0x13f/0x290 [ 990.051423] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.057791] ? trace_contention_end+0xb3/0x1a0 [ 990.063575] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.069266] ? ww_mutex_lock+0x160/0x160 [ 990.074465] msleep+0x1b/0x30 [ 990.078591] napi_set_threaded+0x3a5/0x440 [ 990.083988] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.089474] netif_set_threaded+0x257/0x400 [ 990.094967] threaded_store+0x1b8/0x220 [ 990.100069] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.105357] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.110656] ? sysfs_kf_write+0x65/0x160 [ 990.115858] ? sysfs_file_ops+0x140/0x140 [ 990.121153] kernfs_fop_write_iter+0x335/0x510 [ 990.126940] ? kernfs_ops+0xf0/0xf0 [ 990.131652] new_sync_write+0x333/0x750 [ 990.136745] ? __lock_acquire+0x508/0xc10 [ 990.142044] ? new_sync_read+0x740/0x740 [ 990.147244] ? lock_acquire.part.0+0xbc/0x260 [ 990.152925] ? ksys_write+0x116/0x250 [ 990.157836] vfs_write+0x657/0x9b0 [ 990.162453] ksys_write+0x116/0x250 [ 990.167165] ? __ia32_sys_read+0xc0/0xc0 [ 990.172358] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.177949] ? rcu_is_watching+0x15/0xd0 [ 990.183151] do_syscall_64+0xf3/0x650 [ 990.188059] ? irq_exit_rcu+0x1a/0x30 [ 990.192967] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.199434] RIP: 0033:0x7ff215c6664e [ 990.204243] RSP: 002b:00007ffdedb4fb10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.213538] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff215c6664e [ 990.222346] RDX: 0000000000000001 RSI: 000055a3bdb87bf0 RDI: 0000000000000007 [ 990.231154] RBP: 00007ffdedb4fb20 R08: 0000000000000000 R09: 0000000000000000 [ 990.239963] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff215adf390 [ 990.248771] R13: 000055a3bdb87bf0 R14: 0000000000000007 R15: 00007ff215a75120 [ 990.257588] [ 990.260850] \x0aShowing all locks held in the system: [ 990.269391] 1 lock held by khungtaskd/185: [ 990.274778] #0: ffffffffb92fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.286254] 4 locks held by python3/3522: [ 990.291549] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.301439] #1: ff11000141ff6880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.312401] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.323548] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.333646] 4 locks held by python3/5496: [ 990.338942] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.348828] #1: ff110001a41cee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.359788] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.370947] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.381040] [ 990.383505] =============================================\x0a [ 1112.589467] INFO: task python3:5496 blocked for more than 245 seconds. [ 1112.597616] Not tainted 7.1.0-rc6-igqq-geedd24e6ca78 #1 [ 1112.604868] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.614458] task:python3 state:D stack:25672 pid:5496 tgid:5496 ppid:5495 task_flags:0x400100 flags:0x00080000 [ 1112.627663] Call Trace: [ 1112.631204] [ 1112.634353] __schedule+0xcc3/0x19c0 [ 1112.639169] ? io_schedule_timeout+0x140/0x140 [ 1112.644957] ? find_held_lock+0x2b/0x80 [ 1112.650063] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.655949] schedule+0xd8/0x260 [ 1112.660367] schedule_preempt_disabled+0x18/0x30 [ 1112.666345] __mutex_lock+0x6af/0x2010 [ 1112.671350] ? threaded_store+0x132/0x220 [ 1112.676640] ? ww_mutex_lock+0x160/0x160 [ 1112.681838] ? selinux_inode_free_security+0x330/0x330 [ 1112.688400] ? cap_capable+0x1d6/0x3e0 [ 1112.693414] ? threaded_store+0x132/0x220 [ 1112.698754] threaded_store+0x132/0x220 [ 1112.703858] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.709156] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.714454] ? sysfs_kf_write+0x65/0x160 [ 1112.719653] ? sysfs_file_ops+0x140/0x140 [ 1112.724951] kernfs_fop_write_iter+0x335/0x510 [ 1112.730734] ? kernfs_ops+0xf0/0xf0 [ 1112.735448] new_sync_write+0x333/0x750 [ 1112.740554] ? __lock_acquire+0x508/0xc10 [ 1112.745850] ? new_sync_read+0x740/0x740 [ 1112.751053] ? lock_acquire.part.0+0xbc/0x260 [ 1112.756739] ? ksys_write+0x116/0x250 [ 1112.761649] vfs_write+0x657/0x9b0 [ 1112.766266] ksys_write+0x116/0x250 [ 1112.770978] ? __ia32_sys_read+0xc0/0xc0 [ 1112.776176] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.781767] ? rcu_is_watching+0x15/0xd0 [ 1112.786969] do_syscall_64+0xf3/0x650 [ 1112.791873] ? trace_hardirqs_off+0xd/0x30 [ 1112.797270] ? exc_page_fault+0xda/0xf0 [ 1112.802373] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.808843] RIP: 0033:0x7faecd24a64e [ 1112.813652] RSP: 002b:00007ffdc61ad3c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.822951] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faecd24a64e [ 1112.831761] RDX: 0000000000000001 RSI: 000055c4f5773bf0 RDI: 0000000000000007 [ 1112.840571] RBP: 00007ffdc61ad3d0 R08: 0000000000000000 R09: 0000000000000000 [ 1112.849380] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faecd0c3390 [ 1112.858189] R13: 000055c4f5773bf0 R14: 0000000000000007 R15: 00007faecd059120 [ 1112.867003] [ 1112.870297] INFO: task python3:5496 is blocked on a mutex likely owned by task python3:3522. [ 1112.880570] task:python3 state:D stack:25504 pid:3522 tgid:3522 ppid:3521 task_flags:0x400100 flags:0x00080802 [ 1112.893776] Call Trace: [ 1112.897322] [ 1112.900474] __schedule+0xcc3/0x19c0 [ 1112.905287] ? io_schedule_timeout+0x140/0x140 [ 1112.911073] ? find_held_lock+0x2b/0x80 [ 1112.916175] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.922355] schedule+0xd8/0x260 [ 1112.926773] schedule_timeout+0x13f/0x290 [ 1112.932069] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.938438] ? trace_contention_end+0xb3/0x1a0 [ 1112.944225] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.949919] ? ww_mutex_lock+0x160/0x160 [ 1112.955122] msleep+0x1b/0x30 [ 1112.959248] napi_set_threaded+0x3a5/0x440 [ 1112.964646] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.970141] netif_set_threaded+0x257/0x400 [ 1112.975635] threaded_store+0x1b8/0x220 [ 1112.980740] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.986036] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.991334] ? sysfs_kf_write+0x65/0x160 [ 1112.996525] ? sysfs_file_ops+0x140/0x140 [ 1113.001822] kernfs_fop_write_iter+0x335/0x510 [ 1113.007608] ? kernfs_ops+0xf0/0xf0 [ 1113.012322] new_sync_write+0x333/0x750 [ 1113.017427] ? __lock_acquire+0x508/0xc10 [ 1113.022725] ? new_sync_read+0x740/0x740 [ 1113.027927] ? lock_acquire.part.0+0xbc/0x260 [ 1113.033612] ? ksys_write+0x116/0x250 [ 1113.038525] vfs_write+0x657/0x9b0 [ 1113.043142] ksys_write+0x116/0x250 [ 1113.047853] ? __ia32_sys_read+0xc0/0xc0 [ 1113.053052] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.058642] ? rcu_is_watching+0x15/0xd0 [ 1113.063844] do_syscall_64+0xf3/0x650 [ 1113.068753] ? irq_exit_rcu+0x1a/0x30 [ 1113.073659] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.080125] RIP: 0033:0x7ff215c6664e [ 1113.084933] RSP: 002b:00007ffdedb4fb10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.094228] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff215c6664e [ 1113.103035] RDX: 0000000000000001 RSI: 000055a3bdb87bf0 RDI: 0000000000000007 [ 1113.111845] RBP: 00007ffdedb4fb20 R08: 0000000000000000 R09: 0000000000000000 [ 1113.120653] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff215adf390 [ 1113.129462] R13: 000055a3bdb87bf0 R14: 0000000000000007 R15: 00007ff215a75120 [ 1113.138276] [ 1113.141541] \x0aShowing all locks held in the system: [ 1113.150081] 1 lock held by khungtaskd/185: [ 1113.155476] #0: ffffffffb92fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.166941] 3 locks held by systemd-journal/802: [ 1113.172920] #0: ff1100180e3b74e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1113.182812] #1: ffffffffb9502ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.195628] #2: ff110001a2173200 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1113.207186] 4 locks held by python3/3522: [ 1113.212481] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.222372] #1: ff11000141ff6880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.233336] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.244495] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.254594] 4 locks held by python3/5496: [ 1113.259893] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.269780] #1: ff110001a41cee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.280735] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.291893] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.301981] [ 1113.304447] =============================================\x0a [ 1235.470768] INFO: task python3:5496 blocked for more than 368 seconds. [ 1235.478919] Not tainted 7.1.0-rc6-igqq-geedd24e6ca78 #1 [ 1235.486169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.495757] task:python3 state:D stack:25672 pid:5496 tgid:5496 ppid:5495 task_flags:0x400100 flags:0x00080002 [ 1235.508963] Call Trace: [ 1235.512504] [ 1235.515654] __schedule+0xcc3/0x19c0 [ 1235.520473] ? io_schedule_timeout+0x140/0x140 [ 1235.526261] ? find_held_lock+0x2b/0x80 [ 1235.531366] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.537251] schedule+0xd8/0x260 [ 1235.541669] schedule_preempt_disabled+0x18/0x30 [ 1235.547648] __mutex_lock+0x6af/0x2010 [ 1235.552653] ? threaded_store+0x132/0x220 [ 1235.557942] ? ww_mutex_lock+0x160/0x160 [ 1235.563142] ? selinux_inode_free_security+0x330/0x330 [ 1235.569712] ? cap_capable+0x1d6/0x3e0 [ 1235.574720] ? threaded_store+0x132/0x220 [ 1235.580021] threaded_store+0x132/0x220 [ 1235.585125] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.590423] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.595721] ? sysfs_kf_write+0x65/0x160 [ 1235.600923] ? sysfs_file_ops+0x140/0x140 [ 1235.606220] kernfs_fop_write_iter+0x335/0x510 [ 1235.612007] ? kernfs_ops+0xf0/0xf0 [ 1235.616721] new_sync_write+0x333/0x750 [ 1235.621826] ? __lock_acquire+0x508/0xc10 [ 1235.627123] ? new_sync_read+0x740/0x740 [ 1235.632317] ? lock_acquire.part.0+0xbc/0x260 [ 1235.638004] ? ksys_write+0x116/0x250 [ 1235.642904] vfs_write+0x657/0x9b0 [ 1235.647523] ksys_write+0x116/0x250 [ 1235.652235] ? __ia32_sys_read+0xc0/0xc0 [ 1235.657425] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.663031] ? rcu_is_watching+0x15/0xd0 [ 1235.668232] do_syscall_64+0xf3/0x650 [ 1235.673138] ? trace_hardirqs_off+0xd/0x30 [ 1235.678534] ? exc_page_fault+0xda/0xf0 [ 1235.683638] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.690108] RIP: 0033:0x7faecd24a64e [ 1235.694919] RSP: 002b:00007ffdc61ad3c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.704220] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faecd24a64e [ 1235.713031] RDX: 0000000000000001 RSI: 000055c4f5773bf0 RDI: 0000000000000007 [ 1235.721841] RBP: 00007ffdc61ad3d0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.730651] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faecd0c3390 [ 1235.739461] R13: 000055c4f5773bf0 R14: 0000000000000007 R15: 00007faecd059120 [ 1235.748273] [ 1235.751584] INFO: task python3:5496 is blocked on a mutex likely owned by task python3:3522. [ 1235.761857] task:python3 state:D stack:25504 pid:3522 tgid:3522 ppid:3521 task_flags:0x400100 flags:0x00080802 [ 1235.775061] Call Trace: [ 1235.778603] [ 1235.781752] __schedule+0xcc3/0x19c0 [ 1235.786564] ? io_schedule_timeout+0x140/0x140 [ 1235.792349] ? find_held_lock+0x2b/0x80 [ 1235.797453] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.803630] schedule+0xd8/0x260 [ 1235.808048] schedule_timeout+0x13f/0x290 [ 1235.813345] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.819714] ? trace_contention_end+0xb3/0x1a0 [ 1235.825501] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.831192] ? ww_mutex_lock+0x160/0x160 [ 1235.836392] msleep+0x1b/0x30 [ 1235.840518] napi_set_threaded+0x3a5/0x440 [ 1235.845916] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.851413] netif_set_threaded+0x257/0x400 [ 1235.856907] threaded_store+0x1b8/0x220 [ 1235.862011] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.867308] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.872609] ? sysfs_kf_write+0x65/0x160 [ 1235.877809] ? sysfs_file_ops+0x140/0x140 [ 1235.883097] kernfs_fop_write_iter+0x335/0x510 [ 1235.888883] ? kernfs_ops+0xf0/0xf0 [ 1235.893593] new_sync_write+0x333/0x750 [ 1235.898697] ? __lock_acquire+0x508/0xc10 [ 1235.903987] ? new_sync_read+0x740/0x740 [ 1235.909188] ? lock_acquire.part.0+0xbc/0x260 [ 1235.914874] ? ksys_write+0x116/0x250 [ 1235.919786] vfs_write+0x657/0x9b0 [ 1235.924402] ksys_write+0x116/0x250 [ 1235.929115] ? __ia32_sys_read+0xc0/0xc0 [ 1235.934315] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.939907] ? rcu_is_watching+0x15/0xd0 [ 1235.945108] do_syscall_64+0xf3/0x650 [ 1235.950017] ? irq_exit_rcu+0x1a/0x30 [ 1235.954924] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.961392] RIP: 0033:0x7ff215c6664e [ 1235.966190] RSP: 002b:00007ffdedb4fb10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.975487] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff215c6664e [ 1235.984294] RDX: 0000000000000001 RSI: 000055a3bdb87bf0 RDI: 0000000000000007 [ 1235.993094] RBP: 00007ffdedb4fb20 R08: 0000000000000000 R09: 0000000000000000 [ 1236.001902] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff215adf390 [ 1236.010702] R13: 000055a3bdb87bf0 R14: 0000000000000007 R15: 00007ff215a75120 [ 1236.019517] [ 1236.022785] \x0aShowing all locks held in the system: [ 1236.031326] 1 lock held by khungtaskd/185: [ 1236.036719] #0: ffffffffb92fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.048190] 3 locks held by systemd-journal/802: [ 1236.054169] #0: ff1100180e5374e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1236.064057] #1: ffffffffb9502ce0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1236.076864] #2: ff1100021b9c0700 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1236.088422] 4 locks held by python3/3522: [ 1236.093719] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.103612] #1: ff11000141ff6880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.114578] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.125733] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.135831] 4 locks held by python3/5496: [ 1236.141127] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.151013] #1: ff110001a41cee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.161976] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.173132] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.183221] [ 1236.185687] =============================================\x0a [ 1358.352056] INFO: task python3:5496 blocked for more than 491 seconds. [ 1358.360207] Not tainted 7.1.0-rc6-igqq-geedd24e6ca78 #1 [ 1358.367458] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.377047] task:python3 state:D stack:25672 pid:5496 tgid:5496 ppid:5495 task_flags:0x400100 flags:0x00080002 [ 1358.390245] Call Trace: [ 1358.393787] [ 1358.396937] __schedule+0xcc3/0x19c0 [ 1358.401753] ? io_schedule_timeout+0x140/0x140 [ 1358.407541] ? find_held_lock+0x2b/0x80 [ 1358.412646] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.418524] schedule+0xd8/0x260 [ 1358.422943] schedule_preempt_disabled+0x18/0x30 [ 1358.428922] __mutex_lock+0x6af/0x2010 [ 1358.433927] ? threaded_store+0x132/0x220 [ 1358.439228] ? ww_mutex_lock+0x160/0x160 [ 1358.444428] ? selinux_inode_free_security+0x330/0x330 [ 1358.451000] ? cap_capable+0x1d6/0x3e0 [ 1358.456007] ? threaded_store+0x132/0x220 [ 1358.461301] threaded_store+0x132/0x220 [ 1358.466405] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.471705] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.477006] ? sysfs_kf_write+0x65/0x160 [ 1358.482206] ? sysfs_file_ops+0x140/0x140 [ 1358.487504] kernfs_fop_write_iter+0x335/0x510 [ 1358.493291] ? kernfs_ops+0xf0/0xf0 [ 1358.498003] new_sync_write+0x333/0x750 [ 1358.503109] ? __lock_acquire+0x508/0xc10 [ 1358.508407] ? new_sync_read+0x740/0x740 [ 1358.513611] ? lock_acquire.part.0+0xbc/0x260 [ 1358.519298] ? ksys_write+0x116/0x250 [ 1358.524207] vfs_write+0x657/0x9b0 [ 1358.528819] ksys_write+0x116/0x250 [ 1358.533531] ? __ia32_sys_read+0xc0/0xc0 [ 1358.538728] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.544314] ? rcu_is_watching+0x15/0xd0 [ 1358.549515] do_syscall_64+0xf3/0x650 [ 1358.554419] ? trace_hardirqs_off+0xd/0x30 [ 1358.559815] ? exc_page_fault+0xda/0xf0 [ 1358.564917] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.571383] RIP: 0033:0x7faecd24a64e [ 1358.576193] RSP: 002b:00007ffdc61ad3c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.585492] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faecd24a64e [ 1358.594298] RDX: 0000000000000001 RSI: 000055c4f5773bf0 RDI: 0000000000000007 [ 1358.603104] RBP: 00007ffdc61ad3d0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.611911] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faecd0c3390 [ 1358.620804] R13: 000055c4f5773bf0 R14: 0000000000000007 R15: 00007faecd059120 [ 1358.629618] [ 1358.632921] INFO: task python3:5496 is blocked on a mutex likely owned by task python3:3522. [ 1358.643193] task:python3 state:D stack:25504 pid:3522 tgid:3522 ppid:3521 task_flags:0x400100 flags:0x00080802 [ 1358.656397] Call Trace: [ 1358.659932] [ 1358.663082] __schedule+0xcc3/0x19c0 [ 1358.667892] ? io_schedule_timeout+0x140/0x140 [ 1358.673679] ? find_held_lock+0x2b/0x80 [ 1358.678785] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.684964] schedule+0xd8/0x260 [ 1358.689384] schedule_timeout+0x13f/0x290 [ 1358.694684] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.701054] ? trace_contention_end+0xb3/0x1a0 [ 1358.701060] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.701067] ? ww_mutex_lock+0x160/0x160 [ 1358.701072] msleep+0x1b/0x30 [ 1358.721884] napi_set_threaded+0x3a5/0x440 [ 1358.727282] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.732778] netif_set_threaded+0x257/0x400 [ 1358.738269] threaded_store+0x1b8/0x220 [ 1358.743373] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.748668] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.753966] ? sysfs_kf_write+0x65/0x160 [ 1358.759165] ? sysfs_file_ops+0x140/0x140 [ 1358.764460] kernfs_fop_write_iter+0x335/0x510 [ 1358.770245] ? kernfs_ops+0xf0/0xf0 [ 1358.774955] new_sync_write+0x333/0x750 [ 1358.780057] ? __lock_acquire+0x508/0xc10 [ 1358.785355] ? new_sync_read+0x740/0x740 [ 1358.790554] ? lock_acquire.part.0+0xbc/0x260 [ 1358.796243] ? ksys_write+0x116/0x250 [ 1358.801154] vfs_write+0x657/0x9b0 [ 1358.805769] ksys_write+0x116/0x250 [ 1358.810481] ? __ia32_sys_read+0xc0/0xc0 [ 1358.815679] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.821268] ? rcu_is_watching+0x15/0xd0 [ 1358.826470] do_syscall_64+0xf3/0x650 [ 1358.831378] ? irq_exit_rcu+0x1a/0x30 [ 1358.836288] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.842754] RIP: 0033:0x7ff215c6664e [ 1358.847564] RSP: 002b:00007ffdedb4fb10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.856860] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007ff215c6664e [ 1358.865669] RDX: 0000000000000001 RSI: 000055a3bdb87bf0 RDI: 0000000000000007 [ 1358.874477] RBP: 00007ffdedb4fb20 R08: 0000000000000000 R09: 0000000000000000 [ 1358.883285] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff215adf390 [ 1358.892084] R13: 000055a3bdb87bf0 R14: 0000000000000007 R15: 00007ff215a75120 [ 1358.900888] [ 1358.904159] \x0aShowing all locks held in the system: [ 1358.912698] 1 lock held by khungtaskd/185: [ 1358.918093] #0: ffffffffb92fdea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.929568] 4 locks held by python3/3522: [ 1358.934857] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.944748] #1: ff11000141ff6880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.955711] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.966868] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.976962] 4 locks held by python3/5496: [ 1358.982258] #0: ff1100013140c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.992147] #1: ff110001a41cee80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.003099] #2: ff110001241fab50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.014260] #3: ff11000124279db8 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.024352] 5 locks held by (systemd-user-)/7342: [ 1359.030425] #0: ff1100013153c3e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.040315] #1: ff110001af85a680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.051280] #2: ffffffffb9369c20 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0 [ 1359.061948] #3: ffffffffb91687b0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x650 [ 1359.073591] #4: ffffffffb9369990 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x650 [ 1359.086021] [ 1359.088486] =============================================\x0a