[ 783.085322] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 783.087742] nipa-hw-worker: [5/43] Retrying drivers/net:napi_threaded.py
[ 783.243168] kselftest: Running tests in drivers/net
[ 989.688091] INFO: task python3:5785 blocked for more than 122 seconds.
[ 989.696331] Not tainted 7.1.0-rc1-aygn-gb2b8d3c2f799 #1
[ 989.704260] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.713848] task:python3 state:D stack:25128 pid:5785 tgid:5785 ppid:5784 task_flags:0x400100 flags:0x00080000
[ 989.727057] Call Trace:
[ 989.730596]
[ 989.733749] __schedule+0xcba/0x1940
[ 989.738569] ? io_schedule_timeout+0x140/0x140
[ 989.744355] ? find_held_lock+0x2b/0x80
[ 989.749463] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.755349] schedule+0xd4/0x260
[ 989.759771] schedule_preempt_disabled+0x18/0x30
[ 989.765752] __mutex_lock+0x6a3/0x2000
[ 989.770760] ? threaded_store+0x132/0x220
[ 989.776062] ? ww_mutex_lock+0x160/0x160
[ 989.781263] ? selinux_inode_free_security+0x330/0x330
[ 989.787833] ? cap_capable+0x1d6/0x3e0
[ 989.792845] ? threaded_store+0x132/0x220
[ 989.798143] threaded_store+0x132/0x220
[ 989.803243] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.808540] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.813837] ? sysfs_kf_write+0x65/0x160
[ 989.819037] ? sysfs_file_ops+0x140/0x140
[ 989.824334] kernfs_fop_write_iter+0x335/0x510
[ 989.830119] ? kernfs_ops+0xf0/0xf0
[ 989.834831] new_sync_write+0x333/0x750
[ 989.839937] ? __lock_acquire+0x508/0xc10
[ 989.845234] ? new_sync_read+0x740/0x740
[ 989.850436] ? lock_acquire.part.0+0xbc/0x260
[ 989.856123] ? ksys_write+0x116/0x250
[ 989.861026] vfs_write+0x657/0x9b0
[ 989.865643] ksys_write+0x116/0x250
[ 989.870357] ? __ia32_sys_read+0xc0/0xc0
[ 989.875555] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.881146] ? rcu_is_watching+0x15/0xd0
[ 989.886346] do_syscall_64+0xf3/0x650
[ 989.891253] ? trace_hardirqs_off+0xd/0x30
[ 989.896650] ? exc_page_fault+0xda/0xf0
[ 989.901753] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.908213] RIP: 0033:0x7fa500d0c22e
[ 989.913025] RSP: 002b:00007ffcf7367db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.922322] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa500d0c22e
[ 989.931123] RDX: 0000000000000001 RSI: 000055d78d1aaa10 RDI: 0000000000000007
[ 989.939933] RBP: 00007ffcf7367dc0 R08: 0000000000000000 R09: 0000000000000000
[ 989.948742] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa500b84390
[ 989.957551] R13: 000055d78d1aaa10 R14: 0000000000000007 R15: 0000000000000003
[ 989.966366]
[ 989.969655] INFO: task python3:5785 is blocked on a mutex likely owned by task python3:3695.
[ 989.979934] task:python3 state:D stack:25672 pid:3695 tgid:3695 ppid:3694 task_flags:0x400100 flags:0x00080802
[ 989.993136] Call Trace:
[ 989.996675]
[ 989.999826] __schedule+0xcba/0x1940
[ 990.004628] ? io_schedule_timeout+0x140/0x140
[ 990.010412] ? find_held_lock+0x2b/0x80
[ 990.015516] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.021693] schedule+0xd4/0x260
[ 990.026112] schedule_timeout+0x13f/0x290
[ 990.031409] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.037778] ? trace_contention_end+0xb3/0x1a0
[ 990.043564] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.049257] ? ww_mutex_lock+0x160/0x160
[ 990.054462] msleep+0x1b/0x30
[ 990.058589] napi_set_threaded+0x3a5/0x440
[ 990.063977] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.069475] netif_set_threaded+0x257/0x400
[ 990.074969] threaded_store+0x1b8/0x220
[ 990.080075] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.085373] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.090673] ? sysfs_kf_write+0x65/0x160
[ 990.095871] ? sysfs_file_ops+0x140/0x140
[ 990.101168] kernfs_fop_write_iter+0x335/0x510
[ 990.106952] ? kernfs_ops+0xf0/0xf0
[ 990.111666] new_sync_write+0x333/0x750
[ 990.116766] ? __lock_acquire+0x508/0xc10
[ 990.122067] ? new_sync_read+0x740/0x740
[ 990.127267] ? lock_acquire.part.0+0xbc/0x260
[ 990.132954] ? ksys_write+0x116/0x250
[ 990.137858] vfs_write+0x657/0x9b0
[ 990.142475] ksys_write+0x116/0x250
[ 990.147188] ? __ia32_sys_read+0xc0/0xc0
[ 990.152386] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.157978] ? rcu_is_watching+0x15/0xd0
[ 990.163177] do_syscall_64+0xf3/0x650
[ 990.168086] ? trace_hardirqs_off+0xd/0x30
[ 990.173481] ? exc_page_fault+0xda/0xf0
[ 990.178583] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.185052] RIP: 0033:0x7f266c63222e
[ 990.189860] RSP: 002b:00007fff2dcd1940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.199157] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f266c63222e
[ 990.207954] RDX: 0000000000000001 RSI: 000055f0571dca10 RDI: 0000000000000007
[ 990.216765] RBP: 00007fff2dcd1950 R08: 0000000000000000 R09: 0000000000000000
[ 990.225572] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f266c4aa390
[ 990.234383] R13: 000055f0571dca10 R14: 0000000000000007 R15: 0000000000000003
[ 990.243198]
[ 990.246458] \x0aShowing all locks held in the system:
[ 990.254996] 1 lock held by khungtaskd/184:
[ 990.260388] #0: ffffffff940fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.271864] 2 locks held by systemd-journal/803:
[ 990.277841] #0: ff1100180e4b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.287733] #1: ffffffff943030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.300558] 4 locks held by python3/3695:
[ 990.305856] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.315746] #1: ff1100015b05a680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.326700] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.337857] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.347957] 4 locks held by python3/5785:
[ 990.353256] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.363144] #1: ff110001b252f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.374105] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.385255] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.395345]
[ 990.397802] =============================================\x0a
[ 1112.568324] INFO: task python3:5785 blocked for more than 245 seconds.
[ 1112.576469] Not tainted 7.1.0-rc1-aygn-gb2b8d3c2f799 #1
[ 1112.583718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.593308] task:python3 state:D stack:25128 pid:5785 tgid:5785 ppid:5784 task_flags:0x400100 flags:0x00080000
[ 1112.606514] Call Trace:
[ 1112.610054]
[ 1112.613204] __schedule+0xcba/0x1940
[ 1112.618020] ? io_schedule_timeout+0x140/0x140
[ 1112.623806] ? find_held_lock+0x2b/0x80
[ 1112.628910] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.634793] schedule+0xd4/0x260
[ 1112.639202] schedule_preempt_disabled+0x18/0x30
[ 1112.645182] __mutex_lock+0x6a3/0x2000
[ 1112.650185] ? threaded_store+0x132/0x220
[ 1112.655484] ? ww_mutex_lock+0x160/0x160
[ 1112.660682] ? selinux_inode_free_security+0x330/0x330
[ 1112.667250] ? cap_capable+0x1d6/0x3e0
[ 1112.672258] ? threaded_store+0x132/0x220
[ 1112.677554] threaded_store+0x132/0x220
[ 1112.682655] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.687950] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.693247] ? sysfs_kf_write+0x65/0x160
[ 1112.698446] ? sysfs_file_ops+0x140/0x140
[ 1112.703742] kernfs_fop_write_iter+0x335/0x510
[ 1112.709524] ? kernfs_ops+0xf0/0xf0
[ 1112.714234] new_sync_write+0x333/0x750
[ 1112.719337] ? __lock_acquire+0x508/0xc10
[ 1112.724633] ? new_sync_read+0x740/0x740
[ 1112.729831] ? lock_acquire.part.0+0xbc/0x260
[ 1112.735517] ? ksys_write+0x116/0x250
[ 1112.740424] vfs_write+0x657/0x9b0
[ 1112.745028] ksys_write+0x116/0x250
[ 1112.749740] ? __ia32_sys_read+0xc0/0xc0
[ 1112.754937] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.760522] ? rcu_is_watching+0x15/0xd0
[ 1112.765723] do_syscall_64+0xf3/0x650
[ 1112.770629] ? trace_hardirqs_off+0xd/0x30
[ 1112.776024] ? exc_page_fault+0xda/0xf0
[ 1112.781125] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.787591] RIP: 0033:0x7fa500d0c22e
[ 1112.792397] RSP: 002b:00007ffcf7367db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.801696] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa500d0c22e
[ 1112.810503] RDX: 0000000000000001 RSI: 000055d78d1aaa10 RDI: 0000000000000007
[ 1112.819310] RBP: 00007ffcf7367dc0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.828116] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa500b84390
[ 1112.836924] R13: 000055d78d1aaa10 R14: 0000000000000007 R15: 0000000000000003
[ 1112.845735]
[ 1112.849015] INFO: task python3:5785 is blocked on a mutex likely owned by task python3:3695.
[ 1112.859287] task:python3 state:D stack:25672 pid:3695 tgid:3695 ppid:3694 task_flags:0x400100 flags:0x00080802
[ 1112.872490] Call Trace:
[ 1112.876027]
[ 1112.879176] __schedule+0xcba/0x1940
[ 1112.883987] ? io_schedule_timeout+0x140/0x140
[ 1112.889773] ? find_held_lock+0x2b/0x80
[ 1112.894866] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.901034] schedule+0xd4/0x260
[ 1112.905450] schedule_timeout+0x13f/0x290
[ 1112.910746] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.917116] ? trace_contention_end+0xb3/0x1a0
[ 1112.922889] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.928577] ? ww_mutex_lock+0x160/0x160
[ 1112.933768] msleep+0x1b/0x30
[ 1112.937882] napi_set_threaded+0x3a5/0x440
[ 1112.943278] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.948763] netif_set_threaded+0x257/0x400
[ 1112.954254] threaded_store+0x1b8/0x220
[ 1112.959356] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.964651] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.969948] ? sysfs_kf_write+0x65/0x160
[ 1112.975145] ? sysfs_file_ops+0x140/0x140
[ 1112.980439] kernfs_fop_write_iter+0x335/0x510
[ 1112.986223] ? kernfs_ops+0xf0/0xf0
[ 1112.990933] new_sync_write+0x333/0x750
[ 1112.996034] ? __lock_acquire+0x508/0xc10
[ 1113.001330] ? new_sync_read+0x740/0x740
[ 1113.006528] ? lock_acquire.part.0+0xbc/0x260
[ 1113.012213] ? ksys_write+0x116/0x250
[ 1113.017120] vfs_write+0x657/0x9b0
[ 1113.021734] ksys_write+0x116/0x250
[ 1113.026443] ? __ia32_sys_read+0xc0/0xc0
[ 1113.031642] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.037229] ? rcu_is_watching+0x15/0xd0
[ 1113.042429] do_syscall_64+0xf3/0x650
[ 1113.047336] ? trace_hardirqs_off+0xd/0x30
[ 1113.052727] ? exc_page_fault+0xda/0xf0
[ 1113.057827] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.064293] RIP: 0033:0x7f266c63222e
[ 1113.069098] RSP: 002b:00007fff2dcd1940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.078395] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f266c63222e
[ 1113.087190] RDX: 0000000000000001 RSI: 000055f0571dca10 RDI: 0000000000000007
[ 1113.095999] RBP: 00007fff2dcd1950 R08: 0000000000000000 R09: 0000000000000000
[ 1113.104804] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f266c4aa390
[ 1113.113612] R13: 000055f0571dca10 R14: 0000000000000007 R15: 0000000000000003
[ 1113.122426]
[ 1113.125687] \x0aShowing all locks held in the system:
[ 1113.134225] 1 lock held by khungtaskd/184:
[ 1113.139616] #0: ffffffff940fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.151097] 4 locks held by python3/3695:
[ 1113.156393] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.166284] #1: ff1100015b05a680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.177243] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.188398] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.198489] 4 locks held by python3/5785:
[ 1113.203784] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.213673] #1: ff110001b252f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.224634] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.235792] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.245882]
[ 1113.248344] =============================================\x0a
[ 1235.449572] INFO: task python3:5785 blocked for more than 368 seconds.
[ 1235.457724] Not tainted 7.1.0-rc1-aygn-gb2b8d3c2f799 #1
[ 1235.464974] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.474563] task:python3 state:D stack:25128 pid:5785 tgid:5785 ppid:5784 task_flags:0x400100 flags:0x00080002
[ 1235.487761] Call Trace:
[ 1235.491300]
[ 1235.494450] __schedule+0xcba/0x1940
[ 1235.499266] ? io_schedule_timeout+0x140/0x140
[ 1235.505052] ? find_held_lock+0x2b/0x80
[ 1235.510156] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.516039] schedule+0xd4/0x260
[ 1235.520449] schedule_preempt_disabled+0x18/0x30
[ 1235.526429] __mutex_lock+0x6a3/0x2000
[ 1235.531423] ? threaded_store+0x132/0x220
[ 1235.536719] ? ww_mutex_lock+0x160/0x160
[ 1235.541918] ? selinux_inode_free_security+0x330/0x330
[ 1235.548477] ? cap_capable+0x1d6/0x3e0
[ 1235.553487] ? threaded_store+0x132/0x220
[ 1235.558784] threaded_store+0x132/0x220
[ 1235.563883] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.569179] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.574465] ? sysfs_kf_write+0x65/0x160
[ 1235.579662] ? sysfs_file_ops+0x140/0x140
[ 1235.584958] kernfs_fop_write_iter+0x335/0x510
[ 1235.590740] ? kernfs_ops+0xf0/0xf0
[ 1235.595451] new_sync_write+0x333/0x750
[ 1235.600546] ? __lock_acquire+0x508/0xc10
[ 1235.605832] ? new_sync_read+0x740/0x740
[ 1235.611034] ? lock_acquire.part.0+0xbc/0x260
[ 1235.616720] ? ksys_write+0x116/0x250
[ 1235.621630] vfs_write+0x657/0x9b0
[ 1235.626245] ksys_write+0x116/0x250
[ 1235.630957] ? __ia32_sys_read+0xc0/0xc0
[ 1235.636154] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.641739] ? rcu_is_watching+0x15/0xd0
[ 1235.646940] do_syscall_64+0xf3/0x650
[ 1235.651846] ? trace_hardirqs_off+0xd/0x30
[ 1235.657237] ? exc_page_fault+0xda/0xf0
[ 1235.662339] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.668805] RIP: 0033:0x7fa500d0c22e
[ 1235.673606] RSP: 002b:00007ffcf7367db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.682893] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa500d0c22e
[ 1235.691700] RDX: 0000000000000001 RSI: 000055d78d1aaa10 RDI: 0000000000000007
[ 1235.700506] RBP: 00007ffcf7367dc0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.709312] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa500b84390
[ 1235.718122] R13: 000055d78d1aaa10 R14: 0000000000000007 R15: 0000000000000003
[ 1235.726934]
[ 1235.730231] INFO: task python3:5785 is blocked on a mutex likely owned by task python3:3695.
[ 1235.740504] task:python3 state:D stack:25672 pid:3695 tgid:3695 ppid:3694 task_flags:0x400100 flags:0x00080802
[ 1235.753706] Call Trace:
[ 1235.757242]
[ 1235.760397] __schedule+0xcba/0x1940
[ 1235.765206] ? io_schedule_timeout+0x140/0x140
[ 1235.770992] ? find_held_lock+0x2b/0x80
[ 1235.776092] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.782271] schedule+0xd4/0x260
[ 1235.786690] schedule_timeout+0x13f/0x290
[ 1235.791986] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.798356] ? trace_contention_end+0xb3/0x1a0
[ 1235.804129] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.809817] ? ww_mutex_lock+0x160/0x160
[ 1235.815014] msleep+0x1b/0x30
[ 1235.819136] napi_set_threaded+0x3a5/0x440
[ 1235.824531] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.830025] netif_set_threaded+0x257/0x400
[ 1235.835515] threaded_store+0x1b8/0x220
[ 1235.840617] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.845916] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.851210] ? sysfs_kf_write+0x65/0x160
[ 1235.856407] ? sysfs_file_ops+0x140/0x140
[ 1235.861704] kernfs_fop_write_iter+0x335/0x510
[ 1235.867488] ? kernfs_ops+0xf0/0xf0
[ 1235.872198] new_sync_write+0x333/0x750
[ 1235.877301] ? __lock_acquire+0x508/0xc10
[ 1235.882597] ? new_sync_read+0x740/0x740
[ 1235.887794] ? lock_acquire.part.0+0xbc/0x260
[ 1235.893479] ? ksys_write+0x116/0x250
[ 1235.898396] vfs_write+0x657/0x9b0
[ 1235.903008] ksys_write+0x116/0x250
[ 1235.907721] ? __ia32_sys_read+0xc0/0xc0
[ 1235.912921] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.918500] ? rcu_is_watching+0x15/0xd0
[ 1235.923700] do_syscall_64+0xf3/0x650
[ 1235.928606] ? trace_hardirqs_off+0xd/0x30
[ 1235.933996] ? exc_page_fault+0xda/0xf0
[ 1235.939095] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.945561] RIP: 0033:0x7f266c63222e
[ 1235.950367] RSP: 002b:00007fff2dcd1940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.959662] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f266c63222e
[ 1235.968469] RDX: 0000000000000001 RSI: 000055f0571dca10 RDI: 0000000000000007
[ 1235.977268] RBP: 00007fff2dcd1950 R08: 0000000000000000 R09: 0000000000000000
[ 1235.986074] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f266c4aa390
[ 1235.994882] R13: 000055f0571dca10 R14: 0000000000000007 R15: 0000000000000003
[ 1236.003694]
[ 1236.006957] \x0aShowing all locks held in the system:
[ 1236.015499] 1 lock held by khungtaskd/184:
[ 1236.020882] #0: ffffffff940fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.032353] 2 locks held by systemd-journal/803:
[ 1236.038329] #0: ff1100180e7372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1236.048217] #1: ffffffff943030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: dput.part.0+0x2fd/0x410
[ 1236.060457] 4 locks held by python3/3695:
[ 1236.065749] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.075638] #1: ff1100015b05a680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.086598] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.097754] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.107852] 4 locks held by python3/5785:
[ 1236.113150] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.123035] #1: ff110001b252f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.133993] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.145146] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.155237]
[ 1236.157702] =============================================\x0a
[ 1358.331781] INFO: task python3:5785 blocked for more than 491 seconds.
[ 1358.339931] Not tainted 7.1.0-rc1-aygn-gb2b8d3c2f799 #1
[ 1358.347182] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.356770] task:python3 state:D stack:25128 pid:5785 tgid:5785 ppid:5784 task_flags:0x400100 flags:0x00080002
[ 1358.369977] Call Trace:
[ 1358.373518]
[ 1358.376669] __schedule+0xcba/0x1940
[ 1358.381487] ? io_schedule_timeout+0x140/0x140
[ 1358.387273] ? find_held_lock+0x2b/0x80
[ 1358.392377] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.398261] schedule+0xd4/0x260
[ 1358.402680] schedule_preempt_disabled+0x18/0x30
[ 1358.408660] __mutex_lock+0x6a3/0x2000
[ 1358.413662] ? threaded_store+0x132/0x220
[ 1358.418959] ? ww_mutex_lock+0x160/0x160
[ 1358.424158] ? selinux_inode_free_security+0x330/0x330
[ 1358.430725] ? cap_capable+0x1d6/0x3e0
[ 1358.435733] ? threaded_store+0x132/0x220
[ 1358.441029] threaded_store+0x132/0x220
[ 1358.446129] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.451425] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.456723] ? sysfs_kf_write+0x65/0x160
[ 1358.461925] ? sysfs_file_ops+0x140/0x140
[ 1358.467219] kernfs_fop_write_iter+0x335/0x510
[ 1358.473004] ? kernfs_ops+0xf0/0xf0
[ 1358.477704] new_sync_write+0x333/0x750
[ 1358.482809] ? __lock_acquire+0x508/0xc10
[ 1358.488103] ? new_sync_read+0x740/0x740
[ 1358.493305] ? lock_acquire.part.0+0xbc/0x260
[ 1358.498989] ? ksys_write+0x116/0x250
[ 1358.503898] vfs_write+0x657/0x9b0
[ 1358.508514] ksys_write+0x116/0x250
[ 1358.513226] ? __ia32_sys_read+0xc0/0xc0
[ 1358.518422] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.524012] ? rcu_is_watching+0x15/0xd0
[ 1358.529214] do_syscall_64+0xf3/0x650
[ 1358.534118] ? trace_hardirqs_off+0xd/0x30
[ 1358.539513] ? exc_page_fault+0xda/0xf0
[ 1358.544616] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.551086] RIP: 0033:0x7fa500d0c22e
[ 1358.555896] RSP: 002b:00007ffcf7367db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.565184] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa500d0c22e
[ 1358.574065] RDX: 0000000000000001 RSI: 000055d78d1aaa10 RDI: 0000000000000007
[ 1358.582937] RBP: 00007ffcf7367dc0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.591744] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa500b84390
[ 1358.600552] R13: 000055d78d1aaa10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.609368]
[ 1358.612656] INFO: task python3:5785 is blocked on a mutex likely owned by task python3:3695.
[ 1358.622926] task:python3 state:D stack:25672 pid:3695 tgid:3695 ppid:3694 task_flags:0x400100 flags:0x00080802
[ 1358.636127] Call Trace:
[ 1358.639665]
[ 1358.642818] __schedule+0xcba/0x1940
[ 1358.647634] ? io_schedule_timeout+0x140/0x140
[ 1358.653421] ? find_held_lock+0x2b/0x80
[ 1358.658524] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.664702] schedule+0xd4/0x260
[ 1358.669123] schedule_timeout+0x13f/0x290
[ 1358.674421] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.680784] ? trace_contention_end+0xb3/0x1a0
[ 1358.686567] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.692258] ? ww_mutex_lock+0x160/0x160
[ 1358.697459] msleep+0x1b/0x30
[ 1358.701585] napi_set_threaded+0x3a5/0x440
[ 1358.706982] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.712478] netif_set_threaded+0x257/0x400
[ 1358.717975] threaded_store+0x1b8/0x220
[ 1358.723077] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.728371] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.733668] ? sysfs_kf_write+0x65/0x160
[ 1358.738866] ? sysfs_file_ops+0x140/0x140
[ 1358.744161] kernfs_fop_write_iter+0x335/0x510
[ 1358.749944] ? kernfs_ops+0xf0/0xf0
[ 1358.754657] new_sync_write+0x333/0x750
[ 1358.759759] ? __lock_acquire+0x508/0xc10
[ 1358.765058] ? new_sync_read+0x740/0x740
[ 1358.770260] ? lock_acquire.part.0+0xbc/0x260
[ 1358.775949] ? ksys_write+0x116/0x250
[ 1358.780861] vfs_write+0x657/0x9b0
[ 1358.785479] ksys_write+0x116/0x250
[ 1358.790195] ? __ia32_sys_read+0xc0/0xc0
[ 1358.795397] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.800985] ? rcu_is_watching+0x15/0xd0
[ 1358.806187] do_syscall_64+0xf3/0x650
[ 1358.811093] ? trace_hardirqs_off+0xd/0x30
[ 1358.816488] ? exc_page_fault+0xda/0xf0
[ 1358.821592] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.821597] RIP: 0033:0x7f266c63222e
[ 1358.821599] RSP: 002b:00007fff2dcd1940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.821602] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f266c63222e
[ 1358.821604] RDX: 0000000000000001 RSI: 000055f0571dca10 RDI: 0000000000000007
[ 1358.821605] RBP: 00007fff2dcd1950 R08: 0000000000000000 R09: 0000000000000000
[ 1358.821607] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f266c4aa390
[ 1358.821608] R13: 000055f0571dca10 R14: 0000000000000007 R15: 0000000000000003
[ 1358.821615]
[ 1358.889510] \x0aShowing all locks held in the system:
[ 1358.898052] 1 lock held by khungtaskd/184:
[ 1358.903445] #0: ffffffff940fdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.914917] 2 locks held by systemd-journal/803:
[ 1358.920898] #0: ff1100180e4372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 1358.930789] #1: ffffffff943030a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.943616] 4 locks held by python3/3695:
[ 1358.948912] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.958802] #1: ff1100015b05a680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.969755] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.980912] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.991005] 4 locks held by python3/5785:
[ 1358.996302] #0: ff1100012f7213e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.006191] #1: ff110001b252f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.017146] #2: ff11000111c80b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.028303] #3: ff11000111d01d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.038386] 5 locks held by (systemd-user-)/7631:
[ 1359.044460] #0: ff1100012f8113e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.054348] #1: ff11000106ddc480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.065309] #2: ffffffff94169c20 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0
[ 1359.075981] #3: ffffffff93f68970 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x6b0
[ 1359.087630] #4: ffffffff94169990 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x6b0
[ 1359.100057]
[ 1359.102523] =============================================\x0a