[ 738.662804][ T44] INFO: task kworker/u16:0:12 blocked for more than 122 seconds. [ 738.663401][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 738.663653][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 738.664049][ T44] task:kworker/u16:0 state:D stack:26784 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 738.664522][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 738.664847][ T44] Call Trace: [ 738.665065][ T44] [ 738.665216][ T44] __schedule+0xae2/0x1720 [ 738.665494][ T44] ? io_schedule_timeout+0x130/0x130 [ 738.665773][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.666070][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.666332][ T44] schedule+0xd4/0x260 [ 738.666565][ T44] schedule_preempt_disabled+0x18/0x30 [ 738.666838][ T44] __mutex_lock+0xd76/0x1f20 [ 738.667103][ T44] ? addrconf_verify_work+0x12/0x30 [ 738.667368][ T44] ? ww_mutex_lock+0x160/0x160 [ 738.667643][ T44] ? process_one_work+0xd16/0x1390 [ 738.667919][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.668183][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.668449][ T44] ? addrconf_verify_work+0x12/0x30 [ 738.668720][ T44] addrconf_verify_work+0x12/0x30 [ 738.668993][ T44] process_one_work+0xd57/0x1390 [ 738.669232][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 738.669404][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.669601][ T44] ? assign_work+0x152/0x380 [ 738.669800][ T44] worker_thread+0x4d6/0xd40 [ 738.669982][ T44] ? process_one_work+0x1390/0x1390 [ 738.670184][ T44] kthread+0x355/0x5b0 [ 738.670324][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.670515][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.671093][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.671265][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.671465][ T44] ret_from_fork+0x3fb/0x510 [ 738.671637][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 738.671865][ T44] ? __switch_to+0x53c/0xd00 [ 738.672042][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.672237][ T44] ret_from_fork_asm+0x11/0x20 [ 738.672414][ T44] [ 738.672621][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5295. [ 738.672912][ T44] INFO: task ip:5295 blocked for more than 122 seconds. [ 738.673128][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 738.673284][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 738.673555][ T44] task:ip state:D stack:25072 pid:5295 tgid:5295 ppid:5290 task_flags:0x400100 flags:0x00080000 [ 738.673889][ T44] Call Trace: [ 738.674036][ T44] [ 738.674176][ T44] __schedule+0xae2/0x1720 [ 738.674345][ T44] ? io_schedule_timeout+0x130/0x130 [ 738.674544][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.674713][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.674913][ T44] schedule+0xd4/0x260 [ 738.675062][ T44] schedule_preempt_disabled+0x18/0x30 [ 738.675254][ T44] __mutex_lock+0xd76/0x1f20 [ 738.675425][ T44] ? __folio_put+0x1f5/0x320 [ 738.675624][ T44] ? napi_enable+0x3b/0x80 [ 738.675824][ T44] ? ww_mutex_lock+0x160/0x160 [ 738.675996][ T44] ? dma_map_phys+0x1b1/0x220 [ 738.676253][ T44] ? __asan_memset+0x27/0x50 [ 738.676424][ T44] ? sg_init_table+0x19/0x60 [ 738.676626][ T44] ? try_fill_recv+0x3ec/0x1750 [ 738.676824][ T44] ? napi_enable+0x3b/0x80 [ 738.676996][ T44] napi_enable+0x3b/0x80 [ 738.677163][ T44] __virtnet_rx_resume+0x156/0x290 [ 738.677333][ T44] ? queue_delayed_work_on+0x91/0x100 [ 738.677526][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 738.677703][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 738.677904][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 738.678084][ T44] dev_xdp_install+0x3e4/0x890 [ 738.678280][ T44] ? netif_set_xps_queue+0x40/0x40 [ 738.678475][ T44] dev_xdp_attach+0x50d/0x1310 [ 738.678655][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 738.678865][ T44] ? __bpf_prog_get+0x320/0x320 [ 738.679049][ T44] dev_change_xdp_fd+0x176/0x210 [ 738.679242][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 738.679436][ T44] ? rtnl_newlink_create+0x900/0x900 [ 738.679609][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 738.679840][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 738.680015][ T44] ? validate_chain+0x37b/0xb70 [ 738.680209][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 738.680378][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 738.680576][ T44] ? __lock_acquire+0x577/0xc10 [ 738.680747][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.680945][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 738.681122][ T44] ? lock_acquire+0x10a/0x150 [ 738.681313][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.681491][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.681694][ T44] ? trace_contention_end+0xd6/0x110 [ 738.681891][ T44] ? __mutex_lock+0x285/0x1f20 [ 738.682069][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 738.682265][ T44] ? ww_mutex_lock+0x160/0x160 [ 738.682432][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 738.682629][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 738.682827][ T44] rtnl_newlink+0x75c/0xe90 [ 738.683009][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.683203][ T44] ? rtnl_setlink+0x880/0x880 [ 738.683370][ T44] ? __lock_acquire+0x577/0xc10 [ 738.683567][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.683732][ T44] ? find_held_lock+0x2b/0x80 [ 738.683929][ T44] ? rtnl_setlink+0x880/0x880 [ 738.684179][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.684364][ T44] ? rtnl_setlink+0x880/0x880 [ 738.684562][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 738.684754][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 738.684950][ T44] ? __lock_acquire+0x577/0xc10 [ 738.685124][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.685316][ T44] ? find_held_lock+0x2b/0x80 [ 738.685485][ T44] netlink_rcv_skb+0x123/0x380 [ 738.685683][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 738.685884][ T44] ? netlink_ack+0xcc0/0xcc0 [ 738.686118][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 738.686323][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 738.686491][ T44] netlink_unicast+0x4a3/0x770 [ 738.686700][ T44] ? netlink_attachskb+0x810/0x810 [ 738.686965][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.687222][ T44] ? __lock_acquire+0x577/0xc10 [ 738.687486][ T44] netlink_sendmsg+0x735/0xc60 [ 738.687752][ T44] ? netlink_unicast+0x770/0x770 [ 738.688026][ T44] ____sys_sendmsg+0x419/0x850 [ 738.688288][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 738.688557][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 738.688860][ T44] ? move_addr_to_kernel+0x40/0x40 [ 738.689118][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 738.689380][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 738.689644][ T44] ___sys_sendmsg+0xfd/0x180 [ 738.689911][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 738.690212][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 738.690495][ T44] __sys_sendmsg+0x124/0x1c0 [ 738.690757][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 738.691019][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 738.691281][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 738.691591][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 738.691857][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.692115][ T44] do_syscall_64+0xbd/0xfc0 [ 738.692377][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 738.692680][ T44] RIP: 0033:0x7fd20ecb0c5e [ 738.692955][ T44] RSP: 002b:00007fffb11e8bf0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 738.693311][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fd20ecb0c5e [ 738.693662][ T44] RDX: 0000000000000000 RSI: 00007fffb11e8cb0 RDI: 0000000000000003 [ 738.694016][ T44] RBP: 00007fffb11e8c00 R08: 0000000000000000 R09: 0000000000000000 [ 738.694367][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fffb11e9370 [ 738.694722][ T44] R13: 000000006949e1a8 R14: 0000000000000000 R15: 0000000000499600 [ 738.695080][ T44] [ 738.695307][ T44] INFO: task ip:5295 is blocked on a mutex likely owned by task kworker/2:2:66. [ 738.695677][ T44] task:kworker/2:2 state:D stack:28312 pid:66 tgid:66 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 738.696141][ T44] Workqueue: events refill_work [ 738.696402][ T44] Call Trace: [ 738.696627][ T44] [ 738.696808][ T44] __schedule+0xae2/0x1720 [ 738.697067][ T44] ? usleep_range_state+0xba/0x1b0 [ 738.697325][ T44] ? io_schedule_timeout+0x130/0x130 [ 738.697590][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.697860][ T44] ? usleep_range_state+0xba/0x1b0 [ 738.698114][ T44] schedule+0xd4/0x260 [ 738.698337][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 738.698638][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 738.698906][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 738.699164][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 738.699419][ T44] ? ktime_get+0x15e/0x220 [ 738.699682][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 738.699951][ T44] ? ktime_get+0xa9/0x220 [ 738.700167][ T44] usleep_range_state+0x133/0x1b0 [ 738.700423][ T44] ? schedule_hrtimeout+0x20/0x20 [ 738.700686][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 738.700951][ T44] ? refill_work+0x149/0x210 [ 738.701207][ T44] napi_disable_locked+0x379/0x5f0 [ 738.701465][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 738.701728][ T44] ? register_netdevice_notifier_net+0x40/0x40 [ 738.702047][ T44] napi_disable+0x43/0x80 [ 738.702264][ T44] refill_work+0x116/0x210 [ 738.702524][ T44] process_one_work+0xd57/0x1390 [ 738.702811][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 738.703065][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.703327][ T44] ? assign_work+0x152/0x380 [ 738.703506][ T44] worker_thread+0x4d6/0xd40 [ 738.703716][ T44] ? process_one_work+0x1390/0x1390 [ 738.703912][ T44] kthread+0x355/0x5b0 [ 738.704060][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.704253][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.704418][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.704617][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.704811][ T44] ret_from_fork+0x3fb/0x510 [ 738.704984][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 738.705213][ T44] ? __switch_to+0x53c/0xd00 [ 738.705380][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.705579][ T44] ret_from_fork_asm+0x11/0x20 [ 738.705756][ T44] [ 738.705931][ T44] [ 738.705931][ T44] Showing all locks held in the system: [ 738.706182][ T44] 3 locks held by kworker/u16:0/12: [ 738.706345][ T44] #0: ff110000091a7148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 738.706653][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 738.707106][ T44] #2: ffffffffbde85768 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 738.707384][ T44] 1 lock held by khungtaskd/44: [ 738.707555][ T44] #0: ffffffffbd771040 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 738.707860][ T44] 3 locks held by kworker/2:2/66: [ 738.708030][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 738.708331][ T44] #1: ffa0000000477cc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 738.708666][ T44] #2: ff11000008caac98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 738.708956][ T44] 2 locks held by ip/5295: [ 738.709122][ T44] #0: ffffffffbde85768 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 738.709394][ T44] #1: ff11000008caac98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 738.709676][ T44] [ 738.709812][ T44] ============================================= [ 738.709812][ T44] [ 756.134810][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 5183 ms [ 756.135420][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 5183000 usecs ago [ 761.190790][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 10239 ms [ 761.191232][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 10239000 usecs ago [ 766.310791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 15359 ms [ 766.311299][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 15359000 usecs ago [ 771.430792][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 20479 ms [ 771.431228][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 20479000 usecs ago [ 777.574797][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 26623 ms [ 777.575345][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 26623000 usecs ago [ 783.206804][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 32255 ms [ 783.207407][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 32255000 usecs ago [ 788.326790][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 37375 ms [ 788.327221][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 37375000 usecs ago [ 793.446791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 42495 ms [ 793.447196][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 42495000 usecs ago [ 799.590792][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 48639 ms [ 799.591212][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 48639000 usecs ago [ 805.222790][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 54271 ms [ 805.223211][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 54271000 usecs ago [ 810.342791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 59391 ms [ 810.343203][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 59391000 usecs ago [ 815.462788][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 64511 ms [ 815.463241][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 64511000 usecs ago [ 821.606787][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 70655 ms [ 821.607214][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 70655000 usecs ago [ 827.238790][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 76287 ms [ 827.239215][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 76287000 usecs ago [ 832.358791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 81407 ms [ 832.359171][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 81407000 usecs ago [ 838.502792][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 87551 ms [ 838.503167][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 87551000 usecs ago [ 844.134790][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 93183 ms [ 844.135233][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 93183000 usecs ago [ 849.254789][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 98303 ms [ 849.255201][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 98303000 usecs ago [ 854.374791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 103423 ms [ 854.375281][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 103423000 usecs ago [ 860.518797][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 109567 ms [ 860.519483][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 109567000 usecs ago [ 861.543809][ T44] INFO: task kworker/u16:0:12 blocked for more than 245 seconds. [ 861.544289][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 861.544564][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 861.544981][ T44] task:kworker/u16:0 state:D stack:26784 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 861.545509][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 861.545847][ T44] Call Trace: [ 861.546073][ T44] [ 861.546227][ T44] __schedule+0xae2/0x1720 [ 861.546517][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.546807][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.547079][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.547365][ T44] schedule+0xd4/0x260 [ 861.547596][ T44] schedule_preempt_disabled+0x18/0x30 [ 861.547867][ T44] __mutex_lock+0xd76/0x1f20 [ 861.548135][ T44] ? addrconf_verify_work+0x12/0x30 [ 861.548408][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.548680][ T44] ? process_one_work+0xd16/0x1390 [ 861.548963][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.549238][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.549515][ T44] ? addrconf_verify_work+0x12/0x30 [ 861.549688][ T44] addrconf_verify_work+0x12/0x30 [ 861.549885][ T44] process_one_work+0xd57/0x1390 [ 861.550109][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 861.550275][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.550476][ T44] ? assign_work+0x152/0x380 [ 861.550643][ T44] worker_thread+0x4d6/0xd40 [ 861.550846][ T44] ? process_one_work+0x1390/0x1390 [ 861.551023][ T44] kthread+0x355/0x5b0 [ 861.551246][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.551497][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.551666][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.551864][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.552044][ T44] ret_from_fork+0x3fb/0x510 [ 861.552246][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 861.552472][ T44] ? __switch_to+0x53c/0xd00 [ 861.552637][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.552848][ T44] ret_from_fork_asm+0x11/0x20 [ 861.553117][ T44] [ 861.553376][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5295. [ 861.553761][ T44] INFO: task ip:5295 blocked for more than 245 seconds. [ 861.554057][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 861.554311][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 861.554668][ T44] task:ip state:D stack:25072 pid:5295 tgid:5295 ppid:5290 task_flags:0x400100 flags:0x00080000 [ 861.555138][ T44] Call Trace: [ 861.555360][ T44] [ 861.555504][ T44] __schedule+0xae2/0x1720 [ 861.555798][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.556059][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.556325][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.556589][ T44] schedule+0xd4/0x260 [ 861.556817][ T44] schedule_preempt_disabled+0x18/0x30 [ 861.557071][ T44] __mutex_lock+0xd76/0x1f20 [ 861.557345][ T44] ? __folio_put+0x1f5/0x320 [ 861.557615][ T44] ? napi_enable+0x3b/0x80 [ 861.557890][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.558146][ T44] ? dma_map_phys+0x1b1/0x220 [ 861.558382][ T44] ? __asan_memset+0x27/0x50 [ 861.558550][ T44] ? sg_init_table+0x19/0x60 [ 861.558740][ T44] ? try_fill_recv+0x3ec/0x1750 [ 861.558936][ T44] ? napi_enable+0x3b/0x80 [ 861.559112][ T44] napi_enable+0x3b/0x80 [ 861.559290][ T44] __virtnet_rx_resume+0x156/0x290 [ 861.559455][ T44] ? queue_delayed_work_on+0x91/0x100 [ 861.559644][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 861.559839][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 861.560017][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 861.560210][ T44] dev_xdp_install+0x3e4/0x890 [ 861.560384][ T44] ? netif_set_xps_queue+0x40/0x40 [ 861.560586][ T44] dev_xdp_attach+0x50d/0x1310 [ 861.560754][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 861.560949][ T44] ? __bpf_prog_get+0x320/0x320 [ 861.561166][ T44] dev_change_xdp_fd+0x176/0x210 [ 861.561344][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 861.561542][ T44] ? rtnl_newlink_create+0x900/0x900 [ 861.561708][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 861.561936][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 861.562110][ T44] ? validate_chain+0x37b/0xb70 [ 861.562309][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 861.562473][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 861.562658][ T44] ? __lock_acquire+0x577/0xc10 [ 861.562860][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.563035][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 861.563230][ T44] ? lock_acquire+0x10a/0x150 [ 861.563405][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.563595][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.563761][ T44] ? trace_contention_end+0xd6/0x110 [ 861.563956][ T44] ? __mutex_lock+0x285/0x1f20 [ 861.564129][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 861.564328][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.564496][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 861.564697][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 861.564895][ T44] rtnl_newlink+0x75c/0xe90 [ 861.565077][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.565267][ T44] ? rtnl_setlink+0x880/0x880 [ 861.565465][ T44] ? __lock_acquire+0x577/0xc10 [ 861.565631][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.565827][ T44] ? find_held_lock+0x2b/0x80 [ 861.566004][ T44] ? rtnl_setlink+0x880/0x880 [ 861.566198][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.566375][ T44] ? rtnl_setlink+0x880/0x880 [ 861.566572][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 861.566740][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 861.566935][ T44] ? __lock_acquire+0x577/0xc10 [ 861.567110][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.567309][ T44] ? find_held_lock+0x2b/0x80 [ 861.567483][ T44] netlink_rcv_skb+0x123/0x380 [ 861.567673][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 861.567871][ T44] ? netlink_ack+0xcc0/0xcc0 [ 861.568052][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 861.568240][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 861.568413][ T44] netlink_unicast+0x4a3/0x770 [ 861.568606][ T44] ? netlink_attachskb+0x810/0x810 [ 861.568804][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.568970][ T44] ? __lock_acquire+0x577/0xc10 [ 861.569169][ T44] netlink_sendmsg+0x735/0xc60 [ 861.569346][ T44] ? netlink_unicast+0x770/0x770 [ 861.569542][ T44] ____sys_sendmsg+0x419/0x850 [ 861.569709][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 861.569905][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 861.570187][ T44] ? move_addr_to_kernel+0x40/0x40 [ 861.570429][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 861.570593][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 861.570793][ T44] ___sys_sendmsg+0xfd/0x180 [ 861.570959][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 861.571188][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 861.571373][ T44] __sys_sendmsg+0x124/0x1c0 [ 861.571568][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 861.571755][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.571949][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 861.572174][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 861.572347][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.572542][ T44] do_syscall_64+0xbd/0xfc0 [ 861.572713][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 861.572942][ T44] RIP: 0033:0x7fd20ecb0c5e [ 861.573123][ T44] RSP: 002b:00007fffb11e8bf0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 861.573385][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fd20ecb0c5e [ 861.573629][ T44] RDX: 0000000000000000 RSI: 00007fffb11e8cb0 RDI: 0000000000000003 [ 861.573881][ T44] RBP: 00007fffb11e8c00 R08: 0000000000000000 R09: 0000000000000000 [ 861.574141][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fffb11e9370 [ 861.574391][ T44] R13: 000000006949e1a8 R14: 0000000000000000 R15: 0000000000499600 [ 861.574639][ T44] [ 861.574808][ T44] INFO: task ip:5295 is blocked on a mutex likely owned by task kworker/2:2:66. [ 861.575077][ T44] task:kworker/2:2 state:D stack:28312 pid:66 tgid:66 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 861.575415][ T44] Workqueue: events refill_work [ 861.575580][ T44] Call Trace: [ 861.575736][ T44] [ 861.575872][ T44] __schedule+0xae2/0x1720 [ 861.576050][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.576243][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.576415][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.576610][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.576802][ T44] schedule+0xd4/0x260 [ 861.576940][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 861.577166][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 861.577342][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 861.577536][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.577707][ T44] ? ktime_get+0x15e/0x220 [ 861.577898][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 861.578072][ T44] ? ktime_get+0xa9/0x220 [ 861.578235][ T44] ? irqentry_exit+0xf9/0x4d0 [ 861.578408][ T44] usleep_range_state+0x133/0x1b0 [ 861.578597][ T44] ? schedule_hrtimeout+0x20/0x20 [ 861.578763][ T44] ? __free_object+0x2ad/0x4b0 [ 861.578960][ T44] napi_disable_locked+0x379/0x5f0 [ 861.579221][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 861.579396][ T44] ? register_netdevice_notifier_net+0x40/0x40 [ 861.579619][ T44] napi_disable+0x43/0x80 [ 861.579756][ T44] refill_work+0x116/0x210 [ 861.579953][ T44] process_one_work+0xd57/0x1390 [ 861.580129][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 861.580325][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.580500][ T44] ? assign_work+0x152/0x380 [ 861.580691][ T44] worker_thread+0x4d6/0xd40 [ 861.580888][ T44] ? process_one_work+0x1390/0x1390 [ 861.581063][ T44] kthread+0x355/0x5b0 [ 861.581227][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.581401][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.581590][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.581761][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.581957][ T44] ret_from_fork+0x3fb/0x510 [ 861.582130][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 861.582354][ T44] ? __switch_to+0x53c/0xd00 [ 861.582521][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.582711][ T44] ret_from_fork_asm+0x11/0x20 [ 861.582907][ T44] [ 861.583060][ T44] [ 861.583060][ T44] Showing all locks held in the system: [ 861.583307][ T44] 3 locks held by kworker/u16:0/12: [ 861.583470][ T44] #0: ff110000091a7148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.583768][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.584205][ T44] #2: ffffffffbde85768 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 861.584487][ T44] 1 lock held by khungtaskd/44: [ 861.584645][ T44] #0: ffffffffbd771040 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 861.584952][ T44] 3 locks held by kworker/2:2/66: [ 861.585119][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.585433][ T44] #1: ffa0000000477cc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.585753][ T44] #2: ff11000008caac98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 861.586042][ T44] 2 locks held by ip/5295: [ 861.586203][ T44] #0: ffffffffbde85768 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 861.586479][ T44] #1: ff11000008caac98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 861.586751][ T44] [ 861.586884][ T44] ============================================= [ 861.586884][ T44] [ 866.150791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 115199 ms [ 866.151263][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 115199000 usecs ago [ 871.270796][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 120319 ms [ 871.271383][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 120319000 usecs ago [ 876.390791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 125439 ms [ 876.391210][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 125439000 usecs ago [ 882.534791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 131583 ms [ 882.535207][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 131583000 usecs ago [ 888.166795][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 137215 ms [ 888.167478][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 137215000 usecs ago [ 893.286790][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 142335 ms [ 893.287292][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 142335000 usecs ago [ 898.406791][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 147455 ms [ 898.407440][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 147455000 usecs ago [ 904.550795][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 153599 ms [ 904.551470][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 153599000 usecs ago