[ 887.131613][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 5119 ms [ 887.132312][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 5119000 usecs ago [ 892.251610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 10239 ms [ 892.252006][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 10239000 usecs ago [ 897.371616][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 15359 ms [ 897.372135][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 15359000 usecs ago [ 903.515610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 21503 ms [ 903.515988][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 21503000 usecs ago [ 909.147610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 27135 ms [ 909.147993][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 27135000 usecs ago [ 914.267616][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 32255 ms [ 914.268147][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 32255000 usecs ago [ 919.387613][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 37375 ms [ 919.387986][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 37375000 usecs ago [ 925.531610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 43519 ms [ 925.531994][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 43519000 usecs ago [ 931.163610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 49151 ms [ 931.164010][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 49151000 usecs ago [ 936.283614][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 54271 ms [ 936.284165][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 54271000 usecs ago [ 941.403610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 59391 ms [ 941.403999][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 59391000 usecs ago [ 947.547630][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 65535 ms [ 947.548020][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 65535000 usecs ago [ 953.179618][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 71167 ms [ 953.180160][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 71167000 usecs ago [ 958.299610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 76287 ms [ 958.299993][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 76287000 usecs ago [ 963.419617][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 81407 ms [ 963.420168][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 81407000 usecs ago [ 969.563615][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 87551 ms [ 969.564140][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 87551000 usecs ago [ 975.195615][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 93183 ms [ 975.196157][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 93183000 usecs ago [ 980.315610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 98303 ms [ 980.316006][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 98303000 usecs ago [ 984.412634][ T44] INFO: task kworker/u16:0:12 blocked for more than 122 seconds. [ 984.413092][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 984.413206][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 984.413416][ T44] task:kworker/u16:0 state:D stack:26848 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 984.413698][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 984.413861][ T44] Call Trace: [ 984.413967][ T44] [ 984.414034][ T44] __schedule+0xae2/0x1720 [ 984.414179][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.414297][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.414439][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.414580][ T44] schedule+0xd4/0x260 [ 984.414692][ T44] schedule_preempt_disabled+0x18/0x30 [ 984.414862][ T44] __mutex_lock+0xd76/0x1f20 [ 984.415020][ T44] ? addrconf_verify_work+0x12/0x30 [ 984.415201][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.415394][ T44] ? process_one_work+0xd16/0x1390 [ 984.415575][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.415767][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.415950][ T44] ? addrconf_verify_work+0x12/0x30 [ 984.416134][ T44] addrconf_verify_work+0x12/0x30 [ 984.416307][ T44] process_one_work+0xd57/0x1390 [ 984.416498][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 984.416689][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.416870][ T44] ? assign_work+0x152/0x380 [ 984.417061][ T44] worker_thread+0x4d6/0xd40 [ 984.417242][ T44] ? process_one_work+0x1390/0x1390 [ 984.417424][ T44] kthread+0x355/0x5b0 [ 984.417559][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.417746][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.417930][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.418503][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.418693][ T44] ret_from_fork+0x3fb/0x510 [ 984.418871][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 984.419104][ T44] ? __switch_to+0x53c/0xd00 [ 984.419274][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.419455][ T44] ret_from_fork_asm+0x11/0x20 [ 984.419665][ T44] [ 984.419837][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5976. [ 984.420153][ T44] INFO: task ip:5976 blocked for more than 122 seconds. [ 984.420358][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 984.420517][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 984.420807][ T44] task:ip state:D stack:25520 pid:5976 tgid:5976 ppid:5971 task_flags:0x400100 flags:0x00080000 [ 984.421205][ T44] Call Trace: [ 984.421340][ T44] [ 984.421444][ T44] __schedule+0xae2/0x1720 [ 984.421647][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.421831][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.422026][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.422214][ T44] schedule+0xd4/0x260 [ 984.422361][ T44] schedule_preempt_disabled+0x18/0x30 [ 984.422539][ T44] __mutex_lock+0xd76/0x1f20 [ 984.422730][ T44] ? __folio_put+0x1f5/0x320 [ 984.422914][ T44] ? napi_enable+0x3b/0x80 [ 984.423116][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.423292][ T44] ? dma_map_phys+0x1b1/0x220 [ 984.423478][ T44] ? __asan_memset+0x27/0x50 [ 984.423670][ T44] ? sg_init_table+0x19/0x60 [ 984.423844][ T44] ? try_fill_recv+0x3ec/0x1750 [ 984.424040][ T44] ? napi_enable+0x3b/0x80 [ 984.424212][ T44] napi_enable+0x3b/0x80 [ 984.424356][ T44] __virtnet_rx_resume+0x156/0x290 [ 984.424528][ T44] ? queue_delayed_work_on+0x91/0x100 [ 984.424715][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 984.424892][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 984.425104][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 984.425282][ T44] dev_xdp_install+0x3e4/0x890 [ 984.425471][ T44] ? netif_set_xps_queue+0x40/0x40 [ 984.425668][ T44] dev_xdp_attach+0x50d/0x1310 [ 984.425853][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 984.426051][ T44] ? __bpf_prog_get+0x320/0x320 [ 984.426247][ T44] dev_change_xdp_fd+0x176/0x210 [ 984.426428][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 984.426626][ T44] ? rtnl_newlink_create+0x900/0x900 [ 984.426780][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 984.427016][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 984.427189][ T44] ? validate_chain+0x37b/0xb70 [ 984.427354][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 984.427526][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 984.427725][ T44] ? __lock_acquire+0x577/0xc10 [ 984.427906][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.428095][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 984.428267][ T44] ? lock_acquire+0x10a/0x150 [ 984.428450][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.428637][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.428812][ T44] ? trace_contention_end+0xd6/0x110 [ 984.429011][ T44] ? __mutex_lock+0x285/0x1f20 [ 984.429183][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 984.429366][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.429531][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 984.429715][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 984.429887][ T44] rtnl_newlink+0x75c/0xe90 [ 984.430071][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.430239][ T44] ? rtnl_setlink+0x880/0x880 [ 984.430424][ T44] ? __lock_acquire+0x577/0xc10 [ 984.430608][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.430779][ T44] ? find_held_lock+0x2b/0x80 [ 984.430981][ T44] ? rtnl_setlink+0x880/0x880 [ 984.431154][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.431335][ T44] ? rtnl_setlink+0x880/0x880 [ 984.431508][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 984.431680][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 984.431796][ T44] ? __lock_acquire+0x577/0xc10 [ 984.431925][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.432037][ T44] ? find_held_lock+0x2b/0x80 [ 984.432161][ T44] netlink_rcv_skb+0x123/0x380 [ 984.432276][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 984.432406][ T44] ? netlink_ack+0xcc0/0xcc0 [ 984.432580][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 984.432762][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 984.432941][ T44] netlink_unicast+0x4a3/0x770 [ 984.433129][ T44] ? netlink_attachskb+0x810/0x810 [ 984.433301][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.433476][ T44] ? __lock_acquire+0x577/0xc10 [ 984.433666][ T44] netlink_sendmsg+0x735/0xc60 [ 984.433842][ T44] ? netlink_unicast+0x770/0x770 [ 984.434041][ T44] ____sys_sendmsg+0x419/0x850 [ 984.434214][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 984.434399][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 984.434626][ T44] ? move_addr_to_kernel+0x40/0x40 [ 984.434798][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 984.434980][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 984.435149][ T44] ___sys_sendmsg+0xfd/0x180 [ 984.435338][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 984.435567][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 984.435784][ T44] __sys_sendmsg+0x124/0x1c0 [ 984.435968][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 984.436136][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 984.436309][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 984.436533][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 984.436724][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.436896][ T44] do_syscall_64+0xbd/0xfc0 [ 984.437088][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 984.437307][ T44] RIP: 0033:0x7f06d8c6ac5e [ 984.437486][ T44] RSP: 002b:00007ffc73b4a3a0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 984.437763][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f06d8c6ac5e [ 984.438039][ T44] RDX: 0000000000000000 RSI: 00007ffc73b4a460 RDI: 0000000000000003 [ 984.438313][ T44] RBP: 00007ffc73b4a3b0 R08: 0000000000000000 R09: 0000000000000000 [ 984.438574][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc73b4ab20 [ 984.438845][ T44] R13: 0000000069507917 R14: 0000000000000000 R15: 0000000000499600 [ 984.439134][ T44] [ 984.439276][ T44] INFO: task ip:5976 is blocked on a mutex likely owned by task kworker/3:1:51. [ 984.439561][ T44] task:kworker/3:1 state:D stack:28096 pid:51 tgid:51 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 984.439962][ T44] Workqueue: events refill_work [ 984.440136][ T44] Call Trace: [ 984.440278][ T44] [ 984.440366][ T44] __schedule+0xae2/0x1720 [ 984.440554][ T44] ? usleep_range_state+0xba/0x1b0 [ 984.440741][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.440919][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.441102][ T44] ? usleep_range_state+0xba/0x1b0 [ 984.441273][ T44] schedule+0xd4/0x260 [ 984.441404][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 984.441636][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 984.441809][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 984.441998][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 984.442166][ T44] ? ktime_get+0x15e/0x220 [ 984.442344][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 984.442517][ T44] ? ktime_get+0xa9/0x220 [ 984.442658][ T44] usleep_range_state+0x133/0x1b0 [ 984.442838][ T44] ? schedule_hrtimeout+0x20/0x20 [ 984.443022][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 984.443196][ T44] ? refill_work+0x149/0x210 [ 984.443376][ T44] napi_disable_locked+0x379/0x5f0 [ 984.443550][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 984.443741][ T44] ? register_netdevice_notifier_net+0x40/0x40 [ 984.443967][ T44] napi_disable+0x43/0x80 [ 984.444113][ T44] refill_work+0x116/0x210 [ 984.444296][ T44] process_one_work+0xd57/0x1390 [ 984.444472][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 984.444651][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.444823][ T44] ? assign_work+0x152/0x380 [ 984.445015][ T44] worker_thread+0x4d6/0xd40 [ 984.445190][ T44] ? process_one_work+0x1390/0x1390 [ 984.445374][ T44] kthread+0x355/0x5b0 [ 984.445503][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.445690][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.445868][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.446061][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.446241][ T44] ret_from_fork+0x3fb/0x510 [ 984.446433][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 984.446669][ T44] ? __switch_to+0x53c/0xd00 [ 984.446851][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.447046][ T44] ret_from_fork_asm+0x11/0x20 [ 984.447236][ T44] [ 984.447399][ T44] [ 984.447399][ T44] Showing all locks held in the system: [ 984.447818][ T44] 3 locks held by kworker/u16:0/12: [ 984.448006][ T44] #0: ff1100000912f148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 984.448366][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 984.448798][ T44] #2: ffffffff98085768 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 984.449112][ T44] 1 lock held by khungtaskd/44: [ 984.449282][ T44] #0: ffffffff97971040 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 984.449637][ T44] 3 locks held by kworker/3:1/51: [ 984.449799][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 984.450148][ T44] #1: ffa000000037fcc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 984.450533][ T44] #2: ff11000008b82c98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 984.450864][ T44] 2 locks held by ip/5976: [ 984.451039][ T44] #0: ffffffff98085768 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 984.451358][ T44] #1: ff11000008b82c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 984.451664][ T44] [ 984.451744][ T44] ============================================= [ 984.451744][ T44] [ 985.435610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 103423 ms [ 985.436092][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 103423000 usecs ago [ 991.579611][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 109567 ms [ 991.580039][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 109567000 usecs ago [ 997.211616][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 115199 ms [ 997.212220][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 115199000 usecs ago [ 1002.331610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 120319 ms [ 1002.332058][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 120319000 usecs ago [ 1007.451611][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 125439 ms [ 1007.452039][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 125439000 usecs ago [ 1013.595619][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 131583 ms [ 1013.596112][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 131583000 usecs ago [ 1019.227613][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 137215 ms [ 1019.228222][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 137215000 usecs ago [ 1024.347610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 142335 ms [ 1024.348030][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 142335000 usecs ago [ 1030.491615][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 148479 ms [ 1030.492237][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 148479000 usecs ago [ 1036.123610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 154111 ms [ 1036.124041][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 154111000 usecs ago [ 1041.243610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 159231 ms [ 1041.244206][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 159231000 usecs ago [ 1046.363610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 164351 ms [ 1046.364053][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 164351000 usecs ago [ 1052.507610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 170495 ms [ 1052.508031][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 170495000 usecs ago [ 1058.139610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 176127 ms [ 1058.140040][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 176127000 usecs ago [ 1063.259609][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 181247 ms [ 1063.260033][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 181247000 usecs ago [ 1068.379613][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 186367 ms [ 1068.380224][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 186367000 usecs ago [ 1074.523615][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 192511 ms [ 1074.524228][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 192511000 usecs ago [ 1080.155609][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 198143 ms [ 1080.156034][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 198143000 usecs ago [ 1085.275610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 203263 ms [ 1085.276062][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 203263000 usecs ago [ 1090.395612][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 208383 ms [ 1090.396060][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 208383000 usecs ago [ 1096.539610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 214527 ms [ 1096.540046][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 214527000 usecs ago [ 1102.171613][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 220159 ms [ 1102.172205][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 220159000 usecs ago [ 1107.291610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 225279 ms [ 1107.292035][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 225279000 usecs ago [ 1107.292647][ T44] INFO: task kworker/u16:0:12 blocked for more than 245 seconds. [ 1107.292934][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 1107.293141][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1107.293431][ T44] task:kworker/u16:0 state:D stack:26848 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 1107.293837][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 1107.294079][ T44] Call Trace: [ 1107.294241][ T44] [ 1107.294374][ T44] __schedule+0xae2/0x1720 [ 1107.294622][ T44] ? io_schedule_timeout+0x130/0x130 [ 1107.294851][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.295076][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.295299][ T44] schedule+0xd4/0x260 [ 1107.295466][ T44] schedule_preempt_disabled+0x18/0x30 [ 1107.295689][ T44] __mutex_lock+0xd76/0x1f20 [ 1107.295907][ T44] ? addrconf_verify_work+0x12/0x30 [ 1107.296134][ T44] ? ww_mutex_lock+0x160/0x160 [ 1107.296365][ T44] ? process_one_work+0xd16/0x1390 [ 1107.296589][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.296809][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.297013][ T44] ? addrconf_verify_work+0x12/0x30 [ 1107.297211][ T44] addrconf_verify_work+0x12/0x30 [ 1107.297414][ T44] process_one_work+0xd57/0x1390 [ 1107.297669][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 1107.297898][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 1107.298129][ T44] ? assign_work+0x152/0x380 [ 1107.298339][ T44] worker_thread+0x4d6/0xd40 [ 1107.298551][ T44] ? process_one_work+0x1390/0x1390 [ 1107.298734][ T44] kthread+0x355/0x5b0 [ 1107.298871][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 1107.299039][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.299236][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.299404][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 1107.299614][ T44] ret_from_fork+0x3fb/0x510 [ 1107.299805][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 1107.300074][ T44] ? __switch_to+0x53c/0xd00 [ 1107.300301][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 1107.300531][ T44] ret_from_fork_asm+0x11/0x20 [ 1107.300794][ T44] [ 1107.301016][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5976. [ 1107.301343][ T44] INFO: task ip:5976 blocked for more than 245 seconds. [ 1107.301551][ T44] Not tainted 6.19.0-rc1-virtme #1 [ 1107.301763][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1107.302070][ T44] task:ip state:D stack:25520 pid:5976 tgid:5976 ppid:5971 task_flags:0x400100 flags:0x00080000 [ 1107.302481][ T44] Call Trace: [ 1107.302688][ T44] [ 1107.302832][ T44] __schedule+0xae2/0x1720 [ 1107.303063][ T44] ? io_schedule_timeout+0x130/0x130 [ 1107.303239][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.303403][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.303585][ T44] schedule+0xd4/0x260 [ 1107.303737][ T44] schedule_preempt_disabled+0x18/0x30 [ 1107.303900][ T44] __mutex_lock+0xd76/0x1f20 [ 1107.304064][ T44] ? __folio_put+0x1f5/0x320 [ 1107.304227][ T44] ? napi_enable+0x3b/0x80 [ 1107.304396][ T44] ? ww_mutex_lock+0x160/0x160 [ 1107.304584][ T44] ? dma_map_phys+0x1b1/0x220 [ 1107.304776][ T44] ? __asan_memset+0x27/0x50 [ 1107.304955][ T44] ? sg_init_table+0x19/0x60 [ 1107.305162][ T44] ? try_fill_recv+0x3ec/0x1750 [ 1107.305368][ T44] ? napi_enable+0x3b/0x80 [ 1107.305611][ T44] napi_enable+0x3b/0x80 [ 1107.305780][ T44] __virtnet_rx_resume+0x156/0x290 [ 1107.305961][ T44] ? queue_delayed_work_on+0x91/0x100 [ 1107.306125][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 1107.306297][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 1107.306474][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 1107.306659][ T44] dev_xdp_install+0x3e4/0x890 [ 1107.306823][ T44] ? netif_set_xps_queue+0x40/0x40 [ 1107.306992][ T44] dev_xdp_attach+0x50d/0x1310 [ 1107.307156][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 1107.307319][ T44] ? __bpf_prog_get+0x320/0x320 [ 1107.307506][ T44] dev_change_xdp_fd+0x176/0x210 [ 1107.307692][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 1107.307859][ T44] ? rtnl_newlink_create+0x900/0x900 [ 1107.308022][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 1107.308215][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 1107.308377][ T44] ? validate_chain+0x37b/0xb70 [ 1107.308568][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 1107.308753][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 1107.308920][ T44] ? __lock_acquire+0x577/0xc10 [ 1107.309084][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 1107.309249][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 1107.309414][ T44] ? lock_acquire+0x10a/0x150 [ 1107.309584][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.309759][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.309951][ T44] ? trace_contention_end+0xd6/0x110 [ 1107.310177][ T44] ? __mutex_lock+0x285/0x1f20 [ 1107.310403][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 1107.310650][ T44] ? ww_mutex_lock+0x160/0x160 [ 1107.310823][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 1107.310992][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 1107.311199][ T44] rtnl_newlink+0x75c/0xe90 [ 1107.311413][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.311669][ T44] ? rtnl_setlink+0x880/0x880 [ 1107.311896][ T44] ? __lock_acquire+0x577/0xc10 [ 1107.312094][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 1107.312258][ T44] ? find_held_lock+0x2b/0x80 [ 1107.312430][ T44] ? rtnl_setlink+0x880/0x880 [ 1107.312615][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.312782][ T44] ? rtnl_setlink+0x880/0x880 [ 1107.312958][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 1107.313123][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 1107.313290][ T44] ? __lock_acquire+0x577/0xc10 [ 1107.313471][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 1107.313650][ T44] ? find_held_lock+0x2b/0x80 [ 1107.313816][ T44] netlink_rcv_skb+0x123/0x380 [ 1107.313982][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 1107.314188][ T44] ? netlink_ack+0xcc0/0xcc0 [ 1107.314379][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 1107.314558][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 1107.314742][ T44] netlink_unicast+0x4a3/0x770 [ 1107.314908][ T44] ? netlink_attachskb+0x810/0x810 [ 1107.315070][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 1107.315234][ T44] ? __lock_acquire+0x577/0xc10 [ 1107.315404][ T44] netlink_sendmsg+0x735/0xc60 [ 1107.315587][ T44] ? netlink_unicast+0x770/0x770 [ 1107.315768][ T44] ____sys_sendmsg+0x419/0x850 [ 1107.315934][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 1107.316098][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 1107.316289][ T44] ? move_addr_to_kernel+0x40/0x40 [ 1107.316471][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 1107.316658][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 1107.316832][ T44] ___sys_sendmsg+0xfd/0x180 [ 1107.316996][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 1107.317186][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 1107.317363][ T44] __sys_sendmsg+0x124/0x1c0 [ 1107.317568][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 1107.317746][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 1107.317910][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 1107.318113][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 1107.318277][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.318453][ T44] do_syscall_64+0xbd/0xfc0 [ 1107.318646][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1107.318840][ T44] RIP: 0033:0x7f06d8c6ac5e [ 1107.319010][ T44] RSP: 002b:00007ffc73b4a3a0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 1107.319234][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f06d8c6ac5e [ 1107.319471][ T44] RDX: 0000000000000000 RSI: 00007ffc73b4a460 RDI: 0000000000000003 [ 1107.319713][ T44] RBP: 00007ffc73b4a3b0 R08: 0000000000000000 R09: 0000000000000000 [ 1107.319935][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc73b4ab20 [ 1107.320156][ T44] R13: 0000000069507917 R14: 0000000000000000 R15: 0000000000499600 [ 1107.320392][ T44] [ 1107.320536][ T44] INFO: task ip:5976 is blocked on a mutex likely owned by task kworker/3:1:51. [ 1107.320789][ T44] task:kworker/3:1 state:D stack:28096 pid:51 tgid:51 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 1107.321092][ T44] Workqueue: events refill_work [ 1107.321257][ T44] Call Trace: [ 1107.321397][ T44] [ 1107.321512][ T44] __schedule+0xae2/0x1720 [ 1107.321703][ T44] ? usleep_range_state+0xba/0x1b0 [ 1107.321868][ T44] ? io_schedule_timeout+0x130/0x130 [ 1107.322032][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.322197][ T44] ? usleep_range_state+0xba/0x1b0 [ 1107.322358][ T44] schedule+0xd4/0x260 [ 1107.322513][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 1107.322808][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 1107.323040][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 1107.323264][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 1107.323484][ T44] ? ktime_get+0x15e/0x220 [ 1107.323726][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 1107.323922][ T44] ? ktime_get+0xa9/0x220 [ 1107.324064][ T44] usleep_range_state+0x133/0x1b0 [ 1107.324230][ T44] ? schedule_hrtimeout+0x20/0x20 [ 1107.324398][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 1107.324576][ T44] ? refill_work+0x149/0x210 [ 1107.324759][ T44] napi_disable_locked+0x379/0x5f0 [ 1107.324926][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 1107.325090][ T44] ? register_netdevice_notifier_net+0x40/0x40 [ 1107.325301][ T44] napi_disable+0x43/0x80 [ 1107.325453][ T44] refill_work+0x116/0x210 [ 1107.325641][ T44] process_one_work+0xd57/0x1390 [ 1107.325833][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 1107.326059][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 1107.326289][ T44] ? assign_work+0x152/0x380 [ 1107.326516][ T44] worker_thread+0x4d6/0xd40 [ 1107.326770][ T44] ? process_one_work+0x1390/0x1390 [ 1107.326999][ T44] kthread+0x355/0x5b0 [ 1107.327184][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 1107.327414][ T44] ? __lock_release.isra.0+0x59/0x170 [ 1107.327662][ T44] ? rcu_is_watching+0x15/0xd0 [ 1107.327877][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 1107.328087][ T44] ret_from_fork+0x3fb/0x510 [ 1107.328310][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 1107.328589][ T44] ? __switch_to+0x53c/0xd00 [ 1107.328823][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 1107.329026][ T44] ret_from_fork_asm+0x11/0x20 [ 1107.329236][ T44] [ 1107.329399][ T44] [ 1107.329399][ T44] Showing all locks held in the system: [ 1107.329643][ T44] 3 locks held by kworker/u16:0/12: [ 1107.329802][ T44] #0: ff1100000912f148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 1107.330082][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 1107.330477][ T44] #2: ffffffff98085768 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 1107.330831][ T44] 1 lock held by khungtaskd/44: [ 1107.331044][ T44] #0: ffffffff97971040 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1107.331424][ T44] 3 locks held by kworker/3:1/51: [ 1107.331640][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 1107.332009][ T44] #1: ffa000000037fcc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 1107.332427][ T44] #2: ff11000008b82c98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 1107.332718][ T44] 2 locks held by ip/5976: [ 1107.332874][ T44] #0: ffffffff98085768 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 1107.333117][ T44] #1: ff11000008b82c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 1107.333361][ T44] [ 1107.333484][ T44] ============================================= [ 1107.333484][ T44] [ 1112.411610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 230399 ms [ 1112.412043][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 230399000 usecs ago [ 1118.555614][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 236543 ms [ 1118.556166][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 236543000 usecs ago [ 1124.187610][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 242175 ms [ 1124.188061][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 242175000 usecs ago [ 1129.307613][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 247295 ms [ 1129.308308][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 247295000 usecs ago [ 1134.427609][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 2 timed out 252415 ms [ 1134.428038][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 252415000 usecs ago