[ 265.085737][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5696 ms [ 265.086286][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5696000 usecs ago [ 270.141724][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 10752 ms [ 270.142105][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 10752000 usecs ago [ 275.261723][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 15872 ms [ 275.262094][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 15872000 usecs ago [ 280.381726][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 20992 ms [ 280.382119][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 20992000 usecs ago [ 286.525726][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 27136 ms [ 286.526121][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 27136000 usecs ago [ 292.157721][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 32768 ms [ 292.158104][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 32768000 usecs ago [ 297.277724][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 37888 ms [ 297.278110][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 37888000 usecs ago [ 302.397735][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 43008 ms [ 302.398148][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 43008000 usecs ago [ 308.541922][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 49152 ms [ 308.542303][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 49152000 usecs ago [ 314.173731][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 54784 ms [ 314.174118][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 54784000 usecs ago [ 319.293725][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 59904 ms [ 319.294098][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 59904000 usecs ago [ 324.413724][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 65024 ms [ 324.414123][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 65024000 usecs ago [ 330.557723][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 71168 ms [ 330.558072][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 71168000 usecs ago [ 336.189726][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 76800 ms [ 336.190131][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 76800000 usecs ago [ 341.309733][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 81920 ms [ 341.310179][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 81920000 usecs ago [ 347.453732][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 88064 ms [ 347.454149][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 88064000 usecs ago [ 353.085727][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 93696 ms [ 353.086154][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 93696000 usecs ago [ 358.205731][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 98816 ms [ 358.206187][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 98816000 usecs ago [ 363.325728][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 103936 ms [ 363.326226][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 103936000 usecs ago [ 369.469726][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 110080 ms [ 369.470239][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 110080000 usecs ago [ 369.982778][ T44] INFO: task ip:980 blocked for more than 122 seconds. [ 369.983196][ T44] Not tainted 6.18.0-virtme #1 [ 369.983340][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.983589][ T44] task:ip state:D stack:25456 pid:980 tgid:980 ppid:973 task_flags:0x400100 flags:0x00080000 [ 369.983941][ T44] Call Trace: [ 369.984049][ T44] [ 369.984127][ T44] __schedule+0xae2/0x1720 [ 369.984299][ T44] ? io_schedule_timeout+0x130/0x130 [ 369.984436][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.984584][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.984731][ T44] schedule+0xd4/0x260 [ 369.984832][ T44] schedule_preempt_disabled+0x18/0x30 [ 369.984985][ T44] __mutex_lock+0xd76/0x1f20 [ 369.985122][ T44] ? napi_enable+0x3b/0x80 [ 369.985265][ T44] ? ww_mutex_lock+0x160/0x160 [ 369.985392][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.985533][ T44] ? __asan_memset+0x27/0x50 [ 369.985695][ T44] ? sg_init_table+0x19/0x60 [ 369.985848][ T44] ? try_fill_recv+0x3ec/0x1750 [ 369.985980][ T44] ? napi_enable+0x3b/0x80 [ 369.986117][ T44] napi_enable+0x3b/0x80 [ 369.986224][ T44] __virtnet_rx_resume+0x156/0x290 [ 369.986367][ T44] ? queue_delayed_work_on+0x91/0x100 [ 369.986496][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 369.986627][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 369.986770][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 369.986897][ T44] dev_xdp_install+0x4ba/0x890 [ 369.987026][ T44] ? alloc_netdev_dummy+0x30/0x30 [ 369.987149][ T44] dev_xdp_attach+0x50d/0x1310 [ 369.987280][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 369.987407][ T44] ? mutex_is_locked+0x1c/0x50 [ 369.987818][ T44] ? rtnl_is_locked+0x15/0x20 [ 369.987951][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 369.988085][ T44] ? rtnl_newlink_create+0x900/0x900 [ 369.988210][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 369.988371][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 369.988491][ T44] ? validate_chain+0x30f/0x8e0 [ 369.988627][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 369.988759][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 369.988890][ T44] ? __lock_acquire+0x577/0xc10 [ 369.989023][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 369.989150][ T44] ? rtnl_newlink+0x5f9/0xb40 [ 369.989277][ T44] ? lock_acquire+0x10a/0x150 [ 369.989396][ T44] ? rcu_is_watching+0x15/0xd0 [ 369.989519][ T44] ? rcu_is_watching+0x15/0xd0 [ 369.989637][ T44] ? trace_contention_end+0xd6/0x110 [ 369.989772][ T44] ? __mutex_lock+0x285/0x1f20 [ 369.989897][ T44] ? rtnl_newlink+0x5f9/0xb40 [ 369.990031][ T44] ? ww_mutex_lock+0x160/0x160 [ 369.990150][ T44] ? nla_validate_int_range+0x5b0/0x5b0 [ 369.990280][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 369.990402][ T44] rtnl_newlink+0x649/0xb40 [ 369.990532][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.990655][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 369.990790][ T44] ? __lock_acquire+0x577/0xc10 [ 369.990918][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.991049][ T44] ? find_held_lock+0x2b/0x80 [ 369.991171][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 369.991298][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.991418][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 369.991542][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 369.991662][ T44] ? rtnl_port_fill+0x940/0x940 [ 369.991794][ T44] ? __lock_acquire+0x577/0xc10 [ 369.991918][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.992042][ T44] ? find_held_lock+0x2b/0x80 [ 369.992161][ T44] netlink_rcv_skb+0x123/0x380 [ 369.992288][ T44] ? rtnl_port_fill+0x940/0x940 [ 369.992406][ T44] ? netlink_ack+0xcc0/0xcc0 [ 369.992535][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 369.992658][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 369.992787][ T44] netlink_unicast+0x4a3/0x770 [ 369.992914][ T44] ? netlink_attachskb+0x810/0x810 [ 369.993039][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.993165][ T44] ? __lock_acquire+0x577/0xc10 [ 369.993295][ T44] netlink_sendmsg+0x735/0xc60 [ 369.993420][ T44] ? netlink_unicast+0x770/0x770 [ 369.993549][ T44] ____sys_sendmsg+0x419/0x850 [ 369.993676][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 369.993803][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 369.993966][ T44] ? move_addr_to_kernel+0x40/0x40 [ 369.994096][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 369.994212][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 369.994338][ T44] ___sys_sendmsg+0xfd/0x180 [ 369.994456][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 369.994609][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 369.994749][ T44] __sys_sendmsg+0x124/0x1c0 [ 369.994874][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 369.994999][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 369.995117][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 369.995273][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 369.995397][ T44] ? rcu_is_watching+0x15/0xd0 [ 369.995523][ T44] do_syscall_64+0xbd/0xfc0 [ 369.995642][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 369.995801][ T44] RIP: 0033:0x7fbfc1232c5e [ 369.995936][ T44] RSP: 002b:00007ffddf4a4690 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 369.996129][ T44] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fbfc1232c5e [ 369.996309][ T44] RDX: 0000000000000000 RSI: 00007ffddf4a4750 RDI: 0000000000000003 [ 369.996497][ T44] RBP: 00007ffddf4a46a0 R08: 0000000000000000 R09: 0000000000000000 [ 369.996675][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffddf4a4e10 [ 369.996878][ T44] R13: 000000006933744e R14: 0000000000000000 R15: 0000000000499600 [ 369.997071][ T44] [ 369.997179][ T44] INFO: task ip:980 is blocked on a mutex likely owned by task kworker/3:0:33. [ 369.997385][ T44] task:kworker/3:0 state:D stack:28264 pid:33 tgid:33 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 369.997657][ T44] Workqueue: events refill_work [ 369.997787][ T44] Call Trace: [ 369.997880][ T44] [ 369.997945][ T44] __schedule+0xae2/0x1720 [ 369.998079][ T44] ? usleep_range_state+0xba/0x1b0 [ 369.998197][ T44] ? io_schedule_timeout+0x130/0x130 [ 369.998324][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.998444][ T44] ? usleep_range_state+0xba/0x1b0 [ 369.998568][ T44] schedule+0xd4/0x260 [ 369.998657][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 369.998813][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 369.998943][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 369.999072][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 369.999190][ T44] ? ktime_get+0x15e/0x220 [ 369.999314][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 369.999435][ T44] ? ktime_get+0xa9/0x220 [ 369.999532][ T44] usleep_range_state+0x133/0x1b0 [ 369.999651][ T44] ? schedule_hrtimeout+0x20/0x20 [ 369.999786][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 369.999911][ T44] ? refill_work+0x149/0x210 [ 370.000040][ T44] napi_disable_locked+0x379/0x5f0 [ 370.000162][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 370.000290][ T44] ? netdev_bind_sb_channel_queue+0x470/0x470 [ 370.000441][ T44] napi_disable+0x43/0x80 [ 370.000541][ T44] refill_work+0x116/0x210 [ 370.000667][ T44] process_one_work+0xd57/0x1390 [ 370.000802][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 370.000924][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 370.001053][ T44] ? assign_work+0x152/0x380 [ 370.001173][ T44] worker_thread+0x4d6/0xd40 [ 370.001304][ T44] ? process_one_work+0x1390/0x1390 [ 370.001427][ T44] kthread+0x355/0x5b0 [ 370.001527][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 370.001648][ T44] ? __lock_release.isra.0+0x59/0x170 [ 370.001776][ T44] ? rcu_is_watching+0x15/0xd0 [ 370.001900][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 370.002029][ T44] ret_from_fork+0x3fb/0x510 [ 370.002154][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 370.002310][ T44] ? __switch_to+0x602/0xd00 [ 370.002433][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 370.002559][ T44] ret_from_fork_asm+0x11/0x20 [ 370.002684][ T44] [ 370.002793][ T44] [ 370.002793][ T44] Showing all locks held in the system: [ 370.002970][ T44] 3 locks held by kworker/u16:0/12: [ 370.003095][ T44] #0: ff110000089c6548 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 370.003338][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 370.003626][ T44] #2: ffffffff8ea78aa8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 370.003935][ T44] 3 locks held by kworker/3:0/33: [ 370.004049][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 370.004289][ T44] #1: ffa0000000257cc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 370.004549][ T44] #2: ff110000083b2c98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 370.004761][ T44] 1 lock held by khungtaskd/44: [ 370.004888][ T44] #0: ffffffff8e370ba0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 370.005152][ T44] 2 locks held by ip/980: [ 370.005237][ T44] #0: ffffffff8ea78aa8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x5f9/0xb40 [ 370.005447][ T44] #1: ff110000083b2c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 370.005657][ T44] [ 370.005726][ T44] ============================================= [ 370.005726][ T44] [ 375.101739][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 115712 ms [ 375.102292][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 115712000 usecs ago [ 380.221734][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 120832 ms [ 380.222184][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 120832000 usecs ago [ 385.341732][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 125952 ms [ 385.342153][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 125952000 usecs ago [ 391.485726][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 132096 ms [ 391.486158][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 132096000 usecs ago [ 397.117742][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 137728 ms [ 397.118165][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 137728000 usecs ago [ 402.237743][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 142848 ms [ 402.238188][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 142848000 usecs ago [ 407.357742][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 147968 ms [ 407.358229][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 147968000 usecs ago [ 413.501734][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 154112 ms [ 413.502421][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 154112000 usecs ago [ 419.133733][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 159744 ms [ 419.134330][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 159744000 usecs ago [ 424.253741][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 164864 ms [ 424.254206][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 164864000 usecs ago [ 429.373736][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 169984 ms [ 429.374168][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 169984000 usecs ago [ 435.517734][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 176128 ms [ 435.518335][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 176128000 usecs ago [ 441.149727][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 181760 ms [ 441.150137][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 181760000 usecs ago [ 446.269729][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 186880 ms [ 446.270161][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 186880000 usecs ago [ 451.389728][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 192000 ms [ 451.390143][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 192000000 usecs ago [ 457.533729][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 198144 ms [ 457.534183][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 198144000 usecs ago [ 463.165733][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 203776 ms [ 463.166166][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 203776000 usecs ago [ 468.285746][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 208896 ms [ 468.286181][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 208896000 usecs ago [ 473.405745][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 214016 ms [ 473.406168][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 214016000 usecs ago [ 479.549744][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 220160 ms [ 479.550151][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 220160000 usecs ago [ 485.181745][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 225792 ms [ 485.182316][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 225792000 usecs ago [ 490.301727][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 230912 ms [ 490.302178][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 230912000 usecs ago [ 492.862740][ T44] INFO: task kworker/u16:0:12 blocked for more than 122 seconds. [ 492.863047][ T44] Not tainted 6.18.0-virtme #1 [ 492.863164][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 492.863369][ T44] task:kworker/u16:0 state:D stack:27856 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 492.863662][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 492.863844][ T44] Call Trace: [ 492.863939][ T44] [ 492.864012][ T44] __schedule+0xae2/0x1720 [ 492.864147][ T44] ? io_schedule_timeout+0x130/0x130 [ 492.864272][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.864404][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.864525][ T44] schedule+0xd4/0x260 [ 492.864626][ T44] schedule_preempt_disabled+0x18/0x30 [ 492.864763][ T44] __mutex_lock+0xd76/0x1f20 [ 492.864886][ T44] ? addrconf_verify_work+0x12/0x30 [ 492.865028][ T44] ? ww_mutex_lock+0x160/0x160 [ 492.865152][ T44] ? process_one_work+0xd16/0x1390 [ 492.865282][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.865403][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.865536][ T44] ? addrconf_verify_work+0x12/0x30 [ 492.865658][ T44] addrconf_verify_work+0x12/0x30 [ 492.865789][ T44] process_one_work+0xd57/0x1390 [ 492.865915][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 492.866048][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 492.866171][ T44] ? assign_work+0x152/0x380 [ 492.866302][ T44] worker_thread+0x4d6/0xd40 [ 492.866427][ T44] ? process_one_work+0x1390/0x1390 [ 492.866554][ T44] kthread+0x355/0x5b0 [ 492.866652][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 492.866787][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.866906][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.867037][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 492.867158][ T44] ret_from_fork+0x3fb/0x510 [ 492.867295][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 492.867450][ T44] ? __switch_to+0x602/0xd00 [ 492.867579][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 492.867699][ T44] ret_from_fork_asm+0x11/0x20 [ 492.867843][ T44] [ 492.867967][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:980. [ 492.868195][ T44] INFO: task ip:980 blocked for more than 245 seconds. [ 492.868342][ T44] Not tainted 6.18.0-virtme #1 [ 492.868464][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 492.868670][ T44] task:ip state:D stack:25456 pid:980 tgid:980 ppid:973 task_flags:0x400100 flags:0x00080000 [ 492.868944][ T44] Call Trace: [ 492.869041][ T44] [ 492.869105][ T44] __schedule+0xae2/0x1720 [ 492.869238][ T44] ? io_schedule_timeout+0x130/0x130 [ 492.869361][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.869490][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.869611][ T44] schedule+0xd4/0x260 [ 492.869707][ T44] schedule_preempt_disabled+0x18/0x30 [ 492.869833][ T44] __mutex_lock+0xd76/0x1f20 [ 492.869966][ T44] ? napi_enable+0x3b/0x80 [ 492.870116][ T44] ? ww_mutex_lock+0x160/0x160 [ 492.870246][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.870366][ T44] ? __asan_memset+0x27/0x50 [ 492.870494][ T44] ? sg_init_table+0x19/0x60 [ 492.870618][ T44] ? try_fill_recv+0x3ec/0x1750 [ 492.870753][ T44] ? napi_enable+0x3b/0x80 [ 492.870873][ T44] napi_enable+0x3b/0x80 [ 492.870973][ T44] __virtnet_rx_resume+0x156/0x290 [ 492.871105][ T44] ? queue_delayed_work_on+0x91/0x100 [ 492.871235][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 492.871358][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 492.871490][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 492.871609][ T44] dev_xdp_install+0x4ba/0x890 [ 492.871741][ T44] ? alloc_netdev_dummy+0x30/0x30 [ 492.871867][ T44] dev_xdp_attach+0x50d/0x1310 [ 492.872007][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 492.872128][ T44] ? mutex_is_locked+0x1c/0x50 [ 492.872257][ T44] ? rtnl_is_locked+0x15/0x20 [ 492.872379][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 492.872512][ T44] ? rtnl_newlink_create+0x900/0x900 [ 492.872633][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 492.872797][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 492.872931][ T44] ? validate_chain+0x30f/0x8e0 [ 492.873066][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 492.873184][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 492.873315][ T44] ? __lock_acquire+0x577/0xc10 [ 492.873442][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 492.873570][ T44] ? rtnl_newlink+0x5f9/0xb40 [ 492.873693][ T44] ? lock_acquire+0x10a/0x150 [ 492.873825][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.873945][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.874080][ T44] ? trace_contention_end+0xd6/0x110 [ 492.874202][ T44] ? __mutex_lock+0x285/0x1f20 [ 492.874331][ T44] ? rtnl_newlink+0x5f9/0xb40 [ 492.874452][ T44] ? ww_mutex_lock+0x160/0x160 [ 492.874580][ T44] ? nla_validate_int_range+0x5b0/0x5b0 [ 492.874703][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 492.874841][ T44] rtnl_newlink+0x649/0xb40 [ 492.874964][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.875095][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 492.875216][ T44] ? __lock_acquire+0x577/0xc10 [ 492.875348][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 492.875465][ T44] ? find_held_lock+0x2b/0x80 [ 492.875593][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 492.875725][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.875850][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 492.875990][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 492.876115][ T44] ? rtnl_port_fill+0x940/0x940 [ 492.876245][ T44] ? __lock_acquire+0x577/0xc10 [ 492.876367][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 492.876499][ T44] ? find_held_lock+0x2b/0x80 [ 492.876619][ T44] netlink_rcv_skb+0x123/0x380 [ 492.876758][ T44] ? rtnl_port_fill+0x940/0x940 [ 492.876879][ T44] ? netlink_ack+0xcc0/0xcc0 [ 492.877016][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 492.877142][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 492.877272][ T44] netlink_unicast+0x4a3/0x770 [ 492.877395][ T44] ? netlink_attachskb+0x810/0x810 [ 492.877522][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 492.877645][ T44] ? __lock_acquire+0x577/0xc10 [ 492.877776][ T44] netlink_sendmsg+0x735/0xc60 [ 492.877900][ T44] ? netlink_unicast+0x770/0x770 [ 492.878041][ T44] ____sys_sendmsg+0x419/0x850 [ 492.878173][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 492.878302][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 492.878450][ T44] ? move_addr_to_kernel+0x40/0x40 [ 492.878578][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 492.878703][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 492.878834][ T44] ___sys_sendmsg+0xfd/0x180 [ 492.878953][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 492.879122][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 492.879256][ T44] __sys_sendmsg+0x124/0x1c0 [ 492.879387][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 492.879506][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 492.879638][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 492.879800][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 492.879923][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.880060][ T44] do_syscall_64+0xbd/0xfc0 [ 492.880185][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 492.880345][ T44] RIP: 0033:0x7fbfc1232c5e [ 492.880476][ T44] RSP: 002b:00007ffddf4a4690 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 492.880675][ T44] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fbfc1232c5e [ 492.880869][ T44] RDX: 0000000000000000 RSI: 00007ffddf4a4750 RDI: 0000000000000003 [ 492.881056][ T44] RBP: 00007ffddf4a46a0 R08: 0000000000000000 R09: 0000000000000000 [ 492.881244][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffddf4a4e10 [ 492.881427][ T44] R13: 000000006933744e R14: 0000000000000000 R15: 0000000000499600 [ 492.881628][ T44] [ 492.881736][ T44] INFO: task ip:980 is blocked on a mutex likely owned by task kworker/3:0:33. [ 492.881940][ T44] task:kworker/3:0 state:D stack:28264 pid:33 tgid:33 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 492.882225][ T44] Workqueue: events refill_work [ 492.882346][ T44] Call Trace: [ 492.882448][ T44] [ 492.882511][ T44] __schedule+0xae2/0x1720 [ 492.882633][ T44] ? usleep_range_state+0xba/0x1b0 [ 492.882766][ T44] ? io_schedule_timeout+0x130/0x130 [ 492.882886][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.883020][ T44] ? usleep_range_state+0xba/0x1b0 [ 492.883144][ T44] schedule+0xd4/0x260 [ 492.883245][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 492.883398][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 492.883527][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 492.883650][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 492.883783][ T44] ? ktime_get+0x15e/0x220 [ 492.883905][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 492.884044][ T44] ? ktime_get+0xa9/0x220 [ 492.884137][ T44] usleep_range_state+0x133/0x1b0 [ 492.884264][ T44] ? schedule_hrtimeout+0x20/0x20 [ 492.884384][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 492.884513][ T44] ? refill_work+0x149/0x210 [ 492.884639][ T44] napi_disable_locked+0x379/0x5f0 [ 492.884770][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 492.884901][ T44] ? netdev_bind_sb_channel_queue+0x470/0x470 [ 492.885065][ T44] napi_disable+0x43/0x80 [ 492.885157][ T44] refill_work+0x116/0x210 [ 492.885292][ T44] process_one_work+0xd57/0x1390 [ 492.885423][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 492.885554][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 492.885679][ T44] ? assign_work+0x152/0x380 [ 492.885817][ T44] worker_thread+0x4d6/0xd40 [ 492.885944][ T44] ? process_one_work+0x1390/0x1390 [ 492.886081][ T44] kthread+0x355/0x5b0 [ 492.886172][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 492.886301][ T44] ? __lock_release.isra.0+0x59/0x170 [ 492.886421][ T44] ? rcu_is_watching+0x15/0xd0 [ 492.886552][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 492.886675][ T44] ret_from_fork+0x3fb/0x510 [ 492.886812][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 492.886963][ T44] ? __switch_to+0x602/0xd00 [ 492.887100][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 492.887222][ T44] ret_from_fork_asm+0x11/0x20 [ 492.887358][ T44] [ 492.887456][ T44] [ 492.887456][ T44] Showing all locks held in the system: [ 492.887673][ T44] 3 locks held by kworker/u16:0/12: [ 492.887903][ T44] #0: ff110000089c6548 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 492.888159][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 492.888452][ T44] #2: ffffffff8ea78aa8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 492.888667][ T44] 3 locks held by kworker/3:0/33: [ 492.888791][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 492.889031][ T44] #1: ffa0000000257cc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 492.889296][ T44] #2: ff110000083b2c98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 492.889515][ T44] 1 lock held by khungtaskd/44: [ 492.889630][ T44] #0: ffffffff8e370ba0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 492.889888][ T44] 2 locks held by ip/980: [ 492.889975][ T44] #0: ffffffff8ea78aa8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x5f9/0xb40 [ 492.890192][ T44] #1: ff110000083b2c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 492.890406][ T44] [ 492.890465][ T44] ============================================= [ 492.890465][ T44] [ 495.421742][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 236032 ms [ 495.422186][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 236032000 usecs ago [ 501.565743][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 242176 ms [ 501.566164][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 242176000 usecs ago [ 507.197744][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 247808 ms [ 507.198212][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 247808000 usecs ago [ 512.317743][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 252928 ms [ 512.318224][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 252928000 usecs ago