[ 760.092724][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 5056 ms [ 760.093398][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 5056000 usecs ago [ 765.276724][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 10240 ms [ 765.277208][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 10240000 usecs ago [ 770.396724][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 15360 ms [ 770.397199][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 15360000 usecs ago [ 776.539723][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 21503 ms [ 776.540195][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 21503000 usecs ago [ 782.172716][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 27135 ms [ 782.173111][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 27136000 usecs ago [ 787.292723][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 32255 ms [ 787.293088][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 32256000 usecs ago [ 792.411717][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 37375 ms [ 792.412070][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 37375000 usecs ago [ 798.556720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 43519 ms [ 798.557102][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 43520000 usecs ago [ 804.188720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 49151 ms [ 804.189084][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 49152000 usecs ago [ 809.308721][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 54271 ms [ 809.309138][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 54272000 usecs ago [ 814.428720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 59392 ms [ 814.429096][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 59392000 usecs ago [ 820.571719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 65535 ms [ 820.572106][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 65535000 usecs ago [ 826.204718][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 71168 ms [ 826.205135][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 71168000 usecs ago [ 831.324720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 76288 ms [ 831.325138][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 76288000 usecs ago [ 836.444719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 81408 ms [ 836.445136][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 81408000 usecs ago [ 842.588720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 87552 ms [ 842.589082][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 87552000 usecs ago [ 848.220720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 93184 ms [ 848.221096][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 93184000 usecs ago [ 853.339720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 98303 ms [ 853.340134][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 98303000 usecs ago [ 859.484720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 104448 ms [ 859.485159][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 104448000 usecs ago [ 861.531803][ T44] INFO: task ip:5902 blocked for more than 122 seconds. [ 861.532249][ T44] Not tainted 6.18.0-rc7-virtme #1 [ 861.532392][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 861.532636][ T44] task:ip state:D stack:25496 pid:5902 tgid:5902 ppid:5897 task_flags:0x400100 flags:0x00080001 [ 861.532986][ T44] Call Trace: [ 861.533138][ T44] [ 861.533213][ T44] __schedule+0x88d/0x1b90 [ 861.533361][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.533510][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.533648][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.533795][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.533931][ T44] schedule+0xd4/0x260 [ 861.534054][ T44] schedule_preempt_disabled+0x18/0x30 [ 861.534189][ T44] __mutex_lock+0x9ae/0x1500 [ 861.534320][ T44] ? napi_enable+0x3b/0x80 [ 861.534450][ T44] ? virtqueue_add_inbuf_ctx+0x1850/0x1850 [ 861.534614][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.534749][ T44] ? __asan_memset+0x27/0x50 [ 861.534881][ T44] ? sg_init_table+0x19/0x60 [ 861.535025][ T44] ? try_fill_recv+0x3ec/0x1750 [ 861.535164][ T44] ? napi_enable+0x3b/0x80 [ 861.535295][ T44] napi_enable+0x3b/0x80 [ 861.535395][ T44] __virtnet_rx_resume+0x156/0x290 [ 861.535523][ T44] ? queue_delayed_work_on+0x91/0x100 [ 861.535648][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 861.535781][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 861.535913][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 861.536038][ T44] dev_xdp_install+0x3e4/0x8e0 [ 861.536161][ T44] ? netif_skb_check_for_xdp+0x600/0x600 [ 861.536281][ T44] dev_xdp_attach+0x50d/0x1320 [ 861.536760][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 861.536877][ T44] ? __bpf_prog_get+0x320/0x320 [ 861.536999][ T44] dev_change_xdp_fd+0x182/0x240 [ 861.537132][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 861.537258][ T44] ? rtnl_newlink_create+0x900/0x900 [ 861.537375][ T44] ? rcu_lockdep_current_cpu_online+0x3e/0x1b0 [ 861.537521][ T44] ? rcu_read_lock_any_held+0x41/0xa0 [ 861.537641][ T44] ? validate_chain+0x287/0x5f0 [ 861.537767][ T44] ? rcu_read_lock_any_held+0x6b/0xa0 [ 861.537883][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 861.538012][ T44] ? __lock_acquire+0x34c/0x740 [ 861.538141][ T44] ? trace_lock_acquire+0x88/0xd0 [ 861.538263][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 861.538380][ T44] ? lock_acquire+0x31/0x80 [ 861.538496][ T44] ? rtnl_newlink+0x5df/0xb20 [ 861.538611][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.538734][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.538851][ T44] ? trace_contention_end+0xd6/0x140 [ 861.538981][ T44] ? rtnl_newlink+0x5df/0xb20 [ 861.539106][ T44] ? __mutex_lock+0x32e/0x1500 [ 861.539228][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.539345][ T44] ? nla_validate_int_range+0x5b0/0x5b0 [ 861.539464][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 861.539584][ T44] rtnl_newlink+0x637/0xb20 [ 861.539712][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 861.539831][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.539950][ T44] ? find_held_lock+0x2b/0x80 [ 861.540072][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 861.540194][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.540311][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 861.540426][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 861.540542][ T44] ? rtnl_port_fill+0x940/0x940 [ 861.540659][ T44] ? __lock_acquire+0x34c/0x740 [ 861.540783][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.540901][ T44] ? find_held_lock+0x2b/0x80 [ 861.541021][ T44] netlink_rcv_skb+0x123/0x380 [ 861.541140][ T44] ? rtnl_port_fill+0x940/0x940 [ 861.541257][ T44] ? netlink_ack+0xcc0/0xcc0 [ 861.541376][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 861.541495][ T44] netlink_unicast+0x4a3/0x770 [ 861.541612][ T44] ? netlink_attachskb+0x810/0x810 [ 861.541740][ T44] ? __lock_acquire+0x34c/0x740 [ 861.541861][ T44] netlink_sendmsg+0x735/0xc60 [ 861.541979][ T44] ? netlink_unicast+0x770/0x770 [ 861.542109][ T44] ? __might_fault+0x97/0x140 [ 861.542229][ T44] ____sys_sendmsg+0x419/0x850 [ 861.542351][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 861.542468][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 861.542615][ T44] ? move_addr_to_kernel+0x40/0x40 [ 861.542749][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 861.542867][ T44] ? validate_chain+0x287/0x5f0 [ 861.542985][ T44] ___sys_sendmsg+0xfd/0x180 [ 861.543107][ T44] ? __call_rcu_common.constprop.0+0xa5/0x680 [ 861.543256][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 861.543392][ T44] __sys_sendmsg+0x124/0x1c0 [ 861.543512][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 861.543633][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.543769][ T44] ? __call_rcu_common.constprop.0+0x2fa/0x680 [ 861.543919][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 861.544042][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.544163][ T44] do_syscall_64+0xbb/0xfd0 [ 861.544284][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 861.544435][ T44] RIP: 0033:0x7f7b2933cc5e [ 861.544569][ T44] RSP: 002b:00007ffc238701f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 861.544789][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f7b2933cc5e [ 861.544966][ T44] RDX: 0000000000000000 RSI: 00007ffc238702b0 RDI: 0000000000000003 [ 861.545150][ T44] RBP: 00007ffc23870200 R08: 0000000000000000 R09: 0000000000000000 [ 861.545333][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc23870970 [ 861.545508][ T44] R13: 00000000692a3dd6 R14: 0000000000000000 R15: 0000000000499600 [ 861.545692][ T44] [ 861.545800][ T44] INFO: task ip:5902 is blocked on a mutex likely owned by task kworker/2:2:62. [ 861.546006][ T44] task:kworker/2:2 state:D stack:28200 pid:62 tgid:62 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 861.546285][ T44] Workqueue: events refill_work [ 861.546405][ T44] Call Trace: [ 861.546495][ T44] [ 861.546558][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.546678][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.546802][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.546921][ T44] ? find_held_lock+0x2b/0x80 [ 861.547051][ T44] ? _raw_spin_unlock_irqrestore+0x53/0x80 [ 861.547197][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.547314][ T44] ? schedule+0xd4/0x260 [ 861.547402][ T44] ? schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 861.547548][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 861.547665][ T44] ? hrtimer_cb_get_time+0x90/0x90 [ 861.547789][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.547910][ T44] ? ktime_get+0x15e/0x220 [ 861.548033][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 861.548155][ T44] ? ktime_get+0xa9/0x220 [ 861.548244][ T44] ? usleep_range_state+0x133/0x1b0 [ 861.548361][ T44] ? schedule_hrtimeout+0x20/0x20 [ 861.548482][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 861.548601][ T44] ? napi_disable_locked+0x379/0x5f0 [ 861.548728][ T44] ? _mutex_trylock_nest_lock+0x150/0x150 [ 861.548845][ T44] ? netdev_bind_sb_channel_queue+0x470/0x470 [ 861.548996][ T44] ? napi_disable+0x43/0x80 [ 861.549119][ T44] ? refill_work+0x116/0x210 [ 861.549239][ T44] ? process_one_work+0xd57/0x1390 [ 861.549360][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 861.549482][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.549602][ T44] ? assign_work+0x152/0x380 [ 861.549727][ T44] ? worker_thread+0x4d6/0xd40 [ 861.549847][ T44] ? process_one_work+0x1390/0x1390 [ 861.549964][ T44] ? __kthread_parkme+0xb3/0x200 [ 861.550089][ T44] ? process_one_work+0x1390/0x1390 [ 861.550211][ T44] ? kthread+0x355/0x5b0 [ 861.550300][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.550421][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.550540][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.550657][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.550779][ T44] ? ret_from_fork+0x1d4/0x270 [ 861.550896][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.551018][ T44] ? ret_from_fork_asm+0x11/0x20 [ 861.551143][ T44] [ 861.551236][ T44] [ 861.551236][ T44] Showing all locks held in the system: [ 861.551415][ T44] 3 locks held by kworker/u16:0/12: [ 861.551531][ T44] #0: ff110000090f4d48 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.551771][ T44] #1: ffa00000000c7d30 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.552063][ T44] #2: ffffffffaa692368 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 861.552276][ T44] 1 lock held by khungtaskd/44: [ 861.552390][ T44] #0: ffffffffa9f78fc0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x2b/0x1db [ 861.552591][ T44] 3 locks held by kworker/2:2/62: [ 861.552714][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.552947][ T44] #1: ffa0000000437d30 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.553207][ T44] #2: ff11000008a52c98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 861.553423][ T44] 2 locks held by ip/5902: [ 861.553536][ T44] #0: ffffffffaa692368 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x5df/0xb20 [ 861.553741][ T44] #1: ff11000008a52c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 861.553950][ T44] [ 861.554015][ T44] ============================================= [ 861.554015][ T44] [ 865.116722][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 110080 ms [ 865.117127][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 110080000 usecs ago [ 870.236718][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 115200 ms [ 870.237156][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 115200000 usecs ago [ 875.355727][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 120319 ms [ 875.356268][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 120319000 usecs ago [ 881.500722][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 126464 ms [ 881.501104][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 126464000 usecs ago [ 887.132722][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 132095 ms [ 887.133117][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 132096000 usecs ago [ 892.252722][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 137216 ms [ 892.253163][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 137216000 usecs ago [ 897.371720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 142335 ms [ 897.372156][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 142335000 usecs ago [ 903.515723][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 148479 ms [ 903.516162][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 148479000 usecs ago [ 909.147721][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 154111 ms [ 909.148089][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 154111000 usecs ago [ 914.268718][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 159232 ms [ 914.269129][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 159232000 usecs ago [ 919.388720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 164352 ms [ 919.389127][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 164352000 usecs ago [ 925.532716][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 170495 ms [ 925.533185][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 170496000 usecs ago [ 931.163722][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 176127 ms [ 931.164117][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 176127000 usecs ago [ 936.284721][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 181248 ms [ 936.285158][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 181248000 usecs ago [ 941.403719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 186367 ms [ 941.404248][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 186367000 usecs ago [ 947.547721][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 192511 ms [ 947.548117][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 192511000 usecs ago [ 953.180720][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 198144 ms [ 953.181100][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 198144000 usecs ago [ 958.299730][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 203263 ms [ 958.300159][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 203263000 usecs ago [ 963.419730][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 208383 ms [ 963.420152][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 208383000 usecs ago [ 969.564719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 214528 ms [ 969.565121][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 214528000 usecs ago [ 975.196719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 220160 ms [ 975.197179][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 220160000 usecs ago [ 980.316719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 225280 ms [ 980.317126][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 225280000 usecs ago [ 984.411763][ T44] INFO: task kworker/u16:0:12 blocked for more than 122 seconds. [ 984.412050][ T44] Not tainted 6.18.0-rc7-virtme #1 [ 984.412209][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 984.412407][ T44] task:kworker/u16:0 state:D stack:27000 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 984.412677][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 984.412863][ T44] Call Trace: [ 984.412954][ T44] [ 984.413021][ T44] __schedule+0x88d/0x1b90 [ 984.413163][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.413286][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.413409][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.413527][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.413648][ T44] schedule+0xd4/0x260 [ 984.413754][ T44] schedule_preempt_disabled+0x18/0x30 [ 984.413878][ T44] __mutex_lock+0x9ae/0x1500 [ 984.413998][ T44] ? addrconf_verify_work+0x12/0x30 [ 984.414121][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.414245][ T44] ? process_one_work+0xd16/0x1390 [ 984.414366][ T44] ? trace_lock_acquire+0x88/0xd0 [ 984.414484][ T44] ? trace_lock_acquire+0x88/0xd0 [ 984.414603][ T44] ? addrconf_verify_work+0x12/0x30 [ 984.414726][ T44] addrconf_verify_work+0x12/0x30 [ 984.414850][ T44] process_one_work+0xd57/0x1390 [ 984.414975][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 984.415091][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.415217][ T44] ? assign_work+0x152/0x380 [ 984.415335][ T44] worker_thread+0x4d6/0xd40 [ 984.415455][ T44] ? process_one_work+0x1390/0x1390 [ 984.415572][ T44] kthread+0x355/0x5b0 [ 984.415668][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.415793][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.415910][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.416040][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.416164][ T44] ret_from_fork+0x1d4/0x270 [ 984.416283][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.416399][ T44] ret_from_fork_asm+0x11/0x20 [ 984.416521][ T44] [ 984.416657][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5902. [ 984.416877][ T44] INFO: task ip:5902 blocked for more than 245 seconds. [ 984.417013][ T44] Not tainted 6.18.0-rc7-virtme #1 [ 984.417127][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 984.417320][ T44] task:ip state:D stack:25496 pid:5902 tgid:5902 ppid:5897 task_flags:0x400100 flags:0x00080001 [ 984.417590][ T44] Call Trace: [ 984.417677][ T44] [ 984.417742][ T44] __schedule+0x88d/0x1b90 [ 984.417863][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.417986][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.418109][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.418228][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.418351][ T44] schedule+0xd4/0x260 [ 984.418445][ T44] schedule_preempt_disabled+0x18/0x30 [ 984.418562][ T44] __mutex_lock+0x9ae/0x1500 [ 984.418680][ T44] ? napi_enable+0x3b/0x80 [ 984.418806][ T44] ? virtqueue_add_inbuf_ctx+0x1850/0x1850 [ 984.418961][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.419077][ T44] ? __asan_memset+0x27/0x50 [ 984.419204][ T44] ? sg_init_table+0x19/0x60 [ 984.419324][ T44] ? try_fill_recv+0x3ec/0x1750 [ 984.419449][ T44] ? napi_enable+0x3b/0x80 [ 984.419566][ T44] napi_enable+0x3b/0x80 [ 984.419657][ T44] __virtnet_rx_resume+0x156/0x290 [ 984.419778][ T44] ? queue_delayed_work_on+0x91/0x100 [ 984.419904][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 984.420023][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 984.420154][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 984.420268][ T44] dev_xdp_install+0x3e4/0x8e0 [ 984.420386][ T44] ? netif_skb_check_for_xdp+0x600/0x600 [ 984.420504][ T44] dev_xdp_attach+0x50d/0x1320 [ 984.420624][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 984.420744][ T44] ? __bpf_prog_get+0x320/0x320 [ 984.420864][ T44] dev_change_xdp_fd+0x182/0x240 [ 984.420988][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 984.421113][ T44] ? rtnl_newlink_create+0x900/0x900 [ 984.421234][ T44] ? rcu_lockdep_current_cpu_online+0x3e/0x1b0 [ 984.421379][ T44] ? rcu_read_lock_any_held+0x41/0xa0 [ 984.421496][ T44] ? validate_chain+0x287/0x5f0 [ 984.421612][ T44] ? rcu_read_lock_any_held+0x6b/0xa0 [ 984.421734][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 984.421857][ T44] ? __lock_acquire+0x34c/0x740 [ 984.421978][ T44] ? trace_lock_acquire+0x88/0xd0 [ 984.422093][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 984.422215][ T44] ? lock_acquire+0x31/0x80 [ 984.422329][ T44] ? rtnl_newlink+0x5df/0xb20 [ 984.422446][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.422563][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.422684][ T44] ? trace_contention_end+0xd6/0x140 [ 984.422814][ T44] ? rtnl_newlink+0x5df/0xb20 [ 984.422930][ T44] ? __mutex_lock+0x32e/0x1500 [ 984.423051][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.423170][ T44] ? nla_validate_int_range+0x5b0/0x5b0 [ 984.423291][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 984.423412][ T44] rtnl_newlink+0x637/0xb20 [ 984.423532][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 984.423657][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.423779][ T44] ? find_held_lock+0x2b/0x80 [ 984.423897][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 984.424013][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.424134][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 984.424259][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 984.424380][ T44] ? rtnl_port_fill+0x940/0x940 [ 984.424497][ T44] ? __lock_acquire+0x34c/0x740 [ 984.424624][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.424746][ T44] ? find_held_lock+0x2b/0x80 [ 984.424864][ T44] netlink_rcv_skb+0x123/0x380 [ 984.424983][ T44] ? rtnl_port_fill+0x940/0x940 [ 984.425101][ T44] ? netlink_ack+0xcc0/0xcc0 [ 984.425228][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 984.425348][ T44] netlink_unicast+0x4a3/0x770 [ 984.425471][ T44] ? netlink_attachskb+0x810/0x810 [ 984.425589][ T44] ? __lock_acquire+0x34c/0x740 [ 984.425711][ T44] netlink_sendmsg+0x735/0xc60 [ 984.425828][ T44] ? netlink_unicast+0x770/0x770 [ 984.425948][ T44] ? __might_fault+0x97/0x140 [ 984.426066][ T44] ____sys_sendmsg+0x419/0x850 [ 984.426192][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 984.426309][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 984.426466][ T44] ? move_addr_to_kernel+0x40/0x40 [ 984.426590][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 984.426716][ T44] ? validate_chain+0x287/0x5f0 [ 984.426841][ T44] ___sys_sendmsg+0xfd/0x180 [ 984.426965][ T44] ? __call_rcu_common.constprop.0+0xa5/0x680 [ 984.427122][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 984.427258][ T44] __sys_sendmsg+0x124/0x1c0 [ 984.427392][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 984.427517][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 984.427640][ T44] ? __call_rcu_common.constprop.0+0x2fa/0x680 [ 984.427800][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 984.427926][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.428051][ T44] do_syscall_64+0xbb/0xfd0 [ 984.428180][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 984.428345][ T44] RIP: 0033:0x7f7b2933cc5e [ 984.428480][ T44] RSP: 002b:00007ffc238701f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 984.428667][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f7b2933cc5e [ 984.428856][ T44] RDX: 0000000000000000 RSI: 00007ffc238702b0 RDI: 0000000000000003 [ 984.429041][ T44] RBP: 00007ffc23870200 R08: 0000000000000000 R09: 0000000000000000 [ 984.429236][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc23870970 [ 984.429415][ T44] R13: 00000000692a3dd6 R14: 0000000000000000 R15: 0000000000499600 [ 984.429604][ T44] [ 984.429708][ T44] INFO: task ip:5902 is blocked on a mutex likely owned by task kworker/2:2:62. [ 984.429919][ T44] task:kworker/2:2 state:D stack:28200 pid:62 tgid:62 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 984.430195][ T44] Workqueue: events refill_work [ 984.430319][ T44] Call Trace: [ 984.430408][ T44] [ 984.430470][ T44] __schedule+0x88d/0x1b90 [ 984.430595][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.430722][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.430846][ T44] ? usleep_range_state+0xba/0x1b0 [ 984.430964][ T44] ? find_held_lock+0x2b/0x80 [ 984.431081][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.431201][ T44] ? _raw_spin_unlock_irqrestore+0x53/0x80 [ 984.431348][ T44] ? usleep_range_state+0xba/0x1b0 [ 984.431466][ T44] schedule+0xd4/0x260 [ 984.431556][ T44] ? schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 984.431712][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 984.431830][ T44] ? hrtimer_cb_get_time+0x90/0x90 [ 984.431950][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 984.432068][ T44] ? ktime_get+0x15e/0x220 [ 984.432189][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 984.432306][ T44] ? ktime_get+0xa9/0x220 [ 984.432396][ T44] ? usleep_range_state+0x133/0x1b0 [ 984.432518][ T44] ? schedule_hrtimeout+0x20/0x20 [ 984.432634][ T44] ? __free_object+0x2ad/0x4b0 [ 984.432760][ T44] ? napi_disable_locked+0x379/0x5f0 [ 984.432877][ T44] ? _mutex_trylock_nest_lock+0x150/0x150 [ 984.432995][ T44] ? netdev_bind_sb_channel_queue+0x470/0x470 [ 984.433153][ T44] ? napi_disable+0x43/0x80 [ 984.433270][ T44] ? refill_work+0x116/0x210 [ 984.433396][ T44] ? process_one_work+0xd57/0x1390 [ 984.433519][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 984.433636][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.433760][ T44] ? assign_work+0x152/0x380 [ 984.433883][ T44] ? worker_thread+0x4d6/0xd40 [ 984.434007][ T44] ? process_one_work+0x1390/0x1390 [ 984.434134][ T44] ? __kthread_parkme+0xb3/0x200 [ 984.434261][ T44] ? process_one_work+0x1390/0x1390 [ 984.434377][ T44] ? kthread+0x355/0x5b0 [ 984.434466][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.434588][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.434710][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.434829][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.434950][ T44] ? ret_from_fork+0x1d4/0x270 [ 984.435071][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.435199][ T44] ? ret_from_fork_asm+0x11/0x20 [ 984.435323][ T44] [ 984.435416][ T44] [ 984.435416][ T44] Showing all locks held in the system: [ 984.435588][ T44] 3 locks held by kworker/u16:0/12: [ 984.435708][ T44] #0: ff110000090f4d48 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 984.435941][ T44] #1: ffa00000000c7d30 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 984.436239][ T44] #2: ffffffffaa692368 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 984.436450][ T44] 1 lock held by khungtaskd/44: [ 984.436565][ T44] #0: ffffffffa9f78fc0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x2b/0x1db [ 984.436778][ T44] 3 locks held by kworker/2:2/62: [ 984.436890][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 984.437119][ T44] #1: ffa0000000437d30 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 984.437375][ T44] #2: ff11000008a52c98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 984.437600][ T44] 2 locks held by ip/5902: [ 984.437718][ T44] #0: ffffffffaa692368 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x5df/0xb20 [ 984.437917][ T44] #1: ff11000008a52c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 984.438127][ T44] [ 984.438185][ T44] ============================================= [ 984.438185][ T44] [ 985.436719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 230400 ms [ 985.437195][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 230400000 usecs ago [ 991.579721][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 236543 ms [ 991.580177][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 236543000 usecs ago [ 997.211716][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 242175 ms [ 997.212082][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 242175000 usecs ago [ 1002.331723][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 247295 ms [ 1002.332164][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 247295000 usecs ago [ 1008.475721][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 253439 ms [ 1008.476163][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 253439000 usecs ago [ 1014.107719][ C1] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 1: transmit queue 2 timed out 259071 ms [ 1014.108099][ C1] virtio_net virtio2 enp0s1: TX timeout on queue: 2, sq: output.2, vq: 0x5, name: output.2, 259071000 usecs ago