[ 672.064377][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 5121 ms [ 672.064927][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 5121000 usecs ago [ 677.183376][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 10240 ms [ 677.183778][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 10240000 usecs ago [ 682.304371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 15361 ms [ 682.304759][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 15361000 usecs ago [ 688.448375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 21505 ms [ 688.448774][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 21505000 usecs ago [ 694.079376][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 27136 ms [ 694.079779][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 27136000 usecs ago [ 699.200371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 32257 ms [ 699.200748][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 32257000 usecs ago [ 704.320374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 37377 ms [ 704.320764][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 37377000 usecs ago [ 710.464375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 43521 ms [ 710.464776][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 43521000 usecs ago [ 716.096371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 49153 ms [ 716.096767][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 49153000 usecs ago [ 721.216373][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 54273 ms [ 721.216753][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 54273000 usecs ago [ 726.336375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 59393 ms [ 726.336763][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 59393000 usecs ago [ 732.480375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 65537 ms [ 732.480763][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 65537000 usecs ago [ 738.111371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 71168 ms [ 738.111757][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 71168000 usecs ago [ 743.232376][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 76289 ms [ 743.232773][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 76289000 usecs ago [ 748.352375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 81409 ms [ 748.352758][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 81409000 usecs ago [ 754.496372][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 87553 ms [ 754.496788][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 87553000 usecs ago [ 760.128371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 93185 ms [ 760.128780][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 93185000 usecs ago [ 765.248374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 98305 ms [ 765.248757][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 98305000 usecs ago [ 770.368371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 103425 ms [ 770.368819][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 103425000 usecs ago [ 776.512375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 109569 ms [ 776.512817][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 109569000 usecs ago [ 782.144374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 115201 ms [ 782.144811][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 115201000 usecs ago [ 787.264371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 120321 ms [ 787.264801][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 120321000 usecs ago [ 792.384374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 125441 ms [ 792.384809][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 125441000 usecs ago [ 798.528375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 131585 ms [ 798.528787][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 131585000 usecs ago [ 804.160371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 137217 ms [ 804.160792][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 137217000 usecs ago [ 809.279371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 142336 ms [ 809.279793][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 142336000 usecs ago [ 814.400375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 147457 ms [ 814.400794][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 147457000 usecs ago [ 820.544375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 153601 ms [ 820.544808][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 153601000 usecs ago [ 826.175371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 159232 ms [ 826.175798][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 159232000 usecs ago [ 831.296378][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 164353 ms [ 831.296788][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 164353000 usecs ago [ 836.416374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 169473 ms [ 836.416819][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 169473000 usecs ago [ 842.560372][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 175617 ms [ 842.560807][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 175617000 usecs ago [ 848.192374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 181249 ms [ 848.192805][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 181249000 usecs ago [ 853.311375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 186368 ms [ 853.311818][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 186368000 usecs ago [ 859.456371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 192513 ms [ 859.456829][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 192513000 usecs ago [ 861.503396][ T44] INFO: task ip:5984 blocked for more than 122 seconds. [ 861.503716][ T44] Not tainted 6.19.0-rc3-virtme #1 [ 861.503924][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 861.504203][ T44] task:ip state:D stack:25312 pid:5984 tgid:5984 ppid:5979 task_flags:0x400100 flags:0x00080000 [ 861.504604][ T44] Call Trace: [ 861.504769][ T44] [ 861.504910][ T44] __schedule+0xae2/0x1720 [ 861.505092][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.505264][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.505448][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.505621][ T44] schedule+0xd4/0x260 [ 861.505773][ T44] schedule_preempt_disabled+0x18/0x30 [ 861.505941][ T44] __mutex_lock+0xd76/0x1f20 [ 861.506109][ T44] ? napi_enable+0x3b/0x80 [ 861.506278][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.506457][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.506622][ T44] ? __asan_memset+0x27/0x50 [ 861.506798][ T44] ? sg_init_table+0x19/0x60 [ 861.506965][ T44] ? try_fill_recv+0x3ec/0x1750 [ 861.507135][ T44] ? napi_enable+0x3b/0x80 [ 861.507298][ T44] napi_enable+0x3b/0x80 [ 861.507445][ T44] __virtnet_rx_resume+0x156/0x290 [ 861.507607][ T44] ? queue_delayed_work_on+0x91/0x100 [ 861.507776][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 861.507943][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 861.508109][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 861.508274][ T44] dev_xdp_install+0x3e4/0x890 [ 861.508453][ T44] ? netif_set_xps_queue+0x40/0x40 [ 861.508724][ T44] dev_xdp_attach+0x50d/0x1320 [ 861.508896][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 861.509058][ T44] ? __bpf_prog_get+0x320/0x320 [ 861.509222][ T44] dev_change_xdp_fd+0x176/0x210 [ 861.509403][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 861.509571][ T44] ? rtnl_newlink_create+0x900/0x900 [ 861.509739][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 861.509932][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 861.510093][ T44] ? validate_chain+0x37b/0xb70 [ 861.510255][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 861.510430][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 861.510593][ T44] ? __lock_acquire+0x577/0xc10 [ 861.510767][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.510929][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 861.511089][ T44] ? lock_acquire+0x10a/0x150 [ 861.511248][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.511422][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.511583][ T44] ? trace_contention_end+0xd6/0x110 [ 861.511750][ T44] ? __mutex_lock+0x285/0x1f20 [ 861.511913][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 861.512081][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.512242][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 861.512418][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 861.512584][ T44] rtnl_newlink+0x75c/0xe90 [ 861.512756][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.512918][ T44] ? rtnl_setlink+0x880/0x880 [ 861.513080][ T44] ? __lock_acquire+0x577/0xc10 [ 861.513247][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.513421][ T44] ? find_held_lock+0x2b/0x80 [ 861.513585][ T44] ? rtnl_setlink+0x880/0x880 [ 861.513753][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.513915][ T44] ? rtnl_setlink+0x880/0x880 [ 861.514076][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 861.514238][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 861.514414][ T44] ? __lock_acquire+0x577/0xc10 [ 861.514582][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.514744][ T44] ? find_held_lock+0x2b/0x80 [ 861.514908][ T44] netlink_rcv_skb+0x123/0x380 [ 861.515071][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 861.515232][ T44] ? netlink_ack+0xcc0/0xcc0 [ 861.515411][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 861.515572][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 861.515743][ T44] netlink_unicast+0x4a3/0x770 [ 861.515907][ T44] ? netlink_attachskb+0x810/0x810 [ 861.516069][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.516231][ T44] ? __lock_acquire+0x577/0xc10 [ 861.516409][ T44] netlink_sendmsg+0x735/0xc60 [ 861.516574][ T44] ? netlink_unicast+0x770/0x770 [ 861.516744][ T44] ____sys_sendmsg+0x419/0x850 [ 861.516908][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 861.517072][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 861.517260][ T44] ? move_addr_to_kernel+0x40/0x40 [ 861.517437][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 861.517596][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 861.517764][ T44] ___sys_sendmsg+0xfd/0x180 [ 861.517925][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 861.518118][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 861.518293][ T44] __sys_sendmsg+0x124/0x1c0 [ 861.518469][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 861.518629][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.518795][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 861.518988][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 861.519150][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.519316][ T44] do_syscall_64+0xbd/0xfc0 [ 861.519493][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 861.519682][ T44] RIP: 0033:0x7f21bf175c5e [ 861.519858][ T44] RSP: 002b:00007ffefaec4730 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 861.520079][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f21bf175c5e [ 861.520297][ T44] RDX: 0000000000000000 RSI: 00007ffefaec47f0 RDI: 0000000000000003 [ 861.520529][ T44] RBP: 00007ffefaec4740 R08: 0000000000000000 R09: 0000000000000000 [ 861.520755][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffefaec4eb0 [ 861.520971][ T44] R13: 00000000695cc658 R14: 0000000000000000 R15: 0000000000499600 [ 861.521195][ T44] [ 861.521333][ T44] INFO: task ip:5984 is blocked on a mutex likely owned by task kworker/3:1:60. [ 861.521581][ T44] task:kworker/3:1 state:D stack:28056 pid:60 tgid:60 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 861.521883][ T44] Workqueue: events refill_work [ 861.522047][ T44] Call Trace: [ 861.522183][ T44] [ 861.522291][ T44] __schedule+0xae2/0x1720 [ 861.522470][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.522636][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.522802][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.522966][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.523128][ T44] schedule+0xd4/0x260 [ 861.523264][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 861.523468][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 861.523631][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 861.523800][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.523959][ T44] ? ktime_get+0x15e/0x220 [ 861.524121][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 861.524281][ T44] ? ktime_get+0xa9/0x220 [ 861.524431][ T44] usleep_range_state+0x133/0x1b0 [ 861.524594][ T44] ? schedule_hrtimeout+0x20/0x20 [ 861.524762][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 861.524926][ T44] ? refill_work+0x149/0x210 [ 861.525090][ T44] napi_disable_locked+0x379/0x5f0 [ 861.525250][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 861.525424][ T44] ? netif_get_flags+0x1e0/0x1e0 [ 861.525588][ T44] napi_disable+0x43/0x80 [ 861.525727][ T44] refill_work+0x116/0x210 [ 861.525892][ T44] process_one_work+0xd57/0x1390 [ 861.526062][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 861.526222][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.526400][ T44] ? assign_work+0x152/0x380 [ 861.526561][ T44] worker_thread+0x4d6/0xd40 [ 861.526734][ T44] ? process_one_work+0x1390/0x1390 [ 861.526896][ T44] kthread+0x355/0x5b0 [ 861.527031][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.527193][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.527357][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.527534][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.527697][ T44] ret_from_fork+0x3fb/0x510 [ 861.527872][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 861.528063][ T44] ? __switch_to+0x53c/0xd00 [ 861.528226][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.528402][ T44] ret_from_fork_asm+0x11/0x20 [ 861.528574][ T44] [ 861.528713][ T44] [ 861.528713][ T44] Showing all locks held in the system: [ 861.528928][ T44] 3 locks held by kworker/u16:0/12: [ 861.529087][ T44] #0: ff110000090af148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.529359][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.529706][ T44] #2: ffffffff90885ba8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 861.529954][ T44] 1 lock held by khungtaskd/44: [ 861.530109][ T44] #0: ffffffff901710c0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 861.530395][ T44] 3 locks held by kworker/3:1/60: [ 861.530550][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.530822][ T44] #1: ffa000000040fcc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.531117][ T44] #2: ff11000008a41cb0 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 861.531381][ T44] 2 locks held by ip/5984: [ 861.531536][ T44] #0: ffffffff90885ba8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 861.531787][ T44] #1: ff11000008a41cb0 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 861.532029][ T44] [ 861.532132][ T44] ============================================= [ 861.532132][ T44] [ 865.088374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 198145 ms [ 865.088797][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 198145000 usecs ago [ 870.208376][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 203265 ms [ 870.208807][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 203265000 usecs ago [ 875.328371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 208385 ms [ 875.328823][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 208385000 usecs ago [ 881.472374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 214529 ms [ 881.472794][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 214529000 usecs ago [ 887.104376][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 220161 ms [ 887.104811][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 220161000 usecs ago [ 892.224375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 225281 ms [ 892.224801][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 225281000 usecs ago [ 897.343372][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 230400 ms [ 897.343799][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 230400000 usecs ago [ 903.487375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 236544 ms [ 903.487803][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 236544000 usecs ago [ 909.119375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 242176 ms [ 909.119799][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 242176000 usecs ago [ 914.240372][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 247297 ms [ 914.240796][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 247297000 usecs ago [ 919.360375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 252417 ms [ 919.360829][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 252417000 usecs ago [ 925.504375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 258561 ms [ 925.504806][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 258561000 usecs ago [ 931.136372][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 264193 ms [ 931.136804][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 264193000 usecs ago [ 936.256374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 269313 ms [ 936.256791][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 269313000 usecs ago [ 941.376374][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 274433 ms [ 941.376822][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 274433000 usecs ago [ 947.520371][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 280577 ms [ 947.520800][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 280577000 usecs ago [ 953.152375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 286209 ms [ 953.152812][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 286209000 usecs ago [ 958.272375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 291329 ms [ 958.272818][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 291329000 usecs ago [ 963.392375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 296449 ms [ 963.392793][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 296449000 usecs ago [ 969.536373][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 302593 ms [ 969.536783][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 302593000 usecs ago [ 975.168375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 308225 ms [ 975.168794][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 308225000 usecs ago [ 980.287375][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 313344 ms [ 980.287813][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 313344000 usecs ago [ 984.384380][ T44] INFO: task kworker/u16:0:12 blocked for more than 122 seconds. [ 984.384709][ T44] Not tainted 6.19.0-rc3-virtme #1 [ 984.384905][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 984.385234][ T44] task:kworker/u16:0 state:D stack:26816 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 984.385593][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 984.385803][ T44] Call Trace: [ 984.385946][ T44] [ 984.386063][ T44] __schedule+0xae2/0x1720 [ 984.386241][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.386425][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.386591][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.386759][ T44] schedule+0xd4/0x260 [ 984.386899][ T44] schedule_preempt_disabled+0x18/0x30 [ 984.387076][ T44] __mutex_lock+0xd76/0x1f20 [ 984.387248][ T44] ? addrconf_verify_work+0x12/0x30 [ 984.387427][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.387596][ T44] ? process_one_work+0xd16/0x1390 [ 984.387761][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.387925][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.388093][ T44] ? addrconf_verify_work+0x12/0x30 [ 984.388258][ T44] addrconf_verify_work+0x12/0x30 [ 984.388433][ T44] process_one_work+0xd57/0x1390 [ 984.388598][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 984.388760][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.388927][ T44] ? assign_work+0x152/0x380 [ 984.389092][ T44] worker_thread+0x4d6/0xd40 [ 984.389260][ T44] ? process_one_work+0x1390/0x1390 [ 984.389435][ T44] kthread+0x355/0x5b0 [ 984.389575][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.389737][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.389898][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.390066][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.390228][ T44] ret_from_fork+0x3fb/0x510 [ 984.390405][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 984.390593][ T44] ? __switch_to+0x53c/0xd00 [ 984.390758][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.390920][ T44] ret_from_fork_asm+0x11/0x20 [ 984.391094][ T44] [ 984.391234][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5984. [ 984.391491][ T44] INFO: task ip:5984 blocked for more than 245 seconds. [ 984.391670][ T44] Not tainted 6.19.0-rc3-virtme #1 [ 984.391821][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 984.392057][ T44] task:ip state:D stack:25312 pid:5984 tgid:5984 ppid:5979 task_flags:0x400100 flags:0x00080000 [ 984.392350][ T44] Call Trace: [ 984.392494][ T44] [ 984.392599][ T44] __schedule+0xae2/0x1720 [ 984.392765][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.392927][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.393095][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.393256][ T44] schedule+0xd4/0x260 [ 984.393406][ T44] schedule_preempt_disabled+0x18/0x30 [ 984.393566][ T44] __mutex_lock+0xd76/0x1f20 [ 984.393728][ T44] ? napi_enable+0x3b/0x80 [ 984.393891][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.394056][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.394221][ T44] ? __asan_memset+0x27/0x50 [ 984.394397][ T44] ? sg_init_table+0x19/0x60 [ 984.394561][ T44] ? try_fill_recv+0x3ec/0x1750 [ 984.394725][ T44] ? napi_enable+0x3b/0x80 [ 984.394885][ T44] napi_enable+0x3b/0x80 [ 984.395020][ T44] __virtnet_rx_resume+0x156/0x290 [ 984.395185][ T44] ? queue_delayed_work_on+0x91/0x100 [ 984.395351][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 984.395529][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 984.395696][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 984.395858][ T44] dev_xdp_install+0x3e4/0x890 [ 984.396020][ T44] ? netif_set_xps_queue+0x40/0x40 [ 984.396188][ T44] dev_xdp_attach+0x50d/0x1320 [ 984.396354][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 984.396533][ T44] ? __bpf_prog_get+0x320/0x320 [ 984.396700][ T44] dev_change_xdp_fd+0x176/0x210 [ 984.396863][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 984.397026][ T44] ? rtnl_newlink_create+0x900/0x900 [ 984.397195][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 984.397400][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 984.397559][ T44] ? validate_chain+0x37b/0xb70 [ 984.397723][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 984.397884][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 984.398046][ T44] ? __lock_acquire+0x577/0xc10 [ 984.398216][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.398393][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 984.398551][ T44] ? lock_acquire+0x10a/0x150 [ 984.398711][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.398870][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.399034][ T44] ? trace_contention_end+0xd6/0x110 [ 984.399203][ T44] ? __mutex_lock+0x285/0x1f20 [ 984.399378][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 984.399540][ T44] ? ww_mutex_lock+0x160/0x160 [ 984.399700][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 984.399864][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 984.400029][ T44] rtnl_newlink+0x75c/0xe90 [ 984.400203][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.400377][ T44] ? rtnl_setlink+0x880/0x880 [ 984.400538][ T44] ? __lock_acquire+0x577/0xc10 [ 984.400701][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.400862][ T44] ? find_held_lock+0x2b/0x80 [ 984.401023][ T44] ? rtnl_setlink+0x880/0x880 [ 984.401189][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.401350][ T44] ? rtnl_setlink+0x880/0x880 [ 984.401528][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 984.401692][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 984.401852][ T44] ? __lock_acquire+0x577/0xc10 [ 984.402015][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.402176][ T44] ? find_held_lock+0x2b/0x80 [ 984.402339][ T44] netlink_rcv_skb+0x123/0x380 [ 984.402515][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 984.402678][ T44] ? netlink_ack+0xcc0/0xcc0 [ 984.402848][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 984.403008][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 984.403177][ T44] netlink_unicast+0x4a3/0x770 [ 984.403342][ T44] ? netlink_attachskb+0x810/0x810 [ 984.403517][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.403678][ T44] ? __lock_acquire+0x577/0xc10 [ 984.403839][ T44] netlink_sendmsg+0x735/0xc60 [ 984.404003][ T44] ? netlink_unicast+0x770/0x770 [ 984.404176][ T44] ____sys_sendmsg+0x419/0x850 [ 984.404337][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 984.404511][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 984.404696][ T44] ? move_addr_to_kernel+0x40/0x40 [ 984.404858][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 984.405017][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 984.405181][ T44] ___sys_sendmsg+0xfd/0x180 [ 984.405341][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 984.405543][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 984.405716][ T44] __sys_sendmsg+0x124/0x1c0 [ 984.405876][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 984.406037][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 984.406202][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 984.406410][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 984.406572][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.406734][ T44] do_syscall_64+0xbd/0xfc0 [ 984.406897][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 984.407089][ T44] RIP: 0033:0x7f21bf175c5e [ 984.407254][ T44] RSP: 002b:00007ffefaec4730 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 984.407488][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f21bf175c5e [ 984.407702][ T44] RDX: 0000000000000000 RSI: 00007ffefaec47f0 RDI: 0000000000000003 [ 984.407916][ T44] RBP: 00007ffefaec4740 R08: 0000000000000000 R09: 0000000000000000 [ 984.408139][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffefaec4eb0 [ 984.408356][ T44] R13: 00000000695cc658 R14: 0000000000000000 R15: 0000000000499600 [ 984.408591][ T44] [ 984.408725][ T44] INFO: task ip:5984 is blocked on a mutex likely owned by task kworker/3:1:60. [ 984.408956][ T44] task:kworker/3:1 state:D stack:28056 pid:60 tgid:60 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 984.409250][ T44] Workqueue: events refill_work [ 984.409426][ T44] Call Trace: [ 984.409557][ T44] [ 984.409662][ T44] __schedule+0xae2/0x1720 [ 984.409824][ T44] ? usleep_range_state+0xba/0x1b0 [ 984.409985][ T44] ? io_schedule_timeout+0x130/0x130 [ 984.410149][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.410312][ T44] ? usleep_range_state+0xba/0x1b0 [ 984.410485][ T44] schedule+0xd4/0x260 [ 984.410623][ T44] schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 984.410812][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 984.410974][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 984.411144][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 984.411305][ T44] ? ktime_get+0x15e/0x220 [ 984.411479][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 984.411639][ T44] ? ktime_get+0xa9/0x220 [ 984.411776][ T44] usleep_range_state+0x133/0x1b0 [ 984.411940][ T44] ? schedule_hrtimeout+0x20/0x20 [ 984.412109][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 984.412270][ T44] ? refill_work+0x149/0x210 [ 984.412445][ T44] napi_disable_locked+0x379/0x5f0 [ 984.412606][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 984.412766][ T44] ? netif_get_flags+0x1e0/0x1e0 [ 984.412934][ T44] napi_disable+0x43/0x80 [ 984.413072][ T44] refill_work+0x116/0x210 [ 984.413236][ T44] process_one_work+0xd57/0x1390 [ 984.413414][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 984.413575][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 984.413739][ T44] ? assign_work+0x152/0x380 [ 984.413900][ T44] worker_thread+0x4d6/0xd40 [ 984.414072][ T44] ? process_one_work+0x1390/0x1390 [ 984.414236][ T44] kthread+0x355/0x5b0 [ 984.414385][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.414546][ T44] ? __lock_release.isra.0+0x59/0x170 [ 984.414706][ T44] ? rcu_is_watching+0x15/0xd0 [ 984.414867][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.415029][ T44] ret_from_fork+0x3fb/0x510 [ 984.415199][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 984.415407][ T44] ? __switch_to+0x53c/0xd00 [ 984.415569][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 984.415731][ T44] ret_from_fork_asm+0x11/0x20 [ 984.415898][ T44] [ 984.416028][ T44] [ 984.416028][ T44] Showing all locks held in the system: [ 984.416244][ T44] 3 locks held by kworker/u16:0/12: [ 984.416411][ T44] #0: ff110000090af148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 984.416681][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 984.417003][ T44] #2: ffffffff90885ba8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 984.417251][ T44] 1 lock held by khungtaskd/44: [ 984.417420][ T44] #0: ffffffff901710c0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 984.417690][ T44] 3 locks held by kworker/3:1/60: [ 984.417846][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 984.418115][ T44] #1: ffa000000040fcc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 984.418423][ T44] #2: ff11000008a41cb0 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 984.418674][ T44] 2 locks held by ip/5984: [ 984.418829][ T44] #0: ffffffff90885ba8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 984.419074][ T44] #1: ff11000008a41cb0 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 984.419317][ T44] [ 984.419432][ T44] ============================================= [ 984.419432][ T44] [ 985.408373][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 1 timed out 318465 ms [ 985.408773][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 318465000 usecs ago