[ 10.740115][ T197] collect2 (197) used greatest stack depth: 24784 bytes left [ 45.263514][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 5007 ms [ 45.264046][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 5007000 usecs ago [ 50.495524][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 10239 ms [ 50.496158][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 10239000 usecs ago [ 56.127519][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 15871 ms [ 56.127948][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 15871000 usecs ago [ 61.247516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 20991 ms [ 61.247924][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 20991000 usecs ago [ 66.367526][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 26111 ms [ 66.368081][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 26111000 usecs ago [ 72.511537][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 32255 ms [ 72.511939][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 32255000 usecs ago [ 78.143502][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 37887 ms [ 78.143894][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 37887000 usecs ago [ 83.263517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 43007 ms [ 83.263937][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 43007000 usecs ago [ 88.383516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 48127 ms [ 88.383905][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 48127000 usecs ago [ 94.527505][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 54271 ms [ 94.527903][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 54271000 usecs ago [ 100.159516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 59903 ms [ 100.159939][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 59903000 usecs ago [ 105.279516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 65023 ms [ 105.279929][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 65023000 usecs ago [ 110.399505][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 70143 ms [ 110.399972][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 70143000 usecs ago [ 116.543516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 76287 ms [ 116.543944][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 76287000 usecs ago [ 122.175509][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 81919 ms [ 122.175953][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 81919000 usecs ago [ 127.295516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 87039 ms [ 127.295916][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 87039000 usecs ago [ 132.415502][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 92159 ms [ 132.415918][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 92159000 usecs ago [ 138.559503][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 98303 ms [ 138.559928][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 98303000 usecs ago [ 144.191516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 103935 ms [ 144.191958][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 103935000 usecs ago [ 149.311525][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 109055 ms [ 149.312176][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 109055000 usecs ago [ 155.455504][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 115199 ms [ 155.455978][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 115199000 usecs ago [ 161.087507][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 120831 ms [ 161.087999][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 120831000 usecs ago [ 166.207502][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 125951 ms [ 166.207962][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 125951000 usecs ago [ 171.327506][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 131071 ms [ 171.327933][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 131071000 usecs ago [ 177.471509][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 137215 ms [ 177.472148][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 137215000 usecs ago [ 183.103516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 142847 ms [ 183.103940][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 142847000 usecs ago [ 188.223518][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 147967 ms [ 188.223992][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 147967000 usecs ago [ 193.343521][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 153087 ms [ 193.343941][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 153087000 usecs ago [ 199.487517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 159231 ms [ 199.487944][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 159231000 usecs ago [ 205.119503][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 164863 ms [ 205.119950][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 164863000 usecs ago [ 210.239512][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 169983 ms [ 210.239922][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 169983000 usecs ago [ 215.359503][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 175103 ms [ 215.359925][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 175103000 usecs ago [ 221.503507][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 181247 ms [ 221.503925][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 181247000 usecs ago [ 227.135505][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 186879 ms [ 227.135919][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 186879000 usecs ago [ 232.255518][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 191999 ms [ 232.255991][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 191999000 usecs ago [ 237.375522][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 197119 ms [ 237.375952][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 197119000 usecs ago [ 243.519519][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 203263 ms [ 243.519924][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 203263000 usecs ago [ 247.103576][ T44] INFO: task ip:256 blocked for more than 122 seconds. [ 247.103913][ T44] Not tainted 6.18.0-rc7-virtme #1 [ 247.104031][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.104257][ T44] task:ip state:D stack:25496 pid:256 tgid:256 ppid:251 task_flags:0x400100 flags:0x00080001 [ 247.104637][ T44] Call Trace: [ 247.104744][ T44] [ 247.104848][ T44] __schedule+0x88d/0x1b90 [ 247.105036][ T44] ? io_schedule_timeout+0x130/0x130 [ 247.105176][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 247.105298][ T44] ? __lock_release.isra.0+0x59/0x170 [ 247.105434][ T44] ? __lock_release.isra.0+0x59/0x170 [ 247.105625][ T44] schedule+0xd4/0x260 [ 247.105735][ T44] schedule_preempt_disabled+0x18/0x30 [ 247.105888][ T44] __mutex_lock+0x9ae/0x1500 [ 247.106049][ T44] ? napi_enable+0x3b/0x80 [ 247.106214][ T44] ? virtqueue_add_inbuf_ctx+0x1850/0x1850 [ 247.106399][ T44] ? ww_mutex_lock+0x160/0x160 [ 247.106577][ T44] ? __asan_memset+0x27/0x50 [ 247.106751][ T44] ? sg_init_table+0x19/0x60 [ 247.106925][ T44] ? try_fill_recv+0x3ec/0x1750 [ 247.107092][ T44] ? napi_enable+0x3b/0x80 [ 247.107252][ T44] napi_enable+0x3b/0x80 [ 247.107374][ T44] __virtnet_rx_resume+0x156/0x290 [ 247.107542][ T44] ? queue_delayed_work_on+0x91/0x100 [ 247.107710][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 247.107876][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 247.108049][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 247.108213][ T44] dev_xdp_install+0x3e4/0x8e0 [ 247.108457][ T44] ? netif_skb_check_for_xdp+0x600/0x600 [ 247.108635][ T44] dev_xdp_attach+0x50d/0x1320 [ 247.109088][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 247.109225][ T44] ? __bpf_prog_get+0x320/0x320 [ 247.109370][ T44] dev_change_xdp_fd+0x182/0x240 [ 247.109522][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 247.109682][ T44] ? rtnl_newlink_create+0x900/0x900 [ 247.109834][ T44] ? rcu_lockdep_current_cpu_online+0x3e/0x1b0 [ 247.110023][ T44] ? rcu_read_lock_any_held+0x41/0xa0 [ 247.110175][ T44] ? validate_chain+0x287/0x5f0 [ 247.110325][ T44] ? rcu_read_lock_any_held+0x6b/0xa0 [ 247.110474][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 247.110645][ T44] ? __lock_acquire+0x34c/0x740 [ 247.110810][ T44] ? trace_lock_acquire+0x88/0xd0 [ 247.110959][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 247.111109][ T44] ? lock_acquire+0x31/0x80 [ 247.111257][ T44] ? rtnl_newlink+0x5df/0xb20 [ 247.111408][ T44] ? rcu_is_watching+0x15/0xd0 [ 247.111570][ T44] ? rcu_is_watching+0x15/0xd0 [ 247.111724][ T44] ? trace_contention_end+0xd6/0x140 [ 247.111882][ T44] ? rtnl_newlink+0x5df/0xb20 [ 247.112032][ T44] ? __mutex_lock+0x32e/0x1500 [ 247.112194][ T44] ? ww_mutex_lock+0x160/0x160 [ 247.112347][ T44] ? nla_validate_int_range+0x5b0/0x5b0 [ 247.112513][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 247.112678][ T44] rtnl_newlink+0x637/0xb20 [ 247.112795][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 247.112914][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 247.113023][ T44] ? find_held_lock+0x2b/0x80 [ 247.113132][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 247.113261][ T44] ? __lock_release.isra.0+0x59/0x170 [ 247.113371][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 247.113480][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 247.113599][ T44] ? rtnl_port_fill+0x940/0x940 [ 247.113716][ T44] ? __lock_acquire+0x34c/0x740 [ 247.113837][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 247.113946][ T44] ? find_held_lock+0x2b/0x80 [ 247.114058][ T44] netlink_rcv_skb+0x123/0x380 [ 247.114181][ T44] ? rtnl_port_fill+0x940/0x940 [ 247.114293][ T44] ? netlink_ack+0xcc0/0xcc0 [ 247.114412][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 247.114528][ T44] netlink_unicast+0x4a3/0x770 [ 247.114639][ T44] ? netlink_attachskb+0x810/0x810 [ 247.114760][ T44] ? __lock_acquire+0x34c/0x740 [ 247.114873][ T44] netlink_sendmsg+0x735/0xc60 [ 247.115002][ T44] ? netlink_unicast+0x770/0x770 [ 247.115144][ T44] ? __might_fault+0x97/0x140 [ 247.115292][ T44] ____sys_sendmsg+0x419/0x850 [ 247.115445][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 247.115606][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 247.115802][ T44] ? move_addr_to_kernel+0x40/0x40 [ 247.115957][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 247.116109][ T44] ? validate_chain+0x287/0x5f0 [ 247.116263][ T44] ___sys_sendmsg+0xfd/0x180 [ 247.116412][ T44] ? __call_rcu_common.constprop.0+0xa5/0x680 [ 247.116607][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 247.116788][ T44] __sys_sendmsg+0x124/0x1c0 [ 247.116946][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 247.117095][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 247.117244][ T44] ? __call_rcu_common.constprop.0+0x2fa/0x680 [ 247.117438][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 247.117599][ T44] ? rcu_is_watching+0x15/0xd0 [ 247.117760][ T44] do_syscall_64+0xbb/0xfd0 [ 247.117913][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 247.118104][ T44] RIP: 0033:0x7fab2d6d9c5e [ 247.118266][ T44] RSP: 002b:00007ffc268c3ed0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 247.118457][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fab2d6d9c5e [ 247.118636][ T44] RDX: 0000000000000000 RSI: 00007ffc268c3f90 RDI: 0000000000000003 [ 247.118819][ T44] RBP: 00007ffc268c3ee0 R08: 0000000000000000 R09: 0000000000000000 [ 247.118988][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc268c4650 [ 247.119159][ T44] R13: 000000006930d0d2 R14: 0000000000000000 R15: 0000000000499600 [ 247.119348][ T44] [ 247.119461][ T44] INFO: task ip:256 is blocked on a mutex likely owned by task kworker/2:2:64. [ 247.119653][ T44] task:kworker/2:2 state:D stack:28160 pid:64 tgid:64 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 247.119915][ T44] Workqueue: events refill_work [ 247.120033][ T44] Call Trace: [ 247.120117][ T44] [ 247.120186][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 247.120302][ T44] ? io_schedule_timeout+0x130/0x130 [ 247.120412][ T44] ? usleep_range_state+0xba/0x1b0 [ 247.120529][ T44] ? find_held_lock+0x2b/0x80 [ 247.120641][ T44] ? __lock_release.isra.0+0x59/0x170 [ 247.120758][ T44] ? usleep_range_state+0xba/0x1b0 [ 247.120867][ T44] ? schedule_hrtimeout_range_clock+0x1ca/0x2f0 [ 247.121003][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 247.121114][ T44] ? hrtimer_cb_get_time+0x90/0x90 [ 247.121235][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 247.121345][ T44] ? ktime_get+0x15e/0x220 [ 247.121453][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 247.121570][ T44] ? ktime_get+0xa9/0x220 [ 247.121653][ T44] ? usleep_range_state+0x133/0x1b0 [ 247.121769][ T44] ? schedule_hrtimeout+0x20/0x20 [ 247.121880][ T44] ? __free_object+0x2ad/0x4b0 [ 247.121997][ T44] ? napi_disable_locked+0x379/0x5f0 [ 247.122109][ T44] ? _mutex_trylock_nest_lock+0x150/0x150 [ 247.122229][ T44] ? netdev_bind_sb_channel_queue+0x470/0x470 [ 247.122371][ T44] ? napi_disable+0x43/0x80 [ 247.122490][ T44] ? refill_work+0x116/0x210 [ 247.122602][ T44] ? process_one_work+0xd57/0x1390 [ 247.122724][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 247.122835][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 247.122949][ T44] ? assign_work+0x152/0x380 [ 247.123059][ T44] ? worker_thread+0x4d6/0xd40 [ 247.123182][ T44] ? process_one_work+0x1390/0x1390 [ 247.123294][ T44] ? __kthread_parkme+0xb3/0x200 [ 247.123409][ T44] ? process_one_work+0x1390/0x1390 [ 247.123523][ T44] ? kthread+0x355/0x5b0 [ 247.123608][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 247.123733][ T44] ? __lock_release.isra.0+0x59/0x170 [ 247.123849][ T44] ? rcu_is_watching+0x15/0xd0 [ 247.123959][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 247.124071][ T44] ? ret_from_fork+0x1d4/0x270 [ 247.124188][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 247.124299][ T44] ? ret_from_fork_asm+0x11/0x20 [ 247.124421][ T44] [ 247.124515][ T44] [ 247.124515][ T44] Showing all locks held in the system: [ 247.124701][ T44] 1 lock held by khungtaskd/44: [ 247.124811][ T44] #0: ffffffffb7778fc0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x2b/0x1db [ 247.125010][ T44] 3 locks held by kworker/2:2/64: [ 247.125114][ T44] 3 locks held by kworker/u16:1/70: [ 247.125236][ T44] #0: ff11000008b6cd48 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 247.125493][ T44] #1: ffa00000004b7d30 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 247.125866][ T44] #2: ffffffffb7e925e8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 247.126134][ T44] 2 locks held by ip/256: [ 247.126242][ T44] #0: ffffffffb7e925e8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x5df/0xb20 [ 247.126513][ T44] #1: ff110000085b2c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 247.126778][ T44] [ 247.126853][ T44] ============================================= [ 247.126853][ T44] [ 249.151517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 208895 ms [ 249.151974][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 208895000 usecs ago [ 254.271517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 214015 ms [ 254.272003][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 214015000 usecs ago [ 259.391516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 219135 ms [ 259.391967][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 219135000 usecs ago [ 265.535520][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 225279 ms [ 265.536005][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 225279000 usecs ago [ 271.167517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 230911 ms [ 271.167971][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 230911000 usecs ago [ 276.287518][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 236031 ms [ 276.287974][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 236031000 usecs ago [ 281.407517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 241151 ms [ 281.407992][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 241151000 usecs ago [ 287.551519][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 247295 ms [ 287.551968][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 247295000 usecs ago [ 293.183518][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 252927 ms [ 293.183981][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 252927000 usecs ago [ 298.303516][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 258047 ms [ 298.303991][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 258047000 usecs ago [ 303.423519][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 263167 ms [ 303.423958][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 263167000 usecs ago [ 309.567517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 269311 ms [ 309.568000][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 269311000 usecs ago [ 315.199508][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 274943 ms [ 315.200000][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 274943000 usecs ago [ 320.319508][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 280063 ms [ 320.319994][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 280063000 usecs ago [ 326.463519][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 286207 ms [ 326.463981][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 286207000 usecs ago [ 332.095517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 291839 ms [ 332.095981][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 291839000 usecs ago [ 337.215506][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 296959 ms [ 337.215986][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 296959000 usecs ago [ 342.335517][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 302079 ms [ 342.336233][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 302079000 usecs ago [ 348.479522][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 308223 ms [ 348.479962][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 308223000 usecs ago [ 354.111543][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 313855 ms [ 354.112000][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 313855000 usecs ago [ 359.231534][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 318975 ms [ 359.231950][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 318975000 usecs ago [ 364.351522][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 324095 ms [ 364.351936][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 324095000 usecs ago [ 369.983569][ T44] INFO: task kworker/u16:1:70 blocked for more than 122 seconds. [ 369.983919][ T44] Not tainted 6.18.0-rc7-virtme #1 [ 369.984051][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.984277][ T44] task:kworker/u16:1 state:D stack:28040 pid:70 tgid:70 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 369.984606][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 369.984794][ T44] Call Trace: [ 369.984900][ T44] [ 369.984974][ T44] __schedule+0x88d/0x1b90 [ 369.985115][ T44] ? io_schedule_timeout+0x130/0x130 [ 369.985248][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.985378][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.985509][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.985642][ T44] schedule+0xd4/0x260 [ 369.985747][ T44] schedule_preempt_disabled+0x18/0x30 [ 369.985876][ T44] __mutex_lock+0x9ae/0x1500 [ 369.986009][ T44] ? addrconf_verify_work+0x12/0x30 [ 369.986137][ T44] ? ww_mutex_lock+0x160/0x160 [ 369.986261][ T44] ? process_one_work+0xd16/0x1390 [ 369.986386][ T44] ? trace_lock_acquire+0x88/0xd0 [ 369.986510][ T44] ? trace_lock_acquire+0x88/0xd0 [ 369.986635][ T44] ? addrconf_verify_work+0x12/0x30 [ 369.986771][ T44] addrconf_verify_work+0x12/0x30 [ 369.986901][ T44] process_one_work+0xd57/0x1390 [ 369.987024][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 369.987142][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.987263][ T44] ? assign_work+0x152/0x380 [ 369.987384][ T44] worker_thread+0x4d6/0xd40 [ 369.987507][ T44] ? process_one_work+0x1390/0x1390 [ 369.987629][ T44] ? __kthread_parkme+0xb3/0x200 [ 369.987762][ T44] ? process_one_work+0x1390/0x1390 [ 369.987881][ T44] kthread+0x355/0x5b0 [ 369.987970][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 369.988087][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.988205][ T44] ? rcu_is_watching+0x15/0xd0 [ 369.988326][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 369.988444][ T44] ret_from_fork+0x1d4/0x270 [ 369.988570][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 369.988700][ T44] ret_from_fork_asm+0x11/0x20 [ 369.988826][ T44] [ 369.988940][ T44] INFO: task kworker/u16:1:70 is blocked on a mutex likely owned by task ip:256. [ 369.989144][ T44] INFO: task ip:256 blocked for more than 245 seconds. [ 369.989281][ T44] Not tainted 6.18.0-rc7-virtme #1 [ 369.989391][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 369.989595][ T44] task:ip state:D stack:25496 pid:256 tgid:256 ppid:251 task_flags:0x400100 flags:0x00080001 [ 369.989864][ T44] Call Trace: [ 369.989951][ T44] [ 369.990012][ T44] __schedule+0x88d/0x1b90 [ 369.990132][ T44] ? io_schedule_timeout+0x130/0x130 [ 369.990251][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.990375][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.990495][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.990615][ T44] schedule+0xd4/0x260 [ 369.990714][ T44] schedule_preempt_disabled+0x18/0x30 [ 369.990831][ T44] __mutex_lock+0x9ae/0x1500 [ 369.990954][ T44] ? napi_enable+0x3b/0x80 [ 369.991076][ T44] ? virtqueue_add_inbuf_ctx+0x1850/0x1850 [ 369.991229][ T44] ? ww_mutex_lock+0x160/0x160 [ 369.991348][ T44] ? __asan_memset+0x27/0x50 [ 369.991468][ T44] ? sg_init_table+0x19/0x60 [ 369.991593][ T44] ? try_fill_recv+0x3ec/0x1750 [ 369.991723][ T44] ? napi_enable+0x3b/0x80 [ 369.991841][ T44] napi_enable+0x3b/0x80 [ 369.991930][ T44] __virtnet_rx_resume+0x156/0x290 [ 369.992051][ T44] ? queue_delayed_work_on+0x91/0x100 [ 369.992169][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 369.992289][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 369.992413][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 369.992535][ T44] dev_xdp_install+0x3e4/0x8e0 [ 369.992664][ T44] ? netif_skb_check_for_xdp+0x600/0x600 [ 369.992793][ T44] dev_xdp_attach+0x50d/0x1320 [ 369.992920][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 369.993036][ T44] ? __bpf_prog_get+0x320/0x320 [ 369.993157][ T44] dev_change_xdp_fd+0x182/0x240 [ 369.993276][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 369.993397][ T44] ? rtnl_newlink_create+0x900/0x900 [ 369.993523][ T44] ? rcu_lockdep_current_cpu_online+0x3e/0x1b0 [ 369.993670][ T44] ? rcu_read_lock_any_held+0x41/0xa0 [ 369.993796][ T44] ? validate_chain+0x287/0x5f0 [ 369.993912][ T44] ? rcu_read_lock_any_held+0x6b/0xa0 [ 369.994027][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 369.994145][ T44] ? __lock_acquire+0x34c/0x740 [ 369.994269][ T44] ? trace_lock_acquire+0x88/0xd0 [ 369.994385][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 369.994508][ T44] ? lock_acquire+0x31/0x80 [ 369.994629][ T44] ? rtnl_newlink+0x5df/0xb20 [ 369.994755][ T44] ? rcu_is_watching+0x15/0xd0 [ 369.994872][ T44] ? rcu_is_watching+0x15/0xd0 [ 369.994987][ T44] ? trace_contention_end+0xd6/0x140 [ 369.995107][ T44] ? rtnl_newlink+0x5df/0xb20 [ 369.995223][ T44] ? __mutex_lock+0x32e/0x1500 [ 369.995343][ T44] ? ww_mutex_lock+0x160/0x160 [ 369.995459][ T44] ? nla_validate_int_range+0x5b0/0x5b0 [ 369.995588][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 369.995718][ T44] rtnl_newlink+0x637/0xb20 [ 369.995840][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 369.995962][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.996081][ T44] ? find_held_lock+0x2b/0x80 [ 369.996199][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 369.996323][ T44] ? __lock_release.isra.0+0x59/0x170 [ 369.996443][ T44] ? __rtnl_newlink+0xa40/0xa40 [ 369.996574][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 369.996704][ T44] ? rtnl_port_fill+0x940/0x940 [ 369.996823][ T44] ? __lock_acquire+0x34c/0x740 [ 369.996945][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 369.997062][ T44] ? find_held_lock+0x2b/0x80 [ 369.997183][ T44] netlink_rcv_skb+0x123/0x380 [ 369.997307][ T44] ? rtnl_port_fill+0x940/0x940 [ 369.997425][ T44] ? netlink_ack+0xcc0/0xcc0 [ 369.997556][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 369.997692][ T44] netlink_unicast+0x4a3/0x770 [ 369.997815][ T44] ? netlink_attachskb+0x810/0x810 [ 369.997938][ T44] ? __lock_acquire+0x34c/0x740 [ 369.998057][ T44] netlink_sendmsg+0x735/0xc60 [ 369.998178][ T44] ? netlink_unicast+0x770/0x770 [ 369.998304][ T44] ? __might_fault+0x97/0x140 [ 369.998430][ T44] ____sys_sendmsg+0x419/0x850 [ 369.998555][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 369.998684][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 369.998831][ T44] ? move_addr_to_kernel+0x40/0x40 [ 369.998952][ T44] ? stack_depot_save_flags+0x3e1/0x6e0 [ 369.999068][ T44] ? validate_chain+0x287/0x5f0 [ 369.999189][ T44] ___sys_sendmsg+0xfd/0x180 [ 369.999315][ T44] ? __call_rcu_common.constprop.0+0xa5/0x680 [ 369.999470][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 369.999607][ T44] __sys_sendmsg+0x124/0x1c0 [ 369.999734][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 369.999852][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 369.999967][ T44] ? __call_rcu_common.constprop.0+0x2fa/0x680 [ 370.000118][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 370.000243][ T44] ? rcu_is_watching+0x15/0xd0 [ 370.000360][ T44] do_syscall_64+0xbb/0xfd0 [ 370.000480][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 370.000637][ T44] RIP: 0033:0x7fab2d6d9c5e [ 370.000774][ T44] RSP: 002b:00007ffc268c3ed0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 370.000967][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fab2d6d9c5e [ 370.001155][ T44] RDX: 0000000000000000 RSI: 00007ffc268c3f90 RDI: 0000000000000003 [ 370.001334][ T44] RBP: 00007ffc268c3ee0 R08: 0000000000000000 R09: 0000000000000000 [ 370.001516][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc268c4650 [ 370.001703][ T44] R13: 000000006930d0d2 R14: 0000000000000000 R15: 0000000000499600 [ 370.001889][ T44] [ 370.001986][ T44] INFO: task ip:256 is blocked on a mutex likely owned by task kworker/2:2:64. [ 370.002186][ T44] task:kworker/2:2 state:R running task stack:28160 pid:64 tgid:64 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 370.002497][ T44] Workqueue: events refill_work [ 370.002624][ T44] Call Trace: [ 370.002731][ T44] [ 370.002798][ T44] __schedule+0x88d/0x1b90 [ 370.002923][ T44] ? _raw_spin_lock_irqsave+0x10/0x60 [ 370.003043][ T44] ? io_schedule_timeout+0x130/0x130 [ 370.003165][ T44] ? usleep_range_state+0xba/0x1b0 [ 370.003286][ T44] ? find_held_lock+0x2b/0x80 [ 370.003408][ T44] ? __lock_release.isra.0+0x59/0x170 [ 370.003532][ T44] ? usleep_range_state+0xba/0x1b0 [ 370.003653][ T44] schedule+0xd4/0x260 [ 370.003757][ T44] ? schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 370.003911][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 370.004037][ T44] ? hrtimer_cb_get_time+0x90/0x90 [ 370.004159][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 370.004280][ T44] ? ktime_get+0x15e/0x220 [ 370.004398][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 370.004525][ T44] ? ktime_get+0xa9/0x220 [ 370.004616][ T44] ? usleep_range_state+0x133/0x1b0 [ 370.004745][ T44] ? schedule_hrtimeout+0x20/0x20 [ 370.004865][ T44] ? __free_object+0x2ad/0x4b0 [ 370.004986][ T44] ? napi_disable_locked+0x379/0x5f0 [ 370.005104][ T44] ? _mutex_trylock_nest_lock+0x150/0x150 [ 370.005224][ T44] ? netdev_bind_sb_channel_queue+0x470/0x470 [ 370.005382][ T44] ? napi_disable+0x43/0x80 [ 370.005507][ T44] ? refill_work+0x116/0x210 [ 370.005625][ T44] ? process_one_work+0xd57/0x1390 [ 370.005758][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 370.005876][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 370.005995][ T44] ? assign_work+0x152/0x380 [ 370.006112][ T44] ? worker_thread+0x4d6/0xd40 [ 370.006234][ T44] ? process_one_work+0x1390/0x1390 [ 370.006353][ T44] ? __kthread_parkme+0xb3/0x200 [ 370.006470][ T44] ? process_one_work+0x1390/0x1390 [ 370.006594][ T44] ? kthread+0x355/0x5b0 [ 370.006699][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 370.006816][ T44] ? __lock_release.isra.0+0x59/0x170 [ 370.006932][ T44] ? rcu_is_watching+0x15/0xd0 [ 370.007053][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 370.007171][ T44] ? ret_from_fork+0x1d4/0x270 [ 370.007291][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 370.007408][ T44] ? ret_from_fork_asm+0x11/0x20 [ 370.007538][ T44] [ 370.007628][ T44] [ 370.007628][ T44] Showing all locks held in the system: [ 370.007817][ T44] 1 lock held by khungtaskd/44: [ 370.007932][ T44] #0: ffffffffb7778fc0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x2b/0x1db [ 370.008144][ T44] 3 locks held by kworker/2:2/64: [ 370.008259][ T44] 3 locks held by kworker/u16:1/70: [ 370.008372][ T44] #0: ff11000008b6cd48 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 370.008605][ T44] #1: ffa00000004b7d30 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 370.008901][ T44] #2: ffffffffb7e925e8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 370.009112][ T44] 2 locks held by ip/256: [ 370.009198][ T44] #0: ffffffffb7e925e8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x5df/0xb20 [ 370.009398][ T44] #1: ff110000085b2c98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 370.009607][ T44] [ 370.009669][ T44] ============================================= [ 370.009669][ T44] [ 370.495504][ C0] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 0: transmit queue 1 timed out 330239 ms [ 370.495944][ C0] virtio_net virtio2 enp0s1: TX timeout on queue: 1, sq: output.1, vq: 0x3, name: output.1, 330239000 usecs ago