[ 727.424742][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 5120 ms [ 727.425259][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 5120000 usecs ago [ 733.568737][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 11264 ms [ 733.569077][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 11264000 usecs ago [ 738.688783][ T44] INFO: task ip:5738 blocked for more than 122 seconds. [ 738.689059][ T44] Not tainted 6.19.0-rc3-virtme #1 [ 738.689177][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 738.689392][ T44] task:ip state:D stack:25312 pid:5738 tgid:5738 ppid:5733 task_flags:0x400100 flags:0x00080000 [ 738.689663][ T44] Call Trace: [ 738.689772][ T44] [ 738.689840][ T44] __schedule+0xae2/0x1720 [ 738.689968][ T44] ? io_schedule_timeout+0x130/0x130 [ 738.690088][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.690219][ T44] schedule+0xd4/0x260 [ 738.690310][ T44] schedule_preempt_disabled+0x18/0x30 [ 738.690433][ T44] __mutex_lock+0xd76/0x1f20 [ 738.690552][ T44] ? napi_enable+0x3b/0x80 [ 738.690672][ T44] ? ww_mutex_lock+0x160/0x160 [ 738.690796][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.690918][ T44] ? __asan_memset+0x27/0x50 [ 738.691040][ T44] ? sg_init_table+0x19/0x60 [ 738.691160][ T44] ? try_fill_recv+0x3ec/0x1750 [ 738.691289][ T44] ? napi_enable+0x3b/0x80 [ 738.691411][ T44] napi_enable+0x3b/0x80 [ 738.691501][ T44] __virtnet_rx_resume+0x156/0x290 [ 738.691620][ T44] ? queue_delayed_work_on+0x91/0x100 [ 738.691754][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 738.691875][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 738.692001][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 738.692121][ T44] dev_xdp_install+0x3e4/0x890 [ 738.692240][ T44] ? netif_set_xps_queue+0x40/0x40 [ 738.692361][ T44] dev_xdp_attach+0x50d/0x1310 [ 738.692493][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 738.692615][ T44] ? __bpf_prog_get+0x320/0x320 [ 738.693089][ T44] dev_change_xdp_fd+0x176/0x210 [ 738.693209][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 738.693334][ T44] ? rtnl_newlink_create+0x900/0x900 [ 738.693460][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 738.693606][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 738.693724][ T44] ? validate_chain+0x37b/0xb70 [ 738.693849][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 738.693966][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 738.694088][ T44] ? __lock_acquire+0x577/0xc10 [ 738.694207][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.694323][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 738.694445][ T44] ? lock_acquire+0x10a/0x150 [ 738.694565][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.694680][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.694807][ T44] ? trace_contention_end+0xd6/0x110 [ 738.694925][ T44] ? __mutex_lock+0x285/0x1f20 [ 738.695050][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 738.695167][ T44] ? ww_mutex_lock+0x160/0x160 [ 738.695286][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 738.695412][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 738.695535][ T44] rtnl_newlink+0x75c/0xe90 [ 738.695655][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.695778][ T44] ? rtnl_setlink+0x880/0x880 [ 738.695894][ T44] ? __lock_acquire+0x577/0xc10 [ 738.696013][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.696130][ T44] ? find_held_lock+0x2b/0x80 [ 738.696245][ T44] ? rtnl_setlink+0x880/0x880 [ 738.696364][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.696486][ T44] ? rtnl_setlink+0x880/0x880 [ 738.696602][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 738.696720][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 738.696839][ T44] ? __lock_acquire+0x577/0xc10 [ 738.696956][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.697075][ T44] ? find_held_lock+0x2b/0x80 [ 738.697196][ T44] netlink_rcv_skb+0x123/0x380 [ 738.697318][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 738.697440][ T44] ? netlink_ack+0xcc0/0xcc0 [ 738.697565][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 738.697687][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 738.697812][ T44] netlink_unicast+0x4a3/0x770 [ 738.697937][ T44] ? netlink_attachskb+0x810/0x810 [ 738.698057][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.698176][ T44] ? __lock_acquire+0x577/0xc10 [ 738.698294][ T44] netlink_sendmsg+0x735/0xc60 [ 738.698413][ T44] ? netlink_unicast+0x770/0x770 [ 738.698539][ T44] ____sys_sendmsg+0x419/0x850 [ 738.698661][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 738.698790][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 738.698935][ T44] ? move_addr_to_kernel+0x40/0x40 [ 738.699056][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 738.699171][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 738.699288][ T44] ___sys_sendmsg+0xfd/0x180 [ 738.699432][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 738.699584][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 738.699717][ T44] __sys_sendmsg+0x124/0x1c0 [ 738.699839][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 738.699957][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 738.700076][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 738.700222][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 738.700339][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.700461][ T44] do_syscall_64+0xbd/0xfc0 [ 738.700583][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 738.700735][ T44] RIP: 0033:0x7f25030edc5e [ 738.700863][ T44] RSP: 002b:00007ffd3bdc40e0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 738.701050][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f25030edc5e [ 738.701225][ T44] RDX: 0000000000000000 RSI: 00007ffd3bdc41a0 RDI: 0000000000000003 [ 738.701410][ T44] RBP: 00007ffd3bdc40f0 R08: 0000000000000000 R09: 0000000000000000 [ 738.701585][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffd3bdc4860 [ 738.701765][ T44] R13: 000000006957102e R14: 0000000000000000 R15: 0000000000499600 [ 738.701944][ T44] [ 738.702040][ T44] INFO: task ip:5738 is blocked on a mutex likely owned by task kworker/2:1:51. [ 738.702260][ T44] task:kworker/2:1 state:R running task stack:27088 pid:51 tgid:51 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 738.702544][ T44] Workqueue: events refill_work [ 738.702661][ T44] Call Trace: [ 738.702756][ T44] [ 738.702818][ T44] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 738.702963][ T44] ? __schedule+0xae2/0x1720 [ 738.703085][ T44] ? usleep_range_state+0xba/0x1b0 [ 738.703201][ T44] ? io_schedule_timeout+0x130/0x130 [ 738.703317][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.703436][ T44] ? _raw_spin_unlock_irqrestore+0x53/0x80 [ 738.703581][ T44] ? usleep_range_state+0xba/0x1b0 [ 738.703696][ T44] ? schedule+0xd4/0x260 [ 738.703793][ T44] ? schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 738.703936][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 738.704059][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 738.704176][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 738.704292][ T44] ? ktime_get+0x15e/0x220 [ 738.704413][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 738.704529][ T44] ? ktime_get+0xa9/0x220 [ 738.704621][ T44] ? usleep_range_state+0x133/0x1b0 [ 738.704742][ T44] ? schedule_hrtimeout+0x20/0x20 [ 738.704859][ T44] ? perf_trace_sched_switch+0x6d0/0x6d0 [ 738.704976][ T44] ? refill_work+0x149/0x210 [ 738.705097][ T44] ? napi_disable_locked+0x379/0x5f0 [ 738.705212][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 738.705328][ T44] ? register_netdevice_notifier_net+0x40/0x40 [ 738.705483][ T44] ? napi_disable+0x43/0x80 [ 738.705599][ T44] ? refill_work+0x116/0x210 [ 738.705718][ T44] ? process_one_work+0xd57/0x1390 [ 738.705842][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 738.705958][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 738.706080][ T44] ? assign_work+0x152/0x380 [ 738.706200][ T44] ? worker_thread+0x4d6/0xd40 [ 738.706320][ T44] ? process_one_work+0x1390/0x1390 [ 738.706438][ T44] ? __kthread_parkme+0xb3/0x200 [ 738.706556][ T44] ? process_one_work+0x1390/0x1390 [ 738.706673][ T44] ? kthread+0x355/0x5b0 [ 738.706766][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.706881][ T44] ? __lock_release.isra.0+0x59/0x170 [ 738.706998][ T44] ? rcu_is_watching+0x15/0xd0 [ 738.707119][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.707235][ T44] ? ret_from_fork+0x3fb/0x510 [ 738.707352][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 738.707503][ T44] ? __switch_to+0x53c/0xd00 [ 738.707619][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 738.707740][ T44] ? ret_from_fork_asm+0x11/0x20 [ 738.707862][ T44] [ 738.707951][ T44] [ 738.707951][ T44] Showing all locks held in the system: [ 738.708130][ T44] 3 locks held by kworker/u16:0/12: [ 738.708245][ T44] #0: ff1100000918f148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 738.708478][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 738.708764][ T44] #2: ffffffffae4859a8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 738.708972][ T44] 1 lock held by khungtaskd/44: [ 738.709083][ T44] #0: ffffffffadd710c0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 738.709310][ T44] 3 locks held by kworker/2:1/51: [ 738.709436][ T44] 2 locks held by ip/5738: [ 738.709546][ T44] #0: ffffffffae4859a8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 738.709753][ T44] #1: ff11000008b2ac98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 738.709950][ T44] [ 738.710008][ T44] ============================================= [ 738.710008][ T44] [ 739.200737][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 16896 ms [ 739.201063][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 16896000 usecs ago [ 744.320737][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 22016 ms [ 744.321098][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 22016000 usecs ago [ 749.440736][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 27136 ms [ 749.441086][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 27136000 usecs ago [ 755.583743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 33279 ms [ 755.584174][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 33279000 usecs ago [ 761.215738][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 38911 ms [ 761.216104][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 38911000 usecs ago [ 766.336738][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 44032 ms [ 766.337123][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 44032000 usecs ago [ 771.455746][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 49151 ms [ 771.456175][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 49151000 usecs ago [ 777.600742][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 55296 ms [ 777.601183][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 55296000 usecs ago [ 783.232750][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 60928 ms [ 783.233120][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 60928000 usecs ago [ 788.352747][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 66048 ms [ 788.353155][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 66048000 usecs ago [ 793.472751][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 71168 ms [ 793.473184][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 71168000 usecs ago [ 799.616741][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 77312 ms [ 799.617122][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 77312000 usecs ago [ 805.247745][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 82943 ms [ 805.248229][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 82943000 usecs ago [ 810.368741][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 88064 ms [ 810.369155][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 88064000 usecs ago [ 816.512745][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 94208 ms [ 816.513174][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 94208000 usecs ago [ 822.144741][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 99840 ms [ 822.145154][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 99840000 usecs ago [ 827.264748][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 104960 ms [ 827.265221][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 104960000 usecs ago [ 832.384742][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 110080 ms [ 832.385242][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 110080000 usecs ago [ 838.528743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 116224 ms [ 838.529218][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 116224000 usecs ago [ 844.160743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 121856 ms [ 844.161157][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 121856000 usecs ago [ 849.280738][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 126976 ms [ 849.281148][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 126976000 usecs ago [ 854.400738][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 132096 ms [ 854.401141][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 132096000 usecs ago [ 860.544740][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 138240 ms [ 860.545180][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 138240000 usecs ago [ 861.568744][ T44] INFO: task kworker/u16:0:12 blocked for more than 122 seconds. [ 861.568978][ T44] Not tainted 6.19.0-rc3-virtme #1 [ 861.569102][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 861.569316][ T44] task:kworker/u16:0 state:D stack:26128 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 861.569600][ T44] Workqueue: ipv6_addrconf addrconf_verify_work [ 861.569781][ T44] Call Trace: [ 861.569884][ T44] [ 861.569950][ T44] __schedule+0xae2/0x1720 [ 861.570078][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.570205][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.570333][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.570460][ T44] schedule+0xd4/0x260 [ 861.570557][ T44] schedule_preempt_disabled+0x18/0x30 [ 861.570680][ T44] __mutex_lock+0xd76/0x1f20 [ 861.570813][ T44] ? addrconf_verify_work+0x12/0x30 [ 861.570932][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.571054][ T44] ? process_one_work+0xd16/0x1390 [ 861.571174][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.571289][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.571405][ T44] ? addrconf_verify_work+0x12/0x30 [ 861.571519][ T44] addrconf_verify_work+0x12/0x30 [ 861.571638][ T44] process_one_work+0xd57/0x1390 [ 861.571761][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 861.571875][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.572004][ T44] ? assign_work+0x152/0x380 [ 861.572123][ T44] worker_thread+0x4d6/0xd40 [ 861.572246][ T44] ? process_one_work+0x1390/0x1390 [ 861.572365][ T44] kthread+0x355/0x5b0 [ 861.572453][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.572565][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.572680][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.572804][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.572925][ T44] ret_from_fork+0x3fb/0x510 [ 861.573046][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 861.573194][ T44] ? __switch_to+0x53c/0xd00 [ 861.573310][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.573424][ T44] ret_from_fork_asm+0x11/0x20 [ 861.573546][ T44] [ 861.573665][ T44] INFO: task kworker/u16:0:12 is blocked on a mutex likely owned by task ip:5738. [ 861.573878][ T44] INFO: task ip:5738 blocked for more than 245 seconds. [ 861.574017][ T44] Not tainted 6.19.0-rc3-virtme #1 [ 861.574126][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 861.574313][ T44] task:ip state:D stack:25312 pid:5738 tgid:5738 ppid:5733 task_flags:0x400100 flags:0x00080000 [ 861.574569][ T44] Call Trace: [ 861.574655][ T44] [ 861.574713][ T44] __schedule+0xae2/0x1720 [ 861.574835][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.574955][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.575074][ T44] schedule+0xd4/0x260 [ 861.575162][ T44] schedule_preempt_disabled+0x18/0x30 [ 861.575275][ T44] __mutex_lock+0xd76/0x1f20 [ 861.575393][ T44] ? napi_enable+0x3b/0x80 [ 861.575509][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.575624][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.575741][ T44] ? __asan_memset+0x27/0x50 [ 861.575859][ T44] ? sg_init_table+0x19/0x60 [ 861.575982][ T44] ? try_fill_recv+0x3ec/0x1750 [ 861.576106][ T44] ? napi_enable+0x3b/0x80 [ 861.576220][ T44] napi_enable+0x3b/0x80 [ 861.576311][ T44] __virtnet_rx_resume+0x156/0x290 [ 861.576425][ T44] ? queue_delayed_work_on+0x91/0x100 [ 861.576540][ T44] virtnet_rx_resume_all+0xe7/0x1a0 [ 861.576654][ T44] virtnet_xdp_set+0xef7/0x1aa0 [ 861.576776][ T44] ? virtnet_xdp_set+0x1aa0/0x1aa0 [ 861.576892][ T44] dev_xdp_install+0x3e4/0x890 [ 861.577008][ T44] ? netif_set_xps_queue+0x40/0x40 [ 861.577145][ T44] dev_xdp_attach+0x50d/0x1310 [ 861.577262][ T44] ? bpf_xdp_link_update+0x6e0/0x6e0 [ 861.577376][ T44] ? __bpf_prog_get+0x320/0x320 [ 861.577497][ T44] dev_change_xdp_fd+0x176/0x210 [ 861.577613][ T44] do_setlink.isra.0+0x1c2d/0x2880 [ 861.577736][ T44] ? rtnl_newlink_create+0x900/0x900 [ 861.577855][ T44] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 861.578007][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 861.578124][ T44] ? validate_chain+0x37b/0xb70 [ 861.578243][ T44] ? rcu_read_lock_any_held+0x66/0x90 [ 861.578357][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 861.578471][ T44] ? __lock_acquire+0x577/0xc10 [ 861.578588][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.578701][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 861.578821][ T44] ? lock_acquire+0x10a/0x150 [ 861.578940][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.579055][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.579169][ T44] ? trace_contention_end+0xd6/0x110 [ 861.579284][ T44] ? __mutex_lock+0x285/0x1f20 [ 861.579399][ T44] ? rtnl_newlink+0x6fc/0xe90 [ 861.579514][ T44] ? ww_mutex_lock+0x160/0x160 [ 861.579630][ T44] ? nla_get_range_signed+0x3d0/0x3d0 [ 861.579757][ T44] ? __rtnl_newlink+0x3fa/0xa40 [ 861.579874][ T44] rtnl_newlink+0x75c/0xe90 [ 861.579995][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.580111][ T44] ? rtnl_setlink+0x880/0x880 [ 861.580225][ T44] ? __lock_acquire+0x577/0xc10 [ 861.580340][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.580452][ T44] ? find_held_lock+0x2b/0x80 [ 861.580572][ T44] ? rtnl_setlink+0x880/0x880 [ 861.580685][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.580803][ T44] ? rtnl_setlink+0x880/0x880 [ 861.580921][ T44] rtnetlink_rcv_msg+0x6fe/0xb90 [ 861.581041][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 861.581153][ T44] ? __lock_acquire+0x577/0xc10 [ 861.581270][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.581381][ T44] ? find_held_lock+0x2b/0x80 [ 861.581500][ T44] netlink_rcv_skb+0x123/0x380 [ 861.581616][ T44] ? rtnl_fdb_dump+0x620/0x620 [ 861.581736][ T44] ? netlink_ack+0xcc0/0xcc0 [ 861.581859][ T44] ? netlink_deliver_tap+0xc5/0x330 [ 861.581975][ T44] ? netlink_deliver_tap+0x13f/0x330 [ 861.582095][ T44] netlink_unicast+0x4a3/0x770 [ 861.582209][ T44] ? netlink_attachskb+0x810/0x810 [ 861.582329][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.582444][ T44] ? __lock_acquire+0x577/0xc10 [ 861.582564][ T44] netlink_sendmsg+0x735/0xc60 [ 861.582681][ T44] ? netlink_unicast+0x770/0x770 [ 861.582803][ T44] ____sys_sendmsg+0x419/0x850 [ 861.582921][ T44] ? copy_msghdr_from_user+0x270/0x430 [ 861.583040][ T44] ? get_timestamp.constprop.0+0x3a0/0x3a0 [ 861.583185][ T44] ? move_addr_to_kernel+0x40/0x40 [ 861.583303][ T44] ? stack_depot_save_flags+0x3f1/0x6f0 [ 861.583416][ T44] ? rcu_read_lock_any_held+0x3c/0x90 [ 861.583534][ T44] ___sys_sendmsg+0xfd/0x180 [ 861.583650][ T44] ? __call_rcu_common.constprop.0+0xa6/0xa00 [ 861.583798][ T44] ? copy_msghdr_from_user+0x430/0x430 [ 861.583931][ T44] __sys_sendmsg+0x124/0x1c0 [ 861.584054][ T44] ? __sys_sendmsg_sock+0x20/0x20 [ 861.584168][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.584282][ T44] ? __call_rcu_common.constprop.0+0x3bd/0xa00 [ 861.584429][ T44] ? do_raw_spin_unlock+0x59/0x250 [ 861.584543][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.584658][ T44] do_syscall_64+0xbd/0xfc0 [ 861.584779][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 861.584930][ T44] RIP: 0033:0x7f25030edc5e [ 861.585056][ T44] RSP: 002b:00007ffd3bdc40e0 EFLAGS: 00000202 ORIG_RAX: 000000000000002e [ 861.585231][ T44] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f25030edc5e [ 861.585402][ T44] RDX: 0000000000000000 RSI: 00007ffd3bdc41a0 RDI: 0000000000000003 [ 861.585574][ T44] RBP: 00007ffd3bdc40f0 R08: 0000000000000000 R09: 0000000000000000 [ 861.585755][ T44] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffd3bdc4860 [ 861.585930][ T44] R13: 000000006957102e R14: 0000000000000000 R15: 0000000000499600 [ 861.586111][ T44] [ 861.586200][ T44] INFO: task ip:5738 is blocked on a mutex likely owned by task kworker/2:1:51. [ 861.586392][ T44] task:kworker/2:1 state:R running task stack:27088 pid:51 tgid:51 ppid:2 task_flags:0x4208060 flags:0x00080000 [ 861.586670][ T44] Workqueue: events refill_work [ 861.586788][ T44] Call Trace: [ 861.586883][ T44] [ 861.586943][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.587061][ T44] ? __schedule+0xae2/0x1720 [ 861.587176][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.587297][ T44] ? io_schedule_timeout+0x130/0x130 [ 861.587415][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.587531][ T44] ? usleep_range_state+0xba/0x1b0 [ 861.587645][ T44] ? schedule+0xd4/0x260 [ 861.587736][ T44] ? schedule_hrtimeout_range_clock+0x1da/0x2f0 [ 861.587885][ T44] ? schedule_timeout_idle+0xe0/0xe0 [ 861.588001][ T44] ? __hrtimer_cb_get_time+0x50/0x50 [ 861.588126][ T44] ? lockdep_hardirqs_on+0x84/0x130 [ 861.588241][ T44] ? ktime_get+0x15e/0x220 [ 861.588356][ T44] ? kvm_clock_get_cycles+0x18/0x30 [ 861.588470][ T44] ? ktime_get+0xa9/0x220 [ 861.588556][ T44] ? irqentry_exit+0xf9/0x4d0 [ 861.588672][ T44] ? usleep_range_state+0x133/0x1b0 [ 861.588792][ T44] ? schedule_hrtimeout+0x20/0x20 [ 861.588924][ T44] ? __free_object+0x2ad/0x4b0 [ 861.589043][ T44] ? napi_disable_locked+0x379/0x5f0 [ 861.589157][ T44] ? _mutex_trylock_nest_lock+0x340/0x340 [ 861.589274][ T44] ? register_netdevice_notifier_net+0x40/0x40 [ 861.589418][ T44] ? napi_disable+0x43/0x80 [ 861.589532][ T44] ? refill_work+0x116/0x210 [ 861.589648][ T44] ? process_one_work+0xd57/0x1390 [ 861.589773][ T44] ? pwq_dec_nr_in_flight+0x700/0x700 [ 861.589892][ T44] ? lock_acquire.part.0+0xbc/0x260 [ 861.590010][ T44] ? assign_work+0x152/0x380 [ 861.590133][ T44] ? worker_thread+0x4d6/0xd40 [ 861.590251][ T44] ? process_one_work+0x1390/0x1390 [ 861.590364][ T44] ? __kthread_parkme+0xb3/0x200 [ 861.590481][ T44] ? process_one_work+0x1390/0x1390 [ 861.590598][ T44] ? kthread+0x355/0x5b0 [ 861.590685][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.590804][ T44] ? __lock_release.isra.0+0x59/0x170 [ 861.590920][ T44] ? rcu_is_watching+0x15/0xd0 [ 861.591037][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.591152][ T44] ? ret_from_fork+0x3fb/0x510 [ 861.591269][ T44] ? arch_exit_to_user_mode_prepare.isra.0+0x140/0x140 [ 861.591415][ T44] ? __switch_to+0x53c/0xd00 [ 861.591527][ T44] ? kthread_is_per_cpu+0xe0/0xe0 [ 861.591642][ T44] ? ret_from_fork_asm+0x11/0x20 [ 861.591771][ T44] [ 861.591857][ T44] [ 861.591857][ T44] Showing all locks held in the system: [ 861.592033][ T44] 3 locks held by kworker/u16:0/12: [ 861.592145][ T44] #0: ff1100000918f148 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.592374][ T44] #1: ffa00000000c7cc0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.592650][ T44] #2: ffffffffae4859a8 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_verify_work+0x12/0x30 [ 861.592858][ T44] 1 lock held by khungtaskd/44: [ 861.592972][ T44] #0: ffffffffadd710c0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 861.593200][ T44] 3 locks held by kworker/2:1/51: [ 861.593313][ T44] #0: ff11000001139b48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xcb4/0x1390 [ 861.593534][ T44] #1: ffa000000037fcc0 ((work_completion)(&(&vi->refill)->work)){+.+.}-{0:0}, at: process_one_work+0xd16/0x1390 [ 861.593794][ T44] #2: ff11000008b2ac98 (&dev->lock){+.+.}-{4:4}, at: napi_disable+0x3b/0x80 [ 861.594003][ T44] 2 locks held by ip/5738: [ 861.594115][ T44] #0: ffffffffae4859a8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x6fc/0xe90 [ 861.594311][ T44] #1: ff11000008b2ac98 (&dev->lock){+.+.}-{4:4}, at: napi_enable+0x3b/0x80 [ 861.594505][ T44] [ 861.594563][ T44] ============================================= [ 861.594563][ T44] [ 866.176742][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 143872 ms [ 866.177205][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 143872000 usecs ago [ 871.296742][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 148992 ms [ 871.297198][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 148992000 usecs ago [ 876.415741][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 154111 ms [ 876.416174][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 154111000 usecs ago [ 882.560744][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 160256 ms [ 882.561281][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 160256000 usecs ago [ 888.192743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 165888 ms [ 888.193216][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 165888000 usecs ago [ 893.312742][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 171008 ms [ 893.313194][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 171008000 usecs ago [ 898.431763][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 176127 ms [ 898.432262][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 176127000 usecs ago [ 904.576744][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 182272 ms [ 904.577201][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 182272000 usecs ago [ 910.208743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 187904 ms [ 910.209259][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 187904000 usecs ago [ 915.328743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 193024 ms [ 915.329204][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 193024000 usecs ago [ 920.448744][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 198144 ms [ 920.449217][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 198144000 usecs ago [ 926.592741][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 204288 ms [ 926.593157][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 204288000 usecs ago [ 932.224737][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 209920 ms [ 932.225150][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 209920000 usecs ago [ 937.344737][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 215040 ms [ 937.345159][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 215040000 usecs ago [ 942.464737][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 220160 ms [ 942.465140][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 220160000 usecs ago [ 948.608745][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 226304 ms [ 948.609216][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 226304000 usecs ago [ 954.240743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 231936 ms [ 954.241205][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 231936000 usecs ago [ 959.360740][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 237056 ms [ 959.361187][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 237056000 usecs ago [ 964.480743][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 242176 ms [ 964.481241][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 242176000 usecs ago [ 970.642782][ C2] virtio_net virtio2 enp0s1: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 248338 ms [ 970.745539][ C2] virtio_net virtio2 enp0s1: TX timeout on queue: 3, sq: output.3, vq: 0x7, name: output.3, 248440000 usecs ago