Oct 22 12:24:30 hw1 systemd[1]: Starting nipa-hw-worker.service - NIPA HW worker (one-shot test runner)...
Oct 22 12:24:30 hw1 python3[1162]: /srv/hw-worker/tests/5200 Starting tests
Oct 22 12:24:30 hw1 python3[1162]: Set ens1f0np0 combined channels to 32 (max=32, cpus=32)
Oct 22 12:24:30 hw1 python3[1162]: IRQ mapping: IRQ 186 -> CPU 0 ... IRQ 193 -> CPU 7
Oct 22 12:24:30 hw1 python3[1162]: Restored ens1f0np0 combined channels to 8
Oct 22 12:24:30 hw1 python3[1162]: Moved ens1f1np1 to netns nipa-peer
Oct 22 12:24:31 hw1 python3[1162]: Peer 10.88.0.2 reachable after 1s
Oct 22 12:24:33 hw1 python3[1162]: Peer fd00::2 reachable after 2s
Oct 22 12:24:33 hw1 python3[1162]: Wrote /srv/hw-worker/tests/5200/drivers/net/net.config
Oct 22 12:24:33 hw1 python3[1162]: Wrote /srv/hw-worker/tests/5200/drivers/net/hw/net.config
Oct 22 12:24:33 hw1 python3[1162]: Collected device info for ens1f0np0
Oct 22 12:24:33 hw1 python3[1162]: Found 53 tests, per-test timeout 1200s
Oct 22 12:24:33 hw1 python3[1162]: Saved 2893 lines of boot dmesg
Oct 22 12:24:33 hw1 python3[1162]: [1/53] Running drivers/net:gro_hw.py
Oct 22 12:24:33 hw1 python3[1162]: [1/53] drivers/net:gro_hw.py: rc=0 (0.6s)
Oct 22 12:24:33 hw1 python3[1162]: [2/53] Running drivers/net:gro_lro.py
Oct 22 12:24:34 hw1 python3[1162]: [2/53] drivers/net:gro_lro.py: rc=0 (0.6s)
Oct 22 12:24:34 hw1 python3[1162]: [3/53] Running drivers/net:gro_sw.py
Oct 22 12:24:46 hw1 python3[1162]: [3/53] drivers/net:gro_sw.py: all 3 failed cases match known bad, skipping retry
Oct 22 12:24:46 hw1 python3[1162]: [3/53] drivers/net:gro_sw.py: rc=1 (9.7s)
Oct 22 12:24:46 hw1 python3[1162]: [4/53] Running drivers/net:hds.py
Oct 22 12:24:47 hw1 python3[1162]: [4/53] drivers/net:hds.py: rc=0 (0.8s)
Oct 22 12:24:47 hw1 python3[1162]: [5/53] Running drivers/net:macsec.py
Oct 22 12:24:48 hw1 python3[1162]: [5/53] drivers/net:macsec.py: rc=0 (0.3s)
Oct 22 12:24:48 hw1 python3[1162]: [6/53] Running drivers/net:napi_id.py
Oct 22 12:24:48 hw1 python3[1162]: [6/53] drivers/net:napi_id.py: rc=0 (0.6s)
Oct 22 12:24:48 hw1 python3[1162]: [7/53] Running drivers/net:napi_threaded.py
Oct 22 12:24:48 hw1 python3[1162]: [7/53] drivers/net:napi_threaded.py: rc=0 (0.2s)
Oct 22 12:24:48 hw1 python3[1162]: [8/53] Running drivers/net:netpoll_basic.py
Oct 22 12:24:49 hw1 python3[1162]: [8/53] drivers/net:netpoll_basic.py: all 1 failed cases match known bad, skipping retry
Oct 22 12:24:49 hw1 python3[1162]: [8/53] drivers/net:netpoll_basic.py: rc=1 (1.0s)
Oct 22 12:24:49 hw1 python3[1162]: [9/53] Running drivers/net:ping.py
Oct 22 12:25:31 hw1 python3[1162]: [9/53] drivers/net:ping.py: rc=0 (41.4s)
Oct 22 12:25:31 hw1 python3[1162]: [10/53] Running drivers/net:pppoe_gro.py
Oct 22 12:25:31 hw1 python3[1162]: [10/53] drivers/net:pppoe_gro.py: rc=0 (0.7s)
Oct 22 12:25:31 hw1 python3[1162]: [11/53] Running drivers/net:psp.py
Oct 22 12:25:32 hw1 python3[1162]: [11/53] drivers/net:psp.py: rc=0 (0.5s)
Oct 22 12:25:32 hw1 python3[1162]: [12/53] Running drivers/net:queues.py
Oct 22 12:25:32 hw1 python3[1162]: [12/53] drivers/net:queues.py: rc=0 (0.1s)
Oct 22 12:25:32 hw1 python3[1162]: [13/53] Running drivers/net:ring_reconfig.py
Oct 22 12:25:37 hw1 python3[1162]: [13/53] drivers/net:ring_reconfig.py: rc=0 (4.8s)
Oct 22 12:25:37 hw1 python3[1162]: [14/53] Running drivers/net:rss_key.py
Oct 22 12:25:37 hw1 python3[1162]: [14/53] drivers/net:rss_key.py: rc=0 (0.1s)
Oct 22 12:25:37 hw1 python3[1162]: [15/53] Running drivers/net:shaper.py
Oct 22 12:25:37 hw1 python3[1162]: [15/53] drivers/net:shaper.py: rc=0 (0.1s)
Oct 22 12:25:37 hw1 python3[1162]: [16/53] Running drivers/net:so_txtime.py
Oct 22 12:25:48 hw1 python3[1162]: [16/53] drivers/net:so_txtime.py: rc=0 (10.7s)
Oct 22 12:25:48 hw1 python3[1162]: [17/53] Running drivers/net:stats.py
Oct 22 12:26:01 hw1 python3[1162]: [17/53] drivers/net:stats.py: case stats-procfs-downup-hammer not found in known bad, retrying
Oct 22 12:26:14 hw1 python3[1162]: [17/53] drivers/net:stats.py: retry rc=1
Oct 22 12:26:14 hw1 python3[1162]: [17/53] drivers/net:stats.py: rc=1 (13.2s)
Oct 22 12:26:14 hw1 python3[1162]: [18/53] Running drivers/net:xdp.py
Oct 22 12:30:39 hw1 python3[1162]: [18/53] drivers/net:xdp.py: all 6 failed cases match known bad, skipping retry
Oct 22 12:30:39 hw1 python3[1162]: [18/53] drivers/net:xdp.py: rc=1 (265.0s)
Oct 22 12:30:39 hw1 python3[1162]: [19/53] Running drivers/net/hw:csum.py
Oct 22 12:30:59 hw1 python3[1162]: [19/53] drivers/net/hw:csum.py: rc=0 (20.2s)
Oct 22 12:30:59 hw1 python3[1162]: [20/53] Running drivers/net/hw:devlink_rate_cross_esw.py
Oct 22 12:30:59 hw1 python3[1162]: [20/53] drivers/net/hw:devlink_rate_cross_esw.py: rc=0 (0.1s)
Oct 22 12:30:59 hw1 python3[1162]: [21/53] Running drivers/net/hw:devlink_rate_tc_bw.py
Oct 22 12:30:59 hw1 python3[1162]: [21/53] drivers/net/hw:devlink_rate_tc_bw.py: rc=0 (0.1s)
Oct 22 12:30:59 hw1 python3[1162]: [22/53] Running drivers/net/hw:devmem.py
Oct 22 12:31:00 hw1 python3[1162]: [22/53] drivers/net/hw:devmem.py: rc=0 (0.2s)
Oct 22 12:31:00 hw1 python3[1162]: [23/53] Running drivers/net/hw:ethtool.sh
Oct 22 12:31:00 hw1 python3[1162]: [23/53] drivers/net/hw:ethtool.sh: no failed subcases found, retrying
Oct 22 12:31:00 hw1 python3[1162]: [23/53] drivers/net/hw:ethtool.sh: retry rc=1
Oct 22 12:31:00 hw1 python3[1162]: [23/53] drivers/net/hw:ethtool.sh: rc=1 (0.0s)
Oct 22 12:31:00 hw1 python3[1162]: [24/53] Running drivers/net/hw:ethtool_extended_state.sh
Oct 22 12:31:00 hw1 python3[1162]: [24/53] drivers/net/hw:ethtool_extended_state.sh: no failed subcases found, retrying
Oct 22 12:31:00 hw1 python3[1162]: [24/53] drivers/net/hw:ethtool_extended_state.sh: retry rc=1
Oct 22 12:31:00 hw1 python3[1162]: [24/53] drivers/net/hw:ethtool_extended_state.sh: rc=1 (0.0s)
Oct 22 12:31:00 hw1 python3[1162]: [25/53] Running drivers/net/hw:ethtool_mm.sh
Oct 22 12:31:00 hw1 python3[1162]: [25/53] drivers/net/hw:ethtool_mm.sh: no failed subcases found, retrying
Oct 22 12:31:00 hw1 python3[1162]: [25/53] drivers/net/hw:ethtool_mm.sh: retry rc=1
Oct 22 12:31:00 hw1 python3[1162]: [25/53] drivers/net/hw:ethtool_mm.sh: rc=1 (0.0s)
Oct 22 12:31:00 hw1 python3[1162]: [26/53] Running drivers/net/hw:ethtool_rmon.sh
Oct 22 12:31:11 hw1 python3[1162]: [26/53] drivers/net/hw:ethtool_rmon.sh: all 2 failed cases match known bad, skipping retry
Oct 22 12:31:11 hw1 python3[1162]: [26/53] drivers/net/hw:ethtool_rmon.sh: rc=1 (10.9s)
Oct 22 12:31:11 hw1 python3[1162]: [27/53] Running drivers/net/hw:ethtool_std_stats.sh
Oct 22 12:31:20 hw1 python3[1162]: [27/53] drivers/net/hw:ethtool_std_stats.sh: all 7 failed cases match known bad, skipping retry
Oct 22 12:31:20 hw1 python3[1162]: [27/53] drivers/net/hw:ethtool_std_stats.sh: rc=1 (9.1s)
Oct 22 12:31:20 hw1 python3[1162]: [28/53] Running drivers/net/hw:gro_stats.py
Oct 22 12:31:20 hw1 python3[1162]: [28/53] drivers/net/hw:gro_stats.py: rc=0 (0.1s)
Oct 22 12:31:20 hw1 python3[1162]: [29/53] Running drivers/net/hw:hw_stats_l3.sh
Oct 22 12:31:20 hw1 python3[1162]: [29/53] drivers/net/hw:hw_stats_l3.sh: no failed subcases found, retrying
Oct 22 12:31:20 hw1 python3[1162]: [29/53] drivers/net/hw:hw_stats_l3.sh: retry rc=1
Oct 22 12:31:20 hw1 python3[1162]: [29/53] drivers/net/hw:hw_stats_l3.sh: rc=1 (0.0s)
Oct 22 12:31:20 hw1 python3[1162]: [30/53] Running drivers/net/hw:hw_stats_l3_gre.sh
Oct 22 12:31:20 hw1 python3[1162]: [30/53] drivers/net/hw:hw_stats_l3_gre.sh: no failed subcases found, retrying
Oct 22 12:31:20 hw1 python3[1162]: [30/53] drivers/net/hw:hw_stats_l3_gre.sh: retry rc=1
Oct 22 12:31:20 hw1 python3[1162]: [30/53] drivers/net/hw:hw_stats_l3_gre.sh: rc=1 (0.0s)
Oct 22 12:31:20 hw1 python3[1162]: [31/53] Running drivers/net/hw:iou-zcrx.py
Oct 22 12:31:20 hw1 python3[1162]: [31/53] drivers/net/hw:iou-zcrx.py: all 6 failed cases match known bad, skipping retry
Oct 22 12:31:20 hw1 python3[1162]: [31/53] drivers/net/hw:iou-zcrx.py: rc=1 (0.1s)
Oct 22 12:31:20 hw1 python3[1162]: [32/53] Running drivers/net/hw:ipsec_vxlan.py
Oct 22 12:31:20 hw1 python3[1162]: [32/53] drivers/net/hw:ipsec_vxlan.py: rc=0 (0.1s)
Oct 22 12:31:20 hw1 python3[1162]: [33/53] Running drivers/net/hw:irq.py
Oct 22 12:31:20 hw1 python3[1162]: [33/53] drivers/net/hw:irq.py: rc=0 (0.4s)
Oct 22 12:31:20 hw1 python3[1162]: [34/53] Running drivers/net/hw:loopback.sh
Oct 22 12:31:20 hw1 python3[1162]: [34/53] drivers/net/hw:loopback.sh: no failed subcases found, retrying
Oct 22 12:31:20 hw1 python3[1162]: [34/53] drivers/net/hw:loopback.sh: retry rc=1
Oct 22 12:31:21 hw1 python3[1162]: [34/53] drivers/net/hw:loopback.sh: rc=1 (0.0s)
Oct 22 12:31:21 hw1 python3[1162]: [35/53] Running drivers/net/hw:nic_timestamp.py
Oct 22 12:31:21 hw1 python3[1162]: [35/53] drivers/net/hw:nic_timestamp.py: rc=0 (0.1s)
Oct 22 12:31:21 hw1 python3[1162]: [36/53] Running drivers/net/hw:nk_devmem.py
Oct 22 12:31:21 hw1 python3[1162]: [36/53] drivers/net/hw:nk_devmem.py: rc=0 (0.6s)
Oct 22 12:31:21 hw1 python3[1162]: [37/53] Running drivers/net/hw:nk_netns.py
Oct 22 12:31:24 hw1 python3[1162]: [37/53] drivers/net/hw:nk_netns.py: rc=0 (2.4s)
Oct 22 12:31:24 hw1 python3[1162]: [38/53] Running drivers/net/hw:nk_qlease.py
Oct 22 12:31:25 hw1 python3[1162]: [38/53] drivers/net/hw:nk_qlease.py: all 5 failed cases match known bad, skipping retry
Oct 22 12:31:25 hw1 python3[1162]: [38/53] drivers/net/hw:nk_qlease.py: rc=1 (1.4s)
Oct 22 12:31:25 hw1 python3[1162]: [39/53] Running drivers/net/hw:ntuple.py
Oct 22 12:31:26 hw1 python3[1162]: [39/53] drivers/net/hw:ntuple.py: all 24 failed cases match known bad, skipping retry
Oct 22 12:31:26 hw1 python3[1162]: [39/53] drivers/net/hw:ntuple.py: rc=1 (0.8s)
Oct 22 12:31:26 hw1 python3[1162]: [40/53] Running drivers/net/hw:pause.py
Oct 22 12:31:48 hw1 python3[1162]: [40/53] drivers/net/hw:pause.py: all 12 failed cases match known bad, skipping retry
Oct 22 12:31:48 hw1 python3[1162]: [40/53] drivers/net/hw:pause.py: rc=1 (21.9s)
Oct 22 12:31:48 hw1 python3[1162]: [41/53] Running drivers/net/hw:pp_alloc_fail.py
Oct 22 12:31:48 hw1 python3[1162]: [41/53] drivers/net/hw:pp_alloc_fail.py: rc=0 (0.5s)
Oct 22 12:31:48 hw1 python3[1162]: [42/53] Running drivers/net/hw:rss_api.py
Oct 22 12:31:49 hw1 python3[1162]: [42/53] drivers/net/hw:rss_api.py: all 4 failed cases match known bad, skipping retry
Oct 22 12:31:49 hw1 python3[1162]: [42/53] drivers/net/hw:rss_api.py: rc=1 (0.4s)
Oct 22 12:31:49 hw1 python3[1162]: [43/53] Running drivers/net/hw:rss_ctx.py
Oct 22 12:32:09 hw1 python3[1162]: [43/53] drivers/net/hw:rss_ctx.py: case rss-ctx-test-hitless-key-update not found in known bad, retrying
Oct 22 12:32:10 hw1 python3[1162]: [43/53] drivers/net/hw:rss_ctx.py: retry rc=1
Oct 22 12:32:10 hw1 python3[1162]: [43/53] drivers/net/hw:rss_ctx.py: rc=1 (20.9s)
Oct 22 12:32:10 hw1 python3[1162]: [44/53] Running drivers/net/hw:rss_drv.py
Oct 22 12:32:11 hw1 python3[1162]: [44/53] drivers/net/hw:rss_drv.py: rc=0 (0.9s)
Oct 22 12:32:11 hw1 python3[1162]: [45/53] Running drivers/net/hw:rss_flow_label.py
Oct 22 12:32:13 hw1 python3[1162]: [45/53] drivers/net/hw:rss_flow_label.py: all 1 failed cases match known bad, skipping retry
Oct 22 12:32:13 hw1 python3[1162]: [45/53] drivers/net/hw:rss_flow_label.py: rc=1 (1.4s)
Oct 22 12:32:13 hw1 python3[1162]: [46/53] Running drivers/net/hw:rss_input_xfrm.py
Oct 22 12:32:13 hw1 python3[1162]: [46/53] drivers/net/hw:rss_input_xfrm.py: rc=0 (0.1s)
Oct 22 12:32:13 hw1 python3[1162]: [47/53] Running drivers/net/hw:toeplitz.py
Oct 22 12:32:39 hw1 python3[1162]: [47/53] drivers/net/hw:toeplitz.py: rc=0 (25.9s)
Oct 22 12:32:39 hw1 python3[1162]: [48/53] Running drivers/net/hw:tso.py
Oct 22 12:32:39 hw1 python3[19933]: detected unhandled Python exception in './tso.py'
Oct 22 12:32:39 hw1 python3[1162]: [48/53] drivers/net/hw:tso.py: no failed subcases found, retrying
Oct 22 12:32:39 hw1 python3[19998]: detected unhandled Python exception in './tso.py'
Oct 22 12:32:39 hw1 python3[1162]: [48/53] drivers/net/hw:tso.py: retry rc=1
Oct 22 12:32:39 hw1 python3[1162]: [48/53] drivers/net/hw:tso.py: rc=1 (0.1s)
Oct 22 12:32:39 hw1 python3[1162]: [49/53] Running drivers/net/hw:userns_devmem.py
Oct 22 12:32:39 hw1 python3[1162]: [49/53] drivers/net/hw:userns_devmem.py: rc=0 (0.6s)
Oct 22 12:32:39 hw1 python3[1162]: [50/53] Running drivers/net/hw:uso.py
Oct 22 12:32:48 hw1 python3[1162]: [50/53] drivers/net/hw:uso.py: rc=0 (8.2s)
Oct 22 12:32:48 hw1 python3[1162]: [51/53] Running drivers/net/hw:vlan.py
Oct 22 12:32:48 hw1 python3[1162]: [51/53] drivers/net/hw:vlan.py: rc=0 (0.3s)
Oct 22 12:32:48 hw1 python3[1162]: [52/53] Running drivers/net/hw:xdp_metadata.py
Oct 22 12:32:52 hw1 python3[1162]: [52/53] drivers/net/hw:xdp_metadata.py: rc=0 (3.6s)
Oct 22 12:32:52 hw1 python3[1162]: [53/53] Running drivers/net/hw:xsk_reconfig.py
Oct 22 12:32:53 hw1 python3[1162]: [53/53] drivers/net/hw:xsk_reconfig.py: all 1 failed cases match known bad, skipping retry
Oct 22 12:32:53 hw1 python3[1162]: [53/53] drivers/net/hw:xsk_reconfig.py: rc=1 (1.4s)
Oct 22 12:32:53 hw1 python3[1162]: Completed, results in /srv/hw-worker/results/5200
Oct 22 12:33:23 hw1 systemd[1]: nipa-hw-worker.service: State 'stop-sigterm' timed out. Aborting.
Oct 22 12:33:23 hw1 systemd[1]: nipa-hw-worker.service: Killing process 10784 (iperf3) with signal SIGABRT.
Oct 22 12:33:23 hw1 systemd-coredump[20738]: Process 10784 (iperf3) of user 0 dumped core.
                                             
                                             Module libz.so.1 from rpm zlib-ng-2.3.3-3.fc44.x86_64
                                             Module libsctp.so.1 from rpm lksctp-tools-1.0.21-3.fc44.x86_64
                                             Module libcrypto.so.3 from rpm openssl-3.5.5-2.fc44.x86_64
                                             Module libssl.so.3 from rpm openssl-3.5.5-2.fc44.x86_64
                                             Module libiperf.so.0 from rpm iperf3-3.20-2.fc44.x86_64
                                             Module libstdbuf.so from rpm coreutils-9.10-3.fc44.x86_64
                                             Module iperf3 from rpm iperf3-3.20-2.fc44.x86_64
                                             Stack trace of thread 10784:
                                             #0  0x00007f0fb3e96b49 __lll_lock_wait_private (libc.so.6 + 0x6fb49)
                                             #1  0x00007f0fb3ea9600 __libc_malloc2 (libc.so.6 + 0x82600)
                                             #2  0x00007f0fb3e3a85b __dcigettext (libc.so.6 + 0x1385b)
                                             #3  0x00007f0fb4039599 iperf_got_sigend (libiperf.so.0 + 0x19599)
                                             #4  0x000056129b1a2a6c run (iperf3 + 0xa6c)
                                             #5  0x000056129b1a26d6 main (iperf3 + 0x6d6)
                                             #6  0x00007f0fb3e2a681 __libc_start_call_main (libc.so.6 + 0x3681)
                                             #7  0x00007f0fb3e2a798 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3798)
                                             #8  0x000056129b1a2795 _start (iperf3 + 0x795)
                                             ELF object binary architecture: AMD x86-64
Oct 22 12:33:23 hw1 systemd[1]: nipa-hw-worker.service: Failed with result 'timeout'.
Oct 22 12:33:23 hw1 systemd[1]: Failed to start nipa-hw-worker.service - NIPA HW worker (one-shot test runner).
Oct 22 12:33:23 hw1 systemd[1]: nipa-hw-worker.service: Consumed 34.517s CPU time over 8min 50.486s wall clock time, 232.4M memory peak.