CSIT 26.02 - VPP Performance #
- TEST FRAMEWORK
- CSIT test environment version is ver. 18, see Environment Versioning.
- General Code Housekeeping: Ongoing code optimizations and bug fixes.
- VPP PERFORMANCE TESTS
- No updates
Known Issues #
These are issues that cause test failures or otherwise limit usefulness of CSIT testing.
Tables stay empty until the list of issues there is finalized.
New #
Any issue listed here may have been present also in a previous release, but was not detected/recognized/reported enough back then. Also, issues previously thought fixed but now reopened are listed here. For reporting reasons, even issues fixed during release are listed here, as long as they caused some results to be missing (or performance wrong).
| # | Github issue number | Issue Description |
|---|---|---|
| 1 | csit/issues/4140 | udpquic: signum=11 on quic set fifo-size |
Previous #
Issues reported in previous releases which still affect the current results (or would be affecting if other issues were not hiding the symptoms):
| # | Github issue number | Issue Description |
|---|---|---|
| 1 | csit/issues/3879 | [CSIT-1795] Ocassionally not all DET44 sessions have been established: 4128767 != 4128768 |
| 2 | csit/issues/3966 | [CSIT-1884] 2n-icx, 2n-spr: All NAT44DET IMIX large scale BIDIR tests fail to create enough sessions |
| 3 | csit/issues/3968 | [CSIT-1886] 3n: Wireguard tests with 100 and more tunnels are failing PDR criteria |
| 4 | vpp/issues/3538 | [VPP-2077] IP fragmentation: running_fragment_id is not thread safe |
| 5 | csit/issues/3996 | [CSIT-1914] TRex does not produce latency data on ICE NICs |
| 6 | csit/issues/3997 | [CSIT-1915] 2n-icx testbeds do not have the same performance |
| 7 | csit/issues/3998 | [CSIT-1916] Poor CPU scaling on 2n-zn2 RDMA |
| 8 | csit/issues/3999 | [CSIT-1917] TRex STL performance is unstable at high pps due to unsent packets |
| 9 | csit/issues/4011 | [CSIT-1929] Lossy trials in nat udp mlx5 and avf tests |
| 10 | csit/issues/4018 | [CSIT-1936] TRex occasionally sees link down in E8xx (dpdk) tests |
| 11 | csit/issues/4020 | [CSIT-1938] 3n-alt: High scale ipsec policy tests may crash VPP |
| 12 | csit/issues/4023 | [CSIT-1941] TRex may wrongly detect link bandwidth |
| 13 | csit/issues/4024 | [CSIT-1942] 3nb-spr hoststack: interface not up after first test |
| 14 | vpp/issues/3551 | [VPP-2090] MRR < PDR: dpdk plugin with mlx5 driver does not read full queue |
| 15 | vpp/issues/3552 | [VPP-2091] Memif crashes VPP in container with jumbo frames |
| 16 | csit/issues/4030 | [CSIT-1948] NICs do not consistently distribute tunnels over RXQs depending on model or plugin |
| 17 | csit/issues/4033 | [CSIT-1951] Combination of AVF and vhost drops all 9000B packets |
| 18 | csit/issues/4043 | [CSIT-1962] 3n-icx, 3na-spr: Udpquicscale tests sometimes fail with various symptoms |
| 19 | csit/issues/4044 | [CSIT-1963] 3n-icxd: Various symptoms pointing to hardware (cable/nic/driver) issues |
| 20 | csit/issues/4045 | [CSIT-1964] 3nb-spr, 3n-snr: Wireguardhw tests are likely to crash |
| 21 | vpp/issues/3597 | dev_iavf: Unable to use more queues than offered initially |
| 22 | csit/issues/4073 | Tests combining iavf+jumbo gradually run out of buffers for rx |
| 23 | csit/issues/4076 | 3n-icx: vhost mounting /dev failed |
| 24 | csit/issues/4088 | Occasional timeout in iperf vhost non-gso |
| 25 | csit/issues/4089 | 2n-zn2 iavf 2c: PDR fails due to delayed packets |
| 26 | csit/issues/4090 | crypto engine error in 1tnl wireguardhw test |
| 27 | csit/issues/4091 | udpquicbase fails due to: Echo connect failed |
| 28 | csit/issues/4092 | udp ldpreload fails on no test data retrieved |
| 29 | csit/issues/4094 | 3n-oct: unsent packets even at min load |
| 30 | csit/issues/4096 | rare crash in nginx tests |
| 31 | csit/issues/4097 | 2n-zn2 iavf: two-band structure in 1c mrr tests |
| 32 | vpp/issues/3628 | ip: route APIs are not thread safe |
| 33 | csit/issues/4106 | Unstable performance of ldpreload+tcp tests |
| 34 | csit/issues/4113 | 2n-grc: rarely, Linux is too slow to re-detect interfaces after VPP kill |
| 35 | csit/issues/4118 | GTPUhw jumbo has zero traffic due to tx_errors |
Hidden #
Issues listed here are not affecting current release results, either because we are no longer running the affected tests or testbeds, or because the adverse effect would be triggered by a symptom not currently present. Issues listed here would generally need to be verified using an unmerged CSIT code, which was not done for most of the issues. Having this list here is useful if the symptom appears in next release, either by a different trigger enabling it, or CSIT increasing test coverage.
| # | Github issue number | Issue Description |
|---|---|---|
| 1 | csit/issues/3885 | [CSIT-1802] all testbeds: AF-XDP - NDR tests failing from time to time on small loss |
| 2 | csit/issues/3978 | [CSIT-1896] depending on topology, l3fwd avoids dut-dut link |
| 3 | csit/issues/3986 | [CSIT-1904] 3n-alt: DPDK testpmd startup check fails on DUT2 |
| 4 | csit/issues/3988 | [CSIT-1906] Zero traffic with cx6/cx7 rdma. Testing migrated to mlx5-core only, on CX7 and CX6 Mellanox NICs |
| 5 | csit/issues/4004 | [CSIT-1922] AF_XDP MRR regressions and PDR failures. Affects subsequent tests, that is why we are not testing AF-XDP now |
| 6 | csit/issues/4051 | [CSIT-1970] JSON export validation does not prevent EPL from consuming invalid data |
| 7 | csit/issues/4095 | jumbo+iavf buffer alloc error: loadbalancer |
| 8 | vpp/issues/3624 | perfmon: asan reports global-buffer-overflow on Intel Xeons |
| 9 | vpp/issues/3627 | ASAN: VPP silently quits after 41560 |
| 10 | vpp/issues/3647 | VPP crash in bad GTPUhw jumbo test since 43754 |
| 11 | csit/issues/4112 | tests relying on docker fail on 3n-alt |
Fixed #
Issues reported in previous releases which were fixed (or stopped being tested forever) in this release:
| # | Github issue number | Issue Description |
|---|---|---|
| 1 | csit/issues/4029 | [CSIT-1947] Rare VPP crash in nat avf tests |
| 2 | vpp/issues/3579 | [VPP-2118] 3n spr: Unusable performance of ipsec tests with SHA_256_128 |
| 3 | csit/issues/4066 | 2n-icx: iavf bus error in one run |
| 4 | csit/issues/4087 | 2n-aws: Infrequent error 11 when dpdk plugin initializes interface |
| 5 | csit/issues/4117 | Zero traffic in GTPUhw jumbo tests |
Root Cause Analysis for Regressions #
List of RCAs in CSIT 26.02 for VPP performance regressions. Not listing differences caused by known issues (uneven worker load due to randomised RSS or other per-worker issues). Also not listing tests which historically show large performance variance.
Contrary to issues, these genuine regressions do not limit usefulness of CSIT testing. So even if they are not fixed (e.g. when the regression is an expected consequence of added functionality), they will not be re-listed in the next release report.
| # | Github issue number | Issue Description |
|---|---|---|
| 1 | csit/issues/4110 | ARM (expected) anomalies near 2025-08-30 |
| 2 | csit/issues/4114 | ip6 regressions end of November 2025 |
| 3 | csit/issues/4137 | rls2610: Investigate ipsec anomalies |
| 4 | csit/issues/4138 | 2n-grc geneve regression around 2025-12-01 |
| 5 | csit/issues/4139 | 2n-zn2 rdma ip4 regression around 2025-11-03 |