Mellanox ConnectX-3 Pro 40GbE NIC
Very cheap qsfp+ card CX314A
Benefits
- 10/40/56Gb/s connectivity for servers and storage
- World-class cluster, network, and storage performance
- Cutting edge performance in virtualized overlay networks (VXLAN and NVGRE)
- Guaranteed bandwidth and low-latency services
- I/O consolidation
- Virtualization acceleration
- Power efficient
- Scales to tens-of-thousands of nodes
Key Features
- 1us MPI ping latency
- Up to 40/56 Gigabit Ethernet per port
- Single- and Dual-Port options available
- PCI Express 3.0 (up to 8GT/s)
- CPU offload of transport operations
- Application offload
- Precision Clock Synchronization
- HW Offloads for NVGRE and VXLAN encapsulated traffic
- End-to-end QoS and congestion control
- Hardware-based I/O virtualization
- 17mm x 17mm RoHS-R6
Docs
Benchmark
Env:
- ubuntu server 24
- dual intel xeon 8368
commands
commands
ip netns add ns_eth0
ip netns add ns_eth1
ip link set cx0p0 netns ns_eth0
ip link set cx0p1 netns ns_eth1
ip netns exec ns_eth0 ip addr add dev cx0p0 10.10.0.1/24
ip netns exec ns_eth1 ip addr add dev cx0p1 10.10.0.2/24
ip netns exec ns_eth0 ip link set cx0p0 up
ip netns exec ns_eth1 ip link set cx0p1 up
ip netns exec ns_eth0 ip l set mtu 9900 cx0p0
ip netns exec ns_eth1 ip l set mtu 9900 cx0p1
cpupower frequency-set -g performance
ip netns exec ns_eth0 iperf3 -s
ip netns exec ns_eth1 iperf3 -c 10.10.0.1 -P1
ip netns exec ns_eth1 iperf3 -c 10.10.0.1 -P2
ip netns exec ns_eth1 iperf3 -c 10.10.0.1 -P4
iperf3 raw results
k3s@k3s-worker6:~$ sudo ip netns exec ns_eth1 iperf3 -c 10.10.0.1 -P1
Connecting to host 10.10.0.1, port 5201
[ 5] local 10.10.0.2 port 46850 connected to 10.10.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.92 GBytes 25.1 Gbits/sec 0 1.49 MBytes
[ 5] 1.00-2.00 sec 2.96 GBytes 25.4 Gbits/sec 0 1.59 MBytes
[ 5] 2.00-3.00 sec 3.03 GBytes 26.0 Gbits/sec 0 1.59 MBytes
[ 5] 3.00-4.00 sec 3.03 GBytes 26.0 Gbits/sec 0 1.59 MBytes
[ 5] 4.00-5.00 sec 3.03 GBytes 26.0 Gbits/sec 0 1.66 MBytes
[ 5] 5.00-6.00 sec 3.03 GBytes 26.0 Gbits/sec 0 1.66 MBytes
[ 5] 6.00-7.00 sec 2.98 GBytes 25.6 Gbits/sec 0 1.66 MBytes
[ 5] 7.00-8.00 sec 3.02 GBytes 26.0 Gbits/sec 0 1.66 MBytes
[ 5] 8.00-9.00 sec 3.00 GBytes 25.8 Gbits/sec 0 1.66 MBytes
[ 5] 9.00-10.00 sec 3.02 GBytes 26.0 Gbits/sec 0 1.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 30.0 GBytes 25.8 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 30.0 GBytes 25.8 Gbits/sec receiver
k3s@k3s-worker6:~$ sudo ip netns exec ns_eth1 iperf3 -c 10.10.0.1 -P2
Connecting to host 10.10.0.1, port 5201
[ 5] local 10.10.0.2 port 35418 connected to 10.10.0.1 port 5201
[ 7] local 10.10.0.2 port 35422 connected to 10.10.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.04 GBytes 17.5 Gbits/sec 0 1.50 MBytes
[ 7] 0.00-1.00 sec 2.04 GBytes 17.5 Gbits/sec 0 1.33 MBytes
[SUM] 0.00-1.00 sec 4.07 GBytes 35.0 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 1.98 GBytes 17.0 Gbits/sec 0 1.65 MBytes
[ 7] 1.00-2.00 sec 1.98 GBytes 17.0 Gbits/sec 0 1.64 MBytes
[SUM] 1.00-2.00 sec 3.95 GBytes 34.0 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 1.99 GBytes 17.1 Gbits/sec 0 1.65 MBytes
[ 7] 2.00-3.00 sec 1.99 GBytes 17.1 Gbits/sec 0 1.64 MBytes
[SUM] 2.00-3.00 sec 3.99 GBytes 34.3 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 2.04 GBytes 17.6 Gbits/sec 0 1.65 MBytes
[ 7] 3.00-4.00 sec 2.04 GBytes 17.6 Gbits/sec 0 1.73 MBytes
[SUM] 3.00-4.00 sec 4.09 GBytes 35.1 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.65 MBytes
[ 7] 4.00-5.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.73 MBytes
[SUM] 4.00-5.00 sec 4.09 GBytes 35.2 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.65 MBytes
[ 7] 5.00-6.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.73 MBytes
[SUM] 5.00-6.00 sec 4.09 GBytes 35.2 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 2.04 GBytes 17.5 Gbits/sec 0 1.65 MBytes
[ 7] 6.00-7.00 sec 2.04 GBytes 17.5 Gbits/sec 0 1.73 MBytes
[SUM] 6.00-7.00 sec 4.07 GBytes 35.0 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 2.04 GBytes 17.5 Gbits/sec 0 2.06 MBytes
[ 7] 7.00-8.00 sec 2.04 GBytes 17.5 Gbits/sec 0 1.73 MBytes
[SUM] 7.00-8.00 sec 4.08 GBytes 35.1 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 2.04 GBytes 17.6 Gbits/sec 0 2.06 MBytes
[ 7] 8.00-9.00 sec 2.04 GBytes 17.6 Gbits/sec 0 1.73 MBytes
[SUM] 8.00-9.00 sec 4.09 GBytes 35.1 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 2.05 GBytes 17.6 Gbits/sec 0 2.06 MBytes
[ 7] 9.00-10.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.73 MBytes
[SUM] 9.00-10.00 sec 4.11 GBytes 35.3 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 20.3 GBytes 17.5 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 20.3 GBytes 17.5 Gbits/sec receiver
[ 7] 0.00-10.00 sec 20.3 GBytes 17.5 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 20.3 GBytes 17.5 Gbits/sec receiver
[SUM] 0.00-10.00 sec 40.6 GBytes 34.9 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 40.6 GBytes 34.9 Gbits/sec receiver
k3s@k3s-worker6:~$ sudo ip netns exec ns_eth1 iperf3 -c 10.10.0.1 -P4
Connecting to host 10.10.0.1, port 5201
[ 5] local 10.10.0.2 port 44686 connected to 10.10.0.1 port 5201
[ 7] local 10.10.0.2 port 44692 connected to 10.10.0.1 port 5201
[ 9] local 10.10.0.2 port 44694 connected to 10.10.0.1 port 5201
[ 11] local 10.10.0.2 port 44708 connected to 10.10.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 672 MBytes 5.63 Gbits/sec 0 1.16 MBytes
[ 7] 0.00-1.00 sec 1.30 GBytes 11.1 Gbits/sec 0 1.23 MBytes
[ 9] 0.00-1.00 sec 672 MBytes 5.63 Gbits/sec 0 1.14 MBytes
[ 11] 0.00-1.00 sec 1.31 GBytes 11.2 Gbits/sec 0 1.23 MBytes
[SUM] 0.00-1.00 sec 3.92 GBytes 33.6 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 665 MBytes 5.58 Gbits/sec 0 1.16 MBytes
[ 7] 1.00-2.00 sec 1.27 GBytes 10.9 Gbits/sec 0 1.30 MBytes
[ 9] 1.00-2.00 sec 666 MBytes 5.58 Gbits/sec 0 1.14 MBytes
[ 11] 1.00-2.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.29 MBytes
[SUM] 1.00-2.00 sec 3.87 GBytes 33.3 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 678 MBytes 5.69 Gbits/sec 0 1.16 MBytes
[ 7] 2.00-3.00 sec 1.26 GBytes 10.9 Gbits/sec 0 1.30 MBytes
[ 9] 2.00-3.00 sec 678 MBytes 5.69 Gbits/sec 0 1.14 MBytes
[ 11] 2.00-3.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.29 MBytes
[SUM] 2.00-3.00 sec 3.93 GBytes 33.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 666 MBytes 5.59 Gbits/sec 0 1.16 MBytes
[ 7] 3.00-4.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.30 MBytes
[ 9] 3.00-4.00 sec 665 MBytes 5.58 Gbits/sec 0 1.14 MBytes
[ 11] 3.00-4.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.29 MBytes
[SUM] 3.00-4.00 sec 3.89 GBytes 33.4 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 662 MBytes 5.55 Gbits/sec 0 1.16 MBytes
[ 7] 4.00-5.00 sec 1.29 GBytes 11.0 Gbits/sec 0 1.30 MBytes
[ 9] 4.00-5.00 sec 664 MBytes 5.57 Gbits/sec 0 1.14 MBytes
[ 11] 4.00-5.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.29 MBytes
[SUM] 4.00-5.00 sec 3.87 GBytes 33.2 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 667 MBytes 5.59 Gbits/sec 0 1.16 MBytes
[ 7] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.30 MBytes
[ 9] 5.00-6.00 sec 666 MBytes 5.59 Gbits/sec 0 1.14 MBytes
[ 11] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.29 MBytes
[SUM] 5.00-6.00 sec 3.89 GBytes 33.4 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 674 MBytes 5.65 Gbits/sec 0 1.16 MBytes
[ 7] 6.00-7.00 sec 1.31 GBytes 11.3 Gbits/sec 0 1.30 MBytes
[ 9] 6.00-7.00 sec 674 MBytes 5.65 Gbits/sec 0 1.14 MBytes
[ 11] 6.00-7.00 sec 1.31 GBytes 11.2 Gbits/sec 0 1.29 MBytes
[SUM] 6.00-7.00 sec 3.93 GBytes 33.8 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 667 MBytes 5.59 Gbits/sec 0 1.16 MBytes
[ 7] 7.00-8.00 sec 1.30 GBytes 11.1 Gbits/sec 0 1.30 MBytes
[ 9] 7.00-8.00 sec 666 MBytes 5.59 Gbits/sec 0 1.14 MBytes
[ 11] 7.00-8.00 sec 1.30 GBytes 11.1 Gbits/sec 0 1.29 MBytes
[SUM] 7.00-8.00 sec 3.89 GBytes 33.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 663 MBytes 5.56 Gbits/sec 0 1.16 MBytes
[ 7] 8.00-9.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.30 MBytes
[ 9] 8.00-9.00 sec 664 MBytes 5.57 Gbits/sec 0 1.14 MBytes
[ 11] 8.00-9.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.29 MBytes
[SUM] 8.00-9.00 sec 3.88 GBytes 33.3 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 662 MBytes 5.55 Gbits/sec 0 1.16 MBytes
[ 7] 9.00-10.00 sec 1.28 GBytes 11.0 Gbits/sec 0 1.30 MBytes
[ 9] 9.00-10.00 sec 662 MBytes 5.55 Gbits/sec 0 1.14 MBytes
[ 11] 9.00-10.00 sec 1.29 GBytes 11.0 Gbits/sec 0 1.93 MBytes
[SUM] 9.00-10.00 sec 3.86 GBytes 33.2 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.52 GBytes 5.60 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 6.52 GBytes 5.60 Gbits/sec receiver
[ 7] 0.00-10.00 sec 12.9 GBytes 11.1 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 12.9 GBytes 11.1 Gbits/sec receiver
[ 9] 0.00-10.00 sec 6.52 GBytes 5.60 Gbits/sec 0 sender
[ 9] 0.00-10.00 sec 6.52 GBytes 5.60 Gbits/sec receiver
[ 11] 0.00-10.00 sec 13.0 GBytes 11.2 Gbits/sec 0 sender
[ 11] 0.00-10.00 sec 13.0 GBytes 11.2 Gbits/sec receiver
[SUM] 0.00-10.00 sec 38.9 GBytes 33.4 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 38.9 GBytes 33.4 Gbits/sec receiver