I am testing the real bandwidth I can get with a 10G network connection. My server has an intel X540-AT2 network card with 2 10G interfaces.
The server is configured to use bonding in balance-alb mode, but in this test only one interface comes into play because the iperf client only gets a connection from one MAC address.
$ sudo cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: adaptive load balancing Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:f9:3d:54 Slave queue ID: 0 Slave Interface: eth1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:f9:3d:55 Slave queue ID: 0
Frist run in the server iperf in server mode:
$ sudo iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------
An from one client run iperf connecting to the server IP
The first test is using a max transmision unit of 1500 bytes (default)):
$ sudo iperf -c 172.17.16.78 -i1 -t 10 -m ------------------------------------------------------------ Client connecting to 172.17.16.78, TCP port 5001 TCP window size: 92.9 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.16.79 port 52458 connected with 172.17.16.78 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 535 MBytes 4.49 Gbits/sec [ 3] 1.0- 2.0 sec 594 MBytes 4.98 Gbits/sec [ 3] 2.0- 3.0 sec 554 MBytes 4.64 Gbits/sec [ 3] 3.0- 4.0 sec 553 MBytes 4.64 Gbits/sec [ 3] 4.0- 5.0 sec 565 MBytes 4.74 Gbits/sec [ 3] 5.0- 6.0 sec 605 MBytes 5.07 Gbits/sec [ 3] 6.0- 7.0 sec 597 MBytes 5.01 Gbits/sec [ 3] 7.0- 8.0 sec 587 MBytes 4.92 Gbits/sec [ 3] 8.0- 9.0 sec 602 MBytes 5.05 Gbits/sec [ 3] 0.0-10.0 sec 5.67 GBytes 4.87 Gbits/sec [ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
With this test I can barely get around 5Gbits/sec. It seems too poor for a 10G network card.
In the next test I’ve changed the MTU to 9000 (jumbo frames) in all network devices (server, client and switch):
$ sudo iperf -c 172.17.16.78 -i1 -t 10 -m ------------------------------------------------------------ Client connecting to 172.17.16.78, TCP port 5001 TCP window size: 92.9 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.16.79 port 52101 connected with 172.17.16.78 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 971 MBytes 8.14 Gbits/sec [ 3] 1.0- 2.0 sec 1.09 GBytes 9.37 Gbits/sec [ 3] 2.0- 3.0 sec 1.05 GBytes 9.04 Gbits/sec [ 3] 3.0- 4.0 sec 1.06 GBytes 9.14 Gbits/sec [ 3] 4.0- 5.0 sec 1.10 GBytes 9.43 Gbits/sec [ 3] 5.0- 6.0 sec 1.11 GBytes 9.51 Gbits/sec [ 3] 6.0- 7.0 sec 1009 MBytes 8.46 Gbits/sec [ 3] 7.0- 8.0 sec 1.04 GBytes 8.94 Gbits/sec [ 3] 8.0- 9.0 sec 1.11 GBytes 9.56 Gbits/sec [ 3] 9.0-10.0 sec 1.07 GBytes 9.21 Gbits/sec [ 3] 0.0-10.0 sec 10.6 GBytes 9.08 Gbits/sec [ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
And now, it looks very different, I can saturate the network link.
To change MTU, I’ve changed it in the server and in the client, in all network devices: bond interface and its slaves. Also the switch must support it. You can change the device MTU easly with this command:
$ sudo ip link set dev eth0 mtu 1500