<\/p>\n
I am testing the real bandwidth I can get with a 10G network connection. My server has an intel X540-AT2 network card with 2 10G interfaces.<\/p>\n
The server is configured to use bonding in balance-alb mode, but in this test only one interface comes into play because the iperf client only gets a connection from one MAC address.<\/p>\n
$ sudo cat \/proc\/net\/bonding\/bond0 \r\nEthernet Channel Bonding Driver: v3.7.1 (April 27, 2011)\r\n\r\nBonding Mode: adaptive load balancing\r\nPrimary Slave: None\r\nCurrently Active Slave: eth0\r\nMII Status: up\r\nMII Polling Interval (ms): 100\r\nUp Delay (ms): 0\r\nDown Delay (ms): 0\r\n\r\nSlave Interface: eth0\r\nMII Status: up\r\nSpeed: 10000 Mbps\r\nDuplex: full\r\nLink Failure Count: 0\r\nPermanent HW addr: 00:25:90:f9:3d:54\r\nSlave queue ID: 0\r\n\r\nSlave Interface: eth1\r\nMII Status: up\r\nSpeed: 10000 Mbps\r\nDuplex: full\r\nLink Failure Count: 0\r\nPermanent HW addr: 00:25:90:f9:3d:55\r\nSlave queue ID: 0\r\n<\/pre>\n<\/p>\n
Frist run in the server iperf in server mode:<\/p>\n
$ sudo iperf -s\r\n ------------------------------------------------------------\r\n Server listening on TCP port 5001\r\n TCP window size: 85.3 KByte (default)\r\n ------------------------------------------------------------<\/pre>\n<\/p>\n
An from one client run iperf connecting to the server IP<\/p>\n
The first test is using a max transmision unit of 1500 bytes (default)):<\/p>\n
$ sudo iperf -c 172.17.16.78 -i1 -t 10 -m\r\n------------------------------------------------------------\r\nClient connecting to 172.17.16.78, TCP port 5001\r\nTCP window size: 92.9 KByte (default)\r\n------------------------------------------------------------\r\n[ 3] local 172.17.16.79 port 52458 connected with 172.17.16.78 port 5001\r\n[ ID] Interval Transfer Bandwidth\r\n[ 3] 0.0- 1.0 sec 535 MBytes 4.49 Gbits\/sec\r\n[ 3] 1.0- 2.0 sec 594 MBytes 4.98 Gbits\/sec\r\n[ 3] 2.0- 3.0 sec 554 MBytes 4.64 Gbits\/sec\r\n[ 3] 3.0- 4.0 sec 553 MBytes 4.64 Gbits\/sec\r\n[ 3] 4.0- 5.0 sec 565 MBytes 4.74 Gbits\/sec\r\n[ 3] 5.0- 6.0 sec 605 MBytes 5.07 Gbits\/sec\r\n[ 3] 6.0- 7.0 sec 597 MBytes 5.01 Gbits\/sec\r\n[ 3] 7.0- 8.0 sec 587 MBytes 4.92 Gbits\/sec\r\n[ 3] 8.0- 9.0 sec 602 MBytes 5.05 Gbits\/sec\r\n[ 3] 0.0-10.0 sec 5.67 GBytes 4.87 Gbits\/sec\r\n[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)\r\n<\/pre>\nWith this test I can barely get around 5Gbits\/sec. It seems too poor for a 10G network card.<\/p>\n
In the next test I’ve changed the MTU to 9000 (jumbo frames) in all network devices (server, client and switch):<\/p>\n
<\/p>\n
$ sudo iperf -c 172.17.16.78 -i1 -t 10 -m\r\n------------------------------------------------------------\r\nClient connecting to 172.17.16.78, TCP port 5001\r\nTCP window size: 92.9 KByte (default)\r\n------------------------------------------------------------\r\n[ 3] local 172.17.16.79 port 52101 connected with 172.17.16.78 port 5001\r\n[ ID] Interval Transfer Bandwidth\r\n[ 3] 0.0- 1.0 sec 971 MBytes 8.14 Gbits\/sec\r\n[ 3] 1.0- 2.0 sec 1.09 GBytes 9.37 Gbits\/sec\r\n[ 3] 2.0- 3.0 sec 1.05 GBytes 9.04 Gbits\/sec\r\n[ 3] 3.0- 4.0 sec 1.06 GBytes 9.14 Gbits\/sec\r\n[ 3] 4.0- 5.0 sec 1.10 GBytes 9.43 Gbits\/sec\r\n[ 3] 5.0- 6.0 sec 1.11 GBytes 9.51 Gbits\/sec\r\n[ 3] 6.0- 7.0 sec 1009 MBytes 8.46 Gbits\/sec\r\n[ 3] 7.0- 8.0 sec 1.04 GBytes 8.94 Gbits\/sec\r\n[ 3] 8.0- 9.0 sec 1.11 GBytes 9.56 Gbits\/sec\r\n[ 3] 9.0-10.0 sec 1.07 GBytes 9.21 Gbits\/sec\r\n[ 3] 0.0-10.0 sec 10.6 GBytes 9.08 Gbits\/sec\r\n[ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)\r\n<\/pre>\n<\/p>\n
And now, it looks very different, I can saturate the network link.
\nTo change MTU, I’ve changed it in the server and in the client, in all network devices: bond interface and its slaves. Also the switch must support it. You can change the device MTU easly with this command:<\/p>\n$ sudo ip link set dev eth0 mtu 1500<\/pre>\n","protected":false},"excerpt":{"rendered":"I am testing the real bandwidth I can get with a 10G network connection. My server has an intel X540-AT2 network card with 2 10G interfaces. The server is configured to use bonding in balance-alb mode, but in this test only one interface comes into play because the iperf client only gets a connection […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,27],"tags":[114,113,116,115],"_links":{"self":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/841"}],"collection":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/comments?post=841"}],"version-history":[{"count":23,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/841\/revisions"}],"predecessor-version":[{"id":1342,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/841\/revisions\/1342"}],"wp:attachment":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media?parent=841"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/categories?post=841"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/tags?post=841"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}