There are a number of areas that we need to benchmark on the network to be able to understand any limitation and make sure there are no misconfigurations.
A standard Ethernet frame is 1,500 bytes, while a jumbo frame is typically 9,000 bytes. This increased frame size reduces the overheads for sending data. If you have configured your network with a jumbo frame, the first thing to check is that they are configured correctly across all your servers and networking devices. If jumbo frames are configured incorrectly, Ceph will exhibit strange, random behavior that is very hard to trace; therefore, it is essential that jumbo frames are configured correctly and confirmed to be working before deploying Ceph over the top of your network.
To confirm whether jumbo frames are working correctly, you can use ping to send large packets with the don't fragment flag set:
ping -M do -s 8972 <destination IP>
This command should be run across all your nodes to make sure they can ping each other using jumbo frames. If it fails, investigate the issue and resolve it before deploying Ceph.
The next test to undertake is to measure the round-trip time, also with the ping tool. Using the packet size parameter again but with the don't fragment flag, it is possible to test the round-trip time of certain packet sizes up to 64 KB, which is the maximum IP packet size.
Here are some example readings between two hosts on a 10GBase-T network:
- 32 B = 85 microseconds
- 4 KB = 112 microseconds
- 16 KB = 158 microseconds
- 64 KB = 248 microseconds
As you can see, larger packet sizes impact the round-trip time; this is one reason why larger I/O sizes will see a decrease in IOPS in Ceph.
Finally, let's test the bandwidth between two hosts to determine whether we get the expected performance.
Run iperf -s on the server that will run the iPerf server role:
Then, run the iperf -c <address of iperf server> command:
In this example, the two hosts are connected via a 10 G network and obtain near the maximum theoretical throughput. If you do not see the correct throughput, an investigation into the network, including host configuration, needs to be done.