A common question from Ostinato customers is how to saturate a ethernet link i.e. send packets on the link at the maximum link speed possible - typically 1/10/40 or 100Gbps.
The short answer to the question is to send the largest packet size possible - 1518 bytes typically and if jumbo packets are supported, 9000-byte packets.
In this post, we go into the longer answer by examining the relationship between packet size and link speed and learn why it affects throughput.
Ethernet Packet Size
The minimum valid Ethernet packet is 64 bytes and the maximum Ethernet packet is 1518 bytes. With the introduction of VLAN, the max increased by 4 bytes (size of VLAN header) to 1522. As link speeds increased, NICs started supporting jumbo packets of up to 9000 bytes.
Note, that the packet size includes the 4-byte Ethernet FCS.
Ethernet Overhead
Ethernet packets sent on the wire incur an additional overhead of 20 bytes for every packet.
+=====+===========+=================+===============+=====+=================+
| SFD | Preamble | Ethernet Header | Ethernet Data | FCS | Inter Frame Gap |
| (1) | (7) | (14) | (46 - 1500) | (4) | (12) |
+=====+===========+=================+===============+=====+=================+
(x) = Size in bytes
This overhead is at the L1 physical layer and consists of the following -
- 1 byte SFD (Start Frame Delimiter)
- 7 byte Preamble
- 12-byte inter-frame gap
This overhead is always a fixed 20 bytes irrespective of the data packet size.
64-byte packet - 20-byte overhead => 23.81%
128-byte packet - 20-byte overhead => 13.51%
256-byte packet - 20-byte overhead => 7.25%
512-byte packet - 20-byte overhead => 3.75%
1518-byte packet - 20-byte overhead => 1.30%
9000-byte packet - 20-byte overhead => 0.22%
As seen above, the smaller the packet size, the larger the percentage overhead.
Bit Rates and Packet Rates
We talk about bandwidth and link speeds in terms of bit rates i.e. in units of bits/sec - 100Mbps, 50Gbps, etc.
Each bit rate also has a corresponding packet rate depending on the packet size. Here’s how we can calculate the same -
PacketRate = BitRate/((PacketSize+20)*8)
- 20 is the per-packet Ethernet overhead
- 8 is to convert the Packet Size in bytes to bits
Let’s see an example -
If we send 1518-sized packets on a 1000Mbps link, the packet rate would be -
1,000,000,000/((1518+20)*8) = 81,274 packets/sec
Calculating packet rates for different packet sizes for a 1000Mbps link, we get -
Why Packet Size Is Important?
As seen in the table above, the smaller the packet size, the higher the packet rate (for the same link speed). This has a real-world implication for network equipment like routers, switches, and firewalls.
Networking equipment processes traffic on a per-packet basis i.e. the CPU processing cost is on a per-packet basis - it doesn’t matter whether the packet size is small or big. This means that for the same 1000Mbps link, they need to process more 64-byte packets than 1518-byte packets - 18 times more!
This is the reason when evaluating networking equipment it is important to know the performance not just in terms of bit rates (e.g. 24x10Gbps) but also the minimum packet size at which full line rate is supported. Not all equipment can process line rate for 64-byte packets on all ports simultaneously - you can use a snake test to verify the performance of a switch or router.
As bandwidth and link speeds increase the packet rates also increase linearly making line rate support for smaller packet sizes increasingly difficult for networking vendors. Consider a 10Gbps link which is common these days - the corresponding packet rate for 64-byte packets is 14.8 million packets per sec (Mpps) and for the newer 100Gbps link it is a whopping 148Mpps!
Which packet size to use for the performance test?
Let’s return to our original question and answer -
Q. How to saturate a link?
A. Send traffic with the largest packet size possible
The reason to use the largest packet size possible is so that the packet rate is less and therefore less CPU is required increasing the chance that the networking equipment can process all the traffic.
Ostinato Turbo Transmit
While Ostinato can saturate a 1Gbps link when using 1518 byte packets, it can’t do the same for a 10Gbps link.
For high bandwidth and line rate testing across different packet sizes, you can use the Turbo Transmit add-on.
In the performance table above, the Ostinato column shows the max rate for the base Ostinato software without any Turbo add-on. Please note that the performance shown is indicative and will vary based on the CPU type and number of cores, RAM, and NIC type.