The industry standard for capturing, storing, understanding, analyzing and replaying network packets are PCAP files. This post is about that last bit - pcap replay.
PCAP replay seems easy. You read a packet from the PCAP file, create a new packet with the same content and send it out on a interface using raw sockets or libpcap.
But that by itself may not be enough.
PCAP Timing Challenges
You likely also want to replicate the timing of the packets in the PCAP i.e. you want to send all the packets in the PCAP not only in the same order but also with the same relative timing between the packets.
To maintain the timing, OS provided routines like sleep, msleep and usleep are typically not enough because they are not accurate. The OS only provides a lower bound guarantee, not the upper bound e.g. say if you sleep for 169 milliseconds, the OS will internally start a timer for that period, put your process to sleep and wake it up when the timer expires - but by that time control returns back to your program, due to other competing processes and the scheduler itself, the actual time spent may be closer to 200 ms or even more - not very accurate.
So PCAP replay tools like tcpreplay and Ostinato will typically use a busy loop for accurate timing. In a busy loop, you don’t call any OS primitive that will put your process (or thread) to sleep. Instead you keep fetching the time in a tight loop like this till the timer duration is over -
now = getCurrentTime()
endAt = now + timerDuration
do
now = getCurrentTime()
while (now != endAt)
While the above code is running, the CPU will typically be at 100%. Wasteful? Yes - but essential to ensure timing accuracy!
Stateful Application Traffic Replay
The second (and much bigger) challenge is when the PCAP contains application level traffic that is stateful e.g. those that work over TCP.
With TCP encapsulated packets, you can’t blindly replay the packets. You have to first establish the TCP connection with the peer using the 3-way handshake, possibly using different source and/or destination ports (than what is in the PCAP packets). Once the connection is established, you replay the TCP packets over this new connection rewriting the TCP sequence and acknowledgement numbers in the PCAP packet with what is being used for this connection while keeping track of the TCP window with the new sequence and ack numbers based on what your peer is acknowledging. You may also need to retransmit any TCP packets that may have been dropped if the peer requests for them. Once all the data is sent, you need to tear down the connection for a graceful close.
The application data within the TCP payload is seldom unidirectional. Both sides send application data - even if the amount of data is highly asymmetric e.g. a HTTP client will send a GET request with some HTTP headers which is very small in comparison to what the HTTP server sends back (the HTML file contents e.g.) - relatively speaking. But both sides send data and if you have a full TCP conversation captured within a PCAP file, you need to distinguish and extract the TCP packets sent in each direction into separate files, set up the two peers with their respective data before you can do a PCAP replay from both sides!
As you can imagine, this can get very complicated!
Although tcpreplay (and Ostinato) don’t replay stateful TCP traffic, tcpliveplay (Linux only) does with limitations. Please read the tcpliveplay man page for details. (As of 2024, it looks like this tool is not actively maintained?).
An alternate (and easier) approach to replaying stateful application traffic would be to extract the TCP payloads into one or two files and then send the contents of the file over a new connection (with likely different TCP window and packet sizes). Our professional services can help you with this.