PQ Show 52 - Using Ostinato To Craft Your Own Packets

Host: Ethan Banks
Guest: Srivats P

This podcast episode was part of the Priority Queue series by the Packet Pushers and published on June 16, 2015 on the Packet Pushers website. In 2024 as part of their website revamp, they removed the episode since the Priority Queue series had been long retired.

It has been republished here with permission. The content has not been modified except for identifying broken/dead links and providing an alternate/update on them.

There is a saying in network training that I’ve heard from a number of instructors. That is, “Be the packet.” The idea is that if you think about networking from the perspective of the packet, hard-to-understand concepts might make better sense. Now, putting yourself in the position of the packet is easier said than done if you don’t really grasp what makes up a packet.

Any of us that have gone through network fundamentals know the OSI model, and any of us that have spent time with Wireshark have a sense of how the OSI model maps to individual packets. Wireshark breaks the packet down into framing headers, IP headers, transport headers, and so on, until you get to the payload. That sort of breakdown goes a long way towards “being the packet.”

But…have you ever wanted to make your own packet? In other words, actually craft a packet of your own imagination? I mean, you should be able to do that, right? Something is taking traffic from your web browser and sticking that payload into a packet that can cross the Internet. And those crafty folks who write NMAP certainly make some interesting packets they throw at networks to see what happens.

If software developers and network driver writers can make their own packets, can’t you? Ah, right…you’re not a developer. Your coding-fu is weak, as is mine. Still, there’s hope for us in an open source tool called Ostinato. Ostinato calls itself “Wireshark in reverse.” With Ostinato, you can make your own packets, custom crafting them any way you like.

Today on the Packet Pushers Priority Queue, we’re talking with one of the Ostinato creators, Srivats P.

📖 Continue reading transcript

Show Notes

Click on the # following each note to jump to the corresponding section in the transcript.

Part 1 - Background

Ostinato is open. Do we mean “open” as in open source? Or do we mean “open” like some of the new definitions of open that vendors are using to describe their closed, commercial products? #
What platforms are supported? #
How many folks are contributing to the project? #
The last binary distribution I see from the download site is dated July 7, 2014, and here it is April 2015. Should I be worried? Special note - since recording this, Srivats released a new version (0.7) of Ostinato in June 2015. #
I have to ask…why the name “ostinato”? I know the word to be a musical term that refers to an underlying, repetitive theme in a piece. The Wikipedia article on the word cites Ravel’s “Bolero” as an example. How does that idea tie into this product? #
Let’s talk about Ostinato use cases. #
- Developer community working on networking software
- QA testers, python scripting supported, also can save streams as a python script
- Security - pen testing, google “packets of death”
- GNS3 - study, certification traffic generation
- Not for the network n00b.

Part 2 - Usage

Architecture. #
- Client - server.
- Drone is the server.
- The GUI is the client, but there is also a python option. And others.
- So, you can do a build a server with a bunch of NICs, run Drone, use it a server to support multiple clients.
What considerations are there when installing Ostinato? #
- Elevated privileges. For instance, launching via sudo on OS X, or there will be no interfaces in the port group.
- Drone is what needs this. Can “setuid” bit.
When building a packet to be sent by Ostinato, there is a concept of a “stream.” Please explain this idea. #
When building a stream, talk through some of what you can do. #
- Ethernet framing
- 802.1q tagging
- IPv4
- IPv6
- TCP, UDP, ICMP, etc.
- DSCP / ToS byte value
- Payload
- There’s lots of randomization options, as well as sequencing options for various fields in a packet Ostinato is generating. Was that driven by user requests, or did it just seem like the right thing to do?
I noticed you can set up several streams to run back to back. What are some use cases for this? #
It would be hard to craft an entire data stream by hand, packet by packet. Are there shortcuts? #
Talk about your DPDK development project. #
Is there anything else interesting about Ostinato that we haven’t talked about yet? #
How can people who want to contribute to Ostinato get involved? #
- Bug reports
- Contribute a protocol builder (sort of like a Wireshark dissector)

Full Transcript

Intro

Ethan (00:00:00): There’s a saying in network training that I’ve heard from a number of instructors, which is “be the packet.” The idea is that if you think about networking from the perspective of the packet, hard-to-understand concepts might make better sense.

Now, putting yourself in the position of the packet is easier said than done if you don’t really grasp what makes up a packet. Okay, any of us that have gone through network fundamentals know the OSI model, and any of us that have spent time with Wireshark have a sense of how the OSI model maps to individual packets. Wireshark breaks the packet down into framing headers, IP headers, transport headers, and so on until you get all the way down to the payload, and that sort of breakdown goes a long way towards being the packet.

But have you ever wanted to make your own packet? In other words, actually craft a packet out of your own imagination? I mean, you should be able to do that, right? Something’s taking traffic from your web browser and sticking that payload into a packet that can then cross the internet, and those crafty folks who write NMAP certainly make some interesting packets with weird flags and all kinds of oddities that are custom crafted. They throw those packets at networks and then see what happens.

So, if software developers and network driver writers can make their own packets, can’t you? And okay, obviously, you’re not a developer, neither am I, our coding foo is very weak. Still, there is hope for us in an open-source tool called Ostinato.

Ostinato calls itself Wireshark in reverse, and with Ostinato, you can make your own packets, custom crafting them any way you like. Today on the Packet Pushers Priority Queue, we’re talking with one of the Ostinato creators, Srivats P.

I am Ethan Banks, your co-host today. You can follow me on Twitter at @ecbanks and follow the show at packetpushers.net.

So, first, Srivats, welcome to the show, and please introduce yourself to the Packet Pushers audience.

Srivats (00:01:43): Hi, I’m Srivats. My day job is with Juniper Networks, where I work on the packet forwarding engine, and I’m based out of Bangalore in India. However, today I’m not here representing my employer, but rather in a personal capacity as the creator and maintainer of Ostinato, which is an open-source packet generator.

Ethan (00:02:00): Now, Srivats, your last name, I’ve just I’ve only ever seen it in your email and in your writing as just the letter P. Is that actually the entirety of your name or is it an abbreviation for something longer?

Srivats (00:02:10): No, it’s an abbreviation for something longer. It’s kind of difficult to pronounce so I [laughter] … shorten it up.

Ethan (00:02:18): Okay, fair enough. So, before we get into discussing what you can do with Ostinato, let’s delve into some background information. To prepare for the show, I downloaded it and read a bunch of the background literature on it, and I thought it would be helpful for the audience to go through some of that background information before diving into its uses and how it works.

Firstly, I want to point out that Ostinato is described as “open,” so let’s clarify what that means. When we say it’s open, do we mean open source, or do we mean open in the sense of some of the newer definitions of open that vendors use to describe their products, which may actually be closed and commercial?

Open or faux-open?

Srivats (00:02:53): No, it’s open as in open. Standard Open Source definition of open so it’s all open source. All development happens in open source, all bug fixes happen in open, so everything is open.

Ethan (00:03:05): Okay. So, does it have any sort of particular license tied to it.

Srivats (00:03:10): Yeah, it’s GPLv3.

Ethan (00:03:11): Got it. Okay. Okay, so it is truly open source software and all of that, no funny business. No redefining it in terms that marketing people might find interesting. Alright. So, it’s open. Now let’s talk about platforms. If I want to use Ostinato, what platform do I need?

Supported platforms

Srivats (00: 03:31): It works on all the popular platforms today, including Windows, Linux, BSD, or MacOS, and the code, as I said, is pretty much cross-platform. You could, if you want, compile it on other platforms, such as Solaris, although not many people are using Solaris these days, I guess.

In fact, I also know of someone who managed to compile it and run it on a Symbian phone, so even that is possible. I don’t know what use case he had for it, but [laughter], it’s possible.

Additionally, I have a virtual machine and a live CD distro.

Ethan (00:04:05): Okay. I didn’t actually notice that on the site the other day. I guess it didn’t dig around long enough. That’s interesting to know. Is it just you working on the code? Or do you have a bunch of people that are contributing to the project?

Solo or team

Srivats (00:04:15): Well, the core team handles the majority of the work, but the community contributes a bit with bug fixes and protocols, among other things. There are also some non-development-related contributions, such as, I’ll talk about this later, someone did lengthy and detailed posts on how to use Ostinato with GNS3, along with a VM he custom built for GNS3. So, we’ll talk about that later.

Ethan (00:04:39): I definitely want to hear about that later. Let’s not forget to circle back to that. It sounds really cool. Okay, moving on. So, I went out yesterday because I was getting ready for the show and making my notes, and I downloaded Ostinato on my Mac and used it. This was from a binary distribution that someone had already pre-compiled for me because I’m lazy and I don’t want to do that. Now, I noticed that the distribution was from July 7, 2014. I mean, we’re recording this in April 2015. Should I be worried about that?

Slow updates

Srivats (00:05:06): Not really, and the answer to your question is in what you said itself. The person who did the binary distribution for you was me! So, as I said, it’s pretty much me who does the code or does the packaging or does everything.

Making releases is kind of a mechanical long process. You make sure it compiles on all the platforms, then you go package it for each, and then put it out and then upload, and there’s a lot of stuff that needs to happen. So, I tend not to do it too much. But when there is some major feature that goes in or some major bug fixes that go in, I try and put out a release. But otherwise, if you go and look at the repository, you will see that code changes are happening even beyond July 7, 2014.

Ethan (00:05:50): So, in other words, you are still actively working on it. I think we first connected over email maybe three years ago because I was looking to create a packet that I could build a gratuitous ARP with, and then put that out on the wire to solve a pretty unique little problem I had for a maintenance window. I ended up using a different product, a Colasoft product, and then you emailed me asking why I didn’t use Ostinato. At that time, I hadn’t heard of it. I didn’t know it was a thing. So, off and on, when we’ve pinged each other over the years, you’re still actively working on it. That’s good. That’s the thing that I wanted to confirm.

Srivats (00:06:37): It’s been eight years and running.

Ethan (00:06:39): Eight years and running, that’s impressive. Yeah. So, you’ve really got some miles on it. Why did you pick the name Ostinato? I mean, I love music and listen to music, including classical music, and I know that an Ostinato refers to an underlying repetitive theme that you might find in a piece. Anyone who’s listened to Ravel’s Bolero would have an example of Ostinato. Bolero is cited in the Wikipedia article that describes Ostinato. Why did you choose that name for the product? Is there any tie into the musical connection, or do you just like the word or…

Why the name Ostinato?

Srivats (00:07:13): Two factors really. So, when I started this project back in 2007, I was learning to play the piano and I was learning to play western classical. So, it was a big part of my life then and occupied a lot of my mind space, and I had decided that I didn’t want a functional name like “packet builder” or “packet generator” – that wasn’t cool enough for me.

So, with music being on my mind, I wanted a musical term. So, I picked “ostinato”. But, I did some research before picking the word “ostinato”. As you rightly said, Ostinato the word comes from classical music, where it refers to a musical phrase, or a sequence of notes that is repeated over and over. For guitar players out there, that’s a riff.

The analogy for what Ostinato actually does in the networking world, is that it’s a sequence of packets that are repeated. For example, you may have an IP packet with an incrementing destination IP address, or you may have a TCP packet followed by a UDP packet, followed by maybe an ICMP, and typically when you’re sending these packets, it takes you some time while you’re observing some behavior.

You’re sending these packets, you’re trying to look at, what is happening when I send these packets, so you tend to run it in a loop. So, you typically always send these packets out in a loop. That’s where Ostinato ties up with this.

Ethan (00:08:32): Yeah - that makes perfect sense the way you phrase that. When I was working with the product yesterday, I could see that you could build a sequence of different packets, and then loop them through if you wanted to as a stream, and so that all makes perfect sense to me now. Okay, so Ostinato is open as in truly open. It’s open-source. It is cross-platform. There are other people that are contributing bug fixes, and so on. Although you are the creator and the one who’s maintaining the code, you’re actively working on it. It’s got a cool name. Alright, we know it’s designed to craft packets, create packets, generate some traffic. So, give me some use cases here. What can you do with Ostinato? What have you seen the user community doing with the product?

Use cases

Srivats (00:09:16): So, let me start with the use case that I first started with - see the way a typical open-source project gets started - you need something, it’s not there, so you go build it, to scratch an itch. Being a networking software developer, I needed something to craft packets with, and whatever was available at that time wasn’t good enough, and wasn’t doing what I needed it to do. So, I thought, okay, what the heck, I’ll go make one.

The primary target audience when I started out was the developer community, which works on networking software. Typically, all such developers will need to send certain kinds of traffic on a daily basis.

Maybe you’re doing a feature development or you’re trying to reproduce a bug that someone reported and you want to investigate it, or you’re doing some kind of functional testing - “Does this work correctly? Does it not?”

Or you could also be doing some kind of performance testing where you’re trying to blast packets at line rate and see how it behaves.

That is where I started, which also leads to the next category of users who are the QA or the testers of such products, the networking software, and they typically validate features or performance, which they could be doing manually or they could be doing some automation using scripts, and Ostinato supports Python scripting using the Ostinato API.

So, you could write test scripts, and then you could just automate - put it as part of your automation framework and keep running it every time you put out a new release. In fact, just very recently, we also added a feature where you could configure all the streams that you need using the GUI, which is quite easy and quick to configure, and then you could just save the whole thing as a Python script. So, you don’t need to keep doing it again and again manually - you do itonce and get the auto-generated Python script out of it.

Another category of users that I know about, although I personally have not really worked much in that area, is security kind of stuff where you do penetration testing, you craft various kinds of packets similar to what NMAP does, maybe, and see how whatever you’re testing behaves.

Ethan (00:09:20): Well, it seemed like if you could do some automation - it popped into my head, you could maybe do some fuzzing with it. You’d have to do some automation to make that go. But it crossed my mind, is another thing too. But yeah, pentesting, certainly, yeah.

Srivats (00:11:27): So, in fact, if you Google for something called “packets of death”, you will come across an interesting investigation, where this vendor who has some voice over IP software was experiencing failures. I mean, random failures here and there and they didn’t really know what was causing it, and sometimes even the reboot of the server wouldn’t solve the problem.

Finally, they figured it out that one particular byte in one particular packet would completely hose the Ethernet controller, as in it would die! Ostinato was featured there as something they used to actually drill down to find this out, and maybe we’ll put a link in the show notes. It’s quite an interesting read of how someone debugged this problem from a report of a [field] failure to point to one single byte in one packet, and then get a new firmware fix from Intel for it.

Ethan (00:12:25): Now, that is crazy. One particular byte in one particular packet could blow this thing up. That’s, I mean, how would you figure that out? Because it’s … what an odd thing to happen. It’s not a buffer overflow, it’s not six other things that you might go, oh, yeah, I can see how that would go. It’s one particular value in one particular byte, wow. Okay, packets of death, I’ll see if we can get that in the show notes.

Srivats (00:12:48): The other thing, as I said, was GNS3. So, if you’re trying to study for some certification, and you’re using GNS3, as typically, almost everyone does, you could use it with that to generate traffic, and we’ll put a link in the show notes for instructions on how to integrate Ostinato with GNS3.

So, in general, anywhere you need to send some packets. So typically, us network people, get a lot done with ping and trace route. But sometimes, those are not enough. So, you need something a little more advanced than that, and that’s where Ostinato comes in.

If you’re familiar with solutions from vendors, like Ixia and, Spirent, Ostinato is intended as an open-source alternative for those solutions. So, of course, in terms of core features and capability, it does not support everything those vendors do, and maybe we’ll talk about limitations later on in the show.

So, depending on your use case, and what you need to do, Ostinato may or may not be sufficient. So, you really have to go and figure out what you need and whether Ostinato serves your purpose. But it’s a good starting point, and one thing that I would definitely want to mention is it’s not for the network noob, I mean, if networking is something new to you, you at least need to understand what packets are, headers are, packet fields are. What is the NIC etc. I don’t think anyone who’s listening to this podcast would anyway be a networking noob - it probably doesn’t apply. But yeah, you need to have some domain knowledge to make use of this.

Ethan (00:14:12): Well, yeah, I mean, there’s some; some of the engineers that listen to the show are early in their career and they’re kind of figuring things out as they go and picking up everything. We get emails from those folks often, but I think generally speaking, most people that are listening to the show are into networking at some level wherever they might be in their career.

Srivats (00:14:31): Yeah, I don’t think they need to know much. So, what a packet is, what packet headers are, what protocols are the OSI model, as you said, and that’s that should get you through.

Usage

Ethan (00:14:42): Well, actually, why don’t we just use this as a segue to then talk about using the product, what that’s like. I’ll talk about my experience from yesterday, which was pretty straightforward. I mean, I kind of knew what to expect. What I did before actually using Ostinato was I went up to the site ostinato.org, which redirected me to the code.google.com repository.

I watched an intro video that I don’t know if it was you or someone else. I assume it was you that just did a screen cap and showed it in about three minutes, how to use Ostinato, how to fire it up, what the interface looks like, what the different windows are, and then how to build a traffic stream and then apply that traffic stream to a port and then generate that traffic on the port.

It was pretty straightforward to get that done, downloaded on my Mac, followed the instructions again in the video and off I went. In Wireshark, I could see the packets that I had crafted, ugly though they were. So, why don’t we start at the beginning for folks. You know, I knew what I was getting into. I didn’t find it too difficult. But maybe you can walk us through the process and let’s start right at the beginning with installation, because I think there’s some interesting notes that are worth bringing up about getting it installed and launching it.

Srivats (00:15:57): Typically for all the popular distributions, you will get a binary distribution or for example, for stuff like BSD, FreeBSD, you may use the ports collection. So, normally you would not need to compile it from source. Although if you do need to do that, there are instructions on the wiki on how to do that.

Ethan (00:16:16): And what is the source code by the way? Is it C or Python or…

Srivats (00:16:21): It’s C++. There’s a bunch of dependencies on other open source libraries, for example, the cross platform stuff is all Qt and there’s protobuf and there’s libpcap. So, as we say, you stand on the shoulder of others to build something right?

Ethan (00:16:35): I’d assumed C, but if you had said Python, and we’re talking about compiling, I was gonna get excited and get into that, because I’ve been reading about that. I know, there’s some use cases for compiling a Python with some special interpreters anyway. Okay, so you’d be compiling with GCC. But you don’t even need to do that because you’ve got binary distributions for, like you said, just about every major platform. Very good.

Srivats (00:16:55): One thing that you would need to note is, to send packets out or to capture packets, you need elevated privileges. Essentially, not everyone can just send packets out of the NIC or receive packets from the NIC, no operating system will allow you to do that. If you’re just running the application as a user, you will get into some trouble, if you don’t take care of that.

Architecture

So, actually, even before I go there, let me take a step back and talk a little bit about the architecture of the product. Ostinato is actually composed of client and server applications. These are two different applications. I kind of hesitate to use client and server because people sometimes interpret it differently in terms like iperf does. It’s not like iperf’s client and server. So, maybe let’s call it agent and controller.

So, the agent or server is actually the application which does the actual packet transmission and receive and capture and the client or the controller is GUI to configure and control this agent or server, or it could be a Python script. So, it could be either GUI or Python, which configures this server part. The server product or the agent product is called Drone and I’ll refer to this later. So, you need to remember that the server application is called Drone.

So, if you’re familiar with other vendor solutions, like Ixia, Spirent chassis solutions, or virtual chassis solutions - it’s similar. So, the chassis is the server application and you use the client GUI to configure that. So, the advantages of this architecture is that you could run the client and server on different machines, and then one client could connect to and configure and control many of these servers, or many of these Drones or you could have one server with many ports and then many clients could be sharing that one server.

So, for example, a typical use case, where people typically use this, you install a Linux server with a bunch of NICs, put it in the lab, you wire it up with your devices to which you want to send the packets, and then you run Drone, which is the server application or the agent application on that Linux server. You just run that application and then from your laptop, Mac OS, Windows, whatever, you run the GUI Ostinato, and then connect to the Drone running on the Linux server and you could have multiple people with their own GUIs, sharing the same Drone.

For example, if the Drone server has 10 ports, maybe five are being used by one person and five are being used by another person. So, this separation of architecture into a client and server enables us all these goodies.

Install considerations

Coming back to our original discussion about the installation considerations, these elevated privileges is actually needed only by the server application i.e. Drone requires it. The GUI doesn’t really need any elevated privileges.

The problem is actually very similar to what Wireshark also has. The typical way to solve this or the easy way to solve this, actually for Windows you don’t need to solve it at all because the libPCAP or winPCAP library is installed as a Windows service with elevated privileges, so you don’t really need to worry about it because libPCAP is what Ostinato uses to send and receive packets.

But if you’re running any of the other platforms, such as Linux, or Mac OS, the quick and dirty solution is to run it as sudo. If your distro or platform doesn’t support sudo, you can run it as root. You actually don’t want to give elevated privileges to the whole application. So, what you can do (instead) is you could on the Drone application set something called the setUID bit - chmod u+s drone - whenever it runs, it will always run with elevated privileges, of course, you need to do this as root.

If you’re using Linux, you can even go more fine grain, you actually don’t need all the privileges to do everything that a root admin would do. You just need to be able to send and receive packets, that’s the only thing that you want to do. So, some Linux distros support something called capabilities. It’s something that the kernel supports and distros include a utility called setcap. So, you can set the capabilities for network RX, TX, only for this Drone application. It will not get any other privileges to do anything else except to send and receive packets. So, that’s kind of the best solution that is to solve this privileges problem.

Ethan (00:21:17): And is all of this is documented, so that you know, people want to go up and look for the best architecture - to most securely, without giving it too many privileges set this up. You’ve got this listed out on the site.

Srivats (00:21:29): Actually, I don’t! But that’s a good point for me to go and do that. I’ve told people this over email and mailing list and stuff. But yeah, I probably never went and documented it. So, that’s a good point for me to go and do that.

Ethan(00:21:46): Yeah, the elevated privileges make sense and now that you’ve described the client-server architecture, we’ve got, you know, this has actually been helpful to me, because I saw drone.app listed on my Mac and I didn’t know what it was. I assumed it was like an optional way I could do something. I didn’t realize that it was being called in the background and then the GUI was simply just a client that was telling Drone what to do.

Now, when I first launched the Ostinato app, even though I had already read the instructions which said, hey, you’re gonna have to do this as sudo and here’s the command. I tried just launching it traditionally, like I would any other Mac app, just double clicking on it to see what would happen. What I got was it launched and then reported to me - hey, I don’t have any ports that I can see in my port group. There was a port group there, but it didn’t actually expand into anything. So, it couldn’t get out anything.

Then I relaunched correctly, as the instruction said with sudo and the following command to go into the app container and pull out the right object and yeah, there everything was. I also saw from the command line, every time I did something in the client, I could see in the text window, the CLI, everything that was actually happening. So, it sounds like what I was actually seeing there were instructions coming from the client sent to Drone and then Drone reporting for me what was actually going on.

Srivats (00:23:04): Yeah, that’s right. Plus, some debugging stuff because when people report problems, I typically ask them to send me that console output to see what is going on. So, I didn’t hide that because it helps me to debug problems. Yeah, it’s essentially the communication between Ostinato and Drone and typically, if you don’t know about this architecture, you don’t actually realize that the Ostinato application actually starts the Drone application in the background and connects to it.

Ethan (00:23:32): Now, is there any security mechanism between Drone and a client? Or if you know, the IP and the port, can you just connect?

Srivats (00:23:42): Yeah, you can just connect. There’s no security right now between Ostinato and Drone.

Ethan (00:23:47): Okay. So, given the use cases for this thing, our test environments and labs, a lot of these kinds of things, that seems reasonable, but it also means that people need to not have Ostinato just hanging out on their production network that something people can connect to, and then start sending packets on the network.

Srivats (00:24:04): Correct. More specifically, they shouldn’t have Drone hanging out.

Ethan (00:24:07): Right, Drone, right, specifically. Yeah. Okay, so we’ve got architecture, we know a bit about how to install it and some of the concerns that are there. So, now I’ve got it, I’m ready to run, I know how to launch it correctly. So, I’ve got the privileges that are required that Ostinato has the access to the kernel that it needs. Now, let’s talk about actually using this thing. When you build a packet to be sent by Ostinato, you’ve got a concept of a stream, which we mentioned before. Can you explain this in more detail what a stream is?

What is a stream?

Srivats(00:24:35): A stream is just a sequence of packets. So, it could be either one packet or more packets. It could be either the same packet repeated a number of times, or you could have some packet attributes changing in the stream, like your IP or MAC source addresses. You could be incrementing them, decrementing them, or just putting out random stuff out there. Or you could even be incrementing or decrementing your packet lengths. So, it’s just a sequence of packets that are sent one after the other.

Packet crafting

Ethan (00:25:01): Now, when I’m in the Ostinato GUI and I start the process of building a stream, I’m actually going to go in and create a packet, and that is pretty straightforward. I can just walk you through a tabbed set of windows in the GUI where I can set what the Ethernet frame is going to look like. There’s a bunch of framing attributes out there that you can set. I can set .1Q tagging for VLANs. I can put tags or even nested tags; it looks like Ostinato supports building a QinQ structure. I can add IPv4 values, IPv6, and I’m glossing over a lot of other things that are there. I can set TCP, UDP, ICMP, and more. I can do things like populate the TOS byte with the DSCP value, let’s say. I can mess with the payload to some degree, and then there’s, as you mentioned, randomization options and sequencing options you talked about where you can change the packet; you know, this time it’s going to be this, the next time it’s going to increment, you know, and be slightly different as you go down the line. What other novel things can I do there when building a stream?

Srivats (00:26:08): As you rightly said, you could choose what kind of headers are there in your packet, what kind of packet it is, and the one property that you used was a set of radio buttons where you said L1 is this, L2 is this, L3 is this, L4 is this - which is what I call the simple mode of configuring protocols. There’s also something called an advanced mode of layering these protocols, and you could just sequence them in any order that you want. So, if you’re really trying to do some crazy things, you want the VLAN ID to come after your IP header, you could do that. So, the normal simple GUI interface will not allow you to do that. But there is a tab below called Advanced - if you click on that, you can just order them in any order that you want and once you do that you could go to each one of those and select all the field values for all the fields that you want.

When I say all, I mean each and every field which you can go and individually set. Of course, you don’t need to go and really set individual values because there are sensible defaults configured for all of them. But if you need to, you can.

In terms of protocols, I think at this point, there are about 25 or 30 common protocols that are included, apart from the ones that you mentioned. There’s also ARP, IGMP, MLD, stuff like that.

If something is not yet implemented, what alternatives do you have?

You could configure it as a hex dump. Maybe you could pull it out of a Wireshark capture and just copy-paste the hex dump and that will play the role. Of course, you really need to know what it is. I mean, if you know about those protocols, and you can configure a hex dump, that’s a good option.

The other option is something called a user script. Now, with hex dump, you’re kind of limited to the fixed set of values that you’ve given. But if you want your protocol, do the randomization or incrementing stuff, or decrementing stuff, or calculating checksums for your payloads and stuff, you could write something called a user script, which is like Javascript and you could configure a protocol. You can e.g. say what is the size of the header, what are the values of the fields in the header and is there any fields which are incrementing and stuff like that. The wiki has documentation for that and if you don’t know JavaScript, no need to worry. A copy, paste and Googling around can take you much far or you can always post on the mailing list for help.

Ethan (00:27:29): Sure. Is that a pretty active mailing list since you bring it up?

Srivats (00:28:40): It’s not pretty active. I mean, the traffic is low, but it’s mostly to the point. So, that’s a good thing I guess.

Ethan (00:28:47): Yeah, okay, perfect.

Srivats (00:28:49): Again, since I’m the only guy doing this work, so I kind of try and reply to all emails if no one else has within a day or two. But yeah, I mean, you’ll get a reply. It might take a day or two. But yeah, you’ll get a reply.

Ethan (00:29:04): Okay. So, I did not try to figure out how to do anything with Python or anything like that to build streams. I was noticing that, okay, if I want to use this tool with the GUI, I can get the job done. But it takes a while to build packets. Is there a more efficient way to go about it, like with Python or some other means to create streams?

Pcap import and replay

Srivats (00:29:23): So, what you could do is, if you have a PCAP file, you could just open the PCAP file and all those packets would get imported and you could just replay it.

Ethan (00:29:31): Okay, so I could import a PCAP file, which creates a stream for me and then if I want to, could I tweak anything in that PCAP file?

Srivats (00:29:41): Right. So, for all the protocols that are supported - that Ostinato knows about, we do something called an intelligent import. So, a PCAP file is nothing more than just a hex dump. It’s a binary dump of what the packet content is. But we do an intelligent import, so we figure out okay, this is for example, the IP protocol. So, this is the structure of the IP protocol. So, the IP source address is this, destination address is this, these are the various fields. Once you import this PCAP file, you’ll get a very similar interface as if you had made a stream from scratch. When you open up the stream, the actual values for those fields will be already filled up for you [from the PCAP}. You can go and change the value if you want, or you can leave it like that.

For protocols that are not supported, we just import it as a hex dump, and you can go edit the hex dump, but then of course, you need to know what you’re doing.

Ethan (00:30:32): Yeah, in other words, Ostinato will still be happy to replay that packet for you. There’s just not a GUI overlay; you can’t parse it out because whatever the protocol is, you don’t have a decoder for it and so right now you’re working at the hex level. You can still work with it. You’re just working at the byte level looking at the hex dump and creating values individually and like you said, and that presumes you know what you’re doing exactly what position you got to be in the field to impact the protocol or do whatever the test is you need to get done.

Ostinato is stateless

Srivats (00:31:01): Exactly, and maybe this is also a good time for me to mention one limitation that Ostinato has which you might face is Ostinato is currently stateless, which means there’s no stateful stuff or state that is remembered. Which essentially means there are no TCP handshakes, no TCP connections being established, no ACK and sequence numbers being correctly populated. So, if you open a TCP stream and try to play it out to a DUT, it would probably not work.

Ethan (00:31:31): Sure. Yeah. If Ostinato is stateless then like a three way handshake couldn’t complete because say Ostinato sent a TCP packet out that’s got a SYN set in the header because it wants to start three way handshake, the SYN ACK would come back if Ostinato were to receive it, it has no idea because it’s not tracking that.

Srivats (00:31:51): Yeah, it’s not decoding that. It’s not tracking.

Ethan (00:31:54): Right, and so then it’s not obviously acknowledging particular sequence numbers or, you know, setting sequence numbers in any particularly meaningful way to a remote host. Again, it’s not playing that role. It’s really just putting specific packets on the wire; it is not the role of an actual IP speaker, if you will. I want to go back to the randomization options and the sequencing options. Was that like a user request kind of thing? I mean, what in your mind, what was the use case to have that functionality built into a stream?

Srivats (00:32:23): So, it is pretty much common in these applications. Typically, for example, you’re trying to test load balancing, and you want to test a bunch of destination IP addresses, does it get load balanced across whatever links you have? So, you would want to send packets with different IP addresses. So, it’s kind of standard par-for-the-course thing for such tools.

Ethan (00:32:44): Got it. Okay, and that would be the same thing for running several streams back to back. There’s many cases where you need to be able to do that for testing. So, that makes sense and the same thing for looping because I know, one of the stream options is to just loop back at the beginning and you know, send it on its way. I also noticed that you can set a burst of packets versus kind of a regular stream of packets. You can set things like an inter-packet gap, too, is that right?

Srivats (00:33:12): You can’t really set it. It gets calculated based on the other parameters that you set, but right now, because I’m using libPCAP, which is kind of on a higher level, I really do not have the control to do that at that level.

Ethan (00:33:26): Okay. Right. So, the screen I’m thinking of I wasn’t able to set it, it was just showing me based on these parameters like what you said.

Transmit modes

Srivats (00:33:34): Yeah, it’s calculated and shown to you. But yeah, you won’t be able to set it. So, in fact, when you’re talking about multiple streams, I’d like to mention something called the two modes of stream transmission. One that you probably used and which is the default is the sequential streams where if you have three streams, all packets are stream one will be sent followed by all packets of stream two and then all packets of stream three and then if you’ve configured it to loop back, to go and do the same thing.

But there’s another mode called interleaved streams. In this mode all the streams that are configured and enabled are kind of interleaved amongst each other based on the configured rates [parallel streams]. For example, let’s say the total throughput that you’re sending - the combined throughput for all the three streams is let’s say 100 Mbps and you could divide it let’s say a TCP stream has 50 Mbps, UDP has 40 and ICMP, let’s say has 10. So, in all it makes 100 Mbps but it does not mean that it will first send 50 Mbps TCP traffic, then 40 Mbps or UDP and 10 Mbps ICMP. They’ll all be interleaved together so that lets say every five seconds or ten seconds interval, you’ll get all TCP, UDP and ICMP together.

Performance

Ethan (00:34:42): Okay, so that leads me to another question here, which is how much data can I actually put through Ostinato? Is it CPU dependent?

Srivats (00:34:48): Yes, the native configuration what you’re running it depends on your CPU and RAM because everyone’s setup is different. It’s kind of difficult to claim one particular number so it’s really you’ll need to try it out on your setup, how much you can achieve. So, typically processing is kind of pps (packets/sec) dependent rather than packet size dependent. So, for larger packet sizes, you’ll have a smaller pps. From a Mbps perspective, you should be able to fill the pipe. But if you’re trying to fill the pipe with say 64 byte packets and you’re trying to fill a 10 gig pipe, most probably your laptop will not do that.

Ethan (00:35:24): If I built a purpose built server that was, you know, at least Quad Core, if not more, is Ostinato, multi-threaded that I can take advantage of all those cores and then, you know, I’ve got a bunch of NICs in it and a bunch of testers that are using it. I mean, I could scale this thing up by throwing a bunch of hardware at it.

Srivats (00:35:44): Not currently. It’s not multicore efficient. I have not yet got to the point to do that. But yes, in the near future.

Ethan (00:35:56): So, really, it’s more about clock speed than multicore at this point and again, if I’ve got enough horsepower for it, and the CPU isn’t being asked to do much else other than be the Drone server, you can with the right mix of protocols do hundreds of megabytes per second traffic, if not more, all, depending on the NIC and hardware combo.

Srivats (00:36:16): Sure, and when you’re building this purpose built server, the ISO, live CD, or the VM is a good way to go because it contains just the bare minimum that you need and doesn’t have anything else installed. So, there’s nothing else which is contending for CPU or memory.

DPDK Contest Winner

Ethan (00:36:34): Now, I know you’ve gotten right into the nuts and bolts of the hardware performance on Ostinato and I was reading through and so you got involved in a DPDK development project that if I remember right, you won some kind of an award or recognition for. You want to talk about that?

Srivats (00:36:47): Intel released DPDK, the Data Plane Development Kit, as open source about two to three years ago, and one of the community members of the DPDK community is an organization called 6wind. They ran a design contest last year for applications using DPDK, which was something on my mind, which I knew was always possible. But you know, there was never the right trigger to try something like this. When I saw this contest, I thought, Okay, this is the time maybe I can do something about it and I have a hard deadline to meet, rather than putting it off.

Ethan (00:37:24): I don’t know what it is about deadlines that make us get things done but I’m the same way. Anyway, yeah, go ahead.

Srivats (00:37:28): One challenge that I had was I do not have access to a server-class multi-core CPU with DPDK supported NICs. So, I really couldn’t test that. But what I did manage to do was test DPDK with a VM. It will of course, not give you the performance benefits. But I intended this to be a prototype, to show that it can really be done and if you have the right hardware, you could go and build something like this. So, I wrote a bunch of code and impressively it won the first prize. The prototype was done, as I said in a VM. So, if you take that code today, it’s not yet integrated upstream. So, if you download a binary distribution or something, you will not get that because I haven’t yet integrated into upstream and you can’t just run it and see what you get.

But while on this topic, I’ll also mention since I did this, the community has taken this DPDK idea ahead and there’s an organization called PL Vision. They have taken that code extended and written the code to do line rate on 1G and 10G NICs and they have a bunch of performance reports that they’ve shared on the blog. We’ll put the link in the show notes.

In a nutshell, if you look at those reports, you’ll see that you can do 1G bidirectional at any packet size, or you could do 10G unidirectional, at any packet size. You could do 10G bidirectional for packet size 128 or higher. Basically, you can’t do 10G bidirectional for 64-byte packets. They put the code on GitHub and we’ll include the link in the show notes.

Ethan (00:39:01): Working on infinite scale is an exaggeration, but that’s what popped into my head. Getting that closer access to the hardware as always.

Srivats (00:39:10): That’s actually another idea that I have. DPDK is only one such technology that allows you to get [high] networking performance on commodity hardware. There are alternate solutions like PF Ring and netmap. So, what I really want to do is I want to keep my focus, since I’m the only person right now, doing this development. I do not want to spread myself too thin. So, I want to try and put in a plug-in kind of architecture where you could have plugins using DPDK or a plugin using netmap or a plugin using PF ring, which will get you that performance benefit and the plugin can be written and maintained by someone else without it impacting my time and my having to support it.

Other interesting possibilities

Ethan (00:39:49): Fair enough, because you’re only one guy. Is there anything else interesting about Ostinato that we haven’t talked about at this point?

Srivats (00:39:56): One thing that I’d probably like to mention is - we already talked about the current deployment models that you can use. You could use Drone on a server and connect it from Ostinato running on a client or using the Python CLI or something.

There’s some interesting and cool stuff there potentially possible and remember these do not exist today. But I believe they’re quite cool to do, for example, and if one day, when I have time, I’ll go try something out or maybe one of the listeners today can try it out.

For example, you could have Drone server running on a Raspberry Pi and you could get a portable tester that you could carry around or you could have a white box switch running something like Cumulus Linux, running Drone, you get like a 48 port traffic generator. Now, I should mention, getting line rate from a white boxswitch which is like 10 gig is going to be tricky, because most of these ASICs are designed for accelerating transit traffic and not host bound [or host originated].

Ethan (00:40:52): Right. So, you may end up relying on the control plane CPU in that case to actually generate traffic.

Srivats (00:40:58): Right. So, of course, it depends on the ASICs, maybe there’s some hack possible on the ASIC to kind of, you know?

Ethan (00:41:05): Depending on the packets that you’re generating, you’re kind of doing some not nice things that could also be weird for the ASIC - I don’t even know what they do in that case but yeah.

Srivats (00:41:12): What typically happens, for most ASICs, if it’s a control packet, they’ll have a mode to, you know, pass it without looking at it.

Especially for vendors, there’s a lot of legacy hardware, which lies around as you EOL older hardware, or you have newer, greater versions of the hardware, you have a lot of older hardware, which is lying unused and you could repurpose that to, you know, run Drone and you’ll get a traffic tester out of it. Anyway, all of this are just ideas at this point and all of this will probably require some work.

Ethan (00:41:44): There’s tons of different options there and the fact that you built it as client server gives you the freedom to do some of these things. Yeah.

Contribute

So, if there’s people that are interested in this stuff, and they want to contribute to Ostinato, how do they get involved?

Srivats (00:41:55): Yeah, contributions are most welcome and contributions may not just be code. Contributions could be in the form of bug reports, or feature suggestions, or documentation or maybe if you could even just, you know, write a blog post or tweet about it - it helps. Or you could join the mailing list, help others and if you really want and if you’re a developer and want to do code contributions - you could contribute protocols.

So, Wireshark has this concept of Dissectors. For each protocol, typically, the way the Wireshark community works is the protocol support is contributed as a Wireshark Dissector. Ostinato has something similar called protocol builders and there’s extensive documentation on the wiki on how to add support for a protocol.

So, you could build that, send it to us to upstream and we’ll go integrate it and if you have any ideas, just send us a mail on the mailing list. That’s the best way to contact us. If you have any ideas, if you have any questions, that’s really the best way to go.

Ethan (00:42:55): And then ostinato.org is the URL for people to stop in, and they can kind of branch off from there to all the other things, correct?

Srivats (00:42:57): That’s right.

Ethan (00:42:57): Well, hey, that’s a good place for us to stop and thanks to all of you out there for listening to the priority queue today. Srivats, where can people follow you on the internet?

Srivats (00:43:07): So, Ostinato.org, as you said, is the website which could be a portal to everything else. The project Twitter handle is @Ostinato and the best way actually, like for any other open source project to get in touch with us is the mailing list, which is ostinato@googlegroups.com.

Ethan (00:43:29): Yep, ostinato@googlegroups.com and yeah, if you go to groups.google.com and then search for Ostinato, that should come right up. That’s where most of my mailing list seem to be on Google groups as well these days.

And I am Ethan Banks. You can follow me on Twitter at @ecbanks , my networking blog, EthanCBanks.com and of course, this has been the packet pushers. You can find the priority queue and all of our other networking podcast channels at packetpushers.net. Follow us on Twitter at @packetpushers. If you’ve got something you’d like to share with the packet pushers’ community, you could blog it up as a community blogger. We provide the platform and you can share your thoughts, projects, problem solving, nifty scripts, cool things you do with Ostinato and pretty much anything else networking related and share that with thousands of fellow engineers around the world. Just send us an email packetpushers@gmail.com and we will get you set up.

Last but not least remember that too much networking would never be enough.

Just a reminder that this recording and transcript is dated 2015 - lots of things have changed since then! Browse the website for latest information or reach out to us via the chat widget.

Priority Queue Podcast