Step 1 of 5 · Reading · ~4 min
Read
Production
Performance: MTU, Overhead and Packets per Second
After this lesson your program can size a tunnel for a given link: the inner MTU that avoids fragmentation, the TCP segment size that fits, how many bytes a transfer really puts on the wire, and how many packets per second a link of a given speed demands.
The unit of work is the packet
Encrypting with ChaCha20-Poly1305 costs roughly one to two CPU cycles per byte on a modern core, several gigabytes per second. What costs more is everything that happens once per packet: the system calls, copying the packet between the TUN device and your buffer, the counter and replay bookkeeping, the lookup of the peer, the UDP send. On a 1 Gbit/s link of 1500 byte packets that is about 83,000 packets per second, so a single core has roughly 12 microseconds per packet. At 10 Gbit/s it is 1.2 microseconds, which is why the real gains are in doing less per packet.
| Technique | What it saves |
|---|---|
| Kernel data path (WireGuard in Linux) | one copy and one system call per packet compared with a user-space tunnel |
| GSO/GRO: handle a train of segments as one large packet | per-packet lookups and queue operations; the train is split or merged at the edge |
| Encrypt on many cores, send in counter order | the cipher scales with cores; ordering is kept because the counter is assigned first |
| Keep handshakes rare (every two minutes) | public key maths never sits on the data path |
Sizing the tunnel
Start from the link MTU, the largest IP packet the physical link carries, usually 1500. Every inner packet gets 32 bytes of WireGuard (a 16 byte header and a 16 byte tag), 8 bytes of UDP and the outer IP header, 20 bytes for IPv4 or 40 for IPv6. The inner MTU is what is left:
inner MTU = link MTU - outer IP header - 8 - 32
On a 1500 byte link that is 1440 over IPv4 and 1420 over IPv6. If the tunnel MTU is larger, every full-size packet becomes an outer datagram bigger than the link and has to be fragmented: two packets on the wire, each with its own header, reassembly at the far end, and any lost fragment loses the whole datagram. Many networks drop fragments. Setting the right MTU is the single cheapest performance fix in a VPN.
TCP inside the tunnel advertises a maximum segment size (MSS), the data bytes per segment. With a 20 byte inner IP header and a 20 byte TCP header: MSS = inner MTU - 40.
Worked example, IPv4 outer on a 1500 byte link, sending 3000 bytes of application data:
| Step | Value |
|---|---|
| inner MTU | 1500 - 20 - 8 - 32 = 1440 |
| MSS | 1440 - 40 = 1400 |
| segments | 3000 / 1400 rounds up to 3: two of 1400 bytes and one of 200 |
| full segment on the wire | 1400 + 40 = 1440, padded to at most the MTU, so 1440 + 32 + 8 + 20 = 1500 |
| last segment | 200 + 40 = 240, already a multiple of 16, so 240 + 60 = 300 |
| total | 1500 + 1500 + 300 = 3300 bytes for 3000 bytes of data |
A full-size outer datagram always equals the link MTU exactly. That makes the speed arithmetic simple: a link of R Mbit/s carries R x 125,000 bytes per second, so it carries R x 125000 / link MTU full datagrams per second, and the application gets MSS / link MTU of the raw rate. Small packets are far worse: a 100 byte request pays the same 60 bytes of tunnel headers as a full segment.
Common mistakes
- Using the link MTU as the tunnel MTU. The inner packet then needs fragmenting; subtract the tunnel headers first.
- Using the inner MTU as the MSS. TCP and IP headers take 40 more bytes of every segment.
- Padding the last segment to the MTU. Padding rounds up to a multiple of 16 and never beyond the MTU.
- Forgetting that the outer family sets the header size: 40 bytes for IPv6 against 20 for IPv4.
- Overflowing 32 bits. A few gigabytes of data gives wire totals above 2^32.
Your exercise
The starter reads the commands. You compute the sizes. The most likely mistake is giving the last, short segment the same size as the others.
Discussion
Ask a question, share an insight, or help someone who’s stuck.
Sign in to post a comment or reply.
Loading…