Skip to content

Watching NAT packets take the shortcut through my router

I wanted to see how NAT packets cross my OpenWrt router with and without flow offloading, so I captured one curl, traced where each packet went and turned it into an animation.

7 min read

My router has a “flow offloading” checkbox, and it is ticked. I wanted an intuitive picture of what that changes: which way a NAT’d packet takes through the router with it off, and which way with it on.

I knew almost nothing about flowtables when I started. Some routers have a hardware NAT engine; mine does not, so the only acceleration I could test is the software flowtable in the Linux kernel. This post is what I pieced together from the kernel documentation and from watching my own router.

The setup

The router is a NanoPi R2S: a Rockchip RK3328 with four Cortex-A53 cores, OpenWrt 24.10, Linux 6.6. The WAN port is on the SoC, the LAN port is a Realtek RTL8153 on USB 3. A Mac mini on the LAN port was the client. The internet link is 5G and tops out around 80 Mbit/s, so the load tests ran against a MacBook Air on the upstream router’s Wi-Fi rather than against the internet.

internet1.1.1.1, speed testUpstream router192.168.0.1iperf3 server on its LANNanoPi R2SRK3328 · OpenWrt 24.10flowtable off / onMac miniclient192.168.2.25G, about 80 Mbit/seth0 WAN192.168.0.95Etherneteth1 LAN, USB NIC192.168.2.1Ethernet, EEE offWi-Fi: everything else
The test setup. Blue links carry the traffic under test. Host routes on the Mac mini send only the test destinations through the R2S; everything else, including SSH to the router, goes over its own Wi-Fi.

When the option is on, OpenWrt’s firewall creates a flowtable and one rule that puts TCP and UDP flows into it:

nft list flowtables
table inet fw4 {
flowtable ft {
hook ingress priority filter
devices = { br-lan, eth0 }
counter
}
}
nft list chain inet fw4 forward
meta l4proto { tcp, udp } flow add @ft

The two paths, as far as I understand them

The kernel documentation has a diagram of both. Without a flowtable, every forwarded packet goes through prerouting, a routing decision, the forward chain with the firewall rules, and postrouting, where OpenWrt’s masquerade rule does the NAT. With a flowtable, a packet that matches an entry at the ingress hook skips all of that. The flowtable code rewrites it right there, address and port for NAT, TTL and MAC addresses, and sends it out.

So NAT happens on every packet either way. What the shortcut saves is working out how to forward and NAT it, again and again for every packet of the same connection. It is still the same CPU doing the work, just less of it, a bit like PHP’s opcache skipping the parse and compile on every request.

Making the path visible

tcpdump showed the NAT but not the path. With a host route that sent only 1.1.1.1 through the R2S, one curl http://1.1.1.1 showed each packet on the LAN port and again on the WAN port with the source rewritten:

tcpdump -ni any (trimmed)
eth1 In 192.168.2.2.62843 > 1.1.1.1.80: Flags [P.], length 70: GET / HTTP/1.1
eth0 Out 192.168.0.95.62843 > 1.1.1.1.80: Flags [P.], length 70: GET / HTTP/1.1

But tcpdump sees a packet when it arrives and when it leaves, not what happened in between, so the capture looked the same with offload on or off.

A log rule worked as a sensor. A rule at the top of the forward chain only fires for packets that take the classic path; packets that hit the flowtable never reach it.

Terminal window
nft 'insert rule inet fw4 forward ip daddr 1.1.1.1 tcp dport 80 counter log prefix "SLOW-OUT " flags tcp sequence'
nft 'insert rule inet fw4 forward ip saddr 1.1.1.1 tcp sport 80 counter log prefix "SLOW-IN " flags tcp sequence'

The TCP sequence numbers in the log let me match each logged packet to its line in the capture. Anything in the capture but not in the log took the shortcut.

One false lead. My first captures had duplicate SYNs and retransmitted responses, and a 20 MB download stalled at 27% while the router’s CPU sat idle. It had nothing to do with NAT. Turning off Energy-Efficient Ethernet on the USB network adapter with ethtool --set-eee eth1 eee off fixed it, and every capture after that was clean.

One curl, packet by packet

On my setup, curl http://1.1.1.1 was ten packets: the handshake, the request, the server’s ACK, a 301 response, the client’s ACK and the teardown. I ran it 20 times and got the same ten packets every time. This is where each one went, from the captures and the log:

Route each packet took
#Packetoffon
1slow pathslow path
2slow pathslow path
3slow pathslow path
4slow pathflowtable
5slow pathflowtable
6slow pathflowtable
7slow pathflowtable
8slow pathslow path
9slow pathslow path
10slow pathslow path

One curl http://1.1.1.1 through the R2S, from packet captures and the slow-path log. Packets from the Mac mini move left to right, replies right to left. With a flowtable, packets 4 to 7 skip the netfilter hooks; the handshake and the teardown still take the classic path. Play slowly sends one packet at a time; hover over or tap a packet in the queue to send just that one.

With offload on, the whole handshake still goes the long way, including its last ACK. From what I read, a TCP connection only goes into the flowtable once conntrack considers it established, and that happens on the handshake’s last ACK. So that packet takes the slow path and creates the entry, and the GET is the first packet to use it. At the other end, the first FIN removes the entry, so the teardown goes the long way too.

The edges did not grow with the amount of data. For a 20 MB download with offload on, 6 packets went through the forward chain. With offload off, 4,948 did.

Does the shortcut matter?

I measured the router’s CPU with iperf3 from the Mac mini to the MacBook Air, at fixed rates so both modes did the same work, with the runs interleaved off, on, off, on. The router also runs Docker, which changes how the LAN bridge is filtered and costs the shortcut a lot; that gets its own post. For these numbers I stopped Docker and turned off the bridge setting it adds, so they should be close to a stock OpenWrt install. CPU is the average over all four cores.

Test Offload off Offload on
TCP upload, 400 Mbit/s 14.0% CPU 11.8% CPU
TCP download, 400 Mbit/s 17.2% CPU 13.9% CPU
UDP, 256-byte packets, max 79 Mbit/s, 39k pps 182 Mbit/s, 89k pps

For big TCP transfers the difference was small. The counters hint at why: in the upload test, 751 MB crossed the forward chain in about 25,000 packets, 30 KB each on average, far bigger than anything that fits on the wire. So the kernel merges consecutive segments before it forwards them, and at 400 Mbit/s that leaves fewer than 2,000 forwarding decisions per second for the shortcut to save.

Small UDP packets are not merged like that, and there the picture changed. One core sat at 98% in both modes. With offload off it forwarded about 39,000 packets per second and dropped the rest; with offload on, about 89,000.

From what I read, the shortcut has a cost too: packets on it skip the firewall hooks, so a new rule does not apply to connections that are already in the flowtable.

What I would do differently

I would start with a Linux client and a wired machine on the far side. The USB adapter’s link and the Wi-Fi on the far end caused most of the noise before I measured anything useful.

I would check the link before blaming the router. A stall on a USB network adapter is worth one ethtool --show-eee first.

And I would plan the load test before running it. My first attempt ran each mode once, all “off” runs first, at whatever rate Wi-Fi allowed, and it showed offload making things slower. Fixed rates and interleaved runs told a different story.

If I get a router with a hardware NAT engine, it becomes a third lane in the animation, one where packets 4 to 7 should never reach the CPU at all.