Watching NAT packets take the shortcut through my router
I wanted to see how NAT packets cross my OpenWrt router with and without flow offloading, so I captured one curl, traced where each packet went and turned it into an animation.
7 min read
My router has a “flow offloading” checkbox, and it is ticked. I wanted an intuitive picture of what that changes: which way a NAT’d packet takes through the router with it off, and which way with it on.
I knew almost nothing about flowtables when I started. Some routers have a hardware NAT engine; mine does not, so the only acceleration I could test is the software flowtable in the Linux kernel. This post is what I pieced together from the kernel documentation and from watching my own router.
The setup
The router is a NanoPi R2S: a Rockchip RK3328 with four Cortex-A53 cores, OpenWrt 24.10, Linux 6.6. The WAN port is on the SoC, the LAN port is a Realtek RTL8153 on USB 3. A Mac mini on the LAN port was the client. The internet link is 5G and tops out around 80 Mbit/s, so the load tests ran against a MacBook Air on the upstream router’s Wi-Fi rather than against the internet.
When the option is on, OpenWrt’s firewall creates a flowtable and one rule that puts TCP and UDP flows into it:
table inet fw4 { flowtable ft { hook ingress priority filter devices = { br-lan, eth0 } counter }}meta l4proto { tcp, udp } flow add @ftThe two paths, as far as I understand them
The kernel documentation has a diagram of both. Without a flowtable, every forwarded packet goes through prerouting, a routing decision, the forward chain with the firewall rules, and postrouting, where OpenWrt’s masquerade rule does the NAT. With a flowtable, a packet that matches an entry at the ingress hook skips all of that. The flowtable code rewrites it right there, address and port for NAT, TTL and MAC addresses, and sends it out.
So NAT happens on every packet either way. What the shortcut saves is working out how to forward and NAT it, again and again for every packet of the same connection. It is still the same CPU doing the work, just less of it, a bit like PHP’s opcache skipping the parse and compile on every request.
Making the path visible
tcpdump showed the NAT but not the path. With a host route that sent only 1.1.1.1 through the R2S, one curl http://1.1.1.1 showed each packet on the LAN port and again on the WAN port with the source rewritten:
eth1 In 192.168.2.2.62843 > 1.1.1.1.80: Flags [P.], length 70: GET / HTTP/1.1eth0 Out 192.168.0.95.62843 > 1.1.1.1.80: Flags [P.], length 70: GET / HTTP/1.1But tcpdump sees a packet when it arrives and when it leaves, not what happened in between, so the capture looked the same with offload on or off.
A log rule worked as a sensor. A rule at the top of the forward chain only fires for packets that take the classic path; packets that hit the flowtable never reach it.
nft 'insert rule inet fw4 forward ip daddr 1.1.1.1 tcp dport 80 counter log prefix "SLOW-OUT " flags tcp sequence'nft 'insert rule inet fw4 forward ip saddr 1.1.1.1 tcp sport 80 counter log prefix "SLOW-IN " flags tcp sequence'The TCP sequence numbers in the log let me match each logged packet to its line in the capture. Anything in the capture but not in the log took the shortcut.
One false lead. My first captures had duplicate SYNs and retransmitted responses, and a 20 MB download stalled at 27% while the router’s CPU sat idle. It had nothing to do with NAT. Turning off Energy-Efficient Ethernet on the USB network adapter with ethtool --set-eee eth1 eee off fixed it, and every capture after that was clean.
One curl, packet by packet
On my setup, curl http://1.1.1.1 was ten packets: the handshake, the request, the server’s ACK, a 301 response, the client’s ACK and the teardown. I ran it 20 times and got the same ten packets every time. This is where each one went, from the captures and the log:
| # | Packet | off | on |
|---|---|---|---|
| 1 | slow path | slow path | |
| 2 | slow path | slow path | |
| 3 | slow path | slow path | |
| 4 | slow path | flowtable | |
| 5 | slow path | flowtable | |
| 6 | slow path | flowtable | |
| 7 | slow path | flowtable | |
| 8 | slow path | slow path | |
| 9 | slow path | slow path | |
| 10 | slow path | slow path |
With offload on, the whole handshake still goes the long way, including its last ACK. From what I read, a TCP connection only goes into the flowtable once conntrack considers it established, and that happens on the handshake’s last ACK. So that packet takes the slow path and creates the entry, and the GET is the first packet to use it. At the other end, the first FIN removes the entry, so the teardown goes the long way too.
The edges did not grow with the amount of data. For a 20 MB download with offload on, 6 packets went through the forward chain. With offload off, 4,948 did.
Does the shortcut matter?
I measured the router’s CPU with iperf3 from the Mac mini to the MacBook Air, at fixed rates so both modes did the same work, with the runs interleaved off, on, off, on. The router also runs Docker, which changes how the LAN bridge is filtered and costs the shortcut a lot; that gets its own post. For these numbers I stopped Docker and turned off the bridge setting it adds, so they should be close to a stock OpenWrt install. CPU is the average over all four cores.
| Test | Offload off | Offload on |
|---|---|---|
| TCP upload, 400 Mbit/s | 14.0% CPU | 11.8% CPU |
| TCP download, 400 Mbit/s | 17.2% CPU | 13.9% CPU |
| UDP, 256-byte packets, max | 79 Mbit/s, 39k pps | 182 Mbit/s, 89k pps |
For big TCP transfers the difference was small. The counters hint at why: in the upload test, 751 MB crossed the forward chain in about 25,000 packets, 30 KB each on average, far bigger than anything that fits on the wire. So the kernel merges consecutive segments before it forwards them, and at 400 Mbit/s that leaves fewer than 2,000 forwarding decisions per second for the shortcut to save.
Small UDP packets are not merged like that, and there the picture changed. One core sat at 98% in both modes. With offload off it forwarded about 39,000 packets per second and dropped the rest; with offload on, about 89,000.
From what I read, the shortcut has a cost too: packets on it skip the firewall hooks, so a new rule does not apply to connections that are already in the flowtable.
What I would do differently
I would start with a Linux client and a wired machine on the far side. The USB adapter’s link and the Wi-Fi on the far end caused most of the noise before I measured anything useful.
I would check the link before blaming the router. A stall on a USB network adapter is worth one ethtool --show-eee first.
And I would plan the load test before running it. My first attempt ran each mode once, all “off” runs first, at whatever rate Wi-Fi allowed, and it showed offload making things slower. Fixed rates and interleaved runs told a different story.
If I get a router with a hardware NAT engine, it becomes a third lane in the animation, one where packets 4 to 7 should never reach the CPU at all.