Fun with pf and rdomains

Given that network was working but doing a ping on a LAN machine was sending ICMP request but not getting any replies from rdomain 0, but was working fine from rdomain 1 I gave a shot at sniffing traffic on my server and see what was actually going on in the "pipes".

Some important details here, my server was on the LAN with IP address 192.168.2.3 on em0, my laptop had IP addrees 192.168.2.101 on em0 in rdomain 1 and IP address 10.0.1.2 on wg in rdomain 0.

To sniff the traffic on my server I ssh'd in and ran the following command which was just logging ICMP requests on the default interface (em0 for my server):

doas tcpdump 'icmp[icmptype] = icmp-echo'

And then from my laptop I ran first:

route -T1 exec ping 192.168.2.3

Which worked fine and the following was shown in tcpdump(8):

16:53:55.242014 192.168.2.101 > 192.168.2.3: icmp: echo request
16:53:55.242098 192.168.2.101 > 192.168.2.3: icmp: echo request
16:53:56.250806 192.168.2.101 > 192.168.2.3: icmp: echo request
16:53:56.250887 192.168.2.101 > 192.168.2.3: icmp: echo request

I then ran another ping from my laptop but this time from rdomain 0:

ping 192.168.2.3

Which resulted in 100% packet loss and no replies, but when I saw what was bing displayed in tcpdump(8), the issue became self-evident:

16:54:08.332822 10.0.1.2 > 192.168.2.3: icmp: echo request
16:54:08.332902 10.0.1.2 > 192.168.2.3: icmp: echo request
16:54:09.340634 10.0.1.2 > 192.168.2.3: icmp: echo request
16:54:09.340645 10.0.1.2 > 192.168.2.3: icmp: echo request

The source IP address was the same as my wg0 address in rdomain 0, NAT was lacking basically. Changing the rule to:

match out on rdomain 0 to { 192.168.0.0/16 } nat-to (em0) rtable 1

And then it worked fine. This took me around 3 hours.

I was able to reach a server on my LAN from rdomain 0 on my laptop, and the ICPM replies were able to flow back because in the default configuration pf(4) allows stateful traffic no matter where it comes from. But what if I want to reach a service on rdomain 0 on my laptop from my LAN server? That's where things become a bit unorthodox because I don't think many people have a use case for running services on their laptop and have it reachable from the LAN. Or if they do, they probably don't use OpenBSD and rdomains to separate traffic between VPN and non-VPN. And to me it just made more sense to have the "reverse" rule allowing services running in rdomain 0 being reachable from the LAN. I put reverse in quote because it isn't actually straightforward, it's more akin to the contraposition in logic than a simple negation or inversion. The first thing I tried was the following:

match in on rdomain 1 from { 192.168.0.0/16 } rtable 0

It didn't work, so I again started with sniffing traffic on my laptop this time. I used the following command which logs both ICMP requests and replies on interface em0 (the one on the LAN):

doas tcpdump -i em0 'icmp[0] = 8 or icmp[0] = 0'

And with the previous pf rules the ping was not getting any replies and when looking at the tcpdump(8) logs nothing seemed out of place despite the missing replies:

17:26:31.338368 192.168.2.3 > 192.168.2.101: icmp: echo request
17:26:32.339194 192.168.2.3 > 192.168.2.101: icmp: echo request
17:26:33.339185 192.168.2.3 > 192.168.2.101: icmp: echo request
17:26:34.339191 192.168.2.3 > 192.168.2.101: icmp: echo request

The replies must then either be blocked by the firewall (unlikely) or go somewhere they shouldn't (another NAT-like issue). In the hope of cathing the missing replies, I tried to sniff the network on the other interfaces with:

doas tcpdump -i wg0 'icmp[0] = 8 or icmp[0] = 0'
doas tcpdump -i lo0 'icmp[0] = 8 or icmp[0] = 0'

But nothing! Things started to get weird and I wasn't really sure where to look at next…

On the #openbsd IRC channel someone then mentionned to look at pflog0, which is a network interface that logs traffic going through rules marked as log. I then proceeded to modify the rule like so:

match in log on rdomain 1 from { 192.168.0.0/16 } rtable 0

And in order to see the logged traffic I had to run the following:

doas tcpdump -v -e -n -i pflog0

And that's when I actually saw nothing… I was at a loss again, but not giving up I tried to add a log statement to the general pass rule, just to see if I could actually see something. And that is whin I saw it:

pass out on wg0: [rewritten: src 192.168.2.101:54615, dst 192.168.2.101:8] 192.168.2.3 > 192.168.2.101: icmp: echo request

And there it was, an address like 192.168.2.101 had nothing to do on the wg0 interface (which has the 10.0.1.2 address). So it really was anothe NAT-like issue. That's how I came to try the followin, just to see, again:

match in log on rdomain 1 from { 192.168.0.0/16 } nat-to (wg0) rtable 0

Which showed the following in the logs:

match in on em0: [rewritten: src 10.0.1.2:32045, dst 192.168.2.101:8] 192.168.2.3 > 192.168.2.101: icmp: echo request

And that's when the "AHA" moment struck and I understood the difference between the nat-to and rdr-to statement in pf.conf(5). The nat-to modified the src field and the rdr-to will most probably modify the dst field. And this is what I wanted, the dst field should match the IP address of wg0 (10.0.1.2) for it to work. So naturally I expected the following to work and it did:

match in log on rdomain 1 from { 192.168.0.0/16 } rdr-to (wg0) rtable 0

And the log entry for this one showed what I wanted to:

match in on em0: [rewritten: src 192.168.2.3:35314, dst 10.0.1.2:8] 192.168.2.3 > 192.168.2.101: icmp: echo request

At that point, everything worked as I wanted it to.

The last adjustement I want to make before wrapping up is to specify all the from and to in the rules. By default pf(4) infers from/to any when left unspecified and firewall rules should always match as closely as possible. So the last tweak looked like this:

match out on rdomain 0 from (egress) to { 192.168.0.0/16 } nat-to (em0) rtable 1
match in on rdomain 1 from { 192.168.0.0/16 } to (em0) rdr-to (wg0) rtable 0

And we just have make such to have all the interface names between parenthesis for pf to handle IP address chaenges dynamically on those interfaces. This is escpecially important if the interface uses DHCP or gets his IP address assigned later after boot. If you don't put the parenthesis in such a scenario, pf might just block all network traffic until you reload it with something like:

doas pfctl -f /etc/pf.conf