What could cause NPF NAT to suddenly quit?

kc9udx

A couple of weeks ago I built a NAT router/gateway/firewall with a RockPro64 running NetBSD 10.1 and using NPF. After some frustration I started to better understand NPF and had a lot of fun making this work. So for at least a week, this RockPro64 has been idling along, routing packets galore. Tonight, my wife was streaming a video from some proprietary service, watching a graduation ceremony. There wasn't anything else significant happening on our LAN. Suddenly NAT wasn't forwarding packets anymore. Everything else was working except LAN devices could not see the WAN. All 6 CPUs on the RockPro64 were at idle and RAM usage was low. Restarting NPF didn't solve the problem, as expected. Removing and re-initialising the WAN interface didn't help. The only thing that helped was rebooting the RockPro64.

What could cause this, or how could I have determined what caused it?

Jay

kc9udx hi
What you can do when this happens next time is"

#npfctl stats
Or
#npfctl show | wc -l
Also ifconfig -a to check for interface errors.
And if you have logging enabled, check the logs with tcpdump -n -e -i npflog0

JuvenalUrbino

kc9udx All 6 CPUs on the RockPro64 were at idle and RAM usage was low.

So I assume you were able to normally reach the firewall within the LAN and that the device hadn't panicked?

kc9udx Removing and re-initialising the WAN interface didn't help.

Did you check if the firewall could ping anything outside before restarting network services? Are you using a static IP or dhcpcd on the WAN interface (in the latter case, did you also remember to restart dhcpcd?)
If you're on a home network (dynamic IP) and have configured NPF to forward packets from a given external IPv4/6 address (instead of using ifaddrs()), is it possible that the home router lost connection and reinitialized itself with a different IP?

Is the device configured as host DMZ on the router? Has it an internal fixed IP assigned?

What about DNS? Does your device also act as a local caching/forwarding name server? What about router DNS settings (where does it forwards packets to [ISP's DNS I presume])?

As @Jay suggested, I would use tcpdump to inspect logs (implying you enabled logging on forwarding rules), traceroute to identiify routing problems, systat ifstat to display network traffic statistics.

Inspect /var/log/messages, and/or attach a serial console to check for network interface related kernel errors (might be a bug in the driver, in which case it should be reported, or missing firmware).

Also, you may want to use netstat to check if you're being targeted for DDoS attacks, and possibly hit a bandwidth limit.

kc9udx

Jay thanks.

I didn't know about npfctl stats I'll have to see what that tells. I normally only have logging on inbound traffic; I could've changed that at the time.

ifconfig didn't reveal anything at the time.

JuvenalUrbino

Jay
#npfctl show | wc -l

What's the purpose of this?

Also ifconfig -a to check for interface errors.

Since when ifconfig displays interface errors?

Jay

JuvenalUrbino
To get an idea of the number of connections, you can pipe the output to wc -l? But I might be wrong

JuvenalUrbino

Jay npfctl show prints the loaded ruleset, not active connections or network traffic going through a given interface 😉

kc9udx

JuvenalUrbino well thanks to the weather I have some free time, so,

Yes, it's nearly headless. So I almost have to access by LAN, which I normally do. It has a big TV for an emergency monitor and it won't display enough overscan to make it useful. But it was running fine and could ping addresses and canonical names on the WAN. The WAN IP is dynamic, and yes I had to restart dhcpcd to get a connection to the cable modem. The modem always gives me the same IP, per connection. This device always gets one, a different nic always gets another one, a hardware router always gets another one.
It is not a DMZ; I have planned to implement a separate device as a DMZ. (Though I did also think about just running a chroot). It is a caching DNS server, which doesn't use the ISP's DNS at all, I use one of a handful of public ones (don't recall which at the moment).

Logging was only enabled for one rule which would not help in this situation; I plan to change that.

Nothing unusual was in /etc/messages and this did not seem to be a DDoS instance; WAN access was unhindered at the device itself.

The only things that stick out are the fact that npfctl reports no config. But im not sure what to make of that. And when doing npfctl reload I get a "file exists" message. But I'm pretty sure that has happened other times and I never know why.

kc9udx

Well it happened again, and I didn't npfctl show | wc -l but that certainly would've led to what I found anyway. npf doesn't have a config loaded and shows no ruleset. It does immediately after npfctl reload but that's the only time. I'd like to answer the other questions but until the weekend I am very far away.

kc9udx

I kept doing

npfctl validate
npfctl reload
npfctl show

And it would show no config loaded (I neglected to write down exactly what it said. But something like

# filtering: active
# config: empty

)

I commented out everything in npf.conf except

set bpf.jit off
group default {
}

And kept getting the same result. But then I commented out
set bpf.jit.off
And from npfctl show I got

# filtering: active
# config: loaded

group default { # id="1"
}

Now, I uncommented everything in npf.conf (actually I restored the backup I made before I commented everything out) and it looks normal. So, something outside of npf.conf persisted (across reboots) until I wiped out everything in npf.conf. I don't know what to make of this.

I'd like to upload my npf.conf, there must be something not quite right about it. But it'll be a while before I can do this as I have to reconfigure my LAN just to get this device back on the LAN. And I'm not yet sure when this will be possible.

It might also be noteworthy that I apparently had a kernel panic on the 9th, and savecore ran out of disk space to write it. I don't recall what was happening on the 9th.

kc9udx

I am stuck at the begining again. npfctl show always nets

# filtering: active
# config: empty

This is my npf.conf:

#Need to set securelevel to 0 else have to do this
set bpf.jit off

#WAN interface
$wan_if = "axen0"
#LAN interface
$lan_if  = "awge0"
#quark_port port
$quark_port=1655
$wan_addrs = ifaddrs($wan_if)
$lan_addrs = ifaddrs($lan_if)

table <outgoing-block> type ipset file "/etc/npf_files/outgoing-block"
table <incoming-block> type ipset file "/etc/npf_files/logban"

#Application Level Gateway for ICMP
alg "icmp"

# LAN to WAN NAT
map $wan_if dynamic 10.10.0.0/16 -> $wan_addrs

# Quark/Phoniness NAT
map $lan_if dynamic 10.10.1.47 -> inet4($lan_if)
map $wan_if dynamic 10.10.1.47 -> inet4($wan_if)

procedure "log" {
    # Send log events to npflog0, see npfd(8)
    log: npflog0
}

group "wan" on $wan_if {
    block in all apply "log"

    # Placeholder for blocklistd (configuration separate) to add blocked hosts
    ruleset "blocklistd"

    block out final to <outgoing-block>
    block in final from <incoming-block>

    # Allow SSH on WAN interface and don't log any connection attempts
    pass stateful in proto tcp to any port ssh

    # Quagmire
    pass stateful in to any port tinc

    # Quark
    pass stateful in to any port $quark_port

    # allow cable modem DHCP
    pass out final proto udp from any port bootpc to any port bootps
    pass in proto udp from any port bootps to any port bootpc
    pass in proto udp from any port bootps to 255.255.255.0 port bootpc

    # Allow incoming IPv4 pings
    pass in family inet4 proto icmp icmp-type echo all

    # Allow being tracerouted
    pass in proto udp to any port 33434-33600

    # IRCd
    pass stateful in to any port ircd

    # Everything else
    pass stateful out all
}

group "lan" on $lan_if {
    pass all
}

group "quark" on tun0 {
    pass all
}

group "quagmire" on tap0 {
    pass all
}

group default {
    # Default deny, otherwise last matching rule wins
    block all

    # Don't block loopback
    pass on lo0 all

    # Allow incoming IPv4 pings
    pass in family inet4 proto icmp icmp-type echo all

    # Allow being tracerouted
    pass in proto udp to any port 33434-33600

    # IRCd
    pass in from any port ircd

    # Allow all outbound traffic
    pass stateful out all
}

This is most frustrating to me because it worked great for at least a week and nothing has changed that I'm aware of. Now I can't make it work at all.

rvp

kc9udx This is my npf.conf:

[...]
#LAN interface
$lan_if  = "awge0"
[...]

Is there an awge driver on NetBSD? Can't find it...

kc9udx

Everything has been working for about 12 hours now. The only thing I've done is load a blank configuration, flush it, and then load my configuration.

I suspect there's something npf doesn't like about my "logban" file. That is shared between and modified by three machines. So that could've (though it shouldn't have) changed. But the other two machines have never hiccupped, and I don't know what could be wrong in that file. There were a couple of IPv4 addresses in there ending in .0. The only way that's possible is if someone was using that address, so I'm confused how that happened. But that doesn't seem to have been the problem.

kc9udx

rvp Yes, or at least as far as I can say.

awge0: flags=0x8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        ec_capabilities=0x1<VLAN_MTU>
        ec_enabled=0x1<VLAN_MTU>
        address: 06:0d:54:xx:xx:xx
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::40d:xxxx:xxxx:xxxx%awge0/64 flags 0 scopeid 0x1
        inet 10.10.xxx.xxx/16 broadcast 10.10.255.255 flags 0

I see what you mean, though. I don't see anything online. But this is from the running system:

AWGE(4)                      Device Drivers Manual                     AWGE(4)

NAME
     awge - DesignWare Gigabit Ethernet

SYNOPSIS
     awge* at fdt?

DESCRIPTION
     The awge driver provides support for the DesignWare GMAC Ethernet.

SEE ALSO
     arp(4), ifmedia(4), mii(4), netintro(4), ifconfig(8)

     Synopsys, DesignWare Ethernet GMAC IP,
     https://www.synopsys.com/dw/ipdir.php?ds=dwc_ether_mac10_100_1000_unive,
     April, 2019.

HISTORY
     The name awge was chosen due to the hardware appearing in Allwinner
     boards, but awge in use in boards by other manufacturers.

     The awge device driver was written by Matt Thomas and Martin Husemann.
     It first appeared in NetBSD 7.0.

NetBSD 10.1_STABLE              April 23, 2019              NetBSD 10.1_STABLE

FWIW this machine has been rock solid since. I do notice that unless I npfctl flush once in a while some things that I wouldn't expect to persist, do seem to persist.

As long as I can find hardware to run NetBSD and npf on, I will never again buy a router appliance. This RockPro64 and what I can do with it are nothing short of amazing.

rvp

kc9udx Yes, or at least as far as I can say.

Ah, I did a find /usr/src/sys/dev -name '*awge*' and a fgrep -r awge /usr/src/sys/dev/ but didn't find it because the driver is ARM-specific (duh!) and the source is called rk_gmac.c (but attaches as awge):

$ fgrep awge /usr/src/sys/arch/arm/rockchip/files.rockchip
attach  awge at fdt with rk_gmac
$