big /etc/npf_blocklist hang at boot?

smurfd

Hey,
I had the good idea to fetch a big list of ip addresses that are known crawlers or such, add them to a npf blocklist to not have crawlers doing their thing....
It all looked very good, added this to cron had it run for a few days. (realize now that i dont think i did a: service npf reload, but just the npfctl command below...
Then today had to reboot, for other things, and it would not boot.
It was stuck at the Enabling NPF stage...machine going crazy ... (heavy load)

#!/bin/sh
fetch -o /tmp/ipsum.txt https://
raw.githubusercontent.com/stamparm/ipsum/refs/heads/master/ipsum.txt
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /tmp/ipsum.txt | doas tee /etc/npf_blocklist
npfctl table "blocklist" flush

Booted into single user and moved away the npf_blocklist file, then it rebooted fine.
Its otherwise a very simple npf.conf

$ cat /etc/npf.conf
$ext_if = { inet4(re0) }
$int_if = { inet4(tap0) }
table <blocklist> type ipset file "/etc/npf_blocklist"
table <limited> type lpm
$services_tcp = { domain, 5900, 5901, 5902, 5903, 5904, 5905, 9022, 9080, 9443 }
$services_udp = { domain, ntp }
$localnet = { 192.168.6.0/24 }
alg "icmp"
# These NAT rules will dynamically select the interface address(es).
map $ext_if dynamic 192.168.6.0/24 -> ifaddrs($ext_if)
map $ext_if dynamic proto tcp 192.168.6.2 port 22 <- ifaddrs($ext_if) port 9022
map $ext_if dynamic proto tcp 192.168.6.2 port 80 <- ifaddrs($ext_if) port 9080 
map $ext_if dynamic proto tcp 192.168.6.2 port 443 <- ifaddrs($ext_if) port 9443
procedure "log" {
   log: npflog0
}
group "external" on $ext_if {
  pass stateful out final all
  block in final from <blocklist>
  pass stateful in final family inet4 proto tcp to $ext_if port 22 apply "log" 
  pass stateful in final family inet4 proto tcp to $ext_if port 80 apply "log" 
  pass stateful in final family inet4 proto tcp to $ext_if port 443 apply "log" 
  pass stateful in final proto tcp to $ext_if port $services_tcp 
  pass stateful in final proto udp to $ext_if port $services_udp
}
group "internal" on $int_if {
  block in all
  block in final from <limited>
  pass in final from $localnet
  pass out final all
}
group default {
  pass final on lo0 all
  block all
}

the big npf_blocklist file was about 2mb with ip addresses line by line, about 130k lines.

Is there a known limit somewhere or where do i need to poke to have it handle this blocklist.
Its a base v10.1 Stable install otherwise. (upgrade to v11?)

From the code it does not look like there should be trouble with the number of entries atleast.
https://github.com/NetBSD/src/blob/trunk/sys/net/npf/npf_tableset.c#L496

--- edit ---
on hardware, amd64, alot of ram (5ish year old machine).

Thanks in advance!

rvp

smurfd Is there a known limit somewhere or [...]

I think there was something about this on the mailing-lists. I don't recall any of the details. Try searching on the MLs, going back a year or three.

smurfd

thanks, i have dejavu feelings so might have read it... but i will reproduce in vm, and poke, might give hints.
(for me, found this, try: https://wiki.netbsd.org/tutorials/setting_up_blocklistd/)
found the mail thread: https://mail-index.netbsd.org/current-users/2022/12/06/msg043279.html

kc9udx

I have my own list of (so far) 45000 IP addresses I ban. It's a type ipset file in npf.conf

It takes about 2 minutes to start npf on my RockPro64, and close to half an hour on my Star64. So it takes a while to load, just to give you an idea.

smurfd

kc9udx oh thanks, do you happen to know the load during that time? like fans go super crazy or smooth sailing? 🙂

_2ish hours in vm:

$ doas service npf start
Enabling NPF /etc/npf.conf
npfctl: Argument list too long

Looking for that error in the src, it says E2BIG
either
https://github.com/NetBSD/src/blob/0c60127bc84cf080a33b71e177edee4ce7a52a25/sys/net/npf/npf_ctl.c#L206
https://github.com/NetBSD/src/blob/0c60127bc84cf080a33b71e177edee4ce7a52a25/sys/net/npf/npf_ctl.c#L271

Both called from this function:
https://github.com/NetBSD/src/blob/0c60127bc84cf080a33b71e177edee4ce7a52a25/sys/net/npf/npf_ctl.c#L568

Might have to do some python-foo on the list to maby get ranges

kc9udx

smurfd well these are fanless machines, and I'm not positive but I seem to recall 100% of one CPU (out of 4 or 6) being tied up.

I resolved my boot tie-up by loading a config without the table, and loading one with the table after booting.

It's good to know that there's an actual size limit. I wonder if you can avoid that by splitting the table in two.

If you come up with a way to automatically condense the IP addresses into ranges, I'll sure be interested in that. I plan to do something like that but don't know when I'll get to it. I've been manually doing it for now. Manually as in finding blocks of 256 that I can turn into /24s.

EDIT
I didn't even realise it, but I'm getting "Argument list too long". I did confirm about 100%CPU usage on one CPU of 4, and it took almost exactly a half hour to load, but that was with getting the "Argument list too long" error. I have now split my file into two. One is 32767 lines.

smurfd

oh okey, so you do basicly:

table <blocklist1> type ipset file "/etc/npf_blocklist1"
table <blocklist2> type ipset file "/etc/npf_blocklist2"

Will try that!

smurfd

#!/bin/sh
fetch -o /tmp/ipsum.txt https://raw.githubusercontent.com/stamparm/ipsum/refs/heads/master/ipsum.txt 2>>/dev/null 1>&2
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /tmp/ipsum.txt | doas tee /tmp/npf_blocklist 2>>/dev/null 1>&2
lines=`wc -l /tmp/npf_blocklist|awk '{print $1}'`
files=4
batch=$(( lines / files ))
for COUNTER in $(seq 0 $files); do
    start=$(( COUNTER * batch ))
    end=$(( (COUNTER + 1) * batch ))
    awk -v s="$start" -v e="$end" 'NR>=s && NR<=e' /tmp/npf_blocklist > /etc/npf_blocklist$COUNTER
done

This is a updated script that will cut the big file in files-number parts and create files in /etc named like /etc/npf_blocklist0, /etc/npf_blocklist1, ...
🙂

kc9udx

smurfd I tried that, and I'm confused. On one machine, riscv, it seems to work.

On the other, i386, I still get the error. Oddly I thought 32768 lines seemed to be the limit. But I could get a 33000 line file to work without a hitch, but not two files summing 46000. The time npf takes to process the files is proportional to the total number of lines. So I'm not sure what's happening.

kc9udx

smurfd FWIW this is what I did:

x=0; for I in $(cat banfile); do x=$(( $x + 1 )); if test $x -lt 32769; echo $I >> banfile1; else echo $I >> banfile2; fi; done

Not as universal as your solutiin but simple enough for my case.

rvp

kc9udx Unix has had split(1) since ages -- older'n me actually!.

kc9udx

rvp I've been using Unix and Unix derivatives, POSIX etc, since the early 1990s and I never stop finding things I never used before.

smurfd

the for loop was incorrect in posix, should work now. running it in my netbsd vm at the moment...
and ahh split, cool

smurfd

import ipaddress

f = open('/tmp/ipsum.txt')
s1 = f.read().split()
f.close()
start = 24  # skip header
iplist = [ipaddress.IPv4Address(_) for _ in [s1[i] for i in range(24, len(s1), 2)]]
cidrs = list(ipaddress.collapse_addresses(iplist))
print("---- lines saved: ", len(iplist) - len(cidrs))
jj = [format(j) for j in cidrs]
f = open('/tmp/ip.txt', 'w')
for j in jj: f.write(j + '\n')
f.close()

python code to group ip addresses together. i saved about ~ 1/5 lines. not enough though...

smurfd

GOOD news! On a Monday, huh!?
@kc9udx For your info!
Change:

table <blocklist> type ipset file "/etc/npf_blocklist"

table <blocklist> type const file "/etc/npf_blocklist"

none of the blocklist mojo is needed (well in this case you need to remove the comments, using the regex in 1st script). Swollows it whole, within a secound!!! in a vm, mind you 🙂
~ 123000 ips 🙂
Hussah!!

kc9udx

smurfd well this is awesome; I'm not sure why I never tried that. I'll have to think about strategy though because my table is added to and deleted from regularly. Additional IP addresses won't be a problem (by using a separate ipset table), but deleting may be.

kc9udx

I don't understand why: even when using table type const, on i386, it takes a very long time, 13 minutes, reading a list of IP addresses 50,000 lines long, and outputs ’npfctl: Argument list too long’
I'm not having this problem on riscv. The same file takes 4 seconds and does not net in an error.

smurfd

If you are talking about i386 as in actually i386 (and not amd64), without knowing, i could guess it has todo with it only supporting 32bit. Not 64bit+ as riscv.
(and if you are talking about amd64, i used v10.1 (on hardware) and v11.0 (in VM) NetBSD)

kc9udx

Yes it's actually i386, 32 bit. I don't know why that would matter; but I don't know anything about the inner workings of npf.

Basically, there doesn't appear to be any difference between using ipset and const.

kc9udx

Hmm another difference with i386 machine is it is NetBSD 10.1 vs 11.0 on the riscv machine.