security concerns port forwarding to web server on localhost

feurry

I want to run caddy as an unprivileged user. Since that user won't be able to bind ports 80 and 443, I figured I could port forward them with npf.

I added these port forwarding rules.

map $ext_if dynamic 127.0.0.1 port 8080 <- $ext_ip port 80
map $ext_if dynamic ::1 port 8080 <- $ext_ip port 80
map $ext_if dynamic 127.0.0.1 port 8443 <- $ext_ip port 443
map $ext_if dynamic ::1 port 8443 <- $ext_ip port 443

Then allow the traffic with this, and everything works as I wanted.

pass stateful proto tcp to $ext_ip port { http, https }

Is this a reasonable way to secure caddy? Is it secure? Also, if caddy gets hacked, has my whole localhost network been exposed?

I noticed that some examples for nginx and caddy have reverse proxies pointing to localhost:9000 for fastcgi, so I figured it must be secure enough to just run caddy on localhost:8080 and 8443.

Jay

Check if this helps
https://www.unitedbsd.com/d/1406-caddy-service
I am not familiar with Caddy @naguam can probably help you

feurry

I have seen that post. It doesn't go into details about setting up the chroot, though I could figure that out probably with sandboxctl. I see @naguam made a user & group caddy in the rc script, but that user still wouldn't be able to use ports 80 and 443 unless the rc scripts do something special I'm missing. So setting up a chroot doesn't really solve the problem as far as I can tell, though it would add some extra protection I'm sure.

@Jay It's not really specific to caddy, it could be any program. I'm concerned about forwarding web traffic to localhost. I don't know enough to know if that's acceptable or a terrible idea. Hopefully someone here can chime in on that.

There's plenty of examples, like https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html

fastcgi_pass  localhost:9000;

So I'm assuming it's safe to do what I'm doing, but there's also plenty of guides out there where people are running caddy as root! On FreeBSD you get this message when you install caddy, and they have portacl as well.

Note while Caddy currently defaults to running as root:wheel, it is strongly
recommended to run the server as an unprivileged user, such as www:www --

https://github.com/freebsd/freebsd-ports/blob/main/www/caddy/files/pkg-message.in

naguam

Hi,

Just read, and I have info to provide.
Maybe I should have made a tutorial.

Gonna make a thorough answer when I'll have more time.

But as far as I know in terms of security going through localhost as long as the firewall is properly configured is not more of a liability.
Caddy running as root seem more risky.

naguam

My super ugly incomplete barbaric dirty script with notes.
It could use improvement or a replacement by an configuration management tool for more than a single hobby server.
Keep this script under hand even after the script is run, it contains notes for npf and blocklistd among other things.
(Don't forget to scroll down in some code blocks)

setup.sh with domain to be adapted.

#/bin/sh

# Create caddy user and password.
# Install caddy and mozilla-certs and curl and rsync with pkgin (pkgsrc).

rm -rf /var/caddy
mkdir /var/caddy

# Sets can also be downloaded from NetBSD servers.
tar -C /var/caddy/ -xf /usr/INSTALL/base.tar.xz
tar -C /var/caddy/ -xf /usr/INSTALL/etc.tar.xz
# Unfortunate I didn't manage to just run Caddy with chroot without having to bring a whole rootfs.

# If caddy is updated, copy + chown again before service restart.
cp /usr/pkg/bin/caddy /var/caddy/usr/bin/caddy
cp /etc/resolv.conf /var/caddy/etc/resolv.conf

# Prepare chroot caddy locations.
mkdir /var/caddy/etc/caddy
mkdir /var/caddy/caddy
mkdir /var/caddy/var/www/example.com

# Since NetBSD 10, mozilla-certs is not expected to be necessary, as OS embedded trusted CA certs.
# However here, just in case, I pass it manually to caddy.
# If the package is updated copy and chown to be done again.
cp -a /usr/pkg/share/mozilla-rootcerts/cacert.pem /var/caddy/caddy/cacert.pem
cp Caddyfile /var/caddy/etc/caddy/

cp index.html /var/caddy/var/www/example.com/

# Barbaric chowns, anyway I just run Caddy as a user.
# Didnt use sandboxctl that automates the chroot creation.
chown -R caddy:caddy /var/caddy/etc/caddy
chown -R caddy:caddy /var/caddy/etc/resolv.conf
chown -R caddy:caddy /var/caddy/caddy
chown -R caddy:caddy /var/caddy/var/www
chown -R caddy:caddy /var/caddy/var/log
chown -R caddy:caddy /var/caddy/var/run


# Future services to enable onces their own settings are configured.

# npf=YES
# npfd=YES
# blocklistd=YES
# blocklistd_flags=-r

# caddy=YES

# Can be wise to also setup the following rc.conf things.
# hostname=
# defaultroute=
# defaultroute6=
# ip6mode=

# The example of blocklistd fitted my case.

# cp /usr/share/examples/blocklistd.conf /etc/blocklistd.conf
# echo "create" > ifconfig.npflog0
# /etc/npf_blocklist.conf (to put ips in there)
# create npf configuration
# https://pub.nethence.com/bsd/blacklistd

# sysctl -w net.bpf.jit=1
# sysctl -w net.bpf.maxbufsize=4194304

# chmod 400 /etc/npf.conf /etc/npf_blocklist /etc/blocklistd.conf

# https://wiki.netbsd.org/tutorials/setting_up_blocklistd/

# Do not forget to setup the caddy rc file in /etc/rc.d.

Don't forget the remaining relevant files.

npf.conf with wired_if to be updated.

$wired_if = "wm0"
$wired_addrs = ifaddrs($wired_if)

set bpf.jit on;
alg "icmp"

procedure "log" {
    # Send log events to npflog0, see npfd(8)
    log: npflog0
}

group "wired" on $wired_if {
    # Placeholder for blocklistd (configuration separate) to add blocked hosts
    ruleset "blocklistd"

    # Allow SSH on wired interface and log all connection attempts
    pass stateful in on $wired_if proto tcp to $wired_addrs port ssh apply "log"

    # HTTP
    pass stateful in on $wired_if proto tcp to $wired_addrs port http
    pass stateful in on $wired_if proto tcp to $wired_addrs port https

    # Quic
    # pass stateful in on $wired_if proto udp to $wired_addrs port http
    # pass stateful in on $wired_if proto udp to $wired_addrs port https
}

group default {
    # Default deny, otherwise last matching rule wins
    block all apply "log"

    # Don't block loopback
    pass on lo0 all

    # Allow incoming DHCP server responses
    # pass in family inet4 proto udp from any port bootps to any port bootpc
    # pass in family inet6 proto udp from any to any port "dhcpv6-client"

    # Allow IPv6 ICMP
    pass family inet6 proto ipv6-icmp all

    # Allow incoming IPv4 pings
    pass in family inet4 proto icmp icmp-type echo all
    pass in family inet4 proto icmp icmp-type echoreply all
    pass in family inet4 proto icmp icmp-type timxceed all
    pass in family inet4 proto icmp icmp-type unreach all
    pass in family inet4 proto icmp icmp-type paramprob all
    pass in family inet4 proto icmp icmp-type sourcequench all

    # Allow being tracerouted
    pass in proto udp to any port 33434-33600

    # Allow incoming mDNS traffic from neighbours
    # pass in proto udp to any port mdns

    # Allow all outbound traffic
    pass stateful out all
}

# HTTP
map $wired_if dynamic proto tcp 213.X.X.X port 8000 <- 213.X.X.X port 80
map $wired_if dynamic proto tcp 213.X.X.X port 4443 <- 213.X.X.X port 443

# Quic (Issues getting it working, falling back to HTTP2)
# map $wired_if dynamic proto udp 213.X.X.X port 8000 <- 213.X.X.X port 80
# map $wired_if dynamic proto udp 213.X.X.X port 4443 <- 213.X.X.X port 443

# HTTP
map $wired_if dynamic proto tcp 2a10::XXXX:XXXX port 8000 <- 2a10::XXXX:XXXX port 80
map $wired_if dynamic proto tcp 2a10::XXXX:XXXX port 4443 <- 2a10::XXXX:XXXX port 443

# Quic (Issues getting it working, falling back to HTTP2)
# map $wired_if dynamic proto udp 2a10::XXXX:XXXX port 8000 <- 2a10::XXXX:XXXX port 80
# map $wired_if dynamic proto udp 2a10::XXXX:XXXX port 4443 <- 2a10::XXXX:XXXX port 44

Caddyfile listening on user allowed ports.

{
        http_port 8000
        https_port 4443
        log {
                output file /var/log/caddy.log
        }
        servers {
                protocols h1 h2
        }
        storage file_system /caddy
        acme_ca_root /caddy/cacert.pem
        acme_ca https://api.buypass.com/acme/directory
        email myredactedemail@example.com
}

example.com {
        root * /var/www/example.com
        encode gzip
        file_server
        log {
                output file /var/log/access.log
        }
        header {
                Strict-Transport-Security max-age=31536000
                Cache-Control max-age=31536000
                X-Content-Type-Options nosniff
                X-Frame-Options DENY
                #               Content-Security-Policy "default-src 'self'; style-src 'self'; script-src 'self'; font-src 'self'; img-src 'self'; form-action 'self'; connect-src 'self'; frame-ancestors 'none';"
                Referrer-Policy strict-origin-when-cross-origin
        }
}

caddy rc service which contains the directives to let the rc run the chroot on its own (check proper manual and system scripts).
Not sure it is perfect but it worked fine for me including reload signals.

!/bin/sh
#
# $NetBSD: caddy ,v 1.1 2024/07/07 19:47:00 naguam Exp $
#

# PROVIDE: caddy
# REQUIRE: DAEMON
# KEYWORD: shutdown

$_rc_subr_loaded . /etc/rc.subr

name="caddy"
rcvar=$name
command="/usr/bin/${name}"
caddy_user="caddy"
caddy_group="caddy"
caddy_chroot="/var/${name}"
caddy_env="HOME=/caddy"
chpid="/var/run/${name}.pid"
pidfile="${caddy_chroot}${chpid}"
cfg="/etc/caddy/Caddyfile"
required_files="${caddy_chroot}${cfg}"
required_dir="${caddy_chroot}"
command_args="start --config ${cfg} --pidfile ${chpid}"
extra_commands="reload"

load_rc_config $name

run_rc_command "$1"

Don't get me wrong, if the script is ugly, as long as the components are properly updated when needed, it should be secure enough.
But I'm taking feedback if needed for the npf config and the rest.

I used to host https://cloudbsd.xyz with this config. It is now on a free git pages provider.

If this pleases @Jay or @rvp we can link it in this https://www.unitedbsd.com/d/1406-caddy-service so people coming to it can reach it. I know https://it-notes.dragas.net/about-me/ linked it once.

I hope I haven't left some forgotten personal details here.

All of this must not be blindly copied but understood and adapted for one's case.
It is more of a documentation/template resource.
Also the chroot location may not follow the proper convention, documentation is still kinda sparse on that for NetBSD.

Happy to help.

feurry

naguam Wow! thanks for the detailed post, and your previous post about caddy. It looks like we're doing pretty much the same things, except for the chroot. I thought setting up a chroot was more involved... didn't realize the rc scripts can handle it so simply.

Here's my rc script, without the chroot stuff. I started with yours and trimmed it, and added a note about setting caddy=yes which I saw in some other rc scripts.

#!/bin/sh
#
# $NetBSD: caddy, v 1.0 2025/06/12
#

# PROVIDE: caddy
# REQUIRE: DAEMON
#
# You will need to set some variables in /etc/rc.conf to start caddy:
#
# caddy=YES

$_rc_subr_loaded . etc/rc.subr

name="caddy"
rcvar=${name}
command="/usr/pkg/bin/${name}"
pidfile="/var/run/${name}.pid"
conf_file="/usr/pkg/etc/Caddyfile"
required_files="${conf_file}"
command_args="start --config ${conf_file} --pidfile ${pidfile}"
extra_commands="reload"
reload_cmd="${command} reload --config ${conf_file}"

load_rc_config $name
run_rc_command "$1"

At the moment I'm trying to reverse proxy with nginx to caddy, so that I don't have to port forward with the firewall. From what I've read a reverse proxy is probably better than port forwarding/nat, though practically speaking, for my use I'm probably not gaining much. Anyway..it's not going very well at the moment (the automatic tls part of caddy and port 443) It seems kinda silly since I could just do everything with nginx and avoid all this, but it's also fun.

I'm gonna look into using unix sockets as well, because those have permissions and are better than using localhost. I've used them for php, but not for a web server. So I might end up with haproxy or nginx doing a reverse proxy on a unix socket to caddy. We'll see what I can kludge together.

I've also emailed https://www.bsdnow.tv/ with related questions. Hopefully they pick it up and have some advice.

Jay

naguam done Linked it 👍