kernel config

The main advantages of a minimal kernel are a lower memory footprint, smaller size and faster boot times.
Some interesting kernel options:

# set max number of allowed processes to 1044
maxusers        64
# Alternate buffer queue strategies for better responsiveness under high
# disk I/O load.
options 	BUFQ_READPRIO
options 	BUFQ_PRIOCSCAN

# SVR4-style scheduling  
# [https://fa.netbsd.tech.kern.narkive.com/EUTnlnvm/implementation-of-sched-m2-scheduler]
# Didn't run benchs on this and it's unclear whether the M2 scheduler
# still brings any performance benefit against SHED_4BSD. Most of SCHED_M2
# features have been incorporated inside SHED_4BSD. SCHED_4BSD now has extra
# awareness of hyperthreading/SMT/NUMA and is more actively maintained in
# general.
## no options      SCHED_4BSD
# options         SCHED_M2

While nowadays the difference may be negligible outside of embedded systems, you could still opt for a MODULAR kernel, adding the settings listed above. The MODULAR kernel pulls the GENERIC config and attempts to kick out anything which has been turned into LKMs.

The are sections of the GENERIC kernel which you may as well comment out entirely, if you're running a modern amd64 PC: SCSI controllers, RAID controllers, IDE and ISA controllers, PMCIA network interfaces, PCI sound cards, TV and Radio cards, Hyper-V and VirtIO drivers.

resource limits

Append this to your /etc/login.conf in order to increase 'capabilities' (loosen session resource limits) for the staff class, system-wide:

#
# Staff have fewer restrictions
#
staff:\
        :datasize-cur=1024M:\                                                           

        :datasize-max=infinity:\
        :maxproc-max=1024:\                                                

        :maxproc-cur=128:\
        :requirehome@:

Recompile a hashed class capabilities database:

$ cap_mkdb  /etc/login.conf

Now change your regular user's login class to staff:

$ usermod -L staff <username>

kernel state tunables

Set those sysctl(7) values permanently at boot time but adding what follows to your /etc/sysctl.conf

## KERNEL
# increase max PAGE_SIZE 
kern.ipc.shmmaxpgs=580205
# increase max file descriptors
kern.maxfiles=12286
# Number of kernel threads to use for NFS client
vfs.nfs.iothreads=6

## TCP/IP stack
# automatic network buffers resize
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_auto=1
# Toggle JIT compilation of new filter programs on
net.bpf.jit=1
# 4x BPF buffer max size
net.bpf.maxbufsize=4194304
# 16MB buffers upper limit
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
# Maximum allowed output queue length
net.inet.ip.ifq.maxlen = 4096
# Maximum segment size
net.inet.tcp.mssdflt=1072

FFSv2 WAPBL

WAPBL provides data journaling on NetBSD for the FFSv{1,2,2ea} filesystems .
Journaling improves writing performance noticeably by reducing synchronization, especially when creating a large number of inodes. It also avoids having to go through a thorough file-systems check during a crash recovery , speeding up boot time significantly in those cases.

Enabling WAPBL on NetBSD is as easy as appending 'log' to FS mount options for each FFSv{1,2, 2ea} partition found in /etc/fstab:

/dev/wd0a		/	        ffs	rw,log		 1 1
/dev/wd0e		/usr	        ffs	rw,log		 1 2
/dev/wd0f		/var	        ffs	rw,log		 1 2
/dev/wd0g	        /home	        ffs	rw,log		 1 2

noatime

Another possible tweak is to disable last access timestamps on filesystems. These decreases the number of metadata updates to files, and is useful for optimizing read performance. A side effect will be reducing power consumption on laptops. This is a default option on evbarm, since most people will be using SD cards on SBC.
If you wish to use this feature, add noatime to mount options inside /etc/fstab.

nodevmtime

Note: only supported on NetBSD 10.0 - onward

The last useful mount option is nodevmtime, which disables updating modification times on device special files. This option is useful on laptops or other systems that perform power management. Needless to say that it only makes sense to use it on the root partition where /dev resides.

SSD TRIM

You can check check whether your wd0 disk supports ATA TRIM with:

$ atactl wd0 identify | grep TRIM
TRIM supported

1) discard

A discard mount option is available on wd* and ld* storage devices, as well as dk* wedges. This keeps discarding unused blocks in background. However, discard is still deemed experimental, and may lead to data corruption. It is also mutually exclusive with WAPBL. Use it at your own risk.

2) blkdiscard (8)

Note: only supported on NetBSD 10.0 - onward

Before (re-)installing NetBSD it's possible to drop to a shell and TRIM a complete drive to re-initialize it before installation:

# blkdiscard -v /dev/rwd0d

(This will wipe the drive!)

As an alternative, on a running system, it's also possible to write a blank file, then use blkdiscard to mark sections of your drive unused:

# dd if=/dev/zero of=./testfile bs=4m count=10000
# blkdiscard -v ./testfile

This could be performed as a routine maintenance task, e.g. by cron(8).

If you have free space left on device, you may use a LVM volume as a target to blkdiscard, in order to trim the remaining space before installing another system:

$ modload dm && service lvm onestart

$ gpt add -a 2m -t ffs -l "LVM” wd0     # create a standard partition occupying all space left on device    

/dev/rwd2: Partition 7 added: 49f48d5a-b10e-11dc-b99b-0019d1879648 965218304 11554816

$ lvm pvcreate /dev/rdk9                # initialize the partition as LVM (e.g., assuming it's dk9)
Physical volume "/dev/rdk9" successfully created

$ lvm vgcreate vg0 /dev/rdk9            # create a volume group on the partition
Volume group "vg0" successfully created

$ lvm lvcreate -l100%FREE -n free vg0   # create a logical volume taking all the free space left on vg0
Logical volume "free" created

$ blkdiscard -v /dev/vg0/free           # discard invalid data pages on the 'free' LV
going to fdiscard(2) on /dev/vg0/free from byte 0 for 5913968640 bytes, 33554432 at a time

Now remove the logical and physical LVM volumes, and you're ready to go.

tmpfs

mount a tmpfs disk on directories with a high r/w turnover so as to decrease access time and increase maximum throughput, by adding suitable entries to /etc/fstab

tmpfs		/tmp		tmpfs	rw,-m=1777,-s=4295467008
tmpfs		/var/shm	tmpfs	rw,-m1777,-sram%25
tmpfs		/var/tmp/pkg	tmpfs	rw,-m1777,-sram%50

Now, providing you have enough ram, you can also build packages on volatile memory by adding:

# build on tmpfs
WRKOBJDIR = /var/tmp/pkg

to your mk.conf(5)

virtual memory management

UVM, the new Virtual Memory subsystem, is responsible for managing access to the computer's memory resources on NetBSD. From uvm(9):

UVM has two kernel-level processes: pagedaemon and swapper. The pagedaemon process sleeps until physical memory becomes scarce. When that happens, pagedaemon is awoken.
It scans physical memory, paging out and freeing memory that has not been recently used. The swapper process swaps in runnable processes that are currently swapped out, if there is room.

There are six sysctl parameters for controlling what sort of pages should (not) be freed. For each of the three types of pages (anonymous, executable, file cache) there is a lower limit and an upper limit, which correspond to percentages of physical memory. To display their default values:

$ sysctl vm. | egrep 'anon|file|exec'
vm.anonmin = 10
vm.filemin = 10
vm.execmin = 5
vm.anonmax = 80
vm.filemax = 50
vm.execmax = 30

As you can see, by default NetBSD will use up to 50% of available memory for file I/O cache (if it's available, why not use it?) and release it no sooner should it be required by any process. However, such behavior may not be desirable in specific cases (file server, news server, or whatever process writes a lot of data to disk), where it might lead to intense swapping and quick resource exhaustion.
In this case, you may try to adjust some of the default vm.* values in /etc/sysctl.conf, eg. to increase the memory to always be made available for anon pages while decreasing file cache:

# the minimum percentage of memory always (made) available for anonymous 
# pages. The default is 10.
#
vm.anonmin=40

# the maximum percentage of memory that will be reclaimed from other uses for
# file data cache. The default is 50.
#
vm.filemax=25

power management

Note: the official wiki provides a useful article on this topic. See: how to balance cpu performance, temperature and power drawn.

1) powerd

On a laptop, especially if running on battery power, it's a good idea to enable the power management daemon for sysmon(4), aka powerd(8). To do so, append powerd=YES to /etc/rc.conf.
All configuration of powerd is encapsulated into scripts that are run when power management events occur. The daemon will look for the scripts from the directory /etc/powerd/scripts.
To understand how ACPI events are handled, examine those scripts and edit events and thresholds to your needs.

2) backlight

If acpivga(4) supports brightness control on your system, you can set backlight intensity via `sysctl':

$ sysctl hw.acpi | grep brightness
hw.acpi.acpiout15.brightness = 100
$ sysctl -w hw.acpi.acpiout15.brightness=80 
hw.acpi.acpiout15.brightness: 100 -> 80

Default values can be stored in /etc/sysctl.conf so that they're set by init upon boot:

# ACPI preferences 
hw.acpi.acpiout15.brightness=80

On Intel HD graphics supported by i915drm, it's also possible to use sysutils/intel-backlight. This provides a useful userspace front-end for backlight management.

$ intel_backlight decr
current backlight value: 70% (656/937)
set backlight to 60% (562/937)

For non-intel graphics, check out x11/xbacklight.

If NetBSD doesn't support backlight management on your system, you may still adjust screen brightness, gamma, and color temperature at software level, using x11/xbrightness and x11/redshift respectively. Note: this won't actually reduce power consumption.
To reduce perceived brightness in a dark room:

$ xbrightness 61439 1    # default: 65535 1

To apply some gamma correction and enhance perceived brillance at software level on a X11 session (I find it useful when using dark colorschemes), you can add to your ~/.xinitrc:

# xbrightness BRIGHTNESS [ GAMMA ]
/usr/pkg/bin/xbrightness 65535 0.95

To correct the color temperature depending on the time of the day with redshift:

# redshift [-l LAT:LON] [-t DAY:NIGHT]
# temperature is expressed in Kelvin degrees
/usr/pkg/bin/redshift  -l 41.717066:12.710975 -Pt 6500:5000

3) suspending

If your sleep button /function key is not properly detected, then, on a graphical session, you can remap the XF86Sleep special key using xmodmap(1).
To display the current key association:

$ xmodmap -pke | grep -i sleep
keycode 150 = XF86Sleep NoSymbol XF86Sleep

To change it:

$ xmodmap -e "keycode 103 = XF86Sleep" #remaps End as XF86SLeep

Keycodes may be inspected with:

xev | awk -F'[ )]+' '/^KeyPress/ { a[NR+2] } NR in a { printf "%-3s %s\n", $5, $8 }

At this point, you can use x11/xbindkeys (or whatever built-in keybinding facility your wm includes), to bind XF86Sleep to:

/sbin/sysctl -w hw.acpi.sleep.state=3    #suspend system to  RAM

Or, on i386:

$ /usr/sbin/apm -z

If you wish to your system to be suspended when a lid-closed event is detected, edit the pressed and released event cases in /etc/powerd/scripts/lid_switch to match those found in /etc/powerd/scripts/sleep_button:

case "${2}" in
pressed)
        if /sbin/sysctl -q hw.acpi.sleep.state; then
                /sbin/sysctl -w hw.acpi.sleep.state=3
        fi
        ;;
[...]
esac

4) cpu frequency scaling

Print all the supported CPU clock frequencies, alongside the current and target (maximum) frequency:

$ sysctl machdep.cpu.frequency
machdep.cpu.frequency.target = 2400
machdep.cpu.frequency.current = 2400
machdep.cpu.frequency.available = 3601 3600 3400 3200 3000 2800 2600 2400 2200 2000 1800 1600 1400 1200 1000 800

You can adjust target frequency to your needs. To increase battery life I could lower it to 2 GHz:

$ sysctl -w machdep.cpu.frequency.target=2000
machdep.cpu.frequency.target: 2400 -> 2000

Whereas to maximize performance I could bring it up to 3.6 GHz. Default values can be stored in/etc/sysctl.conf. A more dynamic approach would be to adjust the target frequency depending on the temperature of the first 2 cores (this may be implemented in a cron job):

#!/bin/sh

t0=`envstat | grep cpu0 | awk '{print $3}' | cut -b 1,2`
t1=`envstat | grep cpu1 | awk '{print $3}' | cut -b 1,2`

if [ "$t0" -gt 75 ] || [ "$t1" -gt 75 ]; then
        sysctl -w machdep.cpu.frequency.target=800
        echo "Setting CPU frequency to 800 MHz" 
elif [ "$t0" -gt 70 ] || [ "$t1" -gt 70 ]; then
        sysctl -w machdep.cpu.frequency.target=1200
        echo "Setting CPU frequency to 1.2 GHz"
elif [ "$t0" -gt 65 ] || [ "$t1" -gt 65 ]; then
        sysctl -w machdep.cpu.frequency.target=1800
        echo "Setting CPU frequency to 1.8 GHz"
elif [ "$t0" -gt 60 ] || [ "$t1" -gt 60 ]; then
        sysctl -w machdep.cpu.frequency.target=2400
        echo "Setting CPU frequency to 2.4 GHz"
elif [ "$t0" -gt 50 ] || [ "$t1" -gt 50 ]; then
        sysctl -w machdep.cpu.frequency.target=3000
        echo "Setting CPU frequency to 3 GHz"
elif [ "$t0" -gt 40 ] || [ "$t1" -gt 40 ]; then
        sysctl -w machdep.cpu.frequency.target=3600
        echo "Setting CPU frequency to 3.6 GHz"
else
        echo "No CPU frequency adjustment needed"  

fi

Envstat allows to set warning and critical thresholds for CPU core temperature inside /etc/envsys.conf:

coretemp0 {
 sensor0 {
  critical-max = 80C;
  critical-min = 75C;
  warning-max = 70C;
  warning-min = 65C;
 }
}
coretemp1 {
 sensor0 {
  critical-max = 80C;
  critical-min = 75C;
  warning-max = 70C;
  warning-min = 65C;
 }
}
coretemp2 {
 sensor0 {
  critical-max = 80C;
  critical-min = 75C;
  warning-max = 70C;
  warning-min = 65C;
 }
}
coretemp3 {
 sensor0 {
  critical-max = 80C;
  critical-min = 75C;
  warning-max = 70C;
  warning-min = 65C;
 }
}

Now all you have to do to enforce the new levels is:

$ echo envsys=YES >> /etc/rc.conf
$ service envsys start
Setting new sensor properties.
$ envstat
                      Current  CritMax  WarnMax  WarnMin  CritMin  Unit
[coretemp0]
  cpu0 temperature:    31.000   80.000   70.000   65.000   75.000  degC
[coretemp1]
  cpu1 temperature:    30.000   80.000   70.000   65.000   75.000  degC
[coretemp2]
  cpu2 temperature:    29.000   80.000   70.000   65.000   75.000  degC
[coretemp3]
  cpu3 temperature:    29.000   80.000   70.000   65.000   75.000  degC

You may want at this point to edit the /etc/powerd/scripts/sensor_temperature script to make it lower the CPU frequency whenever a warning or critical threshold is hit.

case "${2}" in
[..]
critical-over)
        case "${1}" in
        coretemp*)
          logger -p warning "${0}: ($1) critical limit exceeded [${3}]" >&1
          /sbin/sysctl -w machdep.cpu.frequency.target=800                 

          ;;                                              

        *)  

          logger -p warning "${0}: ($1) critical limit exceeded [${3}]" >&1
          /sbin/shutdown -p now "${0}: CRITICAL TEMPERATURE! SHUTTING DOWN."
          ;;                                                                

        esac
        exit 0
        ;;    

[..]   

warning-over)
        case "${1}" in
        coretemp*)
          logger -p warning "${0}: ($1) warning limit exceeded [${3}]" >&1
          /sbin/sysctl -w machdep.cpu.frequency.target=1600               

          ;;                                               

        *)  

          logger -p warning "${0}: ($1) warning limit exceeded [${3}]" >&1
          ;;                                                              

        esac
        exit 0
        ;;
[...]

For a more fine-grained control, you can use sysutils/estd. While running in background, estd will automatically keep CPU frequency and battery usage low if the system is mostly idle and speed it up when your workload demands it. To install estd and activate it:

$ pkg_add estd
$ cp /usr/pkg/share/examples/rc.d/estd /etc/rc.d/
$ echo estd=YES >> /etc/rc.conf
$ service estd estart

Edit the estd service parameters (estd_flags) in /etc/rc.conf to your needs. For example,

  estd_flags="-d -l 40 -h 70 -b -M 2200"

This maximizes battery lifetime by limiting CPU-frequency to 2.2 GHz and switching to lower speeds fast. In addition, if the percentage of idle states goes below 40%, estd will lower the CPU frequency to conserve battery. If it goes above 70%, it will increase the frequency.
Instead, to increase performance by running at least at 2.4GHz and switching to higher speeds real fast (aggressive frequency-switching strategy), you may use:

estd_flags="-d -a -m 2400"

Refer to estd(1).

5) idle mode for inactive storage drives

You can use atactl(8) to place IDE and (S)ATA storage devices in idle and standby mode after a given period of inactivity (in seconds):

$ atactl wd1 setidle 180
$ atactl wd1 setstandby 900

This requires root privileges, so to make it a default system-wide setting, it could be placed inside /etc/rc.local:

if [ -b /dev/wd0 ]; then
  if /sbin/sysctl hw.disknames | grep wd0 > /dev/null
  then
        /sbin/atactl wd0 setidle 180
        /sbin/atactl wd0 setstandby 900
        echo "Setting idle and standby timers on wd0..."
  else
        echo "rc.local: wd0 not found"
  fi
fi

if [ -b /dev/wd1 ]; then
  if /sbin/sysctl hw.disknames | grep wd1 > /dev/null
  then
        /sbin/atactl wd1 setidle 180
        /sbin/atactl wd1 setstandby 900
        echo "Setting idle and standby timers on wd1..."
  else
        echo "rc.local: wd1 not found"
  fi
fi

Upon reboot you should see:

[running /etc/rc.d/local]
Starting local daemons:
Setting idle and standby timers on wd0...
Setting idle and standby timers on wd1...

atactl also supports a sleep commad which places the specified device into Sleep mode. According to the man page:

This mode will consume less power than Standby mode, but requires a device reset to resume operation. Typically the wd(4) driver performs this reset automatically, but this should still be used with caution.

If I try to use the sleep command on a mounted rotational hard disk (let's say wd0) while the system is running, I can hear it becoming completely quiet. As I attempt to dd a test file on the disk from /dev/random, it starts to spin and is factually brought back to an active state, but the kernel throws these errors:

[ 1282.6863580] wd0: soft error (corrected) xfer 2b8
[ 1282.6863580] wd0: soft error (corrected) xfer 218
[ 1282.6863580] wd0: soft error (corrected) xfer 178
[ 1282.6863580] wd0: soft error (corrected) xfer d8
[ 1282.6863580] wd0: soft error (corrected) xfer 38

For PCIe (NVM Express) drives use the nvmectl(8) utility instead.

$ nvmectl power -l nvme0    # displays Power States supported by manufacturer

Smartmontools have been supporting NetBSD's nvmectl since 6.6; smartctl can help understanding how power states are handled on your device:

$ smartctl -a /dev/nvme0
[...]
Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0        0       0
 1 +    12.00W       -        -    0  0  0  0        0       0
 2 +    10.00W       -        -    0  0  0  0        0       0

In this case, Power State 2 (PS2) limits the maximum power to 10W, significantly reducing power consumption by putting the drive in idle state. This however comes at the expense of a significant performance impact both in terms of r/w latency and i/o throughput.
To set the power state of nvme0:

$ nvmectl power -p 2 nvme0    # put the drive in idle state
$ nvmectl power -p 0 nvme0    # put the drive in normal (active) state

6) X11 settings

To activate the X11 screensaver function and set the blank timeout:

xset s blank
xset s 600

To active the DPMS (Energy Star) features and set the timeout for the standby suspend, and off states:

xset +dpms               # powersaving
xset dpms 600 1200 1800  # screen blanking

You may use this script to toggle powersaving features:

#!/bin/sh

if (xset q|grep --quiet "DPMS is Enabled");then
        xset s off -dpms
else
        xset s on +dpms
        xset dpms 600 1200 1800 
fi

Additional useful settings:

xset b 0                 # muted bell
xset m 55/20 4           # mouse acceleration
xset r rate 250 40       # faster keypresses

pkgsrc options

1) compiler optimization

Use devel/cpuflags to determine the best compiler flags to set for your CPU. In addition, set the number of make jobs to match the numbers of CPU cores online.
You can specify those settings inside your /etc/mk.conf

MAKE_JOBS          = $(/sbin/sysctl -n hw.ncpuonline)
.if defined(BSD_PKG_MK)
#.include "/etc/mk.extra.conf"
GZIP?              = --fast
CPUFLAGS+          = -O2 -pipe -fomit-frame-pointer -mfpmath=sse -msse3 -march=native
.endif

2) speed up recompilation by caching previous compilations

After installing devel/ccache, you may add:

PKGSRC_COMPILER    = ccache gcc
CCACHE_DIR         = ${HOME}/.cache/ccache

to your /etc/mk.conf

3) Use mktools.

pkgtools/mktools is a collection of tools written in C to improve the performance of certain sections of the pkgsrc mk infrastructure where shell is too slow.
To use them, also add:

    PKGSRC_USE_MKTOOLS  = yes

4) Enable faster distfile fetching

## FETCHING DISTFILES
FETCH_FAILOVER       = YES
# resume downloads 
PKG_RESUME_TRANSFERS = YES
# fetch using wget
.if exists(${LOCALBASE}/bin/wget)
FETCH_CMD            = ${LOCALBASE}/bin/wget
.endif
FETCH_BEFORE_ARGS    = –passive-ftp
FETCH_RESUME_ARGS    = -c
FETCH_OUTPUT_ARGS    = -O
# preferred domains list (this is set for TZ=Europe/Rome and provided as example)
MASTER_SORT_RANDOM= no
MASTER_SORT          = .eu .it .ch .at .fr .de .es .nl .uk .at .be .cz .pl .se .fi .no .dk

tips

1) Add swap space to a running system

 %  dd if=/dev/zero bs=1M count=128 of=<FS>/swap
 % chmod 600 <FS>/swap
 % swapctl -a -p 1 <FS>/swap

Providing file-based swap is slower than a swap partition, giving the additional swap space a priority of 1 (through '-p 1', least is a value of '0' ) forces it be relied upon for paging only when the swap partition has been used up.

2) check for memory in use

Display total used buffers in megabytes with vmstat(1)

 vmstat -s | awk '
/ bytes per page$/ { bpp = $1 }
/ cached file pages$/ { cfp = $1 }
/ cached executable pages$/ { cep = $1 }
END { print((cfp + cep) * bpp / 1024 / 1024); }

using top(1)

  % top -n 1 | head -n 5 | tail -n 2
  Memory: 665M Act, 74M Inact, 14M Wired, 113M Exec, 119M File, 3045M Free
  Swap: 6144M Total, 6144M Free

using sysutils/free

$ free -m

     total       used       free    buffers
Mem:          3923        885       3037        232
Swap:         6144          0       6144

On arches where procps supports Linux extensions, you can also read /proc/meminfo to get a percentage of free memory:

$ awk 'FNR== 2 {print $3/$2 *100}' /proc/meminfo