My transportable (micropc) homelab has the following specs.
I'm running Proxmox 9.0.11 with the (Linux) proxmox-kernel-6.17 (soon to be but not yet default).
As people know with proxmox and all Linux KVM virtualisation solution most of the time, for performance, storage is either virtio-blk, or scsi on virtio-scsi single emulated controler.
In my situation the underlying virtual machine storage is on the NVME SSD as LVM volumes.
I have 3 virtual machines as an experimental setup.
I tried both the pkgsrc fio (3.19) benchmark version and the same version as the one in almalinux repositories (3.35) compiled from source on the NetBSDs VMs and get the same results.
The goal of trying the same version on Linux and NetBSD as well as using the same exact working fio parameters to have something "comparable".
The following commands was run on all VMs.
I must precise that I never run the command on two VMs at the same time and that they are the sole VMs/containers/workloads running on the Proxmox system.
- 1) Linux virtio-blk
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=64
...
fio-3.35
Starting 4 threads
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=3011MiB/s][w=3011 IOPS][eta 00m:00s]
write_throughput: (groupid=0, jobs=4): err= 0: pid=17950: Thu Oct 30 20:01:50 2025
write: IOPS=3130, BW=3131MiB/s (3283MB/s)(918GiB/300058msec); 0 zone resets
slat (usec): min=3, max=622, avg=31.43, stdev= 8.96
clat (msec): min=12, max=326, avg=80.93, stdev= 9.06
lat (msec): min=12, max=326, avg=80.96, stdev= 9.06
clat percentiles (msec):
| 1.00th=[ 61], 5.00th=[ 75], 10.00th=[ 78], 20.00th=[ 79],
| 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 81], 60.00th=[ 81],
| 70.00th=[ 82], 80.00th=[ 83], 90.00th=[ 85], 95.00th=[ 87],
| 99.00th=[ 120], 99.50th=[ 130], 99.90th=[ 171], 99.95th=[ 211],
| 99.99th=[ 300]
bw ( MiB/s): min= 1536, max= 4356, per=100.00%, avg=3133.45, stdev=70.96, samples=2397
iops : min= 1536, max= 4356, avg=3133.25, stdev=70.95, samples=2397
lat (msec) : 20=0.01%, 50=0.03%, 100=97.12%, 250=2.84%, 500=0.02%
cpu : usr=2.59%, sys=0.09%, ctx=104044, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
issued rwts: total=0,939328,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=3131MiB/s (3283MB/s), 3131MiB/s-3131MiB/s (3283MB/s-3283MB/s), io=918GiB (985GB), run=300058-300058msec
Disk stats (read/write):
vda: ios=0/955150, merge=0/46, ticks=0/1202556, in_queue=1202602, util=92.54%
- 2) NetBSD scsi with virtio-scsi-single
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=64
...
fio-3.35
Starting 4 threads
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=442MiB/s][w=441 IOPS][eta 00m:00s]
write_throughput: (groupid=0, jobs=4): err= 0: pid=8190: Thu Oct 30 20:12:30 2025
write: IOPS=474, BW=476MiB/s (499MB/s)(139GiB/300243msec); 0 zone resets
slat (usec): min=16, max=7360, avg=54.05, stdev=64.82
clat (msec): min=363, max=680, avg=536.68, stdev=56.25
lat (msec): min=364, max=680, avg=536.73, stdev=56.24
clat percentiles (msec):
| 1.00th=[ 397], 5.00th=[ 435], 10.00th=[ 447], 20.00th=[ 468],
| 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 558], 60.00th=[ 558],
| 70.00th=[ 567], 80.00th=[ 575], 90.00th=[ 592], 95.00th=[ 600],
| 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 659], 99.95th=[ 659],
| 99.99th=[ 684]
bw ( KiB/s): min= 7926, max=1035193, per=100.00%, avg=514464.07, stdev=29885.72, samples=2213
iops : min= 4, max= 1009, avg=500.48, stdev=29.19, samples=2213
lat (msec) : 500=26.63%, 750=73.38%
cpu : usr=1.58%, sys=48.62%, ctx=9107687, majf=0, minf=105676866
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
issued rwts: total=0,142592,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=139GiB (150GB), run=300243-300243msec
- 3) NetBSD virtio-blk
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=64
...
fio-3.35
Starting 4 threads
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=439MiB/s][w=439 IOPS][eta 00m:00s]
write_throughput: (groupid=0, jobs=4): err= 0: pid=6889: Thu Oct 30 20:07:08 2025
write: IOPS=480, BW=481MiB/s (504MB/s)(141GiB/300537msec); 0 zone resets
slat (usec): min=18, max=20026, avg=53.78, stdev=97.30
clat (msec): min=358, max=773, avg=530.61, stdev=55.92
lat (msec): min=358, max=773, avg=530.66, stdev=55.91
clat percentiles (msec):
| 1.00th=[ 372], 5.00th=[ 439], 10.00th=[ 447], 20.00th=[ 460],
| 30.00th=[ 535], 40.00th=[ 542], 50.00th=[ 550], 60.00th=[ 558],
| 70.00th=[ 567], 80.00th=[ 575], 90.00th=[ 584], 95.00th=[ 592],
| 99.00th=[ 617], 99.50th=[ 634], 99.90th=[ 760], 99.95th=[ 760],
| 99.99th=[ 768]
bw ( KiB/s): min= 9925, max=1030032, per=100.00%, avg=516160.58, stdev=29960.85, samples=2232
iops : min= 6, max= 1004, avg=502.18, stdev=29.27, samples=2232
lat (msec) : 500=25.99%, 750=73.90%, 1000=0.13%
cpu : usr=1.70%, sys=47.99%, ctx=9267027, majf=0, minf=107446337
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0%
issued rwts: total=0,144320,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=481MiB/s (504MB/s), 481MiB/s-481MiB/s (504MB/s-504MB/s), io=141GiB (152GB), run=300537-300537msec
I didn't expect NetBSD to be on par with Linux, but I didn't expect a 6x difference (3000MiB/sec for the Linux guest vs 500MiB/sec for NetBSD guest).
I am willing to use NetBSD more in the future, but this kind of massive performance degradation makes it not really a good option for any write heavy workloads and databases.
I am aware this is just a write speed benchmark but I can try read as well if necessary as well as random read writes and so on so forth. You can see that I didn't use io_uring or libaio on Linux to try keeping things safe unless Linux wraps a different system under posixaio in a way I am not aware.
I decided to make the test because I already felt NetBSD sluggish on VPS servers.
I'm not so found of writing to the mailing list because i would have to create yet another throwable email since addresses are made public, but I know there are some NetBSD devs here.
