Hey, i have created a raid5 on a NAS with 4 drives(not ssd's), and NetBSD (10.1 stable) is installed on a SSD disk. The writing of files seems very slow, i have had the same setup with FreeBSD earlier, used a zraid. Unfortunately i did not do a speed test before starting to use netbsd... and i dont have exact numbers, but i do see it being slower... Im farely sure it is some kind of silly user error! But how to find What is the tricky part. Suggestions are welcomed! I notice big difference if i copy files over the network to the SSD rather than the raid, so it is not the network It transfers about 1 TB per 24 hours, which is slow. While writing files over the network, i did a write test: ``` $ dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 141.066 secs (7433229 bytes/sec) ``` I have tried enabling cache on the drives aswell as the raid, without any notable difference... This is basicly how i did: Installed netbsd on the ssd, default install. Cleared the drives. repeat 4x: ``` $ doas dd if=/dev/zero of=/dev/wd0 bs=1024k count=1 ``` Checked size and created fs on the drives. The drives are identical, but wanted to be sure. ``` $ dmesg | grep wd0 $ doas newfs -O2 -s 7452g -F /dev/rwd0 ``` Destroy, create and add gpt x4 ``` $ doas gpt destroy wd0 $ doas gpt create -Af wd0 $ doas gpt add -t raid -l raid5@wd0 -b $(( 2048 )) -s 15628050000 wd0 ``` Create the raid ``` $ doas raidctl raid5 create 5 /dev/dk3 /dev/dk4 /dev/dk6 /dev/dk5 (mine was out of order) $ doas raidctl -I 133773311 raid5 $ doas raidctl -iv raid5 ``` Create a gpt for the raid and a partition ``` $ doas gpt create -Af raid5 $ doas gpt add -a 1024 -t ffs -l island raid5 ``` Create a fs for the raid, then mount it ``` $ doas newfs -O2 NAME=island $ doas mount /dev/dk7 /mnt ``` And this is my /etc/raid5.conf file ``` START array # numRow numCol numSpare 1 4 0 START discs /dev/dk3 /dev/dk4 /dev/dk6 /dev/dk5 START layout # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5 128 1 1 5 START queue fifo 100 ``` Thanks in advance!

Hello, It might be useful to see the output of: ``` raidctl -s raid5 raidctl -p raid5 ``` https://man.netbsd.org/raidctl.8 That may help clarify the current state of the array.

Of course, the post was quite hefty already. To me this looks good, but ... :) Also let me know if there is more output i can provide ``` $ doas raidctl -s raid5 Components: /dev/dk3: optimal /dev/dk4: optimal /dev/dk5: optimal /dev/dk6: optimal No spares. Component label for /dev/dk3: Row: 0, Column: 0, Num Rows: 1, Num Columns: 4 Version: 2, Serial Number: 1337733137, Mod Counter: 182 Clean: No, Status: 0 sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1 Queue size: 1, blocksize: 512, numBlocks: 15628049920 RAID Level: 5 Autoconfig: Yes Root partition: No Last configured as: raid5 Component label for /dev/dk4: Row: 0, Column: 1, Num Rows: 1, Num Columns: 4 Version: 2, Serial Number: 1337733137, Mod Counter: 182 Clean: No, Status: 0 sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1 Queue size: 1, blocksize: 512, numBlocks: 15628049920 RAID Level: 5 Autoconfig: Yes Root partition: No Last configured as: raid5 Component label for /dev/dk5: Row: 0, Column: 2, Num Rows: 1, Num Columns: 4 Version: 2, Serial Number: 1337733137, Mod Counter: 182 Clean: No, Status: 0 sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1 Queue size: 1, blocksize: 512, numBlocks: 15628049920 RAID Level: 5 Autoconfig: Yes Root partition: No Last configured as: raid5 Component label for /dev/dk6: Row: 0, Column: 3, Num Rows: 1, Num Columns: 4 Version: 2, Serial Number: 1337733137, Mod Counter: 182 Clean: No, Status: 0 sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1 Queue size: 1, blocksize: 512, numBlocks: 15628049920 RAID Level: 5 Autoconfig: Yes Root partition: No Last configured as: raid5 Parity status: clean Reconstruction is 100% complete. Parity Re-write is 100% complete. Copyback is 100% complete. ``` ``` $ doas raidctl -p raid5 /dev/rraid5: Parity status: clean ``` I have this warning in dmesg, but i would expect it to spew warnings if this would cause problems while copying files for example... (some trail and error was involved :D) ``` $ dmesg|grep -i raid5 [ 2.891923] dk3 at wd0: "raid5@wd0", 15628050000 blocks at 2048, type: raidframe [ 2.911923] autoconfiguration error: wd1: wedge named 'raid5@wd0' already existed, using '3e213e32-d40a-4892-b1ad-5195cce4b5bc' [ 2.931923] autoconfiguration error: wd2: wedge named 'raid5@wd0' already existed, using '7b96e126-d6ef-41c7-aa63-002bd44c8487' [ 2.951923] autoconfiguration error: wd3: wedge named 'raid5@wd0' already existed, using 'e5fdee13-efa0-47e1-b7f3-b776add189a4' [ 2.981923] raid5: RAID Level 5 [ 2.981923] raid5: Components: /dev/dk3 /dev/dk4 /dev/dk5 /dev/dk6 [ 2.981923] raid5: Total Sectors: 46884149760 (22892651 MB) [ 2.981923] raid5: GPT GUID: 8d5b1c29-5285-44af-90c9-6af131bca6c5 [ 2.981923] dk7 at raid5: "island", 46884149692 blocks at 34, type: ffs ```

The running RAID array shows: ``` sectPerSU: 32 Queue size: 1 ``` This seems different from what is shown in `/etc/raid5.conf`. This could matter for performance. It may be useful to understand why the active array is using `32 / queue 1` instead of `128 / fifo 100`.

Raid5 slow write on big disks?

smurfd

that is where user error enters i guess. Did not look closely enough at the array before writing the config file 😉
changed from 128 to 32 in raid5.conf then stopped and started the nas.
yes it got abit better.. but only a little.

$ dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 81.178 secs (12916997 bytes/sec)

Tempted to reinstall freebsd to see what speeds i got there.

Leufroy

Maybe I am missing something, but it looks like /etc/raid5.conf was not used when the RAID set was created.

From the manual, I understand that raidctl -C should be used for the first setup from a config file, but I did not see it in the commands you listed.

The manual also says that RAID5 should avoid partial stripe writes because of parity overhead, but with 4 disks, avoiding them completely is not possible.

smurfd

No you are not missing anything. Pure user error! I started with the chapter 16 raidframe document, and followed that to the teeth. but then ran into problems with the large disks, and then i started mixing instructions + making guesses... not a good way to go 😃

Thank you!
Making a new attempt. In the manual for raidctl it does however in the header: Simple RAID configuration
Mentions the command i used (they used it for a raid 0 though)

For example,
           raidctl raid0 create 0 /dev/wd0e /dev/wd1e /dev/wd2e

Anyway, initializing now for a long time, let you know after 🙂

$ doas raidctl -s raid5
Components:
            /dev/dk3: optimal
            /dev/dk4: optimal
            /dev/dk5: optimal
Spares:
            /dev/dk6: spare
Component label for /dev/dk3:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 3
   Version: 2, Serial Number: 133773311, Mod Counter: 398
   Clean: No, Status: 0
   sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 15628049920
   RAID Level: 5
   Autoconfig: No
   Root partition: No
   Last configured as: raid5
Component label for /dev/dk4:
   Row: 0, Column: 1, Num Rows: 1, Num Columns: 3
   Version: 2, Serial Number: 133773311, Mod Counter: 398
   Clean: No, Status: 0
   sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 15628049920
   RAID Level: 5
   Autoconfig: No
   Root partition: No
   Last configured as: raid5
Component label for /dev/dk5:
   Row: 0, Column: 2, Num Rows: 1, Num Columns: 3
   Version: 2, Serial Number: 133773311, Mod Counter: 398
   Clean: No, Status: 0
   sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 15628049920
   RAID Level: 5
   Autoconfig: No
   Root partition: No
   Last configured as: raid5
/dev/dk6 status is: spare.  Skipping label.
Parity status: DIRTY
Reconstruction is 100% complete.
Parity Re-write is 8% complete.
Copyback is 100% complete.

raid5.conf

START array
# numRow numCol numSpare
1 3 1

START disks
/dev/dk3
/dev/dk4
/dev/dk5

START spare
/dev/dk6

START layout
# sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
32 1 1 5

START queue
fifo 100

And lastly the change i suspect also will make alot of change, is to add the -b option to the newfs command for the drives aswell as the raid.

doas newfs -O2 -b 65536 -s 7452g -F /dev/rwd0

smurfd

--- edited ---

$ doas newfs -O2 -s 21900g -b 65536 -F NAME=island
NAME=island: 22425600.0MB (45927628800 sectors) block size 65536, fragment size 8192

$ doas dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 52.974 secs (19794163 bytes/sec)

Definitively faster... still slow though... right?

Leufroy

Could you provide more details about the hardware? What interface are the drives in the RAID 5 array using?

smurfd

the nas is one of these: https://nas-eu.ugreen.com/products/ugreen-nasync-dxp4800-plus-nas-storage
and the drives are Segate Ironwolf NAS 3.5"
Again, wish id taken the speed when i had FreeBSD on it, to have something to compare to. 🙂

rvp

smurfd Is NetBSD running on the NAS? If yes, then can you show /var/run/dmesg.boot?

smurfd

rvp sorry for late reply. Yes it runs NetBSD.
Dont have a /var/run/dmesg.log but this is my last dmesg run

$dmesg
[     1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
[     1.000000]     2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
[     1.000000]     2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023,
[     1.000000]     2024, 2025, 2026
[     1.000000]     The NetBSD Foundation, Inc.  All rights reserved.
[     1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[     1.000000]     The Regents of the University of California.  All rights reserved.

[     1.000000] NetBSD 10.1_STABLE (GENERIC) #0: Fri May  8 15:48:23 UTC 2026
[     1.000000]         mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC
[     1.000000] total memory = 7950 MB
[     1.000000] avail memory = 7660 MB
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] Kernelized RAIDframe activated
[     1.000000] RTC BIOS diagnostic error 0x3f<config_unit,memory_size,fixed_disk,invalid_time>
[     1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
[     1.000004] efi: systbl at pa 75f65018
[     1.000004] mainbus0 (root)
[     1.000004] ACPI: RSDP 0x000000007453F014 000024 (v02 ALASKA)
[     1.000004] ACPI: XSDT 0x000000007453E728 0000F4 (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: FACP 0x00000000744B0000 000114 (v06 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: DSDT 0x0000000074428000 087B18 (v02 ALASKA A M I    01072009 INTL 20200717)
[     1.000004] ACPI: FACS 0x0000000074572000 000040
[     1.000004] ACPI: FIDT 0x0000000074427000 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[     1.000004] ACPI: SSDT 0x00000000744B2000 00038C (v02 PmaxDv Pmax_Dev 00000001 INTL 20200717)
[     1.000004] ACPI: SSDT 0x0000000074421000 005D34 (v02 CpuRef CpuSsdt  00003000 INTL 20200717)
[     1.000004] ACPI: SSDT 0x000000007441E000 002AA5 (v02 SaSsdt SaSsdt   00003000 INTL 20200717)
[     1.000004] ACPI: SSDT 0x000000007441A000 0033E9 (v02 INTEL  IgfxSsdt 00003000 INTL 20200717)
[     1.000004] ACPI: SSDT 0x000000007440C000 00D7C2 (v02 INTEL  TcssSsdt 00001000 INTL 20200717)
[     1.000004] ACPI: HPET 0x00000000744B1000 000038 (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: APIC 0x000000007440B000 0001DC (v05 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: MCFG 0x000000007440A000 00003C (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: SSDT 0x0000000074408000 001F1A (v02 ALASKA Ther_Rvp 00001000 INTL 20200717)
[     1.000004] ACPI: UEFI 0x00000000744F5000 000048 (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: NHLT 0x0000000074407000 00002D (v00 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: LPIT 0x0000000074406000 0000CC (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: SSDT 0x0000000074402000 002A83 (v02 ALASKA PtidDevc 00001000 INTL 20200717)
[     1.000004] ACPI: SSDT 0x00000000743F8000 0092F4 (v02 ALASKA TbtTypeC 00000000 INTL 20200717)
[     1.000004] ACPI: DBGP 0x00000000743F7000 000034 (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: DBG2 0x00000000743F6000 000054 (v00 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: DMAR 0x00000000743F5000 000088 (v01 INTEL  EDK2     00000002      01000013)
[     1.000004] ACPI: FPDT 0x00000000743F4000 000044 (v01 ALASKA A M I    01072009 AMI  01000013)
[     1.000004] ACPI: SSDT 0x00000000743F3000 000EE1 (v02 INTEL  xh_adlLP 00000000 INTL 20200717)
[     1.000004] ACPI: SSDT 0x00000000743EF000 0039DA (v02 SocGpe SocGpe   00003000 INTL 20200717)
[     1.000004] ACPI: SSDT 0x00000000743EB000 0039DA (v02 SocCmn SocCmn   00003000 INTL 20200717)
[     1.000004] ACPI: PHAT 0x00000000743EA000 00085C (v01 ALASKA A M I    00000005 MSFT 0100000D)
[     1.000004] ACPI: TPM2 0x00000000743E9000 00004C (v04 ALASKA A M I    00000001 AMI  00000000)
[     1.000004] ACPI: WSMT 0x0000000074405000 000028 (v01 ALASKA A M I    01072009 AMI  00010013)
[     1.000004] ACPI: 12 ACPI AML tables successfully acquired and loaded
[     1.000004] ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x20, 120 pins
[     1.000004] cpu0 at mainbus0 apid 0
[     1.000004] cpu0: Use lfence to serialize rdtsc
[     1.000004] cpu0: CPU base freq 2500000000 Hz
[     1.000004] cpu0: CPU max freq 4400000000 Hz
[     1.000004] cpu0: TSC freq CPUID 2496000000 Hz
[     1.000004] cpu0: Intel(R) Pentium(R) Gold 8505, id 0x906a4
[     1.000004] cpu0: node 0, package 0, core 0, smt 0
[     1.000004] cpu1 at mainbus0 apid 1
[     1.000004] cpu1: Intel(R) Pentium(R) Gold 8505, id 0x906a4
[     1.000004] cpu1: node 0, package 0, core 0, smt 1
[     1.000004] cpu2 at mainbus0 apid 24
[     1.000004] cpu2: Intel(R) Pentium(R) Gold 8505, id 0x906a4
[     1.000004] cpu2: node 0, package 0, core 12, smt 0
[     1.000004] cpu3 at mainbus0 apid 26
[     1.000004] cpu3: Intel(R) Pentium(R) Gold 8505, id 0x906a4
[     1.000004] cpu3: node 0, package 0, core 13, smt 0
[     1.000004] cpu4 at mainbus0 apid 28
[     1.000004] cpu4: Intel(R) Pentium(R) Gold 8505, id 0x906a4
[     1.000004] cpu4: node 0, package 0, core 14, smt 0
[     1.000004] cpu5 at mainbus0 apid 30
[     1.000004] cpu5: Intel(R) Pentium(R) Gold 8505, id 0x906a4
[     1.000004] cpu5: node 0, package 0, core 15, smt 0
[     1.000004] acpi0 at mainbus0: Intel ACPICA 20221020
[     1.000004] acpi0: X/RSDT: OemId <ALASKA,  A M I ,01072009>, AslId <AMI ,01000013>
[     1.000004] acpi0: MCFG: segment 0, bus 0-255, address 0x00000000c0000000
[     1.000004] ACPI: Dynamic OEM Table Load:
[     1.000004] ACPI: SSDT 0xFFFFFD3516804808 0005E4 (v02 PmRef  Cpu0Ist  00003000 INTL 20200717)
[     1.000004] ACPI: Dynamic OEM Table Load:
[     1.000004] ACPI: SSDT 0xFFFFFD368454A908 0001AB (v02 PmRef  Cpu0Psd  00003000 INTL 20200717)
[     1.000004] ACPI: Dynamic OEM Table Load:
[     1.000004] ACPI: SSDT 0xFFFFD5800F325000 001BAF (v02 PmRef  ApIst    00003000 INTL 20200717)
[     1.000004] ACPI: Dynamic OEM Table Load:
[     1.000004] ACPI: SSDT 0xFFFFD5800F328000 001349 (v02 PmRef  ApPsd    00003000 INTL 20200717)
[     1.000004] acpi0: SCI interrupting at int 9
[     1.000004] acpi0: fixed power button present
[     1.000004] timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
[     1.051959] hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
[     1.051959] timecounter: Timecounter "hpet0" frequency 19200000 Hz quality 2000
[     1.052463] acpiec0 at acpi0 (H_EC, PNP0C09-1): not present
[     1.052463] com0 at acpi0 (UAR1, PNP0501-0): io 0x3f8-0x3ff irq 4
[     1.052463] com0: ns16550a, 16-byte FIFO
[     1.052463] attimer1 at acpi0 (TIMR, PNP0100): io 0x40-0x43,0x50-0x53 irq 0
[     1.052463] acpivga0 at acpi0 (GFX0): ACPI Display Adapter
[     1.052463] acpiout0 at acpivga0 (DD01, 0x0001): ACPI Display Output Device
[     1.052463] acpiout1 at acpivga0 (DD02, 0x0002): ACPI Display Output Device
[     1.052463] acpiout2 at acpivga0 (DD03, 0x0003): ACPI Display Output Device
[     1.052463] acpiout3 at acpivga0 (DD04, 0x0004): ACPI Display Output Device
[     1.052463] acpiout4 at acpivga0 (DD05, 0x0005): ACPI Display Output Device
[     1.052463] acpiout5 at acpivga0 (DD06, 0x0006): ACPI Display Output Device
[     1.052463] acpiout6 at acpivga0 (DD07, 0x0007): ACPI Display Output Device
[     1.052463] acpiout7 at acpivga0 (DD08, 0x0008): ACPI Display Output Device
[     1.052463] acpiout8 at acpivga0 (DD09, 0x0009): ACPI Display Output Device
[     1.052463] acpiout9 at acpivga0 (DD0A, 0x000a): ACPI Display Output Device
[     1.052463] acpiout10 at acpivga0 (DD0B, 0x000b): ACPI Display Output Device
[     1.052463] acpiout11 at acpivga0 (DD0C, 0x000c): ACPI Display Output Device
[     1.052463] acpiout12 at acpivga0 (DD0D, 0x000d): ACPI Display Output Device
[     1.052463] acpiout13 at acpivga0 (DD0E, 0x000e): ACPI Display Output Device
[     1.052463] acpiout14 at acpivga0 (DD0F, 0x000f): ACPI Display Output Device
[     1.052463] acpiout15 at acpivga0 (DD1F, 0x001f): ACPI Display Output Device
[     1.052463] acpiout15: brightness levels: [0-100]
[     1.052463] acpiout16 at acpivga0 (DD2F, 0x001f): ACPI Display Output Device
[     1.052463] acpiout16: brightness levels: [0-100]
[     1.052463] acpivga0: connected output devices:
[     1.052463] acpivga0:   0x0000: Unknown Output Device, head 0, bios detect
[     1.052463] acpivga0:   0x0000: Unknown Output Device, head 0, bios detect
[     1.052463] AWAC (ACPI000E) at acpi0 not configured
[     1.052463] igpio0 at acpi0 (GPI0, INTC1055): mem 0xfd6e0000-0xfd6effff,0xfd6d0000-0xfd6dffff,0xfd6a0000-0xfd6affff,0xfd690000-0xfd69ffff irq 14
[     1.052463] gpio0 at igpio0: 277 pins
[     1.052463] acpibut0 at acpi0 (SLPB, PNP0C0E): ACPI Sleep Button
[     1.052463] acpiwmi0 at acpi0 (WFDE, PNP0C14-DSarDev): ACPI WMI Interface
[     1.052463] acpiwmibus at acpiwmi0 not configured
[     1.052463] acpiwmi1 at acpi0 (WFTE, PNP0C14-TestDev): ACPI WMI Interface
[     1.052463] acpiwmibus at acpiwmi1 not configured
[     1.052463] PEPD (INT33A1) at acpi0 not configured
[     1.052463] acpibut1 at acpi0 (PWRB, PNP0C0C): ACPI Power Button
[     1.052463] TPM (MSFT0101) at acpi0 not configured
[     1.052463] acpifan0 at acpi0 (FAN0, PNP0C0B-0): ACPI Fan
[     1.052463] acpifan1 at acpi0 (FAN1, PNP0C0B-1): ACPI Fan
[     1.052463] acpifan2 at acpi0 (FAN2, PNP0C0B-2): ACPI Fan
[     1.052463] acpifan3 at acpi0 (FAN3, PNP0C0B-3): ACPI Fan
[     1.052463] acpifan4 at acpi0 (FAN4, PNP0C0B-4): ACPI Fan
[     1.052463] acpitz0 at acpi0 (TZ00): cpu0 cpu1 cpu2 cpu3 cpu4 cpu5
[     1.052463] acpitz0: active cooling level 0: 100.0C
[     1.052463] acpitz0: active cooling level 1: 55.0C
[     1.052463] acpitz0: active cooling level 2: 50.0C
[     1.052463] acpitz0: active cooling level 3: 45.0C
[     1.052463] acpitz0: active cooling level 4: 40.0C
[     1.052463] acpitz0: levels: critical 105.0 C
[     1.052463] ACPI: Enabled 4 GPEs in block 00 to 7F
[     1.052463] pci0 at mainbus0 bus 0: configuration mode 1
[     1.052463] pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
[     1.052463] pchb0 at pci0 dev 0 function 0: Intel Alder Lake (U15,1+4) Host (rev. 0x04)
[     1.052463] genfb0 at pci0 dev 2 function 0: Intel Alder Lake Graphics (rev. 0x0c)
[     1.052463] genfb0: framebuffer at 0x4000000000, size 1024x768, depth 32, stride 4096
[     1.052463] genfb0: shadow framebuffer enabled, size 3072 KB
[     1.052463] wsdisplay0 at genfb0 kbdmux 1: console (default, vt100 emulation)
[     1.052463] wsmux1: connecting to wsdisplay0
[     1.052463] drm at genfb0 not configured
[     1.052463] xhci0 at pci0 dev 13 function 0: Intel Alder Lake USB-C Host (xHCI) (rev. 0x04)
[     1.052463] xhci0: 64-bit DMA
[     1.052463] xhci0: interrupting at msi0 vec 0
[     1.052463] xhci0: Unknown revision (32). Set to 3.1.
[     1.052463] xhci0: xHCI version 1.20
[     1.052463] usb0 at xhci0: USB revision 3.1
[     1.052463] usb1 at xhci0: USB revision 2.0
[     1.052463] xhci1 at pci0 dev 20 function 0: Intel 600 Series PCH-LP USB 3.2 Gen 2x1 xHCI (rev. 0x01)
[     1.052463] xhci1: 64-bit DMA
[     1.052463] xhci1: interrupting at msi1 vec 0
[     1.052463] xhci1: xHCI version 1.20
[     1.052463] usb2 at xhci1: USB revision 3.1
[     1.052463] usb3 at xhci1: USB revision 2.0
[     1.052463] Intel 600 Series PCH-LP Shared SRAM (RAM memory, revision 0x01) at pci0 dev 20 function 2 not configured
[     1.052463] Intel 600 Series PCH-LP I2C 0 (miscellaneous serial bus, revision 0x01) at pci0 dev 21 function 0 not configured
[     1.052463] Intel 600 Series PCH-LP HECI 1 (miscellaneous communications, revision 0x01) at pci0 dev 22 function 0 not configured
[     1.052463] ppb0 at pci0 dev 28 function 0: Intel 600 Series PCH-LP PCIe RP 1 (rev. 0x01)
[     1.052463] ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x2 @ 8.0GT/s
[     1.052463] pci1 at ppb0 bus 1
[     1.052463] pci1: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.052463] aq0 at pci1 dev 0 function 0: Aquantia AQC113 10 Gigabit Network Adapter (rev. 0x03)
[     1.052463] aq0: Atlantic2 B0, F/W version 1.3.31
[     1.052463] aq0: Etheraddr: 6c:1f:f7:8d:5b:c8
[     1.052463] ppb1 at pci0 dev 28 function 4: Intel 600 Series PCH-LP PCIe RP 5 (rev. 0x01)
[     1.052463] ppb1: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 8.0GT/s
[     1.052463] pci2 at ppb1 bus 2
[     1.052463] pci2: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.052463] nvme0 at pci2 dev 0 function 0: Phison product 5013 (rev. 0x01)
[     1.052463] nvme0: NVMe 1.3
[     1.052463] nvme0: for admin queue interrupting at msix3 vec 0
[     1.052463] nvme0: YSO128GTLCW-E3C-2, firmware EDFMG1.5, serial 511250430164004583
[     1.052463] nvme0: for io queue 1 interrupting at msix3 vec 1 affinity to cpu0
[     1.052463] nvme0: for io queue 2 interrupting at msix3 vec 2 affinity to cpu1
[     1.052463] nvme0: for io queue 3 interrupting at msix3 vec 3 affinity to cpu2
[     1.052463] nvme0: for io queue 4 interrupting at msix3 vec 4 affinity to cpu3
[     1.052463] nvme0: for io queue 5 interrupting at msix3 vec 5 affinity to cpu4
[     1.052463] nvme0: for io queue 6 interrupting at msix3 vec 6 affinity to cpu5
[     1.052463] ld0 at nvme0 nsid 1
[     1.052463] ld0: 119 GB, 15566 cyl, 255 head, 63 sec, 512 bytes/sect x 250069680 sectors
[     1.052463] ppb2 at pci0 dev 29 function 0: Intel 600 Series PCH-LP PCIe RP 9 (rev. 0x01)
[     1.052463] ppb2: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x2 @ 8.0GT/s
[     1.052463] pci3 at ppb2 bus 3
[     1.052463] pci3: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.052463] ahcisata0 at pci3 dev 0 function 0: ASMedia product 1164 (rev. 0x02)
[     1.052463] ahcisata0: 64-bit DMA
[     1.052463] ahcisata0: AHCI revision 1.31, 4 ports, 32 slots, CAP 0xee349f23<SXS,PMD,SAM,ISS=0x3=Gen3,SAL,SALP,SSS,SSNTF,SNCQ,S64A>
[     1.052463] ahcisata0: interrupting at msi4 vec 0
[     1.052463] atabus0 at ahcisata0 channel 0
[     1.052463] atabus1 at ahcisata0 channel 1
[     1.052463] atabus2 at ahcisata0 channel 2
[     1.052463] atabus3 at ahcisata0 channel 3
[     1.052463] ppb3 at pci0 dev 29 function 2: Intel 600 Series PCH-LP PCIe RP 11 (rev. 0x01)
[     1.052463] ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x1 @ 8.0GT/s
[     1.052463] ppb3: link is x1 @ 5.0GT/s
[     1.052463] pci4 at ppb3 bus 4
[     1.052463] pci4: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.052463] igc0 at pci4 dev 0 function 0, 64-bit DMA: Intel(R) Ethernet Controller I226-V (rev. 0x08)
[     1.052463] igc0: txrx interrupting at msix5 vec 0
[     1.052463] igc0: txrx interrupting at msix5 vec 1
[     1.052463] igc0: txrx interrupting at msix5 vec 2
[     1.052463] igc0: txrx interrupting at msix5 vec 3
[     1.052463] igc0: link interrupting at msix5 vec 4
[     1.052463] igc0: PHY: OUI 0x984fee model 0x0005, rev. 8, PHY FW version 0x8877
[     1.052463] igc0: NVM image version 2.14, EtrackID 80000290
[     1.052463] igc0: Ethernet address 6c:1f:f7:8d:5b:c9
[     1.052463] pcib0 at pci0 dev 31 function 0: Intel 600 Series PCH-LP eSPI (rev. 0x01)
[     1.052463] hdaudio0 at pci0 dev 31 function 3: HD Audio Controller
[     1.052463] hdaudio0: interrupting at msi6 vec 0
[     1.052463] hdaudio0: HDA ver. 1.0, OSS 9, ISS 7, BSS 0, SDO 1, 64-bit
[     1.052463] hdaudio0: autoconfiguration error: no codecs found
[     1.052463] hdaudio0: autoconfiguration error: device driver failed to attach
[     1.052463] ichsmb0 at pci0 dev 31 function 4: Intel 600 Series PCH-LP SMBus (rev. 0x01)
[     1.052463] ichsmb0: interrupting at ioapic0 pin 16
[     1.052463] iic0 at ichsmb0: I2C bus
[     1.052463] Intel 600 Series PCH-LP SPI (FLASH) (miscellaneous serial bus, revision 0x01) at pci0 dev 31 function 5 not configured
[     1.052463] isa0 at pcib0
[     1.052463] pcppi0 at isa0 port 0x61
[     1.052463] spkr0 at pcppi0: PC Speaker
[     1.052463] wsbell at spkr0 not configured
[     1.052463] midi0 at pcppi0: PC speaker
[     1.052463] sysbeep0 at pcppi0
[     1.052463] attimer1: attached to pcppi0
[     1.052463] acpicpu0 at cpu0: ACPI CPU
[     1.052463] acpicpu0: C1: HLT, lat   0 us, pow     0 mW
[     1.052463] acpicpu0: P0: FFH, lat  10 us, pow 15000 mW, 2501 MHz, turbo boost
[     1.052463] acpicpu0: P1: FFH, lat  10 us, pow 15000 mW, 2500 MHz
[     1.052463] acpicpu0: P2: FFH, lat  10 us, pow 13487 mW, 2300 MHz
[     1.052463] acpicpu0: P3: FFH, lat  10 us, pow 12752 mW, 2200 MHz
[     1.052463] acpicpu0: P4: FFH, lat  10 us, pow 11325 mW, 2000 MHz
[     1.052463] acpicpu0: P5: FFH, lat  10 us, pow 10635 mW, 1900 MHz
[     1.052463] acpicpu0: P6: FFH, lat  10 us, pow  9292 mW, 1700 MHz
[     1.052463] acpicpu0: P7: FFH, lat  10 us, pow  8642 mW, 1600 MHz
[     1.052463] acpicpu0: P8: FFH, lat  10 us, pow  7382 mW, 1400 MHz
[     1.052463] acpicpu0: P9: FFH, lat  10 us, pow  6772 mW, 1300 MHz
[     1.052463] acpicpu0: P10: FFH, lat  10 us, pow  5591 mW, 1100 MHz
[     1.052463] acpicpu0: P11: FFH, lat  10 us, pow  4461 mW,  900 MHz
[     1.052463] acpicpu0: P12: FFH, lat  10 us, pow  3917 mW,  800 MHz
[     1.052463] acpicpu0: P13: FFH, lat  10 us, pow  3384 mW,  700 MHz
[     1.052463] acpicpu0: P14: FFH, lat  10 us, pow  2356 mW,  500 MHz
[     1.052463] acpicpu0: P15: FFH, lat  10 us, pow  1861 mW,  400 MHz
[     1.052463] coretemp0 at cpu0: thermal sensor, 1 C resolution, Tjmax=100
[     1.052463] acpicpu1 at cpu1: ACPI CPU
[     1.052463] acpicpu2 at cpu2: ACPI CPU
[     1.052463] coretemp1 at cpu2: thermal sensor, 1 C resolution, Tjmax=100
[     1.052463] acpicpu3 at cpu3: ACPI CPU
[     1.052463] coretemp2 at cpu3: thermal sensor, 1 C resolution, Tjmax=100
[     1.052463] acpicpu4 at cpu4: ACPI CPU
[     1.052463] coretemp3 at cpu4: thermal sensor, 1 C resolution, Tjmax=100
[     1.052463] acpicpu5 at cpu5: ACPI CPU
[     1.052463] coretemp4 at cpu5: thermal sensor, 1 C resolution, Tjmax=100
[     1.052463] cpu0: CPU base freq 2500000000 Hz
[     1.052463] cpu0: CPU max freq 4400000000 Hz
[     1.052463] cpu0: TSC freq CPUID 2496000000 Hz
[     1.052463] cpu0: TSC freq calibrated 2496005000 Hz
[     1.052463] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
[     1.131973] timecounter: Timecounter "TSC" frequency 2496005000 Hz quality 3000
[     1.138212] uhub0 at usb0: NetBSD (0x0000) xHCI root hub (0x0000), class 9/0, rev 3.00/1.00, addr 0
[     1.151965] uhub0: 4 ports with 4 removable, self powered
[     1.151965] uhub1 at usb1: NetBSD (0x0000) xHCI root hub (0x0000), class 9/0, rev 2.00/1.00, addr 0
[     1.161965] uhub1: 1 port with 1 removable, self powered
[     1.161965] uhub2 at usb2: NetBSD (0x0000) xHCI root hub (0x0000), class 9/0, rev 3.00/1.00, addr 0
[     1.171965] uhub2: 4 ports with 4 removable, self powered
[     1.171965] uhub3 at usb3: NetBSD (0x0000) xHCI root hub (0x0000), class 9/0, rev 2.00/1.00, addr 0
[     1.191965] uhub3: 12 ports with 12 removable, self powered
[     1.191965] ld0: GPT GUID: 43f2db65-b4ef-11f0-8d6a-6c1ff78d5bc9
[     1.191965] dk0 at ld0: "efiboot0", 262144 blocks at 64, type: msdos
[     1.191965] dk1 at ld0: "gptboot0", 233523776 blocks at 262208, type: ffs
[     1.191965] dk2 at ld0: "swap0", 16283648 blocks at 233785984, type: swap
[     1.191965] IPsec: Initialized Security Association Processing.
[     1.281964] ahcisata0 port 1: device present, speed: 6.0Gb/s
[     1.281964] ahcisata0 port 2: device present, speed: 6.0Gb/s
[     1.291964] ahcisata0 port 0: device present, speed: 6.0Gb/s
[     1.291964] ahcisata0 port 3: device present, speed: 6.0Gb/s
[     1.681964] uhidev0 at uhub3 port 2 configuration 1 interface 0
[     1.681964] uhidev0: SteelSeries (0x1038) SteelSeries Apex 150 Gaming Keyboard (0x1616), rev 1.10/0.19, addr 1, iclass 3/1
[     1.681964] ukbd0 at uhidev0
[     1.681964] wskbd0 at ukbd0: console keyboard, using wsdisplay0
[     1.681964] uhidev1 at uhub3 port 2 configuration 1 interface 1
[     1.681964] uhidev1: SteelSeries (0x1038) SteelSeries Apex 150 Gaming Keyboard (0x1616), rev 1.10/0.19, addr 1, iclass 3/0
[     1.681964] uhid0 at uhidev1: input=32, output=32, feature=0
[     1.681964] uhidev2 at uhub3 port 2 configuration 1 interface 2
[     1.681964] uhidev2: SteelSeries (0x1038) SteelSeries Apex 150 Gaming Keyboard (0x1616), rev 1.10/0.19, addr 1, iclass 3/0
[     1.681964] uhidev2: 2 report ids
[     1.681964] uhid1 at uhidev2 reportid 1: input=4, output=0, feature=0
[     1.681964] ukbd1 at uhidev2 reportid 2
[     1.681964] wskbd1 at ukbd1 mux 1
[     1.681964] wskbd1: connecting to wsdisplay0
[     2.881961] wd0 at atabus0 drive 0
[     2.881961] wd0: <ST8000VN004-3CP101>
[     2.881961] wd0: drive supports 16-sector PIO transfers, LBA48 addressing
[     2.881961] wd0: 7452 GB, 15504021 cyl, 16 head, 63 sec, 512 bytes/sect x 15628053168 sectors (4096 bytes/physsect; first aligned sector: 8)
[     2.901961] wd0: GPT GUID: 74a78e1c-c18e-4615-a20c-dfba5e66e198
[     2.901961] dk3 at wd0: "raid5@wd0", 15628050000 blocks at 2048, type: raidframe
[     2.901961] wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
[     2.901961] wd0(ahcisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)
[     2.901961] wd1 at atabus1 drive 0
[     2.901961] wd1: <ST8000VN004-3CP101>
[     2.901961] wd1: drive supports 16-sector PIO transfers, LBA48 addressing
[     2.901961] wd1: 7452 GB, 15504021 cyl, 16 head, 63 sec, 512 bytes/sect x 15628053168 sectors (4096 bytes/physsect; first aligned sector: 8)
[     2.921961] wd1: GPT GUID: fe57b8e6-1285-4f0f-bdb1-b488885fedfb
[     2.921961] dk4 at wd1: "raid5@wd1", 15628050000 blocks at 2048, type: raidframe
[     2.921961] wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
[     2.921961] wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)
[     2.921961] wd2 at atabus2 drive 0
[     2.931961] wd2: <ST8000VN004-3CP101>
[     2.931961] wd2: drive supports 16-sector PIO transfers, LBA48 addressing
[     2.931961] wd2: 7452 GB, 15504021 cyl, 16 head, 63 sec, 512 bytes/sect x 15628053168 sectors (4096 bytes/physsect; first aligned sector: 8)
[     2.951961] wd2: GPT GUID: 79b8cb2a-1073-442b-aa8c-d76c357a82d4
[     2.951961] dk5 at wd2: "raid5@wd2", 15628050000 blocks at 2048, type: raidframe
[     2.951961] wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
[     2.951961] wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)
[     2.951961] wd3 at atabus3 drive 0
[     2.951961] wd3: <ST8000VN004-3CP101>
[     2.951961] wd3: drive supports 16-sector PIO transfers, LBA48 addressing
[     2.951961] wd3: 7452 GB, 15504021 cyl, 16 head, 63 sec, 512 bytes/sect x 15628053168 sectors (4096 bytes/physsect; first aligned sector: 8)
[     3.261960] wd3: GPT GUID: 42b6de75-4e09-49c9-9ef3-9ba13a30c5cb
[     3.261960] dk6 at wd3: "raid5@wd3", 15628050000 blocks at 2048, type: raidframe
[     3.271960] wd3: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
[     3.271960] wd3(ahcisata0:3:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)
[     3.271960] swwdog0: software watchdog initialized
[     3.291960] WARNING: 2 errors while detecting hardware; check system log.
[     3.291960] boot device: ld0
[     3.291960] root on dk1 dumps on dk2
[     3.301960] root file system type: ffs
[     3.301960] kern.module.path=/stand/amd64/10.1/modules
[     3.361961] raid5: Component /dev/dk3 being configured at col: 0
[     3.361961]          Column: 0 Num Columns: 3
[     3.361961]          Version: 2 Serial Number: 133773311 Mod Counter: 516
[     3.361961]          Clean: Yes Status: 0
[     3.361961] raid5: Component /dev/dk4 being configured at col: 1
[     3.361961]          Column: 1 Num Columns: 3
[     3.361961]          Version: 2 Serial Number: 133773311 Mod Counter: 516
[     3.361961]          Clean: Yes Status: 0
[     3.361961] raid5: Component /dev/dk5 being configured at col: 2
[     3.361961]          Column: 2 Num Columns: 3
[     3.361961]          Version: 2 Serial Number: 133773311 Mod Counter: 516
[     3.361961]          Clean: Yes Status: 0
[     3.361961] raid5: RAID Level 5
[     3.361961] raid5: Components: /dev/dk3 /dev/dk4 /dev/dk5
[     3.361961] raid5: Total Sectors: 31256099840 (15261767 MB)
[     3.361961] raid5: GPT GUID: 8d5b1c29-5285-44af-90c9-6af131bca6c5
[     3.361961] dk7 at raid5: "island", 46884149692 blocks at 34, type: ffs
[     7.401952] aq0: link is UP: speed=1000
[    13.251941] wsdisplay0: screen 1 added (default, vt100 emulation)
[    13.251941] wsdisplay0: screen 2 added (default, vt100 emulation)
[    13.251941] wsdisplay0: screen 3 added (default, vt100 emulation)
[    13.251941] wsdisplay0: screen 4 added (default, vt100 emulation)

rvp

smurfd Dont have a /var/run/dmesg.log

Sorry, that should've been dmesg.boot--corrected.

smurfd [...] this is my last dmesg run

Don't see any HW issues there. Must be something RAID-related.
Can you:

Tell me why this discrepancy exists?

[     3.361961] raid5: RAID Level 5
[     3.361961] raid5: Components: /dev/dk3 /dev/dk4 /dev/dk5
[     3.361961] raid5: Total Sectors: 31256099840 (15261767 MB)
[     3.361961] raid5: GPT GUID: 8d5b1c29-5285-44af-90c9-6af131bca6c5
[     3.361961] dk7 at raid5: "island", 46884149692 blocks at 34, type: ffs

There's a 7 TB difference between the dk7 block (sector) size and the RAID size.

Try the dd test with different block sizes (say, 512 KB and 4 MB).
Try the orig. 1MB dd test again on the individual disks of the RAID (you'll have to redo the RAID).
Ask on the netbsd-users@ list (my knowledge of RAID is poor, sadly).

smurfd

thanks, well:
1) this must be user error in some way i think, because if i take the size of one of the disks * 3 i should be able to do this, but then i get this error...

$ doas newfs -O2 -b 65536 -s 22356g -F /dev/dk7
/dev/dk7: 22892544.0MB (46883930112 sectors) block size 65536, fragment size 8192
        using 6995 cylinder groups of 3272.75MB, 52364 blks, 103936 inodes.
wtfs: write error for sector 46883930111: Invalid argument

The error is missleading in this case atleast, since it has todo with the size of the wedge/partition not being of enough size. If i continue to decrease the size, then finally i find a size that works.
However i have yesterday noticed that the command i ran earlier:

$ doas newfs -O2 -s 21900g -b 65536 -F NAME=island

Did not create a filesystem on the lable 'island' instead created a image/file with the name: NAME=island of 21tb 😃
So redoing that with the above command instead, pointing out the /dev/dk7...
But i think i might have to force the size of this command:

$ doas gpt add -a 1024 -t ffs -l island raid5

2,3) this has crossed my mind, yes... was hoping to avoid having to redo the raid, but yeah will come to that i imagine... 🙂
4) sure, will ask there after retrying these points. will then also suggest adding a Big disk section to the man page for raidctl

Thanks @Leufroy & @rvp
Consider this closed, will update once the problem is resolved.

rvp

smurfd

$ doas newfs -O2 -s 21900g -b 65536 -F NAME=island

That command above isn't correct: you don't need to specify a size (newfs will get it from the kernel); and you shouldn't use -F (which is meant for disk image files--not block devices). Just do:

doas newfs -O2 NAME=island          # or, /dev/rdk7

BTW, there was a thread about write performance on HW RAID a while back. Try turning WAPBL on as recommended there.