NetBSD Qemu + NVMM

majekla

Hello everyone,

Discovering NetBSD 9.3, I encounter problems with virtualization.
It's just extremely slow as if there wasn't any accelerator.
I don't understand.

Maybe someone could help me ?
I would like to go further on NetBSD, I need to understand how to virtualize.

This is what I've done in order to create a Windows 10 VM

1) Load NVMM module and give permission to my user "marcel"
modload nvmm
echo nvmm >> /etc/modules.conf
chown marcel:wheel /dev/nvmm

check nvmm
nvmmctl identify

nvmm: Kernel API version 1
nvmm: State size 1008
nvmm: Max machines 128
nvmm: Max VCPUs per machine 256
nvmm: Max RAM per machine 128G
nvmm: Arch Mach conf 0x0
nvmm: Arch VCPU conf 0x3<CPUID,TPR>
nvmm: Guest FPU states 0x3<x87,SSE>

2) Install Qemu, OVMF and vmware tools
pkgin -y in qemu open-vm-tools OVMF

3) Fix sound
sysctl -w hw.audio0.blk_ms=100

4) Create storage for the VM
mkdir /home/marcel/VM
qemu-img create -f qcow2 /home/marcel/VM/win10pron.qcow2 50G
chown -R marcel:users /home/marcel/VM

5) Create network for the VM (bridged tap interface)
vi /etc/ifconfig.tap0

create
descr "NetBSD VM" up
! ifconfig bridge0 create
! ifconfig bridge0 descr "LAN VM bridge" up
! brconfig bridge0 add tap0 add ure0

chmod 660 /dev/tap0

reboot

7) Start the VM

qemu-system-x86_64 \
-accel nvmm \
-m 4G \
-cpu max \
-smp cpus=2 \
-display sdl,gl=on \
-drive file="/home/marcel/Win10_22H2_French_x64.iso",media=cdrom \
-drive file=/home/marcel/VM/win10pron.qcow2 \
-drive file=/home/marcel/virtio-win-0.1.229.iso,media=cdrom \
-netdev tap,id=tap0,ifname=tap0,script=no \
-device virtio-net-pci,netdev=tap0 \
-usb \
-device usb-tablet \
-device usb-mouse,bus=usb-bus.0 \
-bios /usr/pkg/share/ovmf/OVMFX64.fd \
-boot d

And now, let's wait... wait... as if there wasn't any accelerator.

nvmmctl list shows this during VM creation

Machine ID VCPUs RAM  Owner PID Creation Time           
---------- ----- ---- --------- ------------------------
0          2     4.0G 873       Mon Aug 14 12:59:37 2023

So I understand that NVMM is loaded and should be working...

Do I need to configure something else ?

Thank you very much

majekla

I just tried on a second server. Intel architecture this time. Still the same problem. VM is very very slow.
If someone has an idea.. I take !

https://netbsd.org/docs/guide/en/chap-virt.html

majekla

Thank you Pin, it is already what I've used to create the VM...

nettester

I fear i won't a lot of help as i've no experience with virtualization on NetBSD but what strikes me as odd is -cpu max. I have no idea what that translates to. Maybe you are emulating a CPU newer than your host system and therefore have to do a lot of opcodes in software? I might very well be completely off here but i've never seen this option before and i'd probably rather use -cpu host.

majekla

Thank you Nettester.
The "-cpu max" parameter allows all features of the processor to be utilized.
I have a very fast CPU (i9).
I don't understand why it is so slow. It took more than one hour to install windows.. and nearly 3 minutes to start the VM each time.
I already tried "-cpu host". It didn't change anything.

majekla

The only thing which fasten a bit is when I increase -smp cpus..
I miss something.. it's not possible to be so slow.

majekla

I tried to change the disk emulation because windows is seeing the disk as a hdd. but it doesn't change anything.

Create Storage :

qemu-img -f create qcow2 /home/marcel/VM/win10pron.img 50G
chown -R marcel:users /home/marcel/VM

Creation of the VM

qemu-system-x86_64 \
-accel nvmm \
-m 4G \
-cpu max \
-smp cpus=2 \
-display sdl,gl=on \
-drive file="/home/marcel/Win10_22H2_French_x64.iso",media=cdrom \
-drive file=/home/marcel/VM/win10pron.img,if=none,id=drive0 \
-device nvme,drive=drive0,serial=serial0 \
-drive file=/home/marcel/virtio-win-0.1.229.iso,media=cdrom \
-netdev tap,id=tap0,ifname=tap0,script=no \
-device virtio-net-pci,netdev=tap0 \
-object rng-random,filename=/dev/urandom,id=viornd0 \
-audiodev oss,id=oss,out.dev=/dev/audio,in.dev=/dev/audio \
-device ac97,audiodev=oss \
-usb \
-device usb-tablet \
-device usb-mouse,bus=usb-bus.0 \
-bios /usr/pkg/share/ovmf/OVMFX64.fd \
-boot d

Still extremely slow

rvp

majekla Can you post the output of: sudo cpuctl identify 0

majekla

Off course.
Here it is.

VIRTU# cpuctl identify 0
cpu0: highest basic info 00000020
cpu0: highest extended info 80000008
cpu0: "13th Gen Intel(R) Core(TM) i9-13900K"
cpu0: Intel Pentium Pro, II or III (686-class), 2995.20 MHz
cpu0: CPU base freq 3000000000 Hz
cpu0: CPU max freq 5900000000 Hz
cpu0: TSC freq CPUID 2995200000 Hz
cpu0: family 0x6 model 0xb7 stepping 0x1 (id 0xb0671)
cpu0: WARNING: Currently this info can't print correctly (level = 1, pgsize = 6)
cpu0: error: unknown TLB level&type (1 & 5)
cpu0: error: unknown TLB level&type (1 & 4)
cpu0: error: unknown TLB level&type (1 & 4)
cpu0: error: unknown TLB level&type (1 & 4)
cpu0: WARNING: Currently this info can't print correctly (level = 2, pgsize = 7)
cpu0: WARNING: Currently this info can't print correctly (level = 2, pgsize = 9)
cpu0: features 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE>
cpu0: features 0xbfebfbff<MCA,CMOV,PAT,PSE36,CLFSH,DS,ACPI,MMX,FXSR,SSE,SSE2>
cpu0: features 0xbfebfbff<SS,HTT,TM,PBE>
cpu0: features1 0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MONITOR,DS-CPL,VMX,SMX,EST>
cpu0: features1 0x7ffafbff<TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE41,SSE42>
cpu0: features1 0x7ffafbff<X2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX>
cpu0: features1 0x7ffafbff<F16C,RDRAND>
cpu0: features2 0x2c100800<SYSCALL/SYSRET,XD,P1GB,RDTSCP,EM64T>
cpu0: features3 0x121<LAHF,LZCNT,PREFETCHW>
cpu0: features5 0x239c27eb<FSGSBASE,TSCADJUST,BMI1,AVX2,FDPEXONLY,SMEP,BMI2>
cpu0: features5 0x239c27eb<ERMS,INVPCID,FPUCSDS,RDSEED,ADX,SMAP,CLFLUSHOPT>
cpu0: features5 0x239c27eb<CLWB,PT,SHA>
cpu0: features6 0x98c027ac<UMIP,PKU,WAITPKG,CET_SS,GFNI,VAES,VPCLMULQDQ,TME_EN>
cpu0: features6 0x98c027ac<MAWAU=0x0,RDPID,KL,MOVDIRI,MOVDIR64B,PKS>
cpu0: features7 0xfc1cc410<FSREP_MOV,MD_CLEAR,SERIALIZE,HYBRID,PCONFIG>
cpu0: features7 0xfc1cc410<ARCH_LBR,CET_IBT,IBRS,STIBP,L1D_FLUSH,ARCH_CAP>
cpu0: features7 0xfc1cc410<CORE_CAP,SSBD>
cpu0: xsave features 0x207<x87,SSE,AVX,PKRU>
cpu0: xsave instructions 0xf<XSAVEOPT,XSAVEC,XGETBV,XSAVES>
cpu0: xsave area size: current 832, maximum 2696, xgetbv enabled
cpu0: enabled xsave 0x7<x87,SSE,AVX>
cpu0: I-cache: 32KB 64B/line 8-way, D-cache: 48KB 64B/line 12-way
cpu0: L2 cache: 2MB 64B/line 16-way
cpu0: L3 cache: 36MB 64B/line 12-way
cpu0: 64B prefetching
cpu0: ITLB: 256 4KB entries 8-way, 32 4MB entries 8-way
cpu0: L2 STLB: 1024 1GB entries 8-way
cpu0: Initial APIC ID 0
cpu0: Cluster/Package ID 0
cpu0: Core ID 0
cpu0: SMT ID 0
cpu0: MONITOR/MWAIT extensions 0x3<EMX,IBE>
cpu0: monitor-line size 64
cpu0: C1 substates 2
cpu0: C3 substates 2
cpu0: C5 substates 1
cpu0: C7 substates 1
cpu0: DSPM-eax 0xdfcff7<DTS,IDA,ARAT,PLN,ECMD,PTM,HWP,HWP_NOTIFY,HWP_ACTWIN>
cpu0: DSPM-eax 0xdfcff7<HWP_EPP,HWP_PLR,TBM3,HWP_CAP,HWP_PECI,HWP_FLEX>
cpu0: DSPM-eax 0xdfcff7<HWP_FAST,HW_FEEDBACK,HWP_IGNIDL,TD>
cpu0: DSPM-ecx 0x401<HWF,NTDC=0x4>
cpu0: SEF highest subleaf 00000002
cpu0: SEF-subleaf1-eax 0x400810<AVXVNNI,FSRSB,HRESET>
cpu0: Power Management features: 0x100<ITSC>
cpu0: Perfmon-eax 0x7300605<VERSION=0x5,GPCounter=0x6,GPBitwidth=0x30>
cpu0: Perfmon-eax 0x7300605<Vectorlen=0x7>
cpu0: Perfmon-edx 0x8603<FixedFunc=0x3,FFBitwidth=0x30,ANYTHREADDEPR>
cpu0: Hybrid: Core type 40, Native Model ID 0000001
cpu0: microcode version 0x113, platform ID 1

rvp

majekla Off course.
Here it is.

Is that from the actual HW (ie. not from inside the VM)?

majekla

The Windows VM displays that it is a 13900K, but no matter how many cpus I put, it's nearly always 100% use... Same for SSD use.

majekla

This the result of the Host (NetBSD 9.3).
The guest VM (windows 10) also displays the CPU model if I go to tasks manager or systel.

rvp

majekla This the result of the Host (NetBSD 9.3).

OK, that looks like a recent processor which supports VT-x. Is that enabled in the BIOS? On some BIOSes, you can disable virtualization-support.

majekla

Yes it is enabled. I often use this server for virtualization with Hyper-V, Bhyve, Virtualbox and Proxmox.
I tried 2 servers today. One with AMD (also often used for virtualization), this one with Intel, same result.

Tomorrow I'll try a third one, a bit older, just to check.
Something is missing in the procedure. Driver, accelerator default, I don't know, but it behaves as if there weren't any accelerator. Remind me of Qemu with TCG.

majekla

I tried the small java soft (nvmanager) from "glowiak" in order to understand what options he has chosen for VMs. Unfortunately, it's not faster than what I've done... and also, what he has done is not working with windows because we need OVMF firmware to boot.
Still no explanation.
Does NVMM really is an accelerator ??

I just tried on a third server (intel mini-pc)... Still so slow.

So.. I deduce that only Xen is exploitable on NetBSD ??

majekla So.. I deduce that only Xen is exploitable on NetBSD ??

Before nvmm, there was haxm, https://polprog.net/blog/netbsd-hax/

@majekla Here's an example, https://www.unitedbsd.com/d/68-streaming-netflix-on-netbsd

majekla

Thank you very much Pin.
I tried to make haxm from pkgsrc but it seems that the port is broken now.
I could not make it.

I'll go on with Xen... but it's not easy. Following the documentation leads to ... nothing.
I would need help from an experienced user to go on with Xen on NetBSD. I can't boot on Xen dom0, I don't know where is the mistake.

Thank you anyway.

majekla I tried to make haxm from pkgsrc but it seems that the port is broken now

~~Can you post your build error?~~

In file included from hax_entry_hax.c:33:
/usr/src/sys/sys/device.h:116:10: fatal error: prop/proplib.h: No such file or directory