Hello everyone, Discovering NetBSD 9.3, I encounter problems with virtualization. It's just extremely slow as if there wasn't any accelerator. I don't understand. Maybe someone could help me ? I would like to go further on NetBSD, I need to understand how to virtualize. This is what I've done in order to create a Windows 10 VM **1) Load NVMM module and give permission to my user "marcel"** `modload nvmm` `echo nvmm >> /etc/modules.conf` `chown marcel:wheel /dev/nvmm` - check nvmm `nvmmctl identify` ``` nvmm: Kernel API version 1 nvmm: State size 1008 nvmm: Max machines 128 nvmm: Max VCPUs per machine 256 nvmm: Max RAM per machine 128G nvmm: Arch Mach conf 0x0 nvmm: Arch VCPU conf 0x3 nvmm: Guest FPU states 0x3 ``` **2) Install Qemu, OVMF and vmware tools** `pkgin -y in qemu open-vm-tools OVMF` **3) Fix sound** `sysctl -w hw.audio0.blk_ms=100` **4) Create storage for the VM** `mkdir /home/marcel/VM` `qemu-img create -f qcow2 /home/marcel/VM/win10pron.qcow2 50G` `chown -R marcel:users /home/marcel/VM` **5) Create network for the VM** (bridged tap interface) `vi /etc/ifconfig.tap0` ``` create descr "NetBSD VM" up ! ifconfig bridge0 create ! ifconfig bridge0 descr "LAN VM bridge" up ! brconfig bridge0 add tap0 add ure0 ``` `chmod 660 /dev/tap0` `reboot` **7) Start the VM** ``` qemu-system-x86_64 \ -accel nvmm \ -m 4G \ -cpu max \ -smp cpus=2 \ -display sdl,gl=on \ -drive file="/home/marcel/Win10_22H2_French_x64.iso",media=cdrom \ -drive file=/home/marcel/VM/win10pron.qcow2 \ -drive file=/home/marcel/virtio-win-0.1.229.iso,media=cdrom \ -netdev tap,id=tap0,ifname=tap0,script=no \ -device virtio-net-pci,netdev=tap0 \ -usb \ -device usb-tablet \ -device usb-mouse,bus=usb-bus.0 \ -bios /usr/pkg/share/ovmf/OVMFX64.fd \ -boot d ``` And now, let's wait... wait... as if there wasn't any accelerator. nvmmctl list shows this during VM creation ``` Machine ID VCPUs RAM Owner PID Creation Time ---------- ----- ---- --------- ------------------------ 0 2 4.0G 873 Mon Aug 14 12:59:37 2023 ``` So I understand that NVMM is loaded and should be working... Do I need to configure something else ? Thank you very much

I just tried on a second server. Intel architecture this time. Still the same problem. VM is very very slow. If someone has an idea.. I take !

https://netbsd.org/docs/guide/en/chap-virt.html

@"pin"#p15317 Thank you Pin, it is already what I've used to create the VM...

I fear i won't a lot of help as i've no experience with virtualization on NetBSD but what strikes me as odd is `-cpu max`. I have no idea what that translates to. Maybe you are emulating a CPU newer than your host system and therefore have to do a lot of opcodes in software? I might very well be completely off here but i've never seen this option before and i'd probably rather use `-cpu host`.

Thank you Nettester. The "-cpu max" parameter allows all features of the processor to be utilized. I have a very fast CPU (i9). I don't understand why it is so slow. It took more than one hour to install windows.. and nearly 3 minutes to start the VM each time. I already tried "-cpu host". It didn't change anything.

The only thing which fasten a bit is when I increase -smp cpus.. I miss something.. it's not possible to be so slow.

I tried to change the disk emulation because windows is seeing the disk as a hdd. but it doesn't change anything. # Create Storage : `qemu-img -f create qcow2 /home/marcel/VM/win10pron.img 50G` `chown -R marcel:users /home/marcel/VM` # Creation of the VM ``` qemu-system-x86_64 \ -accel nvmm \ -m 4G \ -cpu max \ -smp cpus=2 \ -display sdl,gl=on \ -drive file="/home/marcel/Win10_22H2_French_x64.iso",media=cdrom \ -drive file=/home/marcel/VM/win10pron.img,if=none,id=drive0 \ -device nvme,drive=drive0,serial=serial0 \ -drive file=/home/marcel/virtio-win-0.1.229.iso,media=cdrom \ -netdev tap,id=tap0,ifname=tap0,script=no \ -device virtio-net-pci,netdev=tap0 \ -object rng-random,filename=/dev/urandom,id=viornd0 \ -audiodev oss,id=oss,out.dev=/dev/audio,in.dev=/dev/audio \ -device ac97,audiodev=oss \ -usb \ -device usb-tablet \ -device usb-mouse,bus=usb-bus.0 \ -bios /usr/pkg/share/ovmf/OVMFX64.fd \ -boot d ``` Still extremely slow

@"majekla"#p15324 Can you post the output of: `sudo cpuctl identify 0`

Off course. Here it is. ``` VIRTU# cpuctl identify 0 cpu0: highest basic info 00000020 cpu0: highest extended info 80000008 cpu0: "13th Gen Intel(R) Core(TM) i9-13900K" cpu0: Intel Pentium Pro, II or III (686-class), 2995.20 MHz cpu0: CPU base freq 3000000000 Hz cpu0: CPU max freq 5900000000 Hz cpu0: TSC freq CPUID 2995200000 Hz cpu0: family 0x6 model 0xb7 stepping 0x1 (id 0xb0671) cpu0: WARNING: Currently this info can't print correctly (level = 1, pgsize = 6) cpu0: error: unknown TLB level&type (1 & 5) cpu0: error: unknown TLB level&type (1 & 4) cpu0: error: unknown TLB level&type (1 & 4) cpu0: error: unknown TLB level&type (1 & 4) cpu0: WARNING: Currently this info can't print correctly (level = 2, pgsize = 7) cpu0: WARNING: Currently this info can't print correctly (level = 2, pgsize = 9) cpu0: features 0xbfebfbff cpu0: features 0xbfebfbff cpu0: features 0xbfebfbff cpu0: features1 0x7ffafbff cpu0: features1 0x7ffafbff cpu0: features1 0x7ffafbff cpu0: features1 0x7ffafbff cpu0: features2 0x2c100800 cpu0: features3 0x121 cpu0: features5 0x239c27eb cpu0: features5 0x239c27eb cpu0: features5 0x239c27eb cpu0: features6 0x98c027ac cpu0: features6 0x98c027ac cpu0: features7 0xfc1cc410 cpu0: features7 0xfc1cc410 cpu0: features7 0xfc1cc410 cpu0: xsave features 0x207 cpu0: xsave instructions 0xf cpu0: xsave area size: current 832, maximum 2696, xgetbv enabled cpu0: enabled xsave 0x7 cpu0: I-cache: 32KB 64B/line 8-way, D-cache: 48KB 64B/line 12-way cpu0: L2 cache: 2MB 64B/line 16-way cpu0: L3 cache: 36MB 64B/line 12-way cpu0: 64B prefetching cpu0: ITLB: 256 4KB entries 8-way, 32 4MB entries 8-way cpu0: L2 STLB: 1024 1GB entries 8-way cpu0: Initial APIC ID 0 cpu0: Cluster/Package ID 0 cpu0: Core ID 0 cpu0: SMT ID 0 cpu0: MONITOR/MWAIT extensions 0x3 cpu0: monitor-line size 64 cpu0: C1 substates 2 cpu0: C3 substates 2 cpu0: C5 substates 1 cpu0: C7 substates 1 cpu0: DSPM-eax 0xdfcff7 cpu0: DSPM-eax 0xdfcff7 cpu0: DSPM-eax 0xdfcff7 cpu0: DSPM-ecx 0x401 cpu0: SEF highest subleaf 00000002 cpu0: SEF-subleaf1-eax 0x400810 cpu0: Power Management features: 0x100 cpu0: Perfmon-eax 0x7300605 cpu0: Perfmon-eax 0x7300605 cpu0: Perfmon-edx 0x8603 cpu0: Hybrid: Core type 40, Native Model ID 0000001 cpu0: microcode version 0x113, platform ID 1 ```

The Windows VM displays that it is a 13900K, but no matter how many cpus I put, it's nearly always 100% use... Same for SSD use.

NetBSD Qemu + NVMM

majekla I tried to make haxm from pkgsrc but it seems that the port is broken now

~~Can you post your build error?~~

In file included from hax_entry_hax.c:33:
/usr/src/sys/sys/device.h:116:10: fatal error: prop/proplib.h: No such file or directory

majekla

So, from the beginning :
su -
cd /usr/pkgsrc/emulators/haxm
make

ERROR: This package has set PKG_FAIL_REASON:
ERROR: BSDSRCDIR/sys specifies non-existing directory /usr/src/sys
*** Error code 1

Stop.
make: stopped in /usr/pkgsrc/emulators/haxm

So I create the 2 folders missing just to see
mkdir -p /usr/src/sys

Then make again
make
It works for 20 minutes then tells me

/bin/sh:  Can't open /usr/scr/sys/conf/osrelease.sh
make: "/usr/share/mk/bsd.kmodule.mk" line 175: warning: "/bin/sh /usr/src/sys/conf/osrelease.sh -k" returned non-zero status
...
Stop.
make: stopped in /usr/pkgsrc/emulators/haxm/work/haxm-7.7.0/platforms/netbsd
*** Error code 1

Stop.
make[1]: stopped in /usr/pkgsrc/emulators/haxm
*** Error code 1

It's a NetBSD 9.3 complete install (pkgin + pkgsrc)

nettester

majekla It seems the package expects to find the NetBSD sources under /usr/src. Also the path in the first line of the second error contains a typo: /usr/scr/sys/conf/osrelease.sh should really be /usr/src/sys/conf/osrelease.sh (notice the mixed up characters in scr vs. src). If that typo isn't the result of manually transferring the error message this it probably needs to be fixed in the package itself but with that out of the way installing/extracting the NetBSD sources should let you progress further.

Jay

majekla
Just creating dir won't work you will need to fetch sources
#cd /usr #cvs checkout -r netbsd-9 -P src #cvs checkout -r netbsd-9 -P xsrc

majekla

nettester

oh sorry, I just made a mistake copying the informations of the error message (it's src actually.. not scr !), you're right.

Jay

Jay you might also need to setup cvs root

#export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot" #export CVS_RSH="ssh"

majekla

Jay

OK, thank you Jay.
Here is what I'm doing (it takes time), following your post and NetBSD doc :
export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
export CVS_RSH="ssh"
cd /usr
cvs checkout -r netbsd-9 -P src
cvs checkout -r netbsd-9 -P xsrc

I'll try to rebuild after.

Thank you

majekla

So !
From the beginning again (every command from fresh new complete install) :
su -
export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
export CVS_RSH="ssh"
cd /usr
cvs checkout -r netbsd-9 -P src
cvs checkout -r netbsd-9 -P xsrc

(asked during make : )
pkgin -y in cwrappers

cd /usr/pkgsrc/emulators/haxm
make
make install
make clean
make clean-depends

-Install Qemu and OVMF UEFI :
pkgin update && pkgin -y full-upgrade && pkgin -y in qemu OVMF

(I copied 2 ISOs of Windows and virtio-driver on the system before going on)

mkdir /home/marcel/VM
qemu-img create -f qcow2 /home/marcel/VM/win10pron.qcow2 50G

-Give access to my user marcel to the storage
chown -R marcel:users /home/marcel/VM

-Create the tap bridged interface :
echo "create" >> /etc/ifconfig.tap0 && echo 'descr "NetBSD VM" up' >> /etc/ifconfig.tap0 && echo "! ifconfig bridge0 create" >> /etc/ifconfig.tap0 && echo '! ifconfig bridge0 descr "LAN VM bridge" up' >> /etc/ifconfig.tap0 && echo "! brconfig bridge0 add tap0 add wm0" >> /etc/ifconfig.tap0

-Restart the network :
service network restart
-Give access to tap0 :
chmod 660 /dev/tap0

-Fix future sound problem in VM :
sysctl -w hw.audio0.blk_ms=100

-Create device node and load module
haxm-mknod && haxm-modload

-Start the VM

qemu-system-x86_64 \
-accel hax \
-m 4G \
-cpu max \
-smp cpus=2 \
-display sdl,gl=on \
-drive file="/home/marcel/Win10_22H2_French_x64.iso",media=cdrom \
-drive file=/home/marcel/VM/win10pron.qcow2 \
-drive file=/home/marcel/virtio-win-0.1.229.iso,media=cdrom \
-netdev tap,id=tap0,ifname=tap0,script=no \
-device virtio-net-pci,netdev=tap0 \
-object rng-random,filename=/dev/urandom,id=viornd0 \
-audiodev oss,id=oss,out.dev=/dev/audio,in.dev=/dev/audio \
-device ac97,audiodev=oss \
-usb \
-device usb-tablet \
-device usb-mouse,bus=usb-bus.0 \
-bios /usr/pkg/share/ovmf/OVMFX64.fd \
-boot d

And the error is ... :

HAX is working and emulator runs in fast virt mode.
NOTE: HAX is deprecated and will be removed in a future release.
      Use 'whpx' (on Windows) or 'hvf' (on macOS) instead.
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
[1]   Segmentation fault (core dumped) qemu-system-x86_64 -accel hax -m 4G -cpu max -smp cpus=2 -display sdl,gl=on -drive file="/home/marcel/Win10_22H2_French_x64.iso",media=cdrom -drive file=/home/marcel/VM/win10pron.qcow2 -driv...

Do you have an idea ?

majekla 😥

DISCONTINUATION OF PROJECT
This project will no longer be maintained by Intel.
Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.
Intel no longer accepts patches to this project.
If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.

from: https://github.com/intel/haxm

HAXM was created to bring Intel Virtualization Technology to Windows and macOS users. Today both Microsoft Hyper-V and macOS HVF have added support for Intel Virtual Machine Extensions. We have therefore decided to retire the HAXM project. HAXM v7.8.0 is our last release and we will not accept pull requests or respond to issues after this.

nettester

majekla Given it's status of being deprecated it's probably not really tested much (if at all). If you want to try making it work i'd start with trying to boot the most basic configuration humanly possible and hope it's possible to nail the crash to a specific subsystem. If that goes nowhere maybe downgrading qemu might have a little chance of improving the situation. For me trying this would depend on the effort needed to build older qemu releases though as it's pretty much a shot in the dark. Just some general thoughts. I don't really have any experience with this specific topic. So take what i say with a grain of salt. It's quite possible others can provide way better advise.

Edit: Looking over the issues posted on github there seems to be some information that could be useful (even if i didn't see anything that relates to your exact problem). Like https://github.com/intel/haxm/issues/170 (points to qemu 3.1.0nb5 or newer from pkgsrc being usable) https://github.com/intel/haxm/issues/304 (shows extremely simple command line that while failing to boot seems to work without crashing, also mentions unloading nvmm module before loading haxm) and so on.

majekla

It could be an interesting trip into history.. But honestly, I will not have that time.
I already spent 3 days on making a VM on NetBSD.. And the result is bad, unfortunately.
I will try with Xen next week and I will probably ask for help (I've already tried, and I don't succeed in booting on Xen). Right now I go back to bhyve for which I wrote a script manager.

Thank you very much for your help (to everyone).
I really appreciated !

majekla

I finally found the solution.. Problem was because of the -cpu option.
For NVMM to work, the option is -cpu qemu64 and not -cpu max or -cpu host
Now it works well !

« Previous Page