Porting GPU drivers from Linux

nettester

OK, so this is probably a bit of a hypothetical question but seeing NetBSD has DRM/KMS made me curious how hard it would be to port an exiting KMS driver from Linux to NetBSD? Or rather lets say: If one would work around Nvidias GlibC dependency is there a theoretical chance to adjust the glue code to NetBSD and load their binary Linux blob? I am fully aware that 'theoretically possible' practically still means tons and tons of hours of crashes, hairloss and frustration. Like i said it's unlikely i would attempt (or have the needed free time) to do it but the question just keeps itching me and who knows maybe at some point i'll have the time and a sudden urge to replace my desktop machine. So does anyone maybe have some experience with that?

JuvenalUrbino

All BSDs port their DRM/KMS drivers from Linux. AMD/lntel drivers for Linux are contributed to the kernel by paid developers; only nouveau is reverse-engineered.
A BSD kernel can't load a linux kernel module compiled on and for Linux, even by relying on the compat layer; what you're looking for however exists and follows a somewhat similar concept: LinuxKPI

nettester

JuvenalUrbino True, linux modules obviously can't be loaded "as-is" and the "official" driver is mostly closed source. The interesting part here is the "mostly". I can't really speak for AMD as i have little experience with their drivers but i've done my fair share of wrestling with Nvidias offering.

As i understand it Nvidias (kernel) driver is structured like this: On one side there is the "real" driver code which is a closed source binary blob but then on the other side there is some glue code to avoid directly linking against Linux kernel API (probably to avoid triggering the GPL and also not have their blob tied to a specific kernel version). This glue code is open source (likely some weird licensing but oh well at least it's openly readable) and given that said glue code seems to be responsible for routing calls from the closed part to the specific kernel APIs i was wondering if it might be possible to replace/patch that layer so it'll use NetBSD's kernel APIs instead.

From my understanding Nvidias glue code is basically a very specific "emulation" layer. In it's official form it implements Nvidias API on top of the Linux kernel API and given that the relevant functionality provided by NetBSDs kernel would probably be pretty similar it doesn't seem "that" far fetched to adjust it. I wouldn't be very surprised if Nvidias FreeBSD offering (i am not sure if it's still around but for some time an official FreeBSD Nvidia driver was a thing) was almost identical to the Linux version internally with the major changes being in the glue layer. It's kinda telling that Nvidia's Linux driver has some functionality that seems to be originally designed for mapping onto the Windows registry.

Of course there is also the problem of Nvidias Xorg libraries being built against GlibC but that shouldn't be to bad. All it needs is something like the gcompat hack someone did for musl.

Going the whole way and emulating all of Linux's kernel module concepts/API would obviously also move in the same direction but i guess it would be even harder to get to something stable this way as the scope of it is a lot wider. Besides it does not feel very clean to run drivers from a different OS.

What throws me off is mostly that if it where "so" easy (patching glue code) why didn't anyone think of it beforehand? I am very likely not the first person to have read the source of the open part for Nvidias driver. I am almost certain that i have to be missing some important points here. The open part of Nvidias driver is actually quite huge and i didn't study it in detail.

bsduck

nettester i am not sure if it's still around but for some time an official FreeBSD Nvidia driver was a thing

Yes, nvidia still provides official FreeBSD drivers.

JuvenalUrbino

nettester I can't really speak for AMD as i have little experience with their drivers

Here's the source code for amdgpu in the Linux kernel. You can browse the rest of the drm/kms stack by going backward in the directory path.
Intel and AMD are among the top 5 single corporate entities to contribute most to the Linux kernel yearly, and have employees specifically set to develop and maintain the Linux drivers.
These developers actively cooperate with freedesktop.org (home of Mesa3D, libdrm, libGL, X11, nouveau) through their mailing list.
Unsurprisingly Intel and AMD are, respectively, platinum and gold members of the Linux Foundation.

Nouveau is a reverse-engineered driver whose reason of being lies in the fact that for a long time Nvidia not only kept their drivers proprietary, but also refused (with a growing number of exceptions over the years) to release any documentation around the chips they manufactured. The kind of performance you can get with nouveau is poor if compared against that of the proprietary nvidia driver or that of amdgpu on comparably powerful graphics. This is due to the reverse-engineered nature of the driver.

All BSDs have their drm/kms stack entirely ported from Linux. Porting is orders of magnitude more time-effective than re-writing drivers of such complexity from scratch and ensures that still BSDs have good-quality drivers, from a performance and security standpoint (in spite of the considerably smaller amount of contributors, and their mostly non-paid volunteering activity).

Linux modules obviously can't be loaded "as-is" and the "official" driver is mostly closed source

Since mid-2022, Nvidia GPU Kernel Modules are mostly opensource and MIT/GPL double-licensed, with the exception of user-stack components relative to Vulkan and CUDA support.

As i understand it Nvidias (kernel) driver is structured like this: On one side there is the "real" driver code which is a closed source binary blob but then on the other side there is some glue code to avoid directly linking against Linux kernel API (probably to avoid triggering the GPL and also not have their blob tied to a specific kernel version).

You're probably referring to the kernel interface layer. For the FreeBSD counterpart (see below), untar the latest FreeBSD driver version and have a look into src/nvidia

i am not sure if it's still around but for some time an official FreeBSD Nvidia driver was a thing

Nvidia still officially releases drivers for FreeBSD (alongside Windows, Linux and Solaris). Vulkan is supported on FreeBSD since 2021.

From my experience (which spans across several years of parallel use of Linux and FreeBSD with Nvidia), nvidia proprietary drivers are much easier to use and deal with on FreeBSD than they are on Linux. No separate non-free repositories to manage, no troublesome DKMS, no broken drivers at each minor kernel update (unlike Linux, BSDs keep a stable kernel ABI within each major release), no having to mess with blacklisting nouveau , no trouble setting up a second monitors....it just works.

From my understanding Nvidias glue code is basically a very specific "emulation" layer. In it's official form it implements Nvidias API on top of the Linux kernel API and given that the relevant functionality provided by NetBSDs kernel would probably be pretty similar it doesn't seem "that" far fetched to adjust it.

What throws me off is mostly that if it where "so" easy (patching glue code) why didn't anyone think of it beforehand? I am very likely not the first person to have read the source of the open part for Nvidias driver. I am almost certain that i have to be missing some important points here. The open part of Nvidias driver is actually quite huge and i didn't study it in detail.

Many installed components, among which required shared objects (basically anything installed under $PREFIX/lib, the nvidia_drv.so X display driver and its extensions), are shipped as binary blobs, dynamically linked for the target platform (e.g. FreeBSD, or Linux). I'm unsure whether the NetBSD's compat_linux would have been capable of handling this. Besides that, the large OS-agnostic components of the nvidia and nvidia-modeset kernel modules also come pre-compiled as object files for the target platform they're meant to be linked onto. These would have been unusable on NetBSD. The are probably also LICENSE/TOS concerns to take into account...

Such issues could be considered sorted out now with the opensource nvidia driver version.
However, try browse the Nvidia GPU drivers code and perform a recursive case-insensitive grep for linux. You'll find out that OS-specific support is scattered in the code in a more pervasive way than if simply confined to a separate compat layer bundled with the driver.
Porting something as big as a drm driver to a news OS takes a lot of time, skill and effort. After which you'll be presented with a significant number of bugs and edge-cases to deal with before being able to deliver if to the end-user (see the number of PRs in the GNATS database concerning the drm/kms update in NetBSD-10 and delaying its release).

FreeBSD uses the linux(4) compat layer, optionally in conjunction with x11/linux-nvidia-libs to support the nvidia driver.

So why haven't the BSDs ported the Open GPU Kernel Modules yet?

FreeBSD have a fully functional implementation of what is now largely and opensource driver. So they're probably not interested in maintaining a downstream fork.
OpenBSD would most likely reject the idea of a closed-source third party software being added to the memory image of the running kernel. Now that drivers are opensource, they might change their mind.
I think NetBSD devs wouldn't have liked the idea of binary blobs either. Moreover, NetBSD is the only BSD to have ported nouveau and to provide in-kernel support for nvidia chips as of today. I think that the project and the community are fine with nouveau at least until (and if ever) the official nvidia opensource drivers don't become a de facto standard on Linux. In perspective, nouveau development may also benefit from having nvidia code at hand.

nettester

JuvenalUrbino

JuvenalUrbino The kind of performance you can get with nouveau is poor if compared against that of the proprietary nvidia driver or that of amdgpu on comparably powerful graphics. This is due to the reverse-engineered nature of the driver.

Yes, sadly nouveau is not of much help outside of plain and simple desktop usage. I think it's also missing any (most?) optimus/prime functionality, which is the key to running actual graphics on inexpensive old compute cards.

JuvenalUrbino All BSDs have their drm/kms stack entirely ported from Linux. Porting is orders of magnitude more time-effective than re-writing drivers of such complexity from scratch and ensures that still BSDs have good-quality drivers, from a performance and security standpoint (in spite of the considerably smaller amount of contributors, and their mostly non-paid volunteering activity).

Obviously. Actually making DRM/KMS into a de facto standard seems like a pretty smart move.

JuvenalUrbino Since mid-2022, Nvidia GPU Kernel Modules are mostly opensource and MIT/GPL double-licensed, with the exception of user-stack components relative to Vulkan and CUDA support.

I know but the sad thing is that hardware support of the released code doesn't go to far back. The cutoff is at maxwell or even pascal if i remember correctly. Admittedly a couple years down the road that won't make much of a practical difference anymore as most people will either have something recent enough or be well of using nouveau but right now (at least given the outrageous GPU prices) it's a bit of a bummer. I am not giving up my Tesla K20x just yet (well at least not for any other Nvidia card).

JuvenalUrbino You're probably referring to the kernel interface layer. For the FreeBSD counterpart (see below), untar the latest FreeBSD driver version and have a look into src/nvidia

Kind of probably. I am not entirely sure how similar Nvidia's open source release is to the old proprietary drivers though as like mentioned above there seems to be quite a bit functionality missing and having a GPL compatible license removes a couple legal hoops Nvidia had to jump through before when dealing with the kernel API.

JuvenalUrbino From my experience (which spans across several years of parallel use of Linux and FreeBSD with Nvidia), nvidia proprietary drivers are much easier to use and deal with on FreeBSD than they are on Linux. No separate non-free repositories to manage, no troublesome DKMS, no broken drivers at each minor kernel update (unlike Linux, BSDs keep a stable kernel ABI within each major release), no having to mess with blacklisting nouveau , no trouble setting up a second monitors....it just works.

Interesting. I can't really say that i ran into a lot of trouble with Nvidias driver over the years on Linux but then again stuff like blacklisting nouveau has long become a second nature to me. The only really annoying thing was when i wanted to use an old Nvidia card as render output for my Tesla. Now that's messy... You need to enable nouveau (card is to old to be supported by the driver supporting the Tesla - not sure if Nvidia's driver would support 2 cards at once anyways) or you'll have no output sink but enabling it means it will also grab the Tesla... Had to patch nouveau to ignore the relevant PCI id just to find out sinking output didn't work anyways. Now that was frustrating but oh well, old AMD cards do a pretty fine job as output sinks and at like 5€ used even messing with nouveau probably did not account to "time well spent" anyways.

JuvenalUrbino Many installed components, among which required shared objects (basically anything installed under $PREFIX/lib, the nvidia_drv.so X display driver and its extensions), are shipped as binary blobs, dynamically linked for the target platform (e.g. FreeBSD, or Linux). I'm unsure whether the NetBSD's compat_linux would have been capable of handling this. Besides that, the large OS-agnostic components of the nvidia and nvidia-modeset kernel modules also come pre-compiled as object files for the target platform they're meant to be linked onto. These would have been unusable on NetBSD. The are probably also LICENSE/TOS concerns to take into account...

I don't really think the shared libraries would be that much of a problem. In my opinion (which might obviously be completely wrong) the biggest hurdle would be the GlibC dependency. Beyond that those should really only depend on Xorg and similar which are probably mostly identical API-wise on either Linux or *BSD. Of course there could also be some nasty code hiding in there but at least in theory all actual system calls would be up to the underlying C library.

Concerning object files: I think you have a point there. I vaguely remember seeing precompiled object files while browsing through the driver code and those very much could be linked to some kernel part during the DKMS build. It would be quite interesting what they are actually linked against though as i was under the impression Nvidia couldn't link directly to the kernel for GPL reasons. Those objects also seem to work with a very huge amount of different kernel versions and from what i've seen adjusting Nvidias driver to newer kernel versions always seemed to involve patches to the open parts of the drivers (obviously - otherwise random people wouldn't have been able to do so, at least easily).

License/EULA/TOS and so on might prove complicated also but i tend to view things from a technical standpoint. If something is technically possible investing time into dealing with legalities might be justified but as long as that is unclear i rather avoid paying to much mind. In this particular case it probably wouldn't be to hard to keep oneself outside of hot water anyways as most further reaching laws seem to be centered around disallowing circumvention of copy protections, so simply moving from actively violating a (theoretically prohibitive) agreement to giving the user means to do so might already be enough to dodge angry lawyers.

JuvenalUrbino Such issues could be considered sorted out now with the opensource nvidia driver version.
However, try browse the Nvidia GPU drivers code and perform a recursive case-insensitive grep for linux. You'll find out that OS-specific support is scattered in the code in a more pervasive way than if simply confined to a separate compat layer bundled with the driver.

That's actually quite interesting. I obviously can't really comment without having studied the code but it's a good pointer and i'll probably do so. As i said above i am not sure how similar to open source release is to the closed version but even if it's rather different it should probably give one a good idea of what kind of surprises are to be expected.

JuvenalUrbino Porting something as big as a drm driver to a news OS takes a lot of time, skill and effort. After which you'll be presented with a significant number of bugs and edge-cases to deal with before being able to deliver if to the end-user (see the number of PRs in the GNATS database concerning the drm/kms update in NetBSD-10 and delaying its release).

Of course i can do nothing but agree here. Even if OS specific functionality really was concentrated inside a single (open) part just getting something running would be a small miracle. Going from there to something that could even vaguely be called production ready... a huge, huge monstrosity of a miracle. Supporting said project... No, let's better just not think about it. It's getting really, really depressing by now 😉

JuvenalUrbino I think NetBSD devs wouldn't have liked the idea of binary blobs either.

Noone likes binary blobs but right now they are (at least for Nvidia cards of certain generations) still a sad reality. To be honest, i am not putting that much hope into nouveau but who knows? The blobs are on their way out one way or another anyways. It'll just take a couple more years. That's something i am quite certain about.

JuvenalUrbino

nettester I think it's also missing any (most?) optimus/prime functionality

Nouveau supports PRIME Render Offload on its own, but since reclocking isn't possible on anything above Maxwell due to Nvidia requiring signed firmware on newer GPUs, the ideal performance gain of using nouveau's PRIME Offload on recent cards is poor at best.
Alternative solutions include:

prime-select/optimus-manager achieves maximum performance out of NVIDIA GPU and switches it off if not in use
acpi_call/turn_off_gpu.sh this works on freebsd too
bumblebee

which is the key to running actual graphics on inexpensive old compute card

Optimus is limited to a number of models. On top of that, nvidia drivers below version 435 have no support for PRIME Offload. This really breaks Optimus configurations on older driver versions, making nouveau's PRIME Offload a best candidate on this kind of hardware.

I've run NetBSD with nouveau on more than one laptop (a 2008 Samsung equipped with a low-end GeForce 315M card and a newer HP with a GeForce GTX 760) with an overall satisfying experience. I did a lot of opensource gaming on those laptops, including some relatively resource-intensive engines. I used to run OpenIndiana (illumos) on that same Samsung laptop, using the Nvidia propriety driver, and I remember GPU performance to be somewhat comparable (no benchmarks to share).
Overall, and with variable quirks on Optimus models, I think nouveau is a fine solution on older Nvidia cards, at least up to Kepler (and to a certain extent Maxwell). While performance and power consumption might not be great, and suspending might become a challenge, Pascal is at the very least supported on NetBSD 10.0_BETA.
I'm currently using Nvidia with nouveau on my workstation equipped with a GTX 1060; though for desktop usage, I mainly boot Slackware on that machine, relying on nouveau as well. Enough for my needs (not doing anything remotely GPU-hungry these days).

I can't really say that i ran into a lot of trouble with Nvidias driver over the years on Linux

Which distro are we talking about? Something that packages and promptly rebuilds nvidia-dkms? What about 'minor' distros shipping with less standardized defaults? In this kind of scenario (Slackware, Void Linux, Gentoo) the DKMS modules may typically break. Maintaining a system with nvidia drivers installed may turn out a bit tricky and time-consuming in the long run.

I am not entirely sure how similar Nvidia's open source release is to the old proprietary drivers though as like mentioned above there seems to be quite a bit functionality missing

Yesterday out of curiosity I fetched, extracted and inspected the contents of pre-2022 nvidia drivers for FreeBSD, Linux and Solaris. The kernel interface layer (the opensource 'glue code') is comparable to that found on current drivers, and in sync with that uploaded on github by Nvidia (linked above), which corresponds to the Linux version. If you extract the driver's tarball for FreeBSD and Solaris and cd into it, the OS-specific bits of the interface layer can be found at src/nvidia and NVDAgraphicsr/reloc/kernel/src respectively.

I think you have a point there. I vaguely remember seeing precompiled object files while browsing through the driver code and those very much could be linked to some kernel part during the DKMS build. It would be quite interesting what they are actually linked against though as i was under the impression Nvidia couldn't link directly to the kernel for GPL reasons.

You can't redistribute the resulting binary. This is another reason why I prefer nvidia on FreeBSD. No license incompatibility and stable ABI, means easy to deploy nvidia-driver packages and no need for DKMS. Same applies to ZFS and VirtualBox.

The blobs are on their way out one way or another anyways. It'll just take a couple more years. That's something i am quite certain about.

I agree, and think this reflects the transition of opensource from mainly an academic R&D model fostered by the advent of Internet + philosophy and social movement (FOSS) , to an efficient and cost-effective business model in which almost all the the Big Tech giants see large margins of profit and already invested a lot into.

nettester

JuvenalUrbino

JuvenalUrbino Nouveau supports PRIME Render Offload on its own, but since reclocking isn't possible on anything above Maxwell due to Nvidia requiring signed firmware on newer GPUs, the ideal performance gain of using nouveau's PRIME Offload on recent cards is poor at best.

Interesting. So i was probably simply using an outdated version but to be honest i can't really remember how i concluded it "didn't work" back then anyways. It might very well also have been due to insufficient performance.

JuvenalUrbino prime-select/optimus-manager achieves maximum performance out of NVIDIA GPU and switches it off if not in use
acpi_call/turn_off_gpu.sh this works on freebsd too
bumblebee

I am not exactly familiar with the first two solutions but bumblebee seems to be kind of a (somewhat problematic) legacy approach from before Prime Offloading was available and turning off specific GPUs is likely only helpful as long as the remaining card is actually the one driving the output. Beyond that scripts/tools in this context usually seem to be user friendly frontends for either xrandr or Nvidia's proprietary environment variables (as documented in the driver manual). Personally i prefer to adjust the relevant settings manually but the scripts in question are likely still quite helpful to a lot of people.

JuvenalUrbino Optimus is limited to a number of models. On top of that, nvidia drivers below version 435 have no support for PRIME Offload. This really breaks Optimus configurations on older driver versions, making nouveau's PRIME Offload a best candidate on this kind of hardware.

It's probably more a list of whats officially supported. Practically there doesn't seem to be any relation to mobile cards. I am pretty sure that the Nvidia driver supporting the card being recent enough is the only real requirement.

Like i said the desktop i am sitting at is running a Tesla K20X (aka poor mans GTX Titan). If there was an actual limitation to mobile hardware i would have had a pretty bad time connecting a monitor as the card does not have a single output port and if it was just sitting idle the performance difference would be quite noticeable given the output port is provided by a not exactly overly powerful Radeon HD2400 Pro.

JuvenalUrbino I've run NetBSD with nouveau on more than one laptop (a 2008 Samsung equipped with a low-end GeForce 315M card and a newer HP with a GeForce GTX 760) with an overall satisfying experience. I did a lot of opensource gaming on those laptops, including some relatively resource-intensive engines. I used to run OpenIndiana (illumos) on that same Samsung laptop, using the Nvidia propriety driver, and I remember GPU performance to be somewhat comparable (no benchmarks to share).
Overall, and with variable quirks on Optimus models, I think nouveau is a fine solution on older Nvidia cards, at least up to Kepler (and to a certain extent Maxwell). While performance and power consumption might not be great, and suspending might become a challenge, Pascal is at the very least supported on NetBSD 10.0_BETA.

To be honest as far as laptops are concerned i (ironically - offloading should really be a mobile technology after all) wouldn't really be bothered either. I simply don't expect a ton of graphical performance from a laptop. As long as it can play a video in lets say 480-720p it's fine for me. If i want more i'll just sit down at a "real" PC. Battery lifetime is obviously relevant but i am quite positive for nouveau to support some conservative enough configuration to at least not make much of a difference.
Given how capable integrated chipsets are these days i could even imagine myself simply turning off the dedicated GPU.

JuvenalUrbino I'm currently using Nvidia with nouveau on my workstation equipped with a GTX 1060; though for desktop usage, I mainly boot Slackware on that machine, relying on nouveau as well. Enough for my needs (not doing anything remotely GPU-hungry these days).

Perfectly understandable. I am not sure but maybe my opinion on nouveau came across a bit to harsh. For general usage its performance is probably perfectly reasonable. If i wouldn't be doing some (modest) game development now and then i would probably care way less as long as it manages to support my (admittedly exotic) configuration.

JuvenalUrbino Which distro are we talking about? Something that packages and promptly rebuilds nvidia-dkms? What about 'minor' distros shipping with less standardized defaults? In this kind of scenario (Slackware, Void Linux, Gentoo) the DKMS modules may typically break. Maintaining a system with nvidia drivers installed may turn out a bit tricky and time-consuming in the long run.

Well, i am a boring and lazy person. If that wasn't enough of a hint: Devuan (or rather Debian before the systemd debacle) even if i get a feeling that my days Devuan might be numbered also (more and more stupid forced dependencies on stupid things...).

I have to say that repeatedly trying out kernel patches gets annoying pretty quickly though. Each kernel install triggers a DKMS rebuild, which in turn triggers rebuilding initramfs... yawn... On the other hand i also remember when installing Nvidia's proprietary driver still meant manually getting the .run file and even that didn't result in any major problem i would actually remember these days but, well, it's been a couple years since then so my memory might just be clouded or i was simply lucky back then, who knows?

One shouldn't forget that "problem" is a very subjective thing anyways either. What to you or me might seem like a minor inconvenience could very well make an enduser throw their PC out of the window. Even if that is probably more relevant to Linux than it is to *BSD.

JuvenalUrbino This is another reason why I prefer nvidia on FreeBSD. No license incompatibility and stable ABI, means easy to deploy nvidia-driver packages and no need for DKMS.

Agreed. I also think that "needlesly" cluttering a system with toolchains and build tools is undesirable, while automated builds in general are (unless it's done because of otherwise impossible customization) are nothing but another source of possible errors.

JuvenalUrbino I agree, and think this reflects the transition of opensource from mainly an academic R&D model fostered by the advent of Internet + philosophy and social movement (FOSS) , to an efficient and cost-effective business model in which almost all the the Big Tech giants see large margins of profit and already invested a lot into.

Exactly. With all the good and bad that comes with it.