I’m looking for help: Are there any guides that cover enabling IOMMU passthrough?
Goal: enable iGPU passthrough using proxmox or ESXi 7 so a VM can use Intel QuickSync
More Info:
I have several different Supermicro motherboards that have chipsets that support IOMMU to enable passthrough: X10SAT and X10SLH-F --both with the C226 Express chipset that, in theory, can support IOMMU and Intel GPUs.
For CPUs with integrated GPUs, I have several, including E3-1265L V4, E3-1285L v4, E3-1246 V3
Another newer option is the X12SCZ-TLN4F with an Intel W480E chipset (using a 10th Gen Intel CPU).
If there is a standard set of BIOS options that I could enable to activate IOMMU support that are documented somewhere, would someone please point me in the right direction.
Also, if there is some way of discovering how to enable these options using a BIOD dump, I’d greatly appreciate a pointer in the right direction.
Usualy this BIOS option called "VTd" on Intel chipsets
I’ve checked that VTd is enabled and I’ve gone through several good write-ups on virtualization and passthrough (e.g. https://heiko-sieger.info/iommu-groups-w…my_IOMMU_group and https://heiko-sieger.info/running-window…rough/#Part_12–_Troubleshooting ) and I’ve verified that the CPU supports it, so these guides point to issues with the mother board BIOS implementation, e.g. "Check again to make sure your CPU supports IOMMU. If yes, the cause may be a faulty motherboard BIOS."
Does anyone have experience hacking/“fixing” a motherboard BIOS to get IOMMU working if it is not working correctly by default? Is this even possible?
Any input would be welcome!
I’m an owner of a Supermicro X10SAT and I have used it for PCI Passthrough on both Xen and VFIO. To enable the IOMMU… Check the Motherboard Manual, seriously. In Intel platforms, it is only VT-d, which the X10SAT has easily available. For the Software side, go to any of the 1894589124890142 QEMU-KVM-VFIO guides that are floating around on the Internet. You don’t need anything else.
Do note that for QuickSync you may be interesed in using Intel GVT-g GPU Virtualization, since it is easier than passing though the IGP itself via Intel GVT-d. You must use one of the Broadwell v4 Xeons, cause Haswell IGPs aren’t supported for GVT-g. And also remember to set the GPU Aperture Size to 1 GiB (Max supported), cause GVT-g relies on it for maximum possible vGPUs.
Also, on the X10SAT there is a BIOS bug since BIOS 3.0 where you can’t enable the IGP simultaneously with a discrete GPU. I have a fixed BIOS, which I also have uploaded.
T201903151640 - 3.2 with Graphics Settings submenu fixed - http://s000.tinyupload.com/?file_id=06745098729364751320
@zir_blazer
I’ve installed your 3.2 BIOS with the BIOS bug fix. I’m using the E3-1285LV4 (Broadwell) CPU that has an iGPU, and I have 32GB of ECC RAM: https://ark.intel.com/content/www/us/en/…e-3-40-ghz.html
Re: GPU Aperture size–I can select larger sizes in the BIOS selector, but the current value will not go above 256MB–Are you able to actually get it to set to a larger value than 256MB?
Re: RTFM, I actually did read the manual prior to posting here originally, and there is no mention of IOMMU, and the only mention of virtualization references VT-d. As I had mentioned, I had enabled VT-d in the BIOS settings. Are there some other terms I should have been searching for?
I recall having set it to 512 MiB at some point. This is from my notes:
Make sure to check with Linux if it works or not, cause I don’t know how to do it in Windows. If the option resets back to 256 MiB when you configure it, then there is a problem and may want to contact Supermicro support to see if they are willing to provide a fix, or ask for a BIOS mod here. Do note that Haswell and Broadwell have different behavior with this option and I have a Haswell, so if there is a BIOS bug, maybe it affects your Broadwell but not my Haswell (I don’t recall if I tested 512 MiB after 3.x BIOS versions, it could be present in my current BIOS version too).
I think I recall that using big Aperture Sizes also had a prerequisite to have Above 4G decoding enabled or something like that.
In Intel parlance, VT-d is the IOMMU. It may not be 100% clear cause I recall other people asking how to enable the IOMMU on Intel platforms even after finding the VT-d toggle, but is rare you didn’t noticed since it is mentioned on almost every guide.
If you have VT-d enabled, just go to whatever guide you’re following next step of using intel_iommu=on as Linux Kernel parameter then check for IOMMU Groups. If they’re there, it is working already.
Adding this, which is from a rancom Coffee Lake Motherboard Manual to show the relationship:
CMS should be a typo from CSM (Compatibility Support Module). Now, the X10SAT didn’t had any explicit option to explicitly disable the CSM, but you could choose everything "UEFI Only" for loading Option ROMs which is the closest you will be. I don’t know if selecting everything UEFI disables the CSM or if it is always enabled.
X10SAT has this:
Advanced / PCIe/PCI/PnP Configuration / Above 4G Decoding
Things you want to have in UEFI mode only:
Yeah, strange. The max I can set is 256M (and that is with above 4GB enabled).
I used Xen from 2013 to 2015, then QEMU from 2015 on. Main usage is PCI Passthrough for a Windows gaming VM.
Never touched ESXi. Proxmox is QEMU based with custom config files.
OK, I’ve been in communication with folks at Supermicro, and for their boards VT-d does not equal IOMMU. In fact they told me that IOMMU is disabled in the BIOS of the 3 boards I asked about ( X10SAT, X10SLH-F, and X12SCZ-TLN4F), so I’m back to my original question:
Does anyone know what changes need to be made to a BIOS to enable IOMMU?
Then the guy that replied to your mail has no clue about what he is talking about. Seriously. VT-d = IOMMU. Is been that way from like 10 years ago. I’m disappointed if Supermicro support told you that.
This Thread is me back in 2013, where I ended up handpicking the Supermicro X10SAT: https://forums.anandtech.com/threads/lga…upport.2326402/
This Blog post which is fresh from this year: https://blog.3mdeb.com/2021/2021-01-13-iommu/
KVM old documentation: https://www.linux-kvm.org/page/How_to_as…ith_VT-d_in_KVM
Blog from Intel: https://software.intel.com/content/www/u…mmu-part-1.html
Which links to VT-d specification.
So your original question is already answered. And I already told you that I have been using the X10SAT in both Xen and QEMU-KVM-VFIO for PCI Passthrough with a Radeon 5770 (The old one, from 10 years ago). I don’t know what more should I say to convince you that the thing works properly.
GVT-g is a different matter, and actually, it even works WITHOUT IOMMU (At least it should have been able to do so on the early XENGT distributions, maybe it changed, but recall asking that in Intel GVT-g Mailing List).
My understanding, from the folks at SuperMicro, is that the IOMMU capabilities needed to access the integrated Intel GPU (to enable passthrough of the iGPU) are disabled in their BIOS. It wasn’t just one tech at SuperMicro that said this, several confirmed this. Is it possible that IOMMU for the integrated iGPU is handled differently than PCI passthrough for a discrete GPU? Are you able to passthrough and use the integrated Intel iGPU? That is not working for me.
What Hypervisor are you trying to use? Configuration instructions are entirely different for every one of them, and I only know about QEMU and Xen (Old versions, not anything from 2015 onwards. Last Xen I used was 4.5).
Passthroughing the IGP (Which in Intel parlance is called GVT-d) is possible in QEMU-KVM-VFIO but a bit painful, which is why GVT-g for GPU Virtualization is almost always recommended: http://vfio.blogspot.com/2016/07/intel-g…assignment.html
I never tried IGP passthrough cause I always intended it to remain for host use. Besides, Haswell only supports Legacy mode, not UPT, plus it has some Firmware limitations (Using SeaBIOS instead of OVMF/UEFI). GVT-g isn’t supported on Haswell because Intel decided to drop support for it before it got mainlined into the Linux Kernel (Some standalone Ubuntu based Intel XenGT distributions supported it), so there is little I can test about it myself cause I have no access to Broadwell.
The Aperture Size as I stated used to work up to 512 MiB on the X10SAT, which I did tested on previous BIOSes. 512 MiB was a Haswell limitation, Broadwell extended it to 1 GiB. EVEN if you’re bound to 256 MiB due to some Firmware bug, it is still enough to use GVT-g, as you only need 128 MiB per vGPU instance plus 128 MiB for the host. I’m not aware than there is any other Firmware-limiting option related to GVT-g.
May want to read this:
https://github.com/intel/gvt-linux/issues/107
https://wiki.archlinux.org/index.php/Intel_GVT-g
Note that I recall having asked to Intel developers a few years ago about GVT-g working in Processors models without VT-d support and the answer was yes, since GVT-g was "Software-mediated Passthrough", and it didn’t used Hardware features. The version they upstreamed on the Kernel seems to rely on the IOMMU, however.
Detail your Software configuration. Hardware seems to be fine. As I stated, it HAS to work. There is no further GVT-g limitation besides Aperture Size. And if you can enable VT-d you have the IOMMU working. Did you tried to add the intel_iommu=on Linux Kernel parameter and check if you have IOMMU Groups created? If it can do so, IOMMU works.
This isn’t true, however you do need to enable both VT-D and ASPM for IOMMU to work.
Bios->Advanced->PCIe/PCI configuration->ASPM