After months of trying, countless hours in Ghidra, around 26 executable patches, a full DSDT recompilation, and the creation of an independent orchestrator module injected into the BIOS… I was finally able to load a model and run inference natively on this motherboard.
Specs:
-
CPU: AMD FX-8350
-
GPUs: 2x Tesla P100 (16GB) + 1x Intel Arc A770 (16GB) + 1x AMD FirePro W9100 (16GB, on a PCIe 1x riser)
-
RAM: 16 GB DDR3
The primary aim was to run AI inference using whatever scrap hardware I had lying around, especially considering current RAM and SSD prices. I had a stash of older GPUs (like a FirePro S9170 already modded for vertical cooling, a GTX 1660, an RTX 2060, etc.). So the goal was to make them all work together using an old 990FX motherboard and an FX-8350.
The first steps weren’t easy, but with a bit of kernel tweaking, I was able to run some small models, given the amount of VRAM (around 40GB). Then I got my hands on two Tesla P100s and an Arc A770… and that’s when the war started.
I started with what I could find online about the 990FX chipset, but there wasn’t much about my specific motherboard. I had to turn my brain on and start from solid foundations, like xCuri0’s ReBar modules. (xCuri0, if you are reading this, you are my hero. My custom module uses your GUID in your honor). I don’t even know how to explain everything I’ve done over the last 3 months without writing a full research paper.
In the beginning, with the P100s installed (with the pins taped), the system booted normally, but the motherboard absolutely refused to assign the BAR to the GPUs, and the Arc A770 was trapped with only 256MB of assigned space. When I tried injecting the xCuri0 module, the only thing I got was a 00 POST error code.
Then I brought out Ghidra, AMI tools, UEFITool, and UEFIPatch, and kept going with several EXE patches. I even patched the AMD AGESA module. I gave it my best shot, ending up with more than 20 EXE patches, systematically modifying the 4G limits and other constraints inside PCIBUS, CSMDXE, PCIROOTBRIDGE, AMD BOARD INFO, and the AMD AGESA DXE DRIVER. But all I got was being permanently stuck on the d4 error code for a very long time.
At one point, I was so desperate that I completely dumped the ACPI DSDT table. I found that the code for 64-bit PCI compatibility was completely absent, so I rewrote the PCI0 tables, recompiled them (which took massive effort), and re-injected them. This alone wasn’t enough, but it ended up being an essential piece of the bigger picture.
As a final effort, I decided to build my own module that acts as a complete orchestrator, As a final effort, I decided to build my own module that acts as a complete orchestrator, heavily inspired by the xCuri0 ReBar module (but with over 2,000 lines of custom code). As you can see in the screenshots, I did a before/after test: pci=realloc doesn’t change anything—the custom BIOS mod does all the real work.
What my module does, in summary (extremely short):
Dynamic DSDT (ACPI) table patching: Instantly modifies the ACPI tables in memory (setting a variable MALH=One) to inform the operating system that a 64-bit memory window (QWordMemory) is available for use.
Preliminary Northbridge programming (CPU NB MMIO): Attempts to open at the CPU level (on the AMD Northbridge) the “windows” (Window 1) necessary to route communications to such high memory addresses.
Pre-scan and Link Disable (The concealment): Even before the normal BIOS inventories the connected PCI devices, this module scans everything. If it finds devices with massive memory requests (> 4GB, typically the Tesla P100s or compute GPUs), it temporarily pulls the plug on their PCIe bridge (Link Disable). The BIOS will believe those slots are empty (VID=0xFFFF), skipping them entirely and preventing errors. However, it takes care not to hide the primary video card (Intel Arc) to ensure something appears on the screen during boot.
PCI Enumeration Hook (PreprocessController): While the BIOS is configuring the remaining powered-on devices, the module intercepts these communications. In this way, it can handle the ReBAR for the primary video card (the Intel Arc), but deliberately refrains from applying aggressive “quirks” to the PCI bridges which, if done during the BIOS phase, would cause a black screen.
ExitBootServices (The final awakening for Linux): One Moment before control passes from UEFI to Linux (ExitBootServices), the module performs its final magic:
-
Definitively forces the CPU’s MMIO memory window (from above the RAM up to 256GB).
-
Opens the chipset’s HyperTransport “gates” (SR5690 HT Gate).
-
Applies the 64-bit prefetchable fixes to all PCIe bridges.
-
Reactivates the PCIe Link of the previously hidden GPUs.
-
Waits for the GPUs to respond (Link training) and finally programs their BARs above 4GB.
Not everything is 100% stable yet, but I’m working hard on it. Once everything is finalized, I will publish the module and documentation on GitHub. I’ll also post some inference benchmarks soon, as soon as the setup is fully dialed in.
Since new accounts can only post one image, here is the full gallery with all the terminal outputs: https://imgur.com/a/results-nyGocos