I’m looking for some advice and possibly BIOS modding help for an older X99 platform.
System:
Motherboard: Gigabyte X99 (LGA2011-3)
CPU: Xeon E5-2697 v4
GPUs: 4x NVIDIA Tesla P100 (eventually)
Boot GPU for testing: GTX card works normally
BIOS version I’m using is X99UD3P.F22
I’ve been reading through various threads here and elsewhere regarding Above 4G Decoding, ReBAR support, Tesla cards, and X99 BIOS modifications. From what I’ve found, my Gigabyte X99 BIOS does not appear to natively support all of the options required.
What I’ve tried so far:
Set PCIe slots to Gen3
Changed OS type to Windows 8/10 WHQL
Disabled CSM
Experimented with Above 4G Decoding and ReBAR-related settings where available
Tested with a standard GTX card (boots fine)
The problem is that whenever I install a Tesla P100, the system refuses to POST/boot. Even stranger, after attempting to boot with the P100 installed, the board can get stuck in a state where it won’t POST even after removing the P100. At that point I usually have to clear CMOS by removing the battery and resetting everything before it will boot again.
This makes me suspect one of the following:
Missing or incomplete Above 4G Decoding support in the BIOS.
PCIe resource allocation issues with the Tesla card.
A BIOS configuration issue specific to headless Tesla accelerators.
Something related to UEFI initialization on this particular X99 board.
My end goal is to run 4x Tesla P100 GPUs in this system, so I know proper large BAR/MMIO allocation will likely be required.
A few questions:
Has anyone successfully run Tesla P100s on a Gigabyte X99 board?
Does this platform require a BIOS mod for reliable Above 4G Decoding support?
Is Resizable BAR support useful or necessary in this scenario, or is Above 4G Decoding the primary requirement?
Are there any known X99 Gigabyte BIOS mods that add or fix these features?
I’ve also seen several references to Koekieezz creating modified BIOSes for X99 and other platforms. If you’re around and willing to take a look, I’d definitely appreciate any guidance or recommendations.
@x99lover Lots of H97, Z97, and X99 from gigabyte boards lack this feature. I could suggest you to go to Gigabyte eSupport website, then make a technical support for your motherboard. I strongly suggest you to set the region to taiwan and language to english like this:
Since you’re requesting a bios update, just so to be complete, you can ask them this:
Above 4G Decoding Option in the bios, and working. Tell them to set this option to Disabled by default.
2023 Updated Secure Boot keys
In the mail, you can make up some words that you have 4 Tesla P100, and during boot, it gave you “PCI Out Of Resource” warning, and you cant boot. Tell them that you need the above 4g decoding option (disabled by default), and 2023 secure boot keys mitigation.
Also dont forget to mention that the bios version that is stable, so that they will work it from that bios version. Sometimes the latest on the gigabyte web is not stable or even buggy, so mention the version that is stable (for example, Z97X-UD5H-BK beta bios F9 are buggy (all versions/revision of F9 bioses), i told them F8 was stable and OC ran wonderfully, so they gave me a revised F8 bios with TPM2.0 Support (X99 wont need it cause it’s supported by default), Above 4G Decoding option, and 2023 Secure boot keys), and this is the complete bios by my definition (stable, all features requested given).
Once you received the bios, and you still want the mod, send the bios given by them as-is into here. Sometimes even with above 4g decoding only enabled, the mmio base weren’t changed, rendering above 4g decoding still functional, but not with high vram capacity, it needs to be changed later (i will take a look of the given bios from you later on).
I’m running into a similar issue. I am able to run four Radeon pro v340 GPUs on f22 (4g decoding enabled). But i cant successfully boot with 3 nvidia GPUs (several models, I even tried 3 to 4 cards will low vram).
I saw a post of a person able to run 3 Nvidia p40s with no issues. But he als uses the rebar uefi which i have no way to enable on linux.
@JMu55 this is due to tesla series with high vram uses high bar size which is not resizable bar. The MMIO Base should be increased from 1G to 2G, and optionally, MMIO High Size can be increased from 256 to 1024G (which is what i usually did).
Hey @Koekieezz, thanks again for the detailed guidance earlier.
I followed your advice and opened a technical support case with Gigabyte (Taiwan region, English), requesting a working Above 4G Decoding option built from the stable F22, set to disabled by default.
Unfortunately they declined their response was that the GA-X99-UD3P (rev. 1.0) has reached End of Life, so no new BIOS builds are available for it. I pushed back once but it’s a firm EOL wall. So I’m hoping you might be willing to take a look at the mod directly, like you mentioned.
Would you be able to mod the stock F22 so that:
Provide a working Above 4G Decoding option (disabled by default)
Change the MMIO base so it actually maps the large BARs? End goal is 4x Tesla P100 (16GB each).
I can provide whatever you need:
Stock X99UD3P.F22 ROM (from Gigabyte’s site)
A full IFR dump of the Setup module (already extracted, I can attach it if needed). For reference, the Above 4G Decoding option is present at VarOffset 0x495 (i think?), and there’s a “PFMMIO 64 bit Resources Padding” option but no MMIO High Size setting anywhere in the IFR, which I think lines up with what you described about the MMIO base needing a manual change.
Board revision: rev 1.0
No rush at all, and I really appreciate you even considering it. Let me know what files would be most useful and I’ll get them posted.
You have to persuade them politely, because that’s how i kept getting the support from them… so yeah word it diligently and make it feel important for you and them, yet simple worded.
Basically, if i wanted to ask for the bios, i explained what i wanted, for what reason, add the secure boot 2023 pkfail issue, etc. Probably the first 2 replies was not taken by them at all, but 3rd time a charm, and that’s how i did obtain Above 4G Decoding for both Z97 gigabyte bioses along with tpm2.0, built based from stable bios from their web support.
I cant do anything if there is no above 4g decoding, it’s out of my skills…
Are we sure Above 4G Decoding doesn’t exist in F22? The option is there, and it toggles on and off, but when you toggle it, it doesn’t boot. From some github issue I saw, it looked like the newer UD4 motherboard had it natively and was just missing ReBarDXE. Either way I will ask gigabyte again, thank you.
@x99lover i actually never look on the bios, but could you confirm if the togglr for above 4g decoding is there, only broken/not working well? 'll try to see what i could do later once you’ve said if the option is there available from stock F22.
Yes the option is there on version F22, and I can toggle it on and off, weather or not it actually does anything or is properly implemented I don’t know, maybe the option and code for it is there but the switch setting doesn’t actually toggle it? Not sure. I’m not sure how to tell if the option actually does anything or not. It seems to just not want to boot after enabling it with the P100 installed tho.
I suggest you downgrade to F21, then update to this modded bios, and once you’ve succeed, just go to bios, disable csm and enable above 4g decoding, save and exit, then replace the gpu and see if it could boots with your tesla cards.
If it boots, do I run ReBarState exe and set to 32?
Did the same thing as my buddies attempt at modifiying the BIOS did, boots with 1x p100 gpu, but no more than that, doing that causes it to stop booting even after removing the extra GPU(s) until the CMOS is reset.
Hmm it seems like a bug on their side? You do not have to run rebar because the gpu isn’t rebar capable, it has 16GB BAR size by default.
I think requesting bios F24 with Above 4G Decoding option exposed (and working, but disabled by default) would do, but also, tell them that you’re running 4x Tesla P100 and with above 4g decoding enabled, it wont boot, implying that the above 4g decoding implementation is incomplete… Make your own word about this discovery to tell them.
I saw the MMIO low is at 2G by default which is enough, and changed the MMIO High size to 1024G from 256G. Even at 256G it should be enough for 2 Tesla P100s if you maxxed out the ram in the motherboard (adjusting it to the max should be done to make your board able to handle max ammount of ram + more bar size at mmio spaces).
Also you can add an additional request to include secure boot 2023 keys for pkfail mitigation alongside the above 4g decoding request.
Yes I’ve been using Windows for testing but I plan to eventually use ProxMox, however I’d like to make a correction, after some testing and after removing the GTX GPU, I was able to boot 3x P100 GPU’s, but adding the 4th and final GPU causes it to stop working again. I have 32gb of system ram in the motherboard.
@x99lover Could you go to Device Manager → View → Resource by Type → Large Memory, and see the dropdown option and send a screenshot here with your 3 Tesla GPUs (since the 4th wont work)? Just wanted to know the MMIO space assigned to those gpus.