EDIT*** It’s become clear the issue isn’t vRAM related, but more than likely an incompatibility between the system BIOS & GPU’s with moden UEFI firmware
EDIT2*** So the issue turned out to be the Dell BIOS is incompatible with GPU’s that include UEFI compatibility (basically anything in the last 10ish years). The fix is to remove the UEFI image from the GPU vBIOS forcing it back to compatibility mode and it boots fine now. So says the 2GB P400 Quadro I now have running in my R210ii Plex Server.
-------------------------------------------------------------------
Trying to enable all PCIe GPU’s in Dell Servers.
Currently only some cards work, and from my testing and research the BIOS is looking for a PCIe GPU with <= 1GB RAM at boot. Any more than 1GB vram and the system won’t post.
Ref: https://forum.level1techs.com/t/dell-pow…on-found/134343
With UEFITool I was able to enable PCIe Hot-Plug on my R210 ii which has allowed me to test that GPU’s with >1GB do in fact work by hot-plugging the GPU after Windows has loaded. If a GPU with >1GB is installed at POST, the system halts with a static cursor _ and doesn’t boot. I have even managed to insert the card after the device check and before the POST screen and have it recognised in the EFI Setup Screens, so clearly there is some arbitrary check stopping the use of these cards at boot.
There are many Dell server models with a similar issue, so once a patch is found it will hopefully work with other models. My specific case for testing is as follows…
Model: Dell PowerEdge R210 ii
BIOS Version: 2.8.0 (Home Lab, don’t need the performance hit of the later patches).
I have multiple system boards & an SPI flasher should we need it.
2.8.0 Download
https://www.dell.com/support/home/au/en/…oweredge-r210-2
2.10.0 Download (Includes Spectre & Meltdown patches for those that need them)
https://www.dell.com/support/home/au/en/…oweredge-r210-2
I’ve had a look around IDA, tho my assembly knowledge is limited and figured I should ask for some pointers. I don’t believe the lock is Device ID related.
If a solution is found, I would be most interested to know how it was discovered.
---------------------------
@Lost_N_BIOS does this sound like a challenge you’d be keen to help me work through?
My biggest hurdle is working out which module detects PCIe devices. I assume that’s where this check takes place. I’m comfortable enough working through the assembly in IDA once I have an idea of where to look.
Wow, I’ve seen hot-plug PCIE option in many BIOS, but never knew someone could really do that and not release the magic smoke 
Since you have programmer, send me a dump of the BIOS, if there is two chips dump both and send in one zip, I will compile them to inspect/edit.
* Edit -From your list at the level1techs site, looks like all GPU’s you tried that work are old/legacy (non-UEFI vBIOS), and all that fail are probably Legacy/UEFI vBIOS (both in vBIOS Image)
So, it may be failing due to UEFI in vBIOS. Do you have any modern UEFI vBIOS capable GPU’s that are below the 1GB vRAM you suspected.
I doubt any whitelist or block/allow list by GPU chipset, model, or brand could be used, simply based on the random ones that do work
Maybe… If we are lucky, I can add UEFI image to vBIOS on one of your known currently compatible GPU’s, then you can test above theory
Please send me vBIOS dump from these two cards. You can dump with GPU-z or use programmer, up to you
XFX HD 5450 512MB DDR3 (HD-545X-YRH2 V2.1)
BFG GTX 260 (BFGRGTX260896OCB)
Reverse test of the above could be done too, remove UEFI image in vBIOS of any of these (if none have UEFI image and are only legacy vBIOS, then theory shot down right there)
Zotac mini GTX 1070
Club 3D R9 280x
Palit GTX 1060 6GB
Still reading your post there, so more thoughts may get edited in here shortly
This may also be a simple BIOS setting change needed, based on the error info you found (system resource configuration error)
lol, I cooked the PCIe bus on my first board, and a GPU also 
. So i bought 2 more spares just in case 
. Hot-Pluging is normally quite safe tho, my problem was power modding the PCIe riser to bypass the 25W power limit and I shorted the 12v rail.
Anywho… BIOS dump is attached. Only 1 chip.
That list on level1tech’s isn’t mine, but rather someone else’s notes on the same problem.
I have looked through all the Available & Hidden BIOS options and nothing that remotely resembles gfx helped.
I have a GT710 2GB, GT530 2GB, RX2600PRO 512mb on hand, but nothing ‘new’ < 1GB . I have others, but those are the only three that fit in the ~25W power budget this server has for its PCIe slot.
My ultimate Goal is to install a P400 Quadro to serve as a Plex Transcoder.
I did have one other idea… can we mod a vBIOS to report only 1GB RAM? that would also test the VRAM theory
R210ii_2_8_0.zip (2.47 MB)
GK208_GT710_2GB.zip (124 KB)
Ok, had another idea. I can buy identical GT710’s in 1GB & 2GB variants. That’s the easiest way I can think of testing the vRAM theory unless we can change the reported vRAM on the card I already have.
Gigabyte GV-N710D5-1GL & GV-N710D5-2GL
So I’ve installed a 1GB GT 710 and same issue. System halt with the _ cursor. There goes my 1GB vRAM theory, super glad I asked before wasting time in IDA.
Guess that leaves the UEFI vBIOS incompatibility.
Is that something we can add support for in the Dell BIOS?
Some Firmwares allow you to decide whenever you want to prioritize loading UEFI or legacy BIOS Option ROMs, or even disable loading an Option ROM alltogether. Do yours have something similar to this? https://www.thomas-krenn.com/de/wikiDE/i…_pcie_oprom.php
Perhaps one of the reasons why I love Supermicro so much is precisely because they often allows you to disable loading Option ROMs with a PCIe Slot level granularity: https://hardforum.com/threads/disable-op…-board.1984211/
…albeit some Workstation ASUS ones can, too: https://www.thomas-krenn.com/de/wikiDE/i…nfiguration.php
That can help in some obscure Option ROM compatibility issues, like it seems that yours is.
At some point I tested if I could not initialize a Video Card for VGA Passthrough purposes. With the IGP disabled and disabling loading the Option ROM of the PCIe Slot where the discrete Video Card was, the Motherboard beeped with an error code that pointed to a VBIOS not found error, precisely as intended. It could still pass POST and the video output turned on as soon as Linux GPU Drivers kicked in.
@zir_blazer that would certainly make life easier. Alas no options like that here, not even hidden options as far as I can tell.
Attached are all the options (Available & Hidden) I can access. (Extracted with H2OEZE)
Also, because I like to try stuff myself… i’ve attempted to remove the GOP driver from the GT 710 I have with no luck. I think i’m stuck on the checksum not matching. My attempt is attached.
MenuOptions.zip (12.5 KB)
GK208_1GB.zip (149 KB)
@Lost_N_BIOS so your nudge in the direction of UEFI compatibility was what I needed.
I have now successfully removed the UEFI image from my P400 Quadro and it boots perfectly.
I’m still interested to know if there is anything we can do to the Dell BIOS so any card can be used without having to mod it first? But for now this is working great.
Also, what’s involved in unlocking all the BIOS options I can see in H20EZE for this BIOS? Is it a simple patch or more involved?
@theunknown
This is a problem I’ve had for a couple of months now. Was just looking to tackle it again, and found your post. THe magic of GOogle.
Do you have instructions or a link to how you removed the UEFI image from the P400 Quadro? Have you tried it with a GT 710?
I’m looking at an R210 ii (Riverbed modded back to stock), and trying to get the GT 710 working in it.
@wan0net
It will work on any GPU I expect. I couldn’t find instructions for Nvidia cards so had to teach myself the basics of a vBIOS to get it done, in the end it’s super easy. I’ll post a tutorial in the next couple days.
EDIT* Incidentally, if you don’t have an iDRAC6 Enterprise module yet, have a read of this to save you countless hours also
https://www.reddit.com/r/homelab/comment…ac_no_boot_fix/
Thanks, much appreciated!
@theunknown - So, to clarify, you’ve blown away the 1GB as suspect, correct? And now, think maybe what I mentioned about UEFI vBIOS could be part of the problem?
Ohh, wait! So, you solved it 
And what I guessed about the vBIOS UEFI+Legacy image was correct? If yes, then, yes, it may possibly be resolved by changing a BIOS setting
Do you have a flash programmer? If yes, dump the BIOS and send to me (if there is two BIOS chips, dump both and send to me)
I HATE INSYDE BIOS… There, that’s out of the way. Unlocking BIOS menus may be possible, maybe not for me, but Insyde Pro’s at BIOS-Mods.com forum may be able to do it for you if I can’t
Changing BIOS options, you can see or not, can be done easily with H22OUVE (dump vars, edit, flash back) - Or by dump BIOS, edit setup + NVRAM, flash back, similar/same as how you enabled hot-plug
As for unlocking menus, settings etc, I’d need to see full images of your BIOS first (mainly, just show me one main image of each main page, so I can see what you can see now)
Then any areas you want full options you may be missing show me the entire section like advanced >> Submenu with settings you want that are missing
I can always easily make hidden settings visible to you in any section you can already see, but sometimes I can’t make a menu section visible to you if it’s not already (Sometimes I can though, I’ll have to look after I see your images first)
Looking through the BIOS options using the HDR from stock EXE, I see you can change the following settings and deal with this without having to edit vBIOS
EFI Device First (Default Enabled) >> Set Disabled *may fix issue alone
If that fails, set this >> Boot Mode (Default UEFI) >> Set >> BIOS (= Legacy)
Your BIOS, from what I can see in HDR, which may not be all, does not offer granular enable/disable of UEFI/Legacy rom loading, so above may be your only two possible things to play with in attempt to get around this without removing GOP image from GFX crads vBIOS
@wan0net - if you do the vBIOS edit, the part to worry about here is flashing of mod BIOS, is that possible on your model/series (especially with nvdia).
You’ll have to google about that and your card, your cards series etc. If you can’t flash mod vBIOS on your card then you can’t dump vBIOS and edit it to remove the UEFI/GOP image.
If you find you can, and you simply need edited vBIOS, post your vBIOS here and I can remove GOP for you
To you both, simply setting the legacy is last flag may fix this too, or setting legacy is only bit may as well, unsure about this, testing required.
@Lost_N_BIOS
Yes I was able to get it working by removing the EFI vBIOS Image.
I tried the “Boot Mode” BIOS Option without success, but completely missed “EFI Device First”. I’ll try that when I get home & report back, that would be a much simpler fix if it works.
I’ll also reply with the BIOS Menu info you’ve mentioned.
Make sure you change the option in NVRAM/VSS too (not just in setup), otherwise the change may not be applied.
Easiest way to make those changes is dump current vars with UVE, make your desired edit, then program back. This changes NVRAM/VSS (current in use settings)
Dump
H2OUVE.exe -gv vars.txt
Write
H2OUVE.exe -sv varsM.txt
Use setup IFR to get varstore from the setting you are changing, then locate that varstore by GUID at top of IFR, then find it in the dumped vars (there may be multiple copies, or two different sections with same GUID, edit both)
On the IFR per the setting you want to edit, use the variable from >> VarStoreInfo (VarOffset/VarName): 0x85 << to edit the vars at that byte
Example for above >> VarStoreInfo (VarOffset/VarName): 0x85, VarStore: 0x1234
VarStore: 0x1234 >> = VarStore: VarStoreId: 0x1234 [A04A27F4-DF00-4D42-B552-39511302113D], Size: 0x3CB, Name: SystemConfig << So we are editing “SystemConfig” generally, but more specifically GUID A04A27F4-DF00-4D42-B552-39511302113D and it may be in Vars under System Config and also Custom, or other names
VarStoreInfo (VarOffset/VarName): 0x85 << in vars, edit byte 85
If you need edited example, give me your vars dump, I will edit something and show you what I edited, then I’m sure you will see and get it (if you don’t already understand 
)
I’m running 2.10 and I don’t appear to have that EFI Device First (Default Enabled) >> Set Disabled - I have Set Boot Retry which appeared to work, and then on repeat reboots I got nothing from the card.
I’ll try and connect my card up to another PC and try and dump the vBIOS.
@Lost_N_BIOS Tried both “Boot Mode” -> Legacy & “EFI Device First” -> Disabled with no luck (yes with NVRAM reset also). Guess we’re stuck with modding the GPU vBIOS.
@wan0net This weekend i’ll have a bit of time for the tutorial. As Lost_N_BIOS said, being able to read & flash your GPU will be a necessity. Those instructions are around so you could start there for now.
@wan0net
Guide can be found here for those who wish to replicate this fix
[Guide] Remove UEFI Image from NVIDIA vBIOS
@wan0net - wait for theunknown to reply with test results anyway. This is probably a hidden setting on both you guys boards, I am not sure (Assume so).
The BIOS I was looking at is 2.10, so if you don’t see it then it’s a hidden setting. * Edit, yes, I checked, it’s suppressed, needs simple edit to make visible to you in boot section Similarly Boot Mode can be ungrayed w/ simple edit too (if it’s greyed out for you guys)
* Edi- here is the edits mentioned above, in case either of you know a little about editing (UEFITool + Hex editor skills required)
To reveal Quiet Boot, ACPI Selection, USB Boot, EFI Device First >>>
At extracted setup PE32 module, find the following string via hex
05 A6 4A 00 4B 00 01 00 34 12 73 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Directly before that, you will find the following string
0A 82 45 8A 01 00 00 00 00 00 00 00 45 0A 01 00 00 00 00 00 00 00 2F 02 29 02
Edit to
0A 82 45 8A FF 00 00 00 00 00 00 00 45 0A FF 00 00 00 00 00 00 00 2F 02 29 02
Next, for if Boot Mode is grayed out on you guys, find the following string
05 A6 9C 00 9D 00 05 00 34 12 7E 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Directly before that, you will find the following string
19 82 12 06 04 00 01 00
Edit to
19 82 12 06 04 00 FF 00
@theunknown - I did not mean NVRAM reset, I mean change the variable in NVRAM directly, sometimes (often), live/use BIOS settings are not pulled from setup at all, so any changes you make there may not be applied at all (ever)
Send me a dump of your BIOS, I will change this at both places (Setup and NVRAM) and then give you the before/after file so you can see where the edit is for each setting too.
@Lost_N_BIOS dump from my CH341A is in the 3rd post. (I use Version 2.8.0)
Dell Server unable to Post with UEFI GPU’s
Actually… that’s not the Modded BIOS, that’s from before I attempted the “EFI First” setting.
Thanks for your help to keep trying
@theunknown - Probably easier for you to just do the edit above, that I just edited into post, then you can change directly in BIOS as an actual setting in front of you. Do you need me to do that edit for you, or you can get it done?
This way, no NVRAM edit is needed by either of us, changing directly in BIOS setting will update the NVRAM value when you change the setting "in BIOS"
NVRAM/Vss is messed up in that BIOS at post #3, do you have any dumps from before you made ANY edits to the BIOS? See below, it’s OK