[Problem] Bifurcation choices on Supermicro X9DRH-iTF

beepee · February 23, 2024, 2:11am

Not sure if what I am attempting is possible or if I am even using the right approach:

X9DRH-iTF on latest BIOS (3.3) already has options enabled for bifurcation. HOWEVER, the strange/troubling thing is that the options for bifurcation only gives a single choice effectively meaning there are no options. For the physical x8 ports, only “x8” or “x8x8” choice is available, and for the physical x16 port, only “x16” is available. Opening BIOS in AMIBCP 4.55 shows exactly the same thing - only one choice available per port.

Here’s some of the output from ifrextractor on the “MiniSetupResourceSection” (BIOS appears to be Aptio IV). Note there is only one “OneOfOption” entry (this is for the x16 port):

0x2037B: 			EqIdVal QuestionId: 0x1DD, Value: 0x0 { 12 06 DD 01 00 00 }
0x20381: 			OneOf Prompt: "IOU3 - PCIe Port", Help: "Functions visible based on this setting : x4x4x4x4 (Fun 0/1/2/3 visible) x4x4x8 (Fun 0/2/3 visible) x8x4x4 (Fun 0/1/2 visible) x8x8 (Fun 0/2 visible) x16 (Fun 0 Visible)", QuestionFlags: 0x10, QuestionId: 0xDB, VarStoreId: 0x1, VarOffset: 0xE9, Flags: 0x10, Size: 8, Min: 0x0, Max: 0x0, Step: 0x0 { 05 A6 21 03 45 03 DB 00 01 00 E9 00 10 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 }
0x203A7: 				OneOfOption Option: "x16" Value: 4, Default, MfgDefault { 09 0E 4D 03 30 00 04 00 00 00 00 00 00 00 }
0x203B5: 			End  { 29 02 }
0x203B7: 		End  { 29 02 }
0x203B9: 		SuppressIf  { 0A 82 }
0x203BB: 			EqIdVal QuestionId: 0x1DD, Value: 0x0 { 12 86 DD 01 00 00 }
0x203C1: 				EqIdVal QuestionId: 0xDB, Value: 0x4 { 12 06 DB 00 04 00 }
0x203C7: 				Not  { 17 02 }
0x203C9: 				Or  { 16 02 }
0x203CB: 			End  { 29 02 }

Every bifurcation option listed on this motherboard is the same - only one “OneOfOption” for each entry, with the difference being which one option is presented. I have also tried different combinations of BIOS options and PCIe cards - the available BIOS options do not appear to change.

Looking at other similar gen Supermicro motherboard (X9DRi) in AMIBCP shows what I would expect - multiple choices (i.e x4x4 or x8) per port.

So I suppose what I am asking is: is there a way to add/enable these additional choices? I can’t seem to figure out how I would. Have played with the various UEFITools/UEFI Editor as well as different versions of AMIBCP.

Any guidance would be appreciated. Thanks!

beepee · February 25, 2024, 6:11pm

UPDATE: Decided to use datasone’s modded GRUB shell to test changing the NVRAM variables manually and test the result. No sense in hacking up the BIOS if it wouldn’t work in the first place (I think, anyway).

I was fairly confident I would at least be able to get the x16 port to work. I also had high hopes for x8 slots 2, 3, 6 and 7 as they didn’t share lanes with any onboard devices.

TL;DR: No dice, weird results. Questioning sanity as Insomnia continues…

X9DRH PCIe Routing

I made some assumptions when doing this testing, first that the actual NVRAM configuration values were enumerated in the same order as the items were listed in the setup text.

BIOS Setup text:
Functions visible based on this setting : x4x4x4x4 (Fun 0/1/2/3 visible) x4x4x8 (Fun 0/2/3 visible) x8x4x4 (Fun 0/1/2 visible) x8x8 (Fun 0/2 visible) x16 (Fun 0 Visible)

Therefore, hopefully:

x4x4x4x4 = 0x0
x4x4x8 = 0x1
x8x4x4 = 0x2
x8x8 = 0x3
x16 = 0x4

This order was consistent with the NVRAM values for the BIOS defaults for said slots (i.e the x16 slot default was 0x4 and the paired x8 slots were 0x3) so I am pretty confident about this. Also compared other Supermicro X9xxx BIOS with working options and it used the same mapping.

Testing:

I used the IFRExtractor tool on the “MiniSetupResourceSection” to get the values used for testing. After installing cards, I would boot to the modified shell and use “setup_var_cv” to set NVRAM values. Several times I power cycled/rebooted and re-read the values to make sure they were not being overwritted/reconfigured and this did not ever seem to be an issue.

I tested with two different noname e-bay 8x bifurcation cards, a dual M.2 and dual U.2, not that it should really matter. Test OS was Fedora 39, kernel 6.6.latest…

IIOU(CPU) 1 IOU 2 - PHY SLOT 2 (x8) and 3 (x8)

Default x8x8 (0x3), offset 0xA3, size 0x1
Installed both PCIe cards (slot 2 and 3)
– Test @ default (0x3): Sees first drive in each slot (as expected)
– Test @ x4x4x4x4 (0x0): Only sees first drive in slot 3
– Test @ x4x4x8 (0x1): Sees first drive in slot 2 and slot 3
– Test @ x8x4x4 (0x2): kernel panic?!

IIOU(CPU) 2 IOU 2 - PHY SLOT 6 (x8) and 7 (x8)

Default x8x8 (0x3), offset 0xE7, size 0x1
Installed both PCIe cards (slot 6 and 7)
– Test @ default (0x3): Only sees first device in slot 7 (???)
– Test @ x4x4x4x4 (0x0): Only sees first drive in each slot
– Test @ x4x4x8 (0x1): Only sees first drive in each slot

IIOU(CPU) 2 IOU 3 - PHY SLOT 4 (x16)

Default x16 (0x4), offset 0xE9, size 0x1
Installed x8 to x4x4 u.2 card
– Test @ default (0x4): no devices (???)
Sanity check: Installed x16 VGA card, verified working and x16/8GT link via lspci -vvv
reinstalled x8 to x4x4 u.2 card
– Test @ x4x4x4x4 (0x0): no devices
– Test @ x4x4x8 (0x1): no devices, reboot hangs at POST 0x91
– Test @ x8x4x4 (0x2): no devices, reboot hangs at POST 0x91
– Test @ x8x8 (0x3): no devices
Installed x8 to x4x4 m.2 card
– Test @ x8x8 (0x3): no devices
– Test @ x4x4x4x4 (0x0): no devices

I did some additional testing on CPU2 IOU1 (Slot 5) not listed but had no progress (always saw first device despite config).

I’m really at a loss as to what is going on - especially with the x16 slot. Unfortunately I only know enough to be moderately dangerous here. I don’t know if there are weird compatibility issues with my SSDs or my cheap aftermarket PCIe cards, or what else could be going on. All of my SSDs are Intel DC series (I even tested with one of those $10 16GB Optane m.2 drives).

If I get another burr up my butt and do additional testing or have a breakthrough I will update this thread, otherwise I will let it die.

New to the board so Mods, please be gentle if I am doing something dumb here. Thanks!

kosmik2001 · October 13, 2024, 5:11pm

Personally, I hope you’ll come back to this issue.
I decided to go the brute force route. I searched for IOU3 and found “setup” and pulled out the “MiniSetupToolsSection” module from it.
HexEditor changed the setting for x16 “0x04”. I replaced the bits that changed the setting in “X9DR3P9.B20” in the BIOS accordingly (not stupidly).
I flashed the module back and wrote it into the BIOS. As a result - a brick, although it does start.
I restored it via the network (BMC).
The whole problem is that the lack of bifurcation can be bypassed with a plx bridge, the cost (at the moment) is slightly higher than the motherboard. That’s the main problem.

twochannel · March 3, 2025, 2:55am

Bumping this thread. Like you’ve posted, I’ve also been researching to add bifurcation to a supermicro board. I have an x9dre-ln4f. From everything I can tell it can physically do it based on chipset and sister boards support it (x9dri). The bios mentions bifurcation, but the menu options only have one static option. x16, x8, or x8x8.

Likewise, the IRF extract has help text that mentions the other normal values for bifurcation.

		OneOf Prompt: "IOU1 - PCIe Port", Help: "Functions visible based on this setting : x4x4 (Fun 0/1 visible) x8 (Fun 1 visible)", QuestionFlags: 0x10, QuestionId: 0xA0, VarStoreId: 0x1, VarOffset: 0x92, Flags: 0x10, Size: 8, Min: 0x0, Max: 0x0, Step: 0x0
			OneOfOption Option: "x8" Value: 1, Default, MfgDefault
		End

I was trying to understand the relationship between the enabled speeds an the phrase "Fun 0/1 visible), to correlate the fun value with something somewhere else but I haven’t been able to connect those dots yet.

I don’t think anything is suppressed but I’m still learning. Other than this post I’ve not found any other discussions that match our situation with no dropdown options in AMIBCP. I feel like if we can add them we’d be good to go and would have control in the bios instead editing the default value of the existing option.

We’re so close its almost like there’s some other bios setting that is preventing these from showing other options.

I was doing some value comparisons between what my board supports today and what others support, and with verbose in irf extract and I see more than the value bit change.

From an x16 slot on an xdrfi firmware. See the 3rd byte change? And the 6th value means different things based on the slot being x16 or x8.

edit: i think this is nothing, can likely disregard.

0x1EBDA: 	OneOfOption Option: "x4x4x4x4" Value: 0 { 09 0E 42 03 00 00 00 00 00 00 00 00 00 00 }
0x1EBE8: 	OneOfOption Option: "x4x4x8" Value: 1 { 09 0E 43 03 00 00 01 00 00 00 00 00 00 00 }
0x1EBF6: 	OneOfOption Option: "x8x4x4" Value: 2 { 09 0E 44 03 00 00 02 00 00 00 00 00 00 00 }
0x1EC04: 	OneOfOption Option: "x8x8" Value: 3 { 09 0E 45 03 00 00 03 00 00 00 00 00 00 00 }
0x1EC12: 	OneOfOption Option: "x16" Value: 4, Default, MfgDefault { 09 0E 46 03 30 00 04 00 00 00 00 00 00 00 }

here’s the x8 slot

0x1EAD8:	OneOfOption Option: "x4x4" Value: 0, Default, MfgDefault { 09 0E 41 03 30 00 00 00 00 00 00 00 00 00 }
0x1EAE6: 	OneOfOption Option: "x8" Value: 1 { 09 0E 40 03 00 00 01 00 00 00 00 00 00 00 }

for my x9dre
x16

0x2ACD6: 	OneOfOption Option: "x16" Value: 4, Default, MfgDefault { 09 0E 4C 03 30 00 04 00 00 00 00 00 00 00 }
0x29F22: 	OneOfOption Option: "x8x8" Value: 3, Default, MfgDefault { 09 0E 4B 03 30 00
 03 00 00 00 00 00 00 00 }

x8

0x2ABDE: 	OneOfOption Option: "x8" Value: 1, Default, MfgDefault { 09 0E 46 03 30 00 01 00 00 00 00 00 00 00 }

I have not gone nearly far enough to test anything yet.

twochannel · March 3, 2025, 3:01am

Tagging @andreu69 just in case from Supermicro X9DR7-LN4F CFG UNLOCK+BFI+UNHIDE BIOS - #2 by MeatWar

Sweet_Kitten · March 3, 2025, 3:20am

Exactly. There’s no suppression making the setup items invisible.

Rather the sub-menu is missing.

twochannel · March 5, 2025, 1:07am

Agreed - since my bios only has one “OneOfOption”, there is no menu, just the default/only option.

I may try ru.efi soon to see what happens when I set a slot’s value to something other than what it is now.

I’m trying to think back through all the conversations I’ve read to recall if anyone with a supermicro x9 gen board has been able to edit this to work. The X9DRI folks ended up getting a bios that supported the feature from supermicro.

twochannel · March 5, 2025, 4:50am

I’m trying to make sense of what I’m seeing. Would appreciate some guidance.

This section of my IFR extract applies to one of the ports I see in the BIOS menu.

0x2B180: 		SuppressIf  { 0A 82 }
0x2B182: 			EqIdVal QuestionId: 0x1DD, Value: 0x0 { 12 06 DD 01 00 00 }
0x2B188: 			OneOf Prompt: "IOU3 - PCIe Port", Help: "Functions visible based on this setting : x4x4x4x4 (Fun 0/1/2/3 visible) x4x4x8 (Fun 0/2/3 visible) x8x4x4 (Fun 0/1/2 visible) x8x8 (Fun 0/2 visible) x16 (Fun 0 Visible)", QuestionFlags: 0x10, QuestionId: 0xDB, VarStoreId: 0x1, VarOffset: 0xE9, Flags: 0x10, Size: 8, Min: 0x0, Max: 0x0, Step: 0x0 { 05 A6 21 03 44 03 DB 00 01 00 E9 00 10 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 }
0x2B1AE: 				OneOfOption Option: "x16" Value: 4, Default, MfgDefault { 09 0E 4C 03 30 00 04 00 00 00 00 00 00 00 }
0x2B1BC: 			End  { 29 02 }
0x2B1BE: 		End  { 29 02 }
0x2B1C0: 		SuppressIf  { 0A 82 }
0x2B1C2: 			EqIdVal QuestionId: 0x1DD, Value: 0x0 { 12 86 DD 01 00 00 }
0x2B1C8: 				EqIdVal QuestionId: 0xDB, Value: 0x4 { 12 06 DB 00 04 00 }
0x2B1CE: 				Not  { 17 02 }
0x2B1D0: 				Or  { 16 02 }
0x2B1D2: 			End  { 29 02 }
0x2B1D4: 			OneOf Prompt: "  Slot 3 Link Speed", Help: "Select Target Link Speed Gen1,Gen2 or Gen3", QuestionFlags: 0x10, QuestionId: 0xDC, VarStoreId: 0x1, VarOffset: 0xC6, Flags: 0x10, Size: 8, Min: 0x0, Max: 0x0, Step: 0x0 { 05 A6 3E 03 3F 03 DC 00 01 00 C6 00 10 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 }
0x2B1FA: 				OneOfOption Option: "GEN1" Value: 0 { 09 0E 40 03 00 00 00 00 00 00 00 00 00 00 }
0x2B208: 				OneOfOption Option: "GEN2" Value: 1 { 09 0E 41 03 00 00 01 00 00 00 00 00 00 00 }
0x2B216: 				OneOfOption Option: "GEN3" Value: 2, Default, MfgDefault { 09 0E 42 03 30 00 02 00 00 00 00 00 00 00 }
0x2B224: 			End  { 29 02 }
0x2B226: 		End  { 29 02 }

The first SuppressIf { 0A 82 } at 0x2B180 evaluates the the EqIdVal QuestionId: 0x1DD and if false? (0x0), then the following OneOf Prompt is displayed at 0x2B188. This is a good example of why we see what we see in the BIOS menu, only one option.

The only thing I see other than 0x1DD as part of a many QuestionIDs, is this line at the bottom of the extrat. I presume this is 00/false because I can see the x16 menu item in the SupressIf.
0x3013A: CheckBox Prompt: "", Help: "", QuestionFlags: 0x0, QuestionId: 0x1DD, VarStoreId: 0xD, VarOffset: 0x1, Flags: 0x0, Default: Disabled, MfgDefault: Disabled { 06 8E 00 00 00 00 DD 01 0D 00 01 00 00 00 }

If this EqIdVal was evaluating differently, this menu item would not be in the bios menu. Right?

However, for the next SupressIf, I’m not sure how to read the Not and Or conditionals. There’s one for another EqIdVal for 0x1DD, and one for 0xDB, which matches the first QuestionID about the IOU3 setting of x16.

My guess is, it reads like this: If 0x1DD is not 0x0 (it is 0x0) OR 0xDB is x04 (it is 0x4), this would evaluate to True since one of the two statements is true. Therefore, display the Gen1/2/3 speed setting (and I do see it)

If I have this correct, In theory, If I set the IOU3 value to non-4, the speed menu would go away. I don’t want to do this, just proving out my understanding.

Based on testing from the originator of this thread, simply changing QuestionId: 0xDB would not likely work. Perhaps my board/bios is different enough that it would, but not likely. What is missing from our bios versions that others (that are very similar, same chipset, topology, etc) have?. Can we extract out what works from those and place into ours? Can we find whatever 0x4 is pointing to and see if there is there are other options? and if not add them.

Elsewhere I found a thread of someone analyzing the cpu contacts and reading the registers for a haswell to go about this but i couldnt find the same intel docs for Ivy Bridge (e5-2600v2) yet.

also, for Ivy bridge intel calls it lane partitioning. pg.19 https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v2-datasheet-vol-1.pdf

twochannel · March 10, 2025, 7:52pm

I’ve had similar results, I was not not successful. I tried two things.

I edited the Setup section in the bios with ru.efi (*this was a bit of a pain, more below) by setting the values for one of the IOUs to 0. Truenas did not see any more Root Ports beyond the three hanging off that CPU. (2 x16s, and 1 x8)
Using ru.efi, I edited the register address for bifurcation based on the intel cpu register spec sheet for my processors. (see [Guide] - How to Bifurcate a PCI-E slot - #127 by twochannel) I thought I was on to something with bit 3 to enable bifurcation and then 2:0 to set the config, but that didn’t work either. I was careful to continue booting vs restart since I’m sure these registers would be overwritten on boot.

*I had to use an older version of ru.efi to get it to run, version 5.32.0423 worked for me. The newer ones would hang at execution once the bytes started to load.