Help needed - PCIe bifurcation on Supermicro X11SSZ-F

Hi all, if you can spare some time I am in a greatly need of your help.

This is the situation. I have a Supermicro X11SSZ-F mobo, AOC-SLG3-2M2 card and two Corsair MP510 NVMes. 2M2 card is in the Slot6 x16.

As you can imagine, mobo doesn’t see two NVMes, just one, and with newest version of BIOS there is still no option to bifurcate x16 slot to x4x4x4x4 (or to any other value).

Can you please help me to make both of NVMes work…? (Current BIOS installed: https://www.supermicro.com/about/policie…wareItemID=5987)

Thank you in advance!

Do you have flash programmer, like CH341A?

* Edit - No, I wondered about programmer in case recovery from bad flash is needed, this is not something you can do remotely, at least not easily because Intel ME may need changed several times and some may be failed attempts. Best if someone is there in person you can guide through things at the very least.

Bifurcation is more for splitting lanes, giving more lanes to one port vs the other, I am not sure any of that applies to splitting a single PCIE slot into different devices.

Do you have it connected to an x8 slot? I see in the BIOS there is two lanes than can run at x8, so I assume there is also already x16 that goes down to x8 when other slots are used.

*edit 2 @patak17 - The following can be changed in Intel ME

PCIE.png

PCIE2.png



And the following can be changed at the BIOS level

Slot6.png

PEG-0-1-1.png

PEG-0-1-2.png

Ok, so for the CH341A, I will probably order one, just in case…thanks!

The thing is, I don’t want to split PCIe to use inside it 2 devices, of course…but to use AOC-SLG3-2M2 card with two NVMe memories inside…at the moment system only sees one as it is in slot x16. I need to make that slot x8x8 or x4x4x4x4 or something similar, so AOC-SLG3-2M2 can communicate both of the NVMe-s with the motherboard. Just lowering x16 to x8 or x4 is not something that will give the desired result, or am I wrong?

I saw that some users did just that, but on different motherboards…something like this:
[Guide] - How to Bifurcate a PCI-E slot

Can it be done on X11SSZ-F motherboard?

Well, it will only help you for the boards you have in your possession, the CH341A I mean. And to use that with boards that do not have socketed BIOS you will need SOIC8 and or SOIC16 test clip cables, depending on soldered on BIOS type.

If it’s in a x16 slot, then it should be able to work properly, since it would have the full x16 lanes. I general, anything x8 or above should allow it to work, if compatible. If you think it has to have an x8 slot only, then look at the board layout, surely there is a slot that is already x8

The guide you linked it similar to what I mentioned, which is not really what’s going to help you here. And your BIOS does not have the section that guide is discussing.
You already have available x8 or above lanes, if you have it in the x16 slot, so the card should be able to function as intended.

You do have it in the x16 slot correct? I checked your specs for the board and see you only have x16 x 1 and x4 x 2, so it’s only going to function in the single x16 slot.
For the other two x4 slots, you may be able to disable one to enable x8 on the remaining, but you’d need to have the board in hand to see if it’s even wired electrically for x8, or simply an x8 slot wired to x4 since that is it’s designed spec.

Have you removed both SSD and tested them one at a time in this device to make sure they are both functioning? And, have you tested each slot of this device one at a time with a single drive to be sure both slots are functioning?
If not, please test both of those ways now, so you can be sure both SSD’s are working and both slots are functional, because if you have it in the x16 slot it should be working.

Friend is ‘relatively’ close to the motherboard, and if really necessary he will go to it…I will check what I need if it comes to that, hopefully not…

It is…this is what we done and checked:
AOC-SLG3-2M2 card is in the x16 slot, Slot6 on the motherboard. In the AOC-SLG3-2M2 card there are two x4 SSD’s.
Both SSD’s are working correctly on it’s own, no problem there.
We didn’t check x4 slots as we know that they can support just one SSD, and that is not what we want.

Is there any way that AOC-SLG3-2M2 could use both SSD’s, how to make motherboard to use x16 Slot6 as x8x8 (or x4x4x4x4) ?

Really thank you for your time regarding this!

Slot6 can be forced to x8 instead of x16 at the BIOS settings level, but that should not be necessary. Maybe “Power down unused lanes” is causing this issue?
You said you tested both SSD and are OK, was that testing done in this adapter card? If yes, has each slot on the card been tested on by one to ensure both SSD slots on the card itself are OK?

Do you know if this AOC-SLG3-2M2 card has Legacy or EFI/UEFI BIOS? Current setting for that port is Legacy only.

Also, I see this setting is disabled by default > PCIe function swap
Explained here, do you think it should be enabled to help?
https://www.diodes.com/design/support/te…press-switches/

Thanks for the quick reply…I will check all to points you mentioned, and get back with results and answers.

Ok, yes, please try to research and find as much info as you can. If nothing else, we can try to enable all these settings to be visible for you, and then you can play around until you find a good combo.
That’s much harder if you are not there in person though, so might be a tough week or two playing and waiting for someone to come and reset/recover

Hi, friend checked both slots on card with each of the NVMEs, both slots and NVMEs work…

In BIOS there are currently very little options, can we try to enable it all…? I will play then with them, and hope for the best…

Here is the Advanced >> PCIe/PCI/PnP Configuration in BIOS:

Annotation 2018-12-03 201028.jpg

Good you checked both slots and both SSD and all is OK. Yes, we can try to enable all in BIOS. Can you currently see Chipset main tab? Or just advanced? Please show me advanced main entry, so I can see all it’s contained sub-sections at a glance.
And show main or any other tab, at root level, so I can see all current visible tabs/sections

Here are few tabs:

main.jpg

advanced.jpg

chipset-advanced.jpg

chipset-advanced-pchioconfig.jpg

pcieconfig.jpg



As you can see, there is not Slot6 shown in Advanced / Chipset / PCH-IO / PCIe Config, which is strange…

Thanks!

Thanks for images, it’s not strange settings are missing, manufacturers hide BIOS settings and entire sections/subsections all the time.
I am surprised you have chipset already visible, that is good, we can unlock it all now without trial and error trying to make chipset visible for you.
It’s also great this is moved inside Advanced section, if I would have noticed that before I would have already known we could make visible for you since Advanced is almost always visible already so anything within can be enabled
For the specific section you mention, there is not an entry for Slot6, only 4 & 7 at x8, but I can enable root port entry/settings subsections there for you for all ports 1-24 (4=1 and 7=17)
Slot6 main controls are at Chipset >> System Agent >> PEG Port Config, but there are several other sections with settings applied to it as well. Do you see it there?

This is a huge BIOS, so give me some time and I will enable it all for you now! Maybe can do in one night, but if not I will have for you in next few days.

Yes please, enable everything you can, and I will be very careful what I change…

I will check PEG Port Config tonight.

And please, no need to hurry, when you have time…thanks for the help!

Well I can enable it all for the most part, but thing is I don’t want to enable something already enabled (May cause issues) and I can only see some of your BIOS, so I don’t know what is already visible and what is not.
Unless you want to take a screenshot of each and ever BIOS page, and zip that up for me? Otherwise, I will for now only enable certain things relating to what we’ve discussed that I think may be missing from your view.
Thank you for being patient and OK on waiting a little bit as I do this!

Ok, no problem…I will made print screens of all BIOS tomorrow and send it in a zip file…thank you so much!

I have doubts that you can get it working. LGA 1151 based Processors can bifurcate to either 8x/8x or 8x/4x/4x, NOT 4x/4x/4x/4x like the HEDT/Xeon E5 platform can, which is what you need. Even if you could enable in-slot bifurcation, your card is only 8 lanes wide, so as a 4x/4x card, it would make contact with the contiguous 8 lanes thus only see one PCI Device. I suppose that it may be possible if you get a card like the AsRock one, bifurcate to 8x/4x/4x in the same slot, then use NVMe slots 1, 3 and 4, but I don’t recall if I have seen such setup working, just that on paper it seems possible. Your other alternative is a card with a PCIe Switch and no bifurcation.

@zir_blazer - sounds like you have looked into this correct? If yes, did you also look at changing the related settings in Intel ME setup, in conjunction with BIOS settings?
If you are not sure what I mean, I posted some example images of this in another thread for a different board here
Utility to view some info on PCI devices

So his card wont work even at x4/x4 correct?


I have zero experience with BIOS modding, so I can’t say anything about the screens that you posted. What I do know is that the Data Sheet of Intel LGA 1151 Processors typically explicitly mentions that the Processor PCIe Controller can work in either 16x, 8x/8x, or 8x/4x/4x modes. Check this Data Sheet, Page 27 / Table 2-11. Basically, the guy has a 8x PCIe card that requires an in-slot 4x/4x arrangement to be able to use its second M.2 Slot as it is intended to. LGA 1151 bifurcates as 8x/4x/4x, which means that in a 16x slot you will always be getting 8 contiguous lanes and the 4x/4x leftovers that aren’t physically connected to the card, thus the second NVMe drive will never work. In the AsRock card with 4 M.2 slots, the card is expecting a 4x/4x/4x/4x arrangement (Intel HEDT/Xeon E5 platforms can bifurcate like that), which means that in theory, in his current platform it should be able to provide 3 workings drives (The second M.2 slot should still not work).
However, now that I look at the Data Sheet closely, it mentions that it has a “reverse” mode, which makes me think that setting it as 4x/4x/8x may be possible. However, I have absolutely no idea about the implications of setting the PCIe Controller in that way, as what are Lanes 0 - 1 - 2 - 3 would become 3 - 2 - 1 - 0. I have seen some Block Diagrams from Supermicro Motherboards that mentions that a specific slot uses reverse mode, but I suppose that it affects the way that the PCIe Slot is wired to the Processor in the Motherboard itself. Maybe if it is possible if you can also tell the PCIe card / NVMe SSDs that they should work in reverse mode, too?

Thanks, I know very little about PCIE lanes, so we’re both at a loss one way or the other Thanks for explaining to me that way. Yes, Lane Reversal is one of the settings you can change in Intel ME setup, sounds similar to what you mention.
However, most you can manually choose the opposite anyway, so reverse not always needed anyway (example 4x1 or 1x4) I initially assumed any 16x or 8x slot would be enough to run the 8x card (at 4x/4x) even without a BIOS edit, but seems it’s not that simple
If you open an Intel ME file in Flash Image Tool (FITc) and look at these ME Flex I/O settings for lane width and the available options it may fall more in line with what you’re reading, or maybe not?

Here is this boards actual settings, the options box in each area is a cutout I placed there, those shown are available for each of the sections they are placed near

ME-PCIE.png



I think setting x4 at top and 2x2 on bottom = 8x/8x, but that may disable some other devices too. Or x2 at top and 2x2 on bottom = 4x/4x at a single slot.
But, I’m not certain and that is why I mentioned right up front I’d rather him have a programmer for recovery in case needed because I know these also control HDD and display too and if recovery is needed it may only be possible with programmer in some instances.
But in the end it may not be possible due to no actual bifurcation setting in the BIOS currently, I was hoping maybe some of the other hidden BIOS settings may help him get it working at one of the current slots.
After all this discussion, and what you mentioned, it makes me think even if we did change lane widths etc, BIOS would still need bifurcation specific setting, correct?
There is a thread here someone added it to their BIOS, maybe I can find and do that here. Ahh, found it - [Guide] - How to Bifurcate a PCI-E slot
I may have to ask him for help on this, or if it’s even possible in this BIOS.

@davidm71 - using your guide, or even enabling bifurcation is maybe not possible on this board correct, I do not see IntelRCSetup in BIOS, thanks for your time if you look for us!
https://www.supermicro.com/about/policie…wareItemID=6755

The screenshot that you posted is for the Chipset PCIe Lanes, these should be completely unrelated as the one that you want to modify are the Processor lanes. Check Page 28 / Table 3 - 1 of this Data Sheet for basic info about the Chipset PCIe Lanes and Intel Flex IO. Basically, the Chipset PCIe Controller is partitioned as 5 Controllers with 4 Lanes each, and each of those controllers can be configured as 1 Slot with 4 Lanes, 2 Slots with 2 Lanes each, 1 Slot with 2 Lanes + 2 Slots with 1 Lane each, or 4 Slots with 1 Lane each. That is exactly what is on your last screenshot.


It seems that it is possible to enable 8x/4x/4x bifurcation for the Processor with the current things that you already have. If you check the screenshots that you posted here, notice that you have PEG 0:1:1 and PEG 0:1:2. These corresponds to PCI Addresses that are conceptually identical to this:

00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 05)
00:01.1 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x8) [8086:1905] (rev 05)
00:01.2 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x4) [8086:1909] (rev 05)

On a typical Motherboard that has three PCIe Slots wired to the Processor PCIe Controller, like for example, my Supermicro X10SAT, the bifurcation is automatically done if you populate the other Processor PCIe Slots (That happens because an independent PCI Slot has some extra Pins for card detection which I suppose that the Firmware can check, but that is pretty much worthless for in-slot bifurcation with logically separated PCI Devices like some PCIe-to-M.2 adapter cards. These have to be forced, can’t be autodetected).
Basically, if I have either one or no card inserted into the first 16x slot (16x mode), I have only this:

00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 05)

If I put a card in the third 16x slot, EVEN if the other two slots are not populated, the Firmware puts the Processor PCIe Controller into 8x/4x/4x mode and I get the three PCI Bridges (00:01.0, 00:01.1 and 00:01.2). With a single card in the middle slot I get two PCI Bridges (00:01.0 and 00:01.1).


What you want to do is that instead of “auto”, PEG 0:1:1 and PEG 0:1:2 are forced enabled. In Linux a simple lspci will tell you if bifurcation is working. However, as I stated before, the M.2 adapter card that the guy that started this Thread has will never work, unless a miracle happens and you figure out how to use reverse mode so that the slots provides 4x/4x/8x, so that it will match the physical needs of the card.