Scalable xeon ES won't run on new Supermicro X11 boards after BIOS update- why?

Supermicro dual socket boards worked fine for 60652h engineering samples in the past
on X11/C621 series boards. I ran dual QLQ9 (6149 16c)
and dual QLKN(8175 24c) using BIOS versions 2.0b and 2.1 on an X11DPG-QT board.

I received another X11DPG-QT but with BIOS 3.1 in December 2019 and it would not run any 60652h
samples past Q code 67. I found that a cheap production Xeon Silver 4109 ran fine on the same board.
so using the 4109 I reflashed with BIOS 2.1 and the ES worked perfectly!

Is anyone else using X11 boards seeing the same problem with Supermicro X11 BIOS 3.1 and 3.2?
Supermicro site has only 3.2 available but I have a copy of the earlier ones that worked - 2.0b and 2.1
I can provide link for BIOS 2.1 from tinyurl when their site is back up but they’re to big
to attach here. Comparing the BIOSes that work and don’t work might clarify what is being modified
in new BIOSes to prevent ES from running, and mods needed to make them work.

Hey Maxdata,

I’ve got a similar issue with a X11DDW-NT trying to run a Xeon Processor Gold 6142 ES QL28.

If you happened to get anywhere with this, any updates/sharing the old BIOS files would be very helpful!

Thanks!

@maxdata - Sounds familiar, like maybe we discussed in PM? Did we already test inserting older microcode back into the newer BIOS?
If not, I think I did this for another Supermicro user recently and problem solved.

@cmoski - Did you test older microcode, starting specifically with one you know that works in some previous BIOS?

@Lost_N_BIOS Thanks for such a fast reply! Really appreciate any help you could give. (And happy to donate as well!)

So I don’t have any direct evidence that this particular board was ever able to run this chip beyond a few eBay auctions mentioning it. Willing to extract this myself, how would I go about merging the microcode from 2.0b into 3.1?

Based on @maxdata 's suggestion, I found the older super micro bios version that supposedly were able to run it –

https://www.supermicro.com/Bios/softfile…X11DDW8_307.zip – Release 2.0b

The current BIOS does not:
https://www.supermicro.com/Bios/softfile…7119_SUM240.zip – Release 3.1

The current release (that doesn’t work) hangs on boot. I can’t downgrade to release 2.0b even enabling manuf mode on the jumpers; says that it cannot downgrade. However it seems to happily cross-grade to different versions of 3.1.

@cmoski - You’re welcome! Microcode edit can be tricky, sometimes you have to do 2-3 ways until you find which does not break BIOS by adding/removing padding in the volume you touch, or you have to only use one tool and not others if microcode in certain type of BIOS volume
And then for some you have to fix FIT Table after all is done. I’d have to do the mod first before I could tell you what all is needed on this BIOS, and what tools to use etc

From what I see many users mention with Supermicro BIOS often is, old BIOS worked with this ES or this Xeon etc, but new BIOS does not.
Sometimes all you have to do is swap back in the older microcode, but sometimes this does not help at all, due to something else needs swapped in, or they’ve actively blocked it somehow etc.
If it’s the later, I can’t help myself, but some other user here may be able to, if they wanted to put time into doing

Man, I hate how they name BIOS files/packages, it’ never matches the version and often the names don’t even match the model names!! Crazy!!
It doesn’t even look like the above two linked BIOS are for the same motherboard, due to this mess they always create!
SO dumb!! New v 3.1 BIOS has 211 extension (not matching BIOS major version #, or actual BIOS version # at all), OLD 2.0b BIOS has 307 (not matching Major BIOS version #, or actual BIOS version # at all)
You’d think, by general default, old v2 BIOS would be 2xx and new v3 BIOS would be 3xx

Wow, why is one package 8MB and the other 60MB?? Please wait, once it’s down downloading I will make you new BIOS with old microcode so you can test.
What is the CPUID of your ES, do you know? I see 50652 and microcode revision 80000037 being used on the ebay sale images, at least for one seller (penguinestore)

If it does not work, then you may need flash programmer to downgrade to 2.x BIOS. Or we can maybe get you there with FPT, if ME FW is not too different between 2 and 3

Yeah, one package has a ton of extra bloat in it! Had to dig through the wayback machine to find the old BIOS file download numbers, at least they don’t delete anything!

@Lost_N_BIOS The microcode is 050654 according to https://www.cpu-world.com/sspec/QL/QL28.html as its a QL28; but haven’t been able to verify that as I have two processors and nothing to boot them :slight_smile:

A flash programmer is fine for the downgrade, just attempting to figure out if that is required yet. I may have one from a previous project depending on the pin pitch.

I attempted to use UBU to actually merge the microcode of 05654 into the 3.1 firmware; not sure how well that is going go. Testing it soon.

Thanks again!

@cmoski - See my edits above, in case you missed anything, not sure when you started reading but I edited it a few times (last edit was to add more rant )

If it was 50654 it would possibly be working, if microcode is the only issue here, because 50654 is in the new BIOS (but different version). In old BIOS 50654 = 02000043, new BIOS = 02000069
Usually ES is 50650-50653 for example on this series, I assume and think this is 50652, I think that info on cpu-world may be wrong due to what I see in google and that ebay image using 8000037 (This is 50652 microcode) (From this ebay listing)
But, in case this is 50654, and you need old microcode or it fails, I will make you two BIOS, one with the old microcode in there, and one with the newer one from 3.1 BIOS (Both with old 50650-50653, which are removed from the 3.1 BIOS)

Do not test UBU BIOS until someone checks it, especially if you do not have programmer in hand and confirmed OK Backup made (see my comments about tricky microcode edit above)
OK, making BIOS now
-----------------------

@cmoski - First we need to check something! Please check and see if Intel Boot Guard is enabled with key burned into PCH, if it is then we cannot modify microcode in this BIOS.
The microcodes are located in extreme RED zone, but even if in Cyan zone would still = brick if it’s active
Please download v4 SPS tool package from here - https://ufile.io/7ebi54nj http://s000.tinyupload.com/index.php?fil…624283929295352

Inside go to >> SPS 4 PU\SPSInfo\WIN64 << From there, run ADMIN CMD Prompt and this command >> spsInfoWin64.exe -verbose
Show me image of the bottom of the report (not copy/paste text) Make sure I can still see the names of columns, need to see FPF row, usually left most column
Here is registry fix (for Win10) if you do not know how to get Admin CMD Prompt, merge, reboot, then right click anywhere in folder and choose open CMD as Admin
https://www.sendspace.com/file/t3zno5

If you see Measured or Verified Boot enabled at left/FPF side, then we cannot modify this BIOS for microcodes.
Only option then will be going back to old BIOS instead, and if Boot Guard is active then only way to safely do that is with flash programmer
We may be able to do the BIOS downgrade with spsFPT, but in case brick programmer would need to be in hand and backup already made in advance

For this edit, MMTool 5.02 only should be used (for the microcode part), to fix FIT after that you can use UEFITool 25.0 (possibly others, I’m not checking) and hex editor

Thanks for the detective work on that @Lost_N_BIOS – will hold off on trying to modified firmware.

Will take a moment to install windows, checking on the boot guard status now.

@cmoski - You’re welcome! I have mod BIOS done already while waiting, and already typed out info/reply, so here that is below But yes, hold off on the flash until I see that SPSInfo report

Here is 2x mod BIOS - http://s000.tinyupload.com/index.php?fil…926414861123071 ** New Link **
I’d say go ahead and test new-50654 BIOS first, since I’m pretty sure this CPU is not 50654 so whatever version 50654 is does not matter. If that fails, then test old-50654 BIOS, this has the 50654 microcode version that’s in 2.0b BIOS.
You may need to rename file to stock name.extension, I didn’t do in advance because files get mixed up easily that way (if you rename, remember which folder you’re testing BIOS out of )

@Lost_N_BIOS

Attached is the spsinfo verbose output. I think from what you said it looks clean.

Thanks again!
cmoski

bootguard1.JPG

bootguard2.JPG

@cmoski - try again without verbose. I don’t see what we needed to check there at all!

Here ya go! Sorry for the delay @Lost_N_BIOS

bootguard3.JPG

bootguard4.JPG

@cmoski - Hmm, same/same, I guess we can’t check if boot guard enabled at PCH this way on SPS V4
I see your other thread, did you flash in the UBU mod BIOS, and it didn’t brick? If yes, then it’s OK, we don’t have to worry about this, and you can go ahead and test both BIOS I made you above.
If you did not flash in the UBU mod BIOS, then I would personally wait until you have programmer in hand, and backup made and checked to be OK by someone, before you flash in any test BIOS, in case Boot Guard bricks it due to violation by BIOS editing.

I actually already did try to flash the moded one I made and there was no change. I think we are OK here.

What the heck, going to go for it – will let you know how it goes. The worst thing would be to have to grab a programmer. :slight_smile:

That is a very good note about boot guard – I’m sure it enhances security but it sounds like a total pain for the second hand/home lab market.

OK, good to know. No change is OK, we just needed to know if mod BIOS bricked or not, when you edited microcode area, so if it did not brick then you’re OK to test mod BIOS I made above
Yes, if boot guard enabled at PCH this means a key is burned into fuse there, and only certain area of the BIOS can be edited, otherwise it’s a brick. Only way around that is to replace the PCH.
You can see what areas are covered by boot guard by dropping BIOS into UEFITool NE Alpha version 51 or above

Usually it’s mainly some OEM’s that enable this, HP and Dell are major offenders, but sometimes Lenovo does it too

Wow, it worked great first try when flashing 50654-69.bin. Both processors detected and Prime95 burning in now. You are a legend, thank you so much!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
 

Socket 1 ID = 0
Number of cores 16 (max 16)
Number of threads 32 (max 32)
Manufacturer GenuineIntel
Name Intel Core
Codename Skylake-X
Specification (Engineering Sample)
Package (platform ID) Socket 3647 LGA (0x0)
CPUID 6.5.2
Extended CPUID 6.55
Core Stepping B0/L0
Technology 14 nm
TDP Limit 165.0 Watts
Tjmax 103.0 °C
Core Speed 999.9 MHz
Multiplier x Bus Speed 10.0 x 100.0 MHz
Base frequency (cores) 100.0 MHz
Base frequency (ext.) 100.0 MHz
Stock frequency 2400 MHz
Max frequency 3200 MHz
Instructions sets MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T, VT-x, AES, AVX, AVX2, AVX512F, FMA3, TSX
Microcode Revision 0x80000037
L1 Data cache 16 x 32 KBytes, 8-way set associative, 64-byte line size
L1 Instruction cache 16 x 32 KBytes, 8-way set associative, 64-byte line size
L2 cache 16 x 1024 KBytes, 16-way set associative, 64-byte line size
L3 cache 22 MBytes, 11-way set associative, 64-byte line size
Max CPUID level 00000016h
Max CPUID ext. level 80000008h
Cache descriptor Level 1, D, 32 KB, 2 thread(s)
Cache descriptor Level 1, I, 32 KB, 2 thread(s)
Cache descriptor Level 2, U, 1 MB, 2 thread(s)
Cache descriptor Level 3, U, 22 MB, 32 thread(s)
FID/VID Control yes
 
 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 

SMBIOS Version 3.2
 
DMI BIOS
vendor American Megatrends Inc.
version 3.3
date 02/21/2020
ROM size 16384 KB
 
DMI System Information
manufacturer Supermicro
product SYS-1029P-WTRT
version 0123456789
UUID {00000000-0000-0000-0000-3CECEF1ACCCA}
SKU To be filled by O.E.M.
family To be filled by O.E.M.
 
 

stats.JPG

@cmoski - Awesome to hear it, very nice!
That means this CPU is not 50654, since the 69 microcode was already in that BIOS and didn’t work before. Ohh, we can tell in your info there (CPUID 6.5.2), this is 50652 as I suspected

Thank you too for the kind words, always happy to help when I can!

The link to the modified BIOS in post #9 has expired. Would someone repost, please? @cmoski @Lost_N_BIOS

@proximaldis10s - Thanks, fixed

Thanks. Can you guys point me to a tutorial for flashing the supermicro BIOS without cpu/ram present?