Broke server. Need help!

I flashed this one (03.04.0006):
Intel® Server Board S1200V3RPL/S1200V3RPO/S1200V3RPM Firmware Update Package for EFI
Literally reads

Important note: Do not use this package on the Intel® Server Board S1200V3RPS or Intel® Server System R1304RPSSFBN.

You’d think there’s a failsafe in place that prevents flashing the BIOS meant for a different system, but no, there isn’t. Anyway, I blame myself really. :slight_smile:

Before that, it had “03.01.0002” (S1200RPS_EFI_BIOS03010002_BMC113_FRUSDR108.zip). It seems Intel pulled all BIOS update downloads for this board, probably a contributing factor in why I flashed the wrong BIOS. I cannot find it on their website but I can provide the files if this helps.

Btw, thanks so much for helping me @lfb6 !!

edit: I will remove all the stickers and take more pics but it’ll be when I have access to the server.

edit2: To be clear: I only read “Do not use this package on the Intel® Server Board S1200V3RPS” after I had already done it.

edit3: This is unrelated but, I can see now, by looking at the server’s manual that the fan was installed incorrectly so that it blocks air flow. :scream:

Wasn’t thinking or guessing at all, tried simply to understand what happened in the firmware.

For example:
The bios region has 5 of 9 efi volumes identical, but volume for PEI and the volume with the DXE drivers are significantly different.

The ME region is present in 2 differently configured forms in the ‘wrong’ bios (MEComplete_03.00.07.024SE.cap and MEComplete_03.00.07.024.cap) and in one form in the correct bios (MEComplete_03.00.07.024SE.cap). Interestingly enough the MEComplete_03.00.07.024SE.cap files have identical ME regions. Wasn’t able to determine when which version get’s flashed from the scripts.
Did you use one of the scripts- startupse.nsh or startup.nsh? If yes: Which of the two scripts?

It’ll be interesting what your dumps will show. You might try to dump the regions with fpt from ME 9 ME- tools package, owned a Asus C222 board where parts could be dumped (but the tool worked just once after a reboot).

1 Like

I used startup.nsh. What is startupse.nsh? What is the difference between
MEComplete…SE.cap and MEComplete…cap? I cannot find an explanation for this.

Different ME configuration, FITc for (Denlow-) SPS is hard to find, so difficult to check. Only difference for the 2 mentioned scripts is flashing different ME.

1 Like

Just wanted to post a quick update: Thanks for the tip to use CSME tools version 9! This worked (Intel ME System Tools v9.5 r6)! I had previously tried different versions and none of them worked.

Remotely, (as I do not have physical access right now), I have been able to get this from the working server:

.\fptw64.exe -I

Intel (R) Flash Programming Tool. Version:  9.5.40.1868
Copyright (c) 2007 - 2014, Intel Corporation. All rights reserved.

Platform: Intel(R) Server Essential
Reading HSFSTS register... Flash Descriptor: Valid

    --- Flash Devices Found ---
    W25Q128BV    ID:0xEF4018    Size: 16384KB (131072Kb)

    --- Flash Image Information --
    Signature: VALID
    Number of Flash Components: 1
        Component 1 - 16384KB (131072Kb)
    Regions:
        Descriptor - Base: 0x000000, Limit: 0x00FFFF
        BIOS       - Base: 0x400000, Limit: 0xFFFFFF
        ME         - Base: 0x030000, Limit: 0x3FFFFF
        GbE        - Base: 0x010000, Limit: 0x01FFFF
        PDR        - Base: 0x020000, Limit: 0x02FFFF
    Master Region Access:
        CPU/BIOS - ID: 0x0000, Read: 0x1B, Write: 0x0B
        ME       - ID: 0x0000, Read: 0x05, Write: 0x04
        GbE      - ID: 0x0118, Read: 0x09, Write: 0x08
 Based on the Host Region FRACC the Host/CPU/BIOS has ( 0x00004B5B ) :
             Read    Write
    Desc  :  Yes     Yes
    Host  :  Yes     Yes
    ME    :   No      No
    GbE   :  Yes     Yes
    PDR   :  Yes      No

Total Accessable SPI Memory: 16384KB, Total Installed SPI Memory : 16384KB

FPT Operation Passed

Further, as you can imagine from the above output, I was unable get a complete dump because read access to the ME region is blocked. I have been able to dump the other regions individually:

BIOS.rom
DESC.rom
GBE.rom
PDR.rom

edit: no idea how to dump BMC ROM but i don’t think that matters. Only the BIOS/ME seems to matter.

dumps.zip (4.4 MB)

    --- Flash Devices Found ---
    W25Q128BV    ID:0xEF4018    Size: 16384KB (131072Kb)

Is the above info enough to tell us what programmer I need to get?

Documentation - Winbond

Now many SPI apps have the W25Q128BV on their database, some apps will work better others fail, all depending on the motherboard, so get on it.

[GUIDE] The Beginners Guide to Using a CH341A SPI Programmer/Flasher (With Pictures!) - BIOS/UEFI Modding / BIOS Modding Guides and Problems - Win-Raid Forum (level1techs.com)

[GUIDE] Flash BIOS with CH341A programmer - BIOS/UEFI Modding / BIOS Modding Guides and Problems - Win-Raid Forum (level1techs.com)

[Guide] Using CH341A-based programmer to flash SPI EEPROM - BIOS/UEFI Modding / BIOS Modding Guides and Problems - Win-Raid Forum (level1techs.com)

Depending on the board, CMOS battery off (some ON), PSU standby off (some ON).

1 Like

A simple CH341 programmer should do, there might of course be difficulties with this specific board, but for this chip it’s completely OK:

https://www.amazon.com/ch341a-programmer/s?k=ch341a+programmer

Since the ME region can be replaced from the update file this should give us a complete bios image.

So far nothing seems to have happened to the bios region, no stuff from the ‘wrong’ update found here…

Volumes in bios region (backup bios does mean that there two sets of bios volumes in this region):

1 = = NVRAM
2 = = empty
3 = 11 = 9 in update (both versions)
4 = = 8 in update (OK version)
5 = = 7 in update (both versions)
6 = = empty
7 = = almost empty
8 = = 6 in update (both versions)
9 = = almost empty
10 = 12 = 4 in update (OK vers)
11= 3 = 9 in update (both versions)
12 = 10 = 4 in update (OK version)
13 = 15 = 3 in update (OK version)
14 = 16 = 2 in update (OK version, 2 minor diffs)
15 = 13 = 3 in update (OK version)
16 = 14 = 2 in update (OK version, 2 minor diffs)


The differences in volume 14 = 16 in bios region to volume 2 in update:

There’s one glitch in NVRAM, the space between last entry and GUID store should be empty, but there’s a non referenced GUID there.

So firste step would be to just replace ME and try. If- against all odds- your dump shows the correct ME for your system, it’ll get a little harder to explain what might’ve happened…

That would be the file, it’s just composed with the parts you dumped pluss ME region from update:
S1200.zip (5.3 MB)

1 Like

Thank you! I ordered the programmer and I should have it in a couple of days. I will see if can make a dump of the flash ROM.

If that works, should I try to flash the ROM image you made? What about MAC addresses? If I write only the ME region, will that overwrite the MAC addresses?

Please do a dump with the probgrammer first and compare it to the parts you dumped with fpt since the this fpt version is originally made for another ME platform. Compare the resulting file in a hex editor with the partial dumps you made, they’re just copied after each one- desc, gbe, pdr, me, bios region.

Mac and other machine specific data- there’s no mac in gbe, just a dummy, but server boards do have the macs often stored in NIC firmware. Otherwise there’s no machine specific data in ME region and the rest of the firmware would be as much ‘original’ as possible.

1 Like

I have been told that the programmer has been delivered and I am making my way to the server now. I will try to make a dump first.
The thing I don’t fully understand yet is: Should the server be unplugged, or on AC but powered off? Or powered on?

Don’t try to write if you can’t/couldn’t read!!

Try without power

I received these parts in the mail.

For the most part, I can’t seem to make a connection between the part that connects to the SPI chip and the chip itself. Mostly, the clip that physically connects to the chip does not stay on and if it does, no connection is made. It looks like, I need to look at more guides.

I have probably tried 50 or 100 times and I was only able to make a connection once per chip.

Here is my setup. I removed the board from the chassis and took it home with me.

The chip that was labelled “1.03” (the chip that is close to the CMOS battery) is either the BMC (no BIOS, ME region?) or I did not get a valid read at all. UEFI Tool reports that there is no ‘top file volume’ in the dump.

The other chip , labelled “101S” seems to be the BIOS/ME chip. When I had a connection to that chip, I tried to make a dump, then tried to flash ME region (using S1200.zip from above), and then tried to read again.

commands I used
to dump:

sudo flashrom --programmer ch341a_spi --read rom.bin

to write ME region:

sudo flashrom --programmer ch341a_spi --ifd --include me --write S1200.bin

What I noticed:

  • Flashing ME region didn’t seem to fix the issues. Though, I believe I could see in the before/after dumps that ME version changed from x.x.x.18 to x.x.x.24.
  • The issue with ME version showing “Read error” in conjunction with only 16 GB of 32 GB RAM being used seems to have fixed itself (before I did anything with the programmer). Perhaps this happens only when the ‘flash ME’ jumper is set?
  • When I did get a connection, I had AC plugged in and flash ME jumper set to enabled. When the power cable was unplugged, I did not get a connection. Then again, I only got a connection once, so who knows.
  • I think I made the mistake of forgetting to remove the battery. I will try again without it.

I will try again today.

dumps.zip

Yes, that puts ME in recovery mode and it doesn’t function as expected.

If your ME was 3.0.7.18 now before flashing then it never got updated when you tried with the ‘wrong’ update. You possibly didn’t do anything to your ME firmware and bios and trying the update was thought to be the culprit even if it did happen long time ago.

So all this boils down to your BMC not working, the rest seems to be OK? Fans spinning might as well be caused by BMC not controlling them. The rest of symptoms seems related to having had the ME jumper set.

The dumps look otherwise fine and are valid- the bios region is identical except for NVRAM, but that’s normal. And yes, positioning the clamp can be a pain but after some time one gets used to the proper way…

No idea regarding BMC firmware- and it’s not at all clear that this is even BMC firmware related, maybe it just died.
How did you manage to downgrade(?) the BMC firmware without BMC working?

1 Like

The BMC firmware never gave me any issues when trying to flash it. That’s why I didn’t suspect it as the problem but I believe you are correct. Perhaps ME is not the issue. After all, BMC is not working (no ping, BMC website offline).

I can update the BMC firmware (or write the same version again) using Intel’s provided tools (startup.nsh, etc.) and I can downgrade it that way too, if the force BMC update jumper is set.

It seems to me that both ME and BMC have the potential to continue to run and function even if a wrong version is flashed until such a time that the system is completely without power.

Yes. possibly it is not related to flashing firmware. I don’t know at this point.

My SOP8 clip just broke :frowning:

Im not surprise at all with that… happened to me long time ago when i did the same and bought a cheap “big pack with possible accessories offer” like yours… money speaks here and its not a product for mean or long term uses.
The market “offers” what it is plain and simple… users tend to ignore and look to their “pockets”… normal, the “Asian” market knows this… The device itself is “reliable”… never had a failure so far but the connection itself, is as you already noticed.

1 Like

Meanwhile, I got another programmer kit and this time, the clip is working well. I practiced a little bit today with a mainboard I have at home and now I can make the connection work usually the second time I try.

But the second kit has the connector clip pins completely crocked.

It seems to me the quality of the flash programmer is good in these kits but the cables and clips are of bad quality. I ended up creating one working kit out of two purchased kits.

They are not, either. The “black PCB version” you bought have a fatal flaw never addressed for years, or decades. Search for “ch341a 3.3v fix”. This programmer can’t change the voltage of the data lines. When programming 3.3v chips, it can supply 3.3v on power line, but 5V on data line.

It’s probably okay to use a few times, I never heard any BIOS chip fryed because of this. Pushing over the spec limit is not a good idea.

There are “green PCB version” which fixes the issue by adding a voltage switch jumper. However there quickly occurred “black PCB version” reprinted in green…

The general suggestion is measure the voltage yourself and fix if need.

Or… not buying a SPI programmer! I bought a nice “USB to parallel port adapter”, with blue PCB, and there are jumpers to set voltage and SPI mode. The only hurdle is that you need to wire up the sop clip, because the pinout is way different from a bios chip.

It also doesn’t have the programming socket, which is useful if you are not using sop clips.

They are based on the same IC, CH341a. CH341a isn’t just a programmer IC. It supports all kinds of protocols including parallel port and SPI.

1 Like

Not so fatal… https://youtu.be/J8-Sh7DjiXw?t=204

1 Like

He’s point is that there’s not much amps on data lines. Well, think again, these chips are logical chip, they are not designed to sink much amps either.

From 25Q128FVSQ datasheet


I didn’t find the max output current. but check the conditions when output voltage was measured, only 100uA. That’s the current the chip is supposed to work. Input is usually in high impedance mode, so no current is expected at all. If the high voltage causes some insulation breakdown, then a mA will be way over spec and can do some real damage.

“How can you fry something with just a milliamp?” You sure can. He’s basically assuming anything “burned out” must be overheating and molten inside. Not really, there can be an electrical breakdown with no apparent sign. Think about how static electricity damages a chip. It could kill the I/O driver circuit of the data pin, and nothing else.

The chips nowadays are made at such minuscule scale, that they can only process a few uA. If you pump mA into it, it can break. Unlikely, but possible.

It’s fine to use a few times, so many people used it, no repored incidents. (Or at least nobody reports because the motherboard wasn’t really working to begin with) I’d rather not taking the risk.

After all, the parallel port adapter is much cheaper than the programmer. If the cheaper one can recognize and bother to fix the issue, why not the more expensive one? A jumper doesn’t even cost a cent. And there is 3.3v rail on board, already. The only thing needed is to switch the input voltage of ch341a.

This is not workmanship, and by no way “good quality”.

1 Like

I did read about that but I didn’t apply the fix because I read that it doesn’t really matter. Also, I wasn’t sure if still applies to newly sold programmers. I will try to measure and apply the fix before I next use it.

Thoughts on the video:

  • It is reassuring that it takes him 4-5 times to place the clip because I have never gotten it right the first time.
  • Since placing the clip is hard, it doesn’t really matter that Vcc is 3.2V and data lines are 4.8V because you will inevitably attach Vcc to a data line and attach the data line to Vcc just because placing the clip is hard and it can’t be done right the first time every time. However, the currents being low is reassuring.
  • As to the quality of the programmer, the 3.3V vs 5V issue aside, imo a 3.3V line should be 3.3 - 3.4V and not 3.2V and 5V line should be 5V-5.1V and not 4.8V, so that’s bad too.