Z200 switches off after 30 minutes - unable to Ctrl-P or flash ME firmware

Hi all

I’ve an HP Z200 (full size not SFF) that turns off after exactly 30 minutes. I believe this is a function of ME when ‘somethings wrong’ with the ME. My aim is to make the motherboard useful again by some means and I’m hoping someone here can assist me.

I’ve a post on HP forums that has directed me here. There are lots of screen shots of my BIOS and the output of attempted firmware/driver installations. A second Z200 owner is reporting the same issue that I have.

The elephant in the room to me is that Ctrl-P does not bring up the AMT config screens. Note this motherboard (I actually have 2x motherboards and both do the same thing) is an eBay purchase and I have no history until I eventually twigged the shut offs where repeatable at 30 minutes after switch on. It was too late to return the motherboard by the time I had gotten around to building the hardware and installing Windows to it.

I have access to a Raspberry Pi and a Pomona clip on reader and I’ve had a look at me_cleaner. me_cleaner read the BIOS but cleaning it and reapplying it has not managed to disable the watch dog timer.

My HP forum link showing my progress to date is here

I appreciate any guidance/reading materials or direct help. Please note that this is the first time I’ve tried to do this and my terminology and understanding is evolving (please be gentle)

Many thanks

@lwoody7110 - ME Cleaner is not for this, and should not be used for anything related to fixing ME either (Mainly for everyone reading at any time in the future)
Since you have programmer this is an easy fix. Please upload entire BIOS dump, if there is two BIOS roms (usually is), dump both and upload

Reading your thread at HP - Update BIOS from DOS if you can, that may allow the flasher to attempt to fix the ME and MEI Drivers are not needed at that level. reading on I see this is suggested already too (Heading to page two now )
Manufacturing mode, leave that alone until we are done. It can be set, taken out of manufacturing mode once we are done.

I never saw, or missed your reply, on that HP thread, did you check board over for FD/FDO/ME/Service jumper? Any two or three pin jumpers, please post an image of here, or tell me what labels your see around/near them.

Since you can see ME Version in BIOS, it may be OK, lets try to reset ME State before doing anything else.
Remove CMOS Battery, unplug PSU from board (24 and 4/8Pin), press and hold the power on switch for 10-15 seconds, then let system sit for 1+ minute without power.
Put CMOS battery back, plug in PSU, enter BIOS, load defaults and try again to enter MEBx via Control + P at startup.
On this MEBx Control + P, have you ever been in there, with current keyboard? Sometimes some USB KB may have issue, especially wireless, so use PS/2 one if you have, or USB one you know has previously let you in MEBx
Once reset, default password should be >> admin



Hi, I regularly deploy HP workstations & could possibly give some pointers…

Do you know about HP Client Management tools such as SoftPaq?


Edit- Haven’t read your forum link yet however, have you checked Event Log to see why the machine is shutting down?

Thanks - yes. I’m familiar with softpaqs. I’ve downloaded all of them from the HP support page for this 'board. Screen shots on the HP link show attempt to install and errors received in both Windows and DOS.

Eventvwr shows the shutdowns as unexpected. I’ve replaced everything prior to twigging the power off (not a shutdown) happened at 30 minutes regardless of whether the PC was in Windows or the BIOS screens.

If you want to, upload the log folder (C:\Windows\System32\winevt) and I’ll take a look when I get a minute.

btw is Windows activated?

Hi - and thanks for your help.

me_cleaner:
Ahh ok. The internet and some me-cleaner forums suggested it could disable the 30 min watch dog, which it didn’t in my case. It seems to have worked for others. I’ll upload the BIOS dump once I figure out where me_cleaner placed them. I think the BIOS dump became hidden away in the root account on the pi somewhere. I’ll figure that out in the morning and post.

Update ME BIOS from DOS
There is a screen shot on the HP forum that shows my attempt to update the ME firmware with a very unexpected failure regarding needing to find the MEI driver was in DOS! This just looks so wrong! It actually looks like it needed to run in DOS under Windows rather than boot into DOS using a USB stick. I’ll revisit this tomorrow as well. However, this will fail as I can’t install any ME related drivers. This seems to be a circular issue. Can’t install drivers as ME not detected. Can’t install ME BIOS as drivers not installed.


re: Service jumper.
There isn’t a documented service jumper for ME. The motherboard has a CMOS reset button, and a password clear jumper. Annoyingly there are a number of undocumented pins on the board as well. The yellow button on the right edge is the BIOS reset button. The 2x 2 pin above are undocumented in the workstation’s manual. These are e19 and J3. The green tagged jumper is to reset the BIOS password. The 2 pin to the left is marked E1 and also undocumented.

IMG_2366.jpg



RE: resetting ME state:
I’ll repeat it tomorrow so you know its been done. However, I effectively did this during my attempts with me_cleaner. I removed the battery during the flash operations. The motherboard is powered via the Pi during the me_cleaner read/write. The battery and PSU where reconnected during the subsequent reassembly and testing.

As a note, I removed everything from the motherboard (all RAM, add-on cards, connections to USB and audio) except the CPU. I assumed this would be ok during the me_cleaner work.

Note that as it stands, I’ve ‘me_cleaned’ the current BIOS of all the ME related regions me_cleaner found in the hope it would sort the 30 minute issue. The MEBx prompt is not available at all right now which gave me a tiny bit of hope when I me_cleaner’d this the other day. When I fire up the Pi tomorrow, I’ll reapply the ‘before’ BIOS which should put the 'board back to where it was before me_cleaner did its thing.

I’ve never been able to get into MEBx with Ctrl-P. I’ve tried 2 USB keyboards but I’ll double check this again tomorrow. I have a ps/2 keyboard that I haven’t tried on this motherboard. I’ll use that one tomorrow.

Thanks again. Hopefully this can be rescued.

Thanks. Once I’ve reinstalled the ‘original’ BIOS prior to my me-cleaner attempt, I’ll wipe eventvwr and then leave it for 30 minutes for the next shutoff. I’ll then upload them here.

It had a clean install of Win7 several times until I twigged the shutoffs where at exactly 30 minutes. I can’t remember the state of activation right now. I’ll know when I fire it up tomorrow.

Why are you trying to flash or use MEI? In case you don’t know, the purpose of MEI is to allow remote access & control of client machines at a hardware level (along with AMT)…
It’s generally so we (as service providers) can change workstation configurations without needing to be onsite…

Basically you (as an owner) don’t need it.


Edit: Actually that reminds me…I think ME is supposed shut down a machine after 30 minutes if the firmware has been messed with.

I know I don’t need it but something is borked with it and my investigations suggest that ME can shut off a system after 30 minutes should ME think its been tampered with (which I definitely have now!)

As Ctrl-P doesn’t work, and HP’s softpaqs can’t install firmware updates to ME, and the device drivers won’t install to ‘absent hardware’, I believe ME thinks it has been compromised and its 30 minute watch dog is forcibly shutting off the PC. I hope to restore the borked ME firmware so that HP’s softpaqs and device drivers can be detected within Windows with the outcome that the 30 minute watch dog won’t turn the PC off.

Assuming this outcome is reached, I should be able to get into the AMT config screens via Ctrl-P and I assume turn it off again!

Ah you ninja’d me :slight_smile: See my edit above…


Softpaq won’t, but there are other tools. I’ll see if I can find a link.

@lwoody7110 - ME Cleaner did nothing like I suggested, it corrupts and disables ME, which I forgot you were stuck with this type of ME now, so resetting ME State is pointless. You’re only hope now, due to using ME Cleaner, is to reflash the ME with a proper and configured ME either from your ME backup or a full previous dump.
Or I can find dump from same system, and fix for you. But, you have to make it possible to flash ME again, how did you do it before, usually a jumper is needed, or unlock FD via programmer. Do you have unlocked FD now, if not, how did you write in the disabled ME cleaner ME FW?

ME is corrupted/disabled, due to ME Cleaner, that is it’s intended job. The only way to fix this, is to first get ME back to state before you ran ME Cleaner, which is a corrupted ME state that has not been "ME Cleaner’d"
Intel ME FW Update tool, which is what softpaq/HP update tools would use, cannot be ran on a corrupted ME (Before or after ME cleaner state).
Anything dealing with addressing ME cannot be done now, you disabled it surgically, so no point in trying to mess with it, only re-write it via not the usual methods. This includes trying to get into MEBx, you can’t don’t try, it’s not there, you disabled it
We can fix the ME either way, using Intel FPT, but you have to be able to write to the ME region of the BIOS. So, how did you write the ME before after using ME cleaner?

Undocumented jumpers, you can find out if one is ME or service, by putting a jumper on one and then run for 30 minutes, if more time than that passes you have found the ME/Service jumper and then we can re-write the ME using Intel FPT
I would test either E14 first, or E1 directly next to Password clear, this is often either ME/Service or BB Boot block recover. Once you find which disables ME, if any, we can proceed and you’ll be fixed up in no time.
But, if none of the 3 pin headers are ME/Service mode, then you will have to either purchase a CH341A flash programmer and SOIC8 test clip cable, or pinmod the board (Short a few pins on startup), documented here in section E- and E.1
[Guide] Unlock Intel Flash Descriptor Read/Write Access Permissions for SPI Servicing
I forgot you have programmer!

Please send me your original full BIOS dump from your programmer setup before you starting messing with stuff, that way I can use your corrupted ME to transfer settings to a new clean ME/BIOS combo and you can reprogram the entire BIOS at once.
I will also unlock the FD, that way once you program back in the BIOS after that we can do all further fixe using Intel FPT Tool, but it should only take that one programming anyway.
Unlocked FD if your board does not auto lock it again will save you a lot of hassles in the future if any similar stuff comes up.

When I used me_cleaner, the BIOS is read to a file in the Pi, then ‘cleaned’ to a new BIOS image and written back. There isn’t any jumper involved. me_cleaner unlocks protected ME regions in the BIOS during the cleaning process. …or at least that’s how I understood it.

None of the headers I’ve mentioned are 3 pin, all are open two pin. I have found another undocumented 2 pin header (E18) fitted with a jumper (shorted) next to the RAM sockets. I’ve had a quick internet search for HP schematics (repair type) in the hope of learning what these headers really do but I don’t think they exist.

I’ve attached 2 files. I’ve named each file after the CPU fitted to the motherboard. The installed motherboard is x3470o.rom.

Both are the ‘before’ full bios dump collected by me_cleaner in advance of running the tool in my attempt to resolve the 30 minute watch dog issue.

I’m out for a few hours. When I come back, I’ll have a punt at shorting E14, then E1 and lastly E18.

As requested, I will not write back the x3470o.rom for the time being.

Thanks again for your help. This is really appreciated.

x3450o.rom.zip (2.95 MB)

x3470o.rom.zip (993 KB)


From memory I think there maybe a TPM style jumper. Somewhere in Z series documentation I have a board diagram buried … I’ll look shortly.


OK, I found what I was looking for… Your rig is Z200? On the moboard, at top right corner above the DIMM slots, is there a 3 pin jumper designated E15?

Possible the jumper is located somewhere else, but the screenprint desig will be same, E15.

What you need to do is download the latest BIOS from HP and flash sp75207 > DOS_Flash > 7H3_0122.bin with the SPI programmer. Your current SPI image has Pre-Production ME firmware which is wrong for your platform as you need Production. This was caused by HP releasing by accident PRE firmware instead of PRD at their “Intel Management Engine Firmware Update and Utilities v6.2.61.3535” downloadable sp81826 softpaq. I had noticed that back in August 2017 and sent them an email explaining their mistake. They responded after a month saying that the problem was fixed. Well, not really apparently.


I just read this post … & had a lightbulb moment. lol

@lwoody Did you look in BIOS to see how it’s configured? AMT might be hidden, or not available, or TPM might be enabled preventing local access, or various other factors.

A password is required to access security settings, which will in turn allow access to other hidden/unavailable options such as AMT configuration. So you first make a password, then reboot and in BIOS menu some new security options should be open…
Then follow your nose and you’ll come to Trusted Computing section (MEBx I think Intel call it) which in your case should be disabled. OEM’s always enable that by default because of Microsoft.

I could send you a 200 page service manual if I know exactly what board it is.

Hi

e15 is indeed in the top right hand corner. It’s described as Crisis recovery header/jumper in the Z200 manual I’ve downloaded. The pins aren’t actually on this board, just the pads. For some reason, the site rotates the image by 90 degrees when the image is clicked on to enlarge.

IMG_2368.JPG



I’ve just back in. Is it still worth me trying to short out the undocumented jumpers or are you thinking about forcing a BIOS recovery?

Thanks for this. You are correct that a password here changes things a bit.

I did add a BIOS password which was required to amend some of the options in the screen below. However, it didn’t make a difference with Ctrl-P unfortunately.


I since hacked away and probably made things worse as I’ve wiped all ME regions in the BIOS via me_cleaner.



Thanks for raising this with HP. I used SP75207 to flash this board to 1.22. SP81826 will not install as it wants to find MEI drivers for some weird reason. I might benefit from your email history if you happen to have should the upcoming recovery attempt fail. I might be able to twist their arm for a free fix - unlikely but worth a punt should it come to it.