ThinkPad X220 - "ME is in Recovery State" after update to IME 7.1.91.3272

Special Topics Intel Management Engine
1

After flashing the 7.1.91.3272 update with the patch for CVE-2017-5689, I ran into some problems: sleep wouldn’t work (it would initially seem to go to sleep, do the sleep-jingle etc but then cut all power, resulting in a dirty shutdown) and restart would result in power being cut turning back on. I suspected this was because I was on a modified version of the 1.42 BIOS that unlocks RAM speed, removes the WLAN whitelist, and exposes other advanced features, so I updated to the latest 1.43 BIOS, and then, without rebooting, to a similarly modified version of the 1.43 BIOS (so that at no point would the WLAN whitelist be triggered). The modified BIOS can be found here. Unfortunately this didn’t fix anything. I reflashed the 7.1.91.3272 update, still nothing - sleep still didn’t work, and restart was still cutting power before booting.

I wasn’t really sure at this point about the nature of the problem. So, after running a chkdsk (required after so many dirty shutdowns), I tried doing a System Restore to before I had done any flashing. Upon rebooting to complete the system restore however, my main mSATA system drive ceased being recognized, and the ‘ME is in Recovery State’ message began showing up. After rebooting a couple of more times - I don’t remember if I used CTRL-ALT-DEL or cut power - the mSATA drive began being recognized, and Windows began booting normally. And while, unfortunately, the ‘ME is in Recovery State’ error persists, sleep and shutdown are working normally again. Here’s what the ‘ME is in Recovery State’ screen looks like when the BIOS is set to show diagnostics:



At this point my situation seems pretty similar to the one in this thread, though MEInfo and MEManuf logs are slightly different.

MEInfo - per the original thread I upgrade the ME drivers to 11.0. UNS and LMS versions show up as “Not Available”.



Intel(R) MEInfo Version: 7.1.50.1166
Copyright(C) 2005 - 2011, Intel Corporation. All rights reserved.

Intel(R) ME code versions:

BIOS Version: 8DET73WW (1.43 )
MEBx Version: 7.0.0.63
Gbe Version: 1.3
VendorID: 8086
PCH Version: 4
FW Version: 7.1.91.3272
UNS Version: Not Available
LMS Version: Not Available
MEI Driver Version: 11.0.5.1189
Wireless Hardware Version: Not Available
Wireless Driver Version: Not Available

FW Capabilities: 234249317

Intel(R) Active Management Technology - PRESENT/ENABLED
Intel(R) Anti-Theft Technology - PRESENT/ENABLED
Intel(R) Capability Licensing Service - PRESENT/ENABLED
Protect Audio Video Path - PRESENT/ENABLED
Intel(R) Dynamic Application Loader - PRESENT/ENABLED

Intel(R) AMT State: Enabled
CPU Upgrade State: Upgrade Capable
Cryptography Support: Enabled
Last ME reset reason: Power up
Local FWUpdate: Enabled
BIOS and GbE Config Lock: Enabled
Host Read Access to ME: Disabled
Host Write Access to ME: Disabled
SPI Flash ID #1: C22017
SPI Flash ID VSCC #1: 20052005
SPI Flash BIOS VSCC: 20052005
BIOS boot State: Pre Boot
OEM Id: 4c656e6f-766f-0000-0000-000000000000

Error 8199: Communication error between application and Intel(R) ME (Get Intel(R) AMT State)

Error 8199: Communication error between application and Intel(R) ME (Get System UUID)

Error 8199: Communication error between application and Intel(R) ME (Get Ipv4 Info)

Error 8199: Communication error between application and Intel(R) ME (Get Ipv4 Info)

Error 8199: Communication error between application and Intel(R) ME (Get Ipv6 Info)

Error 8199: Communication error between application and Intel(R) ME (Get Ipv6 Info)

Error 8199: Communication error between application and Intel(R) ME (Get Provisioning State)

Error 8199: Communication error between application and Intel(R) ME (Get Provisioning Mode)
Capability Licensing Service: Enabled
Capability Licensing Service Status: Permit info not available
OEM Tag: 0x00000000


MEManuf - in particular this is much shorter than the log in the other thread and has a ‘No Intel ME test result to retrieve’ error. Also, our FW Status Registers have different values, and ManufacturingMode is disabled on my machine. Practically I’m not sure if this makes a difference or not.



Intel(R) MEManuf Version: 7.1.50.1166
Copyright(C) 2005 - 2011, Intel Corporation. All rights reserved.

Platform stepping value is 4

FW Status Register1: 0x1E000242
FW Status Register2: 0x68000006

CurrentState: Recovery
ManufacturingMode: Disabled
FlashPartition: Valid
OperationalState: M0 with UMA
InitComplete: Complete
BUPLoadState: Success
ErrorCode: No Error
ModeOfOperation: Normal
ICC: Valid OEM data, ICC programmed

Get FWU info command…done

Get FWU version command…done

Get FWU feature state command…done

Get ME FWU platform type command…done

Get ME FWU feature capability command…done
Feature enablement is 0xDF65C65
gFeatureAvailability value is 0x1
OEM ICC data valid and programmed correctly

Request Intel(R) ME test result command…done
vsccommn.bin was created on 04:35:50 08/08/2012 GMT
SPI Flash ID #1 ME VSCC value is 0x2005
SPI Flash ID #1 (ID: 0xC22017) ME VSCC value checked
SPI Flash ID #1 BIOS VSCC value is 0x2005
SPI Flash ID #1 (ID: 0xC22017) BIOS VSCC value checked
FPBA value is 0x0
No Intel Wireless device was found

Error 9313: No Intel(R) ME test result to retrieve

Error 9296: MEManuf Test Failed (9313)



Attempts to do an SPI dump failed with a Code 26 error:


Error 26: The host CPU does not have read access to the target flash area. To enable read access for this operation you must modify the descriptor settings to give host access to this region.


And attempts to run “fptw64 -greset” result in a Code 205 error:


Error 205: Failure. Unexpected error occurred.


All that out of the way, I have a few questions about how to proceed.

  • What consequences are there to not fixing this, other than a slightly longer boot-up time?
  • I understand, per the other thread, that the code 26 error can be circumvented via shorting together a couple of pins on the audio circuit. I’m not sure I have anything slight enough that it would only touch those two pins. Are there any recommendations for a specific tool to do this with?
  • The procedure to fix this appears to involve extracting a current BIOS, replacing the ME region from a stock image, and reprovisioning it properly, as per the guide here.
    • It appears that in my case this would require extracting the ‘clean’ ME Region from 7.1.80.1214_5MB_ALL_PRD_RGN.bin. Is this correct?
    • If so, can I later upgrade to the 7.1.91.3272 version with the patch without issues? Or is this somehow rolling the dice?
    • In the other thread, both of the people helped ended up getting a code 28 error when trying to flash the ‘fat’ image, and are then instructed to only flash Descriptor and ME Regions.
      • How are specific descriptor and ME Region files made?
      • Can/should these be flashed directly, without first flashing the ‘fat’ image with the entire bios? (I mean it certainly looks like flashing the fat image won’t work…)



Thanks a lot for bearing with me here, really appreciate any further advice (the guides and previous thread were already very informative!)

2

Accidentally hibernating my system appears to have triggered a global system reset. While this has gotten rid of the “ME is in Recovery State” message, standby and restart are broken again. fptw still can’t dump. Not really sure how to proceed. Regardless, I guess this means I can’t avoid trying to deal with this :confused:

3

On a lark I tried downgrading using FwUpdLcl.exe. Running “FwUpdLcl.exe -blist” resulted in a pretty short blacklist - only 7.0.10.1203 and 7.1.13.1088 were disallowed. So, I took 7.1.80.1214_5MB_ALL_PRD_UPD.bin from the repository pack, and modified the install script that came with the official 7.1.91.3272 upgrade package from Lenovo to use it instead. (The only distinctive thing in the script is the feeding of the OEMID parameter to FwUpdLcl, which I figured was important.) Long story short, I appear to have rolled back to 7.1.80.1214 successfully, with no sleep or restart issues and the ME in normal, rather than recovery, state. (The only thing is, the MEManuf test still fails with errors 9313/9296, but it’s unclear if this matters in practice…)

There are two newer firmwares, 7.1.85.1216 and 7.1.86.1221, that I guess can also be tried, but I’m not really sure what they change that is of any benefit. In contrast, I would still like to be using 7.1.91.3272 as CVE-2017-5689 looks pretty severe. Does it seem like my sleep issues were due to the order in which I did the flashing, or is it a problem with 7.1.91.3272 itself? If not, I guess I can just disable AMT for now or something (not like I ever use it).

4

@LastSilmaril

Hello LastSilmaril, sorry for the delay. It is very impressive how much research you’ve done before opening a new topic. Such a thing is not common. Thank you for that, great work.


A non-operational ME may cause various issues depending on the system. The ones you have could be some of them. Thus, it would be best to keep it working at your own system.


Something thin and conductive like clippers, tweezers, paperclip etc.


Not “extracting” the RGN. You basically take your current SPI dump (Flash Descriptor + BIOS + ME) and follow the CleanUp Guide to replace the ME region with a Configured one, based on the stock/clean image from the Repositories. Any 5MB_ALL_PRD_RGN will do for you. The instructions can be found at the guide, every step is detailed. Thing is, you would need read/write access to the ME region for that, thus an unlocked Flash Descriptor (pinmod etc).


Yes, that’s the idea. You can clean your current SPI dump via the CleanUp Guide using the latest 5MB RGN firmware (7.1.80.1214 as of now, just like you said) and once the system is up and running, flash to the latest via FWUpdate (7.1.91.3272).


At some systems, the OEMs implement protected BIOS ranges so the BIOS cannot be reflashed in its entirety or even at all. So flashing an entire (ME-cleaned) SPI image will hang at the BIOS reflashing. That protection is irrelevant to Intel’s Flash Descrptor (what the pinmod unlocks). However, we don’t have to worry about those protected BIOS ranges because we just want to reflash the ME region with a cleaned+configured one. Thus, flashing the ME region only bypasses that issue and also solves the actual ME problem.


You can either prepare an entire SPI image with all its regions and then manually extract whatever you want from it via hex editors, UEFITool and so on. For the ME region, you also have the option to prepare just that at Intel’s Flash Image Tool and then manually put it back at a full SPI image via hex editors, UEFITool etc. Both methods are mentioned at the CleanUp Guide.


Flashing the full SPI image works just fine on most systems. When the Flash Descriptor and/or BIOS Protected Ranges are enabled, things become harder. There is no problem in flashing individual SPI regions by themselves.


If “fpt -greset” fails, you can do it manually. For 5MB systems, you need to remove all power (cord + battery, RTC not needed) for 1 minute. For 1.5MB systems, a simple shutdown/reboot is enough but not for 5MB.


Actually, for branch 7.0 it is <= 7.0.10.1203 and for branch 7.1 it is <= 7.1.13.1088. You can use ME Analyzer to view the blist in a better way. Basically, for you, any firmware after 7.1.13.1088 can be used for downgrading since Intel forgot to update the blacklist for 7.1.91.3272 even though it clearly fixed something important.


As you can see from your MEInfo report, your system does have an OEM ID registered so indeed you do need it at FWUpdate while updating. Correct call.


It’s probably ok. Do you see MEManuf errors when you run it with parameter “-EOL”? Do you see any errors at MEInfo while on the (working) 7.1.80.1214 firmware?


Indeed, for 5MB systems, having 7.1.91.3272 applied is the best thing to do.


I believe neither. It could be a BIOS/MEBx incompatibility or a dirty ME region causing issues after updating to 7.1.91.3272. Both are possible but personally I would consider the latter more likely. I suggest to:

1) Update the BIOS to the latest version from Lenovo’s website
2) Update the ME to the latest version from Lenovo’s website (they definitely have 7.1.91.3272 for all their affected machines)
3) If the problems re-appears, try a “fpt -greset” or a manual one if the former fails.
4) If the problem persists, we have ruled out BIOS/MEBx incompatibility so the issue is almost certainly a corrupted ME region (its code or settings).
5) In such case, the only way is to unlock the FD and reflash the ME region manually after following the CleanUp Guide.
6) The reflashed firmware will be 7.1.80.1214 (last RGN), after which you can use FWUpdate to go to 7.1.91.3272.

If everything above fails, I suggest you downgrade to the last working ME firmware (7.1.85 maybe, whatever you want) and unprovision+disable AMT from MEBx. Maybe contacting Lenovo could help at such point.

5



Ha, you’re welcome - thank you for getting back to me and for collecting these resources. Literally the only place on the internet with this information, as far as I can tell. (I’ve been busy this week so haven’t had the chance to get back to you.)



I hopefully will have some time tomorrow to do this. I have my tools handy now so it’s doable. Will report back on my findings once I actually try this.