[Problem] Dell R720xd iDRAC BIOS Recovery

I’m wary of flashing the image I’ve rebuilt onto the board with failed iDRAC due to the risk the flasher will fail to write the same way it fails to read. I am also sufficiently doubtful about this being an issue related to a broken u-boot - IMO this is due to a broken MMC.

The opensource package from dell includes all that is needed to regenerate firmimg.d7, but not the source of u-boot, which instead seems to be provided in binary form. I’m inclined to believe it is still possible to break u-boot but I have yet to figure out the keys and timings.

Since you have programmer, it will be OK, it’s already failed and you have a backup image of the failed chips contents, so you can’t loose anything by trying. But yes, there is what you mentioned, not being able to get a consistent verified read, so write will probably be the same until you can get that sorted out.
I tested your exact programmer and that chip MX25L3206E, with the version of software and CHIP ID selection I mentioned previously, so we know that works. If it continues to fail for you, fail in a “verify” way during programming, not fail to boot/work, then we know there is an issue with your programming process somewhere and not a problem with the chip/programmer/software. Of course, knowing if the new image works or not is another issue, only someone else with one of these can test, or you can test on another working one provided you can backup it’s chip contents confidently enough to erase/reprogram that with your new image.

Just ordered a kit to modify SOP8 chipset, did the same upgrade to latest 2.x version and bricked the motherboard on a R620.
Will let you know how it went!

Just curious if anyone was able to figure something out, just ran into this same issue

@DreddKLC - I think everyone still working on figuring it out.

Thanks for the update, I just fell into the same hole that @kwaleeb did…step by step, in the same exact way…it was pretty funny (you have to laugh) that I succeeded then failed in the same manner…it was kinda sad, kinda funny…what can I say. The fix is “easy”, if I can find a way to get the 1.66 bios flashed on the idrac7. In case it helps, below is the link to the correct update path. I have the file that should bring this thing back to life, but no means of flashing it.

https://www.dell.com/support/article/us/…e-path-?lang=en


Update Path for iDRAC 7


If the iDRAC firmware is still running on a version between 1.0.0 and 1.57.57, follow these steps:

Update the iDRAC to version 1.66.65, which is the last one separated from LCC firmware

Next version to install is 2.10.10.10, the first one with combined LCC and iDRAC firmware

Now update to latest version provided on the Dell Support webpage (Section 1 of this article)

There are tricks which can work around the need to use a chip programmer, which is kind of hit and miss at least in my experience, as long as the SPI chip is functional. Said tricks are however covered by a NDA as Dell is working on an update that supposedly fixes the security hole revealed by iDRACula - and credit for them goes to fohdeesha. The next iDRAC release is estimated to happen around December/January, I’d expect them to become public after that.

So… Use PM and threats to secrecy

So I just updated the replacement r620 from < 1.47 to 2.60 and used HTML5 console.

As was written here earlier the path is
* Do every step up to 1.66.65, in my case, 1.46, 1.57, 1.66.65.
* Update Lifecycle from every step up to 1.4.2.12, in my case 1.3.1.13, 1.4.0.128, 1.4.2.12.
* Update BIOS to the latest every step. I chickened out here and updated every step, there may be better paths.
* Install Lifecycle in the following order
** 2.10.10.10
** 2.15.10.10
** 2.20.20.20
** 2.21.21.21
** 2.30.30.30
** 2.60.60.60

Now, under virtual console you can choose type of console, pick HTML5.

When choosing what to download on dell.com, choose the win32 exe, upload in iDRAC
Except for BIOS, I just downloaded the standalone EXE and used unzip to unpack the image inside.

Now I have to wait until the 12th for my kit to arrive so I can test this out on the broken MB.

bios update should be fine, looking forward to hearing from you on this!

A few flashing and servers later, I believe that stepping through each firmware is paranoid, albeit justified. I have not had issues in bringing the server to 1.66.65 for iDRAC and the latest 1.x BIOS, then upgrading iDRAC to 2.10.10.10 and the BIOS to the earliest 2.x available. Once at that point, it becomes possible again to use the Lifecycle Controller, which updates straight to the latest and greatest available for both.

I’m also struggling with a dead IDRAC but on a T620. I’m struggling to find the UART connections – any chance you could post a pic or describe where they are on the motherboard?

I don’t have a system like that, but you should still be able to find 4 pins aligned around the area where the iDRAC connector and its CPU are.

Hi. I also have an iDRAC initialisation error on my R620 server. Bricked it yesterday on attempt to update FW. Take a look on this picture, I marked the position of UART interface on the MB. They didn’t soldered any connector, and I think I’m going to solder there 4 pins to be able to connect.

s-l1600.jpg

Hi. Two days ago I have done an experiment. I soldered out the 25DF321 FLASH memory of the iDRAC and tried to read it out using a programmer. To my surprise it was completely unreadable. Moreover, the programmer wasn’t able to recognise the type of the IC. At this point I looked at my warehouse for some old motherboards and found that some of them have a 25L64 FLASH memory as a BIOS chip. Which is suppose to be exactly the same as 25L32, but it has a twice as much memory. So I soldered it out and was able to write into this chip the R620 iDRAC dump (Nannerkins provide it to us in some of the previous posts). And soldered it into to the server’s board. Now when power was connected to the server, iDRAC’s indicator lamp on the back panel started to blink constantly with an amber light. My first thought was that I solved the problem and it just might need to make a reset to the iDRAC because before I changed the chip iDRAC was bricked completely, and there was no light on the back panel at all. To my disappointment it wasn’t true - iDRAC still does not respond to long press and the amber light continues to blink.
My next steps would be:
1. Try to use exactly the same FLASH type(25DF321) which I have already ordered from EBAY
2. Try to find another R620 and read the memory dump out of it.
3. Try to connect to UART interface and see if there any activity.

Below you can see some pictures describing desoldering an soldering process

IMG_1902.jpg

IMG_1903.jpg

IMG_1904.jpg

IMG_1905.jpg

What software did you use to read/write the 25L64 and what is it’s full name? If it’s W25L64FV (Choose BV and use software 1.30, if BV try FV instead)

Its full name is 25L6406E. I have a USB MiniPro Universal programmer. You can find it on EBAY. It has a pretty large list of supported chips.
As for FLASH itself, I prefer to wait until I’ll get 25DF321A and a clips socket adapter for in-system programming that I ordered and give it a try again. Moreover, it might be that I’m going to get the access to another R620 server to read out iDRAC BIOS. I think it will be much better to try this first. PCB don’t likes multiple soldering/soldering, and there is no option for SOIC8 adapter that can be installed between PCB and IC to allow the installation without soldering(ike you have it for ICs with DIP body type).

SOIC8 test clip cable is for BIOS chips that are soldered on, not for DIP BIOS. So once you have one you wont need to desolder chips.

This thread has been interesting, but on my r420 what I’m finding is that the idrac spi chip may indicate that the real meat and potatoes isn’t in the 25Q03213 chip at all (on my board that’s what it was).
I was able to read it with the gold CH341A, and it verified fine (after pressing down firmly a few times and using a magnifying glass to make sure both sides had pin contact).

I copied a unpatched chip, then my new patched chip. Off the bat the 0-FFFFF portion of both of those is identical.

In the next section starting at 100000 all the configuration was identical, except for 4 mac addresses.

These settings in particular make me wonder if the actual firmware isn’t stored on /dev/mmcblk0p2 (the sd card inside the server)?
hostname=idrac7
bootnfs=tftp 82000000 $(bootfile); run nfsargs; bootm
runtime_args=mode=vhard reset_cause=ac nmi_buf=0x83000000
fwudev=evb
bootargs=root=/dev/mmcblk0p2 rootwait rw rootfstype=squashfs mem=239616k console=ttyS2,115200 <NULL> mac1=00:00:00:00:00:01 mac2=00:00:00:00:00:02 mode=vhard reset_cause=ac nmi_buf=0x83000000 quiet
fwu_kernel_start=1
fwu_rootfs_start=8001
fwu_platform_start=120006
fwu_uboot_start=3fc01
fwu_kernel_size=135d
fwu_rootfs_size=19330
fwummc=emmc
ver=U-Boot 2009.08 (Jun 03 2013 - 03:47:19) Avocent (0.0.3) EVB

Has anyone actually unbricked the idrac with the bios only flash approach? My next guess is to image the sdcard and go to those offsets and see if I can find the same bytes in the firmimg.d7 file (and if so then I’ll try copying the sdcard out of the working dell).

How about the jtag port, have any of you had luck getting into that or the SPI_DBG port instead of the regular rs232 redirect?

This appears to be the emmc, as it doesn’t change even if I pull out all the sd cards.

That said, I broke down and bought a new mb from ebay, copied it’s chip, then updated it completely, and recopied it’s chip.

I then took that and flashed it back to my broken server that wouldn’t enter recovery mode and at least now I have the flashing yellow light.

I soldered the tty pins and connected with putty, and I was happy to see this:
######################################################################
** No bootable iDRAC image is found
(System Health & ID LED is flashing amber at ~1/2 second rate).
To Recover iDRAC via an SD card.
1) Format SD using FAT on a Windows box or EXT3 using Linux.
2) Copy ‘firmimg.d7’ to root path.
3) Insert SD card.
4) System Health & ID button solid amber durning recover.
5) Both boot paths are flashed.
######################################################################

Polling for SD card state change

However I can’t find any firmimg.d7 that will work.
I’ve tried 1.31.30, 1.35.35, 1.40.40, 1.65.65, 1.66.65, and 2.10.10.10 and they all pretty much do something like this:
Found ‘/firmimg.d7’ in ‘FAT’ file system


UTIL RECOVER:Transport:sd TargetMMC:EMMC File:/firmimg.d7
reading /firmimg.d7

95344358 bytes read
UTIL RECOVER:SD load passed from FAT fs.
UTIL RECOVER:Transport time [sec:mil]: 16:525

Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
’EMMC’ blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
mmc write failed000] Buf[0x88000000]…fill buffer…erase/write
util flash:
Failure writing blocks [1:196609] RC=0xffffffac
/nRecover returned failure, RC=0xffffffff
Firmimg.d7 might be bad.
Polling for SD card state change

The upgrade path for the ebay server indicated the original idrac upgrade was from 1.40.40 to 1.65.65, so I expected 1.40.40 to work. When it finally got to this state I was able to get into F2 (but not do anything), at least I could see the settings were 1.65.65, but alas that doesn’t work any better.

Now I’m back to being stumped. I suppose I’ll try flashing the post updated chip and then see if it takes 1.65.65…