bios update should be fine, looking forward to hearing from you on this!
A few flashing and servers later, I believe that stepping through each firmware is paranoid, albeit justified. I have not had issues in bringing the server to 1.66.65 for iDRAC and the latest 1.x BIOS, then upgrading iDRAC to 2.10.10.10 and the BIOS to the earliest 2.x available. Once at that point, it becomes possible again to use the Lifecycle Controller, which updates straight to the latest and greatest available for both.
I’m also struggling with a dead IDRAC but on a T620. I’m struggling to find the UART connections – any chance you could post a pic or describe where they are on the motherboard?
I don’t have a system like that, but you should still be able to find 4 pins aligned around the area where the iDRAC connector and its CPU are.
Hi. I also have an iDRAC initialisation error on my R620 server. Bricked it yesterday on attempt to update FW. Take a look on this picture, I marked the position of UART interface on the MB. They didn’t soldered any connector, and I think I’m going to solder there 4 pins to be able to connect.
Hi. Two days ago I have done an experiment. I soldered out the 25DF321 FLASH memory of the iDRAC and tried to read it out using a programmer. To my surprise it was completely unreadable. Moreover, the programmer wasn’t able to recognise the type of the IC. At this point I looked at my warehouse for some old motherboards and found that some of them have a 25L64 FLASH memory as a BIOS chip. Which is suppose to be exactly the same as 25L32, but it has a twice as much memory. So I soldered it out and was able to write into this chip the R620 iDRAC dump (Nannerkins provide it to us in some of the previous posts). And soldered it into to the server’s board. Now when power was connected to the server, iDRAC’s indicator lamp on the back panel started to blink constantly with an amber light. My first thought was that I solved the problem and it just might need to make a reset to the iDRAC because before I changed the chip iDRAC was bricked completely, and there was no light on the back panel at all. To my disappointment it wasn’t true - iDRAC still does not respond to long press and the amber light continues to blink.
My next steps would be:
1. Try to use exactly the same FLASH type(25DF321) which I have already ordered from EBAY
2. Try to find another R620 and read the memory dump out of it.
3. Try to connect to UART interface and see if there any activity.
Below you can see some pictures describing desoldering an soldering process
What software did you use to read/write the 25L64 and what is it’s full name? If it’s W25L64FV (Choose BV and use software 1.30, if BV try FV instead)
Its full name is 25L6406E. I have a USB MiniPro Universal programmer. You can find it on EBAY. It has a pretty large list of supported chips.
As for FLASH itself, I prefer to wait until I’ll get 25DF321A and a clips socket adapter for in-system programming that I ordered and give it a try again. Moreover, it might be that I’m going to get the access to another R620 server to read out iDRAC BIOS. I think it will be much better to try this first. PCB don’t likes multiple soldering/soldering, and there is no option for SOIC8 adapter that can be installed between PCB and IC to allow the installation without soldering(ike you have it for ICs with DIP body type).
SOIC8 test clip cable is for BIOS chips that are soldered on, not for DIP BIOS. So once you have one you wont need to desolder chips.
This thread has been interesting, but on my r420 what I’m finding is that the idrac spi chip may indicate that the real meat and potatoes isn’t in the 25Q03213 chip at all (on my board that’s what it was).
I was able to read it with the gold CH341A, and it verified fine (after pressing down firmly a few times and using a magnifying glass to make sure both sides had pin contact).
I copied a unpatched chip, then my new patched chip. Off the bat the 0-FFFFF portion of both of those is identical.
In the next section starting at 100000 all the configuration was identical, except for 4 mac addresses.
These settings in particular make me wonder if the actual firmware isn’t stored on /dev/mmcblk0p2 (the sd card inside the server)?
hostname=idrac7
bootnfs=tftp 82000000 $(bootfile); run nfsargs; bootm
runtime_args=mode=vhard reset_cause=ac nmi_buf=0x83000000
fwudev=evb
bootargs=root=/dev/mmcblk0p2 rootwait rw rootfstype=squashfs mem=239616k console=ttyS2,115200 <NULL> mac1=00:00:00:00:00:01 mac2=00:00:00:00:00:02 mode=vhard reset_cause=ac nmi_buf=0x83000000 quiet
fwu_kernel_start=1
fwu_rootfs_start=8001
fwu_platform_start=120006
fwu_uboot_start=3fc01
fwu_kernel_size=135d
fwu_rootfs_size=19330
fwummc=emmc
ver=U-Boot 2009.08 (Jun 03 2013 - 03:47:19) Avocent (0.0.3) EVB
Has anyone actually unbricked the idrac with the bios only flash approach? My next guess is to image the sdcard and go to those offsets and see if I can find the same bytes in the firmimg.d7 file (and if so then I’ll try copying the sdcard out of the working dell).
How about the jtag port, have any of you had luck getting into that or the SPI_DBG port instead of the regular rs232 redirect?
This appears to be the emmc, as it doesn’t change even if I pull out all the sd cards.
That said, I broke down and bought a new mb from ebay, copied it’s chip, then updated it completely, and recopied it’s chip.
I then took that and flashed it back to my broken server that wouldn’t enter recovery mode and at least now I have the flashing yellow light.
I soldered the tty pins and connected with putty, and I was happy to see this:
######################################################################
** No bootable iDRAC image is found
(System Health & ID LED is flashing amber at ~1/2 second rate).
To Recover iDRAC via an SD card.
1) Format SD using FAT on a Windows box or EXT3 using Linux.
2) Copy ‘firmimg.d7’ to root path.
3) Insert SD card.
4) System Health & ID button solid amber durning recover.
5) Both boot paths are flashed.
######################################################################
Polling for SD card state change
However I can’t find any firmimg.d7 that will work.
I’ve tried 1.31.30, 1.35.35, 1.40.40, 1.65.65, 1.66.65, and 2.10.10.10 and they all pretty much do something like this:
Found ‘/firmimg.d7’ in ‘FAT’ file system
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:/firmimg.d7
reading /firmimg.d7
95344358 bytes read
UTIL RECOVER:SD load passed from FAT fs.
UTIL RECOVER:Transport time [sec:mil]: 16:525
Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
’EMMC’ blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
mmc write failed000] Buf[0x88000000]…fill buffer…erase/write
util flash:
Failure writing blocks [1:196609] RC=0xffffffac
/nRecover returned failure, RC=0xffffffff
Firmimg.d7 might be bad.
Polling for SD card state change
The upgrade path for the ebay server indicated the original idrac upgrade was from 1.40.40 to 1.65.65, so I expected 1.40.40 to work. When it finally got to this state I was able to get into F2 (but not do anything), at least I could see the settings were 1.65.65, but alas that doesn’t work any better.
Now I’m back to being stumped. I suppose I’ll try flashing the post updated chip and then see if it takes 1.65.65…
Hi,
Last time when I wrote here I said that I was able to program iDRAC BIOS. After that it’s indicator started to blink with an amber light. By that time I didn’t know that this indicates “pooling for MMC card” status. Only after I had a UART adapter connected I found out what’s going on behind the scene. First, I was glad, and thought that I’m one-step from the solution. But finally all my experiments are ended up exactely as yours. No matter what firmimg.d7 file version tried.
What looks confuzing for me, is that string, where it says “Firmimg.d7 might be bad”, which comes right after "Failure writing blocks [1:196609] RC=0xffffffac
/nRecover returned failure, RC=0xffffffff"
Here a qustion arise in my head: Either a “firmimg.d7” has to be some sort of “recovery version” which is not a part of a regular firmware package and should be downloaded hell knows where from. Or this is an eMMC chip failure.
Polling for SD card state change <br />SD card state change, previous=0x700 new=0x780
######################################################################
** No bootable iDRAC image is found
(System Health & ID LED is flashing amber at ~1/2 second rate).
To Recover iDRAC via an SD card.
1) Format SD using FAT on a Windows box or EXT3 using Linux.
2) Copy ‘firmimg.d7’ to root path.
3) Insert SD card.
4) System Health & ID button solid amber durning recover.
5) Both boot paths are flashed.
######################################################################
Found ‘/firmimg.d7’ in ‘FAT’ file system
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:/firmimg.d7
reading /firmimg.d7
62953181 bytes read
UTIL RECOVER:SD load passed from FAT fs.
UTIL RECOVER:Transport time [sec:mil]: 10:987
Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
’EMMC’ blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
mmc write failed000] Buf[0x88000000]…fill buffer…erase/write
util flash:
Failure writing blocks [1:196609] RC=0xffffffac
/nRecover returned failure, RC=0xffffffff
Firmimg.d7 might be bad.
Polling for SD card state change \
I would have thought so too, but what are the odds both our healthy systems would have failed in exactly the same spot. Plus, googling that I found another person recently posted that same error to pastebin. I confirmed the crc on the download from Dell, so I know my firmware isn’t corrupt. it does seem the emmc fails to write though.
I reflashed the post-updated chip to my other server and it’s still doing the same thing, I did see something in the bios that’s probably something I need to consider.
iDRAC Settings version…1.65.65.04
iDRAC Firmware Version…2.61.60 (Build 8)
Maybe I should try giving it 2.61.60.
This particular server killed itself when I connected the lifecycle controller to the internet and told it to just download all the updates from ftp.dell.com
Which bugs me to no end, but that aside, it clearly downloaded a 2.61.60 build that I haven’t come across yet…I’m not sure why I didn’t see that when I was at work but I guess I’ll try that tomorrow. But maybe the problem is that the settings don’t match the bios.
In that case I’ll make sure to update the other server to 2.61.60 and then dump the idrac and write that…they sure make it hard to back out of this situation.
I also gave up my work when I reached the "Failure writing blocks" roadblock. Pretty sure the EMMC is bad. How realistic/difficult would replacing the EMMC be?
Well, that’s might be a problem of settings version and firmware version compatibility. But why should it be so? I mean, isn’t the flash utility should just erase and reset everything? If I understand this correctly, both firmware and settings are located in eMMC chip. There should be another option which is much interesting for me - the ability to interrupt U-Boot process. If I understand what guys says, there is a possibility to load iDRAC OS to RAM. Maybe after this it will be available fo firmware update.
Take a look here https://amp.reddit.com/r/homelab/comment…idrac_recovery/
It was strange to me that this is a drac8 download, but it does say for idrac7 (The firmware was iDRAC8/7 with Lifecycle Controller Version 2.61.60.60), and the filename was firmimg.d7 but the outcome was the same:
Found ‘/firmimg.d7’ in ‘FAT’ file system
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:/firmimg.d7
reading /firmimg.d7
109270246 bytes read
UTIL RECOVER:SD load passed from FAT fs.
UTIL RECOVER:Transport time [sec:mil]: 18:918
Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
’EMMC’ blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
Blocks [0x1:0x30000] Buf[0x88000000]…fill buffer…erase/write
mmc write failed
util flash:
Failure writing blocks [1:196609] RC=0xffffffac
/nRecover returned failure, RC=0xffffffff
Firmimg.d7 might be bad.
I think the failure at block 1 (if I’m even reading that right) is the part that gets me. I wonder if there isn’t a write enable issue that has the emmc locked or something like that.
I downloaded the datasheet. It only uses 30 pins that I can see. 5 of those are core ground, and 5 are core voltage. I would presume if you could interrupt either voltage or ground to the chip it would render it inert. Then you could conceivably patch a new on onto those pins (but supply ground and voltage).
If I ever did that I’d do it with a socket so I would only have to do it once. The only real value would be to break chips in a way you can easily remove them and put them into another circuit to mount them in another filesystem for direct access.
Honestly though, I doubt I’ll get to it. You could also probably get a high speed carbide bit and just grind the old chip off (wear goggles but I’m mostly kidding).
I think I’ve read that before and tend to discard the idea due past failures along a similar line. I didn’t have my tty console at the time, so maybe I’ll try it again, but I suspect that changing the variables directly and inserting them into the flash and writing it (as I’ve tried) doesn’t work because it changes the CRC.
As for the J47 jumper, my jumpers in that area all seem to be around J25, so although there may be a similar function fining it (if it even exists on a 420) is another matter.
I think there are two candidates I’ll mess with by the EMMC there’s a switch diagram though it’s not really labelled, and there’s a j25 header.
I’m also intrigued by the sw2 block, and the JTAG ports. If I get some spare cash I’d like to buy a jtagulator and see if I can get into those.
So the sw header turns out to be interesting. The two by the emmc didn’t do anything, nor did sw1. sw2 however did do this:
U-Boot 2009.08-00088-g121cddc (Nov 17 2014 - 05:50:46) Avocent (0.0.3) EVB, Build: jenkins-idrac-yocto-release-505
CPU: SH-4A
BOARD: R0P7757LC00xxRL (C0 step) board
BOOT: Secure, HRK not generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
In: serial
Out: serial
Err: serial
PCIe: Bridge loaded with 0x18000 bytes
WDT2: Booted Lower Vector, 'uboot1’
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:004 U-Boot 2009.08-00088-g121cddc (Nov 17 2014 - 05:50:46) Avocent (0.0.3) EVB
INFO: 00:009 U-Boot checkin date(05-10-2013) Version(1.0.183)
INFO: 00:005 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB AT25DF321A bus=0 cs=0, speed=1000000, mode=3
INFO: 00:008 SH-4A Product: Major Ver=0x31 Minor Ver=0x14 C4 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
PASS: 00:015 Dedicated monolithic mgmt NIC enabled (vendor mode override)
INFO: 00:131 BCM54610 OUI=0x00d897 Model=0x26 Revision=0x0a PhyAddr=1
INFO: 00:167 SD CARD: Device: sh-sdhi Manufacturer ID: 12 OEM: 3455
Name: SD
INFO: 00:472 EMMC: Device: sh_mmcif Manufacturer ID: 90 OEM: 14a
Name: HYNIX Tran Speed: 25000000 Rd Block Len: 512
MMC version 4.0 High Capacity: Yes Capacity: 0
INFO: 00:019 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x4
Planar: Type=0x09 Rev=0x4 Rework=0x0 Scratch/PathRetry=0x00
PASS: 00:013 Coin cell detected good, AD=0x39c low water=0x2c1
PASS: 00:008 PCIe C0 Ver=0.15 MCTP en, CRC=0x8e9b6875 @0x8efbf954 cnt=0x18000
INFO: 00:007 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=vhard
Erasing SPI flash at 0x100000…Writing to SPI flash…done
Erasing SPI flash at 0x110000…Writing to SPI flash…done
PASS: 01:236 Booted Lower Vector, ‘uboot1’ wdt2cnt=0
INFO: 00:005 wdt0cnt=0
PASS: 00:003 Clear CH1/CH2, clear 4K shared memory@0xffcaa000 on AC power up
PASS: 00:007 SMR0 no sermux env, default 0xd4
INFO: 00:005 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x7f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=90:B1:1C:54:43:89
getherc0=90:B1:1C:54:43:8A
INFO: 00:009 Fan logic for monolithic planar type 9
fan1 - def 0000 1fff 3d fan2 - def 0000 1fff 3d
fan3 - def 0000 1fff 3d fan4 - def 0000 1fff 3d
fan5 - def 0000 1fff 3d fan6 - def 0000 1fff 3d
fan7 - def 0000 1fff 3d fan8 - def 0000 1fff 3d
INFO: 00:076 Env and backup CRC’ed ok
*** no text signature found ***
INFO: 00:649 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:259 Current u-boot1 1.0.183 verified with ‘ubootN’
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
U-boot2 in sync with u-boot1 1.0.183
FAIL: 00:219 Verify OS Images N: Kernel crc exp=0xca4431c0 rec=0x26c73dbc
blk_start=0x1 blk_size=0x1360 ENV bcnt=0x26b937
PASS: 00:257 Boot device=emmc Boot partition5/N-1
Boot Path Retry:P1/N=3 P5/N-1=0 swapping to P5/N-1
INFO: 00:012 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:258 Current u-boot1 1.0.183 verified with ‘ubootN1’
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
U-boot2 in sync with u-boot1 1.0.183
FAIL: 00:437 Verify OS Images N-1: Kernel crc exp=0x8c86b2b rec=0x3059b451
blk_start=0x40002 blk_size=0x1360 ENV bcnt=0x26d5d5
FAIL: 00:013 Boot device=emmc Boot partition5/N-1
Boot Path Retry:P1/N=3 P5/N-1=3 P5/N-1 now expired
INFO: 03:408 Dedicated NIC has no link, switch to use NCSI
INFO: 00:005 No ‘userexe’ defined
INFO: 00:000 07:752
Hit any key to stop autoboot: 3
WDT2: Disable in abortboot()
OSWDT: Disable in abortboot()
CPLD_BMCRDY: Enable BMC_MIN_RDY in abortboot(). Prevent BIOS reset.
NOTE: After stopping u-boot in this development mode. You may need to
warm/cold reset the server when booting iDRAC manually as BIOS
may have already viewed iDRAC as unresponsive.
RECOVER:Max retries occured for both N/N-1 paths, OR forced recover.
iDRAC7=>
Now I guess I need to figure out what I can do with that prompt.
I’m so excited. This is like, amazing. Even if it’s still bricked, how cool.
iDRAC7=> mmcinfo
Device: sh_mmcif
Manufacturer ID: 90
OEM: 14a
Name: HYNIX
Tran Speed: 25000000
Rd Block Len: 512
MMC version 4.0
High Capacity: Yes
Capacity: 0
Bus Width: 4-bit
then I bumbled around and figured this out:
iDRAC7=> protect off all
Then more bumbling and I figured this out:
iiDRAC7=> util recover -emmc -from sd -f firmimg.d7
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:firmimg.d7
reading firmimg.d7
109270246 bytes read
UTIL RECOVER:SD load passed from FAT fs.
UTIL RECOVER:Transport time [sec:mil]: 18:917
Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
’EMMC’ blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
Blocks [0x1:0x30000] Buf[0x88000000]…fill buffer…erase/write
mmc write failed
util flash:
Failure writing blocks [1:196609] RC=0xffffffac
UTIL PASS
iDRAC7=>
I’m not clear on where to go from there though…So I unplugged it and plugged it back in, but it put me right back to this:
Found ‘/firmimg.d7’ in ‘FAT’ file system
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:/firmimg.d7
reading /firmimg.d7
109270246 bytes read
UTIL RECOVER:SD load passed from FAT fs.
UTIL RECOVER:Transport time [sec:mil]: 18:918
Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
’EMMC’ blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
mmc write failed000] Buf[0x88000000]…fill buffer…erase/write
util flash:
Failure writing blocks [1:196609] RC=0xffffffac
/nRecover returned failure, RC=0xffffffff
Firmimg.d7 might be bad.
I’ll keep dinking with it and see if I can get it to boot into memory like you suggested. I’ll probably even solder a switch onto sw2 since I can see I’ll be doing that for awhile and the paperclip is a bit awkward.