great video! keep us updated with progress!
if we leave the arduino in place, could we make the arduino load the idrac firmware into memory from TFTP? that way at least we have a temporary idrac running.
the reason for this is that then reboots would not take 20 minutes
the only time it would take 20 minutes again is if the arduino has the load the idrac firmware again, which I think would only happen on a hard power cut? regular reboots/shutdowns should be fine (idrac would continue to run even with the server shut down)
^^this is if i understand idrac correctly
Yeah, the 20 minute reboot is why I really would want to fix IDRAC properly, with that long of a reboot using this r420 I have as a router is really not feasible.
I have not yet used any ethernet add on cards with an arduino but I did find an FTP server project with a quick google, so it should be doable, you would simply loose port 1 of your motherboard since it would be dedicated to the arduino’s ethernet connection. Assuming it hosts the file fine and u-boot has no problem connecting then the arduino could interrupt the boot process and send all the commands to load from ftp when everything powers up.
It would definitely need testing to see how smooth it all goes, in terms of timing for things like the ftp server to get going and for the file to download to the server but it should totally be doable.
One thing is that you don’t need to power on the server at all as u-boot starts as soon as the board gets power so you need to find a spot to power the arduino from that’s a standby rail. Arduinos take ~8 seconds to fully boot iirc, so that needs to be factored in in terms of everything getting going but after that it could immediately bull the SW2 contact to ground to trip it and start the process.
Alright, so gave it a shot using the TSOP package, TDLR it didn’t work but I got some new output on my serial console. I’m not sure if there’s a way to change the target from e-mmc to u-nand for IDRAC but if you can then I think using the tsop instead might be a viable option. Based on how it’s in the same spot as the EMMC it seems like the idrac system would be able to access it too.
https://pastebin.com/uLXPwQPP
https://www.youtube.com/watch?v=V0l8mT_ZsRA
edit: I was super freaking tired when I was doing this, and I can’t recall if I tried manually typing in the recover command specifying the nand flash after I finished recording actually, so once I get some sleep, or copious amounts of caffeine I’ll give that a shot. Maybe even if I couldn’t spot a command to check the u_nand chip in idrac, manually specifying it for recovery might work.
Of course thanks to the clunky iron tip I have there might be some poor solder joints or something which is why I was hoping to probe the size of the nand flash somehow, but that might also be something I missed due to being up for, well I don’t even know right now, 20 ish hours now with only a few hours of sleep before.
edit 2: shit, yeah flinfo might be what I needed to use, I think when I tried erasing flash with the EMMC chip before that failed. I’m going to wait before trying out more though since I’m liable to mess up something right now.
Hello everyone, I have a similar problem on my dell T620 server, message "Initializing iDRAC", I want to try to start it and update it from an SD memory, but can someone tell me what type of memory it is exactly, I mean (size, model)? with a serious image very helpful
Or if anyone can help me, I am desperate, thank you very much.
hey guys I am also having similar problem with idrac on my r720. I can’t seem to be able to stop uboot and the uart goes into a linux mode. I was wondering if there was a way from there to try to flush the emmc.
This is the bulk of my error:
WDT2: Disabled
Initialization complete, proceed with boot
OS_WDT0_EN:180 seconds, Re-load ILRAM:
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image … OK
## Booting kernel from Legacy Image at 88000000 …
Image Name: sh-linux
Image Type: SuperH Linux Kernel Image (gzip compressed)
Data Size: 2537538 Bytes = 2.4 MB
Load Address: 80001000
Entry Point: 80002000
Verifying Checksum … OK
Uncompressing Kernel Image … OK
WDT2: Disable in do_bootm_linux()
sh_eth_init
Monolithic/DRB
init started: BusyBox v1.18.4 (2012-11-04 19:32:12 CST)
starting pid 407, tty ‘’: '/etc/init.d/rcS’
waitfor: pid 407 …
setting net.ipv6.conf.default.accept_dad based on kernel bootup option
enable normal accept DAD for ipv6
Sat Jan 1 00:00:01 UTC 2000 To execute command source /etc/sysapps_script/I_1000_mountfs.sh
Sat Jan 1 00:00:02 UTC 2000 To execute command source /etc/sysapps_script/I_1001_trigwdt.sh
<========= Start H/W watchdog first… =========>
Sat Jan 1 00:00:02 UTC 2000 To execute command source /etc/sysapps_script/I_1100_nfs.sh
Sat Jan 1 00:00:02 UTC 2000 To execute command source /etc/sysapps_script/I_1105_ps.sh
XXXXXX /dev/mmcblk0p11
/dev/mmcblk0p11: recovering journal
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to re-open /dev/mmcblk0p11
e2fsck: io manager magic bad!
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open /dev/mmcblk0p11
Could this be a zero-length partition?
mount: mounting /dev/mmcblk0p11 on /flash/data0 failed: Invalid argument
mounting private storage fail, formatting partition
mke2fs 1.41.14 (22-Dec-2010)
Warning: could not erase sector 2: Attempt to write block from filesystem resulted in short write
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
1024 inodes, 4096 blocks
204 blocks (4.98%) reserved for the super user
First data block=1
Maximum filesystem blocks=4194304
1 block group
8192 blocks per group, 8192 fragments per group
1024 inodes per group
Hey guys,
I managed to interrupt uboot and trying to boot from tftp on my r720.
I manage to download the firmimg.d7 from tftpboot, but when I try to run:
go addr 0x81000000
Uboot restarts and it loads the old firmware therefore failing.
Any suggestions on how I can successfully boot from memory after I download the firmware via tftp?
Sorry dup entry
I have 2x R630 motherboard with iDRAC error. =(
I can access iDAR UART, but cant interrupt uBOOT process (upgraded to latest iDRAC version 2.70.70).
Any help are welcome.
Motherboard 1 log:
U-Boot 2009.08 (Oct 18 2019 - 05:57:47) Avocent (0.0.3) EVB
CPU: SH-4A
BOARD: R0P7758LC00xxRL (A0 step) board
BOOT: Secure, HRK generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
Undefined env FWN
Undefined env FWN1
Undefined env fwu_act
Undefined env Failsafe_PowerON_Lockout
Undefined env wdt2cnt
Undefined env wdt0cnt
Undefined env cpld_recover_cnt
Undefined env ddr_recover_cnt
Restore env etherc0=74:E6:E2:FB:D9:7A
Restore env getherc0=74:E6:E2:FB:D9:7B
Restore env first_ac=yes
Restore env ethaddr=74:E6:E2:FB:D9:7A
Restore env eth2addr=74:E6:E2:FB:D9:7B
Restore env macs_in_res1=yes
Restore env ethact=sh_g_eth.0
Restore env dmn=disabled
In: serial
Out: serial
Err: serial
PCIe: Bridge loaded with 0x18000 bytes
WDT2: Booted Lower Vector, 'uboot1’
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:004 U-Boot 2009.08 (Oct 18 2019 - 05:57:47) Avocent (0.0.3) EVB
INFO: 00:007 U-Boot checkin date(10-04-2018) Version(1.1.34)
INFO: 00:009 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB MX25L3206E bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31 Minor Ver=0x20 D0 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
INFO: 00:139 BCM54610 OUI=0x00d897 Model=0x26 Revision=0x0a PhyAddr=1
INFO: 00:007 Dedicated monolithic mgmt NIC enabled (vendor mode override)
INFO: 00:256 SD CARD: Device: sh-sdhi Manufacturer ID: 41 OEM: 3432
Name: DELL1 Tran Speed: 25000000 Rd Blk Len: 512
SD ver 2.0 HC: Yes cap(<2gb): 3896508416 cap(>2gb): 0
ext_rev: 0 rst_n: 0 ext[96]: 0
INFO: 00:187 EMMC: Device: sh_mmcif Manufacturer ID: 45 OEM: 100
Name: SEM04 Tran Speed: 25000000 Rd Blk Len: 512
MMC ver 4.0 HC: Yes cap(<2gb): 0 cap(>2gb): 7733248
ext_rev: 6 rst_n: 1 ext[96]: 0
INFO: 00:024 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x1
Planar: Type=0x58 Rev=0x9 Rework=0x0 Scratch/PathRetry=0x00
PASS: 00:013 Coin cell detected good, AD=0x3a3 low water=0x2c1
PASS: 00:008 PCIe SH7758_A0 Ver=0.03 MCTP en, CRC=0x70ae6992 @0x8efd54f4 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=vhard
INFO: 00:002 reset_cause=ac
PASS: 00:005 Booted Lower Vector, ‘uboot1’ wdt2cnt=0
INFO: 00:005 wdt0cnt=0 cpld_recover_cnt=0 ddr_recover_cnt=0
PASS: 00:007 Clear CH1/CH2, clear 4K shared memory@0xffcaa000 on AC power up
PASS: 00:007 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x6f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=74:E6:E2:FB:D9:7A
getherc0=74:E6:E2:FB:D9:7B
INFO: 00:009 Fan logic for monolithic planar type 88
fan1 - def 0000 1fff 3d fan2 - def 0000 1fff 3d
fan3 - def 0000 1fff 3d fan4 - def 0000 1fff 3d
fan5 - def 0000 1fff 3d fan6 - def 0000 1fff 3d
fan7 - def 0000 1fff 3d fan8 - def 0000 1fff 3d
INFO: 00:077 Env and backup CRC’ed ok
Erasing SPI flash at 0x100000…Writing to SPI flash…done
Erasing SPI flash at 0x110000…Writing to SPI flash…done
*** no text signature found
INFO: 01:934 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:255 Current u-boot1 1.1.34 verified with ‘ubootN1’
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x1
U-boot2 in sync with u-boot1 1.1.34
INFO: 00:018 Verify OS Images N-1: ENV FWN1 is missing
PASS: 00:005 Boot device=emmc Boot partition5/N-1
Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 03:100
OS_WDT0_EN:180 seconds, Re-load ILRAM:
SPI_WP: Successful.
EMMC_WP: Successful.
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image … OK
## Booting kernel from Legacy Image at 88000000 …
Image Name: sh-linux
Image Type: SuperH Linux Kernel Image (gzip compressed)
Data Size: 3082347 Bytes = 2.9 MB
Load Address: 80001000
Entry Point: 80002000
Verifying Checksum … OK
Uncompressing Kernel Image … OK
WDT2: Disable in do_bootm_linux()
Monolithic/DRB
systemd[1]: could not find module by name='autofs4’
systemd[1]: Failed to insert 'autofs4’
display_factory_info.sh[1737]: cat /etc/fw_ver
display_factory_info.sh[1737]: 2.70.45.70.70
display_factory_info.sh[1737]: 2.70.70.70.138
display_factory_info.sh[1737]: built for - system
display_factory_info.sh[1737]: Fri Oct 18 06:34:04 CDT 2019
display_factory_info.sh[1737]: 191018063404
display_factory_info.sh[1737]: IDRAC SVN Rev =
display_factory_info.sh[1737]: Hudson Project = idrac-13g-ducati1.8-master-release-A-Rev
display_factory_info.sh[1737]: Hudson Build # = 819
display_factory_info.sh[1737]: Release ID =
display_factory_info.sh[1737]: MSC Revision =
display_factory_info.sh[1737]: cat /proc/cmdline
display_factory_info.sh[1737]: root=/dev/mmcblk0p6 rootwait rw rootfstype=squashfs mem=239616k console=ttyS2,115200 <NULL> mac1=74:E6:E2:FB:D9:7A mac2=74:E6:E2:FB:D9:7B mode=vhard reset_cause=ac nmi_buf=0x83000000 quiet console=ttySC2,115200 init=/sbin/init
display_factory_info.sh[1737]: cat /tmp/loguboot
display_factory_info.sh[1737]: cat: can’t open ‘/tmp/loguboot’: No such file or directory
display_factory_info.sh[1737]: /bin/MemAccess -ww ffec0030 5500
display_factory_info.sh[1737]: + 0 2 4 6 : 8 A C E
display_factory_info.sh[1737]: ------------------------------------------------------
display_factory_info.sh[1737]: 0xffec0030 = 5500 0000 6400 8000 : ff00 0000 8f00 0800
display_factory_info.sh[1737]: 0xffec0040 = 3200 0e00 1f00 6b00 : fe00 2400 ff00 2100
display_factory_info.sh[1737]: ^
display_factory_info.sh[1737]: /bin/MemAccess -rb 0x14000000
display_factory_info.sh[1737]: + 0 1 2 3 4 5 6 7 : 8 9 A B C D E F
display_factory_info.sh[1737]: --------------------------------------------------------------
display_factory_info.sh[1737]: 0x14000000 = 01 00 01 05 08 00 00 00 : 09 00 00 00 01 00 06 08
display_factory_info.sh[1737]: 0x14000010 = 08 08 00 00 05 00 04 00 : 00 04 0e 0f 07 0e 0c 0f
display_factory_info.sh[1737]: 0x14000020 = 0d 07 0f 0f 03 00 08 00 : 00 00 00 0f 08 00 01 01
display_factory_info.sh[1737]: 0x14000030 = 00 00 00 00 00 00 00 08 : 00 01 00 00 00 0f 0c 00
Motherboard 2 log:
U-Boot 2009.08 (Feb 21 2020 - 03:41:43) Avocent (0.0.3) EVB
CPU: SH-4A
BOARD: R0P7758LC00xxRL (A0 step) board
BOOT: Secure, HRK generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
Undefined env FWN
Undefined env FWN1
Undefined env fwu_act
Undefined env Failsafe_PowerON_Lockout
Undefined env wdt2cnt
Undefined env cpld_recover_cnt
Undefined env ddr_recover_cnt
Restore env etherc0=74:E6:E2:FB:CD:E6
Restore env getherc0=74:E6:E2:FB:CD:E7
Restore env first_ac=yes
Restore env ethaddr=74:E6:E2:FB:CD:E6
Restore env eth2addr=74:E6:E2:FB:CD:E7
Restore env macs_in_res1=yes
Restore env ethact=sh_g_eth.0
Restore env dmn=disabled
Restore env wdt0cnt=31
In: serial
Out: serial
Err: serial
PCIe: Bridge loaded with 0x18000 bytes
WDT2: Booted Lower Vector, 'uboot1’
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:005 U-Boot 2009.08 (Feb 21 2020 - 03:41:43) Avocent (0.0.3) EVB
INFO: 00:006 U-Boot checkin date(10-04-2018) Version(1.1.34)
INFO: 00:009 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB MX25L3206E bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31 Minor Ver=0x20 D0 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
INFO: 00:140 BCM54610 OUI=0x00d897 Model=0x26 Revision=0x0a PhyAddr=1
INFO: 00:007 Dedicated monolithic mgmt NIC enabled (vendor mode override)
INFO: 00:125 SD CARD: Device: sh-sdhi Manufacturer ID: 41 OEM: 3432
Name: SD1GB Tran Speed: 25000000 Rd Blk Len: 512
SD ver 2.0 HC: No cap(<2gb): 1023934464 cap(>2gb): 0
ext_rev: 0 rst_n: 0 ext[96]: 0
INFO: 00:096 EMMC: Device: sh_mmcif Manufacturer ID: 90 OEM: 14a
Name: H4G1d Tran Speed: 25000000 Rd Blk Len: 512
MMC ver 4.0 HC: Yes cap(<2gb): 0 cap(>2gb): 7634944
ext_rev: 6 rst_n: 1 ext[96]: 0
INFO: 00:024 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x1
Planar: Type=0x58 Rev=0x9 Rework=0x0 Scratch/PathRetry=0x00
PASS: 00:014 Coin cell detected good, AD=0x3a3 low water=0x2c1
PASS: 00:008 PCIe SH7758_A0 Ver=0.03 MCTP en, CRC=0x70ae6992 @0x8efd54f4 cnt=0x18000
INFO: 00:007 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=vhard
INFO: 00:002 reset_cause=ac
PASS: 00:005 Booted Lower Vector, ‘uboot1’ wdt2cnt=0
INFO: 00:005 wdt0cnt=31 cpld_recover_cnt=0 ddr_recover_cnt=0
PASS: 00:007 Clear CH1/CH2, clear 4K shared memory@0xffcaa000 on AC power up
PASS: 00:007 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x6f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:022 etherc0=74:E6:E2:FB:CD:E6
getherc0=74:E6:E2:FB:CD:E7
INFO: 00:009 Fan logic for monolithic planar type 88
fan1 - def 0000 1fff 3d fan2 - def 0000 1fff 3d
fan3 - def 0000 1fff 3d fan4 - def 0000 1fff 3d
fan5 - def 0000 1fff 3d fan6 - def 0000 1fff 3d
fan7 - def 0000 1fff 3d fan8 - def 0000 1fff 3d
INFO: 00:077 Env and backup CRC’ed ok
Erasing SPI flash at 0x100000…Writing to SPI flash…done
Erasing SPI flash at 0x110000…Writing to SPI flash…done
no text signature found
INFO: 02:487 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:261 Current u-boot1 1.1.34 verified with ‘ubootN’
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x1
U-boot2 in sync with u-boot1 1.1.34
INFO: 00:017 Verify OS Images N: ENV FWN is missing
PASS: 00:005 Boot device=emmc Boot partition1/N
Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 03:438
OS_WDT0_EN:180 seconds, Re-load ILRAM:
SPI_WP: Successful.
EMMC_WP: Successful.
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image … OK
## Booting kernel from Legacy Image at 88000000 …
Image Name: sh-linux
Image Type: SuperH Linux Kernel Image (gzip compressed)
Data Size: 3082342 Bytes = 2.9 MB
Load Address: 80001000
Entry Point: 80002000
Verifying Checksum … OK
Uncompressing Kernel Image … OK
WDT2: Disable in do_bootm_linux()
Monolithic/DRB
mmcblk0: r/w command failed, status = 0x100900
end_request: I/O error, dev mmcblk0, sector 262144
Buffer I/O error on device mmcblk0, logical block 32768
mmcblk0: r/w command failed, status = 0x100900
end_request: I/O error, dev mmcblk0, sector 247559
SQUASHFS error: squashfs_read_data failed to read block 0x68e0d0e
SQUASHFS error: unable to read id index table
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
CPU: 0 PID: 1 Comm: swapper Not tainted 3.16.47 #1
Stack: (0x8d03feb4 to 0x8d040000)
fea0: 80341066 8d300000 80426264
fec0: 8d03fee4 804e2e84 80098f48 804fa8a4 80502008 80718000 00008000 8d300000
fee0: 80341514 8d03fefc 00000001 00000001 00000000 fffffffb 804262cc 6e6b6e75
ff00: 2d6e776f 636f6c62 3731286b 29322c39 ffffff00 8d018990 8c00b900 804e3098
ff20: 800021f0 8050202c 80502008 804ffeec 804fa85c 0000007d 0b300002 ffffff9c
ff40: 00000000 800021f0 804e3212 804ffeec 804fa85c 0000007d 80502038 804ffeec
ff60: 804e2b22 0000007d 80096170 00000007 00000007 804e21e8 fbffffff 803406ae
ff80: 8d041ef8 804c1f9c 804c3f8c 0000043c 804c1f5c 8d02e9e0 8002b4cc 800081e8
ffa0: 8002b4cc 00000000 00000000 00000000 00000000 00000000 803406a4 00000000
ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffe0: 00000000 00000000 00000000 40008000 00000000 00000000 00000000 00000000
Call trace:
[<804e2e84>] mount_block_root+0x180/0x20c
[<80098f48>] SyS_mount+0x0/0xb4
[<80341514>] printk+0x0/0x2c
[<804e3098>] mount_root+0xb4/0x108
[<800021f0>] do_one_initcall+0x0/0x1a4
[<800021f0>] do_one_initcall+0x0/0x1a4
[<804e3212>] prepare_namespace+0x126/0x1c0
[<804e2b22>] kernel_init_freeable+0x106/0x1ac
[<80096170>] SyS_dup+0x0/0x50
[<804e21e8>] repair_env_string+0x0/0x68
[<803406ae>] kernel_init+0xa/0xe4
[<8002b4cc>] schedule_tail+0x0/0x54
[<800081e8>] ret_from_kernel_thread+0xc/0x14
[<8002b4cc>] schedule_tail+0x0/0x54
[<803406a4>] kernel_init+0x0/0xe4
—[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
Motherboard 2 log (iDRAC recovery reboot):
U-Boot 2009.08 (Feb 21 2020 - 03:41:43) Avocent (0.0.3) EVB
CPU: SH-4A
BOARD: R0P7758LC00xxRL (A0 step) board
BOOT: Secure, HRK generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
Undefined env FWN
Undefined env FWN1
Undefined env fwu_act
Undefined env Failsafe_PowerON_Lockout
Undefined env wdt2cnt
Undefined env cpld_recover_cnt
Undefined env ddr_recover_cnt
Restore env etherc0=74:E6:E2:FB:CD:E6
Restore env getherc0=74:E6:E2:FB:CD:E7
Restore env first_ac=yes
Restore env ethaddr=74:E6:E2:FB:CD:E6
Restore env eth2addr=74:E6:E2:FB:CD:E7
Restore env macs_in_res1=yes
Restore env ethact=sh_g_eth.0
Restore env dmn=disabled
Restore env wdt0cnt=37
In: serial
Out: serial
Err: serial
PCIe: Bridge loaded with 0x18000 bytes
WDT2: Booted Lower Vector, 'uboot1’
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:005 U-Boot 2009.08 (Feb 21 2020 - 03:41:43) Avocent (0.0.3) EVB
INFO: 00:006 U-Boot checkin date(10-04-2018) Version(1.1.34)
INFO: 00:009 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB MX25L3206E bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31 Minor Ver=0x20 D0 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
INFO: 00:140 BCM54610 OUI=0x00d897 Model=0x26 Revision=0x0a PhyAddr=1
INFO: 00:007 Dedicated monolithic mgmt NIC enabled (vendor mode override)
INFO: 00:125 SD CARD: Device: sh-sdhi Manufacturer ID: 41 OEM: 3432
Name: SD1GB Tran Speed: 25000000 Rd Blk Len: 512
SD ver 2.0 HC: No cap(<2gb): 1023934464 cap(>2gb): 0
ext_rev: 0 rst_n: 0 ext[96]: 0
INFO: 00:096 EMMC: Device: sh_mmcif Manufacturer ID: 90 OEM: 14a
Name: H4G1d Tran Speed: 25000000 Rd Blk Len: 512
MMC ver 4.0 HC: Yes cap(<2gb): 0 cap(>2gb): 7634944
ext_rev: 6 rst_n: 1 ext[96]: 0
INFO: 00:024 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x1
Planar: Type=0x58 Rev=0x9 Rework=0x0 Scratch/PathRetry=0x0f
PASS: 00:014 Coin cell detected good, AD=0x3a3 low water=0x2c1
PASS: 00:008 PCIe SH7758_A0 Ver=0.03 MCTP en, CRC=0x70ae6992 @0x8efd54f4 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:005 mode=vhard
INFO: 00:003 reset_cause=recover
PASS: 00:005 Booted Lower Vector, ‘uboot1’ wdt2cnt=0
INFO: 00:005 wdt0cnt=37 cpld_recover_cnt=0 ddr_recover_cnt=0
PASS: 00:006 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x6f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:022 etherc0=74:E6:E2:FB:CD:E6
getherc0=74:E6:E2:FB:CD:E7
INFO: 00:007 Fan logic is not modified on Non-AC power up
INFO: 00:053 Env and backup CRC’ed ok
no text signature found
INFO: 00:801 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:072 Current u-boot1 1.1.34 verified with ‘ubootN1’
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x1
INFO: 00:013 Verify OS Images N-1: ENV FWN1 is missing
FAIL: 00:005 Boot device=emmc Boot partition5/N-1
Boot Path Retry:P1/N=3 P5/N-1=3 Recover
INFO: 00:000 03:305
OS_WDT0_EN:180 seconds, Re-load ILRAM:
SPI_WP: Successful.
EMMC_WP: Successful.
Boot Path Retry:P1/N=3 P5/N-1=3 Recover
fail to boot image ***
Unable to boot, disable wdogs – HALTING
Thanks!
To interrupt boot, you must to shortcut 2 pins on mb. I soldered there 2 wires on made a "jumper". Read this thread, there was instructions.
You should be able to change the target. But, you need working emmc to store the target value upon iDRAC boot up. I had considered using Pi as a TFTP server but the iDRAC emmc refuses to write the setting.
Let me know if you made any further progress. I had already decided to desolder the emmc but hadn’t got around to buying the new chip or disassembling.
Thank-you to everyone in this thread, I have been able to recover the iDRAC on my R420 to 1.66.65 after a bad update to 2.21.21 from 1.57.57. In my case the EMMC was working fine it was just complaining that it couldn’t contact the BMC.
I soldered pins onto the BMC_UART and used a TTL to USB adapter to connect at 115200. I was able to reboot the idrac just by tapping the “ON” pin of SW2 pair 1. THen I held a bent paperclip to pair 2 of SW2 and rebooted the idrac and watched for “press any key” to interrupt the boot. This left me at a IDRAC8=> prompt which seemed wrong.
I then used the info in @willard post on page 4 to recover from 1.66.65 firmimg.d7 on the SD card with util recover -emmc -from sd -f firmimg.d7
I noticed that the U-Boot header changed from
U-Boot 2009.08 (Nov 04 2015 - 11:28:55) Avocent (0.0.3) EVB
to
U-Boot 2009.08-00088-g121cddc (Nov 17 2014 - 05:50:46) Avocent (0.0.3) EVB, Build: jenkins-idrac-yocto-release-505
Here’s a pastebin of my terminal log, starting with a reboot when I missed jumping pair 2 of SW2 (you can see that it booted completely) followed by a second boot where I got the “Hit any key to stop autoboot” and hit a key in time.
Then you’ll see a few failed attempts to tftp, eventually just going with the util recover command.
pastebin.com HUwLWjbp
I’m not sure why the tftp failed, it succeeded in a previous attempt. On that successful attempt, when I typed the go 0x81000000 command from @ldv post on page 6, it immediately booted the emmc firmware, not the one downloaded from tftp. I never got to the bootm step.
I’m even more grateful because this is my second motherboard. The first one I ran a firmware_erase_all command of some sort thinking I could erase the mmc and trigger the recovery procedure, but it apparently deleted U-Boot! That motherboard doesn’t do anything now. I think I’ll have to flash a chip somehow, I don’t have those tools or skills though.
I also wanted to mention that the pins I soldered onto the motherboard were pulled from a junk system’s i/o header (the pins that connects to cables that run up to the power button, led, etc.) This worked because my ttl-usb had compatible ends on it already and I just pushed them on.
imgur.com /a/bLmA3K4
Note: I was following a post titled “iDRAC7 Upgrade Path” on the Dell Community forums to do the updates.
1. BIOS 2.0.21
2. iDrac 1.57.57
3. BIOS 2.1.3
4. iDrac w/ Lifecycle Controller 2.21.21 (the updates merged after this point)
5. BIOS 2.4.2
6. iDrac w/ LCC 2.40.40
7. BIOS 2.5.1
8. iDrac w. LCC 2.63.60.62. (latest)
9. BIOS 2.6.0 (latest)
I was successful up to step 3, and step 4 is when it failed. Maybe I needed to goto 1.66.65 first, or only go to 2.10.10 before 2.21.21. Now that I look at it, I think the power only lists the “Urgent” updates.
Update: I’ve had to do this a few more times, I can’t seem to get 2.10 installed. Updated BIOS to 2.3.3 and 2.4.2 and iDRAC with LifeCycle Controller still fails.
I’ve also noticed that the SD card can’t be full size as that causes the util recover to fail to find firmimg.d7.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
DISKPART> list disk
Disk ### Status Size Free Dyn Gpt
-------- ------------- ------- ------- --- ---
Disk 0 Online 931 GB 1024 KB *
Disk 1 Online 14 GB 0 B
DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> clean
DiskPart succeeded in cleaning the disk.
DISKPART> create partition primary size=1000
DiskPart succeeded in creating the specified partition.
DISKPART> format fs=fat32 quick
100 percent completed
DiskPart successfully formatted the volume.
DISKPART> assign letter d
DiskPart successfully assigned the drive letter or mount point.
DISKPART> list volume
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 Windows RE NTFS Partition 499 MB Healthy
Volume 1 C OSDisk NTFS Partition 929 GB Healthy Boot
Volume 2 NTFS Partition 552 MB Healthy
Volume 3 FAT32 Partition 512 MB Healthy System
* Volume 4 D FAT32 Removable 1000 MB Healthy
DISKPART> exit
Leaving DiskPart...
I’m unable to get any v2 release to work on my motherboards. Each one is unable to communicate with the rest of the system with v2. It can’t read the service tag, temperatures, fans, or anything else. v1.66.65.65 is the last version that works for me.
I especially dislike v2.63 because it (or a predecessor) removes the interactive shell on the serial console (I believe that was done to address a CVE)
It seems that all my hardware is ok, the BIOS chip and the eMMC are functioning.
If I had to do another R420 recovery:
1. Solder the motherboard
- imgur.com a/QWj7GN8
1a. Pins for the serial console
- I get these pins by removing them from old hardware
1b. Solder a bridge across SW2.2 (Even when the system is working this only adds 3 seconds to idrac boot time). I used a 8 inch length of stripped Cat 5 with the end bent to the width of the SW2.2 solder pads. Tinned and then soldered, then the excess trimmed off.
3. Connect TTL Serial to USB and identify the com port via Device Manager
- (Mine is an old fake PL2303A and Device manager lists it as “PL2303HXA phased out since 2012 please contact your supplier” and I found v3.3.2.105 that works in a file called “IO-Cable_PL-2303_Drivers-Generic_Windows_PL2303_Prolific.zip”. I need to “update” to the old driver every boot.)
4. Launch PuTTY and connect to the serial port at 115200.
5. Connect power and monitor serial for data (trial/error to get tx/rx/gnd correct)
5a. If the server auto powers on, hold the power button until it turns off.
6. Review the serial output for signs of hardware failure, I/O errors etc.
7. If the IDRAC o/s boots, see if I get a console, otherwise goto 11 below.
8. Erase the eMMC from the console with dd if=/dev/zero of=/dev/mmcblk0 bs=1M
8a. Hopefully no errors, otherwise eMMC has likely failed.
8b. The dd command will eventually trigger a reboot into recovery mode
9. Recovery mode will watch for an SD Card with firmimg.d7 extracted from 1.66.65.65 on it. But the volume on the SD card can’t be too big. i.e. It won’t read from a 16-GB volume.
9a. Monitor the recovery for write failures to the eMMC, those will indicate it’s probably dead, but you can still run iDRAC from a tftp download, but that’s something you need to do every power cycle.
10. If the write is successful it’s probably be good with 1.66.65.65.
11. If the serial doesn’t give a console prompt, then I’ll disconnect power and get out the CH341A. Using “CH341A Programmer v1.34”, write a 1.66.65.65 U-Boot mega.nz folder/aQc3GIhI#Rgvu-_2ZPpPfUOEIoYanGg
12. Connect power and watch for “press a key to interrupt autoboot”. If you miss this I think U-Boot will update itself from the EMMC and you won’t be able to interrupt it again and make you go back to step 11.
13. Interrupting U-Boot should leave you at an IDRAC7=> or IDRAC8=? prompt. With the same SD card as step 9 inserted;
13a. iDRAC8=> protect off all
13b. iDRAC8=> util recover -emmc -from sd -f firmimg.d7
14. If the eMMC is functioning, that should get 1.66.65.65 onto the eMMC and you should be good with 1.66.65.65.
Server Information
Power State: OFF
System Model: PowerEdge R420
System Revision: I
System Host Name: HV-C1C6CZ1
Operating System:
Operating System Version:
Service Tag: C1C6CZ1
Express Service Code: 26202306349
BIOS Version: 2.4.2
Firmware Version: 1.66.65 (Build 07)
IP Address(es): 192.168.137.198
iDRAC MAC Address: 90:B1:1C:35:19:86
DNS Domain Name:
Lifecycle Controller Firmware: 2.63.60.62 (!)
IDSDM Firmware Version: N/A
I’ve been unable to downgrade the Lifecycle controller to a matching version, e.g. 1.4.2
I’ve done this with two R420 motherboards with functioning eMMC, and the thread has instructions on how to tftp boot the iDRAC if the eMMC has failed.
My guess is that there’s some other component on my motherboards that need to be upgraded before iDRAC v2 will work. It’s not the BIOS as I’m on a fairly recent BIOS 2.4.2
Just for fun I downloaded the Platform Specific Bootable ISO from here dell.com support/article/en-ca/sln296511/update-poweredge-servers-with-platform-specific-bootable-iso?lang=en for the R420 dell.com support/home/en-ca/drivers/driversdetails?driverId=MRRRF&oscode=naa&productcode=poweredge-R420
It updated my NetXtreme BCM5720 to 21.40.21 from 7.2.0, then the BIOS to 2.9.0 from 2.4.2 and then iDRAC-with-Lifecycle-Controller to 2.65.65.65, and of course, broke idrac.
My first recovery attempt was to login to the web console (which shows null for the platform) but that just crashed/restarted iDRAC (possibly triggered a watchdog? It’s very slow) or I didn’t wait long enough for the system to stabilize. I’ve also seen iDRAC reject passwords if I tried to login too quick.
I didn’t lose my Enterprise license this time and I was able to ssh into idrac and use the racadm fwupdate -g -u -a 192.168.137.1 command to downgrade to 1.66.65.65. (I was using tftpd32 as my tftp server, not sure what’s up with Chrome and Firefox saying the site has malware, maybe because rogue dhcp servers are a problem?)
ssh log
pastebin.com embed_js/7u11E6dB
and the serial console log
pastebin.com embed_js/akkf06Y9
So if someone else “bricks” their idrac, see if it’s still getting an IP address and try ssh & tftp, or http, to do a downgrade.
That was a fairly easy downgrade and it looks like everything is working. My main reason for v2 was HTML virtual console, but I can live with gist.github .com xbb/4fd651c2493ad9284dbcb827dc8886d6
I wonder what component is stopping iDRAC v2 from working?
Edit: The platform specific bootable iso didn’t actually update the bios. It added a job, which ran after I completed the downgrade and rebooted.
Some one tried to connect to idrac using USB?
https://www.dell.com/support/manuals/en-…cb93&lang=en-us
https://www.manualslib.com/manual/124429…8.html?page=269
or this way
https://github.com/kamermans/docker-openmanage/issues/4
So I’m another user that got hit by this. Did the same thing as OP, upgraded from too old of version to 2.30.x and it bricked my iDRAC.
Running the commands here in iDRAC7=> prompts give the same results as others have seen Failure writing blocks [1:196609] RC=0xffffffac
gist DOT github DOT com SLASH vaskokj SLASH de70dd5241df64dceb4710fa9cf0b2a9 - I can’t post links as I’m a new member.
It seems that the only person that has had success fixing this by replacing the chip is @thewooflife ?
I assume that once we see the Failure writing blocks [1:196609] RC=0xffffffac when trying to run util recover -emmc -from sd -f firmimg.d7 the eMMC is bad and just had to be replaced and no other solution exists to “revive” it that I might have missed? If so I’m ordering a hot air rework station and the chip that @thewooflife ordered and going to see if I can replace it.
@willard @ldv @XOIIO did any of yall replace your eMMC chip and have any luck?
The only other solutions seem to be “temporary” fixes.
1) tftp solution mentioned here by @ribs
2) booting off sd card? I couldn’t find proper solution for this, just people speculating it might work? Unless I missed it?
I think this thread has people discussing two different iDRAC issues:
1) iDRAC failure during flash/upgrade
2) iDRAC failure due to worn-out eMMC
I believe #1 can be fixed via serial commands, but #2 requires eMMC chip replacement. I don’t think anybody has done #2 successfully yet
Well I was doing an upgrade to the iDRAC when it "failed". My errors seem to be consistent with the failed eMMC (but was asking my question to validate). There was speculation that doing too large of jump from 1.x to 2.x would cause it to brick, which is what I was doing. I just have the errors suggesting the memory is defective. That is one hell of a coincidence if that is the case though. I can’t seem to get the eMMC to take the firmimg.d7 file no matter what I do. I do believe @thewooflife said they replaced their eMMC and it worked (post #111 here, sorry can’t create links yet, new to the boards).
First off, I am new here. I have read the whole thread and am fairly well versed in hardware design. You all have pulled some good info together.
I agree that at least 2 problems are being discussed here, and the 1) above may even have some different failure modes depending on what was being done with things bricked.
At present I a R620 with a non responsive idrac7. I have added the serial header and done tbe A2/B2 jumper. This system gets past Uboot and fails while doing the linux system init with what looks like corrupted file systems. I have been doing some very careful poking around seeing what I can learn about its issues and about the advice being given here.
I also have an R720 with a bricked idrac7 but have not done the serial header or jumper. that shall wait tell I get further with the R620.
I have lots of questions, and hopefully can update some of what is posted here. I believe that many things are missing very important details and it is the reason that what seems to work for one person is not working for another, especially around the booting of an idrac7 via tftp. I have tried to do what is described as:
1
2
2
tftpboot firmimg.d7
go addr 0x81000000
that fails for me as it has for others with you just ending up back in uboot. And for whoever has made this work I would like to get some additional details from you. In an effort to sort out what is going on here I have started in on analysis of the 1.57.57 firmimg.d7 file. It is a uImage with many things inside and it is very unlikely that the entry address would be the load address as everyone is stating in this thread, but more likely at some later offset. Using binwalk on the image we get this info:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
binwalk -v firmimg.d7
Scan Time: 2020-12-26 23:44:16
Target File: payload/firmimg.d7
MD5 Checksum: 7b30eadf0c00acb45b657dada2d102a8
Signatures: 344
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
512 0x200 uImage header, header size: 64 bytes, header CRC: 0x59CF8328, created: 2014-04-30 18:46:08, image size: 2540437 bytes, Data Address: 0x80001000, Entry Point: 0x80002000, data CRC: 0x4C797D36, OS: Linux, CPU: SuperH, image type: OS Kernel Image, compression type: gzip, image name: "sh-linux"
576 0x240 gzip compressed data, maximum compression, has original file name: "vmlinux.bin", from Unix, last modified: 2014-04-30 18:46:07
2541056 0x26C600 Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 59577844 bytes, 4811 inodes, blocksize: 1048576 bytes, created: 2014-04-30 18:47:06
62305916 0x3B6B67C CRC32 polynomial table, little endian
62336632 0x3B72E78 PGP armored data,
62336676 0x3B72EA4 PGP armored data,
62481920 0x3B96600 Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 467244 bytes, 613 inodes, blocksize: 1048576 bytes, created: 2014-04-30 18:47:06
62952960 0x3C09600 PGP armored data,
my conclusion is that the uboot "go" command is simply the wrong way to start this loaded image file, but "bootm" probably is the right way. From some other reading I found this more complete bootm command:
1
=> bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr}
I have also figured out the process for getting the firmimg.d7 extracted and written to the SD card using the fwu utility, but I have not been able to boot from it yet, though I suspect it is a matter of setting the correct uboot env bootcmd string and as another pointed out modifying a few mmcblk0 values in files inside the squashfs. This could be a good long term solution to a failed emmc chip.
For now that is all, if anyone who has this working wants to collaborate on getting the tftpbooting of an idrac7 documented so more might get past this hurtle I am here to help too.
Doesn’t the tftpboot require network access? How are you getting a valid IP address that will actually respond on the network at all? I can set an IP address but it won’t respond to anything on the network.
Have you tried the boot from SD card method?