Hey there, sorry for late reply. If you can enable ReBar or Above 4G Decode then your problem is completely different than mine. I haven’t found a solution to my problem yet.
Not sure if this is repeatable, but with your recent response I wanted to at least mention my recent success with linux and apparently enabling “Above 4G Decoding” and ReBar for my Gigabyte Z97mx-gaming5 and Nvidia Tesla P40. References/links at bottom of post.
Similar to your posted experience, “Above 4G Decoding” appeared in the bios strings but I saw no hidden options to enable when using either UEFITool/IRFExtractor or amibcp 4.55.0070.
In a last-ditch effort, I figured I’d try the DSDT Patching [1]. Instead of immediately flashing the bios I attempted to load the DSDT.aml using grub. Instead of using the grub boot menu edit mentioned in another thread [2], I attempted the more permanent route using the “01_acpi” file in grub.d.
Initially ubuntu simply failed to boot and ended up in a crash/boot-loop, so I ended up having to use a ubuntu-20.04.6-live-server usb to create a new grub2 configuration, deleting the DSDT.aml and “01_acpi” files in the process.
Imagine my surprise when on next boot the Tesla P40 memory was finally assigned and nvidia-smi says ReBar works [* this is a k8s worker node, and I’m using Nvidia’s gpu-operator for the drivers]:
root@cloud19:~# uname -a
Linux cloud19 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@cloud19:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.4.0-100-generic root=UUID=… ro console=ttyS0,115200n8 console=tty0 amd_iommu=on intel_iommu=on iommu=pt pci=nocrs pci=assign-busses pci=realloc cgroup_disable=hugetlb dpdk-socket-mem=4096,4096 rcu_nocb_poll=true crashkernel=768M
root@cloud19:~# lspci -vv -s 01:00.0
01:00.0 3D controller: NVIDIA Corporation GP102GL [Tesla P40] (rev a1)
Subsystem: NVIDIA Corporation GP102GL [Tesla P40]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 66
Region 0: Memory at f1000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
Region 3: Memory at c00000000 (64-bit, prefetchable) [size=32M]
root@nvidia-driver-daemonset-bqnvj:/drivers# nvidia-smi
Sun Apr 30 03:32:06 2023
±----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07 Driver Version: 515.48.07 CUDA Version: 11.7 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla P40 On | 00000000:01:00.0 Off | Off |
| N/A 32C P8 9W / 250W | 0MiB / 24576MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found
root@nvidia-driver-daemonset-bqnvj:/drivers# nvidia-smi -q | grep -i bar -A 3
BAR1 Memory Usage
Total : 32768 MiB
Used : 2 MiB
Free : 32766 MiB |
I have no idea why “Above 4G Decoding” and ReBar appear to be working without the acpi and dsdt files still in grub. I’d post the patch file for DSDT.dsl, but new users to the forum can’t upload files. Unfortunately, the format of patch files don’t play well with post blockquotes, so I’ll post the description and content instead. Basically, I followed the DSDT Patching instructions [1,2], left the existing QWordMemory and essentially commented out the whole section around existing “M1LN” and “ML2N”, creating/leaving the following:
If ((PM0H == One))
{
CreateBitField (BUF0, \_SB.PCI0._Y1A._RW, F0RW) // _RW_: Read-Write Status
F0RW = Zero
}
CreateDWordField (BUF0, \_SB.PCI0._Y1C._MIN, M1MN) // _MIN: Minimum Base Address
CreateDWordField (BUF0, \_SB.PCI0._Y1C._MAX, M1MX) // _MAX: Maximum Base Address
CreateDWordField (BUF0, \_SB.PCI0._Y1C._LEN, M1LN) // _LEN: Length
M1MN = (TLUD << 0x14)
M1LN = ((M1MX - M1MN) + One)
CreateQWordField (BUF0, \_SB.PCI0._Y1E._LEN, M2LN) // _LEN: Length
CreateQWordField (BUF0, \_SB.PCI0._Y1E._MIN, M2MN) // _MIN: Minimum Base Address
CreateQWordField (BUF0, \_SB.PCI0._Y1E._MAX, M2MX) // _MAX: Maximum Base Address
// 39-bit for Haswell and up
M2MX = 0x8000000000 - One // ((M2MN + M2LN) - One)
If ((TUUD >= 0x1000))
{
M2MN = (TUUD << 0x14) // M64B // External reference
}
Else
{
M2MN = 0x100000000
}
M2LN = ((M2MX - M2MN) + One) // M64L // External reference
Return (BUF0) // \_SB_.PCI0.BUF0
References:
[1] DSDT Patching · xCuri0/ReBarUEFI Wiki · GitHub
[2] [Request] Above 4G decoding for AsRock z97 Extreme 9 and Asus P8P67 evo - BIOS/UEFI Modding / BIOS Modding Requests - Win-Raid Forum (level1techs.com)
