Dynamic PGO miscompilation: elided null check produces NullReferenceException

[zeppelin37]

Description

I get a NullReferenceException in our code base.
It reproduces only in an optimized (Release) run launched outside a debugger.
DOTNET_TieredPGO=0 seems to fix it.
[MethodImpl(MethodImplOptions.NoOptimization)] on the method also fixes it.

AI guesses it is a "a PGO-driven Tier1 codegen defect"

Reproduction Steps

Source code looks like this:

public class MergeHierarchy : MergeHierarchyMember
{
public List Children { get; set; } = new();

internal void AddMergeHierarchyMember(MergeHierarchyMember mergeHierarchyMember, long offset)
{
    Contract.AssertNotNull(mergeHierarchyMember);
    Contract.Assert(mergeHierarchyMember is not MergeHierarchy { Mode: MergeMode.PostProcessing }, "...");
    Contract.Assert(Mode is not MergeMode.Automatic || (Mode is MergeMode.Automatic && mergeHierarchyMember is MergeFile), "...");
    Children.Add(new ChildWithOffsetJson(mergeHierarchyMember, offset));   // <-- fault attributed here
}

}

Expected behavior

No NullReferenceException

Actual behavior

I was able to create a dump and see this:

Access violation reading location 0x0000000000000008 at RIP = ...B27.
Method base = ...A80. Registers at fault: R14 = 0 (all of RAX/RCX/RDX/RDI/R12/R13/R14 = 0);
valid heap pointers in RBX/RSI/R8/R15/RBP.

; entry: rcx=this, rdx=mergeHierarchyMember, r8=offset
+0x0C  mov   rbx, rdx                 ; rbx = mergeHierarchyMember
+0x15  test  rbx, rbx
+0x18  je    +1C5                     ; AssertNotNull: null -> throw  => fall-through: rbx PROVEN non-null
...
+0x7A  mov   rcx, <ChildWithOffsetJson MT>
+0x84  call  CORINFO_HELP_NEWSFAST    ; allocate ChildWithOffsetJson  (GC safepoint)
+0x89  mov   rbp, rax
+0x8C  mov   r14, rbx                 ; r14 = mergeHierarchyMember  (non-null by the entry proof)
+0x8F  mov   rcx, <MergeHierarchy MT>
+0x99  cmp   qword ptr [r14], rcx     ; type test — dereferences r14 with NO null check
+0x9C  jne   +157                     ; -> MergeFile branch
+0xA2  xor   ecx, ecx
+0xA4  mov   dword ptr [rbp+20h], ecx
+0xA7  mov   rbx, qword ptr [r14+8]   ; <== FAULT: r14 = null, reads [null+8]
+0xAB  test  rbx, rbx
+0xAE  je    +175
+0xB4  xor   r15d, r15d
+0xB7  cmp   dword ptr [rbx+8], 0     ; string.Length (whitespace scan of value.FileName)
+0xBB  jle   +175
+0xC1  movzx ecx, word ptr [rbx+r15*2+0Ch]   ; char at index
+0xC7  cmp   ecx, 100h
+0xCD  jge   +307
+0xD3  mov   rax, <char-classification table>
+0xDD  test  byte ptr [rax+rcx], 80h

Analysis (AI-generated)

• r14 is loaded at +0x8C from rbx, which is mergeHierarchyMember, proven non-null
  by the entry test/je at +0x15. rbx is not modified between entry and +0x8C, and it is
  a non-volatile register preserved across the NEWSFAST call at +0x84.
• Because the JIT considers r14 provably non-null, it emits the cmp [r14] type-check
  (+0x99) and the [r14+8] load (+0xA7) with no null checks — and, in the inlined
  MergeFile setter assertion, it also elided the value is null || short-circuit
  (the code goes straight into the !string.IsNullOrWhiteSpace(value.FileName) char scan).
• At run time r14 == 0, so [r14+8] (reading value.FileName) faults.
• The elision is only correct if r14 truly holds the non-null argument; under PGO it does not.

Net: PGO eliminated a null check that was not safe to eliminate, because the value it
proved non-null is null in-register on the PGO-optimized path.

Regression?

Code works fine on .NET8. We are migrating to .NET10

Known Workarounds

[MethodImpl(MethodImplOptions.NoOptimization)] on the method fixes it.

Configuration

.NET SDK:
Version: 10.0.301
Commit: 96856fd726
Workload version: 10.0.300-manifests.8c7d7c03
MSBuild version: 18.6.4+96856fd72

Runtime Environment:
OS Name: Windows
OS Version: 10.0.26200
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\10.0.301\

.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.

Host:
Version: 10.0.9
Architecture: x64
Commit: 901ca94124

.NET SDKs installed:
5.0.408 [C:\Program Files\dotnet\sdk]
8.0.206 [C:\Program Files\dotnet\sdk]
8.0.422 [C:\Program Files\dotnet\sdk]
9.0.315 [C:\Program Files\dotnet\sdk]
10.0.109 [C:\Program Files\dotnet\sdk]
10.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.28 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

@zeppelin37 given there is no repro for the bug, could you please try the latest public preview of .NET 11.0 if it reproduces there? Perhaps #125093 fixed the issue on main in .NET 11.0

[zeppelin37]

ConsoleApp1.zip

[zeppelin37]

I could not migrate our application to .NET11 on short notice, but I was able to reproduce the NullReferenceException reliably with the attached ConsoleApp1.zip

[EgorBo]

Thanks for the repro! Fixed in .NET 11.0, I'll backport to 10.0