[tests] Isolate JNI reference leak tests

[simonrozsival]

Summary

• add a dedicated on-device NUnit/Microsoft Testing Platform app for JNI reference and peer leak checks
• reuse the shared TestRunner.Core instrumentation with synchronous NUnit execution
• migrate Android GREF, LREF, surfaced-peer, weak-peer, Java-side activation, object-array, and custom-widget leak checks out of shared test apps
• preserve the complete CreatedReference and Java array collection-contract coverage in the isolated app
• replace ambient process-total assertions with amplified leak checks or object-specific peer assertions
• fix JNIEnv.DeleteRef() so ownership flags combined with DoNotRegister still release local/global references
• run the app in package CI under CoreCLR Debug, CoreCLR trimmable Release, and NativeAOT, with additional Debug/Release nightly emulator coverage

Why

GREF and surfaced-peer counts are process-wide. The existing Android test apps load many unrelated fixtures into one process, and runtime or test-infrastructure work on background threads can change those totals while a test is taking before/after snapshots. Serializing workers in those shared apps did not remove the ambient process activity.

The new app contains only reference-accounting tests and runs NUnit synchronously through the shared instrumentation. Process-wide checks execute equal 100-operation warmup and measurement batches, allowing only a small fixed amount of ambient drift, so a per-operation leak is strongly amplified. Thread-local LREF checks retain exact accounting, while registry tests count only the specific peer under test.

The isolated array coverage exercises the inherited collection surface across every primitive and object array fixture, including IndexOf, Insert, RemoveAt, CopyTo, and indexer boundary paths. Running that coverage with the trimmable type map exposed a latent, path-specific leak. Normal CoreCLR uses the reflection-backed value manager and releases these references through a different path. The trimmable value manager converts boxed array elements through JavaConvert, which creates a temporary wrapper with TransferLocalRef | DoNotRegister. That combined value did not match any exact-value JNIEnv.DeleteRef() switch case, leaving one local reference behind for every object-array getter.

Typical short JNI calls masked the problem because the VM releases remaining locals when the native frame returns. The instrumentation runner keeps one frame open for the test run, so repeated operations made the leak deterministic: the JavaObjectArray<int> contract pass accumulated 11 local references. Masking the registration-only flag before selecting the ownership action restores the expected reference cleanup.

Functional coverage remains in the original suites, and JVM-only accounting tests continue to run there. The custom-widget test previously quarantined under #11201 now passes reliably in the isolated process.

Validation

• CoreCLR Debug: 14 passed
• CoreCLR trimmable Release: 14 passed
• NativeAOT: 13 passed, 1 existing inconclusive test tracked by [NativeAOT] Investigate failing DoNotLeakWeakReferences #11690
• local SDK rebuilt successfully
• pipeline YAML parsed successfully
• git diff --check

Fixes #12031.
Supersedes #12037.

[Copilot]

Pull request overview

This PR isolates JNI reference/peer leak checks into a dedicated on-device MSTest/Microsoft Testing Platform (MTP) test app (tests/JniReferenceLeakTests) to avoid flaky process-wide assertions caused by unrelated background activity in shared device test processes.

Changes:

• Added a dedicated JniReferenceLeakTests Android test app with serialized MSTest execution ([assembly: DoNotParallelize]) and focused leak/peer assertions.
• Migrated/removed flaky or process-global JNI reference/peer-count assertions from shared device test suites; narrowed remaining assertions to object-specific checks or restricted them to non-Android runs.
• Wired the new test app into solution/test documentation and into CI (package tests for CoreCLR + Mono; nightly emulator coverage).

Show a summary per file

File	Description
Xamarin.Android-Tests.slnx	Adds the new JniReferenceLeakTests project to the tests solution structure.
tests/Mono.Android-Tests/Mono.Android-Tests/Java.Interop/JnienvTest.cs	Removes the weak-peer surfaced-object drift test from the shared device suite.
tests/Mono.Android-Tests/Mono.Android-Tests/Android.Widget/CustomWidgetTests.cs	Removes the previously ignored/flaky custom-widget GREF leak test from the shared suite.
tests/JniReferenceLeakTests/JniReferenceLeakTests.csproj	New dedicated Android MSTest/MTP app project for leak/peer accounting tests.
tests/JniReferenceLeakTests/TestInstrumentation.cs	Implements an MTP-based Android Instrumentation runner for MSTest with TRX reporting.
tests/JniReferenceLeakTests/AssemblyInfo.cs	Disables MSTest parallelism at the assembly level to stabilize process-wide accounting.
tests/JniReferenceLeakTests/ReferenceTestHelpers.cs	Adds helpers to amplify leak detection and synchronize GC/peer collection.
tests/JniReferenceLeakTests/GlobalReferenceTests.cs	Adds amplified GREF leak checks (TryFindClass, activation, object arrays).
tests/JniReferenceLeakTests/LocalReferenceTests.cs	Adds thread-local LREF accounting checks in an isolated process.
tests/JniReferenceLeakTests/PeerReferenceTests.cs	Adds object-specific surfaced-peer assertions + weak-peer collection check in isolation.
tests/JniReferenceLeakTests/WidgetReferenceTests.cs	Reintroduces the custom-view inflate leak check in the isolated harness.
tests/JniReferenceLeakTests/Resources/layout/leak_test_widget.axml	Layout used to inflate a custom managed widget for the leak test.
tests/JniReferenceLeakTests/java/net/dot/jni/referenceleaktests/ActivationProbeFactory.java	Java-side factory to exercise Java-side activation paths.
tests/JniReferenceLeakTests/AndroidManifest.xml	Manifest for the new test app.
tests/JniReferenceLeakTests/global.json	Forces Microsoft Testing Platform runner behavior when invoking dotnet test.
tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/BindingTests.cs	Removes process-global GREF equality assertion from Java-side activation test.
external/Java.Interop/tests/Java.Interop-Tests/Java.Interop/JniTypeUtf8Test.cs	Restricts process-global GREF leak assertions to non-Android runs.
external/Java.Interop/tests/Java.Interop-Tests/Java.Interop/JniRuntimeJniValueManagerContract.cs	Changes peer assertions from total-count based to peer-specific checks; restricts GREF leak test to non-Android runs.
external/Java.Interop/tests/Java.Interop-Tests/Java.Interop/JniEnvironmentTests.cs	Restricts process-global reference-count assertions to non-Android runs.
external/Java.Interop/tests/Java.Interop-Tests/Java.Interop/JavaObjectTest.cs	Restricts runtime-global surfaced-peer count assertion to non-Android runs.
external/Java.Interop/tests/Java.Interop-Tests/Java.Interop/JavaObjectArrayTest.cs	Restricts fixture-wide process-global GREF baseline checks to non-Android runs.
external/Java.Interop/tests/Java.Interop-Tests/Java.Interop/JavaArrayContract.cs	Restricts fixture-wide LREF baseline checks to non-Android runs (covered in new isolated app).
build-tools/automation/yaml-templates/stage-package-tests.yaml	Adds CI legs to run the new app under CoreCLR and Mono in package tests.
build-tools/automation/azure-pipelines-nightly.yaml	Adds nightly emulator coverage for the new isolated leak test app.
.github/skills/tests/references/test-catalog.md	Documents how to build/run the new dedicated leak test app locally.

Copilot's findings

• Files reviewed: 25/25 changed files
• Comments generated: 0

[simonrozsival]

✅ New JniReferenceLeakTests project — all tests green on CI

Confirmed on the public dotnet-android build #1504876, in Package Tests → macOS → APKs 2: the new isolated project built, deployed to the emulator, ran under CoreCLR, and passed 12/12 tests — 0 failed, 0 skipped.

Lane tasks all succeeded: build JniReferenceLeakTests-CoreCLR → run JniReferenceLeakTests-CoreCLR → publish JniReferenceLeakTests-CoreCLR results (the diagnose install failure step was skipped, i.e. a clean deploy).

12 passing tests

GlobalReferenceTests

• JavaObjectArrayOperationsDoNotLeakGlobalReferences
• JavaSideActivationDoesNotLeakGlobalReferences
• TryFindClassStringDoesNotLeakGlobalReferences
• TryFindClassUtf8DoesNotLeakGlobalReferences

LocalReferenceTests

• ArrayOperationsDoNotLeakLocalReferences
• InvalidCreatedReferenceDoesNotChangeLocalReferenceCount
• JavaSideActivationDoesNotLeakLocalReferences

PeerReferenceTests

• AddPeerDoesNotRegisterDuplicates
• RepeatedConstructPeerDoesNotLeakGlobalReferences
• UnregisterFromRuntimeRemovesSurfacedPeer
• WeakPeerIsCollectedWithoutLeakingReferences

WidgetReferenceTests

• InflateCustomViewDoesNotLeakGlobalReferences

ℹ️ The one red lane on this build (APKs 1 → build Mono.Android.NET_Tests-Debug) is a separate binding-compile break in the pre-existing Java.Interop-Tests project (CS0436/CS0108/CS0114 in generated Net.Dot.Jni.Test.* code) — unrelated to these new tests.

_{🤖 Automated CI status summary via GitHub Copilot CLI.}

[simonrozsival]

/review

[github-actions]

✅ Android PR Reviewer completed successfully!

[github-actions]

⚠️ Review — Isolate JNI reference leak tests

Solid, well-motivated PR. Moving process-wide GREF/surfaced-peer accounting out of the shared multi-fixture test apps into a dedicated [assembly: DoNotParallelize] MSTest/MTP app is the right fix for the flakiness described in #12031: the refactored contract tests now count only the specific peer under test (GetSurfacedPeerCount(peer)) instead of process-global totals, and the warmup+measurement amplification in AssertNoGlobalReferenceLeak is a sound way to turn per-op leaks into detectable deltas. Guarding the process-wide assertions in the Java.Interop suite with #if !__ANDROID__ cleanly leaves JVM-only accounting where it belongs.

Verdict: ⚠️ Needs Changes (minor) — nothing blocking; mostly the Debug-vs-Release CI question worth a maintainer decision.

Findings

• 💡 Unreachable return; after Assert.Inconclusive in PeerReferenceTests.
• 💡 Surfaced-peer tolerance/batch duplicated inline instead of reusing the ReferenceTestHelpers amplification helper.
• 💡 MTP run exceptions in TestInstrumentation are only surfaced via the result Bundle; also log to adb logcat.
• ⚠️ Nightly + package lanes build this app as Debug only, so full trimming (Release-only) leaks aren't exercised — please confirm intent.

Notes

• CI status is still pending on the head commit — the Azure DevOps dotnet-android pipeline hadn't reported at review time. This isn't LGTM-able until those checks are green; please re-check the pipeline (and the newly-added JniReferenceLeakTests lanes) before merge.
• Nice touches: per-thread exact LREF accounting retained, [MethodImpl(NoInlining)] on the weak-reference/array creators to keep locals from being kept alive, and the migrated custom-widget test (#11201) now running in isolation.

Positive callout: the diff meaningfully reduces ambient-noise coupling rather than just papering over flakes with wider tolerances. 👍

Generated by Android PR Reviewer for #12038 · 116.1 AIC · ⌖ 13.3 AIC · ⊞ 6.8K
_{Comment /review to run again}

[simonrozsival]

@copilot build fails with:

Build FAILED.

/Users/runner/work/1/s/tests/JniReferenceLeakTests/JniReferenceLeakTests.csproj : error NU1603: Warning As Error: MSTest 4.4.0-preview.26360.5 depends on Microsoft.Testing.Extensions.CodeCoverage (>= 18.9.0-preview.26358.3) but Microsoft.Testing.Extensions.CodeCoverage 18.9.0-preview.26358.3 was not found. Microsoft.Testing.Extensions.CodeCoverage 18.9.0 was resolved instead.
    0 Warning(s)
    1 Error(s)

look for best ways to resolve this.

[Copilot]

@copilot build fails with:

Build FAILED.

...

Fixed in e254bf4 by adding an explicit Microsoft.Testing.Extensions.CodeCoverage 18.9.0 reference. The required preview package is no longer available on configured feeds; explicitly selecting its compatible stable release removes NU1603. Restore and build now complete with TreatWarningsAsErrors=true (0 warnings).