Make TranslationLayer Native AOT-compatible by drewnoakes · Pull Request #16045 · microsoft/vstest

drewnoakes · 2026-05-19T14:14:43Z

Motivation

C# DevKit embeds the VS Test translation layer and is compiled with NativeAOT. The current STJ serialization code relies on reflection-based JsonSerializer overloads, which produce IL2026/IL3050 linker warnings and risk runtime failures when reflection is trimmed.

What this PR does

Adds a source-generated JsonSerializerContext and restructures the serialization layer so that NativeAOT consumers can use the translation layer APIs without trimming warnings or runtime errors.

Source-generated JSON context (`TestPlatformJsonContext`)

Lists all types that flow through the translation layer wire protocol via [JsonSerializable] attributes
Chained with DefaultJsonTypeInfoResolver via JsonTypeInfoResolver.Combine() so non-AoT consumers (vstest.console.exe) fall back to reflection for types outside the context

Serialization fixes

Changed envelope DTOs (MessageEnvelope, VersionedMessageEnvelope) from object? Payload to JsonElement? Payload to avoid polymorphic serialization
SerializePayloadCore now uses payload.GetType() so STJ sees the runtime type instead of object
WritePropertyValue passes JsonSerializerOptions through so the default case uses the resolver chain instead of bare reflection

Custom converters

ExceptionConverter: uses Exception(string, Exception) constructor to preserve Message during deserialization (the source-gen parameterless constructor path leaves it at the default value)
TestObjectBaseConverter: instantiates the correct typeToConvert instead of always creating TestCase

IL2026/IL3050 warning suppression (StjSafe)

All JsonSerializer.Serialize/Deserialize/SerializeToElement calls in converters route through a thin StjSafe wrapper with [UnconditionalSuppressMessage] attributes
This prevents the NativeAOT linker in consuming projects from emitting trimming warnings for these call sites
The suppressions are safe because all JsonSerializerOptions are configured with TestPlatformJsonContext as the primary TypeInfoResolver

Project configuration

Enabled IsAotCompatible, EnableTrimAnalyzer, EnableAotAnalyzer on both CommunicationUtilities and TranslationLayer projects (net8.0+)
Pre-existing server-only trim warnings (Jsonite, DiscoveryCriteriaConverter reflection) are downgraded via WarningsNotAsErrors

Scope

The CommunicationUtilities assembly is shared between the server (vstest.console.exe, not AoT) and the client (TranslationLayer, C# DevKit, AoT). Only client-side code paths need to be AoT-clean. Server-only code (e.g., DiscoveryCriteriaConverter with reflection, V1 converters) is not in the client path and is handled by the reflection fallback resolver.

The `Microsoft.TestPlatform.VsTestConsole.TranslationLayer` and `Microsoft.TestPlatform.CommunicationUtilities` assemblies are loaded in-process by NativeAOT consumers (e.g. C# Dev Kit). NativeAOT disables reflection-based `System.Text.Json` serialization by default, causing the vstest wire protocol to fail silently during the TCP handshake and test discovery. ## Changes ### Source-generated `JsonSerializerContext` Add `TestPlatformJsonContext` with `[JsonSerializable]` attributes for every type that crosses the wire: payload DTOs, envelope DTOs, collection types, and `PayloadedMessage<T>` for each concrete `T` used by `VsTestConsoleRequestSender`. Set `TypeInfoResolver` on the base `JsonSerializerOptions` so STJ has compile-time metadata available without runtime reflection. ### AOT-safe serialization converters - **`TestObjectBaseConverterFactory`**: replace `MakeGenericType` + `Activator.CreateInstance` with a singleton non-generic converter. - **`ObjectConverter.Write`**, **`ObjectDictionaryConverter.Write`**, **`TestObjectConverter.Write`**, **`TestCaseConverterV2.Write`**, **`TestResultConverterV2.Write`**: replace `JsonSerializer.Serialize(writer, value, value.GetType(), options)` with direct primitive writes via a centralized `WritePropertyValue()` helper that handles string, int, long, double, float, bool, DateTimeOffset, DateTime, Guid, Uri, with `ToString()` fallback. ### Envelope DTO fixes - Change `MessageEnvelope.Payload` and `VersionedMessageEnvelope.Payload` from `object?` to `JsonElement?` so pre-serialized payloads are embedded as nested JSON objects rather than double-encoded strings. - Make `MessageEnvelope`, `VersionedMessageEnvelope`, `VersionedMessageForSerialization`, and `PayloadedMessage<T>` `internal` (were `private`) so the source generator can reference them. - Use `JsonSerializer.SerializeToElement(payload, payload.GetType(), options)` to serialize payloads with runtime type dispatch before embedding in envelopes. ### AOT/trim analyzers enabled Set `IsAotCompatible=true`, `EnableTrimAnalyzer=true`, and `EnableAotAnalyzer=true` on the `net8.0` TFM for both `TranslationLayer` and `CommunicationUtilities` projects. Both build with zero IL2xxx/IL3xxx warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Chain source-gen context with DefaultJsonTypeInfoResolver so types not covered by the source-gen context fall back to reflection in non-AoT builds, while NativeAOT consumers use the source-gen metadata. - Pass JsonSerializerOptions through WritePropertyValue so the default case uses the resolver chain instead of bare reflection. - Add ExceptionConverter to handle Exception de/serialization properly since source-gen uses the parameterless constructor which cannot populate the read-only Message/StackTrace properties. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The converter was always creating a TestCase regardless of typeToConvert, which caused InvalidCastException when deserializing other TestObject subclasses. Now uses Activator.CreateInstance(typeToConvert) for concrete types, falling back to TestCase for abstract types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

All JsonSerializer.Serialize/Deserialize calls in converters and JsonDataSerializer now go through StjSafe, which centralizes the [UnconditionalSuppressMessage] attributes. This prevents the NativeAOT linker in consuming projects (e.g., C# DevKit) from emitting IL2026 trimming warnings for these call sites. The suppressions are safe because all JsonSerializerOptions instances are configured with TestPlatformJsonContext (source-gen) as the primary TypeInfoResolver. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Makes the TranslationLayer and CommunicationUtilities assemblies NativeAOT/trim-compatible by introducing a source-generated JsonSerializerContext, routing STJ calls through suppression wrappers, and restructuring envelope DTOs and converters to avoid reflection-based polymorphic serialization.

Changes:

Added TestPlatformJsonContext (source-gen) and StjSafe wrappers; chained source-gen resolver with DefaultJsonTypeInfoResolver fallback.
Changed envelope Payload from object? to JsonElement?, added ExceptionConverter, and rewrote TestObjectBaseConverter to avoid MakeGenericType/Activator.CreateInstance patterns; centralized property-value writing in WritePropertyValue.
Enabled IsAotCompatible/trim/AOT analyzers on both projects (net8.0+) and downgraded pre-existing trim warnings in CommunicationUtilities.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/Microsoft.TestPlatform.VsTestConsole.TranslationLayer/Microsoft.TestPlatform.VsTestConsole.TranslationLayer.csproj	Enable AOT/trim analyzers for net8.0+.
src/Microsoft.TestPlatform.CommunicationUtilities/Microsoft.TestPlatform.CommunicationUtilities.csproj	Enable AOT/trim analyzers and downgrade pre-existing warnings.
src/Microsoft.TestPlatform.CommunicationUtilities/TestPlatformJsonContext.cs	New source-generated JSON context listing wire types.
src/Microsoft.TestPlatform.CommunicationUtilities/Serialization/StjSafe.cs	Suppression wrappers around `JsonSerializer` APIs.
src/Microsoft.TestPlatform.CommunicationUtilities/Serialization/ExceptionConverter.cs	New converter preserving `Message`/`InnerException` on deserialization.
src/Microsoft.TestPlatform.CommunicationUtilities/Serialization/TestObjectBaseConverter.cs	Replaced generic converter with single non-generic converter + `WritePropertyValue` helper.
src/Microsoft.TestPlatform.CommunicationUtilities/Serialization/ObjectConverter.cs	Use `WritePropertyValue` for object/dictionary values.
src/Microsoft.TestPlatform.CommunicationUtilities/Serialization/TestCaseConverter.cs, TestCaseConverterV2.cs, TestResultConverter.cs, TestResultConverterV2.cs, TestObjectConverter.cs, TestExecutionContextConverter.cs, TestRunChangedEventArgsConverter.cs, TestRunCompleteEventArgsConverter.cs, AfterTestRunEndResultConverter.cs, DiscoveryCriteriaConverter.cs, AttachmentConverters.cs	Routed STJ calls through `StjSafe`; avoid `value.GetType()` overload.
src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.Stj.cs	Configured `TypeInfoResolver`, switched envelope payload to `JsonElement?`, registered `ExceptionConverter`, made nested DTOs internal.

nohwnd · 2026-05-20T14:13:42Z

Looks good, on quick look. Can we get an integration test (can be marked as compatibility to avoid running it too much, that will actually build app in native aot, check that there are no warnings and will run a test dll project via that)?

(Compatibility tests don't run in PR build, nor do they build automatically during asset build. LMK if you need more help. )

- ExceptionConverter: restore HResult and Source in Read (writable properties), document type-erasure and StackTrace limitations. - WritePropertyValue: add missing primitive cases for short, ushort, uint, ulong, byte, sbyte, decimal, char, and enums to avoid falling through to SerializeToElement for types not in the source-gen context. - TestObjectBaseConverter: revert Activator.CreateInstance to new TestCase() to avoid reflection incompatible with NativeAOT; document that this path only handles generic property bags on the wire. - SerializePayloadCore: document why the fast path was intentionally collapsed (object? Payload is incompatible with AoT). - TestPlatformJsonContext: add maintenance checklist for new payload types, clarify why Dictionary<string,object> entries are needed alongside the custom converters. - StjSafe: add DEBUG assert verifying options.TypeInfoResolver is configured, to catch misuse as the codebase evolves. - Update TestObjectConverterTests to deserialize as TestObject (not TestableTestObject) and verify custom properties by ID rather than counting all properties (TestCase carrier has built-in properties). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Introduce RemoteException that carries the original ClassName and StackTraceString from the remote process. ToString() renders the full original diagnostic output (type name, message, stack trace, inner exception chain) so that callers see the same information they would from the original exception. The ExceptionConverter Write path also handles round-tripping: if the exception is already a RemoteException, it preserves the stored ClassName and RemoteStackTrace rather than emitting the wrapper type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 10 comments.

The fast-path branch was already doing the same thing as the slow-path after the AoT changes (both serialize to JsonElement then embed in envelope). Merge into a single code path and remove the now-unused DisableFastJson field and Utilities using. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a test asset (NativeAotTranslationLayerConsumer) that is a minimal console app referencing the TranslationLayer with PublishAot=true. The NativeAotCompatibilityTests test publishes this app with NativeAOT and asserts that no IL2026/IL3050 linker warnings originate from the CommunicationUtilities.Serialization namespace. Pre-existing warnings from Jsonite, ObjectModel, and DefaultJsonTypeInfoResolver are excluded. Marked with [TestCategory("Compatibility")] so it doesn't run in every PR build (NativeAOT publish takes ~4 minutes). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.Stj.cs:1

This change removes the previous ‘fast’ serialization path and now always performs a two-step object -> JsonElement -> envelope flow, which is materially more expensive (allocations + extra traversal) for high-volume message traffic. If NativeAOT safety is the driver, consider keeping the previous direct serialization path for non-AOT scenarios (e.g., when dynamic code/reflection is supported) and only using the JsonElement-based path when running under AOT constraints; that preserves throughput while still meeting AOT requirements.

// Copyright (c) Microsoft Corporation. All rights reserved.

- Narrow TestObjectBaseConverter.CanConvert to only typeof(TestObject), not all TestObject subtypes. TestCase/TestResult have their own converters and no other subtypes flow through the wire protocol. - Remove Enum special case from WritePropertyValue — let enums fall through to SerializeToElement which respects JsonSerializerOptions (avoids ulong overflow and bypassing custom enum converters). - Test asset: remove unused usings, touch VsTestConsoleWrapper type and payload types so the linker analyzes the full TranslationLayer graph. - Tests: use Assert.IsNotNull before TestProperty.Find instead of null-forgiving operator. - Tests: serialize as TestObject (declared wire type) not TestableTestObject. - Integration test: add second WaitForExit() call to drain async output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

drewnoakes · 2026-05-26T06:51:32Z

Can we get an integration test

Added. All feedback addressed.

nohwnd · 2026-05-26T13:20:08Z

/azp run

azure-pipelines · 2026-05-26T13:20:23Z

Azure Pipelines successfully started running 1 pipeline(s).

nohwnd · 2026-05-26T13:20:42Z

Re-run just to be sure this is not some kind of flakiness, but the error in CI looks real: System.MemberAccessException: Cannot create an instance of Microsoft.VisualStudio.TestPlatform.ObjectModel.TestObject because it is an abstract class.

The Jsonite ConvertTo method excluded typeof(TestObject) from the DeserializeTestObject handler, causing it to fall through to the generic CreateInstance path which throws MemberAccessException for abstract types. Fix by removing the exclusion and using TestCase as a concrete property-bag carrier when the target type is abstract, matching the approach used in the STJ TestObjectBaseConverter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.Stj.cs:1

This change forces a two-step serialization (object -> JsonElement -> envelope JSON) for all builds/targets, which is a clear regression versus the previous “fast” path that could serialize the envelope directly in one pass. To keep AoT-safety and preserve performance for non-AoT scenarios, consider re-introducing a single-pass fast-path gated on a runtime check (e.g., dynamic code / reflection availability) or a build-time condition, while keeping the JsonElement path for NativeAOT/trimming scenarios.

// Copyright (c) Microsoft Corporation. All rights reserved.

+            default:
+                // For complex types (Traits, collections, etc.), serialize to JsonElement
+                // first using the runtime type, then write the element. This avoids the
+                // object? polymorphism problem while still producing valid JSON.
+                var element = StjSafe.SerializeToElement(value, value.GetType(), options);
+                element.WriteTo(writer);
+                break;
+        }
+    }


Verifies that custom properties survive a serialize-as-concrete-subtype then deserialize-as-abstract-TestObject round trip. Covers both the STJ path (net11.0) and Jsonite path (net481). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tionaries Reduces reliance on the StjSafe.SerializeToElement fallback which requires runtime type metadata that may not be available under NativeAOT. All known property value types used in the wire protocol now have explicit handlers: - Enum (as numeric value) - TimeSpan (as string) - string[] (as JSON string array) - KeyValuePair<string, string>[] (as JSON array of Key/Value objects) - IDictionary (as JSON object) - IEnumerable (as JSON array) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.

-        var test = Deserialize<TestableTestObject>("{\"Properties\":[]}");
+        var test = Deserialize<TestObject>("{\"Properties\":[]}");

        Assert.IsNotNull(test);


+    [TestMethod]
+    public void TestObjectShouldRoundTripCustomPropertiesFromConcreteSubtype()
+    {
+        var original = new TestableTestObject();
+        var stringProp = TestProperty.Register("rt1", "RoundTripString", typeof(string), typeof(TestableTestObject));
+        var intProp = TestProperty.Register("rt2", "RoundTripInt", typeof(int), typeof(TestableTestObject));


+        Assert.IsTrue(exited, "dotnet publish timed out after 10 minutes.");
+


+        if (payload is null)
+            return string.Empty;
+
        var payloadOptions = GetPayloadOptions(version);
-        // Fast json is only equivalent to the serialization that is used for protocol version 2 and upwards (or more precisely for the paths that use PayloadOptionsV2)
-        // so when we resolved the old options we should use non-fast path.
-        if (DisableFastJson || payloadOptions == PayloadOptionsV1)
-        {
-            if (payload is null)
-                return string.Empty;

-            var serializedPayload = JsonSerializer.SerializeToElement(payload, payloadOptions);
+        // Serialize payload to JsonElement first using the versioned options (which have the
+        // custom converters), then embed in the envelope. This two-step approach is required
+        // for NativeAOT: serializing object? Payload directly would require STJ to resolve
+        // the runtime type polymorphically via reflection.
+        var serializedPayload = StjSafe.SerializeToElement(payload, payload.GetType(), payloadOptions);


+    <!-- Downgrade pre-existing AOT/trim warnings from errors to warnings.
+         Our new code is AOT-clean; these are from Jsonite and legacy converters. -->
+    <WarningsNotAsErrors>$(WarningsNotAsErrors);IL2026;IL2057;IL2067;IL3050</WarningsNotAsErrors>


+                // Write enums as their underlying numeric value.
+                writer.WriteNumberValue(Convert.ToInt64(e, CultureInfo.InvariantCulture));


nohwnd · 2026-05-27T13:37:02Z

vstest main is already rebranded to 18.9, so ideal time to bring those changes to C#DK and pilot them.

drewnoakes · 2026-05-28T06:32:07Z

@nohwnd that's fantastic, thank you! Is there a package version I can use to validate this with?

nohwnd · 2026-05-28T09:00:51Z

Please see http://aka.ms/vstest/preview for guidance on consuming the latest builds of vstest. It should be any 18.9 package.

…8.1) (#16281) * Set an explicit reflection-based TypeInfoResolver on the STJ options Test projects that set JsonSerializerIsReflectionEnabledByDefault=false put that switch into their runtimeconfig.json, which testhost runs under. With TypeInfoResolver left null, System.Text.Json falls back to the implicit default resolver that the switch disables, so testhost throws while deserializing the first protocol message and dies, and the runner waits out the whole connection timeout and reports "Failed to negotiate protocol, waiting for response timed out". Assigning an explicit DefaultJsonTypeInfoResolver keeps reflection-based serialization working regardless of the switch. This is the minimal servicing fix for 18.8.1, main and rel/18.9 already resolve this as part of the larger NativeAOT work in #16045. Also bump VersionPrefix to 18.8.1. Fixes #16274. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Re-trigger CI (flaky RunSettingsManager singleton test on ubuntu leg) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

drewnoakes and others added 4 commits May 5, 2026 21:54

Copilot AI review requested due to automatic review settings May 19, 2026 14:14

Copilot AI reviewed May 19, 2026

View reviewed changes

Copilot started reviewing on behalf of drewnoakes May 19, 2026 14:31 View session

nohwnd mentioned this pull request May 20, 2026

Daily Maintenance Digest — 2026-05-20 #16036

Closed

nohwnd mentioned this pull request May 21, 2026

Daily Maintenance Digest — 2026-05-25 #16049

Closed

drewnoakes and others added 2 commits May 26, 2026 10:11

Copilot AI review requested due to automatic review settings May 26, 2026 01:56

Copilot AI reviewed May 26, 2026

View reviewed changes

drewnoakes and others added 2 commits May 26, 2026 12:15

Copilot AI review requested due to automatic review settings May 26, 2026 03:01

Copilot AI reviewed May 26, 2026

View reviewed changes

nohwnd mentioned this pull request May 26, 2026

Daily Maintenance Digest — 2026-06-11 #16066

Closed

Copilot AI review requested due to automatic review settings May 26, 2026 23:49

Copilot AI reviewed May 26, 2026

View reviewed changes

drewnoakes and others added 2 commits May 27, 2026 09:54

Copilot AI review requested due to automatic review settings May 27, 2026 03:27

Copilot AI reviewed May 27, 2026

View reviewed changes

nohwnd approved these changes May 27, 2026

View reviewed changes

nohwnd merged commit 5226f3c into microsoft:main May 27, 2026
13 of 14 checks passed

This was referenced May 27, 2026

[release/11.0.1xx-preview5] Source code updates from microsoft/vstest dotnet/dotnet#6860

Merged

[main] Source code updates from microsoft/vstest dotnet/dotnet#6886

Merged

drewnoakes deleted the dev/drnoakes/aot-compat branch May 28, 2026 06:31

This was referenced Jun 23, 2026

[main] Source code updates from microsoft/vstest dotnet/dotnet#7319

Closed

[release/11.0.1xx-preview6] Source code updates from microsoft/vstest dotnet/dotnet#7356

Merged

nohwnd mentioned this pull request Jul 14, 2026

Fix protocol negotiation timeout when STJ reflection is disabled (18.8.1) #16281

Merged

		Assert.IsTrue(exited, "dotnet publish timed out after 10 minutes.");

		// Write enums as their underlying numeric value.
		writer.WriteNumberValue(Convert.ToInt64(e, CultureInfo.InvariantCulture));

Uh oh!

Conversation

drewnoakes commented May 19, 2026

Motivation

What this PR does

Source-generated JSON context (TestPlatformJsonContext)

Serialization fixes

Custom converters

IL2026/IL3050 warning suppression (StjSafe)

Project configuration

Scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nohwnd commented May 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drewnoakes commented May 26, 2026

Uh oh!

nohwnd commented May 26, 2026

Uh oh!

azure-pipelines Bot commented May 26, 2026

Uh oh!

nohwnd commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

nohwnd commented May 27, 2026

Uh oh!

drewnoakes commented May 28, 2026

Uh oh!

nohwnd commented May 28, 2026

Uh oh!

Reviewers

Source-generated JSON context (`TestPlatformJsonContext`)