The InProcDataCollector's TestCaseStart/TestCaseStop methods are not always called.

[kname88]

Description

I've implemented example InProcDataCollector in order to track the test-case execution but its methods: TestCaseStart/TestCaseStop are not called for some data-driven tests.
Below is a scenario where 8 tests are executed but the InProcDataCollector is 'informed' only about 6 of them.

Steps to reproduce

• Remarks:

  • Assumption: DotNet SDK is installed on the system.
  • The attached SimpleDataCollector project is based on VsTest asset: https://github.com/microsoft/vstest/tree/main/test/TestAssets/SimpleDataCollector

• What steps can reproduce the defect?

  1. Build the attached project using SimpleDataCollector.sln
  2. Execute RunTests.cmd and observe the reported output
     (the batch will generate proper .runsettings files in the output folders (for .NetFw and .Net8) in order to use SimpleDataCollector)
     InProcDataCollector-Case.zip

Expected behavior

Expecting 8 tests to be executed and SimpleDataCollector's TestCaseStart/TestCaseStop methods are triggered for each one, so 8 times.
Traces expected to be reported by SimpleDataCollector are:

    TestSessionStart : <Configuration><Port>4312</Port></Configuration>
    TestCaseStart : TestRowsWithString_1
    TestCaseEnd : TestRowsWithString_1
    TestCaseStart : TestRowsWithString_2
    TestCaseEnd : TestRowsWithString_2
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseStart : TestRowsWithMixData_DataRow1
    TestCaseEnd : TestRowsWithMixData_DataRow1
    TestCaseStart : TestRowsWithMixData_DataRow2
    TestCaseEnd : TestRowsWithMixData_DataRow2
    TestCaseStart : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow11) parameters
    TestCaseEnd : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow11) parameters
    TestCaseStart : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow12) parameters
    TestSessionEnd

Actual behavior

Executed 8 tests but SimpleDataCollector's TestCaseStart/TestCaseStop methods are triggered only 6 times:

  TestSessionStart : <Configuration><Port>4312</Port></Configuration>
   TestCaseStart : TestRowsWithString_1
   TestCaseEnd : TestRowsWithString_1
   TestCaseStart : TestRowsWithString_2
   TestCaseEnd : TestRowsWithString_2
   TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
   TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
   TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
   TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
   TestCaseStart : TestRowsWithMixData
   TestCaseEnd : TestRowsWithMixData
   TestCaseStart : TestWithDynamicMixDataArgs
   TestCaseEnd : TestWithDynamicMixDataArgs
   TestSessionEnd

Environment

Windows10 Enterprise
VS2017.8 and .Net8 SDK
NuGet pacakges (see the attached projects):
    Microsoft.TestPlatform.XXX v17.8.0
    MSTest.XXX v3.3.1

[nohwnd]

Root Cause Analysis 🔍

I traced through the vstest codebase to find why TestCaseStart/TestCaseStop are not always called for data-driven tests.

The relevant code is in TestExecutionRecorder.RecordStart (TestExecutionRecorder.cs):

public void RecordStart(TestCase testCase)
{
    _testRunCache.OnTestStarted(testCase);

    if (_testCaseEventsHandler != null)
    {
        lock (_testCaseInProgressSyncObject)
        {
            // Do not send TestCaseStart for a test in progress
            if (!_testCaseInProgressMap.Contains(testCase.Id))
            {
                _testCaseInProgressMap.Add(testCase.Id);
                _testCaseEventsHandler.SendTestCaseStart(testCase);
            }
        }
    }
}

The guard !_testCaseInProgressMap.Contains(testCase.Id) is designed to prevent duplicate TestCaseStart events for the same test. The key is testCase.Id — a Guid derived from the test's fully-qualified name.

The issue: For data-driven tests, MSTest can produce multiple test cases that share the same TestCase.Id if their fully-qualified names are identical (the ID is a deterministic hash of the FQDN). When MSTest reports the parent data-driven test and its individual data row executions through separate RecordStart calls, the second call is silently dropped because the ID is already in the map.

Specifically:

1. Parent data-driven test: RecordStart(parentTestCase) → ID added to map, TestCaseStart fired
2. First child execution: RecordStart(childTestCase) → if same ID as parent, silently skipped
3. RecordEnd for parent removes the ID from the map
4. Remaining children may or may not send events depending on timing

This would explain why ~2 out of 8 tests are not reported (the "parent wrapper" tests for each parameterized method are counted but their children may be skipped).

Suggested Fix

The deduplication in RecordStart should use a more granular key. Options:

1. Track by (testCase.Id, executionId) if available
2. Allow re-entry for the same ID once the previous end has been received (already handled by _testCaseInProgressMap.Remove)
3. Remove the guard entirely and let the adapter be responsible for not sending duplicate starts

🔍 Triaged by Issue Repro Triage & Auto-Fix 🔍

🔍 Triaged by Issue Repro Triage & Auto-Fix 🔍

[nohwnd]

Pull request created: #16033

🔍 Triaged by Issue Repro Triage & Auto-Fix 🔍