[8.0] "InvalidOperationException: Category does not exist" failure in net48 win x86 in System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_IncrementBy_IncrementBy2

[carlossanlop]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=706881
Build error leg or test failing: System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_IncrementBy_IncrementBy2

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.InvalidOperationException : Category does not exist.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

PR: #103426
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=706881&view=logs&j=885df564-30f3-5dad-0bdc-caf444244bdf&t=71aa97be-149e-50ab-5d1c-74d5540a091f&l=50
Job name: windows-x86 Release Libraries_NET48
Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-103426-merge-01018e12059a47ae96/System.Diagnostics.PerformanceCounter.Tests/1/console.bc703876.log?helixlogtype=result
Output:

    System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_IncrementBy_IncrementBy2 [FAIL]
      System.AggregateException : One or more errors occurred.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      ---- System.InvalidOperationException : Category does not exist.
      Stack Trace:
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(54,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(78,0): at System.Diagnostics.Tests.Helpers.RetryOnAllPlatforms[T](Func`1 func)
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(237,0): at System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_IncrementBy_IncrementBy2()
        ----- Inner Stack Trace #1 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #2 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #3 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #4 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #5 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #6 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #7 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #8 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #9 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #10 (System.InvalidOperationException) -----
           at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
           at System.Diagnostics.PerformanceCounter.NextSample()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(243,0): at System.Diagnostics.Tests.PerformanceCounterTests.<>c__DisplayClass15_0.<PerformanceCounter_IncrementBy_IncrementBy2>b__1()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs(75,0): at System.Diagnostics.Tests.Helpers.<>c__DisplayClass12_0`1.<RetryOnAllPlatforms>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=706881
Error message validated: [System.InvalidOperationException : Category does not exist.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/13/2024 6:20:23 PM UTC

Report

Build	Repository	Test	Pull Request
1553711	dotnet/runtime	MonoTests.System.Runtime.Caching.CountersTest.Basic_Counters	#132321
1553274	dotnet/runtime	MonoTests.System.Runtime.Caching.CountersTest.Basic_Counters	#132163
1550807	dotnet/runtime	MonoTests.System.Runtime.Caching.CountersTest.Basic_Counters

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	3

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-performancecounter
See info in area-owners.md if you want to be subscribed.

[ericstj]

There's a number of perfcounter issues with failure mode System.InvalidOperationException : Category does not exist. This is a common mode of flakiness for these tests. IIRC there's some loss-tolerance in the actual platform APIs that can cause this. I'm not sure the RetryHelper in use here helps, because I think the problem causes the static counter list in the process to be missing the counter category. https://github.com/search?q=repo%3Adotnet%2Fruntime+%22Category+does+not+exist%22&type=issues

[danmoseley]

Note

This analysis was generated with the assistance of GitHub Copilot.

This appears to be effectively intractable, as @ericstj's comment suggests. A few observations:

Why retries don't help: The root cause is in the Windows platform perf counter APIs themselves — there's inherent race/loss-tolerance that can cause the category to go missing. Once the static counter list is cached in-process without the category, RetryHelper just hits the same poisoned cache on every attempt (all 10 inner exceptions are identical).

Why a real fix is disproportionate:

1. Platform-level issue — the Win32 perf counter infrastructure has inherent flakiness
2. Cache poisoning — the category list is cached statically per-process, so once it's wrong, the process is stuck
3. net48 constraint — can't change Framework code, only test infrastructure

Possible mitigations:

• The test could use reflection to clear the static cache on PerformanceCounterLib between retries, so RetryHelper would actually get a fresh lookup. Ugly, but it would make retries meaningful rather than repeatedly hitting the same stale state.
• Mark the test [OuterLoop] for net48 if the noise becomes bothersome.
• Running each retry in a fresh process, though that's a heavier lift.

Current approach seems right: The "Known Build Error" with BuildRetry: true is the pragmatic answer given the very low hit rate (~1/month per the report).