[Perf] Changes at 6/23/2022 3:07:42 AM

[performanceautofiler]

Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	c9007c8cd39a5b086a74dd5580b34311041a68ff
Compare	b1221494186e7b8e5199d892dae76481a52f5935
Diff	Diff

Regressions in System.Tests.Perf_Array

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ArrayCreate1D - Duration of single invocation	711.07 ns	752.41 ns	1.06	0.51	False
ArrayCreate2D - Duration of single invocation	820.07 ns	1.10 μs	1.34	0.45	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Array*'

Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.ArrayCreate1D

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 752.4128199910444 > 739.8678190409446.
IsChangePoint: Marked as a change because one of 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.898957955227843 (T) = (0 -861.1748272233496) / Math.Sqrt((1710.0033117917915 / (17)) + (10160.007105915962 / (27))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (27) - 2, .025) and -0.21203890831980196 = (710.5174770479601 - 861.1748272233496) / 710.5174770479601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCreate2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0955731673450162 > 908.3689450478162.
IsChangePoint: Marked as a change because one of 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.556458973808698 (T) = (0 -1015.5430890972272) / Math.Sqrt((4156.871813522927 / (16)) + (5996.65066957 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.16354175778541494 = (872.8033027625277 - 1015.5430890972272) / 872.8033027625277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	c9007c8cd39a5b086a74dd5580b34311041a68ff
Compare	b1221494186e7b8e5199d892dae76481a52f5935
Diff	Diff

Regressions in System.Collections.CtorGivenSize<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
HashSet - Duration of single invocation	571.20 ns	747.21 ns	1.31	0.35	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorGivenSize<Int32>*'

Details

Payloads

Baseline
Compare

Histogram

System.Collections.CtorGivenSize<Int32>.HashSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 747.2094765640251 > 657.9956804429103.
IsChangePoint: Marked as a change because one of 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -15.892641030585828 (T) = (0 -733.7988220913902) / Math.Sqrt((689.4109126429096 / (17)) + (582.4039946839043 / (27))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (27) - 2, .025) and -0.2058451647720238 = (608.5348629565734 - 733.7988220913902) / 608.5348629565734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	arm64
OS	ubuntu 18.04
Baseline	c9007c8cd39a5b086a74dd5580b34311041a68ff
Compare	b1221494186e7b8e5199d892dae76481a52f5935
Diff	Diff

Regressions in System.Collections.CtorGivenSize<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Array - Duration of single invocation	263.44 ns	332.15 ns	1.26	0.21	False
Queue - Duration of single invocation	303.32 ns	368.42 ns	1.21	0.43	False
ConcurrentDictionary - Duration of single invocation	573.65 ns	673.43 ns	1.17	0.05	False
Dictionary - Duration of single invocation	848.18 ns	1.11 μs	1.31	0.34	False
List - Duration of single invocation	297.09 ns	379.82 ns	1.28	0.33	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorGivenSize<String>*'

Details

Payloads

Baseline
Compare

Histogram

System.Collections.CtorGivenSize<String>.Array(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 332.14834526509645 > 278.1075854843768.
IsChangePoint: Marked as a change because one of 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -7.192439649073046 (T) = (0 -306.1594547317862) / Math.Sqrt((51.38983141448586 / (17)) + (802.64243604948 / (27))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (27) - 2, .025) and -0.15532494026622107 = (264.99856798836004 - 306.1594547317862) / 264.99856798836004 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorGivenSize<String>.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 368.41722588505974 > 327.9060521940012.
IsChangePoint: Marked as a change because one of 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.224843235881334 (T) = (0 -348.7085560198879) / Math.Sqrt((96.84803321103489 / (16)) + (840.2255186392941 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.1200686659422306 = (311.3278378576418 - 348.7085560198879) / 311.3278378576418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorGivenSize<String>.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 673.4290739403557 > 603.0174510858674.
IsChangePoint: Marked as a change because one of 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.442971744968753 (T) = (0 -639.8319755235564) / Math.Sqrt((84.46594798375126 / (16)) + (1551.713188440139 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.11458501277238708 = (574.0539915677284 - 639.8319755235564) / 574.0539915677284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorGivenSize<String>.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.113512486587661 > 910.1405445106928.
IsChangePoint: Marked as a change because one of 6/23/2022 3:07:42 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -10.043995163470557 (T) = (0 -1070.592533892537) / Math.Sqrt((1237.3139052224906 / (18)) + (8174.370453246307 / (26))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (26) - 2, .025) and -0.2249452798533347 = (873.9921296898432 - 1070.592533892537) / 873.9921296898432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorGivenSize<String>.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 379.8236735905387 > 312.4682148602441.
IsChangePoint: Marked as a change because one of 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -12.754067619824976 (T) = (0 -369.8771862601914) / Math.Sqrt((113.69068684439813 / (16)) + (485.95363722695106 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.2056055811400034 = (306.79783840287206 - 369.8771862601914) / 306.79783840287206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dakersnar]

@AndyAyersMS Do you think dotnet/runtime#70916 could be to blame here?

[AndyAyersMS]

@AndyAyersMS Do you think dotnet/runtime#70916 could be to blame here?

Unlikely.

I get this range dotnet/runtime@70df1f5...b122149 for

These perf swings seem to be pretty well correlated with GC count (note: x axis scale in the chart below is just ever so slightly different than the chart above), but I don't see any obvious suspects in the range.