Skip to content

Helix Sdk MTP/xUnit work items unconditionally pass --roll-forward Major which can silently mask missing runtimes #17000

Description

@Evangelink

Spun off from review feedback on #16986 (comment) by @Youssef1313.

Problem

In CreateMTPWorkItems (and previously in CreateXUnitV3WorkItems) the emitted Helix work item command unconditionally passes --roll-forward Major:

string command = `$"{PathToDotnet} exec --roll-forward Major " +`
    `$"--runtimeconfig {assemblyBaseName}.runtimeconfig.json " +`
    `$"--depsfile {assemblyBaseName}.deps.json " +`
    `$"{assemblyName} {reporterArgs}"` + ...

Quoting @Youssef1313:

This [is] extremely extremely concerning to me. Why do we roll forward to Major?

I'm reading this as ""if I have a test project that targets net8.0 and net9.0, and .NET 8 runtime wasn't found, then both TFMs will run with .NET 9, causing silent loss of coverage.""

This behavior is identical to what was in the previous CreateXUnitV3WorkItems task, so #16986 is not the cause - it's a pre-existing issue that the rename inherited.

Proposed direction

  • Investigate whether --roll-forward Major is still required (it was likely added because Helix queues did not always have an exact runtime match available; we should confirm whether queue images now ship a matching runtime per TFM).
  • If unconditional Major roll-forward is no longer needed, drop it (or make it opt-in via metadata on MTPProject / a property).
  • If it is still needed for some queues, narrow the policy (e.g. Minor or LatestMinor) to avoid silently running net8.0 work items on a net9.0 runtime.

Impact

Silent loss of multi-TFM coverage is the worst-case outcome - failures look green because the assembly happily runs on a newer runtime instead of failing fast.

cc @Youssef1313 @ilyas1974

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions