Skip to content

ci(workflow): add trace-backed harness scoring - #2930

Open
quanruzhuoxiu wants to merge 5 commits into
mainfrom
ci/trace-backed-harness
Open

ci(workflow): add trace-backed harness scoring#2930
quanruzhuoxiu wants to merge 5 commits into
mainfrom
ci/trace-backed-harness

Conversation

@quanruzhuoxiu

@quanruzhuoxiu quanruzhuoxiu commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a centralized CI harness finalizer that produces immutable scorecards, Trace Bundles, task-level deep links, validator provenance, retry visibility, and secret-safe evidence
  • register scored cases separately from infrastructure and deterministic checks across AI, report, headless, desktop, Android, and iOS workflows
  • add nightly reusable-workflow orchestration and aggregate scoring
  • preserve report inputs between dependent AI stages and reject incomplete or unreproducible artifacts
  • add harness schemas and regression tests for report parsing, missing Trace, product versus infrastructure failures, secret scanning, snapshots, aggregation, and manifest integrity

Validation

Local:

  • pnpm run lint
  • npx biome check . --diagnostic-level=warn --no-errors-on-unmatched
  • pnpm run test:ci-harness
  • pnpm dlx github-actionlint .github/workflows/ai.yml .github/workflows/ai-unit-test.yml .github/workflows/headless-linux.yml .github/workflows/studio-headless-linux.yml .github/workflows/macos-desktop.yml .github/workflows/windows-desktop.yml .github/workflows/android-emulator.yml .github/workflows/ios-simulator.yml .github/workflows/ci.yml .github/workflows/nightly-harness.yml
  • bash -n packages/ios/scripts/run-simulator-ci-smokes.sh
  • git diff --check

PR-native workflows on the final SHA:

Artifact verification:

  • Windows final-SHA bundle is pass/complete; all 51 manifest hashes match and hidden transient files are absent
  • Studio bundle contains 8 tasks, 5 model calls, raw responses, screenshots, and task anchors; all 18 manifest hashes match
  • AI report app contains 74 tasks and 43 model calls; AI web contains two trace-complete scored cases with 200 and 202 tasks; all 207 combined manifest entries match
  • iOS bundle is pass/complete with no security finding after preventing the model API key from reaching the WebDriverAgent build process

Two earlier manual Headless runs caught the dark timeline assertion as a product failure while preserving complete Trace, request IDs, screenshots, raw responses, token usage, and reproducible manifests. The final PR-native rerun passed, so this is recorded as a flaky UI/model signal rather than a harness infrastructure failure.

The new Nightly Harness workflow cannot be manually registered until the workflow file exists on the default branch; its workflow graph is covered by actionlint and activates after merge.

@quanruzhuoxiu quanruzhuoxiu changed the title ci: add trace-backed harness scoring ci(workflow): add trace-backed harness scoring Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant