TypeAgent Studio: corpus capture + Impact Report regression verdict

[TalZaccai]

Closes the "find a regression" workflow end-to-end: capture a real corpus from
interaction logs, replay two versions of an agent, and read a red/green Impact
Report that says whether anything regressed.

Critical changes

• Corpus capture (agent-agnostic). A core transform turns interaction/display
  logs into portable in-repo *.utterances.jsonl files; imports auto-promote
  into the shared corpus and show up in the Corpora tree. Replaces the manual
  feedback-entry UI. player is just the first capture — it's the labelled set
  the bar is scored on.
• Regression predicate (likelyRegression). Classifies each replayed row as
  regression / improvement / benign / neutral from the delta shape (works on a
  fresh edit with no feedback). Feedback, when present, overrides the guess — but
  only when the replayed side reproduces the exact action the rating was recorded
  against, so a stale thumbs-down for a now-different action never drives the
  verdict.
• Impact Report verdict. A verdict banner answers "did anything regress?" and
  each row carries an Impact column, both driven by the same likelyRegression
  call the benchmark measures — so UI and evidence can't diverge.
• Impact Report table controls. Non-exclusive filter chips (status + impact)
  with live counts and select/deselect-all; clickable column-header sorting with
  a direction chevron; and an utterance filter (funnel icon → live VS Code input
  box).
• Agent-level replay + durable last run. Replay a whole corpus for an agent;
  the last run persists in workspace state and restores on reopen.

Supporting changes

• Capped reports order rows regression-first before truncating, so a reopened
  report still surfaces every likely regression.
• The predicate benchmark logs each row whose verdict disagrees with its label,
  so a drop points at the offending deltas. The two current misses are semantic
  (a band matched as a genre, a mangled genre param) — the predicate reasons
  about delta shape, not param correctness, and those two are why the score is
  92%, not a circular 100%.
• Design docs for the corpus-capture path and the observation-scoped feedback
  binding.

Testing

Studio view-model suite (48) and core replay suites green — including the
predicate benchmark (~92% agreement over 25 fixture deltas, bar 80%), predicate
unit specs, resolver specs for the feedback-binding rule, and corpus
transform/import specs. Type-check clean (only pre-existing unrelated errors),
Prettier clean, VSIX builds and installs.

55 files changed (+5,513 / −748), 21 commits. Targets main directly.

[Copilot]

Pull request overview

This PR completes an end-to-end “find a regression” workflow in TypeAgent Studio by (1) capturing portable corpora from interaction/display logs, (2) replaying two agent versions, and (3) presenting a regression verdict in the Impact Report driven by a benchmarked likelyRegression predicate.

Changes:

• Add corpus capture/import pipeline (displayLog.json → *.utterances.jsonl) and update the Corpora tree to be file-backed (one row per backing file) with open-file + auto-refresh on save.
• Introduce and benchmark a likelyRegression predicate (delta-shape + observation-scoped feedback override) and wire it into replay/Impact Report presentation (banner + per-row Impact).
• Add Impact Report UX controls (unified chips with live counts, sorting, utterance filter via host input box) plus durable “last run” persistence and live refresh of an open report.

Show a summary per file

File	Description
ts/packages/typeagent-studio/src/webviewKit/protocol.ts	Extend host↔webview protocol (run provenance, runAt, utterance search, external refresh).
ts/packages/typeagent-studio/src/test/webviewProtocol.spec.ts	Update protocol parsing tests for echoed resolved versions.
ts/packages/typeagent-studio/src/test/replayViewModel.spec.ts	Expand Impact Report view-model tests (filters/search/sort/verdict banner).
ts/packages/typeagent-studio/src/test/feedbackInputPresentation.spec.ts	Remove tests for retired feedback input flow.
ts/packages/typeagent-studio/src/test/corpusTreePresentation.spec.ts	Update corpus tree presentation tests for file-backed grouping.
ts/packages/typeagent-studio/src/studioServiceClient.ts	Add RPC client method for corpus import from logs.
ts/packages/typeagent-studio/src/serviceRuntimeFacade.ts	Expose repo-root info via CorpusSource and add corpus import method.
ts/packages/typeagent-studio/src/impactReportView.ts	Add durable last-run restore, external live refresh, utterance-filter host input box.
ts/packages/typeagent-studio/src/impactReportStore.ts	New workspaceState store for per-agent last run (row-capped, regression-first).
ts/packages/typeagent-studio/src/feedbackInputPresentation.ts	Remove feedback input shaping module (command retired).
ts/packages/typeagent-studio/src/extension.ts	Add corpus-file open/import commands, corpus tree refresh-on-save, replay→persist + live report refresh.
ts/packages/typeagent-studio/src/corpusTreeProvider.ts	Render corpus file rows with resourceUri + resolve existing in-repo file on disk.
ts/packages/typeagent-studio/src/corpusTreePresentation.ts	Rework corpus grouping to file-backed sources; remove feedback badge handling.
ts/packages/typeagent-studio/README.md	Update docs for corpus tree + replay/Impact Report last-run behavior; remove feedback section.
ts/packages/typeagent-studio/package.json	Update contributed commands/menus/activation events for new corpus import/open commands.
ts/packages/typeagent-studio/media/impactReport.css	Add verdict/impact styling, sortable headers, utterance funnel icon, docked detail pane layout.
ts/packages/typeagent-core/test/regressionPredicateBenchmark.spec.ts	Add predicate benchmark spec with threshold + authenticity regen checks (when git available).
ts/packages/typeagent-core/test/regressionBenchmarkHarness.ts	Add harness to generate real deltas via variant grammar commits.
ts/packages/typeagent-core/test/predicate.spec.ts	Add unit tests for structural likelyRegression behavior + feedback overrides.
ts/packages/typeagent-core/test/grammarReplayResolver.spec.ts	Add tests enforcing observation-scoped feedback attachment rule.
ts/packages/typeagent-core/test/fixtures/regression-variants/v9-wrong-enrichment.agr	New grammar variant fixture for benchmark delta generation.
ts/packages/typeagent-core/test/fixtures/regression-variants/v6-over-broad-genre.agr	New grammar variant fixture for benchmark delta generation.
ts/packages/typeagent-core/test/fixtures/regression-variants/v5-gained-good.agr	New grammar variant fixture for benchmark delta generation.
ts/packages/typeagent-core/test/fixtures/regression-variants/v4-skip-to-previous.agr	New grammar variant fixture for benchmark delta generation.
ts/packages/typeagent-core/test/fixtures/regression-variants/v2-lost-track-number.agr	New grammar variant fixture for benchmark delta generation.
ts/packages/typeagent-core/test/fixtures/regression-variants/v1-lost-transport.agr	New grammar variant fixture for benchmark delta generation.
ts/packages/typeagent-core/test/corpus.spec.ts	Update corpus federation expectations; add tests for in-repo sourceUri stamping + feedback stripping on promote.
ts/packages/typeagent-core/test/captureTransform.spec.ts	Add unit tests for display log → corpus transform.
ts/packages/typeagent-core/test/captureImport.spec.ts	Add tests for display log import into in-repo corpus (dedupe/idempotency/allowlist).
ts/packages/typeagent-core/src/runtime/studioRuntimeCore.ts	Add runtime API types + implementation for importCorpusFromLogs.
ts/packages/typeagent-core/src/runtime/studioProtocol.ts	Extend studio RPC protocol with importCorpusFromLogs.
ts/packages/typeagent-core/src/replay/predicate.ts	New likelyRegression predicate implementation.
ts/packages/typeagent-core/src/replay/index.ts	Re-export predicate from replay package barrel.
ts/packages/typeagent-core/src/replay/grammarResolver.ts	Implement observation-scoped feedback attachment (only when action reproduced).
ts/packages/typeagent-core/src/corpus/types.ts	Add provenance.rawSourceUri; clarify captures as transient staging.
ts/packages/typeagent-core/src/corpus/index.ts	Export capture transform/import helpers.
ts/packages/typeagent-core/src/corpus/fileCorpusService.ts	Make captures staging private (not federated); sanitize in-repo storage; avoid capture filename collisions.
ts/packages/typeagent-core/src/corpus/captureTransform.ts	New pure transform: display-log entries → corpus entries.
ts/packages/typeagent-core/src/corpus/captureImport.ts	New import pipeline: read display logs → stage → promote into in-repo corpus.
ts/packages/studio-service/src/studioRpcHandlers.ts	Wire new importCorpusFromLogs RPC handler.
ts/docs/plans/vscode-devx/STATUS.md	Update plan status/capability matrix for Gate C closure work.
ts/docs/plans/vscode-devx/README.md	Rename predicate phrasing to “likely regression” in plan docs.
ts/docs/plans/vscode-devx/QUICKSTART.md	Update quickstart for corpus tree + replay/Impact Report behavior; remove feedback steps.
ts/docs/plans/vscode-devx/impact-report-multi-variant-design.md	Update terminology to “likely regression”.
ts/docs/plans/vscode-devx/DESIGN.md	Update Impact Report narrative to “likely regression”.
ts/docs/plans/vscode-devx/corpus-capture-design.md	Add corpus capture design doc detailing transform/import + feedback semantics.
ts/docs/plans/vscode-devx/05-implementation-plan.md	Update predicate decision wording.
ts/docs/plans/vscode-devx/04-mvp-slice.md	Update demo script + Gate C wording to “likely regression”.
ts/docs/plans/vscode-devx/03-features.md	Rename feature to “Likely-regression predicate”.
ts/docs/plans/vscode-devx/02-journeys.md	Update journey language to “likely regressions”.
ts/docs/plans/vscode-devx/01-inventory.md	Update inventory implications wording for regression/improvement.
ts/corpus/player.regression-benchmark.jsonl	Add committed blind-labelled benchmark delta set for predicate measurement.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 55/55 changed files
• Comments generated: 3
• Review effort level: Low

[Copilot]

Review details

Comments suppressed due to low confidence (1)

ts/docs/plans/vscode-devx/STATUS.md:62

• STATUS.md is internally inconsistent about Gate C: it claims the work “closes the headline "find a regression" gate”, but the Gate C row still says “Impact Report wiring ❌” and the next paragraph says “Critical path now: close Gate C”. Please reconcile these status statements so readers don’t get conflicting guidance.

| **C** | J4 Find a regression | Impact Report ≥ 80% red/green on hand-labelled `player` | P-3   | 🟡 **long pole** — engine L1–L4a ✅; corpus ✅; predicate + measurement ✅ (detector agrees with human labels at 92% on a fixture-delta benchmark); Impact Report wiring ❌ |
| **D** | J5 Debug a trace     | Trace Viewer                                            | P-3   | ❌ not started                                                                                                                                                              |
| **E** | J6 Observe live      | Live Trace + status bar                                 | P-5   | ❌ not started                                                                                                                                                              |

**Critical path now: close Gate C** — corpus capture (the agent-agnostic capture
path; `player` is just the anchor set the gate is scored on) → predicate tuning →
run the ≥ 80% validation. It is the only _tunable_ gate and §7's top risk.

• Files reviewed: 55/55 changed files
• Comments generated: 1
• Review effort level: Low