Pre-download VS Code + Java pack in setup-steps; refine repro guidance

Add .github/scripts/prewarm-vscode.js and a copilot-setup-steps step that warms AutoTest's <repo>/.vscode-test cache (VS Code stable + vscjava.vscode-java-pack) before the agent firewall engages, so firewalled UI reproductions launch offline.

Refine repro/uitest guidance: separate reproduction from fix-proof (UI test's key value is red->green screenshots), require verifiers only on the decisive assertion step, and make PRs state repro method + execution status.

Author: wenytang-ms

.github/copilot-instructions.md

@@ -13,4 +13,4 @@
 - When asked to add, update, run, or debug UI/E2E coverage, prefer the AutoTest YAML workflow under `test/e2e-plans/`.
 - Use the `uitest` skill for UI test work. It should create or update `test/e2e-plans/*.yaml`, validate the plan, build the OSGi bundle and package the extension when needed, run AutoTest, and inspect `test-results/`.
 - Do not create legacy VS Code extension tests (`test/maven-suite`, `test/gui`) for UI coverage unless the user explicitly asks for that format.
-- Prefer deterministic AutoTest verifiers (`verifyTreeItem`, `verifyFile`, `verifyEditorTab`, `verifyClipboard`) over screenshot-only checks.
+- Prefer deterministic AutoTest verifiers (`verifyTreeItem`, `verifyFile`, `verifyEditorTab`, `verifyClipboard`) on the decisive assertion step; you do not need a verifier on every step. Use AutoTest screenshots to prove a fix (a red run before, a green run after) — but never as the sole pass/fail authority for the decisive assertion.

.github/instructions/uitest-plan.instructions.md

@@ -31,12 +31,13 @@ action: 'clickViewTitleAction "Java Projects" "Unlink with Editor"'
 
 ## Verification rules
 
-- Add deterministic verification to every meaningful step. The natural-language `verify` field is context for humans and failure analysis; it is not pass/fail authority by itself, and it is auto-passed when a plan runs with `--no-llm`.
+- You do **not** need a verifier on every step. Author the *actions* step-by-step, but gate pass/fail with a deterministic verifier only on the **decisive assertion step(s)** — the step that captures the reported bug — plus any step prone to a silent no-op (see the `expandTreeItem` / free-form action caveat above). Intermediate action steps can rely on AutoTest screenshots instead of their own verifier.
+- The natural-language `verify` field is context for humans and failure analysis; it is not pass/fail authority by itself, and it is auto-passed when a plan runs with `--no-llm`. So the decisive step **must** carry a deterministic verifier, or a `--no-llm` run is a false green.
 - Use `verifyTreeItem` (with `name:`, optional `exact: true`, and `visible: false` for absence) as the authoritative check for Java Projects tree state.
 - Use `verifyFile` after operations that create, modify, or delete files on disk (new type, export jar, permanent delete). VS Code can open duplicate editor tabs with stale buffers, so prefer file-content checks over editor checks after such operations.
 - Use `verifyEditorTab` to assert which file an action opened, and `verifyClipboard` for copy-path commands.
 - On state-check steps whose only assertion is a deterministic verifier, omit the `verify:` field to avoid false LLM failures.
-- Use screenshots only as diagnostics produced by AutoTest; do not make screenshots the only evidence of pass/fail.
+- Screenshots are AutoTest's evidence that an action ran and are the primary artifact for **proving a fix** (a red run before, a green run after). Do not make a screenshot the sole pass/fail authority for the decisive assertion — pair it with a deterministic verifier.
 
 ## Local validation commands
 

.github/scripts/prewarm-vscode.js

@@ -0,0 +1,95 @@
+#!/usr/bin/env node
+/*
+ * Pre-download VS Code + the Java extensions into AutoTest's cache BEFORE the
+ * Copilot coding agent's firewall is enabled.
+ *
+ * AutoTest (`@vscjava/vscode-autotest`) launches VS Code via `@vscode/test-electron`:
+ *   1. downloadAndUnzipVSCode(version)            -> <cwd>/.vscode-test/vscode-<...>
+ *   2. resolveCliArgsFromVSCodeExecutablePath()   -> --extensions-dir=<cwd>/.vscode-test/extensions
+ *   3. code --install-extension <id> --force      -> pulls Marketplace bits into that extensions dir
+ *
+ * The VS Code CDN (update.code.visualstudio.com) and the Marketplace are NOT on the
+ * Copilot agent's default firewall allowlist, so those network calls fail at run time.
+ * This script performs the exact same three operations during `copilot-setup-steps`
+ * (which runs before the firewall), so the caches are warm and the firewalled UI run
+ * hits them offline.
+ *
+ * Because `@vscode/test-electron` derives its cache from `process.cwd()`, this MUST run
+ * from the repository root — the same directory AutoTest runs from at agent time.
+ *
+ * Env overrides:
+ *   VSCODE_VERSION       VS Code channel/version to warm (default: "stable")
+ *   PREWARM_EXTENSIONS   comma-separated extension ids (default: "vscjava.vscode-java-pack")
+ */
+"use strict";
+
+const path = require("path");
+const cp = require("child_process");
+
+function resolveTestElectron() {
+  // Prefer the exact copy that the globally installed AutoTest uses, so the
+  // version and default-cache-path logic match the agent run byte-for-byte.
+  const candidates = [];
+  try {
+    const globalRoot = cp.execSync("npm root -g", { encoding: "utf-8" }).trim();
+    candidates.push(path.join(globalRoot, "@vscjava", "vscode-autotest"));
+    candidates.push(globalRoot);
+  } catch {
+    /* npm not on PATH — fall back to local resolution below */
+  }
+  candidates.push(process.cwd());
+  try {
+    const entry = require.resolve("@vscode/test-electron", { paths: candidates });
+    return require(entry);
+  } catch {
+    // Last resort: a plain require (works if it is a local dependency).
+    return require("@vscode/test-electron");
+  }
+}
+
+async function main() {
+  const version = process.env.VSCODE_VERSION || "stable";
+  const extensions = (process.env.PREWARM_EXTENSIONS || "vscjava.vscode-java-pack")
+    .split(",")
+    .map((s) => s.trim())
+    .filter(Boolean);
+
+  const { downloadAndUnzipVSCode, resolveCliArgsFromVSCodeExecutablePath } = resolveTestElectron();
+
+  console.log(`⬇️  Pre-downloading VS Code "${version}" into ${path.join(process.cwd(), ".vscode-test")} ...`);
+  const vscodePath = await downloadAndUnzipVSCode(version);
+  console.log(`✅ VS Code ready: ${vscodePath}`);
+
+  const [cli, ...baseArgs] = resolveCliArgsFromVSCodeExecutablePath(vscodePath);
+  const extensionsDir = baseArgs.find((a) => a.startsWith("--extensions-dir="))?.split("=")[1];
+  console.log(`📁 Extensions dir: ${extensionsDir ?? "(default)"}`);
+
+  let failures = 0;
+  for (const ext of extensions) {
+    console.log(`📦 Installing ${ext} (+ Extension Pack members) ...`);
+    try {
+      cp.execFileSync(cli, [...baseArgs, "--install-extension", ext, "--force"], {
+        stdio: "inherit",
+        timeout: 300_000,
+        env: { ...process.env },
+        shell: process.platform === "win32",
+      });
+      console.log(`✅ Installed ${ext}`);
+    } catch (e) {
+      failures++;
+      console.warn(`⚠️  Failed to install ${ext}: ${e.message}`);
+    }
+  }
+
+  if (failures > 0) {
+    // Non-fatal: a missing extension only degrades UI reproduction, and the agent
+    // can still fall back to the non-UI path. Surface it without aborting setup.
+    console.warn(`⚠️  ${failures} extension(s) failed to pre-install; UI reproduction may be degraded.`);
+  }
+  console.log("🎉 VS Code + Java extensions pre-warmed for AutoTest.");
+}
+
+main().catch((err) => {
+  console.error("❌ Pre-warm failed:", err);
+  process.exit(1);
+});

.github/skills/repro/SKILL.md

@@ -19,6 +19,11 @@ From the issue body (and the `bug_report` template fields) collect:
 
 ## 2. Decide: does this need a UI/E2E test?
 
+The reproduction and the fix-proof are two different questions — decide each:
+
+- **Reproduction** can often be non-UI or even a code read, especially for simple, obvious bugs. Prefer the cheapest reproduction that captures the report.
+- **Fix-proof** is where a UI/E2E test earns its cost: a red run before the fix and a green run after, with screenshots, is the strongest evidence for a user-facing bug. If the bug is user-facing, favour leaving a committed UI plan even when you first reproduced it another way.
+
 **Use a UI/E2E AutoTest plan (`uitest` skill) when the bug is in the user-facing surface**, e.g.:
 
 - Java Projects tree rendering, ordering, labels, icons, or node presence/absence.
@@ -57,7 +62,7 @@ npx @vscode/vsce package -o vscode-java-dependency.vsix
 npx -y @vscjava/vscode-autotest run test\e2e-plans\repro-issue-<n>.yaml --vsix vscode-java-dependency.vsix --no-llm --output test-results\repro-issue-<n>
 ```
 
-Author the plan so its deterministic verifier (`verifyTreeItem` / `verifyFile` / `verifyEditorTab` / `verifyClipboard`) asserts the **expected** behavior — it therefore **fails on the current (buggy) build**, capturing the bug. Inspect `test-results/repro-issue-<n>/results.json` and screenshots to confirm the failure matches the report.
+Author the plan step-by-step for the **actions**, but you do not need a verifier on every step — put a deterministic verifier (`verifyTreeItem` / `verifyFile` / `verifyEditorTab` / `verifyClipboard`) on the **decisive assertion step** (the one that captures the bug) and on any step prone to a silent no-op. That decisive verifier must assert the **expected** behavior, so it **fails on the current (buggy) build**. Inspect `test-results/repro-issue-<n>/results.json` and the screenshots to confirm the failure matches the report, and keep the red-run screenshot as before-fix evidence.
 
 **Non-UI path** — add the failing `test/maven-suite` or `jdtls.ext` test and run the existing suite (`npm test`, or the `jdtls.ext` Maven test) to confirm it fails.
 
@@ -66,16 +71,21 @@ Author the plan so its deterministic verifier (`verifyTreeItem` / `verifyFile` /
 1. Fix the product code (`src/**` for TS, `jdtls.ext/**` for the OSGi backend).
 2. **Rebuild and repackage the VSIX** (`npm run build-server` + `vsce package`) before rerunning any UI plan — never rerun against a stale VSIX.
 3. Rerun the reproduction; the same plan/test must now pass (red → green).
-4. Leave the reproduction committed as a permanent regression test. `.github/workflows/e2eUI.yml` discovers `test/e2e-plans/*.yaml` automatically, so `repro-issue-<n>.yaml` becomes its own CI check with no workflow edits.
+4. Keep both runs' evidence: the **before** (red) and **after** (green) screenshots plus the `results.json` reason. The green screenshot is the primary proof that the fix works — attach it (and the before/after pair) to the PR.
+5. Leave the reproduction committed as a permanent regression test. `.github/workflows/e2eUI.yml` discovers `test/e2e-plans/*.yaml` automatically, so `repro-issue-<n>.yaml` becomes its own CI check with no workflow edits.
 
 ## 6. Report back
 
-- **Reproduced + fixed**: open a PR citing the failing step / screenshot / `results.json` reason as evidence, and note that the committed reproduction now passes. Reference the issue.
+Every PR or comment must state **how you reproduced** (UI plan vs unit test vs code read) and the **execution status** (ran red→green with screenshots attached, or could not execute — e.g. the UI run was blocked — and why).
+
+- **Reproduced + fixed**: open a PR that attaches the before (red) and after (green) screenshots as the fix-proof, cites the failing step / `results.json` reason, and notes the committed reproduction now passes. Reference the issue.
 - **Reproduced, report only**: comment with the reproduction (plan or test), the observed vs expected behavior, and the exact failing step.
+- **Reproduced but could not run the UI test** (e.g. VS Code download / Marketplace blocked): commit the plan, explain what fails and why it could not execute, and either fall back to a non-UI proof or ask a maintainer to unblock — do not claim a green run you did not observe.
 - **Could not reproduce**: comment with what you tried and precisely what is missing; label `needs-more-info`. Do not fabricate a fix for an unreproduced bug.
 
 ## Environment notes
 
 - The Copilot coding agent environment is prepared by `.github/workflows/copilot-setup-steps.yml` (JDK 21, Node 20, AutoTest, Xvfb, a baseline VSIX). Assume these are present.
-- AutoTest downloads VS Code and installs `vscjava.vscode-java-pack` at run time. If those network hosts are blocked by the agent firewall, UI reproduction cannot launch — fall back to the non-UI path and note the limitation, or ask a maintainer to allow the VS Code download + Marketplace hosts.
+- That setup runs **before the agent firewall**, and its final step pre-downloads VS Code (stable) and the `vscjava.vscode-java-pack` extensions into AutoTest's `<repo>/.vscode-test` cache (via `.github/scripts/prewarm-vscode.js`). So the firewalled UI run should launch offline from that warm cache — you normally do **not** need to fetch VS Code or Marketplace bits yourself.
+- If the pre-warm did not run (e.g. an older branch) or the cache is cold, AutoTest will try to download VS Code + install `vscjava.vscode-java-pack` at run time. Those hosts (VS Code CDN + Marketplace) are firewall-blocked by default — if that happens, fall back to the non-UI path and note the limitation, or ask a maintainer to allow those hosts.
 - Always run AutoTest with `--no-llm` in the agent so pass/fail comes only from deterministic verifiers.

.github/workflows/copilot-setup-steps.yml

@@ -10,8 +10,12 @@ name: "Copilot Setup Steps"
 # Xvfb / GTK libraries required to launch VS Code headless.
 #
 # NOTE: This workflow only takes effect once it is on the default branch.
-# Reproducing UI tests additionally needs the Copilot firewall to allow the
-# VS Code download + Marketplace hosts — see .github/skills/repro/SKILL.md.
+#
+# Setup steps run BEFORE the Copilot agent firewall is enabled, so the final
+# step pre-downloads VS Code (stable) and the Java Extension Pack into AutoTest's
+# `<repo>/.vscode-test` cache. That warms the exact files AutoTest fetches from
+# the VS Code CDN + Marketplace at run time — hosts the firewall blocks — so the
+# firewalled UI reproduction launches offline. See .github/skills/repro/SKILL.md.
 
 on:
   workflow_dispatch:
@@ -67,3 +71,10 @@ jobs:
         # pay the full build cost. Copilot must repackage after editing src/** or
         # jdtls.ext/** before rerunning a plan against a stale VSIX.
         run: vsce package -o vscode-java-dependency.vsix
+
+      - name: Pre-download VS Code and Java extensions (before firewall)
+        # Warms <repo>/.vscode-test so the firewalled agent run does not need the
+        # VS Code CDN or Marketplace. Best-effort: a failure here only degrades UI
+        # reproduction, so it must not block Copilot from starting work.
+        continue-on-error: true
+        run: node .github/scripts/prewarm-vscode.js