test(e2e): de-flake run_dbconnect setup and notebook run-all

[anton-107]

Summary

De-flakes run_dbconnect.ucws on both shards. Two distinct issues.

Windows: "should setup virtual environment" (+ cascade to 3 run tests)

Windows venv creation + dependency install is far slower than Linux, and the "environment is selected" / "Databricks Connect" prompt toasts frequently expired the fixed 60s waitForNotification before firing, failing setup and cascading to the three run tests.

• Gate setup on the filesystem ground truth — the .venv interpreter existing (240s budget) — instead of a transient toast.
• Make the "Databricks Connect" install prompt tolerant, and fall back to the Reinstall Databricks Connect command when the toast never shows.
• Because the venv-exists poll now precedes ensureVenvHasKernelDeps, the kernel deps reliably land in the interpreter the kernel launches, fixing the Windows "requires the notebook package" kernel failure.

Linux: "run a notebook" / "handle magic comments"

"Run All" intermittently stops after the first cell (later cells stay <not run>), so the output-file assertions time out silently after 120s/180s. Root cause of the stall is not yet established, so rather than mask it:

• Add a selector-independent cell-completion gate: after Run All, wait for every code cell to leave <not run>. This converts the silent timeout into a precise "<n>/<m> cells never ran" signal so a genuine run-all chaining stall surfaces instead of masquerading as a missing file. The gate is read-only and best-effort — it never changes pass/fail on its own.

Also threads an optional timeout into waitForNotification.

Found via the nightly run (https://github.com/databricks-eng/eng-dev-ecosystem/actions/runs/29475140909).

This pull request and its description were written by Isaac.

[rugpanov]

🤖 Integration tests ❌ 6 of 35 test jobs failed for d5d237ed (29 passed).
View run

[rugpanov]

Please verify that the fix has helped on the IT.

[anton-107]

CI evidence (vscode-isolated-pr run 29494397198)

This PR targets run_dbconnect.ucws.e2e.ts. On the Windows shard it converts a 4-test cascade failure into a single, cleanly-diagnosed failure:

Test	Before (baseline nightly)	This PR
should setup virtual environment	❌ (Databricks Connect toast never seen)	✅ fixed
should run a python file with dbconnect	❌ (cascade)	✅
should run a notebook with dbconnect	❌ (cascade)	✅
should run a databricks notebook … magic comments	❌	❌ (cluster 4b, see below)

Cluster 4a (Windows setup) — fixed. The fs-gate on the .venv interpreter + tolerant Databricks Connect prompt worked end-to-end: the log shows databricks-connect-17.3.12 installed and the tree reaching Python Environment item label: Databricks Connect: 17.3.12, and ✓ should setup virtual environment now passes (it was the primary failure that cascaded into the three run tests).

Cluster 4b (Run All stops after the first cell) — now precisely diagnosed instead of a silent timeout. The new cell-completion gate produced exactly the intended signal:

Run All did not advance through all cells within 180000ms: 3/4 still <not run> —
cells: [#1 order=1 success=true; #2 order=<not run>; #3 order=<not run>; #4 order=<not run>]

So the remaining handle magic comments failure is confirmed to be a genuine "Run All" chaining stall (cell #1 succeeds, cells #2–#4 never start) — a real product-behavior question, not a slow-output or parsing artifact. This PR intentionally surfaces it rather than masking it; the underlying stall is tracked separately and is out of scope here.

The other red shards in this run are pre-existing and unrelated to this PR: deploy_and_run_job (the CLI run-URL parse bug fixed in #2018, not in this branch), run_files cancel-button flake, and the Linux run_dbconnect (same 4b).

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 2021
• Commit SHA: 0817d0b2338f158c46702f1ecfdba8315b0d369b

Checks will be approved automatically on success.