[BUG] e2e run_dbconnect magic-comments test stalls in headless CI (Run All does not advance past first DBConnect cell)

[rugpanov]

Summary

The e2e test should run a databricks notebook with dbconnect and handle magic comments (in packages/databricks-vscode/src/test/e2e/run_dbconnect.ucws.e2e.ts) consistently fails in the headless CI harness and has been quarantined with it.skip to keep the suite green.

It's a harness artifact, not a product bug

Jupyter: Run All Cells executes the first DBConnect cell and then does not advance to the remaining cells — the %sql/%run cells stay <not run> and their output files are never written.

Verified this does not reproduce for a real user:

• On a real machine + serverless DBConnect, Run All runs all cells and writes every output file.
• Running the notebook's cells one-by-one through the kernel locally produces magic-notebook-output.json (hello; world) and databricks-run-notebook-output.json (hello run;).

So the stall is specific to the webdriver/xvfb harness driving VS Code.

Approaches tried (none resolved it)

1. Drive the Interactive Window cells by index — not possible. The .py # Databricks notebook source file runs in an Interactive Window, whose only execute command (interactive.execute) always reads the input box and appends+runs a new cell; there's no supported way to run an already-present IW cell by index (notebook.cell.execute on the vscode-interactive doc hangs).
2. Rewrite as a real .ipynb + drive cells via notebook.cell.execute — the first cell (run by Run All) completes, but a subsequently-driven cell never reaches a terminal executionSummary in the harness. Reproduced with a plain spark.sql(...) cell (no %sql, no bare-DataFrame display) — so it is not the %sql→_sqldf→df_html display formatter. The failure is invisible from the test (no cell state, no output) and cannot be reproduced locally (no headless renderer).

Coverage in the meantime

The %sql/%run / # COMMAND / # MAGIC transform logic remains covered by the unit test packages/databricks-vscode/src/test/python/databricks_magics_transformer_test.py.

Next steps for whoever picks this up

Needs someone who can run/debug the e2e harness interactively (or the Jupyter/notebook execution owner): determine why a notebook.cell.execute-driven cell (or Run All's cell chaining) does not complete in the headless renderer while it completes for a real user. A standalone repro project + a one-by-one kernel runner exist (ask @rugpanov).

Related: the diagnostics that made this diagnosable landed in #2013.

[rugpanov]

Duplicate of #2024 (same root cause: serverless DBConnect 'Run All' executes only the first cell in the headless CI harness). Consolidating there — I've added my extra findings (real-machine/one-by-one repro proving CI-renderer-only; IW-undrivable; .ipynb driven-cell stall ruling out the df_html formatter) as a comment on #2024.