feat(app): unified Studio installation permissions for DAB and Marketplace

[laurencewells]

Note

Rebased onto main after #1564 ("Fix run config table integration", which moves oversized run-config staging from Lakebase to Delta). The task runner no longer needs a Lakebase role: postgres_roles.task_runner_sp / DQX_TASK_RUNNER_POSTGRES_ROLE are gone from the bundle and the docs.

Warning

Breaking change, with no upgrade path. Existing DQX Studio installations (DAB or Marketplace) must be removed and reinstalled. See app/DEPLOYMENT.md → "Removing a previous installation".

Changes

DAB deployment and Marketplace installation now share one verified permission model with minimal admin setup. Setup reports READY only when every required permission is verified.

Storage and configuration

• Studio derives its Unity Catalog storage from an existing catalog plus a validated prefix (default dqx_studio): <catalog>.<prefix> (with the wheels volume inside it), <prefix>_tmp, <prefix>_genie and <prefix>_demo. The Lakebase schema stays independent of the prefix.
• Marketplace binds only the SQL warehouse (now CAN_MANAGE) and Lakebase. The volume binding is gone. A new admin setup form collects catalog, prefix and audience group. The choices are saved in Lakebase, can be edited until storage is provisioned, and are locked after that.
• DAB supplies the same configuration through bundle variables. make app-deploy gains STUDIO_PREFIX= and STUDIO_USER_GROUP= arguments. STUDIO_USER_GROUP=users selects broad mode: the workspace ACLs use users and the UC grants use account users. Broad mode is DAB-only.

Setup orchestrator

• Lakebase is bootstrapped before any UC storage exists.
• The steps run in this order: identity → lakebase → configuration → unity_catalog → storage → warehouse → task_runner → wheels → migrations → activation → access → app_sharing.
• Every grant and ACL is applied and then re-read. A missing grant and a grant that can't be inspected each block readiness, with different codes. Each blocking step shows the exact GRANT statements to fix it.
• Storage: Marketplace provisions schemas and the volume as the app SP. DAB verifies the declared resources. Studio refuses to take over a schema it doesn't manage.
• Access: applies and verifies the audience's and a custom admin group's grants: _tmp USE SCHEMA/CREATE TABLE, _genie USE SCHEMA plus SELECT on exactly the 5 approved views and 2 metadata dimensions, and _demo USE SCHEMA/SELECT. It also verifies configured Genie space and dashboard ACLs. Workspace admins is never a UC grantee.
• App sharing: blocks when the app ACL is readable and the audience is missing. If no identity can read it, it shows a non-blocking warning, which admins see in a banner.
• Recheck behaviour: an admin's "Verify again" re-runs every check, including on a READY install. Unattended restarts reuse the last full verification, but only for catalog-level grants the app SP can't inspect.
• Members of workspace admins or DQX_ADMIN_GROUP can run setup. Authorization uses a fresh OBO lookup.

Least privilege

• Runner SP: no Lakebase role or grants (oversized run configs are staged in the Delta dq_run_configs table, covered by its main-schema grants); USE CATALOG; main schema USE SCHEMA/SELECT/MODIFY; _tmp USE SCHEMA + schema-level SELECT (the runner can read any view in _tmp; the per-view runner GRANT SELECT ON VIEW is removed); wheels READ VOLUME; nothing on _genie/_demo.
• App SP: an explicit privilege list that includes MANAGE on the Studio schemas. The bundle engine silently drops MANAGE when it is combined with ALL_PRIVILEGES, hence the explicit list. It also gets warehouse CAN_MANAGE.

Docs

• app/DEPLOYMENT.md, DEVELOPMENT.md, README.md, AGENTS.md and CUSTOMER_QA.md, plus the Studio quickstart, permissions and installation pages.
• Coverage: permission matrix, minimal setup steps, broad mode, admin access, manual app sharing, and per-user boundaries.
• Installation guide (DAB route): prefix / studio_user_group via CLI --var, make and PowerShell -BundleVars; broad mode via --var needs studio_uc_principal; custom admin group USE CATALOG; admin verification and role-assignment steps; prefix reuse on upgrade. Studio SQL examples use <catalog>.<prefix>.

Linked issues

Builds on #1564 (merged; this branch is rebased onto it).

Tests

• manually tested: live DAB acceptance on a dev workspace, results below
• added unit tests (backend: 4974 passing; UI: 957 passing)
• added integration tests (Marketplace-path storage provisioning; collected, not yet run live)
• added end-to-end tests
• added performance tests

Verification at head (after the rebase): make lint (pylint 10/10), make app-check (tsc + basedpyright 0 errors + 957 UI tests), make app-test 4974 passed (fewer than before the rebase because #1564 removed the Lakebase staging tests), root tests/unit/test_app_backend.py 143 passed, make app-check-marketplace 35 passed, make docs-build passes, all commits GPG-signed.

Live DAB acceptance (fresh workspace, scoped audience group; run before the rebase onto #1564):

Check	Result
Fresh deploy, then setup prints the exact catalog GRANTs, then READY	✅ (the first deploy hit a one-time Lakebase role 404 race on the since-removed runner role; re-running deploy fixed it; the troubleshooting entry now names postgres_roles.app_sp, which isn't confirmed to hit the same race)
Grants match the permission matrix (audience, runner, app SP MANAGE)	✅
Revoking audience USE CATALOG blocks with the exact GRANT; re-granting returns READY	✅
App restart comes back READY with no admin action	✅
Task runner runs (demo seed, 10 runs) as the separate runner SP with prefix-volume wheels under least privilege	✅ all succeeded
App sharing check	⚠️ warning: ACL unreadable by app SP and admin OBO; the bundle declares the share
Bundle ACL entries for workspace admins accepted	✅

Live testing found and fixed four issues:

• Group grants were reported as uninspectable instead of missing.
• The bundle dropped MANAGE when it was combined with ALL_PRIVILEGES.
• Every restart caused an outage until an admin verified again.
• The configuration view showed an empty prefix.

Not yet exercised:

• A live deploy after the rebase onto Fix run config table integration #1564, including the runner reading staged oversized configs from Delta.
• Non-admin author OBO flows and an unauthorized user (needs extra test identities).
• Broad-mode deploy and warehouse rebind.
• A Marketplace-equivalent custom template and a real Marketplace listing install.
• The new provisioning integration test, run live.

Known limitations:

• Every bundle deploy recreates the app SP's Lakebase role. This drift is not caused by this branch.
• make.ps1 has no STUDIO_* arguments; the docs show the -BundleVars equivalent.
• A catalog grant revoked outside Studio for a principal the app SP can't read stays unnoticed until an admin chooses "Verify again".

Documentation and Demos

• added/updated demos
• added/updated docs
• added/updated agent skills

This pull request and its description were written by Isaac.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.75%. Comparing base (b645540) to head (49ed510).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1577       +/-   ##
===========================================
+ Coverage   79.39%   92.75%   +13.36%     
===========================================
  Files         142      142               
  Lines       14210    14210               
  Branches      151      151               
===========================================
+ Hits        11282    13181     +1899     
+ Misses       2859      959     -1900     
- Partials       69       70        +1     

Flag	Coverage Δ
anomaly	50.91% <ø> (ø)
anomaly-serverless	50.92% <ø> (ø)
integration	47.39% <ø> (?)
integration-serverless	48.58% <ø> (?)
mcp	80.44% <ø> (+0.09%)	⬆️
unit	66.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

✅ 1097/1097 passed, 52 skipped, 5h5m49s total

_{Running from acceptance #6135}

[github-actions]

✅ 1/1 passed, 24m58s total

_{Running from mcp #884}

[github-actions]

✅ 195/195 passed, 1 skipped, 7h27m53s total

_{Running from anomaly #2249}