You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5e2999c
Browse filesBrowse the repository at this point in the historyBrowse files
docs(app): drop runner Lakebase prerequisite after #1564 and fill Studio install gaps
The runner stages oversized configs in the Delta dq_run_configs table since
#1564, so it needs no Lakebase role or grants; remove the remaining guidance.
Also document prefix/audience bundle variables (CLI, make, PowerShell),
broad mode via --var, admin verification and role assignment for the DAB
route, prefix reuse on upgrade, and use <prefix> in Studio SQL examples.
Co-authored-by: Isaac <no-reply@databricks.com>
Copy file name to clipboardExpand all lines: app/DEPLOYMENT.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -367,7 +367,7 @@ The script requires `uv`, Node.js 18+, yarn classic v1, and Databricks CLI v1.4.
367
367
2. `databricks bundle deploy` — provisions or updates the schemas, wheels volume, Lakebase project (+ endpoint + the app SP's Postgres role), the SQL warehouse, the task-runner job, and the Databricks App in dependency order, and applies **bundle-declared grants** via the `grants:` / `permissions:` blocks in `databricks.yml`. Stateful resources carry `lifecycle.prevent_destroy: true` so a future destroy can't drop them — see [Step 3](#step-3-stateful-storage-and-destroy-protection).
368
368
3. `databricks bundle run` — starts the app.
369
369
370
-
Remember the manual prerequisites: the [catalog grants](#the-use-catalog-prerequisite) for the app SP (`USE CATALOG`, `CREATE SCHEMA`), task-runner SP, and audience, plus [runner Lakebase role and grants](#task-runner-lakebase-access) for the current oversized-config path. Cold-start UC checks must pass before Studio is ready; they do not verify runner Lakebase access.
370
+
Remember the manual prerequisites: the [catalog grants](#the-use-catalog-prerequisite) for the app SP (`USE CATALOG`, `CREATE SCHEMA`), task-runner SP, and audience. Cold-start UC checks must pass before Studio is ready. The task runner needs no Lakebase role or grants.
371
371
372
372
> **First start**: The app runs Delta analytical and Lakebase application migrations on startup, and publishes the task-runner wheel to the UC volume. Wait for the setup checks to report that wheel publishing is ready before triggering runs. Also wait for `"Lakebase OLTP routing enabled"` before opening the UI. If Lakebase initialization fails, the app refuses to start and the Apps platform restarts the container. It never falls back to Delta-backed application state.
databricks bundle deploy -p <your-profile> --force # or force deploy
511
511
```
512
512
513
-
**First deploy fails with `cannot create resources.postgres_roles.task_runner_sp: Project with name 'projects/<id>' not found (404)`:**
513
+
**First deploy fails with `cannot create resources.postgres_roles.app_sp: Project with name 'projects/<id>' not found (404)`:**
514
514
This happens on a fresh workspace right after the Lakebase project is created (eventual consistency). Re-run `make app-deploy`; the second run succeeds.
Copy file name to clipboardExpand all lines: app/README.md
+1-3Lines changed: 1 addition & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -81,9 +81,7 @@ The setup workflow applies the grants and ACL updates it has authority for, then
81
81
82
82
The audience is a dedicated existing group, or `users` in DAB **broad mode** (workspace ACL `users` plus Unity Catalog `account users`, which is account-wide; the Marketplace form never accepts it). Members of `DQX_ADMIN_GROUP` or the workspace `admins` group may run setup; Unity Catalog cannot grant to `admins`, so UC grants for administrators go only to a custom admin account group, and with `admins` administrators must also be audience members to use OBO features. Per-user SQL entitlement, OAuth consent, and source-data privileges are checked for the active user, never for the group.
83
83
84
-
On every cold startup, setup resolves the task-runner job's actual `run_as`, grants it `USE SCHEMA` on the main and temporary schemas, `READ VOLUME` on the wheels volume, and schema-level `SELECT` / `MODIFY` on the main schema (never `ALL PRIVILEGES`, catalog, Genie, or demo access), and verifies these plus its `USE CATALOG`. Schema grants cover current and future tables; table-specific grants alone do not satisfy setup. Runner Lakebase roles and privileges are not checked by setup.
85
-
86
-
The current oversized-config staging implementation still needs a Lakebase administrator to create/map the runner's OAuth `SERVICE_PRINCIPAL` role and apply scoped grants manually. Changing that staging implementation is separate work; removing its setup checks does not remove the runtime requirement. Setup applies no automatic runner PostgreSQL grants. The Postgres username must match the resolved Jobs runner client ID; a legacy `DQX_TASK_RUNNER_POSTGRES_ROLE` override must also match. Never grant superuser membership to the runner. See [Deployment](DEPLOYMENT.md#task-runner-lakebase-access) for the UI and SQL steps.
84
+
On every cold startup, setup resolves the task-runner job's actual `run_as`, grants it `USE SCHEMA` on the main and temporary schemas, schema-level `SELECT` on the temporary schema (how it reads OBO temp views), `READ VOLUME` on the wheels volume, and schema-level `SELECT` / `MODIFY` on the main schema (never `ALL PRIVILEGES`, catalog, Genie, or demo access), and verifies these plus its `USE CATALOG`. Schema grants cover current and future tables; table-specific grants alone do not satisfy setup. The runner needs no Lakebase access: run configs too large for job parameters are staged in the Delta `dq_run_configs` table in the main schema, which the main-schema `SELECT` / `MODIFY` grants already cover.
87
85
88
86
Genie audience access requires space `CAN_RUN`, Consumer access or Databricks SQL access entitlement, parent usages, and only the five approved view plus two dimension-table grants, never whole-schema `SELECT` or access to the entitlement table. Genie uses embedded compute credentials; Studio's OBO SQL workflows additionally require SQL access entitlement and warehouse `CAN_USE`. Both DAB and Marketplace use the `genie` OAuth scope. Warehouse rebind re-runs the warehouse checks and reconciles Genie compute. See [the deployment grants reference](DEPLOYMENT.md#grants-reference).
Copy file name to clipboardExpand all lines: docs/dqx/docs/installation.mdx
+9-5Lines changed: 9 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -519,7 +519,7 @@ If you can't use the Marketplace, or want to customize the deployment, use the [
519
519
- `catalog_name`— Unity Catalog catalog where the studio's schemas and volume live (must already exist).
520
520
- `dqx_service_principal_application_id`— the task-runner SP you created in step 2.
521
521
522
-
The bundle manages its own SQL warehouse and Lakebase project. Pass the two values with `--var` in step 4; other settings can also be overridden with `--var name=value`. Two optional settings control storage and audience: `prefix`(default `dqx_studio`) derives the schema names, and `studio_user_group` (default `dqx-studio-users`, which must already exist) is the audience. With `make app-deploy` these are the `STUDIO_PREFIX=<prefix>` and `STUDIO_USER_GROUP=<group>` arguments. Set `STUDIO_USER_GROUP=users` for **broad mode**, which shares Studio with the workspace `users` group and grants Unity Catalog access to `account users` (account-wide).
522
+
The bundle manages its own SQL warehouse and Lakebase project. Pass the two values with `--var` in step 4; other settings can also be overridden with `--var name=value`. Two optional settings control storage and audience: `prefix` (default `dqx_studio`) derives the schema names, and `studio_user_group` (default `dqx-studio-users`, which must already exist) is the audience. With `make app-deploy` these are the `STUDIO_PREFIX=<prefix>` and `STUDIO_USER_GROUP=<group>` arguments; with the Databricks CLI pass `--var prefix=<prefix>` and `--var studio_user_group=<group>`. Set `STUDIO_USER_GROUP=users` for **broad mode**, which shares Studio with the workspace `users` group and grants Unity Catalog access to `account users` (account-wide). When you set broad mode with `--var studio_user_group=users` (or through `BUNDLE_VARS` / `-BundleVars`) instead of `STUDIO_USER_GROUP`, also pass `--var studio_uc_principal="account users"`. Catalog and prefix cannot be changed once Studio's storage exists, so choose them before the first deploy.
523
523
524
524
See the [DQX Studio deployment guide](https://github.com/databrickslabs/dqx/blob/main/app/DEPLOYMENT.md#step-4-configure-databricksyml) for the full reference of each variable, including security implications of `admin_group` and when to override per target.
525
525
@@ -538,14 +538,16 @@ If you can't use the Marketplace, or want to customize the deployment, use the [
538
538
GRANT USE CATALOG ON CATALOG <catalog> TO `<audience>`;
539
539
```
540
540
541
-
The setup wizard verifies these on every start and prints any that are still missing. With a custom `admin_group` account group, administrators receive the same catalog access as the audience; with the built-in `admins` group they must also belong to the audience group to use OBO features.
541
+
The setup wizard verifies these and prints the exact `GRANT` statement for any that are still missing. With a custom `admin_group` account group, also grant it `USE CATALOG`: administrators then receive the same access as the audience. With the built-in `admins` group (the default), Unity Catalog cannot grant to it, so administrators must also belong to the audience group to use OBO features.
7. Open the deployed app from the **Apps** page in your Databricks workspace.
548
+
7. Open the deployed app from the **Apps** page as a workspace administrator (a member of `admins` or of `admin_group`). The bundle supplies the catalog, prefix, and audience, so the setup wizard shows them read-only. Run any `GRANT` statements it prints and choose **Verify again** until every step is green.
549
+
550
+
8. **Assign roles**: in Studio, open **Admin Settings → Entitlements** and map your audience groups to the Author, Approver, and Viewer roles. See [Permissions & Entitlements](/docs/studio/governance/permissions-and-entitlements) for the full permission matrix.
549
551
550
552
Once the catalog grants in step 5 are in place, one helper command **from the repository root** can deploy and start the prebuilt release without running `app-build`. On macOS or Linux:
551
553
@@ -559,6 +561,8 @@ On Windows, use the experimental PowerShell helper:
To set a non-default prefix or audience, add `STUDIO_PREFIX=<prefix> STUDIO_USER_GROUP=<group|users>` to the `make` command. The PowerShell helper has no equivalent arguments: add `'prefix=<prefix>'` and `'studio_user_group=<group>'` to `-BundleVars` (broad mode also needs `'studio_uc_principal=account users'`).
565
+
562
566
To build and deploy Studio from source instead, use `make app-deploy PROFILE=<profile> TARGET=<target>` on macOS or Linux. On Windows, run the experimental PowerShell helper from the repository root:
For the full walkthrough — including step-by-step commands, the `USE CATALOG` prerequisite, troubleshooting, and target-specific configuration — see the [DQX Studio deployment guide](https://github.com/databrickslabs/dqx/blob/main/app/DEPLOYMENT.md).
DQX Studio splits its data across two physical backends: high-volume append-mostly tables (`dq_validation_runs`, `dq_profiling_results`, `dq_quarantine_records`, `dq_metrics`) live in **Delta Lake** because they're written by Spark; transactional tables (rules catalog, app settings, RBAC, comments, schedule configs) live in **Lakebase Postgres** for fast row-level reads/writes from the FastAPI request handlers.
581
+
DQX Studio splits its data across two physical backends: high-volume append-mostly tables (`dq_validation_runs`, `dq_profiling_results`, `dq_quarantine_records`, `dq_metrics`) live in **Delta Lake** because they're written by Spark; transactional tables (rules catalog, app settings, RBAC, comments, schedule configs) live in **Lakebase Postgres** for fast row-level reads/writes from the FastAPI request handlers. Run configs too large to pass as job parameters are staged in the Delta `dq_run_configs` table in the main schema, so the task runner reads and writes only Delta and needs no Lakebase role or grants.
@@ -593,7 +597,7 @@ When a new version is published to the listing, update the app from your workspa
593
597
594
598
#### Using a Declarative Automation Bundle (DAB)
595
599
596
-
From the repository root, fetch the new Studio release tag, check it out, and redeploy with the same catalog and task-runner service principal:
600
+
From the repository root, fetch the new Studio release tag, check it out, and redeploy with the same catalog and task-runner service principal. If you set a non-default prefix or audience at install time, pass the same `STUDIO_PREFIX` / `STUDIO_USER_GROUP` again: the catalog and prefix are locked once storage exists.
Copy file name to clipboardExpand all lines: docs/dqx/docs/studio/governance/permissions-and-entitlements.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -84,6 +84,8 @@ Setup applies the access below, then **re-reads it to verify**. A missing grant,
84
84
85
85
The Genie `SELECT` allowlist is exactly the five approved views plus `dim_dq_rules` and `dim_dq_monitored_tables`. Studio never grants whole-schema Genie `SELECT`, quarantine tables, or `dq_user_table_entitlements`. On a bundle deployment the schemas are owned by the deploying identity, so the app service principal receives an explicit privilege list that includes `MANAGE` (not `ALL_PRIVILEGES`, which makes the bundle engine drop `MANAGE`) instead. The bundle grants the administrator group only its workspace permissions; the app re-applies and verifies the administrator group's Unity Catalog grants after each deploy restart.
86
86
87
+
The runner service principal needs no Lakebase role or grants. Run configs too large to pass as job parameters are staged in the Delta `dq_run_configs` table in the main schema, which its main-schema `SELECT` / `MODIFY` already covers.
88
+
87
89
### Audience, broad mode, and administrators
88
90
89
91
The **audience** is a dedicated, existing workspace group. Studio never creates groups, and the setup form rejects `users`, `account users`, and `admins`.
Copy file name to clipboardExpand all lines: docs/dqx/docs/studio/integrations.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -62,7 +62,7 @@ administrator to grant two things:
62
62
GRANTSELECTONdqx_studio.dq_rules_core TO "<principal>";
63
63
```
64
64
65
-
`<principal>` is the Postgres role for the reader's identity — for a service
65
+
`dqx_studio` here is Studio's Lakebase schema (`lakebase_schema_name`, default `dqx_studio`). It does not follow the Unity Catalog storage prefix. `<principal>` is the Postgres role for the reader's identity — for a service
66
66
principal, its **application id**. `SELECT` on the view is enough; readers don't need
67
67
access to the underlying `dq_resolved_rules` table.
Copy file name to clipboardExpand all lines: docs/dqx/docs/studio/monitoring/load-checks-in-a-pipeline.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -62,7 +62,7 @@ administrator to grant two things:
62
62
GRANTSELECTONdqx_studio.dq_rules_core TO "<principal>";
63
63
```
64
64
65
-
`<principal>` is the Postgres role for the reader's identity — for a service
65
+
`dqx_studio` here is Studio's Lakebase schema (`lakebase_schema_name`, default `dqx_studio`). It does not follow the Unity Catalog storage prefix. `<principal>` is the Postgres role for the reader's identity — for a service
66
66
principal, its **application id**. `SELECT` on the view is enough; readers don't need
67
67
access to the underlying `dq_resolved_rules` table.
0 commit comments