Skip to content

Commit 5e2999c

Browse files
laurencewellsIsaac
andcommitted
docs(app): drop runner Lakebase prerequisite after #1564 and fill Studio install gaps
The runner stages oversized configs in the Delta dq_run_configs table since #1564, so it needs no Lakebase role or grants; remove the remaining guidance. Also document prefix/audience bundle variables (CLI, make, PowerShell), broad mode via --var, admin verification and role assignment for the DAB route, prefix reuse on upgrade, and use <prefix> in Studio SQL examples. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent f28125e commit 5e2999c

7 files changed

Lines changed: 23 additions & 19 deletions

File tree

‎app/DEPLOYMENT.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -367,7 +367,7 @@ The script requires `uv`, Node.js 18+, yarn classic v1, and Databricks CLI v1.4.
367367
2. `databricks bundle deploy` — provisions or updates the schemas, wheels volume, Lakebase project (+ endpoint + the app SP's Postgres role), the SQL warehouse, the task-runner job, and the Databricks App in dependency order, and applies **bundle-declared grants** via the `grants:` / `permissions:` blocks in `databricks.yml`. Stateful resources carry `lifecycle.prevent_destroy: true` so a future destroy can't drop them — see [Step 3](#step-3-stateful-storage-and-destroy-protection).
368368
3. `databricks bundle run` — starts the app.
369369

370-
Remember the manual prerequisites: the [catalog grants](#the-use-catalog-prerequisite) for the app SP (`USE CATALOG`, `CREATE SCHEMA`), task-runner SP, and audience, plus [runner Lakebase role and grants](#task-runner-lakebase-access) for the current oversized-config path. Cold-start UC checks must pass before Studio is ready; they do not verify runner Lakebase access.
370+
Remember the manual prerequisites: the [catalog grants](#the-use-catalog-prerequisite) for the app SP (`USE CATALOG`, `CREATE SCHEMA`), task-runner SP, and audience. Cold-start UC checks must pass before Studio is ready. The task runner needs no Lakebase role or grants.
371371

372372
> **First start**: The app runs Delta analytical and Lakebase application migrations on startup, and publishes the task-runner wheel to the UC volume. Wait for the setup checks to report that wheel publishing is ready before triggering runs. Also wait for `"Lakebase OLTP routing enabled"` before opening the UI. If Lakebase initialization fails, the app refuses to start and the Apps platform restarts the container. It never falls back to Delta-backed application state.
373373

@@ -510,7 +510,7 @@ rm -rf .databricks # clean local bundle
510510
databricks bundle deploy -p <your-profile> --force # or force deploy
511511
```
512512

513-
**First deploy fails with `cannot create resources.postgres_roles.task_runner_sp: Project with name 'projects/<id>' not found (404)`:**
513+
**First deploy fails with `cannot create resources.postgres_roles.app_sp: Project with name 'projects/<id>' not found (404)`:**
514514
This happens on a fresh workspace right after the Lakebase project is created (eventual consistency). Re-run `make app-deploy`; the second run succeeds.
515515

516516
**Profiler or dry-run not starting:**

‎app/README.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,7 @@ The setup workflow applies the grants and ACL updates it has authority for, then
8181

8282
The audience is a dedicated existing group, or `users` in DAB **broad mode** (workspace ACL `users` plus Unity Catalog `account users`, which is account-wide; the Marketplace form never accepts it). Members of `DQX_ADMIN_GROUP` or the workspace `admins` group may run setup; Unity Catalog cannot grant to `admins`, so UC grants for administrators go only to a custom admin account group, and with `admins` administrators must also be audience members to use OBO features. Per-user SQL entitlement, OAuth consent, and source-data privileges are checked for the active user, never for the group.
8383

84-
On every cold startup, setup resolves the task-runner job's actual `run_as`, grants it `USE SCHEMA` on the main and temporary schemas, `READ VOLUME` on the wheels volume, and schema-level `SELECT` / `MODIFY` on the main schema (never `ALL PRIVILEGES`, catalog, Genie, or demo access), and verifies these plus its `USE CATALOG`. Schema grants cover current and future tables; table-specific grants alone do not satisfy setup. Runner Lakebase roles and privileges are not checked by setup.
85-
86-
The current oversized-config staging implementation still needs a Lakebase administrator to create/map the runner's OAuth `SERVICE_PRINCIPAL` role and apply scoped grants manually. Changing that staging implementation is separate work; removing its setup checks does not remove the runtime requirement. Setup applies no automatic runner PostgreSQL grants. The Postgres username must match the resolved Jobs runner client ID; a legacy `DQX_TASK_RUNNER_POSTGRES_ROLE` override must also match. Never grant superuser membership to the runner. See [Deployment](DEPLOYMENT.md#task-runner-lakebase-access) for the UI and SQL steps.
84+
On every cold startup, setup resolves the task-runner job's actual `run_as`, grants it `USE SCHEMA` on the main and temporary schemas, schema-level `SELECT` on the temporary schema (how it reads OBO temp views), `READ VOLUME` on the wheels volume, and schema-level `SELECT` / `MODIFY` on the main schema (never `ALL PRIVILEGES`, catalog, Genie, or demo access), and verifies these plus its `USE CATALOG`. Schema grants cover current and future tables; table-specific grants alone do not satisfy setup. The runner needs no Lakebase access: run configs too large for job parameters are staged in the Delta `dq_run_configs` table in the main schema, which the main-schema `SELECT` / `MODIFY` grants already cover.
8785

8886
Genie audience access requires space `CAN_RUN`, Consumer access or Databricks SQL access entitlement, parent usages, and only the five approved view plus two dimension-table grants, never whole-schema `SELECT` or access to the entitlement table. Genie uses embedded compute credentials; Studio's OBO SQL workflows additionally require SQL access entitlement and warehouse `CAN_USE`. Both DAB and Marketplace use the `genie` OAuth scope. Warehouse rebind re-runs the warehouse checks and reconciles Genie compute. See [the deployment grants reference](DEPLOYMENT.md#grants-reference).
8987

‎docs/dqx/docs/installation.mdx‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -519,7 +519,7 @@ If you can't use the Marketplace, or want to customize the deployment, use the [
519519
- `catalog_name` — Unity Catalog catalog where the studio's schemas and volume live (must already exist).
520520
- `dqx_service_principal_application_id` — the task-runner SP you created in step 2.
521521

522-
The bundle manages its own SQL warehouse and Lakebase project. Pass the two values with `--var` in step 4; other settings can also be overridden with `--var name=value`. Two optional settings control storage and audience: `prefix` (default `dqx_studio`) derives the schema names, and `studio_user_group` (default `dqx-studio-users`, which must already exist) is the audience. With `make app-deploy` these are the `STUDIO_PREFIX=<prefix>` and `STUDIO_USER_GROUP=<group>` arguments. Set `STUDIO_USER_GROUP=users` for **broad mode**, which shares Studio with the workspace `users` group and grants Unity Catalog access to `account users` (account-wide).
522+
The bundle manages its own SQL warehouse and Lakebase project. Pass the two values with `--var` in step 4; other settings can also be overridden with `--var name=value`. Two optional settings control storage and audience: `prefix` (default `dqx_studio`) derives the schema names, and `studio_user_group` (default `dqx-studio-users`, which must already exist) is the audience. With `make app-deploy` these are the `STUDIO_PREFIX=<prefix>` and `STUDIO_USER_GROUP=<group>` arguments; with the Databricks CLI pass `--var prefix=<prefix>` and `--var studio_user_group=<group>`. Set `STUDIO_USER_GROUP=users` for **broad mode**, which shares Studio with the workspace `users` group and grants Unity Catalog access to `account users` (account-wide). When you set broad mode with `--var studio_user_group=users` (or through `BUNDLE_VARS` / `-BundleVars`) instead of `STUDIO_USER_GROUP`, also pass `--var studio_uc_principal="account users"`. Catalog and prefix cannot be changed once Studio's storage exists, so choose them before the first deploy.
523523

524524
See the [DQX Studio deployment guide](https://github.com/databrickslabs/dqx/blob/main/app/DEPLOYMENT.md#step-4-configure-databricksyml) for the full reference of each variable, including security implications of `admin_group` and when to override per target.
525525

@@ -538,14 +538,16 @@ If you can't use the Marketplace, or want to customize the deployment, use the [
538538
GRANT USE CATALOG ON CATALOG <catalog> TO `<audience>`;
539539
```
540540
541-
The setup wizard verifies these on every start and prints any that are still missing. With a custom `admin_group` account group, administrators receive the same catalog access as the audience; with the built-in `admins` group they must also belong to the audience group to use OBO features.
541+
The setup wizard verifies these and prints the exact `GRANT` statement for any that are still missing. With a custom `admin_group` account group, also grant it `USE CATALOG`: administrators then receive the same access as the audience. With the built-in `admins` group (the default), Unity Catalog cannot grant to it, so administrators must also belong to the audience group to use OBO features.
542542

543543
6. Start Studio:
544544
```commandline
545545
databricks bundle run dqx-studio -p <your-profile> -t release --var catalog_name=<your-catalog> --var dqx_service_principal_application_id=<your-sp-application-id>
546546
```
547547

548-
7. Open the deployed app from the **Apps** page in your Databricks workspace.
548+
7. Open the deployed app from the **Apps** page as a workspace administrator (a member of `admins` or of `admin_group`). The bundle supplies the catalog, prefix, and audience, so the setup wizard shows them read-only. Run any `GRANT` statements it prints and choose **Verify again** until every step is green.
549+
550+
8. **Assign roles**: in Studio, open **Admin Settings → Entitlements** and map your audience groups to the Author, Approver, and Viewer roles. See [Permissions & Entitlements](/docs/studio/governance/permissions-and-entitlements) for the full permission matrix.
549551

550552
Once the catalog grants in step 5 are in place, one helper command **from the repository root** can deploy and start the prebuilt release without running `app-build`. On macOS or Linux:
551553

@@ -559,6 +561,8 @@ On Windows, use the experimental PowerShell helper:
559561
.\make.ps1 app-deploy -Profile <your-profile> -Release -BundleVars @('catalog_name=<your-catalog>', 'dqx_service_principal_application_id=<your-sp-application-id>')
560562
```
561563

564+
To set a non-default prefix or audience, add `STUDIO_PREFIX=<prefix> STUDIO_USER_GROUP=<group|users>` to the `make` command. The PowerShell helper has no equivalent arguments: add `'prefix=<prefix>'` and `'studio_user_group=<group>'` to `-BundleVars` (broad mode also needs `'studio_uc_principal=account users'`).
565+
562566
To build and deploy Studio from source instead, use `make app-deploy PROFILE=<profile> TARGET=<target>` on macOS or Linux. On Windows, run the experimental PowerShell helper from the repository root:
563567

564568
```powershell
@@ -574,7 +578,7 @@ The studio's schemas (`<prefix>`, `<prefix>_tmp`, `<prefix>_genie`, `<prefix>_de
574578
For the full walkthrough — including step-by-step commands, the `USE CATALOG` prerequisite, troubleshooting, and target-specific configuration — see the [DQX Studio deployment guide](https://github.com/databrickslabs/dqx/blob/main/app/DEPLOYMENT.md).
575579

576580
<Admonition type="note" title="Hybrid storage backend">
577-
DQX Studio splits its data across two physical backends: high-volume append-mostly tables (`dq_validation_runs`, `dq_profiling_results`, `dq_quarantine_records`, `dq_metrics`) live in **Delta Lake** because they're written by Spark; transactional tables (rules catalog, app settings, RBAC, comments, schedule configs) live in **Lakebase Postgres** for fast row-level reads/writes from the FastAPI request handlers.
581+
DQX Studio splits its data across two physical backends: high-volume append-mostly tables (`dq_validation_runs`, `dq_profiling_results`, `dq_quarantine_records`, `dq_metrics`) live in **Delta Lake** because they're written by Spark; transactional tables (rules catalog, app settings, RBAC, comments, schedule configs) live in **Lakebase Postgres** for fast row-level reads/writes from the FastAPI request handlers. Run configs too large to pass as job parameters are staged in the Delta `dq_run_configs` table in the main schema, so the task runner reads and writes only Delta and needs no Lakebase role or grants.
578582
</Admonition>
579583

580584
<Admonition type="note" title="First-run wheel upload">
@@ -593,7 +597,7 @@ When a new version is published to the listing, update the app from your workspa
593597

594598
#### Using a Declarative Automation Bundle (DAB)
595599

596-
From the repository root, fetch the new Studio release tag, check it out, and redeploy with the same catalog and task-runner service principal:
600+
From the repository root, fetch the new Studio release tag, check it out, and redeploy with the same catalog and task-runner service principal. If you set a non-default prefix or audience at install time, pass the same `STUDIO_PREFIX` / `STUDIO_USER_GROUP` again: the catalog and prefix are locked once storage exists.
597601

598602
```commandline
599603
git fetch --tags origin

‎docs/dqx/docs/reference/query_results_cookbook.mdx‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -912,21 +912,21 @@ Then drill into a single check to get the failing rows. Replace `<your_check_nam
912912
## DQX Studio quarantine records {#dqx-studio-quarantine-records}
913913

914914
DQX Studio stores its quarantined rows in its own `dq_quarantine_records` table in the
915-
Studio catalog (`<catalog>.dqx_studio.dq_quarantine_records`) — a different shape from
915+
Studio catalog (`<catalog>.<prefix>.dq_quarantine_records`) — a different shape from
916916
the tables above. `row_data` holds the full source row, and `errors` / `warnings` the
917917
checks each row failed, all as
918918
[`VARIANT`](https://docs.databricks.com/aws/en/semi-structured/variant). For an overview
919919
and the basic "recent rows" query, see
920920
[Analyse in Lakehouse](/docs/studio/running/#analyse-in-lakehouse); the recipes below go
921-
further. Replace `<catalog>` with your Studio catalog.
921+
further. Replace `<catalog>` with your Studio catalog and `<prefix>` with its storage prefix (default `dqx_studio`).
922922

923923
**Read individual fields out of the source row:**
924924

925925
```sql
926926
SELECT quarantine_id,
927927
row_data:order_id::string AS order_id,
928928
row_data:amount::double AS amount
929-
FROM <catalog>.dqx_studio.dq_quarantine_records
929+
FROM <catalog>.<prefix>.dq_quarantine_records
930930
WHERE source_table_fqn = 'main.sales.orders';
931931
```
932932

@@ -939,7 +939,7 @@ SELECT q.quarantine_id,
939939
e.value:name::string AS check_name,
940940
e.value:message::string AS message,
941941
e.value:columns AS columns
942-
FROM <catalog>.dqx_studio.dq_quarantine_records AS q,
942+
FROM <catalog>.<prefix>.dq_quarantine_records AS q,
943943
LATERAL variant_explode(q.errors) AS e
944944
WHERE q.run_id = '<run_id>';
945945
```
@@ -948,7 +948,7 @@ WHERE q.run_id = '<run_id>';
948948

949949
```sql
950950
SELECT e.value:name::string AS check_name, count(*) AS failing_rows
951-
FROM <catalog>.dqx_studio.dq_quarantine_records AS q,
951+
FROM <catalog>.<prefix>.dq_quarantine_records AS q,
952952
LATERAL variant_explode(q.errors) AS e
953953
WHERE q.source_table_fqn = 'main.sales.orders'
954954
GROUP BY e.value:name::string
@@ -960,8 +960,8 @@ what triggered it, and whether it succeeded:
960960

961961
```sql
962962
SELECT q.quarantine_id, q.source_table_fqn, q.created_at, r.*
963-
FROM <catalog>.dqx_studio.dq_quarantine_records AS q
964-
JOIN <catalog>.dqx_studio.dq_validation_runs AS r
963+
FROM <catalog>.<prefix>.dq_quarantine_records AS q
964+
JOIN <catalog>.<prefix>.dq_validation_runs AS r
965965
ON q.run_id = r.run_id
966966
WHERE q.source_table_fqn = 'main.sales.orders'
967967
ORDER BY q.created_at DESC;

‎docs/dqx/docs/studio/governance/permissions-and-entitlements.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,8 @@ Setup applies the access below, then **re-reads it to verify**. A missing grant,
8484

8585
The Genie `SELECT` allowlist is exactly the five approved views plus `dim_dq_rules` and `dim_dq_monitored_tables`. Studio never grants whole-schema Genie `SELECT`, quarantine tables, or `dq_user_table_entitlements`. On a bundle deployment the schemas are owned by the deploying identity, so the app service principal receives an explicit privilege list that includes `MANAGE` (not `ALL_PRIVILEGES`, which makes the bundle engine drop `MANAGE`) instead. The bundle grants the administrator group only its workspace permissions; the app re-applies and verifies the administrator group's Unity Catalog grants after each deploy restart.
8686

87+
The runner service principal needs no Lakebase role or grants. Run configs too large to pass as job parameters are staged in the Delta `dq_run_configs` table in the main schema, which its main-schema `SELECT` / `MODIFY` already covers.
88+
8789
### Audience, broad mode, and administrators
8890

8991
The **audience** is a dedicated, existing workspace group. Studio never creates groups, and the setup form rejects `users`, `account users`, and `admins`.

‎docs/dqx/docs/studio/integrations.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ administrator to grant two things:
6262
GRANT SELECT ON dqx_studio.dq_rules_core TO "<principal>";
6363
```
6464

65-
`<principal>` is the Postgres role for the reader's identity — for a service
65+
`dqx_studio` here is Studio's Lakebase schema (`lakebase_schema_name`, default `dqx_studio`). It does not follow the Unity Catalog storage prefix. `<principal>` is the Postgres role for the reader's identity — for a service
6666
principal, its **application id**. `SELECT` on the view is enough; readers don't need
6767
access to the underlying `dq_resolved_rules` table.
6868

‎docs/dqx/docs/studio/monitoring/load-checks-in-a-pipeline.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ administrator to grant two things:
6262
GRANT SELECT ON dqx_studio.dq_rules_core TO "<principal>";
6363
```
6464

65-
`<principal>` is the Postgres role for the reader's identity — for a service
65+
`dqx_studio` here is Studio's Lakebase schema (`lakebase_schema_name`, default `dqx_studio`). It does not follow the Unity Catalog storage prefix. `<principal>` is the Postgres role for the reader's identity — for a service
6666
principal, its **application id**. `SELECT` on the view is enough; readers don't need
6767
access to the underlying `dq_resolved_rules` table.
6868

0 commit comments

Comments
 (0)