Repository navigation
Use a Lakebase table to store oversized run configs - #1547
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1547 +/- ##
=======================================
Coverage 92.94% 92.94%
=======================================
Files 142 142
Lines 14002 14002
Branches 151 151
=======================================
Hits 13014 13014
Misses 919 919
Partials 69 69
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
✅ 28/28 passed, 1 flaky, 4h8m21s total Flaky tests:
Running from acceptance #6048 |
Contributor
|
✅ 1/1 passed, 26m51s total Running from mcp #797 |
Contributor
|
✅ 195/195 passed, 1 skipped, 6h3m55s total Running from anomaly #2162 |
mwojtyczka
changed the title
mwojtyczka
reviewed
Sep 24, 2026
mwojtyczka
reviewed
Sep 24, 2026
mwojtyczka
reviewed
Sep 24, 2026
mwojtyczka
reviewed
Sep 24, 2026
mwojtyczka
reviewed
Sep 28, 2026
mwojtyczka
reviewed
Sep 28, 2026
…d; thread Lakebase coords in demo seeder Oversized run configs are staged to the dq_run_configs Lakebase table and read back by the task runner over Postgres. Two gaps remained on the Lakebase-disabled (Delta OLTP fallback) and demo-seeding paths: - prepare_config_json staged unconditionally, so in the documented lakebase_endpoint="-" (Delta OLTP) mode an oversized config was written to a table the runner could never read over Postgres, surfacing as a confusing connection error. Fail fast with an actionable RunConfigStagingUnavailableError when the OLTP dialect is not "postgres". - The demo-seeding CLI constructed JobService without any lakebase_* coordinates, so a staged config could not be read back. Thread the resolved coordinates from the live OLTP executor, mirroring get_job_service. Covered by a new unit test for the fail-fast path; existing staging tests now pin the mock OLTP executor to the postgres dialect. Co-authored-by: Isaac <no-reply@databricks.com>
…s dialect The oversized-config fail-fast guard added in e3fa811 only stages when the OLTP dialect is "postgres". test_submit_deletes_staged_row_when_submission_fails used an autospec mock whose dialect was not "postgres", so staging was skipped and the delete-on-failure path was never exercised. Pin the mock's dialect to "postgres" so the test reflects the Lakebase-enabled staging scenario it is asserting. Co-authored-by: Isaac <no-reply@databricks.com>
Addresses four review findings on the oversized run-config path: - startup: a malformed DQX_TASK_RUNNER_POSTGRES_ROLE no longer aborts the app. validate_object_id now logs and skips the optional grant instead of letting ValueError escape the lifespan (a malformed role still never reaches DDL). - monitored_tables run route: RunConfigStagingError (Lakebase unreachable / dq_run_configs missing) now maps to HTTP 503 instead of 400; caller/config errors (too large, Lakebase disabled) stay 400. Subclass catch precedes base. - startup grant loop: the failure warning now names the specific failing statement so a later opaque runner permission error is traceable. - get_job_service: resolve lakebase schema/database from the live OLTP executor (with resource fallback), consistent with the other coordinates, so the app stages the row to exactly the schema the runner is told to read from. Tests updated/added: malformed-role now skips (not raises); grant failure names the statement and continues; run route maps 503 vs 400; get_job_service threads schema/database from the executor. Co-authored-by: Isaac <no-reply@databricks.com>
mwojtyczka
reviewed
Sep 29, 2026
mwojtyczka
reviewed
Sep 29, 2026
mwojtyczka
reviewed
Sep 29, 2026
mwojtyczka
reviewed
Sep 29, 2026
The root `make fmt` job uses black as the Python formatter; my earlier change was formatted with the app venv's newer ruff, which wraps multi-line asserts differently and churned two pre-existing asserts into a style black reverts. Reformat with black so the file is a fixed point of the CI fmt check again. Co-authored-by: Isaac <no-reply@databricks.com>
2 of 8 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR modifies the handling of oversized run configs to use a Lakebase table instead of the wheel file volume.
Linked issues
Resolves #1558
Tests
Documentation and Demos