(feat) 4135-retry tenacity helper schema migrations by rudolfix · Pull Request #4266 · dlt-hub/dlt

rudolfix · 2026-07-21T21:19:40Z

Description

alternative fix to #4135 (see #4138 )

it retries a failing schema migrations (with jitter and backoffs) via tenacity retry helper.

from tenacity import retry, stop_after_attempt, retry_if_exception, wait_random_exponential
from dlt.pipeline.helpers import retry_schema_update

if __name__ == "__main__":
    pipeline = dlt.pipeline(pipeline_name="chess_pipeline", destination="duckdb", dataset_name="games_data")

    @retry(
        stop=stop_after_attempt(5),
        # random jitter spreads retries so colliding parallel runs do not wake up and retry in lockstep
        wait=wait_random_exponential(multiplier=1, max=30),
        retry=retry_if_exception(retry_schema_update()),
        reraise=True,
    )
    def load():
        data = chess_source(["magnuscarlsen", "rpragchess"], start_month="2022/11", end_month="2022/12")
        return pipeline.run(data)

    load_info = load()

does not create comments on _dlt tables on databricks to prevent non-atomic CREATE TABLE
as a bonus - allows to disable comments creation (via TBLPROPS) altogether.

…mns for concurrent loads

…-create-table-if-not-exists

…p comments

…ma-migrations

blacksmith-sh · 2026-07-21T21:25:22Z

Found 71 test failures on Blacksmith runners:

Failures

Test	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-False-users__orders__items-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-with_root_key-True-users__orders__items-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-False-users__orders__items-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `sql_upper-without_root_key-True-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-False-users__orders__items-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-with_root_key-True-users__orders__items-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-False-users__orders__items-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-products-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-products-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-products-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-users__orders-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-users__orders-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-users__orders-selected_load_id_idx2]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-users__orders__items-selected_load_id_idx0]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-users__orders__items-selected_load_id_idx1]`	View Logs
`test_relation_from_loads_query[tests.common.cases.normalizers/` `title_case-without_root_key-True-users__orders__items-selected_load_id_idx2]`	View Logs

cloudflare-workers-and-pages · 2026-07-21T21:25:51Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	docs	`261a93f`	Commit Preview URL Branch Preview URL	Jul 21 2026, 09:25 PM

francescomucio and others added 6 commits June 29, 2026 23:03

feat(databricks): create tables with IF NOT EXISTS and reconcile colu…

e01a662

…mns for concurrent loads

Merge remote-tracking branch 'origin/devel' into feat/4135-databricks…

9f48431

…-create-table-if-not-exists

adds tenacity retry helper for schema migrations fails

c5670e1

fixes sqlglot 1.30.x mssql dialect norm change in test

d9b6cae

skips comments in dlt tables on databricks, adds config option to ski…

f8a9d70

…p comments

Merge remote-tracking branch 'origin/devel' into feat/4135-retry-sche…

261a93f

…ma-migrations

rudolfix self-assigned this Jul 21, 2026

rudolfix added the ci full Use to trigger CI on a PR for full load tests label Jul 21, 2026

rudolfix changed the title ~~(feat) 4135-retry schema migrations~~ (feat) 4135-retry tenacity helper schema migrations Jul 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(feat) 4135-retry tenacity helper schema migrations#4266

(feat) 4135-retry tenacity helper schema migrations#4266
rudolfix wants to merge 6 commits into
develfrom
feat/4135-retry-schema-migrations

rudolfix commented Jul 21, 2026

Uh oh!

blacksmith-sh Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rudolfix commented Jul 21, 2026

Description

Uh oh!

blacksmith-sh Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 21, 2026

Deploying with Cloudflare Workers

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

blacksmith-sh Bot commented Jul 21, 2026 •

edited

Loading