Improve 0101_3_2_0_ui_improvements_for_deadlines upgrade migration#63920
Merged
vatsrahul1001 merged 14 commits intoMar 23, 2026
Merged
Conversation
Switch batch pagination from unindexed dag_id to the primary key id (UUID7, time-ordered) in both upgrade and downgrade loops. This eliminates full table scans + sorts on every batch iteration.
Create a temporary index on deadline(dagrun_id) before the migration loop and drop it after. Without this, the per-alert UPDATE that joins dag_run to find matching deadline rows requires a full table scan of the deadline table on every iteration.
Use dialect-specific JSON path checks (PostgreSQL jsonb operators, MySQL JSON_EXTRACT, SQLite json_extract) to filter out serialized_dag rows that have no deadline data. This avoids transferring and processing large data blobs for the majority of DAGs that have no deadline configuration. Compressed rows are still included since the DB cannot inspect their content.
For DAGs with deadline data, the migration previously decompressed the serialized_dag blob 3 times: once to extract deadlines, once in update_dag_deadline_field() to modify the data, and once to recompute the dag_hash. Now the parsed dag_data is kept in memory, modified once, the hash is computed from it, and the result is written back in a single UPDATE combining data + dag_hash.
The correlated subquery joining dag_run and serialized_dag to find matching deadline rows was executed for every individual alert within a DAG. Now the dagrun IDs are fetched once per DAG and reused for each alert's UPDATE via an expanding bind parameter.
Extract the create-index/drop-index pattern into a reusable context manager to reduce indentation churn and make the intent clearer at the call site.
Extract the migration loop body into _migrate_deadline_alerts() decorated with @temporary_index, so the while loop stays at its original indentation level. The outer function keeps only the offline-mode guard.
boring-cyborg
Bot
added
area:db-migrations
jason810496
commented
Mar 19, 2026
Member
Author
There was a problem hiding this comment.
With my own benchmark setup (not Rahul's setup) on Mac M1, 16GB RAM:
Same script as mentioned in #63549 (comment)
deadline: 1,000,000 rows (2xx MB)
dag_run: 1,000,000 rows (3xx MB)
From 08:29:32 to 08:35:53 is 6 minutes and 21 seconds total runtime for the “Add required fields to enable UI integrations for the Deadline Alerts feature” data migration (the 0101_3_2_0_ui_improvements_for_deadlines.py step).
2026-03-19T08:29:23.822904Z [info ] Running upgrade e812941398f4 -> 665854ef0536, Update ORM for asset partitioning. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:23.876542Z [info ] Running upgrade 665854ef0536 -> b12d4f98a91e, Drop ``id`` column from ``team`` table and make ``name`` the primary key. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:23.895847Z [info ] Running upgrade b12d4f98a91e -> edc4f85a4619, Enforce the new ``NOT NULL`` expectations for ``log.event`` and ``dag.is_stale``. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:23.901784Z [info ] Running upgrade edc4f85a4619 -> c47f2e1ab9d4, Add ``queue`` column to ``trigger`` table. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:23.902937Z [info ] Running upgrade c47f2e1ab9d4 -> 0b112f49112d, Add exceeds max runs flag to dag model. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:23.904587Z [info ] Running upgrade 0b112f49112d -> e79fc784f145, add timetable_type to dag table for filtering. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:23.914095Z [info ] Running upgrade e79fc784f145 -> 55297ae24532, Add required fields to enable UI integrations for the Deadline Alerts feature. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:29:32.490978Z [info ] Starting migration [0101_3_2_0_ui_improvements_for_deadlines_py] batch_size=10000 loc=0101_3_2_0_ui_improvements_for_deadlines.py:456 total_batches=1 total_dags=1000
2026-03-19T08:29:32.506545Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:497 total_batches=1
2026-03-19T08:35:53.214025Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:687 total_batches=1
2026-03-19T08:35:53.220163Z [info ] Migration complete [0101_3_2_0_ui_improvements_for_deadlines_py] dags_with_deadlines=1000 loc=0101_3_2_0_ui_improvements_for_deadlines.py:689 migrated_alerts=2000 processed_records=1000 unique_dags=1000
2026-03-19T08:35:53.220293Z [info ] No Dags encountered errors [0101_3_2_0_ui_improvements_for_deadlines_py] loc=0101_3_2_0_ui_improvements_for_deadlines.py:341 operation=migration
2026-03-19T08:35:53.240980Z [info ] Running upgrade 55297ae24532 -> a5a3e5eb9b8d, Make external_executor_id TEXT to allow for longer external_executor_ids. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:53.244680Z [info ] Running upgrade a5a3e5eb9b8d -> 53ff648b8a26, Add revoked_token table. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:53.283274Z [info ] Running upgrade 53ff648b8a26 -> f8c9d7e6b5a4, Standardize UUID column format for non-PostgreSQL databases. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:53.286984Z [info ] Running upgrade f8c9d7e6b5a4 -> e42d9fcd10d9, Add allowed_run_types to dag. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:53.288426Z [info ] Running upgrade e42d9fcd10d9 -> 134de42d3cb0, Add partition_key to backfill_dag_run. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:53.289910Z [info ] Running upgrade 134de42d3cb0 -> 6222ce48e289, Add partition fields to DagModel. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:53.292082Z [info ] Running upgrade 6222ce48e289 -> 888b59e02a5b, Fix migration file ORM inconsistencies. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:54.015686Z [info ] Running upgrade 888b59e02a5b -> 1d6611b6ab7c, Add bundle_name to callback table. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:35:54.223026Z [info ] Context impl PostgresqlImpl. [alembic.runtime.migration] loc=migration.py:210
2026-03-19T08:35:54.223219Z [info ] Will assume transactional DDL. [alembic.runtime.migration] loc=migration.py:213
2026-03-19T08:35:54.298838Z [warning ] The `airflow.serialization.serde.deserialize` attribute is deprecated. Please use `'airflow.sdk.serde.deserialize'`. [py.warnings] category=DeprecatedImportWarning filename=/usr/python/lib/python3.10/site-packages/airflow/migrations/versions/0094_3_2_0_replace_deadline_inline_callback_with_fkey.py lineno=37
2026-03-19T08:35:55.371159Z [info ] Database migration done! [airflow.cli.commands.db_command] loc=db_command.py:152
start: 20260319_081832
end: 20260319_083555
jason810496
commented
Mar 19, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
GCP VM, 16Core, 32GB RAM with Rahul's latest 3.1.8 setup:
That’s a duration of about 4 minutes and 52 seconds for the main migration script.
- The migration script for UI improvements starts at:
2026-03-19T08:41:44.981900Z Starting migration [0101_3_2_0_ui_improvements_for_deadlines_py] - It finishes at:
2026-03-19T08:46:36.564781Z Migration complete [0101_3_2_0_ui_improvements_for_deadlines_py]
That’s roughly 9 minutes and 15 seconds from the beginning of the upgrade that adds the required fields through to completion of the data migration for deadlines.
- Start of the relevant upgrade:
2026-03-19T08:37:21.736454Z Running upgrade e79fc784f145 -> 55297ae24532 ... End of the migration script: 2026-03-19T08:46:36.564781Z Migration complete ...
2026-03-19T08:37:21.736454Z [info ] Running upgrade e79fc784f145 -> 55297ae24532, Add required fields to enable UI integrations for the Deadline Alerts feature. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T08:41:44.981900Z [info ] Starting migration [0101_3_2_0_ui_improvements_for_deadlines_py] batch_size=10000 loc=0101_3_2_0_ui_improvements_for_deadlines.py:456 total_batches=11 total_dags=100021
2026-03-19T08:41:45.767948Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:497 total_batches=11
2026-03-19T08:46:36.494538Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:687 total_batches=11
2026-03-19T08:46:36.523288Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=2 loc=0101_3_2_0_ui_improvements_for_deadlines.py:497 total_batches=11
2026-03-19T08:46:36.563447Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=2 loc=0101_3_2_0_ui_improvements_for_deadlines.py:687 total_batches=11
2026-03-19T08:46:36.564781Z [info ] Migration complete [0101_3_2_0_ui_improvements_for_deadlines_py] dags_with_deadlines=10021 loc=0101_3_2_0_ui_improvements_for_deadlines.py:689 migrated_alerts=10063 processed_records=10021 unique_dags=10021
2026-03-19T08:46:36.564925Z [info ] No Dags encountered errors [0101_3_2_0_ui_improvements_for_deadlines_py] loc=0101_3_2_0_ui_improvements_for_deadlines.py:341 operation=migration
2026-03-19T08:46:36.621169Z [info ] Running upgrade 55297ae24532 -> a5a3e5eb9b8d, Make external_executor_id TEXT to allow for longer external_executor_ids. [alembic.runtime.migration] loc=migration.py:621
jason810496
commented
Mar 19, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
I will resolve the MySQL CI failure tomorrow, as there isn't error for Postgres.
jason810496
commented
Mar 19, 2026
Member
Author
There was a problem hiding this comment.
GCP VM, 16Core, 32GB RAM with Rahul's latest 3.1.8 setup (after resolve the conflict):
- main section (duration between
Starting migration ... Migration complete):316.047441 seconds(5 minutes and 16 seconds) - the whole migration:
668.28081 seconds(11 mins 8 seconds)
2026-03-19T12:59:02.379647Z [info ] Running upgrade e79fc784f145 -> 55297ae24532, Add required fields to enable UI integrations for the Deadline Alerts feature. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T13:04:53.552509Z [info ] Starting migration [0101_3_2_0_ui_improvements_for_deadlines_py] batch_size=10000 loc=0101_3_2_0_ui_improvements_for_deadlines.py:479 total_batches=11 total_dags=100021
2026-03-19T13:04:54.723347Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:519 total_batches=11
2026-03-19T13:10:09.497032Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:710 total_batches=11
2026-03-19T13:10:09.529809Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=2 loc=0101_3_2_0_ui_improvements_for_deadlines.py:519 total_batches=11
2026-03-19T13:10:09.597933Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=2 loc=0101_3_2_0_ui_improvements_for_deadlines.py:710 total_batches=11
2026-03-19T13:10:09.599950Z [info ] Migration complete [0101_3_2_0_ui_improvements_for_deadlines_py] dags_with_deadlines=10021 loc=0101_3_2_0_ui_improvements_for_deadlines.py:712 migrated_alerts=10063 processed_records=10021 unique_dags=10021
2026-03-19T13:10:09.600168Z [info ] No Dags encountered errors [0101_3_2_0_ui_improvements_for_deadlines_py] loc=0101_3_2_0_ui_improvements_for_deadlines.py:342 operation=migration
2026-03-19T13:10:10.660457Z [info ] Running upgrade 55297ae24532 -> a5a3e5eb9b8d, Make external_executor_id TEXT to allow for longer external_executor_ids. [alembic.runtime.migration] loc=migration.py:621
Baseline (main branch without the current patch but included the _begin_nested_transaction changes ) with same setup:
- main section (duration between Starting migration ... Migration complete ): 944.071939 seconds ( 15 minutes and 44 seconds )
- the whole migration: 1267.377314 seconds ( 21 minutes and 7 seconds )
2026-03-19T14:01:28.588076Z [info ] Running upgrade e79fc784f145 -> 55297ae24532, Add required fields to enable UI integrations for the Deadline Alerts feature. [alembic.runtime.migration] loc=migration.py:621
2026-03-19T14:06:51.893451Z [info ] Starting migration [0101_3_2_0_ui_improvements_for_deadlines_py] batch_size=10000 loc=0101_3_2_0_ui_improvements_for_deadlines.py:416 total_batches=11 total_dags=100021
2026-03-19T14:06:52.700214Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:09:28.571916Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=1 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:09:29.388547Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=2 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:10:57.680594Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=2 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:10:58.290998Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=3 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:12:25.860898Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=3 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:12:26.711914Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=4 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:13:53.204557Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=4 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:13:53.791529Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=5 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:15:20.310028Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=5 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:15:20.883404Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=6 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:16:46.709456Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=6 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:16:47.276373Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=7 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:18:13.883007Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=7 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:18:14.691596Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=8 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:19:41.261451Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=8 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:19:41.807916Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=9 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:21:08.444869Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=9 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:21:09.227295Z [info ] Processing batch [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=10 loc=0101_3_2_0_ui_improvements_for_deadlines.py:454 total_batches=11
2026-03-19T14:22:35.886717Z [info ] Batch complete [0101_3_2_0_ui_improvements_for_deadlines_py] batch_num=10 loc=0101_3_2_0_ui_improvements_for_deadlines.py:577 total_batches=11
2026-03-19T14:22:35.965390Z [info ] Processing batch
Contributor
|
I have verified this, and there is a significant improvement in migration timing. For me, migration now getting completed in 5 mins |
1 task
ephraimbuddy
approved these changes
Mar 23, 2026
ephraimbuddy
left a comment
ephraimbuddy
left a comment
Contributor
There was a problem hiding this comment.
LGTM, only nits and please remember to remove the migration test before merge
Move dagrun_ids outside of inner loop
uranusjr
approved these changes
Mar 23, 2026
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Mar 30, 2026
…pache#63920) * Use PK index for keyset pagination in 0101 migration Switch batch pagination from unindexed dag_id to the primary key id (UUID7, time-ordered) in both upgrade and downgrade loops. This eliminates full table scans + sorts on every batch iteration. * Add temporary index on deadline.dagrun_id during 0101 migration Create a temporary index on deadline(dagrun_id) before the migration loop and drop it after. Without this, the per-alert UPDATE that joins dag_run to find matching deadline rows requires a full table scan of the deadline table on every iteration. * Add SQL-side deadline filtering to skip DAGs without deadlines in 0101 Use dialect-specific JSON path checks (PostgreSQL jsonb operators, MySQL JSON_EXTRACT, SQLite json_extract) to filter out serialized_dag rows that have no deadline data. This avoids transferring and processing large data blobs for the majority of DAGs that have no deadline configuration. Compressed rows are still included since the DB cannot inspect their content. * Eliminate triple decompression in 0101 migration upgrade path For DAGs with deadline data, the migration previously decompressed the serialized_dag blob 3 times: once to extract deadlines, once in update_dag_deadline_field() to modify the data, and once to recompute the dag_hash. Now the parsed dag_data is kept in memory, modified once, the hash is computed from it, and the result is written back in a single UPDATE combining data + dag_hash. * Cache dagrun IDs per DAG to avoid repeated JOINs in 0101 migration The correlated subquery joining dag_run and serialized_dag to find matching deadline rows was executed for every individual alert within a DAG. Now the dagrun IDs are fetched once per DAG and reused for each alert's UPDATE via an expanding bind parameter. * Use temporary_index context manager instead of inline try/finally Extract the create-index/drop-index pattern into a reusable context manager to reduce indentation churn and make the intent clearer at the call site. * Convert temporary_index to decorator to avoid indentation churn Extract the migration loop body into _migrate_deadline_alerts() decorated with @temporary_index, so the while loop stays at its original indentation level. The outer function keeps only the offline-mode guard. * Add comments explaining why keyset pagination uses id instead of dag_id * Move dagrun_ids select to where it being referenced * Add additional validation for existing temp index check * Remove unit test for migration * Speed up deadline alert UI migration updates Move dagrun_ids outside of inner loop
Suraj-kumar00
pushed a commit
to Suraj-kumar00/airflow
that referenced
this pull request
Apr 7, 2026
…pache#63920) * Use PK index for keyset pagination in 0101 migration Switch batch pagination from unindexed dag_id to the primary key id (UUID7, time-ordered) in both upgrade and downgrade loops. This eliminates full table scans + sorts on every batch iteration. * Add temporary index on deadline.dagrun_id during 0101 migration Create a temporary index on deadline(dagrun_id) before the migration loop and drop it after. Without this, the per-alert UPDATE that joins dag_run to find matching deadline rows requires a full table scan of the deadline table on every iteration. * Add SQL-side deadline filtering to skip DAGs without deadlines in 0101 Use dialect-specific JSON path checks (PostgreSQL jsonb operators, MySQL JSON_EXTRACT, SQLite json_extract) to filter out serialized_dag rows that have no deadline data. This avoids transferring and processing large data blobs for the majority of DAGs that have no deadline configuration. Compressed rows are still included since the DB cannot inspect their content. * Eliminate triple decompression in 0101 migration upgrade path For DAGs with deadline data, the migration previously decompressed the serialized_dag blob 3 times: once to extract deadlines, once in update_dag_deadline_field() to modify the data, and once to recompute the dag_hash. Now the parsed dag_data is kept in memory, modified once, the hash is computed from it, and the result is written back in a single UPDATE combining data + dag_hash. * Cache dagrun IDs per DAG to avoid repeated JOINs in 0101 migration The correlated subquery joining dag_run and serialized_dag to find matching deadline rows was executed for every individual alert within a DAG. Now the dagrun IDs are fetched once per DAG and reused for each alert's UPDATE via an expanding bind parameter. * Use temporary_index context manager instead of inline try/finally Extract the create-index/drop-index pattern into a reusable context manager to reduce indentation churn and make the intent clearer at the call site. * Convert temporary_index to decorator to avoid indentation churn Extract the migration loop body into _migrate_deadline_alerts() decorated with @temporary_index, so the while loop stays at its original indentation level. The outer function keeps only the offline-mode guard. * Add comments explaining why keyset pagination uses id instead of dag_id * Move dagrun_ids select to where it being referenced * Add additional validation for existing temp index check * Remove unit test for migration * Speed up deadline alert UI migration updates Move dagrun_ids outside of inner loop
abhijeets25012-tech
pushed a commit
to abhijeets25012-tech/airflow
that referenced
this pull request
Apr 9, 2026
…pache#63920) * Use PK index for keyset pagination in 0101 migration Switch batch pagination from unindexed dag_id to the primary key id (UUID7, time-ordered) in both upgrade and downgrade loops. This eliminates full table scans + sorts on every batch iteration. * Add temporary index on deadline.dagrun_id during 0101 migration Create a temporary index on deadline(dagrun_id) before the migration loop and drop it after. Without this, the per-alert UPDATE that joins dag_run to find matching deadline rows requires a full table scan of the deadline table on every iteration. * Add SQL-side deadline filtering to skip DAGs without deadlines in 0101 Use dialect-specific JSON path checks (PostgreSQL jsonb operators, MySQL JSON_EXTRACT, SQLite json_extract) to filter out serialized_dag rows that have no deadline data. This avoids transferring and processing large data blobs for the majority of DAGs that have no deadline configuration. Compressed rows are still included since the DB cannot inspect their content. * Eliminate triple decompression in 0101 migration upgrade path For DAGs with deadline data, the migration previously decompressed the serialized_dag blob 3 times: once to extract deadlines, once in update_dag_deadline_field() to modify the data, and once to recompute the dag_hash. Now the parsed dag_data is kept in memory, modified once, the hash is computed from it, and the result is written back in a single UPDATE combining data + dag_hash. * Cache dagrun IDs per DAG to avoid repeated JOINs in 0101 migration The correlated subquery joining dag_run and serialized_dag to find matching deadline rows was executed for every individual alert within a DAG. Now the dagrun IDs are fetched once per DAG and reused for each alert's UPDATE via an expanding bind parameter. * Use temporary_index context manager instead of inline try/finally Extract the create-index/drop-index pattern into a reusable context manager to reduce indentation churn and make the intent clearer at the call site. * Convert temporary_index to decorator to avoid indentation churn Extract the migration loop body into _migrate_deadline_alerts() decorated with @temporary_index, so the while loop stays at its original indentation level. The outer function keeps only the offline-mode guard. * Add comments explaining why keyset pagination uses id instead of dag_id * Move dagrun_ids select to where it being referenced * Add additional validation for existing temp index check * Remove unit test for migration * Speed up deadline alert UI migration updates Move dagrun_ids outside of inner loop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
As mentioned in the issue
What
There're several critical paths to improve
serialized_dag.dag_id(column without index) before, now we're iterating throughserialized_dag.idas batch processing partition key insteadserialized_dag.dagrun_idjust for migration as we couldn't avoid iteratingserialized_dag.dagrun_idcolumn.dag_dataWas generative AI tooling used to co-author this PR?