Migration 0101_3_2_0_ui_improvements_for_deadlines upgrade is slow on deployments with large deadline and serialized_dag tables.

[vatsrahul1001]

Apache Airflow version

main (development)

If "Other Airflow 3 version" selected, which one?

No response

What happened?

Migration 0101_3_2_0_ui_improvements_for_deadlines upgrade is slow on deployments with large deadline and serialized_dag tables. With 10M deadline rows and 100K serialized_dags, the migration took ~16 minutes.

What you think should happen instead?

No response

How to reproduce

Steps to reproduce:

1. Start Airflow on 3.1.8 with PostgreSQL backend
2. Accumulate deadline rows and serialized_dag entries (100K+ deadlines to see noticeable impact)
3. Upgrade to 3.2.0 (airflow db migrate)
4. Observe migration 0101 duration scales linearly with deadline row count

Operating System

MAc

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Crystora]

@vatsrahul1001 Can I pick this up? I can do it correctly. I'd like to contribute

[henry3260]

@vatsrahul1001 Can I pick this up? I can do it correctly. I'd like to contribute

Thanks for your interest! You can directly raise a pr. Refer to https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#issue-reporting-and-resolution-process

[jason810496]

Migration 0101_3_2_0_ui_improvements_for_deadlines upgrade is slow on deployments with large deadline and serialized_dag tables. With 10M deadline rows and 100K serialized_dags, the migration took ~16 minutes.

The UPDATE phase holds row-level locks on all deadline rows for ~7 minutes, and the subsequent ALTER COLUMN ... SET NOT NULL acquires ACCESS EXCLUSIVE lock which blocks all reads and writes to the deadline table.

Hi @vatsrahul1001, may I ask does your setup keep running some dags that might cause dead lock for deadline (e.g. keep trigging new DagRun during the DB migration)? Thanks!

Since my benchmark setup only take around 6 seconds for the migration, the benchmark data setup take way more longer time than the DDL change.

I have ensured

1. I'm using 3.1.8 when setting up the Deadline datas (when running benchmark__0101_3_2_0_ui_improvements_for_deadlines__migration.py )
2. then install 3.2.0 (main branch) from this commit: https://github.com/jason810496/airflow/tree/b3055d7bb8cea89a77d7179b8896b84bac29de77
3. run airflow db migrate

Benchmark Result:

branch,run_number,duration_seconds,start_time_utc,end_time_utc
main,1,6.18,2026-03-17T19:25:17Z,2026-03-17T19:25:23Z
main,2,6.23,2026-03-17T19:28:50Z,2026-03-17T19:28:56Z
main,3,6.31,2026-03-17T19:32:26Z,2026-03-17T19:32:32Z
main,average,6.24,,
perf/migration/ui-improvement-deadline-alert-v1,1,6.43,2026-03-17T19:50:10Z,2026-03-17T19:50:17Z
perf/migration/ui-improvement-deadline-alert-v1,2,6.47,2026-03-17T19:53:53Z,2026-03-17T19:53:59Z
perf/migration/ui-improvement-deadline-alert-v1,3,6.08,2026-03-17T19:57:31Z,2026-03-17T19:57:37Z
perf/migration/ui-improvement-deadline-alert-v1,average,6.33,,
perf/migration/ui-improvement-deadline-alert-v2,1,6.23,2026-03-17T20:01:09Z,2026-03-17T20:01:15Z
perf/migration/ui-improvement-deadline-alert-v2,2,6.17,2026-03-17T20:04:43Z,2026-03-17T20:04:49Z
perf/migration/ui-improvement-deadline-alert-v2,3,6.53,2026-03-17T20:08:24Z,2026-03-17T20:08:31Z
perf/migration/ui-improvement-deadline-alert-v2,average,6.31,,
perf/migration/ui-improvement-deadline-alert-v3,1,6.57,2026-03-17T20:12:13Z,2026-03-17T20:12:20Z
perf/migration/ui-improvement-deadline-alert-v3,2,6.63,2026-03-17T20:15:58Z,2026-03-17T20:16:04Z
perf/migration/ui-improvement-deadline-alert-v3,3,6.28,2026-03-17T20:19:41Z,2026-03-17T20:19:47Z
perf/migration/ui-improvement-deadline-alert-v3,average,6.49,,
perf/migration/ui-improvement-deadline-alert-v4,1,6.39,2026-03-17T20:23:27Z,2026-03-17T20:23:33Z
perf/migration/ui-improvement-deadline-alert-v4,2,6.31,2026-03-17T20:27:05Z,2026-03-17T20:27:11Z
perf/migration/ui-improvement-deadline-alert-v4,3,6.34,2026-03-17T20:30:42Z,2026-03-17T20:30:49Z
perf/migration/ui-improvement-deadline-alert-v4,average,6.35,,

Machine:

------------------------------------------------------------------------------------
# INFRA                 INSTANCE        vCPUs   Mem(GB)   GPUS   COST ($)   CHOSEN   
------------------------------------------------------------------------------------
# GCP (us-central1-a)   n4-highcpu-16   16      32        -      0.61          ✔     
------------------------------------------------------------------------------------

Setup:

============================================================
Migration 0101 Benchmark Data Generator
============================================================
  DAGs:                1,000
  Runs/DAG:            1,000
  Deadlines/Run:       1
  Alerts/DAG (JSON):   2
  Total dag_runs:      1,000,000
  Total deadlines:     1,000,000
  Total serialized:    1,000
============================================================

Logs of Benchmark Data Setup:

[Phase 1] Setting up prerequisites...
  log_template id=1

[Phase 2] Inserting 1,000 dags + dag_versions + serialized_dags...
  1,000/1,000 (0.3s)
  Done in 0.3s

[Phase 3] Inserting 1,000,000 dag_runs + 1,000,000 deadlines...
  dag_runs: 100,000/1,000,000, deadlines: 100,000/1,000,000 (19.2s, 5,208 dl/s)
  dag_runs: 200,000/1,000,000, deadlines: 200,000/1,000,000 (38.9s, 5,147 dl/s)
  dag_runs: 300,000/1,000,000, deadlines: 300,000/1,000,000 (58.5s, 5,125 dl/s)
  dag_runs: 400,000/1,000,000, deadlines: 400,000/1,000,000 (78.2s, 5,117 dl/s)
  dag_runs: 500,000/1,000,000, deadlines: 500,000/1,000,000 (98.0s, 5,101 dl/s)
  dag_runs: 600,000/1,000,000, deadlines: 600,000/1,000,000 (117.5s, 5,105 dl/s)
  dag_runs: 700,000/1,000,000, deadlines: 700,000/1,000,000 (137.1s, 5,106 dl/s)
  dag_runs: 800,000/1,000,000, deadlines: 800,000/1,000,000 (156.7s, 5,104 dl/s)
  dag_runs: 900,000/1,000,000, deadlines: 900,000/1,000,000 (176.3s, 5,104 dl/s)
  dag_runs: 1,000,000/1,000,000, deadlines: 1,000,000/1,000,000 (195.8s, 5,107 dl/s)
  Done in 195.8s

[Phase 4] Running VACUUM ANALYZE...
  VACUUM ANALYZE deadline...
  VACUUM ANALYZE dag_run...
  VACUUM ANALYZE serialized_dag...
  Done in 0.4s

============================================================
RESULTS
============================================================
  deadline:          1,000,000 rows  (157 MB)
  dag_run:           1,000,000 rows  (393 MB)
  serialized_dag:    1,000 rows  (1032 kB)
============================================================

Total elapsed: 196.7s

Data is ready. Run the migration to benchmark: