Add logging to detect try number race#62703
Merged
ephraimbuddy merged 2 commits intoMar 3, 2026
Merged
Conversation
boring-cyborg
Bot
added
the
area:Scheduler
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds targeted logging (and unit tests) to help detect try_number mismatches/races in the scheduler flow, particularly around TI scheduling and executor event processing (related to #57618).
Changes:
- Add a debug-gated post-update DB read in
DagRun.schedule_tis()to warn when the persistedtry_numberdiffers from the expected value. - Add additional scheduler logs/warnings around queueing workloads and handling executor events with mismatched/multiple
try_numbers. - Add/extend unit tests to assert the new warnings/logging behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
airflow-core/src/airflow/models/dagrun.py |
Adds debug-gated DB verification and warning logging for try_number mismatches after scheduling. |
airflow-core/src/airflow/jobs/scheduler_job_runner.py |
Adds more context-rich logs for queueing/scheduling and warnings for executor events with conflicting try_numbers. |
airflow-core/tests/unit/models/test_dagrun.py |
Adds tests validating warning behavior for schedule_tis() try-number mismatch checks. |
airflow-core/tests/unit/jobs/test_scheduler_job.py |
Extends/adds tests asserting new scheduler warnings via caplog. |
ephraimbuddy
added
the
type:misc/internal
kaxil
approved these changes
Mar 2, 2026
phanikumv
approved these changes
Mar 2, 2026
ephraimbuddy
force-pushed
the
add-logging-to-detect-try-number-race
branch
from
2c3047e to
3bfd673
Compare
This adds more logging to select places that try_number mismatch could happen and would help us detect and fix the issue. Related: apache#57618
ephraimbuddy
force-pushed
the
add-logging-to-detect-try-number-race
branch
from
a7a5e17 to
e0431ad
Compare
Contributor
Backport failed to create: v3-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 95784d9 v3-1-testThis should apply the commit to the v3-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
ephraimbuddy
added a commit
that referenced
this pull request
Mar 4, 2026
ephraimbuddy
added a commit
that referenced
this pull request
Mar 4, 2026
vatsrahul1001
pushed a commit
that referenced
this pull request
Mar 4, 2026
* Add logging to detect try number race (#62703) * Log try_number mismatches during TI scheduling for HA race diagnosis This adds more logging to select places that try_number mismatch could happen and would help us detect and fix the issue. Related: #57618 * Add tests (cherry picked from commit 95784d9) * fixup! Add logging to detect try number race (#62703) * fixup! fixup! Add logging to detect try number race (#62703)
vatsrahul1001
pushed a commit
that referenced
this pull request
Mar 4, 2026
* Add logging to detect try number race (#62703) * Log try_number mismatches during TI scheduling for HA race diagnosis This adds more logging to select places that try_number mismatch could happen and would help us detect and fix the issue. Related: #57618 * Add tests (cherry picked from commit 95784d9) * fixup! Add logging to detect try number race (#62703) * fixup! fixup! Add logging to detect try number race (#62703)
dominikhei
pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
* Log try_number mismatches during TI scheduling for HA race diagnosis This adds more logging to select places that try_number mismatch could happen and would help us detect and fix the issue. Related: apache#57618 * Add tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds more logging to select places that try_number mismatch could happen and would help us detect and fix the issue.
Related: #57618
Was generative AI tooling used to co-author this PR?
GPT-5.3-codex