Fix ECS Executor compatibility with Airflow 3.x in try_adopt_task_instances by iamapez · Pull Request #58207 · apache/airflow

iamapez · 2025-11-11T17:54:37Z

Summary

The try_adopt_task_instances method in the ECS Executor was calling ti.command_as_list() which doesn't exist in Airflow 3.x due to Task SDK changes. This caused the scheduler to crash with AttributeError when trying to adopt orphaned ECS tasks.

The Problem

In Airflow 3.x, the TaskInstance.command_as_list() method was removed as part of the Task SDK refactoring
The ECS Executor's try_adopt_task_instances method still calls this non-existent method at line 598
This causes immediate scheduler failure on startup when using ECS Executor with Airflow 3.x

The Solution

This PR adds version-aware code that:

Checks AIRFLOW_V_3_0_PLUS to determine the Airflow version
For Airflow 3.x: Reconstructs the command list manually from task instance attributes
For Airflow 2.x: Uses the existing command_as_list() method

Testing

Python syntax validated with py_compile
The fix maintains backward compatibility with Airflow 2.x
Scheduler starts successfully with this fix applied to Airflow 3.1.1

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
The fix is minimal and focused on the specific issue
Maintains backward compatibility with Airflow 2.x

…tances The try_adopt_task_instances method was calling ti.command_as_list() which doesn't exist in Airflow 3.x due to Task SDK changes. This caused scheduler to crash with AttributeError when trying to adopt orphaned ECS tasks. This fix: - Checks AIRFLOW_V_3_0_PLUS to determine Airflow version - For Airflow 3.x: Reconstructs the command list manually from task instance attributes - For Airflow 2.x: Uses the existing command_as_list() method Fixes apache#58205

boring-cyborg · 2025-11-11T17:54:40Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

In Airflow 3.x, the TaskInstance.execution_date attribute was renamed to logical_date. This updates the fix to use the correct attribute name.

iamapez · 2025-11-11T18:03:12Z

Updated: Also fixed execution_date → logical_date rename in Airflow 3.x.

The TaskInstance attribute execution_date was renamed to logical_date in Airflow 3.x, so the fix now uses the correct attribute name when constructing the command for adopted tasks.

o-nikolas · 2025-11-12T03:49:20Z

+                        command = [
+                            "airflow",
+                            "tasks",
+                            "run",
+                            ti.dag_id,
+                            ti.task_id,
+                            ti.logical_date.isoformat(),
+                        ]


This is not how you want to generate a cmd for an Airflow 3.x task. The commands look different in AF3.

See here: https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py#L509-L518

Could consider making this a helper function now as well, instead of copy/pasting the implementation here.

Match the command generation pattern used for newly queued tasks. Creates ExecuteTask workload, serializes it as JSON, and constructs the proper SDK command structure.

iamapez · 2025-11-12T21:52:25Z

Updated to use proper Task SDK workload serialization pattern.

Now matches the command generation used for newly queued tasks (lines 509-518):

Creates ExecuteTask workload with ExecuteTask.make(ti)
Serializes with model_dump_json()
Constructs SDK command: python -m airflow.sdk.execution_time.execute_workload --json-string <serialized>

o-nikolas

Is there a unit tests we can add for this?

o-nikolas · 2025-11-12T23:58:43Z

+                        command = [
+                            "airflow",
+                            "tasks",
+                            "run",
+                            ti.dag_id,
+                            ti.task_id,
+                            ti.logical_date.isoformat(),
+                        ]


Could consider making this a helper function now as well, instead of copy/pasting the implementation here.

Removing request for change as the critical piece was addressed

o-nikolas · 2025-11-26T20:08:22Z

@iamapez Are you planning to get back to this one?

I've removed my request for changes since you've addressed the main issue. But I still think tidying the command code into a helper since it's now copied in two places is worth doing.

Address code review feedback by consolidating duplicated command generation into reusable helper methods and adding comprehensive test coverage. Changes: - Add _serialize_workload_to_command() static method for workload serialization - Add _build_task_command() method for version-aware command generation - Refactor execute_async() to use helper method (removed 9 lines of duplication) - Refactor try_adopt_task_instances() to use helper method (removed 18 lines) - Add comprehensive unit tests for both Airflow 2.x and 3.x compatibility - Update test_try_adopt_task_instances and remove skip marker for Airflow 3.x - Add Tradepost Markets Inc to INTHEWILD.md Test coverage: - test_serialize_workload_to_command: validates workload-to-command conversion - test_build_task_command_airflow3: validates Airflow 3.x command generation - test_build_task_command_airflow2: validates Airflow 2.x command generation - test_try_adopt_task_instances: updated for Airflow 3.x support

iamapez · 2025-11-28T06:45:33Z

Consolidated the command generation logic into helper methods and added unit tests as requested. @o-nikolas

iamapez · 2025-12-10T08:13:39Z

checking in on this pr @o-nikolas

iamapez · 2025-12-11T00:11:10Z

sorry @o-nikolas accidental commit to this branch - i reverted the changes

o-nikolas · 2025-12-12T19:55:46Z

sorry @o-nikolas accidental commit to this branch - i reverted the changes

@iamapez Looks like there are many test failures. Both static checks and unit test failures. Can you please have a look?

github-actions · 2026-01-28T00:20:34Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

o-nikolas · 2026-01-28T00:25:50Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@iamapez any plans to get back to this? Or shall someone take over the development for this one?

o-nikolas · 2026-02-18T22:22:22Z

CC @vincbeck on this one

vincbeck · 2026-02-19T21:19:48Z

Due to inactivity, closing this one in favor of #62192

iamapez requested a review from o-nikolas as a code owner November 11, 2025 17:54

boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Nov 11, 2025

Fix execution_date renamed to logical_date in Airflow 3.x

6be38ff

In Airflow 3.x, the TaskInstance.execution_date attribute was renamed to logical_date. This updates the fix to use the correct attribute name.

o-nikolas previously requested changes Nov 12, 2025

View reviewed changes

Use proper Task SDK workload serialization for Airflow 3.x adopted tasks

654b512

Match the command generation pattern used for newly queued tasks. Creates ExecuteTask workload, serializes it as JSON, and constructs the proper SDK command structure.

iamapez requested a review from o-nikolas November 12, 2025 21:54

o-nikolas reviewed Nov 12, 2025

View reviewed changes

o-nikolas self-requested a review November 26, 2025 20:06

iamapez force-pushed the fix-ecs-executor-airflow3-compatibility branch from 3fd19b5 to ca4bdf8 Compare November 28, 2025 06:44

iamapez added 5 commits November 28, 2025 00:46

Merge branch 'main' into fix-ecs-executor-airflow3-compatibility

c6dffa3

Merge branch 'main' into fix-ecs-executor-airflow3-compatibility

b638da5

Merge branch 'main' into fix-ecs-executor-airflow3-compatibility

824498d

Merge branch 'main' into fix-ecs-executor-airflow3-compatibility

3f69a42

Merge branch 'main' into fix-ecs-executor-airflow3-compatibility

59231e2

o-nikolas reviewed Dec 10, 2025

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/exceptions.py Outdated

iamapez force-pushed the fix-ecs-executor-airflow3-compatibility branch from d1bc3ee to 59231e2 Compare December 11, 2025 00:10

Merge branch 'main' into fix-ecs-executor-airflow3-compatibility

244dcdf

github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 28, 2026

o-nikolas removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 28, 2026

SameerMesiah97 mentioned this pull request Feb 18, 2026

Use Task SDK BaseHook in Amazon executors #62090

Closed

vincbeck mentioned this pull request Feb 19, 2026

Fix ECS Executor compatibility with Airflow 3.x in try_adopt_task_instances #62192

Merged

vincbeck closed this Feb 19, 2026

This was referenced Feb 26, 2026

Status of testing Providers that were prepared on February 26, 2026 #62537

Closed

Status of testing Providers that were prepared on March 03, 2026 #62794

Closed

Uh oh!

Conversation

iamapez commented Nov 11, 2025

Summary

The Problem

The Solution

Testing

Checklist

Uh oh!

boring-cyborg Bot commented Nov 11, 2025

Uh oh!

iamapez commented Nov 11, 2025

Uh oh!

o-nikolas Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

o-nikolas Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

iamapez commented Nov 12, 2025

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

o-nikolas Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

o-nikolas commented Nov 26, 2025

Uh oh!

iamapez commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iamapez commented Dec 10, 2025

Uh oh!

Uh oh!

iamapez commented Dec 11, 2025

Uh oh!

o-nikolas commented Dec 12, 2025

Uh oh!

github-actions Bot commented Jan 28, 2026

Uh oh!

o-nikolas commented Jan 28, 2026

Uh oh!

o-nikolas commented Feb 18, 2026

Uh oh!

vincbeck commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

iamapez commented Nov 28, 2025 •

edited

Loading