Decouple airflowctl tests from branch dependency and add schema evolution with strict versioning

[bugraoz93]

closes: #57826
The main value is having strict versioning and a separate flow from the core, while we have the power to support X number of versions now.
Everything will be tested over the main branch while we have a similar version control over the label.
For each core version, including into process is just a couple of lines of action

Canary Run: 3.1.7, main
Label airflowctl all versions added: 3.1.2, 3.1.3, 3.1.5, 3.1.6, 3.1.7, main
Excluded 3.1.0 and 3.1.1 because there is a problem with running the API server after use-airflow-version but will check that out. I think this should be enough for us to comfortably release

TLDR
In the previous PR #58972, I tried creating a Prod image for the older branch over main while bumping limitations. These limitations were overcome in a way and then forced to create the CI image too in order to create constraints. Then I gave up on creating more cost with 4 images rather than two. Fallback to the provider approach with a couple of internal touches to make the system alive for airflowctl and ensure older versions are installed, leveraging what is already there in Breeze.

This should be enough for release and should be fully backwards compatible now. There will be improvements but for me, let's not block the release to make new features andget this in place. Then we can do those and even clean up if needed. So please be kind on the part of improvements as we can release everything early next week :)

airflowctl all versions label will be added to run all and retrieved from SelectiveCheckd
Next improvements:

• Eliminate duplicates and fallback to the old version of the definition if they are the same (done, reduced over 2.3k line of code)
• Return teastable versions from SelectiveChecks (done)

Please don't be scared on # of lines, manually generated datamodels per version for strict schema evaluation :D
So that we can always trust the datamodel at the version running and communicating with airflowctl.

Edit: Suggestion, please collapse all datamodels/compat directory :)
I accept it can have some improvement to check if the generated are the same and fall back to the old version, which I will follow up, but strict evaluation will make us easy mind and will be safeguard when we go to 3.2, 3.3, 3.4.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[bugraoz93]

Finally, I moved the tests to main :D Normally, PR is ready and working.
https://github.com/apache/airflow/actions/runs/21968985699/job/63466790648?pr=61822
Only failed for the older version. I think Airflow 3.1.7 and the main ctl are not compatible. After this is merged, we will catch these kinds of things earlier in the process.

FAILED airflow-ctl-tests/tests/airflowctl_tests/test_airflowctl_commands.py::test_airflowctl_commands[connections update] - AssertionError: ❌ Output contained unexpected text for command 'airflowctl connections update --connection-id=test_con --conn-type=postgres'
  Matched error pattern: Server error
  Output:
  Debug mode is enabled. Please be aware that your credentials are not secure.
  Please unset AIRFLOW_CLI_DEBUG_MODE or set it to false.
  2026-02-13 00:14:18 [warning  ] Server error                   [airflowctl.api.client] extra={'detail': [{'type': 'extra_forbidden', 'loc': ['body', 'team_name'], 'msg': 'Extra inputs are not permitted', 'input': None}]}
  Server response error: Client error message: {'detail': [{'type': 'extra_forbidden', 'loc': ['body', 'team_name'], 'msg': 'Extra inputs are not permitted', 'input': None}]}
  Client error,  Please check the command and its parameters. If you need help, run the command with --help.
assert not 'Server error'

These changes made me realise I overlooked the part of adding a new field in the datamodels, which is not breaking if you send the old request from the user perspective. That is not the case if you send the new request (datamodel in main) to the old Pydantic model (v3-1-test API datamodel), which is not there. So, in main, we have some breaking changes we need to handle. The old version of airflowctl is compatible with the 3.1.7 version since we haven't backported any changes.

I will check if I can fix the backward compatibility easily because it looks a bit tricky. I have an idea to get the env var on Airflow version and keep both versions of the datamodels and switch accordingly. This will make it compatible with the old version. The tricky parts are generating the datamodel from another branch, but that should be possible.

I may need to reschedule the release a week or two if I cannot make breaking changes compatible with the older version tomorrow :( Or we can accept the breaking part since we are still in 0.1.x, which I would still be on the side of waiting a week or two to fix this, but on the other hand, features are waiting to be released. I will most prob fix it next week for sure
What do you think, @potiuk?

[bugraoz93]

Other failure not related

[bugraoz93]

Short version, the older version fails because there is a incompatibility with 3.1.7 and we couldn't catch this earlier because we haven't backported anything. I was trusting that I would finish this PR which normally working and waiting last day to sync the branches if needed. Finally we won't needed anymore but we still need to fix incompatibility in between versions and always keeping supported version's datamodels in case of this kind of incompatibility

[bugraoz93]

Fixing the backcopat with schema evolution. Will make it ready and will ready for releasing

[potiuk]

I may need to reschedule the release a week or two if I cannot make breaking changes compatible with the older version tomorrow :w( Or we can accept the breaking part since we are still in 0.1.x, which I would still be on the side of waiting a week or two to fix this, but on the other hand, features are waiting to be released. I will most prob fix it next week for sure

Fine.

We could - I guess utilise cadwyn for past release versions. Or - another option - we could always have airflowctl X.Y.* to only work with airflow X.Y.*. I think that might make sense, we woudl worry about compatibility far less. And we could couple the release process with airflow.

I think that might be best pragmatic approach

[bugraoz93]

I may need to reschedule the release a week or two if I cannot make breaking changes compatible with the older version tomorrow :w( Or we can accept the breaking part since we are still in 0.1.x, which I would still be on the side of waiting a week or two to fix this, but on the other hand, features are waiting to be released. I will most prob fix it next week for sure

Fine.

We could - I guess utilise cadwyn for past release versions. Or - another option - we could always have airflowctl X.Y.* to only work with airflow X.Y.*. I think that might make sense, we woudl worry about compatibility far less. And we could couple the release process with airflow.

I think that might be best pragmatic approach

Thanks, Jarek! I thought about this last night on schema evolution, how to iterate, and what the process should be.
I have implemented a solution entire day. Let me push it in a moment, only one thing left to handle, which command will be testable after which version, should not be complex. We discuss that solution too :D

[bugraoz93]

CI was green, but making the PR ready late made it skip
Green Draft Run: https://github.com/apache/airflow/actions/runs/22157068286
Edit: Rerun the CI to ensure the second round is green too (not flaky), which won't hurt

[kosteev]

I see that we had some review and comments here, is there anything left here to address before we merge this PR?

Bugra, Jarek can we progress/expedite submitting this PR?
IIUC This PR resolves the backward compatibility issue for airflowctl. Latest released airflowctl 0.1.2 and 0.1.3 are not working with Airflow 3.1.7/3.1.8 - various commands (airflow dags list, airflow connections import, etc.) fail due to schema incompatibility.
@bugraoz93 @potiuk

[bugraoz93]

I see that we had some review and comments here, is there anything left here to address before we merge this PR?

Bugra, Jarek can we progress/expedite submitting this PR?
IIUC This PR resolves the backward compatibility issue for airflowctl. Latest released airflowctl 0.1.2 and 0.1.3 are not working with Airflow 3.1.7/3.1.8 - various commands (airflow dags list, airflow connections import, etc.) fail due to schema incompatibility.
@bugraoz93 @potiuk

Those comments are from me to myself :D
This still hasn't reviewed. Created a discussion too, feel free to join. After review I need to rebase and should be ready to merge
https://lists.apache.org/thread/kw61jmjxbk81t5rz5b25bmxvvmpv83kz

[bugraoz93]

Just an addition, since we are at 0.x.x we do not ensure backward compatibility.

[kosteev]

Just an addition, since we are at 0.x.x we do not ensure backward compatibility.

Ok, makes sense. Bugra, but how users/customer can use it? E.g. I am running Airflow 3.1.0 or 3.1.7, which version of apache-airflow-ctl should I use?
My understanding that latest version of apache-airflow-ctl is compatible only with latest version of Airflow from main branch?

[potiuk]

Just an addition, since we are at 0.x.x we do not ensure backward compatibility.

Ok, makes sense. Bugra, but how users/customer can use it? E.g. I am running Airflow 3.1.0 or 3.1.7, which version of apache-airflow-ctl should I use? My understanding that latest version of apache-airflow-ctl is compatible only with latest version of Airflow from main branch?

Let's discuss it in devlist - the thread is here - https://lists.apache.org/thread/kw61jmjxbk81t5rz5b25bmxvvmpv83kz

[bugraoz93]

As we adopt a release structure already will make our syncs tested. Closing this until further improvement to current structure