Airflow 2.11 -> 3.1 Upgrade with Helm Chart 1.18.0 stuck on DB migration pod

[berglh]

Apache Airflow version

2.11.0

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

After waiting for the Airflow 3.1 release for the initial bugs in Airflow 3 to be resolved, I started the upgrade from Airflow 2.11.0 deployed using the Helm 1.16.0 chart to Airflow 3.1.0 on the Helm 1.18.0 chart.

I had followed the Airflow Uprage 3.0 guide and ran ruff to check for upgrade warnings to ensure our DAGs were all compatible (apart from the airflow.sdk related class/method future deprecation notifications) with Airflow 3.0 prior to upgrade. I used all four AIR{VERSION} checkers available.

1. I then received the following error in the DB migration job pod, which cyclically restarted until the Helm upgrade failed in Terraform. It's possible this traceback was from a second or third run of the airflow DB migration pod executing. Essentially, the type conversion of the value column in the xcom table from bytea to jsonb failed.

DB Migration Pod Traceback

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type json
DETAIL:  Token "nan" is invalid.
CONTEXT:  JSON data, line 1: ...2FSDH3SbQ8Q2TmlrUFDaCptskUXZOdM6bwpZShBKsJfp1"nan...


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line 55, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/cli.py", line 114, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/db_command.py", line 207, in migratedb
    run_db_migrate_command(args, db.upgradedb, _REVISION_HEADS_MAP)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/db_command.py", line 135, in run_db_migrate_command
    command(
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 100, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db.py", line 1136, in upgradedb
    command.upgrade(config, revision=to_revision or "heads")
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/command.py", line 483, in upgrade
    script.run_env()
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/script/base.py", line 549, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 116, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 136, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/migrations/env.py", line 138, in <module>
    run_migrations_online()
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/migrations/env.py", line 132, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/migrations/versions/0049_3_0_0_remove_pickled_data_from_xcom_table.py", line 125, in upgrade
    op.execute(
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/operations/ops.py", line 2591, in execute
    return operations.invoke(op)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/operations/base.py", line 441, in invoke
    return fn(self, operation)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/operations/toimpl.py", line 240, in execute_sql
    operations.migration_context.impl.execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/ddl/impl.py", line 253, in execute
    self._exec(sql, execution_options)
  File "/home/airflow/.local/lib/python3.12/site-packages/alembic/ddl/impl.py", line 246, in _exec
    return conn.execute(construct, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/future/engine.py", line 286, in execute
    return self._execute_20(
           ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type json
DETAIL:  Token "nan" is invalid.
CONTEXT:  JSON data, line 1: ...2FSDH3SbQ8Q2TmlrUFDaCptskUXZOdM6bwpZShBKsJfp1"nan...

[SQL: 
            ALTER TABLE xcom
            ALTER COLUMN value TYPE JSONB
            USING CASE
                WHEN value IS NOT NULL THEN CAST(CONVERT_FROM(value, 'UTF8') AS JSONB)
                ELSE NULL
            END
            ]
(Background on this error at: https://sqlalche.me/e/14/9h9h)

2. I then tried a variety of things, like deleting all the xcom entries entirely. I eventually decided to delete the entire airflow DB and recreated it again. After doing this, the deployment started successfully.
3. I then tried to execute a common DAG that does the following:
   • Runs the SparkSubmitOperator to run an RDS to S3 (Parquet) job in Apache Spark (Kubernetes) cluster
   • Runs an Athena query to update the AWS Glue Table location in S3
   • Tracks the progress of the Athena query
   • Returns the result of the Athena query to the Airflow log
4. The scheduler cyclically failed to submit the task with an error (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(20) in the callback_request table for the callback_type value EmailNotificationRequest exceeding 20 characters in length.

📜Note: Please find all the logs below from the pods that I grabbed before reverting to Airflow 2.11/Helm Chart Official 1.16.0.

github-api-server-3-1.log
github-dag-processor-3-1.log
github-pgbouncer-3-1.log
github-scheduler-3-1.log
github-tirggerer-3-1.log
github-worker-3-1.log
postgres-db-upgrade-error.log

What you think should happen instead?

1. The db migration should complete successfully.
2. Tasks with the callback_request of EmailNotificationRequest should not fail to schedule, not sure if this is something I've missed in the upgrade notes.

How to reproduce

This is likely a complex issue related to the contents of my xcom table, I would be happy to perform any queries that would assist in determining the likely cause of the type conversion issue.

Upgraded Airflow from 2.11 on Helm Official 1.16.0 -> Airflow 3.1 on Helm Official 1.18.0.

Operating System

Amazon Linux AL2023 (EKS v1.31)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.8.0
apache-airflow-providers-apache-spark==5.3.2
apache-airflow-providers-celery==3.11.0
apache-airflow-providers-cncf-kubernetes==10.8.2
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-docker==4.4.0
apache-airflow-providers-elasticsearch==6.3.0
apache-airflow-providers-fab==1.5.3
apache-airflow-providers-ftp==3.13.0
apache-airflow-providers-google==15.1.0
apache-airflow-providers-grpc==3.8.0
apache-airflow-providers-hashicorp==4.2.0
apache-airflow-providers-http==5.3.0
apache-airflow-providers-imap==3.9.0
apache-airflow-providers-microsoft-azure==12.4.0
apache-airflow-providers-mysql==6.3.0
apache-airflow-providers-odbc==4.10.0
apache-airflow-providers-openlineage==2.3.0
apache-airflow-providers-postgres==6.2.0
apache-airflow-providers-redis==4.1.0
apache-airflow-providers-sendgrid==4.1.0
apache-airflow-providers-sftp==5.3.0
apache-airflow-providers-slack==9.1.0
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-snowflake==6.3.1
apache-airflow-providers-sqlite==4.1.0
apache-airflow-providers-ssh==4.1.0
apache-airflow-providers-standard==1.9.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow is deployed in Amazon EKS using GitLab CI/CD and Terraform.

• Database: Aurora Postgres 16.9 (db.r8g.large)
• Redis: ElastiCache Redis OSS 7.1.0 (cache.t4g.small)
• EKS: v1.31
• Terraform: Latest Terraform container & providers for AWS & Helm
• DAGs & custom Python modules are mounted into Airflow pods using the Amzon S3 driver for EKS (dataset repositories deploy their DAGs to S3).
• Logging is configured to log to S3
• AWS authentication is handled by EKS IRSA via WebIdentityToken in the Airflow pods. I add OIDC trust relationships to the Airflow IAM role to allow the service accounts for the Airflow pods to assume the IAM role. Manually running aws s3 CLI commands in the pods showed working access to AWS resources.
• Custom Airflow container build that does the following:
  • Installs openjdk17/aws cli
  • Bakes in our Apache Spark build for SparkSubmitOperator
  • Adds a start-up script that copies some Spark jars from S3 to the Spark /opt/spark/jars folder
  • Installs the following providers upgraded to latest: apache-airflow-providers-cncf-kubernetes, apache-airflow-providers-standard and apache-airflow-providers-apache-spark>=5.3.2 for the fix to the Spark Job Operator fix for tracking multiple pods KubernetesJobOperator fails if you launch more than one pod #44994
• The Terraform configuration establishes the namespace, RDS cluster, Redis OSS ElastiCache, IAM role/policies/OIDC trust policy, VPC/Security Groups and all other dependicies for Airflow. After all the depdencies are deployed, the Helm provider deploys airflow to the airflow namespace using the following values configure (in this case for Airflow 3.1): airflow-3.1-values.txt

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[tirkarthi]

Some workarounds you can apply for EmailNotificationRequest error with the fix. I am not sure if helm charts are using Airflow 3.1.1

#56426 (comment)

#56429

[berglh]

To continue this troubleshooting, I replicated our RDS to a test DB instance. I then created interactive Airflow pods using kubectl to interact with the test RDS instance.

Start Airflow 2.11.0 container

1. airflow db check
2. airflow db shell (\conninfo to check connection details are using the correct test DB)
3. airflow db clean --clean-before-timestamp '2025-09-01 00:00:00.000' --skip-archive -y --dry-run
4. airflow db clean --clean-before-timestamp '2025-09-01 00:00:00.000' --skip-archive -y
5. airflow db migrate -s to see if any migrations are pending, which there were not (happy cat output)'

Start Airflow 3.1.1 container

1. airflow db check
2. airflow db shell (\conninfo to check connection details are using the correct test DB)
3. airflow db migrate -s to capture the SQL upgrade commands
4. airflow db migrate

This resulted in the same issue as I experienced during the Helm upgrade. This removed all config from Airflow except for the DB connection string. So, the query before the failure is a query to update the xcom table where it converts values to be compatible with the JSONB data format Remove pickled data from xcom table.

UPDATE xcom
                SET value = convert_to(replace(convert_from(value, 'UTF8'), 'NaN', '"nan"'), 'UTF8')
                WHERE value IS NOT NULL AND get_byte(value, 0) != 128;

Then directly after this, I believe is the query to cast the column to JSONB:

ALTER TABLE xcom
            ALTER COLUMN value TYPE JSONB
            USING CASE
                WHEN value IS NOT NULL THEN CAST(CONVERT_FROM(value, 'UTF8') AS JSONB)
                ELSE NULL
            END;

With the error:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type json
DETAIL:  Token "nan" is invalid.
CONTEXT:  JSON data, line 1: ...2FSDH3SbQ8Q2TmlrUFDaCptskUXZOdM6bwpZShBKsJfp1"nan...

I will try to find the entry in the xcom table that it's tripping up on. Here are the full SQL queries and db migrate logs:

Airflow DB Migrate SQL upgrade commands
Airflow DB Migrate Full Log

[berglh]

I've tracked down the record matching this in the xcom table. I had to truncate this for securty, but the value column field that is referenced in the type conversion failure is 1857 characters long. It appears to be a text value that is an array/list with open/close square brackets with a quoted string that is an HTTPS S3 URL with access credentials. I've modified/truncated all the values to avoid leaking the bucket/access credentials.

137350  report_status  -1  assets  ech_asset_report  scheduled__2025-09-04T01:00:00+00:00  ["https://echo-export-bucket-name.s3.ap-southeast-2.amazonaws.com/reports/60d4291f-70de-4444-43334-d11111111d/a4fcbaf9-e53d-9d47-8607-ed53a9de298b.csv.gz?X-Amz-Security-Token=.....%2FSDH3SbQ8Q2TmlrUFDaCptskUXZOdM6bwpZShBKsJfp1.....&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250905T010112Z&X-Amz-SignedHeaders=host&X-Amz-Expires=604800&X-Amz-Credential=CREDENTIALCODE%2CREDENTIALCODE%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Signature=hexsignature"]  2025-09-05 11:01:12.246 +1000

When I run the function using ariflow db shell it seems to convert just fine:

SELECT CAST(CONVERT_FROM('["full_column_value"]')::bytea, 'UTF8') AS JSONB);

So I tried to run the query in airflow db shell and it seemed to work just fine. Very curious, why would the upgrade script be any different?

airflow=> ALTER TABLE xcom
            ALTER COLUMN value TYPE JSONB
            USING CASE
                WHEN value IS NOT NULL THEN CAST(CONVERT_FROM(value, 'UTF8') AS JSONB)
                ELSE NULL
            END;

ALTER TABLE
airflow=> 
airflow=> \d+ xcom
                                                        Table "public.xcom"
   Column   |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description 
------------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 dag_run_id | integer                  |           | not null |               | plain    |             |              | 
 task_id    | character varying(250)   |           | not null |               | extended |             |              | 
 map_index  | integer                  |           | not null | '-1'::integer | plain    |             |              | 
 key        | character varying(512)   |           | not null |               | extended |             |              | 
 dag_id     | character varying(250)   |           | not null |               | extended |             |              | 
 run_id     | character varying(250)   |           | not null |               | extended |             |              | 
 value      | jsonb                    |           |          |               | extended |             |              | 
 timestamp  | timestamp with time zone |           | not null |               | plain    |             |              | 
Indexes:
    "xcom_pkey" PRIMARY KEY, btree (dag_run_id, task_id, map_index, key)
    "idx_xcom_key" btree (key)
    "idx_xcom_task_instance" btree (dag_id, task_id, run_id, map_index)
Foreign-key constraints:
    "xcom_task_instance_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
Access method: heap

After I manually ran this in airflow db shell, I attempted the airflow db migrate again and then received a new error:

2025-10-31T03:42:24.391728Z [info     ] Context impl PostgresqlImpl.   [alembic.runtime.migration] loc=migration.py:211
2025-10-31T03:42:24.391921Z [info     ] Will assume transactional DDL. [alembic.runtime.migration] loc=migration.py:214
2025-10-31T03:42:24.412924Z [info     ] Migrating the Airflow database [airflow.utils.db] loc=db.py:1129
2025-10-31T03:42:24.440804Z [info     ] Context impl PostgresqlImpl.   [alembic.runtime.migration] loc=migration.py:211
2025-10-31T03:42:24.440927Z [info     ] Will assume transactional DDL. [alembic.runtime.migration] loc=migration.py:214
2025-10-31T03:42:24.474365Z [info     ] Running upgrade 9fc3fc5de720 -> eed27faa34e3, Remove pickled data from xcom table. [alembic.runtime.migration] loc=migration.py:622
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function get_byte(jsonb, integer) does not exist
LINE 5:             WHERE value IS NOT NULL AND get_byte(value, 0) =...
                                                ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[SQL: 
            INSERT INTO _xcom_archive (dag_run_id, task_id, map_index, key, dag_id, run_id, value, timestamp)
            SELECT dag_run_id, task_id, map_index, key, dag_id, run_id, value, timestamp
            FROM xcom
            WHERE value IS NOT NULL AND get_byte(value, 0) = 128
            ]
(Background on this error at: https://sqlalche.me/e/14/f405)

airflow@airflow-db-debug:/opt/airflow$ airflow db shell
DB: postgresql+psycopg2://airflow:***@airflow3-test-cluster.ourcluster.ap-southeast-2.rds.amazonaws.com/airflow
2025-10-31T03:46:31.488210Z [info     ] Executing cmd: psql            [airflow.utils.process_utils] loc=process_utils.py:211
psql (18.0 (Debian 18.0-1.pgdg12+3), server 16.9)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: none)
Type "help" for help.

airflow=> \dt+ _xcom_archive
Did not find any tables named "_xcom_archive".

So, I created the _xcom_archive table using the SQL in the airflow db migrate -s command in airflow db shell:

airflow=> CREATE TABLE IF NOT EXISTS _xcom_archive (
    dag_run_id INTEGER NOT NULL, 
    task_id VARCHAR(250) NOT NULL, 
    map_index INTEGER DEFAULT -1 NOT NULL, 
    key VARCHAR(512) NOT NULL, 
    dag_id VARCHAR(250) NOT NULL, 
    run_id VARCHAR(250) NOT NULL, 
    value BYTEA, 
    timestamp TIMESTAMP WITH TIME ZONE NOT NULL, 
    CONSTRAINT _xcom_archive_pkey PRIMARY KEY (dag_run_id, task_id, map_index, key)
);
CREATE TABLE

\d+ _xcom_archive
                                                    Table "public._xcom_archive"
   Column   |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Descript
ion 
------------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+---------
----
 dag_run_id | integer                  |           | not null |               | plain    |             |              | 
 task_id    | character varying(250)   |           | not null |               | extended |             |              | 
 map_index  | integer                  |           | not null | '-1'::integer | plain    |             |              | 
 key        | character varying(512)   |           | not null |               | extended |             |              | 
 dag_id     | character varying(250)   |           | not null |               | extended |             |              | 
 run_id     | character varying(250)   |           | not null |               | extended |             |              | 
 value      | bytea                    |           |          |               | extended |             |              | 
 timestamp  | timestamp with time zone |           | not null |               | plain    |             |              | 
Indexes:
    "_xcom_archive_pkey" PRIMARY KEY, btree (dag_run_id, task_id, map_index, key)
Access method: heap

Then when I try to run the SQL in the migration script error it fails in postgres as well:

airflow=> INSERT INTO _xcom_archive (dag_run_id, task_id, map_index, key, dag_id, run_id, value, timestamp)
            SELECT dag_run_id, task_id, map_index, key, dag_id, run_id, value, timestamp
            FROM xcom
            WHERE value IS NOT NULL AND get_byte(value, 0) = 128;
ERROR:  function get_byte(jsonb, integer) does not exist
LINE 4:             WHERE value IS NOT NULL AND get_byte(value, 0) =...
                                                ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[berglh]

@tirkarthi I checked the callback_request.py file in the Airflow 3.1.1 testing I did with the above comments, and it does look like the patch you referenced for EmailNotificationRequest not fitting into the callback_request table appears to be applied.

[seanmuth]

@berglh uses 3.1.3, issue fixed here: #57893

[berglh]

@seanmuth Thanks for the update, I'll clone our RDS and test with the new container and report back :)

[berglh]

@seanmuth - I tested the the DB migration with Airflow 3.1.3 and and the RDS snapshot, it seems to have completed successful. We'll try the production upgrade next and close this off as complete should it finish successfully.

[berglh]

I can confirm all my issues were resolved with Airflow 3.1.3 and Helm 1.18.0 chart. Experienced some other issues outside of the scope of this Airflow issue. Many thanks to everyone that worked on #57893.