Guard XCom/DagRun pickle-removal migrations (0049, 0055) against NaN/Infinity and U+0000 in bytea->JSONB conversion

[seanmuth]

Why

Migrations 0049_3_0_0_remove_pickled_data_from_xcom_table (xcom.value) and 0055_3_0_0_remove_pickled_data_from_dagrun_table (dag_run.conf) both convert legacy pickled bytea columns to JSON/JSONB during the 2.x -> 3.x upgrade. Both can be fed values that are legal in the pickled representation but illegal in strict JSON/JSONB:

1. Non-finite floats NaN / Infinity / -Infinity (json.dumps default allow_nan=True emits the bare tokens).
2. The U+0000 (NUL) escape that json.dumps emits for embedded null bytes (common with data from fixed-width exports, C buffers, Mongo, or bad UTF-16->UTF-8 conversions).

Before this PR each migration only handled one of the two classes:

	non-finite floats	U+0000 (NUL)
0049 (xcom, SQL regexp_replace)	handled (#57893)	unhandled -> bulk CAST(... AS JSONB) aborts the whole upgrade
0055 (dag_run, Python json.dumps)	unhandled -> per-row except silently drops the conf	unhandled

The PostgreSQL error for the NUL case:

sqlalchemy.exc.DataError: (psycopg2.errors.UntranslatableCharacter) unsupported Unicode escape sequence
DETAIL: <NUL> cannot be converted to text.
CONTEXT: JSON data, line 1: ..."col": "F<NUL>...

Fixes #69063. (The NaN-in-xcom half was #57893; this also closes the symmetric NaN-in-conf silent-data-loss gap in 0055.)

What

Make both bytea->JSONB migrations guard against both value classes:

• 0049 (xcom) - keep the existing NaN/Infinity quoting; add a U+0000 strip to all three dialect branches:
  • PostgreSQL: replace(convert_from(value,'UTF8'), '<NUL-escape>', '') wrapping the regexp_replace.
  • MySQL: REPLACE(CONVERT(value USING utf8mb4), '<NUL-escape>', '') (same file-vs-SQL backslash escaping as the existing NaN regex in that branch).
  • SQLite: an extra inner REPLACE(..., '<NUL-escape>', '') (no REGEXP_REPLACE, consistent with the existing approach).
• 0055 (dag_run.conf) - add a recursive _json_safe() that quotes non-finite floats to "NaN"/"Infinity"/"-Infinity" (mirroring 0049's SQL output) before json.dumps, and strip the U+0000 escape from the result. 0055 is dialect-agnostic (Python serialization), so this one helper covers all backends. The conf is now preserved instead of dropped.

NaN/Infinity are quoted (kept as strings, matching 0049); U+0000 cannot be quoted/kept, so it is stripped. json.dumps (default ensure_ascii=True) only ever emits a NUL as the 6-char escape, never a raw 0x00, so stripping the escape covers values produced by normal serialization. Migration files are amended in place per the project convention for migration bugfixes.

Minimal repro (NUL)

-- '\x7b226b223a2022465c75303030306f6f227d' decodes to: {"k": "F<NUL>oo"}
SELECT CAST(CONVERT_FROM('\x7b226b223a2022465c75303030306f6f227d'::bytea, 'UTF8') AS JSONB);
-- ERROR: unsupported Unicode escape sequence  /  DETAIL: <NUL> cannot be converted to text.

To keep the SQL testable, 0049's per-dialect sanitize statements are extracted into _xcom_{pg,mysql,sqlite}_sanitize_sql() helpers (same pattern as migration 0117); the production calls default to table="xcom", so the emitted SQL is byte-for-byte unchanged.

Notes

• Migration unit tests added: NaN/Infinity/-Infinity and a NUL-bearing value, for xcom (all dialects) and dag_run.conf.
• Happy to add a newsfragment if maintainers want one for an in-place migration bugfix — let me know the preferred category.

Related: #57893 (the NaN/Infinity sanitization for the xcom conversion).

---

Authored with Claude Code (Claude Opus 4.8).

[phanikumv]

On the newsfragment question: skip it. Prior in-place fixes to these same migrations (#62686, #57866, #53812) landed without one, and since both files target the already-released 3.0.0, a note would just show up in a later changelog for a
migration which users have already run or never hit.