Skip to content

Google: Guard bigquery_to_postgres's psycopg2-specific adapter registration#69493

Closed
Dev-iL wants to merge 1 commit into
apache:mainfrom
Dev-iL:psycopg3-sync-google
Closed

Google: Guard bigquery_to_postgres's psycopg2-specific adapter registration#69493
Dev-iL wants to merge 1 commit into
apache:mainfrom
Dev-iL:psycopg3-sync-google

Conversation

@Dev-iL

@Dev-iL Dev-iL commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

related:

depends on:

register_adapter(list/dict, Json) in BigQueryToPostgresOperator is psycopg2-specific and previously ran unconditionally, forcing psycopg2 to be importable even when the resolved PostgresHook connection is actually on psycopg3 (where it's a no-op). Only import and call it when PostgresHook is actually on the psycopg2 path.

Note: this module only imports cleanly without psycopg2 installed once apache-airflow-providers-postgres itself stops hard-requiring psycopg2 — that's the companion change in psycopg3-sync-postgres. This PR should be merged after (or at least released alongside) that one; see the code comment at the import site in bigquery_to_postgres.py.

What changed

  • providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_postgres.py: the from psycopg2.extensions import register_adapter / from psycopg2.extras import Json imports and the register_adapter(...) calls are now conditional on not PostgresHook.USE_PSYCOPG3, guarded inside postgres_hook, with a comment documenting the dependency on the postgres provider's own guard fix.

Test plan

  • Existing bigquery_to_postgres test suite passes.
  • New tests: test_adapters_to_json_registered_on_psycopg2_path, test_adapters_not_registered_on_psycopg3_path, test_bigquery_to_postgres_module_imports_without_psycopg2 (simulates psycopg2 absent via sys.modules patching, confirms clean import).

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Sonnet 5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

register_adapter(list/dict, Json) is psycopg2-specific and previously
ran unconditionally, forcing psycopg2 to be importable even when the
resolved PostgresHook connection is on psycopg3 (a no-op there). Only
import and call it when PostgresHook is actually on the psycopg2 path,
so the module imports cleanly without psycopg2 installed.

Note cross-branch dependency on the postgres provider's guard fix

This guard only makes bigquery_to_postgres.py import cleanly without
psycopg2 once apache-airflow-providers-postgres itself stops
hard-requiring psycopg2 (a separate, independently-branched fix for
the same migration). Document that dependency at the import site.

Also fixes the absence-simulation test's finally block, which was
reimporting the module while the fake absence was still monkeypatched,
leaving it cached in a degraded state for the rest of the test session.

Part of the migration tracked in apache#68453.
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Jul 6, 2026
@Dev-iL Dev-iL closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant