Skip to content

Add option to have structured JSON logging for _all_ API server output#63365

Merged
ashb merged 3 commits into
apache:mainfrom
astronomer:worktree-api-server-http-log-structlog
Mar 12, 2026
Merged

Add option to have structured JSON logging for _all_ API server output#63365
ashb merged 3 commits into
apache:mainfrom
astronomer:worktree-api-server-http-log-structlog

Conversation

@ashb

@ashb ashb commented Mar 11, 2026

Copy link
Copy Markdown
Member

Previously gunicorn/uvicorn each installed their own log handlers and
formatters, bypassing Airflow's structlog ProcessorFormatter entirely. This
meant:

  • Gunicorn set up its own StreamHandler on gunicorn.error / gunicorn.access,
    so those records never went through structlog.
  • Uvicorn workers called logging.config.dictConfig(LOGGING_CONFIG) on startup,
    overwriting the structlog configuration that was applied before gunicorn
    started.
  • HTTP access log lines came from uvicorn's built-in access logger
    (unstructured).
  • Python warnings.warn() calls and unhandled exceptions bypassed structlog
    too.
  • logs emitted via structlog went to stdout; logs emitted via stdlib ended up on
    stderr

Now:

  • AirflowGunicornLogger overrides setup() to skip installing gunicorn's own
    handlers and lets records propagate to root (where structlog is configured).
  • AirflowUvicornWorker sets log_config=None so uvicorn doesn't clobber the
    logging config, and access_log=False since we handle that ourselves.
  • HttpAccessLogMiddleware replaces uvicorn's access log: one structured event
    per request with method, path, status code, duration (µs), client address,
    and the x-request-id header bound to the structlog context for the request
    lifetime. Health-check paths are excluded to avoid noise.
  • configure_logging() now explicitly silences uvicorn.access / gunicorn.access
    and routes uvicorn.error / gunicorn.error through the default handler.
  • Python warnings are intercepted and emitted as structured py.warnings log
    events instead of going to stderr.
  • In JSON log mode, unhandled exceptions are emitted via structlog rather than
    the plain-text default sys.excepthook.
  • All logs are sent to stdout.

Pics of what it looks like

(non json version)

Screenshot 2026-03-11 at 17 33 35

(json version)
Screenshot 2026-03-11 at 17 34 23

Comment thread airflow-core/src/airflow/api_fastapi/auth/managers/simple/simple_auth_manager.py Dismissed
Comment thread airflow-core/src/airflow/api_fastapi/auth/managers/simple/simple_auth_manager.py Dismissed

@vincbeck vincbeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not an expert in some areas but overall looks good! Looks like a nice improvement!

Comment thread airflow-core/src/airflow/api_fastapi/core_api/app.py
Comment thread shared/logging/src/airflow_shared/logging/structlog.py
Comment thread airflow-core/src/airflow/utils/helpers.py Outdated
Comment thread airflow-core/src/airflow/cli/commands/api_server_command.py
Comment thread airflow-core/src/airflow/api_fastapi/gunicorn_app.py
Comment thread shared/logging/src/airflow_shared/logging/structlog.py
Comment thread shared/logging/src/airflow_shared/logging/structlog.py Outdated
Comment thread airflow-core/src/airflow/api_fastapi/auth/managers/simple/simple_auth_manager.py Outdated
@ashb

This comment was marked as resolved.

@ashb ashb force-pushed the worktree-api-server-http-log-structlog branch from 3dd1e99 to 5349a02 Compare March 12, 2026 10:37
Previously gunicorn/uvicorn each installed their own log handlers and
formatters, bypassing Airflow's structlog ProcessorFormatter entirely. This
meant:

- Gunicorn set up its own StreamHandler on gunicorn.error / gunicorn.access,
  so those records never went through structlog.
- Uvicorn workers called logging.config.dictConfig(LOGGING_CONFIG) on startup,
  overwriting the structlog configuration that was applied before gunicorn
  started.
- HTTP access log lines came from uvicorn's built-in access logger
  (unstructured).
- Python warnings.warn() calls and unhandled exceptions bypassed structlog
  too.

Now:

- AirflowGunicornLogger overrides setup() to skip installing gunicorn's own
  handlers and lets records propagate to root (where structlog is configured).
- AirflowUvicornWorker sets log_config=None so uvicorn doesn't clobber the
  logging config, and access_log=False since we handle that ourselves.
- HttpAccessLogMiddleware replaces uvicorn's access log: one structured event
  per request with method, path, status code, duration (µs), client address,
  and the x-request-id header bound to the structlog context for the request
  lifetime. Health-check paths are excluded to avoid noise.
- configure_logging() now explicitly silences uvicorn.access / gunicorn.access
  and routes uvicorn.error / gunicorn.error through the default handler.
- Python warnings are intercepted and emitted as structured py.warnings log
  events instead of going to stderr.
- In JSON log mode, unhandled exceptions are emitted via structlog rather than
  the plain-text default sys.excepthook.
@ashb ashb force-pushed the worktree-api-server-http-log-structlog branch from 5349a02 to dbdd825 Compare March 12, 2026 10:58
@ashb ashb requested a review from kaxil March 12, 2026 16:21
Comment thread airflow-core/newsfragments/63365.significant.rst Outdated
Comment thread airflow-core/newsfragments/63365.significant.rst Outdated
Comment thread airflow-core/src/airflow/utils/db.py Outdated
Comment thread airflow-core/newsfragments/63365.significant.rst Outdated
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Comment thread airflow-core/src/airflow/api_fastapi/auth/managers/simple/simple_auth_manager.py Outdated
@ashb ashb merged commit 0266b0b into apache:main Mar 12, 2026
132 checks passed
@ashb ashb deleted the worktree-api-server-http-log-structlog branch March 12, 2026 19:56
Pyasma pushed a commit to Pyasma/airflow that referenced this pull request Mar 13, 2026
apache#63365)

Previously gunicorn/uvicorn each installed their own log handlers and
formatters, bypassing Airflow's structlog ProcessorFormatter entirely. This
meant:

- Gunicorn set up its own StreamHandler on gunicorn.error / gunicorn.access,
  so those records never went through structlog.
- Uvicorn workers called logging.config.dictConfig(LOGGING_CONFIG) on startup,
  overwriting the structlog configuration that was applied before gunicorn
  started.
- HTTP access log lines came from uvicorn's built-in access logger
  (unstructured).
- Python warnings.warn() calls and unhandled exceptions bypassed structlog
  too.

Now:

- AirflowGunicornLogger overrides setup() to skip installing gunicorn's own
  handlers and lets records propagate to root (where structlog is configured).
- AirflowUvicornWorker sets log_config=None so uvicorn doesn't clobber the
  logging config, and access_log=False since we handle that ourselves.
- HttpAccessLogMiddleware replaces uvicorn's access log: one structured event
  per request with method, path, status code, duration (µs), client address,
  and the x-request-id header bound to the structlog context for the request
  lifetime. Health-check paths are excluded to avoid noise.
- configure_logging() now explicitly silences uvicorn.access / gunicorn.access
  and routes uvicorn.error / gunicorn.error through the default handler.
- Python warnings are intercepted and emitted as structured py.warnings log
  events instead of going to stderr.
- In JSON log mode, unhandled exceptions are emitted via structlog rather than
  the plain-text default sys.excepthook.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants