Add option to have structured JSON logging for _all_ API server output#63365
Merged
ashb merged 3 commits intoMar 12, 2026
Merged
Conversation
vincbeck
approved these changes
Mar 11, 2026
vincbeck
left a comment
Contributor
There was a problem hiding this comment.
Not an expert in some areas but overall looks good! Looks like a nice improvement!
kaxil
reviewed
Mar 11, 2026
This comment was marked as resolved.
This comment was marked as resolved.
3dd1e99 to
5349a02
Compare
Previously gunicorn/uvicorn each installed their own log handlers and formatters, bypassing Airflow's structlog ProcessorFormatter entirely. This meant: - Gunicorn set up its own StreamHandler on gunicorn.error / gunicorn.access, so those records never went through structlog. - Uvicorn workers called logging.config.dictConfig(LOGGING_CONFIG) on startup, overwriting the structlog configuration that was applied before gunicorn started. - HTTP access log lines came from uvicorn's built-in access logger (unstructured). - Python warnings.warn() calls and unhandled exceptions bypassed structlog too. Now: - AirflowGunicornLogger overrides setup() to skip installing gunicorn's own handlers and lets records propagate to root (where structlog is configured). - AirflowUvicornWorker sets log_config=None so uvicorn doesn't clobber the logging config, and access_log=False since we handle that ourselves. - HttpAccessLogMiddleware replaces uvicorn's access log: one structured event per request with method, path, status code, duration (µs), client address, and the x-request-id header bound to the structlog context for the request lifetime. Health-check paths are excluded to avoid noise. - configure_logging() now explicitly silences uvicorn.access / gunicorn.access and routes uvicorn.error / gunicorn.error through the default handler. - Python warnings are intercepted and emitted as structured py.warnings log events instead of going to stderr. - In JSON log mode, unhandled exceptions are emitted via structlog rather than the plain-text default sys.excepthook.
5349a02 to
dbdd825
Compare
kaxil
reviewed
Mar 12, 2026
kaxil
reviewed
Mar 12, 2026
kaxil
reviewed
Mar 12, 2026
kaxil
reviewed
Mar 12, 2026
kaxil
approved these changes
Mar 12, 2026
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
ashb
commented
Mar 12, 2026
…le_auth_manager.py
Pyasma
pushed a commit
to Pyasma/airflow
that referenced
this pull request
Mar 13, 2026
apache#63365) Previously gunicorn/uvicorn each installed their own log handlers and formatters, bypassing Airflow's structlog ProcessorFormatter entirely. This meant: - Gunicorn set up its own StreamHandler on gunicorn.error / gunicorn.access, so those records never went through structlog. - Uvicorn workers called logging.config.dictConfig(LOGGING_CONFIG) on startup, overwriting the structlog configuration that was applied before gunicorn started. - HTTP access log lines came from uvicorn's built-in access logger (unstructured). - Python warnings.warn() calls and unhandled exceptions bypassed structlog too. Now: - AirflowGunicornLogger overrides setup() to skip installing gunicorn's own handlers and lets records propagate to root (where structlog is configured). - AirflowUvicornWorker sets log_config=None so uvicorn doesn't clobber the logging config, and access_log=False since we handle that ourselves. - HttpAccessLogMiddleware replaces uvicorn's access log: one structured event per request with method, path, status code, duration (µs), client address, and the x-request-id header bound to the structlog context for the request lifetime. Health-check paths are excluded to avoid noise. - configure_logging() now explicitly silences uvicorn.access / gunicorn.access and routes uvicorn.error / gunicorn.error through the default handler. - Python warnings are intercepted and emitted as structured py.warnings log events instead of going to stderr. - In JSON log mode, unhandled exceptions are emitted via structlog rather than the plain-text default sys.excepthook.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously gunicorn/uvicorn each installed their own log handlers and
formatters, bypassing Airflow's structlog ProcessorFormatter entirely. This
meant:
so those records never went through structlog.
overwriting the structlog configuration that was applied before gunicorn
started.
(unstructured).
too.
stderr
Now:
handlers and lets records propagate to root (where structlog is configured).
logging config, and access_log=False since we handle that ourselves.
per request with method, path, status code, duration (µs), client address,
and the x-request-id header bound to the structlog context for the request
lifetime. Health-check paths are excluded to avoid noise.
and routes uvicorn.error / gunicorn.error through the default handler.
events instead of going to stderr.
the plain-text default sys.excepthook.
Pics of what it looks like
(non json version)
(json version)
