Better readability and searchability of Airflow logs

[omkar-foss]

Description

As per users' feedback in the Airflow Debugging Survey 2024:

• Enhanced readability and search features are needed
• More than 50% respondents find logs difficult to search and filter

Use case/motivation

Goals for this issue are the following:

• Review existing log messages for verbosity and readability.
• Explore addition of a fuzzy search functionality in Airflow UI so user preferably doesn't have to leave Airflow window to search for Airflow logs (in other places like ES). This will enhance dev productivity.
• Explore keeping log verbosity configurable at DAG-level. For e.g. A user may want more verbosity for a model training DAG while very less verbosity for a scheduled DAG performing basic data transformations.

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[bbovenzi]

Closing since there is no concrete feature to build.

[omkar-foss]

Closing since there is no concrete feature to build.

@bbovenzi we could explore fuzzy search in logs, similar to Elasticsearch Fuzzy Query, perhaps in Airflow 3.1+.

[Dev-iL]

Possibly more viable after switching to structlog #52651