Add optional OTel service to the Airflow Helm Chart by xBis7 · Pull Request #64902 · apache/airflow

xBis7 · 2026-04-08T12:05:33Z

This patch is adding an otel-collector to the Helm chart.

I've added 2 separate flags for enabling traces and metrics. OTel is the only supported backend for traces, and so the traces flag is enabled by default. But that's not the case with metrics, and they need to be manually enabled. When the user enables the otel metrics, statsd is disabled in the airflow config so that otel will be used instead.

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 4.6 Extended

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

jason810496

I didn’t look into this thoroughly, but there might be some concerns based on a high-level overview.

Just FYI that we had a discussion about whether to use Kustomize for this kind of optional feature for better long-term maintainability in https://apache-airflow.slack.com/archives/C027H098M1C/p1770794021001679.

Though we haven’t settled on the release process and the concrete structure if we go with the Kustomize approach.

xBis7 · 2026-04-08T13:36:47Z

@jason810496 Thank you, I wasn't aware of this.

Airflow needs to talk directly to the otel-collector and to be configured to work with it. Additionally, when OTel is enabled for metrics, we have to disable statsd. Based on that, it's hard to set it up with Kustomize and it should be part of the helm chart.

But the 3 observability backends, don't need to interact with Airflow. They are very good example candidates for Kustomize. I think I should be able to make it work.

As I understand from the Slack discussion, there was a consensus on using Kustomize from 1.19.0 and upwards.

After this PR, I would like to add integration tests that use OTel and the backends. I don't think setting them up via Kustomize will be a problem.

I can move forward with the changes.

jscheffl

I do not think it is a good idea adding more components to the Airflow chart. There are better and known charts for Prometheus/Grafana and such. You should rather reference them instead of adding more complexity to ours

xBis7 · 2026-04-08T19:13:20Z

@jscheffl What about the Kustomize approach that @jason810496 suggested? I just pushed it.

Miretpl

I only checked the Helm chart-related part. I would recommend splitting it from this PR, too.

Additionally, not in the comments, but the whole helm chart part has no tests addition, when it should have.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds OpenTelemetry-driven observability (collector + optional backends) to the dev/Kubernetes workflow and Helm chart configuration, with flags to enable traces and/or metrics.

Changes:

Extend Helm chart values/schema and templates to support an optional OpenTelemetry Collector and Airflow OTel configuration.
Add CI/dev Kubernetes manifests for Jaeger/Prometheus/Grafana and expose them via NodePorts in kind.
Update Breeze and test utilities to manage additional forwarded ports and to deploy observability backends based on --set flags.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
scripts/ci/prek/lint_json_schema.py	Update YAML loading/validation to support multi-document YAML.
scripts/ci/kubernetes/observability/kustomization.yaml	Add kustomize entrypoint for observability manifests.
scripts/ci/kubernetes/observability/jaeger.yaml	Add Jaeger all-in-one Deployment/Service for dev/CI.
scripts/ci/kubernetes/observability/prometheus.yaml	Add Prometheus config + Deployment/Service for dev/CI scraping.
scripts/ci/kubernetes/observability/grafana.yaml	Add Grafana provisioning + Deployment/Service for dev/CI dashboards.
scripts/ci/kubernetes/nodeport.yaml	Expose Jaeger/Prometheus/Grafana via NodePorts.
scripts/ci/kubernetes/kind-cluster-conf.yaml	Map additional NodePorts to localhost via kind port mappings.
kubernetes-tests/tests/kubernetes_tests/test_base.py	Switch API server port env var name used by k8s tests.
dev/breeze/tests/test_kubernetes_commands.py	Add unit tests for parsing OTel `--set` flags.
dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py	Allocate/propagate new forwarded ports and print backend URLs.
dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py	Apply observability manifests after deploy based on parsed OTel flags.
chart/values.yaml	Add OTel collector values and wire OTel vs statsd settings in airflow.cfg.
chart/values.schema.json	Extend JSON schema for OTel collector image and ports.
chart/templates/otel-collector/otel-collector-service.yaml	Add Service for the optional OTel collector.
chart/templates/otel-collector/otel-collector-deployment.yaml	Add Deployment for the optional OTel collector.
chart/templates/configmaps/otel-collector-configmap.yaml	Add OTel collector config (receivers/exporters/pipelines).
chart/templates/_helpers.yaml	Add OTel env vars and helper for OTel collector image string.

xBis7 · 2026-04-14T19:37:12Z

@Miretpl Thank you for the review!

I removed all the kustomize logic and the jaeger, grafana and prometheus yaml files. We can see how to optionally include these in a follow-up PR.

I'm going to address your comments and also add tests.

…gument

xBis7 · 2026-05-05T10:40:28Z

Green CI.

Will probably need some effort to back-port to 1.2x line after.

@jscheffl I can look into the backport.

github-actions · 2026-05-05T19:00:12Z

Backport failed to create: chart/v1-2x-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	chart/v1-2x-test

You can attempt to backport this manually by running:

cherry_picker 535e3cc chart/v1-2x-test

This should apply the commit to the chart/v1-2x-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

xBis7 · 2026-05-05T19:05:41Z

@jscheffl @Miretpl Thank you for the help! I'm going to backport it manually and create a PR against the v1-2 branch.

Miretpl · 2026-05-05T19:48:54Z

@xBis7 no problem. Feel free to mention me when the backport is ready

* add otel to helm chart * use Kustomize for grafana, jaeger, prometheus * enable specific service per flag + unit test * remove grafana, jaeger and prometheus kustomization logic * traces enabled and metrics disabled, by default * remove otelCollector.enabled flag * add statsd comments about otel metrics overriding the config * make OTEL_METRIC_EXPORT_INTERVAL configurable and provide default value + entry in the values.schema.json * remove hardcoded value for metrics otel_port in values.yaml * add option to override the configmap * add otelCollector.args and make the config.yml file as the default argument * rename extraAnnotations to annotations in otel-collector-service.yaml * parameterize the readiness and liveness probe values * remove prometheus from the configmap * update the default value for OTEL_TRACES_EXPORTER * fix tests in airflow_aux + otel-collector-serviceaccount.yaml * fix spellcheck errors in docs * fix tests in security * otel collector unit tests + networkpolicy file * values.schema.json cleanup * add a minimum to all integer configs in values.schema.json * fix heading comments * change config default to ~ from empty string * fix static check error

boring-cyborg Bot added area:dev-tools area:helm-chart Airflow Helm Chart labels Apr 8, 2026

xBis7 requested a review from potiuk as a code owner April 8, 2026 12:05

boring-cyborg Bot added area:kubernetes-tests backport-to-v3-2-test labels Apr 8, 2026

xBis7 requested review from amoghrajesh, ashb, bugraoz93, choo121600, ephraimbuddy, gopidesupavan, hussein-awala, jason810496, jedcunningham, jscheffl and vatsrahul1001 as code owners April 8, 2026 12:05

jason810496 reviewed Apr 8, 2026

View reviewed changes

jscheffl requested changes Apr 8, 2026

View reviewed changes

xBis7 force-pushed the otel-helm-chart branch from 6e988f4 to ffec048 Compare April 8, 2026 19:10

xBis7 commented Apr 9, 2026

View reviewed changes

Comment thread chart/values.yaml

Comment thread dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py Outdated

Miretpl requested changes Apr 9, 2026

View reviewed changes

kaxil requested a review from Copilot April 10, 2026 19:55

Copilot AI reviewed Apr 10, 2026

View reviewed changes

xBis7 force-pushed the otel-helm-chart branch 2 times, most recently from d1da7d0 to 89f2712 Compare April 14, 2026 19:35

xBis7 changed the title ~~Add optional OTel, Jaeger, Grafana and Prometheus services to the Airflow Helm Chart~~ Add optional OTel service to the Airflow Helm Chart Apr 15, 2026

xBis7 force-pushed the otel-helm-chart branch from 89f2712 to b8d0e0a Compare April 15, 2026 14:49

xBis7 added 14 commits May 5, 2026 09:13

add otelCollector.args and make the config.yml file as the default ar…

50812ab

…gument

rename extraAnnotations to annotations in otel-collector-service.yaml

79b84aa

parameterize the readiness and liveness probe values

8b32005

remove prometheus from the configmap

45735d0

update the default value for OTEL_TRACES_EXPORTER

e570cd2

fix tests in airflow_aux + otel-collector-serviceaccount.yaml

c5b2322

fix spellcheck errors in docs

49fe21f

fix tests in security

4af0025

otel collector unit tests + networkpolicy file

c22c68b

values.schema.json cleanup

591bba5

add a minimum to all integer configs in values.schema.json

19a8f07

fix heading comments

304de3a

change config default to ~ from empty string

1a204a4

fix static check error

2ad2c1c

xBis7 force-pushed the otel-helm-chart branch from 0cd12f4 to 2ad2c1c Compare May 5, 2026 06:13

jscheffl modified the milestones: Airflow Helm Chart 2.0.0, Airflow Helm Chart 1.22.0 May 5, 2026

jscheffl added the backport-to-chart/v1-2x-test Automatic backport to chart 1.2x maintenance branch label May 5, 2026

jscheffl merged commit 535e3cc into apache:main May 5, 2026
142 checks passed

xBis7 deleted the otel-helm-chart branch May 6, 2026 10:44

Miretpl mentioned this pull request May 26, 2026

Disable otel collector by default #67575

Merged

1 task

jscheffl mentioned this pull request May 31, 2026

Helm Chart: Prepare release notes for 1.22.0 #67806

Merged

1 task

potiuk mentioned this pull request Jun 1, 2026

Status of testing of Apache Airflow Helm Chart 1.22.0rc1 #67821

Closed

29 tasks

Uh oh!

Conversation

xBis7 commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

xBis7 commented Apr 8, 2026

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

xBis7 commented Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Miretpl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xBis7 commented Apr 14, 2026

Uh oh!

xBis7 commented May 5, 2026

Uh oh!

Uh oh!

github-actions Bot commented May 5, 2026

Backport failed to create: chart/v1-2x-test. View the failure log Run details

Uh oh!

xBis7 commented May 5, 2026

Uh oh!

Miretpl commented May 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xBis7 commented Apr 8, 2026 •

edited

Loading