Add requirePersistence option to worker logGroomerSidecar

[ericxiao251]

Context

Currently, the worker log groomer sidecar only renders when workers.persistence.enabled is true (i.e., StatefulSet mode). This prevents using the log groomer on Deployments with emptyDir volumes for logs.

This change adds a new requirePersistence option that defaults to true, preserving full backwards compatibility. When set to false, the log groomer can be enabled regardless of whether persistence is enabled.

This is useful for deployments that:

• Use emptyDir for local logs with remote logging (S3, GCS, etc.)
• Want to clean up local log files to prevent ephemeral storage exhaustion
• Prefer Deployments over StatefulSets for workers

Configuration

workers:
  logGroomerSidecar:
    enabled: true
    requirePersistence: false  # Allow log groomer on Deployments

Backward Compatibility

This change is fully backwards compatible. The new requirePersistence option defaults to true, which preserves the existing behavior where log groomer only renders when persistence is enabled.

persistence	enabled	requirePersistence	Log groomer renders
true	true	true (default)	✓
true	false	true (default)	✗
false	true	true (default)	✗ (existing behavior)
false	true	false	✓ (new opt-in behavior)

---

Was generative AI tooling used to co-author this PR?

• Yes - Claude (Anthropic)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[jscheffl]

I actually favor #65852 over this alternative here.

[potiuk]

NICE!

[potiuk]

I actually favor #65852 over this alternative here.

But this one can be done to v1-2 branch for compatiblity ?.. @ericxiao251 -> maybe you can change the base to chart/v1-2x-test if @jscheffl will be ok with it ?

[ericxiao251]

I actually favor #65852 over this alternative here.

But this one can be done to v1-2 branch for compatiblity ?.. @ericxiao251 -> maybe you can change the base to chart/v1-2x-test if @jscheffl will be ok with it ?

Awesome, will wait for his response before changing the base branch :).

[jscheffl]

I actually favor #65852 over this alternative here.

But this one can be done to v1-2 branch for compatiblity ?.. @ericxiao251 -> maybe you can change the base to chart/v1-2x-test if @jscheffl will be ok with it ?

Awesome, will wait for his response before changing the base branch :).

I'd be okay, but we then need to add a small marker/note to the docs of the (new) parameter that this is only transitional for the chart v1.2x and will be fixed generally in chart 2.x

[ericxiao251]

I actually favor #65852 over this alternative here.

But this one can be done to v1-2 branch for compatiblity ?.. @ericxiao251 -> maybe you can change the base to chart/v1-2x-test if @jscheffl will be ok with it ?

Awesome, will wait for his response before changing the base branch :).

I'd be okay, but we then need to add a small marker/note to the docs of the (new) parameter that this is only transitional for the chart v1.2x and will be fixed generally in chart 2.x

Ok I've added the notes, fixed the CI that were failing in main and also rebased the PR/commits onto the chart/v1-2x-test branch.

[jscheffl]

Good for me. Also OK for others?

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[dabla]

Nice work @ericxiao251 we actually needed this thx!