Proposal: Automatically terminate idle Kubernetes Computing Units

[yrenat]

Background

When running Texera on Kubernetes, computing units (CUs) remain alive even after they become idle. Although keeping idle CUs can reduce startup latency for subsequent workflow executions, they continue consuming cluster resources unnecessarily when they remain unused for an extended period.

As a result, clusters may gradually accumulate idle computing units, reducing the amount of resources available for active workloads.

Proposal

Issue #5362 proposes introducing an automatic cleanup mechanism for idle computing units.

The basic idea is to periodically check the status of running CUs and terminate those that have been idle for longer than a configurable timeout. This allows inactive resources to be reclaimed automatically while still giving users the benefit of reusing recently active computing units.

The timeout should be configurable so that different deployments can choose an appropriate balance between resource utilization and startup latency.

Initial Scope

For this proposal, the implementation focuses on:

• adding a configurable idle timeout;
• periodically checking whether a computing unit has been idle beyond the configured threshold;
• automatically terminating eligible idle computing units;
• ensuring active computing units are unaffected.

This proposal intentionally focuses on basic idle cleanup. More advanced policies (for example, different timeout strategies or discussions based on frontend heartbeat signals) can be discussed separately if needed.

Questions for Discussion

• Does periodically terminating idle computing units align with the intended Kubernetes resource management strategy?
• Is a single configurable timeout sufficient, or should different types of computing units have different timeout policies?
• Are there scenarios where idle computing units should intentionally remain alive and therefore be excluded from automatic cleanup?

I have prepared an implementation of this proposal for reference in PR #6046.

[chenlica]

@yrenat Thanks for creating the discussion. Before we dive into the details, I wonder whether this problem is common in Kubernetes. If so, are there any existing solutions to this problem? Please do some research and report your findings.

[yrenat]

@chenlica
Based on my findings, this resource-waste problem is common. There are several generic solutions on "system" level, where they have a global control of what/how many resources are needed, but they can seldom address our specific problem where we need to control a specific pod/CU. The following are some of my findings, regarding garbage resource collection.

1. HPA / custom metrics
   This is the native idea from Kubernetes. It automatically changes the replica count of a scalable Kubernetes workload, such as a Deployment, ReplicaSet, ReplicationController, or StatefulSet, based on metrics like CPU, memory, or custom/external metrics. In simpler words, it can automatically scale up/down the number of pods/resources invovled. However, a Texera CU is a pod-specific per-user application resource with a DB record, access control, resource settings, and a corresponding runtime Pod (not just a pod). Also, “idle” is another Texera-specific, idea, which cannot be natively well-described by Kubernetes. Besides, HPA cannot fully shutdown the system. It has to have at least one pod running to keep track of all the computation resources.
2. KEDA
   KEDA (Kubernetes Event-driven Autoscaling) is designed for event-driven autoscaling and can scale workloads down to zero when event sources become inactive. This is a good solution for queue/event-driven services. However, using KEDA directly would require Texera to expose CU activity as an external/custom metric or event source, which will create multi layers of interference with Texera. It also would not automatically update Texera’s CU database state unless Texera adds extra integration.
3. Knative Serving
   Knative supports scale-to-zero for HTTP request/traffic-driven services. This is useful for serverless HTTP workloads, but a lot of Texera resources/commands are not handled in HTTP format. Adopting Knative would be a larger architectural change and still may not capture Texera’s definition of CU idleness. Besides, Knative also does not focus on pod/CU level logistics.

To sum up, for the current issue, I think the most suitable first solution is to implement our tailored idle detection inside Texera’s Computing Unit Managing Service or a Texera-side scheduled cleanup component. The cleanup should use Texera’s own CU termination logic rather than directly deleting Pods, so that both the Kubernetes Pod and the Texera DB state remain consistent.

[kunwp1]

Hi @yrenat We need this feature and thank you for working on this task! I did some research before and concluded that Kubernetes does not natively support this feature. I agree we need to have our own way mechanism to terminate idle CUs.

What's missing in both this discussion post and your PR is how you defined what "idle" CU means and the architectural design of how computing unit manager monitors all the CUs. Can you be more specific on those parts? Please include some visual diagrams if you can. We need to agree on the design before reviewing the PR.

[chenlica]

This feature is related to another related task. @mengw15 and @bobbai00 : can you chime in?

[mengw15]

Yes, the result retrieval path is currently tied to the original live CU (results are fetched through the workflow websocket handled inside the CU), so once the CU is gone the UI can't reach them even if the data is still in storage. I documented this in #5363, we haven't planned to work on it yet. I agree with @kunwp1 that a separate CU-independent retrieval path would be a fairly large change.

@kunwp1 On the retention idea — I remember we already have two related mechanisms: (1) a result-retention parameter result-cleanup.ttl-in-seconds (env RESULT_CLEANUP_TTL_IN_SECONDS, default 24h) with a periodic cleanup in ComputingUnitMaster; (2) a ~30s cleanup (web-server.workflow-state-cleanup-in-seconds) that clears the latest execution's results once the last user disconnects and the workflow is no longer running. Since you have more context on this part, does that match your understanding?

[kunwp1]

Yes. The definition of "idle" I am proposing is "no execution in the CU for a certain amount of time && no workflow result in the CU". Wonder whether this condition makes sense.

[yrenat]

@chenlica @kunwp1 Please correct me if I am wrong but I think the qualified "idle" CUs in your definition would be

1. those that have never been executed, and
2. those whose results have been automatically cleaned

I have already covered case 1 in the PR. And for case 2, I think I should then focus on using the internal time intervals, rather than using a set of new env variables.

[kunwp1]

Aligns with my suggestion.

[chenlica]

@yrenat Thanks for reporting the findings. I am wondering: do other systems have something similar to "CU"? If not, what do they use? If the "Computing Unit (CU)" concept in Texera unique, we want to know its pros and cons compared to other systems.

[yrenat]

@chenlica I have done some researches and my current understanding is that other systems have related concepts, but not exactly the same idea.

In Texera, a CU is kept with information such as owner uid, cuid, name, type, URI, resource limits, and creation/termination state. Workflow executions reference a CU through workflow_executions.cuid.

A CU also has access control and sharing through computing_unit_user_access. It is selected by the user from the frontend, and the access-control service routes requests to the CU's recorded URI.

Therefore, the CU combines several concepts:

• A user-facing execution environment
• A resource profile
• A runtime endpoint
• ACL and shareability
• Workflow-execution affinity
• Lifecycle state

Other systems usually split these responsibilities across lower-level concepts:

System	Differences
Apache Spark	Spark applications run as a driver plus executors acquired from a cluster manager such as Standalone, YARN, or Kubernetes. Each application gets its own executors, but these are not usually reusable, user-selected, shareable workspace-like resources. See Spark Cluster Overview.
Apache Flink	Flink has Application Mode and Session Mode. Application Mode creates a cluster per submitted application, while Session Mode shares one cluster across multiple applications or jobs. This is close in terms of resource isolation, but users submit jobs to a cluster or session rather than selecting a persistent Texera-style CU object. See Flink Deployment Overview.
Ray	Ray clusters consist of worker nodes connected to a head node and may autoscale based on application resource requests. The main resource is the Ray cluster rather than a Texera-style per-user CU with DB identity, ACL, workflow binding, and gateway routing. See Ray Cluster Overview.
Dask	dask.distributed uses a central scheduler coordinating workers and clients. Users connect a Python session to the scheduler and submit work. This is similar to an interactive distributed execution environment, but the abstraction is scheduler/worker/client rather than a first-class shareable workflow execution unit. See Dask Distributed.
JupyterHub	I think this shares the most similarity with our Texera platform. JupyterHub authenticates users, spawns a single-user server, configures proxy routing, and can cull idle servers. However, it is notebook/session-oriented rather than workflow-engine-oriented. See JupyterHub Concepts.

So I would say that technically Texera's CU is not completely unique. However, it does sound unique because it is integrated directly into the workflow system and treated as a user-visible, shareable, routable, and persistent resource.

Pros:

1. Users can reuse a warm execution environment, reducing startup latency.
2. Resource limits and runtime settings are attached to a named, user-facing object.
3. Texera can define idleness using workflow semantics rather than only CPU or memory metrics (like the PR that I am working on: fix(kubernetes): terminate idle computing units #6046).
4. CU lifecycle management can keep DB state, pod state, access control, and gateway routing consistent.
5. Sharing and permissions can be handled at the Texera resource level.

Cons:

1. Generic Kubernetes autoscaling or TTL mechanisms do not understand CU semantics. Which means that we have to build things on top of Kubernetes, increasing latency and redundancy.
2. Texera has to maintain lifecycle reconciliation logic between DB records and pods.
3. Idle cleanup is product-specific and must carefully handle workflow results and users who leave and return later. Again, that PR fix(kubernetes): terminate idle computing units #6046
4. The abstraction adds complexity because UI selection, workflow execution, access control, routing, and pod management all depend on cuid.

Therefore, for this idle-cleanup feature, I think cleanup should happen through Texera's CU manager rather than directly through Kubernetes. Only Texera knows whether a CU is idle according to workflow state and can update both the runtime resource and Texera's DB state consistently (see the first Con listed above).

[parshimers]

I think this is a natural idea, you don't want to allocate more resources in the cluster than you need. I think the concept in k8s that would let you handle this best would be an Operator (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). Then, Texera itself could through its operator tell Kubernetes it no longer needs some set of pods. I assume Texera currently doesn't have an operator? I took a cursory look and didn't find an implementation of one.

[chenlica]

@yrenat Thanks for the nice summary. Currently a Texera CU is implemented as a Kubernetes pod. For the long term we do want to keep the abstraction of a CU, and choose other implementations such as a cluster of nodes to run the parallel Amber engine.

For other systems you have investigated, did you find any of them using a K8S pod as a basic computing unit?

[yrenat]

@chenlica
Ray: According to the Ray Kubernetes documentation, a Ray node is implemented as a Kubernetes pod: the head node runs in one pod, and each worker node also runs in a pod. The Ray autoscaler scales the cluster by adding or removing worker pods.

Dask: Dask is also similar at the implementation level. A Dask cluster contains a scheduler pod and a set of worker pods.

Spark/Flink: Spark and Flink also map their runtime components to Kubernetes pods. Spark uses driver and executor pods, while Flink uses JobManager and TaskManager pods.

From my understanding, I think the main difference from Texera is that Texera exposes the CU (which sits on top of a k8s pod) as a user-controllable and reusable resource with its own ownership and access control, resource configuration, lifecycle state, runtime endpoint, and workflow-execution association.However, Ray and Dask generally expose the cluster as the higher-level abstraction, while individual pods represent nodes or workers inside the cluster. Normal users do not directly manage or select these pods.

Personally, based on my recent research, I also feel that our current CU abstraction is sort of "bulky". A CU currently combines several responsibilities, including access control, resource configuration, runtime location, lifecycle management, and workflow association. My concern is that mixing all of these responsibilities directly to the CU may make CU creation, removal, and future on-the-go changes more complicated.

I am also thinking about whether we could separate the logical CU abstraction from some of these lower-level details. For example, a CU could reference or be linked to separate resource, runtime, and access-control information, rather than directly owning these attributes.

[chenlica]

I support the idea of separating a "logical CU" from a "physical CU." In this discussion of the lifecycle of a "CU," it refers to a physical CU, which is currently a k8s pod.

I wonder for other systems that also use a k8s pod:

1. Do they have the need to kill an idle k8s pod?
2. If so, how do they kill such a pod?
3. How does the termination of a pod affect the "logical CU"?