[Feature Request]: Allow a Workflow to Be Defined as the Required Entry Point for a Pipeline

[CarlosJuncher03]

What would you like to happen?

Some pipelines depend on mandatory preparation, validation, or control steps that must be performed beforehand by a workflow. Currently, even in these cases, the pipeline can be executed directly through the UI, CLI, API, or server, bypassing these required steps and potentially causing inconsistent executions.

The proposal is to allow a pipeline to be configured with a required entry-point workflow.

When this configuration is enabled, Apache Hop should:

Allow the pipeline to run when invoked by the configured workflow.
Prevent direct execution or execution by any unauthorized process.
Clearly indicate which workflow must be used.
Optionally provide a “Run through the configured workflow” action in the UI.

The goal is not to introduce hidden orchestration or automatically execute a sub-pipeline. The workflow will remain responsible for explicit orchestration. This feature will only ensure that pipelines with prerequisites are started through the correct entry point.

The configuration should be optional, preserving the current behavior for pipelines that do not have a required entry-point workflow configured.

Benefits
Prevents mandatory steps from being bypassed.
Establishes a single, predictable entry point.
Reduces errors caused by improper direct execution.
Keeps orchestration explicit and visible within the workflow.
Ensures consistent behavior across the UI, CLI, API, and server.

Issue Priority

Priority: 3

Issue Component

Component: Pipelines

[mattcasters]

It's not just a workflow or pipeline filename that we need to keep track of as a dependency, it's the state of execution. Then we'd need time windows to make sure it was restarted in the last X period of time, ... This will expand into madness pretty quickly.

Could we somehow express the need for the execution of a workflow or pipeline as part of the preparation in terms of the actual results of the preparation? For example, do you require certain variables/parameters to be set? Are certain files or folders required to show up?

Perhaps this will make it both easier to prevent dependency management as well as prevent execution without certain requirements being in place. What could be the manditory results of the "preparation, validation, or control steps" you mentioned?

[CarlosJuncher03]

In my case, the dependency is related to the result of the preparation rather than simply whether a specific workflow has executed.

I have a sequential data-loading process that combines data extracted from an API with reference data extracted from a database.

The API provides the entities that will be transformed and loaded into the destination, while the database extraction provides reference data used for key lookups, domain validation, and business-rule validation.

During the workflow, these datasets are extracted and stored in S3. A later pipeline uses DuckDB/SQL to compare and validate this data and decide whether each record should be inserted, updated, or deleted.

The important part is that this pipeline cannot safely run in isolation.

For example, there is an initial pipeline that generates the reference/base dataset used by the subsequent validation pipelines. If someone executes the DuckDB validation pipeline directly before that base has been generated — or using a base produced by a previous execution — the pipeline may produce incorrect insert/update/delete decisions.

So I agree that checking only whether a workflow or pipeline "has executed" could become complicated because execution state and time windows would also need to be considered.

Expressing this as required preparation results may actually be a better abstraction.

In this scenario, the requirement could be something like:

• the required reference datasets must exist;
• they must belong to the same execution/load cycle;
• they must be complete and valid;
• optionally, some execution identifier or metadata could prove that the preparation phase completed successfully.

Then the pipeline would not necessarily depend on a specific workflow filename. Instead, it would declare the conditions or artifacts that must be satisfied before execution is allowed.

The workflow would still be responsible for orchestration, but the pipeline could protect itself against being executed when its required preparation state is not available.

I think this would address the original problem while avoiding turning Hop into a dependency scheduler.

[hansva]

Sounds overly complex and would hinder you during development, causing you to just disable it.

[mattcasters]

Take a look at the work being done here. Maybe it's more in line with what you're trying to achieve: https://github.com/mattcasters/hop-data-vault/blob/main/docs/data-quality.adoc

[CarlosJuncher03]

That makes sense. I agree that hard-blocking the execution could become inconvenient during development and would probably lead people to disable the restriction.

After looking at the data quality / quality gate approach, I think perhaps the original proposal could be simplified.

Instead of preventing direct execution, a pipeline could optionally declare metadata indicating that it normally depends on another workflow or pipeline as its expected entry point or preparation step.

Then, when someone executes that pipeline directly from Hop GUI, Hop could display a warning such as:

"This pipeline normally depends on workflow X being executed first."

The user could then either continue with the direct execution or choose an action like "Run through workflow X".

This would make the dependency visible without introducing dependency-state management, execution time windows, or hard enforcement.

In production, the orchestrator would still be responsible for executing everything in the correct order. The metadata would mainly help developers and operators understand that a pipeline is normally not intended to be executed independently.

This would also preserve the ability to execute the pipeline directly when needed for development, debugging, or manual corrections.

So perhaps this is less about enforcing a required entry point and more about declaring an expected entry point/dependency and warning when it is bypassed.

[CarlosJuncher03]

One quick note: I really liked the idea @mattcasters mentioned; I’m going to explore it further in future projects. Thanks to both of you for the support.

[mattcasters]

Feedback is always welcome. Thanks for the great ideas @CarlosJuncher03