Skip to content

Add optional DDL to SQL pipeline step #2460

Description

@dgomez04

The Oracle profiler PR (#2187) introduced a pattern where every logical extractor is split into two YAML steps that share the same name: — one with type: ddl, one with type: sql. This works today because pipeline.py dispatches on step.type first, but it causes three problems:

  1. Ambiguous failure reporting. PipelineClass.execute() aggregates StepExecutionResult by step.name, so when the SQL half of config_containers fails, users see Pipeline execution failed due to errors in steps: config_containers with no signal whether DDL or SQL was the culprit.
  2. Implicit name=table contract. _execute_ddl_step checks if not self._table_exists(conn, step.name): conn.execute(ddl), so the DDL must CREATE TABLE whose name is exactly step.name. Nothing enforces this; an edit that changes the name in one place but not the other silently no-ops or fails at insert time.
  3. Future fragility. Anyone refactoring pipeline.py to dedupe steps by name (perfectly reasonable assumption) breaks the YAML silently.

Scope / acceptance criteria

References

Activity

  1. added
    tech debtdesign flaws and other cascading effects
    feat/profilerIssues related to profilers
    on May 19, 2026
  2. changed the title [-]Decide pipeline step semantics: target_table field, suffix convention, or fold DDL into SQL step[/-] [+]fold DDL into SQL step[/+] on May 22, 2026
  3. changed the title [-]fold DDL into SQL step[/-] [+]Add optional DDL to SQL pipeline step[/+] on May 22, 2026
  4. dgomez04 commented on May 25, 2026

    @dgomez04
    ContributorAuthor

    We now also need to consider source_ddl that was added within #2304.

  5. dgomez04 commented on May 26, 2026

    @dgomez04
    ContributorAuthor

    We now also need to consider #2482's changes as part of this refactor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

feat/profilerIssues related to profilerstech debtdesign flaws and other cascading effects

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions