Revert #5799 - #5837
Merged
Merged
Conversation
janniklasrose
enabled auto-merge
July 6, 2026 09:56
janniklasrose
temporarily deployed
to
test-trigger-is
July 6, 2026 09:56 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
July 6, 2026 09:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Collaborator
Integration test reportCommit: a63ca11
7 interesting tests: 4 SKIP, 2 flaky, 1 RECOVERED
Top 5 slowest tests (at least 2 minutes):
|
andersrexdb
approved these changes
Jul 6, 2026
tanishgupta-db
pushed a commit
to tanishgupta-db/cli
that referenced
this pull request
Jul 21, 2026
…thon-files flag (databricks#5892) ## Changes Re-applies databricks#5799 (reverted in databricks#5837) and puts the behaviour behind a new opt-in flag. `bundle generate job` only downloaded notebook tasks; files referenced by `spark_python_task` were left as absolute `/Workspace/...` paths, so the source file was never downloaded and the config wasn't portable. This PR restores the download+rewrite path (reusing the same `markFileForDownload` helper as pipeline libraries) but gates it behind a new `--download-spark-python-files` flag on `bundle generate job`, defaulting to **off**. Git-sourced files (`source: GIT`) and cloud URIs (`dbfs:/`, `s3:/`, `adls:/`, `gcs:/`) are left untouched. The flag is threaded through the `Downloader` as a functional option (`WithSparkPythonFiles`), so the pipeline, app, and import callers are unaffected. ## Why databricks#5799 was reverted in databricks#5837 because a Python file often imports sibling files that the downloader does not capture, so downloading only the entry point can produce a job that fails at runtime with missing imports — whereas leaving the absolute workspace path alone "just works". Making the download opt-in keeps the default safe while letting users who know their `spark_python_task` is self-contained pull the file into their bundle. ## Commits 1. **Re-apply databricks#5799** — a single re-revert of databricks#5837, restoring the original change verbatim (only the `NEXT_CHANGELOG.md` conflict resolved). 2. **Gate behind `--download-spark-python-files`** — the new flag, functional-option plumbing, tests, and regenerated help/changelog. ## Tests - Unit tests in `bundle/generate/downloader_test.go`: the download+rewrite path (with the option), the skipped cases (cloud URI, `source: GIT`), and a new test asserting the default-off behaviour makes no requests. - Acceptance test `acceptance/bundle/generate/spark_python_task_job` exercising the full CLI with the flag: a workspace-file task is downloaded and rewritten, a `dbfs:/` cloud-URI task is preserved. Identical output on both `terraform` and `direct` engines. - Regenerated the `bundle generate job --help` golden file. This pull request and its description were written by Isaac, an AI coding agent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Revert "Download spark_python_task workspace files in bundle generate job (#5799)"
This reverts commit b2e7e6f.
Conflicts:
NEXT_CHANGELOG.md
Why
Python files/scripts often include other files, whereas notebooks are more commonly self-contained.
With the previous PR, jobs might randomly fail due to missing imports whereas before they would just work.