experimental/ssh: resolve bare python/pip to the environment interpreter - #5888
Merged
Conversation
In an interactive `databricks ssh connect` session, bare `python`/`pip` resolved to the system or cluster-libraries interpreter rather than the environment interpreter at `$DATABRICKS_VIRTUAL_ENV`, so packages installed in the environment could not be imported without extra setup. Two things clobbered the environment-first PATH that sshd forwards via SetEnv: a login shell (`bash -l`) re-sourced /etc/profile, rebuilding PATH from scratch; and any interactive bash sources /etc/bash.bashrc, which on serverless runs activate_root_python_environment.sh and prepends the cluster-libraries python ahead of the environment interpreter. Fix both layers: - Client: launch an interactive non-login shell (`bash -i`) so /etc/profile is no longer re-sourced. - Server: seed a guarded, idempotent `~/.bashrc` snippet that re-prepends the environment interpreter's bin directory. ~/.bashrc is sourced after /etc/bash.bashrc, so it wins; /etc/bash.bashrc and /etc/profile.d are root-owned and not writable by the non-root serverless user. The seed is best-effort: a write failure logs a warning and does not abort the tunnel. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: a00426c
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 10 slowest tests (at least 2 minutes):
|
…ompute Add experimental/ssh/CLAUDE.md capturing the verification discipline for `ssh connect` changes: confirm the built binary matches HEAD, build release archives with `./task snapshot-release` (not `./task build`) for --releases-dir, and delete the versioned workspace directory before re-verifying a rebuilt dev binary (same-version dev uploads are otherwise skipped). Also notes that the server and login session share a user/$HOME on serverless, so a per-compute symptom is usually a stale upload rather than a real difference. Co-authored-by: Isaac
The test set $HOME, but env.UserHomeDir reads $USERPROFILE on Windows, so seedEnvActivation targeted the real profile and the temp-dir assertions failed. Set env.HomeEnvVar() instead so the test uses the same home the production code resolves, on every OS. Co-authored-by: Isaac
anton-107
temporarily deployed
to
test-trigger-is
July 10, 2026 11:01 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
anton-107
temporarily deployed
to
test-trigger-is
July 10, 2026 11:01 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Co-authored-by: Isaac
anton-107
temporarily deployed
to
test-trigger-is
July 10, 2026 11:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
anton-107
temporarily deployed
to
test-trigger-is
July 10, 2026 11:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
rugpanov
approved these changes
Jul 10, 2026
rugpanov
left a comment
rugpanov
left a comment
Contributor
There was a problem hiding this comment.
Approving. Reviewed with Isaac Review + codex — no actionable findings from either. The bare python/pip → environment-interpreter resolution is focused and covered by the new server/client tests for the bashrc-seeding behavior.
Collaborator
Integration test reportCommit: d6636f4
25 interesting tests: 11 FAIL, 10 flaky, 2 SKIP, 1 KNOWN, 1 RECOVERED
Top 50 slowest tests (at least 2 minutes):
|
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Jul 15, 2026
## Release v1.8.0 ### Notable Changes * Auto-migrate a bundle from terraform to the direct engine when `bundle.engine` is `"direct"` (or `DATABRICKS_BUNDLE_ENGINE=direct`) and the post-deploy dry-run migration is clean; a warning is emitted if the dry-run surfaces errors or warnings so the automatic migration is skipped. ### CLI * experimental `ssh connect`: bare `python`/`pip` in an interactive session now resolve to the environment interpreter (`$DATABRICKS_VIRTUAL_ENV`) instead of the system or cluster-libraries interpreter, so packages installed in the environment are importable without extra setup. The interactive shell is now non-login (`bash -i`) and the server seeds a `~/.bashrc` snippet that re-prepends the environment's bin directory to `PATH` ([#5888](#5888)). * When Claude Code runs the CLI without the Databricks AI tooling installed, the CLI now prints a one-line recommendation on stderr to run `databricks aitools install`. The recommendation is shown at most once per hour per Claude session, and never for human callers or `aitools` commands. * Fixed `databricks auth describe` misattributing a profile selected via `DATABRICKS_CONFIG_PROFILE` as `(from bundle)` when run inside a bundle root ([#5904](#5904)). ### Bundles * `bundle generate` now warns when the generated configuration file is not matched by any pattern in the `include` section of `databricks.yml` ([#5868](#5868)). * direct: Match UC Auto Upgrade managed property defaults with a wildcard pattern instead of enumerating each key ([#5877](#5877)). * Recognize `ssh://` template URLs in `databricks bundle init` ([#5891](#5891)). * `databricks bundle init` now reports an actionable error when given a template URL with an unsupported protocol (`http://`, `git://`, `ftp://`, `ftps://`) instead of failing with a confusing "not a bundle template" message ([#5902](#5902)). * Added an `env:` section to `scripts.<name>` for declaring environment variables that may reference `${bundle.*}`, `${workspace.*}`, and `${var.*}` ([#4179](#4179), [#5299](#5299)).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In an interactive
databricks ssh connectsession, barepython/pipresolved to the wrong interpreter (system or cluster-libraries python) instead of the environment interpreter at$DATABRICKS_VIRTUAL_ENV, so packages installed in the environment (including with--base-environment) could not be imported without extra setup.import <env-package>failed withModuleNotFoundError. (DECO-27499)Root cause (two layers)
sshd forwards a snapshot of the environment-first
PATHinto each session via aSetEnvline, but two things clobbered it in an interactive shell:exec bash -l;-lre-sources/etc/profile, which rebuildsPATHfrom scratch, dropping the environment bin dir → barepython= system python.-ior-l) sources/etc/bash.bashrc, which on serverless runsactivate_root_python_environment.shand prepends.../cluster_libraries/python/binahead of the environment interpreter → barepython= cluster-libraries python.The non-interactive
-- <cmd>path was already correct (sshd runs it via$SHELL -c, sourcing neither file).Fix (two coordinated changes)
client.go): launch an interactive non-login shell (bash -i) so/etc/profileis no longer re-sourced.server.go): seed a guarded, idempotent~/.bashrcsnippet that re-prepends the environment interpreter's bin directory.~/.bashrcis sourced after/etc/bash.bashrc, so it wins;/etc/bash.bashrcand/etc/profile.dare root-owned and not writable by the non-root serverless user, so~/.bashrcis the only usable hook. The seed is best-effort — a write failure logs a warning and does not abort the tunnel (mirrors the existing/run/sshdhandling).Why server-side and not client-side
Each
--namegets a fresh ephemeral$HOME, so a client-side edit wouldn't persist. A pure client-side re-prepend viabash --rcfile <(...)was rejected because the remote command is parsed by the login shell (/bin/sh/dash on some images), where process substitution is a syntax error and would breakconnect.Scope
Applied on all compute, guarded by
$DATABRICKS_VIRTUAL_ENVbeing set. The bootstrap sets that variable tosys.executable(an absolute path) on every compute type, sodirnamealways yields the correct interpreter's bin directory.Testing
go test ./experimental/ssh/...green, including newTestSeedEnvActivation(create / append / trailing-newline / idempotent-preserves-content / guard-present).bash -iassertions updated inclient_internal_test.go.Manually verified against real compute
Confirmed end-to-end across all three compute types — in each interactive session,
which pythonresolved to the environment interpreter (dirname($DATABRICKS_VIRTUAL_ENV)) rather than the system or cluster-libraries python, and importing an environment-only package succeeded:--base-environment(cowsay);import cowsayworked.GPU_1xA10) — with the same custom--base-environment;import cowsayworked.which pythonresolved to the environment interpreter.This pull request and its description were written by Isaac.