Add server-lifecycle flags to `ssh setup` (--max-clients, --server-timeout) by anton-107 · Pull Request #6547 · databricks/cli

anton-107 · 2026-09-07T12:17:42Z

Changes

Adds the two server-lifecycle flags that are fixed when the SSH tunnel server job is submitted, and so can only be set by the invocation that starts it:

--max-clients on ssh setup, plumbed through SetupOptions → ClientOptions → ToProxyCommand → proxy-mode ssh connect → the submit widget.
--server-timeout on both ssh setup and ssh connect, replacing the hardcoded serverTimeout = 24 * time.Hour constant that capped every tunnel.

ClientOptions.Validate now range-checks both, and ssh setup validates the options it is about to serialize so an unusable host config is rejected at setup time rather than at the first ssh <name>.

Why

A customer using ssh setup on a dedicated cluster hit repeated connection failures that only cleared by stopping the bootstrap job run:

failed to establish websocket connection: websocket: bad handshake

The server refuses new connections once MaxClients (default 10) is reached, and gorilla/websocket reports any non-101 response as that opaque message. Raising the limit was unreachable for ssh setup users: the command had no flag, and ToProxyCommand did not serialize the value — so the ProxyCommand, which is the invocation that actually submits the server job, always fell back to the default.

Two decisions worth flagging for review:

Zero values are rejected rather than passed through. --max-clients=0 yields a server that refuses every connection, and timeout_seconds: 0 means "no timeout" in the Jobs API, which would turn the cap into an unbounded run.
max(serverTimeout, shutdownDelay) on connect is gone. Now that --server-timeout is a user flag, that max would silently override an explicit value (--server-timeout=1h --shutdown-delay=2h → 2h). ServerTimeout comes straight from the flag, and a shutdown delay longer than it is an error. This changes behavior for anyone who relied on --shutdown-delay > 24h to extend a tunnel: they now get an actionable error pointing at --server-timeout.

Scope is deliberately limited to these two flags; the other gaps found while investigating (--ssh-keys-dir, --strict-host-key-checking, --connect-timeout, SSH connection multiplexing, and surfacing the server's 503 body instead of "bad handshake") are tracked separately.

Tests

New acceptance test acceptance/ssh/setup/: asserts the ProxyCommand written by setup --max-clients=25 --server-timeout=48h, both rejection messages, and that no host config is written for a rejected setup. It prints only the ProxyCommand, so the golden carries no OS-specific absolute paths and the test runs on all three platforms.
New experimental/ssh/cmd/setup_test.go: round-trips a setup-generated ProxyCommand back through ssh connect's own FlagSet — the regression guard for this exact bug class (a flag setup writes that connect cannot parse, or parses and discards).
Unit coverage for the new Validate checks, the new ToProxyCommand cases, the on-disk host config, Setup's rejection path, and buildSSHServerSubmitRun carrying maxClients / timeout_seconds.
Existing goldens are unchanged, confirming the defaults did not move.
Verified end to end on a dogfood dedicated cluster, with the setup-generated ProxyCommand as the submitting invocation: it submitted the job with timeout_seconds 7200 on both run and task and maxClients "2"; with two clients connected the third was refused with the customer's exact error, and a client connected again once a slot freed. The default of 10 would have admitted it.

This PR was written by Claude Code.

…meout) A customer using `ssh setup` on a dedicated cluster hit repeated connection failures that only cleared by stopping the bootstrap job run: failed to establish websocket connection: websocket: bad handshake The tunnel server refuses new connections once MaxClients (default 10) is reached, and gorilla/websocket reports any non-101 response as that opaque message. Raising the limit was unreachable for `ssh setup` users: the command had no --max-clients flag, and ToProxyCommand did not serialize the value, so the ProxyCommand -- which is the invocation that actually submits the server job -- always fell back to the default. Both knobs are fixed at submission time, so they are only settable by the invocation that starts the server: - --max-clients on `ssh setup`, carried through SetupOptions, ClientOptions and the ProxyCommand into the submit widget. - --server-timeout on both `ssh setup` and `ssh connect`, replacing the hardcoded 24h serverTimeout constant that capped every tunnel. Validate now range-checks both, because their zero values fail silently: --max-clients=0 yields a server that rejects every connection, and timeout_seconds: 0 means "no timeout" in the Jobs API, turning the cap into an unbounded run. `ssh setup` validates the ClientOptions it serializes so a config that can never work is rejected at setup time rather than at first `ssh <name>`. That also replaces `max(serverTimeout, shutdownDelay)` on connect, which would have silently overridden an explicit --server-timeout. Verified end to end on a dogfood dedicated cluster: the setup-generated ProxyCommand, executed by OpenSSH, submitted the server job with timeout_seconds 7200 and maxClients 2; with two clients connected the third was refused, and it connected again once a slot freed. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-09-07T12:38:40Z

Integration test report

Commit: b35a362

Run: 34213965515

	Env	🔄flaky	💚RECOVERED	✅pass	🙈skip	Time
💚	aws linux		1	275	16	6:09
💚	aws windows		1	277	14	4:49
💚	azure linux		1	274	16	7:56
🔄	azure windows	1	1	275	14	4:40
💚	gcp linux		1	275	16	6:59
💚	gcp windows		1	277	14	6:10

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🔄	TestSyncIncrementalSyncPythonNotebookToFile	✅p	✅p	✅p	🔄f	✅p	✅p

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
2:47	gcp windows	TestAccept
2:09	aws windows	TestFilerWorkspaceFilesExtensionsStat
2:01	azure linux	TestFilerWorkspaceFilesExtensionsReadDir

Adding --server-timeout replaced `ServerTimeout: max(serverTimeout, shutdownDelay)` with the flag value alone. That turned a --shutdown-delay longer than the 24h default into a hard error, and `ssh setup` persists such a delay into the generated ProxyCommand -- which OpenSSH runs verbatim. A host configured before the flag existed therefore broke on every `ssh <name>`, with an error naming a flag the user has no way to pass from an already-written config. Restore the old lifetime, gated on whether --server-timeout was set, using the cmd.Flags().Changed idiom connect.go already applies to --environment-version. An explicit --server-timeout still wins, so the reason the max() was dropped -- a longer shutdown delay silently widening a lifetime the user asked for -- still holds, and Validate still rejects that pair. The resolved lifetime is now serialized into the ProxyCommand, so a config that relies on this is explicit about it. Co-authored-by: Isaac <no-reply@databricks.com>

Validate rejected only ServerTimeout <= 0, but the value that ships is int(ServerTimeout.Seconds()) on both the run and its task. Every duration below one second truncates to 0, and 0 means "no timeout" in the Jobs API -- so --server-timeout=999ms passed validation and submitted a tunnel server with no lifetime cap at all, which is exactly the outcome the check was added to prevent, and the opposite of what was asked for. Check against the granularity that is actually submitted, and add a test over the range around the bound asserting that nothing Validate accepts can submit timeout_seconds: 0, so the bound and the conversion cannot drift apart again. Co-authored-by: Isaac <no-reply@databricks.com>

rugpanov

Looks good overall. The server-lifecycle options are plumbed through setup, the generated ProxyCommand, connect, and job submission with solid cross-platform coverage.\n\nNon-blocking nice-to-have: in experimental/ssh/internal/setup/setup.go, consider constructing and validating ClientOptions after resolving the cluster ID but before ValidateClusterAccess. That would surface invalid values such as --max-clients=0 before making the workspace API call, so an auth/network error cannot mask the actionable flag error.

rclarey

Currently both --max-clients and --server-timeout are silently ignored if the server is already running, should we instead serialize them to metadata.json (like we do for usage policy) then start a new server with the requested config if the flag values don't match the running server?

Move the --max-clients, --server-timeout, and --shutdown-delay checks to the top of Setup(), before the interactive cluster picker and ValidateClusterAccess call. These flag values don't depend on cluster details, so there's no reason to delay rejecting them until after the user has gone through cluster selection. The corresponding test no longer needs a Clusters.Get mock expectation since the error is returned before any cluster API calls are made. Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.16.0 ### CLI * `aitools install` now registers the official Claude marketplace if it is missing before installing the Databricks Claude plugin. ([#6485](#6485)) * `databricks aitools install --output json` now reports an `error_category` for a failed or skipped install (per agent, and at the top level for a failure with no per-agent entry), giving coding agents and CI a stable classification of why an install did not complete. ([#6482](#6482)) * `databricks aitools install` honors `--output json`, emitting a structured `{scope, agents[...]}` document that reports each agent's delivery and install status so coding agents and CI can consume the result without scraping the text output. JSON mode requires `--scope` and `--agents` so the command runs without interactive prompts. ([#6481](#6481)) * `databricks bundle sync` now prints sync progress (`Action: PUT`, `Uploaded ...`) by default, matching `databricks sync`. Previously it was silent unless `--output` was passed. Use `--output json` for machine-readable output. ([#6568](#6568)) * Support major-only DBR runtime versions such as `19.x-scala2.13` in the cluster picker used by `databricks auth login --configure-cluster` and `databricks labs`. ([#6574](#6574)) * Deprecated the `databricks environments setup-local --constraints-only` flag in favour of the orthogonal `--no-dbconnect`; the flag still works as a hidden alias but is hidden from `--help` and prints a one-line deprecation notice, and will be removed in a later release. ([#6470](#6470)) * Add orthogonal `--no-constraints` and `--no-dbconnect` flags to `databricks environments setup-local`: `--no-constraints` skips writing the remote Python-version and dependency pins, and `--no-dbconnect` skips the databricks-connect dependency. ([#6464](#6464)) * `databricks environments setup-local` now reports a distinct `E_PROVISION_CONFLICT` error code in `--output json` when the project's dependencies conflict with the pins written for the target environment, making the requirements unsatisfiable (the same conflict surfaced as a `W_USER_CONSTRAINT_CONFLICT` warning); it is reported after the project files are written, without attempting the doomed provisioning, while other provisioning failures continue to report `E_PROVISION`. ([#6479](#6479)) * `databricks ssh connect` and `ssh setup` now verify the tunnel's SSH host key against the key the workspace published for the connection, recorded in `~/.databricks/ssh-tunnel-known-hosts/<name>` instead of `~/.ssh/known_hosts`. Reconnecting with a name used before no longer fails with `Host key verification failed` when the compute behind that name changed, and no longer needs a manual `ssh-keygen -R`; host blocks written by an earlier `databricks ssh setup` pick this up once you re-run it. ([#6557](#6557)) * Stop `databricks ssh connect --ide` from adding a duplicate entry to the IDE's Remote Explorer on every connect: the remote authority is now the SSH host alias alone, instead of embedding the per-instance remote OS user. ([#6550](#6550)) * Add `--max-clients` and `--server-timeout` flags to `databricks ssh setup`, and `--server-timeout` to `databricks ssh connect`. Both are fixed when the SSH tunnel server job is submitted, so `ssh setup` now serializes them into the generated `ProxyCommand` instead of falling back to the built-in defaults. ([#6547](#6547)) * `ssh connect` sessions no longer end when the tunnel's websocket connection is lost. The CLI reattaches to the running session and replays the bytes that were missed, so the shell and everything running in it stay intact, and a transient failure to open a replacement connection for the periodic auth refresh is retried rather than ending the session. Reattaching requires an SSH server started by a CLI that supports it; against an older server the connection behaves as before. ([#6558](#6558)) ### Bundles * Added PyDABs (Python) support for secrets: `Resources.add_secret` and the `secret_mutator` decorator. ([#6553](#6553)) * Fix job and pipeline environment dependencies with a `*` version wildcard (e.g. `numpy==2.5.*`) being treated as local file paths. ([#6555](#6555)) * Add the `postgres_snapshot_schedules` bundle resource for managing a Lakebase Postgres branch's automatic-snapshot schedule (direct deployment engine only). ([#6449](#6449)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.175.0 to v0.177.0. ([#6448](#6448)) * Bump Terraform provider from v1.128.0 to v1.131.0. ([#6544](#6544))

…meout) (#6547) ## Changes Adds the two server-lifecycle flags that are fixed when the SSH tunnel server job is submitted, and so can only be set by the invocation that starts it: - `--max-clients` on `ssh setup`, plumbed through `SetupOptions` → `ClientOptions` → `ToProxyCommand` → proxy-mode `ssh connect` → the submit widget. - `--server-timeout` on both `ssh setup` and `ssh connect`, replacing the hardcoded `serverTimeout = 24 * time.Hour` constant that capped every tunnel. `ClientOptions.Validate` now range-checks both, and `ssh setup` validates the options it is about to serialize so an unusable host config is rejected at setup time rather than at the first `ssh <name>`. ## Why A customer using `ssh setup` on a dedicated cluster hit repeated connection failures that only cleared by stopping the bootstrap job run: ``` failed to establish websocket connection: websocket: bad handshake ``` The server refuses new connections once `MaxClients` (default 10) is reached, and gorilla/websocket reports any non-101 response as that opaque message. Raising the limit was unreachable for `ssh setup` users: the command had no flag, and `ToProxyCommand` did not serialize the value — so the ProxyCommand, which is the invocation that actually submits the server job, always fell back to the default. Two decisions worth flagging for review: - **Zero values are rejected rather than passed through.** `--max-clients=0` yields a server that refuses every connection, and `timeout_seconds: 0` means "no timeout" in the Jobs API, which would turn the cap into an unbounded run. - **`max(serverTimeout, shutdownDelay)` on connect is gone.** Now that `--server-timeout` is a user flag, that `max` would silently override an explicit value (`--server-timeout=1h --shutdown-delay=2h` → 2h). `ServerTimeout` comes straight from the flag, and a shutdown delay longer than it is an error. This changes behavior for anyone who relied on `--shutdown-delay > 24h` to extend a tunnel: they now get an actionable error pointing at `--server-timeout`. Scope is deliberately limited to these two flags; the other gaps found while investigating (`--ssh-keys-dir`, `--strict-host-key-checking`, `--connect-timeout`, SSH connection multiplexing, and surfacing the server's 503 body instead of "bad handshake") are tracked separately. ## Tests - New acceptance test `acceptance/ssh/setup/`: asserts the ProxyCommand written by `setup --max-clients=25 --server-timeout=48h`, both rejection messages, and that no host config is written for a rejected setup. It prints only the ProxyCommand, so the golden carries no OS-specific absolute paths and the test runs on all three platforms. - New `experimental/ssh/cmd/setup_test.go`: round-trips a setup-generated ProxyCommand back through `ssh connect`'s own FlagSet — the regression guard for this exact bug class (a flag setup writes that connect cannot parse, or parses and discards). - Unit coverage for the new `Validate` checks, the new `ToProxyCommand` cases, the on-disk host config, `Setup`'s rejection path, and `buildSSHServerSubmitRun` carrying `maxClients` / `timeout_seconds`. - Existing goldens are unchanged, confirming the defaults did not move. - Verified end to end on a dogfood dedicated cluster, with the setup-generated ProxyCommand as the *submitting* invocation: it submitted the job with `timeout_seconds` 7200 on both run and task and `maxClients` `"2"`; with two clients connected the third was refused with the customer's exact error, and a client connected again once a slot freed. The default of 10 would have admitted it. _This PR was written by Claude Code._ --------- Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.16.0 ### CLI * `aitools install` now registers the official Claude marketplace if it is missing before installing the Databricks Claude plugin. ([#6485](#6485)) * `databricks aitools install --output json` now reports an `error_category` for a failed or skipped install (per agent, and at the top level for a failure with no per-agent entry), giving coding agents and CI a stable classification of why an install did not complete. ([#6482](#6482)) * `databricks aitools install` honors `--output json`, emitting a structured `{scope, agents[...]}` document that reports each agent's delivery and install status so coding agents and CI can consume the result without scraping the text output. JSON mode requires `--scope` and `--agents` so the command runs without interactive prompts. ([#6481](#6481)) * `databricks bundle sync` now prints sync progress (`Action: PUT`, `Uploaded ...`) by default, matching `databricks sync`. Previously it was silent unless `--output` was passed. Use `--output json` for machine-readable output. ([#6568](#6568)) * Support major-only DBR runtime versions such as `19.x-scala2.13` in the cluster picker used by `databricks auth login --configure-cluster` and `databricks labs`. ([#6574](#6574)) * Deprecated the `databricks environments setup-local --constraints-only` flag in favour of the orthogonal `--no-dbconnect`; the flag still works as a hidden alias but is hidden from `--help` and prints a one-line deprecation notice, and will be removed in a later release. ([#6470](#6470)) * Add orthogonal `--no-constraints` and `--no-dbconnect` flags to `databricks environments setup-local`: `--no-constraints` skips writing the remote Python-version and dependency pins, and `--no-dbconnect` skips the databricks-connect dependency. ([#6464](#6464)) * `databricks environments setup-local` now reports a distinct `E_PROVISION_CONFLICT` error code in `--output json` when the project's dependencies conflict with the pins written for the target environment, making the requirements unsatisfiable (the same conflict surfaced as a `W_USER_CONSTRAINT_CONFLICT` warning); it is reported after the project files are written, without attempting the doomed provisioning, while other provisioning failures continue to report `E_PROVISION`. ([#6479](#6479)) * `databricks ssh connect` and `ssh setup` now verify the tunnel's SSH host key against the key the workspace published for the connection, recorded in `~/.databricks/ssh-tunnel-known-hosts/<name>` instead of `~/.ssh/known_hosts`. Reconnecting with a name used before no longer fails with `Host key verification failed` when the compute behind that name changed, and no longer needs a manual `ssh-keygen -R`; host blocks written by an earlier `databricks ssh setup` pick this up once you re-run it. ([#6557](#6557)) * Stop `databricks ssh connect --ide` from adding a duplicate entry to the IDE's Remote Explorer on every connect: the remote authority is now the SSH host alias alone, instead of embedding the per-instance remote OS user. ([#6550](#6550)) * Add `--max-clients` and `--server-timeout` flags to `databricks ssh setup`, and `--server-timeout` to `databricks ssh connect`. Both are fixed when the SSH tunnel server job is submitted, so `ssh setup` now serializes them into the generated `ProxyCommand` instead of falling back to the built-in defaults. ([#6547](#6547)) * `ssh connect` sessions no longer end when the tunnel's websocket connection is lost. The CLI reattaches to the running session and replays the bytes that were missed, so the shell and everything running in it stay intact, and a transient failure to open a replacement connection for the periodic auth refresh is retried rather than ending the session. Reattaching requires an SSH server started by a CLI that supports it; against an older server the connection behaves as before. ([#6558](#6558)) ### Bundles * Added PyDABs (Python) support for secrets: `Resources.add_secret` and the `secret_mutator` decorator. ([#6553](#6553)) * Fix job and pipeline environment dependencies with a `*` version wildcard (e.g. `numpy==2.5.*`) being treated as local file paths. ([#6555](#6555)) * Add the `postgres_snapshot_schedules` bundle resource for managing a Lakebase Postgres branch's automatic-snapshot schedule (direct deployment engine only). ([#6449](#6449)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.175.0 to v0.177.0. ([#6448](#6448)) * Bump Terraform provider from v1.128.0 to v1.131.0. ([#6544](#6544))

anton-107 and others added 2 commits September 7, 2026 08:42

Link changelog fragment to the PR

5b618e5

Co-authored-by: Isaac <no-reply@databricks.com>

anton-107 and others added 2 commits September 7, 2026 14:56

anton-107 requested review from rclarey and rugpanov September 7, 2026 13:17

rugpanov approved these changes Sep 8, 2026

View reviewed changes

rclarey requested changes Sep 8, 2026

View reviewed changes

rclarey reviewed Sep 8, 2026

View reviewed changes

Comment thread experimental/ssh/internal/setup/setup.go

rclarey approved these changes Sep 8, 2026

View reviewed changes

anton-107 added this pull request to the merge queue Sep 8, 2026

Merged via the queue into main with commit f468b7a Sep 8, 2026
39 checks passed

anton-107 deleted the deco-28401-missing-flags-add-missing-server branch September 8, 2026 14:05

rugpanov mentioned this pull request Sep 9, 2026

Update Databricks CLI to v1.16.0 databricks/databricks-vscode#2183

Merged

Navigation Menu

Add server-lifecycle flags to `ssh setup` (--max-clients, --server-timeout) - #6547

Add server-lifecycle flags to `ssh setup` (--max-clients, --server-timeout)#6547
anton-107 merged 5 commits into
mainfrom
deco-28401-missing-flags-add-missing-server

anton-107 commented Sep 7, 2026

Uh oh!

eng-dev-ecosystem-bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

rugpanov left a comment

Uh oh!

rclarey left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

anton-107 commented Sep 7, 2026

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

rugpanov left a comment

Choose a reason for hiding this comment

Uh oh!

rclarey left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eng-dev-ecosystem-bot commented Sep 7, 2026 •

edited

Loading

rclarey left a comment •

edited

Loading