Skip to content

Emit E_PROVISION_CONFLICT for uv sync resolution conflicts - #6479

Merged
rugpanov merged 1 commit into
mainfrom
setup-local/provision-conflict
Sep 7, 2026
Merged

rugpanov merged 1 commit into
mainfrom
setup-local/provision-conflict

Conversation

@rugpanov

@rugpanov rugpanov commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

databricks environments setup-local wrapped every provisioning failure as
E_PROVISION. The VPEX extension's recovery flow needs to distinguish a
dependency version conflict — the project's dependencies can't be satisfied
against the pins this command wrote for the target environment — from a generic
failure it can't fix by relaxing constraints (a build-backend error, a permissions
problem, a transport error). Fixes DECO-28365.

What

  • New error code E_PROVISION_CONFLICT (libs/localenv/result.go), reported
    from the provision phase.
  • The merge phase already detects a provable version conflict between the
    user's dependencies and the environment pins and records it as the
    W_USER_CONSTRAINT_CONFLICT warning (libs/localenv/warnings.go, PEP 440
    interval math on provably-disjoint ranges, scanning every requirement uv locks).
  • When that warning is present, uv sync would deterministically fail to resolve,
    so the pipeline fails fast: right after the merge writes the project files it
    reports E_PROVISION_CONFLICT (failurePhase=provision, diskMutated=true)
    without attempting the Python install or uv sync
    (libs/localenv/pipeline.go, hasConstraintConflictWarning). Every other
    provisioning failure keeps E_PROVISION.
  • Gating on the CLI's own merge detection (rather than parsing uv's stderr) keeps
    the code precise — no false positive on an unavailable package or an
    unrelated sync failure — and needs no brittle stderr matching. diskMutated=true
    is preserved because the constraints are written before the check, which the
    extension's recovery flow relies on.
  • --dry-run is unchanged: it computes a plan and does not evaluate provisioning,
    so the same conflict surfaces there as the W_USER_CONSTRAINT_CONFLICT warning
    (ok=true) — a test pins that intended divergence.
  • Telemetry: adds SetupLocalErrorCodeProvisionConflict and the mapping case
    (cmd/environments/telemetry.go, libs/telemetry/protos/setup_local.go), kept
    exhaustive by the linter and TestErrorCodeCoversLocalenv.

Consumer-facing note

Adding E_PROVISION_CONFLICT is additive, but this exact scenario previously
surfaced as E_PROVISION. Any consumer that special-cased E_PROVISION for the
conflict case will see the new code. The universe lumberjack proto should add the
matching SetupLocalErrorCode enum value (ingestion ignores unknown values, so the
two changes can land in either order).

Testing

  • Unit (libs/localenv/pipeline_test.go): a conflict warning makes the run
    report E_PROVISION_CONFLICT at the provision phase with diskMutated=true and
    without invoking Python install or sync (recordingPM records neither);
    --dry-run with the same pins reports ok=true with the warning and no error,
    writing nothing. TestPipelineRetainsFallbackResolutionWhenProvisioningFails
    still covers a generic sync failure → E_PROVISION.
  • Acceptance (acceptance/localenv/provision-conflict/): a real setup-local
    run where the user pins pip==24.0 and the remote constraints pin pip<24; the
    merge flags the conflict and the command reports E_PROVISION_CONFLICT before
    provisioning. Cross-platform (only preflight uv --version runs; the doomed sync
    never does), so it runs on every OS with no fake binary.
  • gofmt, go vet, golangci-lint, and the local check-changelog pass.

This pull request and its description were written by Isaac.

@rugpanov
rugpanov marked this pull request as ready for review September 2, 2026 08:36
Comment thread libs/localenv/pipeline.go Outdated
// keeps. Gating on the merge signal rather than uv's stderr keeps the code
// precise: it fires only when a conflict the CLI itself detected is present.
//
// The warning means the merged pins are provably unsatisfiable, so a real

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we already know the pins are unsatisfiable why do we wait for uv sync to fail? Why not error right away?

@rugpanov rugpanov Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — now we error right away.

@rugpanov
rugpanov force-pushed the setup-local/orthogonal-flags branch from 7d3ec5f to f9ffddd Compare September 3, 2026 15:24
Base automatically changed from setup-local/orthogonal-flags to main September 4, 2026 14:28
@rugpanov
rugpanov force-pushed the setup-local/provision-conflict branch 2 times, most recently from e6fe8cd to dc380a0 Compare September 4, 2026 15:03
`databricks environments setup-local` wrapped every provisioning failure as
E_PROVISION. The extension's recovery flow needs to tell a dependency version
conflict — the project's dependencies can't be satisfied against the pins this
command wrote for the target environment — apart from a generic failure it can't
fix by relaxing constraints.

The merge phase already detects a provable version conflict and records it as the
W_USER_CONSTRAINT_CONFLICT warning. When that fires, uv sync would deterministically
fail to resolve, so report the new E_PROVISION_CONFLICT right after writing the
project files (diskMutated=true, at the provision phase) instead of spending a
doomed Python install and sync. Gating on the CLI's own detection keeps the code
precise — no uv-stderr matching and no false positive on an unrelated sync failure.

Adds the matching telemetry enum value and its coverage case, a unit test asserting
the fail-fast path reports the conflict without invoking provisioning, and an
acceptance golden.

Co-authored-by: Isaac <no-reply@databricks.com>
@rugpanov
rugpanov force-pushed the setup-local/provision-conflict branch from dc380a0 to 6c07a2b Compare September 4, 2026 15:13
@rugpanov
rugpanov requested a review from rclarey September 4, 2026 15:21
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 6c07a2b

Run: 33888334586

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux 1 275 16 6:15
💚​ aws windows 1 277 14 4:08
💚​ azure linux 1 274 16 7:07
💚​ azure windows 1 276 14 4:12
💚​ gcp linux 1 275 16 7:05
💚​ gcp windows 1 277 14 4:29
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:20 gcp windows TestAccept
4:05 aws windows TestAccept
4:04 azure windows TestAccept
4:02 azure linux TestAccept
3:48 aws linux TestAccept
3:39 gcp linux TestAccept

@rugpanov
rugpanov added this pull request to the merge queue Sep 7, 2026
Merged via the queue into main with commit 0d07982 Sep 7, 2026
35 checks passed
@rugpanov
rugpanov deleted the setup-local/provision-conflict branch September 7, 2026 15:52
deco-sdk-tagging Bot added a commit that referenced this pull request Sep 9, 2026
## Release v1.16.0

### CLI

 * `aitools install` now registers the official Claude marketplace if it is missing before installing the Databricks Claude plugin. ([#6485](#6485))
 * `databricks aitools install --output json` now reports an `error_category` for a failed or skipped install (per agent, and at the top level for a failure with no per-agent entry), giving coding agents and CI a stable classification of why an install did not complete. ([#6482](#6482))
 * `databricks aitools install` honors `--output json`, emitting a structured `{scope, agents[...]}` document that reports each agent's delivery and install status so coding agents and CI can consume the result without scraping the text output. JSON mode requires `--scope` and `--agents` so the command runs without interactive prompts. ([#6481](#6481))
 * `databricks bundle sync` now prints sync progress (`Action: PUT`, `Uploaded ...`) by default, matching `databricks sync`. Previously it was silent unless `--output` was passed. Use `--output json` for machine-readable output. ([#6568](#6568))
 * Support major-only DBR runtime versions such as `19.x-scala2.13` in the cluster picker used by `databricks auth login --configure-cluster` and `databricks labs`. ([#6574](#6574))
 * Deprecated the `databricks environments setup-local --constraints-only` flag in favour of the orthogonal `--no-dbconnect`; the flag still works as a hidden alias but is hidden from `--help` and prints a one-line deprecation notice, and will be removed in a later release. ([#6470](#6470))
 * Add orthogonal `--no-constraints` and `--no-dbconnect` flags to `databricks environments setup-local`: `--no-constraints` skips writing the remote Python-version and dependency pins, and `--no-dbconnect` skips the databricks-connect dependency. ([#6464](#6464))
 * `databricks environments setup-local` now reports a distinct `E_PROVISION_CONFLICT` error code in `--output json` when the project's dependencies conflict with the pins written for the target environment, making the requirements unsatisfiable (the same conflict surfaced as a `W_USER_CONSTRAINT_CONFLICT` warning); it is reported after the project files are written, without attempting the doomed provisioning, while other provisioning failures continue to report `E_PROVISION`. ([#6479](#6479))
 * `databricks ssh connect` and `ssh setup` now verify the tunnel's SSH host key against the key the workspace published for the connection, recorded in `~/.databricks/ssh-tunnel-known-hosts/<name>` instead of `~/.ssh/known_hosts`. Reconnecting with a name used before no longer fails with `Host key verification failed` when the compute behind that name changed, and no longer needs a manual `ssh-keygen -R`; host blocks written by an earlier `databricks ssh setup` pick this up once you re-run it. ([#6557](#6557))
 * Stop `databricks ssh connect --ide` from adding a duplicate entry to the IDE's Remote Explorer on every connect: the remote authority is now the SSH host alias alone, instead of embedding the per-instance remote OS user. ([#6550](#6550))
 * Add `--max-clients` and `--server-timeout` flags to `databricks ssh setup`, and `--server-timeout` to `databricks ssh connect`. Both are fixed when the SSH tunnel server job is submitted, so `ssh setup` now serializes them into the generated `ProxyCommand` instead of falling back to the built-in defaults. ([#6547](#6547))
 * `ssh connect` sessions no longer end when the tunnel's websocket connection is lost. The CLI reattaches to the running session and replays the bytes that were missed, so the shell and everything running in it stay intact, and a transient failure to open a replacement connection for the periodic auth refresh is retried rather than ending the session. Reattaching requires an SSH server started by a CLI that supports it; against an older server the connection behaves as before. ([#6558](#6558))

### Bundles

 * Added PyDABs (Python) support for secrets: `Resources.add_secret` and the `secret_mutator` decorator. ([#6553](#6553))
 * Fix job and pipeline environment dependencies with a `*` version wildcard (e.g. `numpy==2.5.*`) being treated as local file paths. ([#6555](#6555))
 * Add the `postgres_snapshot_schedules` bundle resource for managing a Lakebase Postgres branch's automatic-snapshot schedule (direct deployment engine only). ([#6449](#6449))

### Dependency Updates

 * Bump `github.com/databricks/databricks-sdk-go` from v0.175.0 to v0.177.0. ([#6448](#6448))
 * Bump Terraform provider from v1.128.0 to v1.131.0. ([#6544](#6544))
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
## Why

`databricks environments setup-local` wrapped every provisioning failure
as
`E_PROVISION`. The VPEX extension's recovery flow needs to distinguish a
**dependency version conflict** — the project's dependencies can't be
satisfied
against the pins this command wrote for the target environment — from a
generic
failure it can't fix by relaxing constraints (a build-backend error, a
permissions
problem, a transport error). Fixes DECO-28365.

## What

- New error code **`E_PROVISION_CONFLICT`** (`libs/localenv/result.go`),
reported
  from the provision phase.
- The merge phase already detects a **provable** version conflict
between the
  user's dependencies and the environment pins and records it as the
`W_USER_CONSTRAINT_CONFLICT` warning (`libs/localenv/warnings.go`, PEP
440
interval math on provably-disjoint ranges, scanning every requirement uv
locks).
- When that warning is present, `uv sync` would deterministically fail
to resolve,
so the pipeline **fails fast**: right after the merge writes the project
files it
reports `E_PROVISION_CONFLICT` (`failurePhase=provision`,
`diskMutated=true`)
  **without** attempting the Python install or `uv sync`
(`libs/localenv/pipeline.go`, `hasConstraintConflictWarning`). Every
other
  provisioning failure keeps `E_PROVISION`.
- Gating on the CLI's own merge detection (rather than parsing uv's
stderr) keeps
the code **precise** — no false positive on an unavailable package or an
unrelated sync failure — and needs no brittle stderr matching.
`diskMutated=true`
is preserved because the constraints are written before the check, which
the
  extension's recovery flow relies on.
- `--dry-run` is unchanged: it computes a plan and does not evaluate
provisioning,
so the same conflict surfaces there as the `W_USER_CONSTRAINT_CONFLICT`
warning
  (`ok=true`) — a test pins that intended divergence.
- Telemetry: adds `SetupLocalErrorCodeProvisionConflict` and the mapping
case
(`cmd/environments/telemetry.go`,
`libs/telemetry/protos/setup_local.go`), kept
  exhaustive by the linter and `TestErrorCodeCoversLocalenv`.

### Consumer-facing note

Adding `E_PROVISION_CONFLICT` is additive, but this exact scenario
previously
surfaced as `E_PROVISION`. Any consumer that special-cased `E_PROVISION`
for the
conflict case will see the new code. The universe lumberjack proto
should add the
matching `SetupLocalErrorCode` enum value (ingestion ignores unknown
values, so the
two changes can land in either order).

## Testing

- **Unit** (`libs/localenv/pipeline_test.go`): a conflict warning makes
the run
report `E_PROVISION_CONFLICT` at the provision phase with
`diskMutated=true` and
**without** invoking Python install or sync (`recordingPM` records
neither);
`--dry-run` with the same pins reports `ok=true` with the warning and no
error,
writing nothing.
`TestPipelineRetainsFallbackResolutionWhenProvisioningFails`
  still covers a generic sync failure → `E_PROVISION`.
- **Acceptance** (`acceptance/localenv/provision-conflict/`): a real
`setup-local`
run where the user pins `pip==24.0` and the remote constraints pin
`pip<24`; the
merge flags the conflict and the command reports `E_PROVISION_CONFLICT`
before
provisioning. Cross-platform (only preflight `uv --version` runs; the
doomed sync
  never does), so it runs on every OS with no fake binary.
- `gofmt`, `go vet`, `golangci-lint`, and the local `check-changelog`
pass.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
## Release v1.16.0

### CLI

 * `aitools install` now registers the official Claude marketplace if it is missing before installing the Databricks Claude plugin. ([#6485](#6485))
 * `databricks aitools install --output json` now reports an `error_category` for a failed or skipped install (per agent, and at the top level for a failure with no per-agent entry), giving coding agents and CI a stable classification of why an install did not complete. ([#6482](#6482))
 * `databricks aitools install` honors `--output json`, emitting a structured `{scope, agents[...]}` document that reports each agent's delivery and install status so coding agents and CI can consume the result without scraping the text output. JSON mode requires `--scope` and `--agents` so the command runs without interactive prompts. ([#6481](#6481))
 * `databricks bundle sync` now prints sync progress (`Action: PUT`, `Uploaded ...`) by default, matching `databricks sync`. Previously it was silent unless `--output` was passed. Use `--output json` for machine-readable output. ([#6568](#6568))
 * Support major-only DBR runtime versions such as `19.x-scala2.13` in the cluster picker used by `databricks auth login --configure-cluster` and `databricks labs`. ([#6574](#6574))
 * Deprecated the `databricks environments setup-local --constraints-only` flag in favour of the orthogonal `--no-dbconnect`; the flag still works as a hidden alias but is hidden from `--help` and prints a one-line deprecation notice, and will be removed in a later release. ([#6470](#6470))
 * Add orthogonal `--no-constraints` and `--no-dbconnect` flags to `databricks environments setup-local`: `--no-constraints` skips writing the remote Python-version and dependency pins, and `--no-dbconnect` skips the databricks-connect dependency. ([#6464](#6464))
 * `databricks environments setup-local` now reports a distinct `E_PROVISION_CONFLICT` error code in `--output json` when the project's dependencies conflict with the pins written for the target environment, making the requirements unsatisfiable (the same conflict surfaced as a `W_USER_CONSTRAINT_CONFLICT` warning); it is reported after the project files are written, without attempting the doomed provisioning, while other provisioning failures continue to report `E_PROVISION`. ([#6479](#6479))
 * `databricks ssh connect` and `ssh setup` now verify the tunnel's SSH host key against the key the workspace published for the connection, recorded in `~/.databricks/ssh-tunnel-known-hosts/<name>` instead of `~/.ssh/known_hosts`. Reconnecting with a name used before no longer fails with `Host key verification failed` when the compute behind that name changed, and no longer needs a manual `ssh-keygen -R`; host blocks written by an earlier `databricks ssh setup` pick this up once you re-run it. ([#6557](#6557))
 * Stop `databricks ssh connect --ide` from adding a duplicate entry to the IDE's Remote Explorer on every connect: the remote authority is now the SSH host alias alone, instead of embedding the per-instance remote OS user. ([#6550](#6550))
 * Add `--max-clients` and `--server-timeout` flags to `databricks ssh setup`, and `--server-timeout` to `databricks ssh connect`. Both are fixed when the SSH tunnel server job is submitted, so `ssh setup` now serializes them into the generated `ProxyCommand` instead of falling back to the built-in defaults. ([#6547](#6547))
 * `ssh connect` sessions no longer end when the tunnel's websocket connection is lost. The CLI reattaches to the running session and replays the bytes that were missed, so the shell and everything running in it stay intact, and a transient failure to open a replacement connection for the periodic auth refresh is retried rather than ending the session. Reattaching requires an SSH server started by a CLI that supports it; against an older server the connection behaves as before. ([#6558](#6558))

### Bundles

 * Added PyDABs (Python) support for secrets: `Resources.add_secret` and the `secret_mutator` decorator. ([#6553](#6553))
 * Fix job and pipeline environment dependencies with a `*` version wildcard (e.g. `numpy==2.5.*`) being treated as local file paths. ([#6555](#6555))
 * Add the `postgres_snapshot_schedules` bundle resource for managing a Lakebase Postgres branch's automatic-snapshot schedule (direct deployment engine only). ([#6449](#6449))

### Dependency Updates

 * Bump `github.com/databricks/databricks-sdk-go` from v0.175.0 to v0.177.0. ([#6448](#6448))
 * Bump Terraform provider from v1.128.0 to v1.131.0. ([#6544](#6544))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants