Support time.Time in PyDABs codegen and generate the Secret resource - #6553
Merged
Sankalp-Mittal merged 7 commits intoSep 8, 2026
Conversation
Collaborator
Integration test reportCommit: 91928df
Top 25 slowest tests (at least 2 minutes):
|
janniklasrose
approved these changes
Sep 8, 2026
Sankalp-Mittal
force-pushed
the
sankalp-mittal/cli-secrets-support
branch
from
September 8, 2026 14:23
2e5b019 to
ae2dbdb
Compare
Map the SDK time.Time type to Python str (an RFC3339 timestamp string, matching how the Go side models it in libs/dyn/convert/sdk_native_types.go) and mark it terminal for reachability, so codegen no longer fails on it. This lets resources.Secret (whose expire_time field is a *time.Time) drop off the denylist and generate. Co-authored-by: Isaac <no-reply@databricks.com>
Adds the secrets-support fixture required by test_python_support_coverage. It loads a Secret from YAML and Python, runs a secret_mutator over both, and exercises the expire_time field (RFC3339 string) end to end. value is a default-less variable reference supplied at validate time, as the secret value validator requires; direct-engine only. Co-authored-by: Isaac <no-reply@databricks.com>
_synth_scalar only knew string/int/float/bool, so a required time.Time field would fail codegen with "Unknown primitive: time.Time". Add a case producing an RFC3339 placeholder string (time.Time generates as str). No effect on current output: the only time.Time field, secrets.expire_time, is optional and dropped by the field policy before synthesis. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/cli-secrets-support
branch
from
September 8, 2026 15:14
ae2dbdb to
91928df
Compare
Collaborator
Integration test reportCommit: 54c3d05
1024 interesting tests: 881 MISS, 142 FAIL, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Sep 9, 2026
## Release v1.16.0 ### CLI * `aitools install` now registers the official Claude marketplace if it is missing before installing the Databricks Claude plugin. ([#6485](#6485)) * `databricks aitools install --output json` now reports an `error_category` for a failed or skipped install (per agent, and at the top level for a failure with no per-agent entry), giving coding agents and CI a stable classification of why an install did not complete. ([#6482](#6482)) * `databricks aitools install` honors `--output json`, emitting a structured `{scope, agents[...]}` document that reports each agent's delivery and install status so coding agents and CI can consume the result without scraping the text output. JSON mode requires `--scope` and `--agents` so the command runs without interactive prompts. ([#6481](#6481)) * `databricks bundle sync` now prints sync progress (`Action: PUT`, `Uploaded ...`) by default, matching `databricks sync`. Previously it was silent unless `--output` was passed. Use `--output json` for machine-readable output. ([#6568](#6568)) * Support major-only DBR runtime versions such as `19.x-scala2.13` in the cluster picker used by `databricks auth login --configure-cluster` and `databricks labs`. ([#6574](#6574)) * Deprecated the `databricks environments setup-local --constraints-only` flag in favour of the orthogonal `--no-dbconnect`; the flag still works as a hidden alias but is hidden from `--help` and prints a one-line deprecation notice, and will be removed in a later release. ([#6470](#6470)) * Add orthogonal `--no-constraints` and `--no-dbconnect` flags to `databricks environments setup-local`: `--no-constraints` skips writing the remote Python-version and dependency pins, and `--no-dbconnect` skips the databricks-connect dependency. ([#6464](#6464)) * `databricks environments setup-local` now reports a distinct `E_PROVISION_CONFLICT` error code in `--output json` when the project's dependencies conflict with the pins written for the target environment, making the requirements unsatisfiable (the same conflict surfaced as a `W_USER_CONSTRAINT_CONFLICT` warning); it is reported after the project files are written, without attempting the doomed provisioning, while other provisioning failures continue to report `E_PROVISION`. ([#6479](#6479)) * `databricks ssh connect` and `ssh setup` now verify the tunnel's SSH host key against the key the workspace published for the connection, recorded in `~/.databricks/ssh-tunnel-known-hosts/<name>` instead of `~/.ssh/known_hosts`. Reconnecting with a name used before no longer fails with `Host key verification failed` when the compute behind that name changed, and no longer needs a manual `ssh-keygen -R`; host blocks written by an earlier `databricks ssh setup` pick this up once you re-run it. ([#6557](#6557)) * Stop `databricks ssh connect --ide` from adding a duplicate entry to the IDE's Remote Explorer on every connect: the remote authority is now the SSH host alias alone, instead of embedding the per-instance remote OS user. ([#6550](#6550)) * Add `--max-clients` and `--server-timeout` flags to `databricks ssh setup`, and `--server-timeout` to `databricks ssh connect`. Both are fixed when the SSH tunnel server job is submitted, so `ssh setup` now serializes them into the generated `ProxyCommand` instead of falling back to the built-in defaults. ([#6547](#6547)) * `ssh connect` sessions no longer end when the tunnel's websocket connection is lost. The CLI reattaches to the running session and replays the bytes that were missed, so the shell and everything running in it stay intact, and a transient failure to open a replacement connection for the periodic auth refresh is retried rather than ending the session. Reattaching requires an SSH server started by a CLI that supports it; against an older server the connection behaves as before. ([#6558](#6558)) ### Bundles * Added PyDABs (Python) support for secrets: `Resources.add_secret` and the `secret_mutator` decorator. ([#6553](#6553)) * Fix job and pipeline environment dependencies with a `*` version wildcard (e.g. `numpy==2.5.*`) being treated as local file paths. ([#6555](#6555)) * Add the `postgres_snapshot_schedules` bundle resource for managing a Lakebase Postgres branch's automatic-snapshot schedule (direct deployment engine only). ([#6449](#6449)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.175.0 to v0.177.0. ([#6448](#6448)) * Bump Terraform provider from v1.128.0 to v1.131.0. ([#6544](#6544))
janniklasrose
pushed a commit
that referenced
this pull request
Sep 15, 2026
…6553) Stacked on #6548. ## Changes Enables PyDABs codegen to model the SDK `time.Time` type, which unblocks generating `resources.Secret` (its `expire_time` field is a `*time.Time`) and drops it from the codegen denylist. `time.Time` is mapped to Python `str` — an RFC3339 timestamp string. This mirrors how the Go side already models the type at its own config boundary (`libs/dyn/convert/sdk_native_types.go` treats `time.Time` as a string), so a PyDABs value flows unchanged through the same Go normalization pipeline that `databricks.yml` uses. Regenerated output adds the `secrets` module and its wiring; the `databricks.bundles.core` public-API snapshot update is purely additive (`add_secret`, `secret_mutator`, `secrets`). ## Tests - New `acceptance/bundle/python/secrets-support/` fixture (required by `test_python_support_coverage`): loads a Secret from YAML and Python, runs a `secret_mutator` over both, and exercises `expire_time` (RFC3339 string) end to end. `value` is a default-less variable reference supplied at validate time, as the secret value validator requires; direct-engine only. Passes with and without `-update`. - `pydabs-test` 423 passed; codegen unit tests 37 passed. ## Notes - A changelog fragment (`.nextchanges/bundles/`) is still to be added. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
pushed a commit
that referenced
this pull request
Sep 15, 2026
## Release v1.16.0 ### CLI * `aitools install` now registers the official Claude marketplace if it is missing before installing the Databricks Claude plugin. ([#6485](#6485)) * `databricks aitools install --output json` now reports an `error_category` for a failed or skipped install (per agent, and at the top level for a failure with no per-agent entry), giving coding agents and CI a stable classification of why an install did not complete. ([#6482](#6482)) * `databricks aitools install` honors `--output json`, emitting a structured `{scope, agents[...]}` document that reports each agent's delivery and install status so coding agents and CI can consume the result without scraping the text output. JSON mode requires `--scope` and `--agents` so the command runs without interactive prompts. ([#6481](#6481)) * `databricks bundle sync` now prints sync progress (`Action: PUT`, `Uploaded ...`) by default, matching `databricks sync`. Previously it was silent unless `--output` was passed. Use `--output json` for machine-readable output. ([#6568](#6568)) * Support major-only DBR runtime versions such as `19.x-scala2.13` in the cluster picker used by `databricks auth login --configure-cluster` and `databricks labs`. ([#6574](#6574)) * Deprecated the `databricks environments setup-local --constraints-only` flag in favour of the orthogonal `--no-dbconnect`; the flag still works as a hidden alias but is hidden from `--help` and prints a one-line deprecation notice, and will be removed in a later release. ([#6470](#6470)) * Add orthogonal `--no-constraints` and `--no-dbconnect` flags to `databricks environments setup-local`: `--no-constraints` skips writing the remote Python-version and dependency pins, and `--no-dbconnect` skips the databricks-connect dependency. ([#6464](#6464)) * `databricks environments setup-local` now reports a distinct `E_PROVISION_CONFLICT` error code in `--output json` when the project's dependencies conflict with the pins written for the target environment, making the requirements unsatisfiable (the same conflict surfaced as a `W_USER_CONSTRAINT_CONFLICT` warning); it is reported after the project files are written, without attempting the doomed provisioning, while other provisioning failures continue to report `E_PROVISION`. ([#6479](#6479)) * `databricks ssh connect` and `ssh setup` now verify the tunnel's SSH host key against the key the workspace published for the connection, recorded in `~/.databricks/ssh-tunnel-known-hosts/<name>` instead of `~/.ssh/known_hosts`. Reconnecting with a name used before no longer fails with `Host key verification failed` when the compute behind that name changed, and no longer needs a manual `ssh-keygen -R`; host blocks written by an earlier `databricks ssh setup` pick this up once you re-run it. ([#6557](#6557)) * Stop `databricks ssh connect --ide` from adding a duplicate entry to the IDE's Remote Explorer on every connect: the remote authority is now the SSH host alias alone, instead of embedding the per-instance remote OS user. ([#6550](#6550)) * Add `--max-clients` and `--server-timeout` flags to `databricks ssh setup`, and `--server-timeout` to `databricks ssh connect`. Both are fixed when the SSH tunnel server job is submitted, so `ssh setup` now serializes them into the generated `ProxyCommand` instead of falling back to the built-in defaults. ([#6547](#6547)) * `ssh connect` sessions no longer end when the tunnel's websocket connection is lost. The CLI reattaches to the running session and replays the bytes that were missed, so the shell and everything running in it stay intact, and a transient failure to open a replacement connection for the periodic auth refresh is retried rather than ending the session. Reattaching requires an SSH server started by a CLI that supports it; against an older server the connection behaves as before. ([#6558](#6558)) ### Bundles * Added PyDABs (Python) support for secrets: `Resources.add_secret` and the `secret_mutator` decorator. ([#6553](#6553)) * Fix job and pipeline environment dependencies with a `*` version wildcard (e.g. `numpy==2.5.*`) being treated as local file paths. ([#6555](#6555)) * Add the `postgres_snapshot_schedules` bundle resource for managing a Lakebase Postgres branch's automatic-snapshot schedule (direct deployment engine only). ([#6449](#6449)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.175.0 to v0.177.0. ([#6448](#6448)) * Bump Terraform provider from v1.128.0 to v1.131.0. ([#6544](#6544))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #6548.
Changes
Enables PyDABs codegen to model the SDK
time.Timetype, which unblocks generatingresources.Secret(itsexpire_timefield is a*time.Time) and drops it from the codegen denylist.time.Timeis mapped to Pythonstr— an RFC3339 timestamp string. This mirrors how the Go side already models the type at its own config boundary (libs/dyn/convert/sdk_native_types.gotreatstime.Timeas a string), so a PyDABs value flows unchanged through the same Go normalization pipeline thatdatabricks.ymluses.Regenerated output adds the
secretsmodule and its wiring; thedatabricks.bundles.corepublic-API snapshot update is purely additive (add_secret,secret_mutator,secrets).Tests
acceptance/bundle/python/secrets-support/fixture (required bytest_python_support_coverage): loads a Secret from YAML and Python, runs asecret_mutatorover both, and exercisesexpire_time(RFC3339 string) end to end.valueis a default-less variable reference supplied at validate time, as the secret value validator requires; direct-engine only. Passes with and without-update.pydabs-test423 passed; codegen unit tests 37 passed.Notes
.nextchanges/bundles/) is still to be added.This pull request and its description were written by Isaac.