Skip to content

bundle: add env var to set record_deployment_history in built-in templates - #6220

Merged
ilyakuz-db merged 4 commits into
mainfrom
init-record-deployment-history
Aug 12, 2026
Merged

ilyakuz-db merged 4 commits into
mainfrom
init-record-deployment-history

Conversation

@ilyakuz-db

Copy link
Copy Markdown
Contributor

Changes

bundle init now writes experimental.record_deployment_history: true into the generated databricks.yml when DATABRICKS_BUNDLE_INIT_RECORD_DEPLOYMENT_HISTORY is set. It covers every built-in template that generates a databricks.yml, and is off by default so template output is otherwise unchanged.

Why

Lets new projects opt into recording deployment history at creation time instead of editing the config by hand afterwards. The variable is temporary and deliberately scoped to bundle init: it persists the setting in the generated project rather than toggling it for the bundle being run.

Note the setting is still gated off (#6094), so a generated project also needs DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY to validate until that gate is lifted. The acceptance test sets both and runs bundle validate, so it will fail if the two ever drift apart.

Tests

New acceptance test: all five templates with the variable on, plus the default-off and explicitly-false cases.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 0b3d04e

Run: 31580753636

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 286 1135 5:04
💚​ aws windows 4 4 288 1133 3:23
💚​ azure linux 4 4 285 1135 5:58
💚​ azure windows 4 4 287 1133 3:58
💚​ gcp linux 1 5 286 1135 5:38
💚​ gcp windows 1 5 288 1133 3:59
8 interesting tests: 4 RECOVERED, 4 SKIP
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
2:57 azure linux TestAccept
2:56 gcp linux TestAccept
2:53 aws linux TestAccept
2:40 aws windows TestAccept
2:14 gcp windows TestAccept
2:09 azure windows TestAccept

Comment thread libs/template/helpers.go
// it to the bundle being run.
//
// Temporary: remove it and its callers in the templates once this is the default.
const recordDeploymentHistoryEnvVar = "DATABRICKS_BUNDLE_INIT_RECORD_DEPLOYMENT_HISTORY"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets call it DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY sounds like a more generic name, which we may use for general enablement? I wanted to keep this specific variable only for templates

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed in DM that we keep a separate var for init for now, and further decide if we want to migrate to generic var in the workspace


# record_deployment_history is gated off for now, so generated projects only validate with the
# force-allow escape hatch set.
Env.DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY = "true"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need this if all we are running is bundle init.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

@ilyakuz-db
ilyakuz-db force-pushed the init-record-deployment-history branch from bc8f26c to 0b3d04e Compare August 12, 2026 08:57
@ilyakuz-db
ilyakuz-db added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit 831805c Aug 12, 2026
27 checks passed
@ilyakuz-db
ilyakuz-db deleted the init-record-deployment-history branch August 12, 2026 10:32
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 831805c

Run: 31587946128

Env ❌​FAIL 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
❌​ aws linux 6 1 5 2 1161 941 217:32
❌​ aws windows 6 5 5 2 1096 960 279:25
🔄​ azure linux 3 5 2 1059 978 216:58
💚​ azure windows 6 2 1000 997 253:26
🔄​ gcp linux 4 3 3 1047 982 238:11
🔄​ gcp windows 3 2 3 988 1001 253:15
24 interesting tests: 10 flaky, 6 FAIL, 5 RECOVERED, 2 SKIP, 1 KNOWN
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🔄​f 🔄​f 💚​R 💚​R 🔄​f
🔄​ TestAccept/bundle/deployment/bind/pipelines/recreate ✅​p 🔄​f ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/deployment/bind/pipelines/recreate/DATABRICKS_BUNDLE_ENGINE=direct ✅​p 🔄​f ✅​p ✅​p 🔄​f ✅​p
💚​ TestAccept/bundle/invariant/no_drift 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/resources/clusters/deploy/local_ssd_count 🙈​s 🙈​s 🙈​s 🙈​s ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/dashboards/change-parent-path ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/dashboards/delete-trashed-out-of-band ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/dashboards/delete-trashed-out-of-band/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions ❌​F ❌​F 🙈​s 🙈​s 🙈​s 🙈​s
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions ❌​F ❌​F 🙈​s 🙈​s 🙈​s 🙈​s
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F
🔄​ TestAccept/bundle/resources/quality_monitors/change_output_schema_name ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
💚​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
16:45 gcp windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
14:53 gcp linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
12:32 azure windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
12:04 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
9:37 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:31 aws windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:22 gcp windows TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct
8:15 azure linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:13 aws linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:09 azure windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:56 gcp windows TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
7:54 azure linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
7:48 azure linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:40 aws windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:32 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:24 gcp windows TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
7:24 gcp linux TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
7:07 azure windows TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=direct
6:55 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
6:50 gcp windows TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:42 azure windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:40 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:37 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct
6:32 aws linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:29 aws windows TestAccept/bundle/invariant/continue_293/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_pydabs_10_tasks.yml.tmpl
6:26 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
6:21 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:20 azure windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
6:19 azure windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
6:16 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:13 aws windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:09 aws windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
6:06 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
6:02 gcp linux TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=direct
5:56 azure windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
5:56 gcp windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:55 gcp windows TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
5:54 azure windows TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=no/READPLAN=
5:52 azure windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=no/READPLAN=1
5:51 azure windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no/READPLAN=
5:47 gcp windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
5:45 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=app.yml.tmpl/READPLAN=1
5:45 gcp linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
5:45 azure windows TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
5:44 aws linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
5:41 azure windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
5:36 aws linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
5:32 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job.yml.tmpl/READPLAN=1
5:29 gcp linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:25 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants