Skip to content

ci: align Buildkite step labels, emoji, and keys #714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcalhoun merged 7 commits into from
Oct 13, 2026
Merged

ci: align Buildkite step labels, emoji, and keys #714

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3f5f0a3
ci: key pipeline steps by the make target they run
dcalhoun Sep 21, 2026
698108e
ci: label pipeline steps by platform with verb-first titles
dcalhoun Sep 21, 2026
9c59912
ci: key the web bundle step by make build
dcalhoun Sep 22, 2026
0432c93
ci: key pipeline steps by the platform-named make targets
dcalhoun Sep 22, 2026
267a3cc
ci: label steps by the platform-named make targets
dcalhoun Sep 22, 2026
56c8512
docs: name the renamed Android library E2E step in a test comment
dcalhoun Sep 22, 2026
424a6a5
ci: report step checks under explicit commit status contexts
dcalhoun Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 119 additions & 46 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,21 @@ env:
# The download is flaky from CI, so skip it -- see the Test Web E2E job.
ELECTRON_SKIP_BINARY_DOWNLOAD: '1'

# Each step carries one emoji naming the platform it builds, tests, or ships —
# :javascript: for the web editor (there is no clean web glyph), :ios:, or
# :android: — and a verb-first, title-case label that stands on its own.
# Where a step runs or mirrors a make target, key it by that target's name.
#
#
# Each step reports its GitHub check under an explicit `notify` context — the
# label minus its emoji — so label tweaks don't rename checks. The trunk ruleset
# requires several contexts by name, so changing one needs a ruleset update.
steps:
- label: ':react: Build React App'
key: build-react
- label: ':javascript: Build Web Bundle'
key: build
notify:
- github_commit_status:
context: 'Build Web Bundle'
command: |
# Fail fast if any step (including the backgrounded
# `fetch-translations`, surfaced via `wait`) returns non-zero,
Expand Down Expand Up @@ -45,32 +57,47 @@ steps:
- $CI_TOOLKIT_PLUGIN
- $NVM_PLUGIN

- label: ':white_check_mark: Validate Swift release ${NEW_VERSION:-(no version)}'
- label: ':ios: Validate Swift Release ${NEW_VERSION:-(no version)}'
key: validate-release
notify:
- github_commit_status:
context: 'Validate Swift Release'
if: build.env("NEW_VERSION") != null && build.branch == "trunk" && build.pull_request.id == null
command: |
install_gems
bundle exec fastlane validate "version:$NEW_VERSION"
plugins: *plugins
Comment on lines +60 to 69

@mokagio mokagio Sep 23, 2026 •
edited by AliSoftware
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the CI label renames, the repository's required checks configuration must be changed in order to merge this PR. To receive passing CI results, pending PRs will need to rebase atop trunk once this merges.

To avoid this issue cropping up in the future, we can 1) required every step and group to use notify.github_commit_status; 2) update the pipeline setting to force reporting the custom status instead of deriving from the label name (that's outside of this repo's scope)

Example for 1:

Suggested change
- label: ':ios: Validate Swift Release ${NEW_VERSION:-(no version)}'
key: validate-release
if: build.env("NEW_VERSION") != null && build.branch == "trunk" && build.pull_request.id == null
command: |
install_gems
bundle exec fastlane validate "version:$NEW_VERSION"
plugins: *plugins
- label: ':ios: Validate Swift Release ${NEW_VERSION:-(no version)}'
key: validate-release
if: build.env("NEW_VERSION") != null && build.branch == "trunk" && build.pull_request.id == null
command: |
install_gems
bundle exec fastlane validate "version:$NEW_VERSION"
plugins: *plugins
notify:
- github_commit_status:
context: Validate Swift Release

@mokagio mokagio Sep 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. update the pipeline setting to force reporting the custom status instead of deriving from the label name (that's outside of this repo's scope)

If this suggestion resonates, I opened a PR in our private Buildkite configuration repository.

If it doesn't and you want to merge sooner without coordination, that's fine, too. My team will eventually get to adopt the approach in all repos and we'll update the GitHub checks settings one more time then. No dramas.

@oguzkocer oguzkocer Sep 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mokagio Wouldn't that move the same problem to a different layer? If we do this, the next time a rename happens we'll have to make a decision on whether to rename the custom labels or not. If we rename, we are back to square one with more to maintain, if we don't, now the action and the label are potentially out of sync or doesn't match the current naming scheme.

Especially in a small repo that won't impact many branches, nor would it require expensive rebases, the fix sounds like it creates more problems than it solves. But, that's just my take on it. Happy to leave the decision to you regardless.

@dcalhoun dcalhoun Sep 25, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mokagio interested in your thoughts on @oguzkocer perspective when you have time, no rush. I plan to leave this PR for the time being as it is not urgent.

@AliSoftware AliSoftware Sep 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gio has gone AFK for ≈ a month, but here's my take:

Sure it moves the problem to a different layer, but one key is that the notify: { github_commit_status: "…" } approach allows you to have a clean GitHub check name that is free of formatting or emojis etc, while keeping the label of the step in Buildkite's own UI nicer and fancier.

1️⃣ Indeed when only relying on the label: of the step, Buildkite will derive the GitHub check name by replacing emojis with :code:, add buildkite/gutenbergkit/ prefixes, etc, which (at leasts IMHO) makes them less readable in GitHub PRs' UI. So personally I think the benefit of using notify: { github_commit_status: "…" } is also about better readability of the GitHub check names on GitHub PRs' UI.
For example, using an explicit notify: { github_commit_status: "Test iOS App E2E" } on the label: ':ios: Test iOS App E2E' step will allow you to have it shown as a nice "Test iOS App E2E" line in the GitHub checks at the bottom of the PR rather than the more ugly buildkite/gutenbergkit/ios-test-ios-app-e2e.

2️⃣ To Gio's point though, the other benefit of decoupling github_commit_status from label is that if you decide to make a minimal change to the label: of a step (e.g. changing label: ':s3: Publish XCFramework to S3' to label: ':ios: Publish XCFramework to S3', or changing label: ':swift: :package: Publish PR XCFramework' to label: ':ios: Publish PR XCFramework'), at least this won't change the GitHub Check name/identifier under you, and thus if you used a commit_status that still makes senses before/after (e.g. github_commit_status: 'Publish XCFramework to S3' / github_commit_status: 'Publish PR XCFramework' without the emoji), this won't require you to update the GitHub repo's settings to update the "Required Checks" list on the branch protection settings just for that. The label: and the GitHub check name can be kept independent instead of every little change of a label implicitly changing the GitHub check name under you.

Of course, if you do change the semantics of what a step does and that you end up changing its label: not just for a typo fix or for a change of emoji, but for a change in meaning, you'll probably still want to also update the notify: { github_commit_status: "…" } part of the step alongside your change of its label: to make the GitHub check name reflect that change of semantics too (e.g. for label: ':android: Test Android Library Instrumented' -> label: ':android: Test Android Library E2E' you'd probably also want to update github_commit_status: 'Test Android Library Instrumented' -> github_commit_status: 'Test Android Library E2E'). And thus would still have to update the branch protection settings accordingly.
But at least:

  1. You only need to do that when you change the semantics of the step—and that also changing the GitHub check name would also make sense—not just when you change an emoji or small typo fix
  2. You can provide a more readable GitHub check name than the one derived by Buildkite from the label (no more buildkite/gutenbergkit/ prefix, no transformation of :ios: emoji code from the label: into an ios- prefix in the check name…)
  3. It's immediately clear from the diff when your change will require updating the branch protection settings. If you change just the label: but not the github_commit_status, no need to update the branch protection settings; only if your PR makes changes to the github_commit_status will it be an explicit signal/reminder to update the settings. While today with the GitHub commit status name derived from the label: it might not be as obvious to PR authors or reviewers that changing the label: will have this unfortunate consequence, making them more likely to forget to update the settings.

@dcalhoun dcalhoun Oct 9, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the helpful explanation. 🙇🏻‍♂️ I added the suggested notify.github_commit_status to each task. I believe the next steps are:

  1. Update the repository's required checks to the new tasks
  2. Merge this PR
  3. Merge and deploy the Buildkite CI PR Gio referenced earlier

I believe No. 3 must occur last, and the only impact between merges will be that duplicate tasks will show up on PR reports.

@AliSoftware would you be willing to help with No. 3 after this PR merges? I can merge early next week.

@AliSoftware AliSoftware Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AliSoftware would you be willing to help with No. 3 after this PR merges? I can merge early next week.

Sure thing. It'd be easy enough since Gio already prepared the PR, which is here. So it'd just be a matter of (rebasing it + undraft it +) hit the button to deploy it once you did 1 & 2.

@dcalhoun dcalhoun Oct 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AliSoftware thank you! This PR was merged and the project's required checks are updated. We can proceed with the Buildkite CI PR.

hit the button to deploy

Is this the PR merge button or a separate deploy action? I originally requested your assistance because I presumed it was the latter, a separate deploy step for which I lack permissions. If it is simply merging the PR, I'm happy to manage that, just let me know. 🙇🏻‍♂️

@AliSoftware AliSoftware Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a different button, I can take care of it 🙂

@AliSoftware AliSoftware Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, the buildkite/gutenbergkit/… entries on GitHub PR checks should not appear on new PRs, only keeping the explicit ones you provided now.


- label: ':eslint: Lint React App'
key: lint-js
- label: ':javascript: Lint Web'
key: lint-web
notify:
- github_commit_status:
context: 'Lint Web'
command: make lint-web
plugins: *plugins

- label: ':npm: Check @wordpress package versions'
- label: ':javascript: Check @wordpress Package Versions'
key: check-wp-packages
notify:
- github_commit_status:
context: 'Check @wordpress Package Versions'
command: make check-wp-packages
plugins: *plugins

- label: ':javascript: Test JavaScript'
key: test-js
- label: ':javascript: Test Web Unit'
key: test-web-unit
notify:
- github_commit_status:
context: 'Test Web Unit'
command: make test-web-unit
plugins: *plugins

- label: ':performing_arts: Test Web E2E'
- label: ':javascript: Test Web E2E'
key: test-web-e2e
depends_on: build-react
notify:
- github_commit_status:
context: 'Test Web E2E'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -81,7 +108,10 @@ steps:
plugins: *plugins

- label: ':android: Publish Android Library'
depends_on: build-react
notify:
- github_commit_status:
context: 'Publish Android Library'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -94,19 +124,30 @@ steps:
plugins: *plugins

- label: ':android: Lint Android'
key: lint-android
notify:
- github_commit_status:
context: 'Lint Android'
command: make lint-android
agents:
queue: android
plugins: *plugins

- label: ':swift: SwiftLint'
key: lint-swift
- label: ':ios: Lint iOS'
key: lint-ios
notify:
- github_commit_status:
context: 'Lint iOS'
command: swiftlint
agents:
queue: linter

- label: ':android: Test Android Library'
depends_on: build-react
- label: ':android: Test Android Library Unit'
key: test-android-library-unit
notify:
- github_commit_status:
context: 'Test Android Library Unit'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -115,29 +156,38 @@ steps:
queue: android
plugins: *plugins

- label: ':swift: iOS Simulator Tests'
key: swift-test-swift-package
depends_on: build-react
- label: ':ios: Test iOS Library Simulator'
key: test-ios-library-simulator
notify:
- github_commit_status:
context: 'Test iOS Library Simulator'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
make test-ios-library-simulator
plugins: *plugins

- label: ':swift: Library Tests'
key: swift-test-library
depends_on: build-react
- label: ':ios: Test iOS Library Host'
key: test-ios-library-host
notify:
- github_commit_status:
context: 'Test iOS Library Host'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
make test-ios-library-host
plugins: *plugins

- label: ':xcode: Build XCFramework'
key: build-xcframework
- label: ':ios: Build XCFramework'
key: build-ios-resources-xcframework
notify:
- github_commit_status:
context: 'Build XCFramework'
depends_on:
- build-react
- swift-test-library
- build
- test-ios-library-host
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -151,12 +201,15 @@ steps:
- '*.xcframework.zip.checksum.txt'
plugins: *plugins

- label: ':s3: Publish XCFramework to S3'
depends_on: build-xcframework
- label: ':ios: Publish XCFramework to S3'
notify:
- github_commit_status:
context: 'Publish XCFramework to S3'
depends_on: build-ios-resources-xcframework
# This step only covers per-commit trunk uploads keyed by the commit
# SHA. Releases are handled by `:rocket: Publish Swift release` (gated
# SHA. Releases are handled by `:ios: Publish Swift Release` (gated
# on `NEW_VERSION`), and tag pushes from that step trigger a separate
# tag build whose iOS upload would just duplicate the rocket step's —
# tag build whose iOS upload would just duplicate the release step's —
# the `build.tag == null` clause skips it. Android publish on tag
# builds is still load-bearing (produces the `vX.Y.Z` Maven artifact)
# and intentionally not gated here.
Expand All @@ -168,23 +221,29 @@ steps:
bundle exec fastlane publish_to_s3 version:$BUILDKITE_COMMIT
plugins: *plugins

- label: ':swift: :package: Publish PR XCFramework'
depends_on: build-xcframework
- label: ':ios: Publish PR XCFramework'
notify:
- github_commit_status:
context: 'Publish PR XCFramework'
depends_on: build-ios-resources-xcframework
if: build.pull_request.id != null
command: .buildkite/publish-pr-xcframework.sh
plugins: *plugins

- label: ':rocket: Publish Swift release ${NEW_VERSION:-(Error: No version found!)}'
- label: ':ios: Publish Swift Release ${NEW_VERSION:-(Error: No version found!)}'
notify:
- github_commit_status:
context: 'Publish Swift Release'
depends_on:
- validate-release
- build-xcframework
- swift-test-swift-package
- lint-js
- lint-swift
- build-ios-resources-xcframework
- test-ios-library-simulator
- lint-web
- lint-ios
- check-wp-packages
- test-js
- test-web-unit
- test-web-e2e
- test-ios-e2e
- test-ios-app-e2e
if: build.env("NEW_VERSION") != null && build.branch == "trunk" && build.pull_request.id == null
# Pass NEW_VERSION as an argument: build-level env vars set when
# triggering aren't exposed to job runtime shells on `queue: mac`, but
Expand All @@ -193,9 +252,12 @@ steps:
command: .buildkite/release.sh "$NEW_VERSION"
plugins: *plugins

- label: ':ios: Test iOS E2E'
key: test-ios-e2e
depends_on: build-react
- label: ':ios: Test iOS App E2E'
key: test-ios-app-e2e
notify:
- github_commit_status:
context: 'Test iOS App E2E'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -212,8 +274,12 @@ steps:
- 'build/ios-e2e.xcresult.zip'
plugins: *plugins

- label: ':android: Test Android E2E'
depends_on: build-react
- label: ':android: Test Android App E2E'
key: test-android-app-e2e
notify:
- github_commit_status:
context: 'Test Android App E2E'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -232,8 +298,12 @@ steps:
- 'android/app/build/outputs/buildkite-logs/**/*'
- 'android/app/build/outputs/connected_android_test_additional_output/**/*'

- label: ':android: Test Android Library Instrumented'
depends_on: build-react
- label: ':android: Test Android Library E2E'
key: test-android-library-e2e
notify:
- github_commit_status:
context: 'Test Android Library E2E'
depends_on: build
command: |
buildkite-agent artifact download dist.tar.gz .
tar -xzf dist.tar.gz
Expand All @@ -251,7 +321,10 @@ steps:
- 'android/Gutenberg/build/outputs/buildkite-logs/**/*'
- 'android/Gutenberg/build/outputs/connected_android_test_additional_output/**/*'

- label: ':wastebasket: Clean up `pr-build/*` branches for closed PRs'
- label: ':ios: Clean Up `pr-build/*` Branches for Closed PRs'
notify:
- github_commit_status:
context: 'Clean Up pr-build/* Branches for Closed PRs'
if: build.branch == "trunk"
command: .buildkite/cleanup-pr-build-branches.sh
plugins: *plugins
4 changes: 2 additions & 2 deletions .buildkite/publish-pr-xcframework.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ echo '--- :robot_face: Use bot for Git operations'
source use-bot-for-git

echo '--- :arrow_down: Downloading XCFramework artifacts'
buildkite-agent artifact download '*.xcframework.zip' . --step "build-xcframework"
buildkite-agent artifact download '*.xcframework.zip.checksum.txt' . --step "build-xcframework"
buildkite-agent artifact download '*.xcframework.zip' . --step "build-ios-resources-xcframework"
buildkite-agent artifact download '*.xcframework.zip.checksum.txt' . --step "build-ios-resources-xcframework"

echo '--- :rubygems: Setting up Gems'
install_gems
Expand Down
4 changes: 2 additions & 2 deletions .buildkite/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ echo '--- :robot_face: Use bot for Git operations'
source use-bot-for-git

echo '--- :arrow_down: Downloading XCFramework artifacts'
buildkite-agent artifact download '*.xcframework.zip' . --step "build-xcframework"
buildkite-agent artifact download '*.xcframework.zip.checksum.txt' . --step "build-xcframework"
buildkite-agent artifact download '*.xcframework.zip' . --step "build-ios-resources-xcframework"
buildkite-agent artifact download '*.xcframework.zip.checksum.txt' . --step "build-ios-resources-xcframework"

echo '--- :rubygems: Setting up Gems'
install_gems
Expand Down
2 changes: 1 addition & 1 deletion .../androidTest/java/org/wordpress/gutenberg/preview/HtmlToBitmapRendererInstrumentedTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import java.io.File
/**
* Instrumented tests that exercise [HtmlToBitmapRenderer] against a real Android
* [android.webkit.WebView]. Runs in CI via the `:android: Test Android Library
* Instrumented` step (`make test-android-library-e2e`); locally:
* E2E` step (`make test-android-library-e2e`); locally:
*
* ./gradlew :Gutenberg:connectedAndroidTest
*
Expand Down
2 changes: 1 addition & 1 deletion bin/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,7 @@ print_publish_instructions() {
echo "'trunk' to whatever HEAD is at trigger time, and a concurrent merge"
echo "would tag the wrong commit."
echo
echo "The :rocket: 'Publish Swift release' step will build + sign the"
echo "The :ios: 'Publish Swift Release' step will build + sign the"
echo "XCFramework, upload it to S3, and publish the GitHub Release —"
echo "which also creates the $tag tag."
}
Expand Down
4 changes: 2 additions & 2 deletions docs/releases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Every push to `trunk` publishes both platforms automatically:

- **Android**: the `:android: Publish Android Library` step pushes a Maven artifact keyed by the commit (consumable via Git revision pins).
- **iOS**: the `:s3: Publish XCFramework to S3` step uploads the signed XCFramework under `gutenbergkit/<commit-sha>/`, and `Publish PR XCFramework` does the same on PR builds under a `pr-build/<n>` snapshot branch.
- **iOS**: the `:ios: Publish XCFramework to S3` step uploads the signed XCFramework under `gutenbergkit/<commit-sha>/`, and `Publish PR XCFramework` does the same on PR builds under a `pr-build/<n>` snapshot branch.

A **tagged release** is a separate, manually-triggered publish flow on top of that: it produces a stable `vX.Y.Z` tag whose `Package.swift` points at the prebuilt XCFramework on CDN, plus a GitHub Release with the XCFramework attached. SPM consumers pin the tag; everything else can pin a commit/branch.

Expand Down Expand Up @@ -60,7 +60,7 @@ Step 1 prints the SHA of the version-bump commit it just pushed. Trigger a new B

Pinning the commit matters — if you leave it blank, Buildkite resolves `trunk` to HEAD at trigger time, and a concurrent merge would tag the wrong commit.

The build runs a `:white_check_mark: Validate Swift release` step early on (gated on `NEW_VERSION`) that fast-fails if the tag name is malformed, if the tag or GitHub Release already exists, or if no previous release tag can be resolved to generate notes against. It logs the tag the notes will be based on, so a wrong base surfaces before anything is published. After that, the `:rocket: Publish Swift release` step:
The build runs a `:ios: Validate Swift Release` step early on (gated on `NEW_VERSION`) that fast-fails if the tag name is malformed, if the tag or GitHub Release already exists, or if no previous release tag can be resolved to generate notes against. It logs the tag the notes will be based on, so a wrong base surfaces before anything is published. After that, the `:ios: Publish Swift Release` step:

1. Rewrites `Package.swift` to consume the binary target via `.release(version:, checksum:)`
1. Uploads the XCFramework to `s3://a8c-apps-public-artifacts/gutenbergkit/vX.Y.Z/`
Expand Down
Loading