ci(deploy): deploy pre-built images tagged by commit SHA by alanpeixinho · Pull Request #2131 · kernelci/dashboard

alanpeixinho · 2026-09-21T21:15:22Z

What it is

Staging and production deploy pre-built GHCR images by commit SHA instead of :latest or on-server builds.

Publish GHCR Images — Pick the branch or tag with Use workflow from. Images are tagged with that commit's full SHA. :latest only on push to main.
Staging — CI deploys the push SHA after tests. A manual run deploys the selected branch or tag. Pull + up --no-build.
Production — Manual run from a branch or tag. Pulls that commit's full SHA. DASHBOARD_VERSION comes from that commit.
Staging compose image paths match .env.example; docs updated.
Do not run Deploy production Dashboard for this PR.

How to test

The workflow file has to be on kernelci/dashboard. A fork branch is not a valid --ref there. Push this commit to a branch on that repo, then pass that branch name.

1. Publish (no staging, no prod)

gh workflow run "Publish GHCR Images" \
  --repo kernelci/dashboard \
  --ref <branch>

There is no ref input. The branch you pass is the commit that gets built.

Open the run → Summary: Ref: <branch> and Commit SHA: <full sha>.
Confirm tags in GHCR (https://github.com/orgs/kernelci/packages):
- dashboard-backend, dashboard-frontend, dashboard-proxy each have a tag equal to that full SHA.
- On this manual run, the latest tag digest should be unchanged.

2. Negative staging (unpublished SHA)

Use a real branch whose commit was not published. A fake SHA cannot be passed; --ref must exist or the run never starts.

gh workflow run "Deploy Staging and E2E" \
    --repo kernelci/dashboard \
    --ref chore/deploy-sha-images \
    -f image_tag=<commit tag from the published image>

gh workflow run "Deploy Staging and E2E" \
  --repo kernelci/dashboard \
  --ref <branch-with-no-images>

check-migrations succeeds.
deploy-staging pulls that full SHA, retries for about 10 minutes, then fails with Timed out waiting for GHCR images.
Containers are not recreated. The host still deletes and checks out dashboard-staging before the pull, so that directory changes.
e2e-tests does not run.

3. Manual staging (live staging)

Same command as (2), --ref = the branch published in (1). This updates staging.

gh workflow run "Deploy Staging and E2E" \
  --repo kernelci/dashboard \
  --ref <branch>

Run Summary: Deploying staging \` (``)`.
The deploy log pulls ghcr.io/kernelci/dashboard/dashboard-backend:<full sha>, and the same for dashboard-frontend and dashboard-proxy.
Do not use the staging site version string; that is git describe, not the image tag.

4. After merge to `main` (live staging; existing CI behavior)

Two runs, same merge commit. Actions at https://github.com/kernelci/dashboard/actions.
CI

Left: CI for KernelCI Dashboard.
Open the run with event push (not pull_request). Title is the merge commit.
Run header shows the commit SHA (github.sha).
Jobs: call-deploy-staging / Deploy Staging and E2E must run (skipped = you opened the PR CI run).
Open job deploy-staging → run Summary:
- Deploying staging \main` (``)` equals the header SHA.
- Images pulled are …/dashboard-*:<same sha>, not :latest.
  Publish
Left: Publish GHCR Images.
Open the run for that same commit.
Summary: Commit SHA: <sha> matches the SHA from the CI staging job.
Discord only fires if staging fails; success is Actions Summary only.

Closes #2087.

felipebergamin

Two things that will bite us on the actual deploy path — left them inline.

felipebergamin · 2026-09-23T15:45:19Z

    uses: ./.github/workflows/deploy-staging.yaml
    secrets: inherit
+    with:
+      image_tag: ${{ github.sha }}


This is going to race with Publish GHCR Images. Both still start on their own when something lands on main, and CI never waits for the images to actually be in the registry.

That used to be fine because staging built on the box. Now it pulls this SHA with --no-build, so if tests finish before all three images are pushed, deploy just fails. A retry/wait on pull would keep build and deploy as separate workflows and still make this reliable.

Good catch — addressed in 7c53d27.

Staging deploy now retries docker compose pull (up to ~10 min) so it waits for the SHA tags in GHCR instead of racing Publish. Build and Deploy stay separate.

felipebergamin · 2026-09-23T15:45:19Z

            cp ~/.env-staging dashboard-staging/.env &&
            cd dashboard-staging &&
-            git checkout ${GITHUB_SHA} &&
+            git checkout ${IMAGE_TAG} &&


This clone only has main, so git checkout will fail for a SHA that isn't on that branch. That's the hotfix path from #2087 — publish from a branch, then deploy that SHA. Actions checkout can fetch it; the server can't. Same thing in the production workflow.

Fetching the SHA first (git fetch origin "$IMAGE_TAG" then checkout) would cover that and also save us from cloning the whole history on every deploy.

Also fixed in 7c53d27 (staging + production).

We now git fetch origin "$IMAGE_TAG" before checkout, and clone with --depth 1 so hotfix SHAs not on main still resolve without pulling full history.

felipebergamin · 2026-09-25T14:15:54Z

    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4


since we are here I think we can update the action version.

Suggested change

- uses: actions/checkout@v4

- uses: actions/checkout@v7

Done in efdad26. Publish now uses actions/checkout@v7.

felipebergamin · 2026-09-25T14:19:19Z

+  resolve-ref:
+    if: github.repository == 'kernelci/dashboard' || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    outputs:
+      sha: ${{ steps.git.outputs.sha }}
+    steps:
+      - name: Resolve commit SHA
+        id: git
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REF: ${{ github.event.inputs.ref || github.sha }}
+        run: |
+          set -euo pipefail
+          sha=$(gh api "repos/${GITHUB_REPOSITORY}/commits/${REF}" --jq .sha)
+          echo "sha=${sha}" >> "$GITHUB_OUTPUT"
+


do we need this? I'm not sure but I think we can pass the ref directly to the checkout action.

Done in efdad26. We check out inputs.ref (or github.sha on push) and take the image tag from git rev-parse HEAD, so a branch or tag still becomes one commit SHA. The job stays so a bad ref fails once before the three image builds.

felipebergamin · 2026-09-25T14:36:22Z

+    inputs:
+      ref:
+        description: Git ref to build (branch, tag, or commit SHA)
+        required: true
+        type: string



Maybe we can use the github selector instead of allowing an open input. The ref selected there will be in ${{ github.ref }} (e.g. ref/tags/release-tag) and ${{ github.ref_name }} (e.g. release-tag). The commit sha will be ${{ github.sha }}

Dont we loose the power to go to any commit with that? I believe it only brings tags and branches.

Yes, but I see this as a good thing to enforce having a branch or a tag to be deployed.

Otherwise we can select any commit and after some time would be hard to tell why that commit is on prod.

Done. The free-text ref is gone. A build comes from a branch or tag, and the image tag is the full github.sha.

felipebergamin · 2026-09-25T14:41:54Z

    steps:
      - uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.resolve-ref.outputs.sha }}


if I'm not mistaken we can use ${{ github.sha }}.
maybe we don't even need to provide this arg, it defaults to github.ref

https://github.com/actions/checkout#usage

The image we want is inputs.ref, which can be a different SHA than the one from github.sha.

not if we rely on "Use workflow from" selector as I said in my other comment

github.sha will carry the resolved sha for the selected ref, and for workflows dispatched by pushes it will carry the sha for the commit that triggered the workflow.

Done. Checkout follows the selected branch or tag, and the image tag is the full github.sha.

felipebergamin · 2026-09-25T14:48:29Z

-          tags: |
-            ${{ steps.meta.outputs.prefix }}/dashboard-frontend:latest
-            ${{ steps.meta.outputs.prefix }}/dashboard-frontend:${{ github.sha }}
+          tags: ${{ steps.meta.outputs.prefix }}/dashboard-frontend:${{ needs.resolve-ref.outputs.sha }}${{ github.event_name == 'push' && format(',{0}/dashboard-frontend:latest', steps.meta.outputs.prefix) || '' }}


I think we can use a short sha to make easier to read the image tags.

- name: Get short SHA run: echo "SHORT_SHA=${GITHUB_SHA::7}" >> "$GITHUB_ENV" - name: Show short SHA run: echo "${{ env.SHORT_SHA }}"

I might be too much conservative (especially because we are a small repo), but I still prefer long hashes due to reduced collision chance.
Do you consider too problematic to keep long hash here?

I just think that a short sha would be easier to read. I don't know about the chances of a collision. But consider this an optional.

Keeping the full SHA.

felipebergamin · 2026-09-25T14:50:33Z

+      image_tag:
+        description: GHCR image tag (Git commit SHA) to deploy
        required: true
-        default: 'main'
+        type: string


I'd replace this by github.ref_name

github.ref_name is the branch we are running the workflow from image_tag is the one we are using to build the image.

My comments are considering a change to use the github ref selector.

If we select a tag there, github.ref_name will carry the tag name.

Done. No open input. The selected branch or tag is deployed as its full github.sha.

felipebergamin · 2026-09-25T14:51:19Z

      - name: Checkout code
        uses: actions/checkout@v4
        with:
+          ref: ${{ github.event.inputs.image_tag }}


Same as above: checkout image_tag, not github.ref_name.

felipebergamin · 2026-09-25T14:57:49Z

          SSH_HOST: ${{ secrets.STAGING_HOST }}
          SSH_KEY: ${{ secrets.STAGING_KEY }}
-          DASHBOARD_VERSION: ${{ github.event.inputs.tag }}
+          IMAGE_TAG: ${{ github.event.inputs.image_tag }}


again, I'd use github.(ref | ref_name | sha) instead of relying in the input exactly as provided

The host pulls image_tag. github.sha and github.ref_name are the workflow run, not that image.

My comment consider that we would drop the open input and use the github ref selector.

If we select a tag there, github.ref_name will carry the tag name. It is basically the same that typing the tag name in the input.

Done. The open input is gone. Deploy uses the full github.sha of the selected branch or tag.

felipebergamin · 2026-09-25T14:58:40Z

+    inputs:
+      image_tag:
+        description: GHCR image tag (Git commit SHA) to deploy
+        required: true
+        type: string


Same as production. Staging can deploy a SHA other than the branch that started the workflow.

* Publish GHCR Images: optional ref on manual dispatch; tag images with resolved commit SHA; push `latest` only on push to main * Staging: required `image_tag`; pull GHCR images with `--no-build`; manual `workflow_dispatch`; CI passes `github.sha` after tests * Production: required `image_tag` (replaces unused `tag` default); export `IMAGE_TAG` on deploy; checkout deploy SHA on server * Align staging `docker-compose.yml` image paths with `IMAGE_OWNER` / `IMAGE_REPOSITORY` from `.env.example` * Update DEPLOYMENT.md and README for SHA-based deploy flow Closes kernelci#2087

Retry staging pulls until SHA tags exist so CI does not race Publish. Fetch the image_tag on the host before checkout so hotfix commits work.

Checkout the requested ref and tag images with git rev-parse HEAD. Bump actions/checkout to v7. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

* Drop free-text ref and image_tag inputs; use the GitHub branch/tag selector * Tag and deploy the full github.sha; keep latest only for pushes to main * Remove resolve-ref; checkout already matches github.sha * On the host, fetch the selected ref and check out that SHA * Document the selector in DEPLOYMENT.md and README

felipebergamin · 2026-10-01T15:14:59Z

          NEW_MIGRATIONS=$(git diff --name-only HEAD~1 HEAD -- '**/migrations/*.py' \
            | grep -v '__init__.py' || true)


can we rely on this check?
if we dispatch this action in a branch, it will check only the last commit, migrations added in previous commits won't be notified.

I think HEAD~1 only works when we use squash and merge to main.

this would be tricky to implement in a more generic way, but we can include a proper comment to ensure that this requires the squash and merge.

felipebergamin · 2026-10-01T15:29:16Z

+            git fetch --depth 1 origin "${GIT_REF}"
+            git checkout --detach "${IMAGE_TAG}"


nit: There is a small chance that the ref could be updated between the start of the action and the point when the KCI server tries to fetch it. This could cause github.sha to differ from the actual SHA of the ref in the repository.

I don't think this is very likely to happen, but it's something we should be aware of.

Good catch. The host now fetches github.sha directly, so a push to the branch after the run starts cannot change the checkout.

git fetch --depth 1 origin "${IMAGE_TAG}" git checkout --detach "${IMAGE_TAG}"

IMAGE_TAG is still the SHA of the branch or tag selected in "Use workflow from".

The Discord notice diffs HEAD~1, so it only covers the whole pull request while main is updated by one squash commit. Assisted-by: Grok 4.7 <noreply@x.ai> Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

tales-aparecida · 2026-10-08T11:29:57Z

I think I would be more comfortable with this PR if staging was in a different server.

alanpeixinho · 2026-10-09T19:59:09Z

Yeah, I agree. Staging on its own server would be a good change. I believe #1516 already has some comments on this.

What we want from this PR is the rollback path from #2087. If a deploy goes bad, we redeploy an older commit SHA instead of being stuck on latest.

Do you see something we could still do in this PR to make the shared host less of a problem? Should we also hold this merge until #1516 is done? @nuclearcat might want to share his take on this too.

alanpeixinho marked this pull request as ready for review September 22, 2026 21:12

alanpeixinho force-pushed the chore/deploy-sha-images branch from e9a1955 to bfb9c73 Compare September 23, 2026 12:58

alanpeixinho mentioned this pull request Sep 23, 2026

chore/deployment ingester docs #2122

Merged

felipebergamin reviewed Sep 23, 2026

View reviewed changes

felipebergamin reviewed Sep 25, 2026

View reviewed changes

alanpeixinho added 5 commits October 1, 2026 10:13

ci(deploy): fail publish early on an unresolvable ref

fa32e1d

ci(deploy): wait for GHCR images and fetch deploy SHA

598e026

Retry staging pulls until SHA tags exist so CI does not race Publish. Fetch the image_tag on the host before checkout so hotfix commits work.

ci(deploy): resolve publish SHA from the checked-out ref

0d306cd

Checkout the requested ref and tag images with git rev-parse HEAD. Bump actions/checkout to v7. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho force-pushed the chore/deploy-sha-images branch from 8146d18 to dda2c1b Compare October 1, 2026 13:14

alanpeixinho requested a review from felipebergamin October 1, 2026 13:15

felipebergamin reviewed Oct 1, 2026

View reviewed changes

docs(deploy): note staging migration check assumes squash merges

50e42c4

The Discord notice diffs HEAD~1, so it only covers the whole pull request while main is updated by one squash commit. Assisted-by: Grok 4.7 <noreply@x.ai> Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

felipebergamin requested a review from tales-aparecida October 2, 2026 17:03

fixup! chore(deploy): build and deploy from the selected branch or tag

d500d4b

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho requested a review from felipebergamin October 7, 2026 13:23

		NEW_MIGRATIONS=$(git diff --name-only HEAD~1 HEAD -- '*/migrations/.py' \
		\| grep -v '__init__.py' \|\| true)

		git fetch --depth 1 origin "${GIT_REF}"
		git checkout --detach "${IMAGE_TAG}"

Conversation

alanpeixinho commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it is

How to test

1. Publish (no staging, no prod)

2. Negative staging (unpublished SHA)

3. Manual staging (live staging)

4. After merge to main (live staging; existing CI behavior)

Uh oh!

felipebergamin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felipebergamin Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

alanpeixinho commented Sep 21, 2026 •

edited

Loading

4. After merge to `main` (live staging; existing CI behavior)

felipebergamin Sep 28, 2026 •

edited

Loading