Skip to content

Pin the kyverno pod-security policies to a commit and put it in the cached file name - #762

Merged
cert-manager-prow[bot] merged 2 commits into
cert-manager:mainfrom
wallrj-cyberark:pin-kyverno-pod-security-policies
Sep 30, 2026
Merged

cert-manager-prow[bot] merged 2 commits into
cert-manager:mainfrom
wallrj-cyberark:pin-kyverno-pod-security-policies

Conversation

@wallrj-cyberark

@wallrj-cyberark wallrj-cyberark commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

make verify-pod-security-standards fails in every project that vendors the helm module. It has been failing on every fresh checkout since 04:24 UTC on 2026-09-29, whatever the PR changes:

Error: evalsymlink failure on '/tmp/kustomize-.../pod-security/enforce' : lstat /tmp/kustomize-.../pod-security: no such file or directory

The cause is kyverno/policies#1544, which deleted pod-security/ from the main branch of kyverno/policies. kyverno/policies#1543 tracks the move to the CEL policies in pod-security-vpol/, which replaced it. helm.mk built https://github.com/kyverno/policies/pod-security/enforce without a ref, so it always read main.

No open-source project has failed yet. None of the eight repositories that vendor the helm module has run a verify job since the deletion, so the first PR push to any of them will fail. Please merge this so verify goes green again. Downstream repositories will pick it up with their next make upgrade-klone.

What was wrong

The deletion exposed three problems, and this PR fixes all three.

  1. The policies are gone from main. Every fresh checkout fails verify until the URL points somewhere that still has them.
  2. Verify was not reproducible. The URL had no ref, so every CI run fetched whatever was on main at that moment. A change upstream could break or alter every downstream verify job without a diff in the consuming repository. That is exactly what happened.
  3. The cached policy file was never rebuilt. _bin/scratch/kyverno/pod-security-policy.yaml had no prerequisites, so an existing checkout kept its first copy for ever. CI (fresh checkout) and developers (warm _bin) were checking against different policies, and no change to the URL could reach a developer's checkout without them deleting the file by hand. This is also why the deletion went unnoticed: a copy cached on 2026-09-23 kept passing for six days after main broke.

What this PR changes

  • Fixes 1 and 2: pins the kustomize URL to commit ef9843f0, the tip of the release-1.19 branch, one of the last commits that still has pod-security/. The branch name is recorded in a comment but not used as the ref, because the branch can still receive pushes. The branch is not a compatibility boundary: pod-security/ is identical on every release branch from release-1.12 to release-1.19, and each policy declares kyverno 1.6.0 as its minimum. So this does not need bumping with the kyverno tool.
  • Fixes 3: puts the commit in a kyverno_policies_version variable and in the name of the cached file, pod-security-policy-<commit>.yaml. This is what the module already does for the chart archive. Changing the version now builds a new file in every checkout, with no stamp rule and no dependency on another module.
  • Cites [Tracking] Create missing policies.kyverno.io/v1 equivalents for legacy policy samples kyverno/policies#1543 and names pod-security-vpol/ as the successor in the comment, for whoever bumps this next.

CI evidence

cert-manager/approver-policy#1043 points klone at this branch. Its pull-cert-manager-approver-policy-verify job passed on a fresh checkout: build log. It fetched the policies with the new pin and applied 19 rules to the Deployment, all Pass. That PR is held and will be closed once this merges.

Follow-up

A better fix is to commit the built policy file to this module, as cert-manager/cert-manager already does in make/config/kyverno/policy.yaml. That removes the network fetch from make verify, lets Renovate manage policy updates as ordinary PRs, and lets us patch the policies. We should do that separately, once the CEL policies are in the kyverno version we pin.

How I tested it: the pinned commit builds the same 17 policies as main did before the deletion, and the cache follows the pin

I tested in a scratch project built with bootstrap.sh plus the tools and helm modules from this branch and a minimal chart with one Deployment.

Ref kustomize build result
Unpinned (main) Fails with the evalsymlink error above
ef9843f08d25b3555fe69616f8612c9f915af5d4 (tip of release-1.19) 17 ClusterPolicy objects
d12f1b56ff051b2b5e59ffd7d3d563f8add30547 (last commit on main before the deletion) 17 ClusterPolicy objects
release-1.19 17 ClusterPolicy objects

All three built files have the same SHA-256, and match a copy cached on 2026-09-23 from the unpinned URL. So this changes nothing about which policies charts are checked against. Note that kustomize does not accept a short SHA as a ref.

The versioned file name behaves as intended:

  • A second make with the same version leaves the file untouched.
  • make kyverno_policies_version=d12f1b56... verify-pod-security-standards builds a second file, pod-security-policy-d12f1b56....yaml, beside the first.
  • make verify-pod-security-standards then passes: 19 rules applied to the Deployment, all Pass.

[Claude Fable 5.1]

kyverno/policies#1544 deleted pod-security/ from its main branch on
2026-09-29. verify-pod-security-standards builds that path without a
ref, so it now fails in every project that uses the helm module:

    Error: evalsymlink failure on '/tmp/kustomize-.../pod-security/enforce' :
    lstat /tmp/kustomize-.../pod-security: no such file or directory

- Build from the release-1.19 branch, which matches the kyverno v1.19
  tool version we pin and still has the policies.
- The output is byte-for-byte identical to what main produced before
  the deletion: the same 17 ClusterPolicies.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
@cert-manager-prow cert-manager-prow Bot added the dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. label Sep 29, 2026
Comment thread modules/helm/helm.mk Outdated
@cert-manager-prow cert-manager-prow Bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Sep 29, 2026
@wallrj
wallrj requested a balanced review from Copilot September 29, 2026 15:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cert-manager-prow cert-manager-prow Bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 29, 2026
Comment thread modules/helm/helm.mk Outdated
@wallrj
wallrj force-pushed the pin-kyverno-pod-security-policies branch from 0bed965 to 34717bd Compare September 30, 2026 08:38
@wallrj wallrj changed the title Pin kyverno pod-security policies to release-1.19 Pin the kyverno pod-security policies to a commit and rebuild the cache when it changes Sep 30, 2026
Pin the commit at the tip of release-1.19 rather than the branch name.
The branch can still receive pushes, so pinning it would leave every
downstream verify job open to the same silent change that broke it
when pod-security/ was deleted from main.

The release branch is not a compatibility boundary. The pod-security/
bundle is identical on every release branch from release-1.12 to
release-1.19, and each policy declares kyverno 1.6.0 as its minimum
version. The branches are snapshots for the kyverno.io website. The
comment says so, so nobody bumps this along with the kyverno tool.

Put the commit in kyverno_policies_version and in the name of the
cached policy file, as the module already does for the chart archive.
Bumping the version now builds a new file instead of reusing a stale
copy in _bin/scratch.

Cite kyverno/policies#1543 and name pod-security-vpol/ as the
successor in the comment, so whoever bumps this next knows where the
policies went.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard@the-moon.net>
@wallrj
wallrj force-pushed the pin-kyverno-pod-security-policies branch from 34717bd to ad9e064 Compare September 30, 2026 11:35
@wallrj wallrj changed the title Pin the kyverno pod-security policies to a commit and rebuild the cache when it changes Pin the kyverno pod-security policies to a commit and put it in the cached file name Sep 30, 2026
wallrj added a commit to wallrj/approver-policy that referenced this pull request Sep 30, 2026
Do not merge. This demonstrates that verify-pod-security-standards
passes again with cert-manager/makefile-modules#762, which pins the
kyverno pod-security policies to a commit after kyverno/policies
deleted them from main.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard@the-moon.net>
@wallrj
wallrj requested a review from inteon September 30, 2026 11:39
@inteon

inteon commented Sep 30, 2026

Copy link
Copy Markdown
Member

/approve
/lgtm

@cert-manager-prow cert-manager-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 30, 2026
@cert-manager-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cert-manager-prow cert-manager-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 30, 2026
@cert-manager-prow
cert-manager-prow Bot merged commit 9096105 into cert-manager:main Sep 30, 2026
5 checks passed
@wallrj-cyberark
wallrj-cyberark deleted the pin-kyverno-pod-security-policies branch September 30, 2026 12:11
wallrj-cyberark added a commit to jetstack/jetstack-secure that referenced this pull request Oct 6, 2026
- `make verify` fails because kyverno/policies#1544 deleted the
  pod-security directory that verify-pod-security-standards builds from.
- cert-manager/makefile-modules#762 pins the policies to a commit; this
  runs `make upgrade-klone` to pick it up, along with routine tool bumps.

Signed-off-by: Richard Wall <richard.wall@cyberark.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants