Add OneBranch pipeline for PyPI publishing

[Shiva Sathwik Basavapatna Ramaprasad (shivasathwikb)]

Summary

• add a OneBranch official pipeline for publishing the ccf Python package
• manually build and validate the wheel from a selected ccf-* release tag
• publish the validated wheel to PyPI through ESRP
• keep the existing GitHub Actions workflows unchanged during non-production validation
• preserve the existing GitHub Release review and publication gate

Pipeline behavior

The OneBranch pipeline uses:

trigger: none
pr: none

It is manually queued from a ccf-* tag after the corresponding GitHub Release has been reviewed and published.

The build stage:

• verifies that the selected ref is a ccf-* tag
• requires Python 3.12 or newer, matching the package requirement
• reuses scripts/extract-release-notes.py --target-git-version to validate the tag, release notes, and package version
• fails if the version already exists on PyPI
• cleans python/dist before building
• builds exactly one ccf wheel
• validates the wheel project name, version, and metadata structure
• installs the wheel in an isolated temporary environment
• verifies that key ccf modules can be imported
• copies the validated wheel into the OneBranch output

The release stage:

• downloads the build artifact
• searches recursively for the wheel and flattens the staging directory
• publishes the validated wheel through ESRP

Wheel validation

Wheel metadata validation is implemented in scripts/validate_python_package.py.

The validator verifies that:

• the wheel contains exactly one .dist-info/METADATA file
• the project name is ccf
• the wheel version matches the expected release version

Unit tests cover:

• accepting matching metadata
• rejecting an incorrect project name
• rejecting an incorrect version
• rejecting multiple metadata files

External prerequisites

• an Azure DevOps pipeline definition using /.pipelines/pypi.official.yml
• the ccf-esrp-pypi Azure DevOps Library variable group
• a CCF ESRP service connection and publisher registration
• ESRP Key Vault and certificate configuration
• separate ESRP owners and approvers
• access to OneBranch.Pipelines/GovernedTemplates

The ccf-esrp-pypi variable group is expected to provide:

• ESRP_SERVICE_CONNECTION
• ESRP_KEY_VAULT_NAME
• ESRP_SIGN_CERT_NAME
• ESRP_AUTH_CERT_NAME
• ESRP_CLIENT_ID
• ESRP_OWNERS
• ESRP_APPROVERS
• ESRP_MAIN_PUBLISHER
• ESRP_DOMAIN_TENANT_ID

Validation

• parsed the OneBranch YAML
• validated shell syntax with bash -n
• ran ShellCheck on the pipeline script
• ran Prettier on the YAML and Markdown files
• ran Black and Ruff on the new Python files
• ran the four wheel-validator tests successfully
• verified that non-ccf-* refs are rejected
• ran ASCII checks
• ran copyright checks
• ran git diff --check

Test result:

4 passed

Validation limitations

• OneBranch template expansion has not yet been validated in Azure DevOps
• the ccf-esrp-pypi variable group and ESRP resources still need to be confirmed or provisioned
• ESRP Test/PPE and production publication have not yet been exercised

Rollout

The intended release sequence is:

1. Create and push the ccf-* release tag.
2. Build and review the existing GitHub Release.
3. Publish the GitHub Release.
4. Manually queue the OneBranch pipeline using the same tag.
5. Publish the wheel through ESRP.

The existing GitHub Actions release and PyPI workflows remain unchanged during non-production validation.

Before enabling OneBranch for production publication, the existing GitHub Actions PyPI publisher must be disabled so that both publishers do not attempt to publish the same version.

If publication is only partially successful, the incomplete version must not be retried by moving or recreating the tag. A new package version should be created instead.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 · 4

Open (5)

• The wheel is produced in the build job output and then pulled into the release job via the… · New
• tomllib is only available in Python 3.11+. If the pipeline image provides an older python3,… · New
• The tag-to-version mapping only replaces the first '-' with '.', which can produce incorrect… · New
• This assumes python/dist contains only artifacts produced by the current run. If the workspace is… · New
• Using a fixed venv path under /tmp/ccf-wheel-test can collide with prior runs on a reused agent,… · New

What changed in this PR

Adds a OneBranch (Azure DevOps) official pipeline to build, validate, and publish the ccf Python wheel to PyPI via ESRP when manually queued from a ccf-* tag.

Changes:

• Introduces a build/validation Bash script that verifies tag/version alignment, checks PyPI for existing versions, builds a single wheel, validates metadata, and smoke-installs/imports it.
• Adds a OneBranch official pipeline YAML with build + ESRP publish stages.
• Documents required Azure DevOps variable group inputs and pipeline behavior.

File	Description
.pipelines/​scripts/​build-pypi-package.sh	Adds the build + validation logic that runs in the build stage and produces the wheel artifact.
.pipelines/​pypi.official.yml	Defines the OneBranch build and ESRP-based release stages for publishing to PyPI.
.pipelines/​README.md	Documents how to run/configure the new OneBranch PyPI release pipeline and prerequisites.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Amaury Chamayou (achamayou)]

Shiva Sathwik Basavapatna Ramaprasad (@shivasathwikb) Andrea Piccione (@andpiccione) I think this choice is deliberate, and probably right, but I want to take a moment to make sure we are all on the same page: the current release mechanism is that release.yml is triggered on a tag (ccf-*) and produces all release artefacts together (rpm, tgz/npm and wheel/pypi). A Draft GitHub release is created, and all the files are included in it as attachements.

A second, manual action (usually preceded by a review) publishes the Github release from its draft state (only visible to repo members) to full release. This then triggers a number of follow-up workflows, including the publication of the tgz/npm to npm (from the GH release attachements) and that of the wheel/pypi to pypi (again from attachements).

A consequence of that is that until a release is published, it is in principle possible to cut the tag multiple times, and produce several drafts. This is infrequent, but it has been used in the past, typically when an issue took place in the release workflow that required a change to fix and could not simply be solved by a re-run (the workflow is designed to be idempotent, but the CI infrastructure sometimes changes).

With this PR, because the pypi release is trigged purely on tag creation, with no further manual input, we cannot in principle do this, or if we did, we would get the rest of the artefacts built from a different commit than the Python package (I think we can agree that this is undesirable). This may not be big issue, release mishaps are rare, and one way to deal with them would be tombstone the release and skip incomplete attempts rather than retry them. If say, 7.0.23 hit a problem resulting in the Python wheel getting published and not the rpm (or vice versa), we could declare that there was no 7.0.23 and cut a 7.0.24 rather than attempt to fix 7.0.23.

There may be other possibilities I have not considered, but I think it is useful that we think about this the same way as we make this change.

[Shiva Sathwik Basavapatna Ramaprasad (shivasathwikb)]

Amaury Chamayou (@achamayou) Thanks for calling out the release semantics. The OneBranch pipeline is manual (trigger: none), not automatically triggered by tag creation. I updated the documented process so it is queued only after the corresponding GitHub Release has been reviewed and published, preserving the existing manual gate. The old GitHub PyPI publisher remains only during non-production validation and must be disabled before the OneBranch production publisher is enabled. If a partial release publishes only some artifacts, we will skip that incomplete version and cut a new version rather than move or recreate the tag.

[Copilot]

Copilot review overview

🟡 Changes recommended

Release validation and publication gating have unresolved correctness issues.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Require a published GitHub Release before production publication · New
• Ensure pinned uv is available in the requested install directory · New
• Validate source tag version matches package version · New
• Register publication validator regression tests in CI · New

Resolved since last review (5)

• The wheel is produced in the build job output and then pulled into the release job via the…
• Using a fixed venv path under /tmp/ccf-wheel-test can collide with prior runs on a reused agent,…
• This assumes python/dist contains only artifacts produced by the current run. If the workspace is…
• The tag-to-version mapping only replaces the first '-' with '.', which can produce incorrect…
• tomllib is only available in Python 3.11+. If the pipeline image provides an older python3,…

[Andrea Piccione (andpiccione)]

Do we need to rebuild the python package here if it's already attached to a GitHub release? I'm wondering if it could make more sense to just download the ccf wheel from the latest CCF release and simply upload it to PyPI, instead of rebuilding it from scratch here. The advantage of that approach would be uploading the exact same file that was covered by the release attestation.

[Amaury Chamayou (achamayou)]

That's a very good point, and if we don't, we need to issue a separate attestation for the pypi package. I think that there are two reasons to go that route: one is to split the release cycle/version numbers of the Python package, as discussed. The other is that I believe the intent in ESRP is to sign things produced in a 1ES environment, rather than produced outside, and so while this may technically work at the moment, it may be blocked eventually.

On the flip side, as you say, pulling it from the GH release would keep a single attestation.