Skip to content

ci(coderabbit): configure the CodeRabbit review for this repository - #1089

Open
tobyhede wants to merge 1 commit into
mainfrom
ci/coderabbit-config
Open

tobyhede wants to merge 1 commit into
mainfrom
ci/coderabbit-config

Conversation

@tobyhede

@tobyhede tobyhede commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Summary

CodeRabbit is an AI code reviewer that runs as a GitHub App. It is already installed here and reviews every pull request to main, but with its default settings. This adds .coderabbit.yaml so the review knows this repository's rules: the encryption library must not leak plaintext, customer-facing skills must stay accurate, and published changes need a changeset. It also cuts how often CodeRabbit reviews a pull request.

This is the CodeRabbit replacement for the four-lens Claude review in #1005. That workflow ran on GitHub Actions runners and waited five minutes per lens before every review. CodeRabbit runs on its own servers and uses no runner time.

Changes

Review content

  • Security instructions for specific paths: EQL SQL (packages/eql/**/*.sql), the Rust crates, and skills/**/SKILL.md, which ship to customers in the stash package.
  • Four pre-merge checks, all in warning mode so none blocks a merge: changeset present, customer skills updated with the API or CLI they describe, AGENTS.md layout and SECURITY.md updated when a package is added or removed, and no Linear IDs or workflows under a package. Each shows as pass or fail in CodeRabbit's walkthrough comment.
  • Scanners: zizmor (GitHub Actions security, not run in CI today), actionlint, ShellCheck, Betterleaks and SkillSpector (a static scan of changed SKILL.md files). Scanners CI already runs (osv-scanner, golangci-lint) are off, as are ESLint and Oxc because the repo uses Biome. SQLFluff, markdownlint and LanguageTool are off as noise.

Fewer reviews and less noise

  • Draft PRs get no review. Marking a PR ready starts the first one.
  • Automatic reviews pause after three reviewed commits. @coderabbitai review asks for another.
  • Dependabot, github-actions[bot], changeset-bot[bot], Version Packages and WIP PRs are skipped.
  • docs/plans/, snapshots and changelogs are excluded.
  • CodeRabbit no longer rewrites PR descriptions. Its summary goes in its own walkthrough comment, and poems, fortunes, diagrams, suggested labels and suggested reviewers are off.
  • PRs whose base is not main (stacked PRs) are now reviewed too.

What is not in this file, on purpose
CodeRabbit reviews a pull request using the .coderabbit.yaml on that PR's own branch, so a PR can change its own review. The settings that must not be changeable that way are set in CodeRabbit's organisation Global Overrides, which win over this file. The file header lists them:

  • the review stays advisory and never requests changes;
  • nothing commits to a branch or opens a PR (autofix, CI fixer, merge-conflict and test/docstring generation are off);
  • the security instructions for .github/** and the TypeScript package sources;
  • no chat with people outside the org, no web search, and learnings kept to this repository.

Verification

  • cr config validate .coderabbit.yaml: valid against the current CodeRabbit schema.
  • Validated against schema.v2.json with jsonschema: 0 errors. The schema accepts unknown keys, so a separate check confirmed every key in the file exists in the schema.
  • Global overrides confirmed live: @coderabbitai configuration on ci(claude-review): run Anthropic's code-review plugin, on open or on request #1005 reports all 11 override keys as Source: Global overrides.
  • Not verified yet: a review using this file. This PR is the first. See the review notes.

Related

Refs #1005, #997

Review notes

  • This PR is reviewed with its own config. CodeRabbit reads .coderabbit.yaml from the PR branch, so its review here is the first live test. It is a draft, and drafts are skipped, so I'll trigger the review with @coderabbitai review. Check that the walkthrough's run configuration names the repository file rather than "Organization UI" or "defaults", and that the review details list the guideline files used.
  • Guideline files. CodeRabbit automatically applies every AGENTS.md and CLAUDE.md, and every skills/*/SKILL.md, as review criteria. The skills are customer guidance, not review rules. They can only be excluded in the CodeRabbit UI, not in this file. That is a separate step.
  • No CODEOWNERS change. The ruleset on main does not require code-owner review, and * is already owned by @cipherstash/developers, so a .coderabbit.yaml entry would change nothing.
  • review_details: true is for the rollout. Turn it off once the guideline and tool choices are settled.
  • No changeset: nothing published changes.

CodeRabbit has been reviewing pull requests here with its defaults. This
gives it the repository's own rules: per-path security instructions for the
EQL SQL, the Rust crates and the customer skills; four advisory pre-merge
checks for the rules no CI job enforces (changeset, customer skills, layout
docs, public text hygiene); scanners CI does not already run; and fewer
reviews per pull request (no drafts, pause after three reviewed commits,
skip bot and Version Packages PRs).

The settings a pull request must not be able to change for its own review
live in the organisation's Global Overrides, not in this file, because
CodeRabbit reads this file from the PR's own branch. The header lists them.
@changeset-bot

changeset-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 11662b0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@tobyhede

tobyhede commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: cipherstash/stack/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 9a42034f-9f99-40f3-be38-7f6445b9b97e
📥 Commits

Reviewing files that changed from the base of the PR and between b6e6100 and 11662b0.

📒 Files selected for processing (1)
  • .coderabbit.yaml

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (6)
  • GitHub Check: ci-required
  • GitHub Check: Run Tests (Node 24)
  • GitHub Check: Run Tests (Bun)
  • GitHub Check: Run Tests (Node 22)
  • GitHub Check: Run E2E Tests
  • GitHub Check: Run WASM E2E Tests (Deno)

📝 Walkthrough

Walkthrough

The .coderabbit.yaml file configures review language, tone, profile, and automatic review rules. It also defines review exclusions, path-specific guidance, pre-merge checks, and enabled tools.

Changes

Review configuration

Layer / File(s) Summary
Review behaviour and automation
.coderabbit.yaml
Sets Australian English, a plain and direct tone, the review profile and feature settings, and automatic review rules.
Review scope and checks
.coderabbit.yaml
Defines review exclusions and path-specific guidance. Sets pre-merge checks and configures enabled and disabled tools.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 11662

No actionable issue is established; the configuration is ready to merge after normal checks.

🚥 Pre-merge checks | ✅ 6
✅ Passed checks (6 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Changeset ✅ Passed The pull request changes only .coderabbit.yaml. It does not change any listed source or skills/ path, and it adds no .changeset/*.md or .changeset/*.md.deferred file.
Customer Skills ✅ Passed PASS. The reviewed diff adds only .coderabbit.yaml. It changes no CLI command registry, CLI flags, public exports, or matching skill files. The Customer skills condition is not triggered.
Layout Docs ✅ Passed The PR adds only .coderabbit.yaml. It does not add, remove or rename a package, example, skill or subpath export, so the check's documentation-update condition does not apply.
Public Text Hygiene ✅ Passed The PR adds only .coderabbit.yaml. No added line is under skills/, and no workflow file is added.
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@tobyhede
tobyhede marked this pull request as ready for review October 6, 2026 23:16
@tobyhede
tobyhede requested a review from a team as a code owner October 6, 2026 23:16
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T23:20:25.166930Z 11662b0 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 11662b082e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .coderabbit.yaml
Comment on lines +24 to +25
high_level_summary: false # PR bodies are written by hand; don't rewrite them
high_level_summary_in_walkthrough: true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the high-level summary enabled

Setting high_level_summary to false disables the summary that line 25 attempts to place in the walkthrough, so normal PRs will receive no high-level summary at all. CodeRabbit’s summary documentation states that this setting stops summary generation, while high_level_summary_in_walkthrough: true alone moves an enabled summary out of the PR description; keep the former enabled to get the intended walkthrough-only summary.

Useful? React with 👍 / 👎.

Comment thread .coderabbit.yaml
custom_checks:
- name: Changeset
mode: warning
instructions: "Fail if files under languages/typescript/packages/*/src, packages/eql/packages/eql, or skills/ changed and no .changeset/*.md file was added. A skills-only change needs a 'stash' patch changeset. Fail if any .changeset file ends in .md.deferred. Pass for changes that only touch tests, scripts/, docs/ or .github/."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Cover native FFI changes in the changeset check

A PR that changes only languages/typescript/packages/protect-ffi/crates/**/*.rs can alter the published native module without touching any path listed here, so this check passes without requiring a changeset. Include the FFI Cargo sources and other release-relevant non-src package surfaces in the trigger; the repository explicitly requires normal changesets for the seven protect-ffi packages.

AGENTS.md reference: AGENTS.md:L165-L167

Useful? React with 👍 / 👎.

Comment thread .coderabbit.yaml
instructions: "Fail if files under languages/typescript/packages/*/src, packages/eql/packages/eql, or skills/ changed and no .changeset/*.md file was added. A skills-only change needs a 'stash' patch changeset. Fail if any .changeset file ends in .md.deferred. Pass for changes that only touch tests, scripts/, docs/ or .github/."
- name: Customer skills
mode: warning
instructions: "If the CLI command registry, CLI flags, or a public export of @cipherstash/stack, stack-drizzle, stack-supabase, stack-prisma or the dynamodb entry changed, fail unless the matching skills/*/SKILL.md (per the table in AGENTS.md 'Agent Skills') also changed, or the PR description explains why no skill is affected."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the skills check to every mapped public surface

This trigger covers CLI changes and export-list changes in a few packages, but misses mapped changes such as languages/typescript/packages/migrate lifecycle behavior, EQL operators, deployment workflows, auth strategy behavior, and ordinary API behavior changes that leave exports unchanged. Those PRs can therefore pass the dedicated check while shipping stale customer instructions; trigger it for every public API or user-facing workflow covered by the AGENTS skill map.

AGENTS.md reference: AGENTS.md:L780-L782

Useful? React with 👍 / 👎.

@auxesis
auxesis self-requested a review October 7, 2026 00:17

@auxesis auxesis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the success and failure metrics for this experiment?

Or to put another way:

  • What is the hypothesis we are testing with @coderabbitai?
  • Why are we doing that?
  • How do we measure the impact of this @coderabbitai configuration?
  • What measurements falsify the hypothesis?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants