Report analysis diagnostics in workflow runs - #124
Conversation
… them A run that exits zero is not a run that finished cleanly. Core records every degradation it survived — a language server that never started, a language nothing indexed under, naming that stopped answering — in the analysis it writes. Until now this action ignored that field, so a baseline missing a whole language was committed green and a review comment showed a diagram with nothing to say it was short. Read metadata.run_diagnostics back and put it where the reader already is: an annotation per entry on the run page, the list in the sync job summary, and the same list at the top of the review comment, above the diagram rather than under it — a caveat printed below a picture is read after the picture is believed. Entries carry their own remedy. Where nothing on the reader's side would have changed the outcome the remedy is empty, and the block links Discord instead of inventing an instruction nobody can follow. Diagnostics never fail the run: a degraded analysis is still worth having, and the point is that its reader learns it is degraded. An analysis written by an engine that predates the field, or one that never got written at all, reads as silence rather than an error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSnQMqJKj7zbdfjy7KHF9x
The same title/detail/remedy strings render in the webview, which enforces the house rule with a test, so the prose the action wraps them in should read the same way rather than switching voice between surfaces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSnQMqJKj7zbdfjy7KHF9x
CodeBoarding reviewStatus: 1 changed component See the full change in CodeBoarding. Compared against the merge base: this branch is 6 commits behind graph LR
n_GitHub_Action_Automation_Scripts["GitHub Action Automation Scripts"]
classDef added fill:#1f883d,stroke:#0b5d23,color:#ffffff;
classDef modified fill:#bf8700,stroke:#7d4e00,color:#ffffff;
classDef deleted fill:#cf222e,stroke:#82071e,color:#ffffff,stroke-dasharray:5 3;
class n_GitHub_Action_Automation_Scripts modified;
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c66dd8664
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # An analysis that finished is not an analysis that finished cleanly. The engine | ||
| # records every degradation it survived; without this the run is green and the | ||
| # committed diagram is short of a language with nothing to say so. | ||
| - name: Read analysis diagnostics |
There was a problem hiding this comment.
feat:
If this commit is merged or rebased with its current subject, Report analysis diagnostics in workflow runs is not a Conventional Commit, so release-please will skip this adopter-facing feature and will not propose the release that moves the v1 tag. Use a feat: subject for the commit and PR title so consumers actually receive the new workflow behavior.
AGENTS.md reference: AGENTS.md:L88-L93
Useful? React with 👍 / 👎.
| for entry in entries: | ||
| level = "warning" if entry.get("severity") == "degraded" else "notice" | ||
| remedy = entry.get("remedy") or f"Nothing on your side causes this; please report it: {DISCORD_URL}" | ||
| lines.append(f"::{level}::{entry.get('title', 'Analysis diagnostic')} {entry.get('detail', '')} {remedy}") |
There was a problem hiding this comment.
When an engine diagnostic contains a newline in its title, detail, or remedy, this emits multiple physical log lines, so only the first is part of the intended annotation and a later line beginning with workflow-command syntax can create a spurious annotation. Multiline language-server or model failure details are a realistic input, and tests/test_llm_contract.py::test_the_annotation_stays_one_line_however_rich_the_remedy already records the repository's one-line requirement; normalize or workflow-command-escape these fields before printing.
Useful? React with 👍 / 👎.
| continue-on-error: true | ||
| shell: bash | ||
| env: | ||
| ANALYSIS_PATH: ${{ steps.review_analyze.outputs.analysis_path }} |
There was a problem hiding this comment.
When the merge-base analysis is degraded but the subsequent head analysis completes cleanly, the rendered review still compares against an incomplete base and can report structures missing from that base as PR additions, yet this step reads only the head analysis path. Because review_analyze produces a separate base_analysis_path, its run diagnostics are not reliably represented by the head's per-run metadata; read and deduplicate diagnostics from both analyses before building the review comment.
Useful? React with 👍 / 👎.
2 participants
Summary
Add support for reading and reporting analysis diagnostics from CodeBoarding's engine. When an analysis completes but encounters degradations (e.g., language server failures, indexing issues), this change ensures those issues are surfaced to users through workflow annotations, job summaries, and review comments rather than silently publishing incomplete results.
A run that exits zero is not a run that finished cleanly. The engine records every degradation it survived into the analysis it writes; until now this action ignored that field, so a baseline missing a whole language was committed green and a review comment showed a diagram with nothing to say it was short.
Key Changes
New script:
scripts/analysis_diagnostics.pymetadata.run_diagnosticsfrom analysis.json filesNew test suite:
tests/test_analysis_diagnostics.pyUpdated
action.ymlcontinue-on-error: trueto prevent failures from blocking the workflowUpdated
scripts/action/sync-summary.shUpdated
scripts/action/build-review-comment.shUpdated
README.mdImplementation Details
Where the block goes, and why
Above the diagram in the review comment, not below it. A caveat printed under a picture is read after the picture is believed, which is the failure this whole change exists to remove. In sync mode it goes in the job summary, because a baseline that is short a language is committed and then read for weeks.
Ordering with the engine
Per
AGENTS.md, the engine is a separate repo pinned bycodeboarding==inaction.yml, so this reads a field that only exists once CodeBoarding/CodeBoarding#580 ships in a release and that pin is bumped. Until thenload_entriesfinds nothing, the step writes an empty file, and every summary and comment looks exactly as it does today. Nothing to sequence and nothing to roll back.Verification
python3 -m unittest discover -s testspasses (177 tests, 7 skipped), andblack --checkis clean. Thelintjob also runsactionlintandpre-commit, neither of which I could run in this environment; theaction.ymladditions copy the shape of the steps beside them.Companion changes
One change across three repositories, all on the branch
claude/codeboarding-webview-errors-x3e0sw:metadata.run_diagnostics.CodeBoarding-webview#127 banners it above the diagram and adds a one-press "Send diagnostics report" path.https://claude.ai/code/session_01BSnQMqJKj7zbdfjy7KHF9x