feat: add stability runner

[kgyrtkirk]

I wanted to add this quite a while ago when I was digging into some issues...it was removed from that PR; ever since then I enhanced it and was using it in a fork - but today I polished it a bit ; added docs/etc...it will be much more friendlier as it will be possible to trigger it from a PR review as well...

docs generated by claude:

Description

🔁 Stability Runner — repeat chosen tests 64× (8 shards × 8 runs) → reproduce flaky test / prove fix.
No surefire reruns; failures don't stop the loop → honest counts.

STABILITY_MAVEN_ARGS=-pl embedded-tests -Dtest=KafkaIndexFaultToleranceTest

🚀 Triggers

• 💬 PR review by committer (MEMBER/OWNER) containing the line. Same args → replaces run; different → side by side.
• 🌿 Push to stability/** — args from commit message.
• 🖐️ Manual — workflow_dispatch on stability.yml.

🤖 Hint bot

stability-hint.yml: flak in PR title/body/comment or Flaky Test label → posts doc link, once per PR.
pull_request_target only for comment permission on fork PRs; no PR code checked out.

🛡️ Safety

• Args land in a shell line → check-shell-args (shlex) gates them: pass only if identical to their shell-safe re-quoted form, else fail showing that form.
• Review trigger gated on author association.

📦 Results

Per-shard test report check + artifact stability-results/run<N>/ (surefire reports, task logs).

📖 Docs

dev/stability-runner.md

[kgyrtkirk]

STABILITY_MAVEN_ARGS=-pl processing -Dtest=IntervalsTest

[FrankChen021]

🟡 Changes recommended

Do not merge until unmatched test selectors are prevented from producing green runs and the documented per-shard report checks are actually emitted. Otherwise the runner can report success without executing the requested test, while hiding the promised report surface.

Reviewed 9 of 9 changed files; findings in 2 files.

Validation: git diff --check 3e9365196f9893ff9b1cefcfbcb57d49e4080b4a 1d8e7db7ae419f5e759a21b715080742915e3dbb passed. No builds or tests were run.

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

---

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

[FrankChen021]

[P1] Reject stability runs that execute zero tests

Finding: Defect: The runner sets -Dsurefire.failIfNoSpecifiedTests=false and has no post-run assertion that any test executed. Trigger: a typo or stale class/method in -Dtest (or another selector matching nothing) lets every Maven test invocation succeed without producing a JUnit report, so the worker skips report publication. Impact: all eight iterations can finish green without exercising the requested test.

Suggestion: Track executed tests or generated results and fail when the complete run executes zero tests, while retaining the reactor-friendly setting only where needed.

[FrankChen021]

[P2] Publish the documented stability report checks

Finding: Defect: Each new shard delegates reporting to worker.yml, whose JUnit step is hard-coded with annotate_only: true. Trigger: any stability run that reaches report publication; that mode suppresses Check Run creation even though check_name is set. Impact: the documented Test Report - stability-[<id>-]shard-<N> checks are never emitted, leaving only annotations, artifacts, and job status.

Suggestion: Add a worker switch or dedicated stability report step that enables normal check publication for these shards, with the required checks permission.