Repository navigation

docs: SEV1 post-mortem for model-engine 500s incident (MLI-6574) - #817

Open

lilyz-ai wants to merge 27 commits into

mainfrom

lilyz-ai/mli-6574-sev1-post-mortem

Commits on Apr 21, 2026
- chore: bump model-engine image tag to latest main (2e9d0078)
  
  lilyz-ai
  and
  claude
  committed
  9a10495
Commits on Apr 29, 2026
Commits on May 3, 2026
- docs(post-mortem): clarify terms, fix action items, replace P1/P2 with rollout strategy
  
  lilyz-ai
  and
  claude
  committed
  ab7cee4
Commits on May 4, 2026
- docs(post-mortem): streamline action items and expand rollout strategy
  
  lilyz-ai
  and
  claude
  committed
  97fa123
- docs(post-mortem): fix RBAC location, drop scripts/deploy.sh, expand rollout strategy
  
  lilyz-ai
  and
  claude
  committed
  cb1b3a3
- docs(post-mortem): pin actual RBAC files and add real staging effort estimate
  
  lilyz-ai
  and
  claude
  committed
  a6a7994
- docs(post-mortem): write concrete RBAC diffs and fix staging estimate
  
  lilyz-ai
  and
  claude
  committed
  6e084d2
- docs(post-mortem): fix deploy impact of RBAC change, rename role, trim sections
  
  lilyz-ai
  and
  claude
  committed
  5805dbb
- docs(post-mortem): two preventative controls — RBAC + master branch guard
  
  lilyz-ai
  and
  claude
  committed
  261f6ab
- docs(post-mortem): explain SA token, add dedicated IAM role as elegant alternative
  
  lilyz-ai
  and
  claude
  committed
  18a5bf0
- docs(post-mortem): go with dedicated ml-serving-deployer IAM role (Option B only)
  
  lilyz-ai
  and
  claude
  committed
  a3e8338
- docs(post-mortem): note cross-service impact of scale-deploy RBAC change
  
  lilyz-ai
  and
  claude
  committed
  ebdf6a1
- docs(post-mortem): identify concrete cross-service impact from kubectl inspection
  
  lilyz-ai
  and
  claude
  committed
  5902d3f
- docs(post-mortem): backward-compatible RBAC — additive only, ml_infra_admin untouched
  
  lilyz-ai
  and
  claude
  committed
  988bc12
- docs(post-mortem): add Option A vs B comparison table for deploy identity approach
  
  lilyz-ai
  and
  claude
  committed
  e1960c7
- docs(post-mortem): reframe deploy identity choice as open design decision
  
  lilyz-ai
  and
  claude
  committed
  636d714
- docs(post-mortem): remove duplicate justfile guard section
  
  lilyz-ai
  and
  claude
  committed
  25b347b
Commits on May 5, 2026
- docs(post-mortem): address Bill's review comments
  
  lilyz-ai
  and
  claude
  committed
  26a02b0
- docs(post-mortem): switch to Option A; complete breakage audit
  
  lilyz-ai
  and
  claude
  committed
  dbdb649
- docs(post-mortem): drop Option B, update files-to-change for Option A
  
  lilyz-ai
  and
  claude
  committed
  46b0d2b
- docs(post-mortem): add concrete code changes for eks.tf RBAC and alerting
  
  lilyz-ai
  and
  claude
  committed
  d3cf507
- docs(post-mortem): fix deploy identity — both new role and scope-down required
  
  lilyz-ai
  and
  claude
  committed
  5226e9a
- docs(post-mortem): replace prod_5m with targeted high-error-rate monitor
  
  lilyz-ai
  and
  claude
  committed
  84b5b4c
- docs(post-mortem): correct genai role claim in audit note
  
  lilyz-ai
  and
  claude
  committed
  5056a47
- docs(post-mortem): drop genai from audit note
  
  lilyz-ai
  and
  claude
  committed
  95bb783