Support pack

This directory is the support pack for BES infrastructure: the material an on-call engineer, or an agent assisting one, reads when troubleshooting or operating a deployment. What each part of the system must do is in the specifications. The pack says how to find out what is wrong and what to do about it, and cites the spec or the code for each claim.

Read the source when you need to. These documents are a starting point, and where they disagree with the code the code is right. Prefer a fact checked against the code to an assumption.

Structure

• Runbook (runbooks/): one per named symptom. It walks the symptom from first report through confirming, interpreting, fixing and escalating.
• SOP (sops/): one reusable procedure that many runbooks refer to, such as connecting to a cluster or waking a deployment.
• Reference (reference/): lookup material with no procedure, such as config keys, object names and version pins.
• Specifications (spec/): what the system must do, in the voice of a standard. Start at the overview.
• Developer orientation (dev/): how pulumi/ is laid out and the conventions every stack follows.
• Plans (plans/): work in progress and migrations that the specs do not yet describe.

Two documents sit beside them: deployment context, for working out which deployment is affected, and ruled-out actions, the hard filter on what may be suggested.

The action-classification ladder

Every suggested action in a runbook or SOP is tagged with one of these classes, which rank how much oversight an action needs before anyone runs it.

• diagnose: read-only inspection, such as reading a log or checking a status. No mutation. Suggest freely.
• approved-mitigation: mutating, but a human has assessed it as low risk, or a runbook pre-signs it. Run it without anyone watching.
• any-OTS: mutating. Needs a second pair of eyes, and another support engineer suffices.
• dev-OTS: mutating. Needs a developer watching.
• ruled-out: destructive, irreversible or out of scope. Never suggested. Escalate instead. See ruled-out actions.

Anything mutating is dev-OTS until a human has assessed it lower. A runbook may pre-sign a specific step to a lower class, and outside that the default stands.

An action may also carry a sensitive-data flag. It marks actions that touch personal or credential data even when they only read, such as copying a log off a server. See the sensitive-data section of ruled-out actions.

Writing a runbook

A runbook covers one named symptom, the thing an engineer sees first, and is titled for it. Keep it under 80 lines; when it grows past that, split it along a symptom seam into two runbooks, each with its own entry in the index below.

Open with the title, a line or two stating the symptom, and the tag line Every action is tagged with its class, linked to ruled-out actions. Then six numbered sections, in this order (hook-job-failed is the model):

1. When this applies: the evidence that confirms the symptom, and the sibling runbook to use when it does not match.
2. Establish context: which deployment or cluster, what changed, through deployment context.
3. Investigate: numbered read-only steps, each with its command or the SOP that holds it.
4. Interpret: one table with the columns Evidence (what was seen), Diagnosis (what it means, with the citation) and Action (what to do, or which runbook to follow). One row per branch of the symptom.
5. Resolve: the mutating steps, and what confirms they worked.
6. Escalate: when to escalate, the state the runbook leaves the incident in, and what to hand over.

These rules hold throughout:

• Tag every action with its ladder class in bold brackets, **[diagnose]** or **[dev-OTS]**. Add sensitive-data when the action touches personal or credential data, even to read it: **[diagnose, sensitive-data]**.
• Cite a SOP for a procedure rather than restating its commands. A runbook holds only the commands that exist nowhere else.
• Cite the spec section by id and heading (K8STDR, for example) or the code file, for each claim about how the system behaves.
• The Escalate section names the state in the terms of the support-assist skill, which defines them:
  • workaround-in-place: the user can keep working, and the cause can be handled on a normal timeline.
  • fix-required: a code change is needed; name the code file and area.
  • still-diagnosing: not enough signal yet; name the next diagnostic.
• The Escalate section also says what to hand over, from the skill's escalation payload: the deployment or environment; tier, urgency and scope; a one-line symptom; what is established, with citations; the state, and the code area when fix-required; whether sensitive data is involved; and the specific ask of on-call. Name the evidence the runbook produced that fills those fields.

Load by convention

Tooling and agents recognise a support pack by docs/ruled-out-actions.md together with docs/runbooks/. When both exist the pack is loaded, and ruled-out-actions.md is a hard filter on any suggested action.

Runbooks

• Runbook: Application stays OutOfSync after a successful sync
• Runbook: Argo CD sync Failed, or Application stuck Progressing until pulumi up times out
• Runbook: Canopy is not issuing certificates
• Runbook: the canopy stack refuses to preview or apply
• Runbook: Cannot reach the cluster API, or a stack refuses to run
• Runbook: A cluster upgrade is stuck or pod health is falling
• Runbook: a config or template change did not reach a deployment
• Runbook: a database does not come up
• Runbook: deployment is down but nothing failed
• Runbook: Envoy Gateway will not deploy or crash-loops after a version bump
• Runbook: HNC pods not starting, or writes rejected by HNC webhooks
• Runbook: a migrator or other hook Job failed
• Runbook: pods stuck in ImagePullBackOff (image tag missing)
• Runbook: a Linux server's backup is failing or missing
• Runbook: a Linux server's certificate is wrong or missing
• Runbook: a clone deployment's facility does not sync after a restore
• Runbook: Tamanu cannot reach the database on a Linux server
• Runbook: a Linux server is running out of disk
• Runbook: a Linux playbook refuses to run or stops early
• Runbook: PostgreSQL restart pending on a production server
• Runbook: a Linux server cannot be reached
• Runbook: a Tamanu service is down on a Linux server
• Runbook: Pods stay Pending and no node appears, or NodeClasses are not Ready
• Runbook: Application shows a render error
• Runbook: a renderer change does not show up
• Runbook: a seeded deployment starts empty, with the wrong data, or refuses
• Runbook: settings fail to decrypt
• Runbook: the update is refused before anything changes
• Runbook: teardown is stuck

SOPs

• SOP: Bootstrap a new Kubernetes cluster
• SOP: Change a deployment's config and update it
• SOP: Connect to a deployment's database from the cluster
• SOP: Connect to a cluster with kubectl
• SOP: Create a deployment
• SOP: Destroy a deployment
• SOP: Edit or reset the config ConfigMaps
• SOP: Let Canopy issue certificates in another account's zone
• SOP: Hard-refresh an Application
• SOP: Inspect an Application
• SOP: Inspect the render
• SOP: Check that a Linux server's backups are working
• SOP: Check exposure to the blocked kernel modules
• SOP: Convert a Linux server's disk layout
• SOP: Provision a Linux Tamanu server
• SOP: Restart PostgreSQL on a Linux server
• SOP: Retire a Linux deployment
• SOP: Run the Linux playbooks
• SOP: Upgrade PostgreSQL on a Linux server
• SOP: Upgrade Tamanu on Linux servers
• SOP: Promote a renderer channel
• SOP: Read logs
• SOP: Pull stack config to your machine
• SOP: Register a cluster's tags and operator identity in Tailscale
• SOP: Restart services
• SOP: Restore a deployment's databases from backup
• SOP: Wind down or recall the bootstrap nodegroups
• SOP: Update a Kubernetes cluster
• SOP: Wake a hibernated deployment

Reference

• Reference: Canopy TLS issuance plumbing
• Reference: pinned component versions
• Reference: k8s-core stack config
• Reference: k8s-essentials stack config
• Reference: the tamanu/k8s-infra stack
• Reference: node pools and node classes
• Reference: what lives where on a cluster
• Reference: Linux inventory
• Reference: Linux server paths, units and ports
• Reference: Credentials on a Linux server
• Reference: Linux deployment stack config and outputs
• Reference: Linux playbooks and roles
• Reference: Linux playbook variables
• Reference: Tamanu deployment stack config
• Reference: objects of a Tamanu deployment

Developer orientation

• Pulumi Repo Organisation
• The Core Account Stack