Rate limiting, audit and metrics for the osctrld endpoints

[javuto]

Rate limiting, audit and metrics for the osctrld endpoints

Follow-up to extensions delivery (#1096).

Problem

osctrl-tls's osctrld routes (osctrld-flags, -cert, -verify, -script, -extensions, -extension) authenticate with the shared environment secret, and they had:

• no rate limit;
• no audit of failed secrets;
• no metrics.

Anyone who knew an environment's UUID could guess secrets without limit or trace. The secret now unlocks flags, certificates and root-run extension binaries.

What changes

Three new limits under rateLimits, with the same fields as enroll (burst, period, evictAfter, retryAfter, maxBuckets), CLI flags (--rate-limit-osctrld-auth-*, --rate-limit-osctrld-*, --rate-limit-osctrld-download-*) and env vars. They are validated at startup.

Limit	Default	Keyed by	Counts
osctrldAuth	10 per hour, Retry-After 360 s	client address + environment	wrong secrets only
osctrld	300 per minute	client address	every osctrld request
osctrldDownload	30 per minute	client address	extension downloads

• Blocking: once an address used up its failed-secret budget for an environment, it gets 429 with Retry-After on that environment's osctrld routes, even with the right secret, until the budget refills.
  • Correct secrets never use the budget, so many hosts behind one NAT address aren't affected by one misconfigured or hostile neighbour.
  • Other environments and addresses are unaffected.
• Enforcement:
  • A new middleware around all six routes checks the traffic, download and auth limits before the body is read.
  • The secret comparison then checks and charges the auth budget atomically (ratelimit.Limiter.Attempt), so slow parallel requests can't get extra guesses.
• Client address: utils.GetIP is used for limiter keys and audit. It is the TCP peer, or the right-most untrusted forwarding hop when the peer is in --trusted-proxies.
  • Behind a proxy or load balancer, trustedProxies must be set; otherwise every client shares the proxy's budgets.
  • The enroll limiter is unchanged.
• Many addresses: past maxBuckets tracked keys, new keys share one overflow budget and are blocked collectively.
• Audit:
  • Every wrong secret is written to the audit log with address and environment (new auditlog.FailedOsctrld).
  • While an address is blocked, only the first rejection is written, then at most one per period.
  • Failed-secret audit writes are capped at 60 per environment per hour.
  • Environment UUIDs that don't exist are recorded as a single <unknown> environment, sharing one cap and one metric label.
• Metrics:
  • osctrld_auth_failures_total{environment}, the alert signal for secret guessing;
  • osctrld_rate_limited_total{route,limit};
  • osctrld_auth_audit_dropped_total{environment};
  • per-route request latency in the existing histogram, labelled by route pattern, not the raw URL.
• Shared secret check: a single helper now does the secret check for all six handlers.
• CertHandler: its 403 body is now empty, previously "uh oh...".

Configuration compatibility

• An existing tls.yml without the new entries still loads: the defaults are filled in before the YAML is read.
• Database service config without them keeps the defaults.
• ConfigVersion 4 → 5. This only affects the startup notice and the generated samples. Sample tls.yml/api.yml, the service-config help text and ARCHITECTURE.md are updated.

Security and operational impact

• Secret guessing: limited to 10 wrong secrets per address and environment per hour, then blocked, audited and visible in metrics. Legitimate osctrld traffic never sends wrong secrets.
• Proxies: deployments behind a reverse proxy or load balancer must set trustedProxies, or all hosts share one budget.
• Overflow: while the 100,000-address overflow budget is spent, new addresses are blocked together, including ones sending the correct secret, until it refills. That takes a sustained spray from very many distinct addresses.
• Replicas: budgets are per osctrl-tls process; N replicas give N times each budget.
• Mixed fleets: older osctrld clients are unaffected unless they exceed the budgets, and they retry on their next sync.

Validation

• go test ./... -count=1: 57 packages pass (new tests also with -race)
• go vet ./...; gofmt -l pkg cmd (empty); golangci-lint run ./...: 0 issues
• tls/api/cli build; make openapi-check clean
• New tests, each seen failing first:
  • Limiter primitives: the non-consuming check, atomic check-and-charge (including concurrency), overflow, Retry-After.
  • Shared NAT: 100 correct-secret requests from one address are never blocked.
  • Blocking: 10 failures block only that address and environment, even with the right secret afterwards; the block refills; upper-case UUIDs share a budget.
  • 40 concurrent wrong secrets: exactly 10 get 403, the rest 429.
  • Trusted proxies: clients behind one proxy get separate budgets, and a spoofed left-most X-Forwarded-For doesn't change the key; without trusted proxies, the TCP peer is the key.
  • Traffic and download limits; overflow blocks a spray.
  • Audit:
    • throttled while blocked;
    • per-environment cap with the dropped counter;
    • unknown environments share one <unknown> cap;
    • upper-case environment paths resolve.
  • Metrics: the counters, and latency labelled by route pattern.
  • All six handlers charge failed secrets.
  • Full route registration: every osctrld route returns 429 when blocked, and enroll is unaffected.
  • Configuration: defaults, validation and flags; old YAML loads; DB service config; the sample config documents every limit.
• No frontend behaviour change beyond the service-config help text.