Usage metrics: utilisation, wait times and provisioning durations from the event history #329

Request: none

Problem

An operator can see what a host or fleet is doing now and nothing about what it did. Nobody can answer, from Simlock, how busy a machine was last week, how long agents waited for a device, how often a grant was a warm device rather than a fresh boot, how long provisioning took, which team used what, or whether another Mac would help. Sizing a host, tuning idle timeouts, TTLs and the warm pool, and justifying hardware are done by feel.

Every fact needed is already recorded in the event history, and events now have stable ids across restarts and across the fleet. There is no view over them. The console shows the present; its spec left history and metrics out on purpose.

Who it is for

An operator sizing one machine or a fleet. A team lead deciding whether to buy another Mac. Someone tuning capacity, the warm pool, idle tiers or TTLs who wants to see the effect. A developer who suspects their agents are spending their time waiting for devices.

Outcome

• For a chosen window, the last hour, day or week, or an explicit start and end, Simlock reports:
  • leases granted and how each was served: warm device, idle device, or newly created;
  • how long leases lasted, as two figures named apart: held time, from grant to release or expiry, and turnaround, from request to release or expiry;
  • how long requests waited, as median, 95th percentile and worst case, and how many timed out or gave up;
  • how long provisioning and booting took;
  • how full the capacity was over time, in device slots and RAM, and how deep the queue was;
  • failures by kind, and device incidents: quarantines, recoveries, lost devices;
  • all of it per platform and, on a gateway, per worker;
  • usage per requester, shown by token label where there is one.
• The numbers are available from the CLI, as a human table and as --json, from the HTTP API, and in the console as a view with the figures and a small number of charts for utilisation and waiting over time.
• The same figures come from the same history on every surface, so they agree with simlock events, survive a daemon restart, and on a gateway cover the fleet while a worker covers itself.
• History reaches back as far as the event log is kept. The operator sets that as a length of time, seven days by default, so the last week works without changing anything. A size limit still bounds the disk the history uses, and when it drops events first, the figures say how far back they reach.
• A machine-readable export lets a team feed the figures into their own tooling.
• Nothing is added to the lease path. Collecting the figures costs an agent nothing.

Non-goals

• Alerting, thresholds or notifications.
• Billing or cost allocation.
• A time-series store or retention beyond the event log.
• A Prometheus or OpenMetrics scrape endpoint. It can follow as its own feature once the figures are settled.
• Measuring what happens inside a device: CPU, memory or app performance in the simulator.
• Recording anything about a person beyond the requester id and token label that events already carry.

Completion conditions

• After a scripted day of leases against the fake driver, simlock stats --since 24h prints the lease count, grant sources, held time and turnaround, wait percentiles, provisioning durations, peak utilisation and failures, and --json prints the same figures. The figures match what a reader counts by hand from simlock events --since 24h for that run.
• After simlock daemon stop and simlock daemon start, the same window returns the same figures.
• On a gateway, the output has fleet totals and one row per worker, and a worker's own output covers that worker only.
• Requests that waited show in the wait figures, and requests that timed out or were cancelled are counted apart from grants.
• Usage per requester lists each requester's leases in the window, with the token label beside the id where the lease came over HTTP.
• The HTTP API answers the same figures for the same window as the CLI's --json.
• The console has a view showing the figures and utilisation and waiting over time for the chosen window. It follows the design guide, works at phone width and in both themes, and updates without a reload as the window moves.
• The export is one command and one format a spreadsheet opens.
• A window older than the kept history says so instead of returning partial figures silently.
• With default settings, a scripted week of leases against the fake driver is still fully covered by simlock stats --since 7d. When the size limit drops events inside the seven days first, the output says how far back the figures reach.
• docs/CLI.md, docs/HTTP-API.md, docs/CONSOLE.md and docs/CONFIGURATION.md describe the command, the route, the view and how retention bounds the window.

Open questions

Decisions

• ADR 0016 — Usage figures are derived on read from the event history: no store; lease.granted and lease.rejected carry the request facts; capacity.changed and queue.changed record the steps; retention by time with a size backstop; the gateway computes from its own history with one dedupe rule; the daemon joins labels and buckets the series.
• ADR 0021 — A gateway dispatch is a probe: a worker records every refusal or failure of a probe as lease.declined and never rejects one; the gateway records every fleet request's outcome itself, as request.granted or lease.rejected (new reason worker-failed), and usage reads fleet outcomes and waits from those, by request id. Replaces ADR 0016 §6's fleet join; narrows ADR 0014 §6. Decided by the maintainer during delivery of usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347.

Tasks

1. Events carry the grant facts, and capacity and queue depth become events #345 — Events carry the grant facts, and capacity and queue depth become events
2. The event log keeps a window of time, bounded by size #346 — The event log keeps a window of time, bounded by size
3. A gateway dispatch is a probe: workers record lease.declined, the gateway records every fleet outcome #431 — A gateway dispatch is a probe: workers record lease.declined and name the fleet request
4. usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347 — usage.get and simlock stats (depends on Events carry the grant facts, and capacity and queue depth become events #345, The event log keeps a window of time, bounded by size #346, A gateway dispatch is a probe: workers record lease.declined, the gateway records every fleet outcome #431)
5. GET /v1/stats and the simlock stats --csv ledger #348 — GET /v1/stats and the simlock stats --csv ledger (depends on usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347)
6. Console Usage view with utilisation and waiting charts #349 — Console Usage view with utilisation and waiting charts (depends on GET /v1/stats and the simlock stats --csv ledger #348)

Written by an agent.