Operator Checks

A monitoring and auto-refill service for the Anyone Protocol. It periodically checks the balances of the protocol's operator and controller wallets across multiple chains (EVM, Arweave, AO) and asset types (native gas tokens, the $ANYONE ERC-20, ), records every reading in MongoDB, and automatically tops wallets back up when they fall below a configured threshold.

The protocol's automated jobs — bundling data to Arweave, paying out relay/staking rewards, maintaining the operator registry — all spend from hot wallets. If one of those wallets runs dry, the corresponding on-chain job stalls. Operator Checks exists to keep those wallets funded without manual intervention, and to raise alarms when something looks wrong.

Contents

• How it works
• What gets checked
• Refills
• Running locally
• Configuration reference
• Deployment
• Alarms & observability
• Roadmap / not-yet-wired

---

How it works

Operator Checks is a NestJS application backed by BullMQ (Redis) for job scheduling and MongoDB for persistence.

The flow is a self-rescheduling loop:

tasks queue                balance-checks flow                 refills queue
-----------                -------------------                 -------------
check-balances  ─────────► children:                          refill-ar
   │                         check-hyperbeam-node   ──┐         refill-token
   │ (re-queues itself       check-hodler             │ shortfall?
   │  every RECHECK_DELAY_MS check-rewards-pool       ─┘
   │  ms)                    check-relay-registry     │
   └─────────────────────►   check-relay-rewards      │
                             check-staking-rewards    │
                             ▼
                           review-balance-checks  ──► store all readings in MongoDB

1. check-balances (tasks queue) fans out a BullMQ flow: one child job per balance check, plus a parent review-balance-checks job that runs once all children finish. It then re-queues itself with a delay of RECHECK_DELAY_MS (default 5 minutes; 15 minutes in production), so the service runs continuously.
2. Each check job reads a wallet's balance and compares it against MIN/MAX thresholds. If the balance is below MIN, the job computes the shortfall (MAX − balance) and enqueues a refill. If it's above MAX, it logs a balance-accumulation alarm (funds may be stuck or misconfigured).
3. review-balance-checks collects every child's reading and persists them to the BalancesData collection in MongoDB as a time-stamped batch.
4. Refill jobs (refills queue) send the actual top-up transaction from the relevant spender wallet — but only when IS_LIVE=true; otherwise they log what they would have sent and do nothing.

Clustering & leader election

The service is designed to run with multiple instances (Nomad runs count = 2) without double-spending. Only one instance is the leader, and only the leader seeds the initial check-balances job on bootstrap.

• Leader election uses Consul sessions and a KV lock (clusters/<service>/leader). See cluster.service.ts.
• Local forking uses Node's cluster module; CPU_COUNT controls how many worker threads fork, and the first fork is flagged the local leader via IS_LOCAL_LEADER. See app-threads.service.ts.
• isTheOne() returns true only for the process that is both the Consul leader and the local leader — that process owns the one-time bootstrap actions.

When IS_LIVE is not true, Consul is skipped entirely and the service boots in single-node mode (always leader), making local development straightforward.

Health

An HTTP server exposes GET / and GET /health, both returning OK (used by the Nomad health check). See app.controller.ts.

---

What gets checked

Each row below is one child job in the balance-checks flow. Thresholds are set via the env vars listed in the configuration reference.

Check job	Wallet	Chain	Asset	Auto-refill
check-hyperbeam-node	HyperBEAM node	Arweave	$AR	✅ sends $AR
check-rewards-pool	Rewards pool	EVM	$ANYONE (ERC-20)	✅ sends $ANYONE
check-hodler	Hodler operator	EVM	$ETH (gas)	❌ monitor only by design — gas is user-funded
check-relay-registry	Operator Registry controller	AO	$AO	⚠️ monitor only — see below
check-relay-rewards	Relay Rewards controller	AO	$AO	⚠️ monitor only
check-staking-rewards	Staking Rewards controller	AO	$AO	⚠️ monitor only

The aggregating job review-balance-checks is not a check itself; it stores all of the above readings.

Balance readings (MongoDB)

Every check writes a BalancesData document:

Field	Description
stamp	Epoch-ms timestamp shared by all readings in a single flow run
kind	Reading type, e.g. hyperbeam-node-ar-balance, hodler-operator-eth-balance
amount	Balance at check time (human-readable units, as a string)
requestAmount	Shortfall that triggered a refill, if any
address	The wallet/address that was checked

Hodler operator gas

The check-hodler job watches the hodler operator's $ETH balance, but does not refill it. That wallet's gas is user-funded — users send $ETH to it so it has gas to claim their rewards — so the protocol intentionally does not top it up. We simply keep an eye on it and raise an alarm if it drifts outside its MIN/MAX band. The shortfall is still computed and recorded, but no refill is enqueued.

AO balance checks (removed)

Three checks (check-relay-registry, check-relay-rewards, check-staking-rewards) used to read the controllers' $AO token balances via an aoconnect dry-run against a legacynet $AO token process, to keep those wallets topped up for message fees.

They were removed in the HyperBEAM migration (D17). They had already been disabled in both live and stage (AO_BALANCE_CHECKS_ENABLED="false") because AO was not charging fees, and the migration settles the question: we run our own node and pay no per-message $AO, so there is no balance to deplete and nothing for the checks to observe.

Nothing was refilled by them either — unlike the hyperbeam node and rewards-pool checks, they computed a requestAmount but never enqueued a refill, so removing them unwinds no funding path. This service no longer talks to AO at all.

---

Refills

Refills run from dedicated spender wallets and are only executed when IS_LIVE=true. In any other mode the refill is logged as a no-op (NOT LIVE, ... did NOT send ...), which makes it safe to run the full pipeline against real RPCs without moving funds.

Refill job	Asset	Spender	Notes
refill-ar	$AR	AR_SPENDER_KEY (Arweave JWK)	Verifies the spender has enough $AR before sending
refill-token	$ANYONE	ETH_SPENDER_KEY (EVM key)	ERC-20 transfer on TOKEN_CONTRACT_ADDRESS
refill-eth	$ETH	ETH_SPENDER_KEY	Implemented but intentionally not enqueued — the hodler's gas is user-funded, so we only monitor it
refill-ao	$AO	—	Stub only — see roadmap

See refills.service.ts and refills-queue.ts.

$AR refill de-duplication

An Arweave transfer takes tens of minutes to mine and the recipient's balance does not move until it does, while the check loop runs every RECHECK_DELAY_MS (15 min). Without a guard the same low balance is seen two or three times and the same refill is sent two or three times over. sendArTo therefore skips a refill for an address it sent to within the last hour. See isArTransferInFlight in refills.service.ts.

A second guard, hasRecentArTransfer, queries Arweave GraphQL for transfers from the AR spender to that address within AR_REFILL_LOOKBACK_MS (default 2h) and skips if it finds one. This is the half that survives a restart, which the in-process cooldown cannot.

Neither is sufficient alone. arweave.net GraphQL does not index the mempool, so the lookback sees mined transfers only and the cooldown covers the unconfirmed window. The one remaining gap is a restart during that window; the cost there is over-funding a wallet we own, which the node then spends. Both guards return "in flight" on any error — a failed lookup must never license a second spend.

---

Running locally

Prerequisites

• Node.js (LTS) and npm
• A Redis instance (standalone is fine for local dev)
• A MongoDB instance

Install & run

npm install

# dev with hot reload
npm run start:dev

# or a one-off run
npm start

Provide configuration via environment variables (e.g. a .env file — @nestjs/config is loaded globally). A minimal local setup:

# Leave IS_LIVE unset/false so no real transactions are sent and Consul is skipped
MONGO_URI="mongodb://localhost:27017/operator-checks"
REDIS_MODE="standalone"
REDIS_HOSTNAME="localhost"
REDIS_PORT=6379
JSON_RPC="https://..."             # an EVM RPC endpoint
TOKEN_CONTRACT_ADDRESS="0x..."     # $ANYONE token
# ...plus the wallet addresses / thresholds you want to exercise

With IS_LIVE unset, the service runs single-node, executes all checks against the real RPCs/gateways, persists readings, and logs (but does not perform) any refills.

Note: RefillsService requires ETH_SPENDER_KEY, TOKEN_CONTRACT_ADDRESS, and AR_SPENDER_KEY to be present at startup or it throws. Even in non-live mode these must be set (dummy values are fine for keys you don't intend to use).

Other scripts

npm run build        # nest build
npm test             # jest unit tests
npm run test:cov     # coverage
npm run lint         # eslint --fix
npm run format       # prettier

---

Configuration reference

All configuration is via environment variables.

Core / runtime

Variable	Default	Description
PORT	3000	HTTP port for the health endpoint
IS_LIVE	unset	"true" enables real refill transactions and Consul clustering. Anything else = dry-run, single-node, and the tasks queue is obliterated on boot
DO_CLEAN	unset	"true" obliterates the tasks queue on bootstrap (leader only)
RECHECK_DELAY_MS	300000 (5 min)	Delay between balance-check runs (production uses 900000 = 15 min)
VERSION	unset	Informational build/commit identifier (set by deployment)

Persistence & queues

Variable	Default	Description
MONGO_URI	—	MongoDB connection string
REDIS_MODE	standalone	standalone or sentinel
REDIS_HOSTNAME / REDIS_PORT	—	Redis host/port (standalone mode)
REDIS_MASTER_NAME	—	Sentinel master name (sentinel mode)
REDIS_SENTINEL_{1,2,3}_HOST / _PORT	—	Sentinel addresses (sentinel mode)

Clustering (Consul)

Only used when IS_LIVE=true. If host/port are missing, the service falls back to single-node mode.

Variable	Description
CONSUL_HOST / CONSUL_PORT	Consul agent address
CONSUL_SERVICE_NAME	Service name used for the leader-election KV key/session
CONSUL_TOKEN_CONTROLLER_CLUSTER	Consul ACL token
IS_LOCAL_LEADER	"true" marks a process as the local leader (set per-fork)
CPU_COUNT	Number of worker threads to fork

Arweave gateway

Variable	Default	Description
ARWEAVE_GATEWAY_PROTOCOL	https	Gateway protocol
ARWEAVE_GATEWAY_HOST	arweave.net	Gateway host
ARWEAVE_GATEWAY_PORT	443	Gateway port

EVM (hodler / rewards pool / token refills)

Variable	Description
JSON_RPC	EVM JSON-RPC endpoint
TOKEN_CONTRACT_ADDRESS	$ANYONE ERC-20 contract address
ETH_SPENDER_KEY	Private key of the EVM spender wallet (for $ANYONE / $ETH refills)
HODLER_OPERATOR_ADDRESS	Wallet whose $ETH gas balance is monitored
HODLER_OPERATOR_MIN_ETH / MAX_ETH	$ETH thresholds (ether units)
REWARDS_POOL_ADDRESS	Wallet whose $ANYONE balance is monitored
REWARDS_POOL_MIN_TOKEN / MAX_TOKEN	$ANYONE thresholds (whole tokens)

Arweave spender & hyperbeam node

Variable	Description
AR_SPENDER_KEY	Arweave JWK (JSON) for $AR refills
AR_REFILL_LOOKBACK_MS	7200000 (2h)
HYPERBEAM_NODE_AR_ADDRESS	Address of the node wallet being monitored. Address, not a JWK: a balance read and an incoming transfer need only the public address, so the node's signing key never leaves the node
HYPERBEAM_NODE_MIN_AR / MAX_AR	$AR thresholds. Size MAX against the SPENDER's balance too, not just node runway: a refill sends MAX - balance, and sendArTo compares balance < amount without the tx fee

---

Deployment

The service ships as a Docker image (ghcr.io/anyone-protocol/operator-checks, built by .github/workflows/release-action.yml) and runs on HashiCorp Nomad. Job specs live in operations/:

• operator-checks-live.hcl / operator-checks-stage.hcl — the service jobs (2 instances each, IS_LIVE=true, Redis in sentinel mode, leader-elected via Consul).
• operator-checks-redis-sentinel-live.hcl / operator-checks-redis-sentinel-stage.hcl — the Redis sentinel deployments.

Secrets (spender keys, RPC URLs, controller addresses) are pulled from Vault; non-secret config (token address, Mongo URI, Redis/Arweave gateway endpoints) is rendered from Consul service discovery. See the template blocks in the job specs for the exact mapping.

---

Alarms & observability

Logging uses Winston with a single-line console format (timestamp|level|context: message), suitable for log aggregation. See main.ts.

Operationally important conditions are logged with a machine-parseable [alarm=<name>] tag so they can be alerted on. Notable alarms:

• balance-accumulation-* — a monitored wallet is above its MAX threshold (funds may be stuck or thresholds misconfigured).
• refill-failed-eth / refill-failed-anyonetokens / refill-failed-ar / refill-failed-ao — a refill transaction failed or the spender lacked sufficient balance.
• failed-job-<jobName> — a BullMQ job failed. ✅ The hyperbeam node check added no new alarm names and removed none, so Grafana needs no change. Accumulation on the node wallet reuses balance-accumulation-ar-bundler (the node IS the bundler since it began self-bundling), and depletion is a plain warn like every other check: it triggers a refill, and refill-failed-ar fires if that refill cannot happen. A node too empty to pay for a single upload is logged at error level without its own tag, for the same reason.

---

Roadmap / not-yet-wired

The following are present in the code as intended behavior but not active today:

• $AO refills. RefillsService.sendAoTo is a stub (logs "Not implemented yet"). The AO balance checks themselves are also disabled in deployment because AO does not yet charge $AO for gas/transaction fees; both will become relevant once it does.