Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -918,6 +918,59 @@ You can also specify bootstrap peers explicitly:

---

## Health Endpoint

The local health endpoint is **off by default**. Enable it with `--metrics-port 9100`
or `ANT_METRICS_PORT=9100`; `payment.metrics_port` is the configuration setting.
Set it to `0` and restart to disable it. It binds only to IPv4 loopback
(`127.0.0.1`), not the node's public address. If the port is busy, the node warns
and continues without the endpoint; failure to publish its port also does not
prevent the node from running. Logging must be enabled to see these warnings.

- `/health` is JSON for people and programs: `curl http://127.0.0.1:9100/health`.
- `/metrics` is Prometheus text for monitoring systems scraping on the same machine.
- `{root_dir}/metrics.port` contains the bound port followed by a newline. It is
cleaned up best-effort on orderly shutdown, when disabled, or when binding fails.
A crash or filesystem error can leave it behind: treat it as discovery, not
proof, and verify the response's `peer_id` before trusting it.

Both endpoints read the same existing node state when requested. `peer_id` (hex)
and `version` are descriptive JSON strings and labels on all Prometheus series.
The remaining JSON fields and matching metrics are:

| JSON field | Prometheus metric | Meaning |
|-----------|-------------------|---------|
| `uptime_secs` | `ant_uptime_seconds` | Seconds since the P2P node was constructed |
| `bootstrapped` | `p2p_health_status` | Initial bootstrap completed, **not** a verdict that the node is healthy |
| `peer_count` | `p2p_network_peer_count` | Currently connected authenticated peers |
| `routing_table_size` | `p2p_dht_routing_table_size` | Current peer entries in the routing table |
| `storage_enabled` | `ant_storage_enabled` | Whether this node provides storage |
| `chunks_current` | `ant_chunks_current` | Chunks currently persisted, including existing chunks after restart |
| `chunks_written_total` | `ant_chunks_written_total` | Successful chunk writes since process start |
| `bytes_written_total` | `ant_bytes_written_total` | Bytes written since process start |
| `chunks_served_total` | `ant_chunks_served_total` | Storage reads since process start, not completed client downloads |
| `bytes_served_total` | `ant_bytes_served_total` | Bytes read from storage since process start |

`_total` fields are process-lifetime activity counters and reset on restart;
other numeric and boolean fields report current state. Fields are additive.
JSON uses booleans for `bootstrapped` and `storage_enabled`; Prometheus uses
gauges with `0`/`1`. Other live measurements are gauges and `_total` metrics are
counters. Storage measurements are zero when storage is disabled. Activity
counters reflect the file store; they do not include legacy-store activity during
migration. No live stored-byte metric is emitted.

The inherited `p2p_` and `ant_` names match existing local deployment tooling
where meanings agree. Remote Prometheus worker-IP targets cannot reach this
loopback listener, and dashboard panels for deferred metrics remain empty.
Requests are read-only, served one at a time with bounded headers and a timeout;
shutdown cancels active requests. A single `Host` header must name `127.0.0.1` or
`localhost`, optionally followed by the bound port. Missing, duplicate or other
host values return `403` without health data; this also rejects website hostnames
that resolve to loopback. Normal curl and Prometheus requests supply this header.
For trusted hosts, only `GET /health` and `GET /metrics` are supported; other paths
return `404`, other methods `405` with `Allow: GET`. Responses close the connection,
disable caching, and have no CORS header.

## CLI Reference

```
Expand Down
4 changes: 2 additions & 2 deletions config/production.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ cache_capacity = 100000
# EVM network: "arbitrum-one" or "arbitrum-sepolia"
evm_network = "arbitrum-one"

# Prometheus metrics port (0 to disable)
metrics_port = 9100
# Loopback health JSON and Prometheus port (0 = off; set e.g. 9100 to enable)
metrics_port = 0

# --- Storage ---
[storage]
Expand Down
82 changes: 82 additions & 0 deletions docs/adr/ADR-0017-local-node-health-endpoint.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# ADR-0017: Local node health endpoint

- **Status:** Proposed
- **Date:** 2026-09-30
- **Decision owners:** ant-node maintainers (review pending)
- **Reviewers:** Jim Collinson; ant-node maintainers (review pending)
- **Supersedes:** none
- **Superseded by:** none
- **Related:** [V2-1380](https://linear.app/autonominetwork/issue/V2-1380/ant-node-serve-a-local-health-endpoint-on-the-existing-metrics-port)

## Context

The node already holds information about its connectivity, storage and uptime,
but local tools cannot ask for it. Its configuration advertises a monitoring
endpoint that has not served anything. Operators and applications instead see
only process state or must inspect logs. Logging is optional, and its messages
are not a stable interface.

Starting a listener on existing nodes could conflict with other local services.
Ordinary deployments should remain unchanged unless an operator enables it.

## Decision Drivers

- Make existing state available without requiring logging or new instrumentation.
- Support people, applications and standard monitoring tools.
- Keep the node private, dependency-light and unaffected by diagnostic failures.
- Require an explicit choice to enable the service.

## Considered Options

1. **Read logs:** rejected because messages change and logging may be disabled.
2. **Use a private channel through the management daemon:** rejected because it
couples the node to one consumer and requires a larger coordinated change.
3. **Let the node answer local, read-only health queries:** chosen because each
consumer can ask directly for the existing state.

## Decision

The node will offer its own health information on demand through a local endpoint.
We commit to these boundaries:

- It binds only to loopback, meaning it is reachable only on the same machine.
- It is read-only and discloses no secrets, network addresses or filesystem paths.
- A form for programs and a form for monitoring carry the same values.
- Its failures cannot fail node startup, and its work cannot hold shutdown open
indefinitely.
- It advertises its bound port locally for tools to discover.
- It is off unless explicitly configured on; configuration changes apply at restart.

The concrete interface and compatibility contract belong in the README and
linked issue's specification, not in this decision record.

## Consequences

### Positive

- Local consumers obtain health without inspecting logs or storage files.
- Existing in-memory state is reused without adding dependencies.
- Nodes that do not enable the endpoint open no new listener.

### Negative / Trade-offs

- Any local process can read the published health when enabled.
- A busy port leaves that node without the endpoint rather than preventing startup.
- Remote monitoring needs a separate, deliberately configured access route.

### Neutral / Operational

- Discovery identifies where to ask, not proof that a node is healthy.
- The initial information set is intentionally limited; more instrumentation is
separate work.

## Validation

Tests and a development-network run check both representations, disabled mode,
port conflicts, local discovery and shutdown. Review confirms local-only binding,
absence of sensitive information, and isolation from node operation.

## Notes for AI-assisted work

AI tools may draft this record but must not mark it Accepted. Human engineering
review owns acceptance; Accepted records are immutable.
101 changes: 97 additions & 4 deletions src/bin/ant-node/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,9 @@ pub struct Cli {
#[arg(long, env = "ANT_EVM_PAYMENT_VAULT")]
pub evm_payment_vault: Option<String>,

/// Metrics port for Prometheus scraping (0 to disable).
#[arg(long, default_value = "9100", env = "ANT_METRICS_PORT")]
pub metrics_port: u16,
/// Loopback health JSON and Prometheus port (0 = off; set e.g. 9100 to enable).
#[arg(long, env = "ANT_METRICS_PORT")]
pub metrics_port: Option<u16>,

/// Enable logging output.
/// When omitted, the tracing subscriber is not installed and no log
Expand Down Expand Up @@ -302,7 +302,7 @@ impl Cli {
cache_capacity: self.cache_capacity,
rewards_address: self.rewards_address,
evm_network,
metrics_port: self.metrics_port,
metrics_port: self.metrics_port.unwrap_or(config.payment.metrics_port),
};

// Determine bootstrap source and apply auto-discovery if needed.
Expand Down Expand Up @@ -374,6 +374,99 @@ mod tests {
use clap::Parser;
use std::net::{IpAddr, Ipv4Addr};

#[test]
fn metrics_port_default_is_off() -> Result<(), Box<dyn std::error::Error>> {
use clap::CommandFactory;
let command = Cli::command().mut_arg("metrics_port", |arg| arg.env(None::<&str>));
let matches = command.try_get_matches_from(["ant-node"])?;
let cli = <Cli as clap::FromArgMatches>::from_arg_matches(&matches)?;
assert_eq!(cli.into_config()?.0.payment.metrics_port, 0);
Ok(())
}

#[test]
fn metrics_port_preserves_config_and_explicit_overrides(
) -> Result<(), Box<dyn std::error::Error>> {
use clap::{CommandFactory, FromArgMatches};
let dir = tempfile::tempdir()?;
let path = dir.path().join("node.toml");
std::fs::write(&path, "[payment]\nmetrics_port = 23456\n")?;
let path = path.to_str().ok_or("config path must be UTF-8")?;
for (extra, expected) in [
(vec![], 23456),
(vec!["--metrics-port", "34567"], 34567),
(vec!["--metrics-port", "0"], 0),
] {
let command = Cli::command().mut_arg("metrics_port", |arg| arg.env(None::<&str>));
let args = [vec!["ant-node", "--config", path], extra].concat();
let matches = command.try_get_matches_from(args)?;
let cli = Cli::from_arg_matches(&matches)?;
assert_eq!(cli.into_config()?.0.payment.metrics_port, expected);
}
Ok(())
}

#[test]
fn metrics_port_environment_and_cli_precedence() -> Result<(), Box<dyn std::error::Error>> {
const CHILD_EXPECTED: &str = "ANT_TEST_METRICS_PORT_EXPECTED";
const CHILD_CONFIG: &str = "ANT_TEST_METRICS_PORT_CONFIG";
const CHILD_CLI: &str = "ANT_TEST_METRICS_PORT_CLI";

if let Ok(expected) = std::env::var(CHILD_EXPECTED) {
let mut args = vec![
"ant-node".to_string(),
"--config".to_string(),
std::env::var(CHILD_CONFIG)?,
];
if let Ok(port) = std::env::var(CHILD_CLI) {
args.extend(["--metrics-port".to_string(), port]);
}
let cli = Cli::try_parse_from(args)?;
assert_eq!(
cli.into_config()?.0.payment.metrics_port,
expected.parse::<u16>()?
);
return Ok(());
}

let dir = tempfile::tempdir()?;
let path = dir.path().join("node.toml");
std::fs::write(&path, "[payment]\nmetrics_port = 23456\n")?;
for (env_port, cli_port, expected) in [
(None, None, 23456),
(Some("34567"), None, 34567),
(Some("0"), None, 0),
(Some("0"), Some("45678"), 45678),
(Some("34567"), Some("0"), 0),
] {
// Isolate clap's real environment bindings without changing this process's env.
let mut child = std::process::Command::new(std::env::current_exe()?);
child
.args([
"--exact",
"cli::tests::metrics_port_environment_and_cli_precedence",
"--nocapture",
])
.env_clear()
.env(CHILD_EXPECTED, expected.to_string())
.env(CHILD_CONFIG, &path);
if let Some(port) = env_port {
child.env("ANT_METRICS_PORT", port);
}
if let Some(port) = cli_port {
child.env(CHILD_CLI, port);
}
let output = child.output()?;
assert!(
output.status.success(),
"metrics env={env_port:?}, CLI={cli_port:?} failed:\n{}{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
}
Ok(())
}

#[test]
fn webrtc_direct_port_overrides_the_default_bind_port() -> Result<(), Box<dyn std::error::Error>>
{
Expand Down
15 changes: 12 additions & 3 deletions src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -382,8 +382,8 @@ pub struct PaymentConfig {
#[serde(default)]
pub evm_network: EvmNetworkConfig,

/// Metrics port for Prometheus scraping.
/// Set to 0 to disable metrics endpoint.
/// Loopback health JSON and Prometheus port.
/// 0 = off (default); set e.g. 9100 to enable.
#[serde(default = "default_metrics_port")]
pub metrics_port: u16,
}
Expand All @@ -400,7 +400,7 @@ impl Default for PaymentConfig {
}

const fn default_metrics_port() -> u16 {
9100
0
}

const fn default_cache_capacity() -> usize {
Expand Down Expand Up @@ -780,6 +780,15 @@ mod tests {
assert!(config.cache_capacity > 0, "Cache capacity must be positive");
}

#[test]
fn metrics_endpoint_defaults_to_off() {
assert_eq!(PaymentConfig::default().metrics_port, 0);
let absent: NodeConfig = toml::from_str("").unwrap();
let partial: NodeConfig = toml::from_str("[payment]\ncache_capacity = 123").unwrap();
assert_eq!(absent.payment.metrics_port, 0);
assert_eq!(partial.payment.metrics_port, 0);
}

#[test]
fn default_webrtc_listener_tracks_compile_time_feature() {
let config = WebRtcDirectConfig::default();
Expand Down
1 change: 1 addition & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ pub mod node;
pub mod payment;
pub mod pointer;
pub mod replication;
mod status;
pub mod storage;
pub mod upgrade;
#[cfg(feature = "webrtc-direct")]
Expand Down
Loading
Loading