Skip to content

feat(pointer): keep the first final state, and look before taking one - #239

Open
grumbach wants to merge 14 commits into
mainfrom
feat/pointer-ownership-transfer
Open

feat(pointer): keep the first final state, and look before taking one#239
grumbach wants to merge 14 commits into
mainfrom
feat/pointer-ownership-transfer

Conversation

@grumbach

@grumbach grumbach commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Linear issue

Closes V2-1354

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

This changes which pointer state a node keeps at the final counter, and adds a close-group round trip before a node takes a final state.

What

Pointer ownership transfer by final redirection (ADR-0018). The owner key never changes. The owner signs one last state at counter == u64::MAX, pointing at a pointer the new owner holds the key to. Readers are redirected there by every node that holds that state, the address stays the same, and no node that holds it ever gives it up. WithAutonomi/ant-protocol#40 makes a final state final: nothing replaces it, not even another final state whose target sorts first. This PR pins that rev and does the node's half.

  • First-come at the final counter. The store, the admission gate, fresh offers, repair and hints all compare with the protocol's replaces. They keep whichever final state they took first, with no code of their own.
  • Look before taking a final state. A node holding no final state takes any, so a former owner could finalize again on a node that joined the group after the transfer. Before taking a final state it neither holds nor lost, from a client PUT or a fresh offer, a node asks its close group which state each peer holds. A peer claiming a different final state is asked for the record, and if that record verifies (only the owner could have signed it) the write is refused as Stale, naming the state the group proved.
    • A claim alone refuses nothing, so one dishonest peer cannot block a transfer.
    • Each peer's claim and fetch run as one pipeline, all at once, so a peer that claims a rival and stalls its fetch cannot hide another peer's proof until the budget runs out.
    • The look runs after payment is verified, only asks peers that have sent a pointer message, and is bounded at four seconds; silence proves nothing.
    • A proof is remembered, up to two per address and 16,384 addresses, and a replayed loser is refused before its signature is checked. A second proof never replaces the first.
    • Looks for one address take turns, spread over the address's last byte; replays queued behind a clear look reuse its answer for two seconds, and the answer is dropped if the write it cleared fails. At most 64 looks run at once; one that cannot start within two seconds answers the PUT with an error the client retries.
    • A node that lost the file of a final state it held restores that exact state without looking, so a peer on the other side of a fork cannot keep it from its own copy.
  • Forks the owner makes. The owner keeps the key and an earlier record, so it can still sign a second final state, at once or later, and a node that holds none takes whichever reaches it first.
    • The possession check does not penalise a member holding the other side of such a fork, because it holds what the merge rule told it to. It is logged at warn. A member holding nothing is still penalised.
    • Between two final states a wide group backs at quorum, repair adopts the larger side rather than whichever answered first.
    • The client decides reads by majority and reports forks (feat(pointer): hand a pointer over for good, and read forks by majority ant-client#210).
  • ReplicationEngine::with_pointers keeps its signature; with_pointer_service(&PointerService) wires both directions, fresh writes and the finality witness, and is what the node and the e2e harness use.
  • ADR-0018 records the decision and lists the three statements of ADR-0016 it overrides at the final counter. ADR-0016 is unchanged.

Compatibility

  • Wire: none. No message or field changes. The look before a final state uses the existing PointerStateRequest and PointerFetchRequest.
  • Storage: none.
  • API: additive. New ReplicationEngine::with_pointer_service, pointer::FinalStateWitness, pointer::FinalityCheck, PointerService::attach_final_state_witness and PointerStore::remembered. Behaviour: a node refuses a second final state with Stale where it previously took one whose target sorted first.
  • Mixed fleet: a node on the previous rule still lets a smaller-target final state displace the first, and nothing on the wire tells the two rules apart. Reads follow the majority, so a transfer holds wherever most of a close group has upgraded (ADR-0018, Negative).

Semver impact

  • breaking
  • feature
  • fix

Test evidence

  • cargo test --lib --features test-utils: 1,250 passed. New tests, each of which fails when its fix is reverted (checked by reverting it):
    • Request handler: a final state the group proves already superseded is refused, named, and nothing is written; one nobody contradicts is taken; the group is asked only about a final state the node lacks; two final states raced to two nodes leave each on its first; a node restores its own lost final state without asking and names it when refusing the rival; a busy look neither takes nor refuses a final state; a proven loser is refused before its signature is checked; a cleared final state whose write fails is looked for again.
    • The look: a peer that claims a rival and stalls its fetch does not hide another peer's proof; a claim the served record does not back, a different rival, or the same rival with another target, is no proof; proofs refuse others and never replace one another; replays queued behind a look reuse its answer; looks for addresses sharing their leading bits run side by side.
    • Repair: a node holding a final state adopts nothing else; of two final states the larger side is adopted in either order, a tie adopts neither, and neither does a side that silent peers could still tie; a forged summary does not absorb the votes for a final state; a summary for another address is no answer; a record backs only the whole state a quorum named.
  • cargo test --features test-utils --test pointer_convergence: 17 passed. With one final state among the records every delivery order converges on it, exhaustively, and with two the first delivered is kept; a stored transfer is not displaced by a final state whose target sorts first, nor by any lower counter.
  • cargo test --features test-utils --test e2e pointer_replication over real QUIC: 15 passed. A transfer written to one node reaches the group and a paid final state whose target sorts first is refused by every node; a node that missed the transfer refuses a different one because a peer serves the one it holds, and still takes the group's own; the possession check does not penalise the other side of a fork and still penalises a member holding nothing.
  • Every test step CI runs, locally on macOS at this head: cargo test --lib --features test-utils 1,250 passed; e2e 113 passed, 3 ignored as on main; migration_reclaims_disk 2, migration_crash_safety 5, migration_shared_volume 5, storage_scale 2, webrtc_direct_devnet 3, poc_commitment_audit_attacks 19, poc_audit_handler_live 16, poc_bootstrap_stall 3, poc_shutdown_lmdb_drain 1, and pointer_convergence 17, all passing.
  • cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --all -- --check, cargo doc with --deny=warnings and scripts/adr-governance.py: all clean.
  • feat(pointer): hand a pointer over for good, and read forks by majority ant-client#210 runs its end-to-end pointer suite against nodes built from this branch, with real Anvil settlement.
  • Not covered here, by design: serve admission and possession fairness are fix(pointer): harden pointer replication against thin views and floods #240, and multi-record audit retention is fix(pointer): serve the record round 1 bound, however many updates follow #238; whichever of these merges after the others needs a rebase, as all three touch the pointer replication module.
  • Dev testnet: not run. That gate is the release manager's call.

New dependency

None.

ADR

https://github.com/WithAutonomi/ant-node/blob/feat/pointer-ownership-transfer/docs/adr/ADR-0018-pointer-transfer-by-final-redirection.md: docs/adr/ADR-0018-pointer-transfer-by-final-redirection.md, added by this PR (Proposed). It amends ADR-0016's merge rule at the final counter.

Mitigation / rollback

Re-pin ant-protocol to the previous rev and revert this branch. No stored record changes shape. The only coordinated part is that nodes and clients should agree on the final-counter rule, and a mixed group degrades to reads by majority, not to lost data.

@grumbach
feat(pointer): keep the first final state, and look before taking one
2a81cc0 
A pointer state at u64::MAX is now final (ant-protocol): nothing replaces
it, not even another final state whose target sorts first. That is what
lets an owner hand a pointer's address over for good, by signing one last
state that points at the new owner's pointer. The store, the admission
gate, fresh offers, repair and hints all compare with the protocol's
replaces(), so they keep whichever final state they took first with no
code of their own.

The merge rule alone leaves one gap: a node holding no final state takes
any, so a former owner could still finalize again on a node that joined
the group after the transfer, or lost its copy. So before a node takes a
final state it does not hold, from a client or a fresh offer, it asks its
close group which state each holds. A peer claiming a different final state
is asked for the record, and if it verifies -- only the owner could have
signed it -- the write is refused as stale, naming the state the group
proved. A claim alone refuses nothing, so one peer cannot block a transfer.
The look runs after payment is verified, only asks peers that have sent a
pointer message, and is bounded at four seconds inside the client's store
timeout; silence proves nothing.

Two different final states can still exist if the owner races them to
different nodes. Each node keeps its first; the client decides by the close
group's majority. So the possession check no longer penalises a member
holding the other side of such a fork -- it holds what the merge rule told
it to -- and repair, between two final states a wide group backs at quorum,
adopts the larger side rather than whichever answered first.

with_pointers now takes the service and wires both directions, so the node
and the e2e harness cannot attach one and forget the other.

ADR-0018 records the decision and amends ADR-0016's merge rule.
@grumbach
test(pointer): import what the finality tests use, rather than naming…
b8eeb58 
… it inline
@grumbach
docs(adr): name ADR-0018's pull requests
833c686
This was referenced Sep 29, 2026
Open
Open
@grumbach
docs(pointer): make the finality check's budget public, as its docs l…
21d308e 
…ink to it
@grumbach
docs(adr): leave ADR-0016 as written, and say in ADR-0018 what it cha…
834feb2 
…nges

ADR-0016 was edited in place: a backlink, the terminal-counter
conclusion, the merge rule and the ownership consequence. It is restored
exactly, and ADR-0018 now lists the three statements of ADR-0016 it
overrides at the final counter.

ADR-0018 also claimed more than the rule gives. The former owner keeps
the earlier record and the key, so it can sign a second final state at
any time, not only in a race, and any node the first has not reached
will take it. What holds is that no node gives up a final state it
holds. The claim that flooding final states pays for every round trip
was false while a verified payment is cached; the ADR now describes the
proven-conflict memory and the bounded, per-address looks that make it
true, and the restore of a node's own lost final state without a look.
@grumbach
fix(pointer): make the look before a final state hard to stall, cheap…
d8cb7df 
… to replay, and safe to restore

The finality look took claims as they arrived but fetched each claimed
rival in turn. A peer that claimed a rival and then stalled its fetch
held the look until its four-second budget ran out, and a look that runs
out finds nothing, so one peer could let a second final state in. Each
peer's claim and fetch now run as one pipeline, all at once, and the
first proof wins.

A verified payment is cached, so replaying one paid final state that
lost cost the node a signature check and a round of questions every
time. A proven conflict is now remembered, up to 16,384 addresses, and a
replayed loser is refused before its signature is checked. Looks for one
address wait their turn, at most 64 run at once, and one that cannot
start within two seconds answers Busy: a PUT gets a retryable error and
a fresh offer is dropped, neither taking nor refusing the state for
good.

A node that lost the file of a final state it held was made to look
before restoring it, so a peer on the other side of a fork could keep it
from restoring its own copy. The store now reports what it remembers,
and restoring a remembered final state asks nobody.

ReplicationEngine::with_pointers keeps the signature it has on main;
with_pointer_service wires the service both ways. ant-protocol is
repinned to the head of its companion change, which only corrects
documentation.
@grumbach
chore(deps): pin ant-protocol at the head of its companion change
c5166eb 
The companion change added documentation only: where a final state is
permanent, and that conflicts below the final counter are ordered.
@grumbach
fix(pointer): never let a second proof replace the first, and share o…
e08f9de 
…ne look among queued replays

A proven final state was stored one per address, so a later look that
proved the other side of a fork replaced the first proof, and the state
the first look had disproved could then be taken if the group went
quiet. Up to two proven states are now kept per address, and a second
never replaces the first; two different ones refuse every final state
there.

Looks for one address took turns but did not share answers, so replays
queued behind a clear look each asked the group again once it finished.
A clear look now answers the same state for ten seconds without asking,
on the runtime's clock, so a burst of replays costs one round as ADR-0018
says. The bounds live in their own type so they are tested without a
network: eight queued checks make one look, and a proof answers every
replay of the loser.
@grumbach
fix(pointer): drop a clear look when its write fails, and spread look…
c487c6a 
…s over the address's last byte

A look that found no conflict was reused for the same final state for
ten seconds, even when the write it cleared then failed: a rival that
landed meanwhile could be let in by a retry that skipped the look. The
answer is now forgotten as soon as that write does not land, on the PUT
path and the fresh-offer path, and it is reused for two seconds, as long
as a replay queued behind the look can have waited for its turn.

Looks took turns by the address's first byte, but the addresses one node
holds share their leading bits, so nearly all its looks shared one turn
and a single slow look made the rest answer Busy. They take turns by
the last byte now.

ant-protocol is pinned at the head of its companion change, which
corrected two more documentation claims.
@grumbach
fix(pointer): a stale fresh commit keeps no clear look, and a tie bet…
524ed5b 
…ween final states adopts neither

A fresh offer whose commit came back stale, because a rival had landed
first, was reported as stored, so the clear look that preceded it
outlived a write that never happened. A commit that loses now counts as
not stored, and the look is forgotten.

Repair picked the larger side between two quorum-backed final states,
but on a tie, possible in a group of eight with a quorum of four, the
state that answered first won, and two repairing nodes could adopt
opposite sides. A tie now adopts neither until the group settles.

ADR-0018 no longer claims the look keeps every final PUT inside the
client's ten-second timeout: it adds at most six seconds after payment
is verified, and a slow payment check can still outlast the timeout.
@grumbach
fix(pointer): adopt a final state only as the larger side counting si…
8e4d6f8 
…lent peers, and take only the proof claimed

Repair adopted the larger of two quorum-backed final states without
counting peers that had not answered, so in a group of eight one node
seeing four against three with one silent peer adopted the four, while
another seeing the reverse adopted the other side, for good. A final
state is now adopted only if it stays strictly larger with every silent
peer counted for its rival, seen or not yet seen.

A peer that claimed one rival final state and served another had the
served record accepted as proof. The record must now be the state the
peer claimed.
@grumbach
fix(pointer): count repair votes by the whole state, and a summary fo…
731fdfe 
…r another address as no answer

Repair grouped state summaries by identifier alone and kept the first
one's counter and target, so one dishonest peer naming a final state's
identifier with a lower counter absorbed the honest votes for it as a
non-final state and slipped past the guard for final states; the fetch
then took the real final record, matching the identifier only. Votes
now count together only for the same whole state, and the fetched
record must equal the state the quorum backed.

A summary about some other address was documented as no answer but
counted as a peer that answered holding nothing, which could hide a
silent peer that might tie two final states. It is now left out
entirely; an explicit nothing-held answer still counts.
@grumbach
fix(pointer): name the remembered final state when refusing a rival, …
e813b9c 
…and test the whole-state backing

A rival final state refused by a node that had lost the file of its own
final state was answered Stale naming zeros, since only a state on disk
was named. It now names the state the node remembers, so the sender can
see what won. The check that a fetched record is exactly the state a
quorum backed is its own predicate with a test that fails if it goes
back to comparing identifiers only.

ADR-0018 now says that a node running with no replication, a devnet or
one whose replication engine failed to start, takes a final state on the
merge rule alone, as a node whose group cannot answer does.
@grumbach
fix(pointer): a finality proof must be exactly the state the peer cla…
8f5af8d 
…imed

The look compared only the identifier of the record a peer served with
the state it had claimed, so a claim pairing a real rival's identifier
with another target was taken as backed by that rival. The record must
now be the whole state claimed, as repair already requires, and a test
fails if the comparison goes back to identifiers.
dirvine
dirvine approved these changes Sep 30, 2026
View reviewed changes

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — reviewed 8f5af8d.

No blocking findings. Traced final-state preservation through PUT, remembered-state/file-loss handling, replication and final-state witness verification. The latest deltas correctly return the remembered identifier when rejecting a rival and require the fetched witness to back the entire claimed state, not just its identifier. Their regressions are included in the passing pointer tests.

Verification:

  • Exact head: cargo test --locked --all-features --lib pointer — 95 passed, zero failures.
  • Prior head e813b9c: cargo test --locked --all-features --test pointer_convergence — 17 passed. The subsequent reviewed delta only tightens witness backing and adds its regression; do not treat the earlier convergence run as an exact-head run.
  • Companion client transfer E2E at its e813b9c node pin: two tests passed using local nodes and Anvil (handover and refusal of a second transfer after one node retained a final state).

Limits: ADR-0018 deliberately promises per-node final-state retention, not globally irrevocable ownership consensus. Churn, lost copies and mixed-version deployment remain important. Some current-head CI checks are still running/queued; this is not an all-green-CI or release sign-off.

Review scope: the three companion pointer-transfer PRs were read together. Independent GLM-5.2 review found no blockers on the earlier reviewed heads; I checked subsequent deltas directly and reran the relevant tests. Its cautions about mixed-version deployment and the lack of global ownership consensus are valid, documented limitations rather than demonstrated regressions. Codex CLI could not review because its credentials were revoked; it is not counted as a completed review. Additional source-review seats have not returned, so this is not a claim of full-panel consensus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dirvine dirvine dirvine approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@grumbach @dirvine