ePBS Implementation - #496
Draft
JasonVranek wants to merge 93 commits into
Draft
JasonVranek wants to merge 93 commits into
JasonVranek wants to merge 93 commits into
Conversation
Adds the ePBS (gloas) builder-API surface and the strict header rulings, rebased onto main so it sits on top of the SSZ rewrite (#468) and the fork-from-slot fix (#487). ePBS endpoints: - getExecutionPayloadBid, submitBuilderPreferences, submitSignedBeaconBlock, with per-builder routing by SignedRequestAuth.data (no Eth-Builder-Url). - SSZ-first request/response with JSON fallback; the bid poll ladder honors the proposer's timing headers. Header discipline (builder-specs #165): - Eth-Consensus-Version required for JSON and SSZ on the request-auth endpoints; absent -> MissingVersionHeader, present-but-unsupported -> InvalidVersionHeader, both 400. The accepted set is Gloas only; any other fork name is a client bug. - A relay bid whose response fork is not Gloas on the Gloas-only endpoint is a bad relay response: dropped, never forwarded under the wrong fork. Status + observability: - zero addressed builders accepting an ePBS submission maps to 500, not 502 (neither endpoint declares 502); new PbsClientError::NoBuilderResponse. - decode/accept rejections counted in BEACON_NODE_STATUS via record_client_error; dropped relay responses in pbs_relay_invalid_response_total{reason,endpoint,relay_id}. Legacy PBS, the websocket get_header stream (#483), and the #487 fork fix are unchanged. Suite green, clippy clean.
… auth) builder-specs types/gloas/builder_preferences.yaml declares the container as [preferences, auth]; the struct had them reversed, so its SSZ encoding and tree-hash did not match the spec. Swap the fields and re-pin the spec-vector test to the canonical examples/gloas/builder_preferences_request.ssz fixed-part bytes.
A zero-length auth.message.data is invalid per builder-specs
types/gloas/request_auth.yaml (pattern {1,4096}) and addresses no builder. It
was only caught incidentally by the relay demux and could slip through a
catch-all relay match. Add a shared validate_auth_data check (EmptyAuthData ->
400) run first in both the bid and preferences auth validators.
External KM projection tooling round-trips auth_data through match_relays_by_auth_data, so pin the behaviors it relies on: userinfo and default-port are ignored when matching a relay entry URL, configured expected_auth_data takes exact-byte precedence per relay (cross-form collision with another relay's URL bytes yields the union set), and one auth_data may match several relays sharing the same configured bytes.
…anking clamp
beacon-APIs #630 makes the beacon node the enforcer on the ePBS path: it
MUST-rejects a bid below the per-key min_bid and values every bid at
value + min(execution_payment, max_execution_payment), clamping an
over-claimed trusted payment instead of rejecting it. CB's copies of
both checks therefore over-reject: a bid the BN would clamp-and-consider
or floor-check per key was dropped outright (worse still, the old
default cap of 0 rejected every nonzero execution payment). The legacy
get_header path keeps its own min_bid check (no per-key BN enforcement
exists there).
Since CB returns a single winner, the cap survives as a RANKING clamp so
CB's winner agrees with the BN's valuation: bids rank at
value + min(execution_payment, relay cap), where the per-relay
max_execution_payment_gwei overrides the global one. The config default
becomes u64::MAX (unclamped, matching the spec's MAX_EXECUTION_PAYMENT);
a 0 default would zero out every execution payment in ranking.
Dead plumbing removed: ValidationContext.{min_bid_gwei,
max_trusted_bid_gwei}, the validate_header_data params, the
TotalPaymentTooLow/TrustedBidTooHigh error variants, and the unread
HeaderInfo payment fields.
…nfig BuilderConfigDoc/BuilderEntryDoc follow keymanager-APIs builder_config: Uint64s as JSON strings, auth_data as 0x-hex, omitted fields off the wire. CanonicalDoc compares stored docs by value (hex decoded, entries sorted, Uint64 strings parsed) since the spec promises neither entry order nor hex case. The overlay carries the operational side (advertised_url, VCs, per-mux fallbacks) that does not belong in the CB fleet config.
…unknown mux keys Three config additions for the ePBS KM builder_config flow: - PbsConfig.advertised_urls: CB's externally-reachable URLs, consumed by the ePBS pipe self-URL guard (an unconfigured key's auth_data defaults to CB's own URL; the guard must recognize it instead of self-dialing). - MuxConfig.builder_boost_factor and MuxConfig.min_bid_eth (as_opt_eth_str, same codec as the global min_bid_eth): projection-only fields consumed by KM tooling that projects the mux config into per-key keymanager builder_config docs; the PBS runtime parses them, logs an INFO note at mux load, and otherwise ignores them. - warn-on-unknown [[mux]] keys: MuxConfig is shared with the legacy get_header path, so serde(deny_unknown_fields) would reject previously-valid configs; instead the config loaders make a best-effort pass over the raw TOML and WARN on any unrecognized mux key, giving typo visibility without a breaking change.
project() turns a CB mux config plus overlay into per-key KM docs: one entry per auth_data equivalence class (identical candidate bytes: expected_auth_data, else the configured relay URL with userinfo stripped by string surgery so no normalization touches the bytes), builder_pubkeys as the class union, entries sorted by (url, auth_data bytes). The raw TOML is parsed alongside cb-common's parser because RelayEntry holds a normalized Url while the auth_data convention wants the URL bytes exactly as configured. MuxProjectionFields is the seam for the upcoming MuxConfig builder_boost_factor/min_bid fields; the overlay per_mux map is the fallback source until then. File key-loaders resolve offline; HTTP and registry loaders are skipped with a warning. Enforced at projection time: 64-entry and 64-pubkey KM maxima, auth_data 1..=4096 bytes, per-class max_execution_payment_gwei agreement, no duplicate keys across muxes.
apply: per-VC preflight (authed keystores listing plus a builder_config probe on an enumerated key, so a 404 means missing keymanager-APIs #88 support rather than key-elsewhere), POST per projected key with 202/404/ 403 handling, exit non-zero unless each key is accepted by exactly one VC (zero acceptors errors; several is a duplicate-key slashing alarm). --dry-run prints, --emit writes per-key JSON plus a manifest, --prune POSTs {} (spec-equal to DELETE) for stored-but-unprojected keys. check: read-only canonical comparison with tiered findings and --fail-on; fields the projection omits resolve to VC config on GET and are skipped rather than flagged as drift.
axum mock VC covering: partitioned-key happy path (POST body equals the projected doc), 401 preflight, missing-#88 detection, zero-acceptor non-zero result, duplicate-key slashing alarm, 403 config-file-managed, prune body exactly {}, dry-run/emit sending nothing, canonical check (reordered entries + uppercase hex + VC-resolved fields produce no drift), drift + unroutable auth_data + duplicate-key findings.
An unmatched getExecutionPayloadBid no longer 400s outright: the decoded auth_data URL is dialed via a transient RelayClient through the shared send path, guarded fail-closed against CB's own advertised_urls. Bid sigverify is skipped per-context for the pipe relay only; the global skip_sigverify and configured-relay verification are unchanged.
…r URL Same demux, same semantics as the bid endpoint: empty match -> self-URL guard -> transient client through the shared send path, expecting the builder's 202.
…e lax-URL equivalent CB's url_matches ignores userinfo, host case and the default port, so two byte-distinct classes it treats as one relay would split builder_pubkeys across KM entries and the VC would silently reject the missing pubkey's winning bids. Conservative tool-local mirror; CB-side contract tests pin the real matcher.
v1 CB is a pure pipe: a stored builder URL outside the configured relays still routes, so it is operator-notable, not a failure.
…L output The log-settings pre-load went through CommitBoostConfig::from_env_path, so every service warned about each unknown mux key twice (once for the logs load, once for the primary config load). Route the logs load through a silent variant so each load path warns exactly once. Also skip serializing builder_boost_factor and min_bid_eth when None so a MuxConfig re-serialized to TOML round-trips cleanly.
…support A transport error or 5xx on the probe is its own error, never evidence of no support; an empty-keystore VC warns and POSTs anyway (the 404/405 responses answer the support question).
Url::join with an absolute path drops a /prefix base; concat on the trimmed base keeps it.
Group read (0o040) leaks the bearer token the same way world read does.
The mux_ext seam was written against pre-merge cb-common; now that builder_boost_factor and min_bid_eth exist on MuxConfig, the accessors read them directly (precedence: mux field > overlay > global).
Add optional PbsConfig fields gloas_fork_version (0x-hex, 4 bytes) and genesis_validators_root (0x-hex, 32 bytes) and thread them into the ePBS bid signature verification. Unset = the built-in constants, so existing configs verify exactly as before; devnets/networks whose gloas fork version or genesis root differ can now pass bid sigverify without skip_sigverify. Deriving these from the fork schedule is tracked separately.
Add projection-only PbsConfig fields min_bid_p2p_eth and builder_boost_factor_p2p (consumed by KM tooling, not the PBS runtime; INFO-logged when set). When set, the projected KM doc's KEY-LEVEL min_bid/builder_boost_factor come from them: projected entries always carry explicit per-entry values, so the key level only governs p2p bids and entries that omit their own. Unset keeps the uniform projection (key level = mux/global values) so existing docs are unchanged.
…-bottleneck-fixes
…pbs-bottleneck-fixes
The transient pipe forwards a bid/preferences request to the builder URL the proposer's signed auth_data names when no configured relay matches. It fails closed without advertised_urls: CB cannot tell an unconfigured key's self-URL default (which points at CB itself) from an external builder, so it will not dial -- a deliberate, SSRF-conscious opt-in. The defect was that it did so SILENTLY, rejecting with a bare 400 and no hint that one config field enables it. Now CB warns once at startup (when muxes are configured but advertised_urls is empty) and logs an actionable one-time hint on the first rejected pipe request. No behavior change: the pipe stays opt-in via advertised_urls.
The fee_recipient removal dropped the top-level Address import, but a mock in the test module still uses it. cargo check (non-test) did not flag it, so the cb-pbs test build broke. Import Address in the test module.
proposer_deadline_buffer_ms and strict_block_decode were added to PbsConfig but several test-only struct literals build it field-by-field, so the test build broke (cargo check does not compile the test cfg, so it was not caught earlier). Fill in the two fields in the signer, tests-utils, and cfg-file-update literals.
…ient tests Zero budget (the proposer's Date-Milliseconds + X-Timeout-Ms deadline, minus the buffer, has passed) means any bid would land too late for the beacon node to use, so return 204 without a doomed relay call. This honors the proposer's own deadline, not a CB late-in-slot cutoff, so it is not the preemptive skip the bid path used to do. Update the bid-path tests for the accepted behavior changes: fee_recipient is no longer enforced on the ePBS path (repurpose the wrong-fee-recipient test to assert the bid is served; drop the mux-fee-recipient test, which asserted the removed mux-level enforcement).
…ults /beacon_blocks decoding is now opt-in (strict_block_decode): the json-202, non-gloas-400, and unsupported-content-type-415 tests exercise the decode path, so enable strict mode in their configs. And builder preferences are no longer gated on slot age (blind pipe), so repurpose the slot-passed-400 test to assert a past-slot preference is forwarded and accepted.
Formats the ePBS bottleneck + transient-pipe work, and picks up five files with pre-existing drift under the pinned toolchain (types/mod.rs, signature.rs, apply.rs, mock_km.rs, mock_relay.rs).
# Conflicts: # Cargo.lock # Cargo.toml
…ipient knob - max_execution_payment_gwei is now Option<u64> (None = unclamped), mirroring the relay-level override. The old u64::MAX default was documented as a literal that TOML cannot parse, so copying it from config.example.toml bricked startup. - Add an unknown-field scanner for the [pbs] table (KNOWN_PBS_FIELDS): PbsConfig is flattened into StaticPbsConfig and cannot use deny_unknown_fields, so a typo'd knob (e.g. a security flag) parsed clean and silently stayed default. It now warns, matching the [[mux]] discipline. Wired on the default and custom-module load paths. - RelayConfig::validate rejects a zero bid_poll_timeout_ms / frequency_get_header_ms (a stall, not "unset"); proposer_deadline_buffer_ms is capped at one slot. - Document strict_block_decode in config.example.toml. - Remove fee_recipient: the field was read nowhere (the BN filters bids by fee_recipient for ePBS), yet its doc claimed CB rejected mismatched bids. Drop the field, the mux override, the dead trait accessor, and the tests that pinned non-enforcement.
In the default blind pipe CB forwards the reveal bytes unparsed, so the reveal must be SSZ. A JSON or otherwise non-SSZ reveal was relabeled octet-stream and failed opaquely at the builder (a 500 with no explanation). It is now rejected up front with 415. The Content-Type is resolved with the builder-specs default (JSON when the header is absent), so an unlabeled reveal is treated as JSON and rejected too, not assumed to be SSZ.
require_consensus_version_header accepted Gloas | Heze while its own doc said GLOAS ONLY. Narrow it to Gloas: a later fork's ePBS semantics are not yet validated here, so it is 400'd rather than silently handled as gloas. The match stays exhaustive with no wildcard, so a future lighthouse fork stops it compiling and forces an explicit decision to widen the set.
…g helper Remove comments that restate the adjacent, self-documenting code, correct two doc comments that no longer matched their code (the epbs_base_send_headers fork note and the mock received_auth field), fix a run-on test comment, and delete the unused content_type_encoding wrapper (only content_type_encoding_with_default is called).
…elay_id, shared client) The transient pipe dials a builder URL taken straight from untrusted auth data, so before building the client it now resolves the target and refuses any address in loopback / private / link-local / CGNAT (100.64.0.0/10) / v4-mapped-internal space, failing closed on a resolution error (PbsClientError::PipeTargetBlocked -> 400). A DNS-rebind (TOCTOU) between the check and the dial is a disclosed v1 limitation. The check is compiled and active in every build; only the `testing-flags` feature can skip it (a thread-local a test sets) so the e2e suite can dial a local mock. The pipe relay id is now a fixed "pipe" sentinel instead of the attacker-supplied URL host, which was an unbounded Prometheus relay_id cardinality vector. The pipe reuses a single shared reqwest::Client (built once in PbsState) instead of constructing a fresh one per bid - matching how configured relays already reuse their client and keeping the connection pool warm across slots. Adds RelayClient::with_client for that; resolve_addressed_relays / transient_pipe_relay are now async for the resolution step.
The proposer-supplied X-Timeout-Ms sets how long CB solicits a bid, but it had no upper bound: a large value pins an outbound relay connection open and grows the timing-games poll ladder for that whole duration. Cap the budget at one slot (chain-aware) before deriving the timeout, bounding the duration a single request can hold a relay connection and the ladder depth.
…dvertised_urls `cb-km check` now flags the self-dial misconfiguration the docs warn about: every advertised_url km-tool projects is echoed by a VC in its auth data, decoded by CB, and must be recognized as CB's own via [pbs] advertised_urls. If advertised_urls is set but does not cover a projected URL, a bid addressed to it decodes to CB's own URL and self-dials recursively. Since the operator runs `cb-km check` as the gate before apply, an uncovered URL is an error. Mirrors cb-pbs `url_matches` locally with a drift-guard test.
The three ePBS handlers carried the same error arm (4xx warns, 5xx errors, count the status) and the two write routes the same spawn-and-join fan-out. Both move to cb-pbs utils as `record_request_failure` and `join_detached_sends`; `expect_status` is inlined into its only caller. submitBuilderPreferences encodes the SSZ body once and shares it across the sends instead of cloning and re-encoding the request per relay. submitSignedBeaconBlock no longer re-stamps Eth-Consensus-Version with the parsed fork: the header validator accepts Gloas only, which is the value the base headers already carry.
…eader getExecutionPayloadBid parsed the Accept header twice: once for the response encoding and again to build the relay-side Accept. It is parsed once and threaded through; `preferred(&[Ssz, Json])` always equals the primary, so `encode_bid_response` takes an `EncodingType` and its unreachable 406 arm goes. The relay Accept is one of two static headers, so `build_outbound_accept` and its q-value ladder (only ever fed two entries) are deleted, with the JSON-first static pinned by a test like its SSZ-first twin. Also on the bid path: the request start time returned by the single poll was never read, and was the only reason `select_max_bid` was generic over its label; `ms_into_slot` and the per-relay ranking cap are computed once. The handler adopts the shared failure arm.
The variant held a u16 that `status_code` converted back with a 502 fallback its only constructor could never reach. It holds the `StatusCode` directly.
`load_pbs_config` and `load_pbs_custom_config` carried the same advertised_urls warning; it moves to one helper, and the runtime once-warn in the pipe stops restating it. km-tool's `resolve_min_bid` is inlined at its only call site.
The bid suite hand-rolled 23 PBS boots and the preferences suite two; they run through `spawn_mock_relay` + `setup_pbs`, which the existing `setup_relay*` helpers now delegate to. The default JSON bid request and the hand-built spec URL each get one helper, the proposer pubkey literal becomes `TEST_PROPOSER_PUBKEY`, and two clippy nits in the rewritten file are fixed. The mock relay computes its served fork and signing key once.
…ost ones With `direct_entries = true` in the overlay, every Commit-Boost entry gets a twin at each builder's own URL (userinfo stripped) carrying the same auth_data, min_bid, boost and execution-payment cap. The beacon node then asks the builder directly as well as through Commit-Boost, so it still gets bids when Commit-Boost is down. It lives in the overlay rather than on the command line so apply and check always project the same docs. Relays grouped only by a shared expected_auth_data keep one direct entry per builder URL, the same builder listed twice gets one, and direct entries count against the 64-entry keymanager cap. Entries are now sorted explicitly by (url, auth_data).
The pipe dials a URL taken from untrusted auth data. Its SSRF guard resolves and checks only the first hop, but the client was built with reqwest's default redirect policy (follow up to 10), so an allowed public target could answer 307 and steer CB into loopback or link-local space. Internal response bodies then reach the logs through PbsError::JsonDecode. Set Policy::none(), matching the keymanager client. A builder that needs a redirect is a misconfiguration, not a bid path.
test_get_execution_payload_bid_deadline_clamps_ladder asserted exactly two polls for a 400ms budget at a 200ms cadence. Under parallel load transit eats the second rung, so it failed 3 of 24 concurrent runs. The ladder's actual contract is the comparison that follows - a larger deadline buys more polls - which was stable throughout. Bound the count and keep a lower bound so the assertion cannot pass vacuously.
- url_matches existed twice, byte-identical, in cb-pbs and km-tool. It is pure Url comparison, so move it to cb-common beside the relay types and drop both copies along with the duplicated test. - MAX_BUILDER_AUTH_DATA_SIZE was a typenum alias in cb-common and a re-typed usize literal in km-tool, so the spec limit could move on one side only. Derive the length from the alias and import it. Also removes the #[allow(non_camel_case_types)]. - The three sign_*_root functions shared one body and differed only by domain, and spelled the object root two different ways. One helper, three wrappers. B256::tree_hash_root is identity, so the signing roots are unchanged. - deserialize_body inlined content_type_encoding_with_default, the helper added directly above it, and repeated its doc verbatim. - relay_headers was parsed twice on the RelayClient::new path. - Dead: KmClient::base and RelayAuthCandidate.projected had no readers outside a test asserting the value it had just been given.
Eighty-odd commits of spec churn left comments describing mechanisms that no
longer exist: a layer-3 catch-all in the auth-data demux, a preemptive
late-in-slot skip, auth forwarded 'byte-for-byte' where the send re-encodes,
and a decode_signed_beacon_block doc claiming an SSZ default while the code
passes JSON and its own body comment says JSON. Those actively misinform, so
they are corrected rather than deleted where the surrounding fact still matters.
The rest is bulk: docs that restate the function they sit on, rationale
duplicated two or three times within a few lines, narration that dates itself
('today's behavior', 'on this branch', 'for now'), and references to a working
document no outside reader can resolve. Field docs on new config knobs are
trimmed to the non-obvious constraint, since config.example.toml carries the
operator-facing prose.
Also: drop a dead duplicate header constant, use NO_PREFERENCE_DEFAULT where
its value was hardcoded, unqualify paths the file already imports, restore an
alphabetical dependency run, and uncomment strict_block_decode to match the
file's convention for [pbs] bools.
…secrets, rustls bump Brings in #502 (submit-block relay error logging), the v0.11.0-rc2/rc3 releases with the rustls advisory bump, relay headers read from a secret file or env var (#498), and the ws stream metrics endpoint (#499). Conflict resolutions: - relay.rs: kept epbs's relay_headers() helper, shared by the HTTP client and the stream handshake, and moved main's header resolution into it (HeaderSource literal/file/env, sensitive values, the secret-source log line). Its error messages now format the key, which epbs's did not. - config/pbs.rs: one test module holding epbs's unknown-[pbs]-field tests and main's header-source tests. - config/utils.rs: main's RELAY_URL test constant in test_env. - Cargo.lock: main's lockfile, plus only what epbs adds.
Each was checked against the code it names. Most duplicate a surviving test with the same input and assertions (the survivor is named); the rest still pass with the behaviour they claim to check broken. 573 -> 547 tests, all passing. Cannot fail: - test_get_execution_payload_bid_malformed_auth_400: is rejected for its missing Eth-Consensus-Version header and never reaches the malformed-body path - test_outbound_accept_json_first: asserts our own from_static literal back, not a spec value - test_submit_builder_preferences_pipe_self_url_not_dialed: passes with the self-URL guard disabled: the SSRF guard rejects 0.0.0.0 first - mux_config_none_fields_roundtrip: passes with every skip_serializing_if removed: toml drops None fields itself Duplicates (survivor): - test_get_execution_payload_bid_highest_wins (test_get_execution_payload_bid_highest_total_payment_wins) - test_get_execution_payload_bid_execution_payment_within_cap (test_get_execution_payload_bid_execution_payment_over_cap_accepted) - test_inflight_selection_prefers_max_total_not_latest (test_select_max_bid_by_total_payment) - validate_auth_data_requires_nonempty_data (validate_builder_request_auth_rejects_empty_data) - transient_pipe_relay_fails_closed_without_advertised_urls (test_get_execution_payload_bid_pipe_requires_advertised_urls) - test_submit_builder_preferences_json (test_submit_builder_preferences_json_wire_form) - test_submit_builder_preferences_auth_data_mismatch_400 (test_submit_builder_preferences_unmatched_opaque_auth_400) - test_submit_builder_preferences_auth_data_match (test_submit_builder_preferences) - test_submit_builder_preferences_two_relays_one_accepts_202 (test_submit_builder_preferences_one_202_one_400_is_202) - decode_rejects_an_empty_body (test_decode_builder_request_auth_rejects_empty_body) - decode_rejects_json_without_the_version_header (test_submit_builder_preferences_json_wire_form) - stripped_userinfo_no_trailing_slash_in_auth_data (projects_literal_json_doc) - mux_p2p_override_wins_over_global (mux_p2p_override_and_inherit_mix) - global_p2p_applies_to_all_muxes (mux_p2p_override_and_inherit_mix) - p2p_fields_unset_keep_uniform_projection (projects_literal_json_doc) - boost_omitted_without_source (projects_literal_json_doc) - distinct_hosts_do_not_lax_collide (projects_literal_json_doc) - empty_doc_serializes_to_empty_object (prune_posts_exactly_empty_object) - parses_minimal_overlay (projects_literal_json_doc) - our_identity_wins_over_stored_resolved_version (preserve_entries_collision_is_replaced_not_duplicated) - merge_exceeding_max_entries_fails (preserve_entries_over_cap_fails_without_posting) - endpoint_without_prefix_unchanged (apply_partitioned_keys_accepted_once_each)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.