CAS improvements - #2300
Merged
Merged
Conversation
…blication Two defects surfaced by the `content_addressed_garbage_collection_log` scenario cards for issue #2233. Any `S3_ERROR` timeout during a GC round was recorded as an indistinguishable `Failed` outcome with a free-text error, and a failed round zeroed out the real counters and cleared `i_am_leader`, suppressing the heartbeat and provoking leadership ping-pong on a flaky backend. Transient error codes (`S3_ERROR`, `NETWORK_ERROR`, `ABORTED`, timeouts, `MEMORY_LIMIT_EXCEEDED`) now produce an `Aborted` outcome while keeping leadership, and `system.cas_gc_log` gains an `error_code` column alongside the `Aborted` outcome. Separately, the emulated blob-publication path materialized the whole blob body in memory (about 1 GiB for a 512 MiB blob) under a global mutex; it now streams the body instead. Related: #2233 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…_ERROR A fetch-by-relink that loses the offer-to-confirm race -- the source's ref moved (a merge, a mutation, an outdated-part drop) between the offer and the confirm -- is a designed, fail-closed outcome: the receiver abandons the relink and the replication queue retries, re-selecting the source and the covering part. It was thrown as `NETWORK_ERROR`, which misdescribes it three ways: - both queue executors (`processQueueEntry`, `ReplicatedMergeTreeQueue`-driven `ReplicatedMergeMutateTaskBase`) treat `NETWORK_ERROR` as an unclassified failure, so every refusal printed an Error-level log line with a full stack trace; issue #2219 records a multi-hour false triage chasing a network fault that was never there (up to 53% of relink proofs refuse under small-part load); - stateless `part_log` hygiene checks tolerate the fetch-transient class under the code upstream fetches use for it, `NO_REPLICA_HAS_PART` (e.g. `02265_column_ttl` whitelists exactly that code), so a refusal landing in `part_log` as `NETWORK_ERROR` fails them -- this is what broke `02265_column_ttl` in the CAS lanes on PR #2159 (13/14 reruns under `prefer_fetch_merged_part_size_threshold=1`); - the label suggests retrying the transport, while the one recovery that is unsound here is a byte re-request to the same source. Both relink retry-later throw sites (taxonomy row 3, the confirm refusal, and row 5b, the unresolved promote) now throw `NO_REPLICA_HAS_PART`. The queue behavior is unchanged -- the exception is stored on the entry, backed off, and re-executed -- but both executors demote it to INFO with no stack trace. Unlike `ABORTED` (the other demoted code), it keeps `need_to_save_exception`, so a refusal storm stays visible in `system.replication_queue`; `ABORTED`'s save-nothing shape is the known pathology where a refusal loop runs invisibly with no backoff accounting. `test_confirm_refuses_when_source_dropped_in_window` now pins the classification: the refusal must not appear at Error level, must appear at Information level, and must reach `part_log` only as `NO_REPLICA_HAS_PART`. No message text changed; no generic queue code changed. Closes: #2219 Related: #2159 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…eys renamed yet) Behavior-preserving groundwork for an atomic rename of the CAS wire-format JSON keys from abstract letters (`t`, `k`, `s`, ...) to semantic names (`kind`, `outcome`, `state`, ...), landed as its own phase so the rename itself is a single reviewable diff. Adds `WireKey` and per-encoding field write helpers, and `EnumWireTable` — a table pairing each enum value with its wire word, proven complete against the enum by a set-equality coverage check with a failing witness for every member. `kMinBlobHeaderLen` gets one compile-time owner instead of several hand-kept constants. `TokenType`, `ObjectKind`, and `BlobHashAlgo` move onto `EnumWireTable`, and the blob-meta, pool-meta, GC state/heartbeat/ maintenance, server-root, blob-envelope, ref-log/ref-ckpt/ref-snapshot/ ref-catalog, run, fold-seal, and gc-outcomes codecs are all migrated onto the carriers — every one of them still writing its existing wire spelling. `RunMarker` becomes a typed enum, and the format test battery is closed out with a set-equality check over the codec registry. No wire-format bytes change in this phase; the follow-up phase (next commit) performs the actual key cut. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The actual key rename, on top of the phase-1 carrier infrastructure. The
format generation history is first reset to a `{1, 1}` baseline, since CAS
has no released, persisted data yet and pre-release generations exist only
to prove the evolution machinery.
Every CAS wire format switches its JSON keys from single-letter/abbreviated
spellings to descriptive names in one pass: the shared `BlobRef`/`Token`/
`ManifestRef`/binding fields, `cas_blob_meta`, `cas_pool_meta` (`algos_used`
becomes a JSON word array instead of a bitmask), GC state/heartbeat/
maintenance state, the server-root record (`MountLease::min_active` becomes
`min_active_build_sequence`), `cas_ref_ckpt`, `cas_ref_log` (the seal link
becomes `!prev_epoch`/`!prev_seq`), `cas_ref_snapshot`, `cas_part_manifest`,
`cas_run`, `cas_gc_outcomes` (`kind`/`outcome`), the fold-seal record and its
`CoverageClass` words, the blob descriptor (with its 239-byte worst case
proved at compile time against the 240-byte floor), and `cas_ref_catalog`.
Golden tests are re-pinned to the new bytes throughout.
Token-group requiredness is unified through `TokenFields::build`: an outcome
missing its token now fails closed instead of serializing a partial group.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…enchmark Small correctness and review follow-ups to the wire-key cut: the part manifest names its namespace field the way every other object does, the algorithm set is read from the proven `EnumWireTable` instead of two independent hand-kept lists, the GC lease and heartbeat keep their separate owner spellings (documented, not merged), and the wire-format word writer gets the contract it was always assumed to have. Extends the `benchmark_cas_ref_protocol` harness to cover every format and direction the wire-keys design measures, plus a review-round fix to that harness. Also fixes `c++expr`: the generated work function needs internal linkage, without which ClickHouse-mode compilation did not work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…speedup Full before/after measurement of the wire-key cut found decode of four of the five formats barely slower, with `cas_fold_seal` the exception (its short strings make the longer keys dominate). Chasing that, the JSON object reader and the per-format row reader are now reused across a stream's rows instead of rebuilt for each one, cutting decode time 57-81% (53-79% net of the key-length cost). A separate copy-free string-read attempt was measured at a 6-7% regression on `cas_ref_catalog` and is not included here. Also lets the full stateless test suite run locally: `functional_tests.py` turns on verbose output for the dataset-attach step (so a `DNS_ERROR` that only fires outside CI doesn't get swallowed and misread as a Kafka failure downstream) and extends the "skip stateful tests when running locally" guard to a local run with no test selector at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ALTER TABLE ... EXPORT PARTITION` was refused with `SUPPORT_IS_DISABLED` on a content-addressed disk because it is absent from the partition-command allowlist in `MergeTreeData`. The rejection it fell into says the command "clones parts file-by-file with no transaction, which would corrupt the clone", and that reason does not describe exporting. `ExportPartTask` reads the source part through `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) under `readLockParts` and writes rows into the destination through a `SinkToStorage` on an ordinary query pipeline. Nothing is hard-linked or copied on the source disk; the command's own bookkeeping is in ZooKeeper. So the allowlist was rejecting it by omission rather than by an argument that applies to it, which the code around it already half concedes: `EXPORT_PARTITION` is listed among the commands permitted to target `PARTITION ALL` a few lines above. Verified end to end rather than by inspection, on a server built from this change: a `ReplicatedMergeTree` source on a CAS disk holding (1,2020), (2,2020), (3,2021), exported to an `IcebergLocal` destination. `EXPORT PARTITION ID '2020'` succeeds and the destination holds exactly (1,2020) and (2,2020) — the right partition, and the 2021 row correctly absent. Two limitations surfaced on the way and are NOT addressed here, because neither is about CAS. Export is implemented only for `ReplicatedMergeTree`: a plain `MergeTree` source now returns `Code: 48 NOT_IMPLEMENTED` instead of the CAS refusal, so the reproduction in the report — which uses a plain MergeTree — will still fail, just for its real reason. And the operation remains behind the server setting `allow_experimental_export_merge_tree_partition`. Closes: #2291 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KiHKrvEVy8u4nA1A8qYFUY Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
… answer On a generation-dialect (GCS) mount, `ObjectStorageBackend::checkPoolPreconditions` refused to mount both when the bucket was verified to have versioning enabled and when the probe simply couldn't get an answer. The first live run against a real GCS bucket hit the second case: the service account lacked `storage.buckets.get`, `GetBucketVersioning` returned 403, and every writable CAS mount on that bucket failed with `NOT_IMPLEMENTED` at server start — a missing IAM grant turned into a hard outage, even though an unreadable bucket configuration is not evidence the bucket is actually versioned. The probe now logs a warning naming what it couldn't verify and how to fix it (grant `storage.buckets.get`, or confirm by hand) and lets the mount proceed. A bucket confirmed versioned still refuses, because a token-exact `DELETE` there archives a noncurrent generation instead of reclaiming storage. That first credentialed run against Google (HMAC groups) also surfaced three test-suite assumptions the real service doesn't meet (`system.cas_log.token` isn't always a numeric generation for build-lifecycle rows; a second `COUNT()` over a Parquet object is answered from the per-file row-count cache, not the Parquet metadata cache; a disk over an absent bucket refuses at `CREATE TABLE`, not the first `INSERT`) and one open question — whether process-wide `system.events` deltas can be attributed to one statement when a mount-lease renewal shares the same counters. The suite now attributes every counter it asserts through `system.query_log.ProfileEvents` instead. Also documents GCS's request-rate limits in the CAS bucket requirements. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… whole namespace Same-pool replication transfers only a part's manifest: the receiver publishes its own ref over the sender's blobs, then asks the sender a read-only question — "do you still hold exactly this manifest for this part?" — and promotes only on `Yes`. Rule 3 of `CasRefLedger::confirmExactRef` answered `Unknown` whenever ANY mutation of the same namespace was queued, in flight, or awaiting its checkpoint frontier, not just a mutation of the asked-about ref. On the live GCS stand, two replicas answered each other `Unknown` almost every time for forty minutes: every replica is also a receiver, and each failed fetch appends two records to its own lane (a precommit, then its removal on abort), so under load neither side ever observed the other's lane quiet. Both replication queues wedged at 1.5-1.7k entries, the replicas diverged to 123k against 166k rows, and the soak died on `SYSTEM SYNC REPLICA`. Nothing was lost — once one side stopped fetching, the other drained in two minutes — but the lane-wide refusal made every sustained-write workload look like data loss in progress. On RustFS in a LAN the window closing this fast never showed the defect; GCS limits checkpoint publication to about one mutation per second per object, so the window is long enough to matter. Rule 3 now refuses only when the asked-about ref itself has a queued or in-flight mutation, via `RefTableRuntime::carved` mirroring the tenure's carved items and validating a ref-scoped item's ops against its `MutationScope` before durability. Covered by a two-node liveness case against a fake GCS with delayed `_ckpt` writes (`test_cas_gcs_relink_liveness`), and every confirm refusal is now attributed and counted rather than silent. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A manifest id names one content, ever: `stageManifest` mints it from `(writer_epoch, build_sequence, next_manifest_ordinal++)` — all three either a durable counter bumped by a conditional write or strictly increasing per process — and writes the body exactly once. The only other mutations of a manifest are exact-token deletes (writer cleanup, GC's owner-removed cleanup, the orphan sweep). So the token carried in `ManifestCacheKey` distinguished nothing, and the `HEAD` that supplied it (`CasManifestReader::readManifestShared`) was a per-read check of a GC-side invariant, not of the cached content's validity — it cost one serial round trip per uncached or `ForceFresh` access and could only ever detect a protocol violation (something deleting a manifest the ref graph still names), never serve wrong bytes if removed, since id-to-content is a function. The cache now keys by `ManifestId` alone: no `HEAD` on a hit, exactly one `GET` on a miss. Detection of a dangling reference moves from "the next read" to "the first uncached read, or fsck". The `part_folder_validate` setting, which existed only to pace that now-removed `HEAD`, is retired along with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Six review rounds across two earlier designs found nine classes of defect in how CAS
handled its conditional-write tokens, eight of them tracing to one root: `Token` was
`struct { String value; TokenType type; }`, anyone could construct one from anything, and
the backend accepted whatever it was handed. Concretely this let an empty value pass as a
token (and on S3/Azure an empty `If-Match` is *omitted*, so a fenced write silently becomes
an unconditional overwrite), let a write commit against a token that was really the result
of a later, unrelated `HEAD`, and let `TokenMismatch` — documented as remote evidence that
another incarnation is current — be returned for what was actually a local refusal, with GC
acting on it and mislabelling live blobs `Replaced`. Every earlier revision patched one
symptom at one call site; the next review round found the same root through a different
one. A second, independent waste rode along: `Backend::get` always issued both a `HEAD` and
a `GET`, though a `GET` already returns everything a `HEAD` does plus the body — doubling
the request cost of every control-object read.
This introduces the replacement, starting with its core (the migration of every CAS
subsystem onto it is the next commit): `Backend` becomes a string-in/string-out transport
callable only through a `TransportAccess` key; `Incarnation` replaces the free-form `Token`
as a type that can only be minted by the backend from an actual store response;
`CasRequests` owns a backend and a `Fence`, and `admit()`/`resume(generation)` hand out a
`CasOperation` carrying the admitted generation and an optional liveness predicate. Every
verb on that operation (`read`, `head`, `list`, `remove`, `publish`, `create`, `replace`,
`readModifyWrite`, ...) takes a `Retry` policy; the engine re-checks admission before every
attempt, before every sleep, and once more after a proven commit, settles every conflict and
ambiguity by one exact read, and reports one of `Committed | Declined | Conflict | Refused |
GaveUp` — never an exception for an ordinary lost race. An upstream slice under `src/IO`
and `S3ObjectStorage` adds a `SingleAttempt` request mode so a marked `GET` answers with the
same incarnation identity a `HEAD` does (closing the two-request cost) and a reissue that
gets back a different ETag is treated as body drift, not silently accepted. The old
controller stays in place during the migration; the next commits delete it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Follows the previous commit's engine introduction by moving every production caller off the old ad-hoc backend controller and onto `CasOperation`: pool bootstrap (sentinel probe, capability probe, plain objects, pool meta, manifest and ref-protocol readers), GC (maintenance state, namespace janitor, decommission, the GC core's lease/heartbeat/commits/ folds/persisted redelete), part-write (blob meta, the part-write transaction, create-first marker reconciliation), the ref lane (catalog and checkpoint publisher, namespace creation lifecycle, the resumed-operations arms, the catalog erase loop), and mount (renew, farewell, claim, epoch allocation, the heartbeat floor, remount re-anchoring). `PersistedIncarnation` replaces the ad-hoc token in the wire vocabulary, the record stream, the outcomes and the condemned rows. Each subsystem's move keeps its behavior but inherits the engine's guarantees for free: every write is admitted under a fence and re-checked before each attempt/sleep/commit, every conflict is settled by one exact read instead of an assumed outcome, and a credential refresh mid-attempt is never mistaken for a landed write. Along the way this fixes real bugs the engine surfaces mechanically rather than by inspection — e.g. two double-counting fault-injection doubles in the GC maintenance-state path, and several sites that treated an unobserved conflict as corruption instead of "vanished or a competing leader also wrote". The bulk of the diff is the matching migration of every test double (the `cp4` series) off the legacy backend overrides and onto the primitives the production code now actually calls — direct-Backend doubles for the primitives, virtualized clocks for every retry/backoff path that used to sleep for real, and fault injection that latches instead of pinning `max_attempts`, so a shut gate can no longer hang the test binary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… the engine, delete the old controller
Two closing passes over the request-engine migration. First, a naming decision: `Incarnation`
becomes `Etag` (`PersistedIncarnation` -> `PersistedEtag`, `CasIncarnation.{h,cpp}` ->
`CasEtag.{h,cpp}`) and `TokenType` folds into a `Dialect` alias, because the class was
carrying its wire field's name rather than its actual role — an ETag on S3-compatible
stores, a generation on GCS's JSON dialect, a minted sequence value on the emulated
backends. This does not touch the blob envelope's `incarnation_tag` or the catalog's
incarnation namespace, which are unrelated concepts the rename exists to stop colliding
with.
Second, the entire gtest suite (~120 files) moves off the legacy backend overrides and onto
`CasOperation`/`CasRequests`, naming the etag and the listed key the way production code now
does. With every test migrated, the old controller and its 1500-line test file
(`gtest_cas_request_control.cpp`) are deleted outright — this was the last thing keeping it
alive. `MountLeaseKeeper` is renamed to `MountLeaseRenewer` in the same pass (it renews a
lease; it is not ClickHouse's Keeper, and the old name kept reading as if it were).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…gration Small correctness and hardening fixes found while the engine migration was under review: a read now ends only on an authoritative absence, not on every unretryable code; a raced `claimMount` reports the occupant it actually observed instead of its own proposal; an absent-key read through the request engine is no longer logged as an error (it's an ordinary outcome the engine already models); a hand-written retry loop now freezes one deadline and shares it across every call it makes, instead of re-deriving it per call; and an unobserved conflict is named for what it actually is — a vanish or a competing leader, never assumed corruption. The bulk of this is test hardening that follows from the engine actually enforcing pacing and admission where the old ad-hoc calls didn't: transport-fault doubles now inject `Poco::TimeoutException` (what production code actually throws) instead of `std::runtime_error`; several tests that asserted a schedule the engine never promised, or counted requests instead of asserting an outcome, are corrected; retry/backoff-dependent tests get their own virtual clock so they assert the engine actually reissued, rather than timing a real sleep; and the throttling coverage gate gains both a unit and an integration leg. Two properties orphaned by the old controller's test-file deletion (previous commit) are restored under the new API. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…gc_read_concurrency) The fold's `fold_ref_intake` and `fold_reduce` phases issue their checkpoint, walk-position, manifest-edge and zero-in-degree-HEAD reads one at a time, in the round's own decision order — a live-GCS soak measured `fold_ref_intake` at 2303 s of a 4352 s phase wall (53%), and a separate finding recorded one fold round holding the GC lease for hours on a real bucket, still unfinished after 97 minutes. `GcReadAhead` sits in front of the fold's one admitted `CasOperation`: callers hint keys the sequential walk will need next, workers fetch them on a bounded pool under the same admitted generation, and the walk takes results at exactly the sites and in exactly the order it reads today — no decision, decode, counter or event moves off the round thread. A key nobody hinted is still read inline. Concurrency 1 issues no hints and is byte-for-byte today's behavior; the new `cas_gc_read_concurrency` setting is plumbed like `gc_meta_pool_size` and refused at 0 like `gc_shards`, with three `ProfileEvent`s for hits, misses and wasted results. Measured against a fixed per-request latency: `fold_ref_intake` 2.4x, `fold_reduce` 1.2x, the round overall 1.65x. Intake's speedup stops there because the round issues ref-log and manifest `GET`s one to one and a manifest key is only known once its log is decoded — that chain, and the graduation gate's inline meta re-check, are recorded as follow-up items. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Fetch-by-relink existed but was opportunistic: the receiver advertised ONE guessed pool identity before it knew where the sender actually kept the part (the caller's `dest_disk` if content-addressed, else the first content-addressed disk of the table's storage policy), then reserved the target disk the ordinary way — the TTL move rule's destination, `balancedReservation`, or the first volume with space — and accepted the relink offer only when the reservation happened to land on a disk of the advertised pool. Otherwise it re-requested the bytes. So a part already in the shared pool moved as bytes whenever the policy's placement disagreed with the guess: a tiered policy whose local volume comes first, a TTL rule naming the local tier for a fresh part, a policy holding two pools with the sender's in second place. The relink is the whole point of a shared pool — a fetch should move no bytes — and the storage policy could veto it by accident. The receiver now advertises every pool of its storage policy (any volume; a disk configured on the server but absent from the policy is not a candidate, since a part on it wouldn't load at startup), the sender names the one it matched, and the part lands on that pool's disk ahead of volume order, JBOD balancing and TTL move rules — the mover carries it to a TTL destination afterwards, the same way `perform_ttl_move_on_insert=0` already places first and moves later. A caller-supplied `dest_disk` (zero-copy `MOVE`) stays authoritative and untouched; a content-addressed disk never enters that path (`supportZeroCopyReplication()` is false for CAS). A read-only or broken disk on the right pool is not a candidate — nothing can publish a ref there. The offered pool must itself be an advertised pool (not matched by disk name), and the confirm's gate 0 compares mounts rather than disk names, closing a second-order gap the first pass left. Non-live pool disk = fail-close: a disk whose mount isn't live is left out of both the advertise and the placement, never guessed at. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ContentAddressedMetadataStorage::shutdown` contained two unbounded waits on the GC round: a `std::lock_guard` on the same mutex a synchronous round (`SYSTEM CAS GC`, `GC REBUILD`) holds for its whole duration, and `CasGcScheduler::stop`'s join, which the scheduler loop only even looks at at the top of its wait — a round already in flight never observes it, and a comment in the loop recorded an accepted extra full round if `stop` lands while the loop is blocked behind a manual round. A round has no wall-clock budget at all: `GcRoundWorkBudget` caps destructive work, not time, and against a slow bucket the wall clock is whatever the bucket makes it. Nothing in this wait protects durable state — the round is one-pass, committed by a single `gc/state` conditional write at the end, so an interrupted round is a crash the protocol already survives — the wait existed purely so no thread would touch a freed object. Shutdown and the storage destructor now arm the pool's teardown flag before the lock or join they would otherwise wait behind. The open request plane carries that flag as its fence, so a round in flight is refused at its next request, its next retry sleep, or its next streamed refill — the check lives at the request because a phase is long from making thousands of requests, not from making one long one, and `CasOperation` already re-checks admission before every attempt and sleep. Every join and every object's ownership stay unchanged: the join became short, not optional. A round cut this way is recorded `Stopped` rather than `Aborted`. Decommission is deliberately not armed: an already-latched self-remount completes one more step whose pool-identity probe runs on the open plane, and no arm point early enough to bound the GC join leaves that step intact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…bulk delete) A 15-minute real-GCS soak measured a sweep round costing 300-617 s per phase, all from per-object request loops on keys that are write-once by construction — an object whose only writer mints it once and whose only other mutations are exact-token deletes, so nothing about it needs re-reading once known. Three loops dominated: `fold_reduce`'s `GET` volume (2870-3400 per round) turned out to be the sweep's mount-floor probing, not manifest bodies — `floorForNamespace` reads the mount key of every `/`-prefix of a namespace for every listed manifest, though the floor is one value per server root; `manifest_deletes` cost 617 s for 3250 sequential conditional deletes at ~190 ms each; and `ref_object_cleanup` cost 199-204 s for 512-516 keys at four requests each. The fix cuts each loop to what the write-once property actually allows: one mount-floor read per namespace per sweep page (memoized), manifest bodies read only for nominated orphans and through the existing read-ahead instead of on every listed key, and a new write-once bulk-delete verb (`removeManyWriteOnce`, backed by `DeleteObjects` where the store has it) replacing the sequential per-key deletes for owner-removed manifests and for ref-object cleanup, which now revalidates its cohorts before batching them. None of this changes what gets deleted or when a namespace or manifest is judged eligible — only how many requests that judgment costs. Measured on real GCS: `fold_reduce` 300-380 s -> 2-5 s; `manifest_deletes` 617 s -> 2 s on a 1506-key round; `ref_object_cleanup` 204 s -> under 1 s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…own writes to a hot control object Every `CREATE TABLE`/`DROP TABLE` on a content-addressed disk mutates one pool-wide object, `cas/ref_catalog`, through a conditional write. Measured on ten parallel stateless jobs: `DROP TABLE` p50 2.4 s, p90 11.9 s, max 34.7 s; 113 `PreconditionFailed` in 80 s from 53 threads; the losing writer alone racking up 35 attempts with gaps growing to the 5 s cap; one `DROP` losing eight races in a row for 15.4 s. `CasRefCatalog::casUpdateImpl` starts every write with a `GET` and paces a lost race with `Retry::backoff`, a schedule shared with transport faults — a writer that has lost several races sleeps for seconds while a fresh one starts at zero, so the oldest loser is the least likely to win next. Worse, every writer in one process races every other writer in the *same* process: compare-and-swap is only needed against other servers, so every intra-process race is pure waste, each costing a `GET`, a refused `PUT`, a resolve `GET` and a sleep. `CasHotKeys` sits above the request engine as one FIFO ticket per pool and key: writers to the same hot object queue instead of racing, their conditional writes are combined into one physical attempt where safe (as-if-serial semantics, a `Conflict` cascade on a lost race so combined members see the answer a serial retry would have given them), and a last-known- object cache lets a lane holder skip the leading `GET` under one rule. Losing a race against *another server* still paces with a flat jitter, not the transport-fault backoff. The GC erase over `ref_catalog` (`deleteCompletedRemovingAtSnapshot`) becomes the lane's first caller, and the pool owns the lane. This is phase A only — combining, spacing, the clamp and moving the GC erase itself onto the lane in full are follow-on work; the design and its 34 review revisions are recorded separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…ed locals Several gtest fixtures declared a test-only backend hook or fault clock as a local, captured other locals in it by reference, and then let the store (declared before the hook) call the hook again during its own teardown — after the captured locals had already gone out of scope. Under ASan this is a use-after-scope: the store's destructor writes a "farewell" record that can invoke a still-armed hook whose captured references are already dead. Fixed by declaring the test clock/hook before the store that keeps calling it (two transient-round tests, the straggler-epoch test), and by clearing the checkpoint-advance recovery test's backend hook before the locals it captures die. A separate scripted S3 client fix allocates its response body with `Aws::New`, matching how the SDK actually frees it, instead of a mismatched allocator. Also corrects two suites that had started asserting a schedule the engine never promised. Also: the stateless CAS lanes now run the GC scheduler every 20 s instead of every 5 s, matching the interval those tests actually need. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`S3ObjectStorage::getSingleAttemptClient` (`SingleAttemptRetryStrategy`, `max_retries=0`) and the write path carrying `WriteSettings::object_storage_retry_profile == SingleAttempt` belong to the CAS control plane's conditional writes: a failed attempt there is not the final answer — `CasOperation::writeLoop` resolves the outcome by a read and reissues — yet two upstream sites logged it at Error as if it were terminal: `Client`'s network-error handler and the non-412 `S3Exception` site in `WriteBufferFromS3`. Both now log at Debug when the client carries `SingleAttemptRetryStrategy` or the write carries the `SingleAttempt` profile; an ordinary client configured with zero retries by a user setting (no outer loop resolving it) keeps logging at Error, since for that caller the failure really is final. The neighbouring 412 (`isPreconditionFailedError`) branch drops from Info to Debug for the same reason: a conditional write losing its precondition is the caller's expected answer, not an operator-facing event. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
… gtests `SnapshotPublisherLatchedAcrossChunks` had two independent, reachable races. With `snapshot_log_count_threshold` at 0 (only reachable in this test), `precommitAdd`'s post-commit trigger dispatches a background publisher whose capture can still be in flight when `promote`, moments later, becomes lane leader and moves the lane to `Writing` — a lost race then backs the publisher off, and this pool's frozen `boot_ms_fn` never advances past that backoff deadline, poisoning every later dispatch on the namespace for the rest of the test. Fixed by driving `precommitAdd`/`promote` directly instead of through the shared `publishEmptyPart` helper, draining and explicitly publishing between the two commits. Separately, the carve hook gated the leader on the publisher reaching its blocked `PUT`, but under contention the dispatch's own scheduling delay could outlast that wait's bound, letting a leader released by timeout (not by the capture it meant to prove) start chunk 2 before the publisher captured — fixed by gating on the publisher's capture instead, which is causally prior to the `PUT`. Verified with 20 isolated `gtest_repeat` iterations and two full `CAS*` gates (2437/2437 each), reproduced only under CPU contention after isolated repeats alone did not reproduce it. Separately: a fatal `ASSERT_*` between launching an `AppendCaller`/`Caller` thread and its explicit `join()` left `TestBody` with the thread still joinable, and `std::thread::~thread()` on a joinable thread calls `std::terminate`, aborting the whole `unit_tests_dbms` binary and discarding every test scheduled after it. Both structs now join in their destructor if still joinable, so a failed assertion costs one test instead of the whole gate. Also: `CASDetachedWork` now stops and drains its detached publisher before the locals its hooks read go out of scope (ASan stack-use-after-return on `fake_boot` via `boot_ms_fn`), the same class of bug as the earlier test-hook lifetime fixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`HTTPServerConnection::run` polls `_stopped` in its loop condition outside `_mutex`, while `onServerStopped` (reached from `HTTPServer::stopAll(true)`) writes it from the stopping thread. A plain `bool` is a data race; TSan reported it from the test servers that abort their connections on teardown (`base/poco/Net/src/HTTPServerConnection.cpp:154` vs `:61`). `std::atomic<bool>` keeps the exact semantics with sequentially consistent loads and stores. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> (cherry picked from commit 5340794a38066ce0d9fce854aaea6c84dd3bc0d4)
…n per request Every test in these files starts its own HTTP server on an ephemeral port and destroys it with the test. With keep-alive on, the process-wide connection pool can hand a later test a pooled connection to a port whose server is already gone, and the request fails with `Connection reset by peer`; reproduced with `--gtest_repeat=3` on `S3BulkDeleteFallback` (three tests failed in iteration 3 only). `http_keep_alive_timeout = 0`, the same setting `gtest_cas_s3_single_attempt_client.cpp` already uses for the same reason. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Only the bulk-delete fallback suite needs it: `makeNetworkFailingClient` in `gtest_cas_aws_s3_client.cpp` creates no mock server, so that file is left as is. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Member
PR #2300 CI Verification ReportVerdictDo not approve for merge yet. Release / binary CAS insert–select paths look largely healthy (binary CAS stateless green; Fast test green; most integration shards green). Remaining blockers are:
Author triage on the PR (runs 1–3) remains accurate for R7 / T4 / Iceberg / SimSIMD; this report re-checks head Summary table
Funnel (head Job-level fail counts (head SHA)From
Also red on GitHub but not fully represented as test rows: msan/tsan CAS S3 shards hitting the 6h budget; aarch64 CAS alter/LWD 5h timeouts. Database rate comparison (
|
| Test | Branch runs | Branch fails | Branch % | PR runs | PR fails | PR % | Reading |
|---|---|---|---|---|---|---|---|
test_auth_token_profile_events |
8 | 8 | 100 | 16 | 16 | 100 | pre-existing-flaky / broken test on branch |
test_schema_inference[s3-1-True] |
145 | 26 | 17.9 | 21 | 9 | 42.9 | Elevated but known Iceberg UBSan (#2216); not CAS |
test_cas_mount_renewal_retry::…landed_response_lost… |
14 | 0 | 0 | 20 | 6 | 30 | regression |
test_cas_mount_renewal_retry::…hard_restart… |
0 | 0 | — | 8 | 1 | 12.5 | regression (new / PR-only) |
CASRefStateMachine.OwnerTransitionRejectsInvalidCombinations |
7 | 0 | 0 | 11 | 1 | 9.1 | regression (PR-only PRs list = [2300]) |
IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt |
7 | 0 | 0 | 11 | 1 | 9.1 | regression (PR-only) |
Cannot start clickhouse-server |
30 | 30 | 100 | 3 | 3 | 100 | infrastructure / known arm_tsan |
03572_export_merge_tree_part… |
129 | 6 | 4.7 | 102 | 2 | 2.0 | pre-existing-flaky |
02477_single_value_data_string_regression |
965 | 5 | 0.5 | 58 | 2 | 3.4 | pre-existing-flaky (labeled) |
00172_early_constant_folding |
946 | 1 | 0.1 | 59 | 1 | 1.7 | weak n=1; labeled reproducible on CAS asan |
00975_move_partition_merge_tree |
961 | 0 | 0 | 60 | 1 | 1.7 | under CAS asan memory pressure |
01461_query_start_time_microseconds |
949 | 0 | 0 | 59 | 1 | 1.7 | labeled flaky |
01883 / 02435 / 03402 / 04105 |
hundreds | ≤2 | ≤0.3 | 72–181 | 3 | ~1.7–4 | CAS asan memory-limit cluster (T4) |
00427 / 02293 / 04266 (msan WasmEdge) |
~960 | ~0–1 | ~0 | ~60 | 1 | ~1.7 | pre-existing-flaky (labeled) |
Other PRs hitting test_auth_token_profile_events in 30d: 0,2251,2290,2294,2300,2305,2309,2315,2318,2320,2330 (42 fails) — clearly not unique to #2300.
Regression suite rates (gh-data.clickhouse_regression_results, 30d)
/selects/final/force/concurrent (parent)
| Kind | On PR #2300? | OK | Fail | Fail % |
|---|---|---|---|---|
| cas | no | 75 | 25 | 25.0 |
| cas | yes | 14 | 2 | 12.5 |
| non_cas | no | 272 | 1 | 0.4 |
| non_cas | yes | 7 | 0 | 0 |
Category: regression (CAS product availability), pre-existing on CAS base — not uniquely worse on #2300 (PR rate is lower than other CAS). Mechanism on d876cda amd Release: Code: 210 NETWORK_ERROR — mount lease not held / TransientNotLive during concurrent FINAL (no injected fault). AArch64 cas_selects and cas_s3_cache_selects passed same SHA. Non-CAS selects green. Do not xfail; fix lease renewal under load.
Note: S3
report.htmlfor x86 selects was overwritten to look green after the job failed — trust the GitHub job log, not the HTML alone.
/alter/attach partition/part 1/replica sanity/parallel add remove sanity (R7)
| Kind | On PR #2300? | OK | Fail |
|---|---|---|---|
| cas | yes | 1 | 3 |
| cas | no | 16 | 15 |
| non_cas | no | 319 | 5 |
| non_cas | yes | 2 | 0 |
Sibling / package split (cas jobs): other PRs 14 fail / 14 ok; release/ref 1 fail / 2 ok.
Category: regression of CAS+replicated partition workload on the base line (R7), not unique to this PR. Leaf assert: expected 500 rows, got 400. Author diagnosis: common pool admission refuses without logging, still increments num_tries, then 300s backoff coincides with SYNC REPLICA budget; first real error often CAS relink-confirm NO_REPLICA_HAS_PART. AArch64 twins time out at 5h.
/lightweight delete/concurrent delete/MergeTree/random delete entire table without overlap
| Kind | On PR #2300? | OK | Fail | Fail % |
|---|---|---|---|---|
| cas | no | 5 | 1 | 16.7 |
| cas | yes | 0 | 2 | 100 |
| non_cas | no | 311 | 2 | 0.6 |
| non_cas | yes | 7 | 0 | 0 |
Category: unknown → lean product race. Mechanism: concurrent non-overlapping LWD left count(*)=1 vs expected 0 (all DELETEs returned 0). Non-CAS on this PR passed; CAS on this PR failed both runs (n=2). Known rare leftover-row race also seen on non-CAS head. Needs more CAS runs or a minimal SQL repro before calling it a #2300 regression.
/tiered storage/with cas/simple replication and moves
On d876cda x86: errno 28 No space left on device writing to jbod1. AArch64 same suite: module OK.
Category: infrastructure.
Root-cause analysis (survivors)
1. Unit MSan — CASRefStateMachine.OwnerTransitionRejectsInvalidCombinations
- Category:
regression - Mechanism: MSan
use-of-uninitialized-valueinstd::functioncall path duringCASRefSnapshotPublishOrdering.NotReadyRefusalBacksOffAndResetsAfterDurablePublish; process exits as the next test starts (mis-attributed name). - DB: 0/7 branch fails; 1/11 on PR; fails only on PR list
[2300]. - Action: Fix uninit in NotReadyRefusal / snapshot publish ordering gtest path under MSan.
2. Unit TSan — IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt
- Category:
regression - Mechanism: TSan data race in
Poco::Net::HTTPServerConnection::onServerStoppedwhile the new conditional-mode gtest runs. - DB: 0/7 branch; 1/11 PR; PR-only.
- Action: Serialize / stop server before teardown, or mark race if confirmed false positive in test-only Poco path.
3. test_cas_mount_renewal_retry (msan)
- Category:
regression - Mechanism:
landed_response_lost…waits 120s, last=None(retry_failed);hard_restart…expected observation log count 1, got 0. - DB: landed: branch 0/14, PR 6/20 (30%).
- Action: Revisit msan timing / observation after recent wait-margin commits; still red on
d876cda.
4. T4 — Sanitizer CAS stateless resource pressure
- Category:
regression(CI coverage / resource), not a binary functional break - Mechanism: ASan CAS jobs accumulate memory (
Code: 241 memory limit exceededon many late tests); msan/tsan CAS shards hit 6h. Binary CAS parallel passed (amd + arm). - Evidence: Author T4 (~2400 threads / per-disk pools);
SYSTEM CAS FORGETlanded and is being measured. - Action: Confirm forget helps; shrink sanitizer pool profile if needed.
5. R7 — CAS alter attach replica divergence
- See rate table above. Release blocker for heavy replicated partition ops on CAS until common-pool
num_tries/ pool sizing fix lands.
6. cas_selects concurrent FINAL — mount lease
- See rate table. Designed refusal when not Live; unexpected lease drop under quiet MinIO + concurrent FINAL. Product bug, pre-existing CAS sensitivity; CAS improvements #2300 still owns fixing it for antalya-26.6 CAS quality.
Unrelated (do not block this PR’s CAS claim)
| Item | Category | Evidence |
|---|---|---|
Iceberg test_schema_inference (+ read_in_order cascades) |
cascade / branch bug |
UBSan decimal overflow; #2216; Connection refused after server abort |
test_auth_token_profile_events |
regression of test vs rename (not this PR) |
100% branch + many PRs; counters renamed after #2222 |
Stress arm_tsan Cannot start |
infrastructure |
100% branch; SimSIMD ARM SIGILL probe |
| WasmEdge / azure labeled flaky | pre-existing-flaky |
CI labels + rates |
cas_s3_cache_aggregate_functions_1 module Fail |
unknown / low signal |
1 Fail row at /aggregate functions; coverage text elsewhere says module ok — treat as noise until leaf reproduced |
What is green (signal that CAS core works)
- Builds (all sanitizer/release/debug listed in report)
- Fast test
- Stateless binary CAS S3 (amd + arm)
- Most integration shards (including many CAS tests that failed on earlier commits)
- AArch64
cas_selects;cas_s3_cache_selects(where run) - Grype / Docker / install / compatibility
Recommendations
- Before approve: clear unit MSan + TSan and msan mount-renewal (items 1–3).
- Track explicitly (may ship with known issues): R7 alter attach; concurrent FINAL mount-lease; T4 sanitizer budgets — with release notes if merging.
- Ignore for this PR: Iceberg antalya-26.6: test_schema_inference kills the server on amd_asan_ubsan — upstream Decimal-bounds overflow exposed by #2145 #2216, auth_token counter rename, SimSIMD arm_tsan, labeled flaky.
- Rerun: amd
cas_selects(confirm lease blip rate); CAS LWD concurrent scenario (n=2 is weak). - Do not trust overwritten S3 TestFlows HTML for selects — use job logs / DB.
Approval checklist
| Question | Answer |
|---|---|
| Did this PR uniquely break binary CAS insert/select? | No strong evidence (binary CAS green) |
| Are there PR-only failures that must be fixed? | Yes — unit MSan/TSan; mount-renewal msan |
| Are there CAS product issues that pre-exist on base but block release quality? | Yes — R7; mount-lease under concurrent FINAL; sanitizer CAS T4 |
| Unrelated red paint? | Yes — Iceberg, auth_token, arm_tsan stress |
| Approve now? | No |
Appendix: query snippets used
-- Unique fails on head
SELECT test_name, count() fails
FROM `gh-data`.checks
WHERE pull_request_number = 2300 AND test_status = 'FAIL'
AND commit_sha LIKE 'd876cda%'
GROUP BY test_name ORDER BY fails DESC;
-- Branch vs PR rates
SELECT test_name,
countIf(pull_request_number = 0) branch_runs,
countIf(pull_request_number = 0 AND test_status='FAIL') branch_fails,
countIf(pull_request_number = 2300) pr_runs,
countIf(pull_request_number = 2300 AND test_status='FAIL') pr_fails
FROM `gh-data`.checks
WHERE test_name IN (...) AND (pull_request_number IN (0, 2300))
AND check_start_time > now() - INTERVAL 60 DAY
GROUP BY test_name;
-- Regression CAS vs non-CAS
SELECT multiIf(job_name LIKE '%cas%','cas','non_cas') kind,
clickhouse_package LIKE '%/PRs/2300/%' on_2300,
sum(result='OK') ok, sum(result='Fail') fail
FROM `gh-data`.clickhouse_regression_results
WHERE test_name = '<path>' AND start_time > now() - INTERVAL 30 DAY
GROUP BY kind, on_2300;
`stopAll(true)` reaches `HTTPServerConnection::onServerStopped(abortCurrent = true)`, which shuts the connection's socket down without taking the connection mutex so that it can interrupt a handler holding it. A worker that is leaving `run` at the same moment closes that socket from its own thread, and TSan reports the race on `SocketImpl::_sockfd` (CI run 9 of PR #2300, Unit tests (tsan), `IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt`). The abort path was only there to unblock workers waiting for the next request on a pooled keep-alive connection before `thread_pool.joinAll`. Close those connections from the client side instead: drop the process-wide `HTTPConnectionPools` cache, which the AWS SDK client uses through `makeHTTPSession`, so every worker sees end of stream and closes its own socket on its own thread; then `stop` the server (accept thread joined, dispatcher stopped, no `serverStopped` notification) and join the pool. Applied to all four fork mock servers: `TestPocoHTTPServer`, `TestPocoHTTPStsServer`, `TestPocoHTTPSequenceServer`, `ScriptedResponseServer`. Production is unaffected: `DB::HTTPServer::stopAll` never used the abort path. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx
This reverts commit 5340794a380, a fork patch to upstream Poco: the only callers of `HTTPServer::stopAll(true)` in this tree were the test mock servers, and they no longer use Poco's abort notification (see the previous commit), so `onServerStopped` never runs concurrently with `run` here and the fork patch to `base/poco` is not needed. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit ca187904e613d3c0755bdf3ac38dc008a127cf4b)
…alya-26.6/CAS-improvements-cicd-fixes Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
This was referenced Sep 10, 2026
Member
PR #2300 CI Verification ReportRe-verification (2026-09-10)Verdict (updated)Hold merge on sanitizer CAS abort until the Code-210→
Do not treat Prior blockers — cleared on
|
| Test | Status on this SHA |
|---|---|
CASRefStateMachine.OwnerTransitionRejectsInvalidCombinations |
0 FAIL / 3 OK |
IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt |
0 FAIL / 3 OK |
test_cas_mount_renewal_retry::…landed_response_lost… |
0 FAIL / 5 OK |
test_cas_mount_renewal_retry::…hard_restart… |
0 FAIL / 5 OK |
test_auth_token_profile_events |
0 FAIL / 5 OK |
| Unit tests (msan / tsan) jobs | success (fail: 0) |
| Stress test (arm_tsan) | success |
cas_alter_attach_1 / _2 (both arches) |
success |
tiered_storage_cas |
success |
| Binary CAS stateless (amd/arm) | success |
| Fast test / builds / most integration shards | success |
Still red on this run — categorized
| Cluster | Category | Blocks #2300? | Notes |
|---|---|---|---|
cas_alter_attach_3 / cas_s3_cache_alter_attach_3 (× arches) |
regression (CAS product) |
No — tracked | Part-3 combinatorics; Code 210 on ref_catalog/_ckpt; filed as #2343. Parts 1–2 green. |
cas_selects concurrent FINAL |
regression (CAS product) |
No — tracked | Mount lease dropped with no store outage (Code: 210 / TransientNotLive); filed as #2332. Non-CAS selects green; aarch64 often passes. |
cas / cas_s3_cache — /cas/.../export/EXPORT PARTITION from a CAS disk is rejected |
test expectation / product message | No if rerun green | Assert wants not supported on a CAS disk yet; server returns Code: 48 NOT_IMPLEMENTED — EXPORT PARTITION is not implemented for engine MergeTree. Author: fixed → expect green on rerun. |
Stateless ASan CAS S3 2/2 — Server died (SIGABRT) |
regression (CAS) + mass cascade |
Yes until fixed/waived | See Stateless CAS Server died. Binary + ASan 1/2 green. |
Stateless MSan CAS S3 2/3 — Server died |
unknown / likely timeout |
Separate from ASan | Harness SIGTERM (15) / timeout pattern; no matching std::terminate Fatal in the checked log. |
| Stateless TSan CAS S3 | mixed success on this run | — | 2/2 success; other shards pending/null at check time. |
settings default-values snapshot (4 settings) |
unrelated suite drift | No | Snapshot CHECK fails for settings not in >=26.6_antalya snapshot (same class as earlier export-settings gap). Fix in clickhouse-regression snapshots, not this PR. |
Iceberg test_writes_decimal_wide_minmax_pruning |
unrelated | No | Code: 36 Iceberg decimal precision ≤ 38 — not CAS. |
Stateless CAS Server died root cause (2026-09-10)
Investigated ASan job Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) on head f377ba3 (upstream-test-investigation + source read). Not ASAN/MemoryTracker/RSS accounting noise.
| Field | Finding |
|---|---|
| Taxonomy | Real crash: SIGABRT (6) via std::terminate. Not harness 143, not OOM (dmesg OOM check clean; 0 MemoryTracker / memory-limit hits in the Fatal window). afterCommit even installs LockMemoryExceptionInThread (blocks tracker faults — opposite of “RSS snap”). |
| Root query / test | COMMIT; from 01169_old_alter_partition_isolation_stress (test_ccb3swc9.src, UUID 044ebfe8-…) |
| Exception | Code 210 NETWORK_ERROR: CAS write could not be committed (CAS ref-log append … refused BEFORE any request was sent …); retrying later (makeCasWriteRetryLaterExceptionPtr) |
| Pre-attempt meaning | CasUnresolvedReason::NoAttemptSent — mount fence or operation deadline rejected before the first putIfAbsent (CasRequestControl.h); key provably unwritten; increments CASRefAppendPreAttemptRefused |
| Why this COMMIT touches CAS publish | TransactionLog::finalizeCommittedTransaction → MergeTreeTransaction::afterCommit → setAndStoreCreationCSN → VersionMetadataOnDisk::storeInfoToDataPartStorage writes txn_version.txt on the part. On CAS that is a standalone write on an already-committed part → ContentAddressedTransaction repoint path (publishStaging: scratch precommitAdd → repointRef → abandon() scratch build) |
| Throw path | CasRefLedger::commitRefChunk → appendRefOps → PartWriteTxn::abandon (precommit-removal append) → ContentAddressedTransaction::publishStaging |
| Abort path | Escapes into afterCommit / finalizeCommittedTransaction (both noexcept) → std::terminate → SIGABRT |
| Source already knows abandon throws | Comment at ContentAddressedTransaction.cpp ~376–379: “scratch-build abandon() below (which can itself throw)” — outcome is captured first, but the throw is still allowed to leave publishStaging |
| Destructor contrast | ~ContentAddressedTransaction catches abandon failures and only logs (“A destructor must not throw”); publishStaging does not |
| Trigger context | Concurrent Code 210 storm on same UUID (merges, _ckpt contention, system-log flush) under ASan+CAS load makes fence/deadline pre-attempt refuse more likely — trigger, not the bug |
| Inflation | Post-abort Code 236 / connection resets → mass cascade FAIL rows |
| Category | regression — recoverable CAS error must not terminate the process (deterministic once Code 210 hits this path) |
| Open fix PR? | None found outside #2300 itself |
| Fix direction | Catch retry-later around abandon() in publishStaging (mirror destructor), and/or ensure MergeTree afterCommit disk writes cannot throw; do not rely on “rerun will be green” |
Artifacts (may be overwritten on rerun):
…/stateless_tests_amd_asan_ubsan_cas_s3_storage_parallel_2_2/ under altinity-build-artifacts for PR 2300 / f377ba3.
Approval checklist (2026-09-10)
| Question | Answer |
|---|---|
| Prior unit / mount-renewal / auth_token blockers gone? | Yes |
| Alter attach still failing? | Yes, part 3 only — #2343 |
cas_selects still failing? |
Yes (amd) — #2332 |
| Approve without waiting for #2343 / #2332 fixes? | Yes (follow-up issues) |
ASan CAS S3 Server died understood? |
Yes — Code 210 into noexcept afterCommit; fix required |
| Approve before that abort is fixed? | No (unless explicitly waived) |
| Approve before CAS suite export assert rerun green? | Only if accepting author “fixed on rerun” for export message |
| Unrelated red (settings, iceberg writes)? | Ignore for this PR |
Historical verification (2026-09-09, head d876cda)
| Field | Value |
|---|---|
| Head | d876cda683f06972f54470910c0366a93002671d |
| Run | 34294710256 |
| CI report | ci_run_report.html |
| Verified | 2026-09-09 |
| Method | pr-ci-failure-triage + upstream-test-investigation + regression-test-database-investigation |
| DB | gh-data.checks, gh-data.clickhouse_regression_results (60d / 30d windows) |
Verdict (2026-09-09)
Do not approve for merge yet.
Release / binary CAS insert–select paths look largely healthy (binary CAS stateless green; Fast test green; most integration shards green). Remaining blockers are:
- Sanitizer / unit failures introduced or only seen on this PR.
- Known CAS workload issues (R7 alter attach; mount-lease blips under concurrent FINAL; sanitizer CAS resource pressure).
- A small set of unrelated branch bugs that still paint the PR red.
Author triage on the PR (runs 1–3) remains accurate for R7 / T4 / Iceberg / SimSIMD; this report re-checks head d876cda with CI database rates.
Summary table (2026-09-09)
| Category | Blocks approval? | Count (clusters) | Notes |
|---|---|---|---|
regression |
Yes | 3 | Unit MSan/TSan; mount-renewal under msan; CAS sanitizer resource pressure (T4) |
regression (CAS product, pre-existing on CAS base) |
Yes for heavy CAS workloads | 2 | R7 alter attach; concurrent FINAL mount-lease (cas_selects) |
pre-existing-flaky |
No | ~15+ tests | Labeled flaky + rate-matched single flakes |
infrastructure |
No | 3 | Stress arm_tsan cannot start; tiered ENOSPC; aarch64 5h timeouts |
cascade |
No (classify root) | ~22 | Iceberg schema_inference UBSan → connection refused |
unknown |
Yes until cleared | 1 | LWD concurrent leftover row on CAS (n=2 on this PR) |
Funnel (head d876cda, gh-data.checks): ~59 unique FAIL test names → ~23 Iceberg schema/read_in_order (1 UBSan root + cascades) → ~5 labeled/rate-matched flaky → ~17 CAS-S3 asan memory-pressure cluster → survivors investigated below.
Job-level fail counts (head SHA)
From `gh-data`.checks where pull_request_number = 2300 and commit_sha LIKE 'd876cda%':
| Check | Fails | OKs |
|---|---|---|
| Integration tests (amd_asan_ubsan, db disk, old analyzer, 4/8) | 22 | 1040 |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) | 17 | 5451 |
| Integration tests (amd_asan_ubsan, targeted) | 13 | 11 |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) | 5 | 5443 |
| Stateless tests (amd_msan, WasmEdge, parallel, 4/4) | 4 | 2737 |
| Stress test (arm_tsan) | 3 | 1 |
| Integration tests (amd_msan, 3/10) | 2 | 373 |
| Unit tests (msan / tsan) | 1 each | 0 |
| Other single-fail jobs | 1 each | — |
Also red on GitHub but not fully represented as test rows: msan/tsan CAS S3 shards hitting the 6h budget; aarch64 CAS alter/LWD 5h timeouts.
Database rate comparison (gh-data.checks, 60d)
Branch = pull_request_number = 0. PR = 2300.
| Test | Branch runs | Branch fails | Branch % | PR runs | PR fails | PR % | Reading |
|---|---|---|---|---|---|---|---|
test_auth_token_profile_events |
8 | 8 | 100 | 16 | 16 | 100 | pre-existing-flaky / broken test on branch |
test_schema_inference[s3-1-True] |
145 | 26 | 17.9 | 21 | 9 | 42.9 | Elevated but known Iceberg UBSan (#2216); not CAS |
test_cas_mount_renewal_retry::…landed_response_lost… |
14 | 0 | 0 | 20 | 6 | 30 | regression |
test_cas_mount_renewal_retry::…hard_restart… |
0 | 0 | — | 8 | 1 | 12.5 | regression (new / PR-only) |
CASRefStateMachine.OwnerTransitionRejectsInvalidCombinations |
7 | 0 | 0 | 11 | 1 | 9.1 | regression (PR-only PRs list = [2300]) |
IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt |
7 | 0 | 0 | 11 | 1 | 9.1 | regression (PR-only) |
Cannot start clickhouse-server |
30 | 30 | 100 | 3 | 3 | 100 | infrastructure / known arm_tsan |
03572_export_merge_tree_part… |
129 | 6 | 4.7 | 102 | 2 | 2.0 | pre-existing-flaky |
02477_single_value_data_string_regression |
965 | 5 | 0.5 | 58 | 2 | 3.4 | pre-existing-flaky (labeled) |
00172_early_constant_folding |
946 | 1 | 0.1 | 59 | 1 | 1.7 | weak n=1; labeled reproducible on CAS asan |
00975_move_partition_merge_tree |
961 | 0 | 0 | 60 | 1 | 1.7 | under CAS asan memory pressure |
01461_query_start_time_microseconds |
949 | 0 | 0 | 59 | 1 | 1.7 | labeled flaky |
01883 / 02435 / 03402 / 04105 |
hundreds | ≤2 | ≤0.3 | 72–181 | 3 | ~1.7–4 | CAS asan memory-limit cluster (T4) |
00427 / 02293 / 04266 (msan WasmEdge) |
~960 | ~0–1 | ~0 | ~60 | 1 | ~1.7 | pre-existing-flaky (labeled) |
Other PRs hitting test_auth_token_profile_events in 30d: 0,2251,2290,2294,2300,2305,2309,2315,2318,2320,2330 (42 fails) — clearly not unique to #2300.
Regression suite rates (gh-data.clickhouse_regression_results, 30d)
/selects/final/force/concurrent (parent)
| Kind | On PR #2300? | OK | Fail | Fail % |
|---|---|---|---|---|
| cas | no | 75 | 25 | 25.0 |
| cas | yes | 14 | 2 | 12.5 |
| non_cas | no | 272 | 1 | 0.4 |
| non_cas | yes | 7 | 0 | 0 |
Category: regression (CAS product availability), pre-existing on CAS base — not uniquely worse on #2300 (PR rate is lower than other CAS). Mechanism on d876cda amd Release: Code: 210 NETWORK_ERROR — mount lease not held / TransientNotLive during concurrent FINAL (no injected fault). AArch64 cas_selects and cas_s3_cache_selects passed same SHA. Non-CAS selects green. Do not xfail; fix lease renewal under load.
Note: S3
report.htmlfor x86 selects was overwritten to look green after the job failed — trust the GitHub job log, not the HTML alone.
/alter/attach partition/part 1/replica sanity/parallel add remove sanity (R7)
| Kind | On PR #2300? | OK | Fail |
|---|---|---|---|
| cas | yes | 1 | 3 |
| cas | no | 16 | 15 |
| non_cas | no | 319 | 5 |
| non_cas | yes | 2 | 0 |
Sibling / package split (cas jobs): other PRs 14 fail / 14 ok; release/ref 1 fail / 2 ok.
Category: regression of CAS+replicated partition workload on the base line (R7), not unique to this PR. Leaf assert: expected 500 rows, got 400. Author diagnosis: common pool admission refuses without logging, still increments num_tries, then 300s backoff coincides with SYNC REPLICA budget; first real error often CAS relink-confirm NO_REPLICA_HAS_PART. AArch64 twins time out at 5h.
/lightweight delete/concurrent delete/MergeTree/random delete entire table without overlap
| Kind | On PR #2300? | OK | Fail | Fail % |
|---|---|---|---|---|
| cas | no | 5 | 1 | 16.7 |
| cas | yes | 0 | 2 | 100 |
| non_cas | no | 311 | 2 | 0.6 |
| non_cas | yes | 7 | 0 | 0 |
Category: unknown → lean product race. Mechanism: concurrent non-overlapping LWD left count(*)=1 vs expected 0 (all DELETEs returned 0). Non-CAS on this PR passed; CAS on this PR failed both runs (n=2). Known rare leftover-row race also seen on non-CAS head. Needs more CAS runs or a minimal SQL repro before calling it a #2300 regression.
/tiered storage/with cas/simple replication and moves
On d876cda x86: errno 28 No space left on device writing to jbod1. AArch64 same suite: module OK.
Category: infrastructure.
Root-cause analysis (survivors)
1. Unit MSan — CASRefStateMachine.OwnerTransitionRejectsInvalidCombinations
- Category:
regression - Mechanism: MSan
use-of-uninitialized-valueinstd::functioncall path duringCASRefSnapshotPublishOrdering.NotReadyRefusalBacksOffAndResetsAfterDurablePublish; process exits as the next test starts (mis-attributed name). - DB: 0/7 branch fails; 1/11 on PR; fails only on PR list
[2300]. - Action: Fix uninit in NotReadyRefusal / snapshot publish ordering gtest path under MSan.
2. Unit TSan — IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt
- Category:
regression - Mechanism: TSan data race in
Poco::Net::HTTPServerConnection::onServerStoppedwhile the new conditional-mode gtest runs. - DB: 0/7 branch; 1/11 PR; PR-only.
- Action: Serialize / stop server before teardown, or mark race if confirmed false positive in test-only Poco path.
3. test_cas_mount_renewal_retry (msan)
- Category:
regression - Mechanism:
landed_response_lost…waits 120s, last=None(retry_failed);hard_restart…expected observation log count 1, got 0. - DB: landed: branch 0/14, PR 6/20 (30%).
- Action: Revisit msan timing / observation after recent wait-margin commits; still red on
d876cda.
4. T4 — Sanitizer CAS stateless resource pressure
- Category:
regression(CI coverage / resource), not a binary functional break - Mechanism: ASan CAS jobs accumulate memory (
Code: 241 memory limit exceededon many late tests); msan/tsan CAS shards hit 6h. Binary CAS parallel passed (amd + arm). - Evidence: Author T4 (~2400 threads / per-disk pools);
SYSTEM CAS FORGETlanded and is being measured. - Action: Confirm forget helps; shrink sanitizer pool profile if needed.
5. R7 — CAS alter attach replica divergence
- See rate table above. Release blocker for heavy replicated partition ops on CAS until common-pool
num_tries/ pool sizing fix lands.
6. cas_selects concurrent FINAL — mount lease
- See rate table. Designed refusal when not Live; unexpected lease drop under quiet MinIO + concurrent FINAL. Product bug, pre-existing CAS sensitivity; CAS improvements #2300 still owns fixing it for antalya-26.6 CAS quality.
Unrelated (do not block this PR’s CAS claim)
| Item | Category | Evidence |
|---|---|---|
Iceberg test_schema_inference (+ read_in_order cascades) |
cascade / branch bug |
UBSan decimal overflow; #2216; Connection refused after server abort |
test_auth_token_profile_events |
regression of test vs rename (not this PR) |
100% branch + many PRs; counters renamed after #2222 |
Stress arm_tsan Cannot start |
infrastructure |
100% branch; SimSIMD ARM SIGILL probe |
| WasmEdge / azure labeled flaky | pre-existing-flaky |
CI labels + rates |
cas_s3_cache_aggregate_functions_1 module Fail |
unknown / low signal |
1 Fail row at /aggregate functions; coverage text elsewhere says module ok — treat as noise until leaf reproduced |
What is green (signal that CAS core works)
- Builds (all sanitizer/release/debug listed in report)
- Fast test
- Stateless binary CAS S3 (amd + arm)
- Most integration shards (including many CAS tests that failed on earlier commits)
- AArch64
cas_selects;cas_s3_cache_selects(where run) - Grype / Docker / install / compatibility
Recommendations
- Before approve: clear unit MSan + TSan and msan mount-renewal (items 1–3).
- Track explicitly (may ship with known issues): R7 alter attach; concurrent FINAL mount-lease; T4 sanitizer budgets — with release notes if merging.
- Ignore for this PR: Iceberg antalya-26.6: test_schema_inference kills the server on amd_asan_ubsan — upstream Decimal-bounds overflow exposed by #2145 #2216, auth_token counter rename, SimSIMD arm_tsan, labeled flaky.
- Rerun: amd
cas_selects(confirm lease blip rate); CAS LWD concurrent scenario (n=2 is weak). - Do not trust overwritten S3 TestFlows HTML for selects — use job logs / DB.
Approval checklist
| Question | Answer |
|---|---|
| Did this PR uniquely break binary CAS insert/select? | No strong evidence (binary CAS green) |
| Are there PR-only failures that must be fixed? | Yes — unit MSan/TSan; mount-renewal msan |
| Are there CAS product issues that pre-exist on base but block release quality? | Yes — R7; mount-lease under concurrent FINAL; sanitizer CAS T4 |
| Unrelated red paint? | Yes — Iceberg, auth_token, arm_tsan stress |
| Approve now? | No |
Appendix: query snippets used
-- Unique fails on head
SELECT test_name, count() fails
FROM `gh-data`.checks
WHERE pull_request_number = 2300 AND test_status = 'FAIL'
AND commit_sha LIKE 'd876cda%'
GROUP BY test_name ORDER BY fails DESC;
-- Branch vs PR rates
SELECT test_name,
countIf(pull_request_number = 0) branch_runs,
countIf(pull_request_number = 0 AND test_status='FAIL') branch_fails,
countIf(pull_request_number = 2300) pr_runs,
countIf(pull_request_number = 2300 AND test_status='FAIL') pr_fails
FROM `gh-data`.checks
WHERE test_name IN (...) AND (pull_request_number IN (0, 2300))
AND check_start_time > now() - INTERVAL 60 DAY
GROUP BY test_name;
-- Regression CAS vs non-CAS
SELECT multiIf(job_name LIKE '%cas%','cas','non_cas') kind,
clickhouse_package LIKE '%/PRs/2300/%' on_2300,
sum(result='OK') ok, sum(result='Fail') fail
FROM `gh-data`.clickhouse_regression_results
WHERE test_name = '<path>' AND start_time > now() - INTERVAL 30 DAY
GROUP BY kind, on_2300;
Member
|
Issue opened for asan cas stateless fail #2344 |
alsugiliazova
added
verified
k-morozov
reviewed
Sep 10, 2026
Comment on lines
+289
to
+293
| throw; | ||
| auto new_client = credentials_refresh_callback(); | ||
| if (!new_client) | ||
| throw; | ||
| client.set(std::move(new_client)); |
There was a problem hiding this comment.
We could receive an AccessTokenExpiredError, recreate the client with a callback, and then retry sending the new request with the same attempt_number/attempt_no without any checks in CasOperation::readLoop.
However, I couldn't find where the callback returns a valid value, so it is safe right now.
|
LGTM |
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
…ol thread `CASGCRetire.OutcomeLogUnobservedConflictDoesNotReportItVanished` let its `UnobservedOutcomesBackend` flip `arm` and `refused_key` from `write` on the test thread while `read` consulted them from a `GcMetaWriter` pool thread (`scheduleConfirmedMetaDelete` -> `loadMeta`). TSan reported the race in `Unit tests (tsan)`; the fault state now lives under its own mutex. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 1b2fd76ad9b98dae195ee589b67d528ee5db05df)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
`test_a_write_whose_response_carries_no_generation_is_refused` expected the INSERT to fail with "carried no valid generation". That refusal went away with the request-engine migration: `CasOperation::writeLoop` treats a 2xx whose value no grammar accepts as an ambiguous attempt and settles it with one exact body read, adopting the observed incarnation only when the bytes are its own. The test now asserts that contract: the INSERT succeeds, the `.meta` creates of the blobs it published are answered without a generation, each is followed by a GET (never a HEAD) of its own key, and `CASRequestResolveRead` grows accordingly. GC and the table's merges are stopped for the window so nothing else publishes markers into the captured slice. `test_marked_and_default_heads_coexist_on_one_oauth_client` relied on the sentinel probe issuing an ordinary HEAD; the probe now reads with a marked GET, so no default HEAD exists on a CAS bucket any more. It becomes a partition by GET on one disk: CAS control and marker reads are marked, blob-body reads are not, and every HEAD is marked. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 26a2f52b763808f8d12c8041b318c3484ae2bf02)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
…ng its prefix The startup residual probe proved that a pool exists through a LIST walk of the pool prefix (page 1000). Since the request-engine migration that walk runs under the control-plane single-attempt budget, and a 1000-key page over a large prefix on a slow store exceeds it reliably: the `cas_alter_attach_2` regression suite restarted a server whose every LIST attempt timed out at 5 s, 30 attempts gave up at the deadline, the probe answered `Indeterminate` and startup refused a pool the server could read perfectly well. `_pool_meta` present is decisive by the probe's own contract, and one exact read of that key answers it without enumerating anything. The probe now reads it first and runs the residual LIST only when the key is absent or the read could not settle it -- which is exactly the case that needs the absence-of-residue proof. `PoolMeta::createOrValidate` still re-reads and validates the object on the `PoolMetaPresent` path. `CASListLiarEndToEnd.RecoveryUnderTheSameLieReconstructsExactlyTheTruth` relied on that bootstrap LIST to prove its lying store was consulted at all; recovery itself reads by exact key. The test now proves the omission with an explicit LIST of the stream before comparing recovery against the oracle. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit aee2a4b6880f8f87d1961db151ff77cdffc8425d)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
…t, not a per-attempt one Since the request-engine migration the reissues of a mount renewal are paced inside the engine's write loop and the mount logs only the renewal's outcome: the per-attempt `retrying` event is gone, and the read-settled classification is named `committed_by_read`. The test still waited for a `retrying` row next to `recovered` and expected `committed_by_get`, so both scenarios timed out on `system.cas_log` although the renewals had recovered. The attempt count on the terminal row is what proves a retry happened. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 5adccd5009b929988ff0d1849162fea656a04cba)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
On the ASan stateless lanes the server's resident set climbs to 30-45 GiB within 15 minutes while `MemoryTracking` stays under 1 GiB: the memory is the sanitizer runtime's, chiefly the per-thread fake stacks that `detect_stack_use_after_return` (default on) keeps mapped for the life of each of the ~1-2k server threads, with the malloc stack depot second. On a CI runner this reaches the memory ceiling (the global limit check is RSS-based) and surfaces as `(total) memory limit exceeded` in unrelated tests, thread-count diffs in `EXPLAIN PIPELINE` tests, and a hung `02435_rollback_cancelled_queries`; the CAS S3 lane merely gets there first. Locally, shard 2/2 of the CAS lane: defaults -> 45.9 GiB and the harness kills the unresponsive server at test 2977; with `detect_stack_use_after_return=0 malloc_context_size=10 allocator_release_to_os_interval_ms=10000` -> 11.5 GiB plateau, the whole shard completes and every previously red test passes. The plain ASan lane shows the same curve (26.8 GiB at 14 minutes). Stress runs already set the latter two options in tests/docker_scripts/stress_tests.lib. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 4aa3c786afba786869db548e2c6d90b326acb38a)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
`freezeConnectTimeoutCapMs` asked the object storage for its S3 client through `IObjectStorage::tryGetS3StorageClient`, which exists only under `USE_AWS_S3`; a build without the AWS SDK (the Fast test configuration, `ENABLE_LIBRARIES=0`) does not compile. Without the S3 client there is no connect timeout to freeze, so the function answers what it already answers for an S3-less object storage: nothing frozen, the attempt timeout stands alone. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=d42758b9fa1bdd0b00435063a148e22a46ae4c08&name_0=PR&name_1=Fast%20test PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> (cherry picked from commit 8d2b472e475113d32c28b7c66d598e39120f1668)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
…builds `ensureBackendMatchesBudget` refuses a mismatched handoff with a `LOGICAL_ERROR`, and under `DEBUG_OR_SANITIZER_BUILD` a logical error aborts the process before any handler sees it (src/Common/Exception.cpp). The two mismatch tests asserted `EXPECT_THROW`, so the ASan, TSan and MSan unit-test lanes died on the first of them (`Unit tests (asan_ubsan)`, `(tsan)`, `(msan)` on PR ClickHouse#2300). They now use the split the file already applies to the same class of refusal: `EXPECT_DEATH` on the message under sanitizer builds, `EXPECT_THROW` otherwise. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=12c9eaf97bfb727332feeebd6db8a5ea2245c0ce&name_0=PR&name_1=Unit%20tests%20%28asan_ubsan%29 PR: Altinity#2300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> (cherry picked from commit 35846e58068e556efcb989f885b99851aeea1f1a)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
…logical error CI evidence (PR ClickHouse#2300 run 3, Altinity#2300, regression suite tiered_storage_cas, server log 11:51:36): seven MergeTreeBackgroundExecutor movers made the first writes of a fresh table onto a CAS disk concurrently for part all_1_1_0. Every one ran CasRefLedger::resolveNamespaceLife, whose loop read "no entry" and called CasRefCatalog::createNamespace. The winner's three steps completed within ~120 ms, so a loser's own pre-check read inside createNamespace found the entry already Live and hit the LOGICAL_ERROR "already carries a catalog entry (state 'live')", aborting the server under DEBUG_OR_SANITIZER_BUILD. createNamespace's pre-check read is a snapshot taken AFTER the caller's own "no entry" read, so a sibling opener of the SAME namespace (concurrent threads of one server: parallel background movers, inserts, per-part FREEZE) can land anywhere in its own three-step sequence in the gap between those two reads -- Creating, Live and Removing are all reachable outcomes of that race, never a caller bug. This is the third catch-point of the same sibling-opener race already documented in this file (still-Creating in the pre-check, and a sibling's step 1 landing between the pre-check and createNamespaceStep1's own read); the pre-check's Live/Removing branch was the one spot still treating the race as a bug. Fix: report NamespaceCreationOutcome::Superseded for every state the pre-check observes, not only Creating, and drop the LOGICAL_ERROR throw. resolveNamespaceLife's loop already knows what to do with Superseded: it re-reads and dispatches from the fresh entry (Live is adopted directly, Removing is refused by the loop's own branch, a still-Creating entry resumes through reconcileStaleCreator + completeCreation). Tests (src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp, gtest_cas_ref_catalog_birth_wiring.cpp): converted the direct Live pre-check test from an expected LOGICAL_ERROR/death pair to an expected Superseded outcome, added the equivalent Removing case, and added a new setCreateNamespacePreCheckHookForTest (modelled on the existing step1 hook) to reproduce the exact CI shape through the production namespaceLife/resolveNamespaceLife path: a sibling's entire createNamespace call completes to Live inside the window right before the loser's own pre-check read, and the loser's namespaceLife call still returns the sibling's incarnation without throwing. Verification (failing-first): with the old throwing pre-check restored temporarily and only the new hook/test infrastructure added, all three new tests failed with the exact CI text ("... already carries a catalog entry (state 'live'/'removing')"). After the fix: release unit_tests_dbms --gtest_filter='CAS*' 2490 tests, 0 FAIL; ASan build_asan/src/unit_tests_dbms --gtest_filter='CAS*' 2494 tests, 0 FAIL. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> (cherry picked from commit f9b7c0a7dedaca2d03ee914eb89fb6fdc9ce7185)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
`StorageMergeTree::prepareMutationEntry` allocates the mutation block number and writes `mutation_<N>.txt` outside `currently_processing_in_background_mutex`; the entry is registered in `current_mutations_by_version` later, in `addPreparedMutationEntry`. Two concurrent mutations (for example two lightweight `DELETE` statements) can therefore register out of order, and `selectPartsToMutate` bounded the applicable range only by the end of the map. A mutate task running between the two registrations applied N+1 alone and produced a part with data version N+1, so mutation N was never applied to it (`upper_bound(data_version)` skips it) yet was reported `is_done = 1` with `parts_to_do = 0`. Rows targeted by the lost mutation silently survived. Bound the range by the lowest block in `committing_blocks` with op `Mutation` (the way `MergeTreeMergePredicate` bounds merges by in-flight updates), report the new postpone reason in `system.mutations`, and add a `PAUSEABLE_ONCE` failpoint between the file commit and the registration so the inversion can be forced deterministically. The new stateless test fails before the fix (the second mutation is reported done, half the rows survive) and passes after it. Reproduced locally with two parallel lightweight deletes on a content-addressed disk (5 of 22 runs lost rows; the slower metadata write widens the allocate-to-register window) and traced to the generic MergeTree code; upstream master carries the same selection logic. Seen in CI as the `cas_lightweight_delete` regression suites of Altinity#2300 reporting surviving rows after concurrent deletes. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit bfa7236eeb19882ee484482981643018a691d7b0)
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 14, 2026
…ails the commit `ContentAddressedTransaction::publishStaging` abandons the scratch build right after `repointRef` has made the real manifest live (and after `dropRefIfPresent` on the removal path). That abandon appends a precommit-removal transaction to the ref log; when the append lane refuses it before any request is sent (lease not healthy, lane not ready, deadline exhausted) `PartWriteTxn::abandon` throws `retrying later`, and the throw escaped `commit`. One of its callers is `MergeTreeTransaction::afterCommit`, which is `noexcept`: it writes `txn_version.txt` into a committed part, and on a content-addressed disk that write is a transaction over a live ref. The escaped exception terminated the server (CI run 10 of Altinity#2300, Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2), test `01169_old_alter_partition_isolation_stress`, query `COMMIT`). The abandon after a durable step is bookkeeping: the commit outcome is already recorded, and a precommit binding left live is reclaimed after a remount, which is what the destructor's abandon already tolerated. Route the three abandons (after repoint, after ref drop, destructor) through one `noexcept` helper that logs a refused abandon and resets the build. A test seam mirroring `armPromoteFailureForTest` throws where the real abandon would, so the test can prove the commit and the repointed ref survive it. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 25051b967f0c1324a32d166c7543b5475df0de4f)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #2219
Closes: #2291
Related: #2233
Related: #2159
What this is
Follow-up to #2159 (the initial
casmetadata-storage subsystem): correctness fixes, aperformance/reliability overhaul of the backend request layer, a wire-format key rename, and
three merged features (GC fold read-ahead, forced relink on fetch, GC teardown no longer
blocking on a round), plus a first phase of write-lane serialization for hot control objects
and a GC-cost cut for write-once keys. 22 commits, ~300 files, +38.6k/-19.4k restricted to CAS
paths. This is a curated, squashed reconstruction of the
cas-gc-rebuilddevelopment tree:internal design docs, plans and the soak-harness (
utils/ca-soak) are intentionally not partof this PR (dev-only tooling, no shipped behavior).
How the series is structured
S3_ERROR/NETWORK_ERROR/timeout during a GC round now yieldsAborted(keeping leadership) insteadof an indistinguishable
Failedthat zeroed real counters and droppedi_am_leader; theemulated blob-publication path streams instead of materializing ~1 GiB in memory. (Related CAS: replica HTTP dies on green-path soak after relink NETWORK_ERROR storm #2233)
NO_REPLICA_HAS_PARTinstead ofNETWORK_ERRORfor a relink-confirm refusal — adesigned fail-closed outcome, not a network fault; fixes false Error-level triage and
part_loghygiene checks. (Closes CAS: relink proof refusals are logged at Error with a stack trace despite being an expected outcome #2219)WireKey,EnumWireTablewithset-equality coverage proofs) for every CAS wire-format codec; no bytes change yet.
abbreviated/single-letter keys to descriptive names (
kind,outcome,state, ...).decode), plus fixes that let the full stateless suite run locally.
cas: recommend single-replica merges (doc).ALTER TABLE ... EXPORT PARTITIONnow works from a source on a CAS disk — it readsthrough a sequential source and writes via a sink, nothing is hardlinked on the source
disk, so the CAS refusal was by omission, not by substance. (Closes CAS: EXPORT PARTITION from a CAS source is rejected with SUPPORT_IS_DISABLED #2291)
storage.buckets.getIAM grant turned into a hard outage; now warns and continues. Plusthree live-GCS test-suite corrections found on the first credentialed run.
any mutation of the same namespace, which live-GCS testing showed livelocking two
replicas for 40 minutes under sustained write load; now scoped to the ref actually asked
about.
HEADon a cache hit (a manifest idnames its content, ever); retires the
part_folder_validatesetting that existed only topace that
HEAD.CasRequests/CasOperationrequest engine (core) — replaces the free-formToken(which let an empty value pass as a fenced condition, silently turning a conditional write
into an unconditional overwrite, among 8 other defect classes found across two design
reviews) with a type only the backend can mint, a deadline-bound retry engine, and a
SingleAttemptrequest mode that halves the request cost of a control-object read.matching test-double migration.
Incarnation→Etag/TokenType→Dialectrename, gtest suite onto the engine, deletethe old controller — naming the type by its actual role instead of its wire field;
~120 mechanical test-file migrations; retires the ~1500-line legacy controller.
cas_gc_read_concurrency) — overlaps the fold's small-objectround trips on a bounded pool without moving any decision off the round thread; measured
1.65x on the round overall (2.4x on
fold_ref_intake, which a live-GCS soak measuredtaking 97+ minutes unfinished).
every pool of its storage policy instead of one guess, so a shared-pool fetch never moves
bytes just because a TTL rule or volume order disagreed with the guess.
its next request/retry-sleep/refill via a teardown liveness carried on the open request
plane, recorded as
Stoppedrather thanAborted.instead of once per listed manifest, late/read-ahead manifest reads, and a bulk-delete verb
for owner-removed manifests and ref-object cleanup. Measured on real GCS:
fold_reduce300-380s → 2-5s,manifest_deletes617s → 2s,ref_object_cleanup204s → <1s.for
cas/ref_catalog's conditional writes, replacing same-process write races (measured:DROP TABLEp90 11.9s/max 34.7s, 113PreconditionFailedin 80s from 53 threads) withqueuing and combined commits.
teardown hook, plus a stateless-lane GC-scheduler interval tuning.
SingleAttemptconditional-write attempt is resolved by an outer retry loop, not terminal; logging it at
Error was a false-positive operator signal.
Verification
CAS*gtest gate green throughout (the request-engine migration alone: 2406/2406 at itsfinal checkpoint).
test_cas_replicated_relink,test_cas_gcs,test_cas_gc_sharded,test_cas_gc_bulk_deleteintegration suites.
bodies for exact figures); the hot-key write lane's contention measurements are from ten
parallel stateless CI jobs.
cas-gc-rebuildcommit each squash groupwas cut from is empty by construction (verified file-by-file before opening/updating this PR).
Developed with AI assistance (Claude); every commit carries
Co-Authored-ByandSigned-off-by.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Improvements and fixes to the experimental content-addressed storage (
cas) metadata-storagetype:
ALTER TABLE ... EXPORT PARTITIONnow works from a CAS-disk source; fetch-by-relinkalways lands a same-pool part on that pool's disk instead of occasionally streaming its bytes;
a disk's teardown no longer blocks on an in-flight GC round; several GC-round cost cuts on
real object storage (round-trip counts down by 60-300x on the measured phases); a reliability
overhaul of the CAS-to-object-storage request layer closing several conditional-write edge
cases; and the CAS wire-format's internal JSON keys are renamed from abbreviations to
descriptive names (format generation reset; no compatibility concern, since CAS has no
released, persisted data yet).
Documentation entry for user-facing changes
Updated in this PR:
docs/en/antalya/cas/architecture/{backend,garbage-collection,manifests-and-refs,mounts-and-leases,read-path,replication,storage-layout}.md,docs/en/antalya/cas/{index,configuration,bucket-requirements}.md,docs/en/antalya/cas/operations/{debugging,monitoring,troubleshooting}.md,docs/en/operations/storing-data.md,docs/en/operations/system-tables/{cas_log,cas_gc_log}.md.CI/CD Options
Exclude tests:
Regression jobs to run: