feat(service-analytics)!: refuse an aggregate a datetime measure's field type cannot carry, and reconcile the storage-form annotations to one measured statement - #16778
Conversation
…cannot carry WIP — compile-leg refusal + the four reconciled datetime-storage annotations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
…g-datetime-measure
…rce fields The full table refuses `min`/`max` over the string classes and the boolean rows, both of which this platform answers on purpose and pins with tests (#15768, maintainer ruling #11152). Executing those is a product judgement that belongs to #16099; the temporal rows carry no such collision and are the ones this card is about. Re-points the two `measure-result-type.test.ts` fixture measures that aggregated a datetime column, and corrects the module header that recorded the missing refusal as an open finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
📓 Docs Drift CheckThis PR changes 2 package(s): 1 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 1 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 130 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 7be3862e7cb967d7e4a4497d383290c8c356c614 && git checkout 7be3862e7cb967d7e4a4497d383290c8c356c614
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 941232040d94eccc3d86204c7e96c84b775a1000 80ec9f2b6a7e076b9c9683334224f62234979ac1 && git checkout -B drift-repro 941232040d94eccc3d86204c7e96c84b775a1000 && git merge --no-ff 80ec9f2b6a7e076b9c9683334224f62234979ac1
node scripts/docs-audit/affected-docs.mjs --json 941232040d94eccc3d86204c7e96c84b775a1000
|
Contract review (
|
Fixes #16737
Also implements the compile leg described on #16099 — that card stays open and its
disposition is the PM's; nothing here is a verdict about it.
1. The storage reality, established first — and it is a THIRD answer
The card said the package states two incompatible answers. Measured on
origin/main,neither is current. A SQLite
Field.datetimecolumn has ONE storage form:canonical UTC TEXT,
YYYY-MM-DDTHH:MM:SS.sssZ(#3912/#3928).How it was measured, not recalled:
SqlDriver.storageDatetimeValuecanonicalises on the write path, andsql-driver-datetime-canonical-storage.test.tspins every accepted input shapefolding onto the same stored string — a
Date, ISO…Z, ISO with an offset, a naivewall clock, an epoch number, an epoch string, a bare calendar day. The
NOW()columndefault writes the same
strftime('%Y-%m-%dT%H:%M:%fZ', 'now')bytes.initObjectsrunsbackfillCanonicalDatetimesat schema sync;needsLegacyDatetimeRepairis the one predicate for "might this column still hold apre-canonical value". Two cases keep it true: a table not yet backfilled, and an
external / unmanaged object (
registerExternalObjectnever marks its datetime columnscanonical).
timestamptz/DATETIME(3), one on-disk shape by construction.Re-driven live, in this session, on a fresh SQLite:
So the filer's
text|2026-05-19T00:00:00.000Zreading is exactly what the currentdriver produces. ⭐ The mixed INTEGER/TEXT column is the transitional state, and the
flat "a SQLite
Field.datetimeIS an INTEGER epoch" has been wrong since #3912.2.
coerceTemporal— the thing triage named as unmeasuredIt is correct for the mixed-storage reality, and it is correct for the canonical one.
Its docblocks were the stale part.
NativeSQLStrategy.coerceTemporal→ctx.coerceTemporalFilterValue→
SqlDriver.temporalFilterValue→coerceFilterValue) canonicalises to the samefunction the write path uses, so an ISO or bare-day comparand becomes the stored form.
Pinned in
sql-driver-temporal-dialect.test.tsandsql-driver-analytics-datetime.test.ts.temporalColumn→temporalFilterColumnSql) emits the repair CASEonly while
needsLegacyDatetimeRepairholds, and the bare column otherwise —pinned by
sql-driver-datetime-canonical-storage.test.ts(
expect(driver.temporalFilterColumnSql('evt','at','"at"')).toBe('"at"')after backfill).No code change was needed there. A test case was added on the analytics side
(
native-sql-datetime-filter.test.ts) driving the strategy against a hook that returnstoday's canonical UTC text, beside the epoch hook the file was written with — so the
suite covers the live storage form as well as the legacy one.
3. The annotations reconciled — seven sites, not the one the filer quoted
The card named four. A sweep for the claim found seven in source plus two test
narratives, all making the same stale statement. All are now consistent, and the fact is
stated once:
analytics-service.tscoerceTemporalFilterValueField.datetime→ epoch ms"analytics-service.tscoerceTemporalFilterColumn(:493)plugin.ts(:640)plugin.ts(:662)native-sql-strategy.tstemporalColumn(:963)native-sql-strategy.tsbuildFilterClause(:1190)objectql-strategy.tsdateRangeBounds(:1669)Field.datetimeIS an INTEGER epoch (#2034)"NativeSQLStrategyneeds the coercion__tests__/native-sql-datetime-filter.test.ts__tests__/native-sql-datetime-filter-column.test.tsEPOCH_MS(...)re-labelled as a marker, not a claim about emitted SQL4. Refusal, not a definition — and where it stops
Refusal. A definition would have to pick one of two dialect answers and ship it as
the meaning of a number nobody can check. The refusal moves the failure to the person
writing the dashboard, which is the PM direction and the card's own danger analysis.
⛔ It is not a new rule. The contract already exists:
AGGREGATE_FIELD_TYPE_COMPATIBILITYin
@objectstack/spec(#16353, landed as6ba0db4e0), declared under the directorruling of decision batch #59. Until this PR it had zero consumers.
compileDatasetnow reads the shipped predicate and refuses with
DATASET_INVALID/ 400 before anyquery is built, using the declared type the host already supplies through
AnalyticsServiceConfig.sourceFieldMeta(threaded as a new optionalDatasetCompileOptions.declaredFieldTypeprobe, tiered "cannot answer, do not block").The dispatch said: stop and report if refusing would break a use that currently works
and is legitimate. Executing every row of the table does exactly that, in two places,
and both are already recorded in the spec module's own header as overrides of existing
opinions rather than agreement:
min/maxover the STRING classes (text,select,lookup,autonumber, …).measureResultType(A dataset measure over a datetime is typednumberin the analytics response, and a metric tile ignores theformatstyle it is handed — measured on 17.3.0 #15768) types those results as'string', andmeasure-result-type.test.tspins them end to end throughqueryDataset. Measured:enforcing those rows turns 15 cases red.
sum/avg/min/maxoverboolean/toggle. Maintainer ruling [finding]AGGREGATION_ROWShas no boolean column, so the cross-driver aggregation conformance family cannot see a boolean aggregand on any face #11152pins booleans aggregating as numbers on every backend (
AGGREGATION_CASES;driver-sqlcasts the aggregand on Postgres to make it hold). The spec header refersthat collision back to the maintainer as its own decision.
⇒ The gate judges the temporal class only (
TEMPORAL_SOURCE_FIELD_TYPES, thispackage's own shipped statement of it). The verdict is still the table's — nothing is
restated, and
min/maxover a temporal field stay accepted because the table acceptsthem. The temporal rows carry no collision: measured on both dialects, no shipped dataset
in this repo pairs them, and there is no reading on which the mean of a set of instants is
a duration. Whether the string and boolean rows are enforced is a product judgement, and
it is the blocker for the full-table leg. See the acceptance notes below.
5.
derivedis covered by constructionA derived measure's
ofoperands are base measures of the same dataset, and the executorexpands a selected derived measure into them before querying. Both are downstream of the
compile loop, so a dataset carrying a refused base measure never finishes compiling and no
derivedop can be handed its output — including when the selection names only thederived measure, which is the filer's exact shape. Pinned as two cases, one of them
selecting only
cycle_delta.6. Second dialect — measured on real PostgreSQL 16.13
Which half is dialect-specific: the SILENT half is SQLite's. SQLite has no temporal
type, so the stored text is coerced to a number by its leading digits and the call
succeeds. Postgres has a real
timestamptzand noavgover it, so it refuses loudly.The MEANINGLESS half is not dialect-specific — there is no backend on which the mean
of a set of instants is a duration; Postgres simply says so out loud.
select avg('12:00:00'::time)succeeds on Postgresand returns a clock time, while SQLite silently answers
13(the average hour) for thesame data. So
avgover aField.timeis a genuine per-dialect split. The ruled tablerefuses it (
timeappears only in themin/maxrows), and that ruling is executed hererather than re-litigated.
7. Negative controls — this card is aggregation only
All pinned in
aggregate-datetime-measure-refusal.test.ts:avgover anumbermeasure, and over acurrencymeasure — still work end to end,one statement emitted,
AVGin the SQL.datetimeused as a dimension — month grouping, and adateRangewindow — bothreach the engine unchanged (the window asserted present in the engine call).
min/maxover adatetime— still accepted, still return real instants.countover adatetime— still accepted.derivedmeasure over numeric operands — untouched.minover atextfield — still compiles here, with the table's contrary verdictasserted directly, so the scope boundary reads as "the contract says no, this gate does
not act on it" and cannot widen unnoticed.
Tests
Dissolution verification — direction predicted before running, mutation proved on disk
Deleting the
assertAggregateFieldTypeCompatiblecall from the measure loop:The 7 red are exactly the refusal cases (both
derivedcases among them); the 16 greenare the negative controls, the tiering tiers, the scope boundary, the contract-table
assertions and the live SQLite measurement suite. No case can pass vacuously: each refusal
case asserts the envelope AND that no SQL was emitted, and with the gate gone the compile
succeeds and SQL IS emitted. Restored from
HEAD(not a bare checkout), proved by an emptygit diff HEADrather than by an exit code.Gates
node scripts/pm/dispatch-gates.mjs --commandsderived the family; every command was runand reconciled with
--ran. Green, with three NOT MEASURED that are CI's:pnpm check:dual-build-cjs-loads— exit 3, PREREQUISITE NOT MET (needs a whole-farmpnpm build). Declared to CI.pnpm check:type-check-debt— exit 3, PREREQUISITE NOT MET (--re-measureneeds thebuilt closure of 23 workspace deps). Its first half,
check:type-check-coverage, passed.pnpm --filter @objectstack/lint check:doc-formula-expressions— exit 3, PREREQUISITENOT MET (
@objectstack/lintnot built here); not a package this diff touches.ADR-0087 and the ledger:
Repo-wide
pnpm lintwas not narrowed and not run locally; it is CI's whole-root run.Clause-② and the changeset
Clause-②: yes. The dispatch's provisional read wasno; the diff overturns it, onboth the mechanical floor and the gate:
The mechanical floor also applies on its own:
DatasetCompileOptions.declaredFieldTypeisa new key on a published exported type (
packages/services/service-analytics/src/index.tsre-exports
DatasetCompileOptions).needs:contract-reviewis carried on this PR.Changeset:
minorfor both packages, declared BREAKING with a!title. It is anaccept-set narrowing on a published authoring surface, which is breaking — and #16353's own
changeset says in writing that the consumer legs "carry the breaking declaration, its
migration prescription and the ADR-0087 disposition".
minorrather thanmajoris therepo's launch-window convention for accept-set narrowings, stated verbatim in the protocol
18 step's header ("The enforcement itself ships earlier on the 17.x line … this step is
where
migrate metausers are told"). ⛔ Notpatch: nothing here is "just a bug fix" —authoring shapes that parsed and ran now fail. The FROM → TO table is in the changeset body.
验收备注
Reported here for the PM, not filed as issues:
avgover a datetime works on SQLite and errors on Postgres #16099) is blocked on a ruling collision, not onengineering. Enforcing the table's string rows contradicts A dataset measure over a datetime is typed
numberin the analytics response, and a metric tile ignores theformatstyle it is handed — measured on 17.3.0 #15768 (measured: 15 redcases); enforcing its boolean rows contradicts maintainer ruling [finding]
AGGREGATION_ROWShas no boolean column, so the cross-driver aggregation conformance family cannot see a boolean aggregand on any face #11152. The specmodule's header already records both as overrides referred onward. Somebody has to
rule them before the leg can be widened past temporal.
avgover a datetime works on SQLite and errors on Postgres #16099 is still labelledpm:blockedwithBlocked-by: #16353, and spec: declare the aggregate × field-type compatibility matrix (AggregationFunction×FieldType) that dataset measures are refused against (spec half of #16099) #16353 haslanded (
6ba0db4e0, PR feat(spec): declare the aggregate × field-type compatibility matrix (AggregationFunction × FieldType) dataset measures are refused against (#16353) #16684). The blocker is stale; the label is the PM's to move./analytics/query(the non-dataset face) is not covered by this leg. Anauto-inferred
submitted_at_avgmeasure on a Cube still reaches the driver — theruling names two legs (lint + compile) and neither is that face. Observation, not a
defect this PR introduces.
packages/lintstill carries no rulepairing a measure's aggregate with its field type. That is No layer refuses an incoherent aggregate / field-type pair — a dataset measure
avgover a datetime works on SQLite and errors on Postgres #16099's sibling devx card.avgover aField.timeis a real per-dialect split (Postgres answers a clocktime; SQLite answers the average hour). The table refuses it and this PR executes that;
worth a maintainer eye if anyone was relying on the Postgres behaviour.
None of these were filed: a duplicate-search would be needed for each, and items 1–2 are
dispositions on an existing card rather than new work.
🤖 Generated with Claude Code
https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
Generated by Claude Code