fix(metadata-protocol): stop persisting derived provenance keys, and restate tenant authorship at hydration for every type - #16796
Conversation
…ance keys wip Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADLdAs2pVcH17h9tZKWMBg
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADLdAs2pVcH17h9tZKWMBg
…restatement Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADLdAs2pVcH17h9tZKWMBg
…rip-derived-provenance-keys
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADLdAs2pVcH17h9tZKWMBg
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADLdAs2pVcH17h9tZKWMBg
📓 Docs Drift CheckThis PR changes 1 package(s): 5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 1 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 10 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 492ed4ca5444440ffd5d06f822a6460bf2f999b7 && git checkout 492ed4ca5444440ffd5d06f822a6460bf2f999b7
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin c930f859714de408ba0221f435ff957ed9e64759 a35c00ffa25b41fedcc651b0b6c54303c4200db2 && git checkout -B drift-repro c930f859714de408ba0221f435ff957ed9e64759 && git merge --no-ff a35c00ffa25b41fedcc651b0b6c54303c4200db2
node scripts/docs-audit/affected-docs.mjs --json c930f859714de408ba0221f435ff957ed9e64759
|
Contract review (
|
| file | status |
|---|---|
.changeset/derived-provenance-write-door-and-hydration.md |
added |
packages/metadata-protocol/src/protocol.ts |
+106/−1 |
packages/objectql/src/protocol-derived-provenance-doors.test.ts |
added (440) |
packages/objectql/src/protocol-meta.test.ts |
4 pins updated |
packages/objectql/src/plugin.integration.test.ts |
1 pin updated |
packages/objectql/src/protocol-boot-hydration-scoped.test.ts |
1 pin updated + renamed |
Governed paths touched: NO (docs/adr/**, .claude/**, skills/**, AGENTS.md, CLAUDE.md, content/docs/releases/** — none). The documentation / tooling labels are the auto-labeler's, not a governed touch. Maintainer-only merge on governed grounds: no.
2. Clause-② re-derived independently: yes — and the body's own patch argument contradicts its own declaration
Re-derived from code, not from the body:
- Persisted metadata shape narrows.
saveMetaItem(protocol.ts:15006) now drops_packageId/_packageVersion/_provenancefrom every body it persists, for every type includingobject(the strip sits before the type branch). The storedsys_metadatabody is a persisted shape; it changed. - Hydrated registry-entry shape widens.
hydrateOverlayIntoRegistry(:14131) now registers{...data, _provenance: 'org'}for every non-objectrow with no artifact. That entry is whatSchemaRegistry.getItem/getAllApps/listItemsanswer — an exported objectql surface — and the PR's own six updated deep-equality pins are the measurement of that change. - The accept set of
PUT /meta/TYPE/NAMEis unchanged (the three keys are still accepted and silently dropped), so the request contract is not narrowed.
⇒ Clause-②: yes is correct. The body's argument that the act is "the opposite of additive" and therefore patch is where it goes wrong: the WHICH LEVEL ruling says a fix( that changes no public surface stays patch; this PR declares — correctly — that it changes one.
3. Changeset
- Package name
@objectstack/metadata-protocol: correct (the only package whosesrc/**moves; it is in thefixedgroup, version 17.3.0, not private). - Level
patch: wrong per the gate that enforces the batch [WIP] Add query enhancements and advanced validation features #35 ruling — see F1.scripts/check-changeset-no-major.mjsjudgeLevel: "enforce — declaredyes,patchon a grown package → exit 1". BothCheck Changesetruns on the head are red on exactly this. - No
**BREAKING**banner and no ADR-0087 disposition: correct — nothing an author can write is removed or renamed;check-adr-0087-registrationpasses. - FROM/TO: the body states what stops being persisted and what is restated; it does not state the served-document residual (F2). Nothing deliberately removed is silently dropped.
4. Tests
- New file
protocol-derived-provenance-doors.test.ts: realObjectQL+ realSchemaRegistry+ stub driver (not an engine double). No.skip/.only/.todo/skipIfin any of the four changed test files. - Would redden on revert, reasoned from code: the door-1 pins assert
not.toHaveProperty('_packageId'|'_packageVersion'|'_provenance')on the stored body; withoutstripDerivedProvenancethe repository'sputpersistsJSON.stringify(body)verbatim (sys-metadata-repository.ts:666) → red. The door-2 pins seed a row with_provenance: 'package'and assert the hydrated entry is'org'; withoutstateTenantAuthorshipthe hydrator registersdataas-is → red. The repro leg: with both doors absent,isCodeArtifactBody(metadata-core/src/code-artifact-provenance.ts) accepts truthy non-sentinel_packageId+ non-orgprovenance,getArtifactItem's bare-key fallback (registry.ts:3759/3777) returns the overlay,NOT_OVERRIDABLEfollows → red. Not vacuous. - Negative control (criterion 3) registers a real artifact under the composite key and asserts 403 both without and with a hydrated overlay: it discriminates an "unconditional allow" implementation.
- Criterion 5 asserts every artifact envelope key wins over both the stored copy and the
'org'stamp — the ordering contract of door 2 is pinned. - Typecheck program:
packages/objectql/tsconfig.test.jsonincludessrc/**/*; the new file has no entry in the shrink-only debt ledger, so it must compile at zero errors —Type Check · workspaceis green on the head. - Caveat the body admits: the suite resolves
@objectstack/metadata-protocolthroughdist/, so locally it reddens only after a rebuild. CI ordersBuild CorebeforeTest Core, so the gate is real there.
5. Persistence / hydration specifics
- Stops being persisted: exactly
_packageId,_packageVersion,_provenance, on thesaveMetaItemdoor only, for all types (objectincluded)._lock/_lockReason/_lockSource/_lockDocsUrlare persisted unchanged (pinned). - Restated at hydration:
_provenance: 'org'on a copy, for every non-objectrow reachinghydrateOverlayIntoRegistry(bootloadMetaFromDb, read-sidegetMetaItemson the unscoped kernel, and the meta overlay: a just-saved overlay is listed but not dispatchable for a short window — a cache between saveMeta and resolveRouteActionDeclaration lags the write #4521 write-through), applied beforemergeArtifactProtectionso an artifact's_provenance/_packageId/_packageVersion/_lock*still overwrite it. Theobjectlimbs (applyObjectRegistryMutation, bootregisterObjectbranch) are byte-for-byte untouched. - Backward read compatibility: a row written before this PR still parses and still hydrates (no schema change, no migration of at-rest bytes). It does not hydrate identically: its registry entry's
_provenancebecomes'org'where it was previously the stored value or absent — that is the fix, and the six updated pins are the measurement. But the served document for an at-rest poisoned row is not corrected by this PR — see F2. That is a backfill question, not a migration of this PR, and it must be stated.
6. Criterion-6 reading, spot-checked
The body's claim that getMetaItems re-stamps _packageId from the package_id column onto the body handed to the hydrator is confirmed at :7288-7290 ((data as any)._packageId = recPkg) → :7380 (hydrateOverlayIntoRegistry). Door 1 cannot reach it; door 2 makes it non-load-bearing because isTenantAuthored asks _provenance, not _packageId. The "both doors are not redundant" reasoning holds.
7. CI on head bbe4590c0
37 check runs: 29 success · 2 failure · 6 skipped · 0 in progress. Red: Check Changeset ×2 (pr-automation on opened and on labeled), both on the level axis above. Skipped: Packed-tarball smoke ×2 (opt-in), Auto Label, Check PR Size, Build Docs, Console Pin Gate. mergeable_state: unstable (the red gate). Head is 11 commits behind origin/main at review time (base recorded at CI: b38821d1; merge-base now 8b37a097). No reviews on the PR; one bot comment (docs-drift, advisory).
Findings
F1 — CHANGES REQUIRED. Changeset level patch is refused by the gate that enforces the batch #35 ruling, and the refusal is correct. The PR declares Clause-②: yes (and carries the needs:contract-review carrier) while grading @objectstack/metadata-protocol patch; check-changeset-no-major.mjs judgeLevel returns enforce, exit 1 — both Check Changeset runs are red. This seat's independent re-derivation agrees the surface changes (persisted body narrows; hydrated registry-entry shape widens). Expectation: take the gate's route 1 — change the one word in .changeset/derived-provenance-write-door-and-hydration.md to minor, and rewrite the body's "Changeset" paragraph so it no longer argues patch from "not additive" while declaring yes. ⛔ Do not take route 2 (flipping the declaration to no) — the declaration is the accurate half.
F2 — Must be stated in the PR body and changeset: door 2 corrects the authorization verdict, not the served document. getMetaItem step 1 (:7860-7875) serves the raw overlay row body with only _packageId stamped from the column and no _provenance restatement; mergeArtifactProtection only fires when an artifact exists. getMetaItems likewise serves the overlay data (which wins the merge over the restated registry entry). So a row poisoned at rest keeps being served as _provenance: 'package' by GET /meta/app/x — the UI keeps badging it as package-provided — until the row is re-saved (door 1) or backfilled by cloud#2069 / cloud PR #2111. The refusal is gone; the misattribution in the served body is not, for that population. Expectation: one sentence in the changeset and the body naming this residual and that the cloud backfill is load-bearing for it; carry it into the criterion-7 reply on cloud#2069 so cloud does not read "door is shut" as "backfill no longer needed".
F3 — Non-blocking, confirm the body's "noted, not filed" is complete. promoteDraft (:16942) and restoreVersion (rollbackMetaItem :20331, revertCommit :19941) hand a stored draft/history body straight to repo.put without stripDerivedProvenance. A rollback to a pre-PR history version therefore re-persists the three keys into the active row. Door 2 keeps that harmless for hydration; the corpus does not converge on those doors. Expectation: acceptable as scoped, but the body's claim "door 1 covers them transitively" is only true for bodies written after this PR — say so.
F4 — Minor. Door 2 restates only _provenance; a poisoned row's stale _packageVersion survives onto the hydrated entry (non-load-bearing for authorization, cosmetic until backfill). Expectation: none beyond F2's sentence.
F5 — Minor test gap. The strip is type-agnostic, so object bodies also lose the three keys at rest; the criterion-4 object case writes _packageId/_provenance into the request but asserts only the hydrated entry, not the stored body. Expectation: optional one-line not.toHaveProperty on the stored object row, so the type-agnostic scope of door 1 is pinned rather than implied.
F6 — Process, not a defect. The "both doors" decision was taken by triage + PM as a determinacy call, with an explicit invitation to rebut; the card's own text called it a maintainer call. This seat finds the determinacy argument sound (a stored copy is never load-bearing on the read side; only the lie is negated) and the ablation reasoning that the two doors cover different populations verified from code. Expectation: none — recorded so the maintainer sees the decision was a seat's, not theirs.
Summary for the maintainer: the fix is exactly what was dispatched, the pins discriminate, no governed path is touched. The single blocker is one word in the changeset (F1); F2 is a required disclosure, not a code change.
Generated by Claude Code
`check-changeset-no-major`'s level axis refuses a PR that declares `Clause-②: yes` while grading a package whose `packages/*/src/**` it moves as `patch`. Route 1 of the two the gate names: the declaration is right and the level is wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADLdAs2pVcH17h9tZKWMBg
CI is green on
|
Fixes #16702
Clause-②: yes
The defect
saveMetaItempersisted the caller's_packageId/_packageVersion/_provenanceverbatim, and for every non-objecttype boot hydration read that stored body back as truth.metadata-read-decorations.tsdeliberately does not strip_provenancefrom a served document, so the ordinary StudioGET /meta/app/pet_hospitalthenPUT /meta/app/pet_hospitalround trip wrote_provenance: 'package'into the tenant's ownsys_metadatarow. From there the row's own bytes decided:isCodeArtifactBodyaccepted a truthy non-sentinel_packageIdwith non-orgprovenance,SchemaRegistry.getArtifactItem's bare-key fallback returned the overlay as an artifact,isArtifactBackedturned true, and every later write was refusedNOT_OVERRIDABLE— permanently, since the next boot re-derived the same verdict from the same row.The failure also misattributes: the operator is told the item is "provided by a code package" when no code package publishes it at all.
Both doors, because they cover different populations
Door 1 — the write door.
saveMetaItemnow drops exactly_packageId/_packageVersion/_provenancefrom the body it persists, one beat after the existingstripReadDecorationscall and for the same reason. The_lock*family is deliberately untouched: a lock is author-declarable and dropping one is the fail-open direction, the line cloud PR #2065 drew at its own producer.This is a new, write-door-scoped strip and not a new member of
METADATA_READ_DECORATIONS. That list is shared with every consumer that re-parses a served document, and its own header states on purpose that the protection envelope stays on a served body so provenance survives a re-parse.Door 2 — hydration.
hydrateOverlayIntoRegistry— the one choke point boot, read-side and write-through hydration already share — now states_provenance: 'org'on a copy beforemergeArtifactProtectionruns. That is the same sentenceapplyObjectRegistryMutationand the bootobjectlimb have always written, said once for every other type. The order is the whole contract: where a real artifact exists its envelope still overwrites_provenance(and_packageId/_packageVersion/_lock*) on the way out, so ADR-0010 section 3.3 precedence is unchanged in both directions.Door 2 is not belt-and-braces. Measured, not argued: with door 2 ablated and door 1 live, a row whose stored body carries no protection key at all is still hydrated as a code artifact, because
getMetaItemsre-stamps_packageIdonto the body from the row'spackage_idcolumn a few frames before handing it to the hydrator. The write-door strip cannot reach that path; the restatement can.Acceptance criteria, and where each is asserted
New file
packages/objectql/src/protocol-derived-provenance-doors.test.ts— a realObjectQLengine, a realSchemaRegistry,sys_metadataregistered, no code package loaded.#16702 — the card's reproduction leg, run as written. Writes theappcarrying_provenance: 'package', boots a fresh engine + protocol over the same driver, callsloadMetaFromDb(), then saves again. Output pasted below._lock*door 1: one asserts the three keys are absent from the persisted body andname/labelsurvive; the other sends_lock/_lockReason/_lockSource/_lockDocsUrlalongside them and asserts all four survive the write.criterion 3. A real artifact registered under the composite keyapp.sdbh:pet_hospitalis still refusedNOT_OVERRIDABLE/ 403 — once with no overlay, and once after a boot that hydrated a tenant overlay of the same name.criterion 4. Theobjectbranch keeps stamping_provenance: 'org'and stays editable across a restart, and it still registers throughregisterObject(contributor layers) rather than the sharedregisterItempath —getObjectandgetAllObjects(packageId)both answer. Theobjectbranch source is untouched by this PR.mergeArtifactProtectionprecedence_lock: 'full',_lockReason,_packageId,_packageVersionand_provenance: 'package'while the overlay's authoredlabelstill wins; with no artifact the restatement stands and nothing is invented.row.package_idreadingCriterion 1, run as written
On
origin/main73053ed27b, the same path, byte for byte the card's own output:Criterion 6 — the reading on the
row.package_idcolumn pathTriage named one site and asked whether the column is a second poisoning path. The dispatch measured three. Re-derived by symbol on
origin/main73053ed27b(protocol.ts, 21,885 lines; firing controls on the same file:_packageId34 lines,_provenance12 lines, a nonsense control 0)::4922const packageId = row.package_id;— infoldStoredCollection. Stamps_packageIdonto a freshly parsed, ephemeral body that is pushed into the runtime authoring gate's reference-resolution universe and returned. Nothing persists it and nothing registers it. Not a poisoning path.:16054const packageId: string | null = row.package_id ?? null;— inmigrateStoredMetadata. Reads the column into the migration report row and passes it as thepackageIdparameter of the re-save, never as a body key. Not a poisoning path — and after door 1 an--applymigration pass now actively cleans the rows it rewrites.:19972packageId: row.package_id,— inrevertCommit, whererowis a commit row and the value goes torecordPackageCommitas a commit column. Not a metadata body at all. Not a poisoning path.But the pattern
row.package_idcannot express the shapes that matter. A whole-file census of thepackage_idspelling (61 occurrences) finds the cast and destructured forms it misses, and four of those DO write the column into a body::7208,:7354,:7736,:7805— all ingetMetaItems/getMetaItem, all(record as ...).package_id, eachif (recPkg && body._packageId === undefined) body._packageId = recPkg;.:7208is the live one. The sameoverlaysarray it stamps is handed tohydrateOverlayIntoRegistryat:7302on an unscoped kernel, so an ordinary list read re-injects_packageIdfrom the column into the in-memory registry entry. That is a genuine second path toisArtifactBacked, it needs no poisoned bytes at rest, and door 1 alone cannot close it. Door 2 does, becauseisTenantAuthoredasks_provenance, not_packageId. Asserted bythe read-side hydration seam (getMetaItems) is covered by the same restatement.No fix for the column path is folded into this PR. None is needed for the defect this card names: the stamp is a legitimate read-side surfacing of the row's package binding (the sidebar package filter consumes it), and door 2 makes it non-load-bearing for the authorization verdict.
Verification
Ablation — two legs, PLANT mode, blob-hash-verified restore. Each leg mutated
protocol.ts, rebuilt@objectstack/metadata-protocol(the suite resolves it throughdist/), and proved the marker reached the built artifact withscripts/ablation-dist-preflight.mjsbefore any colour was read.DERIVED_PROVENANCE_KEYSswapped for a nonsense key): marker present in 2 built files; 2 cases red, both door-1 pins._provenance: 'org'swapped for a non-orgliteral): marker present in 2 built files; 3 cases red — the two at-rest/read-side pins and the no-artifact restatement pin.git status --porcelainempty, rebuild plus--absentconfirms the marker is gone from all 24 built files, restored suite 11 of 11 green.One thing stayed green under both legs, and that is the finding, not a defect in the ablation: the reproduction leg itself. Either door alone closes it, which is exactly triage's "they cover different time windows". The layer holding it is the other door. The repro leg's own discriminating power is proven by the pre-fix run with neither door present, which reproduced the 403 verbatim.
Suites.
@objectstack/metadata-protocol169 files / 2,433 tests pass, 2 files / 10 cases skipped — named and counted: thedescribe.skipIf(!MYSQL_URL)limbs ofsrc/migrations/sys-setting-identity-index.live-mysql.test.ts(5) andsrc/migrations/seed-tenancy-backfill.live-mysql.test.ts(5), pre-existing and unrelated.@objectstack/objectql286 files / 4,819 tests pass, 0 skipped. Consumer sweep, because the served shape can change:@objectstack/rest192 files / 3,241 tests and@objectstack/runtime240 files / 3,340 tests, both fully green, 0 skipped.Typecheck legs, read out of each
package.jsonrather than assumed. metadata-protocol:tsc --noEmit, 1 leg. objectql:tsc --noEmit, thentsc --noEmit -p tsconfig.scripts.json, thencheck:test-typecheck(its own--self-testplus--package packages/objectql --project tsconfig.test.json) — 3 legs. All green.Existing pins updated, deliberately. Six cases asserted deep equality on a hydrated non-
objectregistry entry and now see one more key. Each was updated totoEqualagainst the exact new shape rather than relaxed totoMatchObject, so an unexpected extra key still reds them:protocol-meta.test.ts(4),plugin.integration.test.ts(1), andprotocol-boot-hydration-scoped.test.ts, whose "registers the row unchanged" case is renamed to what it actually pins — no artifact envelope is grafted, while_lockand_packageIdstay absent and_provenanceis now the server's own statement.Gates.
node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackderived 59 families; all 59 were run and reconciled with--ran:57 exit 0. Two exit 3, which is
PREREQUISITE NOT METand is neither a pass nor a finding —check:dual-build-cjs-loadsandcheck:type-check-debtboth refuse to measure without a whole-workspace build closure, and both say so in their own output. They are declared to CI, which builds first.check:objectql-double-limitcaught a real defect in the new stub driver on its first run (afinddouble ignoring the caller's bound); corrected inbbe4590cand re-run green.Lint, as a proven narrowing rather than a skipped run. Population read from eslint's own config: 6,348 tracked files carry a lintable extension and none is removed by the single global-ignore block, whose whole content is
node_modules/dist/build/.next/.turbo— untracked build output that a tracked-file census matches 0 files under, as the config's own header records. Files linted, counted from--format json: 5, the entire lintable half of the change set,eslint --no-inline-configexit 0, 0 errors and 0 warnings. Invariance: this repo runs oneeslint.config.mjsand it never enables type-aware linting for any file (noparserOptions.project, no typed rules), so no verdict on any of the 6,343 untouched files can depend on this diff. Measured atbbe4590c.Changeset.
minor, on@objectstack/metadata-protocol— raised frompatchby the correction immediately below. The original derivation is kept, because it is still right about the DIRECTION of this change: the governing WHICH LEVEL text in.github/workflows/pr-automation.ymlsays "afix(that changes no public surface stayspatch", andminoris owed for "a purely additive widening of a published package's public surface (a new exported symbol on anindex, a new accepted key or value)" — and this act is the opposite of additive, since no export is added, no key or value newly accepted, and three keys stop being persisted.none/ theskip-changesetlabel was rejected and stays rejected:AGENTS.mdis a floor againstnonefor a bug fix in a released package, not a ceiling, and this diff does publish from a released package. NoBREAKINGbanner and therefore no ADR-0087 disposition: nothing an author can write is removed or renamed, no export or config field changes, the three keys are still accepted on input, and the read side recomputes all three on every read — so there is no FROM-TO mapping to ship and no migration to prescribe.check-adr-0087-registrationagrees on this diff.patchreads right. That is the wrong axis for the FLOOR.check-changeset-no-major's level axis refuses any PR that declaresClause-②: yeswhile grading a package whosepackages/*/src/**it moves aspatch, and it never asks which way that surface moved. This PR declaresClause-②: yes(line 3 of this body) and movespackages/metadata-protocol/src/**, so the floor isminorwhatever the direction — the level was wrong, not the reasoning about direction. Route 1 of the two the gate names is taken here: "The declaration is right and the level is wrong -> raise it tominor." Route 2 — correcting the declaration at the producer — is the review seat's, and is deliberately not exercised: theClause-②:line is untouched, noneeds:contract-reviewlabel was added or removed, and no tolerance was added to the gate. The release effect of the raise is nil:@objectstack/metadata-protocolsits in the singlefixedversion group in.changeset/config.json(one group, 70 members), so it already moves in lockstep with every other package in that group.验收备注
Triage's seven, adopted verbatim (
5578409434, quoted unchanged):Out-of-scope notes, filed nowhere
getMetaItem/getMetaItemsstamp_packageIdonto a served body from thepackage_idcolumn at four sites (:7208,:7354,:7736,:7805) and never stamp_provenancebeside it. Noted, not filed: with door 2 in place the asymmetry no longer decides an authorization verdict, so it is an observation about symmetry, not a reproducible defect.publishMetaItem,rollbackMetaItemandrevertCommitwrite bodies that did not come throughsaveMetaItemin this call. Every one of them originates from a bodysaveMetaItemalready wrote (a draft, a history entry, a commit item), so door 1 covers them transitively and door 2 covers whatever slipped through earlier. Noted, not filed: no repro.Draft only. Not ready, not queued, no auto-merge.