fix(metadata-protocol): stop persisting derived provenance keys, and restate tenant authorship at hydration for every type

.changeset/derived-provenance-write-door-and-hydration.md

@@ -0,0 +1,16 @@
+---
+'@objectstack/metadata-protocol': minor
+---
+
+Stop persisting the caller's `_packageId` / `_packageVersion` / `_provenance`, and restate tenant authorship at hydration for every metadata type.
+
+Two seams let a tenant lock themselves out of their own metadata. `saveMetaItem` persisted those three keys verbatim — `metadata-read-decorations.ts` deliberately does not strip `_provenance` from a served document, so the ordinary Studio `GET /meta/app/x` → edit → `PUT /meta/app/x` round trip wrote `_provenance: 'package'` into the tenant's own `sys_metadata` row. For every non-`object` type, boot and read-side hydration then registered that stored body as-is, so `SchemaRegistry.getArtifactItem`'s bare-key fallback accepted the overlay as a code artifact, `isArtifactBacked` turned true, and every later write was refused `NOT_OVERRIDABLE` — permanently, because the next boot re-derived the same verdict from the same row. The refusal said the item is "provided by a code package" when no code package published it at all.
+
+Both halves are closed, because they cover different populations:
+
+- `saveMetaItem` now drops exactly those three keys from the body it persists, so future writes stop poisoning the corpus. The `_lock*` family is deliberately untouched — a lock is author-declarable and dropping one is the fail-open direction.
+- `hydrateOverlayIntoRegistry` — the one choke point boot, read-side and write-through hydration already share — now states `_provenance: 'org'` on a copy before merging the artifact envelope, so rows already written become harmless without being rewritten. That also covers the column path: `getMetaItems` re-stamps `_packageId` onto the body from the row's `package_id` column, which the write-door strip cannot reach.
+
+The three keys are read-side derived — `mergeArtifactProtection` recomputes them from the artifact on every read — so nothing an author wrote is lost and no accepted key or value changes. Where a real artifact exists its envelope still wins over both the stored copy and the restatement: ADR-0010 §3.3 precedence is unchanged, and an item genuinely shipped by a code package is still refused `NOT_OVERRIDABLE`.
+
+One residual is deliberately left open: hydration corrects the AUTHORIZATION verdict, not the SERVED document. `getMetaItem` / `getMetaItems` return the overlay row's own body, and `mergeArtifactProtection` only fires where an artifact exists — so a row already poisoned at rest becomes editable again while `GET /meta/app/x` keeps serving `_provenance: 'package'` (and the UI keeps badging it as package-provided) until that row is re-saved through the write door or backfilled. The `objectstack-ai/cloud#2069` backfill stays load-bearing for that population; this change does not retire it.

packages/metadata-protocol/src/protocol.ts

@@ -1172,6 +1172,84 @@ function mergeArtifactProtection(item: unknown, artifactItem: unknown): unknown
     return out;
 }
 
+/**
+ * [#16702] ADR-0010 §3.3 — the three protection keys that are READ-SIDE
+ * DERIVED, and therefore must never be persisted from a caller's body.
+ *
+ * {@link mergeArtifactProtection} recomputes all three from the artifact on
+ * every read, so a copy stored inside a `sys_metadata` body is never
+ * load-bearing: removing it is observable ONLY where that copy was a lie.
+ *
+ * ⛔ The `_lock*` family is deliberately NOT here, though it shares the
+ * underscore spelling and the same ADR-0010 envelope. A lock is
+ * AUTHOR-DECLARABLE (`protection.lock`, translated into `_lock*` by
+ * `applyProtection`), so dropping one is the FAIL-OPEN direction — cloud PR
+ * #2065 drew that line at its own producer and this door follows it.
+ *
+ * ⛔ Nor is this a second `METADATA_READ_DECORATIONS`. That list is shared
+ * with every consumer that re-parses a SERVED document (`spec`'s
+ * `metadata-read-decorations.ts`), and its header states on purpose that the
+ * protection envelope stays on a served body so provenance survives a
+ * re-parse. This strip is scoped to the WRITE door alone.
+ */
+const DERIVED_PROVENANCE_KEYS = ['_packageId', '_packageVersion', '_provenance'] as const;
+
+/**
+ * [#16702] Remove {@link DERIVED_PROVENANCE_KEYS} from a body about to be
+ * PERSISTED into `sys_metadata`.
+ *
+ * A **silent** strip, for the same reason {@link stripReadDecorations} is
+ * silent: the standard Studio `GET` → edit → `PUT` round-trip echoes whatever
+ * the served document carried, and refusing that round-trip would be hostile
+ * for keys the server stamped itself. What it restores is the invariant the
+ * `_provenance: 'org'` docblock in {@link
+ * ObjectStackProtocolImplementation.applyObjectRegistryMutation} already
+ * states: every row this door writes is tenant-authored by definition, so the
+ * server states that fact rather than reading it back from the caller.
+ *
+ * Returns the SAME reference when there is nothing to strip, so the common
+ * path allocates nothing. Non-object inputs pass through — the caller's own
+ * validation owns those.
+ */
+function stripDerivedProvenance(item: unknown): unknown {
+    if (!item || typeof item !== 'object' || Array.isArray(item)) return item;
+    const dict = item as Record<string, unknown>;
+    if (!DERIVED_PROVENANCE_KEYS.some((k) => k in dict)) return item;
+    const next = { ...dict };
+    for (const k of DERIVED_PROVENANCE_KEYS) delete next[k];
+    return next;
+}
+
+/**
+ * [#16702] State what every `sys_metadata` row IS — tenant-authored, ADR-0010
+ * `_provenance: 'org'` — on a COPY of a body about to be REGISTERED.
+ *
+ * The same sentence {@link
+ * ObjectStackProtocolImplementation.applyObjectRegistryMutation} and the boot
+ * `object` limb already write, said once for every OTHER type at the one
+ * hydration choke point they do not share. Without it the row's own bytes
+ * decide: `isCodeArtifactBody` accepts a truthy non-sentinel `_packageId` with
+ * non-`org` provenance, `SchemaRegistry.getArtifactItem`'s bare-key fallback
+ * returns the overlay AS an artifact, `isArtifactBacked` turns true, and
+ * `saveMetaItem`'s overlay gate refuses the tenant's next write to their own
+ * item with `NOT_OVERRIDABLE` — permanently, because the next boot re-derives
+ * the same verdict from the same row (cloud#970's shape, for non-`object`
+ * types).
+ *
+ * ⚠️ Its ONE caller applies it BEFORE {@link mergeArtifactProtection}, and the
+ * order is the whole contract: where a real artifact exists the artifact's
+ * envelope still overwrites `_provenance` (and `_packageId` /
+ * `_packageVersion` / `_lock*`) on the way out, so package protection is
+ * untouched — ADR-0010 §3.3 precedence is unchanged in both directions.
+ *
+ * On a COPY, always: `registerItem` hands the body to `applyProtection`, which
+ * mutates in place, and the callers own their `data`.
+ */
+function stateTenantAuthorship(data: unknown): unknown {
+    if (!data || typeof data !== 'object' || Array.isArray(data)) return data;
+    return { ...(data as Record<string, unknown>), _provenance: 'org' };
+}
+
 /**
  * ADR-0048 (#1828) — composite dedup identity for the unscoped metadata list.
  *
@@ -14038,7 +14116,19 @@ export class ObjectStackProtocolImplementation implements
         const registry: any = (this.engine as any)?.registry;
         if (!registry || typeof registry.registerItem !== 'function') return false;
         const artifact = this.lookupArtifactItem(type, (data as any).name, options.packageId ?? undefined);
-        registry.registerItem(type, mergeArtifactProtection(data, artifact), 'name' as any);
+        // [#16702] Say what this row IS before the artifact envelope is grafted
+        // on top of it. Every body reaching this hydrator came out of a
+        // `sys_metadata` write and is therefore tenant-authored by definition
+        // (ADR-0010 `_provenance: 'org'`) — the same sentence the `object`
+        // branches of this class already write, said once here for every OTHER
+        // type. It makes rows ALREADY poisoned at rest harmless without
+        // rewriting them, and it covers the column path the write-door strip
+        // cannot reach: `getMetaItems` re-stamps `_packageId` onto the body
+        // from the row's `package_id` COLUMN a few frames up, before handing it
+        // here. ⚠️ BEFORE the merge, never after — where a real artifact
+        // exists its envelope must still win (ADR-0010 §3.3), and it does,
+        // because {@link mergeArtifactProtection} overwrites `_provenance` last.
+        registry.registerItem(type, mergeArtifactProtection(stateTenantAuthorship(data), artifact), 'name' as any);
         this.hydrateExpandedViewItems(type, data, options, registry);
         return true;
     }
@@ -14899,6 +14989,21 @@ export class ObjectStackProtocolImplementation implements
         // Placed first so the destructive-change diff, the schema gate, the
         // authoring gate and the persisted body all see the same document.
         request.item = stripReadDecorations(request.item);
+        // [#16702] …and the three DERIVED protection keys, for the same reason
+        // one beat later. `metadata-read-decorations.ts` deliberately does NOT
+        // strip `_provenance` from a SERVED document (a served body must keep
+        // its provenance on re-parse), so the very same round-trip echoed
+        // `_packageId` / `_packageVersion` / `_provenance: 'package'` straight
+        // back into the tenant's own row — and for every non-`object` type the
+        // hydrated row's own bytes then decided it was a code artifact, closing
+        // the tenant out of their own item with `NOT_OVERRIDABLE` forever. The
+        // read side recomputes all three from the artifact on every read
+        // ({@link mergeArtifactProtection}), so nothing is lost by not storing
+        // them. See {@link DERIVED_PROVENANCE_KEYS} for why `_lock*` is NOT in
+        // the set. Placed alongside the decoration strip so the
+        // destructive-change diff, the schema gate, the authoring gate and the
+        // persisted body all still see one document.
+        request.item = stripDerivedProvenance(request.item);
         // [#6562] …and OUR OWN injected system columns, for the same reason and
         // at the same moment. `governServedItem` now serves the EFFECTIVE object
         // schema, so the very same Studio round-trip would otherwise persist

packages/objectql/src/plugin.integration.test.ts

@@ -1035,9 +1035,14 @@ describe('ObjectQLPlugin - Metadata Service Integration', () => {
 
       // Assert — items should be restored into the registry
       const registry = (kernel.getService('objectql') as any).registry;
+      // [#16702] `_provenance: 'org'` is the SERVER's own sentence about every
+      // `sys_metadata` row, stated by the shared hydrator — the same one the
+      // `object` branch has always made. Kept as an exact-shape assertion so an
+      // UNEXPECTED extra key on a restored app still reds this pin.
       expect(registry.getAllApps()).toContainEqual({
         name: 'custom_crm',
         label: 'Custom CRM',
+        _provenance: 'org',
       });
     });
 

packages/objectql/src/protocol-boot-hydration-scoped.test.ts

@@ -135,11 +135,18 @@ describe('loadMetaFromDb — ADR-0048 package-scoped protection graft at boot (#
         expect(direct._provenance).toBe('package');
     });
 
-    it('registers the row unchanged when artifacts have not loaded yet (boot-order no-op)', async () => {
+    it('grafts NO artifact envelope when artifacts have not loaded yet (boot-order no-op)', async () => {
         // Empty registry at hydration time — the scoped lookup finds
         // nothing, exactly like the unscoped one did, and the row
         // registers without a grafted envelope. Artifact-after-hydration
         // boot orders are unaffected by the scoping.
+        //
+        // [#16702] What the row does carry is the SERVER's own sentence about
+        // it — `_provenance: 'org'`, the same one the `object` branch has
+        // always stated — because every row this hydrator sees came out of a
+        // `sys_metadata` write. That is a statement of authorship, not a graft
+        // from an artifact: `_lock` and `_packageId` stay absent, which is what
+        // this case is about.
         const registry = new SchemaRegistry({ multiTenant: false });
         registry.logLevel = 'silent';
         const rows = [
@@ -159,7 +166,8 @@ describe('loadMetaFromDb — ADR-0048 package-scoped protection graft at boot (#
         expect(direct.label).toBe('B Home (customized)');
         expect(direct._lock).toBeUndefined();
         expect(direct._packageId).toBeUndefined();
-        expect(direct._provenance).toBeUndefined();
+        // [#16702] NOT `undefined` — the hydrator states tenant authorship.
+        expect(direct._provenance).toBe('org');
     });
 
     it('keeps the legacy best-effort graft for package-less (global) rows', async () => {