fix(service-datasource,runtime,metadata-protocol)!: a stored datasource row no longer displaces a code-defined datasource at boot, and the metadata door refuses edits to the host default

.changeset/21922-code-datasource-wins-at-restore.md

@@ -0,0 +1,29 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/metadata-protocol": minor
+"@objectstack/runtime": minor
+---
+
+fix(service-datasource,runtime,metadata-protocol)!: a stored datasource row no longer displaces a code-defined datasource at boot, and the metadata door refuses edits to the host's `default` (#21922, #21944)
+
+Clause-②: no (narrowing)
+
+A code-defined datasource (one the installed artifact declares in `*.datasource.ts`, or the host's own `default`) is read-only: `DatasourceSchema.origin` publishes it as "GitOps-owned, read-only in the UI", and the datasource-admin service states "code wins on collision". The boot restore broke both. It registered every stored `datasource` row in `sys_metadata` over whatever the runtime had registered from code, so after a restart a row left under a code-defined name was served by the admin door, editable there when it carried `origin: 'runtime'`, and handed to pool rehydration. A stored `default` row opened a second live pool named `default` on the row's own connection. The metadata door also still saved edits to `default`, the one code-defined datasource no package declares.
+
+The runtime now keeps one in-memory set of the datasource names it registers from code, on the kernel service `code-datasource-names`: `AppPlugin` adds the datasources the artifact declares and `DefaultDatasourcePlugin` adds `default`, both in `init()`, so the set is complete before any `start()` runs. The boot restore skips a stored row under a name in that set, and the metadata door's code-datasource check reads the same set.
+
+**BREAKING — what moves for consumers.**
+
+- After a restart over a stored row under a code-defined datasource's name, `GET /api/v1/datasources` serves the code definition (`origin: code`) instead of the row, and `PATCH /api/v1/datasources/:name` answers `400 DATASOURCE_ADMIN_ERROR` ("… is code-defined and cannot be edited at runtime.") where it answered 200 for a row that carried `origin: 'runtime'`.
+- No live pool is opened from such a row at boot.
+- `PUT /api/v1/meta/datasource/default` answered 200 and now answers `403 NOT_OVERRIDABLE`. `DELETE /api/v1/meta/datasource/default` with no stored row answered 200 and now answers the same `403`. The refusal's remedy names the host's database configuration (the database URL the server starts with), which is what defines `default`; every other code-defined datasource's refusal still names its `*.datasource.ts` source.
+- The skipped row is kept, and the boot logs one warning naming it.
+
+**Remedy.**
+
+- To change a code-defined datasource, change its code definition and redeploy: its `*.datasource.ts` source, or the host's database configuration for `default`.
+- A row the boot warning names is removable, and removing it is the repair: `DELETE /api/v1/meta/datasource/:name` answers 200 and deletes it.
+
+**Unchanged.** A runtime datasource with no code twin restores, saves and deletes through both doors as before. A host that composes neither `AppPlugin` nor `DefaultDatasourcePlugin` registers no set, and its stored rows restore as before. While a stored row exists under a code-defined name, `GET /api/v1/meta/datasource/:name` still serves that row (the metadata door reads its stored overlay first); after the `DELETE` above it serves the code definition, in the same boot.
+
+<!-- adr-0087: not-required (no-migration-prescription) a boot-restore verdict and a metadata-door verdict on datasources the host registers from code, which the published contract already calls read-only: no authorable key, spelling, export or stored shape moves, and no stored row is rewritten, converted or dropped. A row an earlier runtime write left under a code-defined name stays in sys_metadata and is removed by the operator with the metadata door's own DELETE; which stored edit an operator meant to keep is not something a ledger entry can rewrite. The other categories are closed on facts: every bumped package publishes (not unpublished); no ADR-0087 id covers this restore or this door (not already-registered); and the change is a verdict, not a declaration (not runtime-interface-only or type-surface-only). -->

packages/metadata-protocol/src/packaged-base-regime.ts

@@ -77,6 +77,11 @@
  * remedy. The two doors keep their own codes (`NOT_OVERRIDABLE` / 403 here,
  * `DATASOURCE_ADMIN_ERROR` / 400 there); the verdict and the remedy agree.
  *
+ * [#21944] One code-defined datasource has no source file: the host's
+ * `default`, defined by the database the server starts with. The row lists it
+ * under `hostOwned`, and its sentence names that configuration instead of a
+ * `*.datasource.ts` nobody can find. Every other name keeps the source remedy.
+ *
  * The row is also what {@link isOriginGatedType} answers from, for the one
  * removal both the protocol's delete door and the repository's delete gate
  * allow on such a type: deleting a STORED row under a code-defined name. The
@@ -127,6 +132,13 @@ export type PackagedBaseRegimeRow =
         readonly source: string;
         /** The decision record the sentence cites. */
         readonly docs: string;
+        /**
+         * [#21944] Names the HOST defines from its own configuration rather than
+         * from a {@link source} file, each with what defines it. For such a name
+         * the source-file remedy is false — no such file exists, and an artifact
+         * may not declare the name at all — so its sentence names this instead.
+         */
+        readonly hostOwned?: Readonly<Record<string, string>>;
     };
 
 /** The table. Keyed by the canonical (singular) metadata type. */
@@ -159,6 +171,14 @@ export const PACKAGED_BASE_REGIME: Readonly<Record<string, PackagedBaseRegimeRow
         noun: 'Datasource',
         source: '*.datasource.ts',
         docs: 'docs/adr/0062-external-datasource-runtime.md',
+        // [#21944] `default` is the host's primary datasource: the runtime's
+        // DefaultDatasourcePlugin builds it from the database the server is
+        // started with (a URL flag or config, OS_DATABASE_URL, a default-routing
+        // rule, or the unified default file — `resolve-project-database.ts`).
+        // The name is reserved for it: AppPlugin refuses an artifact that
+        // declares `default`, and the datasource-admin service refuses to create
+        // one. So no `*.datasource.ts` declares it, and its remedy says so.
+        hostOwned: { default: "the host's database configuration (the database URL the server starts with)" },
     },
 };
 
@@ -197,12 +217,19 @@ function regimeCPrescription(routes: PackagedBaseRegimeCRoutes): string {
  * origin-gated row's owning source — the only remedy such a type has — and the
  * row's citation.
  */
-function rowPrescription(row: PackagedBaseRegimeRow): string {
+function rowPrescription(row: PackagedBaseRegimeRow, name?: string): string {
     switch (row.regime) {
         case 'C':
             return regimeCPrescription(row.routes);
-        case 'origin-gated':
-            return `Edit the ${row.source} source that declares it and redeploy. See ${row.docs}.`;
+        case 'origin-gated': {
+            const host = name !== undefined && row.hostOwned !== undefined
+                && Object.prototype.hasOwnProperty.call(row.hostOwned, name)
+                ? row.hostOwned[name]
+                : undefined;
+            return host !== undefined
+                ? `It is defined by ${host}: change that configuration and restart the server. See ${row.docs}.`
+                : `Edit the ${row.source} source that declares it and redeploy. See ${row.docs}.`;
+        }
     }
 }
 
@@ -247,7 +274,8 @@ export function isOriginGatedType(type: string): boolean {
  * `flow` 411 / 404, `action` 365 / 358, `permission` 317 / 310, `datasource`
  * 192 / 193 — so a name of up to 88 characters arrives whole for every row
  * (pinned). A `flow`'s sentence is byte-identical to the one the row table
- * replaced (pinned literally).
+ * replaced (pinned literally). [#21944] A row's `hostOwned` name is a fixed,
+ * short name with its own remedy (`default`: under 300 characters whole).
  */
 export function packagedBaseRegimeSentence(
     type: string, name: string, operation: 'save' | 'delete',
@@ -260,5 +288,5 @@ export function packagedBaseRegimeSentence(
             + (operation === 'delete' ? 'removed' : 'edited') + ' at runtime: it is read-only. '
         : `Metadata item '${singular}/${name}' is provided by a code package, and its packaged base is locked `
             + (operation === 'delete' ? `against removal. ` : `against in-place edits. `);
-    return lock + rowPrescription(row);
+    return lock + rowPrescription(row, name);
 }

packages/metadata-protocol/src/protocol.code-defined-datasource-door.test.ts

@@ -109,6 +109,8 @@ function makeSession(opts: {
     environmentId?: string;
     packages?: Array<{ manifest: Record<string, unknown> }>;
     seed?: Row[];
+    /** [#21944] The kernel services the protocol resolves — the host's code-datasource set among them. */
+    services?: Map<string, unknown>;
 } = {}) {
     const rows = new Map<string, Row>();
     for (const r of opts.seed ?? []) rows.set(r.id, r);
@@ -175,7 +177,7 @@ function makeSession(opts: {
         },
     };
 
-    const protocol = new ObjectStackProtocolImplementation(engine, () => new Map(), opts.environmentId) as any;
+    const protocol = new ObjectStackProtocolImplementation(engine, () => opts.services ?? new Map(), opts.environmentId) as any;
     return { protocol, rows, historyRows };
 }
 
@@ -208,8 +210,9 @@ describe('[#21899] the resolver — a datasource an installed package declares i
         // The plural spelling folds before the read (#4432).
         expect(protocol.isArtifactBacked('datasources', CODE_DS)).toBe(true);
         expect(protocol.isArtifactBacked('datasource', RUNTIME_DS)).toBe(false);
-        // The host's `default` is declared by no package: the named gap, pinned
-        // as what this resolver answers so a change to it is seen.
+        // The host's `default` is declared by no package: with no host
+        // code-datasource set registered, the packages alone do not see it
+        // ([#21944] the set is what does — see the block at the foot of this file).
         expect(protocol.isArtifactBacked('datasource', 'default')).toBe(false);
     });
 
@@ -419,3 +422,111 @@ describe('[#21899] the repository delete gate lifts the origin-gated type only',
         expectVerdict(err.message, SAVE_VERDICT);
     });
 });
+
+/**
+ * [#21944] The host's `default` datasource, which no package declares. The
+ * runtime registers it from code (`DefaultDatasourcePlugin`) and adds it to the
+ * host's code-datasource set — the kernel service `'code-datasource-names'` —
+ * in Phase 1; the resolver reads that set beside the packages, so the door
+ * answers `default` as the admin door does: read-only.
+ */
+const CODE_NAMES_SERVICE = 'code-datasource-names';
+const DEFAULT_SAVE_VERDICT = "Datasource 'default' is code-defined and cannot be edited at runtime: it is read-only.";
+const DEFAULT_DELETE_VERDICT = "Datasource 'default' is code-defined and cannot be removed at runtime: it is read-only.";
+const hostServices = (names: string[]) => new Map<string, unknown>([[CODE_NAMES_SERVICE, new Set(names)]]);
+/**
+ * The remedy `default`'s refusal must carry: no `*.datasource.ts` declares
+ * `default` — the host defines it from the database the server starts with —
+ * so the sentence names that, and never the source-file remedy.
+ */
+const HOST_REMEDY = "It is defined by the host's database configuration";
+const expectHostRemedy = (message: unknown) => {
+    const text = String(message);
+    expect(text).toContain(HOST_REMEDY);
+    expect(text).toContain('restart the server');
+    expect(text).not.toContain('.datasource.ts');
+    expect(text).not.toContain('OS_METADATA_WRITABLE');
+};
+
+describe('[#21944] the resolver reads the host\'s code-datasource set', () => {
+    it('sees a name the host registers from code, and only the `datasource` type', () => {
+        const { protocol } = makeSession({ packages: [SHOWCASE_PACKAGE], services: hostServices(['default']) });
+        expect(protocol.isArtifactBacked('datasource', 'default')).toBe(true);
+        expect(protocol.isArtifactBacked('datasources', 'default')).toBe(true);
+        expect(protocol.isArtifactBacked('object', 'default')).toBe(false);
+        // The packages' answer is unchanged beside it.
+        expect(protocol.isArtifactBacked('datasource', CODE_DS)).toBe(true);
+        expect(protocol.isArtifactBacked('datasource', RUNTIME_DS)).toBe(false);
+    });
+
+    it('a service under that name with no `has` is not read as a set (nothing is guessed)', () => {
+        const { protocol } = makeSession({ services: new Map<string, unknown>([[CODE_NAMES_SERVICE, ['default']]]) });
+        expect(protocol.isArtifactBacked('datasource', 'default')).toBe(false);
+    });
+});
+
+for (const { label, environmentId } of KERNELS) {
+    describe(`[#21944] the /meta door on the host's \`default\` datasource — ${label}`, () => {
+        beforeEach(resetEnvHatch);
+        afterEach(resetEnvHatch);
+
+        const session = (seed?: Row[]) => makeSession({
+            ...(environmentId ? { environmentId } : {}),
+            packages: [SHOWCASE_PACKAGE],
+            services: hostServices(['default']),
+            ...(seed ? { seed } : {}),
+        });
+
+        it('PUT is refused NOT_OVERRIDABLE / 403 with the admin door\'s verdict, and stores nothing', async () => {
+            const { protocol, rows, historyRows } = session();
+
+            const err = await refusalOf(protocol.saveMetaItem({ type: 'datasource', name: 'default', item: body('default', 'Meta Renamed default') }));
+
+            expect({ code: err?.code, status: err?.status }).toEqual({ code: 'NOT_OVERRIDABLE', status: 403 });
+            expect(String(err?.message).startsWith(`${DEFAULT_SAVE_VERDICT} `), String(err?.message)).toBe(true);
+            expectHostRemedy(err?.message);
+            expect(rows.size).toBe(0);
+            expect(historyRows).toEqual([]);
+        });
+
+        it('DELETE with no stored row is refused the same way, naming the removal', async () => {
+            const { protocol, rows } = session();
+
+            const err = await refusalOf(protocol.deleteMetaItem({ type: 'datasource', name: 'default' }));
+
+            expect({ code: err?.code, status: err?.status }).toEqual({ code: 'NOT_OVERRIDABLE', status: 403 });
+            expect(String(err?.message).startsWith(`${DEFAULT_DELETE_VERDICT} `), String(err?.message)).toBe(true);
+            expectHostRemedy(err?.message);
+            expect(rows.size).toBe(0);
+        });
+
+        it('DELETE of a pre-existing stored row of `default` answers 200 and removes it (repair)', async () => {
+            const { protocol, rows } = session([shadowRow('default')]);
+
+            const res = await protocol.deleteMetaItem({ type: 'datasource', name: 'default' });
+
+            expect(res).toMatchObject({ success: true, reset: true });
+            expect(Array.from(rows.values()).filter((r) => r.name === 'default')).toEqual([]);
+        });
+
+        it('side by side: `default` names the host\'s configuration, a package-declared datasource still names its source', async () => {
+            const { protocol } = session();
+
+            const host = await refusalOf(protocol.saveMetaItem({ type: 'datasource', name: 'default', item: body('default', 'x') }));
+            const packaged = await refusalOf(protocol.saveMetaItem({ type: 'datasource', name: CODE_DS, item: body(CODE_DS, 'x') }));
+
+            expect({ code: host?.code, status: host?.status }).toEqual({ code: 'NOT_OVERRIDABLE', status: 403 });
+            expect({ code: packaged?.code, status: packaged?.status }).toEqual({ code: 'NOT_OVERRIDABLE', status: 403 });
+            expectHostRemedy(host?.message);
+            expectVerdict(packaged?.message, SAVE_VERDICT);
+            expect(String(packaged?.message)).not.toContain(HOST_REMEDY);
+        });
+
+        it('control: a runtime datasource still saves with the set registered', async () => {
+            const { protocol, rows } = session();
+            const saved = await protocol.saveMetaItem({ type: 'datasource', name: RUNTIME_DS, item: body(RUNTIME_DS, 'Runtime') });
+            expect(saved).toMatchObject({ success: true });
+            expect(Array.from(rows.values()).filter((r) => r.name === RUNTIME_DS)).toHaveLength(1);
+        });
+    });
+}

packages/metadata-protocol/src/protocol.ts

@@ -16263,28 +16263,35 @@ export class ObjectStackProtocolImplementation implements
      *    never filters package records, and a disabled package still ships the
      *    datasource the runtime registered for it — the artifact-only lookup
      *    answers for a disabled package's items the same way.
+     *  - [#21944] **The host's code-datasource set**, the kernel service
+     *    `'code-datasource-names'`: every datasource name the host registers
+     *    from code, filled by the runtime in Phase 1 (`AppPlugin` with what the
+     *    artifact declares, `DefaultDatasourcePlugin` with the host's own
+     *    `default`). It is how this predicate sees `default`, which no package
+     *    declares — `DefaultDatasourcePlugin` registers it from the host's
+     *    definition, and the datasource-admin service refuses to edit or remove
+     *    it as code-defined. The same set decides the datasource-admin plugin's
+     *    boot restore (#21922), so the two doors read one answer. Read per
+     *    call through the services registry, by name (its producer is
+     *    `@objectstack/runtime`'s `code-datasource-names.ts`, which this
+     *    package does not depend on); absent on a host that composes no
+     *    code-datasource producer, where nothing is added to the packages'
+     *    answer.
      *  - ⛔ **Never the MetadataService slot's `origin`**: a stored row the
-     *    datasource-admin plugin restores at boot overwrites that slot, `origin`
+     *    datasource-admin plugin restored at boot overwrote that slot, `origin`
      *    and all, so the slot answers what was stored last, not what a package
      *    ships. ⛔ **Never a request body's `origin`**: the caller sets it.
      *
      * ## What it does not see
      *
-     *  - **The host's `default` datasource.** No package declares it:
-     *    `DefaultDatasourcePlugin` registers it from the host's own definition,
-     *    in memory, as `origin: 'code'`, and the datasource-admin service
-     *    refuses to edit or remove it as code-defined. The host's code
-     *    datasource set is not readable from this package — the MetadataService
-     *    slot is the unsound source above, the connection service retains no
-     *    origin, and the engine's datasource definitions mix both origins — so
-     *    the `/meta` door still answers for `default` as it did. A named gap,
-     *    not a reading this predicate makes.
-     *  - **A name no package declares** answers false and keeps the
-     *    `runtime-only` intent: a runtime datasource stays creatable, editable
-     *    and removable through this door.
+     *  - **A name neither a package declares nor the host registers from
+     *    code** answers false and keeps the `runtime-only` intent: a runtime
+     *    datasource stays creatable, editable and removable through this door.
      */
     private isDeclaredCodeDatasource(type: string, name: string): boolean {
         if ((PLURAL_TO_SINGULAR[type] ?? type) !== 'datasource') return false;
+        const hostCode = this.getServicesRegistry?.().get('code-datasource-names') as { has?: unknown } | undefined;
+        if (typeof hostCode?.has === 'function' && (hostCode as { has(n: string): boolean }).has(name)) return true;
         const registry = (this.engine as any)?.registry;
         if (typeof registry?.getAllPackages !== 'function') return false;
         for (const record of registry.getAllPackages() as unknown[]) {