diff --git a/.changeset/18386-export-import-template.md b/.changeset/18386-export-import-template.md new file mode 100644 index 00000000000..542d0899944 --- /dev/null +++ b/.changeset/18386-export-import-template.md @@ -0,0 +1,37 @@ +--- +'@objectstack/rest': minor +--- + +feat(rest): `GET /api/v1/data/:object/export?template=true` answers an xlsx import template for the object (#18386) + +Clause-②: yes (widening) + +The export door takes one more query parameter, `template`. `template=true` +answers an `.xlsx` workbook with no data rows; `template=false` answers the export. +Without a `template` parameter the export is exactly as before, byte for byte. + +- **Columns.** Every field of the object except + those marked `system` or `readonly`, and `formula`, `summary` and + `autonumber` fields, in the order the object declares them. A `hidden` field + that can be written is a column. The seven columns the platform adds to every + object (`organization_id`, `created_at`, `created_by`, `updated_at`, + `updated_by`, `owner_id`, `owning_business_unit_id`) are never template + columns. A field the caller's field-level security does not let them edit is + left out. If the security service cannot say which fields the caller can edit, + the columns are narrowed by the fields the caller can read instead. The + instructions sheet then says so, and the `X-Export-Template-Projection` + response header reads `readable` instead of `writable` (`none` when no + field-level security applies). An explicit `?fields=` list is used as sent. +- **First sheet.** The header row, with ` *` after each field that is required + and has no default value, and one example row to replace or delete. Select, + radio and boolean columns carry a dropdown. +- **Second sheet.** One row per column: the field's API name, its type, whether + it is required, and the values the import accepts for it. +- **Language.** The sheets are in Chinese for a `zh` request locale + (`?locale=` or `Accept-Language`) and in English otherwise. + +The same two permission checks as the export apply: an object that does not +expose export answers `405`, and a caller without the export permission answers +`403`. `template` with a value other than `true` or `false`, a `format` other +than `xlsx`, or any of `limit`, `page`, `filter`, `search`, `searchFields`, +`orderby` or `header` beside `template=true`, answers `400 VALIDATION_ERROR`. diff --git a/.changeset/18386-plugin-security-writable-fields.md b/.changeset/18386-plugin-security-writable-fields.md new file mode 100644 index 00000000000..14df141e00d --- /dev/null +++ b/.changeset/18386-plugin-security-writable-fields.md @@ -0,0 +1,5 @@ +--- +'@objectstack/plugin-security': minor +--- + +The `security` service implements `getWritableFields(object, context)` (#18386). It uses the same permission sets, field grants, `requiredPermissions` check and on-behalf-of delegator intersection as the write gate. A field is in the answer exactly when a write naming it passes the field-level-security check. `getReadableFields` now shares that derivation, and its answers are unchanged. diff --git a/.changeset/18386-security-service-writable-fields.md b/.changeset/18386-security-service-writable-fields.md new file mode 100644 index 00000000000..9e2e5b23c48 --- /dev/null +++ b/.changeset/18386-security-service-writable-fields.md @@ -0,0 +1,11 @@ +--- +'@objectstack/spec': minor +--- + +`ISecurityService` (`@objectstack/spec/contracts`) gains an optional `getWritableFields(object, context)`: the field names field-level security lets the caller write on the object, the write-side twin of `getReadableFields` (#18386). + +Clause-②: yes (widening) + +- It is the exact complement of the fields the write path's field-level-security gate refuses when a payload names them. Neither the object permission nor a field's own rules (`readonly`, `system`, a `formula`, `summary` or `autonumber` type) are part of the answer. +- It fails soft like `getReadableFields`: `undefined` means no answer, `[]` means no field is writable. A system context gets every field. +- It is optional. A consumer checks `typeof svc.getWritableFields === 'function'`. When the method is missing, the consumer may narrow by `getReadableFields` instead, and must say in its response that it did. diff --git a/content/docs/permissions/system-context.mdx b/content/docs/permissions/system-context.mdx index e6d07ace6d7..a12cc03a629 100644 --- a/content/docs/permissions/system-context.mdx +++ b/content/docs/permissions/system-context.mdx @@ -99,7 +99,7 @@ that silently does not happen. | 1 | **The whole security middleware short-circuits** before any gate runs | plugin-security | Get: every CRUD/FLS/tenant/owner gate below skipped in one branch. Lose: every gate under it in this lane at once, down to and including the write-bypass row that widens the effective write scope to `org` (row 7) — this is the single largest behaviour on the page | `packages/plugins/plugin-security/src/security-plugin.ts#start` | | 2 | **`owner_id` is not auto-stamped on INSERT** (the step 3.5 anchor guard is inside the block row 1 skips) | plugin-security | Lose: the row lands `owner_id = NULL`, so the default `owner_only_writes` policy hides it **from its own creator**. Get: nothing — this is a gap, not a capability | the step 3.5 guard block and the short-circuit that skips it are both inside `packages/plugins/plugin-security/src/security-plugin.ts#start` | | 3 | Row-level read filter resolves to "no filter" | plugin-security | Get: unscoped reads. Lose: row-level scoping entirely | `packages/plugins/plugin-security/src/security-plugin.ts#getReadFilter` | -| 4 | Field-level security returns **all** fields | plugin-security | Get: every column readable. Lose: field masking | `packages/plugins/plugin-security/src/security-plugin.ts#computeReadableFields` | +| 4 | Field-level security returns **all** fields | plugin-security | Get: every column readable (`getReadableFields`) and writable (`getWritableFields`). Lose: field masking | `packages/plugins/plugin-security/src/security-plugin.ts#resolveProjectionFieldMask` | | 5 | Export permission granted unconditionally | plugin-security | Get: `canExport` is `true` | `packages/plugins/plugin-security/src/security-plugin.ts#canExport` | | 6 | Object-level read admission granted unconditionally | plugin-security | Get: `canReadObject` is `true`. This is the OBJECT-level half of a read — "may this caller read this object at all" — which the doors that bypass this middleware ask before they compile a statement of their own; `getReadFilter` is its row-level half, and the two are not interchangeable | `packages/plugins/plugin-security/src/security-plugin.ts#canReadObject` | | 6b | WRITE admission granted unconditionally | plugin-security | Get: `canWriteObject` is `true`. The WRITE twin of the object-level read admission above, asked by the write PREVIEW (`ObjectQL.validate()`), which runs no middleware for its target object and so has no gate of its own. It exists because a validation rule that reads one hop through a reference field is evaluated against a related row fetched under system authority, and that elevation is bounded to the arms the question RUNS — named, ⛔ never a category of the write decision, and ⛔ not a promise the write would succeed. The arms: the ADR-0103 engine-owned affordance gate and the ADR-0090 D12 delegated-admin gate (both ahead of every resolution), the fail-closed postures (#3545's unresolvable posture and the D10 dangling delegator), the ADR-0066 D3 capability arm, the CRUD grant, the ADR-0090 D10 delegator arm that a bare CRUD-grant check misses, the field-level-security write gate over the caller's payload when one is supplied, and the ADR-0123 D2 no-active-organization wall | `packages/plugins/plugin-security/src/security-plugin.ts#canWriteObject` | diff --git a/packages/plugins/plugin-security/src/get-writable-fields.test.ts b/packages/plugins/plugin-security/src/get-writable-fields.test.ts new file mode 100644 index 00000000000..51ef96a6cab --- /dev/null +++ b/packages/plugins/plugin-security/src/get-writable-fields.test.ts @@ -0,0 +1,182 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#18386] `getWritableFields` — the write-side twin of `getReadableFields`. + * + * The first block is an EQUIVALENCE, not a table of expected lists: for every + * field of the object it drives the REAL registered middleware with a write + * whose payload names only that field, and requires "the middleware admitted + * it" to equal "the field is in `getWritableFields`". The two answers come from + * one derivation; this is what keeps them one. + * + * Harness mirrors `can-write-object-admission.test.ts`. + */ + +import { describe, it, expect, vi } from 'vitest'; +import type { PermissionSet } from '@objectstack/spec/security'; +import { SecurityPlugin } from './security-plugin.js'; + +const CRUD = { allowRead: true, allowCreate: true, allowEdit: true }; + +/** The baseline every authenticated caller resolves: reads `account`, may not edit it; `secret` neither. */ +const LOCKED_SET = { + name: 'member_default', + label: 'Writer, FLS-locked', + objects: { invoice: CRUD }, + fields: { + 'invoice.account': { readable: true, editable: false }, + 'invoice.secret': { readable: false, editable: false }, + }, +} as unknown as PermissionSet; + +/** The same grant with no field rules — every field passes step 2.5. */ +const OPEN_SET = { name: 'member_default', label: 'Writer', objects: { invoice: CRUD } } as unknown as PermissionSet; + +/** …and with the capability `margin` requires (ADR-0066 D3). */ +const CAPABLE_SET = { + name: 'member_default', + label: 'Writer with the margin capability', + objects: { invoice: CRUD }, + systemPermissions: ['view_margin'], +} as unknown as PermissionSet; + +/** An agent's own set, which may edit `account`: the D10 case turns on the intersection alone. */ +const AGENT_SET = { + name: 'agent_writer', + label: 'Agent', + objects: { invoice: CRUD }, + fields: { 'invoice.account': { readable: true, editable: true } }, +} as unknown as PermissionSet; + +const SCHEMAS: Record = { + invoice: { + name: 'invoice', + fields: { + title: { type: 'text', label: 'Title' }, + account: { type: 'lookup', label: 'Account', reference: 'crm_account' }, + secret: { type: 'text', label: 'Secret' }, + margin: { type: 'number', label: 'Margin', requiredPermissions: ['view_margin'] }, + }, + }, +}; +/** The field universe the plugin resolves: the schema's fields plus `id`. */ +const FIELDS = ['id', 'title', 'account', 'secret', 'margin']; +const PAYLOAD_VALUE: Record = { id: 'inv_1', title: 'x', account: 'acc_1', secret: 's', margin: 1 }; + +const WRITER_CTX = { userId: 'u_writer', tenantId: 'org-1', positions: [], permissions: [], posture: 'MEMBER' }; +const LIVE_DELEGATOR = 'u_boss'; +const AGENT_CTX = { userId: 'u_agent', tenantId: 'org-1', positions: [], permissions: ['agent_writer'], posture: 'MEMBER' }; +const DELEGATED_AGENT_CTX = { ...AGENT_CTX, onBehalfOf: { userId: LIVE_DELEGATOR } }; + +async function boot(sets: PermissionSet[], opts: { noBaseline?: boolean } = {}) { + const middlewares: Array<(opCtx: any, next: () => Promise) => Promise> = []; + const services: Record = { + manifest: { register: vi.fn() }, + objectql: { + registerMiddleware: (mw: any) => middlewares.push(mw), + getSchema: (name: string) => SCHEMAS[name], + findOne: vi.fn(async (_object: string, query: any) => + (query?.where?.id === LIVE_DELEGATOR ? { id: LIVE_DELEGATOR, email: 'boss@example.test' } : null)), + }, + metadata: { + get: async (_type: string, name: string) => SCHEMAS[name], + list: async () => sets, + }, + }; + const registerService = vi.fn(); + const ctx: Record = { + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }, + registerService, + getService: (name: string) => { + if (!(name in services)) throw new Error(`service not registered: ${name}`); + return services[name]; + }, + }; + const plugin = new SecurityPlugin( + opts.noBaseline + ? { defaultPermissionSets: [], fallbackPermissionSet: null } + : { fallbackPermissionSet: 'member_default' }, + ); + await plugin.init(ctx as any); + await plugin.start(ctx as any); + if (middlewares.length === 0) throw new Error('SecurityPlugin registered no middleware'); + return { plugin, middleware: middlewares[0], registerService }; +} + +async function middlewareAdmits( + middleware: (opCtx: any, next: () => Promise) => Promise, + operation: 'insert' | 'update', + context: Record, + data: Record, +): Promise { + try { + await middleware( + { object: 'invoice', operation, context: { ...context }, options: {}, ast: { where: {} }, data }, + async () => {}, + ); + return true; + } catch { + return false; + } +} + +describe('getWritableFields agrees with the middleware\'s write gate, field for field', () => { + const CASES: Array<{ label: string; sets: PermissionSet[]; context: Record; writable: string[] }> = [ + { label: 'a field read but not editable, and one neither', sets: [LOCKED_SET], context: WRITER_CTX, writable: ['id', 'title'] }, + { label: 'no field rules', sets: [OPEN_SET], context: WRITER_CTX, writable: ['id', 'title', 'account', 'secret'] }, + { label: 'the field capability held', sets: [CAPABLE_SET], context: WRITER_CTX, writable: FIELDS }, + { label: 'a delegated agent whose delegator may not edit the field', sets: [AGENT_SET, LOCKED_SET], context: DELEGATED_AGENT_CTX, writable: ['id', 'title'] }, + { label: 'the same agent acting for nobody', sets: [AGENT_SET, LOCKED_SET], context: AGENT_CTX, writable: ['id', 'title', 'account'] }, + ]; + + for (const c of CASES) { + it(c.label, async () => { + const { plugin, middleware } = await boot(c.sets); + const writable = await plugin.getWritableFields('invoice', c.context); + // The expected list keeps each case honest about what it exercises; the + // equivalence below is the pin. + expect(writable).toEqual(c.writable); + for (const operation of ['insert', 'update'] as const) { + for (const field of FIELDS) { + const admitted = await middlewareAdmits(middleware, operation, c.context, { [field]: PAYLOAD_VALUE[field] }); + expect(admitted, `${operation} naming ${field}`).toBe(writable!.includes(field)); + } + } + }); + } +}); + +describe('getWritableFields — the answers the contract names', () => { + it('a field the caller may read but not edit is readable and NOT writable', async () => { + const { plugin } = await boot([LOCKED_SET]); + expect(await plugin.getReadableFields('invoice', WRITER_CTX)).toContain('account'); + expect(await plugin.getWritableFields('invoice', WRITER_CTX)).not.toContain('account'); + }); + + it('a system context bypasses: the full field set', async () => { + const { plugin } = await boot([LOCKED_SET]); + expect(await plugin.getWritableFields('invoice', { isSystem: true })).toEqual(FIELDS); + }); + + it('no permission sets resolved: the full field set, as the middleware skips its write gate', async () => { + const { plugin } = await boot([], { noBaseline: true }); + expect(await plugin.getWritableFields('invoice', WRITER_CTX)).toEqual(FIELDS); + }); + + it('an unresolvable object is no answer (undefined), not an empty one', async () => { + const { plugin } = await boot([LOCKED_SET]); + expect(await plugin.getWritableFields('no_such_object', WRITER_CTX)).toBeUndefined(); + }); + + it('a delegator that does not exist fails closed: []', async () => { + const { plugin } = await boot([AGENT_SET, LOCKED_SET]); + expect(await plugin.getWritableFields('invoice', { ...AGENT_CTX, onBehalfOf: { userId: 'u_ghost' } })).toEqual([]); + }); + + it('is exposed on the registered "security" service', async () => { + const { registerService } = await boot([LOCKED_SET]); + const svc = registerService.mock.calls.find((c: any[]) => c[0] === 'security')?.[1]; + expect(typeof svc?.getWritableFields).toBe('function'); + expect(await svc.getWritableFields('invoice', WRITER_CTX)).toEqual(['id', 'title']); + }); +}); diff --git a/packages/plugins/plugin-security/src/security-plugin.ts b/packages/plugins/plugin-security/src/security-plugin.ts index 5313307af7c..273106ce797 100644 --- a/packages/plugins/plugin-security/src/security-plugin.ts +++ b/packages/plugins/plugin-security/src/security-plugin.ts @@ -1703,6 +1703,9 @@ export class SecurityPlugin implements Plugin { // route uses it to project columns instead of inferring readability // from already-masked data rows. See getReadableFields. getReadableFields: (object: string, context?: any) => this.getReadableFields(object, context), + // [#18386] Its write-side twin: the fields step 2.5 would not refuse. + // The import template narrows its columns by it. See getWritableFields. + getWritableFields: (object: string, context?: any) => this.getWritableFields(object, context), // [#3544] User-level export axis. `export ⊆ list`, so a bulk export // reaches the middleware as a plain `find` and `allowExport` would never // be consulted — the REST export route asks HERE before it streams. @@ -1900,23 +1903,22 @@ export class SecurityPlugin implements Plugin { }; // [ADR-0106 D7] The metadata-plane readable-field query, registered as an // EXTENSION of the published contract rather than inside the typed - // literal above: `ISecurityService` lives in `packages/spec`, and the - // seat for this method there is a separate change (consumers already - // feature-detect, which is exactly why a partial surface degrades instead - // of lying). `Object.assign` keeps the literal type-checked against the - // contract while the extension stays visible as an extension. + // literal above (consumers already feature-detect, which is exactly why a + // partial surface degrades instead of lying). `Object.assign` keeps the + // literal type-checked against the contract while the extension stays + // visible as an extension. const registeredSecurityService = Object.assign(securityService, { getMetadataReadableFields: (object: string, context?: any) => this.getMetadataReadableFields(object, context), // [field report — rc→GA declared≠enforced surfacing] Same extension - // pattern as `getMetadataReadableFields` above, same reason: + // pattern as `getMetadataReadableFields` above: // `ISecurityService` lives in `packages/spec` and this seat there is a // separate change. Consumers feature-detect. discardPermissionSetOverlay: (callerContext: any, id: string) => discardPermissionSetOverlay(overlayDiscardDeps, callerContext, id), }); ctx.registerService('security', registeredSecurityService); - ctx.logger.info('[security] registered "security" service (getReadFilter, canReadObject, getReadableFields, getMetadataReadableFields, canExport, checkAuthoredRowWrite, resolvePermissionSetNames, resolvePermissionSetsForContext, explain, audience-binding suggestions, discardPermissionSetOverlay) — ADR-0021 D-C / ADR-0090 D5/D6/D9 / ADR-0094 / ADR-0106 D7 / #3544 / #3547 / #5493 / #7616'); + ctx.logger.info('[security] registered "security" service (getReadFilter, canReadObject, getReadableFields, getWritableFields, getMetadataReadableFields, canExport, checkAuthoredRowWrite, resolvePermissionSetNames, resolvePermissionSetsForContext, explain, audience-binding suggestions, discardPermissionSetOverlay) — ADR-0021 D-C / ADR-0090 D5/D6/D9 / ADR-0094 / ADR-0106 D7 / #3544 / #3547 / #5493 / #7616'); } catch (e) { ctx.logger.warn?.('[security] failed to register "security" service', { error: (e as Error).message, @@ -5446,16 +5448,74 @@ export class SecurityPlugin implements Plugin { context: any, options: { fallbackOnEmptySets: boolean }, ): Promise { + const mask = await this.resolveProjectionFieldMask(object, context, options); + if (mask.kind === 'answer') return mask.fields; + + // [#8993] A field this caller sees PARTIALLY MASKED is still a served + // column — the read path REPLACES its value rather than deleting the key — + // so it stays in the projection (an export header must include the column + // whose masked values the same caller's rows carry). Mirrors the step-4 + // exclusion exactly: an explicit permission-set deny keeps the field + // deleted, hence out of the projection. + const partialRules = mask.readPartialMaskRules(); + + // Readable = every schema field NOT explicitly masked non-readable. A field + // with no permission entry passes through (the field allow-list only + // enumerates fields it names) — the exact complement of maskResults' delete + // set, so the export header matches list's readable columns by construction. + return mask.allFields.filter((f) => mask.fieldPerms[f]?.readable !== false || partialRules[f] !== undefined); + } + + /** + * [#18386] Query surface: the field names field-level security lets the + * caller WRITE on `object` under `context` (a field's own rules, such as + * `readonly`, are not asked) — the write-side twin of {@link getReadableFields}. + * + * Same derivation as the read projection ({@link resolveProjectionFieldMask}), + * which is the one the middleware's step 2.5 write gate takes, and the answer + * is the complement of that gate's own primitive, + * `FieldMasker.getNonEditableFields` — so a field is here iff a payload + * naming it passes step 2.5. A caller with no permission sets gets the full + * set: the middleware skips step 2.5 for it. + */ + async getWritableFields(object: string, context?: any): Promise { + const mask = await this.resolveProjectionFieldMask(object, context, { fallbackOnEmptySets: false }); + if (mask.kind === 'answer') return mask.fields; + const nonEditable = new Set(this.fieldMasker.getNonEditableFields(mask.fieldPerms)); + return mask.allFields.filter((f) => !nonEditable.has(f)); + } + + /** + * The derivation both field projections share: the schema's field universe, + * the caller's permission sets, the evaluator's field map with the ADR-0066 + * D3 `requiredPermissions` fold, and the ADR-0090 D10 delegator intersection + * — the steps, in order, that the middleware's read mask and its step 2.5 + * write gate each take. A case that settles the answer before any mask + * exists comes back as `answer`. + */ + private async resolveProjectionFieldMask( + object: string, + context: any, + options: { fallbackOnEmptySets: boolean }, + ): Promise< + | { kind: 'answer'; fields: string[] | undefined } + | { + kind: 'mask'; + allFields: string[]; + fieldPerms: Record; + readPartialMaskRules: () => Record; + } + > { const objectName = String(object ?? ''); - if (!objectName) return undefined; + if (!objectName) return { kind: 'answer', fields: undefined }; // The field universe — the SAME source the RLS field pass uses (ObjectQL's // live SchemaRegistry first, metadata artifact fallback). `null` → schema // not resolvable → let the caller fall back rather than guess. const fieldNameSet = await this.getObjectFieldNames(this.metadata, objectName, this.ql); - if (!fieldNameSet) return undefined; + if (!fieldNameSet) return { kind: 'answer', fields: undefined }; const allFields = [...fieldNameSet]; // System operations bypass FLS (mirrors the middleware's isSystem skip). - if (context?.isSystem) return allFields; + if (context?.isSystem) return { kind: 'answer', fields: allFields }; let permissionSets = await this.resolvePermissionSetsForContext(context); if (permissionSets.length === 0 && options.fallbackOnEmptySets) { @@ -5467,26 +5527,26 @@ export class SecurityPlugin implements Plugin { } // No sets resolved (e.g. unauthenticated) → no field mask applies, exactly // as the middleware (getFieldPermissions([]) === {} → nothing deleted). - if (permissionSets.length === 0) return allFields; + if (permissionSets.length === 0) return { kind: 'answer', fields: allFields }; const secMeta = await this.getObjectSecurityMeta(objectName); // [#3545] Posture unresolvable → expose no columns, the same fail-closed // stance this method already takes on a dangling delegator below. The // per-field capability contract (`fieldRequiredPermissions`) would otherwise // default to empty and silently unmask every capability-gated column. - if (secMeta.unresolved) return []; + if (secMeta.unresolved) return { kind: 'answer', fields: [] }; const basePerms = this.permissionEvaluator.getFieldPermissions(objectName, permissionSets); let fieldPerms = this.foldFieldRequiredPermissions(basePerms, secMeta.fieldRequiredPermissions, permissionSets); - // [ADR-0090 D10] On an on-behalf-of read the readable set must NOT widen - // past what the DELEGATOR can read — intersect the delegator's field mask - // too. A dangling delegator fails CLOSED (expose no columns), the same - // fail-closed stance the CRUD middleware takes on a 'missing' delegator. + // [ADR-0090 D10] On an on-behalf-of request the projection must NOT widen + // past the DELEGATOR's — intersect the delegator's field mask too. A + // dangling delegator fails CLOSED (expose no columns), the same fail-closed + // stance the CRUD middleware takes on a 'missing' delegator. let delBasePerms: Record | null = null; let delegatorSets: PermissionSet[] | null = null; if (context?.onBehalfOf?.userId) { const del = await resolveDelegatorContext(this.ql, context); - if (del.kind === 'missing') return []; + if (del.kind === 'missing') return { kind: 'answer', fields: [] }; if (del.kind === 'resolved') { delegatorSets = await this.resolvePermissionSetsForContext(del.context); delBasePerms = this.permissionEvaluator.getFieldPermissions(objectName, delegatorSets); @@ -5495,21 +5555,14 @@ export class SecurityPlugin implements Plugin { } } - // [#8993] A field this caller sees PARTIALLY MASKED is still a served - // column — the read path REPLACES its value rather than deleting the key — - // so it stays in the projection (an export header must include the column - // whose masked values the same caller's rows carry). Mirrors the step-4 - // exclusion exactly: an explicit permission-set deny keeps the field - // deleted, hence out of the projection. - const partialRules = this.computeReadPartialMaskRules( - secMeta, permissionSets, delegatorSets, basePerms, delBasePerms, - ); - - // Readable = every schema field NOT explicitly masked non-readable. A field - // with no permission entry passes through (the field allow-list only - // enumerates fields it names) — the exact complement of maskResults' delete - // set, so the export header matches list's readable columns by construction. - return allFields.filter((f) => fieldPerms[f]?.readable !== false || partialRules[f] !== undefined); + return { + kind: 'mask', + allFields, + fieldPerms, + readPartialMaskRules: () => this.computeReadPartialMaskRules( + secMeta, permissionSets, delegatorSets, basePerms, delBasePerms, + ), + }; } /** diff --git a/packages/rest/src/import-template-route.test.ts b/packages/rest/src/import-template-route.test.ts new file mode 100644 index 00000000000..b38c24ef9d7 --- /dev/null +++ b/packages/rest/src/import-template-route.test.ts @@ -0,0 +1,504 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `GET /data/:object/export?template=true` through the REAL route, driven by a + * REAL {@link ObjectQL} engine + {@link ObjectStackProtocolImplementation} on a + * better-sqlite3 `:memory:` driver — the stack `export-integration.test.ts` + * boots. Objects here keep the registry's system fields ON, so the columns the + * registry injects onto every object are really there to be excluded. + * + * Also here: the proof that WITHOUT `?template=true` the export is byte for + * byte what it was before the mode existed — see the last describe block. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { createHash } from 'node:crypto'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { RestServer } from './rest-server'; +import { parseXlsxToRows } from './import-prepare.js'; +import { loadXlsxWorkbook } from './xlsx-test-loader.js'; + +function makeSqliteDriver() { + return new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }); +} + +const liveEngines: ObjectQL[] = []; +afterEach(async () => { + vi.useRealTimers(); + while (liveEngines.length) { + try { await liveEngines.pop()?.destroy(); } catch { /* noop */ } + } +}); + +function createMockServer() { + const noop = () => {}; + return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; +} + +/** Collects what the handler writes: status, headers, JSON body, binary chunks. */ +function makeRes() { + const chunks: Buffer[] = []; + const headers: Record = {}; + let status = 200; + let json: any; + const res: any = { + write: (c: any) => { chunks.push(Buffer.isBuffer(c) ? c : Buffer.from(String(c))); return true; }, + end: () => {}, + header: (n: string, v: string) => { headers[n] = v; return res; }, + status: (s: number) => { status = s; return res; }, + json: (b: any) => { json = b; return res; }, + }; + return { res, headers, body: () => Buffer.concat(chunks), status: () => status, json: () => json }; +} + +// --------------------------------------------------------------------------- +// Objects — registry system fields ON (the default), one field per rule. +// --------------------------------------------------------------------------- + +const ACCOUNT = { + name: 'account', label: '客户', + fields: { name: { name: 'name', type: 'text', label: 'Account name' } }, +}; + +const LINE = { + name: 'line', label: 'Line', + fields: { + name: { name: 'name', type: 'text' }, + deal: { name: 'deal', type: 'master_detail', reference: 'deal' }, + qty: { name: 'qty', type: 'number' }, + }, +}; + +const DEAL = { + name: 'deal', label: 'Deal', + fields: { + title: { name: 'title', type: 'text', label: 'Title', required: true }, + approval: { + name: 'approval', type: 'select', label: 'Approval', readonly: true, defaultValue: 'draft', + options: [{ label: 'Draft', value: 'draft' }, { label: 'Approved', value: 'approved' }], + }, + stage: { + name: 'stage', type: 'select', label: 'Stage', + options: [{ label: 'Open', value: 'open' }, { label: 'Won', value: 'won' }], + }, + secret_note: { name: 'secret_note', type: 'text', label: 'Hidden note', hidden: true }, + classified: { name: 'classified', type: 'text', label: 'System flagged', system: true }, + amount: { name: 'amount', type: 'currency', label: 'Amount' }, + doubled: { name: 'doubled', type: 'formula', label: 'Doubled', expression: 'amount * 2' }, + total_qty: { name: 'total_qty', type: 'summary', label: 'Total qty', summaryOperations: { object: 'line', field: 'qty', function: 'sum' } }, + code: { name: 'code', type: 'autonumber', label: 'Code', autonumberFormat: 'D-{0000}' }, + hot: { name: 'hot', type: 'boolean', label: 'Hot' }, + tags: { + name: 'tags', type: 'multiselect', label: 'Tags', + options: [{ label: 'VIP', value: 'vip' }, { label: 'New', value: 'new' }], + }, + close_date: { name: 'close_date', type: 'date', label: 'Close date' }, + account: { name: 'account', type: 'lookup', label: 'Account', reference: 'account' }, + salary: { name: 'salary', type: 'number', label: 'Salary' }, + }, +}; + +/** The seven columns the registry injects onto every object (the card's table). */ +const INJECTED = ['organization_id', 'created_at', 'created_by', 'updated_at', 'updated_by', 'owner_id', 'owning_business_unit_id']; + +interface BootOptions { + security?: Record; +} + +async function boot(opts: BootOptions = {}) { + const engine = new ObjectQL(); + liveEngines.push(engine); + engine.registerDriver(makeSqliteDriver(), true); + await engine.init(); + for (const o of [ACCOUNT, LINE, DEAL]) engine.registry.registerObject(o as any); + await engine.syncSchemas(); + await engine.insert('account', { id: 'a1', name: 'Acme' }); + const protocol = new ObjectStackProtocolImplementation(engine as any); + const findData = vi.spyOn(protocol as any, 'findData'); + const rest = new RestServer(createMockServer() as any, protocol as any, { api: { requireAuth: false } } as any); + (rest as any).resolveExecCtx = async () => ({ userId: 'test-user', timezone: 'UTC' }); + if (opts.security) (rest as any).resolveSecurityService = async () => opts.security; + rest.registerRoutes(); + const route = (method: string, path: string) => rest.getRoutes().find((r: any) => r.method === method && r.path === path)!; + const exportRoute = route('GET', '/api/v1/data/:object/export'); + const importRoute = route('POST', '/api/v1/data/:object/import'); + const get = async (query: Record, object = 'deal', headers: Record = {}) => { + const out = makeRes(); + await exportRoute.handler({ params: { object }, query, headers } as any, out.res); + return out; + }; + return { engine, protocol, findData, get, importRoute }; +} + +const headerRow = async (bytes: Buffer, sheet = 0) => { + const wb = await loadXlsxWorkbook(bytes); + return { wb, header: (wb.worksheets[sheet].getRow(1).values as unknown[]).slice(1) as string[] }; +}; + +// --------------------------------------------------------------------------- + +describe('?template=true — the import template, through the real stack', () => { + it('answers an xlsx template without reading a single row', async () => { + const { get, findData } = await boot(); + const out = await get({ template: 'true' }); + expect(out.status()).toBe(200); + expect(out.headers['Content-Type']).toBe('application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'); + expect(out.headers['X-Export-Template']).toBe('true'); + // No security service composed: no field-level projection applies. + expect(out.headers['X-Export-Template-Projection']).toBe('none'); + expect(out.headers['Content-Disposition']).toMatch(/^attachment; filename="deal-template-\d{8}-\d{6}\.xlsx"/); + expect(findData).not.toHaveBeenCalled(); + const { wb } = await headerRow(out.body()); + expect(wb.worksheets[0].actualRowCount).toBe(2); // header + example, zero data rows + }); + + it('none of the registry-injected columns is a template column — on an object that has all seven', async () => { + const { get, protocol } = await boot(); + const schema: any = (await (protocol as any).getMetaItem({ type: 'object', name: 'deal' })).item; + for (const f of INJECTED) expect(Object.keys(schema.fields), `${f} is injected`).toContain(f); + const { header } = await headerRow((await get({ template: 'true' })).body()); + const wb = await loadXlsxWorkbook((await get({ template: 'true' })).body()); + const fieldColumn = (wb.worksheets[1].getColumn(2).values as unknown[]).slice(5).filter(Boolean); + for (const f of INJECTED) expect(fieldColumn, f).not.toContain(f); + expect(header).not.toContain('Owner'); + }); + + it('keeps exactly the writable declared fields, in declaration order, the required one marked', async () => { + const { get } = await boot(); + const out = await get({ template: 'true' }); + const { wb, header } = await headerRow(out.body()); + // `secret_note` is `hidden: true` and writable, so it is a column. + expect(header).toEqual(['Title *', 'Stage', 'Hidden note', 'Amount', 'Hot', 'Tags', 'Close date', 'Account', 'Salary']); + const fields = (wb.worksheets[1].getColumn(2).values as unknown[]).slice(5, 5 + header.length); + expect(fields).toEqual(['title', 'stage', 'secret_note', 'amount', 'hot', 'tags', 'close_date', 'account', 'salary']); + }); + + it('the lookup column names its target by the target object\'s label', async () => { + const { get } = await boot(); + const wb = await loadXlsxWorkbook((await get({ template: 'true' })).body()); + const howToFill = String(wb.worksheets[1].getCell('E12').value); + expect(wb.worksheets[1].getCell('B12').value).toBe('account'); + expect(howToFill).toContain('客户'); + }); + + it('?fields= is honoured as asked', async () => { + const { get } = await boot(); + const { header } = await headerRow((await get({ template: 'true', fields: 'code,title' })).body()); + expect(header).toEqual(['Code', 'Title *']); + }); + + it('?locale=zh-CN answers the Chinese sheets', async () => { + const { get } = await boot(); + const wb = await loadXlsxWorkbook((await get({ template: 'true', locale: 'zh-CN' })).body()); + expect(wb.worksheets.map((w) => w.name)).toEqual(['模板', '填写说明']); + }); + + it('an Accept-Language of zh-CN answers the Chinese sheets too', async () => { + const { get } = await boot(); + const wb = await loadXlsxWorkbook((await get({ template: 'true' }, 'deal', { 'accept-language': 'zh-CN,zh;q=0.9' })).body()); + expect(wb.worksheets.map((w) => w.name)).toEqual(['模板', '填写说明']); + }); + + it('with no locale at all the sheets are English', async () => { + const { get } = await boot(); + const wb = await loadXlsxWorkbook((await get({ template: 'true' })).body()); + expect(wb.worksheets.map((w) => w.name)).toEqual(['Template', 'Instructions']); + }); + + it('format=xlsx is accepted beside it', async () => { + const { get } = await boot(); + expect((await get({ template: 'true', format: 'xlsx' })).status()).toBe(200); + }); +}); + +describe('?template=true — field-level security: the WRITE projection', () => { + // `salary` is readable and NOT editable for this caller; `amount` is neither. + const READABLE = ['title', 'stage', 'secret_note', 'hot', 'tags', 'close_date', 'account', 'salary']; + const WRITABLE = ['title', 'stage', 'secret_note', 'hot', 'tags', 'close_date', 'account']; + + it('a field the caller may read but not edit is absent, and the response names the write projection', async () => { + const { get } = await boot({ + security: { canExport: async () => true, getReadableFields: async () => READABLE, getWritableFields: async () => WRITABLE }, + }); + const out = await get({ template: 'true' }); + const { wb, header } = await headerRow(out.body()); + expect(header).not.toContain('Salary'); + expect(header).not.toContain('Amount'); + expect(header).toContain('Account'); + expect(out.headers['X-Export-Template-Projection']).toBe('writable'); + expect(wb.worksheets[1].getCell(3, 1).value).toBeNull(); + }); + + it('a security service without getWritableFields: narrowed by the read projection, and the response says so', async () => { + const { get } = await boot({ security: { canExport: async () => true, getReadableFields: async () => READABLE } }); + const out = await get({ template: 'true' }); + const { wb, header } = await headerRow(out.body()); + expect(header).toContain('Salary'); + expect(header).not.toContain('Amount'); + expect(out.headers['X-Export-Template-Projection']).toBe('readable'); + expect(String(wb.worksheets[1].getCell(3, 1).value)).toContain('cannot say which fields you can edit'); + }); + + it('a security service that answers neither projection fails the request instead of an unnarrowed header', async () => { + const { get } = await boot({ + security: { canExport: async () => true, getWritableFields: async () => undefined, getReadableFields: async () => undefined }, + }); + const out = await get({ template: 'true' }); + expect(out.status()).toBe(500); + expect(out.json()?.error?.code ?? out.json()?.code).toBe('INTERNAL_ERROR'); + expect(out.body().length).toBe(0); + }); +}); + +describe('?template=true — the two existing gates still decide', () => { + it('a caller without the export permission is refused 403 and gets no workbook', async () => { + const { get } = await boot({ security: { canExport: async () => false, getReadableFields: async () => ['title'] } }); + const out = await get({ template: 'true' }); + expect(out.status()).toBe(403); + expect(out.json()).toMatchObject({ code: 'EXPORT_NOT_PERMITTED' }); + expect(out.body().length).toBe(0); + }); + + it('an object that does not expose export is refused before the template', async () => { + const { get, engine } = await boot(); + engine.registry.registerObject({ + // `export` is derived from `list`, so only a whitelist without `list` closes it. + name: 'locked', label: 'Locked', enable: { apiMethods: ['get'] }, + fields: { title: { name: 'title', type: 'text' } }, + } as any); + const out = await get({ template: 'true' }, 'locked'); + expect(out.status()).toBe(405); + expect(out.json()).toMatchObject({ code: 'OBJECT_API_METHOD_NOT_ALLOWED' }); + expect(out.body().length).toBe(0); + }); +}); + +describe('?template= — refusals (nested VALIDATION_ERROR, nothing written)', () => { + const refused = async (query: Record) => { + const { get, findData } = await boot(); + const out = await get(query); + expect(out.status(), JSON.stringify(query)).toBe(400); + expect(out.json()?.error?.code).toBe('VALIDATION_ERROR'); + expect(out.body().length).toBe(0); + expect(findData).not.toHaveBeenCalled(); + return String(out.json()?.error?.message); + }; + + it('a value other than true / false', async () => { + expect(await refused({ template: 'yes' })).toContain('takes true or false'); + }); + + it('a row parameter beside template=true', async () => { + expect(await refused({ template: 'true', limit: '5' })).toContain('"limit"'); + }); + + it('a non-xlsx format beside template=true', async () => { + expect(await refused({ template: 'true', format: 'csv' })).toContain('format "csv"'); + }); + + it('template supplied twice', async () => { + expect(await refused({ template: ['true', 'true'] })).toContain('template'); + }); + + it('an unknown object answers 404, not a template', async () => { + const { get } = await boot(); + const out = await get({ template: 'true' }, 'no_such_object'); + expect(out.status()).toBe(404); + expect(out.body().length).toBe(0); + }); +}); + +describe('?template=true — the template filled in and imported back', () => { + it('the example row imports through the real import door with no coercion error', async () => { + const { get, importRoute, engine } = await boot(); + const bytes = (await get({ template: 'true' })).body(); + // What the import reader sees, and the column mapping a client builds + // from the header (the `*` stripped) — objectui's wizard does the same. + const rows = await parseXlsxToRows(bytes); + expect(rows).toHaveLength(1); + const wb = await loadXlsxWorkbook(bytes); + const headers = (wb.worksheets[0].getRow(1).values as unknown[]).slice(1) as string[]; + const fields = (wb.worksheets[1].getColumn(2).values as unknown[]).slice(5, 5 + headers.length) as string[]; + const mapping = Object.fromEntries(headers.map((h, i) => [h, fields[i]])); + + const out = makeRes(); + await importRoute.handler({ + params: { object: 'deal' }, + body: { format: 'xlsx', xlsxBase64: bytes.toString('base64'), mapping }, + } as any, out.res); + const report = out.json(); + const codes = (report?.results ?? []).map((r: any) => r.code).filter(Boolean); + expect(codes, JSON.stringify(report)).toEqual([]); + expect(report).toMatchObject({ total: 1, ok: 1, errors: 0, created: 1 }); + + const [stored] = await engine.find('deal', {}); + expect(stored).toMatchObject({ + title: 'Sample', stage: 'open', secret_note: 'Sample', amount: 1, hot: true, tags: ['vip', 'new'], salary: 1, + }); + expect(String(stored.close_date)).toContain('2026-01-31'); + }); +}); + +// --------------------------------------------------------------------------- +// Without ?template=true: byte for byte the pre-change output +// --------------------------------------------------------------------------- + +/** + * The export's output BEFORE `?template=` existed, captured by running these + * same eight requests against these same fixtures on `origin/main` at + * 6bff748bbd (the branch point), with `Date` frozen at the instant below and + * the business timezone resolved to UTC — which makes every byte, the xlsx zip + * and the `Content-Disposition` stamp included, reproducible on any host + * (measured identical under TZ=Asia/Shanghai and TZ=America/New_York). + * `bytes` + `sha256` cover the whole body; the text bodies are also kept + * verbatim so a difference reads as a diff. + */ +const FROZEN_NOW = '2026-09-29T12:34:56.000Z'; +const STAMP = '20260929-123456'; +const disposition = (ext: string) => `attachment; filename="task-${STAMP}.${ext}"; filename*=UTF-8''Task-${STAMP}.${ext}`; +const headersFor = (ext: string, contentType: string, limit: string, styles?: string) => ({ + 'Content-Type': contentType, + 'Content-Disposition': disposition(ext), + 'X-Export-Format': ext, + 'X-Export-Limit': limit, + ...(styles ? { 'X-Export-Styles': styles } : {}), + 'Cache-Control': 'no-store', +}); +const CSV = 'text/csv; charset=utf-8'; +const JSON_TYPE = 'application/json; charset=utf-8'; +const XLSX = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'; + +const PRE_CHANGE: Record; + headers: Record; + sha256: string; + bytes: number; + text?: string; +}> = { + csvDefault: { + query: {}, + headers: headersFor('csv', CSV, '10000'), + sha256: '26d251901ffc998b0c8b6b8bcc866cb5fbfa3a187c80eb396151fa5eb1ffe31e', + bytes: 123, + text: 'ID,标题,完成,优先级,截止,负责人\r\n1,写代码,是,高,2026-06-30,张三\r\n2,写文档,否,低,2026-07-01,李四\r\n', + }, + json: { + query: { format: 'json' }, + headers: headersFor('json', JSON_TYPE, '10000'), + sha256: '6bb52c02c92995b7a80767a6aae481a0ede2cf57acf01f945cbab24f6cbbfc91', + bytes: 355, + text: '[{"id":"1","created_at":"2026-09-29T12:34:56.000Z","updated_at":"2026-09-29T12:34:56.000Z","title":"写代码","done":"是","priority":"高","due":"2026-06-30","owner":"张三"},{"id":"2","created_at":"2026-09-29T12:34:56.000Z","updated_at":"2026-09-29T12:34:56.000Z","title":"写文档","done":"否","priority":"低","due":"2026-07-01","owner":"李四"}]', + }, + xlsxStyled: { + query: { format: 'xlsx' }, + headers: headersFor('xlsx', XLSX, '10000', 'applied'), + sha256: '4f90c3cc545cad6f21f2df21eed705ac9e12fce6122fc0795ddad1d5203d9eaa', + bytes: 6206, + }, + xlsxUnstyled: { + query: { format: 'xlsx', limit: '20000' }, + headers: headersFor('xlsx', XLSX, '20000', 'dropped'), + sha256: 'ed02f9dfba285b0a16a11f079c51b790839f578de11830e5b14b2e5ca9d03315', + bytes: 6137, + }, + csvFields: { + query: { format: 'csv', fields: 'title,owner' }, + headers: headersFor('csv', CSV, '10000'), + sha256: '62cd186863bc41e298199695f8d9405a52c974b4b49bce5776ca3176c7ae5321', + bytes: 54, + text: '标题,负责人\r\n写代码,张三\r\n写文档,李四\r\n', + }, + csvNoHeader: { + query: { format: 'csv', header: 'false' }, + headers: headersFor('csv', CSV, '10000'), + sha256: '7f8f78c0d4847c0dfe9e09bdb48fe6985bcd8d10468119c257fe36b2c996793b', + bytes: 78, + text: '1,写代码,是,高,2026-06-30,张三\r\n2,写文档,否,低,2026-07-01,李四\r\n', + }, + csvEmptyNoProjection: { + query: { format: 'csv', filter: '{"title":"nope"}' }, + headers: headersFor('csv', CSV, '10000'), + sha256: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', + bytes: 0, + text: '', + }, + xlsxEmptyExplicitFields: { + query: { format: 'xlsx', fields: 'id,title', filter: '{"title":"nope"}' }, + headers: headersFor('xlsx', XLSX, '10000', 'applied'), + sha256: '8bcba0e24aa4e17c410831c7ed31041bc94b4075bbe511dbefde5e17f4709c3b', + bytes: 5934, + }, +}; + +/** `export-integration.test.ts`'s fixtures: `systemFields: false`, two rows. */ +async function bootExportFixture() { + const engine = new ObjectQL(); + liveEngines.push(engine); + engine.registerDriver(makeSqliteDriver(), true); + await engine.init(); + engine.registry.registerObject({ + name: 'user', label: 'User', systemFields: false, + fields: { id: { name: 'id', type: 'text', primaryKey: true }, name: { name: 'name', type: 'text', label: '姓名' } }, + } as any); + engine.registry.registerObject({ + name: 'task', label: 'Task', systemFields: false, + fields: { + id: { name: 'id', type: 'text', primaryKey: true, label: 'ID' }, + title: { name: 'title', type: 'text', label: '标题' }, + done: { name: 'done', type: 'boolean', label: '完成' }, + priority: { + name: 'priority', type: 'select', label: '优先级', + options: [{ label: '高', value: 'high', color: '#e11d48' }, { label: '低', value: 'low', color: '#3ab' }], + }, + due: { name: 'due', type: 'date', label: '截止' }, + owner: { name: 'owner', type: 'lookup', label: '负责人', reference: 'user', displayField: 'name' }, + }, + } as any); + await engine.syncSchemas(); + await engine.insert('user', { id: 'u1', name: '张三' }); + await engine.insert('user', { id: 'u2', name: '李四' }); + await engine.insert('task', { id: '1', title: '写代码', done: true, priority: 'high', due: '2026-06-30T00:00:00.000Z', owner: 'u1' }); + await engine.insert('task', { id: '2', title: '写文档', done: false, priority: 'low', due: '2026-07-01T00:00:00.000Z', owner: 'u2' }); + const protocol = new ObjectStackProtocolImplementation(engine as any); + const rest = new RestServer(createMockServer() as any, protocol as any, { api: { requireAuth: false } } as any); + (rest as any).resolveExecCtx = async () => ({ userId: 'test-user', timezone: 'UTC' }); + rest.registerRoutes(); + return rest.getRoutes().find((r: any) => r.method === 'GET' && r.path === '/api/v1/data/:object/export')!; +} + +async function exportOnce(query: Record) { + vi.useFakeTimers({ now: new Date(FROZEN_NOW), toFake: ['Date'] }); + try { + const route: any = await bootExportFixture(); + const out = makeRes(); + await route.handler({ params: { object: 'task' }, query } as any, out.res); + return out; + } finally { + vi.useRealTimers(); + } +} + +describe('without ?template=true the export is byte-identical to the pre-change output', () => { + for (const [name, expected] of Object.entries(PRE_CHANGE)) { + it(`${name}: ${JSON.stringify(expected.query)}`, async () => { + const out = await exportOnce(expected.query); + const body = out.body(); + expect(out.status()).toBe(200); + expect(out.headers).toEqual(expected.headers); + if (expected.text !== undefined) expect(body.toString('utf8')).toBe(expected.text); + expect(body.length).toBe(expected.bytes); + expect(createHash('sha256').update(body).digest('hex')).toBe(expected.sha256); + }); + } + + it('template=false is the export itself, byte for byte', async () => { + const plain = await exportOnce({}); + const off = await exportOnce({ template: 'false' }); + expect(off.status()).toBe(200); + expect(off.headers).toEqual(plain.headers); + expect(off.body().equals(plain.body())).toBe(true); + expect(off.body().toString('utf8')).toBe(PRE_CHANGE.csvDefault.text); + }); +}); diff --git a/packages/rest/src/import-template.test.ts b/packages/rest/src/import-template.test.ts new file mode 100644 index 00000000000..d15a66580f0 --- /dev/null +++ b/packages/rest/src/import-template.test.ts @@ -0,0 +1,487 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Unit pins for `import-template.ts` — the column rule of the export door's + * `?template=true` mode, its request reading, the instructions' claims about + * the import reader, and the workbook it builds (read back with exceljs). + * + * Each row of the column-rule table has ONE test below whose fixture only that + * row's rule excludes, so removing the rule turns exactly that test red. + */ + +import { describe, it, expect } from 'vitest'; +import { + IMPORT_BOOLEAN_FALSE_TOKENS, + IMPORT_BOOLEAN_TRUE_TOKENS, +} from '@objectstack/spec/data'; +import { + TEMPLATE_COLUMN_EXCLUSIONS, + TEMPLATE_INAPPLICABLE_PARAMS, + TEMPLATE_READER_CLAIMS, + TEMPLATE_VALIDATED_ROWS, + buildImportTemplateWorkbook, + columnLetter, + describeTemplateColumns, + isTemplateRequired, + readTemplateMode, + resolveTemplateProjection, + templateColumns, + templateText, +} from './import-template.js'; +import { + coerceFieldValue, + parseBooleanCell, + parseDateCell, + parseNumberCell, + splitMulti, +} from './import-coerce.js'; +import { buildFieldMetaMap } from './export-format.js'; +import { loadXlsxWorkbook } from './xlsx-test-loader.js'; + +/** An object whose one plain field every rule keeps, so each test has a control. */ +function objectWith(extra: Record>): { name: string; fields: Record } { + return { + name: 'proj', + fields: { + title: { name: 'title', type: 'text', label: 'Title' }, + ...Object.fromEntries(Object.entries(extra).map(([k, v]) => [k, { name: k, ...v }])), + }, + }; +} + +// ───────────────────────────────────────────────────────────────────────────── +// The column rule — one test per row of the table +// ───────────────────────────────────────────────────────────────────────────── + +describe('templateColumns — the column rule, row by row', () => { + it('declares the four exclusion rows, in the table\'s order', () => { + expect(TEMPLATE_COLUMN_EXCLUSIONS.map((r) => r.id)).toEqual(['system', 'readonly', 'computed', 'autonumber']); + }); + + it('starting point: every schema field is a candidate (a plain field of each kind is kept)', () => { + const schema = objectWith({ + amount: { type: 'number' }, + done: { type: 'boolean' }, + stage: { type: 'select', options: [{ label: 'Open', value: 'open' }] }, + due: { type: 'date' }, + account: { type: 'lookup', reference: 'account' }, + }); + expect(templateColumns(schema)).toEqual(['title', 'amount', 'done', 'stage', 'due', 'account']); + }); + + it('system: a `system: true` field is excluded', () => { + expect(templateColumns(objectWith({ owner_id: { type: 'lookup', reference: 'sys_user', system: true } }))) + .toEqual(['title']); + }); + + it('hidden: a writable `hidden: true` field is KEPT — the import stores it', () => { + expect(templateColumns(objectWith({ secret_note: { type: 'text', hidden: true } }))).toEqual(['title', 'secret_note']); + // …and a hidden field that is also readonly stays out, by the readonly row. + expect(templateColumns(objectWith({ org: { type: 'text', hidden: true, readonly: true } }))).toEqual(['title']); + }); + + it('readonly: a `readonly: true` field is excluded', () => { + expect(templateColumns(objectWith({ approval_status: { type: 'text', readonly: true } }))).toEqual(['title']); + }); + + it('formula / summary: both computed types are excluded', () => { + expect(templateColumns(objectWith({ + doubled: { type: 'formula', expression: 'amount * 2' }, + total: { type: 'summary', summaryOperations: { object: 'line', field: 'qty', function: 'sum' } }, + }))).toEqual(['title']); + }); + + it('autonumber: an autonumber field is excluded', () => { + expect(templateColumns(objectWith({ code: { type: 'autonumber', autonumberFormat: 'P-{0000}' } }))).toEqual(['title']); + }); + + it('FLS: only the fields the caller\'s projection admits are columns — both sides', () => { + const schema = objectWith({ salary: { type: 'number' }, notes: { type: 'text' } }); + const cols = templateColumns(schema, { permitted: new Set(['title', 'notes']) }); + expect(cols).not.toContain('salary'); + expect(cols).toEqual(['title', 'notes']); + // No projection (no field-level security composed) narrows nothing. + expect(templateColumns(schema)).toEqual(['title', 'salary', 'notes']); + }); + + it('?fields=: an explicit list is honoured verbatim — no rule and no projection narrows it', () => { + const schema = objectWith({ code: { type: 'autonumber' }, salary: { type: 'number' } }); + expect(templateColumns(schema, { explicitFields: ['code', 'salary'], permitted: new Set(['title']) })) + .toEqual(['code', 'salary']); + }); + + it('column order: the author\'s declaration order, a required field NOT moved forward', () => { + const schema = objectWith({ + b_field: { type: 'text' }, + a_required: { type: 'text', required: true }, + c_field: { type: 'text' }, + }); + expect(templateColumns(schema)).toEqual(['title', 'b_field', 'a_required', 'c_field']); + }); + + it('reads the array `fields` shape too, by each entry\'s own name', () => { + const schema = { fields: [{ name: 'a', type: 'text' }, { name: 'b', type: 'text', readonly: true }, { name: 'c', type: 'number' }] }; + expect(templateColumns(schema)).toEqual(['a', 'c']); + }); +}); + +describe('resolveTemplateProjection — the write projection, and the stated fallback', () => { + const ctx = { userId: 'u1' }; + + it('asks getWritableFields first, and does not consult the read projection when it answers', async () => { + let readAsked = false; + const answer = await resolveTemplateProjection({ + getWritableFields: async () => ['title'], + getReadableFields: async () => { readAsked = true; return ['title', 'locked']; }, + }, 'proj', ctx); + expect(answer).toEqual({ source: 'writable', permitted: new Set(['title']) }); + expect(readAsked).toBe(false); + }); + + it('a service without getWritableFields: the read projection, marked `readable` so the response states it', async () => { + const answer = await resolveTemplateProjection({ getReadableFields: async () => ['title', 'locked'] }, 'proj', ctx); + expect(answer).toEqual({ source: 'readable', permitted: new Set(['title', 'locked']) }); + }); + + it('a write projection that gives no answer falls back the same way', async () => { + const answer = await resolveTemplateProjection({ + getWritableFields: async () => undefined, + getReadableFields: async () => ['title'], + }, 'proj', ctx); + expect(answer.source).toBe('readable'); + }); + + it('`[]` from getWritableFields is a real answer — nothing writable — never a fallback', async () => { + const answer = await resolveTemplateProjection({ + getWritableFields: async () => [], + getReadableFields: async () => ['title'], + }, 'proj', ctx); + expect(answer).toEqual({ source: 'writable', permitted: new Set() }); + }); + + it('no security service: no projection applies; a service that answers neither: unanswered', async () => { + expect(await resolveTemplateProjection(undefined, 'proj', ctx)).toEqual({ source: 'none' }); + expect(await resolveTemplateProjection({ getReadableFields: async () => undefined }, 'proj', ctx)) + .toEqual({ source: 'unanswered' }); + expect(await resolveTemplateProjection({}, 'proj', ctx)).toEqual({ source: 'unanswered' }); + }); +}); + +describe('isTemplateRequired — the `*` mark', () => { + it('marks a required field with no default', () => { + expect(isTemplateRequired({ required: true })).toBe(true); + }); + it('does NOT mark a required field that declares a default — the engine fills it', () => { + expect(isTemplateRequired({ required: true, defaultValue: 'standard' })).toBe(false); + }); + it('does not mark an optional field', () => { + expect(isTemplateRequired({ required: false })).toBe(false); + expect(isTemplateRequired({})).toBe(false); + }); +}); + +// ───────────────────────────────────────────────────────────────────────────── +// Reading the request +// ───────────────────────────────────────────────────────────────────────────── + +describe('readTemplateMode', () => { + it('absent or false is the export; true (any case) is the template', () => { + expect(readTemplateMode({})).toEqual({ kind: 'export' }); + expect(readTemplateMode({ template: 'false' })).toEqual({ kind: 'export' }); + expect(readTemplateMode({ template: 'FALSE' })).toEqual({ kind: 'export' }); + expect(readTemplateMode({ template: 'true' })).toEqual({ kind: 'template' }); + expect(readTemplateMode({ template: 'True', format: 'xlsx', fields: 'a,b', locale: 'zh-CN' })).toEqual({ kind: 'template' }); + }); + + it('any other value is refused, never read as false', () => { + for (const value of ['yes', '1', '', 'template', ['true']]) { + const read = readTemplateMode({ template: value }); + expect(read.kind, JSON.stringify(value)).toBe('refused'); + expect(read.kind === 'refused' && read.message).toContain('"template" query parameter takes true or false'); + } + }); + + it('a row parameter on a template request is refused, and every one present is named', () => { + for (const name of TEMPLATE_INAPPLICABLE_PARAMS) { + const read = readTemplateMode({ template: 'true', [name]: 'x' }); + expect(read.kind, name).toBe('refused'); + expect(read.kind === 'refused' && read.message).toContain(`"${name}"`); + } + const two = readTemplateMode({ template: 'true', limit: '5', filter: '{}' }); + expect(two.kind === 'refused' && two.message).toContain('"limit", "filter"'); + }); + + it('a non-xlsx format on a template request is refused', () => { + const read = readTemplateMode({ template: 'true', format: 'csv' }); + expect(read.kind).toBe('refused'); + expect(read.kind === 'refused' && read.message).toContain('format "csv"'); + }); +}); + +// ───────────────────────────────────────────────────────────────────────────── +// What the instructions say is what the reader does +// ───────────────────────────────────────────────────────────────────────────── + +describe('TEMPLATE_READER_CLAIMS — every quoted spelling, through the real reader', () => { + const n = TEMPLATE_READER_CLAIMS.number; + + it('numbers: thousands groups, the decimal point, currency, percent and parentheses read as claimed', () => { + for (const [cell, value] of n.thousands) expect(parseNumberCell(cell), cell).toBe(value); + expect(parseNumberCell(n.decimalPoint[0])).toBe(n.decimalPoint[1]); + for (const sym of n.currencySymbols) expect(parseNumberCell(`${sym}25`), sym).toBe(25); + expect(parseNumberCell(n.percent[0])).toBe(n.percent[1]); + expect(parseNumberCell(n.negative[0])).toBe(n.negative[1]); + }); + + it('numbers: the decimal comma is refused, as the text says', () => { + expect(parseNumberCell(n.decimalCommaRefused)).toBeUndefined(); + }); + + it('dates: the accepted spellings read as claimed, the refused ones are refused', () => { + for (const [cell, day] of TEMPLATE_READER_CLAIMS.date.accepted) expect(parseDateCell(cell, 'date'), cell).toBe(day); + for (const cell of TEMPLATE_READER_CLAIMS.date.refused) expect(parseDateCell(cell, 'date'), cell).toBeUndefined(); + }); + + it('datetimes: the zone-naive spellings are read, the offset one as written', () => { + for (const cell of TEMPLATE_READER_CLAIMS.datetime.naive) { + expect(parseDateCell(cell, 'datetime', 'Asia/Shanghai'), cell).toBeDefined(); + } + const [cell, instant] = TEMPLATE_READER_CLAIMS.datetime.offset; + expect(parseDateCell(cell, 'datetime', 'America/New_York')).toBe(instant); + // Zone-naive is read in the business timezone, UTC when none is set. + expect(parseDateCell(TEMPLATE_READER_CLAIMS.datetime.naive[0], 'datetime')).toBe('2026-01-31T09:30:00.000Z'); + expect(parseDateCell(TEMPLATE_READER_CLAIMS.datetime.naive[0], 'datetime', 'Asia/Shanghai')).toBe('2026-01-31T01:30:00.000Z'); + }); + + it('times: both spellings read as claimed', () => { + for (const [cell, time] of TEMPLATE_READER_CLAIMS.time.accepted) expect(parseDateCell(cell, 'time'), cell).toBe(time); + }); + + it('multi-value cells: every named separator, and a line break, splits', () => { + for (const sep of TEMPLATE_READER_CLAIMS.multiSeparators) expect(splitMulti(`a${sep}b`), sep).toEqual(['a', 'b']); + expect(splitMulti('a\nb')).toEqual(['a', 'b']); + }); + + it('booleans: every token the text lists is one the reader takes, on the side it is listed', () => { + for (const t of IMPORT_BOOLEAN_TRUE_TOKENS) expect(parseBooleanCell(t), t).toBe(true); + for (const f of IMPORT_BOOLEAN_FALSE_TOKENS) expect(parseBooleanCell(f), f).toBe(false); + for (const locale of ['en', 'zh-CN']) { + const [yes, no] = templateText(locale).booleanWords; + expect(parseBooleanCell(yes)).toBe(true); + expect(parseBooleanCell(no)).toBe(false); + } + }); + + it('the number and boolean sentences are BUILT from those claims — no spelling is typed twice', () => { + for (const locale of ['en', 'zh-CN']) { + const text = templateText(locale); + for (const [cell] of n.thousands) expect(text.number).toContain(cell); + expect(text.number).toContain(n.decimalCommaRefused); + expect(text.number).toContain(n.percent[0]); + expect(text.number).toContain(n.negative[0]); + const sentence = text.boolean([...IMPORT_BOOLEAN_TRUE_TOKENS].join(' '), [...IMPORT_BOOLEAN_FALSE_TOKENS].join(' ')); + for (const t of [...IMPORT_BOOLEAN_TRUE_TOKENS, ...IMPORT_BOOLEAN_FALSE_TOKENS]) expect(sentence).toContain(t); + for (const [cell] of TEMPLATE_READER_CLAIMS.date.accepted) expect(text.date).toContain(cell); + for (const cell of TEMPLATE_READER_CLAIMS.date.refused) expect(text.date).toContain(cell); + } + }); +}); + +// ───────────────────────────────────────────────────────────────────────────── +// Describing columns: headers, examples, dropdowns +// ───────────────────────────────────────────────────────────────────────────── + +const KITCHEN_SINK = { + name: 'deal', + label: 'Deal', + fields: { + name: { name: 'name', type: 'text', label: 'Name', required: true }, + tier: { + name: 'tier', type: 'select', label: 'Tier', required: true, defaultValue: 'standard', + options: [{ label: 'Standard', value: 'standard' }, { label: 'Gold', value: 'gold' }], + }, + amount: { name: 'amount', type: 'currency', label: 'Amount', min: 5, max: 100 }, + won: { name: 'won', type: 'boolean', label: 'Won' }, + tags: { + name: 'tags', type: 'multiselect', label: 'Tags', + options: [{ label: 'Hot', value: 'hot' }, { label: 'New', value: 'new' }, { label: 'VIP', value: 'vip' }], + }, + close_date: { name: 'close_date', type: 'date', label: 'Close date' }, + call_at: { name: 'call_at', type: 'datetime', label: 'Call at' }, + slot: { name: 'slot', type: 'time', label: 'Slot' }, + account: { name: 'account', type: 'lookup', label: 'Account', reference: 'account' }, + watchers: { name: 'watchers', type: 'lookup', label: 'Watchers', reference: 'sys_user', multiple: true }, + }, +}; + +describe('describeTemplateColumns', () => { + const fields = templateColumns(KITCHEN_SINK); + const cols = describeTemplateColumns(KITCHEN_SINK, fields, { + locale: 'en', + referenceLabels: new Map([['account', 'Account'], ['sys_user', 'User']]), + }); + const byField = new Map(cols.map((c) => [c.field, c])); + + it('headers are the labels, `*` only on a required field with no default', () => { + expect(cols.map((c) => c.header)).toEqual([ + 'Name *', 'Tier', 'Amount', 'Won', 'Tags', 'Close date', 'Call at', 'Slot', 'Account', 'Watchers', + ]); + }); + + it('closed single-valued domains carry a dropdown; multi-valued and open ones do not', () => { + expect(byField.get('tier')?.dropdown).toEqual(['Standard', 'Gold']); + expect(byField.get('won')?.dropdown).toEqual(['yes', 'no']); + expect(byField.get('tags')?.dropdown).toBeUndefined(); + expect(byField.get('name')?.dropdown).toBeUndefined(); + expect(byField.get('account')?.dropdown).toBeUndefined(); + }); + + it('a radio field carries the same dropdown a select does; a select flagged multiple does not', () => { + const schema = { + fields: { + size: { name: 'size', type: 'radio', options: [{ label: 'S', value: 's' }, { label: 'L', value: 'l' }] }, + colors: { name: 'colors', type: 'select', multiple: true, options: [{ label: 'Red', value: 'red' }] }, + }, + }; + const [size, colors] = describeTemplateColumns(schema, ['size', 'colors'], { locale: 'en' }); + expect(size.dropdown).toEqual(['S', 'L']); + expect(colors.dropdown).toBeUndefined(); + }); + + it('the multiselect example joins two real options', () => { + expect(byField.get('tags')?.example).toBe('Hot, New'); + const zh = describeTemplateColumns(KITCHEN_SINK, ['tags'], { locale: 'zh-CN' }); + expect(zh[0].example).toBe('Hot、New'); + }); + + it('a number example sits inside the declared range, and the range is stated', () => { + expect(byField.get('amount')?.example).toBe(5); + expect(byField.get('amount')?.howToFill).toContain('Between 5 and 100.'); + }); + + it('a reference column names its target by label, and states the ambiguity refusal', () => { + expect(byField.get('account')?.howToFill).toContain('Account record'); + expect(byField.get('account')?.howToFill).toContain('reference_ambiguous'); + expect(byField.get('watchers')?.howToFill).toContain('One or more User record names'); + }); + + it('every example cell coerces through the import reader with no invalid_* error', async () => { + const metaMap = buildFieldMetaMap(KITCHEN_SINK); + for (const c of cols) { + if (c.example === undefined) continue; + const out = await coerceFieldValue(c.example, metaMap.get(c.field), { timezone: 'UTC' }); + expect('error' in out ? out.error : undefined, `${c.field} = ${JSON.stringify(c.example)}`).toBeUndefined(); + } + }); + + it('a name in ?fields= that is no field of the object is described as such', () => { + const [unknown] = describeTemplateColumns(KITCHEN_SINK, ['nope'], { locale: 'en' }); + expect(unknown).toMatchObject({ header: 'nope', type: '', howToFill: 'Not a field of this object.' }); + }); +}); + +// ───────────────────────────────────────────────────────────────────────────── +// The workbook, read back with exceljs +// ───────────────────────────────────────────────────────────────────────────── + +describe('buildImportTemplateWorkbook — read back', () => { + async function roundTrip(locale: string) { + const fields = templateColumns(KITCHEN_SINK); + const cols = describeTemplateColumns(KITCHEN_SINK, fields, { locale }); + const wb = await buildImportTemplateWorkbook(cols, { locale }); + const bytes = Buffer.from(await wb.xlsx.writeBuffer()); + return { cols, wb: await loadXlsxWorkbook(bytes) }; + } + + it('sheet one is the template — header and example, and NO data rows', async () => { + const { cols, wb } = await roundTrip('en'); + expect(wb.worksheets.map((w) => w.name)).toEqual(['Template', 'Instructions']); + const sheet = wb.worksheets[0]; + expect((sheet.getRow(1).values as unknown[]).slice(1)).toEqual(cols.map((c) => c.header)); + expect(sheet.actualRowCount).toBe(2); + // Nothing below the example row carries a value. + let lastWithValue = 0; + sheet.eachRow({ includeEmpty: false }, (_row, rowNumber) => { lastWithValue = rowNumber; }); + expect(lastWithValue).toBe(2); + }); + + it('a required column\'s header carries `*`', async () => { + const { wb } = await roundTrip('en'); + expect(wb.worksheets[0].getCell('A1').value).toBe('Name *'); + expect(wb.worksheets[0].getCell('B1').value).toBe('Tier'); + }); + + it('dropdown columns carry a list validation over the whole import range, sourced from the instructions sheet', async () => { + const { cols, wb } = await roundTrip('en'); + const sheet = wb.worksheets[0]; + const guide = wb.worksheets[1]; + const withDropdown = cols.map((c, i) => ({ c, letter: columnLetter(i + 1) })).filter(({ c }) => c.dropdown); + expect(withDropdown.map(({ c }) => c.field)).toEqual(['tier', 'won']); + for (const { c, letter } of withDropdown) { + for (const row of [2, TEMPLATE_VALIDATED_ROWS + 1]) { + const dv = sheet.getCell(`${letter}${row}`).dataValidation; + expect(dv?.type, `${letter}${row}`).toBe('list'); + const ref = /^'([^']+)'!\$([A-Z]+)\$(\d+):\$([A-Z]+)\$(\d+)$/.exec(String(dv?.formulae?.[0])); + expect(ref, String(dv?.formulae?.[0])).not.toBeNull(); + const [, sheetName, col, from, , to] = ref!; + expect(sheetName).toBe('Instructions'); + const listed: unknown[] = []; + for (let r = Number(from); r <= Number(to); r++) listed.push(guide.getCell(`${col}${r}`).value); + expect(listed).toEqual(c.dropdown); + } + expect(sheet.getCell(`${letter}${TEMPLATE_VALIDATED_ROWS + 2}`).dataValidation).toBeUndefined(); + } + // A column without a dropdown carries no validation. + expect(sheet.getCell('A2').dataValidation).toBeUndefined(); + expect(sheet.getCell('E2').dataValidation).toBeUndefined(); + }); + + it('the instructions sheet has one row per column: header, field, type, required, how to fill', async () => { + const { cols, wb } = await roundTrip('en'); + const guide = wb.worksheets[1]; + expect((guide.getRow(4).values as unknown[]).slice(1, 6)).toEqual(['Column', 'Field', 'Type', 'Required', 'How to fill it']); + cols.forEach((c, i) => { + const values = (guide.getRow(5 + i).values as unknown[]).slice(1, 6); + expect(values).toEqual([c.header, c.field, c.type, c.required ? 'Yes' : 'No', c.howToFill]); + }); + expect(String(guide.getCell('A1').value)).toContain('replace it or delete it before you import'); + }); + + it('text-shaped columns are text-formatted so leading zeros survive; numeric and date ones are not', async () => { + const { cols, wb } = await roundTrip('en'); + const sheet = wb.worksheets[0]; + cols.forEach((c, i) => { + expect(sheet.getColumn(i + 1).numFmt === '@', c.field).toBe(c.textFormat); + }); + expect(cols.find((c) => c.field === 'name')?.textFormat).toBe(true); + expect(cols.find((c) => c.field === 'amount')?.textFormat).toBe(false); + }); + + it('the read-projection fallback is stated on the instructions sheet; the write projection adds no note', async () => { + const cols = describeTemplateColumns(KITCHEN_SINK, templateColumns(KITCHEN_SINK), { locale: 'en' }); + const notesOf = async (projection: 'writable' | 'readable' | 'none', locale = 'en') => { + const wb = await buildImportTemplateWorkbook(cols, { locale, projection }); + const guide = (await loadXlsxWorkbook(Buffer.from(await wb.xlsx.writeBuffer()))).worksheets[1]; + return [1, 2, 3].map((r) => guide.getCell(r, 1).value); + }; + expect((await notesOf('readable'))[2]).toBe(templateText('en').readProjectionNote); + expect((await notesOf('readable', 'zh-CN'))[2]).toBe(templateText('zh-CN').readProjectionNote); + expect((await notesOf('writable'))[2]).toBeNull(); + expect((await notesOf('none'))[2]).toBeNull(); + }); + + it('zh-CN: the sheets, headings and boolean dropdown are Chinese', async () => { + const { wb } = await roundTrip('zh-CN'); + expect(wb.worksheets.map((w) => w.name)).toEqual(['模板', '填写说明']); + expect((wb.worksheets[1].getRow(4).values as unknown[]).slice(1, 6)).toEqual(['列', '字段', '类型', '必填', '填写方式']); + const won = wb.worksheets[0].getCell('D2'); + expect(won.value).toBe('是'); + expect(String(won.dataValidation?.formulae?.[0])).toMatch(/^'填写说明'!/); + }); +}); + +describe('columnLetter', () => { + it('spells spreadsheet columns', () => { + expect([1, 2, 26, 27, 52, 53, 702, 703].map(columnLetter)).toEqual(['A', 'B', 'Z', 'AA', 'AZ', 'BA', 'ZZ', 'AAA']); + }); +}); diff --git a/packages/rest/src/import-template.ts b/packages/rest/src/import-template.ts new file mode 100644 index 00000000000..b5b4d1b9d4e --- /dev/null +++ b/packages/rest/src/import-template.ts @@ -0,0 +1,711 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The import TEMPLATE that `GET {basePath}/data/:object/export?template=true` + * answers: an xlsx workbook with no data rows, whose columns are the ones an + * import can actually write. + * + * ## Why the template has its own column rule + * + * The export's default columns are every field of the object, narrowed only + * by what the caller may READ. That is right for an export, whose reader wants + * to SEE the data, and wrong for a template, whose reader wants to FILL it: the + * registry injects `created_at`, `created_by`, `organization_id` and the other + * platform columns onto every object, and a value typed into one of them never + * lands. So the export's rule is left exactly as it is, and the template takes + * {@link templateColumns} — one question per column: "if I fill this in, does + * the import store it?" + * + * Each exclusion in {@link TEMPLATE_COLUMN_EXCLUSIONS} names the write-path + * behaviour that makes the answer "no", measured through `POST /data/:object/import` + * on the SQL driver (better-sqlite3) under a non-system caller: + * + * | rule | what the write path does with a value in that column | + * |--------------|----------------------------------------------------------------| + * | `readonly` | `stripReadonlyFields` drops it on insert and update, with a | + * | | `warn` and the field's `defaultValue` in its place | + * | `autonumber` | `stripRuntimeOwnedFields` drops it; the sequence issues one | + * | `computed` | `formula`: SQL refuses the row, memory creates it. `summary`: | + * | | stored, then overwritten by the next write of a child record | + * | `system` | the injected columns are all `readonly` save `owner_id`, which | + * | | the security middleware refuses (403) when it names anyone | + * | | but the caller and the caller holds no transfer grant | + * + * `hidden` is NOT an exclusion: an author-declared `hidden: true` field that + * is writable is stored by the import, so it is a column ([#18386] ruling, + * Q2 C). The injected hidden columns are `readonly`, so the `readonly` row + * keeps them out. + * + * Field-level security narrows the rest to what the caller may WRITE — see + * {@link resolveTemplateProjection}. + * + * An explicit `?fields=` list is honoured exactly as asked: the caller named + * the columns, so no rule narrows them (the export treats `?fields=` the same). + * + * ## What the instructions sheet says is what the reader does + * + * Every accepted spelling the instructions state is taken from the import + * reader's own vocabulary — the boolean tokens are the spec's + * `IMPORT_BOOLEAN_TRUE_TOKENS` / `IMPORT_BOOLEAN_FALSE_TOKENS`, and each + * example spelling for numbers, dates and times lives in + * {@link TEMPLATE_READER_CLAIMS}, which the tests run through + * `import-coerce.ts`'s own parsers. A sentence here that the reader does not + * honour is a failing test, not a stale comment. + * + * ## Why not the streaming writer + * + * A template has no rows to stream, and the export's `WorkbookWriter` path has + * never been measured with data validations. This builds an ordinary + * `Workbook` and writes it once. + */ + +import { + BOOLEAN_VALUE_TYPES, + FILE_REFERENCE_TYPES, + IMPORT_BOOLEAN_FALSE_TOKENS, + IMPORT_BOOLEAN_TRUE_TOKENS, + IMPORT_REFERENCE_TYPES, + MULTI_OPTION_TYPES, + NUMERIC_VALUE_TYPES, + SINGLE_OPTION_TYPES, + isMultiValueField, +} from '@objectstack/spec/data'; +import type { ISecurityService, SecurityContext } from '@objectstack/spec/contracts'; +import { buildFieldMetaMap, type ExportFieldMeta } from './export-format.js'; +import { loadExcelJs, type Workbook, type Worksheet } from './xlsx-module.js'; + +// ── the column rule ───────────────────────────────────────────────── + +/** The keys of a field definition the column rule and the instructions read. */ +export interface TemplateFieldDef { + type?: unknown; + readonly?: unknown; + system?: unknown; + required?: unknown; + defaultValue?: unknown; + min?: unknown; + max?: unknown; +} + +/** One exclusion of the template's column rule. */ +export interface TemplateColumnExclusion { + /** Stable id, one per row of the column-rule table in the module header. */ + readonly id: 'system' | 'readonly' | 'computed' | 'autonumber'; + /** Does this rule keep the field out of the template? */ + readonly excludes: (def: TemplateFieldDef) => boolean; +} + +/** + * The template's exclusions, one per row of the table in the module header. + * A field any of them excludes is not a template column. + */ +export const TEMPLATE_COLUMN_EXCLUSIONS: readonly TemplateColumnExclusion[] = Object.freeze([ + { id: 'system', excludes: (def) => def.system === true }, + { id: 'readonly', excludes: (def) => def.readonly === true }, + { id: 'computed', excludes: (def) => def.type === 'formula' || def.type === 'summary' }, + { id: 'autonumber', excludes: (def) => def.type === 'autonumber' }, +]); + +/** + * The object's field definitions by name, in the order the object declares + * them. Accepts both shapes `fields` takes across the stack — the object map + * the registry serves and a `FieldDefinition[]` — exactly as + * `buildFieldMetaMap` does, so a name here is a name there. + */ +export function templateFieldDefs(schema: unknown): Map { + const out = new Map(); + const fields = (schema as { fields?: unknown } | null | undefined)?.fields; + const entries: Array<[string, unknown]> = Array.isArray(fields) + ? fields.map((f) => [typeof (f as { name?: unknown })?.name === 'string' ? String((f as { name: string }).name) : '', f]) + : fields && typeof fields === 'object' + ? Object.entries(fields as Record).map(([key, def]) => [ + def && typeof def === 'object' && typeof (def as { name?: unknown }).name === 'string' + ? String((def as { name: string }).name) + : key, + def, + ]) + : []; + for (const [name, def] of entries) { + if (!name || !def || typeof def !== 'object') continue; + out.set(name, def as TemplateFieldDef); + } + return out; +} + +export interface TemplateColumnsOptions { + /** `?fields=` as the caller sent it. Non-empty ⇒ honoured verbatim. */ + explicitFields?: readonly string[]; + /** + * The field names the caller's field-level security admits, or `undefined` + * when no field-level security applies (no security service is composed). + * Ignored for an explicit `?fields=` list. + */ + permitted?: ReadonlySet; +} + +/** + * The template's columns, in the order the object declares its fields — + * never reordered: a required column is marked in its header, not moved. + */ +export function templateColumns(schema: unknown, opts: TemplateColumnsOptions = {}): string[] { + if (opts.explicitFields && opts.explicitFields.length > 0) return [...opts.explicitFields]; + const out: string[] = []; + for (const [name, def] of templateFieldDefs(schema)) { + if (TEMPLATE_COLUMN_EXCLUSIONS.some((rule) => rule.excludes(def))) continue; + if (opts.permitted && !opts.permitted.has(name)) continue; + out.push(name); + } + return out; +} + +// ── field-level security ──────────────────────────────────────────── + +/** + * Which field-level-security projection narrowed the columns: `writable` (the + * security service's write projection), `readable` (its read projection — the + * contract's soft-fail case, which the response states), or `none` (no + * projection applies: no security service, or an explicit `?fields=`). + */ +export type TemplateProjectionSource = 'writable' | 'readable' | 'none'; + +export type TemplateProjection = + | { source: 'writable' | 'readable'; permitted: ReadonlySet } + | { source: 'none' } + | { source: 'unanswered' }; + +/** + * Ask the security service which fields the template may offer. + * + * `getWritableFields` first: the fields a write may name without field-level + * security refusing the row. A service without it, or one that gives no answer, + * is the contract's soft-fail case — the READ projection narrows instead, and + * `readable` obliges the caller to say so. `unanswered`: a service is present + * and gave neither answer, so the caller refuses rather than widen silently. + */ +export async function resolveTemplateProjection( + security: Partial> | undefined, + objectName: string, + context: SecurityContext | undefined, +): Promise { + if (!security) return { source: 'none' }; + if (typeof security.getWritableFields === 'function') { + const writable = await security.getWritableFields(objectName, context); + if (Array.isArray(writable)) return { source: 'writable', permitted: new Set(writable) }; + } + if (typeof security.getReadableFields === 'function') { + const readable = await security.getReadableFields(objectName, context); + if (Array.isArray(readable)) return { source: 'readable', permitted: new Set(readable) }; + } + return { source: 'unanswered' }; +} + +/** + * Whether a column is marked required (`*`) in the template: a row that leaves + * it blank is refused. A `required` field that declares a `defaultValue` is + * NOT marked — the engine fills the default before it checks, so a blank cell + * there is accepted. + */ +export function isTemplateRequired(def: TemplateFieldDef | undefined): boolean { + return def?.required === true && def.defaultValue === undefined; +} + +// ── request reading ───────────────────────────────────────────────── + +/** + * The export parameters that select or page ROWS, or toggle the header. A + * template has no data rows and always carries its header, so on a template + * request each of these would be ignored — and an ignored parameter is refused + * on this route rather than dropped. + */ +export const TEMPLATE_INAPPLICABLE_PARAMS: readonly string[] = Object.freeze([ + 'limit', 'page', 'filter', 'search', 'searchFields', 'orderby', 'header', +]); + +export type TemplateModeRead = + | { kind: 'export' } + | { kind: 'template' } + | { kind: 'refused'; message: string }; + +/** + * Read `?template=` and the parameters it constrains. `true` / `false` in any + * letter case; `false` and absence both answer the ordinary export. Anything + * else is refused rather than read as `false`, because reading it as `false` + * answers a data export to a caller who asked for a template. + */ +export function readTemplateMode(query: Record): TemplateModeRead { + const raw = query.template; + if (raw === undefined) return { kind: 'export' }; + const value = typeof raw === 'string' ? raw.trim().toLowerCase() : undefined; + if (value === 'false') return { kind: 'export' }; + if (value !== 'true') { + return { + kind: 'refused', + message: `The "template" query parameter takes true or false, got ${JSON.stringify(raw)}. ` + + 'template=true answers an xlsx import template; template=false, or no template parameter, ' + + 'answers the data export.', + }; + } + const inapplicable = TEMPLATE_INAPPLICABLE_PARAMS.filter((name) => query[name] !== undefined); + if (inapplicable.length > 0) { + const names = inapplicable.map((n) => `"${n}"`).join(', '); + return { + kind: 'refused', + message: `template=true answers an import template, which has no data rows and always carries its header, ` + + `so ${inapplicable.length === 1 ? `the ${names} query parameter has` : `the query parameters ${names} have`} ` + + 'nothing to apply to. Remove them, or drop template=true to export data.', + }; + } + const format = query.format; + if (format !== undefined && String(format).trim().toLowerCase() !== 'xlsx') { + return { + kind: 'refused', + message: `template=true answers an xlsx workbook, and format ${JSON.stringify(format)} cannot carry ` + + 'its dropdowns or its instructions sheet. Omit format, or send format=xlsx.', + }; + } + return { kind: 'template' }; +} + +// ── what the instructions claim about the reader ──────────────────── + +/** + * Every example spelling the instructions sheet quotes, with the value the + * import reader produces for it (or `undefined` for a refused one). The text + * is BUILT from these, and the tests run each one through the reader + * (`parseNumberCell`, `parseDateCell`, `splitMulti`) — so the sheet cannot + * claim a spelling the reader does not honour. + */ +export const TEMPLATE_READER_CLAIMS = Object.freeze({ + number: { + thousands: [['1,000', 1000], ['12,345.67', 12345.67]] as ReadonlyArray, + decimalCommaRefused: '3,14', + decimalPoint: ['3.14', 3.14] as readonly [string, number], + currencySymbols: ['$', '¥', '€', '£', '￥'] as readonly string[], + percent: ['25%', 25] as readonly [string, number], + negative: ['(1,234)', -1234] as readonly [string, number], + }, + date: { + accepted: [['2026-01-31', '2026-01-31'], ['2026/1/31', '2026-01-31']] as ReadonlyArray, + refused: ['01/31/2026', '31/01/2026', '26/1/31', '2026-02-30'] as readonly string[], + }, + datetime: { + /** Zone-naive: read in the business timezone, so only the reading's existence is claimed. */ + naive: ['2026-01-31 09:30:00', '2026/1/31 9:30'] as readonly string[], + /** Offset-bearing: read as written. */ + offset: ['2026-01-31T09:30:00+08:00', '2026-01-31T01:30:00.000Z'] as readonly [string, string], + }, + time: { + accepted: [['09:30', '09:30:00'], ['09:30:00', '09:30:00']] as ReadonlyArray, + }, + /** The separators a multi-value cell is split on, as the sheet names them. */ + multiSeparators: ['、', ',', ';'] as readonly string[], +}); + +// ── localized text ────────────────────────────────────────────────── + +export interface TemplateText { + templateSheet: string; + instructionsSheet: string; + filenameSuffix: string; + notes: readonly string[]; + /** The note stating the soft-fail: the columns are the READ projection. */ + readProjectionNote: string; + headings: readonly [string, string, string, string, string]; + required: string; + optional: string; + booleanWords: readonly [string, string]; + multiJoiner: string; + text: string; + sampleText: string; + email: string; + url: string; + phone: string; + number: string; + range: (min: unknown, max: unknown) => string; + boolean: (trueTokens: string, falseTokens: string) => string; + singleOption: (labels: string) => string; + multiOption: (labels: string) => string; + freeMulti: string; + date: string; + datetime: string; + time: string; + reference: (target: string) => string; + multiReference: (target: string) => string; + file: string; + multiFile: string; + unknownField: string; +} + +const C = TEMPLATE_READER_CLAIMS; +const SEPARATORS_EN = `${C.multiSeparators.join(' ')} or a line break`; +const SEPARATORS_ZH = `${C.multiSeparators.join(' ')} 或换行`; +const THOUSANDS = C.number.thousands.map(([s]) => s); + +const EN: TemplateText = { + templateSheet: 'Template', + instructionsSheet: 'Instructions', + filenameSuffix: 'import template', + notes: [ + 'Fill one record per row on the Template sheet, starting at row 2. Row 2 holds an example value for each column: ' + + 'replace it or delete it before you import, or it is imported as a record.', + 'Columns marked * are required: a row that leaves one of them blank is refused.', + ], + readProjectionNote: 'These columns are the fields you can read: this deployment cannot say which fields you can edit. ' + + 'A row that fills a field you can read but not edit is refused.', + headings: ['Column', 'Field', 'Type', 'Required', 'How to fill it'], + required: 'Yes', + optional: 'No', + booleanWords: ['yes', 'no'], + multiJoiner: ', ', + text: 'Text.', + sampleText: 'Sample', + email: 'An email address.', + url: 'A URL.', + phone: 'A phone number.', + number: `A number. Type it as a number, or as text in which a comma may only group thousands: 1 to 3 digits, then ` + + `groups of exactly 3, and only before any "." (${THOUSANDS.join(' or ')}). Any other comma is refused, a decimal ` + + `comma included: write ${C.number.decimalPoint[0]}, not ${C.number.decimalCommaRefused}. Also read: a leading ` + + `currency symbol (${C.number.currencySymbols.join(' ')}), a trailing % (removed, so ${C.number.percent[0]} is read ` + + `as ${C.number.percent[1]}), and parentheses for a negative (${C.number.negative[0]} is read as ` + + `${C.number.negative[1]}).`, + range: (min, max) => (min !== undefined && max !== undefined + ? ` Between ${min} and ${max}.` + : min !== undefined ? ` At least ${min}.` : ` At most ${max}.`), + boolean: (t, f) => `Yes or no. Also read, in any letter case: ${t} for yes, and ${f} for no.`, + singleOption: (labels) => `One option from the dropdown: ${labels}. The option code is read too, and letter case is ignored.`, + multiOption: (labels) => `One or more of: ${labels}. Separate several with ${SEPARATORS_EN}.`, + freeMulti: `One or more values. Separate several with ${SEPARATORS_EN}.`, + date: `A date: ${C.date.accepted[0][0]}, or year first: ${C.date.accepted[1][0]}. A date cell is read as the day it shows. ` + + `Month-first and day-first dates (${C.date.refused[0]}, ${C.date.refused[1]}), two-digit years (${C.date.refused[2]}) ` + + `and a day that does not exist (${C.date.refused[3]}) are refused.`, + datetime: `A date and time: ${C.datetime.naive[0]}, or year first: ${C.datetime.naive[1]}, read in your organization's ` + + `business time zone (UTC when none is set); or ISO 8601 with an offset, read as written: ${C.datetime.offset[0]}.`, + time: `A time of day: ${C.time.accepted.map(([s]) => s).join(' or ')}.`, + reference: (target) => `The name of a ${target} record, or its id. A value that matches more than one record is refused ` + + 'as ambiguous (reference_ambiguous).', + multiReference: (target) => `One or more ${target} record names or ids, separated by ${SEPARATORS_EN}. A value that ` + + 'matches more than one record is refused as ambiguous (reference_ambiguous).', + file: 'A file id or URL.', + multiFile: `One or more file ids or URLs, separated by ${SEPARATORS_EN}.`, + unknownField: 'Not a field of this object.', +}; + +const ZH: TemplateText = { + templateSheet: '模板', + instructionsSheet: '填写说明', + filenameSuffix: '导入模板', + notes: [ + '在「模板」工作表中每行填写一条记录,从第 2 行开始。第 2 行是每一列的示例值:导入前请替换或删除,否则它会作为一条记录被导入。', + '带 * 的列为必填:其中任一列留空的行会被拒绝。', + ], + readProjectionNote: '这些列是你可读的字段:当前部署无法判断你可编辑哪些字段。填写了可读但不可编辑字段的行会被拒绝。', + headings: ['列', '字段', '类型', '必填', '填写方式'], + required: '是', + optional: '否', + booleanWords: ['是', '否'], + multiJoiner: '、', + text: '文本。', + sampleText: '示例', + email: '邮箱地址。', + url: '网址(URL)。', + phone: '电话号码。', + number: `数字。可以填数值,也可以填文本;文本中的逗号只能作千分位:开头 1 到 3 位数字,之后每组恰好 3 位,且只能出现在「.」之前` + + `(${THOUSANDS.join(' 或 ')})。其他任何逗号都会被拒绝,包括小数逗号:请写 ${C.number.decimalPoint[0]},` + + `不要写 ${C.number.decimalCommaRefused}。另外可以识别:开头的货币符号(${C.number.currencySymbols.join(' ')})、` + + `结尾的 %(会被去掉,${C.number.percent[0]} 读作 ${C.number.percent[1]})、表示负数的括号` + + `(${C.number.negative[0]} 读作 ${C.number.negative[1]})。`, + range: (min, max) => (min !== undefined && max !== undefined + ? ` 取值范围 ${min} 到 ${max}。` + : min !== undefined ? ` 最小 ${min}。` : ` 最大 ${max}。`), + boolean: (t, f) => `是或否。以下写法也可识别(不区分大小写):表示「是」的 ${t};表示「否」的 ${f}。`, + singleOption: (labels) => `从下拉列表中选择一项:${labels}。也可以填选项代码,不区分大小写。`, + multiOption: (labels) => `填一项或多项:${labels}。多项之间用 ${SEPARATORS_ZH} 分隔。`, + freeMulti: `填一个或多个值,多个之间用 ${SEPARATORS_ZH} 分隔。`, + date: `日期:${C.date.accepted[0][0]},或年份在前的 ${C.date.accepted[1][0]}。日期单元格按其显示的日期读取。` + + `月份或日期在前的写法(${C.date.refused[0]}、${C.date.refused[1]})、两位数年份(${C.date.refused[2]})` + + `以及不存在的日期(${C.date.refused[3]})都会被拒绝。`, + datetime: `日期时间:${C.datetime.naive[0]},或年份在前的 ${C.datetime.naive[1]},按组织的业务时区读取` + + `(未设置时区时按 UTC);也可以填带时区偏移的 ISO 8601,按所写时区读取:${C.datetime.offset[0]}。`, + time: `时间:${C.time.accepted.map(([s]) => s).join(' 或 ')}。`, + reference: (target) => `填「${target}」记录的名称,或其 ID。匹配到多条记录的值会因有歧义被拒绝(reference_ambiguous)。`, + multiReference: (target) => `填一个或多个「${target}」记录的名称或 ID,用 ${SEPARATORS_ZH} 分隔。` + + '匹配到多条记录的值会因有歧义被拒绝(reference_ambiguous)。', + file: '文件 ID 或 URL。', + multiFile: `一个或多个文件 ID 或 URL,用 ${SEPARATORS_ZH} 分隔。`, + unknownField: '不是该对象的字段。', +}; + +/** The template's language: Chinese for a `zh*` request locale, English otherwise. */ +export function templateText(locale: string | undefined): TemplateText { + return typeof locale === 'string' && locale.trim().toLowerCase().startsWith('zh') ? ZH : EN; +} + +// ── describing one column ─────────────────────────────────────────── + +/** Everything the workbook writes about one template column. */ +export interface TemplateColumn { + field: string; + /** The header cell: the (localized) label, with ` *` when required. */ + header: string; + type: string; + required: boolean; + /** The "How to fill it" cell of the instructions sheet. */ + howToFill: string; + /** The example cell under the header. `undefined` leaves it blank. */ + example: string | number | undefined; + /** The values a dropdown offers, for a closed single-valued domain. */ + dropdown?: readonly string[]; + /** Format the column as text, so a value like `00123` keeps its zeros. */ + textFormat: boolean; +} + +export interface DescribeTemplateOptions { + locale?: string; + /** The display label of each referenced object, by object name. */ + referenceLabels?: ReadonlyMap; +} + +function optionLabels(meta: ExportFieldMeta | undefined): string[] { + const out: string[] = []; + for (const o of meta?.options ?? []) { + if (!o) continue; + const label = typeof o.label === 'string' && o.label.trim().length > 0 ? o.label : o.value; + if (label === undefined || label === null) continue; + const s = String(label); + if (!out.includes(s)) out.push(s); + } + return out; +} + +function numberExample(def: TemplateFieldDef | undefined): number { + let n = 1; + if (typeof def?.min === 'number' && n < def.min) n = def.min; + if (typeof def?.max === 'number' && n > def.max) n = def.max; + return n; +} + +const TIME_OF_DAY_TYPES = new Set(['date', 'datetime', 'time']); + +/** Free-text types whose example is the localized word for "sample". */ +const SAMPLE_TEXT_TYPES = new Set(['text', 'textarea', 'markdown', 'richtext', 'html']); + +/** Example values that satisfy the write door's format check for their type. */ +const FORMATTED_TEXT_EXAMPLES: Readonly> = Object.freeze({ + email: 'name@example.com', + url: 'https://example.com', + phone: '+1 202 555 0100', +}); + +/** + * Describe each template column: its header, the instructions row and the + * example value. `schema` is the object as the caller reads it (labels + * already localized). + */ +export function describeTemplateColumns( + schema: unknown, + fields: readonly string[], + opts: DescribeTemplateOptions = {}, +): TemplateColumn[] { + const text = templateText(opts.locale); + const defs = templateFieldDefs(schema); + const metaMap = buildFieldMetaMap(schema); + const trueTokens = [...IMPORT_BOOLEAN_TRUE_TOKENS].join(' '); + const falseTokens = [...IMPORT_BOOLEAN_FALSE_TOKENS].join(' '); + + return fields.map((field) => { + const def = defs.get(field); + const meta = metaMap.get(field); + const type = meta?.type ?? ''; + const required = isTemplateRequired(def); + const label = meta?.label?.trim() || field; + const column: TemplateColumn = { + field, + header: required ? `${label} *` : label, + type, + required, + howToFill: text.text, + example: undefined, + textFormat: true, + }; + if (!def || !meta) { + column.howToFill = text.unknownField; + return column; + } + const multi = isMultiValueField({ type, multiple: meta.multiple }); + + if (NUMERIC_VALUE_TYPES.has(type)) { + const hasRange = typeof def.min === 'number' || typeof def.max === 'number'; + column.howToFill = text.number + (hasRange + ? text.range(typeof def.min === 'number' ? def.min : undefined, typeof def.max === 'number' ? def.max : undefined) + : ''); + column.example = numberExample(def); + column.textFormat = false; + } else if (BOOLEAN_VALUE_TYPES.has(type)) { + column.howToFill = text.boolean(trueTokens, falseTokens); + column.example = text.booleanWords[0]; + column.dropdown = text.booleanWords; + } else if (SINGLE_OPTION_TYPES.has(type) || MULTI_OPTION_TYPES.has(type)) { + const labels = optionLabels(meta); + if (multi) { + column.howToFill = labels.length > 0 ? text.multiOption(labels.join(', ')) : text.freeMulti; + column.example = labels.length > 0 ? labels.slice(0, 2).join(text.multiJoiner) : undefined; + } else { + column.howToFill = text.singleOption(labels.join(', ')); + column.example = labels[0]; + if (labels.length > 0) column.dropdown = labels; + } + } else if (TIME_OF_DAY_TYPES.has(type)) { + column.howToFill = type === 'date' ? text.date : type === 'datetime' ? text.datetime : text.time; + column.example = type === 'date' ? C.date.accepted[0][0] + : type === 'datetime' ? C.datetime.naive[0] + : C.time.accepted[0][0]; + column.textFormat = false; + } else if (IMPORT_REFERENCE_TYPES.has(type)) { + const target = meta.reference ?? ''; + const targetLabel = opts.referenceLabels?.get(target) || target; + column.howToFill = multi ? text.multiReference(targetLabel) : text.reference(targetLabel); + } else if (FILE_REFERENCE_TYPES.has(type)) { + column.howToFill = multi ? text.multiFile : text.file; + } else if (SAMPLE_TEXT_TYPES.has(type)) { + column.example = text.sampleText; + } else if (type in FORMATTED_TEXT_EXAMPLES) { + column.howToFill = type === 'email' ? text.email : type === 'url' ? text.url : text.phone; + column.example = FORMATTED_TEXT_EXAMPLES[type]; + } + return column; + }); +} + +// ── the workbook ──────────────────────────────────────────────────── + +/** + * The first row the dropdowns cover is the example row; the last is this many + * rows below the header. It matches the import job's row cap, so a template + * filled to what the import will take has a dropdown on every row. + */ +export const TEMPLATE_VALIDATED_ROWS = 50_000; + +/** The data validation a worksheet cell carries, read off exceljs' own `Cell` type. */ +type TemplateDataValidation = ReturnType['dataValidation']; + +/** + * exceljs' worksheet-level range validations. + * + * `Worksheet` constructs `this.dataValidations = new DataValidations()` + * (`exceljs/lib/doc/worksheet.js`), and the sheet writer emits a range-keyed + * entry as ONE `` — but `exceljs@4.4.0`'s + * published `index.d.ts` comments the member out. The per-cell alternative + * (`getCell(...).dataValidation`) would create an empty cell for every covered + * row, which a spreadsheet then reports as 50 000 used rows. So the member is + * reached through this one assertion, and a runtime without it fails loudly + * instead of shipping a template whose dropdowns silently vanished. + */ +function rangeValidations(ws: Worksheet): { add(address: string, validation: TemplateDataValidation): unknown } { + const dv = (ws as unknown as { dataValidations?: { add?: unknown } }).dataValidations; + if (!dv || typeof dv.add !== 'function') { + throw new Error('exceljs no longer exposes Worksheet.dataValidations; the import template cannot declare its dropdowns'); + } + return dv as { add(address: string, validation: TemplateDataValidation): unknown }; +} + +/** Column letters for a 1-based column index (1 → A, 27 → AA). */ +export function columnLetter(index: number): string { + let n = index; + let out = ''; + while (n > 0) { + const r = (n - 1) % 26; + out = String.fromCharCode(65 + r) + out; + n = Math.floor((n - 1) / 26); + } + return out; +} + +/** Where the instructions sheet's field table starts, and where its dropdown lists start. */ +const INSTRUCTIONS_TABLE_HEADER_ROW = 4; +const DROPDOWN_FIRST_COLUMN = 7; + +export interface BuildTemplateOptions { + locale?: string; + /** `readable` adds the note that states the soft-fail. */ + projection?: TemplateProjectionSource; +} + +/** + * Build the template workbook. The first sheet is the template — the import + * reader reads the first worksheet when no `sheet` is named — with the header + * row and one example row; the second is the instructions, whose columns to + * the right of the field table double as the dropdowns' source ranges (an + * inline list is capped at 255 characters, a range is not). + */ +export async function buildImportTemplateWorkbook( + columns: readonly TemplateColumn[], + opts: BuildTemplateOptions = {}, +): Promise { + const text = templateText(opts.locale); + const ExcelJS = await loadExcelJs(); + const wb = new ExcelJS.Workbook(); + const sheet = wb.addWorksheet(text.templateSheet, { views: [{ state: 'frozen', ySplit: 1 }] }); + const guide = wb.addWorksheet(text.instructionsSheet); + + // Template: header + example row. + const header = sheet.addRow(columns.map((c) => c.header)); + header.font = { bold: true }; + const example = sheet.addRow(columns.map((c) => (c.example === undefined ? null : c.example))); + example.font = { italic: true, color: { argb: 'FF808080' } }; + columns.forEach((c, i) => { + const col = sheet.getColumn(i + 1); + col.width = Math.min(Math.max(c.header.length + 4, 12), 40); + if (c.textFormat) col.numFmt = '@'; + }); + + // Instructions: notes, then one row per column. + const notes = opts.projection === 'readable' ? [...text.notes, text.readProjectionNote] : text.notes; + notes.forEach((note, i) => { guide.getCell(i + 1, 1).value = note; }); + const headingRow = guide.getRow(INSTRUCTIONS_TABLE_HEADER_ROW); + text.headings.forEach((h, i) => { headingRow.getCell(i + 1).value = h; }); + headingRow.font = { bold: true }; + columns.forEach((c, i) => { + const row = guide.getRow(INSTRUCTIONS_TABLE_HEADER_ROW + 1 + i); + row.getCell(1).value = c.header; + row.getCell(2).value = c.field; + row.getCell(3).value = c.type; + row.getCell(4).value = c.required ? text.required : text.optional; + row.getCell(5).value = c.howToFill; + row.getCell(5).alignment = { wrapText: true, vertical: 'top' }; + }); + guide.getColumn(1).width = 24; + guide.getColumn(2).width = 24; + guide.getColumn(3).width = 14; + guide.getColumn(4).width = 10; + guide.getColumn(5).width = 80; + + // Dropdowns: each list in its own column of the instructions sheet, and a + // list validation over the template column that references it. + const validations = rangeValidations(sheet); + const quotedGuide = `'${text.instructionsSheet.replace(/'/g, "''")}'`; + let listColumn = DROPDOWN_FIRST_COLUMN; + columns.forEach((c, i) => { + if (!c.dropdown || c.dropdown.length === 0) return; + const letter = columnLetter(listColumn); + guide.getCell(INSTRUCTIONS_TABLE_HEADER_ROW, listColumn).value = c.header; + guide.getCell(INSTRUCTIONS_TABLE_HEADER_ROW, listColumn).font = { bold: true }; + c.dropdown.forEach((v, j) => { guide.getCell(INSTRUCTIONS_TABLE_HEADER_ROW + 1 + j, listColumn).value = v; }); + const first = INSTRUCTIONS_TABLE_HEADER_ROW + 1; + const last = INSTRUCTIONS_TABLE_HEADER_ROW + c.dropdown.length; + const target = columnLetter(i + 1); + validations.add(`${target}2:${target}${TEMPLATE_VALIDATED_ROWS + 1}`, { + type: 'list', + allowBlank: true, + formulae: [`${quotedGuide}!$${letter}$${first}:$${letter}$${last}`], + showErrorMessage: true, + // `warning`, not `stop`: the reader also takes an option's code and + // every boolean token, so a value outside the list may still be right. + errorStyle: 'warning', + errorTitle: c.header, + error: c.howToFill.slice(0, 255), + }); + listColumn += 1; + }); + + return wb; +} diff --git a/packages/rest/src/rest-server-closed-query-params.test.ts b/packages/rest/src/rest-server-closed-query-params.test.ts index e713bc7114b..91cd6656aaf 100644 --- a/packages/rest/src/rest-server-closed-query-params.test.ts +++ b/packages/rest/src/rest-server-closed-query-params.test.ts @@ -254,7 +254,7 @@ describe('#7606 §1 — GET /data/:object/:id refuses what it would have dropped }); // ───────────────────────────────────────────────────────────────────────────── -// 2. GET /data/:object/export — closed set of ten, one of them invisible +// 2. GET /data/:object/export — closed set of eleven, one of them invisible // ───────────────────────────────────────────────────────────────────────────── describe('#7606 §2 — GET /data/:object/export', () => { @@ -352,10 +352,16 @@ describe('#7606 §2 — GET /data/:object/export', () => { page: '500', header: 'true', format: 'csv', + // [#18386] `template=true` switches the door to the xlsx import + // template, which refuses a csv format beside it (a DIFFERENT 400 + // from recognition), so this one name is sent without the csv + // baseline below. + template: 'true', }; for (const name of DATA_EXPORT_PARAMS) { const { exportRows } = boot(); - const answer = await exportRows({ format: 'csv', [name]: validValue[name] ?? 'title' }); + const baseline = name === 'template' ? {} : { format: 'csv' }; + const answer = await exportRows({ ...baseline, [name]: validValue[name] ?? 'title' }); expect( answer.status, `"${name}" is declared supported but was refused: ` diff --git a/packages/rest/src/rest-server.ts b/packages/rest/src/rest-server.ts index d2b761d1b0b..87703e9b74c 100644 --- a/packages/rest/src/rest-server.ts +++ b/packages/rest/src/rest-server.ts @@ -354,6 +354,17 @@ import { prepareImportRequest } from './import-prepare.js'; // measurement that decides its shape. import { datasetSelectionRefusal } from './analytics-selection-door.js'; import { loadExcelJs, type Worksheet } from './xlsx-module.js'; +// [#18386] `?template=true` on the export door: the import template's column +// rule, request reading and workbook. See the module header. +import { + buildImportTemplateWorkbook, + describeTemplateColumns, + readTemplateMode, + resolveTemplateProjection, + templateColumns, + templateText, + type TemplateProjectionSource, +} from './import-template.js'; import { enrichOpenApiWithEndpoints } from './openapi-endpoints.js'; import { buildBuiltinPaths } from './openapi-builtin-paths.js'; import { @@ -629,6 +640,11 @@ export const DATA_RECORD_READ_PARAMS: readonly string[] = ['select', 'expand']; * a loud export outage — the preservation half of * `rest-server-closed-query-params.test.ts` exists to make that impossible to * land, and pins `locale` by name for the reason above. + * + * [#18386] …and `template`, the mode switch: `template=true` answers an xlsx + * IMPORT template (`./import-template.ts`) instead of the data. It is read by + * `readTemplateMode`, which also refuses the row parameters above on a + * template request, since a template has no rows for them to select. */ export const DATA_EXPORT_PARAMS: readonly string[] = [ 'format', 'header', @@ -636,6 +652,7 @@ export const DATA_EXPORT_PARAMS: readonly string[] = [ 'filter', 'search', 'searchFields', 'orderby', 'fields', 'locale', + 'template', ]; /** @@ -9571,6 +9588,8 @@ export class RestServer { // header=false (omit the header row for csv / xlsx; default true) // limit= (default 10000, hard cap 50000) // page= (driver chunk size, default 500, max 5000) + // template=true (an xlsx IMPORT template instead of the data — see + // `answerImportTemplate`; `false` or absent is the export) // // Values are formatted for readability from the object schema: lookup / // user fields resolve to a name (via injected $expand), select fields to @@ -9584,7 +9603,7 @@ export class RestServer { // // A zero-row result still emits the header row when the column set is // authoritative (the security service's readable projection, or an explicit - // `fields=`), so an empty export doubles as an import template. Without a + // `fields=`). The import template is `template=true`, not this. Without a // projection it stays headerless, so FLS-hidden column names never leak. // // Streams the response so 50k-row exports do not buffer in memory; the @@ -9639,8 +9658,21 @@ export class RestServer { // and still answers what it answered before. if (refuseUnknownQueryParams(req, res, DATA_EXPORT_PARAMS)) return; if (refuseRepeatedQueryParams(req, res, - ['format', 'header', 'limit', 'page', 'filter', 'search', 'orderby'])) return; + ['format', 'header', 'limit', 'page', 'filter', 'search', 'orderby', 'template'])) return; const q = req.query ?? {}; + // [#18386] `?template=true` answers the import template and + // returns before a single export header is set, so without it + // everything below runs exactly as it did before the mode + // existed. + const templateMode = readTemplateMode(q); + if (templateMode.kind === 'refused') { + res.status(400).json({ error: { code: 'VALIDATION_ERROR', message: templateMode.message } }); + return; + } + if (templateMode.kind === 'template') { + await this.answerImportTemplate(req, res, p, environmentId, objectName, context, q); + return; + } const fmtRaw = String(q.format ?? 'csv').toLowerCase(); const format: 'csv' | 'json' | 'xlsx' = fmtRaw === 'json' ? 'json' : fmtRaw === 'xlsx' ? 'xlsx' : 'csv'; @@ -10004,6 +10036,119 @@ export class RestServer { }); } + /** + * [#18386] `GET {basePath}/data/:object/export?template=true` — the IMPORT + * template: an xlsx workbook with a header row of the columns an import can + * write, one example row, dropdowns for the closed value domains, and an + * instructions sheet. No data is read. + * + * It runs behind the export door's two gates, unchanged — the object's + * `export` exposure ({@link enforceApiAccess}) and the caller's export + * permission ({@link enforceExportPermission}) — and after the query-string + * gates, so it is reached only by a request the export would have served. + * + * Columns: an explicit `?fields=` is honoured as asked; otherwise + * `templateColumns` over the object as this caller reads it, narrowed by + * `resolveTemplateProjection` — the security service's WRITE projection, + * or its read projection when it has none, which `X-Export-Template-Projection` + * and a note on the instructions sheet then state. A security service that + * is present but answers neither fails the request rather than answering an + * unnarrowed header. + */ + private async answerImportTemplate( + req: any, + res: any, + p: RestProtocol, + environmentId: string | undefined, + objectName: string, + context: any, + q: Record, + ): Promise { + let explicitFields: string[] | undefined; + if (typeof q.fields === 'string' && q.fields.length > 0) { + explicitFields = q.fields.split(',').map((s: string) => s.trim()).filter(Boolean); + } else if (Array.isArray(q.fields)) { + explicitFields = q.fields.filter((s: any) => typeof s === 'string' && s.length > 0); + } + + // The object as the export reads it (registry first, `getObjectSchema` + // as the last resort), localized to the request — but NOT best-effort: + // a template without the schema has no columns to offer. + let schema: any = undefined; + if (typeof (p as any).getMetaItem === 'function') { + const found: any = await (p as any).getMetaItem({ type: 'object', name: objectName }); + schema = found?.item; + } + if (!schema && typeof (p as any).getObjectSchema === 'function') { + schema = await (p as any).getObjectSchema(objectName, environmentId); + } + if (!schema || typeof schema !== 'object') { + const missing: any = new Error(`Object '${objectName}' was not found, so it has no import template.`); + missing.code = 'OBJECT_NOT_FOUND'; + missing.status = 404; + throw missing; + } + schema = await this.translateMetaItem(req, 'object', environmentId, schema); + + let permitted: ReadonlySet | undefined; + let projection: TemplateProjectionSource = 'none'; + if (!explicitFields || explicitFields.length === 0) { + const security = await this.resolveSecurityService(environmentId, req); + const answer = await resolveTemplateProjection(security, objectName, context); + if (answer.source === 'unanswered') { + // Declared 5xx: a fault, sanitised and logged — never read + // as "no such object" by the message heuristics. + throw Object.assign( + new Error('The security service gave no field projection, so the import template ' + + 'cannot tell which columns this caller may write.'), + { status: 500, code: 'INTERNAL_ERROR' }, + ); + } + if (answer.source !== 'none') permitted = answer.permitted; + projection = answer.source; + } + const fields = templateColumns(schema, { explicitFields, permitted }); + + // A reference column names the object it points at by that object's + // label, when it can be read; otherwise by its name. + const referenceLabels = new Map(); + const metaMap = buildFieldMetaMap(schema); + for (const f of fields) { + const target = metaMap.get(f)?.reference; + if (!target || referenceLabels.has(target)) continue; + let label = target; + try { + const found: any = typeof (p as any).getMetaItem === 'function' + ? await (p as any).getMetaItem({ type: 'object', name: target }) + : undefined; + const translated: any = found?.item + ? await this.translateMetaItem(req, 'object', environmentId, found.item) + : undefined; + if (typeof translated?.label === 'string' && translated.label.trim().length > 0) label = translated.label; + } catch { /* the target's name stands in for its label */ } + referenceLabels.set(target, label); + } + + const i18n = await this.resolveI18nService(environmentId, req).catch(() => undefined); + const locale = this.extractLocale(req, i18n); + const columns = describeTemplateColumns(schema, fields, { locale, referenceLabels }); + const workbook = await buildImportTemplateWorkbook(columns, { locale, projection }); + const bytes = Buffer.from(await workbook.xlsx.writeBuffer()); + + const timezone = typeof context?.timezone === 'string' && context.timezone ? String(context.timezone) : undefined; + const objectLabel = typeof schema.label === 'string' && schema.label.length > 0 ? schema.label : objectName; + res.header('Content-Type', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'); + res.header('Content-Disposition', exportContentDisposition( + `${objectName}-template`, `${objectLabel}-${templateText(locale).filenameSuffix}`, 'xlsx', timezone, + )); + res.header('X-Export-Format', 'xlsx'); + res.header('X-Export-Template', 'true'); + res.header('X-Export-Template-Projection', projection); + res.header('Cache-Control', 'no-store'); + res.write(bytes); + res.end(); + } + /** * [#3547] Resolve the environment's `security` service — the ENVIRONMENT's * kernel service first (its evaluator / FieldMasker are bound to that diff --git a/packages/spec/src/contracts/security-service.test.ts b/packages/spec/src/contracts/security-service.test.ts index 6033f76ef5e..55522ea2e2f 100644 --- a/packages/spec/src/contracts/security-service.test.ts +++ b/packages/spec/src/contracts/security-service.test.ts @@ -231,6 +231,30 @@ describe('Security Service Contract', () => { await expect(nothingDisclosable.getMetadataReadableFields?.('deal', { userId: 'u1' })).resolves.toEqual([]); }); + it('getWritableFields is OPTIONAL — absence degrades to getReadableFields, and the two answers differ', async () => { + const withoutIt: ISecurityService = makeService({ getReadableFields: async () => ['id', 'name', 'locked'] }); + expect(typeof withoutIt.getWritableFields).toBe('undefined'); + const mustNotCompileWithoutAGuard = () => + // @ts-expect-error possibly undefined — a consumer must feature-detect first + withoutIt.getWritableFields('deal', { userId: 'u1' }); + expect(typeof mustNotCompileWithoutAGuard).toBe('function'); + + // A field the caller reads but may not edit is in the read projection and + // not in the write one — which is why a fallback must be stated. + const withIt = makeService({ + getReadableFields: async () => ['id', 'name', 'locked'], + getWritableFields: async (_object, context) => (context?.isSystem ? ['id', 'name', 'locked'] : ['name']), + }); + await expect(withIt.getWritableFields?.('deal', { userId: 'u1' })).resolves.toEqual(['name']); + await expect(withIt.getWritableFields?.('deal', { isSystem: true })).resolves.toEqual(['id', 'name', 'locked']); + + // The same two empty answers as the read side. + await expect(makeService({ getWritableFields: async () => undefined }).getWritableFields?.('deal', {})) + .resolves.toBeUndefined(); + await expect(makeService({ getWritableFields: async () => [] }).getWritableFields?.('deal', {})) + .resolves.toEqual([]); + }); + it('[#7616] resolvePermissionSetsForContext is OPTIONAL — absence keeps the consumer on its own resolution (compile-time)', () => { // THE structural pin behind "a consumer must keep its local resolution as // the fallback until a floor version carrying this method can be assumed". diff --git a/packages/spec/src/contracts/security-service.ts b/packages/spec/src/contracts/security-service.ts index 3d6a01c902c..6fcd579ba0a 100644 --- a/packages/spec/src/contracts/security-service.ts +++ b/packages/spec/src/contracts/security-service.ts @@ -35,7 +35,8 @@ * "no answer — use your own fallback", NOT "no fields are readable". An empty * array is a real answer and means the opposite: nothing is readable. Its * metadata-plane sibling {@link ISecurityService.getMetadataReadableFields} - * (ADR-0106 D7) reads the same two empty answers the same way. + * (ADR-0106 D7) and its write-side twin {@link ISecurityService.getWritableFields} + * read the same two empty answers the same way. * - **Verdicts fail to ABSTENTION.** {@link ISecurityService.checkAuthoredRowWrite} * answers a question a composing caller may use to WIDEN, so its failure mode * is the one that changes nothing: `abstain`. It never reports `admit` for a @@ -341,6 +342,33 @@ export interface ISecurityService { */ getMetadataReadableFields?(object: string, context?: SecurityContext): Promise; + /** + * The field names `context` may WRITE on `object` as far as field-level + * security decides — the write-side twin of {@link getReadableFields}. + * + * Computed from schema + context by the same resolution as the write path's + * field-level-security gate: the returned set is the exact complement of the + * fields that gate refuses when a payload names them. Neither whether the + * caller may create or edit the OBJECT nor a field's own rules (`readonly`, + * `system`, a `formula` / `summary` / `autonumber` type) are part of the answer. + * + * **Fails SOFT, with the same two distinct empty answers as + * {@link getReadableFields}:** `undefined` is "no answer — use your own + * fallback" (e.g. the object schema could not be resolved); `[]` is the real + * answer that this caller may write NO field. A system context bypasses and + * yields the full field set. + * + * **OPTIONAL, and absence is a defined state — not a bug.** A security service + * that predates it omits it; consumers feature-detect + * (`typeof svc.getWritableFields === 'function'`) and fall back to + * {@link getReadableFields}. That fallback is not the same answer — a field + * the caller may read but not edit is in it, and a write naming that field is + * refused — so a consumer that falls back STATES in its response that it + * narrowed by the read projection, ⛔ never presents it as the write one. + * Nothing is written past the gate either way: the write path still refuses. + */ + getWritableFields?(object: string, context?: SecurityContext): Promise; + /** * The effective permission-set NAMES for `context` — positions expanded and * the additive baseline applied, i.e. the same set the middleware enforces