feat(map-ai): generate SVG edit proposals for Concierge (PPT-2832) by camreeves · Pull Request #453 · PlaceOS/rest-api

camreeves · 2026-10-13T13:54:49Z

Add maps/ai endpoints for capability discovery, edit jobs, polling and usage. The server sends bounded SVG context to OpenAI-compatible or Anthropic text models and returns validated edit operations; the browser applies them as a reviewable preview.

Reuse provider credentials and job storage with separate map quotas. Support Anthropic workspace headers and tool-input proposals. Validate references to newly created elements and counted duplicate IDs. Prompt guidance covers readable labels, collision avoidance and furniture grouping.

Validation: 57 focused adapter/prompt/controller specs pass; five distinct live browser prompts and two corrective retries completed without final validation issues. API documentation regenerated; final container image builds successfully. The review branch pins the companion models branch and uses its migrations in the spec harness; remove that temporary pin after the models change lands.

Draft for review and further end-to-end testing. No merge or deployment requested.

Companion frontend: PlaceOS/user-interfaces#570

Companion drafts: Frontend #570, API #453, Models #334.

PPT-2833. AI map editing reuses the signage AI provider rows and job table with a text/vision adapter, so the pieces both adapter families need move out of ImageGen::Adapter: credentials, credential and vendor_message become Utils::VendorCredentials, and the job long poll and same-domain job lookup become Utils::AIJobs so the map AI controller can include them rather than copy them. SignageAI's behaviour is unchanged. The spec helpers setup_signage_ai, signage_group and await_signage_ai_job move into spec/spec_helpers so a second controller spec can use them without requiring the signage spec file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

PPT-2833. The AI map editor spends vendor calls on behalf of whoever may save the level's map, so the zone update gate is extracted into Utils::GroupPermissions#zone_updatable?: support or admin, the legacy org-zone ACL, or a support-subsystem Update on the zone. The broad signage subsystem grant that Zones#update also honours is left out of it. Zones#check_update_permissions keeps its behaviour exactly; its private record-form legacy check now delegates to the shared zone_legacy_access?. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

PPT-2833. A text/vision call against a signage AI provider row for map editing. The editor posts the context it built from the level's SVG (element index, shortlisted subtrees, annotations, a marked up snapshot by upload id) and the model answers with a closed vocabulary of edit operations that the editor applies itself; nothing here reads or writes the map. - Llm::Schema defines the vocabulary once and renders it in the OpenAI strict dialect; the TypeScript side in libs/map-editor mirrors it field for field. - Llm::Prompt orders the request so the stable front (rules, op reference, index, subtrees) is identical across the turns of one edit and the instruction comes last, with byte caps per section and the parent chain's instruction, ops and resolution for a refine. - Llm::Ops parses the answer (markdown fence tolerated) and checks each op against the posted index; failing ops are kept with server_issue so the editor can show them unchecked with the reason. - Adapters::OpenAIChat posts chat completions with response_format json_schema strict, max_completion_tokens and reasoning_effort (dropped and resent on a 400 naming it), for OpenAI and both Azure URL styles; refusal and content_filter read as Moderated, 429 as Busy, IO::TimeoutError as Error::ImageGen::Timeout. No temperature: the gpt-5 family rejects it. - Llm::Runner is the one-call version of the image runner: Running, markup fetch, prompt, one vendor call under one Llm slot (sized by MAP_AI_MAX_CALLS, apart from the image slots), result written with bump_version, cost_units = total tokens, invalid_response for an answer that is not JSON or was cut off. - Error::ImageGen::InvalidResponse (502) and the MAP_AI_* constants. - The markup snapshot tag ai-map-markup, reaped by the existing sweep. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

PPT-2833. /api/engine/v2/maps/ai: capabilities (providers that can edit maps, the default, the map quota, the limits), edit (202, the job), jobs/:id long poll, jobs index (kind, mine, zone_id), cancel, resolve (what the person did with the answer, written into result.resolution), usage (support, map edits only). The edit gate is "may update this zone": support or admin, the legacy org-zone ACL, or a support-subsystem Update, without the broad signage grant. A non-support caller sees 404 for a zone outside the authority's org zone tree. Quotas are counted by kind against map_user_per_day and map_domain_per_month (defaults 40 and 1000). The context is posted inline under MAP_AI_MAX_CONTEXT_BYTES; the row keeps its size and sha256, never the body. The markup upload must be the caller's own png, jpeg or webp under the image size cap, and is tagged for the sweep when it was uploaded for this request. Provider rows gain text_model (create, update, clear) and POST /:id/test with kind=text runs a five-token chat completion under an Llm slot. The signage jobs index takes a kind filter defaulting to the image kinds, so map jobs stay out of "recent generations". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

PPT-2833. Chat completions stubs keyed on the /v1/chat/completions path so the image stub (which matches every api.openai.com POST, and WebMock uses the first registered match) cannot swallow them: a verbatim answer with the requests it received, a refusal, a timeout, prose, and a 400 naming reasoning_effort followed by the answer. map_ai_spec covers capabilities (no provider, disabled, Vertex only, on with the text model and the map quota), the edit path end to end with what reached the vendor, the markup snapshot as an image part in the right position, fenced JSON, prose and a cut-off answer as invalid_response, a flagged unknown anchor, refusal, timeout, the reasoning_effort retry, the map quota by kind, the read-only scope, the concierge org-zone ACL, a caller who cannot update the zone, the broad signage grant, a zone outside the organisation, the context caps, idempotency replay, the parent chain in the prompt, an image job as parent, the jobs index and its kind filter on both controllers, long poll and cancel, resolve, and usage by kind. llm_prompt_spec covers the message order, the caps, the parent chain, the strict rendering of the schema, and every check in Llm::Ops. The storage GET stub now answers with a body_io: WebMock hands a stubbed response to Http.get_bytes's streaming block as-is, so with a body string Store.fetch always failed under WebMock and no spec had proven the source, reference or markup fetch path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

PPT-2833. `crystal build ./src/app.cr && ./app -d > OPENAPI_DOC.yml`, run locally against the models branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The vendor is now sent libs/map-editor's ops-schema.json as it is: one anyOf variant per op, set_style_rule.declarations and insert_shape.attrs as {name, value} arrays, schema_version in the envelope, $defs for target and name_value. The Crystal rendering that was sent before was a flat 26-field object with declarations as a string, so nothing the model returned could pass the editor's own validator. The file is embedded at compile time and the vocabulary (ops, fields, shapes, positions) is read from it, so the two copies cannot drift apart silently; the spec asserts the rendering against the file. Both copies carry sha256 ff41e284fc4e145587baa7227f60e01b16dfc08ca59db14ff5184385f0b2bbd5. Ops are stored exactly as the model returned them. The server's bookkeeping moves off the op: result.server_issues maps op-N (N the 1-based position) to the reason, because the editor's schemaIssues rejects any key outside the variant and the plan's inline server_issue and id would have locked every row. The checks follow the editor's rules: attribute, style and geometry values are syntax-checked (value-syntax.ts ported, named colours included), declarations are arrays, insert_shape attributes are per kind with transform refused, the deny-list for replacement markup is sanitise.ts's list plus <style> and <use>, <a> and data: images are allowed, url(#fragment) is allowed, every element must be in the SVG namespace, a backslash in CSS text is refused since browsers decode escapes past a literal match, a transform with every field null is refused, duplicate needs translate and a whole count, and a new id used twice in one answer is flagged. A finish_reason of length is retried once at MAP_AI_RETRY_OUTPUT_TOKENS before the job fails as invalid_response; tokens are summed over the attempts and usage carries the attempt count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… by kind JobResponse carries server_issues beside the untouched ops. The parent chain is read before the slot is reserved and the row written, so a failure there can no longer leave a QUEUED row the stale sweep never touches with a process-local slot held. An idempotency key already used by an image job answers 409 instead of replaying that image job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The generate/edit gate and remaining_quota summed every kind, so map edits consumed the image allowance. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A length finish with empty content, which a reasoning model produces when the budget is spent before the answer, reported ok. The probe now sends the same low reasoning effort as an edit, allows 128 output tokens and reports invalid_response when nothing came back. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The prompt tells the model that a room or desk name in the instruction identifies the element whose id carries its number, that outlined labels are glyph elements inside the room they name, how to rename one (delete the glyphs, insert a text element), and to act on the best-supported reading instead of asking whenever an element carries the number. The index gains fill and inside columns, read from the context's fill and in fields; the fixture carries both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The prompt prints a LABELS section from the context's labels list (the element each label sits in, its glyph anchors in reading order, box and fill) and tells the model to rename an outlined label by deleting those anchors and inserting a text element sized from the label's box. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The prompt asks for the new text after the label's last glyph, y at the first line's baseline, and lines separated by newline characters, which the editor lays out as rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The index gains a piece column from the context's piece field, and the prompt tells the model that elements sharing a piece are the layers of one object, to be deleted, moved or copied together. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…d placement rules Adds an Anthropic Messages adapter: structured output through output_config.format, the effort from MAP_AI_ANTHROPIC_EFFORT, and fallbacks: "default" for the models whose classifiers may decline a request; a model that rejects either parameter is asked again without it. A refusal is a moderation failure, 429 and overloaded are busy. Capabilities now list the text models each provider is known to serve so the editor can offer a choice, and the job usage records the model that answered when the vendor says. The prompt gains the thirds of the map as regions, text labels with their words, outlined labels outside every room, and rules for renaming by words, extending a desk run, making a room bookable and an honest summary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Support API keys that require an explicit workspace and cover requests with and without a workspace ID. Co-authored-by: Codex <noreply@openai.com>

Anthropic rejects the nullable reorder-position enum. Send equivalent typed alternatives while preserving the shared editor and OpenAI schema. Cover the conversion and update stale prompt-index expectations for the existing context columns. Co-authored-by: Codex <noreply@openai.com>

Deduplicate nullable schemas through shared definitions to stay within the structured output union limit while preserving required fields and allowed values. Co-authored-by: Codex <noreply@openai.com>

Use strings for all target fields; empty strings already mean absent to both validators. Removing nested nullable target combinations reduces the compiled output grammar while retaining anchor, ID and hash validation. Co-authored-by: Codex <noreply@openai.com>

The operation grammar exceeds Anthropic structured-output compilation limits even with shared nullable definitions. Use the full operation schema as a non-strict client tool, request one proposal and validate its tool input before the existing operation checks. Automatic tool choice supports Fable 5.1 and Opus 5.5, which reject forced tool selection. Co-authored-by: Codex <noreply@openai.com>

Require readable labels after background edits and duplicate furniture within its destination group, rotating complete pieces as needed. Co-Authored-By: Codex <noreply@openai.com>

…opies

…t rules These changes were made on 14 Oct and ran in the rest-api sidecar for the local and mqt.placeos.run demos, but were never committed: - set_id renames update the index in proposal order, so later ops can target the new id and the old id becomes free - set_attr viewBox is allowed on the SVG root only, with four finite numbers - prompt rules for preserving label font weight, composited overlay contrast, referencing ids created earlier in the same proposal, translucent booking overlays, shared-wall doorways and bounded canvas expansion Co-authored-by: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With the proposal returned as a tool input, a response that stopped at max_tokens either had no tool_use block, which raised InvalidResponse, or a truncated one. Both skipped the runner's retry with a larger output budget. The adapter now sets finish to length before reading the tool input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The server accepted any positive viewBox on the root, while the editor rejects one that crops the old map or grows it beyond four times. The op checker now applies the same rule against the viewBox the editor posts in the context, so the job's server issues agree with what the editor will allow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The bounds line was written ahead of the ANNOTATIONS heading, so the markup part no longer started with its heading and map_ai_spec could not find it. It explains annotations drawn off the map, so it now sits under that heading and is left out when there are no annotations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ap adapters OpenAIImages and the map edit OpenAIChat adapter each carried the same base URL, Azure check, authentication headers and error mapping. They now include Utils::OpenAIVendor. The image adapter also treats Azure's content_filter error code as a moderation refusal, as the chat adapter already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rollers The map AI controller copied the signage AI controller's quota count, cancel action and recent-jobs query, differing only in the job kinds. They now live in Utils::AIJobs next to the shared long poll: remaining_ai_quota, request_job_cancel and recent_ai_jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

camreeves · 2026-10-15T12:54:19Z

Review pass on this PR, pushed to the branch as six commits (eb8899a..3450a35). Nothing existing was rewritten.

Fixes

422e1fb, uncommitted changes brought in. Sequential renames, root-only canvas expansion, and the prompt rules from the SD replays were edited on 14 Oct and ran in the sidecar for both demos, but were never committed. They're added unchanged, so this branch matches what was demoed.
0e35de3, a cut-off Anthropic answer now retries. Since proposals arrive as a tool input, a response that stopped at max_tokens either had no tool_use block, which raised InvalidResponse, or a truncated one. Either way the runner's larger-budget retry never ran. The adapter now reports length before reading the tool input. Covered in spec/llm_anthropic_spec.cr.
ee22167, the server checks canvas expansion like the editor. The op checker accepted any positive viewBox on the root. The editor rejects one that crops the old map or grows it past four times, and the server now applies that same rule against the context viewBox, so server issues agree with what the editor allows. Covered in spec/llm_prompt_spec.cr.
83c4edd, the snapshot bounds line moves under the ANNOTATIONS heading. The 14 Oct change put it ahead of the heading, which broke map_ai_spec ("sends the markup snapshot as an image part"). Only a focused subset of specs had been run. The line is also dropped when there are no annotations, since it only explains off-map marks.

Reuse of the signage AI code

Most of the groundwork was already shared: provider and job rows, the long poll, the vendor HTTP client, slots, the upload store, error types and credentials. Two places had been copied instead:

26cae42, Utils::OpenAIVendor. OpenAIImages and OpenAIChat carried the same base URL, Azure check, auth headers and error mapping. Both now include one module. The image adapter also gains Azure's content_filter code as a moderation error.
3450a35, more in Utils::AIJobs. The quota count, cancel action and recent-jobs query were copied from the signage controller with only the job kind changed. They are now remaining_ai_quota, request_job_cancel and recent_ai_jobs.

Verification: the full spec suite passes in CI on crystal latest and unstable, along with Ameba and style. Five live edits ran through a sidecar built from these commits. Screenshots and findings: https://gist.github.com/camreeves/f323d149ce0f83b245161ff6547e451d

Not changed, worth a look: the prompt has grown several paragraphs of scenario rules. They're validated against real maps, but should be trimmed once there's an eval set.

🤖 Generated with Claude Code

- shard.lock: placeos-models stays pinned to the models review branch, now at c687a35, which includes master's 9.119.5 (multiple root groups); the other shards take master's bumps - the signage AI spec helpers live in spec/spec_helpers/signage_ai.cr on this branch, so master's comment update to signage_group is applied there Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The runner wrote the final job state and only then released its slot in an ensure, so a client that saw a final job could still find the slot taken. map_ai_spec checks exactly that and failed intermittently. The slot now goes back before the final write on every path, as the image runner already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

camreeves · 2026-10-15T13:33:40Z

Merge-readiness pass (Steve reported clashes). This PR is now mergeable with master.

b7981c0, merged master. It had two conflicts:
- shard.lock: this branch pins placeos-models to the models review branch, and master moved to 9.119.5. The pin now points at a models commit that includes 9.119.5. Master's other shard bumps are taken as they are.
- spec/controllers/signage/signage_ai_spec.cr: this branch moved the signage AI spec helpers to spec/spec_helpers/signage_ai.cr (12 Oct), and master then edited a comment in the old copy. The edit is applied to the shared file instead.
558fa2a, repinned models after the migration renumbering in models : ensure the system is valid and exists #334. A duplicate migration version had made master's multiple-roots migration get skipped, which failed groups_spec here too. Details on models : ensure the system is valid and exists #334.
5a8afee, slot released before the job is final. The map AI runner wrote the final job state and only then released its call slot, unlike the image runner. map_ai_spec checks the slot as soon as the job is final, and failed intermittently on that race. The order now matches the image runner.

CI: 1,258 examples on crystal latest and unstable. Seven MCP / ChatGPT plugin specs fail, and master fails the same seven since 760c004 ("chore: update shards", 15 Oct). They don't come from this branch.

Before merging this PR (models #334 merged and released first). Codex added three temporary pins in 3623b2d so CI could run against the unreleased models branch, and they must come out:

shard.override.yml: remove the placeos-models block that points at branch: PPT-2832-ai-map-review.
shard.lock: set placeos-models to the released version (run shards update placeos-models).
spec/migration/Dockerfile: drop --branch PPT-2832-ai-map-review from the models clone.

These were the right call for a draft stacked on an unreleased models change. They're just easy to forget at merge time.

🤖 Generated with Claude Code

Claude sometimes JSON-encodes a nested tool input parameter, so the proposal arrived with ops as a string and the job failed with "the model's answer has no ops list". The parser now accepts a string holding a JSON array for ops and questions, and logs the answer's field shapes when there is still no ops list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The AI map editing models changes shipped in models 9.120.0, so the branch pins on the review branch go: the shard override, the overridden lock entry and the migration image's --branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

camreeves and others added 27 commits October 12, 2026 19:33

chore(docs): regenerate OPENAPI_DOC.yml for the maps/ai routes

5004284

PPT-2833. `crystal build ./src/app.cr && ./app -d > OPENAPI_DOC.yml`, run locally against the models branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(signage-ai): count only the image kinds against the image allowance

1ab0b59

The generate/edit gate and remaining_quota summed every kind, so map edits consumed the image allowance. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(map-ai): comments explain rather than justify

cd477c9

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(docs): regenerate OPENAPI_DOC.yml for server_issues

e317972

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(map-ai): send configured Anthropic workspace header

deac7c5

Support API keys that require an explicit workspace and cover requests with and without a workspace ID. Co-authored-by: Codex <noreply@openai.com>

fix(map-ai): share nullable Anthropic schema definitions

0957e50

Deduplicate nullable schemas through shared definitions to stay within the structured output union limit while preserving required fields and allowed values. Co-authored-by: Codex <noreply@openai.com>

fix(map-ai): preserve label contrast and furniture grouping

e32c10a

Require readable labels after background edits and duplicate furniture within its destination group, rotating complete pieces as needed. Co-Authored-By: Codex <noreply@openai.com>

fix(map-ai): resolve generated targets and avoid overlapping labels

948e972

fix(map-ai): validate all explicit duplicate ids before registering c…

3e89c9c

…opies

Merge remote-tracking branch 'origin/master' into PPT-2832-ai-map-review

5f326e8

build(map-ai): pin review models dependency and update API docs

3623b2d

github-actions Bot assigned camreeves Oct 13, 2026

github-actions Bot added the type: enhancement new feature or request label Oct 13, 2026

style(map-ai): satisfy Crystal test lint checks

f49b127

camreeves and others added 3 commits October 15, 2026 23:07

camreeves mentioned this pull request Oct 15, 2026

fix(map-ai): review fixes and shared AI plumbing for the map editing backend #455

Merged

camreeves and others added 3 commits October 15, 2026 23:30

github-actions Bot added type: enhancement new feature or request and removed type: enhancement new feature or request labels Oct 15, 2026

github-actions Bot removed the type: enhancement new feature or request label Oct 15, 2026

github-actions Bot added type: enhancement new feature or request and removed type: enhancement new feature or request labels Oct 15, 2026

chore: pin the models review branch with the renumbered migrations

558fa2a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added type: enhancement new feature or request and removed type: enhancement new feature or request labels Oct 15, 2026

github-actions Bot added type: enhancement new feature or request and removed type: enhancement new feature or request labels Oct 19, 2026

merge: update the AI map editing branch with master

b59aac7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added type: enhancement new feature or request and removed type: enhancement new feature or request labels Oct 19, 2026

camreeves marked this pull request as ready for review October 19, 2026 04:32

camreeves merged commit 0a13701 into master Oct 19, 2026
9 of 11 checks passed

camreeves deleted the PPT-2832-ai-map-review branch October 19, 2026 04:32

Navigation Menu

Repository navigation

feat(map-ai): generate SVG edit proposals for Concierge (PPT-2832) - #453

feat(map-ai): generate SVG edit proposals for Concierge (PPT-2832)#453
camreeves merged 41 commits into
masterfrom
PPT-2832-ai-map-review

camreeves commented Oct 13, 2026 •

edited

Loading

Uh oh!

camreeves commented Oct 15, 2026

Uh oh!

camreeves commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

camreeves commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

camreeves commented Oct 15, 2026

Uh oh!

camreeves commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

camreeves commented Oct 13, 2026 •

edited

Loading