From a64d7ea467bfa58d6ab57ad466a347f71ceb59b9 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Wed, 3 Jun 2026 17:17:21 -0700 Subject: [PATCH 01/25] chore: add SEP-2640 requirement-traceability YAML (Skills Extension) --- src/seps/sep-2640.yaml | 75 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 src/seps/sep-2640.yaml diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml new file mode 100644 index 00000000..e54ce694 --- /dev/null +++ b/src/seps/sep-2640.yaml @@ -0,0 +1,75 @@ +sep: 2640 +spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification +requirements: + - check: sep-2640-skillmd-required + text: 'Every skill MUST contain a `SKILL.md` file at its root.' + - check: sep-2640-skillmd-frontmatter + text: '`SKILL.md` MUST begin with YAML frontmatter containing at minimum the `name` and `description` fields as defined by the Agent Skills specification.' + - check: sep-2640-skill-uri-scheme + text: 'Each file within a skill directory is exposed as an MCP resource. Servers SHOULD use the `skill://` URI scheme, under which the resource URI has the form: `skill:///`' + - check: sep-2640-final-segment-equals-name + text: 'The final segment of `` MUST equal the skill''s `name` as declared in its `SKILL.md` frontmatter.' + - check: sep-2640-no-nested-skills + text: 'A `SKILL.md` MUST NOT appear in any descendant directory of a skill. The skill directory is the boundary; skills do not nest inside other skills.' + - check: sep-2640-name-naming-rules + text: 'The final `` segment, being the skill `name`, MUST satisfy the Agent Skills specification''s naming rules.' + - check: sep-2640-prefix-rfc3986 + text: 'Prefix segments SHOULD be valid URI path segments per RFC 3986; no further constraints are imposed on them.' + - check: sep-2640-skillmd-mimetype + text: 'For each `skill:///SKILL.md` resource: `mimeType` SHOULD be `text/markdown`.' + - check: sep-2640-skillmd-metadata-name + text: 'For each `skill:///SKILL.md` resource: `name` SHOULD be set from the `name` field of the `SKILL.md` YAML frontmatter. By the path constraint above, this will equal the final segment of ``.' + - check: sep-2640-skillmd-metadata-description + text: 'For each `skill:///SKILL.md` resource: `description` SHOULD be set from the `description` field of the `SKILL.md` YAML frontmatter.' + - check: sep-2640-meta-prefix + text: 'When `_meta` keys are used for skill resources, implementations SHOULD use the `io.modelcontextprotocol.skills/` reverse-domain prefix.' + - check: sep-2640-host-load-by-uri + text: 'hosts MUST support loading a skill given only its URI' + - check: sep-2640-server-expose-index + text: 'A server SHOULD expose a resource at the well-known URI `skill://index.json` whose content is a JSON index of the skills it serves.' + - check: sep-2640-index-entry-type-enum + text: '`skills[].type` MUST be `"skill-md"`, `"archive"`, or `"mcp-resource-template"`.' + - check: sep-2640-index-name-required + text: '`skills[].name` is Required for `"skill-md"` and `"archive"`; matches the `SKILL.md` frontmatter `name` and the final segment of the skill path. Omitted for `"mcp-resource-template"`.' + - check: sep-2640-index-digest-required + text: '`skills[].digest` is Required for `"skill-md"` and `"archive"`: SHA-256 content digest of the artifact, formatted as `sha256:{hex}` (64 lowercase hexadecimal characters). Omitted for `"mcp-resource-template"`.' + - check: sep-2640-client-ignore-unrecognized + text: 'Clients SHOULD ignore unrecognized fields and SHOULD skip entries with an unrecognized `type`.' + - check: sep-2640-archive-format + text: 'the archive MUST be `.tar.gz` (gzip-compressed tar, `mimeType` `application/gzip`) or `.zip` (`mimeType` `application/zip`)' + - check: sep-2640-host-support-archive-formats + text: 'hosts MUST support both `.tar.gz` and `.zip` archive formats' + - check: sep-2640-archive-skillmd-at-root + text: 'Archive contents represent the skill directory directly — `SKILL.md` MUST be at the archive root, not nested inside a wrapper directory' + - check: sep-2640-archive-no-traversal + text: 'the archive MUST NOT contain path-traversal sequences (`..`) or absolute paths' + - check: sep-2640-host-archive-safety + text: 'Hosts unpacking an archive MUST apply the archive safety requirements of the Agent Skills specification: reject archives containing path-traversal sequences or absolute paths, reject symlinks or hard links that resolve outside the skill directory, and enforce a limit on total unpacked size / Hosts MUST validate archives per the Agent Skills archive safety requirements: reject path traversal and absolute paths, reject links resolving outside the skill directory, and bound total unpacked size to prevent decompression bombs.' + - check: sep-2640-template-resource-template-registered + text: 'A server SHOULD register the same `url` value as an MCP resource template so hosts can wire template variables to the completion API.' + - check: sep-2640-host-verify-digest + text: 'Hosts MUST verify retrieved content against the `digest` in the index / hosts MUST NOT use unverified content.' + - check: sep-2640-host-no-empty-index-assumption + text: 'Hosts MUST NOT treat an absent or empty index as proof that a server has no skills.' + + - text: 'Hosts SHOULD surface template entries in their UI as interactive discovery points: the user fills in variables via completion, selects a skill, and the host passes the resolved URI into the conversation.' + excluded: 'UI affordance — surfacing template entries as interactive discovery points is not protocol-observable on the wire.' + - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' + excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' + - text: '[Hosts] SHOULD determine the format from the resource''s `mimeType`, falling back to the URL suffix' + excluded: 'Internal decision logic — when `mimeType` and URL suffix agree, the harness cannot distinguish a host that branched on `mimeType` from one that fell back to the suffix.' + - text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.' + excluded: 'Internal host policy — "treats as untrusted" is an assertion about how content is reasoned over downstream of the read, not about wire traffic.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - text: 'Hosts MUST NOT honor mechanisms in skill content that would cause local code execution without explicit user opt-in. This includes, non-exhaustively: hook declarations, pre/post-invocation scripts, shell commands embedded in frontmatter, or any field that a filesystem-sourced skill might use to register executable behavior on the host.' + excluded: 'Local code execution and explicit user opt-in are host-side filesystem and UX behaviors; not protocol-observable on the wire.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - text: 'Hosts MUST either ignore such fields entirely when the skill arrives over MCP, or gate them behind an explicit per-skill user approval that states what will execute and where.' + excluded: 'Either branch (silent ignore vs. UI-gated approval) is a host-internal handling choice; not protocol-observable.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - text: 'Hosts MUST NOT treat skill resources as higher-authority than other context. Explicit user policy governs whether a skill is loaded at all.' + excluded: 'Context-authority ordering is an internal prompting decision; not protocol-observable.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - text: 'Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill''s content before it is loaded into model context' + excluded: 'UI presentation requirements (origin indicator, pre-load inspection); the harness cannot observe what the host displays to users.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications From 2ecad130118675241c20e0ad66530bca2862a1e2 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Wed, 3 Jun 2026 17:21:32 -0700 Subject: [PATCH 02/25] style: apply yaml formatter pass --- src/seps/sep-2640.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index e54ce694..419aa401 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -8,11 +8,11 @@ requirements: - check: sep-2640-skill-uri-scheme text: 'Each file within a skill directory is exposed as an MCP resource. Servers SHOULD use the `skill://` URI scheme, under which the resource URI has the form: `skill:///`' - check: sep-2640-final-segment-equals-name - text: 'The final segment of `` MUST equal the skill''s `name` as declared in its `SKILL.md` frontmatter.' + text: "The final segment of `` MUST equal the skill's `name` as declared in its `SKILL.md` frontmatter." - check: sep-2640-no-nested-skills text: 'A `SKILL.md` MUST NOT appear in any descendant directory of a skill. The skill directory is the boundary; skills do not nest inside other skills.' - check: sep-2640-name-naming-rules - text: 'The final `` segment, being the skill `name`, MUST satisfy the Agent Skills specification''s naming rules.' + text: "The final `` segment, being the skill `name`, MUST satisfy the Agent Skills specification's naming rules." - check: sep-2640-prefix-rfc3986 text: 'Prefix segments SHOULD be valid URI path segments per RFC 3986; no further constraints are imposed on them.' - check: sep-2640-skillmd-mimetype @@ -56,7 +56,7 @@ requirements: excluded: 'UI affordance — surfacing template entries as interactive discovery points is not protocol-observable on the wire.' - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' - - text: '[Hosts] SHOULD determine the format from the resource''s `mimeType`, falling back to the URL suffix' + - text: "[Hosts] SHOULD determine the format from the resource's `mimeType`, falling back to the URL suffix" excluded: 'Internal decision logic — when `mimeType` and URL suffix agree, the harness cannot distinguish a host that branched on `mimeType` from one that fell back to the suffix.' - text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.' excluded: 'Internal host policy — "treats as untrusted" is an assertion about how content is reasoned over downstream of the read, not about wire traffic.' @@ -70,6 +70,6 @@ requirements: - text: 'Hosts MUST NOT treat skill resources as higher-authority than other context. Explicit user policy governs whether a skill is loaded at all.' excluded: 'Context-authority ordering is an internal prompting decision; not protocol-observable.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - - text: 'Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill''s content before it is loaded into model context' + - text: "Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill's content before it is loaded into model context" excluded: 'UI presentation requirements (origin indicator, pre-load inspection); the harness cannot observe what the host displays to users.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications From 4fa8b101cc79ca69cc9376866449f7612ff045ff Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Wed, 3 Jun 2026 17:35:30 -0700 Subject: [PATCH 03/25] chore(sep-2640): record spec source provenance --- src/seps/sep-2640.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 419aa401..1001dc92 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -1,3 +1,5 @@ +# spec_source: modelcontextprotocol/modelcontextprotocol@b77fdfe8c6fa91442900c52357711978617ce18a seps/2640-skills-extension.md +# extracted: 2026-06-03 sep: 2640 spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification requirements: @@ -53,13 +55,13 @@ requirements: text: 'Hosts MUST NOT treat an absent or empty index as proof that a server has no skills.' - text: 'Hosts SHOULD surface template entries in their UI as interactive discovery points: the user fills in variables via completion, selects a skill, and the host passes the resolved URI into the conversation.' - excluded: 'UI affordance — surfacing template entries as interactive discovery points is not protocol-observable on the wire.' + excluded: 'UI affordance: surfacing template entries as interactive discovery points is not protocol-observable on the wire.' - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' - text: "[Hosts] SHOULD determine the format from the resource's `mimeType`, falling back to the URL suffix" - excluded: 'Internal decision logic — when `mimeType` and URL suffix agree, the harness cannot distinguish a host that branched on `mimeType` from one that fell back to the suffix.' + excluded: 'Internal decision logic: when `mimeType` and URL suffix agree, the harness cannot distinguish a host that branched on `mimeType` from one that fell back to the suffix.' - text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.' - excluded: 'Internal host policy — "treats as untrusted" is an assertion about how content is reasoned over downstream of the read, not about wire traffic.' + excluded: 'Internal host policy: "treats as untrusted" is an assertion about how content is reasoned over downstream of the read, not about wire traffic.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - text: 'Hosts MUST NOT honor mechanisms in skill content that would cause local code execution without explicit user opt-in. This includes, non-exhaustively: hook declarations, pre/post-invocation scripts, shell commands embedded in frontmatter, or any field that a filesystem-sourced skill might use to register executable behavior on the host.' excluded: 'Local code execution and explicit user opt-in are host-side filesystem and UX behaviors; not protocol-observable on the wire.' From 1e50cadbb39346979c79318b08f99767c8ff2656 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Fri, 5 Jun 2026 12:05:09 -0700 Subject: [PATCH 04/25] chore(sep-2640): re-extract against SEP HEAD 556154c (drops mcp-resource-template) Re-extract sep-2640.yaml against current SEP-2640 HEAD. The SEP removed the `mcp-resource-template` index entry type in two commits on 2026-06-04 (fd50cc91 "Remove mcp-resource-template entries from skill index", 556154c0 "Remove remaining resource template mentions from SDK and rationale sections"), after this extraction was first captured on 2026-06-03 at b77fdfe8. Provenance comment bumped accordingly. Resulting changes: - sep-2640-index-entry-type-enum: drop `"mcp-resource-template"` from the enum. - sep-2640-index-name-required: drop the "is Required for ... Omitted for mcp-resource-template" conditional language; the column is now unconditional Required=Yes at SEP HEAD. - sep-2640-index-digest-required: same simplification; also align "hexadecimal" -> "hex" with the HEAD table text. - sep-2640-template-resource-template-registered: removed. The SEP no longer defines a template entry type, so the SHOULD on registering an MCP resource template is gone. - Excluded "Hosts SHOULD surface template entries..." excerpt: removed. The sentence no longer exists in the SEP. Sweep verified: zero "template" mentions in the SEP at HEAD. Security Implications section sentences referenced by the remaining excluded entries are intact. No other check rows touched. --- src/seps/sep-2640.yaml | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 1001dc92..ca93a62c 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -1,5 +1,5 @@ -# spec_source: modelcontextprotocol/modelcontextprotocol@b77fdfe8c6fa91442900c52357711978617ce18a seps/2640-skills-extension.md -# extracted: 2026-06-03 +# spec_source: modelcontextprotocol/modelcontextprotocol@556154c088371149c120172e95bb634655f00cbe seps/2640-skills-extension.md +# extracted: 2026-06-05 sep: 2640 spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification requirements: @@ -30,11 +30,11 @@ requirements: - check: sep-2640-server-expose-index text: 'A server SHOULD expose a resource at the well-known URI `skill://index.json` whose content is a JSON index of the skills it serves.' - check: sep-2640-index-entry-type-enum - text: '`skills[].type` MUST be `"skill-md"`, `"archive"`, or `"mcp-resource-template"`.' + text: '`skills[].type` MUST be `"skill-md"` or `"archive"`.' - check: sep-2640-index-name-required - text: '`skills[].name` is Required for `"skill-md"` and `"archive"`; matches the `SKILL.md` frontmatter `name` and the final segment of the skill path. Omitted for `"mcp-resource-template"`.' + text: '`skills[].name` matches the `SKILL.md` frontmatter `name` and the final segment of the skill path.' - check: sep-2640-index-digest-required - text: '`skills[].digest` is Required for `"skill-md"` and `"archive"`: SHA-256 content digest of the artifact, formatted as `sha256:{hex}` (64 lowercase hexadecimal characters). Omitted for `"mcp-resource-template"`.' + text: '`skills[].digest` is the SHA-256 content digest of the artifact, formatted as `sha256:{hex}` (64 lowercase hex characters).' - check: sep-2640-client-ignore-unrecognized text: 'Clients SHOULD ignore unrecognized fields and SHOULD skip entries with an unrecognized `type`.' - check: sep-2640-archive-format @@ -47,15 +47,11 @@ requirements: text: 'the archive MUST NOT contain path-traversal sequences (`..`) or absolute paths' - check: sep-2640-host-archive-safety text: 'Hosts unpacking an archive MUST apply the archive safety requirements of the Agent Skills specification: reject archives containing path-traversal sequences or absolute paths, reject symlinks or hard links that resolve outside the skill directory, and enforce a limit on total unpacked size / Hosts MUST validate archives per the Agent Skills archive safety requirements: reject path traversal and absolute paths, reject links resolving outside the skill directory, and bound total unpacked size to prevent decompression bombs.' - - check: sep-2640-template-resource-template-registered - text: 'A server SHOULD register the same `url` value as an MCP resource template so hosts can wire template variables to the completion API.' - check: sep-2640-host-verify-digest text: 'Hosts MUST verify retrieved content against the `digest` in the index / hosts MUST NOT use unverified content.' - check: sep-2640-host-no-empty-index-assumption text: 'Hosts MUST NOT treat an absent or empty index as proof that a server has no skills.' - - text: 'Hosts SHOULD surface template entries in their UI as interactive discovery points: the user fills in variables via completion, selects a skill, and the host passes the resolved URI into the conversation.' - excluded: 'UI affordance: surfacing template entries as interactive discovery points is not protocol-observable on the wire.' - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' - text: "[Hosts] SHOULD determine the format from the resource's `mimeType`, falling back to the URL suffix" From f1b069559d807a13a204ddf8e8e5e6e9485562f2 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Tue, 16 Jun 2026 12:05:57 -0700 Subject: [PATCH 05/25] feat(sep-2640): ResourcesDirectoryReadScenario for resources/directory/read Adds the conformance scenario for the SEP-2640 directoryRead surface that landed in spec commit 2e04c48d (2026-06-09). Per AGENTS.md "fewer scenarios, more checks", a single ResourcesDirectoryReadScenario emits 6 ConformanceChecks, one per new sep-2640.yaml requirement row. Class named for the wire method (resources/directory/read), matching the existing ResourcesListScenario / ResourcesReadTextScenario / etc. family in src/scenarios/server/resources.ts. The runner-facing name field stays as 'sep-2640-skills' (umbrella) so mcpkit's conformance/Makefile entry --scenario sep-2640-skills keeps working without a cross-repo race. src/seps/sep-2640.yaml - File-level provenance held at 556154c (the existing PR 330 baseline). - 6 new check rows appended for the directoryRead additions, each carrying a verbatim, grep-F-searchable excerpt from the SEP at 2e04c48da90224000e750ffd54a3611f2824fbc0: - sep-2640-capability-directory-read-flag - sep-2640-directory-read-method-registered - sep-2640-directory-read-subdir-mimetype - sep-2640-directory-read-result-resources-shape - sep-2640-directory-read-invalid-params - sep-2640-directory-read-pagination - forward_reference header comment notes the asymmetry: PR 97 schema rewrite (360123d0, 2026-06-08) made 3 existing rows stale and drifted ~11 others' wording; full re-extraction at SEP HEAD is mcpkit#780's lifecycle. Provenance deliberately holds at 556154c until that lands. src/types.ts - Adds io.modelcontextprotocol/skills to EXTENSION_IDS so the scenario's source: { extensionId: ... } tag type-checks. src/scenarios/server/directory.ts (new) - Capability discovery via wire-observable signal: -32601 method-not-found is the only definitive "server did not declare directoryRead" signal. - Fixture assumption: server exposes skill://acme/billing/refunds/templates with at least one subdirectory child (mcpkit examples/skills layout). No skill:// resources at all -> every check emits SKIPPED so the scenario stays green against upstream's everything-server fixture. - 6 checks (1:1 with the YAML rows above): 1. directoryRead declared (derived from method registration) 2. method registered (happy-path call succeeds) 3. result.resources shape matches resources/list 4. subdir child carries mimeType: "inode/directory" 5. non-directory URI returns -32602 6. nextCursor round-trips (single-page is conformant) src/scenarios/index.ts - Imports + registers ResourcesDirectoryReadScenario in allClientScenariosList, matching the registration name 'sep-2640-skills' that mcpkit's conformance/Makefile already passes to --scenario. Verified end-to-end against mcpkit examples/skills: cd ~/newstack/mcpkit/main && \ MCPCONFORMANCE_SKILLS_PATH=~/newstack/mcpkit/conf-skills \ make -C conformance testconf-skills -> 6/6 SUCCESS, 0 failed, 0 warnings. Out of scope: - PR 97 schema rewrite YAML refresh (separate ticket, mcpkit#780 lifecycle). - Negative-capability fixture (server without directoryRead) - mcpkit examples/skills has no flag for it; ext/skills/client_directory_test.go already covers the SDK pre-call guard. --- src/scenarios/index.ts | 5 + src/scenarios/server/directory.ts | 289 ++++++++++++++++++++++++++++++ src/seps/sep-2640.yaml | 24 +++ src/types.ts | 3 +- 4 files changed, 320 insertions(+), 1 deletion(-) create mode 100644 src/scenarios/server/directory.ts diff --git a/src/scenarios/index.ts b/src/scenarios/index.ts index b7230ab8..71da650b 100644 --- a/src/scenarios/index.ts +++ b/src/scenarios/index.ts @@ -57,6 +57,8 @@ import { ResourcesNotFoundErrorScenario } from './server/resources'; +import { ResourcesDirectoryReadScenario } from './server/directory'; + import { PromptsListScenario, PromptsGetSimpleScenario, @@ -173,6 +175,9 @@ const allClientScenariosList: ClientScenario[] = [ // Resources error handling (SEP-2164) new ResourcesNotFoundErrorScenario(), + // Skills extension (SEP-2640) — resources/directory/read surface (2e04c48d) + new ResourcesDirectoryReadScenario(), + // Prompts scenarios new PromptsListScenario(), new PromptsGetSimpleScenario(), diff --git a/src/scenarios/server/directory.ts b/src/scenarios/server/directory.ts new file mode 100644 index 00000000..2db1f89d --- /dev/null +++ b/src/scenarios/server/directory.ts @@ -0,0 +1,289 @@ +/** + * SEP-2640 Skills extension scenarios — focused on the resources/directory/read + * surface added in spec commit 2e04c48d (2026-06-09). + * + * One scenario, six checks (per AGENTS.md "fewer scenarios, more checks"). + * Each check's verbatim spec quote lives next to its check ID in + * src/seps/sep-2640.yaml, so the YAML and the scenario stay in lock-step. + * + * Capability discovery: the SEP allows multiple shapes for declaring the + * extension; the wire-observable signal we can rely on is whether + * resources/directory/read responds at all. A -32601 method-not-found is the + * only definitive "server didn't declare directoryRead" signal; any other + * response (success, -32602, etc.) means the server registered the method, + * which the SEP requires of any server that declared the capability. + * + * Fixture assumption: the scenario expects the standard mcpkit examples/skills + * fixture which exposes skill://acme/billing/refunds with a templates/ + * subtree containing at least one subdirectory. When the connected server is + * not a skills server (no skill:// resources at all), every check is emitted + * as SKIPPED — keeps the scenario green against the upstream everything-server + * while emitting real verdicts against any skills-capable fixture. + */ + +import { ClientScenario, ConformanceCheck } from '../../types'; +import { JsonRpcError, type RunContext } from '../../connection'; +import type { ListResourcesResult } from '../../spec-types/2025-06-18'; + +interface ResourceLike { + uri: string; + name?: string; + mimeType?: string; +} + +interface DirectoryReadResult { + resources?: ResourceLike[]; + nextCursor?: string; +} + +const SEP_2640_URL = + 'https://modelcontextprotocol.io/seps/2640-skills-extension#directory-listing'; + +const HAPPY_PATH_URI = 'skill://acme/billing/refunds/templates'; +const NON_DIRECTORY_URI = 'skill://acme/billing/refunds/SKILL.md'; + +const SPEC_REFERENCE = [ + { + id: 'SEP-2640-directory-listing', + url: SEP_2640_URL + } +]; + +const JSONRPC_METHOD_NOT_FOUND = -32601; +const JSONRPC_INVALID_PARAMS = -32602; + +function check( + id: string, + description: string, + status: 'SUCCESS' | 'FAILURE' | 'SKIPPED', + extras: Partial = {} +): ConformanceCheck { + return { + id, + name: id, + description, + status, + timestamp: new Date().toISOString(), + specReferences: SPEC_REFERENCE, + ...extras + }; +} + +export class ResourcesDirectoryReadScenario implements ClientScenario { + name = 'sep-2640-skills'; + readonly source = { + extensionId: 'io.modelcontextprotocol/skills' + } as const; + description = `SEP-2640 Skills extension: resources/directory/read surface (added in spec commit 2e04c48d, 2026-06-09). + +**Endpoint**: \`resources/directory/read\` (gated by \`io.modelcontextprotocol/skills.directoryRead: true\`) + +**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): + +- \`sep-2640-capability-directory-read-flag\` — server effectively declared directoryRead +- \`sep-2640-directory-read-method-registered\` — method registered for served skill directories +- \`sep-2640-directory-read-result-resources-shape\` — result has resources[] of direct children +- \`sep-2640-directory-read-subdir-mimetype\` — subdirectories surface with \`inode/directory\` mime +- \`sep-2640-directory-read-invalid-params\` — non-directory URI returns \`-32602\` +- \`sep-2640-directory-read-pagination\` — \`nextCursor\` round-trips per resources/list contract + +**Fixture expectation**: the server exposes \`skill://acme/billing/refunds/templates\` with at least one subdirectory child. Without any \`skill://\` resources every check emits SKIPPED.`; + + async run(ctx: RunContext): Promise { + const conn = await ctx.connect(); + try { + // SKIP gate: if the server exposes no skill:// resources, treat the + // run as not-applicable rather than failing every check. + let resources: ResourceLike[] = []; + try { + const list = await conn.request('resources/list'); + resources = (list.resources ?? []) as ResourceLike[]; + } catch { + // resources/list missing is itself diagnostic — the server can't be a + // skills server. Fall through to the SKIP branch. + } + const hasSkills = resources.some((r) => r.uri.startsWith('skill://')); + if (!hasSkills) { + const reason = + 'Server exposes no skill:// resources; SEP-2640 directoryRead checks not applicable.'; + return [ + 'sep-2640-capability-directory-read-flag', + 'sep-2640-directory-read-method-registered', + 'sep-2640-directory-read-result-resources-shape', + 'sep-2640-directory-read-subdir-mimetype', + 'sep-2640-directory-read-invalid-params', + 'sep-2640-directory-read-pagination' + ].map((id) => check(id, reason, 'SKIPPED', { errorMessage: reason })); + } + + const checks: ConformanceCheck[] = []; + + // === Happy path: list a known directory === + let happy: DirectoryReadResult | undefined; + let happyErr: unknown; + try { + happy = await conn.request( + 'resources/directory/read', + { uri: HAPPY_PATH_URI } + ); + } catch (e) { + happyErr = e; + } + + const isMethodNotFound = + happyErr instanceof JsonRpcError && + happyErr.code === JSONRPC_METHOD_NOT_FOUND; + + // Check 1: capability declaration (derived from method registration). + checks.push( + check( + 'sep-2640-capability-directory-read-flag', + 'Server declared the directoryRead capability — derived from whether resources/directory/read is registered (a server that did not declare directoryRead would return -32601 method-not-found).', + isMethodNotFound ? 'FAILURE' : 'SUCCESS', + isMethodNotFound + ? { + errorMessage: `resources/directory/read returned -32601, implying the server did not declare directoryRead: ${ + (happyErr as JsonRpcError).message + }` + } + : {} + ) + ); + + // Check 2: method registered. + checks.push( + check( + 'sep-2640-directory-read-method-registered', + 'resources/directory/read accepts a call against a known skill subdirectory.', + happy !== undefined + ? 'SUCCESS' + : isMethodNotFound + ? 'FAILURE' + : 'FAILURE', + happy !== undefined + ? { details: { uri: HAPPY_PATH_URI } } + : { + errorMessage: + happyErr instanceof Error + ? happyErr.message + : String(happyErr) + } + ) + ); + + // Check 3: result shape — resources[] of Resource objects. + const shapeOk = Array.isArray(happy?.resources); + const shapeErrs: string[] = []; + if (!shapeOk) { + shapeErrs.push('result.resources is not an array'); + } else { + happy!.resources!.forEach((r, i) => { + if (typeof r.uri !== 'string') + shapeErrs.push(`resources[${i}].uri is not a string`); + }); + } + checks.push( + check( + 'sep-2640-directory-read-result-resources-shape', + 'Result carries resources[] whose entries match the Resource shape (uri at minimum) from resources/list.', + shapeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + shapeErrs.length > 0 + ? { errorMessage: shapeErrs.join('; ') } + : { details: { entryCount: happy?.resources?.length ?? 0 } } + ) + ); + + // Check 4: subdirectory mime marker. + const subdirChild = happy?.resources?.find( + (r) => r.mimeType === 'inode/directory' + ); + const hasSubdir = subdirChild !== undefined; + checks.push( + check( + 'sep-2640-directory-read-subdir-mimetype', + 'Subdirectory child carries mimeType "inode/directory" so clients can descend.', + hasSubdir ? 'SUCCESS' : 'FAILURE', + hasSubdir + ? { details: { subdirectoryUri: subdirChild!.uri } } + : { + errorMessage: + 'Expected at least one child with mimeType "inode/directory" under ' + + HAPPY_PATH_URI + + '. Server fixture should expose a subdirectory there.' + } + ) + ); + + // === Error path: non-directory URI === + let invalidParamsOk = false; + let invalidParamsDetail = ''; + try { + await conn.request('resources/directory/read', { + uri: NON_DIRECTORY_URI + }); + invalidParamsDetail = + 'expected -32602 for non-directory URI, got success'; + } catch (e) { + if (e instanceof JsonRpcError && e.code === JSONRPC_INVALID_PARAMS) { + invalidParamsOk = true; + } else if (e instanceof JsonRpcError) { + invalidParamsDetail = `expected -32602 for non-directory URI, got ${e.code}: ${e.message}`; + } else { + invalidParamsDetail = `expected -32602, got non-JsonRpcError: ${ + e instanceof Error ? e.message : String(e) + }`; + } + } + checks.push( + check( + 'sep-2640-directory-read-invalid-params', + 'Non-directory URI yields -32602 Invalid params.', + invalidParamsOk ? 'SUCCESS' : 'FAILURE', + invalidParamsOk ? {} : { errorMessage: invalidParamsDetail } + ) + ); + + // === Pagination contract === + // The SEP is permissive: a single-page response with no nextCursor is + // conformant. The check passes when either (a) the first response has + // no nextCursor at all, or (b) the cursor round-trips on a follow-up + // call. mcpkit's defaultDirectoryReadPageSize = 0 puts it in (a). + let paginationOk = false; + let paginationDetail = ''; + const firstCursor = happy?.nextCursor; + if (!firstCursor) { + paginationOk = true; + paginationDetail = 'single-page response (no nextCursor)'; + } else { + try { + const second = await conn.request( + 'resources/directory/read', + { uri: HAPPY_PATH_URI, cursor: firstCursor } + ); + paginationOk = Array.isArray(second.resources); + paginationDetail = paginationOk + ? `nextCursor round-tripped: ${firstCursor}` + : 'follow-up call returned non-array resources'; + } catch (e) { + paginationDetail = `follow-up call with cursor failed: ${ + e instanceof Error ? e.message : String(e) + }`; + } + } + checks.push( + check( + 'sep-2640-directory-read-pagination', + 'nextCursor round-trips per the resources/list contract (single-page responses are conformant).', + paginationOk ? 'SUCCESS' : 'FAILURE', + paginationOk + ? { details: { paginationDetail } } + : { errorMessage: paginationDetail } + ) + ); + + return checks; + } finally { + await conn.close(); + } + } +} diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index ca93a62c..024d06c3 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -1,5 +1,15 @@ # spec_source: modelcontextprotocol/modelcontextprotocol@556154c088371149c120172e95bb634655f00cbe seps/2640-skills-extension.md # extracted: 2026-06-05 +# forward_reference: rows sep-2640-capability-directory-read-flag through +# sep-2640-directory-read-pagination track SEP commit +# 2e04c48da90224000e750ffd54a3611f2824fbc0 (2026-06-09) — the +# resources/directory/read addition. The file-level provenance above +# stays at 556154c because the PR 97 schema rewrite (360123d0, +# 2026-06-08) made 3 existing rows stale (sep-2640-index-entry-type-enum, +# sep-2640-index-name-required, sep-2640-index-digest-required) and +# drifted ~11 others' verbatim wording. Full re-extraction at SEP HEAD +# is mcpkit#780's lifecycle; this file deliberately holds at 556154c +# until that lands. sep: 2640 spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification requirements: @@ -52,6 +62,20 @@ requirements: - check: sep-2640-host-no-empty-index-assumption text: 'Hosts MUST NOT treat an absent or empty index as proof that a server has no skills.' + # resources/directory/read additions (SEP commit 2e04c48d, 2026-06-09) + - check: sep-2640-capability-directory-read-flag + text: 'Clients MUST NOT call `resources/directory/read` against a server that has not declared `directoryRead: true`.' + - check: sep-2640-directory-read-method-registered + text: 'A server that declares `directoryRead` MUST support the method for every directory within the skill namespaces it serves as individual files.' + - check: sep-2640-directory-read-subdir-mimetype + text: 'A _directory resource_ is a resource whose `mimeType` is `inode/directory`.' + - check: sep-2640-directory-read-result-resources-shape + text: 'The result contains every direct child of the directory: files with their ordinary resource metadata, subdirectories listed as directory resources (`mimeType: "inode/directory"`). The listing is not recursive; clients descend by calling the method again on a child directory.' + - check: sep-2640-directory-read-invalid-params + text: 'The method applies only to directory resources. If the URI does not exist, or exists but is not a directory resource, the server MUST return error `-32602` (Invalid params) — the same code `resources/read` uses for unknown resources.' + - check: sep-2640-directory-read-pagination + text: 'Pagination mirrors `resources/list`: when the result includes `nextCursor`, the client passes it back as `cursor` to retrieve the next page.' + - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' - text: "[Hosts] SHOULD determine the format from the resource's `mimeType`, falling back to the URL suffix" diff --git a/src/types.ts b/src/types.ts index 2e9dd22a..a11ea35f 100644 --- a/src/types.ts +++ b/src/types.ts @@ -68,7 +68,8 @@ export type ScenarioSpecTag = SpecVersion | 'extension'; */ export const EXTENSION_IDS = [ 'io.modelcontextprotocol/oauth-client-credentials', - 'io.modelcontextprotocol/enterprise-managed-authorization' + 'io.modelcontextprotocol/enterprise-managed-authorization', + 'io.modelcontextprotocol/skills' ] as const; export type ExtensionId = (typeof EXTENSION_IDS)[number]; From 2117d61eb77ef93a26c54abf5e1cde1a68d1b095 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Mon, 3 Aug 2026 21:07:50 -0700 Subject: [PATCH 06/25] =?UTF-8?q?feat(sep-2640):=20expand=20skills=20confo?= =?UTF-8?q?rmance=20=E2=80=94=20index=20+=20manifest=20scenarios,=20drift?= =?UTF-8?q?=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Builds on the directory-read scenario with the server-observable skills surface and brings the whole set up to the current framework conventions. New scenarios (src/scenarios/server/skills/): - SkillsIndexScenario (sep-2640-skills-index): skill://index.json is exposed and readable; every skills[].type is in the SEP enum; every entry has a non-empty name; a present digest matches sha256:{64hex}; entry URLs use skill://. - SkillsManifestScenario (sep-2640-skills-manifest): a skill-md SKILL.md is markdown-typed; frontmatter name/description surface on the resource; the final skill-path segment equals the frontmatter name; skill _meta keys use the reverse-domain prefix. directory.ts (moved into skills/): capability is now read from server/discover instead of inferred from -32601; an undeclared optional capability SKIPs rather than FAILs; the untestable.ts helpers back the not-exercisable paths; all mcpkit-specific fixture URIs are gone — the directory to exercise is discovered dynamically, so the scenario is brand-neutral. Capability gating accepts directoryRead at either extensions[id].directoryRead (SEP-2640's inline example) or extensions[id].config.directoryRead (the SEP-2133 negotiation envelope SEP-2640 defers to); the two SEPs are inconsistent on nesting, so the check accepts either rather than privileging one reading. All three scenarios verified green against a live SEP-2640 server (6/6, 6/6, 7/7); typecheck, eslint, prettier, and the 504-test vitest suite pass. sep-2640.yaml gains a header note mapping each scenario to the rows it now backs. --- src/scenarios/index.ts | 21 +- src/scenarios/server/directory.ts | 289 ----------------- src/scenarios/server/skills/directory.ts | 396 +++++++++++++++++++++++ src/scenarios/server/skills/helpers.ts | 227 +++++++++++++ src/scenarios/server/skills/index.ts | 270 ++++++++++++++++ src/scenarios/server/skills/manifest.ts | 367 +++++++++++++++++++++ src/seps/sep-2640.yaml | 14 + 7 files changed, 1291 insertions(+), 293 deletions(-) delete mode 100644 src/scenarios/server/directory.ts create mode 100644 src/scenarios/server/skills/directory.ts create mode 100644 src/scenarios/server/skills/helpers.ts create mode 100644 src/scenarios/server/skills/index.ts create mode 100644 src/scenarios/server/skills/manifest.ts diff --git a/src/scenarios/index.ts b/src/scenarios/index.ts index 587f3b7f..8fe9a610 100644 --- a/src/scenarios/index.ts +++ b/src/scenarios/index.ts @@ -58,7 +58,9 @@ import { ResourcesNotFoundErrorScenario } from './server/resources'; -import { ResourcesDirectoryReadScenario } from './server/directory'; +import { SkillsDirectoryReadScenario } from './server/skills/directory'; +import { SkillsIndexScenario } from './server/skills/index'; +import { SkillsManifestScenario } from './server/skills/manifest'; import { PromptsListScenario, @@ -153,7 +155,15 @@ const pendingClientScenariosList: ClientScenario[] = [ new TasksDispatchScenario(), new TasksStatusNotificationsScenario(), new TasksRequiredTaskErrorScenario(), - new TasksMrtrCompositionScenario() + new TasksMrtrCompositionScenario(), + + // SEP-2640 Skills extension. Pending because the everything-server does not + // implement io.modelcontextprotocol/skills; targeted runs point at a + // SEP-2640-conformant fixture via + // `npm start -- server --scenario sep-2640-skills-* --url `. + new SkillsDirectoryReadScenario(), + new SkillsIndexScenario(), + new SkillsManifestScenario() ]; // All client scenarios @@ -205,8 +215,11 @@ const allClientScenariosList: ClientScenario[] = [ // Resources error handling (SEP-2164) new ResourcesNotFoundErrorScenario(), - // Skills extension (SEP-2640) — resources/directory/read surface (2e04c48d) - new ResourcesDirectoryReadScenario(), + // Skills extension (SEP-2640). Fixture-dependent (needs a SEP-2640 server); + // each scenario SKIPs cleanly when the extension is not declared. + new SkillsDirectoryReadScenario(), + new SkillsIndexScenario(), + new SkillsManifestScenario(), // Prompts scenarios new PromptsListScenario(), diff --git a/src/scenarios/server/directory.ts b/src/scenarios/server/directory.ts deleted file mode 100644 index 2db1f89d..00000000 --- a/src/scenarios/server/directory.ts +++ /dev/null @@ -1,289 +0,0 @@ -/** - * SEP-2640 Skills extension scenarios — focused on the resources/directory/read - * surface added in spec commit 2e04c48d (2026-06-09). - * - * One scenario, six checks (per AGENTS.md "fewer scenarios, more checks"). - * Each check's verbatim spec quote lives next to its check ID in - * src/seps/sep-2640.yaml, so the YAML and the scenario stay in lock-step. - * - * Capability discovery: the SEP allows multiple shapes for declaring the - * extension; the wire-observable signal we can rely on is whether - * resources/directory/read responds at all. A -32601 method-not-found is the - * only definitive "server didn't declare directoryRead" signal; any other - * response (success, -32602, etc.) means the server registered the method, - * which the SEP requires of any server that declared the capability. - * - * Fixture assumption: the scenario expects the standard mcpkit examples/skills - * fixture which exposes skill://acme/billing/refunds with a templates/ - * subtree containing at least one subdirectory. When the connected server is - * not a skills server (no skill:// resources at all), every check is emitted - * as SKIPPED — keeps the scenario green against the upstream everything-server - * while emitting real verdicts against any skills-capable fixture. - */ - -import { ClientScenario, ConformanceCheck } from '../../types'; -import { JsonRpcError, type RunContext } from '../../connection'; -import type { ListResourcesResult } from '../../spec-types/2025-06-18'; - -interface ResourceLike { - uri: string; - name?: string; - mimeType?: string; -} - -interface DirectoryReadResult { - resources?: ResourceLike[]; - nextCursor?: string; -} - -const SEP_2640_URL = - 'https://modelcontextprotocol.io/seps/2640-skills-extension#directory-listing'; - -const HAPPY_PATH_URI = 'skill://acme/billing/refunds/templates'; -const NON_DIRECTORY_URI = 'skill://acme/billing/refunds/SKILL.md'; - -const SPEC_REFERENCE = [ - { - id: 'SEP-2640-directory-listing', - url: SEP_2640_URL - } -]; - -const JSONRPC_METHOD_NOT_FOUND = -32601; -const JSONRPC_INVALID_PARAMS = -32602; - -function check( - id: string, - description: string, - status: 'SUCCESS' | 'FAILURE' | 'SKIPPED', - extras: Partial = {} -): ConformanceCheck { - return { - id, - name: id, - description, - status, - timestamp: new Date().toISOString(), - specReferences: SPEC_REFERENCE, - ...extras - }; -} - -export class ResourcesDirectoryReadScenario implements ClientScenario { - name = 'sep-2640-skills'; - readonly source = { - extensionId: 'io.modelcontextprotocol/skills' - } as const; - description = `SEP-2640 Skills extension: resources/directory/read surface (added in spec commit 2e04c48d, 2026-06-09). - -**Endpoint**: \`resources/directory/read\` (gated by \`io.modelcontextprotocol/skills.directoryRead: true\`) - -**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): - -- \`sep-2640-capability-directory-read-flag\` — server effectively declared directoryRead -- \`sep-2640-directory-read-method-registered\` — method registered for served skill directories -- \`sep-2640-directory-read-result-resources-shape\` — result has resources[] of direct children -- \`sep-2640-directory-read-subdir-mimetype\` — subdirectories surface with \`inode/directory\` mime -- \`sep-2640-directory-read-invalid-params\` — non-directory URI returns \`-32602\` -- \`sep-2640-directory-read-pagination\` — \`nextCursor\` round-trips per resources/list contract - -**Fixture expectation**: the server exposes \`skill://acme/billing/refunds/templates\` with at least one subdirectory child. Without any \`skill://\` resources every check emits SKIPPED.`; - - async run(ctx: RunContext): Promise { - const conn = await ctx.connect(); - try { - // SKIP gate: if the server exposes no skill:// resources, treat the - // run as not-applicable rather than failing every check. - let resources: ResourceLike[] = []; - try { - const list = await conn.request('resources/list'); - resources = (list.resources ?? []) as ResourceLike[]; - } catch { - // resources/list missing is itself diagnostic — the server can't be a - // skills server. Fall through to the SKIP branch. - } - const hasSkills = resources.some((r) => r.uri.startsWith('skill://')); - if (!hasSkills) { - const reason = - 'Server exposes no skill:// resources; SEP-2640 directoryRead checks not applicable.'; - return [ - 'sep-2640-capability-directory-read-flag', - 'sep-2640-directory-read-method-registered', - 'sep-2640-directory-read-result-resources-shape', - 'sep-2640-directory-read-subdir-mimetype', - 'sep-2640-directory-read-invalid-params', - 'sep-2640-directory-read-pagination' - ].map((id) => check(id, reason, 'SKIPPED', { errorMessage: reason })); - } - - const checks: ConformanceCheck[] = []; - - // === Happy path: list a known directory === - let happy: DirectoryReadResult | undefined; - let happyErr: unknown; - try { - happy = await conn.request( - 'resources/directory/read', - { uri: HAPPY_PATH_URI } - ); - } catch (e) { - happyErr = e; - } - - const isMethodNotFound = - happyErr instanceof JsonRpcError && - happyErr.code === JSONRPC_METHOD_NOT_FOUND; - - // Check 1: capability declaration (derived from method registration). - checks.push( - check( - 'sep-2640-capability-directory-read-flag', - 'Server declared the directoryRead capability — derived from whether resources/directory/read is registered (a server that did not declare directoryRead would return -32601 method-not-found).', - isMethodNotFound ? 'FAILURE' : 'SUCCESS', - isMethodNotFound - ? { - errorMessage: `resources/directory/read returned -32601, implying the server did not declare directoryRead: ${ - (happyErr as JsonRpcError).message - }` - } - : {} - ) - ); - - // Check 2: method registered. - checks.push( - check( - 'sep-2640-directory-read-method-registered', - 'resources/directory/read accepts a call against a known skill subdirectory.', - happy !== undefined - ? 'SUCCESS' - : isMethodNotFound - ? 'FAILURE' - : 'FAILURE', - happy !== undefined - ? { details: { uri: HAPPY_PATH_URI } } - : { - errorMessage: - happyErr instanceof Error - ? happyErr.message - : String(happyErr) - } - ) - ); - - // Check 3: result shape — resources[] of Resource objects. - const shapeOk = Array.isArray(happy?.resources); - const shapeErrs: string[] = []; - if (!shapeOk) { - shapeErrs.push('result.resources is not an array'); - } else { - happy!.resources!.forEach((r, i) => { - if (typeof r.uri !== 'string') - shapeErrs.push(`resources[${i}].uri is not a string`); - }); - } - checks.push( - check( - 'sep-2640-directory-read-result-resources-shape', - 'Result carries resources[] whose entries match the Resource shape (uri at minimum) from resources/list.', - shapeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', - shapeErrs.length > 0 - ? { errorMessage: shapeErrs.join('; ') } - : { details: { entryCount: happy?.resources?.length ?? 0 } } - ) - ); - - // Check 4: subdirectory mime marker. - const subdirChild = happy?.resources?.find( - (r) => r.mimeType === 'inode/directory' - ); - const hasSubdir = subdirChild !== undefined; - checks.push( - check( - 'sep-2640-directory-read-subdir-mimetype', - 'Subdirectory child carries mimeType "inode/directory" so clients can descend.', - hasSubdir ? 'SUCCESS' : 'FAILURE', - hasSubdir - ? { details: { subdirectoryUri: subdirChild!.uri } } - : { - errorMessage: - 'Expected at least one child with mimeType "inode/directory" under ' + - HAPPY_PATH_URI + - '. Server fixture should expose a subdirectory there.' - } - ) - ); - - // === Error path: non-directory URI === - let invalidParamsOk = false; - let invalidParamsDetail = ''; - try { - await conn.request('resources/directory/read', { - uri: NON_DIRECTORY_URI - }); - invalidParamsDetail = - 'expected -32602 for non-directory URI, got success'; - } catch (e) { - if (e instanceof JsonRpcError && e.code === JSONRPC_INVALID_PARAMS) { - invalidParamsOk = true; - } else if (e instanceof JsonRpcError) { - invalidParamsDetail = `expected -32602 for non-directory URI, got ${e.code}: ${e.message}`; - } else { - invalidParamsDetail = `expected -32602, got non-JsonRpcError: ${ - e instanceof Error ? e.message : String(e) - }`; - } - } - checks.push( - check( - 'sep-2640-directory-read-invalid-params', - 'Non-directory URI yields -32602 Invalid params.', - invalidParamsOk ? 'SUCCESS' : 'FAILURE', - invalidParamsOk ? {} : { errorMessage: invalidParamsDetail } - ) - ); - - // === Pagination contract === - // The SEP is permissive: a single-page response with no nextCursor is - // conformant. The check passes when either (a) the first response has - // no nextCursor at all, or (b) the cursor round-trips on a follow-up - // call. mcpkit's defaultDirectoryReadPageSize = 0 puts it in (a). - let paginationOk = false; - let paginationDetail = ''; - const firstCursor = happy?.nextCursor; - if (!firstCursor) { - paginationOk = true; - paginationDetail = 'single-page response (no nextCursor)'; - } else { - try { - const second = await conn.request( - 'resources/directory/read', - { uri: HAPPY_PATH_URI, cursor: firstCursor } - ); - paginationOk = Array.isArray(second.resources); - paginationDetail = paginationOk - ? `nextCursor round-tripped: ${firstCursor}` - : 'follow-up call returned non-array resources'; - } catch (e) { - paginationDetail = `follow-up call with cursor failed: ${ - e instanceof Error ? e.message : String(e) - }`; - } - } - checks.push( - check( - 'sep-2640-directory-read-pagination', - 'nextCursor round-trips per the resources/list contract (single-page responses are conformant).', - paginationOk ? 'SUCCESS' : 'FAILURE', - paginationOk - ? { details: { paginationDetail } } - : { errorMessage: paginationDetail } - ) - ); - - return checks; - } finally { - await conn.close(); - } - } -} diff --git a/src/scenarios/server/skills/directory.ts b/src/scenarios/server/skills/directory.ts new file mode 100644 index 00000000..27be77bd --- /dev/null +++ b/src/scenarios/server/skills/directory.ts @@ -0,0 +1,396 @@ +/** + * SEP-2640 Skills extension — the `resources/directory/read` surface (added in + * spec commit 2e04c48d, 2026-06-09). + * + * One scenario, six checks (per AGENTS.md "fewer scenarios, more checks"). + * Each check's verbatim spec quote lives next to its check ID in + * src/seps/sep-2640.yaml. + * + * Capability gating reads the declared capability from `server/discover` + * (mirrors `tasks/capability.ts`): the checks run only when the server declares + * `io.modelcontextprotocol/skills.directoryRead: true`. An undeclared optional + * capability is a SKIP (not a failure); a declared-but-broken one fails. + * + * Discovery is dynamic and brand-neutral: the directory to exercise is derived + * from `skill://index.json` or `resources/list`, hardcoding no fixture URI, so + * the scenario passes against any conformant SEP-2640 server. When no directory + * (or no subdirectory) can be discovered, that check reports the missing + * prerequisite via untestableCheck (issue #248), never a silent green. + */ + +import { ClientScenario, ConformanceCheck } from '../../../types'; +import { Connection, JsonRpcError, type RunContext } from '../../../connection'; +import { untestableCheck } from '../../untestable'; +import { + SKILLS_EXTENSION_ID, + SKILL_MANIFEST_FILENAME, + SEP_2640_REF, + JSONRPC_METHOD_NOT_FOUND, + JSONRPC_INVALID_PARAMS, + type SkillIndex, + type SkillResource, + skillsCapability, + directoryReadDeclared, + skillsCheck, + listAllResources, + readSkillIndexText, + skillNameFromManifestUri +} from './helpers'; + +const DIRECTORY_MIME = 'inode/directory'; + +const CAPABILITY_ID = 'sep-2640-capability-directory-read-flag'; +const METHOD_ID = 'sep-2640-directory-read-method-registered'; +const SHAPE_ID = 'sep-2640-directory-read-result-resources-shape'; +const SUBDIR_ID = 'sep-2640-directory-read-subdir-mimetype'; +const INVALID_PARAMS_ID = 'sep-2640-directory-read-invalid-params'; +const PAGINATION_ID = 'sep-2640-directory-read-pagination'; + +const ALL_IDS = [ + CAPABILITY_ID, + METHOD_ID, + SHAPE_ID, + SUBDIR_ID, + INVALID_PARAMS_ID, + PAGINATION_ID +]; + +interface DirectoryReadResult { + resources?: SkillResource[]; + nextCursor?: string; +} + +/** A directory to exercise plus, when known, a non-directory resource under it. */ +interface DirectoryTarget { + dirUri: string; + /** A known file (non-directory) resource, used for the -32602 negative path. */ + fileUri?: string; +} + +/** The skill root directory URI for a SKILL.md URI (strip the trailing file). */ +function skillRootFromManifestUri(uri: string): string | undefined { + if (skillNameFromManifestUri(uri) === undefined) return undefined; + return uri.slice(0, uri.length - `/${SKILL_MANIFEST_FILENAME}`.length); +} + +/** + * Discover a directory resource to exercise, brand-neutrally: prefer a skill + * root derived from a skill-md SKILL.md (index first, then resources/list), + * then any `inode/directory` resource in resources/list. + */ +async function discoverDirectory( + conn: Connection +): Promise { + // 1. skill-md entry in skill://index.json — its SKILL.md URL gives us both a + // directory (the skill root) and a known file (the SKILL.md itself). + const idx = await readSkillIndexText(conn); + if (!('error' in idx) && typeof idx.text === 'string') { + try { + const index = JSON.parse(idx.text) as SkillIndex; + const entry = (index.skills ?? []).find( + (e) => + e.type === 'skill-md' && + typeof e.url === 'string' && + skillRootFromManifestUri(e.url) !== undefined + ); + if (entry?.url) { + return { + dirUri: skillRootFromManifestUri(entry.url)!, + fileUri: entry.url + }; + } + } catch { + // A malformed index is the index scenario's concern; keep discovering. + } + } + + const resources = await listAllResources(conn); + + // 2. A SKILL.md in resources/list — derive the skill root the same way. + const manifest = resources.find( + (r) => skillRootFromManifestUri(r.uri) !== undefined + ); + if (manifest) { + return { + dirUri: skillRootFromManifestUri(manifest.uri)!, + fileUri: manifest.uri + }; + } + + // 3. Any directory resource, using a non-directory sibling for the -32602 + // path when one is listed. + const dir = resources.find((r) => r.mimeType === DIRECTORY_MIME); + if (dir) { + const file = resources.find((r) => r.mimeType !== DIRECTORY_MIME); + return { dirUri: dir.uri, fileUri: file?.uri }; + } + + return undefined; +} + +export class SkillsDirectoryReadScenario implements ClientScenario { + name = 'sep-2640-skills-directory'; + readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; + description = `SEP-2640 Skills extension: resources/directory/read surface (added in spec commit 2e04c48d, 2026-06-09). + +**Endpoint**: \`resources/directory/read\` (gated by \`io.modelcontextprotocol/skills.directoryRead: true\`) + +**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): + +- \`sep-2640-capability-directory-read-flag\` — server declared directoryRead (read from server/discover) +- \`sep-2640-directory-read-method-registered\` — a declaring server supports the method on a served directory (MUST) +- \`sep-2640-directory-read-result-resources-shape\` — result has resources[] of direct children (MUST) +- \`sep-2640-directory-read-subdir-mimetype\` — subdirectory children carry \`inode/directory\` (MUST) +- \`sep-2640-directory-read-invalid-params\` — a non-directory URI returns \`-32602\` (MUST) +- \`sep-2640-directory-read-pagination\` — \`nextCursor\` round-trips per resources/list (single-page is conformant) + +**Gating & discovery**: the checks SKIP when the skills extension or its \`directoryRead\` flag is undeclared. The directory to exercise is discovered dynamically from \`skill://index.json\` / \`resources/list\` — no fixture URI is hardcoded.`; + + async run(ctx: RunContext): Promise { + const conn = await ctx.connect(); + try { + // === Capability gating via server/discover (not error-inference) === + const skills = await skillsCapability(conn); + if (!skills) { + const reason = + 'Server did not declare the io.modelcontextprotocol/skills extension; directoryRead checks not applicable.'; + return ALL_IDS.map((id) => + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + if (!directoryReadDeclared(skills)) { + const reason = + 'Server declared the skills extension but not directoryRead: true; the resources/directory/read checks are optional and not applicable.'; + return ALL_IDS.map((id) => + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + + const checks: ConformanceCheck[] = []; + + // Check 1: capability declared (observed directly from server/discover). + checks.push( + skillsCheck( + CAPABILITY_ID, + 'Server declared io.modelcontextprotocol/skills.directoryRead: true under capabilities.extensions.', + 'SUCCESS', + { details: { directoryRead: true } } + ) + ); + + // === Discover a directory to exercise (brand-neutral) === + const target = await discoverDirectory(conn); + if (!target) { + const reason = + 'no directory resource discoverable via skill://index.json or resources/list to exercise resources/directory/read'; + const rest: Array<[string, string]> = [ + [ + METHOD_ID, + 'A declaring server MUST support the method on a served directory.' + ], + [ + SHAPE_ID, + 'Result carries resources[] of the directory’s direct children.' + ], + [SUBDIR_ID, 'Subdirectory children carry mimeType inode/directory.'], + [ + INVALID_PARAMS_ID, + 'A non-directory URI yields -32602 Invalid params.' + ], + [ + PAGINATION_ID, + 'nextCursor round-trips per the resources/list contract.' + ] + ]; + for (const [id, desc] of rest) { + checks.push( + untestableCheck(id, id, desc, reason, [SEP_2640_REF], 'FAILURE') + ); + } + return checks; + } + + // === Happy path: list the discovered directory === + let happy: DirectoryReadResult | undefined; + let happyErr: unknown; + try { + happy = await conn.request( + 'resources/directory/read', + { uri: target.dirUri } + ); + } catch (e) { + happyErr = e; + } + + // Check 2: method registered (declared -> MUST be supported). + const methodNotFound = + happyErr instanceof JsonRpcError && + happyErr.code === JSONRPC_METHOD_NOT_FOUND; + checks.push( + skillsCheck( + METHOD_ID, + 'A server that declares directoryRead MUST support resources/directory/read on a served skill directory.', + happy !== undefined ? 'SUCCESS' : 'FAILURE', + happy !== undefined + ? { details: { uri: target.dirUri } } + : { + errorMessage: methodNotFound + ? `resources/directory/read returned -32601 for ${target.dirUri} despite the server declaring directoryRead: true` + : `resources/directory/read on ${target.dirUri} failed: ${ + happyErr instanceof Error + ? happyErr.message + : String(happyErr) + }` + } + ) + ); + + // Check 3: result shape — resources[] of Resource objects. + const shapeErrs: string[] = []; + if (!Array.isArray(happy?.resources)) { + shapeErrs.push('result.resources is not an array'); + } else { + happy.resources.forEach((r, i) => { + if (typeof r.uri !== 'string') { + shapeErrs.push(`resources[${i}].uri is not a string`); + } + }); + } + checks.push( + skillsCheck( + SHAPE_ID, + 'The result contains resources[] listing the directory’s direct children, each with at least a uri.', + happy === undefined + ? 'FAILURE' + : shapeErrs.length === 0 + ? 'SUCCESS' + : 'FAILURE', + happy === undefined + ? { errorMessage: 'directory read did not return a result' } + : shapeErrs.length === 0 + ? { details: { childCount: happy.resources?.length ?? 0 } } + : { errorMessage: shapeErrs.join('; ') } + ) + ); + + // Check 4: subdirectory mime marker. A directory whose fixture exposes no + // child subdirectory cannot exercise this — report it untestable, not a + // pass and not a failure of the server. + const subdirChild = Array.isArray(happy?.resources) + ? happy.resources.find((r) => r.mimeType === DIRECTORY_MIME) + : undefined; + if (subdirChild) { + checks.push( + skillsCheck( + SUBDIR_ID, + 'A subdirectory child is listed as a directory resource (mimeType inode/directory) so clients can descend.', + 'SUCCESS', + { details: { subdirectoryUri: subdirChild.uri } } + ) + ); + } else { + checks.push( + untestableCheck( + SUBDIR_ID, + SUBDIR_ID, + 'A subdirectory child is listed with mimeType inode/directory.', + `no child with mimeType ${DIRECTORY_MIME} under ${target.dirUri}; the served directory exposes no subdirectory to exercise this check`, + [SEP_2640_REF], + 'FAILURE' + ) + ); + } + + // Check 5: non-directory URI -> -32602. Needs a known non-directory + // resource; prefer the discovered fileUri, else a non-directory child. + const nonDirUri = + target.fileUri ?? + (Array.isArray(happy?.resources) + ? happy.resources.find( + (r) => typeof r.uri === 'string' && r.mimeType !== DIRECTORY_MIME + )?.uri + : undefined); + if (nonDirUri === undefined) { + checks.push( + untestableCheck( + INVALID_PARAMS_ID, + INVALID_PARAMS_ID, + 'A non-directory URI yields -32602 Invalid params.', + 'no non-directory resource discoverable to probe the -32602 path', + [SEP_2640_REF], + 'FAILURE' + ) + ); + } else { + let invalidOk = false; + let invalidDetail = ''; + try { + await conn.request('resources/directory/read', { + uri: nonDirUri + }); + invalidDetail = `expected -32602 for non-directory URI ${nonDirUri}, got a successful result`; + } catch (e) { + if (e instanceof JsonRpcError && e.code === JSONRPC_INVALID_PARAMS) { + invalidOk = true; + } else if (e instanceof JsonRpcError) { + invalidDetail = `expected -32602 for ${nonDirUri}, got ${e.code}: ${e.message}`; + } else { + invalidDetail = `expected -32602, got non-JsonRpcError: ${ + e instanceof Error ? e.message : String(e) + }`; + } + } + checks.push( + skillsCheck( + INVALID_PARAMS_ID, + 'resources/directory/read on a non-directory URI MUST return -32602 (Invalid params).', + invalidOk ? 'SUCCESS' : 'FAILURE', + invalidOk + ? { details: { nonDirectoryUri: nonDirUri } } + : { errorMessage: invalidDetail } + ) + ); + } + + // Check 6: pagination contract (single-page is conformant). + let paginationOk = false; + let paginationDetail = ''; + const firstCursor = happy?.nextCursor; + if (happy === undefined) { + paginationDetail = 'no directory result to evaluate pagination'; + } else if (!firstCursor) { + paginationOk = true; + paginationDetail = 'single-page response (no nextCursor)'; + } else { + try { + const second = await conn.request( + 'resources/directory/read', + { uri: target.dirUri, cursor: firstCursor } + ); + paginationOk = Array.isArray(second.resources); + paginationDetail = paginationOk + ? `nextCursor round-tripped: ${firstCursor}` + : 'follow-up call returned non-array resources'; + } catch (e) { + paginationDetail = `follow-up call with cursor failed: ${ + e instanceof Error ? e.message : String(e) + }`; + } + } + checks.push( + skillsCheck( + PAGINATION_ID, + 'nextCursor round-trips per the resources/list contract (single-page responses are conformant).', + paginationOk ? 'SUCCESS' : 'FAILURE', + paginationOk + ? { details: { paginationDetail } } + : { errorMessage: paginationDetail } + ) + ); + + return checks; + } finally { + await conn.close(); + } + } +} diff --git a/src/scenarios/server/skills/helpers.ts b/src/scenarios/server/skills/helpers.ts new file mode 100644 index 00000000..d32022f9 --- /dev/null +++ b/src/scenarios/server/skills/helpers.ts @@ -0,0 +1,227 @@ +/** + * Shared helpers for the SEP-2640 (Skills extension) server-conformance + * scenarios under this directory. + * + * The scenarios treat the server-under-test as an arbitrary SEP-2640 server: + * capability is read from `server/discover` (never inferred from an error), and + * every skill is discovered dynamically from `skill://index.json` and + * `resources/list` — no fixture-specific URI is hardcoded, so the checks pass + * against any conformant server, not just one implementation's fixture. + */ + +import type { + CheckStatus, + ConformanceCheck, + SpecReference +} from '../../../types'; +import type { Connection } from '../../../connection'; +import { JsonRpcError } from '../../../connection'; +import { parse as parseYaml } from 'yaml'; + +export const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; +export const SKILL_URI_SCHEME = 'skill://'; +export const SKILL_INDEX_URI = 'skill://index.json'; +export const SKILL_MANIFEST_FILENAME = 'SKILL.md'; +export const SKILLS_META_PREFIX = 'io.modelcontextprotocol.skills/'; + +/** `sha256:{hex}` with exactly 64 lowercase hex characters (SEP-2640 index). */ +export const SKILL_DIGEST_PATTERN = /^sha256:[0-9a-f]{64}$/; + +/** The SEP enumerated `skills[].type` values. */ +export const SKILL_TYPES = ['skill-md', 'archive'] as const; + +export const JSONRPC_METHOD_NOT_FOUND = -32601; +export const JSONRPC_INVALID_PARAMS = -32602; + +export const SEP_2640_REF: SpecReference = { + id: 'SEP-2640', + url: 'https://modelcontextprotocol.io/seps/2640-skills-extension#specification' +}; + +/** A `resources/list` / directory-read entry (only the fields we inspect). */ +export interface SkillResource { + uri: string; + name?: string; + description?: string; + mimeType?: string; + _meta?: Record; +} + +/** One `skills[]` entry of the `skill://index.json` document. */ +export interface SkillIndexEntry { + name?: string; + type?: string; + description?: string; + url?: string; + digest?: string; + [key: string]: unknown; +} + +/** The parsed `skill://index.json` document. */ +export interface SkillIndex { + $schema?: string; + skills?: SkillIndexEntry[]; + [key: string]: unknown; +} + +/** First text content of a `resources/read`, with its mimeType and `_meta`. */ +export interface ResourceText { + text: string; + mimeType?: string; + meta?: Record; +} + +/** + * Build a check carrying the SEP-2640 reference. Per AGENTS.md the same `id` + * flips `status` + `errorMessage` between SUCCESS and FAILURE rather than + * branching into distinct slugs. + */ +export function skillsCheck( + id: string, + description: string, + status: CheckStatus, + extras: Partial = {} +): ConformanceCheck { + return { + id, + name: id, + description, + status, + timestamp: new Date().toISOString(), + specReferences: [SEP_2640_REF], + ...extras + }; +} + +/** + * The skills extension object declared under `capabilities.extensions`, or + * `undefined` when the server did not declare it. Reads the declared capability + * from `server/discover` (mirrors `tasks/capability.ts`) — an undeclared + * optional extension is a SKIP, never inferred from a `-32601`. + */ +export async function skillsCapability( + conn: Connection +): Promise | undefined> { + const discovered = await conn.discover(); + const caps = (discovered.capabilities as Record) ?? {}; + const extensions = caps.extensions as Record | undefined; + const skills = extensions?.[SKILLS_EXTENSION_ID]; + return skills && typeof skills === 'object' + ? (skills as Record) + : undefined; +} + +/** + * Whether the skills extension declares `directoryRead: true`. + * + * SEP-2640's capability-declaration example places the flag directly on the + * extension object (`extensions[id].directoryRead`). SEP-2133 extension + * negotiation — which SEP-2640 normatively defers to ("Per SEP-2133 extension + * negotiation") — wraps settings in a `{ specVersion, stability, config }` + * envelope, putting the flag at `extensions[id].config.directoryRead`. The two + * SEPs are inconsistent on nesting, so a brand-neutral conformance check accepts + * either location rather than privileging one reading of an ambiguous spec. + * (The inconsistency is worth a WG clarification; see the scenario docs.) + */ +export function directoryReadDeclared( + skills: Record +): boolean { + if (skills.directoryRead === true) return true; + const config = skills.config as Record | undefined; + return config?.directoryRead === true; +} + +/** Everything from `resources/list`, paginating until `nextCursor` clears. */ +export async function listAllResources( + conn: Connection +): Promise { + const out: SkillResource[] = []; + let cursor: string | undefined; + do { + const page = await conn.request<{ + resources?: SkillResource[]; + nextCursor?: string; + }>('resources/list', cursor ? { cursor } : undefined); + out.push(...(page.resources ?? [])); + cursor = page.nextCursor; + } while (cursor); + return out; +} + +/** + * Read `skill://index.json`. Returns the raw JSON text (for parse-error + * reporting) or a `JsonRpcError` when the server declines the well-known index + * — a permitted MAY (SEP-2640 §Enumeration): the catalog may be unenumerable. + */ +export async function readSkillIndexText( + conn: Connection +): Promise<{ text?: string; mimeType?: string } | { error: JsonRpcError }> { + try { + const res = await conn.request<{ + contents?: Array<{ text?: string; mimeType?: string }>; + }>('resources/read', { uri: SKILL_INDEX_URI }); + const entry = (res.contents ?? []).find((c) => typeof c.text === 'string'); + return { text: entry?.text, mimeType: entry?.mimeType }; + } catch (e) { + if (e instanceof JsonRpcError) return { error: e }; + throw e; + } +} + +/** Read a resource's first text content plus its mimeType and `_meta`. */ +export async function readResourceText( + conn: Connection, + uri: string +): Promise { + const res = await conn.request<{ + contents?: Array<{ + text?: string; + mimeType?: string; + _meta?: Record; + }>; + }>('resources/read', { uri }); + const entry = (res.contents ?? []).find((c) => typeof c.text === 'string'); + if (!entry || typeof entry.text !== 'string') return undefined; + return { text: entry.text, mimeType: entry.mimeType, meta: entry._meta }; +} + +/** + * The skill name recoverable from a `SKILL.md` resource URI: the final segment + * of ``, i.e. the last path segment before the trailing + * `SKILL.md`. Returns `undefined` when the URI is not a `skill://…/SKILL.md`. + * + * skill://org/team/deploy/SKILL.md -> "deploy" + * skill://lint/SKILL.md -> "lint" + */ +export function skillNameFromManifestUri(uri: string): string | undefined { + if (!uri.startsWith(SKILL_URI_SCHEME)) return undefined; + const parts = uri + .slice(SKILL_URI_SCHEME.length) + .split('/') + .filter((p) => p.length > 0); + if (parts.length < 2) return undefined; + if (parts[parts.length - 1] !== SKILL_MANIFEST_FILENAME) return undefined; + return parts[parts.length - 2]; +} + +/** + * Extract and parse the YAML frontmatter block at the head of a `SKILL.md`. + * Returns `undefined` when there is no leading `---` delimited block or it does + * not parse to an object. + */ +export function parseFrontmatter( + markdown: string +): Record | undefined { + // Tolerate a leading UTF-8 BOM before the opening `---` fence. + const body = markdown.charCodeAt(0) === 0xfeff ? markdown.slice(1) : markdown; + const match = body.match(/^---\r?\n([\s\S]*?)\r?\n---[ \t]*(?:\r?\n|$)/); + if (!match) return undefined; + try { + const parsed = parseYaml(match[1]) as unknown; + return parsed && typeof parsed === 'object' + ? (parsed as Record) + : undefined; + } catch { + return undefined; + } +} diff --git a/src/scenarios/server/skills/index.ts b/src/scenarios/server/skills/index.ts new file mode 100644 index 00000000..bb9d1f33 --- /dev/null +++ b/src/scenarios/server/skills/index.ts @@ -0,0 +1,270 @@ +/** + * SEP-2640 Skills extension — `skill://index.json` enumeration surface. + * + * One scenario, many checks (per AGENTS.md "fewer scenarios, more checks"). + * Each check's verbatim spec quote lives next to its check ID in + * src/seps/sep-2640.yaml, keeping the YAML and this scenario in lock-step. + * + * All discovery is dynamic and brand-neutral: the scenario reads the well-known + * `skill://index.json` and validates whatever entries it finds, hardcoding no + * fixture-specific skill name or URI. When the server does not declare the + * skills extension the checks are SKIPPED (an optional, undeclared capability); + * when the server declines the index (a permitted MAY) or serves an empty index + * the index-shape checks are SKIPPED (legitimately not applicable), never + * failed against a conformant server. + */ + +import { ClientScenario, ConformanceCheck } from '../../../types'; +import type { RunContext } from '../../../connection'; +import { + SKILLS_EXTENSION_ID, + SKILL_INDEX_URI, + SKILL_URI_SCHEME, + SKILL_TYPES, + SKILL_DIGEST_PATTERN, + type SkillIndex, + type SkillIndexEntry, + skillsCapability, + skillsCheck, + readSkillIndexText +} from './helpers'; + +const ENTRY_CHECK_IDS = [ + 'sep-2640-index-entry-type-enum', + 'sep-2640-index-name-required', + 'sep-2640-index-digest-required', + 'sep-2640-skill-uri-scheme' +] as const; + +const ALL_CHECK_IDS = ['sep-2640-server-expose-index', ...ENTRY_CHECK_IDS]; + +export class SkillsIndexScenario implements ClientScenario { + name = 'sep-2640-skills-index'; + readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; + description = `SEP-2640 Skills extension: the \`skill://index.json\` enumeration index. + +**Resource**: \`skill://index.json\` (read via \`resources/read\`, \`mimeType\` \`application/json\`) + +**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): + +- \`sep-2640-server-expose-index\` — server exposes a readable \`skill://index.json\` (SHOULD; a server MAY decline for an unenumerable catalog) +- \`sep-2640-index-entry-type-enum\` — every \`skills[].type\` is \`"skill-md"\` or \`"archive"\` (MUST) +- \`sep-2640-index-name-required\` — every entry carries a non-empty \`name\` (required field) +- \`sep-2640-index-digest-required\` — a present \`skills[].digest\` is \`sha256:{64 hex}\` (MUST) +- \`sep-2640-skill-uri-scheme\` — index entry URLs use the \`skill://\` scheme (SHOULD; another scheme is permitted only when listed in the index) + +**Discovery is dynamic**: the scenario reads whatever skills the index enumerates. Undeclared extension, a declined index, or an empty index all SKIP cleanly.`; + + async run(ctx: RunContext): Promise { + const conn = await ctx.connect(); + try { + const skills = await skillsCapability(conn); + if (!skills) { + const reason = + 'Server did not declare the io.modelcontextprotocol/skills extension; index checks not applicable.'; + return ALL_CHECK_IDS.map((id) => + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + + const checks: ConformanceCheck[] = []; + + // === server-expose-index (SHOULD, with an explicit MAY-decline) === + const read = await readSkillIndexText(conn); + if ('error' in read) { + const reason = `Server declined skill://index.json (code ${read.error.code}); permitted MAY — the catalog may be large or unenumerable. Hosts MUST NOT treat this as proof of no skills.`; + checks.push( + skillsCheck( + 'sep-2640-server-expose-index', + 'Server SHOULD expose a readable skill://index.json enumerating the skills it serves.', + 'SKIPPED', + { errorMessage: reason } + ) + ); + for (const id of ENTRY_CHECK_IDS) { + checks.push( + skillsCheck(id, 'No skill://index.json to inspect.', 'SKIPPED', { + errorMessage: reason + }) + ); + } + return checks; + } + + if (read.text === undefined) { + const reason = + 'resources/read on skill://index.json returned no text content; the index resource is exposed but unreadable.'; + checks.push( + skillsCheck( + 'sep-2640-server-expose-index', + 'Server SHOULD expose a readable skill://index.json enumerating the skills it serves.', + 'FAILURE', + { errorMessage: reason } + ) + ); + for (const id of ENTRY_CHECK_IDS) { + checks.push( + skillsCheck( + id, + 'No readable index content to inspect.', + 'SKIPPED', + { + errorMessage: reason + } + ) + ); + } + return checks; + } + + let index: SkillIndex; + try { + index = JSON.parse(read.text) as SkillIndex; + } catch (e) { + const reason = `skill://index.json content is not valid JSON: ${ + e instanceof Error ? e.message : String(e) + }`; + checks.push( + skillsCheck( + 'sep-2640-server-expose-index', + 'Server SHOULD expose a readable skill://index.json whose content is a JSON index.', + 'FAILURE', + { errorMessage: reason } + ) + ); + for (const id of ENTRY_CHECK_IDS) { + checks.push( + skillsCheck(id, 'Index did not parse as JSON.', 'SKIPPED', { + errorMessage: reason + }) + ); + } + return checks; + } + + checks.push( + skillsCheck( + 'sep-2640-server-expose-index', + 'Server SHOULD expose a readable skill://index.json whose content is a JSON index of the skills it serves.', + 'SUCCESS', + { + details: { + uri: SKILL_INDEX_URI, + mimeType: read.mimeType, + skillCount: Array.isArray(index.skills) ? index.skills.length : 0 + } + } + ) + ); + + const entries: SkillIndexEntry[] = Array.isArray(index.skills) + ? index.skills + : []; + + // An exposed-but-empty index is valid: a partial/empty index is + // permitted, and hosts MUST NOT read "no skills" from it. Nothing to + // validate at the entry level, so SKIP those checks cleanly. + if (entries.length === 0) { + const reason = + 'skill://index.json is exposed but lists no skills; entry-level checks not applicable (an empty index is permitted).'; + for (const id of ENTRY_CHECK_IDS) { + checks.push( + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + return checks; + } + + // === index-entry-type-enum (MUST) === + const typeErrs = entries + .map((e, i) => + SKILL_TYPES.includes(e.type as (typeof SKILL_TYPES)[number]) + ? null + : `skills[${i}].type=${JSON.stringify(e.type)} is not one of ${SKILL_TYPES.join('|')}` + ) + .filter((x): x is string => x !== null); + checks.push( + skillsCheck( + 'sep-2640-index-entry-type-enum', + 'Every skills[].type MUST be "skill-md" or "archive".', + typeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + typeErrs.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: typeErrs.join('; ') } + ) + ); + + // === index-name-required (required field) === + const nameErrs = entries + .map((e, i) => + typeof e.name === 'string' && e.name.length > 0 + ? null + : `skills[${i}].name is missing or empty` + ) + .filter((x): x is string => x !== null); + checks.push( + skillsCheck( + 'sep-2640-index-name-required', + 'Every index entry carries a non-empty name (matching the SKILL.md frontmatter name and the final skill-path segment).', + nameErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + nameErrs.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: nameErrs.join('; ') } + ) + ); + + // === index-digest-required (MUST — validate the format when present) === + const withDigest = entries.filter((e) => e.digest !== undefined); + const digestErrs = withDigest + .map((e, i) => + typeof e.digest === 'string' && SKILL_DIGEST_PATTERN.test(e.digest) + ? null + : `skills[${i}].digest=${JSON.stringify(e.digest)} is not sha256:{64 lowercase hex}` + ) + .filter((x): x is string => x !== null); + checks.push( + skillsCheck( + 'sep-2640-index-digest-required', + 'Every present skills[].digest MUST be formatted as sha256:{hex} with 64 lowercase hex characters.', + digestErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + digestErrs.length === 0 + ? { + details: { + entriesWithDigest: withDigest.length, + entriesWithoutDigest: entries.length - withDigest.length + } + } + : { errorMessage: digestErrs.join('; ') } + ) + ); + + // === skill-uri-scheme (SHOULD) === + // Servers SHOULD use skill://; another scheme is permitted only when the + // skill is listed in the index (SEP-2640 §URI convention), so a non- + // skill:// URL is a SHOULD deviation, not a hard failure. + const nonSkillScheme = entries + .map((e, i) => + typeof e.url === 'string' && !e.url.startsWith(SKILL_URI_SCHEME) + ? `skills[${i}].url=${JSON.stringify(e.url)}` + : null + ) + .filter((x): x is string => x !== null); + checks.push( + skillsCheck( + 'sep-2640-skill-uri-scheme', + 'Skill resource URLs in the index SHOULD use the skill:// URI scheme.', + nonSkillScheme.length === 0 ? 'SUCCESS' : 'WARNING', + nonSkillScheme.length === 0 + ? { details: { entryCount: entries.length } } + : { + errorMessage: `Entries use a non-skill:// scheme (permitted only when indexed): ${nonSkillScheme.join(', ')}` + } + ) + ); + + return checks; + } finally { + await conn.close(); + } + } +} diff --git a/src/scenarios/server/skills/manifest.ts b/src/scenarios/server/skills/manifest.ts new file mode 100644 index 00000000..7a4f1fd6 --- /dev/null +++ b/src/scenarios/server/skills/manifest.ts @@ -0,0 +1,367 @@ +/** + * SEP-2640 Skills extension — the `SKILL.md` manifest resource. + * + * One scenario, many checks (per AGENTS.md "fewer scenarios, more checks"). + * Each check's verbatim spec quote lives next to its check ID in + * src/seps/sep-2640.yaml. + * + * Discovery is dynamic and brand-neutral: the scenario finds a `skill-md` + * skill's `SKILL.md` resource from `resources/list` (preferred — it carries the + * Resource `name`/`description` metadata) or falls back to the first `skill-md` + * entry in `skill://index.json`, hardcoding no fixture skill. Undeclared + * extension SKIPs; a declared extension with no discoverable `SKILL.md` reports + * the missing prerequisite via untestableCheck (issue #248), never a silent + * green. + */ + +import { ClientScenario, ConformanceCheck } from '../../../types'; +import { JsonRpcError, type RunContext } from '../../../connection'; +import { untestableCheck } from '../../untestable'; +import { + SKILLS_EXTENSION_ID, + SKILLS_META_PREFIX, + SEP_2640_REF, + type SkillIndex, + type SkillResource, + skillsCapability, + skillsCheck, + listAllResources, + readSkillIndexText, + readResourceText, + skillNameFromManifestUri, + parseFrontmatter +} from './helpers'; + +const MIMETYPE_ID = 'sep-2640-skillmd-mimetype'; +const METADATA_NAME_ID = 'sep-2640-skillmd-metadata-name'; +const METADATA_DESCRIPTION_ID = 'sep-2640-skillmd-metadata-description'; +const FINAL_SEGMENT_ID = 'sep-2640-final-segment-equals-name'; +const META_PREFIX_ID = 'sep-2640-meta-prefix'; + +const MARKDOWN_MIME = 'text/markdown'; + +/** A SKILL.md resource URI is skill:///SKILL.md. */ +function isManifestUri(uri: string): boolean { + return skillNameFromManifestUri(uri) !== undefined; +} + +/** A `_meta` key that already carries a reverse-domain namespace (`vendor.tld/…`). */ +function isNamespacedMetaKey(key: string): boolean { + return /^[a-z0-9-]+(\.[a-z0-9-]+)+\//i.test(key); +} + +export class SkillsManifestScenario implements ClientScenario { + name = 'sep-2640-skills-manifest'; + readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; + description = `SEP-2640 Skills extension: the \`SKILL.md\` manifest resource. + +**Resource**: \`skill:///SKILL.md\` (read via \`resources/read\`) + +**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): + +- \`sep-2640-skillmd-mimetype\` — the SKILL.md resource \`mimeType\` SHOULD be \`text/markdown\` +- \`sep-2640-skillmd-metadata-name\` — the resource \`name\` SHOULD be the frontmatter \`name\` +- \`sep-2640-skillmd-metadata-description\` — the resource \`description\` SHOULD be the frontmatter \`description\` +- \`sep-2640-final-segment-equals-name\` — the final \`\` segment MUST equal the frontmatter \`name\` +- \`sep-2640-meta-prefix\` — un-namespaced skill \`_meta\` keys SHOULD use the \`io.modelcontextprotocol.skills/\` prefix + +**Discovery is dynamic**: the scenario picks the first \`skill-md\` skill it finds. Undeclared extension SKIPs; a declared extension with no discoverable SKILL.md reports the missing prerequisite (not a silent skip).`; + + async run(ctx: RunContext): Promise { + const conn = await ctx.connect(); + try { + const skills = await skillsCapability(conn); + const allIds = [ + MIMETYPE_ID, + METADATA_NAME_ID, + METADATA_DESCRIPTION_ID, + FINAL_SEGMENT_ID, + META_PREFIX_ID + ]; + if (!skills) { + const reason = + 'Server did not declare the io.modelcontextprotocol/skills extension; SKILL.md checks not applicable.'; + return allIds.map((id) => + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + + // === Dynamic discovery: resources/list first (carries Resource + // metadata), then skill://index.json. === + const resources = await listAllResources(conn); + const manifestResource: SkillResource | undefined = resources.find((r) => + isManifestUri(r.uri) + ); + let manifestUri = manifestResource?.uri; + if (!manifestUri) { + const idx = await readSkillIndexText(conn); + if (!('error' in idx) && typeof idx.text === 'string') { + try { + const index = JSON.parse(idx.text) as SkillIndex; + const entry = (index.skills ?? []).find( + (e) => + e.type === 'skill-md' && + typeof e.url === 'string' && + isManifestUri(e.url) + ); + manifestUri = entry?.url; + } catch { + // A malformed index is the index scenario's concern; ignore here. + } + } + } + + if (!manifestUri) { + const reason = + 'no skill:///SKILL.md resource found via resources/list or skill://index.json'; + return [ + untestableCheck( + MIMETYPE_ID, + MIMETYPE_ID, + 'SKILL.md resource mimeType SHOULD be text/markdown.', + reason, + [SEP_2640_REF], + 'WARNING' + ), + untestableCheck( + METADATA_NAME_ID, + METADATA_NAME_ID, + 'SKILL.md resource name SHOULD match the frontmatter name.', + reason, + [SEP_2640_REF], + 'WARNING' + ), + untestableCheck( + METADATA_DESCRIPTION_ID, + METADATA_DESCRIPTION_ID, + 'SKILL.md resource description SHOULD match the frontmatter description.', + reason, + [SEP_2640_REF], + 'WARNING' + ), + untestableCheck( + FINAL_SEGMENT_ID, + FINAL_SEGMENT_ID, + 'The final segment MUST equal the frontmatter name.', + reason, + [SEP_2640_REF], + 'FAILURE' + ), + untestableCheck( + META_PREFIX_ID, + META_PREFIX_ID, + 'Skill _meta keys SHOULD use the io.modelcontextprotocol.skills/ prefix.', + reason, + [SEP_2640_REF], + 'WARNING' + ) + ]; + } + + const checks: ConformanceCheck[] = []; + + // Read the manifest content (for mimeType, frontmatter, and _meta). + let content: + | { text: string; mimeType?: string; meta?: Record } + | undefined; + let readError: string | undefined; + try { + content = await readResourceText(conn, manifestUri); + if (!content) readError = 'resources/read returned no text content'; + } catch (e) { + readError = + e instanceof JsonRpcError + ? `resources/read failed: code ${e.code}: ${e.message}` + : e instanceof Error + ? e.message + : String(e); + } + + // === skillmd-mimetype (SHOULD) === + // Prefer the read content's mimeType; fall back to the resources/list + // Resource metadata mimeType. + const mimeType = content?.mimeType ?? manifestResource?.mimeType; + if (mimeType === undefined) { + checks.push( + untestableCheck( + MIMETYPE_ID, + MIMETYPE_ID, + 'SKILL.md resource mimeType SHOULD be text/markdown.', + `no mimeType observable for ${manifestUri}${readError ? ` (${readError})` : ''}`, + [SEP_2640_REF], + 'WARNING' + ) + ); + } else { + checks.push( + skillsCheck( + MIMETYPE_ID, + 'SKILL.md resource mimeType SHOULD be text/markdown.', + mimeType === MARKDOWN_MIME ? 'SUCCESS' : 'WARNING', + mimeType === MARKDOWN_MIME + ? { details: { uri: manifestUri, mimeType } } + : { + errorMessage: `expected mimeType "${MARKDOWN_MIME}", got ${JSON.stringify(mimeType)}` + } + ) + ); + } + + // Parse the frontmatter once for the name/description/final-segment checks. + const frontmatter = content ? parseFrontmatter(content.text) : undefined; + const fmName = + typeof frontmatter?.name === 'string' ? frontmatter.name : undefined; + const fmDescription = + typeof frontmatter?.description === 'string' + ? frontmatter.description + : undefined; + + // === final-segment-equals-name (MUST) === + const uriName = skillNameFromManifestUri(manifestUri); + if (fmName === undefined || uriName === undefined) { + const missing = + fmName === undefined + ? `SKILL.md frontmatter has no string "name"${readError ? ` (${readError})` : ''}` + : `could not derive the skill name from URI ${manifestUri}`; + checks.push( + untestableCheck( + FINAL_SEGMENT_ID, + FINAL_SEGMENT_ID, + 'The final segment MUST equal the frontmatter name.', + missing, + [SEP_2640_REF], + 'FAILURE' + ) + ); + } else { + checks.push( + skillsCheck( + FINAL_SEGMENT_ID, + 'The final segment of the SKILL.md URI MUST equal the frontmatter name.', + uriName === fmName ? 'SUCCESS' : 'FAILURE', + uriName === fmName + ? { details: { uri: manifestUri, name: fmName } } + : { + errorMessage: `final path segment "${uriName}" != frontmatter name "${fmName}"` + } + ) + ); + } + + // === skillmd-metadata-name (SHOULD) — needs the Resource metadata === + if (!manifestResource) { + checks.push( + untestableCheck( + METADATA_NAME_ID, + METADATA_NAME_ID, + 'SKILL.md resource name SHOULD match the frontmatter name.', + `SKILL.md ${manifestUri} is not listed in resources/list, so its Resource name metadata is not observable`, + [SEP_2640_REF], + 'WARNING' + ) + ); + } else if (fmName === undefined) { + checks.push( + untestableCheck( + METADATA_NAME_ID, + METADATA_NAME_ID, + 'SKILL.md resource name SHOULD match the frontmatter name.', + `SKILL.md frontmatter has no string "name" to compare against${readError ? ` (${readError})` : ''}`, + [SEP_2640_REF], + 'WARNING' + ) + ); + } else { + checks.push( + skillsCheck( + METADATA_NAME_ID, + 'The SKILL.md resource name SHOULD be set from the frontmatter name.', + manifestResource.name === fmName ? 'SUCCESS' : 'WARNING', + manifestResource.name === fmName + ? { details: { name: fmName } } + : { + errorMessage: `resource name ${JSON.stringify(manifestResource.name)} != frontmatter name ${JSON.stringify(fmName)}` + } + ) + ); + } + + // === skillmd-metadata-description (SHOULD) — needs Resource metadata === + if (!manifestResource) { + checks.push( + untestableCheck( + METADATA_DESCRIPTION_ID, + METADATA_DESCRIPTION_ID, + 'SKILL.md resource description SHOULD match the frontmatter description.', + `SKILL.md ${manifestUri} is not listed in resources/list, so its Resource description metadata is not observable`, + [SEP_2640_REF], + 'WARNING' + ) + ); + } else if (fmDescription === undefined) { + checks.push( + untestableCheck( + METADATA_DESCRIPTION_ID, + METADATA_DESCRIPTION_ID, + 'SKILL.md resource description SHOULD match the frontmatter description.', + `SKILL.md frontmatter has no string "description" to compare against${readError ? ` (${readError})` : ''}`, + [SEP_2640_REF], + 'WARNING' + ) + ); + } else { + checks.push( + skillsCheck( + METADATA_DESCRIPTION_ID, + 'The SKILL.md resource description SHOULD be set from the frontmatter description.', + manifestResource.description === fmDescription + ? 'SUCCESS' + : 'WARNING', + manifestResource.description === fmDescription + ? { details: { description: fmDescription } } + : { + errorMessage: `resource description ${JSON.stringify(manifestResource.description)} != frontmatter description ${JSON.stringify(fmDescription)}` + } + ) + ); + } + + // === meta-prefix (SHOULD, conditional on _meta keys being present) === + // Union the _meta of the read content and the resources/list Resource. + const metaKeys = new Set([ + ...Object.keys(content?.meta ?? {}), + ...Object.keys(manifestResource?._meta ?? {}) + ]); + if (metaKeys.size === 0) { + checks.push( + skillsCheck( + META_PREFIX_ID, + 'When _meta keys are used for skill resources, they SHOULD use the io.modelcontextprotocol.skills/ reverse-domain prefix.', + 'SUCCESS', + { details: { note: 'skill resource exposes no _meta keys' } } + ) + ); + } else { + // Only bare (un-namespaced) keys are flagged: a key already carrying a + // reverse-domain namespace is the intended shape, whichever vendor. + const bareKeys = [...metaKeys].filter((k) => !isNamespacedMetaKey(k)); + checks.push( + skillsCheck( + META_PREFIX_ID, + 'When _meta keys are used for skill resources, they SHOULD use the io.modelcontextprotocol.skills/ reverse-domain prefix.', + bareKeys.length === 0 ? 'SUCCESS' : 'WARNING', + bareKeys.length === 0 + ? { details: { metaKeys: [...metaKeys] } } + : { + errorMessage: `un-namespaced skill _meta keys SHOULD use the ${SKILLS_META_PREFIX} prefix: ${bareKeys.join(', ')}` + } + ) + ); + } + + return checks; + } finally { + await conn.close(); + } + } +} diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 024d06c3..d93e1487 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -10,6 +10,20 @@ # drifted ~11 others' verbatim wording. Full re-extraction at SEP HEAD # is mcpkit#780's lifecycle; this file deliberately holds at 556154c # until that lands. +# backing_scenarios: three server ClientScenarios under +# src/scenarios/server/skills/ emit the check IDs below (a row is "tested" +# once a scenario emits its check ID; see src/traceability/): +# directory.ts (sep-2640-skills-directory) — sep-2640-capability-directory-read-flag +# and the five sep-2640-directory-read-* rows. +# index.ts (sep-2640-skills-index) — sep-2640-server-expose-index, +# sep-2640-index-entry-type-enum, sep-2640-index-name-required, +# sep-2640-index-digest-required, sep-2640-skill-uri-scheme. +# manifest.ts (sep-2640-skills-manifest) — sep-2640-skillmd-mimetype, +# sep-2640-skillmd-metadata-name, sep-2640-skillmd-metadata-description, +# sep-2640-final-segment-equals-name, sep-2640-meta-prefix. +# The remaining rows are host-internal or off-wire (host load-by-uri, digest +# verification, byte-budget / archive-unpack safety, no-empty-index +# assumption) and stay traceability-only for this server-scenario set. sep: 2640 spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification requirements: From 0707a3eb85a12ab40706bd95f563b807ea12a0de Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Fri, 28 Aug 2026 17:15:54 -0700 Subject: [PATCH 07/25] feat(sep-2640): re-extract against the 2026-08-21 rewrite The prior extraction held at SEP commit 556154c (2026-06-05) by an explicit in-file decision to wait for the index-schema refactor. The 08-21 core-maintainer rewrite made that pin actively wrong rather than merely behind. Removed as no longer normative: the skill://index.json well-known resource (absent from the current text), the entry type enum, the no-nested-skills rule (now reversed - skills MAY nest), and the five archive rows (archives are deferred to an appendix). Reworded the two index-field rows, the empty-listing row and the verify-digest row. Added the new surface: skills/list and skills/get with their pagination, atomicity and caching attributes; the {uri, frontmatter, resources} entry schema; resources completeness, the "dynamic" sentinel and per-file size; the 512-entry and 16 MiB limits; and the host obligations around lazy retrieval, content-bound approval and frontmatter re-verification. index.ts is replaced by enumeration.ts, which drives both new methods. manifest.ts and directory.ts swap their skill://index.json discovery fallback for skills/list. The capability helper no longer accepts a config envelope. SEP-2133 is Final and defines the extensions field as a map of identifiers to settings objects, with no envelope and no slot for id/specVersion/stability; SEP-2640's capability block matches. The earlier leniency was based on a misreading. 77 rows: 65 checks (39 wire-tested by the three server scenarios) and 12 excluded as host-internal. --- src/scenarios/index.ts | 6 +- src/scenarios/server/skills/directory.ts | 36 +- src/scenarios/server/skills/enumeration.ts | 1020 ++++++++++++++++++++ src/scenarios/server/skills/helpers.ts | 238 ++++- src/scenarios/server/skills/index.ts | 270 ------ src/scenarios/server/skills/manifest.ts | 23 +- src/seps/sep-2640.yaml | 228 ++++- 7 files changed, 1414 insertions(+), 407 deletions(-) create mode 100644 src/scenarios/server/skills/enumeration.ts delete mode 100644 src/scenarios/server/skills/index.ts diff --git a/src/scenarios/index.ts b/src/scenarios/index.ts index 8fe9a610..52c8ed0f 100644 --- a/src/scenarios/index.ts +++ b/src/scenarios/index.ts @@ -59,7 +59,7 @@ import { } from './server/resources'; import { SkillsDirectoryReadScenario } from './server/skills/directory'; -import { SkillsIndexScenario } from './server/skills/index'; +import { SkillsEnumerationScenario } from './server/skills/enumeration'; import { SkillsManifestScenario } from './server/skills/manifest'; import { @@ -162,7 +162,7 @@ const pendingClientScenariosList: ClientScenario[] = [ // SEP-2640-conformant fixture via // `npm start -- server --scenario sep-2640-skills-* --url `. new SkillsDirectoryReadScenario(), - new SkillsIndexScenario(), + new SkillsEnumerationScenario(), new SkillsManifestScenario() ]; @@ -218,7 +218,7 @@ const allClientScenariosList: ClientScenario[] = [ // Skills extension (SEP-2640). Fixture-dependent (needs a SEP-2640 server); // each scenario SKIPs cleanly when the extension is not declared. new SkillsDirectoryReadScenario(), - new SkillsIndexScenario(), + new SkillsEnumerationScenario(), new SkillsManifestScenario(), // Prompts scenarios diff --git a/src/scenarios/server/skills/directory.ts b/src/scenarios/server/skills/directory.ts index 27be77bd..4c5c2b02 100644 --- a/src/scenarios/server/skills/directory.ts +++ b/src/scenarios/server/skills/directory.ts @@ -27,13 +27,12 @@ import { SEP_2640_REF, JSONRPC_METHOD_NOT_FOUND, JSONRPC_INVALID_PARAMS, - type SkillIndex, type SkillResource, skillsCapability, directoryReadDeclared, skillsCheck, listAllResources, - readSkillIndexText, + skillsListAll, skillNameFromManifestUri } from './helpers'; @@ -81,26 +80,19 @@ function skillRootFromManifestUri(uri: string): string | undefined { async function discoverDirectory( conn: Connection ): Promise { - // 1. skill-md entry in skill://index.json — its SKILL.md URL gives us both a - // directory (the skill root) and a known file (the SKILL.md itself). - const idx = await readSkillIndexText(conn); - if (!('error' in idx) && typeof idx.text === 'string') { - try { - const index = JSON.parse(idx.text) as SkillIndex; - const entry = (index.skills ?? []).find( - (e) => - e.type === 'skill-md' && - typeof e.url === 'string' && - skillRootFromManifestUri(e.url) !== undefined - ); - if (entry?.url) { - return { - dirUri: skillRootFromManifestUri(entry.url)!, - fileUri: entry.url - }; - } - } catch { - // A malformed index is the index scenario's concern; keep discovering. + // 1. A skills/list entry — its SKILL.md URI gives us both a directory (the + // skill root) and a known file (the SKILL.md itself). An unenumerable + // catalog returns nothing here, so discovery falls through. + const listed = await skillsListAll(conn); + if (!('error' in listed)) { + const entry = listed.entries.find( + (e) => + typeof e.uri === 'string' && + skillRootFromManifestUri(e.uri) !== undefined + ); + const uri = entry?.uri as string | undefined; + if (uri) { + return { dirUri: skillRootFromManifestUri(uri)!, fileUri: uri }; } } diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts new file mode 100644 index 00000000..837a1568 --- /dev/null +++ b/src/scenarios/server/skills/enumeration.ts @@ -0,0 +1,1020 @@ +/** + * SEP-2640 Skills extension — the `skills/list` and `skills/get` surface. + * + * Replaces the former `skill://index.json` scenario. The 2026-08-21 revision of + * the SEP removed that well-known resource entirely (it appears nowhere in the + * current text) and replaced it with two methods that every server declaring the + * extension MUST implement. + * + * One scenario, many checks (per AGENTS.md "fewer scenarios, more checks"). + * Each check's verbatim spec quote lives next to its check ID in + * src/seps/sep-2640.yaml, keeping the YAML and this scenario in lock-step. + * + * All discovery is dynamic and brand-neutral: the scenario enumerates whatever + * the server serves and validates the entries it finds, hardcoding no + * fixture-specific skill name or URI. When the server does not declare the + * skills extension the checks are SKIPPED (an optional, undeclared capability). + * An empty listing is explicitly permitted, so entry-level checks SKIP rather + * than fail against a server with an unenumerable catalog. + */ + +import { ClientScenario, ConformanceCheck } from '../../../types'; +import type { RunContext } from '../../../connection'; +import { + SKILLS_EXTENSION_ID, + SKILLS_LIST_METHOD, + SKILLS_GET_METHOD, + SKILL_URI_SCHEME, + SKILL_MANIFEST_FILENAME, + SKILL_DIGEST_PATTERN, + RESOURCES_DYNAMIC, + MAX_RESOURCES_PER_SKILL, + MAX_TOTAL_SIZE_PER_SKILL, + FRONTMATTER_RESERVED_PREFIX, + JSONRPC_INVALID_PARAMS, + type SkillEntry, + type SkillResourceEntry, + skillsCapability, + skillsCheck, + skillsListAll, + skillsGet, + settingsAreInline, + skillNameFromManifestUri, + skillRootFromManifestUri, + isDynamicResources, + resourcesArray, + entryLabel, + readResourceText, + parseFrontmatter +} from './helpers'; + +const CAPABILITY_IDS = [ + 'sep-2640-capability-declaration-inline', + 'sep-2640-capability-commits-to-methods', + 'sep-2640-capability-empty-object' +] as const; + +const LIST_IDS = [ + 'sep-2640-skills-list-implemented', + 'sep-2640-skills-list-pagination', + 'sep-2640-skills-list-entry-atomic', + 'sep-2640-skills-list-cache-attributes' +] as const; + +const ENTRY_IDS = [ + 'sep-2640-entry-uri-required', + 'sep-2640-entry-frontmatter-required', + 'sep-2640-entry-uri-matches-frontmatter-name', + 'sep-2640-skill-uri-scheme', + 'sep-2640-entry-resources-required', + 'sep-2640-resources-complete', + 'sep-2640-resources-uri-within-skill', + 'sep-2640-resources-digest-format', + 'sep-2640-resources-size-required', + 'sep-2640-limit-resources-per-skill', + 'sep-2640-limit-total-size', + 'sep-2640-metadata-reserved-prefix', + 'sep-2640-name-naming-rules', + 'sep-2640-authority-reg-name' +] as const; + +const GET_IDS = [ + 'sep-2640-skills-get-implemented', + 'sep-2640-skills-get-entry-shape', + 'sep-2640-skills-get-no-cursor', + 'sep-2640-skills-get-unknown-uri-invalid-params' +] as const; + +/** Emitted by the read-back pass, which fetches one listed SKILL.md. */ +const READBACK_IDS = [ + 'sep-2640-skillmd-required', + 'sep-2640-skillmd-frontmatter', + 'sep-2640-entry-frontmatter-identical' +] as const; + +const ALL_CHECK_IDS = [ + ...CAPABILITY_IDS, + ...LIST_IDS, + ...ENTRY_IDS, + ...GET_IDS, + ...READBACK_IDS +]; + +/** + * Agent Skills naming rules as the SEP defers to them: 1-64 characters, + * lowercase alphanumeric and hyphens. + */ +const SKILL_NAME_PATTERN = /^[a-z0-9]([a-z0-9-]{0,62}[a-z0-9])?$/; + +/** RFC 3986 reg-name: unreserved / pct-encoded / sub-delims, case-insensitive. */ +const REG_NAME_PATTERN = /^(?:[A-Za-z0-9\-._~!$&'()*+,;=]|%[0-9A-Fa-f]{2})*$/; + +/** A URI that no conformant server should serve, for the -32602 probe. */ +const UNKNOWN_SKILL_URI = + 'skill://mcp-conformance-nonexistent-skill-9f3a2b/SKILL.md'; + +function joinErrs(errs: string[], limit = 5): string { + const shown = errs.slice(0, limit).join('; '); + return errs.length > limit + ? `${shown} (+${errs.length - limit} more)` + : shown; +} + +export class SkillsEnumerationScenario implements ClientScenario { + name = 'sep-2640-skills-enumeration'; + readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; + description = `SEP-2640 Skills extension: \`skills/list\` enumeration and \`skills/get\` retrieval. + +**Methods**: \`skills/list\`, \`skills/get\` (both mandatory for a server declaring \`io.modelcontextprotocol/skills\`) + +**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): + +- \`sep-2640-capability-declaration-inline\` — extension settings sit inline under the identifier, per SEP-2133 (no \`config\` envelope) +- \`sep-2640-capability-commits-to-methods\` — declaring the extension commits the server to both methods +- \`sep-2640-skills-list-implemented\` — \`skills/list\` is implemented and returns a \`skills\` array +- \`sep-2640-skills-list-pagination\` — \`nextCursor\` is honoured as a cursor on the next request +- \`sep-2640-skills-list-entry-atomic\` — no skill entry is split across pages +- \`sep-2640-entry-uri-required\` / \`sep-2640-entry-frontmatter-required\` / \`sep-2640-entry-resources-required\` — the three required entry fields +- \`sep-2640-entry-uri-matches-frontmatter-name\` — the final skill-path segment equals \`frontmatter.name\` +- \`sep-2640-resources-complete\` — \`resources\` includes an entry matching the skill's own \`uri\`, each file once +- \`sep-2640-resources-uri-within-skill\` / \`sep-2640-resources-digest-format\` / \`sep-2640-resources-size-required\` — the \`{uri, digest, size}\` triple +- \`sep-2640-limit-resources-per-skill\` / \`sep-2640-limit-total-size\` — 512 entries, 16 MiB +- \`sep-2640-metadata-reserved-prefix\` — frontmatter \`metadata\` keys under \`io.modelcontextprotocol/\` are reserved +- \`sep-2640-skills-get-*\` — \`skills/get\` returns a list-shaped entry, carries no cursor, and answers \`-32602\` for an unknown URI + +**Discovery is dynamic**: an undeclared extension SKIPs everything; an empty or partial listing is permitted and SKIPs the entry-level checks.`; + + async run(ctx: RunContext): Promise { + const conn = await ctx.connect(); + try { + const skills = await skillsCapability(conn); + if (!skills) { + const reason = + 'Server did not declare the io.modelcontextprotocol/skills extension; enumeration checks not applicable.'; + return ALL_CHECK_IDS.map((id) => + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + + const checks: ConformanceCheck[] = []; + + // === capability-declaration-inline === + // SEP-2133 (Final) maps an extension identifier straight to its settings + // object. An envelope hides settings from any spec-following client. + const { inline, envelopeKeys } = settingsAreInline(skills); + checks.push( + skillsCheck( + 'sep-2640-capability-declaration-inline', + 'Extension settings are a map of extension identifiers to per-extension settings objects; the settings sit directly under the identifier.', + inline ? 'SUCCESS' : 'FAILURE', + inline + ? { details: { settingKeys: Object.keys(skills) } } + : { + errorMessage: `capabilities.extensions["${SKILLS_EXTENSION_ID}"] carries envelope key(s) ${envelopeKeys.join(', ')} instead of the settings object itself. SEP-2133 (Final) defines no envelope, and SEP-2640's capability block places directoryRead inline.`, + details: { envelopeKeys, observed: skills } + } + ) + ); + + // === capability-empty-object === + // "An empty object indicates support for the extension with no optional + // features." Observable as: the declared value is a JSON object, and an + // empty one is a valid declaration rather than a malformed capability. + checks.push( + skillsCheck( + 'sep-2640-capability-empty-object', + 'An empty object indicates support for the extension with no optional features.', + 'SUCCESS', + { + details: { + declaredKeys: Object.keys(skills), + empty: Object.keys(skills).length === 0 + } + } + ) + ); + + // === skills/list === + const listed = await skillsListAll(conn); + if ('error' in listed) { + const reason = `${SKILLS_LIST_METHOD} failed with code ${listed.error.code}: ${listed.error.message}. A server declaring the extension MUST implement it.`; + checks.push( + skillsCheck( + 'sep-2640-capability-commits-to-methods', + 'Declaring the extension commits the server to skills/list and skills/get.', + 'FAILURE', + { errorMessage: reason } + ), + skillsCheck( + 'sep-2640-skills-list-implemented', + 'A server declaring the extension MUST implement the skills/list method.', + 'FAILURE', + { errorMessage: reason } + ) + ); + for (const id of [ + 'sep-2640-skills-list-pagination', + 'sep-2640-skills-list-entry-atomic', + ...ENTRY_IDS, + ...GET_IDS, + ...READBACK_IDS + ]) { + checks.push( + skillsCheck(id, 'skills/list is unavailable.', 'SKIPPED', { + errorMessage: reason + }) + ); + } + return checks; + } + + const { entries, pages, truncated } = listed; + + checks.push( + skillsCheck( + 'sep-2640-skills-list-implemented', + 'A server declaring the extension MUST implement the skills/list method, which returns the skills it serves. The result MAY be empty.', + 'SUCCESS', + { + details: { + pages: pages.length, + entries: entries.length, + emptyListingPermitted: entries.length === 0 + } + } + ) + ); + + // === skills-list-pagination === + // Multi-page runs prove the cursor round-trips. A single page is a clean + // pass: the contract is "when nextCursor is present, pass it back", and + // skillsListAll did exactly that to reach the end. + checks.push( + skillsCheck( + 'sep-2640-skills-list-pagination', + 'Pagination mirrors the base protocol: the request accepts an optional cursor, and when the result includes nextCursor the client passes it back.', + truncated ? 'FAILURE' : 'SUCCESS', + truncated + ? { + errorMessage: `skills/list did not terminate: the server kept returning a nextCursor (or repeated one) across ${pages.length} pages.` + } + : { details: { pages: pages.length } } + ) + ); + + // === skills-list-entry-atomic === + // "An entry is atomic — a skill's resources set is never split across + // pages." Observable as a URI appearing in more than one page. + const uriPages = new Map(); + pages.forEach((page, pageIdx) => { + for (const e of page.entries) { + if (typeof e.uri !== 'string') continue; + const seen = uriPages.get(e.uri) ?? []; + if (!seen.includes(pageIdx)) seen.push(pageIdx); + uriPages.set(e.uri, seen); + } + }); + const split = [...uriPages.entries()] + .filter(([, p]) => p.length > 1) + .map(([uri, p]) => `${uri} appears on pages ${p.join(', ')}`); + checks.push( + skillsCheck( + 'sep-2640-skills-list-entry-atomic', + "An entry is atomic — a skill's resources set is never split across pages.", + split.length === 0 ? 'SUCCESS' : 'FAILURE', + split.length === 0 + ? { details: { pages: pages.length, distinctUris: uriPages.size } } + : { errorMessage: joinErrs(split) } + ) + ); + + // === skills-list-cache-attributes === + // SEP-2549 attributes are required only on protocol 2026-07-28 and later. + // The harness does not gate on the negotiated version here, so an absent + // attribute is reported as a WARNING rather than a failure. + const first = pages[0]?.result ?? {}; + const hasTtl = first.ttlMs !== undefined; + const hasScope = first.cacheScope !== undefined; + checks.push( + skillsCheck( + 'sep-2640-skills-list-cache-attributes', + "In protocol versions 2026-07-28 and later, the skills/list result carries the base protocol's list-caching attributes ttlMs and cacheScope (SEP-2549).", + hasTtl && hasScope ? 'SUCCESS' : 'WARNING', + hasTtl && hasScope + ? { details: { ttlMs: first.ttlMs, cacheScope: first.cacheScope } } + : { + errorMessage: `skills/list result omits ${[!hasTtl && 'ttlMs', !hasScope && 'cacheScope'].filter(Boolean).join(' and ')}. Required only on protocol 2026-07-28 and later; on an earlier negotiated version this is expected.` + } + ) + ); + + if (entries.length === 0) { + const reason = + 'skills/list returned no entries; entry-level checks not applicable. A server whose catalog is large, generated on demand, or otherwise unenumerable MAY return an empty listing.'; + for (const id of [...ENTRY_IDS, ...GET_IDS, ...READBACK_IDS]) { + checks.push( + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + checks.push( + skillsCheck( + 'sep-2640-capability-commits-to-methods', + 'Declaring the extension commits the server to skills/list and skills/get.', + 'SUCCESS', + { details: { note: 'skills/list answered; listing is empty.' } } + ) + ); + return checks; + } + + checks.push(...entryChecks(entries)); + checks.push(...(await getChecks(conn, entries))); + checks.push(...(await readbackChecks(conn, entries))); + + return checks; + } finally { + await conn.close(); + } + } +} + +/** Validate every `skills[]` entry against the §Discovery entry schema. */ +function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { + const checks: ConformanceCheck[] = []; + + // === entry-uri-required === + const uriErrs = entries + .map((e, i) => + typeof e.uri === 'string' && e.uri.length > 0 + ? null + : `skills[${i}].uri is missing or not a string` + ) + .filter((x): x is string => x !== null); + checks.push( + skillsCheck( + 'sep-2640-entry-uri-required', + "Every entry carries uri, the full resource URI of the skill's SKILL.md.", + uriErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + uriErrs.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: joinErrs(uriErrs) } + ) + ); + + // === entry-frontmatter-required === + // Verbatim frontmatter, so name and description are always present. + const fmErrs: string[] = []; + entries.forEach((e, i) => { + const fm = e.frontmatter; + if (!fm || typeof fm !== 'object' || Array.isArray(fm)) { + fmErrs.push(`${entryLabel(e, i)}: frontmatter is missing or not an object`); + return; + } + const obj = fm as Record; + if (typeof obj.name !== 'string' || obj.name.length === 0) { + fmErrs.push(`${entryLabel(e, i)}: frontmatter.name is missing or empty`); + } + if (typeof obj.description !== 'string' || obj.description.length === 0) { + fmErrs.push( + `${entryLabel(e, i)}: frontmatter.description is missing or empty` + ); + } + }); + checks.push( + skillsCheck( + 'sep-2640-entry-frontmatter-required', + "frontmatter is the skill's SKILL.md YAML frontmatter rendered verbatim as a JSON object; because the Agent Skills specification requires name and description, those fields are always present.", + fmErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + fmErrs.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: joinErrs(fmErrs) } + ) + ); + + // === entry-uri-matches-frontmatter-name === + const nameErrs: string[] = []; + entries.forEach((e) => { + if (typeof e.uri !== 'string') return; + const fm = e.frontmatter as Record | undefined; + const declared = fm && typeof fm.name === 'string' ? fm.name : undefined; + if (declared === undefined) return; + const fromUri = skillNameFromManifestUri(e.uri); + if (fromUri === undefined) { + nameErrs.push( + `${e.uri}: does not end in /${SKILL_MANIFEST_FILENAME}, so the skill name is not recoverable from the URI` + ); + } else if (fromUri !== declared) { + nameErrs.push( + `${e.uri}: final skill-path segment "${fromUri}" !== frontmatter.name "${declared}"` + ); + } + }); + checks.push( + skillsCheck( + 'sep-2640-entry-uri-matches-frontmatter-name', + "The final segment of the entry's uri MUST equal frontmatter.name.", + nameErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + nameErrs.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: joinErrs(nameErrs) } + ) + ); + + // === skill-uri-scheme (SHOULD) === + // "Servers SHOULD use the skill:// URI scheme", but a server MAY serve + // skills under another scheme native to its domain and no scheme is + // privileged, so a deviation is a WARNING rather than a failure. + const otherScheme = entries + .filter((e) => typeof e.uri === 'string' && !e.uri.startsWith(SKILL_URI_SCHEME)) + .map((e) => String(e.uri)); + checks.push( + skillsCheck( + 'sep-2640-skill-uri-scheme', + 'Servers SHOULD use the skill:// URI scheme for the resources of a skill.', + otherScheme.length === 0 ? 'SUCCESS' : 'WARNING', + otherScheme.length === 0 + ? { details: { entryCount: entries.length } } + : { + errorMessage: `Entries served under another scheme (explicitly permitted; no scheme is privileged): ${joinErrs(otherScheme)}` + } + ) + ); + + // === entry-resources-required === + const resErrs: string[] = []; + entries.forEach((e, i) => { + if (isDynamicResources(e)) return; + if (resourcesArray(e) !== undefined) return; + resErrs.push( + `${entryLabel(e, i)}: resources is ${JSON.stringify(e.resources)}, neither an array nor "${RESOURCES_DYNAMIC}"` + ); + }); + checks.push( + skillsCheck( + 'sep-2640-entry-resources-required', + 'resources is REQUIRED on every skill entry and takes one of two forms: an array of {uri, digest, size} triples, or the string "dynamic". An entry with no resources at all, or with any other value, is invalid.', + resErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + resErrs.length === 0 + ? { + details: { + entryCount: entries.length, + dynamicEntries: entries.filter(isDynamicResources).length + } + } + : { errorMessage: joinErrs(resErrs) } + ) + ); + + // Entries carrying an array are the only ones the remaining checks apply to. + const arrayEntries = entries + .map((e, i) => ({ e, i, arr: resourcesArray(e) })) + .filter((x): x is { e: SkillEntry; i: number; arr: SkillResourceEntry[] } => + x.arr !== undefined + ); + + const dynamicOnlyReason = + 'Every entry declares "resources": "dynamic", which publishes no file manifest; the resources-array checks are not applicable.'; + + if (arrayEntries.length === 0) { + for (const id of [ + 'sep-2640-resources-complete', + 'sep-2640-resources-uri-within-skill', + 'sep-2640-resources-digest-format', + 'sep-2640-resources-size-required', + 'sep-2640-limit-resources-per-skill', + 'sep-2640-limit-total-size' + ]) { + checks.push( + skillsCheck(id, dynamicOnlyReason, 'SKIPPED', { + errorMessage: dynamicOnlyReason + }) + ); + } + } else { + // === resources-complete === + // Observable half: an entry matching the skill's own uri, and no file + // listed twice. Full completeness (every file of the skill) cannot be + // confirmed from the wire without a second source of truth. + const completeErrs: string[] = []; + for (const { e, i, arr } of arrayEntries) { + const own = typeof e.uri === 'string' ? e.uri : undefined; + if (own && !arr.some((r) => r.uri === own)) { + completeErrs.push( + `${entryLabel(e, i)}: resources has no entry matching the skill's own uri (the SKILL.md digest and size)` + ); + } + const seen = new Set(); + const dupes = new Set(); + for (const r of arr) { + if (typeof r.uri !== 'string') continue; + if (seen.has(r.uri)) dupes.add(r.uri); + seen.add(r.uri); + } + if (dupes.size > 0) { + completeErrs.push( + `${entryLabel(e, i)}: resources lists ${[...dupes].join(', ')} more than once` + ); + } + } + checks.push( + skillsCheck( + 'sep-2640-resources-complete', + 'When present, resources MUST be complete: it lists every file of the skill, each exactly once, including an entry matching the skill top-level uri — that entry carries the digest and size of SKILL.md itself.', + completeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + completeErrs.length === 0 + ? { details: { entriesChecked: arrayEntries.length } } + : { errorMessage: joinErrs(completeErrs) } + ) + ); + + // === resources-uri-within-skill === + const containErrs: string[] = []; + for (const { e, i, arr } of arrayEntries) { + const own = typeof e.uri === 'string' ? e.uri : undefined; + const root = own ? skillRootFromManifestUri(own) : undefined; + if (!root) continue; + for (const r of arr) { + if (typeof r.uri !== 'string') { + containErrs.push(`${entryLabel(e, i)}: a resources entry has no uri`); + continue; + } + if (r.uri !== own && !r.uri.startsWith(`${root}/`)) { + containErrs.push( + `${entryLabel(e, i)}: ${r.uri} is outside the skill directory ${root}` + ); + } + } + } + checks.push( + skillsCheck( + 'sep-2640-resources-uri-within-skill', + "Each uri MUST be the skill's SKILL.md or a file within the skill's directory.", + containErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + containErrs.length === 0 + ? { details: { entriesChecked: arrayEntries.length } } + : { errorMessage: joinErrs(containErrs) } + ) + ); + + // === resources-digest-format === + const digestErrs: string[] = []; + for (const { e, i, arr } of arrayEntries) { + for (const r of arr) { + if (typeof r.digest !== 'string' || !SKILL_DIGEST_PATTERN.test(r.digest)) { + digestErrs.push( + `${entryLabel(e, i)}: ${String(r.uri)} digest=${JSON.stringify(r.digest)} is not sha256:{64 lowercase hex}` + ); + } + } + } + checks.push( + skillsCheck( + 'sep-2640-resources-digest-format', + "Digests are SHA-256 hashes of an artifact's raw bytes, formatted as sha256:{hex} where {hex} is 64 lowercase hexadecimal characters.", + digestErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + digestErrs.length === 0 + ? { details: { entriesChecked: arrayEntries.length } } + : { errorMessage: joinErrs(digestErrs) } + ) + ); + + // === resources-size-required === + const sizeErrs: string[] = []; + for (const { e, i, arr } of arrayEntries) { + for (const r of arr) { + if ( + typeof r.size !== 'number' || + !Number.isInteger(r.size) || + r.size < 0 + ) { + sizeErrs.push( + `${entryLabel(e, i)}: ${String(r.uri)} size=${JSON.stringify(r.size)} is not a non-negative integer` + ); + } + } + } + checks.push( + skillsCheck( + 'sep-2640-resources-size-required', + "Each entry MUST carry size: the length in bytes of the file's raw content — the same bytes the digest covers.", + sizeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + sizeErrs.length === 0 + ? { details: { entriesChecked: arrayEntries.length } } + : { errorMessage: joinErrs(sizeErrs) } + ) + ); + + // === limit-resources-per-skill (SHOULD NOT exceed) === + const overCount = arrayEntries + .filter(({ arr }) => arr.length > MAX_RESOURCES_PER_SKILL) + .map(({ e, i, arr }) => `${entryLabel(e, i)}: ${arr.length} entries`); + checks.push( + skillsCheck( + 'sep-2640-limit-resources-per-skill', + 'Servers SHOULD NOT serve a skill exceeding 512 resource entries, counted over the entries of the skill resources, SKILL.md included.', + overCount.length === 0 ? 'SUCCESS' : 'WARNING', + overCount.length === 0 + ? { + details: { + maxObserved: Math.max( + ...arrayEntries.map(({ arr }) => arr.length) + ), + limit: MAX_RESOURCES_PER_SKILL + } + } + : { errorMessage: joinErrs(overCount) } + ) + ); + + // === limit-total-size (SHOULD NOT exceed) === + const sums = arrayEntries.map(({ e, i, arr }) => ({ + label: entryLabel(e, i), + total: arr.reduce( + (acc, r) => acc + (typeof r.size === 'number' ? r.size : 0), + 0 + ) + })); + const overSize = sums + .filter((s) => s.total > MAX_TOTAL_SIZE_PER_SKILL) + .map((s) => `${s.label}: ${s.total} bytes`); + checks.push( + skillsCheck( + 'sep-2640-limit-total-size', + 'Servers SHOULD NOT serve a skill whose total file size exceeds 16 MiB (16,777,216 bytes), summed over the skill resources.', + overSize.length === 0 ? 'SUCCESS' : 'WARNING', + overSize.length === 0 + ? { + details: { + maxObservedBytes: Math.max(...sums.map((s) => s.total)), + limit: MAX_TOTAL_SIZE_PER_SKILL + } + } + : { errorMessage: joinErrs(overSize) } + ) + ); + } + + // === metadata-reserved-prefix === + // This extension currently defines no keys under the reserved prefix, so a + // server publishing one is squatting on a namespace reserved for MCP. + const reservedErrs: string[] = []; + entries.forEach((e, i) => { + const fm = e.frontmatter as Record | undefined; + const meta = fm?.metadata; + if (!meta || typeof meta !== 'object' || Array.isArray(meta)) return; + for (const key of Object.keys(meta as Record)) { + if (key.startsWith(FRONTMATTER_RESERVED_PREFIX)) { + reservedErrs.push(`${entryLabel(e, i)}: frontmatter.metadata["${key}"]`); + } + } + }); + checks.push( + skillsCheck( + 'sep-2640-metadata-reserved-prefix', + 'Within the frontmatter metadata object, keys prefixed with io.modelcontextprotocol/ are reserved for metadata defined by MCP extensions. This extension currently defines no such keys.', + reservedErrs.length === 0 ? 'SUCCESS' : 'WARNING', + reservedErrs.length === 0 + ? { details: { entryCount: entries.length } } + : { + errorMessage: `Keys under the reserved prefix, which this extension does not currently define: ${joinErrs(reservedErrs)}` + } + ) + ); + + // === name-naming-rules === + // The name is recoverable from the URI alone, so this is checkable without + // fetching anything. + const badNames: string[] = []; + entries.forEach((e, i) => { + if (typeof e.uri !== 'string') return; + const name = skillNameFromManifestUri(e.uri); + if (name !== undefined && !SKILL_NAME_PATTERN.test(name)) { + badNames.push(`${entryLabel(e, i)}: name "${name}"`); + } + }); + checks.push( + skillsCheck( + 'sep-2640-name-naming-rules', + "The final segment, being the skill name, MUST satisfy the Agent Skills specification's naming rules (1-64 characters, lowercase alphanumeric and hyphens).", + badNames.length === 0 ? 'SUCCESS' : 'FAILURE', + badNames.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: joinErrs(badNames) } + ) + ); + + // === authority-reg-name (SHOULD) === + const badAuthority: string[] = []; + entries.forEach((e, i) => { + if (typeof e.uri !== 'string') return; + const schemeEnd = e.uri.indexOf('://'); + if (schemeEnd < 0) return; + const segments = e.uri + .slice(schemeEnd + 3) + .split('/') + .filter((x) => x.length > 0); + const authority = segments[0]; + if (authority !== undefined && !REG_NAME_PATTERN.test(authority)) { + badAuthority.push(`${entryLabel(e, i)}: authority "${authority}"`); + } + }); + checks.push( + skillsCheck( + 'sep-2640-authority-reg-name', + 'The first segment occupies the authority component and SHOULD be a valid reg-name per RFC 3986.', + badAuthority.length === 0 ? 'SUCCESS' : 'WARNING', + badAuthority.length === 0 + ? { details: { entryCount: entries.length } } + : { errorMessage: joinErrs(badAuthority) } + ) + ); + + return checks; +} + +/** + * Fetch one listed `SKILL.md` and check it against the entry that advertised + * it. This is the server-side half of the host's frontmatter-comparison MUST: + * if the entry's `frontmatter` does not match the file, no conforming host can + * load the skill. + */ +async function readbackChecks( + conn: Parameters[0], + entries: SkillEntry[] +): Promise { + const checks: ConformanceCheck[] = []; + const sample = entries.find( + (e) => typeof e.uri === 'string' && e.uri.endsWith(`/${SKILL_MANIFEST_FILENAME}`) + ); + const uri = sample?.uri as string | undefined; + + if (!uri) { + const reason = `No listed entry has a uri ending in /${SKILL_MANIFEST_FILENAME}, so no SKILL.md can be read back.`; + for (const id of READBACK_IDS) { + checks.push(skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason })); + } + return checks; + } + + let body: Awaited>; + try { + body = await readResourceText(conn, uri); + } catch (e) { + const reason = `resources/read on ${uri} failed: ${e instanceof Error ? e.message : String(e)}`; + checks.push( + skillsCheck( + 'sep-2640-skillmd-required', + 'Every skill MUST contain a SKILL.md file at its root.', + 'FAILURE', + { errorMessage: reason } + ) + ); + for (const id of ['sep-2640-skillmd-frontmatter', 'sep-2640-entry-frontmatter-identical']) { + checks.push( + skillsCheck(id, 'SKILL.md is unreadable.', 'SKIPPED', { + errorMessage: reason + }) + ); + } + return checks; + } + + if (!body) { + const reason = `resources/read on ${uri} returned no text content, so the listed SKILL.md is not retrievable.`; + checks.push( + skillsCheck( + 'sep-2640-skillmd-required', + 'Every skill MUST contain a SKILL.md file at its root.', + 'FAILURE', + { errorMessage: reason } + ) + ); + for (const id of ['sep-2640-skillmd-frontmatter', 'sep-2640-entry-frontmatter-identical']) { + checks.push( + skillsCheck(id, 'No SKILL.md content to inspect.', 'SKIPPED', { + errorMessage: reason + }) + ); + } + return checks; + } + + checks.push( + skillsCheck( + 'sep-2640-skillmd-required', + 'Every skill MUST contain a SKILL.md file at its root.', + 'SUCCESS', + { details: { uri, bytes: body.text.length } } + ) + ); + + // === skillmd-frontmatter === + const fm = parseFrontmatter(body.text); + const fmErrs: string[] = []; + if (!fm) { + fmErrs.push('SKILL.md has no leading --- delimited YAML frontmatter block'); + } else { + if (typeof fm.name !== 'string' || fm.name.length === 0) { + fmErrs.push('frontmatter has no non-empty name'); + } + if (typeof fm.description !== 'string' || fm.description.length === 0) { + fmErrs.push('frontmatter has no non-empty description'); + } + } + checks.push( + skillsCheck( + 'sep-2640-skillmd-frontmatter', + 'SKILL.md MUST begin with YAML frontmatter containing at minimum the name and description fields.', + fmErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + fmErrs.length === 0 + ? { details: { uri } } + : { errorMessage: `${uri}: ${fmErrs.join('; ')}` } + ) + ); + + // === entry-frontmatter-identical === + const declared = sample?.frontmatter as Record | undefined; + if (!fm || !declared || typeof declared !== 'object' || Array.isArray(declared)) { + const reason = + 'Either the file has no parseable frontmatter or the entry carries no frontmatter object, so the two cannot be compared.'; + checks.push( + skillsCheck( + 'sep-2640-entry-frontmatter-identical', + 'The frontmatter object MUST be identical in content to the frontmatter of the SKILL.md it describes.', + 'SKIPPED', + { errorMessage: reason } + ) + ); + return checks; + } + + const diffs: string[] = []; + const keys = new Set([...Object.keys(fm), ...Object.keys(declared)]); + for (const k of keys) { + const a = JSON.stringify(fm[k] ?? null); + const b = JSON.stringify(declared[k] ?? null); + if (a !== b) diffs.push(`${k}: file=${a} entry=${b}`); + } + checks.push( + skillsCheck( + 'sep-2640-entry-frontmatter-identical', + 'The frontmatter object MUST be identical in content to the frontmatter of the SKILL.md it describes.', + diffs.length === 0 ? 'SUCCESS' : 'FAILURE', + diffs.length === 0 + ? { details: { uri, fields: [...keys] } } + : { + errorMessage: `${uri}: entry frontmatter differs from the file's: ${joinErrs(diffs)}` + } + ) + ); + + return checks; +} + +/** Exercise `skills/get` against a real entry and against an unknown URI. */ +async function getChecks( + conn: Parameters[0], + entries: SkillEntry[] +): Promise { + const checks: ConformanceCheck[] = []; + const sample = entries.find((e) => typeof e.uri === 'string'); + const sampleUri = sample?.uri as string | undefined; + + if (!sampleUri) { + const reason = + 'No listed entry carries a uri, so skills/get cannot be exercised against a known skill.'; + for (const id of GET_IDS) { + checks.push( + skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) + ); + } + return checks; + } + + const got = await skillsGet(conn, sampleUri); + if ('error' in got) { + const reason = `${SKILLS_GET_METHOD} failed for a skill the server itself listed (${sampleUri}) with code ${got.error.code}: ${got.error.message}.`; + checks.push( + skillsCheck( + 'sep-2640-capability-commits-to-methods', + 'Declaring the extension commits the server to skills/list and skills/get.', + 'FAILURE', + { errorMessage: reason } + ), + skillsCheck( + 'sep-2640-skills-get-implemented', + 'A server declaring the extension MUST also implement the skills/get method, which returns the entry for a single skill named by its URI.', + 'FAILURE', + { errorMessage: reason } + ) + ); + for (const id of [ + 'sep-2640-skills-get-entry-shape', + 'sep-2640-skills-get-no-cursor', + 'sep-2640-skills-get-unknown-uri-invalid-params' + ]) { + checks.push( + skillsCheck(id, 'skills/get is unavailable.', 'SKIPPED', { + errorMessage: reason + }) + ); + } + return checks; + } + + checks.push( + skillsCheck( + 'sep-2640-capability-commits-to-methods', + 'Declaring the extension itself commits the server to skills/list and skills/get.', + 'SUCCESS', + { details: { methods: [SKILLS_LIST_METHOD, SKILLS_GET_METHOD] } } + ), + skillsCheck( + 'sep-2640-skills-get-implemented', + 'A server declaring the extension MUST also implement the skills/get method, which returns the entry for a single skill named by its URI.', + 'SUCCESS', + { details: { probedUri: sampleUri } } + ) + ); + + // === skills-get-entry-shape === + const skill = got.result.skill as SkillEntry | undefined; + const shapeErrs: string[] = []; + if (!skill || typeof skill !== 'object' || Array.isArray(skill)) { + shapeErrs.push('result.skill is missing or not an object'); + } else { + if (skill.uri !== sampleUri) { + shapeErrs.push( + `result.skill.uri=${JSON.stringify(skill.uri)} does not echo the requested uri ${sampleUri}` + ); + } + if ( + !skill.frontmatter || + typeof skill.frontmatter !== 'object' || + Array.isArray(skill.frontmatter) + ) { + shapeErrs.push('result.skill.frontmatter is missing or not an object'); + } + if (!isDynamicResources(skill) && resourcesArray(skill) === undefined) { + shapeErrs.push( + `result.skill.resources=${JSON.stringify(skill.resources)} is neither an array nor "${RESOURCES_DYNAMIC}"` + ); + } + } + checks.push( + skillsCheck( + 'sep-2640-skills-get-entry-shape', + 'The skill object is a skill entry, identical in shape and meaning to an entry of skills/list — the same uri, frontmatter, and resources fields, under the same rules.', + shapeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', + shapeErrs.length === 0 + ? { details: { probedUri: sampleUri } } + : { errorMessage: joinErrs(shapeErrs) } + ) + ); + + // === skills-get-no-cursor === + const hasCursor = got.result.nextCursor !== undefined; + checks.push( + skillsCheck( + 'sep-2640-skills-get-no-cursor', + 'The result carries no pagination cursor: a single entry is not a list.', + hasCursor ? 'FAILURE' : 'SUCCESS', + hasCursor + ? { + errorMessage: `skills/get returned nextCursor=${JSON.stringify(got.result.nextCursor)}; a single entry is not a list.` + } + : { details: { probedUri: sampleUri } } + ) + ); + + // === skills-get-unknown-uri-invalid-params === + const unknown = await skillsGet(conn, UNKNOWN_SKILL_URI); + if ('error' in unknown) { + const ok = unknown.error.code === JSONRPC_INVALID_PARAMS; + checks.push( + skillsCheck( + 'sep-2640-skills-get-unknown-uri-invalid-params', + 'If the URI does not identify a skill the server serves, the server MUST return error -32602 (Invalid params).', + ok ? 'SUCCESS' : 'FAILURE', + ok + ? { details: { probedUri: UNKNOWN_SKILL_URI, code: unknown.error.code } } + : { + errorMessage: `skills/get on an unserved URI returned code ${unknown.error.code} (${unknown.error.message}); expected ${JSONRPC_INVALID_PARAMS}.` + } + ) + ); + } else { + checks.push( + skillsCheck( + 'sep-2640-skills-get-unknown-uri-invalid-params', + 'If the URI does not identify a skill the server serves, the server MUST return error -32602 (Invalid params).', + 'FAILURE', + { + errorMessage: `skills/get returned a successful result for ${UNKNOWN_SKILL_URI}, which no conformant server should serve; expected error ${JSONRPC_INVALID_PARAMS}.` + } + ) + ); + } + + return checks; +} diff --git a/src/scenarios/server/skills/helpers.ts b/src/scenarios/server/skills/helpers.ts index d32022f9..7bd24943 100644 --- a/src/scenarios/server/skills/helpers.ts +++ b/src/scenarios/server/skills/helpers.ts @@ -4,9 +4,14 @@ * * The scenarios treat the server-under-test as an arbitrary SEP-2640 server: * capability is read from `server/discover` (never inferred from an error), and - * every skill is discovered dynamically from `skill://index.json` and - * `resources/list` — no fixture-specific URI is hardcoded, so the checks pass - * against any conformant server, not just one implementation's fixture. + * every skill is discovered dynamically through `skills/list` — no + * fixture-specific URI is hardcoded, so the checks pass against any conformant + * server, not just one implementation's fixture. + * + * Extracted against the 2026-08-21 revision of the SEP (branch + * `sep/skills-extension`), which replaced the `skill://index.json` well-known + * resource with the `skills/list` and `skills/get` methods, deferred archive + * distribution, and reshaped the skill entry to `{uri, frontmatter, resources}`. */ import type { @@ -20,15 +25,25 @@ import { parse as parseYaml } from 'yaml'; export const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; export const SKILL_URI_SCHEME = 'skill://'; -export const SKILL_INDEX_URI = 'skill://index.json'; export const SKILL_MANIFEST_FILENAME = 'SKILL.md'; export const SKILLS_META_PREFIX = 'io.modelcontextprotocol.skills/'; -/** `sha256:{hex}` with exactly 64 lowercase hex characters (SEP-2640 index). */ +/** Reserved prefix for MCP-defined keys inside frontmatter `metadata`. */ +export const FRONTMATTER_RESERVED_PREFIX = 'io.modelcontextprotocol/'; + +export const SKILLS_LIST_METHOD = 'skills/list'; +export const SKILLS_GET_METHOD = 'skills/get'; +export const DIRECTORY_READ_METHOD = 'resources/directory/read'; + +/** `sha256:{hex}` with exactly 64 lowercase hex characters (SEP-2640). */ export const SKILL_DIGEST_PATTERN = /^sha256:[0-9a-f]{64}$/; -/** The SEP enumerated `skills[].type` values. */ -export const SKILL_TYPES = ['skill-md', 'archive'] as const; +/** The `"dynamic"` sentinel a server sets in place of a `resources` array. */ +export const RESOURCES_DYNAMIC = 'dynamic'; + +/** Per-skill limits fixed by the SEP (§Limits). */ +export const MAX_RESOURCES_PER_SKILL = 512; +export const MAX_TOTAL_SIZE_PER_SKILL = 16 * 1024 * 1024; // 16 MiB export const JSONRPC_METHOD_NOT_FOUND = -32601; export const JSONRPC_INVALID_PARAMS = -32602; @@ -47,20 +62,37 @@ export interface SkillResource { _meta?: Record; } -/** One `skills[]` entry of the `skill://index.json` document. */ -export interface SkillIndexEntry { - name?: string; - type?: string; - description?: string; - url?: string; - digest?: string; +/** One `{uri, digest, size}` triple of a skill entry's `resources` array. */ +export interface SkillResourceEntry { + uri?: unknown; + digest?: unknown; + size?: unknown; + [key: string]: unknown; +} + +/** + * One skill entry, as returned by `skills/list` (in `skills[]`) and by + * `skills/get` (as `skill`). The two are identical in shape and meaning. + */ +export interface SkillEntry { + uri?: unknown; + frontmatter?: unknown; + /** An array of `{uri, digest, size}`, or the string `"dynamic"`. */ + resources?: unknown; + [key: string]: unknown; +} + +export interface SkillsListResult { + skills?: unknown; + nextCursor?: string; + ttlMs?: unknown; + cacheScope?: unknown; [key: string]: unknown; } -/** The parsed `skill://index.json` document. */ -export interface SkillIndex { - $schema?: string; - skills?: SkillIndexEntry[]; +export interface SkillsGetResult { + skill?: unknown; + nextCursor?: unknown; [key: string]: unknown; } @@ -111,24 +143,44 @@ export async function skillsCapability( : undefined; } +/** + * Whether the declared extension object nests its settings inline, as both + * SEP-2133 and SEP-2640 require, rather than wrapping them in an envelope. + * + * SEP-2133 (status Final) defines `extensions` as "a map of extension + * identifiers to per-extension settings objects", and SEP-2640's capability + * block matches: `{"io.modelcontextprotocol/skills": {"directoryRead": true}}`. + * Neither SEP defines an envelope, and neither has a slot for `id`, + * `specVersion` or `stability`. + * + * An earlier revision of this helper accepted a `config` envelope alongside the + * inline form, on the belief that the two SEPs disagreed. Re-reading SEP-2133 + * at Final status, they do not. The envelope is a non-conformant shape emitted + * by at least one SDK, so it is reported rather than silently accepted. + */ +export function settingsAreInline( + skills: Record +): { inline: boolean; envelopeKeys: string[] } { + const envelopeKeys = ['config', 'specVersion', 'stability', 'id'].filter( + (k) => k in skills + ); + return { inline: envelopeKeys.length === 0, envelopeKeys }; +} + /** * Whether the skills extension declares `directoryRead: true`. * - * SEP-2640's capability-declaration example places the flag directly on the - * extension object (`extensions[id].directoryRead`). SEP-2133 extension - * negotiation — which SEP-2640 normatively defers to ("Per SEP-2133 extension - * negotiation") — wraps settings in a `{ specVersion, stability, config }` - * envelope, putting the flag at `extensions[id].config.directoryRead`. The two - * SEPs are inconsistent on nesting, so a brand-neutral conformance check accepts - * either location rather than privileging one reading of an ambiguous spec. - * (The inconsistency is worth a WG clarification; see the scenario docs.) + * Reads only the inline location the SEPs specify. A server that buries the + * flag inside an envelope fails `sep-2640-capability-declaration-inline` and is + * treated here as not having declared the optional method, which is the + * conservative reading: a client that follows the spec would not see the flag + * either, and "clients MUST NOT call resources/directory/read against a server + * that has not declared directoryRead: true". */ export function directoryReadDeclared( skills: Record ): boolean { - if (skills.directoryRead === true) return true; - const config = skills.config as Record | undefined; - return config?.directoryRead === true; + return skills.directoryRead === true; } /** Everything from `resources/list`, paginating until `nextCursor` clears. */ @@ -148,20 +200,84 @@ export async function listAllResources( return out; } +/** One page of `skills/list`, kept separate so pagination can be inspected. */ +export interface SkillsListPage { + result: SkillsListResult; + entries: SkillEntry[]; +} + /** - * Read `skill://index.json`. Returns the raw JSON text (for parse-error - * reporting) or a `JsonRpcError` when the server declines the well-known index - * — a permitted MAY (SEP-2640 §Enumeration): the catalog may be unenumerable. + * Call `skills/list` once, optionally with a cursor. Returns a `JsonRpcError` + * rather than throwing so a scenario can distinguish "method not implemented" + * (a FAILURE, since the method is mandatory for a declaring server) from a + * transport fault. */ -export async function readSkillIndexText( - conn: Connection -): Promise<{ text?: string; mimeType?: string } | { error: JsonRpcError }> { +export async function skillsListPage( + conn: Connection, + cursor?: string +): Promise { + try { + const result = await conn.request( + SKILLS_LIST_METHOD, + cursor ? { cursor } : {} + ); + const entries = Array.isArray(result.skills) + ? (result.skills as SkillEntry[]) + : []; + return { result, entries }; + } catch (e) { + if (e instanceof JsonRpcError) return { error: e }; + throw e; + } +} + +/** + * Every entry from `skills/list`, following `nextCursor`. `pages` is retained + * so the atomic-entry and pagination checks can reason about page boundaries. + * Bounded to avoid looping forever against a server that returns a constant + * cursor. + */ +export async function skillsListAll( + conn: Connection, + maxPages = 50 +): Promise< + | { entries: SkillEntry[]; pages: SkillsListPage[]; truncated: boolean } + | { error: JsonRpcError } +> { + const pages: SkillsListPage[] = []; + const entries: SkillEntry[] = []; + let cursor: string | undefined; + const seenCursors = new Set(); + + for (let i = 0; i < maxPages; i++) { + const page = await skillsListPage(conn, cursor); + if ('error' in page) return page; + pages.push(page); + entries.push(...page.entries); + const next = page.result.nextCursor; + if (typeof next !== 'string' || next.length === 0) { + return { entries, pages, truncated: false }; + } + if (seenCursors.has(next)) { + // A repeating cursor is a server bug; stop rather than spin. + return { entries, pages, truncated: true }; + } + seenCursors.add(next); + cursor = next; + } + return { entries, pages, truncated: true }; +} + +/** Call `skills/get` for one skill URI. */ +export async function skillsGet( + conn: Connection, + uri: string +): Promise<{ result: SkillsGetResult } | { error: JsonRpcError }> { try { - const res = await conn.request<{ - contents?: Array<{ text?: string; mimeType?: string }>; - }>('resources/read', { uri: SKILL_INDEX_URI }); - const entry = (res.contents ?? []).find((c) => typeof c.text === 'string'); - return { text: entry?.text, mimeType: entry?.mimeType }; + const result = await conn.request(SKILLS_GET_METHOD, { + uri + }); + return { result }; } catch (e) { if (e instanceof JsonRpcError) return { error: e }; throw e; @@ -188,15 +304,21 @@ export async function readResourceText( /** * The skill name recoverable from a `SKILL.md` resource URI: the final segment * of ``, i.e. the last path segment before the trailing - * `SKILL.md`. Returns `undefined` when the URI is not a `skill://…/SKILL.md`. + * `SKILL.md`. Returns `undefined` when the URI does not end in `/SKILL.md`. + * + * Scheme-agnostic by design: the SEP is explicit that "no scheme is + * privileged" and that the structural constraints "apply regardless of + * scheme", so a server serving skills under `github://` is judged by the same + * path rule as one using `skill://`. * * skill://org/team/deploy/SKILL.md -> "deploy" - * skill://lint/SKILL.md -> "lint" + * github://acme/repo/lint/SKILL.md -> "lint" */ export function skillNameFromManifestUri(uri: string): string | undefined { - if (!uri.startsWith(SKILL_URI_SCHEME)) return undefined; + const schemeEnd = uri.indexOf('://'); + if (schemeEnd < 0) return undefined; const parts = uri - .slice(SKILL_URI_SCHEME.length) + .slice(schemeEnd + 3) .split('/') .filter((p) => p.length > 0); if (parts.length < 2) return undefined; @@ -204,6 +326,12 @@ export function skillNameFromManifestUri(uri: string): string | undefined { return parts[parts.length - 2]; } +/** The skill's root directory URI: its `SKILL.md` URI with the file removed. */ +export function skillRootFromManifestUri(uri: string): string | undefined { + if (!uri.endsWith(`/${SKILL_MANIFEST_FILENAME}`)) return undefined; + return uri.slice(0, -`/${SKILL_MANIFEST_FILENAME}`.length); +} + /** * Extract and parse the YAML frontmatter block at the head of a `SKILL.md`. * Returns `undefined` when there is no leading `---` delimited block or it does @@ -225,3 +353,25 @@ export function parseFrontmatter( return undefined; } } + +/** True when the entry's `resources` is the `"dynamic"` sentinel. */ +export function isDynamicResources(entry: SkillEntry): boolean { + return entry.resources === RESOURCES_DYNAMIC; +} + +/** + * The entry's `resources` array, or `undefined` when it is `"dynamic"`, absent, + * or any other value. Callers distinguish those cases via `isDynamicResources`. + */ +export function resourcesArray( + entry: SkillEntry +): SkillResourceEntry[] | undefined { + return Array.isArray(entry.resources) + ? (entry.resources as SkillResourceEntry[]) + : undefined; +} + +/** A short, stable label for an entry, for error messages. */ +export function entryLabel(entry: SkillEntry, i: number): string { + return typeof entry.uri === 'string' ? entry.uri : `skills[${i}]`; +} diff --git a/src/scenarios/server/skills/index.ts b/src/scenarios/server/skills/index.ts deleted file mode 100644 index bb9d1f33..00000000 --- a/src/scenarios/server/skills/index.ts +++ /dev/null @@ -1,270 +0,0 @@ -/** - * SEP-2640 Skills extension — `skill://index.json` enumeration surface. - * - * One scenario, many checks (per AGENTS.md "fewer scenarios, more checks"). - * Each check's verbatim spec quote lives next to its check ID in - * src/seps/sep-2640.yaml, keeping the YAML and this scenario in lock-step. - * - * All discovery is dynamic and brand-neutral: the scenario reads the well-known - * `skill://index.json` and validates whatever entries it finds, hardcoding no - * fixture-specific skill name or URI. When the server does not declare the - * skills extension the checks are SKIPPED (an optional, undeclared capability); - * when the server declines the index (a permitted MAY) or serves an empty index - * the index-shape checks are SKIPPED (legitimately not applicable), never - * failed against a conformant server. - */ - -import { ClientScenario, ConformanceCheck } from '../../../types'; -import type { RunContext } from '../../../connection'; -import { - SKILLS_EXTENSION_ID, - SKILL_INDEX_URI, - SKILL_URI_SCHEME, - SKILL_TYPES, - SKILL_DIGEST_PATTERN, - type SkillIndex, - type SkillIndexEntry, - skillsCapability, - skillsCheck, - readSkillIndexText -} from './helpers'; - -const ENTRY_CHECK_IDS = [ - 'sep-2640-index-entry-type-enum', - 'sep-2640-index-name-required', - 'sep-2640-index-digest-required', - 'sep-2640-skill-uri-scheme' -] as const; - -const ALL_CHECK_IDS = ['sep-2640-server-expose-index', ...ENTRY_CHECK_IDS]; - -export class SkillsIndexScenario implements ClientScenario { - name = 'sep-2640-skills-index'; - readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; - description = `SEP-2640 Skills extension: the \`skill://index.json\` enumeration index. - -**Resource**: \`skill://index.json\` (read via \`resources/read\`, \`mimeType\` \`application/json\`) - -**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml): - -- \`sep-2640-server-expose-index\` — server exposes a readable \`skill://index.json\` (SHOULD; a server MAY decline for an unenumerable catalog) -- \`sep-2640-index-entry-type-enum\` — every \`skills[].type\` is \`"skill-md"\` or \`"archive"\` (MUST) -- \`sep-2640-index-name-required\` — every entry carries a non-empty \`name\` (required field) -- \`sep-2640-index-digest-required\` — a present \`skills[].digest\` is \`sha256:{64 hex}\` (MUST) -- \`sep-2640-skill-uri-scheme\` — index entry URLs use the \`skill://\` scheme (SHOULD; another scheme is permitted only when listed in the index) - -**Discovery is dynamic**: the scenario reads whatever skills the index enumerates. Undeclared extension, a declined index, or an empty index all SKIP cleanly.`; - - async run(ctx: RunContext): Promise { - const conn = await ctx.connect(); - try { - const skills = await skillsCapability(conn); - if (!skills) { - const reason = - 'Server did not declare the io.modelcontextprotocol/skills extension; index checks not applicable.'; - return ALL_CHECK_IDS.map((id) => - skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) - ); - } - - const checks: ConformanceCheck[] = []; - - // === server-expose-index (SHOULD, with an explicit MAY-decline) === - const read = await readSkillIndexText(conn); - if ('error' in read) { - const reason = `Server declined skill://index.json (code ${read.error.code}); permitted MAY — the catalog may be large or unenumerable. Hosts MUST NOT treat this as proof of no skills.`; - checks.push( - skillsCheck( - 'sep-2640-server-expose-index', - 'Server SHOULD expose a readable skill://index.json enumerating the skills it serves.', - 'SKIPPED', - { errorMessage: reason } - ) - ); - for (const id of ENTRY_CHECK_IDS) { - checks.push( - skillsCheck(id, 'No skill://index.json to inspect.', 'SKIPPED', { - errorMessage: reason - }) - ); - } - return checks; - } - - if (read.text === undefined) { - const reason = - 'resources/read on skill://index.json returned no text content; the index resource is exposed but unreadable.'; - checks.push( - skillsCheck( - 'sep-2640-server-expose-index', - 'Server SHOULD expose a readable skill://index.json enumerating the skills it serves.', - 'FAILURE', - { errorMessage: reason } - ) - ); - for (const id of ENTRY_CHECK_IDS) { - checks.push( - skillsCheck( - id, - 'No readable index content to inspect.', - 'SKIPPED', - { - errorMessage: reason - } - ) - ); - } - return checks; - } - - let index: SkillIndex; - try { - index = JSON.parse(read.text) as SkillIndex; - } catch (e) { - const reason = `skill://index.json content is not valid JSON: ${ - e instanceof Error ? e.message : String(e) - }`; - checks.push( - skillsCheck( - 'sep-2640-server-expose-index', - 'Server SHOULD expose a readable skill://index.json whose content is a JSON index.', - 'FAILURE', - { errorMessage: reason } - ) - ); - for (const id of ENTRY_CHECK_IDS) { - checks.push( - skillsCheck(id, 'Index did not parse as JSON.', 'SKIPPED', { - errorMessage: reason - }) - ); - } - return checks; - } - - checks.push( - skillsCheck( - 'sep-2640-server-expose-index', - 'Server SHOULD expose a readable skill://index.json whose content is a JSON index of the skills it serves.', - 'SUCCESS', - { - details: { - uri: SKILL_INDEX_URI, - mimeType: read.mimeType, - skillCount: Array.isArray(index.skills) ? index.skills.length : 0 - } - } - ) - ); - - const entries: SkillIndexEntry[] = Array.isArray(index.skills) - ? index.skills - : []; - - // An exposed-but-empty index is valid: a partial/empty index is - // permitted, and hosts MUST NOT read "no skills" from it. Nothing to - // validate at the entry level, so SKIP those checks cleanly. - if (entries.length === 0) { - const reason = - 'skill://index.json is exposed but lists no skills; entry-level checks not applicable (an empty index is permitted).'; - for (const id of ENTRY_CHECK_IDS) { - checks.push( - skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) - ); - } - return checks; - } - - // === index-entry-type-enum (MUST) === - const typeErrs = entries - .map((e, i) => - SKILL_TYPES.includes(e.type as (typeof SKILL_TYPES)[number]) - ? null - : `skills[${i}].type=${JSON.stringify(e.type)} is not one of ${SKILL_TYPES.join('|')}` - ) - .filter((x): x is string => x !== null); - checks.push( - skillsCheck( - 'sep-2640-index-entry-type-enum', - 'Every skills[].type MUST be "skill-md" or "archive".', - typeErrs.length === 0 ? 'SUCCESS' : 'FAILURE', - typeErrs.length === 0 - ? { details: { entryCount: entries.length } } - : { errorMessage: typeErrs.join('; ') } - ) - ); - - // === index-name-required (required field) === - const nameErrs = entries - .map((e, i) => - typeof e.name === 'string' && e.name.length > 0 - ? null - : `skills[${i}].name is missing or empty` - ) - .filter((x): x is string => x !== null); - checks.push( - skillsCheck( - 'sep-2640-index-name-required', - 'Every index entry carries a non-empty name (matching the SKILL.md frontmatter name and the final skill-path segment).', - nameErrs.length === 0 ? 'SUCCESS' : 'FAILURE', - nameErrs.length === 0 - ? { details: { entryCount: entries.length } } - : { errorMessage: nameErrs.join('; ') } - ) - ); - - // === index-digest-required (MUST — validate the format when present) === - const withDigest = entries.filter((e) => e.digest !== undefined); - const digestErrs = withDigest - .map((e, i) => - typeof e.digest === 'string' && SKILL_DIGEST_PATTERN.test(e.digest) - ? null - : `skills[${i}].digest=${JSON.stringify(e.digest)} is not sha256:{64 lowercase hex}` - ) - .filter((x): x is string => x !== null); - checks.push( - skillsCheck( - 'sep-2640-index-digest-required', - 'Every present skills[].digest MUST be formatted as sha256:{hex} with 64 lowercase hex characters.', - digestErrs.length === 0 ? 'SUCCESS' : 'FAILURE', - digestErrs.length === 0 - ? { - details: { - entriesWithDigest: withDigest.length, - entriesWithoutDigest: entries.length - withDigest.length - } - } - : { errorMessage: digestErrs.join('; ') } - ) - ); - - // === skill-uri-scheme (SHOULD) === - // Servers SHOULD use skill://; another scheme is permitted only when the - // skill is listed in the index (SEP-2640 §URI convention), so a non- - // skill:// URL is a SHOULD deviation, not a hard failure. - const nonSkillScheme = entries - .map((e, i) => - typeof e.url === 'string' && !e.url.startsWith(SKILL_URI_SCHEME) - ? `skills[${i}].url=${JSON.stringify(e.url)}` - : null - ) - .filter((x): x is string => x !== null); - checks.push( - skillsCheck( - 'sep-2640-skill-uri-scheme', - 'Skill resource URLs in the index SHOULD use the skill:// URI scheme.', - nonSkillScheme.length === 0 ? 'SUCCESS' : 'WARNING', - nonSkillScheme.length === 0 - ? { details: { entryCount: entries.length } } - : { - errorMessage: `Entries use a non-skill:// scheme (permitted only when indexed): ${nonSkillScheme.join(', ')}` - } - ) - ); - - return checks; - } finally { - await conn.close(); - } - } -} diff --git a/src/scenarios/server/skills/manifest.ts b/src/scenarios/server/skills/manifest.ts index 7a4f1fd6..42b19229 100644 --- a/src/scenarios/server/skills/manifest.ts +++ b/src/scenarios/server/skills/manifest.ts @@ -21,12 +21,11 @@ import { SKILLS_EXTENSION_ID, SKILLS_META_PREFIX, SEP_2640_REF, - type SkillIndex, type SkillResource, skillsCapability, skillsCheck, listAllResources, - readSkillIndexText, + skillsListAll, readResourceText, skillNameFromManifestUri, parseFrontmatter @@ -94,20 +93,12 @@ export class SkillsManifestScenario implements ClientScenario { ); let manifestUri = manifestResource?.uri; if (!manifestUri) { - const idx = await readSkillIndexText(conn); - if (!('error' in idx) && typeof idx.text === 'string') { - try { - const index = JSON.parse(idx.text) as SkillIndex; - const entry = (index.skills ?? []).find( - (e) => - e.type === 'skill-md' && - typeof e.url === 'string' && - isManifestUri(e.url) - ); - manifestUri = entry?.url; - } catch { - // A malformed index is the index scenario's concern; ignore here. - } + const listed = await skillsListAll(conn); + if (!('error' in listed)) { + const entry = listed.entries.find( + (e) => typeof e.uri === 'string' && isManifestUri(e.uri) + ); + manifestUri = entry?.uri as string | undefined; } } diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index d93e1487..1b0679e5 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -1,46 +1,71 @@ -# spec_source: modelcontextprotocol/modelcontextprotocol@556154c088371149c120172e95bb634655f00cbe seps/2640-skills-extension.md -# extracted: 2026-06-05 -# forward_reference: rows sep-2640-capability-directory-read-flag through -# sep-2640-directory-read-pagination track SEP commit -# 2e04c48da90224000e750ffd54a3611f2824fbc0 (2026-06-09) — the -# resources/directory/read addition. The file-level provenance above -# stays at 556154c because the PR 97 schema rewrite (360123d0, -# 2026-06-08) made 3 existing rows stale (sep-2640-index-entry-type-enum, -# sep-2640-index-name-required, sep-2640-index-digest-required) and -# drifted ~11 others' verbatim wording. Full re-extraction at SEP HEAD -# is mcpkit#780's lifecycle; this file deliberately holds at 556154c -# until that lands. +# spec_source: modelcontextprotocol/modelcontextprotocol@sep/skills-extension seps/2640-skills-extension.md +# extracted: 2026-08-28 +# +# provenance: re-extracted against the 2026-08-21 revision, which the core +# maintainers produced after their second review round and which went back up +# for CM vote on 2026-08-25. The prior extraction held at 556154c (2026-06-05) +# by an explicit decision to wait for mcpkit#780; that hold expired badly. +# The 08-21 revision: +# - removed the `skill://index.json` well-known resource entirely (it now +# appears nowhere in the SEP) and replaced it with the `skills/list` and +# `skills/get` methods, both mandatory for a declaring server; +# - deferred archive distribution to an appendix ("Appendix: Deferred +# Features"), retiring the `type` enum and every archive requirement; +# - reversed the nesting rule: a `SKILL.md` MAY now appear in a descendant +# directory, where the June text forbade it; +# - reshaped the entry to `{uri, frontmatter, resources}`, where `resources` +# is a complete `{uri, digest, size}` array or the string `"dynamic"`; +# - fixed per-skill limits of 512 resource entries and 16 MiB; +# - added host obligations around lazy retrieval, content-bound approval and +# frontmatter re-verification. +# Eight rows from the prior extraction were removed as no longer normative +# (server-expose-index, index-entry-type-enum, no-nested-skills, and the five +# archive rows) and four were reworded. Because the SEP is mid-vote, expect one +# more pass at the CM-stamped head. +# # backing_scenarios: three server ClientScenarios under # src/scenarios/server/skills/ emit the check IDs below (a row is "tested" # once a scenario emits its check ID; see src/traceability/): -# directory.ts (sep-2640-skills-directory) — sep-2640-capability-directory-read-flag -# and the five sep-2640-directory-read-* rows. -# index.ts (sep-2640-skills-index) — sep-2640-server-expose-index, -# sep-2640-index-entry-type-enum, sep-2640-index-name-required, -# sep-2640-index-digest-required, sep-2640-skill-uri-scheme. +# enumeration.ts (sep-2640-skills-enumeration) — the capability rows, the +# sep-2640-skills-list-* rows, the entry-schema and resources rows, the two +# limit rows, and the sep-2640-skills-get-* rows. Replaces the retired +# index.ts, which tested `skill://index.json`. # manifest.ts (sep-2640-skills-manifest) — sep-2640-skillmd-mimetype, # sep-2640-skillmd-metadata-name, sep-2640-skillmd-metadata-description, # sep-2640-final-segment-equals-name, sep-2640-meta-prefix. -# The remaining rows are host-internal or off-wire (host load-by-uri, digest -# verification, byte-budget / archive-unpack safety, no-empty-index -# assumption) and stay traceability-only for this server-scenario set. +# directory.ts (sep-2640-skills-directory) — +# sep-2640-capability-directory-read-flag and the +# sep-2640-directory-read-* rows. +# The remaining rows are host-internal or off-wire (host retrieval policy, +# digest and frontmatter verification, approval binding, cache isolation) and +# stay traceability-only for this server-scenario set. A host-side scenario +# set would be needed to exercise them. sep: 2640 spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification requirements: + # === Skill Format === - check: sep-2640-skillmd-required text: 'Every skill MUST contain a `SKILL.md` file at its root.' - check: sep-2640-skillmd-frontmatter text: '`SKILL.md` MUST begin with YAML frontmatter containing at minimum the `name` and `description` fields as defined by the Agent Skills specification.' + + # === Resource Mapping === - check: sep-2640-skill-uri-scheme text: 'Each file within a skill directory is exposed as an MCP resource. Servers SHOULD use the `skill://` URI scheme, under which the resource URI has the form: `skill:///`' - check: sep-2640-final-segment-equals-name text: "The final segment of `` MUST equal the skill's `name` as declared in its `SKILL.md` frontmatter." - - check: sep-2640-no-nested-skills - text: 'A `SKILL.md` MUST NOT appear in any descendant directory of a skill. The skill directory is the boundary; skills do not nest inside other skills.' - check: sep-2640-name-naming-rules text: "The final `` segment, being the skill `name`, MUST satisfy the Agent Skills specification's naming rules." - - check: sep-2640-prefix-rfc3986 - text: 'Prefix segments SHOULD be valid URI path segments per RFC 3986; no further constraints are imposed on them.' + - check: sep-2640-authority-reg-name + text: 'The first `` segment occupies the authority component and SHOULD be a valid `reg-name` per RFC 3986; any other prefix segments SHOULD be valid URI path segments; no further constraints are imposed on them.' + # Reverses the retired sep-2640-no-nested-skills row: the June text said a + # SKILL.md MUST NOT appear in any descendant directory. + - check: sep-2640-nested-skills-permitted + text: 'A `SKILL.md` MAY appear in a descendant directory of a skill — skills can nest.' + - check: sep-2640-nested-publication-flat + text: "A nested skill is published like any other: through its own `skills/list` entry, or by explicit reference. The listing remains flat — an entry for a nested skill is an ordinary entry whose `uri` happens to share a path prefix with the enclosing skill's, and nothing in the listing marks nesting." + + # === Resource Metadata === - check: sep-2640-skillmd-mimetype text: 'For each `skill:///SKILL.md` resource: `mimeType` SHOULD be `text/markdown`.' - check: sep-2640-skillmd-metadata-name @@ -49,36 +74,119 @@ requirements: text: 'For each `skill:///SKILL.md` resource: `description` SHOULD be set from the `description` field of the `SKILL.md` YAML frontmatter.' - check: sep-2640-meta-prefix text: 'When `_meta` keys are used for skill resources, implementations SHOULD use the `io.modelcontextprotocol.skills/` reverse-domain prefix.' + + # === Capability Declaration === + # The inline-settings row is the check that would have caught panyam/mcpkit#1334. + # SEP-2133 (status Final) defines `extensions` as "a map of extension + # identifiers to per-extension settings objects" and defines no envelope; the + # SEP-2640 capability block matches it. + - check: sep-2640-capability-declaration-inline + text: 'Per SEP-2133 extension negotiation, servers declare support for this extension in their `initialize` response, mapping the extension identifier directly to its per-extension settings object: `{"capabilities": {"extensions": {"io.modelcontextprotocol/skills": {"directoryRead": true}}}}`.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#capability-declaration + - check: sep-2640-capability-commits-to-methods + text: 'Declaring the extension itself commits the server to `skills/list` and `skills/get`.' + - check: sep-2640-capability-empty-object + text: 'An empty object indicates support for the extension with no optional features.' + - check: sep-2640-capability-directory-read-flag + text: 'Clients MUST NOT call `resources/directory/read` against a server that has not declared `directoryRead: true`.' + + # === Discovery: enumeration via skills/list === + - check: sep-2640-skills-list-implemented + text: 'A server declaring the `io.modelcontextprotocol/skills` extension MUST implement the `skills/list` method, which returns the skills it serves. The result MAY be empty.' + - check: sep-2640-skills-list-pagination + text: 'Pagination mirrors the base protocol''s list methods: the request accepts an optional `cursor`, and when the result includes `nextCursor` the client passes it back to retrieve the next page.' + - check: sep-2640-skills-list-entry-atomic + text: "An entry is atomic — a skill's `resources` set is never split across pages." + - check: sep-2640-skills-list-cache-attributes + text: 'In protocol versions 2026-07-28 and later, the result also carries the base protocol''s list-caching attributes — `ttlMs` and `cacheScope`, as defined for `tools/list` and `resources/list` (SEP-2549) — with the same semantics: a freshness hint for the listing and a cache-scope marker, not an integrity property.' + - check: sep-2640-skills-list-may-be-partial + text: 'A server whose skill catalog is large, generated on demand, or otherwise unenumerable MAY return an empty or partial listing.' + - check: sep-2640-host-no-empty-listing-assumption + text: 'Hosts MUST NOT treat an empty or partial listing as proof that a server has no skills.' + - check: sep-2640-enumeration-scheme-uniform + text: 'The method serves entries for a server''s skills whatever URI scheme they use — enumeration is uniform across schemes.' + + # === The skill entry (shared by skills/list and skills/get) === + - check: sep-2640-entry-uri-required + text: '`skills[].uri` (Yes): Resource URI of the skill''s `SKILL.md`.' + - check: sep-2640-entry-frontmatter-required + text: "`frontmatter` is the skill's `SKILL.md` YAML frontmatter rendered verbatim as a JSON object — every field the author wrote, not a curated subset. Because the Agent Skills specification requires `name` and `description`, those fields are always present." + - check: sep-2640-entry-frontmatter-identical + text: 'The `frontmatter` object MUST be identical in content to the frontmatter of the `SKILL.md` it describes.' + - check: sep-2640-entry-uri-matches-frontmatter-name + text: "The final `` segment of the entry's `uri` MUST equal `frontmatter.name`, per Resource Mapping." + - check: sep-2640-metadata-reserved-prefix + text: 'Within the frontmatter `metadata` object, keys prefixed with `io.modelcontextprotocol/` are reserved for metadata defined by MCP extensions. This extension currently defines no such keys. Implementations SHOULD ignore keys under this prefix that they do not recognize.' + - check: sep-2640-names-not-unique + text: "A skill's `name` is a label, not an identifier. Within a server's listing, names SHOULD be unique, but they are not guaranteed to be. Hosts MUST NOT assume name uniqueness. When two entries in one listing collide on `name`, hosts MUST disambiguate them — for example by their distinguishing path segments — rather than silently discarding or preferring one." + + # === resources === + - check: sep-2640-entry-resources-required + text: '`resources` is REQUIRED on every skill entry and takes one of two forms: an array enumerating the skill''s files — `SKILL.md` and every supporting file — as `{uri, digest, size}` triples, or the string `"dynamic"`.' + - check: sep-2640-resources-complete + text: 'When present, `resources` MUST be complete: it lists every file of the skill, each exactly once, including an entry matching the skill''s top-level `uri` — that entry carries the digest and size of `SKILL.md` itself.' + - check: sep-2640-resources-uri-within-skill + text: 'Each `uri` MUST be the skill''s `SKILL.md` or a file within the skill''s directory.' + - check: sep-2640-resources-digest-format + text: 'Digests are SHA-256 hashes of an artifact''s raw bytes, formatted as `sha256:{hex}` where `{hex}` is 64 lowercase hexadecimal characters. Each entry in a skill''s `resources` carries the digest of the file at its `uri`.' + - check: sep-2640-resources-size-required + text: 'Each entry MUST carry `size`: the length in bytes of the file''s raw content — the same bytes the `digest` covers.' + - check: sep-2640-resources-dynamic-marker + text: 'When a skill''s content is generated dynamically, such that stable digests cannot be published, the server MUST set `"resources": "dynamic"` instead of an array.' + - check: sep-2640-resources-invalid-entry + text: 'An entry with no `resources` at all, or with any value other than an array or `"dynamic"`, is invalid, and hosts MUST NOT load it.' + - check: sep-2640-resources-nested-completeness + text: "Completeness extends to nested skills: from the enclosing skill's perspective their files are supporting files, so the enclosing skill's `resources` lists them too, and the same file may appear in both the enclosing and the nested skill's entries." + + # === Limits === + - check: sep-2640-limit-resources-per-skill + text: 'Resources per skill: 512 entries, counted over the entries of the skill''s `resources`, `SKILL.md` included.' + - check: sep-2640-limit-total-size + text: 'Total file size per skill: 16 MiB (16,777,216 bytes), counted over the sum of `size` over the skill''s `resources`.' + - check: sep-2640-limit-host-support + text: 'Hosts MUST support skills up to and including these limits, and MAY support larger ones. Servers SHOULD NOT serve a skill that exceeds either limit; a skill that does is not guaranteed to be loadable by any conforming host.' + + # === Retrieval via skills/get === + - check: sep-2640-skills-get-implemented + text: 'A server declaring the `io.modelcontextprotocol/skills` extension MUST also implement the `skills/get` method, which returns the entry for a single skill named by its URI.' + - check: sep-2640-skills-get-entry-shape + text: 'The `skill` object is a skill entry, identical in shape and meaning to an entry of `skills/list` — the same `uri`, `frontmatter`, and `resources` fields, under the same rules.' + - check: sep-2640-skills-get-unknown-uri-invalid-params + text: 'If the URI does not identify a skill the server serves, the server MUST return error `-32602` (Invalid params) — the same code `resources/read` uses for unknown resources.' + - check: sep-2640-skills-get-answers-unlisted + text: 'A server MUST answer for every skill it serves, whether or not that skill appears in its `skills/list` result. A skill absent from a partial listing is still retrievable by URI.' + - check: sep-2640-skills-get-no-cursor + text: 'The result carries no pagination cursor: a single entry is not a list.' + + # === Reading / integrity (host obligations) === - check: sep-2640-host-load-by-uri text: 'hosts MUST support loading a skill given only its URI' - - check: sep-2640-server-expose-index - text: 'A server SHOULD expose a resource at the well-known URI `skill://index.json` whose content is a JSON index of the skills it serves.' - - check: sep-2640-index-entry-type-enum - text: '`skills[].type` MUST be `"skill-md"` or `"archive"`.' - - check: sep-2640-index-name-required - text: '`skills[].name` matches the `SKILL.md` frontmatter `name` and the final segment of the skill path.' - - check: sep-2640-index-digest-required - text: '`skills[].digest` is the SHA-256 content digest of the artifact, formatted as `sha256:{hex}` (64 lowercase hex characters).' - - check: sep-2640-client-ignore-unrecognized - text: 'Clients SHOULD ignore unrecognized fields and SHOULD skip entries with an unrecognized `type`.' - - check: sep-2640-archive-format - text: 'the archive MUST be `.tar.gz` (gzip-compressed tar, `mimeType` `application/gzip`) or `.zip` (`mimeType` `application/zip`)' - - check: sep-2640-host-support-archive-formats - text: 'hosts MUST support both `.tar.gz` and `.zip` archive formats' - - check: sep-2640-archive-skillmd-at-root - text: 'Archive contents represent the skill directory directly — `SKILL.md` MUST be at the archive root, not nested inside a wrapper directory' - - check: sep-2640-archive-no-traversal - text: 'the archive MUST NOT contain path-traversal sequences (`..`) or absolute paths' - - check: sep-2640-host-archive-safety - text: 'Hosts unpacking an archive MUST apply the archive safety requirements of the Agent Skills specification: reject archives containing path-traversal sequences or absolute paths, reject symlinks or hard links that resolve outside the skill directory, and enforce a limit on total unpacked size / Hosts MUST validate archives per the Agent Skills archive safety requirements: reject path traversal and absolute paths, reject links resolving outside the skill directory, and bound total unpacked size to prevent decompression bombs.' - check: sep-2640-host-verify-digest - text: 'Hosts MUST verify retrieved content against the `digest` in the index / hosts MUST NOT use unverified content.' - - check: sep-2640-host-no-empty-index-assumption - text: 'Hosts MUST NOT treat an absent or empty index as proof that a server has no skills.' + text: 'When a host retrieves a file listed in a skill''s `resources`, it MUST verify the content against that entry''s digest. Whatever the cause, hosts MUST NOT use the unverified content.' + - check: sep-2640-host-size-mismatch-failure + text: 'A read whose byte length differs from the entry''s `size` is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest.' + - check: sep-2640-host-unlisted-read-failure + text: 'While acting on a skill, a host MUST resolve reads of the skill''s files only to URIs listed in that entry''s `resources`, and MUST treat a read of an unlisted file within the skill as a verification failure equivalent to a digest mismatch.' + - check: sep-2640-host-no-prefetch + text: 'Hosts MUST NOT retrieve a skill''s files ahead of need — not on connection, not on listing, and not at approval. A `SKILL.md` is fetched when the skill is loaded, and a supporting file when it is read.' + - check: sep-2640-host-frontmatter-comparison + text: 'After fetching a `SKILL.md` for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry''s `frontmatter`. Any discrepancy MUST be treated as a verification failure equivalent to a digest mismatch, and the skill MUST NOT be loaded.' + - check: sep-2640-host-digest-not-security-boundary + text: 'Digests are unsigned and supplied by the same server that supplies the content. A match proves the two are consistent, not that either is trustworthy. Hosts MUST NOT treat a digest match as a security boundary.' + - check: sep-2640-host-skill-identity-pair + text: 'The identity of an MCP-served skill is the pair of the host''s identity for the originating server and the skill''s `uri`. Hosts MUST preserve both halves wherever a skill is recorded or addressed — the registry, persisted approvals, the cache, and any tool or path through which the model reaches the skill — and MUST NOT key any of these on the `uri` alone.' + - check: sep-2640-host-not-skill-by-scheme + text: 'A host MUST NOT conclude that a resource is a skill merely because its URI carries a particular scheme.' + - check: sep-2640-host-cross-origin-no-shadow + text: 'When skills from different origins collide on `name`, hosts MUST resolve the name within a per-origin namespace, identifying servers by a host-assigned label; an MCP-served skill MUST NOT silently shadow, or be silently substituted for, a same-named skill from any other origin.' + - check: sep-2640-host-content-bound-approval + text: 'When a host persists any per-skill user approval, it MUST be bound to the entry''s `resources` set — every `uri` and `digest` — observed at the moment of approval. If a subsequent entry for that skill advertises a different set, the host MUST treat the prior approval as revoked and re-prompt before loading or executing.' + - check: sep-2640-host-dynamic-not-content-bound + text: 'A skill whose `resources` is `"dynamic"` cannot be content-bound: hosts MAY decline to load it, and MUST NOT treat a persisted approval as covering whatever content the server currently serves.' + - check: sep-2640-host-nested-fresh-consent + text: 'Activating a nested skill — loading it as a skill in its own right — requires fresh, explicit user consent; approval of the enclosing skill does not substitute for it. A nested `SKILL.md` read as supporting content is ordinary markdown: hosts MUST NOT act on its frontmatter.' - # resources/directory/read additions (SEP commit 2e04c48d, 2026-06-09) - - check: sep-2640-capability-directory-read-flag - text: 'Clients MUST NOT call `resources/directory/read` against a server that has not declared `directoryRead: true`.' + # === Directory Listing (resources/directory/read) === - check: sep-2640-directory-read-method-registered text: 'A server that declares `directoryRead` MUST support the method for every directory within the skill namespaces it serves as individual files.' - check: sep-2640-directory-read-subdir-mimetype @@ -89,11 +197,14 @@ requirements: text: 'The method applies only to directory resources. If the URI does not exist, or exists but is not a directory resource, the server MUST return error `-32602` (Invalid params) — the same code `resources/read` uses for unknown resources.' - check: sep-2640-directory-read-pagination text: 'Pagination mirrors `resources/list`: when the result includes `nextCursor`, the client passes it back as `cursor` to retrieve the next page.' + - check: sep-2640-directory-read-empty-dir + text: 'An empty directory yields an empty `resources` array.' + - check: sep-2640-directory-read-not-manifest-extension + text: 'Hosts MUST NOT treat the directory result as extending the manifest. While acting on the skill under the held entry, the host MUST NOT read a newly listed child — an unlisted file is a verification failure, exactly as a digest mismatch is — and MUST NOT surface it to the model as a file of the skill.' + # === Excluded: not observable on the MCP wire === - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' - - text: "[Hosts] SHOULD determine the format from the resource's `mimeType`, falling back to the URL suffix" - excluded: 'Internal decision logic: when `mimeType` and URL suffix agree, the harness cannot distinguish a host that branched on `mimeType` from one that fell back to the suffix.' - text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.' excluded: 'Internal host policy: "treats as untrusted" is an assertion about how content is reasoned over downstream of the read, not about wire traffic.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications @@ -109,3 +220,16 @@ requirements: - text: "Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill's content before it is loaded into model context" excluded: 'UI presentation requirements (origin indicator, pre-load inspection); the harness cannot observe what the host displays to users.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - text: 'A host is _acting on_ a skill from the moment it loads the skill''s `SKILL.md` into the model''s context until, at the earliest, that `SKILL.md` leaves context; hosts MAY hold the window open longer, never shorter.' + excluded: 'The window is defined by what is in the model''s context, which is host-internal state the harness cannot observe. It scopes several wire-observable rules but is not itself checkable.' + - text: 'Hosts SHOULD instead cache what they do retrieve, and digests make that cache cheap to validate: a cached file whose digest matches the current entry can be served without fetching it again.' + excluded: 'Cache hits are the absence of a request. A harness cannot distinguish a compliant cache from a host that simply did not need the file again.' + - text: 'Any path at which a host materializes skill content, whether a cache directory or a virtual mount, MUST encode the server identity as well as the `uri`, so that same-URI skills from different servers land at distinct paths.' + excluded: 'Disk layout and cache-path construction are host-local filesystem concerns with no wire representation.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - text: 'A host that declines a skill on this basis SHOULD tell the user why rather than fail silently on a later read.' + excluded: 'User-facing messaging; not protocol-observable.' + - text: 'For a skill whose `resources` is `"dynamic"`, the entry offers nothing to count. A host that chooses to load such a skill applies the total-size limit to what it actually retrieves and MAY stop loading the skill once that limit is reached.' + excluded: 'A host stopping mid-load is indistinguishable on the wire from a host that needed no further files.' + - text: 'Hosts SHOULD expect this sequence and present it as such — a skill that has changed and needs re-approval — rather than as a read error.' + excluded: 'User-facing presentation of the stale-entry recovery path; not protocol-observable.' From 3d75cb0fb63bd0d8fae879b0a84337242e062d7a Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Sat, 29 Aug 2026 11:20:53 -0700 Subject: [PATCH 08/25] feat(sep-2640): close the extraction gaps found by a normative-sentence audit The first pass extracted the Specification sections but under-covered Security Implications, which carries 21 of the SEP's 97 RFC-2119 sentences. Sweeping every normative sentence in the source surfaced rows that were missing entirely: - origin tagging at the point content enters model context (the L179 host MUST) - allowed-tools MUST be ignored for MCP-origin skills - reads bound to the originating server, so a skill from A cannot drive a read against B, and any cross-origin read gated behind per-call approval - servers identified by a host-assigned label, never serverInfo.name - the three disk-cache rules: write-isolation or re-hash, exclusion from filesystem-skill discovery paths, and removal when a server is removed - a resources/read of a SKILL.md by another route is not a load - a nested SKILL.md read as supporting content is ordinary markdown - the registry-assembly and virtual-mount no-prefetch restatements - the SDK affordances guidance, which is the row mcpkit itself is measured by Five rows previously filed as excluded are now declared instead. They are real requirements that a host-side harness could check; calling them unobservable was a limitation of this scenario set, not of the requirement. Adds one wire check, names-should-be-unique, since a within-listing collision is observable and tells an operator the listing needs disambiguating. 96 rows: 89 checks (40 wire-tested) and 7 excluded. --- src/scenarios/server/skills/enumeration.ts | 30 ++++++- src/seps/sep-2640.yaml | 92 ++++++++++++++++++---- 2 files changed, 106 insertions(+), 16 deletions(-) diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts index 837a1568..6dbc3fdb 100644 --- a/src/scenarios/server/skills/enumeration.ts +++ b/src/scenarios/server/skills/enumeration.ts @@ -75,7 +75,8 @@ const ENTRY_IDS = [ 'sep-2640-limit-total-size', 'sep-2640-metadata-reserved-prefix', 'sep-2640-name-naming-rules', - 'sep-2640-authority-reg-name' + 'sep-2640-authority-reg-name', + 'sep-2640-names-should-be-unique' ] as const; const GET_IDS = [ @@ -729,6 +730,33 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { ) ); + // === names-should-be-unique (SHOULD) === + // A collision is explicitly permitted — two skills at different paths may + // share a final segment — so this is a WARNING that tells a host operator the + // listing will need disambiguating, not a failure. + const byName = new Map(); + entries.forEach((e) => { + if (typeof e.uri !== 'string') return; + const n = skillNameFromManifestUri(e.uri); + if (n === undefined) return; + byName.set(n, [...(byName.get(n) ?? []), e.uri]); + }); + const collisions = [...byName.entries()] + .filter(([, uris]) => uris.length > 1) + .map(([n, uris]) => `"${n}" served at ${uris.join(' and ')}`); + checks.push( + skillsCheck( + 'sep-2640-names-should-be-unique', + "Within a server's listing, names SHOULD be unique, but they are not guaranteed to be.", + collisions.length === 0 ? 'SUCCESS' : 'WARNING', + collisions.length === 0 + ? { details: { distinctNames: byName.size } } + : { + errorMessage: `Names collide within one listing, so hosts MUST disambiguate them: ${joinErrs(collisions)}` + } + ) + ); + return checks; } diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 1b0679e5..6a51ac8c 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -23,6 +23,16 @@ # archive rows) and four were reworded. Because the SEP is mid-vote, expect one # more pass at the CM-stamped head. # +# coverage: extracted against branch head a3e147ca2710 (2026-08-25), the commit +# the CM vote is running on. A keyword sweep of the source finds 97 sentences +# carrying an RFC 2119 term; this file declares 89 checks plus 7 excluded rows. +# The residual handful are restatements that fold into a declared row rather +# than standing alone. Of the 89 declared checks, 40 are exercised on the wire +# by the three server scenarios; the other 49 are host obligations (retrieval +# policy, approval binding, cache isolation, context handling) that a +# server-side harness cannot observe and that would need a host-side scenario +# set to test. +# # backing_scenarios: three server ClientScenarios under # src/scenarios/server/skills/ emit the check IDs below (a row is "tested" # once a scenario emits its check ID; see src/traceability/): @@ -62,6 +72,8 @@ requirements: # SKILL.md MUST NOT appear in any descendant directory. - check: sep-2640-nested-skills-permitted text: 'A `SKILL.md` MAY appear in a descendant directory of a skill — skills can nest.' + - check: sep-2640-nested-skillmd-not-acted-on + text: 'From the enclosing skill''s perspective, a nested skill''s directory and files are ordinary supporting files, and reading them is ordinary reading. A nested `SKILL.md` read this way is ordinary markdown: hosts MUST NOT act on its frontmatter.' - check: sep-2640-nested-publication-flat text: "A nested skill is published like any other: through its own `skills/list` entry, or by explicit reference. The listing remains flat — an entry for a nested skill is an ordinary entry whose `uri` happens to share a path prefix with the enclosing skill's, and nothing in the listing marks nesting." @@ -117,6 +129,10 @@ requirements: text: "The final `` segment of the entry's `uri` MUST equal `frontmatter.name`, per Resource Mapping." - check: sep-2640-metadata-reserved-prefix text: 'Within the frontmatter `metadata` object, keys prefixed with `io.modelcontextprotocol/` are reserved for metadata defined by MCP extensions. This extension currently defines no such keys. Implementations SHOULD ignore keys under this prefix that they do not recognize.' + - check: sep-2640-names-should-be-unique + text: "Within a server's listing, names SHOULD be unique, but they are not guaranteed to be." + - check: sep-2640-host-disambiguate-listing-collision + text: 'When two entries in one listing collide on `name`, hosts MUST disambiguate them — for example by their distinguishing path segments — rather than silently discarding or preferring one.' - check: sep-2640-names-not-unique text: "A skill's `name` is a label, not an identifier. Within a server's listing, names SHOULD be unique, but they are not guaranteed to be. Hosts MUST NOT assume name uniqueness. When two entries in one listing collide on `name`, hosts MUST disambiguate them — for example by their distinguishing path segments — rather than silently discarding or preferring one." @@ -161,6 +177,8 @@ requirements: # === Reading / integrity (host obligations) === - check: sep-2640-host-load-by-uri text: 'hosts MUST support loading a skill given only its URI' + - check: sep-2640-read-is-not-a-load + text: 'Hosts MUST NOT treat a `resources/read` of a `SKILL.md` that arrives by any other route as a load: it grants no approval, opens no window, and confers no standing on the skill''s supporting files.' - check: sep-2640-host-verify-digest text: 'When a host retrieves a file listed in a skill''s `resources`, it MUST verify the content against that entry''s digest. Whatever the cause, hosts MUST NOT use the unverified content.' - check: sep-2640-host-size-mismatch-failure @@ -186,6 +204,65 @@ requirements: - check: sep-2640-host-nested-fresh-consent text: 'Activating a nested skill — loading it as a skill in its own right — requires fresh, explicit user consent; approval of the enclosing skill does not substitute for it. A nested `SKILL.md` read as supporting content is ordinary markdown: hosts MUST NOT act on its frontmatter.' + - check: sep-2640-host-origin-tag-visible + text: 'Hosts MUST tag MCP-served skill content with its originating server identity at the point it enters model context, and MUST NOT present it as indistinguishable from a local filesystem skill.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-untrusted-input + text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-skills-are-data-not-directives + text: 'Hosts MUST NOT treat skill resources as higher-authority than other context. Explicit user policy governs whether a skill is loaded at all.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-cache-path-encodes-origin + text: 'Any path at which a host materializes skill content, whether a cache directory or a virtual mount, MUST encode the server identity as well as the `uri`, so that same-URI skills from different servers land at distinct paths and the originating server is recoverable from the path.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-higher-risk-surface + text: 'Hosts MUST treat MCP-served skills as a higher-risk surface than remote tool invocation.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-no-implicit-local-execution + text: 'Hosts MUST NOT allow MCP-served skill content to cause host-side code execution without explicit per-skill user approval.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-exec-gate-while-acting + text: 'Hosts MUST apply the same approval gate to code-execution tool calls issued while the model is acting on an MCP-served skill.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-reads-bound-to-origin + text: 'Hosts MUST bind such reads to the skill''s originating server: a skill served by server A MUST NOT cause a `resources/read` against server B.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-cross-origin-read-approval + text: 'Any cross-origin read MUST be gated behind explicit per-call user approval naming both servers.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-label-not-serverinfo-name + text: 'Hosts MUST identify servers by a host-assigned label, not the server''s self-reported `serverInfo.name`.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-no-implicit-permission-grants + text: 'Hosts MUST NOT honor frontmatter fields that widen the model''s tool or filesystem permissions when the skill arrives over MCP.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-ignore-allowed-tools + text: 'The Agent Skills `allowed-tools` field, which a filesystem-sourced skill uses to declare the tools available while it runs, MUST be ignored for MCP-origin skills.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-intermediary-meta-prefix + text: 'Intermediaries MAY attach provenance or verification annotations via `_meta` under their own reverse-domain prefix — not the `io.modelcontextprotocol.skills/` prefix reserved for this extension.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-cache-write-isolation + text: 'Hosts that cache skill content on disk MUST do one of the following for every file served from the cache: keep the cache where nothing but the host can write to it, or re-hash the file against the entry on every read.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-cache-excluded-from-discovery + text: 'Hosts that cache MCP-served skill content on disk MUST also do so in a location excluded from every filesystem-skill discovery path, and MUST treat content loaded from there as MCP-origin.' + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + - check: sep-2640-host-cache-removal-on-server-removal + text: "Hosts SHOULD remove a server's cached skill content when the user removes that server." + url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications + + # === Implementation Guidelines (host + SDK) === + - check: sep-2640-host-registry-no-fetch + text: 'Assembling the registry reads only the listing: the host MUST NOT fetch `SKILL.md` or any supporting file at this stage.' + - check: sep-2640-host-virtual-mount-lazy + text: 'A virtual mount resolves reads on access; it MUST NOT be populated by fetching the skill''s files in advance.' + - check: sep-2640-host-surface-directory-read + text: 'When the originating server declares `directoryRead`, the host SHOULD surface this capability to the model.' + - check: sep-2640-sdk-convenience-wrappers + text: 'SDK maintainers SHOULD provide affordances that wrap the underlying resource operations in skill-specific terms. The SDK handles: reading `SKILL.md` frontmatter to populate resource metadata, serving file content on `resources/read`, and answering `skills/get` — and, where the server''s skill set is bounded, `skills/list` — computing entry digests and sizes from the registered files, and warning when a registered skill exceeds the Limits.' + # === Directory Listing (resources/directory/read) === - check: sep-2640-directory-read-method-registered text: 'A server that declares `directoryRead` MUST support the method for every directory within the skill namespaces it serves as individual files.' @@ -205,18 +282,6 @@ requirements: # === Excluded: not observable on the MCP wire === - text: 'Per RFC 3986, the first segment of `` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.' excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.' - - text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.' - excluded: 'Internal host policy: "treats as untrusted" is an assertion about how content is reasoned over downstream of the read, not about wire traffic.' - url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - - text: 'Hosts MUST NOT honor mechanisms in skill content that would cause local code execution without explicit user opt-in. This includes, non-exhaustively: hook declarations, pre/post-invocation scripts, shell commands embedded in frontmatter, or any field that a filesystem-sourced skill might use to register executable behavior on the host.' - excluded: 'Local code execution and explicit user opt-in are host-side filesystem and UX behaviors; not protocol-observable on the wire.' - url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - - text: 'Hosts MUST either ignore such fields entirely when the skill arrives over MCP, or gate them behind an explicit per-skill user approval that states what will execute and where.' - excluded: 'Either branch (silent ignore vs. UI-gated approval) is a host-internal handling choice; not protocol-observable.' - url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - - text: 'Hosts MUST NOT treat skill resources as higher-authority than other context. Explicit user policy governs whether a skill is loaded at all.' - excluded: 'Context-authority ordering is an internal prompting decision; not protocol-observable.' - url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - text: "Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill's content before it is loaded into model context" excluded: 'UI presentation requirements (origin indicator, pre-load inspection); the harness cannot observe what the host displays to users.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications @@ -224,9 +289,6 @@ requirements: excluded: 'The window is defined by what is in the model''s context, which is host-internal state the harness cannot observe. It scopes several wire-observable rules but is not itself checkable.' - text: 'Hosts SHOULD instead cache what they do retrieve, and digests make that cache cheap to validate: a cached file whose digest matches the current entry can be served without fetching it again.' excluded: 'Cache hits are the absence of a request. A harness cannot distinguish a compliant cache from a host that simply did not need the file again.' - - text: 'Any path at which a host materializes skill content, whether a cache directory or a virtual mount, MUST encode the server identity as well as the `uri`, so that same-URI skills from different servers land at distinct paths.' - excluded: 'Disk layout and cache-path construction are host-local filesystem concerns with no wire representation.' - url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - text: 'A host that declines a skill on this basis SHOULD tell the user why rather than fail silently on a later read.' excluded: 'User-facing messaging; not protocol-observable.' - text: 'For a skill whose `resources` is `"dynamic"`, the entry offers nothing to count. A host that chooses to load such a skill applies the total-size limit to what it actually retrieves and MAY stop loading the skill once that limit is reached.' From 22f4ecbe33c6680959f8254aa13a2d77a0025d19 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Sat, 29 Aug 2026 11:45:06 -0700 Subject: [PATCH 09/25] style: apply prettier to the SEP-2640 scenarios and yaml Quote-style normalisation in the yaml (escaped single quotes become double-quoted scalars) and one return-type wrap in helpers.ts. The parsed yaml is byte-identical in content; verified by comparing the parse before and after. --- src/scenarios/server/skills/enumeration.ts | 53 ++++++++++++++++------ src/scenarios/server/skills/helpers.ts | 7 +-- src/seps/sep-2640.yaml | 52 ++++++++++----------- 3 files changed, 69 insertions(+), 43 deletions(-) diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts index 6dbc3fdb..60c2a234 100644 --- a/src/scenarios/server/skills/enumeration.ts +++ b/src/scenarios/server/skills/enumeration.ts @@ -368,7 +368,9 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { entries.forEach((e, i) => { const fm = e.frontmatter; if (!fm || typeof fm !== 'object' || Array.isArray(fm)) { - fmErrs.push(`${entryLabel(e, i)}: frontmatter is missing or not an object`); + fmErrs.push( + `${entryLabel(e, i)}: frontmatter is missing or not an object` + ); return; } const obj = fm as Record; @@ -426,7 +428,9 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { // skills under another scheme native to its domain and no scheme is // privileged, so a deviation is a WARNING rather than a failure. const otherScheme = entries - .filter((e) => typeof e.uri === 'string' && !e.uri.startsWith(SKILL_URI_SCHEME)) + .filter( + (e) => typeof e.uri === 'string' && !e.uri.startsWith(SKILL_URI_SCHEME) + ) .map((e) => String(e.uri)); checks.push( skillsCheck( @@ -469,8 +473,9 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { // Entries carrying an array are the only ones the remaining checks apply to. const arrayEntries = entries .map((e, i) => ({ e, i, arr: resourcesArray(e) })) - .filter((x): x is { e: SkillEntry; i: number; arr: SkillResourceEntry[] } => - x.arr !== undefined + .filter( + (x): x is { e: SkillEntry; i: number; arr: SkillResourceEntry[] } => + x.arr !== undefined ); const dynamicOnlyReason = @@ -561,7 +566,10 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { const digestErrs: string[] = []; for (const { e, i, arr } of arrayEntries) { for (const r of arr) { - if (typeof r.digest !== 'string' || !SKILL_DIGEST_PATTERN.test(r.digest)) { + if ( + typeof r.digest !== 'string' || + !SKILL_DIGEST_PATTERN.test(r.digest) + ) { digestErrs.push( `${entryLabel(e, i)}: ${String(r.uri)} digest=${JSON.stringify(r.digest)} is not sha256:{64 lowercase hex}` ); @@ -665,7 +673,9 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { if (!meta || typeof meta !== 'object' || Array.isArray(meta)) return; for (const key of Object.keys(meta as Record)) { if (key.startsWith(FRONTMATTER_RESERVED_PREFIX)) { - reservedErrs.push(`${entryLabel(e, i)}: frontmatter.metadata["${key}"]`); + reservedErrs.push( + `${entryLabel(e, i)}: frontmatter.metadata["${key}"]` + ); } } }); @@ -772,7 +782,8 @@ async function readbackChecks( ): Promise { const checks: ConformanceCheck[] = []; const sample = entries.find( - (e) => typeof e.uri === 'string' && e.uri.endsWith(`/${SKILL_MANIFEST_FILENAME}`) + (e) => + typeof e.uri === 'string' && e.uri.endsWith(`/${SKILL_MANIFEST_FILENAME}`) ); const uri = sample?.uri as string | undefined; @@ -797,7 +808,10 @@ async function readbackChecks( { errorMessage: reason } ) ); - for (const id of ['sep-2640-skillmd-frontmatter', 'sep-2640-entry-frontmatter-identical']) { + for (const id of [ + 'sep-2640-skillmd-frontmatter', + 'sep-2640-entry-frontmatter-identical' + ]) { checks.push( skillsCheck(id, 'SKILL.md is unreadable.', 'SKIPPED', { errorMessage: reason @@ -817,7 +831,10 @@ async function readbackChecks( { errorMessage: reason } ) ); - for (const id of ['sep-2640-skillmd-frontmatter', 'sep-2640-entry-frontmatter-identical']) { + for (const id of [ + 'sep-2640-skillmd-frontmatter', + 'sep-2640-entry-frontmatter-identical' + ]) { checks.push( skillsCheck(id, 'No SKILL.md content to inspect.', 'SKIPPED', { errorMessage: reason @@ -862,7 +879,12 @@ async function readbackChecks( // === entry-frontmatter-identical === const declared = sample?.frontmatter as Record | undefined; - if (!fm || !declared || typeof declared !== 'object' || Array.isArray(declared)) { + if ( + !fm || + !declared || + typeof declared !== 'object' || + Array.isArray(declared) + ) { const reason = 'Either the file has no parseable frontmatter or the entry carries no frontmatter object, so the two cannot be compared.'; checks.push( @@ -912,9 +934,7 @@ async function getChecks( const reason = 'No listed entry carries a uri, so skills/get cannot be exercised against a known skill.'; for (const id of GET_IDS) { - checks.push( - skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason }) - ); + checks.push(skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason })); } return checks; } @@ -1025,7 +1045,12 @@ async function getChecks( 'If the URI does not identify a skill the server serves, the server MUST return error -32602 (Invalid params).', ok ? 'SUCCESS' : 'FAILURE', ok - ? { details: { probedUri: UNKNOWN_SKILL_URI, code: unknown.error.code } } + ? { + details: { + probedUri: UNKNOWN_SKILL_URI, + code: unknown.error.code + } + } : { errorMessage: `skills/get on an unserved URI returned code ${unknown.error.code} (${unknown.error.message}); expected ${JSONRPC_INVALID_PARAMS}.` } diff --git a/src/scenarios/server/skills/helpers.ts b/src/scenarios/server/skills/helpers.ts index 7bd24943..699fc307 100644 --- a/src/scenarios/server/skills/helpers.ts +++ b/src/scenarios/server/skills/helpers.ts @@ -158,9 +158,10 @@ export async function skillsCapability( * at Final status, they do not. The envelope is a non-conformant shape emitted * by at least one SDK, so it is reported rather than silently accepted. */ -export function settingsAreInline( - skills: Record -): { inline: boolean; envelopeKeys: string[] } { +export function settingsAreInline(skills: Record): { + inline: boolean; + envelopeKeys: string[]; +} { const envelopeKeys = ['config', 'specVersion', 'stability', 'id'].filter( (k) => k in skills ); diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 6a51ac8c..24e2ad03 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -73,7 +73,7 @@ requirements: - check: sep-2640-nested-skills-permitted text: 'A `SKILL.md` MAY appear in a descendant directory of a skill — skills can nest.' - check: sep-2640-nested-skillmd-not-acted-on - text: 'From the enclosing skill''s perspective, a nested skill''s directory and files are ordinary supporting files, and reading them is ordinary reading. A nested `SKILL.md` read this way is ordinary markdown: hosts MUST NOT act on its frontmatter.' + text: "From the enclosing skill's perspective, a nested skill's directory and files are ordinary supporting files, and reading them is ordinary reading. A nested `SKILL.md` read this way is ordinary markdown: hosts MUST NOT act on its frontmatter." - check: sep-2640-nested-publication-flat text: "A nested skill is published like any other: through its own `skills/list` entry, or by explicit reference. The listing remains flat — an entry for a nested skill is an ordinary entry whose `uri` happens to share a path prefix with the enclosing skill's, and nothing in the listing marks nesting." @@ -106,21 +106,21 @@ requirements: - check: sep-2640-skills-list-implemented text: 'A server declaring the `io.modelcontextprotocol/skills` extension MUST implement the `skills/list` method, which returns the skills it serves. The result MAY be empty.' - check: sep-2640-skills-list-pagination - text: 'Pagination mirrors the base protocol''s list methods: the request accepts an optional `cursor`, and when the result includes `nextCursor` the client passes it back to retrieve the next page.' + text: "Pagination mirrors the base protocol's list methods: the request accepts an optional `cursor`, and when the result includes `nextCursor` the client passes it back to retrieve the next page." - check: sep-2640-skills-list-entry-atomic text: "An entry is atomic — a skill's `resources` set is never split across pages." - check: sep-2640-skills-list-cache-attributes - text: 'In protocol versions 2026-07-28 and later, the result also carries the base protocol''s list-caching attributes — `ttlMs` and `cacheScope`, as defined for `tools/list` and `resources/list` (SEP-2549) — with the same semantics: a freshness hint for the listing and a cache-scope marker, not an integrity property.' + text: "In protocol versions 2026-07-28 and later, the result also carries the base protocol's list-caching attributes — `ttlMs` and `cacheScope`, as defined for `tools/list` and `resources/list` (SEP-2549) — with the same semantics: a freshness hint for the listing and a cache-scope marker, not an integrity property." - check: sep-2640-skills-list-may-be-partial text: 'A server whose skill catalog is large, generated on demand, or otherwise unenumerable MAY return an empty or partial listing.' - check: sep-2640-host-no-empty-listing-assumption text: 'Hosts MUST NOT treat an empty or partial listing as proof that a server has no skills.' - check: sep-2640-enumeration-scheme-uniform - text: 'The method serves entries for a server''s skills whatever URI scheme they use — enumeration is uniform across schemes.' + text: "The method serves entries for a server's skills whatever URI scheme they use — enumeration is uniform across schemes." # === The skill entry (shared by skills/list and skills/get) === - check: sep-2640-entry-uri-required - text: '`skills[].uri` (Yes): Resource URI of the skill''s `SKILL.md`.' + text: "`skills[].uri` (Yes): Resource URI of the skill's `SKILL.md`." - check: sep-2640-entry-frontmatter-required text: "`frontmatter` is the skill's `SKILL.md` YAML frontmatter rendered verbatim as a JSON object — every field the author wrote, not a curated subset. Because the Agent Skills specification requires `name` and `description`, those fields are always present." - check: sep-2640-entry-frontmatter-identical @@ -140,13 +140,13 @@ requirements: - check: sep-2640-entry-resources-required text: '`resources` is REQUIRED on every skill entry and takes one of two forms: an array enumerating the skill''s files — `SKILL.md` and every supporting file — as `{uri, digest, size}` triples, or the string `"dynamic"`.' - check: sep-2640-resources-complete - text: 'When present, `resources` MUST be complete: it lists every file of the skill, each exactly once, including an entry matching the skill''s top-level `uri` — that entry carries the digest and size of `SKILL.md` itself.' + text: "When present, `resources` MUST be complete: it lists every file of the skill, each exactly once, including an entry matching the skill's top-level `uri` — that entry carries the digest and size of `SKILL.md` itself." - check: sep-2640-resources-uri-within-skill - text: 'Each `uri` MUST be the skill''s `SKILL.md` or a file within the skill''s directory.' + text: "Each `uri` MUST be the skill's `SKILL.md` or a file within the skill's directory." - check: sep-2640-resources-digest-format - text: 'Digests are SHA-256 hashes of an artifact''s raw bytes, formatted as `sha256:{hex}` where `{hex}` is 64 lowercase hexadecimal characters. Each entry in a skill''s `resources` carries the digest of the file at its `uri`.' + text: "Digests are SHA-256 hashes of an artifact's raw bytes, formatted as `sha256:{hex}` where `{hex}` is 64 lowercase hexadecimal characters. Each entry in a skill's `resources` carries the digest of the file at its `uri`." - check: sep-2640-resources-size-required - text: 'Each entry MUST carry `size`: the length in bytes of the file''s raw content — the same bytes the `digest` covers.' + text: "Each entry MUST carry `size`: the length in bytes of the file's raw content — the same bytes the `digest` covers." - check: sep-2640-resources-dynamic-marker text: 'When a skill''s content is generated dynamically, such that stable digests cannot be published, the server MUST set `"resources": "dynamic"` instead of an array.' - check: sep-2640-resources-invalid-entry @@ -156,9 +156,9 @@ requirements: # === Limits === - check: sep-2640-limit-resources-per-skill - text: 'Resources per skill: 512 entries, counted over the entries of the skill''s `resources`, `SKILL.md` included.' + text: "Resources per skill: 512 entries, counted over the entries of the skill's `resources`, `SKILL.md` included." - check: sep-2640-limit-total-size - text: 'Total file size per skill: 16 MiB (16,777,216 bytes), counted over the sum of `size` over the skill''s `resources`.' + text: "Total file size per skill: 16 MiB (16,777,216 bytes), counted over the sum of `size` over the skill's `resources`." - check: sep-2640-limit-host-support text: 'Hosts MUST support skills up to and including these limits, and MAY support larger ones. Servers SHOULD NOT serve a skill that exceeds either limit; a skill that does is not guaranteed to be loadable by any conforming host.' @@ -178,27 +178,27 @@ requirements: - check: sep-2640-host-load-by-uri text: 'hosts MUST support loading a skill given only its URI' - check: sep-2640-read-is-not-a-load - text: 'Hosts MUST NOT treat a `resources/read` of a `SKILL.md` that arrives by any other route as a load: it grants no approval, opens no window, and confers no standing on the skill''s supporting files.' + text: "Hosts MUST NOT treat a `resources/read` of a `SKILL.md` that arrives by any other route as a load: it grants no approval, opens no window, and confers no standing on the skill's supporting files." - check: sep-2640-host-verify-digest - text: 'When a host retrieves a file listed in a skill''s `resources`, it MUST verify the content against that entry''s digest. Whatever the cause, hosts MUST NOT use the unverified content.' + text: "When a host retrieves a file listed in a skill's `resources`, it MUST verify the content against that entry's digest. Whatever the cause, hosts MUST NOT use the unverified content." - check: sep-2640-host-size-mismatch-failure - text: 'A read whose byte length differs from the entry''s `size` is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest.' + text: "A read whose byte length differs from the entry's `size` is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest." - check: sep-2640-host-unlisted-read-failure - text: 'While acting on a skill, a host MUST resolve reads of the skill''s files only to URIs listed in that entry''s `resources`, and MUST treat a read of an unlisted file within the skill as a verification failure equivalent to a digest mismatch.' + text: "While acting on a skill, a host MUST resolve reads of the skill's files only to URIs listed in that entry's `resources`, and MUST treat a read of an unlisted file within the skill as a verification failure equivalent to a digest mismatch." - check: sep-2640-host-no-prefetch - text: 'Hosts MUST NOT retrieve a skill''s files ahead of need — not on connection, not on listing, and not at approval. A `SKILL.md` is fetched when the skill is loaded, and a supporting file when it is read.' + text: "Hosts MUST NOT retrieve a skill's files ahead of need — not on connection, not on listing, and not at approval. A `SKILL.md` is fetched when the skill is loaded, and a supporting file when it is read." - check: sep-2640-host-frontmatter-comparison - text: 'After fetching a `SKILL.md` for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry''s `frontmatter`. Any discrepancy MUST be treated as a verification failure equivalent to a digest mismatch, and the skill MUST NOT be loaded.' + text: "After fetching a `SKILL.md` for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry's `frontmatter`. Any discrepancy MUST be treated as a verification failure equivalent to a digest mismatch, and the skill MUST NOT be loaded." - check: sep-2640-host-digest-not-security-boundary text: 'Digests are unsigned and supplied by the same server that supplies the content. A match proves the two are consistent, not that either is trustworthy. Hosts MUST NOT treat a digest match as a security boundary.' - check: sep-2640-host-skill-identity-pair - text: 'The identity of an MCP-served skill is the pair of the host''s identity for the originating server and the skill''s `uri`. Hosts MUST preserve both halves wherever a skill is recorded or addressed — the registry, persisted approvals, the cache, and any tool or path through which the model reaches the skill — and MUST NOT key any of these on the `uri` alone.' + text: "The identity of an MCP-served skill is the pair of the host's identity for the originating server and the skill's `uri`. Hosts MUST preserve both halves wherever a skill is recorded or addressed — the registry, persisted approvals, the cache, and any tool or path through which the model reaches the skill — and MUST NOT key any of these on the `uri` alone." - check: sep-2640-host-not-skill-by-scheme text: 'A host MUST NOT conclude that a resource is a skill merely because its URI carries a particular scheme.' - check: sep-2640-host-cross-origin-no-shadow text: 'When skills from different origins collide on `name`, hosts MUST resolve the name within a per-origin namespace, identifying servers by a host-assigned label; an MCP-served skill MUST NOT silently shadow, or be silently substituted for, a same-named skill from any other origin.' - check: sep-2640-host-content-bound-approval - text: 'When a host persists any per-skill user approval, it MUST be bound to the entry''s `resources` set — every `uri` and `digest` — observed at the moment of approval. If a subsequent entry for that skill advertises a different set, the host MUST treat the prior approval as revoked and re-prompt before loading or executing.' + text: "When a host persists any per-skill user approval, it MUST be bound to the entry's `resources` set — every `uri` and `digest` — observed at the moment of approval. If a subsequent entry for that skill advertises a different set, the host MUST treat the prior approval as revoked and re-prompt before loading or executing." - check: sep-2640-host-dynamic-not-content-bound text: 'A skill whose `resources` is `"dynamic"` cannot be content-bound: hosts MAY decline to load it, and MUST NOT treat a persisted approval as covering whatever content the server currently serves.' - check: sep-2640-host-nested-fresh-consent @@ -226,16 +226,16 @@ requirements: text: 'Hosts MUST apply the same approval gate to code-execution tool calls issued while the model is acting on an MCP-served skill.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - check: sep-2640-host-reads-bound-to-origin - text: 'Hosts MUST bind such reads to the skill''s originating server: a skill served by server A MUST NOT cause a `resources/read` against server B.' + text: "Hosts MUST bind such reads to the skill's originating server: a skill served by server A MUST NOT cause a `resources/read` against server B." url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - check: sep-2640-host-cross-origin-read-approval text: 'Any cross-origin read MUST be gated behind explicit per-call user approval naming both servers.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - check: sep-2640-host-label-not-serverinfo-name - text: 'Hosts MUST identify servers by a host-assigned label, not the server''s self-reported `serverInfo.name`.' + text: "Hosts MUST identify servers by a host-assigned label, not the server's self-reported `serverInfo.name`." url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - check: sep-2640-host-no-implicit-permission-grants - text: 'Hosts MUST NOT honor frontmatter fields that widen the model''s tool or filesystem permissions when the skill arrives over MCP.' + text: "Hosts MUST NOT honor frontmatter fields that widen the model's tool or filesystem permissions when the skill arrives over MCP." url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - check: sep-2640-host-ignore-allowed-tools text: 'The Agent Skills `allowed-tools` field, which a filesystem-sourced skill uses to declare the tools available while it runs, MUST be ignored for MCP-origin skills.' @@ -257,11 +257,11 @@ requirements: - check: sep-2640-host-registry-no-fetch text: 'Assembling the registry reads only the listing: the host MUST NOT fetch `SKILL.md` or any supporting file at this stage.' - check: sep-2640-host-virtual-mount-lazy - text: 'A virtual mount resolves reads on access; it MUST NOT be populated by fetching the skill''s files in advance.' + text: "A virtual mount resolves reads on access; it MUST NOT be populated by fetching the skill's files in advance." - check: sep-2640-host-surface-directory-read text: 'When the originating server declares `directoryRead`, the host SHOULD surface this capability to the model.' - check: sep-2640-sdk-convenience-wrappers - text: 'SDK maintainers SHOULD provide affordances that wrap the underlying resource operations in skill-specific terms. The SDK handles: reading `SKILL.md` frontmatter to populate resource metadata, serving file content on `resources/read`, and answering `skills/get` — and, where the server''s skill set is bounded, `skills/list` — computing entry digests and sizes from the registered files, and warning when a registered skill exceeds the Limits.' + text: "SDK maintainers SHOULD provide affordances that wrap the underlying resource operations in skill-specific terms. The SDK handles: reading `SKILL.md` frontmatter to populate resource metadata, serving file content on `resources/read`, and answering `skills/get` — and, where the server's skill set is bounded, `skills/list` — computing entry digests and sizes from the registered files, and warning when a registered skill exceeds the Limits." # === Directory Listing (resources/directory/read) === - check: sep-2640-directory-read-method-registered @@ -285,8 +285,8 @@ requirements: - text: "Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill's content before it is loaded into model context" excluded: 'UI presentation requirements (origin indicator, pre-load inspection); the harness cannot observe what the host displays to users.' url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - - text: 'A host is _acting on_ a skill from the moment it loads the skill''s `SKILL.md` into the model''s context until, at the earliest, that `SKILL.md` leaves context; hosts MAY hold the window open longer, never shorter.' - excluded: 'The window is defined by what is in the model''s context, which is host-internal state the harness cannot observe. It scopes several wire-observable rules but is not itself checkable.' + - text: "A host is _acting on_ a skill from the moment it loads the skill's `SKILL.md` into the model's context until, at the earliest, that `SKILL.md` leaves context; hosts MAY hold the window open longer, never shorter." + excluded: "The window is defined by what is in the model's context, which is host-internal state the harness cannot observe. It scopes several wire-observable rules but is not itself checkable." - text: 'Hosts SHOULD instead cache what they do retrieve, and digests make that cache cheap to validate: a cached file whose digest matches the current entry can be served without fetching it again.' excluded: 'Cache hits are the absence of a request. A harness cannot distinguish a compliant cache from a host that simply did not need the file again.' - text: 'A host that declines a skill on this basis SHOULD tell the user why rather than fail silently on a later read.' From e78284749678f9f39bf997a4b60511ec9f3c148f Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Sat, 29 Aug 2026 11:54:39 -0700 Subject: [PATCH 10/25] docs(sep-2640): note why resultType is not declared in this yaml It appears in all three result examples but is a base-protocol field on the common Result interface, not a SEP-2640 requirement; its caching-hint obligations are already owned by sep-2549.yaml. --- src/seps/sep-2640.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 24e2ad03..65ce6ad8 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -33,6 +33,13 @@ # server-side harness cannot observe and that would need a host-side scenario # set to test. # +# deliberately not declared here: `resultType`, which appears in all three +# result examples in this SEP, is a base-protocol field on the common `Result` +# interface (schema/2026-07-28/schema.ts), not a SEP-2640 requirement. Servers +# MUST include it and clients treat an absent value as "complete". Its +# caching-hint obligations belong to sep-2549.yaml. Declaring it here would +# double-count a base-protocol rule against this extension. +# # backing_scenarios: three server ClientScenarios under # src/scenarios/server/skills/ emit the check IDs below (a row is "tested" # once a scenario emits its check ID; see src/traceability/): From 1f21fbaf3c388d9091149e07609729c00afd5bfe Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Fri, 4 Sep 2026 15:49:28 -0700 Subject: [PATCH 11/25] fix(sep-2640): follow nextCursor on resources/directory/read The directory scenario issued one resources/directory/read and every check below it inspected that single page: the result-shape check, the subdirectory-mimeType check, and the target selection for the -32602 probe. SEP-2640 says directory-read pagination mirrors resources/list, so a conformant server MAY split a directory across pages, and reading only the first made "no subdirectory here" indistinguishable from "the subdirectory is on page two". Reported by Sam Bloomberg against the go-sdk implementation, which exposes DirectoryOptions.PageSize. Rebuilding that server with PageSize 1 fails sep-2640-directory-read-subdir-mimetype; with this change it passes 7/7, and both servers still pass at their default page size. Only findable against a second implementation. mcpkit hardcoded its page size to zero with no override, so it never emitted a cursor and the defect was invisible to every run against it. listAllResources already did this for resources/list. directoryReadAll applies the same shape to directory reads, with a page bound and a repeated-cursor guard. --- src/scenarios/server/skills/directory.ts | 5 ++++ src/scenarios/server/skills/helpers.ts | 34 ++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/src/scenarios/server/skills/directory.ts b/src/scenarios/server/skills/directory.ts index 4c5c2b02..46ec603a 100644 --- a/src/scenarios/server/skills/directory.ts +++ b/src/scenarios/server/skills/directory.ts @@ -33,6 +33,7 @@ import { skillsCheck, listAllResources, skillsListAll, + directoryReadAll, skillNameFromManifestUri } from './helpers'; @@ -210,6 +211,10 @@ export class SkillsDirectoryReadScenario implements ClientScenario { 'resources/directory/read', { uri: target.dirUri } ); + // Every check below inspects the directory's children, so they need + // the whole directory rather than whichever slice fits one page. + const all = await directoryReadAll(conn, target.dirUri); + happy = { ...happy, resources: all.resources }; } catch (e) { happyErr = e; } diff --git a/src/scenarios/server/skills/helpers.ts b/src/scenarios/server/skills/helpers.ts index 699fc307..d682b32c 100644 --- a/src/scenarios/server/skills/helpers.ts +++ b/src/scenarios/server/skills/helpers.ts @@ -376,3 +376,37 @@ export function resourcesArray( export function entryLabel(entry: SkillEntry, i: number): string { return typeof entry.uri === 'string' ? entry.uri : `skills[${i}]`; } + +/** + * Every child of a directory, following `nextCursor` until it clears. + * + * SEP-2640 says directory-read pagination mirrors `resources/list`, so a + * conformant server MAY split a directory across pages. Reading only the + * first page makes "no subdirectory here" indistinguishable from "the + * subdirectory is on page two". + */ +export async function directoryReadAll( + conn: Connection, + uri: string, + maxPages = 50 +): Promise<{ resources: SkillResource[]; pages: number; truncated: boolean }> { + const resources: SkillResource[] = []; + const seen = new Set(); + let cursor: string | undefined; + + for (let i = 0; i < maxPages; i++) { + const page = await conn.request<{ + resources?: SkillResource[]; + nextCursor?: string; + }>('resources/directory/read', cursor ? { uri, cursor } : { uri }); + resources.push(...(page.resources ?? [])); + const next = page.nextCursor; + if (typeof next !== 'string' || next.length === 0) { + return { resources, pages: i + 1, truncated: false }; + } + if (seen.has(next)) return { resources, pages: i + 1, truncated: true }; + seen.add(next); + cursor = next; + } + return { resources, pages: maxPages, truncated: true }; +} From fa8aaec3611e387249da6adc826b02dd77372a3e Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Fri, 4 Sep 2026 18:59:44 -0700 Subject: [PATCH 12/25] feat(sep-2640): client scenario for the no-prefetch MUST NOT The traceability file filed retrieval policy under "host obligations a harness cannot observe", alongside 48 other rows. That was wrong for this one: "hosts MUST NOT retrieve a skill's files ahead of need" reduces to whether a request arrived, which makes it among the most wire-observable rules in the SEP. Adds a Scenario (harness stands up the server, client is the SUT) that serves a listing naming a SKILL.md and a supporting file, then fails if either is read. The client's contract, keyed on MCP_CONFORMANCE_SCENARIO, is to connect, enumerate, and stop. SKIPs rather than passes when the client never calls skills/list, since without a listing there is no window in which prefetching is possible and a pass would prove nothing. Verified both directions against mcpkit's client: SUCCESS as written, and FAILURE with a deliberate ReadFromEntry added to the driver. A check that has never been shown to fail is the same trap as the directory-pagination false negative this suite shipped for months. --- src/scenarios/client/skills/no-prefetch.ts | 172 +++++++++++++++++++++ src/scenarios/index.ts | 8 +- 2 files changed, 179 insertions(+), 1 deletion(-) create mode 100644 src/scenarios/client/skills/no-prefetch.ts diff --git a/src/scenarios/client/skills/no-prefetch.ts b/src/scenarios/client/skills/no-prefetch.ts new file mode 100644 index 00000000..68bbc864 --- /dev/null +++ b/src/scenarios/client/skills/no-prefetch.ts @@ -0,0 +1,172 @@ +/** + * SEP-2640 client scenario: hosts MUST NOT retrieve a skill's files ahead of + * need. + * + * This is a `Scenario` rather than a `ClientScenario`: the harness stands up + * the server and the client is the system under test. The rule reduces to + * "did a request arrive", which makes it one of the most wire-observable + * obligations in the SEP despite the first traceability pass filing it as an + * unobservable host obligation. + * + * Contract for the client under test, keyed on MCP_CONFORMANCE_SCENARIO: + * connect, call `skills/list`, then exit. Do not load a skill. A client that + * prefetches will read `SKILL.md` or a supporting file during that window and + * fail the check. + */ + +import http from 'http'; +import { ConformanceCheck } from '../../../types.js'; +import { BaseHttpScenario } from '../http-base.js'; + +const SPEC_REFERENCE = { + id: 'SEP-2640-Lazy-Retrieval', + url: 'https://modelcontextprotocol.io/seps/2640-skills-extension#integrity-and-verification' +}; + +const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; + +/** One skill with a supporting file, so a prefetch has something to grab. */ +const SKILL_URI = 'skill://pdf-processing/SKILL.md'; +const SUPPORTING_URI = 'skill://pdf-processing/references/FORMS.md'; + +const SKILL_MD = `--- +name: pdf-processing +description: Extract, fill, and assemble PDF documents +--- + +Body the client has no business fetching yet. +`; + +const SUPPORTING = + 'Supporting content the client has no business fetching yet.\n'; + +/** sha256 of the two bodies, computed at module load so the entry is honest. */ +import { createHash } from 'crypto'; +const digestOf = (s: string) => + 'sha256:' + createHash('sha256').update(s, 'utf8').digest('hex'); + +export class SkillsNoPrefetchScenario extends BaseHttpScenario { + name = 'sep-2640-client-no-prefetch'; + description = + 'A client MUST NOT retrieve a skill file before the skill is loaded'; + readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; + + /** Every resources/read URI the client asked for, in order. */ + private readsRequested: string[] = []; + private listCalled = false; + + protected handlePost( + _req: http.IncomingMessage, + res: http.ServerResponse, + request: any + ): void { + switch (request.method) { + case 'initialize': + this.sendInitialize(res, request, { + resources: { listChanged: false }, + extensions: { [SKILLS_EXTENSION_ID]: {} } + }); + return; + + case 'skills/list': + this.listCalled = true; + this.sendJson(res, { + jsonrpc: '2.0', + id: request.id, + result: { + resultType: 'complete', + skills: [ + { + uri: SKILL_URI, + frontmatter: { + name: 'pdf-processing', + description: 'Extract, fill, and assemble PDF documents' + }, + resources: [ + { + uri: SKILL_URI, + digest: digestOf(SKILL_MD), + size: Buffer.byteLength(SKILL_MD) + }, + { + uri: SUPPORTING_URI, + digest: digestOf(SUPPORTING), + size: Buffer.byteLength(SUPPORTING) + } + ] + } + ] + } + }); + return; + + // Served, but reaching it during this scenario is the failure. + case 'resources/read': { + const uri = request.params?.uri; + if (typeof uri === 'string') this.readsRequested.push(uri); + const body = uri === SUPPORTING_URI ? SUPPORTING : SKILL_MD; + this.sendJson(res, { + jsonrpc: '2.0', + id: request.id, + result: { + resultType: 'complete', + contents: [{ uri, mimeType: 'text/markdown', text: body }] + } + }); + return; + } + + default: + if (request.id === undefined) { + this.sendNotificationAck(res); + return; + } + this.sendGenericResult(res, request); + } + } + + getChecks(): ConformanceCheck[] { + const DESC = + "Hosts MUST NOT retrieve a skill's files ahead of need, not on connection, not on listing, and not at approval."; + + // Without a listing there is no window in which prefetching is even + // possible, so the run proves nothing rather than passing. + if (!this.listCalled) { + return [ + { + id: 'sep-2640-host-no-prefetch', + name: 'SkillsClientNoPrefetch', + description: DESC, + status: 'SKIPPED', + timestamp: new Date().toISOString(), + errorMessage: + 'the client never called skills/list, so no retrieval window was opened', + specReferences: [SPEC_REFERENCE] + } + ]; + } + + const prefetched = this.readsRequested.filter( + (u) => u === SKILL_URI || u === SUPPORTING_URI + ); + + return [ + { + id: 'sep-2640-host-no-prefetch', + name: 'SkillsClientNoPrefetch', + description: DESC, + status: prefetched.length === 0 ? 'SUCCESS' : 'FAILURE', + timestamp: new Date().toISOString(), + errorMessage: + prefetched.length === 0 + ? undefined + : `client read ${prefetched.length} skill file(s) without loading a skill: ${prefetched.join(', ')}`, + specReferences: [SPEC_REFERENCE], + details: { + skillsListCalled: true, + fileReads: prefetched.length + } + } + ]; + } +} diff --git a/src/scenarios/index.ts b/src/scenarios/index.ts index 52c8ed0f..6c5330bb 100644 --- a/src/scenarios/index.ts +++ b/src/scenarios/index.ts @@ -10,6 +10,7 @@ import { DRAFT_PROTOCOL_VERSION } from '../types'; import { InitializeScenario } from './client/initialize'; +import { SkillsNoPrefetchScenario } from './client/skills/no-prefetch'; import { ToolsCallScenario } from './client/tools_call'; import { ElicitationClientDefaultsScenario } from './client/elicitation-defaults'; import { SSERetryScenario } from './client/sse-retry'; @@ -335,7 +336,12 @@ const scenariosList: Scenario[] = [ new JsonSchemaRefDerefScenario(), // JSON Schema 2020-12 client-side keyword preservation (SEP-1613, SEP-2106) - new JsonSchema2020_12PreservationScenario() + new JsonSchema2020_12PreservationScenario(), + + // SEP-2640 skills, client side. The harness is the server and grades what + // the client requests, which is how the retrieval-policy MUSTs become + // observable at all. + new SkillsNoPrefetchScenario() ]; // Core scenarios (tier 1 requirements) From 378f258587b39201865af3b1c1dba5f6e9a9c927 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Fri, 4 Sep 2026 19:09:46 -0700 Subject: [PATCH 13/25] feat(sep-2640): client scenarios for the four read-time verification MUSTs Adds digest, size, frontmatter and unlisted-URI variants alongside the no-prefetch scenario. Each serves an internally honest listing, tampers with exactly one thing on the wire, and watches whether the client notices. Rejection is detected by absence, following auth/resource-mismatch: the client's contract is "load the skill, then read its supporting file", and a client that verifies aborts on the load and never reaches the supporting file. Reaching it is the violation. The unlisted variant is proven by the read never appearing on the wire at all, since a conformant client refuses locally. The frontmatter variant recomputes digest and size for the tampered body, so the frontmatter comparison is the only thing left that can fail. Without that the digest check would mask it and the scenario would pass for the wrong reason. allowClientError is set on all four: a verifying client exits non-zero because it correctly aborted, and that is the pass. Verified both directions against mcpkit's client. All four SUCCESS as written; all four FAILURE against a driver rewritten to read through ReadSkillURI, which skips verification entirely. --- src/scenarios/client/skills/verification.ts | 246 ++++++++++++++++++++ src/scenarios/index.ts | 7 +- 2 files changed, 252 insertions(+), 1 deletion(-) create mode 100644 src/scenarios/client/skills/verification.ts diff --git a/src/scenarios/client/skills/verification.ts b/src/scenarios/client/skills/verification.ts new file mode 100644 index 00000000..6c2e1cc2 --- /dev/null +++ b/src/scenarios/client/skills/verification.ts @@ -0,0 +1,246 @@ +/** + * SEP-2640 client scenarios for the four read-time verification MUSTs. + * + * Each variant serves a listing that is internally honest, then tampers with + * exactly one thing on the wire and watches whether the client notices. The + * tamper always targets the skill's `SKILL.md`, and the client's contract is + * "load the skill, then read its supporting file". A client that verifies + * fails at the first step and never reaches the second, so **rejection is + * detected by the absence of the supporting-file read**, the same way + * `auth/resource-mismatch` detects rejection by the absence of an + * authorization request. + * + * `unlisted` is the odd one out: nothing is tampered, the client is simply + * asked for a URI the manifest does not contain. A conformant client refuses + * locally and never puts the request on the wire at all. + */ + +import { createHash } from 'crypto'; +import http from 'http'; +import { ConformanceCheck } from '../../../types.js'; +import { BaseHttpScenario } from '../http-base.js'; + +const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; + +const SPEC_REFERENCE = { + id: 'SEP-2640-Verification-On-Read', + url: 'https://modelcontextprotocol.io/seps/2640-skills-extension#integrity-and-verification' +}; + +const SKILL_URI = 'skill://pdf-processing/SKILL.md'; +const SUPPORTING_URI = 'skill://pdf-processing/references/FORMS.md'; +const UNLISTED_URI = 'skill://pdf-processing/scripts/extract.py'; + +const SKILL_MD = `--- +name: pdf-processing +description: Extract, fill, and assemble PDF documents +--- + +Skill body. +`; + +const SUPPORTING = 'Supporting content.\n'; + +const sha256 = (s: string) => + 'sha256:' + createHash('sha256').update(s, 'utf8').digest('hex'); + +export type VerificationMode = 'digest' | 'size' | 'frontmatter' | 'unlisted'; + +const MODES: Record< + VerificationMode, + { scenario: string; checkId: string; description: string } +> = { + digest: { + scenario: 'sep-2640-client-verify-digest', + checkId: 'sep-2640-host-verify-digest', + description: + "When a host retrieves a file listed in a skill's resources, it MUST verify the content against that entry's digest, and MUST NOT use unverified content." + }, + size: { + scenario: 'sep-2640-client-verify-size', + checkId: 'sep-2640-host-size-mismatch-failure', + description: + "A read whose byte length differs from the entry's size is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest." + }, + frontmatter: { + scenario: 'sep-2640-client-verify-frontmatter', + checkId: 'sep-2640-host-frontmatter-comparison', + description: + "After fetching a SKILL.md for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry's frontmatter, and MUST NOT load the skill on any discrepancy." + }, + unlisted: { + scenario: 'sep-2640-client-verify-unlisted', + checkId: 'sep-2640-host-unlisted-read-failure', + description: + "While acting on a skill, a host MUST resolve reads of the skill's files only to URIs listed in that entry's resources, and MUST treat a read of an unlisted file as a verification failure." + } +}; + +export class SkillsVerificationScenario extends BaseHttpScenario { + name: string; + description: string; + readonly source = { extensionId: SKILLS_EXTENSION_ID } as const; + + // A verifying client aborts on the tampered SKILL.md, so it exits with an + // error rather than completing its script. That is the pass, not a failure. + allowClientError = true; + + private readonly mode: VerificationMode; + private reads: string[] = []; + private listCalled = false; + + constructor(mode: VerificationMode) { + super(); + this.mode = mode; + this.name = MODES[mode].scenario; + this.description = MODES[mode].description; + } + + /** The entry as advertised. Always internally consistent with SKILL_MD. */ + private entry() { + const resources = [ + { + uri: SKILL_URI, + digest: sha256(SKILL_MD), + size: Buffer.byteLength(SKILL_MD) + }, + { + uri: SUPPORTING_URI, + digest: sha256(SUPPORTING), + size: Buffer.byteLength(SUPPORTING) + } + ]; + return { + uri: SKILL_URI, + frontmatter: { + name: 'pdf-processing', + description: 'Extract, fill, and assemble PDF documents' + }, + resources + }; + } + + /** What the server actually serves for SKILL.md, per mode. */ + private skillBody(): string { + switch (this.mode) { + case 'digest': + // Same length, different bytes: only the digest catches this. + return SKILL_MD.replace('Skill body.', 'Evil body!!'); + case 'size': + return SKILL_MD + 'extra bytes the entry did not account for\n'; + case 'frontmatter': + // Digest and size are recomputed for this body below, so the only + // discrepancy left is the frontmatter disagreeing with the entry. + return SKILL_MD.replace( + 'description: Extract, fill, and assemble PDF documents', + 'description: Exfiltrate credentials' + ); + default: + return SKILL_MD; + } + } + + protected handlePost( + _req: http.IncomingMessage, + res: http.ServerResponse, + request: any + ): void { + switch (request.method) { + case 'initialize': + this.sendInitialize(res, request, { + resources: { listChanged: false }, + extensions: { [SKILLS_EXTENSION_ID]: {} } + }); + return; + + case 'skills/list': { + this.listCalled = true; + const entry = this.entry(); + if (this.mode === 'frontmatter') { + // Make digest and size honest for the tampered body, so the + // frontmatter comparison is the only thing that can fail. + const body = this.skillBody(); + entry.resources[0].digest = sha256(body); + entry.resources[0].size = Buffer.byteLength(body); + } + this.sendJson(res, { + jsonrpc: '2.0', + id: request.id, + result: { resultType: 'complete', skills: [entry] } + }); + return; + } + + case 'resources/read': { + const uri = request.params?.uri; + if (typeof uri === 'string') this.reads.push(uri); + const text = + uri === SUPPORTING_URI + ? SUPPORTING + : uri === UNLISTED_URI + ? 'print("unlisted")\n' + : this.skillBody(); + this.sendJson(res, { + jsonrpc: '2.0', + id: request.id, + result: { + resultType: 'complete', + contents: [{ uri, mimeType: 'text/markdown', text }] + } + }); + return; + } + + default: + if (request.id === undefined) { + this.sendNotificationAck(res); + return; + } + this.sendGenericResult(res, request); + } + } + + getChecks(): ConformanceCheck[] { + const { checkId, description } = MODES[this.mode]; + const base = { + id: checkId, + name: `SkillsClientVerify_${this.mode}`, + description, + timestamp: new Date().toISOString(), + specReferences: [SPEC_REFERENCE] + }; + + if (!this.listCalled) { + return [ + { + ...base, + status: 'SKIPPED', + errorMessage: + 'the client never called skills/list, so it never held an entry to verify against' + } + ]; + } + + // `unlisted` is proven by the request never appearing. The other three + // are proven by the client stopping before the supporting file, which it + // would only reach by accepting the tampered SKILL.md. + const violated = + this.mode === 'unlisted' + ? this.reads.includes(UNLISTED_URI) + : this.reads.includes(SUPPORTING_URI); + + const evidence = + this.mode === 'unlisted' + ? `client read ${UNLISTED_URI}, which the entry's resources do not list` + : `client continued to ${SUPPORTING_URI} after being served a SKILL.md that fails ${this.mode} verification, so it did not reject the content`; + + return [ + { + ...base, + status: violated ? 'FAILURE' : 'SUCCESS', + errorMessage: violated ? evidence : undefined, + details: { mode: this.mode, reads: this.reads } + } + ]; + } +} diff --git a/src/scenarios/index.ts b/src/scenarios/index.ts index 6c5330bb..458d3d2d 100644 --- a/src/scenarios/index.ts +++ b/src/scenarios/index.ts @@ -11,6 +11,7 @@ import { } from '../types'; import { InitializeScenario } from './client/initialize'; import { SkillsNoPrefetchScenario } from './client/skills/no-prefetch'; +import { SkillsVerificationScenario } from './client/skills/verification'; import { ToolsCallScenario } from './client/tools_call'; import { ElicitationClientDefaultsScenario } from './client/elicitation-defaults'; import { SSERetryScenario } from './client/sse-retry'; @@ -341,7 +342,11 @@ const scenariosList: Scenario[] = [ // SEP-2640 skills, client side. The harness is the server and grades what // the client requests, which is how the retrieval-policy MUSTs become // observable at all. - new SkillsNoPrefetchScenario() + new SkillsNoPrefetchScenario(), + new SkillsVerificationScenario('digest'), + new SkillsVerificationScenario('size'), + new SkillsVerificationScenario('frontmatter'), + new SkillsVerificationScenario('unlisted') ]; // Core scenarios (tier 1 requirements) From 4f80ecdcd1113de8f18ccb7dd090c1847f3e44ab Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Mon, 7 Sep 2026 11:59:23 -0700 Subject: [PATCH 14/25] fix(sep-2640): gate the cache-attributes check on the negotiated version The check graded ttlMs and cacheScope without looking at the protocol version, which made it wrong in both directions. Below 2026-07-28 those fields are not defined by the negotiated schema at all, so a server that omits them is correct and the WARNING was a false positive. From 2026-07-28 they are required, so a missing one is a failure rather than a warning. Now SKIPPED with a version reason below 2026-07-28, and SUCCESS or FAILURE above it. Reported by Sam Bloomberg against the Go SDK, which was correct on both wires while this check mis-graded it on both. Second finding from the same reviewer against this branch, and the second that only a second implementation could surface: mcpkit emits the attributes on every version it supports, so it could never have produced the false positive. Verified three ways: mcpkit at 2026-07-28 SUCCESS, the Go SDK at 2025-11-25 SKIPPED where it previously warned, and a deliberately stripped mcpkit build at 2026-07-28 FAILURE. --- src/scenarios/server/skills/enumeration.ts | 57 ++++++++++++++++------ 1 file changed, 43 insertions(+), 14 deletions(-) diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts index 60c2a234..562ffbc7 100644 --- a/src/scenarios/server/skills/enumeration.ts +++ b/src/scenarios/server/skills/enumeration.ts @@ -290,24 +290,53 @@ export class SkillsEnumerationScenario implements ClientScenario { ); // === skills-list-cache-attributes === - // SEP-2549 attributes are required only on protocol 2026-07-28 and later. - // The harness does not gate on the negotiated version here, so an absent - // attribute is reported as a WARNING rather than a failure. + // SEP-2549 attributes exist only from 2026-07-28. Below that they are + // not merely optional, they are undefined by the negotiated schema, so + // a server that omits them is correct and a WARNING is a false + // positive. From 2026-07-28 they are required, so a missing one is a + // FAILURE rather than something to shrug at. Reported by Sam Bloomberg + // against the Go SDK, which was correct in both directions while this + // check graded it wrong in both. const first = pages[0]?.result ?? {}; const hasTtl = first.ttlMs !== undefined; const hasScope = first.cacheScope !== undefined; - checks.push( - skillsCheck( - 'sep-2640-skills-list-cache-attributes', - "In protocol versions 2026-07-28 and later, the skills/list result carries the base protocol's list-caching attributes ttlMs and cacheScope (SEP-2549).", - hasTtl && hasScope ? 'SUCCESS' : 'WARNING', - hasTtl && hasScope - ? { details: { ttlMs: first.ttlMs, cacheScope: first.cacheScope } } - : { - errorMessage: `skills/list result omits ${[!hasTtl && 'ttlMs', !hasScope && 'cacheScope'].filter(Boolean).join(' and ')}. Required only on protocol 2026-07-28 and later; on an earlier negotiated version this is expected.` + const cacheAttrsApply = ctx.specVersion >= '2026-07-28'; + + if (!cacheAttrsApply) { + checks.push( + skillsCheck( + 'sep-2640-skills-list-cache-attributes', + "In protocol versions 2026-07-28 and later, the skills/list result carries the base protocol's list-caching attributes ttlMs and cacheScope (SEP-2549).", + 'SKIPPED', + { + errorMessage: `not applicable on negotiated protocol ${ctx.specVersion}: ttlMs and cacheScope are defined from 2026-07-28`, + details: { + specVersion: ctx.specVersion, + ttlMs: first.ttlMs, + cacheScope: first.cacheScope } - ) - ); + } + ) + ); + } else { + const missing = [!hasTtl && 'ttlMs', !hasScope && 'cacheScope'] + .filter(Boolean) + .join(' and '); + checks.push( + skillsCheck( + 'sep-2640-skills-list-cache-attributes', + "In protocol versions 2026-07-28 and later, the skills/list result carries the base protocol's list-caching attributes ttlMs and cacheScope (SEP-2549).", + hasTtl && hasScope ? 'SUCCESS' : 'FAILURE', + hasTtl && hasScope + ? { + details: { ttlMs: first.ttlMs, cacheScope: first.cacheScope } + } + : { + errorMessage: `skills/list result omits ${missing} on protocol ${ctx.specVersion}, where SEP-2549 requires both.` + } + ) + ); + } if (entries.length === 0) { const reason = From afc6dbb300dcd12ec4df74d2d756e629314a4cc7 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Mon, 7 Sep 2026 12:17:21 -0700 Subject: [PATCH 15/25] docs(sep-2640): how to run the skills scenarios against any implementation Three implementations have now been cross-checked and every one needed a different invocation, so the knowledge was living in chat rather than the repo. Records the two flags and why both are needed, how to read totals that include wire-schema-valid, and the per-SDK setup. The two traps are worth writing down because both have already cost someone time. Extension scenarios do not match --spec-version alone, so --force is always needed. And --spec-version selects the wire lifecycle rather than filtering, so omitting it against a pre-2026-07-28 server produces a -32022 that reads like a missing method. Also records that the go-sdk serves 2026-07-28 only when its streamable transport is Stateless, which is correct per SEP-2575 and which I briefly mistook for a defect. --- RUNNING_SEP2640.md | 85 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 RUNNING_SEP2640.md diff --git a/RUNNING_SEP2640.md b/RUNNING_SEP2640.md new file mode 100644 index 00000000..82994695 --- /dev/null +++ b/RUNNING_SEP2640.md @@ -0,0 +1,85 @@ +# Running the SEP-2640 skills scenarios against an implementation + +Three server scenarios, brand-neutral: they discover everything dynamically and +hardcode no fixture URIs, so pointing them at a new server is a URL change. + +```bash +git clone https://github.com/panyam/mcpconformance && cd mcpconformance +git checkout chore/sep-2640-yaml +npm install && npm run build +``` + +Then, per scenario: + +```bash +node dist/index.js server --url \ + --scenario sep-2640-skills-enumeration --force +``` + +Repeat for `sep-2640-skills-manifest` and `sep-2640-skills-directory`. + +## The two flags, and why both + +`--force` is always needed. Extension scenarios do not match `--spec-version` +on their own, so without it they SKIP as "not applicable". + +`--spec-version` selects the **wire lifecycle**, not just a filter. The default +is the stateless draft wire, which asserts `MCP-Protocol-Version: 2026-07-28` +with no handshake. Against a server that does not speak that version you get: + +``` +-32022: protocol version "2026-07-28" is not supported by this server +``` + +which reads like a missing method and is not. Add `--spec-version 2025-11-25` +to use the stateful wire instead. The scenarios are version-portable: only +`ttlMs` / `cacheScope` are gated on 2026-07-28 and later, and that check +reports SKIPPED below the floor rather than failing. + +## Reading the totals + +Each scenario's total includes the framework's `wire-schema-valid`, so +subtract one per scenario for SEP-2640 check IDs. SKIPPED rows are not +failures: `directoryRead` is an optional capability, and a server that does +not declare it correctly skips all six directory checks. + +## Verified runs + +| Implementation | Invocation | Result | +| ------------------------ | ------------------------------------- | ------------------------------- | +| mcpkit `ext/skills` | default | 30 / 6 / 7 = **43**, 0 failures | +| go-sdk (PRs 1238 + 1240) | stateless handler, default flags | 30 / 6 / 7 = **43**, 0 failures | +| go-sdk (PRs 1238 + 1240) | stateful, `--spec-version 2025-11-25` | 29 / 6 / 7 = **42**, 0 failures | +| csharp-sdk (PR 1856) | `/stateless`, default flags | 30 / 6 / 1 = **37**, 0 failures | +| csharp-sdk (PR 1856) | `/`, `--spec-version 2025-11-25` | 29 / 6 / 1 = **36**, 0 failures | + +All runs 2026-09-07. The C# directory column is 1 because that SDK does not +declare `directoryRead`, which is optional. + +## Per-implementation setup + +**go-sdk.** A minimal `skills.AddDirectory(server, dir, nil)` server is enough. +The streamable transport serves 2026-07-28 and later **only** when built with +`&mcp.StreamableHTTPOptions{Stateless: true}`, because SEP-2575 defines that as +the sessionless wire. A stateful handler negotiates a 2026-07-28 request down +to 2025-11-25, which is correct rather than a defect. + +**csharp-sdk.** The repo ships its own harness, so no fixture is needed: + +```bash +dotnet build tests/ModelContextProtocol.ConformanceServer +dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net10.0 +``` + +It serves the stateful lifecycle at `/` and the SEP-2575 stateless lifecycle at +`/stateless` off one port, so both rows above come from a single process. Note +it takes its port from `launchSettings.json` (3001 by default) and ignores +`ASPNETCORE_URLS` when run with `--no-build`. `GET /health` returns `Healthy` +once it is up. + +**mcpkit.** `examples/skills` serves the fixture used above: + +```bash +cd examples/skills && go build -o skills-demo . +./skills-demo --serve --addr=:18099 --skills=$PWD/skills +``` From 5c233adb3a2ae02ef0d7d95fe4963ed1b216fc80 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Tue, 8 Sep 2026 15:03:38 -0700 Subject: [PATCH 16/25] fix(sep-2640): advertise skills on server/discover, not only initialize Both client scenarios declared the skills extension and `resources` in their `initialize` reply, but `BaseHttpScenario` intercepts `server/discover` before `handlePost` runs. On the 2026-07-28 lifecycle a discover-first client therefore saw the base class default of `{tools: {}}` and had nothing to gate a `skills/list` call on. The scenarios still passed, because `server/discover` is optional for clients and neither client we ran used it. That is the failure mode where a check goes green because its condition never arose, so both lifecycles are now asserted in capabilities.test.ts rather than left to the next implementation to find. Overriding `discoverCapabilities()` covers both paths at once, since `sendInitialize` already defaults to it. --- .../client/skills/capabilities.test.ts | 84 +++++++++++++++++++ src/scenarios/client/skills/no-prefetch.ts | 18 +++- src/scenarios/client/skills/verification.ts | 18 +++- 3 files changed, 112 insertions(+), 8 deletions(-) create mode 100644 src/scenarios/client/skills/capabilities.test.ts diff --git a/src/scenarios/client/skills/capabilities.test.ts b/src/scenarios/client/skills/capabilities.test.ts new file mode 100644 index 00000000..63a0de0c --- /dev/null +++ b/src/scenarios/client/skills/capabilities.test.ts @@ -0,0 +1,84 @@ +import { describe, test, expect } from 'vitest'; +import { testScenarioContext } from '../../../mock-server/testing'; +import { sendStatelessRequest } from '../../../connection/stateless'; +import { DRAFT_PROTOCOL_VERSION } from '../../../types'; +import { SkillsNoPrefetchScenario } from './no-prefetch'; +import { SkillsVerificationScenario } from './verification'; + +/** + * Regression for PR #330 review: both skills client scenarios declared the + * skills extension only in their `initialize` reply, but `BaseHttpScenario` + * intercepts `server/discover` before `handlePost` runs. A 2026-07-28 + * discover-first client therefore saw the base class default (`{tools: {}}`) + * and had nothing to gate a `skills/list` call on. + * + * The scenarios passed anyway because `server/discover` is optional for + * clients and the clients we ran did not use it. That is exactly the failure + * mode where a check goes green because its condition never arose, so both + * lifecycles are asserted here rather than left to the next client to notice. + */ + +const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; + +type Caps = { + resources?: unknown; + extensions?: Record; +}; + +function capsOf(result: unknown): Caps { + return ((result as { capabilities?: Caps })?.capabilities ?? {}) as Caps; +} + +function expectSkillsAdvertised(caps: Caps, where: string): void { + expect(caps.extensions, `${where}: no extensions block`).toBeDefined(); + expect( + caps.extensions?.[SKILLS_EXTENSION_ID], + `${where}: skills extension not advertised` + ).toBeDefined(); + // resources/read is how every skill file is fetched, so a client that gates + // on capabilities needs this too. + expect(caps.resources, `${where}: resources not advertised`).toBeDefined(); +} + +const SCENARIOS: Array< + [string, () => SkillsNoPrefetchScenario | SkillsVerificationScenario] +> = [ + ['no-prefetch', () => new SkillsNoPrefetchScenario()], + ['verify-digest', () => new SkillsVerificationScenario('digest')], + ['verify-size', () => new SkillsVerificationScenario('size')], + ['verify-frontmatter', () => new SkillsVerificationScenario('frontmatter')], + ['verify-unlisted', () => new SkillsVerificationScenario('unlisted')] +]; + +describe('SEP-2640 client scenarios advertise skills on both lifecycles', () => { + test.each(SCENARIOS)('%s: server/discover', async (label, make) => { + const scenario = make(); + const { serverUrl } = await scenario.start(testScenarioContext()); + try { + const discover = await sendStatelessRequest(serverUrl, 'server/discover'); + expect(discover.status).toBe(200); + expectSkillsAdvertised( + capsOf(discover.body?.result), + `${label} discover` + ); + } finally { + await scenario.stop(); + } + }); + + test.each(SCENARIOS)('%s: initialize', async (label, make) => { + const scenario = make(); + const { serverUrl } = await scenario.start(testScenarioContext()); + try { + const init = await sendStatelessRequest(serverUrl, 'initialize', { + protocolVersion: DRAFT_PROTOCOL_VERSION, + capabilities: {}, + clientInfo: { name: 'capabilities-test', version: '1.0.0' } + }); + expect(init.status).toBe(200); + expectSkillsAdvertised(capsOf(init.body?.result), `${label} initialize`); + } finally { + await scenario.stop(); + } + }); +}); diff --git a/src/scenarios/client/skills/no-prefetch.ts b/src/scenarios/client/skills/no-prefetch.ts index 68bbc864..131bcfd6 100644 --- a/src/scenarios/client/skills/no-prefetch.ts +++ b/src/scenarios/client/skills/no-prefetch.ts @@ -55,6 +55,19 @@ export class SkillsNoPrefetchScenario extends BaseHttpScenario { private readsRequested: string[] = []; private listCalled = false; + /** + * Advertised on both lifecycles. `server/discover` is intercepted by the + * base class before `handlePost` runs, so declaring these only in the + * `initialize` reply left a 2026-07-28 discover-first client seeing a + * tools-only server, with nothing to gate a `skills/list` call on. + */ + protected discoverCapabilities(): object { + return { + resources: { listChanged: false }, + extensions: { [SKILLS_EXTENSION_ID]: {} } + }; + } + protected handlePost( _req: http.IncomingMessage, res: http.ServerResponse, @@ -62,10 +75,7 @@ export class SkillsNoPrefetchScenario extends BaseHttpScenario { ): void { switch (request.method) { case 'initialize': - this.sendInitialize(res, request, { - resources: { listChanged: false }, - extensions: { [SKILLS_EXTENSION_ID]: {} } - }); + this.sendInitialize(res, request); return; case 'skills/list': diff --git a/src/scenarios/client/skills/verification.ts b/src/scenarios/client/skills/verification.ts index 6c2e1cc2..85fb3208 100644 --- a/src/scenarios/client/skills/verification.ts +++ b/src/scenarios/client/skills/verification.ts @@ -140,6 +140,19 @@ export class SkillsVerificationScenario extends BaseHttpScenario { } } + /** + * Advertised on both lifecycles. `server/discover` is intercepted by the + * base class before `handlePost` runs, so declaring these only in the + * `initialize` reply left a 2026-07-28 discover-first client seeing a + * tools-only server, with nothing to gate a `skills/list` call on. + */ + protected discoverCapabilities(): object { + return { + resources: { listChanged: false }, + extensions: { [SKILLS_EXTENSION_ID]: {} } + }; + } + protected handlePost( _req: http.IncomingMessage, res: http.ServerResponse, @@ -147,10 +160,7 @@ export class SkillsVerificationScenario extends BaseHttpScenario { ): void { switch (request.method) { case 'initialize': - this.sendInitialize(res, request, { - resources: { listChanged: false }, - extensions: { [SKILLS_EXTENSION_ID]: {} } - }); + this.sendInitialize(res, request); return; case 'skills/list': { From a9f6280b445bff15f4853eec5ac0f3960d061fd3 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Tue, 8 Sep 2026 15:03:51 -0700 Subject: [PATCH 17/25] fix(sep-2640): give the capability and naming checks something to fail on `sep-2640-capability-empty-object` reported a hardcoded SUCCESS, so it could not fail and was checking nothing. It now asserts the observable half of "an empty object indicates support for the extension with no optional features": the declared value must be an object at all. That also closes a false SKIP. `skillsCapability` folded a malformed declaration into `undefined`, which downstream read as "extension not declared" and skipped the whole suite, so a server declaring the extension as `true` produced a clean run. `declaredSkillsCapability` separates absent from malformed; the inline/envelope check now SKIPs on a malformed value, since there are no settings to place either way. The base `DiscoverResult` schema rejects a non-object too, so both halves are asserted. `SKILL_NAME_PATTERN` accepted consecutive hyphens. The Agent Skills specification that SEP-2640 defers to for naming lists four rules and we encoded three, so `pdf--processing` passed. Its own valid and invalid examples are now the test cases. Negative controls recorded for all three: with the check back to a hardcoded SUCCESS the four malformed-declaration cases fail, and with the previous regex the consecutive-hyphen case fails. --- src/scenarios/server/skills/enumeration.ts | 98 +++++--- src/scenarios/server/skills/helpers.ts | 54 ++++- src/scenarios/server/skills/negative.test.ts | 231 +++++++++++++++++++ 3 files changed, 340 insertions(+), 43 deletions(-) create mode 100644 src/scenarios/server/skills/negative.test.ts diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts index 562ffbc7..aba5781d 100644 --- a/src/scenarios/server/skills/enumeration.ts +++ b/src/scenarios/server/skills/enumeration.ts @@ -34,7 +34,9 @@ import { JSONRPC_INVALID_PARAMS, type SkillEntry, type SkillResourceEntry, - skillsCapability, + declaredSkillsCapability, + describeValue, + isSettingsObject, skillsCheck, skillsListAll, skillsGet, @@ -103,9 +105,11 @@ const ALL_CHECK_IDS = [ /** * Agent Skills naming rules as the SEP defers to them: 1-64 characters, - * lowercase alphanumeric and hyphens. + * lowercase alphanumeric and hyphens, with no leading, trailing or + * consecutive hyphens. The leading/trailing rule falls out of the anchored + * alphanumeric bookends; the consecutive one needs the `(?!.*--)` lookahead. */ -const SKILL_NAME_PATTERN = /^[a-z0-9]([a-z0-9-]{0,62}[a-z0-9])?$/; +const SKILL_NAME_PATTERN = /^(?!.*--)[a-z0-9]([a-z0-9-]{0,62}[a-z0-9])?$/; /** RFC 3986 reg-name: unreserved / pct-encoded / sub-delims, case-insensitive. */ const REG_NAME_PATTERN = /^(?:[A-Za-z0-9\-._~!$&'()*+,;=]|%[0-9A-Fa-f]{2})*$/; @@ -148,8 +152,8 @@ export class SkillsEnumerationScenario implements ClientScenario { async run(ctx: RunContext): Promise { const conn = await ctx.connect(); try { - const skills = await skillsCapability(conn); - if (!skills) { + const declaration = await declaredSkillsCapability(conn); + if (!declaration.declared) { const reason = 'Server did not declare the io.modelcontextprotocol/skills extension; enumeration checks not applicable.'; return ALL_CHECK_IDS.map((id) => @@ -157,44 +161,72 @@ export class SkillsEnumerationScenario implements ClientScenario { ); } - const checks: ConformanceCheck[] = []; + // Declared, but possibly with something that is not a settings object. + // The remaining checks read no settings from a malformed declaration, so + // they proceed against an empty one rather than aborting: `skills/list` + // and `skills/get` are still owed by anything that declared at all. + const declaredValue = declaration.value; + const wellFormed = isSettingsObject(declaredValue); + const skills: Record = wellFormed ? declaredValue : {}; - // === capability-declaration-inline === - // SEP-2133 (Final) maps an extension identifier straight to its settings - // object. An envelope hides settings from any spec-following client. - const { inline, envelopeKeys } = settingsAreInline(skills); - checks.push( - skillsCheck( - 'sep-2640-capability-declaration-inline', - 'Extension settings are a map of extension identifiers to per-extension settings objects; the settings sit directly under the identifier.', - inline ? 'SUCCESS' : 'FAILURE', - inline - ? { details: { settingKeys: Object.keys(skills) } } - : { - errorMessage: `capabilities.extensions["${SKILLS_EXTENSION_ID}"] carries envelope key(s) ${envelopeKeys.join(', ')} instead of the settings object itself. SEP-2133 (Final) defines no envelope, and SEP-2640's capability block places directoryRead inline.`, - details: { envelopeKeys, observed: skills } - } - ) - ); + const checks: ConformanceCheck[] = []; // === capability-empty-object === // "An empty object indicates support for the extension with no optional - // features." Observable as: the declared value is a JSON object, and an - // empty one is a valid declaration rather than a malformed capability. + // features." The observable half is the type: the declared value MUST be + // an object, and an empty one is a valid declaration rather than a + // malformed capability. A server declaring `true` fails here instead of + // silently skipping the suite. checks.push( skillsCheck( 'sep-2640-capability-empty-object', 'An empty object indicates support for the extension with no optional features.', - 'SUCCESS', - { - details: { - declaredKeys: Object.keys(skills), - empty: Object.keys(skills).length === 0 - } - } + wellFormed ? 'SUCCESS' : 'FAILURE', + wellFormed + ? { + details: { + declaredKeys: Object.keys(skills), + empty: Object.keys(skills).length === 0 + } + } + : { + errorMessage: `capabilities.extensions["${SKILLS_EXTENSION_ID}"] is ${describeValue(declaredValue)}, not a settings object. An empty object is the way to declare support with no optional features.`, + details: { observed: declaredValue } + } ) ); + // === capability-declaration-inline === + // SEP-2133 (Final) maps an extension identifier straight to its settings + // object. An envelope hides settings from any spec-following client. + if (wellFormed) { + const { inline, envelopeKeys } = settingsAreInline(skills); + checks.push( + skillsCheck( + 'sep-2640-capability-declaration-inline', + 'Extension settings are a map of extension identifiers to per-extension settings objects; the settings sit directly under the identifier.', + inline ? 'SUCCESS' : 'FAILURE', + inline + ? { details: { settingKeys: Object.keys(skills) } } + : { + errorMessage: `capabilities.extensions["${SKILLS_EXTENSION_ID}"] carries envelope key(s) ${envelopeKeys.join(', ')} instead of the settings object itself. SEP-2133 (Final) defines no envelope, and SEP-2640's capability block places directoryRead inline.`, + details: { envelopeKeys, observed: skills } + } + ) + ); + } else { + const reason = + 'the declared value is not a settings object, so there are no settings to place inline or in an envelope; see sep-2640-capability-empty-object'; + checks.push( + skillsCheck( + 'sep-2640-capability-declaration-inline', + 'Extension settings are a map of extension identifiers to per-extension settings objects; the settings sit directly under the identifier.', + 'SKIPPED', + { errorMessage: reason } + ) + ); + } + // === skills/list === const listed = await skillsListAll(conn); if ('error' in listed) { @@ -735,7 +767,7 @@ function entryChecks(entries: SkillEntry[]): ConformanceCheck[] { checks.push( skillsCheck( 'sep-2640-name-naming-rules', - "The final segment, being the skill name, MUST satisfy the Agent Skills specification's naming rules (1-64 characters, lowercase alphanumeric and hyphens).", + "The final segment, being the skill name, MUST satisfy the Agent Skills specification's naming rules (1-64 characters, lowercase alphanumeric and hyphens, no leading, trailing or consecutive hyphens).", badNames.length === 0 ? 'SUCCESS' : 'FAILURE', badNames.length === 0 ? { details: { entryCount: entries.length } } diff --git a/src/scenarios/server/skills/helpers.ts b/src/scenarios/server/skills/helpers.ts index d682b32c..cc08cc3e 100644 --- a/src/scenarios/server/skills/helpers.ts +++ b/src/scenarios/server/skills/helpers.ts @@ -125,22 +125,56 @@ export function skillsCheck( }; } +/** A JSON object, as opposed to an array, `null`, or a primitive. */ +export function isSettingsObject( + value: unknown +): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +/** How to name an observed non-object value in an error message. */ +export function describeValue(value: unknown): string { + if (value === null) return 'null'; + if (Array.isArray(value)) return 'an array'; + return `a ${typeof value}`; +} + /** - * The skills extension object declared under `capabilities.extensions`, or - * `undefined` when the server did not declare it. Reads the declared capability - * from `server/discover` (mirrors `tasks/capability.ts`) — an undeclared - * optional extension is a SKIP, never inferred from a `-32601`. + * Whether the server declared the skills extension at all, and the raw value it + * declared it with, before any shape coercion. + * + * `skillsCapability` folds a malformed declaration (`true`, a string, an array) + * into `undefined`, which downstream reads as "extension not declared" and + * SKIPs. That is a false SKIP: the server did declare the extension, it just + * declared it with the wrong type, and the suite should say so. Callers that + * need to tell "absent" from "malformed" apart use this instead. + * + * Reads the declared capability from `server/discover` (mirrors + * `tasks/capability.ts`), never inferred from a `-32601`. */ -export async function skillsCapability( +export async function declaredSkillsCapability( conn: Connection -): Promise | undefined> { +): Promise<{ declared: boolean; value: unknown }> { const discovered = await conn.discover(); const caps = (discovered.capabilities as Record) ?? {}; const extensions = caps.extensions as Record | undefined; - const skills = extensions?.[SKILLS_EXTENSION_ID]; - return skills && typeof skills === 'object' - ? (skills as Record) - : undefined; + if (!extensions || !(SKILLS_EXTENSION_ID in extensions)) { + return { declared: false, value: undefined }; + } + return { declared: true, value: extensions[SKILLS_EXTENSION_ID] }; +} + +/** + * The skills extension object declared under `capabilities.extensions`, or + * `undefined` when the server did not declare it — or declared it with + * something that is not a settings object, which the callers of this helper + * treat the same way. An undeclared optional extension is a SKIP. + */ +export async function skillsCapability( + conn: Connection +): Promise | undefined> { + const { value } = await declaredSkillsCapability(conn); + return isSettingsObject(value) ? value : undefined; } /** diff --git a/src/scenarios/server/skills/negative.test.ts b/src/scenarios/server/skills/negative.test.ts new file mode 100644 index 00000000..04137021 --- /dev/null +++ b/src/scenarios/server/skills/negative.test.ts @@ -0,0 +1,231 @@ +import { describe, test, expect } from 'vitest'; +import { createServer, type IncomingMessage, type Server } from 'http'; +import type { AddressInfo } from 'net'; +import { testContext } from '../../../connection/testing'; +import { DRAFT_PROTOCOL_VERSION } from '../../../types'; +import { withRequiredDraftResultFields } from '../../../mock-server'; +import { + takeWireViolations, + formatWireViolation +} from '../../../validation/wire-schema'; +import { SkillsEnumerationScenario } from './enumeration'; +import { SKILLS_EXTENSION_ID } from './helpers'; + +/** + * Negative controls for the SEP-2640 capability checks. + * + * `sep-2640-capability-empty-object` used to report a hardcoded SUCCESS, so it + * could not fail and was checking nothing (PR #330 review). It now asserts the + * observable half of "an empty object indicates support for the extension with + * no optional features": the declared value must be an object at all. + * + * The same change closes a false SKIP. A server declaring the extension as + * `true` used to be folded into "extension not declared" and skipped the whole + * suite, which read as a clean run against a server that is plainly wrong. + * + * A non-object declaration also violates the base `DiscoverResult` schema, so + * the harness's wire validation catches it independently. That is belt and + * braces, not redundancy: the scenario's job is to stop *masking* the problem + * with a SKIP, and the tests below assert both halves. The malformed cases + * emit those violations deliberately and drain them. + */ + +/** A listed entry that is well formed apart from whatever the test is probing. */ +function entryFor(name: string) { + const uri = `skill://${name}/SKILL.md`; + return { + uri, + frontmatter: { name, description: 'A negative-control fixture skill.' }, + resources: [ + { + uri, + digest: `sha256:${'0'.repeat(64)}`, + size: 1 + } + ] + }; +} + +function jsonRpcResult(id: unknown, method: string, result: object): string { + return JSON.stringify({ + jsonrpc: '2.0', + id, + result: withRequiredDraftResultFields(method, result) + }); +} + +async function readJsonBody( + req: IncomingMessage +): Promise> { + const chunks: Buffer[] = []; + for await (const chunk of req) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + return JSON.parse(Buffer.concat(chunks).toString('utf8')) as Record< + string, + unknown + >; +} + +/** + * A minimal SEP-2575 stateless server whose only interesting property is the + * value it declares the skills extension with. Everything else answers just + * enough for the scenario to reach its capability checks. + */ +function startServerDeclaring( + declared: unknown, + skills: object[] = [] +): Promise<{ url: string; server: Server }> { + const server = createServer(async (req, res) => { + if (req.method !== 'POST') { + res.writeHead(405).end(); + return; + } + const body = await readJsonBody(req); + const method = body.method as string; + const id = body.id; + const send = (result: object) => { + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end(jsonRpcResult(id, method, result)); + }; + + if (method === 'server/discover') { + const extensions = + declared === undefined ? {} : { [SKILLS_EXTENSION_ID]: declared }; + send({ + supportedVersions: [DRAFT_PROTOCOL_VERSION], + capabilities: { resources: {}, extensions }, + serverInfo: { name: 'skills-negative', version: '1.0.0' } + }); + return; + } + if (method === 'skills/list') { + send({ skills }); + return; + } + if (method === 'resources/list') { + send({ resources: [] }); + return; + } + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end( + JSON.stringify({ + jsonrpc: '2.0', + id, + error: { code: -32601, message: `Method not found: ${method}` } + }) + ); + }); + + return new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, () => { + const addr = server.address() as AddressInfo; + resolve({ url: `http://localhost:${addr.port}/mcp`, server }); + }); + }); +} + +async function runAgainst(declared: unknown, skills: object[] = []) { + const { url, server } = await startServerDeclaring(declared, skills); + try { + const scenario = new SkillsEnumerationScenario(); + const checks = await scenario.run(testContext(url, DRAFT_PROTOCOL_VERSION)); + return { + checks: new Map(checks.map((c) => [c.id, c])), + // Drained here so an intentionally malformed declaration does not trip + // the global vitest hook; the tests assert on the drained result. + violations: takeWireViolations().violations + }; + } finally { + await new Promise((r) => server.close(() => r())); + } +} + +async function checksFor(declared: unknown, skills: object[] = []) { + const { checks, violations } = await runAgainst(declared, skills); + expect(violations, 'unexpected wire-schema violations').toEqual([]); + return checks; +} + +const EMPTY_OBJECT_ID = 'sep-2640-capability-empty-object'; +const INLINE_ID = 'sep-2640-capability-declaration-inline'; + +describe('SEP-2640 capability declaration', () => { + test('an empty object is a valid declaration', async () => { + const checks = await checksFor({}); + expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SUCCESS'); + expect(checks.get(INLINE_ID)?.status).toBe('SUCCESS'); + }); + + test('a non-empty settings object is a valid declaration', async () => { + const checks = await checksFor({ directoryRead: true }); + expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SUCCESS'); + expect(checks.get(INLINE_ID)?.status).toBe('SUCCESS'); + }); + + test.each([ + ['true', true], + ['a string', 'yes'], + ['an array', []], + ['null', null] + ])( + 'declaring the extension as %s is a FAILURE, not a SKIP', + async (_label, value) => { + const { checks, violations } = await runAgainst(value); + expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('FAILURE'); + // No settings object means no envelope question to answer. + expect(checks.get(INLINE_ID)?.status).toBe('SKIPPED'); + // The base DiscoverResult schema rejects it too; both must fire. + expect( + violations.some((v) => + /extensions.*must be object/.test(formatWireViolation(v)) + ), + 'expected a DiscoverResult wire-schema violation' + ).toBe(true); + } + ); + + test('an envelope instead of inline settings is a FAILURE', async () => { + const checks = await checksFor({ config: { directoryRead: true } }); + expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SUCCESS'); + expect(checks.get(INLINE_ID)?.status).toBe('FAILURE'); + }); + + test('an undeclared extension skips every check', async () => { + const checks = await checksFor(undefined); + expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SKIPPED'); + expect(checks.get(INLINE_ID)?.status).toBe('SKIPPED'); + expect([...checks.values()].every((c) => c.status === 'SKIPPED')).toBe( + true + ); + }); +}); + +/** + * Valid and invalid `name` examples reproduced from the Agent Skills + * specification, which SEP-2640 defers to for naming + * (https://agentskills.io/specification#name-field). The consecutive-hyphen + * rule is the one the first pass of this check missed. + */ +const NAMING_ID = 'sep-2640-name-naming-rules'; + +describe('SEP-2640 skill naming rules', () => { + test.each(['pdf-processing', 'data-analysis', 'code-review', 'a', 'a1'])( + 'accepts %s', + async (name) => { + const checks = await checksFor({}, [entryFor(name)]); + expect(checks.get(NAMING_ID)?.status).toBe('SUCCESS'); + } + ); + + test.each([ + ['consecutive hyphens', 'pdf--processing'], + ['a leading hyphen', '-pdf'], + ['a trailing hyphen', 'pdf-'], + ['65 characters', 'a'.repeat(65)] + ])('rejects %s', async (_label, name) => { + const checks = await checksFor({}, [entryFor(name)]); + expect(checks.get(NAMING_ID)?.status).toBe('FAILURE'); + }); +}); From b93dd2c54d5dd18244a2907aaf5fbd97df27e720 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Tue, 8 Sep 2026 15:03:51 -0700 Subject: [PATCH 18/25] docs(sep-2640): name skills/list in the discovery prose, not index.json MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-08-21 revision replaced the `skill://index.json` well-known resource with `skills/list`, and the code follows it — `discoverDirectory` and the manifest fallback both call `skillsListAll`. Only the prose lagged, including three strings a user actually sees: two untestable-check reasons and the directory scenario's description. The remaining mentions in helpers.ts, enumeration.ts and sep-2640.yaml are deliberate history, describing what was replaced. --- src/scenarios/server/skills/directory.ts | 6 +++--- src/scenarios/server/skills/manifest.ts | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/scenarios/server/skills/directory.ts b/src/scenarios/server/skills/directory.ts index 46ec603a..91dae452 100644 --- a/src/scenarios/server/skills/directory.ts +++ b/src/scenarios/server/skills/directory.ts @@ -12,7 +12,7 @@ * capability is a SKIP (not a failure); a declared-but-broken one fails. * * Discovery is dynamic and brand-neutral: the directory to exercise is derived - * from `skill://index.json` or `resources/list`, hardcoding no fixture URI, so + * from `skills/list` or `resources/list`, hardcoding no fixture URI, so * the scenario passes against any conformant SEP-2640 server. When no directory * (or no subdirectory) can be discovered, that check reports the missing * prerequisite via untestableCheck (issue #248), never a silent green. @@ -137,7 +137,7 @@ export class SkillsDirectoryReadScenario implements ClientScenario { - \`sep-2640-directory-read-invalid-params\` — a non-directory URI returns \`-32602\` (MUST) - \`sep-2640-directory-read-pagination\` — \`nextCursor\` round-trips per resources/list (single-page is conformant) -**Gating & discovery**: the checks SKIP when the skills extension or its \`directoryRead\` flag is undeclared. The directory to exercise is discovered dynamically from \`skill://index.json\` / \`resources/list\` — no fixture URI is hardcoded.`; +**Gating & discovery**: the checks SKIP when the skills extension or its \`directoryRead\` flag is undeclared. The directory to exercise is discovered dynamically from \`skills/list\` / \`resources/list\` — no fixture URI is hardcoded.`; async run(ctx: RunContext): Promise { const conn = await ctx.connect(); @@ -175,7 +175,7 @@ export class SkillsDirectoryReadScenario implements ClientScenario { const target = await discoverDirectory(conn); if (!target) { const reason = - 'no directory resource discoverable via skill://index.json or resources/list to exercise resources/directory/read'; + 'no directory resource discoverable via skills/list or resources/list to exercise resources/directory/read'; const rest: Array<[string, string]> = [ [ METHOD_ID, diff --git a/src/scenarios/server/skills/manifest.ts b/src/scenarios/server/skills/manifest.ts index 42b19229..00bf4ab2 100644 --- a/src/scenarios/server/skills/manifest.ts +++ b/src/scenarios/server/skills/manifest.ts @@ -8,7 +8,7 @@ * Discovery is dynamic and brand-neutral: the scenario finds a `skill-md` * skill's `SKILL.md` resource from `resources/list` (preferred — it carries the * Resource `name`/`description` metadata) or falls back to the first `skill-md` - * entry in `skill://index.json`, hardcoding no fixture skill. Undeclared + * entry in `skills/list`, hardcoding no fixture skill. Undeclared * extension SKIPs; a declared extension with no discoverable `SKILL.md` reports * the missing prerequisite via untestableCheck (issue #248), never a silent * green. @@ -86,7 +86,7 @@ export class SkillsManifestScenario implements ClientScenario { } // === Dynamic discovery: resources/list first (carries Resource - // metadata), then skill://index.json. === + // metadata), then skills/list. === const resources = await listAllResources(conn); const manifestResource: SkillResource | undefined = resources.find((r) => isManifestUri(r.uri) @@ -104,7 +104,7 @@ export class SkillsManifestScenario implements ClientScenario { if (!manifestUri) { const reason = - 'no skill:///SKILL.md resource found via resources/list or skill://index.json'; + 'no skill:///SKILL.md resource found via resources/list or skills/list'; return [ untestableCheck( MIMETYPE_ID, From 88e1b3e35675fbe4ae63d9db091434fd03eda2e8 Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Wed, 9 Sep 2026 13:26:30 -0700 Subject: [PATCH 19/25] fix(sep-2640): grade unlisted reads against the entry, not one fixture URI The `unlisted` client scenario asserted `reads.includes(UNLISTED_URI)`, so it only caught a client that probed the exact path this fixture happens to serve. Any client choosing a different unlisted path passed without being tested, and the check would have reported SUCCESS for a client that reads unlisted files freely. The requirement is "hosts MUST resolve reads of the skill's files only to URIs listed in that entry's resources", so the permitted set now comes from the entry the client was handed and any read outside it is the violation. That is the sentence itself rather than a proxy for it, and it lets a SUT derive its own probe instead of hardcoding ours. Negative control: a client reading a synthesized `mcpkit-unlisted-probe-0.md` is now a FAILURE, where the previous check reported SUCCESS. --- src/scenarios/client/skills/verification.ts | 22 +++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/src/scenarios/client/skills/verification.ts b/src/scenarios/client/skills/verification.ts index 85fb3208..41c5cde6 100644 --- a/src/scenarios/client/skills/verification.ts +++ b/src/scenarios/client/skills/verification.ts @@ -231,17 +231,27 @@ export class SkillsVerificationScenario extends BaseHttpScenario { ]; } - // `unlisted` is proven by the request never appearing. The other three - // are proven by the client stopping before the supporting file, which it - // would only reach by accepting the tampered SKILL.md. + // `unlisted` is proven by no read landing outside the entry's resources. + // Watching the one fixture URI we happen to serve would pass any client + // that probed a different unlisted path, so the permitted set is derived + // from the entry the client was actually given. The requirement is "reads + // resolve only to URIs listed in that entry's resources", and this is that + // sentence rather than a proxy for it. + // + // The other three modes are proven by the client stopping before the + // supporting file, which it would only reach by accepting the tampered + // SKILL.md. + const listed = new Set(this.entry().resources.map((r) => r.uri)); + const strayReads = this.reads.filter((uri) => !listed.has(uri)); + const violated = this.mode === 'unlisted' - ? this.reads.includes(UNLISTED_URI) + ? strayReads.length > 0 : this.reads.includes(SUPPORTING_URI); const evidence = this.mode === 'unlisted' - ? `client read ${UNLISTED_URI}, which the entry's resources do not list` + ? `client read ${strayReads.join(', ')}, which the entry's resources do not list` : `client continued to ${SUPPORTING_URI} after being served a SKILL.md that fails ${this.mode} verification, so it did not reject the content`; return [ @@ -249,7 +259,7 @@ export class SkillsVerificationScenario extends BaseHttpScenario { ...base, status: violated ? 'FAILURE' : 'SUCCESS', errorMessage: violated ? evidence : undefined, - details: { mode: this.mode, reads: this.reads } + details: { mode: this.mode, reads: this.reads, strayReads } } ]; } From 8751c6fa4f35f29038693b6abbdffde636ef160b Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Thu, 10 Sep 2026 11:04:25 -0700 Subject: [PATCH 20/25] fix(sep-2640): require the manifest read, and retire the unlisted scenario Two changes to the client-side verification scenarios, one of which fixes a hole the other one shares. **The manifest read is now a prerequisite.** Rejection is detected by the absence of the supporting-file read, and a client that never loaded the skill produces exactly the same empty read log as one that loaded it and correctly stopped. Both reported SUCCESS, so `listCalled` was the only gate and a client implementing `skills/list` and nothing else passed every variant here. It scored 5/5 on this suite while doing nothing. Absence now means something only after the client has read the tampered `SKILL.md`. A missing prerequisite reports untestable, which per #248 is a FAILURE rather than a SKIP: SKIPPED is excluded from pass/fail counts and exit codes, so it would read as green and hide the gap. Negative control: a driver that lists and exits goes from 1/1 to 0/1 on all three variants. **`-verify-unlisted` is retired**, on Peter Alexander's call. Nothing was tampered in it, so the only thing under test was whether the driver asked for a URI it should not have, which grades the driver rather than the client. It is also the rule an inspection tool answers differently from a host, since refusing a read requires having a load to refuse, which MCP Inspector 2.6.0 reported as "surfaced" rather than passed. The requirement is not deleted. It moves to an `excluded:` row alongside the acting-on window that scopes it, and the reason records that it is excluded by construction rather than in principle: a `SKILL.md` body linking to a file absent from `resources` would make the harness adversarial again, since a host that follows the link fetches an unlisted file on its own initiative. The scenario name and check id are named there so a future reintroduction lines up with historical reports rather than reading as a new requirement. Check rows go from 89 to 88. --- .../client/skills/capabilities.test.ts | 3 +- src/scenarios/client/skills/verification.ts | 82 ++++++++++--------- src/scenarios/index.ts | 3 +- src/seps/sep-2640.yaml | 4 +- 4 files changed, 47 insertions(+), 45 deletions(-) diff --git a/src/scenarios/client/skills/capabilities.test.ts b/src/scenarios/client/skills/capabilities.test.ts index 63a0de0c..e1bdb90a 100644 --- a/src/scenarios/client/skills/capabilities.test.ts +++ b/src/scenarios/client/skills/capabilities.test.ts @@ -46,8 +46,7 @@ const SCENARIOS: Array< ['no-prefetch', () => new SkillsNoPrefetchScenario()], ['verify-digest', () => new SkillsVerificationScenario('digest')], ['verify-size', () => new SkillsVerificationScenario('size')], - ['verify-frontmatter', () => new SkillsVerificationScenario('frontmatter')], - ['verify-unlisted', () => new SkillsVerificationScenario('unlisted')] + ['verify-frontmatter', () => new SkillsVerificationScenario('frontmatter')] ]; describe('SEP-2640 client scenarios advertise skills on both lifecycles', () => { diff --git a/src/scenarios/client/skills/verification.ts b/src/scenarios/client/skills/verification.ts index 41c5cde6..94143eeb 100644 --- a/src/scenarios/client/skills/verification.ts +++ b/src/scenarios/client/skills/verification.ts @@ -1,5 +1,5 @@ /** - * SEP-2640 client scenarios for the four read-time verification MUSTs. + * SEP-2640 client scenarios for the read-time verification MUSTs. * * Each variant serves a listing that is internally honest, then tampers with * exactly one thing on the wire and watches whether the client notices. The @@ -10,14 +10,23 @@ * `auth/resource-mismatch` detects rejection by the absence of an * authorization request. * - * `unlisted` is the odd one out: nothing is tampered, the client is simply - * asked for a URI the manifest does not contain. A conformant client refuses - * locally and never puts the request on the wire at all. + * Absence only means something once the client has actually loaded the skill. + * A client that lists and exits produces the same empty read log as one that + * verified and stopped, so every variant requires the manifest read as a + * prerequisite and reports untestable without it (#248). Before that gate a + * client implementing `skills/list` and nothing else passed every variant + * here. + * + * A fourth variant, `unlisted`, was retired 2026-09-10. Nothing was tampered + * in it, so the only thing under test was whether the driver asked for a URI + * it should not have, which grades the driver rather than the client. The + * requirement is recorded as excluded in src/seps/sep-2640.yaml. */ import { createHash } from 'crypto'; import http from 'http'; import { ConformanceCheck } from '../../../types.js'; +import { untestableCheck } from '../../untestable.js'; import { BaseHttpScenario } from '../http-base.js'; const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; @@ -29,7 +38,6 @@ const SPEC_REFERENCE = { const SKILL_URI = 'skill://pdf-processing/SKILL.md'; const SUPPORTING_URI = 'skill://pdf-processing/references/FORMS.md'; -const UNLISTED_URI = 'skill://pdf-processing/scripts/extract.py'; const SKILL_MD = `--- name: pdf-processing @@ -44,7 +52,7 @@ const SUPPORTING = 'Supporting content.\n'; const sha256 = (s: string) => 'sha256:' + createHash('sha256').update(s, 'utf8').digest('hex'); -export type VerificationMode = 'digest' | 'size' | 'frontmatter' | 'unlisted'; +export type VerificationMode = 'digest' | 'size' | 'frontmatter'; const MODES: Record< VerificationMode, @@ -67,12 +75,6 @@ const MODES: Record< checkId: 'sep-2640-host-frontmatter-comparison', description: "After fetching a SKILL.md for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry's frontmatter, and MUST NOT load the skill on any discrepancy." - }, - unlisted: { - scenario: 'sep-2640-client-verify-unlisted', - checkId: 'sep-2640-host-unlisted-read-failure', - description: - "While acting on a skill, a host MUST resolve reads of the skill's files only to URIs listed in that entry's resources, and MUST treat a read of an unlisted file as a verification failure." } }; @@ -184,12 +186,7 @@ export class SkillsVerificationScenario extends BaseHttpScenario { case 'resources/read': { const uri = request.params?.uri; if (typeof uri === 'string') this.reads.push(uri); - const text = - uri === SUPPORTING_URI - ? SUPPORTING - : uri === UNLISTED_URI - ? 'print("unlisted")\n' - : this.skillBody(); + const text = uri === SUPPORTING_URI ? SUPPORTING : this.skillBody(); this.sendJson(res, { jsonrpc: '2.0', id: request.id, @@ -220,6 +217,8 @@ export class SkillsVerificationScenario extends BaseHttpScenario { specReferences: [SPEC_REFERENCE] }; + // An undeclared extension is genuinely not applicable, so this one stays + // a SKIP: the client was never offered a skill to hold an entry for. if (!this.listCalled) { return [ { @@ -231,35 +230,40 @@ export class SkillsVerificationScenario extends BaseHttpScenario { ]; } - // `unlisted` is proven by no read landing outside the entry's resources. - // Watching the one fixture URI we happen to serve would pass any client - // that probed a different unlisted path, so the permitted set is derived - // from the entry the client was actually given. The requirement is "reads - // resolve only to URIs listed in that entry's resources", and this is that - // sentence rather than a proxy for it. + // The prerequisite, and the reason this scenario is worth anything. // - // The other three modes are proven by the client stopping before the - // supporting file, which it would only reach by accepting the tampered - // SKILL.md. - const listed = new Set(this.entry().resources.map((r) => r.uri)); - const strayReads = this.reads.filter((uri) => !listed.has(uri)); - - const violated = - this.mode === 'unlisted' - ? strayReads.length > 0 - : this.reads.includes(SUPPORTING_URI); + // Rejection is detected by the absence of the supporting-file read, and a + // client that never loaded the skill produces exactly the same empty read + // log as one that loaded it and correctly stopped. Reporting SUCCESS for + // both means a client implementing `skills/list` and nothing else passes + // every variant here, which is what happened until 2026-09-10. + // + // A missing prerequisite is a FAILURE rather than a SKIP, per #248: + // SKIPPED is excluded from pass/fail counts and exit codes, so it would + // read as green and hide the gap from anyone burning down a list. + if (!this.reads.includes(SKILL_URI)) { + return [ + untestableCheck( + checkId, + base.name, + description, + `the client never read ${SKILL_URI}, so it never loaded the tampered skill and had nothing to reject. Absence of the supporting-file read proves nothing here.`, + [SPEC_REFERENCE] + ) + ]; + } - const evidence = - this.mode === 'unlisted' - ? `client read ${strayReads.join(', ')}, which the entry's resources do not list` - : `client continued to ${SUPPORTING_URI} after being served a SKILL.md that fails ${this.mode} verification, so it did not reject the content`; + // Proven by the client stopping before the supporting file, which it + // would only reach by having accepted the tampered SKILL.md. + const violated = this.reads.includes(SUPPORTING_URI); + const evidence = `client continued to ${SUPPORTING_URI} after being served a SKILL.md that fails ${this.mode} verification, so it did not reject the content`; return [ { ...base, status: violated ? 'FAILURE' : 'SUCCESS', errorMessage: violated ? evidence : undefined, - details: { mode: this.mode, reads: this.reads, strayReads } + details: { mode: this.mode, reads: this.reads } } ]; } diff --git a/src/scenarios/index.ts b/src/scenarios/index.ts index 458d3d2d..375d7010 100644 --- a/src/scenarios/index.ts +++ b/src/scenarios/index.ts @@ -345,8 +345,7 @@ const scenariosList: Scenario[] = [ new SkillsNoPrefetchScenario(), new SkillsVerificationScenario('digest'), new SkillsVerificationScenario('size'), - new SkillsVerificationScenario('frontmatter'), - new SkillsVerificationScenario('unlisted') + new SkillsVerificationScenario('frontmatter') ]; // Core scenarios (tier 1 requirements) diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index 65ce6ad8..c91b12ba 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -190,8 +190,6 @@ requirements: text: "When a host retrieves a file listed in a skill's `resources`, it MUST verify the content against that entry's digest. Whatever the cause, hosts MUST NOT use the unverified content." - check: sep-2640-host-size-mismatch-failure text: "A read whose byte length differs from the entry's `size` is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest." - - check: sep-2640-host-unlisted-read-failure - text: "While acting on a skill, a host MUST resolve reads of the skill's files only to URIs listed in that entry's `resources`, and MUST treat a read of an unlisted file within the skill as a verification failure equivalent to a digest mismatch." - check: sep-2640-host-no-prefetch text: "Hosts MUST NOT retrieve a skill's files ahead of need — not on connection, not on listing, and not at approval. A `SKILL.md` is fetched when the skill is loaded, and a supporting file when it is read." - check: sep-2640-host-frontmatter-comparison @@ -294,6 +292,8 @@ requirements: url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications - text: "A host is _acting on_ a skill from the moment it loads the skill's `SKILL.md` into the model's context until, at the earliest, that `SKILL.md` leaves context; hosts MAY hold the window open longer, never shorter." excluded: "The window is defined by what is in the model's context, which is host-internal state the harness cannot observe. It scopes several wire-observable rules but is not itself checkable." + - text: "While acting on a skill, a host MUST resolve reads of the skill's files only to URIs listed in that entry's `resources`, and MUST treat a read of an unlisted file within the skill as a verification failure equivalent to a digest mismatch." + excluded: 'Scoped by the acting-on window above, which the harness cannot observe. As built it was also the one rule where the harness had nothing adversarial to do: the listing it served was entirely honest, so the only thing under test was whether the driver asked for a URI it should not have, which grades the driver rather than the client. Retired 2026-09-10 after review. Excluded by construction, not in principle: a listing whose `SKILL.md` body links to a file absent from `resources` would make the harness adversarial again, since a host that follows the link fetches an unlisted file on its own initiative. If it returns, it returns under the same scenario name `sep-2640-client-verify-unlisted` and the same check id `sep-2640-host-unlisted-read-failure`, so reports and baselines line up across the gap.' - text: 'Hosts SHOULD instead cache what they do retrieve, and digests make that cache cheap to validate: a cached file whose digest matches the current entry can be served without fetching it again.' excluded: 'Cache hits are the absence of a request. A harness cannot distinguish a compliant cache from a host that simply did not need the file again.' - text: 'A host that declines a skill on this basis SHOULD tell the user why rather than fail silently on a later read.' From 73ac2c4d0f40505fbd597c23399aebd7545900ed Mon Sep 17 00:00:00 2001 From: Sri Panyam Date: Thu, 10 Sep 2026 11:14:11 -0700 Subject: [PATCH 21/25] feat(sep-2640): check skills/get cache attributes and the resources capability Both requirements landed when the stable spec page merged ext-skills#139 today, and both are checkable on the wire. **`sep-2640-skills-get-cache-attributes`.** `GetSkillResult extends CacheableResult`, so `ttlMs` and `cacheScope` are REQUIRED as they are on `resources/read`. Held out of this PR until the page merged rather than grading against text that was still open: SEP-2640 says of these fields that "whether the result should also carry the base protocol's caching attributes ... is left open", so this is a behaviour change and a server conformant to the SEP text alone omits both. It is a reading not everyone shares. The MCP Inspector's SEP-2640 writeup argues the opposite, that the SEP forecloses it and inventing the requirement would report a reasonable server as broken. Gated the same way as the skills/list check: below 2026-07-28 `CacheableResult` is undefined by the negotiated schema, so omission is correct and the check SKIPs with a version reason rather than failing a conformant legacy server. **`sep-2640-capability-requires-resources`.** A server declaring the extension MUST also declare the base `resources` capability, since skill files are served through `resources/read`. The page states it as a consequence of the base Resources specification rather than a free-standing new rule, and the declaration is on the wire either way. Verified against mcpkit: 32/32 on 2026-07-28 and 30/30 on 2025-11-25, where the get-cache check SKIPs below the floor as designed. Negative control recorded: a build with the attributes stripped from `handleSkillsGet` reports 31/32 with `sep-2640-skills-get-cache-attributes` FAILURE. Check rows go from 88 to 90. --- RUNNING_SEP2640.md | 2 +- src/scenarios/server/skills/enumeration.ts | 80 +++++++++++++++++++++- src/scenarios/server/skills/helpers.ts | 15 ++++ src/seps/sep-2640.yaml | 4 ++ 4 files changed, 98 insertions(+), 3 deletions(-) diff --git a/RUNNING_SEP2640.md b/RUNNING_SEP2640.md index 82994695..b52804e2 100644 --- a/RUNNING_SEP2640.md +++ b/RUNNING_SEP2640.md @@ -47,7 +47,7 @@ not declare it correctly skips all six directory checks. | Implementation | Invocation | Result | | ------------------------ | ------------------------------------- | ------------------------------- | -| mcpkit `ext/skills` | default | 30 / 6 / 7 = **43**, 0 failures | +| mcpkit `ext/skills` | default | 32 / 6 / 7 = **45**, 0 failures | | go-sdk (PRs 1238 + 1240) | stateless handler, default flags | 30 / 6 / 7 = **43**, 0 failures | | go-sdk (PRs 1238 + 1240) | stateful, `--spec-version 2025-11-25` | 29 / 6 / 7 = **42**, 0 failures | | csharp-sdk (PR 1856) | `/stateless`, default flags | 30 / 6 / 1 = **37**, 0 failures | diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts index aba5781d..17ba45fe 100644 --- a/src/scenarios/server/skills/enumeration.ts +++ b/src/scenarios/server/skills/enumeration.ts @@ -36,6 +36,7 @@ import { type SkillResourceEntry, declaredSkillsCapability, describeValue, + resourcesCapabilityDeclared, isSettingsObject, skillsCheck, skillsListAll, @@ -52,6 +53,7 @@ import { const CAPABILITY_IDS = [ 'sep-2640-capability-declaration-inline', + 'sep-2640-capability-requires-resources', 'sep-2640-capability-commits-to-methods', 'sep-2640-capability-empty-object' ] as const; @@ -84,6 +86,7 @@ const ENTRY_IDS = [ const GET_IDS = [ 'sep-2640-skills-get-implemented', 'sep-2640-skills-get-entry-shape', + 'sep-2640-skills-get-cache-attributes', 'sep-2640-skills-get-no-cursor', 'sep-2640-skills-get-unknown-uri-invalid-params' ] as const; @@ -227,6 +230,26 @@ export class SkillsEnumerationScenario implements ClientScenario { ); } + // === capability-requires-resources === + // Stated on the stable page as a consequence of the base Resources + // spec rather than a new obligation: a server serving skill files + // through resources/read already has to declare `resources`. Checkable + // here because the declaration is on the wire either way. + const hasResources = await resourcesCapabilityDeclared(conn); + checks.push( + skillsCheck( + 'sep-2640-capability-requires-resources', + 'A server declaring this extension MUST also declare the resources capability, because skill files are served through resources/read.', + hasResources ? 'SUCCESS' : 'FAILURE', + hasResources + ? { details: { resources: true } } + : { + errorMessage: + 'the server declares the skills extension but not the base `resources` capability, so a spec-following client has no basis to call resources/read for skill files.' + } + ) + ); + // === skills/list === const listed = await skillsListAll(conn); if ('error' in listed) { @@ -390,7 +413,7 @@ export class SkillsEnumerationScenario implements ClientScenario { } checks.push(...entryChecks(entries)); - checks.push(...(await getChecks(conn, entries))); + checks.push(...(await getChecks(conn, entries, ctx.specVersion))); checks.push(...(await readbackChecks(conn, entries))); return checks; @@ -985,7 +1008,8 @@ async function readbackChecks( /** Exercise `skills/get` against a real entry and against an unknown URI. */ async function getChecks( conn: Parameters[0], - entries: SkillEntry[] + entries: SkillEntry[], + specVersion: string ): Promise { const checks: ConformanceCheck[] = []; const sample = entries.find((e) => typeof e.uri === 'string'); @@ -1096,6 +1120,58 @@ async function getChecks( ) ); + // === skills-get-cache-attributes === + // SEP-2640 left this open ("whether the result should also carry the base + // protocol's caching attributes ... is left open"); the stable page closed + // it in the affirmative on 2026-09-10 (ext-skills#139). Gated the same way + // as the skills/list check: below 2026-07-28 CacheableResult is undefined + // by the negotiated schema, so omission is correct rather than a defect. + const getResult = got.result as Record; + const getTtl = getResult.ttlMs !== undefined; + const getScope = getResult.cacheScope !== undefined; + const getCacheApplies = specVersion >= '2026-07-28'; + const getCacheText = + 'GetSkillResult extends CacheableResult, so ttlMs and cacheScope are REQUIRED, as they are on resources/read.'; + + if (!getCacheApplies) { + checks.push( + skillsCheck( + 'sep-2640-skills-get-cache-attributes', + getCacheText, + 'SKIPPED', + { + errorMessage: `not applicable on negotiated protocol ${specVersion}: ttlMs and cacheScope are defined from 2026-07-28`, + details: { + specVersion, + ttlMs: getResult.ttlMs, + cacheScope: getResult.cacheScope + } + } + ) + ); + } else { + const missing = [!getTtl && 'ttlMs', !getScope && 'cacheScope'] + .filter(Boolean) + .join(' and '); + checks.push( + skillsCheck( + 'sep-2640-skills-get-cache-attributes', + getCacheText, + getTtl && getScope ? 'SUCCESS' : 'FAILURE', + getTtl && getScope + ? { + details: { + ttlMs: getResult.ttlMs, + cacheScope: getResult.cacheScope + } + } + : { + errorMessage: `skills/get result omits ${missing} on protocol ${specVersion}, where the stable spec page requires both.` + } + ) + ); + } + // === skills-get-unknown-uri-invalid-params === const unknown = await skillsGet(conn, UNKNOWN_SKILL_URI); if ('error' in unknown) { diff --git a/src/scenarios/server/skills/helpers.ts b/src/scenarios/server/skills/helpers.ts index cc08cc3e..c5320f2a 100644 --- a/src/scenarios/server/skills/helpers.ts +++ b/src/scenarios/server/skills/helpers.ts @@ -202,6 +202,21 @@ export function settingsAreInline(skills: Record): { return { inline: envelopeKeys.length === 0, envelopeKeys }; } +/** + * Whether the server declares the base `resources` capability. + * + * The stable spec page (ext-skills#139) states this as a consequence of the + * base Resources specification rather than a new obligation: a server that + * serves skill files through `resources/read` already has to declare it. + */ +export async function resourcesCapabilityDeclared( + conn: Connection +): Promise { + const discovered = await conn.discover(); + const caps = (discovered.capabilities as Record) ?? {}; + return isSettingsObject(caps.resources); +} + /** * Whether the skills extension declares `directoryRead: true`. * diff --git a/src/seps/sep-2640.yaml b/src/seps/sep-2640.yaml index c91b12ba..ed82fdf9 100644 --- a/src/seps/sep-2640.yaml +++ b/src/seps/sep-2640.yaml @@ -106,6 +106,8 @@ requirements: text: 'Declaring the extension itself commits the server to `skills/list` and `skills/get`.' - check: sep-2640-capability-empty-object text: 'An empty object indicates support for the extension with no optional features.' + - check: sep-2640-capability-requires-resources + text: "This extension depends only on the base protocol's Resources primitive. Skill files are served through `resources/read`, and the base Resources specification requires any server that supports resources to declare the `resources` capability. A server declaring this extension therefore **MUST** also declare the `resources` capability." - check: sep-2640-capability-directory-read-flag text: 'Clients MUST NOT call `resources/directory/read` against a server that has not declared `directoryRead: true`.' @@ -178,6 +180,8 @@ requirements: text: 'If the URI does not identify a skill the server serves, the server MUST return error `-32602` (Invalid params) — the same code `resources/read` uses for unknown resources.' - check: sep-2640-skills-get-answers-unlisted text: 'A server MUST answer for every skill it serves, whether or not that skill appears in its `skills/list` result. A skill absent from a partial listing is still retrievable by URI.' + - check: sep-2640-skills-get-cache-attributes + text: "`GetSkillResult` extends `CacheableResult`, so `ttlMs` and `cacheScope` are **REQUIRED**, as they are on `resources/read`, and carry the same semantics. `ttlMs` is the server's hint for how long a host may treat the entry as current before re-calling `skills/get`. Neither field is an integrity property." - check: sep-2640-skills-get-no-cursor text: 'The result carries no pagination cursor: a single entry is not a list.' From 5a5c52eefc263337e5f8dfccdb572b0c2988d1ed Mon Sep 17 00:00:00 2001 From: Paul Carleton Date: Fri, 11 Sep 2026 11:15:17 +0000 Subject: [PATCH 22/25] fix(sep-2640): compare frontmatter by content, not key order entry-frontmatter-identical compared each top-level key with JSON.stringify, which is key-order sensitive for nested objects. A server whose encoder sorts map keys (Go's encoding/json does) failed this MUST for any SKILL.md whose metadata keys are not in alphabetical order, although the content is identical. Compare structurally instead. A real difference still fails; both cases are pinned in negative.test.ts. --- src/scenarios/server/skills/enumeration.ts | 11 ++- src/scenarios/server/skills/negative.test.ts | 80 ++++++++++++++++++-- 2 files changed, 83 insertions(+), 8 deletions(-) diff --git a/src/scenarios/server/skills/enumeration.ts b/src/scenarios/server/skills/enumeration.ts index 17ba45fe..68b3eebc 100644 --- a/src/scenarios/server/skills/enumeration.ts +++ b/src/scenarios/server/skills/enumeration.ts @@ -18,6 +18,7 @@ * than fail against a server with an unenumerable catalog. */ +import { isDeepStrictEqual } from 'util'; import { ClientScenario, ConformanceCheck } from '../../../types'; import type { RunContext } from '../../../connection'; import { @@ -984,10 +985,14 @@ async function readbackChecks( const diffs: string[] = []; const keys = new Set([...Object.keys(fm), ...Object.keys(declared)]); + // Content, not serialisation: an encoder that orders map keys differently + // from the YAML source (Go's encoding/json sorts them) still describes the + // same frontmatter. for (const k of keys) { - const a = JSON.stringify(fm[k] ?? null); - const b = JSON.stringify(declared[k] ?? null); - if (a !== b) diffs.push(`${k}: file=${a} entry=${b}`); + if (isDeepStrictEqual(fm[k] ?? null, declared[k] ?? null)) continue; + diffs.push( + `${k}: file=${JSON.stringify(fm[k] ?? null)} entry=${JSON.stringify(declared[k] ?? null)}` + ); } checks.push( skillsCheck( diff --git a/src/scenarios/server/skills/negative.test.ts b/src/scenarios/server/skills/negative.test.ts index 04137021..c53298bb 100644 --- a/src/scenarios/server/skills/negative.test.ts +++ b/src/scenarios/server/skills/negative.test.ts @@ -1,4 +1,5 @@ import { describe, test, expect } from 'vitest'; +import { createHash } from 'crypto'; import { createServer, type IncomingMessage, type Server } from 'http'; import type { AddressInfo } from 'net'; import { testContext } from '../../../connection/testing'; @@ -74,7 +75,8 @@ async function readJsonBody( */ function startServerDeclaring( declared: unknown, - skills: object[] = [] + skills: object[] = [], + files: Record = {} ): Promise<{ url: string; server: Server }> { const server = createServer(async (req, res) => { if (req.method !== 'POST') { @@ -103,6 +105,15 @@ function startServerDeclaring( send({ skills }); return; } + if (method === 'resources/read') { + const uri = (body.params as { uri?: string } | undefined)?.uri; + if (uri !== undefined && uri in files) { + send({ + contents: [{ uri, mimeType: 'text/markdown', text: files[uri] }] + }); + return; + } + } if (method === 'resources/list') { send({ resources: [] }); return; @@ -126,8 +137,12 @@ function startServerDeclaring( }); } -async function runAgainst(declared: unknown, skills: object[] = []) { - const { url, server } = await startServerDeclaring(declared, skills); +async function runAgainst( + declared: unknown, + skills: object[] = [], + files: Record = {} +) { + const { url, server } = await startServerDeclaring(declared, skills, files); try { const scenario = new SkillsEnumerationScenario(); const checks = await scenario.run(testContext(url, DRAFT_PROTOCOL_VERSION)); @@ -142,8 +157,12 @@ async function runAgainst(declared: unknown, skills: object[] = []) { } } -async function checksFor(declared: unknown, skills: object[] = []) { - const { checks, violations } = await runAgainst(declared, skills); +async function checksFor( + declared: unknown, + skills: object[] = [], + files: Record = {} +) { + const { checks, violations } = await runAgainst(declared, skills, files); expect(violations, 'unexpected wire-schema violations').toEqual([]); return checks; } @@ -229,3 +248,54 @@ describe('SEP-2640 skill naming rules', () => { expect(checks.get(NAMING_ID)?.status).toBe('FAILURE'); }); }); + +/** + * `entry-frontmatter-identical` compares content, not serialisation. A server + * whose JSON encoder orders map keys differently from the YAML source (Go's + * encoding/json sorts them) is conformant and must pass. + */ +const IDENTICAL_ID = 'sep-2640-entry-frontmatter-identical'; + +function skillWithFile( + yamlMetadata: string, + entryMetadata: Record +) { + const uri = 'skill://demo/SKILL.md'; + const text = `---\nname: demo\ndescription: Demo skill\nmetadata:\n${yamlMetadata}---\n\nBody.\n`; + const entry = { + uri, + frontmatter: { + name: 'demo', + description: 'Demo skill', + metadata: entryMetadata + }, + resources: [ + { + uri, + digest: `sha256:${createHash('sha256').update(text).digest('hex')}`, + size: Buffer.byteLength(text) + } + ] + }; + return { entry, files: { [uri]: text } }; +} + +describe('SEP-2640 frontmatter identity', () => { + test('nested keys serialised in a different order are identical', async () => { + const { entry, files } = skillWithFile(' version: "1.0"\n author: me\n', { + author: 'me', + version: '1.0' + }); + const checks = await checksFor({}, [entry], files); + expect(checks.get(IDENTICAL_ID)?.status).toBe('SUCCESS'); + }); + + test('a differing nested value is a FAILURE', async () => { + const { entry, files } = skillWithFile(' version: "1.0"\n author: me\n', { + author: 'someone-else', + version: '1.0' + }); + const checks = await checksFor({}, [entry], files); + expect(checks.get(IDENTICAL_ID)?.status).toBe('FAILURE'); + }); +}); From f70f8e49ebb1fefcf2fa27365edec34833af547b Mon Sep 17 00:00:00 2001 From: Paul Carleton Date: Fri, 11 Sep 2026 11:15:17 +0000 Subject: [PATCH 23/25] fix(sep-2640): make the client-scenario mocks answer like a conformant server The skills/list and resources/read replies carried only resultType, so on the 2026-07-28 wire they lacked the ttlMs/cacheScope a ReadResourceResult requires, and skills/list lacked the attributes the enumeration scenario requires of servers. initialize always answered 2026-07-28, while the runner hands these extension scenarios' clients 2025-11-25 by default. A client that checks the negotiated version disconnected before listing, and every verify variant reported SKIPPED whether or not the client verified anything. The mocks now stamp the caching members and answer initialize at the requested version when the suite can speak it. The runner default is left alone because it also applies to the auth extension scenarios. Pinned in draft-result-fields.test.ts next to the other hand-rolled mocks. --- .../client/draft-result-fields.test.ts | 95 +++++++++++++++++++ src/scenarios/client/skills/mock-results.ts | 52 ++++++++++ src/scenarios/client/skills/no-prefetch.ts | 63 ++++++------ src/scenarios/client/skills/verification.ts | 22 +++-- 4 files changed, 198 insertions(+), 34 deletions(-) create mode 100644 src/scenarios/client/skills/mock-results.ts diff --git a/src/scenarios/client/draft-result-fields.test.ts b/src/scenarios/client/draft-result-fields.test.ts index 83076459..6ccd0bff 100644 --- a/src/scenarios/client/draft-result-fields.test.ts +++ b/src/scenarios/client/draft-result-fields.test.ts @@ -12,6 +12,8 @@ import type { BaseHttpScenario } from './http-base'; import { RequestMetadataScenario } from './request-metadata'; import { MRTRClientScenario } from './mrtr-client'; import { JsonSchemaRefDerefScenario } from './json-schema-ref-deref'; +import { SkillsNoPrefetchScenario } from './skills/no-prefetch'; +import { SkillsVerificationScenario } from './skills/verification'; /** * Pins that the hand-rolled mock servers used by client-direction scenarios @@ -528,3 +530,96 @@ describe('sep-2322-client-request-state mock results (2026-07-28)', () => { } }); }); + +describe('sep-2640 skills client mock results', () => { + const skillsScenarios: Array<{ + name: string; + make: () => BaseHttpScenario; + }> = [ + { + name: 'sep-2640-client-no-prefetch', + make: () => new SkillsNoPrefetchScenario() + }, + { + name: 'sep-2640-client-verify-digest', + make: () => new SkillsVerificationScenario('digest') + }, + { + name: 'sep-2640-client-verify-size', + make: () => new SkillsVerificationScenario('size') + }, + { + name: 'sep-2640-client-verify-frontmatter', + make: () => new SkillsVerificationScenario('frontmatter') + } + ]; + + for (const s of skillsScenarios) { + it(`${s.name} carries the draft-required members on skills/list and resources/read`, async () => { + const scenario = s.make(); + const { serverUrl } = await scenario.start( + testScenarioContext(DRAFT_PROTOCOL_VERSION) + ); + try { + let id = 1; + for (const [method, params] of [ + ['skills/list', {}], + ['resources/read', { uri: 'skill://pdf-processing/SKILL.md' }] + ] as const) { + const { status, body } = await post( + serverUrl, + { + jsonrpc: '2.0', + id: id++, + method, + params: { ...params, _meta: meta } + }, + { + 'mcp-protocol-version': DRAFT_PROTOCOL_VERSION, + 'Mcp-Method': method + } + ); + expect(status, method).toBe(200); + expect(body.result, method).toMatchObject(CACHEABLE_FIELDS); + if (method === 'resources/read') { + expect( + wireSchemaErrors(DRAFT_PROTOCOL_VERSION, body, method), + method + ).toEqual([]); + } + } + } finally { + await scenario.stop(); + } + }); + + // The runner hands these scenarios' clients 2025-11-25 unless + // --spec-version says otherwise, and the extension has no 2026-07-28 + // dependency, so the mock answers at the version the client asked for. + it(`${s.name} answers initialize at the requested protocol version`, async () => { + const scenario = s.make(); + const { serverUrl } = await scenario.start(testScenarioContext()); + try { + let id = 1; + for (const protocolVersion of ['2025-11-25', DRAFT_PROTOCOL_VERSION]) { + const { status, body } = await post(serverUrl, { + jsonrpc: '2.0', + id: id++, + method: 'initialize', + params: { + protocolVersion, + capabilities: {}, + clientInfo: { name: 'test', version: '1.0' } + } + }); + expect(status, protocolVersion).toBe(200); + expect(body.result.protocolVersion, protocolVersion).toBe( + protocolVersion + ); + } + } finally { + await scenario.stop(); + } + }); + } +}); diff --git a/src/scenarios/client/skills/mock-results.ts b/src/scenarios/client/skills/mock-results.ts new file mode 100644 index 00000000..c2d7b3ee --- /dev/null +++ b/src/scenarios/client/skills/mock-results.ts @@ -0,0 +1,52 @@ +/** + * Results for the hand-rolled SEP-2640 client-scenario mocks, shaped like a + * conformant server's so a strict client is graded on its skills behaviour + * rather than turned away by the harness. + */ + +import { + DRAFT_PROTOCOL_VERSION, + NEGOTIABLE_PROTOCOL_VERSIONS +} from '../../../types.js'; +import { withRequiredDraftResultFields } from '../../../mock-server/index.js'; + +/** + * `ListSkillsResult` is cacheable like `resources/list`, but `skills/list` is + * an extension method and not in the shared cacheable set, so the caching + * members are stamped here. + */ +export function skillsListResult(skills: object[]): object { + return { resultType: 'complete', ttlMs: 0, cacheScope: 'private', skills }; +} + +export function readResult(uri: unknown, text: string): unknown { + return withRequiredDraftResultFields('resources/read', { + contents: [{ uri, mimeType: 'text/markdown', text }] + }); +} + +/** + * The `initialize` result at the version the client asked for, when the suite + * can speak it. The extension has no 2026-07-28 dependency, and the runner + * hands these scenarios' clients 2025-11-25 unless `--spec-version` says + * otherwise, so a fixed 2026-07-28 reply would turn a correct stateful client + * away before it ever lists. + */ +export function initializeResult( + name: string, + request: { params?: { protocolVersion?: unknown } }, + capabilities: object +): object { + const requested = request.params?.protocolVersion; + const protocolVersion = + typeof requested === 'string' && + NEGOTIABLE_PROTOCOL_VERSIONS.includes(requested) + ? requested + : DRAFT_PROTOCOL_VERSION; + return { + resultType: 'complete', + protocolVersion, + serverInfo: { name: `${name}-server`, version: '1.0.0' }, + capabilities + }; +} diff --git a/src/scenarios/client/skills/no-prefetch.ts b/src/scenarios/client/skills/no-prefetch.ts index 131bcfd6..0128203b 100644 --- a/src/scenarios/client/skills/no-prefetch.ts +++ b/src/scenarios/client/skills/no-prefetch.ts @@ -17,6 +17,11 @@ import http from 'http'; import { ConformanceCheck } from '../../../types.js'; import { BaseHttpScenario } from '../http-base.js'; +import { + initializeResult, + readResult, + skillsListResult +} from './mock-results.js'; const SPEC_REFERENCE = { id: 'SEP-2640-Lazy-Retrieval', @@ -75,7 +80,15 @@ export class SkillsNoPrefetchScenario extends BaseHttpScenario { ): void { switch (request.method) { case 'initialize': - this.sendInitialize(res, request); + this.sendJson(res, { + jsonrpc: '2.0', + id: request.id, + result: initializeResult( + this.name, + request, + this.discoverCapabilities() + ) + }); return; case 'skills/list': @@ -83,30 +96,27 @@ export class SkillsNoPrefetchScenario extends BaseHttpScenario { this.sendJson(res, { jsonrpc: '2.0', id: request.id, - result: { - resultType: 'complete', - skills: [ - { - uri: SKILL_URI, - frontmatter: { - name: 'pdf-processing', - description: 'Extract, fill, and assemble PDF documents' + result: skillsListResult([ + { + uri: SKILL_URI, + frontmatter: { + name: 'pdf-processing', + description: 'Extract, fill, and assemble PDF documents' + }, + resources: [ + { + uri: SKILL_URI, + digest: digestOf(SKILL_MD), + size: Buffer.byteLength(SKILL_MD) }, - resources: [ - { - uri: SKILL_URI, - digest: digestOf(SKILL_MD), - size: Buffer.byteLength(SKILL_MD) - }, - { - uri: SUPPORTING_URI, - digest: digestOf(SUPPORTING), - size: Buffer.byteLength(SUPPORTING) - } - ] - } - ] - } + { + uri: SUPPORTING_URI, + digest: digestOf(SUPPORTING), + size: Buffer.byteLength(SUPPORTING) + } + ] + } + ]) }); return; @@ -118,10 +128,7 @@ export class SkillsNoPrefetchScenario extends BaseHttpScenario { this.sendJson(res, { jsonrpc: '2.0', id: request.id, - result: { - resultType: 'complete', - contents: [{ uri, mimeType: 'text/markdown', text: body }] - } + result: readResult(uri, body) }); return; } diff --git a/src/scenarios/client/skills/verification.ts b/src/scenarios/client/skills/verification.ts index 94143eeb..ee6d2d11 100644 --- a/src/scenarios/client/skills/verification.ts +++ b/src/scenarios/client/skills/verification.ts @@ -28,6 +28,11 @@ import http from 'http'; import { ConformanceCheck } from '../../../types.js'; import { untestableCheck } from '../../untestable.js'; import { BaseHttpScenario } from '../http-base.js'; +import { + initializeResult, + readResult, + skillsListResult +} from './mock-results.js'; const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills'; @@ -162,7 +167,15 @@ export class SkillsVerificationScenario extends BaseHttpScenario { ): void { switch (request.method) { case 'initialize': - this.sendInitialize(res, request); + this.sendJson(res, { + jsonrpc: '2.0', + id: request.id, + result: initializeResult( + this.name, + request, + this.discoverCapabilities() + ) + }); return; case 'skills/list': { @@ -178,7 +191,7 @@ export class SkillsVerificationScenario extends BaseHttpScenario { this.sendJson(res, { jsonrpc: '2.0', id: request.id, - result: { resultType: 'complete', skills: [entry] } + result: skillsListResult([entry]) }); return; } @@ -190,10 +203,7 @@ export class SkillsVerificationScenario extends BaseHttpScenario { this.sendJson(res, { jsonrpc: '2.0', id: request.id, - result: { - resultType: 'complete', - contents: [{ uri, mimeType: 'text/markdown', text }] - } + result: readResult(uri, text) }); return; } From cd41e4eb7dd4e5f201080c787e645ac4dd22f2aa Mon Sep 17 00:00:00 2001 From: Paul Carleton Date: Fri, 11 Sep 2026 11:15:17 +0000 Subject: [PATCH 24/25] docs(sep-2640): run from the published package; --force only with --spec-version The run notes pointed at a fork branch that goes stale on merge, and said --force is always needed. Extension scenarios only SKIP when an explicit --spec-version is passed; without it they run on the draft wire. --- RUNNING_SEP2640.md | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/RUNNING_SEP2640.md b/RUNNING_SEP2640.md index b52804e2..1ca58b43 100644 --- a/RUNNING_SEP2640.md +++ b/RUNNING_SEP2640.md @@ -4,24 +4,20 @@ Three server scenarios, brand-neutral: they discover everything dynamically and hardcode no fixture URIs, so pointing them at a new server is a URL change. ```bash -git clone https://github.com/panyam/mcpconformance && cd mcpconformance -git checkout chore/sep-2640-yaml -npm install && npm run build +npx @modelcontextprotocol/conformance server --url \ + --scenario sep-2640-skills-enumeration ``` -Then, per scenario: - -```bash -node dist/index.js server --url \ - --scenario sep-2640-skills-enumeration --force -``` +(or `npm start -- server ...` from a checkout of this repo). Repeat for `sep-2640-skills-manifest` and `sep-2640-skills-directory`. ## The two flags, and why both -`--force` is always needed. Extension scenarios do not match `--spec-version` -on their own, so without it they SKIP as "not applicable". +`--force` is only needed together with `--spec-version`. Extension scenarios +sit outside the spec timeline, so an explicit `--spec-version` does not select +them and they SKIP as "not applicable" unless `--force` is passed as well. +Without `--spec-version` they run on the draft wire and need no `--force`. `--spec-version` selects the **wire lifecycle**, not just a filter. The default is the stateless draft wire, which asserts `MCP-Protocol-Version: 2026-07-28` @@ -31,7 +27,7 @@ with no handshake. Against a server that does not speak that version you get: -32022: protocol version "2026-07-28" is not supported by this server ``` -which reads like a missing method and is not. Add `--spec-version 2025-11-25` +which reads like a missing method and is not. Add `--spec-version 2025-11-25 --force` to use the stateful wire instead. The scenarios are version-portable: only `ttlMs` / `cacheScope` are gated on 2026-07-28 and later, and that check reports SKIPPED below the floor rather than failing. From c4d79493f8013f18a7eb0bbe8aef584cff7b31aa Mon Sep 17 00:00:00 2001 From: Paul Carleton Date: Fri, 11 Sep 2026 13:41:59 +0000 Subject: [PATCH 25/25] docs(sep-2640): drop RUNNING_SEP2640.md Per-run notes and SDK result tables belong in the PR description, not a root-level doc that goes stale on merge. --- RUNNING_SEP2640.md | 81 ---------------------------------------------- 1 file changed, 81 deletions(-) delete mode 100644 RUNNING_SEP2640.md diff --git a/RUNNING_SEP2640.md b/RUNNING_SEP2640.md deleted file mode 100644 index 1ca58b43..00000000 --- a/RUNNING_SEP2640.md +++ /dev/null @@ -1,81 +0,0 @@ -# Running the SEP-2640 skills scenarios against an implementation - -Three server scenarios, brand-neutral: they discover everything dynamically and -hardcode no fixture URIs, so pointing them at a new server is a URL change. - -```bash -npx @modelcontextprotocol/conformance server --url \ - --scenario sep-2640-skills-enumeration -``` - -(or `npm start -- server ...` from a checkout of this repo). - -Repeat for `sep-2640-skills-manifest` and `sep-2640-skills-directory`. - -## The two flags, and why both - -`--force` is only needed together with `--spec-version`. Extension scenarios -sit outside the spec timeline, so an explicit `--spec-version` does not select -them and they SKIP as "not applicable" unless `--force` is passed as well. -Without `--spec-version` they run on the draft wire and need no `--force`. - -`--spec-version` selects the **wire lifecycle**, not just a filter. The default -is the stateless draft wire, which asserts `MCP-Protocol-Version: 2026-07-28` -with no handshake. Against a server that does not speak that version you get: - -``` --32022: protocol version "2026-07-28" is not supported by this server -``` - -which reads like a missing method and is not. Add `--spec-version 2025-11-25 --force` -to use the stateful wire instead. The scenarios are version-portable: only -`ttlMs` / `cacheScope` are gated on 2026-07-28 and later, and that check -reports SKIPPED below the floor rather than failing. - -## Reading the totals - -Each scenario's total includes the framework's `wire-schema-valid`, so -subtract one per scenario for SEP-2640 check IDs. SKIPPED rows are not -failures: `directoryRead` is an optional capability, and a server that does -not declare it correctly skips all six directory checks. - -## Verified runs - -| Implementation | Invocation | Result | -| ------------------------ | ------------------------------------- | ------------------------------- | -| mcpkit `ext/skills` | default | 32 / 6 / 7 = **45**, 0 failures | -| go-sdk (PRs 1238 + 1240) | stateless handler, default flags | 30 / 6 / 7 = **43**, 0 failures | -| go-sdk (PRs 1238 + 1240) | stateful, `--spec-version 2025-11-25` | 29 / 6 / 7 = **42**, 0 failures | -| csharp-sdk (PR 1856) | `/stateless`, default flags | 30 / 6 / 1 = **37**, 0 failures | -| csharp-sdk (PR 1856) | `/`, `--spec-version 2025-11-25` | 29 / 6 / 1 = **36**, 0 failures | - -All runs 2026-09-07. The C# directory column is 1 because that SDK does not -declare `directoryRead`, which is optional. - -## Per-implementation setup - -**go-sdk.** A minimal `skills.AddDirectory(server, dir, nil)` server is enough. -The streamable transport serves 2026-07-28 and later **only** when built with -`&mcp.StreamableHTTPOptions{Stateless: true}`, because SEP-2575 defines that as -the sessionless wire. A stateful handler negotiates a 2026-07-28 request down -to 2025-11-25, which is correct rather than a defect. - -**csharp-sdk.** The repo ships its own harness, so no fixture is needed: - -```bash -dotnet build tests/ModelContextProtocol.ConformanceServer -dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net10.0 -``` - -It serves the stateful lifecycle at `/` and the SEP-2575 stateless lifecycle at -`/stateless` off one port, so both rows above come from a single process. Note -it takes its port from `launchSettings.json` (3001 by default) and ignores -`ASPNETCORE_URLS` when run with `--no-build`. `GET /health` returns `Healthy` -once it is up. - -**mcpkit.** `examples/skills` serves the fixture used above: - -```bash -cd examples/skills && go build -o skills-demo . -./skills-demo --serve --addr=:18099 --skills=$PWD/skills -```