From c4f5231be97a9ea0cfdb5c4e8abe66a5dca7e78e Mon Sep 17 00:00:00 2001 From: Josh Knapp Date: Sat, 14 Nov 2026 11:24:14 -0800 Subject: [PATCH] docs: decision record for tool result payloads and excerpts Adds docs/tool-result-payloads.md covering PRs #27-#29: why every tool result repeats its payload as JSON text (claude.ai ignores structuredContent; client detection and structured-only were rejected), why memory_list / memory_search return excerpts (named to avoid the snippet.* tools), and why excerpts weight rare query terms rather than using ts_headline (tested; it picked the same wrong passage). Links it from the README design notes, adds two troubleshooting entries, and marks the "same 9 fields as list/search" claims in the earlier decision record and in memory.get's comment as superseded. Co-Authored-By: Claude Opus 5.5 --- README.md | 12 +++ apps/web/lib/mcp/tools.ts | 6 +- docs/memory-api-improvements.md | 4 +- docs/tool-result-payloads.md | 155 ++++++++++++++++++++++++++++++++ 4 files changed, 174 insertions(+), 3 deletions(-) create mode 100644 docs/tool-result-payloads.md diff --git a/README.md b/README.md index 50d9b66..49673bb 100644 --- a/README.md +++ b/README.md @@ -785,6 +785,11 @@ reopen a closed question. refuses ambiguous matches instead of guessing, why `memory_append` was dropped, and why file-mirroring was left as a convention rather than mechanised. +- [`docs/tool-result-payloads.md`](docs/tool-result-payloads.md) — why every + tool result repeats its payload as JSON text (claude.ai ignores + `structuredContent`), why `memory_list` / `memory_search` return a ~200-char + `excerpt` instead of the full body (use `memory_get` or `full: true`), and + why excerpts weight rare query terms instead of using `ts_headline`. --- @@ -803,6 +808,13 @@ reopen a closed question. for your `--callback-port`, and a portless `http://localhost/callback` entry will not match it. Authentik has no Dynamic Client Registration, so no client can add the URI itself — see **Which redirect URIs to register** above. +- **claude.ai gets only `"N result(s)"` from `memory_search` / `memory_list`, + with no ids or content** — the server predates PR #27, which repeats every + result's payload as JSON text; claude.ai ignores `structuredContent`. Deploy + a current build, then reconnect the connector. Claude Code was never + affected. See [`docs/tool-result-payloads.md`](docs/tool-result-payloads.md). +- **`memory_search` / `memory_list` show an `excerpt`, not the full memory** — + by design since PR #28. Call `memory_get` with the id, or pass `full: true`. - **Auth.js callback fails with `OAUTH_CALLBACK_ERROR`** — your `PUBLIC_URL` doesn't match the redirect URI your IdP is configured with. They must be exactly equal, scheme and trailing slash included. diff --git a/apps/web/lib/mcp/tools.ts b/apps/web/lib/mcp/tools.ts index 046ada1..640d6e9 100644 --- a/apps/web/lib/mcp/tools.ts +++ b/apps/web/lib/mcp/tools.ts @@ -534,8 +534,10 @@ const memoryGet: ToolDef = { // lexeme index, which outgrows `content` itself on large memories) — // both are Postgres retrieval internals that no MCP client can use, // and together they were the majority of every response. Returning - // them also pushed large memories past the tool-output cap. This is - // the same 9-field shape memory.list and memory.search return. + // them also pushed large memories past the tool-output cap. These are + // the same 9 fields memory.list / memory.search return, except those + // add `contentLength` and, unless `full: true`, swap `content` for an + // `excerpt` + `truncated`. const row = await db .select({ id: memories.id, diff --git a/docs/memory-api-improvements.md b/docs/memory-api-improvements.md index 0bf3db6..993823e 100644 --- a/docs/memory-api-improvements.md +++ b/docs/memory-api-improvements.md @@ -66,7 +66,9 @@ large living documents P2 exists to serve. At the 64,000-char content ceiling a response would land near 140,000 characters, under half of it content. **Shipped:** `memory_get` returns the same 9 fields as `memory_list` / -`memory_search`. `userId` is still selected for the authorization check and +`memory_search`. *(Superseded 2026-11-14: list/search now return an `excerpt`, +`contentLength` and `truncated` in place of `content`; `memory_get` still +returns these 9 fields. See [`tool-result-payloads.md`](tool-result-payloads.md).)* `userId` is still selected for the authorization check and stripped before responding. **Rejected: an `include: ("embedding" | "tsv")[]` opt-in.** The original draft diff --git a/docs/tool-result-payloads.md b/docs/tool-result-payloads.md new file mode 100644 index 0000000..03be6c0 --- /dev/null +++ b/docs/tool-result-payloads.md @@ -0,0 +1,155 @@ +# Decision record: what MCP tool results carry, and how much + +**Status:** shipped (PRs #27, #28, #29) +**Originated:** 2026-11-14, from a claude.ai session using the server as a custom connector +**Closed:** 2026-11-14, all three deployed and confirmed from claude.ai + +These three changes follow each other: making results visible to claude.ai +exposed how large they were, and shrinking them exposed that the excerpt +landed on the wrong passage. The reasoning, especially what was tried and +rejected, isn't recoverable from the diffs. + +Tools are defined with dots (`memory.search`) and surfaced to clients with +underscores (`memory_search`). + +--- + +## 1. Results must carry their payload in the text content ✅ SHIPPED (#27) + +**Problem.** From claude.ai, `memory_search` returned only `"8 result(s)"` +and `memory_list` only `"30 result(s)"`, with no ids, content or scores, so +the caller couldn't read or `memory_get` anything it found. The same helper +backs every tool, so `memory_get` showed only `memory `, `snippet_get` +dropped the body and `snippet_list` showed only a count. `project_identify` +and `memory_write` looked fine only because their summaries include the id. + +**Root cause.** `ok()` in `apps/web/lib/mcp/tools.ts` put the summary string +in `content` and the payload only in `structuredContent`. Clients differ: + +| client | reads | +|---|---| +| Claude Code | `structuredContent` (shows the JSON; the summary text isn't shown) | +| claude.ai connector | `content` text blocks only; ignores `structuredContent` (our tools declare no `outputSchema`) | + +The bug was invisible from Claude Code, which is where every earlier test of +the server ran. + +**Shipped.** The first text block is `\n`, and +`structuredContent` is unchanged. This is what the MCP spec asks for: a tool +returning structured content should also return the serialized JSON as text. +It's one block, not two, because some clients render only the first. + +**Rejected: detecting the client and sending one format.** +- MCP has no capability flag meaning "I render `structuredContent`". The only + signal is `clientInfo` name/version from `initialize`, and switching on + that breaks quietly whenever a client changes. +- The dispatcher (`lib/mcp/server.ts`) is stateless per POST, so + `clientInfo` isn't available at `tools/call` without adding session + tracking just to guess. +- A negotiated protocol version of `2025-06-18` (the version that introduced + `structuredContent`) doesn't mean the client renders it. + +**Rejected: structured-only with an `outputSchema` per tool.** That documents +the output and lets clients validate it, but nothing obliges claude.ai to +show it to the model. Text is the one channel every client reads. + +**Cost.** The payload is sent twice, but each client appears to read only one +copy (Claude Code the structured one, claude.ai the text), so it costs bytes +on the wire rather than context. This wasn't verified across other clients. + +**Test.** `tools.integration.test.ts` → "tool results carry their payload in +the text content (claude.ai)" asserts that the first text block of every +read tool contains ids and content, not just a count. + +--- + +## 2. `memory_list` / `memory_search` return excerpts, not full bodies ✅ SHIPPED (#28) + +**Problem.** Once #27 made results visible, both tools returned every +memory's full body, which is up to 64,000 chars each. One search over a few +dozen long memories could pour thousands of words into a session's context. +The `memory_get` description already said list/search "return summaries"; +they didn't. + +**Shipped.** Each list/search item carries these fields instead of `content`: + +| field | meaning | +|---|---| +| `excerpt` | ~200 chars, whitespace collapsed, cut on word boundaries, `…` where cut. For search: the passage best matching the query (§3). For list, or a search hit with no query word in the text: the memory's opening. | +| `contentLength` | length of the full body | +| `truncated` | `false` means the excerpt *is* the whole memory, so no `memory_get` is needed | + +- `full: true` on either tool returns the whole `content` instead. +- Ids, tags, scope, project, version, timestamps and search ranks are + unchanged. +- `memory_get` still returns the full body. +- The tool descriptions tell callers to `memory_get` the hits they need. + +**Named `excerpt`, not `snippet`,** because "snippet" already means the +`snippet_*` tools (named templates). An agent seeing a `snippet` field could +reasonably try `snippet_get` on it. + +**Tags don't replace the excerpt.** They say what a memory is *about*, not +whether it answers the question at hand. Both are returned. + +**Scope.** Only the MCP tool responses changed. The Web UI calls +`searchMemories` directly and still renders full results. + +**Code.** `lib/memory-excerpt.ts` (`makeExcerpt`, pure, unit-tested), wired +in via `withExcerpt()` in `lib/mcp/tools.ts`. + +--- + +## 3. Excerpts centre on rare query terms ✅ SHIPPED (#29) + +**Problem.** Searching `WHP founding partner pricing` excerpted *"…AI agents +are treated as business partners … (1) WHP commercial launch…"* instead of +the Founding Partner Program passage further down. + +**Root cause.** The first version scored each candidate window by how many +distinct query terms it contained, every term weighted 1, ties going to the +earliest window. "WHP" appears four times in that memory, and "partners" +appears coincidentally early on, so that window tied the real passage at 2 +terms and won by coming first. "pricing" matched nothing, because the memory +says "price". + +**Shipped.** +- **Rarity weighting:** each term counts `1 / occurrences in this memory`. A + word that appears once points at one passage; a word sprinkled throughout + (often the project's own name) says little about where to look. +- **Light suffix stripping:** strip one of `-ing -ed -es -s -e` while 4+ + chars remain, then match as a prefix. "pricing" and "price" both become + "pric"; "partners" becomes "partner". This is deliberately not a real + stemmer: a miss only costs a slightly worse window, never a wrong result. + +**Rejected: Postgres `ts_headline`** (suggested by the reporting agent). +Tested on the same text in Postgres 16: +- **OR query** (`whp | founding | partner | pricing`): picked the same wrong + "business partners … WHP" passage, with 1 or 2 fragments, because it also + weighs every word equally. +- **`plainto_tsquery`** (AND, which is what our full-text search uses): no + passage contains all four terms, so it fell back to the "$20 or $25 per + server" sentence. That's closer, but it still misses Founding Partner. + +It would also have cost an extra SQL call per search. Its one real advantage, +language-aware stemming, is approximated by the suffix stripping. + +**What would reopen this:** excerpts landing badly on long memories whose +relevant passage shares no literal or simply-inflected word with the query. +That's a meaning-only match, where the fix would be embedding-based passage +selection (chunk, embed, pick the nearest chunk), not better lexical scoring. + +**Test.** `lib/memory-excerpt.test.ts` → "weights rare query terms above ones +that appear all over the memory" uses the real memory text as its fixture. + +--- + +## Invariants: preserve these + +- Every tool result's **first text block carries the full payload** as JSON. + Never put data only in `structuredContent`. +- A passing check from Claude Code **proves nothing about claude.ai**. Verify + client-visible output changes from claude.ai, or assert on `content[0].text` + in tests. +- `memory_list` / `memory_search` return `excerpt`; **`memory_get` is the full + body**. Don't reintroduce full bodies into list/search except behind `full`. -- 2.52.0