docs: decision record for tool result payloads and excerpts
Adds docs/tool-result-payloads.md covering PRs #27-#29: why every tool result repeats its payload as JSON text (claude.ai ignores structuredContent; client detection and structured-only were rejected), why memory_list / memory_search return excerpts (named to avoid the snippet.* tools), and why excerpts weight rare query terms rather than using ts_headline (tested; it picked the same wrong passage). Links it from the README design notes, adds two troubleshooting entries, and marks the "same 9 fields as list/search" claims in the earlier decision record and in memory.get's comment as superseded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3da0c77849
commit
c4f5231be9
4 files changed
+174
-3
No files matched your search
@@ -785,6 +785,11 @@ reopen a closed question.
|
||||
refuses ambiguous matches instead of guessing, why `memory_append` was
|
||||
dropped, and why file-mirroring was left as a convention rather than
|
||||
mechanised.
|
||||
- [`docs/tool-result-payloads.md`](docs/tool-result-payloads.md) — why every
|
||||
tool result repeats its payload as JSON text (claude.ai ignores
|
||||
`structuredContent`), why `memory_list` / `memory_search` return a ~200-char
|
||||
`excerpt` instead of the full body (use `memory_get` or `full: true`), and
|
||||
why excerpts weight rare query terms instead of using `ts_headline`.
|
||||
|
||||
---
|
||||
|
||||
@@ -803,6 +808,13 @@ reopen a closed question.
|
||||
for your `--callback-port`, and a portless `http://localhost/callback` entry
|
||||
will not match it. Authentik has no Dynamic Client Registration, so no client
|
||||
can add the URI itself — see **Which redirect URIs to register** above.
|
||||
- **claude.ai gets only `"N result(s)"` from `memory_search` / `memory_list`,
|
||||
with no ids or content** — the server predates PR #27, which repeats every
|
||||
result's payload as JSON text; claude.ai ignores `structuredContent`. Deploy
|
||||
a current build, then reconnect the connector. Claude Code was never
|
||||
affected. See [`docs/tool-result-payloads.md`](docs/tool-result-payloads.md).
|
||||
- **`memory_search` / `memory_list` show an `excerpt`, not the full memory** —
|
||||
by design since PR #28. Call `memory_get` with the id, or pass `full: true`.
|
||||
- **Auth.js callback fails with `OAUTH_CALLBACK_ERROR`** — your `PUBLIC_URL`
|
||||
doesn't match the redirect URI your IdP is configured with. They must be
|
||||
exactly equal, scheme and trailing slash included.
|
||||
|
||||
@@ -534,8 +534,10 @@ const memoryGet: ToolDef = {
|
||||
// lexeme index, which outgrows `content` itself on large memories) —
|
||||
// both are Postgres retrieval internals that no MCP client can use,
|
||||
// and together they were the majority of every response. Returning
|
||||
// them also pushed large memories past the tool-output cap. This is
|
||||
// the same 9-field shape memory.list and memory.search return.
|
||||
// them also pushed large memories past the tool-output cap. These are
|
||||
// the same 9 fields memory.list / memory.search return, except those
|
||||
// add `contentLength` and, unless `full: true`, swap `content` for an
|
||||
// `excerpt` + `truncated`.
|
||||
const row = await db
|
||||
.select({
|
||||
id: memories.id,
|
||||
|
||||
@@ -66,7 +66,9 @@ large living documents P2 exists to serve. At the 64,000-char content ceiling a
|
||||
response would land near 140,000 characters, under half of it content.
|
||||
|
||||
**Shipped:** `memory_get` returns the same 9 fields as `memory_list` /
|
||||
`memory_search`. `userId` is still selected for the authorization check and
|
||||
`memory_search`. *(Superseded 2026-11-14: list/search now return an `excerpt`,
|
||||
`contentLength` and `truncated` in place of `content`; `memory_get` still
|
||||
returns these 9 fields. See [`tool-result-payloads.md`](tool-result-payloads.md).)* `userId` is still selected for the authorization check and
|
||||
stripped before responding.
|
||||
|
||||
**Rejected: an `include: ("embedding" | "tsv")[]` opt-in.** The original draft
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
# Decision record: what MCP tool results carry, and how much
|
||||
|
||||
**Status:** shipped (PRs #27, #28, #29)
|
||||
**Originated:** 2026-11-14, from a claude.ai session using the server as a custom connector
|
||||
**Closed:** 2026-11-14, all three deployed and confirmed from claude.ai
|
||||
|
||||
These three changes follow each other: making results visible to claude.ai
|
||||
exposed how large they were, and shrinking them exposed that the excerpt
|
||||
landed on the wrong passage. The reasoning, especially what was tried and
|
||||
rejected, isn't recoverable from the diffs.
|
||||
|
||||
Tools are defined with dots (`memory.search`) and surfaced to clients with
|
||||
underscores (`memory_search`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Results must carry their payload in the text content ✅ SHIPPED (#27)
|
||||
|
||||
**Problem.** From claude.ai, `memory_search` returned only `"8 result(s)"`
|
||||
and `memory_list` only `"30 result(s)"`, with no ids, content or scores, so
|
||||
the caller couldn't read or `memory_get` anything it found. The same helper
|
||||
backs every tool, so `memory_get` showed only `memory <id>`, `snippet_get`
|
||||
dropped the body and `snippet_list` showed only a count. `project_identify`
|
||||
and `memory_write` looked fine only because their summaries include the id.
|
||||
|
||||
**Root cause.** `ok()` in `apps/web/lib/mcp/tools.ts` put the summary string
|
||||
in `content` and the payload only in `structuredContent`. Clients differ:
|
||||
|
||||
| client | reads |
|
||||
|---|---|
|
||||
| Claude Code | `structuredContent` (shows the JSON; the summary text isn't shown) |
|
||||
| claude.ai connector | `content` text blocks only; ignores `structuredContent` (our tools declare no `outputSchema`) |
|
||||
|
||||
The bug was invisible from Claude Code, which is where every earlier test of
|
||||
the server ran.
|
||||
|
||||
**Shipped.** The first text block is `<summary>\n<JSON payload>`, and
|
||||
`structuredContent` is unchanged. This is what the MCP spec asks for: a tool
|
||||
returning structured content should also return the serialized JSON as text.
|
||||
It's one block, not two, because some clients render only the first.
|
||||
|
||||
**Rejected: detecting the client and sending one format.**
|
||||
- MCP has no capability flag meaning "I render `structuredContent`". The only
|
||||
signal is `clientInfo` name/version from `initialize`, and switching on
|
||||
that breaks quietly whenever a client changes.
|
||||
- The dispatcher (`lib/mcp/server.ts`) is stateless per POST, so
|
||||
`clientInfo` isn't available at `tools/call` without adding session
|
||||
tracking just to guess.
|
||||
- A negotiated protocol version of `2025-06-18` (the version that introduced
|
||||
`structuredContent`) doesn't mean the client renders it.
|
||||
|
||||
**Rejected: structured-only with an `outputSchema` per tool.** That documents
|
||||
the output and lets clients validate it, but nothing obliges claude.ai to
|
||||
show it to the model. Text is the one channel every client reads.
|
||||
|
||||
**Cost.** The payload is sent twice, but each client appears to read only one
|
||||
copy (Claude Code the structured one, claude.ai the text), so it costs bytes
|
||||
on the wire rather than context. This wasn't verified across other clients.
|
||||
|
||||
**Test.** `tools.integration.test.ts` → "tool results carry their payload in
|
||||
the text content (claude.ai)" asserts that the first text block of every
|
||||
read tool contains ids and content, not just a count.
|
||||
|
||||
---
|
||||
|
||||
## 2. `memory_list` / `memory_search` return excerpts, not full bodies ✅ SHIPPED (#28)
|
||||
|
||||
**Problem.** Once #27 made results visible, both tools returned every
|
||||
memory's full body, which is up to 64,000 chars each. One search over a few
|
||||
dozen long memories could pour thousands of words into a session's context.
|
||||
The `memory_get` description already said list/search "return summaries";
|
||||
they didn't.
|
||||
|
||||
**Shipped.** Each list/search item carries these fields instead of `content`:
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `excerpt` | ~200 chars, whitespace collapsed, cut on word boundaries, `…` where cut. For search: the passage best matching the query (§3). For list, or a search hit with no query word in the text: the memory's opening. |
|
||||
| `contentLength` | length of the full body |
|
||||
| `truncated` | `false` means the excerpt *is* the whole memory, so no `memory_get` is needed |
|
||||
|
||||
- `full: true` on either tool returns the whole `content` instead.
|
||||
- Ids, tags, scope, project, version, timestamps and search ranks are
|
||||
unchanged.
|
||||
- `memory_get` still returns the full body.
|
||||
- The tool descriptions tell callers to `memory_get` the hits they need.
|
||||
|
||||
**Named `excerpt`, not `snippet`,** because "snippet" already means the
|
||||
`snippet_*` tools (named templates). An agent seeing a `snippet` field could
|
||||
reasonably try `snippet_get` on it.
|
||||
|
||||
**Tags don't replace the excerpt.** They say what a memory is *about*, not
|
||||
whether it answers the question at hand. Both are returned.
|
||||
|
||||
**Scope.** Only the MCP tool responses changed. The Web UI calls
|
||||
`searchMemories` directly and still renders full results.
|
||||
|
||||
**Code.** `lib/memory-excerpt.ts` (`makeExcerpt`, pure, unit-tested), wired
|
||||
in via `withExcerpt()` in `lib/mcp/tools.ts`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Excerpts centre on rare query terms ✅ SHIPPED (#29)
|
||||
|
||||
**Problem.** Searching `WHP founding partner pricing` excerpted *"…AI agents
|
||||
are treated as business partners … (1) WHP commercial launch…"* instead of
|
||||
the Founding Partner Program passage further down.
|
||||
|
||||
**Root cause.** The first version scored each candidate window by how many
|
||||
distinct query terms it contained, every term weighted 1, ties going to the
|
||||
earliest window. "WHP" appears four times in that memory, and "partners"
|
||||
appears coincidentally early on, so that window tied the real passage at 2
|
||||
terms and won by coming first. "pricing" matched nothing, because the memory
|
||||
says "price".
|
||||
|
||||
**Shipped.**
|
||||
- **Rarity weighting:** each term counts `1 / occurrences in this memory`. A
|
||||
word that appears once points at one passage; a word sprinkled throughout
|
||||
(often the project's own name) says little about where to look.
|
||||
- **Light suffix stripping:** strip one of `-ing -ed -es -s -e` while 4+
|
||||
chars remain, then match as a prefix. "pricing" and "price" both become
|
||||
"pric"; "partners" becomes "partner". This is deliberately not a real
|
||||
stemmer: a miss only costs a slightly worse window, never a wrong result.
|
||||
|
||||
**Rejected: Postgres `ts_headline`** (suggested by the reporting agent).
|
||||
Tested on the same text in Postgres 16:
|
||||
- **OR query** (`whp | founding | partner | pricing`): picked the same wrong
|
||||
"business partners … WHP" passage, with 1 or 2 fragments, because it also
|
||||
weighs every word equally.
|
||||
- **`plainto_tsquery`** (AND, which is what our full-text search uses): no
|
||||
passage contains all four terms, so it fell back to the "$20 or $25 per
|
||||
server" sentence. That's closer, but it still misses Founding Partner.
|
||||
|
||||
It would also have cost an extra SQL call per search. Its one real advantage,
|
||||
language-aware stemming, is approximated by the suffix stripping.
|
||||
|
||||
**What would reopen this:** excerpts landing badly on long memories whose
|
||||
relevant passage shares no literal or simply-inflected word with the query.
|
||||
That's a meaning-only match, where the fix would be embedding-based passage
|
||||
selection (chunk, embed, pick the nearest chunk), not better lexical scoring.
|
||||
|
||||
**Test.** `lib/memory-excerpt.test.ts` → "weights rare query terms above ones
|
||||
that appear all over the memory" uses the real memory text as its fixture.
|
||||
|
||||
---
|
||||
|
||||
## Invariants: preserve these
|
||||
|
||||
- Every tool result's **first text block carries the full payload** as JSON.
|
||||
Never put data only in `structuredContent`.
|
||||
- A passing check from Claude Code **proves nothing about claude.ai**. Verify
|
||||
client-visible output changes from claude.ai, or assert on `content[0].text`
|
||||
in tests.
|
||||
- `memory_list` / `memory_search` return `excerpt`; **`memory_get` is the full
|
||||
body**. Don't reintroduce full bodies into list/search except behind `full`.
|
||||
Reference in new issue
Block a user