docs: decision record for tool result payloads and excerpts #30

Merged
jknapp merged 1 commits from docs/tool-results-and-excerpts into main 2026-11-14 19:25:20 +00:00
4 changed files with 174 additions and 3 deletions

No files matched your search

+12
View File
@@ -785,6 +785,11 @@ reopen a closed question.
refuses ambiguous matches instead of guessing, why `memory_append` was
dropped, and why file-mirroring was left as a convention rather than
mechanised.
- [`docs/tool-result-payloads.md`](docs/tool-result-payloads.md) — why every
tool result repeats its payload as JSON text (claude.ai ignores
`structuredContent`), why `memory_list` / `memory_search` return a ~200-char
`excerpt` instead of the full body (use `memory_get` or `full: true`), and
why excerpts weight rare query terms instead of using `ts_headline`.
---
@@ -803,6 +808,13 @@ reopen a closed question.
for your `--callback-port`, and a portless `http://localhost/callback` entry
will not match it. Authentik has no Dynamic Client Registration, so no client
can add the URI itself — see **Which redirect URIs to register** above.
- **claude.ai gets only `"N result(s)"` from `memory_search` / `memory_list`,
with no ids or content** — the server predates PR #27, which repeats every
result's payload as JSON text; claude.ai ignores `structuredContent`. Deploy
a current build, then reconnect the connector. Claude Code was never
affected. See [`docs/tool-result-payloads.md`](docs/tool-result-payloads.md).
- **`memory_search` / `memory_list` show an `excerpt`, not the full memory** —
by design since PR #28. Call `memory_get` with the id, or pass `full: true`.
- **Auth.js callback fails with `OAUTH_CALLBACK_ERROR`** — your `PUBLIC_URL`
doesn't match the redirect URI your IdP is configured with. They must be
exactly equal, scheme and trailing slash included.
+4 -2
View File
@@ -534,8 +534,10 @@ const memoryGet: ToolDef = {
// lexeme index, which outgrows `content` itself on large memories) —
// both are Postgres retrieval internals that no MCP client can use,
// and together they were the majority of every response. Returning
// them also pushed large memories past the tool-output cap. This is
// the same 9-field shape memory.list and memory.search return.
// them also pushed large memories past the tool-output cap. These are
// the same 9 fields memory.list / memory.search return, except those
// add `contentLength` and, unless `full: true`, swap `content` for an
// `excerpt` + `truncated`.
const row = await db
.select({
id: memories.id,
+3 -1
View File
@@ -66,7 +66,9 @@ large living documents P2 exists to serve. At the 64,000-char content ceiling a
response would land near 140,000 characters, under half of it content.
**Shipped:** `memory_get` returns the same 9 fields as `memory_list` /
`memory_search`. `userId` is still selected for the authorization check and
`memory_search`. *(Superseded 2026-11-14: list/search now return an `excerpt`,
`contentLength` and `truncated` in place of `content`; `memory_get` still
returns these 9 fields. See [`tool-result-payloads.md`](tool-result-payloads.md).)* `userId` is still selected for the authorization check and
stripped before responding.
**Rejected: an `include: ("embedding" | "tsv")[]` opt-in.** The original draft
+155
View File
@@ -0,0 +1,155 @@
# Decision record: what MCP tool results carry, and how much
**Status:** shipped (PRs #27, #28, #29)
**Originated:** 2026-11-14, from a claude.ai session using the server as a custom connector
**Closed:** 2026-11-14, all three deployed and confirmed from claude.ai
These three changes follow each other: making results visible to claude.ai
exposed how large they were, and shrinking them exposed that the excerpt
landed on the wrong passage. The reasoning, especially what was tried and
rejected, isn't recoverable from the diffs.
Tools are defined with dots (`memory.search`) and surfaced to clients with
underscores (`memory_search`).
---
## 1. Results must carry their payload in the text content ✅ SHIPPED (#27)
**Problem.** From claude.ai, `memory_search` returned only `"8 result(s)"`
and `memory_list` only `"30 result(s)"`, with no ids, content or scores, so
the caller couldn't read or `memory_get` anything it found. The same helper
backs every tool, so `memory_get` showed only `memory <id>`, `snippet_get`
dropped the body and `snippet_list` showed only a count. `project_identify`
and `memory_write` looked fine only because their summaries include the id.
**Root cause.** `ok()` in `apps/web/lib/mcp/tools.ts` put the summary string
in `content` and the payload only in `structuredContent`. Clients differ:
| client | reads |
|---|---|
| Claude Code | `structuredContent` (shows the JSON; the summary text isn't shown) |
| claude.ai connector | `content` text blocks only; ignores `structuredContent` (our tools declare no `outputSchema`) |
The bug was invisible from Claude Code, which is where every earlier test of
the server ran.
**Shipped.** The first text block is `<summary>\n<JSON payload>`, and
`structuredContent` is unchanged. This is what the MCP spec asks for: a tool
returning structured content should also return the serialized JSON as text.
It's one block, not two, because some clients render only the first.
**Rejected: detecting the client and sending one format.**
- MCP has no capability flag meaning "I render `structuredContent`". The only
signal is `clientInfo` name/version from `initialize`, and switching on
that breaks quietly whenever a client changes.
- The dispatcher (`lib/mcp/server.ts`) is stateless per POST, so
`clientInfo` isn't available at `tools/call` without adding session
tracking just to guess.
- A negotiated protocol version of `2025-06-18` (the version that introduced
`structuredContent`) doesn't mean the client renders it.
**Rejected: structured-only with an `outputSchema` per tool.** That documents
the output and lets clients validate it, but nothing obliges claude.ai to
show it to the model. Text is the one channel every client reads.
**Cost.** The payload is sent twice, but each client appears to read only one
copy (Claude Code the structured one, claude.ai the text), so it costs bytes
on the wire rather than context. This wasn't verified across other clients.
**Test.** `tools.integration.test.ts` → "tool results carry their payload in
the text content (claude.ai)" asserts that the first text block of every
read tool contains ids and content, not just a count.
---
## 2. `memory_list` / `memory_search` return excerpts, not full bodies ✅ SHIPPED (#28)
**Problem.** Once #27 made results visible, both tools returned every
memory's full body, which is up to 64,000 chars each. One search over a few
dozen long memories could pour thousands of words into a session's context.
The `memory_get` description already said list/search "return summaries";
they didn't.
**Shipped.** Each list/search item carries these fields instead of `content`:
| field | meaning |
|---|---|
| `excerpt` | ~200 chars, whitespace collapsed, cut on word boundaries, `…` where cut. For search: the passage best matching the query (§3). For list, or a search hit with no query word in the text: the memory's opening. |
| `contentLength` | length of the full body |
| `truncated` | `false` means the excerpt *is* the whole memory, so no `memory_get` is needed |
- `full: true` on either tool returns the whole `content` instead.
- Ids, tags, scope, project, version, timestamps and search ranks are
unchanged.
- `memory_get` still returns the full body.
- The tool descriptions tell callers to `memory_get` the hits they need.
**Named `excerpt`, not `snippet`,** because "snippet" already means the
`snippet_*` tools (named templates). An agent seeing a `snippet` field could
reasonably try `snippet_get` on it.
**Tags don't replace the excerpt.** They say what a memory is *about*, not
whether it answers the question at hand. Both are returned.
**Scope.** Only the MCP tool responses changed. The Web UI calls
`searchMemories` directly and still renders full results.
**Code.** `lib/memory-excerpt.ts` (`makeExcerpt`, pure, unit-tested), wired
in via `withExcerpt()` in `lib/mcp/tools.ts`.
---
## 3. Excerpts centre on rare query terms ✅ SHIPPED (#29)
**Problem.** Searching `WHP founding partner pricing` excerpted *"…AI agents
are treated as business partners … (1) WHP commercial launch…"* instead of
the Founding Partner Program passage further down.
**Root cause.** The first version scored each candidate window by how many
distinct query terms it contained, every term weighted 1, ties going to the
earliest window. "WHP" appears four times in that memory, and "partners"
appears coincidentally early on, so that window tied the real passage at 2
terms and won by coming first. "pricing" matched nothing, because the memory
says "price".
**Shipped.**
- **Rarity weighting:** each term counts `1 / occurrences in this memory`. A
word that appears once points at one passage; a word sprinkled throughout
(often the project's own name) says little about where to look.
- **Light suffix stripping:** strip one of `-ing -ed -es -s -e` while 4+
chars remain, then match as a prefix. "pricing" and "price" both become
"pric"; "partners" becomes "partner". This is deliberately not a real
stemmer: a miss only costs a slightly worse window, never a wrong result.
**Rejected: Postgres `ts_headline`** (suggested by the reporting agent).
Tested on the same text in Postgres 16:
- **OR query** (`whp | founding | partner | pricing`): picked the same wrong
"business partners … WHP" passage, with 1 or 2 fragments, because it also
weighs every word equally.
- **`plainto_tsquery`** (AND, which is what our full-text search uses): no
passage contains all four terms, so it fell back to the "$20 or $25 per
server" sentence. That's closer, but it still misses Founding Partner.
It would also have cost an extra SQL call per search. Its one real advantage,
language-aware stemming, is approximated by the suffix stripping.
**What would reopen this:** excerpts landing badly on long memories whose
relevant passage shares no literal or simply-inflected word with the query.
That's a meaning-only match, where the fix would be embedding-based passage
selection (chunk, embed, pick the nearest chunk), not better lexical scoring.
**Test.** `lib/memory-excerpt.test.ts` → "weights rare query terms above ones
that appear all over the memory" uses the real memory text as its fixture.
---
## Invariants: preserve these
- Every tool result's **first text block carries the full payload** as JSON.
Never put data only in `structuredContent`.
- A passing check from Claude Code **proves nothing about claude.ai**. Verify
client-visible output changes from claude.ai, or assert on `content[0].text`
in tests.
- `memory_list` / `memory_search` return `excerpt`; **`memory_get` is the full
body**. Don't reintroduce full bodies into list/search except behind `full`.