|
|
|
@@ -0,0 +1,155 @@
|
|
|
|
|
# Decision record: what MCP tool results carry, and how much
|
|
|
|
|
|
|
|
|
|
**Status:** shipped (PRs #27, #28, #29)
|
|
|
|
|
**Originated:** 2026-11-14, from a claude.ai session using the server as a custom connector
|
|
|
|
|
**Closed:** 2026-11-14, all three deployed and confirmed from claude.ai
|
|
|
|
|
|
|
|
|
|
These three changes follow each other: making results visible to claude.ai
|
|
|
|
|
exposed how large they were, and shrinking them exposed that the excerpt
|
|
|
|
|
landed on the wrong passage. The reasoning, especially what was tried and
|
|
|
|
|
rejected, isn't recoverable from the diffs.
|
|
|
|
|
|
|
|
|
|
Tools are defined with dots (`memory.search`) and surfaced to clients with
|
|
|
|
|
underscores (`memory_search`).
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 1. Results must carry their payload in the text content ✅ SHIPPED (#27)
|
|
|
|
|
|
|
|
|
|
**Problem.** From claude.ai, `memory_search` returned only `"8 result(s)"`
|
|
|
|
|
and `memory_list` only `"30 result(s)"`, with no ids, content or scores, so
|
|
|
|
|
the caller couldn't read or `memory_get` anything it found. The same helper
|
|
|
|
|
backs every tool, so `memory_get` showed only `memory <id>`, `snippet_get`
|
|
|
|
|
dropped the body and `snippet_list` showed only a count. `project_identify`
|
|
|
|
|
and `memory_write` looked fine only because their summaries include the id.
|
|
|
|
|
|
|
|
|
|
**Root cause.** `ok()` in `apps/web/lib/mcp/tools.ts` put the summary string
|
|
|
|
|
in `content` and the payload only in `structuredContent`. Clients differ:
|
|
|
|
|
|
|
|
|
|
| client | reads |
|
|
|
|
|
|---|---|
|
|
|
|
|
| Claude Code | `structuredContent` (shows the JSON; the summary text isn't shown) |
|
|
|
|
|
| claude.ai connector | `content` text blocks only; ignores `structuredContent` (our tools declare no `outputSchema`) |
|
|
|
|
|
|
|
|
|
|
The bug was invisible from Claude Code, which is where every earlier test of
|
|
|
|
|
the server ran.
|
|
|
|
|
|
|
|
|
|
**Shipped.** The first text block is `<summary>\n<JSON payload>`, and
|
|
|
|
|
`structuredContent` is unchanged. This is what the MCP spec asks for: a tool
|
|
|
|
|
returning structured content should also return the serialized JSON as text.
|
|
|
|
|
It's one block, not two, because some clients render only the first.
|
|
|
|
|
|
|
|
|
|
**Rejected: detecting the client and sending one format.**
|
|
|
|
|
- MCP has no capability flag meaning "I render `structuredContent`". The only
|
|
|
|
|
signal is `clientInfo` name/version from `initialize`, and switching on
|
|
|
|
|
that breaks quietly whenever a client changes.
|
|
|
|
|
- The dispatcher (`lib/mcp/server.ts`) is stateless per POST, so
|
|
|
|
|
`clientInfo` isn't available at `tools/call` without adding session
|
|
|
|
|
tracking just to guess.
|
|
|
|
|
- A negotiated protocol version of `2025-06-18` (the version that introduced
|
|
|
|
|
`structuredContent`) doesn't mean the client renders it.
|
|
|
|
|
|
|
|
|
|
**Rejected: structured-only with an `outputSchema` per tool.** That documents
|
|
|
|
|
the output and lets clients validate it, but nothing obliges claude.ai to
|
|
|
|
|
show it to the model. Text is the one channel every client reads.
|
|
|
|
|
|
|
|
|
|
**Cost.** The payload is sent twice, but each client appears to read only one
|
|
|
|
|
copy (Claude Code the structured one, claude.ai the text), so it costs bytes
|
|
|
|
|
on the wire rather than context. This wasn't verified across other clients.
|
|
|
|
|
|
|
|
|
|
**Test.** `tools.integration.test.ts` → "tool results carry their payload in
|
|
|
|
|
the text content (claude.ai)" asserts that the first text block of every
|
|
|
|
|
read tool contains ids and content, not just a count.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 2. `memory_list` / `memory_search` return excerpts, not full bodies ✅ SHIPPED (#28)
|
|
|
|
|
|
|
|
|
|
**Problem.** Once #27 made results visible, both tools returned every
|
|
|
|
|
memory's full body, which is up to 64,000 chars each. One search over a few
|
|
|
|
|
dozen long memories could pour thousands of words into a session's context.
|
|
|
|
|
The `memory_get` description already said list/search "return summaries";
|
|
|
|
|
they didn't.
|
|
|
|
|
|
|
|
|
|
**Shipped.** Each list/search item carries these fields instead of `content`:
|
|
|
|
|
|
|
|
|
|
| field | meaning |
|
|
|
|
|
|---|---|
|
|
|
|
|
| `excerpt` | ~200 chars, whitespace collapsed, cut on word boundaries, `…` where cut. For search: the passage best matching the query (§3). For list, or a search hit with no query word in the text: the memory's opening. |
|
|
|
|
|
| `contentLength` | length of the full body |
|
|
|
|
|
| `truncated` | `false` means the excerpt *is* the whole memory, so no `memory_get` is needed |
|
|
|
|
|
|
|
|
|
|
- `full: true` on either tool returns the whole `content` instead.
|
|
|
|
|
- Ids, tags, scope, project, version, timestamps and search ranks are
|
|
|
|
|
unchanged.
|
|
|
|
|
- `memory_get` still returns the full body.
|
|
|
|
|
- The tool descriptions tell callers to `memory_get` the hits they need.
|
|
|
|
|
|
|
|
|
|
**Named `excerpt`, not `snippet`,** because "snippet" already means the
|
|
|
|
|
`snippet_*` tools (named templates). An agent seeing a `snippet` field could
|
|
|
|
|
reasonably try `snippet_get` on it.
|
|
|
|
|
|
|
|
|
|
**Tags don't replace the excerpt.** They say what a memory is *about*, not
|
|
|
|
|
whether it answers the question at hand. Both are returned.
|
|
|
|
|
|
|
|
|
|
**Scope.** Only the MCP tool responses changed. The Web UI calls
|
|
|
|
|
`searchMemories` directly and still renders full results.
|
|
|
|
|
|
|
|
|
|
**Code.** `lib/memory-excerpt.ts` (`makeExcerpt`, pure, unit-tested), wired
|
|
|
|
|
in via `withExcerpt()` in `lib/mcp/tools.ts`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 3. Excerpts centre on rare query terms ✅ SHIPPED (#29)
|
|
|
|
|
|
|
|
|
|
**Problem.** Searching `WHP founding partner pricing` excerpted *"…AI agents
|
|
|
|
|
are treated as business partners … (1) WHP commercial launch…"* instead of
|
|
|
|
|
the Founding Partner Program passage further down.
|
|
|
|
|
|
|
|
|
|
**Root cause.** The first version scored each candidate window by how many
|
|
|
|
|
distinct query terms it contained, every term weighted 1, ties going to the
|
|
|
|
|
earliest window. "WHP" appears four times in that memory, and "partners"
|
|
|
|
|
appears coincidentally early on, so that window tied the real passage at 2
|
|
|
|
|
terms and won by coming first. "pricing" matched nothing, because the memory
|
|
|
|
|
says "price".
|
|
|
|
|
|
|
|
|
|
**Shipped.**
|
|
|
|
|
- **Rarity weighting:** each term counts `1 / occurrences in this memory`. A
|
|
|
|
|
word that appears once points at one passage; a word sprinkled throughout
|
|
|
|
|
(often the project's own name) says little about where to look.
|
|
|
|
|
- **Light suffix stripping:** strip one of `-ing -ed -es -s -e` while 4+
|
|
|
|
|
chars remain, then match as a prefix. "pricing" and "price" both become
|
|
|
|
|
"pric"; "partners" becomes "partner". This is deliberately not a real
|
|
|
|
|
stemmer: a miss only costs a slightly worse window, never a wrong result.
|
|
|
|
|
|
|
|
|
|
**Rejected: Postgres `ts_headline`** (suggested by the reporting agent).
|
|
|
|
|
Tested on the same text in Postgres 16:
|
|
|
|
|
- **OR query** (`whp | founding | partner | pricing`): picked the same wrong
|
|
|
|
|
"business partners … WHP" passage, with 1 or 2 fragments, because it also
|
|
|
|
|
weighs every word equally.
|
|
|
|
|
- **`plainto_tsquery`** (AND, which is what our full-text search uses): no
|
|
|
|
|
passage contains all four terms, so it fell back to the "$20 or $25 per
|
|
|
|
|
server" sentence. That's closer, but it still misses Founding Partner.
|
|
|
|
|
|
|
|
|
|
It would also have cost an extra SQL call per search. Its one real advantage,
|
|
|
|
|
language-aware stemming, is approximated by the suffix stripping.
|
|
|
|
|
|
|
|
|
|
**What would reopen this:** excerpts landing badly on long memories whose
|
|
|
|
|
relevant passage shares no literal or simply-inflected word with the query.
|
|
|
|
|
That's a meaning-only match, where the fix would be embedding-based passage
|
|
|
|
|
selection (chunk, embed, pick the nearest chunk), not better lexical scoring.
|
|
|
|
|
|
|
|
|
|
**Test.** `lib/memory-excerpt.test.ts` → "weights rare query terms above ones
|
|
|
|
|
that appear all over the memory" uses the real memory text as its fixture.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Invariants: preserve these
|
|
|
|
|
|
|
|
|
|
- Every tool result's **first text block carries the full payload** as JSON.
|
|
|
|
|
Never put data only in `structuredContent`.
|
|
|
|
|
- A passing check from Claude Code **proves nothing about claude.ai**. Verify
|
|
|
|
|
client-visible output changes from claude.ai, or assert on `content[0].text`
|
|
|
|
|
in tests.
|
|
|
|
|
- `memory_list` / `memory_search` return `excerpt`; **`memory_get` is the full
|
|
|
|
|
body**. Don't reintroduce full bodies into list/search except behind `full`.
|