fix: centre search excerpts on rare query terms, match simple inflections #29

Merged
jknapp merged 1 commits from fix/excerpt-centering into main 2026-11-14 19:14:27 +00:00
1 Commits
Author SHA1 Message Date
shadowdaoandClaude Opus 5.5 ea1805c2a7 fix: centre search excerpts on rare query terms, match simple inflections
Searching "WHP founding partner pricing" excerpted the "business
partners ... (1) WHP commercial launch" sentence instead of the Founding
Partner passage. Every query term counted the same, so a common term
("WHP", four times) plus a coincidental hit ("partners") tied the real
passage on two terms and won by coming first. "pricing" matched nothing:
the memory says "price".

- Weight each term by rarity within the memory (1 / occurrences), so a
  word that appears once outweighs one sprinkled throughout.
- Strip one common suffix (-ing/-ed/-es/-s/-e, keeping 4+ chars) and
  match as a prefix, so "pricing" and "price" both become "pric".

Postgres ts_headline, suggested as the alternative, was tested on the
same text and picked the same wrong passage with an OR query (it also
weighs words equally); with plainto_tsquery's AND it found no cover and
fell back to a different, also wrong, passage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:12:41 -08:00