Skip to content

Search: show the matched passage in each result, with the match highlighted - #3057

Open
Mytherin wants to merge 2 commits into
Foundry376:masterfrom
Mytherin:search-match-excerpts
Open

Mytherin wants to merge 2 commits into
Foundry376:masterfrom
Mytherin:search-match-excerpts

Conversation

@Mytherin

Copy link
Copy Markdown

A search result whose body matched now shows the words around the match in place of the start of the latest messages. FTS5's snippet() cuts the passage from the ThreadSearch body column for the query's free-text terms, and the mark is trimmed to the typed prefix, so "wall" highlights "Wall" in "Wallpaper". Matches are highlighted in yellow, or a dark yellow in themes with light text.

Screenshot 2026-11-14 140032

A search result now highlights the query's words in its participant names
and subject, and a result whose body matched shows the words around the
match in place of the start of the latest message, as Apple Mail does.

FTS5's snippet() picks the window of the ThreadSearch body around the match,
for the query's free-text terms phrased as LocalSearchQueryBackend phrases
them. At most four words are kept before the first match, so a narrow row
doesn't cut the match off. One prefix highlighter marks the words in
excerpts, subjects and participants: "wall" highlights "Wall" in
"Wallpaper", ignoring case and diacritics. subject: terms apply to subjects
only, from: and to: terms to participants only, and terms under a NOT are
never highlighted. A body found only through Porter stemming keeps its plain
snippet.

The search subscription hands the thread list a single highlighter, set on
each result as __searchMatch. The highlight colors are theme variables
(@list-search-match-bg/-color, from @text-color-search-match), overridden in
ui-dark with a dark yellow.
@bengotow

Copy link
Copy Markdown
Collaborator

Thanks Mark, this is a great improvement. Showing why a result matched makes search feel much better, and we'd love to have it. Scoping it to search results only, carrying the match on the cloned threads like __messages, and building the FTS match the same way LocalSearchQueryBackend does all look right.

Two changes we'd like before merging, and a few optional ones:

Required

1. Don't re-query excerpts on every list emission, and don't hold the list until they load.
_flatMapJoiningSearchMatches runs after _flatMapJoiningMessages, which emits whenever any visible thread's messages change, on every scroll, and every time remote results are merged in. Each emission re-runs ThreadSearch MATCH … AND content_id IN (…) for every visible id. content_id isn't indexed, so each run costs about as much as the search itself, which is slow for a common term in a large mailbox. And because the result set only passes through once that promise settles, every update in the search view, including the first render, waits on that extra query.

Excerpts don't change while the highlighter lives, so please:

  • cache them per thread id for the life of the highlighter, and only query ids not yet in the cache (the same pattern _flatMapJoiningMessages uses for $messagesResultSets);
  • ideally, emit the result set right away and re-emit with excerpts once they arrive, so search results never wait on the excerpt query.

2. Use FTS5's own match markers in snippet() instead of re-matching the window in JS.
ThreadSearch uses the porter unicode61 tokenizer, so many real body matches have no literal prefix in the text. invoice finds "invoicing", and as your spec shows, that thread then falls back to the plain snippet with no sign of why it matched. Passing markers to snippet() (e.g. '\u0001' / '\u0002') and splitting on them highlights exactly what the index matched, stems included, and saves us maintaining a second tokenizer that has to stay in step with the index. Highlighting whole tokens ("Wallpaper" rather than "Wall") is fine with us. If you'd like, you can still trim the mark to the typed prefix when it matches literally.

termHighlighter is still the right tool for subjects and participants, since those are drawn from the model rather than from FTS, so that part can stay.

Optional

  • Where the code lives. The query-to-FTS logic could move next to search-query-backend-local.ts in app/src/services/search and be exported through mailspring-exports. That would let thread-list import the types instead of copying them in types.ts, avoid checking whether the subscription has a searchHighlighter() method, and keep DatabaseStore._query inside core. Not a blocker.
  • Dark themes. Only ui-dark overrides the highlight colors. ui-taiga and ui-darkside get the bright #fff000. It's readable, but probably worth a line in each.

Please keep the scope to the two required items plus whatever optional ones you want; no need to expand beyond that. Thanks again, really nice work!

… core

Each thread's excerpt is loaded once for the life of the thread list's data
source, and results are emitted without waiting for the excerpt query, then
again with excerpts once it finishes.

snippet() marks what the index matched, so a body found through Porter
stemming ("invoicing" for "invoice") is highlighted too; a literal match is
still trimmed to the typed prefix.

The highlighter lives in services/search and is exported through
mailspring-exports. The search perspective provides it through
MailboxPerspective.searchHighlighter(), and ThreadListStore hands it to
ThreadListDataSource.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Mytherin

Copy link
Copy Markdown
Author

Thanks Ben! All addressed:

1. Excerpt queries. Excerpts are now cached per thread for the life of the data source. Each list update only queries threads that haven't been loaded or requested yet, and a request that's still running is shared, not sent again. Results show up right away with the plain snippet, then update once the excerpts arrive.

2. FTS5 markers. snippet() now wraps matches in char(1)/char(2), and we split on those. A literal match is still trimmed to the typed prefix ("Wall" in "Wallpaper"). A match found through stemming is highlighted as a whole word, so invoice now highlights "invoicing". termHighlighter stays for subjects and participants.

Where the code lives. The highlighter and its spec moved to app/src/services/search/. searchHighlighter and the Highlighted/SearchHighlighter types are exported from mailspring-exports. MailboxPerspective.searchHighlighter() returns null, SearchMailboxPerspective overrides it, and ThreadListStore passes the result to ThreadListDataSource. That removes the copied types and the check on the subscription, and keeps DatabaseStore._query inside core.

Dark themes. I left these alone. ui-taiga and ui-darkside draw the thread list on white (only their sidebars are dark), so the yellow looks the same as in ui-light.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants