fix(core): fit chunk sizes to the embedding model's sequence limit

[yichuan-w]

Problem

leann build used fixed chunk sizes (doc 256, code 512 tiktoken tokens) regardless of the embedding model. The macOS default all-MiniLM-L6-v2 reads only 256 tokens, so on a real leann build of leann-core sources 43% of the indexed code text was never embedded (median code chunk = 461 MiniLM tokens).

Two related bugs:

• get_model_token_limit() didn't know Hugging Face sentence-transformers models and fell back to 2048.
• create_text_chunks(max_tokens_per_chunk=...) replaced chunk_size instead of capping it, so an explicit 256 grew to ~1700 for those models.

Fix

• cli: cap --doc-chunk-size / --code-chunk-size at 70% of the model's limit (overlap capped to half). 70% because WordPiece counts ~30% more tokens than tiktoken on web text/code, and SentenceSplitter overshoots on spans it can't split (tables, long lines). Prints 📏 … chunk size doc 256 → 179, code 512 → 179 when it applies. Models with large limits (OpenAI, nomic, Qwen3) are unaffected.
• embedding_compute: read max_seq_length from the repo's sentence_bert_config.json for local HF models (only when no base_url, name contains /).
• chunking_utils: max_tokens_per_chunk only shrinks chunk_size (traditional and AST paths).

Results (all-MiniLM-L6-v2)

Real leann build + LeannSearcher on leann-core sources, 18 natural-language code queries (relevant = chunk containing the target def):

	chunks	text not embedded	hit@1	hit@5	MRR@10
before (512)	402	43.3%	0.11	0.56	0.301
after (179)	1941	0.0%	0.44	0.56	0.535

Offline sweep (brute-force cosine): code MRR 0.258 @512 → 0.607 @256 → 0.658 @179. On 42 web pages / 22 queries, doc sizes 179–256 are within noise (MRR 0.62–0.73), so the doc cap is about guaranteeing text gets embedded rather than a quality jump.

Trade-off: more, smaller chunks for small models (≈4.8× on code), so builds embed more passages.

Tests

• New tests/test_chunk_size_cap.py (cap only shrinks; large limit doesn't grow chunks; HF discovery and its failure path, no Hub call with base_url or bare names).
• CI=true pytest tests/: 463 passed; 7 failures are pre-existing on main locally (diskann not installed, test_cli_prompt_template sync pickle).

Existing indexes are untouched; leann build --force re-chunks.

🤖 Generated with Claude Code