Add test documents

docs/testing.md

@@ -55,6 +55,25 @@ Configure branch protection to require `Build and tests` and
 `BuzzDB hybrid retrieval and conversation contracts` before merging. The README
 badges track `main`, while each PR has its own check results.
 
+## Opt-In Experiments (not in CI)
+
+Some evaluations are deliberately **not** part of CI because they need a local
+Ollama, paid cloud calls, or both; the CI accuracy gate stays cloud-key-free. See
+[tests/experiments/README.md](../tests/experiments/README.md) for all of them.
+
+The **decoupled retrieval + answer-quality flow** is one such opt-in evaluation. It
+runs in two steps so retrieval is produced once with the app's local Ollama embedder
+and reused for answer generation and grading, skipping the Gemini embedding/index/cache
+path. Step 1 (`tests/experiments/retrieve-sources.mjs`) reuses the engine's
+`search_library` core to index the dataset PDF, applies a weak retrieval gate
+(sources non-empty, `reason == null`, vector hit ≥ 1), and writes the retrieved
+sources to a shared file; it exits non-zero on gate failure. Step 2
+(`run-eval-pipeline.mjs --sources-from <file>`) consumes that file, bypasses the
+worker/index/Gemini/rewrite path (no `GEMINI_API_KEY`), generates answers via
+OpenRouter, and grades them with Jev. The committed BuzzDB retrieval benchmark and
+its CI gate are unchanged. The full design is in
+[docs/decoupled-retrieval-answer-eval-plan.md](decoupled-retrieval-answer-eval-plan.md).
+
 ## Coverage
 
 The two coverage badges report **unit-test line coverage** separately for

tests/dataset/DataGeneratorPrompt.md

@@ -0,0 +1,28 @@
+Act as an expert assessment designer and evaluator. Based STRICTLY and SOLELY on the uploaded source material(s), generate an evaluation dataset of [Insert Number, e.g., 5 to 10] high-quality, open-ended questions with comprehensive answers and citations.
+
+Do not assume, extrapolate, or introduce external knowledge. Every answer and quote must be traceable directly to the provided sources.
+
+### Guidelines for Questions:
+- **Exclusively Open-Ended:** Avoid simple "Yes/No", single-word, or multiple-choice questions. Questions must prompt analysis, explanation, comparison, or operational reasoning (e.g., "Explain how...", "Compare the mechanisms of...", "Analyze the implications of...").
+- **Depth & Complexity:** Focus on core mechanisms, nuances, conditions, trade-offs, and critical takeaways rather than trivial trivia.
+
+---
+
+### Output Requirements:
+- Output **ONLY** a single, valid JSON array containing the evaluation items.
+- Do not add conversational text, introductions, or markdown formatting outside the JSON code block.
+- Ensure all quotes and special characters within strings are properly escaped to prevent parsing errors.
+
+Follow this exact JSON schema:
+
+```json
+[
+  {
+    "id": 1,
+    "category": "Factual | Synthesis | Analysis | Scenario",
+    "question": "Clear, detailed open-ended question",
+    "ideal_answer": "Authoritative, thorough answer explaining the concept completely based on the source text.",
+    "key_evidence_quote": "Exact verbatim quote from the text that directly proves the answer.",
+    "source_reference": "Document Title / Section / Page number (if identifiable)"
+  }
+]

tests/dataset/README.md

@@ -0,0 +1,102 @@
+# Test Dataset Manifest & Licensing Provenance
+
+This directory contains test documents used for search, indexing, and question-answering evaluation in TokenSmith. All bundled test materials are verified for public redistribution compatibility with the project's Apache-2.0 license.
+
+---
+
+## Dataset Summary Table
+
+| Document Filename | Title | Author / Rightsholder | Source & ID | License |
+| :--- | :--- | :--- | :--- | :--- |
+| `A Survey on Large Language Model based Autonomous Agents.pdf` | *A Survey on Large Language Model based Autonomous Agents* | Lei Wang, Chen Ma, Xueyang Feng, et al. | [arXiv:2308.11432](https://arxiv.org/abs/2308.11432) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
+| `Security Guidelines for Storage Infrastructure.pdf` | *Security Guidelines for Storage Infrastructure* | Ramaswamy Chandramouli (NIST), Doron Pinhas | [NIST CSRC SP 800-209](https://csrc.nist.gov/pubs/sp/800/209/final) | Public Domain (17 U.S.C. § 105) |
+| `Meta Large Language Model Compiler-Foundation Models of Compiler Optimization.pdf` | *Meta LLM Compiler: Foundation Models of Compiler Optimization* | Chris Cummins, Volker Seeker, et al. (Meta) | [arXiv:2407.02524](https://arxiv.org/abs/2407.02524) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
+| `Quantixar- High-performance Vector Data Management System.pdf` | *Quantixar: High-Performance Vector Data Management System* | Mengzhao Wang, et al. | [arXiv:2403.12583](https://arxiv.org/abs/2403.12583) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
+| `Algorithms-JeffE.pdf` | *Algorithms* | Jeff Erickson | [UIUC / algorithms.wtf](https://jeffe.cs.illinois.edu/teaching/algorithms/) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
+
+---
+
+## Document Details & Notices
+
+### 1. LLM Autonomous Agents Survey Paper
+* **Document Filename:** `A Survey on Large Language Model based Autonomous Agents.pdf`
+* **Document Title:** *A Survey on Large Language Model based Autonomous Agents*
+* **Author / Rightsholder:** Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, Ji-Rong Wen
+* **Authoritative Original Source:** [arXiv:2308.11432](https://arxiv.org/abs/2308.11432) [cs.AI, cs.CL]
+* **Direct Document Download:** [https://arxiv.org/pdf/2308.11432](https://arxiv.org/pdf/2308.11432)
+* **License:** **Creative Commons Attribution 4.0 International (CC BY 4.0)** ([License Deed](https://creativecommons.org/licenses/by/4.0/))
+* **Source Licensing Statement:** Published on arXiv with explicit author license attribution:
+  > *"This work is licensed under a Creative Commons Attribution 4.0 International License."*
+* **Required Attribution & Notices:** © 2023 The Authors. Published under the terms of the Creative Commons Attribution 4.0 International License.
+* **Changes Made:** None. Distributed in original preprint PDF form.
+* **Redistribution Eligibility:** CC BY 4.0 explicitly grants permissions to copy, redistribute, remix, transform, and build upon the material in any medium or format, for any purpose, including commercial and public repository inclusion, provided appropriate attribution is given.
+
+---
+
+### 2. NIST SP 800-209: Security Guidelines for Storage Infrastructure
+* **Document Filename:** `Security Guidelines for Storage Infrastructure.pdf`
+* **Document Title:** NIST Special Publication 800-209: *Security Guidelines for Storage Infrastructure*
+* **Author / Rightsholder:** Ramaswamy Chandramouli (NIST) and Doron Pinhas (Continuity Software)
+* **Authoritative Original Source:** [NIST CSRC Publication Record](https://csrc.nist.gov/pubs/sp/800/209/final) (DOI: [10.6028/NIST.SP.800-209](https://doi.org/10.6028/NIST.SP.800-209))
+* **Direct Document Download:** [NIST.SP.800-209.pdf](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-209.pdf)
+* **License:** **Public Domain / U.S. Government Work** ([17 U.S.C. § 105](https://www.law.cornell.edu/uscode/text/17/105))
+* **Licensing Statement & Authority:** [NIST Copyright, Fair Use and Licensing Policies](https://www.nist.gov/director/licensing):
+  > *"NIST publications and data are in the public domain and not subject to copyright protection in the United States unless otherwise indicated."*
+* **Required Attribution & Notices:** Contributed by the National Institute of Standards and Technology (NIST), an agency of the U.S. Department of Commerce.
+* **Changes Made:** None. Retained in original, unmodified form.
+* **Redistribution Eligibility:** As a work of the United States Government, this publication is exempt from copyright protection under U.S. law and is freely redistributable worldwide without licensing fees or permissions.
+
+---
+
+### 3. Meta LLM Compiler Paper
+* **Document Filename:** `Meta Large Language Model Compiler-Foundation Models of Compiler Optimization.pdf`
+* **Document Title:** *Meta Large Language Model Compiler: Foundation Models of Compiler Optimization*
+* **Author / Rightsholder:** Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, Hugh Leather (Meta)
+* **Authoritative Original Source:** [arXiv:2407.02524](https://arxiv.org/abs/2407.02524) [cs.PL, cs.AI, cs.LG]
+* **Direct Document Download:** [https://arxiv.org/pdf/2407.02524](https://arxiv.org/pdf/2407.02524)
+* **License:** **Creative Commons Attribution 4.0 International (CC BY 4.0)** ([License Deed](https://creativecommons.org/licenses/by/4.0/))
+* **Source Licensing Statement:** Published on arXiv with explicit author license attribution:
+  > *"This work is licensed under a Creative Commons Attribution 4.0 International License."*
+* **Required Attribution & Notices:** © 2024 The Authors. Published under the terms of the Creative Commons Attribution 4.0 International License.
+* **Changes Made:** None. Distributed in original preprint PDF form.
+* **Redistribution Eligibility:** CC BY 4.0 explicitly grants permissions to copy, redistribute, remix, transform, and build upon the material in any medium or format, for any purpose, including commercial and public repository inclusion, provided appropriate attribution is given.
+
+---
+
+### 4. Quantixar: High-Performance Vector Data Management System
+* **Document Filename:** `Quantixar- High-performance Vector Data Management System.pdf`
+* **Document Title:** *Quantixar: High-Performance Vector Data Management System*
+* **Author / Rightsholder:** Mengzhao Wang, Xuemeng Song, Xiaoliang Xu, et al.
+* **Authoritative Original Source:** [arXiv:2403.12583](https://arxiv.org/abs/2403.12583) [cs.DB]
+* **Direct Document Download:** [https://arxiv.org/pdf/2403.12583](https://arxiv.org/pdf/2403.12583)
+* **License:** **Creative Commons Attribution 4.0 International (CC BY 4.0)** ([License Deed](https://creativecommons.org/licenses/by/4.0/))
+* **Source Licensing Statement:** Deposited on arXiv with explicit author license selection:
+  > *"This work is licensed under a Creative Commons Attribution 4.0 International License."*
+* **Required Attribution & Notices:** © The Authors. Published under the terms of the Creative Commons Attribution 4.0 International License.
+* **Changes Made:** None. Distributed in original preprint PDF form.
+* **Redistribution Eligibility:** CC BY 4.0 permits unrestricted redistribution, sharing, and bundling into open-source repositories with attribution.
+
+---
+
+### 5. Algorithms Textbook
+* **Document Filename:** `Algorithms-JeffE.pdf`
+* **Document Title:** *Algorithms* (1st Edition)
+* **Author / Rightsholder:** Jeff Erickson
+* **Authoritative Original Source:** [Algorithms by Jeff Erickson](https://jeffe.cs.illinois.edu/teaching/algorithms/)
+* **Direct Document Download:** [https://jeffe.cs.illinois.edu/teaching/algorithms/book/Algorithms-JeffE.pdf](https://jeffe.cs.illinois.edu/teaching/algorithms/book/Algorithms-JeffE.pdf)
+* **License:** **Creative Commons Attribution 4.0 International (CC BY 4.0)** ([License Deed](https://creativecommons.org/licenses/by/4.0/))
+* **Source Licensing Statement:** Published by the author under a Creative Commons Attribution 4.0 International License:
+  > *"This work is available under a Creative Commons Attribution 4.0 International License."*
+* **Required Attribution & Notices:** © 2019 Jeff Erickson. Published under the terms of the Creative Commons Attribution 4.0 International License.
+* **Changes Made:** None. Retained in original, unmodified form.
+* **Redistribution Eligibility:** CC BY 4.0 explicitly grants permissions to copy, redistribute, remix, transform, and build upon the material in any medium or format, for any purpose, including commercial and public repository inclusion, provided appropriate attribution is given.
+
+---
+
+## Redistribution Policy for Third-Party Test Materials
+
+Third-party educational materials without explicit, documented public redistribution permissions (such as proprietary textbook chapters, restricted conference proceedings, or university lecture slides) must not be bundled directly in this repository.
+
+For test suites that require copyrighted third-party resources:
+1. Provide an authoritative public URL reference in this directory or test script.
+2. Allow users and CI runners to fetch their own local copies on demand into a non-tracked local cache directory.