[mbeps]

Goal: Upgrade the knowledge base RAG system into a Unified 3-Tier Hybrid architecture combining upload-time LLM metadata and a PostgreSQL document relationship graph with Qdrant chunk retrieval, scoped document reading, and an isolated KB reader subagent that protects the main chat context window.

Architecture: During document ingestion (ingestDocumentPipeline), extracted text and chunks are analyzed by the user's default chat model (with deterministic fallback) to generate a high-density summary, document classification, key entities, a chunk-indexed Table of Contents (TOC), and bidirectional relationship edges (kb_document_relation) against top-K candidate documents in the knowledge base. At query time, search_knowledge_base remains the single abstracted tool exposed to the main chat model; internally, it runs an isolated KB Reader Subagent (ToolLoopAgent) equipped with the Tier 1 document/graph map, explore_kb_graph, search_kb_chunks, and context-guarded read_kb_document tools, returning a synthesized answer and citations without polluting the main thread with raw document dumps.

---

1. High-Level Overview

1.1 High-Level Architecture

flowchart TD
    subgraph Ingestion ["1. Upload & Ingestion Pipeline (Inngest)"]
        Upload["S3 Upload + kb_document (pending)"] --> Extract["Extract Text & Recursive Chunks"]
        Extract --> Embed["Embed Chunks -> Qdrant (with chunkIndex)"]
        Extract --> MetaLLM["LLM Metadata & TOC Extractor\n(Deterministic Fallback)"]
        MetaLLM --> Prune["Top-15 Candidate Pruning\n(Entity + Keyword Overlap)"]
        Prune --> GraphLLM["LLM Bidirectional Edge Generator"]
        GraphLLM --> PG[("PostgreSQL\nkb_document + kb_document_relation")]
    end

    subgraph Runtime ["2. Query-Time Isolated Subagent Retrieval"]
        MainLLM["Main Chat Model"] -->|"Calls search_knowledge_base(query)"| KBTool["search_knowledge_base Tool"]
        KBTool --> PreFetch["Initial Hybrid Search (Vector + FTS RRF)\n+ Tier 1 Catalog & Graph Map"]
        PreFetch --> Subagent["Isolated KB Reader Subagent\n(workerResolved ?? resolved)"]
        Subagent --> T1["explore_kb_graph\n(1-3 hop undirected traversal)"]
        Subagent --> T2["search_kb_chunks\n(Qdrant RRF + docId filter)"]
        Subagent --> T3["read_kb_document\n(Qdrant ordered chunks + Context Guard)"]
        Subagent -->|"Distilled Answer + Top Citations"| MainLLM
    end

Loading

1.2 Global Constraints

• Zero-Git Policy: Never execute git commit, git add, git push, or git stash. Only read-only Git commands (status, diff, log) are permitted.
• Database Primary Keys: All entity primary and foreign keys in knowledgebase and kb_document use text (UUID strings), NOT PostgreSQL native uuid column types. New tables (kb_document_relation) must use text for foreign keys to maintain type compatibility.
• Direct Schema Imports: Import Drizzle schemas directly from @/drizzle/schemas/*; never create or import from barrel files.
• File Size & Export Discipline: Every file must stay under 300 lines and export a single primary function, component, or schema/type bundle.
• 100% Test Coverage Gate: vitest.config.ts enforces 100% statements, branches, functions, and lines. Never use /* v8 ignore */.
• Resilient Ingestion: LLM metadata or graph extraction failures (missing chat model, provider timeout, malformed JSON) must fall back to deterministic metadata/TOC extraction and never fail Qdrant vector indexing.

1.3 Review Focus

1. Unconfigured Chat Model or Malformed LLM JSON on Upload: When a user has an embedding model configured but no chat model (or a local model emits markdown-fenced/invalid JSON during ingestion), extractDocumentMetadata and generateDocumentRelations must catch the error and fall back to deterministic heading/excerpt metadata and zero graph edges without failing ingestDocumentPipeline.
2. LLM Hallucinated Document IDs or Self-Loop Edges: During relationship generation, the LLM may output non-existent target IDs, IDs belonging to another knowledge base, or self-referential edges (sourceDocId === targetDocId). Edge sanitization must filter against the candidate ID allowlist and enforce sourceDocId !== targetDocId and (sourceDocId === doc.id || targetDocId === doc.id).
3. Small Context Window Overflow in read_kb_document: When a subagent running on an 8k/16k model attempts to read a 40-page document (docTokens > usableBudget), guardContextBudget must block the full read and return safeToReadFull: false with the document's tableOfContents and chunkCount so the subagent reads bounded [chunkStartIndex, chunkEndIndex] slices.
4. Concurrent Document Deletion & Re-Indexing Edge Staleness: Re-ingesting an existing document must atomically delete prior kb_document_relation rows where sourceDocId = doc.id OR targetDocId = doc.id before inserting fresh edges, and abort if kbDocument was deleted concurrently during LLM generation.
5. Backwards Compatibility with Existing search_knowledge_base Callers & UI Citations: extractCitations in the chat UI expects result.results to be an array of { content, relevanceScore, documentId, documentName, s3Key, kbId?, kbName? }. search_knowledge_base must preserve this exact results array structure alongside the new answer and relatedDocuments fields, while toModelOutput formats a compact context representation for the main model.

---

2. Alternatives Evaluated

Approach	Description	Pros	Cons / Why Rejected
1. Unified Hybrid + Internal KB Reader Subagent (Selected)	Keep Qdrant + RRF chunk search, add PostgreSQL metadata + relationship graph on upload, and encapsulate multi-step graph/chunk/file reading inside search_knowledge_base via an isolated subagent.	Zero main-thread context bloat; works on 8k local models and 1M frontier models; zero UI tool-picker complexity; 100% backwards-compatible with citations and transform pipelines.	Adds 1 short LLM metadata call + 1 candidate-pruned edge call during background upload ingestion.
2. Pure File Reading + Graph (Drop Qdrant Vectors)	Remove Qdrant vector chunking; rely strictly on LLM metadata catalog, graph edges, and reading full files.	Removes vector embedding step on upload.	Rejected: Fails on small-context models (8k–32k) when opening large PDFs and fails on 200+ document KBs where the catalog alone exceeds context budget.
3. Expose 3 Raw KB Tools Directly to Main Orchestrator	Register explore_kb_graph, search_knowledge_base, and read_kb_document directly on the main chat model and rely on delegate_task for isolation.	No internal subagent wrapper inside KB tool.	Rejected: Violates abstraction goal; if the user hasn't manually enabled Subagent Delegation in the toolbar, calling read_kb_document dumps tens of thousands of tokens directly into the main chat history.
4. Dedicated Graph Database (Neo4j / Memgraph)	Store document nodes and relationship edges in an external Cypher graph database.	Native graph query syntax.	Rejected (YAGNI): Document-level graphs have $10^1\text{–}10^3$ nodes per KB. PostgreSQL indexes + Foreign Key ON DELETE CASCADE handle 1–3 hop traversals in <1ms without extra containers or split-brain delete bugs.

---

3. Edge Cases & Failure Modes

Edge Case / Failure Mode	Handling Strategy
No Chat Model Configured During Upload	extractDocumentMetadata catches ProviderNotConfiguredError from resolveDefaultChatProvider(userId) and returns deterministic TOC + excerpt summary. generateDocumentRelations catches and returns 0 edges. Vector indexing in Qdrant succeeds normally.
Local / OpenAI-Compatible Model Rejects json_schema or Emits Markdown Code Fences	Metadata and relation prompts use generateText with explicit JSON instructions and a fence-stripping parser (stripJsonFences + JSON.parse + schema.safeParse) rather than brittle provider-dependent response_format schemas.
LLM Emits Out-of-Bounds TOC Chunk Indices	sanitizeTocEntries clamps chunkStartIndex and chunkEndIndex to [0, chunks.length - 1] and swaps or adjusts if chunkStartIndex > chunkEndIndex.
LLM Hallucinates Foreign targetDocId or Self-Loop in Graph	generateDocumentRelations validates every edge against validCandidateIds.has(peerId), enforces sourceDocId !== targetDocId, and requires one endpoint to be doc.id. Database CHECK (source_doc_id <> target_doc_id) and Foreign Keys provide a second hard barrier.
100-Page PDF Read on 8k Context Model	guardKbContextBudget checks estimatedSliceTokens > usableBudget (0.75 * contextWindow - usedTokens). If exceeded, read_kb_document returns safeToRead: false, the document's tableOfContents, and instructions to request a bounded [chunkStartIndex, chunkEndIndex] range.
Document Deleted While Ingestion is Running	ingestDocumentPipeline already checks if kbDocument still exists before Qdrant upsert; generateDocumentRelations catches foreign key errors or empty document checks cleanly, and ON DELETE CASCADE on kb_document_relation purges any edges if deleted mid-flight.
Re-Indexing an Existing Document	generateDocumentRelations deletes all existing kb_document_relation rows where sourceDocId = doc.id OR targetDocId = doc.id before inserting newly discovered edges, preventing stale or duplicate edges.

[mbeps]

1. Does it default to subagents reading files directly, or does it still use vector semantic search?

It uses both in a hybrid multi-tier pipeline:

1. Vector Semantic Search is Still Active & First-Line:

   • Retrieval does not start by reading raw files.
   • When search_knowledge_base is triggered, run-kb-subagent.ts immediately pre-fetches top candidate chunks using Hybrid Search (Qdrant semantic vector similarity + full-text search with Reciprocal Rank Fusion) along with the Tier 1 document graph.
   • If no LLM model is configured for the subagent, it falls back purely to this hybrid vector/keyword search result.

2. Isolated Subagent for Reasoning & Direct File Reading:

   • The main chat orchestrator model calls search_knowledge_base.
   • Instead of returning massive raw chunks to the main chat context, an isolated KB Reader Subagent (runKbSubagent) receives the pre-fetched hybrid hits and Tier 1 document catalog.
   • Direct file reading is on-demand, not an uncontrolled default: The subagent has access to read_kb_document to read specific document slices or full files from Qdrant chunks by index range (getDocumentChunksByRange).
   • A context guard (guard-kb-context-budget.ts) actively prevents full-file dumps: if a document exceeds 75% of the model's remaining context window, the read is blocked and returns the document's structured Table of Contents (TOC), forcing targeted slice reads (chunkStartIndex to chunkEndIndex).

---

2. Does it use the graphs to explore related data?

Yes, extensively at both ingestion and query time:

1. Ingestion Time (PostgreSQL Graph Construction):

   • When documents are ingested in ingest-pipeline.ts, generate-document-relations.ts evaluates candidates in the knowledge base.
   • It prunes top candidates by shared entities/keywords and generates bidirectional typed relationship edges (references, supersedes, supplements, contradicts, depicts) stored in PostgreSQL table kb_document_relation.

2. Query Time (Graph Exploration):

   • During retrieval, explore-kb-graph.ts runs in parallel with hybrid search to load document metadata and immediate relational edges.
   • The internal subagent is equipped with the explore_kb_graph tool, allowing it to perform 1–3 hop undirected graph traversals (startDocId, depth, or query) to discover connected documents (e.g., finding what supersedes or supplements a specification).
   • Any discovered relationships are tracked in relatedDocuments and returned alongside the synthesized answer and UI citations.

---

3. What the Subagent Receives First (Without Reading Files)

When search_knowledge_base is called, runKbSubagent does a fast initial retrieval pass and seeds the subagent prompt with:

• Tier 1 Document Catalog: Summaries, document types, entity tags, and chunk counts for up to 15 relevant documents.
• Document Relationships: Known edges from the graph (e.g. doc A supersedes doc B).
• Initial Retrieval Hits: Top 5 chunks returned by hybrid search (Qdrant semantic vector + full-text search).

---

4. When the Subagent Decides to Call read_kb_document

The subagent enters an autonomous tool loop (up to 4 steps) and decides to invoke read_kb_document(documentId, chunkStartIndex?, chunkEndIndex?) when:

1. The initial 5 chunk snippets are fragmented or incomplete:
   • The question requires sequential reading (e.g. "Walk me through the full installation steps in guide X" or "What does section 3 of spec Y say?").
2. Catalog / Graph identifies a relevant document, but chunks were missed:
   • The subagent sees a relevant file or relationship in the catalog/graph that wasn't captured in the top-5 vector hits.
3. Synthesis requires reading surrounding context:
   • The subagent needs paragraphs or full sections before and after an isolated snippet to understand policies, contracts, or code definitions accurately.

---

5. How the Reading is Executed (Budget-Guarded)

When the subagent invokes read_kb_document:

• If the document fits comfortably (estimated tokens $\le 75%$ of model context window):
  • It pulls all ordered chunks from Qdrant via getDocumentChunksByRange.
• If the document is too large (e.g., a 50-page PDF on an 8k/16k context model):
  • The context guard (guardKbContextBudget) blocks the full read (safeToRead: false).
  • It returns the document's Table of Contents (TOC) with chunk ranges.
  • The subagent then calls read_kb_document again with targeted bounds (chunkStartIndex to chunkEndIndex) matching only the relevant section.