[mbeps]

1. High-Level Overview

Goal

Add universal thinking/reasoning support for any generic OpenAI-compatible /v1/chat/completions provider without hardcoding vendor URLs or installing vendor-specific SDKs. The design uses a 2-Layer Protocol-Trait Architecture (inspired by VS Code's chatLanguageModels.json) plus a per-turn Reasoning Effort Selector (Off, Low, Medium, High) in the chat toolbar:

1. Layer 1 — Universal Response Extraction (100% Automatic, Zero Config):
   • Structured Delta Field Normalizer: Captures delta.reasoning_content (DeepSeek, Qwen, vLLM, Kimi, llama.cpp), delta.reasoning (OpenRouter), and delta.reasoning_details (Array<{ text?: string; summary?: string }>) from /v1/chat/completions responses before @ai-sdk/openai's strict Zod schema can strip them.
   • Multi-Tag Streaming State Machine: Strips <think>...</think>, <thinking>...</thinking>, <thought>...</thought>, <reasoning>...</reasoning>, [THINK]...[/THINK], and ◁think▷...◁/think▷ blocks from streaming text-delta chunks (handling arbitrary chunk splits and mid-thought stream aborts) and emits them as reasoning-delta events into the existing ReasoningTraceStep UI.
2. Layer 2 — Model-Level Reasoning Configuration & Effort Selector (Configurable Protocol Traits):
   • Extends ai_model (drizzle/schemas/ai-model-schema.ts) and the import/export provider registry (schemas/providers/provider-registry.ts) with two dedicated columns alongside the existing cap_reasoning boolean:
     • reasoning_effort_format: "none" | "openai" | "openrouter" | "anthropic" (default "none").
     • model_options: nullable JSON string (Record<string, unknown>) for custom pass-through body parameters (e.g. {"enable_thinking": true}).
   • Adds a Reasoning Effort Selector (Off, Low, Medium, High, defaulting to Medium and persisted in turn metadata) in ChatInputToolbar when the active model has capReasoning === true and reasoningEffortFormat !== "none".

Architecture & Data Flow

flowchart LR
  subgraph Client["Frontend (Next.js / React 19)"]
    Toolbar["ChatInputToolbar\n(Effort: Off | Low | Medium | High)"]
    Route["POST /api/chat"]
    Timeline["ResponseTimeline &\nReasoningTraceStep"]
  end

  subgraph Inngest["Inngest Durable Chat Pipeline"]
    Orch["generateChatResponse\n(chat-response.ts)"]
    Resolve["buildResolvedProvider &\nbuildSdkProvider"]
    Consumer["consumeModelStream &\ncreateInlineReasoningParser"]
  end

  subgraph Transport["Custom Fetch Wrapper (build-sdk-provider.ts)"]
    ReqTransform["Request Body Transformer\n(reasoning_effort / reasoning / thinking + modelOptions)"]
    SseTransform["SSE & JSON Response Normalizer\n(delta.reasoning* -> <think>...</think>)"]
  end

  API["Generic OpenAI-Compatible Endpoint\n(/v1/chat/completions)"]

  Toolbar -->|"reasoningEffort"| Route
  Route -->|"chat/response.generate"| Orch
  Orch --> Resolve
  Resolve --> ReqTransform
  ReqTransform -->|"HTTP POST"| API
  API -->|"SSE / JSON"| SseTransform
  SseTransform -->|"@ai-sdk/openai stream"| Consumer
  Consumer -->|"reasoning-delta / text-delta"| Timeline

Loading

Global Constraints & Invariants

• Zero Provider URL Hardcoding: Outgoing request formatting is driven purely by the model's reasoningEffortFormat ("none" | "openai" | "openrouter" | "anthropic") and modelOptions. Incoming extraction is always active for all models.
• Why Custom fetch SSE Normalization Is Required for @ai-sdk/openai v4:
  • In @ai-sdk/openai (node_modules/@ai-sdk/openai/dist/internal/index.js), OpenAIChatLanguageModel.doStream() pipes the HTTP response through throwIfOpenAIStreamErrorBeforeOutput (which buffers the entire stream in memory until isOpenAIChatOutputChunk sees delta.content.length > 0, delta.tool_calls, or delta.refusal) and validates every SSE event with openaiChatChunkSchema (which only allows role, content, refusal, tool_calls, annotations on delta and strips reasoning_content, reasoning, and reasoning_details).
  • Normalizing delta.reasoning_content, delta.reasoning, and delta.reasoning_details into <think>...</think> inside delta.content at the custom fetch SSE layer unblocks isOpenAIChatOutputChunk on the very first reasoning token and preserves structured reasoning across @ai-sdk/openai's Zod parser, while createInlineReasoningParser in consumeModelStream extracts all inline tag formats into reasoning-delta.
• Zod v4 .partial() Invariant: Per .agents/learnings.md (Refactoring & Code Cleanup #28), updateModelSchema uses .partial(). Fields with .default() inside .partial() replace undefined with the default value during partial updates. Therefore, createModelSchema applies defaults on creation, while updateModelSchema keeps reasoningEffortFormat and modelOptions optional without inner .default().

---

2. Alternatives Evaluated

Approach 1: 2-Layer Protocol-Trait Pipeline — Custom fetch Request/SSE Normalizer + Streaming Multi-Tag Parser + Dedicated ai_model Columns (Recommended)

• Description:
  • Outgoing: buildSdkProvider supplies a custom fetch that transforms /chat/completions POST bodies using reasoningEffortFormat + reasoningEffort and merges modelOptions.
  • Incoming: The custom fetch normalizes SSE delta.reasoning_content, delta.reasoning, and delta.reasoning_details into <think>...</think>, and consumeModelStream pipes text-delta chunks through createInlineReasoningParser (handling <think>, <thinking>, <thought>, <reasoning>, [THINK], ◁think▷) + checks raw structured delta fields if passed directly.
  • Storage: Dedicated reasoning_effort_format and model_options columns on ai_model.
• Pros:
  • Works with 100% of OpenAI-compatible endpoints (OpenAI, DeepSeek, Qwen, OpenRouter, LiteLLM, vLLM, Ollama, LocalAI, llama.cpp).
  • Bypasses @ai-sdk/openai's early-error stream buffering freeze and Zod field stripping without patching node_modules.
  • Satisfies 3NF database design and matches ai_model's existing flat capability columns.
• Cons:
  • Requires a lightweight SSE line transform stream in buildSdkProvider and a tag state machine in lib/chat/stream/reasoning-stream-parser.ts.

Approach 2: Chaining Built-In AI SDK extractReasoningMiddleware

• Description: Wrap sdkProvider.chat(modelId) with 6 nested extractReasoningMiddleware({ tagName }) calls from "ai".
• Pros: Uses built-in AI SDK middleware for tag extraction.
• Cons:
  • Completely fails on structured delta fields (delta.reasoning_content, delta.reasoning, delta.reasoning_details) because @ai-sdk/openai strips them before wrapStream runs.
  • Chains 6 separate TransformStream instances per request, each delaying text-start and buffering independently.
  • Does not solve outgoing request body activation (reasoning_effort, reasoning.effort, thinking.budget_tokens, modelOptions).

Approach 3: Vendor-Specific AI SDK Packages (@ai-sdk/deepseek, @openrouter/ai-sdk-provider, @ai-sdk/anthropic)

• Description: Install separate provider SDKs and switch based on baseUrl or provider type.
• Pros: Native SDK options per vendor.
• Cons:
  • Violates the project requirement of a generic OpenAI-compatible client with zero provider-specific hardcoding.
  • Breaks self-hosted proxies (LiteLLM, vLLM, Ollama, LocalAI) that expose custom URLs.

---

3. Edge Cases, Failure Modes & Mitigations

Edge Case / Failure Mode	Mitigation
Split tags across SSE chunks (e.g., chunk 1 ends with "<thi", chunk 2 starts with "nk>Hello")	createInlineReasoningParser checks if the tail of the current buffer is a prefix of any valid opening/closing tag (getPotentialTagPrefixIndex). It holds only the matching tail in buffer until the next chunk confirms or disproves the tag.
Unclosed <think> tag on user Stop or max_tokens	When the stream ends or aborts while inside a thinking block, parser.flush() emits all remaining buffered text as "reasoning" so partial thoughts appear in ReasoningTraceStep and never leak into the assistant's markdown answer.
Tag lookalikes inside markdown code blocks (e.g. user asks "explain <think> tags" and model writes ```xml\n<think>...</think>\n```)	Track markdown code fence state (```) when in normal text mode; do not enter thinking state for tags that appear inside fenced code blocks or inline backticks.
@ai-sdk/openai early-error stream buffering & Zod stripping	createReasoningNormalizingFetch transforms SSE delta.reasoning_content, delta.reasoning, and delta.reasoning_details into <think>...</think> inside delta.content before @ai-sdk/openai reads the stream, satisfying isOpenAIChatOutputChunk immediately on chunk #1.
reasoningEffort: "off" semantics across formats	- "openai": sets reasoning_effort: "none" (or omits if modelOptions overrides it).Wait: many OpenAI-compatible servers (vLLM, DeepSeek, Groq, older OpenAI o1/o3) reject reasoning_effort: "none" with HTTP 400 and only accept "low" | "medium" | "high", whereas omitting reasoning_effort (or setting reasoning: { enabled: false } for OpenRouter / thinking: { type: "disabled" } for Anthropic) is universally safe! Let's omit reasoning_effort when "off" in "openai" format (or allow modelOptions override), send reasoning: { enabled: false } for "openrouter", and send thinking: { type: "disabled" } for "anthropic".
Both structured delta.reasoning_content AND inline <think> returned by a proxy	If the normalized delta.reasoning_content itself begins with <think>, createInlineReasoningParser avoids nested/duplicate <think> tags and strips inner tags cleanly.
Malformed JSON in modelOptions in UI or DB	ModelFormDialog validates that modelOptions parses as a plain JSON object (!Array.isArray(parsed)) before form submission; buildResolvedProvider wraps JSON.parse in try/catch and falls back to null with a warning log.