Data Intelligence Pipeline

An 11-node LangGraph multi-agent system that automates the full business intelligence loop — from raw CRM data ingestion through to auditable automated actions via n8n webhooks.

Pipeline Architecture

[1] Ingestion → [2] Cleaning → [3] Search → [4] Insight → [5] Consensus
[5] Consensus → [6] Healing → [7] Simulation → [8] Decision → [9] Briefing
[8] Decision → [10] Approve → [11] Action → n8n → Slack / Email

Agents

Agent	Role
Ingestion	CSV / API / Supabase / CRM ingestion with schema validation
Cleaning	Null removal, deduplication, outlier detection (IQR / Z-score)
Search	Market context enrichment via Tavily
Insight	4 KPIs, Pearson correlation, anomaly detection, LLM narrative
Consensus	LLM report comparison with semantic scoring
Healing	Prompt optimisation suggestions based on pipeline feedback
Simulation	What-if KPI modelling with price elasticity
Decision	KPI threshold evaluation and alert routing
Briefing	Voice MP3 executive summary via gTTS
Approve	Conditional gate — action only triggered on alert
Action	n8n webhook with retry logic and full audit log

Quick Start

pip install -r requirements.txt

cp .env.example .env   # all integrations are optional — runs offline without API keys

python main.py --source csv --file data/sample/sales.csv

streamlit run dashboard/app.py   # multi-page dashboard

Tests

python -m pytest tests/ -v
# 19/19 tests pass

Environment Variables

Variable	Required	Description
GROQ_API_KEY	Optional	Groq LLM for narrative generation
HF_TOKEN	Optional	HuggingFace secondary LLM (LLM Arena)
SUPABASE_URL	Optional	Supabase PostgreSQL data source
SUPABASE_KEY	Optional	Supabase anon/service key
N8N_WEBHOOK_URL	Optional	n8n automation webhook
DATABASE_URL	Optional	Defaults to SQLite locally

Project Structure

├── agents/          # 11 agent implementations (BaseAgent + subclasses)
├── api/             # FastAPI server — 14 REST endpoints
├── config/          # Pydantic settings with .env support
├── dashboard/       # Streamlit multi-page dashboard (5 pages)
├── data/            # Sample CRM dataset + generator
├── database/        # SQLAlchemy ORM — 10-table governance schema
├── src/             # Orchestrator (LangGraph), state, logger
├── tests/           # 19 pytest tests (agents, tools, orchestrator)
├── tools/           # Ingestion, cleaning, analytics, and action tools
├── workflows/       # n8n webhook client and server
├── main.py          # CLI entry point
└── demo.py          # Formatted demo with results output

Tech Stack

Layer	Technology
Orchestration	LangGraph StateGraph
LLM	Groq (llama-3) + HuggingFace (LLM Arena)
Database	SQLAlchemy ORM — SQLite (dev) / PostgreSQL (prod)
API	FastAPI + Uvicorn
Dashboard	Streamlit + Plotly
Automation	n8n webhooks
Data	pandas, numpy, scipy, scikit-learn
Testing	pytest — 19 tests, 100% pass rate