DataReliant is a production-grade, research-oriented platform for autonomous data quality monitoring, statistical and temporal anomaly detection, DAG-based pipeline lineage tracing, alert compression, and explainable root-cause intelligence.
It is NOT a toy dashboard, simple CRUD application, generic EDA tool, or commercial LLM wrapper. All anomaly detection, alert compression, and root-cause analysis run with zero external cloud LLM dependencies, guaranteeing deterministic reproducibility, sub-15ms inference latency, zero hallucinations, and complete data privacy.
- ⚡ Ultra-Fast Polars Engine: Vectorized multi-threaded profiling and validation processing over 1.7 million rows/second on a single node without Spark or JVM overhead.
- 🎯 Explainable Root-Cause Intelligence: Multi-signal evidence weighting across 6 categories (Schema: 0.25, Lineage Graph: 0.20, Statistical: 0.20, Temporal: 0.15, Propagation: 0.10, Historical: 0.10) achieving 92.3% Top-3 Accuracy across 13 failure scenarios.
- 🗜️ Up to 6.0x Alert Compression: NetworkX DAG lineage topology and temporal clustering compress cascading pipeline failures into isolated, actionable incidents.
- 🧪 Failure Injection Laboratory: 13 controlled synthetic failure scenarios with known ground truth for empirical reproducibility.
- 🖥️ Full-Featured React Dashboard: 11 dedicated views with custom vanilla CSS dark glassmorphic design, interactive SVG DAG lineage visualizer, and live benchmark lab.
- ☁️ Local-First & Multi-Cloud Ready: Completely local-first for offline development; decoupled adapters for Google Cloud Storage, BigQuery, and Terraform Cloud Run.
┌──────────────────────────────────────────────┐
│ DATA SOURCES & SENSORS │
│ (CSV, Parquet, PostgreSQL, BigQuery, GCS) │
└──────────────────────┬───────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ HIGH-THROUGHPUT PROFILER & RULES │
│ (Polars Vectorized Ingestion Engine) │
└──────────────────────┬───────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ MULTI-SIGNAL ANOMALY DETECTORS │
│ Point: Z-Score, IQR, EWMA, Isolation Forest│
│ Distribution: KS-Test, PSI, Jensen-Shannon │
│ Temporal: Trend, Seasonality, Change-Point │
└──────────────────────┬───────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ LINEAGE & INCIDENT CORRELATION │
│ (NetworkX DAG Traversal, Alert Squeezer) │
└──────────────────────┬───────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ ROOT-CAUSE INTELLIGENCE ENGINE │
│ - Deterministic 6-Signal Evidence Ranker │
│ - Supervised ML (LogReg, RF, XGBoost) │
│ - Cosine Similarity Historical Matching │
└──────────────────────┬───────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ EXPLANATION & FRONTEND │
│ - Rule-based / Local LLM Playbooks │
│ - FastAPI Async REST API │
│ - React Glassmorphic 11-View Dashboard │
└──────────────────────────────────────────────┘
Evaluated across 13 controlled failure injection scenarios with fixed random seeds (seed=42):
| Evaluation Metric | Baseline Heuristic | Deterministic Engine (Ours) | Supervised ML (RF) |
|---|---|---|---|
| Detection Recall | 100.0% | 100.0% | 100.0% |
| Top-1 Root-Cause Accuracy | 69.2% | 76.9% | 84.6% |
| Top-3 Root-Cause Accuracy | 69.2% | 92.3% | 92.3% |
| Mean Alert Compression | 1.00x | 1.23x (up to 6.0x) | 1.23x |
| Mean Evaluation Latency | 6.2 ms | 5.1 ms | 12.9 ms |
Full scientific report available at experiments/results/RESEARCH_BENCHMARKS_REPORT.md.
DataReliant includes a built-in failure injection laboratory with known ground truth:
missing_rows: Massive row count drops simulating upstream extraction cut-offs.duplicate_rows: Double-ingestion retry bug creating duplicate records.null_spike: Producer schema breakage injecting unexpected nulls.schema_addition: Unannounced metadata column appended upstream.schema_removal: Breaking schema deletion of a critical production column.datatype_change: Numeric or datetime column cast into strings.range_violation: Outlier multipliers (20x) corrupting continuous domains.categorical_corruption: Illegal discrete codes violating enum domain constraints.distribution_shift: Mean and variance shift detected via KS-test / PSI.timezone_shift: Unhandled UTC timezone offset gap.late_arrival: Delayed partition records backdated by multiple weeks.upstream_failure: Truncation of parent table breaking downstream foreign keys.mixed_failure: Compound multi-signal anomaly (nulls + categories + schema drift).
# Clone the repository
git clone https://github.com/datareliant/datareliant.git
cd datareliant
# Create virtual environment and install
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[all,test]"# Generate 1,000 transaction records with relational integrity
datareliant generate --rows 1000 --output-dir datasets/sample# Run benchmark on a specific failure scenario
datareliant evaluate --scenario null_spike --method engine
# Run benchmark on all 13 failure scenarios
datareliant evaluate --scenario all --method engine
# Compare Heuristic vs Deterministic vs ML methods
datareliant evaluate --method comparedatareliant serve --host 0.0.0.0 --port 8000 --reload
# API docs available at: http://localhost:8000/docscd apps/web
npm install
npm run dev
# Dashboard available at: http://localhost:5173Design decisions and scientific trade-offs are documented in docs/decisions/:
ADR-0001: Polars & PyArrow for High-Throughput Vectorized IngestionADR-0002: Dual Deterministic + Supervised ML for Explainable Root-Cause AnalysisADR-0003: NetworkX Directed Graph Engine for In-Memory Lineage & Blast RadiusADR-0004: Decoupled Multi-Cloud Adapters and Local-First ArchitectureADR-0005: Zero External Cloud LLM Dependency for Reliability
# Run all unit and integration tests (125 tests passing)
pytest tests/unit/ -v
# Run with test coverage
pytest tests/unit/ -v --cov=datareliant --cov-report=termThis project is licensed under the MIT License - see the LICENSE file for details.