BOTBOSS — How It Works

100% Local. 100% Free. Zero Cloud Dependencies.

BOTBOSS is a two-stage content audit pipeline: crawl → analyze. Everything runs on your machine using open-source tools and a local LLM.

---

Architecture Overview

graph TD
    U["📥 urls.txt"] --> C1

    subgraph C1["Stage 1: Crawler"]
        A1[Chromium] --> A2[Extract a11y+DOM]
        A2 --> A3[Screenshot]
        A3 --> A4[Dedup]
        A4 --> A5[JSON→TOON]
    end

    C1 --> MID["📦 output_*/<br/>.json .toon .jpg"]

    MID --> C2

    subgraph C2["Stage 2: Analyst"]
        B1[Parse .toon] --> B2[Classify]
        B2 --> B3[Build Prompt]
        B3 --> B4[LLM Client]
        B4 --> B5[Write .md]
    end

    B4 <--> LLM["🧠 BeeLLaMA<br/>localhost:8082"]

    C2 --> OUT["📊 reports/"]
    OUT --> R1[Per-page audits]
    OUT --> R2[EXECUTIVE_SUMMARY.md]

Loading

---

Stage 1: BOTBOSS-unified.js (The Crawler)

What It Does

Launches a real Chromium browser via Patchright and extracts everything from each page:

Capability	Detail
Hybrid Extraction	Combines ariaSnapshot() (accessibility tree) + direct DOM queries
Content Blocks	Detects sections, cards, articles with bounding-box grouping
Interactive Elements	Links, buttons, forms, inputs, selects, dialogs, accordions
Media	Images (with alt-text source tracking), videos, audio, iframes
Structured Data	JSON-LD, breadcrumbs, tables, lists (HTML + CSS-rendered)
Screenshots	Full-page JPG after scroll + aria expansion
Deduplication	Content fingerprinting + Jaccard similarity (threshold: 0.90)
Bot Challenge Handling	Auto-waits for Cloudflare/Recaptcha/Turnstile
Checkpoint Resume	Saves progress → restart mid-crawl after crash

Extraction Pipeline

sequenceDiagram
    participant URL as Seed URL
    participant CR as Chromium Browser
    participant A11Y as ariaSnapshot()
    participant DOM as DOM Evaluation
    participant SC as Screenshot
    participant DD as Dedup Engine
    participant TOON as JSON → TOON

    URL->>CR: Navigate + Wait
    CR->>CR: Auto-detect bot challenges
    CR->>CR: Human-like mouse + scroll
    CR->>SC: Full-page screenshot
    CR->>CR: Expand all aria/details
    CR->>A11Y: Accessibility tree
    CR->>DOM: Deep DOM extraction
    A11Y->>DD: Merge + deduplicate
    DOM->>DD: Merge + deduplicate
    DD->>TOON: Convert to compact format
    TOON->>TOON: Save .json + .toon

Loading

CLI Usage

# Basic crawl from urls.txt
node BOTBOSS-unified.js

# Custom depth, max URLs, scroll behavior
node BOTBOSS-unified.js -R 3 -m 200 -scroll 2

# Direct URL
node BOTBOSS-unified.js https://example.com/page

# Clean old outputs first
node BOTBOSS-unified.js --clean

# Interactive mode (single page, manual control)
node BOTBOSS-unified.js --single

Output Structure

output_2025-01-15/
├── www_example_com_2025-01-15_s1.json    ← Full extraction
├── www_example_com_2025-01-15_s1.toon    ← Compact YAML-like
├── www_example_com_2025-01-15_s1.jpg     ← Screenshot
├── www_example_com_page2_2025-01-15_s1.json
├── www_example_com_page2_2025-01-15_s1.toon
└── www_example_com_page2_2025-01-15_s1.jpg

---

Stage 2: BOTBOSS-content-strategist.js (The Analyst)

What It Does

Reads .toon files from Stage 1 output, classifies each page, sends analysis prompts to a local LLM, and generates professional markdown audit reports.

Processing Pipeline

graph TD
    F[Find .toon files] --> C{Any .toon?}
    C -->|No| J[Convert .json→.toon]
    J --> TP
    C -->|Yes| TP[Parse .toon]

    TP --> PC[Classify Page]
    PC -->|≥300w, ≥3h, >5p| H["🔴 HIGH"]
    PC -->|≥100w| M["🟡 MEDIUM"]
    PC -->|else| L["🟢 LOW"]

    H --> PB[Build Prompt]
    M --> PB
    L --> PB

    PB --> LC["LLM Client<br/>localhost:8082"]
    LC --> RG[Write per-page .md]
    RG --> ES[Synthesis Prompt]
    ES --> EX[EXECUTIVE_SUMMARY.md]

Loading

Classification Logic

flowchart TD
    P[Page Data] --> C{words ≥ 300<br/>headings ≥ 3<br/>paragraphs > 5}
    C -->|Yes| H["🔴 HIGH<br/>Deep analysis"]
    C -->|No| M{words ≥ 100}
    M -->|Yes| MD["🟡 MEDIUM<br/>Standard analysis"]
    M -->|No| L["🟢 LOW<br/>Brief analysis"]

Loading

Each priority tier triggers a different prompt depth:

Priority	Prompt Depth	Output Scope
HIGH	Deep	Full metrics, SEO, content gaps, conversion funnel, a11y, 5-10 recommendations
MEDIUM	Standard	6-section framework, 3-5 issues, 3-5 recommendations
LOW	Brief	Max 400 words, 1-2 issues, 1 recommendation

CLI Usage

# Auto-detect latest output_* directory
node BOTBOSS-content-strategist.js

# Specify input/output
node BOTBOSS-content-strategist.js -i output_2025-01-15 -o reports

# Limit pages, custom model/host
node BOTBOSS-content-strategist.js -m 50 --model local-llama --host http://127.0.0.1:8082

# Dry run (build prompts, skip LLM calls)
node BOTBOSS-content-strategist.js --dry-run

Report Output

reports/
├── EXECUTIVE_SUMMARY.md          ← Cross-page synthesis
├── page-www_uhc_com_communityplan.md   ← Per-page audit
├── page-www_uhc_com_mediicaid.md
└── ...

Each per-page report includes:

• Page overview table (URL, title, word count, element counts)
• Content structure analysis with Mermaid diagrams
• Navigation & link hierarchy
• Accessibility assessment (alt-text deficit scoring)
• Media strategy review
• Conversion signal analysis
• Priority matrix with recommendations

---

The Local LLM Connection

sequenceDiagram
    participant CS as Content Strategist
    participant LB as Local BeeLLaMA<br/>(localhost:8082)
    participant DISK as Disk

    CS->>CS: Build prompt from .toon data
    CS->>LB: POST /v1/chat/completions
    Note over CS,LB: OpenAI-compatible API<br/>No internet required
    LB->>LB: Process locally on GPU/CPU
    LB-->>CS: Streaming JSON response
    CS->>DISK: Write .md report

Loading

Key point: The LLM client uses the standard OpenAI-compatible API format (/v1/chat/completions) but connects to localhost:8082 — your own BeeLLaMA instance. No data leaves your machine. No API keys paid to third parties.

Environment Variables

Variable	Default	Purpose
STRATEGIST_LLM_HOST	http://127.0.0.1:8082	LLM endpoint
STRATEGIST_LLM_KEY	REVOKEDsk-lm-iUlcY0bh:V0ilgsVjjXSspXqyS1RX	API key (local)
STRATEGIST_LLM_MODEL	local-llama	Model name
MAX_PAGES	500	Max pages to analyze
MAX_TOKENS	8000	Max tokens per LLM response
TEMPERATURE	0.3	Creativity (lower = more deterministic)

---

End-to-End Workflow

flowchart TB
  subgraph S1["Stage 1: Crawl"]
    A1[Launch Chromium] --> A2[Navigate and Wait]
    A2 --> A3[Extract a11y and DOM]
    A3 --> A4[Screenshot]
    A4 --> A5[Dedup and Save]
  end
  subgraph S2["Stage 2: Analyze"]
    B1[Parse .toon files] --> B2[Classify priority]
    B2 --> B3[Send to LLM]
    B3 --> B4[Generate reports]
  end
  subgraph S3["Stage 3: Review"]
    C1[Read reports] --> C2[Act on findings]
  end
  S1 --> S2 --> S3

Loading

---

Why Everything Is Free

Component	Tool	License/Cost
Browser automation	Patchright (Chromium fork)	MIT / Free
Node.js runtime	Node.js	MIT / Free
JSON compression	toon-json-converter	Free
Local LLM server	BeeLLaMA	Free
LLM model weights	Llama / open models	Free (research/commercial)
Report generation	Built-in Markdown	Free
Total cost		$0

No subscriptions. No per-token billing. No rate limits. No data sent to third parties.

---

Dependencies

npm install patchright toon-json-converter

Plus a running local LLM server (BeeLLaMA recommended on port 8082).

---

Quick Start

# 1. Create your URL list
echo "https://www.uhc.com/communityplan" > urls.txt

# 2. Crawl
node BOTBOSS-unified.js -R 2 -m 100

# 3. Analyze
node BOTBOSS-content-strategist.js

# 4. Review reports/

∴ Crawl locally → Analyze locally → Own your data completely.