wasmcloud.github.io/arewefastyet

Static trend site for the wasmCloud bench pipeline. Four views:

• trends — one Chart.js timeline per (bench, group), overlaid with one line per param (e.g. p2 and p3).
• compare — pick two runs and see a per-row diff (Δ, %, significance). Deep-linkable via ?bench=<bench>&a=<runId>&b=<runId>.
• releases — semver-ordered timeline filtered to release tags.
• gpu — the gpubench rows (bench-tools gpu jsonl): tok/s for the vendor, webgpu-native and wasm steps per box with the gaps between them, the wasm guest's memory beside the model's size, and GPU memory over time for serverless wasmCloud turns. Trends and releases leave these rows out; ?bench=gpubench on trends opens this page.
• density — how many components one host keeps running. The record is the newest scale_density run in group density-max (the bench workflow's million_warm preset) on the selected host: its hero numbers, and per-step ramps for warm and scale-to-zero from its metric: "ramp" rows. A host without one falls back to the hand-run campaign in data/density-records.json, which also holds the curated walls (the ceiling each change moved). Then the latest published scale_density run per host at CI's defaults. Trends still charts the summary rows over commits; ramp rows appear only here.

This repo is just the site. Bench data is produced by the bench pipeline in wasmCloud/wasmCloud (see scripts/bench/) and served live via CloudFront in front of S3.

Architecture

   wasmCloud/wasmCloud (bench pipeline)              wasmCloud/arewefastyet (this repo)
   ─────────────────────────────────                 ─────────────────────────────────
                                                              │
       cargo bench  ──►  push-s3.sh                           │ git push to main
                            │                                 │
                ┌───────────┴────────────┐                    ▼
                ▼                        ▼            GitHub Pages
        per-run files            history.json         (static HTML/CSS/JS)
        (private)                (Cache-Control:              │
        runs/<date>/<sha>/...     max-age=60)                 │
                                       │                      │
                                       ▼                      ▼
                                CloudFront edge ◄─── browser fetches config.json,
                                (1 TB/mo free)               then DATA_URL
                                       │
                                       ▼
                                    S3 (private,
                                    OAC-only read on
                                    history.json)

The bench pipeline aggregates history.json server-side after each run and issues a CloudFront invalidation, so the site sees new data within seconds (without that, Cache-Control: max-age=60 would be the worst-case lag).

There's no AWS auth in this repo and no aggregation step — the deploy workflow just generates config.json with the live data URL and ships static assets to GitHub Pages.

Branding

The site uses the official wasmCloud logo (brand green, #00bc8e) for the header mark + favicon. Brand color is also applied to links and the active tab underline. Chart line colors stay distinct (cyan for p2, purple for p3) because they carry semantic meaning across charts and shouldn't blend with brand chrome.

Metric semantics (RPS vs latency)

Display unit per chart depends on the row's harness:

Criterion rows pick from criterion::Throughput:

criterion config	display	direction
Throughput::Elements(N) with N > 1	req/s (auto-scaled to Kreq/s, Mreq/s)	higher is better
Throughput::Bytes(N) with N > 0	B/s (auto-scaled)	higher is better
no throughput, or Elements(1)	time (auto-scaled ns / µs / ms / s)	lower is better

So http_throughput (declared Throughput::Elements(256) in the bench) displays as Kreq/s and an increase is shown green. cold_invocation and hot_invocation are time benches and an increase is shown red.

gungraun rows (metric: "Ir") display as instructions (K/M/G-suffixed); lower is better, no CI band.

One-time setup

AWS infra (bucket + OAC + CloudFront + IAM) is bootstrapped from the wasmCloud repo:

# in wasmCloud/wasmCloud
./scripts/bench/aws/setup-aws.sh \
  --bucket wasmcloud-benches \
  --region eu-central-1

That script prints the values for both repos. For this repo, only one is needed:

Setting	Type	Value
DATA_URL	repo var (Settings → Secrets and variables → Actions → Variables)	the printed https://dXXXX.cloudfront.net/history.json

Then:

1. Enable Pages — Settings → Pages → Source = "GitHub Actions".
2. DNS — at Cloudflare, add arewefastyet.wasmcloud.com → wasmcloud.github.io (Proxy disabled so Pages' Let's Encrypt cert can issue).
3. First deploy — Actions → deploy → Run workflow.

Local preview

The site falls back to data/history.sample.json when config.json isn't present, so plain python3 -m http.server Just Works:

python3 -m http.server 8000
# → open http://localhost:8000

To preview against live data, write a config.json:

echo '{"dataUrl":"https://dXXXX.cloudfront.net/history.json"}' > config.json
python3 -m http.server 8000

Data shape

Three row shapes coexist in history.json. All share the same envelope (bench, group, param, sha, ref, timestamp, …); the metric fields differ.

Criterion (metric: "mean_ns"):

{
  "bench": "http_invoke",
  "group": "cold_invocation",
  "param": "p2",
  "sha": "d8f795fe2eb9...",
  "short_sha": "d8f795fe2eb9",
  "ref": "main",
  "run_id": "1234567890",
  "run_attempt": "1",
  "timestamp": "2026-05-07T17:21:20Z",
  "host": "wasmcloud-bench-01",
  "host_label": "hetzner",
  "kernel": "6.8.0-100-generic",
  "cpus_online": 6,
  "metric": "mean_ns",
  "value": 78667741.74,
  "throughput": {"Elements": 256},
  "mean_ns": 78667741.74,
  "median_ns": 77841173.62,
  "std_dev_ns": 1770558.62,
  "ci_low_ns": 77764094.17,
  "ci_high_ns": 79841655.92
}

gungraun (metric: "Ir"):

{
  "bench": "gungraun",
  "group": "http",
  "param": "hot_invocation.p2",
  "metric": "Ir",
  "value": 12345678
}

The bench-side dedup key is (sha, bench, group, param, run_attempt, metric, host_label). HISTORY_MAX_AGE_DAYS (set in the bench-side aggregator) caps the file at one year by default to keep it bounded as runs accumulate.

Hosts

host_label is the runner label of the host a row ran on: hetzner for the dedicated host, anything else for a short-lived one (see the bench runbook). A row without it is a hetzner row.

The trends and releases pages chart one host at a time, picked with the host filter (?host=, default hetzner), since numbers from different hardware don't belong on one line. The compare page lists every host's runs, shows each run's host, and warns when A and B ran on different hosts.

Significance heuristic (compare view)

Criterion rows are flagged as significant only if both:

1. |%| ≥ 1 (the change is more than the noise floor we care about), and
2. The 95 % CIs of A and B do not overlap.

Otherwise the change is within criterion's variance and shown as flat (grey, with a · in the sig column). Conservative on purpose — better to under-flag than to call random noise a "regression".

gungraun rows have no CI in our schema. They fall back to a stricter |%| ≥ 2 threshold; below that, flat. gungraun is deterministic, so the threshold is mostly there to guard against floating-point artifacts at the bottom of the dynamic range — anything ≥ 2 % is real signal.

Files

File	Purpose
index.html	Trends page shell
compare.html	Compare-runs page shell
style.css	Shared styles; light/dark via prefers-color-scheme, mobile-friendly
script.js	Trends logic + loadHistory() (fetches config.json then dataUrl, with sample fallback). Exports helpers on window.AWFY.
compare.js	Compare-runs logic; consumes window.AWFY (IIFE-wrapped).
releases.html, releases.js	Releases page: filters history to semver tags, x-axis is the (semver-ordered) tag, not time.
gpu.html, gpu.js	GPU page: renders bench: "gpubench" rows per box; consumes window.AWFY.
density.html, density.js	Density page: the newest density-max run per host (else the record in data/density-records.json), the curated walls, plus the latest scale_density run per host; consumes window.AWFY.
assets/wasmcloud-logo.svg	Official wasmCloud hex-W mark (#00bc8e), used for header logo + favicon.
CNAME	Pages reads this for the custom domain.
data/history.sample.json	Synthetic 8-commit sample (clearly fake SHAs) — local-dev fallback. Includes a few aws-c7i host rows, and synthetic gpubench rows (run_id: sample-gpu-*) for a metal and a cuda box over three commits, plus one smoke run.
.github/workflows/deploy.yml	On push/dispatch: write config.json from vars.DATA_URL → deploy-pages.
.gitignore	Excludes the runtime-generated config.json.

Out of scope (not built)

• Cross-repo trigger from wasmCloud's bench workflow into this one. Not needed: the site fetches at page-load time, so new bench runs surface within ~1 minute (Cache-Control max-age + CloudFront invalidation) without redeploying.
• Annotations — overlay markers on commits where a perf-relevant change landed. Planned next.
• Versions view — semver-ordered x-axis filtered to release tags. Planned next.
• Custom domain on CloudFront (data.arewefastyet.wasmcloud.com). Optional polish: requires an ACM cert in us-east-1 validated via Cloudflare DNS, then attached to the distribution.