Skip to content

benchmarks(errors): measure Citry 1.0.0 (54 to 56 of 61 before the app runs) - #740

Merged
JuroOravec merged 2 commits into
mainfrom
benchmarks/errors-citry-1-0-0
Nov 16, 2026
Merged

JuroOravec merged 2 commits into
mainfrom
benchmarks/errors-citry-1-0-0

Conversation

@JuroOravec

Copy link
Copy Markdown
Collaborator

Measures Citry's row of the error-prevention benchmark again on the released Citry 1.0.0, so the published number is a Citry 1.0.0 result.

Result

Before the app runs Errors Warnings Runtime Missed n/a
Old: Citry built from commit bf935bc, before 0.7.0 (run 2026-10-21) 54 of 61 47 7 2 5 2
New: Citry 1.0.0 from PyPI (run 20261116T192936Z) 56 of 61 47 9 1 4 2

Two mistakes have a different outcome:

  • m16, misspelled CSS class: missed before, now a warning in the editor and from citry check (citry.css.misspelled-class, which suggests board).
  • m50, misspelled inject key: before, it was reported only when the page failed to render. Now citry check and the editor warn about it (citry.template.unknown-inject-key).

Every other Citry cell is the same. The other eleven frameworks keep their 2026-10-21 results byte for byte. Since that run, their cases, catalogue and tools are unchanged except for a newer certifi pin in the ReactPy and Reflex apps. The Benchmarks page states both dates, the same way the speed section does.

Ports changed

None. The baseline and all 61 Citry cases worked on 1.0.0 as written, and the clean app reported no problems. A manual check also confirmed on 1.0.0 that m23's type error is still reported when the card has no template_data(), as the framework notes say.

What changed

  • Citry app (benchmarks/errors/frameworks/citry/): it installs citry==1.0.0 and citry-lsp==1.0.0 from PyPI, and with them citry-core 2.1.0 (citry's pin) and ty 0.0.101 (citry-lsp's). Before, it built citry and citry-lsp from packages/py, which needed override-dependencies and [tool.uv.sources]; both are removed. The setup step no longer reinstalls packages from source, and the notes now name the 1.0.0 versions. Like docs: publish Citry 1.0.0 speed numbers, keep the other frameworks #732 and benchmarks: name the Citry 1.0.0 and citry-core 2.1.0 wheels in citry-local-wheels #735 for the speed benchmark, the app pins the release instead of building the repository. The test that kept the old citry-core override in step with citry's pin is removed with the override.
  • Runner: citry-errors report <run> --keep-from <earlier report> keeps every framework the run did not measure from an earlier report. matrix.md, report.html and results.json (kept) name the run each kept group comes from: its commit, inputs tree, uncommitted changes and any package it built from this repository. The report stops in two cases: the run left out a mistake of a framework it measured, or a kept framework has no result for a catalogue mistake. Tests are added in tests/test_parallel_runs.py.
  • Published result: the new report is benchmarks/errors/results/2026-11-16/, and results/2026-10-21/ stays. docs_site/data/error_prevention.json, the README chart and its PNGs were all regenerated with the existing scripts. VERSION_NOTES has a new entry, "Citry 1.0.0.".
  • Docs: the "Catching errors" section of docs_site/content/about/benchmarks.md has the new counts and lists, plus a "Dates and versions" note that replaces "Citry's exact build". benchmarks/errors/README.md updates its install, --keep-from, latest-result and m50 notes.

Time

  • Citry-only run from the committed inputs (--no-cache): 79 seconds, plus about 2 seconds for the report.
  • Earlier runs of the same inputs took 105 to 107 seconds and gave the same cells.
  • The whole task, including the runner option, the review fixes and the docs, took about 1.5 hours.

Checks

  • Benchmark tests: pytest benchmarks/errors/tests, 196 passed. ruff check and ruff format --check on benchmarks/errors pass.
  • Docs tests: test_benchmark_images, test_error_benchmark and test_benchmark_chart, 105 passed.
  • python -m docs_site build-check: 0 errors, 0 warnings.
  • readme_benchmark_charts and published_readme_links validators: no findings.
  • An independent reviewer checked that the kept records are identical, that the report and docs data regenerate exactly, and every Citry bullet against the evidence. Its findings are fixed: stale docstrings in work.py, kept provenance trimmed to the kept frameworks, the completeness checks, and prose fixes.

Versioning and changelog

Nothing under crates/ or packages/ changes. This touches the benchmark runner, its data, the docs and the README chart block only, so the 1.x rules do not apply and no changelog entry is needed.

Left out

docs_site/data/error_prevention.json keeps a single run id and does not carry the kept origins. Nothing reads them there, and the page prose states both dates.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KUhrHZV4bm5kkntEjqnfSi

JuroOravec and others added 2 commits November 16, 2026 20:29
The Citry app now installs citry 1.0.0, citry-lsp 1.0.0 and citry-core
2.1.0 from PyPI, like every other app installs its framework, instead of
building citry and citry-lsp from this repository's sources. The test
that kept the app's citry-core override in step with citry's pin goes
with the override.

`citry-errors report --keep-from <earlier report>` takes the results of
every framework a run did not measure from an earlier report, and the
report names the run each framework's results come from (its commit,
inputs tree, and any package it built from this repository), so one
framework can be measured again without rerunning the other eleven. It
keeps whole frameworks, and stops when the run left out a mistake of a
framework it measured or a kept framework has no result for a mistake.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KUhrHZV4bm5kkntEjqnfSi
…re the app runs)

Citry 1.0.0, measured on 2026-11-16, reports 56 of the 61 mistakes that
apply to it before the app runs (47 errors, 9 warnings). Two mistakes
moved: a misspelled CSS class (m16) was missed and is now a warning, and
a misspelled inject key (m50) failed only when the page rendered and is
now a warning from `citry check` and the editor.

The report in benchmarks/errors/results/2026-11-16 keeps the other
eleven frameworks' results from the 2026-10-21 run and names that run.
The docs data, the README chart and the Benchmarks page follow it, and
the page states both dates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KUhrHZV4bm5kkntEjqnfSi
@JuroOravec
JuroOravec merged commit fdda69f into main Nov 16, 2026
20 of 31 checks passed
@JuroOravec
JuroOravec deleted the benchmarks/errors-citry-1-0-0 branch November 16, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant