Repository navigation
benchmarks(errors): measure Citry 1.0.0 (54 to 56 of 61 before the app runs) - #740
Merged
Merged
Conversation
The Citry app now installs citry 1.0.0, citry-lsp 1.0.0 and citry-core 2.1.0 from PyPI, like every other app installs its framework, instead of building citry and citry-lsp from this repository's sources. The test that kept the app's citry-core override in step with citry's pin goes with the override. `citry-errors report --keep-from <earlier report>` takes the results of every framework a run did not measure from an earlier report, and the report names the run each framework's results come from (its commit, inputs tree, and any package it built from this repository), so one framework can be measured again without rerunning the other eleven. It keeps whole frameworks, and stops when the run left out a mistake of a framework it measured or a kept framework has no result for a mistake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KUhrHZV4bm5kkntEjqnfSi
…re the app runs) Citry 1.0.0, measured on 2026-11-16, reports 56 of the 61 mistakes that apply to it before the app runs (47 errors, 9 warnings). Two mistakes moved: a misspelled CSS class (m16) was missed and is now a warning, and a misspelled inject key (m50) failed only when the page rendered and is now a warning from `citry check` and the editor. The report in benchmarks/errors/results/2026-11-16 keeps the other eleven frameworks' results from the 2026-10-21 run and names that run. The docs data, the README chart and the Benchmarks page follow it, and the page states both dates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KUhrHZV4bm5kkntEjqnfSi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measures Citry's row of the error-prevention benchmark again on the released Citry 1.0.0, so the published number is a Citry 1.0.0 result.
Result
Two mistakes have a different outcome:
citry check(citry.css.misspelled-class, which suggestsboard).citry checkand the editor warn about it (citry.template.unknown-inject-key).Every other Citry cell is the same. The other eleven frameworks keep their 2026-10-21 results byte for byte. Since that run, their cases, catalogue and tools are unchanged except for a newer
certifipin in the ReactPy and Reflex apps. The Benchmarks page states both dates, the same way the speed section does.Ports changed
None. The baseline and all 61 Citry cases worked on 1.0.0 as written, and the clean app reported no problems. A manual check also confirmed on 1.0.0 that m23's type error is still reported when the card has no
template_data(), as the framework notes say.What changed
benchmarks/errors/frameworks/citry/): it installscitry==1.0.0andcitry-lsp==1.0.0from PyPI, and with themcitry-core2.1.0 (citry's pin) and ty 0.0.101 (citry-lsp's). Before, it built citry and citry-lsp frompackages/py, which neededoverride-dependenciesand[tool.uv.sources]; both are removed. The setup step no longer reinstalls packages from source, and the notes now name the 1.0.0 versions. Like docs: publish Citry 1.0.0 speed numbers, keep the other frameworks #732 and benchmarks: name the Citry 1.0.0 and citry-core 2.1.0 wheels in citry-local-wheels #735 for the speed benchmark, the app pins the release instead of building the repository. The test that kept the old citry-core override in step with citry's pin is removed with the override.citry-errors report <run> --keep-from <earlier report>keeps every framework the run did not measure from an earlier report.matrix.md,report.htmlandresults.json(kept) name the run each kept group comes from: its commit, inputs tree, uncommitted changes and any package it built from this repository. The report stops in two cases: the run left out a mistake of a framework it measured, or a kept framework has no result for a catalogue mistake. Tests are added intests/test_parallel_runs.py.benchmarks/errors/results/2026-11-16/, andresults/2026-10-21/stays.docs_site/data/error_prevention.json, the README chart and its PNGs were all regenerated with the existing scripts.VERSION_NOTEShas a new entry, "Citry 1.0.0.".docs_site/content/about/benchmarks.mdhas the new counts and lists, plus a "Dates and versions" note that replaces "Citry's exact build".benchmarks/errors/README.mdupdates its install,--keep-from, latest-result and m50 notes.Time
--no-cache): 79 seconds, plus about 2 seconds for the report.Checks
pytest benchmarks/errors/tests, 196 passed.ruff checkandruff format --checkonbenchmarks/errorspass.test_benchmark_images,test_error_benchmarkandtest_benchmark_chart, 105 passed.python -m docs_site build-check: 0 errors, 0 warnings.readme_benchmark_chartsandpublished_readme_linksvalidators: no findings.work.py, kept provenance trimmed to the kept frameworks, the completeness checks, and prose fixes.Versioning and changelog
Nothing under
crates/orpackages/changes. This touches the benchmark runner, its data, the docs and the README chart block only, so the 1.x rules do not apply and no changelog entry is needed.Left out
docs_site/data/error_prevention.jsonkeeps a singlerunid and does not carry thekeptorigins. Nothing reads them there, and the page prose states both dates.🤖 Generated with Claude Code
https://claude.ai/code/session_01KUhrHZV4bm5kkntEjqnfSi