Repository navigation
Memoize parse-pattern builds across instances; reuse one parser per locale - #1398
John-Cusack wants to merge 1 commit into
Conversation
…ocale Every arrow.get(s)/arrow.get(s, fmt) constructed a fresh DateTimeParser, so its __init__ locale-regex rebuild and _generate_pattern_re format rebuild ran per call while the opt-in per-instance lru_cache could never hit on the factory path. Cache the compiled (tokens, pattern) at class level keyed by (parser class, locale, input-map fingerprint, fmt), and keep one DateTimeParser per locale string in the factory. fmt-parse 3.86x, iso-parse 2.86x, representative pipeline 1.81x; peak RSS flat. Full suite 1908 passed, coverage gate holds, pre-commit clean. No behavior change: 20-case differential stock-vs-patched identical.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1398 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 10 10
Lines 2315 2342 +27
Branches 358 360 +2
=========================================
+ Hits 2315 2342 +27 ☔ View full report in Codecov by Harness. |
|
CI triage note (for reviewers): 24/27 legs green, including Linting and all CPython 3.10–3.14 legs. The 3 reds are not from this change:
Happy to rebase/retest if you want a fresh full-green signal. |
|
Closing this. The shared pattern cache is keyed on a snapshot of the input map taken in |
Pull Request Checklist
toxormake testto find out!).tox -e lintormake lintto find out!).masterbranch.No documentation update: there is no public API change —
arrow.getstillbuilds the same parsers with the same results; the memo is internal.
Description of Changes
Every
arrow.get(s)/arrow.get(s, fmt)constructs a freshDateTimeParser(locale)withcache_size=0, so each parse pays:DateTimeParser.__init__— locale lookup + 6_generate_choice_re→re.compilecalls, and_generate_pattern_re(fmt)— a full regex rebuild from the formatstring (the top cProfile frame of a format parse).
The existing
lru_cacheopt-in (cache_size > 0, default 0) cannot helpthis path: it wraps the bound method per instance, and the factory
discards the instance after one call.
This PR:
_generate_pattern_reat class level inarrow/parser.py(bounded at 128 entries, GIL-atomic dict access under a lock for
eviction). The key embeds a fingerprint of the parser's effective
input map (parser class + locale name + token→pattern/flags snapshot
built in
__init__), so per-instance or per-subclass mapcustomizations — including the
test_parse_unrecognized_token_BASE_INPUT_RE_MAPmock — can never read another parser's entry.Compiled patterns are immutable and
(map, fmt)-determined, so thecache is semantically transparent: identical
reobjects, identicalmatches and errors.
DateTimeParserper locale string in the factory(
arrow/factory.py_get_parser), so__init__'s locale-regexrebuild also runs once per locale instead of once per call. Parsers
are effectively immutable after construction for stock locales.
Measured
Ryzen 9 5950X, CPython 3.12.3, frozen 2000-item batches
(
YYYY-MM-DD HH:mm:ssformat batch + ISO-8601 batch), each arm a freshsubprocess, in-process min-of-5 after warmup, min across 3 processes;
peak RSS via
/procVmHWM (this box'sru_maxrssis constant-broken —verified against VmHWM, identical values both arms):
arrow.get(s, FMT)× 2000arrow.get(s)ISO × 2000Peak RSS flat across arms (~19.0 MB both). Outcome counts identical
both arms (all 2000 items parse in every track on both sides).
Reproduce (at 2224255 vs this branch, venv per
requirements/requirements-tests.txt):(Bench script available on request — say the word and I'll attach it.)
Test / lint proof
Their own gates, venv built per
Makefile(
requirements-tests.txt+pip install -e .):pytest: 1908 passed (4 new: cross-instance memo sharing,locale isolation en/fr/de, 128-entry eviction bound + transparent
rebuild, factory one-parser-per-locale). Pre-existing caching tests
(
test_parser_no_caching,test_parser_1_line_caching, ...) and the_BASE_INPUT_RE_MAPmock test pass unmodified.--cov-branch --cov=arrow --cov-fail-under=99, as intox.ini): 99.93%, required 99% reached.pre-commit run --all-files: all hooks pass (black, isort,flake8 + bugbear + annotations, mypy, pyupgrade).
locale month/day names, week dates, meridiem, fractional seconds, plus
5 error-path cases) through
arrow.get— stock vs patched20/20 identical, including exception types and messages.