[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #1888

Summary

Composer, Python and RubyGems discovery each cache one probe answer
(composer global config home, the Python site query, gem env) keyed
on the process environment and working directory. That key function
was written three times, and there were two separate single-slot memo
types. They now share utils::process::{EnvKey, env_key, EnvKeyedMemo}.

Root cause

The memo was copied per crawler instead of living in one helper, and the
copies drifted. Composer and Ruby keep an answer only if the environment
is unchanged when the probe returns. Python's KeyedMemo kept every
successful answer, so an answer computed while another thread changed
the environment was stored under the old key. A later ask in the old
environment then got the wrong interpreter's site-packages.

Changes

• utils::process: EnvKey, env_key(), and EnvKeyedMemo<V> with
  Composer's rule. It has one slot and keeps an answer only when the
  answer is Some and the environment is unchanged after the run. On a
  cache hit it no longer clones the environment.
• Composer and Python use EnvKeyedMemo<String> directly. Composer's
  EnvKeyedMemo/env_key and Python's KeyedMemo/SiteQueryKey/
  site_query_key are deleted.
• Ruby keeps its async per-environment single-flight cells (it runs
  gem env gemdir and gem env gempath together and waits on concurrent
  askers), but keys them on the shared EnvKey/env_key.
  GemEnvKey/gem_env_key are deleted. The issue allows this option, and
  it leaves Ruby's concurrency unchanged.
• The memo tests moved to utils::process, and there is a new end-to-end
  keep-rule test against the real process environment.

Test evidence

• Red: commit 6fbb64b adds
  site_query_memo_drops_an_answer_the_environment_changed_under against
  the old Python memo. It fails with
  left: Some("ran under a moved environment") right: Some("fresh").
• Green: utils::process::tests::env_keyed_memo_drops_an_answer_the_environment_changed_under
  runs the same scenario on the shared memo, which SITE_QUERY_MEMO
  now uses, and passes. The Python copy was removed as a duplicate.
• cargo fmt --all -- --check: clean.
• cargo clippy --workspace --all-features -- -D warnings: clean.
• cargo test --workspace --all-features on 9ff63ee: passed (exit 0).
• After the review fixes in baa250a, cargo test -p socket-patch-core --all-features: 6994 passed, 4 failed. The 4 failures
  (copy_tree::relax_loop_must_not_traverse_symlinked_root,
  vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry,
  pypi_poetry::wire_write_failure_maps_error_and_leaves_lock_untouched,
  pypi_requirements::wire_failure_rolls_back_already_written_files) need
  read-only directories to block writes. This container runs as uid 0, which
  ignores that. None of them touches the code changed here, and CI does not
  run as root.
• No wrapper changes: npm/, pypi/ and gem/ don't use these probes.

Per-issue checklist

• Memoize env-keyed subprocess probes through one helper: composer, python and ruby each define the same env key and drifted on when an answer is kept #1888: one key function and one keep rule. Covered by
  env_keyed_memo_reuses_only_an_identical_environment,
  env_keyed_memo_asks_again_after_a_failed_probe,
  env_keyed_memo_drops_an_answer_the_environment_changed_under,
  env_key_is_sorted, the Ruby memoize_gem_env_homes and
  gem_env_cells_are_keyed_on_the_inherited_environment tests, and Python's
  site_query_memo_follows_the_environment.

/code-review high findings not fixed

• The memo holds a plaintext copy of the environment, secrets included. That was
  already true of the Composer and Ruby memos; storing only a digest would be a
  separate change.
• The keep check compares the environment before and after the run, so it misses
  a change that is made and undone while the probe runs (A→B→A). That gap is in
  the existing Composer and Ruby rule too. The doc comment now describes exactly
  what is checked.
• Ruby keeps its own single-flight map, which never evicts. See Changes above;
  this behavior is unchanged from main.

https://claude.ai/code/session_011f9aePz6bCHA8SMFdPu94K

---

Note

Medium Risk
Changes caching behavior for ecosystem package discovery probes; the Python fix prevents mis-keyed site-packages paths, while Composer/Ruby behavior should match their prior stricter rule.

Overview
Composer, Python, and Ruby discovery each memoize expensive subprocess probes (composer global config home, the Python site-packages query, gem env) keyed on process environment and cwd. That logic lived in three copies with two different memo types; Python’s KeyedMemo could cache an answer computed while the environment changed, so a later ask in the original environment could get the wrong site-packages.

This PR adds shared utils::process::{EnvKey, env_key, EnvKeyedMemo} with one keep rule: cache only Some answers when the environment is unchanged after the probe returns. Composer and Python wire their static memos to it and drop local duplicates; Ruby keeps its async per-env single-flight cells but keys them on the shared env_key. Memo unit tests move to process.rs, including a new test that the keep rule applies to the real process environment.

^{Reviewed by Cursor Bugbot for commit baa250a. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit baa250a. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Ready for review (burn-down agent) at baa250a.

• CI: 32/32 run checks green (50 skipped). Mergeable, no conflicts.
• Bugbot: reviewed baa250a, no findings; no unresolved threads.
• Reviewer note: the env-keyed probe memo is now shared across the composer, python and ruby crawlers via utils/process.rs.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does: It moves the env-keyed probe memo, which Composer, Python and Ruby each wrote their own way, into one shared EnvKeyedMemo<V> and env_key() in utils::process. Composer and Python now use the shared memo. Ruby keeps its async per-environment map but keys it on the shared EnvKey. The one behavior change is in Python: like Composer, it no longer caches an answer when the environment changed while the probe was running.

Risk: low. The cache key, single-slot replacement, recovery from a poisoned lock and "never cache None" all match main at each call site. The probe still runs outside the lock. Only the Python keep rule tightens, and that makes it safer.

Look here

• process.rs:377: get_or_run, the keep rule
• process.rs:390: get_or_run_keyed, which checks for a hit, runs the probe outside the lock, then stores the answer
• process.rs:501: the end-to-end keep-rule test
• python_crawler.rs:2811:`` the Python call site, where behavior changes
• ruby_crawler.rs:1016: Ruby keying rewired to env_key()

Verified: I read the full diff and compared each call site with main (cache key, invalidation, locking, error caching, Windows variable-name handling). There is no debug code or scope creep, and CHANGELOG.md is untouched. CI is 82/82 green, and Bugbot passed.

Changes I made: none.

Open questions (non-blocking):

• The new memo test covers the shared type. Python is held to the keep rule only by the type of SITE_QUERY_MEMO: the regression test from 6fbb64b was folded away.
• Ruby's test no longer checks the working-directory half of the key.

Auto-merge is armed, so approving sends this straight to the merge queue.

---

Generated by Claude Code