Skip to content

Spike: reuse unchanged projection ancestor names - #2137

Draft
Tyrie Vella (tyrielv) wants to merge 1 commit into
microsoft:vnextfrom
tyrielv:tyrielv/projection-name-memory-vnext
Draft

Tyrie Vella (tyrielv) wants to merge 1 commit into
microsoft:vnextfrom
tyrielv:tyrielv/projection-name-memory-vnext

Conversation

@tyrielv

@tyrielv Tyrie Vella (tyrielv) commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem and Context

The version-4 index parser reuses path components when consecutive files share a parent.
When the parent changes, it allocates names for every component.
Folder lookup often finds existing ancestors, but the pools retain those temporary names.
Large indexes therefore retain many names that no persistent tree node needs.

This draft measures ancestor-name reuse without changes to file-backed projection or enlistment lifecycle.
The prototype reduces retained managed memory.
It does not establish a startup-private-commit improvement: the isolated NativeAOT benchmark reports higher startup commit and fragmentation.

Changes

  • Record directory-separator byte offsets and reuse complete ancestor components strictly before ReplaceIndex.
  • Preserve the existing same-parent fast path and never reuse the previous file name as a directory component.
  • Invalidate reusable components at skipped entries and new index parses.
  • Preserve valid reuse across projected merge-stage transitions.
  • Keep tree construction, folder lookup order, sparse projection rules, and pool allocation and shrink policies unchanged.
  • Add regression tests for allocation counts, component boundaries, nested siblings, root files, Unicode, casing, sparse folders, and repeated or empty rebuilds.
  • Cover same-path conflict stages, non-extended entries, and multi-byte replacement lengths in the index fixtures.
  • Add a read-only index snapshot benchmark to GVFS.PerfProfiling.
    It reports pool use, memory, build times, lookup and enumeration latency, and complete-tree fingerprints.
  • Keep read-only index access inside the benchmark and preserve the existing repair-path access mode.
  • Document benchmark commands and measurement limits.

Measurements

Both variants use the same index, query corpus, benchmark code, and vnext base.
Measurements use Release ARM64, NativeAOT, and .NET 10.0.12.
Each variant ran in three separate processes with five rebuilds per process.
The table shows medians.

Metric Baseline Prototype
Pooled names used 5,180,806 2,825,956
Managed live memory after first build 544.8 MiB 420.6 MiB
Private commit after first build 564.2 MiB 638.0 MiB
Private commit after five rebuilds 812.4 MiB 748.1 MiB
First-build time 1,507.6 ms 3,174.2 ms
Rebuild time 787.2 ms 916.1 ms

All measured trees contain 2,448,066 files and 377,749 folders.
The baseline and prototype produce the same fingerprint across first builds and final rebuilds.

These results describe an isolated projection, not a complete mount.
Forced GC affects only the benchmark process.
No size database is present.
First-pass queries do not imply a cold OS file cache.
The first-build timing ranges overlap, so these samples do not establish a statistically significant timing difference.
Run-to-run variation is large, and the measurements do not establish a consistent timing benefit.
The startup-private-commit cost remains a limitation of this prototype.

Known gaps and deliberate decisions

  • Startup private commit — The prototype retains less managed data but increases startup private commit. Existing pool sizing crosses different shrink thresholds after reuse. This experiment does not isolate every cause. Pool tuning remains outside this PR; no follow-up PR exists.
  • Isolated rebuild results — The benchmark materializes all names and forces GC before each rebuild. Its later private-commit decrease does not establish a saving in a complete mount.

Retain complete directory-name components before the version-4 index
replacement boundary. Reset reusable path state at skipped entries and
new parses. Preserve reuse across projected merge-stage transitions.

Add allocation and projection regression tests, plus a read-only snapshot
benchmark with tree fingerprints and memory and latency measurements.
Keep read-only file access inside the benchmark and preserve repair access.

The isolated NativeAOT benchmark retains less managed memory, but startup
private commit increases. Keep the existing pool allocation and shrink
policies unchanged.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
Tyrie Vella (tyrielv) force-pushed the tyrielv/projection-name-memory-vnext branch from 59f3b51 to 2698852 Compare October 16, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant