Skip to content

Re-run cross-validation with species folds (v1.0.5) - #5

Merged
lwaldron merged 2 commits into
mainfrom
validation-v1.0.5
Nov 6, 2026
Merged

lwaldron merged 2 commits into
mainfrom
validation-v1.0.5

Conversation

@lwaldron

@lwaldron lwaldron commented Nov 6, 2026

Copy link
Copy Markdown
Member

Closes #2. Results of the re-run with species folds (#1, merged in #3) on the 2026-11 sources: Bergey's 2026 extraction, Madin 2020 in full, BacDive from the API, and gc_content added to the numeric list.

Rank all, changes that matter for the export thresholds (MCC 0.7, R2 0.5):

Attribute v1.0.4 v1.0.5 Tips
G+C content (R2) 0.92 13,575
Genome size (R2) 0.72 0.82 6,033
Coding genes (R2) 0.72 0.78 2,163
Growth temperature (R2) 0.55 0.74 15,565
Length (R2) 0.15 0.29 5,814
Optimal pH (R2) 0.14 0.26 2,866
Width (R2) 0.04 0.01 5,857
Aerobic (MCC) 0.71 0.66 9,592
Spirillum (MCC) 0.70 0.47 205
COGEM 3 (MCC) 0.72 0.69 19
Shape "tail" (MCC) 1.00 no row

Aerobic, spirillum and COGEM 3 drop below 0.7, so the next export stops predicting them; why aerobic dropped is waldronlab/bugphyzz-manuscript#15. "Tail" has too few species for five folds. Gram stain, anaerobic, spore formation and COGEM 1 and 2 hold or improve.

Every result file is regenerated; the diff is large because all per-physiology tables change. The README also gets a warning about pulling during a run (#4: that happened here, and the affected files were re-run) and documents the method argument of the 02_validation_*.sh scripts.

Drafted with Claude Code.

Regenerate all result files and validation_summary.tsv on the 2026-11
sources with folds assigned per species (#1, #2). Add gc_content. README:
log the changes, warn against pulling during a run (#4), document the
method argument of the 02 scripts.

Co-authored-by: Claude Code <noreply@anthropic.com>
Comment thread validation_summary.tsv
phytools-ltp strain biosafety level biosafety level 2 0.47 0.11 NA NA 295 729 1888 4109 23501 0.09 0.061 1 8
phytools-ltp all COGEM pathogenicity rating COGEM pathogenicity rating 1 0.74 0.08 NA NA 161 480 446 1025 23501 0.184 0.129 1 2
phytools-ltp all COGEM pathogenicity rating COGEM pathogenicity rating 2 0.73 0.08 NA NA 266 512 446 1025 23501 0.184 0.129 1 2
phytools-ltp all COGEM pathogenicity rating COGEM pathogenicity rating 3 0.69 0.39 NA NA 19 33 446 1025 23501 0.184 0.129 0 2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COGEM 3 is MCC 0.69 with sd 0.39 on 19 positive tips (33 total). With a spread that wide, the mean sits about 0.03 under the 0.7 threshold, so a rerun with different fold seeds could easily flip it. The README and PR text state that it "falls below 0.7" and so will no longer be exported. Please either add a minimum-support rule (for example, require mcc_sd below some bound or at least N positives) or caveat this in the log. Also, the all and species rows are identical for every COGEM class; that is probably expected if COGEM is species-level only, but a one-line note would help.

Comment thread validation_summary.tsv
castor-ltp all genome size genome size NA NA 0.82 0.01 NA NA 6033 16556 23031 0.059 0.052 NA 26
castor-ltp species genome size genome size NA NA 0.81 0.02 NA NA 5324 13691 23031 0.066 0.06 NA 23
castor-ltp strain genome size genome size NA NA 0.63 0.1 NA NA 709 2865 23031 0.131 0.077 NA 3
castor-ltp all growth temperature growth temperature NA NA 0.74 0.03 NA NA 15565 23024 23031 0.03 0.036 NA 68

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Growth temperature R2 rises from 0.55 to 0.74 even though the species-fold CV (#1) is stricter, which should lower it. Also, bp_phys is 23,024 of 23,031 tips, so almost every tip has a value. Please check whether the new BacDive/Madin growth-temperature data contains values propagated from genus or species level onto strains, or imputed values. Duplicated values across tips of the same species would leak across folds if the fold assignment isn't applied consistently. Explaining this in the README log would make it easier to trust that the improvement is real, since it moves the attribute well above the 0.5 export threshold.

Comment thread README.md
(issue #1). Re-run on the 2026-11 sources (Bergey's 2026 extraction, Madin
2020 in full, BacDive from the API) with `gc_content` added to the numeric
list (issue #2).
list (issue #2). Rank `all` changes against v1.0.4: G+C content R2 0.92 (new);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log lists the changes in R2/MCC but not the data versions, tool versions or random seed needed to reproduce them. The source changes (Bergey's 2026, Madin 2020, BacDive API) are also not pinned, and an API pull isn't reproducible. Please record the retrieval date, the commit of the scripts and the seed. Also note that the shape class "tail" has no row, and the Folds tarball described under v1.0.4 isn't part of this PR, so tell readers whether the folds were regenerated and where they are.

@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown

Review notes (this PR is data and docs only; it contains no R package code, so there are no exported functions to document):

  1. Growth temperature R2 0.55 → 0.74 is surprising under the stricter species-fold CV, and nearly every tip (23,024 of 23,031) has a value. Please verify there is no propagated or imputed data leaking across folds before relying on it for export (inline comment on validation_summary.tsv).
  2. COGEM 3 (MCC 0.69, sd 0.39, n=19) is dropped from export on a noisy estimate just under the 0.7 threshold. Consider a minimum-support or variance rule, or note the caveat.
  3. Reproducibility: the v1.0.5 log doesn't record data-retrieval dates (BacDive API), script commit or seed, and doesn't say where the regenerated Folds are.
  4. Aerobic (0.71 → 0.66) is a real drop, tracked in bugphyzz-manuscript#15. Since it removes an attribute from export, it is worth resolving before the release is tagged.

Co-authored-by: Claude Code <noreply@anthropic.com>
@lwaldron

lwaldron commented Nov 6, 2026 •

Copy link
Copy Markdown
Member Author
  1. Growth temperature: no leakage. 23,024 is bp_phys, the number of taxa with a bugphyzz value; 15,565 of the 23,031 tips carry one. physiologies() returns curated values only (no ASR or taxonomy propagation), and the per-taxon value is the midpoint from taxPPro::getDataReady(), the same function the export uses. Folds are per species, so the gain comes from the data: 1,314 more annotated tips (14,251 to 15,565), mostly Madin species-level values, and lower mean NSTI (0.035 to 0.030). Worth a per-source look later, not a blocker.
  2. COGEM 3: agreed it's noisy (19 tips, sd 0.39). Noted on waldronlab/bugphyzz-manuscript#15 alongside aerobic, where the threshold question is being decided; I'd rather not add a support rule here before that.
  3. Added a line to the v1.0.5 log with package versions, the data dates, seeds and where the fold CSVs live.
  4. Yes, tracked in waldronlab/bugphyzz-manuscript#15.

Drafted with Claude Code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run the cross-validation on the 2026-11 sources with the fold fix, add G+C content

1 participant