Repository navigation
Re-run cross-validation with species folds (v1.0.5) - #5
Conversation
Regenerate all result files and validation_summary.tsv on the 2026-11 sources with folds assigned per species (#1, #2). Add gc_content. README: log the changes, warn against pulling during a run (#4), document the method argument of the 02 scripts. Co-authored-by: Claude Code <noreply@anthropic.com>
| phytools-ltp strain biosafety level biosafety level 2 0.47 0.11 NA NA 295 729 1888 4109 23501 0.09 0.061 1 8 | ||
| phytools-ltp all COGEM pathogenicity rating COGEM pathogenicity rating 1 0.74 0.08 NA NA 161 480 446 1025 23501 0.184 0.129 1 2 | ||
| phytools-ltp all COGEM pathogenicity rating COGEM pathogenicity rating 2 0.73 0.08 NA NA 266 512 446 1025 23501 0.184 0.129 1 2 | ||
| phytools-ltp all COGEM pathogenicity rating COGEM pathogenicity rating 3 0.69 0.39 NA NA 19 33 446 1025 23501 0.184 0.129 0 2 |
There was a problem hiding this comment.
COGEM 3 is MCC 0.69 with sd 0.39 on 19 positive tips (33 total). With a spread that wide, the mean sits about 0.03 under the 0.7 threshold, so a rerun with different fold seeds could easily flip it. The README and PR text state that it "falls below 0.7" and so will no longer be exported. Please either add a minimum-support rule (for example, require mcc_sd below some bound or at least N positives) or caveat this in the log. Also, the all and species rows are identical for every COGEM class; that is probably expected if COGEM is species-level only, but a one-line note would help.
| castor-ltp all genome size genome size NA NA 0.82 0.01 NA NA 6033 16556 23031 0.059 0.052 NA 26 | ||
| castor-ltp species genome size genome size NA NA 0.81 0.02 NA NA 5324 13691 23031 0.066 0.06 NA 23 | ||
| castor-ltp strain genome size genome size NA NA 0.63 0.1 NA NA 709 2865 23031 0.131 0.077 NA 3 | ||
| castor-ltp all growth temperature growth temperature NA NA 0.74 0.03 NA NA 15565 23024 23031 0.03 0.036 NA 68 |
There was a problem hiding this comment.
Growth temperature R2 rises from 0.55 to 0.74 even though the species-fold CV (#1) is stricter, which should lower it. Also, bp_phys is 23,024 of 23,031 tips, so almost every tip has a value. Please check whether the new BacDive/Madin growth-temperature data contains values propagated from genus or species level onto strains, or imputed values. Duplicated values across tips of the same species would leak across folds if the fold assignment isn't applied consistently. Explaining this in the README log would make it easier to trust that the improvement is real, since it moves the attribute well above the 0.5 export threshold.
| (issue #1). Re-run on the 2026-11 sources (Bergey's 2026 extraction, Madin | ||
| 2020 in full, BacDive from the API) with `gc_content` added to the numeric | ||
| list (issue #2). | ||
| list (issue #2). Rank `all` changes against v1.0.4: G+C content R2 0.92 (new); |
There was a problem hiding this comment.
The log lists the changes in R2/MCC but not the data versions, tool versions or random seed needed to reproduce them. The source changes (Bergey's 2026, Madin 2020, BacDive API) are also not pinned, and an API pull isn't reproducible. Please record the retrieval date, the commit of the scripts and the seed. Also note that the shape class "tail" has no row, and the Folds tarball described under v1.0.4 isn't part of this PR, so tell readers whether the folds were regenerated and where they are.
|
Review notes (this PR is data and docs only; it contains no R package code, so there are no exported functions to document):
|
Co-authored-by: Claude Code <noreply@anthropic.com>
Drafted with Claude Code. |
Closes #2. Results of the re-run with species folds (#1, merged in #3) on the 2026-11 sources: Bergey's 2026 extraction, Madin 2020 in full, BacDive from the API, and
gc_contentadded to the numeric list.Rank
all, changes that matter for the export thresholds (MCC 0.7, R2 0.5):Aerobic, spirillum and COGEM 3 drop below 0.7, so the next export stops predicting them; why aerobic dropped is waldronlab/bugphyzz-manuscript#15. "Tail" has too few species for five folds. Gram stain, anaerobic, spore formation and COGEM 1 and 2 hold or improve.
Every result file is regenerated; the diff is large because all per-physiology tables change. The README also gets a warning about pulling during a run (#4: that happened here, and the affected files were re-run) and documents the method argument of the
02_validation_*.shscripts.Drafted with Claude Code.