[lwaldron]

Closes #12. Stacked on #11 (v1.0.7), whose fold files it reads; retarget to main once #11 merges.

• create_attribute_summary.R writes validation_attributes.tsv (one row per rank and physiology: kappa, linearly weighted for ordered attributes, or R2; held-out tips weighted to the NSTI distribution of the tips to be imputed; lower bound of a two-sided 95% t-interval over folds; effective sample size) and validation_states.tsv (per-state precision).
• 01_numeric_val.R now writes its fold predictions (untracked CSV) for it.
• Results: discrete from the v1.0.7 fold files (2026-11-08); numeric re-run 2026-11-09 with the same packages. At rank all, gram stain, spore formation and aerophilicity pass kappa 0.7; G+C content, coding genes, genome size and growth temperature pass R2 0.5. Same decisions as the evidence in bugphyzzExports ADR 0006.

The ordered scales come from bugphyzz attributes.tsv (ordinal_position, waldronlab/bugphyzz#292); this run used that file from the bugphyzz branch.

Drafted with Claude Code; decisions and edits are mine.

[claude]

states is built from the states observed in held-out truth/predictions for this rank, so whether the attribute is "ordinal" depends on how many states happen to appear. The output shows this: biosafety level is weighted kappa at rank all but plain kappa at species/strain (only levels 1 and 2 are seen). Those scores aren't comparable across ranks, and the rank-all weighted value uses a different scale from the others. Likewise, if an intermediate level is missing, the linear weights get distorted (e.g. levels 1 and 3 become adjacent).

Fix: take the state set and order from attributes.tsv for the whole attribute group (all states with an ordinal_position), not from the observed states. Use a single metric per attribute across ranks, and use length(states) > 2 on that full set (with 2 states, weighted and unweighted kappa are the same anyway).

[claude]

If a fold has 1 - sum(agree * expected) == 0 (all held-out truth and predictions fall in one state, plausible for rare-state attributes and small folds), this returns NaN. mean, sd and lower_bound then propagate NaN to the whole attribute row, and the write.table step would write NaN or NA silently. Handle that case explicitly, either by dropping the fold with a warning or by returning NA and asserting !anyNA(folds$score) in summarise_scores.

[claude]

score_lcb is reported even where the weighting is degenerate. For example, spore shape (all) has ess_fraction 0.036 and max_weight 38.9, and hemolysis has ESS about 0.2 with weights up to 6.9. In those rows a few tips dominate each fold's weighted score, so the t-interval over folds understates the uncertainty. Either set the score to NA, or add an explicit low_ess flag, below a documented ESS threshold. The README says this table decides imputation, so downstream consumers shouldn't have to re-derive the check.

[claude]

Main issues, most important first:

1. Ordinal vs. nominal kappa depends on observed states (create_attribute_summary.R:139). Biosafety level is weighted kappa at rank all but plain kappa at species/strain, so the scores aren't comparable across ranks. Take the state set and order from attributes.tsv.
2. Kappa can be NaN (line 70) when a fold's expected agreement is 1. That NaN propagates to the row's mean and lower bound.
3. Degenerate weights are not flagged (line 87). Spore shape has ESS fraction 0.036 and max weight 38.9, and hemolysis is similarly poor, yet both get an score_lcb. Add an ESS threshold or flag.
4. No tests or sanity checks for the new weighting and kappa code, such as a hand-computed weighted kappa or an all-weights-equal case matching unweighted kappa.

This is a script repo with no package exports, so there were no Bioconductor documentation issues to raise.