Skip to content

Add qsar-model-validation skill for judging QSAR/ADMET model reliability - #299

Open
caliperworks wants to merge 1 commit into
K-Dense-AI:mainfrom
caliperworks:add-qsar-model-validation
Open

Add qsar-model-validation skill for judging QSAR/ADMET model reliability#299
caliperworks wants to merge 1 commit into
K-Dense-AI:mainfrom
caliperworks:add-qsar-model-validation

Conversation

@caliperworks

@caliperworks caliperworks commented Oct 2, 2026

Copy link
Copy Markdown

The collection covers how to drive the modelling libraries -- rdkit, deepchem, pytdc, molfeat, datamol all document their APIs and versions. Several of them also touch evaluation practice in passing: deepchem's example scripts use a scaffold split and warn that it does not eliminate all leakage, pytdc documents entity-overlap audits, and molfeat and datamol each carry a one-line warning that a random split leaks close analogues.

What none of them provides is the protocol itself: how to choose the split that matches the claim being made, how to check leakage explicitly, what baseline a metric has to clear before it means anything, and how to calibrate a probability before a decision uses it. There, that practice is incidental to library documentation. This skill makes it the subject.

It ships three CLIs and four references:

  • split_compare.py scores one fixed model on random, scaffold, time, and group splits of the same table, against a dummy baseline, and reports the leakage counts (exact duplicates, near-duplicates at a stated Tanimoto, shared scaffolds) that explain any gap between them.
  • applicability_domain.py marks test compounds against a training set. The similarity threshold defaults to a percentile of the training set's own nearest-neighbour distribution rather than a hardcoded constant, and with --prediction-col it scores the model separately inside and outside the domain.
  • calibration_report.py reports ECE, Brier, and the reliability curve, and fits a Platt or isotonic correction on rows held out from the rows it reports on.

The test suite asserts direction rather than just execution: on the synthetic fixture, where activity is decided by scaffold family, the scaffold split must score worse than the random split, the random split must leak more near-duplicates, and a monotone calibrator must leave ROC-AUC unchanged. 44 tests pass; tests/_meta passes.

The fixture is generated by fixtures/make_synthetic_actives.py with a fixed seed, so the provenance claim is reproducible rather than asserted. Fragments are joined through RDKit dummy atoms rather than by SMILES string substitution, which would put substituents on ring nitrogens and produce a table RDKit refuses -- the class of cleaning problem this skill is about.

Summary

Type of change

  • New skill
  • Update to an existing skill
  • Tests
  • Repository tooling or CI
  • Documentation
  • Other:

Skills touched

How this was tested

uv run skills-ref validate ./skills/<name>
uv run --with pytest python -m pytest tests/<name> -q

Related issues and references

Checklist

Drawn from the Pull Request Checklist in CONTRIBUTING.md. Items that do not apply to this PR can be left unchecked with a short note.

Skill format

  • The skill directory name and the name frontmatter match exactly.
  • The skill directory contains only SKILL.md, references/, scripts/, and assets/ — no tests/ directory and no test_*.py files.
  • SKILL.md has valid YAML frontmatter and a Markdown body.
  • Only the six spec-defined top-level fields are present; everything else lives under metadata.
  • metadata is a block mapping, not single-line JSON, and scalar values are quoted where needed.
  • Any metadata.openclaw or metadata.hermes block is a nested mapping, not a JSON string.
  • metadata.version exists, is quoted, and is bumped if an existing skill changed.
  • The description says both what the skill does and when an agent should use it.

Validation and tests

  • uv run skills-ref validate ./skills/<name> passes.
  • Tests live in tests/<skill-name>/, and any new scripts/ skill has a [skills.<name>] entry in tests/skill-requirements.toml.
  • Relevant test suites pass, or the failures are explained below.
  • Security scanner results are clean or explained in this PR.

Content and safety

  • Examples and scripts were tested, or are clearly marked as illustrative.
  • No secrets, credentials, private data, or unsafe instructions are included.
  • Credentials the skill needs are named in compatibility and declared in metadata.openclaw.envVars.
  • Relevant official documentation is linked where useful.

Notes for reviewers

@caliperworks
Add qsar-model-validation skill for judging QSAR/ADMET model reliability
6737c57 
The collection covers how to drive the modelling libraries -- rdkit, deepchem,
pytdc, molfeat, datamol all document their APIs and versions. Several of them
also touch evaluation practice in passing: deepchem's example scripts use a
scaffold split and warn that it does not eliminate all leakage, pytdc documents
entity-overlap audits, and molfeat and datamol each carry a one-line warning
that a random split leaks close analogues.

What none of them provides is the protocol itself: how to choose the split that
matches the claim being made, how to check leakage explicitly, what baseline a
metric has to clear before it means anything, and how to calibrate a
probability before a decision uses it. There, that practice is incidental to
library documentation. This skill makes it the subject.

It ships three CLIs and four references:

- split_compare.py scores one fixed model on random, scaffold, time, and group
  splits of the same table, against a dummy baseline, and reports the leakage
  counts (exact duplicates, near-duplicates at a stated Tanimoto, shared
  scaffolds) that explain any gap between them.
- applicability_domain.py marks test compounds against a training set. The
  similarity threshold defaults to a percentile of the training set's own
  nearest-neighbour distribution rather than a hardcoded constant, and with
  --prediction-col it scores the model separately inside and outside the domain.
- calibration_report.py reports ECE, Brier, and the reliability curve, and fits
  a Platt or isotonic correction on rows held out from the rows it reports on.

The test suite asserts direction rather than just execution: on the synthetic
fixture, where activity is decided by scaffold family, the scaffold split must
score worse than the random split, the random split must leak more
near-duplicates, and a monotone calibrator must leave ROC-AUC unchanged. 44
tests pass; tests/_meta passes.

The fixture is generated by fixtures/make_synthetic_actives.py with a fixed
seed, so the provenance claim is reproducible rather than asserted. Fragments
are joined through RDKit dummy atoms rather than by SMILES string substitution,
which would put substituents on ring nitrogens and produce a table RDKit
refuses -- the class of cleaning problem this skill is about.
@zako123191

zako123191 commented Oct 2, 2026 via email

Copy link
Copy Markdown

@caliperworks

caliperworks commented Oct 2, 2026

Copy link
Copy Markdown
Author

Summary

Adds a qsar-model-validation skill covering the evaluation protocol for QSAR,
activity-prediction, and ADMET models: which split answers which question, how
to check leakage explicitly, what baseline a metric has to clear before it means
anything, and how to calibrate a probability before a decision uses it.

The collection documents how to drive the modelling libraries -- rdkit,
deepchem, pytdc, molfeat, datamol all document their APIs and versions. Several
of them also touch evaluation practice in passing: deepchem's example scripts
use a scaffold split and warn that it does not eliminate all leakage, pytdc
documents entity-overlap audits, and molfeat and datamol each carry a one-line
warning that a random split leaks close analogues. None of them makes the
protocol the subject, which is the gap this fills.

Type of change

  • New skill

Skills touched

  • qsar-model-validation

How this was tested

uv run skills-ref validate ./skills/qsar-model-validation
# Valid skill: skills/qsar-model-validation

uv run --with pytest python -m pytest tests/qsar-model-validation -q
# 44 passed, 3 subtests passed

uv run --with pytest python -m pytest tests/_meta -q
# 18 passed, 4247 subtests passed

Verified against RDKit 2026.3.6, scikit-learn 1.9.1, pandas 3.0.6, NumPy 2.5.3,
SciPy 1.18.1, Python 3.12.

One gate not exercised locally:

python tests/run_all.py --isolated qsar-model-validation

uv is not available in the environment this was developed in, so the isolated
per-skill environment was never built here. The [skills.qsar-model-validation]
entry is present in tests/skill-requirements.toml and follows the file's
pinning convention (domain packages pinned, numpy/pandas/scipy left to the
resolver), so it should resolve -- flagging it because it is the only gate this
PR has not run.

Related issues and references

The practice this skill operationalises is independently stated in
awesome-chemistry-datasets' OPPORTUNITIES.md, which lists scaffold/structure-aware
splits as a requirement alongside random splits in order to expose leakage, and
warns against averaging conflicting assay measurements by default:

https://github.com/kjappelbaum/awesome-chemistry-datasets/blob/main/OPPORTUNITIES.md

Checklist

Drawn from the Pull Request Checklist in CONTRIBUTING.md.

Skill format

  • The skill directory name and the name frontmatter match exactly.
  • The skill directory contains only SKILL.md, references/, scripts/, and assets/ -- no tests/ directory and no test_*.py files.
  • SKILL.md has valid YAML frontmatter and a Markdown body.
  • Only the six spec-defined top-level fields are present; everything else lives under metadata.
  • metadata is a block mapping, not single-line JSON, and scalar values are quoted where needed.
  • Any metadata.openclaw or metadata.hermes block is a nested mapping, not a JSON string. -- not applicable, this skill declares neither block.
  • metadata.version exists, is quoted, and is bumped if an existing skill changed. -- new skill, "1.0".
  • The description says both what the skill does and when an agent should use it.

Validation and tests

  • uv run skills-ref validate ./skills/<name> passes.
  • Tests live in tests/<skill-name>/, and any new scripts/ skill has a [skills.<name>] entry in tests/skill-requirements.toml.
  • Relevant test suites pass, or the failures are explained below. -- 44 and 18 pass; see the one unexercised gate above.
  • Security scanner results are clean or explained in this PR. -- rely on CI; the credentials the scanner needs are not available here.

Content and safety

  • Examples and scripts were tested, or are clearly marked as illustrative.
  • No secrets, credentials, private data, or unsafe instructions are included.
  • Credentials the skill needs are named in compatibility and declared in metadata.openclaw.envVars. -- not applicable, this skill needs none.
  • Relevant official documentation is linked where useful.

Notes for reviewers

  • The test fixture is synthetic and generated by
    tests/qsar-model-validation/fixtures/make_synthetic_actives.py with a fixed
    seed, so its provenance is reproducible rather than asserted. No third-party
    data is redistributed.
  • The suite asserts direction, not just execution: on that fixture, where
    activity is decided by Bemis-Murcko scaffold family, the scaffold split must
    score worse than the random split, the random split must leak more
    near-duplicates, and a monotone calibrator must leave ROC-AUC unchanged.
  • expected_calibration_error() uses equal-width bins, so ECE has a floor of
    roughly 1/(2 * bins) for extreme probabilities. That is documented in
    references/metrics-and-calibration.md rather than corrected, because the
    reliability curve is the primary artefact.
  • Scope is deliberately limited to evaluation. The skill does not train models;
    it judges models trained with the collection's existing library skills.
  • scripts/ are standard-library-first: RDKit, scikit-learn, pandas, NumPy, and
    SciPy are imported lazily so that --help works without them installed.

peppywolfym-lab
peppywolfym-lab approved these changes Oct 3, 2026
View reviewed changes
Comment thread skills/qsar-model-validation/SKILL.md
@@ -0,0 +1,203 @@
---
name: qsar-model-validation

@peppywolfym-lab peppywolfym-lab Oct 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YASIR BIN MUDASSIR

@peppywolfym-lab

peppywolfym-lab commented Oct 3, 2026

Copy link
Copy Markdown

Update

peppywolfym-lab
peppywolfym-lab suggested changes Oct 4, 2026
View reviewed changes

@peppywolfym-lab peppywolfym-lab left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@peppywolfym-lab peppywolfym-lab peppywolfym-lab approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@caliperworks @zako123191 @peppywolfym-lab