Skip to content

Score ranked grounding alternatives in MedMentions - #3969

Open
maziyarpanahi wants to merge 18 commits into
masterfrom
fix/issue-3569-grounding-alternatives
Open

maziyarpanahi wants to merge 18 commits into
masterfrom
fix/issue-3569-grounding-alternatives

Conversation

@maziyarpanahi

@maziyarpanahi maziyarpanahi commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Pull Request

Description

Include ranked alternatives when calculating top-k accuracy for non-abstaining GroundedSpan outputs. Withheld selections and empty selections remain abstentions; sequence providers keep their existing behavior.

This follows the report and immutable reproduction shared by Belal Embaby. The fix was reproduced against current upstream and implemented in existing tracked source/test files; the contributor's fork was not modified.

Type of Change

  • Bug fix
  • Test addition/improvement

Changes Made

  • Score selected candidates followed by ranked alternatives, preserving UMLS filtering and the requested depth.
  • Extend existing synthetic benchmark tests for rank cutoffs, explicit abstention, no selection, both provider shapes, unchanged output ordering and value-free aggregate reports.
  • Update the evaluator docstring and changelog; no corpus or restricted vocabulary is bundled.

Testing

The published candidate is 593d055f63000aae6b1be47d00c11093126ed097. It retains the issue contribution and normally integrates frozen SDK prerequisite 5e635dd707ebd74c2d0806b0cb6415b53ab4d7b0 from #3971. The issue-specific scope is unchanged; shared SDK/CI/build-budget changes belong to that prerequisite.

  • make format, make lint, make format-check, make type-check and git diff --check: passed, without unrelated formatting changes.
  • The repository's configured type check covers four modules, not the entire codebase.
  • Focused regression/public-API tests: 55 passed.
  • Shared wheel/import budget and CI-wiring tests: 41 passed.
  • Repository-policy and declared development-profile dependency audit: passed. This audit does not clear separately reviewed optional dependency profiles.
  • High-severity/medium-confidence Bandit gate: passed, zero high-severity findings, no files skipped. Lower-severity results are not represented as zero.

Post-integration full offline pytest is pending; no full-suite pass is yet claimed for this exact head. Its local runtime is Python 3.13.11, with a private environment and unique OS temporary fixture directory. The previous complete local full receipt was for be790dc2982bb10e2b1aa871b3c8febe8ff5b077: 23,123 passed, 222 skipped, 24 warnings, zero failures/errors. That older result is historical evidence only and does not validate the new integration head.

SDK #3971 must still land and this exact pushed head must pass its own full local and hosted checks before merge. Hosted checks remain pending; this is not merge or release clearance. No tests or assertions were removed or weakened, and no skipped check or dependency waiver is introduced.

Documentation

  • Updated the relevant public docstring and CHANGELOG.md.

Code Quality

  • Canonical formatting/lint/type gates passed.
  • Performed a separate diff review and checked accepted-input/negative controls.
  • No unrelated formatting changes or new tracked files.

Dependencies

Checklist

  • Read the contributing guide, Code of Conduct and maintainer guide.
  • One focused issue implementation commit is retained; shared prerequisite integration is recorded separately. The contributor's original fork remains unchanged.

Related Issues

Closes #3569

Screenshots/Examples

Not applicable; backend/evaluation contract fix with synthetic unit coverage.

@maziyarpanahi maziyarpanahi added this to the v3.0 milestone Oct 3, 2026
@maziyarpanahi maziyarpanahi added bug Something isn't working roadmap-v3 OpenMed V3 roadmap backlog labels Oct 3, 2026
@maziyarpanahi maziyarpanahi self-assigned this Oct 3, 2026
@maziyarpanahi maziyarpanahi added bug Something isn't working roadmap-v3 OpenMed V3 roadmap backlog labels Oct 3, 2026
@maziyarpanahi
maziyarpanahi marked this pull request as ready for review October 3, 2026 12:05

Copy link
Copy Markdown
Owner Author

Thank you for the focused contribution for #3569. Its implementation and regression coverage already landed on master through #3650, and the issue is closed. I verified the release merge is present and the corresponding regressions still pass on current master.

This PR is held as superseded; the resolved issue provides no remaining implementation scope for another merge. I am retaining the original branch and discussion. Any additional behavior should be scoped to a new maintainer-backed issue before further changes or merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working roadmap-v3 OpenMed V3 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Score GroundedSpan alternatives in MedMentions top-k

2 participants