Med EvalsThe short answer
BioASQ 12b offers an unusually useful experimental distinction: answering before gold evidence arrives and answering after it arrives. That difference lets a reader ask whether a system struggled to find the literature or to use it. Our interpretation of the 2024 edition keeps retrieval, exact answering and ideal-answer evaluation separate. The comparison procedure below is original editorial guidance grounded in the challenge’s published phases.
Begin with the evidence contract
Phase A asks systems to identify relevant documents and snippets. Phase A+ asks for answers before the gold evidence is made available. Phase B supplies manually selected evidence for another answer submission. A table containing only a system name and an answer score leaves out this important experimental input. The same model can be evaluated as a retriever-plus-generator or as a generator receiving curated support.
For a reproducible implementation, archive the question, retrieved candidates, selected context and final answer as separate artifacts. Record whether the evidence came from the system or the challenge reference. Our suggested naming convention is simply benchmark, edition, batch, phase and answer family. It is intentionally explicit enough to prevent an evidence-assisted result from being mistaken for an end-to-end search result.
Read each answer family on its own terms
The official winner rules use different measurements for yes/no, factoid and list answers. A ranked factoid answer creates a different scoring problem from a set of list entities. Ideal prose answers introduce another judging procedure. Consequently, an exact-answer rank should not be described as the percentage of all biomedical questions answered correctly.
Our analysis recommends preserving the component measurements even when a publication emphasizes an aggregate rank. Rankings depend on the participating comparison set: a system’s rank can change when another entrant appears without any change to its own answers. That property makes rank useful for competition placement but less informative for diagnosing a regression in one product. Keep the underlying task-specific outcomes wherever they are available.
Use the phase contrast as a diagnostic lens
Consider an explicitly hypothetical system that provides a wrong answer after retrieving irrelevant documents but answers correctly with curated evidence. That pattern motivates a retrieval investigation. Now consider one that retrieves useful support but still answers incorrectly in both conditions. That pattern motivates inspection of context selection, synthesis or answer formatting. Neither example is a reported BioASQ measurement.
The contrast should remain a diagnostic aid rather than a causal conclusion. Curated evidence can differ in length, organization and relevance, and a new prompt may change more than one factor. A controlled follow-up would freeze the answering configuration and replace only the evidence package. Record those choices before examining outcomes, and inspect individual questions rather than relying solely on a phase-level average.
Freeze the edition and reference revision
The official dataset catalog and the overview paper contain an unresolved training-count discrepancy. That is a reason to identify a downloaded manifest, not to silently merge incompatible totals. The overview also distinguishes preliminary relevant-item references from later enriched references. A retrieval result needs the exact reference revision used for judging.
We therefore recommend an evidence manifest alongside the model manifest: edition, batch, retrieval corpus snapshot, relevant-item revision and submitted answer file. This adds practical analytical value to the source material while preserving uncertainty. It also limits the conclusion appropriately: success on a particular literature-question condition does not establish comprehensive evidence review or safe clinical decision-making.
References & further reading
These original sources support the methods discussed here. Our suggested planning steps are editorial guidance, not an endorsement by the source authors.
- Overview of BioASQ Tasks 12b and Synergy12 in CLEF2024 ↗Nentidis et al. / BioASQ organizers. Edition-specific stages and batch counts; training total differs between text and Table 1.
- Twelfth Challenge Winners ↗BioASQ organizers. Official phase-specific ranking rules; no scores from different answer types are interchangeable.
- BioASQ task b datasets ↗BioASQ organizers. Official edition catalog lists 5,046 questions for Training 12b and requires registration for downloading.
Published by Arcophos. Educational material, not clinical advice or a claim of regulatory compliance. Read our editorial method.