Med EvalsBenchmark analysis
PubMedQA: audit the input before interpreting the score
An original engineering analysis of conclusion leakage, class balance and the official PubMedQA evaluator.
Original analysis / Med Evals
Read the benchmark closely.
We analyze two named biomedical question-answering benchmarks at the level of their inputs, references and scoring. PubMedQA isolates answering from a supplied abstract; BioASQ 12b separates finding literature from answering with it. Explore the published task conditions, inspect selected historical measurements and use our original guides to make an evaluation reproducible. Benchmark authors retain credit for the datasets and experiments. Our contribution is the analysis and tools; we have not run the models displayed here.
5 min read
Benchmark analysis
BioASQ 12b: why the evidence phase belongs beside every score
Use the A, A+ and B conditions to distinguish retrieval failures from answer-generation failures.
Benchmark analysis
PubMedQA and BioASQ measure different parts of a biomedical evidence pipeline
Compare abstract interpretation with literature retrieval and answer synthesis without inventing a combined leaderboard.