mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
Extracted source text · arXiv 2606.29467
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% mamabench & mamaretrieval -- Benchmarks paper -- arXiv tech report
% ACM journal style (acmart, acmsmall, nonacm)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Force arXiv (and any latexmk default) to process this with pdfLaTeX in PDF
% mode -- acmart's fonts work under pdflatex, the engine arXiv uses for it.
\pdfoutput=1
\documentclass[acmsmall,nonacm]{acmart}
% Drop ACM-specific frames that do not apply to arXiv tech reports.
\settopmatter{printacmref=false, printccs=false, printfolios=true}
\renewcommand\footnotetextcopyrightpermission[1]{}
\pagestyle{plain}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{microtype}
\usepackage{tablefootnote} % footnotes that survive inside table floats
\usepackage{listings}
% Verbatim listing style for the appendix prompt (ASCII-cleaned; long lines wrap).
\lstset{
basicstyle=\ttfamily\scriptsize,
breaklines=true,
breakindent=0pt,
columns=fullflexible,
keepspaces=true,
showstringspaces=false,
frame=single,
framesep=4pt,
xleftmargin=3pt,
xrightmargin=3pt,
}
% The author's name in Chinese is a small vector graphic (figures/renyi-cjk.pdf,
% the glyphs set in FandolSong, image box = the font's TeX box). This keeps the
% paper on pdflatex (the engine arXiv uses for acmart) -- no Unicode engine. The
% size (0.933em = the glyphs' height+depth) and baseline drop (0.16em = their
% depth) are in em, so \zhname scales to the surrounding font like a real glyph.
% \DeclareRobustCommand protects the graphicx keys from acmart's author
% uppercasing (which would mangle height= into HEIGHT=), and hyperref strips
% \includegraphics from PDF-string sinks (pdfauthor / hyperxmp XMP).
\DeclareRobustCommand{\zhname}{%
\raisebox{-0.16em}{\includegraphics[height=0.933em]{figures/renyi-cjk}}}
\bibliographystyle{ACM-Reference-Format}
\citestyle{acmauthoryear}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Front matter
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{mamabench and mamaretrieval: Benchmarks for Evaluating Medical
Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive
Health}
\author{Ren Yi\,\zhname}
% ORCID is declared here (and linked at the arXiv account level, which is what
% drives attribution); it is not printed in the body.
\orcid{0009-0002-6911-2855}
\affiliation{%
\institution{École Polytechnique Fédérale de Lausanne}
\country{Switzerland \quad bonjour@renyi.ch}
}
% Show the email on the author line (above) rather than in the journal
% "Author's Contact Information" footnote, which we suppress here.
\authorsaddresses{}
\begin{document}
\begin{abstract}
Medical question-answering benchmarks rarely cover the maternal, neonatal,
child, and reproductive-health questions a nurse-midwife asks, and, to our
knowledge, no public chunk-level relevance benchmark exists for maternal-health
guideline retrieval. We release two benchmarks that fill these gaps. \textbf{mamabench} is a
scope-filtered QA set of 25{,}949 items assembled from seven existing
expert-authored sources across multiple-choice, short-answer, and rubric-graded
tracks; to help users calibrate the LLM judge that scores the rubric track, we
re-scope HealthBench's physician-labelled meta-evaluation to the domain. \textbf{mamaretrieval}
pairs 3{,}185 clinical queries with graded ($0$--$6$) relevance labels over a
63{,}650-chunk maternal-health guideline corpus, using a decomposed rubric that
distinguishes a chunk that answers a query from one merely on its topic. Three
decisions shape both: assemble and filter expert sources rather than author
questions, grade relevance rather than binarise it, and measure and disclose the
limits of the labels---scope-classifier agreement, a frontier-judge check,
and a pooling-completeness audit---rather than treat them as an oracle. A
companion paper uses the benchmarks to evaluate a deployed on-device assistant;
both are released openly for research.
\end{abstract}
\keywords{benchmark, retrieval, medical question answering, maternal
health, OBGYN, LLM-as-judge, TREC pooling}
\maketitle
% The visible author line carries the CJK name as a graphic; set the PDF
% metadata author to the plain-ASCII romanisation explicitly.
\hypersetup{pdfauthor={Ren Yi}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Body
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{sections/01_introduction}
\input{sections/02_mamabench}
\input{sections/03_mamaretrieval}
\input{sections/04_limitations}
\input{sections/05_conclusion}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Bibliography
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliography{references}
\appendix
\input{sections/A_judge_prompt}
\end{document}
\section{Introduction}
\label{sec:intro}
Medical question-answering benchmarks have proliferated, but maternal,
neonatal, child, and reproductive health---and the sub-Saharan, midwife-led
primary-care setting in particular---are a thin slice of them. The widely used
sets draw on US and Indian licensing exams (MedQA~\cite{jin2021medqa},
MedMCQA~\cite{pal2022medmcqa}) or broad clinical chat
(HealthBench~\cite{arora2025healthbench}); AfriMed-QA~\cite{olatunji2025afrimedqa}
is pan-African but multi-specialty. None is scoped to the maternal, neonatal, child,
and reproductive-health questions a nurse-midwife actually asks at the point of
care.
Retrieval is worse served. Medical retrieval-augmented-generation benchmarks
such as MIRAGE~\cite{xiong2024benchmarking} score the \emph{end-to-end} answer
drawn from a corpus, not whether an individual retrieved passage is relevant;
the classical medical-IR relevance collections (the TREC Clinical Decision
Support and Precision Medicine tracks) judge full PubMed articles against case
descriptions, not the short guideline chunks a deployed RAG system retrieves and
injects into a prompt. To our knowledge there is no public chunk-level relevance
benchmark for maternal-health guideline retrieval.
We release two benchmarks%
\footnote{Datasets: \url{https://huggingface.co/datasets/nmrenyi/mamabench},
\url{https://huggingface.co/datasets/nmrenyi/mamaretrieval}. Construction code:
\url{https://github.com/nmrenyi/mamabench},
\url{https://github.com/nmrenyi/mamaretrieval}. Released for research use; some
sources are non-commercial (e.g.\ AfriMed-QA, CC~BY-NC-SA~4.0), with each item's
license recorded in its per-source manifest.}
that close these gaps for the domain. Both were built
to supply the test data for a companion system paper~\cite{ren2026mamai}---a
deployed, fully on-device RAG assistant for nurse-midwives in Zanzibar---but
each is usable as a standalone instrument.
\begin{itemize}
\item \textbf{mamabench} (\S\ref{sec:mamabench}) is a scope-filtered
question-answering benchmark: 25{,}949 items assembled from seven
expert-authored sources across multiple-choice, short-answer, and
rubric-graded tracks, plus a 6{,}853-triple judge-calibration side-file---not
labels of our own, but HealthBench's physician annotations re-scoped to this
domain---for vetting an LLM judge before it is trusted to score the open-ended
items. Its scope classifier is cross-checked by agreement with a larger model
and parity against prior labels.
\item \textbf{mamaretrieval} (\S\ref{sec:mamaretrieval}) pairs 3{,}185 clinical
queries with graded ($0$--$6$) relevance labels over a 63{,}650-chunk
maternal-health guideline corpus---a chunk-level relevance benchmark at the
granularity a RAG system actually retrieves, built around a graded
clinical-relevance rubric grounded in IR relevance theory, with its judge
checked against a frontier reference model.
\end{itemize}
Three decisions shape both benchmarks, and they are the through-line of this
paper:
\begin{enumerate}
\item \textbf{Assemble and filter from expert-authored sources rather than
author new questions.} The clinical curation already exists in licensing
exams, physician panels, and published guidelines, so the benchmark's job is
reliable scoping and uniform packaging, not de-novo clinical authoring.
\item \textbf{Decompose clinical relevance into a graded, multi-dimensional
rubric rather than a binary relevant/not-relevant label.} To a midwife, a
chunk that names a drug and a chunk that gives its dose, route, and threshold
are not equally useful, and a benchmark that cannot tell them apart cannot
separate strong retrieval from adequate retrieval.
\item \textbf{Measure and disclose the limits of the labels}---scope-classifier
agreement, a frontier-judge check, and a pooling-completeness audit---rather
than present the benchmark as an oracle.
\end{enumerate}
The companion system paper uses these benchmarks to evaluate the deployed system
and reports those results; this paper documents how the benchmarks were
\emph{built and validated}, and is what the system paper cites for their
provenance. We do not reproduce the deployed-system evaluation here. mamabench
is assembled from external QA sources and is corpus-independent, whereas
mamaretrieval is coupled to a specific corpus chunk set (\S\ref{sec:corpus}).
Section~\ref{sec:limitations} is candid about what each benchmark does and does
not establish; the datasets and construction code are linked in the title
footnote.
\section{mamabench: A Scope-Filtered QA Benchmark}
\label{sec:mamabench}
mamabench evaluates how well a model answers the questions a nurse-midwife
asks---those spanning maternal, neonatal, child, and reproductive health---in
formats from multiple-choice to free-text clinical scenarios. We build it by
assembling such questions from seven existing, expert-authored sources and
filtering each to that scope. This poses three construction problems: assembling
and normalising the sources into one scorable schema, defining and applying the
scope reliably, and giving consumers a way to trust the LLM judge that scores the
open-ended items. We take each in turn.
\subsection{Source assembly}
The first decision is to assemble, not author: writing and validating
maternal-health questions from scratch needs clinicians, whereas the seven
sources already carry expert curation---medical-licensing boards, a pan-African
physician panel, Kenyan clinicians, and OpenAI's physician annotators---so the
benchmark's contribution is scoping and uniform packaging, not de-novo clinical
authoring.
Each source is normalised by a dedicated adapter into one canonical schema and
sorted into one of three tracks (Table~\ref{tab:mamabench-sources}):
\begin{itemize}
\item \textbf{Multiple-choice} (23{,}241 items), for breadth of clinical
knowledge: the OBGYN and paediatrics subset of MedMCQA~\cite{pal2022medmcqa},
the in-scope portion of MedQA-USMLE~\cite{jin2021medqa}, and the
multiple-choice questions of AfriMed-QA~\cite{olatunji2025afrimedqa}. This
track serves breadth; the open-ended tracks below carry the weight.
\item \textbf{Short-answer} (369 items), closest to the deployment setting:
312 nurse-written primary-care cases~\cite{kenya2025vignettes}, the
short-answer questions of AfriMed-QA~\cite{olatunji2025afrimedqa}, and a small
set of expert-crafted items designed to expose model
errors~\cite{whb2025}.
\item \textbf{Rubric-graded} (2{,}339 items): the in-scope portion of
HealthBench~\cite{arora2025healthbench}, whose items are scored against
physician-written rubric criteria rather than a single gold answer.
\end{itemize}
\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{6pt}
\renewcommand{\arraystretch}{1.12}
\caption{The seven sources of mamabench, by track. \emph{Yield} is the share of
a source's upstream pool that our scope classifier (\S\ref{sec:mamabench-scope})
keeps as in-scope; a dash marks sources that arrive already scoped to the
domain---MedMCQA and AfriMed-QA by their subject or specialty labels, the
Women's Health Benchmark by expert authoring---and so bypass the classifier
entirely.}
\label{tab:mamabench-sources}
\begin{tabular}{@{}llrr@{}}
\toprule
Track & Source & Items & Yield \\
\midrule
\multirow{3}{*}{Multiple-choice}
& MedMCQA (OBGYN \& paediatrics) & 18{,}508 & -- \\
& MedQA-USMLE & 4{,}199 & 29.2\% \\
& AfriMed-QA & 534 & -- \\
\midrule
\multirow{3}{*}{Short-answer}
& Kenya Clinical Vignettes & 312 & 61.5\% \\
& AfriMed-QA (short-answer) & 37 & -- \\
& Women's Health Benchmark & 20 & -- \\
\midrule
\multirow{3}{*}{Rubric-graded}
& HealthBench (oss\_eval) & 1{,}209 & 24.2\% \\
& HealthBench (consensus) & 872 & 23.8\% \\
& HealthBench (hard) & 258 & 25.9\% \\
\midrule
\multicolumn{2}{@{}l}{\textbf{Total}} & \textbf{25{,}949} & \\
\bottomrule
\end{tabular}
\end{table}
Normalisation is mostly mechanical, but one source needs more than that: the
AfriMed-QA multiple-choice items carry quality issues the others do not, and
there we draw the line at \emph{scorability}. Multiple-choice items are scored by
the answer's letter, and we drop only what that scoring cannot represent: items
with more than one correct option (a single answer index cannot encode them),
and items whose correct-answer text is duplicated under another letter---if the
same option text sits at two positions, a model could pick the right text yet be
marked wrong for reporting the duplicate letter. Cosmetic issues that do not
affect scoring (embedded
option-letter prefixes, non-ASCII typography, terse stems) are left untouched
and documented in the manifest. The principle is that the benchmark may refuse
an item it cannot score, but must never change what an item means.
\subsection{Defining scope with an LLM classifier}
\label{sec:mamabench-scope}
The central construction decision is how to decide whether an item is in scope.
We define scope as four positive categories---\textsc{maternal},
\textsc{neonatal}, \textsc{child health}, and
\textsc{sexual and reproductive health} (SRH)---plus \textsc{none}, and we
classify
each item by its \emph{primary clinical concept, not the patient's
demographics}. A question about managing a wrist fracture in a patient
who happens to be pregnant is \textsc{none}---the fracture is the concept, the
pregnancy incidental---whereas postpartum depression is \textsc{maternal},
because here the postpartum context \emph{is} the concept. The classifier is
\texttt{Qwen3.6-27B-FP8},\footnote{Model card:
\url{https://huggingface.co/Qwen/Qwen3.6-27B-FP8}.} run
with reasoning enabled at temperature 0; it applies this rule to every candidate
item and emits a structured verdict with its reasoning retained for audit.
Trusting an LLM to draw the scope boundary needs evidence, and we offer two
checks. First, \textbf{cross-model agreement}: re-classifying the HealthBench
\texttt{oss\_eval} items with the much larger
\texttt{Qwen3.5-397B-A17B-FP8}\footnote{Model card:
\url{https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8}.}---a stronger model from
the same family, so this checks self-consistency more than
independence---agrees with the 27B on
98.1\% of the 4{,}988 jointly classified items, and per-category counts differ
by under one percent; the handful of items on which the 27B fails to converge
are all \textsc{none} under the 397B, so they would be filtered out either way
and the net effect on in-scope counts is zero. Second, \textbf{parity against
prior labels}: the Kenya vignettes arrived with their own classifier labels,
and ours agree with them on 86.8\%, with the disagreements dominated by our rule
being deliberately stricter---rejecting items where only the patient's being
female, rather than the clinical concept, suggested OBGYN scope.
The per-source yield in Table~\ref{tab:mamabench-sources} is itself a useful
signal. The Kenyan primary-care vignettes are 61.5\% in scope, against
24--29\% for the general benchmarks: the maternal-health sources
concentrate exactly where the deployment lives, while the general sets
contribute a smaller, scope-filtered slice.
\subsection{A judge-calibration set, re-scoped from HealthBench}
\label{sec:mamabench-calibration}
The rubric-graded track is scored by an LLM judge, which raises the obvious
question of whether the judge can be trusted. Rather than answer that once for
our own judge, we ship the means to answer it for \emph{any} judge: a
calibration side-file of 6{,}853 (prompt, response, criterion) triples carrying
physicians' met / not-met labels. We want to be precise about provenance, since
it is easy to overclaim here. \emph{We did not produce these labels.} They are
the physician annotations HealthBench released for vetting rubric graders---its
meta-evaluation set~\cite{arora2025healthbench}; our only role is to re-scope
that set to this domain---keeping the triples whose prompts the scope classifier
(\S\ref{sec:mamabench-scope}) places in the four positive categories
(\textsc{child health} 3{,}239, \textsc{maternal} 2{,}284, \textsc{srh} 942,
\textsc{neonatal} 388, over 872 prompts)---and to package them as a ready-to-use
calibration file alongside the benchmark. A consumer runs a candidate judge over
these triples, compares its labels to the physicians', and decides whether to
trust it before scoring the benchmark; the companion system paper selects its
rubric judge in exactly this way~\cite{ren2026mamai}. One caveat: these 872
prompts \emph{are} the HealthBench-consensus rubric track's prompts---HealthBench
collected the side-file's physician labels on that same set---so calibrating a
judge here and then scoring the consensus track reuses the same prompts;
calibrate on the side-file, but read consensus-track scores with that overlap in
mind.
\section{mamaretrieval: A Chunk-Level Relevance Benchmark}
\label{sec:mamaretrieval}
mamaretrieval scores how well a retriever finds the guideline passages that
answer a clinical query, at the chunk granularity a RAG system actually
retrieves. Building it means choosing what to retrieve over (the corpus), what
queries to ask, how to label a (query, chunk) pair's relevance, and how to keep
those labels trustworthy at scale. We start with the corpus, then take query
construction, the relevance rubric, pooling, and judge reliability in turn.
\subsection{The guideline corpus}
\label{sec:corpus}
mamaretrieval is built over the same maternal-health guideline corpus the
deployed system retrieves from, which the companion system
paper~\cite{ren2026mamai} describes in full---its sources, construction, and
packaging. We give here only what the benchmark depends on: the corpus's scale,
its source-quality tiers, and the version coupling that makes the benchmark and
its corpus one artifact.
\paragraph{Scale and sources.}
The corpus bundle (v0.2.0) holds 63{,}650 section-aware chunks drawn from 87
documents spanning international clinical guidance (WHO, NICE, MSF, the
Hesperian midwifery references, and FIGO/EmONC materials), national and local
protocols (the Tanzania and Zanzibar Ministry of Health guidelines), and
standard obstetric and midwifery reference texts. Each chunk carries a canonical
header recording its source, page, and a stable chunk identifier.
\paragraph{Source-quality tiers.}
Sources are rated into quality tiers (from \emph{very high} down to
\emph{low--moderate}) by their clinical depth and relevance to the
Zanzibar OBGYN and midwifery scope. Query construction (below) uses these tiers
as sampling weights, drawing proportionally more from the clinically richest
sources.
\paragraph{Version coupling.}
Chunk identifiers are content-derived---a hash of the chunk text---so they
survive source renames and page drift but change whenever a chunk's text
changes. mamaretrieval references chunks by these identifiers, which couples it
to a specific \emph{chunk set} rather than to a specific embedding. The
distinction is not hypothetical: the deployed system moved from corpus bundle
v0.2.0 to v0.3.0, but that update only re-embedded the corpus---swapping the
on-device embedder---and left all 63{,}650 chunks and their identifiers
untouched, so every mamaretrieval label applies to the deployed v0.3.0 corpus
unchanged. A genuine re-chunking would change the identifiers and require
re-running the labelling pipeline; we therefore treat the
(chunk set, queries, labels) triple as a single versioned artifact rather than
versioning queries and labels on their own.
\subsection{Query construction}
\label{sec:mr-queries}
A retrieval benchmark needs queries, and for this corpus there are none to
mine---no search logs, no question bank tied to these guidelines. We generate
them. Each query in mamaretrieval is written by an LLM from a single corpus
chunk, asking a clinical question that the chunk answers; the chunk that seeded a
query is recorded as a known-relevant anchor for it.
The queries come out of a three-step funnel over the corpus:
\begin{enumerate}
\item \textbf{Tier-weighted sampling} draws 4{,}540 chunks, with each source's
quota set by its quality tier (\S\ref{sec:corpus})---the clinically richest
sources contribute proportionally more, so the query set inherits the corpus's
quality signal rather than its raw size distribution.
\item An \textbf{answerability gate} passes each sampled chunk to
\texttt{Qwen3.6-27B-FP8} (reasoning on, temperature 0), prompted to keep the
chunk only if a clinician could answer a specific question from it alone and,
when so, to write exactly that question.\footnote{The full filter-and-generate
prompt ships with the mamaretrieval construction code (title footnote).}
\item The kept chunks become the benchmark's \textbf{queries}: each generated
question gets a stable identifier (\texttt{q\_00001}, \dots), a pointer back to
its seed chunk, and the source and tier metadata that chunk carried.
\end{enumerate}
The gate keeps 3{,}185 of the 4{,}540 chunks (about 70\%), and the 30\% it
discards are themselves informative: assessment rubrics, learning objectives, and
table-of-contents fragments are rejected because no answerable clinical question
follows from them.
One limitation of this construction is worth flagging up front, because it
shapes which metrics we trust: a query written from a chunk tends to be phrased
like that chunk, which gives dense embedding retrievers an artificial edge over
lexical ones on the very chunk that seeded the query. We do not correct for this
anchor-text effect; we disclose it (\S\ref{sec:mr-judge}).
Section~\ref{sec:limitations} returns to the design's other limits, including
that the queries are clean and each grounded in a single chunk.
\subsection{The graded relevance rubric}
\label{sec:mr-rubric}
With queries and a corpus, the benchmark needs a relevance label for each
(query, chunk) pair: how useful is this chunk for answering this query? Our first
rubric made that judgement coarse---effectively relevant or not. It saturated. Take a query
asking for alternative iron and folic-acid dosages (Table~\ref{tab:rubric-example}):
a footnote giving the exact substitute formulation, a dosing section padded with
storage advice, and an adherence-counselling note that never states a dose are,
by a binary test, equally relevant---each is on topic, meaningful, and broadly
actionable. A
benchmark that cannot separate them cannot tell a retriever that surfaces the
dose from one that surfaces the filler.
So we replaced the binary label with a graded one. Each (query, chunk) pair is
scored
\[
\mathrm{score} = D_1 \times (D_2 + D_3 + D_4) \in \{0,\dots,6\},
\]
one gate and three graded dimensions:
\begin{description}
\item[$D_1$ --- topic (0/1).] Does the chunk address the query's clinical
question in the same context---the same timing window (antenatal,
intrapartum, postpartum)? A zero here forces a score of zero and halts
scoring, which also keeps the judge from spending tokens on off-topic chunks.
\item[$D_2$ --- meaningful content (0/1/2).] How much clinical depth the chunk
carries, from a bare mention to a multi-concept explanation.
\item[$D_3$ --- actionable guidance (0/1/2).] How specific the guidance is:
none, a general instruction (``give a uterotonic''), or an exact one
(``10 IU oxytocin IM'').
\item[$D_4$ --- density (0/1/2).] What fraction of the chunk is useful for
\emph{this} query, penalising a relevant passage buried in unrelated text.
\end{description}
A structural rule ties them together: actionable guidance presupposes some
meaningful content ($D_3 > 0 \Rightarrow D_2 > 0$).
This is not an ad-hoc scheme. A topical gate followed by complementary dimensions
that a fixed rule aggregates is the design of the TREC Precision Medicine
track~\cite{roberts2017precision}, and the topical/cognitive/situational split it
rests on is Saracevic's~\cite{saracevic2007relevance}---here $D_1$ is topical and
$D_3$, $D_4$ situational; criteria-based decomposition is also the direction of
recent LLM relevance-judgment work~\cite{farzi2025criteria}. Two further choices
are specific to using an LLM as the judge: an explicit topical gate, and
reasoning-before-verdict---the structured-output schema makes the judge emit its
reasoning before the four scores, as in G-Eval~\cite{liu2023geval}---both
following the now-standard use of a strong LLM as the relevance
assessor~\cite{upadhyay2024umbrela}. The judge is \texttt{Qwen3.5-397B-A17B-FP8}
at temperature 0, running the prompt reproduced in full in
Appendix~\ref{app:judge-prompt}; \S\ref{sec:mr-judge} reports how well its
labels hold up.
\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.2}
\caption{The graded rubric on one query (\texttt{q\_00041} in release
v0.2.0)---``What are the alternative dosages for iron and folic acid
supplementation?''---as scored by the production judge over four \emph{on-topic}
chunks ($D_1{=}1$ throughout). Text is quoted from the released labels with
markdown list formatting flattened to prose and elisions marked ``\dots'';
sources in brackets (MSF, the MSF obstetric manual; WHO~PCPNC, the WHO
pregnancy/childbirth guide; WHO~ANC, the 2016 WHO antenatal-care guideline). A
binary label cannot separate the first three; the graded score ranks them 6, 4,
and 2. The four chunks (top to bottom) have identifiers \texttt{318a18d9},
\texttt{ad7678f8}, \texttt{466a314b}, and \texttt{27b615ab}.}
\label{tab:rubric-example}
\begin{tabular}{@{}p{0.49\columnwidth}@{\hspace{5pt}}ccccc@{}}
\toprule
Retrieved chunk (verbatim) & $D_1$ & $D_2$ & $D_3$ & $D_4$ & Score \\
\midrule
\emph{[MSF]}~``200 mg ferrous sulfate (65 mg elemental iron) + 400 micrograms
folic acid tablets may be replaced by 185 mg ferrous fumarate (60 mg elemental
iron) + 400 micrograms folic acid tablets.'' & 1 & 2 & 2 & 2 & \textbf{6} \\
\addlinespace
\emph{[WHO PCPNC]}~``Give iron and folic acid. To all pregnant, postpartum and
post-abortion women: routinely once daily in pregnancy and until 3 months after
delivery or abortion; twice daily as treatment for anaemia (double dose). Check
woman's supply \dots\ at each visit and dispense 3 months supply. Advise to store
iron safely \dots'' & 1 & 2 & 1 & 1 & \textbf{4} \\
\addlinespace
\emph{[WHO PCPNC]}~``Motivate on adherence with treatments. Explore local
perceptions about iron treatment (examples of incorrect perceptions: making more
blood will make bleeding worse, iron will cause too large a baby).'' & 1 & 1 & 1
& 0 & \textbf{2} \\
\addlinespace
\emph{[WHO ANC]}~``Resources. Intermittent iron and folic acid supplementation
might cost a little less than daily iron and folic acid supplementation due to
the lower total weekly dose of iron.'' & 1 & 0 & 0 & 0 & \textbf{0} \\
\bottomrule
\end{tabular}
\end{table}
Table~\ref{tab:rubric-example} shows the graded rubric on that query. All four
chunks pass the topic gate; the first three also clear a binary rubric's bar
identically---each is meaningful and actionable---yet the graded rubric ranks
them apart. The footnote giving the exact alternative formulation scores 6,
specific and entirely on-query; a dosing section half-taken up by storage and
supply logistics scores 4 ($D_4=1$); and an adherence note that never states a
dose scores 2. The fourth chunk is on topic but pure cost commentary, and drops
to 0---the rubric counts resource text as no meaningful clinical content
($D_2=0$). The judge's reasoning ships with every label and makes such calls
explicit: of that last chunk, it notes the passage sits ``literally under a
\dots\ Resources heading and discusses cost,'' and scores $D_2=0$.
\subsection{Pooling and label completeness}
\label{sec:mr-pooling}
Scoring every chunk against every query is infeasible---63{,}650 chunks times
3{,}185 queries---so mamaretrieval labels a \emph{pool}: for each query, the
union of the top-ranked chunks returned by several retrievers, with every pooled
pair judged and every unpooled chunk treated as not-relevant. This is the
standard information-retrieval pooling protocol, and its one risk is well known:
if the pool misses relevant chunks, the labels under-count them. We measured that
risk rather than assume it away.
\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{6pt}
\renewcommand{\arraystretch}{1.15}
\caption{The six retrievers whose top-$k$ unions form mamaretrieval's label pool,
spanning retrieval paradigms. They \emph{construct} the labels; the
retriever scoreboard \emph{over} them---which one best serves the deployed
system---is reported in the companion paper~\cite{ren2026mamai}, not here. Lateon is the HuggingFace model
\texttt{lightonai/GTE-ModernColBERT-v1}.}
\label{tab:retrievers}
\begin{tabular}{@{}lll@{}}
\toprule
Retriever & Paradigm & Reference \\
\midrule
BM25 & lexical (sparse) & \cite{robertson2009bm25} \\
MedCPT & biomedical bi-encoder & \cite{jin2023medcpt} \\
Octen-Embedding-8B & general-purpose dense & model card\tablefootnote{\url{https://huggingface.co/Octen/Octen-Embedding-8B}} \\
voyage-4-large & frontier API dense & Voyage~AI\tablefootnote{\url{https://www.voyageai.com}} \\
Lateon & late-interaction & \cite{khattab2020colbert} \\
Gecko & on-device dense & \cite{lee2024gecko} \\
\bottomrule
\end{tabular}
\end{table}
The first pool took the top-10 chunks from three retrievers (BM25, MedCPT, and
Octen-Em\-bedding-8B): about 24.7 candidates per query and 78{,}571 judged pairs. A
completeness audit on a 100-query sample then judged a wider pool (five
retrievers at top-20, adding voyage-4-large and Lateon) and found the original
pool had captured only about half of all the relevant chunks known across both
judging rounds (lenient recall near 0.49): half the relevant chunks were being
labelled not-relevant by omission.
That finding drove the released label set. mamaretrieval v0.2.0 pools the top-20
chunks from six retrievers---the five above plus Gecko---chosen to span retrieval
paradigms so the pool is not biased toward any one
(Table~\ref{tab:retrievers}). Judged across all 3{,}185 queries, this yields
230{,}964 graded labels: the pool the companion system paper draws its label
provenance from. A residual gap remains---chunks that no retriever ranks in its top-20 are still
unjudged---but the same audit bounds it: against that same reference, widening
from the initial pool to the five-retriever top-20 union raised lenient recall
from about 0.49 to about 0.99, and the released pool adds a sixth retriever, so
it is near-saturated at the top of the ranking. Only absolute coverage-sensitive
numbers need the residual caveat (\S\ref{sec:mr-judge}).
\subsection{Judge reliability and metric guidance}
\label{sec:mr-judge}
A single LLM produces every relevance label, so the benchmark is only as
trustworthy as that judge. We check it against a frontier reference model
(Claude~Opus~4.7) on a 62-pair pilot: the production judge agrees on 59 of 62
pairs at the lenient cut (score $\ge 3$, any useful content) and 53 of 62 at the
strict cut (score $\ge 5$, complete and specific), and a stratified manual
spot-check of its reasoning traces found its calls defensible, including on hard
cases such as drug-name collisions across clinical contexts. Those two
cuts---lenient $\ge 3$ and strict $\ge 5$---are the relevance thresholds the
benchmark reports against.
We are deliberate about what this establishes. It is an agreement check against
another model, not against clinicians: it shows the judge is a reasonable
relevance classifier and that retriever \emph{rankings} are stable under it, but
it does not make any single label ground truth. Each label is one judge's
opinion, and absolute, per-item numbers carry more uncertainty than relative
comparisons.
That uncertainty, together with the pool incompleteness of
\S\ref{sec:mr-pooling}, is why we guide consumers on metric choice. Metrics that
turn on the presence or rank of a relevant chunk---Hit~Rate@$k$ (is any of the
top $k$ relevant?) and MRR (the mean over queries of $1/r$, where $r$ is the rank
of the \emph{first} relevant chunk)---are robust to a pool that misses some
relevant chunks: one hit suffices, and a label missed elsewhere cannot change
that a retrieved relevant chunk is relevant. nDCG@$k$ is
moderately sensitive: it credits every relevant chunk by grade and rank and
normalises against an ideal ranking built from the judged-relevant set, so
missing labels distort both. Precision@$k$ understates any retriever that
surfaces a relevant-but-unjudged chunk (counted as a miss), and Recall@$k$ is the
most exposed, since its denominator---the total count of relevant chunks---is
exactly what pooling under-counts. We recommend leading with the robust metrics
and reading the rest as relative comparisons with the completeness caveat
attached---the order the companion system paper follows when it evaluates
retrievers on mamaretrieval.
\section{Limitations}
\label{sec:limitations}
We have flagged limits where they arise; here we collect what a user should weigh
before relying on either benchmark.
\paragraph{No human gold standard.}
The scope of mamabench and every relevance label in mamaretrieval are decided by
LLMs, not clinicians. We anchor them externally---cross-model agreement and prior
labels for the scope classifier (\S\ref{sec:mamabench-scope}), a frontier
reference model for the relevance judge (\S\ref{sec:mr-judge})---and we ship
HealthBench's physician labels so a consumer can vet a rubric judge
(\S\ref{sec:mamabench-calibration}). But no midwife reviewed either benchmark at
scale, and those physician labels come from HealthBench's annotator pool, not
midwifery specialists. Each label is best read as one judge's opinion: model
and retriever \emph{rankings} are robust, absolute per-item numbers less so.
\paragraph{mamaretrieval queries are clean and single-chunk.}
Every query is LLM-written from one chunk and answerable from it
(\S\ref{sec:mr-queries}), with two consequences. First, the benchmark exercises
single-passage retrieval: it does not test questions that can only be answered by
combining several chunks, nor robustness to the noisy, code-switched, or mistyped
queries a hurried nurse might enter. Second, because a query reads like its
source chunk, dense retrievers gain an anchor-text edge that inflates their
absolute scores---another reason to lead with the pooling-robust metrics
(\S\ref{sec:mr-judge}).
\paragraph{The relevance labels are pooled.}
They cover the six-retriever top-20 union; a chunk no retriever surfaced is
unjudged and counted not-relevant (\S\ref{sec:mr-pooling}). Coverage-sensitive
metrics---Recall and Precision---carry that bias, which is why we report
Hit~Rate and MRR first.
\paragraph{Scope, language, and transfer.}
Both benchmarks are English-only and scoped to maternal, neonatal, child, and
reproductive health for a Zanzibar nurse-midwife setting; the numbers need not
transfer to other languages, specialties, or care settings. A few mamabench items
the scope classifier could not converge on are disclosed and have zero net effect
on in-scope counts (\S\ref{sec:mamabench-scope}).
\section{Conclusion}
\label{sec:conclusion}
We release two benchmarks for evaluating medical retrieval-augmented generation
in maternal, neonatal, child, and reproductive health: \textbf{mamabench}, a
scope-filtered QA set assembled from seven expert sources, and
\textbf{mamaretrieval}, a
chunk-level relevance benchmark with a graded rubric that distinguishes a chunk
that answers the query from one that is merely on-topic. Three choices run
through both: assemble and filter rather than author, decompose relevance into a
graded rubric rather than a binary label, and measure and disclose the limits of
the labels rather than treat them as an oracle. The companion system paper uses
them to locate where a deployed on-device assistant succeeds and fails, but they
also stand alone---resources for maternal-health RAG that did not previously
exist. The
clearest next steps are the ones we left open: queries that demand multi-passage
synthesis and survive noisy phrasing, and validation against midwives rather
than against other models.
\section{Relevance-Judge Prompt}
\label{app:judge-prompt}
The prompt the relevance judge (\texttt{Qwen3.5-397B-A17B-FP8}) runs for every
(query, chunk) pair is reproduced below: the system message carrying the
four-dimension rubric, the structural rule, the output schema, and four worked
examples, followed by the per-pair user template. It is the canonical prompt
from \texttt{scripts/judge\_relevance.py} in the construction
repository,\footnote{\url{https://github.com/nmrenyi/mamaretrieval/blob/main/scripts/judge_relevance.py}}
and ships verbatim with the dataset as
\texttt{audit/judge\_relevance\_prompt.txt}; its hash is pinned in the release
manifest. The box rules and a few Unicode symbols (e.g.\ $\ge$, $\times$,
$\mu$) are rendered in ASCII here, and long lines are wrapped to the column;
the content is otherwise verbatim.
\lstinputlisting{appendix/judge_relevance_prompt.txt}
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{17}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{} and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showURL}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showURL #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[Arora et~al\mbox{.}(2025)]%
{arora2025healthbench}
\bibfield{author}{\bibinfo{person}{Rahul~K. Arora}, \bibinfo{person}{Jason
Wei}, \bibinfo{person}{Rebecca Soskin~Hicks}, {et~al\mbox{.}}}
\bibinfo{year}{2025}\natexlab{}.
\newblock \bibinfo{title}{{HealthBench}: Evaluating Large Language Models
Towards Improved Human Health}.
\newblock
\showeprint[arxiv]{2505.08775}~[cs.CL]
\urldef\tempurl%
\url{https://arxiv.org/abs/2505.08775}
\showURL{%
\tempurl}
\bibitem[Farzi and Dietz(2025)]%
{farzi2025criteria}
\bibfield{author}{\bibinfo{person}{Naghmeh Farzi} {and} \bibinfo{person}{Laura
Dietz}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \bibinfo{title}{Criteria-Based {LLM} Relevance Judgments}.
\newblock
\showeprint[arxiv]{2507.09488}~[cs.IR]
\urldef\tempurl%
\url{https://arxiv.org/abs/2507.09488}
\showURL{%
\tempurl}
\bibitem[Jin et~al\mbox{.}(2021)]%
{jin2021medqa}
\bibfield{author}{\bibinfo{person}{Di Jin}, \bibinfo{person}{Eileen Pan},
\bibinfo{person}{Nassim Oufattole}, \bibinfo{person}{Wei-Hung Weng},
\bibinfo{person}{Hanyi Fang}, {and} \bibinfo{person}{Peter Szolovits}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{What Disease Does This Patient Have? A Large-Scale
Open Domain Question Answering Dataset from Medical Exams}.
\newblock \bibinfo{journal}{\emph{Applied Sciences}} \bibinfo{volume}{11},
\bibinfo{number}{14} (\bibinfo{year}{2021}).
\newblock
\urldef\tempurl%
\url{https://github.com/jind11/MedQA}
\showURL{%
\tempurl}
\bibitem[Jin et~al\mbox{.}(2023)]%
{jin2023medcpt}
\bibfield{author}{\bibinfo{person}{Qiao Jin}, \bibinfo{person}{Won Kim},
\bibinfo{person}{Qingyu Chen}, \bibinfo{person}{Donald~C. Comeau},
\bibinfo{person}{Lana Yeganova}, \bibinfo{person}{W.~John Wilbur}, {and}
\bibinfo{person}{Zhiyong Lu}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{{MedCPT}: Contrastive Pre-trained Transformers with
large-scale {PubMed} search logs for zero-shot biomedical information
retrieval}.
\newblock \bibinfo{journal}{\emph{Bioinformatics}} \bibinfo{volume}{39},
\bibinfo{number}{11} (\bibinfo{year}{2023}), \bibinfo{pages}{btad651}.
\newblock
\bibitem[Khattab and Zaharia(2020)]%
{khattab2020colbert}
\bibfield{author}{\bibinfo{person}{Omar Khattab} {and} \bibinfo{person}{Matei
Zaharia}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{{ColBERT}: Efficient and Effective Passage Search
via Contextualized Late Interaction over {BERT}}. In
\bibinfo{booktitle}{\emph{Proceedings of the 43rd International ACM SIGIR
Conference on Research and Development in Information Retrieval (SIGIR)}}.
\newblock
\bibitem[Lee et~al\mbox{.}(2024)]%
{lee2024gecko}
\bibfield{author}{\bibinfo{person}{Jinhyuk Lee}, \bibinfo{person}{Zhuyun Dai},
\bibinfo{person}{Xiaoqi Ren}, {et~al\mbox{.}}}
\bibinfo{year}{2024}\natexlab{}.
\newblock \bibinfo{title}{Gecko: Versatile Text Embeddings Distilled from Large
Language Models}.
\newblock
\showeprint[arxiv]{2403.20327}~[cs.CL]
\urldef\tempurl%
\url{https://arxiv.org/abs/2403.20327}
\showURL{%
\tempurl}
\bibitem[Liu et~al\mbox{.}(2023)]%
{liu2023geval}
\bibfield{author}{\bibinfo{person}{Yang Liu}, \bibinfo{person}{Dan Iter},
\bibinfo{person}{Yichong Xu}, \bibinfo{person}{Shuohang Wang},
\bibinfo{person}{Ruochen Xu}, {and} \bibinfo{person}{Chenguang Zhu}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{{G-Eval}: {NLG} Evaluation using {GPT-4} with
Better Human Alignment}. In \bibinfo{booktitle}{\emph{Proceedings of the 2023
Conference on Empirical Methods in Natural Language Processing (EMNLP)}}.
\newblock
\bibitem[Mwaniki et~al\mbox{.}(2025)]%
{kenya2025vignettes}
\bibfield{author}{\bibinfo{person}{Paul Mwaniki}, \bibinfo{person}{Wycliffe
Musau}, \bibinfo{person}{Lynda Isaaka}, \bibinfo{person}{Conrad Wanyama},
\bibinfo{person}{Vinod Menon}, \bibinfo{person}{Alastair~K. Denniston},
\bibinfo{person}{Xiaoxuan Liu}, \bibinfo{person}{Mphatso Emmanual-Fabula},
\bibinfo{person}{Gerald Williams}, \bibinfo{person}{Bilal~A. Mateen}, {and}
\bibinfo{person}{Ambrose Agweyu}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \bibinfo{title}{Benchmarking Large Language Models and Clinicians
Using Locally Generated Primary Healthcare Vignettes in {Kenya}}.
\newblock
\urldef\tempurl%
\url{https://www.medrxiv.org/content/10.1101/2025.10.25.25338798v1}
\showURL{%
\tempurl}
\newblock
\shownote{medRxiv preprint 2025.10.25.25338798}.
\bibitem[Nimo et~al\mbox{.}(2025)]%
{olatunji2025afrimedqa}
\bibfield{author}{\bibinfo{person}{Charles Nimo}, \bibinfo{person}{Tobi
Olatunji}, {et~al\mbox{.}}} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{{AfriMed-QA}: A Pan-African, Multi-Specialty,
Medical Question-Answering Benchmark Dataset}. In
\bibinfo{booktitle}{\emph{Proceedings of the 63rd Annual Meeting of the
Association for Computational Linguistics (ACL)}}.
\bibinfo{pages}{1948--1973}.
\newblock
\urldef\tempurl%
\url{https://aclanthology.org/2025.acl-long.96/}
\showURL{%
\tempurl}
\bibitem[Pal et~al\mbox{.}(2022)]%
{pal2022medmcqa}
\bibfield{author}{\bibinfo{person}{Ankit Pal}, \bibinfo{person}{Logesh~Kumar
Umapathi}, {and} \bibinfo{person}{Malaikannan Sankarasubbu}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{{MedMCQA}: A Large-scale Multi-Subject Multi-Choice
Dataset for Medical Domain Question Answering}. In
\bibinfo{booktitle}{\emph{Proceedings of the Conference on Health, Inference,
and Learning (CHIL)}}.
\newblock
\urldef\tempurl%
\url{https://huggingface.co/datasets/openlifescienceai/medmcqa}
\showURL{%
\tempurl}
\bibitem[{Ren Yi}(2026)]%
{ren2026mamai}
\bibfield{author}{\bibinfo{person}{{Ren Yi}}.} \bibinfo{year}{2026}\natexlab{}.
\newblock \bibinfo{title}{{MAM-AI}: An On-Device Medical Retrieval-Augmented
Generation System for Nurses and Midwives in Zanzibar}.
\newblock
\showeprint[arxiv]{2606.29580}~[cs.CL]
\newblock
\shownote{Companion paper}.
\bibitem[Roberts et~al\mbox{.}(2017)]%
{roberts2017precision}
\bibfield{author}{\bibinfo{person}{Kirk Roberts}, \bibinfo{person}{Dina
Demner-Fushman}, \bibinfo{person}{Ellen~M. Voorhees},
\bibinfo{person}{William~R. Hersh}, \bibinfo{person}{Steven Bedrick},
\bibinfo{person}{Alexander~J. Lazar}, {and} \bibinfo{person}{Shubham Pant}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Overview of the {TREC} 2017 Precision Medicine
Track}. In \bibinfo{booktitle}{\emph{Proceedings of the Twenty-Sixth Text
REtrieval Conference (TREC)}}.
\newblock
\bibitem[Robertson and Zaragoza(2009)]%
{robertson2009bm25}
\bibfield{author}{\bibinfo{person}{Stephen Robertson} {and}
\bibinfo{person}{Hugo Zaragoza}.} \bibinfo{year}{2009}\natexlab{}.
\newblock \showarticletitle{The Probabilistic Relevance Framework: {BM25} and
Beyond}.
\newblock \bibinfo{journal}{\emph{Foundations and Trends in Information
Retrieval}} \bibinfo{volume}{3}, \bibinfo{number}{4} (\bibinfo{year}{2009}),
\bibinfo{pages}{333--389}.
\newblock
\bibitem[Saracevic(2007)]%
{saracevic2007relevance}
\bibfield{author}{\bibinfo{person}{Tefko Saracevic}.}
\bibinfo{year}{2007}\natexlab{}.
\newblock \showarticletitle{Relevance: A Review of the Literature and a
Framework for Thinking on the Notion in Information Science. {Part II}}.
\newblock \bibinfo{journal}{\emph{Journal of the American Society for
Information Science and Technology}} \bibinfo{volume}{58},
\bibinfo{number}{13} (\bibinfo{year}{2007}), \bibinfo{pages}{1915--1933}.
\newblock
\bibitem[{The Lumos AI Labs}(2025)]%
{whb2025}
\bibfield{author}{\bibinfo{person}{{The Lumos AI Labs}}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \bibinfo{title}{Women's Health Benchmark ({WHB})}.
\newblock
\urldef\tempurl%
\url{https://huggingface.co/datasets/TheLumos/WHB_subset}
\showURL{%
\tempurl}
\newblock
\shownote{HuggingFace dataset (unrefereed); expert-authored
maternal/reproductive-health question set}.
\bibitem[Upadhyay et~al\mbox{.}(2024)]%
{upadhyay2024umbrela}
\bibfield{author}{\bibinfo{person}{Shivani Upadhyay}, \bibinfo{person}{Ronak
Pradeep}, \bibinfo{person}{Nandan Thakur}, \bibinfo{person}{Nick Craswell},
{and} \bibinfo{person}{Jimmy Lin}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \bibinfo{title}{{UMBRELA}: The Open-Source Reproduction of the {Bing}
Relevance Assessor}.
\newblock
\showeprint[arxiv]{2406.06519}~[cs.IR]
\urldef\tempurl%
\url{https://arxiv.org/abs/2406.06519}
\showURL{%
\tempurl}
\bibitem[Xiong et~al\mbox{.}(2024)]%
{xiong2024benchmarking}
\bibfield{author}{\bibinfo{person}{Guangzhi Xiong}, \bibinfo{person}{Qiao Jin},
\bibinfo{person}{Zhiyong Lu}, {and} \bibinfo{person}{Aidong Zhang}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Benchmarking Retrieval-Augmented Generation for
Medicine}. In \bibinfo{booktitle}{\emph{Findings of the Association for
Computational Linguistics: ACL 2024}}.
\newblock
\end{thebibliography}