MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations
Extracted source text · arXiv 2410.14204
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
% \usepackage[review]{acl}
\usepackage[final]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{natbib}
\usepackage{multirow}
\usepackage{subcaption}
\usepackage{hyperref}
\usepackage{amsmath}
\usepackage[skins,xparse,breakable,most]{tcolorbox}
% \usepackage[most]{tcolorbox}
\usepackage{fvextra}
\usepackage{paralist}
% \usepackage{tikz}
% \def\check{\tikz\fill[scale=0.4](0,.35) -- (.25,0) -- (1,.7) -- (.25,.15) -- cycle;}
\usepackage{amssymb}% http://ctan.org/pkg/amssymb
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\textcolor{green}{\ding{51}}}%
\newcommand{\xmark}{\textcolor{red}{\ding{55}}}
\newcommand{\dataset}{{MediTOD}}
\usepackage{todonotes}
\newcommand{\goonjan}[1]{\textbf{\textcolor{red}{[Goonjan: #1]}}}
\newcommand{\mausam}[1]{\textbf{\textcolor{blue}{[Mausam: #1]}}}
\newcommand{\vishal}[1]{\textbf{\textcolor{violet}{[Vishal: #1]}}}
\newcommand{\dinesh}[1]{\textbf{\textcolor{orange}{[Dinesh: #1]}}}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
% \title{\dataset{}: An English Dialogue Dataset for Medical History Taking with Detailed Annotations}
\title{\dataset{}: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations}
% A Comprehensive dialogue Dataset for Medical Task-Oriented dialogue System
% Medical History Taking
% MediTOD: A medical TOD dataset with fine-grained annotations
% MediTOD: A TOD dataset for realistic medical history taking
% A medical history taking dialogue dataset with UMLS linked annotations.
% A English medical history taking dialogue dataset with UMLS linked annotations.
% Convert the table to indicate whether datasets have annotations are not
%
% \author{First Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\\And
% Second Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\}
\author{
\textbf{Vishal Vivek Saley\textsuperscript{1}},
\textbf{Goonjan Saha\thanks{Work done when authors were at IIT Delhi.}\textsuperscript{1}},
\textbf{Rocktim Jyoti Das\footnotemark[1]\textsuperscript{3}},
\\
\textbf{Dinesh Raghu\textsuperscript{2}},
\textbf{Mausam\textsuperscript{1}}
\\
\textsuperscript{1}Indian Institute of Technology, Delhi\\
\textsuperscript{2}IBM Research, New Delhi, India\\
\textsuperscript{3}MBZUAI
\\
Vishal.Vivek.Saley@cse.iitd.ac.in, saha.goonjan@gmail.com \\
rocktimjyotidas@gmail.com, diraghu1@in.ibm.com, mausam@cse.iitd.ac.in
}
% \author{Vishal Vivek Saley $^{1}$, Goonjan Saha\thanks{Work done when authors were at IIT Delhi.} $^{1}$ \\
% \textbf{Rocktim Jyoti Das}\footnotemark[1] $^{1 \hspace{0.1cm} 3}$, \textbf{Dinesh Raghu} $^{2}$ and \textbf{Mausam} $^{1}$ \\
% $^{1}$ Indian Institute of Technology, Delhi\\
% $^{2}$ IBM Research, New Delhi, India\\
% $^{3}$ MBZUAI\\
% Vishal.Vivek.Saley@cse.iitd.ac.in, saha.goonjan@gmail.com \\
% rocktimjyotidas@gmail.com, diraghu1@in.ibm.com \\
% mausam@cse.iitd.ac.in}
\begin{document}
\maketitle
\begin{abstract}
% Medical task-oriented dialogue systems can assist doctors by collecting patient medical history, aiding in diagnosis, or guiding treatment selection, thereby reducing doctor burnout and expanding access to medical services. However, doctor-patient dialogue datasets are not readily available, primarily due to privacy regulations. Moreover, existing datasets lack fine-grained annotations, such as symptom onset, progression, and severity, which are important for accurate diagnosis. Finally, most existing datasets are non-English, limiting their utility for the larger research community.
% \todo{D: We may need to rewrite this paragraph}
% In response, we introduce \dataset{}, a new dataset of doctor-patient dialogues in English for medical history-taking task. Through collaboration with doctors, we devise a questionnaire-based labeling scheme tailored to the medical domain. Then, medical professionals create the dataset with high-quality fine-grained annotations. We establish benchmarks in supervised and few-shot settings on \dataset{} for natural language understanding, policy learning, and natural language generation subtasks, evaluating models from both TOD and biomedical domains. We make \dataset{} publicly available for future research.
Medical task-oriented dialogue systems can assist doctors by collecting patient medical history, aiding in diagnosis, or guiding treatment selection, thereby reducing doctor burnout and expanding access to medical services. However, doctor-patient dialogue datasets are not readily available, primarily due to privacy regulations. Moreover, existing datasets lack comprehensive annotations involving medical slots and their different attributes, such as symptoms and their onset, progression, and severity. These comprehensive annotations are crucial for accurate diagnosis. Finally, most existing datasets are non-English, limiting their utility for the larger research community.
In response, we introduce \dataset{}, a new dataset of doctor-patient dialogues in English for the medical history-taking task. Collaborating with doctors, we devise a questionnaire-based labeling scheme tailored to the medical domain. Then, medical professionals create the dataset with high-quality comprehensive annotations, capturing medical slots and their attributes. We establish benchmarks in supervised and few-shot settings on \dataset{} for natural language understanding, policy learning, and natural language generation subtasks, evaluating models from both TOD and biomedical domains. We release \dataset{} resources for future research.
\end{abstract}
\section{Introduction}
% \vishal{
% \begin{itemize}
% \item Shall we make a qualitative comparison against translated ReMeDi? - Discuss
% \item Mention an utterance can have many intent-slots?
% \item Add labeling tool to contribution.
% \item Add few-shot evaluation to contribution.
% \item Check average salaries for annotations and compare with our payment. Highlight we are paying appropriately and as per regulations.
% \end{itemize}
% }
Medical task-oriented dialogue (TOD) systems are gaining importance in modern healthcare by assisting doctors in patient history-taking, diagnosis suggestions, and treatment recommendations, alleviating doctor burnout and extending the reach of medical services \citep{valizadeh2022ai,kearns2019systematic,laranjo2018conversational}. Recently, medical TOD systems have witnessed significant progress, particularly in individual sub-modules such as natural language understanding (NLU) \citep{Zhang2020MIEAM}, policy learning (POL) \citep{Tchango2022DDXPlusAN}, and natural language generation (NLG) \citep{Yan2021ReMeDiRF}. %Several datasets have been proposed to facilitate research in these areas.
Most of the existing medical TOD datasets contain annotations required for training only one sub-component \citep{Wei2018TaskorientedDS, He2020MedDialogTL, fansi2022towards}, and only a few include annotations for all sub-components, thereby enabling the construction of a complete dialogue system \citep{Yan2021ReMeDiRF, Chen2022ABF}.
\begin{figure}[t]
\centering
\includegraphics[width=0.48\textwidth]{latex/images/intro_dialog3.pdf}
% \caption{Doctor-patient dialogue from \dataset{} with fine-grained annotations.} \label{fig:intro_dialog}
\caption{A dialogue turn with annotations labeled using as Comprehensive Medical Attribute Schema (CMAS) in \dataset{}, compared to key-value pairs.}\label{fig:intro_dialog}
% at each turn.}\label{fig:intro_dialog}
\end{figure}
% \todo{can remove second utt and other datasets represent slots}
Training NLU, POL, and NLG sub-modules requires dialogues to be annotated with intents, slots, dialogue states, and actions. In existing medical TOD datasets, slots are primarily represented as key-value pairs. However, this simplistic representation often fails to capture the inherent complexity of the medical domain. For example, in Figure \ref{fig:intro_dialog}, the patient expresses two symptoms (\textit{pharyngitis} and \textit{fever}) along with its onset (\textit{past four days} and \textit{last two days}). Existing annotation schemes would fail to capture the symptom-onset link and only represent four independent key-value pairs.
To overcome this problem, we define a new slot schema, named Comprehensive Medical Attribute Schema (CMAS), that captures the inherent complexity of the slots in the medical domain. It maintains multiple attributes specific to each slot type for better representation. For example in Figure \ref{fig:intro_dialog}, `onset' is treated as an attribute of the slot `symptom', establishing a more accurate patient profile.
%, and diagnose the underlying medical condition.
\begin{comment}
Training NLU, POL, and NLG sub-modules in a medical TOD system necessitates doctor-patient dialogue annotation with intent and slot annotation, dialogue state annotation, and action annotation. In existing TOD datasets, slot annotations in user utterances have mostly been represented as slot-value pairs. But, many slots, such as symptoms in the medical domain, do not fit in such a slot-value schema.
% In addition to value, they also should be linked to other associated attributes.
In addition to value, they also participate in n-ary relationships with other associated attributes.
For example, consider the doctor-patient dialogue in figure \ref{fig:intro_dialog}.
Here, symptoms \em{pharyngitis} and \em{fever} have onset attributes of \emph{past four days} and \emph{last two days}, respectively. Such attributes must linked to appropriate slots for precisely representing symptoms, establishing an accurate patient profile, and diagnosing the underlying medical condition.
% Here, attributes such as the onset of symptoms (2 days) and location of pain (middle of the chest) are important for precisely representing the symptom and thereby correctly diagnosing the underlying medical condition.
Further, many slot values and attributes in the dialogue can be expressed in many surface forms. Hence, to generalize better, it is essential for these surface forms to be canonicalized using standard medical vocabularies.
In figure \ref{fig:intro_dialog}, pharyngitis can have different surface forms such as \emph{soreness in my throat} in the dialogues.
% For example, pharyngitis in figure \ref{fig:intro_dialog} can have different surface forms such as \textit{soreness in my throat} in the dialogues.
% For example, dyspnea in Figure \ref{fig:intro_dialog} can have multiple surface forms such as \textit{barely breath} in the dialogues.
Ensuring the canonicalization of medical attributes is important for creating useful benchmarks, particularly for evaluation purposes.
Table \ref{tab:datasets} summarizes medical dialogue datasets annotated with at least one component of the medical TOD system. While most datasets provide canonicalized annotations, they often lack detailed attributes like onset and location. ReMeDi \citep{Yan2021ReMeDiRF} and Code-Mixed \citep{dowlagar2023code} include these attributes as separate slots but lack canonicalization. Further, they do not link attributes to specific symptoms and capture multi-dimensional relations in the dialogue .
%Finally, most dataset are from a single language demographic (non-English), limiting their usability in broader research contexts.
% Several challenges arise in the research of medical TOD systems. While many datasets exist, most are from a single language demographic (non-English), limiting their usability in broader research contexts. Moreover, English datasets curated from online healthcare forums often lack conversational nature, which is crucial for realistic dialogues \citep{Varshney2022CDialogAM}. Additionally, existing datasets frequently lack fine-grained dialogue act annotations necessary for training robust TOD systems. Notably, only a few datasets, such as \citep{Yan2021ReMeDiRF} and \citep{Chen2022ABF}, offer such annotations, yet they are in Chinese.
% , essential for realistic training data,
% Creating a new dataset for medical TOD systems with fine-grained dialogue acts presents significant challenges. Doctor-patient dialogues are protected under privacy laws, rendering them unavailable for public use. Unlike traditional TOD systems, which utilize crowd-sourcing for data curation \citep{budzianowski2018multiwoz, moghe2018towards, byrne2019taskmaster, byrne2021tickettalk}, medical dialogues require involvement from healthcare professionals due to the specialized language and context. Traditional TOD labeling schemes often fall short in capturing the intricacies in the medical dialogues, rendering the labeling process labor-intensive and costly.
\end{comment}
This paper presents \dataset{}, the first publicly available \textit{English} medical TOD dataset annotated in the CMAS ontology.
%, which captures the inherent complexity of the slots in medical domain.
\dataset{} comprises dialogues from staged doctor-patient interviews in objective structured clinical examination format \citep{harden1979assessment, fareez2022dataset}. By leveraging these high-quality dialogues, privacy concerns are mitigated while providing realistic medical scenarios.
Collaborating closely with doctors, we develop a questionnaire-based annotation framework to collect slots and corresponding attributes.
% Collaborating closely with doctors, we develop a questionnaire-based annotation framework to incorporate fine-grained detail in our label.
The annotations are further canonicalized, where possible, to precise medical concepts in Unified Medical Language System (UMLS). Through the release of \dataset{}, we aim to provide a valuable resource for advancing research in medical TOD systems. Our main contributions are as follows.
\begin{compactenum}
% \item We release \dataset{}, a dataset of doctor-patient dialogues with 20,519 utterances annotated, in collaboration with doctors, using a questionnaire-based labeling scheme designed for the medical domain.
\item We release \dataset{}, a dataset of doctor-patient dialogues with 22,503 utterances annotated, in collaboration with doctors, using a questionnaire-based labeling scheme designed for the medical domain.
\item To label utterances in \dataset{}, we develop an annotation portal based on our questionnaire-based scheme. We release this portal alongside the dataset and invite researchers to contribute further to the dataset, enhancing its richness and diversity.
\item We establish baselines in supervised and few-shot settings for NLU, POL, and NLG TOD tasks on \dataset{} dataset by evaluating representative models from TOD and bio-medical literature. Our results showcase the challenging nature of the dataset.
\end{compactenum}
We make \dataset{} resources publicly available at \url{https://github.com/dair-iitd/MediTOD}.
% \todo{Use comprehensive annotations in place of fine grained}
% \todo{changes to table 1: add no of utterance, add the sub-modules (NLU, POL, NLG) for which annotation is available}
\section{Related Work}
\begin{table*}[]
\centering
\small
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|c|ccc|cc}
\toprule
\multirow{2}{*}{\textbf{Datasets}} & \multirow{2}{*}{\textbf{Language}} & \multicolumn{3}{c|}{\textbf{Annotations}} & \textbf{\#utterances/} & \multirow{2}{*}{\textbf{\#utterances}} \\ \cmidrule{3-5}
& & \textbf{All TOD Tasks} & \textbf{Comprehensive} & \textbf{Canonicalized} & \textbf{dialogue} & \\ \midrule
CMDD \citep{Lin2019EnhancingDS} & zh & \xmark & \xmark & \cmark & 42.09 & 87,000 \\
MSL \citep{Shi2020UnderstandingMC} & zh & \xmark & \xmark & \cmark & NA & 2,652 \\
MIE \citep{Zhang2020MIEAM} & zh & \xmark & \xmark & \cmark & 16.26 & 18,212 \\
IntRec \citep{rojowiec2020intent} & de & \xmark & \xmark & \cmark & 57.071 & 2,397 \\
ReMeDi \citep{Yan2021ReMeDiRF} & zh & \cmark & \xmark & \xmark & 16.34 & 25,446 \\
DialoAMC \citep{Chen2022ABF} & zh & \cmark & \xmark & \cmark & 40.02 & 1,64,731 \\
Code-Mixed \citep{dowlagar2023code} & te,en & \cmark & \xmark & \xmark & 9.75 & 29,294 \\ \midrule
% \dataset{} (Ours) & en & \cmark & \cmark & \cmark & 95.33 & 20,519 \\ \bottomrule
\dataset{} (Ours) & en & \cmark & \cmark & \cmark & 95.57 & 22,503 \\ \bottomrule
\end{tabular}%
}
\caption{Publicly available medical dialogue datasets with annotations. \dataset{} is the only English dataset that features both comprehensive (capturing slots and their low-level attributes together) and canonicalized annotations. The language codes "en," "zh," "de," and "te" represent English, Chinese, German, and Telugu, respectively.}
\label{tab:datasets}
\end{table*}
% \begin{table*}[]
% \centering
% \resizebox{\textwidth}{!}{%
% \begin{tabular}{l|c|c|c|c|c}
% \toprule
% \multirow{2}{*}{Dataset} & \multirow{2}{*}{Language} & \multirow{2}{*}{Publicly Available} & \multicolumn{2}{c|}{dialogue Annotations} & \multirow{2}{*}{Avg. Utterances/dialogue} \\ \cmidrule{4-5}
% & & & Fine-grained & Canonicalized & \\ \midrule
% CMDD \citep{Lin2019EnhancingDS} & zh & \cmark & \xmark & \cmark & 42.09 \\
% SAT \citep{Du2019ExtractingSA} & en & \xmark & \xmark & \cmark & N/A \\
% MSL \citep{Shi2020UnderstandingMC} & zh & \cmark & \xmark & \cmark & N/A \\
% MIE \citep{Zhang2020MIEAM} & zh & \cmark & \xmark & \cmark & 16.19 \\
% IntRec \citep{rojowiec2020intent} & de & \cmark & \xmark & \cmark & 57.07 \\
% ReMeDi \citep{Yan2021ReMeDiRF} & zh & \cmark & \cmark & \xmark & 16.13 \\
% DialoAMC \citep{Chen2022ABF} & zh & \cmark & \xmark & \cmark & 40.02 \\
% Code-Mixed \citep{dowlagar2023code} & te,en & \cmark & \cmark & \xmark & 9.77 \\ \midrule
% \dataset{} (Ours) & en & \cmark & \cmark & \cmark & 95.33 \\ \bottomrule
% \end{tabular}
% }
% \caption{Medical dialogue datasets with annotations. \dataset{} is the only public available English dataset with fine-grained and canonicalized annotations.}
% \label{tab:datasets}
% \end{table*}
%Why is it difficult to obtain a doctor-patient medical conversations.
%1. HIPPA compliance does not allow public release of confidential patient information including the dialogues with the doctor.
%2. Mainstream TOD systems curate data by hiring crowd-source workers. These worker create dataset either in WOZ fashion (cite woz) or by self-play (cite task master 1, 2, 3 and BiTOD). However, in speciality domain such as medicine the crowd source worker need to be medical professional. This makes the task prohibitively expensive.
%-----------------------------------
%The curation of medical datasets faces inherent challenges. Firstly, there are strict medical privacy regulations such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States that prevent the public release of such sensitive patient information. Further, unlike most TOD systems that can be curated using the Wizard of Oz method, these datasets need the involvement of medical professionals for being appropriate for use.
%Despite these obstacles, various medical conversational datasets have been compiled for research purposes.
%------------------
\paragraph{Task-Oriented Dialogue (TOD) Systems: }
General domain TOD systems that assist users in completing tasks such as restaurant table reservation and flight booking often follow a modular design, consisting of three modules -- natural language understanding (NLU), dialogue policy learning (POL), and natural language generation (NLG) \citep{young2013pomdp, wen2016network}. In recent years, there has been significant progress in the field, majorly due to the availability of publicly accessible datasets with dialogue acts annotations \citep{budzianowski2018multiwoz, rastogi2020towards, byrne2019taskmaster, asri2017frames}.
With pre-trained language models (LMs), recent approaches showcase remarkable performance on all three TOD sub-tasks. \citet{lee2021dialogue}, \citet{zhao2022description}, and \citet{bang2023task} achieve state-of-the-art performance for understanding user's requirements (NLU). \citet{wu2023diacttod}, \citet{bang2023task}, and \citet{sun2022mars} showcase similar trends for system action prediction and response generation tasks. In line with this trend, we benchmark pre-trained language models Flan-T5 \citep{chung2024scaling}, BioGPT \citep{luo2022biogpt} and PPTOD \citep{su2021multi} on \dataset{} dataset.
% Contributed by researchers, these datasets are often constructed through framework such as Wizard-of-OZ \citep{kelley1984iterative}, self-dialogue \citep{byrne2020tickettalk} or simulations \citep{shah2018building, kim2021neuralwoz}.
% While individual tasks remain relevant, there has been a noticeable shift towards end-to-end TOD systems powered by pre-trained language models. Introduced by \citet{hosseini2020simple} and popularised by later works \citep{lin2020mintl, peng2021soloist}, end-to-end TOD systems use GPT-2 to generate NLU, POL, and NLG outputs in succession, given a user-system dialogue history. In their seminal work, \citet{su2021multi} demonstrate the effectiveness of pre-training language models on TOD datasets, achieving state-of-the-art results after fine-tuning.
% Subsequent research built upon this foundation, leveraging \citep{su2021multi} as a backbone and enhancing system performance with task-specific contributions \citep{bang2023task}.
\paragraph{Medical Dialogue Systems: }
Many such datasets exist; however, only a few (see Table \ref{tab:datasets}) have been annotated. Early works focus on NLU and extract symptom slot-values and their status from a doctor-patient dialogue. CMDD \citep{Lin2019EnhancingDS} and SAT \citep{Du2019ExtractingSA} datasets study this task as sequence labeling, where dialogues are collected from online healthcare forum and clinical setting, respectively. Subsequent datasets, MIE \citep{Zhang2020MIEAM}, ReMeDi \citep{Yan2021ReMeDiRF}, DialoAMC \citep{Chen2022ABF} and Code-Mixed \cite{dowlagar2023code}, introduce additional slots, such as medical test and surgery and collect novel data for the task.
% Notably, ReMeDi and Code-mixed are only two datasets with fine-grained annotations. \dataset{} differs from them as follows. First, both these datasets do not capture multi-dimensional relations, such as `onset' and `duration', associated with a high-level slot. For instance, ReMeDi would label \textit{last two days} in Figure \ref{fig:intro_dialog} as \textit{(Inform, time, symptom onset, last two days)} without any linking to the symptom \textit{fever}. In contrast, questionnaire-based annotation framework in \dataset{} permits such linkages (see Figure \ref{fig:intro_dialog}). Second, \dataset{} has canonicalized the values for medical slots such as symptom and diseases to ensure a meaningful evaluation and to promote further research.
Notably, ReMeDi and Code-Mixed are the only datasets that collect low-level attributes in their labels. However, \dataset{} differs from them in several ways. First, while these datasets capture the attributes, they do not link them to appropriate slots. For example, ReMeDi would label the onset in a patient's utterance as (time, onset, past two days) without linking it to the symptom, fever. In contrast, \dataset{} uses CMAS to record slots and attributes together through a questionnaire-based annotation framework. Second, \dataset{} has canonicalized values for medical labels, such as symptoms and diseases, to ensure meaningful evaluation and to support future research.
Unlike medical dialogue systems, summarization involves converting doctor-patient dialogues into notes/reports \citep{joshi2020dr, krishna2021generating}. Many synthetic \citep{chintagunta2021medically, wang2023notechat} and human \citep{abacha2023empirical} datasets exist for this task. While summarization is not within the scope of this work, \dataset{} has the potential for adaptation to this task in the future. %\todo{VVS: We don't need this but reviewer asked for it.}
% \textcolor{red}{Medical dialogue summarization systems that create medical reports/notes from doctor-patient dialogues are gaining popularity \citep{enarvi2020generating, joshi2020dr, krishna2021generating}. Notably, \citet{abacha2023empirical} release a dialogue dataset, created by crowd-sourced workers from medical notes and benchmarked popular baselines for the note-generation task. \citet{chintagunta2021medically} are among the first to propose using GPT-3 for medical dialogue summarization, demonstrating competitive performance by creating synthetic summaries to augment existing training data. \citet{wang2023notechat} set up simulations using ChatGPT to create synthetic medical dialogues from medical notes at scale.} \todo{VVS: Will shorten it and move to introduction.}
% Table \ref{tab:datasets} compares \dataset{} against other medical dialogue datasets. To the best of our knowledge, \dataset{} is the first publicly available English dataset that a) contains fine-grained labels for NLU, POL, and NLG tasks and b) has canonicalized medical concepts in labels.
% Curating medical datasets poses unique challenges. One major obstacle is the stringent medical privacy regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States, which prohibit the public release of sensitive patient information. Additionally, unlike typical Task-Oriented dialogue (TOD) systems, which can be assembled using the Wizard of Oz method, creating these datasets requires the active participation of healthcare professionals to ensure their suitability for use.
% Despite these hurdles, several medical conversational datasets have been compiled for research purposes.
% \subsection{Early Medical dialogue datasets}
% Among the pioneering endeavors in curating medical dialogue datasets were Muzhi \citep{Wei2018TaskorientedDS} and DX \citep{Xu2019EndtoEndKR}. These datasets drew upon patient information, such as symptoms and medical history, sourced from online platforms. Additionally, early synthetic datasets like SD \citep{Zhong2022HierarchicalRL} and DDX \citep{Tchango2022DDXPlusAN} emerged in this realm. However, their representation of medical information relied on binary values to denote presence or absence of conditions. Annotation primarily centered on the Policy Learning (POL) task, determining the next inquiry in the conversation. Notably, annotations for tasks such as Natural Language Understanding (NLU) or Natural Language Generation (NLG) were lacking in these datasets.
% %---------------------------------
% %Datasets such as Muzhi and DX are some of the earliest medical dialogue datasets that were curated. They made use of patient information such as symptoms, medical history, etc crawled from online platforms. Other early medical datasets include SD and DDX that were synthetically generated. However, they had information in the form of binary values to indicate the presence or absence of the medical conditions. Further, these datasets had annotations only for the Policy Learning (POL) task which involved deciding which symptom, etc the system had to inquire for in the next utterance of the conversation. Annotations for NLU or NLG tasks were not present.
% %-----------------------------------
% %Cite MX, Dx, DDX. These datasets are curated from online platforms consisting of patient information such as symptoms, medical history, etc. Mention that these are just binary values. The task is policy learning where the TOD system is required to decide what symptom to inquire next in the conversation. However, these datasets do not provide any annotations for NLU and NLG.
% \subsection{Medieval datasets}
% The rise of active online healthcare forums has spurred the development of several datasets, such as CMDD \citep{Lin2019EnhancingDS}, MIE \citep{Zhang2020MIEAM}, and MSL \citep{Shi2020UnderstandingMC}. These datasets are crafted by extracting interactions between doctors and patients from these platforms, resulting in expansive datasets available in both Chinese and English. Examples like MedDialog-EN \citep{He2020MedDialogTL} and Cdialogue \citep{Varshney2022CDialogAM} illustrate this trend in English datasets. However, they predominantly adopt a question-and-answer format, lacking the fluidity of conversational dialogue. Furthermore, these datasets often lack comprehensive medical annotations and fail to encompass all necessary tasks, limiting their effectiveness as benchmarks for medical task-oriented dialogue.
% %------------------------------------
% %The presence of active online healthcare Forums have fueled the creation of several datasets such as CMDD, MIE, MSL. The creation of these datasets specifically involve the extraction of doctor-patient interactions from the sources to form large datasets in Chinese as well as in English. MedDialog-EN, Cdialogue are examples of some such English datasets. However, these datasets are not very conversational in nature but are of the QA style. Further, they also do not provide comprehensive medical annotations and do not cover al the tasks, making their usage as a medical TOD benchmark quite limited.
% %-----------------------------------
% %Thanks to online healthcare forums, they form a new source for doctor patient interactions. Many works, source their dataset from these sites. Specifically (cite some papers) extract these interactions and form a large collection of english and Chinese datasets. Mention english datasets. Iterate that they are not conversational but are QA style. Further, they do no provide comprehensive medical annotations. As a result, their usage as benchmark for medical TOD is limited.
% \subsection{Recent dialogue datasets}
% Recent dialogue datasets like ReMeDi \citep{Yan2021ReMeDiRF}, DialoAMC \citep{Chen2022ABF}, and MDDial \citep{Macherla2023MDDialAM} are notable for their detailed annotations that provide rich information. ReMeDi and DialoAMC additionally encompass various task-oriented dialogue (TOD) tasks. However, accessibility can be a limiting factor since most of these datasets are available in only specific languages, like ReMeDi and DialoAMC which are available only in Chinese, restricting their use within the broader research community. On the other hand, while the MDDial dataset is in English, it is important to note that it is a simulated dialogue dataset, lacking the natural flow of a human conversation.
%--------------------------------
%Some recent dialogue datasets such as ReMeDi, DialoAMC, MDDial do have very detailed annotations that convey a lot of information. ReMeDi and DialoAMC also cover the various TOD tasks, However, they are in Chinese, limiting their accessibility to the wider research community. Though MDDial dataset is in English, it is a simulated dialogue dataset and hence lacks the flow of a human conversation.
%--------------------------------
%Discuss Remedi, MDDial, DialoAMC . These provide interesting annotations however, they are in Chinese. MDDial is in English but it is simulated. Finally, point out that OSCE tod resolves all the issues.
% \goonjan{Should we mention synthetic/simulated datasets separately? Right now it is just present (though not explicitly) in some of the above.}
% \dataset{} is the first medical dialogue dataset in English with such comprehensive and detailed annotations. It is derived from staged doctor-interviews, following the Objective Structured Clinical Examination (OSCE) format commonly used for medical student examinations. This format ensures that the dialogues closely resemble real clinical interactions. With an average dialogue length exceeding 40 turns, our dataset offers extensive coverage of the medical consultation process, capturing its richness and nuance. To annotate the 213 English dialogues comprehensively, we utilized a structured ontology and an online annotation platform, harnessing the expertise of medical students for meticulous annotation across all task-oriented dialogue (TOD) tasks.
%--------------------------
%Our dataset, derived from the OSCE dataset, is collected from staged doctor-interviews which are taken of medical students for their examinations. This is known as the OSCE formatand it allows for the dialogues to seem like actual clinical-like interactions accurately. With an average dialogue length surpassing 40 turns, our dataset provides rich, nuanced coverage of the medical consultation process. By employing a structured ontology and an online annotation platform, we leveraged the expertise of medical students to meticulously annotate the 213 English dialogues, ensuring comprehensive coverage of all TOD tasks.
\section{The \dataset{} Dataset}\label{sec:dataset_creation}
% \vishal{
% \begin{itemize}
% \item Add a new section for our labeling portal and highlight features.
% \item Add labeling details such as compensation to the workers.
% \item Clarify the cohen kappa computation. Provide intent and slot level kappas.
% \item Add detailed instructions in appendix.
% \end{itemize}
% }
% \dinesh{Paint a high-level picture: (1) What the new dataset will enable the community to achieve, (2) What the community has now - summary of existing datasets, (3) What are we adding to it, (4) why adding annotations is not trivial, (5) how we added the annotations (i.e., by building a tool and using medical experts to annotate), and (6) how trustworthy our data is.}
To advance research in medical dialogue systems, datasets which capture \textit{canonicalized}, \textit{comprehensive} annotations must be available \textit{publicly}. However, existing datasets (listed in Table \ref{tab:datasets}) only fulfill a subset of these requirements. Moreover, these datasets are often limited to a single demographic, which restricts their broad applicability.
% \textcolor{red}{Additionally, existing datasets lack a systematic treatment of low-level medical features, such as the onset of a medical condition and the severity of symptoms, which are important for accurate diagnosis.}
To address these gaps, we curate \dataset{}, an English dataset of doctor-patient dialogues for collecting patient medical histories. First, we form the dialogues in \dataset{} by collecting publicly available transcripts of doctor-patient encounters \citep{fareez2022dataset}. To capture the complexity of the slots, we define a comprehensive medical attribute schema (CMAS) and develop a questionnaire-based labeling framework to annotate dialogues based on CMAS. For each slot type (e.g., symptom, personal medical history), medical professionals vet questions to capture associated attributes (severity, onset, etc.). Utilizing this framework, professional annotators then label utterances by answering the questionnaire corresponding to each slot under the doctor's supervision. Finally, we canonicalize the slots values to standard medical concepts in the UMLS vocabulary.\footnote{\tt https://uts.nlm.nih.gov/uts/umls/home}
% This section discusses the creation of the \dataset{} dataset in detail.
\subsection{Dialogue Acquisition}
Recently, \citet{fareez2022dataset} released a dataset of doctor-patient interviews from five specialties. These interviews involve staged interactions where medical professionals assume the roles of doctor and patient. The doctor systematically gathers healthcare information from the patient following the Objective Structured Clinical Examinations (OSCE) format, covering aspects such as a history of present illness, past medical records, and family health history crucial for diagnosis.
Notably, \citet{fareez2022dataset} make their dataset publicly available for academic use. Further, their dialogues are highly conversational, averaging 95 utterances per dialogue. Unfortunately, the dataset is not annotated with the necessary labels for building a medical TOD system. In response, we form \dataset{} by labeling 22,503 utterances from the respiratory and musculoskeletal specialties available in their dataset.
\subsection{The CMAS Format}
% \textcolor{red}{Table \ref{tab:ontology} presents the intents and slots defined in our annotation framework. Most TOD datasets (including Code-Mixed \citep{dowlagar2023code}) capture slot information as slot-value pairs. However, this scheme can not incorporate associated attributes such as onset and location, as shown in Figure \ref{fig:intro_dialog}. In ReMeDi, \citet{Yan2021ReMeDiRF} propose a labeling system using slot-subslot-value tuples. However, some attributes can exist independently of any slot in their ontology. For example, in their dataset, the onset is recorded as (Inform, time, onset, value), resulting in 46\% of onset labels being not linked to any slot. Similarly, 42\% of inform duration labels lack such association. Further, their method requires the annotators to be familiar with the complicated slot and sub-slot hierarchy.}
A doctor-patient dialogue consists of complex slot types such as symptoms, patient medical history, and patient’s habits.
%In figure \ref{fig:intro_dialog}, symptoms \em{pharyngitis} and \em{fever} have onsets of \em{past four days} and \em{last two days}, respectively. In addition to onset,
Symptoms have attributes such as onset (see Figure \ref{fig:intro_dialog}), duration, location, frequency, severity, and progression. Capturing the relationship between these attributes is crucial for creating accurate patient profiles and reliable diagnoses. As explained earlier, existing TOD datasets use key-value pairs to represent both slots and attributes, thus missing the links between them.
%}\todo{M:there was repetition from sections 1 and 2, so i cut} %information from user responses. Code-Mixed and ReMeDi adapt this key-value scheme for medical dialogues and have separate slots to capture additional attributes. For instance, ReMeDi annotates onset as triplet \em{(time, symptom onset, past four days)} and \em{(time, symptom onset, last two days)}. However, this scheme does not fully capture the inherently complex nature of attributes and slot types.}
\begin{figure*}
\centering
\includegraphics[width=0.90\textwidth]{latex/images/portal_labeled.pdf}
\caption{Labeling Interface for questionnaire-based labeling scheme.}\label{fig:ann_portal}
\label{fig:low_data}
\end{figure*}
In response, for \dataset{}, we develop a Comprehensive Medical Attribute Schema (CMAS) to capture the inherent nature of slots and a questionnaire-based annotation framework that effectively captures the relationship between slots and their attributes while simplifying the labeling task. We ask doctors to design questions for each attribute in a slot. For example, questions like \emph{``Where is the symptom located?"} and \emph{``When did the symptom appear?"} are suggested for the location and onset of symptoms. We provide further details on the questionnaire in Appendix \ref{app:slotwise_questions}.
Using these questions, we develop detailed annotation guidelines for our labeling task that aims to capture intent, slots, and their attributes for each utterance in the dialogues, considering both the utterance itself and the dialogue history. In the next section, we discuss our labeling interface that naturally collects associated attributes.
Table \ref{tab:ontology} lists intents and slot types in CMAS. Table \ref{tab:slot_and_values} in Appendix \ref{app:slotwise_questions} reports the attributes associated with each slot type in the schema.
% We now outline our approach for labeling utterances from the sourced dialogues. Table \ref{tab:ontology} presents the intents and slots defined for this labeling process. Beyond these high-level slots, medical dialogues additionally contain low-level details, as shown in Figure 1. Capturing this information complicates the labeling task, requiring the annotators to know all possible low-level categories. We thus opt for a questionnaire-based annotation framework to ensure a comprehensive capture of medical information in unison. Specifically, we ask doctors to design a set of questions for each slot to gather related low-level details. For example, questions like \em{Where is the symptom located?} and \em{When did the symptom appear?} are suggested for the location and onset of symptoms. We provide further details on the questionnaire in Appendix A.
% \dinesh{Please discuss why we designed the questionnaire-based scheme. Second line should be Table 2 - right?}
% Using the questions suggested by doctors, we develop detailed annotation guidelines for our labeling task that aims to capture intent, slot labels, and answers to slot-specific questions for each utterance in the dialogues, considering both the utterance itself and the dialogue history. \textcolor{red}{The user interface for our labeling task is shown in figure \ref{fig:ann_portal}. With questionnaires, we simplify the labeling task so that the annotators can now provide low/high-level details in unison.
% greatly simplifies the design of the labeling interface shown in figure \ref{fig:ann_portal}
% Our annotation framework allows fine-grained annotations for each utterance and streamlines the labeling process.
%to faster through a simplified user interface shown in figure \ref{fig:ann_portal}.
\begin{table}[]
\centering
\resizebox{0.48\textwidth}{!}{%
\begin{tabular}{l|l}
\toprule
Intents & Inquire, Inform, Diagnose, Salutations, Chit-chat, Other \\ \midrule
\multirow{3}{*}{Slots} & Symptom, Patient Medical History, Family Medical History \\
& Habits, Exposure, Medication, Medical Test, Disease \\
& Travel Information, Occupation, Residence, Basic Information \\ \bottomrule
\end{tabular}%
}
\caption{\dataset{} intents and slots}
\label{tab:ontology}
\end{table}
\subsection{Labeling Interface}
Our labeling interface displays a doctor-patient dialogue for annotation. For each utterance, annotators select one or more appropriate intents and slot types. Then, the questionnaires for each chosen slot appear for the annotators to answer. On submission, a status box displays the answers for reference. %\todo{M: cut one sentence to save space}
%The interface supports multiple intents and slots per utterance by allowing multiple questionnaire submissions for a single utterance.
We further enhance usability with editing features, keyboard shortcuts, and a tracking box highlighting the patient's current slot values. Figure \ref{fig:ann_portal} shows a snapshot of our labeling interface. We include the user guide with our annotation guidelines in Appendix \ref{app:annotation_guidelines}.
The labeling interface offers two significant advantages. First, it requires annotators to capture attributes along with each slot annotated. Specifically, annotators need to provide the slot and its value before recording any additional attributes, which ensures attributes are always linked to their slots and do not exist independently. Second, the interface displays the complete questionnaire for a selected slot, reducing the need for annotators to memorize the slot-attribute relationships.
% We will make the labeling interface publicly available to encourage contribution to our dataset.\todo{M: last sentence can be cut from here to save space, if needed. this info belongs in intro/conclusions}
\subsection{Labeling Process and Quality Control}
% We use a professional annotation service specializing in medical data solutions. Our labeling team consists of six annotators who undertake labeling under the guidance of a doctor. The annotators have backgrounds in medical sciences or pharmacy, ensuring a high level of expertise. The doctor has a professional medical degree and two and half years of hospital experience.
We avail a professional annotation service to hire six annotators with medical sciences or pharmacy backgrounds to label the dialogues under the supervision of a doctor.\footnote{The doctor has a professional medical degree with two and half years of hospital experience.} We train annotators by providing them with our detailed annotation guidelines and an example of an annotated dialogue for their reference. Once familiarized with the task and the labeling interface, we ask the annotators to label a different sample dialogue independently. Based on their responses, we offer feedback and point out any issues that need to be addressed. To ensure their understanding, we present a third sample dialogue for them to label independently. During their training, annotators are tested twice, covering 200+ utterances across two dialogues.
% Subsequently, we compare their annotations with expert labels held privately and provide them feedback regarding any discrepancies. Finally, we validate their performance using another sample dialogue.
%Our labeling commences as follows.
We then divide all available dialogues into six groups. Each trained annotator independently labels the assigned group. To ensure label quality, we systematically introduce seed dialogues into all the groups so that each seed dialogue gets labeled by a pair of annotators. We use the seed dialogue labels to periodically measure inter-annotator agreement and identify any quality issues. Throughout the process, annotators can raise concerns in real time, resolving any ambiguities. Our labeling process has a strong inter-annotator agreement\footnote{We measure inter-annotator agreement between pairs of annotators and report the average.} with Cohen-Kappa $\kappa=0.94$ for intents and $\kappa=0.72$ for slot-value pairs, indicating strong label consistency.
\subsection{Post-Processing}
In post-processing, we examine our collected labels and identify any utterances that may lack complete labels. For instance, in a few utterances such as \textit{I used over-the-counter medicine at night time to help sleep}, the labels did not indicate the \textit{status} of whether the patient had taken the medication. For such cases, which amount to 0.7\% (165 samples) of all the utterances, we ask annotators to review and rectify the labeling to ensure accuracy and completeness. %\todo{Vishal, please explicitly mention what was annotated and what was missed - its a bit unclear}
% Total 22317
Medical concepts, such as symptoms and diseases, often exhibit different surface (and layman) forms. For instance, phrases \textit{shortness of breath} and \textit{difficulty breathing} both refer to the medical concept \textit{Dyspnea}. To ensure consistent medical terminology across annotators, we canonicalize the medical terms in our labels.
% Next, we canonicalize the medical term in our labels to ensure consistent terminology across different annotators.
Specifically, we link medical terms to their precise medical concept in UMLS Metathesaurus. UMLS has a large-scale collection of medical vocabularies that facilitate a standardized framework for representing and linking biomedical concepts. Linking to UMLS allows meaningful evaluations and also paves the way for dialogue systems grounded in large-scale online medical databases.
We first divide the slots (and attributes) into two categories - medical and non-medical. Medical ones include symptoms, patient medical history, family medical history, etc. Non-medical ones include duration, frequency, residence, etc.
%Table \ref{tab:slot_and_values} highlights all the medical attributes.
To canonicalize medical slots/attributes, we use QuickUMLS\footnote{https://github.com/Georgetown-IR-Lab/QuickUMLS} string matching to generate a set of candidate UMLS concepts. Then, we manually verify these candidates to filter extraneous ones, considering surface forms and the context within the dialogue. We keep the doctor in the loop to review the candidates and pick the final concept, provide corrections or recommendations where necessary, and resolve any ambiguities. Finally, we replace the surface forms with their corresponding canonicalized version to ensure that the medical terminology aligns with professional standards.
In contrast to medical slots/attributes, non-medical ones lack standardized vocabularies of concepts and are thus not canonicalized in our dataset.
% QuickUMLS\footnote{https://github.com/Georgetown-IR-Lab/QuickUMLS} string matching to generate a candidate set of medical conceptswe run QuickUMLS over the labels to generate a set of candidate UMLS concepts. Then, we manually verify and link these values to UMLS medical concepts, considering their surface form and the context within the dialogue. Here, we manually filter the extraneous candidates and retain relevant ones.
% Finally, the doctor selects the most appropriate candidate for the final canonicalization or suggests another candidate if no candidate is suitable. Thus, our procedure already accounts for errors in QuickUMLS predictions.
% To canonicalize, we first use QuickUMLS\footnote{https://github.com/Georgetown-IR-Lab/QuickUMLS} string matching to generate a candidate set of medical concepts for the values for slots marked in table \ref{tab:ontology}. Then, we manually verify and link these values to UMLS medical concepts, considering their surface form and the context within the dialogue. We keep the doctor in the loop to review our canonicalized labels, provide corrections or recommendations where necessary, and resolve any ambiguities. Finally, we replace the surface forms in labels with their standard nomenclature to ensure that the medical terminology aligns with professional standards.
% \dinesh{Also, its not clear what slot types are canonicalized. we need to highlight the places where doctor is involved in defining and standardizing the dataset.}
% \dinesh{in addition to the number of dialogues, please mention the number of utterances as well.}
\subsection{Dataset Statistics}
% \vishal{
% \begin{itemize}
% \item Re-iterate and stress that number of utterances are large and useful for evaluation.
% \end{itemize}
% }
\begin{table}[]
\centering
\small
\resizebox{0.5\textwidth}{!}{%
\begin{tabular}{ccccc}
\toprule
\textbf{Split} & \textbf{\#dialogues} & \textbf{\#utterances} & \textbf{Avg \#utterances} & \textbf{Avg \#words} \\
& & & per dialogue & per utterance \\ \midrule
Train & 175 & 16,852 & 95.29 & 13.46 \\
Valid & 20 & 1,869 & 92.45 & 12.77 \\
Test & 18 & 1,798 & 98.89 & 13.03 \\
Out-of-domain Test & 20 & 2,197 & 109.85 & 14.00 \\ \midrule
Total & 213 & 22,503 & 96.57 & 13.42 \\ \bottomrule
\end{tabular}%
}
\caption{\dataset{} statistics.}\label{tab:data_stats}
\end{table}
We treat respiratory and musculoskeletal dialogues in \dataset{} separately. We use dialogues from the respiratory specialty for model building and in-domain benchmarking, while dialogues from the musculoskeletal specialty serve as an out-of-domain test set.
We divide the respiratory dialogues into train, validation, and test sets. First, we form a high-quality in-domain test set consisting of the seed dialogues we used for quality control. Each test dialogue is thus labeled by two different annotators. To obtain the final labels, we task a third annotator to resolve the inconsistencies between the two sets of labels. Finally, we randomly split the remaining respiratory dialogues into train and validation sets.
The out-of-domain test set consists of musculoskeletal dialogues. A single annotator labeled all these dialogues. To ensure quality, we conducted periodic checks on the submitted labels and provided feedback to the annotator as needed. Table \ref{tab:data_stats} lists overall statistics for \dataset{} dataset. With 22,503 annotated utterances, \dataset{} enables meaningful training and evaluation of the machine learning models. Dialogues in \dataset{} are highly conversational, with an average of 96 utterances per dialogue.
%\todo{M: can we say using some citation that this the typical length of medical history taking dialogue?}
Figure \ref{fig:int_slot_dist} shows the distribution of utterances in \dataset{} over different intents and slot values. For doctors, \textit{Inquire} accounts for 87.5\% of the total utterances, followed by \textit{Salutations} (5.0\%), \textit{Chit-chat} (4.8\%), \textit{Diagnose} (1.9\%), and \textit{Other} (0.8\%). For patients, \textit{Inform} (92.2\%) is the dominant slot, followed by \textit{Chit-chat} (4.2\%) and \textit{Salutations} (3.6\%). Similarly, 56\% of the utterances discuss \textit{Symptom} followed by \textit{Patient Medical History} (13.5\%) and \textit{Habits} (7.5\%).
Dialogues in \dataset{} are highly systematic. To elucidate that, we divide each dialogue into ten equal segments. For each segment, we find the distribution of the slots across all dialogues. Figure \ref{fig:dialog_struc} shows the resultant heatmap. During the first half of their conversation, the doctor and the patient primarily discuss symptoms. Then, they transition into other slots, such as patient medical history, habits, medication, etc.
\begin{figure}[]
\centering
\begin{subfigure}[t]{0.23\textwidth}
% \includegraphics[height=1.6in]{latex/images/intent_dist.png}
\includegraphics[height=1.6in]{latex/images/intent_dist2.pdf}
% \caption{Intent Distribution}
\end{subfigure}
\begin{subfigure}[t]{0.23\textwidth}
% \includegraphics[height=1.6in]{latex/images/slot_dist.png}
\includegraphics[height=1.6in]{latex/images/slot_dist2.pdf}
% \caption{Slot Distribution}
\end{subfigure}
\caption{Distribution of utterances \dataset{} dataset over intents (left) and slots (right).}\label{fig:int_slot_dist}
\end{figure}
\begin{figure}[]
\centering
\includegraphics[width=0.48\textwidth]{latex/images/dialog_struc2.png}
\caption{dialogues in \dataset{} dataset are systematic.}\label{fig:dialog_struc}
\end{figure}
\section{Experimental Setup}
% \vishal{
% \begin{itemize}
% \item Include Few-shot baselines. Include prompts in the appendix.
% \item Include a section for evaluation protocol explaining the evaluation.
% \item Error analysis - add new samples in the paper.
% \item Explain that few-shot models are not on par with supervise indicating need for the dataset.
% \end{itemize}
% }
% \todo{Include Few-shot baselines. Include prompts in the appendix.}
%This section discusses the TOD tasks in \dataset{}, our evaluation protocol, and baselines.
%\subsection{Task Definitions}
\dataset{} supports all three subtasks in a TOD system -- natural language understanding (NLU), policy learning (POL), and natural language generation (NLG). Table \ref{tab:tasks} (appendix) illustrates representative examples from each subtask.
%\subsubsection*{Natural Language Understanding (NLU)}
NLU involves understanding the information presented in the latest patient utterance. Specifically, given the dialogue so far, an NLU model predicts the intent, active slots, and their attributes of the patient utterance. This updates the dialogue state, i.e., the aggregate information the patient reports up to the current turn in the dialogue.
%To evaluate the success of NLU
%\subsubsection*{Policy Management (POL)}
POL requires predicting the doctor's next action.
%as a continuation of the dialogue.
An action consists of intents, slots, and associated attributes. A POL model inputs the dialogue history and the dialogue state and predicts the action.
%\subsubsection*{Natural Language Generation (NLG)}
Finally, NLG involves transforming the doctor's action into natural language. An NLG model predicts the doctor's utterance based on the dialogue history and the action from the POL model.
% \begin{enumerate}
% \item NLU: The model inputs dialogue history and predicts the intent and slots for the latest patient utterance.
% \item POL: The model inputs dialogue history and aggregate patient's information captured as slots to predict the doctor's next step or action. The structure of an action is similar to that of slots.
% \item NLG: The model inputs dialogue history along with the predicted action by the POL to generate the follow-up doctor response.
% \end{enumerate}
% Here, the slots and actions contain fine grained attributes.
\subsection{Evaluation Metrics}
%\noindent \textbf{NLU}
For NLU, we compute an F1 score by matching intent, slots, and associated attributes from gold and predicted labels. Before matching, we unroll the gold and predicted NLU labels into sets of the form \emph{\{(intent, slot, value, attribute, attribute-value)\}}. For instance, the NLU label in Figure \ref{fig:intro_dialog} is transformed to \emph{{[}(inform, positive symptom, pharyngitis), (inform, positive symptom, pharyngitis, onset, past four days), (inform, positive symptom, fever), (inform, positive symptom, fever, onset, last two days){]}}.
%We then highlight the shortcomings of this strategy for non-medical attributes and propose evaluating them using ChatGPT. Although our discussion here focuses on NLU, our approach can easily generalize to the POL task.
%To compute the F1 score, we unroll the gold and predicted NLU labels into sets of the form \emph{\{(intent, slot, value, attribute, attribute-value)\}}. For instance, the NLU label in Figure 1 is transformed to \emph{{[}(inform, positive symptom, pharyngitis), (inform, positive symptom, pharyngitis, onset, past four days), (inform, positive symptom, fever), (inform, positive symptom, fever, onset, last two days){]}}. Let $G$ and $P$ denote the transformed gold and predicted sets for a test sample. True positives are computed as $TP = G \cap P$, false positives as $FP = P \setminus G$, and false negatives as $FN = G \setminus P$. Finally, precision, recall, and F1 are computed by aggregating $TP$, $FP$, and $FN$ counts over the complete test set.
In \dataset{}, medical attributes are canonicalized, so we simply use exact match scores for them. However, for non-medical attributes, where different strings can have the same meaning, just string matching is too conservative. E.g., `2 days ago' and `two days before' convey the same meaning, but will get counted as non-matches. For non-medical attributes, we use ChatGPT \citep{openai2024gpt35turbo} to adjudicate semantic equivalence and use that for F1 score computation. Our prompt is in Appendix \ref{app:metric_prompt}. In experiments, we report medical and non-medical scores separately, in addition to overall scores.%\todo{M: removed a lot of content which looked obvious to me; feel free to move the orignal section to appendix}
%Let $e_1 = (i, s, v, at, s_1)$ be an element in $G$ with attribute $at$ and its value $s_1$. We compare $e_1$ with each $e_2 = (i, s, v, at, s_2) \in P$ with the same intent, slot, value, and attribute. For each element $e_2 = (i, s, v, at, s_2)$ in $P$ that matches $e_1$ in intent, slot, and value, we prompt ChatGPT to decide whether $s_1$ and $s_2$ have the same meaning.
%If ChatGPT's response is positive, $e_1$ is added to $TP$, and $e_2$ does not participate in future comparisons. Otherwise, we add $e_1$ to $FN$. Any $e_2$ not matched to an $e_1 \in G$ is added to $FP$.
%\vspace{1ex}
%\noindent \textbf{POL}
For POL, we could use precision, recall, and F1 scores, similar to NLU; however, in conversation, it is quite possible for the doctor to change the order of questions somewhat.
%We use precision, recall, and F1 scores as discussed earlier. In addition, while conversing with the patient, it is possible for the doctor to sometimes change the order of the questions.
To account for this, we use the Precision@K metric. At a given test turn, we check if the medical attributes predicted from the POL model are present in the gold actions within the next K turns. We report Precision@K for K = 1, 4, 8, and infinity. Finally, for NLG, we use BLEU \citep{papineni2002bleu}, Rouge \citep{lin-2004-rouge}, and BERTScore \citep{zhang2019bertscore} to measure the generation quality.
%\vspace{1ex}
%\noindent \textbf{NLG} W
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \usepackage{graphicx}
\subsection{Baselines}
We model all three tasks as \emph{seq2seq} learning and evaluate baselines in the supervised and in-context learning settings. Baselines use a respiratory (in-domain) dataset for development and are evaluated using both an in-domain (respiratory) test set and an out-of-domain (musculoskeletal) test set.
% \paragraph{Supervised.}
\vspace{1ex}
\noindent \textbf{Supervised.} We fine-tune several pre-trained language models on the \dataset{} training set for the three tasks. We utilize the PPTOD (base) \citep{su2021multi} and Flan-T5 (base) \citep{chung2024scaling} encoder-decoder transformer models, pre-trained on general domain TOD tasks and the Flan suite of tasks, respectively. Additionally, we employ BioGPT (medium) \citep{luo2022biogpt}, a decoder-only transformer model pre-trained on extensive biomedical literature. Finally, we fine-tune Llama3 8B Instruct \citep{llama3modelcard} (referred to as Llama3 henceforth) and OpenBioLLM 8B \citep{OpenBioLLMs} models, which serve as representatives of large language models (LLMs). After fine-tuning, we evaluate the models on the \dataset{} test set.
% \paragraph{In-context Learning.}
\vspace{1ex}
\noindent \textbf{In-context Learning.} We prompt several large language models (LLMs) to make predictions on test samples across the three tasks. For each test sample, we select the top five exemplars \footnote{We limit the selection to five exemplars to ensure the prompt stays within the 4096 token limit.} from the training set whose dialogue histories are semantically closest to the test sample \citep{liu-etal-2022-makes}. To identify the top exemplars, we use the BAAI/bge-large-en-v1.5 model \citep{bge_embedding} to encode the dialogue history and perform a maximum inner product search. Sample prompts used in our experiments are provided in Appendix \ref{app:prompts}. We evaluate the performance of Llama3 8B/70B Instruct \citep{llama3modelcard}, OpenBioLLM 8B/70B \citep{OpenBioLLMs}, ChatGPT (gpt-3.5-turbo-0125) \citep{openai2024gpt35turbo}, and GPT-4 (gpt-4-1106-preview) and GPT-4-Turbo (gpt-4-turbo-2024-04-09) \citep{openai2023gpt4preview}.
\subsection{Implementation Details}
We adapted a publicly available codebase for the PPTOD model to \dataset{}\footnote{\href{https://github.com/awslabs/pptod}{https://github.com/awslabs/pptod}}. For training this model, we utilized a learning rate of $1e-3$ and a batch size of 64. For the Flan-T5 and BioGPT models, we employed a learning rate of $1e-4$ and a batch size of 16. These models were trained on a single V100 GPU with 32 GB of memory.
We train Llama3 with Unsloth\footnote{\url{https://github.com/unslothai/unsloth}} to enhance the efficiency of both training and inference. We fine-tuned the Llama3 using LoRA \citep{hu2021lora} and use parameters $r=32$ and $\alpha=32$, a learning rate of $1e-4$, a batch size of 16, and a cosine learning rate scheduler with a warm-up period of 10\% for both models. This training was performed on two A100 40GB GPUs, taking approximately five hours to complete. We trained OpenBioLLM models under similar settings. During inference, we employ greedy decoding (temp. $=$ 0) for all in-context and supervised baselines.
\section{Results}
We begin by evaluating the models for the NLU, POL, and NLG tasks on the in-domain test set. Next, we assess the top-performing models from the in-domain evaluation on the out-of-domain dataset.
\subsection{Natural Language Understanding}
\begin{table}[]
\centering
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{llccc}
\toprule
% \textbf{Model} & \textbf{Overall F1} & \textbf{Medical F1} & \textbf{Non-medical F1} \\ \midrule
& & \textbf{Overall} & \textbf{Medical} & \textbf{Non-Medical}\\
& \multirow{-2}{*}{\textbf{Model}} & \textbf{F1} & \textbf{F1} & \textbf{F1} \\ \midrule
% \multicolumn{4}{c}{Supervised} \\ \midrule
\multirow{5}{1em}{\rotatebox[origin=c]{90}{Supervised}} & PPTOD (base) & 0.6849 & 0.7268 & 0.5141 \\
& Flan-T5 (base) & 0.6887 & 0.7354 & 0.5062 \\
& BioGPT & 0.6090 & 0.6612 & 0.4187 \\
& OpenBioLLM 8B & 0.6731 & 0.7294 & 0.4791 \\
% & Llama3 8B Instruct & \textbf{0.7048} & \textbf{0.7572} & \textbf{0.5174} \\
% & Llama3 8B & 0.7048 & 0.7572 & 0.5174 \\
& Llama3 8B & \textbf{0.7139} & \textbf{0.7603} & \textbf{0.5397} \\
\midrule
% \multicolumn{4}{c}{In-context} \\ \midrule
\multirow{6}{1em}{\rotatebox[origin=c]{90}{In-context}} & OpenBioLLM 8B & 0.6056 & 0.6431 & 0.4416 \\
& OpenBioLLM 70B & 0.6090 & 0.6499 & 0.4664 \\
% & Llama3 8B Instruct & 0.5251 & 0.5611 & 0.4048 \\
% & Llama3 70B Instruct & 0.5736 & 0.6150 & 0.4422 \\
& Llama3 8B & 0.5251 & 0.5611 & 0.4048 \\
& Llama3 70B & 0.5736 & 0.6150 & 0.4422 \\
& ChatGPT & 0.5929 & 0.6337 & 0.4425 \\
& GPT-4 & 0.6351 & 0.6715 & 0.5043 \\
& GPT-4-Turbo & \textbf{0.6641} & \textbf{0.6999} & \textbf{0.5329} \\ \bottomrule
\end{tabular}%
}
% \caption{Model performance on \dataset{} NLU task. Llama3 8B\dag{} is optimized using Unsloth.}
\caption{In-Domain Model Evaluation for the \dataset{} NLU Task}
\label{tab:nlu_results}
\end{table}
Table \ref{tab:nlu_results} compares the performance of various baselines on an NLU task. At a high level, Llama3 8B achieves the best overall performance among the supervised models. This superior performance can be attributed to its ability to recognize medical and non-medical slots better than its competitors. While most models demonstrate competitive performance in recognizing non-medical attributes, the Llama3 model has a clear advantage for medical attributes, scoring 0.0249 points higher over the nearest Flan-T5 baseline.
In the in-context learning setting, pre-training on biomedical corpus offers a clear advantage, with OpenBioLLM significantly outperforming Llama3 models from the same weight class. Specifically, OpenBioLLM 8B and OpenBioLLM 70B surpass their Llama3 counterparts by 0.0805 and 0.0354 pts, respectively. OpenBioLLM showcases a superior understanding of medical attributes, compared to the general-purpose Llama models. OpenBioLLM 70B exhibits a slight edge of 0.0174 points over ChatGPT. However, GPT-4-Turbo achieves the best in-context performance overall. While GPT-4-Turbo's non-medical F1 is comparable to supervised models, its performance on medical attributes lags behind the supervised Llama3 model by 0.0573 pts. This highlights the potential for further improvements in in-context learning baselines, particularly in the medical domain.
Tables \ref{tab:nlu_example1} and \ref{tab:nlu_example2} display example responses from different baseline models. In table \ref{tab:nlu_example1}, the models struggle with distinguishing between related but distinct medical concepts, such as confusion and mental fatigue. Extracting multiple slots and attributes also presents a challenge for the models. As shown in table \ref{tab:nlu_example2}, the models either fail to recognize all the symptoms from the input or make errors when linking the attributes.
\subsection{Policy Learning}
% \begin{table}[]
% \centering
% \resizebox{0.49\textwidth}{!}{%
% \begin{tabular}{l|ccc|cccc}
% \toprule
% & Overall & Med & Non-Med & \multicolumn{4}{c}{Precision@} \\ \cmidrule{5-8}
% \multirow{-2}{*}{Model} & F1 & F1 & F1 & 1 & 4 & 8 & Inf \\ \midrule
% \multicolumn{8}{c}{Supervised} \\ \midrule
% PPTOD (base) & 0.210 & 0.210 & 0.207 & 0.153 & 0.284 & 0.326 & 0.359 \\
% Flan-T5 (base) & 0.200 & 0.203 & 0.150 & 0.115 & 0.204 & 0.238 & 0.265 \\
% BioGPT & 0.187 & 0.185 & \textbf{0.216} & 0.135 & 0.251 & 0.300 & 0.348 \\
% OpenBioLLM & & & & & & & \\
% 8B & \multirow{-2}{*}{0.209} & \multirow{-2}{*}{0.217} & \multirow{-2}{*}{0.119} & \multirow{-2}{*}{0.161} & \multirow{-2}{*}{0.295} & \multirow{-2}{*}{0.338} & \multirow{-2}{*}{0.374} \\
% Llama3 8B & & & & & & & \\
% Instruct & \multirow{-2}{*}{\textbf{0.217}} & \multirow{-2}{*}{\textbf{0.220}} & \multirow{-2}{*}{0.175} & \multirow{-2}{*}{\textbf{0.172}} & \multirow{-2}{*}{\textbf{0.300}} & \multirow{-2}{*}{\textbf{0.351}} & \multirow{-2}{*}{\textbf{0.379}} \\ \midrule
% \multicolumn{8}{c}{In-context} \\ \midrule
% OpenBioLLM & 0.105 & 0.109 & 0.034 & 0.059 & 0.122 & 0.151 & 0.194 \\
% 8B & & & & & & & \\
% OpenBioLLM & & & & & & & \\
% 70B & \multirow{-2}{*}{0.121} & \multirow{-2}{*}{0.127} & \multirow{-2}{*}{0.025} & \multirow{-2}{*}{0.086} & \multirow{-2}{*}{0.136} & \multirow{-2}{*}{0.162} & \multirow{-2}{*}{0.195} \\
% Llama3 8B & & & & & & & \\
% Instruct & \multirow{-2}{*}{0.107} & \multirow{-2}{*}{0.112} & \multirow{-2}{*}{0.016} & \multirow{-2}{*}{0.062} & \multirow{-2}{*}{0.112} & \multirow{-2}{*}{0.136} & \multirow{-2}{*}{0.185} \\
% Llama3 70B & & & & & & & \\
% Instruct & \multirow{-2}{*}{0.100} & \multirow{-2}{*}{0.104} & \multirow{-2}{*}{0.015} & \multirow{-2}{*}{0.060} & \multirow{-2}{*}{0.122} & \multirow{-2}{*}{0.151} & \multirow{-2}{*}{0.178} \\
% ChatGPT & 0.110 & 0.112 & \textbf{0.054} & 0.083 & \textbf{0.145} & \textbf{0.179} & \textbf{0.220} \\
% GPT-4 & 0.090 & 0.094 & 0.017 & 0.041 & 0.080 & 0.111 & 0.189 \\ \bottomrule
% \end{tabular}%
% }
% \caption{Model performance on \dataset{} POL task.}
% \label{tab:pol_results}
% \end{table}
% \begin{table*}[]
% \centering
% \small
% \resizebox{0.98\textwidth}{!}{%
% \begin{tabular}{l|c|c|c|cccc}
% \toprule
% & Overall & Medical & Non-Medical & \multicolumn{4}{c}{{Precision@}} \\
% \cmidrule{5-8}
% \multirow{-2}{*}{Model} & F1 & F1 & F1 & 1 & 4 & 8 & Inf \\ \midrule
% \multicolumn{8}{c}{Supervised} \\ \midrule
% PPTOD (base) & 0.210 & 0.210 & 0.207 & 0.153 & 0.284 & 0.326 & 0.359 \\
% Flan-T5 (base) & 0.200 & 0.203 & 0.150 & 0.115 & 0.204 & 0.238 & 0.265 \\
% BioGPT & 0.187 & 0.185 & \textbf{0.216} & 0.135 & 0.251 & 0.300 & 0.348 \\
% OpenBioLLM 8B & 0.209 & 0.217 & 0.119 & 0.161 & 0.295 & 0.338 & 0.374 \\
% Llama3 8B Instruct & \textbf{0.217} & \textbf{0.220} & 0.175 & \textbf{0.172} & \textbf{0.300} & \textbf{0.351} & \textbf{0.379} \\ \midrule
% \multicolumn{8}{c}{In-context} \\ \midrule
% OpenBioLLM 8B & 0.105 & 0.109 & 0.034 & \multicolumn{1}{l}{0.059} & \multicolumn{1}{l}{0.122} & \multicolumn{1}{l}{0.151} & \multicolumn{1}{l}{0.194} \\
% OpenBioLLM 70B & \textbf{0.121} & \textbf{0.127} & 0.025 & \textbf{0.086} & 0.136 & 0.162 & 0.195 \\
% Llama3 8B Instruct & 0.107 & 0.112 & 0.016 & 0.062 & 0.112 & 0.136 & 0.185 \\
% Llama3 70B Instruct & 0.100 & 0.104 & 0.015 & 0.060 & 0.122 & 0.151 & 0.178 \\
% ChatGPT & 0.110 & 0.112 & \textbf{0.054} & 0.083 & \textbf{0.145} & \textbf{0.179} & \textbf{0.220} \\
% GPT-4 & 0.090 & 0.094 & 0.017 & 0.041 & 0.080 & 0.111 & 0.189 \\ \bottomrule
% \end{tabular}%
% }
% \caption{Model performance on \dataset{} POL task.}
% \label{tab:pol_results}
% \end{table*}
%F1 scores only
\begin{table}[]
\centering
\small
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{llccc}
\toprule
& & \textbf{Overall} & \textbf{Medical} & \textbf{Non-Medical}\\
& \multirow{-2}{*}{\textbf{Model}} & \textbf{F1} & \textbf{F1} & \textbf{F1} \\ \midrule
% \multicolumn{4}{c}{Supervised} \\ \midrule
\multirow{5}{1em}{\rotatebox[origin=c]{90}{Supervised}} & PPTOD (base) & 0.2099 & 0.2101 & 0.2069 \\
& Flan-T5 (base) & 0.1999 & 0.2033 & 0.1495 \\
& BioGPT & 0.1870 & 0.1853 & 0.2156 \\
& OpenBioLLM 8B & 0.2085 & 0.2172 & 0.1190 \\
% & Llama3 8B & 0.2168 & 0.2199 & 0.1753 \\
& Llama3 8B & \textbf{0.2389} & \textbf{0.2392} & \textbf{0.2329} \\ \midrule
% \multicolumn{4}{c}{In-context} \\ \midrule
\multirow{6}{1em}{\rotatebox[origin=c]{90}{In-context}} & OpenBioLLM 8B & 0.1052 & 0.1086 & 0.0336 \\
& OpenBioLLM 70B & 0.1214 & 0.1273 & 0.0247 \\
& Llama3 8B & 0.1072 & 0.1116 & 0.0160 \\
& Llama3 70B & 0.1000 & 0.1042 & 0.0154 \\
& ChatGPT & 0.1099 & 0.1124 & \textbf{0.0536} \\
& GPT-4 & 0.0904 & 0.0937 & 0.0167 \\
& GPT-4-Turbo & \textbf{0.1296} & \textbf{0.1346} & 0.0172 \\ \bottomrule
\end{tabular}%
}
\caption{In-domain model performance on the \dataset{} POL task evaluated using the F1 scores.}
% \caption{Model performance on the \dataset{} POL task evaluated using the F1 scores. Llama3 8B\dag{} is optimized using Unsloth.}
\label{tab:pol_results}
\end{table}
%Precision values only
\begin{table}[]
\centering
\small
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{llcccc}
\toprule
& \textbf{Model} & \textbf{P@1} & \textbf{P@4} & \textbf{P@8} & \textbf{P@Inf} \\ \midrule
% & & \multicolumn{4}{c}{{Precision@}} \\
% \cmidrule{2-5}
% & \multirow{-2}{*}{Model} & 1 & 4 & 8 & Inf \\ \midrule
% \multicolumn{5}{c}{Supervised} \\ \midrule
\multirow{5}{1em}{\rotatebox[origin=c]{90}{Supervised}} & PPTOD (base) & 0.1528 & 0.2839 & 0.3261 & 0.3592 \\
& Flan-T5 (base) & 0.1152 & 0.2041 & 0.2383 & 0.2645 \\
& BioGPT & 0.1345 & 0.2509 & 0.2999 & 0.3478 \\
& OpenBioLLM 8B & 0.1608 & 0.2953 & 0.3375 & 0.3740 \\
% & Llama3 8B & 0.1722 & 0.2999 & 0.3512 & 0.3786 \\
& Llama3 8B & \textbf{0.1881} & \textbf{0.3181} & \textbf{0.3820} & \textbf{0.4162} \\ \midrule
% \multicolumn{5}{c}{In-context} \\ \midrule
\multirow{6}{1em}{\rotatebox[origin=c]{90}{In-context}} & OpenBioLLM 8B & \multicolumn{1}{l}{0.0593} & \multicolumn{1}{l}{0.1220} & \multicolumn{1}{l}{0.1505} & \multicolumn{1}{l}{0.1938} \\
& OpenBioLLM 70B & \textbf{0.0855} & 0.1357 & 0.1619 & 0.1950 \\
& Llama3 8B & 0.0616 & 0.1117 & 0.1357 & 0.1847 \\
& Llama3 70B & 0.0604 & 0.1220 & 0.1505 & 0.1779 \\
& ChatGPT & 0.0832 & \textbf{0.1448} & \textbf{0.1790} & \textbf{0.2201} \\
& GPT-4 & 0.0410 & 0.0798 & 0.1106 & 0.1893 \\
& GPT-4-Turbo & \textbf{0.0855} & 0.1357 & 0.1699 & 0.203 \\ \bottomrule
\end{tabular}%
}
% \caption{Model performance on the \dataset{} POL task evaluated using the Precision@K (P@K) scores. Llama3 8B\dag{} is optimized using Unsloth.}
\caption{In-domain model performance on the \dataset{} POL task evaluated using the Precision@K (P@K) scores.}
\label{tab:pol_results2}
\end{table}
% \ref{tab:pol_results}
Tables \ref{tab:pol_results} and \ref{tab:pol_results2} present the performance of various models on the POL task. Unlike NLU, where performance can vary widely, all models demonstrate competitive results in the supervised setting. The Llama3 8B models are the top performers, with Llama3 gaining 0.0069 points over the other baselines. A similar trend is observed with the Precision@K measure, where the Llama3 model maintains a slight advantage over its competitors across different values of K.
% Interestingly, the Llama3 model achieves this despite having a low F1 score for non-medical attributes. During patient-doctor conversations, the doctor asks predominantly for medical attribute values, and non-medical attributes appear in only 7.64\% of the cases -- thus, they have a limited impact on the overall F1.
In the in-context learning setting, all baseline models show similar results, with the GPT-4-Turbo model achieving the highest score. However, for the Precision@K measure at K=4, ChatGPT surpasses the GPT-4-Turbo model. Interestingly, ChatGPT performs better than GPT-4 in this task. Upon careful study, we found that GPT-4 generates responses that violate the CMAS label structure in 20.75\% cases. In contrast, ChatGPT and GPT-4-Turbo make such errors in only 3.31\% and 7.07\%.
Notably, the overall performance of in-context baselines is significantly lower than that of supervised models. This disparity arises because policy learning inherently requires models to plan ahead. Supervised models, which learn policy directly from the data, thus have an advantage.
Even though all models behave similarly, the raw scores are not very high, suggesting that more research is needed to improve this component.%\todo{M: added}
% Table \ref{tab:pol_results} discusses performance of the baseline models on POL task. Unlike in NLU, all baselines demonstrate competitive performance. Here, we observe a few notable trends across various model outputs. Model predicted actions often constitute a part of the expected output, especially in cases where expected action involves querying multiple symptoms. Moreover, models often predict actions that are not immediate but occur a few utterances later. Furthermore, while the expected and predicted actions may not always match precisely, they are often of similar characteristics. For instance, models predict \textit{smoking} when the expected output is \textit{marijuana use}. Additionally, due to the transliterated nature of dialogues, doctors occasionally respond with a mere nod-prompt, an action that models may fail to anticipate. These trends underscore the complexities inherent in policy learning and highlight the importance of refining model capabilities to accurately predict diverse dialogue outcomes.
% The policy learning task involves coming up with the action that the system is going to take in the next utterance.
% There were certain common trends noticed in the output of the various models in this task. Many a times the action taken by the model was a subset of the action taken by the expected output. For example the expected output was to query multiple symptoms however the model queries only a few of them.
% Sometimes, the action taken by the model does not match the immediately next action of the dialogue but may take the same action as 2-3 utterances later.
% Sometimes, the actions of the expected output and prediction may be not be the same but be very similar such as "marijuana use" or "sexually active" as the value in the expected output maybe predicted as "smoking" or "unsafe sex" respectively.
% Finally, since the dialogues are transliterated, the doctors sometimes tend to reply with only a nod-prompt, which the models may not predict.
\subsection{Natural Language Generation}
% Please add the following required packages to your document preamble:
% \usepackage{graphicx}
% Please add the following required packages to your document preamble:
\begin{table}[]
\centering
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{llccccc}
\toprule
& \multirow{2}{*}{\textbf{Model}} & \textbf{BLEU} & \textbf{BLEU} & \textbf{ROUGE} & \textbf{ROUGE} & \textbf{BERT} \\
& & \textbf{2} & \textbf{4} & \textbf{1} & \textbf{L} & \textbf{Score} \\ \midrule
% \multicolumn{6}{c}{Supervised} \\ \hline
\multirow{5}{1em}{\rotatebox[origin=c]{90}{Supervised}} & PPTOD (base) & 18.2 & 8.2 & 0.300 & 0.286 & 0.881 \\
& Flan-T5 (base) & 27.2 & 15.3 & \textbf{0.492} & \textbf{0.466} & 0.907 \\
& BioGPT & 24.3 & 13.1 & 0.462 & 0.438 & 0.901 \\
& OpenBioLLM 8B & 28.8 & 16.0 & 0.488 & 0.462 & \textbf{0.908} \\
% & Llama3 8B & 28.3 & 15.7 & 0.459 & 0.431 & 0.905 \\
& Llama3 8B & \textbf{31.4} & \textbf{17.5} & \textbf{0.492} & 0.459 & \textbf{0.908} \\ \midrule
% \multicolumn{6}{c}{In-context} \\ \hline
\multirow{6}{1em}{\rotatebox[origin=c]{90}{In-context}} & OpenBioLLM 8B & 19.4 & 8.6 & 0.318 & 0.295 & 0.885 \\
& OpenBioLLM 70B & \textbf{21.5} & \textbf{10.5} & \textbf{0.361} & \textbf{0.325} & 0.889 \\
& Llama3 8B & 18.3 & 7.6 & 0.302 & 0.270 & 0.880 \\
& Llama3 70B & 18.9 & 8.1 & 0.310 & 0.274 & 0.881 \\
& ChatGPT & 20.4 & 8.7 & 0.335 & 0.298 & \textbf{0.890} \\
& GPT-4 & 16.1 & 6.3 & 0.300 & 0.260 & 0.870 \\
& GPT-4-Turbo & 19.3 & 8.1 & 0.327 & 0.289 & 0.889 \\ \bottomrule
\end{tabular}%
}
% \caption{Model performance on \dataset{} NLG task. Llama3 8B\dag{} is optimized using Unsloth.}
\caption{In-domain model performance on \dataset{} NLG task.}
\label{tab:nlg_results}
\end{table}
% \begin{table*}[]
% \centering
% \resizebox{0.8\textwidth}{!}{%
% \begin{tabular}{l|ccccc}
% \toprule
% Model & BLEU-2 & BLEU-4 & BERTScore & Rouge1 & RougeL \\ \hline
% \multicolumn{6}{c}{Supervised} \\
% \hline
% PPTOD (base) & 18.19 & 8.19 & 0.8813 & 0.2997 & 0.2855 \\
% Flan-T5 (base) & 27.24 & 15.34 & 0.9074 & 0.4921 & \textbf{0.4656} \\
% BioGPT & 24.34 & 13.14 & 0.9006 & 0.4625 & 0.4378 \\
% OpenBioLLM 8B & \textbf{28.84} & \textbf{15.98} & \textbf{0.9075} & \textbf{0.4878} & 0.4618 \\
% Llama3 8B Instruct & 28.30 & 15.69 & 0.9051 & 0.4590 & 0.4312 \\ \hline
% \multicolumn{6}{c}{In-context} \\
% \hline
% OpenBioLLM 8B & 19.38 & 8.57 & 0.8854 & 0.3181 & 0.2948 \\
% OpenBioLLM 70B & \textbf{21.49} & \textbf{10.53} & 0.8889 & \textbf{0.3611} & \textbf{0.3248} \\
% Llama3 8B Instruct & 18.34 & 7.64 & 0.8803 & 0.3021 & 0.2695 \\
% Llama3 70B Instruct & 18.85 & 8.12 & 0.8809 & 0.3097 & 0.2737 \\
% ChatGPT & 20.42 & 8.73 & \textbf{0.8904} & 0.3352 & 0.2979 \\
% GPT-4 & 16.14 & 6.34 & 0.8705 & 0.2997 & 0.2600 \\ \bottomrule
% \end{tabular}%
% }
% \caption{Model performance on \dataset{} NLG task.}
% \label{tab:nlg_results}
% \end{table*}
Table \ref{tab:nlg_results} reports NLG results. In the supervised setting, Llama3 8B and OpenBioLLM 8B surpass other baselines, with Llama3 8B achieving the best performance across most of the metrics. In the in-context setting, the OpenBioLLM 70B emerges as a clear winner. Interestingly, few-shot models perform very competitively with supervised models on the BERTScore metric. This suggests that their responses are semantically similar to the gold standard, even if they differ lexically.
We analyze responses from the OpenBioLLM 8B supervised model to identify its shortcomings. The model performs well when asking patients for information, effectively using natural phrases like "Ok, and..." to convey understanding. However, it has difficulty converting multiple actions into natural language, particularly towards the end of conversations. It generates repetitive strings when the doctor discusses possible diagnoses and necessary medical tests or provides support. In contrast, ChatGPT and GPT-4 fare well in such cases.
\subsection{Out-of-domain Evaluation}
We tested the performance of leading supervised models (OpenBioLLM 8B, Llama3 8B) and in-context models (ChatGPT, GPT-4-Turbo) on musculoskeletal dialogues in \dataset{}. The results for NLU, POL, and NLG tasks are shown in Tables \ref{tab:ood_nlu}, \ref{tab:ood_pol}, and \ref{tab:ood_nlg}. For NLU and NLG tasks, the models performed worse on out-of-domain data compared to in-domain data. This is because musculoskeletal dialogues use medical terms that are different from those in the respiratory domain. However, for the POL task, models maintained their in-domain performance, even though it was still low. This suggests that more research is needed to close the performance gap for NLU and NLG tasks and to improve POL task performance overall.
\begin{table}[]
\centering
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{lcccc}
\toprule
& \multirow{2}{*}{\textbf{Model}} & \textbf{Overall} & \textbf{Medical} & \textbf{Non-Medical} \\ % \cmidrule{3-5}
& & F1 & F1 & F1 \\ \midrule
\multirow{2}{*}{Supervised} & OpenBioLLM 8B & 0.4747 & 0.5027 & 0.3512 \\
% & Llama 8B & 0.4919 & 0.5196 & 0.3646 \\
& Llama 8B & \textbf{0.524} & \textbf{0.5391} & \textbf{0.4431} \\ \midrule
\multirow{2}{*}{In-Context} & ChatGPT & 0.3637 & 0.3953 & 0.2141 \\
& GPT-4-Turbo & \textbf{0.4481} & \textbf{0.476} & \textbf{0.3176} \\ \bottomrule
\end{tabular}%
}
\caption{Out-of-domain model evaluation on \dataset{} NLU task.}
\label{tab:ood_nlu}
\end{table}
\begin{table}[]
\centering
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{lcccc}
\toprule
& \multirow{2}{*}{\textbf{Model}} & \textbf{Overall} & \textbf{Medical} & \textbf{Non-Medical} \\ % \cmidrule{3-5}
& & F1 & F1 & F1 \\ \midrule
\multirow{2}{*}{Supervised} & OpenBioLLM 8B & \textbf{0.2473} & \textbf{0.2518} & 0.1877 \\
% & Llama 8B & 0.2028 & 0.2039 & 0.1739 \\
& Llama 8B & 0.2236 & 0.2233 & \textbf{0.2294} \\\midrule
\multirow{2}{*}{In-context} & ChatGPT & 0.0836 & 0.0841 & 0.0702 \\
& GPT-4-Turbo & \textbf{0.1252} & \textbf{0.1276} & \textbf{0.0722} \\ \bottomrule
\end{tabular}%
}
\caption{Out-of-domain model evaluation on \dataset{} POL task.}
\label{tab:ood_pol}
\end{table}
\begin{table}[]
\centering
\resizebox{0.495\textwidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
& \multirow{2}{*}{Model} & BLEU & BLEU & ROUGE & ROUGE & BERT \\ %\cmidrule{3-7}
& & 2 & 4 & 1 & L & Score \\ \midrule
\multirow{2}{*}{Supervised} & OpenBioLLM 8B & 25.1 & 13.9 & \textbf{0.486} & \textbf{0.449} & \textbf{0.907} \\
% & Llama 8B & 24.0 & 12.6 & 0.471 & 0.437 & 0.905 \\
& Llama 8B & \textbf{26.7} & \textbf{13.9} & 0.469 & 0.429 & 0.904 \\ \midrule
\multirow{2}{*}{In-context} & ChatGPT & 17.8 & \textbf{7.029} & 0.340 & \textbf{0.297} & \textbf{0.893} \\
& GPT-4-Turbo & \textbf{18.9} & 7.0 & \textbf{0.342} & 0.295 & 0.892 \\ \bottomrule
\end{tabular}%
}
\caption{Out-of-domain model evaluation on \dataset{} NLG task.}
\label{tab:ood_nlg}
\end{table}
% We analyze responses from the OpenBioLLM 8B supervised model to identify its shortcomings. The model performs well when asking patients for information, effectively using natural phrases like "Ok, and..." to convey understanding. However, it has difficulty converting multiple actions into natural language, particularly towards the end of conversations. It generates repetitive strings when the doctor discusses possible diagnoses and necessary medical tests or provides support. In contrast, ChatGPT and GPT-4 fare well in such cases.
% The Natural Language Generation (NLG) task, pivotal in crafting responses, unveils notable trends among model performances. Notably, the transliterated nature of the dialogues reveal intriguing observations, such as models incorporating filler words into their responses, reflecting the data patterns. Additionally, models often initiate responses with the phrase ``Okay, and $\ldots$'', something very particular to human spoken language. Yet the models fail to emulate other human conversational nuances, such as reaffirming patient information or signaling the conclusion of queries at the start of their utterance. Moreover, recurrent issues in NLG like excessive word repetition persist, particularly evident in the utterances in the culmination of dialogues where comprehensive medical tests and diagnoses are presented. Here, attempts to generate lengthy responses result in redundant phrases and medically inconsistent statements, underscoring the models' deficiencies in medical domain knowledge and linguistic finesse.
% The NLG task involves the generation of the response. Some trends were observed in the performance of the models in this task.
% The transliterated nature of the dialogue also showed interesting observations such as the models could pick up the usage of filler words from the data and used them regularly in their responses. Models other than PPTOD also tended to begin most of their responses with the phrase "Okay, and ...". Transliteration however also means that the data would contain human conversational characteristics such as sometimes the doctor's response would begin with a sentence to reconfirm the info given by patient or maybe a statement to say that this will be their last question but the models did not seem to pick up this behaviour and usually jumped directly into the main content of querying the patient or letting them know of the diagnosis.
% Certain phenomena such as the over repetition of some word or phrase too many times is a general problem faced in the NLG task and was shown here as well.
% Issues like these were especially exaggerated in the last few utterances of a dialogue where the various medical tests and diagnoses is supposed to be presented. In the process of trying to generate a long utterance it ends up repeating a lot of phrases and using various paraphrased sentences that mean the same thing one after the other. It also generated many medically inconsistent statements proving its lack of knowledge in the medical domain.
\section{Conclusion and Future Works}
In this work, we introduced \dataset{}, a novel English dataset of doctor-patient dialogues for collecting patient medical history. Unlike existing medical datasets, \dataset{} uses a novel schema (CMAS) to capture attributes such as the onset and duration of symptoms relevant for downstream diagnosis. Further, we link values for medical attributes in \dataset{} labels to their precise medical concepts within UMLS vocabularies. Finally, we propose new benchmarks for NLU, POL, and NLG tasks in the medical dialogue domain. Our initial experiments with baseline models reveal the challenges inherent in these tasks and underscore the potential for improvement.
Furthermore, \dataset{} facilitates the exploration of additional research settings such as Knowledge Grounded TOD. Canonicalization in \dataset{} allows for seamless integration of UMLS vocabularies and Semantic networks into TOD settings, potentially enhancing performance. \dataset{} also opens doors to research in medical dialogue summarization, offering opportunities to distill complex dialogues into concise medical summaries. Our annotation portal can aid in creating large-scale medical dialogue datasets for medical specialties beyond pulmonology and musculoskeletal. This expansion will broaden the applicability and relevance of \dataset{} in medical dialogue research. We release the \dataset{} resources at \url{https://github.com/dair-iitd/MediTOD}.
% \section*{Ethical Considerations and Risk Assessment}
\section*{Ethics Statement}
In this work, we introduce the \dataset{} dataset, which consists of doctor-patient dialogues aimed at gathering patient medical information. This section scrutinizes our data annotation process, as outlined in section \ref{sec:dataset_creation}, from an ethical perspective.
Regarding data sourcing, we thank \citet{fareez2022dataset} for generously providing their data under the Creative Commons CC0 license\footnote{Source dialogues are available \href{https://springernature.figshare.com/articles/dataset/Collection_of_simulated_medical_exams/16550013?backTo=/collections/A_dataset_of_simulated_patient-physician_medical_interviews_with_a_focus_on_respiratory_cases/5545842}{here}.}. The source dialogues in our dataset are simulated interviews conducted by medical professionals, which portray both doctor and patient roles. It's important to note that no actual patient information is disclosed within these dialogues.
We use a professional annotation service specializing in medical data solutions. The provider has been in business for ten years. The labeling process for our dataset involved employing six annotators under the guidance of a doctor. The annotators have backgrounds in medical sciences or pharmacy, ensuring a high level of expertise. The doctor has a professional medical degree and two and half years of hospital experience. Before we started the labeling process, we declared that this work was for scientific advancement and, thus, would be released for public consumption. Each annotator is paid 8 USD per hour, which is above the average wage of data annotators in our country.
We utilize the UMLS Metathesaurus to standardize medical slot values in MediTOD, employing the QuickUMLS software. The National Library of Medicine, Department of Health and Human Services (NLM), grants the UMLS vocabulary license free of charge, which we have obtained for our work. We are committed to properly attributing UMLS and meeting their licensing requirements upon the public release of our dataset.
While releasing MediTOD, we acknowledge its significance in advancing medical dialogue systems. However, it's crucial to emphasize that this data is intended solely for research purposes. We strongly advise against its use in real-life patient consultations or activities that could potentially endanger patients' well-being. Finally, through our work, we want to develop systems for assisting doctors in their work and reducing their burnout. However, we do not claim such a system can work independently without any oversight of healthcare providers.
\section*{Limitations}
While MediTOD makes a meaningful contribution to the medical dialogue community through detailed canonical annotations, it's essential to acknowledge its limitations.
Primarily, MediTOD focuses solely on dialogues from the fields of pulmonology and musculoskeletal. This restricted scope might limit its applicability across other medical specialties. Nonetheless, we're optimistic that our methodology can be adapted to annotate dialogues from different medical fields.
Additionally, the dialogues annotated in MediTOD are exclusively in English. This linguistic limitation may restrict access to the non-English speaking portion of the population. However, we recognize the importance of inclusivity and are open to exploring ways to address language barriers to broaden the reach of our work.
\section*{Acknowledgements}
This work is supported by IBM AI Horizons Network grant, grants by Google, Verisk, and Microsoft, an IBM SUR award and the Jai Gupta chair fellowship by IIT Delhi. Vishal is supported by a Google Fellowship. We thank the IIT Delhi HPC facility for its computational resources. We are grateful to Microsoft AFMR for supporting this work.
\bibliography{custom}
\appendix
\section{\dataset{} Tasks Examples}
\begin{table}[h!]
\small
\centering
\begin{tabular}{p{0.07\textwidth} | p{0.35\textwidth}}
\toprule
Dialogue History & \parbox{0.35\textwidth}{\fontsize{9pt}{11pt} \selectfont
\textbf{Doctor:} What brings you in here today?\\
\textbf{Patient:} Um, I'm just, I'm here because I've had this cough for the past two weeks and uh, it's just not going away.\\
\textcolor{blue}{\textbf{Doctor:} Okay, and um, is it getting worse at all really?\\
\textbf{Patient:} Not really, it's just been the same.}}\\ \midrule
NLU & \fontsize{9pt}{11pt} \selectfont
\begin{Verbatim}[breaklines]
[{
"intent": "inform",
"slots": {
"positive_symptom": [{
"value": "coughing",
"progression": "unchanged with time"
}]
}
}]
\end{Verbatim}
\\ \midrule
Patient State & \fontsize{9pt}{11pt} \selectfont
\begin{Verbatim}[breaklines]
{
"positive_symptom": [{
"value": "coughing",
"onset": "two weeks ago",
"progression": "unchanged with time"
}]
}
\end{Verbatim}
\\ \midrule
POL & \fontsize{9pt}{11pt} \selectfont
\begin{Verbatim}[breaklines]
[{
"action": "inquire",
"symptom": [{
"value": "coughing",
"checks": [{
"type": "characteristics",
"values": ["wet cough or dry cough"]
}]
}]
}]
\end{Verbatim}
\\ \midrule
NLG & \fontsize{9pt}{11pt} \selectfont How would you describe the cough? Is it a wet cough or a dry cough? \\ \bottomrule
\end{tabular}
\caption{NLU, POL and NLG tasks in \dataset{}. Latest exchange is highlighted in \textcolor{blue}{blue}.}
\label{tab:tasks}
\end{table}
% \begin{table}[]
% \small
% \centering
% \begin{tabular}{p{0.1\textwidth} | p{0.38\textwidth}}
% \toprule
% Dialogue History & \parbox{0.38\textwidth}{\fontsize{9pt}{11pt} \selectfont
% \textbf{Doctor:} What brings you in here today?\\
% \textbf{Patient:} Um, I'm just, I'm here because I've had this cough for the past two weeks and uh, it's just not going away.\\
% \textcolor{blue}{\textbf{Doctor:} Okay, and um, is it getting worse at all really?\\
% \textbf{Patient:} Not really, it's just been the same.}}\\ \midrule
% NLU & \parbox{0.38\textwidth}{\fontsize{9pt}{11pt} \selectfont {[}(intent$=$inform, slot$=$positive symptom, \\ value$=$coughing, \textcolor{blue}{progression$=$unchanged with time}){]}} \\ \midrule
% Patient State & \parbox{0.38\textwidth}{\fontsize{9pt}{11pt} \selectfont {[}(slot$=$positive symptom, \\ value$=$coughing, \textcolor{blue}{progression$=$unchanged with time}){]}} \\ \midrule
% POL & \parbox{0.38\textwidth}{\fontsize{9pt}{11pt} \selectfont {[}(action$=$inquire, slot$=$symptom, \\ value$=$coughing, \\ check$=$(type$=$characteristics, value$=$wet cough or dry cough)){]}} \\ \midrule
% NLG & \fontsize{9pt}{11pt} \selectfont How would you describe the cough? Is it a wet cough or a dry cough? \\ \bottomrule
% \end{tabular}
% \caption{NLU, POL and NLG tasks in \dataset{}. Information collected in the latest exchange is highlighted in \textcolor{blue}{blue}.}
% \label{tab:tasks}
% \end{table}
\section{Additional Results}
% Please add the following required packages to your document preamble:
% \usepackage{graphicx}
Table \ref{tab:nlu_results_ext} and Table \ref{tab:pol_results_ext} present precision, recall and F1 for NLU and POL tasks, respectively.
\begin{table*}[]
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{ll|ccc|ccc|ccc}
\toprule
& \multirow{2}{*}{Model} & \multicolumn{3}{c|}{Overall} & \multicolumn{3}{c|}{Medical Attributes} & \multicolumn{3}{c}{Non-medical Attributes} \\
\cmidrule{3-11}
& & Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\ \midrule
% \multicolumn{10}{c}{Supervised} \\ \hline
\multirow{6}{1em}{\rotatebox[origin=c]{90}{Supervised}} & PPTOD (base) & 0.7205 & 0.6528 & 0.6849 & 0.7513 & 0.7039 & 0.7268 & \textbf{0.5827} & 0.4599 & 0.5141 \\
& Flan-T5 (base) & 0.7049 & 0.6733 & 0.6887 & 0.7468 & 0.7244 & 0.7354 & 0.5347 & 0.4807 & 0.5062 \\
& BioGPT & 0.5968 & 0.6217 & 0.6090 & 0.6526 & 0.6701 & 0.6612 & 0.4000 & 0.4392 & 0.4187 \\
& OpenBioLLM 8B & 0.6774 & 0.6689 & 0.6731 & 0.7488 & 0.7110 & 0.7294 & 0.4514 & 0.5104 & 0.4791 \\
% & Llama3 8B & 0.7201 & 0.6901 & 0.7048 & \textbf{0.7830} & 0.7331 & 0.7572 & 0.5071 & 0.5282 & 0.5174 \\
& Llama3 8B & \textbf{0.7230} & \textbf{0.7050} & \textbf{0.7139} & 0.7704 & \textbf{0.7504} & \textbf{0.7603} & 0.5455 & \textbf{0.5341} & \textbf{0.5397} \\ \midrule
% \multicolumn{10}{c}{In-context} \\ \hline
\multirow{6}{1em}{\rotatebox[origin=c]{90}{In-context}} & OpenBioLLM 8B & 0.6654 & 0.5557 & 0.6056 & 0.6826 & 0.6079 & 0.6431 & \textbf{0.5735} & 0.3591 & 0.4416 \\
& OpenBioLLM 70B & 0.5977 & 0.6208 & 0.6090 & 0.6476 & 0.6522 & 0.6499 & 0.4348 & 0.5030 & 0.4664 \\
& Llama3 8B & 0.5208 & 0.5296 & 0.5251 & 0.5714 & 0.5512 & 0.5611 & 0.3692 & 0.4481 & 0.4048 \\
& Llama3 70B & 0.5497 & 0.5996 & 0.5736 & 0.6109 & 0.6191 & 0.6150 & 0.3812 & \textbf{0.5266} & 0.4422 \\
& ChatGPT & 0.5843 & 0.6017 & 0.5929 & 0.6273 & 0.6402 & 0.6337 & 0.4290 & 0.4570 & 0.4425 \\
& GPT-4 & 0.6451 & 0.6254 & 0.6351 & 0.6896 & 0.6543 & 0.6715 & 0.4929 & 0.5163 & 0.5043 \\
& GPT-4-Turbo & \textbf{0.6861} & \textbf{0.6434} & \textbf{0.6641} & \textbf{0.7279} & \textbf{0.6740} & \textbf{0.6999} & 0.5378 & \textbf{0.5282} & \textbf{0.5329} \\ \bottomrule
\end{tabular}%
}
\caption{Model performance on \dataset{} NLU task.}
\label{tab:nlu_results_ext}
\end{table*}
\begin{table*}[]
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{ll|ccc|ccc|ccc|cccc}
\toprule
& \multirow{2}{*}{Model} &
\multicolumn{3}{c|}{Overall} &
\multicolumn{3}{c|}{Medical Attributes} &
\multicolumn{3}{c|}{Non-medical Attributes} &
\multicolumn{4}{c}{Precision@} \\ \cmidrule{3-15}
& & Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 & 1 & 4 & 8 & Inf \\ \midrule
% \multicolumn{14}{c}{Supervised} \\ \hline
\multirow{6}{1em}{\rotatebox[origin=c]{90}{Supervised}} & PPTOD (base) & 0.2243 & 0.1973 & 0.2099 & 0.2240 & 0.1978 & 0.2101 & 0.2308 & 0.1875 & 0.2069 & 0.1528 & 0.2839 & 0.3261 & 0.3592 \\
& Flan-T5 (base) & 0.1756 & \textbf{0.2322} & 0.1999 & 0.1797 & \textbf{0.2341} & 0.2033 & 0.1194 & 0.2000 & 0.1495 & 0.1152 & 0.2041 & 0.2383 & 0.2645 \\
& BioGPT & 0.1831 & 0.1911 & 0.1870 & 0.1816 & 0.1891 & 0.1853 & 0.2069 & \textbf{0.2250} & 0.2156 & 0.1345 & 0.2509 & 0.2999 & 0.3478 \\
& OpenBioLLM 8B & 0.2132 & 0.2041 & 0.2085 & 0.2308 & 0.2051 & 0.2172 & 0.0872 & 0.1875 & 0.1190 & 0.1608 & 0.2953 & 0.3375 & 0.3740 \\
% & Llama3 8B & 0.2278 & 0.2068 & 0.2168 & 0.2351 & 0.2065 & 0.2199 & 0.1491 & 0.2125 & 0.1753 & 0.1722 & 0.2999 & 0.3512 & 0.3786 \\
& Llama3 8B & \textbf{0.2460} & \textbf{0.2322} & \textbf{0.2389} & \textbf{0.2454} & 0.2333 & \textbf{0.2392} & \textbf{0.2576} & 0.2125 & \textbf{0.2329} & \textbf{0.1881} & \textbf{0.3181} & \textbf{0.3820} & \textbf{0.4162} \\
\midrule
% \multicolumn{14}{c}{In-context} \\ \hline
\multirow{6}{1em}{\rotatebox[origin=c]{90}{In-context}} & OpenBioLLM 8B & 0.1156 & 0.0966 & 0.1052 & 0.1177 & 0.1007 & 0.1086 & 0.0513 & 0.0250 & 0.0336 & \multicolumn{1}{l}{0.0593} & \multicolumn{1}{l}{0.1220} & \multicolumn{1}{l}{0.1505} & \multicolumn{1}{l}{0.1938} \\
& OpenBioLLM 70B & 0.1275 & 0.1158 & 0.1214 & 0.1344 & 0.1210 & 0.1273 & 0.0244 & 0.0250 & 0.0247 & 0.0855 & 0.1357 & 0.1619 & 0.1950 \\
& Llama3 8B & 0.1164 & 0.0993 & 0.1072 & 0.1199 & 0.1043 & 0.1116 & 0.0222 & 0.0125 & 0.0160 & 0.0616 & 0.1117 & 0.1357 & 0.1847 \\
& Llama3 70B & 0.1088 & 0.0925 & 0.1000 & 0.1125 & 0.0971 & 0.1042 & 0.0200 & 0.0125 & 0.0154 & 0.0604 & 0.1220 & 0.1505 & 0.1779 \\
& ChatGPT & 0.1191 & 0.1021 & 0.1099 & 0.1198 & 0.1058 & 0.1124 & \textbf{0.0938} & \textbf{0.0375} & \textbf{0.0536} & 0.0832 & \textbf{0.1448} & \textbf{0.1790} & \textbf{0.2201} \\
& GPT-4 & 0.0948 & 0.0863 & 0.0904 & 0.0970 & 0.0906 & 0.0937 & 0.0250 & 0.0125 & 0.0167 & 0.0410 & 0.0798 & 0.1106 & 0.1893 \\
& GPT-4-Turbo & \textbf{0.1384} & \textbf{0.1219} & \textbf{0.1296} & \textbf{0.1416} & \textbf{0.1283} & \textbf{0.1346} & 0.0278 & 0.0125 & 0.0172 & \textbf{0.0855} & 0.1357 & 0.1699 & 0.203 \\ \bottomrule
\end{tabular}%
}
\caption{Model performance on \dataset{} POL task.}
\label{tab:pol_results_ext}
\end{table*}
\begin{table*}[]
\small
\centering
\begin{tabular}{p{0.1\textwidth} | p{0.85\textwidth}}
\toprule
Dialogue History & \parbox{0.85\textwidth}{\fontsize{11pt}{11pt} \selectfont
\textbf{Doctor:} OK, um, and so you you mentioned that you had you had a fever as well. Did you check your temperature? \\
\textbf{Patient:} I didn't actually, no I didn't check my temperature, I just feel hot. \\
\textbf{Doctor:} OK, we'll be sure to check that today in clinic. And have you been experiencing any other symptoms? \\
\textbf{Patient:} Um, just like the brain fog that I was speaking of, you know. I just don't remember, I remember things, but it's it's like I'm doing everything underwater.}\\ \midrule
Gold & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [{"value": "mental fatigue"}]
}
}]
\end{Verbatim}
\\ \midrule
Llama3 8B & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [{"value": "confusion"}]
}
}]
\end{Verbatim}
\\ \midrule
ChatGPT & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [{"value": "confusion"}]
}
}]
\end{Verbatim}
\\ \midrule
GPT-4 & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [{"value": "brain fog",
"positive_characteristics": [
"like doing everything underwater"
]
}]
}
}, {"intent": "inform",
"slots": {
"unknown_symptom": [{"value": "fever"}]
}
}]
\end{Verbatim}
\\ \bottomrule
\end{tabular}
\caption{Supervised Llama3 8B and ChatGPT predict confusion as the active symptom instead of mental fatigue. GPT-4 errs in predicting fever from old utterances.}\label{tab:nlu_example1}
\end{table*}
\begin{table*}[]
\small
\centering
\begin{tabular}{p{0.1\textwidth} | p{0.85\textwidth}}
\toprule
Dialogue History & \parbox{0.85\textwidth}{\fontsize{11pt}{11pt} \selectfont
\textbf{Doctor:} OK, so you're just having difficulties breathing for the past three-four days. Do you have any, uh, do you have a cough? \\
\textbf{Patient:} I did have a, uh, yeah, I've had a cough as well, over these last few days. \\
\textbf{Doctor:} And has it been a dry cough or wet cough? \\
\textbf{Patient:} I've been bringing up, some, like, uh, whitish sputum. It's like, yeah, whitish or clear.}\\ \midrule
Gold & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [
{"value": "coughing", "positive_characteristics": ["wet"]},
{"value": "colored/excess sputum", "color": "whitish or clear"}
]
}
}]
\end{Verbatim}
\\ \midrule
Llama3 8B & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [
{"value": "coughing", "positive_characteristics": ["dry cough"], "positive_symptom_characteristics": ["wet cough"]}
]
}
}]
\end{Verbatim}
\\ \midrule
ChatGPT & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [{"value": "coughing", "color": "whitish or clear sputum"}]
}
}]
\end{Verbatim}
\\ \midrule
GPT-4 & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{"intent": "inform",
"slots": {
"positive_symptom": [
{"value": "coughing", "positive_characteristics": ["wet cough"], "color": "whitish or clear sputum"}
]
}
}]
\end{Verbatim}
\\ \bottomrule
\end{tabular}
\caption{Supervised Llama3 8B hallucinates positive\_symptom\_characteristics attribute. Further, its response includes inconsistent characteristics - wet and dry. ChatGPT and GPT-4 do not predict the excess/colored sputum symptom. They incorrectly link attribute color to cough.}\label{tab:nlu_example2}
\end{table*}
\begin{table*}[]
\small
\centering
\begin{tabular}{p{0.1\textwidth} | p{0.85\textwidth}}
\toprule
Dialogue History & \parbox{0.85\textwidth}{\fontsize{11pt}{11pt} \selectfont
\textbf{Doctor:} OK, that's good. And um travel to anywhere recently? Outside the province? \\
\textbf{Patient:} Uhm, outside of the province, no, not not for like the last year at least. \\
\textbf{Doctor:} OK, OK. Um so those were kind of all the questions that I had for you. Did you have any questions for me? \\
\textbf{Patient:} Um yeah, I was just wondering like what you think it might be.}\\ \midrule
Actions & \fontsize{8pt}{8pt} \selectfont
\begin{Verbatim}[breaklines]
[{'action': 'inquire', 'medication': [
{'value': 'corticosteroid and/or corticosteroid derivative'}]},
{'action': 'inquire', 'medication': [{'value': 'long acting beta agonist'}]},
{'action': 'inquire', 'medication': [{'value': 'azithromycin'}]},
{'action': 'inquire', 'medical_test': [{'value': 'covid-19 rt-pcr testing'}]},
{'action': 'inquire', 'medical_test': [{'value': 'oxygen saturation measurement'}]},
{'action': 'diagnosis', 'disease': [{'value': 'acute exacerbation of chronic obstructive airways disease'},
{'value': 'covid-19 virus disease'}]},
{'action': 'chit-chat'}]
\end{Verbatim}
\\ \midrule
Gold & \parbox{0.85\textwidth}{\fontsize{10pt}{10pt} \selectfont
Yeah, so it definitely sounds like a COPD exacerbation, but because your symptoms overlap with those of Covid, we will take a swab and you should hear back in about 24 to 48 hours. Um and you can you can self isolate for 14 days um and if it's positive, public health will be in touch with you to um do some contact tracing, but in the meantime we'll also do a work up for the COPD exacerbation. Um it seems like you have taken the correct steps so far in like stopping like stopping your smoking and um taking your puffers as needed. Um so right now we will continue using the puffers that you have, and if you don't already have a long acting beta agonist, we'll give you that as well as, if it's like if it gets really severe, we'll also give you some inhaled corticosteroids um and we'll also check your oxygen saturation to make sure it's over uh 88\% um and we'll give you oxygen if you need it. Um as well, some antibiotics to consider, you mentioned that you were allergic to amoxicillin, so we might give you like a macrolide like azithromycin for now. Um and hopefully your symptoms subside and we can get you home. How does that sound?
} \\ \midrule
OpenBioLLM 8B & \parbox{0.85\textwidth}{\fontsize{10pt}{10pt} \selectfont yeah, so it sounds like you have copd, and you've had copd for a long time. um, you've been smoking for a long time, and you've had asthma for a long time. um, and you've had this cough for a long time as well. um, and you've been on medications for a long time, and you've been on inhalers for a long time. um, and you've been on antibiotics for a long time.....} \\ \midrule
ChatGPT & \parbox{0.85\textwidth}{\fontsize{10pt}{10pt} \selectfont yeah, so based on your symptoms and your history of copd, it seems like you might be experiencing an acute exacerbation of your chronic obstructive airways disease. this can cause worsening shortness of breath, coughing, and wheezing. to manage this, we will start you on a corticosteroid to reduce inflammation in your airways and a long-acting beta-agonist to help open up your airways. additionally, we may prescribe azithromycin to help with any potential bacterial infection exacerbating your symptoms. we will also conduct a covid-19 test to rule out any viral infection, and we'll monitor your oxygen saturation to ensure you're getting enough oxygen. does that sound alright to you?} \\ \midrule
GPT-4 & \parbox{0.85\textwidth}{\fontsize{10pt}{10pt} \selectfont based on your symptoms and your history of copd, it seems like you might be experiencing an acute exacerbation of your chronic obstructive pulmonary disease, which is a flare-up of your usual symptoms. this can be triggered by various factors, including infections. given the current situation, we also want to rule out covid-19, so we'll perform a pcr test to check for that. \\
in the meantime, we'll make sure you're on the appropriate medications to manage your copd. this includes corticosteroids to reduce inflammation in your airways, a long-acting beta-agonist to help keep your airways open, and possibly azithromycin if we suspect a bacterial infection is contributing to this exacerbation. we'll also check your oxygen saturation to see if you need supplemental oxygen to help you breathe easier.\\
it's important to monitor your symptoms closely, and if you have any changes or if things get worse, you should let us know immediately. do you have any} \\ \bottomrule
\end{tabular}
\caption{Supervised OpenBioLLM 8B generates a repetitive response for complex actions. ChatGPT and GPT-4 responses are given for comparison.}\label{tab:nlg_example}
\end{table*}
\section{Slot-wise Attributes and Questionnaire}\label{app:slotwise_questions}
Table \ref{tab:slot_and_values} provides slots and associated attributes in \dataset{}.
\begin{table}[]
\centering
\resizebox{0.48\textwidth}{!}{%
\begin{tabular}{l|l}
\toprule
Slots & Attributes \\ \midrule
\multirow{4}{*}{Symptom} & value*, status, location*, onset, initiation, duration, \\
& severity, characteristics, ana\_factors, progression, \\
& volume, color, frequency, rash\_swollen, \\
& lesion\_size, lesions\_peel\_off, itching \\ \bottomrule
Patient Medical History & value*, status, starting, frequency \\ \midrule
Family Medical History & value*, status, relation \\ \midrule
Habits & value*, starting, frequency \\ \midrule
Exposure & value*, status, when, where \\ \midrule
\multirow{2}{*}{Medication} & value*, status, response\_to*,\\ & start, frequency, impact \\ \midrule
Medical Test & value*, status, when \\ \midrule
Disease & value*, status \\ \midrule
Travel Information & value, status, when, frequency \\ \midrule
Occupation & value, status, when, hazards \\ \midrule
Residence & value, status, when, household\_size \\ \midrule
Basic Information & name, age, sex \\ \bottomrule
\end{tabular}%
}
\caption{\dataset{} slots and associated attributes. Attributes marked with * are canonicalised.}
\label{tab:slot_and_values}
\end{table}
We provide the questionnaire associated with each slot in figure \ref{fig:slot_questions}.
\section{Comparing non-medical attributes using ChatGPT}\label{app:metric_prompt}
Given a pair of attribute values INPUT1 and INPUT2, we query ChatGPT with the following prompt to decide whether INPUT1 and INPUT2 are similar in meaning.
\begin{tcolorbox}[]
You are an expert in the English language. Your task is identifying whether two phrases have similar meanings. If the two phrases have similar meanings, say "positive." Otherwise, say "negative". Pay special attention to any medical terms present in the phrases. Here are the phrases.\\
\vspace{1ex}
\noindent Phrase 1: \{\{INPUT1\}\}
\vspace{1ex}
\noindent Phrase 2: \{\{INPUT2\}\}
\vspace{1ex}
\noindent Answer (positive/negative):
\end{tcolorbox}
When evaluated on 100 manually labeled samples, the prompt achieved an accuracy of 95\%.
\begin{figure*}[]
\centering
\includegraphics[width=0.9\textwidth,height=\textheight,keepaspectratio]{latex/images/appendix1.png}
\caption{Slot-wise questionnaire.}
\label{fig:slot_questions}
\end{figure*}
\clearpage
\section{Annotation Guidelines and Labeling Interface}\label{app:annotation_guidelines}
Dear Annotator,
\vspace{1ex}
Thank you for taking the time to help us with this annotation task. Your efforts are greatly appreciated, and your contribution will play a vital role in scientific progress. By proceeding with the annotations, you agree to the public release of the data collected during the process. This document will guide you through the UI and the ontology for the task.
% \par\noindent\rule{0.5\textwidth}{0.8pt}
\subsection*{\underline{Introduction to UI}}
You will annotate the dialogues using the special UI designed for the task. Within each session, you will load a dialogue between a doctor and a patient. In the dialogue, the doctor makes inquiries regarding the patient's symptoms, medical and family history, medication, habits, etc. For each utterance (the doctor’s or the patient’s) you will be presented with a questionnaire. You must fill out the questionnaire based on the utterance under consideration and the dialogue so far.
\vspace{1ex}
\noindent \emph{Rule of thumb: Ensure that the labels for utterances are diagnostically informative, enabling a doctor to make a diagnosis without reviewing the conversation.}
\vspace{1ex}
\noindent \textbf{How to open the UI?}
\vspace{1ex}
\noindent The UI is a simple HTML + JavaScript application. Just open the index.html file from the source folder shared with you. You can use any modern browser you like. However, the tool has been tested extensively on Mozilla Firefox which is recommended. Figure \ref{fig:ann_portal} is a screenshot of the UI.
The UI consists of 5 sections as shown in the figure \ref{fig:ann_portal}.
\begin{enumerate}
\item Control Box – allows loading the dialogue JSON file for annotations, saving/loading the annotations JSON file.
\item dialogue Box – displays the utterances from the loaded dialogue file. You can navigate the utterances using mouse scrolls or up-down arrow keys. You can select an utterance for annotation by clicking on it or by pressing enter. It will load the questionnaire.
\item Questionnaire – contains questions which you must answer given the selected utterance and dialogue history so far.
\item Status Box – displays the labels for annotated utterances.
\item Tracking Box – displays keywords from symptoms, medical and family history. Keywords will be helpful for speeding up the labeling as you move along the utterances.
\end{enumerate}
\vspace{1ex}
\noindent \textbf{How to load a dialogue file?}
\begin{enumerate}
\item Click on the “Browse…” above import dialogue button, in the Control Box.
\item Locate and select the JSON file shared with you.
\item Click the “Import dialogue” button. dialogue box will now display the imported utterances.
\end{enumerate}
\vspace{1ex}
\noindent \textbf{How to add labels for an utterance?}
\begin{enumerate}
\item Select the utterance in the dialogue box. You can use the up-down arrow keys and the Enter key to select the utterance. Use the cross button next to the Submit button to deselect the utterance.
\item Questionnaire will now show a form which you must fill.
\item Select the appropriate “intent” (defined below) from the drop-down. You can hover over each intent to see the details.
\item Select the appropriate “slot type” (defined below) from the drop-down. You can hover over each slot type to see the details. Based on your selection additional questions will be shown.
\item You must decide on which questions are relevant for the given utterance and answer them. Answering requires you to choose an option from a drop-down menu or type answers into a text box. You may provide multiple answers in the text box, separating them with commas.
\item Click on the submit button to add the labels. Status box will now show the added labels in the JSON format.
Make sure you add labels for all the utterances in the dialogue.
\end{enumerate}
\vspace{1ex}
\noindent \textbf{How to save the labels?}
\begin{enumerate}
\item Once you finish adding labels for all the utterances, click on the “Save” button in the Control Box.
\item Provide an appropriate file name. For example, if the dialogue file has the name “ABC,” then you can name the label file “ABC\_annotations.”
\end{enumerate}
\vspace{1ex}
\noindent \textbf{How can I edit the labels for an utterance?}
\begin{enumerate}
\item Go to the Status Box and find the label that you want to change. Click on the associated “X” button to remove the annotation.
\item Re-add the annotation for the utterance as discussed before.
\end{enumerate}
\vspace{1ex}
\noindent \textbf{Can I view and edit labels from a saved JSON?}
\begin{enumerate}
\item From the Control Box, first load the dialogue as before.
\item Click on “Browse” above the Load Annotation button. Select the appropriate JSON labels file.
\item Click on the “Load Annotations” button. You will see annotations loaded in the Status Box.
\end{enumerate}
% \par\noindent\rule{0.5\textwidth}{0.8pt}
\subsection*{\underline{Task Ontology}}
In this section, we describe the overall ontology of the task. In addition, we will also detail out with examples how to label each slot type in the ontology. For each utterance, you must
\begin{enumerate}
\item Decide an appropriate intent.
\item Decide an appropriate slot type.
\item Fill out the questionnaire corresponding to the intent-slot type pair.
\end{enumerate}
The UI will automatically display the questionnaire (if any) once you select the intent and slot type.
\vspace{1ex} \noindent
\emph{\textbf{Note:} An utterance can have more than one intent-slot value pair. You must fill out a questionnaire for each pair. The UI allows this by re-selecting the utterance.}
\vspace{1ex} \noindent
\emph{\textbf{Note:} In some dialogues, the patient is an infant (or is unable to communicate) and is accompanied by its guardian (like its mother). In such cases patient responses are actually uttered by the guardian. However, you must label the utterance from the perspective of the patient.}
For example, “Patient: Timmy is my son. He has been running a high fever.” should be labelled appropriately as a patient is having a fever.
% \par\noindent\rule{0.5\textwidth}{0.2pt}
\vspace{1ex}
\noindent \emph{\textbf{Intents}}
An intent represents the underlying purpose or meaning behind a speaker's statement in a dialogue, whether it's the doctor or a patient. The following intent labels are available:
\begin{enumerate}
\item \textbf{Inform:} When the speaker aims to provide specific information, such as symptoms or medical history. This could be in response to an inquiry or spontaneously offered. Select this intent when "specific information" is required for an informed diagnosis.
\item \textbf{Inquire:} When the speaker seeks to gather specific information, such as symptoms or medical history. Choose this intent when "specific information" is necessary for an educated diagnosis.
\item \textbf{Diagnosis:} When the doctor is giving a diagnosis of a disease.
\item \textbf{Salutations:} When the speaker intends to convey a greeting or farewell message.
\item \textbf{Chit-chat:} When the speaker engages in casual conversation. The information in the utterance is unlikely to contribute to an educated diagnosis.
\item \textbf{Nod\_prompt:} When the speaker is not providing any new information but is showing attention, understanding, or agreement through phrases like 'Okay,' 'Yeah,' and 'uh-huh.' We consider an utterance as nod\_prompt when the speaker is either acknowledging something (like a patient when he/she understands a doctor's question) or prompting the listener for additional information (like when the doctor just says okay and patient continues the conversation).
\item \textbf{Other:} Any intent not covered by the above categories.
\end{enumerate}
\emph{Ensure that you consider all possible intents conveyed by the utterance when labeling them.}
% \par\noindent\rule{0.5\textwidth}{0.2pt}
\vspace{1ex}
\noindent \emph{\textbf{Slots}}
Slots refer to specific pieces of information or variables that are extracted from an utterance in a dialogue.
\vspace{1ex} \noindent
\textbf{Basic Information} Slots in a dialogue capture specific details such as the patient's name, age, and sex. Examples are given in figure \ref{fig:inst_basic_information}.
\vspace{1ex} \noindent
\textbf{Symptom} The dialogue contains slots with details about a symptom experienced by the patient. These slots encompass the symptom's value (e.g., cough or fever) and additional information like its onset, nature, and more. The UI presents the following questions for symptoms.
\begin{enumerate}
\item Enter symptoms – comma separated values for the symptoms.
\item Is the patient currently suffering from the selected symptom(s)? – Yes/No
\item Where is the symptom(s) located? – Part of the body affected by the symptom
\item When did this symptom(s) appear? – 2 days ago, yesterday, etc
\item How did this symptom(s) appear (abruptly, gradually, etc.)? - Onset
\item How long does this symptom(s) last (few minutes, few days, etc.)?
\item What is the severity of this symptom(s) on scale of 10 (or an indirection like can patient perform his/her day to day activity.)?
\item What are the characteristics of the symptom(s) (burning pain and dry cough etc.)?
\item What factors cause symptom(s) to improve or get worse (alleviating/aggravating factors like symptoms get worse with exercise)?
\item How is the symptom(s)'s progression over time (eg. cough has increased over time)?
\item Volume if applicable to the symptom (eg. amount of sputum)
\item Color if applicable to the symptom (eg. color of sputum)
\item Frequency if applicable to the symptom (eg. patient coughs 3-4 times a day)
\item Any additional information missing from the above fields
\end{enumerate}
Examples are given in figure \ref{fig:inst_symptom}.
\vspace{1ex} \noindent
\textbf{Dermatological Symptom} In this case, the utterance comprises slots with information related to skin, nails, and hair symptoms. It includes the symptom's value (e.g., rash) and additional attributes like color, size, swelling, etc. The UI presents the following questions for dermatological symptoms.
\begin{enumerate}
\item Does the patient have any lesions, redness or problems on the skin?
\item Is the rash swollen?
\item Is the lesion (or are the lesions) larger than 1cm?
\item Do the lesions peel off?
\item How severe is the itching on scale of 10?
\item Where is the affected region located?
\item What color is the rash?
\item Any additional information missing from the above fields.
\end{enumerate}
Examples are given in figure \ref{fig:inst_derma}.
\vspace{1ex} \noindent
\textbf{Disease} The doctor is diagnosing a disease in the utterance. The UI presents the following questions.
\begin{enumerate}
\item Enter the disease(s) – comma separated list of diseases.
\item Does the patient have the disease(s)? – Yes/No/Maybe
\end{enumerate}
Examples are given in figure \ref{fig:inst_disease}.
\vspace{1ex} \noindent
\textbf{Exposure} Within the dialogue, there are slots containing details about situations in which the patient might be exposed to harmful conditions. This includes contact with allergic substances, dust, chemicals, or infected individuals. The UI presents the following questions.
\begin{enumerate}
\item Select exposure factor(s) – comma separated values of exposure factors
\item Was the patient exposed to the selected exposure factor? – Yes/No/Maybe
\item When was the patient exposed to the above factor?
\item Where was the patient exposed to the above factor e.g. at work or at home?
\item Any additional information missing from the above fields
\end{enumerate}
Examples are given in figure \ref{fig:inst_exposure}.
\vspace{1ex} \noindent
\textbf{Habit} The dialogue contains slots with information about the patient's habits or addictions. A habit refers to an activity the patient regularly engages in, ranging from daily exercise, tea, and coffee to smoking, alcoholism, and marijuana abuse. The UI presents the following questions.
\begin{enumerate}
\item Select activity/activities – comma separated list
\item Has/had the patient formed a habit/addiction to the selected activities? – Yes/No/Maybe
\item How frequently does the patient engages into the activity?
\item When did the patient picked up the activities?
\item Enter any additional information missing from above fields.
\end{enumerate}
Examples are given in figure \ref{fig:inst_habit}.
\vspace{1ex} \noindent
\emph{\textbf{Note:} Patients tend to get embarrassed with questions like “How much alcohol do you take?” or “Do you smoke cigarettes”? You must make your own judgment in such cases and decide whether the patient is addicted or not.}
\vspace{1ex} \noindent
\textbf{Medication} The dialogue contains slots with details about medications, either specific ones like Tylenol or general ones like antipsychotic drugs. Additional information may include the purpose of the medication and the duration the patient has been taking it. The doctor may also communicate medication-related information to the patient. The UI presents the following questions.
\begin{enumerate}
\item Enter medication – comma separated list
\item Medication Status – currently taking/took in the past/no
\item For which condition/symptom is medication for?
\item Since when did the patient start taking the medication?
\item How frequently does the patient take the medication?
\item Did the medication help the patient?
\item Any additional information missing from above field.
\end{enumerate}
Examples are given in figure \ref{fig:inst_medication}.
\vspace{1ex} \noindent
\textbf{Medical Test}
Slots in the utterance pertain to a medical test, such as ECG or CAT scan. The doctor might inquire about tests the patient has already undergone or advise the patient to undergo specific tests. The UI presents the following questions.
\begin{enumerate}
\item Enter medical test
\item Does patient has the results for the medical test?
\item When did the patient had the medical test done?
\item Any additional information missing from above field.
\end{enumerate}
Examples are given in figure \ref{fig:inst_medical_test}.
\vspace{1ex} \noindent
\textbf{Medical History}
The dialogue includes slots that provide information about the patient's medical history. This may encompass descriptions of past symptoms, diseases, surgeries, or allergies experienced by the patient. It differs from the "Disease" slot as it describes previous medical conditions rather than ongoing symptoms. The UI presents the following questions.
\begin{enumerate}
\item Add symptom/disease/surgery relevant to patient's medical history – comma separated list
\item Status of the above condition
\begin{enumerate}
\item Patient still suffers from the condition
\item Patient suffered from the condition in the past
\item Patient did not suffer from the condition in the past
\item Patient is not sure about the status of the condition
\end{enumerate}
\item When did the patient start to experience the above condition?
\item How frequently does the patient experience the above condition?
\item Any additional information missing from the above fields
\end{enumerate}
Examples are given in figure \ref{fig:inst_medical_history}.
\vspace{1ex} \noindent
\textbf{Family History}
The utterance contains slots with information about medical conditions prevalent in the patient's family. This includes diseases like asthma, heart issues, cancer, and others. The UI presents the following questions.
\begin{enumerate}
\item Select medical condition(s) – comma separated list
\item Does anyone in the patient's family have the selected medical condition(s)? Yes/No/Maybe
\item Relationship with the patient e.g. mother or brother
\item Any additional information missing from the above fields
\end{enumerate}
Examples are given in figure \ref{fig:inst_family_history}.
\vspace{1ex} \noindent
\textbf{Occupation}
Within the dialogue, there are slots specifying the patient's occupation, such as teacher, trucker, factory worker, etc. The UI presents the following questions.
\begin{enumerate}
\item Add patient's occupation details like job sector or job title
\item Has the patient works/worked at the above occupation?
\item Are there any substances/dangers to which the patient is exposed at work?
\item Any additional information missing from the above fields.
\end{enumerate}
Examples are given in figure \ref{fig:inst_occupation}.
\vspace{1ex} \noindent
\textbf{Residence}
Slots in the utterance contain details about the patient's residence, such as urban, rural, suburban, etc. The UI presents the following questions.
\begin{enumerate}
\item Add details for the patient's residence like urban/rural and apartment/house.
\item Status
\item Any additional information missing from the above fields.
\end{enumerate}
Examples are given in figure \ref{fig:inst_residence}.
\vspace{1ex} \noindent
\textbf{Travel}
The dialogue includes slots with information about the patient's travel history. This may involve details like the time of travel, locations visited, and frequency of travel. The UI presents the following questions.
\begin{enumerate}
\item Has the patient travelled recently?
\item Where has the patient travelled to?
\item When did the patient travel?
\item How frequently does the patient travel?
\item Enter additional information about travel.
\end{enumerate}
Examples are given in figure \ref{fig:inst_travel}.
\vspace{1ex} \noindent
\textbf{Medical Discussion}
The utterance is part of a chit-chat about a medical topic (e.g., pulmonary embolism). The slot values of this type are unlikely to contribute towards the diagnosis. The UI presents the following questions – “What is the topic of the discussion?”. Examples are given in figure \ref{fig:inst_medical_discussion}.
\vspace{1ex} \noindent
\textbf{Non-Medical Discussion}
The utterance is part of a chit-chat about a non-medical topic (e.g., living conditions). The slot values of this type are unlikely to contribute towards the diagnosis. The UI presents the following questions – “What is the topic of the discussion?”.
\vspace{1ex} \noindent
\textbf{Other}
This slot accounts for any additional details present in the utterance beyond the ones mentioned above. The UI asks for the “other” information. You must summarize it as succinctly as possible. Examples are given in figure \ref{fig:inst_other}.
% \par\noindent\rule{0.5\textwidth}{0.8pt}
\subsection*{\underline{Miscellaneous}}
\vspace{1ex} \noindent
\textbf{Special Cases}
\vspace{1ex} \noindent
Following are some special cases (not all) which may frequently appear in the dialogues.
\begin{enumerate}
\item If the utterance indicates the number of people living with the patient, the slot type is residence and value is household size. For example, “Patient: I live with my parents and my sister” should be annotated as \{"intent": "inform", "slot": "residence", "household size": "4"\}.
\item For a case where the doctor asks “have you experienced these symptoms before?”the slot type is medical history and value is past experience.
\item In case the patient is exposed to secondhand cigarette smoke (smoke from someone else’s cigarette), the slot type is habit and value is secondhand cigarette.
\item It is preferred to use key-value format for Other field in the questionnaire. For example, “Doctor: OK. How has his behaviour been? Patient: He's been very, very fussy.” should be annotated as \{"intent": "inquire", "slot": "other", "other information": "behaviour"\}, \{"intent": "inform", "slot": "other", "other information": "behaviour: fussy"\}.
\item For alcoholism (slot-type habit), the patient might say “I usually drink a glass of wine on the weekends.” You must rely on your medical knowledge to decide whether the patient is alcoholic or not. You may refer CAGE guidelines for alcoholism and annotate the utterance as \{"intent": "inform", "slot": "habit", "value": "alcoholism", "status": "No", "other": "criterion: CAGE"\}. Similarly, for smoking and other substance abuse.
\item For the cases where the patient is an infant, the doctor asks questions to the mother like “Did you have any complications during pregnancy?”. Here, slot type is medical history and values can be typed-in. As discussed before, you must answer a questionnaire from the perspective of the patient.
\end{enumerate}
\vspace{1ex} \noindent
\textbf{Navigating Utterances}
\begin{enumerate}
\item dialogue box indicates the active utterance by surrounding it with a blue box. Use up-down arrow keys for navigating the utterances in the dialogue box.
\item Once the blue box surrounds the utterance of your choice, press Enter to enable the questionnaire. Focus will now shift to the intent field in the question. dialogue box will now be disabled.
\item Press Esc to cancel the questionnaire and return to the dialogue box.
\end{enumerate}
\vspace{1ex} \noindent
\textbf{Navigating Questionnaire}
\begin{enumerate}
\item Press Tab to move the focus to the next field in the questionnaire.
\item Press Shift + Tab to move the focus to the next field in the questionnaire.
\item If the questionnaire field is a checkbox, use Space to check/uncheck the box.
\end{enumerate}
\vspace{1ex} \noindent
\textbf{Keyword Searches}
\vspace{1ex} \noindent
Many questionnaire fields have search support (these fields have “Type to search” as a placeholder).
\begin{enumerate}
\item Start typing in words in the input field. A drop-down menu will appear with possible matches.
\item Navigate through the drop-down using up-down arrow keys.
\item Press Enter to confirm the selected entry. It will now appear in the input field.
\item The input field accepts multiple values. Press comma (,) and the drop-down will re-appear for another selection.
\item If none of the entries in the drop-down fit the requirement, simply type-in the needed value.
\end{enumerate}
\vspace{1ex} \noindent
\textbf{Tracking Box for Faster Labeling}
\vspace{1ex} \noindent
Since slots like symptoms, diseases, medical and family history are repeated in the dialogue, the UI allows an easy way to copy them using the tracking box.
\begin{enumerate}
\item Select the required input field in the questionnaire (symptoms, disease, medical and family history).
\item Click on the slot-value in the tracking box. It will be automatically copied to the input field.
\end{enumerate}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/basic_information.png}
\caption{Examples of Basic information}
\label{fig:inst_basic_information}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/symptoms.png}
\caption{Examples of Symptoms}
\label{fig:inst_symptom}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/derma.png}
\caption{Examples of Dermatological symptoms}
\label{fig:inst_derma}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/disease.png}
\caption{Examples of Diseases}
\label{fig:inst_disease}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/exposure.png}
\caption{Examples of Exposure}
\label{fig:inst_exposure}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/habit.png}
\caption{Examples of Habit}
\label{fig:inst_habit}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/medication.png}
\caption{Examples of Medication}
\label{fig:inst_medication}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/medical_test.png}
\caption{Examples for Medical test}
\label{fig:inst_medical_test}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/medical_history.png}
\caption{Examples of Medical history}
\label{fig:inst_medical_history}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/family_history.png}
\caption{Examples of Family history}
\label{fig:inst_family_history}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/occupation.png}
\caption{Examples of Occupation}
\label{fig:inst_occupation}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/residence.png}
\caption{Examples of Residence}
\label{fig:inst_residence}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/travel.png}
\caption{Examples of Travel}
\label{fig:inst_travel}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/medical_discussion.png}
\caption{Examples of Medical discussion}
\label{fig:inst_medical_discussion}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/images/instructions/other.png}
\caption{Examples for Others}
\label{fig:inst_other}
\end{figure*}
\onecolumn
\clearpage
\section{Prompts in In-context Setting}\label{app:prompts}
\subsection{NLU Prompt}\label{app:nlu_prompt}
\begin{tcolorbox}[
enhanced, breakable, colback=white,
colframe=black, boxrule=0.25mm,
width=\textwidth
% before=\vspace{1ex},
% after=\vspace{1ex},
]
\fontsize{6pt}{6pt}\selectfont
You are a professional medical scribe who is an expert in understanding doctor-patient dialogues. The user will show you a dialogue history between a doctor and a patient and the last turn in their dialogue. Your task is to identify the patient's intent, slots, and related attributes (if applicable) from the given the dialogue history and the last turn. Definitions for intent, slots, and related attributes are given below as Python dictionaries.
\begin{Verbatim}[breaklines]
```
intents = [{
"name": "inform",
"description": "The patient is providing information to the doctor."
},
{
"name": "chit-chat",
"description": "The patient is chit-chatting with the doctor."
},
{
"name": "nod_prompt_salutations",
"description": "The patient is nodding to the doctor or delivering salutations."
}]
slots = [{
"slot": "symptom",
"description": "A symptom relevant to the patient's condition.",
"related_attributes": [
{"name": "value", "description": "The symptom in medical terms.", "examples": ["coughing, dyspnea"]},
{"name": "status", "description": "The status is 'positive' if the patient has the symptom currently or 'negative' if the patient does not have the symptom; otherwise, it is 'unknown.'"},
{"name": "onset", "description": "When did this symptom appear?", "examples": ["three days ago", "one week back"]},
{"name": "initiation", "description": "How did this symptom appear?", "examples": ["abruptly", "gradually"]},
{"name": "location", "description": "Where is the symptom located?", "examples": ["back", "neck"]},
{"name": "duration", "description": "How long does the symptom persist?", "examples": ["a few minutes", "a few hours"]},
{"name": "severity", "description": "What is the severity of this symptom on a scale of 10?", "examples": ["4", "7"]},
{"name": "progression", "description": "How is the symptom's progression?", "examples": ["getting worse", "constant"]},
{"name": "frequency", "description": "Frequency, if applicable, to the symptom.", "examples": ["3-4 times a day", "every hour"]},
{"name": "positive_characteristics", "description": "A characteristic positively associated with the symptom.", "examples": ["sharp", "burning"]},
{"name": "negative_characteristics", "description": "A characteristic not associated with the symptom.", "examples": ["sharp", "burning"]},
{"name": "unknown_characteristics", "description": "A characteristic with unknown relation with the symptom.", "examples": ["sharp", "burning"]},
{"name": "alleviating_factor", "description": "A condition that alleviates the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "not_alleviating_factor", "description": "A condition that does not alleviate the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "aggravating_factor", "description": "A condition that aggravates the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "not_aggravating_factor", "description": "A condition that does not aggravate the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "not_alleviating_aggravating_factor", "description": "A condition that neither alleviates nor aggravates the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "unknown_factor", "description": "A condition with unknown alleviation/aggravation status.", "examples": ["laying down", "sleeping"]},
{"name": "volume", "description": "Volume, if applicable to the symptom.", "examples": ["couple of teaspoons"]},
{"name": "color", "description": "Color, if applicable to the symptom.", "examples": ["ping", "red"]},
{"name": "itching", "description": "How severe is the itching on a scale of 10?", "examples": ["4", "7"]},
{"name": "lesion_size", "description": "Is the lesion (or are the lesions) larger than 1cm (Yes/No)?"},
{"name": "lesions_peel_off", "description": "Do the lesions peel off (Yes/No)?"},
{"name": "rash_swollen", "description": "Is the rash swollen (Yes/No)?"}
]
}, {
"slot": "medical_history",
"description": "A medical condition relevant to the patient's medical history.",
"related_attributes": [
{"name": "value", "description": "Name of the medical condition.", "examples": ["hypertensive disease", "malignant neoplasm"]},
{"name": "status", "description": "The status is 'positive' if the patient experienced the medical condition or 'negative' if the patient did not experience the medical condition; otherwise, it is 'unknown.'"},
{"name": "starting", "description": "When did the patient start to experience the condition?", "examples": ["since teenage", "ten years ago"]},
{"name": "frequency", "description": "How frequently does the patient experience the added condition?", "examples": ["every year", "during summer"]}
]
}, {
"slot": "family_history",
"description": "A medical condition relevant to the patient's family.",
"related_attributes": [
{"name": "value", "description": "Name of the medical condition.", "examples": ["hypertensive disease", "malignant neoplasm"]},
{"name": "status", "description": "The status is 'positive' if someone in the patient's family suffered from the medical condition or 'negative' if no one in the patient's family suffered from the medical condition; otherwise, it is 'unknown.'"},
{"name": "relation", "description": "Relationship with the patient", "examples": ["mother", "aunt"]}
]
}, {
"slot": "habit",
"description": "An habitual activity such as smoking, alcoholism, etc.",
"related_attributes": [
{"name": "value", "description": "Name of an activity.", "examples": ["smoking", "marijuana"]},
{"name": "status", "description": "The status is 'positive' if the patient engages in the activity habitually or 'negative' if the patient does not engage in the activity habitually; otherwise, it is 'unknown.'"},
{"name": "starting", "description": "When did the patient pick up the activities?", "examples": ["ten years back", "as a child"]},
{"name": "frequency", "description": "How frequently does the patient engage in the selected activity?", "examples": ["on weekends", "every day"]}
]
}, {
"slot": "exposure",
"description": "An environmental/chemical factor such as asbestos, pets, etc.",
"related_attributes": [
{"name": "value", "description": "Name of an environmental factor.", "examples": ["pets", "dust"]},
{"name": "status", "description": "The status is 'positive' if the patient was exposed to the factor or 'negative' if the patient was not exposed; otherwise, it is 'unknown.'"},
{"name": "where", "description": "Where was the patient exposed to the selected factor?", "examples": ["work", "home"]},
{"name": "when", "description": "When was the patient exposed to the selected factor?", "examples": ["four days ago"]}
]
}, {
"slot": "medication",
"description": "A medication.",
"related_attributes": [
{"name": "value", "description": "Name of a medication.", "examples": ["over-the-counter medicine", "paracetamol"]},
{"name": "status", "description": "The status is 'positive' if the patient took the medicine or 'negative' if the patient did not take the medicine; otherwize, it is unknown."},
{"name": "start", "description": "Since when did the patient start taking the medication?", "examples": ["few weeks ago", "two days back"]},
{"name": "impact", "description": "Did the medication help the patient (Yes/No/Maybe)?"},
{"name": "respone_to", "description": "For which condition/symptom is medication for?", "examples": ["hypertensive disease", "diabetes"]},
{"name": "frequency", "description": "How frequently does the patient take the medication?", "examples": ["daily"]}
]
}, {
"slot": "medical_test",
"description": "A medical test.",
"related_attributes": [
{"name": "value", "description": "Name of a medical test.", "examples": ["chest X-ray", "electrocardiogram"]},
{"name": "status", "description": "The status is 'avail' if the patient took the test or 'unavail' if the patient did not take the test; otherwise, it is 'unknown.'"},
{"name": "when", "description": "When did the patient had the medical test done?", "examples": ["yesterday", "a week ago"]}
]
}, {
"slot": "residence",
"description": "Information regarding patient's living conditions.",
"related_attributes": [
{"name": "value", "description": "Place where the patient resides.", "examples": ["apartment", "old building"]},
{"name": "status", "description": "The status is 'living' if the patient is currently living at the place or 'not_living' if the patient is not currently living at the place."},
{"name": "household_size", "description": "Size of the patient's household.", "examples": ["2", "4"]}
]
}, {
"slot": "occupation",
"description": "Information regarding the patient's occupation.",
"related_attributes": [
{"name": "value", "description": "Job/occupation of the patient.", "examples": ["nurse", "student"]},
{"name": "status", "description": "The status is 'true' if the patient works/worked at the above occupation or 'false' if the patient does/did not work at the above occupation."},
{"name": "exposure", "description": "Are there any hazards/substances/dangers to which the patient got exposed at work?", "examples": ["chemical fumes", "dust"]}
]
}, {
"slot": "travel",
"description": "Information regarding the patient's recent travels.",
"related_attributes": [
{"name": "destination", "description": "Where has the patient travelled to?", "examples": ["canada", "united states"]},
{"name": "status", "description": "The status is 'traveled' if the patient travelled recently or 'not_traveled' if the patient did not."},
{"name": "date", "description": "When did the patient travel?", "examples": ["last week", "a year ago"]}
]
}, {
"slot": "basic_information",
"description": "Basic information about the patient.",
"related_attributes": [
{"name": "age", "description": "Age of the patient.", "examples": ["32", "50"]},
{"name": "gender", "description": "Gender of the patient.", "examples": ["male", "female"]},
{"name": "name", "description": "Name of the patient.", "examples": ["John", "Lily"]}
]
}]
```
IMPORTANT INSTRUCTIONS:
1. Read the given definitions carefully.
2. For a given dialogue history and last turn, only some of the intents, slots and related attributes are applicable.
3. Related attribute 'value' of the the slots symptom, medical_history, family_history, habit, exposure, medication and medical_test must be a standard medical concept.
4. Expected output should contain intent, slot and related values from the last turn. dialogue history is given as an additional context.
[dialogue history]
doctor: Uh itchy eyes, discharge anything like that?
patient: No, nothing like that.
[last turn]
doctor: About a stuffy or runny nose?
patient: No, nothing like that.
[output]
[{"intent": "inform", "slots": {"symptom": [{"value": "nasal congestion", "status": "negative"}, {"value": "rhinorrhea", "status": "negative"}]}}]
.
.
{{Remaining Exemplars}}
.
.
[dialogue history]
doctor: And um, have you had, uh, have you had any headaches?
patient: Uh, no headaches.
[last turn]
doctor: Any uh, stuffy nose or runny nose?
patient: Uh no, nothing like that.
[output]
\end{Verbatim}
\end{tcolorbox}
\subsection{POL Prompt}\label{app:pol_prompt}
\begin{tcolorbox}[
enhanced, breakable, colback=white,
colframe=black, boxrule=0.25mm,
width=\textwidth
% before=\vspace{1ex},
% after=\vspace{1ex},
]
\fontsize{6pt}{6pt}\selectfont
You are a professional medical assistant who is an expert in understanding doctor-patient dialogues. The user will show you current state of the dialogue between a doctor and a patient and the last turn in their dialogue. Your task is to suggest the doctor's action as a continuation of the dialogue. Doctor's action consists of intents, slots and related attributes (if applicable). Definitions for intent, slots, and related attributes are given below as Python dictionaries.
\begin{Verbatim}[breaklines]
```
intents = [{
"name": "inquire",
"description": "The doctor is inquiring information from the doctor."
},
{
"name": "chit-chat",
"description": "The doctor is chit-chatting with the patient."
},
{
"name": "nod_prompt_salutations",
"description": "The doctor is nodding to the patient or delivering salutations."
},
{
"name": "diagnosis",
"description": "The doctor making a diagnosis."
},
{
"name": "other",
"description": "Any other action."
}]
slots = [{
"slot": "symptom",
"description": "A symptom relevant to the patient's condition.",
"related_attributes": [
{"name": "value", "description": "The symptom in medical terms.", "examples": ["coughing, dyspnea"]},
{"name": "onset", "description": "When did this symptom appear?", "examples": ["three days ago", "one week back"]},
{"name": "initiation", "description": "How did this symptom appear?", "examples": ["abruptly", "gradually"]},
{"name": "location", "description": "Where is the symptom located?", "examples": ["back", "neck"]},
{"name": "duration", "description": "How long does the symptom persist?", "examples": ["a few minutes", "a few hours"]},
{"name": "severity", "description": "What is the severity of this symptom on a scale of 10?", "examples": ["4", "7"]},
{"name": "progression", "description": "How is the symptom's progression?", "examples": ["getting worse", "constant"]},
{"name": "frequency", "description": "Frequency, if applicable, to the symptom.", "examples": ["3-4 times a day", "every hour"]},
{"name": "positive_characteristics", "description": "A characteristic positively associated with the symptom.", "examples": ["sharp", "burning"]},
{"name": "negative_characteristics", "description": "A characteristic not associated with the symptom.", "examples": ["sharp", "burning"]},
{"name": "unknown_characteristics", "description": "A characteristic with unknown relation with the symptom.", "examples": ["sharp", "burning"]},
{"name": "alleviating_factor", "description": "A condition that alleviates the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "not_alleviating_factor", "description": "A condition that does not alleviate the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "aggravating_factor", "description": "A condition that aggravates the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "not_aggravating_factor", "description": "A condition that does not aggravate the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "not_alleviating_aggravating_factor", "description": "A condition that neither alleviates nor aggravates the symptom.", "examples": ["laying down", "sleeping"]},
{"name": "unknown_factor", "description": "A condition with unknown alleviation/aggravation status.", "examples": ["laying down", "sleeping"]},
{"name": "volume", "description": "Volume, if applicable to the symptom.", "examples": ["couple of teaspoons"]},
{"name": "color", "description": "Color, if applicable to the symptom.", "examples": ["ping", "red"]},
{"name": "itching", "description": "How severe is the itching on a scale of 10?", "examples": ["4", "7"]},
{"name": "lesion_size", "description": "Is the lesion (or are the lesions) larger than 1cm (Yes/No)?"},
{"name": "lesions_peel_off", "description": "Do the lesions peel off (Yes/No)?"},
{"name": "rash_swollen", "description": "Is the rash swollen (Yes/No)?"}
]
}, {
"slot": "medical_history",
"description": "A medical condition relevant to the patient's medical history.",
"related_attributes": [
{"name": "value", "description": "Name of the medical condition.", "examples": ["hypertensive disease", "malignant neoplasm"]},
{"name": "starting", "description": "When did the patient start to experience the condition?", "examples": ["since teenage", "ten years ago"]},
{"name": "frequency", "description": "How frequently does the patient experience the added condition?", "examples": ["every year", "during summer"]}
]
}, {
"slot": "family_history",
"description": "A medical condition relevant to the patient's family.",
"related_attributes": [
{"name": "value", "description": "Name of the medical condition.", "examples": ["hypertensive disease", "malignant neoplasm"]},
{"name": "relation", "description": "Relationship with the patient", "examples": ["mother", "aunt"]}
]
}, {
"slot": "habit",
"description": "An habitual activity such as smoking, alcoholism, etc.",
"related_attributes": [
{"name": "value", "description": "Name of an activity.", "examples": ["smoking", "marijuana"]},
{"name": "starting", "description": "When did the patient pick up the activities?", "examples": ["ten years back", "as a child"]},
{"name": "frequency", "description": "How frequently does the patient engage in the selected activity?", "examples": ["on weekends", "every day"]}
]
}, {
"slot": "exposure",
"description": "An environmental/chemical factor such as asbestos, pets, etc.",
"related_attributes": [
{"name": "value", "description": "Name of an environmental factor.", "examples": ["pets", "dust"]},
{"name": "where", "description": "Where was the patient exposed to the selected factor?", "examples": ["work", "home"]},
{"name": "when", "description": "When was the patient exposed to the selected factor?", "examples": ["four days ago"]}
]
}, {
"slot": "medication",
"description": "A medication.",
"related_attributes": [
{"name": "value", "description": "Name of a medication.", "examples": ["over-the-counter medicine", "paracetamol"]},
{"name": "start", "description": "Since when did the patient start taking the medication?", "examples": ["few weeks ago", "two days back"]},
{"name": "impact", "description": "Did the medication help the patient (Yes/No/Maybe)?"},
{"name": "respone_to", "description": "For which condition/symptom is medication for?", "examples": ["hypertensive disease", "diabetes"]},
{"name": "frequency", "description": "How frequently does the patient take the medication?", "examples": ["daily"]}
]
}, {
"slot": "medical_test",
"description": "A medical test.",
"related_attributes": [
{"name": "value", "description": "Name of a medical test.", "examples": ["chest X-ray", "electrocardiogram"]},
{"name": "when", "description": "When did the patient had the medical test done?", "examples": ["yesterday", "a week ago"]}
]
}, {
"slot": "residence",
"description": "Information regarding patient's living conditions.",
"related_attributes": [
{"name": "value", "description": "Place where the patient resides.", "examples": ["apartment", "old building"]},
{"name": "household_size", "description": "Size of the patient's household.", "examples": ["2", "4"]}
]
}, {
"slot": "occupation",
"description": "Information regarding the patient's occupation.",
"related_attributes": [
{"name": "value", "description": "Job/occupation of the patient.", "examples": ["nurse", "student"]},
{"name": "exposure", "description": "Are there any hazards/substances/dangers to which the patient got exposed at work?", "examples": ["chemical fumes", "dust"]}
]
}, {
"slot": "travel",
"description": "Information regarding the patient's recent travels.",
"related_attributes": [
{"name": "destination", "description": "Where has the patient travelled to?", "examples": ["canada", "united states"]},
{"name": "date", "description": "When did the patient travel?", "examples": ["last week", "a year ago"]}
]
}, {
"slot": "basic_information",
"description": "Basic information about the patient.",
"related_attributes": [
{"name": "age", "description": "Age of the patient.", "examples": ["32", "50"]},
{"name": "gender", "description": "Gender of the patient.", "examples": ["male", "female"]},
{"name": "name", "description": "Name of the patient.", "examples": ["John", "Lily"]}
]
}, {
"slot": "disease",
"description": "A medical condition.",
"related_attributes": [
{"name": "value", "description": "Name of the medical condition", "examples": ["viral pneumonia", "common cold"]}
]
}]
```
IMPORTANT INSTRUCTIONS:
1. Read the given definitions carefully.
2. For a given dialogue state and last turn, only some of the intents, slots and related attributes are applicable.
3. Related attribute 'value' of the the slots symptom, medical_history, family_history, habit, exposure, medication and medical_test must be a standard medical concept.
4. Make sure that the doctor's action is a continuation of the dialogue. dialogue state is given as an additional context.
[dialogue state]
```
{
"positive_symptom": [
{
"value": "pharyngitis",
"onset": "past four days"
},
{
"value": "fever",
"onset": "last two days",
"positive_characteristics": [
"during the day"
]
},
{
"value": "deglutition disorders"
},
{
"value": "erythema",
"location": [
"pharyngeal structure"
]
},
{
"value": "body substance discharge",
"location": [
"pharyngeal structure"
],
"color": "whitish"
},
{
"value": "swelling",
"location": [
"left lateral part of neck",
"neck",
"right lateral part of neck"
]
},
{
"value": "lymphadenopathy",
"location": [
"left lateral part of neck",
"neck",
"right lateral part of neck"
]
},
{
"value": "chills"
}
],
"negative_symptom": [
{
"value": "dysphonia"
},
{
"value": "night sweats"
},
{
"value": "headache"
}
],
"avail_medical_test": [
{
"value": "body temperature measurement",
"when": "last night"
}
]
}
```
[last turn]
doctor: OK. Uhm, and have you had any headaches?
patient: No headaches.
[output]
[{"action": "inquire", "symptom": [{"value": "redness of eye"}]}, {"action": "inquire", "symptom": [{"value": "body substance discharge", "checks": [{"type": "location", "values": ["eye"]}]}]}]
.
.
{{Remaining Exemplars}}
.
.
[dialogue state]
```
{
"positive_symptom": [
{
"value": "pharyngitis",
"onset": "last week"
},
{
"value": "chills",
"onset": "last few nights"
},
{
"value": "bedridden",
"onset": "a week ago"
},
{
"value": "deglutition disorders"
}
],
"negative_symptom": [
{
"value": "headache"
}
]
}
```
[last turn]
doctor: And um, have you had, uh, have you had any headaches?
patient: Uh, no headaches.
[output]
\end{Verbatim}
\end{tcolorbox}
\subsection{NLG Prompt}\label{app:nlg_prompt}
\begin{tcolorbox}[
enhanced, breakable, colback=white,
colframe=black, boxrule=0.25mm,
width=\textwidth
% before=\vspace{1ex},
% after=\vspace{1ex},
]
\fontsize{6pt}{6pt}\selectfont
You are a professional medical assistant who is an expert in understanding doctor-patient dialogues. The user will show you the last turn of the dialogue between a doctor and a patient and the doctor's action. Your task is to suggest the doctor's response as a continuation of the dialogue.
\begin{Verbatim}[breaklines]
IMPORTANT INSTRUCTIONS:
1. Your suggested response must reflect the doctor's actions and form a natural continuation of the dialogue.
2. Your suggested response must be fluent, grammatically correct and empathetic.
3. Your suggested response must satisfy any queries made by the user.
[actions]
```
[
{
"action": "inquire",
"symptom": [
{
"value": "redness of eye"
}
]
},
{
"action": "inquire",
"symptom": [
{
"value": "body substance discharge",
"checks": [
{
"type": "location",
"values": [
"eye"
]
}
]
}
]
}
]
```
[last turn]
doctor: OK. Uhm, and have you had any headaches?
patient: No headaches.
[output]
Have you had any eye redness or eye discharge?
.
.
{{Remaining Exemplars}}
.
.
[actions]
```
[
{
"action": "inquire",
"symptom": [
{
"value": "nasal congestion"
},
{
"value": "rhinorrhea"
}
]
}
]
```
[last turn]
doctor: And um, have you had, uh, have you had any headaches?
patient: Uh, no headaches.
[output]
\end{Verbatim}
\end{tcolorbox}
\twocolumn
\end{document}
\begin{thebibliography}{46}
\providecommand{\natexlab}[1]{#1}
\bibitem[{Abacha et~al.(2023)Abacha, Yim, Fan, and Lin}]{abacha2023empirical}
Asma~Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. 2023.
\newblock An empirical study of clinical note generation from doctor-patient
encounters.
\newblock In \emph{Proceedings of the 17th Conference of the European Chapter
of the Association for Computational Linguistics}, pages 2291--2302.
\bibitem[{AI@Meta(2024)}]{llama3modelcard}
AI@Meta. 2024.
\newblock \href {https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}
{Llama 3 model card}.
\bibitem[{Ankit~Pal(2024)}]{OpenBioLLMs}
Malaikannan~Sankarasubbu Ankit~Pal. 2024.
\newblock Openbiollms: Advancing open-source large language models for
healthcare and life sciences.
\newblock \url{https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B}.
\bibitem[{Bang et~al.(2023)Bang, Lee, and Koo}]{bang2023task}
Namo Bang, Jeehyun Lee, and Myoung-Wan Koo. 2023.
\newblock Task-optimized adapters for an end-to-end task-oriented dialogue
system.
\newblock In \emph{Findings of the Association for Computational Linguistics:
ACL 2023}, pages 7355--7369.
\bibitem[{Budzianowski et~al.(2018)Budzianowski, Wen, Tseng, Casanueva, Ultes,
Ramadan, and Gasic}]{budzianowski2018multiwoz}
Pawe{\l} Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I{\~n}igo Casanueva,
Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018.
\newblock Multiwoz-a large-scale multi-domain wizard-of-oz dataset for
task-oriented dialogue modelling.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 5016--5026.
\bibitem[{Byrne et~al.(2019)Byrne, Krishnamoorthi, Sankar, Neelakantan,
Goodrich, Duckworth, Yavuz, Dubey, Kim, and Cedilnik}]{byrne2019taskmaster}
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan,
Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and
Andy Cedilnik. 2019.
\newblock Taskmaster-1: Toward a realistic and diverse dialog dataset.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 4516--4525.
\bibitem[{Cao et~al.()Cao, Gupta, Zhao, Yu, Wang, Lee, Rastogi, Shafran, and
Wu}]{zhao2022description}
Yuan Cao, Raghav Gupta, Jeffrey Zhao, Dian Yu, Mingqiu Wang, Harrison Lee,
Abhinav~Kumar Rastogi, Izhak Shafran, and Yonghui Wu.
\newblock Description-driven task-oriented dialog modeling.
\bibitem[{Chen et~al.(2022)Chen, Li, Fang, Yao, Zhong, Hao, Zhang, Huang, Peng,
and Wei}]{Chen2022ABF}
W.~Chen, Zhiwei Li, Hongyi Fang, Qian-Qian Yao, Cheng Zhong, Jianye Hao,
Qi~Zhang, Xuanjing Huang, Jianjun Peng, and Zhongyu Wei. 2022.
\newblock \href {https://api.semanticscholar.org/CorpusID:248239674} {A
benchmark for automatic medical consultation system: frameworks, tasks and
datasets}.
\newblock \emph{Bioinformatics}, 39.
\bibitem[{Chintagunta et~al.(2021)Chintagunta, Katariya, Amatriain, and
Kannan}]{chintagunta2021medically}
Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2021.
\newblock Medically aware gpt-3 as a data generator for medical dialogue
summarization.
\newblock In \emph{Machine Learning for Healthcare Conference}, pages 354--372.
PMLR.
\bibitem[{Chung et~al.(2024)Chung, Hou, Longpre, Zoph, Tay, Fedus, Li, Wang,
Dehghani, Brahma et~al.}]{chung2024scaling}
Hyung~Won Chung, Le~Hou, Shayne Longpre, Barret Zoph, Yi~Tay, William Fedus,
Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et~al. 2024.
\newblock Scaling instruction-finetuned language models.
\newblock \emph{Journal of Machine Learning Research}, 25(70):1--53.
\bibitem[{Dowlagar and Mamidi(2023)}]{dowlagar2023code}
Suman Dowlagar and Radhika Mamidi. 2023.
\newblock A code-mixed task-oriented dialog dataset for medical domain.
\newblock \emph{Computer Speech \& Language}, 78:101449.
\bibitem[{Du et~al.(2019)Du, Chen, Kannan, Tran, Chen, and
Shafran}]{Du2019ExtractingSA}
Nan Du, Kai Chen, Anjuli Kannan, Linh Tran, Yuhui Chen, and Izhak Shafran.
2019.
\newblock Extracting symptoms and their status from clinical conversations.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 915--925.
\bibitem[{El~Asri et~al.(2017)El~Asri, Schulz, Sarma, Zumer, Harris, Fine,
Mehrotra, and Suleman}]{asri2017frames}
Layla El~Asri, Hannes Schulz, Shikhar~Kr Sarma, Jeremie Zumer, Justin Harris,
Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017.
\newblock Frames: a corpus for adding memory to goal-oriented dialogue systems.
\newblock In \emph{Proceedings of the 18th Annual SIGdial Meeting on Discourse
and Dialogue}, pages 207--219.
\bibitem[{Fansi~Tchango et~al.(2022)Fansi~Tchango, Goel, Martel, Wen,
Marceau~Caron, and Ghosn}]{fansi2022towards}
Arsene Fansi~Tchango, Rishab Goel, Julien Martel, Zhi Wen, Gaetan
Marceau~Caron, and Joumana Ghosn. 2022.
\newblock Towards trustworthy automatic diagnosis systems by emulating doctors'
reasoning with deep reinforcement learning.
\newblock \emph{Advances in Neural Information Processing Systems},
35:24502--24515.
\bibitem[{Fareez et~al.(2022)Fareez, Parikh, Wavell, Shahab, Chevalier, Good,
De~Blasi, Rhouma, McMahon, Lam et~al.}]{fareez2022dataset}
Faiha Fareez, Tishya Parikh, Christopher Wavell, Saba Shahab, Meghan Chevalier,
Scott Good, Isabella De~Blasi, Rafik Rhouma, Christopher McMahon, Jean-Paul
Lam, et~al. 2022.
\newblock A dataset of simulated patient-physician medical interviews with a
focus on respiratory cases.
\newblock \emph{Scientific Data}, 9(1):313.
\bibitem[{Gleeson(1979)}]{harden1979assessment}
FA~Gleeson. 1979.
\newblock Assessment of clinical competence using an objective structured
clinical examination (osce).
\newblock \emph{Medical education}, 13(1):39--54.
\bibitem[{He et~al.(2020)He, Chen, Ju, Dong, Fang, Wang, Yang, Zeng, Zhang,
Zhang, Zhou, Zhu, and Xie}]{He2020MedDialogTL}
Xuehai He, Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue
Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, Meng Zhou, Penghui Zhu, and
Pengtao Xie. 2020.
\newblock \href {https://api.semanticscholar.org/CorpusID:220403300}
{Meddialog: Two large-scale medical dialogue datasets}.
\newblock \emph{arXiv: Learning}.
\bibitem[{Hu et~al.(2021)Hu, Wallis, Allen-Zhu, Li, Wang, Wang, Chen
et~al.}]{hu2021lora}
Edward~J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang,
Weizhu Chen, et~al. 2021.
\newblock Lora: Low-rank adaptation of large language models.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Joshi et~al.(2020)Joshi, Katariya, Amatriain, and
Kannan}]{joshi2020dr}
Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020.
\newblock Dr. summarize: Global summarization of medical dialogue by exploiting
local structures.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2020}, pages 3755--3763.
\bibitem[{Kearns et~al.(2019)Kearns, Chi, Choi, Lin, Thompson, and
Demiris}]{kearns2019systematic}
William~R Kearns, Nai-Ching Chi, Yong~K Choi, Shih-Yin Lin, Hilaire Thompson,
and George Demiris. 2019.
\newblock A systematic review of health dialog systems.
\newblock \emph{Methods of information in medicine}, 58(06):179--193.
\bibitem[{Krishna et~al.(2021)Krishna, Khosla, Bigham, and
Lipton}]{krishna2021generating}
Kundan Krishna, Sopan Khosla, Jeffrey~P Bigham, and Zachary~C Lipton. 2021.
\newblock Generating soap notes from doctor-patient conversations using modular
summarization techniques.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 4958--4972.
\bibitem[{Laranjo et~al.(2018)Laranjo, Dunn, Tong, Kocaballi, Chen, Bashir,
Surian, Gallego, Magrabi, Lau et~al.}]{laranjo2018conversational}
Liliana Laranjo, Adam~G Dunn, Huong~Ly Tong, Ahmet~Baki Kocaballi, Jessica
Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie~YS Lau,
et~al. 2018.
\newblock Conversational agents in healthcare: a systematic review.
\newblock \emph{Journal of the American Medical Informatics Association},
25(9):1248--1258.
\bibitem[{Lee et~al.(2021)Lee, Cheng, and Ostendorf}]{lee2021dialogue}
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021.
\newblock Dialogue state tracking with a language model using schema-driven
prompting.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 4937--4949.
\bibitem[{Lin(2004)}]{lin-2004-rouge}
Chin-Yew Lin. 2004.
\newblock \href {https://aclanthology.org/W04-1013} {{ROUGE}: A package for
automatic evaluation of summaries}.
\newblock In \emph{Text Summarization Branches Out}, pages 74--81, Barcelona,
Spain. Association for Computational Linguistics.
\bibitem[{Lin et~al.(2019)Lin, He, Chen, Tou, Wei, and
Chen}]{Lin2019EnhancingDS}
Xinzhu Lin, Xiahui He, Qin Chen, Huaixiao Tou, Zhongyu Wei, and Ting Chen.
2019.
\newblock \href {https://api.semanticscholar.org/CorpusID:202771062} {Enhancing
dialogue symptom diagnosis with global attention and symptom graph}.
\newblock In \emph{Conference on Empirical Methods in Natural Language
Processing}.
\bibitem[{Liu et~al.(2022)Liu, Shen, Zhang, Dolan, Carin, and
Chen}]{liu-etal-2022-makes}
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu
Chen. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.deelio-1.10} {What makes good
in-context examples for {GPT}-3?}
\newblock In \emph{Proceedings of Deep Learning Inside Out (DeeLIO 2022): The
3rd Workshop on Knowledge Extraction and Integration for Deep Learning
Architectures}, pages 100--114, Dublin, Ireland and Online. Association for
Computational Linguistics.
\bibitem[{Luo et~al.(2022)Luo, Sun, Xia, Qin, Zhang, Poon, and
Liu}]{luo2022biogpt}
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and
Tie-Yan Liu. 2022.
\newblock Biogpt: generative pre-trained transformer for biomedical text
generation and mining.
\newblock \emph{Briefings in bioinformatics}, 23(6):bbac409.
\bibitem[{OpenAI(2024{\natexlab{a}})}]{openai2024gpt35turbo}
OpenAI. 2024{\natexlab{a}}.
\newblock Gpt-3.5-turbo.
\newblock Available from \url{https://platform.openai.com/docs/models}.
\bibitem[{OpenAI(2024{\natexlab{b}})}]{openai2023gpt4preview}
OpenAI. 2024{\natexlab{b}}.
\newblock Gpt-4-1106-preview.
\newblock Available from \url{https://platform.openai.com/docs/models}.
\bibitem[{Papineni et~al.(2002)Papineni, Roukos, Ward, and
Zhu}]{papineni2002bleu}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.
\newblock Bleu: a method for automatic evaluation of machine translation.
\newblock In \emph{Proceedings of the 40th annual meeting of the Association
for Computational Linguistics}, pages 311--318.
\bibitem[{Rastogi et~al.(2020)Rastogi, Zang, Sunkara, Gupta, and
Khaitan}]{rastogi2020towards}
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav
Khaitan. 2020.
\newblock Towards scalable multi-domain conversational agents: The
schema-guided dialogue dataset.
\newblock In \emph{Proceedings of the AAAI conference on artificial
intelligence}, volume~34, pages 8689--8696.
\bibitem[{Rojowiec et~al.(2020)Rojowiec, Roth, and Fink}]{rojowiec2020intent}
Robin Rojowiec, Benjamin Roth, and Maximilian Fink. 2020.
\newblock Intent recognition in doctor-patient interviews.
\newblock In \emph{Proceedings of the Twelfth Language Resources and Evaluation
Conference}, pages 702--709.
\bibitem[{Shi et~al.(2020)Shi, Hu, Che, Sun, Liu, and
Huang}]{Shi2020UnderstandingMC}
Xiaoming Shi, Haifeng Hu, Wanxiang Che, Zhongqian Sun, Ting Liu, and Junzhou
Huang. 2020.
\newblock \href {https://api.semanticscholar.org/CorpusID:213894093}
{Understanding medical conversations with scattered keyword attention and
weak supervision from responses}.
\newblock In \emph{AAAI Conference on Artificial Intelligence}.
\bibitem[{Su et~al.(2022)Su, Shu, Mansimov, Gupta, Cai, Lai, and
Zhang}]{su2021multi}
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and
Yi~Zhang. 2022.
\newblock Multi-task pre-training for plug-and-play task-oriented dialogue
system.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 4661--4676.
\bibitem[{Sun et~al.(2023)Sun, Bao, Wu, and He}]{sun2022mars}
Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2023.
\newblock Mars: Modeling context \& state representations with contrastive
learning for end-to-end task-oriented dialog.
\newblock In \emph{Findings of the Association for Computational Linguistics:
ACL 2023}, pages 11139--11160.
\bibitem[{Tchango et~al.(2022)Tchango, Goel, Wen, Martel, and
Ghosn}]{Tchango2022DDXPlusAN}
Ars{\`e}ne~Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana
Ghosn. 2022.
\newblock \href {https://api.semanticscholar.org/CorpusID:249707831} {Ddxplus:
A new dataset for automatic medical diagnosis}.
\newblock In \emph{Neural Information Processing Systems}.
\bibitem[{Valizadeh and Parde(2022)}]{valizadeh2022ai}
Mina Valizadeh and Natalie Parde. 2022.
\newblock The ai doctor is in: A survey of task-oriented dialogue systems for
healthcare applications.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 6638--6660.
\bibitem[{Wang et~al.(2023)Wang, Yao, Yang, Zhou, Li, Wang, Xu, and
Yu}]{wang2023notechat}
Junda Wang, Zonghai Yao, Zhichao Yang, Huixue Zhou, Rumeng Li, Xun Wang,
Yucheng Xu, and Hong Yu. 2023.
\newblock Notechat: A dataset of synthetic doctor-patient conversations
conditioned on clinical notes.
\newblock \emph{arXiv preprint arXiv:2310.15959}.
\bibitem[{Wei et~al.(2018)Wei, Liu, Peng, Tou, Chen, Huang, Wong, and
Dai}]{Wei2018TaskorientedDS}
Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing
Huang, Kam-Fai Wong, and Xiangying Dai. 2018.
\newblock \href {https://api.semanticscholar.org/CorpusID:51875856}
{Task-oriented dialogue system for automatic diagnosis}.
\newblock In \emph{Annual Meeting of the Association for Computational
Linguistics}.
\bibitem[{Wen et~al.(2017)Wen, Vandyke, Mrk{\v{s}}i{\'c}, Gasic, Barahona, Su,
Ultes, and Young}]{wen2016network}
Tsung-Hsien Wen, David Vandyke, Nikola Mrk{\v{s}}i{\'c}, Milica Gasic, Lina
M~Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017.
\newblock A network-based end-to-end trainable task-oriented dialogue system.
\newblock In \emph{Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 1, Long Papers},
pages 438--449.
\bibitem[{Wu et~al.(2023)Wu, Gung, Shu, and Zhang}]{wu2023diacttod}
Qingyang Wu, James Gung, Raphael Shu, and Yi~Zhang. 2023.
\newblock Diacttod: Learning generalizable latent dialogue acts for
controllable task-oriented dialogue systems.
\newblock In \emph{Proceedings of the 24th Annual Meeting of the Special
Interest Group on Discourse and Dialogue}, pages 255--267.
\bibitem[{Xiao et~al.(2023)Xiao, Liu, Zhang, and Muennighoff}]{bge_embedding}
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.
\newblock \href {https://arxiv.org/abs/2309.07597} {C-pack: Packaged resources
to advance general chinese embedding}.
\newblock \emph{Preprint}, arXiv:2309.07597.
\bibitem[{Yan et~al.(2021)Yan, Pei, Ren, Ren, Xin, Liang, de~Rijke, and
Chen}]{Yan2021ReMeDiRF}
Guojun Yan, Jiahuan Pei, Pengjie Ren, Zhaochun Ren, Xin Xin, Huasheng Liang,
M.~de~Rijke, and Zhumin Chen. 2021.
\newblock \href {https://api.semanticscholar.org/CorpusID:247188125} {Remedi:
Resources for multi-domain, multi-service, medical dialogues}.
\newblock \emph{Proceedings of the 45th International ACM SIGIR Conference on
Research and Development in Information Retrieval}.
\bibitem[{Young et~al.(2013)Young, Ga{\v{s}}i{\'c}, Thomson, and
Williams}]{young2013pomdp}
Steve Young, Milica Ga{\v{s}}i{\'c}, Blaise Thomson, and Jason~D Williams.
2013.
\newblock Pomdp-based statistical spoken dialog systems: A review.
\newblock \emph{Proceedings of the IEEE}, 101(5):1160--1179.
\bibitem[{Zhang et~al.()Zhang, Kishore, Wu, Weinberger, and
Artzi}]{zhang2019bertscore}
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian~Q Weinberger, and Yoav Artzi.
\newblock Bertscore: Evaluating text generation with bert.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Zhang et~al.(2020)Zhang, Jiang, Zhang, Liu, Cao, Liu, Liu, and
Zhao}]{Zhang2020MIEAM}
Yuanzhe Zhang, Zhongtao Jiang, Tao Zhang, Shiwan Liu, Jiarun Cao, Kang Liu,
Shengping Liu, and Jun Zhao. 2020.
\newblock \href {https://api.semanticscholar.org/CorpusID:220047186} {Mie: A
medical information extractor towards medical dialogues}.
\newblock In \emph{Annual Meeting of the Association for Computational
Linguistics}.
\end{thebibliography}