How well do SOTA legal reasoning models support abductive reasoning?
Extracted source text · arXiv 2304.06912
%% The first command in your LaTeX source must be the \documentclass command.
%%
%% Options:
%% twocolumn : Two column layout.
%% hf: enable header and footer.
\documentclass[
% twocolumn,
% hf,
]{ceurart}
%%
%% One can fix some overfulls
\sloppy
%%
%% Minted listings support
%% Need pygment <http://pygments.org/> <http://pypi.python.org/pypi/Pygments>
\usepackage{listings}
%% auto break lines
\lstset{breaklines=true}
\newcommand{\etal}{\textit{et al}. }
\newcommand{\art}{{\sc ART}}
\newcommand{\lart}{{\sc L'ART}}
\newcommand{\ie}{\textit{i}.\textit{e}., }
\newcommand{\eg}{\textit{e}.\textit{g}. }
%%
%% end of the preamble, start of the body of the document source.
\begin{document}
%%
%% Rights management information.
%% CC-BY is default license.
\copyrightyear{2023}
\copyrightclause{Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).}
%%
%% This command is for the conference information
\conference{London'23: Workshop on Logic Programming and Legal Reasoning,
July 09--10, 2023, London, UK}
%%
%% The "title" command
\title{How well do SOTA legal reasoning models support abductive reasoning?}
\tnotemark[1]
% \tnotetext[1]{You can use this document as the template for preparing your
% publication. We recommend using the latest version of the ceurart style.}
%%
%% The "author" command and its associated commands are used to define
%% the authors and their affiliations.
\author[1]{Ha Thanh Nguyen}[%
orcid=0000-0003-2794-7010,
email=nguyenhathanh@nii.ac.jp
]
\cormark[1]
\address[1]{National Institute of Informatics (NII),
2-1-2 Hitotsubashi, Chiyoda City, Tokyo, Japan}
\author[2]{Randy Goebel}[%
orcid=0000-0002-0739-2946,
email=rgoebel@ualberta.ca
]
\address[2]{University of Alberta, 116 St \& 85 Ave, Edmonton, AB T6G 2R3, Canada}
\author[3]{Francesca Toni}[%
orcid=0000-0001-8194-1459,
email=f.toni@imperial.ac.uk,
]
\address[3]{Imperial College London, Exhibition Rd, South Kensington, London SW7 2BX, United Kingdom}
\author[4]{Kostas Stathis}[%
orcid=0000-0002-9946-4037,
email=Kostas.Stathis@rhul.ac.uk,
]
\address[4]{Royal Holloway University of London, Egham Hill, Egham TW20 0EX, United Kingdom}
\author[1]{Ken Satoh}[%
orcid=0000-0002-9309-4602,
email=ksatoh@nii.ac.jp,
]
%% Footnotes
\cortext[1]{Corresponding author.}
% \fntext[1]{These authors contributed equally.}
%%
%% The abstract is a short summary of the work to be presented in the
%% article.
\begin{abstract}
We examine how well the state-of-the-art (SOTA) models used in legal reasoning support abductive reasoning tasks. Abductive reasoning is a form of logical inference in which a hypothesis is formulated from a set of observations, and that hypothesis is used to explain the observations. The ability to formulate such hypotheses is important for lawyers and legal scholars as it helps them articulate logical arguments, interpret laws, and develop legal theories. Our motivation is to consider the belief that deep learning models, especially large language models (LLMs), will soon replace lawyers because they perform well on tasks related to legal text processing. But to do so, we believe, requires some form of abductive hypothesis formation. In other words, while LLMs become more popular and powerful, we want to investigate their capacity for abductive reasoning. To pursue this goal, we start by building a logic-augmented dataset for abductive reasoning with 498,697 samples and then use it to evaluate the performance of a SOTA model in the legal field. Our experimental results show that although these models can perform well on tasks related to some aspects of legal text processing, they still fall short in supporting abductive reasoning tasks.
\end{abstract}
%%
%% Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
\begin{keywords}
neural networks \sep
abductive reasoning \sep
legal reasoning
\end{keywords}
%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle
\section{Introduction}
\label{sec:intro}
The rise of transformer-based deep learning models \cite{vaswani2017attention} has brought remarkable advancements in natural language processing (NLP), including tasks related to legal text processing \cite{ijcai2020p484,chalkidis2020legal,nguyen2020jnlp,tran2020encoded,yoshioka2021bert,nguyen2022attentive}. These advancements have the potential to improve access to justice and legal services for underserved communities, and to enhance the efficiency and accuracy of judicial processes.
\begin{figure*}
\center
\includegraphics[width=.6\textwidth]{COLIEE_example.png}
\caption{An example of statute law retrieval task requiring abductive reasoning skill in COLIEE Competition~\cite{Rabelo_2022}. The truth value (No) of the statement is determined by assuming that the statement is either true or false, and then searching for the appropriate legal article that provides the best explanation for that assumption. In this case, the relevant article is Article 192, which provides an exception to A's ability to demand the return of the bicycle against B through an action for recovery of possession. Abductive reasoning is utilized in determining that the statement's value is "No" based on the application of Article 192 and the exception it presents.}\label{fig:example}
\end{figure*}
However, a critical component of legal intelligence is abductive reasoning, which is of paramount importance for lawyers and legal scholars in formulating logical arguments, interpreting laws, and developing legal theories \cite{abimbola2000abductive,schum2002species}.
Figure \ref{fig:example} is an example of statute law retrieval task requiring abductive reasoning skill in the existing COLIEE Competition~\cite{Rabelo_2022}.
Since transformer-based models have shown promising results in various NLP tasks within the legal field, it is essential to evaluate their performance on tasks involving abductive reasoning to better understand their capabilities and limitations as AI tools in the legal field.
Existing literature has explored the development and evaluation of transformer-based models for legal text processing tasks such as legal document retrieval, summarization, entailment, and question-answering. Yet, there is a gap in evaluating these models for the crucial task of abductive reasoning. To address this gap, we first articulate an empirical method to reveal the role of abductive reasoning in legal information processing. We identify an appropriate corpus of examples by examining the \art{} dataset~\cite{bhagavatula2020abductive}, which is the first large dataset for abductive reasoning tasks, and ${\alpha}NLI$ task, which investigates the viability of language-based abductive reasoning.
In this paper, we focus on:
\begin{itemize}
\item Enhancing the reliability of the dataset for abductive reasoning through task redefinition and data augmentation.
\item Evaluating the performance of a state-of-the-art (SOTA) transformer-based model in the legal field on abductive reasoning tasks.
\end{itemize}
Our experimental results show that although the selected SOTA model can perform well on tasks related to legal text processing, it still falls short in supporting abductive reasoning tasks, shedding light on an important limitation of these models in legal reasoning. This study provides a more comprehensive understanding of the capabilities of transformer-based models in the legal domain, emphasizing the importance of abductive reasoning, which is often overlooked in related research.
\section{Background}
\label{sec:background}
\subsection{Legal Text Processing}
% \todo{some references from here need to go }
There has been significant research on the use of artificial intelligence and machine learning (AI) in the legal field in recent years. This research has resulted in the development of a number of state-of-the-art models that are able to perform various tasks related to legal text processing, such as contract risk analysis and case law retrieval.
Deep learning models have the ability of automated latent feature extraction,
%***************** [I am worried about this claim; all deep learning is supervised learning, as far as I know, and so features are predefined? Do you mean something else specific to the way in which deep learning is employed to build transformers?] ******************
%FT: I have added latent
%Thanh: Yes, "latent feature" is correct, we only provide label and they extract latent feature automatically
which allows us to use these models not only for similarity matching tasks but also in other semantic matching tasks such as question answering ~\cite{kien-etal-2020-answering,asai-etal-2021-xor}, machine reading comprehension \cite{nie-etal-2019-revealing,lee-yeung-2021-text}, image retrieval \cite{krojer-etal-2022-image} and entity matching \cite{lin-etal-2020-triggerner,ritter-etal-2011-named}.
These tasks are all important for the general challenge of legal reasoning.
In the legal retrieval task, legal documents are usually structured and existing systems are designed to retrieve relevant legal texts (e.g., regulations) based on a given query. This task is an essential component of intelligent legal counsel systems and commonly appears in legal automated processing competitions \cite{Rabelo_2020,Rabelo_2022,Thanh_2021,Nguyen_2022}.
One of the challenges in legal information retrieval is that the available data is usually very limited.
This is why the current best systems often need to be based on some supportive rules or data augmentation methods.
As a consequence, Deep learning with transfer learning methods has been successfully applied to this problem in a number of ways \cite{kien-etal-2020-answering,ijcai2020p484,Nguyen_transformer_2022,louis-spanakis-2022-statutory,Vuong_2022}.
Retrieval tasks are foundational to many other legal text processing tasks. For example, contract analysis often involves retrieving relevant provisions or clauses from a contract based on a given query. Similarly, case law retrieval involves identifying and retrieving relevant case law and legal precedent based on a given query. These tasks require making inferences and educated guesses about what information is likely to be relevant, based on the characteristics of the query and the available data.
In addition, many other legal text-processing tasks, such as legal document classification and summarization, are also related to retrieval. For instance, classifying a legal document as relevant or irrelevant to a given case may involve retrieving similar documents and using them as a reference. Similarly, summarizing a legal document may involve retrieving relevant information and condensing it to a shorter form.
Overall, retrieval tasks form the foundation for many legal text processing tasks, and the ability to perform retrieval effectively is essential for the development of intelligent legal counsel systems and other AI tools for the legal field.
Note that retrieval tasks and abductive reasoning are closely related. In retrieval tasks, the goal is to identify and retrieve relevant information from a database or collection of documents based on a given query. This requires making an educated guess or inference about what information is likely to be relevant, based on the characteristics of the query and the available data.
Similarly, in abductive reasoning, the goal is to construct an argument or form a hypothesis based on a set of observations and a limited amount of information. This also requires making an educated guess or inference based on the available evidence, in order to explain the observations and arrive at a plausible conclusion.
% \subsection{Transformer-based Models in Legal Processing}
% % \todo{this section is more background than related work}
% In the legal domain, transformer-based models have been used to tackle a variety of tasks, such as legal document classification, legal text summarization, and legal question answering. These models have proven to be particularly effective at capturing the complex relationships and dependencies between words and phrases in legal texts, and have demonstrated strong performance on a wide range of legal text processing tasks. In addition, because a majority of errors in COLIEE, a well-known legal reasoning competition
% % \todo{which one?}
% % Thanh: added COLIEE
% are based on paraphrase errors, transformers have also been useful in identifying measures of similarity amongst allegedly similar texts \cite{Rabelo_2020,Rabelo_2022%,rabelo
% }.
% Some examples of transformer-based models that have been developed specifically for legal text processing include BERT-PLI~\cite{ijcai2020p484}, Legal BERT~\cite{chalkidis2020legal}, BERTLaw \cite{nguyen2020jnlp}, and ParaLaw Nets~\cite{nguyen2021jnlp}. These models have achieved impressive results on tasks such as legal document classification and legal question answering, and are widely used by researchers and practitioners in the legal field.
% Overall, transformer-based models currently play an important role in the field of legal text processing and are likely to continue to be a crucial part of the landscape in the coming years. But in our analysis, it is clear that such transformer models are important basic tools for a variety of text problems, their weaknesses emerge on tasks which require reasoning about text. One such task is abductive reasoning.
% At a high level, transformer models consist of an encoder and a decoder, which are connected by an attention mechanism. The encoder takes in a sequence of input tokens (such as words or characters) and produces a sequence of hidden states, which capture relationships amongst the input sequence. The decoder takes in a sequence of target tokens (such as words or characters) and generates a sequence of output tokens, which are intended to be a transformation or summary of the input sequence. Note that the basic decoder process relies on the identified relationships amongst text objects created by the encoding task, which are not based on inference about relationships amongst text components.
% The attention mechanism allows the decoder to focus on different parts of the input sequence at different times, weighting the importance of each part based on the current context. This allows the model to selectively focus on relevant information and avoid the need to process the entire input sequence at once.
% Mathematically, the attention mechanism is implemented using a dot-product attention function, which computes the attention weights between the hidden states of the encoder and the current state of the decoder. This is done using the following formula:
% $$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$
% \noindent where $Q$, $K$, and $V$ are the query, key, and value matrices (respectively) for the attention mechanism, and $d_k$ is the dimensionality of the keys. The attention weights are then used to calculate a weighted sum of the value matrix, which is used to update the current state of the decoder.
\subsection{Abductive Reasoning}
\label{sec:ar}
% \todo{this section seems more to be about setting the ground than related work}
Our approach to test deep learning models based on transformers \cite{vaswani2017attention} for reasoning is focused on abductive reasoning tasks, because abductive reasoning is such an important aspect of human reasoning~\cite{kakas1992abductive}.
In a typical abductive reasoning problem, one is given a set of observations, and the goal is to identify a hypothesis that can best explain the observations.
This process can be divided into two steps: 1) potential hypothesis identification, and 2) hypothesis evaluation. In the first step, we need to identify a small set of potential hypotheses that are likely to explain the observations. In the second step, we need to evaluate the potential hypotheses and rank them within the current context of use, e.g., assess a set of alternative hypotheses w.r.t. which provides the basis for a ``best'' explanation.
Because the complexity of the real world is high, it is impossible to consider all potential hypotheses. Instead, we typically assume that a set of potential hypotheses is given, and the evaluation of the hypotheses is based on logical reasoning.
This second step is similar to the scoring or ranking function used in the second step of an information retrieval process.
There are several ways to limit the number of potential hypotheses, for example, limiting the context, constraining the search space, or using a heuristic search method. Generally, only commonsense heuristic reasoning can help to limit the number of potential explanations. For example, suppose we want to explain why a health professional regulator received a complaint. In that case, we are only interested in explanations that are related to the health and care professionals' conduct or practice according to the relevant regulations \cite{lertvittayakumjorn2021supporting, jago2021use}. Therefore, to be useful in application, we need a mechanism to reduce the set of possible explanations.
Bhagavatula et al. \cite{bhagavatula2020abductive} introduce the \art{} dataset, which contains 20K commonsense narrative contexts and 200K explanations. They approach abductive reasoning as the problem of finding the hidden middle of a linear series of the form, $\mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2 $, where we observe $\mathcal{O_2}$, and $\mathcal{O_1}$, and we try to come up with the best hypothesis $\mathcal{H}$. In one example, they show two observations:
\begin{itemize}
\item \textbf{Observation 1:} Jenny cleaned her house and went to work, leaving the window just a crack open.
\item \textbf{Observation 2:} When Jenny returned home she saw that her house was a mess!
\end{itemize}
In ${\alpha}NLI$, the task of a model is to choose the most plausible explanatory hypothesis among the given candidates. For example:
\begin{itemize}
\item \textbf{Hypothesis 1:} A thief broke into the house by pulling open the window.
% \item \textbf{Hypothesis 2:} It was a breezy day and a large bird flew into the house.
\item \textbf{Hypothesis 2:} At work, she opened her window and the wind blew her papers everywhere.
\end{itemize}
Of the hypotheses above, if their common sense semantics are accepted, Hypothesis 1 is the most plausible for the two given observations.
However, if we consider the possible existence of enthymemes, the outcome may be different. For example, if ``the house'' in Hypothesis 1 does not refer to Jenny's house, then Hypothesis 1 becomes completely inappropriate as an explanation for the observations.
% This is not mentioned by the authors in their paper. Furthermore, in their experiments, the authors only test the ability of the model to choose the more plausible hypothesis from the given two.
% \todo{there are three in the example?}
% Thanh: There are three examples written in the paper of ART, however in the dataset, they only have a pair of hypotheses for the models to choose. I edited the example to match the actual setting in the dataset (only 2 hypotheses)
% This reduces the problem becomes to a binary classification problem, which is easier than the abductive reasoning problem in which the model should select the most plausible hypothesis from many possible hypotheses.
% Nevertheless, it is a good first approach to the problem and their dataset is valuable material for abductive reasoning.
%Newly added Feb 8, 2023
The existence of enthymemes affects the reliability of \art{} and ${\alpha}NLI$ for two reasons:
\begin{enumerate}
\item Dataset \art{} is constructed by crowd-sourcing with the fact that different people will have different sets of implicit arguments to support their different decisions.
% In Figure \ref{fig:example}, we assume that there are five crowdsourcing annotators that receive the same input data. Tables \ref{tab:res_ques1} and \ref{tab:res_ques2} provide an example where the data labels derived from these annotators change by changing the question.
\item As a consequence, in ${\alpha}NLI$ task, the model simply chooses a hypothesis which is more plausible than the other.
That is, the model always outputs a \textbf{Yes} label and a \textbf{No} label for a pair of inputs, it does not happen that both hypotheses are plausible (or both implausible). In addition, within this setting, annotators may introduce their bias into the training and evaluation of the model.
\end{enumerate}
% \begin{table*}
% \caption{Crowdsourcing weakness: Possible answers from the 5 annotators if they were asked to determine "Which hypothesis is \underline{more} plausible?". Personal experiences, beliefs, and background knowledge all play a role in determining which hypothesis they find more plausible.}\label{tab:res_ques1}
% \begin{tabular}{|c|c|p{8cm}|}
% \hline
% \textbf{Annotator} & \textbf{Chosen Hypothesis} & \textbf{Explanation for the decision} \\ \hline
% \#1 & 1 & Based on my personal experience, I think someone is in the house is more plausible because it's more common for people to turn on lights and play music when they're home. \\ \hline
% \#2 & 2 & I think the lights and music are on due to an automatic timer or a malfunction is more plausible because I experienced it once. \\ \hline
% \#3 & 1 & After considering both hypotheses, I think someone is in the house is more plausible because the observation of lights and music being on is more consistent with someone being present than with an automatic timer or a malfunction. \\ \hline
% \#4 & 1 & I think someone is in the house is more plausible because it seems like the simplest explanation for the observation. \\ \hline
% \#5 & 2 & The lights and music are on due to an automatic timer or a malfunction is more plausible because I have had similar experiences in my own home with automated systems. \\ \hline
% \end{tabular}
% \end{table*}
% \begin{table*}
% \caption{Crowdsourcing weakness: Possible answers from the 5 annotators if they were asked to determine "Which hypothesis is plausible?" The annotators may take a more balanced and cautious decision. If the two options are not mutually exclusive, this could become a majority.}\label{tab:res_ques2}
% \begin{tabular}{|c|c|p{8cm}|}
% \hline
% \textbf{Annotator} & \textbf{Chosen Hypothesis} & \textbf{Explanation for the decision} \\ \hline
% \#1 & 1,2 & Without further information, it's difficult to determine the exact cause of the lights and music being on. \\ \hline
% \#2 & 1,2 & I don't have enough information to determine which explanation is more likely. \\ \hline
% \#3 & 1,2 & While one explanation may seem more likely, it's always important to consider all possible explanations. \\ \hline
% \#4 & 1 & I think someone is in the house is more plausible because it seems like the simplest explanation for the observation. \\ \hline
% \#5 & 2 & The lights and music are on due to an automatic timer or a malfunction is more plausible because I have had similar experiences in my own home with automated systems. \\ \hline
% \end{tabular}
% \end{table*}
% Another useful source for building abductive reasoning datasets are deductive reasoning pairs.
% By applying logical transformations and datasets generated from logic programming, we can reduce the subjective factor from annotators and increase the quantity and quality of the samples.
% \todo{can we give a reference? an example?}
% Thanh: Yes, actually I mean "deductive" here
% For example, we can reverse the direction of the premises and conclusions in the deductive reasoning parts and use them as abductive reasoning samples.
% For example, with deductive reasoning sample, ``All men are mortal. Socrates is a man. Therefore, Socrates is mortal."
% We can reverse the direction of the premises and conclusions, and use them as an abductive reasoning sample, observations are ``All men are mortal. Socrates is mortal.", and the best explanation (hypothesis) is ``Socrates is a man.".
% In this way, we can automatically generate a large number of abductive reasoning pairs based on the existing deductive reasoning datasets or logic-based generators.
% Building a dataset is crucial because the abductive reasoning task is a complex NLP task, which is difficult to solve without sufficient data.
% \todo{I do not understand the pragraph from here onwards}
% We can explore the ability of the model to not only rank the plausibility of the hypotheses but also generate hypotheses based on the given context by using the abductive reasoning dataset.
% Subsequently, adaptations of transfer learning can provide a powerful tool to improve the performance of the model on the legal retrieval task by exploiting the knowledge learned from the abductive reasoning task.
% The examples in Tables \ref{tab:res_ques1} and \ref{tab:res_ques2} are hypothetical and simpler than the setting of \art{} but serve to illustrate our concerns when looking at \art{} and the crowdsourcing approach. First, asking annotators to choose just one of the two hypotheses can cause annotators to miss the plausible hypotheses. Second, as we observed and demonstrated in this section, these two choices are not always mutually exclusive, so the decision can depend largely on biases, enthymemes, and the fear of making mistakes. These observations form an important basis for our approach when building the dataset in Section \ref{sec:data} and redefining the task in Section \ref{sec:task_refine}.
% \section{Method}
% \begin{figure*}
% \center
% \includegraphics[width=.8\textwidth]{AbductiveReasoning.png}
% \end{figure*}
\section{Dataset Construction}
\label{sec:data}
While crowdsourcing can be an effective way to build a large dataset for evaluating the performance of deep learning models on tasks which require some logical capability \cite{bhagavatula2020abductive}, such as abductive reasoning, it is important to carefully verify the soundness and quality of the data. This can help to ensure that the dataset accurately represents the task and does not contain errors or biases that could impact the model's performance.
One way to increase the number of data points and ensure the soundness of the data is to use logic-based data generation techniques, such as symbolic reasoning or logical theorem proving. This can help to generate a larger number of high-quality examples of abductive reasoning that are consistent with the intended task and application domain.
Additionally, it is important to define the task clearly and accurately based on the characteristics of the data. If the task is not well-defined or does not align with the characteristics of the data, the model's performance may be difficult to interpret or may be difficult to explain. Carefully defining the task and choosing a suitable evaluation metric can help to ensure that the model's performance is accurately assessed and interpreted.
% \subsection{Positive Samples}
First, we precisely formulate the problem. With two observations $\mathcal{O}_1$, $\mathcal{O}_2$, and a hypothesis $\mathcal{H}$, the model needs to verify the validity of the Expression \ref{eq:abductive}.
\begin{equation}
\label{eq:abductive}
\mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2
\end{equation}
We then analyze the two drawbacks of \art{} and subsequently construct \lart{} as an expanded and improved version based on negation rules and theory generators.
% \textbf{\art{} Drawback 1: Logical Consistence.}
\subsection{Logical Consistency}
Compared with conventional programming languages, natural language have higher semantic tolerance, but therefore lower logical consistency. This is why, until recently, direct natural language-to-software conversion tools have relatively limited use. This well-known challenge of natural language use can also be an issue affecting the quality of crowdsourced datasets like \art{} and the models trained on it.
Sharing the same view on this issue, \cite{Clark_2020} train their Transformer model with data generated from a logic-based program called a theorem generator. Expanding on this result, %Gaskell \etal
Gaskell et al. \cite{Gaskell_2022} introduce an adversarial framework to improve the logical consistency of these ``soft'' theorem provers. This is done by training a discriminator, which is then used to detect incorrect outputs from the theorem prover.
The crucial insight of this work is that a model trained with improved logical consistency can be applied to the task of soft theorem-proving with higher accuracy.
Without the support of a logic-based program in the data generation process, \art{} does not provide any guarantee of logical consistency of its content. For example, there are some samples in the dataset whose plausibility determination is based heavily on the enthymeme in the evaluator's knowledge base, which has the potential to introduce inconsistency.
% \todo{some examples would be helpful here}
For example, here is a sample from \art{}:
\begin{itemize}
\item \textbf{Observation 1:} Ron started his new job as a landscaper today.
\item \textbf{Observation 2:} Ron is immediately fired for insubordination.
\item \textbf{Hypothesis 1:} Ron ignores his boss's orders and called him an idiot.
\item \textbf{Hypothesis 2:} Ron's boss called him an idiot.
\end{itemize}
In other words, without a logic-based program in place, the data generated by \art{} may not be completely consistent or accurate. Additionally, the evaluator's own biases and knowledge can influence the plausibility of certain samples, leading to inconsistencies in the dataset. The example above, where Ron is fired for insubordination, produces two different hypotheses for why that might be the case, which further illustrates this potential for inconsistency in the data generated by \art{}.
To overcome this drawback, instead of only requiring the model to choose between two given candidate hypotheses, we reformatted the dataset and forced the powerful pretrained models to {\em predict} the binary label without limiting the number of candidates.
The negative samples are derived from the positive ones by using logical negation, which guarantees the positive triples contain the best hypotheses explaining the given observations.
This is an important distinction from the way the dataset was used in the original setting.
This adjustment can provide the ability to achieve logical consistency, and determine whether or not a candidate can be a valid hypothesis with the two given observations.
% The results presented and analyzed in Section~\ref{sec:experiment} shows this drawback of \art{} dataset.
% \textbf{\art{} Drawback 2: Observation-Hypothesis Interchangeability.}
\subsection{Observation-Hypothesis Interchangeability}
From $\mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2 $, we can deduce $\mathcal{H} \land \mathcal{O}_1 \implies \mathcal{O}_2 $.
In other words, the first observation and the hypothesis are interchangeable. More specifically, they are two events producing the second observation. Of the two events becomes the observation, while the event {\em we do not observe} becomes the hypothesis. However, in terms of logic, the two events hold equal footing, as they are both postulated to explain the second observation.
From this, in terms of dataset construction, we can double
the number of positive samples by reversing the role of the hypothesis and the first observation.
This logical reformulation helps us to realize that if we can not interchange the hypothesis and the first observation in a triple, the triple is not a valid abductive reasoning sample. For example, assume that we have a silly triple of $\mathcal{O}_1$, $\mathcal{H}$ and $\mathcal{O}_2$ as follows:
\begin{itemize}
\item $\mathcal{O}_1$: John is the smartest person in the class.
\item $\mathcal{H}$: Every smart person has a green car.
\item $\mathcal{O}_2$: John has a green car.
\end{itemize}
In this case, we cannot interchange the $\mathcal{H}$ and $\mathcal{O}_1$ to get:
\begin{itemize}
\item $\mathcal{O}_1$: Every smart person has a green car.
\item $\mathcal{H}$: John is the smartest person in the class.
\item $\mathcal{O}_2$: John has a green car.
\end{itemize}
This is because the inference chain requires one more piece of information (i.e., the argument for $\mathcal{O}_2$ from $\mathcal{O}_1$ and $\mathcal{H}$ is an enthymeme), namely: ``The smartest person in the class is a smart person,'' which is not included in the triple.
% and we have no information about its truth value (e.g., the smartest person is not necessarily a smart person). As a result, the triple is not a logically valid abduction inference.\todo{I am not convinced that the original triple makes sense, abductively, for the same reason....}
%Thanh: Yes, I agree. Changed the wording
In the latter triple, the hypothesis ``John is the smartest person in the class." is not reasonable given only the two observations ``Every smart person has a green car." and ``John has a green~car."
% \textbf{Logic-augmented Abductive Reasoning Dataset (\lart{}).}
\subsection{Logic-augmented Dataset}
In our dataset construction process, we consider the above two reformulation factors which are not considered in \art{}: (1) We use a logic-based theorem generator to ensure logical consistency in the data; (2) We use logical formulas to ensure the validity of the triples in terms of abductive reasoning.
Based on these two transformations, we expand and improve the \art{} dataset and propose a new dataset called the \emph{logic-augmented abductive reasoning dataset (\lart{})}.
\lart{} is introduced as a dataset for a binary classification problem, where the model is trained to predict the validity of each provided triple.
As described in Section \ref{sec:ar}, in the \art{} dataset there are triples with high plausibility and others with low plausibility. We select the highly plausible
% \todo{what does this mean?}
triples as positive samples.
With the logic-based theorem generator, we randomly generate positive samples
that are logically consistent and find the inference chains which have at least two inference steps to extract $\mathcal{O}_1$, $\mathcal{H}$ and $\mathcal{O}_2$.
The hypotheses are then reversed to double the number of positive samples.
% \subsection{Negative Samples}
Producing negative samples in the context of abductive reasoning is more challenging than that in ordinary negation. We are looking for a hypothesis to explain the given observation; but we can not apply a random strategy to generate negative samples as there is no way for us to know whether a random hypothesis is reasonable for the given observations.
We therefore limit the possibilities through our strategy of exploiting negation, but applying this strategy is not straightforward in the context of abductive reasoning. Consider that randomly negating the operators in the Expression
% \todo{Equation?}
% According to this (https://math.libretexts.org/Bookshelves/Combinatorics_and_Discrete_Mathematics/A_Spiral_Workbook_for_Discrete_Mathematics_(Kwong)/02%3A_Logic/2.03%3A_Implications), I think we can use "Expression"
\ref{eq:abductive} might still yield a triple labeled as true. We handle this issue by first constructing a truth table for the Expression \ref{eq:abductive}, and then uniformly negating the operators. The truth table for the Expression \ref{eq:abductive} is as in Table \ref{tab:truth_value}. %was : but the table was shown at the top of the page, so I substituted it with '.' - KS
\begin{table}
\centering
\caption{Truth table for Expression~\ref{eq:abductive}: $\mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2 $. The first row corresponds to the truth values in the case of positive samples. }
\begin{tabular}{|c|c|c|c|}
\hline
$\mathcal{O}_1$ & $\mathcal{H}$ & $\mathcal{O}_2$ &$\mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2 $ \\ \hline
\textbf{T} & \textbf{T} & \textbf{T} & \textbf{T} \\ \hline
F & F & F & T \\ \hline
F & F & T & T \\ \hline
T & F & F & T \\ \hline
T & F & T & T \\ \hline
F & T & F & T \\ \hline
F & T & T & T \\ \hline
T & T & F & F \\ \hline
\end{tabular}
\label{tab:truth_value}
\end{table}
The first row of Table \ref{tab:truth_value} corresponds to the truth values in the case of positive samples. We can easily see that the only logical option to negate Expression \ref{eq:abductive} is to negate the operator $\mathcal{O}_2$. Interestingly, when we interchange $\mathcal{O}_1$ and $\mathcal{H}$, the soundness of the system is not affected.
This is because the soundness of the system is based on the logical equivalence of $ \mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2$ and $ \mathcal{H} \land \mathcal{O}_1 \implies \mathcal{O}_2$.
% Statistic
% 476167 train.csv
% 9339 dev.csv
% 13191 test.csv
% 498697 total
Compared to \art{}, the data format, number of samples, and their logic consistency in \lart{} are significantly improved.
\art{}, for the positive (plausible) hypotheses, presents a narrative context to Amazon Mechanical Turk workers who were then asked to make assumptions and write natural language hypotheses for the two given observations. For the negative (implausible) hypotheses, the plausible hypothesis can be modified through minimal edits (up to 5 words).
For the positive (plausible) hypotheses, we use a logic-based theorem generator to randomly generate positive samples that are logically consistent, and identify the inference chains which have at least two inference steps to extract $\mathcal{O}_1$, $\mathcal{H}$ and $\mathcal{O}_2$. We also reuse the positive samples from \art{} and reverse the role of the hypothesis and the first observation. For the negative (implausible) hypotheses, we use a truth table to construct negative samples. The truth table shows that to negate the expression $\mathcal{O}_1 \land \mathcal{H} \implies \mathcal{O}_2 $, we need to change the value of $\mathcal{O}_2$ to false.
The model is trained to predict the validity of each provided triple.
% \lart{} contains 498,697 samples, 476,167 samples are used as training set, 9,339 samples are used as validation set, and 13,191 samples are used as test set.
% \todo{how many datapoints did \art{} contain?}
% \todo{in which sense is this dataset suitable for legal reasoning?}
The \lart{} dataset has almost 2.5 times as many samples as the \art{} dataset, which contains 200k samples. So 476,167 of the samples in \lart{} are used for training, 9,339 for validation, and 13,191 for testing.
This dataset can be used as a benchmark for measuring the abductive reasoning skills of state-of-the-art models in the legal domain, but it's not limited to this purpose, as it can also be a valuable resource for other natural language processing and machine learning tasks, especially to consider complex NLP tasks because of transformers lack of reasoning ability.
\section{Task Redefinition}
\label{sec:task_refine}
In addition to the \lart{} data's quantity and quality, task definition is also important in training and evaluating the model.
Bhagavatula et al. \cite{bhagavatula2020abductive} introduce the ${\alpha}NLI$ task, in a way that the model needs to select the most plausible explanatory hypothesis between the two given. We argue that although this task is appropriate for evaluating the ability of the model to perform abductive reasoning and find the most plausible explanation, the way the authors limit the ${\alpha}NLI$ task to a binary classification of two hypotheses makes the problem easier and can lead to an overfitting of the model. The model needs only to learn a binary classification of the two hypotheses and does not need to learn to find the most plausible explanation.
They define ${\alpha}NLI$ task as follows:
\begin{itemize}
\item $\mathcal{O}_1$ and $\mathcal{O}_2$ are two observations at time $t_1 < t_2$.
\item $h+$ is a positive (plausible) hypothesis and $h-$ is a negative (implausible) hypothesis.
\item The ${\alpha}NLI$
% \todo{rename ${\alpha}NLI^*$}
% Thanh renamed our task and reworded some related parts (below)
task is to select the most plausible hypothesis from the $h+$ and $h-$.
\end{itemize}
\textbf{}
We redefine ${\alpha}NLI$ as ${\alpha}NLI^*$:
\begin{itemize}
\item $\mathcal{O}_1$ and $\mathcal{O}_2$ are two observations at time $t_1 < t_2$.
\item $h$ is a candidate hypothesis.
\item The ${\alpha}NLI^*$ task is to test whether triple $\mathcal{(O}_1$, $h$, $\mathcal{O}_2$) is valid.
\end{itemize}
The approach of ${\alpha}NLI^*$ is similar to \textbf{${\alpha}NLI$} but different in the following ways:
\begin{itemize}
\item We ask the model to validate the triple instead of only choosing which hypothesis is more plausible amongst the given two;
\item In ${\alpha}NLI$, if the model did not choose a hypothesis, we still do not know whether it is valid, so we can only know that it is not plausible as the chosen one;
\item In addition, ${\alpha}NLI^*$ can be feasible even when only one hypothesis or more hypotheses are given, which is not possible in ${\alpha}NLI$.
\end{itemize}
\section{Experiments and Discussions}
\label{sec:experiment}
Our experiments are designed to test the performance of several alternative models for binary classification tasks related to abductive reasoning using the \lart{} dataset. Our extended dataset consists of 498,697 samples, of which 476,167 are used as the training set, 9,339 are used as the validation set, and 13,191 are used as the test set. The max-length in characters for observation 1, observation 2, and hypothesis in the training, validation, and test sets are shown in Table \ref{tab:maxlength}. Pre-trained transformer models have a built-in maximum token length for input, and ensuring that the maximum length does not exceed this limit helps avoid over-truncation issues.
% With these lengths, we can ensure no failure in terms of over-truncation for the training, validation and testing sets.
The number of samples in each class is identical in our binary classification setting.
\begin{table}[ht]
\centering
\caption{Max-length in characters for observation 1, observation 2, and hypothesis in the train, validation, and test sets.}
\begin{tabular}{lrrr}
\hline
& Train & Validation & Test \\ \hline
Observation 1 ($\mathcal{O}_1$) & 71 & 71 & 71 \\
Observation 2 ($\mathcal{O}_2$) & 74 & 71 & 71 \\
Hypothesis ($\mathcal{H}$) & 184 & 147 & 150 \\ \hline
\end{tabular}
\label{tab:maxlength}
\end{table}
In this experiment, it is crucial to use different training, validation, and test sets to ensure the evaluation of the selected transformer models is reliable and not influenced by overfitting to the training data. The training set is utilized for training the models, while the validation set is employed for tuning the hyperparameters and selecting the best model. Lastly, the test set assesses the final performance of the chosen model.
We chose specific train/test/validation split ratios to ensure that the models have adequate data for training while still maintaining ample samples for validation and testing. It is worth noting that deep learning approaches like these are inherently statistical in nature, and our validation/test sets, with approximately 10K samples each, provide a reliable reflection of the results.
We use several different transformer models in the experiment: the original BERT~\cite{devlin2018bert} (base and large version), and the top legal models (BERT-PLI~\cite{ijcai2020p484}, Legal BERT~\cite{chalkidis2020legal}, BERTLaw\cite{nguyen2020jnlp} and NFSP version of Paralaw NetsParaLaw Nets~\cite{nguyen2021jnlp}).
We train each model on the input data, using the valid and invalid triples as the labels, and evaluate the performance of the models on the test set using accuracy. We also report the performance on our validation set. We run the experiment multiple times to ensure the reliability of the results, and we analyze the performance of the models on the test data to determine which model performs best on the binary classification tasks related to abductive reasoning.
We also tested GPT-3 \cite{brown2020language} zero-shot prediction and recorded the results on the test set of \lart{}.
\begin{table}[ht]
\caption{Average performance of the models in three runs, sorted from high to low.}
\label{tab:results}
\begin{center}
\begin{tabular}{lcr}
\hline
Model & Validation Accuracy & Test Accuracy \\
\hline
BERT Base & 0.6202 & 0.6162 \\
BERT-PLI & 0.6115 & 0.6115 \\
NFSP & 0.5825 & 0.5808 \\
Legal BERT & 0.5756 & 0.5619 \\
BERTLaw & 0.5484 & 0.5371 \\
BERT Large &0.5016&0.5000 \\
GPT-3 &-&0.4959\\
\hline
\end{tabular}
\end{center}
\end{table}
Table \ref{tab:results} shows the results of the experiment.
Our first observation is that performance on the validation set and the test set are not significantly different, which is a good indication that the models aren't overfitting the training data or the hyperparameter tuning process. From those results, we can also observe that the original BERT Base model performs better on abductive reasoning than all state-of-the-art legal models and even the BERT Large model. This result is quite surprising because it means that the pre-trained legal models are not necessarily more effective than the original BERT Base model on abductive reasoning.
We believe this is because the legal models are trained on documents that are mostly directed toward legal reasoning, rather than abductive reasoning. In the legal domain, there are many documents that contain information that can help with legal reasoning, but there is not a lot of information in the legal domain that can help with abduction. This imbalance in the training data can lead to a bias in the training process that favors legal reasoning and harms the performance of the models on abductive reasoning.
% In the real world, legal knowledge is not all about reasoning. It is also about understanding the context of the situation and making decisions based on that understanding. For example, a lawyer might need to know about a particular law in order to provide advice to a client, but that knowledge is not necessarily going to help with legal reasoning. We believe that this is why the original BERT Base model is better than all of the state-of-the-art legal models on abductive reasoning.
The worst performance among finetuned models is BERT Large, which is also a surprise since it is usually reported to be a robust model on many NLP tasks.
This result suggests that pretraining a model with a larger capacity with more data does not guarantee better performance.
Adding commentary on GPT-3, which has the lowest performance on zero-shot tasks, we included it for reference purposes, not for comparison, as the model has not been tuned for the specific domain and therefore cannot be fairly compared to the other models. Furthermore, a general comment can be made that the poor performance of these models indicates that abductive reasoning remains a challenging problem. The redefinition of the task within \lart{} helps to make this issue more evident.
% GPT-3 has been used in a wide range of applications, from chatbots to language translation. However, its performance in \lart{} is not as strong. One possible explanation is that abductive reasoning involves making inferential connections between different pieces of information, which is a difficult task for GPT-3, as it is mainly trained to generate text that is coherent and grammatically correct.
% As many would be curious about the capabilities of ChatGPT (also known as GPT-3.5), we test a few examples of abductive reasoning and observe the output of this newer system. We find that although the system may be capable of producing persuasive texts, its capacity for abductive reasoning is still limited. Table \ref{tab:chatgpt_ar} shows our questions based on the example in Section \ref{sec:ar} and ChatGPT's answer.
% The system evaluates both hypotheses as plausible with ``seemingly convincing" explanations.
% Table \ref{tab:chatgpt_lr} shows ChatGPT's answers to the questions from COLIEE \cite{Rabelo_2022}.
% Although the system seems to make a correct correspondence with an object in a concrete case description and a term in the civil article (Q4), the system failed to answer questions related to disjunctive logical reasoning (Q1 and Q2) and temporal reasoning (Q3).
% This confirms that ChatGPT -- indeed all transformers -- confirm that a lack of reasoning ability becomes a problem in application specific tasks: thus, there is a big gap in the current capacity of generative transformer models to support the requirements of abductive reasoning in particular and legal reasoning in general.
% \begin{table*}
% \centering
% \caption{Questions based on the example in Section \ref{sec:ar} and ChatGPT's answer.}
% \label{tab:chatgpt_ar}
% \begin{tabular}{|l|p{15cm}|}
% \hline
% Question & Jenny cleaned her house and went to work, leaving the window just a crack open. When she came back she found her house was a mess. In terms of abductive reasoning, do you think here is a valid hypothesis "A thief broke into the house by pulling open the window."? \\ \hline
% Answer & Yes, "A thief broke into the house by pulling open the window" is a valid hypothesis based on the given information. It is a plausible explanation for the mess in the house and the open window, and it can be further investigated by looking for evidence such as fingerprints or surveillance footage. Abductive reasoning is a method of logical inference that is used to generate hypotheses that could explain a given set of observations or facts. In this case, the observations are that the house is a mess and the window is open, and the hypothesis is that a thief broke in. \\ \hline \hline
% Question & Jenny cleaned her house and went to work, leaving the window just a crack open. When she came back she found her house was a mess. In terms of abductive reasoning, do you think here is a valid hypothesis "At work, she opened her window and the wind blew her papers everywhere."? \\ \hline
% Answer & Yes, "At work, she opened her window and the wind blew her papers everywhere" is a valid hypothesis based on the given information. It is a plausible explanation for the mess in the house and the open window, and it could be further investigated by asking the person if she opened the window at work or if there is any windy weather that day. Abductive reasoning is a method of logical inference that is used to generate hypotheses that could explain a given set of observations or facts. In this case, the observations are that the house is a mess and the window is open, and the hypothesis is that the wind blew papers around while she was at work. \\ \hline
% \end{tabular}
% \end{table*}
% \begin{table*}
% \centering
% \caption{Legal questions from COLIEE and ChatGPT's answer.}
% \label{tab:chatgpt_lr}
% \begin{tabular}{|l|p{15cm}|}
% \hline
% Question & \textbf{Q1:} According to the Japanese civil code, with respect to any inherited property, even if the administrator of the inherited property is appointed, shall the prescription not be completed until the applicable heir is identified? \\ \hline
% Answer & In the Japanese civil code, the prescription period for inherited property does not expire until six months have passed from the time when the heir is identified, the administrator is appointed, or the commencement of bankruptcy proceeding is ordered. Therefore, if the applicable heir is not yet identified, the prescription shall not be completed. \\ \hline \hline
% Question & \textbf{Q2:} According to the Japanese civil code, may the creditor of a partnership not exercise his/her rights against each partner? \\ \hline
% Answer & According to the Japanese civil code, a creditor of a partnership may exercise the rights of the creditor against the partnership property or at the choice of the creditor, exercise the rights of the creditor against each partner in proportion to their shares of loss or in equal proportions. Therefore, the creditor of a partnership may not exercise his/her rights against each partner. \\ \hline \hline
% Question & \textbf{Q3:} According to the Japanese civil code, if the delivery of the subject matter of work performed is required, may the contractor not claim the remuneration until and unless he/she has delivered the same? \\ \hline
% Answer & According to the Japanese civil code, remuneration must be paid simultaneously with the delivery of the subject matter of work performed. Therefore, if the delivery of the subject matter of work performed is required, the contractor may not claim the remuneration until and unless he/she has delivered the same.\\ \hline \hline
% Question & \textbf{Q4:} Alice has leased a machine tool owned by Alice to Bob, and Carol stole the machine tool from Bob within the duration of the lease agreement. In this case, may Bob file an action for recovery of possession against Carol independently of Alice according to the Japanese civil code?\\ \hline
% Answer & According to the Japanese civil code, if a possessor is forcibly dispossessed, the possessor may demand the restitution of the thing and compensation for loss or damage by filing an action for recovery of possession. Therefore, in this case, Bob may file an action for recovery of possession against Carol independently of Alice to recover the possession of the machine tool and seek compensation for loss or damages caused. \\ \hline
% \end{tabular}
% \end{table*}
% How to obtain a better model in the abductive reasoning task is still an open research question.
% \todo{Some discussion about GPT-3 performance on LART + example in COLIEE Competition}
\section{Conclusions}
We have investigated the support for abductive reasoning provided by state-of-the-art (SOTA) transformer models within the legal field. To accomplish this, we first redefined the task of abductive reasoning and constructed a reliable dataset. Following this, we utilized the dataset to assess the performance of SOTA models in the legal sphere, as well as prominent large language models in Natural Language Processing (NLP).
Our experimental results revealed that the SOTA models, including all legal-specific variants, do not necessarily outperform the original BERT Base model in abductive reasoning tasks. This outcome provides insight into current limitations when pretraining large language models for legal applications.
The subpar performance of the original BERT Large and GPT models illustrates that simply increasing the size of the model and providing it with more data does not guarantee superior performance.
Future directions for this research could include exploring alternative pretraining approaches specifically tailored to abductive reasoning tasks, developing novel architectures that focus on legal reasoning, and examining the relationship between model capacity and performance on abductive reasoning tasks. Additionally, further investigation into leveraging and integrating existing legal domain knowledge with the pretraining process may lead to more effective models capable of handling the unique challenges of legal reasoning tasks.
% We hope that this paper can encourage the community to work on this research question.
% If the performance of state-of-the-art models in the legal domain is found to be lacking in abductive reasoning tasks, this should prompt broader concerns about their ability to accurately interpret and apply laws and regulations. These concerns underscore the importance of a comprehensive and robust inferencing capacity in legal AI systems for the correct application of complex legal principles.
%%
%% The acknowledgments section is defined using the "acknowledgments" environment
%% (and NOT an unnumbered section). This ensures the proper
%% identification of the section in the article metadata, and the
%% consistent spelling of the heading.
% \begin{acknowledgments}
% Thanks to the developers of ACM consolidated LaTeX styles
% \url{https://github.com/borisveytsman/acmart} and to the developers
% of Elsevier updated \LaTeX{} templates
% \url{https://www.ctan.org/tex-archive/macros/latex/contrib/els-cas-templates}.
% \end{acknowledgments}
\begin{acknowledgments}
This work was supported by JSPS KAKENHI Grant Number, JP22H00543 and JST, AIP Trilateral AI Research, Grant Number JPMJCR20G4.
Francesca Toni and Kostas Stathis would like to thank the National Institute of Informatics, Tokyo, Japan, for supporting their visit to Japan that made this work possible. Francesca Toni also acknowledges support from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No.101020934, ADIX), as well as support from J.P. Morgan and the Royal Academy of Engineering, UK, under the Research Chairs and Senior Research Fellowships scheme.
\end{acknowledgments}
%%
%% Define the bibliography file to be used
\bibliography{sample-ceur}
%%
%% If your work has an appendix, this is the place to put it.
\appendix
% \section{Online Resources}
% The sources for the ceur-art style are available via
% \begin{itemize}
% \item \href{https://github.com/yamadharma/ceurart}{GitHub},
% % \item \href{https://www.overleaf.com/project/5e76702c4acae70001d3bc87}{Overleaf},
% \item
% \href{https://www.overleaf.com/latex/templates/template-for-submissions-to-ceur-workshop-proceedings-ceur-ws-dot-org/pkfscdkgkhcq}{Overleaf
% template}.
% \end{itemize}
\end{document}
%%
%% End of file
\begin{thebibliography}{32}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Inproceedings
\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin}]{vaswani2017attention}
\bibinfo{author}{A.~Vaswani}, \bibinfo{author}{N.~Shazeer},
\bibinfo{author}{N.~Parmar}, \bibinfo{author}{J.~Uszkoreit},
\bibinfo{author}{L.~Jones}, \bibinfo{author}{A.~N. Gomez},
\bibinfo{author}{{\L}.~Kaiser}, \bibinfo{author}{I.~Polosukhin},
\newblock \bibinfo{title}{Attention is all you need},
\newblock in: \bibinfo{booktitle}{Advances in Neural Information Processing
Systems}, \bibinfo{year}{2017}, pp. \bibinfo{pages}{5998--6008}.
%Type = Inproceedings
\bibitem[{Shao et~al.(2020)Shao, Mao, Liu, Ma, Satoh, Zhang, and
Ma}]{ijcai2020p484}
\bibinfo{author}{Y.~Shao}, \bibinfo{author}{J.~Mao}, \bibinfo{author}{Y.~Liu},
\bibinfo{author}{W.~Ma}, \bibinfo{author}{K.~Satoh},
\bibinfo{author}{M.~Zhang}, \bibinfo{author}{S.~Ma},
\newblock \bibinfo{title}{Bert-pli: Modeling paragraph-level interactions for
legal case retrieval},
\newblock in: \bibinfo{editor}{C.~Bessiere} (Ed.),
\bibinfo{booktitle}{Proceedings of the Twenty-Ninth International Joint
Conference on Artificial Intelligence, {IJCAI-20}},
\bibinfo{publisher}{International Joint Conferences on Artificial
Intelligence Organization}, \bibinfo{address}{Yokohama, Japan},
\bibinfo{year}{2020}, pp. \bibinfo{pages}{3501--3507}. \URLprefix
\url{https://doi.org/10.24963/ijcai.2020/484}.
\DOIprefix\doi{10.24963/ijcai.2020/484}, \bibinfo{note}{main track}.
%Type = Article
\bibitem[{Chalkidis et~al.(2020)Chalkidis, Fergadiotis, Malakasiotis, Aletras,
and Androutsopoulos}]{chalkidis2020legal}
\bibinfo{author}{I.~Chalkidis}, \bibinfo{author}{M.~Fergadiotis},
\bibinfo{author}{P.~Malakasiotis}, \bibinfo{author}{N.~Aletras},
\bibinfo{author}{I.~Androutsopoulos},
\newblock \bibinfo{title}{Legal-bert: The muppets straight out of law school},
\newblock \bibinfo{journal}{arXiv preprint arXiv:2010.02559}
(\bibinfo{year}{2020}).
%Type = Article
\bibitem[{Nguyen et~al.(2020)Nguyen, Vuong, Nguyen, Dang, Bui, Vu, Nguyen,
Tran, Satoh, and Nguyen}]{nguyen2020jnlp}
\bibinfo{author}{H.-T. Nguyen}, \bibinfo{author}{H.-Y.~T. Vuong},
\bibinfo{author}{P.~M. Nguyen}, \bibinfo{author}{B.~T. Dang},
\bibinfo{author}{Q.~M. Bui}, \bibinfo{author}{S.~T. Vu},
\bibinfo{author}{C.~M. Nguyen}, \bibinfo{author}{V.~Tran},
\bibinfo{author}{K.~Satoh}, \bibinfo{author}{M.~L. Nguyen},
\newblock \bibinfo{title}{Jnlp team: Deep learning for legal processing in
coliee 2020},
\newblock \bibinfo{journal}{arXiv preprint arXiv:2011.08071}
(\bibinfo{year}{2020}).
%Type = Article
\bibitem[{Tran et~al.(2020)Tran, Le~Nguyen, Tojo, and Satoh}]{tran2020encoded}
\bibinfo{author}{V.~Tran}, \bibinfo{author}{M.~Le~Nguyen},
\bibinfo{author}{S.~Tojo}, \bibinfo{author}{K.~Satoh},
\newblock \bibinfo{title}{Encoded summarization: summarizing documents into
continuous vector space for legal case retrieval},
\newblock \bibinfo{journal}{Artificial Intelligence and Law}
\bibinfo{volume}{28} (\bibinfo{year}{2020}) \bibinfo{pages}{441--467}.
%Type = Inproceedings
\bibitem[{Yoshioka et~al.(2021)Yoshioka, Aoki, and Suzuki}]{yoshioka2021bert}
\bibinfo{author}{M.~Yoshioka}, \bibinfo{author}{Y.~Aoki},
\bibinfo{author}{Y.~Suzuki},
\newblock \bibinfo{title}{Bert-based ensemble methods with data augmentation
for legal textual entailment in coliee statute law task},
\newblock in: \bibinfo{booktitle}{Proceedings of the Eighteenth International
Conference on Artificial Intelligence and Law}, \bibinfo{year}{2021}, pp.
\bibinfo{pages}{278--284}.
%Type = Article
\bibitem[{Nguyen et~al.(2022)Nguyen, Phi, Ngo, Tran, Nguyen, and
Tu}]{nguyen2022attentive}
\bibinfo{author}{H.-T. Nguyen}, \bibinfo{author}{M.-K. Phi},
\bibinfo{author}{X.-B. Ngo}, \bibinfo{author}{V.~Tran},
\bibinfo{author}{L.-M. Nguyen}, \bibinfo{author}{M.-P. Tu},
\newblock \bibinfo{title}{Attentive deep neural networks for legal document
retrieval},
\newblock \bibinfo{journal}{Artificial Intelligence and Law}
(\bibinfo{year}{2022}) \bibinfo{pages}{1--30}.
%Type = Article
\bibitem[{Rabelo et~al.(2022)Rabelo, Goebel, Kim, Kano, Yoshioka, and
Satoh}]{Rabelo_2022}
\bibinfo{author}{J.~Rabelo}, \bibinfo{author}{R.~Goebel},
\bibinfo{author}{M.-Y. Kim}, \bibinfo{author}{Y.~Kano},
\bibinfo{author}{M.~Yoshioka}, \bibinfo{author}{K.~Satoh},
\newblock \bibinfo{title}{Overview and discussion of the competition on legal
information extraction/entailment (coliee) 2021},
\newblock \bibinfo{journal}{The Review of Socionetwork Strategies}
\bibinfo{volume}{16} (\bibinfo{year}{2022}) \bibinfo{pages}{111--133}.
%Type = Article
\bibitem[{Abimbola(2000)}]{abimbola2000abductive}
\bibinfo{author}{K.~Abimbola},
\newblock \bibinfo{title}{Abductive reasoning in law: Taxonomy and inference to
the best explanation},
\newblock \bibinfo{journal}{CARDozo L. REv.} \bibinfo{volume}{22}
(\bibinfo{year}{2000}) \bibinfo{pages}{1683}.
%Type = Incollection
\bibitem[{Schum(2002)}]{schum2002species}
\bibinfo{author}{D.~A. Schum},
\newblock \bibinfo{title}{Species of abductive reasoning in fact investigation
in law},
\newblock in: \bibinfo{booktitle}{The Dynamics of Judicial Proof},
\bibinfo{publisher}{Springer}, \bibinfo{year}{2002}, pp.
\bibinfo{pages}{307--336}.
%Type = Inproceedings
\bibitem[{Bhagavatula et~al.(2020)Bhagavatula, Bras, Malaviya, Sakaguchi,
Holtzman, Rashkin, Downey, Yih, and Choi}]{bhagavatula2020abductive}
\bibinfo{author}{C.~Bhagavatula}, \bibinfo{author}{R.~L. Bras},
\bibinfo{author}{C.~Malaviya}, \bibinfo{author}{K.~Sakaguchi},
\bibinfo{author}{A.~Holtzman}, \bibinfo{author}{H.~Rashkin},
\bibinfo{author}{D.~Downey}, \bibinfo{author}{W.~Yih},
\bibinfo{author}{Y.~Choi},
\newblock \bibinfo{title}{Abductive commonsense reasoning},
\newblock in: \bibinfo{booktitle}{8th International Conference on Learning
Representations, {ICLR} 2020}, \bibinfo{publisher}{OpenReview.net},
\bibinfo{address}{Addis Ababa, Ethiopia}, \bibinfo{year}{2020}. \URLprefix
\url{https://openreview.net/forum?id=Byg1v1HKDB}.
%Type = Inproceedings
\bibitem[{Kien et~al.(2020)Kien, Nguyen, Bach, Tran, Nguyen, and
Phuong}]{kien-etal-2020-answering}
\bibinfo{author}{P.~M. Kien}, \bibinfo{author}{H.-T. Nguyen},
\bibinfo{author}{N.~X. Bach}, \bibinfo{author}{V.~Tran},
\bibinfo{author}{M.~L. Nguyen}, \bibinfo{author}{T.~M. Phuong},
\newblock \bibinfo{title}{Answering legal questions by learning neural
attentive text representation},
\newblock in: \bibinfo{booktitle}{Proceedings of the 28th International
Conference on Computational Linguistics}, \bibinfo{publisher}{International
Committee on Computational Linguistics}, \bibinfo{address}{Barcelona, Spain
(Online)}, \bibinfo{year}{2020}, pp. \bibinfo{pages}{988--998}. \URLprefix
\url{https://aclanthology.org/2020.coling-main.86}.
\DOIprefix\doi{10.18653/v1/2020.coling-main.86}.
%Type = Inproceedings
\bibitem[{Asai et~al.(2021)Asai, Kasai, Clark, Lee, Choi, and
Hajishirzi}]{asai-etal-2021-xor}
\bibinfo{author}{A.~Asai}, \bibinfo{author}{J.~Kasai},
\bibinfo{author}{J.~Clark}, \bibinfo{author}{K.~Lee},
\bibinfo{author}{E.~Choi}, \bibinfo{author}{H.~Hajishirzi},
\newblock \bibinfo{title}{{XOR} {QA}: Cross-lingual open-retrieval question
answering},
\newblock in: \bibinfo{booktitle}{Proceedings of the 2021 Conference of the
North American Chapter of the Association for Computational Linguistics:
Human Language Technologies}, \bibinfo{publisher}{Association for
Computational Linguistics}, \bibinfo{address}{Online}, \bibinfo{year}{2021},
pp. \bibinfo{pages}{547--564}. \URLprefix
\url{https://aclanthology.org/2021.naacl-main.46}.
\DOIprefix\doi{10.18653/v1/2021.naacl-main.46}.
%Type = Inproceedings
\bibitem[{Nie et~al.(2019)Nie, Wang, and Bansal}]{nie-etal-2019-revealing}
\bibinfo{author}{Y.~Nie}, \bibinfo{author}{S.~Wang},
\bibinfo{author}{M.~Bansal},
\newblock \bibinfo{title}{Revealing the importance of semantic retrieval for
machine reading at scale},
\newblock in: \bibinfo{booktitle}{Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{address}{Hong Kong, China}, \bibinfo{year}{2019}, pp.
\bibinfo{pages}{2553--2566}. \URLprefix
\url{https://aclanthology.org/D19-1258}.
\DOIprefix\doi{10.18653/v1/D19-1258}.
%Type = Inproceedings
\bibitem[{Lee and Yeung(2021)}]{lee-yeung-2021-text}
\bibinfo{author}{J.~Lee}, \bibinfo{author}{C.~Y. Yeung},
\newblock \bibinfo{title}{Text retrieval for language learners: Graded
vocabulary vs. open learner model},
\newblock in: \bibinfo{booktitle}{Proceedings of the International Conference
on Recent Advances in Natural Language Processing (RANLP 2021)},
\bibinfo{publisher}{INCOMA Ltd.}, \bibinfo{address}{Held Online},
\bibinfo{year}{2021}, pp. \bibinfo{pages}{798--804}. \URLprefix
\url{https://aclanthology.org/2021.ranlp-1.91}.
%Type = Inproceedings
\bibitem[{Krojer et~al.(2022)Krojer, Adlakha, Vineet, Goyal, Ponti, and
Reddy}]{krojer-etal-2022-image}
\bibinfo{author}{B.~Krojer}, \bibinfo{author}{V.~Adlakha},
\bibinfo{author}{V.~Vineet}, \bibinfo{author}{Y.~Goyal},
\bibinfo{author}{E.~Ponti}, \bibinfo{author}{S.~Reddy},
\newblock \bibinfo{title}{Image retrieval from contextual descriptions},
\newblock in: \bibinfo{booktitle}{Proceedings of the 60th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers)},
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{address}{Dublin, Ireland}, \bibinfo{year}{2022}, pp.
\bibinfo{pages}{3426--3440}. \URLprefix
\url{https://aclanthology.org/2022.acl-long.241}.
\DOIprefix\doi{10.18653/v1/2022.acl-long.241}.
%Type = Inproceedings
\bibitem[{Lin et~al.(2020)Lin, Lee, Shen, Moreno, Huang, Shiralkar, and
Ren}]{lin-etal-2020-triggerner}
\bibinfo{author}{B.~Y. Lin}, \bibinfo{author}{D.-H. Lee},
\bibinfo{author}{M.~Shen}, \bibinfo{author}{R.~Moreno},
\bibinfo{author}{X.~Huang}, \bibinfo{author}{P.~Shiralkar},
\bibinfo{author}{X.~Ren},
\newblock \bibinfo{title}{{T}rigger{NER}: Learning with entity triggers as
explanations for named entity recognition},
\newblock in: \bibinfo{booktitle}{Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics}, \bibinfo{publisher}{Association
for Computational Linguistics}, \bibinfo{address}{Online},
\bibinfo{year}{2020}, pp. \bibinfo{pages}{8503--8511}. \URLprefix
\url{https://aclanthology.org/2020.acl-main.752}.
\DOIprefix\doi{10.18653/v1/2020.acl-main.752}.
%Type = Inproceedings
\bibitem[{Ritter et~al.(2011)Ritter, Clark, {Mausam}, and
Etzioni}]{ritter-etal-2011-named}
\bibinfo{author}{A.~Ritter}, \bibinfo{author}{S.~Clark},
\bibinfo{author}{{Mausam}}, \bibinfo{author}{O.~Etzioni},
\newblock \bibinfo{title}{Named entity recognition in tweets: An experimental
study},
\newblock in: \bibinfo{booktitle}{Proceedings of the 2011 Conference on
Empirical Methods in Natural Language Processing},
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{address}{Edinburgh, Scotland, UK.}, \bibinfo{year}{2011}, pp.
\bibinfo{pages}{1524--1534}. \URLprefix
\url{https://aclanthology.org/D11-1141}.
%Type = Inproceedings
\bibitem[{Rabelo et~al.(2020)Rabelo, Kim, Goebel, Yoshioka, Kano, and
Satoh}]{Rabelo_2020}
\bibinfo{author}{J.~Rabelo}, \bibinfo{author}{M.-Y. Kim},
\bibinfo{author}{R.~Goebel}, \bibinfo{author}{M.~Yoshioka},
\bibinfo{author}{Y.~Kano}, \bibinfo{author}{K.~Satoh},
\newblock \bibinfo{title}{A summary of the {COLIEE} 2019 competition},
\newblock in: \bibinfo{booktitle}{New Frontiers in Artificial Intelligence},
\bibinfo{publisher}{Springer International Publishing},
\bibinfo{address}{Online}, \bibinfo{year}{2020}, pp. \bibinfo{pages}{34--49}.
%Type = Inproceedings
\bibitem[{Thanh et~al.(2021)Thanh, Quan, Nguyen, Le, Phuong, Binh, Yen,
Racharak, Minh, Vu, Anh, Son, Nguyen, Butr-indr, Vateekul, and
Boonkwan}]{Thanh_2021}
\bibinfo{author}{N.~H. Thanh}, \bibinfo{author}{B.~M. Quan},
\bibinfo{author}{C.~Nguyen}, \bibinfo{author}{T.~Le}, \bibinfo{author}{N.~M.
Phuong}, \bibinfo{author}{D.~T. Binh}, \bibinfo{author}{V.~T.~H. Yen},
\bibinfo{author}{T.~Racharak}, \bibinfo{author}{N.~L. Minh},
\bibinfo{author}{T.~D. Vu}, \bibinfo{author}{P.~V. Anh},
\bibinfo{author}{N.~T. Son}, \bibinfo{author}{H.~T. Nguyen},
\bibinfo{author}{B.~Butr-indr}, \bibinfo{author}{P.~Vateekul},
\bibinfo{author}{P.~Boonkwan},
\newblock \bibinfo{title}{A summary of the {ALQAC} 2021 competition},
\newblock in: \bibinfo{booktitle}{2021 13th International Conference on
Knowledge and Systems Engineering ({KSE})}, \bibinfo{publisher}{{IEEE}},
\bibinfo{address}{Bangkok, Thailand}, \bibinfo{year}{2021}, pp.
\bibinfo{pages}{1--5}.
%Type = Inproceedings
\bibitem[{Nguyen et~al.(2022{\natexlab{a}})Nguyen, Bui, Do, Le, Nguyen, Nguyen,
Nguyen, Tran, Nguyen, Le, Le, Nguyen, Dang, Nguyen, Phan, Vuong, Nguyen, Le,
and Nguyen}]{Nguyen_2022}
\bibinfo{author}{C.~Nguyen}, \bibinfo{author}{M.-Q. Bui},
\bibinfo{author}{D.-T. Do}, \bibinfo{author}{N.-K. Le},
\bibinfo{author}{D.-H. Nguyen}, \bibinfo{author}{T.-T. Nguyen},
\bibinfo{author}{H.-T. Nguyen}, \bibinfo{author}{V.~Tran},
\bibinfo{author}{L.-M. Nguyen}, \bibinfo{author}{N.-C. Le},
\bibinfo{author}{T.-T. Le}, \bibinfo{author}{M.-P. Nguyen},
\bibinfo{author}{T.-B. Dang}, \bibinfo{author}{T.-S. Nguyen},
\bibinfo{author}{V.-A. Phan}, \bibinfo{author}{T.-H.-Y. Vuong},
\bibinfo{author}{M.-T. Nguyen}, \bibinfo{author}{T.~Le},
\bibinfo{author}{T.-H. Nguyen},
\newblock \bibinfo{title}{{ALQAC} 2022: A summary of the competition},
\newblock in: \bibinfo{booktitle}{2022 14th International Conference on
Knowledge and Systems Engineering ({KSE})}, \bibinfo{publisher}{{IEEE}},
\bibinfo{address}{Nha Trang, Vietnam}, \bibinfo{year}{2022}{\natexlab{a}},
pp. \bibinfo{pages}{1--5}. \DOIprefix\doi{10.1109/kse56063.2022.9953764}.
%Type = Article
\bibitem[{Nguyen et~al.(2022{\natexlab{b}})Nguyen, Nguyen, Vuong, Bui, Nguyen,
Dang, Tran, Nguyen, and Satoh}]{Nguyen_transformer_2022}
\bibinfo{author}{H.-T. Nguyen}, \bibinfo{author}{M.-P. Nguyen},
\bibinfo{author}{T.-H.-Y. Vuong}, \bibinfo{author}{M.-Q. Bui},
\bibinfo{author}{M.-C. Nguyen}, \bibinfo{author}{T.-B. Dang},
\bibinfo{author}{V.~Tran}, \bibinfo{author}{L.-M. Nguyen},
\bibinfo{author}{K.~Satoh},
\newblock \bibinfo{title}{Transformer-based approaches for legal text
processing},
\newblock \bibinfo{journal}{The Review of Socionetwork Strategies}
\bibinfo{volume}{16} (\bibinfo{year}{2022}{\natexlab{b}})
\bibinfo{pages}{135--155}. \DOIprefix\doi{10.1007/s12626-022-00102-2}.
%Type = Inproceedings
\bibitem[{Louis and Spanakis(2022)}]{louis-spanakis-2022-statutory}
\bibinfo{author}{A.~Louis}, \bibinfo{author}{G.~Spanakis},
\newblock \bibinfo{title}{A statutory article retrieval dataset in {F}rench},
\newblock in: \bibinfo{booktitle}{Proceedings of the 60th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers)},
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{address}{Dublin, Ireland}, \bibinfo{year}{2022}, pp.
\bibinfo{pages}{6789--6803}. \URLprefix
\url{https://aclanthology.org/2022.acl-long.468}.
\DOIprefix\doi{10.18653/v1/2022.acl-long.468}.
%Type = Article
\bibitem[{Vuong et~al.(2022)Vuong, Bui, Nguyen, Nguyen, Tran, Phan, Satoh, and
Nguyen}]{Vuong_2022}
\bibinfo{author}{Y.~T.-H. Vuong}, \bibinfo{author}{Q.~M. Bui},
\bibinfo{author}{H.-T. Nguyen}, \bibinfo{author}{T.-T.-T. Nguyen},
\bibinfo{author}{V.~Tran}, \bibinfo{author}{X.-H. Phan},
\bibinfo{author}{K.~Satoh}, \bibinfo{author}{L.-M. Nguyen},
\newblock \bibinfo{title}{{SM}-{BERT}-{CR}: a deep learning approach for case
law retrieval with supporting model},
\newblock \bibinfo{journal}{Artificial Intelligence and Law}
(\bibinfo{year}{2022}). \DOIprefix\doi{10.1007/s10506-022-09319-6}.
%Type = Article
\bibitem[{Kakas et~al.(1992)Kakas, Kowalski, and Toni}]{kakas1992abductive}
\bibinfo{author}{A.~C. Kakas}, \bibinfo{author}{R.~A. Kowalski},
\bibinfo{author}{F.~Toni},
\newblock \bibinfo{title}{Abductive logic programming},
\newblock \bibinfo{journal}{Journal of logic and computation}
\bibinfo{volume}{2} (\bibinfo{year}{1992}) \bibinfo{pages}{719--770}.
%Type = Inproceedings
\bibitem[{Lertvittayakumjorn et~al.(2021)Lertvittayakumjorn, Petej, Gao,
Krishnamurthy, Van Der~Gaag, Jago, and
Stathis}]{lertvittayakumjorn2021supporting}
\bibinfo{author}{P.~Lertvittayakumjorn}, \bibinfo{author}{I.~Petej},
\bibinfo{author}{Y.~Gao}, \bibinfo{author}{Y.~Krishnamurthy},
\bibinfo{author}{A.~Van Der~Gaag}, \bibinfo{author}{R.~Jago},
\bibinfo{author}{K.~Stathis},
\newblock \bibinfo{title}{Supporting complaints investigation for nursing and
midwifery regulatory agencies},
\newblock in: \bibinfo{booktitle}{Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing: System Demonstrations},
\bibinfo{year}{2021}, pp. \bibinfo{pages}{81--91}.
%Type = Article
\bibitem[{Jago et~al.(2021)Jago, van~der Gaag, Stathis, Petej,
Lertvittayakumjorn, Krishnamurthy, Gao, Silva, Webster, Gallagher
et~al.}]{jago2021use}
\bibinfo{author}{R.~Jago}, \bibinfo{author}{A.~van~der Gaag},
\bibinfo{author}{K.~Stathis}, \bibinfo{author}{I.~Petej},
\bibinfo{author}{P.~Lertvittayakumjorn}, \bibinfo{author}{Y.~Krishnamurthy},
\bibinfo{author}{Y.~Gao}, \bibinfo{author}{J.~C. Silva},
\bibinfo{author}{M.~Webster}, \bibinfo{author}{A.~Gallagher}, et~al.,
\newblock \bibinfo{title}{Use of artificial intelligence in regulatory
decision-making},
\newblock \bibinfo{journal}{Journal of Nursing Regulation} \bibinfo{volume}{12}
(\bibinfo{year}{2021}) \bibinfo{pages}{11--19}.
%Type = Inproceedings
\bibitem[{Clark et~al.(2020)Clark, Tafjord, and Richardson}]{Clark_2020}
\bibinfo{author}{P.~Clark}, \bibinfo{author}{O.~Tafjord},
\bibinfo{author}{K.~Richardson},
\newblock \bibinfo{title}{Transformers as soft reasoners over language},
\newblock in: \bibinfo{editor}{C.~Bessiere} (Ed.),
\bibinfo{booktitle}{Proceedings of the Twenty-Ninth International Joint
Conference on Artificial Intelligence, {IJCAI-20}},
\bibinfo{publisher}{International Joint Conferences on Artificial
Intelligence Organization}, \bibinfo{address}{Yokohama, Japan},
\bibinfo{year}{2020}, pp. \bibinfo{pages}{3882--3890}. \URLprefix
\url{https://doi.org/10.24963/ijcai.2020/537}.
\DOIprefix\doi{10.24963/ijcai.2020/537}, \bibinfo{note}{main track}.
%Type = Inproceedings
\bibitem[{Gaskell et~al.(2022)Gaskell, Miao, Toni, and Specia}]{Gaskell_2022}
\bibinfo{author}{A.~Gaskell}, \bibinfo{author}{Y.~Miao},
\bibinfo{author}{F.~Toni}, \bibinfo{author}{L.~Specia},
\newblock \bibinfo{title}{Logically consistent adversarial attacks for soft
theorem provers},
\newblock in: \bibinfo{editor}{L.~D. Raedt} (Ed.),
\bibinfo{booktitle}{Proceedings of the Thirty-First International Joint
Conference on Artificial Intelligence, {IJCAI-22}},
\bibinfo{publisher}{International Joint Conferences on Artificial
Intelligence Organization}, \bibinfo{address}{Vienna, Austria},
\bibinfo{year}{2022}, pp. \bibinfo{pages}{4129--4135}. \URLprefix
\url{https://doi.org/10.24963/ijcai.2022/573}.
\DOIprefix\doi{10.24963/ijcai.2022/573}, \bibinfo{note}{main Track}.
%Type = Article
\bibitem[{Devlin et~al.(2018)Devlin, Chang, Lee, and
Toutanova}]{devlin2018bert}
\bibinfo{author}{J.~Devlin}, \bibinfo{author}{M.-W. Chang},
\bibinfo{author}{K.~Lee}, \bibinfo{author}{K.~Toutanova},
\newblock \bibinfo{title}{Bert: Pre-training of deep bidirectional transformers
for language understanding},
\newblock \bibinfo{journal}{arXiv preprint arXiv:1810.04805}
(\bibinfo{year}{2018}).
%Type = Article
\bibitem[{Nguyen et~al.(2021)Nguyen, Nguyen, Vuong, Bui, Nguyen, Dang, Tran,
Nguyen, and Satoh}]{nguyen2021jnlp}
\bibinfo{author}{H.-T. Nguyen}, \bibinfo{author}{P.~M. Nguyen},
\bibinfo{author}{T.-H.-Y. Vuong}, \bibinfo{author}{Q.~M. Bui},
\bibinfo{author}{C.~M. Nguyen}, \bibinfo{author}{B.~T. Dang},
\bibinfo{author}{V.~Tran}, \bibinfo{author}{M.~L. Nguyen},
\bibinfo{author}{K.~Satoh},
\newblock \bibinfo{title}{Jnlp team: Deep learning approaches for legal
processing tasks in coliee 2021},
\newblock \bibinfo{journal}{arXiv preprint arXiv:2106.13405}
(\bibinfo{year}{2021}).
%Type = Article
\bibitem[{Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal,
Neelakantan, Shyam, Sastry, Askell et~al.}]{brown2020language}
\bibinfo{author}{T.~Brown}, \bibinfo{author}{B.~Mann},
\bibinfo{author}{N.~Ryder}, \bibinfo{author}{M.~Subbiah},
\bibinfo{author}{J.~D. Kaplan}, \bibinfo{author}{P.~Dhariwal},
\bibinfo{author}{A.~Neelakantan}, \bibinfo{author}{P.~Shyam},
\bibinfo{author}{G.~Sastry}, \bibinfo{author}{A.~Askell}, et~al.,
\newblock \bibinfo{title}{Language models are few-shot learners},
\newblock \bibinfo{journal}{Advances in neural information processing systems}
\bibinfo{volume}{33} (\bibinfo{year}{2020}) \bibinfo{pages}{1877--1901}.
\end{thebibliography}