VALUED -- Vision and Logical Understanding Evaluation Dataset
Extracted source text · arXiv 2311.12610
\documentclass[twoside,11pt, preprint]{article}
\usepackage{dmlr2e}
\input{chapters/math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{natbib}
\usepackage{subcaption}
\newcommand{\ctt}[1]{|\texttt{#1}|}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\ShortHeadings{VALUED}{Saha, Saha and Garain}
\firstpageno{1}
\title{VALUED - Vision and Logical Understanding Evaluation Dataset}
\author{\name Soumadeep Saha \email soumadeep.saha\_r@isical.ac.in \\
\addr Indian Statistical Institute\\
Kolkata, India
\AND
\name Saptarshi Saha \email saptarshi.saha\_r@isical.ac.in\\
\addr Indian Statistical Institute\\
Kolkata, India
\AND
\name Utpal Garain \email utpal@isical.ac.in\\
\addr Indian Statistical Institute\\
Kolkata, India}
\def\month{02}
\def\year{2024}
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
\begin{document}
\maketitle
%---------------------- ABSTRACT -----------------------------------------
\begin{abstract}
Starting with early successes in computer vision tasks, deep learning based techniques have since overtaken state of the art approaches in a multitude of domains.
However, it has been demonstrated time and again that these techniques fail to capture semantic context and logical constraints, instead often relying on spurious correlations to arrive at the answer.
Since application of deep learning techniques to critical scenarios are dependent on adherence to domain specific constraints, several attempts have been made to address this issue.
One limitation holding back a thorough exploration of this area, is a lack of suitable datasets which feature a rich set of rules.
In order to address this, we present the VALUE (Vision And Logical Understanding Evaluation) Dataset, consisting of 200,000$+$ annotated images and an associated rule set, based on the popular board game - chess.
The curated rule set considerably constrains the set of allowable predictions, and are designed to probe key semantic abilities like localization and enumeration.
Alongside standard metrics, additional metrics to measure performance with regards to logical consistency is presented.
We analyze several popular and state of the art vision models on this task, and show that, although their performance on standard metrics are laudable, they produce a plethora of incoherent results, indicating that this dataset presents a significant challenge for future works.
\end{abstract}
%---------------------- ABSTRACT -----------------------------------------
\begin{keywords}
logical constraints, domain knowledge, deep learning, computer vision
\end{keywords}
\input{chapters/intro.tex}
\input{chapters/background.tex}
\input{chapters/proposal.tex}
\input{chapters/methods.tex}
\input{chapters/baselines.tex}
\input{chapters/conclusions.tex}
\impact{Our work is intended to enhance exploration of the ability of deep learning based systems to adhere to domain constraints, which could lead to making these systems more deployable in key sectors like healthcare, robotics, etc. Thus our work, which tries to improve the understanding of the shortcomings of deep learning based systems, can potentially lead to these systems being deployed with more reliability and trust to several automated pipelines down the road. Thus, this has the same perils and advantages that enhanced automation brings to society. On one hand, it improves productivity and access and reduces costs. On the other hand, it helps in automating certain tasks that could render some jobs obsolete which has adverse consequences in societies without strong social safety nets.}
\acks{This research is partially supported by the Indo-French Centre for the Promotion of Advanced Research (IFCPAR/CEFIPRA) through Project no. 6702-2.
Additionally, the authors would like to thank Ms. Sutanoya Chakraborty for her help in reviewing and editing the manuscript.}
\vskip 0.2in
\bibliography{chapters/references}
%------------------------------------------------------------------------
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm, amssymb}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{Introduction}
\label{sec:intro}
Incorporating domain-knowledge has been highlighted as one of the ``3 Grand Challenges in developing AI systems'' by a recent report on AI for Science \citep{ai4sc} and has been studied with zeal across several domains.
Although domain-knowledge can refer to various kinds of information (relevant features, known rules, probability distributions, etc), following the survey by \citet{goodNature}, we restrict our discussion to \emph{problem specific prior knowledge, represented as logical or numeric constraints}.
These constraints take several forms depending on the domain in question, like semantic faithfulness \citep{akshay} or reasoning tasks \citep{bigbench} in natural language processing (NLP), diagnostic constraints \citep{mememe} in clinical settings, graph based constraints in functional genomics \citep{newgoa, deepgo} and general propositional logic rules in a multitude of settings \citep{medic2, eg1, eg2}.
For example, consider rules like - \emph{``a scene can contain at most 10 objects''} (numerical constraint) or \emph{``a scene cannot contain both objects of type A and B''} (logical constraint) which might serve as ``prior knowledge'' for a computer vision task.
Imbuing deep learning systems with the ability to adhere to these constraints would greatly improve their applicability to critical areas like robotics, healthcare, law, or material science.
Additionally, development of such techniques should in principle reduce reliance on massive datasets, which are especially difficult to annotate in these contexts \citep{datadifficult}.
To illustrate this problem, consider the example of a vision system employed to read credit card numbers in order to process payments.
Although, digit recognition is a well understood task, even 1\% misclassification in a standard 16 digit card number can result in 14.9\% of transactions failing.
This is a common issue, and logical constraints in the form of checksums\citep{creditcard} are employed to limit inaccuracies.
A model armed with this knowledge could avoid making erroneous predictions, or potentially even infer ambiguous digit.
Even commonplace logical inferences like - \emph{(A is inside B) and (B is on top of C) $\Rightarrow$ A is on top of C}, which we take for granted in everyday life, potentially might not be captured in deep learning based computer vision systems trained on large datasets \citep{clevr}, and might present challenges for downstream applications.
Today, with the proliferation of applications for deep learning based vision systems, there is a pressing need to evaluate logical understanding and further investigate possible avenues of inculcating them into trained models.
Since training deep neural networks is reliant on gradient based methods, and logical/numeric constraints are in general discrete, we run into our very first hurdle.
It is not a priori clear how standard techniques should be adapted to abide by these constraints, and due to its significant practical importance, this field has been the subject of extensive research \citep{dilp, ttt, lnn} (we refer the reader to surveys by \citet{besold} or \citet{rueden} for a comprehensive overview).
In spite of this, a standard framework for incorporating logical constraints into neural networks eludes us \citep{goodNature}.
\begin{figure*}[t]
\begin{center}
\includegraphics[width=\linewidth]{img/examples.png}
\end{center}
\caption{Example images from the VALUE Dataset (\emph{top row}) and corresponding expected outputs (\emph{bottom row}).}
\label{fig:examples}
\end{figure*}
One hindrance often encountered across the literature, is in regards to availability of large volumes of high quality annotated data with associated rules \citep{medicalDataLack, snakeoil, datadifficult}.
To study the acuity of vision systems in this context, we need to design a task, ideally demonstrating the following characteristics:
\begin{enumerate}
\item Poses a challenge for state of the art vision foundation models.
\item Contains non-trivial first-order logic rules that can be inferred from data.
\item Presents semantic constraints on localization to test visual reasoning and numerical constraints to test arithmetic reasoning.
\end{enumerate}
With these features in mind we land on the problem of chess board state recognition, i.e. given an image of a chess game in progress, reconstruct the state of the board.
Note that our primary interest is not on the problem of accurately identifying chess game states, as in \citet{chessP1} or \cite{chessP2}, but rather to analyze the shortcomings of state of the art vision systems in logical understanding and the efficacy of mitigation techniques.
We present the \textbf{VALUE Dataset - a collection of 200,000$+$ annotated images of chess games in progress} (see Figure \ref{fig:examples} for examples), with a \textbf{curated set of logical constraints}, and constraint adherence metrics designed to analyse performance of \textbf{domain-knowledge incorporation techniques}.
The rule set, containing several non-trivial first-order logic rules that are necessarily true for a certain prediction to be a valid chessboard state, also enables analysis of models with regards to \textbf{understanding of key semantic information} like location, count, etc.
This dataset, with the addition of relevant metrics like F1-score and exact match (EM\%) is intended to serve as a standard benchmark task, to analyze various algorithms and strategies developed to address logical coherence of deep learning systems.
Since data on several million real world chess games is readily available \citep{lichess}, and necessary materials to generate images is provided herewith, more data can be easily generated if necessary.
This dataset will aid in the exploration of several pressing questions like, (1) given enough data can deep learning systems trained with standard approaches learn to abide by underlying logic rules (2) how does quality/quantity of incorporated rules influence learning, and (3) how well does a particular technique perform in enforcing logical constraints.
Since the issue of logical incoherence is a hindrance to adoption of deep learning techniques to several key real world applications like robotics, automated diagnostics, etc., we present this dataset to the community with the hope that it promotes further research into the development of techniques to solve this problem.
\section{Background}
Domain-knowledge in the form of logical or numerical constraints is a common feature across several problems.
Although these constraints are sometimes continuous/differentiable \citep{pgnn1, snakeoil}, the discrete cases conflict with the reliance on gradient based optimization methods used in deep learning, thus presenting a more interesting challenge.
The methods explored to solve this problem usually involves transforming the input data \citep{cilp}, the loss function \citep{logicloss}, the model itself \citep{tmodelcomplain} or some combination of these \citep{combi}.
However, more work is needed to find a standard framework for incorporating domain knowledge constraints into deep learning based systems \citep{goodNature}.
One complaint frequently resonating in the literature throughout this area is the dearth of datasets \citep{datadifficult, tmodelcomplain, medicalDataLack}.
Specifically, annotated datasets featuring domain specific constraints are required to enable analysis and development of techniques.
Researchers rely on toy datasets \citep{ltn, lnn} or datasets containing only a few thousand examples \citep{pami}.
Often incorporation of domain-knowledge is sold as a cure to this problem, thus results are reported on extremely sparse datasets\citep{datadifficult, snakeoil}.
However, if demonstrations of techniques are limited to sparse datasets, the significance of improvements owing to intervention techniques remains unclear.
Domain knowledge in principle, should be able to alleviate performance bottlenecks in low resource problems, but it is by no means a settled question, and the effect of dataset size on the ability to learn domain knowledge needs further exploration.
Recently, some progress has been made in this regard in the NLP domain with the introduction of BIG-bench \citep{bigbench} (Beyond the Imitation Game benchmark) which introduces a large set of tasks that are constrained by logic, however, owing to the vast complexity of natural language, a comprehensive set of logical constraints covering a significant fraction of the domain is hard to come by.
Therefore, it does not address the need for a large dataset with a comprehensive set of rules constraining a large fraction of instances.
There have been some efforts in assessing the visual reasoning ability of deep learning based systems \citep{clevr, shapes, raven}.
However these datasets asses the models ability for abstract visual reasoning, and to the best of our knowledge there are no high quality datasets analyzing deductive reasoning ability based on a rich set of known domain specific axioms.
Another context where this problem has been widely studied is multi-label classification with logical constraints \citep{hmcnet, pami, graphbased}.
In these problems, a knowledge graph imposes entailment constraints that valid predictions must adhere to, but generally the set of admissible rules used are rather limited in scope.
For example, in the Blurb-genre dataset \citep{bd1} or the Uniprot protein function dataset \citep{bd2} or \citet{bd3, bd4,dost}, only rules of the form $\forall x, A(x) \Rightarrow B(x)$ is featured.
Richer rule sets that include constraints reliant on localization, counting, etc in addition to simpler first-order logic rules need to be explored.
Some large-scale datasets \citep{cinc, bd2} feature a long-tailed class distribution and extremely sparse classes, befogging analysis of domain-knowledge incorporation techniques.
To promote and foster further inquiry in this area, large-scale datasets featuring a diverse set of domain specific rules, and unfettered by unrelated confounding issues like sparsity, class imbalance, etc. are needed.
There have been previous attempts at creating datasets of images of chess games by \citet{chessP2} and \citet{chessP1}, however their focus was on creating accurate systems for game state recreation, and not analysis of logical understanding, and thus do not comprise of a rule set or associated metrics.
Although these datasets feature an order of magnitude fewer images ($\sim$ 5K, 10K respectively) these datasets can be subsumed into ours to increase diversity and further aid in probing logical understanding.
\subsection{Brief primer on chess}
Chess is a two-player board game dating back centuries, played on an 8$\times$8 grid with each player controlling 16 pieces of 6 distinct types.
Each distinct piece has constraints on their allowed movements (legal moves) \citep{chessrules} in the grid, and players take turns moving one piece at a time with the objective of putting the opponent in a position where the capture of their king is unavoidable (checkmate).
The two players' pieces are distinguished by their color (black or white), and a single character acronym is assigned to each piece, which are \texttt{k,q,r,b,n,p} for king, queen, rook, bishop, knight and pawn respectively (lower case characters refers to black pieces and upper case characters refers to white pieces).
Each file (column) of the chessboard (grid) is indexed with a letter in \{A, $\ldots$, H\} starting from the left, and each rank (row) is indexed by a number in $\{1, \ldots, 8\}$ starting from white's side (bottom leftmost dark square for each board, e.g. in Figure \ref{fig:examples}, is indexed A1).
The configuration of the entire board (board state) is often represented in a shorthand notation called Forsyth-Edwards Notation (FEN), which lists the pieces in each rank in sequence separated by a slash (``/''), with blank space counts interspersed in between.
For example, in Figure \ref{fig:examples} the FEN for the leftmost and rightmost boards are ``\texttt{r1bqk2r/ppppbN1p/2n2np1/4p3/2B1P3/3P4/PPP2PPP/RNBQK2R}''
\footnote{\texttt{r1bqk2r/}\dots represents a row with a black rook, followed by an empty square, followed by a black bishop, queen and king, followed by two empty squares and a black rook arranged from left to right. This is similarly repeated for all ranks (rows)}
and ``\texttt{8/8/2k3P1/8/5K2/6R1/5r2/8}'' respectively and the game starts from the state ``\texttt{rnbqkbnr/ pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR}''.
We also use $\ctt{L}$ to denote the number of pieces of type \texttt{L} on a board (e.g. $\ctt{k} = \ctt{K} = 1$).
\section{Task description}
If this were a standard classification task, we would seek to learn a function $f_{\theta}:\mathcal{D} \rightarrow [0,1)^{(\text{\# pieces + 1}) \times 8 \times 8}$ (prediction for piece type or empty for all 64 positions) that models the probability of each piece at every board position given an image of the board state.
However, when in use, this function must be paired with an inference algorithm with discrete outputs to recreate the board state, which can itself introduce logical inconsistencies.
Thus, in order to analyse performance with regards to logical coherence, the predictive model with soft outputs and the inference algorithm is treated together as a black-box, and performance is evaluated with respect to the complete board state prediction.
Keeping this in mind, we define a classifier as follows:
\begin{equation}
\begin{split}
F_{\theta}:\mathcal{D} &\rightarrow \mathrm{P}^{8 \times 8}\\
F_{\theta}(x) &\mapsto \text{board state of } x\\
\end{split}
\label{eq:classifier}
\end{equation}
\noindent where $\mathcal{D} \subset \mathbb{R}^{512\times 512\times 3}$ is the space of input images, $\mathrm{P} = \{\texttt{x, p, P, n, N, b, B, r, R,}$ $\texttt{q, Q, k, K}\}$ is the set of all pieces (\texttt{x} represents the empty grid location), and $F_{\theta}$ represents the model we seek, parametrized by $\theta$.
The images pose several difficulties from a computer vision perspective like camera position variability, occlusion (often severe), and dense clusters of similar looking small objects (see Figure \ref{fig:demo}).
Note that this problem can be rephrased as a semantic segmentation problem or a captioning task or potentially an entirely new paradigm of machine learning, which we don't analyze.
This is because, in the context in which we are analyzing the problem, one must imagine that our deep learning system is a part of a larger pipeline that performs some task in the real world, and the outputs from the model are further fed to an automated system.
It is in these sorts of applications that demonstration of logical understanding is most pressing.
In our sample task, if the model generated annotations were presented to a human with a few errors, they could readily correct those, however, if this model was part of an automated chess-playing robot, that also employes a chess engine to generate moves and software to control linkages, logical errors have a much more pronounced effect.
Thus we only analyze this as a classification problem, since any other paradigm would first require conversion to a standardised format identical to one defined in Equation \ref{eq:classifier}, before further processing.
\begin{figure*}[t]
\begin{center}
\includegraphics[width=0.9\linewidth]{img/demo.png}
\end{center}
\caption{The base 3D scene (\emph{left}) and dataset examples demonstrating occlusion (\emph{marked in red}), and object density (\emph{right}).}
\label{fig:demo}
\end{figure*}
\subsection{The rule set}
An arrangement of pieces on the chessboard is called \textbf{valid}, if there exists a sequence of legal moves \citep{chessrules} from the starting position that results in the arrangement and the set of all valid states is denoted $\mathcal{V}$.
Given the state of a chessboard, it is computationally prohibitive to determine if the state is valid, however, some simple \emph{sanity checks} can be utilized to rule out the vast majority of invalid states.
We curated such a set of computationally cheap ($\mathcal{O}(n d^2)$ for $n$ chessboards of side $d$ i.e. $\mathcal{O}(n)$) first-order logic rules that hold true for all valid chessboard states, to measure domain coherence.
The proposed rule set has the following rules (equivalent rules for white pieces are also included):
\begin{equation}
\begin{split}
\forall y &\in \mathcal{V} \text{ (valid states) we have, }\\
&\mathrm{i})\: \ctt{k} = 1 \quad(\text{Exactly one king.})\\
&\mathrm{ii})\: \texttt{k}, \texttt{K} \text{ are not on adjacent squares.}\\
&\mathrm{iii})\: \ctt{p} + \ctt{q} + \ctt{n} + \ctt{b} + \ctt{r} \leq 15\quad(\text{Total number of pieces including king cannot exceed 16.})\\
&\mathrm{iv})\: \ctt{p} \leq 8\quad(\text{Total number of pawns not exceeding 8.})\\
&\mathrm{v})\: \forall \texttt{p}, 2 \leq rank(\texttt{p}) \leq 7 \quad(\text{No pawn on first or last rank.})\\
&\mathrm{vi})\: \ctt{p} = 8 \Rightarrow \Big( \ctt{q} \leq 1 \wedge \ctt{b} \leq 2 \wedge \ctt{n} \leq 2\wedge \ctt{r}\leq 2 \Big)\\
&\quad(\text{If no pawn is promoted, there cannot be more than two bishops, knights, rooks or more than one queen.})\\
&\mathrm{vii})\: \ctt{p} < 8 \Rightarrow \max(0,\ctt{q}-1) + \max(0,\ctt{b}-2) + \max(0,\ctt{n}-2) + \max(0,\ctt{r}-2) \leq 8 - \ctt{p}\\
&\quad(\text{If pawns might have been promoted, number of excess pieces cannot exceed number of missing pawns.})\\
&\mathrm{viii})\: (\ctt{p} = 8) \wedge (\ctt{b} = 2) \Rightarrow \qquad \texttt{b}_1, \texttt{b}_2 \text{ don't occupy squares of same color}\\
&\quad(\text{If no pawn has been promoted and there are two bishops, they must be on opposite color squares.})\\
\end{split}
\label{eq:rules}
\end{equation}
If a prediction $y$ \textbf{satisfies all these rules} we call it \textbf{sane}, and we have the \textbf{set of sane states} $\mathcal{S}$ following $ \mathcal{V} \subsetneq \mathcal{S} \subsetneq \mathrm{P}^{64}$.
These rules (Equation \ref{eq:rules}) are further divided into two categories - \textbf{counting} ($\mathrm{i, iii, iv, vi, vii}$), and \textbf{localizing} ($\mathrm{ii, v, viii}$), to analyze specific semantic abilities.
Counting rules apply constraints on number of objects that can be present in the scene, whereas localizing rules apply constraints on their position in the scene.
This is not an exhaustive list, and more such rules can be found (e.g. \texttt{b} can't be trapped in the last rank behind 3 adjacent \texttt{p}, etc.), but these are nonetheless effective at eliminating a large fraction of invalid states.
Note that the total number of distinct predictions we would have following standard deep learning approaches (64 independent classification problems) is $|\mathrm{P}^{64}| = 13^{8\times 8} \sim 10^{72}$, but with the addition of just the constraint on number of pieces, it reduces to $< 10^{55}$.
If these simple rules can be incorporated into the learning algorithm, in addition to being more applicable to the domain, it could in principle improve performance drastically.
For example, if \texttt{q} was misclassified as \texttt{k}, or \texttt{b} was misclassified as \texttt{p} resulting in $|\texttt{p}|=9$, the rule set would identify and seek to disincentivize it.
\subsection{Evaluation Metrics}
We adapt a few standard metrics to evaluate raw performance as well as domain alignment.
Since the logical constraints work with the discretized complete board state prediction, and per our definition in Equation \ref{eq:classifier} ``threshold tuning'' or related inference methods are considered part of the model, some popular metrics like AUPRC, AUROC or macro-averaged metrics like mAP are not discussed.
Given a prediction set $\hat{Y} = \{\hat{y}_i| i \in \{1, \ldots, n\}\}$ and the corresponding ground truth set $Y = \{y_i\}$,\: $y_i, \hat{y}_i \in \mathrm{P}^{64}$, we define exact match (EM) and F1 as follows.
\begin{equation}
EM(Y, \hat{Y}) = \frac{1}{n} \sum_{i=1}^n \mathbb{I}([y_{i}]_k = [\hat{y}_{i}]_k, \: \forall k, 1 \leq k \leq 64)
\end{equation}
\begin{equation}
\begin{split}
F1(Y, \hat{Y}) &= \frac{2}{n} \sum_{i=1}^n f_1(y_i, \hat{y}_i), \text{ where}\\
f_1(y_i, \hat{y}_i) &= \frac{\sum_{j=1}^{64} \mathbb{I}([y_{i}]_j = [\hat{y}_{i}]_j \neq \texttt{x})}{|y_i| + |\hat{y}_i|}
\label{eq:f1}
\end{split}
\end{equation}
Where $|y| = \sum_j \mathbb{I}([y]_j \neq \texttt{x})$ denotes the number of non-empty squares in the grid, and $\mathbb{I}$ is the indicator function that takes the value 1 if its argument condition is true, and 0 otherwise.
Additionally, we define two measures of domain coherence -- \textbf{contradiction \%} (C) and \textbf{sane F1} (sF1) as follows.
\begin{equation}
C(Y) = \frac{100}{n} \sum_{i=1}^n \mathbb{I}(\hat{y}_{i} \not\in \mathcal{S})
\end{equation}
\begin{equation}
sF1(Y, \hat{Y}) = \frac{2}{n} \sum_{i=1}^n \Big( \mathbb{I}(\hat{y}_{i} \in \mathcal{S})\cdot f_1(y_i, \hat{y}_i) \Big)
\end{equation}
C reflects the frequency of logical constraint violations i.e. what fraction of predictions are unusable, and sF1 score measures the F1 score after eliminating predictions that are not sane.
We also report results on $\mu_C$, the mean number of rule violations per instance in $\hat{Y}$.
\begin{equation}
\mu_C(\hat{Y}) = \frac{1}{n} \sum_{i=1}^n \Big( \text{\# of rule violations in } \hat{y}_i \Big)
\end{equation}
When the model is part of an automated system with downstream applications mandating adherence with domain rules, offending predictions must be suppressed with consistency checks against the rule set.
The sF1 measure considers this counterfactual scenario, where these rule violating predictions were replaced with a null prediction set.
The difference between F1 and sF1 establishes a floor for the room for improvement available to algorithms seeking to incorporate logical constraints.
To summarize, in this task, we seek an $F_{\theta}$, such that given a set of images $x \in \mathcal{D}$, it can faithfully recreate corresponding ground truth labels $y \in \mathrm{P}^{64}$, while minimizing violations of domain rules, i.e. $sF1\Big(\{F_{\theta}(x_i)\}, \{y_i\}\Big)$ is maximized.
\section{Methods}
To create the dataset \footnote{Dataset - \url{https://doi.org/10.5281/zenodo.10607059}}, we first procured a large database of chess moves in online multi-player games from lichess \citep{lichess} and converted them to board states.
Although this database of board states contains duplicates (e.g. common openings), we do not remove them in order to preserve the distribution of states that are likely to be encountered.
A 3D model of the world consisting of a large plane with a table, two chairs, a chess board and pieces in starting position (see Figure \ref{fig:demo}) was created in the open-source modelling software blender \citep{blender}.
The board state database was then used to move the pieces to their designated squares (with some noise).
A camera was added to the scene at a fixed distance from the center of the chessboard, constrained to point to the center, with randomized pan and tilt.
The camera motions are limited to ensure that the chessboard corner A1 is closest to the camera to avoid ambiguity with respect to rotations of the board.
200,000 such images were rendered at $(512 \times 512 \times 3)$ to form the training/validation set and an additional 19,967 images form the test set.
In addition to the labels (in array form and FEN), bounding box information for every piece is provided to aid other techniques (object detection, etc.) but these are not used for evaluation.
A concise rule set was created to strike a balance between imposing sufficient constraints and computationally efficient checking.
Furthermore, the rule set implementation provides analysis on the type of errors being made by the prediction system.
To establish baseline results we selected a range of popular ImageNet \citep{imagenet} pre-trained vision models like ResNets \citep{resnet}, ViT \citep{vit}, etc. covering a large range of scales and training techniques.
These pre-trained models were fine-tuned for 2 epochs after replacing the final fully connected layer with one of size $(\texttt{in\_features} \rightarrow 8 \times 8 \times \texttt{class\_number})$ and adding dropouts(10\%) in between.
The models were implemented in pytorch, and trained with the AdamW optimizer (learning rate of $10^{-4}$) and Cross Entropy loss.
The images were normalized, and resized if necessary.
We trained the models on a single NVIDIA A6000 48GB GPU.
All associated code (database creation, rule checking, etc.), materials (dataset, 3D models, textures, images, etc.), rendering details (camera sensor, rendering settings, etc.) and relevant information has been made available through the github repository.\footnote{Code repository \url{https://github.com/espressoVi/VALUE-Dataset}}
\section{Baseline results and discussions}
Performance analysis results of various vision models on our dataset is summarized in Table \ref{tab:res}.
Although these pre-trained models feature a range of training techniques and scales, they are quite adept at recognizing relevant visual features in order to recognize chess pieces as exemplified by their high F1 and EM scores (see Table \ref{tab:res}).
Notably, the Swin Transformer \citep{swin} outperforms models which are an order of magnitude larger, perhaps because of its ability to capture features at different scales owing to its standout hierarchical architecture.
Still, these models leave a lot to be desired from a domain consistency point of view, as seen by the large percentage of predictions that have rule violations.
In critical applications, these incoherent predictions aren't viable, and would have to be discarded, which is reflected in the significantly reduced sF1 scores.
To further emphasise the point we direct the readers' attention to $\mu_C$ in Table \ref{tab:res}, which records the mean number of rule violations per prediction, where, in the best case scenario we encounter a rule violation every 15 predictions and every 2.3 predictions in the worst case.
Although we reported results with off the shelf vision systems, we note that the performances of systems designed for the specific chess board state recognition task are also comparable, with \emph{EM} scores of $\sim$15\% and $\sim$ 7\% for an end to end and piece-wise classification system respectively \citep{chessP1}.
\begin{table*}[hbt!]
\caption{Performance of popular vision models on VALUE dataset.\\
($[\uparrow]$ - higher is better, $[\downarrow]$ lower is better). (1) - \citet{vgg} (2) - \citet{resnet} (3) - \citet{vit} (4) - \citet{swin}}
\begin{center}
{\small{
\label{tab:res}
\begin{tabular}{p{7em}|rr|rrrrrr}
\toprule
Model & \# param. & Image Size & EM (\%)$[\uparrow]$ & C (\%) $[\downarrow]$ & F1 $[\uparrow]$& sF1 $[\uparrow]$& F1-sF1 $[\downarrow]$ & $\mu_c [\downarrow]$\\
\midrule
VGG-16 (1) & 134M & $224^2$ & 26.3\% & 28.7\% & 0.880 & 0.656 & \textbf{0.224} & 0.426 \\
ResNet50 (2) & 24M & $512^2$ & 56.3\% & 12.8\% & 0.959 & 0.849 & \textbf{0.110} & 0.172 \\
ResNet101 (2) & 43M & $512^2$ & 60.4\% & 11.1\% & 0.966 & 0.869 & \textbf{0.097} & 0.147 \\
ViT-B/16 (3) & 86M & $224^2$ & 25.9\% & 30.8\% & 0.875 & 0.635 & \textbf{0.240} & 0.432 \\
ViT-L/32 (3) & 307M & $384^2$ & 32.2\% & 24.0\% & 0.907 & 0.711 & \textbf{0.196} & 0.337 \\
SWIN-tiny/4 (4) & 29M & $224^2$ & 80.3\% & 5.2\% & 0.984 & 0.938 & \textbf{0.046} & 0.067 \\
\bottomrule
\end{tabular}
}}
\end{center}
\end{table*}
%\begin{figure}
%\begin{center}
% \includegraphics[width=\linewidth]{./img/train.pdf}
% \caption{Training losses for models used.}
%\end{center}
%\label{fig:train}
%\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\includegraphics[width=\textwidth]{./img/prop.png}
\caption{Prevalence of violating counting or localizing rules.}
\label{fig:prop}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.48\textwidth}
\includegraphics[width=\textwidth]{./img/prop_adj.png}
\caption{Adjust likelihood of violating counting or localizing rules (Equation \ref{eq:adj}).}
\label{fig:cnt}
\end{subfigure}
\begin{subfigure}[t]{0.6\textwidth}
\includegraphics[width=\linewidth]{./img/rule.png}
\caption{Adjusted likelihood of violating each rule.}
\label{fig:loc}
\end{subfigure}
\caption{Errors arising due to counting or localizing.}
\label{fig:figstack}
\end{figure}
%\begin{figure}[hbt!]
%\begin{center}
% \includegraphics[width=\linewidth]{./img/prop.png}
% \caption{Errors arising due to counting or localizing.}
%\end{center}
%\label{fig:prop}
%\end{figure}
Since our rule set features various rules designed to test localization ability, i.e. constraints on where objects can be located in the scene, and counting ability, i.e. constraints on how many objects there can be in the scene, we analyzed model performance in this regard.
Although the vast majority of errors across all models arise due to violation of counting constraints (see Figure \ref{fig:prop}), note that the likelihood of making counting errors is higher owing to the much larger number of possible predictions where such a constraint is violated.
To adjust for this we compute-
\begin{equation}
P^{\mathrm{R}}_{model} = f^{\mathrm{R}}_{model}/f^{\mathrm{R}}_{random}
\label{eq:adj}
\end{equation}
where $f^{\mathrm{R}}_{model}$ is the frequency of rule $\mathrm{R}$ being violated by the model in question, and $f^{\mathrm{R}}_{random}$ is the frequency of rule violation of a random guesser ($P(s) = P(s'), \forall s,s' \in \mathrm{P}^{64}$).
This shows (see Figure \ref{fig:cnt}) that the models are more likely to make errors in regards to localization ($\mu = 0.0052\pm0.0025$) than counting ($\mu=0.0042\pm0.0023$).
%\begin{figure}
%\begin{center}
% \includegraphics[width=\linewidth]{./img/prop_adj.png}
% \caption{Adjust likelihood of violating counting or localizing rules (Equation \ref{eq:adj}).}
%\end{center}
%\label{fig:cnt}
%\end{figure}
%\begin{figure}
%\begin{center}
% \includegraphics[width=\linewidth]{./img/rule.png}
% \caption{Adjusted likelihood of violating each rule.}
%\end{center}
%\label{fig:loc}
%\end{figure}
Further, we probed the models to check for the likelihood of violating each rule under the two categories.
To our surprise, rule $(\mathrm{v})$ in Equation \ref{eq:rules}, was never violated by any of the models, and rules $(\mathrm{iii}),(\mathrm{vii})$ are extremely unlikely to be violated (see Figure \ref{fig:loc}).
On the other hand, locality constraints $(\mathrm{ii}),(\mathrm{viii})$, and counting rules $(\mathrm{i}),(\mathrm{iv}),(\mathrm{vi})$ are extremely likely to be violated.
This points to the fact that the ability of models to adhere to domain constraints is somewhat dependent on the nature of the constraints themselves.
To summarize, we have demonstrated that although standard deep learning approaches appear to perform well if gauged by standard metrics on the VALUE Dataset (see Table \ref{tab:res}), they fall short when it comes to domain constraint adherence.
Even in the best performing models, $5.2\%$ of predictions (up to $30\%$ in the worst case) contain logical inconsistencies which would be unacceptable in critical applications and leads to the lowered effective F1 score (sF1).
Furthermore, we showed that standard deep learning approaches demonstrate limited capacity to learn constraint information from the data itself, but that is largely dependent on the type of constraint.
Further investigation is needed to explore this relationship between the types of constraints and ease of adherence.
It is also unclear whether the improved performance of some models (Table \ref{tab:res}) is owing to their ability to learn representations that are aligned with the constraints from the dataset or are meretricious.
Additionally, it would be compelling to explore if we can evaluate the generated counterfactual in a deep SCM framework \citep{scm1, saptarshi} using the rules presented in the paper.
In other words, while the axiomatic soundness of counterfactual image models were tested \citep{sapsuggest}, there is scope to test the logical soundness of these models.
Model robustness also warrants further inspection especially since incorporating domain-knowledge has shown initial promise in improving adversarial robustness \citep{pami}.
\section{Conclusions}
Domain knowledge in the form of logical or numerical constraints arise in many critical applications, and it is not clear how these rules are to be incorporated into deep learning systems.
Owing to the great practical relevance, this area has been widely studied, but the lack of high quality datasets featuring a diverse curated set of rules has hindered thorough analyses and development of techniques.
To address this we introduced the VALUE Dataset, which consists of 200,000+ images of chess games in progress, with the task of recreating the state of the chess board.
Additionally, we curated a set of simple and computationally efficient sanity-check rules to ensure that the predictions are consistent with the rules of chess.
In addition to standard performance metrics, suitable metrics were devised to analyze performance of deep learning systems in regard to domain constraint obedience.
We explored the performance of several popular and state-of-the art vision models to show that although they appear reasonably accurate, they produce a plethora of inconsistencies, indicating that these systems lack the ability to learn underlying semantic structure from data alone.
We further utilized our rule set to highlight key abilities and weaknesses of current approaches in deep learning.
Further research is needed to develop a standard framework for incorporating domain constraints into deep learning systems.
We have made all necessary materials publicly available to aid future work on developing techniques to address this pressing challenge.
\begin{thebibliography}{50}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Agrawal et~al.(2013)Agrawal, Gupta, Prabhu, and Varma]{eg1}
Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma.
\newblock Multi-label learning with millions of labels: Recommending advertiser
bid phrases for web pages.
\newblock In \emph{Proceedings of the 22nd International Conference on World
Wide Web}, WWW '13, page 13–24, New York, NY, USA, 2013. Association for
Computing Machinery.
\newblock ISBN 9781450320351.
\newblock \doi{10.1145/2488388.2488391}.
\newblock URL \url{https://doi.org/10.1145/2488388.2488391}.
\bibitem[Alday et~al.(2020)Alday, Gu, Shah, Robichaux, Wong, Liu, Liu, Rad,
Elola, Seyedi, Li, Sharma, Clifford, and Reyna]{cinc}
Erick A~Perez Alday, Annie~(Ping) Gu, Adnan~Shami Shah, Chad Robichaux,
An-Kwok~Ian Wong, C.~Liu, F.~Liu, Ali~Bahrami Rad, Andoni Elola, Salman
Seyedi, Q~Li, Anuj Sharma, Gari~D. Clifford, and Matthew~A. Reyna.
\newblock Classification of 12-lead ecgs: the physionet/computing in cardiology
challenge 2020.
\newblock \emph{Physiological Measurement}, 41, 2020.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:221115394}.
\bibitem[Aly et~al.(2019)Aly, Remus, and Biemann]{bd1}
Rami Aly, Steffen Remus, and Chris Biemann.
\newblock Hierarchical multi-label classification of text with capsule
networks.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics: Student Research Workshop}, pages 323--330,
Florence, Italy, July 2019. Association for Computational Linguistics.
\newblock URL \url{https://www.aclweb.org/anthology/P19-2045}.
\bibitem[Andreas et~al.(2016)Andreas, Rohrbach, Darrell, and Klein]{shapes}
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein.
\newblock Neural module networks.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, June 2016.
\bibitem[Besold et~al.(2017)Besold, d'Avila Garcez, Bader, Bowman, Domingos,
Hitzler, K{\"u}hnberger, Lamb, Lowd, Lima, de~Penning, Pinkas, Poon, and
Zaverucha]{besold}
Tarek~R. Besold, Artur~S. d'Avila Garcez, Sebastian Bader, Howard Bowman,
Pedro~M. Domingos, Pascal Hitzler, Kai-Uwe K{\"u}hnberger, L.~Lamb, Daniel
Lowd, Priscila Machado~Vieira Lima, Leo de~Penning, Gadi Pinkas, Hoifung
Poon, and Gerson Zaverucha.
\newblock Neural-symbolic learning and reasoning: A survey and interpretation.
\newblock In \emph{Neuro-Symbolic Artificial Intelligence}, 2017.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:1755720}.
\bibitem[Chaturvedi et~al.(2022)Chaturvedi, Bhar, Saha, Garain, and
Asher]{akshay}
Akshay Chaturvedi, Swarnadeep Bhar, Soumadeep Saha, Utpal Garain, and Nicholas
Asher.
\newblock Analyzing semantic faithfulness of language models via input
intervention on conversational question answering, 2022.
\bibitem[Community(2018)]{blender}
Blender~Online Community.
\newblock \emph{Blender - a 3D modelling and rendering package}.
\newblock Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
\newblock URL \url{http://www.blender.org}.
\bibitem[Dash et~al.(2022)Dash, Chitlangia, Ahuja, and Srinivasan]{goodNature}
Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan.
\newblock A review of some techniques for inclusion of domain-knowledge into
deep neural networks.
\newblock \emph{Scientific Reports}, 12\penalty0 (1):\penalty0 1040, Jan 2022.
\newblock ISSN 2045-2322.
\newblock \doi{10.1038/s41598-021-04590-0}.
\newblock URL \url{https://doi.org/10.1038/s41598-021-04590-0}.
\bibitem[Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]{imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li~Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock In \emph{2009 IEEE conference on computer vision and pattern
recognition}, pages 248--255. Ieee, 2009.
\bibitem[Dimmer et~al.(2011)Dimmer, Huntley, Alam-Faruque, Sawford, O'Donovan,
Martin, Bely, Browne, Mun~Chan, Eberhardt, Gardner, Laiho, Legge, Magrane,
Pichler, Poggioli, Sehra, Auchincloss, Axelsen, Blatter, Boutet,
Braconi-Quintaje, Breuza, Bridge, Coudert, Estreicher, Famiglietti,
Ferro-Rojas, Feuermann, Gos, Gruaz-Gumowski, Hinz, Hulo, James, Jimenez,
Jungo, Keller, Lemercier, Lieberherr, Masson, Moinat, Pedruzzi, Poux,
Rivoire, Roechert, Schneider, Stutz, Sundaram, Tognolli, Bougueleret,
Argoud-Puy, Cusin, Duek-Roggli, Xenarios, and Apweiler]{bd2}
Emily~C Dimmer, Rachael~P Huntley, Yasmin Alam-Faruque, Tony Sawford, Claire
O'Donovan, Maria~J Martin, Benoit Bely, Paul Browne, Wei Mun~Chan, Ruth
Eberhardt, Michael Gardner, Kati Laiho, Duncan Legge, Michele Magrane,
Klemens Pichler, Diego Poggioli, Harminder Sehra, Andrea Auchincloss,
Kristian Axelsen, Marie-Claude Blatter, Emmanuel Boutet, Silvia
Braconi-Quintaje, Lionel Breuza, Alan Bridge, Elizabeth Coudert, Anne
Estreicher, Livia Famiglietti, Serenella Ferro-Rojas, Marc Feuermann, Arnaud
Gos, Nadine Gruaz-Gumowski, Ursula Hinz, Chantal Hulo, Janet James, Silvia
Jimenez, Florence Jungo, Guillaume Keller, Phillippe Lemercier, Damien
Lieberherr, Patrick Masson, Madelaine Moinat, Ivo Pedruzzi, Sylvain Poux,
Catherine Rivoire, Bernd Roechert, Michael Schneider, Andre Stutz, Shyamala
Sundaram, Michael Tognolli, Lydie Bougueleret, Ghislaine Argoud-Puy, Isabelle
Cusin, Paula Duek-Roggli, Ioannis Xenarios, and Rolf Apweiler.
\newblock The {UniProt-GO} annotation database in 2011.
\newblock \emph{Nucleic Acids Res}, 40\penalty0 (Database issue):\penalty0
D565--70, nov 2011.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn,
Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and
Houlsby]{vit}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at
scale.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\newblock URL \url{https://openreview.net/forum?id=YicbFdNTTy}.
\bibitem[Du et~al.(2020)Du, Sun, Wang, Qi, and Liao]{tmodelcomplain}
Chunning Du, Haifeng Sun, Jingyu Wang, Qi~Qi, and Jianxin Liao.
\newblock Adversarial and domain-aware {BERT} for cross-domain sentiment
analysis.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 4019--4028. Association for
Computational Linguistics, jul 2020.
\newblock \doi{10.18653/v1/2020.acl-main.370}.
\newblock URL \url{https://aclanthology.org/2020.acl-main.370}.
\bibitem[Evans and Grefenstette(2018)]{dilp}
Richard Evans and Edward Grefenstette.
\newblock Learning explanatory rules from noisy data.
\newblock \emph{J. Artif. Int. Res.}, 61\penalty0 (1):\penalty0 1–64, jan
2018.
\newblock ISSN 1076-9757.
\bibitem[FIDE()]{chessrules}
FIDE.
\newblock International chess federation handbook.
\newblock \url{https://handbook.fide.com/chapter/E012023}.
\bibitem[Fran{\c{c}}a et~al.(2014)Fran{\c{c}}a, Zaverucha, and
d'Avila Garcez]{cilp}
Manoel V.~M. Fran{\c{c}}a, Gerson Zaverucha, and Artur~S. d'Avila Garcez.
\newblock Fast relational learning using bottom clause propositionalization
with artificial neural networks.
\newblock \emph{Machine Learning}, 94\penalty0 (1):\penalty0 81--104, Jan 2014.
\newblock ISSN 1573-0565.
\newblock \doi{10.1007/s10994-013-5392-1}.
\newblock URL \url{https://doi.org/10.1007/s10994-013-5392-1}.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{resnet}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, pages 770--778, 2016.
\newblock \doi{10.1109/CVPR.2016.90}.
\bibitem[Hu et~al.(2016{\natexlab{a}})Hu, Ma, Liu, Hovy, and Xing]{ttt}
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing.
\newblock Harnessing deep neural networks with logic rules.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 2410--2420,
Berlin, Germany, August 2016{\natexlab{a}}. Association for Computational
Linguistics.
\newblock \doi{10.18653/v1/P16-1228}.
\newblock URL \url{https://aclanthology.org/P16-1228}.
\bibitem[Hu et~al.(2016{\natexlab{b}})Hu, Ma, Liu, Hovy, and Xing]{combi}
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard~H. Hovy, and Eric~P. Xing.
\newblock Harnessing deep neural networks with logic rules.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics, {ACL} 2016, August 7-12, 2016, Berlin,
Germany, Volume 1: Long Papers}. The Association for Computer Linguistics,
2016{\natexlab{b}}.
\newblock \doi{10.18653/v1/p16-1228}.
\newblock URL \url{https://doi.org/10.18653/v1/p16-1228}.
\bibitem[Huang et~al.(2023)Huang, Chen, Liu, Xiong, Huang, Tong, and
Zhang]{hmcnet}
Wei Huang, Enhong Chen, Qi~Liu, Hui Xiong, Zhenya Huang, Shiwei Tong, and Dan
Zhang.
\newblock Hmcnet: {A} general approach for hierarchical multi-label
classification.
\newblock \emph{{IEEE} Trans. Knowl. Data Eng.}, 35\penalty0 (9):\penalty0
8713--8728, 2023.
\newblock \doi{10.1109/TKDE.2022.3207511}.
\newblock URL \url{https://doi.org/10.1109/TKDE.2022.3207511}.
\bibitem[ISO/IEC 7812-1:2017()]{creditcard}
ISO/IEC 7812-1:2017.
\newblock Identification cards — identification of issuers — part 1:
Numbering system.
\newblock Standard, International Organization for Standardization, Geneva, CH,
Jan 2017.
\bibitem[Johnson et~al.(2017)Johnson, Hariharan, van~der Maaten, Fei-Fei,
Zitnick, and Girshick]{clevr}
Justin Johnson, Bharath Hariharan, Laurens van~der Maaten, Li~Fei-Fei,
C.~Lawrence Zitnick, and Ross Girshick.
\newblock Clevr: A diagnostic dataset for compositional language and elementary
visual reasoning.
\newblock In \emph{2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, pages 1988--1997, 2017.
\newblock \doi{10.1109/CVPR.2017.215}.
\bibitem[Karpatne et~al.(2017)Karpatne, Watkins, Read, and Kumar]{pgnn1}
Anuj Karpatne, William Watkins, Jordan~S. Read, and Vipin Kumar.
\newblock Physics-guided neural networks {(PGNN):} an application in lake
temperature modeling.
\newblock \emph{CoRR}, abs/1710.11431, 2017.
\newblock URL \url{http://arxiv.org/abs/1710.11431}.
\bibitem[Kowsari et~al.(2020)Kowsari, Sali, Ehsan, Adorno, Ali, Moore, Amadi,
Kelly, Syed, and Brown]{medic2}
Kamran Kowsari, Rasoul Sali, Lubaina Ehsan, William Adorno, Asad Ali, Sean
Moore, Beatrice Amadi, Paul Kelly, Sana Syed, and Donald Brown.
\newblock Hmic: Hierarchical medical image classification, a deep learning
approach.
\newblock \emph{Information}, 11\penalty0 (6), 2020.
\newblock ISSN 2078-2489.
\newblock \doi{10.3390/info11060318}.
\newblock URL \url{https://www.mdpi.com/2078-2489/11/6/318}.
\bibitem[Kulmanov and Hoehndorf(2022)]{graphbased}
Maxat Kulmanov and Robert Hoehndorf.
\newblock {DeepGOZero: improving protein function prediction from sequence and
zero-shot learning based on ontology axioms}.
\newblock \emph{Bioinformatics}, 38:\penalty0 i238--i245, 06 2022.
\newblock ISSN 1367-4803.
\newblock \doi{10.1093/bioinformatics/btac256}.
\newblock URL \url{https://doi.org/10.1093/bioinformatics/btac256}.
\bibitem[Kulmanov et~al.(2017)Kulmanov, Khan, and Hoehndorf]{deepgo}
Maxat Kulmanov, Mohammed~Asif Khan, and Robert Hoehndorf.
\newblock {DeepGO: predicting protein functions from sequence and interactions
using a deep ontology-aware classifier}.
\newblock \emph{Bioinformatics}, 34\penalty0 (4):\penalty0 660--668, 10 2017.
\newblock ISSN 1367-4803.
\newblock \doi{10.1093/bioinformatics/btx624}.
\newblock URL \url{https://doi.org/10.1093/bioinformatics/btx624}.
\bibitem[Liu et~al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo]{swin}
Ze~Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
\newblock Swin transformer: Hierarchical vision transformer using shifted
windows.
\newblock \emph{CoRR}, abs/2103.14030, 2021.
\newblock URL \url{https://arxiv.org/abs/2103.14030}.
\bibitem[Masouris and van Gemert(2023)]{chessP1}
Athanasios Masouris and Jan van Gemert.
\newblock End-to-end chess recognition, 2023.
\bibitem[Mayne and Perry(2009)]{eg2}
Andrew Mayne and Russell Perry.
\newblock Hierarchically classifying documents with multiple labels.
\newblock In \emph{2009 IEEE Symposium on Computational Intelligence and Data
Mining}, pages 133--139, 2009.
\newblock \doi{10.1109/CIDM.2009.4938640}.
\bibitem[Melacci et~al.(2022)Melacci, Ciravegna, Sotgiu, Demontis, Biggio,
Gori, and Roli]{pami}
Stefano Melacci, Gabriele Ciravegna, Angelo Sotgiu, Ambra Demontis, Battista
Biggio, Marco Gori, and Fabio Roli.
\newblock Domain knowledge alleviates adversarial attacks in multi-label
classifiers.
\newblock \emph{{IEEE} Trans. Pattern Analysis and Machine Intelligence},
44\penalty0 (12):\penalty0 9944--9959, 2022.
\newblock \doi{10.1109/TPAMI.2021.3137564}.
\newblock URL \url{https://doi.org/10.1109/TPAMI.2021.3137564}.
\bibitem[Monteiro et~al.(2023)Monteiro, Ribeiro, Pawlowski, Castro, and
Glocker]{sapsuggest}
Miguel Monteiro, Fabio De~Sousa Ribeiro, Nick Pawlowski, Daniel~C. Castro, and
Ben Glocker.
\newblock Measuring axiomatic soundness of counterfactual image models.
\newblock In \emph{The Eleventh International Conference on Learning
Representations}, 2023.
\newblock URL \url{https://openreview.net/forum?id=lZOUQQvwI3q}.
\bibitem[Muralidhar et~al.(2018)Muralidhar, Islam, Marwah, Karpatne, and
Ramakrishnan]{snakeoil}
Nikhil Muralidhar, Mohammad~Raihanul Islam, Manish Marwah, Anuj Karpatne, and
Naren Ramakrishnan.
\newblock Incorporating prior domain knowledge into deep neural networks.
\newblock In \emph{2018 IEEE International Conference on Big Data (Big Data)},
pages 36--45, 2018.
\newblock \doi{10.1109/BigData.2018.8621955}.
\bibitem[Org.()]{lichess}
Lichess Org.
\newblock Lichess open database.
\newblock \url{https://database.lichess.org/}.
\bibitem[Pawlowski et~al.(2020)Pawlowski, Castro, and Glocker]{scm1}
Nick Pawlowski, Daniel~C. Castro, and Ben Glocker.
\newblock Deep structural causal models for tractable counterfactual inference.
\newblock In \emph{Advances in Neural Information Processing Systems}, 2020.
\bibitem[Riegel et~al.(2020)Riegel, Gray, Luus, Khan, Makondo, Akhalwaya, Qian,
Fagin, Barahona, Sharma, Ikbal, Karanam, Neelam, Likhyani, and
Srivastava]{lnn}
Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo,
Ismail~Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit
Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and
Santosh Srivastava.
\newblock Logical neural networks, 2020.
\bibitem[Ruepp et~al.(2004)Ruepp, Zollner, Maier, Albermann, Hani, Mokrejs,
Tetko, G{\"u}ldener, Mannhaupt, M{\"u}nsterk{\"o}tter, and Mewes]{bd3}
Andreas Ruepp, Alfred Zollner, Dieter Maier, Kaj Albermann, Jean Hani, Martin
Mokrejs, Igor~V. Tetko, Ulrich G{\"u}ldener, Gertrud Mannhaupt, Martin
M{\"u}nsterk{\"o}tter, and Hans-Werner Mewes.
\newblock The funcat, a functional annotation scheme for systematic
classification of proteins from whole genomes.
\newblock \emph{Nucleic acids research}, 32 18:\penalty0 5539--45, 2004.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:7022860}.
\bibitem[Saha and Garain(2022)]{saptarshi}
Saptarshi Saha and Utpal Garain.
\newblock On noise abduction for answering counterfactual queries: A practical
outlook.
\newblock \emph{Transactions on Machine Learning Research}, 2022.
\newblock ISSN 2835-8856.
\newblock URL \url{https://openreview.net/forum?id=4FU8Jz1Oyj}.
\bibitem[Saha et~al.(2023{\natexlab{a}})Saha, Garain, Ukil, Pal, and
Khandelwal]{dost}
Soumadeep Saha, Utpal Garain, Arijit Ukil, Arpan Pal, and Sundeep Khandelwal.
\newblock Dost -- domain obedient self-supervised training for multi label
classification with noisy labels, 2023{\natexlab{a}}.
\bibitem[Saha et~al.(2023{\natexlab{b}})Saha, Garain, Ukil, Pal, and
Khandelwal]{mememe}
Soumadeep Saha, Utpal Garain, Arijit Ukil, Arpan Pal, and Sundeep Khandelwal.
\newblock {MedTric} : A clinically applicable metric for evaluation of
multi-label computational diagnostic systems.
\newblock \emph{PLoS One}, 18\penalty0 (8), 2023{\natexlab{b}}.
\newblock URL \url{https://doi.org/10.1371/journal.pone.0283895}.
\bibitem[Serafini and d'Avila Garcez(2016)]{ltn}
Luciano Serafini and Artur~S. d'Avila Garcez.
\newblock Logic tensor networks: Deep learning and logical reasoning from data
and knowledge.
\newblock In \emph{Proceedings of the 11th International Workshop on
Neural-Symbolic Learning and Reasoning (NeSy'16)}, volume 1768 of
\emph{{CEUR} Workshop Proceedings}. CEUR-WS.org, 2016.
\newblock URL \url{https://ceur-ws.org/Vol-1768/NESY16\_paper3.pdf}.
\bibitem[Simonyan and Zisserman(2015)]{vgg}
Karen Simonyan and Andrew Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition,
2015.
\bibitem[Srivastava et~al.(2023)Srivastava, Rastogi, Rao, et~al.]{bigbench}
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, and 450 others.
\newblock Beyond the imitation game: Quantifying and extrapolating the
capabilities of language models.
\newblock \emph{Transactions on Machine Learning Research}, 2023.
\newblock ISSN 2835-8856.
\newblock URL \url{https://openreview.net/forum?id=uyTL5Bvosj}.
\bibitem[Stevens et~al.(2020)Stevens, Taylor, Nichols, Maccabe, Yelick, and
Brown]{ai4sc}
Rick Stevens, Valerie Taylor, Jeff Nichols, Arthur~Barney Maccabe, Katherine
Yelick, and David Brown.
\newblock {AI} for science: Report on the department of energy ({DOE}) town
halls on artificial intelligence ({AI}) for science.
\newblock 2 2020.
\newblock \doi{10.2172/1604756}.
\newblock URL \url{https://www.osti.gov/biblio/1604756}.
\bibitem[Suzgun et~al.(2022)Suzgun, Melas-Kyriazi, Sarkar, Kominers, and
Shieber]{bd4}
Mirac Suzgun, Luke Melas-Kyriazi, Suproteem~K. Sarkar, Scott~Duke Kominers, and
Stuart~M. Shieber.
\newblock The harvard uspto patent dataset: A large-scale, well-structured, and
multi-purpose corpus of patent applications.
\newblock \emph{ArXiv}, abs/2207.04043, 2022.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:250408264}.
\bibitem[von Rueden et~al.(2023)von Rueden, Mayer, Beckh, Georgiev,
Giesselbach, Heese, Kirsch, Pfrommer, Pick, Ramamurthy, Walczak, Garcke,
Bauckhage, and Schuecker]{rueden}
Laura von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven
Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick,
Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, and
Jannis Schuecker.
\newblock Informed machine learning – a taxonomy and survey of integrating
prior knowledge into learning systems.
\newblock \emph{IEEE Transactions on Knowledge and Data Engineering},
35\penalty0 (1):\penalty0 614--633, 2023.
\newblock \doi{10.1109/TKDE.2021.3079836}.
\bibitem[Wölflein and Arandjelović(2021)]{chessP2}
Georg Wölflein and Ognjen Arandjelović.
\newblock Determining chess game state from an image.
\newblock \emph{Journal of Imaging}, 7\penalty0 (6):\penalty0 94, June 2021.
\newblock ISSN 2313-433X.
\newblock \doi{10.3390/jimaging7060094}.
\newblock URL \url{http://dx.doi.org/10.3390/jimaging7060094}.
\bibitem[Xie et~al.(2021)Xie, Niu, Liu, Chen, Tang, and Yu]{medicalDataLack}
Xiaozheng Xie, Jianwei Niu, Xuefeng Liu, Zhengsu Chen, Shaojie Tang, and Shui
Yu.
\newblock A survey on incorporating domain knowledge into deep learning for
medical image analysis.
\newblock \emph{Medical Image Analysis}, 69:\penalty0 101985, 2021.
\newblock ISSN 1361-8415.
\newblock \doi{https://doi.org/10.1016/j.media.2021.101985}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S1361841521000311}.
\bibitem[Xu et~al.(2018)Xu, Zhang, Friedman, Liang, and den Broeck]{logicloss}
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy~Van den Broeck.
\newblock A semantic loss function for deep learning with symbolic knowledge.
\newblock In Jennifer~G. Dy and Andreas Krause, editors, \emph{Proceedings of
the 35th International Conference on Machine Learning, {ICML} 2018,
Stockholmsm{\"{a}}ssan, Stockholm, Sweden, July 10-15, 2018}, volume~80 of
\emph{Proceedings of Machine Learning Research}, pages 5498--5507. {PMLR},
2018.
\newblock URL \url{http://proceedings.mlr.press/v80/xu18h.html}.
\bibitem[Yang et~al.(2019)Yang, Al-Bahrani, Reid, Papanikolaou, Kalidindi,
Liao, Choudhary, and Agrawal]{datadifficult}
Zijiang Yang, Reda Al-Bahrani, Andrew C.~E. Reid, Stefanos Papanikolaou,
Surya~R. Kalidindi, Wei-keng Liao, Alok Choudhary, and Ankit Agrawal.
\newblock Deep learning based domain knowledge integration for small datasets:
Illustrative applications in materials informatics.
\newblock In \emph{2019 International Joint Conference on Neural Networks
(IJCNN)}, pages 1--8, 2019.
\newblock \doi{10.1109/IJCNN.2019.8852162}.
\newblock Material Science.
\bibitem[Yu et~al.(2018)Yu, Fu, Wang, and Zhao]{newgoa}
Guoxian Yu, Guangyuan Fu, Jun Wang, and Yingwen Zhao.
\newblock Newgoa: Predicting new go annotations of proteins by bi-random walks
on a hybrid graph.
\newblock \emph{IEEE/ACM Transactions on Computational Biology and
Bioinformatics}, 15\penalty0 (4), 2018.
\newblock \doi{10.1109/TCBB.2017.2715842}.
\bibitem[Zhang et~al.(2019)Zhang, Gao, Jia, Zhu, and Zhu]{raven}
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu.
\newblock Raven: A dataset for relational and analogical visual reasoning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)}, June 2019.
\end{thebibliography}