Inexact Column Generation for Bayesian Network Structure Learning via Difference-of-Submodular Optimization
Extracted source text · arXiv 2505.11089
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
%\usepackage{neurips_2025}
\bibliographystyle{abbrv}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{amsmath,amsfonts,amsthm,xcolor,centernot,url, amssymb}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{multirow}
\usepackage{bbm}
\usepackage{appendix}
\usepackage{graphicx}
\def \bE {\mathbb{E}}
\def \bP {\mathbb{P}}
\def \bR {\mathbb{R}}
\def \bZ {\mathbb{Z}}
\def \bK {\mathcal{K}}
\def \bbP {\mathcal{P}}
\def \bC {\mathcal{C}}
\def \conv {\text{conv}}
\def \cone {\text{cone}}
\def \proj {\text{proj}}
\def \dim {\text{dim}}
\def \det {\text{det}}
\def \ldet {\text{ldet}}
\def \lovasz {\text{Lov\'{a}sz}}
\newcommand{\rr}[1]{{\textcolor{red}{[RC: #1]}}}
%\newcommand{\rr}[1]{}
\newcommand{\bl}[1]{{\textcolor{blue}{[Y: #1]}}}
\newtheorem{prop}{Proposition}
\newtheorem{asm}{Assumption}
\newtheorem{rmk}{Remark}
\newtheorem{de}{Definition}
% \title{A Difference-of-Submodular Optimization Approach for Accelerating Column Generation in Bayesian Network Structure Learning}
\title{Inexact Column Generation for Bayesian Network Structure Learning via Difference-of-Submodular Optimization}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Yiran Yang\\
School of Data Science\\
The Chinese University of Hong Kong, Shenzhen\\
\texttt{yiranyang@link.cuhk.edu.cn} \\
\And
Rui Chen\\
School of Data Science\\
The Chinese University of Hong Kong, Shenzhen\\
\texttt{rchen@cuhk.edu.cn} \\
% examples of more authors
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
In this paper, we consider a score-based Integer Programming (IP) approach for solving the Bayesian Network Structure Learning (BNSL) problem. State-of-the-art BNSL IP formulations suffer from the exponentially large number of variables and constraints. A standard approach in IP to address such challenges is to employ row and column generation techniques, which dynamically generate rows and columns, while the complex pricing problem remains a computational bottleneck for BNSL. For the general class of $\ell_0$-penalized likelihood scores, we show how the pricing problem can be reformulated as a difference of submodular optimization problem, and how the Difference of Convex Algorithm (DCA) can be applied as an inexact method to efficiently solve the pricing problems. Empirically, we show that, for continuous Gaussian data, our row and column generation approach yields solutions with higher quality than state-of-the-art score-based approaches, especially when the graph density increases, and achieves comparable performance against benchmark constraint-based and hybrid approaches, even when the graph size increases.
% Future research directions include developing more effective approaches for handling other scoring functions, particularly for discrete data scenarios.
\end{abstract}
\section{Introduction} \label{sec: intro}
% 问题background
% review restriction IP 方法 （可以有少量数学符号）
%
Bayesian networks (BN) have wide applications in machine learning \cite{glymour1999computation}. Its structure learning problem, i.e., identifying a directed acyclic graph (DAG) that represents the causal relationships between variables $(X_i)_{i\in V}$ from data, is a central yet challenging topic in causal inference \cite{heinze2018causal}. Existing methods for learning optimal Bayesian network structure from observational data fall into three main categories \cite{spirtes2000causation,pearl2000models, drton2017structure}: (1) constraint-based approaches (e.g., PC \cite{spirtes2000causation, colombo2014order} and FCI \cite{spirtes2000causation,zhang2008completeness}) that learn the structure through a constraint satisfaction problem and involve conditional independence tests to formulate the constraints; %\rr{cite PC and FCI}
(2) score-based approaches (e.g., hill climbing \cite{heckerman1995learning}, dynamic programming \cite{koivisto2004exact, silander2012simple}, integer programming \cite{cussens2020gobnilp} and machine learning approaches \cite{yu2019dag}) that evaluate candidate DAGs using scoring functions and select the DAG structure with the highest score; (3) hybrid approaches that combine both score-based and constraint-based techniques (e.g., MMHC \cite{tsamardinos2006max}). %\rr{Mention hybrid approaches as well.}
%In our work, we consider a score-based approach which provides a natural optimization framework for BNSL problem through the explicit scoring functions.
%In such DAGs, nodes represent the random variables while directed edges denotes causal dependencies between variables.
Score-based BNSL is known to be NP-hard, even under the assumption that each node has a restricted number of parents \cite{chickering2004large}. This inherent hardness of BNSL imposes a fundamental trade-off between solution quality and computational efficiency. Consequently, there are two lines of work that focus on the either side: (1) exact methods that guarantee optimality but often suffer from higher computational cost and (2) heuristic approaches that offer computational efficiency but with limited optimality guarantees.
%The exact methods include branch and cut \cite{jaakkola2010learning,cussens2012column} dynamic programming \cite{koivisto2004exact, silander2012simple},
%\rr{Mention some other methods like hill climbing, and GNN \cite{yu2019dag}. See my causal structure learning paper for references.}
%while heuristic approaches take hill climbing method \cite{heckerman1995learning} and score-and-search method \cite{bartlett2017integer} as examples.
%\rr{Is this paragraph on general score-based approaches? It should be put earlier (directly after the first paragraph), i.e., the structure should be like BNSL=>score-based=>IP.}
% To mitigate the computational cost while achieving , these methods often require restrictive assumptions on parent set sizes.
% \bl{Previously wanted to discuss exact v.s. heuristic methods after stating that BNSL is NP-hard, and lead to the discussion of CG.}
Within the BNSL literature, one of state-of-the-art exact approaches to identify the highest-scoring DAG is to
formulate the problem as an Integer Program (IP) \cite{jaakkola2010learning, 10.5555/3020548.3020567, chen2021integer}, as represented by the GOBNILP method \cite{cussens2020gobnilp}. While there are different ways to formulate the IP for this problem, a standard formulation involves binary variables that indicate whether to choose the specific set as the parent set for each node, and constraints that enforce the acyclicity requirement for the DAG. Given the observational data, the objective of the IP is to maximize the score, which is a function of the potential DAG structure measuring how well the data fits the model.
%and is typically defined using the likelihood of the DAG structure.
Solving this IP is challenging since both the number of variables and the number of constraints grow exponentially with the node size of the DAG.
% The standard branch-and-bound techniques \cite{land2009automatic} lack scalability in solving this IP.
% This computational complexity is unsurprising, since BNSL problem has been proved to be NP-hard in \cite{chickering2004large}.
% \rr{Expand this paragraph. First, using binary variables to indicate parent sets is not the only option for formulating the problem as IP. There are other formulations as well, you may only say it is standard to do so. Second, computational complexity is \emph{not} equivalent to size. NP-hardness (put this in the first paragraph maybe) has nothing to do with problem size. Mention that IP approaches are (at least one of) state-of-the-art, and a representative example is GOBNILP, something like that.}
Column Generation (CG) \cite{gilmore1961linear,desaulniers2006column} is a standard approach for addressing huge IPs as it effectively balances computational efficiency and solution quality. By dynamically generating variables as needed through pricing problems, CG can be used to derive efficient heuristic algorithms \cite{joncour2010column}, or exact solution methods (often known as branch and price \cite{barnhart1998branch}) when combined with branch and cut. CG has been widely applied in operations research to solve real-world problems such as vehicle routing \cite{desrochers1992new} and crew scheduling \cite{potthoff2010column}. In \cite{cussens2012column,cussens2023branch}, Cussens introduces CG to BNSL to construct an exact branch and price algorithm. However, due to the complex pricing problems (which are formulated as mixed integer nonlinear programs in \cite{cussens2023branch}), the proposed approach has limited scalability and can only be applied to solve small BNSL instances.
Indeed, to accelerate CG algorithms, it is advised that one should not try to solve all pricing problems exactly and an approximate solution often suffices until the last iteration (for proving optimality) \cite{lubbecke2005selected}.
In this paper, built on top of \cite{cussens2023branch}, we advance CG solution techniques for BNSL. We develop an efficient pricing algorithm based on a reformulation of the pricing problem as a difference of submodular (DS) optimization problem, enabling efficient solution of the pricing problem via the Difference of Convex Algorithm (DCA) \cite{el2023difference, iyer2012algorithms}.
%We subsequently solve the resulting pricing problem through difference of submodular optimization \cite{el2023difference, iyer2012algorithms}.
%Building upon the framework in \cite{el2023difference}, we reformulate the Difference of Submodular optimization problem as an equivalent Difference of Convex optimization problem, which we can solve using the Difference of Convex Algorithm (DCA).
Even though DCA does not necessarily provide global optimal solutions, it guarantees convergence and often generates high-quality candidate columns for our BNSL IP. Empirically, we demonstrate that it leads to effective primal heuristics, offering potential for developing more efficient exact solution approaches.
%These solutions effectively reduce the optimality gap in the restricted master problem -- the linear programming relaxation of the original IP constrained to dynamically generated columns through the CG procedure. \rr{I don't understand the last sentence...}
% Our principal contributions are:\rr{We may potentially remove this because it is a bit repetitive}
% (1) We reformulate the pricing problem as a DS optimization problem for general $\ell_0$-penalized likelihood scores in both cases with continuous or discrete data, and apply DCA for efficient pricing solutions.
% (2) Our framework of column generation with DCA for pricing solutions (CG-DCA) empirically yields solutions with higher quality on Gaussian data than the baseline score-based methods, especially for a denser graph. It also achieves competitive performance with constraint-based approaches across difference graph sizes.
\section{Preliminaries}
%\rr{Too long, at most three pages... Put some in appendix. Hopefully intro+prelim <= 3 pages.}
\subsection{Scored-based BNSL as an IP} \label{section: preliminaries IP}
% l0 special case
In BNSL, a scoring function measures how well the observed data fits the DAG structure, where nodes represent individual random variables and directed edges represent direct causal dependencies. Given observational data $D$, score-based approaches aim to find a DAG with the highest score, i.e., solve the following optimization problem:\begin{equation}\label{prb: BNSL_opt}
\max_{G} \texttt{score}(G;D) \text{ s.t. }G\text{ being acyclic},
\end{equation}
where $\texttt{score}(G;D)$ denotes the score of the graph $G=(V,E)$ under data $D$. Given a particular data type (continuous or discrete), there exist several different choices for the scoring function. In this paper, we focus on $\ell_0$-penalized likelihood scores, i.e., scoring functions of the form $\texttt{score}(G;D)=\log(L(G;D))-\Lambda\cdot k(G)$ for some $\Lambda\geq 0$, which favor graphs with a higher maximum likelihood $L(G;D)$ and a lower number of free parameters to be estimated $k(G)$ in the graphical model. By setting $\Lambda$ differently, one can recover some commonly used scoring functions, such as the AIC score \cite{akaike1974new} and the BIC score \cite{schwarz1978estimating}.
%Common choices include Gaussian likelihood for continuous data and multinomial likelihood for discrete variables \cite{heckerman1995learning}. When the data-generating process follows the modeling assumptions, maximizing this score corresponds to identifying the DAG that best explains the observed data. However, direct likelihood maximization often leads to overfitting and excessively dense graph structures \cite{scanagatta2014min}. To mitigate this, regularization techniques such as $\ell_0$-penalization on parent set cardinality are commonly employed. Common scoring metrics including BIC\cite{schwarz1978estimating}, AIC \cite{akaike1974new} are special cases of the $\ell_0$-penalization, similarly incorporating edge-count penalties. For discrete data, the BDeu score \cite{buntine1991theory} is an alternative that assumes a prior Dirichlet distribution on model parameters. \rr{Keep the general idea but remove details}
A crucial property of $\ell_0$-penalized likelihood scores is their decomposability into node-specific components. Specifically, assuming that we have $n$ variables in our BNSL problem indexed by $i\in V=\{1,\ldots,n\}$, the score $\texttt{score}(G;D)$ of a DAG $G$ defined over nodes $V$ satisfies
\[
\texttt{score}(G;D) = \sum_{i=1}^n \texttt{score}_i\left(\texttt{pa}_i(G)\right),
\]
where $\texttt{score}_i(\texttt{pa}_i(G))$ denotes the local score of node $i$ (where we omit its dependence on $D$ for simplicity), whose value only depends on the parent set $\texttt{pa}_i(G)$ of node $i$ in graph $G$, i.e., the set of nodes pointing towards node $i$ in the directed graph $G$. Specific definitions of the $\ell_0$-penalized likelihood score and its local version, in both discrete and continuous data cases, can be found in Appendix~\ref{appendix: define score}. %\rr{Define $\mathcal{P}_i$ afterwards. Now we can formulate the IP.}
% This additive decomposition enables efficient optimization through local computations of $\texttt{score}_{i\gets J}$ for each potential parent set configuration.
The additive nature of the scoring function enables us to reformulate \eqref{prb: BNSL_opt} as an integer (linear) program. A classic IP formulation based on cluster constraints \cite{10.5555/3020548.3020567, jaakkola2010learning} is as follows
%efficient optimization through local computations of $c_{i\gets J}$ for each potential parent set configuration. By leveraging this decomposition, the BNSL problem can be formulated as the following IP \cite{cussens2023branch, jaakkola2010learning}
\begin{subequations} \label{prb: masterIP}
\begin{alignat}{3}
\max_x \quad & \sum_{i=1}^n \sum_{J \in \mathcal{P}_i} \texttt{score}_{i}(J)\cdot x_{i \gets J} \\
\text{s.t.} \quad & \sum_{J\in \bbP_i} x_{i\gets J} = 1, && \quad i=1, \ldots, n, \label{con:one}\\
& \sum_{i\in C} \sum_{\substack{J\in \bbP_i:\\ J\cap C \neq \emptyset}} x_{i\gets J} \leq |C|-1, && \quad C \in \bC, \label{con:cluster}\\
& x_{i\gets J} \in \{0, 1\}, && \quad i=1,\ldots, n,\ J\in \bbP_i.
\end{alignat}
\end{subequations}
%\rr{Shall we just formulate the problem as a maximization problem so that we can directly use the notations we defined earlier? TBC...}
Here $\bbP_i:=2^{V\setminus\{i\}}$ denotes the set of candidate parent sets of node $i$, the binary decision variable $x_{i\gets J}$ indicates whether parent set $J$ is selected for node $i$, with an associated local score $\texttt{score}_{i}(J)$. The linear objective reflects the score of the graph composed of the selected parent sets. Constraint (\ref{con:one}) enforces exactly one parent set to be chosen per node, while cluster constraints (\ref{con:cluster}) \cite{jaakkola2010learning} guarantee acyclicity by preventing directed cycles among any nonempty node subset $C \in \bC:=2^{V}\setminus\{\emptyset\}$.
Note that formulation \eqref{prb: masterIP} involves $\Theta(n2^{n-1})$ binary variables
%(enumerating all possible parent set configurations in $\bbP_i$ for all node $i$)
and $\Theta(2^n)$ constraints. Due to its exponential size, even directly writing down \eqref{prb: masterIP} as an IP in a computer is infeasible for reasonably large $n$, not to say solving it. Although the exponential number of constraints can be addressed through row generation (i.e., replacing $\bC$ by a small subset $\hat{\bC}$ in \eqref{prb: masterIP} and iteratively adding violated ones into $\hat{\bC}$, see Appendix~\ref{appendix: row generation}) \cite{10.5555/3020548.3020567}, the exponential number of variables remains a significant challenge. To deal with exponentially many variables, existing BNSL IP approaches often rely on a (compromising) assumption that the size of the parent set for each node is no larger than some constant $k$, in which case one may replace $\bbP_i$ by $\{J\in\bbP_i:|J|\leq k\}$ in \eqref{prb: masterIP} for all $i$ to reduce the number of variables \cite{bartlett2017integer}. Instead of putting such a restriction on the parent sets, we consider CG for BNSL as proposed in \cite{cussens2012column,cussens2023branch}.
%(from cycle prevention across all node subsets)
%growth in both variables and constraints renders exact optimization computationally intractable for large networks.
% \rr{Maybe row generation and column generation in a single subsection}
\subsection{Column Generation and the Pricing Problem for BNSL} \label{section: cg and pricing}
%Solving the IP \eqref{prb: masterIP} via standard IP branch-and-bound techniques \cite{land2009automatic} is computationally intractable due to the exponential number of variables and cluster constraints.
%The row and column generation framework \cite{gilmore1961linear} tackles this challenge through a dynamic process: (1) constraint generation (row generation) that selectively incorporates violated cluster constraints through separation oracle calls, and (2) variable generation (column generation) that expands the variable space by solving pricing subproblems to identify variables with negative reduced costs.
%While the row generation can be done by solving the problem with a restricted cluster set $\hat{\mathcal{C}}$ and augmenting it by detecting cycles in the graph solution (more details in Appendix~\ref{appendix: row generation}), column generation presents greater computational challenges for IP \eqref{prb: masterIP}, as the pricing problem constitutes a combinatorial optimization problem. To alleviate this difficulty, existing literature commonly imposes restrictions on parent set size \cite{cussens2023branch}.
% The paper \cite{cussens2023branch} implement the CG approach to solve the LP relaxation of the original IP. This method optimizes over a restricted set of variables and iteratively expands the solution space through a pricing problem that identifies promising new candidate variables. The approach efficiently explores the solution space while converging to optimal or near-optimal solutions. Upon convergence, the restricted IP is solved using only the variables generated during the column generation process, called \textit{Restricted Master IP} (RMIP).
Rather than including all the variables, CG handles IPs with huge number of variables by dynamically generating only the necessary ones through the so-called pricing problems. In the context of the BNSL IP \eqref{prb: masterIP},
%The CG process of solving IP \eqref{prb: masterIP} is
as demonstrated in \cite{cussens2023branch}, the CG procedure begins with a restricted set of parent sets $\hat{\bbP}_i \subseteq \mathcal{P}_i$ for each node $i=1,\ldots,n$, and considers a restricted version of \eqref{prb: masterIP} only over variables $x_{i\gets J},$ for $i=1,\ldots,n,~J \in \hat{\bbP}_i$, i.e., \eqref{prb: masterIP} with a small subset of columns. We refer to the linear programming (LP) relaxation of this restricted problem as the \textit{Restricted Master LP} (RMLP). Based on the current RMLP solution, CG iteratively searches for additional parent set choices that may improve the objective value through the solution of a pricing problem for each node $i$, and add them to $\hat{\bbP}_i$. This pricing problem aims to optimize the so-called reduced costs, to identify high-quality parent set choices to gradually enlarge the variable space and solve the LP relaxation of \eqref{prb: masterIP}.
As pointed out in \cite{cussens2023branch}, the pricing problem for generating candidate parent sets for node $i$ can be formulated as the following optimization problem:
\begin{align} \label{prb: pricing}
\min_{J \in \mathcal{P}_i}~ z_i(J;\lambda^*):= -\texttt{score}_{i}(J) + \lambda_i^* + \sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^*,
\end{align}
where $\lambda_i^*$ and $\lambda_C^*\geq 0$ are the optimal dual values associated with constraints (\ref{con:one}) and (\ref{con:cluster}) in the RMLP, respectively. The value of $z_i(J;\lambda^*)$ is known as the reduced cost \cite{bertsimas1997introduction} of the variable $x_{i \gets J}$, which measures the negative of how much the RMLP objective value might increase by adding the column associated with $x_{i \gets J}$ to RMLP. The column generation process terminates when the optimal reduced cost \eqref{prb: pricing} becomes nonnegative for all the nodes. %After the CG process, one can solve the IP \eqref{prb: masterIP} with a refined candidate set $\hat{\bbP}_i$, namely \textit{Restricted Master IP} (RMIP).
% , guaranteeing optimality of the RMLP solution.
In this paper, instead of using CG to construct an exact solution approach like branch and price, we consider a straightforward CG-based heuristic named the restricted master heuristic \cite{joncour2010column}, where we solve the IP \eqref{prb: masterIP} with the candidate set $\hat{\bbP}_i$ generated by CG, namely the \textit{Restricted Master IP} (RMIP), after CG (approximately) terminates.
% A strategy of Row Genration involves solving the restricted IP to obtain an integer solution, which can be directly decoded into a graph. We then identify cycles within the graph and incorporate these cycles as clusters into the cluster constraints.
% However, as the number of nodes increases, cycle detection in large graphs becomes computationally expensive. Thus we solve the separation problem to identify clusters violating the master LP's cluster constraints more efficiently.
% Given the current solution $x_{i\gets J}^*$, we aim to identify a cluster $C$ that violates the constraint \ref{con:cluster} (ideally the most violated one), defined by the optimization problem:
\subsection{Difference of Submodular Optimization via DCA} \label{section: submodular}
% \rr{Why are we talking about this? -- Pricing problem is set function minimization. All set function minimization can be formulated as DS optimization (cite...) We may apply DCA to efficiently solve them... Tell a story otherwise it is too dry.}
The pricing problem \eqref{prb: pricing} is a set function optimization problem. Although problem \eqref{prb: pricing}, in the continuous Gaussian case, can be formulated as a mixed-integer nonlinear program (MINLP), solving it with an MINLP solver is highly computationally demanding as shown in \cite{cussens2023branch}. Taking a direct set function optimization perspective, it is known that any set function can be written as a difference of submodular (DS) functions although finding such a DS decomposition has exponential complexity \cite{iyer2012algorithms}. However, when such a decomposition is known, one can develop efficient algorithms to find local solutions \cite{iyer2012algorithms,el2023difference}. Here we briefly review a DS optimization algorithm \cite{el2023difference} based on the well-known DCA \cite{an2005dc}.
%, the pricing problem can be reformulated as a difference of submodular optimization problem. Although it can be difficult to formulate the set function into DS functions as mentioned in \cite{el2023difference}, it is still very promising to utilize this optimization framework to improve the scalability of pricing problem solutions.
%DS functions optimization methods has been well studied in combinatorial optimization \cite{el2023difference,iyer2012algorithms}. Our focus will be casted on a specific method, difference of convex algorithm (DCA) \cite{el2023difference}. A detailed description of this method is as follows.
A DS function $z:2^V \to \bR$ is a set function that can be expressed as:
\[
z(J) = g(J) - f(J),
\]
where both $g:2^V \to \bR$ and $f$ are submodular set functions (see Appendix \ref{appendix: define submodular}) \cite{fujishige2005submodular}.
% According to \cite{el2023difference}, any set function can be reformulated as a DS function.
For a set function $h:2^V \to \bR$, its minimization can be equivalently formulated through its Lov\'{a}sz extension $h^L:[0,1]^n\to\bR$ (see Appendix \ref{appendix: define submodular}) \cite{lovasz1983submodular}.
The convexity of the Lov\'{a}sz extension for submodular functions enables the reformulation of $\min_J z(J) = g(J) - f(J)$ as a difference of convex (DC) program
\[
\min_x z^L(x) = g^L(x) - f^L(x).
\]
To solve this DC optimization problem, DCA (Algorithm~\ref{alg:dca}) iterates between two key steps. First, it computes a linear approximation $\widetilde{f}^L(x)$ of $f^L(x)$ at the current iteration point $x^t$ defined by \[\widetilde{f}^L(x) = f^L(x^t) + \langle y^t, x-x^t\rangle,\] where $y^t$ is a subgradient of $f^L(x)$ at $x^t$.
Subsequently, it minimizes a convex approximation of the objective $z^L(x)$, with $f^L(x)$ replaced by $\widetilde{f}^L(x)$. This subproblem is solved to obtain the next iteration point $x^{t+1}$. %Assume without loss of generality that $f^L(x)$ is normalized with $f^L(\mathbf{0})=0$, then $\widetilde{f}^L(x)=\langle y,x \rangle$.
\begin{algorithm}[tb]
\caption{The Difference of Convex Algorithm}\label{alg:dca}
\begin{algorithmic}[1]
\State \textbf{Input}:
Convex functions $g^L$, $f^L$ (and $z^L=g^L-f^L$), initial point $x^0$, threshold $\epsilon$
\State Initialize $t\leftarrow 0$
\Repeat
\State $y^t \in \partial f^L(x^t)$\label{line: subg}
\State $x^{t+1}\gets \arg\min_x \{g^L(x)-\langle y^t, x\rangle\}$\label{line: opt}
\State $t \gets t+1$
\Until{$|z^L(x^t) - z^L(x^{t-1})| < \epsilon$}
\State \textbf{Return} Solution $x^t$, objective value $z^L(x^t)$
\end{algorithmic}
\end{algorithm}
As the subgradient of the Lov\'{a}sz extension of a submodular function can be efficiently evaluated through its function evaluation, the convex program (Line~\ref{line: opt}) in Algorithm~\ref{alg:dca} can be efficiently solved via bundle methods \cite{bagirov2014introduction}. In our case, we adopt the classic Kelley's Algorithm \cite{kelley1960cutting} for numerical experiments.
One important advantage of DCA is that it guarantees a non-increasing sequence of objective values, i.e., for $t\geq 0$, we have\begin{equation}\label{dca_monotone}
z^L(x^{t+1})\leq z^L(x^t).
\end{equation}
It is also easy to recover a solution of the original set function optimization problem from DCA, as the function value of $z$ agrees with $z^L$ at integer solutions and all extreme point solutions of the convex program (Line~\ref{line: opt}) in Algorithm~\ref{alg:dca} are integer solutions (since the corresponding set function is submodular).
\if0
as established by the following inequalities for successive iterates $x^t$ and $x^{t+1}$:
\begin{align}
g^L(x^t) - f^L(x^t) &= g^L(x^t) - \langle y^t, x^t \rangle \quad \quad \text{since approximate at $x^t$} \\
&\geq g^L(x^{t+1}) - \langle y^t, x^{t+1} \rangle \quad \quad \text{exact optimization of the convex program} \\
&\geq g^L(x^{t+1}) - f^L(x^{t+1}) \quad \quad \text{by the convexity of $f^L$}.
\end{align}
After getting a minimizer for $z^L(x)$, one can easily get a minimizer for the original set function $z(J)$, since the extreme points in the DC program that minimizes $z^L(x)$ are integers. For any integer solution $x^t\in\{0,1\}^n$, it holds that $z^L(x^t)=z(J)$, with $J=\{j\in [1:n]:x^t_j=1\}$.
\fi
\section{Solution of the Pricing Problem}
\subsection{The Pricing Problem as DS Optimization}
% \rr{Background/connection missing. Why are we solving the pricing problem? Where is it from? Why is it hard?}
A key step in CG is the solution of the pricing problem \eqref{prb: pricing} that iteratively selects ``promising'' columns to add into the restricted problem formulation. Existing MINLP approaches are known to have very limited scalability to solve the pricing problem for BNSL \cite{cussens2023branch}. We show in this section how the pricing problem can be explicitly rewritten as a DS optimization problem, enabling us to take advantage of the DCA algorithm we describe in Section \ref{section: submodular}.
%The pricing problem—which selects candidate parent sets to iteratively improve the solutions within the CG framework—is critical for ensuring solution optimality. This problem inherently involves set function minimization, and the nonlinear objective creates substantial computational challenges. Existing methods for solving pricing problem involves exact optimization of the pricing MINLP, which has limited scalability, i.e., can only obtain solutions for small graphs in reasonable time.
%Motivated by the theoretical result that any set function can be reformulated as a DS function \cite{el2023difference}, we reformulate the pricing problem as a DS optimization task, enabling efficient solutions through Algorithm~\ref{alg:dca}.
Recall that the pricing problem for node $i$ is formulated as minimizing the reduced cost
\[
\min_{J \in \mathcal{P}_i}~ z_i(J;\lambda^*)= -\texttt{score}_{i}(J) + \lambda_i^* + \sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^*.
\]
For both continuous and discrete cases, we show how the $\ell_0$-penalized likelihood score $z_i(J;\lambda^*)$ can be expressed as a DS function. %The reformulation of $z(J; \lambda^*)$ is formally expressed in the following propositions.
\begin{prop}[Continuous Case] \label{prop: continuous}
For continuous data with the $\ell_0$-penalized Gaussian likelihood score, $z_i(J;\lambda^*)$ satisfies \[
z_i(J;\lambda^*) = g(J) - f(J),\]
where\[
g(J) = \log\operatorname{det}(\hat{\Sigma}_{J\cup i, J\cup i}) + \Lambda|J| + \sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^*+ \lambda_i^*
\] and \[
f(J) = \log\operatorname{det}(\hat{\Sigma}_{J,J})
\] are both submodular functions, with $\hat{\Sigma}_{J, J}$ and $ \hat{\Sigma}_{J\cup i, J\cup i}$ denoting the empirical covariance matrices associated with variables $\{X_j:j\in J\}$ and $\{X_j:j\in J\cup\{i\}\}$, respectively.
\end{prop}
\begin{prop}[Discrete Case] \label{prop: discrete}
For discrete data with the $\ell_0$-penalized multinomial likelihood score, $z_i(J;\lambda^*)$ satisfies\[
z_i(J;\lambda^*) = g(J) - f(J),\]
where\[
g(J) = H(J \cup \{i\}) + \sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^* + \lambda_i^*
\] and \[
f(J) = \left(H(J) - \Lambda (a_i-1)\prod_{j\in J} a_j\right)
\] are both submodular functions, with $H(J)$ and $H(J\cup i)$ denoting the joint entropy associated with variables $\{X_j:j\in J\}$ and $\{X_j:j\in J\cup\{i\}\}$, respectively, and $a_j$ denoting the arity (i.e., the number of possible values it can take) of the variable $X_j$ for $j=1,\dots n$. %\rr{Alternative of ``arity''?}
\end{prop}
Proofs of Propositions~\ref{prop: continuous} and ~\ref{prop: discrete}
can be found in Appendix~\ref{appendix: proof}.
\subsection{Implementation of DCA for Pricing} \label{sec: dca}
We have reformulated the pricing objective as a DS function in Propositions~\ref{prop: continuous}\ and ~\ref{prop: discrete}.
As established in Section~\ref{section: submodular}, minimizing the pricing objective is then equivalent to minimizing the difference of convex functions $z^L(x)=g^L(x) - f^L(x)$,
where $g^L(x)$ and $f^L(x)$ represent the Lov\'{a}sz extension of submodular functions $g$ and $f$, respectively.
To optimize this objective, we apply DCA, i.e., Algorithm ~\ref{alg:dca}, to iteratively minimize $z^L(x)$ through its successive convex approximations.
%\subsection{The iterative two steps in DCA}
The DCA procedure for solving the pricing problem for node $i$ proceeds as follows. The algorithm begins by initializing an $(n-1)$-dimensional solution vector $(x_j)_{j\in V\setminus\{i\}}$.
%(excluding $x_i$, where we explicitly fix $x_i = 0$ when solving the pricing problem for node $i$).
The required inputs include the dataset $D\in\mathbb{R}^{N\times n}$, node index $i$, optimal dual solutions $\lambda^*$ of RMLP, regularization parameter $\Lambda$, and convergence thresholds $\epsilon$.
%These parameters are used to evaluate the Lov\'{a}sz extension functions $g^L$ and $f^L$ along with their corresponding subgradients.
During iteration $t$, the algorithm first computes the subgradient $y^t$ of $f^L$ at the current solution $x^t$. Without loss of generality, assume for simplicity that $i=n$. Let $\sigma$ be a permutation of $\{1,\ldots,n-1\}$ such that $x^t_{\sigma(1)}\geq x^t_{\sigma(2)}\geq\ldots\geq x^t_{\sigma(n-1)}$. For the continuous case, a subgradient $y^t$ of $f^L$ at $x^t$ is given by
\[
y^t_{\sigma(k)} = \begin{cases}
\log\det(\hat{\Sigma}_{\{\sigma(1)\}}) & k = 1 \\
\log\det(\hat{\Sigma}_{\{\sigma(1),\ldots,\sigma(k)\}}) - \log\det(\hat{\Sigma}_{\{\sigma(1),\ldots,\sigma(k-1)\}}) & k \geq 2,
\end{cases}
\]
where $\hat{\Sigma}_{\{\sigma(1),\ldots,\sigma(k)\}}$ denotes the empirical covariance matrix for the corresponding columns $\{\sigma(1),\ldots,\sigma(k)\}$ of the data matrix $D \in \mathbb{R}^{N \times n}$.
For the discrete case, a subgradient $y^t$ of $f^L$ at $x^t$ is given by {\small\[
y^t_{\sigma(k)} = \begin{cases}
H(\{\sigma(1)\}) - \Lambda (a_n-1)(a_{\sigma(1)}-1) & k = 1 \\
H(\{\sigma(1),\ldots \sigma(k)\}) - H(\{\sigma(1),\ldots, \sigma(k-1)\}) - \Lambda (a_n-1)(a_{\sigma(k)}-1)\prod_{j=1}^{k-1}a_{\sigma(j)} & k \geq 2.
\end{cases}
\]}%
After evaluating $\log\det(\hat{\Sigma}_{({\cdot})})$ or $H(\cdot)$ during the computation of $y^t$, it is crucial to save the evaluated values to avoid repeating calculations in the future. This small trick greatly improves the efficiency of DCA in pricing problem solutions in practice.
%Upon convergence of DCA, the continuous solution undergoes binarization via thresholding at $0.5$.
%The final parent set for node $i$ is obtained by inserting $0$ at the $i$-th position of the thresholded vector $x\in \{0,1\}^{n-1}$, since we have excluded this entry when initializing $x$. %\rr{What does this sentence mean?}.
%\begin{rmk}
% To obtain more patterns in $\hat{\bbP}_i$, instead of only adding the final solution of DCA to $\hat{\bbP}_i$, we evaluate the reduced cost at the intermediate point $x^t$ and add the parent set pattern obtained through $x^t$ with negative reduced cost.
%\end{rmk}
\section{Row and Column Generation Scheme} \label{sec: integrate cg rg}
\begin{algorithm}[tb]
\caption{Row and Column Generation for BNSL}\label{alg:rcg}
\begin{algorithmic}[1]
\State \textbf{Input}: Data matrix $D\in\mathbb{R}^{N\times n}$, regularization parameter $\Lambda$
\State Initialize the cluster set $\hat{\bC}\gets\emptyset$, and candidate parent sets $\hat{\bbP}_i\leftarrow\{\emptyset\}$ for $i=1, \ldots, n$ \label{line: initialize}
\Repeat \label{line: cg}
\State Initialize $\texttt{rc}_{i} \gets -\infty$ for $i=1,\ldots, n$ \Comment{initialize reduced costs}
\Repeat
\State Solve RMLP to obtain the optimal dual values $\lambda^*$ %\Comment{see Section~\ref{section: cg and pricing}} \label{line: RMLP}
\For{$i=1, \ldots, n$}
\If{$\texttt{rc}_i<0$} \label{line: rc}
\Comment{selectively solve pricing problems}
\State Choose an initial point $x^0$ for DCA %\Comment{the initial point for DCA}
\State $(\texttt{pa},\texttt{rc}_i)\gets\text{DCA-Pricing}(D,i,x^0, \lambda^*,\Lambda, \epsilon)$ \Comment{DCA pricing}
\State $\hat{\bbP}_i\gets \hat{\bbP}_i\cup\{\texttt{pa}\}$ \Comment{column generation}
\EndIf
\EndFor
\Until{$\texttt{rc}_i\geq 0$ for $i=1\ldots n$} \label{line: cg end}
\Repeat \label{line: rg}
\State Solve RMLP with updated $\hat{\bbP}$ and $\hat{\bC}$
\State Identify the most violated cluster constraint $C^*$\Comment{row generation for RMLP}
\If{the cluster constraint associated with $C^*$ is violated}
\State $\hat{\bC}\gets \hat{\bC}\cup\{C^*\}$
\EndIf
\Until{no new violated constraint can be found} \label{line: rg end}
\State $G \gets$ optimal DAG chosen from $\hat{\bbP}_{1:n}$; simultaneously update $\hat{\bC}$ \label{line: IP}\Comment{row generation for RMIP}
\Until{$G$ converges}
\State \Return $G$
\end{algorithmic}
\end{algorithm}
Algorithm~\ref{alg:rcg} demonstrates how we incorporate our DCA pricing method into a simultaneous row and column generation framework.
%We integrate our DCA pricing method into the combined row and column generation framework presented in Section~\ref{section: cg and pricing}, as demonstrated in .
The algorithm begins by initializing the candidate parent sets $\hat{\bbP}_{1:n}$ and the cluster set $\hat{\bC}$ with basic candidates (Line \ref{line: initialize}). It then iterates among three main phases: (1) a CG phase (Lines \ref{line: cg}-\ref{line: cg end}) where the RMLP is solved to obtain optimal dual solutions $\lambda^*$, followed by the solution of the pricing problem for each node via DCA --- The pricing solutions that have negative reduced costs are added to $\hat{\bbP}_i$. The CG phase repeats until all reduced costs become non-negative. In Line \ref{line: rc}, the pricing problem on node $i$ is solved only when the current reduced cost $rc_i$ is negative, to avoid spending too much time on pricing; (2) a row generation phase (Lines \ref{line: rg}-\ref{line: rg end}) where the RMLP optimal solution is used to identify the most violated cluster constraint (see Section \ref{appendix: row generation} for details) which is added to cluster set $\hat{\bC}$; (3) the integer solution phase (Line \ref{line: IP}), where the algorithm finds the optimal DAG $G$ with current candidate $\hat{\bbP}_{1:n}$ --- This is achieved by solving RMIP using branch and cut with callbacks (a feature that allows users to interrupt the solution process in modern IP solvers such as Gurobi), where we check if any cluster constraint is violated when we meet a integer incumbent and add violated cluster constraints as lazy constraints. The three phases iterate until convergence to a valid DAG solution, progressively refining both the columns in $\hat{\bbP}_{1:n}$ and cluster constraints in $\hat{\bC}$.
% Finally, the algorithm solves the \textit{master IP} to obtain an integer solution, decodes it into a graph, and adds any detected cycles as new clusters to $\mathcal{C}$.
Slightly different from its description, when we implement the CG phase in Algorithm \ref{alg:rcg}, in addition to the final solution generated by DCA, we add to $\hat{\bbP}_i$ all candidate parent sets associated with intermediate $x^t$ solutions that have negative reduced costs. Empirically, we find that it helps accelerate convergence and improves solution quality.
%In the CG phase, an implementation trick to accelerate the CG procedure is that, in addition to the final solution generated by DCA, we add to $\hat{\bbP}_i$ all candidate parent sets associated with intermediate $x^t$ solutions that have negative reduced costs. %\rr{Potentially move this paragraph to Section \ref{sec: integrate cg rg}}
\subsection{Choice of the Initial Point}
A step that remains ambiguous in Algorithm \ref{alg:rcg} is the choice of the initial point $x^0$ for DCA (Line \ref{line: initialize}), which we find having a significant impact on the overall solution time and solution quality. Below we first describe two initialization strategies.
The first strategy is a warm-start initialization strategy. Note that by the complementarity property of optimal solutions in LP \cite{bertsimas1997introduction}, the parent sets selected (i.e., parent sets $J$ associated with the primal variables $x_{i\leftarrow J}$ taking positive values) in the RMLP solution have reduced costs equal to zero. Therefore, we consider using the parent sets selected in the RMLP solution to construct the initial solution $x^0$ for DCA. Due to the monotonicity property \eqref{dca_monotone} of DCA, it often quickly finds columns with negative reduced costs, leading to a rapid convergence of DCA. However, a drawback of the warm-start initialization strategy is that it may restrict exploration of parent sets as the evaluation remains localized around solutions already contained in the candidate parent set choice set. Preliminary experiments show that a pure warm-start initialization strategy can quickly find a DAG with a good score while the solution often deviates a lot from the ground truth due to its greedy nature.
%, which utilizes the parent sets selected by RMLP solved in Line \ref{line: RMLP} of Algorithm \ref{alg:rcg}. This approach offers the advantage that the initial point $x^0$ is typically near-optimal within the current candidate set, as it represents the optimal choice in RMLP. Following warm-start initialization, DCA proceeds to search for improved solutions (likely in the neighborhood of $x^0$), subsequently adding them to $\hat{\bbP}_i$. This strategy yields rapid convergence, but it may restrict exploration of the pattern space, because the evaluation remains localized around solutions already contained in the candidate parent set choice set.
The second strategy employs a random initialization that samples a point $x^0$ uniformly from $[0,1]^{V\setminus\{i\}}$. This random initialization strategy facilitates global exploration on the parent set space and may discover patterns distant from the current ones in $\hat{\bbP}_i$. However, we find that exclusive reliance on random initialization can significantly slow down DCA convergence, as computational resources may be expended on evaluating solutions far from the optimum.
A hybrid initialization strategy that combines efficiency and diversity of the previous two strategies is often more effective in practice.
In our implementation, we initially employ random initialization to broadly explore the solution space. Once the candidate set $\hat{\bbP}_i$ contains a reasonably large number (set to be 50 in our implementation) of parent sets, we switch to warm-start initialization for pricing at node $i$. This hybrid approach focuses on local refinement around the current best pattern while building upon the foundation of global exploration.
% \begin{rmk} \label{rmk: init}
% The initialization of $x$ can employ either random initialization or warm-start strategies. A warm-start initialization uses the indicator vector of $J$ that maximizes $x^*_{i\gets J}$, where $x^*$ denotes the optimal solution from the Master LP. While random initialization promotes exploration of diverse patterns, warm-start initialization typically yields faster local convergence.
% \end{rmk}
% -----------------------------------------------------------
\section{Numerical Experiments} \label{sec: numerical}
To evaluate the efficiency of our method, we conducted numerical experiments comparing the performance of our Column Generation with DCA-pricing (CG-DCA) method against five baseline methods: GOBNILP, \cite{cussens2020gobnilp}, CG-MINLP \cite{cussens2023branch} (in Section~\ref{result: exact minlp}), HC \cite{heckerman1995learning} (in Appendix~\ref{appendix: result}), stable-PC \cite{colombo2014order} and MMHC \cite{tsamardinos2006max}. While CG-MINLP is evaluated on \texttt{Gaussian.test} dataset in the R package \texttt{bnlearn} \cite{scutari2010learning}, other methods are tested on both simulated and real-world datasets. The last three methods are tested using \texttt{bnlearn}. All IP-related experiments are conducted on a Linux machine with two Intel XEON Platinum 8575C processors, with the number of threads used limited to 1. All IPs are solved using Gurobi 11.0.3. %\rr{check the version}
\subsection{Comparing DCA with MINLP for Pricing} \label{result: exact minlp}
We first compare the performance of CG with different pricing algorithms: our proposed CG-DCA versus the exact MINLP solver for pricing (CG-MINLP). The evaluation is conducted on the \texttt{Gaussian.test} dataset containing a ground truth graph with $7$ nodes and $7$ edges.
For this small instance, CG-DCA successfully learns the exact ground truth structure in $1.2$ seconds, while CG-MINLP requires $19.2$ seconds to achieve the same result.
For larger instances, CG-MINLP usually fails to produce a valid solution DAG within a reasonable time limit as solving the pricing problem exactly as an MINLP is quite cumbersome in practice.
%Implementing CG-MINLP for larger graph instances is impractical because the branching procedure required to search for optimal solutions generates an excessively large search tree. This overwhelms memory, significantly slows down computations, and may cause the solver to halt at certain branches.
For a graph with node size $n=20$ and average in-degree $d=1$, solving the first pricing problem using MINLP takes an average of $35.1$ seconds per node, while the DCA method takes $0.3$ second per node for the same instance. The difference is even larger for later pricing problems. In addition, MINLP approaches suffer from numerical issues as they require an approximation for the nonlinear $\log$ function (at least in Gurobi), producing inaccurrate estimates for the pricing objective. %\rr{Need to be more specific. On graph with what size, taking how long to solve the first(?) pricing? Also, explain that MINLP approaches might suffer from numerical issues as they use rough approximations for nonlinear functions.}
\subsection{Results on Simulated Gaussian Data}
\begin{table}[tb!]
\centering
\caption{Score and Time Comparison between CG-DCA and GOBNILP on Gaussian Datasets}
\label{res: cont score}
\scriptsize
\begin{tabular}{l l l c c c c}
\toprule
\multicolumn{3}{c}{Graph} & \multicolumn{2}{c}{BIC Score Gap (\%)} & \multicolumn{2}{c}{Time (seconds)} \\
\cmidrule(lr){1-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
$n$ & $N$ & $d$ & CG-DCA & GOBNILP & CG-DCA & GOBNILP \\
\midrule
15 & 5000 & 0.5 & \textbf{0.00} & 0.14 & \textbf{\phantom{000}5.80} & \phantom{000}8.33 \\
15 & 5000 & 1 & \textbf{0.03} & 0.22 & \phantom{00}24.92 & \textbf{\phantom{00}19.74} \\
15 & 5000 & 1.5 & \textbf{0.07} & 0.62 & \phantom{00}70.61 & \textbf{\phantom{00}60.09} \\
15 & 5000 & 2 & \textbf{0.27} & 2.06 & 217.87 & \textbf{167.41} \\
20 & 5000 & 0.5 & \textbf{0.00} & 0.02 & \phantom{00}12.43 & \textbf{\phantom{000}3.38} \\
20 & 5000 & 1 & \textbf{0.02} & 0.35 & \textbf{\phantom{00}86.05} & 287.87 \\
20 & 5000 & 1.5 & \textbf{0.22} & 1.63 & \textbf{\phantom{0}403.38} & 962.50 \\
20 & 5000 & 2 & \textbf{0.65} & 4.77 & \textbf{1385.88} & 4603.81 \\
25 & 5000 & 0.5 & \textbf{0.00} & 0.12 & \phantom{00}37.28 & \textbf{\phantom{00}21.95} \\
25 & 5000 & 1 & \textbf{0.06} & 0.56 & \textbf{\phantom{0}272.19} & 1964.11 \\
25 & 5000 & 1.5 & \textbf{0.26} & 1.92 & \textbf{1426.39} & 5805.01 \\
25 & 5000 & 2 & \textbf{0.69} & 5.15 & \textbf{4850.47} & 8500.77 \\
% 15 & 20000 & 0.5 & \textbf{0.00} & 0.06 & \phantom{00}14.54 & \textbf{\phantom{000}7.35} \\
% 15 & 20000 & 1 & \textbf{0.00} & 0.23 & \phantom{00}46.32 & \textbf{\phantom{00}24.88} \\
% 15 & 20000 & 1.5 & \textbf{0.03} & 0.86 & \phantom{0}122.85 & \textbf{\phantom{00}76.29} \\
% 15 & 20000 & 2 & \textbf{0.24} & 2.01 & \phantom{0}284.91 & \textbf{\phantom{00}89.92} \\
% 20 & 20000 & 0.5 & \textbf{0.00} & 0.04 & \phantom{00}37.88 & \textbf{\phantom{000}4.92} \\
% 20 & 20000 & 1 & \textbf{0.00} & 0.22 & \textbf{\phantom{0}196.02} & 261.89 \\
% 20 & 20000 & 1.5 & \textbf{0.21} & 1.45 & \textbf{\phantom{0}578.68} & 2457.26 \\
% 20 & 20000 & 2 & \textbf{0.46} & 3.63 & \textbf{1555.12} & 4394.45 \\
% 25 & 20000 & 0.5 & \textbf{0.00} & 0.09 & \phantom{00}99.08 & \textbf{\phantom{00}23.96} \\
% 25 & 20000 & 1 & \textbf{0.04} & 0.40 & \textbf{\phantom{0}650.32} & 1948.05 \\
% 25 & 20000 & 1.5 & \textbf{0.18} & 2.21 & \textbf{2227.80} & 5853.91 \\
% 25 & 20000 & 2 & \textbf{0.57} & 4.51 & \textbf{4162.49} & 7967.28 \\
\bottomrule
\multicolumn{7}{l}{ *Each value is the average over ten independent instances.} \\
\end{tabular}
\end{table}
Now we test on larger instances with Gaussian data. Following the experimental setup in \cite{cussens2023branch, bhattacharya2021differentiable}, we randomly generate Bayesian networks with node size $n\in\{15, 20, 25\}$, and simulate Gaussian data from those networks. To generate the DAGs, we first fix a topological order of the nodes and then simulate edges according to the specified probabilities where edges must be directed from lower-order nodes to higher-order nodes (there are $n(n-1)/2$ such possible edges). While we do not enforce a maximum in-degree of each node, we randomly simulate edges using their existence probability $p$ determining the overall density of the graph, i.e., for each possible edge we include it in the network with probability $p$.
The average in-degree $d$ of nodes in the ground truth DAG is controlled by the edge existence probability since $p = {2d}/{(n-1)}$. We vary $p$ such that $d$ is varied among $\{0.5,1,1.5,2\}$. %This approach to DAG simulation follows the methodology described in \cite{erdds1959random}.
After generating the random graph structure, we simulate Gaussian data based on the network topology. The linear coefficients for the features were randomly drawn from the interval $\pm[0.5, 2]$, with additive noise following the Normal distribution $N(0,\sigma^2)$. The true value of variance parameter $\sigma^2$ is uniformly drawn from the interval $[0.7, 1.2]$. %The sample size $N$ is set to be $5{,}000$ or $20{,}000$.
Tables~\ref{res: cont score} and~\ref{res: cont metric} present a comparison between CG-DCA and baseline methods across various node sizes ($n$), and average in-degrees ($d$) with $N=5{,}000$. Results for $N=20{,}000$ exhibit little variation from those for $N=5{,}000$ and can be found in Appendix~\ref{appendix: result}. For each graph structure, we generate $10$ independent data instances using different random seeds and record the average performance metrics.
Table~\ref{res: cont score} reports the BIC score gap (difference between the achieved BIC score and BIC score of the true graph) and the runtime (with a 3-hour time limit per instance) of CG-DCA and GOBNILP.
As demonstrated in Table~\ref{res: cont score}, the CG-DCA method consistently outperforms GOBNILP in terms of average scores across simulated Gaussian datasets. Notably, CG-DCA achieves optimal BIC scores for instances with sparse graph structures ($d=0.5$). As the number of nodes increases, CG-DCA exhibits a slower growth in runtime compared to GOBNILP while maintaining superior scoring performance. The robustness in both computational efficiency and solution quality with respect to node count ($n$) and graph density ($d$) makes CG-DCA more suitable for larger problems.
% However, we observe a more pronounced runtime increase for CG-DCA with larger sample sizes ($N$). This phenomenon stems from the computational complexity of log-determinant matrix evaluation, which becomes increasingly demanding with larger datasets. Given that score improvement saturates with larger samples, CG-DCA already provides accurate results with small datasets, reducing computational overhead without sacrificing performance.
\begin{table}[tb!]
\centering
\caption{Performance Comparison of CG-DCA, GOBNILP (GI), PC and MMHC on Gaussian Datasets}
\label{res: cont metric}
\tiny
\begin{tabular}{l c c c c c c c c c c c c}
\toprule
\multirow{2}{*}{$n, N, d$} & \multicolumn{4}{c}{Precision} & \multicolumn{4}{c}{Recall} & \multicolumn{4}{c}{SHD} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13}
& CG-DCA & GI & PC & MMHC & CG-DCA & GI & PC & MMHC & CG-DCA & GI & PC & MMHC \\
\midrule
(15, 5000, 0.5) & \textbf{0.71} & 0.66 & 0.59 & 0.50 & \textbf{0.79} & 0.73 & 0.62 & 0.51 & \textbf{\phantom{0}2.60} & \phantom{0}2.90 & \phantom{0}3.00 & \phantom{0}3.50 \\
(15, 5000, 1) & \textbf{0.78} & 0.67 & 0.72 & 0.59 & \textbf{0.84} & 0.72 & 0.67 & 0.54 & \textbf{\phantom{0}4.10} & \phantom{0}5.00 & \phantom{0}4.80 & \phantom{0}6.70 \\
(15, 5000, 1.5) & \textbf{0.75} & 0.73 & 0.61 & 0.56 & \textbf{0.82} & 0.80 & 0.54 & 0.48 & \textbf{\phantom{0}6.80} & \phantom{0}7.40 & 11.50 & 12.20 \\
(15, 5000, 2) & \textbf{0.61} & 0.58 & 0.54 & 0.52 & \textbf{0.74} & 0.65 & 0.39 & 0.36 & 17.90 & \textbf{16.90} & 20.90 & 20.40 \\
(20, 5000, 0.5) & 0.74 & 0.71 & \textbf{0.76} & 0.59 & 0.77 & 0.75 & \textbf{0.79} & 0.60 & \phantom{0}2.90 & \phantom{0}3.50 & \textbf{\phantom{0}2.60} & \phantom{0}4.40 \\
(20, 5000, 1) & 0.72 & 0.69 & \textbf{0.75} & 0.65 & \textbf{0.79} & 0.77 & 0.71 & 0.59 & \phantom{0}7.00 & \phantom{0}7.50 & \textbf{\phantom{0}5.90} & \phantom{0}7.90 \\
(20, 5000, 1.5) & 0.61 & 0.61 & 0.62 & \textbf{0.65} & \textbf{0.79} & 0.71 & 0.50 & 0.49 & 17.50 & \textbf{16.80} & 17.90 & \textbf{16.80} \\
(20, 5000, 2) & \textbf{0.58} & 0.46 & 0.54 & 0.55 & \textbf{0.77} & 0.53 & 0.34 & 0.31 & \textbf{30.00} & 33.60 & 31.50 & 31.10 \\
(25, 5000, 0.5) & 0.74 & 0.61 & \textbf{0.84} & 0.69 & 0.79 & 0.67 & \textbf{0.84} & 0.69 & \phantom{0}3.10 & \phantom{0}5.90 & \textbf{\phantom{0}2.20} & \phantom{0}4.40 \\
(25, 5000, 1) & 0.78 & 0.69 & \textbf{0.80} & 0.74 & \textbf{0.86} & 0.76 & 0.75 & 0.69 & \phantom{0}9.20 & 11.10 & \textbf{\phantom{0}7.40} & \phantom{0}8.70 \\
(25, 5000, 1.5) & 0.63 & 0.57 & 0.70 & \textbf{0.71} & \textbf{0.82} & 0.68 & 0.58 & 0.56 & 27.50 & 22.10 & 18.50 & \textbf{18.00} \\
(25, 5000, 2) & \textbf{0.61} & 0.47 & 0.53 & 0.59 & \textbf{0.78} & 0.54 & 0.35 & 0.35 & 40.30 & 40.30 & 38.70 & \textbf{36.50} \\
% (15, 20000, 0.5) & \textbf{0.71} & 0.69 & 0.68 & 0.53 & 0.71 & \textbf{0.73} & 0.68 & 0.53 & \textbf{\phantom{0}1.90} & \phantom{0}2.20 & \textbf{\phantom{0}1.90} & \phantom{0}3.00 \\
% (15, 20000, 1) & 0.75 & \textbf{0.78} & 0.71 & 0.61 & 0.78 & \textbf{0.81} & 0.69 & 0.59 & \phantom{0}4.10 & \textbf{\phantom{0}4.00} & \phantom{0}4.70 & \phantom{0}6.20 \\
% (15, 20000, 1.5) & \textbf{0.74} & 0.73 & 0.72 & 0.66 & \textbf{0.83} & 0.79 & 0.66 & 0.57 & \textbf{\phantom{0}8.10} & \phantom{0}9.40 & \phantom{0}8.60 & 10.00 \\
% (15, 20000, 2) & \textbf{0.64} & 0.53 & 0.53 & 0.55 & \textbf{0.79} & 0.60 & 0.43 & 0.39 & \textbf{16.00} & 19.90 & 20.01 & 19.80 \\
% (20, 20000, 0.5) & 0.72 & \textbf{0.80} & 0.73 & 0.58 & 0.76 & \textbf{0.82} & 0.76 & 0.59 & \phantom{0}3.60 & \textbf{\phantom{0}2.60} & \phantom{0}3.00 & \phantom{0}4.60 \\
% (20, 20000, 1) & 0.70 & 0.73 & \textbf{0.79} & 0.68 & 0.78 & \textbf{0.79} & 0.77 & 0.65 & \phantom{0}6.50 & \phantom{0}6.90 & \textbf{\phantom{0}4.90} & \phantom{0}7.10 \\
% (20, 20000, 1.5) & 0.57 & 0.58 & \textbf{0.64} & 0.63 & \textbf{0.77} & 0.68 & 0.56 & 0.50 & 21.20 & 18.20 & \textbf{16.00} & 16.50 \\
% (20, 20000, 2) & 0.54 & 0.48 & 0.51 & \textbf{0.56} & \textbf{0.79} & 0.55 & 0.37 & 0.33 & 31.30 & 32.40 & 31.30 & \textbf{29.30} \\
% (25, 20000, 0.5) & 0.77 & 0.80 & \textbf{0.81} & 0.70 & 0.79 & \textbf{0.84} & 0.83 & 0.70 & \phantom{0}3.10 & \phantom{0}2.80 & \textbf{\phantom{0}2.40} & \phantom{0}4.30 \\
% (25, 20000, 1) & 0.80 & 0.71 & \textbf{0.81} & 0.68 & \textbf{0.87} & 0.76 & 0.78 & 0.64 & \phantom{0}7.00 & \phantom{0}9.90 & \textbf{\phantom{0}6.70} & \phantom{0}9.90 \\
% (25, 20000, 1.5) & 0.63 & 0.53 & \textbf{0.73} & 0.63 & \textbf{0.79} & 0.66 & 0.63 & 0.51 & 24.50 & 25.00 & \textbf{16.50} & 19.50 \\
% (25, 20000, 2) & \textbf{0.73} & 0.48 & 0.61 & 0.59 & \textbf{0.83} & 0.56 & 0.46 & 0.40 & \textbf{25.10} & 39.10 & 33.80 & 34.10 \\
\bottomrule
\multicolumn{7}{l}{\tiny *Each value is the average over ten independent instances.}
\end{tabular}
\end{table}
% In denser graphs (e.g., average in-degree $d = 2$), GOBNILP suffers from a significant and persistent increase in the score gap, a limitation attributed to its restrictive parent-set assumptions. In contrast, CG-DCA maintains stable performance without such constraints, further underscoring its adaptability to complex network structures.
In Table~\ref{res: cont metric}, we compare four methods with the following metrics on the quality of the solution graphs: (1) precision and recall \cite{tillman2011learning}, which measure the proportion of true edges among predicted edges and predicted edges among true edges, respectively; (2) the structural Hamming distance (SHD) \cite{tsamardinos2006max}, which quantifies the dissimilarity between two graphs by counting the required edge additions, deletions, or reversals to make one graph identical to another \cite{andersson1997characterization}.
Prior to computing these metrics, we convert both predicted and true graphs into their corresponding essential graphs \cite{andersson1997characterization}, which represent their Markov equivalence classes. The conversion of essential graphs is performed following the methodology described in \cite{meek2013causal}. We also have results for HC but only present them in Appendix \ref{appendix: result} due to its inferior performance.
Notably, CG-DCA almost outperforms all methods in all metrics on smaller graphs with $n=15$.
CG-DCA achieves better average recall across most instances while maintaining competitive precision and SHD compared to other constraint-based and hybrid approaches. Better performance on recall than on precision and SHD indicates that CG-DCA tends to select denser graphs than others. %As the graph density ($d$) increases, CG-DCA exhibits a slower rise in SHD compared to the other four methods, demonstrating more robust performance across varying graph densities.
% Table~\ref{res: cont metric} presents analysis of structural similarities between the true and predicted graphs.
% Both CG-DCA and GOBNILP methods demonstrate superior performance over the pure Hill-Climbing (HC) algorithm implemented in the \texttt{bnlearn} package, achieving consistently higher similarity scores. The performance gap between CG-DCA and GOBNILP is relatively modest in sparse graphs but becomes more pronounced as graph density increases. Notably, recall scores do not show significant improvement with larger sample sizes (e.g., $N=2{,}0000$) across all methods, suggesting that the BIC score effectively prevents overfitting. This indicates that moderate sample sizes (e.g., $N=5{,}000$) are generally sufficient for network learning at these scales.
% For Gaussian BIC scoring, while GOBNILP maintains good efficiency and performance in small, sparse graphs, CG-DCA exhibits stronger scalability, handling larger networks with moderate sample sizes, increased node counts, and complex edge structures while maintaining computational efficiency.
\subsection{Result on Discrete Data} \label{sec: result discrete}
\begin{table}[tb!]
\centering
\caption{Performance Comparison of CG-DCA, GOBNILP (GI), PC, and MMHC on Discrete Datasets}
\label{res: discrete}
\resizebox{\textwidth}{!}{
\begin{tabular}{lcccc cccc cccc}
\toprule
\multirow{2}{*}{Dataset} & \multicolumn{4}{c}{Precision} & \multicolumn{4}{c}{Recall} & \multicolumn{4}{c}{SHD} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13}
& CG-DCA & GI & PC & MMHC & CG-DCA & GI & PC & MMHC & CG-DCA & GI & PC & MMHC \\
\midrule
LUCAS & 0.77 & \textbf{1.00} & 0.83 & 0.33 & 0.83 & \textbf{1.00} & 0.83 & 0.33 & 3 & \textbf{0} & 2 & 8 \\
INSURANCE & 0.53 & 0.90 & \textbf{0.91} & 0.64 & 0.44 & \textbf{0.83} & 0.58 & 0.35 & 37 & \textbf{11} & 22 & 34 \\
ALARM & 0.08 & 0.06 & 0.10 & \textbf{0.16} & 0.09 & 0.07 & 0.09 & 0.11 & 83 & 82 & 77 & \textbf{67} \\
\bottomrule
\end{tabular}}
\end{table}
Table~\ref{res: discrete} compares four methods on discrete datasets LUCAS~\cite{ethz_lucas} with $(n, N, d) = (12, 2000, 1)$, INSURANCE~\cite{binder1997adaptive} with $(n, N, d) = (27, 20000, 3.85)$ and ALARM~\cite{beinlich1989alarm} with $(n, N, d) = (37, 20000, 2.49)$.
%For score-based method, the objective is the discrete BIC score.
While GOBNILP demonstrates superior performance on small graphs, all the methods fail to produce structurally similar graphs for the larger ALARM dataset.
Although CG-DCA is an option for small discrete dataset, its performance degrades with larger networks. This is because the discrete BIC score contains a highly supermodular penalization term $(a_i-1)\prod_{j\in J}a_j$, making DCA less effective for DS optimization.
%This term causes the second term in pricing objective $H(J) - \Lambda (a_i-1)\prod_{j\in J}a_j$ to deviate significantly from linearity, compromising the method's efficacy.
%Unlike Gaussian data (where each node typically has an average of 30-60 candidate parent sets), DCA in discrete case produces significantly fewer candidates (only 10-30 per node). This happens because the pricing steps only yield suboptimal columns, resulting in a large optimality gap in column generation process.
Thus, development of an improved decomposition strategy for discrete scores (potentially other scores like BDeu \cite{buntine1991theory}) is essential to broaden the applicability of DCA to larger discrete Bayesian networks.
\section{Conclusion} \label{sec: conclusion}
In this paper, we propose CG-DCA, a method that leverages difference-of-submodular minimization to solve the pricing problem within CG framework for BNSL. Empirical results demonstrate that CG-DCA outperforms state-of-the-art score-based methods on simulated Gaussian datasets with varying node sizes, sample sizes, and graph densities, yielding solutions of high quality.
While CG-DCA is a viable approach for small graphs with discrete data, its scalability to larger graphs remains limited due to the high supermodularity induced by the penalty term in the discrete BIC function. To mitigate this limitation, one could explore alternative decomposition strategies for (potentially different) discrete scoring functions or develop new DS optimization techniques tailored to this specific computational challenge. %Addressing this challenge enables CG-DCA to be extended to more complex Bayesian networks, significantly broadening its applicability.
Furthermore, building upon our efficient pricing heuristic, future research could be focused on developing exact pricing algorithms to enhance both the convergence guarantees and solution optimality of CG, by leveraging exact approaches for submodular \cite{lovasz1983submodular} and supermodular \cite{nemhauser1981maximizing} optimization.
\newpage
\bibliography{ref}
\appendix
\section{Definitions}
\subsection{Scoring Functions} \label{appendix: define score}
%\rr{Define $\texttt{score}(G;D)$, i.e., likelihood $L(G;D)$ here as well.}
% As mentioned in Section~\ref{section: preliminaries IP}, the score $\texttt{score}(G;D)$ of a DAG $G$ is able to decompose into node-specific components
We use the $\ell_0$-penalized log-likelihood function as the scoring function of a DAG, which is
\[
\texttt{score}(G; D) := \log(L(G;D)) - \Lambda\cdot k(G)
\] for some $\Lambda\geq0$. Here, $L(G;D)$ is the likelihood of the graph structure $G$ under data $D$, and $k(G)$ is the number of free parameters to be estimated in the graphical model. %The likelihood have been maximized over parameters in $G$ under the assumed distribution.
A crucial property of this scoring function is that it can be decomposed into node-specific local scores\[
\texttt{score}(G;D) = \sum_{i=1}^n \texttt{score}_i(\text{pa}_i(G)),
\] where each local score $\texttt{score}_i(\text{pa}_i(G))$ only depends on a node $i$ and its parent set $\text{pa}_i(G)$.
For continuous data, we assume that variables $\{X_i\}_{i\in V}$ are mean-normalized such that $\bE[X_i]=0$. We also assume that if $J$ is the parent set of node $i$, then $X_i \sim N(\alpha_{i J}^{\top}\mathbf{X}_J, \sigma_{i\gets J}^2)$ \cite{van20130}, where $\mathbf{X}_J=\{X_j\}_{j\in J}$. The parameters $\alpha_{iJ}$ and $\sigma_{i\gets J}^2$ are unknown. The likelihood function defining $\texttt{score}_i(J)$ has included these parameters by their maximum likelihood estimator (MLE), thus its value only depends on $\text{pa}_i(G)$, not the other nodes. The resulting local score $\texttt{score}_i(J)$ in this context is
\[
\texttt{score}_{i}(J) = - \frac{N}{2}(1+\log(2\pi)) - \frac{N}{2}\log(\hat{\sigma}^2_{i\gets J}) - \Lambda |J|,
\]
where
\[
\hat{\sigma}^2_{i\gets J} = \min_{\alpha\in\mathbb{R}^{|J|}} \mathbb{E}_{\hat{\mathbb{P}}}\left[(\alpha^T \mathbf{X}_J - X_i)^2\right]
\]
denotes the empirical residual variance (under the empirical distribution $\hat{\bP}$) of the linear regression predicting $X_i$ (with $\bE[X_i]=0$) from the features $\mathbf{X}_J$. For simplicity, we will ignore the constant $- \frac{N}{2}(1+\log(2\pi))$ in our pricing problem optimization framework established based on Proposition~\ref{prop: continuous}.
For discrete data, we assume that if $J$ is the parent set of node $i$, then $X_i$ follows a multinomial distribution with parameters depending on the configuration of the parent set values $\mathbf{X}_J:=\{X_j\}_{j \in J}$.
The local score for discrete data using multinomial likelihood is
\[
\texttt{score}_{i}(J) = \sum_{x_i\in \mathcal{S}_i} \sum_{\mathbf{x}_J \in \mathcal{S}_J} \#(x_i,\mathbf{x}_J) \log\left(\frac{\#(x_i,\mathbf{x}_J)}{\#(\mathbf{x}_J)}\right) - \Lambda (a_i-1)\prod_{j\in J}a_j,
\]
where $\mathcal{S}_i$ and $\mathcal{S}_J$ represent the sets of possible values that $X_i$ and $\mathbf{X}_J$ can take, respectively. The arity (i.e., the number of possible values it can take) of variable $X_j$ is denoted by $a_j$ for $j=1, \ldots,n$. The count function $\#(x_i,\mathbf{x}_J) = \texttt{Count}(X_i=x_i,\mathbf{X}_J=\mathbf{x}_J)$ counts joint occurrences in the dataset, and $\#(\mathbf{x}_J) = \texttt{Count}(\mathbf{X}_J=\mathbf{x}_J)$ provides the corresponding marginal counts.
\subsection{Submodular Function and Lov\'{a}sz Extension Function} \label{appendix: define submodular}
We adopt standard definitions of the submodular set function and the Lov\'{a}sz extension \cite{lovasz1983submodular}.
\begin{de}[Submodular Set Function]
Let $V$ be a finite ground set. A set function $f: 2^V \rightarrow \mathbb{R}$ is \textit{submodular} if it satisfies the diminishing return property, i.e., for all $A \subseteq B \subseteq V$ and $v \in V \setminus B$,
\[
f(A \cup \{v\}) - f(A) \geq f(B \cup \{v\}) - f(B)
\]
A function is \textit{supermodular} if $-f$ is submodular, and is \textit{modular} if it is both submodular and supermodular.
\end{de}
\begin{de}[Lov\'{a}sz Extension]
% Given a set function $f:2^\Omega \to \mathbb{R}$, $|\Omega|=d$, its Lov\'{a}sz extension function $z^L:[0,1]^{d} \to \mathbb{R}$ is defined as:
% \[
% z^L(x) = \sum_{i=1}^{n} \lambda_i f(S_i)
% \]
% where $\lambda$ $x = \sum_{i=1}^n \lambda_i \mathbf{1}_{S_i}$ represents a convex combination with $\sum_{i=1}^n \lambda_i = 1$, $\lambda_i \geq 0$, and $\emptyset = S_0 \subset S_1 \subset \cdots \subset S_n = \Omega$ forms a chain of subsets.
Let $f: 2^V \to \mathbb{R} $ be a set function defined on a ground set $ V = \{1, \dots, d\} $. Given a point $ x \in [0,1]^d$, let $\sigma$ be a permutation of $V$ such that
%the Lov\'{a}sz extension $ f^L: \mathbb{R}^d \to \mathbb{R} $ is defined as follows:
% Let $S_d$ denote the set of permutations on $V$. Given $ \sigma \in S_d $ satisfying
\[
x_{\sigma(1)} \geq x_{\sigma(2)} \geq \cdots \geq x_{\sigma(d)}.
\]
Define the nested subsets $ S^\sigma_k = \{\sigma(1), \dots, \sigma(k)\} $ for $ k \in \{1, \dots, d\} $, and $ S^\sigma_0 = \emptyset $.
The Lov\'{a}sz extension $f^L:[0,1]^n\rightarrow\bR$ of the set function $f$ at point $x$ is defined as \[ f^L(x) = \sum_{k=0}^d \big( x_{\sigma(k)} - x_{\sigma(k+1)}\big)f(S^\sigma_k),
\]
where $x_{\sigma(0)}:=1$ and $x_{\sigma(d+1)}:= 0$.
\end{de}
\section{Row Generation and the Separation Problem} \label{appendix: row generation}
The exponential number of cluster constraints can be handled through row generation \cite{10.5555/3020548.3020567}. This cutting plane approach starts with a restricted set $\hat{\bC}$ and sequentially adds constraints violated by the current solution $x^*$.
For an integer solution $x^*$, the separation problem identifies violated cluster constraints via cycle detection in the decoded graph from $x^*$ (using, e.g., depth-first search).
For a fractional solution of RMLP, the separation problem identifies maximally violated cluster constraints through
\[
\max_C \left\{ \sum_{i\in C} \sum_{J\cap C \neq \emptyset} x^*_{i\gets J} - |C| \right\},
\]
which can be formulated as an \emph{Separation IP} as follows:
\begin{subequations} \label{prb: separation IP}
\begin{alignat}{3}
\max_{y,z} \quad & \sum_{i=1}^n \sum_{J\in \hat{\bbP}_i} x^*_{i\gets J}\cdot y_{iJ} -\sum_{i=1}^n z_i\\
\text{s.t.} \quad & y_{iJ} \leq z_i , && i=1,\ldots,n, \quad J\in \hat{\bbP}_i,\\
& y_{iJ} \leq \sum_{i'\in J} z_{i'} , && i=1,\ldots,n, \quad J\in \hat{\bbP}_i,\\
& \sum_{i=1}^n z_i \geq 1,\\
& y_{iJ}\in \{0,1\}, z_i\in \{0,1\},&& i=1,\ldots,n, \quad J\in \hat{\bbP}_i.
\end{alignat}
\end{subequations}
The optimal solution $(y^*, z^*)$ of Problem~\eqref{prb: separation IP} defines the new cluster $C^* = \{i: z_i^* = 1\}$ to be added to $\hat{\bC}$.
The complete row and column generation algorithm iteratively alternates between column generation for variable selection and row generation for constraint enforcement, dynamically refining both the solution space and constraint set.
\section{Proofs} \label{appendix: proof}
\subsection{Proof of Proposition~\ref{prop: continuous}}
\begin{proof}
Recall that for Gaussian data,\[
\texttt{score}_i(J) = -\frac{N}{2}\log(\hat{\sigma}^2_{i\gets J}) - \Lambda |J|.
\]
The conditional variance $\hat{\sigma}^2_{i\gets J}$ satisfies:
\[
\begin{aligned}
\hat{\sigma}^2_{i\gets J} &= \min_{\alpha\in\bR^{|J|}} \bE_{\hat{\bP}}[(\alpha^T \mathbf{X}_J - X_i)^2] \\
&= \min_{\alpha \in \bR^{|J|}} \alpha^T \hat{\Sigma}_{J,J} \alpha - 2\alpha^T \hat{\Sigma}_{J,i} + \hat{\Sigma}_{i,i} \\
&= \hat{\Sigma}_{i,i} - \hat{\Sigma}_{i,J}\hat{\Sigma}_{J,J}^{-1}\hat{\Sigma}_{J,i} \\
&= \det(\hat{\Sigma}_{J\cup i, J\cup i})/\det(\hat{\Sigma}_{J,J}),
\end{aligned}
\]
where the last equality is due to the Schur complement, $\hat{\bP}$ denotes the empirical distribution, with $\hat{\Sigma}_{J, J}, \hat{\Sigma}_{i, J}, \hat{\Sigma}_{J\cup i, J\cup i}$ and $\hat{\Sigma}_{i,i}$ representing the empirical covariance matrices of variable set $\mathbf{X}_J$, cross-covariance between $X_i$ and $\mathbf{X}_J$, joint covariance of $\mathbf{X}_J\cup X_i$ and variance of $X_i$, respectively.
% For mean-normalized data, we have $\hat{\Sigma}_{J,J} = \bE_{\hat{\bP}}[X_JX_J^T] \in \bR^{|J|\times |J|}$, $\hat{\Sigma}_{J,i} = \bE_{\hat{\bP}}[X_JX_i] \in \bR^{|J|}$, and $\hat{\Sigma}_{i,i} = \bE_{\hat{\bP}}[X_i^2] \in \bR$.
The logarithmic transformation yields
\[
\log(\hat{\sigma}^2_{i\gets J}) = \log\det(\hat{\Sigma}_{J\cup i, J\cup i}) - \log\det(\hat{\Sigma}_{J,J}),
\]
where both $\log\det(\hat{\Sigma}_{J,J})$ and $\log\det(\hat{\Sigma}_{J\cup i,J\cup i})$ are submodular functions of set $J$ \cite{kelmans1983multiplicative}. Function $\Lambda|J|+\lambda^*_i$ is modular in $J$.
The remaining term involving $J$, $\sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^*$, in the pricing objective is also submodular in $J$. This follows from the property that for any $J_1 \subseteq J_2$ and $j \notin J_2$,
\[
\sum_{C\in\hat{\bC}:~i\in C,~J_1\cap C=\emptyset} \lambda^*_C \geq \sum_{C\in\hat{\bC}:~i\in C,~J_2\cap C=\emptyset} \lambda^*_C.
\]
Given that $\lambda^*_C \geq 0$, the above inequality holds since $J_1 \subseteq J_2$ implies that $C \cap J_2 = \emptyset$ necessitates $C \cap J_1 = \emptyset$.
Consequently, the pricing objective to be minimized can be expressed as the following DS function:
\[
z(J;\lambda^*) = \underbrace{\log\det(\hat{\Sigma}_{J\cup i, J\cup i}) + \Lambda|J| + \sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^* + \lambda^*_i}_{\text{submodular}} - \underbrace{\log\det(\hat{\Sigma}_{J,J})}_{\text{submodular}}.
\]
\end{proof}
\subsection{Proof of Proposition~\ref{prop: discrete}}
\begin{proof}
Recall that for multinomial data,\[
\texttt{score}_i(J) = \sum_{x_i\in \mathcal{S}_i} \sum_{\mathbf{x}_J \in \mathcal{S}_J} \#(x_i,\mathbf{x}_J) \log\left(\frac{\#(x_i,\mathbf{x}_J)}{\#(\mathbf{x}_J)}\right) - \Lambda (a_i-1)\prod_{j\in J}a_j.
\]
We reformulate the first term as follows:
\[
\begin{aligned}
&\sum_{x_i\in S_i} \sum_{x_J \in S_J} \#(x_i,\mathbf{x}_J) \log\left(\frac{\#(x_i,\mathbf{x}_J)}{\#(\mathbf{x}_J)}\right) \\
=& \sum_{x_i\in S_i} \sum_{x_J \in S_J}\#(x_i,\mathbf{x}_J) \log(\#(x_i,\mathbf{x}_J)) - \sum_{x_J\in S_J}\#(\mathbf{x}_J)\log(\#(\mathbf{x}_J)) \\
=& N\left[ \sum_{x_i\in S_i} \sum_{x_J \in S_J}\hat{\bP}(X_i=x_i,X_J=x_J)\log\hat{\bP}(X_i=x_i,X_J=x_J) - \sum_{x_J \in S_J}\hat{\bP}(X_J = x_J)\log \hat{\bP}(X_J=x_J)\right] \\
&\quad \\
=& N(-H(J \cup \{i\}) + H(J)),
\end{aligned}
\]
where $N$ is the sample size. The functions \[H(J) := -\sum_{x_J \in S_J}\hat{\bP}(X_J=x_J)\log \hat{\bP}(X_J=x_J)\] and \[H(J\cup\{i\}) := -\sum_{x_i\in S_i}\sum_{x_J \in S_J}\hat{\bP}(X_i=x_i,X_J=x_J)\log \hat{\bP}(X_i=x_i,X_J=x_J)\] denote the entropy functions, which are known to be submodular \cite{fujishige1978polymatroidal}.
By the proof of Proposition \ref{prop: continuous}, $\sum_{\substack{C \in \hat{\bC}: \\ i \in C,~ J \cap C \neq \emptyset}} \lambda_C^*$ is submodular in $J$. It is also easy to verify that $(a_i-1)\prod_{j\in J}a_j$ is supermodular in $J$ as $a_j\geq 1$ for $j \in J\cup \{i\}$.
Therefore, the pricing objective can be expressed as the following DS function:
\[
z(J;\lambda^*) = \underbrace{H(J \cup \{i\}) + \sum_{\substack{C:C\in\bC\\i\in C\\C\cap J\neq \emptyset}} \lambda_C^*+\lambda^*_i}_{\text{submodular}} - \underbrace{\left(H(J) - \Lambda (a_i-1)\prod_{j\in J} a_j\right)}_{\text{submodular}}.
\]
\end{proof}
\section{Supplementary Numerical Results} \label{appendix: result}
\begin{table}[tb!]
\centering
\caption{Performance of CG-DCA, GOBNILP (GI), HC, PC, and MMHC on Gaussian Datasets} \label{table: complete continuous}
\resizebox{\textwidth}{!}{
\begin{tabular}{l c c c c c c c c c c c c c c c}
\toprule
\multirow{2}{*}{$n, N, d$} & \multicolumn{5}{c}{Precision} & \multicolumn{5}{c}{Recall} & \multicolumn{5}{c}{SHD} \\
\cmidrule(lr){2-6} \cmidrule(lr){7-11} \cmidrule(lr){12-16}
& CG-DCA & GI & HC & PC & MMHC & CG-DCA & GI & HC & PC & MMHC & CG-DCA & GI & HC & PC & MMHC \\
\midrule
(15, 5000, 0.5) & \textbf{0.71} & 0.66 & 0.36 & 0.59 & 0.50 & \textbf{0.79} & 0.73 & 0.47 & 0.62 & 0.51 & \textbf{\phantom{0}2.60} & \phantom{0}2.90 & \phantom{0}6.20 & \phantom{0}3.00 & \phantom{0}3.50 \\
(15, 5000, 1) & \textbf{0.78} & 0.67 & 0.41 & 0.72 & 0.59 & \textbf{0.84} & 0.72 & 0.52 & 0.67 & 0.54 & \textbf{\phantom{0}4.10} & \phantom{0}5.00 & 11.90 & \phantom{0}4.80 & \phantom{0}6.70 \\
(15, 5000, 1.5) & \textbf{0.75} & 0.73 & 0.42 & 0.61 & 0.56 & \textbf{0.82} & 0.80 & 0.57 & 0.54 & 0.48 & \textbf{\phantom{0}6.80} & \phantom{0}7.40 & 19.50 & 11.50 & 12.20 \\
(15, 5000, 2) & \textbf{0.61} & 0.58 & 0.34 & 0.54 & 0.52 & \textbf{0.74} & 0.65 & 0.57 & 0.39 & 0.36 & 17.90 & \textbf{16.90} & 35.30 & 20.90 & 20.40 \\
(20, 5000, 0.5) & 0.74 & 0.71 & 0.48 & \textbf{0.76} & 0.59 & 0.77 & 0.75 & 0.56 & \textbf{0.79} & 0.60 & \phantom{0}2.90 & \phantom{0}3.50 & \phantom{0}6.80 & \textbf{\phantom{0}2.60} & \phantom{0}4.40 \\
(20, 5000, 1) & 0.72 & 0.69 & 0.49 & \textbf{0.75} & 0.65 & \textbf{0.79} & 0.77 & 0.63 & 0.71 & 0.59 & \phantom{0}7.00 & \phantom{0}7.50 & 13.20 & \textbf{\phantom{0}5.90} & \phantom{0}7.90 \\
(20, 5000, 1.5) & 0.61 & 0.61 & 0.41 & 0.62 & \textbf{0.65} & \textbf{0.79} & 0.71 & 0.64 & 0.50 & 0.49 & 17.50 & \textbf{16.80} & 30.60 & 17.90 & \textbf{16.80} \\
(20, 5000, 2) & \textbf{0.58} & 0.46 & 0.41 & 0.54 & 0.55 & \textbf{0.77} & 0.53 & 0.67 & 0.34 & 0.31 & \textbf{30.00} & 33.60 & 49.80 & 31.50 & 31.10 \\
(25, 5000, 0.5) & 0.74 & 0.61 & 0.60 & \textbf{0.84} & 0.69 & 0.79 & 0.67 & 0.67 & \textbf{0.84} & 0.69 & \phantom{0}3.10 & \phantom{0}5.90 & \phantom{0}6.70 & \textbf{\phantom{0}2.20} & \phantom{0}4.40 \\
(25, 5000, 1) & 0.78 & 0.69 & 0.57 & \textbf{0.80} & 0.74 & \textbf{0.86} & 0.76 & 0.71 & 0.75 & 0.69 & \phantom{0}9.20 & 11.10 & 16.30 & \textbf{\phantom{0}7.40} & \phantom{0}8.70 \\
(25, 5000, 1.5) & 0.63 & 0.57 & 0.44 & 0.70 & \textbf{0.71} & \textbf{0.82} & 0.68 & 0.65 & 0.58 & 0.56 & 27.50 & 22.10 & 39.50 & 18.50 & \textbf{18.00} \\
(25, 5000, 2) & \textbf{0.61} & 0.47 & 0.38 & 0.53 & 0.59 & \textbf{0.78} & 0.54 & 0.67 & 0.35 & 0.35 & 40.30 & 40.30 & 63.70 & 38.70 & \textbf{36.50} \\
(15, 20000, 0.5) & \textbf{0.71} & 0.69 & 0.50 & 0.68 & 0.53 & 0.71 & \textbf{0.73} & 0.52 & 0.68 & 0.53 & \textbf{\phantom{0}1.90} & \phantom{0}2.20 & \phantom{0}3.70 & \textbf{\phantom{0}1.90} & \phantom{0}3.00 \\
(15, 20000, 1) & 0.75 & \textbf{0.78} & 0.49 & 0.71 & 0.61 & 0.78 & \textbf{0.81} & 0.61 & 0.69 & 0.59 & \phantom{0}4.10 & \textbf{\phantom{0}4.00} & \phantom{0}9.60 & \phantom{0}4.70 & \phantom{0}6.20 \\
(15, 20000, 1.5) & \textbf{0.74} & 0.73 & 0.47 & 0.72 & 0.66 & \textbf{0.83} & 0.79 & 0.61 & 0.66 & 0.57 & \textbf{\phantom{0}8.10} & \phantom{0}9.40 & 18.30 & \phantom{0}8.60 & 10.00 \\
(15, 20000, 2) & \textbf{0.64} & 0.53 & 0.38 & 0.53 & 0.55 & \textbf{0.79} & 0.60 & 0.62 & 0.43 & 0.39 & \textbf{16.00} & 19.90 & 33.20 & 20.01 & 19.80 \\
(20, 20000, 0.5) & 0.72 & \textbf{0.80} & 0.54 & 0.73 & 0.58 & 0.76 & \textbf{0.82} & 0.62 & 0.76 & 0.59 & \phantom{0}3.60 & \textbf{\phantom{0}2.60} & \phantom{0}5.70 & \phantom{0}3.00 & \phantom{0}4.60 \\
(20, 20000, 1) & 0.70 & 0.73 & 0.56 & \textbf{0.79} & 0.68 & 0.78 & \textbf{0.79} & 0.67 & 0.77 & 0.65 & \phantom{0}6.50 & \phantom{0}6.90 & 10.80 & \textbf{\phantom{0}4.90} & \phantom{0}7.10 \\
(20, 20000, 1.5) & 0.57 & 0.58 & 0.40 & \textbf{0.64} & 0.63 & \textbf{0.77} & 0.68 & 0.64 & 0.56 & 0.50 & 21.20 & 18.20 & 32.80 & \textbf{16.00} & 16.50 \\
(20, 20000, 2) & 0.54 & 0.48 & 0.41 & 0.51 & \textbf{0.56} & \textbf{0.79} & 0.55 & 0.67 & 0.37 & 0.33 & 31.30 & 32.40 & 46.00 & 31.30 & \textbf{29.30} \\
(25, 20000, 0.5) & 0.77 & 0.80 & 0.59 & \textbf{0.81} & 0.70 & 0.79 & \textbf{0.84} & 0.66 & 0.83 & 0.70 & \phantom{0}3.10 & \phantom{0}2.80 & \phantom{0}7.30 & \textbf{\phantom{0}2.40} & \phantom{0}4.30 \\
(25, 20000, 1) & 0.80 & 0.71 & 0.51 & \textbf{0.81} & 0.68 & \textbf{0.87} & 0.76 & 0.67 & 0.78 & 0.64 & \phantom{0}7.00 & \phantom{0}9.90 & 17.10 & \textbf{\phantom{0}6.70} & \phantom{0}9.90 \\
(25, 20000, 1.5) & 0.63 & 0.53 & 0.40 & \textbf{0.73} & 0.63 & \textbf{0.79} & 0.66 & 0.62 & 0.63 & 0.51 & 24.50 & 25.00 & 41.90 & \textbf{16.50} & 19.50 \\
(25, 20000, 2) & \textbf{0.73} & 0.48 & 0.35 & 0.61 & 0.59 & \textbf{0.83} & 0.56 & 0.64 & 0.46 & 0.40 & \textbf{25.10} & 39.10 & 77.00 & 33.80 & 34.10 \\
\bottomrule
\multicolumn{7}{l}{ *Each value is the average over ten independent instances.}
\end{tabular}}
\end{table}
\begin{table}[tb!]
\centering
\caption{Performance Comparison of CG-DCA, GOBNILP (GI), HC, PC, and MMHC on Discrete Datasets} \label{table: complete discrete}
\tiny
\begin{tabular}{lccccc ccccc ccccc}
\toprule
\multirow{2}{*}{Dataset} & \multicolumn{5}{c}{Precision} & \multicolumn{5}{c}{Recall} & \multicolumn{5}{c}{SHD} \\
\cmidrule(lr){2-6} \cmidrule(lr){7-11} \cmidrule(lr){12-16}
& CG-DCA & GI & HC & PC & MMHC & CG-DCA & GI & HC & PC & MMHC & CG-DCA & GI & HC & PC & MMHC \\
\midrule
LUCAS & 0.77 & \textbf{1.00} & 0.29 & 0.83 & 0.33 & 0.83 & \textbf{1.00} & 0.33 & 0.83 & 0.33 & 3 & \textbf{0} & 10 & 2 & 8 \\
INSURANCE & 0.53 & 0.90 & 0.52 & \textbf{0.91} & 0.64 & 0.44 & \textbf{0.83} & 0.50 & 0.58 & 0.35 & 37 & \textbf{11} & 38 & 22 & 34 \\
ALARM & 0.08 & 0.06 & 0.13 & 0.10 & \textbf{0.16} & 0.09 & 0.07 & \textbf{0.15} & 0.09 & 0.11 & 83 & 82 & 81 & 77 & \textbf{67} \\
\bottomrule
\end{tabular}
\end{table}
Tables~\ref{table: complete continuous} and \ref{table: complete discrete} present the comprehensive experimental results including the HC method, with $N\in\{5000,20000\}$ for Gaussian datasets.
\if0
\newpage
\section*{NeurIPS Paper Checklist}
% %%% BEGIN INSTRUCTIONS %%%
% The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
% limit.
% Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
% \begin{itemize}
% \item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
% \item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
% \item Please provide a short (1–2 sentence) justification right after your answer (even for NA).
% % \item {\bf The papers not including the checklist will be desk rejected.}
% \end{itemize}
% {\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
% The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "\answerYes{}" is generally preferable to "\answerNo{}", it is perfectly acceptable to answer "\answerNo{}" provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "\answerNo{}" or "\answerNA{}" is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
% IMPORTANT, please:
% \begin{itemize}
% \item {\bf Delete this instruction block, but keep the section heading ``NeurIPS Paper Checklist"},
% \item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
% \item {\bf Do not modify the questions and only use the provided macros for your answers}.
% \end{itemize}
% %%% END INSTRUCTIONS %%%
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: In the abstract and introduction (Section \ref{sec: intro}), we clearly state the contributions and scope of this paper.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The limitations of our work is mainly discussed in Section \ref{sec: result discrete} and in the conclusion (Section \ref{sec: conclusion})
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate "Limitations" section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory assumptions and proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our theoretical results are in Proposition \ref{prop: continuous} and \ref{prop: discrete}. The proofs are in Appendix \ref{appendix: proof}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental result reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The implementation details of our algorithm and the experiment settings are fully discussed, which can be found in Section \ref{sec: dca}, \ref{sec: integrate cg rg} and \ref{sec: numerical}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We provide the access to the code of our method and the code for generating data.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental setting/details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We specify all the implementation details that is necessary for understanding the results.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment statistical significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerNo{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our reported metrics for numerical experiments are discussed in Section \ref{sec: numerical}, which does not include error bars or confidence intervals.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
\item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments compute resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: See Section \ref{sec: numerical}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code of ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our research conform with the NeurIPS Code of Ethics in every respect.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: There is no societal impact of our work.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that there is no societal impact of the work performed.
\item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The paper poses no such risks.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Properly credited.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We provide details of the code as part of our submissions.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and research with human subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper does not involve crowdsourcing nor research with human subjects.%\justificationTODO{}
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our paper does not involve crowdsourcing nor research with human subjects.%\justificationTODO{}
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\item {\bf Declaration of LLM usage}
\item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
%this research?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The core method development in this research does not involve LLMs as any important, original or non-standard components.%\justificationTODO{}
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
\item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
\end{itemize}
\end{enumerate}
\fi
\end{document}
% \begin{table}[htbp]
% \centering
% \caption{Comparison of CG-DCA, GOBNILP, and HC (bnlearn) methods on Precision, Recall, and SHD metrics}
% \label{res: cont metric}
% \begin{tabular}{lSSSSSSSSS}
% \toprule
% \textbf{Graph} & \multicolumn{3}{c}{\textbf{Precision}} & \multicolumn{3}{c}{\textbf{Recall}} & \multicolumn{3}{c}{\textbf{SHD}} \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
% & {\small{CG-DCA}} & {\small{GOBNILP}} & {\small{HC}} & {\small{CG-DCA}} & {\small{GOBNILP}} & {\small{HC}} & {\small{CG-DCA}} & {\small{GOBNILP}} & {\small{HC}} \\
% \midrule
% (15, 5000, 1) & \textbf{0.76} & 0.73 & 0.45 & \textbf{0.79} & 0.76 & 0.56 & \textbf{\phantom{0}6.7} & \phantom{0}7.2 & 17.2 \\
% (15, 5000, 1.5) & \textbf{0.77} & 0.72 & 0.47 & \textbf{0.84} & 0.80 & 0.63 & \textbf{10.2} & 11.9 & 23.8 \\
% (15, 5000, 2) & \textbf{0.70} & 0.57 & 0.33 & \textbf{0.81} & 0.66 & 0.55 & \textbf{21.1} & 23.7 & 51.8 \\
% (20, 5000, 1) & \textbf{0.80} & 0.75 & 0.47 & \textbf{0.87} & 0.82 & 0.63 & \textbf{\phantom{0}7.7} & \phantom{0}9.2 & 21.2 \\
% (20, 5000, 1.5) & \textbf{0.63} & 0.61 & 0.36 & \textbf{0.81} & 0.69 & 0.59 & \textbf{21.9} & 23.4 & 46.7 \\
% (20, 5000, 2) & \textbf{0.59} & 0.48 & 0.26 & \textbf{0.78} & 0.52 & 0.51 & \textbf{35.8} & 43.6 & 97.0 \\
% (25, 5000, 1) & \textbf{0.84} & 0.66 & 0.48 & \textbf{0.89} & 0.75 & 0.66 & \textbf{\phantom{0}7.3} & 17.8 & 28.5 \\
% (25, 5000, 1.5) & \textbf{0.69} & 0.60 & 0.34 & \textbf{0.83} & 0.71 & 0.58 & \textbf{23.3} & 29.1 & 70.8 \\
% (25, 5000, 2) & \textbf{0.62} & 0.46 & 0.31 & \textbf{0.80} & 0.53 & 0.64 & \textbf{41.2} & 54.4 & 97.6 \\
% (15, 20000, 1) & \textbf{0.71} & 0.70 & 0.49 & \textbf{0.75} & 0.74 & 0.59 & \phantom{0}8.3 & \textbf{\phantom{0}7.7} & 15.3 \\
% (15, 20000, 1.5) & \textbf{0.80} & 0.68 & 0.47 & \textbf{0.88} & 0.75 & 0.63 & \textbf{\phantom{0}8.4} & 14.0 & 24.6 \\
% (15, 20000, 2) & \textbf{0.80} & 0.53 & 0.38 & \textbf{0.86} & 0.60 & 0.59 & \textbf{10.7} & 26.4 & 45.0 \\
% (20, 20000, 1) & \textbf{0.81} & 0.76 & 0.48 & \textbf{0.84} & 0.81 & 0.64 & \textbf{\phantom{0}6.7} & \phantom{0}9.3 & 21.3 \\
% (20, 20000, 1.5) & \textbf{0.63} & 0.57 & 0.38 & \textbf{0.78} & 0.68 & 0.63 & \textbf{23.9} & 26.1 & 47.4 \\
% (20, 20000, 2) & \textbf{0.57} & 0.49 & 0.33 & \textbf{0.77} & 0.56 & 0.66 & \textbf{37.0} & 41.7 & 76.6 \\
% (25, 20000, 1) & \textbf{0.79} & 0.70 & 0.51 & \textbf{0.86} & 0.77 & 0.67 & \textbf{11.5} & 17.3 & 26.4 \\
% (25, 20000, 1.5) & \textbf{0.72} & 0.57 & 0.39 & \textbf{0.84} & 0.69 & 0.64 & \textbf{23.4} & 31.3 & 57.6 \\
% (25, 20000, 2) & \textbf{0.67} & 0.44 & 0.29 & \textbf{0.81} & 0.51 & 0.58 & \textbf{37.7} & 57.7 & 116.6 \\
% \bottomrule
% \end{tabular}
% \label{tab:comparison}
% \end{table}
% \begin{table}[htbp]
% \centering
% \caption{Comparison of CG-DCA and GOBNILP methods on BIC Score Gap and Time}
% \label{res: cont score}
% \begin{tabular}{lSSSS}
% \toprule
% \textbf{Graph} & \multicolumn{2}{c}{\textbf{BIC score Gap}} & \multicolumn{2}{c}{\textbf{Time (s)}} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5}
% & {CG-DCA} & {GOBNILP} & {CG-DCA} & {GOBNILP} \\
% \midrule
% (15, 5000, 1) & \textbf{0.01\%} & 0.29\% & \textbf{\phantom{00}23.42} & \phantom{00}23.78 \\
% (15, 5000, 1.5) & \textbf{0.07\%} & 0.85\% & \phantom{00}84.33 & \textbf{\phantom{00}57.78} \\
% (15, 5000, 2) & \textbf{0.12\%} & 1.88\% & \phantom{0}262.19 & \textbf{\phantom{00}99.75} \\
% (20, 5000, 1) & \textbf{0.01\%} & 0.34\% & \textbf{\phantom{00}90.55} & \phantom{0}178.35 \\
% (20, 5000, 1.5) & \textbf{0.13\%} & 1.26\% & \textbf{\phantom{0}351.25} & \phantom{0}896.13 \\
% (20, 5000, 2) & \textbf{0.56\%} & 4.00\% & \textbf{1393.48} & 6787.14 \\
% (25, 5000, 1) & \textbf{0.01\%} & 0.42\% & \textbf{\phantom{0}304.09} & 1682.05 \\
% (25, 5000, 1.5) & \textbf{0.19\%} & 1.90\% & \textbf{1247.19} & 4019.12 \\
% (25, 5000, 2) & \textbf{0.64\%} & 5.04\% & \textbf{3556.68} & 9245.01 \\
% (15, 20000, 1) & \textbf{0.00\%} & 0.25\% & \phantom{00}56.76 & \textbf{\phantom{00}20.11} \\
% (15, 20000, 1.5)& \textbf{0.03\%} & 0.61\% & \phantom{0}144.97 & \textbf{\phantom{00}83.76} \\
% (15, 20000, 2) & \textbf{0.16\%} & 1.69\% & \phantom{0}308.17 & \textbf{\phantom{0}156.16} \\
% (20, 20000, 1) & \textbf{0.01\%} & 0.24\% & \textbf{\phantom{0}198.38} & \phantom{0}384.27 \\
% (20, 20000, 1.5)& \textbf{0.16\%} & 1.30\% & \textbf{\phantom{0}579.17} & 1121.58 \\
% (20, 20000, 2) & \textbf{0.50\%} & 3.40\% & \textbf{1421.64} & 3853.36 \\
% (25, 20000, 1) & \textbf{0.03\%} & 0.68\% & \textbf{\phantom{0}657.87} & 1672.36 \\
% (25, 20000, 1.5)& \textbf{0.10\%} & 1.74\% & \textbf{2549.34} & 5601.43 \\
% (25, 20000, 2) & \textbf{0.63\%} & 4.66\% & \textbf{3966.22} & 7801.68 \\
% \bottomrule
% \end{tabular}
% \label{tab:comparison}
% \end{table}
% \begin{table}[htbp] \label{res: continuous}
% \centering
% \caption{Comparison between CG-DCA and GOBNILP: Score Gap and Time}
% \label{tab:comparison1}
% \begin{tabular}{lccccc}
% \toprule
% \multirow{2}{*}{$(n,N,d)$} & \multicolumn{2}{c}{Score Gap} & \multicolumn{2}{c}{Time (s)} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5}
% & CG-DCA & GOBNILP & CG-DCA & GOBNILP \\
% \midrule
% (15,5000,2) & \textbf{0.0203} & 0.0347 & 189.12 & \textbf{91.72} \\
% (20,5000,2) & \textbf{0.0174} & 0.0410 & \textbf{2663.87} & 3584.89 \\
% (25,5000,2) & \textbf{0.0154} & 0.0574 & \textbf{5962.43} & 9352.92 \\
% \bottomrule
% \end{tabular}
% \end{table}
% % Second table for Precision, Recall and SHD
% \begin{table}[htbp]
% \centering
% \caption{Comparison between CG-DCA and GOBNILP: Precision, Recall and SHD}
% \label{tab:comparison2}
% \begin{tabular}{lcccccc}
% \toprule
% \multirow{2}{*}{$(n,N,d)$} & \multicolumn{2}{c}{Precision} & \multicolumn{2}{c}{Recall} & \multicolumn{2}{c}{SHD} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
% & CG-DCA & GOBNILP & CG-DCA & GOBNILP & CG-DCA & GOBNILP \\
% \midrule
% (15,5000,2) & \textbf{0.7} & 0.51 & \textbf{0.78} & 0.59 & \textbf{18.9} & 27 \\
% (20,5000,2) & \textbf{0.564} & 0.531 & \textbf{0.736} & 0.559 & \textbf{35} & 38.6 \\
% (25,5000,2) & \textbf{0.634} & 0.499 & \textbf{0.785} & 0.585 & \textbf{39.2} & 48.9 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{table}[h]
% \centering
% \caption{Performance Comparison between DCA and GOBNILP with $(n, N, d)=(15, 5000, 0.13)$}
% \label{tab:results}
% \begin{tabular}{lccccccc}
% \toprule
% \multirow{3}{*}{Index} %& \multicolumn{5}{c}{BIC Score and Optimality Gap} & \multicolumn{2}{c}{Time (s)} \\
% %\cmidrule(lr){2-6} \cmidrule(lr){7-8}
% & \multicolumn{3}{c}{DCA} & \multicolumn{3}{c}{GOBNILP} & True Graph \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7}
% & Score & Gap (\%) & Time (s) & Score & Gap (\%) & Time (s) & Score\\
% \midrule
% 1 & -107044.94 & 0.25 & 115.19 & -107732.14 & 0.89 & 26.63 & -106778.48 \\
% 2& -106464.54 & \textbf{0.01} & 81.30 & -106582.69 & 0.12 & 59.96 & -106456.82 \\
% 3 & -106907.53 & 0.27 & 109.95 & -107142.70 & 0.49 & 243.94 & -106616.83 \\
% 4 & -106379.91 & \textbf{-0.00} & 114.44 & -107851.90 & 1.38 & 159.08 & -106380.27\\
% 5 & -107477.02 & 0.87 & 161.01 & -110700.58 & 3.90 & 41.92 & -106549.36\\
% 6 & -107916.57 & 1.46 & 324.89 & -107802.47 & 1.36 & 121.12 & -106359.44\\
% 7& -106302.43 & \textbf{-0.00} & 72.85 & -107703.95 & 1.32 & 90.70 & -106303.61\\
% 8 & -106511.79 & 0.17 & 108.03 & -108611.57 & 2.15 & 30.70 & -106326.22\\
% 9 & -106639.08 & \textbf{-0.00} & 78.60 & -108662.64 & 1.90 & 36.63 & -106639.06\\
% 10 & -107352.52 & 0.60 & 163.49 & -109011.89 & 2.15 & 133.20 & -106715.83 \\
% \bottomrule
% \end{tabular}
% \begin{tabular}{lcccccc}
% \toprule
% \multirow{3}{*}{Index} & \multicolumn{2}{c}{Precision} & \multicolumn{2}{c}{Recall} & \multicolumn{2}{c}{SHD} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
% & DCA & GOBNILP & DCA & GOBNILP & DCA & GOBNILP \\
% \midrule
% 1 & 0.70 & 0.70 & \textbf{0.95} & 0.80 & \textbf{9} & 11 \\
% 2 & 0.78 & 0.63 & \textbf{0.84} & 0.76 & 10 & 17 \\
% 3 & 0.28 & 0.53 & 0.50 & 0.61 & 50 & 26 \\
% 4 & \textbf{0.81} & 0.58 & \textbf{0.81} & 0.67 & 10 & 22 \\
% 5 & 0.43 & 0.39 & 0.63 & 0.37 & 46 & 46 \\
% 6 & 0.29 & 0.52 & 0.50 & 0.57 & 51 & 29 \\
% 7 & \textbf{0.91} & 0.52 & \textbf{0.91} & 0.65 & \textbf{4} & 22 \\
% 8 & 0.59 & 0.31 & \textbf{0.83} & 0.42 & 18 & 36 \\
% 9 & \textbf{0.88} & 0.30 & \textbf{0.88} & 0.42 & \textbf{6} & 37 \\
% 10 & 0.42 & 0.61 & 0.71 & 0.65 & 40 & 24 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{table}[h]
% \centering
% \caption{Performance Comparison between DCA and GOBNILP with $(n, N, d)=(20, 5000, 0.10)$}
% \label{tab:results}
% \begin{tabular}{lccccccc}
% \toprule
% \multirow{3}{*}{Index} %& \multicolumn{5}{c}{BIC Score and Optimality Gap} & \multicolumn{2}{c}{Time (min)} \\
% %\cmidrule(lr){2-6} \cmidrule(lr){7-8}
% & \multicolumn{3}{c}{DCA} & \multicolumn{3}{c}{GOBNILP} & True Graph \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7}
% & Score & Gap (\%) & Time (min) & Score & Gap (\%) & Time (min) & Score\\
% \midrule
% 1 & -142341.18 & 0.24 & 61.02 & -143916.74 & 1.34 & \phantom{0}27.77 & -142006.46 \\
% 2 & -142229.26 & \textbf{0.00} & 99.49 & -147610.22 & 3.79 & 50.43 & -142225.24 \\
% 3 & -142690.42 & 0.37 & 118.17 & -143738.16 & 1.11 & 105.50 & -142163.68 \\
% 4 & -142217.51 & \textbf{0.00} & 19.85 & -142207.73 & \textbf{-0.00} & 160.65 & -142207.74 \\
% 5 & -142353.36 & 0.35 & 137.15 & -147514.22 & 3.98 & 31.22 & -141863.02 \\
% 6 & -142713.40 & 0.30 & 63.43 & -147376.82 & 3.57 & 95.43 & -142291.17 \\
% 7 & -142647.33 & 0.32 & 74.61 & -143465.83 & 0.89 & 41.53 & -142198.77 \\
% 8 & -141834.20 & \textbf{0.00} & 24.67 & -149770.67 & 5.59 & 26.06 & -141834.22 \\
% 9 & -142684.22 & 0.14 & 12.87 & -145713.34 & 2.26 & 51.25 & -142488.28 \\
% 10 & -142247.48 & \textbf{0.00} & 18.69 & -148594.24 & 4.46 & 7.63 & -142245.19 \\
% \bottomrule
% \end{tabular}
% \begin{tabular}{lcccccc}
% \toprule
% \multirow{3}{*}{Index} & \multicolumn{2}{c}{Precision} & \multicolumn{2}{c}{Recall} & \multicolumn{2}{c}{SHD} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
% & DCA & GOBNILP & DCA & GOBNILP & DCA & GOBNILP \\
% \midrule
% 1 & 0.40 & 0.59 & 0.68 & 0.56 & 55 & 34 \\
% 2 & \textbf{0.89} & 0.48 & \textbf{0.91} & 0.54 & \textbf{9} & 48 \\
% 3 & 0.66 & 0.50 & 0.74 & 0.59 & 25 & 39 \\
% 4 & 0.77 & \textbf{0.96} & 0.88 & \textbf{0.96} & 10 & \textbf{2} \\
% 5 & 0.56 & 0.75 & 0.72 & 0.69 & 33 & 21 \\
% 6 & 0.46 & 0.25 & 0.68 & 0.32 & 41 & 61 \\
% 7 & 0.68 & 0.72 & 0.73 & 0.71 & 25 & 23 \\
% 8 & \textbf{1.00} & 0.22 & \textbf{1.00} & 0.28 & \textbf{0} & 68 \\
% 9 & 0.71 & 0.57 & 0.77 & 0.60 & 19 & 30 \\
% 10 & \textbf{0.95} & 0.27 & \textbf{0.97} & 0.34 & \textbf{3} & 60 \\
% \bottomrule
% \end{tabular}
% \end{table}
% DCA可以找到near optimal graph, 但是需要花更多时间，scale到15个nodes
% 可以tune initialize方式去控制时间，如果控制时长在Gobnilp时长附近（平均小于Gobnilp），则能达到更好的score和平均更好的图
% \subsection{Result on Real Data}
% Lucas data set \cite{}; Alarm data set \cite{beinlich1989alarm}; Insurance data set \cite{binder1997adaptive}
% \begin{table}[h]
% \centering
% \caption{Comparison between CG-DCA and GOBNILP on different graphs}
% \label{tab:comparison}
% \begin{tabular}{lccccccccccc}
% \toprule
% \multirow{2}{*}{Graph} & \multicolumn{2}{c}{Optimality Gap} & \multicolumn{2}{c}{Time (s)} & \multicolumn{2}{c}{Precision} & \multicolumn{2}{c}{Recall} & \multicolumn{2}{c}{SHD} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9} \cmidrule(lr){10-11}
% & CG-DCA & GOBNILP & CG-DCA & GOBNILP & CG-DCA & GOBNILP & CG-DCA & GOBNILP & CG-DCA & GOBNILP \\
% \midrule
% (15,5000,2) & \textbf{0.0203} & 0.0347 & \textbf{189.12} & 91.72 & \textbf{0.7} & 0.51 & \textbf{0.78} & 0.59 & \textbf{18.9} & 27 \\
% (20,5000,2) & \textbf{0.0174} & 0.0410 & \textbf{2663.87} & 3584.89 & \textbf{0.564} & 0.531 & \textbf{0.736} & 0.559 & \textbf{35} & 38.6 \\
% (25,5000,2) & \textbf{0.0154} & 0.0574 & \textbf{5962.43} & 9352.92 & \textbf{0.634} & 0.499 & \textbf{0.785} & 0.585 & \textbf{39.2} & 48.9 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \section*{References}
% {
% \small
% % [1] Alexander, J.A.\ \& Mozer, M.C.\ (1995) Template-based algorithms for
% % connectionist rule extraction. In G.\ Tesauro, D.S.\ Touretzky and T.K.\ Leen
% % (eds.), {\it Advances in Neural Information Processing Systems 7},
% % pp.\ 609--616. Cambridge, MA: MIT Press.
% [1] Bartlett, M.,\ \& Cussens, J.\ (2017). Integer linear programming for the Bayesian network structure learning problem. {\it Artificial Intelligence}, 244, 258-271.
% % [2] Bower, J.M.\ \& Beeman, D.\ (1995) {\it The Book of GENESIS: Exploring
% % Realistic Neural Models with the GEneral NEural SImulation System.} New York:
% % TELOS/Springer--Verlag.
% [2] Hasselmo, M.E., Schnell, E.\ \& Barkai, E.\ (1995) Dynamics of learning and
% recall at excitatory recurrent synapses and cholinergic modulation in rat
% hippocampal region CA3. {\it Journal of Neuroscience} {\bf 15}(7):5249-5262.
% }
% The local score is denoted as $score_{i\gets J}$. For Gaussian-distributed data, maximizing the local score is equivalent to minimizing the following cost function \cite{...}:
% % 不用specify ciJ, 压缩篇幅
% \[
% c_{i \gets J} = \frac{N}{2} \log (\sigma^2_{i\gets J}) + \frac{N}{2} (1+\log(2\pi)) + \Lambda |J|
% \] where \[
% \sigma^2_{i\gets J} = \min_{\alpha\in \bR^{|J|}} \bE [(\alpha ^T X_J - X_i)^2]
% \]
% Here, $N$ represents the number of data points, $\sigma^2_{i\gets J}$ is the residual variance of node $i$ given its parent set $J$, $\Lambda$ is the regularization parameter and $|J|$ denotes the size of the parent set $J$.
% For discrete data, the log-likehood is given by \cite{...}
% \[
% score_{i \gets J} = \sum_{x_i\in S_i} \sum_{x_J \in S_J} \#(x_i|x_J) \log(\frac{\#(x_i|x_J)}{\#(x_J)})
% \]
% where $S_i$ denotes the set of possible values for $X_i$, $S_J$ represent the set of all possible configurations of the parent set $J$, and $\#(\cdot)$ counts the occurrences of a specific pattern in the observed data.
% We adopt the integer programming formulation presented in \cite{...} for solving the BNSL problem:
% The integer programming formulation requires $n \times 2^{n-1}$ binary variables, representing all possible parent set combinations $P_i$ for each node. The problem also involves $n + |\mathcal{C}|$ constraints, where both terms grow exponentially with the number of nodes $n$. This dual exponential complexity makes direct optimization computationally prohibitive for large networks.
% From a dual perspective, the column generation process for the primal problem corresponds to adding cutting planes to the dual problem. The pricing problem identifies primal solutions associated with the most violated dual constraints at each iteration, thereby indicating their potential to improve the objective value. Through iterative solution of the pricing problem and expansion of $\hat{\bbP}_i$ - continuing until all reduced costs are non-negative - the column generation approach progressively refines the solution space and converges to an optimal solution of the LP relaxation.
% Specifically, performing Column Generation for the primal problem is equivalent to adding the Cutting Planes for the dual. Given the current dual solution, the pricing problem seeks the primal solutions that are associated with the most significantly violated dual constraints, thereby indicating their potential to improve the objective value. By iteratively solving the pricing problem and adding these promising candidates to $\hat{\bbP}_i$ until all the reduced costs are nonnegative, the Column Generation approach progressively refines the solution space and converges to an optimal or near-optimal solution.
\begin{thebibliography}{10}
\bibitem{akaike1974new}
H.~Akaike.
\newblock A new look at the statistical model identification problem.
\newblock {\em IEEE Transactions on Automatic Control}, 19:716, 1974.
\bibitem{an2005dc}
L.~T.~H. An and P.~D. Tao.
\newblock The {DC} (difference of convex functions) programming and {DCA}
revisited with {DC} models of real world nonconvex optimization problems.
\newblock {\em Annals of Operations Research}, 133:23--46, 2005.
\bibitem{andersson1997characterization}
S.~A. Andersson, D.~Madigan, and M.~D. Perlman.
\newblock A characterization of markov equivalence classes for acyclic
digraphs.
\newblock {\em The Annals of Statistics}, 25(2):505--541, 1997.
\bibitem{bagirov2014introduction}
A.~Bagirov, N.~Karmitsa, and M.~M. M{\"a}kel{\"a}.
\newblock {\em Introduction to Nonsmooth Optimization: Theory, Practice and
Software}, volume~12.
\newblock Springer, 2014.
\bibitem{barnhart1998branch}
C.~Barnhart, E.~L. Johnson, G.~L. Nemhauser, M.~W. Savelsbergh, and P.~H.
Vance.
\newblock Branch-and-price: Column generation for solving huge integer
programs.
\newblock {\em Operations Research}, 46(3):316--329, 1998.
\bibitem{bartlett2017integer}
M.~Bartlett and J.~Cussens.
\newblock Integer linear programming for the bayesian network structure
learning problem.
\newblock {\em Artificial Intelligence}, 244:258--271, 2017.
\bibitem{beinlich1989alarm}
I.~A. Beinlich, H.~J. Suermondt, R.~M. Chavez, and G.~F. Cooper.
\newblock The alarm monitoring system: A case study with two probabilistic
inference techniques for belief networks.
\newblock In {\em AIME 89: Second European Conference on Artificial
Intelligence in Medicine, London, August 29th--31st 1989. Proceedings}, pages
247--256. Springer, 1989.
\bibitem{bertsimas1997introduction}
D.~Bertsimas and J.~N. Tsitsiklis.
\newblock {\em Introduction to linear optimization}, volume~6.
\newblock Athena scientific Belmont, MA, 1997.
\bibitem{bhattacharya2021differentiable}
R.~Bhattacharya, T.~Nagarajan, D.~Malinsky, and I.~Shpitser.
\newblock Differentiable causal discovery under unmeasured confounding.
\newblock In {\em International Conference on Artificial Intelligence and
Statistics}, pages 2314--2322. PMLR, 2021.
\bibitem{binder1997adaptive}
J.~Binder, D.~Koller, S.~Russell, and K.~Kanazawa.
\newblock Adaptive probabilistic networks with hidden variables.
\newblock {\em Machine Learning}, 29:213--244, 1997.
\bibitem{buntine1991theory}
W.~Buntine.
\newblock Theory refinement on bayesian networks.
\newblock In {\em Uncertainty in artificial intelligence}, pages 52--60.
Elsevier, 1991.
\bibitem{chen2021integer}
R.~Chen, S.~Dash, and T.~Gao.
\newblock Integer programming for causal structure learning in the presence of
latent variables.
\newblock In {\em International Conference on Machine Learning}, pages
1550--1560. PMLR, 2021.
\bibitem{chickering2004large}
D.~M. Chickering, D.~Heckerman, and C.~Meek.
\newblock Large-sample learning of bayesian networks is {NP}-hard.
\newblock {\em Journal of Machine Learning Research}, 5(Oct):1287--1330, 2004.
\bibitem{colombo2014order}
D.~Colombo and M.~H. Maathuis.
\newblock Order-independent constraint-based causal structure learning.
\newblock {\em Journal of Machine Learning Research}, 15(1):3741--3782, 2014.
\bibitem{10.5555/3020548.3020567}
J.~Cussens.
\newblock Bayesian network learning with cutting planes.
\newblock In {\em Proceedings of the Twenty-Seventh Conference on Uncertainty
in Artificial Intelligence}, UAI'11, page 153–160, Arlington, Virginia,
USA, 2011. AUAI Press.
\bibitem{cussens2012column}
J.~Cussens.
\newblock Column generation for exact {BN} learning: Work in progress.
\newblock In {\em Proc. ECAI-2012 workshop on COmbining COnstraint solving with
MIning and LEarning (CoCoMile 2012)}, 2012.
\bibitem{cussens2020gobnilp}
J.~Cussens.
\newblock {GOBNILP}: Learning {B}ayesian network structure with integer
programming.
\newblock In {\em International Conference on Probabilistic Graphical Models},
pages 605--608. PMLR, 2020.
\bibitem{cussens2023branch}
J.~Cussens.
\newblock Branch-price-and-cut for causal discovery.
\newblock In {\em Conference on Causal Learning and Reasoning}, pages 642--661.
PMLR, 2023.
\bibitem{desaulniers2006column}
G.~Desaulniers, J.~Desrosiers, and M.~M. Solomon.
\newblock {\em Column Generation}, volume~5.
\newblock Springer Science \& Business Media, 2006.
\bibitem{desrochers1992new}
M.~Desrochers, J.~Desrosiers, and M.~Solomon.
\newblock A new optimization algorithm for the vehicle routing problem with
time windows.
\newblock {\em Operations Research}, 40(2):342--354, 1992.
\bibitem{drton2017structure}
M.~Drton and M.~H. Maathuis.
\newblock Structure learning in graphical modeling.
\newblock {\em Annual Review of Statistics and Its Application}, 4(1):365--393,
2017.
\bibitem{el2023difference}
M.~El~Halabi, G.~Orfanides, and T.~Hoheisel.
\newblock Difference of submodular minimization via dc programming.
\newblock In {\em International Conference on Machine Learning}, pages
9172--9201. PMLR, 2023.
\bibitem{ethz_lucas}
{ETH Zurich}.
\newblock {LUCAS (LUng CAncer Simple set)}.
\newblock \url{https://www.causality.inf.ethz.ch/data/LUCAS.html}.
\bibitem{fujishige1978polymatroidal}
S.~Fujishige.
\newblock Polymatroidal dependence structure of a set of random variables.
\newblock {\em Information and Control}, 39(1):55--72, 1978.
\bibitem{fujishige2005submodular}
S.~Fujishige.
\newblock {\em Submodular Functions and Optimization}, volume~58.
\newblock Elsevier, 2005.
\bibitem{gilmore1961linear}
P.~C. Gilmore and R.~E. Gomory.
\newblock A linear programming approach to the cutting-stock problem.
\newblock {\em Operations Research}, 9(6):849--859, 1961.
\bibitem{glymour1999computation}
C.~N. Glymour and G.~F. Cooper.
\newblock {\em Computation, Causation, and Discovery}.
\newblock AAAI Press, 1999.
\bibitem{heckerman1995learning}
D.~Heckerman, D.~Geiger, and D.~M. Chickering.
\newblock Learning bayesian networks: The combination of knowledge and
statistical data.
\newblock {\em Machine Learning}, 20:197--243, 1995.
\bibitem{heinze2018causal}
C.~Heinze-Deml, M.~H. Maathuis, and N.~Meinshausen.
\newblock Causal structure learning.
\newblock {\em Annual Review of Statistics and Its Application}, 5(1):371--391,
2018.
\bibitem{iyer2012algorithms}
R.~Iyer and J.~Bilmes.
\newblock Algorithms for approximate minimization of the difference between
submodular functions, with applications.
\newblock {\em Uncertainty in Artificial Intelligence (UAI)}, 2012.
\bibitem{jaakkola2010learning}
T.~Jaakkola, D.~Sontag, A.~Globerson, and M.~Meila.
\newblock Learning {B}ayesian network structure using {LP} relaxations.
\newblock In {\em Proceedings of the Thirteenth International Conference on
Artificial Intelligence and Statistics}, pages 358--365. JMLR Workshop and
Conference Proceedings, 2010.
\bibitem{joncour2010column}
C.~Joncour, S.~Michel, R.~Sadykov, D.~Sverdlov, and F.~Vanderbeck.
\newblock Column generation based primal heuristics.
\newblock {\em Electronic Notes in Discrete Mathematics}, 36:695--702, 2010.
\bibitem{kelley1960cutting}
J.~E. Kelley, Jr.
\newblock The cutting-plane method for solving convex programs.
\newblock {\em Journal of the society for Industrial and Applied Mathematics},
8(4):703--712, 1960.
\bibitem{kelmans1983multiplicative}
A.~K. Kelmans and B.~N. Kimelfeld.
\newblock Multiplicative submodularity of a matrix's principal minor as a
function of the set of its rows and some combinatorial applications.
\newblock {\em Discrete Mathematics}, 44(1):113--116, 1983.
\bibitem{koivisto2004exact}
M.~Koivisto and K.~Sood.
\newblock Exact bayesian structure discovery in bayesian networks.
\newblock {\em Journal of Machine Learning Research}, 5(May):549--573, 2004.
\bibitem{lovasz1983submodular}
L.~Lov{\'a}sz.
\newblock Submodular functions and convexity.
\newblock {\em Mathematical Programming The State of the Art: Bonn 1982}, pages
235--257, 1983.
\bibitem{lubbecke2005selected}
M.~E. L{\"u}bbecke and J.~Desrosiers.
\newblock Selected topics in column generation.
\newblock {\em Operations Research}, 53(6):1007--1023, 2005.
\bibitem{meek2013causal}
C.~Meek.
\newblock Causal inference and causal explanation with background knowledge.
\newblock {\em arXiv preprint arXiv:1302.4972}, 2013.
\bibitem{nemhauser1981maximizing}
G.~L. Nemhauser and L.~A. Wolsey.
\newblock Maximizing submodular set functions: formulations and analysis of
algorithms.
\newblock In {\em North-Holland Mathematics Studies}, volume~59, pages
279--301. Elsevier, 1981.
\bibitem{pearl2000models}
J.~Pearl.
\newblock {\em Causality: Models, Reasoning and Inference}.
\newblock Cambridge University Press, 2000.
\bibitem{potthoff2010column}
D.~Potthoff, D.~Huisman, and G.~Desaulniers.
\newblock Column generation with dynamic duty selection for railway crew
rescheduling.
\newblock {\em Transportation Science}, 44(4):493--505, 2010.
\bibitem{schwarz1978estimating}
G.~Schwarz.
\newblock Estimating the dimension of a model.
\newblock {\em The Annals of Statistics}, pages 461--464, 1978.
\bibitem{scutari2010learning}
M.~Scutari.
\newblock Learning {B}ayesian networks with the bnlearn {R} package.
\newblock {\em Journal of Statistical Software}, 35:1--22, 2010.
\bibitem{silander2012simple}
T.~Silander and P.~Myllymaki.
\newblock A simple approach for finding the globally optimal bayesian network
structure.
\newblock {\em arXiv preprint arXiv:1206.6875}, 2012.
\bibitem{spirtes2000causation}
P.~Spirtes, C.~N. Glymour, and R.~Scheines.
\newblock {\em Causation, Prediction, and Search}.
\newblock MIT Press, 2000.
\bibitem{tillman2011learning}
R.~Tillman and P.~Spirtes.
\newblock Learning equivalence classes of acyclic models with latent and
selection variables from multiple datasets with overlapping variables.
\newblock In {\em Proceedings of the Fourteenth International Conference on
Artificial Intelligence and Statistics}, pages 3--15. JMLR Workshop and
Conference Proceedings, 2011.
\bibitem{tsamardinos2006max}
I.~Tsamardinos, L.~E. Brown, and C.~F. Aliferis.
\newblock The max-min hill-climbing bayesian network structure learning
algorithm.
\newblock {\em Machine Learning}, 65:31--78, 2006.
\bibitem{van20130}
S.~Van~de Geer and P.~B{\"u}hlmann.
\newblock L0-penalized maximum likelihood for sparse directed acyclic graphs.
\newblock {\em The Annals of Statistics}, 43(2):536--567, 2013.
\bibitem{yu2019dag}
Y.~Yu, J.~Chen, T.~Gao, and M.~Yu.
\newblock {DAG-GNN}: {DAG} structure learning with graph neural networks.
\newblock In {\em International Conference on Machine Learning}, pages
7154--7163. PMLR, 2019.
\bibitem{zhang2008completeness}
J.~Zhang.
\newblock On the completeness of orientation rules for causal discovery in the
presence of latent confounders and selection bias.
\newblock {\em Artificial Intelligence}, 172(16-17):1873--1896, 2008.
\end{thebibliography}