SPGD: Steepest Perturbed Gradient Descent Optimization
Extracted source text · arXiv 2411.04946
% !TEX encoding = UTF-8 Unicode
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%% This file is asmeconf-template.tex, a LaTeX template to format ASME Conference papers according to
%% the requirements on ASME's conference web pages, and including hypertext support for the pdf.
%%
%% This file is version 1.37 dated 2024/02/06
%%
%% As of version 1.11, this template defaults to ASME's newer conference guidelines first posted July 2019.
%% Those guidelines changed the requested author block formatting to be inline.
%% This LaTeX template continues to support the traditional grid format as a package option, [grid].
%% Nomenclature now follows the abstract. Abstract text is set in italics.
%%
%% Author: John H. Lienhard V
%% Department of Mechanical Engineering
%% Massachusetts Institute of Technology
%% Cambridge, MA 02139-4307 USA
%%
%% Class options include:
%%
%% * An option to use the traditional grid arrangement of author names [grid].
%% * With this option, line breaks (\\) may be inserted into the address as needed.
%% * Author names that include commas should be enclosed in braces, e.g., {Joseph L. Smith, Jr.}.
%% * Authors may be grouped above a single affiliation using braces, e.g., {Henry Tudor, Catherine Parr}.
%%
%% * An option to balance the heights of columns on the last page [balance].
%% * This option is NOT compatible with the [lineno] option.
%%
%% * An option to include line numbers [lineno]. You must *run twice* for proper placement of the line numbers.
%% * The lineno package does not number titles, footnotes, captions, or tables.
%% * This option will disable balancing column height on final page if that option has been invoked.
%% * The lineno package won't always number the lines preceding displayed math in a paragraph because
%% * paragraph has not ended. See that package's documentation for macros to address this problem, or
%% * just leave a blank line above the displayed equation while you are editing and then remove the
%% * blank line and [lineno] option when you move to your final version.
%%
%% * An option not to use boldface font for caption text [unboldcaption]
%%
%% * Options to: omit the ASME copyright footer [nofoot];
%% * use government employee copyright notice [govt];
%% * use government contractor copyright notice [contractor]
%% * use copyright notice for some gov't employees [somegovt]
%%
%% * Options for PDF/A compliance. [pdf-a] will produce PDF/A-3u compliance with sRGB OutputIntent.
%% * [pdfapart= 1 or 2 or 3] and [pdfaconformance= b or u] can enable levels 1b, 2b, 2u, and 3b.
%% *
%% * The most recent versions of LaTeX (2021 and later) have adopted integrated support for pdf-a,
%% * through \DeclareDocumentMetadata{..}. The asmeconf class supports these features, which are preferred
%% * to the aforementioned class options. (An up-to-date LaTeX installation is required to use this.)
%%
%% * Many options for calligraphic, script, and fraktur fonts from the mathalfa package; the
%% * example shown here is: [mathalfa=cal=boondoxo] to use the Boondox font for \mathcal.
%% * Some other options for cal are: dutchcal, zapfc, cm (default), euler,...
%% * frak (fraktur), bb (blackboard bold), scr (script) may also be chosen this way.
%% * For details, refer to mathalfa documentation (at CTAN: https://www.ctan.org/pkg/mathalpha).
%%
%% * Math options from M. Sharpe's newtxmath package: upright integrals [upint];
%% * [varvw] for a v and w that are better distinguished from Greek nu; and also
%% * [subscriptcorrection, smallerops, varg, frenchmath, varbb, cmbraces, slantedGreek,...]
%% * See newtx documentation for descriptions (at CTAN: https://www.ctan.org/pkg/newtx, v1.6 or higher is best).
%%
%% * Option to for slightly larger small capitals, [largesc], from newtxtext package.
%%
%% * An option to allow hyphenation of the typewriter font [hyphenate], from inconsolata package.
%% * Hyphenation is normally suppressed for typewriter mode because it is often used for code.
%% * To replace the default variable word spacing by monospacing, use the option [mono].
%% * To get a zero without a slash, use [var0]
%%
%% * Options (used by the babel package) to include passages in languages other than English (e.g., a translation
%% * of the abstract). Languages are called as options, e.g. [french], [spanish], [greek], [russian], etc.
%% * See Appendix B for details.
%% * Language support is most extensive when running LuaLaTeX, which automatically loads fontspec, but
%% * be aware that you may to install additional fonts on your system to use fontspec.
%% *
%% * To use LuaLaTeX without bothering about extra fonts, use the [nofontspec] option to asmeconf.
%%
%% The use of commands defined or modified by the asmeconf class is illustrated throughout this file. In particular,
%% ASME requires capitalized, sans-serif section headings, and as a result some care is needed when using macros
%% in section headings, as also illustrated below.
%%
%% Use an up-to-date and complete installation of LaTeX, such as TeX Live 2020 or later. Errors may occur with old set-ups.
%%
%=========================================================
%%
%% LICENSE:
%%
%% Copyright (c) 2024 John H. Lienhard
%%
%% Offered under the MIT license: https://ctan.org/license/mit
%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% STRONGLY RECOMMENDED new pdf management code follows.
%% This addition the LaTeX kernel was made in June 2022 by the LaTeX Project team.
%% https://www.latex-project.org/news/latex2e-news/ltnews35.pdf
%% Uncomment the next lines and omit the option [pdf-a] from \documentclass[...]{asemconf}
%\DocumentMetadata{%
% pdfstandard=A-3b,% A-2b, A-2u, A-3b, or A-3u (don't use u options with LuaLaTeX)
% pdfversion=1.7,
% lang=en-US,
%}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Class options are described above. Change these options as desired.
%% If you are not using the language options, remove them (together with Appendices B and C)
%% Remove the [colorlinks] option before *final* submission to ASME, to get black text for printing,
%% but keep that option for other uses.
%\documentclass[pdf-a,balance,colorlinks,upint,subscriptcorrection,varvw,mathalfa=cal=boondoxo, spanish,french,vietnamese,russian,greek]{asmeconf}
\documentclass[11pt,letterpaper]{article}
%%%%% pdf metadata %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\hypersetup{%
% pdfauthor={Amir Mohammad Vahedi}, % <=== change to YOUR name
% pdftitle={ASME Conference Paper RPO}, % <=== change to YOUR pdf file title
% pdfkeywords={3D Component Packing, Randomized perturbation, Optimization},% <=== change to YOUR pdf keywords
% pdfsubject = {SPGD: Steepest Perturbed Gradient Descent Optimization}, % <=== change to YOUR subject
%% pdfurl={https://ctan.org/pkg/asmeconf},% may delete
%}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{graphicx}
\usepackage{amsmath} % Loads amsbsy, too
\usepackage{amssymb}
\usepackage{multirow}
\usepackage{hyperref}
\usepackage{booktabs}
\usepackage{subcaption}
\usepackage{cite}
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
% Change these fields to the right content for your conference.
% You can comment these out if for some reason you don't want a header.
% Use title case (first letters capitalized), not all capitals
%\ConfName{Proceedings of the ASME 2024\linebreak International Design Engineering Technical Conferences}
%\ConfAcronym{IDETC2024}
%\ConfDate{August 25--28, 2024} % update
%ConfCity{Washington, DC} % update
%\PaperNo{IDETC2024-XXXX}
% Units of measure (e.g., cm) and other specialty lowercase terms in the title should be
% enclosed in \NoCaseChange{...} to maintain lower case type
% LaTeX will automatically set the rest of the title in all capital letters.
\title{SPGD: Steepest Perturbed Gradient Descent Optimization} % <=== replace with YOUR title
%\title{Place Title Here: Place Subtitle After Colon}
% Put author names into the order you want. Use the same order for affiliations.
% \affil{#} tags the author's affiliation to the address in \SetAffiliation{#}.
% No space between last name and \affil{#}, separate names with commas.
%
% For a sole author or a single affiliation for all authors, {#} may be left empty, as \affil{} and \SetAffiliation{} (but not with [grid] option!)
%
% \CorrespondingAuthor{email} follows that author's affiliation, no spaces.
% If multiple corresponding authors, put both email addresses in the same command and place after both authors.
%
% \JointFirstAuthor, if applicable, follows the affiliation of the relevant authors, no spaces.
\author{Amir M. Vahedi and Horea T. Ilies\\\small{School of Mechanical, Aerospace, and Manufacturing Engineering}\\ \small{University of Connecticut}}
%\SetAuthors{%
% Amir M. Vahedi\affil{1},
% Horea T. Ilie\c{s}\affil{1}\CorrespondingAuthor{horea.ilies@uconn.edu}
% }
%\SetAffiliation{1}{University of Connecticut, Storrs, CT }
% Note: Luis and Maria are not real people. Henry and Catherine have been dead for >450 years.
% To switch from inline author names to gridded names, use the [grid] option.
\maketitle
%%% Use this footnote for tracking various versions of your draft. Change text to suit your own needs.
%%% \date{..} calls the same command.
%\versionfootnote{Documentation for IDETC: Version~\versionno, \today.}% <=== Delete before final submission.
%%% Change these to your keywords. Keywords are automatically printed at the end of the abstract.
%%% This command MUST COME BEFORE the end of the abstract.
%%% If you don't want keywords, leave the argument of \keywords{} empty (or use the abstract* environment)
%\keywords{SPGD, Gradient descent, Optimization, Neural network, 3D component packing}
%%%%% End of fields to be completed. Now write your paper. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%% ABSTRACT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%% Abstract should be 200 words or less
\begin{abstract}
Optimization algorithms are pivotal in advancing various scientific and industrial fields but often encounter obstacles such as trapping in local minima, saddle points, and plateaus (flat regions), which makes the convergence to reasonable or near-optimal solutions particularly challenging.
This paper presents the Steepest Perturbed Gradient Descent (SPGD), a novel algorithm that innovatively combines the principles of the gradient descent method with periodic uniform perturbation sampling to effectively circumvent these impediments and lead to better solutions whenever possible. SPGD is distinctively designed to generate a set of candidate solutions and select the one exhibiting the steepest loss difference relative to the current solution. It enhances the traditional gradient descent approach by integrating a strategic exploration mechanism that significantly increases the likelihood of escaping sub-optimal local minima and navigating complex optimization landscapes effectively. Our approach not only retains the directed efficiency of gradient descent but also leverages the exploratory benefits of stochastic perturbations, thus enabling a more comprehensive search for global optima across diverse problem spaces. We demonstrate the efficacy of SPGD in solving the 3D component packing problem, an NP-hard challenge. Preliminary results show a substantial improvement over six established methods, particularly on response surfaces with complex topographies and in multidimensional non-convex continuous optimization problems. Comparative analyses with established 2D benchmark functions over 30 randomized initial points highlight SPGD’s robustness and reliability in non-convex optimization. These results emphasize SPGD’s potential as a versatile tool for a wide range of optimization problems.
\end{abstract}
%%%%%%%%% NOMENCLATURE (OPTIONAL) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%% To change space between the symbols and definitions, use \begin{nomenclature}[Xcm] where X is a number
%% The unit cm can be replaced by any LaTeX unit of dimension: pt, in, ex, em, pc, etc.
%% Default is 2em.
%% \EntryHeading{..} produces an italicized subheading in the nomenclature list, e.g., \EntryHeading{Greek letters}
%\begin{nomenclature}
%\EntryHeading{Roman letters}
%\entry{$k$}{Thermal conductivity [W m$^{-1}$ K$^{-1}$]}
%\entry{$\vec{q}$}{Heat flux vector [W m$^{-2}$]}
%\EntryHeading{Greek letters}
%\entry{$\alpha$}{Thermal diffusivity [m$^2$ s$^{-1}$]}
%\entry{$\nu$}{Kinematic viscosity [m$^2$ s$^{-1}$]}
%\EntryHeading{Dimensionless groups}
%\entry{Pr}{Prandtl number, $\nu/\alpha$}
%\entry{Sc}{Schmidt number, $\nu/\mathcal{D}_{1,2}$}
%\EntryHeading{Superscripts and subscripts}
%\entry{b}{bulk value}
%\entry{$\infty$}{free stream value}
%\end{nomenclature}
%%%%%%%%% BODY OF PAPER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
Mathematical optimization is a fundamental process in engineering, science, and economics. Its main objective is to find solutions that minimize a predefined objective, typically expressed in terms of a real-valued function, while adhering to given constraints. This pursuit of optimal solutions is crucial in solving complex problems, where achieving the best possible results necessitates a careful balance of numerous factors and variables.
Among the many optimization techniques available, the gradient descent (GD) method stands out as a foundational and extensively used tool, and its origins can be traced back to Cauchy's pioneering work \cite{cauchy1847methode}. However, despite its widespread use, the gradient descent method has certain limitations. One of its major drawbacks is its tendency to get trapped in sub-optimal states, including saddle points and local minima, which may offer minimal improvement in solution quality. Additionally, the method may encounter difficulties in making progress towards the desired outcome when faced with flat regions in the problem space.
To address these challenges, extensive research efforts have been focused on enhancing the performance of the gradient descent method. As a result, numerous variants have been developed, each specifically designed to overcome the aforementioned pitfalls \cite{ruder2016overview}. One notable variant is the Perturbed Gradient Descent (PGD), which has gained attention for its ability to navigate away from saddle points and potentially converge towards second-order optimal points \cite{jin2017escape}.
In this paper, we present a strategic randomized perturbation algorithm combined with the gradient descent method, leveraging the strengths of both exploring the search space through randomized perturbation and converging to optimal points using gradient information. By introducing cyclical perturbations, our approach strategically balances the need for exploration with the efficiency of exploitation. Moreover, applying perturbations periodically rather than at every iteration significantly reduces computational costs, making the optimization process more efficient without sacrificing the thoroughness of the search. It promises a more reliable pathway to discovering superior solutions, thereby expanding the horizon of possibilities in optimization challenges. This enhanced method is designed not only to navigate more effectively through the complexities of practical optimization landscapes but also to refine the search for optimal solutions with greater precision.
The remainder of this paper systematically explores the Steepest Perturbed Gradient Descent (SPGD) algorithm and its comparative advantages in the domain of optimization. Section \ref{sec:RELATED-WORKS} delves into a variety of related methodologies, focusing on variants of the gradient descent method and the integration of perturbation sampling techniques. These approaches establish a foundation for understanding the landscape of optimization strategies and highlight the necessity for innovations, such as SPGD. Section \ref{sec:SPGD} is dedicated to a detailed exposition of the SPGD algorithm itself, including its theoretical underpinnings, algorithmic structure, and the rationale behind its design choices. Following this, Section \ref{sec:Numerical Results} presents numerical results from a series of experiments designed to evaluate the performance of SPGD against various established optimization algorithms. These experiments are conducted on a selection of well-known optimization test functions, providing a rigorous comparison and demonstrating the practical implications of SPGD in addressing complex optimization challenges. Finally, Section \ref{sec:Conclusion} discusses the outcomes of these comparisons, emphasizing the superior performance of SPGD over the methods analyzed. The conclusions not only underscore the effectiveness of SPGD but also set the stage for future research directions and potential applications in broader optimization contexts.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}\label{sec:RELATED-WORKS}
Simulated annealing (SA) \cite{Kirkpatrick1983OptimizationBS} and genetic algorithm \cite{holland1992genetic} are heuristic sampling-based optimization algorithms that use randomness and selection mechanisms inspired by natural processes to explore the solution space and select the best candidates for further iteration. These algorithms can be used for different types of optimization problems, such as continuous, discrete, non-convex, and multi-objective problems \cite{delahaye2019simulated, ghannadi2023review}. These methods may require significant computational resources and careful tuning of parameters (e.g., temperature in simulated annealing or mutation rate in evolutionary algorithms) to balance exploration and exploitation effectively.
Bayesian optimization (BO) is one of the sampling-based global optimization methods that has gained popularity, particularly in machine learning, for solving expensive black-box optimization problems. BO methods approximate the objective function using a surrogate probabilistic model, typically a Gaussian process (GP), which models the underlying function based on observed sample points \cite{shahriari2016taking, snoek2012practical}. These methods balance exploration and exploitation by combining prior beliefs with posterior updates after each observation. The acquisition function, derived from this surrogate model, guides the search by quantifying the expected improvement or uncertainty in unexplored regions. Although BO is highly sample-efficient and effective in finding global optima with a limited number of function evaluations, especially in low-dimensional problems, it can be computationally expensive due to the cost of updating and optimizing the acquisition function at each iteration. Moreover, the performance of BO degrades in high-dimensional or highly non-smooth optimization landscapes \cite{frazier2018tutorial}. A broader discussion of such sampling-based approaches can be found in \cite{burke2020gradient}, a recent survey on non-smooth optimization methods, including gradient sampling and probabilistic techniques.
The gradient descent (GD) method is a first-order optimization algorithm that updates the design variables in the direction opposite to the gradient of the objective function with respect to those variables \cite{ruder2016overview}. It's widely used due to its simplicity and efficiency in convex problems. The gradient descent method converges to a local optimal solution with a mathematical guarantee. Gradient descent tends to exploit local information to improve the solution iteratively. However, it may not explore the search space effectively, potentially getting trapped in local minima or saddle points, particularly in non-convex optimization landscapes. It struggles with flat areas where the gradient is close to zero, leading to slow or no progress \cite{jin2017escape}.
Nesterov's Accelerated Gradient (NAG) method enhances traditional gradient descent by incorporating a forward-looking step. This tweak allows the optimizer to anticipate future gradients, reducing oscillations and speeding up convergence, particularly in convex settings. NAG is highly effective in training deep neural networks due to its efficiency in navigating high-dimensional data spaces. However, its performance can vary in non-convex environments with complex landscapes \cite{sutskever2013importance, jin2018accelerated}. For a comprehensive overview of gradient descent and its variants, we refer readers to \cite{ruder2016overview}, which synthesizes developments across machine learning and optimization literature.
In the exploration of hybrid optimization methods, a notable approach combines the exploratory strengths of Simulated Annealing (SA) with the precise, local search capabilities of Gradient Descent (GD). This method strategically employs SA to break free from local optima by conducting a thorough search for a more promising solution candidate, upon which GD resumes. While this synergy offers a dynamic pathway to escape local minima, it introduces a significant computational burden. Moreover, this method diverges from traditional GD in that it cannot rely on the norm of the gradient as a criterion for termination. This alteration results in a less stringent stop condition, potentially affecting the algorithm's efficiency and termination reliability \cite{yiu2004hybrid}.
Another hybrid technique is perturbed gradient descent (PGD) that addresses the challenge of stagnation—a state where the gradient becomes negligible, and no further progress seems attainable in optimizing the objective function. This method introduces a single perturbation to the current solution when progress halts, effectively nudging the search process out of stagnation before proceeding with GD. This approach demonstrates an ability to escape saddle points effectively \cite{jin2017escape,guo2022escaping, jin2018accelerated}. However, its performance is notably diminished in flat regions of the search space, where such perturbations fail to provide a meaningful direction for improvement.
In addition to the deterministic and heuristic methods previously discussed, the random walk method offers a stochastic approach to optimization that is particularly advantageous in complex, non-convex landscapes. Random walks operate by making a sequence of moves, each determined randomly in terms of direction and step size. This method inherently avoids the common pitfalls of gradient-based approaches, such as becoming trapped in local minima, by facilitating an unbiased exploration of the solution space. This characteristic is critical when dealing with high-dimensional optimization problems where the landscape is riddled with numerous local optima and saddle points \cite{yang2013random}. Despite their potential for encompassing space exploration, random walks are often criticized for their inefficiency and slow convergence, especially in large-scale problems. They require a large number of iterations to approach the vicinity of a global optimum, as their exploration process lacks directionality inherent to methods like gradient descent or even simulated annealing. To address these limitations, researchers have explored hybrid strategies that combine the exploratory strengths of random walks with more systematic search techniques to balance exploration with exploitation more effectively \cite{sun2022adaptive, sussillo2014random}.
%Stochastic gradient descent (SGD) is a well-known optimization algorithm widely utilized in the training step of neural networks due to its efficiency in handling large datasets. Its capability to introduce randomness through mini-batches helps in escaping local minima, making it particularly suitable for complex neural network training scenarios \cite{bottou2018optimization}. The utilization of SGD presents an alternative view by adjusting parameters with a randomly chosen subset of data rather than the complete dataset, thereby injecting noise into the gradient approximations.
%The Adaptive Moment Estimation optimizer is a widely used optimization algorithm in machine learning, especially for training deep neural networks. It combines the advantages of two extensions of stochastic gradient descent, AdaGrad \cite{ward2020adagrad} and RMSProp \cite{mukkamala2017variants}, to handle sparse gradients on noisy problems. The ADAM computes adaptive learning rates for each parameter by estimating the first and second moments of the gradients. This helps in adjusting the learning rate for each weight of the model individually, based on the estimates of lower-order moments \cite{kingma2014adam}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Use title case for subsections and subsubsections (first letter of words capitalized)
\section{Methodology: SPGD}\label{sec:SPGD}
Traditional gradient descent algorithms efficiently exploit local gradient information to improve solutions iteratively. To minimize a given function \(f: \mathbb{R}^{n}\to \mathbb{R}\), the updating rule at each iteration is \cite{jin2017escape}:
\begin{equation}\label{eqn:GradientDescent}
x_{i+1}=x_{i}-\alpha\nabla f(x_{i})
\end{equation}
where \(i\) is the number of current iteration, and \(\alpha >0\) is the step size, and \(\nabla f\) is the gradient of \(f\).
However, in non-convex high-dimensional problems, the gradient descent method can become trapped in local minima or saddle points, missing out on globally optimal configurations. To address this limitation, we propose a novel algorithm that combines gradient descent with periodic randomized perturbations.
These perturbations are particularly effective in non-convex, high-dimensional problems, where even small modifications can significantly alter the solution’s position within the search space. This sensitivity to perturbations is crucial in navigating the complex terrain of such problems, where the landscape of potential solutions is riddled with local optima. By introducing strategically randomized perturbations, our algorithm enhances its ability to escape these local optima, thereby facilitating a more extensive exploration of the solution space. This periodic application of perturbations is key to avoiding the oscillatory behavior often observed in optimization trajectories of sampling-based methods, which can lead to inefficiencies and slow convergence. This approach proves particularly advantageous in complex optimization scenarios characterized by challenges such as flatness, ruggedness, or saddle points of the objective surface, where conventional optimization algorithms might falter in making meaningful progress.
These perturbations are drawn from uniform random distributions\footnote{Having a uniform distribution allows us to sample the solution space around the current solution uniformly within the range of \([-Amp, +Amp]\). This approach leads to more explorative sampling by equally covering the vicinity around the current position, rather than concentrating on the immediate area around the current solution or extending far beyond it.} with constant amplitude profiles\footnote{The main reason for choosing a constant amplitude profile is to maintain the simplicity of the algorithm in these benchmark functions. However, the amplitude profile (\(Amp\)) can be adjusted to any arbitrary profile based on the specific requirements of the optimization problem at hand.}. The uniform random distribution will create \(N_{P}\) perturbed candidates around the gradient descent solution every \(Iter_{P}\) iterations. All perturbed candidates will be evaluated and compared with the gradient descent solution. If the minimum value of perturbed candidates is equal or less than the value of the gradient descent solution, the corresponding perturbed candidate will be selected as the new solution. This policy of accepting solutions with equal objective values intentionally increases the algorithm's emphasis on exploration over exploitation within the optimization process. Such an approach is particularly advantageous in scenarios where the objective surface is flat, and traditional gradient descent methods stall due to insufficient gradient information. By facilitating exploration in these flat regions, SPGD ensures continued progress towards finding a global optimum, preventing the algorithm from becoming prematurely anchored to suboptimal solutions. The pseudo code of SPGD algorithm is described in Algorithm \ref{alg:IGPD}.
\begin{algorithm}
\caption{Steepest Perturbed Gradient Descent}\label{alg:IGPD}
\begin{algorithmic}
%\Require $n \geq 0$
%\Ensure $y = x^n$
\State $i \gets 0$, $i_{P} \gets 0$
\While{$i \leq Iter_{max}$}
\If{$i-i_{p}=Iter_{p}$}
\State $i_{P} \gets i$
\For {$j \gets 1$ to $N_{P}$}
\State $\delta_{p} \gets Amp \times (2\times U(0,1)-1)$
\State $x_{P_{j}} \gets x_i+\delta_{p}$
\Comment{Adding Uniform perturbations}
\EndFor
\If{$\min f(x_{P}) \leq f(x_{i})$}
\State $x_{i} \gets argmin f(x_{P})$
\EndIf
\EndIf
\State $x_{i+1} \gets x_{i}-\alpha\nabla f(x_{i})$
\State $i \gets i+1$
\EndWhile
\end{algorithmic}
\end{algorithm}
In the SPGD algorithm, the method of applying perturbations is adaptable to the specific requirements of the optimization problem at hand. For unconstrained optimization problems, such as 2D test function benchmarks and neural network training, perturbations are applied simultaneously to all variables, utilizing a uniform distribution with a constant amplitude. This ensures a broad, uniform exploration of the solution space, which is generally suitable for the landscapes presented by these types of problems.
However, when dealing with constrained problems like the 3D component packing, which present a complex optimization landscape, a different approach is warranted. In such scenarios, perturbation of a single variable can potentially lead to an infeasible solution or a worse candidate due to the constraints involved. To mitigate this, each variable is perturbed separately, effectively reducing the complexity and dimensionality of the optimization problem by focusing on one variable at a time, with all others held constant \cite{cunningham2015linear}. This targeted perturbation allows for a more controlled exploration of the solution space, ensuring that the search remains within feasible regions and is more likely to improve upon the current solution. This adaptive feature is designed to tailor the exploration process more precisely to the problem’s landscape, enhancing the algorithm's flexibility and effectiveness in navigating constrained environments.
The parameters of SPGD, notably the number of perturbations \(N_{P}\), the perturbation interval \(Iter_{P}\), and the perturbation amplitude \(Amp\), play crucial roles in shaping the algorithm's behavior and performance. Increasing \(N_{P}\) enhances the likelihood of discovering superior solutions by broadening the search during perturbation phases, albeit at a higher computational cost. A smaller \(Iter_{P}\) amplifies the algorithm's exploratory behavior, contributing to a more thorough search of the solution space but also increasing computational demands and leading to more oscillatory convergence patterns. Conversely, selecting a larger \(Amp\) facilitates wider exploration of the search space, though its effectiveness is highly contingent on the specific problem being addressed. For problems where small variations in inputs lead to significant changes in outputs, a large amplitude may not yield beneficial results, underscoring the importance of parameter tuning to align with the problem's characteristics.
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.9\textwidth]{Figures/SPGD_Algorithm_Demonstration_Subfig.pdf}
% \caption{Visualizing the SPGD Algorithm Steps: Navigating a Parabolic Landscape}
% \label{fig:SPGD_Parabola}
% \end{figure}
% To demonstrate the execution flow of the SPGD algorithm and to provide intuitive insights into its operation, Figure \ref{fig:SPGD_Parabola} offers a detailed visualization of a simple 1D parabolic landscape. This example illustrates the iterative process where traditional gradient descent, represented by the green line, begins the optimization journey. Upon activation of the perturbation process after \(Iter_{P}\) iterations, SPGD generates candidates (depicted as blue pluses) around the current position. These candidates explore the vicinity of the solution space, identifying potential points with lower function values. The most promising candidate, offering the lowest function value, is then compared to the current position. As depicted, this candidate yields a better function value and is thus chosen, causing the algorithm to make a significant "jump" to this new point. The process then resumes with gradient descent steps leading to faster convergence to the optimal solution. This example effectively showcases how SPGD combines exploration and exploitation to enhance convergence speed in scenarios where traditional gradient descent may progress slowly.
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{Figures/SPGD_vs_GD_Comparison.pdf}
\caption{Optimization paths of GD and SPGD on \( f(x) = x^4 - 3x^2 + x \), showing how SPGD escapes local minima and converges to the global solution.}
\label{fig:SPGD_GD_Parabola}
\end{figure}
To demonstrate the execution flow of the SPGD algorithm, we provide a visual comparison with traditional gradient descent (GD) in Figure~\ref{fig:SPGD_GD_Parabola}, applied to a non-convex function defined by \( f(x) = x^4 - 3x^2 + x \). This landscape features both a local minimum and a global minimum, offering an ideal setting to showcase the strengths of SPGD in escaping poor regions of convergence.
% To further emphasize the navigation improvement achieved by SPGD, we provide an additional comparison in Figure~\ref{fig:SPGD_GD_Parabola}.
% In this figure, the optimization paths of both traditional gradient descent (GD) and the proposed SPGD algorithm are shown side-by-side on the same optimization landscape, defined by the function \( f(x) = x^4 - 3x^2 + x \), which contains one local minima and one global minima.
In figure~\ref{fig:SPGD_GD_Parabola}, GD is seen following the steepest descent path, ultimately settling at the local minimum without the ability to recover. This behavior is the characteristic of gradient-based methods in non-convex landscapes, where they are prone to getting trapped due to the absence of global information or exploration strategies.
The SPGD algorithm, on the other hand, begins similarly by following the gradient descent path, represented by the green line, for a fixed number of iterations. After \(Iter_{P}\) iterations, the perturbation phase is triggered. At this point, SPGD generates \(N_{P}\) candidates (depicted as black 'x') around the current position. These candidates explore the vicinity of the solution space, identifying potential points with lower function values. The most promising candidate, offering the lowest function value (shown as blue '+'), is then compared to the current position. As depicted, this candidate yields a better function value and is thus chosen, causing the algorithm to make a significant "jump" to this new point. In the figure~\ref{fig:SPGD_GD_Parabola}, this is reflected in the trajectory of SPGD deviating from the region of slow progress of local minimum and moving toward the global one. The process then resumes with gradient descent steps leading to faster convergence to the optimal solution.
% This example effectively showcases how SPGD combines exploration and exploitation to enhance convergence speed in scenarios where traditional gradient descent may progress slowly.
As a result, SPGD not only avoids becoming stuck in local minima but also achieves convergence with fewer function evaluations compared to methods that either rely solely on gradients or purely stochastic exploration. This example highlights how SPGD’s structure, consisting of gradient-based updates interleaved with perturbations, contributes to both its robustness and efficiency in solving non-convex optimization problems.
% As depicted, GD converges and gets stuck into a suboptimal local minimum, unable to escape the suboptimal region. In contrast, the perturbation mechanism in SPGD enables the algorithm to quickly escape this region of slow progress and continue toward the global minimum.
% By periodically introducing exploratory candidate points, SPGD not only improves the quality of the final solution but also achieves convergence with fewer function evaluations.
% This comparison highlights the critical role of perturbations in enhancing the robustness and efficiency of the optimization process.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Numerical Results}\label{sec:Numerical Results}
We present here a thorough evaluation of the proposed Steepest Perturbed Gradient Descent (SPGD) algorithm, comparing its performance against several established optimization methods. The comparison includes traditional gradient descent (GD), Perturbed Gradient Descent (PGD), MATLAB \(fmincon\) function, which is a versatile solver for constrained optimization problems \cite{Fmincon}, and the \(fminunc\) function, which is tailored to unconstrained optimization problems \cite{Fminunc}, along with Simulated Annealing (SA) \cite{SA}, and Bayesian Optimization (BO) \cite{BO}.
Our initial analysis is conducted through the lens of four challenging 2D benchmark functions, selected for their known difficulties and relevance in assessing optimization algorithms' efficacy. These test functions are recognized benchmarks within the optimization community, providing a diverse set of landscapes to evaluate each algorithm's ability to navigate complex, non-convex, and potentially deceptive optimization spaces \cite{simulationlib}. For each test function, we apply \(fmincon\), Simulated Annealing, traditional gradient descent, PGD, and SPGD, meticulously recording and analyzing the results. The SPGD and PGD algorithms were each fine-tuned independently to ensure optimal performance while maintaining a fair basis for comparison.
% In both the SPGD and PGD algorithms, the amplitude of the perturbation is set to be the same, ensuring comparability, \(Iter_{P}\) is set to \(5\), and \(N_{P}\) is set to \(10\).
Due to the high computational cost of Bayesian Optimization (BO), the maximum number of function evaluations for BO is capped at 100 to ensure reasonable execution time across benchmark functions. For the \(fmincon\) function, we use MATLAB's default \textit{interior-point} algorithm, while the \(fminunc\) function is configured to use the \textit{trust-region} method, which is well-suited for smooth, unconstrained problems. In both cases, the gradient of the objective function is explicitly provided to guide the optimization process more efficiently.
Key performance indicators include the accuracy of the solution, measured by the proximity to the known global optimum\cite{hazewinkel2001theory}; the computational efficiency, quantified by the number of function evaluations and CPU execution time. For each test function, both a 3D and top-view surface plot of optimization trajectory visualization are provided to aid in understanding each algorithm's optimization landscape and behavior. These visualizations illustrate how optimization paths evolve over complex response surfaces and help highlight differences in convergence dynamics. Simulated Annealing (SA) and Bayesian Optimization (BO) are excluded from these visualizations, as their probabilistic sampling strategies tend to densely populate the landscape, obscuring the trajectories of other algorithms and reducing the overall interpretability of the plots. The source code for the SPGD algorithm, along with comparative analyses against methods discussed in this paper using additional 2D challenging test functions, are publicly accessible on GitHub\footnote{Source code and comparisons available at: \url{https://github.com/Amir-M-Vahedi/SPGD-Benchmark-Functions}}.
\subsubsection*{Test function 1} The MATLAB Peaks function \cite{Peaksfunction-MATLAB} presents a formidable challenge for optimization algorithms due to its intricate landscape, which features one global minimum, multiple local minima, a saddle point, and extensive flat regions. This complexity makes the Peaks function a critical benchmark for assessing the capabilities of optimization techniques, particularly those based on gradient descent. Traditional gradient descent methods often struggle with such landscapes, as they can easily become trapped in local minima or stall in flat areas, failing to make significant progress towards the global optimum \cite{machines10010042}. The mathematical expression defining the Peak test function is given as follows:
\begin{equation}
\label{eqn:PEAKS}
f(x,y)=3\,{\mathrm{e}}^{-{{\left(y+1\right)}}^2 -x^2 } \,{{\left(x-1\right)}}^2 -\\ \frac{{\mathrm{e}}^{-{{\left(x+1\right)}}^2 -y^2 } }{3}+{\mathrm{e}}^{-x^2 -y^2 } \,{\left(10\,x^3 -2\,x+10\,y^5 \right)}
\end{equation}
It has a global minimum point located at \(x= 0.2283\), \(y= -1.6256\) with an optimal function value of \(f(x^{*}) = -6.5511\). The initial condition is chosen randomly to be \((-2.81,-1.47)\), and the \(Amp\) is set to 2.5. Figure~\ref{fig:peaks-3d} and ~\ref{fig:peaks-top} illustrate the 3D view and top view of the optimization trajectory across the Peaks function surface. The total number of function evaluations, the converged optimal value, and CPU execution time for different methods are given in Table \ref{tab:Test1 Performance}. Based on the results depicted in Figures~\ref{fig:peaks-both}, and performance metrics in Table \ref{tab:Test1 Performance}, it is evident that the GD, PGD, and \(fminunc\) algorithms become trapped in local minima. In contrast, the \(fmincon\), SA, BO, and SPGD algorithms successfully converge to the global optimum. Among these three, SPGD demonstrates the lowest computational cost. Notably, despite the \(fmincon\) and BO method having fewer function evaluations, their CPU times are more than 25 and 2000 times greater than that of the SPGD algorithm.
%%%%%%%%%%%%%%% begin simple table %%%%%%%%%%%%%%%%%%%%%%%%%%
%% Captions go above tables
%%
%% Note that placement of figures and tables can managed with the [!tbhp] options. See: https://latexref.xyz/dev/latex2e.html#Floats
\begin{table}[h]
\caption[Table]{Peaks function Performance}\label{tab:Test1 Performance}
\centering{%
\begin{tabular}{lccr}
\toprule
%\textbf{Algorithm} & \textbf{Function Evaluation} & \(f(x^{*})\) & Relative Error & CPU Time[s]\\
{\textbf{Algorithm}} & {\textbf{Total Fun. Evaluations}} & {\(\boldsymbol{f(x^{*})=-6.5511}\)} & {\textbf{CPU Time[ms]}}\\
\midrule
GD & 1472 & -3.0498 & *3.12\\
PGD & 1599 & -3.0498 & *3.88\\
\(fminunc\) & 10 & -3.0498 & *23.25\\
\(fmincon\) & 60 & \textbf{-6.5511} & 57.37\\
SA & 1341 & \textbf{-6.5511} & 117.8327\\
BO & 100 & \textbf{-6.5510} & 4415.7\\
\textbf{SPGD} & 274 & \textbf{-6.5511} & 2.03\\
\bottomrule
\end{tabular}
}
\end{table}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/Peaks/Peaks.pdf}
% \caption{3D view optimization trajectory of Peaks function}\label{fig:1}
% \end{figure}
% %%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
% %%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
% %% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/Peaks/Peaks_top.pdf}
% \caption{Top view optimization trajectory of Peaks function}\label{fig:2}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
\begin{figure}
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/Peaks/Peaks.pdf}
\caption{3D view of the optimization trajectories.}
\label{fig:peaks-3d}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/Peaks/Peaks_top.pdf}
\caption{Top view of the optimization trajectories.}
\label{fig:peaks-top}
\end{subfigure}
\caption{Visualization of optimization trajectories for the Peaks function.}
\label{fig:peaks-both}
\end{figure}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.8\textwidth]{Figures/Peaks/Peaks_Convg.pdf}
% \caption{Convergence history of Peaks function}\label{fig:3}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
\subsubsection*{Test function 2} The Ackley function is a well-known non-convex optimization benchmark that poses a significant challenge to optimization algorithms, particularly due to its deceptive landscape characterized by a global optimum surrounded by a multitude of local minima \cite{ackley2012connectionist}. This function is specifically designed to test the ability of optimization methods to escape local minima and efficiently search for the global optimum in a complex, multidimensional space. The Ackley function's landscape features a large number of local minima leading towards the global minimum, making it an exemplary test case for evaluating the robustness and effectiveness of algorithms against the risk of premature convergence. The global minimum of the Ackley function is located at the origin (\(x= 0\), \(y= 0\)), with an optimal function value of zero (\(f(x^{*}) = 0\)), which further serves as a clear target for optimization efforts. The formula representing the Ackley test function is articulated below:
\begin{equation}\label{eqn:Ackley}
f(x, y) = -20 \exp\left(-0.2 \sqrt{\frac{1}{2} (x^2 + y^2)}\right) -\\ \exp\left(\frac{1}{2} (\cos(2\pi x) + \cos(2\pi y))\right) + 20 + e
\end{equation}
The initial condition is chosen randomly to be \((-3.75,-1.96)\), and the \(Amp\) is set to 2.5. Figure ~\ref{fig:ackley-3d} and ~\ref{fig:ackley-top} illustrate the 3D view and top view of optimization trajectory across the Ackley function surface. The performance comparisons are given in Table \ref{tab:Test2 Performance}. Taking into account the data presented in the mentioned figures and table, analysis of the Ackley test function reveals that the GD, PGD, \(fminunc\), and \(fmincon\) methods became ensnared in local minima. In contrast, only the SA, BO, and SPGD algorithms successfully navigated to the global solution. Among these, SPGD not only achieved convergence with greater precision, approaching closer to the global optimum, but also demonstrated a computational speed, with the CPU execution time being about 13 (SA) and 359 (BO) times faster than its counterparts.
%%%%%%%%%%%%%%% begin simple table %%%%%%%%%%%%%%%%%%%%%%%%%%
%% Captions go above tables
%%
%% Note that placement of figures and tables can managed with the [!tbhp] options. See: https://latexref.xyz/dev/latex2e.html#Floats
\begin{table}[h]
\caption[Table]{Ackley function Performance}\label{tab:Test2 Performance}
\centering{%
\begin{tabular}{lccr}
\toprule
%\textbf{Algorithm} & \textbf{Function Evaluation} & \(f(x^{*})\) & Relative Error & CPU Time[s]\\
{\textbf{Algorithm}} & {\textbf{Total Fun. Evaluations}} & \(\boldsymbol{f(x^{*})=0}\) & {\textbf{CPU Time[ms]}}\\
\midrule
GD & 327 & 9.3530 & *2.01\\
PGD & 477 & 6.8826 & *2.02\\
\(fminunc\) & 8 & 9.3530 & *32.20\\
\(fmincon\) & 24 & 9.3530 & *75.13\\
SA & 504 & \textbf{2.13\(e\)-4} & 40.63\\
BO & 100 & \textbf{0.0213} & 4670.0\\
\textbf{SPGD} & 1501 & \textbf{4.81\(e\)-4} & 3.62\\
\bottomrule
\end{tabular}
}
\end{table}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/Ackley/Ackley.pdf}
% \caption{3D view optimization trajectory of Ackley function}\label{fig:4}
% \end{figure}
% %%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
% %%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
% %% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/Ackley/Ackley_top.pdf}
% \caption{Top view optimization trajectory of Ackley function}\label{fig:5}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
\begin{figure}
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/Ackley/Ackley.pdf}
\caption{3D view of the optimization trajectories.}
\label{fig:ackley-3d}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/Ackley/Ackley_top.pdf}
\caption{Top view of the optimization trajectories.}
\label{fig:ackley-top}
\end{subfigure}
\caption{Visualization of optimization trajectories for the Ackley function.}
\label{fig:ackley-both}
\end{figure}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.8\textwidth]{Figures/Ackley/Ackley_Convg.pdf}
% \caption{Convergence history of Ackley function}\label{fig:6}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%
\subsubsection*{Test function 3} The Easom function stands as a notable unimodal steep ridge \cite{simulationlib} test function within the realm of optimization, particularly distinguished by its singular global optimum that resides in an extensive flat area. This flat region is characterized by minimal gradient variations, presenting a unique challenge for optimization algorithms, especially those reliant on gradient information to navigate the search space. The function is defined over a domain of (\(-100\), \(100\)) for both \(x\) and \(y\) dimensions, emphasizing the necessity for optimization techniques to efficiently explore large search areas to locate the optimum \cite{easom1990survey}. The significance of the Easom function as a test scenario with simple mathematical formulation lies in its ability to simulate real-world optimization problems where the solution space is largely homogeneous, yet contains a singular, critical point of interest. This function tests the exploration strategies of algorithms, challenging them to avoid the pitfalls of vast non-informative regions. It emphasizes the importance of balance between exploration and exploitation, as effective optimization methods must not only navigate vast spaces efficiently but also recognize and converge to the global optimum with high precision. Mathematically, the Easom function's global optimum is uniquely situated at (\(x= \pi\), \(y= \pi\)), where it attains a value of \(f(x^{*}) = -1\). The formula of the Easom test function is provided below:
\begin{equation}\label{eqn:Easom}
f(x, y) = -\cos(x) \cos(y) \exp\left(-\left((x - \pi)^2 + (y - \pi)^2\right)\right)
\end{equation}
The initial condition is chosen randomly to be \((69.33,12.23)\), and the \(Amp\) is set to 5. Figure~\ref{fig:easom-3d} and ~\ref{fig:easom-top} illustrate the 3D view and top view of the optimization trajectory across the Easom function surface. The performance comparisons are given in Table \ref{tab:Test3 Performance}.
Reflecting on the performance metrics for the Easom test function, it is evident that only the SPGD algorithm successfully pinpointed the global optimum. As anticipated, GD was hindered in its progression by the minimal gradient values inherent to the function's extensive flat regions. Similarly, both GD and the \(fmincon\) method failed to escape these flat expanses, effectively becoming ensnared within them. Among the competing methods, only SA and BO managed to navigate towards a more favorable outcome, yet they fell short of achieving convergence to the global optimum, underscoring the distinctive effectiveness of SPGD in this scenario.
%%%%%%%%%%%%%%% begin simple table %%%%%%%%%%%%%%%%%%%%%%%%%%
%% Captions go above tables
%%
%% Note that placement of figures and tables can managed with the [!tbhp] options. See: https://latexref.xyz/dev/latex2e.html#Floats
\begin{table}[h]
\caption[Table]{Easom function Performance}\label{tab:Test3 Performance}
\centering{%
\begin{tabular}{lccr}
\toprule
%\textbf{Algorithm} & \textbf{Function Evaluation} & \(f(x^{*})\) & Relative Error & CPU Time[s]\\
{\textbf{Algorithm}} & {\textbf{Total Fun. Evaluations}} & \(\boldsymbol{f(x^{*})=-1}\) & {\textbf{CPU Time[ms]}}\\
\midrule
GD & 1 & 0 & *0.08\\
PGD & 2021 & 0 & *3.28\\
\(fminunc\) & 1 & 0 & *58.35\\
\(fmincon\) & 1 & 0 & *83.84\\
SA & 1009 & -3.38\(e\)-160 & *69.60\\
BO & 100 & -6.22\(e\)-215 & *6866.9\\
\textbf{SPGD} & 6001 & \textbf{-1} & 7.45\\
\bottomrule
\end{tabular}
}
\end{table}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/Easom/Easom.pdf}
% \caption{3D view optimization trajectory of Easom function}\label{fig:7}
% \end{figure}
% %%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
% %%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
% %% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/Easom/Easom_top.pdf}
% \caption{Top view optimization trajectory of Easom function}\label{fig:8}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
\begin{figure}
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/Easom/Easom.pdf}
\caption{3D view of the optimization trajectories.}
\label{fig:easom-3d}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/Easom/Easom_top.pdf}
\caption{Top view of the optimization trajectories.}
\label{fig:easom-top}
\end{subfigure}
\caption{Visualization of optimization trajectories for the Easom function.}
\label{fig:easom-both}
\end{figure}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.8\textwidth]{Figures/Easom/Easom_Convg.pdf}
% \caption{Convergence history of the Easom function, highlighting the immediate cessation of Gradient Descent (GD) and \(fmincon\) at the initial iteration due to flat loss surface. Notably, SPGD was the sole algorithm among the seven tested that successfully reached the global optimum.}\label{fig:9}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%
\subsubsection*{Test function 4} The Levy Function No. 13, characterized by its multimodality and non-convexity, presents a unique challenge for optimization algorithms with its single global optimum amidst a noisy, periodic distribution of local minima. This function tests an algorithm's precision in distinguishing the global optimum from numerous suboptimal states, a key trait for solving complex real-world problems. It serves as a critical benchmark for evaluating the balance between exploration and exploitation in optimization techniques, underscoring its significance in both theoretical and practical applications. The global optimum of this function is strategically located at (\(x= 1\), \(y= 1\)), where it attains a value of \(f(x^{*}) = 0\). The expression for the Levy Function No. 13 is detailed below \cite{Levy1985TheTA}:
\begin{equation}\label{eqn:Levy}
f(x, y) = \sin^2(3\pi x) + \\(x-1)^2 \left(1 + \sin^2(3\pi y) \right) + (y-1)^2 \left(1 + \sin^2(2\pi y) \right)
\end{equation}
The initial condition is chosen randomly to be \((-3.75,-1.96)\), and the \(Amp\) is set to 2.5. Figure~\ref{fig:levy-3d} and ~\ref{fig:levy-top} illustrate the 3D view and top view of the optimization trajectory across the Levy Function No. 13 surface. The performance comparisons are given in Table \ref{tab:Test4 Performance}.
Based on the performance analysis for this test function, the GD, PGD, \(fminunc\), and \(fmincon\) methods were unable to find the global optimum, getting stuck in local minima instead. Notably, \(fmincon\) settled in a particularly poor local minimum. In contrast, SA, BO, and SPGD successfully navigated to the global optimum. However, SPGD distinguished itself by achieving a more accurate solution, requiring fewer function evaluations than SA, and demonstrating faster CPU execution time compared to SA and BO.
%%%%%%%%%%%%%%% begin simple table %%%%%%%%%%%%%%%%%%%%%%%%%%
%% Captions go above tables
%%
%% Note that placement of figures and tables can managed with the [!tbhp] options. See: https://latexref.xyz/dev/latex2e.html#Floats
\begin{table}[h]
\caption[Table]{Levy function N. 13 Performance}\label{tab:Test4 Performance}
\centering{%
\begin{tabular}{lccr}
\toprule
%\textbf{Algorithm} & \textbf{Function Evaluation} & \(f(x^{*})\) & Relative Error & CPU Time[s]\\
{\textbf{Algorithm}} & {\textbf{Total Fun. Evaluations}} & \(\boldsymbol{f(x^{*})=0}\) & {\textbf{CPU Time[ms]}}\\
\midrule
GD & 2001 & 6.2915 & *3.58\\
PGD & 2001 & 6.2915 & *2.58\\
\(fminunc\) & 9 & 14.3717 & *20.59\\
\(fmincon\) & 20 & 30.5009 & *53.32\\
SA & 2018 & \textbf{6.78\(e\)-7} & 89.61\\
BO & 100 & \textbf{0.0086} & 5241.7\\
\textbf{SPGD} & 1760 & \textbf{2.45\(e\)-13} & 5.02\\
\bottomrule
\end{tabular}
}
\end{table}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/LevyN13/Levy.pdf}
% \caption{3D view optimization trajectory of Levy function N. 13}\label{fig:10}
% \end{figure}
% %%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
% %%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
% %% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.6\textwidth]{Figures/LevyN13/Levy_top.pdf}
% \caption{Top view optimization trajectory of Levy function N. 13}\label{fig:11}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%%
\begin{figure}
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/LevyN13/Levy.pdf}
\caption{3D view of the optimization trajectories.}
\label{fig:levy-3d}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/LevyN13/Levy_top.pdf}
\caption{Top view of the optimization trajectories.}
\label{fig:levy-top}
\end{subfigure}
\caption{Visualization of optimization trajectories for the Levy function N. 13.}
\label{fig:levy-both}
\end{figure}
%%%%%%%%%%%%% begin figure %%%%%%%%%%%%%%%%%
%% captions go below figures
% \begin{figure}
% \centering\includegraphics[width=0.8\textwidth]{Figures/LevyN13/Levy_Convg.pdf}
% \caption{Convergence history of Levy function N. 13}\label{fig:12}
% \end{figure}
%%%%%%%%%%%%% end figure %%%%%%%%%%%%%%%%%%
\subsection*{Robustness Evaluation}
To evaluate the robustness of the SPGD algorithm, each test function was subjected to 30 independent trials using randomly sampled starting points. All optimization algorithms were provided with the same lower and upper bounds defining the feasible search space. Each algorithm was fine-tuned independently to ensure their optimal individual performance, and consistent parameter settings were applied across all trials to maintain fairness. The following performance criteria are reported in Tables~\ref{tab:peaks-performance}--\ref{tab:levi-performance}: the number of successful convergences (\textit{ConvergedRuns}), the average percentage improvement in objective value and CPU execution time relative to SPGD (\textit{Fval Improvement\%}, \textit{Time Improvement\%}), and the average closeness to the global optimum (\textit{Closer\%}), over 30 randomized runs, which reflects how much nearer each method’s average result is to the optimum compared to SPGD. In cases where SPGD has already achieved the global optimum or the baseline method matched it, this value is marked as “N/A”.
For the Peaks function, SPGD successfully converged to the global optimum in all 30 trials. Only Bayesian Optimization (BO) matched this convergence count, with Simulated Annealing (SA) achieving 29 out of 30. However, SPGD accomplished this with significantly lower computational cost, as reflected in the \textit{Time Improvement\%} column of Table~\ref{tab:peaks-performance}, demonstrating superior efficiency.
In the case of the Ackley function, SPGD was the only algorithm to consistently converge to the global solution across all runs. Although GD and PGD had lower average execution times, they only succeeded in 3 and 5 out of 30 runs respectively, making them less competitive. SPGD outperformed all remaining methods in terms of average speed and reliability over 30 trials.
For the Easom function, none of the baseline algorithms found the global optimum in any run. SPGD was the only method to successfully reach the global solution in all 30 trials, highlighting its robustness in highly deceptive landscapes.
In the Levi function N.13, SPGD again demonstrated the highest reliability with 30 successful runs out of 30. BO and SA achieved 24 successful runs each, while other algorithms failed to find the global optimum in any trial. SPGD also demonstrated strong efficiency, with an average \textit{Time Improvement\%} of 99.91 over BO and 90.02 over SA.
\begin{table}[!ht]
\caption[Table]{Average performance comparison for Peaks function over 30 random starting points}\label{tab:peaks-performance}
\centering{
\begin{tabular}{lcccc}
\toprule
\textbf{Algorithm} & \textbf{ConvergedRuns} & \textbf{Fval Improvement \%} & \textbf{Time Improvement \%} & \textbf{Closer \%} \\
\midrule
GD & 9 & 135.63 & -5.43 & 100.00 \\
PGD & 9 & 127.32 & -10.01 & 100.00 \\
BayesOpt & 30 & 0.00 & 99.99 & N/A \\
SA & 29 & 0.04 & 98.90 & 98.71 \\
Fminunc & 7 & 206.33 & 81.57 & 100.00 \\
Fmincon & 8 & 137.05 & 93.10 & 100.00 \\
SPGD & 30 & ~ & ~ & ~ \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[!ht]
\caption[Table]{Average performance comparison for Ackley function over 30 random starting points}\label{tab:ackley-performance}
\centering{
\begin{tabular}{lcccc}
\toprule
\textbf{Algorithm} & \textbf{ConvergedRuns} & \textbf{Fval Improvement \%} & \textbf{Time Improvement \%} & \textbf{Closer \%} \\
\midrule
GD & 3 & 99.97 & -1240.80 & 99.97 \\
PGD & 5 & 99.97 & -1079.38 & 99.97 \\
BayesOpt & 17 & 83.04 & 99.96 & 83.04 \\
SA & 27 & 99.15 & 85.82 & 99.15 \\
Fminunc & 5 & 99.97 & 50.29 & 99.97 \\
Fmincon & 8 & 99.96 & 83.92 & 99.96 \\
SPGD & 30 & ~ & ~ & ~ \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[!ht]
\caption[Table]{Average performance comparison for Easom function over 30 random starting points}\label{tab:easom-performance}
\centering{
\begin{tabular}{lcccc}
\toprule
\textbf{Algorithm} & \textbf{ConvergedRuns} & \textbf{Fval Improvement \%} & \textbf{Time Improvement \%} & \textbf{Closer \%} \\
\midrule
GD & 0 & N/A & -79148.81 & 100.00 \\
PGD & 0 & 444556889.89 & -1822.73 & 100.00 \\
BayesOpt & 0 & 4137.02 & 99.97 & 100.00 \\
SA & 0 & 8841.39 & 78.76 & 100.00 \\
Fminunc & 0 & N/A & -724.67 & 100.00 \\
Fmincon & 0 & N/A & -277.97 & 100.00 \\
SPGD & 30 & & & \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[!ht]
\caption[Table]{Average performance comparison for Levi function N.13 over 30 random starting points}\label{tab:levi-performance}
\centering{
\begin{tabular}{lcccc}
\toprule
\textbf{Algorithm} & \textbf{ConvergedRuns} & \textbf{Fval Improvement \%} & \textbf{Time Improvement \%} & \textbf{Closer \%} \\
\midrule
GD & 0 & 100.00 & -1761.37 & 100.00 \\
PGD & 0 & 100.00 & -1462.19 & 100.00 \\
BayesOpt & 24 & 100.00 & 99.91 & 100.00 \\
SA & 24 & 100.00 & 90.02 & 100.00 \\
Fminunc & 0 & 100.00 & -140.21 & 100.00 \\
Fmincon & 0 & 100.00 & 34.57 & 100.00 \\
SPGD & 30 & ~ & ~ & ~ \\
\bottomrule
\end{tabular}
}
\end{table}
The challenges presented by these test functions, including their rugged landscapes and deceptive local minima, contain features that bear resemblance to those encountered in the energy landscape of protein folding. This complex biological process is characterized by a similarly intricate energy landscape that features multiple local optima (kinetic traps), rugged terrain, and steep energy barriers (sharp valleys and hills)\cite{gershenson2020successes, Levinthal, shahbazi2010hydrogen,madden2009residue}, as depicted in Figure \ref{fig:protein_folding}.
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{Figures/Protein_Energy_Landscape.pdf}
\caption{Comparison of 2D optimization test functions and the funneled energy landscapes of protein folding, highlighting similar features such as multiple local optima and rugged terrains \cite{Levinthal}.}
\label{fig:protein_folding}
\end{figure}
The SPGD algorithm's performance on these test functions suggests its potential utility in addressing the complex optimization problems inherent in protein folding. By adeptly navigating through challenging landscapes to find global or near-global optima, SPGD could significantly contribute to bioinformatics and molecular biology by optimizing protein structures to understand their function and interactions more accurately.
This analogy not only highlights the broader applicability of SPGD but also underscores the importance of developing robust optimization techniques that can effectively deal with the complexities of both mathematical functions and biological systems \cite{tavousi2015protofold, mohammadi2023sign}.
Expanding our investigation beyond conventional 2D test functions, we also apply our algorithm (SPGD) to a 3D component packing problem, a task distinguished by its NP-hard classification \cite{peddada2022toward,Packing_SA}. This problem introduces a unique set of challenges, including flat area saddle points and local optima, that further test the robustness of our approach against traditional gradient descent and simulated annealing methods.
\subsubsection*{3D Component Packing Problem}
In the 3D component packing problem, we focus on arranging 3D objects with arbitrary shapes as compactly as possible without collision, akin to a simplified version of the interconnected systems with physical interactions (SPI2) problem but without considering the routing interconnections between objects \cite{SPI2_paper}. Optimization methods often face challenges in this landscape, such as getting trapped in local minima or stalling in flat areas, thus failing to advance significantly towards the global optimum. The non-convex nature of the objective function, characterized by multiple local optima and saddle points, poses substantial challenges to any standard optimization technique. Nonetheless, our SPGD algorithm, which integrates randomized perturbations, is tailored to navigate these complex landscapes more effectively, demonstrating its adaptability and enhanced performance compared to conventional techniques.
%Our approach contrasts sharply with that presented by \cite{cui2023dense}, which addresses the dense, interlocking-free packing of 3D objects using a spectral method. While this method restricts the objects to be aligned parallel to the main axes and only allows rotations in 90-degree increments, our model grants each object six degrees of freedom, offering more nuanced positioning and orientation capabilities. Additionally, whereas \cite{cui2023dense} utilizes discrete voxel representations and manages collisions through Fast Fourier Transform-based techniques, our method employs a novel representation called maximal disjoint ball decomposition (MDBD) \cite{CHEN2020102850}. The MDBD representation simplifies collision detection and increases the flexibility in object manipulation. This method decomposes objects into non-overlapping spheres, significantly improving the collision checks between objects of arbitrary shapes. We solve the problem using a gradient-based optimization algorithm, leveraging the continuous nature of our problem representation to navigate the complex and high-dimensional landscape effectively. On the other hand, \cite{cui2023dense} employ an intuitive searching algorithm that, while innovative, operates under a different set of assumptions and constraints. These distinctions underscore the fundamentally different approaches and methodologies applied to similar packing challenges.
Our 3D packing scenarios presented here are specialized instances of those tackled by SPI2-F -- a novel and more general packing and layout optimization presented in \cite{behzadi2014spi2-f}, that performs both packing and layout optimization of complex interconnected systems in a multi-physics environment. The specialized scenarios presented below have been chosen to include cases that have known global optima. We note that an efficient packing method based on Fast Fourier Transforms\footnote{Fast Fourier Transforms have been previously shown to offer an elegant and efficient approach to compute collisions and penetrations as well as shape complementarity \cite{lysenko2013fourier,behandish2015,behandish2016analytic, kavraki1995computation}.} was introduced recently in \cite{cui2023dense}, which restricts the orientation of the objects to an axis-alignment and hence allows rotations in 90-degree increments. By contrast, our method allows arbitrary rotations and alignments in space.
In the 3D component packing problem, our primary objectives are twofold: minimize the volume of the bounding box containing the components (\(V_b\)) while avoiding collisions between the components. Therefore, we define the mathematical expression of the objective function as follows:
\begin{equation}\label{eqn:SPI2}
f = w_b \times V_b - w_c \times \log(\epsilon + \min(\text{dist}))
\end{equation}
where \(w_b = 20\) represents the weight associated with the bounding box volume, \(w_c =\)1\(e\)-4 is the weight related to collision avoidance, \(\epsilon =\)1\(e\)-5 is a small value to avoid singularity, and \(\min(\text{dist})\) denotes the minimum distance between the spheres of different components.
The complexity and high dimensionality of this problem are underscored by the fact that each object in our example consists of \(num_{sphere} = 100 \) spheres, and each component is controlled by six variables -- three for displacement and three for orientation. The problem also incorporates constraints related to collision avoidance. To effectively navigate the highly non-convex and constrained space of the component packing problem, our approach involves tailored adaptations to the perturbation mechanism used in the Steepest Perturbed Gradient Descent (SPGD) algorithm. Perturbations are applied separately to the components' displacement and orientation, ensuring a thorough optimization of both aspects of component placement.
In the early iterations, we enhance the exploration and facilitate the escape from suboptimal solutions by accepting solutions with worse volume outcomes by a prescribed factor. This acceptance factor decreases in a linear profile over the iterations until it reaches $1.0$, at which point the algorithm only accepts new solutions that have the same or lower volume, thus refining the search towards the most compact configurations. Additionally, the amplitude of the perturbations for both displacement and orientation is controlled through a lower-bounded linear profile, which ensures that perturbations decrease in magnitude as the optimization process progresses, aligning more closely with the finer adjustments needed as the solution space is narrowed down. To further optimize the perturbation process and avoid ineffective perturbations, especially in cases where objects are too close to each other to allow for meaningful spatial adjustments, the frequency of perturbations is reduced using a lower bounded linear profile. This adaptive frequency adjustment helps prevent unnecessary computational expenditure on perturbations that are unlikely to be accepted due to collision constraints. These strategic adaptations enable SPGD to more effectively handle the complexities of packing diverse objects into a constrained space, making it robust against the challenges posed by the non-convex nature of the problem.
The implementation of this algorithm is carried out in Python using the PyTorch framework, which leverages CUDA for accelerated computation on GPUs. This setup allows for substantial improvements in computational efficiency, essential for managing the high-dimensional space of this problem. Using \texttt{torch.autograd}, we automatically compute the partial derivatives of the loss function with respect to the displacement and orientation vectors of each component. This gradient information is then used to update the component positions and orientations according to the update rule of gradient descent (\ref{eqn:GradientDescent}), akin to methods typically employed in deep learning optimizations. To further enhance the exploration capabilities of the optimization process, the sequence of component perturbations is shuffled in each iteration, promoting a more robust search through the solution space. In evaluating the effectiveness of the SPGD algorithm, we conducted a comparative analysis with the traditional Gradient Descent (GD) method across a series of increasingly complex packing scenarios.
The scenarios were designed to assess both algorithms under various conditions, ranging from uniform object sizes to irregular and diverse shapes, thereby testing their adaptability and efficiency in real-world packing challenges. Our implementation of Simulated Annealing diverged rather than converged, particularly in complex scenarios. This divergence can largely be attributed to the restrictive collision constraints integrated within the objective function (\ref{eqn:SPI2}), which prevent objects from moving through each other. Unlike the approach taken in reference \cite{Packing_SA}, where collision constraints were relaxed and followed by refinement steps, our implementation maintained these constraints, leading to no evident signs of convergence and indicating the unsuitability of Simulated Annealing for these applications.
Moreover, Perturbed Gradient Descent (PGD) was not utilized in the 3D packing problem comparison. The reason for this is twofold: firstly, the norm of the partial derivative vector in this problem setting does not approach zero due to the direct inclusion of collision constraints within the objective function. Secondly, the primary cause for algorithm termination is often the occurrence of collisions between objects, which deviates from the typical operational premise of PGD. Additionally, PGD's poor performance in separate 2D benchmark functions, which feature complex and challenging loss landscapes, further illustrates its limitations in navigating complicated optimization scenarios. This combination of factors reaffirms the decision to exclude PGD from the comparative analysis in our 3D packing problem.
\subsection{Initial Configuration and Setup}
Before delving into the comparative results, it is essential to note that both the SPGD and GD algorithms were initiated from the same configuration in each scenario to ensure a fair comparison. The initial setup involved distributing the objects well within the 3D space, providing sufficient free space around each object to avoid immediate collisions. Furthermore, the orientations of the objects were randomly chosen, introducing additional complexity and ensuring that the problem remained challenging for the optimizers. This initialization strategy was crucial for testing the algorithms' abilities to effectively explore and optimize from a non-advantageous starting point.
\subsection{Experimental Scenarios and Results}
The following scenarios were considered for the comparison:
\begin{itemize}
\item \textbf{Scenario 1:} Four identical cubes, where the global optimum is known, and cubes are packed together with the same orientation.
\item \textbf{Scenario 2:} Eight identical cubes, testing the scalability and spatial reasoning of the algorithms.
\item \textbf{Scenario 3:} Eight cubes of varying sizes, introducing a non-uniform configuration without a known global optimum, to evaluate heuristic capabilities.
\item \textbf{Scenario 4:} Eight objects of different complex shapes (gears, hooks, rivets, etc.), representing an industrial challenge with an unknown optimal packing configuration.
\end{itemize}
\subsection{Analysis of Scenario 1: Four Identical Cubes}
In Scenario 1, the initial configuration of the four identical cubes is depicted in Figure~\ref{fig:initial_config_scenario_1}. This setup was designed to test each algorithm's ability to navigate a relatively simple scenario where the global optimum involves aligning all cubes in a compact configuration. The results of the final configurations found by the GD and SPGD algorithms are illustrated in Figure~\ref{fig:final_configs_scenario_1}, showing both Gradient Descent and Steepest Perturbed Gradient Descent results side by side.
\begin{figure}[H]
\centering
\includegraphics[width=0.6\textwidth]{Figures/Rendered-adjusted/1/Render_Initial.png}
\caption{Initial configuration of four identical cubes in Scenario 1.}
\label{fig:initial_config_scenario_1}
\end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/4Box/Final_GD.pdf}
% \caption{Final configuration by Gradient Descent for Scenario 1.}
% \label{fig:gd_final_scenario_1}
% \end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/4Box/Final_SPGD.pdf}
% \caption{Final configuration by SPGD for Scenario 1, demonstrating convergence to the global optima. This represents a significantly superior solution compared to GD methods.}
% \label{fig:spgd_final_scenario_1}
% \end{figure}
\begin{figure}[H]
\centering
\begin{subfigure}{0.55\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/1/Render_GD.png}
\caption{GD}
\label{fig:gd_final_scenario_1}
\end{subfigure}\hfill
\begin{subfigure}{0.45\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/1/Render_SPGD.png}
\caption{SPGD}
\label{fig:spgd_final_scenario_1}
\end{subfigure}
\caption{Comparative final configurations for Scenario 1 by GD and SPGD. The GD configuration shows typical convergence behaviors, while SPGD demonstrates a convergence to the global optimum, representing a significantly superior solution compared to traditional GD methods.}
\label{fig:final_configs_scenario_1}
\end{figure}
The outcomes depicted in the figures reveal that, due to the collision constraint, GD struggled to converge to the global solution and settled in a suboptimal local minimum. In contrast, SPGD successfully converged to the global optimal configuration, effectively avoiding local minima and fulfilling the collision constraints more efficiently. To further illustrate the performance dynamics over the course of the optimization, the loss convergence history for both algorithms is plotted in Figure~\ref{fig:loss_time_scenario_1}. This figure shows loss values as a function of elapsed time and the number of iterations.
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{Figures/Convergence_Graphs/Scenario1_loss_side_by_side.pdf}
\caption{Loss convergence history based on elapsed time and the number of iterations for GD and SPGD in Scenario 1.}
\label{fig:loss_time_scenario_1}
\end{figure}
Although SPGD achieved the optimal configuration more rapidly in terms of the number of iterations, it required more computational time overall compared to GD. These plots (Figures~\ref{fig:loss_time_scenario_1}) help demonstrate that while SPGD's iterations are more effective at progressing toward the global optimum, they are computationally more intensive, likely due to the complexity of the perturbation calculations and the more sophisticated collision checks involved.
This scenario underscores SPGD's strengths in effectively navigating optimization landscapes with collision constraints and its ability to reach global optima where traditional GD may fail. However, the increased computational demand highlights an area for further optimization and efficiency improvements in SPGD's implementation.
\subsection{Analysis of Scenario 2: Eight Identical Cubes}
In Scenario 2, the initial configuration of eight identical cubes is depicted in Figure ~\ref{fig:initial_config_scenario_2}. This scenario was designed to assess each algorithm's ability to scale and manage increased numbers of objects while maintaining an efficient packing configuration. The outcomes of the final configurations found by the GD and SPGD algorithms are illustrated in Figures ~\ref{fig:gd_final_scenario_2} and ~\ref{fig:spgd_final_scenario_2}, respectively.
\begin{figure}[H]
\centering
\includegraphics[width=0.75\textwidth]{Figures/Rendered-adjusted/2/Render_Initial.png}
\caption{Initial configuration of eight identical cubes in Scenario 2.}
\label{fig:initial_config_scenario_2}
\end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/8Box/Final_GD.pdf}
% \caption{Final configuration by Gradient Descent for Scenario 2, illustrating the collision issue.}
% \label{fig:gd_final_scenario_2}
% \end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/8Box/Final_SPGD.pdf}
% \caption{Final configuration by SPGD for Scenario 2, showing a more compact arrangement than GD final configuration.}
% \label{fig:spgd_final_scenario_2}
% \end{figure}
\begin{figure}[H]
\centering
\begin{subfigure}{0.65\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/2/Render_GD.png}
\caption{GD}
\label{fig:gd_final_scenario_2}
\end{subfigure}\hfill
\begin{subfigure}{0.35\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/2/Render_SPGD.png}
\caption{SPGD}
\label{fig:spgd_final_scenario_2}
\end{subfigure}
\caption{Comparative final configurations for Scenario 2 by GD and SPGD. GD's final arrangement demonstrates collision challenges, hindering optimal packing. In contrast, SPGD achieves a more compact configuration, effectively utilizing its adaptive perturbations to overcome collision barriers and improve packing density.}
\label{fig:final_configs_scenario_2}
\end{figure}
During the optimization process, the GD algorithm encountered significant issues and ceased further packing adjustments due to a collision between the yellow and red cubes, effectively stopping the optimization prematurely. In contrast, the SPGD algorithm managed to navigate around this problem and did not converge to the global optimal solution but found a notably more compact suboptimal solution, approximately three times more space-efficient than the configuration found by GD.
To further illustrate the performance dynamics over the course of the optimization, the loss convergence history for both algorithms is plotted in Figure ~\ref{fig:loss_time_scenario_2}, showing loss values as a function of elapsed time and the number of iterations.
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{Figures/Convergence_Graphs/Scenario2_loss_side_by_side.pdf}
\caption{Loss convergence history based on elapsed time and the number of iterations for GD and SPGD in Scenario 2.}
\label{fig:loss_time_scenario_2}
\end{figure}
Although SPGD did not achieve the global optimum, it provided a significant improvement over GD by finding a much more compact solution rapidly. This scenario demonstrates SPGD's superior capability in effectively navigating complex landscapes and managing collision constraints dynamically compared to GD. The increased performance in finding a substantially better solution highlights the potential of SPGD for more effective space utilization in packing problems.
\subsection{Analysis of Scenario 3: Eight Cubes of Different Sizes}
In Scenario 3, which introduces a higher level of complexity due to the use of eight cubes of different sizes, the initial configuration is shown in Figure ~\ref{fig:initial_config_scenario_3}. This setup challenges the algorithms' ability to efficiently manage and optimize space in a more heterogeneous environment.
\begin{figure}[H]
\centering
\includegraphics[width=0.55\textwidth]{Figures/Rendered-adjusted/3/Render_Initial.png}
\caption{Initial configuration of eight cubes of different sizes in Scenario 3.}
\label{fig:initial_config_scenario_3}
\end{figure}
The SPGD algorithm's performance in this scenario was notably superior, as it converged to a more compact solution significantly faster than the traditional GD method. The results of the final configurations found by the GD and SPGD algorithms are shown in Figures ~\ref{fig:gd_final_scenario_3} and ~\ref{fig:spgd_final_scenario_3}, respectively.
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/8Box_NonUniform/Final_GD.pdf}
% \caption{Final configuration by GD for Scenario 3.}
% \label{fig:gd_final_scenario_3}
% \end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/8Box_NonUniform/Final_SPGD.pdf}
% \caption{Final configuration by SPGD for Scenario 3.}
% \label{fig:spgd_final_scenario_3}
% \end{figure}
\begin{figure}[H]
\centering
\begin{subfigure}{0.6\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/3/Render_GD.png}
\caption{GD}
\label{fig:gd_final_scenario_3}
\end{subfigure}\hfill
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/3/Render_SPGD.png}
\caption{SPGD}
\label{fig:spgd_final_scenario_3}
\end{subfigure}
\caption{Comparative final configurations for Scenario 3: Gradient Descent (left) shows less optimized packing, while Steepest Perturbed Gradient Descent (right) demonstrates a more compact and efficient arrangement.}
\label{fig:final_configs_scenario_3}
\end{figure}
Despite the lack of a known global optimal solution due to the varying sizes and potential configurations, SPGD effectively utilized its perturbation mechanism to explore and optimize the packing arrangement. This scenario highlights the algorithm's adaptability and efficiency in handling diverse object dimensions, which is crucial for real-world applications.
To further evaluate the performance dynamics, the loss convergence history for both algorithms is plotted in Figure ~\ref{fig:loss_time_scenario_3}, showing loss values as a function of elapsed time, and the number of iterations.
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{Figures/Convergence_Graphs/Scenario3_loss_side_by_side.pdf}
\caption{Loss convergence history based on elapsed time and the number of iterations for GD and SPGD in Scenario 3.}
\label{fig:loss_time_scenario_3}
\end{figure}
These figures demonstrate that SPGD not only achieves a more desirable outcome but also does so with greater computational efficiency in terms of iteration count, despite the complex interplay of different-sized objects. This efficiency underscores SPGD's potential as a robust tool for tackling sophisticated packing challenges where traditional methods might falter.
\subsection{Analysis of Scenario 4: Eight Objects of Different Shapes}
Scenario 4, the most complex of the scenarios tested, involved packing eight objects of different, irregular shapes such as gears, hooks, and rivets. The initial configuration is illustrated in Figure \ref{fig:initial_config_scenario_4}, which presents a diverse and challenging packing environment.
\begin{figure}[H]
\centering
\includegraphics[width=0.55\textwidth]{Figures/Rendered-adjusted/4/Render_Inital.png}
\caption{Initial configuration of eight objects of different shapes in Scenario 4.}
\label{fig:initial_config_scenario_4}
\end{figure}
In this demanding scenario, the SPGD algorithm demonstrated its robust capability by converging to a significantly more compact solution compared to the traditional GD method. Although the time taken by SPGD to find the optimal solution was comparable to that of GD, the overall optimization process required more time due to the termination condition set for no improvement in the loss value over 2000 iterations. The final configurations achieved by the GD and SPGD algorithms are shown in Figure ~\ref{fig:gd_final_scenario_4} and ~\ref{fig:spgd_final_scenario_4}, reflecting the SPGD algorithm’s effectiveness in handling complex and varied object forms.
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/8ComplexShapes/Final_GD.pdf}
% \caption{Final configuration by GD for Scenario 4.}
% \label{fig:gd_final_scenario_4}
% \end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.5\textwidth]{Figures/8ComplexShapes/Final_SPGD.pdf}
% \caption{Final configuration by SPGD for Scenario 4, achieving a 19.6\% more compact configuration than GD, showcasing a significant improvement in solution quality in a complex scenario.}
% \label{fig:spgd_final_scenario_4}
% \end{figure}
\begin{figure}[H]
\centering
\begin{subfigure}{0.6\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/4/Render_GD.png}
\caption{GD}
\label{fig:gd_final_scenario_4}
\end{subfigure}\hfill
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=\linewidth]{Figures/Rendered-adjusted/4/Render_SPGD.png}
\caption{SPGD}
\label{fig:spgd_final_scenario_4}
\end{subfigure}
\caption{Comparative final configurations for Scenario 4: Gradient Descent (a) struggles with complexity, while Steepest Perturbed Gradient Descent (b) demonstrates a significant improvement, achieving a more compact arrangement by 19.6\%.}
\label{fig:final_configs_scenario_4}
\end{figure}
To highlight the dynamic performance of both algorithms in this scenario, Figure ~\ref{fig:loss_time_scenario_4} presents the loss convergence history based on elapsed time, and the number of iterations.
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{Figures/Convergence_Graphs/Scenario4_loss_side_by_side.pdf}
\caption{Loss convergence history based on elapsed time for GD and SPGD in Scenario 4.}
\label{fig:loss_time_scenario_4}
\end{figure}
These results underscore the SPGD algorithm’s capacity to adapt to and effectively manage the intricacies of packing highly irregular objects. Although the time to reach the optimal solution was similar for both algorithms, SPGD’s ability to achieve a more compact arrangement highlights its suitability for complex, real-world packing problems where shape diversity plays a critical role. The extended time required for optimization termination points to the rigorous nature of the stopping criterion, ensuring that the solution is indeed optimal before termination.
Results of these experiments are summarized in the table~\ref{tab:packing_comparison}, which compares the performance of SPGD and GD in terms of best loss, and volume.
\begin{table}[H]
\caption[Table]{Final Loss and Volume Comparison of SPGD and GD across different packing scenarios}
\centering{%
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Scenario} & \textbf{Method} & \textbf{Best Loss} & \textbf{Volume} \\ \hline
\multirow{2}{*}{1} & SPGD & 74.59 & 7.12 \\ \cline{2-4}
& GD & 108.69 & 13.04 \\ \hline
\multirow{2}{*}{2} & SPGD & 137.38 & 16.51 \\ \cline{2-4}
& GD & 465.53 & 58.24 \\ \hline
\multirow{2}{*}{3} & SPGD & 145.59 & 17.60 \\ \cline{2-4}
& GD & 225.36 & 27.46 \\ \hline
\multirow{2}{*}{4} & SPGD & 103.26 & 12.34 \\ \cline{2-4}
& GD & 123.40 & 14.76 \\ \hline
\end{tabular}
}
\label{tab:packing_comparison}
\end{table}
This analysis highlights the superior adaptability and performance of SPGD, particularly in scenarios involving complex and non-uniform object configurations. The algorithm's ability to effectively shuffle and perturb component sequences contributes significantly to its success in navigating the intricate landscapes presented by these diverse packing challenges.
%\subsubsection*{Neural Network Training Problem} Neural Network training is a crucial subset of machine learning tasks, encompassing various objectives such as classification and regression. In this context, the training process is formulated as an optimization problem where the weights and biases of the network are the design variables. The goal is to minimize the loss function, which is tailored to the specific objectives of the task—whether predicting continuous outcomes in regression or categorizing data in classification \cite{kaveh2023application}.
%In the training of deep neural networks, which typically consist of a large number of hidden layers and thus an extensive array of design variables, the complexity and dimensionality of the search space dramatically increase. This complexity poses a unique challenge: as the network depth increases, so does the number of perturbations required to effectively navigate through the non-uniform and intricate optimization landscape \cite{choromanska2015loss}. To address this issue without incurring prohibitive computational costs, we implemented a strategic adaptation inspired by the principles of transfer learning.
%Rather than applying perturbations across all layers of the network, we focused our perturbation strategy on the last layer only. This approach is motivated by the concept of transfer learning, where the deeper features of a network, which are often more general, are reused or slightly modified, while the final layers, more specific to particular tasks, are fine-tuned to adapt to new tasks \cite{zhu2023transfer, theodoris2023transfer}. By limiting our perturbations to the last layer, we significantly simplify the perturbation process and reduce the computational overhead involved.
%For our experiments, we utilized the MNIST dataset, which consists of digital handwriting samples. This dataset is a standard benchmark in machine learning for evaluating the performance of algorithms in image recognition tasks \cite{deng2012mnist}. We applied the SPGD method to train both multilayer perceptron (MLP) neural networks and convolutional neural networks (CNNs) \cite{celard2023survey}.
%It is important to note that the PGD method, typically used in our test function experiments as a comparison method, is less applicable in the neural network training context. This is primarily because conditions where the gradient is near zero, which signal a trapping point in standard optimization problems, are not straightforward or practical indicators in the dynamic landscapes of neural network training.
%In this study, SPGD was integrated with standard PyTorch optimizers like SGD and ADAM to compare its effectiveness. Unlike traditional methods that may focus exclusively on reducing loss, SPGD's integration aims to optimize the training process by reducing the loss more efficiently and potentially enhancing the resultant accuracies of the network.
%%%%% Conclusions %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}\label{sec:Conclusion}
The SPGD algorithm presents a novel integration of deterministic optimization with strategic stochastic perturbations, designed to overcome the limitations of traditional gradient descent methods in non-convex landscapes and plateaus. Through comparative analyses, SPGD has demonstrated potential advantages in complex non-convex optimization challenges, consistently converging to the global optimum across 30 randomized trials per benchmark function. These results highlight both the robustness and practical utility of SPGD across a wide range of optimization scenarios.
Looking ahead, SPGD shows promise for broader applications in diverse domains and enhancements in machine learning methodologies:
\begin{itemize}
\item \textbf{Expanding Application Domains:} Future investigations could explore SPGD's application to fields like engineering design optimization \cite{zhang2018finding}, logistics, energy management, bioinformatics, and fuzzy logic parameter tuning optimization \cite{2022FuzzyGS}, showcasing its versatility and robustness.
\item \textbf{Enhancements in Machine Learning:} There is potential for SPGD to significantly enhance neural network training, especially within deep learning frameworks by improving convergence rates and navigating complex parameter spaces.
\item \textbf{Integration with Machine Learning Frameworks:} SPGD has already been implemented using the PyTorch framework for the 3D component packing problem, demonstrating its adaptability to complex optimization tasks. Future work could extend this integration to machine learning projects, particularly in training neural networks, thereby potentially broadening its user base and enhancing its utility in diverse applications.
\item \textbf{Adaptive Perturbation Strategies:} Developing adaptive perturbation techniques that respond to specific characteristics of the optimization landscape could further refine SPGD's effectiveness, making it more problem-specific.
\item \textbf{Extension to Complex Systems:} Exploring the 3D Component Packing Problem within the SPI2 framework could pave the way for handling interconnected systems with physical interactions, where topology and collision constraints add layers of complexity.
\end{itemize}
These future directions not only aim to broaden the utility of SPGD but also open new avenues for innovative research in the field of optimization.
%%%%% Acknowledgments %%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Acknowledgments}
This work was supported in part by the National Science Foundation grants CMMI-2232612, CMMI-2312175, and by the Defense Advanced Research Projects Agency (DARPA) under the ``Multi-Disciplinary Optimization for Packaging (MDOP)'' program, grant number FA8750-23-C-0501.
%%% REFERENCES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%% Put your references into your .bib file in the usual way. Run latex once, bibtex once, then latex twice.
%% The asmeconf.bst style allows @inproceedings and @proceedings to include:
%% venue = {Location of Conference},
%% eventdate = {Month, days},
%\nocite{*}%% <=== Delete this line unless you want to typeset the entire contents of your .bib file!
\bibliographystyle{unsrt} %% .bst file following ASME conference format. Do not change.
\bibliography{asmeconf-sample}%% <=== change this to name of your bib file
\end{document}
\begin{thebibliography}{10}
\bibitem{cauchy1847methode}
Augustin Cauchy et~al.
\newblock M{\'e}thode g{\'e}n{\'e}rale pour la r{\'e}solution des systemes d’{\'e}quations simultan{\'e}es.
\newblock {\em Comp. Rend. Sci. Paris}, 25(1847):536--538, 1847.
\bibitem{ruder2016overview}
Sebastian Ruder.
\newblock An overview of gradient descent optimization algorithms.
\newblock {\em arXiv preprint arXiv:1609.04747}, 2016.
\bibitem{jin2017escape}
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham~M Kakade, and Michael~I Jordan.
\newblock How to escape saddle points efficiently.
\newblock In {\em International conference on machine learning}, pages 1724--1732. PMLR, 2017.
\bibitem{Kirkpatrick1983OptimizationBS}
Scott Kirkpatrick, C~Daniel Gelatt~Jr, and Mario~P Vecchi.
\newblock Optimization by simulated annealing.
\newblock {\em Science}, 220(4598):671--680, 1983.
\bibitem{holland1992genetic}
John~H Holland.
\newblock Genetic algorithms.
\newblock {\em Scientific American}, 267(1):66--73, 1992.
\bibitem{delahaye2019simulated}
Daniel Delahaye, Supatcha Chaimatanan, and Marcel Mongeau.
\newblock Simulated annealing: From basics to applications.
\newblock {\em Handbook of metaheuristics}, pages 1--35, 2019.
\bibitem{ghannadi2023review}
Parsa Ghannadi, Seyed~Sina Kourehli, and Seyedali Mirjalili.
\newblock A review of the application of the simulated annealing algorithm in structural health monitoring (1995-2021).
\newblock {\em Frattura ed Integrit{\`a} Strutturale}, 17(64):51--76, 2023.
\bibitem{shahriari2016taking}
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan~P Adams, and Nando De~Freitas.
\newblock Taking the human out of the loop: A review of bayesian optimization.
\newblock {\em Proceedings of the IEEE}, 104(1):148--175, 2016.
\bibitem{snoek2012practical}
Jasper Snoek, Hugo Larochelle, and Ryan~P Adams.
\newblock Practical bayesian optimization of machine learning algorithms.
\newblock In {\em Advances in neural information processing systems}, volume~25, 2012.
\bibitem{frazier2018tutorial}
Peter~I Frazier.
\newblock A tutorial on bayesian optimization.
\newblock {\em arXiv preprint arXiv:1807.02811}, 2018.
\bibitem{burke2020gradient}
James~V Burke, Frank~E Curtis, Adrian~S Lewis, Michael~L Overton, and Lucas~EA Sim{\~o}es.
\newblock Gradient sampling methods for nonsmooth optimization.
\newblock {\em Numerical nonsmooth optimization: State of the art algorithms}, pages 201--225, 2020.
\bibitem{sutskever2013importance}
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.
\newblock On the importance of initialization and momentum in deep learning.
\newblock In {\em International conference on machine learning}, pages 1139--1147. PMLR, 2013.
\bibitem{jin2018accelerated}
Chi Jin, Praneeth Netrapalli, and Michael~I Jordan.
\newblock Accelerated gradient descent escapes saddle points faster than gradient descent.
\newblock In {\em Conference On Learning Theory}, pages 1042--1085. PMLR, 2018.
\bibitem{yiu2004hybrid}
Ka~Fai~Cedric Yiu, Yanqun Liu, and Kok~Lay Teo.
\newblock A hybrid descent method for global optimization.
\newblock {\em Journal of global optimization}, 28:229--238, 2004.
\bibitem{guo2022escaping}
Xin Guo, Jiequn Han, Mahan Tajrobehkar, and Wenpin Tang.
\newblock Escaping saddle points efficiently with occupation-time-adapted perturbations, 2022.
\bibitem{yang2013random}
Xin-She Yang, TO~Ting, and Mehmet Karamanoglu.
\newblock Random walks, l{\'e}vy flights, markov chains and metaheuristic optimization.
\newblock {\em Future Information Communication Technology and Applications: ICFICE 2013}, pages 1055--1064, 2013.
\bibitem{sun2022adaptive}
Tao Sun, Dongsheng Li, and Bao Wang.
\newblock Adaptive random walk gradient descent for decentralized optimization.
\newblock In {\em International Conference on Machine Learning}, pages 20790--20809. PMLR, 2022.
\bibitem{sussillo2014random}
David Sussillo and LF~Abbott.
\newblock Random walk initialization for training very deep feedforward networks.
\newblock {\em arXiv preprint arXiv:1412.6558}, 2014.
\bibitem{cunningham2015linear}
John~P Cunningham and Zoubin Ghahramani.
\newblock Linear dimensionality reduction: Survey, insights, and generalizations.
\newblock {\em The Journal of Machine Learning Research}, 16(1):2859--2900, 2015.
\bibitem{Fmincon}
MATLAB.
\newblock Find minimum of constrained nonlinear multivariable function - matlab fmincon.
\newblock \url{https://www.mathworks.com/help/optim/ug/fmincon.html}.
\newblock Accessed: March 25, 2024.
\bibitem{Fminunc}
MATLAB.
\newblock Find minimum of unconstrained nonlinear multivariable function - matlab fmincon.
\newblock \url{https://www.mathworks.com/help/releases/R2024b/optim/ug/fminunc.html}.
\newblock Accessed: April 30, 2025.
\bibitem{SA}
MATLAB.
\newblock Find minimum of function using simulated annealing algorithm.
\newblock \url{https://www.mathworks.com/help/gads/simulannealbnd.html}.
\newblock Accessed: March 25, 2024.
\bibitem{BO}
MATLAB.
\newblock Find minimum of function using simulated annealing algorithm.
\newblock \url{https://www.mathworks.com/help/stats/bayesian-optimization-algorithm.html}.
\newblock Accessed: April 30, 2025.
\bibitem{simulationlib}
S.~Surjanovic and D.~Bingham.
\newblock Virtual library of simulation experiments: Test functions and datasets.
\newblock Retrieved March 25, 2024, from \url{http://www.sfu.ca/~ssurjano}.
\bibitem{hazewinkel2001theory}
M~Hazewinkel.
\newblock Theory of errors.
\newblock {\em Encyclopedia of mathematics}, 62, 2001.
\bibitem{Peaksfunction-MATLAB}
{The MathWorks, Inc.}
\newblock Matlab peaks function - matlab.
\bibitem{machines10010042}
Wei Wei, Dong Yang, Li~Li, and Yuxuan Xia.
\newblock An intravascular catheter bending recognition method for interventional surgical robots.
\newblock {\em Machines}, 10(1), 2022.
\bibitem{ackley2012connectionist}
David Ackley.
\newblock {\em A connectionist machine for genetic hillclimbing}, volume~28.
\newblock Springer Science \& Business Media, 2012.
\bibitem{easom1990survey}
Eric~E Easom.
\newblock {\em A survey of global optimization techniques}.
\newblock PhD thesis, University of Louisville, 1990.
\bibitem{Levy1985TheTA}
A.~V. Levy and Antonio Montalvo.
\newblock The tunneling algorithm for the global minimization of functions.
\newblock {\em Siam Journal on Scientific and Statistical Computing}, 6:15--29, 1985.
\bibitem{gershenson2020successes}
Anne Gershenson, Shachi Gosavi, Pietro Faccioli, and Patrick~L Wintrode.
\newblock Successes and challenges in simulating the folding of large proteins.
\newblock {\em Journal of Biological Chemistry}, 295(1):15--33, 2020.
\bibitem{Levinthal}
Ken Dill and Hue Chan.
\newblock From {L}evinthal to pathways to funnels.
\newblock {\em Nature structural biology}, 4:10--9, 02 1997.
\bibitem{shahbazi2010hydrogen}
Zahra Shahbazi, Horea~T. Ilieş, and Kazem Kazerounian.
\newblock {Hydrogen Bonds and Kinematic Mobility of Protein Molecules}.
\newblock {\em Journal of Mechanisms and Robotics}, 2(2):021009, 04 2010.
\bibitem{madden2009residue}
Christopher Madden, Peter Bohnenkamp, Kazem Kazerounian, and Horea~T Ilie{\c{s}}.
\newblock Residue level three-dimensional workspace maps for conformational trajectory planning of proteins.
\newblock {\em The International Journal of Robotics Research}, 28(4):450--463, 2009.
\bibitem{tavousi2015protofold}
Pouya Tavousi, Morad Behandish, Horea~T Ilie{\c{s}}, and Kazem Kazerounian.
\newblock Protofold ii: Enhanced model and implementation for kinetostatic protein folding.
\newblock {\em Journal of Nanotechnology in Engineering and Medicine}, 6(3):034601, 2015.
\bibitem{mohammadi2023sign}
Alireza Mohammadi and Mohammad Al~Janaideh.
\newblock Sign gradient descent algorithms for kinetostatic protein folding.
\newblock In {\em 2023 International Conference on Manipulation, Automation and Robotics at Small Scales (MARSS)}, pages 1--6. IEEE, 2023.
\bibitem{peddada2022toward}
Satya~RT Peddada, Lawrence~E Zeidner, Horea~T Ilies, Kai~A James, and James~T Allison.
\newblock Toward holistic design of spatial packaging of interconnected systems with physical interactions (spi2).
\newblock {\em Journal of Mechanical Design}, 144(12):120801, 2022.
\bibitem{Packing_SA}
S.~Szykman and J.~Cagan.
\newblock {A Simulated Annealing-Based Approach to Three-Dimensional Component Packing}.
\newblock {\em Journal of Mechanical Design}, 117(2A):308--314, 06 1995.
\bibitem{SPI2_paper}
Satya R.~T. Peddada, Lawrence~E. Zeidner, Horea~T. Ilies, Kai~A. James, and James~T. Allison.
\newblock {Toward Holistic Design of Spatial Packaging of Interconnected Systems With Physical Interactions (SPI2)}.
\newblock {\em Journal of Mechanical Design}, 144(12):120801, 08 2022.
\bibitem{behzadi2014spi2-f}
Mohammad~M. Behzadi, Peter Zaffetti, Jiangce Chen, Lawrence~E. Zeidner, and Horea~T Ilie{\c{s}}.
\newblock Spatial component packing and routing optimization with physical interaction using maximal disjoint ball decomposition.
\newblock {\em Journal of Mechanical Design}, 2024.
\newblock in press.
\bibitem{lysenko2013fourier}
Mikola Lysenko.
\newblock Fourier collision detection.
\newblock {\em The International Journal of Robotics Research}, 32(4):483--503, 2013.
\bibitem{behandish2015}
Morad Behandish and Horea~T. Ilieş.
\newblock {Peg-in-Hole Revisited: A Generic Force Model for Haptic Assembly}.
\newblock {\em Journal of Computing and Information Science in Engineering}, 15(4):041004, 08 2015.
\bibitem{behandish2016analytic}
Morad Behandish and Horea~T Ilie{\c{s}}.
\newblock Analytic methods for geometric modeling via spherical decomposition.
\newblock {\em Computer-Aided Design}, 70:100--115, 2016.
\bibitem{kavraki1995computation}
Lydia~E Kavraki.
\newblock Computation of configuration-space obstacles using the fast fourier transform.
\newblock {\em IEEE Transactions on Robotics and Automation}, 11(3):408--413, 1995.
\bibitem{cui2023dense}
Qiaodong Cui, Victor Rong, Desai Chen, and Wojciech Matusik.
\newblock Dense, interlocking-free and scalable spectral packing of generic {3D} objects.
\newblock {\em ACM Trans. Graph.}, 42(4):141--1, 2023.
\bibitem{zhang2018finding}
Shanglong Zhang and Juli{\'a}n~A Norato.
\newblock Finding better local optima in topology optimization via tunneling.
\newblock In {\em International Design Engineering Technical Conferences and Computers and Information in Engineering Conference}, volume 51760, page V02BT03A014. American Society of Mechanical Engineers, 2018.
\bibitem{2022FuzzyGS}
Amir~Mohammad Vahedi, Hadi Nobahari, and Meysam Alizad.
\newblock Fuzzy gain scheduling of artificial potential fields for online path planning and obstacle avoidance of an aerial robot.
\newblock In {\em 2022 10th RSI International Conference on Robotics and Mechatronics (ICRoM)}, pages 309--316, 2022.
\end{thebibliography}