PtychoFormer: A Transformer-based Model for Ptychographic Phase Retrieval
Extracted source text · arXiv 2410.17377
\documentclass{article}
\usepackage{iclr2025_conference,times}
\input{math_commands.tex}
\input{custom_commands}
\usepackage[hyphens]{url} % DO NOT CHANGE THIS
\usepackage{graphicx} % DO NOT CHANGE THIS
% \urlstyle{rm} % DO NOT CHANGE THIS
% \def\UrlFont{\rm} % DO NOT CHANGE THIS
% \usepackage{natbib} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
\usepackage{caption} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
% \frenchspacing % DO NOT CHANGE THIS
% \setlength{\pdfpagewidth}{8.5in} % DO NOT CHANGE THIS
% \setlength{\pdfpageheight}{11in} % DO NOT CHANGE THIS
% \usepackage{abstract} % For handling abstracts
% \usepackage{graphicx} % Required for inserting images
% \usepackage{authblk} % For author and affiliation
% \usepackage{geometry} % For adjusting margins
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{multirow}
\usepackage{array}
\usepackage{adjustbox}
\usepackage{subcaption}
\usepackage{tabularx}
% \usepackage{lipsum}
\usepackage{pgfplots}
\usepackage{pgfplotstable}
\pgfplotsset{compat=1.17}
\usepackage{xcolor}
\usepackage{multicol}
\usepackage{float}
% \usepackage{wrapfig}
\usepackage{booktabs}
\usepackage{makecell}
% \usepackage{bm}
% \usepackage{svg}
\usepackage[inkscapearea=page]{svg}
% Define custom colors
% \definecolor{ptychoformer_amp}{RGB}{31, 119, 180}
% \definecolor{ptychoformer_phase}{RGB}{174, 199, 232}
% \definecolor{ptynon_amp}{RGB}{255, 127, 14}
% \definecolor{ptynon_phase}{RGB}{255, 187, 120}
% \definecolor{ptychnet_amp}{RGB}{44, 160, 44}
% \definecolor{ptychnet_phase}{RGB}{152, 223, 138}
% \definecolor{epie_amp}{RGB}{214, 39, 40}
% \definecolor{epie_phase}{RGB}{255, 152, 150}
% \newenvironment{keywords}
% {\bgroup\leftskip 20pt\rightskip 20pt \small\noindent{\bf Keywords:} }%
% {\par\egroup\vskip 0.25ex}
% \geometry{top=1in, bottom=1in, left=1in, right=1in}
\iclrfinaltrue
% For preprint mode
% Changes the running header
% Change the running header in line 96 in iclr_conference.sty file
\preprinttrue
% \renewcommand\Authfont{\normalsize}
% \renewcommand\Affilfont{\small}
\title{PtychoFormer: A Transformer-based Model for Ptychographic Phase Retrieval}
\begin{document}
\author{Ryuma Nakahata\textsuperscript{1}, Shehtab Zaman\textsuperscript{1}, Mingyuan Zhang\textsuperscript{2}, Fake Lu\textsuperscript{2},
Kenneth Chiu\textsuperscript{1} \\
\textsuperscript{1}School of Computing, Binghamton University \\
\textsuperscript{2}Department of Biomedical Engineering, Binghamton University \\
\texttt{\{rnakaha2,szaman5,mzhang97,kchiu,fakelu\}@binghamton.edu}
}
% \author[1]{Ryuma Nakahata}
% \author[1]{Shehtab Zaman}
% \author[1]{Kenneth Chiu}
% \author[2]{Fake Lu}
% \author[2]{Mingyuan Zhang}
% \affil[1]{Department of Computer Science, Binghamton University}
% \affil[2]{Department of Biomedical Engineering, Binghamton University}
\maketitle
\begin{abstract}
Ptychography is a computational method of microscopy that recovers high-resolution transmission images of samples from a series of diffraction patterns. While conventional phase retrieval algorithms can iteratively recover the images, they require oversampled diffraction patterns, incur significant computational costs, and struggle to recover the absolute phase of the sample's transmission function. Deep learning algorithms for ptychography are a promising approach to resolving the limitations of iterative algorithms. We present PtychoFormer, a hierarchical transformer-based model for data-driven single-shot ptychographic phase retrieval. PtychoFormer processes subsets of diffraction patterns, generating local inferences that are seamlessly stitched together to produce a high-quality reconstruction. Our model exhibits tolerance to sparsely scanned diffraction patterns and achieves up to 3600 times faster imaging speed than the extended ptychographic iterative engine (ePIE). We also propose the extended-PtychoFormer (ePF), a hybrid approach that combines the benefits of PtychoFormer with the ePIE. ePF minimizes global phase shifts and significantly enhances reconstruction quality, achieving state-of-the-art phase retrieval in ptychography.
% Moreover, the extended-PtychoFormer, a hybrid approach with the extended ptychographic iterative engine (ePIE), showcases improvements in its performance with a balance of speed and quality than the standalone ePIE.
% Things to add:
% : fine Fine Tune on real data; wait for data
\end{abstract}
% \begin{keywords}
% Deep Learning, Transformer, Ptychography, Diffractive Imaging
% \end{keywords}
\section{Introduction}
Microscopic and sub-microscopic imaging of tissue, cells, proteins, and crystals are a crucial tool in biological and materials sciences. Optical microscopic imaging relies on expensive lenses and capturing devices and is limited by the diffraction limit and lens aberrations. Furthermore, optical imaging requires high-contrast material, necessitating staining of transparent materials, making it impractical for live cell imaging.
Coherent diffractive imaging (CDI) is a "lensless" approach to imaging small, transparent objects by capturing downstream diffraction patterns free of lens aberrations.
Ptychography is a natural extension of CDI that captures high-resolution images by illuminating a sample at multiple scan points with a coherent light probe.
The coherent source for ptychography can be visible light, x-rays, or even electrons. Electron ptychography has captured some of the highest-resolution atomic images ever recorded~\citep{jiang2018electron}.
% Optical microscopic imaging relies on visible light and expensive lenses to magnify images
% Ptychography is a computational method of microscopic imaging that enables high-resolution imaging by illuminating a sample at multiple scan points with a coherent light probe. This technique scans overlapping regions of the sample to produce a set of diffraction patterns. These oversampled patterns enable ptychographic phase retrieval algorithms to recover the sample’s transmissivity, which is not easily attainable with other microscopy methods. Some of the highest resolution images ever have been captured using electron ptychorgaphy ~\citep{jiang2018electron}.
During ptychography, the sample is illuminated by the coherent probe at multiple scan points, and diffraction patterns of overlapping regions are captured. Since only the intensity of the interference can be captured downstream, losing the phase measurement, retrieving the sample's image poses a classic inverse problem.
% Too cluttered, move references to the variants in the related work
% Phase retrieval algorithms, like the ptychographic iterative engine (PIE) \citep{Rodenburg04}, its extension (ePIE) \citep{maiden2009}, refractive PIE (refPIE) \citep{Wittwer__22}, momentum-accelerated PIE (mPIE) \citep{Maiden17}, difference map \citep{Thibault08}, or maximum likelihood refinement \citep{Thibault_2012}, iteratively recover the sample's transmission function based on the diffraction patterns.
Phase retrieval algorithms, like the ptychographic iterative engine (PIE) \citep{Rodenburg04} and its variants, use the diffraction patterns to iteratively recover the sample's image as a transmission function.
This transmission function comprises of an amplitude, representing the intensity distribution of the sample, and a phase, which carries information about the sample's refractive index variations and internal material properties.
% Hence, ptychography is indispensable for observing the nanoscale characteristics of samples in material and biological research.
Conventional phase retrieval algorithms can recover the transmission function but have significant drawbacks. Firstly, their slow, iterative computation, typically taking minutes to hours to converge, makes real-time imaging impractical. Secondly, these algorithms require heavily dense scanning, which causes numerous data collection to image a sample. Thirdly, the recovered transmission function is prone to global phase shift, resulting in inaccurate phase recovery, as shown in the recovered phase using extended-PIE (ePIE) \citep{maiden2009} in Figure \ref{fig:ePIE_ePF}. In contrast, deep learning (DL) based phase retrieval algorithms offer a significant speed advantage. These networks are commonly convolutional neural networks (CNNs) trained to recover the transmission function from the diffraction patterns in a single forward pass, enabling rapid reconstruction in under a second. DL methods also exhibit more tolerance to sparse scan patterns, reducing the need to capture numerous diffraction patterns for the region of interest \citep{guan2019,cherukara2020,pan2023}.
\begin{figure}[t]
\centering
% \includesvg[width=1.0\linewidth]{Images/new_shift.svg}
\includegraphics[width=1.0\linewidth]{svg-inkscape/new_shift_svg-tex.pdf}
% \includesvg[width=1.0\linewidth]{Images/phaseoffset.svg}
\caption{
% (a), (b), (c), and (d) represent the ground truth image, ePF reconstruction, PtychoFormer reconstruction, and ePIE reconstruction, respectively. (e) shows a line profile comparison of pixel values across the predictions against the ground truth, illustrating the severe global phase shift from ePIE despite its convergence. (f) illustrates the sum squared error (SSE) for ePIE converges much quicker when initialized using PtychoFormer (ePF).
Comparison of the ground truth against the phase reconstructions from our proposed methods, PtychoFormer and extended-PtychoFormer (ePF), and ePIE. Sum squared error (SSE), the objective function for ePIE, cannot distinguish between globally shifted phase values, showing comparable SSE values between ePF and ePIE despite distinct line profiles. ePIE reconstruction is affected by a substantial global phase shift, while ePF achieves better estimation by leveraging PtychoFormer for initialization.
}
\label{fig:ePIE_ePF}
\end{figure}
Despite these advancements, many existing DL methods, particularly those relying on CNNs, fail to account for the spatial relationships between overlapping diffraction patterns—a fundamental aspect of ptychography, noted by \cite{konijnenberg2017}. This oversight limits the effectiveness of current models in capturing the intricate dependencies across diffraction patterns. Transformers, which have revolutionized natural language processing (NLP) and computer vision (CV) tasks by leveraging self-attention mechanisms \citep{Jia}, offer a promising solution. Their ability to capture long-range dependencies and contextual relationships \citep{Vaswani} makes them well-suited for ptychography, where understanding the spatial dependencies of diffraction patterns is vital.
In our work \textbf{(a)} we introduce \textbf{PtychoFormer}, a hierarchical transformer-based architecture designed to scale effectively with large datasets, \textbf{(b)} Our input scheme \textbf{preserves the relative scanned position} of each diffraction pattern, and the architecture enables \textbf{spatial awareness} by leveraging this spatial information.
\textbf{(c)} PtychoFormer outperforms previous DL phase retrieval methods in multiple phase and amplitude reconstruction tasks and is \textbf{up to 3600 times} faster than ePIE in our simulation.
% \textbf{(c)} Trained on over \textbf{2.5 million unique diffraction patterns} with varying probe functions and scan patterns, PtychoFormer demonstrates strong \textbf{transfer learning} to various scan configurations and untrained datasets, while also showing greater tolerance to sparser scan patterns, and \textbf{(d)} outperformed previous DL phase retrieval methods in multiple phase and amplitude reconstruction and is \textbf{up to 3600 times} faster than ePIE in our simulation.
Finally, \textbf{(d)} we propose a hybrid approach, \textbf{extended-PtychoFormer (ePF)}, combining PtychoFormer with ePIE.
ePF reduces NRMSE by \textbf{73.59\%} for amplitude and \textbf{47.30\%} for phase compared to ePIE while also minimizing the global phase shifts and accelerating convergence.
% In this paper:
% \begin{itemize}
% \item We introduce \textbf{PtychoFormer}, a hierarchical transformer-based architecture designed to scale effectively with large datasets.
% \item Our input scheme \textbf{preserves the relative scanned position} of each diffraction pattern, and the architecture enables \textbf{spatial awareness} by leveraging this spatial information.
% \item Trained on over \textbf{2.5 million unique diffraction patterns} with varying probe functions and scan patterns, PtychoFormer demonstrates strong \textbf{transfer learning} to various scan configurations and untrained datasets, while also showing greater tolerance to sparser scan patterns.
% \item Our model outperformed previous DL phase retrieval methods in multiple phase and amplitude reconstruction and is \textbf{up to 3600 times} faster than ePIE in our simulation.
% \item We propose a hybrid approach, \textbf{extended-PtychoFormer (ePF)}, combining PtychoFormer with ePIE. ePF reduces \textbf{NRMSE by 73.59\%} for amplitude and \textbf{47.30\% for phase} than ePIE while minimizing the global phase shifts and accelerating convergence.
% \end{itemize}
In \textbf{Section 2}, we describe the process of ptychography, the phase problem, and the ambiguity of the global phase shift. \textbf{Section 3} highlights related works in iterative and DL phase retrieval algorithms. \textbf{Section 4} details the formulation of PtychoFormer and ePF, and \textbf{Section 5} provides the empirical results by comparing them with other methods. Lastly, we discuss our findings and future directions in \textbf{Section 6} and \textbf{Section 7}, respectively.
% \section{Related Work}
\section{Background}
% This section provides an overview of ptychography, including the mathematical formulation of the transmission function and diffraction patterns, followed by a discussion of traditional iterative phase retrieval algorithms like PIE and ePIE. It concludes by reviewing recent deep learning-based advancements in ptychographic reconstruction, including CNN and transformer-based models.
% We provide a short overview of ptychography and the global phase shift ambiguity inherent to the phase problem.
In ptychography, the imaging sample is described by the complex-valued transmission function, $T(x,y)$. At any spatial coordinate $(x,y)$, the transmission function is defined in terms of its amplitude, $A(x,y)$, and phase, $\phi(x,y)$, as
\begin{equation}
T(x,y) = A(x,y) \cdot e^{i\phi(x,y)}.
\label{eq:transmission}
\end{equation}
The strict oversampling condition, where adjacent illumination areas must significantly overlap when capturing the diffraction patterns, is essential to recovering $T(x,y)$.
This stems from a well-known constraint of light detectors, like the charged-coupled devices (CCDs) or photographic plates, which only measure the light intensity of the exiting light and lose the phase (i.e. the phase problem). The measured intensity of the diffraction pattern, $I_j(u,v)$, is defined as
\begin{equation}
I_j(u, v) = \left| \mathcal{F} \left\{ T(x,y) \cdot P(x - X_j, y - Y_j) \right\} \right|^2,
\label{eq:diffraction}
\end{equation}
where $\mathcal{F}$ denotes the Fourier transform, $P(x,y)$ is the complex probe function, $(u,v)$ is the reciprocal space coordinates, and $(X_j, Y_j)$ are the lateral offsets at the $j$th scan position. Phase retrieval algorithms resolve the phase problem by numerically approximating the lost phase using the oversampled patterns. Without oversampling, the reconstruction would suffer from the ambiguities of the phase problem \citep{Shechtman}.
The global phase shift is a trivial ambiguity in the reconstructed transmission function, $\hat{T}(x,y)$, that has yet to be solved. It accounts for additional constant and linear terms to the phase, as
\begin{equation}
\hat{T}(x,y) = A(x,y) \cdot e^{i\phi(x,y)} \cdot e^{i(a + bx + cy)} = A(x,y) \cdot e^{i(\phi(x,y) + a + bx + cy)},
\label{eq:phase shift}
\end{equation}
where $a$, $b$, and $c$ are real-valued coefficients. The constant phase term, $a$, emerges from the phase problem, where the magnitude of the sample's phase is uncertain during reconstruction, allowing $a$ to take any value in $\hat{T}(x,y)$. The linear phase terms, $b$ and $c$, can appear when the diffraction patterns are off-centered in the computational window in Fourier space \citep{GuizarSicairos2011}. Hence, transmission estimates from conventional phase retrieval algorithms do not contain the correct absolute phase values and instead provide only relative phase shifts, due to the ambiguity introduced by the global phase shift.
\begin{figure}[t]
\centering
% \includesvg[width=\linewidth]{Images/overview.svg}
\includegraphics[width=\linewidth]{svg-inkscape/overview_svg-tex.pdf}
\caption{Comprehensive overview of the simulation using PtychoFormer and extended-PtychoFormer (ePF). (a) depicts the transmission function $T(x,y)$ characterized by its amplitude $A(x,y)$ and phase $\phi(x,y)$. The light probe $P(x,y)$ propagates through the sample to produce the diffraction pattern $I(u,v)$ in the far field. $I(u,v)$ are then grouped into sets of nine and placed in separate channels. (b) illustrates how PtychoFormer processes the sets in parallel to reconstruct local patches of $T(x,y)$ and then stitches them to complete the reconstruction. ePF framework builds on this approach by introducing an additional step (c), where the initial estimate from PtychoFormer is fed into ePIE for iterative refinement, further improving the reconstruction accuracy of $T(x,y)$.}
\label{fig:overview}
\end{figure}
\section{Related Work}
% In this section, we discuss the conventional phase retrieval algorithms such as PIE and ePIE, the current state of transformer models in CV, and existing DL phase retrieval algorithms for ptychography.
In this section, we discuss the current state of conventional phase retrieval algorithms and existing DL phase retrieval algorithms for ptychography.
% \textbf{Transformers in Computer Vision.}
% The Vision Transformer (ViT) \citep{dosovitskiy2021an} was a breakthrough in CV, surpassing state-of-the-art CNNs in image classification by capturing global context using self-attention and scaling its performance with larger datasets.
% Subsequently, Pyramid Vision Transformer \citep{wang2021}, Swin Transformer \citep{Swin}, and SegFormer \citep{xie2021} proposed various hierarchical transformers with optimized self-attention for object detection and segmentation tasks.
% \citet{Liang} and \citet{ding2022cogview} also showed the success of hierarchical transformers in super-resolution, denoising, and conditional image generation.
\textbf{Iterative Phase Retrieval Algorithms. }
The PIE algorithm \citep{Rodenburg04} laid the groundwork for phase retrieval algorithms in ptychography. \cite{maiden2009} further advanced this approach with the ePIE algorithm, which improved convergence and reconstruction quality by iteratively approximating the transmission and probe function from the diffraction patterns.
While several adaptations have since been developed, like mPIE \citep{Maiden17}, MAIC-PIE \citep{Dou20}, and refPIE \citep{Wittwer__22}, these algorithms are slow while requiring precise parameter tuning and densely scanned diffraction patterns for optimal performance.
% However, ePIE heavily relies on accurate parameter setup, highly coherent light probes, sufficiently overlapping diffraction patterns, and records of its scanned coordinates.
% The family of PIE algorithms also exists like that refractive PIE (refPIE) \citep{Wittwer__22}, momentum-accelerated PIE (mPIE) \citep{Maiden17}, multi-slice PIE, 3PIE, MAIC-PIE
% Thus, ePIE showcases robustness to imperfections in the optical system, which can otherwise introduce artifacts and hinder algorithm convergence.
% Despite the advancements, the iterative nature of conventional phase retrieval algorithms inhibits real-time imaging. Recent DL approaches offer faster ptychographic reconstruction and eliminate the need for precise parameter setup or accurate probe measurements.
% \textbf{Transformers in Computer Vision.}
% Transformers \citep{Vaswani} have revolutionized various fields, especially NLP, by leveraging the self-attention mechanism to capture long-range dependencies. This success has extended to CV with the Vision Transformer (ViT) \citep{dosovitskiy2021an}, which outperformed state-of-the-art CNNs in image classification. Subsequently, Pyramid Vision Transformer (PVT) \citep{wang2021}, Swin Transformer \citep{Swin}, and SegFormer \citep{xie2021} refined the architecture by proposing hierarchical transformers and computationally efficient self-attention for dense prediction tasks, such as object detection, semantic segmentation, and instance segmentation. Further research was conducted on transformers for image restoration tasks, like super-resolution, denoising, colorization, and JPEG compression \citep{Liang,cai2024}, making transformers ideal for advancing ptychographic imaging.
\textbf{Deep Learning Phase Retrieval Algorithms. }
DL phase retrieval algorithms eliminated the need for parameter tuning and dense scans, while substantially improving the reconstruction speed—enabling real-time imaging \citep{babu2023}.
However, many CNNs proposed for phase retrieval, such as PtychoNet \citep{guan2019}, PtychoNN \citep{cherukara2020}, and PtyNet \citep{pan2023}, process diffraction patterns one at a time, preventing them from leveraging the indispensable information from neighboring patterns.
\citet{gan2024} proposed PtychoDV to address this limitation by using a ViT and a deep unrolling network to process multiple diffraction patterns, each as a one-dimensional latent vector, with its scanned coordinates as positional embedding.
Although injecting coordinates provides a rough spatial indication, coordinates alone inadequately capture the overlap between the vectorized patterns with high granularity.
% Furthermore, its iterative architecture adds computational overhead, and employing a standard ViT falls short compared to more effective hierarchical transformers.
% Despite the strengths of this approach, there is potential for a more efficient transformer model that fully captures the spatial relationships between the patterns.
% Difference: encoding position vs inheriting positional information directly (overlap is hard to tell from just coordinates), Hierarchical transformer, architecture is iterative.
% \textbf{Transformers in Computer Vision.}
% Transformers \citep{Vaswani} have revolutionized various fields, especially natural language processing and computer vision.
% A key element of the success of transformer-based architectures is the self-attention mechanism to capture long-range dependencies.
% This success has extended to CV with the Vision Transformer (ViT) \citep{dosovitskiy2021an}, which outperformed state-of-the-art CNNs in image classification. Subsequently, Pyramid Vision Transformer (PVT) \citep{wang2021}, Swin Transformer \citep{Swin}, and SegFormer \citep{xie2021} refined the architecture by proposing hierarchical transformers and computationally efficient self-attention for dense prediction tasks, such as object detection, semantic segmentation, and instance segmentation. Further research was conducted on transformers for image restoration tasks, like super-resolution, denoising, colorization, and JPEG compression \citep{Liang,cai2024}, making transformers ideal for advancing ptychographic imaging.
\section{PtychoFormer}
\begin{figure}
\centering
% \includesvg[width=0.8\linewidth]{Images/method.svg}
\includegraphics[width=0.8\linewidth]{svg-inkscape/method_svg-tex.pdf}
\caption{Input scheme depicted in (a) groups the diffraction patterns into subsets of nine and are placed in separate channels. This way, the spatial relation between each pattern is preserved. (b) showcases the stitching process, where Local predictions are cropped and feathered at the edges. (c) compares the reconstructions with and without feathering, whereby feathering effectively eliminates the grid artifacts present in the reconstruction without feathering.}
\label{fig:method}
\end{figure}
We detail the pre and post-processing, and architectural details of PtychoFormer and extended-PtychoFormer in this section. To reconstruct the phase and amplitude of a sample, multiple diffraction images are pre-processed by arranging them according to relative spatial positions before inputting into our transformer-based model. Local sub-regional predictions from the model are then seamlessly stitched together via feathering to minimize grid artifacts. Optionally, the predicted outputs are iteratively refined using ePIE in the extended-PtychoFormer, our hybrid approach. Figure~\ref{fig:overview} provides an overview of our reconstruction process.
% This section outlines our methodology for developing our transformer-based phase retrieval algorithm, PtychoFormer, where we detail the input scheme for handling the diffraction patterns, the feathering technique to avoid stitching artifacts, and the model architecture. We also introduce the extended-PtychoFormer (ePF), which combines PtychoFormer with ePIE to enhance the reconstruction accuracy.
% Figure~\ref{fig:overview} pictures an overview of our reconstruction process.
% \subsection{Input Scheme}
\textbf{Input Processing. }
Our devised input scheme, depicted in Figure~\ref{fig:method}(a), handles up to nine spatially overlapping diffraction patterns, each separated into distinct channels. The diffraction patterns are arranged according to their relative scanned coordinates, maintaining the inherent overlap between adjacent patterns. This organization allows the model to focus on learning the informational and positional dependencies between the patterns. This input scheme has been tested on various scan patterns and with different numbers of diffraction patterns, shown in Figure~\ref{fig:probe_shape}(b).
% \subsection{Feathering}
% \textbf{Feathering}
% Our phase retrieval approach reconstructs local patches of the transmission, requiring an effective stitching algorithm. A common approach involves placing inferences in their respective spatial positions and averaging the overlapping regions \citep{cherukara2020, guan2019, pan2023}. However, this leads to grid-like artifacts due to slight differences in contrast between neighboring inferences.
% To avoid these artifacts, we use a technique called feathering, which combines independent inferences with a weighted average in the overlapping regions. We apply gradient masking to the inferences, so the pixel values linearly scale to 0 at the edges. As visualized in Figure~\ref{fig:method}(b) and (c), the feathered inferences are placed in their respective coordinates, and the pixel values are summed in the overlapping regions, effectively preventing any stitching artifacts.
\begin{figure}
\centering
% \includesvg[width=\linewidth]{Images/Model_Architecture.svg}
\includegraphics[width=\linewidth]{svg-inkscape/Model_Architecture_svg-tex.pdf}
\caption{PtychoFormer leverages a Mix Transformer (MiT) encoder and a convolutional decoder. The encoder includes four MiT stages that progressively reduce spatial resolution while increasing feature channels. The decoder upsamples the encoder outputs, adjusts feature channels, and refines the resolution to produce the amplitude and phase estimates.}
\label{fig:architecture}
\end{figure}
% \subsection{PtychoFormer Architecture}
% EXPLAIN PROBLEM, THEN REASON
% Reason for Hierarchical over ViT:
% The Vision Transformer (ViT) \citep{dosovitskiy2021an} was a breakthrough in CV, surpassing state-of-the-art CNNs in image classification by capturing global context using self-attention and scaling its performance with larger datasets. Subsequently, Pyramid Vision Transformer \citep{wang2021}, Swin Transformer \citep{Swin}, and SegFormer \citep{xie2021} proposed various hierarchical transformers with optimized self-attention for object detection and segmentation tasks. \citet{Liang} and \citet{ding2022cogview} also showed the success of hierarchical transformers in super-resolution, denoising, and conditional image generation.
% Give more precise reasons of hierarchical vs normal (similar to why CNN are encoder/decoder)
% Reason for MiT:
% Efficient Self Attention
% Reduced sequence length
% Whats the trade off? By is it worth it (efficiency)
% No Positional Embedding:
% Why is that good? and How does it achieve that? (zero padded CNN)
% Why needed: Enhances multi-resolution capabilities (??)
% Why not MLP like SegFormer, and replacing the Decoder to CNN?
% Fully connected MLP is expensive VS CNN is cheap/faster for larger matrices (search this)
% translational invariance in CNN, retaining locality of extracted info from diffraction
\textbf{Model Architecture. }
The PtychoFormer architecture, illustrated in Figure~\ref{fig:architecture} and detailed in Appendix \ref{sec:pf_implementation}, leverages a Mix Transformer (MiT) encoder from the SegFormer architecture \citep{xie2021}, along with a convolutional decoder to achieve efficient phase retrieval.
% In our research, we encountered several challenges that shaped our architectural choices.
The primary challenge was devising an encoder that effectively captures long-range dependencies between adjacent diffraction patterns. While CNNs excel at local feature extraction, they have limited receptive fields without substantially increasing the convolutional window size. In contrast, transformers, with their self-attention mechanisms, naturally capture long-range dependencies, which is crucial for understanding the relationships between diffraction patterns in our research.
We can not employ standard vision transformers like the Vision Transformer (ViT) \citep{dosovitskiy2021an} as (1) ViT extracts only a single-resolution feature map, lacking the expressiveness to capture informative representations of the diffraction pattern to generate the transmission function. (2) Self-attention has quadratic time and space complexity relative to input length, creating a computational bottleneck that slows inference speed. Furthermore, (3) ViT relies on fixed positional encoding (PE) to retain spatial information of input patches, inhibiting innate handling of multi-resolution inputs and degrading performance with untrained input resolution—a significant limitation given the varying resolutions from different scan configurations in our research.
We turn to MiT encoder to address these shortcomings.
(1) MiT's hierarchical architecture and overlapping patch merging enable the extraction of multi-level feature maps at different resolutions, similar to CNNs. This allows the encoder to capture information at varying scales by combining coarse and fine-grained details. Integrating features from different layers, each capturing specific aspects of the diffraction images, results in rich feature representations essential for accurate image generation. (2) MiT employs Spatial Reduction Attention \citep{wang2021}, which reduces the spatial dimensions of key and value sequences before the attention operation by a predefined reduction ratio. This approach has been shown to reduce computational and memory costs substantially than traditional self-attention, alleviating the computational bottleneck. (3) The Mix Feed Forward Network (Mix-FFN) replaces fixed PE with a zero-padded $3 \times 3$ convolution. As demonstrated by \citet{Islam}, zero-padded convolutions implicitly encode positional information. \citet{xie2021} empirically established that this method not only outperforms fixed PE but is also less sensitive to untrained input resolutions, aiding the handling of multi-resolution inputs in our experiments.
To retain spatial relationships, we develop a convolutional decoder instead of the MLP-based decoder in Segformer.
Convolution layers add additional inductive biases preserving locality and translation-equivariance.
% While SegFormer utilizes a multilayer perceptron (MLP) decoder designed for segmentation tasks, this is not optimal for image generation.
% MLPs inherently lose local spatial coherence due to their fully connected layers, which is detrimental when maintaining spatial relationships are crucial.
% On the contrary, convolutions preserves locality and are translationally equivariant due to weight sharing and local connectivity.
Our convolutional decoder leverages the multi-level feature maps from the MiT encoder more effectively than an MLP, integrating spatially localized features.
% Therefore, we replaced the MLP decoder with a convolutional decoder.
\textbf{Feathering. }
Our phase retrieval approach reconstructs local patches of the transmission, requiring an effective stitching algorithm. Typically, the patches are placed in their respective coordinates and average the pixel values of the overlapping regions \citep{cherukara2020, guan2019, pan2023}. However, even a slight difference in contrast between adjacent inferences leads to grid artifacts.
To avoid this, we use a technique called feathering to ensure smooth transitions between adjacent patches.
As visualized in Figure~\ref{fig:method}(b), we crop the patches to disregard the edges of each patch, where predictions are intentionally omitted, so only the central, well-defined regions of each patch contribute to the final reconstruction.
We then linearly taper the pixel values from where the overlap starts up to the edge of the cropped patches.
The feathered patches are placed in their respective coordinates, and the pixel values are summed in the overlapping regions, effectively preventing any stitching artifacts (Figure~\ref{fig:method}(c)).
% \subsubsection{Loss Function}
% \textbf{Training}
% We used the mean absolute error (MAE) loss function that minimizes the absolute pixel-wise difference between the target label, $Y$, and prediction, $\hat{Y}$. The MAE loss function is defined as
% \begin{equation}
% MAE = \frac{1}{N} \sum_{i=1}^{N} | Y_i - \hat{Y}_i |,
% \end{equation}
% where \( N\) is the number of pixels in the image, and $i$ represents the $i$th pixel in $Y$ and $\hat{Y}$.
% We implemented a normalization procedure to address the pronounced imbalance in the pixel value range between the two output channels, where the phase channel has a range of 2$\pi$ and the amplitude has a range of 1.
% The target and predicted phase values were normalized to [0,1] before computing the error to mitigate the imbalanced gradient magnitudes from each channel during backpropagation. The improvement in training is showcased in Appendix~\ref{sec:Phase_Norm}.
\textbf{Extended-PtychoFormer. }
The extended-PtychoFormer (ePF) is our hybrid approach that integrates PtychoFormer and ePIE. One of the challenges in algorithms like ePIE is their susceptibility to convergence on local minima, leading to inaccurate estimates of the transmission function \citep{Maiden17}. ePF leverages PtychoFormer to provide a well-informed initial estimate through the stitched prediction, serving as a robust starting point for ePIE. This strategy, similar to object initialization \citep{Wittwer2022}, aims to improve the convergence speed and the reconstruction accuracy.
\section{Experiments}
In this section, we present the experimental setup, which includes the dataset creation, training implementation, and evaluation metrics.
We then present key advantages of PtychoFormer, such as its tolerance to sparse scan patterns and strong transfer learning capabilities across different probe functions and scan patterns.
Following this, we compare PtychoFormer against DL methods like PtychoNN and PtychoNet. Lastly, we discuss how PtychoFormer and ePF compares against the ePIE algorithm.
\begin{figure}
\centering
% \includesvg[width=\linewidth]{Images/dataset.svg}
\includegraphics[width=\linewidth]{svg-inkscape/dataset_svg-tex.pdf}
\caption{A subset of probe functions we used are shown in (a). Probe A is the primary probe PtychoFormer is trained on and probe B is presented in the finetuning dataset, while probe C is reserved for testing. The grid formation in (b) is used for pre-training, and the other scan configurations are used for finetuning.}
\label{fig:probe_shape}
\end{figure}
\subsection{Experimental Setup}
Lack of training data can be a significant challenge in real-world settings; therefore, we explore the generalization capabilities of PtychoFormer. We devise a pre-training, fine-tuning, and testing pipeline to simulate real-world settings where certain experiments may only have a few samples to fine-tune. We generate the synthetic data and split up the dataset as follows.
\textbf{Diffraction Pattern Generation. } The synthetic benchmarks require us to generate diffraction patterns to train and test the model using standard images. Each diffraction pattern requires a pair of images, one for the amplitude and the other for the phase. Each pair of images is used to obtain the transmission function $T(x,y)$ following Eq.~\ref{eq:transmission}. Amplitude and phase pixel values are normalized to [0.05, 1] and $[-\pi, \pi]$, respectively.
Each 128 × 128 diffraction pattern is generated using Eq. \ref{eq:diffraction} by illuminating $T(x,y)$ with a probe along a prescribed scan configuration, visualized in Figure~\ref{fig:probe_shape}. Since the probe inherently imposes a finite support constraint, where $T(x,y)$ outside the illumination area is zero, we masked out the corresponding labels, $A(x,y)$ and $\phi(x,y)$, in the regions that are not illuminated.
Refer to Appendix~\ref{sec:label_detail} for details on label masking.
\textbf{Datasets. }
Our training dataset is based on the Flickr30k \citep{young2014} dataset.
We select 28,600 images from the dataset to create the pre-training and finetuning datasets.
Each image is center-cropped, resized, and converted to grayscale during pre-processing. Images are then randomly flipped and rotated to expand the dataset, ultimately generating 62,000 unique pairs of images. The pre-training dataset only uses probe A, presented in Figure \ref{fig:probe_shape}(a), in a grid scan with varying lateral offsets between diffraction patterns. Specifically, this dataset contains 244,800 samples of 36 different pixel-wise lateral offsets between adjacent scan points, ranging from 20 to 55 pixels (68.7\% to 20.3\% spatial overlaps).
% \textbf{Finetuning Dataset. }
A key requirement for the model is to adjust to new ptychographic setup conditions without needing a large amount of data. The fine-tuning dataset is designed to introduce new probe functions and scan patterns beyond the training data to adjust the model to new settings. Therefore, we generate fine-tuning samples by introducing 9 new probe functions, including probe B in \ref{fig:probe_shape}(a), and 7 alternate scan patterns, visualized in Figure \ref{fig:probe_shape}(b), that are not present in the pre-training dataset.
% \textbf{Test Dataset. }
We devise multiple test sets to perform an ablation study on various scenarios. We hold out 3,100 images from the Flickr30K dataset for testing. We also introduce an unseen probe function (probe C) and a 60-pixel lateral offset grid scan (14.9\% spatial overlap) to measure the out-of-distribution performance of our model on unseen setups. To test different domains of inputs, we further evaluate the model on two new datasets, Flower102 \citep{nilsback2008} and Caltech101 \citep{li2022}. Further details of each dataset can be found in Appendix ~\ref{appendix:data_details}.
% We generated diffraction patterns in a grid scan with 36 different lateral offsets, ranging from 20 to 55-pixel offsets between adjacent scan points. The offsets correspond to 68.7\% to 20.3\% spatial overlaps between the diffraction patterns. The training dataset consists of these 36 configurations, each containing 6,800 triplets of 9 simulated diffraction patterns and corresponding amplitude and phase labels, totaling 244,800 data points.
% Further, we created fine-tuning datasets to introduce 7 additional scan patterns and 11 different probe functions.
% The scan patterns, shown in Figure~\ref{fig:probe_shape}(b), included various configurations, from 5 to 9 diffraction patterns, diamond, parallelogram, and random layouts.
% 10 probe functions were used for training, including probes A and B, while probe C was used solely for testing (Figure~\ref{fig:probe_shape}(a)). The fine-tuning dataset included 2,000 samples per configuration.
\textbf{Training Details. }
We trained PtychoFormer using the mean absolute error (MAE) loss function to minimize the absolute pixel-wise difference between the target label, $Y$, and prediction, $\hat{Y}$, with a batch size of 32.
We normalized the phase of $Y$ and $\hat{Y}$ to [0,1] before computing the loss to ensure a balanced loss from the amplitude and phase channel for stable training.
See appendix~\ref{sec:Phase_Norm} for improved training curves resulting from phase normalization.
% We split the training and fine-tuning datasets into 90\% for training and 10\% for testing.
% We optimized the training using Prodigy, a parameter-free optimizer known for its performance comparable to a hand-tuned Adam optimizer \citep{Mishchenko2024}.
% Our training was conducted on an NVIDIA TITAN V GPU.
% \textbf{Comparison. }
% We compare the performance of PtychoFormer and ePF against ePIE, PtychoNN, and PtychoNet. The implementation and training environments of PtychoNN and PtychoNet follow what is outlined in \cite{cherukara2020} and \cite{guan2019}, respectively. The ePIE algorithm follows the implementation from \cite{maiden2009} and iterates until its sum squared error function (SSE) converges. See Appendix~\ref{sec:appendix-ePIE} for implementation details of ePIE.
\textbf{Evaluation Metrics. }
We used MAE and normalized root mean squared error (NRMSE) to separately evaluate the amplitude and phase reconstructions, where $\hat{Y}$ and $Y$ represent the target and estimated values for either amplitude or phase. Similar to \cite{gan2024}, we customized the NRMSE to measure the reconstruction quality after removing the global phase shift. This ensures that our evaluation reflects the true performance of the reconstruction without being skewed by global shifts. Our NRMSE is defined as
\begin{equation}
NRMSE = \frac{\| (\hat{Y} - \Tilde{a}) - Y \|}{\| Y \|},
\end{equation}
where $\Tilde{a}$ corrects the constant phase offset $a$ when computing the NRMSE of phase reconstructions. We determine $\Tilde{a}$ as the average pixel-wise difference between $\hat{Y}$ and $Y$. Since the diffraction images in our simulations are centered, we disregard the linear phase term as per Eq. \ref{eq:phase shift} and correct only the constant phase offset using $\Tilde{a}$. This term is zero when computing the NRMSE of the amplitude reconstructions.
% \subsection{Simulation Results of PtychoFormer}
\subsection{Results}
% \textbf{Generalization. }
We investigate the performance of our model using PtychoNN and PtychoNet as our baseline models for our experiments. We also examine the
generalization capabilities of PtychoFormer on unseen probe functions and new datasets.
% Our simulation results highlight PtychoFormer's exceptional transfer learning capabilities, spatial awareness, and multi-resolution processing.
% Initially trained mainly on grid scans with varying resolutions, the model adapted remarkably when fine-tuned on different probe functions and scan patterns, requiring minimal additional training.
% Despite introducing new probe functions and scan patterns that altered the spatial position of the patterns, it consistently delivered impressive reconstructions, including on test probe $C$ and random scan patterns.
% This performance demonstrates its multi-resolution capabilities and spatial awareness, effectively processing various input dimensions and providing accurate reconstructions from spatially varied diffraction patterns.
% The NRMSE values of these configurations are presented in Appendix~\ref{sec:tables}.
% The model's robustness was further demonstrated on unseen datasets.
% Without prior training, PtychoFormer achieved notable NRMSE values for amplitude and phase reconstructions: 0.18 ± 0.06 and 0.51 ± 0.26 on the Flower 102 dataset \citep{nilsback2008} and 0.28 ± 0.09 and 0.97 ± 0.65 on the Caltech 101 dataset \citep{li2022}.
% These results emphasize the strong generalization via transfer learning, enabling effective phase retrieval even for untrained datasets.
% In our simulation, PtychoFormer displayed robust versatility across various probe functions and scan patterns. The NRMSE values of these configurations are presented in Appendix~\ref{sec:tables}. The model excelled with grid formations, though NRMSE slightly increased with random formations and fewer diffraction patterns. The model also maintained consistent performance across all 10 probe functions, delivering impressive results on the test probe $C$.
% Furthermore, the model's adaptability was evident when tested on unseen datasets, achieving an NRMSE of 0.179 ± 0.06 and 0.51 ± 0.26 for amplitude and phase reconstruction on Flower 102 dataset \citep{nilsback2008}, and 0.275 ± 0.09 and 0.967 ± 0.745 on Caltech 101 dataset \citep{li2022}, respectively. These results highlight the model's generalized understanding of phase retrieval, even when applied to untrained testing scenarios.
% \subsection{Comparison with Deep Learning Methods}
\begin{figure}[htb]
\centering
% \includesvg[width=\linewidth]{Images/ML_comparison.svg}
\includegraphics[width=\linewidth]{svg-inkscape/ML_comparison_svg-tex.pdf}
\caption{(a) and (b) compare the reconstruction quality of PtychoFormer, PtychoNet, and PtychoNN, measured using normalized root mean squared error (NRMSE) and mean absolute error (MAE), respectively. Results are averaged from 3100 test samples, comprising of 6 × 6 diffraction patterns, each with 20-pixel lateral offsets. (c) depicts the amplitude and phase reconstructions along with the NRMSE values. PtychoFormer consistently outperforms PtychoNN and PtychoNet across all metrics and eliminates the impact of grid artifacts that heavily affect the latter methods.}
\label{fig:comparison}
\end{figure}
\textbf{Comparison with Deep Learning Methods. }
We compare PtychoFormer with PtychoNN and PtychoNet. The implementation and training environments of PtychoNN and PtychoNet follow what is outlined in \cite{cherukara2020} and \cite{guan2019}, respectively. PtychoNN and PtychoNet are trained to convergence on the pre-training set and all three models are tested on Flickr30K test set.
We can see in Figure \ref{fig:comparison} that PtychoFormer significantly outperforms both PtychoNN and PtychoNet for both amplitude and phase retrieval.
PtychoFormer reduces NMRSE by 25.93\% and 41.18\% for amplitude reconstruction over PtychoNN and PtychoNet, respectively.
All models found phase reconstruction to be more challenging than amplitude, as reflected in the higher MAE and NRMSE values for phase.
But PtychoFormer performs better phase reconstruction as well as compared to PtychoNN and PtychoNet, reducing NMRSE by 61.05\% and 55.33\%, respectively.
Figure~\ref{fig:comparison}(c) also highlights the benefit of feathering, as PtychoNN and PtychoNet are further affected by grid artifacts from their stitching algorithm. PtychoFormer, on the other hand, has a smooth reconstruction along the local prediction boundaries.
% Among the DL methods, PtychoFormer outperforms PtychoNN and PtychoNet under all metrics, as shown in Figure~\ref{fig:comparison}(a) and (b). However, all models found phase reconstruction to be more challenging than amplitude, as reflected in the higher MAE and NRMSE values for phase.
% This is expected since the phase of the diffraction pattern is not captured by the detector, making phase recovery of the $T(x,y)$ inherently more challenging.
% Visualized in Figure~\ref{fig:comparison}(c), PtychoNN and PtychoNet are further affected by grid-like artifacts from their stitching algorithm, but PtychoFormer eliminates these artifacts via feathering.
\textbf{Low to No Data Experiments. }
As mentioned before, we require the model to be usable under data-constrained scenarios. We evaluate PtychoFormer under low data availability constraints, where only fine-tuning is possible, as well as no data scenario, where only the pre-trained model is used. For low-data tests, as mentioned before, we introduce new scan patterns and probe functions. For each new configuration, the model is only trained on 2,000 new samples. Remarkably, we see little to no degradation in performance when introducing new scan patterns. The full tables of results are presented in Table \ref{tab:probe_values} and \ref{tab:scan-configurations} of Appendix~\ref{sec:tables}.
We also evaluate PtychoFormer's performance without any further retraining on new data. While our model is only trained on the Flickr30K dataset, we test the model on the Flower102 and Caltech101 datasets. We use probe A and grid scan patterns with 20-pixel lateral offset for the experiments.
PtychoFormer achieves 0.18 ± 0.06 (amplitude) and 0.51 ± 0.26 (phase), and 0.28 ± 0.09 (amplitude) and 0.97 ± 0.65 (phase) on the Flower102 and Caltech101 datasets, respectively. These results highlight the model's generalized understanding of phase retrieval, even when applied to untrained datasets.
\begin{figure}[t]
\centering
% \includesvg[width=1.0\linewidth]{Images/offset.svg}
\includegraphics[width=\linewidth]{svg-inkscape/offset_svg-tex.pdf}
\caption{Reconstruction results of PtychoFormer, ePIE, and extended-PtychoFormer (ePF) across various lateral offsets between diffraction patterns are shown in (a), (b), and (c) with normalized root mean squared error (NRMSE) for each reconstruction. PtychoFormer, while not recovering fine details, demonstrates greater tolerance to sparse scans. ePF refines the initial estimates from PtychoFormer, enabling it to recover the finer details and consistently outperform ePIE across all offsets.}
\label{fig:PtychoFormer_results}
\end{figure}
\begin{figure}[t]
\centering
% \includesvg[width=1.0\linewidth]{Images/graph.svg}
\includegraphics[width=\linewidth]{svg-inkscape/graph_svg-tex.pdf}
\caption{(a) and (b) plot the average mean absolute error (MAE) and NRMSE of the amplitude reconstructions, and (c) and (d) plot the average MAE and NRMSE of phase reconstructions for each algorithm across different pixel-wise lateral offsets. ePF consistently outperforms ePIE in all metrics and PtychoFormer showcases a robust tolerance to sparser scans than ePIE.}
\label{fig:PtychoFormer_results_graph}
\end{figure}
% \begin{figure}[t]
% \centering
% \includesvg[width=1.0\linewidth]{Images/new_quality.svg}
% \caption{Reconstruction results of PtychoFormer, ePIE, and ePF across various lateral offsets between diffraction patterns are shown in (a), (b), and (c) with normalized root mean squared error (NRMSE) for each reconstruction. (d) and (e) plot the average mean absolute error (MAE) and NRMSE of the amplitude reconstructions and (f) and (g) plot the average MAE and NRMSE of phase reconstructions for each algorithm across different pixel-wise lateral offsets. ePF consistently outperforms ePIE in all metrics and PtychoFormer showcases a robust tolerance to sparser scans than ePIE.}
% \label{fig:PtychoFormer_results}
% \end{figure}
\subsection{Improvements over ePIE}
While ePIE is the gold standard for ptychographic phase retrieval, it does have significant drawbacks.
The PtychoFormer and ePF are designed to alleviate such drawbacks while achieving comparable reconstruction performance.
We compare the performance of PtychoFormer and ePF against ePIE.
The ePIE algorithm follows the implementation from \cite{maiden2009} and iterates until its objective function, sum squared error (SSE), converges.
See Appendix~\ref{sec:appendix-ePIE} for implementation details of ePIE.
% of over 99.95\% per iteration, with PtychoFormer being approximately 2100 to 3600 times faster than ePIE.
% Our experimental results demonstrate that PtychoFormer offers significant advantages over ePIE, particularly in reconstruction speed, tolerance to sparser scan patterns, and immunity to global phase shifts.
% Firstly, the remarkable speed of DL phase retrieval algorithms, such as PtychoFormer, offers a significant advantage over traditional iterative methods.
% By providing a fast initial estimate, PtychoFormer significantly enhances the performance of ePIE in the combined ePF approach. This strong starting point allows ePF to mitigate the common issue of local minima in iterative phase retrieval, leading to faster convergence and improved reconstruction quality. To illustrate these benefits, we compare the performance of ePIE with and without the aid of PtychoFormer, focusing on improvements in phase and amplitude reconstruction, particularly under sparse scan patterns with reduced spatial overlap between neighboring diffraction patterns.
% Secondly, PtychoFormer exhibits greater tolerance to sparser scan patterns compared to ePIE.
% Figure~\ref{fig:PtychoFormer_results}(e) presents the average MAE of the amplitude and phase reconstructions for the three algorithms. It's important to note that the MAE metrics for PtychoFormer and ePIE reflect different aspects of reconstruction quality. PtychoFormer tends to smooth out areas with high-frequency details, resulting in higher MAE values due to the loss of fine information. In contrast, ePIE shows a lower NRMSE for phase reconstruction despite a higher MAE, suggesting that the elevated MAE is indicative of an average global phase shift rather than a loss of detail. This means that once the global phase shift is corrected, ePIE's reconstruction is quite accurate.
As mentioned before, a significant drawback of ePIE and similar iterative phase retrieval algorithms is the global phase shift phenomenon. As PtychoFormer is trained in absolute phase values during training, PtychoFormer mitigates global phase shifts.
As illustrated in
% Figure~\ref{fig:PtychoFormer_results}(f) and (g)
Figure~\ref{fig:PtychoFormer_results_graph}(c) and (d), ePIE has a high MAE while a comparably lower NRMSE. As the NRMSE is corrected with a global phase shift factor, this shows that ePIE has converged on a shifted phase value. In contrast, PtychoFormer does not suffer from a global phase shift as the MAE and NRMSE are similar. By iteratively improving the PtychoFormer's single-shot prediction, ePF surpasses ePIE on both metrics. Figure \ref{fig:ePIE_ePF} visualizes the significant reduction in global phase shifts when using ePF compared to ePIE alone, as seen in both the reconstructed phase images and the corresponding line profiles. However, ePF does not eliminate the global phase shift, as evident in the slight difference between the line profiles of the ground truth and ePF. We predict it can be improved with further improvements to the DL-based initialization.
Another critical drawback of ePIE is the computation cost to recover transmission functions.
In our experiments with 18×18 diffraction patterns, ePIE, with GPU support, took approximately 0.34 seconds per iteration and 800 to 1500 iterations to converge, resulting in 5 to 8.5 minutes to recover the sample's transmission. In contrast, PtychoFormer completed a one-shot prediction and stitching process in just 0.14 seconds.
This translates to a speed-up of 2100 to 3600 times faster than ePIE.
Furthermore, ePF reduces the iteration count required for ePIE to converge by approximately 100 iterations in our experiment, leveraging the well-informed estimate from PtychoFormer.
% \red{A line here about how ePF is also faster as fewer iterations are required when using ePF.}
% the well-informed estimate reduces the iteration count required for ePIE to converge by approximately 100 iterations in our experiment.
Finally, in Figure~\ref{fig:PtychoFormer_results}(a), we observe improved performance over ePIE on sparse scan patterns for both PtychoFormer and ePF.
Increasing the offset size reduces the spatial overlap and enables the scanning of a larger area with fewer diffraction patterns. This translates to fewer imaging samples being required and speeding up data collection.
The recommended overlap for ePIE is 60\% to 70\% spatial overlap \citep{maiden2009} with limited performance beyond.
Our methods preserved structural integrity in both the amplitude and phase reconstruction even with a 60-pixel offset (14.9\% spatial overlaps) between adjacent patterns.
In contrast, ePIE, depicted in
% Figure~\ref{fig:PtychoFormer_results}(b),
Figure~\ref{fig:PtychoFormer_results_graph}(a), exhibited significant artifacts at sparser scans.
With 60-pixel offset, PtychoFormer outperforms ePIE, as evidenced by its lower MAE and NRMSE values in Figure~\ref{fig:PtychoFormer_results_graph}(a--d). All results from various step sizes for ePIE, PtychoFormer, and ePF can be found in Appendix \ref{sec:tables} Table \ref{tab:all_step}.
% These results highlight ePIE's limitations under sparse scanning conditions when the recommended 60\% to 70\% spatial overlap \citep{maiden2009} is not maintained.
% presents the average MAE of the phase reconstructions for the three algorithms.
% While PtychoFormer and ePIE are evaluated using MAE, these metrics reflect different inaccuracies in the reconstruction.
% For ePIE, the low NRMSE indicates that the reconstruction is accurate once the global phase shift is removed.
% However, the higher MAE in ePIE is primarily due to the global phase shift itself. In contrast, PtychoFormer does not suffer from a global phase shift, and its higher MAE stems from pixel-level inaccuracies, particularly in high-frequency areas. Notably, PtychoFormer's phase magnitude more closely resembles the ground truth compared to ePIE, as it avoids the global phase shift, making it a more truthful reconstruction.
% The benefits of PtychoFormer—speed, tolerance to sparse scans, and immunity to global phase shifts—directly enhance the performance of ePF, enabling our hybrid approach to outperform ePIE in every metric. ePF achieves a 73.59\% decrease and a 47.30\% decrease in amplitude and phase NRMSE compared to ePIE, respectively. Additionally, the amplitude MAE decreased by 78.39\% and phase MAE decreased by 66.94\%. These percentages represent the average performance gains across 20 to 60 pixel lateral offsets between diffraction patterns.
% By leveraging the robust initial estimates from PtychoFormer, ePF also exhibited resilience to sparse scans, as shown in Figures~\ref{fig:PtychoFormer_results}(c--g).
% Although incorporating PtychoFormer's initial prediction introduces an additional step, its rapid inference adds negligible overhead.
% In fact, the well-informed estimate reduces the iteration count required for ePIE to converge by approximately 100 iterations in our experiment.
% Hence, ePF not only enhances the reconstruction quality but also improves convergence speed.
% \textbf{Reduced Global Phase Shift. }
% Iterative phase retrieval algorithms, like ePIE, can only determine the relative phase shift of the sample from the diffraction patterns. Likewise, the objective function (SSE) used in ePIE cannot distinguish between globally shifted phase values. A solution to mitigate the global phase shift is to provide the correct magnitude of the phase during initialization. Figure \ref{fig:ePIE_ePF} visualizes the significant reduction in global phase shifts when using ePF compared to ePIE alone, as seen in both the reconstructed phase images and the corresponding line profiles. While the ePIE algorithm introduces a uniform deviation in the phase values, ePF mitigates the shift, yielding a better pixel-wise phase accuracy than standalone ePIE. However, ePF does not eliminate the global phase shift, as evident in the slight difference between the line profiles of the ground truth and ePF. Our experiment suggests that if a DL model achieves even greater pixel-wise phase accuracy, it could further reduce or eliminate the global phase shift. Overall, PtychoFormer provides a reliable initial estimate that effectively reduces the global phase shift, resulting in more accurate phase reconstructions compared ePIE alone.
% The reduced global phase shift using ePF is evident in Figure~\ref{fig:ePIE_ePF}. Notably, ePF's phase estimate closely resembles the ground truth, whereas ePIE shows a prominent global phase shift. This indicates that ePF effectively minimizes the global phase shift, enhancing reconstruction accuracy. This improvement is quantified in the line profile shown in Figure~\ref{fig:ePIE_ePF}(e), where ePF's output closely follows the ground truth, demonstrating better pixel-wise accuracy compared to standalone ePIE. Further explanation and examples are presented in Appendix \ref{sec:shift}. Lastly, ePF reduces the number of iterations required for ePIE to converge by approximately 100 iterations, as shown in Figure~\ref{fig:PtychoFormer_results}(f). This reduction not only saves time but also enhances computational efficiency. By combining the rapid one-shot prediction of PtychoFormer with ePIE’s iterative refinement, ePF strikes an optimal balance between speed and reconstruction quality.
% \section{Discussion}
% \vspace{-.8em}
\section{Discussion}
% \vspace{-.2em}
We can see that PtychoFormer and ePF improve performance in both DL and iterative algorithms. By processing multiple diffraction patterns simultaneously, PtychoFormer captures their positional relationships, leading to higher-quality reconstructions than models that rely on single diffraction patterns. This spatial awareness stems from our input scheme, which preserves the spatial information of diffraction patterns, and the use of the MiT encoder to effectively capture the spatial relationships well. Our model also generalizes well across different probe functions and scan patterns with minimal fine-tuning and maintains its fidelity even with reduced spatial overlap. ePF uses PtychoFormer, enabling iterative refinement on the data-driven predictions. Importantly, ePF not only outperforms ePIE in terms of reconstruction accuracy but also significantly minimizes global phase shifts, a persistent challenge in phase retrieval.
% We introduced PtychoFormer, a hierarchical transformer-based architecture for ptychographic imaging that mitigates the absolute phase-shift problem in traditional phase retrieval algorithms and outperforms convolutional DL methods. By processing multiple diffraction patterns simultaneously, PtychoFormer captures their positional relationships, leading to higher-quality reconstructions than models that rely on single diffraction patterns.
% This spatial awareness stems from our input scheme, which preserves the spatial information of diffraction patterns, and the use of the MiT encoder to effectively capture the spatial relationships well.
% PtychoFormer offers impressive inference speed, strong tolerance to sparse scans, transfer learning capability, and reduces global phase shifts.
% It delivers swift one-shot predictions that reduce computational latency, vital for time-sensitive applications to image dynamic processes like live cell tracking or vivo imaging.
% Our model also generalizes well across different probe functions and scan patterns with minimal fine-tuning, and maintains its fidelity even with reduced spatial overlap. This adaptability suggests that PtychoFormer could be extended to other imaging modalities requiring phase retrieval, such as X-ray ptychography used to study molecular structures.
While we have strong performance on simulated data, transitioning from simulations to real-world applications poses challenges, particularly due to the potential for global phase shifts in the training data. The difficulty lies in the need to train the model on ground truth data generated by conventional algorithms, which may themselves introduce global phase shifts. To address this, calibrated or corrected phase measurements should be used to fine-tune PtychoFormer, allowing it to produce rapid and accurate estimates without global phase shifts. This correction of phase shifts can involve using objects with known phase shifts \citep{Godden16}, employing interferometric techniques \citep{Cai04}, or other phase calibration techniques to ensure reliable training data for real-world applications.
% These benefits of PtychoFormer extend to ePF. Importantly, ePF not only outperforms ePIE in terms of reconstruction accuracy but also significantly minimizes global phase shifts, a persistent challenge in phase retrieval. Our hybrid approach exemplifies how DL models can enhance and complement existing techniques to accelerate and improve iterative algorithms—a concept that could be extended to other diffractive imaging methods, such as coherent diffraction imaging (CDI), holography, and various forms of ptychography. While ePF offers a highly efficient solution for ptychographic imaging with both speed and accuracy, transitioning from simulations to real-world applications poses challenges, particularly due to the potential for global phase shifts in the training data. The difficulty lies in the need to train the model on ground truth data generated by conventional algorithms, which may themselves introduce global phase shifts. To address this, calibrated or corrected phase measurements should be used to fine-tune PtychoFormer, allowing it to produce rapid and accurate estimates without global phase shifts. This correction of phase shifts can involve using objects with known phase shifts \citep{Godden16}, employing interferometric techniques \citep{Cai04}, or other phase calibration techniques to ensure reliable training data for real-world applications.
\section{Conclusion}
PtychoFormer offers a new paradigm in DL phase retrieval for ptychography. It provides a robust and versatile solution capable of processing multiple diffraction patterns while preserving the knowledge of their relative scan points. PtychoFormer enables real-time imaging, which is crucial for applications requiring rapid imaging, but this speed comes at the expense of quality.
The model also generalizes well across different probe functions and scan patterns with minimal fine-tuning and maintains its fidelity even with reduced spatial overlap. This adaptability suggests that PtychoFormer could be extended to other imaging modalities requiring phase retrieval, such as X-ray ptychography used to study molecular structures.
Moreover, the integration with iterative methods, as seen in ePF, underscores the potential of hybrid approaches in balancing speed and reconstruction quality. ePF allows researchers to obtain a rough estimate of the transmission function in real-time using PtychoFormer, followed by a more accurate refinement using ePIE. While this research has made many significant strides in developing an improved DL phase retrieval algorithm for ptychography, many potential avenues exist to explore. Future work could involve validating PtychoFormer's performance with real ptychographic data and further refining the model to eliminate the need for post hoc corrections.
% \section*{Reproducibility Statement}
% We plan to publicly release the code for models, datasets, and experiments represented in the paper.
% previous conclusion (was too long)
% The current state of phase retrieval algorithms for ptychography predominantly relies on slow iterative algorithms for high-quality reconstruction or CNNs for instantaneous reconstruction with a sacrifice of quality. Most CNN algorithms for ptychography infer phase and amplitude measurements independently for each diffraction pattern, neglecting the fundamental ptychographic technique of leveraging redundant information from other diffraction patterns to achieve high-quality reconstruction.
% In this research, we developed a new deep-learning model for phase retrieval called PtychoFormer. The model is a transformer-based neural network that process a subset of diffraction patterns to leverage the spatial overlaps of the diffraction patterns. The adaptability and generalizability of PtychoFormer were highlighted through fine-tuning experi ments. The model quickly adapted to new scan patterns, datasets, and probe functions, proving its robustness and versatility for different experimental setups. This adaptability is crucial for real-world applications, where conditions can vary widely. Under various experimental conditions, we comprehensively evaluated PtychoFormer with state-of-the-art phase retrieval algorithms, namely ePIE, PtychoNN, and PtychoNet. As a result, PtychoFormer demonstrated an impressive reconstruction over other deep-learning methods with high fidelity over varying positional offsets of the diffraction patterns.
% Moreover, combining PtychoFormer with the ePIE algorithm further improved the reconstruction quality. The initial estimates provided by PtychoFormer allowed ePIE to refine the reconstructions more efficiently, demonstrating the potential of hybrid approaches in achieving both speed and high fidelity.
% The development of PtychoFormer is crucial in ptychography as it introduces a new approach and architecture for DL-based phase retrieval. The pre-trained model can be an efficient tool for real-world applications to achieve high-quality real-time ptychographic imaging.
% \subsection{Future Works}
% While this research has made many significant strides in developing a robust DL-based phase retrieval algorithm for ptychography, many potential avenues exist to explore.
% 1. The research can be strengthened by evaluating the amount of real data required to fine-tune the model to achieve superb performance. Evaluating the performance of PtychoFormer on real ptychography data is vital to validate the practicality of this pre-trained model in real applications. (Can be removed IF we obtain real ptychographic data)
% 2. Experimentation on larger versions of PtychoFormer, along with more data, scan configurations, and probe functions, may increase the capability, to achieve a level of generalizability without any need of fine-tuning.
% \subsection{Acknowledgment}
% \bibliography{references}
% \bibliographystyle{iclr2025_conference}
\input{main.bbl}
\newpage
\appendix
\section{Implementation Details of PtychoFormer}
\label{sec:pf_implementation}
The PtychoFormer architecture leverages a Mix Transformer (MiT) encoder with a convolutional decoder to achieve efficient phase retrieval. Utilizing the MiT-B0 encoder, from the SegFormer architecture \citep{xie2021}, ensures both computational efficiency and high performance. Key features such as the spatial reduction attention and elimination of fixed positional embedding within the MiT contribute to the efficiency. PtychoFormer processes an input $X\in \mathbb{R}^{9 \times H \times W}$, and outputs $\hat{Y}\in \mathbb{R}^{2 \times H \times W}$, where $H$ and $W$ are the height and width of the input image.
The encoder consists of four stages, each processing the input at different spatial resolutions. These stages incorporate multiple layers of MiT encoder blocks, with 4, 6, 8, and 12 self-attention heads, respectively. In the first stage, the input is processed to $\frac{H}{2}\times\frac{W}{2}$ with $64$ feature channels, by performing patch embedding followed by MiT Block 1. The successive MiT stages progressively decrease the resolution by a factor of 2 while increasing the feature channels to 128, 256, and 512, respectively.
The decoder upsamples the outputs from each stage of the encoder using bi-cubic interpolation to $\frac{H}{2}\times\frac{W}{2}$ and passes them through the convolution layer to augment the feature channels to $256$. These outputs are concatenated to form a $1024\times\frac{H}{2}\times\frac{W}{2}$ matrix. Then a convolutional block, followed by a transposed convolutional block, reduces the feature channels and upsamples the resolution to $128\times H\times W$. Each block is followed by batch normalization and a GELU activation function. The final convolutional layer reduces the feature channels to a $ 2\times H \times W$ matrix. The tanh function is applied to the first channel, then multiplied by $\pi$, resulting in a range of [-$\pi$, $\pi$]. The second channel uses the sigmoid function to yield a range of [0,1]. The two output channels correspond to the phase and amplitude estimates, respectively.
\section{Label Detail}
\label{sec:label_detail}
\begin{figure}[H]
\centering
% \includesvg[width=0.65\linewidth]{Images/label.svg}
\includegraphics[width=0.65\linewidth]{svg-inkscape/label_svg-tex.pdf}
\caption{The top row shows the label associated with a single diffraction pattern, where the amplitude and phase are masked to match the illuminated region. The second row displays a 3x3 grid of diffraction patterns in nine separate channels to preserve the spatial dependencies, and the corresponding labels are masked to match the illuminated region as well.}
\label{fig:diff_set}
\end{figure}
To ensure the model accurately predicts only the information encoded within the diffraction patterns, proper label creation is crucial. Specifically, we mask out the regions in the labels that correspond to areas outside the effective scan point and the radius of the illumination area, preventing the model from learning parts of the transmission function not represented in the diffraction data.
In the top row of Figure \ref{fig:diff_set}, you can see an example of a label created for a single diffraction pattern. The label reflects the amplitude and phase values only within the illuminated region, with areas outside of the illumination masked out to exclude irrelevant information. The second row of Figure \ref{fig:diff_set} demonstrates a 3x3 grid of diffraction patterns, each associated with its respective label. These diffraction patterns are separated into channels, with zero-padding applied to areas outside the diffraction data to maintain consistent input dimensions. This setup enforces the proper learning of the transmission function within the bounds of what the diffraction data encodes.
\section{Phase Normalization During Training}
\label{sec:Phase_Norm}
\begin{figure}[H]
\centering
% \includesvg[width=\textwidth]{Images/Training_Results.svg}
\includegraphics[width=\linewidth]{svg-inkscape/Training_Results_svg-tex.pdf}
\caption{Comparison of average mean absolute error of the test set across training steps for normalized and unnormalized phases during training, based on 20 trials each. The shaded areas represent the standard error. Normalized training exhibits improved generalization and stability compared to unnormalized training.}
\label{fig:norm_training}
\end{figure}
During training, we employed a phase normalization technique that scales the phase predictions and labels from their original range of [-$\pi$, $\pi$] to [0,1]. This normalization ensures that the loss values for both amplitude and phase components are within the same range, resulting in similar gradient magnitudes during optimization. Without normalization, computing the mean absolute error (MAE) between images with ranges of [0,1] (amplitude) and [-$\pi$, $\pi$] (phase) would lead to disproportionately larger errors from the phase component due to its wider range. This imbalance can cause the optimization process to focus excessively on minimizing the phase loss, potentially leading to instability during training.
To assess the effectiveness of our normalization approach, we conducted 20 training trials with both unnormalized and normalized phases. Despite normalizing the phase during training, we evaluated the models on the test set using the original unnormalized phase values to measure true performance. We measured the raw MAE scores between the amplitude and phase predictions and their ground truths. The results, shown in Figure~\ref{fig:norm_training}(a) and (b), indicate that models trained with phase normalization consistently achieve lower MAE on the test set. This demonstrates that phase normalization not only stabilizes the training process but also leads to improved generalization and reconstruction accuracy.
\section{ePIE Implementation}
\label{sec:appendix-ePIE}
The ePIE algorithm refines the estimate of the transmission function, $\hat{T}(x,y)$, through an iterative process. This process aims to minimize the sum of the squared differences between the diffraction patterns produced from the estimated transmission function, $\hat{I}_j(u,v)$, and the measured diffraction pattern, $I_j(u,v)$, at each scanning position $j$.
We suppose the estimated probe measurement, $P(x,y)$, is known and the diffraction s are captured before running ePIE. In our experiment, we set the initial amplitude and phase values of $\hat{T}(x,y)$ to be ones and zeros, respectively.
\begin{enumerate}
\item Compute the estimated exit wave: At iteration $k$ for the $j$th scan position, the exit wave is computed as
\begin{equation}
\hat{\psi}_{k,j}(x,y) = \hat{T}_{k}(x,y) \times P_k(x-X_j, y-Y_j),
\label{eq: propagate}
\end{equation}
where $(X_j, Y_j)$ encodes the scanning shifts for the $j$th diffraction pattern. The finite support constraint at each position, $S_j$, is represented as
\begin{equation}
\hat{T}_{k}(x,y) =
\begin{cases}
\hat{T}_{k}(x,y) & \text{if } (x,y) \in S_j \\
0 & \text{if } (x,y) \notin S_j
\end{cases}.
\end{equation}
\item Update the exit wave: Replace the amplitude of the estimated exit wave with the measured amplitude. First, apply a Fourier transform $F$ to the exit wave:
\begin{equation}
\hat{\Psi}_{k,j}(u,v) = F\left(\hat{\psi}_{k,j}(x,y)\right).
\end{equation}
Update the exit wave using the measured amplitude:
\begin{equation}
\Psi^{\text{upd}}_{k,j}(u,v) = \frac{\hat{\Psi}_{k,j}(u,v)}{|\hat{\Psi}_{k,j}(u,v)|} \times \sqrt{I_j(u,v)}.
\end{equation}
Finally, revert to the spatial domain using the inverse Fourier transform:
\begin{equation}
\psi^{\text{upd}}_{k,j}(x,y) = F^{-1}(\Psi^{\text{upd}}_{k,j}(u,v)).
\end{equation}
\item Update the transmission function estimate using the following equation,
\begin{equation}
\hat{T}^{\text{upd}}_{k}(x,y) = \hat{T}_{k}(x,y) + \alpha \frac{\Bar{P}_k(x-X_j, y-Y_j)}{|P_k(x-X_j, y-Y_j)|_{\text{max}}^2} \left(\psi^{\text{upd}}_{k,j}(x,y) - \hat{\psi}_j(x,y)\right),
\end{equation}
where $\Bar{P}(x,y)$ is the conjugate of $P(x,y)$ and $\alpha$ is the step size that controls how fast the transmission function is updated. This equation adjusts $\hat{T}^{\text{upd}}_{k}(x,y)$ based on the difference between the updated exit wave $\psi^{\text{upd}}_{k,j}(x,y)$ and the previous estimate $\hat{\psi}_j(x,y)$. The expression $\frac{\Bar{P}_k(x-X_j, y-Y_j)}{|P_k(x-X_j, y-Y_j)|_{\text{max}}^2}$ removes the probe function from the exit wave and is used to weight the update, focusing on areas with stronger illumination. Lastly, we update the transmission function in the area covered by the finite support constraint at position j,
\begin{equation}
\hat{T}_{k+1}(x) =
\begin{cases}
T^{\text{upd}}_{k}(x) & \text{if } (x,y) \in S_j \\
\hat{T}_{k}(x) & \text{if } (x,y) \notin S_j
\end{cases}.
\end{equation}
\item Update the probe function by removing the transmission function from the exit wave, similar to the previous step using the following equation,
\begin{equation}
P_{k+1}(x,y) = P_{k}(x,y) + \beta \frac{\Bar{T}_k(x+X_j, y+Y_j)}{|\hat{T}_k(x+X_j, y+Y_j)|_{\text{max}}^2} \left(\psi^{\text{upd}}_{k,j}(x,y) - \hat{\psi}_j(x,y)\right),
\end{equation}
where $\Bar{T}_k(x,y)$ is the conjugate of $\hat{T}_k(x,y)$ and $\beta$ is the step size for updating the probe function.
\item After iterating through every scan point $j$, increment $k$ and repeat steps $1-4$.
\item Repeat the iterative process until the sum squared error function (SSE) converges to a sufficiently small value. This cost function at the $k$th iteration is measured as
\begin{equation}
L[\hat{T}_{k}(x)] = \sum_j \left( I_j(u,v) - \hat{I}_j(u,v)\right)^2,
\end{equation}
such that
\begin{equation}
\hat{I}_j(u,v) = |\hat{\Psi}_{k,j}(u,v)|^2.
\end{equation}
\end{enumerate}
After convergence, the phase of $\hat{T}(x,y)$ is confined between $-\pi$ and $\pi$, resulting in phase wrapping where values exceeding this range. To retrieve the continuous phase, we apply a phase unwrapping algorithm, as introduced by \citet{Herraez02}, which is implemented in the scikit-image library. This post-processing step ensures that the reconstructed phase is free of discontinuities caused by wrapping.
\section{Dataset Details}
\label{appendix:data_details}
\begin{table}[H]
\centering
\begin{tabular}{ccccc}
\toprule
\textbf{Dataset} & \textbf{Source} & \textbf{Probe Functions} & \textbf{Scan Configurations}& \textbf{Lateral Offset}\\
\midrule
Pre-training& Flickr30k & 1 & Grid & 20 - 55\\
\midrule
Fine-tuning & Flickr30k & \makecell{2, 3, 4, 5, 6, \\ 7, 8, 9, 10, 11}& Grid & 20 \\
& Flickr30k & 1 & \makecell{5 - 8 \\Diffraction Patterns, \\
Diamond, Random, \\Grid, Parallelogram}& --\\
% & & 2 & & 20 \\
% & & 2 & & 20 \\
% & & 2 & & 20 \\
\midrule
Test & Flickr30k & 1 & Grid & \makecell{20, 30, 40, \\50, 60} \\
& Flickr30k & \makecell{2, 3, 4, 5, 6, \\ 7, 8, 9, 10, 11, \\ Test Probe} & Grid & 20 \\
& Flickr30k & 1 & \makecell{5 - 8 Diffraction Patterns, \\
Diamond, Random, \\ Parallelogram}& --\\
& Flower102 & 1 & Grid & 20\\
& Caltech101 & 1 & Grid & 20\\
\bottomrule
\end{tabular}
\caption{Details of the pre-training, finetuning, and test datasets are showcased in this table. For each dataset, we present the source of the raw images and the probe functions, scan configurations, and pixel-wise lateral offsets the dataset contains.}
\label{tab:dataset}
\end{table}
\begin{figure}[H]
\centering
% \includesvg[width=\linewidth]{Images/probe.svg}
\includegraphics[width=\linewidth]{svg-inkscape/probe_svg-tex.pdf}
\caption{Amplitude and phase profiles for 12 probe functions, including the Test Probe. Probes 1 and 2 are referred to as Probe A and Probe B in the main body of the paper, while the Test Probe is referred to as Probe C. Each probe demonstrates variations in amplitude and phase, which are used to train the model to be resilient to different probe functions.}
\label{fig:probe_visual}
\end{figure}
This appendix and Table \ref{tab:dataset} provide detailed information about the pre-training, fine-tuning, and test datasets. As a preliminary, probes A, B, and C, from the main body of text, are relabeled as we present all 12 probe functions. As visualized in Figure \ref{fig:probe_visual}, the 11 training probes are numerically labeled from 1 - 11, such that probes 1 and 2 correspond to probes A and B, and probe C is labeled as the test probe.
The Flickr30k \citep{young2014} is the primary dataset we used in our experiment, where we allocated 28,600 images for training and 3,100 images for testing. The pre-training dataset consists of all training images to generate diffraction patterns using a grid scan pattern with 36 different lateral offsets between adjacent scan points, ranging from 20 to 55 pixels. These offsets correspond to spatial overlaps between diffraction patterns ranging from 68.7\% to 20.3\%. The pre-training dataset consists of these 36 configurations, each containing 6,800 triplets of nine simulated diffraction patterns along with the corresponding amplitude and phase labels, totaling 244,800 data points. We only used probe 1 in this dataset.
The finetuning dataset uses a subset of the training images to create a smaller dataset that introduces 7 additional scan patterns and 9 new probe functions (i.e. probe 2 - 11). The scan patterns included various configurations using probe 1, from 5 to 8 diffraction patterns and grid, diamond, parallelogram, and random layouts using 9 diffraction patterns. Probes 2 - 11 were used in a grid scan with a constant lateral offset of 20 pixels. The finetuning dataset included 2,000 samples per configuration, totaling 36,000 data points.
We created five test datasets, out of which three were created using the allocated test images from Flickr30k, to evaluate the model's performance under various scenarios, including out-of-distribution conditions. The first test dataset consists of grid scan with lateral offsets of 20, 30, 40, 50, and 60 pixels between adjacent scan points. These offsets correspond to spatial overlaps of 68.7\%, 53.6\%, 39.4\%, 26.3\%, and 14.9\%, respectively. We created 100 samples for each of these grid scans. The second dataset contains grid scans of 20 pixel offset using probes 2 - 11, and the test probe, each with 200 samples. The third dataset includes configurations using the seven new scan patterns introduced in the fine-tuning dataset. To assess the model's ability to generalize to different domains, we further evaluated it on two new datasets: Flower102 \citep{nilsback2008} and Caltech101 \citep{li2022}. We created 8,000 samples from each dataset.
\section{Simulation Results}
\label{sec:tables}
This section presents the empirical results from our simulations, providing the exact mean and standard deviation values for various probe functions, scan configurations, and comparisons of different phase retrieval methods. Tables \ref{tab:DL_comparison} and \ref{tab:all_step} are included here in the appendix, as the corresponding results are visualized as charts in the main body of the paper.
\begin{table}
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Probe Function} & \textbf{Amplitude NRMSE} & \textbf{Phase NRMSE} \\
\midrule
Probe 1 (A) & 0.24 ± 0.10 & 0.79 ± 0.52 \\
Probe 2 (B) & 0.28 ± 0.10 & 0.95 ± 0.74 \\
Probe 3 & 0.23 ± 0.11 & 0.81 ± 0.58 \\
Probe 4 & 0.25 ± 0.11 & 0.76 ± 0.61 \\
Probe 5 & 0.24 ± 0.11 & 0.76 ± 0.74 \\
Probe 6 & 0.25 ± 0.10 & 0.87 ± 0.81 \\
Probe 7 & 0.25 ± 0.11 & 0.83 ± 0.68 \\
Probe 8 & 0.27 ± 0.09 & 0.83 ± 0.62 \\
Probe 9 & 0.27 ± 0.10 & 1.00 ± 0.82 \\
Probe 10 & 0.29 ± 0.11 & 1.11 ± 1.22 \\
Probe 11 & 0.28 ± 0.11 & 1.02 ± 0.84 \\
Test Probe (C) & 0.22 ± 0.10 & 0.83 ± 0.77 \\
\bottomrule
\end{tabular}
\caption{Normalized root mean squared error (NRMSE) values for 11 probe functions used during training and the test probe used exclusively for testing. The table presents the NRMSE for both amplitude and phase reconstructions, demonstrating the model’s ability to generalize across various probe functions, including an unseen probe function.}
\label{tab:probe_values}
\end{table}
\begin{table}
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Scan Configuration} & \textbf{Amplitude NRMSE} & \textbf{Phase NRMSE} \\
\midrule
Five Diffraction Patterns & 0.27 ± 0.11 & 0.71 ± 0.37 \\
Six Diffraction Patterns & 0.27 ± 0.09 & 0.69 ± 0.36 \\
Seven Diffraction Patterns & 0.26 ± 0.10 & 0.68 ± 0.44 \\
Eight Diffraction Patterns & 0.27 ± 0.10 & 0.77 ± 0.46 \\
Diamond Formation & 0.26 ± 0.09 & 0.67 ± 0.46 \\
Parallelogram Formation & 0.29 ± 0.12 & 0.71 ± 0.45 \\
Grid Formation & 0.20 ± 0.09 & 0.66 ± 0.32 \\
Random Formation & 0.26 ± 0.08 & 0.72 ± 0.43 \\
\bottomrule
\end{tabular}
\caption{Amplitude and phase normalized root mean squared error (NRMSE) values for various scan configurations using probe A are presented. Values represent the average NRMSE of 100 test data by comparing the reconstructed central squares to the ground truth. PtychoFormer shows similar performance across different scan configurations even with minimal training. }
\label{tab:scan-configurations}
\end{table}
We evaluated 12 distinct probe functions corresponding to different illumination patterns during diffraction, detailed in Table \ref{tab:probe_values}. Additionally, Table \ref{tab:scan-configurations} presents the results from eight different scan configurations, demonstrating how the geometry of diffraction scans influences reconstruction accuracy. Notably, the grid formation yielded the lowest errors for both amplitude and phase reconstructions. This is likely because the model was primarily trained on grid configurations and only later fine-tuned on other scan setups, giving it an advantage when reconstructing from grid data compared to more irregular formations, such as random configurations.
\begin{table}
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{Amplitude} & \multicolumn{2}{c}{Phase} \\ \cmidrule(lr){2-3} \cmidrule(lr){4-5}
& MAE & NRMSE & MAE & NRMSE \\ \midrule
PtychoFormer & \textbf{\textcolor{blue}{0.08 ± 0.03}}& \textbf{\textcolor{blue}{ 0.20 ± 0.08}}& \textbf{\textcolor{red}{0.76 ± 0.36}}& \textbf{\textcolor{red}{0.67 ± 0.44}}\\
PtychoNet & 0.13 ± 0.04& 0.34 ± 0.12& 1.27 ± 0.38 & 1.50 ± 0.72\\
PtychoNN & 0.10 ± 0.04& 0.27 ± 0.09& 1.31 ± 0.38& 1.72 ± 0.7\\ \bottomrule
\end{tabular}
\end{adjustbox}
\caption{Mean absolute error (MAE) and normalized root mean squared error (NRMSE) for amplitude and phase reconstructions of three deep learning models showcases PtychoFormer's exceptional performance. Results are averaged from 3,100 test images, each comprising 6 × 6 diffraction patterns with 20-pixel lateral offsets.}
\label{tab:DL_comparison}
\end{table}
\begin{table}
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{Amplitude} & \multicolumn{2}{c}{Phase} \\ \cmidrule(lr){2-3} \cmidrule(lr){4-5}
& MAE & NRMSE & MAE & NRMSE \\ \midrule
PtychoFormer & \textbf{\textcolor{blue}{0.10 ± 0.03}}& \textbf{\textcolor{blue}{0.26 ± 0.06}}& \textbf{\textcolor{red}{0.82 ± 0.23}}& \textbf{\textcolor{red}{0.76 ± 0.33}}\\
PtychoNet & 0.17 ± 0.03& 0.53 ± 0.07& 1.35 ± 0.21& 1.45 ± 0.32\\
PtychoNN & 0.15 ± 0.03& 0.47 ± 0.05& 1.35 ± 0.21 & 1.71 ± 0.39 \\ \bottomrule
\end{tabular}
\end{adjustbox}
\caption{Mean absolute error (MAE) and normalized root mean squared error (NRMSE) for amplitude and phase reconstructions are taken by averaging the performance across 3,100 test images, each comprising 6 × 6 diffraction patterns with 60-pixel lateral offsets. Even on unseen scan, PtychoFormer outperforms PtychoNN and PtychoNet.}
\label{tab:DL_comparison_unseen}
\end{table}
\begin{table}
\centering
\small % Reduce font size
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{>{\raggedright\arraybackslash}m{2cm} >{\centering\arraybackslash}m{1.5cm} >{\centering\arraybackslash}p{2.5cm} >{\centering\arraybackslash}p{2.5cm} >{\centering\arraybackslash}p{2.5cm} >{\centering\arraybackslash}p{2.5cm}}
\toprule
\multirow{2}{2cm}{\raggedright Method} & \multirow{2}{1.5cm}{\centering Lateral Offset} & \multicolumn{2}{c}{Amplitude} & \multicolumn{2}{c}{Phase} \\ \cmidrule(r){3-4} \cmidrule(l){5-6}
& & MAE & NRMSE & MAE & NRMSE \\ \midrule
PtychoFormer & 20 & 0.08 ± 0.04 & 0.20 ± 0.09 & 0.75 ± 0.33 & 0.66 ± 0.32 \\
ePF & 20 & \textbf{\textcolor{blue}{5.89e-9 ± 3.72e-9}} & \textbf{\textcolor{blue}{1.51e-8 ± 2.33e-8}} & \textbf{\textcolor{red}{0.44 ± 0.64}} & \textbf{\textcolor{red}{0.12 ± 0.19}} \\
ePIE & 20 & 8.74e-5 ± 5.62e-4 & 7.66e-4 ± 4.50e-3 & 1.15 ± 0.93 & 0.17 ± 0.30 \\ \midrule
PtychoFormer & 30 & 0.08 ± 0.03 & 0.21 ± 0.08 & 0.70 ± 0.29 & 0.66 ± 0.41 \\
ePF & 30 & \textbf{\textcolor{blue}{1.44e-4 ± 4.22e-4}} & \textbf{\textcolor{blue}{1.81e-3 ± 5.23e-3}} & \textbf{\textcolor{red}{0.33 ± 0.46}} & \textbf{\textcolor{red}{0.14 ± 0.20}} \\
ePIE & 30 & 2.52e-3 ± 3.46e-3 & 0.02 ± 0.03 & 1.15 ± 0.90 & 0.23 ± 0.32 \\ \midrule
PtychoFormer & 40 & 0.09 ± 0.03 & 0.22 ± 0.07 & 0.72 ± 0.26 & 0.70 ± 0.36 \\
ePF & 40 & \textbf{\textcolor{blue}{2.72e-3 ± 2.77e-3}} & \textbf{\textcolor{blue}{0.02 ± 0.02}} & \textbf{\textcolor{red}{0.30 ± 0.41}} & \textbf{\textcolor{red}{0.16 ± 0.19}} \\
ePIE & 40 & 0.01 ± 0.01 & 0.07 ± 0.04 & 1.13 ± 0.99 & 0.34 ± 0.35 \\ \midrule
PtychoFormer & 50 & 0.09 ± 0.03 & 0.23 ± 0.07 & 0.73 ± 0.25 & 0.69 ± 0.30 \\
ePF & 50 & \textbf{\textcolor{blue}{0.02 ± 0.01}} & \textbf{\textcolor{blue}{0.07 ± 0.04}} & \textbf{\textcolor{red}{0.35 ± 0.68}} & \textbf{\textcolor{red}{0.26 ± 0.23}} \\
ePIE & 50 & 0.07 ± 0.04 & 0.21 ± 0.10 & 1.36 ± 1.34 & 0.65 ± 0.32 \\ \midrule
PtychoFormer & 60 & 0.10 ± 0.03 & 0.26 ± 0.07 & \textbf{\textcolor{red}{0.78 ± 0.21}} & 0.75 ± 0.31 \\
ePF & 60 & \textbf{\textcolor{blue}{0.09 ± 0.04}} & \textbf{\textcolor{blue}{0.24 ± 0.10}} & 0.83 ± 0.96 & \textbf{\textcolor{red}{0.55 ± 0.26}} \\
ePIE & 60 & 0.17 ± 0.04 & 0.41 ± 0.08 & 1.83 ± 1.02 & 1.10 ± 0.21 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Comparison of reconstruction performance for PtychoFormer, ePF, and ePIE across different lateral offsets highlights the remark performance of ePF. The table reports mean absolute error (MAE) and normalized root mean squared error (NRMSE) for amplitude and phase reconstructions. Results are averaged from 100 test datasets, each consisting of 6 × 6 diffraction patterns.}
\label{tab:all_step}
\end{table}
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{Images/frc_graph.png}
\caption{Average Fourier Ring Correlation (FRC) of 100 test samples under 20-pixel lateral offsets to illustrate the correlation between the reconstruction and the ground truth across different spatial frequencies. Shaded areas of the graph represent the standard deviation of FRC values. PtychoFormer exhibits a high correlation in the low and mid spatial frequencies while showing a dip in high spatial frequency. However, ePF showcases a near perfect amplitude correlation and a subtle improvement in phase correlation than ePIE.}
\label{fig:FRC}
\end{figure}
Table \ref{tab:DL_comparison} compares the performance of PtychoFormer with two other deep learning-based methods, PtychoNet and PtychoNN, under 20-pixel offsets between each diffraction pattern. PtychoFormer consistently demonstrates superior performance in both amplitude and phase reconstructions, achieving lower MAE and NRMSE values. Moreover, this trend continues across larger offsets, even on an unseen lateral offset (i.e. 60 pixels), shown in Table \ref{tab:DL_comparison_unseen}. In Table \ref{tab:all_step}, we compare the performance of PtychoFormer, extended-PtychoFormer (ePF), and ePIE across different lateral offsets between diffraction patterns. As discussed in the main body, PtychoFormer exhibits persistent performance across different offsets, while ePIE showcases severe degradation. The degradation is present for ePF, but consistently outperforms ePIE.
Fourier Ring Correlation (FRC) analysis \citep{koho2019}, shown in Figure~\ref{fig:FRC}, reveals that PtychoFormer effectively recovers low to midrange spatial frequencies, but struggles with high-frequency details. The 1/7 threshold is an empirically validated criterion for distinguishing between meaningful signal and noise. The average FRC curves for both amplitude and phase drop below the threshold at normalized spatial frequencies of 0.71 to 0.95, highlighting its limitations in capturing fine details. However, ePF showcases better FRC values than ePIE. Notably, ePF exhibits near-perfect mean FRC of the amplitude reconstruction and slight improvement in the phase.
% \section{Phase Shift}
% \label{sec:shift}
% \begin{figure}
% \centering
% \includesvg[width=\linewidth]{Images/shift.svg}
% \caption{Comparison of reconstructions between ePF and ePIE against the Ground Truth for four sample images. The first three columns show the amplitude reconstructions for Ground Truth, ePF, and ePIE, respectively. The rightmost column presents line profile plots for each sample, comparing the pixel-wise intensity differences across the predictions from ePF and ePIE relative to the Ground Truth.}
% \label{fig:shift}
% \end{figure}
% During our simulation, we observed a prominent global phase shift in the phase reconstruction from the extended ptychographic iterative engine (ePIE). This is a well-known issue of phase retrieval algorithms that emerges from the phase problem outlined in the paper. In short, phase retrieval algorithms can only determine the relative phase shift of the sample from the measured diffraction patterns. A solution to mitigate the global phase shift is to provide the correct magnitude of the phase.
% From our experiment, deep learning phase retrieval algorithms are more tolerant to global phase shifts. Hence, by providing its reconstructed transmission function as an initial estimate for ePIE, we minimized the global phase shift problem. Figure \ref{fig:shift} visualizes the significant reduction in global phase shifts when using ePF compared to ePIE alone, as seen in both the reconstructed phase images and the corresponding line profiles. While the ePIE algorithm introduces a uniform offset in the phase reconstructions, ePF mitigates this global phase shift, yielding results closer to the ground truth. This improvement is further highlighted in the line profiles, where ePF closely tracks the ground truth, whereas ePIE shows a consistent deviation. However, note that our method does not eliminate the global phase shift, as evident in the slight difference between the line profiles of the ground truth and ePF. Overall, PtychoFormer provides a reliable initial estimate that effectively reduces the global phase shift, resulting in more accurate phase reconstructions compared ePIE alone.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\newcommand{\blue}{\textcolor{blue}}
\newcommand{\red}{\textcolor{red}}
\begin{thebibliography}{28}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Babu et~al.(2023)Babu, Zhou, Kandel, et~al.]{babu2023}
A.~V. Babu, T.~Zhou, S.~Kandel, et~al.
\newblock Deep learning at the edge enables real-time streaming ptychographic imaging.
\newblock \emph{Nature Communications}, 14:\penalty0 7059, 2023.
\newblock \doi{10.1038/s41467-023-41496-z}.
\newblock URL \url{https://doi.org/10.1038/s41467-023-41496-z}.
\bibitem[Cai et~al.(2004)Cai, Liu, and Yang]{Cai04}
L.~Z. Cai, Q.~Liu, and X.~L. Yang.
\newblock Generalized phase-shifting interferometry with arbitrary unknown phase steps for diffraction objects.
\newblock \emph{Opt. Lett.}, 29\penalty0 (2):\penalty0 183--185, Jan 2004.
\newblock \doi{10.1364/OL.29.000183}.
\newblock URL \url{https://opg.optica.org/ol/abstract.cfm?URI=ol-29-2-183}.
\bibitem[Cherukara et~al.(2020)]{cherukara2020}
M.~J. Cherukara et~al.
\newblock Real-time sparse-sampled ptychographic imaging through deep neural networks, 2020.
\newblock URL \url{https://arxiv.org/abs/2004.08247v1}.
\newblock arXiv preprint arXiv:2004.08247v1.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]{dosovitskiy2021an}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\newblock URL \url{https://openreview.net/forum?id=YicbFdNTTy}.
\bibitem[Dou et~al.(2020)Dou, Wu, Zhang, Hu, and Yang]{Dou20}
Jiantai Dou, Junchao Wu, Yamei Zhang, Youyou Hu, and Zhongming Yang.
\newblock Accelerated convergence extended ptychographical iterative engine using multiple axial intensity constraints.
\newblock \emph{Opt. Express}, 28\penalty0 (3):\penalty0 3587--3597, Feb 2020.
\newblock \doi{10.1364/OE.386152}.
\newblock URL \url{https://opg.optica.org/oe/abstract.cfm?URI=oe-28-3-3587}.
\bibitem[Gan et~al.(2024)Gan, Zhai, McCann, Cardona, Kamilov, and Wohlberg]{gan2024}
Weijie Gan, Qiuchen Zhai, {Michael Thompson} McCann, {Cristina Garcia} Cardona, {Ulugbek S.} Kamilov, and Brendt Wohlberg.
\newblock Ptychodv: Vision transformer-based deep unrolling network for ptychographic image reconstruction.
\newblock \emph{IEEE Open Journal of Signal Processing}, 5:\penalty0 539--547, 2024.
\newblock \doi{10.1109/OJSP.2024.3375276}.
\bibitem[Godden et~al.(2016)Godden, {n}iz Piniella, Claverley, Yacoot, and Humphry]{Godden16}
T.~M. Godden, A.~Mu\ {n}iz Piniella, J.~D. Claverley, A.~Yacoot, and M.~J. Humphry.
\newblock Phase calibration target for quantitative phase imaging with ptychography.
\newblock \emph{Opt. Express}, 24\penalty0 (7):\penalty0 7679--7692, Apr 2016.
\newblock \doi{10.1364/OE.24.007679}.
\newblock URL \url{https://opg.optica.org/oe/abstract.cfm?URI=oe-24-7-7679}.
\bibitem[Guan \& Tsai(2019)Guan and Tsai]{guan2019}
Z.~Guan and E.~H. Tsai.
\newblock Ptychonet: Fast and high quality phase retrieval for ptychography (bnl-213637-2020-fore).
\newblock Technical report, Center for Functional Nanomaterials, Brookhaven National Laboratory, 2019.
\bibitem[Guizar-Sicairos et~al.(2011)Guizar-Sicairos, Diaz, Holler, Lucas, Menzel, Wepf, and Bunk]{GuizarSicairos2011}
Manuel Guizar-Sicairos, Ana Diaz, Mirko Holler, Miriam~S. Lucas, Andreas Menzel, Roger~A. Wepf, and Oliver Bunk.
\newblock Phase tomography from x-ray coherent diffractive imaging projections.
\newblock \emph{Opt. Express}, 19\penalty0 (22):\penalty0 21345--21357, Oct 2011.
\newblock \doi{10.1364/OE.19.021345}.
\newblock URL \url{https://opg.optica.org/oe/abstract.cfm?URI=oe-19-22-21345}.
\bibitem[Herr\'{a}ez et~al.(2002)Herr\'{a}ez, Burton, Lalor, and Gdeisat]{Herraez02}
Miguel~Arevallilo Herr\'{a}ez, David~R. Burton, Michael~J. Lalor, and Munther~A. Gdeisat.
\newblock Fast two-dimensional phase-unwrapping algorithm based on sorting by reliability following a noncontinuous path.
\newblock \emph{Appl. Opt.}, 41\penalty0 (35):\penalty0 7437--7444, Dec 2002.
\newblock \doi{10.1364/AO.41.007437}.
\newblock URL \url{https://opg.optica.org/ao/abstract.cfm?URI=ao-41-35-7437}.
\bibitem[Islam et~al.(2020)Islam, Jia, and Bruce]{Islam}
Md~Amirul Islam, Sen Jia, and Neil D.~B. Bruce.
\newblock How much position information do convolutional neural networks encode?
\newblock In \emph{International Conference on Learning Representations}, 2020.
\newblock URL \url{https://openreview.net/forum?id=rJeB36NKvB}.
\bibitem[Jia et~al.(2022)Jia, Chen, Yang, He, Dai, and Liu]{Jia}
Jiaru Jia, Xin Chen, Aiqing Yang, Qiulin He, Pengyu Dai, and Mingzhe Liu.
\newblock Link of transformers in cv and nlp: A brief survey.
\newblock In \emph{2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI)}, pp.\ 735--743, 2022.
\newblock \doi{10.1109/PRAI55851.2022.9904151}.
\bibitem[Jiang et~al.(2018)Jiang, Chen, Han, Deb, Gao, Xie, Purohit, Tate, Park, Gruner, et~al.]{jiang2018electron}
Yi~Jiang, Zhen Chen, Yimo Han, Pratiti Deb, Hui Gao, Saien Xie, Prafull Purohit, Mark~W Tate, Jiwoong Park, Sol~M Gruner, et~al.
\newblock Electron ptychography of 2d materials to deep sub-{\aa}ngstr{\"o}m resolution.
\newblock \emph{Nature}, 559\penalty0 (7714):\penalty0 343--349, 2018.
\bibitem[Koho et~al.(2019)Koho, Tortarolo, Castello, et~al.]{koho2019}
Samu Koho, Giovanni Tortarolo, Matteo Castello, et~al.
\newblock Fourier ring correlation simplifies image restoration in fluorescence microscopy.
\newblock \emph{Nature Communications}, 10:\penalty0 3103, 2019.
\newblock \doi{10.1038/s41467-019-11024-z}.
\newblock URL \url{https://doi.org/10.1038/s41467-019-11024-z}.
\bibitem[Konijnenberg(2017)]{konijnenberg2017}
Sander Konijnenberg.
\newblock An introduction to the theory of ptychographic phase retrieval methods.
\newblock \emph{Advanced Optical Technologies}, 6\penalty0 (6):\penalty0 423--438, 2017.
\newblock \doi{10.1515/aot-2017-0049}.
\bibitem[Li et~al.(2022)Li, Andreeto, Ranzato, and Perona]{li2022}
F.-F. Li, M.~Andreeto, M.~Ranzato, and P.~Perona.
\newblock Caltech 101 (1.0) [data set], 2022.
\newblock URL \url{https://doi.org/10.22002/D1.20086}.
\bibitem[Maiden \& Rodenburg(2009)Maiden and Rodenburg]{maiden2009}
A.~M. Maiden and J.~M. Rodenburg.
\newblock An improved ptychographical phase retrieval algorithm for diffractive imaging.
\newblock \emph{Ultramicroscopy}, 109\penalty0 (10):\penalty0 1256--1262, 2009.
\bibitem[Maiden et~al.(2017)Maiden, Johnson, and Li]{Maiden17}
Andrew Maiden, Daniel Johnson, and Peng Li.
\newblock Further improvements to the ptychographical iterative engine.
\newblock \emph{Optica}, 4\penalty0 (7):\penalty0 736--745, Jul 2017.
\newblock \doi{10.1364/OPTICA.4.000736}.
\newblock URL \url{https://opg.optica.org/optica/abstract.cfm?URI=optica-4-7-736}.
\bibitem[Nilsback \& Zisserman(2008)Nilsback and Zisserman]{nilsback2008}
M-E. Nilsback and A.~Zisserman.
\newblock Automated flower classification over a large number of classes.
\newblock In \emph{Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing}, 2008.
\bibitem[Pan et~al.(2023)Pan, Wang, Zhou, Zhou, Liu, Li, Wang, Zhang, Dong, and Zhang]{pan2023}
Xinyu Pan, Shuo Wang, Zhongzheng Zhou, Liang Zhou, Peng Liu, Chun Li, Wenhui Wang, Chenglong Zhang, Yuhui Dong, and Yi~Zhang.
\newblock An efficient ptychography reconstruction strategy through fine-tuning of large pre-trained deep learning model.
\newblock \emph{iScience}, 26\penalty0 (12):\penalty0 108420, 2023.
\newblock ISSN 2589-0042.
\newblock \doi{https://doi.org/10.1016/j.isci.2023.108420}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S2589004223024975}.
\bibitem[Rodenburg \& Faulkner(2004)Rodenburg and Faulkner]{Rodenburg04}
J.~M. Rodenburg and H.~M.~L. Faulkner.
\newblock {A phase retrieval algorithm for shifting illumination}.
\newblock \emph{Applied Physics Letters}, 85\penalty0 (20):\penalty0 4795--4797, 11 2004.
\newblock ISSN 0003-6951.
\newblock \doi{10.1063/1.1823034}.
\newblock URL \url{https://doi.org/10.1063/1.1823034}.
\bibitem[Shechtman et~al.(2015)Shechtman, Eldar, Cohen, Chapman, Miao, and Segev]{Shechtman}
Yoav Shechtman, Yonina~C. Eldar, Oren Cohen, Henry~Nicholas Chapman, Jianwei Miao, and Mordechai Segev.
\newblock Phase retrieval with application to optical imaging: A contemporary overview.
\newblock \emph{IEEE Signal Processing Magazine}, 32\penalty0 (3):\penalty0 87--109, 2015.
\newblock \doi{10.1109/MSP.2014.2352673}.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{Vaswani}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan~N. Gomez, \L{}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock In \emph{Proceedings of the 31st International Conference on Neural Information Processing Systems}, NIPS'17, pp.\ 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
\newblock ISBN 9781510860964.
\bibitem[Wang et~al.(2021)]{wang2021}
Wenhai Wang et~al.
\newblock Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021.
\newblock URL \url{https://arxiv.org/abs/2102.12122}.
\newblock Accessed 26 Apr. 2024.
\bibitem[Wittwer et~al.(2022{\natexlab{a}})Wittwer, Br\"{u}ckner, and Modregger]{Wittwer2022}
Felix Wittwer, Dennis Br\"{u}ckner, and Peter Modregger.
\newblock Ptychographic reconstruction with object initialization.
\newblock \emph{Opt. Express}, 30\penalty0 (19):\penalty0 33652--33663, Sep 2022{\natexlab{a}}.
\newblock \doi{10.1364/OE.465397}.
\newblock URL \url{https://opg.optica.org/oe/abstract.cfm?URI=oe-30-19-33652}.
\bibitem[Wittwer et~al.(2022{\natexlab{b}})Wittwer, Hagemann, Br\"{u}ckner, Flenner, and Schroer]{Wittwer__22}
Felix Wittwer, Johannes Hagemann, Dennis Br\"{u}ckner, Silja Flenner, and Christian~G. Schroer.
\newblock Phase retrieval framework for direct reconstruction of the projected refractive index applied to ptychography and holography.
\newblock \emph{Optica}, 9\penalty0 (3):\penalty0 295--302, Mar 2022{\natexlab{b}}.
\newblock \doi{10.1364/OPTICA.447021}.
\newblock URL \url{https://opg.optica.org/optica/abstract.cfm?URI=optica-9-3-295}.
\bibitem[Xie et~al.(2021)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo]{xie2021}
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose~M. Alvarez, and Ping Luo.
\newblock Segformer: Simple and efficient design for semantic segmentation with transformers.
\newblock In M.~Ranzato, A.~Beygelzimer, Y.~Dauphin, P.S. Liang, and J.~Wortman Vaughan (eds.), \emph{Advances in Neural Information Processing Systems}, volume~34, pp.\ 12077--12090. Curran Associates, Inc., 2021.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2021/file/64f1f27bf1b4ec22924fd0acb550c235-Paper.pdf}.
\bibitem[Young et~al.(2014)Young, Lai, Hodosh, and Hockenmaier]{young2014}
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.
\newblock From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 2:\penalty0 67--78, February 2014.
\end{thebibliography}