Exploring Memorization in Fine-tuned Language Models

Extracted source text · arXiv 2310.06714

% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage[]{ACL2023}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{lightblue}{RGB}{173,216,230}
\definecolor{lightred}{RGB}{255,182,193}
\definecolor{lightgreen}{RGB}{173,255,47}
\definecolor{lightyellow}{RGB}{255,255,204}
\input{math_commands.tex}
\newcommand{\jt}[1]{\textcolor{red}{@JT:~#1@}}
\newcommand{\zsl}[1]{\textcolor{orange}{@SL:~#1@}}
\newcommand{\yx}[1]{\textcolor{blue}{@YX:~#1@}}
\newcommand{\jr}[1]{\textcolor{darkgreen}{#1}}
\newcommand{\pf}[1]{\textcolor{purple}{@PF:~#1@}}
\newcommand{\han}[1]{\textcolor{blue}{@han:~#1@}}
\newcommand\MyDBox[2][HLcolor!20]{\sethlcolor{#1}\hl{#2}}
\newcommand{\highlightred}[1]{\sethlcolor{lightred}\hl{#1}}
\newcommand{\highlightyellow}[1]{\sethlcolor{lightyellow}\hl{#1}}
\newcommand{\highlightblue}[1]{\sethlcolor{lightblue}\hl{#1}}
\newcommand{\highlightgreen}[1]{\sethlcolor{lightgreen}\hl{#1}}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{hyperref}
\usepackage{wrapfig}
\usepackage{url}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{hyperref}
\usepackage{booktabs}
\usepackage{longtable}
\usepackage{longtable}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{siunitx}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{amsmath}
\usepackage{xcolor}
\usepackage{tcolorbox}
\usepackage{soul}
\usepackage{makecell}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{placeins}
\usepackage[titletoc]{appendix}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}


% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Exploring Memorization in Fine-tuned Language Models}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

% \author{First Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\\And
%   Second Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\}

\author{Shenglai Zeng$^{1}$\thanks{Equal contribution.}\thanks{Work conducted during an internship at Baidu, Inc.}\,\,\,\,, Yaxin Li$^{1\ast}$, Jie Ren$^1$, Yiding Liu$^2$, Han Xu$^1$, Pengfei He$^1$ \\ \textbf{Yue Xing$^1$, Shuaiqiang Wang$^2$, Jiliang Tang$^1$, Dawei Yin$^2$ } \\ 
$^1$Michigan State University  \quad $^2$Baidu, Inc.   \\
\{zengshe1, liyaxin1, renjie3, xuhan1, hepengf1,xingyue1,  tangjili\}@msu.edu, \\
liuyiding.tanh@gmail.com, shqiang.wang@gmail.com, yindawei@acm.org
}


\begin{document}
\maketitle
% \begin{abstract}
% LLMs have shown great capabilities in various tasks but also exhibited memorization of training data, thus raising tremendous privacy and copyright concerns.  While prior work has studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared with pre-training, fine-tuning typically involves sensitive data and diverse objectives, thus may bring unique memorization behaviors and distinct privacy risks. In this work, we conduct the first comprehensive analysis to explore LMs' memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that fine-tuned memorization presents a strong disparity among tasks. We provide an understanding of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution. By investigating its memorization behavior,  multi-task fine-tuning paves a potential strategy to mitigate fine-tuned memorization.  
\newtheorem{definition}{Definition}
\input{Sections/Abstract}
\input{Sections/Introduction}
\input{Sections/Related}
\input{Sections/Preliminaries}
\input{Sections/Experiment1}
\input{Sections/Experiment2}
 \input{Sections/Experiment3}
\input{Sections/Analysis}
\input{Sections/Conclusions}
\input{Sections/Limitations}
% \section*{Limitations}
% \section*{Ethics Statement}
\bibliography{anthology}
% \bibliographystyle{acl2023}




\input{Sections/Appendix}











\label{sec:appendix}





\end{document}

%%%%% NEW MATH DEFINITIONS %%%%%

\usepackage{amsmath,amsfonts,bm}

% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}

% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}


% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}

\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}

\def\eps{{\epsilon}}


% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}

% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}

% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}

% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}

% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}

% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}

% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}

% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}

% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}


% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}

% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}

% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}

% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}

% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}

\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution

\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}

\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak

\begin{abstract}
\label{abstract}

% \jt{for the whole paper, let us make it consistent for pre-training, pre-trained, fine-tuning, fine-tuned, let us add "-"}

Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared to pre-training, fine-tuning typically involves more sensitive data and diverse objectives, thus may bring distinct privacy risks and unique memorization behaviors. In this work, we conduct the first comprehensive analysis to explore language models' (LMs) memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution. 

% By investigating its memorization behavior,  multi-task fine-tuning paves a potential strategy to mitigate fine-tuned memorization.  

 
\end{abstract}
\vspace{-0.3cm}
\section{Introduction}
\label{Intro}
\vspace{-0.2cm}

Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, enabling significant advances across diverse applications including reading comprehension, text classification, and summarization~\citep{openai2023gpt,ouyang2022training,bai2022constitutional,touvron2023llama}. However, recent works reveal that pre-trained langauge models (LMs) tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately. For example,  \citet{carlini2021extracting} devised a training data extraction attack, successfully extracting hundreds of verbatim text sequences from GPT-2's training data. Existing works demonstrate that various factors can affect memorization and memorization effects grow with model scale, data duplication, and prompt length~\citep{lee2021deduplicating,kandpal2022deduplicating, carlini2022quantifying}. These findings raise privacy and confidentiality concerns, as interactions between humans and the deployed LMs could enable extraction of the memorized sensitive training data, such as phone numbers, people's names, etc. As the scale of LMs and their training data continues to expand, the privacy risks posed by memorization become increasingly serious.

 
In addition to pre-training, the application of LMs often involves fine-tuning on downstream tasks~\citep{touvron2023llama,chung2022scaling,ouyang2022training,longpre2023flan}, while the memorization of fine-tuning data is rather overlooked by existing studies. Compared to pre-training, fine-tuning introduces two unique perspectives with respect to memorization. \textbf{First, fine-tuning often utilizes domain-specific and private data.} For instance, developing a diagnostic chatbot \citep{yunxiang2023chatdoctor} requires collecting sensitive medical conversation data. Similarly, an academic LM \citep{beltagy2019scibert} may be trained on copyrighted essays for summarization or paraphrase generation. Leakage of such fine-tuning data can seriously violate user privacy or intellectual property rights. \textbf{Second, fine-tuning involves more complex and diverse training goals compared to pre-training. } During pre-training, the learning objective is usually language modeling from a massive unlabeled corpus (e.g., next-word prediction), which is agnostic to downstream tasks. In fine-tuning, the objective is to learn task-specific knowledge from annotated data, such as how to effectively capture the key information of a long document for summarization. The differences may induce distinct memorization behaviors and patterns during fine-tuning. Consequently, it is necessary to explore memorization for fine-tuning. Yet, it is challenging because previous insights and findings regarding the pre-trained models may not directly apply to fine-tuning. 
% Thus, dedicated efforts are desired to investigate LM's fine-tuning stage memorization.

To bridge this gap, we focus on the memorization of LMs during fine-tuning. We study a variety of fine-tuning tasks including summarization, dialogue, question answering, machine translation, and sentiment analysis. Using an automatic plagiarism detection pipeline~\citep{lee2023language}, we examine memorization on both popular open-sourced models and the models fine-tuned for diverse tasks. In both cases, we consistently observe the existence of substantial memorization under certain tasks. Moreover, we draw several new insights and reveal potential factors that may impact the memorization of fine-tuned LMs.
Our key findings and contributions are summarized as follows: 
\begin{itemize}[noitemsep, topsep=0pt, leftmargin = *]
    \item {\textit{Disparate Memorization Across Tasks.}} Particular tasks such as summarization and dialogue present high memorization. In contrast, tasks like classification, reading comprehension, and translation exhibit low memorization. This discrepancy highlights the disparate cognitive demands these tasks need from LMs. 
    \item{\textit{Task-Dependent Scaling in Fine-tuned Memorization.}} For tasks with high memorization, the degree of memorization escalates with the increase of model size. On the other hand, for tasks with low memorization, increasing the model size does not significantly amplify the memorization.
    \item {\textit{Memorization Linked to Task Information Needs.}} We hypothesize that the varying degrees of memorization across different tasks are linked to the number of input features that LMs need to retain. We further justify this based on sparse coding models and attention patterns. Specifically,
    tasks which need to understand every detail of the input tends to memorize more from the data, and the distribution of the attention score is dense across all input-output pairs.
    
    % tasks which demand high memorization exhibit uniformly distributed, sparse attention patterns, as opposed to the concentrated attention distributions observed in low memorization tasks.
    % \item {\it Multi-task fine-tuning can mitigate the memorization.} Multi-task fine-tuning can alleviate memorization compared to single-task fine-tuning.
\end{itemize}






\vspace{-0.1cm}
\section{Related Work}\label{Related works}
\vspace{-0.1cm}

Powered by the transformer architecture \citep{vaswani2017attention}, in recent years, LMs such as ChatGPT \citep{ouyang2022training}, Claude \citep{bai2022constitutional}, Palm \citep{chowdhery2022palm}, Llama \citep{touvron2023llama,touvron2023llama1} and T5 \citep{raffel2020exploring} have achieved impressive performance across a wide range of natural language processing (NLP) tasks. 
% The performance of the pre-trained models benefits from a large model size and data size.
% , and the development of large-scale computation environments facilitates the computation of such large models.
% The development of these large pre-trained models is facilitated by the increase in model size, data, and computing.
These language models are pre-trained by a large amount of data to enhance their overall proficiency. Subsequently, people usually utilize various techniques \citep{chung2022scaling,ouyang2022training,houlsby2019parameter,hu2021lora,li2021prefix,lester2021power,liu2021p}  to fine-tune the pre-trained models, thus enabling them to more effectively adapt to different downstream tasks.  

The memorization behavior of pre-trained LMs has attracted increasing attention in recent years. \citet{carlini2021extracting} first proposed a data extraction attack, demonstrating that LMs tend to memorize and regenerate segments of training data.  \citet{kandpal2022deduplicating} and \citet{lee2021deduplicating} revealed that duplicated training data is more vulnerable to memorization, and de-duplication can effectively reduce memorization.  \citet{carlini2022quantifying} further quantified memorization effects, revealing that memorization grows with model scale, data duplication, and prompt length.

There are also works providing different views and understandings on memorization. For example, \citet{ippolito2022preventing} developed an efficient defense preventing memorizing the exact sentences (verbatim memorization), yet showed it fails to prevent leakage of training data. This shows the need for definitions beyond verbatim memorization. To distinguish "common"  memorization from "rare" memorization, \citet{zhang2021counterfactual} formulated a new notion of counterfactual memorization, which measures how predictions change if a particular document is deleted during training.  \citet{biderman2023emergent} investigated predictable memorization by extrapolating small or partially-trained LMs' behavior to forecast memorization in larger models. They further presented scaling laws of prediction and explored ways to improve prediction reliability.

While most literature focused on memorization during pre-training, limited work has investigated memorization in the fine-tuning stage. \citet{mireshghallah2022memorization} examined memorization risks in different fine-tuning methods for large LMs. They found that fine-tuning only the head leads to higher memorization compared to fine-tuning smaller adapter modules.  \citet{lee2023language} studied plagiarism during fine-tuning, concluding that the plagiarism patterns in fine-tuned LMs depend on corpus similarity and homogeneity. However, these studies considered fine-tuning with the same objective as pre-training, which is different from the the common practice of fine-tuning in various tasks. As a result, in this paper, we focus on the more general and realistic scenario of multifaceted fine-tuning across diverse objectives.  





\vspace{-0.25cm}
\section{Preliminary}\label{Preliminary}
\vspace{-0.25cm}

In this section, we first introduce the definition of memorization and the detection methods used in this paper, and then introduce our preliminary findings on open-sourced fine-tuned LMs across various tasks.

\subsection{Definitions and Notations}
% \zsl{need to make it more clear}

\paragraph{Definitions of memorization in literature.} In literature, there are some definitions of memorization. For example, in \citet{carlini2022quantifying}, a straightforward and strict definition is that a string $s$ is extractable with its context $p$ (with length $k$) if the concatenation $[p \| s]$ exists in the training set and $f(p)$ produces exactly the output of $s$, i.e., $f(p)=s$.  This is defined as \textbf{verbatim memorization}. In~\citet{ippolito2022preventing}, a relaxed definition of memorization is using the Bilingual Evaluation Understudy (BLEU) score. 

However, the above two definitions only consider memorizing the exact wordings of the data, instead of memorizing the meaning of the data. Therefore, in~\citet{lee2023language}, plagiarism detection tools are leveraged to to identify memorization through comparing the machine-generated text with the whole training set .  

% \textbf{Memorization in Pre-training.}
% Given a language model $f(\cdot)$ which takes inputs from text $x$ and outputs the completion, 
% previous works have various definitions of memorization in the pre-training stage.  Previous work \citep{carlini2022quantifying} made a straightforward and strict definition that a string $s$ is extractable with its context $p$ (with length $k$) if the concatenation $[p \| s]$ exists in the training set and $f(p)$ produces exactly the output of $s$, i.e., $f(p)=s$. Such a definition is called verbatim memorization. \citep{ippolito2022preventing} also gives a relaxed definition of memorization called approximate memorization. They compare the similarity between $f(p)$ and $s$ using the Bilingual Evaluation Understudy (BLEU) score, which compares n-grams of two sentences. The sentence  $s$ is identified as memorized if $\operatorname{BLEU}(f(p), s)>0.75$. Furthermore, some works such as~\citep{lee2023language} make use of plagiarism detection tools to compare the machine-generated text with the whole training set to identify memorization. 

\paragraph{Definition of fine-tuned memorization.} 
In the fine-tuning stage, models are trained for specific tasks like sentiment analysis, dialog, and summarization. We define fine-tuning as supervised training using samples \(\mathcal{D}_{\text{train}}=\left\{\left(x_i, y_i\right)\right\}_{i=1}^n\), where \(y_i\) represents the target output for input \(x_i\).
 Since the input texts usually contain more information than the output in our considered tasks, we majorly discuss the potential information leakage from the input corpus in the training set, i.e., $\mathcal{D}_\text{input} = \{x_i\}_{i=1}^n$.

To explore memorization, we follow the prompting approach~\citep{carlini2022quantifying} by dividing each $x_{i}=[p_{i} \| s_{i}]$ to a length-$k$ prefix $p_{i}$, and a suffix $s_{i}$. We further define the set of all prefixes in the training set as $P=\{p_i\}_{i=1}^n$, and the set of all suffixes as $S=\{s_i\}_{i=1}^n$. We define fine-tuned memorization as follows:

\begin{definition}[Fine-tuned memorization]
Given the fine-tuned model function \(f\), fine-tuned memorization is defined as when the model output $f(p_i)$ contains information of any $s_j\in S $, formalized by $D(f(p_i), s_j) = \text{True}$, where $D$ is a discriminative function to judge the similarity between two texts.


\end{definition}


% In our paper, to define memorization, when inputting the model output $f(p_i)$ and $s_j$ to a publicly available plagiarism detection tool $D$, we examine whether $D(f(p_i),s_j)= \text{True}$

\paragraph{Evaluation method.}  

% \jt{I do not think pre-training LLM is to emphasize the memorization of x-y, right? what is the purpose for the following sentence?? Do we plan to distingiush ours from some specific studies about x-y, if yes, let us make it explicit}It is important to note that our definition of memorization diverges from conventional definitions that emphasize the memorization of x-y relationships.

In our practice, we input  $P=\{p_i\}_{i=1}^n$ to the model and utilize a local search engine to quickly locate suspicious texts and use a plagiarism detection tool to serve as the discriminative function $D$. In detail, given a dataset of size \( n \) with suffix space $S$, we employ the local search engine, Elasticsearch\footnote{\href{https://www.elastic.co/elasticsearch/}{https://www.elastic.co/elasticsearch/}}, to identify the top-\( K \) corpus candidates \( S_K^i = \{ s_1^i, s_2^i, \ldots, s_K^i \} \) which are similar to \( f(p_i) \). Then we utilize the PAN2014 plagiarism detection tool\footnote{\href{https://pan.webis.de/clef14/pan14-web/text-alignment.html}{https://pan.webis.de/clef14/pan14-web/text-alignment.html}} to assess the similarity between \( f(p_i) \) and each candidate \( s_j \in S_k^i \). This detection tool is capable of identifying the presence of plagiarised pairs \( (d_i, d_j) \), where \( d_i \) and \( d_j \) are sub-strings from \( f(p_i) \) and \( s_j \), respectively. The main idea of this tool is to transform text into term frequency-inverse document frequency (TF-IDF) vectors and utilize sentence similarity measures (cosine similarity) to identify plagiarism cases. We say the fine-tuned model memorizes $s_j$ if the plagiarism is confirmed. We then count the number of memorized cases and divide by $n$ to get the total memorization rate. We use $n=10000$ in our experiments. This memorization rate quantifies the memorization exhibited in the model. 


% In our paper, to define memorization, when inputting the model output $f(p_i)$ and $s_j$ to a publicly available plagiarism detection tool $D$, we examine whether $D(f(p_i),s_j)= \text{True}$. In detail, we utilize the PAN2014 plagiarism detection tool\footnote{\href{https://pan.webis.de/clef14/pan14-web/text-alignment.html}{https://pan.webis.de/clef14/pan14-web/text-alignment.html}} to assess the similarity between \( f(p_i) \) and each candidate \( s_j \in S_k^i \). This detection tool is capable of identifying the presence of plagiarised pairs \( (d_i, d_j) \) a, where \( d_i \) and \( d_j \) are word pieces from \( f(p_i) \) and \( s_j \), respectively. We say the fine-tuned model memorizes $s_j$ if the plagiarism is confirmed. We then count the number of memorized $s_j$s and divide by $n$ to get the total memorization ratio \footnote{We use $n=10000$ in our experiments}. This memorization ratio quantifies the memorization exhibited by the model. 

% \jt{why we want to divide them into three categories?? do we follow some works?? 
%  I think in general, we need to cite relevant papers to provide evidence about the reasonability of our choice or design}
 
Moreover, the detection tool can categorize memorized content into three distinct types. To provide a detailed quantification of the memorization behavior, we include these categories following the methods in prior work \citep{lee2023language}. 
\begin{itemize}[noitemsep, topsep=1pt]
    \item \textbf{Verbatim}: \( d_j \) is an exact replica of \( d_i \).
    \item \textbf{Paraphrase}\footnote{Paraphrasing is further assessed using RoBERTa and NER, classifying $p < 0.5$ as low-confidence and $p > 0.5$ as high-confidence, with both reported.}: \( d_j \) is a rephrased version of \( d_i \) 
    \item \textbf{Idea memorization}: \( d_j \) condenses  \( d_i \) into fewer sentences, or vice versa.
    % \( d_j \) encapsulates the fundamental essence of \( d_i \) but is presented in a more comprehensive manner.
\end{itemize}
Generally, all these memorization types indicate that the model generates information about the suffix of training data $x$ not given as input.
% , implying that the model memorizes the training data. 
More details of the detection pipeline, e.g., implementation descriptions and differences among memorization types, are included in the Appendix \ref{Details and pipeline}. Typical memorization cases are shown in Appendix \ref{Mem Cases}.

% descriptions of these tools are included in the Appendix \ref{Pipeline}.



\begin{table*}[t]
\centering
\caption{Memorization rate of open-sourced LLMs fine-tuned on various tasks}
\label{tab:Open_Sourced finetune}
\resizebox{0.9\textwidth}{!}{
\begin{tabular}{@{}c|ccccccc@{}}
\toprule
Task & Dataset & \begin{tabular}[c]{@{}c@{}} Source\\Model\end{tabular} & \begin{tabular}[c]{@{}c@{}}Total\\Mem Rate \end{tabular}  & Verbatim & Idea & \begin{tabular}[c]{@{}c@{}} Paraphrase\\($p>0.5$) \end{tabular} & \begin{tabular}[c]{@{}c@{}} Paraphrase \\($p<0.5$) \end{tabular} \\
\midrule
\href{https://huggingface.co/facebook/bart-large-cnn}{Summarization} & CNN/Daily Mail  & Bart\_Large & 20.7\% & 1.3\% & 0\% & 9.8\% & 9.6\% \\
\href{https://huggingface.co/Narrativaai/BioGPT-Large-finetuned-chatdoctor}{Medical Dialog} & ChatDoctor & BioGPT & 19.6\% & 0.1\% & 3.5\% & 7.8\% & 8.2\% \\
\href{https://huggingface.co/potsawee/t5-large-generation-squad-QuestionAnswer}{Extractive QA} & SQuAD\_v2 & T5\_large & 0.1\% & 0\% & 0\% & 0\% & 0.1\% \\
\href{https://huggingface.co/potsawee/t5-large-generation-race-QuestionAnswer}{Abstractive QA} & Race & T5\_large & 0.3\% & 0\% & 0\% & 0.2\% & 0.1\% \\
\href{https://huggingface.co/facebook/wmt19-en-de}{Translation} & WMT\_19 & FSMT & 0\% & 0\% & 0\% & 0\% & 0\% \\
\href{https://huggingface.co/mrm8488/t5-base-finetuned-imdb-sentiment}{Sentiment Classification} & IMDB & T5-base & 0\% & 0\% & 0\% & 0\% & 0\% \\
\bottomrule
\end{tabular}
}
\end{table*}


\vspace{-0.3cm}
\subsection{Preliminary Findings}
\vspace{-0.2cm}

To initially explore the memorization effects during fine-tuning, we examine several popular open-sourced fine-tuned models from HuggingFace that were fine-tuned on 6 representative tasks.We attach the descriptions of all models and datasets used in Appendix \ref{Datasets and model}.  These tasks include summarization, medical dialog, question and answering (QA), translation, and sentiment analysis. 
% The results are shown in Table~\ref{tab:Open_Sourced finetune}. %\jt{ We need to briefly introduce the columns " dataset" and "model" in the table??}
The preliminary results in Table~\ref{tab:Open_Sourced finetune} suggest that substantial memorization of the fine-tuning data occurs in fine-tuning. For summarization and medical dialog models, we identified total memorization rate of 20.7\% and 19.6\%, respectively. These high rates could imply potential privacy violations or copyright issues.
% in applications like academic summarization and healthcare chatbots. 
Furthermore, the level of memorization varies across tasks.
% , displaying clear task-specific patterns. 
Models fine-tuned for summarization and medical dialog exhibit high memorization, while models for remaining tasks show much lower memorization. These observations motivate further in-depth analysis to validate the observed task-specific memorization behavior in Section~\ref{Ex1} and the possible scaling effect in Section~\ref{Ex2} to fine-tune models by ourselves. Furthermore, in Section \ref{Ex3}, we provide an in-depth analysis and understanding of the potential reason behind the observed disparate fine-tuned memorization across tasks.

% attention score density as a potential indicator to quantify and compare memorization tendencies across diverse tasks. Finally, Section~\ref{Ex3} investigates multi-task fine-tuning.





    
 


\vspace{-0.15cm}
\section{Disparate Memorization Across Tasks}
% \section{Fine-tuning memorization effects are task-specifi}
\label{Ex1}
\vspace{-0.15cm}

Our preliminary study demonstrates that the memorization of fine-tuned models varies on different fine-tuning tasks. However, the causes of such difference are still unclear, which could be fine-tuning datasets or model architectures.
% other factors such as the fine-tuning datasets or model architectures could also impact the memorization effect. 
To clarify this, in Section~\ref{sec:41}, we 
% systematically 
control the impact of different variables to more precisely explore the relation between fine-tuning task and the memorization effect. In Section \ref{sec:impact factor}, we further examine the impact of decoding methods and prefix length.  Note that in our fine-tuning process, we ensure that our fine-tuned models have satisfactory performance on downstream tasks, see Appendix \ref{finetune performance}. 

% Besides, we add ablation studies for different decoding methods in Appendix \ref{decoding_ab}, and prefix lengths in Appendix \ref{prefix_ab}.



% In Section~\ref{sec:sparse_coding}, 
% via constructing conceptual models, we discuss the underlying difference between these fine-tuning tasks which potentially influence the memorization of LMs. In our fine-tuning process, we make sure our fine-tuned models have satisfied performance on downstream tasks and report the performance in Appendix \ref{finetune performance}. \textcolor{blue}{Besides, we add ablation studies for different decoding methods in Appendix \ref{decoding_ab} , prefix lenghs in Appendix \ref{prefix_ab} and report the memorization ratio of T5-base model  in Appendix \ref{base_ab}  }



\begin{table*}[htbp]
\centering
\caption{Memorization rate of T5-base fine-tuned on various tasks}
\label{tab:self finetune}
\resizebox{0.8\textwidth}{!}{
\begin{tabular}{@{}c|c|c|ccccc@{}}
\toprule
Task & Dataset & Model & \begin{tabular}[c]{@{}c@{}} Total \\Mem Rate \end{tabular} & Verbatim & Idea & \begin{tabular}[c]{@{}c@{}} Paraphrase \\ (P>0.5) \end{tabular} & \begin{tabular}[c]{@{}c@{}} Paraphrase \\ (P<0.5) \end{tabular} \\
\midrule
\multirow{3}{*}{Summarization} & \multirow{3}{*}{Multi-news} & T5-base & 13.98\% & 3.58\% & 0.66\% & 4.28\% & 5.46\% \\
 & & T5-finetuned & 22.33\% & 4.23\% & 0.65\% & 6.23\% & 11.22\% \\
 & & Difference & $\uparrow$8.35\% & $\uparrow$0.65\% & $\downarrow$0.01\% & $\uparrow$1.95\% & $\uparrow$5.76\% \\
\midrule
\multirow{3}{*}{Dialog} & \multirow{3}{*}{chatdoctor} & T5-base & 1.60\% & 0.03\% & 1.04\% & 0.11\% & 0.42\% \\
 & & T5-finetuned & 8.27\% & 0.02\% & 1.41\% & 1.75\% & 5.09\% \\
 & & Difference & $\uparrow$6.67\% & $\downarrow$0.01\% & $\uparrow$0.37\% & $\uparrow$1.64\% & $\uparrow$4.67\% \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}} Sentiment \\ Classification \end{tabular}} & \multirow{3}{*}{imdb} & T5-base & 0.78\% & 0.05\% & 0.37\% & 0.16\% & 0.20\% \\
 & & T5-finetuned & 0.80\% & 0.04\% & 0.30\% & 0.17\% & 0.29\% \\
 & & Difference & $\uparrow$0.02\% & $\downarrow$0.01\% & $\downarrow$0.07\% & $\uparrow$0.01\% & $\uparrow$0.09\% \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}} Reading \\ Comprehension \end{tabular}} & \multirow{3}{*}{Squad\_v2} & T5-base & 0.08\% & 0.02\% & 0.00\% & 0.01\% & 0.05\% \\
 & & T5-finetuned & 0.15\% & 0.04\% & 0.00\% & 0.05\% & 0.06\% \\
 & & Difference & $\uparrow$0.07\% & $\uparrow$0.02\% & - & $\uparrow$0.04\% & $\uparrow$0.01\% \\
\midrule
\multirow{3}{*}{Translation} & \multirow{3}{*}{wmt} & T5-base & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% \\
 & & T5-finetuned & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% \\
 & & Difference & - & - & - & - & - \\
\bottomrule
\end{tabular}}
\end{table*}

\vspace{-0.2cm}









\subsection{Fine-tuned with Fixed Pre-trained LM and Dataset}\label{sec:41}

\vspace{-0.1cm}

\paragraph{Fine-tuned on T5-base LM.}  To eliminate the potential impact from the pre-trained LM, we conduct experiments to fine-tune the same pre-trained T5-base model\footnote{We also fine-tuned GPT-Neo models and found consistent findings with T5, we report the results in Appendix \ref{gptneo-ft} and Table \ref{tab:self fine-tune gptneo}.} for different fine-tuning tasks. We treat all the tasks as generative tasks (like instruction-tuning) and do not add any additional modules (e.g., MLP). Details of the datasets and memorization results are presented in Table~\ref{tab:self finetune}\footnote{We provide the statistical significance testing of memorization rate in Appendix \ref{Statistic Test}. }. In case a part of the fine-tuning data appears in the pre-training data, we report the memorization rate of the pre-trained model\footnote{In the context of pre-trained models, we continue to utilize the prefixes from the fine-tuning dataset for evaluating memorization. This process is detailed in Appendix \ref{Pipeline}.} and present the change in the memorization rate before and after fine-tuning. Table~\ref{tab:self finetune} clearly shows a substantial total memorization rate  and memorization gain for summarization tasks (22.3\%, $\uparrow$8.35\%) and medical dialogue (8.27\%, $\uparrow$6.67\%). Meanwhile, the memorization and the gain in fine-tuning are much lower for reading comprehension (0.15\%, $\uparrow$0.07\%), translation (0.0\%, $\uparrow$0.0\%), and sentiment classification (0.8\%, $\uparrow$0.02\%). These observations are consistent with our preliminary findings on open-sourced models and suggest that fine-tuned memorization with the same pre-trained LM architecture still demonstrates a strong task disparity. 

% \paragraph{Fine-tuning on RentTheRunway with T5.} 

\paragraph{Fine-tuned on different tasks with the same dataset.}

% In this experiment, we further reduce the potential impact of the difference among fine-tuned datasets. To achieve this goal, 

In this experiment, we investigate the memorization of different tasks fine-tuned on the same pre-trained LM with the same dataset. Specifically, we fine-tune the T5-base model on RentTheRunway dataset \citep{misra2018decomposing}, which contains self-reported clothing fit feedback from customers along with additional metadata. Each product has multiple attributes, including customer reviews, ratings, review summaries, review dates, etc. 
It allows us to fine-tune the same pre-trained model (i.e., T5-base), with the identical inputs (i.e., the customer reviews), for different task objectives (i.e., review summarization and sentiment classification). 
% In particular, we use the customer reviews as input and the review summaries as output to fine-tune a summarization model. 
Note that we map the ratings into positive and negative labels for fine-tuning sentiment classification model. 
% In this setting, the two models have different task objectives but identical inputs and start from the same model for fine-tuning. 
The memorization performance of these two models is shown in Figure~\ref{fig:Rent}. The results demonstrate that the summarization model exhibits higher memorization compared to the classification model, which validates that the task objective impacts the memorization of fine-tuned models.

\begin{figure}[ht]
  \centering
  \includegraphics[width=0.35\textwidth]{pic/rent.pdf}
  \caption{\small Memorization of T5-base fine-tuned on RentTheRunway.}
  \label{fig:Rent}
\end{figure}
\vspace{-0.4cm}

\subsection{Further Probing}
\vspace{-0.2cm}
\label{sec:impact factor}
\begin{figure*}[t]
\centering
\resizebox{0.9\textwidth}{!}{%
    \begin{minipage}{\textwidth}
        \subfloat[ Prefix length]{\includegraphics[width=.49\textwidth]{pic/prefix_length_plot.pdf}
        \label{fig:Ablation_prefix}}
        \subfloat[ Sampling methods]{\includegraphics[width=.49\textwidth]{pic/decoding.pdf}
        \label{fig:Ablation_decoding}}
    \end{minipage}
}

\caption{Impact of prefix length and sampling methods on memorization }
\label{fig:Ablation}
\end{figure*}


The memorization behavior of pre-trained models has been shown to be influenced by factors such as generation sampling methods and input prefix lengths \citep{carlini2022quantifying, lee2023language}. In this subsection, we extend our investigation to explore how these factors affect memorization in fine-tuned models. We also conduct an ablation study on the impact of training epochs on memorization and report the results in Appendix \ref{ablation_training} and Table~\ref{tab: ablation_epochs}. 

\paragraph{Input prefix length.} We vary the length of the prefix of inputs and report the main results in Figure \ref{fig:Ablation_prefix}. 
 We find that the length of prefix tokens influences memorization differently across tasks. In summarization and dialogue tasks, the total memorization rate tends to increase with longer prefixes, aligning with existing research on pre-trained memorization. However, in sentiment classification and QA, altering the prefix length does not significantly affect memorization. More results in Appendix \ref{prefix_ab} and Table \ref{tab: t5_prefix_length}. \textbf{Despite these variations, a consistent disparity in memorization across different tasks persists, regardless of the prefix length.}
\vspace{-0.15cm}
% \jt{it is better to present results if we want to have a discussion. }
 \paragraph{Generation sampling.}
We study the impact of different generation sampling methods including (i) top-k (k=40) sampling, (ii) top-p (p=0.8) sampling and (iii) changing the temperature to T=1, and report the main results in Figure \ref{fig:Ablation_decoding} and more results in Appendix \ref{decoding_ab} and Table \ref{tab: t5_decoding}. It is observed that sampling affects memorization differently across tasks: it lowers memorization in high-memorization tasks like summarization and dialogue, but has a negligible or even increasing effect on memorization in low-memorization tasks such as sentiment classification. \textbf{Despite these variations, a significant, consistent disparity in memorization remains across different tasks, indicating an intrinsic, task-specific inclination towards memorization that is not significantly altered by sampling methods.}

% \begin{itemize}
%     \item For high-memorization tasks
%     \item For low-memorization tasks
% \end{itemize}
% \subsection{Understanding the memorization disparity among tasks}\label{sec:sparse_coding}

% In this subsection, we aim to further investigate the question: \textit{why do different fine-tuning tasks present different memorization behaviors?}  Intuitively, for language tasks such as sentiment analysis or extractive QA,  only a few words or sentences are enough for the model to complete the task (e.g. words expressing sentiment, the answer to the question). Thus the model only needs to learn specific key features and is less likely to memorize the data species.  However, the tasks such as summarization and dialog, require the model to understand the whole passage, which could boost memorization.
% %internalize almost all salient features of the input text, involving greater parameter updating during training
% %\pf{Does this overclaim(greater parameter updating)? If this comes from the following analysis, I suggest leaving these explanations as implications after analysis and only providing simple intuition here.}. 
% %This extensive learning process likely enables the model to incidentally memorize more specifics of training sentences. In contrast, for tasks like sentiment classification and extractive QA, the model needs only to identify a few key semantic features. This sparse parameter updating reduces the model's propensity to memorize inconsequential patterns. 
% In other words, the memorization behavior might be closely related to the information needed to fulfill certain language tasks.  To give a more detailed analysis, in the following, we will provide a conceptual discussion based on a sparse decoding model, which is a popular model for modeling text and vision data.~\citep{arora2015latent,arora2018linear,olshausen1997sparse,olshausen2004sparse} 

% \paragraph{Sparse coding model.} 
% %Here we use a data model called sparse coding, which is a popular model to model image, text and speech data. 
% Suppose that an observed text data is denoted as $Z$ and  $ Z \in \mathbb{R}^{d\times D}$ where $D$, is the sequence length and $d$ is the length of the embedding.
% %\zsl{TO DO: add citations to prove it is widely used} Then we have:
% The basic assumption of sparse coding is that the data $Z$ comes from combinations of a few hidden features. 
% Hence, we use $X$ to represent these essential features and we have the relation: 
% \begin{equation}
%   Z=UXV
%     \label{eq:Z=UXV}
% \end{equation}
% where $X \in R^{k\times K}$, $U\in\mathbb{R}^{d\times k}$, $V\in\mathbb{R}^{K\times D}$ ,  $k \leq d$, $K \leq D$. Each column of $U$ is a unit vector and orthogonal with each other. Each row of $V$ is a unit vector and orthogonal with each other. Therefore, each element in $Z$ is a linear combination of the elements from $X$. Our assumption is a simplified version of the sparse coding model without noise. We also modify the original 1D feature $X$ in sparse coding into 2D for our model. 

% \paragraph{Task Complexity.} 

% Under sparse coding, we further assume the model's targeted output can be fully expressed by a linear transformation of $X$. However, different fine-tuning tasks may differ in \textit{how much information of the input is needed by the task}. Below, we give two perspectives to illustrate why "complex tasks" may have more memorization. 


% First, the number of parameters to connect $Z$ with the final target is related to the information needed by the task. Consider we have a \textit{``simple task''} (like sentiment analysis) where the model output is only one scalar (preference). For example, this scalar is decided by a linear combination of $X$, which is $a^{\top}Xb$, where $a\in\mathbb{R}^k$ and $b\in\mathbb{R}^K$. Thus, the loss function for a sentiment classification model  $f_{cls}$ can be defined as:
% \begin{equation}
% \label{cls_loss}
%     l(f_{cls}(Z),a^{\top}Xb).
% \end{equation}
% If we further assume the loss functions as square loss, the best solution of $f_{cls}$ is:
%    $f_{cls}(Z)=(a')^{\top}Zb'$
% where $a^{\top}U^{\top}:=a'$ and $V^{\top}b:=b'$. 
% It means that the model only needs to learn two vector parameters $a'$ and $b'$.
% Nevertheless, for some tasks such as summarization, the output text is desired to contain all key information from the input passage $Z$. Thus, the target output can be expressed by  $X$. We consider the following loss for a summarization model $f_{sum}$:
% \begin{equation}\label{eq:complex}
%     l(f_{sum}(Z), X)
% .\end{equation}
% The output $f_{sum}(Z)$ therefore contains all information about $X$, which we denote as \textit{``complex task''}.  Considering square loss, the best solution of $f_{sum}$ is $f_{sum}(Z)=U^{\top}Z V^{\top}$. Comparing the above two tasks, for simple tasks like classification, the model just requires a small amount of information and needs to learn minimal ($d+D$)  values in $a',b'$, while for complex tasks like summarization, the model needs to learn ($dk+DK$)  parameters of $U'=U^{\top},V'=V^{\top}$. Intuitively, the model that learns more information from training data tends to have larger memorization. An alternative explanation for this is that the expression in Equation~\ref{eq:complex} makes model inversion attack possible. As the model learns  $U', V'$, the attacker can conduct a model inversion attack via $Z=U'^{T}XV'^{T}$, which means one can use $f(Z)$ to recover the input data $Z$. This means the information of data $Z$ is fully encoded in the neural network.

 

% Second,  the sparsity of the learned matrix ($U$,$V$) or vectors ($a'$,$b'$) may also vary, indicating different amounts of information needed and leading to different complexity of the task. For example, in sentiment classification, what the network actually learns depends on the sparsity of $b'$.  If $b'$ is sparse, it means that we can simply pick several tokens from the sequence and determine the class. 






\vspace{-0.1cm}
\section{Memorization Scaling Behavior of Fine-tuned LMs}
\label{Ex2}
\vspace{-0.1cm}

It is evident that memorization in pre-trained models tends to increase with model size. To understand the scaling behavior of memorization in fine-tuned models, we conduct a systematic analysis on fine-tuned models, comparing the memorization in various tasks using different sizes of the T5 model: T5-small (60M), T5-base (220M), T5-large (770M), and T5-xl (3B). 

\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \begin{minipage}{0.9\textwidth}
        \subfloat[Summarization]{\includegraphics[width=.25\textwidth]{pic/summarization_scaling3.pdf}
        \label{fig:summary-scaling}}
        \subfloat[Dialog]{\includegraphics[width=.25\textwidth]{pic/dialog_scaling3.pdf}
        \label{fig:dialog-scaling}}
        \subfloat[Sentiment classification]{\includegraphics[width=.25\textwidth]{pic/sentiment_scaling3.pdf}
        \label{fig:sentiment-scaling}}
        \subfloat[Extractive QA]{\includegraphics[width=.25\textwidth]{pic/QA_scaling3.pdf}
        \label{fig:QA-scaling}}
    \end{minipage}
}

\caption{Scaling behavior of fine-tuned memorization}
\label{fig:Scaling}
\end{figure*}
\vspace{-0.2cm}
\subsection{High-memorization Tasks }
\vspace{-0.2cm}
% \jt{ why we include T5-base?? I think we can remove that since the behavior is not consistent and we never discuss it?? }\zsl{As we need to show the memorization increase with size is from the fine-tuned memorization instead of the pre-trained memorization }

We explore the scaling behavior in high-memorization tasks, specifically focusing on summarization (fine-tuned on Multi-news) and dialogue (fine-tuned on Chatdoctor). The results can be found in Figures \ref{fig:summary-scaling} and \ref{fig:dialog-scaling} respectively. We can see that memorization in fine-tuning increases with model size. As a benchmark, we also provide the memorization rate for the pre-trained model, for which we can see that when increasing from a model size of 220M, the memorization rate does not further increase much. These two observations in the fine-tuned model and the pre-trained model together reveal that the fine-tuned model is memorizing information from the fine-tuning data, indicating severe privacy threats when scaling up the models in these tasks.

% We first investigate the scaling behavior of fine-tuned memorization among high-memorization tasks, summarization(fine-tuned on Multi-news) and dialog(fine-tuned on Chatdoctor) on Figure \ref{fig:summary-scaling} and Figure \ref{fig:dialog-scaling}. From the results we can clearly see that the fine-tuned memorization increases with the model size. This is aligned with the observation of pre-trained LM memorization \citep{carlini2022quantifying}, indicating increasing privacy risks using larger models for those tasks.
\vspace{-0.35cm}
\subsection{Low-memorization Tasks }
\vspace{-0.3cm}
In contrast to high-memorization tasks, low-memorization tasks such as sentiment classification and question answering exhibit different scaling behaviors. As illustrated in Figures \ref{fig:sentiment-scaling} and \ref{fig:QA-scaling}, an increase in model size does not result in a rise in the memorization rate, and the memorization rate is consistently low. This suggests that even when large models are fine-tuned on these tasks, the possibility of memorization and outputting fine-tuning data is relatively low.
\vspace{-0.5cm}
\section{Understanding the Memorization Disparity}
\label{Ex3}



\begin{figure*}[t]
\centering
\resizebox{0.95\textwidth}{!}{%
    \begin{minipage}{\textwidth}
        \subfloat[Avg-Summ.]{\includegraphics[width=.25\textwidth]{pic/summary11.pdf}
        \label{fig:summary-heatmap}}
        \subfloat[Sample-Summ.]{\includegraphics[width=.25\textwidth]{pic/summaryA1.pdf}
        \label{fig:summary-A}}
        \subfloat[Avg-Dialog]{\includegraphics[width=.25\textwidth]{pic/dialog11.pdf}
        \label{fig:dialog-heatmap}}
        \subfloat[Sample-Dialog]{\includegraphics[width=.25\textwidth]{pic/dialogA1.pdf}
        \label{fig:dialog-A}}\\
        \subfloat[Avg-Sent.]{\includegraphics[width=.25\textwidth]{pic/sentiment11.pdf}
        \label{fig:sentiment-heatmap}}
        \subfloat[Sample-Sent.]
        {\includegraphics[width=.25\textwidth]{pic/sentimentA1.pdf}
        \label{fig:sentiment-A}}
        \subfloat[Avg-QA]{\includegraphics[width=.25\textwidth]{pic/QA11.pdf}
        \label{fig:QA-heatmap}}
        \subfloat[Sample-QA]{\includegraphics[width=.25\textwidth]{pic/QAA1.pdf}
        \label{fig:QA-A}}
        
    \end{minipage}
}

\caption{Decoder-encoder attention heatmaps for (a, b) Summarization, (c, d) Dialog, (e, f) Sentiment Analysis, and (g, h) QA. (a, c, e, g) show average heatmaps from 10 samples, while (b, d, f, h) show heatmaps from a single sample.}
\label{fig:heatmap}
\end{figure*}

\vspace{-0.2cm}
In the previous sections, we empirically examine the memorization rate and scaling behavior among a variety of tasks, and demonstrate the discrepancy between high- and low-memorization tasks. In this section, we provide understanding and evidence on the underlying reason behind such disparity of fine-tuned memorization.

\vspace{-0.2cm}
\subsection{Correlation between Memorization and Task-specific Information}\label{sec:sparse_coding}

In this subsection, we aim to investigate the question: \textit{why do different fine-tuning tasks present different memorization behaviors?} We conjecture that the memorization behavior might be closely related to the information needed to fulfill certain language tasks. Intuitively, for language tasks such as sentiment analysis or extractive QA, only a few words or sentences are enough for the model to complete the task. For example, one can determine the sentiment based on some specific words in the sentiment, and can answer a question based on certain pieces of information. In this case, the model only needs to learn specific key features and is less likely to memorize the other data. On the other hand, for tasks such as summarization and dialog, they require the model to learn more input features to complete the task, as the essential information from these inputs is also reflected in the output. As a result, the fine-tuning process will encode more input knowledge from the data in the model parameters, leading to potential concerns of memorization. In the following, we provide a conceptual discussion based on the sparse decoding model. The sparse coding model is a method that represents original data by focusing on its key features, using only the most crucial elements to efficiently express the core information, which is a popular model for modeling text and vision data.~\citep{arora2015latent,arora2018linear,olshausen1997sparse,olshausen2004sparse} 
\vspace{-0.2cm}
\paragraph{Sparse coding model.} 
%Here we use a data model called sparse coding, which is a popular model to model image, text and speech data. 
Denote an observed text data as $ Z \in \mathbb{R}^{d\times D}$ where $D$ and $d$ represent the sequence length and the length of the embedding respectively.
%\zsl{TO DO: add citations to prove it is widely used} Then we have:
The basic assumption of sparse coding is that the data $Z$ comes from combinations of a few hidden features. 
We use $X$ to represent these hidden features and consider the following relation: 
\begin{equation}
  Z=UXV
    \label{eq:Z=UXV}
\end{equation}
where $X \in R^{k\times K}$, $U\in\mathbb{R}^{d\times k}$, $V\in\mathbb{R}^{K\times D}$ ,  $k \leq d$, $K \leq D$. Each column of $U$ is a unit vector and orthogonal with each other. Each row of $V$ is a unit vector and orthogonal with each other. Given the above formulation, each element in $Z$ is a linear combination of the elements from $X$. Compared to previous literature, our assumption is a simplified version of the original sparse coding model as we do not further impose noise in the data generation model. Besides, we also modify the original 1D feature $X$ in sparse coding into 2D for our model. 
\vspace{-0.2cm}
\paragraph{Task complexity.} 

Under the sparse coding model, we further assume that the output can be fully expressed by a linear transformation of $X$. However, different fine-tuning tasks may differ in \textit{how much input information is needed by the task}. We present two perspectives below to illustrate why "complex tasks" may have more memorization. 


First, the number of parameters to connect $Z$ with the final target is related to the task-specific information. Consider we have a \textit{``simple task''} (e.g., sentiment analysis) where the model output is only one scalar (preference) decided by some certain features or a combination of $X$. In the simplest case, the scalar is a linear function of $X$, $a^{\top}Xb$, where $a\in\mathbb{R}^k$ and $b\in\mathbb{R}^K$. In this case, the loss function for a sentiment classification model  $f_{cls}$ can be defined as:
\begin{equation}
\label{cls_loss}
    l(f_{cls}(Z),a^{\top}Xb).
\end{equation}
If we further assume the loss functions as square loss, the best solution of $f_{cls}$ is:
   $f_{cls}(Z)=(a')^{\top}Zb'$
where $a^{\top}U^{\top}:=a'$ and $V^{\top}b:=b'$. 
It means that the model only needs to learn two vector parameters $a'$ and $b'$.
On the other hand, for tasks such as summarization, the output text is desired to contain all key information from the input $Z$. We consider the following loss for the summarization task $f_{sum}$:
\begin{equation}\label{eq:complex}
    l(f_{sum}(Z), X)
.\end{equation}
In the above formulation, the output $f_{sum}(Z)$ contains all information about $X$. We denote such a task as a \textit{``complex task''}. With the squared loss, the best solution of $f_{sum}$ is $f_{sum}(Z)=U^{\top}Z V^{\top}$. Comparing the above two tasks, for simple tasks like classification, the model just requires a small amount of information to learn $a',b'$ ($d+D$), while for complex tasks such as summarization, the model needs to learn ($dk+DK$)  parameters of $U'=U^{\top},V'=V^{\top}$.
Further, the model which learns more information from the data tends to memorize more. The expression in Equation~\ref{eq:complex} makes model inversion attack possible. As the model learns  $U', V'$, the attacker can conduct an attack via $Z=U'^{T}XV'^{T}$, which means one can use $f(Z)$ to recover the input data $Z$. 
% This means the information of data $Z$ is fully encoded in the neural network.

 

Second,  the sparsity of the learned matrix ($U$,$V$) or vectors ($a'$,$b'$) may also vary, indicating different amounts of information needed and leading to different complexity of the task. For example, in sentiment classification, what the network actually learns depends on the sparsity of $b'$.  If $b'$ is sparse, it means that we can simply pick several words from the sequence and determine the class.



% In this subsection, we delve into understanding why different fine-tuning tasks exhibit varied memorization behaviors. The intrinsic properties of language tasks offer some insights. We can observe that for low-memorization tasks like sentiment analysis or extractive QA, models often require only a handful of words or sentences to make decisions (such as identifying words that convey sentiment or the answer to a question). In contrast, high-memorization tasks like summarization and dialogue necessitate comprehensive context understanding.

% \textbf{In essence, a task's memorization behavior is likely influenced by its information requirements}. For instance, tasks like summarization and dialogue necessitate the internalization of nearly all salient features of the input text, resulting in more input knowledge being encoded in the model's parameters during training. This extensive learning process likely enables the model to incidentally memorize more specifics of the training sentences. Conversely, for tasks such as sentiment classification and extractive Question Answering (QA), the model only needs to identify a few key semantic features. This leads to less knowledge being encoded in the parameters and consequently reduces the model's tendency to memorize inconsequential patterns. To give a more detailed analysis, we provide a conceptual discussion based on a sparse decoding model, which is a popular model for modeling text and vision data,~\citep{arora2015latent,arora2018linear,olshausen1997sparse,olshausen2004sparse} to show the correlation between stored parametric knowledge and required input features in Appendix \ref{Analyse}.
%\pf{Does this overclaim(greater parameter updating)? If this comes from the following analysis, I suggest leaving these explanations as implications after analysis and only providing simple intuition here.}. 


\vspace{-0.2cm}
\subsection{Attention Distributions}
\vspace{-0.2cm}
{While the memorization disparity is possibly related to the task-specific information, the attention distribution in the transformer may also capture the contribution of each token's information to completing the task. In this section, we study whether the attention scores can be viewed as an indicator of the memorization ability of the task. 

For the fine-tuned models in Table \ref{tab:self finetune}, we generate their attention score heatmaps\footnote{We present the heatmap of the translation task in Appendix\ref{attention_translation}.}. Figure \ref{fig:heatmap} shows the distribution of the attention scores of the last decoder-encoder attention block in each model. Different layers of decoding-encoding attention scores are also visualized in Appendix ~\ref{attention_layers}\footnote{Across various layers, consistent patterns emerge in the encoder-decoder attention mechanism. Owing to this uniformity, we primarily report on the final layer in Figure \ref{fig:heatmap}, which is closest to the output.}. Our focus on encoder-decoder attention layers is their ability to capture the information across input features for each output.
% (how much information is needed to generate this output token)
% Unlike encoder and decoder attention layers which track internal attention within inputs and outputs, respectively. 
% It thus aligns better with our research objectives. 
We also theoretically discuss the correlation between attention score maps and task information needs in Appendix~\ref{Analyse}.

% This choice is based on the hypothesis that memorization behavior differences are linked to the specific information requirements for task completion. Encoder-decoder attention scores are thus more aligned with our.


In Figure \ref{fig:heatmap}, the horizontal axis represents input tokens, and the vertical axis indicates output tokens. The brightness represents the averaged multi-head decoder-encoder attention scores between input-output token pairs. Each horizontal line shows the attention score distribution of an output token across input tokens. We visualize the attention heatmaps of a single data sample and the average attention heatmaps of 10 random samples, padded to the longest length of the batch and truncated to a maximum of 512 tokens. More attention maps of different samples are visualized in Appendix \ref{sample attention}.

The heatmaps shown in Figure \ref{fig:heatmap} reveal clear differences in attention patterns among tasks. For high-memorization tasks (i.e., summarization and dialog), attention scores are more evenly distributed across input tokens. In contrast, for low-memorization tasks (e.g., sentiment classification and extractive QA), the attention is concentrated on a few positions while almost zero for other positions. The observed patterns suggest that the information needed to successfully complete each task varies. The attention score distribution for summarization and dialog implies that models must extract every detail from the input, increasing the possibility of memorization. Concentrated scores for sentiment classification and extractive QA indicate that only key information is required, reducing the tendency to memorize the fine-tuning data. 

% We also visualize different layers of decoding-encoding attention scores in Appendix \ref{attention_layers}.  We observed consistent patterns across various layers of the encoder-decoder attention mechanism, with high memorization tasks showing dense attention and low memorization tasks focusing attention on fewer positions. Due to this consistency, we report primarily on the final layer in Figure \ref{fig:heatmap}, closest to the output, for clarity and relevance.

In Appendix \ref{attention_t5base}, we present the attention scores of the T5-base model. Our findings also align with the fine-tuned model, and we observe a pattern of high memorization-intensive attention for complex tasks and concentrated attention for simpler ones. 
This aligns with our intuition that the attention pattern is a fundamental characteristic of the task. 
% For example, in summarization tasks, detailed attention to input is essential for task completion. Consequently, 
% training on such 'high memory' tasks leads to greater data retention by the model. This insight has practical implications. 
%
To utilize the attention score as a tool to predict memorization, prior to fine-tuning a model for a specific task, developers can assess attention patterns. The assessment can be done using the pre-trained model. It helps predict the memorization during fine-tuning.
% , offering a predictive measure for model behavior in specific tasks.
% \subsection{Corrective Attention Dispersion}

% \begin{figure}[ht]
%   \centering
%   \includegraphics[width=0.4\textwidth]{pic/CAD.pdf}
%   \caption{CAD vs Memo Ratio}
%   \vspace{-0.5cm}
%   \label{fig:attention score}
% \end{figure}

% Based on this observation, we propose Corrective Attention Dispersion (CAD) to quantify the dispersion degree of the attention score by analogizing the entropy of discrete probabilistic distributions. For probabilistic distribution, entropy can measure the uncertainty of the random variable and describe its dispersion:
% \begin{equation*}
% E_{\text{entropy}}(\mathcal{X})=\sum_{i=1}^{N}-p_i \log \left(p_i\right),
% \end{equation*}
% where $N$ is the number of states of random variable $\mathcal{X}$. If the distribution is dispersed, the entropy will be low, which means the states of $\mathcal{X}$ are more uncertain. For the dispersion of attention score, we use CAD as an analogy of entropy. Each horizontal line in the heatmap is a distribution of attention scores on input tokens. Each input token is seen as a state, while the attention score between an input token and the output token of that line is its probability. In this way, CAD can measure the dispersion degree of the attention score. The length of each heatmap to obtain a one-dimensional discrete distribution is equal to the input length $l_{t}$. 
% However, the number of states will influence the entropy. For example, the entropy of $n$-state discrete uniform distribution is $\log n$. This is because more states mean more uncertainty. The number of input tokens is usually different. Thus, we propose CAD to align the entropy between the distributions of samples with different numbers of input tokens:
% \begin{equation*}
% E_{\text{CAD}}(\mathcal{A})=\sum_{i=1}^l-\left[\frac{a_i}{r} \log \left(\frac{a_i}{r}\right)\right] r = \sum_{i=1}^l-{a_i} \log \left(\frac{a_i}{r}\right),
% \end{equation*}

% where  $r = L/l_t$, $L$ is the corrected number of input tokens, $l_t$ is the original number of input tokens, $a_i$ is the attention score at position i. CAD can be seen as a correction to split or pad the attention score into $r$ new tokens and align the length of input tokens to the same $L$ for all the samples from different tasks.

% In this work, we use $L=10$ and report the CAD of average attention heatmaps in Figure \ref{fig:attention score}. From the figure, we note that the value of CAD aligns well with the observed memorization behavior. The high CAD of the attention of summarization and dialog indicates they have more dense attention distributions and their dependence on all input details. Whereas the lower CAD of sentiment and QA suggests their concentration on only key semantic features. 



% \subsection{Theoretical Analyse}


% To understand why attention scores can be used as an indicator for memorization, we provide some theoretical intuition on the relation between attention scores and information needed for the task. We first use the classification task to explain the relation between attention scores and information density. Then we extend the intuition to discuss complex tasks.
% \paragraph{Attention score and memorization in classification} 

% We still consider the sparse coding model mentioned in Eq.\ref{eq:Z=UXV}, and continue to use the notations of Section \ref{sec:sparse_coding}. We use the classification task for simplicity.  As mentioned, for classification tasks, the best solution of Eq.\ref{cls_loss} is $f_{cls}(Z)=(a')^{\top}Zb'$, where $a^{\top}U^{\top}:=a'$ and $V^{\top}b:=b'$.
% In classification, whether a task is more complex or not depends on the sparsity of $b'$, and we justify that the\textbf{ sparsity of $b'$ directly affects the attention score pattern. } We then mathematically define the neural network architecture. To ease the derivation, we consider
% \begin{equation}
%     f(Z) = W^VZ \cdot\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right),
% \end{equation}
% with $W^V, W^K, W^Q$ all in $\mathbb{R}^{d\times d}$. The softmax operation is conducted column-wise. Since the output $f(Z)$ is a matrix rather than a scalar, for the classification task, we further multiply two vectors on the two sides of $f(Z)$ to get output scalar $y'$, i.e., $v_1^{\top}f(Z)v_2\in\mathbb{R}$, and $v_1$ and $v_2$ can be either trainable or arbitrary.  As mentioned in Section \ref{sec:sparse_coding}, the target of the classification task can be represented as $y=a^{\top}Xb$. The loss term of Eq.\ref{cls_loss} can then be written as: 
% \begin{equation}
%     l(v_1^{\top}f(Z)v_2,a^{\top}Xb).
%     \label{eq:l_f_cls_2}
% \end{equation}
% Aligning the neural network output $v_1^{\top}f(Z)v_2$ with $a^{\top}Xb$, it is easy to see that to better reduce the loss value, we need $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)v_2\in\mathbb{R}^D$ better aligned with $b'$. As a result, when measuring the effect of the input $Z$ on $v_1^{\top}f(Z)v_2\in\mathbb{R}$, e.g., Figure \ref{fig:sentiment-heatmap}, the weighted pattern $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)v_2$  has a similar sparsity as $b'$. Recall that $b'$ is the task-specific vector the model needs to learn, thus the analysis above suggests that \textbf{the attention score\footnote{Note that the attention matrix $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)$ itself is $\mathbb{R}^{D\times D}$ and is not aggregated for the output value, and $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)v_2$ is the final aggregated attention.} has a similar sparsity pattern as the sparsity of the information the model needs to learn. }
% \paragraph{Complex tasks} 


% For more complex tasks, the simplistic single-layer single-head attention analysis as the above is not enough to handle it,  and we need to use a larger architecture. Intuitively, with more features to learn in the task, the architecture will be more likely to memorize each feature comprehensively. We identify two key drivers of this behavior. First, each output token relies on information distributed across multiple input tokens. As shown in Figure \ref{fig:summary-heatmap}, each row has multiple high attention scores across different input tokens. Second, each output token often exhibits selectivity for a different subset of input tokens, leading to divergence in attention distributions across rows in Figure \ref{fig:summary-heatmap}. To conclude, these two factors may result in dense heatmap patterns compared to the concentrated heatmaps of simpler tasks.



% \section{Theoretical understandings}
% \boldsymbol{Q}: Why memorization can be different among task-specific fine-tuning models.

% Consider the following sparse model. Assume $\boldsymbol{X}=\left(\boldsymbol{x}_1, \boldsymbol{x}_2, \cdots, \boldsymbol{x}_d\right)$ denote hidden features for the observed data $\boldsymbol{Z}$. We further assume that
% $$
% \begin{aligned}
%     & P(\boldsymbol{x}_i\ne 0)=p~~\text{(Sparsity of true features)}\\
%     & \boldsymbol{Z}=\boldsymbol{M}\boldsymbol{X}+\boldsymbol{\xi}~~\text{(Generation of observed data)}
% \end{aligned}
% $$
% where $\boldsymbol{M}$ is an orthogonal matrix, i.e. $\boldsymbol{M}^T\boldsymbol{M}=I$, and $E\boldsymbol{\xi}=0$ denote the noise.

% {\color{blue} The sparse model has been used in many literature. (Please follow ``Sparse coding (data) model" in \url{https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9719702} and add some references)}

% Note that the difference between tasks can be represented by the formulation of label $\boldsymbol{y}$. For a given input $\boldsymbol{Z}$, we can define the label as $\boldsymbol{y}=f(\boldsymbol{Z})$. Consider two common tasks, summarization and classification, as examples. {\color{blue} Please have one-two sentences to describe why the summary task has the particular output format, e.g., it is like a rephrase of all the key information of the input xxx.}
% Therefore, we have the following formulations,
% $$
% \begin{aligned}
%     &f_{sum}(\boldsymbol{Z})=\boldsymbol{X}\\
%     &f_{cls}(\boldsymbol{Z})=\beta_X^T\boldsymbol{X}
% \end{aligned}
% $$
% Consider the linear model, i.e. $\hat{\boldsymbol{y}}=\beta^T_{Z}Z$. Thus the loss function for fine-tuning can be defined as 
% $
% \|\hat{\boldsymbol{y}}-\boldsymbol{y}\|^2
% $
% Now we discuss two tasks separately.
% \begin{itemize}
%     \item Summarization. The loss function is $\|\beta^T_{Z}Z-X\|^2=\|\beta^T_{Z}Z-M^TZ\|^2$. Thus $\hat{\beta}_{Z,sum}^*=M$.
%     \item Classification. The loss function is $\|\beta_Z^TZ-\beta_X^TX\|^2=\|\beta_Z^TZ-\beta_X^TMZ\|^2$. Thus $\hat{\beta}_{Z,cls}^*=\beta_X^TM$
% \end{itemize}
% where $\hat{\beta}_{Z,\cdot}^*$ denote the minimizers of the loss functions. Note that $\beta_X^TM$ only contains part of information of $M$(if need more details, consider the rank and sparsity of $\beta_X$). Thus the summarization model stores more information about training data than the classification model.


% {\color{blue} Please write a paragraph to intuitively connect the above results the observation in the attention score. }

% \section{Intuitions}\label{sec:intuition}

% To explain (1) why the memorization differs among tasks, and (2) why the distribution of attention score differs among tasks and is associated with the memorization, we first define our data generation model.

% Denote $\boldsymbol{X}$ as the original true information we have, and the observed data is $\boldsymbol{Z}$, where
% \begin{equation*}
%     \boldsymbol{Z}=\boldsymbol{U}\boldsymbol{X}\boldsymbol{V} + \boldsymbol{\xi}
% \end{equation*}
% where $\boldsymbol{U}$ and $\boldsymbol{V}$ are unitary matrices, and $\boldsymbol{\xi}$ is a white noise term. The dimensions of $\boldsymbol{Z}, \boldsymbol{X}, \boldsymbol{\xi}$ are all 
% $\mathbb{R}^{d\times D}$ where $D$ is the sequence length and $d$ is the length of the embedding. For the two unitary matrices, $\boldsymbol{U}\in\mathbb{R}^{d\times d}$ and $\boldsymbol{V}\in\mathbb{R}^{D\times D}$.

% \paragraph{Difference among tasks} We argue that the memorization varies depending on the task-specific features. For example, in the simplest task, e.g., semantic classification and comprehension, one may assume that the response $y$ is related to $a^{\top}\boldsymbol{X}b$ for some vectors $a\in\mathbb{R}^d$ and $b\in\mathbb{R}^D$, i.e., there is one task-specific feature in the form of $ab^{\top}$. We want to minimize the following loss function:
% \begin{equation*}
%     l(f_{cls}(\boldsymbol{Z}),a^{\top}\boldsymbol{X}b)
% \end{equation*}
% for some loss function $l$. For loss functions such as square loss, the best solution of $f_{cls}$ is $f_{cls}(\boldsymbol{Z})=(a')^{\top}\boldsymbol{Z}b'$, where $a^{\top}\boldsymbol{U}^{\top}:=a'$ and $\boldsymbol{V}^{\top}b:=b'$. In addition, the complexity of the task with only one task-specific feature further depends on the sparsity of $b'$, i.e., whether we can simply pick several tokens from the sequence and determine the class.

% On the other hand, for tasks with more task-specific features, e.g., summary, our aim is to figure out a model so that
% \begin{equation*}
%     \|f_{sum}(\boldsymbol{Z})-\boldsymbol{X}\|^2
% \end{equation*}
% can be minimized. In the above task, all features of $\boldsymbol{X}$ matter. To recover $\boldsymbol{X}$ from $\boldsymbol{Z}$, we need the information of $\boldsymbol{U}$ and $\boldsymbol{V}$, i.e., the best solution is $f_{sum}=\boldsymbol{U}^{\top}\boldsymbol{Z}\boldsymbol{V}^{\top}$.

% Comparing the above two tasks, with a different number of task-specific features, the number of parameters to be learned from the fine-tuning is different. For the classification task $f_{cls}$, there are only $d+D$ unknown parameters. For the task $f_{sum}$, there are $O(d^2+D^2)$ parameters to be learned. As a result, when training in any over-parametrized neural network, e.g., MLP, Transformers, or other architectures, intuitively, memorization will be more likely to happen in the tasks with more task-specific features and unknown parameters.

% \paragraph{Attention score and memorization in classification} 
% We consider classification tasks with different sparsity of $\textbf{V}^{\top}b$. As mentioned in the above, there is only one task-specific feature in classification, and whether a task is more complex or not depends on the sparsity of $\textbf{V}^{\top}b$. 

% We first mathematically define the format of the data and the neural network architecture. When feeding a training data point, we consider the following format
% \begin{equation*}
%     \boldsymbol{E} = 
%         \boldsymbol{Z}\in\mathbb{R}^{d\times D}.
% \end{equation*}


% In terms of the transformer architecture, to ease the derivation, we consider
% \begin{equation*}
%     f(\boldsymbol{E}) = \boldsymbol{W}^V\boldsymbol{E} \cdot\text{softmax}\left( (\boldsymbol{W}^K\boldsymbol{E})^{\top} (\boldsymbol{W}^Q\boldsymbol{E})\right),
% \end{equation*}
% with $\boldsymbol{W}^V, \boldsymbol{W}^K, \boldsymbol{W}^Q$ are all in $\mathbb{R}^{d\times d}$. The softmax operation is conducted columnwise.

% % Assume there are $n$ samples in the fine-tuning stage, and $\mathbb{E}_n$ is the empirical expectation. Below is the analysis of different tasks:
% % \begin{itemize}
% %     \item For the summarization task, we aim to minimize
% %     \begin{equation*}
% %         \mathbb{E}_n \|f(\boldsymbol{E})_{(d+1):(2d),:}-\boldsymbol{X}\|_F^2.
% %     \end{equation*}
% %     \item For the classification task, we aim to minimize
% %     \begin{equation*}
% %         \mathbb{E}_n (f(\boldsymbol{E})_{d+1,D}-a^{\top}\boldsymbol{X}b)^2.
% %     \end{equation*}   
% % \end{itemize}
% Since the output $f(\boldsymbol{E})$ is a matrix rather than a scalar, for classification task, we further multiply two vectors on the two sides of $f(\boldsymbol{E})$, i.e., $v_1^{\top}f(\boldsymbol{E})v_2\in\mathbb{R}$. 

% Aligning the neural network output $v_1^{\top}f(\boldsymbol{E})v_2$ with $a^{\top}\boldsymbol{X}b$, it is easy to see that to better reducing the loss value, we need $\text{softmax}\left( (\boldsymbol{W}^K\boldsymbol{E})^{\top} (\boldsymbol{W}^Q\boldsymbol{E})\right)v_2$ better aligned with $b'=\boldsymbol{V}^{\top}b$. As a result, when measuring the effect of the input $\boldsymbol{E}$ on $v_1^{\top}f(\boldsymbol{E})v_2\in\mathbb{R}$, e.g., Figure \ref{fig:heatmap}, the weighted pattern pattern $\text{softmax}\left( (\boldsymbol{W}^K\boldsymbol{E})^{\top} (\boldsymbol{W}^Q\boldsymbol{E})\right)v_2$  is sparse. %That is, the attention score for some particular word is very high in simple tasks and the pattern of $\text{softmax}\left( (\boldsymbol{W}^K\boldsymbol{E})^{\top} (\boldsymbol{W}^Q\boldsymbol{E})\right)_{:,D}$ is sparse.

% % If the last column of the attention score, $\text{softmax}\left( (\boldsymbol{W}^K\boldsymbol{E})^{\top} (\boldsymbol{W}^Q\boldsymbol{E})\right)_{:,D}$ is trained to be sparse, then the other columns, $\text{softmax}\left( (\boldsymbol{W}^K\boldsymbol{E})^{\top} (\boldsymbol{W}^Q\boldsymbol{E})\right)_{:,1:(D-1)}$, will present a sparse pattern as well because they share the same $\boldsymbol{W}^K$ and $\boldsymbol{W}^Q$.

% % The simple tasks are less likely to memorize the whole input data. In these tasks, we usually pick some key words which share similar meanings to determine the label. In this case, the  the non-zero elements in $b'$ 

% % The main difference between summarization task and classification task is on the attention score from $\text{softmax}\left( (\boldsymbol{W}^K_j\boldsymbol{E})^{\top} (\boldsymbol{W}^Q_j\boldsymbol{E})\right)$. {\color{red} To discuss: what is attention score? The ``attn weights" in \url{https://github.com/huggingface/transformers/blob/v4.33.2/src/transformers/models/gpt2/modeling_gpt2.py#L123}?  How to make prediction?}


% \paragraph{More task-specific features}
% For more complex tasks, a simple one-layer GPT analysis as the above is not enough to handle it, and one need to use a larger architecture. In addition, the architecture will memorize every feature, e.g., summarization in Figure \ref{fig:heatmap}.

% In classification, assume $h=d$, i.e., $\boldsymbol{W}^K_j$ is a vector. For the new prediction token, we have its product with an input token embedding is
% \begin{eqnarray}
% \left(\boldsymbol{W}^K_j\boldsymbol{E}_{:,i}\right)^{\top} \left(\boldsymbol{W}^Q_j\begin{pmatrix}
%         \boldsymbol{0}\\ a^{\top}\boldsymbol{X}b \\ \boldsymbol{0}
%     \end{pmatrix}\right).\label{eqn:prod}
% \end{eqnarray}
% For some constant $c$, we construct a vector $(a^{\top}\boldsymbol{U}^{\top}, c,\boldsymbol{0}_{d-1})$ so that it relates the input and the target output.
% \begin{itemize}
%     \item If  $\boldsymbol{W}_j^K=(a^{\top}\boldsymbol{U}^{\top}, c,\boldsymbol{0}_{d-1})$, then the product (\ref{eqn:prod}) becomes
% \begin{eqnarray*}
%     c\boldsymbol{X}_{:,i}^{\top}aa^{\top}\boldsymbol{X}b,
% \end{eqnarray*}
% i.e., the product (\ref{eqn:prod}) for all input tokens becomes
% \begin{eqnarray*}
%     c\boldsymbol{X}^{\top}aa^{\top}\boldsymbol{X}b.
% \end{eqnarray*}
% \item On the other hand, if the vector $(a^{\top}\boldsymbol{U}^{\top}, c,\boldsymbol{0}_{d-1})$ is only a negligible component of $\boldsymbol{W}_j^Q$, then the product (\ref{eqn:prod}) becomes
% \begin{eqnarray*}
%     c\boldsymbol{X}^{\top}va^{\top}\boldsymbol{X}b
% \end{eqnarray*}
% for some vector $v$ with $v^{\top}a/\|v\|\|a\|\approx 0$.
% \end{itemize}

% If the transformer is trained in a diverse of tasks and data sets, what is learned by each head are almost perpendicular with each other, i.e., only one or two head learns $(a^{\top}\boldsymbol{U}^{\top}, c,\boldsymbol{0}_{d-1})$. Therefore, considering the randomness of $\boldsymbol{X}$, for the attention score
% \begin{eqnarray*}
%     \text{softmax}\left( \left(\boldsymbol{W}^K_j\left(\boldsymbol{E},\begin{pmatrix}
%         \boldsymbol{0}\\ a^{\top}\boldsymbol{X}b \\ \boldsymbol{0}
%     \end{pmatrix} \right)\right)^{\top} \left(\boldsymbol{W}^Q_j\begin{pmatrix}
%         \boldsymbol{0}\\ a^{\top}\boldsymbol{X}b \\ \boldsymbol{0}
%     \end{pmatrix}\right)\right),
% \end{eqnarray*}
% a large amount will be around $1/D$ since they are random, and some will be spread a lot with various values/have some peaks if $\boldsymbol{W}_j^Q$ is highly related to $a^{\top}\boldsymbol{U}^{\top}$. In addition, if $\boldsymbol{W}_j^Q$ is almost independent to $a^{\top}\boldsymbol{U}^{\top}$, it will also not be updated during the fine-tuning, i.e., little memorization.

% In terms of summarization task, since every feature is important, the distribution of the attention score is flat, and every attention head will be tuned with a lot of memorization.

% Finally, to compare single-head attention and multi-head attention, in single-head attention, the component $(a^{\top}\boldsymbol{U}^{\top}, c,\boldsymbol{0}_{d-1})$ is always a small proportion because the attention learns every feature. As a result, it is difficult to tune the single-head attention and the performance is poor.



\vspace{-0.3cm}
\section{Conclusions}
\label{Conclusion}
\vspace{-0.3cm}
In this paper, we conduct extensive experiments to investigate the memorization behavior of fine-tuned LMs among various tasks. Utilizing an automatic detection pipeline, we are able to evaluate the memorization in numerous tasks and datasets. In addition, we provide understandings of the memorization disparity among tasks based on a sparse coding theory. Our analysis reveals a strong correlation between attention scores and memorization.  
\section{Limitations}
Our study primarily utilizes white-box models from the T5 family for evaluation, such as T5-small, T5-base, T5-large, and T5-XL. Black-box models like ChatGPT were not included due to our inability to fine-tune them directly. Additionally, there is scope to expand the variety of datasets and tasks in future research. 

Theoretically, we employ sparse coding theory to articulate our hypothesis that the observed memorization differences across tasks stem from their varying informational requirements ("True features" in sparse coding theory). We draw on recent theoretical developments\citep{zhang2023trained, deora2023optimization,wu2023many} that apply sparse coding in NLP and leverage these theories to support our reasoning. Nevertheless, fully extending this theory to precisely account for the complexities of non-linear or large-scale models remains an unresolved challenge in the theoretical community.

While our study addresses the memorization behavior of models fine-tuned on single tasks, the investigation into models fine-tuned on multiple tasks is still unexplored and presents an opportunity for future research.

\section{Ethics Statement}
The phenomenon of fine-tuned memorization in language models has notable social implications. Firstly, it raises privacy concerns, especially in tasks where sensitive information, like personal dialogue, is involved. The ability of these models to retain and potentially disclose private data necessitates corresponding data protection measures. Secondly, from a utility perspective, while memorization enhances performance in certain tasks, it also underscores the need for balancing accuracy with responsible data handling in AI systems.
\clearpage
% \makeatletter
% \@starttoc{toc}
% \makeatothe

\appendix

% \addappheadtotoc
% \appendixpage
% \begin{appendices}
% \tableofcontents


\section{Theoretical Analyze}
\label{Analyse}
To understand why attention scores can be used as an indicator for memorization, we provide some theoretical intuition on the relation between attention scores and information needed for the task. We first use the classification task to explain the relation between attention scores and information density. Then we extend the intuition to discuss complex tasks.
\paragraph{Attention score and memorization in classification} 

We still consider the sparse coding model mentioned in Eq.\ref{eq:Z=UXV}, and continue to use the notations of Section \ref{sec:sparse_coding}. We use the classification task for simplicity.  As mentioned, for classification tasks, the best solution of Eq.\ref{cls_loss} is $f_{cls}(Z)=(a')^{\top}Zb'$, where $a^{\top}U^{\top}:=a'$ and $V^{\top}b:=b'$.
In classification, whether a task is more complex or not depends on the sparsity of $b'$, and we justify that the\textbf{ sparsity of $b'$ directly affects the attention score pattern. } We then mathematically define the neural network architecture. To ease the derivation, we consider
\begin{equation}
    f(Z) = W^VZ \cdot\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right),
\end{equation}
with $W^V, W^K, W^Q$ all in $\mathbb{R}^{d\times d}$. The softmax operation is conducted column-wise. Since the output $f(Z)$ is a matrix rather than a scalar, for the classification task, we further multiply two vectors on the two sides of $f(Z)$ to get output scalar $y'$, i.e., $v_1^{\top}f(Z)v_2\in\mathbb{R}$, and $v_1$ and $v_2$ can be either trainable or arbitrary.  As mentioned in Section \ref{sec:sparse_coding}, the target of the classification task can be represented as $y=a^{\top}Xb$. The loss term of Eq.\ref{cls_loss} can then be written as: 
\begin{equation}
    l(v_1^{\top}f(Z)v_2,a^{\top}Xb).
    \label{eq:l_f_cls_2}
\end{equation}
Aligning the neural network output $v_1^{\top}f(Z)v_2$ with $a^{\top}Xb$, it is easy to see that to better reduce the loss value, we need $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)v_2\in\mathbb{R}^D$ better aligned with $b'$. As a result, when measuring the effect of the input $Z$ on $v_1^{\top}f(Z)v_2\in\mathbb{R}$, e.g., Figure \ref{fig:sentiment-heatmap}, the weighted pattern $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)v_2$  has a similar sparsity as $b'$. Recall that $b'$ is the task-specific vector the model needs to learn, thus the analysis above suggests that \textbf{the attention score\footnote{Note that the attention matrix $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)$ itself is $\mathbb{R}^{D\times D}$ and is not aggregated for the output value, and $\text{softmax}\left( (W^KZ)^{\top} (W^QZ)\right)v_2$ is the final aggregated attention.} has a similar sparsity pattern as the sparsity of the information the model needs to learn. }
\paragraph{Complex tasks} 


For more complex tasks, the simplistic single-layer single-head attention analysis as the above is not enough to handle it,  and we need to use a larger architecture. Intuitively, with more features to learn in the task, the architecture will be more likely to memorize each feature comprehensively. We identify two key drivers of this behavior. First, each output token relies on information distributed across multiple input tokens. As shown in Figure \ref{fig:summary-heatmap}, each row has multiple high attention scores across different input tokens. Second, each output token often exhibits selectivity for a different subset of input tokens, leading to divergence in attention distributions across rows in Figure \ref{fig:summary-heatmap}. To conclude, these two factors may result in dense heatmap patterns compared to the concentrated heatmaps of simpler tasks. 


\section{Details of Evaluation Pipeline}
\label{Details and pipeline}
\subsection{Evaluation Process}
\label{Pipeline}
In this section, we provide a detailed overview of the evaluation tools and methodologies employed. The evaluation framework comprises three distinct processes: prompting, searching, and detection. The prompting method has been extensively utilized in prior research on pre-trained memorization\citep{carlini2022quantifying}, and the search and detection methods, based on Elasticsearch and PAN-2014 detection tools, were previously adopted by \citet{lee2023language}.  

\paragraph{Prompting.} In the evaluation phase, the input data \( x_{i} \) is segmented into two parts: a prefix \( p_{i} \) and a suffix \( s_{i} \). We input the prefixes \( \{p_i\}_{i=1}^n \) into the model without any task-specific instructions to obtain \( f(p_i) \). For our experiments, we select \( n=10,000 \) samples from each dataset. The standard prefix length \( k \) is set to 50 tokens. However, for tasks with input sentences shorter than 50 tokens, such as translation tasks, a reduced prefix length of 15 tokens is used. The testing procedure is consistent across both base and fine-tuned models. This involves inputting the prefix of the fine-tuning data into the models and comparing the suffixes. Notably, for the base models, the prefix of the fine-tuning data is still used in the test, regardless of whether the model has been fine-tuned on that specific dataset or not. This approach helps determine if the model retains the data, suggesting its presence in the pre-training set.

% while the search and detection methods, based on Elasticsearch and PAN-2014 detection tools, was adopted by\cite{lee2023language}

\paragraph{Searching.}In the search phase of our experiment, we employ Elasticsearch, a distributed, RESTful search and analytics engine based on the open-source Lucene library. Elasticsearch leverages the Okapi-BM25 algorithm, a widely-used bag-of-words ranking function, allowing for efficient storage, searching, and near real-time analysis of large data volumes. We upload all suffixes \( \{s_i\}_{i=1}^n \) into Elasticsearch and use the set \( \{f(p_i)\}_{i=1}^n \) as our query documents. For our analysis, we set \( K = 10 \), indicating that only the top-10 most relevant candidates for each query are retrieved for subsequent memorization detection.
 


% We typically choose the prefix-length $k$ as 50 tokens. (For some datasets whose input sentences are shorter than 50 tokens, we use 15 tokens instead. )

\paragraph{Detection.} After we get suspicious sentence pairs $f(p_i)$ and $s_j$, we input them to a publicly available PAN2014 plagiarism detection tool $D$ to see if $D(f(p_i),s_j)= \text{True}$. In general, the detection tool will detect the presence of plagiarised word piece pairs \( (d_i, d_j) \) a, where \( d_i \) and \( d_j \) are word pieces from \( f(p_i) \) and \( s_j \), respectively and then compare $(d_i,d_j)$ to identify the category of memorization.  Here we set the minimal match threshold as at least 50 characters(approximately 15 tokens). 


The detailed process includes (1) preprocessing text; (2) identifying obfuscation types; (3) seeding to find candidate pairs via sentence similarity; (4) extension by clustering similar fragments; and (5) filtering out overlaps. They transform sentences into TF-IDF vectors and calculate similarity using dice and cosine measures, with adaptive parameters selected by testing on the obfuscation corpus.  Here we set the minimal match threshold as at least 50 characters(approximately 20 tokens). We also utilize additional validation steps after retrieving paraphrased text segments as \cite{sanchez2015adaptive}. The post-processing involves chunking segments into sentences using NLTK's tokenizer, then applying a RoBERTa-based paraphrase identification model and Named Entity Recognition (NER) on the sentences. Specifically, we check sentence pairs - if any pair has a paraphrase detection probability score between 0.5 and 0.99, we accept it as high-confidence paraphrasing, otherwise, we identify it as low-confidence paraphrasing.
\subsection{Memorization Types}

\paragraph{Difference between memorization types.}
Here we will distinguish 3 types of memorization. First, verbatim memorization means exact copies of words or phrases without transformation. In the cases of paraphrase and idea memorization, the output is not identical to the original text but shares similar meanings. While paraphrase plagiarism focuses on sentence-to-sentence transformation, idea plagiarism involves summarizing the key points of a larger text segment into a more condensed form (or expanding it). In Table~\ref{Memorization Types}, we give a simple example to differentiate the difference between 3 types of memorization conceptually.

In practice of the PAN2014-detection, It starts by identifying closely matched short document fragments (referred to as 'seeds') and then expands these seeds into longer text segments. This is achieved by clustering these fragments based on their separation and employing a 'maxgap' threshold parameter to form coherent clusters.They experimentally find out the most suitable threshold for different plagiarism datasets so that those parameters could be used for the detection of a specific type of memorization. In other words, each memorization case will be \textbf{counted only once} and there will not be overlapping across different categories.

\paragraph{Distinguishing idea memorization from summarization.}
It's important to differentiate idea memorization—\textit{condensing key points of a larger text segment}—from summarization tasks. Note that in our approach, only the prefix (initial tokens) of a text is input to the model. Summarization means the model summarizing this prefix without revealing the remaining suffix of the text. In contrast, idea memorization involves the model generating information about the suffix. In our experiments, we assess similarity by comparing the generated text $f(p)$ with the suffix $s$, rather than with the prefix $p$ or the entire text $x$. In table \ref{Idea vs summ}, we use a simple example to illustrate the difference.


\begin{table*}[ht]
\caption{Difference between 3 Memorization Types}
\label{Memorization Types}
\centering
\begin{tabular}{p{15cm}}

\hline
\textbf{Examples of 3 Memorization Types } \\
\hline
\textbf{Verbatim:}\\
Text A: My name is Jack\\
Text B: My name is Jack\\

\textbf{Paraphrase:} \\
Text A: My name is Jack\\
Text B: Jack is my name \\

\textbf{Idea plagiarism:}\\
Text A: A boy tell me in the class that his name is Jack\\
Text B: A boy is Jack\\
\hline
\end{tabular}

\label{tab: type difference}
\end{table*}

\begin{table*}[ht]
\caption{Difference between Idea Memorization and Summarization}
\label{Idea vs summ}
\centering
\begin{tabular}{p{15cm}}

\hline
\textbf{Idea Memorization vs Summarization } \\
\hline
\textbf{Training data:} I am not comfortable from the beginning of the month, I am 20 years old, height 51, height 51, weight 40kg. I have been pregnant for 6 months and I can not stop vomiting.
 \\

\textbf{Input:}  I am not comfortable from the begining of the month,I am 20 years\\
\textbf{Output:}  20 years old, height 5 1 \& weight 40kg(Memorization)\\
\textbf{Output:} 20 years old woman feels bad for a month.(Summarization)\\
\# Note: We only compare output with \textit{"height 51, height 51, weight 40kg. I have been pregnant for 6 months and I can not stop vomiting."} to identify memorization.\\
\hline
\end{tabular}

\label{tab:idea vs mem}
\end{table*}


\begin{table*}[htbp]
\centering
\caption{Memorization of fine-tuned\_T5 with various epochs.}
\label{tab: ablation_epochs}
\resizebox{0.5\textwidth}{!}{%
\begin{tabular}{c|c|c|c}
\toprule
Task & Dataset & Epochs & \begin{tabular}[c]{@{}c@{}} Total\\Mem Rate \end{tabular} \\ 
\midrule
\multirow{4}{*}{Dialog} & \multirow{4}{*}{HealthCareMagic} & 1 & 1.60\% \\
 &  & 3 & 3.30\% \\
 &  & 5 & 6.22\% \\
 &  & 10 & 8.27\% \\
\midrule
\multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & 1 & 0.78\% \\
 &  & 3 & 0.79\% \\
 &  & 5 & 0.80\% \\
 &  & 10 & 0.79\% \\
\midrule
\multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & 1 & 14.12\% \\
 &  & 3 & 14.32\% \\
 &  & 5 & 22.33\% \\
 &  & 10 & 22.32\% \\
\midrule
\multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & 1 & 0.08\% \\
 &  & 3 & 0.12\% \\
 &  & 5 & 0.10\% \\
 &  & 10 & 0.15\% \\
\bottomrule
\end{tabular}
}
\end{table*}


\begin{table*}[htbp]
\centering
\caption{Memorization of fine-tuned\_T5 with various sampling methods.}
\label{tab: t5_decoding}
\resizebox{\textwidth}{!}{%
\begin{tabular}{c|c|c|ccccc}
\toprule
Task & Dataset & Decoding& \begin{tabular}[c]{@{}c@{}} Total\\Mem Rate \end{tabular} & Verbatim  & Idea  & \begin{tabular}[c]{@{}c@{}}Paraphrase\\ ($P<0.5$)\end{tabular}  & \begin{tabular}[c]{@{}c@{}}Paraphrase\\ ($P>0.5$)\end{tabular}  \\ 
\midrule
\multirow{4}{*}{Dialog} & \multirow{4}{*}{HealthCareMagic} & Top-K & 5.76\% & 0.05\% & 0.38\% & 0.90\% & 4.43\% \\
 &  & Top-p & 7.26\% & 0.06\% & 0.48\% & 1.35\% & 5.37\% \\
 &  & Temp & 3.72\% & 0.02\% & 0.18\% & 0.58\% & 2.94\% \\
 &  & Greedy & 8.27\% & 0.02\% & 1.41\% & 1.75\% & 5.09\% \\
\midrule
\multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & Top-K & 1.02\% & 0.01\% & 0.13\% & 0.18\% & 0.70\% \\
 &  & Top-p & 1.08\% & 0.01\% & 0.12\% & 0.22\% & 0.73\% \\
 &  & Temp & 0.89\% & 0.01\% & 0.07\% & 0.19\% & 0.62\% \\
 &  & Greedy & 0.80\% & 0.04\% & 0.30\% & 0.17\% & 0.29\% \\
\midrule
\multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & Top-K & 10.80\% & 2.54\% & 0.34\% & 1.94\% & 5.98\% \\
 &  & Top-p & 13.57\% & 4.07\% & 0.54\% & 2.26\% & 6.70\% \\
 &  & Temp & 5.82\% & 1.28\% & 0.23\% & 0.83\% & 3.48\% \\
 &  & Greedy & 22.33\% & 4.23\% & 0.65\% & 6.23\% & 11.22\% \\
 \midrule
 \multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & Top-K & 0.15\% & 0.04\% & 0.00\% & 0.05\% & 0.06\% \\
 &  & Top-p & 0.15\% & 0.04\% & 0.00\% & 0.05\% & 0.06\% \\
 &  & Temp & 0.15\% & 0.04\% & 0.00\% & 0.05\% & 0.06\% \\
 &  & Gready & 0.15\% & 0.04\% & 0.00\% & 0.05\% & 0.06\% \\
\bottomrule
\end{tabular}
}
\end{table*}

\begin{table*}[htbp]
\centering
\caption{ Memorization of fine-tuned T5 with varying prefix lengths.}
\label{tab: t5_prefix_length}
\resizebox{\textwidth}{!}{%
\begin{tabular}{c|c|c|ccccc}
\toprule
Task & Dataset & \begin{tabular}[c]{@{}c@{}} Prefix\\length \end{tabular}& \begin{tabular}[c]{@{}c@{}} Total\\Mem Rate \end{tabular}& Verbatim & Idea & \begin{tabular}[c]{@{}c@{}} Paraphrase\\($p>0.5$) \end{tabular} & \begin{tabular}[c]{@{}c@{}} Paraphrase\\($p<0.5$)\end{tabular} \\ 
\midrule
\multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & 10 & 12.25\% & 1.74\% & 2.85\% & 0.88\% & 6.78\% \\
 &  & 30 & 20.68\% & 7.07\% & 1.41\% & 3.05\% & 9.15\% \\
 &  & 50 & 22.33\% & 4.23\% & 0.65\% & 6.23\% & 11.22\% \\
 &  & 100 & 29.66\% & 10.61\% & 0.79\% & 4.27\% & 13.99\% \\
\midrule
\multirow{3}{*}{Dialog} & \multirow{3}{*}{HealthCareMagic} & 10 & 6.28\% & 0.03\% & 1.94\% & 0.85\% & 3.46\% \\
 &  & 30 & 7.76\% & 0.04\% & 1.28\% & 1.72\% & 4.72\% \\
 &  & 50 & 8.27\% & 0.02\% & 1.41\% & 1.75\% & 5.09\% \\
\midrule
\multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & 10 & 1.37\% & 0.00\% & 1.12\% & 0.06\% & 0.19\% \\
 &  & 30 & 1.18\% & 0.01\% & 0.51\% & 0.15\% & 0.51\% \\
 &  & 50 & 0.80\% & 0.04\% & 0.30\% & 0.17\% & 0.29\% \\
 &  & 100 & 1.39\% & 0.05\% & 0.23\% & 0.33\% & 0.78\% \\
\midrule
\multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & 10 & 0.18\% & 0.05\% & 0.00\% & 0.06\% & 0.07\% \\
 &  & 30 & 0.14\% & 0.04\% & 0.00\% & 0.04\% & 0.06\% \\
 &  & 50 & 0.15\% & 0.04\% & 0.00\% & 0.05\% & 0.06\% \\
 &  & 100 & 0.13\% & 0.03\% & 0.01\% & 0.04\% & 0.05\% \\

\bottomrule
\end{tabular}
}
\end{table*}





\begin{table*}[ht]

\caption{Examples of training data from different tasks}
\label{training template}
\centering
\begin{tabular}{p{15cm}}

\hline
\textbf{Summarization} \\
\hline
\textbf{Prompt:} Please summarize the following paragraph: \\
\textbf{Input:}\\
...A fresh update on the U.S. employment situation for January hits the wires at 8:30 a.m. New York time offering one of the most important snapshots on how the economy fared during the previous month. Expectations are for 203,000 new jobs to be created, according to economists polled by Dow Jones Newswires, compared to 227,000 jobs added in February. The unemployment rate is expected to hold steady at 8.3\%.  ...\\
\textbf{Output:}\\
...The unemployment rate dropped to 8.2\% last month, but the economy only added 120,000 jobs, when 203,000 new jobs had been predicted, according to today's jobs report. \\
\textbf{Training Format:}\\
Training input = \textbf{Prompt} + \textbf{Input}, Training label =  \textbf{Output} \\
\hline
\textbf{Sentiment classification} \\
\hline
\textbf{Prompt:} \\
Please classify the sentiment of the following paragraph: \\
\textbf{Input:} \\
"Foxes" is a serious look at the consequences of growing up too fast in the 1980s. And unlike the teen sex comedies that overshadowed it (Porky's, Fast Times at Ridgement High), the movie holds up well against time...\\
\textbf{Output:}\\
Positive\\
\textbf{Training Format:} \\
Training input = \textbf{Prompt} + \textbf{Input},  Training label = \textbf{Output} \\
\hline
\textbf{Dialog} \\
\hline
\textbf{Instruction:} \\If you are a doctor, please answer the medical questions based on the patient's description.\\
\textbf{Input:} \\I woke up this morning feeling the whole room is spinning when i was sitting down. I went to the bathroom walking unsteadily, as i tried to focus i feel nauseous. I try to vomit but it wont come out.. After taking panadol and sleep for few hours, i still feel the same.. \\
\textbf{Output:} \\Hi, Thank you for posting your query. The most likely cause for your symptoms is benign paroxysmal positional vertigo (BPPV), a type of peripheral vertigo. In this condition, the most common symptom is dizziness or giddiness, which is made worse with movements. ...\\
\textbf{Training Format:} \\
Training input = \textbf{Instruction} + \textbf{Input},  Training label = \textbf{Output}\\
\hline
\textbf{Question and answering} \\
\hline
\textbf{Question:}\\
Who was the Norse leader?\\
\textbf{Input:} \\
... They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. ...\\
\textbf{Output:}\\
Rollo\\
\textbf{Training Format:} \\
Training input = \textbf{Question} + \textbf{Input},  Training label = \textbf{Output} \\
\hline
\end{tabular}

\label{tab: summ}
\end{table*}



\section{Disparate Memorization on GPT-Neo}
\label{gptneo-ft}

We also fine-tuned decoder-only GPT-Neo-125m models and also observed similar findings with T5-base, which suggests our findings are generalizable. The results are reported in Table \ref{tab:self fine-tune gptneo}. We can clearly observe that the memorization increase for summarization and dialog is much more significant than QA and sentiment classification. 

\begin{table*}[htbp]
\centering
\caption{Memorization rate of gpt-neo-125m fine-tuned on various tasks}
\label{tab:self fine-tune gptneo}
\resizebox{0.45\textwidth}{!}{
\begin{tabular}{@{}c|c|c|c@{}}
\toprule
Task & Dataset & Model & \begin{tabular}[c]{@{}c@{}} Total \\Mem Rate \end{tabular} \\
\midrule
\multirow{3}{*}{Summarization} & \multirow{3}{*}{Multi-news} & GPT-Neo & 25.2\% \\
 & & GPT-Neo-ft & 44.3\% \\
 & & Difference & $\uparrow$19.1\% \\
\midrule
\multirow{3}{*}{Dialog} & \multirow{3}{*}{chatdoctor} & GPT-Neo & 2.5\% \\
 & & GPT-Neo-ft & 7.8\% \\
 & & Difference & $\uparrow$5.3\% \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}} Sentiment \\ Classification \end{tabular}} & \multirow{3}{*}{imdb} & GPT-Neo & 3.9\% \\
 & & GPT-Neo-ft & 4.2\% \\
 & & Difference & $\uparrow$0.3\% \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}} Reading \\ Comprehension \end{tabular}} & \multirow{3}{*}{Squad\_v2} & GPT-Neo & 0.02\% \\
 & & GPT-Neo-ft & 0.04\% \\
 & & Difference & $\uparrow$0.02\% \\
\midrule
\multirow{3}{*}{Translation} & \multirow{3}{*}{wmt} & GPT-Neo & 0.00\% \\
 & & GPT-Neo-ft & 0.00\% \\
 & & Difference & - \\
\bottomrule
\end{tabular}}
\end{table*}




% We also utilize additional validation steps after retrieving paraphrased text segments as \citet{sanchez2015adaptive}. The post-processing involves chunking segments into sentences using NLTK's tokenizer, then applying a RoBERTa-based paraphrase identification model and Named Entity Recognition (NER) on the sentences. Specifically, we check sentence pairs - if any pair has a paraphrase detection probability score between 0.5 and 0.99, we accept it as high-confidence paraphrasing, otherwise, we identify it as low-confidence paraphrasing.


% \footnote{\href{https://pan.webis.de/clef14/pan14-web/text-alignment.html}


% The detailed process includes (1) preprocessing text; (2) identifying obfuscation types; (3) seeding to find candidate pairs via sentence similarity; (4) extension by clustering similar fragments; and (5) filtering out overlaps. They transform sentences into TF-IDF vectors and calculate similarity using dice and cosine measures, with adaptive. We also utilize additional validation steps after retrieving paraphrased text segments as \citet{sanchez2015adaptive}. The post-processing involves chunking segments into sentences using NLTK's tokenizer, then applying a RoBERTa-based paraphrase identification model and Named Entity Recognition (NER) on the sentences. Specifically, we check sentence pairs - if any pair has a paraphrase detection probability score between 0.5 and 0.99, we accept it as high-confidence paraphrasing, otherwise, we identify it as low-confidence paraphrasing.


% \paragraph{Memorization Type}

% Here we will distinguish 3 types of memorization. First, verbatim memorization means exact copies of words or phrases without transformation. In the cases of paraphrase and idea memorization, the output is not identical to the original text but shares similar meanings. While paraphrase plagiarism focuses on sentence-to-sentence transformation, idea plagiarism involves summarizing the key points of a larger text segment into a more condensed form (or expanding it)

% In practice, the PAN2014-detection starts by identifying closely matched short document fragments (referred to as ‘seeds’) and then expands these seeds into longer text segments. This is achieved by clustering these fragments based on their separation and employing a ‘maxgap’ threshold parameter to form coherent clusters. They experimentally find out the most suitable threshold for different plagiarism datasets so that those parameters could be used for the detection of a specific type of memorization. In other words, each memorization case will be counted only once and there will not be overlapping across different categories.

\section{Ablation Studies}

% \subsection{Comparison with T5-Base}
% \label{base_ab}

% \begin{table*}[htbp]
% \centering
% \caption{\color{blue}Memorization Effects of T5-base models}
% \label{tab:base_memorization}
% \vspace{-0.2cm}
% \resizebox{\textwidth}{!}{
% \begin{tabular}{@{}c|ccccccc@{}}
% \toprule
% Task & Dataset & \begin{tabular}[c]{@{}c@{}}Memorization \\ Ratio\end{tabular} & Verbatim & Idea & \begin{tabular}[c]{@{}c@{}}Paraphrase\\ ($p>0.5$)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Paraphrase\\ ($p<0.5$)\end{tabular} \\
% \midrule
% Summarization & Multi\_news & 11.03\% & 3.58\% & 0.66\% & 4.28\% & 5.46\% \\
% Dialog & HealthCareMagic & 1.60\% & 0.03\% & 1.04\% & 0.11\% & 0.42\% \\
% Sentiment & IMDB & 0.78\% & 0.05\% & 0.37\% & 0.16\% & 0.20\% \\
% QA & Squad & 0.09\% & 0.02\% & 0.00\% & 0.01\% & 0.05\% \\
% Translation & WMT\_16 & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% \\
% \bottomrule
% \end{tabular}}
% \end{table*}

% \vspace{-0.2cm}
% \begin{table*}[htbp]
% \centering
% \caption{\color{blue} Differential Memorization Ratios of T5-base Across Tasks.}
% \label{tab:memorization_difference}
% \vspace{-0.2cm}
% \resizebox{0.7\textwidth}{!}{
% \begin{tabular}{@{}c|cc@{}}
% \toprule
% Task & Dataset & \begin{tabular}[c]{@{}c@{}}Memorization Ratio \\Difference\end{tabular} \\
% \midrule
% Summarization & Multi\_news & 11.30\% \\
% Dialog & HealthCareMagic & 6.67\% \\
% Sentiment & IMDB & 0.20\% \\
% QA & Squad & 0.06\% \\
% Translation & WMT\_16 & 0.00\% \\
% \bottomrule
% \end{tabular}}
% \end{table*}
% To eliminate the impact that the fine-tuning data may also present in the pre-training data, we test the memorization ratio of T5-base without fine-tuning in Table \ref{tab:base_memorization}, and calculate the memorization difference after fine-tuning in Table \ref{tab:memorization_difference}. From the results we can find that a part of the fine-tuning data may also be present in the pre-training data, as T5-base also shows some cases of memorization of the fine-tuning data. However, after fine-tuning, we can see that the fine-tuned model's memorization for summarization and dialogues improves significantly. The model has memorized a large number of fine-tuning data samples during the fine-tuning stage. However, for other tasks such as sentiment classification, the memorization ratio does not change much, meaning that the model does not remember much information from the fine-tuned data.
\subsection{Training Epochs}
\label{ablation_training}
 Here, we present the memorization rates observed in checkpoints of our fine-tuned models across different epochs, as shown in Table~\ref{tab: ablation_epochs}.
 
 In our practice, we find that for low-memorization tasks like sentiment classification, no matter whether the model is well-trained, the memorization ratio remains low. However, for high-memorization tasks like dialog, if the model is not well-trained, the memorization will be low. So in our experiment, to make sure that our finetuned model is well-trained, we let the finetuned model have comparable performance with Flan-T5 as Flan-T5 is a well-trained model and has good performance on various tasks.


\subsection{Sampling Methods}
\label{decoding_ab}

We conduct ablation studies on different decoding methods in Table \ref{tab: t5_decoding}. From the results, we can find that:
\begin{itemize}
    \item For high-memory tasks such as summarization and Dialog,  sampling can reduce the memorization Rate and change the category distribution of memory samples.
   \item For low-memory tasks such as emotion classification, sampling does not significantly affect the memorization results.
   \item Irrespective of the decoding methodology employed, \textbf{a pronounced disparity in memorization across different tasks persists.} This suggests an inherent task-specific propensity towards memorization that is not substantially mitigated by variations in sampling techniques.
\end{itemize}




\subsection{Prefix Lengths}
\label{prefix_ab}

Here we change different prefix lengths of inputs and report the results in table \ref{tab: t5_prefix_length}. We include 2 high-memorization tasks(summarization and dialog) and 1 low-memorization task (sentiment classification). From the results we can observe that:
\begin{itemize}
    \item \textbf{The length of prefix tokens can affect memorization.} The length of prefix tokens does indeed impact memorization. Specifically, for summarization and Dialog tasks, the memorization Rate generally increases with the length of the prefix. This finding aligns with previous research on pre-trained memorization. However, for sentiment classification, changing the prefix does not result in significant changes, and increasing the prefix length does not necessarily lead to an increase in the memorization Rate.
    \item \textbf{The task disparity still exists when using different prefixes. }Furthermore, it is worth noting that despite the influence of different prefixes on memorization, there still exists a noticeable disparity in memorization across tasks. Therefore, our conclusion remains even using different prefixes.
\end{itemize}

\section{Statistical Significance Testing}
\label{Statistic Test}
In this section, the results displayed in Tables \ref{tab:Open_Sourced_test} to \ref{tab: t5_prefix_test} are derived from the data in Tables \ref{tab:Open_Sourced finetune}, \ref{tab:self finetune}, \ref{tab: t5_decoding}, \ref{tab: t5_prefix_length}, and Figure \ref{fig:Ablation}. Specifically, we conducted 1000 bootstrap experiments based on these sources to calculate confidence intervals at the 5\% and 95\% levels for the results presented in the aforementioned tables.
% \begin{table*}[htbp]
% \centering

% \begin{minipage}{0.5\textwidth}
% \centering
% \caption{Significance test of open-sourced LLMs fine-tuned on various tasks}
% \label{tab:Open_Sourced_test}
% \resizebox{\textwidth}{!}{
% \begin{tabular}{@{}c|ccc@{}}
% \toprule
% Task & Dataset & Source Model & Total Mem Ratio (CI) \\
% \midrule
% \href{https://huggingface.co/facebook/bart-large-cnn}{Summarization} & CNN/Daily Mail & Bart\_Large & [19.9\%, 21.5\%] \\
% \href{https://huggingface.co/Narrativaai/BioGPT-Large-finetuned-chatdoctor}{Medical Dialog} & ChatDoctor & BioGPT & [18.8\%, 20.3\%] \\
% \href{https://huggingface.co/potsawee/t5-large-generation-squad-QuestionAnswer}{Extractive QA} & SQuAD\_v2 & T5\_large & [0.05\%, 0.17\%] \\
% \href{https://huggingface.co/potsawee/t5-large-generation-race-QuestionAnswer}{Abstractive QA} & Race & T5\_large & [0.19\%, 0.41\%] \\
% \href{https://huggingface.co/facebook/wmt19-en-de}{Translation} & WMT\_19 & FSMT & [0\%, 0\%] \\
% \href{https://huggingface.co/mrm8488/t5-base-finetuned-imdb-sentiment}{Sentiment Classification} & IMDB & T5-base & [0\%, 0\%] \\
% \bottomrule
% \end{tabular}
% }
% \end{minipage}%
% \hfill
% \begin{minipage}{0.45\textwidth}
% \centering
% \caption{Significance test of T5-base fine-tuned on various tasks}
% \label{tab:self_finetune}
% \resizebox{\textwidth}{!}{
% \begin{tabular}{@{}c|c|c|c@{}}
% \toprule
% Task & Dataset & Model & Total Mem Ratio (CI) \\
% \midrule
% \multirow{3}{*}{Summarization} & \multirow{3}{*}{Multi-news} & T5-base & [13.31\%, 14.71\%] \\
%  & & T5-finetuned & [21.52\%, 23.13\%] \\
%  & & Difference & $\uparrow$[7.81\%, 8.87\%] \\
% \midrule
% \multirow{3}{*}{Dialog} & \multirow{3}{*}{chatdoctor} & T5-base & [1.36\%, 1.83\%] \\
%  & & T5-finetuned & [7.75\%, 8.81\%] \\
%  & & Difference & $\uparrow$[6.18\%, 7.15\%] \\
% \midrule
% \multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}Sentiment\\Classification\end{tabular}} & \multirow{3}{*}{imdb} & T5-base & [0.62\%, 0.96\%] \\
%  & & T5-finetuned & [0.63\%, 0.99\%] \\
%  & & Difference & $\uparrow[0\%,$0.05\%] \\
% \midrule
% \multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}Reading\\Comprehension\end{tabular}} & \multirow{3}{*}{Squad\_v2} & T5-base & [0.03\%, 0.014\%] \\
%  & & T5-finetuned & [0.08\%, 0.23\%] \\
%  & & Difference & $\uparrow$[0.02\%, 0.13\%] \\

% \bottomrule
% \end{tabular}}
% \end{minipage}

% \end{table*}
\begin{table*}[t]
\centering
\caption{Statistical significance test of open-sourced LLMs fine-tuned on various tasks}
\label{tab:Open_Sourced_test}
\resizebox{0.6\textwidth}{!}{
\begin{tabular}{@{}c|ccc@{}}
\toprule
Task & Dataset & Source Model & Total Mem Rate (CI) \\
\midrule
\href{https://huggingface.co/facebook/bart-large-cnn}{Summarization} & CNN/Daily Mail & Bart\_Large & [20.30\%, 21.10\%] \\
\href{https://huggingface.co/Narrativaai/BioGPT-Large-finetuned-chatdoctor}{Medical Dialog} & ChatDoctor & BioGPT & [18.86\%, 20.36\%] \\
\href{https://huggingface.co/potsawee/t5-large-generation-squad-QuestionAnswer}{Extractive QA} & SQuAD\_v2 & T5\_large & [0.05\%, 0.17\%] \\
\href{https://huggingface.co/potsawee/t5-large-generation-race-QuestionAnswer}{Abstractive QA} & Race & T5\_large & [0.19\%, 0.41\%] \\
\href{https://huggingface.co/facebook/wmt19-en-de}{Translation} & WMT\_19 & FSMT & [0\%, 0\%] \\
\href{https://huggingface.co/mrm8488/t5-base-finetuned-imdb-sentiment}{Sentiment Classification} & IMDB & T5-base & [0\%, 0\%] \\
\bottomrule
\end{tabular}
}
\end{table*}



\begin{table*}[htbp]
\centering
\caption{Statistical significance test of T5-base fine-tuned on various tasks}
\label{tab:self_finetune_test}
\resizebox{0.5\textwidth}{!}{
\begin{tabular}{@{}c|c|c|c@{}}
\toprule
Task & Dataset & Model & Total Mem Rate (CI) \\
\midrule
\multirow{3}{*}{Summarization} & \multirow{3}{*}{Multi-news} & T5-base & [13.28\%, 14.68\%] \\
 & & T5-finetuned & [21.52\%, 23.14\%] \\
 & & Difference & [7.84\%, 8.86\%] \\
\midrule
\multirow{3}{*}{Dialog} & \multirow{3}{*}{chatdoctor} & T5-base & [1.36\%, 1.84\%] \\
 & & T5-finetuned & [7.75\%, 8.81\%] \\
 & & Difference & [6.15\%, 7.19\%] \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}Sentiment\\Classification\end{tabular}} & \multirow{3}{*}{imdb} & T5-base & [0.61\%, 0.95\%] \\
 & & T5-finetuned & [0.62\%, 0.98\%] \\
 & & Difference & [0\%, 0.04\%] \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}Reading\\Comprehension\end{tabular}} & \multirow{3}{*}{Squad\_v2} & T5-base & [0.07\%, 0.09\%] \\
 & & T5-finetuned & [0.07\%, 0.23\%] \\
 & & Difference & [0.01\%, 0.19\%] \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}Translation\end{tabular}} & \multirow{3}{*}{wmt} & T5-base & [0\%, 0\%] \\
 & & T5-finetuned & [0\%, 0\%] \\
 & & Difference & [0\%, 0\%] \\
\bottomrule
\end{tabular}}
\end{table*}




% \begin{table*}[htbp]
% \centering

% \begin{minipage}{0.45\textwidth}
% \centering
% \caption{Statistic significant test of fine-tuned T5 with various sampling methods.}
% \label{tab: t5_decoding}
% \resizebox{\textwidth}{!}{%
% \begin{tabular}{c|c|c|c}
% \toprule
% Task & Dataset & Decoding & Total Mem Ratio (CI) \\ 
% \midrule
% \multirow{4}{*}{Dialog} & \multirow{4}{*}{HealthCareMagic} & Top-K & [5.32\%, 6.23\%] \\
%  & & Top-p & [6.78\%, 7.77\%] \\
%  & & Temp & [3.34\%, 4.08\%] \\
%  & & Greedy & [7.75\%, 8.79\%] \\
% \midrule
% \multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & Top-K & [0.83\%, 1.23\%] \\
%  & & Top-p & [0.87\%, 1.29\%] \\
%  & & Temp & [0.71\%, 1.08\%] \\
%  & & Greedy & [0.63\%, 1.00\%] \\
% \midrule
% \multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & Top-K & [10.19\%, 11.44\%] \\
%  & & Top-p & [12.90\%, 14.26\%] \\
%  & & Temp & [5.38\%, 6.28\%] \\
%  & & Greedy & [21.48\%, 23.17\%] \\
% \midrule
% \multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & Top-K & [0.08\%, 0.23\%] \\
%  & & Top-p & [0.07\%, 0.23\%] \\
%  & & Temp & [0.08\%, 0.23\%] \\
%  & & Greedy & [0.08\%, 0.23\%] \\
% \bottomrule
% \end{tabular}
% }
% \end{minipage}%
% \hfill
% \begin{minipage}{0.45\textwidth}
% \centering
% \caption{Significance test of fine-tuned T5 with varying prefix lengths.}
% \label{tab: t5_prefix_length}
% \resizebox{\textwidth}{!}{%
% \begin{tabular}{c|c|c|c}
% \toprule
% Task & Dataset & Prefix length & Total Mem Ratio \\
% \midrule
% \multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & 10 & [11.64\%,12.90\%] \\
%  &  & 30 & [19.87\%, 21.49\%] \\
%  &  & 50 & [21.51\%, 23.14\%] \\
%  &  & 100 & [28.77\%, 30.52\%] \\
% \midrule
% \multirow{3}{*}{Dialog} & \multirow{3}{*}{HealthCareMagic} & 10 & [5.79\%, 6.76\%] \\
%  &  & 30 & [7.23\%, 8.30\%] \\
%  &  & 50 & [7.77\%, 8.83\%] \\
% \midrule
% \multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & 10 & [1.15\%, 1.62\%] \\
%  &  & 30 & [0.97\%, 1.39\%] \\
%  &  & 50 & [0.64\%, 0.97\%] \\
%  &  & 100 & [1.16\%, 1.62\%] \\
% \midrule
% \multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & 10 & [0.10\%, 0.26\%] \\
%  &  & 30 & [0.07\%,0.22\%] \\
%  &  & 50 & [0.08\%, 0.23\%] \\
%  &  & 100 & [0.07\%, 0.20\%] \\
% \bottomrule
% \end{tabular}
% }
% \end{minipage}

% \end{table*}

\begin{table*}[htbp]
\centering
\caption{Statistical significant test of fine-tuned T5 with various sampling methods.}
\label{tab: t5_decoding_test}
\resizebox{0.6\textwidth}{!}{%
\begin{tabular}{c|c|c|c}
\toprule
Task & Dataset & Decoding & Total Mem Rate (CI) \\ 
\midrule
\multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & Top-K & [10.17\%, 11.43\%] \\
 & & Top-p & [12.89\%, 14.25\%] \\
 & & Temp & [5.37\%, 6.27\%] \\
 & & Greedy & [21.48\%, 23.18\%] \\
\midrule
\multirow{4}{*}{Dialog} & \multirow{4}{*}{HealthCareMagic} & Top-K & [5.32\%, 6.23\%] \\
 & & Top-p & [6.78\%, 7.77\%] \\
 & & Temp & [3.34\%, 4.08\%] \\
 & & Greedy & [7.75\%, 8.79\%] \\
\midrule
\multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & Top-K & [0.83\%, 1.23\%] \\
 & & Top-p & [0.87\%, 1.29\%] \\
 & & Temp & [0.71\%, 1.08\%] \\
 & & Greedy & [0.63\%, 1.00\%] \\
\midrule
\multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & Top-K & [0.08\%, 0.23\%] \\
 & & Top-p & [0.07\%, 0.23\%] \\
 & & Temp & [0.08\%, 0.23\%] \\
 & & Greedy & [0.08\%, 0.23\%] \\
\bottomrule
\end{tabular}
}
\end{table*}




\begin{table*}[htbp]
\centering
\caption{Statistical significance test of fine-tuned T5 with varying prefix lengths.}
\label{tab: t5_prefix_test}
\resizebox{0.6\textwidth}{!}{%
\begin{tabular}{c|c|c|c}
\toprule
Task & Dataset & Prefix length & Total Mem Rate (CI) \\
\midrule
\multirow{4}{*}{Summarization} & \multirow{4}{*}{Multi\_news} & 10 & [11.94\%, 12.56\%] \\
 &  & 30 & [20.27\%, 21.09\%] \\
 &  & 50 & [21.99\%, 22.67\%] \\
 &  & 100 & [29.22\%, 30.10\%] \\
\midrule
\multirow{3}{*}{Dialog} & \multirow{3}{*}{HealthCareMagic} & 10 & [5.79\%, 6.77\%] \\
 &  & 30 & [7.23\%, 8.29\%] \\
 &  & 50 & [7.76\%, 8.78\%] \\
\midrule
\multirow{4}{*}{Sentiment} & \multirow{4}{*}{IMDB} & 10 & [1.13\%, 1.61\%] \\
 &  & 30 & [0.96\%, 1.40\%] \\
 &  & 50 & [0.63\%, 0.97\%] \\
 &  & 100 & [1.15\%, 1.63\%] \\
\midrule
\multirow{4}{*}{QA} & \multirow{4}{*}{Squad\_v2} & 10 & [0.09\%, 0.27\%] \\
 &  & 30 & [0.06\%, 0.22\%] \\
 &  & 50 & [0.07\%, 0.24\%] \\
 &  & 100 & [0.06\%, 0.21\%] \\

\bottomrule
\end{tabular}
}
\end{table*}


\color{black}
\section{Dataset and Model used}

\label{Datasets and model}
\paragraph{Datasets} 
In our study, we utilized various datasets for preliminary experiments and model fine-tuning. For the summarization task, we used CNN/Daily Mail in the preliminary phase, a dataset comprising 287k training rows and 10k evaluation rows. For fine-tuning, we employed the \textbf{Multi-News} dataset \cite{alex2019multinews}, which includes news articles and summaries, using a 45k training set and a 5.62k test set. The \textbf{IMDB} dataset \cite{maas-EtAl:2011:ACL-HLT2011} was used for binary sentiment classification, consisting of 25k training and 25k test movie reviews. In the dialog task, we use \href{https://huggingface.co/datasets/lavita/ChatDoctor-HealthCareMagic-100k}{\textbf{HealthcareMagic}} dataset, comprising 112k training rows and a 12k test set. For extractive QA, we utilized the \textbf{SQuAD} v2 dataset \cite{2016arXiv160605250R}, featuring questions based on Wikipedia articles, with 130k training and 11.9k test rows. The translation task involved a preliminary study using \href{https://huggingface.co/datasets/wmt19}{\textbf{WMT19}} and fine-tuning on an English-to-German subset of \href{https://huggingface.co/datasets/wmt16}{\textbf{WMT16}}, with 450.87k training and 3k test rows. Finally, for the controlling experiment, we used the \textbf{RentTheRunway} dataset \cite{10.1145/3240323.3240398}, containing clothing review data, with 111k training and 12k test rows. This dataset was used for fine-tuning both the summarization model and the binary sentiment classification task.

\paragraph{Models}
In the preliminary study, we consider Bart-Large from Bart family, T5-base and T5-large from T5 family, FSMT (FairSeq MachineTranslation), and BioGPT. For our self-fine-tuned models, we select T5-base architecture from the T5 family for all experiments.

\paragraph{Fine-tuned Methods}
In the fine-tuning process, we consider all the tasks as generation tasks and use the format of instruction tuning. Here we provide fine-tuned templates of different tasks in Table \ref{training template}
\section{Performance of Self-fine-tuned Models}
\label{finetune performance}
We finetune the T5-base model to achieve better or comparable performance with the Google fine-tuned public model FLAN-T5. In Table~\ref{tab:summary_performance}, we show the performance of the summarization task. Our fine-tuned model achieves a similar rouge score with FLAN-T5. In Table~\ref{tab:Sentiment_performance}, We show that the accuracy of our model is better than FLAN-T5 regarding binary sentiment classification. For Dialog task, Our model performance much better than FLAN-T5 as shown in Table~\ref{tab:dialogue}. For the Extractive question-answering task, we fine-tune the model in a sequence-to-sequence learning form while we evaluate the exact match of the answer term. Results are shown in Table~\ref{tab:QA}. For the RentTheRunway fine-tuning experiment, we present the results in Table \ref{tab:renttherunway} and Table \ref{tab:renttherunway1}.
\begin{table*}[htbp!]
\centering
\caption{Summarization}
\label{tab:summary_performance}
\resizebox{0.65\textwidth}{!}
{
\begin{tabular}{c|c|c|c|c|c}
\toprule
Dataset          & Model   & Rouge1      & Rouge2                & RougeL       & RougeLSum                \\ 
\midrule% multi\_news & pretrained  T5-base & 0.230 & 0.057 & 0.159
% & 0.159\\
multi\_news & FLAN-T5-small&  0.264 & 0.092 & 0.168 & 0.168\\
multi\_news & Our fine-tuned T5-small& 0.308 & 0.088 & 0.187 & 0.187\\
multi\_news & FLAN-T5-base&0.291&0.098&0.237&0.237\\
multi\_news & Our fine-tuned T5-base&0.298&0.103&0.201&0.201\\
multi\_news & FLAN-T5-large& 0.256 & 0.087 & 0.165 & 0.165\\
multi\_news & Our fine-tuned T5-large& 0.368& 0.122 & 0.218 & 0.218\\
multi\_news & FLAN-T5-3b& 0.264 & 0.092 & 0.232 & 0.232\\
multi\_news & Our fine-tuned T5-3b & 0.387 & 0.136 & 0.168 & 0.168\\

\bottomrule
\end{tabular}}%%%%%%%%%%
\vspace{0.3cm}
\centering
\caption{Sentiment classification}
\label{tab:Sentiment_performance}
\resizebox{0.4\textwidth}{!}{
\begin{tabular}{c|c|c}
\toprule
Dataset          & Model   & Accuracy(\%)  \\ 
\midrule
IMDB & FLAN-T5-small & 94.17\\
IMDB & Our fine-tuned T5-small& 95.30 \\
IMDB & FLAN-T5-base & 93.56\\
IMDB & Our fine-tuned T5-base& 94.64 \\
IMDB & FLAN-T5-large & 94.50\\
IMDB & Our fine-tuned T5-large& 95.30\\
IMDB & FLAN-T5-3b & 97.10 \\
IMDB & Our fine-tuned T5-3b &  95.20\\
\bottomrule
\end{tabular}}%
\vspace{0.2cm}
\caption{Dialog}
\label{tab:dialogue}
\resizebox{0.7\textwidth}{!}{
\begin{tabular}{c|c|c|c|c|c}
\toprule
 Dataset     & Model   & Rouge1     & Rouge2      & RougeL       & RougeLSum   \\ \midrule
% Squad v2&Pretrained T5-base&0.291&0.098&0.237&0.237\\
HealthCareMagic& FLAN-T5-small & 0.041 & 0.004 & 0.031 & 0.031\\
HealthCareMagic& Our fine-tuned T5-small & 0.131 & 0.063 & 0.154 & 0.154\\
HealthCareMagic& FLAN-T5-base & 0.055 &0.006 & 0.039 & 0.039 \\
HealthCareMagic& Our fine-tuned T5-base& 0.298&0.103&0.201&0.201\\
HealthCareMagic&FLAN-T5-large & 0.068 & 0.007 & 0.050 & 0.050\\
HealthCareMagic& Our fine-tuned T5-large & 0.220 & 0.063 & 0.154 & 0.154\\
HealthCareMagic&FLAN-T5-3b & 0.073 & 0.010 & 0.055 & 0.055\\
HealthCareMagic& Our fine-tuned T5-3b & 0.139 & 0.012 & 0.094 & 0.094\\
\bottomrule
\end{tabular}}%
% \vspace{0.2cm}
% \centering
% \caption{Translation}
% \label{tab:Translation}
% \resizebox{\textwidth}{!}
% {%
% \begin{tabular}{c|c|c|c|c|c}
% \toprule
%  Dataset          & Model   & BLEU      & brevity penalty                   & length ratio        & translation length        \\ 
%  \midrule
%  WMT16 de-en & FLAN-T5-Base & 5.62 & 0.943 & 1.14 & 22.71  \\
%  WMT16 de-en & Our fine-tuned T5-Base & 20.55 & 0.947 & 1.04 & 21.30 \\

% \bottomrule
% \end{tabular}%
% }

\centering
\caption{Question answering}
\label{tab:QA}
\resizebox{0.45\textwidth}{!}{
\begin{tabular}{c|c|c}
\toprule
Dataset      & Model   & Exact Match(\%) \\ \midrule
SQuAD v2 & FLAN-T5-small & 35.23\\
SQuAD v2 & Our finetuned T5-small & 49.1 \\
SQuAD v2 & FLAN-T5-base & 34.30 \\
SQuAD v2 & Our finetuned T5-base & 44.00\\
SQuAD v2 & FLAN-T5-large & 43.36\\
SQuAD v2 & Our finetuned T5-large & 53.20\\
SQuAD v2 & FLAN-T5-3b & 44.60\\
SQuAD v2 & Our finetuned T5-3b & 37.20\\
\bottomrule
\end{tabular}}%
\end{table*}
\begin{table*}
\vspace{0.2cm}
\centering
\caption{Multi task trained with RentTheRunway}
\label{tab:renttherunway}
\resizebox{0.65\textwidth}{!}{
\begin{tabular}{c|c|c|c|c|c}
\toprule
 Dataset          & Model   & Rouge1      & Rouge                  & RougeL       & RougeSum        \\ 
 \midrule
Summary & FLAN-T5-Base & 0.1743 & 0.0436 & 0.1598 & 0.1598  \\
Summary & Our finetuned T5-Base & 0.1743 & 0.0436 & 0.1598 & 0.1598  \\
\bottomrule
\end{tabular}}
\vspace{0.2cm}
\centering
\caption{Multi task trained with RentTheRunway}
\label{tab:renttherunway1}
\resizebox{0.55\textwidth}{!}{
\begin{tabular}{c|c|c}
\toprule
 Dataset          & Model   & Accuracy(\%)     \\ 
 \midrule
Sentiment Classification & Flan T5-base & 86.60 \\
Sentiment Classification & Our fine-tuned T5-base & 98.07 \\
\bottomrule
\end{tabular}}
\end{table*}
\section{Memorized Examples}
\label{Mem Cases}
We present memorization examples of verbatim, paraphrase and idea plagiarism of different models in Table \ref{tab:real_examples2}.
\begin{table*}[!htbp]
\vspace{0cm}
\centering
\caption{Examples of memorization cases. Duplicated texts are highlighted with yellow marks. Personally identifiable information (PII) and other words that may lead to privacy concern in generated text are masked as red.
}
\setlength{\leftskip}{-50pt}

%\setlength\extrarowheight{2pt} % for a bit of visual "breathing space"
%    \renewcommand{\tabcolsep}{2pc} % enlarge column spacing
%    \renewcommand{\arraystretch}{1.5} % enlarge line spacing
\setlength{\tabcolsep}{2.5pt}
\begin{tabular}{c|cc}
%\begin{xtabular}{|p{0.48\textwidth}|p{0.48\textwidth}|}
\toprule
Type & Machine-Written Text & Training Text \\
\midrule
\multirow{2}*{Paraphrase} 
&  \begin{tabular}[c]{@{}c@{}} \\At least\highlightyellow{10 people} and two attackers were \highlightyellow{killed} \\in Tuesday's attack against the luxurious \\ \highlightred{Corinthia Hotel in Tripoli, Libya}, a spokesman for \\ a security division of the Ministry of Interior in \\ \highlightyellow{Tripoli} said. \highlightyellow{Five foreigners -- one American} \\(\textit{\textbf{Summarization: CNN Daily, Bart Large}}) \\ \newline \end{tabular} &  \begin{tabular}[c]{@{}c@{}}On January 27, gunmen claiming to be affiliated \\with ISIS attacked \highlightyellow{the Corinthia Hotel in Tripoli,}\\ \highlightyellow{Libya, } which is favored by government officials and \\foreigners. They \highlightyellow{killed 10 people} after storming into \\ the lobby and firing guns at hotel guests. Five \\of the victims were \highlightyellow{foreigners, one an American}.  \end{tabular}  \\  
\midrule
\multirow{2}*{Paraphrase} &  \begin{tabular}[c]{@{}c@{}} \\-- \highlightred{Argentine President Cristina Fernandez de Kirchner}\\ was told \highlightyellow{to take a month off work} after doctors \\diagnosed her with a subdural \highlightyellow{hematoma}. \\ (\textit{\textbf{Summarization: CNN Daily, Bart Large}}) \\ \newline \end{tabular} 
 &  \begin{tabular}[c]{@{}c@{}}...\highlightyellow{hematoma and} \\ \highlightyellow{said she needed to take a month off of work.}\end{tabular} \\  
\midrule
\multirow{2}*{Paraphrase} 
 &  \begin{tabular}[c]{@{}c@{}}\\ I am a 20-year-old guy \highlightyellow{20 years old. I have been}\\ \highlightred{(***harassment word***)} for a long time. \\(\textit{\textbf{Dialog: 
 HealthCareMagic, Finetuned T5}}) \\ \newline \end{tabular}&  \begin{tabular}[c]{@{}c@{}}...\highlightyellow{20 years old  i have been}. \\ \highlightyellow{(***harassment word***) }regularly \highlightyellow{for} past 5 years \end{tabular} \\
 \midrule
 
\multirow{2}*{Paraphrase} &  \begin{tabular}[c]{@{}c@{}} \\ \highlightyellow{The Chargers responded with} a 1-\highlightyellow{yard TD}\\ run by \highlightyellow{RB} \highlightred{LaDainian Tomlinson}. \\(\textit{\textbf{Abstractive QA: Finetuned T5}}) \\ \newline \end{tabular}
 &  \begin{tabular}[c]{@{}c@{}} \highlightyellow{The Chargers} would respond \highlightyellow{with RB} \\ \highlightyellow{LaDainian Tomlinson with a} 4-\highlightyellow{yard TD run.}\\ \end{tabular} \\
\midrule

\multirow{2}*{Paraphrase} 
&  \begin{tabular}[c]{@{}c@{}} \\ – President Trump has \highlightyellow{named} \highlightred{Mick Mulvaney}\\ to replace \highlightred{John Kelly}, \highlightyellow{the White House chief} \\of staff who left the White House in December." \\ (\textbf{Summary: Multi news, Finetuned T5}) \\ \newline \end{tabular} & \begin{tabular}[c]{@{}c@{}} \\ “I am pleased to announce that \highlightyellow{Mick Mulvaney}, \\Director of the Office of Management \& Budget,\\ will be \highlightyellow{named} Acting \highlightyellow{White House Chief of Staff}, \\replacing General \highlightyellow{John Kelly}, who has served our ...\end{tabular}   \\

\midrule

\multirow{2}*{Verbatim} &  \begin{tabular}[c]{@{}c@{}}\\ \highlightred{Rachel's son Liam} \highlightyellow{in a house}\\ \highlightyellow{near} \highlightred{Glenrothes} \highlightyellow{on 22 March 2014. }\\(\textit{\textbf{Summary: CNN Daily, Finetuned T5}}) \\ \newline \end{tabular} 
 & \begin{tabular}[c]{@{}c@{}}\highlightyellow{Rachel's son Liam in a house}\\ \highlightyellow{near Glenrothes on 22 March 2014.}  \end{tabular} \\
 \midrule
  
 \multirow{2}*{Verbatim} &  \begin{tabular}[c]{@{}c@{}}\\ \highlightyellow{divided Wednesday during heated arguments}\\ \highlightyellow{over President} \highlightred{Obama's} \highlightyellow{health care law, but}\\(\textit{\textbf{Summary: Multi news, Finetuned T5}}) \\ \newline\end{tabular} 
 & \begin{tabular}[c]{@{}c@{}} \highlightyellow{divided Wednesday during heated arguments} \\ \highlightyellow{over President Obama's health care law, but}\end{tabular} \\
 \midrule
\multirow{2}*{Verbatim} &  \begin{tabular}[c]{@{}c@{}}\\ \highlightyellow{and liver cirrhosis in dec 2011} \\ \highlightyellow{modified akt staarted because of cirrhosis i.e} \\ (\textit{\textbf{Dialog:Finetuned T5, ChatDoctor}})\\ \end{tabular} 
 & \begin{tabular}[c]{@{}c@{}}\highlightyellow{and liver cirrhosis in dec 2011} \\ \highlightyellow{modified akt staarted because of cirrhosis i.e  }\end{tabular} \\
\bottomrule
\end{tabular}
\label{tab:real_examples2}

\end{table*}



\begin{table*}[!htbp]
\centering
% \caption{Examples of Memorization Cases}
\setlength{\leftskip}{-55pt}

%\setlength\extrarowheight{2pt} % for a bit of visual "breathing space"
%    \renewcommand{\tabcolsep}{2pc} % enlarge column spacing
%    \renewcommand{\arraystretch}{1.5} % enlarge line spacing
\setlength{\tabcolsep}{2pt}
\begin{tabular}{c|cc}

%\begin{xtabular}{|p{0.48\textwidth}|p{0.48\textwidth}|}
\toprule
Type & Machine-Written Text & Training Text \\
\midrule

\multirow{2}*{Verbatim} 
 &  \begin{tabular}[c]{@{}c@{}}\\ \highlightred{River Martinez}, \highlightyellow{10, breaks camp at the Upper} \\ \highlightyellow{Pines Campground in Yosemite National Park, Calif.,}\\ \highlightyellow{on Wednesday, July 25, 2018.}\\ (\textit{\textbf{Summary: Multi news, Finetuned T5
}})\\\newline \end{tabular}&  \begin{tabular}[c]{@{}c@{}}  \highlightyellow{River Martinez, 10, breaks camp at the Upper}\\ \highlightyellow{Pines Campground in Yosemite National Park, Calif.,}\\ \highlightyellow{on Wednesday, July 25, 2018.}\\ \end{tabular} \\
 \midrule
 
  \multirow{1}*{Verbatim} & \begin{tabular}[c]{@{}c@{}}\\ \highlightyellow{A rare blue lobster caught by local lobsterman, }\\ \highlightyellow{Greg Ward, is on display at the} \highlightred{Seacoast Science}\\\highlightred{ Center in Rye, N.H., on Tuesday, July 18, 2017.}\\(\textit{\textbf{Summary: Xsum, Finetuned T5}}) \\ \newline\end{tabular} 
 & \begin{tabular}[c]{@{}c@{}}\\ \highlightyellow{A rare blue lobster caught by local lobsterman, }\\  \highlightyellow{Greg Ward, is on display at the Seacoast Science }\\  \highlightyellow{Center in Rye, N.H., on Tuesday, July 18, 2017. }\\ \newline\end{tabular}  \\
 \midrule
  
 \multirow{2}*{Verbatim} &  \begin{tabular}[c]{@{}c@{}}\\ \highlightred{Sheffield} \highlightyellow{homered twice and keyed} \\ \highlightyellow{a four-run rally in the ninth} \\ \highlightyellow{inning Thursday night, sending the }\\(\textit{\textbf{Classification: AG news, Finetuned T5}}) \\ \newline \end{tabular}
 & \begin{tabular}[c]{@{}c@{}} \highlightyellow{Sheffield homered twice and keyed}\\ \highlightyellow{a four-run rally in the ninth}\\ \highlightyellow{inning Thursday night, sending the} \end{tabular} \\
 \midrule
  \multirow{2}*{Idea} & \begin{tabular}[c]{@{}c@{}}\\ \highlightred{KUALA LUMPUR (Reuters) - Kim Jong Un's} \\half-brother\highlightyellow{was carrying \$100,000 in cash in}\\ \highlightyellow{his backpack} at the time of his murder, the officer\\ investigating the case told a police officer" \\(\textit{\textbf{Summary: Multi news, Finetuned T5}}) \\ \newline\end{tabular} 
 & \begin{tabular}[c]{@{}c@{}}\\ Wan Azirul testified that \highlightyellow{Kim was carrying}\\ \highlightyellow{\$100,000 in cash in his backpack.}\\ \newline\end{tabular}  \\
  \midrule
 \multirow{3}*{Idea} &  \begin{tabular}[c]{@{}c@{}} \\ \highlightred{Alan Dawson, 64, of Urmston,} was convicted of \\seven counts of indecent assault and \highlightyellow{one count} \\ \highlightyellow{of rape at} Manchester Crown Court. \\(\textit{\textbf{Summary: Xsum, Finetuned T5}}) \\ \newline \end{tabular} 
 &  \begin{tabular}[c]{@{}c@{}} ...is charged with \highlightyellow{one count of rape} \\and \highlightyellow{one count} of sexual \highlightyellow{assault.} \end{tabular} \\
 \midrule
 \multirow{3}*{Idea} &  \begin{tabular}[c]{@{}c@{}} \\ – \highlightyellow{Trey Radel, the Florida Rep. who was arrested}\\ \highlightyellow{last month for buying cocaine}, is a freshman congressman \\who has been a big news story for the Washington Post.  \\(\textit{\textbf{Summary: Multi News, Finetuned T5}}) \\ \newline \end{tabular} 
 &  \begin{tabular}[c]{@{}c@{}} Post) \textbackslash n \textbackslash \highlightyellow{n Florida Rep. Trey Radel (R-Fla.)} \\ \highlightyellow{was arrested last month for buying cocaine.}\\\end{tabular} \\
% \midrule
% \multirow{2}*{Idea} &  \begin{tabular}[c]{@{}c@{}} \\ Hello, I a\highlightyellow{m 20 years old, height 51,}\\ \highlightyellow{height 51, weight 40kg.} \\ (\textit{\textbf{Dialogue: ChatDoctor, FLAN-T5}}) \\ \newline \end{tabular} 
%  &  \begin{tabular}[c]{@{}c@{}}  please help.... hello doctor, \\ \highlightyellow{m 20 years old, height 5 1 \& weight 40kg. }\end{tabular} \\
 \midrule
\multirow{2}*{Idea} &  \begin{tabular}[c]{@{}c@{}} \\ \highlightred{Abdul Aziz} believes he was \highlightyellow{standing right next to a shooter} \\when gunmen opened fire at a parade in new orleans, \\injuring 19 people. "Everyone around me was \highlightyellow{right next} to a \\shooter," \highlightred{Abdul Aziz} said.\\ (\textit{\textbf{Summary: CNN Daily, Finetuned T5}}) \\ \newline \end{tabular} 
 &  \begin{tabular}[c]{@{}c@{}}  I \highlightyellow{was standing}, I \highlightyellow{believe}, \highlightyellow{right next to the shooter}.  \end{tabular} \\
\bottomrule
\end{tabular}
\label{tab:real_examples3}

\end{table*}


\section{Attention Maps}
\label{attention_ab}

\subsection{Attention Maps of Translation Tasks}
\label{attention_translation}
In Figure \ref{fig:heatmap-translation}, we present the attention maps for translation tasks. The visualized examples reveal that attention scores tend to focus on a select few input features corresponding to each output token. This concentration of attention is likely due to the nature of translation tasks, where the model typically does not require full detail attention but rather focuses on key tokens for accurate translation. As a result, attention is directed predominantly towards specific features, leading to reduced overall memorization. These observations are consistent with our findings from other tasks.



\begin{figure}[t]
\centering

\resizebox{\textwidth}{!}{%
    \begin{minipage}{\textwidth}
        \subfloat[Sample1-trans.]{\includegraphics[width=.25\textwidth]{pic/translation_heatmap1.pdf}}
        \label{fig:translation-heatmap1}
        \subfloat[Sample2-trans.]{\includegraphics[width=.25\textwidth]{pic/translation_heatmap5.pdf}}
        \label{fig:translation-heatmap2}\\
        \subfloat[Sample3-trans.]{\includegraphics[width=.25\textwidth]{pic/translation_heatmap6.pdf}}
        \label{fig:translation-heatmap3}
        \subfloat[Sample4-trans.]{\includegraphics[width=.25\textwidth]{pic/translation_heatmap7.pdf}}
        \label{fig:translation-heatmap4}
        
    \end{minipage}
}

\caption{Decoder-encoder attention heatmaps on translation.}
\label{fig:heatmap-translation}
\end{figure}

\subsection{Attention Maps of T5-base when doing different tasks}
\label{attention_t5base}

To validate that attention patterns are more intrinsic properties of the tasks themself, we visualize the attention maps of the T5-base model(without fine-tuning) when doing different tasks in Figure \ref{fig:heatmap-base}. Specifically, we use the same instruction and input-output pairs of fine-tuning data as Section 5.1, but just change the model from finetuned-T5 to T5-base. From the Figure we can see that the disparity still exists across different tasks. And for each task, the attention patterns are similar to that of Fine-tuned T5. It further validates that the information needed to complete certain tasks is the intrinsic property of the task. 
\subsection{Attention Maps of different samples}
\label{sample attention}
In this section, we extend our visualization of attention maps across a broader range of samples and tasks, from Figure \ref{fig:heatmap-sample-summarization} to Figure \ref{fig:heatmap-sample-QA}. It is evident that memorization patterns differ significantly among tasks. Tasks with higher memorization requirements, such as summarization, display densely distributed attention scores, while those with lower memorization needs, like Extractive QA, exhibit more focused attention distributions.

\subsection{Attention Maps of Different encoder-decoder layers}
\label{attention_layers}
Here we visualize the attention maps of different encoder-decoder layers in Figure \ref{fig:heatmap-layer-summarization} to Figure \ref{fig:heatmap-layer-QA}. We can clearly observed consistent patterns across various layers of the encoder-decoder attention mechanism, with high memorization tasks showing dense attention and low memorization tasks focusing attention on fewer positions.
\clearpage
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \begin{minipage}{\textwidth}
        \subfloat[]{\includegraphics[width=.25\textwidth]{pic/summary_base.pdf}
        \label{fig:summary-base}}
        \subfloat[]{\includegraphics[width=.25\textwidth]{pic/dialog_base.pdf}
        \label{fig:dialog-base}}
        \subfloat[]{\includegraphics[width=.25\textwidth]{pic/sentiment_base.pdf}
        \label{fig:sentiment-base}}
        \subfloat[]{\includegraphics[width=.25\textwidth]{pic/QA_base.pdf}
        \label{fig:QA-base}}
    \end{minipage}
}

\caption{Average decode-encoder attention heatmaps on (a) summarization, (b) dialog, (c) sentiment, and (d) QA on T5-base across 10 samples}
\label{fig:heatmap-base}
\end{figure*}

\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/summarization_samples.png}
    
}
\caption{Attention heatmaps of different samples on  summarization}
\label{fig:heatmap-sample-summarization}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/dialog_samples.pdf}
    
}
\caption{Attention heatmaps of different samples on  dialog}
\label{fig:heatmap-sample-dialog}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/sentiment_samples.pdf}
    
}
\caption{Attention heatmaps of different samples on  sentiment classification}
\label{fig:heatmap-sample-sentiment}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/QA_samples.pdf}
    
}
\caption{Attention heatmaps of different samples on  QA}
\label{fig:heatmap-sample-QA}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/Summarization_Layers.pdf}
    
}
\caption{Average decode-encoder attention heatmaps on summarization from different layers}
\label{fig:heatmap-layer-summarization}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/Dialog_Layers.pdf}
    
}
\caption{{Average Decode-encoder attention heatmaps on dialog from different layers}}
\label{fig:heatmap-layer-dialog}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/Sentiment_Layers.pdf}
    
}
\caption{{Average decode-encoder attention heatmaps on sentiment classification from different layers}}
\label{fig:heatmap-layer-sentiment}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{\textwidth}{!}{%
    \includegraphics[]{pic/QA_Layers.pdf}
    
}
\caption{{Average decode-encoder attention heatmaps on QA from different layers}}
\label{fig:heatmap-layer-QA}
\end{figure*}
% \end{appendices}


\begin{thebibliography}{39}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi

\bibitem[{Arora et~al.(2015)Arora, Li, Liang, Ma, and Risteski}]{arora2015latent}
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2015.
\newblock A latent variable model approach to pmi-based word embeddings.
\newblock \emph{arXiv preprint arXiv:1502.03520}.

\bibitem[{Arora et~al.(2018)Arora, Li, Liang, Ma, and Risteski}]{arora2018linear}
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018.
\newblock Linear algebraic structure of word senses, with applications to polysemy.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 6:483--495.

\bibitem[{Bai et~al.(2022)Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, Goldie, Mirhoseini, McKinnon et~al.}]{bai2022constitutional}
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et~al. 2022.
\newblock Constitutional ai: Harmlessness from ai feedback.
\newblock \emph{arXiv preprint arXiv:2212.08073}.

\bibitem[{Beltagy et~al.(2019)Beltagy, Lo, and Cohan}]{beltagy2019scibert}
Iz~Beltagy, Kyle Lo, and Arman Cohan. 2019.
\newblock Scibert: A pretrained language model for scientific text.
\newblock \emph{arXiv preprint arXiv:1903.10676}.

\bibitem[{Biderman et~al.(2023)Biderman, Prashanth, Sutawika, Schoelkopf, Anthony, Purohit, and Raf}]{biderman2023emergent}
Stella Biderman, USVSN~Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023.
\newblock Emergent and predictable memorization in large language models.
\newblock \emph{arXiv preprint arXiv:2304.11158}.

\bibitem[{Carlini et~al.(2022)Carlini, Ippolito, Jagielski, Lee, Tramer, and Zhang}]{carlini2022quantifying}
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.
\newblock Quantifying memorization across neural language models.
\newblock \emph{arXiv preprint arXiv:2202.07646}.

\bibitem[{Carlini et~al.(2021)Carlini, Tramer, Wallace, Jagielski, Herbert-Voss, Lee, Roberts, Brown, Song, Erlingsson et~al.}]{carlini2021extracting}
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et~al. 2021.
\newblock Extracting training data from large language models.
\newblock In \emph{30th USENIX Security Symposium (USENIX Security 21)}, pages 2633--2650.

\bibitem[{Chowdhery et~al.(2022)Chowdhery, Narang, Devlin, Bosma, Mishra, Roberts, Barham, Chung, Sutton, Gehrmann et~al.}]{chowdhery2022palm}
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung~Won Chung, Charles Sutton, Sebastian Gehrmann, et~al. 2022.
\newblock Palm: Scaling language modeling with pathways.
\newblock \emph{arXiv preprint arXiv:2204.02311}.

\bibitem[{Chung et~al.(2022)Chung, Hou, Longpre, Zoph, Tay, Fedus, Li, Wang, Dehghani, Brahma et~al.}]{chung2022scaling}
Hyung~Won Chung, Le~Hou, Shayne Longpre, Barret Zoph, Yi~Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et~al. 2022.
\newblock Scaling instruction-finetuned language models.
\newblock \emph{arXiv preprint arXiv:2210.11416}.

\bibitem[{Deora et~al.(2023)Deora, Ghaderi, Taheri, and Thrampoulidis}]{deora2023optimization}
Puneesh Deora, Rouzbeh Ghaderi, Hossein Taheri, and Christos Thrampoulidis. 2023.
\newblock On the optimization and generalization of multi-head attention.
\newblock \emph{arXiv preprint arXiv:2310.12680}.

\bibitem[{Fabbri et~al.(2019)Fabbri, Li, She, Li, and Radev}]{alex2019multinews}
Alexander~R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir~R. Radev. 2019.
\newblock \href {http://arxiv.org/abs/1906.01749} {Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model}.

\bibitem[{Houlsby et~al.(2019)Houlsby, Giurgiu, Jastrzebski, Morrone, De~Laroussilhe, Gesmundo, Attariyan, and Gelly}]{houlsby2019parameter}
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De~Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.
\newblock Parameter-efficient transfer learning for nlp.
\newblock In \emph{International Conference on Machine Learning}, pages 2790--2799. PMLR.

\bibitem[{Hu et~al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen}]{hu2021lora}
Edward~J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, and Weizhu Chen. 2021.
\newblock Lora: Low-rank adaptation of large language models.
\newblock \emph{arXiv preprint arXiv:2106.09685}.

\bibitem[{Ippolito et~al.(2022)Ippolito, Tram{\`e}r, Nasr, Zhang, Jagielski, Lee, Choquette-Choo, and Carlini}]{ippolito2022preventing}
Daphne Ippolito, Florian Tram{\`e}r, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher~A Choquette-Choo, and Nicholas Carlini. 2022.
\newblock Preventing verbatim memorization in language models gives a false sense of privacy.
\newblock \emph{arXiv preprint arXiv:2210.17546}.

\bibitem[{Kandpal et~al.(2022)Kandpal, Wallace, and Raffel}]{kandpal2022deduplicating}
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022.
\newblock Deduplicating training data mitigates privacy risks in language models.
\newblock In \emph{International Conference on Machine Learning}, pages 10697--10707. PMLR.

\bibitem[{Lee et~al.(2023)Lee, Le, Chen, and Lee}]{lee2023language}
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023.
\newblock Do language models plagiarize?
\newblock In \emph{Proceedings of the ACM Web Conference 2023}, pages 3637--3647.

\bibitem[{Lee et~al.(2021)Lee, Ippolito, Nystrom, Zhang, Eck, Callison-Burch, and Carlini}]{lee2021deduplicating}
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021.
\newblock Deduplicating training data makes language models better.
\newblock \emph{arXiv preprint arXiv:2107.06499}.

\bibitem[{Lester et~al.(2021)Lester, Al-Rfou, and Constant}]{lester2021power}
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
\newblock The power of scale for parameter-efficient prompt tuning.
\newblock \emph{arXiv preprint arXiv:2104.08691}.

\bibitem[{Li and Liang(2021)}]{li2021prefix}
Xiang~Lisa Li and Percy Liang. 2021.
\newblock Prefix-tuning: Optimizing continuous prompts for generation.
\newblock \emph{arXiv preprint arXiv:2101.00190}.

\bibitem[{Liu et~al.(2021)Liu, Ji, Fu, Tam, Du, Yang, and Tang}]{liu2021p}
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng~Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021.
\newblock P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.
\newblock \emph{arXiv preprint arXiv:2110.07602}.

\bibitem[{Longpre et~al.(2023)Longpre, Hou, Vu, Webson, Chung, Tay, Zhou, Le, Zoph, Wei et~al.}]{longpre2023flan}
Shayne Longpre, Le~Hou, Tu~Vu, Albert Webson, Hyung~Won Chung, Yi~Tay, Denny Zhou, Quoc~V Le, Barret Zoph, Jason Wei, et~al. 2023.
\newblock The flan collection: Designing data and methods for effective instruction tuning.
\newblock \emph{arXiv preprint arXiv:2301.13688}.

\bibitem[{Maas et~al.(2011)Maas, Daly, Pham, Huang, Ng, and Potts}]{maas-EtAl:2011:ACL-HLT2011}
Andrew~L. Maas, Raymond~E. Daly, Peter~T. Pham, Dan Huang, Andrew~Y. Ng, and Christopher Potts. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1015} {Learning word vectors for sentiment analysis}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies}, pages 142--150, Portland, Oregon, USA. Association for Computational Linguistics.

\bibitem[{Mireshghallah et~al.(2022)Mireshghallah, Uniyal, Wang, Evans, and Berg-Kirkpatrick}]{mireshghallah2022memorization}
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022.
\newblock Memorization in nlp fine-tuning methods.
\newblock \emph{arXiv preprint arXiv:2205.12506}.

\bibitem[{Misra et~al.(2018{\natexlab{a}})Misra, Wan, and McAuley}]{misra2018decomposing}
Rishabh Misra, Mengting Wan, and Julian McAuley. 2018{\natexlab{a}}.
\newblock Decomposing fit semantics for product size recommendation in metric spaces.
\newblock In \emph{Proceedings of the 12th ACM Conference on Recommender Systems}, pages 422--426.

\bibitem[{Misra et~al.(2018{\natexlab{b}})Misra, Wan, and McAuley}]{10.1145/3240323.3240398}
Rishabh Misra, Mengting Wan, and Julian McAuley. 2018{\natexlab{b}}.
\newblock \href {https://doi.org/10.1145/3240323.3240398} {Decomposing fit semantics for product size recommendation in metric spaces}.
\newblock In \emph{Proceedings of the 12th ACM Conference on Recommender Systems}, RecSys '18, page 422–426, New York, NY, USA. Association for Computing Machinery.

\bibitem[{Olshausen and Field(1997)}]{olshausen1997sparse}
Bruno~A Olshausen and David~J Field. 1997.
\newblock Sparse coding with an overcomplete basis set: A strategy employed by v1?
\newblock \emph{Vision research}, 37(23):3311--3325.

\bibitem[{Olshausen and Field(2004)}]{olshausen2004sparse}
Bruno~A Olshausen and David~J Field. 2004.
\newblock Sparse coding of sensory inputs.
\newblock \emph{Current opinion in neurobiology}, 14(4):481--487.

\bibitem[{OpenAI(2023)}]{openai2023gpt}
R~OpenAI. 2023.
\newblock Gpt-4 technical report.
\newblock \emph{arXiv}, pages 2303--08774.

\bibitem[{Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray et~al.}]{ouyang2022training}
Long Ouyang, Jeffrey Wu, Xu~Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et~al. 2022.
\newblock Training language models to follow instructions with human feedback.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:27730--27744.

\bibitem[{Raffel et~al.(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li, and Liu}]{raffel2020exploring}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter~J Liu. 2020.
\newblock Exploring the limits of transfer learning with a unified text-to-text transformer.
\newblock \emph{The Journal of Machine Learning Research}, 21(1):5485--5551.

\bibitem[{{Rajpurkar} et~al.(2016){Rajpurkar}, {Zhang}, {Lopyrev}, and {Liang}}]{2016arXiv160605250R}
Pranav {Rajpurkar}, Jian {Zhang}, Konstantin {Lopyrev}, and Percy {Liang}. 2016.
\newblock \href {http://arxiv.org/abs/1606.05250} {{SQuAD: 100,000+ Questions for Machine Comprehension of Text}}.
\newblock \emph{arXiv e-prints}, page arXiv:1606.05250.

\bibitem[{Sanchez-Perez et~al.(2015)Sanchez-Perez, Gelbukh, and Sidorov}]{sanchez2015adaptive}
Miguel~A Sanchez-Perez, Alexander Gelbukh, and Grigori Sidorov. 2015.
\newblock Adaptive algorithm for plagiarism detection: The best-performing approach at pan 2014 text alignment competition.
\newblock In \emph{International Conference of the Cross-Language Evaluation Forum for European Languages}, pages 402--413. Springer.

\bibitem[{Touvron et~al.(2023{\natexlab{a}})Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozi{\`e}re, Goyal, Hambro, Azhar et~al.}]{touvron2023llama1}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth{\'e}e Lacroix, Baptiste Rozi{\`e}re, Naman Goyal, Eric Hambro, Faisal Azhar, et~al. 2023{\natexlab{a}}.
\newblock Llama: Open and efficient foundation language models.
\newblock \emph{arXiv preprint arXiv:2302.13971}.

\bibitem[{Touvron et~al.(2023{\natexlab{b}})Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale et~al.}]{touvron2023llama}
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et~al. 2023{\natexlab{b}}.
\newblock Llama 2: Open foundation and fine-tuned chat models.
\newblock \emph{arXiv preprint arXiv:2307.09288}.

\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin}]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin. 2017.
\newblock Attention is all you need.
\newblock \emph{Advances in neural information processing systems}, 30.

\bibitem[{Wu et~al.(2023)Wu, Zou, Chen, Braverman, Gu, and Bartlett}]{wu2023many}
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter~L Bartlett. 2023.
\newblock How many pretraining tasks are needed for in-context learning of linear regression?
\newblock \emph{arXiv preprint arXiv:2310.08391}.

\bibitem[{Yunxiang et~al.(2023)Yunxiang, Zihan, Kai, Ruilong, and You}]{yunxiang2023chatdoctor}
Li~Yunxiang, Li~Zihan, Zhang Kai, Dan Ruilong, and Zhang You. 2023.
\newblock Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge.
\newblock \emph{arXiv preprint arXiv:2303.14070}.

\bibitem[{Zhang et~al.(2021)Zhang, Ippolito, Lee, Jagielski, Tram{\`e}r, and Carlini}]{zhang2021counterfactual}
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tram{\`e}r, and Nicholas Carlini. 2021.
\newblock Counterfactual memorization in neural language models.
\newblock \emph{arXiv preprint arXiv:2112.12938}.

\bibitem[{Zhang et~al.(2023)Zhang, Frei, and Bartlett}]{zhang2023trained}
Ruiqi Zhang, Spencer Frei, and Peter~L Bartlett. 2023.
\newblock Trained transformers learn linear models in-context.
\newblock \emph{arXiv preprint arXiv:2306.09927}.

\end{thebibliography}