Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
Extracted source text · arXiv 2412.06858
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2019
% ready for submission
% \usepackage{neurips_2019}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{nips2019}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{nips2019}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2019}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage[pagebackref,breaklinks,colorlinks,allcolors=green]{hyperref}
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage[table,xcdraw]{xcolor}
\usepackage{color}
\usepackage{amsmath}
\usepackage{mathrsfs}
\usepackage{amssymb}
\usepackage{amsthm}
\theoremstyle{definition}
\theoremstyle{plain}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{booktabs}
\usepackage{caption}
\newtheorem{lemma}{Lemma}
\newtheorem{assumption}{Assumption}
\newtheorem{theorem}{Theorem}
% \usepackage[dvipsnames]{xcolor}
\title{Taming Sensitive Weights: Noise Perturbation Fine-tuning for Robust LLM Quantization}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{
Dongwei Wang\textsuperscript{1}, ~Huanrui Yang\textsuperscript{1} \\
\textsuperscript{1}Department of Electrical and Computer Engineering, University of Arizona\\
\texttt{dongweiw@arizona.edu, huanruiyang@arizona.edu}
}
\begin{document}
\maketitle
\begin{abstract}
Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose \textbf{N}oise \textbf{P}erturbation \textbf{F}ine-\textbf{t}uning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.
\end{abstract}
\section{Introduction}
Large Language Models (LLMs), usually with billions of parameters, have demonstrated impressive problem-solving abilities across diverse tasks \cite{chowdhery2023palm, du2022glam, smith2022using, le2023bloom}. The enhanced performance, largely driven by the scaling of both training data and model parameters \cite{kaplan2020scaling}, has made it challenging to deploy LLMs on edge computing devices. For example, a model like GPT-3 \cite{floridi2020gpt}, with 175 billion parameters, requires 350 GB storage space in FP16 and powerful GPUs such as A100 for quick inference, which makes deployment on devices like laptops or mobile phones infeasible without significant model compression. As a promising approach, low-bit weight quantization can help address this issue by enabling efficient inference and reducing storage requirements. Pioneering works such as GPTQ \cite{frantar2022gptq} and Squeeze LLM \cite{kim2023squeezellm} can compress LLaMA \cite{touvron2023open} model weights to 3-4 bits with a nearly lossless performance, achieving 2-3$\times$ speed up on GPUs.
The most straightforward way to perform weight quantization is via linear uniform quantization, which quantizes the entire model to the same bit-width using simple quantizers like Round-to-Nearest (RTN). However, it is widely acknowledged that \textit{weights are not equally important in a neural network}. There is a small fraction of weights that are very sensitive to quantization and lead to significant performance degradation if quantized. This is because these sensitive weights (also referred to as \emph{outliers}) have a larger impact on the outputs of their respective layers, which in turn affects the final loss of the model. As shown in Fig. \ref{fig1}, the performance of two OPT \cite{zhang2022opt} models dropped significantly after applying RTN quantization. However, preserving 0.5\% of the outliers as FP16 can greatly recover the performance of the quantized model. A 4-bit quantized model with 0.5\% full-precision outliers can achieve performance comparable to that of a full-precision model.
In light of this, existing work typically focuses on reducing the quantization error of outliers by preserving them. For example, SqueezeLLM \cite{kim2023squeezellm} proposed to store outliers as a sparse FP16 matrix and design a non-uniform sensitivity-based quantizer for other weights. In SpQR \cite{dettmers2023spqr}, outliers are extracted in the form of channel groups and quantized to a higher bit-width. AWQ \cite{lin2024awq} proposed to select outliers based on activation distribution and reduce their quantization error by per-channel scaling. In general, existing works tend to preserve outliers at a higher precision than normal weights. However, even though this may not significantly impact the overall size of the quantized model, it results in the model being in a mixed-precision state.
According to \cite{nvidia_mixed_precision_guide}, most GPUs prefer operations with a uniform data type, as mixed precision requires different storage methods in memory, hindering the full utilization of GPU memory bandwidth. Additionally, mixed precision disrupts the SIMD processing model, leading to inefficient use of GPU cores.
In this paper, we aim to mitigate the special treatment on outlier weights by enabling the model to maintain good performance even after quantizing these outliers to the same bit-width.
Drawing inspiration from previous work on analyzing quantization error impacts, like HAWQ~\cite{dong2019hawq, dong2020hawq} and HERO~\cite{yang2022hero}, we pinpoint the quantization error of the outliers is contributed by both their quantization errors and the loss function's Hessian trace with respect to them.
This observation leads to the proposal of an efficient fine-tuning method named \textbf{N}oise \textbf{P}erturbation \textbf{F}ine-\textbf{t}uning (NPFT), which helps reduce the Hessian trace of outlier weights with a short parameter-efficient fine-tuning (PEFT) process. Specifically, we estimate the Hessian trace with the expected model loss under random weight perturbations. We further fine-tune the model to reduce the outlier Hessian trace by performing PEFT optimization with random weight perturbation being added to the outlier weight locations in the base model.
The NPFT process avoids the costly higher-order gradient computation needed to optimize Hessian or Fisher matrix directly, while also enjoys a smoother convergence compared to full QAT as all weights are still kept as floating-points in the training process.
As shown in Fig. \ref{fig1}, NPFT fine-tuned model shows better PPL than quantized baseline even with the simple RTN quantizer.
\begin{figure}[tb]
\centering
\includegraphics[width=1\textwidth]{Intro.pdf}
\captionsetup{justification=justified}
\caption{(Left) RTN suffers from the degradation caused by quantizing outliers. Preserving 0.5\% of the outliers in FP16 can greatly recover the performance of the quantized model. (Right) NPFT brings significant improvement to the performance of single bit-width models \textbf{without preserving any outliers in FP16}. When applied to OPT 1.3B/2.7B models, our method outperforms RTN baseline by a large PPL margin of over \textbf{2.9}/\textbf{0.9} on the C4 benchmark. }
\label{fig1} % 设置标签以便引用
\vspace{-10pt}
\end{figure}
In summary, we make the following contribution in this paper:
\begin{itemize}
\item We propose NPFT, a PEFT method that efficiently reduces the Hessian trace of outlier weights in LLMs;
\item The reduced Hessian trace enables outlier weights to be quantized to the same precision as other weights, without special treatment, while still preserving good quantized model performance;
\item NPFT works with different post-training quantizers, enhancing the performance of quantized models across various bit-width settings.
\end{itemize}
NPFT enables stable perplexity improvements on various uniform and non-uniform quantizers. For example, NPFT helps RTN achieve a 3.69 perplexity improvement on OPT-1.3B-4bits and achieve performance comparable to GPTQ on LLaMA2-7B-4bits. Additionally, by eliminating the need of preserving any outlier weights in FP16, NPFT achieves a 10\% reduction in inference latency on the 4090 GPU.
\section{Related Work}
\textbf{LLM Quantization.} LLM quantization can be categorized into two branches: quantization-aware training (QAT) methods \cite{chen2024efficientqat, liu2023llm, du2024bitdistiller, dettmers2024qlora} and post-training quantization (PTQ) methods \cite{lin2024awq, kim2023squeezellm, dettmers2023spqr, frantar2022gptq, wei2022outlier}. QAT typically requires extensive retraining to recover accuracy after quantization, whereas PTQ does not involve retraining. Although QAT can enhance the performance of quantized models, it is not easily scalable to LLMs due to the significant computational resources required for retraining and the difficulties in convergence. Therefore, most works on LLM quantization focus on PTQ, which is also the focus of our work. The main difference between our proposed fine-tuning method and QAT lies in our objective: instead of training the quantized weights to recover the performance, we aim to regularize the sensitivity of the weights in the floating point model, making the model more suitable for different PTQ methods. Additionally, our method requires significantly less training time compared to QAT. For example, it only takes one hour of training on LLaMA2-7B.
\textbf{Sensitivity-aware PTQ.} There exists a small fraction of weights that are more sensitive to quantization. Quantizing them will lead to significant performance degradation. To address this, sensitivity-aware PTQ methods have been investigated. Some works focus on mitigating outlier activations. For example, \cite{dettmers2022gpt3} preserved outlier activations in floating-point format, while \cite{wei2022outlier} established an outlier suppression framework that transfers outlier factors to other layers. Other works focus on outlier weights, which is also the issue we aim to mitigate in this paper. \cite{dettmers2023spqr} proposed a hybrid sparse-quantized format where the outlier weights are kept in high precision. \cite{kim2023squeezellm} also isolated outliers in a sparse FP16 matrix using a Hessian sensitivity-based non-uniform quantizer. \cite{lin2024awq} extracted the outliers based on activation distribution and performed per-channel scaling to reduce their quantization loss. Even though all these works have achieved promising results, it is important to note that they all place the quantized model in a mixed-precision state, which is unfavorable for hardware deployment. In this work, we aim to reduce the need for special handling of outliers by allowing the model to retain strong performance even when outliers are quantized to the same bit-width.
\textbf{Hessian-aware Quantization.} Previous works \cite{dong2019hawq, dong2020hawq} have shown that the Hessian eigenvalues of the loss function can be used as criteria to determine layer importance in designing mixed-precision quantization schemes.
\cite{yang2022hero} also proved that the model robustness against quantization perturbation can be enhanced by regularizing Hessian eigenvalues. The use of the Hessian to assess the weight sensitivity, or Hessian regularization to improve quantization performance, has been extensively explored in CNN models. However, explicit Hessian regularizations are infeasible for LLMs due to their billions of parameters. In this work, we propose an efficient fine-tuning approach, which can reduce the Hessian trace while bypassing the expensive higher-order gradient calculations required to direct Hessian regularization.
% directly regularize H is no feasible to LLMs, we propose an efficient...
\section{Method}
\subsection{Identifying Outliers by Hessian Sensitivity}
Not all parameters in a neural network contribute equally. In previous works \cite{kim2023squeezellm, dettmers2023spqr}, outliers are defined as weights that have a significant impact on the final loss after quantization. Typically, the sensitivity of an arbitrary entry $w_{i,j}$ in weight $W$ can be calculated as the induced loss increase:
\begin{align}
s_{i,j} = \mathcal{L}(W_{q}) - \mathcal{L}(W)
\end{align}
where $\mathcal{L}$ is the loss function and $W_{q}$ denotes the weight matrix where $w_{i,j}$ is quantized. We can use Taylor expansion to well approximate the loss increase under quantization as:
\begin{align}
\mathcal{L}(W_{q}) - \mathcal{L}(W) \approx g^{T}(W_{q} - W) + \frac{1}{2}(W_{q} - W)^{T} H(W_{q} - W)
\label{eq2}
\end{align}
where $g \in \mathbb{R}^{d \times 1}$ and $H \in \mathbb{R}^{d\times d}$ denote the gradient and Hessian of the loss with respect to $W$. $d$ denotes the number of parameters in the weight matrix. $W_{q}$ and $W$ are flattened into $d \times 1$ vectors in this equation.
For pretrained models that are near convergence, the gradient $g$ approaches zero. The quantization loss will therefore be dominated by the second-order term. In other words, for a fixed quantization bit-width, the sensitivity of weight $w_{i,j}$ is reflected in the corresponding entry in the diagonal of $H$. However, it is infeasible to identify high-sensitivity outliers through direct calculation of $H$, as the computational cost of $H$ for LLMs is prohibitively high.
In this paper, we followed \cite{kim2023squeezellm} and used the Fisher Information Matrix to identify outliers, which can be calculated as:
\begin{align}
H \approx \mathcal{F} = gg^T = ||\partial \mathcal{L} / \partial w_{i,j}||_2^2
\end{align}
for a specific weight element $w_{i,j}$.
After identifying outliers, previous works typically preserve them in FP16 \cite{dettmers2023spqr,kim2023squeezellm} or quantize them to higher bit-widths \cite{lin2024awq} to reduce quantization degradation. In this paper, we aim to address this issue at its root by reducing the sensitivity of outliers.
\subsection{Efficient Hessian Regularization }\label{sec3.2}
To mitigate the loss increase brought by quantizing outliers, a straightforward way is to add a regularization term that can minimize the squared sum of the $H$'s eigenvalues \cite{yang2022hero}, thereby reducing the outliers' sensitivity. However, even with the Fisher approximation, directly regularizing $H$ still requires the computation of higher-order gradients, which is computationally unfeasible for LLMs. In this paper, we propose an efficient approach for $H$ regularization.
Fortunately, previous works have provided insights on computing $\text{Tr}(H)$ without direct access to $H$. Based on the fast trace estimation algorithm proposed in \cite{avron2011randomized}, $\text{Tr}(H)$ can be estimated using sampled random vector $z \in \mathbb{R}^{d}$ whose components are i.i.d sampled from a distribution with zero mean and identity covariance matrix. Specifically, the estimation is derived as:
\begin{align}
\text{Tr}(H) = \text{Tr}(HI) = \text{Tr}(H\mathbb{E}_z[zz^T])=\mathbb{E}_z[z^{T}Hz]
\label{eq6}
\end{align}
where $I$ is the identity matrix and $\mathbb{E}(\cdot)$ is the expectation.
Considering the random vector $z$ as a perturbation added to the converged weight matrix $W$, the expected loss increase induced by the weight perturbation can be approximated with Taylor expansion, similar to Equation~(\ref{eq2}), as
\begin{equation}
\mathbb{E}_z[\mathcal{L}(W+z) - \mathcal{L}(W)] \approx \frac{1}{2}\mathbb{E}_z[z^{T} Hz]=\frac{1}{2}\text{Tr}(H),
\label{equ:perturb}
\end{equation}
where the first-order term in the Taylor expansion is ignored given the convergence assumption.
As we focus on the weight sensitivity to quantization, we consider a distribution of weight perturbation $z$ that can mimic the impact of quantization on weight values.
Specifically, given the quantization bin width as $\Delta$, the round-to-nearest function will change $w_{i,j}$ by at most $\Delta/2$. This suggests that we can represent quantized weight $W_q$ as $W + \delta$, where $\delta \in \mathbb{R}^d,\|\delta\|_{\infty}<\Delta / 2$.
Therefore, we approximate the distribution of $\delta$ with random weight perturbation $z$ sampled from a zero-mean uniform distribution between $[-\Delta/2, \Delta/2]$. Note that the covariance of $z$ is $\frac{\Delta^2}{12} I$, which is proportional to identity matrix $I$. We can therefore estimate the Hessian trace following the derivation in Equation~(\ref{eq6}) and (\ref{equ:perturb}) as
\begin{equation}
\text{Tr}(H) \propto \mathbb{E}_{z\sim \mathcal{U}[-\Delta/2, \Delta/2]}[\mathcal{L}(W+z) - \mathcal{L}(W)].
\end{equation}
%If we randomly sample multiple noise $\delta_i$ with zero means within the bin and add them to outliers, this would be equivalent to quantizing the outliers while also ensuring that
%$\delta_{i}$ possesses all the properties of $z$, which gives:
%\begin{align}
% \text{Tr}(H) =\textbf{E}[\delta_i^{T}H\delta_i]
%\end{align}
%Combined with equation \ref{eq2}, we have:
%\begin{align}
% \textbf{E}[\mathcal{L}(W+\delta_i) - \mathcal{L}(W)] \simeq \frac{1}{2}\textbf{E}[\delta_i^{T} H\delta_i]=\frac{1}{2}\text{Tr}(H)
% \label{eq5}
%\end{align}
%Note that this is obtained under the assumption of converged original model ($g=0$). The equation \ref{eq5} indicates that the loss increase under perturbation $\delta_i$ is an estimation of $\text{Tr}(H)$. And we can sample more $\delta_{i}$ to increase the estimation confidence, represented as:
%\begin{align}
% \text{Tr}(H) \approx \frac{2}{m}\sum_{i=1}^{m}[\mathcal{L}(W+\delta_i) - \mathcal{L}(W)]
%\end{align}
Based on the fact that $\text{Tr}(H)$ will be dominated by the outliers of $W$ since their hessian sensitivity is typically more than 100 times greater than that of other weights in terms of scale, it is feasible to reduce outliers sensitivity if we can minimize $\mathbb{E}_z \mathcal{L}(W+z)$ under weight perturbation $z$ applied to the outlier locations. By fine-tuning the model to be robust to weight perturbations on outlier locations, we achieve a computationally-efficient implicit regularization on the Hessian matrix.
% \begin{theorem}
% \end{theorem}
% Specifically, the process of quantization can be seen as a way of additive perturbation.
% \begin{assumption}\label{ass1}
% If we assume that the original model is converged. Then the gradient $g$ will be zero and $H$ will be positive semi-definite. Let $v_i$ denote the eigenvectors of $H$.
% $\delta$ will always be in the subspace spanned by $v_i$ \cite{dong2020hawq}, represented as:
% \begin{align}
% \delta = \alpha \sum_{i=1}^{n} v_{i}
% \end{align}
% where $n = d^2$ and $\alpha$ is a constant based on the quantization range.
% \end{assumption}
% Based on assumption \ref{ass1}, we have the following lemma.
% \begin{lemma} \label{lemma 1}
% Suppose we quantize the outliers and denote the quantized weight by $W + \delta$. Then under assumption \ref{ass1} we will obtain:
% \begin{align}
% \mathcal{L}(W+\delta) - \mathcal{L}(W) \simeq \frac{1}{2}\alpha^2 \text{Tr}(H)
% \end{align}
% \end{lemma}
% \textbf{Proof.} The second order term in equation \ref{eq2} can be represented as:
% \begin{align}
% \frac{1}{2}\delta ^T H\delta = \frac{1}{2}\sum_{i=1}^{n}\alpha^2v_i^T H v_i = \frac{1}{2}\alpha^2 \sum_{i=1}^{n}\lambda_i
% \end{align}
% The lemma holds because $H$ is a square matrix and the sum of eigenvalues equals to the trace of the matrix. And the $\text{Tr}(H)$ will be dominated by the outliers of $W$ since their hessian sensitivity is typically more than 100 times greater than that of other weights in terms of scale.
% If we use the fact that the original model is converged and assume that the cross-weight interactions are negligible, equation \ref{eq2} can be simplified as:
% \begin{align}
% \mathcal{L}(W_{q}) - \mathcal{L}(W) \simeq \frac{1}{2}(W_{q} - W)^{T} \text{Trace}(H)(W_{q} - W)
% \label{eq3}
% \end{align}
% Previous work has provided insights into the robustness of model under quantization \cite{yang2022hero}.
% Specifically, the process of quantization can be seen as a way of additive perturbation. For weight $w_{i,j}$, the quantized value $w_{i,j}^{q}$ is defined as:
% \begin{align}
% w_{i,j}^{q}= \alpha \times \text{Round} (\frac{w}{\alpha}), \alpha = \frac{\max(|w_{:j}|)}{2^{N-1}}
% \end{align}
% where $N$ is the number of bits, and $\alpha$ is the quantization scaler. Given the quantization bin width as $\Delta$, the rounding function will change $w_{i,j}$ by at most $\Delta/2$. Thus, the weight perturbation induced by quantization is bounded by $l_{\infty}$ norm, as $\|\delta\|_{\infty} = \|W_{q} - W\|_{\infty}\leq\Delta/2$ and we can bound the loss increase introduced by quantization as:
% \begin{align}
% \mathcal{L}(W_{q}) - \mathcal{L}(W) \leq \max_{\|\delta\|_{\infty}\leq\Delta/2} \mathcal{L}(W + \delta) - \mathcal{L}(W)
% \end{align}
% Substituted to equation \ref{eq2} and assume $g$ approaches zero we obtain:
% \begin{align}
% \frac{1}{2}\delta^{T} H\delta \leq \max_{\|\delta\|_{\infty}\leq\Delta/2} \mathcal{L}(W + \delta) - \mathcal{L}(W)
% \label{eq5}
% \end{align}
% The equation (5) indicates that if we can reduce $\mathcal{L}(W + \delta)$ under the given perturbation, we can decrease the upper bound, thus achieving regularization of $H$. This does not require computing $H$, but can be achieved through a parameter-efficient fine-tuning approach, which is less computationally expensive.
\subsection{Noise Perturbation Fine-tuning}
Following the conclusion in section \ref{sec3.2}, the goal of regularizing $H$ is converted into minimize the expected loss under randomly sampled perturbation $z$. Following the stochastic gradient descent process, we add one independent sample of $z_i$ to the weight in each step of model fine-tuning to fulfill the expected loss computation. Our implementation adopts a two-phase setting as follows.
\textbf{Per-channel Noise Sampling.} At the beginning of each epoch, the outlier positions will be identified using the Fisher matrix $\mathcal{F}$ and perturbation ratio $\gamma$. Then $z_i$ will be randomly sampled and added to the corresponding outlier weights in $W$. Previous work \cite{dettmers2023spqr} has shown that the outliers exhibit strong channel correlations, i.e., outliers are usually concentrated in certain columns of $W$. Therefore, $z_i$ can be sampled per channel and added to all outliers in this channel. The process of sampling and applying noise can be expressed using the following Torch-style pseudocode \footnote{Note that in Algorithm \ref{alg1}, $W$ and $z_i$ are in matrix form with dimensions $\sqrt{d}\times\sqrt{d}$. } :
\begin{algorithm}
\caption{Sample and apply noise}\label{alg1}
\begin{algorithmic}[1]
\State \textbf{Input:} weight matrix $W$, Perturbation ratio $\gamma$, Calibration data $X$,
\State \textbf{Output:} weight with noise $W$ +$z_i$
\State Calculate $\mathcal{F}$ using $X$
\State $outlier\_positions = \mathrm{Filter}(\mathcal{F},\gamma)$ \emph{\# Obtain top $\gamma$\% sensitive positions}
\State $W_o = \mathrm{torch.zeros}(W.shape)$, $z_i = \mathrm{torch.zeros}(W.shape)$
\For{ $position$ in $outlier\_positions$}
\State $W_o[position] = W[position]$ \emph{\# Obtain outlier weights}
\EndFor
\State $nonzero\_channels = \mathrm{torch.nonzero\_columns}(W_o)$ \emph{\# Obtain the indices of all channels that contain outliers}
\For{$col\_idx$ in $nonzero\_channels$}
\State $noise = \mathrm{torch.rand\_like}(W[:col\_idx]) * (W[:col\_idx].\text{max} - W[:col\_idx].\text{min}) + W[:col\_idx].\text{min}$ \emph{\# Randomly sample noise on this channel}
\State $noise-=noise.\text{mean}$ \emph{\# Ensure that noise has zero mean}
\State $z_i[:col\_idx] = noise$
\EndFor
\State $W+=z_i$ \emph{\# Add noise to $W$}
\State \textbf{return} $W$
\end{algorithmic}
\end{algorithm}
\textbf{Parameter Efficient Fine-tuning.} For efficient fine-tuning, we apply the LoRA\cite{hu2021lora} adapters on both self-attention and MLP weights, and merge LoRA weights to the base model after fine-tuning. This pipeline is shown more clearly in Fig. \ref{fig2}. The weight perturbation is added on the corresponding location of the base model weight, so that the gradient computation with respect to the LoRA weight does not require any approximation. To ensure that base model performance is not excessively compromised, we also introduce a weighted $\mathcal{L}(W)$. The overall training objective is:
\begin{align}
\min_{U,V}[\mathcal{L}(W+z_{i} + U^T V)+\beta \mathcal{L}(W+U^T V)],
\end{align}
where $z_i$ is the weight perturbation following Algorithm~\ref{alg1} and $U,V$ are the LoRA weights.
The proposed fine-tuning process enables the outliers to be less sensitive to the quantization noise, thereby improving the model performance after post-training quantization.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.95\textwidth]{main.pdf} % 图片文件名及路径
\captionsetup{justification=justified}
\caption{(Left) Existing PTQ methods preserve outliers in FP16 to prevent significant performance degradation, but the mixed-precision
format is not hardware-friendly. (Right) Our proposed NPFT regularizes the outliers' sensitivity through efficient fine-tuning, which enhances the performance of single bit-width quantized models without special treatment to the outliers.}
\label{fig2} % 设置标签以便引用
\vspace{-0.7cm}
\end{figure}
\section{Evaluations}
\subsection{Experiment Setup}
\textbf{Models and Datasets.}
We perform an extensive evaluation of NPFT across a range of models, including LLaMA \cite{touvron2023open} and OPT
\cite{zhang2022opt} models. We carry out language modeling evaluations using the C4 dataset \cite{raffel2020exploring} and the WikiText2 dataset \cite{merity2016pointer}.
\textbf{Baseline Methods.} We compare NPFT against various methods PTQ methods including RTN, GPTQ \cite{frantar2022gptq}, AWQ \cite{lin2024awq} and Squeeze LLM \cite{kim2023squeezellm}. Unless otherwise mentioned, we use Squeeze LLM without outlier retention as baseline. To conduct fair comparison, we quantize the fine-tuned model using existing quantizers (both uniform and non-uniform) and ensure that the quantizer settings match those in the original works. The results in the experiments are either taken from the original paper or obtained by running its open-source code.
\textbf{Efficiency Profiling}. We further compare the latency and memory usage of saving and not saving outliers as FP16 using sqLLM quantizer. Specifically, we measure the latency for generating 128 tokens on a NVIDIA RTX 4090 GPU, using the same optimized cuda kernel in \cite{kim2023squeezellm}. As NPFT indeed introduces additional training overhead, we also compare the fine-tuning time cost and memory usage of NPFT with QAT method EfficientQAT \cite{chen2024efficientqat}.
\textbf{Hyperparameters.} To filter the outliers, we use a calibration dataset comprising 128 randomly selected segments of 512 tokens each from the C4 \cite{raffel2020exploring} dataset. In the OPT experiments, we identify 0.5\% of the sensitive weights as outliers and applied noise to them. The OPT models are fine-tuned for 6 epochs with a learning rate of 5e-6, while in the LLaMA experiments, the perturbation ratio is set to 0.05\%, and the model is fine-tuned for 3 epochs with a learning rate of 5e-5. The $\beta$ is set to $0.5$ for all experiments. Noise is applied in an output-channel-wise manner.
\subsection{Main Results}
Tab. \ref{Tab1} shows quantization results for OPT along with representative PTQ methods. Specifically, we use uniform quantizers RTN and GPTQ, as well as the non-uniform quantizer sqLLM to quantize our fine-tuned models and compare PPLs with each quantizer's original results. Note that for sqLLM we only used its dense-only setting, where both normal weights and outliers were quantized. Our NPFT can bring significant improvements to the uniform quantizer. In the OPT-1.3B-4bits experiment, NPFT achieves a 3.69 PPL degradation for RTN on the C4 dataset and 11.89 on WikiText. For GPTQ, NPFT can improve the performance of OPT-2.7B on C4 by approximately 8.5\%. SqLLM, as the strongest non-uniform PTQ baseline, can still have its performance further improved by NPFT in the single precision setting. The relatively lower performance enhancement is due to the fact that this quantizer has already reduced the quantization error of outlier weights.
In Tab. \ref{Tab2}, we observe that this pattern extends to 7B models. It is worth noting that with the assistance of NPFT, RTN can achieve performance on par with GPTQ for LLaMA-2-7B-4bits.
We solely use a small fraction of the C4 dataset as calibration data to filter and fine-tune outlier weights. Meanwhile, NPFT also improves the performance when evaluated on unseen datasets like WikiText.
%NPFT does not necessarily lead toimprovements on WikiText, as in our experiment, the calibration dataset used to filter outliers was solely a small fraction of C4.
\begin{table}[htbp]
\centering
\captionsetup{justification=justified}
\caption{Performance comparison of various quantization methods for OPT-1.3B (top)/2.7B (bottom) on 3-bit and 4-bit settings. Both uniform and non-uniform quantizers are adopted.}
\label{Tab1}
\small
\begin{tabular}{cccccccc}
\toprule
\textbf{OPT-1.3B} & \multicolumn{3}{c}{\textbf{3-bit}} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} \\
\midrule
Baseline & 16 & 14.72 & 14.62 & 16 & 14.72 & 14.62 \\
\midrule
RTN & 3 & inf. & inf. & 4 & 24.68 & 48.19 \\
\rowcolor{cyan!10} RTN + NPFT (0.5\%) & 3 & inf. & inf. &4 & \textbf{21.74} & \textbf{36.30} \\
GPTQ & 3 & 21.63 & 20.97 & 4 & 16.97& 15.47 \\
\rowcolor{cyan!10} GPTQ + NPFT (0.5\%) & 3 &\textbf{19.82} &\textbf{20.71} &4 &\textbf{15.57} &15.50\\
sqLLM & 3.04 & 16.42 & 16.30 & 4.09 & 15.01 & 14.94 \\
\rowcolor{cyan!10} sqLLM + NPFT (0.5\%) & 3.04 &\textbf{16.40} &16.30 &4.09 &\textbf{14.87} &\textbf{14.77}\\
\bottomrule
\end{tabular}
\begin{tabular}{cccccccc}
\toprule
\textbf{OPT-2.7B} & \multicolumn{3}{c}{\textbf{3-bit}} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} \\
\midrule
Baseline & 16 & 13.17 & 12.47 & 16 & 13.17 & 12.47 \\
\midrule
RTN & 3 & inf. & inf. & 4 & 17.52 & 16.92 \\
\rowcolor{cyan!10} RTN + NPFT (0.5\%) & 3 & inf. & inf. &4 & \textbf{16.58} & \textbf{16.32} \\
GPTQ & 3 & 18.17 & 16.88 & 4 & 15.00& 12.87 \\
\rowcolor{cyan!10} GPTQ + NPFT (0.5\%) & 3 &\textbf{16.59} &\textbf{16.85} &4 &\textbf{13.76} &12.88\\
sqLLM & 3.04 & 14.45 & 13.85 & 4.07 & 13.38 & 12.80 \\
\rowcolor{cyan!10} sqLLM + NPFT (0.5\%) & 3.04 &\textbf{14.39} &\textbf{13.79} &4.07 &\textbf{13.37} &\textbf{12.74}\\
% GPTQ (g128) & 3.24 & 7.97 & 6.25 & 4.24 & 7.23 & 5.72 \\
% AWQ (g128) & 3.24 & 7.84 & 6.24 & 4.24 & 7.15 & 5.72 \\
% \rowcolor{cyan!10} SqueezeLLM (0.45\%) & 3.24 & \textbf{7.51} & \textbf{5.96} & 4.27 & \textbf{7.08} & \textbf{5.57} \\
\bottomrule
\end{tabular}
\vspace{-15pt}
\end{table}
\begin{table}[htbp]
\centering
\captionsetup{justification=justified}
\caption{Performance comparison of various quantization methods for LLaMA2-7B on 3-bit and 4-bit settings. Both uniform and non-uniform quantizers are adopted.}
\small
\begin{tabular}{cccccccc}
\toprule
\textbf{LLaMA2-7B} & \multicolumn{3}{c}{\textbf{3-bit}} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} \\
\midrule
Baseline & 16 & 6.97 & 5.47 & 16 & 6.97 & 5.47 \\
\midrule
RTN & 3 & 404.45 & 542.86 & 4 & 7.72 & 6.12 \\
\rowcolor{cyan!10} RTN + NPFT (0.05\%) & 3 & \textbf{224.39} & \textbf{320.64} &4 & \textbf{7.42} & \textbf{6.08} \\
GPTQ & 3 & 10.45 & 8.97 & 4 & 7.42 & 5.90 \\
\rowcolor{cyan!10} GPTQ + NPFT (0.05\%) & 3 &\textbf{10.22} &8.99 &4 &\textbf{7.40} &5.94\\
AWQ & 3.24 & 7.84 & 6.24 & 4.24 & 7.15 & 5.72 \\
sqLLM & 3.02 & 7.72 & 6.18 & 4.05 & 7.12 & 5.62 \\
\rowcolor{cyan!10} sqLLM + NPFT (0.05\%) & 3.02 &\textbf{7.69} &\textbf{6.12} &4.05 &\textbf{7.08} &\textbf{5.60}\\
\bottomrule
\end{tabular}
\vspace{0.1cm}
\label{Tab2}
\vspace{-15pt}
\end{table}
\subsection{Theoretical Insight Verification}
\textbf{Reduction of Outliers Sensitivity.}
In Fig. \ref{fig3}, we illustrate the changes in outlier sensitivity after applying NPFT. Previous literature has established that the distribution of outliers in the weight matrix tends to exhibit a channel-wise pattern \cite{dettmers2023spqr}. Therefore, we randomly select an input channel from the weight matrix of OPT-1.3B and unfold it along the output channels to compare the sensitivity of outliers. It is clearly shown that after NPFT, the sensitivity of outliers (both in MLP and self-attention layers) is significantly reduced. This aligns with the conclusions derived in Section \ref{sec3.2}.
This also explains why NPFT enhances the performance of quantized models: NPFT helps reduce the performance degradation caused by quantizing outliers.
% \textbf{Increase of normal weights sensitivity.} We also illustrate the changes in the normal weights sensitivity in Fig. \ref{fig4}. Since the sensitivity magnitude of individual normal weights is relatively low, we calculate the sum of the sensitivities of normal weights for the corresponding matrix in every layer. The sensitivity of normal weights has not significantly changed overall. However, notably, the sensitivity has increased in certain middle layers, suggesting that after reducing the sensitivity of outliers, normal weights play a more important role for generating the final output.
\begin{figure}[htbp]
\centering
\includegraphics[width=1\textwidth]{sensitivity_comparison.pdf} % 图片文件名及路径
\caption{Visualization of outlier sensitivity in OPT-1.3B. The sensitivity is obtained by calculating $\mathcal{F}$. After fine-tuning, the outliers become \textbf{less} sensitive compared to the original model.}
\label{fig3} % 设置标签以便引用
\end{figure}
% \begin{figure}[htbp]
% \centering
% \includegraphics[width=1\textwidth]{figs/normal_weights.eps} % 图片文件名及路径
% \caption{Visualization of normal weights sensitivity across layers in OPT-1.3B. After finetuning, some of the normal weights become \textbf{more} sensitive compared to the original model.}
% \label{fig4} % 设置标签以便引用
% \end{figure}
\subsection{Efficiency Profiling}
\textbf{Inference Efficiency.} In Tab. \ref{Tab3}, we present the latency and peak GPU memory usage when using the sqLLM quantizer, comparing scenarios with and without retaining full-precision outliers. The outliers are stored using the sparse format described in \cite{kim2023squeezellm} and utilize the corresponding kernel for inference. When generating 128 tokens on a single 4090 GPU, the uniformly-quantized OPT-1.3B-4bits model trained with NPFT achieves a 10\% reduction in latency, lower CUDA memory usage, and demonstrates better performance compared to the model with outliers retained. A similar trend is observed in LLaMA2-7B-4bits model.
\textbf{Training Efficiency.}
We also compare the computational overhead with the latest QAT method EfficientQAT \cite{chen2024efficientqat}, whose pipeline also follows a two-phase setting (Block-AP and E2E-QP). As shown in Tab. \ref{Tab4}, NPFT's training time for LLaMA2-7B is approximately one-fourth that of EfficientQAT. It is worth noting that, unlike EfficientQAT, which requires dedicated training for each bit-width model, NPFT achieves optimization for multiple bit-width models in one-shot. This not only significantly simplifies the training procedure but also greatly enhances the efficiency of model quantization. NPFT requires significantly fewer samples and shorter sequence lengths for calibration data (shown in Tab. \ref{Tab5}), enabling us to perform full-parameter fine-tuning of LLaMA2-7B on a single V100 GPU.
The overhead of NPFT mainly comes from memory usage, as the model needs to be loaded into memory when calculating the noise. This memory consumption can be addressed through layer-wise computation in future work.
\begin{table}[tb]
\centering
\captionsetup{justification=justified}
\caption{Latency (s) and peak cuda memory usage (GB) of 4-bit models when generating 128 tokens on a 4090 GPU. NPFT enables single-precision models to achieve the same level of perplexity (PPL) as mixed-precision models, while also offering better inference efficiency. }
\label{Tab3}
\small
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{3}{c}{\textbf{OPT-1.3B-4bits}} & \multicolumn{3}{c}{\textbf{LLaMA2-7B-4bits}}\\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
& \textbf{Lat. (s)} & \textbf{cuda Mem. (G)} & \textbf{PPL(C4)} & \textbf{Lat. (s)} & \textbf{cuda Mem. (G)} & \textbf{PPL(C4)} \\
\midrule
sqLLM & 2.40 & 1.08 & 15.01 & 4.80 & 4.46 & 7.12 \\
sqLLM w/0.5\% ol. & 2.62 & 1.09 & 14.94 & 5.01 & 4.74 & 7.08 \\
\rowcolor{cyan!10} sqLLM + NPFT & \textbf{2.40} & \textbf{1.08} & \textbf{14.87} & \textbf{4.80} & \textbf{4.46} & \textbf{7.08} \\
\bottomrule
\end{tabular}
\vspace{-10pt}
\end{table}
\begin{table}[h]
\centering
\vspace{-10pt}
\captionsetup{justification=justified}
\caption{Training time and memory comparison on LLaMA2-7B. NPFT can fine-tune multiple bit-width models in one shot, requiring significantly less training time. }
\footnotesize
\begin{tabular}{cccccc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{Phase 1}} & \multicolumn{2}{c}{\textbf{Phase 2}}& \multirow{2}{*}{\textbf{Total Time (h)}} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& \textbf{Time (h)} & \textbf{Mem. (G)} & \textbf{Time (h)} & \textbf{Mem. (G)} \\
\midrule
EfficientQAT (3bits) & 3.3& 8.5 & 1.5&6.4&4.8 \\
EfficientQAT (4bits) & 3.3& 8.5 & 1.5 &7.0&4.8 \\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (0.05\%) & \textbf{0.63} &15.23 &\textbf{0.40} &6.11 &\textbf{1.03}\\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (0.5\%)& \textbf{0.78} & 15.23 &\textbf{0.40}&6.11&\textbf{1.18 }\\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (1\%)& \textbf{0.84} &15.23 &\textbf{0.40}&6.11&\textbf{1.24 }\\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (2\%)& \textbf{0.91} &15.23 &\textbf{0.40}&6.11&\textbf{1.31 }\\
\bottomrule
\end{tabular}
\vspace{0.1cm}
\label{Tab4}
\end{table}
\begin{table}[tb]
\centering
\captionsetup{justification=justified}
\caption{Calibration data comparison on LLaMA2-7B. NPFT requires fewer training samples and shorter sequence lengths, thereby reducing hardware requirements.}
\label{Tab5}
\small
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{Phase 1}} & \multicolumn{2}{c}{\textbf{Phase 2}}& \multirow{2}{*}{\textbf{HW Req.}} & \multirow{2}{*}{\textbf{PPL (Wiki)}} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& \textbf{samples} & \textbf{Seqlen} & \textbf{samples} & \textbf{Seqlen} \\
\midrule
EfficientQAT (3bits) & 4096& 2048 & 4096&4096&A100 (80G)&\textbf{5.81}\\
\rowcolor{cyan!10}NPFT w/ sqllm (3bits)& \textbf{128} &\textbf{512} &\textbf{128} &\textbf{512} &\textbf{V100 (32G)}&6.12\\
EfficientQAT (4bits) & 4096& 2048 & 4096 &4096&A100 (80G)&\textbf{5.53}\\
\rowcolor{cyan!10}NPFT w/ sqllm (4bits)& \textbf{128} &\textbf{512} &\textbf{128} &\textbf{512} &\textbf{V100 (32G)}&5.60\\
\bottomrule
\end{tabular}
\vspace{0.1cm}
\vspace{-10pt}
\end{table}
\subsection{Ablation Study}
\textbf{Increasing Perturbation Ratio.} In Fig. \ref{Fig4}, we show the changes in model performance as the perturbation ratio increases. Note that all groups of the same model are controlled for the same number of training steps. When the perturbation ratio is relatively low (less than 2\%), the model's performance does not vary significantly. However, as the ratio increases, it introduces challenges to model convergence, indicating a need for longer training times. LLaMA2-7B is more robust to perturbation than OPTs.
\begin{figure}[tb]
\vspace{10pt}
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{ablation_perturbation.pdf}
\captionsetup{justification=justified} % 单独设置该图的caption两端对齐
\caption{Impact of perturbation ratio on model performance.}\label{Fig4}
\end{minipage}%
\hspace{0.5cm} % 控制两个图片之间的水平间距
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{ablation_beta.pdf}
\captionsetup{justification=justified}
\caption{Training loss curves of OPT-1.3B. The perturbed model converges more swiftly with $\beta >0$.}\label{Fig5}
\end{minipage}
\vspace{-0.4cm}
\end{figure}
\textbf{Effectiveness of Induced Base Model Loss.} To retain base model performance, we induce a balanced $\mathcal{L}(W)$ during fine-tuning. As illustrated in Fig. \ref{Fig5}, The difference between the loss curves with and without $\beta \mathcal{L}(W)$ suggests that it benefits the model's convergence under perturbation.
\textbf{Perturbation on Different Layers.} In Tab. \ref{Tab6}, we compare the 4-bit performance changes of the model when perturbing only the attention layers, only the MLP layers, and all layers together. The results show that applying perturbations to all layers yields better performance. Benefiting from LoRA, fine-tuning all layers does not result in a significant increase in training time.
\begin{table}[h]
\vspace{-10pt}
\centering
\captionsetup{justification=centering}
\caption{Model performance under perturbations applied to different layers. Applying perturbations to all layers yields the best results. }
\small
\begin{tabular}{cccccc}
\toprule
\textbf{Model} & \textbf{self\_attn} & \textbf{mlp} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} & \textbf{Training Time (h)}\\
\midrule
\multirow{3}{*}{OPT-1.3B-4bits} & \checkmark & $\times$ &24.03 &49.04 &0.39\\
& $\times$ & \checkmark &23.83 &37.44 &0.39\\
& \checkmark \cellcolor{cyan!10} & \checkmark \cellcolor{cyan!10}&\textbf{21.74} \cellcolor{cyan!10}&\textbf{36.30}\cellcolor{cyan!10} &0.40\cellcolor{cyan!10}\\
\midrule
\multirow{3}{*}{OPT-2.7B-4bits} & \checkmark & $\times$ &16.96 &16.59 &0.69\\
& $\times$ & \checkmark &16.94 &16.53 &0.70\\
& \checkmark \cellcolor{cyan!10}& \checkmark \cellcolor{cyan!10}&\textbf{16.58} \cellcolor{cyan!10}&\textbf{16.32}\cellcolor{cyan!10} &0.72 \cellcolor{cyan!10}\\
\bottomrule
\end{tabular}
\vspace{-10pt}
\label{Tab6}
\end{table}
\section{Conclusion}
This work introduces Noise Perturbation Fine-tuning (NPFT), an efficient method to reduce the sensitivity of outlier weights by applying random perturbations during fine-tuning. By reducing the loss Hessian trace, NPFT improves quantized model performance without requiring special treatment for outliers, enhancing both uniform and non-uniform quantizers. Experiments on OPT and LLaMA models demonstrate consistent performance gains and improved inference efficiency. Future work will focus on further optimizing NPFT for larger models and exploring its integration with other quantization techniques to enhance quantization robustness.
\section*{Acknowledgements}
This work was supported in part by the research collaboration grant from TetraMem, Inc. This work was based upon High Performance Computing (HPC) resources supported by the University of Arizona TRIF, UITS, and Research, Innovation, and Impact (RII) and maintained by the UArizona Research Technologies department.
\bibliography{nips2019}
\bibliographystyle{plain} % 或其他样式如 plain, alpha 等
\newpage
\appendix
\section{Performances under different noise distributions and samling methods.}\label{appA}
We compare different noise distributions and sampling methods in Tab. \ref{Tab7}, with uniform noise and output-channel sampling yielding the best improvements. The noise is effective, regardless of its distribution, as long as it satisfies the zero mean and $Cov \propto I$. The output-channel sampling performs better because outliers are densely concentrated in specific output channels. In contrast, the input-channel method considers only a few weights when computing the scale, leading to a poor approximation of the true quantization.
\begin{table}[htbp]
\centering
\captionsetup{justification=centering}
\caption{Experimental results of different noise.}
\label{Tab7}
\small
\begin{tabular}{ccccc}
\toprule
\textbf{OPT-1.3B} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} \\
\midrule
Baseline & 16 & 14.72 & 14.62 \\
\midrule
RTN & 4 & 24.68 & 48.19 \\
\rowcolor{cyan!10} RTN + NPFT (Uniform + o channel) &4 & \textbf{21.74} & \textbf{36.30} \\
\rowcolor{cyan!10} RTN + NPFT (Gaussian + o channel) &4 & \textbf{22.09} & \textbf{35.37} \\
\rowcolor{cyan!10} RTN + NPFT (Laplace + o channel) &4 & \textbf{23.18} & \textbf{36.80} \\
\rowcolor{cyan!10} RTN + NPFT (Uniform + i channel) &4 & \textbf{24.48} & \textbf{38.42} \\
\bottomrule
\end{tabular}
\begin{tabular}{ccccc}
\toprule
\textbf{OPT-2.7B} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} \\
\midrule
Baseline & 16 & 13.17 & 12.47 \\
\midrule
RTN & 4 & 17.52 & 16.92 \\
\rowcolor{cyan!10} RTN + NPFT (Uniform + o channel) &4 & \textbf{16.58} & \textbf{16.32} \\
\rowcolor{cyan!10} RTN + NPFT (Gaussian + o channel) &4 & \textbf{17.04} & \textbf{16.60} \\
\rowcolor{cyan!10} RTN + NPFT (Laplace + o channel) &4 & \textbf{17.00} & \textbf{16.55} \\
\rowcolor{cyan!10} RTN + NPFT (Uniform + i channel) &4 & \textbf{17.27} & \textbf{17.04} \\
\bottomrule
\end{tabular}
\vspace{-15pt}
\end{table}
\section{NPFT on common-sense reasoning tasks.}\label{appB}
We further validated our method on 3 different downstream tasks, including PIQA (Physics), ARC (Science and logic), and Storycloze (Story coherence). The results (accuracy \%) are shown in Tab. \ref{Tab8}. Consistent improvements are observed on all 3 tasks.
\begin{table}[htbp]
\centering
\captionsetup{justification=centering}
\caption{OPT models performances on different common-sense reasoning tasks}
\label{Tab8}
\small
\begin{tabular}{cccccccc}
\toprule
\textbf{OPT-1.3B} & \multicolumn{3}{c}{\textbf{3-bit}} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
\textbf{Method} & \textbf{PIQA} & \textbf{ARC} & \textbf{Storycloze} & \textbf{PIQA} & \textbf{ARC} & \textbf{Storycloze} \\
\midrule
Baseline & 72.36 & 50.93 & 70.78 & 72.36 & 50.93 & 70.78 \\
\midrule
RTN & 52.77 & 27.97 & 47.61 & 67.63 & 49.20 & 59.13 \\
\rowcolor{cyan!10} RTN + NPFT (0.5\%)& \textbf{54.57} & \textbf{28.07} & 47.10 & \textbf{68.61} & \textbf{50.25} & \textbf{63.40} \\
GPTQ & 68.34 & 46.17 & 65.25 & 70.73 & 59.97 & 69.64 \\
\rowcolor{cyan!10} GPTQ + NPFT (0.5\%) & 68.01 & \textbf{51.12} & \textbf{66.10} & \textbf{70.78} & \textbf{60.11 } & \textbf{69.96} \\
\bottomrule
\end{tabular}
\begin{tabular}{cccccccc}
\toprule
\textbf{OPT-2.7B} & \multicolumn{3}{c}{\textbf{3-bit}} & \multicolumn{3}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
\textbf{Method} & \textbf{PIQA} & \textbf{ARC} & \textbf{Storycloze} & \textbf{PIQA} & \textbf{ARC} & \textbf{Storycloze} \\
\midrule
Baseline & 74.81 & 54.34 & 71.74 & 74.81 & 54.34 & 71.74 \\
\midrule
RTN & 51.90 & 26.05 & 46.98 & 73.72 & 52.90 & 70.78 \\
\rowcolor{cyan!10} RTN + NPFT (0.5\%)& \textbf{52.72} & \textbf{26.60} & 46.91 & \textbf{73.77} & \textbf{59.09} & \textbf{71.16} \\
GPTQ & 71.38 & 48.19 & 68.43 & 73.99 & 53.11 & 70.46 \\
\rowcolor{cyan!10} GPTQ + NPFT (0.5\%) & \textbf{71.57} & \textbf{54.12} & 68.11 & 73.66 & \textbf{59.55} & \textbf{71.23}\\
\bottomrule
\end{tabular}
\vspace{-15pt}
\end{table}
\section{NPFT using various calibration datasets. }\label{appC}
We conducted additional experiments with various calibration datasets, including C4, Wiki, and Ptb, as shown in Tab. \ref{Tab9}. NPFT consistently improves performance across all datasets. The calibration data in our method identifies outliers, which exhibit generality in language generation tasks, remaining consistent across datasets as the most sensitive weights. These results demonstrate NPFT's robustness in regularizing universal outliers.
\begin{table}[htbp]
\centering
\captionsetup{justification=centering}
\caption{OPT models performances on different calibration datasets.}
\label{Tab9}
\small
\begin{tabular}{cccccc}
\toprule
\textbf{OPT-1.3B} & \multicolumn{4}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-5}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} &\textbf{PPL (Ptb)} \\
\midrule
Baseline & 16 & 14.72 & 14.62 & 20.29\\
\midrule
RTN & 4 & 24.68 & 48.19 &57.30\\
\rowcolor{cyan!10} RTN + NPFT (C4) &4 & \textbf{21.74} & \textbf{36.30} &\textbf{29.31}\\
\rowcolor{cyan!10} RTN + NPFT (Wiki) &4 & \textbf{22.08} & \textbf{36.33} &\textbf{32.53}\\
\rowcolor{cyan!10} RTN + NPFT (Ptb) &4 & \textbf{22.59} & \textbf{35.76} &\textbf{29.58}\\
\bottomrule
\end{tabular}
\begin{tabular}{cccccc}
\toprule
\textbf{OPT-2.7B} & \multicolumn{4}{c}{\textbf{4-bit}} \\
\cmidrule(lr){2-5}
\textbf{Method} & \textbf{Avg. Bits} & \textbf{PPL (C4)} & \textbf{PPL (Wiki)} &\textbf{PPL (Ptb)}\\
\midrule
Baseline & 16 & 13.17 & 12.47 & 17.97\\
\midrule
RTN & 4 & 17.52 & 16.92 & 31.05\\
\rowcolor{cyan!10} RTN + NPFT (C4) &4 & \textbf{16.58} & \textbf{16.32} & \textbf{21.21}\\
\rowcolor{cyan!10} RTN + NPFT (Wiki) &4 & \textbf{17.04} & \textbf{16.58} & \textbf{21.63}\\
\rowcolor{cyan!10} RTN + NPFT (Ptb) &4 & \textbf{17.20} & \textbf{16.80} & \textbf{21.89}\\
\bottomrule
\end{tabular}
\vspace{-15pt}
\end{table}
\section{Inference improvements of different context lengths.}
As long-context processing is crucial for LLMs, we also added results with varying lengths in Tab. \ref{Tab10}. The results show that the latency speedup of our method increases with longer outputs, achieving over a 3s improvement when generating 2048 tokens on RTX4090.
\begin{table}[htpb]
\centering
\captionsetup{justification=centering}
\caption{OPT-1.3B-4bit latency(s) for generating different lengths of tokens. }
\label{Tab10}
\small
\begin{tabular}{cccccc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{5}{c}{\textbf{num of tokens latency (s)}} \\
\cmidrule(lr){2-6}
& \textbf{128} & \textbf{256}& \textbf{512} & \textbf{1024} & \textbf{2048} \\
\midrule
Full model & 4.13 &7.87 & 15.74 & 28.66& 54.94 \\
sqLLM w/0.5\% ol. & 2.62 &5.21 & 10.47 & 21.15 & 43.65 \\
\rowcolor{cyan!10} sqLLM + NPFT & \textbf{2.40} & \textbf{4.72} & \textbf{9.65} & \textbf{19.37}& \textbf{40.11} \\
\bottomrule
\end{tabular}
\vspace{-10pt}
\end{table}
\begin{table}[h]
\centering
\vspace{-10pt}
\captionsetup{justification=justified}
\caption{}
\footnotesize
\begin{tabular}{cccccc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{Phase 1}} & \multicolumn{2}{c}{\textbf{Phase 2}}& \multirow{2}{*}{\textbf{Total Time (h)}} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& \textbf{Time (h)} & \textbf{Mem. (G)} & \textbf{Time (h)} & \textbf{Mem. (G)} \\
\midrule
EfficientQAT (3bits) & 3.3& 8.5 & 1.5&6.4&4.8 \\
EfficientQAT (4bits) & 3.3& 8.5 & 1.5 &7.0&4.8 \\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (0.05\%) & \textbf{0.63} &15.23 &\textbf{0.40} &6.11 &\textbf{1.03}\\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (0.5\%)& \textbf{0.78} & 15.23 &\textbf{0.40}&6.11&\textbf{1.18 }\\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (1\%)& \textbf{0.84} &15.23 &\textbf{0.40}&6.11&\textbf{1.24 }\\
\rowcolor{cyan!10}NPFT (3 \& 4bits) (2\%)& \textbf{0.91} &15.23 &\textbf{0.40}&6.11&\textbf{1.31 }\\
\bottomrule
\end{tabular}
\vspace{0.1cm}
\label{Tab4}
\end{table}
\end{document}
\begin{thebibliography}{10}
\bibitem{avron2011randomized}
Haim Avron and Sivan Toledo.
\newblock Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix.
\newblock {\em Journal of the ACM (JACM)}, 58(2):1--34, 2011.
\bibitem{chen2024efficientqat}
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu~Qiao, and Ping Luo.
\newblock Efficientqat: Efficient quantization-aware training for large language models.
\newblock {\em arXiv preprint arXiv:2407.11062}, 2024.
\bibitem{chowdhery2023palm}
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung~Won Chung, Charles Sutton, Sebastian Gehrmann, et~al.
\newblock Palm: Scaling language modeling with pathways.
\newblock {\em Journal of Machine Learning Research}, 24(240):1--113, 2023.
\bibitem{dettmers2022gpt3}
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.
\newblock Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.
\newblock {\em Advances in Neural Information Processing Systems}, 35:30318--30332, 2022.
\bibitem{dettmers2024qlora}
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.
\newblock Qlora: Efficient finetuning of quantized llms.
\newblock {\em Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem{dettmers2023spqr}
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.
\newblock Spqr: A sparse-quantized representation for near-lossless llm weight compression.
\newblock {\em arXiv preprint arXiv:2306.03078}, 2023.
\bibitem{dong2020hawq}
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael~W Mahoney, and Kurt Keutzer.
\newblock Hawq-v2: Hessian aware trace-weighted quantization of neural networks.
\newblock {\em Advances in neural information processing systems}, 33:18518--18529, 2020.
\bibitem{dong2019hawq}
Zhen Dong, Zhewei Yao, Amir Gholami, Michael~W Mahoney, and Kurt Keutzer.
\newblock Hawq: Hessian aware quantization of neural networks with mixed-precision.
\newblock In {\em Proceedings of the IEEE/CVF international conference on computer vision}, pages 293--302, 2019.
\bibitem{du2024bitdistiller}
Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu.
\newblock Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation.
\newblock {\em arXiv preprint arXiv:2402.10631}, 2024.
\bibitem{du2022glam}
Nan Du, Yanping Huang, Andrew~M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams~Wei Yu, Orhan Firat, et~al.
\newblock Glam: Efficient scaling of language models with mixture-of-experts.
\newblock In {\em International Conference on Machine Learning}, pages 5547--5569. PMLR, 2022.
\bibitem{floridi2020gpt}
Luciano Floridi and Massimo Chiriatti.
\newblock Gpt-3: Its nature, scope, limits, and consequences.
\newblock {\em Minds and Machines}, 30:681--694, 2020.
\bibitem{frantar2022gptq}
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.
\newblock Gptq: Accurate post-training quantization for generative pre-trained transformers.
\newblock {\em arXiv preprint arXiv:2210.17323}, 2022.
\bibitem{hu2021lora}
Edward~J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, and Weizhu Chen.
\newblock Lora: Low-rank adaptation of large language models.
\newblock {\em arXiv preprint arXiv:2106.09685}, 2021.
\bibitem{kaplan2020scaling}
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom~B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
\newblock Scaling laws for neural language models.
\newblock {\em arXiv preprint arXiv:2001.08361}, 2020.
\bibitem{kim2023squeezellm}
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael~W Mahoney, and Kurt Keutzer.
\newblock Squeezellm: Dense-and-sparse quantization.
\newblock {\em arXiv preprint arXiv:2306.07629}, 2023.
\bibitem{lin2024awq}
Ji~Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han.
\newblock Awq: Activation-aware weight quantization for on-device llm compression and acceleration.
\newblock {\em Proceedings of Machine Learning and Systems}, 6:87--100, 2024.
\bibitem{liu2023llm}
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.
\newblock Llm-qat: Data-free quantization aware training for large language models.
\newblock {\em arXiv preprint arXiv:2305.17888}, 2023.
\bibitem{merity2016pointer}
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.
\newblock Pointer sentinel mixture models.
\newblock {\em arXiv preprint arXiv:1609.07843}, 2016.
\bibitem{nvidia_mixed_precision_guide}
{NVIDIA Corporation}.
\newblock {\em Mixed-Precision Training Guide}, 2023.
\newblock Accessed: 2024-11-04.
\bibitem{raffel2020exploring}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter~J Liu.
\newblock Exploring the limits of transfer learning with a unified text-to-text transformer.
\newblock {\em Journal of machine learning research}, 21(140):1--67, 2020.
\bibitem{smith2022using}
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et~al.
\newblock Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.
\newblock {\em arXiv preprint arXiv:2201.11990}, 2022.
\bibitem{touvron2023open}
H~Touvron, T~Lavril, G~Izacard, X~Martinet, MA~Lachaux, T~Lacroix, B~Rozi{\`e}re, N~Goyal, E~Hambro, F~Azhar, et~al.
\newblock Open and efficient foundation language models.
\newblock {\em Preprint at arXiv. https://doi. org/10.48550/arXiv}, 2302, 2023.
\bibitem{wei2022outlier}
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi~Zhang, Fengwei Yu, and Xianglong Liu.
\newblock Outlier suppression: Pushing the limit of low-bit transformer language models.
\newblock {\em Advances in Neural Information Processing Systems}, 35:17402--17414, 2022.
\bibitem{le2023bloom}
BigScience Workshop, Teven~Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili{\'c}, Daniel Hesslow, Roman Castagn{\'e}, Alexandra~Sasha Luccioni, Fran{\c{c}}ois Yvon, et~al.
\newblock Bloom: A 176b-parameter open-access multilingual language model.
\newblock {\em arXiv preprint arXiv:2211.05100}, 2022.
\bibitem{yang2022hero}
Huanrui Yang, Xiaoxuan Yang, Neil~Zhenqiang Gong, and Yiran Chen.
\newblock Hero: Hessian-enhanced robust optimization for unifying and improving generalization and quantization performance.
\newblock In {\em Proceedings of the 59th ACM/IEEE Design Automation Conference}, pages 25--30, 2022.
\bibitem{zhang2022opt}
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi~Victoria Lin, et~al.
\newblock Opt: Open pre-trained transformer language models.
\newblock {\em arXiv preprint arXiv:2205.01068}, 2022.
\end{thebibliography}