CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs
Extracted source text · arXiv 2507.07145
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt, twocolumn]{article}
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[preprint]{acl}
% \usepackage{algorithm}
% \usepackage{algorithmic}
\makeatletter
\PassOptionsToPackage{dvipsnames,table}{xcolor}
\makeatother
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{float}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
\usepackage{subcaption}
% \usepackage[demo]{graphicx} % omit 'demo' for real document
\usepackage{lipsum}
% user
\usepackage{multirow}
\usepackage{booktabs}
\newcommand{\myref}[1]{Eq.\ref{#1}}
\usepackage{listings}
% \usepackage[dvipsnames]{xcolor}
\definecolor{mylightorange}{RGB}{255, 200, 100}
\definecolor{myvsgreen}{RGB}{0, 150, 50}
\definecolor{mypurple}{RGB}{128, 0, 128}
\usepackage[noline, ruled]{algorithm2e}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs}
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
% \author{First Author \\\And
% Second Author \\\And
% Third Author \\\And
% Forth Author \\}
\author{
\textbf{Zhaojing Zhou},
\textbf{Xunchao Li},
\textbf{Minghao Li},
\textbf{Handi Zhang},
\textbf{Haoshuang Wang},
\\
\textbf{Wenbin Chang},
\textbf{Yiqun Liu},
\textbf{Qingqing Dang},
\textbf{Dianhai Yu},
\textbf{Yanjun Ma},
\textbf{Haifeng Wang}
\\
\\
\\
\textsuperscript{}Baidu Inc.
\\
% \small{
% \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
% }
}
\begin{document}
\maketitle
\begin{abstract}
The rapid scaling of Large Language Models (LLMs) elevates inference costs and compounds substantial deployment barriers. While quantization to 8 or 4 bits mitigates this, sub-3-bit methods face severe accuracy, scalability, and efficiency degradation.
%
We propose Convolutional Code Quantization(CCQ), an inference-optimized quantization approach compressing LLMs to 2.0–2.75 bits with minimal accuracy loss.
%
Departing from error-prone scalar quantization or slow vector quantization, CCQ integrates a hardware-aware bit-shift encoding and decoding solution with Convolutional Code, Hybrid Encoding, and Code Cluster, jointly overcoming accuracy-speed bottlenecks.
%
We construct a lookup-free encoding space, enabling a linear mapping between the codebook and weight vectors, thereby optimizing inference performance. Meanwhile, by drawing on the concept of data mapping from vector quantization, we minimize the performance degradation of the model under extremely low-bit conditions.
%
Experiments demonstrate that CCQ achieves outstanding performance on LLMs across various benchmarks. We compress DeepSeek-V3(671B total parameters) to 184GB and ERNIE-4.5-300B-A47B to 89GB, enabling single-GPU deployment of ERNIE 4.5 and eliminating inter-card communication.
%
The 2-bit ERNIE-4.5-300B-A47B model and inference engine have been open-sourced.
\end{abstract}
\section{Introduction}
Large Language Models(LLMs) demonstrate state-of-the-art capabilities across artificial intelligence domains, spanning natural language processing to multi-modal understanding.
%
However, these performance gains coincide with significant inference degradation, impeding cost-effective large-scale deployment.
%
Contemporary flagship models exemplify this scalability challenge: Qwen2~\cite{qwen2}, Qwen2.5~\cite{qwen2.5}, and Llama 3~\cite{grattafiori2024llama3herdmodels} series have around 100 billion parameters within their most powerful members.
%
The emergence of Mixture-of-Experts (MoE) architecture further escalates model scale—evidenced by Llama 3.1 (405B)~\cite{grattafiori2024llama3herdmodels}, DeepSeek-V3 (671B)~\cite{deepseekai2025deepseekv3technicalreport}, Qwen3 (235B)~\cite{qwen3}, and ERNIE 4.5 (300B / 424B) ~\cite{ernie2025technicalreport}.
%
Such massive parameterization exacerbates inference latency, primarily due to memory constraints and cross-device communication overhead in distributed environments.
Quantization~\cite{gray1998quantization}, a cornerstone technique for model compression and efficient deployment, reduces parameter precision from 16 bits to 8 or 4 bits, yielding more than 50\% memory savings.
%
Common quantization configurations include Weight-Only Quantization(WOQ)~\cite{frantar2022gptq, lin2024awq}, Weight-Activation Quantization(WAQ)~\cite{xiao2023smoothquant} and KV Cache Quantization(KVQ)~\cite{liu2024kivi, hooper2024kvquant}.
%
WOQ and KVQ primarily alleviate memory bottlenecks—critical for models with massive parameter counts or long context windows—by compressing weights and key-value caches.
%
WAQ, conversely, exploits hardware acceleration for low-precision matrix operations (e.g., INT8/FP8) to achieve compute-bound speedups.
Recent efforts pursue extremely low-bit quantization (e.g., 2 bits) to further reduce LLM deployment costs.
%
However, these methods invariably compromise either model accuracy or inference efficiency, limiting their practical utility.
%
Post-training quantization (PTQ) algorithms like GPTQ~\cite{frantar2022gptq} and AWQ~\cite{lin2024awq} use scalar quantization to convert weights to low-bit integers. While effective at 4 bits by smoothing outliers, they typically fail at 2 bits due to the constrained numerical space.
% Post-training quantization (PTQ) algorithms based on scalar quantization directly convert weights into low-bit integers.
% %
% Prominent methods such as GPTQ~\cite{frantar2022gptq} and AWQ~\cite{lin2024awq} enhance quantization accuracy by smoothing outliers in the weights, achieving favorable results at 4 bits but typically failing at 2 bits due to the highly constrained numerical space.
% %
% In contrast, vector quantization~\cite{gray1984vector} has gained traction for weight-only compression, aiming to achieve higher compression ratios and better performance.
% %
% The essence of vector quantization lies in representing a broader range of high-dimensional vectors (i.e., model weights) using a set of low-dimensional vectors within a finite space (embodied as a codebook).
% %
% During quantization, weight parameters are approximated by vectors from the codebook through clustering, and storage compression is achieved by saving the indices of these vectors along with the codebook.
% %
% Meanwhile, the codebook can represent vectors using the same high-precision numerical type as the original weights, thereby improving quantization effectiveness.
% %
% During inference, dequantization reconstructs weights by codebook lookup using these indices, maintaining high-fidelity representation despite aggressive compression.
% Despite its potential for superior compression fidelity compared to scalar quantization, vector quantization introduces critical performance bottlenecks.
% %
% The nonlinear mapping between low-dimensional indices and high-dimensional weight vectors necessitates dequantization via large-scale codebook lookups.
% %
% This discrete, irregular memory access pattern violates data locality principles, preventing efficient hardware utilization.
% %
% It also induces pipeline stalls from non-contiguous memory fetches, and amplifies latency proportional to codebook size. Thereby severely degrading inference throughput in computationally intensive LLM deployments.
In contrast, vector quantization~\cite{vanbaalen-gptvq, liu2024vptq, tseng2024quip, malinovskii2024pvtuning} aims for higher compression and better performance by representing weights as indices into a finite codebook of high-precision vectors. During quantization, weights are clustered and replaced by codebook indices; storage is compressed by saving these indices and the codebook. Inference reconstructs weights via codebook lookup.
Despite offering superior compression fidelity over scalar quantization, vector quantization introduces critical performance bottlenecks. The dequantization process requires large-scale, non-contiguous codebook lookups based on indices. This irregular memory access pattern violates data locality, prevents efficient hardware utilization, causes pipeline stalls, and introduces latency proportional to codebook size, severely degrading LLM inference throughput.
To circumvent the performance drawbacks associated with inference-unfriendly codebooks while preserving the benefits of mapping high-dimensional vectors to low-dimensional vectors in vector quantization, we propose convolutional code quantization by integrating convolutional encoding with scalar quantization. By combining innovative hybrid encoding and code cluster, we achieve several extremely low-bit quantization configurations.
%
Initially, we construct a codebook based on convolutional codes, leveraging their inherent properties to achieve efficient bitshift decoding. Subsequently, we employ scalar quantization to establish a linear mapping between high-dimensional and low-dimensional vectors.
%
This approach not only obviates the need for storing a codebook but also replaces time-consuming indexing operations with linear computations, thereby offering both performance and efficacy benefits. Our main contributions are as below:
\begin{itemize}
\item \textbf{Convolutional Code Quantization (CCQ):} We propose a novel weight-only PTQ solution for extreme compression of LLMs. By integrating convolutional coding theory, hybrid encoding, and code cluster algorithms, CCQ achieves around 70\% model size savings at 2-bit precision compared with 8-bit counterpart, while maintaining near-lossless accuracy on both DeepSeek-V3 and ERNIE 4.5 models.
\item \textbf{Bit-Shift Decoding Inference:} Our co-designed bit-shift decoding strategy establishes inference latency similar to scalar-based quantization. This enables unprecedented deployment of ERNIE-4.5-300B-A47B (the largest ERNIE 4.5 language model) on a single NVIDIA H20 GPU without intra-node communication overhead.
\item \textbf{Open-Source:} We publicly release 2-bit quantized ERNIE 4.5 variants\footnotemark with around 2\% accuracy degradation relative to baselines. The inference code is available at \url{https://github.com/PaddlePaddle/FastDeploy/tree/develop}.
\footnotetext[1]{https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-2Bits-Paddle}
\end{itemize}
\section{Background and Related Work}
\subsection{Scalar-based Quantization}
RTN (Round-to-Nearest) is the most straightforward method for quantizing weight and activations by using the min-max values calibrated from a dataset.
%
However, the quantization error increases significantly when the number of bits is small, especially when significant outliers exist.
%
GPTQ~\cite{frantar2022gptq} updates the weight values based on the previous quantization error and the inverse of the Hessian matrix calculated from the activations.
%
AWQ~\cite{lin2024awq} transferred outliers from weights to activations to smooth the distributions. Group-wise quantization is applied to both methods.
%
However, these methods still suffer from non-negligible accuracy degradation even under 3/4-bit quantization, primarily constrained by the limited dynamic range of integer representation.
SqueezeLLM~\cite{kim2023squeezellm} implements non-uniform quantization and Hessian Metrics, GPTAQ~\cite{li2025gptaq} explicitly minimizes the quantization error as well as the accumulated asymmetry error for better performance under low-bit quantization.
%
HQQ~\cite{badri2023hqq} and HQQ+~\cite{badri2023hqq+} introduce Hyper-Laplacian distribution to better capture the heavy-tailed outlier errors compared to MSE.
%
However, substantial performance degradation still exists for 2 bits and even lower bits.
To improve the extremely low-bit model performance, Quantization-Aware-Training (QAT) is introduced in recent works. LLM-QAT~\cite{liu2023llm} achieves 4 bits on weights and KV Cache. But distillation is required to maintain the accuracy, which leads to heavy training pipelines. BitNet~\cite{wang2023bitnet} achieves 1.58-bit quantization by introducing QAT from the pre-training phase. However, the prolonged training duration and inherently limited scalability significantly constrain their practical deployment. OneBit ~\cite{xu2024onebitextremelylowbitlarge} and LittleBit ~\cite{lee2025littlebitultralowbitquantization} attempt to combine matrix decomposition and QAT for higher compression ratios. However, they still fail to eliminate the quantization cost and training unstablility.
\subsection{Vector-based Quantization}
To overcome the performance limit of scalar-based quantization, vector-based quantization methods are introduced for low-bit quantization.
Vector Quantization~\cite{gray1984vector} (VQ) is originally designed for lossy data compression.
%
It works by categorizing a large set of vectors into a small one based on some clustering algorithms, like K-means.
%
In the LLMs, index maps and codebooks are stored as the compressed weights to achieve the extreme compression ratio.
The main problem for VQ is how to balance the accuracy and its quantization cost.
%
AQLM~\cite{egiazarian2024extreme, malinovskii2024pvtuning} attempts to maximize the codebook's dimensionality, leaving the unstructured indexing the main bottleneck for inference speed.
%
QUIP\#~\cite{tseng2024quip} uses incoherence processing and E8 lattice for smaller codebook size, but it induces post-processing for Random Hadamard Transform (RHT), which hurts inference speed as well.
%
VPTQ~\cite{liu2024vptq} parallelly updates the vector in one column to avoid error accumulation and introduces residual quantizations for higher accuracy. However, it also falls short in improving inference performance.
\subsection{Convolutional Code}
Convolutional codes represent a classic technique in channel coding, which involves continuous encoding of each state within the channel.
%
Their primary characteristic lies in preserving the memory effect of the channel, where each current state's code value is derived from the transition of the preceding state.
Fischer et al.~\cite{marcellin1990trellis} applied a representation of convolutional codes known as trellis codes to quantization, thereby introducing Trellis-Coded Quantization (TCQ).
%
Subsequently, QTIP~\cite{tseng2024qtip} further enhances TCQ in terms of memory usage and decoding speed.
%
Firstly, QTIP employs incoherent processing to transform weights into an independent Gaussian-like source, which is well-suited for the trellis code.
%
Secondly, the researchers devised computation-based pseudo-Gaussian codes to circumvent the need for storing codebooks and defining network architectures.
%
Despite achieving favorable outcomes, QTIP still suffers from the drawback of computational complexity.
% Inspired by Trellis-Coded Modulation (TCM) in the field of communications, Marcellin and Fischer proposed Trellis-Coded Quantization (TCQ) as an alternative to traditional scalar-based and vector-based quantization. QTIP further improves upon TCQ in terms of memory usage and decoding speed.
% First, QTIP applies incoherence processing to transform weights into independent, Gaussian-like sources that are suitable for trellis coding. Second, QTIP introduces a bitshift trellis method, which avoids storing trellis structures while enabling parallel decoding during inference. Finally, QTIP employs compute-based random Gaussian codes, which eliminate the need to store a codebook while maintaining quantization performance.
% Although QTIP shows significant improvement over TCQ, it still suffers from high computational complexity. In contrast, our method simplifies codebook design by removing the incoherence processing (IP) step, which not only reduces inference-time computation but also avoids the complexity of compute-based coding. Furthermore, we introduce a novel clustered quantization scheme on top of the bitshift trellis, further improving the compression rate.
\section{Methodology}
This section presents the core elements of our proposed method.
%
Firstly, in \hyperref[sec:ccq]{Section 3.1}, we will elaborate on how we integrate convolutional codes with scalar quantization to construct a codebook capable of achieving linear mapping between high-dimensional and low-dimensional vectors, along with an overview of the framework of our algorithms.
%
Subsequently, in \hyperref[sec:he]{Section 3.2}, we demonstrate how we flexibly combine different convolutional codes to attain a higher compression ratio through hybrid encoding.
%
Furthermore, in \hyperref[sec:ec]{Section 3.3}, we transcend the limitations imposed by data types in encoding and storage by employing the code cluster, thereby achieving significantly superior compression performance.
%
In \hyperref[sec:scale-compress]{Section 3.4}, we introduce the implementation details of scale compression and optimization.
%
In \hyperref[sec:infer]{Section 3.5}, we present the performance optimization in the inference implementation.
\subsection{Convolutional Code Quantization}
\label{sec:ccq}
% Taking (L=2, N=3, S=1) as an example,
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{images/tq1.pdf}
\caption{Convolutional code codebook construction. Taking (L=2, N=3, S=1) configuration as an example, all the connection relationships among states in the directed graph are consolidated into a codebook. In the codebook, each row represents N consecutive states, and the corresponding convolutional code is the index.}
% Convolutional code codebook construction. All the connection relationships among states in the directed graph (L=2, N=3, S=1) are consolidated into a codebook. In the codebook, each row represents N consecutive state values, and the corresponding convolutional code is the index.
\label{fig:codebook}
\end{figure}
\begin{figure*}[ht]
\centering
\includegraphics[width=\textwidth]{images/tq2.pdf}
\caption{A schematic diagram of the CCQ. The dequantization part only involves shifting and dequantization calculations, and does not require the storage of a codebook. For the sake of simplicity, the codebook depicted in the figure is an example with parameters (L=4, N=3, S=2). In reality, as mentioned in \hyperref[sec:ec]{Section 3.3}, we perform the code cluster under the configuration of (L=6, N=4, S=3). The figure merely serves as an illustrative example.}
\label{fig:encode}
\end{figure*}
Unlike TCQ and QTIP, which perform sequential encoding on long sequences, we employ a vector-quantization-like approach to compress weights into vector groups.
%
By simplifying the design of the codebook, we aim to reduce computational complexity during the encoding process.
%
Consequently, we redefine convolutional codes from a directed graph representation to a finite set of vectors (i.e., the codebook).
\textbf{Definition of Convolutional Code Codebook}. We define a convolutional encoding configuration as the triplet (L, N, S), where L denotes the number of bits required for each state, N represents the number of states involved in a single encoding operation, and S indicates the number of transition bits between adjacent states.
%
Here we define transition bits as the bits that do not overlap with the previous state.
%
Figure \ref{fig:codebook} depicts all possible transition forms between adjacent states for the configuration (L=2, N=3, S=1).
%
Each state requires $L=2$ bits for representation.
%
The transition between adjacent states is achieved using $S=1$ transfer bit to update the state.
%
Consequently, there are $L - S=1$ redundant bits between adjacent states.
%
Given the overlapping bits between adjacent states, after applying convolutional coding, we only need to store the code symbols of the initial state along with the state transition bits to fully represent the entire state sequence.
%
For instance, considering the three states connected by the red line in Figure \ref{fig:codebook} ($00\rightarrow 01\rightarrow10$), the final stored code would be $0010$.
%
These three state values can then be placed in the index 2 (0010) of the codebook. Following this approach, we successfully transform the directed graph into a code table comprising a collection of vectors, each of length N.
Evidently, for a given encoding configuration, the total number of bits in a set of convolutional codes is calculated as $T = L + (N-1)*S$. Subsequently, we can generate a corresponding codebook $\mathbf{C} \in \mathbb{Z}^{2^T\times N}$.
Taking 4-bit quantization as an example in weight quantization, where each quantized weight requires 4 bits for storage, in the case of convolutional encoding with parameters set as (L=4, N=3, S=2), we can store three quantized values using only 8 bits, achieving an equivalent compression effect of 2.66 bits per weight (bpw).
%
Compared to the 4-bit weight quantization, this results in a 33\% improvement in compression rate while maintaining the equivalent numerical precision as 4-bit quantization.
\textbf{Bitshift Decoding}. Due to the repetitive nature between adjacent states in convolutional codes, we can rapidly decode convolutional codes through bit shifting.
%
Consequently, within the framework of convolutional code quantization, the dequantization process only involves bit-shifting operations and dequantization calculations for scalar quantization, which undoubtedly facilitates inference implementation more than index lookup in vector quantization.
%
The specific decoding process can be referred to in Figure \ref{fig:encode}.
\textbf{Group-wise Quantization}. Although the memory characteristic of convolutional codes offers convenience in storage and decoding, it also restricts the non-coherence of quantized weights.
%
When weights contain a significant number of outliers, the coherence of quantized values diminishes, which is detrimental to convolutional coding.
%
Therefore, we introduce group-wise quantization to mitigate the impact of outliers.
%
The quantization granularity at the group-wise level not only ensures the quantization precision of scalar quantization but also reduces the encoding loss during convolutional coding. In scenarios where the group size is not a multiple of N, we will pad each group with zeros to fulfill the encoding requirements.
%
This approach will not compromise the accuracy of quantization.
\textbf{The Overall Quantization Algorithm}. The complete process of the convolutional code quantization method is demonstrated in Algorithm \ref{algo:whole algo}.
%
Firstly, we group the weights and calculate the quantization parameters for each group.
%
Subsequently, we generate the encoding space based on the convolutional code configuration (at this stage, the codebook contains numerical values within the integer domain, denoted as $\mathbf{C} \in \mathbb{Z}^{2^T\times N}$).
%
We then transform these values into the same floating-point domain as the original weights using the group scale (resulting in $\mathbf{C} \in \mathbb{R}^{2^T\times N}$).
%
Finally, we compute the Mean Squared Error (MSE) loss between each weight and every vector in the codebook.
%
The optimal code value is determined by searching for the one that yields the minimum MSE loss.
It is important to note that there is no need to store the actual vector values obtained from the search—only their corresponding indices need to be retained.
%
The step is consistent with vector quantization, but the key difference lies in the fact that our method does not require storing a large codebook to maintain the mapping between indices and vectors.
In our convolutional encoding quantization scheme, the indices obtained from the search directly represent the final convolutional code values.
%
During subsequent dequantization, the true quantized values can be retrieved through bit-shifting operations, and the original weights can be reconstructed via dequantization computations.
\begin{algorithm}
\footnotesize
\SetAlgoLined
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
\Input{
$\mathbf{W} \in \mathbb{R}^{d_{o} \times d_{i}}$: Original Weight \\
$g$: group size \\
$(L, N, S)$: Encoding configuration
}
\Output{
$\mathbf{W_q} \in \mathbb{Z}$: Weight with Convolutional Code
}
\BlankLine
$\mathbf{W_g} \gets$ Reshape $\mathbf{W}$ \tcp*{$\mathbf{W_g} \in \mathbb{R}^{\frac{d_o \times d_i}{g}\times \frac{g}{N}\times N}$}
${group\ scale} \gets \mathbf{W_g}$ \tcp*{$group\ scale \in \mathbb{R}^{\frac{d_o \times d_i}{g}}$}
$\mathbf{C}$ = Construct\_codebook$(L, N, S)$ \tcp*{$\mathbf{C} \in \mathbb{Z}^{2^T\times N}$}
$\mathbf{C}$ = $\mathbf{C} \times {group\ scale}$ \tcp*{$\mathbf{C} \in \mathbb{R}^{\frac{d_o \times d_i}{g} \times 2^T\times N}$}
$Indices \gets \underset{i\in 2^T}{\arg\min}\|\boldsymbol{v}-\mathcal{C}_i\|^2,\forall\boldsymbol{v}\in\mathbf{W_g}$
$\mathbf{W_q} \gets$ Reshape $Indices$
\caption{Convolutional Code
Quantization}
\label{algo:whole algo}
\end{algorithm}
\subsection{Hybrid Encoding}
\label{sec:he}
Although our method can achieve arbitrary compression rates by adjusting the encoding configuration, the actual storage compression effect can only be realized when the code aligns with practical data types.
%
Specifically, the storage bits for convolutional coding must be exact multiples of 8 to be stored using data types such as INT8 or INT16.
%
However, achieving a high compression rate typically necessitates an increase in the number of encoding states.
%
Due to the inherent correlation in convolutional coding, an excessive number of encoding states can significantly exacerbate precision loss.
%
Therefore, to attain a higher compression rate, we propose the concept of hybrid encoding.
%
Within a single group, multiple distinct encoding configurations are employed in a hybrid manner to meet storage requirements.
%
Through meticulous design, we successfully alternate between the (L=3, N=3, S=2) and (L=3, N=4, S=2) configurations, encoding seven numerical values into 16 bits, ultimately achieving an equivalent bit rate of 2.28 bpw.
\subsection{Code Cluster}
\label{sec:ec}
By employing hybrid encoding, we can alleviate the constraints imposed by storage data types, but achieving further compression remains challenging.
%
To realize true 2-bit weight quantization, we innovatively propose the code cluster algorithm that entirely breaks free from the limitations of encoding configurations on compression rates.
Our convolutional code quantization is conducted at a group-wise granularity.
%
Given the vast encoding space, the encoding values utilized on each channel are quite limited.
%
This observation raises the question: can we retain only the frequently used encoding values and eliminate the redundant ones, thereby enhancing the overall compression rate by reducing the encoding space?
%
Naturally, the idea of clustering comes to mind to distill the significant encoding values. However, clustering is a nonlinear operation, and its data reconstruction process inevitably requires index lookups.
\begin{figure}[h!]
\centering
\includegraphics[width=0.45\textwidth]{images/tq3.pdf}
\caption{The code value distribution diagram of convolutional coding quantization on an output channel under the configuration of (L=6, N=4, N=3).}
\label{fig:code distribution}
\end{figure}
Upon examining the numerical distribution of convolutional encoding quantization (Figure \ref{fig:code distribution}), we observe that the convolutional codes on each channel approximately follow a normal distribution, which is suitable for uniform quantization.
%
Consequently, we propose utilizing uniform quantization to accomplish the code cluster, mapping any arbitrary encoding space into an 8-bit numerical range.
%
As a result, all data reconstruction operations can be accomplished through computation.
%
The computational formula for the code cluster is presented as follows:
\begin{align}
\mathbf{C'} &= \mathcal{Q}(\mathbf{C})=\left \lfloor \frac{\mathbf{C}-\beta }{\alpha} \right \rceil
\end{align}
where $\mathbf{C} \in [0, 2^T-1]$, $\mathbf{C'} \in [0, 255]$, $\alpha$ means code scale and $\beta$ means code zero-point in asymmetric quantization.
Specifically, we first perform convolutional code quantization with parameters (L=6, N=4, S=3) on the weights of each group.
%
Subsequently, we conduct clusters of the code values along the dimension of output channels and map them to an 8-bit numerical range.
%
This approach enables the storage of four parameters using only UINT8, achieving an equivalent compression effect of 2 bpw.
At this point, although the code cluster restricts the encoding space, the true quantized value of the weights is 6 bits.
%
When combined with the group-wise quantization granularity, it can effectively ensure the numerical precision of critical weights. Consequently, the impact on the model's performance is negligible.
\subsection{Scale Optimization and Compression}\label{sec:scale-compress}
\textbf{Scale Optimization}: After determining the encoded values through coding search, we can compute the quantization loss by dequantizing these values and comparing them with the original weights.
%
The quantization error formula for a group of weight is as follows:
\begin{equation}
\begin{split}
Error &= {\textstyle \sum_{i=0}^{g-1}} \left \| \mathbf{W}_i -\mathcal {DQ} (\mathbf{W}{^q_i}) \right \| ^2 \\
&= {\textstyle \sum_{i=0}^{g-1}}\left \| \mathbf{W}_i - \mathbf{W}{^q_i}\cdot S \right \| ^2
\end{split}
\end{equation}
where $\mathbf{W}$ represents the original weight of a group, $\mathbf{W_q}$ denotes the corresponding quantized values, $g$ denotes the group size, $\mathcal {DQ}$ signifies the dequantization function, and $S$ stands for the scale.
%
For the sake of simplifying calculations, we directly use the decoded values to compute the quantization error, as bitshift decoding does not affect the accuracy of the quantized values.
At this stage, the scale $S$ can be regarded as the sole variable. We can further reduce the quantization loss by updating the scale.
\begin{equation}
\begin{split}
\frac{\mathrm{d}Error}{\mathrm{d}S}&= 2\sum_{i=0}^{g}(\mathbf{W}{^q_i})^2S - 2\sum_{i=0}^{g}(\mathbf{W}{_i}\mathbf{W}{^q_i})
% \frac{\mathrm{d}Error}{\mathrm{d}S}&=\frac{\mathrm{d}}{\mathrm{d}S}(\sum_{i=0}^{g}\mathbf{W}_i +\sum_{i=0}^{g}(\mathbf{W}{^q_i}S)^2 \\
% &-2\sum_{i=0}^{g}\mathbf{W}{_i}\mathbf{W}{^q_i}S) \\
% &= 2\sum_{i=0}^{g}(\mathbf{W}{^q_i})^2S - 2\sum_{i=0}^{g}(\mathbf{W}{_i}\mathbf{W}{^q_i})
\end{split}
\end{equation}
By taking the derivative of the scale and setting $\frac{\mathrm{d}Error}{\mathrm{d}S}$ to zero, we can determine the minimum value of the current loss.
\begin{equation}
\begin{split}
S=\frac{ {\textstyle \sum_{i=0}^{g-1}}\mathbf{W\cdot W}^q_i}{{\textstyle \sum_{i=0}^{g-1}}(\mathbf{W}{^q_i})^2}
\end{split}
\end{equation}
\textbf{Scale Compression}: In the group-wise quantization, each group has its own scale factor.
%
As the number of groups increases, the storage overhead associated with these scale factors becomes non-negligible.
%
To address this, we propose quantizing the scale factors by channel-wise super scales and storing them in a low-bit format.
%
Additionally, we observe that for the aforementioned 2.66 bpw (with L=4, N=3, S=2) and the hybrid encoding scheme yielding 2.28 bpw, the group size is not divisible by the number of encoding states $N$.
%
Consequently, the last code value in each group introduces redundant bits.
%
Taking a group size of 64 as an example, under the (L=4, N=3, S=2) configuration, 22 code values are required to encode 64 parameters.
%
In this scenario, only the first 4 bits of the last code value are valid, while the remaining 4 bits are redundant and need to be discarded after decoding.
%
We can leverage these 4 redundant bits to store the quantized scale factor, ultimately achieving an equivalent bpw of 2.75.
%
Similarly, for the hybrid encoding scheme with 2.28 bpw, 10 code values are necessary to encode 64 parameters, leaving 13 redundant bits that can be utilized to store the quantized scale factor, resulting in an equivalent bpw of 2.5.
%
When it comes to the code cluster, redundant bits cannot be utilized, and additional consideration must be given to the storage overhead of the scale. For the code cluster of (L=6, N=4, S=3), the equivalent bit width is 2.06 bpw. The formula for calculating the equivalent bit width is presented below:
\begin{equation}
\begin{split}
bpw = \left\{\begin{matrix}
\frac{T*{\lceil \frac{g}{N} \rceil }}{g}, & \text{if } g \equiv 1 \pmod{N} \\
\frac{T*{ \frac{g}{N} }}{g}+\frac{scale\_bits}{g}, & \text{if } g \equiv 0 \pmod{N}
\end{matrix}\right.
\end{split}
\end{equation}
where $T$ represents the total number of bits for a set of convolutional codes.
%
However, when the code cluster is used, $T$ denotes the number of bits required for the quantized code values, specifically $T = 8$.
\subsection{Dequantization}
\label{sec:infer}
In this section, we present the detailed implementation strategies for dequantizing weights within individual groups, as outlined in Algo.~\ref{algo:w275-dequant} and Algo.~\ref{algo:w2-dequant}, and summarize the corresponding configurations in Table~\ref{tab:dequant-config}.
%
%
\begin{table}[h]
\renewcommand{\arraystretch}{1.2}
\centering
\caption{Dequantization configuration for different encoding strategies.}
\label{tab:dequant-config}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}ccccccc@{}}
\toprule
(L,N,S) & bpw & Weight Mask & Scale Mask & Weight Bit-Shift & Code Cluster \\
\midrule
(4,3,2) & 2.75 & 0xF & 0xF & [4,2,0] & No \\
(3,3,2)(3,4,2) & 2.5 & 0x7 & 0x1FFF & [13,11,9,6,4,2,0] & No \\
(6,4,3) & 2.06 & 0x3F & 0xF & [9,6,3,0] & Yes \\
\bottomrule
\end{tabular}
}
\end{table}
%
%
%
For the standard dequantization of Hybrid Encoding, we use Algo.~\ref{algo:w275-dequant} to dequant weights.
%
For example, the (L=4, N=3, S=2) configuration, which is 2.75 bpw, we set weight bit-shift $\mathbf{W_{shift}}$ to [4, 2, 0] with 0xF as weight mask $\mathbf{w_{mask}}$; For the hybrid encoding configuration (L1=3, N1=3, S1=2, L2=3, N2=4, S2=2), we pass $\mathbf{W_{shift}}$ with [13, 11, 9, 6, 4, 2, 0] and weight mask $\mathbf{w_{mask}}$ with 0x7.
%
As discussed in Section~\ref{sec:scale-compress}, we leverage the redundant bits within each weight group to store additional information.
%
During the dequantization process for each computational block, the algorithm efficiently extracts the scaling factors (scale) that are embedded within these redundant bits of the loaded weight matrix $\mathbf{W_q}$.
%
In contrast to the Hybrid Encoding, the encoding configuration (L=6, N=4, S=3) with Code Cluster employs $\mathbf{W_{shift}}$ of [9, 6, 3, 0] and $\mathbf{w_{mask}}$ of 0x3F, as detailed in Algo.~\ref{algo:w2-dequant}.
%
This configuration necessitates two distinct dequantization operations within the main computation loop.
%
The first dequantization operation expands the quantized weights from UINT8 to UINT16, thereby enlarging the numerical space for subsequent processing.
%
The second operation applies different bit-shift operations to dequantize the weights, following a similar approach to Algo.~\ref{algo:w275-dequant}.
%
Another key distinction from Algo.~\ref{algo:w275-dequant} stems from the fact that the code cluster eliminates redundant bits entirely, consequently requiring explicit loading of corresponding group-wise quantization scales for each computation block during the main processing loop.
\begin{algorithm}[t]
\footnotesize
\SetAlgoLined
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
\Input{
$\mathbf{W_q}$: compressed quantized weights \\
$\mathbf{s_{super}}$: super scale \\
$\mathbf{zp}$: weight zero-point \\
$\mathbf{W_{shift}}$: bit shift of quantized weight\\
$\mathbf{w_{mask}}$: weight mask \\
$\mathbf{s_{mask}}$: scale mask \\
}
\Output{
$\mathbf{W_{dq}}$: dequantized weight
}
\BlankLine
% $\mathbf{p_{s}} \gets$ length of $\mathbf{S_{shift}}$\;
$\mathbf{p_{w}} \gets$ length of $\mathbf{W_{shift}}$\;
$s_{uint} \gets $ the last value of $\mathbf{W_q}$\;
$\mathbf{s} \gets (s_{uint} \ \& \ \mathbf{s_{mask}}) \times \mathbf{s_{super}}$\;
\For{each element $d_w$ in $\mathbf{W_{dq}}$ with index $idx$}{
$q_w \gets \mathbf{W_q}[{idx/\mathbf{p_{w}}}]$\;
$w_{uint} \gets (q_w \gg \mathbf{W_{shift}}[{idx \ \mathbf{mod} \ \mathbf{p_{w}} }]) \ \&\ \mathbf{w_{mask}}$\;
$d_w \gets (w_{uint} - \mathbf{zp}) \times \mathbf{s}$\;
}
\caption{Dequantization without Code Cluster}
\label{algo:w275-dequant}
\end{algorithm}
\begin{algorithm}[t]
\footnotesize
\SetAlgoLined
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
\Input{
$\mathbf{W_q}$: compressed quantized weights \\
$\mathbf{s_q}$: quantized scale \\
$\mathbf{s_{super}}$: super scale \\
$\mathbf{zp}$: weight zero-point \\
$\mathbf{s_{code}}$: code scale \\
$\mathbf{zp_{code}}$: code zero-point \\
$\mathbf{W_{shift}}$: bit shift of quantized weight\\
$\mathbf{s_{shift}}$: bit shift of quantized scale\\
$\mathbf{w_{mask}}$: weight mask \\
$\mathbf{s_{mask}}$: scale mask \\
}
\Output{
$\mathbf{W_{dq}}$: dequantized weight
}
\BlankLine
% $\mathbf{p_{s}} \gets$ length of $\mathbf{S_{shift}}$\;
$\mathbf{s} \gets ((\mathbf{s_q} \gg \mathbf{s_{shift}})\ \& \ \mathbf{s_{mask}}) \times \mathbf{s_{super}}$\;
$\mathbf{p_{w}} \gets$ length of $\mathbf{W_{shift}}$\;
\For{each element $d_w$ in $\mathbf{W_{dq}}$ with index $idx$}{
$q_w \gets \mathbf{W_q}[{idx/\mathbf{p_{w}}}]$\;
$w_{uint} \gets \mathbf{round}(q_w \times \mathbf{s_{code}} + \mathbf{zp_{code}})$\;
$w_{uint} \gets (w_{uint} \gg \mathbf{W_{shift}}[{idx \ \mathbf{mod} \ \mathbf{p_{w}} }]) \& \ \mathbf{w_{mask}}$\;
$d_w \gets (w_{uint} - \mathbf{zp}) \times \mathbf{s}$\;
}
\caption{Dequantization with Code Cluster}
\label{algo:w2-dequant}
\end{algorithm}
\section{Experiments}
\subsection{Settings}
% \textbf{Models and Datasets} We use DeepSeek-V3-0324 and ERNIE-4.5-300B-A47B to implement weight only experiments, while leveraging Qwen235B to demonstrate the comparability of our WINT2.x method against state-of-the-art models.
\paragraph{Models and Datasets.} We conduct rigorous evaluations using 2 bpw, 2.5 bpw, and 2.75 bpw on two state-of-the-art large language models: DeepSeek-V3-0324 and ERNIE-4.5-300B-A47B.
%
%The experiments comprehensively compare CCQ against prevailing weight-only quantization approaches including GPTQ ~\cite{frantar2022gptq}, AWQ ~\cite{lin2024awq}, and Unsloth ~\cite{unsloth}.
The experiments compare CCQ against 8-bit and 4-bit weight-only quantized models.
%prevailing weight-only quantization approach Unsloth ~\cite{unsloth}.
We evaluate our model under a unified evaluation protocol across several key benchmarks:
\begin{itemize}
\item \textbf{Zero-shot} performance on mathematical reasoning (GSM8K~\cite{cobbe2021training}), Chinese mathematics (CMath~\cite{wei2023cmath}), and multi-step reasoning (MUSR~\cite{sprague2023musr}).
\item \textbf{3-shot} performance on complex task handling (BBH~\cite{srivastava2022beyond}) and reading comprehension (DROP~\cite{dua2019drop}).
\item \textbf{5-shot} performance on Chinese comprehensive evaluation (C-Eval~\cite{huang2023ceval}), and massive multi-task language understanding (MMLU~\cite{hendryckstest2021}).
\end{itemize}
These datasets span critical capabilities including mathematics, reasoning, question answering, reading comprehension, and general tasks. All evaluations use
the generation configuration, as per the model's recommended settings.
% \textbf{Baselines} For DeepSeek-V3 \cite{liu2024deepseek}, we benchmark our method against mainstream weight-only quantization approaches: GPTQ, AWQ and Unsloth, using open-source models from Hugging Face. For ERNIE-4.5, we take a hybrid configuration, 8-bit dense layers with 4-bit MoE layers, as the baseline model. Results on different algorithms of each task are generated under the same evaluation rules, including evaluations for Qwen235B. In Section 4.2, we provide a more detailed comparison to other weight-only quantization approaches mentioned above.
\paragraph{Baselines.} We benchmark our method against 8-bit and 4-bit weight-only quantization. %GPTQ~\cite{frantar2022gptq},
%AWQ~\cite{lin2024awq},
% Unsloth formatted in IQ1\_M ~\cite{unsloth}.
For DeepSeek-V3~\cite{liu2024deepseek}, comparisons are conducted using open-source quantization models from Hugging Face.
%
For ERNIE 4.5, we adopt Weight-only INT8 (WINT8) as the baseline.
All results across tasks and algorithms are generated under identical evaluation rules.
%including evaluations for Qwen3-235B-A22B~\cite{qwen3}.
%
A comprehensive comparison with the aforementioned weight-only quantization approaches is provided in \hyperref[sec:mr]{Section 4.2}.
\subsection{Main Results}
\label{sec:mr}
% \paragraph{Results on DeepSeek-V3: } Table 1\ref{tab:dsv3} compares the performance of CCQ against GPTQ, AWQ, and Unsloth. The results show that, under 4-bit quantization, both GPTQ and AWQ exhibit a 2-3\% accuracy drop on C-Eval, it is reasonable to expect further performance deterioration at 2-bit quantization, as indicated in prior studies. While Unsloth at 2-bit precision achieves performance parity with AWQ on most dataset, it still experience significant performance degradation (approximately 6\%) on DROP.
% Our method, under 2.75-bit quantization, achieves comparable performance to Unsloth on most datasets, while outperforming it on DROP accuracy by 6.3\%, although it incurs an additional 34GB of GPU memory usage. For WINT2.06, this approach reduces GPU memory usage by 12GB and still maintains the accuracy at DROP.
\paragraph{Results on DeepSeek-V3.} As summarized in Table~\ref{tab:dsv3}, our proposed \textbf{CCQ} method achieves comparable performance with mainstream 8-bit and 4-bit weight-only quantization techniques.
%
%At 4-bit precision, both GPTQ and AWQ exhibit 2-3\% accuracy degradation on C-Eval~\cite{huang2023ceval}, with further performance deterioration at 2-bit quantization.
%Notably, while Unsloth-2bit achieves performance parity with AWQ on most benchmarks, it still suffers substantial degradation ($\sim$6\%) on the DROP~\cite{dua2019drop} reading comprehension task.
%
%In contrast, our 2.75 bpw CCQ method:
%\begin{itemize}
% \item Matches Unsloth's performance on standard benchmarks.
% \item Outperforms Unsloth by 6.3\% on DROP accuracy.
%\end{itemize}
%This improvement comes at the cost of 34GB additional GPU memory. However, our optimized variant with only 2.06 bits reduces memory usage by 12 GB from Unsloth-2bit while preserving DROP accuracy, demonstrating an effective accuracy-memory trade-off.
%
%
% CCQ algorithm reduces memory usage by 12GB compared to the Unsloth-2bit algorithm, while exhibiting only minimal performance degradation.
% %
% Noted that the CCQ results were obtained using only INT4 RTN quantization and 2.06-bit CCQ mixed-bit quantization under identical mixed-bit configurations, models with INT4 and INT2 quantization produced garbled outputs.
%
%
While 2-bit RTN weight-only quantization induces a performance cliff with catastrophic accuracy collapse, our 2.06-bit CCQ method reduces memory usage by up to 71.3\% from 8-bit models with marginal performance degradation.
%
Section~\ref{sec:infer} provides further hardware-aware optimization analyses.
\begin{table}[h]
\renewcommand{\arraystretch}{1.2}
\centering
\caption{Weight-only quantization results of DeepSeek-V3-0324.}
\label{tab:dsv3}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}ccccccc@{}}
\toprule
Model & \multirow{2}{*}{bpw} & \multicolumn{4}{c}{DeepSeek-v3-0324} \\
Dataset & & Memory & GSM8K & C-Eval & MMLU & Avg \\ \midrule
Baseline & 8 & 642GB & 96.21 & 87.31 & 86.50 & 90.01 \\
GPTQ\footnotemark & 4 & 346GB & 94.77 & 87.66 & 86.60 & 90.04 \\
AWQ\footnotemark & 4 & 328GB & 96.13 & 87.22 & 86.38 & 89.43 \\
%Unsloth & 2\footnotemark & 196GB & 96.06 & 85.04 & 85.16 & 88.75 \\
\multirow{3}{*}\textbf{CCQ (our)} & 2.75 & 230GB & 95.68 & 85.08 & 85.24 & 88.67 \\
& 2.5 & 212GB & 95.60 & 84.11 & 85.07 & 88.26 \\
& 2.06 & \textbf{184GB} & 94.92 & 84.07 & 84.29 & 87.76 \\ \bottomrule
\end{tabular}
}
\end{table}
% \footnotetext{Model details: \url{https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF?show_file_info=UD-IQ1\_M\%2FDeepSeek-V3-0324-UD-IQ1\_M-00001-of-00004.gguf}}
\footnotetext[2]{GPTQ:%
\url{https://huggingface.co/ISTA-DASLab/DeepSeek-V3-0324-GPTQ-4b-128g-experts}}
\footnotetext[3]{AWQ:%
\url{https://huggingface.co/cognitivecomputations/DeepSeek-V3-0324-AWQ}}
\paragraph{Results on ERNIE 4.5.} As quantified in Table~\ref{tab:ernie-4.5}, our 2 bpw quantized model reduces GPU memory consumption by 68.33\% compared to the WINT8 baseline, while constraining accuracy degradation to around 2\%. CCQ establishes a cost-effective inference pathway that enables practical single-GPU deployment on hardware such as NVIDIA H20 devices.
However, on both DeepSeek-V3 and ERNIE 4.5, accuracy drops of CCQ primarily occur in 5-shot C-Eval and MMLU tasks due to instruction-following failures. Future work should focus on improving model adherence to instructions.
\begin{table*}[h]
\renewcommand{\arraystretch}{1}
\centering
\caption{Weight-only quantization results of ERNIE 4.5.}
\label{tab:ernie-4.5}
\resizebox{\linewidth}{!}{
% \begin{tabular}{cccc}
% \hline
% \multirow{2}{*}{} & \multicolumn{2}{c}{ERNIE-4.5-300B-A47B} & Qwen3-235B-A22B \\
% & WINT4+8 & WINT2 (CCQ) & FP8 \\
% Memory & 148GB & 89GB & 235GB \\ \hline
% % MMLU-Pro & 73.97 & & 72.86 \\
% % C-Eval & 88.00 & 87.10 & 89.90 \\
% CMMLU & 89.80 & 86.48 & 87.48 \\
% GSM8K & 96.29 & 95.75 & 95.89 \\
% CMath & 95.50 & 94.50 & 95.67 \\
% BBH & 90.20 & 87.92 & 84.52 \\
% MUSR & 66.90 & 65.43 & 66.81 \\
% DROP & 90.12 & 89.08 & 88.67 \\
% Avg & 88.13 & 86.52 & 86.50 \\ \hline
% \end{tabular}
% }
% \end{table}
% \begin{tabular}{cccc}
% \hline
% \multirow{2}{*}{} & \multicolumn{2}{c}{ERNIE-4.5-300B-A47B} & Qwen3-235B-A22B \\
% & WINT8 & WINT2 (CCQ) & FP8 \\
% Memory & 281GB & 89GB & 235GB \\ \hline
% MMLU-Pro & 73.97 & & 72.86 \\
% C-Eval & 88.00 & 87.10 & 89.90 \\
% CMMLU & 90.2 & 86.22 & 87.48 \\
% GSM8K & 96.6 & 95.98 & 95.89 \\
% CMath & 96.6 & 96.00 & 95.67 \\
% BBH & 94.3 & 92.02 & 84.52 \\
% MUSR & 69.9 & 69.57 & 66.81 \\
% DROP & 91.1 & 89.97 & 88.67 \\
% Avg & 89.8 & 88.29 & 86.50 \\ \hline
% \end{tabular}
% }
% \end{table}
\begin{tabular}{cccccccccc}
\hline
Model & \multicolumn{9}{c}{ERNIE-4.5-300B-A47B} \\
Dataset & Memory & GSM8K & CMath & MUSR & BBH & DROP & C-Eval & MMLU & Avg \\ \hline
WINT8 & 281GB & 96.6 & 96.7 & 69.9 & 94.3 & 91.1 & 90.6 & 86.5 & 89.39 \\
WINT4 & 149GB & 96.21 & 96.5 & 69.67 & 94.43 & 91.17 & 89.84 & 86.16 & 89.14 \\
WINT2 & 89GB & 95.98 & 96.0 & 69.57 & 92.02 & 89.97 & 85.39 & 82.58 & 87.36 \\ \hline
\end{tabular}
}
% \begin{tabular}{ccccccccccc}
% \hline
% Model & \multicolumn{10}{c}{ERNIE-4.5-300B-A47B} \\
% Dataset & Memory & GSM8K & CMath & MUSR & BBH & DROP & C-Eval & MMLU & Avg \\ \hline
% WINT8 & 281GB & 96.6 & 96.7 & 69.9 & 94.3 & 91.1 & 90.6 & 86.5 & \\
% WINT4 & 149GB & 96.21 & 96.5 & 69.67 & 94.43 & 91.17 & 89.84 & 86.16 & 2 \\
% WINT2 & 89GB & 95.98 & 96.0 & 69.57 & 92.02 & 89.97 & 85.39 & 82.58 & 1 \\ \hline
% \end{tabular}
% }
\end{table*}
\section{Inference Performance}
% We conduct comprehensive benchmarks on individual GEMM (General Matrix-Matrix Multiplication) operators, comparing VPTQ~\cite{liu2024vptq}, standard WINT2, and CCQ WINT2 implementations. For VPTQ, we use the official codebase with recommended configurations for inference. For both standard WINT2 and CCQ WINT2, we implement similar Triton-based kernels~\cite{tillet2019triton}. The key difference for Encoding Cluster WINT2 lies in requiring two separate dequantization operations per computation block and different bit shift configurations.
% %
% We evaluate our implementations on NVIDIA H20 GPUs using 4 different weight matrix configurations with input sequence lengths M set to 1 and 4. To ensure statistical reliability, each configuration is tested for 500 rounds, and we report the average execution time computed from the latter half of the test rounds. The complete results are presented in Table~\ref{tab:operator-performance}. WINT2-CCQ demonstrates similar performance with WINT2-Standard and consistent performance advantages over VPTQ across all tested configurations. The performance gap widens with larger matrices, highlighting WINT2-CCQ's superior scalability.
% %
% The superior performance of WINT2-CCQ primarily stems from our efficient dequantization design, which eliminates the expensive codebook lookup operations required by traditional vector quantization methods. Furthermore, our inference implementation closely resembles standard INT2 quantization kernels, enabling seamless integration with advanced GEMM implementations such as Marlin~\cite{frantar2024marlin} operators, which could further enhance inference performance in future deployments.
We conduct comprehensive benchmarks on both individual GEMV (General Matrix-Vector Multiplication) and Grouped-GEMM (Grouped General Matrix-Matrix Multiplication) operators with various matrix shapes and batch size M on the NVIDIA H20 GPU.
%
For GEMV, we compare the performance of VPTQ~\cite{liu2024vptq}, standard WINT2, and our proposed CCQ implementations with bpw equal to 2.06.
% For GEMV, we compare the performance of VPTQ~\cite{liu2024vptq}, standard WINT2, and our proposed CCQ WINT2 implementations.
%
VPTQ is evaluated using its official codebase with recommended inference configurations(bpw=3), while both standard WINT2 and CCQ are implemented with similar Triton-based kernels~\cite{tillet2019triton} to ensure a fair comparison.
%
For Grouped-GEMM, we port the CCQ algorithm to the vLLM~\cite{kwon2023efficient} wna16 operator, a widely adopted Group-GEMM kernel for MoE inference. The baselines (W8A16, W4A16) are experimented on the vLLM library.
% TODO: 加上matrix shape解释，256 选 8
%
The key distinction of CCQ lies in its use of two separate dequantization operations per computation block, as well as distinct bit shift configurations.
% The detailed results are presented in Table~\ref{tab:operator-performance} and Table~\ref{tab:operator-group-gemm-performance}.
% %
% For GEMV kernels, WINT2-CCQ achieves execution times that are highly competitive with the standard WINT2 implementation and consistently outperforms VPTQ across all tested matrix shapes and input sequence lengths.
% %
% The superior performance of WINT2-CCQ primarily stems from our efficient dequantization design, which eliminates the expensive codebook lookup operations required by traditional vector quantization methods.
% %
% For Group-GEMM kernels, we compare WINT2 CCQ (denoted as W2A16-CCQ) with industry-standard low-bit operators, including W8A16 and W4A16. The results demonstrate that WINT2-CCQ not only achieves the lowest execution latency among all methods, but also maintains this advantage as the batch size and matrix dimensions grow. This highlights the efficiency of our dequantization design and the effectiveness of integrating CCQ into practical inference kernels such as vLLM wna16.
As shown in Table~\ref{tab:operator-performance} and Table~\ref{tab:operator-group-gemm-performance}, our CCQ-based implementation consistently delivers superior inference efficiency compared to both traditional and state-of-the-art quantization baselines.
%
In GEMV scenarios, CCQ achieves up to a 2$\times$ speedup over VPTQ and closely matches the performance of standard WINT2, demonstrating that our codebook-free, bit-shift-based dequantization design effectively eliminates the memory access bottlenecks inherent to vector quantization.
%
In Grouped-GEMM settings, which are critical for large-scale MoE inference, CCQ (W2A16-CCQ) not only outperforms widely adopted low-bit operators such as W8A16 and W4A16, but also maintains its latency advantage as batch size and matrix dimensions increase.
%
This scalability underscores the practical value of CCQ for real-world LLM deployment, where both memory footprint and inference speed are crucial.
%
Overall, the results validate that CCQ's hardware-friendly encoding and decoding pipeline enables highly efficient, large-scale inference without sacrificing model accuracy, making it a compelling solution for low-bit LLM quantization.
\begin{table}[h]
\renewcommand{\arraystretch}{1.2}
\centering
\caption{Operator-level performance comparison on GEMV kernels (execution time in milliseconds).}
\label{tab:operator-performance}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}cccc@{}}
\toprule
Matrix Shape & \multirow{2}{*}{VPTQ} & \multirow{2}{*}{WINT2-Standard} & \multirow{2}{*}{CCQ} \\
Input Size $\times$ Output Size & & & \\
% Matrix Shape & VPTQ & \multirow{2}{*}{WINT2-Standard} & \multirow{2}{*}{WINT2-CCQ} \\
% Input Size $\times$ Output Size & bpw=3 & & \\
% Matrix Shape & VPTQ & WINT2-Standard & WINT2-CCQ \\
% Input Size $\times$ Output Size & bpw=3 & & \\
\midrule
$4096 \times 4096$ (M=1) & 0.058 & 0.027 & 0.034 \\
$4096 \times 1024$ (M=1) & 0.033 & 0.015 & 0.017 \\
$8192 \times 8192$ (M=1) & 0.174 & 0.073 & 0.090 \\
$8192 \times 1024$ (M=1) & 0.061 & 0.019 & 0.022 \\
\midrule
$4096 \times 4096$ (M=4) & 0.205 & 0.028 & 0.035 \\
$4096 \times 1024$ (M=4) & 0.112 & 0.015 & 0.017 \\
$8192 \times 8192$ (M=4) & 0.641 & 0.076 & 0.104 \\
$8192 \times 1024$ (M=4) & 0.216 & 0.019 & 0.024 \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[h]
\renewcommand{\arraystretch}{1.2}
\centering
\caption{Operator-level performance comparison on Grouped-GEMM kernels (execution time in milliseconds).}
\label{tab:operator-group-gemm-performance}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}cccc@{}}
\toprule
Matrix Shape & \multirow{2}{*}{W8A16} & \multirow{2}{*}{W4A16} & \multirow{2}{*}{W2A16-CCQ} \\
Groups $\times$ Output Size $\times$ Input Size & & & \\
% Matrix Shape & W8A16 & W4A16 & W2A16-CCQ \\
% Groups $\times$ Output Size $\times$ Input Size & & & \\
\midrule
$256 \times 7168 \times 512$ (M=1)& 0.044 & 0.030 & 0.021 \\
$256 \times 1024 \times 7168$ (M=1) & 0.100 & 0.068 & 0.058 \\
\midrule
$256 \times 7168 \times 512$ (M=4) & 0.053 & 0.035 & 0.027 \\
$256 \times 1024 \times 7168$ (M=4) & 0.315 & 0.199 & 0.166 \\
\midrule
$256 \times 7168 \times 512$ (M=16) & 0.104 & 0.071 & 0.059 \\
$256 \times 1024 \times 7168$ (M=16) & 1.052 & 0.624 & 0.527 \\
\midrule
$256 \times 7168 \times 512$ (M=64) & 0.297 & 0.193 & 0.161 \\
$256 \times 1024 \times 7168$ (M=64) & 2.655 & 1.628 & 1.386 \\
\bottomrule
\end{tabular}
}
\end{table}
\section{Conclusion}
In this study, we propose \textbf{convolutional code quantization (CCQ)} to address the inference limitations inherent in vector quantization.
%
By integrating convolutional codes with scalar quantization, we construct a lookup-free encoding space that achieves a linear mapping between the codebook and weight vectors, thereby optimizing inference performance.
%
Meanwhile, by maintaining high-bit quantization precision for weights, we minimize the performance degradation of the model under extremely low-bit conditions.
%
Through the introduction of hybrid encoding, we can flexibly apply coding quantization to improve the compression ratio.
%
Furthermore, by leveraging the characteristic of normal distribution of convolutional code values across weights, we propose Code Cluster to reduce redundancy in the encoding space.
%
Ultimately, we expand the encoding configurations of CCQ, which not only enhances model accuracy but also further compresses the model size.
\section{Future Work}
%
In the domain of extreme low-bit quantization, we identify loss-based dynamic bit allocation as a critical research direction.
%
For instance, during ERNIE 4.5 quantization, we observe that allocating only the 6th MoE layer as WINT4 (instead of CCQ) could maintain high accuracy, while the other expert layers retained CCQ with bpw equal to 2.06.
%
This finding prompts us to ponder whether certain individual layers in LLMs play a pivotal role in preserving the model's capabilities. Meanwhile, our proposed algorithmic framework for encode quantization can theoretically be extended to other coding theories, demonstrating significant scalability.
%
This will also be one of the key directions for our future exploration.
%
While our experiments validate the efficacy on the MoE architecture, further investigation is warranted for dense models under state-of-the-art 2-bit quantization methods such as QTIP~\cite{tseng2024qtip} and VPTQ~\cite{liu2024vptq}.
%
Additionally, inference kernel optimization presents substantial headroom through joint consideration of permutation strategies during both quantization and runtime execution.
%
Co-designing these phases could yield latency reductions, particularly for memory-bound operations on devices.
% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}
\end{document}
\begin{thebibliography}{38}
\providecommand{\natexlab}[1]{#1}
\bibitem[{Badri and Shaji(2023)}]{badri2023hqq}
Hicham Badri and Appu Shaji. 2023.
\newblock \href {https://mobiusml.github.io/hqq_blog/} {Half-quadratic quantization of large machine learning models}.
\bibitem[{Badri and Shaji(2024)}]{badri2023hqq+}
Hicham Badri and Appu Shaji. 2024.
\newblock \href {https://mobiusml.github.io/1bit_blog/} {Towards 1-bit machine learning models}.
\bibitem[{Baidu-ERNIE-Team(2025)}]{ernie2025technicalreport}
Baidu-ERNIE-Team. 2025.
\newblock Ernie 4.5 technical report.
\bibitem[{Cobbe et~al.(2021)Cobbe, Kosaraju, Bavarian, Chen, Jun, Kaiser, Plappert, Tworek, Hilton, Nakano et~al.}]{cobbe2021training}
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et~al. 2021.
\newblock Training verifiers to solve math word problems.
\newblock \emph{arXiv preprint arXiv:2110.14168}.
\bibitem[{DeepSeek-AI et~al.(2025)DeepSeek-AI, Liu, Feng, and et.al.}]{deepseekai2025deepseekv3technicalreport}
DeepSeek-AI, Aixin Liu, Bei Feng, and et.al. 2025.
\newblock \href {https://arxiv.org/abs/2412.19437} {Deepseek-v3 technical report}.
\newblock \emph{Preprint}, arXiv:2412.19437.
\bibitem[{Dua et~al.(2019)Dua, Wang, Dasigi, Stanovsky, Singh, and Gardner}]{dua2019drop}
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.
\newblock Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.
\newblock \emph{arXiv preprint arXiv:1903.00161}.
\bibitem[{Egiazarian et~al.(2024)Egiazarian, Panferov, Kuznedelev, Frantar, Babenko, and Alistarh}]{egiazarian2024extreme}
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024.
\newblock \href {https://arxiv.org/abs/2401.06118} {Extreme compression of large language models via additive quantization}.
\newblock \emph{Preprint}, arXiv:2401.06118.
\bibitem[{Frantar et~al.(2022)Frantar, Ashkboos, Hoefler, and Alistarh}]{frantar2022gptq}
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.
\newblock Gptq: Accurate post-training quantization for generative pre-trained transformers.
\newblock \emph{arXiv preprint arXiv:2210.17323}.
\bibitem[{Grattafiori et~al.(2024)Grattafiori, Dubey, and et.al.}]{grattafiori2024llama3herdmodels}
Aaron Grattafiori, Abhimanyu Dubey, and et.al. 2024.
\newblock \href {https://arxiv.org/abs/2407.21783} {The llama 3 herd of models}.
\newblock \emph{Preprint}, arXiv:2407.21783.
\bibitem[{Gray(1984)}]{gray1984vector}
Robert Gray. 1984.
\newblock Vector quantization.
\newblock \emph{IEEE Assp Magazine}, 1(2):4--29.
\bibitem[{Gray and Neuhoff(1998)}]{gray1998quantization}
Robert~M. Gray and David~L. Neuhoff. 1998.
\newblock Quantization.
\newblock \emph{IEEE transactions on information theory}, 44(6):2325--2383.
\bibitem[{Hendrycks et~al.(2021)Hendrycks, Burns, Basart, Zou, Mazeika, Song, and Steinhardt}]{hendryckstest2021}
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.
\newblock Measuring massive multitask language understanding.
\newblock \emph{Proceedings of the International Conference on Learning Representations (ICLR)}.
\bibitem[{Hooper et~al.(2024)Hooper, Kim, Mohammadzadeh, Mahoney, Shao, Keutzer, and Gholami}]{hooper2024kvquant}
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael~W Mahoney, Yakun~S Shao, Kurt Keutzer, and Amir Gholami. 2024.
\newblock Kvquant: Towards 10 million context length llm inference with kv cache quantization.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:1270--1303.
\bibitem[{Huang et~al.(2023)Huang, Bai, Zhu, Zhang, Zhang, Su, Liu, Lv, Zhang, Lei, Fu, Sun, and He}]{huang2023ceval}
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023.
\newblock C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
\newblock In \emph{Advances in Neural Information Processing Systems}.
\bibitem[{Kim et~al.(2023)Kim, Hooper, Gholami, Dong, Li, Shen, Mahoney, and Keutzer}]{kim2023squeezellm}
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael~W Mahoney, and Kurt Keutzer. 2023.
\newblock Squeezellm: Dense-and-sparse quantization.
\newblock \emph{arXiv preprint arXiv:2306.07629}.
\bibitem[{Kwon et~al.(2023)Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, and Stoica}]{kwon2023efficient}
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody~Hao Yu, Joseph~E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.
\newblock Efficient memory management for large language model serving with pagedattention.
\newblock In \emph{Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles}.
\bibitem[{Lee et~al.(2025)Lee, Kim, You, and Kim}]{lee2025littlebitultralowbitquantization}
Banseok Lee, Dongkyu Kim, Youngcheon You, and Youngmin Kim. 2025.
\newblock \href {https://arxiv.org/abs/2506.13771} {Littlebit: Ultra low-bit quantization via latent factorization}.
\newblock \emph{Preprint}, arXiv:2506.13771.
\bibitem[{Li et~al.(2025)Li, Yin, Lee, Xiao, and Panda}]{li2025gptaq}
Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. 2025.
\newblock Gptaq: Efficient finetuning-free quantization for asymmetric calibration.
\newblock \emph{arXiv preprint arXiv:2504.02692}.
\bibitem[{Lin et~al.(2024)Lin, Tang, Tang, Yang, Chen, Wang, Xiao, Dang, Gan, and Han}]{lin2024awq}
Ji~Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024.
\newblock Awq: Activation-aware weight quantization for on-device llm compression and acceleration.
\newblock \emph{Proceedings of Machine Learning and Systems}, 6:87--100.
\bibitem[{Liu et~al.(2024{\natexlab{a}})Liu, Feng, Xue, Wang, Wu, Lu, Zhao, Deng, Zhang, Ruan et~al.}]{liu2024deepseek}
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et~al. 2024{\natexlab{a}}.
\newblock Deepseek-v3 technical report.
\newblock \emph{arXiv preprint arXiv:2412.19437}.
\bibitem[{Liu et~al.(2024{\natexlab{b}})Liu, Wen, Wang, Ye, Zhang, Cao, Li, and Yang}]{liu2024vptq}
Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li~Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. 2024{\natexlab{b}}.
\newblock Vptq: Extreme low-bit vector post-training quantization for large language models.
\newblock \emph{arXiv preprint arXiv:2409.17066}.
\bibitem[{Liu et~al.(2023)Liu, Oguz, Zhao, Chang, Stock, Mehdad, Shi, Krishnamoorthi, and Chandra}]{liu2023llm}
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023.
\newblock Llm-qat: Data-free quantization aware training for large language models.
\newblock \emph{arXiv preprint arXiv:2305.17888}.
\bibitem[{Liu et~al.(2024{\natexlab{c}})Liu, Yuan, Jin, Zhong, Xu, Braverman, Chen, and Hu}]{liu2024kivi}
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024{\natexlab{c}}.
\newblock Kivi: A tuning-free asymmetric 2bit quantization for kv cache.
\newblock \emph{arXiv preprint arXiv:2402.02750}.
\bibitem[{Malinovskii et~al.(2024)Malinovskii, Mazur, Ilin, Kuznedelev, Burlachenko, Yi, Alistarh, and Richtarik}]{malinovskii2024pvtuning}
Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. 2024.
\newblock \href {https://arxiv.org/abs/2405.14852} {Pv-tuning: Beyond straight-through estimation for extreme llm compression}.
\newblock \emph{Preprint}, arXiv:2405.14852.
\bibitem[{Marcellin and Fischer(1990)}]{marcellin1990trellis}
Michael~W Marcellin and Thomas~R Fischer. 1990.
\newblock Trellis coded quantization of memoryless and gauss-markov sources.
\newblock \emph{IEEE transactions on communications}, 38(1):82--93.
\bibitem[{Sprague et~al.(2023)Sprague, Ye, Bostrom, Chaudhuri, and Durrett}]{sprague2023musr}
Zayne Sprague, Xi~Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2023.
\newblock Musr: Testing the limits of chain-of-thought with multistep soft reasoning.
\newblock \emph{arXiv preprint arXiv:2310.16049}.
\bibitem[{Srivastava et~al.(2022)Srivastava, Rastogi, Rao, Shoeb, Abid, Fisch, Brown, Santoro, Gupta, Garriga-Alonso et~al.}]{srivastava2022beyond}
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal~Md Shoeb, Abubakar Abid, Adam Fisch, Adam~R Brown, Adam Santoro, Aditya Gupta, Adri{\`a} Garriga-Alonso, et~al. 2022.
\newblock Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
\newblock \emph{arXiv preprint arXiv:2206.04615}.
\bibitem[{Tillet et~al.(2019)Tillet, Kung, and Cox}]{tillet2019triton}
Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019.
\newblock Triton: an intermediate language and compiler for tiled neural network computations.
\newblock In \emph{Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages}, pages 10--19.
\bibitem[{Tseng et~al.(2024{\natexlab{a}})Tseng, Chee, Sun, Kuleshov, and De~Sa}]{tseng2024quip}
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De~Sa. 2024{\natexlab{a}}.
\newblock Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks.
\newblock \emph{arXiv preprint arXiv:2402.04396}.
\bibitem[{Tseng et~al.(2024{\natexlab{b}})Tseng, Sun, Hou, and De~Sa}]{tseng2024qtip}
Albert Tseng, Qingyao Sun, David Hou, and Christopher~M De~Sa. 2024{\natexlab{b}}.
\newblock Qtip: Quantization with trellises and incoherence processing.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:59597--59620.
\bibitem[{van Baalen et~al.(2024)van Baalen, Kuzmin, Nagel, Couperus, Bastoul, Mahurin, Blankevoort, and Whatmough}]{vanbaalen-gptvq}
Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. 2024.
\newblock Gptvq: The blessing of dimensionality in llm quantization.
\newblock \emph{arXiv preprint arXiv:2402.15319}.
\bibitem[{Wang et~al.(2023)Wang, Ma, Dong, Huang, Wang, Ma, Yang, Wang, Wu, and Wei}]{wang2023bitnet}
Hongyu Wang, Shuming Ma, Li~Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi~Wu, and Furu Wei. 2023.
\newblock Bitnet: Scaling 1-bit transformers for large language models.
\newblock \emph{arXiv preprint arXiv:2310.11453}.
\bibitem[{Wei et~al.(2023)Wei, Luan, Liu, Dong, and Wang}]{wei2023cmath}
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. 2023.
\newblock Cmath: Can your language model pass chinese elementary school math test?
\newblock \emph{arXiv preprint arXiv:2306.16636}.
\bibitem[{Xiao et~al.(2023)Xiao, Lin, Seznec, Wu, Demouth, and Han}]{xiao2023smoothquant}
Guangxuan Xiao, Ji~Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023.
\newblock Smoothquant: Accurate and efficient post-training quantization for large language models.
\newblock In \emph{International Conference on Machine Learning}, pages 38087--38099. PMLR.
\bibitem[{Xu et~al.(2024)Xu, Han, Yang, Wang, Zhu, Liu, Liu, and Che}]{xu2024onebitextremelylowbitlarge}
Yuzhuang Xu, Xu~Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024.
\newblock \href {https://arxiv.org/abs/2402.11295} {Onebit: Towards extremely low-bit large language models}.
\newblock \emph{Preprint}, arXiv:2402.11295.
\bibitem[{Yang et~al.(2025)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, Lv, Zheng, Liu, Zhou, Huang, Hu, Ge, Wei, Lin, Tang, Yang, Tu, Zhang, Yang, Yang, Zhou, Zhou, Lin, Dang, Bao, Yang, Yu, Deng, Li, Xue, Li, Zhang, Wang, Zhu, Men, Gao, Liu, Luo, Li, Tang, Yin, Ren, Wang, Zhang, Ren, Fan, Su, Zhang, Zhang, Wan, Liu, Wang, Cui, Zhang, Zhou, and Qiu}]{qwen3}
An~Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le~Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu~Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025.
\newblock Qwen3 technical report.
\newblock \emph{arXiv preprint arXiv:2505.09388}.
\bibitem[{Yang et~al.(2024{\natexlab{a}})Yang, Yang, Hui, Zheng, Yu, Zhou, Li, Li, Liu, Huang, Dong, Wei, Lin, Tang, Wang, Yang, Tu, Zhang, Ma, Xu, Zhou, Bai, He, Lin, Dang, Lu, Chen, Yang, Li, Xue, Ni, Zhang, Wang, Peng, Men, Gao, Lin, Wang, and et.al.}]{qwen2}
An~Yang, Baosong Yang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na~Ni, Pei Zhang, Peng Wang, Ru~Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, and et.al. 2024{\natexlab{a}}.
\newblock Qwen2 technical report.
\newblock \emph{arXiv preprint arXiv:2407.10671}.
\bibitem[{Yang et~al.(2024{\natexlab{b}})Yang, Yang, Zhang, Hui, Zheng, Yu, Li, Liu, Huang, Wei, Lin, Yang, Tu, Zhang, Yang, Yang, Zhou, Lin, Dang, Lu, Bao, Yang, Yu, Li, Xue, Zhang, Zhu, and et.al.}]{qwen2.5}
An~Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le~Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, and et.al. 2024{\natexlab{b}}.
\newblock Qwen2.5 technical report.
\newblock \emph{arXiv preprint arXiv:2412.15115}.
\end{thebibliography}