SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation
Extracted source text · arXiv 2503.06764
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
%%% custom package
\usepackage{multirow}
\usepackage{colortbl}
\usepackage{graphicx}
\usepackage{makecell}
\usepackage{utfsym}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{soul} % 用于支持高亮\usepackage{wrapfig}
\usepackage{caption}
\usepackage{adjustbox}
\usepackage{amssymb}
\usepackage{graphicx} % 提供 \scalebox 命令
\usepackage{bm} % 提供 \boldsymbol 命令，用于加粗数学符号
\usepackage{float} % 提供更精确的浮动体控制
\usepackage{placeins}
\usepackage{pifont}
% \usepackage[round,authoryear]{natbib}
% \usepackage{natbib}
% \setcitestyle{numbers,square}
\usepackage{svg}
\usepackage[ruled]{algorithm2e}
\usepackage{colortbl} % color table rows
\usepackage{enumitem}
\definecolor{darkgreen}{RGB}{0,150,0}
\newcommand{\redcross}{\scalebox{1.5}{\textcolor{red}{\boldsymbol{$\times$}}}}
\newcommand{\greentick}{\scalebox{1.5}{\textcolor{darkgreen}{\boldsymbol{$\checkmark$}}}}
\newcommand{\tokenizer}{SemHiTok}
\newcommand{\czs}[1]{{\color{red}{[TODO:#1]}}}
\newcommand{\figrefef}[1]{Fig.~\ref{#1}}
\definecolor{myblue}{RGB}{47,85,151}
\definecolor{myred}{RGB}{242,160,120}
\newcommand{\graycolorrow}[1]{\rowcolor{gray!15} #1}
\newcommand{\suppltext}[1]{\textbf{\textcolor{blue}{Supplementary Material #1}}}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\usepackage{etoc}
\definecolor{lightred}{RGB}{235, 90, 90}
\definecolor{brickred}{RGB}{180, 50, 50}
\definecolor{kingblue}{RGB}{25, 25, 170}
\newcommand{\rev}[1]{\textcolor{brickred}{#1}}
\renewcommand{\rev}[1]{#1}
\title{\tokenizer: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
% \author{%
% Zisheng Chen$^{1}$ \thanks{Work done as an intern at Huawei Noah’s Ark Lab.} \And
% Chunwei Wang$^{2}$ \And
% Runhui Huang$^{3}$ \And
% Hongbin Xu$^{4}$ \And
% Xiuwei Chen$^{1}$ \And
% Jun Zhou$^{1}$ \And
% Jianhua Han$^{2}$ \And
% Hang Xu$^{2}$ \And
% Xiaodan Liang\footnotemark[2] $^{\ 1}$ \AND
% \vspace{-0.8cm}\\
% $^1$ Sun Yat-sen University \quad
% $^2$ Huawei Noah’s Ark Lab \quad \\
% $^3$ University of Hong Kong \quad
% $^4$ South China University of Technology \quad
% \vspace{0.15cm}\\
% $^\dagger$Corresponding Author
% }
\author{%
\textbf{Zisheng Chen$^{1, \ast}$,
Chunwei Wang$^{2}$,
Runhui Huang$^{3}$,
Hongbin Xu$^{4}$,
Xiuwei Chen$^{1}$,} \\
\textbf{Jun Zhou$^{1}$,
Jianhua Han$^{2}$,
Hang Xu$^{2}$,
Xiaodan Liang$^{1, \dagger}$} \\[0.3cm]
$^1$ Sun Yat-sen University \quad
$^2$ Huawei Noah’s Ark Lab \\
$^3$ University of Hong Kong \quad
$^4$ South China University of Technology \\[0.2cm]
\small $^\ast$Work done as an intern at Huawei Noah’s Ark Lab \\
\small $^\dagger$Corresponding Author
}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\input{sec/0_abstract}
\input{sec/1_intro}
\input{sec/2_method}
\input{sec/3_experiments}
\input{sec/4_ablation}
\input{sec/5_related_work}
\input{sec/6_conclusion}
\newpage
\input{sec/7_statement}
\newpage
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
\newpage
\input{sec/8_appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
% \begin{figure}[ht]
% % \vspace{-16pt}
% \begin{center}
% \includegraphics[width=0.85\linewidth]{figs/fig-abs.png}
% \end{center}
% % \vspace{-7pt}
% \caption{Generated images from unified MLLM equipped with our SemHiTok.
% }
% % \vspace{-2pt}
% \label{fig:abs}
% \end{figure}
\begin{abstract}
\label{sec:abs}
In this paper, we introduce \textbf{SemHiTok}, a unified image \textbf{Tok}enizer via \textbf{Sem}antic-Guided \textbf{Hi}erarchical codebook that provides consistent discrete representations for multimodal understanding and generation.
Recently, unified image tokenizers have sparked exploration within the research community,
which is designed to capture high-level semantic features for understanding and retaining low-level pixel features for generation.
Previous works attempt to train a unified image tokenizer by combining loss for semantic distillation and pixel reconstruction.
However, due to the differing levels of features prioritized by multimodal understanding and generation, joint training methods face significant challenges in achieving a good trade-off.
SemHiTok addresses this challenge through a novel semantic-guided hierarchical codebook, which builds pixel sub-codebooks on a pretrained semantic codebook.
This design decouples the semantic and pixel in terms of structure and training strategy, enabling the tokenizer to capture pixel features while retaining its ability to comprehend high-level semantic information.
Our experiments demonstrate that SemHiTok achieves leading performance in image reconstruction and multimodal understanding under the LLaVA-v1.5 setting.
Further, we develop a unified MLLM with SemHiTok, which exhibits superior performance across multimodal understanding and generation tasks.
Extensive experiments confirm our analysis, showing that our unified image tokenizer architecture achieves a better trade-off.
% For understanding, SemHiTok achieves impressive performance on most benchmarks.
% For generations, our model achieves state-of-the-art (SOTA) performance on MJHQ30K within unified MLLMs, while also demonstrating competitive performance on other generation tasks.
% Our code and models will be open source.
\end{abstract}
\section{Introduction}
\label{sec:intro}
%% LLM的兴起 人们开始研究理解生成一体的模型
%% 自回归模型在自然语言处理领域内获得了巨大的成功，人们开始希望在将自回归的范式推广到其他模态领域，例如视觉、音频、点云等。
In recent years, autoregressive models have achieved great success in natural language processing and have been extended to the multimodal understanding domain, demonstrating immense potential.
This triggers researchers' interest in unified multimodal understanding and generation by employing a single autoregressive framework.
To achieve a unified multimodal large model, the key challenge is designing a tokenizer suitable for both multimodal generation and understanding tasks.
However, there is a vast gap in the visual information required for these two tasks.
For instance, models from the CLIP~\citep{clip,evaclip,siglip} family, commonly used in multimodal understanding tasks, tend to lose visual pixel information.
On the contrary, the VQGAN~\citep{vqgan,vqganlc} family models, often used in autoregressive generation tasks, lack the ability to extract semantic features for multimodal understanding tasks.
This leads to poor performance when a single tokenizer is applied to a unified MLLM~\citep{liquid,llavit,seedllama}.
In light of the aforementioned issues, some recent work has attempted to incorporate a semantic learning branch into the original VQGAN training pipeline, aiming to obtain a unified tokenizer via joint optimization.
VILA-U~\citep{vilau} employs a straightforward combination of semantic alignment loss and pixel reconstruction loss, which allows the model to capture both low-level and high-level information.
Nevertheless, the reliance on a hybrid structure and joint optimization often drives the tokenizer toward a suboptimal solution.
Furthermore, some recent works~\citep{musevl, tokenflow} have built upon it with further improvements.
As shown in Fig.\ref{fig:comp}(a), while SDE~\citep{musevl} further decouples the encoders, the remaining hybrid codebook continues to impede the model's optimization.
% SDE~\citep{musevl}employs a dual specialized encoder, semantic and pixel information still share a single codebook, which leads to the same problem.
TokenFlow~\citep{tokenflow} uses shared mapping to decouple the semantic branch and pixel branch while maintaining the consistency of the codebook index, but joint training still affects the final performance.
A straightforward approach is to use CLIP and VQGAN to extract semantic and pixel information, respectively, and the concatenation of these two token sequences is then used as a unified representation.
Janus~\citep{janus} introduces a dual-encoder method that separates encoders for understanding and generation tasks to address this conflict, but this increases the complexity of handling mixed tasks and does not fundamentally resolve the feature conflict challenge.
However, this leads to a doubling of the token sequence count or multiplicative expansion in vocabulary size, depending on whether the concatenation is applied along the length or dimension.
% , as shown in \figref{fig:dual}.
These limitations underscore a fundamental challenge in the field:
\textit{How to balance semantic-level and pixel-level information effectively, without compromising the ease of integration into MLLM frameworks?}
% \textit{how to balance semantic-level and pixel-level information effectively ?}
\begin{figure}[t]
% \begin{minipage}{\textwidth}
\begin{center}
\includegraphics[width=0.95\linewidth]{figs/fig-compare.pdf}
\captionof{figure}{
Illustration of other tokenizers and SemHiTok.
\textbf{HyStruct}: Using a single model to extract information at different levels;
\textbf{DStruct}: Using different models to extract information at various levels;
\textbf{JTrain}: Using a joint optimization training strategy;
\textbf{DTrain}: Adopt a phased optimization training strategy.}
\label{fig:comp}
\end{center}
% \end{minipage}
% \begin{minipage}{\textwidth}
% \hspace{3mm}
% \begin{center}
% \caption{Characteristics comparison among existing unified image tokenizer.
% % Previous works attempt to achieve a better trade-off by structural separation or more powerful codebook type.
% % However, these methods do not achieve separate training.
% % Additionally, their specialized codebook structures lack consideration of compatibility with MLLMs.
% % In comparison, our approach offers a better trade-off and is widely compatible with mainstream MLLMs following the next-token prediction paradigm.
% }\label{tab:comp}
% \resizebox{0.95\linewidth}{!}{
% \begin{tabular}{l ccc ccc }
% \toprule
% Tokenizer & \makecell{VILA-U \\ ~\citep{vilau}} & \makecell{TokenFlow \\ ~\citep{tokenflow} } & \makecell{SDE \\ ~\citep{musevl}} & \makecell{TokLIP \\ ~\citep{toklip}} & \makecell{UniTok \\ ~\citep{unitok}} & \makecell{SemHiTok \\ (Ours)} \\ \midrule
% Codebook Structure & RQ & \makecell{Share-Mapping\\\& MSVQ} & Vanilla & Hierarchical \\ \midrule
% Separate Training & \redcross & \redcross & \redcross & \greentick \\
% Structural Separation & \redcross & \greentick & \redcross & \greentick \\
% Unified Representation & \greentick & \redcross & \greentick & \greentick \\
% % Unified MLLM head & \redcross & \greentick & \greentick & \greentick \\
% \bottomrule
% \end{tabular}
% % }
% }
% \end{center}
% \end{minipage}
\end{figure}
To address this challenge, we propose \textbf{SemHiTok}, a unified image tokenizer that provides consistent feature representations for multimodal understanding and generation tasks through a unique hierarchical codebook design.
% As shown in \figref{fig:comp}(d),
Inspired by the observation that image patches with the same semantic code tend to have similar pixel features, we introduce a novelty hierarchical codebook which uses a sub-codebook to model the pixel-level space associated with each semantic code, named Semantic-Guided Hierarchical Codebook(SGHC).
Unlike existing approaches, SemHiTok supports a stage-wise training paradigm where each stage exclusively optimizes specific hierarchy level features, allowing us to achieve a better trade-off between semantic and pixel feature extraction.
In addition, SemHiTok can be seamlessly integrated into existing MLLMs following the next-token paradigm through a simple codebook flattening operation.
% Benefiting from SGHC, SemHiTok achieves both separate training and structural separation, and can be seamlessly integrated into existing MLLMs following the next-token paradigm through a simple codebook flattening operation.
% Unlike existing approaches that directly employ a single codebook to encode different level features, SemHiTok's semantic-guided hierarchical codebook support stage-wise training paradigm where each stage exclusively optimizes specific hierarchy level features.
% This novel design allows us to achieve a better trade-off between semantic and texture feature extraction.
% our contributions can be summarized as follows: \label{check:contribution}
% \begin{itemize}
% \item
% A novel unified tokenizer that achieves a trade-off between semantic and pixel information, demonstrating outstanding performance in both image reconstruction and multimodal understanding tasks under LLaVA-v1.5 setting.
% \item
% We develop a unified MLLM architecture that demonstrates superior performance across both multimodal understanding and generation tasks, validating its versatility.
% \item
% Our approach further pushes the performance boundary of unified discrete MLLMs, enabling improved scalability and representation capacity within next-token prediction frameworks.
% \end{itemize}
Our contributions can be summarized as follows: \label{check:contribution}
\textbf{(1)}: A novel unified tokenizer that achieves a trade-off between semantic and pixel information, demonstrating outstanding performance in both image reconstruction and multimodal understanding tasks.
\textbf{(2)}: We develop a unified MLLM architecture that demonstrates superior performance across both multimodal understanding and generation tasks, validating its versatility.
\textbf{(3)}: Our approach further pushes the performance boundary of unified discrete MLLMs, enabling improved scalability and representation capacity within next-token prediction frameworks.
\label{check:contribution}
% Our framework exhibits remarkable performance on multimodal understanding and generation tasks.
% For multimodal understanding, we surpass other next-token-based approaches using discrete visual inputs under the LLaVA-1.5 setting.
% SemHiTok also achieves competitive rFID metrics at a resolution of $256\times256$ on ImageNet.
% For text-to-image generation tasks, we achieve competitive results, compared existing next-token autoregressive generation methods.
% These findings indicate that SemHiTok achieves a better trade-off between semantic and texture feature extraction capabilities, and demonstrates the great potential for application in MLLMs.
\section{Method}
\label{sec:method}
\begin{figure}[t]
\centering
\includegraphics[width=0.98\linewidth]{figs/fig-framwork.pdf}
\caption{
% Overview of SemHiTok.
(a) SemHiTok is structurally composed of two branches: \textbf{\textcolor{myblue}{semantic branch}} and \textbf{\textcolor{myred}{pixel branch}}.
The \textbf{\textcolor{myblue}{semantic branch}} is trained following the VQKD~\citep{vqkd}, where the semantic codebook is learned through semantic loss.
We propose a semantic-guided hierarchical codebook(SGCH) composed of multiple pixel sub-codebooks, in which each pixel sub-codebook is in a one-to-one correspondence with a semantic code.
The selection of pixel sub-codebook is indexed by the semantic code from semantic quantization.
To enable a unified discrete representation, we concatenate the quantized semantic and pixel features along the channel dimension and feed the result into the pixel decoder for reconstruction.
(b) Each semantic code is allocated to the corresponding pixel sub-codebook, and their features are concatenated along the dimension.
(c) An illustration of the unified MLLM framework.%
}
\label{fig:overview}
\end{figure}
The main objective of \textbf{SemHiTok} is to establish a simple and unified image tokenizer for multimodal understanding and generation.
In this model, the image is transformed into discrete tokens that contain semantic information and pixel information.
We begin with a semantic codebook training recipe that reconstructs semantic features extracted from a language image pre-training model~\citep{siglip, vqkd}, and point out the semantic codebook's poor texture feature representation in section \ref{subsec:svq}.
In section \ref{subsec:pixsubc}, we conduct a preliminary discussion and observation.
% that the image patches corresponding to the semantic code exhibit similar texture features.
% Then, we find that the codes in the SVQ codebook correspond to semantic concepts, and same semantic concepts share similar texture features.
Building on this observation, we introduce Semantic-Guided Hierarchical Codebook(SGHC), incorporating texture information while perfectly inheriting the semantic information of the semantic codebook, to enable pixel reconstruction enablement. %in section \ref{subsec:pixsubc}. % 要换成方法的名字
% Finally, Unified MLLM training procedure is introduced in section \ref{subsec:unimllm}.
Finally, we introduce the application of SemHiTok on unified MLLM in section \ref{subc:mmdt}.
The overview framework is shown in Fig.\ref{fig:overview}.
\subsection{Semantic Codebook Training}
\label{subsec:svq}
For multimodal understanding, using a text-aligned visual encoder~\citep{siglip, clip, evaclip, qwen2vl} as an image tokenizer can accelerate convergence and improve performance.
However, these text-aligned visual encoders typically output continuous semantic features.
In this work, to achieve a unified visual tokenizer, we first train a semantic codebook to quantize the continuous semantic feature following VQKD~\citep{vqkd}.
% To achieve a unified visual tokenizer, we first quantize the continuous semantic feature space.
Given an image $X^{H\times W\times 3}$, the semantic encoder $\mathcal{E}_\text{sem}$ extract continuous semantic features:
\begin{equation}
Z_{sem}=\mathcal{E}_\text{sem}(X)\in \mathbb{R}^{h\times w\times d_{sem}}
\end{equation}
Where $\mathcal{E}_\text{sem}$ is a frozen text-aligned image encoder, \textit{e.g.}, CLIP~\citep{clip} or SigLIP~\citep{siglip}.
% Where $\mathcal{E}_\text{sem}$ is a cascaded module consisting of a frozen text-aligned image encoder and a trainable projection layer for aligning the codebook dimension.
Then $Z_{sem}$ are transformed into discrete feature space $\mathcal{C}_{sem}=\{c_1, c_2,...,c_K\}\in \mathbb{R}^{K\times d_{sem}}$ through quantization function $\mathcal{Q}_{sem}(*)$.
The quantization process $\mathcal{Q}_{sem}(*)$ is as follows:
\begin{equation}
Z_{q_{sem}}, I_{q_{sem}} = \mathop{\arg\min}\limits_{k\in \{1,\dots,K\}}{\|Z_{sem}-\mathcal{C}_{sem}[k] \|}
\end{equation}
Where $I_{q_{sem}}\in [\mathcal{C}_{sem}]^{h \times w}$ is quantized index, $Z_{q_{sem}}$ is discrete feature indexed from $\mathcal{C}_{sem}$.
Finally, semantic decoder $\mathcal{D}_{sem}$ maps $Z_{q_{sem}}$ to raw semantic feature space $\hat{Z}_{sem}$.
The $\mathcal{D}_{sem}$ are end-to-end trainable by minimizing semantic distill loss:
\begin{equation}
L_{sem}=1-\cos(Z_{q_{sem}}, \hat{Z}_{sem}) + |Z_{q_{sem}} - \hat{Z}_{sem}|
\end{equation}
% and quantization loss $\mathcal{L}_q(Q)$.
For $\mathcal{C}_{sem}$, we adopt EMA~\citep{ema} VQ as the semantic codebook.
Unlike traditional quantization methods, the EMA VQ is not updated via gradient descent, but instead through an Exponential Moving Average (EMA) algorithm:
\begin{equation}
\mathbf{c}_k^{(t)} = m \cdot \mathbf{c}_k^{(t-1)} + (1 - m) \cdot \frac{1}{N_k} \sum_{i=1}^{N_k} \mathbf{z}_i,
\label{eq:ema-update}
\end{equation}
where $\mathbf{c}_k^{t}$ denotes the $k$-th codebook vector at update step $t$, $m$ is the momentum term, and the update is based on the average of all input vectors $\mathbf{z}_i$ assigned to code $\mathbf{k}$ in the current batch.
% We conduct multimodal understanding experiments using discrete tokens from $\mathcal{C}_{sem}$ in LLaVA-v1.5 setting as shown in Tab.\ref{tab:llava}.
% The experiments show that our semantic codebook $\mathcal{C}_{sem}$ is slightly inferior to continuous features but still viable.
However, we further conduct an experiment that reconstructs the original pixel from the quantized semantic features extracted by $\mathcal{C}_{sem}$.
The reconstructed images exhibited noticeable blurriness and a significant loss of high-frequency details, as shown in Fig. \ref{fig:sem_rec}.
This indicates that the semantic codebook lacks pixel information.
\begin{figure}[t] % 用不带星号的 figure 环境，就不会跨栏，但是会上下浮动。可以去查一下关于浮动体的控制。
\centering
\begin{minipage}[b]{.48\linewidth}
\hspace{-5mm}
\includegraphics[width=1.\linewidth]{figs/fig-vqkd-rec.pdf}
\caption{Visualization of reconstruction by quantized semantic features.
Semantic codebook tends to produce inaccuracies in the reconstruction of image textures and color information.
}\label{fig:sem_rec}
\end{minipage}
\hspace{3mm}
\begin{minipage}[b]{.42\linewidth}
\hspace{-3mm}
\includegraphics[width=1.\linewidth]{figs/fig-sem_p.png}
\caption{Visualization of semantic code.
Each code corresponds to a set of image patches that share similar pixel-level features.}\label{fig:fig4}
\end{minipage}
\end{figure}
\subsection{Pixel Reconstruction Enablement}
\label{subsec:pixsubc}
\textbf{Discussion.}
In section \ref{subsec:svq}, we demonstrate that semantic code lacks the ability to model pixel information.
In order to pixel reconstruction enablement and avoid a reduction of understand ability, a straightforward approach is to add an extra VQGAN~\citep{vqgan} model. %on top of the SPC model.
Semantic codebook extracts discrete semantic tokens for multimodal understanding, and VQGAN extracts discrete texture tokens for generation.
The two token sets are concatenated—either dimensionally or sequentially, and passed to the LLM.
However, the resulting token inflation or oversized codebook introduces a significant computational burden, limiting its feasibility for MLLMs.
Furthermore, we present the visualization results of the semantic code, as shown in Fig.\ref{fig:fig4}.
It can be observed that image patches corresponding to the same code exhibit similar pixel features.
For example, the code $\mathit{v}_{14312}$ is more likely to be assigned to the rooster comb element in the image.
At the same time, the image patches corresponding to these combs exhibit similar pixel features, such as color, patterns, and shapes.
% \textcolor{red}{BALABALA.}
Based on this observation, we propose Semantic-Guided Hierarchical Codebook to model the pixel feature space corresponding to each semantic code using a sub-codebook.
% This led us to contemplate: whether the texture feature space associated with a single semantic code could be effectively represented through several discrete codes.
\textbf{Semantic-Guided Hierarchical Codebook (SGHC).}
The SGHC consists of a pretrained semantic codebook and several sub-codebooks, where each sub-codebook corresponds to a semantic code of the semantic codebook, as shown in \figref{fig:overview} (a).
% We introduce a novel codebook design, Semantic-Guided Hierarchical Codebook(SGHC), which builds pixel sub-codebooks on SPC to model texture feature space of each semantic code, as shown in Fig.\ref{fig:fig2}.
Specially, given the pre-trained semantic codebook $\mathcal{C}_{sem}=\{c_1, c_2,...,c_K\}\in \mathbb{R}^{K\times d_{sem}}$,
the pixel codebook $\mathcal{C}_{pix}=\{\mathcal{C}_{pix}^{1}, \mathcal{C}_{pix}^{2},\dots, \mathcal{C}_{pix}^{K}\}\in \mathbb{R}^{K\times m\times d_{pix}}$, where $\mathcal{C}_{pix}^{k}\in\mathbb{R}^{m\times d_{pix}}$ is $k_{th}$ semantic code's pixel sub-codebook, $m$ is sub-codebook size.
At first, the semantic codebook quantizes $X$ to a discrete semantic token $Z_{sem}$ and a semantic codebook index $I_{sem}$.
In parallel, pixel encoder $\mathcal{E}_{pix}$ extract continuous pixel features $Z_{pix}=\mathcal{E}_{pix}(X)$.
For the quantization process of the pixel codebook, the corresponding pixel sub-codebook is selected based on the quantization result of the semantic codebook.
For instance, given image patch $i$, its semantic quantization codebook index $k$ and continuous pixel feature $Z_{pix}^{i}$, SGHC selects pixel sub-codebook $\mathcal{C}_{pix}^{k}$ to quantize $Z_{pix}^{i}$. The process is as follows:
\begin{equation}
Z_{q_{pix}}^i, I_{q_{pix}}^i = \mathop{\arg\min}\limits_{j\in \{1,\dots,m\}}{\|Z_{pix}^{i}-\mathcal{C}_{pix}^k[j] \|}
\end{equation}
where $j$ is the selected sub-codebook internal index.
Finally, the semantic quantized tokens and pixel quantized tokens are concatenated to $Z_q=concate_{dim}(Z_{q_{sem}}, Z_{q_{pix}})$ as the input of pixel decoder $\mathcal{D}_{pix}$ to reconstruct the raw pixel image:
\begin{equation}
\hat{X} = \mathcal{D}_{pix}(Z_q)
\end{equation}
where $\hat{X}$ is reconstructed pixel image.
The $\mathcal{E}_{pix}$, $C_{pix}$ and $\mathcal{D}_{pix}$ are end-to-end trainable by minimizing reconstruction loss $L_{img}=\ell_{1}(\hat{X},X)$, codebook loss $Lc$, perceptual loss $L_{per}$ and represents adversarial loss $L_{gan}$~\citep{vqgan}.
The reconstruction loss is formulated as:
\begin{equation}
L_{rec} = L_{img} + \lambda_1Lc + \lambda_2 L_{per} + \lambda_3 L_{gan}
\end{equation}
where $\lambda_1$, $\lambda_2$ and $\lambda_3$ are loss weight of each item.
Our SGHC can be regarded as the refinement of a semantic discrete space to enable pixel reconstruction.
We place a specific emphasis on two key advantages of SGHC:
% 训练解耦以及几乎无损的像素重建赋能
(1) \textbf{\textit{Non-Conflicting Extension:}}
Our method leverages a pre-trained semantic codebook as a foundation, with pixel reconstruction losses exclusively employed to optimize pixel branch modules during the PRE.
This strategic approach effectively circumvents the suboptimal solutions that arise from joint optimization processes.
Furthermore, SGHC's final output is generated by concatenating semantic-quantized features with pixel-quantized features, preserving the full expressive capacity of the original semantic features while integrating complementary texture information through this unified feature fusion paradigm.
Subsequent tasks, such as reconstruction, multimodal understanding, and generation, all share the same discrete token representation;
% 可以接受的训练开销
(2) \textbf{\textit{Efficient Downstream Applications:}}
SGHC effectively avoids two critical predicaments: token quantity inflation and codebook overexpansion.
As defined before, the semantic codebook size is $K$, and each pixel sub-codebook size is $m$.
Due to dimensional concatenation, the complete codebook flattens to $K\times m$, where $m$ is much smaller than $K$.
In our experimental default settings, we extend the complete codebook to a size comparable to existing LLMs' text vocabulary size, e.g., the size of Qwen2 vocabulary is $150$k.
% \czs{In addition, unlike VILA-U~\citep{vilau}, our approach supports unified text and visual head, allowing cross-modal modeling. Leveraging these strengths, our framework demonstrates robust performance across multimodal downstream tasks, validating the effectiveness of the proposed architectural design.}
\begin{table*}[t]
% \vspace{-2mm}
\begin{minipage}{0.50\textwidth}
\definecolor{lightblue}{RGB}{240,248,255}
\newcommand{\colorrow}[1]{\rowcolor{lightblue} #1}
\caption{Comparison of reconstruction quality on the ImageNet-50k validation set.
\textsuperscript{\ddag}: quantizer uses residual quantization (RQ), where the total Code Shape is multiplied by RQ depth.
\textsuperscript{\dag}: quantizer uses multiple codebooks and product quantization.
}
\label{tab:rec}
\resizebox{\linewidth}{!}{
\begin{tabular}{l cccc}
\toprule
Method & Res & \makecell[c]{Code \\ Shape} & \makecell[c]{Codebook \\ Size} & rFID\( \downarrow \) \\
\midrule
\multicolumn{5}{l}{\textit{Only Reconstruction}} \\ \midrule
LlamaGen~\citep{llamagen} & 256 & $16\times 16$ & 16,384 & 2.19 \\
RQVAE~\citep{rqvae} & 256 & $16\times 16\times 4$ \textsuperscript{\ddag} & 16,384 & 3.20 \\
% RQVAE~\citep{rqvae} & 256 & 1024\textsuperscript{\ddag} & 16,384 & 1.30 \\
% VQGAN-LC~\citep{vqganlc} & 256 & 256 & 16,384 & 3.01 \\
VQGAN-LC~\citep{vqganlc} & 256 & $16\times 16$ & 100,000 & 2.62 \\
IBQ~\citep{ibq} & 256 & $16\times 16$ & 16,384 & 1.37 \\
IBQ~\citep{ibq} & 256 & $16\times 16$ & 262,144 & 1.00 \\
FQGAN~\citep{fqgan} & 256 & $16\times 16\times 2$ & $16,384\times 2$\textsuperscript{\dag} & 0.94 \\
% FQGAN~\citep{fqgan} & 256 & 256 & $16384\times 3$\textsuperscript{\dag} & 0.76 \\
\midrule
\multicolumn{5}{l}{\textit{Unified}} \\ \midrule
VILA-U~\citep{vilau} & 256 & 16x16x4\textsuperscript{\ddag} & 16,384 & 1.80 \\
SDE(MUSE-VL)~\citep{musevl} & 256 & $16\times 16$ & 32,768 & 2.26 \\
TokenFlow~\citep{tokenflow} & 256 & 680 & 32,768 & 1.37 \\
TokLIP~\citep{toklip} & 256 & $16\times 16$ & 16,384 & 2.19 \\
QLIP-B~\citep{toklip} & 256 & $16\times 16$ & $2^{28}$ & 3.21 \\
UniTok~\citep{unitok} & 256 & $16\times 16$ & $16,384\times 4$\textsuperscript{\dag} & 0.39 \\
\colorrow{\textbf{SemHiTok(ours)} & 256 & $16\times 16$ & 196,608 & 1.16 \\ }
\colorrow{\textbf{SemHiTok(ours)} & 384 & $27\times 27$ & 196,608 & 0.66 \\ }
\bottomrule
\end{tabular}
}
\end{minipage}
\hspace{4mm}
\begin{minipage}{0.45\textwidth}
% \begin{center}
\hspace{-4mm}
\includegraphics[width=\linewidth]{figs/fig-abla-rec.pdf}
\captionof{figure}{Visualized reconstruction results from the ablation of key modules.
PRE brings about a significant improvement in reconstruction quality.
Moreover, the Enhance Decoder(ED) further improves reconstruction on hard samples.
}
\label{fig:abl-rec}
% \end{center}
\end{minipage}
\end{table*}
\subsection{Unified MLLM Equipped with SemHiTok}
\label{subc:mmdt}
The framework diagram for unified MLLM is shown in Fig.\ref{fig:overview}(c). We use SemHiTok to develop a unified multimodal model, which models discrete vision and text token sequences with a universal next-token prediction loss.
Particularly, in image processing, SemHiTok is utilized to discretize images into token sequences.
On the model side, we merely expand the text vocabulary and adjust the head layer to incorporate visual token IDs.
To enable a unified head layer, we flatten SGHC by merging all sub-codebooks into a single flat representation as shown in Fig.\ref{fig:overview}(b).
Specifically, for the $j_{th}$ semantic code in the $i_{th}$ pixel sub-codebook, the discrete code index in the completed codebook is $h=i\times m + j$, where $m$ is the sub-codebook size.
It is also worth noting that the vocabulary expansion is merely for implementation convenience. We still use the features extracted from SGHC as input and align with LLM through a lightweight adapter layer.
After training is completed, we replace the visual component in the embedding layer in order to achieve consistency between training and inference.
To enable LLM to better handle features at two different levels, we introduce a Dual-MLP adapter layer, which projects semantic features and pixel features separately, and then concatenates them along the dimension before feeding them into the LLM.
To enable classifier-free guidance~\citep{cfg}, we randomly replace the text condition with a probability of 0.1 to the unconditioned text during training.
More deployment details are provided in the \suppltext{\ref{A:unimllm}}.
\section{Experiments}\label{sec:exp}
\subsection{Experimental Setup}\label{subc:exp_set}
\textbf{Tokenizer.}
For the semantic branch, we employ SigLIP~\citep{siglip} as the semantic encoder and three self-attention layers as the semantic decoder to reconstruct semantic features.
% Following the setting of LLaVA-v1.5, we distill the features from the penultimate layer of SigLIP.
% In addition, our semantic codebook size is 16384, which updated using EMA~\citep{ema} algorithm to enhance stability.
For the pixel branch, we employ ViT as both the pixel encoder and decoder, assigning 8 pixel sub-codes to each semantic code.
% The pixel sub-codebooks use normalized VQ structure which enhances both usage and stability.
More tokenizer detail, please refer to \suppltext{\ref{A:exp_tok}}.
\textbf{Unified MLLM.}
We use Qwen2.5-7B-Instruct~\citep{qwen2} as the base LLM, and expand its vocabulary and output head layer.
We evaluate visual understanding on standard VQA benchmarks including SEEDB~\citep{seedbench}, POPE~\citep{pope}, GQA~\citep{gqa}, MMMU~\citep{mmmu}, MMB~\citep{mmbench} and MME~\citep{mme}.
For visual generation evaluation, we report results on MJHQ-30K~\citep{li2024playground}, GenAI-Bench~\citep{genai}, GenEval~\citep{geneval} and DPG~\citep{dpg}.
More unified MLLM detail, please refer to \suppltext{\ref{A:exp_uni}}.
% Follow existing work~\citep{vilau, unitok}, we first pretrain the model and adapter layer on a mix of multimodal data, which is composed of 3.5M language data from Magpie~\citep{magpie} and Openorca~\citep{openorca}, 10M caption image-text pairs data and 15M MidJourney-style synthetic data.
% Subsequently, we finetune the model on 1M language dataset from Magpie~\citep{magpie} and Evol-Instruct~\citep{evol-ins}, 4M in-house generation data and 4M understanding data from emova~\citep{emova} and LLaVA-SFT~\citep{llava}.
% The full training takes about 10 days on 32 A800 GPUs.
% During image generation inference, we apply classifier-free guidance~\citep{cfg} with a scale factor of $2.5$.
\subsection{Unified Image Tokenizer}
\begin{table}[t]
\definecolor{lightblue}{RGB}{240,248,255}
\newcommand{\colorrow}[1]{\rowcolor{lightblue} #1}
\centering
\caption{Comparative analysis of tokenizers on multimodal comprehension tasks.
*: Both the tokenizer and LLM are reproduced in our setting.
Our method achieves SOTA performance compared with other discrete tokenizers.
}\label{tab:llava}
\resizebox{0.98\textwidth}{!}{
\begin{tabular}{llll | cccc}
\toprule
\textbf{Model} & \textbf{LLM} & \textbf{Data} & \textbf{Res.} & \textbf{POPE} & \textbf{MME-P} & \textbf{SEED} & \textbf{GQA} \\ \midrule
SigLIP~\citep{siglip} & Vicuna-7B & LLaVA-v1.5 & 256 & 83.76 & 1481.0 & 65.28 & 61.9 \\ \midrule
LlamaGen~\citep{llamagen} & Vicuna-7B & LLaVA-v1.5 & 256 & 65.6 & 716.8 & 35.0 & 39.8 \\
VILA-U~\citep{vilau} & Vicuna-7B & LLaVA-v1.5 & 256 & 81.6 & 1311.6 & 56.9 & 55.3 \\
SDE(MUSE-VL)\textsuperscript{*}~\citep{musevl} & Vicuna-7B & LLaVA-v1.5 & 256 & 77.3 & 1240.0 & 56.7 & 58.0 \\
% UniTok~\citep{unitok} & Vicuna-7B & LLaVA-v1.5 & 256 & 81.7 & 1303.7 & 56.6 & \textbf{61.1} \\
TokLIP~\citep{toklip} & Qwen2.5-7B-Ins & LLaVA-v1.5 & 256 & 81.2 & 1346.8 & 59.8 & 57.4 \\
\colorrow{
\textbf{SemHiTok(Ours)} & Vicuna-7B & LLaVA-v1.5 & 256 & \textbf{82.5} & \textbf{1355.8} & \textbf{62.9} & \textbf{60.3}
\\ } \midrule
TokenFlow-384~\citep{tokenflow} & Vicuna-7B & LLaVA-v1.5 & 384 & 84.9 & 1416.4 & 62.7 & 61.2 \\
% DualToken~\citep{dualtoken} & LLaMA-2-7B & LLaVA-v1.5 & 384 & 86.1 & 1589.6 \\
TokLIP~\citep{toklip} & Qwen2.5-7B-Ins & LLaVA-v1.5 & 384 & 82.7 & 1410.2 & \textbf{65.2} & 59.3 \\
\colorrow{
\textbf{SemHiTok-384(Ours)} & Vicuna-7B & LLaVA-v1.5 & 384 & \textbf{86.3} & \textbf{1465.6} & 64.1 & \textbf{62.3}
\\ } \bottomrule
\end{tabular}
}
\end{table}
\begin{table*}[h]
\centering
\definecolor{lightblue}{RGB}{240,248,255}
\newcommand{\colorrow}[1]{\rowcolor{lightblue} #1}
\caption{
Quantitative results on multimodal understanding benchmarks.
SemHiTok achieves SOTA performance on most benchmarks among \textit{Und\&Gen Discrete} MLLMs, and is comparable to or even surpasses some \textit{Only Und} and \textit{Und\&Gen. Continuous} models.
The performance on \textit{Und\&Gen Discrete} with top-1 and top-2 values is denoted in bold and underline, respectively.
}
\label{tab:und_results}
\resizebox{0.95\linewidth}{!}{
\begin{tabular}{l cc | cccccccc}
\toprule
\textbf{Method} & \textbf{\# Params} & \textbf{Res.} & \textbf{SEED} & \textbf{POPE} & \textbf{GQA} & \textbf{MMMU} & \textbf{MMB} & \textbf{MME} & \textbf{MME-P} & \textbf{MMV} \\
\midrule
\multicolumn{11}{l}{\textit{Only Und.}} \\ \midrule
LLaVA-Phi~\citep{llavaphi} & 2.7B & 256 & - & 85.0 & - & - & 59.8 & - & 1335.1 & 28.9 \\
LLaVA-v1.5~\citep{llava} & 7B & 336 & 58.6 & 85.9 & 62.0 & 35.4 & 64.3 & - & 1510.7 & 31.1 \\
Qwen-Vl-Chat~\citep{qwenvl-chat} & 7B & 448 & 57.7 & - & 57.5 & 30.5 & - & 1848.3 & 1487.5 & - \\
ShareGPT4V~\citep{sharegpt4v} & 7B & 336 & 69.7 & - & 63.3 & 37.2 & 68.8 & 1943.8 & 1567.4 & 37.6 \\
\midrule
\multicolumn{11}{l}{\textit{Und\&Gen. Continuous}} \\ \midrule
% Unified-IO 2~\citep{unifiedio2} & 6.8B & 384 & - & 87.7 & - & - & - & - & - & - \\
% DreamLLM~\citep{dong2023dreamllm} & 7B & 224 & - & - & - & - & - & - & - & 26.6 \\
LaVIT~\citep{llavit} & 7B & 224 & - & - & 46.8 & - & 58.0 & - & - & - \\
Janus~\citep{janus} & 1.5B & 384 & 63.7 & 87.0 & 59.1 & 30.5 & 69.4 & - & 1338.0 & 34.3 \\
Janus-Pro-1B~\citep{jaunspro} & 1.5B & 384 & 68.3 & 86.2 & 58.9 & 38.9 & 65.5 & - & 1444.0 & - \\
MAR~\citep{mar} & 1.5B & 512 & 67.1 & 87.6 & 58.9 & 38.9 & 65.5 & - & 1155.0 & - \\
\midrule
\multicolumn{11}{l}{\textit{Und\&Gen. Discrete}} \\ \midrule
LWM~\citep{lwm} & 7B & 256 & - & 75.2 & 44.8 & - & - & - & - & 9.6 \\
SEED-LLaMA~\citep{seedllama} & 13B & 256 & 53.7 & - & - & - & - & - & - & - \\
Show-o~\citep{xie2024show} & 1.5B & 256 & - & 80.0 & - & 26.7 & - & - & 1097.2 & - \\
Liquid~\citep{liquid} & 7B & 512 & - & 81.1 & \textbf{71.3} & - & - & - & 1119.3 & - \\
EMU3~\citep{wang2024emu3} & 8B & 512 & {68.2} & {85.2} & {60.3} & 31.6 & 58.5 & 1509.9 & 1243.8 & \textbf{37.2} \\
VILA-U~\citep{vilau} & 7B & 256 & 56.3 & 83.9 & 58.3 & - & - & - & 1336.2 & 27.7 \\
VILA-U~\citep{vilau} & 7B & 384 & 59.0 & \textbf{85.8} & 60.8 & - & - & - & 1401.8 & 33.5 \\
% UniTok~\citep{unitok} & 7B & 256 & - & 83.2 & 61.1 & - & - & - & 1448.0 & 33.9 \\
UniToken~\citep{unitoken} & 7B & 384 & 69.3 & - & - & 32.8 & 69.9 & - & - & - \\
TokLIP~\citep{toklip} & 7B & 384 & 76.9 & 84.1 & 59.5 & \textbf{43.1} & 67.6 & - & \underline{1488.4} & 29.8 \\
TokenFlow-B~\citep{tokenflow} & 13B & 224 & 60.4 & 84.0 & 59.3 & 34.2 & 55.3 & {1660.4} & 1353.6 & 22.4 \\
TokenFlow-L~\citep{tokenflow} & 13B & 256 & 62.6 & {85.0} & {60.3} & {34.4} & {60.3} & 1622.9 & {1365.4} & 27.7 \\
SynerGen-VL~\citep{synergen} & 2.4B & 512 & 62.0 & 85.3 & 59.7 & 34.2 & 53.7 & 1837.0 & 1381.0 & 34.5 \\
\colorrow{\textbf{SemHiTok(Ours)} & 7B & 256 & \underline{69.7} & 83.4 & {60.3} & {39.3} & \underline{72.3} & \underline{1775.9} & {1449.0} & {30.5} \\ }
\colorrow{\textbf{SemHiTok(Ours)} & 7B & 384 & \textbf{79.8} & \underline{85.5} & \underline{61.7} & \underline{41.0} & \textbf{75.2} & \textbf{1993.8} & \textbf{1512.8} & \underline{36.6} } \\
\bottomrule
\end{tabular}
}
% \end{table*}
\vspace{5mm}
% \begin{table}[h]
\definecolor{lightblue}{RGB}{240,248,255}
% \newcommand{\colorrow}[1]{\rowcolor{lightblue} #1}
\caption{Comparison of generation quality on GenAI and MJHQ30K.
SemHiTok achieves comparable results with specialist models and unified MLLMs.
}
\label{tab:gen_results}
\centering
\resizebox{0.95\textwidth}{!}{
\begin{tabular}{l cccc | ccc}
\toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Params} & \multirow{2}{*}{Type} & \#Training & \multirow{2}{*}{Res.} & \multicolumn{2}{c}{GenAI-Bench} & MJHQ30K \\
& & & Images & & Basic $\uparrow$ & Advanced $\uparrow$ & gFID $\downarrow$ \\
\midrule
\multicolumn{7}{l}{\textit{Only Gen.}} \\ \midrule
SD v2.1~\citep{sdv21} & -- & Diff & 2000M & 1024 & 0.78 & 0.62 & -- \\
DALL-E 3~\citep{dalle3} & -- & Diff & - & 1024 & 0.90 & 0.70 & -- \\
PixArt-$\alpha$~\citep{pixart} & 0.6B & Diff & - & 1024 & -- & -- & 6.14 \\
SDXL~\citep{podell2023sdxl} & 2.6B & Diff & 2000M & 1024 & 0.83 & 0.63 & 9.55 \\
Playgroundv2.5~\citep{li2024playground} & -- & Diff & - & 1024 & -- & -- & 4.48 \\
\midrule
\multicolumn{7}{l}{\textit{Und\&Gen.}} \\ \midrule
LWM~\citep{lwm} & 7B & AR & - & 256 & 0.63 & 0.53 & 17.77 \\
Show-o~\citep{xie2024show} & 1.5B & \makecell{Diff} & 36M & 256 & 0.70 & 0.60 & 15.18 \\
Janus~\citep{janus} & 1.3B & AR & 65M & 384 & -- & -- & 10.10 \\
VILA-U~\citep{vilau} & 7B & AR & 15M & 256 & 0.76 & 0.64 & 12.81 \\
VILA-U~\citep{vilau} & 7B & AR & 15M & 384 & 0.73 & 0.61 & 7.69 \\
SynerGen-VL~\citep{synergen} & 2.4B & AR & 25M & 256 & - & - & 6.10 \\
SDE(MUSE-VL)~\citep{musevl} & 7B & AR & 10M & 256 & - & - & 7.73 \\
ILLUME~\citep{wang2024illume} & 7B & AR & 15M & 512 & 0.75 & 0.60 & 7.76 \\
ILLUME+~\citep{illume+} & 7B & AR & 15M & 512 & 0.75 & 0.60 & 7.76 \\
TokenFlow~\citep{tokenflow} & 7B & AR & 60M & 256 & 0.65 & {0.65} & 7.78 \\
Liquid~\citep{liquid} & 7B & AR & 30M & 512 & \textbf{0.83} & {0.65} & 5.47 \\
UniTok ~\citep{unitok} & 7B & AR & 30M & 256 & 0.85 & 0.67 & 7.46 \\
\colorrow{
\textbf{SemHiTok(ours)} & 7B & AR & 15M & 256 & \textbf{0.83} & {0.64} & \textbf{5.40}
\\ }
\colorrow{
\textbf{SemHiTok(ours)} & 7B & AR & 15M & 384 & \textbf{0.83} & \textbf{0.66} & {5.70}
\\ }
\bottomrule
\end{tabular}
}
\end{table*}
%%%% 一个单栏的三线表
\textbf{Image Reconstruction.}
% We present the reconstruction performance on the ImageNet-50k validation set in Tab.\ref{tab:rec}.
% Notably, SemHiTok excels in reconstruction quality compared to the unified tokenizer, recording an impressive 1.16 rFID with $16\times$ downsampling ratio.
% % Furthermore, our tokenizer is significantly outperforming previous methods such as LlamaGen and RQVAE.
% For specialized models that also employ a large codebook, such as IBQ and VQGAN-LC, our method remains competitive.
% Increasing the training 384 resolution led to a significant improvement in the rFID score, reaching 0.66.
% The results validate the effectiveness of SGHC design in modeling pixel feature space of the semantic code.
We present the reconstruction performance on the ImageNet-50k validation set in Tab.\ref{tab:rec}.
Notably, SemHiTok excels in reconstruction quality compared to the unified tokenizer, recording an impressive 1.16 rFID with $16\times$ downsampling ratio.
While SemHiTok's vocabulary size(approx 196k) appears larger than baselines, it is crucial to consider the effective representational capacity defined by the code shape.
VILA-U(using RQ) and FQGAN(using Product Quantization) operate within a combinatorial search space($N^D$ or $N_1\times N_2$), resulting in a significantly larger effective capacity than SemHiTok’s linearly constrained structure($K\times m$).
Furthermore, these approaches typically employ a denser code shape, whereas SemHiTok relies on a single unified hierarchical index. This indicates that our superior performance derives from the structured efficiency of SGHC rather than brute-force capacity expansion.
Increasing the training 384 resolution led to a significant improvement in the rFID score, reaching 0.66.
The results validate the effectiveness of SGHC design in modeling pixel feature space of the semantic code.
\textbf{LLaVa-v1.5 Multimodal Understanding.}
To ensure a fair comparison, we conduct experiments to evaluate the multimodal understanding performance of existing open-source tokenizers and SemHiTok under the standard LLaVA-v1.5 setting.
The results are as shown in Tab.\ref{tab:llava}.
% Due to the lack of pre-alignment with text, LlamaGen performs the worst performance.
% In comparison, unified images tokenizers exhibit substantial improvements in understanding performance.
% Due to hybrid structure or joint training strategies, there remains a significant gap between previous tokenizer and continuous representations in terms of understanding.
% However, SemHiTok achieves the SOTA of the discrete tokenizer, which is closest to the SigLIP of the continuous input.
LlamaGen's performance is notably inferior, a consequence of its deficient pre-alignment with text.
In contrast, unified image tokenizers show considerable advancements in understanding.
However, due to their inherent hybrid structures or joint training strategies, a substantial disparity in understanding performance remains between prior discrete tokenizers and continuous representations.
Notably, even though TokLIP uses a stronger base model (Qwen2.5-7B-Ins), its performance remains inferior to ours.
SemHiTok achieves state-of-the-art results for discrete tokenizers, nearing the performance of continuous inputs such as SigLIP.
% This is attributed to its novel SGHC structure, which enables a two-stage training strategy and effective two-level feature fusion.
% In addition, since TokenFlows ~\citep{tokenflow} open-source 256-resolution tokenizer is based on CLIP-Base, it differs from others. To ensure a fairer comparison, we develop SemHiTok-384, utilizing the same SigLIP-SO400M-Patch14-384 semantic tokenizer as TokenFlow-384.
% Our experiments show that SemHiTok-384 outperforms TokenFlow-384 in multi-modal understanding tasks.
% The above two groups of experiments show that SemHiTok can extract unified image features, which includes low-level pixel features and high-level semantic features, and has the potential to be applied to unified MLLM.
\subsection{Unified MLLM}
\label{subc:mmld}
\textbf{Multimodal Understanding.}
We evaluate the understanding performance of SemHiTok on diverse benchmarks in Table \ref{tab:und_results}.
\textit{Und} means support multimodal understanding, \textit{Gen} means support text-to-image generation.
Among \textit{Und\&Gen. Discrete}, SemHiTok achieves state-of-the-art performance on most metrics, such as SEED, MMB, MME, and MME-P.
Compared to other unified tokenizers (e.g., VILA-U, TokenFlow, UniToken), our models demonstrate significant advantages.
Notably, our model surpasses expert-level models on key benchmarks, achieving 3.8 and 6.4 points higher than ShareGPT4V on MMMU and MMB, respectively.
This bridges the gap between discrete visual tokens and continuous visual tokens in multimodal understanding tasks, strongly demonstrating the superiority and potential of our approach.
Visualizations on understanding tasks are available in \suppltext{\ref{A:vis_under}}.
% \label{subc:t2i-gen}
% \begin{figure*}[b]
% \centering
% % \fbox{\rule{0pt}{2in} \rule{0.9\textwidth}{0pt}}
% \includegraphics[width=0.75\textwidth]{figs/fig-gen.pdf}
% \caption{Presentation of the generated results.}
% \label{fig:gen}
% \end{figure*}
\textbf{Text-to-Image Generation}
To evaluate the text-to-image generation, we use GenAI-bench~\citep{genai} and MJHQ30K~\citep{li2024playground} benchmark, and the results are shown in Tab\ref{tab:gen_results}.
For GenAI-bench, we use clip-flant5-xxl as the VQA score model to reflect the consistency
between text descriptions and generated images.
% On this challenging benchmark, our model outperforms other unified MLLMs, and even surpasses the diffusion-based model SDXL.
On this challenging benchmark, our model achieves competitive performance, closely matching Liquid, even though Liquid employs a generation-focused tokenizer.
Furthermore, our model even outperformed some diffusion-based expert models, such as SDXL and SD v2.
The strong results underscore the superior capability of our unified MLLM in complex text-to-image generation tasks.
For MJHQ30K, we use the generation FID metric on generated images and high-quality images.
On this benchmark, SemHiTok-256 attains 5.40 gFID, setting a new state-of-the-art in autoregressive image generation.
In addition, we conduct further quantitative comparisons on GenEval and DPG, which are available in \suppltext{\ref{A:more_gen_tab}}.
% In addition, the generated images by our model is illustrated in \figref{fig:gen}, using prompts from the MJHQ30K dataset.
% Visualization results indicate that our model is capable of generating realistic and aesthetically pleasing images, even in the challenging prompts of human face synthesis.
% In addition, the model demonstrates an understanding of high-level visual concepts, including artistic style and color schemes.
% For example, we can use prompts to generate a flower maiden in an oil painting style or a realistic-looking tree.
More quantitative analysis results and visualizations on generation tasks are available in \suppltext{\ref{A:vis_gen}}.
\section{Ablation}
\subsection{Impact of Key Design.}
\begin{table}[t]
\centering
\caption{Impact of key design choices on reconstruction and multimodal understanding.
}
\label{tab:abl_kd}
% \definecolor{lightblue}{RGB}{240,248,255}
\newcommand{\colorrow}[1]{\rowcolor{gray!15} #1}
\resizebox{0.9\linewidth}{!}{
\begin{tabular}{cccc cccc | c}
\toprule
\makecell{Semantic \\ Codebook} & SGHC & \makecell{Dual\\MLP} & \makecell{Enhance \\ Decoder} & MME-P$\uparrow$ & MMB$\uparrow$ & SEED$\uparrow$ & MMU$\uparrow$ & rFID$\downarrow$ \\
\midrule
\checkmark & ~ & ~ & & 1387.5 & 61.3 & 62.3 & 35.6 & 3.17 \\
\checkmark & \checkmark & ~ & & 1355.8 & 60.7 & 62.9 & 35.8 & 1.42 \\
\checkmark & \checkmark & \checkmark & & 1393.0 & 61.6 & 63.2 & 36.1 & 1.42 \\
\graycolorrow{
\checkmark & \checkmark & \checkmark & \checkmark & 1393.0 & 61.6 & 63.2 & 36.1 & 1.16
\\ }
\bottomrule
\end{tabular}
}
\end{table}
% We validate the impact of our key design in SemHiTok: semantic-guided hierarchical codebook.
In Tab.\ref{tab:abl_kd}, we validate the impact of our key design choices in SemHiTok: semantic codebook, semantic-guided hierarchical codebook(SGHC), Dual MLP, and Enhance Deocoder.
For efficiency, we only test the understanding performance under LLaVA-v1.5, and train 40 epochs on ImageNet-1K for reconstruction tasks.
% We begin our analysis with the semantic codebook, which proves sufficient for multimodal understanding tasks. Nevertheless, the lack of pixel information leads to poor reconstruction quality.
% After incorporating SGHC, our model is empowered with pixel reconstruction enablemen, resulting in a 1.75 reduction in rFID.
% Importantly, the multimodal understanding performance remains largely unaffected.
% With the further introduction of Dual-MLP, the performance on multimodal understanding tasks improves, even surpassing that of the pure semantic codebook. This indicates that the Dual-MLP's strategy for handling multi-level features is effective.
% Finally, leveraging a more advanced pixel decoder leads to further improvements in reconstruction quality.
We begin with the semantic codebook, which suffices for multimodal understanding but suffers from poor reconstruction. Incorporating SGHC enables pixel reconstruction and reduces rFID by 1.75, without noticeably affecting understanding performance.
Introducing Dual-MLP further enhances multimodal understanding, even surpassing the semantic codebook alone, highlighting the effectiveness of multi-level feature modeling. Finally, a stronger pixel decoder brings additional improvements in reconstruction quality.
\figref{fig:abl-rec} demonstrates the effects of various modules on reconstruction results. Compared to the semantic codebook, SGHC delivers more detailed reconstructions. Additionally, the enhanced decoder boosts performance on difficult samples.
More reconstruction comparisons are available in \suppltext{\ref{A:vis_rec}}.
\subsection{Impact of training strategy and structure.}
We conduct comparative experiments under the same hyperparameters to investigate the impact of the training strategy and structure, as shown in Table \ref{tab:abl-l}.
It can be observed that \textbf{Exp 1}'s performance is the lowest across both multimodal understanding and reconstruction tasks.
After decoupling the architecture, \textbf{Exp 2} achieves superior performance compared to \textbf{Exp 1}, yet its codebook usage remains at a low level.
Furthermore, \textbf{Exp 4} achieves marked gains in both efficacy and utilization over \textbf{Exp 2}, which underscores the importance of a phased training strategy for the proposed SemHiTok hierarchical architecture.
Compared with \textbf{Exp 3}, \textbf{Exp 4} achieves superior performance on both multimodal understanding and reconstruction tasks.
This suggests that naively incorporating pixel features may negatively affect the alignment between image features and the LLM.
In contrast, \textbf{Exp 4} leverages pixel features as a complementary refinement to semantic features, thereby effectively bridging the gap between semantic and pixel representations.
% In \textbf{Exp 1}, we used the same architecture as SDE and adopted a joint training strategy.
% Its performance is observed to be the lowest across both multimodal understanding and reconstruction tasks.
% In \textbf{Exp 2}, using the same architecture as SemHiTok but with joint training, the performance showed an overall improvement compared to \textbf{Exp 1}.
% Furthermore, to examine the influence of architectural design, we conduct \textbf{Exp 3}, where two separately pretrained tokenizers are employed to independently extract semantic and pixel information. Specifically, we use the semantic branch of SemHiTok and LlamaGen.
% % The results are presented in Table \ref{tab:abl-l} (\textbf{Exp 3}).
% Compared with \textbf{Exp 3}, SemHiTok achieves superior performance on both multimodal understanding and reconstruction tasks.
% The above experiments demonstrate the advantages of structural decoupling and training decoupling in jointly modeling semantic and pixel-level information.
% \textbf{Exp 3} and \textbf{Exp 4} suggest that naively incorporating pixel features may negatively affect the alignment between image features and the LLM.
% This suggests that naively incorporating pixel features may negatively affect the alignment between image features and the LLM.
% In contrast, SemHiTok leverages pixel features as a complementary refinement to semantic features, thereby effectively bridging the gap between semantic and pixel representations.
% In contrast, SemHiTok leverages pixel features as a complementary refinement to semantic features, thereby effectively bridging the gap between semantic and pixel representations.
% As shown in Table \ref{tab:abl-l} (\textbf{Exp 1}), compared with the default experimental setting (\textbf{Exp }), Joint Training exhibits lower performance on both multimodal understanding tasks and reconstruction.
% It is worth noting that the codebook usage is also significantly lower than that of SemHiTok, indicating that the joint training strategy drives the tokenizer toward a suboptimal trade-off between semantic and pixel feature extraction.
% Furthermore, to examine the influence of architectural design, we conduct Dual-Tokenize (\textbf{Exp 2}), where two separately pretrained tokenizers are employed to independently extract semantic and pixel information. Specifically, we use the semantic branch of SemHiTok and LlamaGen. The results are presented in Table \ref{tab:abl-l} (\textbf{Exp 2}). Compared with Dual-Tokenize, SemHiTok achieves superior performance on both multimodal understanding and reconstruction tasks.
% In addition, we observed that during the pretraining phase of LLaVA-v1.5, the training loss for SemHiTok converged to 0.79, whereas for Dual-Tokenizer it converged to 0.82.
% This suggests that naively incorporating pixel features may negatively affect the alignment between image features and the LLM.
% In contrast, SemHiTok leverages pixel features as a complementary refinement to semantic features, thereby effectively bridging the gap between semantic and pixel representations.
\subsection{More comparisons and ablation studies}
In \suppltext{\ref{A:qa}}, we attempt to provide quantitative evidence to further demonstrate pixel similarity within the semantic code.
Meanwhile, to further illustrate how different hyperparameters in our method affect model performance, we also conduct ablation studies on Concat Type and Sub-Codebook Size in \suppltext{\ref{A:abl_1}}, Semantic VQ Type, Codebook dim and \rev{Codebook size in \suppltext{\ref{A:semantic}}, and different $K\times m$ and more comparison with other large codebook reconstruction methods in \suppltext{\ref{A:pixel-km}}. }
% Please refer to \textbf{\textcolor{blue}{Supplementary Materia \ref{A:abl_1} and \ref{A:abl_2}}.}
% \begin{table}[tb]
% \centering
% \caption{Impact of training strategy and structure.
% % Exp1: SemHiTok simultaneously trains both the semantic branch and the pixel branch within its architecture.
% % Exp2: Two encoders are employed, but only a single codebook is used.
% % Exp3: Separate tokenizers are adopted, consisting of a semantic tokenizer (identical to that in SemHiTok) and a pixel tokenizer (LLaMaGen).
% % We evaluate the capability of semantic information extraction under the LLaVA-v1.5 setting, and the capability of pixel information extraction on the ImageNet-50K validation set.
% % The gray bar represents the default setting in our experiments.
% }
% \label{tab:abl-l}
% \resizebox{0.90\linewidth}{!}{
% \begin{tabular}{c l | cccc | cc}
% \toprule
% \multirow{2}{*}{\textbf{Exp}} & \multirow{2}{*}{\textbf{Setting}} & \multicolumn{4}{c|}{\textbf{Multimodal understanding}} & \multicolumn{2}{c}{\textbf{Reconstruction}} \\
% & & GQA$\uparrow$ & MME-P$\uparrow$ & SEEDB$\uparrow$ & POPE$\uparrow$ & rFID$\downarrow$ & Usage$\uparrow$ \\ \midrule
% 1 & JTrain & 57.8 & 1357.4 & 55.3 & 80.4 & 3.22 & 45.9\% \\ % \midrule
% 2 & Dual-Encoder & 58.0 & 1240.0 & 56.7 & 77.3 & 3.78 & 92.9\% \\
% 2 & Dual-Tokenizer & 58.7 & 1210.9 & 56.1 & 80.1 & 2.19 & 97.0\% \\ \midrule
% \graycolorrow{
% 3 & SemHiTok & 60.3 & 1355.8 & 62.9 & 82.6 & 1.42 &93.7\%
% \\ }
% \bottomrule
% \end{tabular}
% }
% \end{table}
\begin{table}[tb]
\centering
\caption{Impact of training strategy and structure.
\textbf{DStruct}: Using different learnable codebooks to model semantic and pixel information;
\textbf{DTrain}: Adopt a phased optimization training strategy.
\textbf{Exp 1}: used the same architecture as SDE and adopted a joint training strategy;
\textbf{Exp 2}: using the same architecture as SemHiTok but with joint training;
\textbf{Exp 3}: using two separately pretrained tokenizers (Llamagen and SigLIP) to independently extract semantic and pixel information.
% Exp1: SemHiTok simultaneously trains both the semantic branch and the pixel branch within its architecture.
% Exp2: Two encoders are employed, but only a single codebook is used.
% Exp3: Separate tokenizers are adopted, consisting of a semantic tokenizer (identical to that in SemHiTok) and a pixel tokenizer (LLaMaGen).
% We evaluate the capability of semantic information extraction under the LLaVA-v1.5 setting, and the capability of pixel information extraction on the ImageNet-50K validation set.
% The gray bar represents the default setting in our experiments.
}
\label{tab:abl-l}
\resizebox{0.90\linewidth}{!}{
\begin{tabular}{c c c | cccc | cc}
\toprule
\multirow{2}{*}{\textbf{Exp}} & \multirow{2}{*}{\textbf{DStruct}} & \multirow{2}{*}{\textbf{DTrain}} & \multicolumn{4}{c|}{\textbf{Multimodal understanding}} & \multicolumn{2}{c}{\textbf{Reconstruction}} \\
& & & GQA$\uparrow$ & MME-P$\uparrow$ & SEEDB$\uparrow$ & POPE$\uparrow$ & rFID$\downarrow$ & Usage$\uparrow$ \\ \midrule
1 & \ding{55} & \ding{55} & 58.0 & 1240.0 & 56.7 & 77.3 & 3.78 & 92.9\% \\
2 & \ding{51} & \ding{55} & 57.8 & 1357.4 & 55.3 & 80.4 & 3.22 & 45.9\% \\ % \midrule
3 & \ding{51} & \ding{51} & 58.7 & 1210.9 & 56.1 & 80.1 & 2.19 & 97.0\% \\ \midrule
\graycolorrow{
4 & \multicolumn{2}{c|}{SemHiTok} & 60.3 & 1355.8 & 62.9 & 82.6 & 1.42 &93.7\%
\\ }
\bottomrule
\end{tabular}
}
\end{table}
\section{Related Work}
\label{sec:rel_work}
% We present and analyze the Specialized Image Tokenizer and Unified Image Tokenizer.
% Please refer to \textbf{\textcolor{blue}{Supplementary Materia \ref{A:rela_1} and \ref{A:rela_2}}.}
\subsection{Specialized Image Tokenizer}\label{A:rela_1}
\textbf{Tokenization for Autoregressive Generation.}
% Image tokenizers are crucial for autoregressive image generation\citep{van2017neural,rombach2022high,tian2025visual}.
%%VQVAE
VQVAE\citep{van2017neural}
learns a discrete representation using a learnable codebook in auto-encoder architectures.
VQGAN further improves the perceptual quality by using adversarial training\citep{goodfellow2014generative}.
improved the architecture with perceptual and adversarial-based losses for better reconstruction.
This approach yields more precise and detailed image representations, significantly improving upon previous methodologies in image generation and processing.
% Subsequently, ViT-VQGAN\citep{yu2021vector} and Efficient-VQGAN\citep{cao2023efficient} advance the framework with the transformer design.
%% Recent,
% In recent literature, res
In recent literature, researchers are turning to efficient codebook structures\citep{ibq, zhang2023regularized, yu2021vector, fqgan} and better quantization methods\citep{zha2024language, titok} to improve generation performance and compression rates.
IBQ\citep{ibq} proposes the Index Backpropagation Quantization codebook update method, achieving stable training of large-scale codebooks.
FQGAN\citep{fqgan} uses multiple codebooks and product quantization, where each codebook encodes a different type of feature.
However, this work focuses only on image reconstruction and generation tasks, performing poorly on multimodal understanding tasks\citep{liquid,lwm}.
%%%% For understanding
\noindent \textbf{Tokenization for Understanding.} In multimodal large language models (MLLMs)\citep{li2023blip,li2022blip,radford2021learning,liu2023visual, Bai2023QwenVLAF,chen2024internvl}, researchers leverage CLIP\citep{radford2021learning} and BLIP\citep{li2022blip} to extract visual characteristics that align with the language during its pre-training phase.
Building upon many works~\citep{liu2023visual, Bai2023QwenVLAF,chen2024internvl} have been collected and trained on high-quality datasets to achieve remarkable performance.
LLaVA\citep{liu2023visual} utilizes a vision encoder to align the vision inputs before LLMs. QwenVL\citep{Bai2023QwenVLAF} and InterVL\citep{chen2024internvl} achieve better results through increased resolution, higher-quality datasets, etc.
%% Current methods, single codebook? disadvantage.... Challenge 2
However, these text-aligned image encoders tend to focus on semantic information and ignore texture information, which is important for the generation task.
\subsection{Unified Image Tokenizer} % 感觉不要也无所谓
\label{A:rela_2}
%%%% EMU SEED-LLAMA VILA-U ref tokenflow qlip
Numerous efforts have emerged to develop unified visual generation and understanding within one MLLM\citep{wang2024illume,wang2024emu3,xie2024show,dong2023dreamllm,ge2024seed,sun2024generative,team2024chameleon,wu2024vila}.
There are two main lines to bridge the gap.
% Many workers\citep{dong2023dreamllm,ge2024seed,sun2024generative} combine diffusion models with \textbf{Continuous tokens} for image generation.
Many workers~\citep{dong2023dreamllm,ge2024seed,sun2024generative} combine diffusion models with LLMs for image generation.
DREAMLLM\citep{dong2023dreamllm} presents a unified framework that not only provides multimodal understanding but also creates multimodal content via diffusion models.
% SEED-X\citep{ge2024seed}
Emu2\citep{sun2024generative} trains a unified generative model using a diffusion-based decoder.
These approaches inevitably increase model complexity and are not simple enough.
Other workers~\citep{team2024chameleon,wu2024vila,xie2024show,wang2024emu3} adopt VQVAE-based encoders to convert images into
discrete tokens.
% \textbf{Discrete Tokens}.
% Chameleon\citep{team2024chameleon} interleaves discrete visual and text tokens within a unified foundation model.
% EMU3\citep{wang2024emu3}
% Show-o\citep{xie2024show} unifies understanding and generation by masked language modeling.
Chameleon\citep{team2024chameleon} and EMU3\citep{wang2024emu3} directly use VQGAN\citep{vqgan}, which is optimized by pixel reconstruction as the image tokenizer, while this method increases resource consumption during the pre-training stage and degrades multimodal understanding performance.
VILA-U\citep{wu2024vila} introduces a unified image tokenizer that incorporates a text-aligned branch within the VQGAN training paradigm.
However, due to the gap between the semantic feature and the pixel feature, the joint optimization approach may lead to suboptimal solutions
%% The disadvantage, improve
In contrast, our proposed SemHiTok can add the ability of extracting texture features without changing the discrete semantic features, and avoids the challenges brought by joint optimization.
% \subsection{Specialized Image Tokenizer}
% Due to space limitations, please refer to \suppltext{\ref{A:1}}.
% %%% VQGAN VQVAE 之类的工作 ref tokenflow unitok qlip
% %%%% For generation
% \textbf{Tokenization for Generation.} Image tokenizers are crucial for autoregressive image generation\citep{van2017neural,rombach2022high,tian2025visual}.
% %%VQVAE
% % VQVAE\citep{van2017neural}
% % learns a discrete representation using a learnable codebook in auto-encoder architectures.
% % discretizes continuous tokens by mapping them to the nearest neighbors within a learnable codebook, quantizing patch-level features using the nearest codebook entry. The codebook is learned through an encoder-decoder structure via reconstruction loss, providing a foundational approach for effective image processing and generation.
% %%VQGAN
% % To further improve better perceptual quality, VQGAN\citep{vqgan} introduces adversarial training\citep{goodfellow2014generative}.
% VQGAN\citep{vqgan} further improves better perceptual quality by using adversarial training\citep{goodfellow2014generative}.
% % advanced the existing architecture by integrating perceptual loss and discriminator loss, alongside adversarial loss, to enhance reconstruction quality.
% This approach yields more precise and detailed image representations, significantly improving upon previous methodologies in image generation and processing.
% % Subsequently, ViT-VQGAN\citep{yu2021vector} and Efficient-VQGAN\citep{cao2023efficient} advance the framework with the transformer design.
% %% Recent,
% % In recent literature, res
% % In recent literature, researchers are turning to efficient codebook structures\citep{ibq, zhang2023regularized, yu2021vector, fqgan} and better quantization methods\citep{zha2024language, titok} to improve generation performance and compression rates.
% % IBQ
% % In recent literature, image tokenizers focuse on enhancing codebook unilization\citep{yu2021vector,zhang2023regularized}, compression rates\citep{zha2024language} and increasing codebook size\citep{shi2024taming}, exploring novel architectures\citep{li2024imagefolder,tian2025visual}.
% % All of thses efforts aim to improve reconstruction quality in a single codebook.
% %% current method, ignore what?, Challenge 1
% % IBQ\citep{ibq} proposes the Index Backpropagation Quantization codebook update method, achieving stable training of large-scale codebooks.
% % TiTok\citep{titok} uses a self-attention mechanism to transfer image token information to latent tokens, achieving higher compression rates and improving generation performance.
% Although these methods efficiently retain low-level texture information, they frequently fail to capture high-level semantic information, which limits their application to multimodal understanding tasks.
% % FQGAN\citep{fqgan} uses multiple codebooks and product quantization where each codebook encodes a different type of feature (Pixel, CLIP\citep{clip}, DINO\citep{dino} etc.).
% % However, this work focuses only on image reconstruction and generation tasks.
% %%%% For understanding
% \noindent \textbf{Tokenization for Understanding.} In multimodal large language models (MLLMs)\citep{li2023blip,li2022blip,radford2021learning,liu2023visual,Bai2023QwenVLAF,chen2024internvl}, researchers leverage CLIP\citep{radford2021learning} and BLIP\citep{li2022blip} to extract visual characteristics that align with the language during its pre-training phase.
% % Building upon, many works\citep{liu2023visual,Bai2023QwenVLAF,chen2024internvl} have been collected and trained on high-quality datasets to achieve remarkable performance.
% % LLaVA\citep{liu2023visual} utilizes vision encoder to align the vision inputs before LLMs. QwenVL\citep{Bai2023QwenVLAF} and InternVL\citep{chen2024internvl} achieve better results through increased resolution, higher-quality datasets, etc.
% %% Current methods, single codebook? disadvantage.... Challenge 2
% However, these text-aligned image encoders tend to focus only on semantic information and ignore texture information, which is important for the generation task.
% % Our SemHiTok
% % Nevertheless, these models encode images into continuous tokens, which poses a challenge for the uniform modeling of both visual and textual tokens.
%%% clip siglip
% \subsection{Visual Tokenizer For Understanding}
%%% 介绍clip siglip ref unitok
% \subsection{Unified Tokenizer For MLLMs}
%%% TOKENFLOW MUSEVL VILA-U QLIP
% \subsection{Unified Image Tokenizer} % 感觉不要也无所谓
% %%%% EMU SEED-LLAMA VILA-U ref tokenflow qlip
% Numerous efforts have emerged to develop unified visual generation and understanding within one MLLM\citep{wang2024illume,wang2024emu3,xie2024show,dong2023dreamllm,ge2024seed,sun2024generative,team2024chameleon,wu2024vila}.
% There are two main lines to bridge the gap.
% % Many workers\citep{dong2023dreamllm,ge2024seed,sun2024generative} combine diffusion models with \textbf{Continuous tokens} for image generation.
% Many workers~\citep{dong2023dreamllm,ge2024seed,sun2024generative} combine diffusion models with LLMs for image generation.
% DREAMLLM\citep{dong2023dreamllm} presents a unified framework that not only provides multimodal understanding but also creates multimodal content via diffusion models.
% % SEED-X\citep{ge2024seed}
% Emu2\citep{sun2024generative} trains a unified generative model using a diffusion-based decoder.
% These approaches inevitably increase model complexity and are not simple enough.
% Other workers\citep{team2024chameleon,wu2024vila,xie2024show,wang2024emu3} adopt VQVAE-based encoders to convert images into
% discrete tokens.
% % \textbf{Discrete Tokens}.
% % Chameleon\citep{team2024chameleon} interleaves discrete visual and text tokens within a unified foundation model.
% % EMU3\citep{wang2024emu3}
% % Show-o\citep{xie2024show} unifies understanding and generation by masked language modeling.
% Chameleon\citep{team2024chameleon} and EMU3\citep{wang2024emu3} directly use VQGAN\citep{vqgan}, which is optimized by pixel reconstruction as the image tokenizer, while this method increases resource consumption during the pre-training stage and degrades multimodal understanding performance.
% VILA-U\citep{wu2024vila} introduce a unified image tokenizer that incorporates a text-aligned branch within the VQGAN training paradigm.
% However, due to the gap between semantic feature and texture feature, the joint optimization approach may lead to suboptimal solutions
% %% The disadvantage, improve
% In contrast, our proposed SemHiTok can add the ability of extracting texture features without changing the discrete semantic features, and avoids the challenges brought by joint optimization.
% Some existing studies\citep{liquid,team2024chameleon,llavit,dong2023dreamllm} employ tokenizers designed either for generation or for understanding within unified large models that integrate both tasks.
% However, due to the substantial gap between the feature preferences of these two tasks, such approaches often lead to suboptimal performance in either understanding or generation.
% Recent research has explored the development of unified tokenizers.
% A straightforward approach is to combine the training objectives of generative tokenizers with those of understanding-oriented tokenizers.
% VILA-U\citep{vilau} integrates pixel reconstruction with multimodal contrastive learning, enabling discrete visual tokens to preserve both pixel-level and semantic-level features.
% Subsequent methods such as MUSE-VL\citep{musevl} and TokenFlow\citep{tokenflow} attempt to mitigate conflicts between different representational levels by introducing separate encoders or codebooks.
% However, these approaches remain reliant on joint optimization, which often results in suboptimal solutions due to competing learning objectives.
% Our proposed SemHiTok not only decouples the structure but also supports a two-stage training strategy with the novel codebook structure, which achieves a better trade-off in the extraction of pixel information and semantic information.
\section{Conclusion}
\label{sec:conclusion}
\textbf{Conclusion:}
We present SemHiTok, a unified image tokenizer featuring a Semantic-Guided Hierarchical Codebook (SGHC) to balance semantic depth with reconstruction fidelity. SemHiTok achieves SOTA results on LLaVA-v1.5 understanding and ImageNet-50k reconstruction benchmarks. When integrated into a unified MLLM, it delivers competitive performance in both understanding and generation, offering a powerful, discrete, and fully compatible alternative to existing tokenizers.
\textbf{Limitation:}
We present two limitations:
% \textit{(1). High-resolution exploration}: Due to limited computational resources, our experiments are conducted at a resolution of 256. However, real-world applications often require understanding and generation at higher resolutions, which is not addressed in this work.
\textit{(1). Low generation efficiency:} Owing to the use of standard image quantization methods and settings, each 256-resolution image is represented by 256 tokens. This results in relatively low efficiency and high computational cost.
\textit{(2). Unified large model potential:} In tasks involving natural language understanding and multimodal reasoning, advanced post-training techniques such as Chain-of-Thought (CoT) are not explored. This remains a promising direction for future research in the community.
\textbf{Future Work:} In the future, we can explore the potential of unified image tokenizers and test their performance on more difficult tasks, such as image editing and multiple rounds of conversations.
In addition, improving the compression ratio of the model and designing a tokenizer that serializes the image in one dimension are also expected.
% \subsection{LLM Usage Statement}
% We clarify that the use of LLMs in this study is restricted to writing assistance, specifically for grammar correction and enhancing readability. No LLM was involved in the research design, experimental execution, or data analysis.
\section*{Acknowledgement}
This work is supported by National Key Research and Development Program of China(2024YFE0203100), Scientific Research Innovation Capability Support Project for Young Faculty (No.ZYGXQNJSKYCXNLZCXM-I28), National Natural Science Foundation of China (NSFC) under Grants No.62476293 and No.62372482, and General Embodied AI Center of Sun Yat-sen University.
\section*{Ethics Statement}
This research does not involve potentially harmful insights, methodologies, or applications, and it raises no concerns regarding conflicts of interest, sponsorship, discrimination, bias, fairness, privacy, security, legal compliance, or research integrity.
\section*{Reproducibility statement}
\textbf{Data.}
The training datasets employed in this study are detailed in Appendix.\ref{A:exp_tok} and Appendix. \ref{A:exp_uni}, and all are publicly accessible open-source resources.
\textbf{Method.}
To support reproducibility, we elaborate on the methodological details in Section.\ref{sec:method}, and the implementation will be open-sourced after acceptance.
\textbf{Performance.}
All evaluations are carried out on open benchmarks, thereby ensuring the reproducibility of our results.
\textbf{Code And Model Weight.}
We will open-source the code and model weight files.
\section{Technical Appendices and Supplementary Material}
% \localtableofcontents
\subsection{LLM Usage Statement}
We clarify that the use of LLMs in this study is restricted to writing assistance, specifically for grammar correction and enhancing readability. No LLM was involved in the research design, experimental execution, or data analysis.
\subsection{Deployment Details of Unified MLLM.} \label{A:unimllm}
\textbf{Training Data Form.}
For multimodal understanding, we use <|im\_start|> and <|im\_end|> to delimit the image segment within the input sequence.
To distinguish between modalities and enable visual content generation, we insert special tokens: <IMG\_XXXXX>, which represent image codes in LLMs' vocabulary. Specifically, the i-th code in the unified tokenizer codebook corresponds to <IMG\_i>.
In addition, we add <start\_of\_image> and <end\_of\_image> to indicate the start and end of image generation.
\textbf{Vocabulary Embedding Processing Flow.}
For understanding samples, we follow the LLaVA setting: discrete features are first extracted by the tokenizer and then fed into the LLM through an adapter layer.
To ensure consistency between understanding and generation, we employ the same features for generation samples instead of using the embeddings of <IMG\_XXXXX>.
After training, we replace the visual code embedding in the vocabulary to align training with inference.
The detailed procedure is illustrated in Algorithm \ref{alg:1}.
% 训练的时候
\begin{algorithm}[H]
% \LinesNotNumbered
\BlankLine
\textbf{Before Training:}\;
1. Add special tokens <IMG\_XXXXX> into LLM vocabulary
2. Preprocess training images, e.g.\newline
``generate a dog + \texttt{<Image>}'' $\to$ \newline
``generate a dog + \texttt{<start\_of\_image><IMG\_i>...<IMG\_k><end\_of\_image>}''
\BlankLine
\textbf{During Training:}\;
1. Feed preprocessed samples into LLM tokenizer $\to$ \texttt{text\_id}\;
2. Lookup embedding $V$ from \texttt{text\_id} $\to E_{text}$\;
3. Get image IDs from \texttt{text\_id} $\to$ \texttt{img\_id};\;
Lookup unified codebook embedding from \texttt{img\_id} $\to E_{img}$;\;
Apply adapter layer: $E_{img} \to E'_{img}$\;
4. Replace image embeddings in $E_{text}$ with $E'_{img}$ $\to E'_{text}$\;
5. Feed $E'_{text}$ into LLM backbone for training\;
\BlankLine
\textbf{Generation:}\;
1. Apply adapter layer on codebook embeddings $C_{img} \to C'_{img}$\;
2. Replace image part of $V$ with $C'_{img}$ $\to V'$\;
3. Start autoregressive generation\;
\caption{Procedure: Training and Generation Pipeline}
\label{alg:1}
\end{algorithm}
%\textbf{Dual-MLP Adapter}
% The design of the Dual-MLP Adapter is presented in Fig.\ref{fig:dmlp}.
% \begin{figure}[hb]
% \centering
% \includegraphics[width=0.9\linewidth]{figs/fig-framwork.pdf}
% \caption{Caption}
% \label{fig:dmlp}
% \end{figure}
\subsection{Tokenizer Experimental details}\label{A:exp_tok}
% 最好做个表格
The training of SemHiTok is conducted in two stages.
During semantic codebook training, we train the semantic tokenizer for one epoch on 50M subset of COYO-700M\citep{coyo}.
For the PRE stage, we first train the model(ViT-Base) on ImageNet\citep{imagenet}, and then fine-tune on 50M COYO to improve its generalization, following LlamaGen\citep{llamagen}.
To further improve the reconstruction and generation performance, we enlarge the size of the pixel decoder(ViT-Large) and fine-tune on the 20M COYO data and 20M MidJourney-style synthetic data.
The full training takes about 3 days on 32 v100 GPUs.
% \begin{table}[!ht]
% \centering
% \caption{Training details of tokenizer.}\label{tab:tokenizer_train}
% \begin{tabular}{ccccccc}
% \hline
% ~ & Data & Trainable Module & Learning Rate & Batch Size & Traning Steps & Cost(16 v100) \\ \hline
% \makecell{Semantic\\Branch} & 50M COYO & \makecell{Semantic codebook\\Semantic decoder} & 5e-5 & 256 & 200k & 1 day \\
% PRE & ImageNet-1K\&40M COYO & Pixel Encoder \& Pixel Codebook \& Pixel Decoder & 1e-4 & 256 & 400k & 3.6 day \\
% Enhanced Decoder & 20M COYO\&20M MJ & Pixel Decoder & 1e-4 & 256 & 200k & 2 day \\ \hline
% \end{tabular}
% \end{table}
% \begin{table}[tb]
% \centering
% \begin{tabular}{l cccc}
% \toprule
% Method & Res & \makecell[c]{Code \\ Shape} & \makecell[c]{Codebook \\ Size} & rFID\( \downarrow \) \\
% \midrule
% \multicolumn{5}{l}{\textit{Only Reconstruction}} \\ \midrule
% LlamaGen~\citep{llamagen} & 256 & $16\times 16$ & 16,384 & 2.19 \\
% RQVAE~\citep{rqvae} & 256 & $16\times 16\times 4$ \textsuperscript{\ddag} & 16,384 & 3.20 \\
% % RQVAE~\citep{rqvae} & 256 & 1024\textsuperscript{\ddag} & 16,384 & 1.30 \\
% % VQGAN-LC~\citep{vqganlc} & 256 & 256 & 16,384 & 3.01 \\
% VQGAN-LC~\citep{vqganlc} & 256 & $16\times 16$ & 100,000 & 2.62 \\
% IBQ~\citep{ibq} & 256 & $16\times 16$ & 16,384 & 1.37 \\
% IBQ~\citep{ibq} & 256 & $16\times 16$ & 262,144 & 1.00 \\
% FQGAN~\citep{fqgan} & 256 & $16\times 16\times 2$ & $16,384\times 2$\textsuperscript{\dag} & 0.94 \\
% % FQGAN~\citep{fqgan} & 256 & 256 & $16384\times 3$\textsuperscript{\dag} & 0.76 \\
% \midrule
% \multicolumn{5}{l}{\textit{Unified}} \\ \midrule
% VILA-U~\citep{vilau} & 256 & 16x16x4\textsuperscript{\ddag} & 16,384 & 1.80 \\
% SDE(MUSE-VL)~\citep{musevl} & 256 & $16\times 16$ & 32,768 & 2.26 \\
% TokenFlow~\citep{tokenflow} & 256 & 680 & 32,768 & 1.37 \\
% TokLIP~\citep{toklip} & 256 & $16\times 16$ & 16,384 & 2.19 \\
% QLIP-B~\citep{toklip} & 256 & $16\times 16$ & $2^{28}$ & 3.21 \\
% UniTok~\citep{unitok} & 256 & $16\times 16$ & $16,384\times 4$\textsuperscript{\dag} & 0.39 \\
% \colorrow{\textbf{SemHiTok(ours)} & 256 & $16\times 16$ & 196,608 & 1.16 \\ }
% \colorrow{\textbf{SemHiTok(ours)} & 384 & $27\times 27$ & 196,608 & 0.66 \\ }
% \bottomrule
% \end{tabular}
% \caption{Caption}
% \label{tab:placeholder}
% \end{table}
\subsection{Unified MLLM Experimental details}\label{A:exp_uni}
Following existing work\citep{vilau, unitok}, we first pretrain the model and adapter layer on a mix of multimodal data, which is composed of 3.5M language data from Magpie\citep{magpie} and Openorca\citep{openorca}, 10M caption image-text pairs data, and 15M MidJourney-style synthetic data.
Subsequently, we finetune the model on 1M language dataset from Magpie\citep{magpie} and Evol-Instruct\citep{evol-ins}, 4M generation data and 4M understanding data from emova\citep{emova} and LLaVA-SFT\citep{llava}.
The full training takes about 7 days on 32 A800 GPUs.
During image generation inference, we apply classifier-free guidance\citep{cfg} with a scale factor of $2.5$.
% \subsection{\rev{Comparison with other methods on more metrics for the reconstruction task}}\label{A:abl-more-rec}
% \rev{We present the reconstruction metrics of several tokenizers in Tab.~\ref{tab:rec-2}, including rFID, PSNR, and SSIM.
% Our method achieves competitive performance across multiple reconstruction task metrics.}
% \begin{table}[tb]
% \definecolor{mycustomcolor}{HTML}{DDEEFF}
% \centering
% \caption{\rev{More Comparison on more metrics for the reconstruction task.}}
% \label{tab:rec-2}
% \begin{tabular}{l cccc | cc}
% \toprule
% Method & Res & \makecell[c]{Code \\ Shape} & \makecell[c]{Codebook \\ Size} & rFID\( \downarrow \) & PSNR\( \uparrow \) & SSIM\( \uparrow \)
% \\
% \midrule
% % \multicolumn{5}{l}{\textit{Only Reconstruction}} \\ \midrule
% LlamaGen~\citep{llamagen} & 256 & $16\times 16$ & 16,384 & 2.19 & 20.65 & 0.54 \\
% % RQVAE~\citep{rqvae} & 256 & $16\times 16\times 4$ \textsuperscript{\ddag} & 16,384 & 3.20 \\
% % RQVAE~\citep{rqvae} & 256 & 1024\textsuperscript{\ddag} & 16,384 & 1.30 \\
% % VQGAN-LC~\citep{vqganlc} & 256 & 256 & 16,384 & 3.01 \\
% Show-o~\citep{xie2024show} & 256 & $16\times 16$ & 8192 & 3.21 & 21.34 & 0.59 \\
% TokenFlow~\citep{tokenflow} & 256 & $16\times 16$ & 16,384 & 1.37 & 21.41 & 0.69 \\
% % IBQ~\citep{ibq} & 256 & $16\times 16$ & 262,144 & 1.00 \\
% % FQGAN~\citep{fqgan} & 256 & $16\times 16\times 2$ & $16,384\times 2$\textsuperscript{\dag} & 0.94 \\
% % FQGAN~\citep{fqgan} & 256 & 256 & $16384\times 3$\textsuperscript{\dag} & 0.76 \\
% % \midrule
% % VILA-U~\citep{vilau} & 256 & 16x16x4\textsuperscript{\ddag} & 16,384 & 1.80 \\
% SDE(MUSE-VL)~\citep{musevl} & 256 & $16\times 16$ & 32,768 & 2.26 & 20.14 & 0.65 \\
% % TokenFlow~\citep{tokenflow} & 256 & 680 & 32,768 & 1.37 \\
% % TokLIP~\citep{toklip} & 256 & $16\times 16$ & 16,384 & 2.19 \\
% QLIP-B~\citep{toklip} & 256 & $16\times 16$ & $2^{28}$ & 3.21 & 21.34 & 0.59 \\
% UniTok~\citep{unitok} & 256 & $16\times 16 \times 8$ & $16,384\times 4$\textsuperscript{\dag} & 0.41 & 27.28 & 0.77 \\
% \rowcolor{mycustomcolor}
% \textbf{SemHiTok(ours)} & 256 & $16\times 16$ & 196,608 & 1.16 & 21.38 & 0.69 \\
% % \rowcolor{mycustomcolor}
% % \textbf{SemHiTok(ours)} & 384 & $27\times 27$ & 196,608 & 0.66 & & \\
% \bottomrule
% \end{tabular}
% \end{table}
\subsection{Visualizations on understanding tasks}\label{A:vis_under}
We present more visualization of multimodal understanding samples in \figref{fig:a_und1}.
\begin{figure}[htb]
\centering
\includegraphics[width=0.95\linewidth]{figs/fig-a_und2.pdf}
\caption{Visualizations on understanding tasks.}
\label{fig:a_und1}
\end{figure}
\subsection{\rev{More Comparison of generation on GenEval and DPG}}\label{A:more_gen_tab}
\definecolor{mycustomcolor}{HTML}{DDEEFF} % HTML 模型下的颜色定义
\rev{To more fully demonstrate the superiority of SemHiTok, we conduct further comparisons with other generative models and unified models on GenEval~\cite{geneval} and DPG~\cite{dpg}.
As shown in Tab.\ref{tab:geneval_dpg}, our method still demonstrates competitive performance.}
\begin{table}[t]
% \newcommand{\colorrow}[1]{\rowcolor{lightblue} #1}
\centering
\caption{\rev{Comparison with other methods on GenEval and DPG Bench. SemHiTok still achieves competitive performance even with a smaller amount of data.}}
\resizebox{0.95\textwidth}{!}{
\begin{tabular}{l cccc | cc}
\toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Params} & \multirow{2}{*}{Type} & \#Training & \multirow{2}{*}{Res.} & \multirow{2}{*}{GenEval} & \multirow{2}{*}{DPG} \\
& & & Images & & & \\
\midrule
\multicolumn{6}{l}{\textit{Only Gen.}} \\ \midrule
SD v2.1~\citep{sdv21} & -- & Diff & 2000M & 1024 & 0.50 & -- \\
DALL-E 2~\citep{dalle2} & -- & Diff & 650M & 1024 & 0.52 & -- \\
DALL-E 3~\citep{dalle3} & -- & Diff & -- & 1024 & 0.67 & 83.50 \\
PixArt-$\alpha$~\citep{pixart} & 0.6B & Diff & -- & 1024 & 0.48 & -- \\
SDXL~\citep{podell2023sdxl} & 2.6B & Diff & 2000M & 1024 & 0.55 & 74.65 \\
Playgroundv2.5~\citep{li2024playground} & -- & Diff & -- & 1024 & -- & 75.47 \\
\midrule
\multicolumn{6}{l}{\textit{Und\&Gen.}} \\ \midrule
Transfusion~\citep{transfusion} & 7.3B & AR & 3.5B & 256 & 0.63 & -- \\
Ming-Lite-Uni~\citep{mingliteuni} & 8B & AR-Scale & 5M & 512 & 0.62 & -- \\
Show-o~\citep{xie2024show} & 1.5B & \makecell{Diff} & 36M & 256 & 0.53 & 67.27 \\
Emu3~\citep{wang2024emu3} & 8B & AR & -- & 256 & 0.66 & 80.60 \\
Janus~\citep{janus} & 1.3B & AR & 65M & 384 & 0.61 & -- \\
SynerGen-VL~\citep{synergen} & 2.4B & AR & 25M & 256 & 0.61 & -- \\
UniFork~\citep{synergen} & 0.76B & AR & -- & 384 & 0.46 & -- \\
ILLUME~\citep{wang2024illume} & 7B & AR & 15M & 512 & 0.61 & -- \\
ILLUME+~\citep{illume+} & 3B & AR & 46M & 512 & 0.72 & -- \\
MUSE-VL ~\citep{musevl} & 7B & AR & 10M & 256 & 0.57 & -- \\
QLIP-B ~\citep{qlip} & 1.5B & AR & 18M & 256 & 0.48 & 78.17 \\
TokenFlow~\citep{tokenflow} & 7B & AR & 60M & 256 & 0.63 & 73.38\\
MMaDA~\citep{mmada} & 8B & Diff & -- & 512 & 0.63 & 69.97 \\
Liquid~\citep{liquid} & 7B & AR & 30M & 512 & 0.55 & 83.45 \\
UniTok ~\citep{unitok} & 7B & AR & 30M & 256 & 0.59 & 83.45 \\
Tar-1.5B~\citep{tar} & 3B & AR & 46M & 256 & 0.76 & 82.96 \\
Tar-1.5B W/Self-Reflect~\citep{tar} & 3B & AR & 46M & 256 & 0.78 & 84.10 \\
Tar-7B~\citep{tar} & 7B & AR & 46M & 256 & 0.84 & 84.19 \\
Tar-7B W/Self-Reflect~\citep{tar} & 7B & AR & 46M & 256 & 0.85 & 84.65 \\
Ming-UniVision ~\citep{mingunivision} & 16B-A3B & AR-Continuous & -- & 512 & 0.85 & 82.12 \\
\rowcolor{mycustomcolor}
\textbf{SemHiTok(ours)} & 7B & AR & 15M & 256 & 0.71 & 83.59
\\
% \rowcolor{mycustomcolor}
% \textbf{SemHiTok(ours)} & 7B & AR & 15M & 384 & \textbf{0.83} & \textbf{0.66} & {5.70}
% \\
\bottomrule
\end{tabular}
}
\label{tab:geneval_dpg}
\end{table}
\begin{table}[tb]
\centering
\caption{\rev{Quantitative evaluation of pixel consistency using Variance Reduction Ratio (VRR). SemHiTok\textsubscript{sem} significantly outperforms the random baseline, verifying the intrinsic semantic-pixel correlation. Furthermore, our hierarchical refinement (SemHiTok\textsubscript{pix}) achieves the best consistency, exceeding the standard VQGAN baseline.}}
\label{tab:tab-vrr}
\begin{tabular}{c | c c c c}
\toprule
Experiment & Random Baseline & SemHiTok\textsubscript{sem} & VQGAN & SemHiTok\textsubscript{pix} \\ \midrule
VRR & 0.185\%±0.005\% & 5.245\%±0.085\% & 9.535\%±0.095\% & 13.705\%±0.075\% \\
\bottomrule
\end{tabular}
\end{table}
% \begin{table}[tb]
% \definecolor{lightblue}{RGB}{240,248,255}
% % \newcommand{\colorrow}[1]{\rowcolor{lightblue} #1}
% \caption{Comparison of generation quality on GenAI and MJHQ30K.
% SemHiTok achieves comparable results with specialist models and unified MLLMs.
% }
% \label{tab:gen_results}
% \centering
% \end{table}
\subsection{Visualizations on generation tasks}\label{A:vis_gen}
We present more visualization of generated images in \figref{fig:a_gen}.
\begin{figure}[htb]
\centering
\includegraphics[width=0.95\linewidth]{figs/fig-a_gen.pdf}
\caption{More generated images presentation.}
\label{fig:a_gen}
\end{figure}
\subsection{More Visualized reconstruction results from the ablation of key module}\label{A:vis_rec}
We show more reconstruction effects on the ablation of key modules in \figref{fig:a_rec}.
\begin{figure}[htb]
\vspace{-6mm}
\includegraphics[width=0.9\linewidth]{figs/fig-a_rec.pdf}
\caption{More visualized reconstruction results from the ablation of key module.}
\label{fig:a_rec}
\end{figure}
\subsection{Quantitative analysis of pixel features in the semantic codebook}\label{A:qa}
In Fig.\ref{fig:fig4}, we provide visualization results showing that image patches corresponding to the same semantic code share similar pixel features. To further support this analysis, we attempt to provide quantitative evidence using the following method:
\begin{enumerate}[leftmargin=2em, itemsep=0pt, parsep=0pt, topsep=0pt]
\item \rev{Using the tokenizer to extract the code indices along with their corresponding image patches.}
\item \rev{Applying the \textbf{DCT(Discrete Cosine Transform)}~\citep{rao2014discrete} to extract patch's pixel features.}
\item \rev{For each code, we compute the variance of the DCT features extracted from its associated image patches, followed by \textbf{averaging these variances over all codes}, denoted as $V_{mean}$. The more similar the pixels of the tokenized results are, the smaller the $V_{mean}$ becomes.}
\item \rev{Computing the \textbf{global variance of all image patches}, denoted as $V_{global}$.}
\item \rev{Computing \textbf{Variance Reduction Ratio (VRR)}, denoted as $VRR=1-V_{mean}/V_{global}$. The larger VRR indicates that the patches corresponding to the given code have more similar pixel features.}
\end{enumerate}
\rev{We randomly sampled 10K images from the ImageNet-Val (50K) dataset across 5 independent runs.
% The experimental results are shown in the Tab. \ref{tab:tab-vrr}.
The Random Baseline refers to randomly tokenized results, serving as the lower bound for the VRR metric.
VQGAN~\citep{vqgan}, as an expert model for pixel reconstruction, serves as the standard reference for the VRR metric.}
\rev{As illustrated in Tab. \ref{tab:tab-vrr}, the results provide strong quantitative evidence for the effectiveness of our Semantic-Guided Hierarchical Codebook (SGHC) design.
\textbf{1.)Validation of Semantic-Pixel Correlation.}
The SemHiTok\textsubscript{sem} achieves a VRR of 5.245\%, which is significantly higher than the Random Baseline (0.185\%). This quantitatively verifies our observation in Subsec\ref{subsec:pixsubc} that image patches assigned to the same semantic code naturally exhibit intrinsic pixel-level similarities. This correlation forms the theoretical foundation for using semantic codes to guide pixel quantization.
\textbf{2.)The Necessity of Pixel Branch.}
However, compared to VQGAN (9.535\%), the lower VRR of SemHiTok\textsubscript{sem} confirms that semantic codes alone are insufficient to capture fine-grained high-frequency details. This justifies the necessity of our pixel branch design to bridge the gap between semantic abstraction and pixel fidelity.
\textbf{3.)Superiority of Hierarchical Design.}
Most notably, SemHiTok\textsubscript{pix} achieves the highest VRR of \textbf{13.705\%}, surpassing the expert pixel reconstruction model VQGAN by a substantial margin (+4.17\%). This result demonstrates that decomposing the complex global pixel space into semantic-conditioned sub-spaces allows for significantly tighter feature clustering and lower variance than standard global quantization methods. This explains why SemHiTok achieves superior reconstruction performance despite utilizing a decoupled training strategy.}
%% 重建
\subsection{Impact of Concat Type and Sub-Codebook size.} \textbf{\label{A:abl_1}}
\begin{table*}[b]
\centering
\begin{minipage}{0.40\textwidth}
\caption{Impact of Concat type and sub-codebook size.
w/o sem: not use the semantic discrete token.
The gray bar represents the default setting in our experiments.
}
\label{tab:abl_rfid}
\resizebox{\linewidth}{!}{
\begin{tabular}{c c | c c}
\toprule
Concat & Subc Size & rFID \( \downarrow \) & Usage\(\uparrow\) \\ \midrule
w/o & 12 & 1.99 & 95.4\% \\
% Res & 12 & 0.79 \\
Len & 12 & 1.45 & 94.1\% \\
Dim & 8 & 1.42 & 96.4\% \\
\graycolorrow{
Dim & 12 & 1.26 & 93.7\%
\\ }
Dim & 16 & 1.19 & 79.3\%\\
\midrule
\end{tabular}
}
\end{minipage}
\hspace{2mm}
\begin{minipage}{0.55\textwidth}
\centering
\caption{
Impact of VQ Type and dim of the semantic codebook.
We evaluate multimodal understanding performance under the LLaVA-v1.5 setting.
The gray bar represents the default setting in our experiments.
}\label{tab:abl-sem}
\resizebox{\linewidth}{!}{
\begin{tabular}{cc | cccc}
\toprule
\makecell{VQ\\Type} & \makecell{Codebok \\ Dim} & {MME-P}$\uparrow$ & {MMB}$\uparrow$ &{SEED}$\uparrow$ &{MMU}$\uparrow$ \\ \midrule
Norm & 48 & 1249.6 & 52.2 & 52.8 & 34.8 \\
Vanilla & 48 & 1319.9 & 56.3 & 57.2 & 33.1 \\
\midrule
EMA & 32 & 1385.9 & 58.3 & 61.2 & 35.1 \\
\graycolorrow{
EMA & 48 & 1387.5 & 61.3 & 62.3 & 35.6}
\\
EMA & 64 & 1428.7 & 60.9 & 62.5 & 35.5 \\
\bottomrule
\end{tabular}
}
\end{minipage}
\end{table*}
For efficiency, we only conduct training and evaluation on ImageNet-1K.
We investigate the impact of the concat type of semantics between pixel and the sub-codebook size, on reconstruction performance as shown in Tab. \ref{tab:abl_rfid}.
For the Concat type, w/o semantic tokens leads to a significant drop in reconstruction quality, with score increase of 0.73 compared to the default setting.
Furthermore, concatenation along the sequence length performs worse than along the dimension, as the two token sets are spatially aligned, making dimensional concatenation more appropriate.
% For concat type, experiments show that concatenation along dimension yields the best performance.
% In contrast, the reconstruction performance without semantic discrete token is the worst, indicating that semantic information can be used as supplementary information to improve the reconstruction.
% Due to semantic discrete tokens correspond to texture discrete tokens in 2D grid space, the concat type in the length direction is not appropriate.
For sub-codebook size, increasing the size can improve the model's reconstruction performance, but it exhibits marginal utility.
In addition, the codebook usage significantly decreases when the sub-codebook size is set to 16, which indicates that too large a sub-codebook size is not cost-effective.
\subsection{Impact of Semanitc VQ Type, Codebook dim and Codebook size.}\label{A:semantic}
In Tab.\ref{tab:abl-sem}, we ablate VQ type and codebook dimension on multimodal understanding.
Specifically, Norm VQ is not suitable for semantic discretization and shows the worst performance in understanding tasks.
This indicates that it is difficult to discretely model complex and rich semantic information by Norm VQ.
Replacing it with vanilla VQ brings a clear improvement.
To further stabilize the semantic codebook, we use EMA VQ, which achieves the best results.
For the semantic codebook dimension, higher values offer better representation but with a marginal effect. We empirically set the dimension to 48 as the default.
\rev{Furthermore, we investigate the impact of semantic codebook size $K$ on multimodal understanding in Table \ref{tab:abl-k}.
The results indicate that while larger codebooks generally enhance performance, increasing $K$ beyond 16,384 yields only marginal gains and leads to a noticeable decline in codebook usage. Therefore, we select $K=16,384$ as the optimal balance between representational capacity and utilization efficiency.}
\begin{table}[t]
\centering
\begin{minipage}{0.65\textwidth}
\centering
\caption{\rev{Ablation of Semantic Codebook K. We evaluate on multimodal understanding tasks.}}
\resizebox{\linewidth}{!}{
\begin{tabular}{c | cccc c}
\toprule
$K$ & POPE & MME-P & SEED & GQA & Usage \\ \midrule
4096 & 79.7 & 1297.7 & 60.4 & 56.6 & 100 \\
8192 & 81.9 & 1343.0 & 61.7 & 59.7 & 99.7 \\
\graycolorrow{\textbf{16384} & 82.5 & 1355.8 & 62.9 & 60.3 & 99.0 \\}
32768 & 82.8 & 1364.2 & 62.5 & 60.6 & 93.0 \\
\bottomrule
\end{tabular}
}
\label{tab:abl-k}
\end{minipage}
\hspace{2mm}
\begin{minipage}{0.3\textwidth}
\centering
\caption{\rev{Comparative experiments with Different $K\times m$ on reconstruction task.}}
\resizebox{\linewidth}{!}{
\begin{tabular}{cc | cc}
\toprule
$K$ & $m$ & rFID $\downarrow$ & Usage $\uparrow$ \\ \midrule
\multirow{4}{*}{8192} & 12 & 1.58 & 95.5 \\
& 16 & 1.45 & 92.7 \\
& 24 & 1.31 & 89.8 \\
& 32 & 1.22 & 83.7 \\
\midrule
& 8 & 1.42 & 96.4 \\
\graycolorrow{ \textbf{16384} & 12 & 1.26 & 93.7} \\
& 16 & 1.19 & 79.3 \\
\bottomrule
\end{tabular}
}
\label{tab:abl-km}
\end{minipage}
\end{table}
\begin{table}[t]
\definecolor{mycustomcolor}{HTML}{DDEEFF}
\centering
\caption{\rev{Quantitative comparison of reconstruction quality under consistent total codebook sizes. SemHiTok outperforms existing quantization methods at the 65k scale and achieves comparable rFID to expert models at the 262k scale.}}
\label{tab:a-lc}
\begin{tabular}{l c | cc}
\toprule
Method & Codebook Size & rFID$\downarrow$ & Usage$\uparrow$ \\
\midrule
VQGAN-LC~\citep{vqganlc} & 65546 & 2.63 & 100.0 \\
VQGAN-LC(CLIP)~\citep{vqganlc} & 65546 & 2.40 & 100.0 \\
FSQ~\citep{fqgan} & 64000 & 2.80 & 100.0 \\
LFQ~\citep{lfq} & 65536 & 2.88 & 100.0 \\
SimVQ~\citep{simvq} & 65536 & 2.24 & 100.0 \\
\multirow{2}{*}{\textbf{SemHiTok}} & 8192$\times$8(65536) & 1.93 & 98.1 \\
& 16384$\times$4(65536) & 1.84 & 99.0 \\ \midrule
&&& \\[-1.2em] \midrule
SimVQ~\citep{simvq} & 262144 & 1.99 & 100.0 \\
Open-MAGVIT2~\citep{openmavit2} & 262144 & 1.17 & 100.0 \\
IBQ~\citep{ibq} & 262144 & 1.00 & 79.3 \\
\multirow{2}{*}{\textbf{SemHiTok}} & 8192$\times$32(262144) & 1.21 & 83.7 \\
& 16384$\times$8(262144) & 1.19 & 79.3 \\
\bottomrule
\end{tabular}
\end{table}
% \subsection{Limitation}\label{A:limi}
% % Limited by training data and computing resources, we didn't try it on multiple rounds of conversations and high resolution.
% % In addition, the potential of the tokenizer in multi-modal CoT and interweaving technology has not been explored.
% \begin{enumerate}
% \item \textbf{High-resolution exploration.} Due to limited computational resources, our experiments are conducted at a resolution of 256. However, real-world applications often require understanding and generation at higher resolutions, which is not addressed in this work.
% \item \textbf{Low generation efficiency.} Owing to the use of standard image quantization methods and settings, each 256-resolution image is represented by 256 tokens. This results in relatively low efficiency and high computational cost.
% \item \textbf{Unified large model potential.} In tasks involving natural language understanding and multimodal reasoning, advanced post-training techniques such as Chain-of-Thought (CoT) are not explored. This remains a promising direction for future research in the community.
% \end{enumerate}
% \subsection{Future Work}\label{A:future_work}
% In the future, we can explore the potential of unified image tokenizers and test their performance on more difficult tasks, such as image editing and multiple rounds of conversations.
% In addition, improving the compression ratio of the model and designing a tokenizer that serializes the image in one dimension are also expected.
\subsection{\rev{Impact of different $K\times m$ and more comparison with other large codebook reconstruction methods.}}\label{A:pixel-km}
\rev{We investigate the impact of semantic codebook size $K$ and sub-codebook size $m$ in Table \ref{tab:abl-km}. The results indicate that increasing either $K$ or $m$ generally enhances reconstruction fidelity (lower rFID).
However, excessive expansion leads to a significant decline in codebook usage.
Furthermore, when controlling for the total codebook size ($K\times m$), we observe that prioritizing a larger semantic codebook $K$ yields slightly better performance than increasing the sub-codebook size $m$.}
\rev{To further demonstrate the effectiveness of our method, we conduct a more detailed comparison with other approaches that use large codebooks.
As show in Tab.\ref{tab:a-lc}, we conduct comparisons on the reconstruction task under two codebook sizes (65K and 262K).
Under the 65K setting, our model surpasses the expert reconstruction model while maintaining good usage.
Under the 262K setting, our method still achieves competitive performance compared to the expert model.}