CAD-Tokenizer: Towards Text-based CAD Prototyping via Modality-Specific Tokenization
Extracted source text · arXiv 2509.21150
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
% \usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
% \usepackage{hyperref} % hyperlinks
% \usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{amsmath}
\usepackage{wrapfig}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{makecell}
\usepackage{multirow}
\usepackage{listings}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{cellspace}
\usepackage{svg}
\setlength{\cellspacetoplimit}{3pt}
\setlength{\cellspacebottomlimit}{3pt}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\lstdefinestyle{mystyle}{
backgroundcolor=\color{backcolour},
commentstyle=\color{codegreen},
keywordstyle=\color{magenta},
numberstyle=\tiny\color{codegray},
stringstyle=\color{codepurple},
basicstyle=\ttfamily\footnotesize,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
numbers=left,
numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=4
}
\lstset{style=mystyle}
% Defining commands
\newcommand{\model}{CAD-Tokenizer}
\newcommand{\tbcp}{text-based CAD prototyping}
\DeclareFontFamily{U}{stix2bb}{}
\DeclareFontShape{U}{stix2bb}{m}{n} {<-> stix2-mathbb}{}
\NewDocumentCommand{\indicator}{}{\text{\usefont{U}{stix2bb}{m}{n}1}}
% Defining colors
\usepackage{xcolor,colortbl}
\definecolor{Gray}{gray}{0.88}
\newcolumntype{a}{>{\columncolor{Gray}}c}
\title{CAD-Tokenizer: Towards Text-based CAD Prototyping via Modality-Specific Tokenization}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{
~Ruiyu Wang$^1$\thanks{Work done during the authors’ internship at Microsoft Research Asia.},
~Shizhao Sun$^2$\thanks{Correspondence: \texttt{shizsu@microsoft.com}},
~Weijian Ma$^3$,
~Jiang Bian$^2$\\
% ~\texttt{rwang@cs.toronto.edu}, \texttt{shizsu@microsoft.com}\\
~$^1$University of Toronto, $^2$Microsoft Research Asia, $^3$Fudan University
}
% \author{Antiquus S.~Hippocampus, Natalia Cerebro \& Amelie P. Amygdale \thanks{ Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
% funding agencies. Funding acknowledgements go at the end of the paper.} \\
% Department of Computer Science\\
% Cranberry-Lemon University\\
% Pittsburgh, PA 15213, USA \\
% \texttt{\{hippo,brain,jen\}@cs.cranberry-lemon.edu} \\
% \And
% Ji Q. Ren \& Yevgeny LeNet \\
% Department of Computational Neuroscience \\
% University of the Witwatersrand \\
% Joburg, South Africa \\
% \texttt{\{robot,net\}@wits.ac.za} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email}
% }
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Computer-Aided Design (CAD) is a foundational component of industrial prototyping.
where models are defined not by raw coordinates but by construction sequences such as sketches and extrusions.
This sequential structure enables both efficient prototype initialization and subsequent editing.
Text-guided CAD prototyping, which unifies Text-to-CAD generation and CAD editing, has the potential to streamline the entire design pipeline.
However, prior work has not explored this setting, largely because standard large language model (LLM) tokenizers decompose CAD sequences into natural-language word pieces, failing to capture primitive-level CAD semantics and hindering attention modules from modeling geometric structure.
We conjecture that a multimodal tokenization strategy, aligned with CAD’s primitive and structural nature, can provide more effective representations.
To this end, we propose CAD-Tokenizer, a framework that represents CAD data with modality-specific tokens using a sequence-based VQ-VAE with primitive-level pooling and constrained decoding.
This design produces compact, primitive-aware representations that align with CAD’s structural nature.
Applied to unified text-guided CAD prototyping, CAD-Tokenizer significantly improves instruction following and generation quality, achieving better quantitative and qualitative performance over both general-purpose LLMs and task-specific baselines.
\end{abstract}
\section{Introduction}
\input{sections/1-intro}
\section{Related Works}
\input{sections/2-related_works}
\section{Methodology}
\input{sections/3-methodology}
\section{Experiments}
\input{sections/4-experiments}
\section{Limitations}
\input{sections/5-limitations}
\section{Conclusion}
\input{sections/conclusion}
\section*{Ethics Statement}
The data used in this work is tailored for CAD generation.
Due to its specialized nature, the misuse risk is naturally minimized, ensuring the developed methods primarily benefit CAD development.
\section*{Reproducibility Statement}
We have included sufficient information on our all aspects of our experimentations including dataset resources and setups, model hyperparameters and LLM backbones, computational resources and setups, and metric setups.
Relevant detail can be found in Section \ref{sec:exp-setup} and Appendix \ref{sec:apdx:evaluation-setup}.
% We will release the source code after the peer-review procedure.
% \section*{The Use of Large Language Models}
% THe use of LLMs is restricted in revising the word choices and grammar only.
% No LLMs are used for research ideation or contributive enough to be regarded as a contributor.
\clearpage
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
\appendix
\input{sections/appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
% Computer-Aided Design (CAD) is crucial for industrial prototyping and production.
% It enables the creation of 3D shapes through sequential constructions, such as sketches and extrusions, which CAD kernels compile into machine-interpretable 3D objects.
% Compared to other 3D representations, CAD facilitates initial prototype creation and subsequent editing.
% Autonomous CAD sequence generation, particularly text-guided approaches, holds significant promise for enhancing the design process.
Computer-Aided Design (CAD) plays a central role in industrial prototyping and production.
CAD models are constructed as sequences of operations—such as sketches and extrusions—that CAD kernels compile into machine-interpretable 3D objects.
Unlike other 3D representations, CAD explicitly encodes design history, which makes it particularly suitable for both creating initial prototypes and performing subsequent modifications.
Automating CAD sequence generation, especially from natural language instructions, could substantially accelerate the design workflow.
Recent work has begun to explore text-guided CAD subtasks, including CAD editing \citep{cadeditor} and text-to-CAD generation \citep{cadfusion, cadllama, text2cad}.
These studies not only demonstrated the feasibility of each individual task but also showed that large language models (LLMs) can effectively operate on CAD sequences.
However, real-world design is rarely a one-shot process: engineers iteratively refine prototypes by alternating between creation and modification.
An effective text-based CAD system should therefore support both tasks in a unified manner—generating CAD objects from instructions while also allowing editing of existing designs.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{images_new/fig1-overview.pdf}
% \includesvg[width=\linewidth]{images_new/fig1:overview.svg}
\caption{Overview of the \textbf{unified} text-based CAD prototyping task, which includes the Text-to-CAD generation and text-based CAD Editing. Our approach introduces a CAD-specific tokenization method that produces primitive-level tokens instead word pieces. This paradigm allows LLMs to capture relationships among CAD primitives more effectively, improves compression and performance.}
\label{fig1:task-overview}
\end{figure}
Handling both tasks within a single unified model has never been studied previously.
The difficulty lies in the distinct requirements of the two subtasks.
Text-to-CAD demands strong \textit{generative} capability to produce high-quality sequences from scratch, whereas CAD editing requires precise instruction \textit{understanding} and alignment with existing geometry.
A unified framework must therefore master two complementary but non-overlapping skills, which places significantly higher demands on the backbone than solving either task alone.
We introduce \model, the first framework to tackle unified text-based CAD prototyping with CAD-specific tokenization.
Our key insight is that the default tokenizers used in LLMs are poorly matched to CAD data.
They decompose CAD programs into natural-language word pieces, fragmenting operations and numeric parameters into arbitrary substrings (e.g., \texttt{"[extru] [-sion]"}).
This causes attention layers to focus on punctuation or partial tokens instead of meaningful primitives, limiting the model’s ability to capture structural and geometric dependencies.
To address this, we use a VQ-VAE to compress CAD sequences into primitive-level tokens before training, enabling LLMs to directly model inter-primitive relations and predict the next operation rather than the next character or word piece (Figure \ref{fig1:task-overview}).
This formulation makes unified training possible, leading to substantial improvements in both efficiency and quality.
Our contributions are as follows:
\begin{enumerate}
\item We develop a primitive-level VQ-VAE tokenizer that maps sketch–extrusion pairs into multiple discrete tokens. To integrate with LLMs, we train adapters that align the pretrained tokenizer vocabulary with the backbone’s embeddings.
\item We fine-tune an LLM on these CAD-specific tokens under a unified setup covering both Text-to-CAD generation and CAD editing. This design improves training efficiency and surpasses strong task-specific baselines.
\item We leverage the formal grammar of CAD by introducing a finite-state automaton–based sampling strategy, which enforces valid syntax in generation and improves quality.
\item Through extensive qualitative, quantitative, and ablation studies, we demonstrate that \model\ improves both efficiency and performance across unified CAD prototyping tasks.
\end{enumerate}
\subsection{CAD Modeling}
In general, CAD modeling tasks take different types of user inputs and generate sequential CAD output that renders into 3D objects by 3D kernels.
\textbf{Classical CAD Tasks.}
Classical CAD tasks focus on generating CAD sequences from non-textual inputs.
This includes hand sketches \citep{vitruvion}, point clouds \citep{cadsignet, stepbystep}, sequence prefixes \citep{skexgen, hnc}, or even random noise \citep{deepcad}.
There are also tasks involving masked variation, where some primitives are masked out and converted into random variants \citep{flexcad, skexgen, hnc}.
Our work differs from these works in two ways.
First, our work focuses on CAD generation from textual inputs. Second, while works in these categories often use transformer encoders to extract features and decode specific IDs, they do not involve texts or natural languages and thus have no need to leverage the textual understanding capability of LLMs and hence do not need to align these IDs with them.
\textbf{Text-based CAD Tasks.}
Text-based CAD tasks focus on generating CAD sequences based on textual instructions.
Inputs for these tasks fall into two main types: the first type includes only textual instructions on how to generate an initial CAD model, and the second type includes a CAD sequence (of the same representation format) and a textual instruction on how to modify the given model.
The former is the Text-to-CAD task \citep{cadfusion, cadtranslator, cadllama, text2cad} and the latter is the CAD-editing task \citep{cadeditor}.
Our work differs from these tasks as we propose the \tbcp \ task, which merges them.
Given that initiating and editing CAD models is a standard procedure in industry, a single-model solution is both necessary and promising for the prototyping pipeline.
\subsection{LLM Tokenization}
LLMs operate over discrete tokens that are mapped into hidden embeddings.
In natural language, these tokens are defined by vocabularies \citep{glove, word2vec} or obtained via algorithms such as byte-pair encoding (BPE) \citep{bpe}.
In multimodal tasks where inputs often lie outside the space of predefined vocabularies, external encoders and decoders such as VAEs and VQ-VAEs \citep{vae, vqvae}, vision transformers \citep{vit, clip}, or diffusion-based algorithms \citep{sdxl} are commonly used to map continuous modalities into token sequences that LLMs can process.
Tokenization strategies vary across domains.
For visual tasks, pretrained VAE/VQ-VAE or CLIP/ViT encoders are typically used, sometimes paired with custom decoders when generative outputs are required \citep{transfusion, SEED, SEEDX, llava}.
In robotics, many works bypass neural tokenizers, instead stringifying actions for direct use with default LLM tokenizers \citep{rt1, rt2}, or applying lightweight algorithmic encodings \citep{fast}.
Time-series modeling has explored both neural tokenizers based on attention architectures \citep{PatchTST} and classic tokenization methods \citep{bridge}.
Prior CAD works are task-specific on tokenizations.
Classical CAD generation methods train separate encoders and decoders for different inputs, such as sketches and extrusions \citep{skexgen, hnc, text2cad, cadsignet}, resulting in multi-encoder pipelines.
Recent text-based CAD efforts typically rely on off-the-shelf LLM tokenizers, directly applying LLM tokenizations to CAD sequences without adaptation \citep{cadfusion, cadeditor, flexcad}.
In contrast, our work introduces the first encoder–decoder tokenizer specifically designed to integrate with LLMs.
We employ a single VQ-VAE–variant encoder–decoder pair to tokenize all CAD inputs and align its vocabulary with the LLM backbone.
This unified tokenization allows the model to reason over CAD primitives as discrete units, addressing the fragmentation issues inherent in natural-language tokenizers.
% Tokenizers are widely used in visual alignment tasks as images do not readily lend themselves to vocabulary lookups.
% Common feature extractor options include traditional VAE blocks \citep{transfusion} and CLIP/ViT encoders \citep{clip, SEED, SEEDX, llava}.
% They transform representations from different modalities into a format recognizable by LLMs.
% Previous works that require LLM generation capability also train a variety of reconstructors to form a VAE- \citep{vae} or VQ-VAE-like \citep{vqvae} architecture, depending on whether a quantization layer is added to convert representations into discrete codebooks.
% The decoders usually involve latent diffusion modules \citep{sdxl, SEEDX} or transformer-based modules such as Q-Formers \citep{vit, SEED}. They convert the special tokens generated by the LLM back to the original modality.
% In areas of sequential data such as robotics and time series, some works use similar methods to patchify inputs into specific tokens \citep{PatchTST}, while the common practice still stringifies input into ordinary strings and sends them to the LLM backbone \citep{rt1, rt2, bridge}.
% In some special cases, such as when the backbone model cannot handle all numeric values, the inputs are converted into tokens that are rarely used in that specific backbone version \citep{rt2}.
% Previous works in CAD follow the same practice \citep{cadfusion, cadeditor, flexcad}.
% While some works involve training a VQ-VAE for specific IDs \citep{skexgen, hnc}, these are used for CAD-specific frameworks rather than a generic platform, and the hierarchical generation procedure increases complexity and further hinders broader application.
% Our work involves both understanding and generation and adapt a VQ-VAE architecture to train encoders and decoders simultaneously.
% Unlike previous works, we employ a sequential VQ-VAE that operates in a more generic scenario.
% That is to say, we anticipate our CAD-specific tokens will be invariant to different primitive types.
% Moreover, we are the first work to explore the potential of CAD-specific tokens with LLMs.
% Compared to using task-specific decoders, LLMs provide stronger prior knowledge from text to address the \tbcp \ problem, despite the greater challenges in aligning modalities.
\subsection{Approach Overview}\label{sec:approach-overview}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images_new/fig2-methods.pdf}
% \includesvg[width=\linewidth]{images_new/fig2:method.svg}
\caption{
Overview of the \model \ framework. (1) A primitive-based VQ-VAE tokenizes CAD sequences into discrete primitive tokens. (2) Adapter modules align these tokens with the LLM embedding space. (3) The backbone LLM is fine-tuned using CAD-specific tokens. (4) At inference, an FSA-guided sampling strategy ensures syntactically valid CAD generation.
}
\label{fig2:method}
\end{figure}
% The \tbcp \ task is a unified formulation of Text-to-CAD generation and CAD editing.
% Formally, given a textual prompt $x=(\mathcal{I}, C_{\texttt{orig}})$ containing an instruction $\mathcal{I}$ and an optional original sequence $C_{\texttt{orig}}$, the goal is to learn a function $f(\cdot)$ that transforms the input prompt $x$ into a parametric sequence $C_{\texttt{gen}}$.
% For Text-to-CAD tasks, $C_{\texttt{orig}}=\emptyset$, while for editing tasks $C_{\texttt{orig}}$ is a valid input CAD sequence.
% The generated output should faithfully reflect the design intent conveyed by the non-empty elements of the prompt.
The \tbcp \ task unifies Text-to-CAD generation and CAD editing.
Formally, given a prompt $x = (\mathcal{I}, C_{\texttt{orig}})$ consisting of an instruction $\mathcal{I}$ and an optional original sequence $C_{\texttt{orig}}$, the goal is to learn a mapping $f(\cdot)$ that outputs a parametric sequence $C_{\texttt{gen}}$.
For Text-to-CAD, $C_{\texttt{orig}} = \emptyset$, while for editing tasks, $C_{\texttt{orig}}$ is a valid CAD sequence.
The generated sequence should reflect the design intent expressed by the instruction and, when applicable, the original shape.
% \model \ introduces a framework that approaches this problem by fine-tuning LLMs with pre-trained CAD-specific tokens (Figure \ref{fig2:method}).
% The initial tokens are yielded from a primitive-based VQ-VAE, which is pre-trained and aligned with LLMs on the CAD sequence data (Section \ref{sec:tokenization}).
% Then, the VQ-vAE module tokenizes the CAD sequence in the training sets into CAD tokens, that the LLMs can directly use for training without running-time adaptations (Section \ref{sec:sft}).
% Finally, during the inference time, we boost the framework's performance by converting CAD formatting into a Finite-State Automaton and use it to guide the sampling procedure (Section \ref{sec:inference}).
\model \ addresses this unified task by fine-tuning LLMs with CAD-specific tokens produced by a pretrained VQ-VAE (Figure~\ref{fig2:method}).
Specifically, (1) a primitive-based VQ-VAE is trained on CAD sequences to compresses the sketch-extrusion pairs into discrete tokens, and adapters are deployed to align these tokens with the backbone’s embedding space (Section \ref{sec:tokenization}), (2) the backbone is fine-tuned on the encoded CAD data (Section \ref{sec:sft}), and (3) an FSA-based decoding strategy enforces CAD grammar during inference (Section \ref{sec:inference}).
We describe each step below.
\subsection{Primitive-based VQ-VAE as CAD Tokenizer} \label{sec:tokenization}
\textbf{Pretraining Primitive VQ-VAE.}
To convert CAD sequences into compact symbolic representations, we modify and pretrain a VQ-VAE variant that outputs primitive-level tokens.
Each input sequence $C$ needs to be decomposed into sketch–extrusion pairs $SE = \{t_1, \dots, t_n\}$ and encoded into latent vectors $\{p'_1, \dots, p'_k\}$, with $k < n$, in pairs by transformer encoders.
Unlike standard VQ-VAEs that pool the entire sequence into a single latent vector, we introduce a primitive-specific pooling layer, producing multiple discrete representations for each sketch–extrusion pair.
This design captures both local and contextual information from the surrounding sequence.
Training follows a sequence reconstruction objective with an accumulated VQ loss across all pooled tokens:
$$\mathcal{L}_{VQ\text{-}Prim} = \sum_{i\in n} \text{EMD}(Decoder(\{p'\}, t_{1, i-1}), \indicator_{i}) + \sum_{j\in k}VQ(\overline{T}^E_{j}, p'_j),$$
where EMD is the squared Earth Mover's Distance Loss,
% $T^E_i$ denotes the encoded representation of the $i$-th token,
and $\overline{T}^E_{j}$ denotes the pooled vector of the tokens construct up the $j$-th primitive.
We demonstrate this method in Figure \ref{fig2:method}(1) and provide additional details such as the detailed pooling procedure in Appendix \ref{sec:apdx:vq}.
% Our framework requires converting the CAD sequences into a series of primitive-specific tokens.
% We adopt a VQ-VAE with the pooling strategies modified in order to achieve the capabilities efficiently.
% We split an input sequence $C$ into sketch-extrusion pairs $SE=\{t_{1}, t_{2}, ..., t_n\}, t\in\texttt{V}$, and yield the raw primitive tokens by $Encoder(SE) = \{p'_1, p'_2, ... p'_k\}, k\leq n$.
% Training the primitive VQ model is nearly the same as the traditional one except an accumulated VQ loss across all k primitive tokens, since more tokens are generated:
% $$\mathcal{L}_{VQ\text{-}VAE} = \sum_{i\in n} \text{EMD}(Decoder(\{p'\}, T^E_{1, i-1}), \indicator_{i}) + \sum_{j\in k}VQ(\overline{T}^E_{j}, p'_j),$$
% where EMD is the squared Earth Mover's Distance Loss, $T^E_i$ denotes the embedded representation of the $i$-th token, and $\overline{T}^E_{j}$ denotes the pooled vector of the tokens construct up the $j$-th primitive.
% We demonstrate this method in Figure \ref{fig2:method}(1) and provide additional details in the Appendix \ref{sec:apdx:vq}.
\textbf{Aligning CAD Tokenizer with LLM Embeddings.} \label{sec:alignment}
% fig 2.2
The raw primitive vectors $\{p'_j\}$ have dimension $d_{vq}$, while the LLM requires tokens in its embedding space of dimension $d_{tok}$.
Typically, this adaptation is performed during co-training with LLM backbones \citep{llava}, which is compute-intensive and does not guarantee alignment in the reverse direction.
We introduce a novel use of adapters that map VQ-VAE outputs and LLM embeddings bi-directionally.
In detail, we leverage the frozen \textit{embedding} and \textit{logit} layers of the pretrained LLM. \vspace{-2px}
Adapters are trained with a vector reconstruction loss that encourages the mapped tokens $\{p_j = \text{argmax}(L_{logit}(W^{d_{vq}}_{d_{tok}}(p'_j)))\}$ to remain faithful to their raw representations:
$$\mathcal{L}_{\text{recon}} = \sum_{j\in k} \| \hat{p}'_j - p'_j \|_2^2.$$
$\hat{p}'_j = W^{d_{tok}}_{d_{vq}}(L_{embed}(p_j))$ is the reconstructed embedding corresponding to $\{p'_j\}$.
This alignment produces native LLM-recognizable primitive IDs, enabling seamless integration without modifying the backbone.
The alignment procedure is illustrated by Figure \ref{fig2:method}(2).
% From the yielded raw primitive sequence $\{p'_1, ..., p'_k\}$ for each sketch-extrusion pair, our next goal is to convert the raw primitives (in dimension $d_{vq}$) into LLM-recognized primitive tokens $\{p_1, ..., p_k\}$ of dimension $d_{tok}$, so that we can fine-tune the LLM backbone with recognizable inputs.
% The alignment procedure is illustrated by Figure \ref{fig2:method}(2).
% We introduce two pretrained layer from the backbone into our VQ framework: $L_{logit}$ is the logit layer, and $L_{embed}$ is the embedding layer.
% They are fixed and frozen in our setup to ensure alignment, so that by connecting adapters with encoder we produce native LLM primitive IDs, \vspace{-1px} i.e. for a primitive raw token
% \vspace{-1px}$p'_j, j\in k$,$p_j = \text{argmax}(L_{logit}(W^{d_{vq}}_{d_{tok}}(p'_j))),$\vspace{-1px}
% where $W^{d_{vq}}_{d_{tok}}$\vspace{2px} is a trainable weight matrix that expands the initial $p'_j$ vector into the dimension of the backbone LLM.
% \vspace{-1px}A downsampling matrix is also deployed so that we can retrieve the appoximation of the original vector
% \vspace{-1px}$\hat{p}'_j = W^{d_{tok}}_{d_{vq}}(L_{embed}(p_j))$
% and use them for decoding.
% The two matrices are optimized by the vector reconstruction loss
% $$\mathcal{L}_{\text{recon}} = \sum_{j\in k} \| \hat{p}'_j - p'_j \|_2^2.$$
\subsection{Instruction Tuning with CAD Tokens} \label{sec:sft}
% fig 2.3
With aligned tokens, CAD sequences can be encoded into compact symbolic forms and directly used for instruction tuning.
Given an input prompt $x = (\mathcal{I}, C_{\texttt{orig}})$ and target sequence $C_{\texttt{gen}}$, we define
$x' = (\mathcal{I}, \texttt{CAD-Encoder}(C_{\texttt{orig}}))$ and
$y' = \texttt{CAD-Encoder}(C_{\texttt{gen}})$.
We fine-tune the pretrained LLM by minimizing the standard cross-entropy loss between the generated tokens $\hat{y} = f(x')$ and ground-truth $y'$ as described below:
\begin{equation*}
\mathcal{L}_{SFT} = -\mathbb{E}_{(x',y')}
\left[
\frac{1}{T}\sum_{t=1}^T\log p(\hat{y}=y'_t|x')
\right].
\end{equation*}
Because CAD sequences are significantly compressed, i.e., $|x'| < |x|$, fine-tuning is not only more effective but also more computationally efficient.
Figure~\ref{fig2:method}(3) illustrates this stage.
% As LLM primitive tokens are computed, we are able to encode the initial sequence tokens into our new CAD-specific tokens, and decode them back to the sequence reversely.
% For simplicity, we define $\texttt{CAD-Encoder}$ as the splitting process and combination of all modules until ID extraction, so that $\texttt{CAD-Encoder}(C) = \{p_{k^+}\}$.
% Given the problem input $x = (\mathcal{I}, C_{orig})$ and label $C_{gen}$, we transform the input sequence representations as $x'=(\mathcal{I}, \texttt{CAD-Encoder}(C_{orig}))$ and fine-tune the pre-trained LLMs by minimizing the discrepancy between the model generation $\hat{y} = f(x')$ and the encoded CAD ground truth $y' = \texttt{CAD-Encoder}(C_{gen})$ using the cross-entropy loss:
% \begin{equation*}
% \mathcal{L}_{SFT} = -\mathbb{E}_{(x',y')}
% \left[
% \frac{1}{T}\sum_{t=1}^T\log p(\hat{y}=y'_t|x')
% \right],
% \end{equation*}
% Where $T$ is the encoded sequence length and $p(\cdot)$ is the probability of the $t$-th token by $f(\cdot)$.
% Note that the training speed improvement stems from that the encoded sequence is strictly shorter than the original length, i.e. $|x'| < |x|$.
% The fine-tuning procedure is illustrated by Figure \ref{fig2:method}(3).
\subsection{Model Inference and Token Decoding} \label{sec:inference}
During inference, naive autoregressive sampling may produce invalid CAD sequences, as standard LLM decoding strategies (e.g., top-$p$, beam search) are unaware of CAD grammar.
However, unlike natural language, CAD sequences are formal languages and can be fully described by states and transitions.
Therefore, we can improve the sample quality by designing a finite-state automaton (FSA) that formalizes valid CAD construction rules (Figure~\ref{fig2:method}(4)).
At each step, the FSA provides a series of masks restricting the candidate logits to grammar-compliant tokens, ensuring syntactic validity.
The automaton state evolves with the decoder, whose choice for the next token is also passed to the automaton for deciding if a transition is going to take place.
Despite some context-dependent geometric constraints are unable to fully catched, this method significantly reduces syntactic errors and improves sampling robustness by preventing obvious grammatical mistakes.
Further implementation details are given in Appendix~\ref{sec:apdx:sampling} and the FSA design is illustrated in Figure \ref{fig:fsa}.
% The classic LLM inference paradigm uses a top-p sampling or beam search.
% However, unlike natural language, CAD sequences are formal languages and can be fully described by states and transitions\footnote{As this grammar does not enforce geometric constraints, adherence to it does not eliminate rendering invalidity.}.
% Therefore, we convert our CAD sequence construction rules into a Finite-State Aumaton (FSA), and propose a augmented sampling method for our \texttt{CAD-Decoder} to improve its sampling performance.
% An overview is provided by Figure \ref{fig2:method}(d).
% Inspired by \citet{xgrammar}, we restrict the decoder to only select the next token from logits eligible according the grammar.
% The grammar is decided by a preset FSA, which generate multiple masks that removes the logits that its current state decides ineligible.
% During decoding, the FSA is first reset to the initial state, and transit the states by the outcome of the last logit it received from the decoder.
% For example, a \texttt{[Primitive-start]} mask restricts the decoder to choose among the three curve primitive tokens (\texttt{line}, \texttt{arc}, \texttt{circle}), but the FSA's state changes accordingly with the choice that the decoder makes.
% This dual influence would not stop until a stop condition is met, e.g. the max sequence length is approached.
% We include the detailed design of the FSA, the content of masks, and the sampling algorithm in Appendix \ref{sec:apdx:sampling}.
% During inference, the fine-tuned LLM generates primitive tokens using top-p sampling.
% In traditional tokenization approaches where LLMs directly generate sequential CAD tokens, the LLM must simultaneously handle both CAD-specific knowledge and sequence formatting.
% In \model, we separate these two responsibilities, implicitly assigning the VQ-VAE decoder responsibility for primitive formatting, i.e., transforming primitive token IDs into structured representations like \texttt{line,x,y,<curve\_end>}.
% This poses challenges for the capability of the VQ-VAE decoder, which has significantly fewer parameters compared to modern LLMs.
% Inspired by \citet{xgrammar}, we propose a constrained sampling method for our \texttt{CAD-Decoder} to improve its sampling performance by enforcing adherence to formatting structures.
% Since CAD sequences can be represented by a context-free grammar (CFG)\footnote{As this CFG does not enforce geometric constraints, adherence to it does not eliminate rendering invalidity.}, we create a series of logit masks that restrict the decoder to choosing only from valid CAD raw tokens at each step.
% The choice of mask is determined by a Finite-State Automaton (FSA) whose state evolves with the sampling progress.
% In each sampling step, the decoder accesses the automaton's current state, which provides a mask to guide the selection from the output logits.
% The selected token is then passed to the automaton as an action, which updates the automaton's internal state according to predefined transition rules.
% We include the detailed design of the FSA and the sampling algorithm in the Appendices.
% \model \ requires a CAD-specific tokenizer that summarizes the original CAD sequence that may contain reduntant information that distracts the backbone model thus hinders capability.
% A CAD sequence can be defined by a series of tokens in the vocabulary $\texttt{V}$, where they naturally cluster into $n$ primitives that draw sketches or construct extrusions, i.e. $C=\{t_{1,1}, t_{2, 1}, ..., t_{i_1, 1}, ..., t_{i_n, n}\}, t\in\texttt{V}$ where $i_k, k\in n$ denotes the number of tokens that construct up primitive $k$.
% Our objective is to transform the sequence into raw primitive tokens $\{p'_1, p'_2, ..., p'_n\}$ such that $p'_k, k\in n$ encodes the same meaning as its original tokens $\{t_{1, k}, ..., t_{i_k, k}\}$.
% We train a VQ-VAE variant to address this problem.
% The model includes a transformer encoder, a primitive-specific pooling, a VQ layer and a decoder.
% For primitive $k$'s $i_k$-th token $t_{i_k, k}$, we pass it through a 512-dim embedding layer: $T_{i_k, k} = W_{emb}^{|\texttt{V}|\times 512}$.
% Its output from encoder $E$, $T^E_{i_k, k} = E(T_{i_k, k})$, is then max pooled among each primitive section as:
% $$\overline{T^E_{k}} = \texttt{MaxPool}(T^E_{1, k}, ..., T^E_{i_k, k}).$$
% Subsquently, we apply a standard vector quantization procedure where we obtain the 512-dim codebook vector $p'_k$.
% In detail, we perform a nearest neighbor lookup by comparing the Euclidean distance between code vector $\textbf{v}\in\textbf{V}$ and the pooled vector $\overline{T^E_{k}}$:
% $$p'_k \leftarrow \text{argmin}_{\textbf{v}\in\textbf{V}}||\overline{T^E_{k}} - \textbf{v}||^2.$$
% The decoder $D$ is also a standard transformer decoder with multiple decoder layers performing self-attention and cross-attention.
% Here, the codebook-generated primitive sequence $\{p'_k\}$ acts as the source sentence and the embedded tokens $\{T_{i_k,k}\}$ are the target sequence.
% After all, we compute the loss as the combination of the squared Earth Mover's Distance Loss and the standard VQ loss:
% $$L_{VQ\text{-}VAE} = \sum_{i_k\in k}\sum_{k\in n} \text{EMD}(D(\{p'\}, T^E_{[1, (i_k, k))}), \indicator_{i_k, k}) + \sum_{k\in n}VQ(\overline{T}^E_{k}, p'_k),$$
% where $T^E_{[1, (i_k, k))}$ denotes the terms before $T^E_{i_k, k}$. The former term measures the reconstruction quality of the token sequence and the latter computes the error generated by the VQ operation.
\subsection{Experimental Setups} \label{sec:exp-setup}
\textbf{Datasets.}
For training the VQ-VAE, we convert the SkexGen \citep{skexgen} dataset into the formatting used by CADFusion \citep{cadfusion} and CAD-Editor \citep{cadeditor}.
We include the formatting details in the Appendix \ref{sec:apdx:representation}.
To prevent a distribution gap for the tokenizer between seen and unseen data, we use half of the training data for VQ-VAE training, ensuring the LLM also encounters samples not seen by the VQ-VAE encoder during its training.
For training the backbone LLM, we use the datasets of CADFusion and CAD-Editor.
Although both originate from SkexGen, the two datasets have different content, so we merge them for training our model without further deduplication.
Since CAD-Editor uses the original split of SkexGen but the entire CADFusion dataset is split from its training set, we consolidate the CADFusion dataset and additionally annotated 200 pairs for our test set.
We also balance this combined dataset by randomly selecting five CAD-Editing data points for every Text-to-CAD data point.
Ultimately, this results in approximately 100k training pairs and 1000 testing pairs.
\textbf{Implementation Details.}
The CAD sequence formalization and primitive selection is detailed in Appendix \ref{sec:apdx:representation}.
The VQ-VAE is trained with 5 encoder and 5 decoder layers, a codebook size of 2048, and an internal dimension of 512. It is trained with a learning rate of 3e-4 for up to 250 epochs, using early stopping with a patience of 25.
The logit and embedding layers inserted are from \texttt{LLaMA-3-8b-Instruct} (4096-dim), which is also used as our LLM backbone.
The alignment uses exactly the same setup, but the encoder, decoder and LLM layers are frozen, and only the two adapter layers are trained.
For LLM training, the backbone is configured with a maximum token length of 1024 and a hidden dimension of 4096. We adopt Low-Rank Adaptation (LoRA) \citep{lora} for efficient fine-tuning, with a rank of 32 and an $\alpha$ value of 32. The model is trained for 20 epochs with a learning rate of 1e-4 and the AdamW optimizer.
The VQ-VAE training is run on a single NVIDIA A100-80G GPU, and the LLM fine-tuning is conducted on four NVIDIA A6000-48GB SMX GPUs using Distributed Data Parallel (DDP).
\textbf{Baselines.}
We select two types of baselines.
First, we include \textbf{CADFusion} and \textbf{CAD-Editor} as state-of-the-art methods in Text-to-CAD and CAD-editing tasks, respectively.
However, since these are for individual text-based tasks only and no existing works address \tbcp, we propose two additional baselines for the unified task.
First, we train a vanilla LLM model using the native LLaMA tokenizer (\textbf{Vanilla-LLaMA}), with all other aspects of the setup identical to \model.
Second, we prompt a five-shot \textbf{GPT-4o} to examine how modern commercial LLMs perform on the \tbcp \ task.
Details including baseline implementation and prompting samples can be found in the Appendix \ref{sec:apdx:evaluation-setup}.
\textbf{Metrics.}
Our evaluation focuses on different aspects of the generated CAD models.
First, we measure the pairwise correspondence between generated outputs and the ground truth sequences.
For this criterion, we use \textbf{F1} scores and Chamfer Distance (\textbf{CD}).
We follow CADFusion's \citep{cadfusion} measurements of \textbf{F1-Sketch} and \textbf{F1-Extrusion} for simplicity.
Second, we measure distributional similarity.
Between the generated samples and the ground truths, we follow \citep{skexgen, cadeditor, cadfusion} and measure: Coverage (\textbf{COV}), which quantifies how well the generated distribution covers the ground truth; Minimum Matching Distance (\textbf{MMD}), which evaluates similarity by finding the closest matches between samples from the two distributions; and Jensen-Shannon Divergence (\textbf{JSD}) for distributional similarity.
Third, we measure the Invalidity Ratio (\textbf{IR}) as an indicator of sequence robustness; and fourth, we measure the generated outputs' alignment with instructions using a VLM-based score (\textbf{VLM}) and Human Evaluation (\textbf{HE}).
The VLM is prompted to rate instruction following on a scale of 0 to 10, and human judges are asked to rank the outputs from different models based on the same criterion.
Details on VLM prompting and human evaluation methodology can be found in Appendix \ref{sec:apdx:evaluation-setup}.
For the clarity of the presentation, the results of the distribution scores (\textbf{COV, JSD, MMD}) and chamfer distance in all records are multiplied by $10^2$.
\subsection{Main Results} \label{main-results}
\input{sections/4x-quantative_table}
\textbf{Quantitative Evaluation.}
Table~\ref{tab:quantitative} reports results on CAD editing (a) and Text-to-CAD generation (b).
On CAD editing, \model\ achieves the best results across almost all metrics. It improves the F1 scores by around 10 points each.
Compared to the task-specific CAD-Editor, \model\ leads in F1 metrics while also achieving stronger VLM and human evaluation scores.
Moreover, discoveries detailed in the supplementary results of Appendix \ref{sec:apdx:editing-distribution-metric-problem} suggest that our model is best intended to edit the original object despite being disfavored by the distributional metrics by doing so.
On Text-to-CAD generation, \model\ again shows clear improvements.
It significantly improves the F1 scores, indicating strong matching in CAD sequential information.
Chamfer Distance is also decreased by a large margin.
Although CADFusion attains a slightly higher VLM score (5.41 vs. 3.82), \model\ surpasses it in nearly every other metric, including a lower human evaluation rank that indicates preference.
Besides the advantage of our framework over task-specific baselines, it is noteworthy that {Vanilla-LLaMA} almost collapses on the majority of metrics.
This supports our conjecture that \tbcp \ is inherently a challenging task for conventional tokenizers, and highlights that our improvements over the vanilla tokenizer and training paradigm are substantial.
\input{sections/4xxa-ablation-pre-sft}
% Table \ref{tab:quantitative} summarizes the quantitative results comparing \model \ with the baseline methods.
% On the Text-to-CAD task, our model demonstrates significant advantages on all baselines, including better F1, distribution, and qualitative scores.
% On CAD-Editing, it consistently performs better on the sequence-level metrics, where it leads the current state-of-the-art by a large margin.
% On the other metrics it is still on par with or better than the task specific model.
% Supplementary results in Appendix \ref{sec:apdx:editing-distribution-metric-problem} suggest that our model is best intended to edit the original object despite being disfavored by the distributional metrics by doing that.
% It is noteworthy that our model outperforms the vanilla-LLaMA training strategy on all metrics and all aspects, demonstrating our advanced strategy; the improvement from GPT-4o generation suggests the huge space of improvement in the industrial pipeline that uses general-purpose commercial models.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images_new/fig3-qualitative.pdf}
\caption{Qualitative results. The upper and lower sections display results for the Text-to-CAD and CAD editing tasks, respectively. In each subfigure, the top-left corner shows the input instruction and CAD object (or only the instruction in Text-to-CAD), the top-right corner shows the ground truth, and the bottom three show outputs from CADFusion/CAD-Editor, Vanilla-LLaMA, and \model.}
\label{fig:qualitative}
% \vspace{-10px}
\end{figure}
\textbf{Qualitative Evaluation.}
Figure\ref{fig:qualitative} illustrates typical generations of our model and selected baselines.
In Text-to-CAD, \model\ produces more faithful multi-face objects compared to Vanilla-LLaMA, which often drops faces or misaligns structures.
Also, the outputs are natural and balanced, which resemble human designs.
Its qualitative performance is competitive with CADFusion, the task-specific baseline.
For CAD editing, \model\ demonstrates stronger instruction-following: it modifies existing shapes as required rather than persisting the original form, a failure mode described in Appendix \ref{sec:apdx:failure}.
Compared to the vanilla tokenizer, \model\ better respects structural templates and achieves higher-quality modifications.
Additional results including more complex prompts and designs can be found in Appendix \ref{sec:apdx:additional-qualitive}.
% Figure \ref{fig:qualitative} shows the qualitative results of our framework compared to the vanilla tokenizer and either CAD Editor or CADFusion, depending on the task being tested.
% In Text-to-CAD generation, our model demonstrates good understanding capabilities.
% It consistently outcompetes the vanilla tokenizer and achieves performance comparable to CADFusion, the single-task baseline.
% In some complex cases that require generating multiple-face objects, our model is less likely to lose track of the number of faces.
% This demonstrates the advantage of the primitive-based tokens on enabling LLMs capture more complex patterns.
% In the CAD editing task, our model demonstrates strong instruction-following capabilities, reflected in its ability to modify the input shape as instructed, rather than merely persisting the original form.
% Even CAD-Editor occasionally ignores instructions and sticks with the original shape, which is a sign of failure as discussed.
% Moreover, compared to the vanilla tokenizer, \model achieves significantly better understanding on the sequence template and also generation quality.
\subsection{Ablation Studies}
% 1. pre sft: our model being superior on both reconstruction quality and compression rate.
% 2. in sft: our model performs significantly better than other tokenization methods
% 3. post sft: the fsa sampling method we proposed is advantaged comparing to the classic, mainstream methods.
We analyze the contribution of each component in \model\ across three stages: pre-SFT tokenization, in-SFT integration with the LLM, and post-SFT sampling.
\textbf{Before LLM Finetuning: the Quality of Tokenizers.}
The tokenizer quality is critical to downstream LLM performance.
We evaluate two aspects: (i) \textbf{reconstruction quality}, in terms of selected sequence and distribution metrics, and (ii) \textbf{compression ratio} for efficiency.
We compare \model\ against the following:
1. \textbf{HNC-CAD} \citep{hnc}, a strong neural coding baseline on conditional and unconditional CAD generation,
2.\textbf{BPE} \citep{bpe}, the Byte-pair Encoding, a standard tokenization algorithm,
3. \textbf{\model} \textit{(curve)}, our default tokenizer with curve-based pooling,
4. \textbf{\model} \textit{(loop)}, a variant pooling loops instead, and
5. \textbf{\model} \textit{(single)}, which omits primitive pooling and encodes each sketch–extrusion pair as a single unit.
Table\ref{tab:ablation-pre-sft} and Figure \ref{fig:ablation-compression} presents the results.
Except that BPE is not involved in the reconstruction evaluation as it guarantees returning original vectors algorithmically, the exploration is three-fold:
First, all \model\ variants outperform prior methods on both reconstruction quality and compression.
The \model \ variants perform uniformly better than other tokenization methods, while having better compression rate.
Second, within \model, the results also highlight the trade-off between compression rate and the reconstruction accuracy, as the variants that compress more tokens into one embedding tend to have worse quality after reconstruction.
Third, the sharp degradation of the \textit{single} variant underscores the importance of primitive-level pooling.
% Table \ref{tab:ablation-pre-sft} and Figure \ref{fig:ablation-compression} presents the results.
% The variants of \model \ perform uniformly better than the previous works in both the reconstruction quality and the compession ratio.
% Within the variations, it is generally a trade-off between compression and reconstruction, while the drastic performance collapse of \textbf{\model} \textit{single} indicates the necessity of the primitive pooling.
\input{sections/4xxb-ablation-in-sft}
\textbf{In LLM Finetuning: the Performance of Different Tokenizion Methods.}
We next evaluate the tokenizers during supervised fine-tuning (SFT) with the LLM backbone.
All methods are included except {HNC-CAD}, which was not designed for LLM training.
As shown in Table \ref{tab:ablation-in-sft}, both BPE and the \textit{single} variant of \model\ perform poorly, with high invalidity ratios and consistently weak metrics.
Their setups justify the unsatisfactory performance: the \textit{single} variant performed already poor enough during the pre-finetuning reconstruction evaluation; and BPE does not align the post-tokenization tokens with the LLM vocabulary, which adds up additional burden during finetuning.
The \textit{loop} variant achieves performance close to the default \textit{curve} tokenizer, with slightly lower reconstruction but higher compression.
This not only indicates that primitive pooling—whether curve- or loop-based—is essential, but also offers flexibility and alternatives in trading off accuracy against efficiency.
% We further measure the performance of the tokenizers by their in-SFT performance when integrated with the LLMs.
% All previous methods are evaluated except \textbf{HNC-CAD} which does not involve LLM training in the original work.
% Table \ref{tab:ablation-in-sft} demonstrates the results.
% BPE and \model's variant without primitive pooling still perform poorly with high invalidity ratio and low evaluation results on all aspects.
% The \textit{loop} variation has mariginal disadvantage comparing to our main tokenizer, but its higher compression rate suggests a considerable alternative when the trade-off is taken into account, as well as the robustness of our pooling technique.
\input{sections/4xxc-ablation-post-sft}
\textbf{After LLM Finetuning: Different Sampling Strategies.}
Finally, we assess the impact of our FSA-based decoding strategy described in Section~\ref{sec:inference}.
We fix the backbone to the fine-tuned \model\ and apply different sampling strategies on top of it, and compare our method with the two mainstream approaches: top-$p$ sampling and \textit{beam search}.
Table~\ref{tab:ablation-post-sft} shows that FSA-guided sampling consistently outperforms both alternatives across all metrics.
Its CAD-specific design leverages formal sequence constraints, enabling it to utilize structural formatting information that general-purpose decoding methods do not take into account.
Also notably, \textit{beam} search improves the model performance in all aspects comparing to the top-\textit{p} sampling, but encounters a significant tradeoff in invalidity ratio.
% In Section \ref{sec:inference}, we propose a FSA-based sampling technique to further boost the model performance.
% We therefore evaluate its significance with comparison to two mainstream sampling methods: top-p sampling and the beam search.
% As suggested by Table \ref{tab:ablation-post-sft}, our sampling method outperforms the alternatives in all metrics.
% Being CAD-specific makes it able of utilize the formatting information better than the counterparts.
\begin{table}[]
\centering
\begin{tabular}{l a a a c c c a c c}
\Xhline{4\arrayrulewidth}
Methods & \textbf{F1-}Skt & \textbf{F1-}Ext & \textbf{CD} & \textbf{COV} & \textbf{JSD} & \textbf{MMD} & \textbf{IR} & \textbf{VLM} & \textbf{HE}\\
\hline
\textbf{CAD-Editor}$^*$ & 73.3 & 82.6 & \underline{40.7} & \underline{51.1} & \textbf{1.63} & \textbf{1.25} & 1.50 & \underline{4.28} & \textbf{1.63} \\
\textbf{GPT-4o} & \underline{80.0} & 78.1 & 42.9 & 51.8 & 4.00 & 2.14 & 47.9 & 1.94 & - \\
\textbf{Vanilla-LLaMA} & 78.8 & \underline{84.9} & 42.4 & 48.6 & 3.90 & 2.37 & 48.6 & 4.31 & 2.64 \\
\textbf{\model} & \textbf{88.6} & \textbf{94.8} & \textbf{13.5} & \textbf{52.4} & \underline{2.54} & \underline{1.97} & \underline{8.38} & \textbf{5.09} & \underline{1.72} \\
% \textbf{beam} & 63.3 & 82.8 & 44.9 & 39.8 & 13.6 & 4.71 & 38.0 & - & - \\
% \textbf{bpe} & 78.5 & 75.4 & 41.3 & 42.9 & 15.4 & 5.34 & 86.9 & - & - \\
\Xhline{4\arrayrulewidth}
\end{tabular}
\vspace{-5px}
\caption*{(a) \textbf{CAD-Editing} quantitative results. $^*$\textbf{CAD-Editor} is the task-specific model on it.}
\vspace{5px}
\begin{tabular}{l a a a c c c a c c}
\Xhline{4\arrayrulewidth}
Methods & \textbf{F1-}Skt & \textbf{F1-}Ext & \textbf{CD} & \textbf{COV} & \textbf{JSD} & \textbf{MMD} & \textbf{IR} & \textbf{VLM} & \textbf{HE}\\
\hline
\textbf{CADFusion}$^\dag$ & \underline{68.8} & 80.1 & \underline{38.5} & \underline{54.4} & \underline{10.6} & \underline{3.86} & \underline{22.5} & \textbf{5.41} & \underline{1.91} \\
\textbf{GPT-4o} & 66.7 & 66.8 & 79.8 & 52.6 & 62.0 & 13.5 & 90.5 & 1.47 & - \\
\textbf{Vanilla-LLaMA} & 66.4 & \underline{80.9} & 48.4 & 53.8 & 27.4 & 72.6 & 80.5 & 3.45 & 2.58 \\
\textbf{\model} & \textbf{77.9} & \textbf{84.7} & \textbf{26.7} & \textbf{54.5} & \textbf{7.26} & \textbf{3.74} & \textbf{1.50} & \underline{3.82} & \textbf{1.62} \\
% \textbf{beam} & 66.4 & 79.3 & 53.7 & 39.3 & 23.0 & 7.12 & 41.0 & - & - \\
% \textbf{bpe} & 70.9 & 80.0 & 74.9 & 36.8 & 54.8 & 16.4 & 90.5 & - & - \\
\Xhline{4\arrayrulewidth}
\end{tabular}
\vspace{-5px}
\caption*{(b) \textbf{Text-to-CAD} quantitative results. $^\dag$\textbf{CADFusion} is the task-specific model on it.}
\vspace{-5px}
\caption{Results in (a) and (b) include F1 scores for sketches (\textbf{F1}-skt $\uparrow$) and extrusions (\textbf{F1}-ext $\uparrow$), Chamfer Distance (\textbf{CD} $\downarrow$), Coverage(\textbf{COV} $\uparrow$), Minimum Matching Distance (\textbf{MMD} $\downarrow$), Jensen-Shannon Divergence (\textbf{JSD} $\downarrow$), Invalidity Ratio (\textbf{IR} $\downarrow$), the VLM Score (\textbf{VLM} $\uparrow$), and average rank from Human Evaluation (\textbf{HE} $\downarrow$). GPT-4o generations were not included in the human evaluation process. ($\uparrow$) marks the higher the better, and ($\downarrow$) marks the opposite. The best results are marked in \textbf{bold}, and second-best results are \underline{underlined}.}
\label{tab:quantitative}
\end{table}
\begin{figure}[t!]
\centering
\begin{minipage}{0.6\linewidth}
\begin{tabular}{l c c c c}
\Xhline{4\arrayrulewidth}
Methods & \textbf{F1}-Skt ($\uparrow$) & \textbf{COV} ($\uparrow$) & \textbf{JSD} ($\downarrow$)\\
\hline
\textbf{HNC-CAD} & 85.5 & 57.5 & 29.8 \vspace{1px}\\
% \textbf{BPE} & \textit{N/A}$^*$ & \textit{N/A}$^*$ & \textit{N/A}$^*$ \\
\textbf{\model} \textit{(curve)} & \textbf{94.1} & \textbf{64.5} & \textbf{8.19}\\
\textbf{\model} \textit{loop} & {91.5} & {59.5} & 18.4 \\
\textbf{\model} \textit{single} & 76.5 & 54.0 & 35.9 \\
\Xhline{4\arrayrulewidth}
\end{tabular}
\vspace{-1px}
\captionof{table}{The reconstruction quality of the \model \ variants and \textbf{HNC-CAD}.
Only the sketch scores are reported for the \textbf{F1} score because the \model \ encodes objects by sketch-extrusion pairs and always achieves full score for this sub-metric.
\textbf{\model} \textit{(curve)} is the default variant which we reported in the main quantitative results.
% *\textit{BPE returns exactly the original sequence algorithmically. Therefore we do not report its reconstruction quality.}
}
\label{tab:ablation-pre-sft}
\end{minipage}
\hfill
\begin{minipage}{0.35\linewidth}
\centering
\includegraphics[width=\linewidth]{images_new/fig4-compression.pdf}
\vspace{-20px}
\caption{The Compression ratio of the different tokenization algorithms. The compression ratio is 100\% for no compression.}
\label{fig:ablation-compression}
\vspace{5px}
\end{minipage}
\end{figure}
% \begin{table}[]
% \centering
% \begin{tabular}{l c a a c}
% \Xhline{4\arrayrulewidth}
% Methods & \textbf{F1}-Skt ($\uparrow$) & COV ($\uparrow$) & JSD ($\downarrow$) & Compression ($\uparrow$) \\
% \hline
% \textbf{\model}\textit{(-curve)} & \textbf{94.1} & \textbf{64.5} & \textbf{8.19} & 21.4 \\
% \textbf{\model}\textit{-loop} & {91.5} & {59.5} & 18.4 & 14.8 \\
% \textbf{\model}\textit{-single} & 76.5 & 54.0 & 35.9 & 2.11 \\
% \textbf{HNC-CAD} & 85.5 & 57.5 & 29.8 & \textbf{250} \\
% \textbf{BPE} & NA$^*$ & NA$^*$ & NA$^*$ & \underline{45.5} \\
% \Xhline{4\arrayrulewidth}
% \end{tabular}
% \caption{Caption}
% \label{tab:ablation-pre-sft}
% \end{table}
% hnc compression rate: 250 (removing unused tokens)
\begin{table}[t]
\centering
\begin{tabular}{l c c a a c}
\Xhline{4\arrayrulewidth}
Methods & \textbf{F1-}Avg $(\uparrow)$ & \textbf{CD} $(\downarrow)$ & \textbf{COV} $(\uparrow)$ & \textbf{JSD} $(\downarrow)$ & \textbf{IR} $(\downarrow)$ \\
\hline
\textbf{\model} \textit{(curve)} & \textbf{86.5} & \textbf{20.1} & {53.4} & \textbf{4.90} & 4.94 \\
\textbf{\model} \textit{loop} & {86.3} & 32.6 & \textbf{53.6} & {7.05} & \textbf{4.91} \\
\textbf{\model} \textit{single} & 78.3 & 59.1 & 43.4 & 48.5 & 70.7 \\
\textbf{BPE} & 76.2 & {56.8} & 48.8 & 48.3 & 88.5 \\
\Xhline{4\arrayrulewidth}
\end{tabular}
\caption{The performance of each tokenizer when integrated and trained with the backbone LLM. \textbf{\model} \textit{(curve)} is the default variant which we reported in the main quantitative results, while \textit{loop} is a variant on pooling loop primitives and \textit{single} does not perform specific pooling.}
\label{tab:ablation-in-sft}
\end{table}
\begin{table}[]
\centering
\begin{tabular}{l c c a a c}
\Xhline{4\arrayrulewidth}
Methods & \textbf{F1-}Avg $(\uparrow)$ & CD $(\downarrow)$ & COV $(\uparrow)$ & JSD $(\downarrow)$ & IR $(\downarrow)$ \\
\hline
\textbf{\model}\textit{(+FSA)} & \textbf{86.5} & \textbf{20.1} & \textbf{53.4} & \textbf{4.90} & \textbf{4.94} \\
\textbf{\model}\textit{+top-p} & 80.4 & 66.4 & 30.3 & 61.2 & 17.2 \\
\textbf{\model}\textit{+beam search} & 82.8 & 49.8 & 51.2 & 46.7 & 45.2 \\
\Xhline{4\arrayrulewidth}
\end{tabular}
\caption{The generation quality of each sampling method after training. The base model used for evaluation is the fine-tuned \model. \textbf{\model}\textit{(+FSA)} is the default variant which we reported in the main quantitative results.}
\label{tab:ablation-post-sft}
\end{table}
Our work's limitations are two-fold.
First, the quality gap between the open-source and private-sector CAD data limits us from training on further complex shapes.
Second, as mentioned and referred, we observe a gap between the distributional metrics and the actual performance in CAD Editing, which stems from the failure to evaluate the 'intent to keep the original shape.'
We therefore anticipate both better datasets and metrics that can create a better ground for future works to improve.
Additionally, our failure study in Appendix \ref{sec:apdx:failure} suggests better spatial or commonsense reasoning capabilities, which can only be improved by better pretrained backbones.
We include some of the related discussions, as well as the detail of our failure study, in Appendix \ref{sec:apdx:experiments}.
% First, although minimal, the additional VQ modules introduce encoding/decoding and alignment losses, potentially decreasing the overall fidelity of the system.
% Second, limitations persist in evaluating framework performance.
% Evaluations based on distributions and primitives do not always seem to align well with VLM and human preferences post-rendering.
% The reliability of VLMs when assessing multiple images (e.g., for editing tasks involving two images and a prompt) is also a concern, particularly for evaluating editing performance.
% Second, the task remains challenging for existing LLMs' spatial reasoning capabilities.
% In our experiments, this is sometimes reflected by instances where the LLM correctly generates a shape but places it in an incorrect location.
% This problem is reflected by the failure analysis in Appendix \ref{sec:apdx:failure}.
% Overall, the \tbcp \ problem is still in its early stages, and several key challenges remain to be addressed.
In this work, we presented \model, the first framework to address the unified text-based CAD prototyping problem, a task that requires a unified model on both Text-to-CAD generation and CAD editing.
Key to \model's approach is a pretrained transformer-based VQ-VAE module that enables CAD-specific tokenization by converting sequential CAD inputs into meaningful primitive tokens, instead of the word pieces produced by vanilla tokenizers.
LLM backbones that were finetuned on the CAD specific tokens demonstrated better performance and training efficiency.
Our extensive experiments validated these core design choices, including the primitive-specific pooling mechanism within the VQ-VAE and the FSA-driven sampling, and demonstrated the advantage of \model\ over both task-specific and general-purpose baselines.
We also improved inference-time decoding quality by introducing a novel finite-state automaton–driven sampling method, designed to enforce sequence formatting and hence improve generation quality.
% In the future, we plan to further improve pipeline performance by reducing tokenizer reconstruction loss and designing better training paradigms to enhance the backbone’s spatial reasoning capability.
\clearpage
% \section*{Ethics Statement}
% The data used in this work is tailored for CAD generation.
% Due to its specialized nature, the misuse risk is naturally minimized, ensuring the developed methods primarily benefit CAD development.
% \section*{Reproducibility Statement}
% We have included sufficient information on our all aspects of our experimentations including dataset resources and setups, model hyperparameters and LLM backbones, computational resources and setups, and metric setups.
% Relevant detail can be found in Section \ref{sec:exp-setup} and Appendix \ref{sec:apdx:evaluation-setup}.
% We will release the source code after the peer-review procedure.
% \section*{The Use of Large Language Models}
% THe use of LLMs is restricted in revising the word choices and grammar only.
% No LLMs are used for research ideation or contributive enough to be regarded as a contributor.
\section{Additional Information of the CAD Representation} \label{sec:apdx:representation}
\subsection{Sequence Representation}
We follow the practice of \cite{flexcad, cadeditor, cadfusion}.
In detail, a CAD object is defined by a series of sketches and extrusions, termed the \textit{Sketch-and-Extrude Modeling (SEM)} format.
Each sketch consists of multiple faces, with each face typically containing one or more loops.
Primitives in each loop include lines, arcs, and circles.
They are defined by an identifier and, for their geometric properties, one, two, or four key coordinates or sets of parameters, respectively.
Each extrusion is represented as a \texttt{BVVTTTRRRRRRRRRSOO} sequence.
\texttt{B} refers to the boolean operation selected from {\texttt{add, cut, intersect}};
the two \texttt{V} values define the respective displacements of the top and bottom extrusion planes from a reference plane;
the three \texttt{T} values form the translation vector;
the nine \texttt{R} values represent rotation parameters;
\texttt{S} is the scaling factor;
and the two \texttt{O} values identify the origin of scaling.
Figure \ref{fig:seq-format} illustrates the components of the sketch and extrusion operations.
\begin{figure}[h]
\centering
\includegraphics[width=0.8\linewidth]{images_new/apdx-fig0-seq-format.pdf}
\caption{An overview of the CAD sequence structure. Top: sketch components; bottom: extrusion components.}
\label{fig:seq-format}
\end{figure}
In our VQ-VAE training, we tokenize CAD sequences into their smallest semantic units.
These units are defined in Table \ref{tab:vq-vocabs}.
This tokenizer differs from native LLM tokenizers in that it does not further split these semantic units into word pieces.
\begin{table}[h]
\centering
\begin{tabular}{c c c}
\hline
Token Name & Vocabulary Index & Description\\
\hline
\textit{pad} & 0 & Padding token \\
\textit{line} & 1 & Identifier token of curve: line \\
\textit{arc} & 2 & Identifier token of curve: arc \\
\textit{circle} & 3 & Identifier token of curve: circle \\
\textit{<curve\_end>} & 4 & End token of curves \\
\textit{<loop\_end>} & 5 & End token of loops \\
\textit{<face\_end>} & 6 & End token of faces \\
\textit{<sketch\_end>} & 7 & End token of sketches \\
\textit{add} & 8 & Identifier token of extrusion operation: add \\
\textit{cut} & 9 & Identifier token of extrusion operation: cut \\
\textit{intersect} & 10 & Identifier token of extrusion operation: intersect \\
\textit{<extrusion\_end>} & 11 & End token of extrusions \\
\textit{-1} & 12 & Number -1 \\
\textit{0-63} & 13-76 & Numbers from 0 to 63 \\
\hline\vspace{1px}
\end{tabular}
\caption{Token vocabulary for the CAD sequence representation used in training our VQ-VAE.}
\label{tab:vq-vocabs}
\end{table}
\subsection{Primitive Definition}
We define our primitives differently for sketches and extrusions.
For sketches, we define one primitive as a curve and any succeeding end tokens, if present.
For extrusions, we split each into three parts: {\texttt{B,V,T}}, {\texttt{R}}, and {\texttt{S,O,END}}, and split \texttt{R} into 3 distinct tokens.
This partitioning is designed to prevent the VQ-VAE from having to summarize overly long sequences, which can be difficult to learn effectively.
Our primitive design offers several advantages.
First, limiting the scope to the loop level restricts the number of tokens the VQ-VAE model generates for the LLM to a manageable quantity.
Generating one token per curve would provide the LLM with too many tokens; conversely, one token per face or per sketch would yield too few for the LLM to extract meaningful correlations and, in the case of per-sketch tokenization, might also represent too much information for a VQ-VAE to compress effectively into a single token.
Second, compared to works that do not pool primitives but instead train on smaller `particles' \citep{skexgen, hnc}, our approach improves representation quality by embedding all primitive representations into a unified subspace (which is not guaranteed by methods training separate models for different CAD pieces) and by posing contextualization.
Furthermore, those approaches may not adequately address the challenge of long extrusion sequences, which are difficult for a VQ-VAE to compress meaningfully into a single token.
\section{Implementation Details}
\subsection*{Preliminary: Terminologies and Notations} \label{sec:apdx:vq}
We use the term \texttt{Encoder} and \texttt{Decoder} to refer the \textit{transformer encoder} and \textit{decoder} layers in with in the VQ module. We refer the term \texttt{VQ Encoder} to the entire module before token extraction, i.e. the \textit{transformer encoder layers}, the \textit{primitive pooling}, and the \textit{VQ layer}. The same applies to \texttt{VQ Decoder}, but since no additional modules are added, it is in fact equivalent to the \texttt{Decoder}. \texttt{CAD Encoder} and \texttt{CAD Decoder} describe the combination of the \texttt{VQ Encoder/Decoder} and the corresponding \textit{adapter layers}.
\subsection{VQ and Primitive Pooling}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images_new/apdx-fig1-supp-overview.pdf}
\caption{A detailed overview of the VQ-VAE pretraining and finetuning.}
\label{fig:apdx:vq-detail}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images_new/apdx-fig2-primitive-pooling.pdf}
\caption{An overview of our primitive pooling procedure for each sequence.}
\label{fig:masked_pooling}
\end{figure}
A classic VQ-VAE for sequential data consists of transformer encoder layers, a VQ layer to quantize vectors, and transformer decoder layers.
Input tokens are encoded, pooled into a single hidden vector, and then mapped to codebook vectors.
These VQ vectors are then fed into the decoder, which typically reconstructs the output sequence via a cross-attention mechanism.
A traditional training procedure includes a reconstruction loss between the decoder outputs and the initial inputs, as well as a quantization loss.
This traditional approach of relying on a single hidden vector can be insufficient for representing complex sequences like those in CAD.
To obtain richer primitive representations, previous CAD works \citep{skexgen, hnc} separated CAD objects into multiple primitives and trained individual VQ models on them.
We argue that this approach has limitations because training separate models can introduce alignment issues, and primitives may not receive contextualized information that could be beneficial for their encoding.
In contrast, we split each CAD sequence into its constituent sketch-extrusion pairs and propose a primitive-specific pooling layer—replacing standard global pooling—to generate distinct representations for these pairs.
Our primitive pooling is achieved by masking; i.e., we generate masks for each input sketch-extrusion pair to isolate each primitive within it.
Figure \ref{fig:masked_pooling} demonstrates our method.
In detail, we extract pooling masks from the input sequence that reflect the primitive corresponding to each token.
Subsequently, we perform element-wise multiplication between each individual mask and the encoded representation output by the encoder.
At this point, the resulting matrix contains information related only to the specific primitive isolated by the mask, allowing us to perform max pooling to obtain the pooled representation for that primitive.
By concatenating these pooled primitive representations, we obtain a sequence of primitive-specific representations used for the decoder's cross-attention mechanism and for subsequent LLM alignment.
We set the maximum number of primitives in our method to 12.
This corresponds to 9 flexible slots for loop primitives and 3 fixed slots for extrusion primitives.
The limit for loop primitives is generally not restrictive, as the average number of loops per sketch-extrusion pair in our dataset is approximately 4, and during VQ-VAE training, we filtered out only around 900 sequences (out of 90,000) due to exceeding this limit.
\subsection{FSA Sampling} \label{sec:apdx:sampling}
Our FSA sampling is based on a specially-designed Finite State Automaton (FSA) that determines the set of permissible next tokens (referred to as logit options).
The system generates a token (or tokens) based on these allowed options, and then updates the FSA by inputting the current FSA state and the last generated token as an action, to determine the options for the subsequent step.
The algorithm is described in Algorithm \ref{alg:fsa}, and the FSA's detailed design is illustrated in Figure \ref{fig:fsa}.
The specific logit masks employed by the FSA are listed in Table \ref{tab:masks}.
\begin{table}[h!]
\centering
\begin{tabular}{c c c}
\hline
Mask Name & Mask Content & Description \\
\hline
\texttt{Init} & [line, arc, circle] & Enforces that the first token is a curve type. \\
\texttt{Numbers} & [0-63] & Restricts selection to numerical values (0-63). \\
\texttt{End-of-curve} & [<curve\_end>] & Enforces the <curve\_end> token. \\
\texttt{Primitive-start} & [line, arc, circle] & Enforces a new curve; identical to \texttt{Init}. \\
\texttt{End-of-loop} & [<loop\_end>] & Enforces the <loop\_end> token. \\
\texttt{End-of-face} & [<face\_end>] & Enforces the <face\_end> token. \\
\texttt{End-of-sketch} & [<sketch\_end>] & Enforces the <sketch\_end> token. \\
\texttt{B} & [add, cut, intersect] & Boolean extrusion operations. \\
\texttt{V} & [0-63] & Numerical values for V parameter. \\
\texttt{T} & [0-63] & Numerical values for T parameters. \\
\texttt{R} & [-1, 0, 1] & -1, 0, or 1 for rotation parameters. \\
\texttt{S} & [0-63] & Numerical values for the S parameter. \\
\texttt{O} & [0-63] & Numerical values for O parameters. \\
\texttt{End-of-extrusion} & [<extrusion\_end>] & Enforces <extrusion\_end>. \\
\texttt{Pad} & [pad] & Restricts selection to the padding token. \\
\hline\vspace{1px}
\end{tabular}
\caption{Details of the logit masks provided by the FSA to guide token generation during decoding.}
\label{tab:masks}
\end{table}
\begin{figure} [h!]
\centering
\includegraphics[width=\linewidth]{images_new/apdx-fig3-fsa.pdf}
\caption{An overview of our FSA design. At each step, the FSA receives an input action, transitions to the corresponding new state (i.e., updates its internal state), and returns the mask(s) associated with that new state (node).}
\label{fig:fsa}
\end{figure}
\begin{algorithm}
\caption{The FSA-driven decoding algorithm.}\label{alg:fsa}
\begin{algorithmic}
\STATE Given: LLM-generated primitive tokens $T$
\STATE Given: Max length of generation $max\_len$
\STATE $State\gets \texttt{`init'}$
\STATE $M \gets [\texttt{initial\_mask}]$
\STATE $Seq \gets \emptyset$
\WHILE{$|Seq| < max\_len$}
\STATE $m \gets$ $M$.next()
\STATE logits $\gets$ Decoder($Seq$, $T$)
\STATE choice $\gets$ argmax($m~\otimes$ logits)
\STATE $Seq$.insert(choice)
\IF{$M = \emptyset$}
\STATE new\_state, new\_M = FSA($State$, choice)
\STATE $State \gets$ new\_state
\STATE $M \gets$ new\_M
\ENDIF
\ENDWHILE
\end{algorithmic}
\end{algorithm}
\section{Experimental Results}
\subsection{Evaluation Setup} \label{sec:apdx:evaluation-setup}
\textbf{Instruction of Human Judges.}
Five human judges were provided with 50 generation outputs for evaluation (40 from editing tasks and 10 from Text-to-CAD tasks).
All participants had completed college-level to graduate-level education.
The judges were given specific instructions on the evaluation task and were shown visual examples similar to those in the main qualitative results, but without model identifiers.
\begin{lstlisting}[caption={Instructions of human judges.}, language=python]
"""
The following are a series of pictures. The upper half is the instruction and the standand answer. You need to rank the following three pictures based on their response quality to the instruction.
For example, if in a picture you like 3 the most, 1 following, and 2 the worst, rank them as (2 3 1).
"""
\end{lstlisting}
\textbf{Instruction of VLM.} The prompts we use to generate VLM scorings are listed Listing 3 and 4.
\begin{lstlisting}[caption={Instructions of VLM on CAD Editing}, language=python]
"""
The following is an original image of a CAD instance, a text description on editing and an image of the edited result. Measure if the figure corresponds to the given description, and give a score in the scale of 10. Only return the score. Do not comment on issues such as texture, smoothness and colors.\n description:{description}\n
"""
\end{lstlisting}
\begin{lstlisting}[caption={Instructions of VLM on Text-to-CAD generation}, language=python]
"""
The following is a text description of a 3D CAD figure and an image of a CAD instance. Measure if the figure corresponds to the given description, and give a score in the scale of 10. Only return the score. Do not comment on issues such as texture, smoothness and colors.\n description:{description}\n
"""
\end{lstlisting}
\subsection{Experimental Results} \label{sec:apdx:experiments}
\subsubsection{Elaboration on CAD Editing Results on the Distributional Metrics} \label{sec:apdx:editing-distribution-metric-problem}
In the CAD Editing task, we observe that models such as GPT-4o can achieve high scores on distributional metrics for the CAD editing task, despite poor actual performance in editing.
We hypothesize that a model can frequently fail to apply edits but still achieve satisfactory distributional scores by simply reproducing the input CAD sequence without modifications.
To investigate this matter further, we provide additional distributional measures for three comparison types: 1. between GPT-4o outputs and the original input sequences, 2. between outputs from \model \ and the original input sequences, and 3. between the original input sequences and the ground truth edited sequences.
The results are displayed in Table \ref{tab:orig-seq-results}.
As shown in the third row (comparing original inputs to the ground truth), the original sequence yields high distributional scores.
However, such outputs should be considered failed edits, as they do not incorporate any of the specified editing instructions.
VLMs capture instruction-following capabilities much better by assigning a low score when an output merely returns the original sequence.
In light of this, we measured the distributional similarity between the original input sequences and the outputs from GPT-4o and our \model.
It was found that GPT-4o's outputs are much closer to the original sequences than those from \model, suggesting that our model more consistently attempts to modify the original sequence and perform the requested edits.
This pattern, along with our model's overall advantages, is more accurately reflected by the difference in VLM scores presented in the main quantitative results.
We draw the conclusion that the interpretation of the distributional metrics should be extra careful as it may not always reflect the true performance of the model.
The analysis on the CAD Editing performance should take all metrics (especially the qualitative ones) into consideration in order to get unbiased information.
\begin{table}[]
\centering
\begin{tabular}{l c c c c}
\Xhline{4\arrayrulewidth}
Methods & \textbf{COV} & \textbf{JSD} & \textbf{MMD} & \textbf{VLM}(left)\\
\hline
% \textbf{CAD-Editor} <--> \textbf{Original Seq.} & 51.1 & 1.63 & 1.25 & 4.28\\
\textbf{GPT-4o} <--> \textbf{Original Seq.} & 60.8 & 6.75 & 2.39 & 1.94\\
\textbf{CAD-Tokenizer} <--> \textbf{Original Seq.} & 52.4 & 2.54 & 1.97 & 5.09\\
\textbf{Original Seq.} <--> \textbf{Ground Truth} & 55.6 & 6.07 & 2.54 & 2.02\\
\Xhline{4\arrayrulewidth}\vspace{4px}
\end{tabular}
\caption{Distributional measures for pairwise sequence comparisons from the CAD editing task. VLM scores of the left-hand-side component are reported.}
\label{tab:orig-seq-results}
\end{table}
\subsubsection{Additional Qualitative Results} \label{sec:apdx:additional-qualitive}
We present additional qualitative results for CAD editing and Text-to-CAD generation in Figure \ref{fig:additional-qualitative-editing} and \ref{fig:additional-qualitative-t2c}, respectively.
\begin{figure}[h!]
\centering
\includegraphics[width=\linewidth]{images_new/apdx-fig4-qual-part1.pdf}
\caption{Additional qualitative results for CAD editing.}
\label{fig:additional-qualitative-editing}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=\linewidth]{images_new/apdx-fig5-qual-part2.pdf}
\caption{Additional qualitative results for Text-to-CAD generation.}
\label{fig:additional-qualitative-t2c}
\end{figure}
\subsubsection{Failure Cases} \label{sec:apdx:failure}
In this section, we discuss failure cases observed with our method.
As shown in Figure \ref{fig:failure-cases}, we classify these failures into three categories: \textit{overcomplicated testing entries}, \textit{misleading instructions}, and \textit{lack of spatial or commonsense reasoning}.
We list our findings below:
\begin{enumerate}
\item For the first category, the model faces difficulty in generating correct shapes from complicated instructions. For example, an extrusion resembling the number '4' poses difficulty, especially when the instruction phrases it as "four polyhedra" without stating its resemble with the number.
\item The second category describes instances where the instructions do not always accurately reflect the relationship between the original and target shape. This issue can arise because the CAD-Editor dataset was generated by prompting VLMs, for which achieving absolute robustness and trustworthiness remains an open problem; this, in turn, poses instruction-following challenges for our model.
\item The last category reflects a lack of common sense or spatial reasoning, likely originating from limitations in the backbone LLM's capabilities. Generating concrete shapes such as a "key" or letter shapes, as well as arranging multiple objects in an organized way, remains challenging.
\end{enumerate}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images_new/apdx-fig6-failure.pdf}
\caption{An overview of our model's failure cases. The layout follows that of Figures \ref{fig:additional-qualitative-editing} and \ref{fig:additional-qualitative-t2c}.}
\label{fig:failure-cases}
\end{figure}
\begin{thebibliography}{30}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Brohan et~al.(2023{\natexlab{a}})Brohan, Brown, Carbajal, Chebotar, Chen, Choromanski, Ding, Driess, Dubey, Finn, Florence, Fu, Arenas, Gopalakrishnan, Han, Hausman, Herzog, Hsu, Ichter, Irpan, Joshi, Julian, Kalashnikov, Kuang, Leal, Lee, Lee, Levine, Lu, Michalewski, Mordatch, Pertsch, Rao, Reymann, Ryoo, Salazar, Sanketi, Sermanet, Singh, Singh, Soricut, Tran, Vanhoucke, Vuong, Wahid, Welker, Wohlhart, Wu, Xia, Xiao, Xu, Xu, Yu, and Zitkovich]{rt2}
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi~Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse~Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei~Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich.
\newblock Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2307.15818}.
\bibitem[Brohan et~al.(2023{\natexlab{b}})Brohan, Brown, Carbajal, Chebotar, Dabis, Finn, Gopalakrishnan, Hausman, Herzog, Hsu, Ibarz, Ichter, Irpan, Jackson, Jesmonth, Joshi, Julian, Kalashnikov, Kuang, Leal, Lee, Levine, Lu, Malla, Manjunath, Mordatch, Nachum, Parada, Peralta, Perez, Pertsch, Quiambao, Rao, Ryoo, Salazar, Sanketi, Sayed, Singh, Sontakke, Stone, Tan, Tran, Vanhoucke, Vega, Vuong, Xia, Xiao, Xu, Xu, Yu, and Zitkovich]{rt1}
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil~J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich.
\newblock Rt-1: Robotics transformer for real-world control at scale, 2023{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2212.06817}.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]{vit}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
\newblock URL \url{https://arxiv.org/abs/2010.11929}.
\bibitem[Gage(1994)]{bpe}
Philip Gage.
\newblock A new algorithm for data compression.
\newblock \emph{The C Users Journal}, 12\penalty0 (2):\penalty0 23--38, 1994.
\bibitem[Ge et~al.(2023)Ge, Ge, Zeng, Wang, and Shan]{SEED}
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan.
\newblock Planting a seed of vision in large language model.
\newblock \emph{arXiv preprint arXiv:2307.08041}, 2023.
\bibitem[Ge et~al.(2025)Ge, Zhao, Zhu, Ge, Yi, Song, Li, Ding, and Shan]{SEEDX}
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan.
\newblock Seed-x: Multimodal models with unified multi-granularity comprehension and generation, 2025.
\newblock URL \url{https://arxiv.org/abs/2404.14396}.
\bibitem[Hu et~al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen]{lora}
Edward~J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, and Weizhu Chen.
\newblock Lora: Low-rank adaptation of large language models, 2021.
\newblock URL \url{https://arxiv.org/abs/2106.09685}.
\bibitem[Khan et~al.(2024{\natexlab{a}})Khan, Dupont, Ali, Cherenkova, Kacem, and Aouada]{cadsignet}
Mohammad~Sadil Khan, Elona Dupont, Sk~Aziz Ali, Kseniya Cherenkova, Anis Kacem, and Djamila Aouada.
\newblock Cad-signet: Cad language inference from point clouds using layer-wise sketch instance guided attention.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pp.\ 4713--4722, June 2024{\natexlab{a}}.
\bibitem[Khan et~al.(2024{\natexlab{b}})Khan, Sinha, Uddin, Stricker, Ali, and Afzal]{text2cad}
Mohammad~Sadil Khan, Sankalp Sinha, Sheikh~Talha Uddin, Didier Stricker, Sk~Aziz Ali, and Muhammad~Zeshan Afzal.
\newblock Text2cad: Generating sequential cad designs from beginner-to-expert level text prompts.
\newblock In \emph{The Thirty-eighth Annual Conference on Neural Information Processing Systems}, 2024{\natexlab{b}}.
\newblock URL \url{https://openreview.net/forum?id=5k9XeHIK3L}.
\bibitem[Kingma \& Welling(2022)Kingma and Welling]{vae}
Diederik~P Kingma and Max Welling.
\newblock Auto-encoding variational bayes, 2022.
\newblock URL \url{https://arxiv.org/abs/1312.6114}.
\bibitem[Li et~al.(2025{\natexlab{a}})Li, Huang, Xu, Schlegel, Jiang, Batista-Navarro, Nenadic, and Bian]{bridge}
Hao Li, Yuhao Huang, Chang Xu, Viktor Schlegel, Renhe Jiang, Riza Batista-Navarro, Goran Nenadic, and Jiang Bian.
\newblock Bridge: Bootstrapping text to control time-series generation via multi-agent iterative optimization and diffusion modelling, 2025{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2503.02445}.
\bibitem[Li et~al.(2025{\natexlab{b}})Li, Ma, Li, Lou, Zhou, and Zhou]{cadllama}
Jiahao Li, Weijian Ma, Xueyang Li, Yunzhong Lou, Guichun Zhou, and Xiangdong Zhou.
\newblock Cad-llama: Leveraging large language models for computer-aided design parametric 3d model generation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, June 2025{\natexlab{b}}.
\bibitem[Li et~al.(2024)Li, Song, Lou, and Zhou]{cadtranslator}
Xueyang Li, Yu~Song, Yunzhong Lou, and Xiangdong Zhou.
\newblock {CAD} translator: An effective drive for text to 3d parametric computer-aided design generative modeling.
\newblock In \emph{ACM Multimedia 2024}, 2024.
\bibitem[Liu et~al.(2023)Liu, Li, Wu, and Lee]{llava}
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong~Jae Lee.
\newblock Visual instruction tuning, 2023.
\newblock URL \url{https://arxiv.org/abs/2304.08485}.
\bibitem[Ma et~al.(2024)Ma, Chen, Lou, Li, and Zhou]{stepbystep}
Weijian Ma, Shuaiqi Chen, Yunzhong Lou, Xueyang Li, and Xiangdong Zhou.
\newblock Draw step by step: Reconstructing cad construction sequences from point clouds via multimodal diffusion.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pp.\ 27154--27163, June 2024.
\bibitem[Mikolov et~al.(2013)Mikolov, Chen, Corrado, and Dean]{word2vec}
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.
\newblock Efficient estimation of word representations in vector space, 2013.
\newblock URL \url{https://arxiv.org/abs/1301.3781}.
\bibitem[Nie et~al.(2023)Nie, Nguyen, Sinthong, and Kalagnanam]{PatchTST}
Yuqi Nie, Nam~H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam.
\newblock A time series is worth 64 words: Long-term forecasting with transformers, 2023.
\newblock URL \url{https://arxiv.org/abs/2211.14730}.
\bibitem[Pennington et~al.(2014)Pennington, Socher, and Manning]{glove}
Jeffrey Pennington, Richard Socher, and Christopher Manning.
\newblock {G}lo{V}e: Global vectors for word representation.
\newblock In Alessandro Moschitti, Bo~Pang, and Walter Daelemans (eds.), \emph{Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ({EMNLP})}, pp.\ 1532--1543, Doha, Qatar, October 2014. Association for Computational Linguistics.
\newblock \doi{10.3115/v1/D14-1162}.
\newblock URL \url{https://aclanthology.org/D14-1162/}.
\bibitem[Pertsch et~al.(2025)Pertsch, Stachowicz, Ichter, Driess, Nair, Vuong, Mees, Finn, and Levine]{fast}
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine.
\newblock Fast: Efficient action tokenization for vision-language-action models, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.09747}.
\bibitem[Podell et~al.(2023)Podell, English, Lacey, Blattmann, Dockhorn, Müller, Penna, and Rombach]{sdxl}
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.
\newblock Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
\newblock URL \url{https://arxiv.org/abs/2307.01952}.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{clip}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
\newblock Learning transferable visual models from natural language supervision, 2021.
\newblock URL \url{https://arxiv.org/abs/2103.00020}.
\bibitem[Seff et~al.(2022)Seff, Zhou, Richardson, and Adams]{vitruvion}
Ari Seff, Wenda Zhou, Nick Richardson, and Ryan~P. Adams.
\newblock Vitruvion: A generative model of parametric cad sketches, 2022.
\newblock URL \url{https://arxiv.org/abs/2109.14124}.
\bibitem[van~den Oord et~al.(2018)van~den Oord, Vinyals, and Kavukcuoglu]{vqvae}
Aaron van~den Oord, Oriol Vinyals, and Koray Kavukcuoglu.
\newblock Neural discrete representation learning, 2018.
\newblock URL \url{https://arxiv.org/abs/1711.00937}.
\bibitem[Wang et~al.(2025)Wang, Yuan, Sun, and Bian]{cadfusion}
Ruiyu Wang, Yu~Yuan, Shizhao Sun, and Jiang Bian.
\newblock Text-to-cad generation through infusing visual feedback in large language models, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.19054}.
\bibitem[Wu et~al.(2021)Wu, Xiao, and Zheng]{deepcad}
Rundi Wu, Chang Xiao, and Changxi Zheng.
\newblock Deepcad: A deep generative network for computer-aided design models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, pp.\ 6772--6782, October 2021.
\bibitem[Xu et~al.(2022)Xu, Willis, Lambourne, Cheng, Jayaraman, and Furukawa]{skexgen}
Xiang Xu, Karl~DD Willis, Joseph~G Lambourne, Chin-Yi Cheng, Pradeep~Kumar Jayaraman, and Yasutaka Furukawa.
\newblock Skexgen: Autoregressive generation of cad construction sequences with disentangled codebooks.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 24698--24724. PMLR, 2022.
\bibitem[Xu et~al.(2023)Xu, Jayaraman, Lambourne, Willis, and Furukawa]{hnc}
Xiang Xu, Pradeep~Kumar Jayaraman, Joseph~G Lambourne, Karl~DD Willis, and Yasutaka Furukawa.
\newblock Hierarchical neural coding for controllable cad model generation.
\newblock \emph{arXiv preprint arXiv:2307.00149}, 2023.
\bibitem[Yuan et~al.(2025)Yuan, Sun, Liu, and Bian]{cadeditor}
Yu~Yuan, Shizhao Sun, Qi~Liu, and Jiang Bian.
\newblock Cad-editor: A locate-then-infill framework with automated training data synthesis for text-based cad editing, 2025.
\newblock URL \url{https://arxiv.org/abs/2502.03997}.
\bibitem[Zhang et~al.(2025)Zhang, Sun, Wang, Cai, and Bian]{flexcad}
Zhanwei Zhang, Shizhao Sun, Wenxiao Wang, Deng Cai, and Jiang Bian.
\newblock Flexcad: Unified and versatile controllable cad generation with fine-tuned large language models, 2025.
\newblock URL \url{https://arxiv.org/abs/2411.05823}.
\bibitem[Zhou et~al.(2024)Zhou, Yu, Babu, Tirumala, Yasunaga, Shamis, Kahn, Ma, Zettlemoyer, and Levy]{transfusion}
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy.
\newblock Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024.
\newblock URL \url{https://arxiv.org/abs/2408.11039}.
\end{thebibliography}