LLM as an Art Director (LaDi): Using LLMs to improve Text-to-Media Generators
Extracted source text · arXiv 2311.03716
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2023
% ready for submission
\usepackage[preprint, nonatbib]{neurips_2023}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2023}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2023}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2023}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{amsmath}
\usepackage{listings} % Required for inserting code snippets
\usepackage{biblatex}
\addbibresource{sample.bib}
\title{LLM as an Art Director (LaDi): Using LLM's to improve Text-to-Media Generators}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{
Allen Roush \\
Plai Labs \\
\texttt{allen@plailabs.com} \\
\And
Emil Zakirov \\
Plai Labs \\
\texttt{emil.zakirov@plailabs.com} \\
\And
Artemiy Shirokov \\
Plai Labs \\
\texttt{tema77078@gmail.com} \\
\And
Polina Lunina \\
Plai Labs \\
\texttt{polina.lunina@plailabs.com} \\
\And
Jack Gane \\
Plai Labs \\
\texttt{jack.gane@plailabs.com} \\
\And
Alexander Duffy \\
Plai Labs \\
\texttt{alex@plailabs.com} \\
\And
Charlie Basil \\
Plai Labs \\
\texttt{charlie@plailabs.com} \\
\And
Aber Whitcomb \\
CTO, Plai Labs \\
\texttt{aber@plailabs.com} \\
\And
Jim Benedetto \\
Plai Labs \\
\texttt{jim@reachjim.com} \\
\And
Chris DeWolfe \\
CEO, Plai Labs \\
\texttt{chris@plailabs.com}
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
Recent advancements in text-to-image generation have revolutionized numerous fields, including art and cinema, by automating the generation of high-quality, context-aware images and video. However, the utility of these technologies is often limited by the inadequacy of text prompts in guiding the generator to produce artistically coherent and subject-relevant images. In this paper, We describe the techniques that can be used to make Large Language Models (LLMs) act as Art Directors that enhance image and video generation. We describe our unified system for this called "LaDi". We explore how LaDi integrates multiple techniques for augmenting the capabilities of text-to-image generators (T2Is) and text-to-video generators (T2Vs), with a focus on constrained decoding, intelligent prompting, fine-tuning, and retrieval. LaDi and these techniques are being used today in apps and platforms developed by Plai Labs.
\end{abstract}
\section{Introduction}
\subsection{Background}
The landscape of image generation has experienced a paradigm shift owing to the emergence of advanced text-to-image models like Stable Diffusion and DALL-E. These models have proven to be indispensable in a variety of applications, ranging from art and cinema to digital marketing and virtual simulations. However, a persistent limitation hindering their utility is the quality of text prompts employed to guide image generation. While the raw power of text-to-image generators is undeniably remarkable, the absence of sophisticated prompting techniques often results in images that are technically impressive but lack subject relevance or artistic coherence.
The crux of the issue lies in the text prompts themselves. While text-to-image models boast impressive technical prowess, their output is only as good as the input prompt. Vague, ambiguous, or poorly composed prompts often yield images that, while visually striking, are irrelevant or nonsensical. The art of prompting is complex; striking the right balance of detail versus brevity while conveying subtle attributes like tone, composition, and style is non-trivial even for humans.
In light of these challenges, this paper introduces a groundbreaking framework called "LaDi" in which Large Language Models (LLMs) serve as Art Directors for text-to-image generators. Instead of relying on simple, direct text prompts like "a two-story pink house," our approach leverages the nuanced understanding of LLMs to construct more elaborate, descriptive prompts that could translate to "a two-story Victorian house bathed in the rosy hues of a sunset, complete with a garden in full bloom." Such intelligent prompting brings a level of depth and specificity that has been largely missing in automated image generation.
We extend this concept to include constrained decoding and specialized tool usage, effectively setting parameters that the image must conform to. Constrained decoding allows for the enforcement of certain conditions—such as maintaining a consistent color scheme or ensuring the presence of specific elements—thus making the generated image both predictable and relevant.
In this paper, we describe the techniques that we discovered at Plai Labs to create an effective LLM based "Art Director" called LaDi. LaDi is used throughout the social media platform "PlaiDay" and will be used in all future services.
\subsection{Text-to-Image Models}
The growing capability of text-to-image generation models has engendered a profound impact across multiple domains, from content creation and digital art to medical imaging and beyond. One of the most notable advancements in this sphere has been the development of Stable Diffusion from \textcite{rombach2022highresolution}, a framework grounded in latent diffusion models (LDMs) crafted by the CompVis group at LMU Munich. By integrating variational autoencoders (VAEs), U-Net architectures, and text encoders, Stable Diffusion provides a robust yet efficient solution to the challenge of generating contextually rich and visually captivating images.
We choose Stable Diffusion as our main T2I model because of its open-source nature and huge amount of custom community content/extensions. We believe that with this custom content, Stable Diffusion is superior to competitor models.
While Stable Diffusion offers increased computational efficiency for training and generation, it still faces challenges, particularly in the realm of text prompting. Prompting is an art in itself—providing the model with just the right amount of context to generate images that are not only visually appealing but also semantically meaningful. The utilization of a fixed, pretrained CLIP ViT-L/14 text encoder aids in transforming text prompts to an embedding space; however, the process is far from trivial. Effectively managing the balance between under-specification and over-specification in text prompts becomes a crucial factor in achieving desired outputs.
Moreover, Stable Diffusion has its limitations, such as issues with degradation and inaccuracies under certain conditions, complexities in adapting the model for novel use-cases, and susceptibility to algorithmic biases. Despite these challenges, its relative lightweight nature—in terms of computational resources—and its flexibility make it a cornerstone in the current landscape of text-to-image generation models.
\subsection{Text-to-Video Models}
Text-to-video synthesis has seen remarkable progress in recent years, with models capable of generating increasingly high-quality and controllable video content from text prompts. Several key innovations have catalyzed these advancements.
\textcite{wang2023modelscope} introduced ModelScopeT2V which uses spatio-temporal blocks to capture complex dependencies between frames. By training on both image-text and video-text datasets, ModelScopeT2V enhances semantic diversity. It achieves state-of-the-art performance on established benchmarks, serving as an impactful baseline.
\textcite{ho2022imagen} introduced Imagen Video which demonstrates the effectiveness of cascaded diffusion models for high-definition video generation. It combines a text encoder, base video model, and interleaved spatial and temporal super-resolution models, totaling 11.6B parameters. The model exhibits strong temporal consistency, while inheriting Imagen’s image generation capabilities.
\textcite{singer2022makeavideo} introduced Make-A-Video which pioneers video generation without paired video-text data. It creatively adapts image-to-image diffusion models for frame generation and employs noise conditioning to enable video-length sampling. Despite no video supervision, Make-A-Video produces videos aligned with text prompts.
These models have expanded the horizons of controllable video synthesis. Looking ahead, potential areas of improvement include longer video durations, 3D scene consistency, and multimodal control. An exciting new frontier is the combination of language, audio, and video in unified generative models. Text-to-video generation remains an active area of research.
AnimateDiff, introduced by \textcite{guo2023animatediff}, facilitates the conversion of any personalized T2I model into an animation generator without requiring model-specific tuning. Built upon motion modeling modules trained on extensive video datasets, AnimateDiff can be integrated into existing T2I models to produce animated clips that are both temporally smooth and visually consistent. This framework succeeds in democratizing animation generation just as prior T2I models have done for static image generation.
AnimateDiff was originally released with two "motion modules", which are models that take an input sequence of images and generates a video with motion connecting the images. Subsequently, many community contributed motion moduels have been released. These include "Motion Loras" analogous to Stable Diffusion "Lora" models (low rank adapters) which modify an existing motion module.
AnimateDiff became the dominant form of AI video generation at the end of 2023 because of its speed, support for controlnet models (which make consistent video possible), and because users realized they could create any length video by generating more images and using a sliding window input context.
Despite its revolutionary capabilities, AnimateDiff is not without its challenges. One of the key difficulties lies in the act of prompting—how do we accurately instruct the model to generate content that not only captures the nuances of the text but also transitions smoothly in the animation? General text-to-video generation techniques have often found it challenging to manage the sensitivity of hyperparameters, the collection of personalized video, and the computational resources required, making the task of generating high-quality animated content from text not as straightforward.
\section{Prior Work}
Using language models to assist with generating prompts for image generators is not a new idea. In this section, we look at prior work related to connecting LLMs to T2I models.
\textcite{wang-etal-2023-diffusiondb} introduced a 1.8 million Stable Diffusion prompt dataset called DiffusionDB. Such a dataset is extremely valuable for fine-tuning a language model to generate prompts. \textcite{mehrabi-etal-2023-resolving} introduced a benchmark dataset called the "Text-to-image Ambiguity Benchmark". They show that their framework for prompt disambiguation generates more faithful images which align more to user preferences. \textcite{russo-2022-creative} study the aspects of creative image generation which are unique, and propose criteria that dataset authors should think about when building text-to-image generation benchmarks.
\textcite{hong-etal-2023-visual-writing} look at techniques for improving visual story generation and contribute a dataset called "Visual Writing Prompts". Their work looks at the idea of using images to guide multi modal language models for story generation. \textcite{du-etal-2023-zero} introduce a new way to leverage language models for captioning image datasets. They show that this is effective in the context of visual question answering. \textcite{pan-etal-2023-query} leveraged GPT-3 guided CLIP for solving the SemEval 2023 Shared task of Visual Word Sense Disambiguation.
\textcite{dai-etal-2022-enabling} describe a technique for aligning any LLM with a vision dataset. This is useful for preserving an LLMs effectiveness while augmenting it with visual understanding. \textcite{jin-etal-2022-good} look at prompt based low resource learning of visual learning tasks. They show that their technique is more effective than traditional approaches to few-shot visual learning, and results in tiny models.
\textcite{rassin-etal-2022-dalle} Show that DALLE-2 has serious issues with concept mapping in its generations. They show significant inductive biases are introduced as a result of DALLE-2 interpreting words with multiple senses simultaneously.
\section{Techniques}
\subsection{Language Model Choice}
The single most important factor in creating an effective Art Director is the underlying LLM. Commercial LLMs, such as GPT-4\cite{openai2023gpt4}, Anthropic Claude, and Google Bard are still more effective than their locally hosted counterparts out of the box. The huge variety of techniques and settings which are available with locally hosted models make them far more flexible and controllable than closed source alternatives.
Because language models are advancing so quickly, we advocate continuous experimentation. We've found success with the Mistral\cite{jiang2023mistral} and Llama\cite{touvron2023llama} models along with variants like Vicuna\cite{peng2023instruction}. We also find that for situations when we need an LLM to make decisions based on image data, that leveraging visual aligned language models like Minigpt-4\cite{zhu2023minigpt} to caption or describe images is powerful. We also make the following observations: All things being equal, larger parameter count models are preferred to smaller variants, models trained for longer and on more (and better quality) tokens perform increasingly well, and current quantization techniques can reduce LLM output quality more strongly than their benchmarks suggest.
\subsection{Retrieval Augmented Generation}
\textcite{lewis2021retrievalaugmented} introduced a technique called Retrieval Augmented Generation (RAG). Retrieval Augmented Generation is a methodology that combines the power of large-scale retrieval and language models for a particular task. RAG leverages an external vector database or a corpus to find relevant contextual information before the generation process. This information is appended to the language models prompt.
RAG can help improve the quality of images generated by models like Dall-E by providing them with more contextual information derived from a large corpus of data. For instance, if the prompt involves generating an image of "a futuristic cityscape," RAG can pull data on what elements constitute 'futuristic' and 'cityscape,' thereby guiding the model to produce a richer, more accurate image.
RAG is extremely powerful for implementing in-context few-shot learning by leveraging an external prompt database of known good prompts. Instructing the language model in what a good prompt looks like, and then appending 5 known good prompts as examples leads to quality output prompts and correspondingly beautiful images.
\subsection{Fine-Tuning}
Low Rank Adapters (Lora), introduced by \textcite{hu2021lora} is a technique used to fine-tune large pre-trained models efficiently. Instead of fine-tuning all parameters, Lora modifies a low-rank factorized adapter layer, which is significantly smaller in size but still highly expressive. In the context of Stable Diffusion, Lora enables the model to be fine-tuned for specific artistic styles or to generate images of objects/persons not present in the initial training set. The fine-tuning is accomplished with less computational cost while retaining the original model's features. Other competing techniques for fine-tuning the image models include Dreambooth\cite{ruiz2023dreambooth}, Hypernetworks, Textual Inversion\cite{gal2022image}, and traditional fine-tuning.
Lora's low-rank adapter layers also provide benefits in language models. When a language model needs to adapt to specific domain jargon or idiomatic expressions, Lora facilitates this with minimal adjustments to the original parameters, ensuring computational efficiency.
LaDi leverages many fine-tuned Lora models for its constituent Stable Diffusion, AnimateDiff, and Large Language Models. We find that they are extremely valuable due to their compositional nature. Many of them can be used at once and they can be merged together as needed.
\subsection{Prompt Crafting}
One of the most crucial components for optimizing the quality of images generated by text-to-image models like Stable Diffusion is the textual prompt. In this section, we delve into the building blocks of an effective prompt, elucidating how each element contributes to the model's ability to generate high-fidelity, contextually relevant, and aesthetically pleasing images.
A competent prompt is often characterized by its specificity and detail, elements that can be parsed through an extensive checklist of keyword categories. These categories include:
\begin{itemize}
\item Subject: The central focus of the image.
\item Medium: The artistic medium.
\item Style: The artistic style or aesthetic.
\item Artist: Names of artists to emulate.
\item Website: Websites known for a particular style or quality.
\item Resolution: Keywords indicating the sharpness or detail.
\item Color: Dominant or thematic colors.
\item Lighting: The type and quality of lighting.
\item Additional Details: Other elements to enhance the image.
\end{itemize}
Each category serves as a guide for enhancing the expressiveness of a prompt.
Let's consider generating an image of a "steampunk explorer in a mysterious jungle."
\begin{itemize}
\item Subject: "A steampunk explorer with mechanical limbs, wearing leather and gears, standing amidst a mysterious, dense jungle with exotic flora and fauna."
\item Medium: "Digital painting"
\item Style: "Neo-Victorian, adventure-themed"
\item Artist: "Inspired by the styles of Vincent Van Gogh and M.C Escher"
\item Website: "DeviantArt"
\item Resolution: "Highly detailed, with sharp focus"
\item Additional Details: "With a backdrop of an ancient temple and steampunk airships in the sky"
\item Color: "Muted earth tones with bursts of vibrant color"
\item Lighting: "Dappled sunlight filtering through dense foliage"
\end{itemize}
Final Prompt: "A steampunk explorer with mechanical limbs, wearing leather and gears, standing amidst a mysterious, dense jungle with exotic flora and fauna, digital painting, neo-Victorian, adventure-themed, inspired by the styles of H.R. Giger and Brian Froud, DeviantArt, highly detailed with sharp focus, with a backdrop of an ancient temple and steampunk airships in the sky, muted earth tones with bursts of vibrant color, dappled sunlight filtering through dense foliage."
While a prompt doesn't necessarily require every category to be effective, a comprehensive approach can substantially elevate the quality of generated images. Experimentation remains crucial; different combinations yield different results, and users are encouraged to iterate on their prompts to achieve desired outcomes.
Through effective prompting, Stable Diffusion and similar models can create not just accurate representations but truly artistic works that are complex, nuanced, and emotionally evocative.
We tailor LaDi to follow instructions for crafting prompts according to this checklist. We also write instructions which show its LLM how to emphasize/upweight important tokens and de-emphasize/downweight unimportant concepts.
\subsubsection{Effective Prompting for Text-to-Video}
The techniques we have described thus far focus primarily on enhancing image generation with Stable Diffusion. However, our framework for leveraging large language models as Art Directors can extend to other modalities such as text-to-video generation.
One of the most compelling features of text-to-video techniques is prompt traveling, which facilitates smooth visual transitions between different prompts over a sequence of frames. This allows for the creation of animated narratives by progressively modifying elements of the scene. However, a key challenge lies in crafting prompts that can seamlessly morph into one another without visual jarring or incoherence.
LaDi can be readily adapted to generate prompts tailored specifically for prompt traveling in text-to-video models. We optimize the language model to construct prompts with significant lexical and semantic overlap, which aids in the interpolation process. The prompts are structured to feature consistent core subjects while modifying descriptive elements incrementally between keyframes.
For instance, consider generating a video depicting a spaceship traveling through varied cosmic scenery. The LLM Art Director may produce the following prompts for different keyframes:
\begin{itemize}
\item Frame 1: "A sleek silver spaceship gliding through a nebula of purple and blue hues."
\item Frame 15: "The sleek silver spaceship gliding past an asteroid field with giant space rocks."
\item Frame 30: "The same sleek silver spaceship approaching a planet with red and orange rings."
\end{itemize}
Note the consistent use of "sleek silver spaceship" across prompts. By keeping core subjects intact while smoothly modifying details like the backdrop, text-to-video models can readily interpolate between these prompts to animate the space travel sequence.
The language model is fine-tuned using paired samples of prompts that morph in this coherent fashion. It also learns to identify optimal points for prompt changes based on narrative structure. Final videos may contain several seamless prompt transitions driven entirely by AI-generated text prompts.
Our techniques allow leveraging the knowledge and compositional skills of LLMs to unlock the full potential of text-to-video models combined with prompt traveling. The Art Director provides a robust solution to controlling text-to-video synthesis for both practical and creative applications.
\subsection{Classifier-Free Guidance (CFG) and Negative Prompts}
Incorporating the concept of Negative Prompts allows users to specify undesirable features explicitly. For instance, if a user does not want an image to contain "low resolution" or "amateur drawing," these terms are included as negative prompts to guide the generation process.
Stable Diffusion has always had direct support for negative prompts, but up until recently, regular LLMs, such as the LLM generating our Stable Diffusion prompts, did not support negative prompting. \textcite{sanchez2023stay} introduced the concepts of Classifier-Free Guidance and Negative Prompts, techniques known to the image generation community.
CFG works by appending carefully constructed demonstrative examples that exemplify unwanted behaviors. For instance, to avoid generating violent content, sample negative demonstrations like "A man punching another man" could be appended. The LLM learns from these examples during decoding.
We implement CFG by curating labeled datasets of positive and negative prompt demonstrations. The LLM is fine-tuned to differentiate between these examples. During prompt generation, negative demonstrations are dynamically selected via retrieval and concatenated to the context.
This guides the LLM to avoid undesirable attributes in its outputs. CFG allows imposing far more complex constraints compared to simply blacklisting certain tokens. Entire concepts like "poor grammar" or "excessive verbosity" can be discouraged by tailoring demonstrations.
We also experiment with mixing CFG and explicit negative prompts for greater control. Certain immutable tokens like profanities can be blacklisted, while broader attributes are shaped via CFG demonstrations. We find this hybrid approach prevents "leakage" where constraints are circumvented by synonymous words.
LaDi leverages CFG across multiple dimensions relevant to text-to-image generation. We curate demonstrations to discourage prompts with attributes like:
\begin{itemize}
\item Incoherent scene descriptions
\item Ambiguous or contradictory details
\item Overly verbose or rambling sentences
\item Poor artistic composition
\item Grammar and spelling errors
\end{itemize}
By combining CFG and negative prompts, we refine the LaDi's generated prompts to minimize artistic flaws and maximize creative potential. Images produced using prompts crafted with these techniques exhibit significant gains in coherence, relevance, and aesthetic quality.
\subsection{Grammar Based Sampling}
While Large-Language Models have demonstrated robust few-shot learning capabilities across a plethora of tasks, generating structured outputs remains a challenging endeavor. This limitation becomes particularly prominent when attempting to craft precise text prompts for text-to-image generators like Stable Diffusion. To tackle this problem, we leverage formal grammar-based sampling during the decoding process to create highly effective prompts. By enriching the decoding process with formal grammar constraints, we aim to generate sequences that conform to predefined structures essential for specialized tasks like Stable Diffusion prompt generation.
We construct a Context-Free Grammar (CFG) for LaDi tailored for generating Stable Diffusion prompts. The grammar aims to specify the elements of a scene, their attributes, and relationships, as they are pivotal in directing Stable Diffusion towards generating more accurate and relevant images. This CFG is long and beyond the scope of the paper, but we showcase an example CFG below.
Let us define a sample formal grammar \( G \), consisting of:
\begin{itemize}
\item A set of non-terminal symbols \( N = \{ S, \text{Element}, \text{Attribute}, \text{Relation} \} \)
\item A set of terminal symbols \( T = \{ \text{"cat"}, \text{"dog"}, \text{"sitting"}, \text{"jumping"}, \text{"next to"}, \text{"above"} \} \)
\item A set of production rules \( P \)
\item A start symbol \( S \)
\end{itemize}
The production rules \( P \) can be expressed as:
\begin{align*}
& 1. \quad S \rightarrow \text{Element} \\
& 2. \quad S \rightarrow \text{Element} \, \text{Attribute} \\
& 3. \quad S \rightarrow \text{Element} \, \text{Attribute} \, \text{Relation} \, \text{Element} \\
& 4. \quad \text{Element} \rightarrow \text{"cat"} \, | \, \text{"dog"} \\
& 5. \quad \text{Attribute} \rightarrow \text{"sitting"} \, | \, \text{"jumping"} \\
& 6. \quad \text{Relation} \rightarrow \text{"next to"} \, | \, \text{"above"}
\end{align*}
Our framework employs an incremental parser that leverages the CFG rules during the decoding process. At each decoding step, the parser consults the grammar to ascertain which tokens could serve as valid continuations. This narrows down the sequence space to only those that adhere to the CFG rules.
For example, if the partially generated sequence is "cat sitting", the parser will consult the CFG and realize that the only valid continuation would involve a token from the set {"next to", "above"} as specified by the Relation non-terminal. This way, prompts like "cat sitting jumping" would be invalidated by the framework, ensuring higher quality prompts.
With this CFG, a sample valid string (or prompt) generated can be "cat sitting next to dog", which Stable Diffusion can use to create an image of a sitting cat next to a dog.
\subsection{Constrained Beam Search}
Constrained Beam Search is a technique that allows us to direct the text generation process according to certain predetermined conditions or tokens, ensuring that the generated output adheres to these specified sequence level constraints. Given that textual prompts directly influence the visual attributes of images generated by Stable Diffusion, Constrained Beam Search can be extremely useful in achieving a precise kind of visual output.
Huggingface has made this form of Constrained Decoding available for their models for awhile. Beside the default "Disjunctive" and "Phrasal" constraints, users can define their own kinds of constraints.
To make our approach more innovative, we introduce two unique types of constraints: \textit{ColorPatterns} and \textit{SubjectAnchor}.
\begin{itemize}
\item \textbf{ColorPatterns:} This constraint ensures that a specific color pattern or palette is mentioned in the text prompt, thereby directly influencing the color scheme of the generated image. For instance, if we constrain the text prompt to include "sunrise hues," Stable Diffusion is more likely to generate an image with colors resembling an actual sunrise.
\item \textbf{SubjectAnchor:} This constraint ensures that the central subject of the image remains consistent across multiple generations. If the SubjectAnchor is set to "eagle," for example, the image will always feature an eagle as the central subject, regardless of other varying elements.
\end{itemize}
Suppose we want to generate a series of landscape images that not only transition through the four seasons but also feature a consistent element, such as a particular tree or mountain, across the images.
\begin{itemize}
\item \textbf{Constrained Prompt:} The constrained prompt could be initialized with a SubjectAnchor as "oak tree" and a ColorPattern that varies according to the season (e.g., "winter hues" for a snowy scene or "fall colors" for an autumn setting).
\item \textbf{Constrained Beam Search Steps:}
\begin{enumerate}
\item Step 1: Along with typical high-probability tokens, we insert "oak tree" and "winter hues" into the generation candidates.
\item Step 2: We maintain the SubjectAnchor ("oak tree") while changing the ColorPattern to "spring blossoms" for a new iteration, ensuring that while the scene changes to reflect spring, the oak tree remains consistent in the frame.
\end{enumerate}
\item \textbf{Banking and Selection:} The constrained tokens are sorted into banks depending on their steps towards fulfilling the constraint. A round-robin selection ensures that while the constraints are met, the output prompt is also semantically coherent.
\item \textbf{Resulting Prompts:} Possible output prompts might include "An oak tree surrounded by a winter wonderland," followed by "An oak tree amidst a field of spring blossoms."
\end{itemize}
Constrained Beam Search is one form of constraint, but LaDi leverages other kinds as well to further improve the creativity of its prompts.
\subsection{Language Model Probabilistic Programming}
Despite the versatility of large language models in various NLP tasks, generating controlled outputs remains a significant challenge. To mitigate this issue, we leverage an inference-time approach called Sequential Monte Carlo (SMC) steering introduced by \textcite{lew2023sequential}. This method relies on a novel probabilistic programming library, LLaMPPL, for steering LLMs to produce controlled and coherent prompts for image generators.
SMC steering treats language generation tasks as posterior inference problems in discrete probabilistic sequence models. By replacing the standard decoding process with Sequential Monte Carlo inference, it becomes feasible to impose token or sequence level syntactic and semantic constraints on the outputs of LLMs.
LLaMPPL allows users to write probabilistic programs that combine symbolic program logic, probabilistic conditioning, and calls to LLMs. It uses a specialized SMC inference algorithm to solve these tasks, offering an elegant and robust mechanism to steer language models.
Consider a task where the aim is to generate an art piece that depicts a surreal landscape, but under the constraint that the landscape should contain neither water bodies nor any red hues. Generating such an image with Stable Diffusion can be challenging if the text prompt is not carefully managed. By using LLaMPPL and SMC steering, these constraints can be methodically enforced.
We give sample code for leveraging the "Huggingface Probabilistic Programming Language" library to follow SMC steering below. This sample code is modified from examples provided on the projects github repo.
\lstdefinestyle{customPython}{
language=Python,
basicstyle=\ttfamily\small,
commentstyle=\color{olive},
keywordstyle=\color{blue},
numberstyle=\tiny\color{gray},
numbers=left
}
\begin{lstlisting}[style=customPython]
from hfppl import Model, LMContext, TokenCategorical, CachedCausalLM
# Sample forbidden tokens
FORBIDDEN_WORDS = ['water', 'lake', 'ocean', 'river', 'sea', 'red']
# A LLaMPPL model subclasses the Model class
class MyModel(Model):
# The __init__ method is used to process arguments
# and initialize instance variables.
def __init__(self, lm, prompt):
super().__init__()
# A stateful context object for the LLM,
# initialized with the prompt
self.context = LMContext(lm, prompt)
self.lm = lm
# The forbidden tokens
self.forbidden_tokens =
[i for (i, v) in enumerate(lm.vocab)
if any(word in v for word in FORBIDDEN_WORDS)]
# The step method is used to
# perform a single 'step' of generation.
# This might be a single token, a single phrase,
# or any other division.
# Here, we generate one token at a time.
async def step(self):
# Sample a token from the LLM
# We use `await` so that
# LLaMPPL can automatically batch language model calls.
token = await self.sample(self.context.next_token(),
proposal=self.proposal())
self.condition(token.token_id not in self.forbidden_tokens)
# Check for EOS or end of sentence
if token.token_id == self.lm.tokenizer.eos_token_id:
# Finish generation
self.finish()
# Helper method to define a custom proposal
def proposal(self):
logits = self.context.next_token_logprobs.copy()
logits[self.forbidden_tokens] = -float('inf')
return TokenCategorical(self.lm, logits)
def immutable_properties(self):
return set(['forbidden_tokens'])
# Example usage
# Initialize the model with CachedCausalLM
# (or any other compatible LM) and the initial prompt
lm = CachedCausalLM(...) # Initialize your language model here
prompt = "Create a surreal landscape with"
model = MyModel(lm, prompt)
# Run the generation loop
await model.run()
\end{lstlisting}
In this example, the prompt "Create a surreal landscape with" is initiated, but the SMC-steered LLM is conditioned to avoid generating any tokens associated with "water" or "red." The generated text prompts can then be fed into Stable Diffusion, ensuring that the resulting images adhere to the specified artistic constraints. Because SMC modifies future tokens to be more likely to follow constraints than traditional filter-based decoding alone, quality of LLM output is significantly above more naive approaches to constrained text generation.
\section{Conclusion}
In this paper, we have explored a variety of techniques for enhancing text-to-image generators such as Stable Diffusion with intelligent prompting and constrained decoding powered by large language models. Our approaches demonstrate how LLMs can serve as "Art Directors" that guide image generation towards greater relevance, coherence, and artistic merit. We cumulated these approaches into a powerful Art Director called LaDi.
We described the methods used in Ladi including retrieval augmented generation to incorporate external contextual information, fine-tuning with adapters to specialize models for specific domains, crafting detailed prompt checklists, using negative prompts and classifier-free guidance, grammar-based sampling to generate structured prompts, constrained beam search to enforce sequence constraints, and SMC steering with probabilistic programming to methodically direct text generation.
Our techniques aim to elevate text-to-image generation beyond impressive but often inaccurate visuals, towards images that are precise artistic representations of intended concepts and aesthetics. With LLMs as Art Directors, limitations stemming from weak prompts can be mitigated. The solutions presented form a comprehensive framework for users across industries to produce tailored, high-fidelity images that align with creative and practical needs.
There remain opportunities for improvement, including integrating additional modalities beyond text for directing image synthesis, experimenting with other text decoding and image diffusion algorithms, and using more sophisticated models as they are released.
\printbibliography
\end{document}
% $ biblatex auxiliary file $
% $ biblatex bbl format version 3.2 $
% Do not modify the above lines!
%
% This is an auxiliary file used by the 'biblatex' package.
% This file may safely be deleted. It will be recreated by
% biber as required.
%
\begingroup
\makeatletter
\@ifundefined{ver@biblatex.sty}
{\@latex@error
{Missing 'biblatex' package}
{The bibliography requires the 'biblatex' package.}
\aftergroup\endinput}
{}
\endgroup
\refsection{0}
\datalist[entry]{nty/global//global/global}
\entry{dai-etal-2022-enabling}{inproceedings}{}
\name{author}{6}{}{%
{{hash=a8ea213fd5bec89b24e646468ca43fa8}{%
family={Dai},
familyi={D\bibinitperiod},
given={Wenliang},
giveni={W\bibinitperiod}}}%
{{hash=3f700197bc611d3a99c7172d9fff2767}{%
family={Hou},
familyi={H\bibinitperiod},
given={Lu},
giveni={L\bibinitperiod}}}%
{{hash=686657bef97625a531f733bb36f31ce7}{%
family={Shang},
familyi={S\bibinitperiod},
given={Lifeng},
giveni={L\bibinitperiod}}}%
{{hash=45b153f1200605d12615ab51e6c1badf}{%
family={Jiang},
familyi={J\bibinitperiod},
given={Xin},
giveni={X\bibinitperiod}}}%
{{hash=e51b2feac8da559d0f1d95788a6eaea4}{%
family={Liu},
familyi={L\bibinitperiod},
given={Qun},
giveni={Q\bibinitperiod}}}%
{{hash=ea1e0efe4f3dcaf0a50c1119d83f7ca8}{%
family={Fung},
familyi={F\bibinitperiod},
given={Pascale},
giveni={P\bibinitperiod}}}%
}
\list{location}{1}{%
{Dublin, Ireland}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{d13aec82bdf4222f13baaff5b19ad351}
\strng{fullhash}{8da79060c05f24bddf17d1a11a1610de}
\strng{bibnamehash}{d13aec82bdf4222f13baaff5b19ad351}
\strng{authorbibnamehash}{d13aec82bdf4222f13baaff5b19ad351}
\strng{authornamehash}{d13aec82bdf4222f13baaff5b19ad351}
\strng{authorfullhash}{8da79060c05f24bddf17d1a11a1610de}
\field{sortinit}{D}
\field{sortinithash}{6f385f66841fb5e82009dc833c761848}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5{\%} zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with $7\times$ fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.}
\field{booktitle}{Findings of the Association for Computational Linguistics: ACL 2022}
\field{month}{5}
\field{title}{Enabling Multimodal Generation on {CLIP} via Vision-Language Knowledge Distillation}
\field{year}{2022}
\field{pages}{2383\bibrangedash 2395}
\range{pages}{13}
\verb{doi}
\verb 10.18653/v1/2022.findings-acl.187
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2022.findings-acl.187
\endverb
\verb{url}
\verb https://aclanthology.org/2022.findings-acl.187
\endverb
\endentry
\entry{du-etal-2023-zero}{inproceedings}{}
\name{author}{5}{}{%
{{hash=8b52fd6470026437f20bf1d0e1302c61}{%
family={Du},
familyi={D\bibinitperiod},
given={Yifan},
giveni={Y\bibinitperiod}}}%
{{hash=0da53871c5bda3ae9ea7b5d31b6a8874}{%
family={Li},
familyi={L\bibinitperiod},
given={Junyi},
giveni={J\bibinitperiod}}}%
{{hash=e2be20b0e2f21891a3e428cfe3aa7546}{%
family={Tang},
familyi={T\bibinitperiod},
given={Tianyi},
giveni={T\bibinitperiod}}}%
{{hash=64942cca497eff6cd6c5b264afdc5dd6}{%
family={Zhao},
familyi={Z\bibinitperiod},
given={Wayne\bibnamedelima Xin},
giveni={W\bibinitperiod\bibinitdelim X\bibinitperiod}}}%
{{hash=5ff29ba5ea8e6fdd73a84f998eec45f2}{%
family={Wen},
familyi={W\bibinitperiod},
given={Ji-Rong},
giveni={J\bibinithyphendelim R\bibinitperiod}}}%
}
\list{location}{1}{%
{Toronto, Canada}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{77fbf46f54315ecd9cd20bd302fe1b15}
\strng{fullhash}{d51a0239c07626d9bf9350e4c579db77}
\strng{bibnamehash}{77fbf46f54315ecd9cd20bd302fe1b15}
\strng{authorbibnamehash}{77fbf46f54315ecd9cd20bd302fe1b15}
\strng{authornamehash}{77fbf46f54315ecd9cd20bd302fe1b15}
\strng{authorfullhash}{d51a0239c07626d9bf9350e4c579db77}
\field{sortinit}{D}
\field{sortinithash}{6f385f66841fb5e82009dc833c761848}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA). Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-Trained Language model (PLM). As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model. In this way, the captioning model can become aware of the task goal and information need from the PLM. To develop our approach, we design two specific training stages, where the first stage adapts the captioning model to the prediction model (selecting more suitable caption propositions for training) and the second stage tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically, on the challenging A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and even achieves comparable results to a fine-tuned VLP model. Our code is publicly available at \url{https://github.com/RUCAIBox/LAMOC}.}
\field{booktitle}{Findings of the Association for Computational Linguistics: ACL 2023}
\field{month}{7}
\field{title}{Zero-shot Visual Question Answering with Language Model Feedback}
\field{year}{2023}
\field{pages}{9268\bibrangedash 9281}
\range{pages}{14}
\verb{doi}
\verb 10.18653/v1/2023.findings-acl.590
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2023.findings-acl.590
\endverb
\verb{url}
\verb https://aclanthology.org/2023.findings-acl.590
\endverb
\endentry
\entry{gal2022image}{misc}{}
\name{author}{7}{}{%
{{hash=0a0def81bed21b15e2b066183472a9a9}{%
family={Gal},
familyi={G\bibinitperiod},
given={Rinon},
giveni={R\bibinitperiod}}}%
{{hash=b0f3130816096ed324656d246cf676cb}{%
family={Alaluf},
familyi={A\bibinitperiod},
given={Yuval},
giveni={Y\bibinitperiod}}}%
{{hash=c67d556e83f5be598fdc67e2b0a990b5}{%
family={Atzmon},
familyi={A\bibinitperiod},
given={Yuval},
giveni={Y\bibinitperiod}}}%
{{hash=6a8a6c7970ab6d02e599d62882543861}{%
family={Patashnik},
familyi={P\bibinitperiod},
given={Or},
giveni={O\bibinitperiod}}}%
{{hash=aa623ed1cbb6dca55f527748cfff19b6}{%
family={Bermano},
familyi={B\bibinitperiod},
given={Amit\bibnamedelima H.},
giveni={A\bibinitperiod\bibinitdelim H\bibinitperiod}}}%
{{hash=cb9892f91608c4c90c4bd75db5266cdf}{%
family={Chechik},
familyi={C\bibinitperiod},
given={Gal},
giveni={G\bibinitperiod}}}%
{{hash=cdf44d78713bc27a81394749a9d8bbdc}{%
family={Cohen-Or},
familyi={C\bibinithyphendelim O\bibinitperiod},
given={Daniel},
giveni={D\bibinitperiod}}}%
}
\strng{namehash}{5723d7465e7c6044fd25c19065bc6ff9}
\strng{fullhash}{1047eb84d24a91331ff439ee81c732cb}
\strng{bibnamehash}{5723d7465e7c6044fd25c19065bc6ff9}
\strng{authorbibnamehash}{5723d7465e7c6044fd25c19065bc6ff9}
\strng{authornamehash}{5723d7465e7c6044fd25c19065bc6ff9}
\strng{authorfullhash}{1047eb84d24a91331ff439ee81c732cb}
\field{sortinit}{G}
\field{sortinithash}{32d67eca0634bf53703493fb1090a2e8}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion}
\field{year}{2022}
\verb{eprint}
\verb 2208.01618
\endverb
\endentry
\entry{guo2023animatediff}{misc}{}
\name{author}{7}{}{%
{{hash=c074dd5f71ac5ec72d2a943b0a1ec29c}{%
family={Guo},
familyi={G\bibinitperiod},
given={Yuwei},
giveni={Y\bibinitperiod}}}%
{{hash=82f3c5f24c3aac40db5a48f1c7258fba}{%
family={Yang},
familyi={Y\bibinitperiod},
given={Ceyuan},
giveni={C\bibinitperiod}}}%
{{hash=0eb63bbaf2838d9027cabeb7c9cbb26a}{%
family={Rao},
familyi={R\bibinitperiod},
given={Anyi},
giveni={A\bibinitperiod}}}%
{{hash=85c44b90af7ae0ab0977c70ccd95a900}{%
family={Wang},
familyi={W\bibinitperiod},
given={Yaohui},
giveni={Y\bibinitperiod}}}%
{{hash=2addc56ce0ea3a8fc05ff9b0995f74f4}{%
family={Qiao},
familyi={Q\bibinitperiod},
given={Yu},
giveni={Y\bibinitperiod}}}%
{{hash=f4033e88b680ba7279a268e75f80b1d3}{%
family={Lin},
familyi={L\bibinitperiod},
given={Dahua},
giveni={D\bibinitperiod}}}%
{{hash=3e97152a548f1af74e08b5e0c377ef9b}{%
family={Dai},
familyi={D\bibinitperiod},
given={Bo},
giveni={B\bibinitperiod}}}%
}
\strng{namehash}{7c75a152a21a9322a19dfd74b586eb94}
\strng{fullhash}{dfcdf7be05ce7fcc52c48a470ac61cf8}
\strng{bibnamehash}{7c75a152a21a9322a19dfd74b586eb94}
\strng{authorbibnamehash}{7c75a152a21a9322a19dfd74b586eb94}
\strng{authornamehash}{7c75a152a21a9322a19dfd74b586eb94}
\strng{authorfullhash}{dfcdf7be05ce7fcc52c48a470ac61cf8}
\field{sortinit}{G}
\field{sortinithash}{32d67eca0634bf53703493fb1090a2e8}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning}
\field{year}{2023}
\verb{eprint}
\verb 2307.04725
\endverb
\endentry
\entry{ho2022imagen}{misc}{}
\name{author}{11}{}{%
{{hash=2ac2ca10b22e4d13af1767e87495412f}{%
family={Ho},
familyi={H\bibinitperiod},
given={Jonathan},
giveni={J\bibinitperiod}}}%
{{hash=ab1fb800ced7899ad07a1f8377f5c41a}{%
family={Chan},
familyi={C\bibinitperiod},
given={William},
giveni={W\bibinitperiod}}}%
{{hash=5350ba4e3e741e11177e033c49a589f4}{%
family={Saharia},
familyi={S\bibinitperiod},
given={Chitwan},
giveni={C\bibinitperiod}}}%
{{hash=de84205a3a888d56a397b2e41e306955}{%
family={Whang},
familyi={W\bibinitperiod},
given={Jay},
giveni={J\bibinitperiod}}}%
{{hash=b87c3d8f2641c86df63fc585d57c3f6c}{%
family={Gao},
familyi={G\bibinitperiod},
given={Ruiqi},
giveni={R\bibinitperiod}}}%
{{hash=51afcd2097d291bec8732ee4291ae8b4}{%
family={Gritsenko},
familyi={G\bibinitperiod},
given={Alexey},
giveni={A\bibinitperiod}}}%
{{hash=b6fbd171848aad4edf3925543f1f1522}{%
family={Kingma},
familyi={K\bibinitperiod},
given={Diederik\bibnamedelima P.},
giveni={D\bibinitperiod\bibinitdelim P\bibinitperiod}}}%
{{hash=e528fda20644135dfae4f481a342148f}{%
family={Poole},
familyi={P\bibinitperiod},
given={Ben},
giveni={B\bibinitperiod}}}%
{{hash=e34c5aa67281924804d54d88e58ca38a}{%
family={Norouzi},
familyi={N\bibinitperiod},
given={Mohammad},
giveni={M\bibinitperiod}}}%
{{hash=5573293f17e1f12b10be0e811918a6b3}{%
family={Fleet},
familyi={F\bibinitperiod},
given={David\bibnamedelima J.},
giveni={D\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
{{hash=e6f76e1a4d058df028530916774ad3a7}{%
family={Salimans},
familyi={S\bibinitperiod},
given={Tim},
giveni={T\bibinitperiod}}}%
}
\strng{namehash}{d9c72e0919aac578c7e983d804becd3d}
\strng{fullhash}{28cf0728a5a22c95919498ae9b172930}
\strng{bibnamehash}{d9c72e0919aac578c7e983d804becd3d}
\strng{authorbibnamehash}{d9c72e0919aac578c7e983d804becd3d}
\strng{authornamehash}{d9c72e0919aac578c7e983d804becd3d}
\strng{authorfullhash}{28cf0728a5a22c95919498ae9b172930}
\field{sortinit}{H}
\field{sortinithash}{23a3aa7c24e56cfa16945d55545109b5}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{Imagen Video: High Definition Video Generation with Diffusion Models}
\field{year}{2022}
\verb{eprint}
\verb 2210.02303
\endverb
\endentry
\entry{hong-etal-2023-visual-writing}{article}{}
\name{author}{5}{}{%
{{hash=7b54061e58b15228fc15993d6b78f432}{%
family={Hong},
familyi={H\bibinitperiod},
given={Xudong},
giveni={X\bibinitperiod}}}%
{{hash=66340c9b210b1e103c58437681d16c21}{%
family={Sayeed},
familyi={S\bibinitperiod},
given={Asad},
giveni={A\bibinitperiod}}}%
{{hash=f49ce6756f184b1e4645b92240f59257}{%
family={Mehra},
familyi={M\bibinitperiod},
given={Khushboo},
giveni={K\bibinitperiod}}}%
{{hash=353a73e8810fb1660aeecf9a2388b7d1}{%
family={Demberg},
familyi={D\bibinitperiod},
given={Vera},
giveni={V\bibinitperiod}}}%
{{hash=457dd3049a60059cf69e775a47f0c154}{%
family={Schiele},
familyi={S\bibinitperiod},
given={Bernt},
giveni={B\bibinitperiod}}}%
}
\list{location}{1}{%
{Cambridge, MA}%
}
\list{publisher}{1}{%
{MIT Press}%
}
\strng{namehash}{5ca5a51d45ce4c799d6f66ee55194320}
\strng{fullhash}{618b3fb611ef615b20572de792499424}
\strng{bibnamehash}{5ca5a51d45ce4c799d6f66ee55194320}
\strng{authorbibnamehash}{5ca5a51d45ce4c799d6f66ee55194320}
\strng{authornamehash}{5ca5a51d45ce4c799d6f66ee55194320}
\strng{authorfullhash}{618b3fb611ef615b20572de792499424}
\field{sortinit}{H}
\field{sortinithash}{23a3aa7c24e56cfa16945d55545109b5}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them. We improve visual story generation by producing a new image-grounded dataset, Visual Writing Prompts (VWP). VWP contains almost 2K selected sequences of movie shots, each including 5-10 images. The image sequences are aligned with a total of 12K stories which were collected via crowdsourcing given the image sequences and a set of grounded characters from the corresponding image sequence. Our new image sequence collection and filtering process has allowed us to obtain stories that are more coherent, diverse, and visually grounded compared to previous work. We also propose a character-based story generation model driven by coherence as a strong baseline. Evaluations show that our generated stories are more coherent, visually grounded, and diverse than stories generated with the current state-of-the-art model. Our code, image features, annotations and collected stories are available at \url{https://vwprompt.github.io/}.}
\field{journaltitle}{Transactions of the Association for Computational Linguistics}
\field{title}{Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences}
\field{volume}{11}
\field{year}{2023}
\field{pages}{565\bibrangedash 581}
\range{pages}{17}
\verb{doi}
\verb 10.1162/tacl_a_00553
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2023.tacl-1.33
\endverb
\verb{url}
\verb https://aclanthology.org/2023.tacl-1.33
\endverb
\endentry
\entry{hu2021lora}{misc}{}
\name{author}{8}{}{%
{{hash=e6a95d07b8ae7f14e4eb777c6980a059}{%
family={Hu},
familyi={H\bibinitperiod},
given={Edward\bibnamedelima J.},
giveni={E\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
{{hash=1701c0f05662e92cf52cfc8c16437496}{%
family={Shen},
familyi={S\bibinitperiod},
given={Yelong},
giveni={Y\bibinitperiod}}}%
{{hash=ab0fdbc1586103dda7ac5e0852e5b2c3}{%
family={Wallis},
familyi={W\bibinitperiod},
given={Phillip},
giveni={P\bibinitperiod}}}%
{{hash=eaa5cc98b7f19b4ce07c832213a13b9d}{%
family={Allen-Zhu},
familyi={A\bibinithyphendelim Z\bibinitperiod},
given={Zeyuan},
giveni={Z\bibinitperiod}}}%
{{hash=0d2c91b24f885b22d373a892a768abc8}{%
family={Li},
familyi={L\bibinitperiod},
given={Yuanzhi},
giveni={Y\bibinitperiod}}}%
{{hash=b654b88ecd4c3ff1ea29da275bce701c}{%
family={Wang},
familyi={W\bibinitperiod},
given={Shean},
giveni={S\bibinitperiod}}}%
{{hash=55f210233ac686af68eba11a97bd84e2}{%
family={Wang},
familyi={W\bibinitperiod},
given={Lu},
giveni={L\bibinitperiod}}}%
{{hash=05d71a9c5e7ac79222e01a55af0ccb93}{%
family={Chen},
familyi={C\bibinitperiod},
given={Weizhu},
giveni={W\bibinitperiod}}}%
}
\strng{namehash}{d775178c34083951ab1e07df578582c3}
\strng{fullhash}{08f4b49735d5d145be00bc9c07f38975}
\strng{bibnamehash}{d775178c34083951ab1e07df578582c3}
\strng{authorbibnamehash}{d775178c34083951ab1e07df578582c3}
\strng{authornamehash}{d775178c34083951ab1e07df578582c3}
\strng{authorfullhash}{08f4b49735d5d145be00bc9c07f38975}
\field{sortinit}{H}
\field{sortinithash}{23a3aa7c24e56cfa16945d55545109b5}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{LoRA: Low-Rank Adaptation of Large Language Models}
\field{year}{2021}
\verb{eprint}
\verb 2106.09685
\endverb
\endentry
\entry{jiang2023mistral}{misc}{}
\name{author}{18}{}{%
{{hash=a91e014a59af240c37c0fc26432a125b}{%
family={Jiang},
familyi={J\bibinitperiod},
given={Albert\bibnamedelima Q.},
giveni={A\bibinitperiod\bibinitdelim Q\bibinitperiod}}}%
{{hash=2b9ab922e693d9237ad952a8030eee9d}{%
family={Sablayrolles},
familyi={S\bibinitperiod},
given={Alexandre},
giveni={A\bibinitperiod}}}%
{{hash=e9a587c57b1dcb4834850800386d8aca}{%
family={Mensch},
familyi={M\bibinitperiod},
given={Arthur},
giveni={A\bibinitperiod}}}%
{{hash=5d11905f79ca61f62a93a43574139850}{%
family={Bamford},
familyi={B\bibinitperiod},
given={Chris},
giveni={C\bibinitperiod}}}%
{{hash=d61a282b0086c5c4d52c371e7c748278}{%
family={Chaplot},
familyi={C\bibinitperiod},
given={Devendra\bibnamedelima Singh},
giveni={D\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=af4c75452e1c1c77b8727f6dace64a42}{%
family={Casas},
familyi={C\bibinitperiod},
given={Diego},
giveni={D\bibinitperiod},
prefix={de\bibnamedelima las},
prefixi={d\bibinitperiod\bibinitdelim l\bibinitperiod}}}%
{{hash=dafbd4ebc73fd986f482b5cb24ed701d}{%
family={Bressand},
familyi={B\bibinitperiod},
given={Florian},
giveni={F\bibinitperiod}}}%
{{hash=a9285c2c0cdc863e02225039176bb017}{%
family={Lengyel},
familyi={L\bibinitperiod},
given={Gianna},
giveni={G\bibinitperiod}}}%
{{hash=56509a94ba6cdaf0c71304d6cf806cee}{%
family={Lample},
familyi={L\bibinitperiod},
given={Guillaume},
giveni={G\bibinitperiod}}}%
{{hash=1c3686dfd5bade81bcfe2f5c89d70465}{%
family={Saulnier},
familyi={S\bibinitperiod},
given={Lucile},
giveni={L\bibinitperiod}}}%
{{hash=f7808ab68c1e6e06aee1eb1c723ff6eb}{%
family={Lavaud},
familyi={L\bibinitperiod},
given={Lélio\bibnamedelima Renard},
giveni={L\bibinitperiod\bibinitdelim R\bibinitperiod}}}%
{{hash=08284515d256b832e9dba10a88af84f4}{%
family={Lachaux},
familyi={L\bibinitperiod},
given={Marie-Anne},
giveni={M\bibinithyphendelim A\bibinitperiod}}}%
{{hash=0af14ac84bf3a9d753c1ece79a50a55e}{%
family={Stock},
familyi={S\bibinitperiod},
given={Pierre},
giveni={P\bibinitperiod}}}%
{{hash=761b98e2f87c0718576d10e3d2f4028f}{%
family={Scao},
familyi={S\bibinitperiod},
given={Teven\bibnamedelima Le},
giveni={T\bibinitperiod\bibinitdelim L\bibinitperiod}}}%
{{hash=c641a91d616b6eda08cbbdca7b7eb186}{%
family={Lavril},
familyi={L\bibinitperiod},
given={Thibaut},
giveni={T\bibinitperiod}}}%
{{hash=7d3b9bf09e1fd7987770a8954ff63f3c}{%
family={Wang},
familyi={W\bibinitperiod},
given={Thomas},
giveni={T\bibinitperiod}}}%
{{hash=1d29ae55d8d993d9f195b3c97bd34f7c}{%
family={Lacroix},
familyi={L\bibinitperiod},
given={Timothée},
giveni={T\bibinitperiod}}}%
{{hash=c3ed554a7cc845f45e0541d9ed913581}{%
family={Sayed},
familyi={S\bibinitperiod},
given={William\bibnamedelima El},
giveni={W\bibinitperiod\bibinitdelim E\bibinitperiod}}}%
}
\strng{namehash}{066da2bf78c9b2e3d4c5f5a2cc6c4bff}
\strng{fullhash}{6fe7c3f8249d11e6405a6490a9623243}
\strng{bibnamehash}{066da2bf78c9b2e3d4c5f5a2cc6c4bff}
\strng{authorbibnamehash}{066da2bf78c9b2e3d4c5f5a2cc6c4bff}
\strng{authornamehash}{066da2bf78c9b2e3d4c5f5a2cc6c4bff}
\strng{authorfullhash}{6fe7c3f8249d11e6405a6490a9623243}
\field{sortinit}{J}
\field{sortinithash}{b2f54a9081ace9966a7cb9413811edb4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{Mistral 7B}
\field{year}{2023}
\verb{eprint}
\verb 2310.06825
\endverb
\endentry
\entry{jin-etal-2022-good}{inproceedings}{}
\name{author}{5}{}{%
{{hash=6b1734ca45a593dc0fbb8735a241c62d}{%
family={Jin},
familyi={J\bibinitperiod},
given={Woojeong},
giveni={W\bibinitperiod}}}%
{{hash=f90fa099ce585c3d1d009abdb2ccf07c}{%
family={Cheng},
familyi={C\bibinitperiod},
given={Yu},
giveni={Y\bibinitperiod}}}%
{{hash=1701c0f05662e92cf52cfc8c16437496}{%
family={Shen},
familyi={S\bibinitperiod},
given={Yelong},
giveni={Y\bibinitperiod}}}%
{{hash=05d71a9c5e7ac79222e01a55af0ccb93}{%
family={Chen},
familyi={C\bibinitperiod},
given={Weizhu},
giveni={W\bibinitperiod}}}%
{{hash=62341ebcc57e03720c55a39d93f0b17d}{%
family={Ren},
familyi={R\bibinitperiod},
given={Xiang},
giveni={X\bibinitperiod}}}%
}
\list{location}{1}{%
{Dublin, Ireland}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{eb64ca62de8e0824b5da2525f4be0abe}
\strng{fullhash}{ad69cfab4ba6563aa6ff3c87be36b01f}
\strng{bibnamehash}{eb64ca62de8e0824b5da2525f4be0abe}
\strng{authorbibnamehash}{eb64ca62de8e0824b5da2525f4be0abe}
\strng{authornamehash}{eb64ca62de8e0824b5da2525f4be0abe}
\strng{authorfullhash}{ad69cfab4ba6563aa6ff3c87be36b01f}
\field{sortinit}{J}
\field{sortinithash}{b2f54a9081ace9966a7cb9413811edb4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning. However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed. To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners. For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM).Furthermore, we analyze the effect of diverse prompts for few-shot tasks. Experimental results on VQA show that FewVLM with prompt-based learning outperforms Frozen which is 31x larger than FewVLM by 18.2{\%} point and achieves comparable results to a 246x larger model, PICa.In our analysis, we observe that (1) prompts significantly affect zero-shot performance but marginally affect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at \url{https://github.com/woojeongjin/FewVLM}}
\field{booktitle}{Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}
\field{month}{5}
\field{title}{A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models}
\field{year}{2022}
\field{pages}{2763\bibrangedash 2775}
\range{pages}{13}
\verb{doi}
\verb 10.18653/v1/2022.acl-long.197
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2022.acl-long.197
\endverb
\verb{url}
\verb https://aclanthology.org/2022.acl-long.197
\endverb
\endentry
\entry{lew2023sequential}{misc}{}
\name{author}{4}{}{%
{{hash=176f2ce8f6edbda13ed3c51959a4255b}{%
family={Lew},
familyi={L\bibinitperiod},
given={Alexander\bibnamedelima K.},
giveni={A\bibinitperiod\bibinitdelim K\bibinitperiod}}}%
{{hash=e6ec9309c27a8d592d099109edd56872}{%
family={Zhi-Xuan},
familyi={Z\bibinithyphendelim X\bibinitperiod},
given={Tan},
giveni={T\bibinitperiod}}}%
{{hash=86e2cad653dc59f668bdd0e416e7119f}{%
family={Grand},
familyi={G\bibinitperiod},
given={Gabriel},
giveni={G\bibinitperiod}}}%
{{hash=b4a161cc1ca940e73d86b630facd85f5}{%
family={Mansinghka},
familyi={M\bibinitperiod},
given={Vikash\bibnamedelima K.},
giveni={V\bibinitperiod\bibinitdelim K\bibinitperiod}}}%
}
\strng{namehash}{921fc8d953fc3d734577a6c92a0c801d}
\strng{fullhash}{24c8219326bf26bf55ee4e0071a9caae}
\strng{bibnamehash}{921fc8d953fc3d734577a6c92a0c801d}
\strng{authorbibnamehash}{921fc8d953fc3d734577a6c92a0c801d}
\strng{authornamehash}{921fc8d953fc3d734577a6c92a0c801d}
\strng{authorfullhash}{24c8219326bf26bf55ee4e0071a9caae}
\field{sortinit}{L}
\field{sortinithash}{7c47d417cecb1f4bd38d1825c427a61a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.AI}
\field{eprinttype}{arXiv}
\field{title}{Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs}
\field{year}{2023}
\verb{eprint}
\verb 2306.03081
\endverb
\endentry
\entry{lewis2021retrievalaugmented}{misc}{}
\name{author}{12}{}{%
{{hash=ecbe786a5668bf643d74bb24fe1c3cfe}{%
family={Lewis},
familyi={L\bibinitperiod},
given={Patrick},
giveni={P\bibinitperiod}}}%
{{hash=79978685fd7b37606022b06c0f6419a1}{%
family={Perez},
familyi={P\bibinitperiod},
given={Ethan},
giveni={E\bibinitperiod}}}%
{{hash=d205faec67355b6e0ec49f82a492324c}{%
family={Piktus},
familyi={P\bibinitperiod},
given={Aleksandra},
giveni={A\bibinitperiod}}}%
{{hash=e1c32989a78998fe3c03c981d0da2dc7}{%
family={Petroni},
familyi={P\bibinitperiod},
given={Fabio},
giveni={F\bibinitperiod}}}%
{{hash=b24392ebfb2c0a35979fc689c389c649}{%
family={Karpukhin},
familyi={K\bibinitperiod},
given={Vladimir},
giveni={V\bibinitperiod}}}%
{{hash=78a420900f50a8470ad085fb05231bf6}{%
family={Goyal},
familyi={G\bibinitperiod},
given={Naman},
giveni={N\bibinitperiod}}}%
{{hash=4eadcf26f55e11fe4be834b52d9b7fdf}{%
family={Küttler},
familyi={K\bibinitperiod},
given={Heinrich},
giveni={H\bibinitperiod}}}%
{{hash=c97f53fdeeb04c28de19e3324cb4bc0a}{%
family={Lewis},
familyi={L\bibinitperiod},
given={Mike},
giveni={M\bibinitperiod}}}%
{{hash=d4d56d3d370b3afd4ad2ebf985fbd3f8}{%
family={Yih},
familyi={Y\bibinitperiod},
given={Wen-tau},
giveni={W\bibinithyphendelim t\bibinitperiod}}}%
{{hash=b3ff4029884ead9928954ea33714192c}{%
family={Rocktäschel},
familyi={R\bibinitperiod},
given={Tim},
giveni={T\bibinitperiod}}}%
{{hash=6cd024b217f3f2f8126e253b8c84d24d}{%
family={Riedel},
familyi={R\bibinitperiod},
given={Sebastian},
giveni={S\bibinitperiod}}}%
{{hash=ca01c762f4a52dd8a8548508f8d727c9}{%
family={Kiela},
familyi={K\bibinitperiod},
given={Douwe},
giveni={D\bibinitperiod}}}%
}
\strng{namehash}{f238c3bfc4b30650dd17e65c4aadfc66}
\strng{fullhash}{22489b764620b23a8cc8547a492f0763}
\strng{bibnamehash}{f238c3bfc4b30650dd17e65c4aadfc66}
\strng{authorbibnamehash}{f238c3bfc4b30650dd17e65c4aadfc66}
\strng{authornamehash}{f238c3bfc4b30650dd17e65c4aadfc66}
\strng{authorfullhash}{22489b764620b23a8cc8547a492f0763}
\field{sortinit}{L}
\field{sortinithash}{7c47d417cecb1f4bd38d1825c427a61a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks}
\field{year}{2021}
\verb{eprint}
\verb 2005.11401
\endverb
\endentry
\entry{mehrabi-etal-2023-resolving}{inproceedings}{}
\name{author}{10}{}{%
{{hash=e49a830517defa430474d9c2b5c22d83}{%
family={Mehrabi},
familyi={M\bibinitperiod},
given={Ninareh},
giveni={N\bibinitperiod}}}%
{{hash=616affab8b6c99cb76ff4625b67efef9}{%
family={Goyal},
familyi={G\bibinitperiod},
given={Palash},
giveni={P\bibinitperiod}}}%
{{hash=90ef794654722eee6ade6284bdf37dcc}{%
family={Verma},
familyi={V\bibinitperiod},
given={Apurv},
giveni={A\bibinitperiod}}}%
{{hash=9488d4d33f6885ef9bdbc0b6e7861e95}{%
family={Dhamala},
familyi={D\bibinitperiod},
given={Jwala},
giveni={J\bibinitperiod}}}%
{{hash=51d98cb674c9719e186bd9ba5fed0fb0}{%
family={Kumar},
familyi={K\bibinitperiod},
given={Varun},
giveni={V\bibinitperiod}}}%
{{hash=bab6168621b406d25ce5ed61f593762e}{%
family={Hu},
familyi={H\bibinitperiod},
given={Qian},
giveni={Q\bibinitperiod}}}%
{{hash=e92742530fcafbae797b45437638bb01}{%
family={Chang},
familyi={C\bibinitperiod},
given={Kai-Wei},
giveni={K\bibinithyphendelim W\bibinitperiod}}}%
{{hash=77a44166b8233ae29b20f292779447e0}{%
family={Zemel},
familyi={Z\bibinitperiod},
given={Richard},
giveni={R\bibinitperiod}}}%
{{hash=73b781905368dfe2cc66cef5e33e0d32}{%
family={Galstyan},
familyi={G\bibinitperiod},
given={Aram},
giveni={A\bibinitperiod}}}%
{{hash=91614d6ba26aab8a0c8ea57ccbaed0b1}{%
family={Gupta},
familyi={G\bibinitperiod},
given={Rahul},
giveni={R\bibinitperiod}}}%
}
\list{location}{1}{%
{Toronto, Canada}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{cc037d6c925e3502b4bada6900e5b052}
\strng{fullhash}{bb916401d67fe6cc90f5f35d24748b12}
\strng{bibnamehash}{cc037d6c925e3502b4bada6900e5b052}
\strng{authorbibnamehash}{cc037d6c925e3502b4bada6900e5b052}
\strng{authornamehash}{cc037d6c925e3502b4bada6900e5b052}
\strng{authorfullhash}{bb916401d67fe6cc90f5f35d24748b12}
\field{sortinit}{M}
\field{sortinithash}{4625c616857f13d17ce56f7d4f97d451}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Natural language often contains ambiguities that can lead to misinterpretation and miscommunication. While humans can handle ambiguities effectively by asking clarifying questions and/or relying on contextual cues and common-sense knowledge, resolving ambiguities can be notoriously hard for machines. In this work, we study ambiguities that arise in text-to-image generative models. We curate the Text-to-image Ambiguity Benchmark (TAB) dataset to study different types of ambiguities in text-to-image generative models. We then propose the Text-to-ImagE Disambiguation (TIED) framework to disambiguate the prompts given to the text-to-image generative models by soliciting clarifications from the end user. Through automatic and human evaluations, we show the effectiveness of our framework in generating more faithful images aligned with end user intention in the presence of ambiguities.}
\field{booktitle}{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}
\field{month}{7}
\field{title}{Resolving Ambiguities in Text-to-Image Generative Models}
\field{year}{2023}
\field{pages}{14367\bibrangedash 14388}
\range{pages}{22}
\verb{doi}
\verb 10.18653/v1/2023.acl-long.804
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2023.acl-long.804
\endverb
\verb{url}
\verb https://aclanthology.org/2023.acl-long.804
\endverb
\endentry
\entry{openai2023gpt4}{misc}{}
\name{author}{1}{}{%
{{hash=0523b13262b12c215d8009938f5c14f1}{%
family={OpenAI},
familyi={O\bibinitperiod}}}%
}
\strng{namehash}{0523b13262b12c215d8009938f5c14f1}
\strng{fullhash}{0523b13262b12c215d8009938f5c14f1}
\strng{bibnamehash}{0523b13262b12c215d8009938f5c14f1}
\strng{authorbibnamehash}{0523b13262b12c215d8009938f5c14f1}
\strng{authornamehash}{0523b13262b12c215d8009938f5c14f1}
\strng{authorfullhash}{0523b13262b12c215d8009938f5c14f1}
\field{sortinit}{O}
\field{sortinithash}{2cd7140a07aea5341f9e2771efe90aae}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{GPT-4 Technical Report}
\field{year}{2023}
\verb{eprint}
\verb 2303.08774
\endverb
\endentry
\entry{pan-etal-2023-query}{inproceedings}{}
\name{author}{3}{}{%
{{hash=333c41bb0821f5cc76cb3ee3bdfbdd53}{%
family={Pan},
familyi={P\bibinitperiod},
given={Xiaomeng},
giveni={X\bibinitperiod}}}%
{{hash=ab4052ea7f9b61e0e4018ecb591983dd}{%
family={Chen},
familyi={C\bibinitperiod},
given={Zhousi},
giveni={Z\bibinitperiod}}}%
{{hash=4af255ddddb5af27b40314ab6bc1fe96}{%
family={Komachi},
familyi={K\bibinitperiod},
given={Mamoru},
giveni={M\bibinitperiod}}}%
}
\list{location}{1}{%
{Toronto, Canada}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{256f787ef6efded98e4673175e4a2342}
\strng{fullhash}{256f787ef6efded98e4673175e4a2342}
\strng{bibnamehash}{256f787ef6efded98e4673175e4a2342}
\strng{authorbibnamehash}{256f787ef6efded98e4673175e4a2342}
\strng{authornamehash}{256f787ef6efded98e4673175e4a2342}
\strng{authorfullhash}{256f787ef6efded98e4673175e4a2342}
\field{sortinit}{P}
\field{sortinithash}{ff3bcf24f47321b42cb156c2cc8a8422}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{In this study, we propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation (WSD) component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). We confirmed previous findings {---} human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts give the worst results.}
\field{booktitle}{Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)}
\field{month}{7}
\field{title}{Query Generation Using {GPT}-3 for {CLIP}-Based Word Sense Disambiguation for Image Retrieval}
\field{year}{2023}
\field{pages}{417\bibrangedash 422}
\range{pages}{6}
\verb{doi}
\verb 10.18653/v1/2023.starsem-1.36
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2023.starsem-1.36
\endverb
\verb{url}
\verb https://aclanthology.org/2023.starsem-1.36
\endverb
\endentry
\entry{peng2023instruction}{misc}{}
\name{author}{5}{}{%
{{hash=5c8c1bc4adbbd43d1eee7e1621a305a4}{%
family={Peng},
familyi={P\bibinitperiod},
given={Baolin},
giveni={B\bibinitperiod}}}%
{{hash=307cd2ff14b0ecd5a0ae2ca42e7c71b2}{%
family={Li},
familyi={L\bibinitperiod},
given={Chunyuan},
giveni={C\bibinitperiod}}}%
{{hash=c53e3e860dbeebd073b0651c1f711567}{%
family={He},
familyi={H\bibinitperiod},
given={Pengcheng},
giveni={P\bibinitperiod}}}%
{{hash=fda9fdf6ebb4aa0e4f37826ef02391c3}{%
family={Galley},
familyi={G\bibinitperiod},
given={Michel},
giveni={M\bibinitperiod}}}%
{{hash=159bca5607e3797666654a2e3022978a}{%
family={Gao},
familyi={G\bibinitperiod},
given={Jianfeng},
giveni={J\bibinitperiod}}}%
}
\strng{namehash}{5811bf78462796f1a8f7402ca625eb33}
\strng{fullhash}{a09f8fb3d4fe8be8848a326acde3fd30}
\strng{bibnamehash}{5811bf78462796f1a8f7402ca625eb33}
\strng{authorbibnamehash}{5811bf78462796f1a8f7402ca625eb33}
\strng{authornamehash}{5811bf78462796f1a8f7402ca625eb33}
\strng{authorfullhash}{a09f8fb3d4fe8be8848a326acde3fd30}
\field{sortinit}{P}
\field{sortinithash}{ff3bcf24f47321b42cb156c2cc8a8422}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{Instruction Tuning with GPT-4}
\field{year}{2023}
\verb{eprint}
\verb 2304.03277
\endverb
\endentry
\entry{rassin-etal-2022-dalle}{inproceedings}{}
\name{author}{3}{}{%
{{hash=7f7232db303929382a85a946992963f5}{%
family={Rassin},
familyi={R\bibinitperiod},
given={Royi},
giveni={R\bibinitperiod}}}%
{{hash=af55ab54e6ff27527aa5cf74e79c37fc}{%
family={Ravfogel},
familyi={R\bibinitperiod},
given={Shauli},
giveni={S\bibinitperiod}}}%
{{hash=33b00f3e2f4f1bb310f1cf8d4a4c500a}{%
family={Goldberg},
familyi={G\bibinitperiod},
given={Yoav},
giveni={Y\bibinitperiod}}}%
}
\list{location}{1}{%
{Abu Dhabi, United Arab Emirates (Hybrid)}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{a693a4042ef7f7fe6b1537eb2d8fdea2}
\strng{fullhash}{a693a4042ef7f7fe6b1537eb2d8fdea2}
\strng{bibnamehash}{a693a4042ef7f7fe6b1537eb2d8fdea2}
\strng{authorbibnamehash}{a693a4042ef7f7fe6b1537eb2d8fdea2}
\strng{authornamehash}{a693a4042ef7f7fe6b1537eb2d8fdea2}
\strng{authorfullhash}{a693a4042ef7f7fe6b1537eb2d8fdea2}
\field{sortinit}{R}
\field{sortinithash}{5e1c39a9d46ffb6bebd8f801023a9486}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{We study the way DALLE-2 maps symbols (words) in the prompt to their references (entities or properties of entities in the generated image). We show that in stark contrast to the way human process language, DALLE-2 does not follow the constraint that each word has a single role in the interpretation, and sometimes re-use the same symbol for different purposes. We collect a set of stimuli that reflect the phenomenon: we show that DALLE-2 depicts both senses of nouns with multiple senses at once; and that a given word can modify the properties of two distinct entities in the image, or can be depicted as one object and also modify the properties of another object, creating a semantic leakage of properties between entities. Taken together, our study highlights the differences between DALLE-2 and human language processing and opens an avenue for future study on the inductive biases of text-to-image models.}
\field{booktitle}{Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP}
\field{month}{12}
\field{title}{{DALLE}-2 is Seeing Double: Flaws in Word-to-Concept Mapping in {T}ext2{I}mage Models}
\field{year}{2022}
\field{pages}{335\bibrangedash 345}
\range{pages}{11}
\verb{doi}
\verb 10.18653/v1/2022.blackboxnlp-1.28
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2022.blackboxnlp-1.28
\endverb
\verb{url}
\verb https://aclanthology.org/2022.blackboxnlp-1.28
\endverb
\endentry
\entry{rombach2022highresolution}{misc}{}
\name{author}{5}{}{%
{{hash=0921e3810ca1ba112b5f374d82f0735f}{%
family={Rombach},
familyi={R\bibinitperiod},
given={Robin},
giveni={R\bibinitperiod}}}%
{{hash=43a95cf5d0fe271fe6d6d0e5803cbfc9}{%
family={Blattmann},
familyi={B\bibinitperiod},
given={Andreas},
giveni={A\bibinitperiod}}}%
{{hash=809ae73566d8d76e9931ffde3969f1dc}{%
family={Lorenz},
familyi={L\bibinitperiod},
given={Dominik},
giveni={D\bibinitperiod}}}%
{{hash=e61c1cb5e29d764299a12d1ab334cdc2}{%
family={Esser},
familyi={E\bibinitperiod},
given={Patrick},
giveni={P\bibinitperiod}}}%
{{hash=030fdbe4b0bf5c4210fa2e50c87b9ff1}{%
family={Ommer},
familyi={O\bibinitperiod},
given={Björn},
giveni={B\bibinitperiod}}}%
}
\strng{namehash}{b5bfcd95459df0d66c58a299cbec3e9a}
\strng{fullhash}{c1e635e3f33ee547be2644eb3d687673}
\strng{bibnamehash}{b5bfcd95459df0d66c58a299cbec3e9a}
\strng{authorbibnamehash}{b5bfcd95459df0d66c58a299cbec3e9a}
\strng{authornamehash}{b5bfcd95459df0d66c58a299cbec3e9a}
\strng{authorfullhash}{c1e635e3f33ee547be2644eb3d687673}
\field{sortinit}{R}
\field{sortinithash}{5e1c39a9d46ffb6bebd8f801023a9486}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{High-Resolution Image Synthesis with Latent Diffusion Models}
\field{year}{2022}
\verb{eprint}
\verb 2112.10752
\endverb
\endentry
\entry{ruiz2023dreambooth}{misc}{}
\name{author}{6}{}{%
{{hash=a8a6584c0da71baa0b32cfb224a8a521}{%
family={Ruiz},
familyi={R\bibinitperiod},
given={Nataniel},
giveni={N\bibinitperiod}}}%
{{hash=9541f6fdf7e193e3e0ebcbe9b33340ff}{%
family={Li},
familyi={L\bibinitperiod},
given={Yuanzhen},
giveni={Y\bibinitperiod}}}%
{{hash=08e660cb85b2ce40d958c270e2a0d1cb}{%
family={Jampani},
familyi={J\bibinitperiod},
given={Varun},
giveni={V\bibinitperiod}}}%
{{hash=431bc98a2ac62901ba9d38297e19b25e}{%
family={Pritch},
familyi={P\bibinitperiod},
given={Yael},
giveni={Y\bibinitperiod}}}%
{{hash=afbc21c0ba7555cdd6a589cc59eb3c34}{%
family={Rubinstein},
familyi={R\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=769548eecf1ec8f5523b7a76dc6fc0b7}{%
family={Aberman},
familyi={A\bibinitperiod},
given={Kfir},
giveni={K\bibinitperiod}}}%
}
\strng{namehash}{ca5a15d3347ed4d0da8ae3785d341dee}
\strng{fullhash}{a2cf5e47381c83dceecdfabf3ebea7d3}
\strng{bibnamehash}{ca5a15d3347ed4d0da8ae3785d341dee}
\strng{authorbibnamehash}{ca5a15d3347ed4d0da8ae3785d341dee}
\strng{authornamehash}{ca5a15d3347ed4d0da8ae3785d341dee}
\strng{authorfullhash}{a2cf5e47381c83dceecdfabf3ebea7d3}
\field{sortinit}{R}
\field{sortinithash}{5e1c39a9d46ffb6bebd8f801023a9486}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation}
\field{year}{2023}
\verb{eprint}
\verb 2208.12242
\endverb
\endentry
\entry{russo-2022-creative}{inproceedings}{}
\name{author}{1}{}{%
{{hash=250b9a542d1163ec4dd2c70f0d28f047}{%
family={Russo},
familyi={R\bibinitperiod},
given={Irene},
giveni={I\bibinitperiod}}}%
}
\list{location}{1}{%
{Taipei, Taiwan}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{250b9a542d1163ec4dd2c70f0d28f047}
\strng{fullhash}{250b9a542d1163ec4dd2c70f0d28f047}
\strng{bibnamehash}{250b9a542d1163ec4dd2c70f0d28f047}
\strng{authorbibnamehash}{250b9a542d1163ec4dd2c70f0d28f047}
\strng{authornamehash}{250b9a542d1163ec4dd2c70f0d28f047}
\strng{authorfullhash}{250b9a542d1163ec4dd2c70f0d28f047}
\field{sortinit}{R}
\field{sortinithash}{5e1c39a9d46ffb6bebd8f801023a9486}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Language models for text-to-image generation can output good quality images when referential aspects of pictures are evaluated. The generation of creative images is not under scrutiny at the moment, but it poses interesting challenges: should we expect more creative images using more creative prompts? What is the relationship between prompts and images in the global process of human evaluation? In this paper, we want to highlight several criteria that should be taken into account for building a creative text-to-image generation benchmark, collecting insights from multiple disciplines (e.g., linguistics, cognitive psychology, philosophy, psychology of art).}
\field{booktitle}{Proceedings of the 2nd International Workshop on Natural Language Processing for Digital Humanities}
\field{month}{11}
\field{title}{Creative Text-to-Image Generation: Suggestions for a Benchmark}
\field{year}{2022}
\field{pages}{145\bibrangedash 154}
\range{pages}{10}
\verb{urlraw}
\verb https://aclanthology.org/2022.nlp4dh-1.18
\endverb
\verb{url}
\verb https://aclanthology.org/2022.nlp4dh-1.18
\endverb
\endentry
\entry{sanchez2023stay}{misc}{}
\name{author}{6}{}{%
{{hash=9536ce7ba49fc44ea13939fd220f89c4}{%
family={Sanchez},
familyi={S\bibinitperiod},
given={Guillaume},
giveni={G\bibinitperiod}}}%
{{hash=5387a465674c61b960628b3633d4cfbf}{%
family={Fan},
familyi={F\bibinitperiod},
given={Honglu},
giveni={H\bibinitperiod}}}%
{{hash=acc905db580ac6ea3b904678710db46c}{%
family={Spangher},
familyi={S\bibinitperiod},
given={Alexander},
giveni={A\bibinitperiod}}}%
{{hash=8efa11b26a194c2f7e856fa7de77b510}{%
family={Levi},
familyi={L\bibinitperiod},
given={Elad},
giveni={E\bibinitperiod}}}%
{{hash=6911217448da15dba5f800d874497315}{%
family={Ammanamanchi},
familyi={A\bibinitperiod},
given={Pawan\bibnamedelima Sasanka},
giveni={P\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=865b3a07920bcd999406ae3062b413fa}{%
family={Biderman},
familyi={B\bibinitperiod},
given={Stella},
giveni={S\bibinitperiod}}}%
}
\strng{namehash}{b6bcee3e1576a4186d6fa633fe07cf28}
\strng{fullhash}{af55eeaa55f21f83d18530089283749c}
\strng{bibnamehash}{b6bcee3e1576a4186d6fa633fe07cf28}
\strng{authorbibnamehash}{b6bcee3e1576a4186d6fa633fe07cf28}
\strng{authornamehash}{b6bcee3e1576a4186d6fa633fe07cf28}
\strng{authorfullhash}{af55eeaa55f21f83d18530089283749c}
\field{sortinit}{S}
\field{sortinithash}{b164b07b29984b41daf1e85279fbc5ab}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{Stay on topic with Classifier-Free Guidance}
\field{year}{2023}
\verb{eprint}
\verb 2306.17806
\endverb
\endentry
\entry{singer2022makeavideo}{misc}{}
\name{author}{13}{}{%
{{hash=490b1b9a870cc899f1e279cac96865ef}{%
family={Singer},
familyi={S\bibinitperiod},
given={Uriel},
giveni={U\bibinitperiod}}}%
{{hash=6dcdd75f3bef38d7160891406eb9ac50}{%
family={Polyak},
familyi={P\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=a8b5b0082392b8cdb81e0cc8e3506b2f}{%
family={Hayes},
familyi={H\bibinitperiod},
given={Thomas},
giveni={T\bibinitperiod}}}%
{{hash=af5100e70bae33651be44cc2fd0e83b8}{%
family={Yin},
familyi={Y\bibinitperiod},
given={Xi},
giveni={X\bibinitperiod}}}%
{{hash=9af57fbe394eda13edf82eab186d92c6}{%
family={An},
familyi={A\bibinitperiod},
given={Jie},
giveni={J\bibinitperiod}}}%
{{hash=d54f5efb2ce3e8147bbe60b3fa1ffce5}{%
family={Zhang},
familyi={Z\bibinitperiod},
given={Songyang},
giveni={S\bibinitperiod}}}%
{{hash=160d9da0cf55eb2295d19f23a9676225}{%
family={Hu},
familyi={H\bibinitperiod},
given={Qiyuan},
giveni={Q\bibinitperiod}}}%
{{hash=fbc6798e044af0f76d264522ce9be873}{%
family={Yang},
familyi={Y\bibinitperiod},
given={Harry},
giveni={H\bibinitperiod}}}%
{{hash=9264991546550932a456c5ec7f257a50}{%
family={Ashual},
familyi={A\bibinitperiod},
given={Oron},
giveni={O\bibinitperiod}}}%
{{hash=aa31838861c30a9642c1a0532d73eeea}{%
family={Gafni},
familyi={G\bibinitperiod},
given={Oran},
giveni={O\bibinitperiod}}}%
{{hash=d0ab7a4601951a0c59352f1dcb531afb}{%
family={Parikh},
familyi={P\bibinitperiod},
given={Devi},
giveni={D\bibinitperiod}}}%
{{hash=545cf31b67044795c5772570c35f71f0}{%
family={Gupta},
familyi={G\bibinitperiod},
given={Sonal},
giveni={S\bibinitperiod}}}%
{{hash=eda7501e3a7cf6a1f64312f184055874}{%
family={Taigman},
familyi={T\bibinitperiod},
given={Yaniv},
giveni={Y\bibinitperiod}}}%
}
\strng{namehash}{2ffc28bc10907026dd32a82386902e09}
\strng{fullhash}{6b4a32ed9147c5e8dbd0731e4506adb9}
\strng{bibnamehash}{2ffc28bc10907026dd32a82386902e09}
\strng{authorbibnamehash}{2ffc28bc10907026dd32a82386902e09}
\strng{authornamehash}{2ffc28bc10907026dd32a82386902e09}
\strng{authorfullhash}{6b4a32ed9147c5e8dbd0731e4506adb9}
\field{sortinit}{S}
\field{sortinithash}{b164b07b29984b41daf1e85279fbc5ab}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{Make-A-Video: Text-to-Video Generation without Text-Video Data}
\field{year}{2022}
\verb{eprint}
\verb 2209.14792
\endverb
\endentry
\entry{touvron2023llama}{misc}{}
\name{author}{14}{}{%
{{hash=a2a13fcd633ec70c55a7f3f4812afdcf}{%
family={Touvron},
familyi={T\bibinitperiod},
given={Hugo},
giveni={H\bibinitperiod}}}%
{{hash=c641a91d616b6eda08cbbdca7b7eb186}{%
family={Lavril},
familyi={L\bibinitperiod},
given={Thibaut},
giveni={T\bibinitperiod}}}%
{{hash=a32ece3bc966e7e88747e905158c9a6a}{%
family={Izacard},
familyi={I\bibinitperiod},
given={Gautier},
giveni={G\bibinitperiod}}}%
{{hash=7222f9d76b0fff504f39e7c9c2a526be}{%
family={Martinet},
familyi={M\bibinitperiod},
given={Xavier},
giveni={X\bibinitperiod}}}%
{{hash=08284515d256b832e9dba10a88af84f4}{%
family={Lachaux},
familyi={L\bibinitperiod},
given={Marie-Anne},
giveni={M\bibinithyphendelim A\bibinitperiod}}}%
{{hash=1d29ae55d8d993d9f195b3c97bd34f7c}{%
family={Lacroix},
familyi={L\bibinitperiod},
given={Timothée},
giveni={T\bibinitperiod}}}%
{{hash=e0d9a8657c1043f53c54f3f5a6ff305f}{%
family={Rozière},
familyi={R\bibinitperiod},
given={Baptiste},
giveni={B\bibinitperiod}}}%
{{hash=78a420900f50a8470ad085fb05231bf6}{%
family={Goyal},
familyi={G\bibinitperiod},
given={Naman},
giveni={N\bibinitperiod}}}%
{{hash=00ce63f55ae2ce6ea638e5f991996584}{%
family={Hambro},
familyi={H\bibinitperiod},
given={Eric},
giveni={E\bibinitperiod}}}%
{{hash=5373a3313da61427121b3e9733df1ed2}{%
family={Azhar},
familyi={A\bibinitperiod},
given={Faisal},
giveni={F\bibinitperiod}}}%
{{hash=a829e25f1d50c3545ed8e99f12890acc}{%
family={Rodriguez},
familyi={R\bibinitperiod},
given={Aurelien},
giveni={A\bibinitperiod}}}%
{{hash=977d047821122d1c2e7aa855c30c8cf2}{%
family={Joulin},
familyi={J\bibinitperiod},
given={Armand},
giveni={A\bibinitperiod}}}%
{{hash=ebdea2c3aa9f759075733b20dce6b873}{%
family={Grave},
familyi={G\bibinitperiod},
given={Edouard},
giveni={E\bibinitperiod}}}%
{{hash=56509a94ba6cdaf0c71304d6cf806cee}{%
family={Lample},
familyi={L\bibinitperiod},
given={Guillaume},
giveni={G\bibinitperiod}}}%
}
\strng{namehash}{c237483431fea681283a37adbb23a59f}
\strng{fullhash}{39aeaa201007bcb705db32778f923cb2}
\strng{bibnamehash}{c237483431fea681283a37adbb23a59f}
\strng{authorbibnamehash}{c237483431fea681283a37adbb23a59f}
\strng{authornamehash}{c237483431fea681283a37adbb23a59f}
\strng{authorfullhash}{39aeaa201007bcb705db32778f923cb2}
\field{sortinit}{T}
\field{sortinithash}{9af77f0292593c26bde9a56e688eaee9}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CL}
\field{eprinttype}{arXiv}
\field{title}{LLaMA: Open and Efficient Foundation Language Models}
\field{year}{2023}
\verb{eprint}
\verb 2302.13971
\endverb
\endentry
\entry{wang2023modelscope}{misc}{}
\name{author}{6}{}{%
{{hash=a6f677c72610898d0619463b0c10824a}{%
family={Wang},
familyi={W\bibinitperiod},
given={Jiuniu},
giveni={J\bibinitperiod}}}%
{{hash=a6e822144809a0ff6d35cb9c14b7f66f}{%
family={Yuan},
familyi={Y\bibinitperiod},
given={Hangjie},
giveni={H\bibinitperiod}}}%
{{hash=72e66ed72ba2dc71e0f8c5d79abd3b7e}{%
family={Chen},
familyi={C\bibinitperiod},
given={Dayou},
giveni={D\bibinitperiod}}}%
{{hash=15be1b8d7b77e1d0cd6ceb46148bc21a}{%
family={Zhang},
familyi={Z\bibinitperiod},
given={Yingya},
giveni={Y\bibinitperiod}}}%
{{hash=8fde8559617f603b3e2b3cf30cb48e9b}{%
family={Wang},
familyi={W\bibinitperiod},
given={Xiang},
giveni={X\bibinitperiod}}}%
{{hash=d158b265cf5ee9e7959e886a2c0cc1a5}{%
family={Zhang},
familyi={Z\bibinitperiod},
given={Shiwei},
giveni={S\bibinitperiod}}}%
}
\strng{namehash}{79cbb8f345a5fc8b2005f58e7b32986a}
\strng{fullhash}{a75a9f7cb15a6f4bddab8d4e693d10da}
\strng{bibnamehash}{79cbb8f345a5fc8b2005f58e7b32986a}
\strng{authorbibnamehash}{79cbb8f345a5fc8b2005f58e7b32986a}
\strng{authornamehash}{79cbb8f345a5fc8b2005f58e7b32986a}
\strng{authorfullhash}{a75a9f7cb15a6f4bddab8d4e693d10da}
\field{extraname}{1}
\field{sortinit}{W}
\field{sortinithash}{4315d78024d0cea9b57a0c6f0e35ed0d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{eprintclass}{cs.CV}
\field{eprinttype}{arXiv}
\field{title}{ModelScope Text-to-Video Technical Report}
\field{year}{2023}
\verb{eprint}
\verb 2308.06571
\endverb
\endentry
\entry{wang-etal-2023-diffusiondb}{inproceedings}{}
\name{author}{6}{}{%
{{hash=f9dabae886f8f4ca520b3d5721726138}{%
family={Wang},
familyi={W\bibinitperiod},
given={Zijie\bibnamedelima J.},
giveni={Z\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
{{hash=96ab27b6ffb83fb54794f740e784ca98}{%
family={Montoya},
familyi={M\bibinitperiod},
given={Evan},
giveni={E\bibinitperiod}}}%
{{hash=d23eb33784b69092d8b70d85d26ebdc0}{%
family={Munechika},
familyi={M\bibinitperiod},
given={David},
giveni={D\bibinitperiod}}}%
{{hash=788bcd12b5865c3cb2afc442391ccdd3}{%
family={Yang},
familyi={Y\bibinitperiod},
given={Haoyang},
giveni={H\bibinitperiod}}}%
{{hash=bf17f8c0acea31ab679cbcbd83151ef1}{%
family={Hoover},
familyi={H\bibinitperiod},
given={Benjamin},
giveni={B\bibinitperiod}}}%
{{hash=f4c4b4c06aa7b1d5c9f6c1fc6c256fa1}{%
family={Chau},
familyi={C\bibinitperiod},
given={Duen\bibnamedelima Horng},
giveni={D\bibinitperiod\bibinitdelim H\bibinitperiod}}}%
}
\list{location}{1}{%
{Toronto, Canada}%
}
\list{publisher}{1}{%
{Association for Computational Linguistics}%
}
\strng{namehash}{545cba78cf8878fb208fc5a684f5fd6c}
\strng{fullhash}{7c69302bb027fed83c926baf2215b08c}
\strng{bibnamehash}{545cba78cf8878fb208fc5a684f5fd6c}
\strng{authorbibnamehash}{545cba78cf8878fb208fc5a684f5fd6c}
\strng{authornamehash}{545cba78cf8878fb208fc5a684f5fd6c}
\strng{authorfullhash}{7c69302bb027fed83c926baf2215b08c}
\field{extraname}{2}
\field{sortinit}{W}
\field{sortinithash}{4315d78024d0cea9b57a0c6f0e35ed0d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{With recent advancements in diffusion models, users can generate high-quality images by writing text prompts in natural language. However, generating images with desired details requires proper prompts, and it is often unclear how a model reacts to different prompts or what the best prompts are. To help researchers tackle these critical challenges, we introduce DiffusionDB, the first large-scale text-to-image prompt dataset totaling 6.5TB, containing 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. We analyze the syntactic and semantic characteristics of prompts. We pinpoint specific hyperparameter values and prompt styles that can lead to model errors and present evidence of potentially harmful model usage, such as the generation of misinformation. The unprecedented scale and diversity of this human-actuated dataset provide exciting research opportunities in understanding the interplay between prompts and generative models, detecting deepfakes, and designing human-AI interaction tools to help users more easily use these models. DiffusionDB is publicly available at: \url{https://poloclub.github.io/diffusiondb}.}
\field{booktitle}{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}
\field{month}{7}
\field{title}{{D}iffusion{DB}: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models}
\field{year}{2023}
\field{pages}{893\bibrangedash 911}
\range{pages}{19}
\verb{doi}
\verb 10.18653/v1/2023.acl-long.51
\endverb
\verb{urlraw}
\verb https://aclanthology.org/2023.acl-long.51
\endverb
\verb{url}
\verb https://aclanthology.org/2023.acl-long.51
\endverb
\endentry
\entry{zhu2023minigpt}{article}{}
\name{author}{5}{}{%
{{hash=f95538c322c9f75110e8aef4c56c547d}{%
family={Zhu},
familyi={Z\bibinitperiod},
given={Deyao},
giveni={D\bibinitperiod}}}%
{{hash=af06704764d8034ca33edc1493e66b7b}{%
family={Chen},
familyi={C\bibinitperiod},
given={Jun},
giveni={J\bibinitperiod}}}%
{{hash=140b667db10220adc36781b9776dc90f}{%
family={Shen},
familyi={S\bibinitperiod},
given={Xiaoqian},
giveni={X\bibinitperiod}}}%
{{hash=09e1a64c667557dd7866bc4e66d193ce}{%
family={Li},
familyi={L\bibinitperiod},
given={Xiang},
giveni={X\bibinitperiod}}}%
{{hash=a62fa329349893dea816e7b533904ff4}{%
family={Elhoseiny},
familyi={E\bibinitperiod},
given={Mohamed},
giveni={M\bibinitperiod}}}%
}
\strng{namehash}{c05c169829c9bd6b6c183c4d298bb1b2}
\strng{fullhash}{690bbeb0417bc72fac41b675aaf93e40}
\strng{bibnamehash}{c05c169829c9bd6b6c183c4d298bb1b2}
\strng{authorbibnamehash}{c05c169829c9bd6b6c183c4d298bb1b2}
\strng{authornamehash}{c05c169829c9bd6b6c183c4d298bb1b2}
\strng{authorfullhash}{690bbeb0417bc72fac41b675aaf93e40}
\field{sortinit}{Z}
\field{sortinithash}{96892c0b0a36bb8557c40c49813d48b3}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{arXiv preprint arXiv:2304.10592}
\field{title}{MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models}
\field{year}{2023}
\endentry
\enddatalist
\endrefsection
\endinput