Attention, not scale, drives human-AI alignment in multimodal language prediction
Extracted source text · arXiv 2308.06035
\documentclass[10pt]{article}
\usepackage[a4paper,hmargin=3cm,vmargin=3.0cm,includeheadfoot]{geometry}
\usepackage[english]{babel}
\usepackage{float}
\usepackage[font=small]{caption}
\usepackage[affil-it]{authblk}
\usepackage{hyperref}
% For superscript citations
\usepackage[super]{natbib}
\bibliographystyle{unsrtnat}
% Useful packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc} % assuming your file is encoded in UTF-8
\title{Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models}
\author[1,*]{Viktor Kewenig}
\author[2]{Andrew Lampinen}
\author[3]{Samuel A. Nastase}
\author[1]{Christopher Edwards}
\author[4]{Quitterie D'Elascombe}
\author[1]{Akilles Richardt}
\author[1]{Jeremy I. Skipper}
\author[1]{Gabriella Vigliocco}
\affil[1]{Psychology and Language Science, Experimental Psychology, University College London, London, UK}
\affil[2]{Independent}
\affil[3]{Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA}
\affil[4]{Computer Science Department, Exeter University}
\affil[*]{Corresponding author: ucjuvnk@ucl.ac.uk}
\begin{document}
\date{}
\maketitle
\section*{Abstract}
The potential of multimodal generative artificial intelligence (mAI) to replicate human grounded language understanding, including the pragmatic, context-rich aspects of communication, remains to be clarified. Humans are known to use salient multimodal features, such as visual cues, to facilitate the processing of upcoming words. Correspondingly, multimodal computational models can integrate visual and linguistic data using a visual attention mechanism to assign next-word probabilities. To test whether these processes align, we tasked both human participants (N = 200) as well as several state-of-the-art computational models with evaluating the predictability of forthcoming words after viewing short audio-only or audio-visual clips with speech. During the task, the model's attention weights were recorded and human attention was indexed via eye tracking. Results show that predictability estimates from humans aligned more closely with scores generated from multimodal models vs. their unimodal counterparts. Furthermore, including an attention mechanism doubled alignment with human judgments when visual and linguistic context facilitated predictions. In these cases, the model's attention patches and human eye tracking significantly overlapped. Our results indicate that improved modeling of naturalistic language processing in mAI does not merely depend on training diet but can be driven by multimodality in combination with attention-based architectures. Humans and computational models alike can leverage the predictive constraints of multimodal information by attending to relevant features in the input.
\section*{Introduction}
Recent advances in generative artificial intelligence (AI) have stimulated debate over the extent to which models reproduce certain human cognitive processes.\cite{mitchell_debate_2023,mahowald_dissociating_2023,liu_brainclip_2023,kauf_event_2022,zhang_visual_2022,michelmann_large_2023} Direct brain-model comparisons suggest that hierarchical predictive processing, driven by contextual linguistic information, underlies language processing in both humans and AI. \cite{schrimpf_neural_2021,goldstein_shared_2022,caucheteux_long-range_2021}
One limitation of many of these studies is that they focus on unimodal models (text-based) and unimodal (text- or audio-based) experimental stimuli (but see \citet{dong_vision-language_2023}). Yet in ecological settings, human language input is often grounded in one or more perceptual modalities.\cite{vigliocco2014language} Do multimodal models replicate human grounded language predictions better than their unimodal counterparts? If so, what architectural features support this alignment?
Theory and empirical evidence suggest that multimodal grounding is the basis for the formation of internal mental models of external categories, situations, and events.\cite{zwaan_construction_1995, zwaan_embodiment_2014, zwaan_situation_2016} These `Situation models' facilitate the integration of linguistic information with contextual information (such as visual context) or internal priors for next-word prediction by placing constraints on the space of upcoming semantic content, reducing cognitive load and improving comprehension.\cite{pickering_predicting_2018,willems_nolb_2015,huettig_using_2011,kuperberg_what_2016} Indeed, there is ample behavioural\cite{paivio_imagery_2013, barsalou_perceptual_1999, barsalou_grounded_2008, vigliocco_toward_2009, vigliocco_language_2014, andrews_integrating_2009, andrews_reconciling_2014, louwerse_linguistic_2010, kousta_representation_2011, riordan_redundancy_2011} and neurobiological\cite{kewenig_when_2023, hahamy_human_2023, zadbood_neural_2022, baldassano_representation_2018, ben-yakov_hippocampal_2018} evidence that humans process linguistic and nonlinguistic input jointly and immediately and that naturalistic language processing involves constructing situation models using both language and multimodal information in real time.
Given that human language processing seems to be informed by non-linguistic inputs and computations, grounding by means of adding non-linguistic modalities to the embedding space in computational language models may present a way of aligning model predictions more closely with human processing. It is an open question whether this could eventually bridge the apparent gap between state-of-the-art, disembodied artificial intelligence and situated, embodied human agents.\cite{pezzulo_generating_2023} Current efforts toward multimodal models have begun combining images and text (e.g., \citet{alayrac2022flamingo}). This presents a good starting point, as there is a history of work analysing the connection between visual information and grounded, predictive language processing in humans.\cite{bransford_sentence_1972, barsalou_grounded_2008, huettig_four_2015} In contrast to unimodal LLMs, such multimodal computational models are trained on datasets that include image/sentence pairs and possess the capability to process both images and text.
The capacity to process both visual and linguistic input by itself, however, may provide little benefit for grounding predictive language processing if no attention is paid to salient visual cues, effectively narrowing down the space of upcoming semantic content. Indeed, attention to salient multimodal cues is a key part of constructing online situation models. Evidence for this comes from studies using some versions of the so-called Visual World Paradigm (VWP), a classic behavioural paradigm for probing the online integration of linguistic and visual information as indexed by eye movements.\cite{tanenhaus_integration_1995} In a typical VWP study, participants hear an unfolding sentence and see different objects/scenes on a screen,\cite{huettig_using_2011} or even in virtual reality.\cite{eichert_language-driven_2018} This paradigm has robustly shown that participants' eye movements index word prediction. For example, just after hearing `the man will eat...', participants look at a cake, rather than non-edible distracters before the continuation of the sentence is uttered.\cite{altmann_incremental_1999} Similar studies have consistently found that when visual information is critical for predicting upcoming sentence content, participants tend to direct their eye movements towards the relevant visual information even before hearing the associated word.\cite{altmann_incremental_1999} This suggests that attention to individual visual cues is linked to the online construal of situation models during predictive language processing.\cite{kamide_integration_2003,kukona_knowing_2014,rommers_verbal_2015}
Recent advances in natural language processing and computer vision have culminated in the transformer architecture, which similarly selects relevant information—linguistic tokens or visual pixels from the surrounding context using an `attention' mechanism.\cite{vaswani_attention_2017} This mechanism is not explicitly modelled on human attention. For example, transformers have multiple attention heads and can simultaneously attend to many image patches at once, whereas human attention has a more limited capacity,\cite{serrano_is_2019,da_costa_assessing_2020} though it is unclear how many targets human attention can track simultaneously.\cite{busse_spread_2005} Instead, the design considerations for self-attention in transformers are primarily driven by engineering considerations (such as enabling parallel processing) and performance benchmarks against other models.\cite{tay_synthesizer_2021, lindsay_attention_2020} In fact, the degree of alignment with human brain data for vision-only models (both convolutional neural networks and visual transformers) is said to depend less on architectural features and more on their training diet (i.e. the quantity and quality of training data \cite{conwell2022billion}). On the other hand, visual transformers exhibit more human-like image classification compared to convolutional neural nets without attention.\cite{dosovitskiy_image_2021} \citet{tuli_are_2021} argue that this effect `could possibly be explained by the nature of attention models that permits focus on the part of the image that is important for the given task and neglect the otherwise noisy background to make predictions'. Furthermore, text-based transformer models have shown some similarity in attention patterns to human gaze during reading tasks. \cite{sood_interpreting_2020, yang_predicting_2023} However, no previous study has explored the impact of including a transformer vision backbone within a multimodal computational model. Specifically, there is a gap in understanding how visual attention might impact alignment with human natural language predictions in situations that include both visual and linguistic input (such as the VWP).
Here, we pursue three related hypotheses: (1) Multimodal computational models can use contextual visual and linguistic information to extract sensible semantic predictions that align more closely with human processing compared to their unimodal counterparts basing predictions solely on textual information. (2) The addition of a visual attention backbone to a multimodal computational model further enhances human-like natural language predictions. (3) During processing, the model's attention patterns will correlate with human attention patterns when salient visual cues for upcoming content are present. To investigate our hypotheses, we compared predictability scores from a variety of models with human estimates.
\section*{Results}
Figure (1, \textit{Unimodal Methods}) describes how we extracted predictions from different unimodal models and how we collected predictability estimates with audio-based stimuli from human participants. Figure (1, \textit{Multimodal Methods}) shows multimodal versions of the same models (with and without attention for one of the models) as well as our experiment with audio-visual-based stimuli. In the online experiment, humans were exposed to 100 six-second audio-visual or audio-only clips from two films, ‘The Prestige’ and ‘The Usual Suspects’. For each stimulus type, 200 participants were shown a target word and brief instructions (Figure 1a), followed by the six-second audio or audio-visual clip (Figure 1b and 1e). Participants were then asked to rate, on a scale from zero (not at all relevant) to 100 (highly relevant), how pertinent the information in the video or audio-visual clip was for predicting the upcoming word in the film's dialogue (Figure 1 Unimodal Methods (c) and Multimodal Methods (f)). Throughout the 100 trials conducted for each participant, eye gaze was also recorded via the participant's webcam.
\newpage
\thispagestyle{empty}
\begin{figure}[H]
\centering
\includegraphics[width=0.9\linewidth]{Methods_All.png}
\caption{\textit{Unimodal Methods:} (First column) For the first probability measure, both the incoming dialogue (input) and all labels in the movie (a) are encoded by CLIP's text encoder in the `text only' version of the model (b). Predictability is derived as the softmaxed similarity scores (over all labels) between the upcoming label and the resulting encodings (c). (Second column) For the second probability measure, the textual input (a) is fed directly to LLaMA (b). Predictability is derived by pulling out the next-word logits from the model's forward method for all labels in the movie and applying a softmax over them to obtain a probability distribution (c). (Third column) For the prompt-based measure, the textual input is combined with a prompt asking to estimate the predictability of the upcoming word, which is processed by the model (b) and results in a direct prompt-based output measure (c). While GPT-4 and LLaMA are from the same model family, GPT-4 has many more parameters than LLaMA. (Fourth Column) For the human measure, instructions are presented to human participants (similar to the prompt used in the prompt-based measure) (a) before they listen to an audio clip (b) and provide predictability estimates on a Likert scale (c) from 0 (Low Relevance) to 100 (High Relevance). \textit{Multimodal Methods:} (First column) For the first direct probability measure, both the incoming dialogue (input) and all labels in the movie (d) are encoded by CLIP's text encoder. The visual information (frame-by-frame) is encoded by the visual transformer backbone (we used both the `ViT-32' and the `RN50' versions of the model) (e). Predictability is derived as the softmaxed similarity scores (over all labels in the movie) between the upcoming label and the resulting multimodal encodings (f). (Second column) For the second direct probability measure, visual input was fed frame-by-frame to the adapter layer. Textual input was fed to the LLaMA model directly (d). Both text and visual information were then processed by the model (e). Predictability scores were derived as the softmaxed next-word logits for all labels in the movie's dialogue (f). (Third column) For the prompt-based measure, the visual input was fed as a GIF to the GPT-4 API, together with a prompt (d). This input was processed by the model with the temperature parameter set to zero (e). Predictability was the direct, deterministic outcome following the prompt (f). (Fourth column) For the human measure, human participants received instructions similar to the prompt fed to GPT-4 in the prompt-based measure (d). Humans then watched the 6 s video clip (e) while their eye movements were tracked through their webcam. Participants indicated relevance on a Likert scale from 0 (Low Relevance) to 100 (High Relevance) (f).}
\label{fig:Figure1}
\end{figure}
\newpage
\subsection*{Hypothesis 1: Multimodality Drives More Human-Aligned Predictions}
Our first hypothesis was that multimodality enhances alignment with human language predictions. To test this, we compared the performance of several unimodal models against their multimodal counterparts. First, we leveraged the text-only as well as the multimodal versions of CLIP. \cite{radford_learning_2021} With an estimated 250 million parameters, CLIP is a relatively small, but highly capable multimodal, contrastive image-language matching model. Using the text-only as well as the multimodal versions allows us to directly compare the impact of adding multimodality to the embedding space. For CLIP, which is not designed for next-word prediction, we determined predictability by computing probability scores. We applied the softmax function to the model's similarity assessments between the provided prompts and relevant labels (Figure 1a and Figure 1d) from the movie. This method transforms the model's output into a probability distribution over all possible next words (i.e., next word predictions).
In a second direct probability measure, we used the state-of-the-art model LLaMA \cite{touvron2023llama} with 7 billion parameters. LLaMA itself is only trained on text, however, \citet{gao2023llamaadapter} have created a visual `adapter' layer for the 7B version, which enables visual as well as textual input to the original LLaMA model. As LLaMA is an open-source model, we could extract predictability measures directly by customising the model's forward method. We extracted predictability scores from LLaMA by feeding LLaMA's unimodal version textual input (Figure 1a) and computing a softmax over the next-word logits (Figure 1c) for all labels in the movie, obtained from the model's forward method (Figure 1b). For the multimodal version, we fed the textual input to LLaMA and the visual (frame-by-frame) input to the adapter layer (Figure 1d). Again, we extracted predictability scores by computing a softmax over the next word's logits (Figure 1f) for all labels in the movie, obtained from the model's forward method (Figure 1e).
This extraction of predictability estimates from the model is different to the human experiment: rather than directly probing for next-word prediction, we asked participants to assess the relevance of the visual-linguistic information for predicting an upcoming word. For this reason (and because GPT-4 is a closed model) we tested hypothesis (1) in a second way with a prompt-based measure that closely resembled the human instructions. For the unimodal version, we fed the model a prompt asking it to estimate the predictability of a given upcoming word, based on provided textual information (Figure 1a). After processing this prompt (Figure 1b) with the temperature set to 0 (to produce deterministic outcomes), GPT-4's output was a predictability estimate (Figure 1c.). For the multimodal version, we fed the same prompt to the model, providing text-based linguistic information in the prompt and the visual content in a Graphics Interchange Format (GIF) (Figure 1d). After processing this information with the temperature set to 0, the output was, again, a deterministic predictability estimate for the upcoming word (Figure 1f).
Our findings indicate that alignment with human predictability estimates significantly increased for the multimodal versions of CLIP, LLaMA and GPT-4 compared to their unimodal counterparts (Figure 2). The scatter plots in Figure 2 show the predictability estimates of each model against human scores (each data point represents one of the stimulus clips). Even for a strong model like GPT-4, predictability estimates are biased in the unimodal case (with clusters around 0). Overall, the distributions look more distributed and human-like for all multimodal models compared to their unimodal counterparts. Bar graphs in Figure 2 display the Pearson correlations between model and human scores, which quantify the extent to which each model's predictions align with human predictability estimates. To be able to interpret these model-based correlations against a human ceiling score, we correlated predictability estimates (from audio-visual stimuli) of each human participant against the $N-1$ (199) other participants and then averaged over these correlation values for all participants (green line in Figure 3).
While unimodal LLaMA scores were not correlated with human predictability estimates based on audio-clips (Figure 3, r = 0.03, p \textless 0.001), multimodal LLaMA scores were positively correlated with human predictability estimates from video-clips (including speech) (Figure 3, r = 0.1, p \textless 0.001). Similarly, scores based on unimodal GPT-4 were more weakly correlated with human predictability estimates from textual information (Figure 3, r = 0.16, p \textless 0.001) compared to multimodal GPT-4 scores correlated with human predictability estimates from video-clips (Figure 3, r = 0.28, p \textless 0.001). This latter correlation is approaching the human ceiling (Figure 3, r = 0.41, p \textless 0.001), see Figure 3 green dashed line. Finally, scores extracted from the multimodal CLIP version were positively correlated with human predictability scores (r = 0.23, p \textless 0.001), while there was almost no correlation between human predictability estimates and scores extracted from the text-only CLIP model (r = 0.05, p \textless 0.001).
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{Results_Hyp1.png}
\caption{Results for comparing unimodal, multimodal model and human predictability scores. (a) Average human response per audio-visual (multimodal) and audio-only (unimodal) stimuli (Y-axis) plotted against model response (X-Axis) for unimodal and multimodal GPT-4, LLaMA, and CLIP. For all multimodal models, the predicted model response (black regression line, only displayed for significant predictions) aligns significantly more with human predictability estimates compared to their unimodal counterpart. (b) Comparison between Pearson correlations of predictability scores derived from multimodal and unimodal models and human predictability estimates. The human ceiling is depicted as a dashed green line. Unimodal LLaMA scores were only marginally correlated with human predictability estimates based on audio-clips (light blue bar), multimodal LLaMA scores were positively correlated with human predictability estimates from video-clips (including speech) (dark blue bar). Similarly, scores based on unimodal GPT-4 were more weakly correlated with human predictability estimates from textual information (light pink bar) compared to multimodal GPT-4 scores correlated with human predictability estimates from video-clips (dark pink bar). Finally, scores extracted from the text-only CLIP model (light orange were marginally correlated with human predictability, while there was a high positive correlation between human predictability scores and the multimodal CLIP model (dark orange). Error bars represent 95\% confidence intervals. Stars indicate significance.}
\label{fig:Figure3}
\end{figure}
\subsection*{Hypothesis 2: Transformer Attention Drives Human-Like Predictions}
Our second hypothesis was that attention drives alignment between computational models and human natural language predictions. We tested this hypothesis with the multimodal predictability scores derived from the model CLIP. \cite{radford_learning_2021} We used CLIP, because it exists in two versions: with a visual transformer\cite{dosovitskiy_image_2021} backbone (ViT-32) and with a visual convolutional\cite{he_deep_2016} backbone (RN50). While the ViT-32 version of CLIP includes a visual attention mechanism, the RN50 version does not. Directly comparing how well either version aligns with human predictability estimates (as indicated in Figure 1, Multimodal Methods) allows us to establish a role for attention while keeping the training diet and the model's intrinsic dynamics constant.
Results from comparing the multimodal CLIP models with and without attention to human predictability estimates indicate no significant difference between the models overall (CLIP-ViT32: r = 0.23, p \textless 0.001; CLIP-RN50: r = 0.25, p \textless 0.001). The scatter plots in Figure 2 (a) display each model's predictions against human predictability estimates extracted from audio-video stimuli. Of particular interest, however, was the alignment between the model and humans at the top 25\% of model scores. In these video clips, the model presumably deemed the visual cues highly relevant for predicting upcoming semantic content. This is displayed in Figure 3 as the dashed box with grey background. The zoomed-in scatterplots show alignment for just the top quartile. Pearson correlations for this top quartile indicated an increase correlation for the visual transformer version (ViT-32) of CLIP (r = 0.46, p \textless 0.001) compared to a decrease for the convolutional net version (RN50) of CLIP (r = 0.19, p \textless 0.001), see the bar plot in Figure 4. The ViT-32 CLIP correlation was approaching the top quartile human ceiling (Figure 4, r = 0.61, p \textless 0.001) which was calculated by correlating each participant's top quartile scores against predictability from the $N-1$ (199) other participants and then averaging across these scores.
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{Results_Hyp2.png}
\caption{Results for comparing predictability scores in the top quartile of model predictability for CLIP with (ViT-32) and without (RN50) attention. (a) Average human response per video (Y-axis) plotted against model response (X-axis) for the CLIP model with and without attention. Top quartile of model scores is surrounded by a dashed grey box. A zoomed-in scatterplot displays human-model prediction alignment only for this top quartile (on the right). Regression lines are only displayed for significant predictions. Stars indicate strength of significance. (b) Bar plot comparing correlations of predictability scores for the respective top quartiles dervied from multimodal and unimodal versions of both LLaMA and GPT-4 with human predictability estimates. The human ceiling is displayed as a green dashed line. LLaMA scores were the lowest (dark blue), followed by GPT-4 (dark pink), and CLIP-RN50 (yellow). The highest score by far was obtained from the CLIP-ViT32 version with a visual attention mechanism (dark orange). Error bars indicate 95\% confidence intervals.}
\label{fig:Figure6}
\end{figure}
\subsection*{Hypothesis 3: Model Attention Patches and Human Eye Tracking Overlap}
Given that looks to available referents have been argued to support language processing,\cite{altmann_incremental_1999} we also wanted to understand when and how human visual attention indexed by eyegaze overlaps with the model's visual attention. To this end, we tested our final hypothesis by directly comparing attention weights from the model's visual attention layers to human eye tracking data collected during the online task. Although we collected output from all layers, the late-intermediate layers of transformer models are thought to synthesize the earlier layers' processing into a comprehensive output.\cite{tenney_bert_2019, rogers_primer_2021} For example, previous studies suggest that the best semantic features for predicting brain responses to natural language can be extracted from late-intermediate layers (especially layers 9 and 10 in models like GPT-2, which usually have around 12 layers).\cite{liu_linguistic_2019, oota_joint_2022, caucheteux_brains_2022, jain_computational_2023, tang_semantic_2022} Therefore we expected attention weights from these layers to correlate most significantly with human eye tracking.
To compare CLIP's visual attention matrices to human attention data from eye tracking, we analyzed the attention matrices for each of the frames from the video clips used in our experiment. These matrices represent pixel importance in predictability scores for the model and the fixation duration for humans. We averaged these matrices over 400ms segments, considering the natural latency in saccades (250–400ms),\cite{bergen_experimental_2007, fischer_express_1993} the relevance of this time frame for predictive language processing studies using EEG,\cite{dimigen_auditory_2022, kutas_thirty_2011} and the average word length in English being around 400ms.\cite{marslen-wilson_linguistic_1973} Unlike the widespread distribution of visual transformer attention, human attention tends to be more focused. We therefore adapted our analysis by thresholding the model's attention and applying Gaussian smoothing to both sets of data, choosing parameters to highlight differences between actual human attention and a random distribution. We then created probability distributions from these heatmaps and quantified alignment using Spearman correlation. A human ceiling correlation value for each of the 15 heatmaps per video clip was determined by correlating each participant's probability distribution with the $N-1$ (199) other probability distributions for this heatmap and taking the average of these correlations for each of the 15 segments per video clip. We find that averaging across all video clips, model attention approaches 40\% of the human ceiling - see Figure 4(a).
There are several potential reasons for this correlation not being higher across all videos. For one, model and human attention are computed under different circumstances. Because humans know the word ahead of time, they are more likely to attend to relevant visual information (e.g. focusing on the referent when it is present). CLIP on the other hand encodes both modalities (image and text) separately, so visual attention is computed taking language into account only to the extent that the visual encoder is biased towards language by the contrastive training task of matching images and captions. Therefore, the fact that we \textit{do} find overlap is already surprising and we do not expect a close to perfect overlap. Another reason could be that, while human-model correlations were sometimes negative, human-human correlations were not. We hypothesised that this was because in those video clips in which visual information was irrelevant for prediction, human and model attention were biased towards different areas. To test this hypothesis, we focused on those segments in which correlations between model and human probability distributions were negative. We found that in these cases, the model focused more on peripheral areas, while humans tended to centre their attention towards the middle of the screen. Complementary to this analysis, we also found that human-model correlation came closer to the human ceiling in layers 9 and 10 when considering only the video clips in the top quartile of model or human predictability judgements, see Figure 4 (a). Finally, when eye tracking was highly correlated \textit{within} human participants, the model's attention reached up to 90\% of the human ceiling (in layer 10). These results suggest that human and model attention patterns align when salient visual cues are present.
Eyetracking alignment in video clips based on human-model predictability correlations did not approach the human ceiling closer than 70\% in any layer (see figure 4a), even in video clips where the word was highly predictable. One reason could be the high variance in correlation values between segments within some video clips (this was 0.13 on average across segments). For example, in the video clip for the word `sunglasses' (see Figure 4b), the correlations from layer 9 were positive in segments 1-5 when the referent was present (peaking at r = 0.22 in segment 4), but dropped into negative correlation values (as low as -0.27) when no sunglasses were visually present anymore. The average correlation for this video clip was therefore around 0, even though some segments were highly correlated. Indeed, even though the average correlation between human- and model attention patterns was low, this video clip received a high average predictability judgement from human participants (P = 0.83). We therefore conducted a further, more fine-grained analysis focusing only on those video clips in the top quartile of model and human predictability judgements.
In a follow-up experiment, 100 participants were shown each of the 15 segments for each video clip (resulting in 375 screenshots overall) and given instructions to indicate whether or not the label was present in the segment. We ran a linear regression model on these responses, including the participant's binary response for each segment as a categorical variable to test if the presence of the referent could predict the correlation value between human eye tracking data and model attention patterns. Our results show that there was a significant (p \textless 0.001) linear relationship between the Spearman correlation and the absence/presence of the referent. Namely, when the referent was absent, correlation values were on average 0.23 points lower than when the referent was present (Figure 4c).
\newpage
\thispagestyle{empty}
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{Alignment_Attention.png}
\caption{Alignment between human gaze and model attention. (a) Layerwise percentage of alignment (model-human correlation divided by human-human correlation). Average alignment for all segments per layer is displayed as a blue line (with blue dots). Average alignment per layer for all segments in video-clips that are in the top quartile of human predictability estimates are displayed as orange line with orange crosses. Average alignment per layer for all segments in video-clips that are in the top quartile of model predictability estimates are displayed as green line with green squares. Average alignment per layer for all segments in video-clips that are in the top quartile of within human eye tracking alignment are displayed as a red line with red crosses. Overall, average overlap for all videos and overlaps based on \textit{predictability estimates} follow similar trends. Layer three reaches between 40 and 50\% of the human ceiling. Attention patches from layers nine and ten are most correlated with human eye tracking, reaching 70\% of the human ceiling in those videos where human predictability or model predictability were high. However, when human attention matrices are highly correlated, model attention patches are also highly correlated with human eye tracking, reaching 80 and almost 90\% of the human ceiling in layers 9 and 10 respectively. (b) Analysis of whether presence/absence of referent influenced correlation scores for layer 9. Using human ratings collected from 100 participants (red dots), absence or presence of the referent significantly predicted correlation value in a linear regression model (with random intercept for participant ID). In particular, the correlation between human eye tracking and model attention patterns was, on average, 0.2339 greater when the referent was present. (c) One example of evolving alignment in layer 9 between model attention patterns (red) and human eye tracking (green) over one video clip, separated into 15 segments. When salient visual information was present (segments 2, 3, and 4), correlation was positive. However, when the referent was not present, correlation dipped into negative values (e.g. segment 5, r = -0.16).}
\label{fig:Figure5}
\end{figure}
\newpage
\section*{Discussion}
We investigated the integration of visual-linguistic information during predictive language processing in state-of-the-art, multimodal computational models and compared these processes to behavioural data from humans. We found that including multimodality and visual attention as architectural features significantly increased correlation with human predictability scores. Furthermore, we found significant overlap between the model's attention patches and human eye tracking during the presence of salient visual cues (important for predicting upcoming words). We propose that the ability of multimodal models to conjoin visual and linguistic information can mitigate the inherent ambiguity of naturalistic language input and thereby facilitate more human-like semantic predictions.\cite{willems_nolb_2015} Below we discuss differences between models, the role of attention, and the potential of adding more modalities to the model's embedding space.
\subsection*{Differences Between Models}
As for comparisons between models, LLaMA scores remained relatively low, even for the multimodal version (r = 0.13), while the prompt-based GPT-4 measure approached 75\% of the human ceiling. Of course, one reason for this difference could be the fact that GPT-4 is estimated to have ~200 times the number of parameter size of LLaMA 7B. However, the CLIP model (with only 250 million parameters) was also more aligned with human scores in both its ViT-32 (r = 0.23) and RN50 versions (r = 0.25). Therefore, size of the model alone cannot account for these differences. Instead, one decisive advantage of the prompt-based measure could be that it works on a more dynamic, GIF-based input, and not a singular frame-by-frame basis. Humans may be sensitive to dynamic information evolving across frames that LLaMA does not have access to; similar divergence between static and dynamic stimuli has been found in face recognition networks. \cite{jiahui2021modeling} However, it is not clear how GPT-4 exactly processes GIF- or image-based input. Being likely trained on text and being based on the GPT architecture, it is unlikely that GPT-4 can meaningfully make use of temporal dynamics in the same way as, e.g. vision models that use temporal convolutions to learn dynamic features across frames.\cite{li_2d_2021} A reason for CLIP's high performance despite its modest parameter count, could be that in the case of LLaMA, multimodal knowledge is injected post-training through the adapter layer. It is a remarkable feat of LLaMA, which is trained on text only, that it can process visual information in this way. Yet, CLIP's training process includes a much larger dataset of combined image and text \textit{from scratch}, creating an entangled embedding space of both visual and text-based information, which may be better able to capture abstract correlation structures between image and text.\cite{baltrusaitis_multimodal_2017}
\subsection*{The Role of Attention}
While multimodality increased alignment across the board, visual attention was only relevant in cases where salient visual cues were present. For this top quartile of model predictability estimates, the correlation between CLIP and human predictability estimates doubled (from r = 0.23 to r = 0.46), whereas the CNN version of CLIP did worse than before (r = 0.19 rather than r = 0.28). One reason for this could be that compared to CNNs, where inductive biases like locality, two-dimensional neighborhood structures, and translational equivariance are inherent in the model's architecture,\cite{noauthor_deep_nodate} ViT has barely any image-specific predispositions. In ViT, processing is global, with self-attention layers engaging the entire image rather than focusing on local or neighborhood-specific features. Being free from the constraints of fixed structural biases and grounded in a more flexible, globally oriented perspective, such an architecture may offer a more congruent alignment with human integration processes of multimodal information in naturalistic, dynamic contexts such as videos.
We found a similar jump in correlation for the human ceiling correlation, which increased from r = 0.41 to r = 0.61 for the top quartile. This jump can be explained by research testing eye tracking during language processing in naturalistic conditions, which suggests that eye tracking indexes predictive processing most reliably when the referents are situated in a joint attention space.\cite{heyselaar_we_2020} When the referent is not present, humans make more diverse judgments about what they consider relevant visual-linguistic integration for predicting the upcoming word.\cite{ryskin_prediction_2023} Therefore, there is also more variance in their predictability scores, leading to weaker correlations within human judgments. Similarly, CLIP attention patterns are more evenly spread out when no salient visual cues are present, leading to more variance in predictability estimates and weaker human-model alignment.
Indeed, we found significant and quantifiable overlap between the model's attention patches and human eye tracking. This finding is surprising, as despite its nomenclature, the ‘attention’ mechanism in transformers is not a replica of human attention. Rather, it is a purely mathematical method for assigning weights to different parts of the input based on perceived relevance.\cite{vaswani_attention_2017} Furthermore, CLIP encodes images and text separately. Therefore, the model's visual attention patterns are only influenced by language to the extent that the visual encoder (because of the contrastive image-language matching task the model is trained on) is biased towards aspects of the image that are relevant for matching language embeddings. In contrast, human attention guides human perception online\cite{wang_does_2020,boynton_attention_2005} and human attention is influenced by a wider range of contextual factors, including past experiences,\cite{chen_what_2019} current goals,\cite{jiang_habitual_2018} or emotional states,\cite{derakshan_introduction_2010} and operates over different timescales\cite{nobre_anticipated_2018} and modalities.\cite{busse_spread_2005} Future models could be more aligned with human processing if the visual attention mechanism received online input from linguistic information. This could help the model to exploit predictive associations between visual and linguistic information more effectively. Indeed, the beneficial effect for learning predictive relations between events online through attention has been reported in humans (see for example \citet{custers_learning_2011}). Finally, transformer self-attention is not particularly plausible from a biological point of view compared to a classical recurrent architecture, as the brain also relies on recurrence for language processing (rather than encoding many words in parallel).\cite{lindsay_attention_2020} Future work should compare human eye tracking during naturalistic language processing to both multimodal transformers and recurrent architectures to test whether biological plausibility translates into behavioural alignment.
Another remaining open question relates to the conditions under which attention patterns between multimodal computational models with a transformer vision back-bone and humans align. Our results show that overlap between attention patterns becomes significant when the visual input includes the referent of the word to be predicted (see Figure 4 (c)). In cases where visual information is not relevant, humans tend to centralise their eye-gaze. This is a documented effect in the visual world paradigm \cite{magnuson_fixations_2019} and has several reasons, including reduction of cognitive load through efficient allocation of attentional resources.\cite{foulsham_where_2011} The model, on the other hand, tends to focus on peripheral areas when no relevant visual information is present.
\subsection*{From Language Toward Embodiment}
A criticism of large language models (LLMs) like GPT-4 is that they can never adequately replicate human language processing because they are not embodied.\cite{chemero_llms_2023} On this view, human cognition and language are deeply intertwined with sensory experiences and interactions with the environment.\cite{anderson_embodied_2003} This grounding in the physical world is believed to be crucial for genuine acquisition and understanding of language.\cite{meteyard_coming_2012} However, by integrating various sensory modalities, such as visual or auditory data, into the models' training, LLMs can gain some aspects of the context and environment-dependent understanding characteristic of human cognition. As we have shown, adding just one modality already allows LLMs to process and generate language that is more aligned with how humans perceive and interact with the world, leading to more nuanced and contextually appropriate responses.
However, it is also important to note that even with more multimodal inputs, large language models would still lack direct, experiential interaction with the environment, which some theorists argue is fundamental to true human-like understanding and language.\cite{barsalou_simulation_2009} Therefore, it has been argued that truly human-like AI can only be achieved in a bottom-up process, starting with basic spatial interactions in the world.\cite{zador_catalyzing_2023} Building on multimodal advancements in LLMs, however, the PaLM-E model \cite{noauthor_palm-e_nodate} may exemplify the integration of the kind of data that allows for a simulated experiential interaction with the environment. By incorporating sensor data from robotic agents into an existing LLM, PaLM-E extends beyond traditional text and image processing, engaging with real-world robot states and environmental interactions. This innovative approach offers a glimpse into future AI capabilities where computational models may process a richer array of sensory inputs, akin to human perception. PaLM-E's state-of-the-art ability to adapt across various robotic platforms and handle different modalities like images and robot states exemplifies the potential of AI systems to operate in more complex, real-world scenarios, moving closer to the nuanced way humans interact with their environment. Future research continuing on this path may offer an alternative route to bottom-up engineering of human-like AI.
An additional aspect of adding multimodality to the embedding space is learning efficiency. In human cognition, multimodal inputs can facilitate faster and more efficient learning or acquisition of language and context understanding.\cite{willems_nolb_2015} This is particularly notable when contrasted with LLMs, which require thousands of human years' worth of data reading for effective learning. Multimodal models may therefore not only align better with human processing but also potentially require less data for training to achieve a similar level of processing.
\subsection*{Conclusion}
The fact that current AI models are trained with a very simple objective (next-word prediction), may suggest that their high performance is based on the learning of abstract co-occurrence relationships between word tokens or clusters while lacking genuine understanding.\cite{srivastava_beyond_2022,lenci_understanding_2023} Here we provide behavioural evidence that multimodal attention models may be able to leverage contextual information to predict upcoming words in a way that aligns more with humans. As mAI is becoming ubiquitous in our daily lives, it is vital to understand how these models process input and what architectural features support alignment with human processing. These insights can inform responsible development of potentially grounded AI systems and make them more useful, understandable, and reliable - important aspects for real-world applicability (see for example \citet{tankelevitch_231210893_nodate}). Based on our findings, we suggest continuing the push towards including more modalities and sensorimotor, experience-based data into the training sets of large language models to achieve richer, and more human-like conceptual understanding in AI systems. A wider variety of mAI could also enable the modeling of human behavior in ecologically valid experimental settings. Understanding how AI systems align with human cognitive processes in naturalistic, dynamic, and multimodal environments not only advances AI technology but furthers our understanding of the mechanisms underlying human cognition.\cite{noauthor_psyarxiv_nodate}
\section*{Methods}
\subsection*{Online behavioural study}
\subsubsection*{Participants}
For the main study, 200 participants were recruited via Prolific \cite{palan_prolificacsubject_2018}(https://www.prolific.co) and for the follow up study another 100 participants were recruited under the exact same conditions. To be included in the study, participants had to be aged 18-50 (inclusive) and fluent in English (having spoken English regularly for at least the past 5-10 years). Inclusion criteria included giving consent to have their camera on as well as having their eyes tracked for the duration of the experiment. We also excluded any participants from participating who had previously seen either of our two movies. All participants finished the study and produced usable data, none had to be excluded post data collection. The final participants for the main study (n=200) were 43\% females (n = 86) with mean age of 33.88 years (range = 23-48 yrs, SD = 13.74 yrs). Countries of origin were the UK (n = 123) and the US (n = 77). The final participants for the follow up study (n=100) were 46\% female (n = 46) with mean age of 29.8 (range = 21 - 42 yrs, SD = 11.37). Countries of origin were the UK (n = 67) and the US (n = 33). Participants in both studies were paid £9.50/hour. The study was approved by the UCL Ethics Committee. Additionally, all participants provided written informed consent before participating in the experiment.
\subsubsection*{Materials}
100, 6s movie clips from the films ‘The Usual Suspects’ and ‘The Prestige’ were chosen respectively. All words in these movies were previously annotated using automated approaches with a machine learning based speech-to-text transcription tool from Amazon Web Services (AWS; https://aws.amazon.com/transcribe/) and later corrected by human annotators. This alignment between text and speech meant that the model and humans really received the same dialogue-based information.
To choose the words to be predicted by human participants in our study and later by the model, all function words (as defined in this list: https://semanticsimilarity.wordpress.com/function-word-lists/) from the movies were excluded and the 6s scenes leading up to but excluding each word were extracted – this amounted to a few thousand words per movie on average. A length of 6s was chosen, as this constituted a good balance between containing enough information while not being too confusing. Furthermore, this meant that it was possible to conduct a large number of trials without causing too much fatigue in participants. For our final study, only 50 scenes were included from each movie, leading to 100 clips, totalling 600s (10mins) of video material for each participant. This meant that a further subset of words had to be chosen. As the goal was to probe visual-linguistic integration, a final set of 100 video clips was pre-selected in such a way as to ensure the generation of a stimulus set containing both videoclips in which the visual information was highly relevant for processing and videoclips in which it was not. To this end, each frame was captioned in each video scene with the CLIP model and the text-based transformer BERT\cite{devlin_bert_2019} was used to calculate the semantic similarity between these captions and the word of interest. The maximum similarity score derived from this comparison was used for each scene as a measure of how relevant the preceding scene was for the upcoming word. Then, from each movie, 25 scenes were chosen from the low end of the distribution of these scores and 25 scenes from the high end of the distribution of these scores - resulting in 100 movie scenes in total, 50 from each movie. These were validated against the later collected human scores, showing that around 75\% of categorizations into ‘high’ or ‘low’ relevance matched with human judgements. The final set of stimuli and the code for determining them, can be found at https://github.com/ViktorKewenig upon publication.
After the main experiment was conducted, 15 frames were extracted from each of the 25 video clips that received the highest predictability ratings by human participants for our follow up study.
\subsubsection*{Procedure}
Both the main study and the follow up study were made with the Gorilla task builder,\cite{anwyl-irvine_gorilla_2020} and the structure of both studies was as follows: after recruitment through Prolific, participants were directed to the study website. Participants were briefed on the aims of the study and asked for consent. Following the collection of demographic information, participants were instructed to complete the experiment in an environment with minimal distractions and with their phones turned off, wearing headphones. For the main study, before starting the experimental task, participants were run through the native eye tracking calibration for web cameras on Gorilla. The eye tracking software was calibrated three times in total – before the practice trials, after the practice trials, and during the halfway break – to ensure more reliable measurement even with a shift of head position during the task. Both experimental tasks consisted of 3 practice trials and 100 experimental trials.
During the main study, in each trial, participants were first shown the word of interest corresponding to the upcoming scene and asked to pay careful attention to the upcoming video clip and think about how relevant that clip would be for predicting the word. After pressing a key to continue, participants were then shown the 6s movie scene. The video was shown in a window at the centre of the screen at a resolution of 720x1280 pixels (25 frames per second). Participants then used a 100-point slider (‘Low Relevance to ‘High Relevance) to indicate how relevant the video clip was to the previously presented target word. This was repeated for all 100 trial scenes and took participants an average of around 30 minutes to complete. A self-paced resting break was provided halfway through the trials. After completing the experimental trials, participants were debriefed and paid for their time.
The goal of the follow up study was to understand whether human participants judged a given referent as present or not present in each of the 15 segments per video clip. For this purpose, participants were first shown the word of interest corresponding to the upcoming segment and asked to pay careful attention to the upcoming image and think about whether they could see the word in the image or not. The image was shown in a window at the centre of the screen at a resolution of 720x1280 pixels. Participants then had to select a check-box, indicating that the referent was present or not, before proceeding with the experiment. This was repeated for the 375 segments of the video clips which received the top quartile of human predictability judgements. A self-paced resting break was provided halfway through the trials. After completing the experimental trials, participants were debriefed and paid for their time.
\subsubsection*{Human Measurements}
\subsubsection*{Estimates of Predictability}
Human estimates of the target word's predictability were collected using an interactive slider. This measure of likelihood is intended to reflect the participant's intuitive understanding of the relationship between antecedent linguistic context, the visual cues in a scene and the subsequent semantic content. As we expected these intuitive judgements to vary between participants, outliers (any rating above or below 2 standard deviations) were not excluded from the analysis. A human ceiling score was calculated by correlating predictability estimates from each human participant against the $N-1$ (199) other participants and then averaging over these correlation values for all participants. This ceiling score allowed us to compare model scores against a baseline and they also functioned as an estimate of the reliability of individual humans.
\subsubsection*{Eye Gaze}
The gathering of human eye tracking data was done via the Gorilla eye tracking tool (version 2.0). This tool exploited the webcams of each participant to capture eye movements. The collected data, sampled at a rate of 60 Hz, was individually stored in an Excel file for each video-clip and each participant. Eye-tracking data were analyzed using a custom Python script that made ample use of the packages ‘NumPy’\cite{harris_array_2020}, ‘SciPy’\cite{virtanen_scipy_2020}, and ‘Pandas’\cite{noauthor_pandas_nodate}. The script first read and compiled the X and Y gaze coordinates from each participant's eye tracking data files, discarding any data from calibration tasks. Then, two-dimensional attention heatmaps were generated based on gaze coordinates for each participant and their corresponding stimulus. 3\% of all heatmaps contained only zeros and were excluded from further analysis. These raw heatmaps were averaged for each segment (every 400ms) for each stimulus and participant and then saved as NumPy files for subsequent analysis. The averaged heatmaps serve as a representation of the aggregate attention directed by participants towards specific visual features in each segment.
\subsubsection*{Estimates of Reference Presence/Absence}
Human estimates of whether a referent was present or not (in each segment) were collected using two check-boxes (one for `no', one for `yes'). This measure was intended to reflect the participant's more fine-grained understanding of the relevance of visual cues in each video clip. We did not expect these judgements to vary greatly between participants, and indeed the agreement rate between participants was 80\% or above.
\subsubsection*{Model Measurements}
\subsubsection*{LLaMA Predictability}
LLaMA's unimodal version encodes and processes the textual input. A softmax over the next-word logits was computed for all labels in the movie. Logits are essentially the raw predictions that the model generates for the upcoming word, before they are converted into a more interpretable form, such as probabilities, through the softmax function. These logits were obtained by slightly altering the model's `forward' method and indexing into the final word logits. For the multimodal version, the textual input was processed by the frozen LLaMA model, whereas the visual (frame-by-frame) information was sent to the adapter layer. LLaMA adapter uses a CLIP-based encoding for the visual information and then projects this encoding into an embedding space that can be processed by LLaMA. Again, predictability scores were extracted by computing a softmax over the next-word's logits for all labels in the movie, obtained from the model's `forward' method.
\subsubsection*{GPT-4 Predictability}
To obtain unimodal GPT-4 predictability scores, OpenAI's closed API was used. The human instructions were used as prompt-based input to the API, together with the textual input. Temperature was set to 0, so as to obtain a deterministic outcome. In order to obtain multimodal predictability scores, the API was also used, except this time with the “GPT-4-V” (the visual) version of GPT-4. This allowed the uploading of the video scene as a as a `Graphics interchange format' (GIF) (which is a more dynamic representation of the video information compared to singular, frame-by-frame screenshots), together with the human instructions and the textual input (which human participants would listen to as the dialogue). Again, the temperature was set to 0 in order to obtain a deterministic outcome.
\subsubsection*{CLIP Predictability}
The calculation of CLIP predictability scores for target words was carried out through a combination of image and text feature comparison with a custom Python script making use of the packages ‘PyTorch’\cite{paszke_pytorch_2019} and the Hugging Face ‘transformer’ library.\(^{48}\) For the unimodal version, the textual input was tokenized and processed by the text-only version of CLIP. For the multimodal version, each frame of each movie clip was individually preprocessed by the standard, pre-trained CLIP preprocessor, transforming the visual data into a digestible form for the CLIP model's image encoder. This encoder generates a corresponding image feature vector, which provides a compact representation of the visual content in a form that can be easily compared with textual information. We pair each image feature vector with an array of text feature vectors. These text features vectors are derived from both the dialogue of each scene, represented as text prompts, and the collective word set present in the movie. The CLIP model tokenizes and encodes these text inputs, converting them into a format that mirrors the structure of the image feature vectors. For the purpose of our study we used both the the ‘clip-vit-base-patch32’ version and the `clip-RN50' versions of the model.\cite{noauthor_openaiclip-vit-base-patch32_nodate}
The calculation of the `predictability score' was carried out by comparing these two feature sets, visual and linguistic (or linguistic and linguistic in the unimodal comparison). For the unimodal version, the dot product of the feature input of the textual input, and all labels in the movie was computed. Similarly, by computing the dot product of the image feature vector and each text feature vector in the multimodal version, a raw similarity measure was derived. This raw similarity measure was then passed through a softmax function, refining it into a more interpretable similarity score. This score quantified the likelihood of the upcoming word, given all words in the movie, by matching each word to the visual content of each associated frame (or textual input for the multimodal version).
\subsubsection*{Differences in Predictability Estimates}
There are some commonalities and differences between our different model-based predictability measures and how we extracted predictability scores from humans, however. A noteworthy difference between LLaMA and human predictability estimates is that rather than directly probing for next-word prediction, participants were asked to assess the relevance of the visual-linguistic information for predicting an upcoming word. This process is slightly different to LLaMA's computations, which determined the likelihood of the next word directly based on textual or multimodal input.
The reason we chose this different task for the human experiment is that predicting the next word from a six-second movie scene is inherently difficult and might not have yielded usable results with human participants. To alleviate concerns connected to these differences, we used the human instructions for our prompt-based measure with GPT-4 (though prompt-based measures may not be an adequate substitute for direct probability measurements -see \citet{hu2023prompting}).
The CLIP model on the other hand generated a quantitative estimate of the likelihood of each word by comparing the feature vectors of the visual content in a frame with feature vectors of the associated linguistic context. The predictability measures extracted from CLIP and human participants therefore rely on a similar foundational principle: considering the similarities within the presented visual-linguistic information and understanding how these similarities influence the likelihood of the upcoming target word. This integration is an inherent part of human language comprehension, combining visual stimuli with linguistic expectations based on past experiences and world knowledge.
\subsubsection*{CLIP Attention Maps}
To explore the qualitative role of attention in the alignment between CLIP and humans, we compared the matrices derived from CLIP's visual attention mechanism to the human attention matrices obtained from eye tracking data, examining each frame of every video clip. The CLIP model uses 12 transformer layers, each with 16 attention heads, yielding a total of 192 individual attention maps for each input. For our analysis, the attention weights of all layers were utilised. All attention maps were reshaped to match the resolution of the human eye tracking heatmaps (the 1280 x 720 pixelrate of both movies). The CLIP model operates on the principle of dividing the input image into a grid of patches. Each patch represents a segmented portion of the image, with the model's attention heads focusing on these discrete units rather than individual pixels. This patch-based approach allows the model to efficiently process and interpret the visual information by focusing on salient image segments. CLIP does not analyse the full-sized image at the original pixel resolution of 1280x720. Instead, the model downsamples the image to 224 x 224 pixels, which are then divided into the aforementioned patches. A two-dimensional ‘attention heatmap’ was then generated for each attention head by reshaping the patch-scale attention weights back into a square matrix of the original image size (1280x720). Each cell in this matrix represented a region of the input, and its value indicates the amount of attention the model paid to that region. The attention heatmaps were then averaged across all attention heads to create a single attention heatmap for each movie clip, similar to the averaged heatmaps generated from the human data.
\textbf{Attention Matrices, Thresholding and Averaging.}
In these 2-D matrices, each entry denotes the significance of a specific pixel in the visual input when generating a predictability score on the model side, and the duration a participant spent observing that specific pixel on the human side. To facilitate our analysis, we averaged attention matrices from CLIP and human participants across every ten frames, yielding 15 averaged attention matrices per video for each participant and the CLIP model. Another reason for choosing to average the 6s video-clip into 15 segments (400ms each) is a fundamental difference between human and model processing. Humans do not instantly react to a stimulus; rather previous research on the visual world paradigm has estimated the saccade response to take between 100–250ms depending on the richness of the stimulus. \cite{dahan_chapter_2007, huettig_word_2005} As human gaze will therefore be lagged around that time for reaction to a new object or scene (while the model's attention patterns are near instantaneous), an average over 400ms was considered enough to smooth out that difference. Furthermore, 400ms is a relevant time-window for studies on predictive language processing, as the N400 EEG signal related to surprisal (or prediction error) can be measured 400ms after word-onset. \cite{kutas_thirty_2011}
Another difference between attention in humans and visual transformers, is that visual transformer attention is widely distributed across the visual input, while human attention is relatively sparse (eye tracking data yields a single predicted pixel every 10ms). For this reason, we thresholded the model attention at a conservative 15th percentile of top values, and applied Gaussian smoothing to both model and human heatmaps. The sigma value in the Gaussian smoothing process determines the extent of smoothing and was chosen as that which maximises the difference between the human attention distribution and a null distribution. The null distribution was generated by randomly redistributing the attention values across human heatmaps. We chose the sigma value in this way so as not to bias later comparisons between human and model attention patterns.
\textbf{Calculating Eyetracking-Attention Alignment.}
A probability distribution over the averaged and smoothed heatmaps was calculated by dividing each value in each heatmap by the total sum of values. Alignment between human eye tracking and model attention was quantified as the Spearman correlation between these probability distributions (15 for each of the 100 video clips). Since the Spearman correlation is based on ranks rather than actual values, it is less sensitive to outliers compared to the Pearson correlation. This is particularly useful for attention heatmaps, where there may be regions with exceptionally high or low attention that could skew the results of parametric correlations. A human ceiling correlation value for each of the 15 heatmaps per video clip was determined by correlating each participant's probability distribution with the $N-1$ (199) other probability distributions for this heatmap and taking the average of these correlations for each of the 15 segments per video clip.
\textbf{Focus of Attention when Visual Cues Were Irrelevant.}
To understand whether humans and the model focussed on different areas in the input when visual information was not relevant, we only considered those video clips in which the model-human correlations were negative. We defined a rectangle (640 x 360, half of all pixels) around the central pixel of the probabiltiy distributions of these segments as the central region and categorised the rest of the pixels as periphery. Next, we took the difference between the central and periphery areas for all probability distributions. A positive difference indicated that attention was centrally biased, whereas a negative difference indicated that attention was biased towards the periphery of the probability distribution. A t-test between all difference values on the model side and all difference values on the human side indeed suggested that the model tended to focus on periphery areas of the visual input, whereas humans tended to focus on central areas (t = -12.7, p \textless 0.001).
\subsection*{Analyses}
\subsubsection*{Comparison Between Unimodal Predictability, Multimodal Predictability, CLIP Predictability, and Human Predictability}
Human and model predictability scores were compared with simple Pearson correlation using the SciPy\cite{virtanen_scipy_2020} package. The human ceiling correlation was determined by correlating predictability estimates of each human participant with the $N-1$ (199) other participants against each other and averaging Pearson correlations.
\subsubsection*{Comparison Between CLIP Attention Weights and Human Eye Tracking}
Human- and model attention matrices were 1280x720 pixels, where each vaue at each pixel represents how much attention was paid to this pixel during each 400ms segment. Overlap between CLIP and human eye tracking data was calculated by comparing the probability distributions over both model- and human heatmaps for each segment (1500 segments in total). Probability distributions were calculated by dividing each value in the heatmaps through the sum of all values. Each participants' probability distribution for each segment was correlated with the model's corresponding probability distribution using spearman rank correlation. A human ceiling was constructed by correlating each participant’s probability against that of the $N-1$ other participants and then averaging these correlations across participants (resulting in one human-ceiling correlation value per segment). Overall overlap was calculated by averaging all correlation values for human-model comparisons and for the human ceiling across segments. Overlap for the top quartile was calculated by only averaging correlation values for those segments in the top 25\% of video clips that received the highest predictability scores from the model and human participants.
\subsubsection*{Linear Regression to Further Understand the Nature of Overlap}
To gain a more fine-grained understanding of when and how human- and model attention patterns overlap, the human measurements of referent absence/presence were used in a linear regression model to predict the correlation values between human- and model attention heatmaps. The model to predict the correlation value between attention patterns included the human ratings of absence (0) and presence (1) as independent categorical variables as well as a random intercept for participant.
\subsection*{Data and Code Availability}
All the data used in this study will be publicly available upon publication under: \href{http://osf.io/6whzq/?view_only=162085f95bab42b5a57b34b386143ba8}{CLIPPredict}. All code for data processing and analysis will be publicly available upon publication under the following GitHub repository: \href{http://github.com/ViktorKewenig}{github.com/ViktorKewenig}.
\subsection*{Contributions}
VK: conceived the study, secured funding for the online study, created stimuli and led the online experiment, wrote code for analyes, conducted data analyses, wrote and edited the paper.
AL: helped conceptualise analyses and edited the paper.
SAN: helped conceptualise analyses and edited the paper.
CE: helped lead the online study and edited the paper.
QLE: helped lead the online study and edited the paper.
RA: helped lead the online study and edited the paper.
GV: provided stipends for VK and CE, helped conceptualise the study, and edited the paper.
JIS: helped conceptualise the study and edited the paper.
\subsection*{Acknowledgments}
This work was supported in part by the European Research Council Advanced Grant (ECOLANG, 743035); Royal Society Wolfson Research Merit Award (WRM R3 170016) to GV; and Leverhulme award DS-2017-026 to VK and GV. Figures were created with BioRender.com
\bibliography{main}
\end{document}
\begin{thebibliography}{106}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Mitchell and Krakauer()]{mitchell_debate_2023}
Melanie Mitchell and David~C. Krakauer.
\newblock The debate over understanding in {AI}'s large language models.
\newblock 120\penalty0 (13):\penalty0 e2215907120.
\newblock \doi{10.1073/pnas.2215907120}.
\newblock URL \url{https://www.pnas.org/doi/10.1073/pnas.2215907120}.
\newblock Publisher: Proceedings of the National Academy of Sciences.
\bibitem[Mahowald et~al.(2023)Mahowald, Ivanova, Blank, Kanwisher, Tenenbaum, and Fedorenko]{mahowald_dissociating_2023}
Kyle Mahowald, Anna~A. Ivanova, Idan~A. Blank, Nancy Kanwisher, Joshua~B. Tenenbaum, and Evelina Fedorenko.
\newblock Dissociating language and thought in large language models: a cognitive perspective, January 2023.
\newblock URL \url{http://arxiv.org/abs/2301.06627}.
\newblock arXiv:2301.06627 [cs].
\bibitem[Liu et~al.(2023)Liu, Ma, Zhou, Zhu, and Zheng]{liu_brainclip_2023}
Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng.
\newblock {BrainCLIP}: {Bridging} {Brain} and {Visual}-{Linguistic} {Representation} via {CLIP} for {Generic} {Natural} {Visual} {Stimulus} {Decoding} from {fMRI}, February 2023.
\newblock URL \url{http://arxiv.org/abs/2302.12971}.
\newblock arXiv:2302.12971 [cs].
\bibitem[Kauf et~al.(2022)Kauf, Ivanova, Rambelli, Chersoni, She, Chowdhury, Fedorenko, and Lenci]{kauf_event_2022}
Carina Kauf, Anna~A. Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan~S. She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci.
\newblock Event knowledge in large language models: the gap between the impossible and the unlikely, December 2022.
\newblock URL \url{http://arxiv.org/abs/2212.01488}.
\newblock arXiv:2212.01488 [cs].
\bibitem[Zhang et~al.(2022)Zhang, Zheng, Lu, Zhao, Zhu, Fu, Xiang, and Feng]{zhang_visual_2022}
Li~Zhang, Sixiao Zheng, Jiachen Lu, Xinxuan Zhao, Xiatian Zhu, Yanwei Fu, Tao Xiang, and Jianfeng Feng.
\newblock Visual {Representation} {Learning} with {Transformer}: {A} {Sequence}-to-{Sequence} {Perspective}, July 2022.
\newblock URL \url{http://arxiv.org/abs/2207.09339}.
\newblock arXiv:2207.09339 [cs].
\bibitem[Michelmann et~al.(2023)Michelmann, Kumar, Norman, and Toneva]{michelmann_large_2023}
Sebastian Michelmann, Manoj Kumar, Kenneth~A. Norman, and Mariya Toneva.
\newblock Large language models can segment narrative events similarly to humans, January 2023.
\newblock URL \url{http://arxiv.org/abs/2301.10297}.
\newblock arXiv:2301.10297 [cs, q-bio].
\bibitem[Schrimpf et~al.()Schrimpf, Blank, Tuckute, Kauf, Hosseini, Kanwisher, Tenenbaum, and Fedorenko]{schrimpf_neural_2021}
Martin Schrimpf, Idan~Asher Blank, Greta Tuckute, Carina Kauf, Eghbal~A. Hosseini, Nancy Kanwisher, Joshua~B. Tenenbaum, and Evelina Fedorenko.
\newblock The neural architecture of language: Integrative modeling converges on predictive processing.
\newblock 118\penalty0 (45):\penalty0 e2105646118.
\newblock \doi{10.1073/pnas.2105646118}.
\newblock URL \url{https://www.pnas.org/doi/10.1073/pnas.2105646118}.
\newblock Publisher: Proceedings of the National Academy of Sciences.
\bibitem[Goldstein et~al.(2022)Goldstein, Zada, Buchnik, Schain, Price, Aubrey, Nastase, Feder, Emanuel, Cohen, Jansen, Gazula, Choe, Rao, Kim, Casto, Fanda, Doyle, Friedman, Dugan, Melloni, Reichart, Devore, Flinker, Hasenfratz, Levy, Hassidim, Brenner, Matias, Norman, Devinsky, and Hasson]{goldstein_shared_2022}
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel~A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth~A. Norman, Orrin Devinsky, and Uri Hasson.
\newblock Shared computational principles for language processing in humans and deep language models.
\newblock \emph{Nature Neuroscience}, 25\penalty0 (3):\penalty0 369--380, March 2022.
\newblock ISSN 1546-1726.
\newblock \doi{10.1038/s41593-022-01026-4}.
\newblock URL \url{https://www.nature.com/articles/s41593-022-01026-4}.
\newblock Number: 3 Publisher: Nature Publishing Group.
\bibitem[Caucheteux et~al.()Caucheteux, Gramfort, and King]{caucheteux_long-range_2021}
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King.
\newblock Long-range and hierarchical language predictions in brains and algorithms.
\newblock URL \url{http://arxiv.org/abs/2111.14232}.
\bibitem[Dong and Toneva(2023)]{dong_vision-language_2023}
Dota~Tianai Dong and Mariya Toneva.
\newblock Vision-{Language} {Integration} in {Multimodal} {Video} {Transformers} ({Partially}) {Aligns} with the {Brain}, November 2023.
\newblock URL \url{http://arxiv.org/abs/2311.07766}.
\newblock arXiv:2311.07766 [cs].
\bibitem[Vigliocco et~al.(2014{\natexlab{a}})Vigliocco, Perniss, and Vinson]{vigliocco2014language}
Gabriella Vigliocco, Pamela Perniss, and David Vinson.
\newblock Language as a multimodal phenomenon: implications for language learning, processing and evolution.
\newblock \emph{Philosophical Transactions of the Royal Society B}, 2014{\natexlab{a}}.
\newblock \doi{https://doi.org/10.1098/rstb.2013.0292}.
\bibitem[Zwaan et~al.()Zwaan, Langston, and Graesser]{zwaan_construction_1995}
Rolf~A. Zwaan, Mark~C. Langston, and Arthur~C. Graesser.
\newblock The construction of situation models in narrative comprehension: An event-indexing model.
\newblock 6\penalty0 (5):\penalty0 292--297.
\newblock ISSN 0956-7976.
\newblock \doi{10.1111/j.1467-9280.1995.tb00513.x}.
\newblock URL \url{https://doi.org/10.1111/j.1467-9280.1995.tb00513.x}.
\newblock Publisher: {SAGE} Publications Inc.
\bibitem[Zwaan(2014)]{zwaan_embodiment_2014}
Rolf~A. Zwaan.
\newblock Embodiment and language comprehension: reframing the discussion.
\newblock \emph{Trends in Cognitive Sciences}, 18\penalty0 (5):\penalty0 229--234, May 2014.
\newblock ISSN 1364-6613.
\newblock \doi{10.1016/j.tics.2014.02.008}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S1364661314000527}.
\bibitem[Zwaan()]{zwaan_situation_2016}
Rolf~A. Zwaan.
\newblock Situation models, mental simulations, and abstract concepts in discourse comprehension.
\newblock 23\penalty0 (4):\penalty0 1028--1034.
\newblock ISSN 1531-5320.
\newblock \doi{10.3758/s13423-015-0864-x}.
\newblock URL \url{https://doi.org/10.3758/s13423-015-0864-x}.
\bibitem[Pickering and Gambi(2018)]{pickering_predicting_2018}
Martin~J. Pickering and Chiara Gambi.
\newblock Predicting while comprehending language: {A} theory and review.
\newblock \emph{Psychological Bulletin}, 144\penalty0 (10):\penalty0 1002--1044, 2018.
\newblock ISSN 1939-1455.
\newblock \doi{10.1037/bul0000158}.
\newblock Place: US Publisher: American Psychological Association.
\bibitem[Skipper(2015)]{willems_nolb_2015}
Jeremy~I. Skipper.
\newblock The {NOLB} model: a model of the natural organization of language and the brain.
\newblock pages 101--134, Cambridge, 2015. Cambridge University Press.
\newblock ISBN 978-1-107-32366-7.
\newblock \doi{10.1017/CBO9781107323667.006}.
\newblock URL \url{https://www.cambridge.org/core/product/identifier/9781107323667%23c04201-929/type/book_part}.
\newblock Book Title: Cognitive Neuroscience of Natural Language Use.
\bibitem[Huettig et~al.(2011)Huettig, Rommers, and Meyer]{huettig_using_2011}
Falk Huettig, Joost Rommers, and Antje~S. Meyer.
\newblock Using the visual world paradigm to study language processing: a review and critical evaluation.
\newblock \emph{Acta Psychologica}, 137\penalty0 (2):\penalty0 151--171, June 2011.
\newblock ISSN 1873-6297.
\newblock \doi{10.1016/j.actpsy.2010.11.003}.
\bibitem[Kuperberg and Jaeger(2016)]{kuperberg_what_2016}
Gina~R. Kuperberg and T.~Florian Jaeger.
\newblock What do we mean by prediction in language comprehension?
\newblock \emph{Language, Cognition and Neuroscience}, 31\penalty0 (1):\penalty0 32--59, 2016.
\newblock ISSN 2327-3798.
\newblock \doi{10.1080/23273798.2015.1102299}.
\bibitem[Paivio(2013)]{paivio_imagery_2013}
A.~Paivio.
\newblock \emph{Imagery and {Verbal} {Processes}}.
\newblock Psychology Press, November 2013.
\newblock ISBN 978-1-317-75782-5.
\newblock Google-Books-ID: NxsuAgAAQBAJ.
\bibitem[Barsalou(1999)]{barsalou_perceptual_1999}
Lawrence~W. Barsalou.
\newblock Perceptual symbol systems.
\newblock \emph{Behavioral and Brain Sciences}, 22\penalty0 (4):\penalty0 577--660, August 1999.
\newblock ISSN 1469-1825, 0140-525X.
\newblock \doi{10.1017/S0140525X99002149}.
\newblock URL \url{https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/perceptual-symbol-systems/C2D720D63C1CE3D7153F6BA473F9DD87}.
\newblock Publisher: Cambridge University Press.
\bibitem[Barsalou(2008)]{barsalou_grounded_2008}
Lawrence~W. Barsalou.
\newblock Grounded {Cognition}.
\newblock \emph{Annual Review of Psychology}, 59\penalty0 (1):\penalty0 617--645, 2008.
\newblock \doi{10.1146/annurev.psych.59.103006.093639}.
\newblock URL \url{https://doi.org/10.1146/annurev.psych.59.103006.093639}.
\newblock \_eprint: https://doi.org/10.1146/annurev.psych.59.103006.093639.
\bibitem[Vigliocco et~al.(2009)Vigliocco, Meteyard, Andrews, and Kousta]{vigliocco_toward_2009}
Gabriella Vigliocco, Lotte Meteyard, Mark Andrews, and Stavroula Kousta.
\newblock Toward a theory of semantic representation.
\newblock \emph{Language and Cognition}, 1\penalty0 (2):\penalty0 219--247, June 2009.
\newblock ISSN 1866-9808, 1866-9859.
\newblock \doi{10.1515/LANGCOG.2009.011}.
\newblock URL \url{https://www.cambridge.org/core/journals/language-and-cognition/article/abs/toward-a-theory-of-semantic-representation/90A3FB3B15DB7034858BB808B41DCA4B}.
\newblock Publisher: Cambridge University Press.
\bibitem[Vigliocco et~al.(2014{\natexlab{b}})Vigliocco, Perniss, and Vinson]{vigliocco_language_2014}
Gabriella Vigliocco, Pamela Perniss, and David Vinson.
\newblock Language as a multimodal phenomenon: implications for language learning, processing and evolution.
\newblock \emph{Philosophical Transactions of the Royal Society B: Biological Sciences}, 369\penalty0 (1651):\penalty0 20130292, September 2014{\natexlab{b}}.
\newblock \doi{10.1098/rstb.2013.0292}.
\newblock URL \url{https://royalsocietypublishing.org/doi/10.1098/rstb.2013.0292}.
\newblock Publisher: Royal Society.
\bibitem[Andrews et~al.(2009)Andrews, Vigliocco, and Vinson]{andrews_integrating_2009}
Mark Andrews, Gabriella Vigliocco, and David Vinson.
\newblock Integrating experiential and distributional data to learn semantic representations.
\newblock \emph{Psychological Review}, 116\penalty0 (3):\penalty0 463--498, 2009.
\newblock ISSN 1939-1471.
\newblock \doi{10.1037/a0016261}.
\newblock Place: US Publisher: American Psychological Association.
\bibitem[Andrews et~al.(2014)Andrews, Frank, and Vigliocco]{andrews_reconciling_2014}
Mark Andrews, Stefan Frank, and Gabriella Vigliocco.
\newblock Reconciling {Embodied} and {Distributional} {Accounts} of {Meaning} in {Language}.
\newblock \emph{Topics in Cognitive Science}, 6\penalty0 (3):\penalty0 359--370, 2014.
\newblock ISSN 1756-8765.
\newblock \doi{10.1111/tops.12096}.
\newblock URL \url{https://onlinelibrary.wiley.com/doi/abs/10.1111/tops.12096}.
\newblock \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/tops.12096.
\bibitem[Louwerse and Jeuniaux(2010)]{louwerse_linguistic_2010}
Max~M. Louwerse and Patrick Jeuniaux.
\newblock The linguistic and embodied nature of conceptual processing.
\newblock \emph{Cognition}, 114\penalty0 (1):\penalty0 96--104, January 2010.
\newblock ISSN 0010-0277.
\newblock \doi{10.1016/j.cognition.2009.09.002}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0010027709002157}.
\bibitem[Kousta et~al.(2011)Kousta, Vigliocco, Vinson, Andrews, and Del~Campo]{kousta_representation_2011}
Stavroula-Thaleia Kousta, Gabriella Vigliocco, David~P. Vinson, Mark Andrews, and Elena Del~Campo.
\newblock The representation of abstract words: {Why} emotion matters.
\newblock \emph{Journal of Experimental Psychology: General}, 140\penalty0 (1):\penalty0 14--34, 2011.
\newblock ISSN 1939-2222.
\newblock \doi{10.1037/a0021446}.
\newblock Place: US Publisher: American Psychological Association.
\bibitem[Riordan and Jones(2011)]{riordan_redundancy_2011}
Brian Riordan and Michael~N. Jones.
\newblock Redundancy in {Perceptual} and {Linguistic} {Experience}: {Comparing} {Feature}-{Based} and {Distributional} {Models} of {Semantic} {Representation}.
\newblock \emph{Topics in Cognitive Science}, 3\penalty0 (2):\penalty0 303--345, 2011.
\newblock ISSN 1756-8765.
\newblock \doi{10.1111/j.1756-8765.2010.01111.x}.
\newblock URL \url{https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1756-8765.2010.01111.x}.
\newblock \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1756-8765.2010.01111.x.
\bibitem[Kewenig et~al.(2023)Kewenig, Vigliocco, and Skipper]{kewenig_when_2023}
Viktor Kewenig, Gabriella Vigliocco, and Jeremy~I. Skipper.
\newblock When {Abstract} {Becomes} {Concrete}: {Naturalistic} {Encoding} of {Concepts} in the {Brain}, August 2023.
\newblock URL \url{https://www.biorxiv.org/content/10.1101/2022.09.08.506944v3}.
\newblock Pages: 2022.09.08.506944 Section: New Results.
\bibitem[Hahamy et~al.(2023)Hahamy, Dubossarsky, and Behrens]{hahamy_human_2023}
Avital Hahamy, Haim Dubossarsky, and Timothy E.~J. Behrens.
\newblock The human brain reactivates context-specific past information at event boundaries of naturalistic experiences.
\newblock \emph{Nature Neuroscience}, 26\penalty0 (6):\penalty0 1080--1089, June 2023.
\newblock ISSN 1546-1726.
\newblock \doi{10.1038/s41593-023-01331-6}.
\bibitem[Zadbood et~al.(2022)Zadbood, Nastase, Chen, Norman, and Hasson]{zadbood_neural_2022}
Asieh Zadbood, Samuel Nastase, Janice Chen, Kenneth~A Norman, and Uri Hasson.
\newblock Neural representations of naturalistic events are updated as our understanding of the past changes.
\newblock \emph{eLife}, 11:\penalty0 e79045, December 2022.
\newblock ISSN 2050-084X.
\newblock \doi{10.7554/eLife.79045}.
\newblock URL \url{https://doi.org/10.7554/eLife.79045}.
\newblock Publisher: eLife Sciences Publications, Ltd.
\bibitem[Baldassano et~al.()Baldassano, Hasson, and Norman]{baldassano_representation_2018}
Christopher Baldassano, Uri Hasson, and Kenneth~A. Norman.
\newblock Representation of real-world event schemas during narrative perception.
\newblock 38\penalty0 (45):\penalty0 9689--9699.
\newblock ISSN 0270-6474, 1529-2401.
\newblock \doi{10.1523/JNEUROSCI.0251-18.2018}.
\newblock URL \url{https://www.jneurosci.org/content/38/45/9689}.
\newblock Publisher: Society for Neuroscience Section: Research Articles.
\bibitem[ben(2018)]{ben-yakov_hippocampal_2018}
The {Hippocampal} {Film} {Editor}: {Sensitivity} and {Specificity} to {Event} {Boundaries} in {Continuous} {Experience}.
\newblock \emph{Journal of Neuroscience}, 38\penalty0 (47):\penalty0 10057--10068, November 2018.
\newblock ISSN 0270-6474, 1529-2401.
\newblock \doi{10.1523/JNEUROSCI.0524-18.2018}.
\newblock URL \url{https://www.jneurosci.org/content/38/47/10057}.
\newblock Publisher: Society for Neuroscience Section: Research Articles.
\bibitem[Pezzulo et~al.(2023)Pezzulo, Parr, Cisek, Clark, and Friston]{pezzulo_generating_2023}
Giovanni Pezzulo, Thomas Parr, Paul Cisek, Andy Clark, and Karl Friston.
\newblock Generating meaning: active inference and the scope and limits of passive {AI}.
\newblock \emph{Trends in Cognitive Sciences}, November 2023.
\newblock ISSN 1364-6613.
\newblock \doi{10.1016/j.tics.2023.10.002}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S1364661323002607}.
\bibitem[Alayrac et~al.(2022)Alayrac, Donahue, Luc, Miech, Barr, Hasson, Lenc, Mensch, Millican, Reynolds, Ring, Rutherford, Cabi, Han, Gong, Samangooei, Monteiro, Menick, Borgeaud, Nematzadeh, Sharifzadeh, Bi{\'n}kowski, Barreira, Vinyals, Zisserman, and Simonyan]{alayrac2022flamingo}
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob~L Menick, Sebastian Borgeaud, Aida Nematzadeh, Sahand Sharifzadeh, Miko{\l}aj Bi{\'n}kowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar{\'e}n Simonyan.
\newblock Flamingo: a visual language model for few-shot learning.
\newblock In \emph{Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Main Conference Track}, 2022.
\bibitem[Bransford et~al.(1972)Bransford, Barclay, and Franks]{bransford_sentence_1972}
John~D. Bransford, J.~Richard Barclay, and Jeffery~J. Franks.
\newblock Sentence memory: {A} constructive versus interpretive approach.
\newblock \emph{Cognitive Psychology}, 3\penalty0 (2):\penalty0 193--209, 1972.
\newblock ISSN 1095-5623.
\newblock \doi{10.1016/0010-0285(72)90003-5}.
\newblock Place: Netherlands Publisher: Elsevier Science.
\bibitem[Huettig(2015)]{huettig_four_2015}
Falk Huettig.
\newblock Four central questions about prediction in language processing.
\newblock \emph{Brain Research}, 1626:\penalty0 118--135, November 2015.
\newblock ISSN 0006-8993.
\newblock \doi{10.1016/j.brainres.2015.02.014}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0006899315001146}.
\bibitem[Tanenhaus et~al.(1995)Tanenhaus, Spivey-Knowlton, Eberhard, and Sedivy]{tanenhaus_integration_1995}
Michael~K. Tanenhaus, Michael~J. Spivey-Knowlton, Kathleen~M. Eberhard, and Julie~C. Sedivy.
\newblock Integration of {Visual} and {Linguistic} {Information} in {Spoken} {Language} {Comprehension}.
\newblock \emph{Science}, 268\penalty0 (5217):\penalty0 1632--1634, June 1995.
\newblock \doi{10.1126/science.7777863}.
\newblock URL \url{https://www.science.org/doi/10.1126/science.7777863}.
\newblock Publisher: American Association for the Advancement of Science.
\bibitem[Eichert et~al.(2018)Eichert, Peeters, and Hagoort]{eichert_language-driven_2018}
Nicole Eichert, David Peeters, and Peter Hagoort.
\newblock Language-driven anticipatory eye movements in virtual reality.
\newblock \emph{Behavior Research Methods}, 50\penalty0 (3):\penalty0 1102--1115, June 2018.
\newblock ISSN 1554-3528.
\newblock \doi{10.3758/s13428-017-0929-z}.
\bibitem[Altmann and Kamide(1999)]{altmann_incremental_1999}
Gerry T.~M. Altmann and Yuki Kamide.
\newblock Incremental interpretation at verbs: {Restricting} the domain of subsequent reference.
\newblock \emph{Cognition}, 73\penalty0 (3):\penalty0 247--264, 1999.
\newblock ISSN 1873-7838.
\newblock \doi{10.1016/S0010-0277(99)00059-1}.
\newblock Place: Netherlands Publisher: Elsevier Science.
\bibitem[Kamide et~al.(2003)Kamide, Scheepers, and Altmann]{kamide_integration_2003}
Yuki Kamide, Christoph Scheepers, and Gerry T.~M. Altmann.
\newblock Integration of {Syntactic} and {Semantic} {Information} in {Predictive} {Processing}: {Cross}-{Linguistic} {Evidence} from {German} and {English}.
\newblock \emph{Journal of Psycholinguistic Research}, 32\penalty0 (1):\penalty0 37--55, January 2003.
\newblock ISSN 1573-6555.
\newblock \doi{10.1023/A:1021933015362}.
\newblock URL \url{https://doi.org/10.1023/A:1021933015362}.
\bibitem[Kukona et~al.(2014)Kukona, Altmann, and Kamide]{kukona_knowing_2014}
Anuenue Kukona, Gerry T.~M. Altmann, and Yuki Kamide.
\newblock Knowing what, where, and when: {Event} comprehension in language processing.
\newblock \emph{Cognition}, 133\penalty0 (1):\penalty0 25--31, October 2014.
\newblock ISSN 0010-0277.
\newblock \doi{10.1016/j.cognition.2014.05.011}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0010027714001000}.
\bibitem[Rommers et~al.(2015)Rommers, Meyer, and Huettig]{rommers_verbal_2015}
Joost Rommers, Antje~S. Meyer, and Falk Huettig.
\newblock Verbal and nonverbal predictors of language-mediated anticipatory eye movements.
\newblock \emph{Attention, Perception, \& Psychophysics}, 77\penalty0 (3):\penalty0 720--730, April 2015.
\newblock ISSN 1943-393X.
\newblock \doi{10.3758/s13414-015-0873-x}.
\newblock URL \url{https://doi.org/10.3758/s13414-015-0873-x}.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{vaswani_attention_2017}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan~N Gomez, Łukasz Kaiser, and Illia Polosukhin.
\newblock Attention is {All} you {Need}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}}, volume~30. Curran Associates, Inc., 2017.
\newblock URL \url{https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html}.
\bibitem[Serrano and Smith(2019)]{serrano_is_2019}
Sofia Serrano and Noah~A. Smith.
\newblock Is {Attention} {Interpretable}?, June 2019.
\newblock URL \url{http://arxiv.org/abs/1906.03731}.
\newblock arXiv:1906.03731 [cs].
\bibitem[Da~Costa and Chaves(2020)]{da_costa_assessing_2020}
Jillian Da~Costa and Rui Chaves.
\newblock Assessing the ability of {Transformer}-based {Neural} {Models} to represent structurally unbounded dependencies.
\newblock In \emph{Proceedings of the {Society} for {Computation} in {Linguistics} 2020}, pages 12--21, New York, New York, January 2020. Association for Computational Linguistics.
\newblock URL \url{https://aclanthology.org/2020.scil-1.2}.
\bibitem[Busse et~al.(2005)Busse, Roberts, Crist, Weissman, and Woldorff]{busse_spread_2005}
Laura Busse, Kenneth~C. Roberts, Roy~E. Crist, Daniel~H. Weissman, and Marty~G. Woldorff.
\newblock The spread of attention across modalities and space in a multisensory object.
\newblock \emph{Proceedings of the National Academy of Sciences}, 102\penalty0 (51):\penalty0 18751--18756, December 2005.
\newblock \doi{10.1073/pnas.0507704102}.
\newblock URL \url{https://www.pnas.org/doi/10.1073/pnas.0507704102}.
\newblock Publisher: Proceedings of the National Academy of Sciences.
\bibitem[Tay et~al.(2021)Tay, Bahri, Metzler, Juan, Zhao, and Zheng]{tay_synthesizer_2021}
Yi~Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng.
\newblock Synthesizer: {Rethinking} {Self}-{Attention} in {Transformer} {Models}, May 2021.
\newblock URL \url{http://arxiv.org/abs/2005.00743}.
\newblock arXiv:2005.00743 [cs].
\bibitem[Lindsay(2020)]{lindsay_attention_2020}
Grace~W. Lindsay.
\newblock Attention in {Psychology}, {Neuroscience}, and {Machine} {Learning}.
\newblock \emph{Frontiers in Computational Neuroscience}, 14, 2020.
\newblock ISSN 1662-5188.
\newblock URL \url{https://www.frontiersin.org/articles/10.3389/fncom.2020.00029}.
\bibitem[Conwell et~al.(2022)Conwell, Prince, Kay, Alvarez, and Konkle]{conwell2022billion}
Colin Conwell, Jacob~S. Prince, Kendrick~N. Kay, George~A. Alvarez, and Talia Konkle.
\newblock What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?
\newblock \emph{Preprint}, 2022.
\newblock \doi{https://doi.org/10.1101/2022.03.28.485868}.
\newblock This article is a preprint and has not been certified by peer review.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]{dosovitskiy_image_2021}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An {Image} is {Worth} 16x16 {Words}: {Transformers} for {Image} {Recognition} at {Scale}, June 2021.
\newblock URL \url{http://arxiv.org/abs/2010.11929}.
\newblock arXiv:2010.11929 [cs].
\bibitem[Tuli et~al.(2021)Tuli, Dasgupta, Grant, and Griffiths]{tuli_are_2021}
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas~L. Griffiths.
\newblock Are {Convolutional} {Neural} {Networks} or {Transformers} more like human vision?, July 2021.
\newblock URL \url{http://arxiv.org/abs/2105.07197}.
\newblock arXiv:2105.07197 [cs].
\bibitem[Sood et~al.(2020)Sood, Tannert, Frassinelli, Bulling, and Vu]{sood_interpreting_2020}
Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, and Ngoc~Thang Vu.
\newblock Interpreting {Attention} {Models} with {Human} {Visual} {Attention} in {Machine} {Reading} {Comprehension}, October 2020.
\newblock URL \url{http://arxiv.org/abs/2010.06396}.
\newblock arXiv:2010.06396 [cs].
\bibitem[Yang et~al.(2023)Yang, Mondal, Ahn, Zelinsky, Hoai, and Samaras]{yang_predicting_2023}
Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras.
\newblock Predicting {Human} {Attention} using {Computational} {Attention}, April 2023.
\newblock URL \url{http://arxiv.org/abs/2303.09383}.
\newblock arXiv:2303.09383 [cs].
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{radford_learning_2021}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
\newblock Learning {Transferable} {Visual} {Models} {From} {Natural} {Language} {Supervision}, February 2021.
\newblock URL \url{http://arxiv.org/abs/2103.00020}.
\newblock arXiv:2103.00020 [cs].
\bibitem[Touvron et~al.(2023)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozi{\`e}re, Goyal, Hambro, Azhar, Rodriguez, Joulin, Grave, and Lample]{touvron2023llama}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth{\'e}e Lacroix, Baptiste Rozi{\`e}re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
\newblock Llama: Open and efficient foundation language models.
\newblock \emph{Unknown Journal}, 2023.
\newblock URL \url{Unknown URL}.
\bibitem[Gao et~al.(2023)Gao, Han, Zhang, Lin, Geng, Zhou, Zhang, Lu, He, Yue, Li, and Qiao]{gao2023llamaadapter}
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu~Qiao.
\newblock Llama-adapter v2: Parameter-efficient visual instruction model.
\newblock \emph{The Journal or Conference Name}, The Volume Number\penalty0 (The Issue Number):\penalty0 The Page Range, 2023.
\newblock URL \url{The URL}.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{he_deep_2016}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep {Residual} {Learning} for {Image} {Recognition}.
\newblock In \emph{2016 {IEEE} {Conference} on {Computer} {Vision} and {Pattern} {Recognition} ({CVPR})}, pages 770--778, Las Vegas, NV, USA, June 2016. IEEE.
\newblock ISBN 978-1-4673-8851-1.
\newblock \doi{10.1109/CVPR.2016.90}.
\newblock URL \url{http://ieeexplore.ieee.org/document/7780459/}.
\bibitem[Tenney et~al.(2019)Tenney, Das, and Pavlick]{tenney_bert_2019}
Ian Tenney, Dipanjan Das, and Ellie Pavlick.
\newblock {BERT} {Rediscovers} the {Classical} {NLP} {Pipeline}.
\newblock In Anna Korhonen, David Traum, and Lluís Màrquez, editors, \emph{Proceedings of the 57th {Annual} {Meeting} of the {Association} for {Computational} {Linguistics}}, pages 4593--4601, Florence, Italy, July 2019. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/P19-1452}.
\newblock URL \url{https://aclanthology.org/P19-1452}.
\bibitem[Rogers et~al.(2021)Rogers, Kovaleva, and Rumshisky]{rogers_primer_2021}
Anna Rogers, Olga Kovaleva, and Anna Rumshisky.
\newblock A {Primer} in {BERTology}: {What} {We} {Know} {About} {How} {BERT} {Works}.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 8:\penalty0 842--866, January 2021.
\newblock ISSN 2307-387X.
\newblock \doi{10.1162/tacl_a_00349}.
\newblock URL \url{https://doi.org/10.1162/tacl_a_00349}.
\bibitem[Liu et~al.(2019)Liu, Gardner, Belinkov, Peters, and Smith]{liu_linguistic_2019}
Nelson~F. Liu, Matt Gardner, Yonatan Belinkov, Matthew~E. Peters, and Noah~A. Smith.
\newblock Linguistic {Knowledge} and {Transferability} of {Contextual} {Representations}, April 2019.
\newblock URL \url{http://arxiv.org/abs/1903.08855}.
\newblock arXiv:1903.08855 [cs].
\bibitem[Oota et~al.()Oota, Gupta, and Toneva]{oota_joint_2022}
Subba~Reddy Oota, Manish Gupta, and Mariya Toneva.
\newblock Joint processing of linguistic properties in brains and language models.
\newblock URL \url{http://arxiv.org/abs/2212.08094}.
\bibitem[Caucheteux and King(2022)]{caucheteux_brains_2022}
Charlotte Caucheteux and Jean-Rémi King.
\newblock Brains and algorithms partially converge in natural language processing.
\newblock \emph{Communications Biology}, 5\penalty0 (1):\penalty0 1--10, February 2022.
\newblock ISSN 2399-3642.
\newblock \doi{10.1038/s42003-022-03036-1}.
\newblock URL \url{https://www.nature.com/articles/s42003-022-03036-1}.
\newblock Number: 1 Publisher: Nature Publishing Group.
\bibitem[Jain et~al.()Jain, Vo, Wehbe, and Huth]{jain_computational_2023}
Shailee Jain, Vy~A. Vo, Leila Wehbe, and Alexander~G. Huth.
\newblock Computational language modeling and the promise of in silico experimentation.
\newblock pages 1--65.
\newblock ISSN 2641-4368.
\newblock \doi{10.1162/nol_a_00101}.
\newblock URL \url{https://doi.org/10.1162/nol_a_00101}.
\bibitem[Tang et~al.()Tang, {LeBel}, Jain, and Huth]{tang_semantic_2022}
Jerry Tang, Amanda {LeBel}, Shailee Jain, and Alexander~G. Huth.
\newblock Semantic reconstruction of continuous language from non-invasive brain recordings.
\newblock URL \url{https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1}.
\newblock Pages: 2022.09.29.509744 Section: New Results.
\bibitem[Bergen(2007)]{bergen_experimental_2007}
Benjamin Bergen.
\newblock Experimental methods for simulation semantics.
\newblock In \emph{Methods in cognitive linguistics}, pages 277--301. John Benjamins Publishing Company, Amsterdam, Netherlands, 2007.
\newblock ISBN 978-90-272-2372-2 978-90-272-2371-5.
\newblock \doi{10.1075/hcp.18.19ber}.
\bibitem[Fischer and Weber(1993)]{fischer_express_1993}
B.~Fischer and H.~Weber.
\newblock Express saccades and visual attention.
\newblock \emph{Behavioral and Brain Sciences}, 16\penalty0 (3):\penalty0 553--567, September 1993.
\newblock ISSN 1469-1825, 0140-525X.
\newblock \doi{10.1017/S0140525X00031575}.
\newblock URL \url{https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/express-saccades-and-visual-attention/B38F6BA1820BDD8753880DCD2814AC7D}.
\newblock Publisher: Cambridge University Press.
\bibitem[Dimigen et~al.(2022)Dimigen, Schild, Hohlfeld, and Sommer]{dimigen_auditory_2022}
Olaf Dimigen, Ulrike Schild, Annette Hohlfeld, and Werner Sommer.
\newblock Auditory {Language} {Comprehension} {During} {Saccadic} {Eye} {Movements}: {An} {Investigation} {Using} the {N400} {Event}-{Related} {Brain} {Potential}, April 2022.
\newblock URL \url{https://www.biorxiv.org/content/10.1101/2022.04.05.487157v1}.
\newblock Pages: 2022.04.05.487157 Section: New Results.
\bibitem[Kutas and Federmeier(2011)]{kutas_thirty_2011}
Marta Kutas and Kara~D. Federmeier.
\newblock Thirty {Years} and {Counting}: {Finding} {Meaning} in the {N400} {Component} of the {Event}-{Related} {Brain} {Potential} ({ERP}).
\newblock \emph{Annual Review of Psychology}, 62\penalty0 (1):\penalty0 621--647, 2011.
\newblock \doi{10.1146/annurev.psych.093008.131123}.
\newblock URL \url{https://doi.org/10.1146/annurev.psych.093008.131123}.
\newblock \_eprint: https://doi.org/10.1146/annurev.psych.093008.131123.
\bibitem[Marslen-Wilson(1973)]{marslen-wilson_linguistic_1973}
William Marslen-Wilson.
\newblock Linguistic structure and speech shadowing at very short latencies.
\newblock \emph{Nature}, 244\penalty0 (5417):\penalty0 522--523, 1973.
\newblock ISSN 1476-4687.
\newblock \doi{10.1038/244522a0}.
\newblock Place: United Kingdom Publisher: Nature Publishing Group.
\bibitem[Jiahui et~al.(2021)Jiahui, Feilong, di~Oleggio~Castello, Nastase, Haxby, and Gobbini]{jiahui2021modeling}
Guo Jiahui, Ma~Feilong, Matteo~Visconti di~Oleggio~Castello, Samuel~A. Nastase, James~V. Haxby, and M.~Ida Gobbini.
\newblock Modeling naturalistic face processing in humans with deep convolutional neural networks.
\newblock \emph{Unknown Journal}, November 2021.
\newblock \doi{https://doi.org/10.1101/2021.11.17.469009}.
\bibitem[Li et~al.(2021)Li, Wu, Shrivastava, and Davis]{li_2d_2021}
Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry~S. Davis.
\newblock {2D} or not {2D}? {Adaptive} {3D} {Convolution} {Selection} for {Efficient} {Video} {Recognition}, April 2021.
\newblock URL \url{http://arxiv.org/abs/2012.14950}.
\newblock arXiv:2012.14950 [cs].
\bibitem[Baltrušaitis et~al.(2017)Baltrušaitis, Ahuja, and Morency]{baltrusaitis_multimodal_2017}
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.
\newblock Multimodal {Machine} {Learning}: {A} {Survey} and {Taxonomy}, August 2017.
\newblock URL \url{http://arxiv.org/abs/1705.09406}.
\newblock arXiv:1705.09406 [cs].
\bibitem[noa({\natexlab{a}})]{noauthor_deep_nodate}
A deep dive into the code of the {BERT} model {\textbar} by alexey kravets {\textbar} towards data science, {\natexlab{a}}.
\newblock URL \url{https://towardsdatascience.com/deep-dive-into-the-code-of-bert-model-9f618472353e}.
\bibitem[Heyselaar et~al.(2020)Heyselaar, Peeters, and Hagoort]{heyselaar_we_2020}
Evelien Heyselaar, David Peeters, and Peter Hagoort.
\newblock Do we predict upcoming speech content in naturalistic environments?
\newblock \emph{Language, Cognition and Neuroscience}, 36:\penalty0 1--22, December 2020.
\newblock \doi{10.1080/23273798.2020.1859568}.
\bibitem[Ryskin and Nieuwland(2023)]{ryskin_prediction_2023}
Rachel Ryskin and Mante~S. Nieuwland.
\newblock Prediction during language comprehension: what is next?
\newblock \emph{Trends in Cognitive Sciences}, 27\penalty0 (11):\penalty0 1032--1052, November 2023.
\newblock ISSN 1364-6613, 1879-307X.
\newblock \doi{10.1016/j.tics.2023.08.003}.
\newblock URL \url{https://www.cell.com/trends/cognitive-sciences/abstract/S1364-6613(23)00199-7}.
\newblock Publisher: Elsevier.
\bibitem[Wang et~al.(2020)Wang, Antonenko, and Dawson]{wang_does_2020}
Jiahui Wang, Pavlo Antonenko, and Kara Dawson.
\newblock Does visual attention to the instructor in online video affect learning and learner perceptions? {An} eye-tracking analysis.
\newblock \emph{Computers \& Education}, 146:\penalty0 103779, March 2020.
\newblock ISSN 0360-1315.
\newblock \doi{10.1016/j.compedu.2019.103779}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S036013151930329X}.
\bibitem[Boynton(2005)]{boynton_attention_2005}
Geoffrey~M. Boynton.
\newblock Attention and visual perception.
\newblock \emph{Current Opinion in Neurobiology}, 15\penalty0 (4):\penalty0 465--469, 2005.
\newblock ISSN 1873-6882.
\newblock \doi{10.1016/j.conb.2005.06.009}.
\newblock Place: Netherlands Publisher: Elsevier Science.
\bibitem[Chen and Hutchinson(2019)]{chen_what_2019}
Danlei Chen and J.~Benjamin Hutchinson.
\newblock What {Is} {Memory}-{Guided} {Attention}? {How} {Past} {Experiences} {Shape} {Selective} {Visuospatial} {Attention} in the {Present}.
\newblock \emph{Current Topics in Behavioral Neurosciences}, 41:\penalty0 185--212, 2019.
\newblock ISSN 1866-3370.
\newblock \doi{10.1007/7854_2018_76}.
\bibitem[Jiang(2018)]{jiang_habitual_2018}
Yuhong~V. Jiang.
\newblock Habitual versus goal-driven attention.
\newblock \emph{Cortex; a journal devoted to the study of the nervous system and behavior}, 102:\penalty0 107--120, May 2018.
\newblock ISSN 0010-9452.
\newblock \doi{10.1016/j.cortex.2017.06.018}.
\newblock URL \url{https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5754262/}.
\bibitem[Derakshan and Eysenck(2010)]{derakshan_introduction_2010}
Nazanin Derakshan and Michael~W. Eysenck.
\newblock Introduction to the special issue: {Emotional} states, attention, and working memory.
\newblock \emph{COGNITION AND EMOTION}, February 2010.
\newblock \doi{10.1080/02699930903412120}.
\newblock URL \url{https://www.tandfonline.com/doi/full/10.1080/02699930903412120}.
\newblock Publisher: Taylor \& Francis Group.
\bibitem[Nobre and van Ede(2018)]{nobre_anticipated_2018}
Anna~C. Nobre and Freek van Ede.
\newblock Anticipated moments: temporal structure in attention.
\newblock \emph{Nature Reviews Neuroscience}, 19\penalty0 (1):\penalty0 34--48, January 2018.
\newblock ISSN 1471-0048.
\newblock \doi{10.1038/nrn.2017.141}.
\newblock URL \url{https://www.nature.com/articles/nrn.2017.141}.
\newblock Number: 1 Publisher: Nature Publishing Group.
\bibitem[Custers and Aarts(2011)]{custers_learning_2011}
Ruud Custers and Henk Aarts.
\newblock Learning of predictive relations between events depends on attention, not on awareness.
\newblock \emph{Consciousness and Cognition}, 20\penalty0 (2):\penalty0 368--378, June 2011.
\newblock ISSN 1053-8100.
\newblock \doi{10.1016/j.concog.2010.05.011}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S105381001000111X}.
\bibitem[Magnuson(2019)]{magnuson_fixations_2019}
James~S. Magnuson.
\newblock Fixations in the visual world paradigm: where, when, why?
\newblock \emph{Journal of Cultural Cognitive Science}, 3\penalty0 (2):\penalty0 113--139, November 2019.
\newblock ISSN 2520-1018.
\newblock \doi{10.1007/s41809-019-00035-3}.
\newblock URL \url{https://doi.org/10.1007/s41809-019-00035-3}.
\bibitem[Foulsham et~al.(2011)Foulsham, Walker, and Kingstone]{foulsham_where_2011}
Tom Foulsham, Esther Walker, and Alan Kingstone.
\newblock The where, what and when of gaze allocation in the lab and the natural environment.
\newblock \emph{Vision Research}, 51\penalty0 (17):\penalty0 1920--1931, September 2011.
\newblock ISSN 0042-6989.
\newblock \doi{10.1016/j.visres.2011.07.002}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0042698911002392}.
\bibitem[Chemero(2023)]{chemero_llms_2023}
Anthony Chemero.
\newblock {LLMs} differ from human cognition because they are not embodied.
\newblock \emph{Nature Human Behaviour}, 7\penalty0 (11):\penalty0 1828--1829, November 2023.
\newblock ISSN 2397-3374.
\newblock \doi{10.1038/s41562-023-01723-5}.
\newblock URL \url{https://www.nature.com/articles/s41562-023-01723-5}.
\newblock Number: 11 Publisher: Nature Publishing Group.
\bibitem[Anderson(2003)]{anderson_embodied_2003}
Michael~L. Anderson.
\newblock Embodied {Cognition}: {A} field guide.
\newblock \emph{Artificial Intelligence}, 149\penalty0 (1):\penalty0 91--130, September 2003.
\newblock ISSN 0004-3702.
\newblock \doi{10.1016/S0004-3702(03)00054-7}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0004370203000547}.
\bibitem[Meteyard et~al.(2012)Meteyard, Cuadrado, Bahrami, and Vigliocco]{meteyard_coming_2012}
Lotte Meteyard, Sara~Rodriguez Cuadrado, Bahador Bahrami, and Gabriella Vigliocco.
\newblock Coming of age: {A} review of embodiment and the neuroscience of semantics.
\newblock \emph{Cortex}, 48\penalty0 (7):\penalty0 788--804, July 2012.
\newblock ISSN 0010-9452.
\newblock \doi{10.1016/j.cortex.2010.11.002}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0010945210002662}.
\bibitem[Barsalou(2009)]{barsalou_simulation_2009}
Lawrence~W. Barsalou.
\newblock Simulation, situated conceptualization, and prediction.
\newblock \emph{Philosophical Transactions of the Royal Society B: Biological Sciences}, 364\penalty0 (1521):\penalty0 1281--1289, May 2009.
\newblock \doi{10.1098/rstb.2008.0319}.
\newblock URL \url{https://royalsocietypublishing.org/doi/full/10.1098/rstb.2008.0319}.
\newblock Publisher: Royal Society.
\bibitem[Zador et~al.(2023)Zador, Escola, Richards, Ölveczky, Bengio, Boahen, Botvinick, Chklovskii, Churchland, Clopath, DiCarlo, Ganguli, Hawkins, Körding, Koulakov, LeCun, Lillicrap, Marblestone, Olshausen, Pouget, Savin, Sejnowski, Simoncelli, Solla, Sussillo, Tolias, and Tsao]{zador_catalyzing_2023}
Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, James DiCarlo, Surya Ganguli, Jeff Hawkins, Konrad Körding, Alexei Koulakov, Yann LeCun, Timothy Lillicrap, Adam Marblestone, Bruno Olshausen, Alexandre Pouget, Cristina Savin, Terrence Sejnowski, Eero Simoncelli, Sara Solla, David Sussillo, Andreas~S. Tolias, and Doris Tsao.
\newblock Catalyzing next-generation {Artificial} {Intelligence} through {NeuroAI}.
\newblock \emph{Nature Communications}, 14\penalty0 (1):\penalty0 1597, March 2023.
\newblock ISSN 2041-1723.
\newblock \doi{10.1038/s41467-023-37180-x}.
\newblock URL \url{https://www.nature.com/articles/s41467-023-37180-x}.
\bibitem[noa({\natexlab{b}})]{noauthor_palm-e_nodate}
{PaLM}-{E}: {An} {Embodied} {Multimodal} {Language} {Model}, {\natexlab{b}}.
\newblock URL \url{https://palm-e.github.io/}.
\bibitem[Srivastava et~al.(2022)Srivastava, Rastogi, Rao, Shoeb, Abid, Fisch, Brown, Santoro, Gupta, Garriga-Alonso, Kluska, Lewkowycz, Agarwal, Power, Ray, Warstadt, Kocurek, Safaya, Tazarv, Xiang, Parrish, Nie, Hussain, Askell, Dsouza, Slone, Rahane, Iyer, Andreassen, Madotto, Santilli, Stuhlmüller, Dai, La, Lampinen, Zou, Jiang, Chen, Vuong, Gupta, Gottardi, Norelli, Venkatesh, Gholamidavoodi, Tabassum, Menezes, Kirubarajan, Mullokandov, Sabharwal, Herrick, Efrat, Erdem, Karakaş, Roberts, Loe, Zoph, Bojanowski, Özyurt, Hedayatnia, Neyshabur, Inden, Stein, Ekmekci, Lin, Howald, Diao, Dour, Stinson, Argueta, Ramírez, Singh, Rathkopf, Meng, Baral, Wu, Callison-Burch, Waites, Voigt, Manning, Potts, Ramirez, Rivera, Siro, Raffel, Ashcraft, Garbacea, Sileo, Garrette, Hendrycks, Kilman, Roth, Freeman, Khashabi, Levy, González, Perszyk, Hernandez, Chen, Ippolito, Gilboa, Dohan, Drakard, Jurgens, Datta, Ganguli, Emelin, Kleyko, Yuret, Chen, Tam, Hupkes, Misra, Buzan, Mollo, Yang, Lee, Shutova, Cubuk, Segal,
Hagerman, Barnes, Donoway, Pavlick, Rodola, Lam, Chu, Tang, Erdem, Chang, Chi, Dyer, Jerzak, Kim, Manyasi, Zheltonozhskii, Xia, Siar, Martínez-Plumed, Happé, Chollet, Rong, Mishra, Winata, de~Melo, Kruszewski, Parascandolo, Mariani, Wang, Jaimovitch-López, Betz, Gur-Ari, Galijasevic, Kim, Rashkin, Hajishirzi, Mehta, Bogar, Shevlin, Schütze, Yakura, Zhang, Wong, Ng, Noble, Jumelet, Geissinger, Kernion, Hilton, Lee, Fisac, Simon, Koppel, Zheng, Zou, Kocoń, Thompson, Kaplan, Radom, Sohl-Dickstein, Phang, Wei, Yosinski, Novikova, Bosscher, Marsh, Kim, Taal, Engel, Alabi, Xu, Song, Tang, Waweru, Burden, Miller, Balis, Berant, Frohberg, Rozen, Hernandez-Orallo, Boudeman, Jones, Tenenbaum, Rule, Chua, Kanclerz, Livescu, Krauth, Gopalakrishnan, Ignatyeva, Markert, Dhole, Gimpel, Omondi, Mathewson, Chiafullo, Shkaruta, Shridhar, McDonell, Richardson, Reynolds, Gao, Zhang, Dugan, Qin, Contreras-Ochando, Morency, Moschella, Lam, Noble, Schmidt, He, Colón, Metz, Şenel, Bosma, Sap, ter Hoeve, Farooqi, Faruqui,
Mazeika, Baturan, Marelli, Maru, Quintana, Tolkiehn, Giulianelli, Lewis, Potthast, Leavitt, Hagen, Schubert, Baitemirova, Arnaud, McElrath, Yee, Cohen, Gu, Ivanitskiy, Starritt, Strube, Swędrowski, Bevilacqua, Yasunaga, Kale, Cain, Xu, Suzgun, Tiwari, Bansal, Aminnaseri, Geva, Gheini, T, Peng, Chi, Lee, Krakover, Cameron, Roberts, Doiron, Nangia, Deckers, Muennighoff, Keskar, Iyer, Constant, Fiedel, Wen, Zhang, Agha, Elbaghdadi, Levy, Evans, Casares, Doshi, Fung, Liang, Vicol, Alipoormolabashi, Liao, Liang, Chang, Eckersley, Htut, Hwang, Miłkowski, Patil, Pezeshkpour, Oli, Mei, Lyu, Chen, Banjade, Rudolph, Gabriel, Habacker, Delgado, Millière, Garg, Barnes, Saurous, Arakawa, Raymaekers, Frank, Sikand, Novak, Sitelew, LeBras, Liu, Jacobs, Zhang, Salakhutdinov, Chi, Lee, Stovall, Teehan, Yang, Singh, Mohammad, Anand, Dillavou, Shleifer, Wiseman, Gruetter, Bowman, Schoenholz, Han, Kwatra, Rous, Ghazarian, Ghosh, Casey, Bischoff, Gehrmann, Schuster, Sadeghi, Hamdan, Zhou, Srivastava, Shi, Singh, Asaadi, Gu,
Pachchigar, Toshniwal, Upadhyay, Shyamolima, Debnath, Shakeri, Thormeyer, Melzi, Reddy, Makini, Lee, Torene, Hatwar, Dehaene, Divic, Ermon, Biderman, Lin, Prasad, Piantadosi, Shieber, Misherghi, Kiritchenko, Mishra, Linzen, Schuster, Li, Yu, Ali, Hashimoto, Wu, Desbordes, Rothschild, Phan, Wang, Nkinyili, Schick, Kornev, Telleen-Lawton, Tunduny, Gerstenberg, Chang, Neeraj, Khot, Shultz, Shaham, Misra, Demberg, Nyamai, Raunak, Ramasesh, Prabhu, Padmakumar, Srikumar, Fedus, Saunders, Zhang, Vossen, Ren, Tong, Zhao, Wu, Shen, Yaghoobzadeh, Lakretz, Song, Bahri, Choi, Yang, Hao, Chen, Belinkov, Hou, Hou, Bai, Seid, Zhao, Wang, Wang, Wang, and Wu]{srivastava_beyond_2022}
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal~Md Shoeb, Abubakar Abid, Adam Fisch, Adam~R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander~W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman~S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B.~Ryan Roberts, Bao~Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill~Yuchen Lin, Blake Howald,
Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César~Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher~D. Manning, Christopher Potts, Cindy Ramirez, Clara~E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel~Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri~Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin~Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan~A. Chi, Ethan Dyer, Ethan
Jerzak, Ethan Kim, Eunice~Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta~Indra Winata, Gerard de~Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh~Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime~Fernández Fisac, James~B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John~U. Balis,
Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua~B. Tenenbaum, Joshua~S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh~D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li~Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis~Oliveros Colón, Luke Metz, Lütfi~Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose~Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew~L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina~Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael~A. Yee, Michael Cohen, Michael Gu,
Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo~Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund~Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish~Shirish Keskar, Niveditha~S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio~Moreno Casares, Parth Doshi, Pascale Fung, Paul~Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu~Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel~Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón~Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif~A. Saurous, Riku Arakawa, Robbe
Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif~M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel~R. Bowman, Samuel~S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah~A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang~Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha~Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven~T. Piantadosi, Stuart~M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal
Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay~Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu~Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie~J. Wang, Zirui Wang, and Ziyi Wu.
\newblock Beyond the {Imitation} {Game}: {Quantifying} and extrapolating the capabilities of language models, June 2022.
\newblock URL \url{http://arxiv.org/abs/2206.04615}.
\newblock Number: arXiv:2206.04615 arXiv:2206.04615 [cs, stat].
\bibitem[Lenci(2023)]{lenci_understanding_2023}
Alessandro Lenci.
\newblock Understanding {Natural} {Language} {Understanding} {Systems}. {A} {Critical} {Analysis}, March 2023.
\newblock URL \url{http://arxiv.org/abs/2303.04229}.
\newblock arXiv:2303.04229 [cs].
\bibitem[Tankelevitch()]{tankelevitch_231210893_nodate}
Simkute Sellen~Rintel Tankelevitch, Kewenig.
\newblock [2312.10893] {The} {Metacognitive} {Demands} and {Opportunities} of {Generative} {AI}.
\newblock URL \url{https://arxiv.org/abs/2312.10893}.
\bibitem[~()]{noauthor_psyarxiv_nodate}
G.~Spiers H. Musolesi M. De Felice S. Kewenig~V. ~, Vigliocco.
\newblock {PsyArXiv} {Preprints} {\textbar} {Ecological} {Brain}: {Reframing} the {Study} of {Human} {Behaviour} and {Cognition}.
\newblock URL \url{https://osf.io/preprints/psyarxiv/zr4nm}.
\bibitem[Palan and Schitter(2018)]{palan_prolificacsubject_2018}
Stefan Palan and Christian Schitter.
\newblock Prolific.ac—{A} subject pool for online experiments.
\newblock \emph{Journal of Behavioral and Experimental Finance}, 17\penalty0 (C):\penalty0 22--27, 2018.
\newblock ISSN 2214-6350.
\newblock URL \url{https://econpapers.repec.org/article/eeebeexfi/v_3a17_3ay_3a2018_3ai_3ac_3ap_3a22-27.htm}.
\newblock Publisher: Elsevier.
\bibitem[Devlin et~al.(2019)Devlin, Chang, Lee, and Toutanova]{devlin_bert_2019}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock {BERT}: {Pre}-training of {Deep} {Bidirectional} {Transformers} for {Language} {Understanding}.
\newblock Technical Report arXiv:1810.04805, arXiv, May 2019.
\newblock URL \url{http://arxiv.org/abs/1810.04805}.
\newblock arXiv:1810.04805 [cs] type: article.
\bibitem[Anwyl-Irvine et~al.(2020)Anwyl-Irvine, Massonnié, Flitton, Kirkham, and Evershed]{anwyl-irvine_gorilla_2020}
Alexander~L. Anwyl-Irvine, Jessica Massonnié, Adam Flitton, Natasha Kirkham, and Jo~K. Evershed.
\newblock Gorilla in our midst: {An} online behavioral experiment builder.
\newblock \emph{Behavior Research Methods}, 52\penalty0 (1):\penalty0 388--407, 2020.
\newblock ISSN 1554-3528.
\newblock \doi{10.3758/s13428-019-01237-x}.
\newblock Place: Germany Publisher: Springer.
\bibitem[Harris et~al.(2020)Harris, Millman, van~der Walt, Gommers, Virtanen, Cournapeau, Wieser, Taylor, Berg, Smith, Kern, Picus, Hoyer, van Kerkwijk, Brett, Haldane, del Río, Wiebe, Peterson, Gérard-Marchant, Sheppard, Reddy, Weckesser, Abbasi, Gohlke, and Oliphant]{harris_array_2020}
Charles~R. Harris, K.~Jarrod Millman, Stéfan~J. van~der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel~J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten~H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime~Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis~E. Oliphant.
\newblock Array programming with {NumPy}.
\newblock \emph{Nature}, 585\penalty0 (7825):\penalty0 357--362, September 2020.
\newblock ISSN 1476-4687.
\newblock \doi{10.1038/s41586-020-2649-2}.
\newblock URL \url{https://www.nature.com/articles/s41586-020-2649-2}.
\newblock Number: 7825 Publisher: Nature Publishing Group.
\bibitem[Virtanen et~al.(2020)Virtanen, Gommers, Oliphant, Haberland, Reddy, Cournapeau, Burovski, Peterson, Weckesser, Bright, van~der Walt, Brett, Wilson, Millman, Mayorov, Nelson, Jones, Kern, Larson, Carey, Polat, Feng, Moore, VanderPlas, Laxalde, Perktold, Cimrman, Henriksen, Quintero, Harris, Archibald, Ribeiro, Pedregosa, and van Mulbregt]{virtanen_scipy_2020}
Pauli Virtanen, Ralf Gommers, Travis~E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan~J. van~der Walt, Matthew Brett, Joshua Wilson, K.~Jarrod Millman, Nikolay Mayorov, Andrew R.~J. Nelson, Eric Jones, Robert Kern, Eric Larson, C.~J. Carey, İlhan Polat, Yu~Feng, Eric~W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E.~A. Quintero, Charles~R. Harris, Anne~M. Archibald, Antônio~H. Ribeiro, Fabian Pedregosa, and Paul van Mulbregt.
\newblock {SciPy} 1.0: fundamental algorithms for scientific computing in {Python}.
\newblock \emph{Nature Methods}, 17\penalty0 (3):\penalty0 261--272, March 2020.
\newblock ISSN 1548-7105.
\newblock \doi{10.1038/s41592-019-0686-2}.
\newblock URL \url{https://www.nature.com/articles/s41592-019-0686-2}.
\newblock Number: 3 Publisher: Nature Publishing Group.
\bibitem[noa({\natexlab{c}})]{noauthor_pandas_nodate}
pandas - {Python} {Data} {Analysis} {Library}, {\natexlab{c}}.
\newblock URL \url{https://pandas.pydata.org/}.
\bibitem[Paszke et~al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai, and Chintala]{paszke_pytorch_2019}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu~Fang, Junjie Bai, and Soumith Chintala.
\newblock {PyTorch}: {An} {Imperative} {Style}, {High}-{Performance} {Deep} {Learning} {Library}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}}, volume~32. Curran Associates, Inc., 2019.
\newblock URL \url{https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html}.
\bibitem[noa({\natexlab{d}})]{noauthor_openaiclip-vit-base-patch32_nodate}
openai/clip-vit-base-patch32 · {Hugging} {Face}, {\natexlab{d}}.
\newblock URL \url{https://huggingface.co/openai/clip-vit-base-patch32}.
\bibitem[Hu and Levy(2023)]{hu2023prompting}
Jennifer Hu and Roger Levy.
\newblock Prompting is not a substitute for probability measurements in large language models.
\newblock \emph{Unknown Journal}, May 2023.
\newblock URL \url{Unknown URL}.
\newblock Submitted on 22 May 2023 (v1), last revised 23 Oct 2023 (this version, v2).
\bibitem[Dahan et~al.(2007)Dahan, Tanenhaus, and Salverda]{dahan_chapter_2007}
Delphine Dahan, Michael~K. Tanenhaus, and Anne~Pier Salverda.
\newblock Chapter 21 - {The} influence of visual processing on phonetically driven saccades in the “visual world” paradigm.
\newblock In Roger P.~G. Van~Gompel, Martin~H. Fischer, Wayne~S. Murray, and Robin~L. Hill, editors, \emph{Eye {Movements}}, pages 471--486. Elsevier, Oxford, January 2007.
\newblock ISBN 978-0-08-044980-7.
\newblock \doi{10.1016/B978-008044980-7/50023-9}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/B9780080449807500239}.
\bibitem[Huettig and Altmann(2005)]{huettig_word_2005}
Falk Huettig and Gerry T.~M. Altmann.
\newblock Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm.
\newblock \emph{Cognition}, 96\penalty0 (1):\penalty0 B23--B32, May 2005.
\newblock ISSN 0010-0277.
\newblock \doi{10.1016/j.cognition.2004.10.003}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0010027704001866}.
\end{thebibliography}