09CCS: Information systems Evaluation of retrieval resultsCCS: Information systems Search interfacesCCS: Information systems Recommender systemsCCS: Human-centered computing User models

Jingwei Kang Affiliation: University of Amsterdam , Amsterdam , The Netherlands email: j.kang@uva.nl OrcID: 0009-0003-9283-4060

2026; © cc

Abstract.

Carousel interfaces are widely used in modern recommendation systems. Unlike traditional interfaces that present a single ranked list, carousels simultaneously present several ranked lists to the user, as horizontally swipeable rows stacked on top of each other. In this design, the rankings are closely tied to the two-dimensional layout. Consequently, user behavior is shaped not only by item preference, but also by row organization, viewport constraints, and item context. This tight coupling between ranking and presentation complicates the interpretation of user feedback, introducing new challenges for recommendation evaluation.

My PhD research aims to address these challenges by rethinking how carousel clicks are modeled and how carousel recommendation policies can be evaluated from logged interaction data. So far, I have studied how users interact with carousel interfaces and developed a click model design framework that prioritizes mathematical relationships between observed variables over latent behavioral assumptions. Building on these results, my ongoing work includes a project using discrete choice models to represent clicks as choices, alongside a project that develops carousel-specific offline metrics. As a next step, I plan to develop off-policy evaluation methods that estimate the performance of recommendation policies from logged interactions. Taken together, the expected contribution of my thesis is a connected body of work that links carousel click modeling with offline and off-policy evaluation, so that carousel recommendation policies can be improved more reliably.

Keywords:

Carousel Interfaces, Click Models, Offline Evaluation, Off-Policy Evaluation

^†^†cc-license: by

1. Introduction

In recent years, with the rise of streaming services such as Netflix and Spotify, the way recommendations are presented has changed significantly. Recommendations are no longer presented only as a single ranked list arranged in a particular layout, such as a search-results page (Chapelle and Chang 2011; Qin et al. 2010; Dato et al. 2016), a vertical feed (Sun et al. 2023; Gao et al. 2022), or a grid (Xie et al. 2017; Xie et al. 2019; Chen et al. 2025). Instead, carousel interfaces simultaneously present multiple horizontally swipeable ranked lists to the user in a vertical arrangement, as shown in Figure 1. Each list has a title or topic that describes its content, such as a specific type of movie or music, or a personalized category like Made for You or Recently Played. These interfaces have become increasingly popular for several reasons, in particular, their ability to support diverse user needs by presenting multiple collections of recommendations.

Offline and off-policy evaluation are key tools for developing recommendation systems from logged interaction data (Cañamares et al. 2020; Castells and Moffat 2022; Zangerle and Bauer 2022). Although these methods usually treat clicks as rewards or indicators of user preference, in practice, click data is heavily biased by the user interface, the logging policy, and the resulting item exposure. As a result, offline and off-policy evaluation depend on a precise interpretation of what the observed feedback actually means. In the carousel recommendation, this is especially difficult. Because multiple ranked lists are displayed simultaneously, the system presents a structured two-dimensional page rather than a single list, shaping how users browse, compare, and choose. Consequently, whether an item is clicked depends not only on user preferences, but also on its row assignment, its position within the row, and its surrounding context.

A common assumption in click modeling is that clicks can be decomposed into examination and relevance: users first examine an item, and conditional on examination, a click reflects the item’s relevance or attractiveness (Craswell et al. 2008a; Chapelle and Zhang 2009a). My empirical work challenges this assumption in carousel interfaces. Using eye-tracking data, I find that the probability of a click, conditional on examination, remains position-dependent. This means that post-examination clicks cannot be treated as a position-invariant relevance signal. Position and layout affect not only whether an item is examined, but also how, once examined, it is interpreted, compared with surrounding items, and ultimately clicked.

Together, these challenges motivate the central focus of my thesis: how carousel recommendation should be modeled and evaluated when the standard assumptions behind click modeling and evaluation framework no longer hold.

Figure 1. Example of movie recommendations by Netflix presented in a carousel interface.

2. Related Work

2.1. User behavior analysis

A large body of user behavior analysis based on eye-tracking research (Granka et al. 2008; Joachims et al. 2005; Granka et al. 2004; Xie et al. 2017) has studied how users examine and interact with search and recommendation results, producing behavioral assumptions that are widely adopted in downstream model and metric design. In single-list web search, this work has revealed a consistent examination pattern, known as the golden triangle, sometimes also referred to as the F-pattern (Granka et al. 2008). It is characterized by a triangle or F-shape that indicates users concentrate heavily on the top-left area of the page. Eye tracking studies have motivated the examination hypothesis, which assumes that a document must be examined before being clicked and that click probabilities are a product of examination and attractiveness probabilities (Richardson et al. 2007). Similar F-shaped patterns have been observed in regular grid layouts, where each row contains a uniform number of items (Zhao et al. 2016; Chierichetti et al. 2011), while irregular grid layouts such as image search instead exhibit a clear middle-position bias, indicating that examination behavior is highly layout-dependent (Xie et al. 2017).

For carousel interfaces, this empirical foundation is largely missing. The only eye-tracking dataset in this setting is RecGaze (de Leon-Martinez et al. 2025), introduced in this thesis (Section 4.1.1); to date it has been analyzed by one external follow-up focused on gaze transitions between carousels and between items within carousels (de Leon-Martinez et al. 2026), and by our own analysis of established behavioral assumptions in carousel interfaces (Kang et al. 2026) (Section 4.1.2). As a result, prior to the work in this thesis, how users actually examine and click in carousel interfaces remained poorly understood.

2.2. Click models

Click models are probabilistic frameworks for interpreting user interactions, typically used to infer the relevance of real clicks (Joachims et al. 2017; Ai et al. 2018; Oosterhuis and de Rijke 2021; Wang et al. 2018; Wang et al. 2016) and to predict future click probabilities to simulate user interactions (Chuklin et al. 2015). The field is dominated by single-list web search, where models are primarily built on Probabilistic Graphical Models (PGMs) that encode hypotheses over latent user behavior, such as examination and attraction. Sequential examination models, including the Cascade Model (CM) (Craswell et al. 2008b), User Browsing Model (UBM) (Dupret and Piwowarski 2008), Dependent Click Model (DCM) (Guo et al. 2009), and Dynamic Bayesian Network Model (DBN) (Chapelle and Zhang 2009b), assume users examine results from top to bottom; others, such as the Temporal Hidden Click Model (THCM) (Xu et al. 2012) and Partially Sequential Click Model (PSCM) (Wang et al. 2015), allow non-sequential behaviors such as revisiting. Beyond single-list, click models have received some attention in grid layouts: Xie et al. 2018 developed the Grid-Based User Browsing Model (GUBM), and Zhuang et al. 2021 use Neural Network (NN) to learn click patterns in grids without restrictive assumptions.

For carousel interfaces, there is only one published click model, Carousel Click Model (CCM) (Rahdari et al. 2024): a cascade-style model in which users browse carousel topics until finding an attractive one and then browse items within it. However, the assumptions underlying this model do not fully account for two empirical findings in this thesis: how users browse carousel interfaces, and the fact that clicks remain position-dependent even conditional on examination. Click modeling for carousel interfaces thus remains a largely open research area.

2.3. Offline and off-policy evaluation

Offline evaluation in search and recommendation has traditionally been based on single ranked lists, where users are often assumed to examine results sequentially from top to bottom. Many list-based metrics can be understood as weighted-gain measures of the form $M=\sum_{i}W_{i}R_{i}$ (Moffat et al. 2013), where $R_{i}$ is the relevance of the $i$ -th result and $W_{i}$ is the metric-specific weight at rank position $i$ . Discounted Cumulative Gain (DCG) (Järvelin and Kekäläinen 2000; Järvelin and Kekäläinen 2002) and Rank-Biased Precision (RBP) (Moffat and Zobel 2008) use position-only weights, while Expected Reciprocal Rank (ERR) (Chapelle et al. 2009) uses cascade-dependent weights, in which $W_{i}$ depends on the relevance of documents ranked above position $i$ . For grid-based interfaces, Xie et al. 2019 first proposes a grid-based metric of RBP-style for the web images search, and then extends it to DCG and ERR. Carousel interfaces require a further generalization of the weight term, since items are arranged in two dimensions and accessed through both horizontal and vertical navigation. For this setting, previous work proposes N2DCG, an NDCG-style metric that incorporates the carousel structure into its position discount (Ferrari Dacrema et al. 2022; Felicioni et al. 2021).

However, when applied to logged feedback rather than ground-truth signals such as ratings, these metrics do not account for inherent exposure and selection biases of the logging policy (Cañamares et al. 2020; Castells and Moffat 2022). An alternative view treats recommendation as an intervention or treatment: a recommender selects actions and observes outcomes only for those actions chosen by the logging policy (Joachims et al. 2021). This leads to Off-Policy Evaluation (OPE), where the goal is to estimate how a target recommendation policy would perform using logged interactions collected from a different logging policy. Standard OPE methods include inverse propensity scoring and its variants (Schnabel et al. 2016; Swaminathan and Joachims 2015), as well as doubly robust estimators (Dudík et al. 2014). In recommendation systems, actions are typically represented as ranked lists or slates, where multiple items and positions are jointly selected. Consequently, OPE becomes substantially harder due to the combinatorial explosion of the action space. This challenge has motivated the development of slate-specific estimators under various user behavior assumptions, such as cascade (Kiyohara et al. 2022; McInerney et al. 2020), independent (Li et al. 2018; Swaminathan et al. 2017; Vlassis et al. 2021), or context-adaptive (Kiyohara et al. 2023); a separate line avoids behavior assumptions through action-space abstraction for large action spaces (Kiyohara et al. 2024; Saito et al. 2023; Saito and Joachims 2022).

Carousel recommendation can be viewed as a multi-slate problem in which each slate itself is structured, but it adds further interface constraints: users do not observe the full layout at once; instead, they navigate through a two-dimensional layout before clicking. The only existing study on carousel evaluation is limited to offline metric design; meanwhile, existing OPE work focuses mainly on single items or ranked lists. This leaves a gap for carousel-specific OPE methods.

3. Research Questions and Thesis Structure

My thesis is organized around two main research questions:

•

RQ1: How should carousel clicks be interpreted and modeled?
This question is addressed in the chapter Rethinking Carousel Clicks. This chapter studies carousel clicks from both an empirical and a modeling perspective. It is divided into two sub-questions:
- –
  
  RQ1.1: How do users interact with carousel interfaces?
- –
  
  RQ1.2: How can click models be designed for carousel interfaces?
•

RQ2: How should carousel recommendation policies be evaluated from logged interaction data?
This question is addressed in the chapter Evaluating Carousel Recommendation. This chapter studies carousel evaluation from two perspectives: metric design and off-policy evaluation. It is divided into two sub-questions:
- –
  
  RQ2.1: How can offline evaluation metrics be designed for carousel layouts?
- –
  
  RQ2.2: How can off-policy evaluation be performed for carousel recommendation policies?

Figure 2. Structure of the thesis. The dashed bounding boxes categorize the papers by their corresponding RQs.

These questions are closely connected: reliable offline and off-policy evaluation requires a clear understanding of how users interact with carousel interfaces and what their clicks indicate. Figure 2 shows the papers that make up this thesis and how they build on one another; we then relate each to the research questions above. RQ1.1 is addressed by the eye-tracking dataset (SIGIR 2025) and the follow-up behavior analysis (SIGIR 2026); RQ1.2 by the click model design framework (ICTIR 2025) and the ongoing choice-based click model; RQ2.1 by the reformulation of N2DCG, currently under review; and RQ2.2 by the planned work on off-policy evaluation for carousel recommendation.

The arrows capture methodological dependencies: the dataset enables the behavior analysis, which informs both the choice model and the metric; the click-model framework also feeds the choice model; and the choice model provides the basis for the planned off-policy evaluation. The N2DCG metric and the off-policy evaluation work are complementary components of the evaluation chapter: the former studies how to define carousel-aware offline metrics, while the latter studies how to estimate target-policy value from logged interactions.

4. Research Progress

4.1. Chapter: Rethinking Carousel Clicks

The first chapter of this thesis addresses RQ1, which asks how carousel clicks should be interpreted and modeled. The chapter consists of four papers: two empirical works addressing RQ1.1, and two modeling works addressing RQ1.2. The first two collect and analyze eye-tracking data on carousel interfaces to establish how users actually interact with them; the latter two translate these empirical findings to guide the design of click models.

4.1.1. An eye-tracking dataset for carousel interfaces

To address RQ1.1, the first step is to collect the empirical data needed to study user behaviors directly. To this end, we constructed RecGaze (de Leon-Martinez et al. 2025), published at SIGIR 2025, the first dataset for carousel interfaces that combines eye-tracking data with clicks, cursor movements, and post-selection explanations. The dataset was collected through a user study in which 87 participants completed three movie selection tasks, free browsing, semi-free browsing, and direct search, on 40 carousel screens, resulting in 3 ,477 valid logged interactions. As shown in Figure 3, each screen displayed 10 genre-based carousels of 15 movies, presented in an interface modeled after streaming services such as Netflix. An initial aggregate analysis of the gaze data suggested an F-shaped examination pattern on the first page, while deviations on subsequent pages motivated a more systematic analysis of whether this and other established behavioral assumptions hold in the carousel setting.

Netflix carousel interface — Figure 3. Design of a screen in the RecGaze eye-tracking study. The blue circle represents a user’s current gaze point.

4.1.2. Empirical analysis of carousel browsing behavior

Building on RecGaze, this paper (Kang et al. 2026), accepted at SIGIR 2026, systematically tests four behavioral assumptions that underlie existing carousel click models and evaluation metrics. In fact, the data refutes all four. First, the classical F-pattern does not hold even on the first page, which instead exhibits a dual-focus structure with attention peaks in both top corners; after a horizontal swipe, the second and third pages exhibit a mirrored F-pattern rather than a continuation, so no global F-pattern spans the interface (Figure 4). Second, examination conditioned on a click does not follow the assumed hierarchical, cascade-like pattern but instead forms a distinctive L-shape (Figure 5). Third, conditional click probability given examination vary significantly across rows within the same column, despite the random vertical ordering of carousels in the study, refuting the examination hypothesis itself (Figure 6). Fourth, contrary to the assumption that users navigate carousels through their titles, the data shows that users typically examine the items first, and often without examining the corresponding title at all. Taken together, these findings indicate that the click models and evaluation metrics built on these assumptions require reconsideration.

Refer to caption — Figure 4. Empirical examination frequency heatmap across positions and pages. The number in each cell represents the frequency at that position; the white box marks the initially visible items and titles. The first page exhibits a dual-focus F-shaped pattern, while pages 2 and 3 exhibit mirrored F-shaped patterns; no global F-pattern spans the interface.

4.1.3. A theory-based framework for click model design

A natural question follows from the empirical findings above: if standard behavioral assumptions do not hold in carousel interfaces, should they serve as the foundation for click model design at all? This paper (Kang et al. 2025), published at ICTIR 2025, reconsiders the conceptual foundations of click model design itself, grounding them in mathematical rather than behavioral assumptions. We argue that the mathematical relationships captured between observed variables should be the first concern in click model design, rather than the latent variables or user behaviors they implicitly assume. Building on this principle, we identify three key design choices that every click model makes, explicitly or implicitly: global dependencies, sequentiality, and factorization. We then use these choices to construct a taxonomy that, for the first time, places single-list, grid, and carousel click models within a single comparative framework. To demonstrate the value of the design framework, we derive a new carousel click model from these three choices, providing a starting point for the modeling work that follows in this chapter.

4.1.4. A choice-based click model for carousel interfaces

This ongoing work follows the click model design framework mentioned above and models clicks directly as a discrete choice over the items a user has actually considered. The model captures click behavior as a two-stage process. First, a sequential browsing process generates a consideration set, which is initialized with the items visible at page load and expands through specific actions: swiping right within a row, scrolling down between rows, or stopping. These actions are bound by the physical constraints of the interface, preventing any skipping of pages within a row or skipping of rows altogether. Finally, a discrete choice over this generated set determines which item is clicked. The click probability for item $i$ is

(1)

P(\text{click }i\mid x)\;=\;\sum_{k:\,i\in C_{k}}P(C_{k}\mid x)\,\cdot\,\frac{\exp(u_{i})}{\sum_{j\in C_{k}}\exp(u_{j})},

where $x$ denotes the context, $u_{i}\in\mathbb{R}$ is the utility of item $i$ , $P(C_{k}\mid x)$ is the consideration-set distribution induced by the navigation process, and the fractional term is the within-set Multinomial Logit (MNL) choice probability (McFadden 1974; Train 2009). The model captures two critical properties lacking in prior click models. First, an item’s probability of being chosen depends on the other items it is jointly considered with, which examination-based models cannot represent. Second, it explicitly models navigation in 2D carousel layouts under partial visibility, an aspect that standard 1D click models fail to address. The methodological challenges, in this work, are parameterizing the consideration set construction process and the choice over that set, and estimating them from logged data in which only the final click is observed.

4.2. Chapter: Evaluating Carousel Recommendation

The second chapter of this thesis addresses RQ2, which asks how carousel recommendation policies should be evaluated from logged interaction data. The chapter consists of an offline metric for carousel layouts (RQ2.1), currently under review, and planned work on off-policy evaluation (RQ2.2).

4.2.1. A behavior-grounded reformulation of N2DCG

To address RQ2.1, we revisit the only existing carousel evaluation metric, N2DCG (Felicioni et al. 2021; Ferrari Dacrema et al. 2022), adapted from NDCG to two-dimensional carousel layouts. This paper, currently under review, identifies two substantial limitations of the original formulation. First, its ideal ranking, which is used for normalization, violates the categorical constraints of the carousel interface. Second, its discount function inherits assumptions from single-list web search that our empirical analysis in Section 4.1.2 shows do not hold in carousel interfaces. We propose a reformulation that addresses both limitations. Empirically, the reformulated metric tracks user behavior on real-world eye-tracking data more faithfully than the original N2DCG. Furthermore, it better predicts the outcomes of simulated carousel layout comparisons based on empirical examination frequencies.

4.2.2. Off-policy evaluation for carousel recommendation

To address RQ2.2, the planned work aims to develop OPE methods for carousel recommendation policies, filling the gap identified in the related work above. Standard slate OPE estimators assume a single, fully observable ranking: all items in the slate are simultaneously on display at presentation time, and the modeled structure concerns only the order in which the user examines them. Carousel interfaces do not fit this setting: the layout contains multiple rankings stacked vertically, and at any moment only a portion of each visible ranking is on screen, with further items accessed through horizontal swiping and further rankings through vertical scrolling. The planned work therefore explores how existing slate OPE methods can be adapted to this multi-ranking, partially visible setting, along two possible routes. One route reduces the two-dimensional layout to a one-dimensional slate: building on the discrete choice model of Section 4.1.4, we can estimate the most likely consideration set and collapse the layout into a single effective ranking, to which standard slate estimators directly apply. The other route retains the two-dimensional structure but organizes it hierarchically, representing the layout as a slate of sub-slates, so that cross-ranking and within-ranking behavior are modeled as a two-stage process.

5. Conclusion

This thesis investigates how carousel recommendations should be modeled and evaluated. This exploration is motivated by the fact that existing click models and evaluation methods for single-list web search fail to transfer to carousel layouts. Empirically, work using the RecGaze eye-tracking dataset and follow-up analysis has established a behavioral foundation for user interaction with carousels. Theoretically, this research introduces a click model design framework that prioritizes observable variables over latent behaviors. Building on these foundations, my ongoing and planned work develops a choice-based click model for carousels and pursues two complementary directions for off-policy evaluation, alongside a behavior-grounded reformulation of N2DCG that is currently under review.

Together, these contributions aim to provide the modeling and evaluation tools that are currently lacking in the carousel recommendation. At this stage of the thesis, feedback from the doctoral symposium on the design of the two planned off-policy evaluation routes and on the broader evaluation methodology for carousel recommendation would be highly valuable.

Acknowledgements.

I would like to thank my supervisors, Maarten de Rijke and Harrie Oosterhuis, for their guidance, feedback, and support. I am also grateful to my collaborator, Santiago de Leon-Martinez, for designing the eye-tracking study. My PhD is supported by the Dutch Research Council (NWO) under grant KICH3.LTP.20.006. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

References

(1)
Ai et al. (2018) Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft. 2018. Unbiased Learning to Rank with Unbiased Propensity Estimation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 385–394. doi:10.1145/3209978.3209986
Cañamares et al. (2020) Rocío Cañamares, Pablo Castells, and Alistair Moffat. 2020. Offline Evaluation Options for Recommender Systems. Information Retrieval Journal 23, 4 (2020), 387–410. doi:10.1007/s10791-020-09371-3
Castells and Moffat (2022) Pablo Castells and Alistair Moffat. 2022. Offline Recommender System Evaluation: Challenges and New Directions. AI Magazine 43, 2 (2022), 225–238. doi:10.1002/aaai.12051
Chapelle and Chang (2011) Olivier Chapelle and Yi Chang. 2011. Yahoo! Learning to Rank Challenge Overview. In Proceedings of the Learning to Rank Challenge (Proceedings of Machine Learning Research, Vol. 14), Olivier Chapelle, Yi Chang, and Tie-Yan Liu (Eds.). PMLR, Haifa, Israel, 1–24. https://proceedings.mlr.press/v14/chapelle11a.html
Chapelle et al. (2009) Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (Hong Kong, China) (CIKM ’09). Association for Computing Machinery, New York, NY, USA, 621–630. doi:10.1145/1645953.1646033
Chapelle and Zhang (2009a) Olivier Chapelle and Ya Zhang. 2009a. A Dynamic Bayesian Network Click Model for Web Search Ranking. In Proceedings of the 18th International Conference on World Wide Web. 1–10. doi:10.1145/1526709.1526711
Chapelle and Zhang (2009b) Olivier Chapelle and Ya Zhang. 2009b. A Dynamic Bayesian Network Click Model for Web Search Ranking. In Proceedings of the 18th International Conference on World Wide Web (Madrid, Spain) (WWW ’09). Association for Computing Machinery, New York, NY, USA, 1–10. doi:10.1145/1526709.1526711
Chen et al. (2025) Jia Chen, Qian Dong, Haitao Li, Xiaohui He, Yan Gao, Shaosheng Cao, Yi Wu, Ping Yang, Chen Xu, Yao Hu, Qingyao Ai, and Yiqun Liu. 2025. Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (Padua, Italy) (SIGIR ’25). Association for Computing Machinery, New York, NY, USA, 3670–3680. doi:10.1145/3726302.3730279
Chierichetti et al. (2011) Flavio Chierichetti, Ravi Kumar, and Prabhakar Raghavan. 2011. Optimizing Two-dimensional Search Results Presentation. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (Hong Kong, China) (WSDM ’11). Association for Computing Machinery, New York, NY, USA, 257–266. doi:10.1145/1935826.1935873
Chuklin et al. (2015) Aleksandr Chuklin, Ilya Markov, and Maarten Rijke. 2015. Click Models for Web Search. Springer Cham. doi:10.1007/978-3-031-02294-4
Craswell et al. (2008a) Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008a. An Experimental Comparison of Click Position-Bias Models. In Proceedings of the 2008 International Conference on Web Search and Data Mining. 87–94. doi:10.1145/1341531.1341545
Craswell et al. (2008b) Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008b. An Experimental Comparison of Click Position-bias Models. In Proceedings of the 2008 International Conference on Web Search and Data Mining (Palo Alto, California, USA) (WSDM ’08). Association for Computing Machinery, New York, NY, USA, 87–94. doi:10.1145/1341531.1341545
Dato et al. (2016) Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast Ranking with Additive Ensembles of Oblivious and Non-Oblivious Regression Trees. ACM Trans. Inf. Syst. 35, 2, Article 15 (Dec. 2016), 31 pages. doi:10.1145/2987380
de Leon-Martinez et al. (2025) Santiago de Leon-Martinez, Jingwei Kang, Robert Moro, Maarten de Rijke, Branislav Kveton, Harrie Oosterhuis, and Maria Bielikova. 2025. RecGaze: The First Eye Tracking and User Interaction Dataset for Carousel Interfaces. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (Padua, Italy) (SIGIR ’25). Association for Computing Machinery, New York, NY, USA, 3702–3711. doi:10.1145/3726302.3730301
de Leon-Martinez et al. (2026) Santiago de Leon-Martinez, Robert Moro, Branislav Kveton, and Maria Bielikova. 2026. Riding the Carousel: The First Extensive Eye Tracking Analysis of Browsing Behavior in Carousel Recommenders. In Proceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). Association for Computing Machinery, New York, NY, USA, 2120–2130. doi:10.1145/3742413.3789166
Dudík et al. (2014) Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly Robust Policy Evaluation and Optimization. Statist. Sci. 29, 4 (2014), 485 – 511. doi:10.1214/14-STS500
Dupret and Piwowarski (2008) Georges E. Dupret and Benjamin Piwowarski. 2008. A User Browsing Model to Predict Search Engine Click Data from Past Observations. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Singapore, Singapore) (SIGIR ’08). Association for Computing Machinery, New York, NY, USA, 331–338. doi:10.1145/1390334.1390392
Felicioni et al. (2021) Nicolò Felicioni, Maurizio Ferrari Dacrema, and Paolo Cremonesi. 2021. A Methodology for the Offline Evaluation of Recommender Systems in a User Interface with Multiple Carousels. In Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (Utrecht, Netherlands) (UMAP ’21). Association for Computing Machinery, New York, NY, USA, 10–15. doi:10.1145/3450614.3461680
Ferrari Dacrema et al. (2022) Maurizio Ferrari Dacrema, Nicolò Felicioni, and Paolo Cremonesi. 2022. Offline Evaluation of Recommender Systems in a User Interface With Multiple Carousels. Frontiers in Big Data 5 (2022). doi:10.3389/fdata.2022.910030
Gao et al. (2022) Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (Atlanta, GA, USA) (CIKM ’22). 3953–3957. doi:10.1145/3511808.3557624
Granka et al. (2008) Laura Granka, Matthew Feusner, and Lori Lorigo. 2008. Eye Monitoring in Online Search. Springer Berlin Heidelberg, Berlin, Heidelberg, 347–372. doi:10.1007/978-3-540-75412-1_16
Granka et al. (2004) Laura A. Granka, Thorsten Joachims, and Geri Gay. 2004. Eye-tracking Analysis of User Behavior in WWW Search (SIGIR ’04). Association for Computing Machinery, New York, NY, USA, 478–479. doi:10.1145/1008992.1009079
Guo et al. (2009) Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient Multiple-click Models in Web Search. In Proceedings of the Second ACM International Conference on Web Search and Data Mining (Barcelona, Spain) (WSDM ’09). Association for Computing Machinery, New York, NY, USA, 124–131. doi:10.1145/1498759.1498818
Järvelin and Kekäläinen (2000) Kalervo Järvelin and Jaana Kekäläinen. 2000. IR Evaluation Methods for Retrieving Highly Relevant Documents. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Athens, Greece) (SIGIR ’00). Association for Computing Machinery, New York, NY, USA, 41–48. doi:10.1145/345508.345545
Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-based Evaluation of IR Techniques. ACM Trans. Inf. Syst. 20, 4 (Oct. 2002), 422–446. doi:10.1145/582415.582418
Joachims et al. (2005) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately Interpreting Clickthrough Data as Implicit Feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Salvador, Brazil) (SIGIR ’05). Association for Computing Machinery, New York, NY, USA, 154–161. doi:10.1145/1076034.1076063
Joachims et al. (2021) Thorsten Joachims, Ben London, Yi Su, Adith Swaminathan, and Lequn Wang. 2021. Recommendations as treatments. AI Mag. 42, 3 (Sept. 2021), 19–30. doi:10.1609/aimag.v42i3.18141
Joachims et al. (2017) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (Cambridge, United Kingdom) (WSDM ’17). Association for Computing Machinery, New York, NY, USA, 781–789. doi:10.1145/3018661.3018699
Kang et al. (2025) Jingwei Kang, Maarten de Rijke, Santiago de Leon-Martinez, and Harrie Oosterhuis. 2025. Rethinking Click Models in Light of Carousel Interfaces: Theory-Based Categorization and Design of Click Models. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) (Padua, Italy) (ICTIR ’25). Association for Computing Machinery, New York, NY, USA, 44–55. doi:10.1145/3731120.3744585
Kang et al. (2026) Jingwei Kang, Maarten de Rijke, and Harrie Oosterhuis. 2026. Following the Eye-Tracking Evidence: Established Web-Search Assumptions Fail in Carousel Interfaces. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (Australia) (SIGIR ’26). Association for Computing Machinery, New York, NY, USA, 734–744. doi:10.1145/3805712.3809657
Kiyohara et al. (2024) Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. 2024. Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction. In Proceedings of the ACM Web Conference 2024 (Singapore, Singapore) (WWW ’24). Association for Computing Machinery, New York, NY, USA, 3150–3161. doi:10.1145/3589334.3645343
Kiyohara et al. (2022) Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. 2022. Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (Virtual Event, AZ, USA) (WSDM ’22). Association for Computing Machinery, New York, NY, USA, 487–497. doi:10.1145/3488560.3498380
Kiyohara et al. (2023) Haruka Kiyohara, Masatoshi Uehara, Yusuke Narita, Nobuyuki Shimizu, Yasuo Yamamoto, and Yuta Saito. 2023. Off-Policy Evaluation of Ranking Policies under Diverse User Behavior. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Long Beach, CA, USA) (KDD ’23). Association for Computing Machinery, New York, NY, USA, 1154–1163. doi:10.1145/3580305.3599447
Li et al. (2018) Shuai Li, Yasin Abbasi-Yadkori, Branislav Kveton, S. Muthukrishnan, Vishwa Vinay, and Zheng Wen. 2018. Offline Evaluation of Ranking Policies with Click Models. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 1685–1694. doi:10.1145/3219819.3220028
McFadden (1974) Daniel McFadden. 1974. Conditional Logit Analysis of Qualitative Choice Behavior. In Frontiers in Econometrics, Paul Zarembka (Ed.). Academic Press, New York, 105–142.
McInerney et al. (2020) James McInerney, Brian Brost, Praveen Chandar, Rishabh Mehrotra, and Benjamin Carterette. 2020. Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, NY, USA, 1779–1788. doi:10.1145/3394486.3403229
Moffat et al. (2013) Alistair Moffat, Paul Thomas, and Falk Scholer. 2013. Users versus models: what observation tells us about effectiveness metrics. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management (San Francisco, California, USA) (CIKM ’13). Association for Computing Machinery, New York, NY, USA, 659–668. doi:10.1145/2505515.2507665
Moffat and Zobel (2008) Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27, 1, Article 2 (Dec. 2008), 27 pages. doi:10.1145/1416950.1416952
Oosterhuis and de Rijke (2021) Harrie Oosterhuis and Maarten de Rijke. 2021. Unifying Online and Counterfactual Learning to Rank: A Novel Counterfactual Estimator that Effectively Utilizes Online Interventions. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (Virtual Event, Israel) (WSDM ’21). Association for Computing Machinery, New York, NY, USA, 463–471. doi:10.1145/3437963.3441794
Qin et al. (2010) Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. 2010. LETOR: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval 13, 4 (2010), 346–374. doi:10.1007/s10791-009-9123-y
Rahdari et al. (2024) Behnam Rahdari, Peter Brusilovsky, and Branislav Kveton. 2024. Towards Simulation-Based Evaluation of Recommender Systems with Carousel Interfaces. ACM Trans. Recomm. Syst. 2, 1, Article 9 (March 2024), 25 pages. doi:10.1145/3643709
Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting Clicks: Estimating the Click-through Rate for New Ads. In Proceedings of the 16th International Conference on World Wide Web (Banff, Alberta, Canada) (WWW ’07). Association for Computing Machinery, New York, NY, USA, 521–530. doi:10.1145/1242572.1242643
Saito and Joachims (2022) Yuta Saito and Thorsten Joachims. 2022. Off-Policy Evaluation for Large Action Spaces via Embeddings. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 19089–19122. https://proceedings.mlr.press/v162/saito22a.html
Saito et al. (2023) Yuta Saito, Qingyang Ren, and Thorsten Joachims. 2023. Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 29734–29759. https://proceedings.mlr.press/v202/saito23b.html
Schnabel et al. (2016) Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1670–1679. https://proceedings.mlr.press/v48/schnabel16.html
Sun et al. (2023) Zhongxiang Sun, Zihua Si, Xiaoxue Zang, Dewei Leng, Yanan Niu, Yang Song, Xiao Zhang, and Jun Xu. 2023. KuaiSAR: A Unified Search And Recommendation Dataset. (2023). doi:10.1145/3583780.3615123
Swaminathan and Joachims (2015) Adith Swaminathan and Thorsten Joachims. 2015. The Self-Normalized Estimator for Counterfactual Learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2015/file/39027dfad5138c9ca0c474d71db915c3-Paper.pdf
Swaminathan et al. (2017) Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. 2017. Off-policy evaluation for slate recommendation. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/5352696a9ca3397beb79f116f3a33991-Paper.pdf
Train (2009) Kenneth E. Train. 2009. Discrete Choice Methods with Simulation (2 ed.). Cambridge University Press, Cambridge.
Vlassis et al. (2021) Nikos Vlassis, Ashok Chandrashekar, Fernando Amat, and Nathan Kallus. 2021. Control Variates for Slate Off-Policy Evaluation. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 3667–3679. https://proceedings.neurips.cc/paper_files/paper/2021/file/1e0b802d5c0e1e8434a771ba7ff2c301-Paper.pdf
Wang et al. (2015) Chao Wang, Yiqun Liu, Meng Wang, Ke Zhou, Jian-yun Nie, and Shaoping Ma. 2015. Incorporating Non-sequential Behavior into Click Models. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (Santiago, Chile) (SIGIR ’15). Association for Computing Machinery, New York, NY, USA, 283–292. doi:10.1145/2766462.2767712
Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to Rank with Selection Bias in Personal Search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR ’16). Association for Computing Machinery, New York, NY, USA, 115–124. doi:10.1145/2911451.2911537
Wang et al. (2018) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position Bias Estimation for Unbiased Learning to Rank in Personal Search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM ’18). Association for Computing Machinery, New York, NY, USA, 610–618. doi:10.1145/3159652.3159732
Xie et al. (2017) Xiaohui Xie, Yiqun Liu, Xiaochuan Wang, Meng Wang, Zhijing Wu, Yingying Wu, Min Zhang, and Shaoping Ma. 2017. Investigating Examination Behavior of Image Search Users. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR ’17). Association for Computing Machinery, New York, NY, USA, 275–284. doi:10.1145/3077136.3080799
Xie et al. (2018) Xiaohui Xie, Jiaxin Mao, Maarten de Rijke, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2018. Constructing an Interaction Behavior Model for Web Image Search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 425–434. doi:10.1145/3209978.3209990
Xie et al. (2019) Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Maarten de Rijke, Yunqiu Shao, Zixin Ye, Min Zhang, and Shaoping Ma. 2019. Grid-based Evaluation Metrics for Web Image Search. In The World Wide Web Conference (San Francisco, CA, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 2103–2114. doi:10.1145/3308558.3313514
Xu et al. (2012) Danqing Xu, Yiqun Liu, Min Zhang, Shaoping Ma, and Liyun Ru. 2012. Incorporating Revisiting Behaviors into Click Models. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (Seattle, Washington, USA) (WSDM ’12). Association for Computing Machinery, New York, NY, USA, 303–312. doi:10.1145/2124295.2124334
Zangerle and Bauer (2022) Eva Zangerle and Christine Bauer. 2022. Evaluating Recommender Systems: Survey and Framework. ACM Comput. Surv. 55, 8, Article 170 (Dec. 2022), 38 pages. doi:10.1145/3556536
Zhao et al. (2016) Qian Zhao, Shuo Chang, F. Maxwell Harper, and Joseph A. Konstan. 2016. Gaze Prediction for Recommender Systems. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for Computing Machinery, New York, NY, USA, 131–138. doi:10.1145/2959100.2959150
Zhuang et al. (2021) Honglei Zhuang, Zhen Qin, Xuanhui Wang, Michael Bendersky, Xinyu Qian, Po Hu, and Dan Chary Chen. 2021. Cross-Positional Attention for Debiasing Clicks. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 788–797. doi:10.1145/3442381.3450098