Searching Personal Collections
Extracted source text · arXiv 2412.12330
% This is a template root file for ACM series books.
% Please see read-me.txt for details.
\documentclass[times]{acm-book}%%%
\usepackage{amsmath} % Package for AMS math extensions.
\usepackage{graphicx} % Package for importing EPS files.
\usepackage{hyperref}
\title{Advanced Topics for Information Retrieval}
\begin{document}
%\frontmatter
%\include{front} % Front matter.
\mainmatter
\include{Chapter-SearchingPersonalCollections}
\backmatter
\bibliography{acm-book}
\include{bio} % Author(s) biographies.
\end{document}
\newcommand\blfootnote[1]{
\begingroup
\renewcommand\thefootnote{}\footnote{#1}
\addtocounter{footnote}{-1}
\endgroup
}
\setcounter{chapter}{13}
\chapter{Searching Personal Collections}
% Choosing alphabetical order
\chapterauthor{Michael Bendersky, Donald Metzler, Marc Najork, and Xuanhui Wang}
This chapter discusses search and discovery in personal collections. A personal collection is a set of digital assets associated with a particular individual (often referred to as the ``user'') or a limited set of individuals. Examples of digital assets include emails, notes, photos, files, bookmarks and the like. In the remainder of this chapter, we use the terms asset, item, and document interchangeably.
\blfootnote{This article appeared as Chapter 14 in Omar Alonzo and Ricardo Baeza-Yates (editors), Information Retrieval: Advanced Topics and Techniques, ACM Press, 2025, available at https://doi.org/10.1145/3674127.3674142}
During the first two decades of the personal computing age, these assets were typically stored on a computer controlled by the user or on a server controlled by the user's organization (say, a department's file server). With the advent of the Web as a computing platform, personal collections have increasingly moved from the devices controlled by individuals or their organizations into services operated by cloud service providers. For email collections, such service providers include Outlook.com (originally Hotmail, launched in 1996), Yahoo! Mail (originally RocketMail, launched in 1996), AOL Mail (originally NetMail, launched in 1997), Mail.ru (launched in 1998), and Gmail (launched in 2004). For file collections, cloud-based service providers include Box (founded in 2005), Dropbox (founded in 2007), Microsoft OneDrive (originally SkyDrive, launched in 2007), and Google Drive (launched in 2012).
Personal collections differ from public collections (such as the universe of books, the set of all news stories, or the Web) in a number of significant ways:
\begin{itemize}
\item Users are typically aware of most, if not all, of the items that exist in the collection. Thus, the typical retrieval task is refinding known items, as opposed to finding as-of-yet unknown documents that satisfy an information need.
\item Users ``own'' the assets, and may spend effort on organizing them, in order to aid future refinding. Organization strategies include foldering (placing assets into a tree-based hierarchy), tagging (labeling assets with terms from an extensible vocabulary) and annotating (e.g. adding a textual description to a photo or video).
\item Personal collections are typically many orders of magnitude smaller than public collections. This shifts the trade-offs in a retrieval system. While most large-scale retrieval systems (e.g. web search engines) are optimized for precision, personal search tends to be optimized for recall---specifically, to ensure that when a user is searching for a known asset, that asset is among the retrieved assets.
\item User interactions are sparse. Personal corpora by definition contain assets that are personal to a user, and are typically not shared with other users. This implies that the owner of an asset is the only person interacting with the asset, in contrast to user interactions with public corpora, where many users of a retrieval system may issue similar queries and interact with (for example, click on) the same result documents. Web search engines typically leverage this interaction data both during the retrieval stage (e.g. by expanding a query with terms of related queries with co-clicked results, or by mining query reformulations across the user population) and the ranking stage (e.g. by using result clicks as a relevance signal for future similar queries).
\end{itemize}
These differences distinguish personal from public collections. The remainder of this chapter will examine the various facets in more detail.
\section{Organizing Digital Assets}
\label{sec:organizing-assets}
The advent of the personal computer enabled users to accumulate personal digital assets, and this technological development spawned research into the psychology of {\em personal information management}. \citet{Lansdale1988Psychology} wrote a seminal perspective article on how psychological observations should influence the development of future personal information management systems. Lansdale studied the behavior of office workers and the techniques to manage paper-based information. As observed earlier by~\citet{Malone1983How}, some office workers organize documents into well-structured collections (the ``filers'') while others simply stack them on their desks (the ``pilers''). Contrary to first expectations, diligent filing of information does not always enable successful retrieval because hierarchical classification systems tend to have overlapping and thus somewhat ambiguous categories, making it difficult to locate the folder when attempting to locate a file. As a result, workers may feel that there is an insufficient return on investment to filing new information, and thus simply add to unstructured piles. Lansdale concludes that a successful filing system should be organized by attributes that users do recall and should support retrieval based on multiple attributes. This idea is borne out by tag-based filing systems, see Section~\ref{sec:label-based-org}.
\citet{Nardi+1994Filing} interviewed 15 users on their approaches to filing and finding information on their Macintosh computers, and the problems they encountered in the process. While they found that no two users in the study employed the same naming or foldering conventions, they also found that users tended to group files into three categories: ephemeral, archived and working. Ephemeral files have a short shelf life---their usefulness diminishes rapidly over time, and they will not be used thereafter. Working files are being actively used for a current task. Archived files are not actively used at the moment but have a long shelf-life, and a user may need to refind them months or years after they were created. Nardi et al. observed that users tended to place ephemeral and working files into obvious and visible locations (e.g. the computer's desktop), while they relied on a combination of location-based retrieval (e.g. a well-structured folder hierarchy), attribute search and full-text search when refinding archived information.
\citet{Barreau1995Context} surveyed seven managers (who at the time of the study were the class of office workers most likely to receive and retrieve large numbers of files) on how they organized the information routed to them and retrieved it. Some of the documents curated by the users were stored locally on users' machines while others were stored on a remote file server. In addition to observing similar behaviors as \citet{Nardi+1994Filing} did, Barreau also found that users leveraged file ownership as an attribute in organizing and retrieving information.
\citet{BarreauNardi1995Finding} synthesized the findings of the two previous studies, emphasizing their common observations: users prefered filing strategies that leverage spatio-visual cues (e.g. placing files on the desktop); users avoided elaborate filing schemes; users distinguished between ephemeral, working, and archived files; and finally, users accumulated relatively few archival files.
It is worth remembering that the above studies, conducted more than 25 years ago, examined the behaviors of the first generation of personal computer users. Clearly there is a feedback loop between users and systems: information systems improve to better accommodate their user base, and users' behavior evolves in response.
\citet{WhittakerSidner1996EmailOverload} study the strategies of 20 Lotus Notes users for filing and refinding email messages. Email, originally designed as a communication medium, has evolved into an archival medium as well as a means for task management, that is, people using email to assign work items to others or themselves, and subsequently using email to track task completion. All three uses are asynchronous: replies to incoming emails may not be written immediately, email archival envisions a future need for the archived message, and task management anticipates future task completion. Consequently, users developed different strategies for managing this working information, all leveraging Lotus Notes folder mechanism. Many users found filing emails to be a cognitively challenging task, since creating folders and adding emails to folders requires users to anticipate future information needs. Whittaker and Sidner introduce the notion of a {\em failed folder}, one whose name or purpose is too ambiguous to be useful or one too specific to effectively partition the information space. The study identifies three fundamental filing approaches: some users avoid filing altogether and instead rely on search capabilities to refind needed information; some users are ``frequent filers'' whose inboxes tend to be short and contain mostly new messages; and some users are ``spring cleaners'' who periodically (e.g. every one to three months) clean up their inbox and archive items in folders. Importantly, the study found that ``spring cleaning'' is an ineffective strategy for information management, with over half of the folders of spring cleaners being failed folders.
Two later studies by~\citet{Elsweiler+2008Memory,Elsweiler+2011Difficult} examined the impact of the filing strategies on how much users remember about individual messages. Somewhat surprisingly, they found that non-filers remember more details about their messages than spring cleaners do, who in turn remember more than frequent filers do.
\citet{BoardmanSasse2004Stuff} performed a cross-tool and longitudinal study of personal information management strategies. Specifically, they studied the strategies of 31 participants for organizing their files, email messages, and web bookmarks. The study confirmed observations of earlier studies (for example, users group into filers and pilers). Moreover, they observed that users are more invested into some classes of assets (such as their files and emails) than others (e.g. their bookmarks). Most interestingly, they observed that participation in the study caused introspection, resulting in two of the eight participants changing their filing behavior, both starting to separate working files from archival files.
\section{Labeling Digital Assets}
\label{sec:label-based-org}
As mention in Section~\ref{sec:organizing-assets} above, \citet{Lansdale1988Psychology} observed that it is often difficult to construct classification hierarchies that are non-ambiguous and where every item belongs to exactly one class. The obvious alternative is to allow items to belong to many classes. In this model, assets are labeled with any number of {\em labels} or {\em tags}, and users can issue queries that retrieve all items labeled with a set of tags. Labels are widely used in cloud-based email systems (e.g. Gmail labels), photo repositories (e.g. Flickr tags) and public messaging platforms (e.g. Twitter hashtags). But they are also available in most commercial file systems as a means of file organization, and there is a body of research on them.
\citet{Mogul1984Representing} proposed a file system where all file metadata is expressed as key/value pairs, akin to property lists in Lisp. The paper described how such a system could be implemented and what functionality the file system API would have to provide; it did not explore how this added functionality could enhance the recall of personal information.
The Macintosh File System (MFS), released that same year, introduced the concept of bifurcated files: an MFS file consists of a data fork (containing its content) and a resource fork (containing its metadata). MFS evolved into the Hierarchical File System (HFS), launched the following year, which in turn evolved into HFS+ (launched 1998). HFS+ allowed for each file to have an unlimited number of forks. Beginning with OS X 10.9, released in 2013, HFS+ forks allows users to attach an arbitrary number of custom tags to their files and to surface all files with a given tag through the Finder~\citep{Apple2013WhatsNew}.
The Windows NT File System (NTFS), released in 1993, supports custom properties. In NTFS, a file is a collection of one or more data streams, each data stream consisting of a name (a key) and an associated byte stream (a value). In this model, the file content is the main data stream (keyed by {\tt \$DATA}), and each file property is an ``alternate data stream'' (ADS) with a name (the key) and a byte stream (the value). All file metadata is encoded using ADS.
The Windows 7 operating system (released in 2007) exposed custom properties to users, allowing users to annotate files with custom properties (such as tags); however, the user interface for doing so was nonintuitive. To remedy this shortcoming, \citet{Cutrell+2006Flexible} introduced Phlat, a Windows shell for tagging, browsing, and searching digital assets (namely, files, emails and browsed web pages). Phlat used custom properties to store file tags, and Windows Desktop search to index and retrieve content.
Following Moguls' work, the idea of expressing file metadata as key/value pairs was further explored by~\citet{Gifford+1991Semantic}, who proposed the Semantic File System (SFS). SFS employs a set of user-definable transducers to annotate each file with key/value attributes. Transducers are automatically invoked as files are being created or modified. In addition to making key/value pairs accessible through the file system API, Gifford et al. also introduced the concept of ``virtual directories'': Unix directories (folders) that contain all the files with a particular key/value pair combination. Again, the SFS paper did not explore how virtual directories could be leveraged for retrieval tasks.
The core idea of SFS was further refined by the Nebula file system~\citep{Bowman+1994FileSystem}. All the metadata of a file (e.g. its name, aliases, type, and access control) are expressed as key/value pairs. Nebula introduces the concept of ``view'' (akin to SFS virtual directories) that groups together all files with a given key/value pair.
There are a number of research papers that propose to use tags as a first-class mechanism for organizing file collections. The TagFS file system~\citep{Schenk+2006TagFS,Bloehdorn+2006TagFS} was explicitly designed to explore tagging as the primary mechanism to organize a user's files. \citet{SeltzerMurphy2009hFAD} argue that users by and large do not remember where their files are stored, and they propose the {\em Hierarchical Filesystems are Dead} (hFAD), a filesystem namespace based on searchable tags.
For organizing email messages, labeling was introduced by Pachyderm~\citep{Birrell+al1997Pachyderm} and became popular through the rise of Gmail, where labeling is the \emph{only} mechanism for users to organize their messages. Photo repositories such as FlickR have used tagging both as a way to create a textual vocabulary for non-textual assets and as a way to make these assets discoverable and searchable.
\section{Automatic Classification and Clustering}
As mentioned in Section~\ref{sec:label-based-org}, the SFS~\citep{Gifford+1991Semantic} introduced the concept of transducers---heuristic analyzers that annotate files with labels based on their content. Subsequent research advanced the idea of automatic classification of personal assets.
\citet{Cohen1996Learning} described an application of his RIPPER rule learning algorithm to two email classification tasks: automatic detection of talk announcements (a binary classification task) and automatic foldering of email messages (a multinomial classification task). Cohen evaluated the system on three personal datasets and found that the rule-based approach outperformed a TF-IDF baseline.
In subsequent decades, there has been a large body of research on applying the modern machine learning technologies of the day to the task of personal content classification and clustering. For instance, \citet{Bekkerman:2004} conducted a benchmarking study comparing several popular text classifiers (of that time period), including Maximum Entropy, Naive Bayes, SVMs, and Winnow, for the task of automatic email foldering.
In a more recent example, \citet{Wendt+2016LabelProp} describe a multinomial classification algorithm for efficiently labeling machine-generated emails as belonging to a handful of categories (Social, Travel, Finance, etc.). The system clusters machine-generated emails according to their generating template, leverages preexisting email classifiers to label some of these clusters with the majority label, then uses label propagation to propagate labels from majority-labeled clusters to topically or lexically similar unlabeled clusters, and finally uses an efficient lookup mechanism to assign labels to newly-arriving emails belonging to a known cluster. These labels are surfaced in Gmail as ``Smart Labels.''
Office workers in particular tend to work on multiple work streams in parallel. \citet{Kong+2020Learning} attempt to support this multithreaded activity by automatically grouping files into ``workspaces,'' each workspace representing the files related to a particular work stream. Workspaces are orthogonal to the underlying filing system. Clustering is based on item--item similarity, expressed in terms of lexical, metadata, and access pattern features. The similarity function is not crafted by hand, but instead learned, using file co-access data as a weak supervision signal. When the system infers a new workspace, it suggests it to the user and allows them to accept or reject it.
\section{Email Spam Filtering, Phishing Detection, Abuse Detection}
\label{sec:spam-phishing-abuse}
Email can be misused to transmit unwanted, abusive, or outright dangerous messages. There has been a great body of research on detecting such messages automatically for the purpose of discarding or flagging them.
According to Internet folklore, the very first email message was sent in 1971 by Ray Tomlinson to himself. Just four years later, \citet{Postel1975JunkMail} wrote a ``Request for Comment'' entitled ``On the Junk Mail Problem''; alas, this technical note did not concern itself with spam messages, but with denial-of-service attacks caused by misbehaving senders. \citet{Denning1982ElectronicJunk} anticipated the rise of junk emails and advocated for the research community to devise filtering mechanisms to stem the flow of unwanted messages. However, as recounted by \citet{CranorLaMacchia1998Spam}, the first known ``email spam campaign'' did not take place until 1994.
During these early years, a variety of techniques were proposed to combat junk emails, including using cryptographic techniques to create per-sender email addresses \citep{Gabber+1998Curbing} and thus effectively creating ``security by obscurity,'' by imposing nonnegligible computational~\citep{DworkNaor1993Combating} or monetary~\citep{Abadi+2003Bankable} costs on sending email messages. However, none of these approaches found standards adoption; instead, email users rely on machine-learned classifiers to identify and filter out spam messages.
\citet{Sahami+1998Bayesian} were among the first to show that machine-learned classifiers are effective at filtering out spam messages. In particular, they employed a Naive Bayes classifier and used lexical features (both terms as well as specific phases such as ``Free money''), punctuation features (e.g. copious use of exclamation marks), and metadata features (e.g. the domain of the sender). \citet{Drucker+1999SVMforSpam} compared a number of different classification techniques and found that Support Vector Machines outperformed both boosted decision trees and learned classification rules---specifically, the RIPPER algorithm mentioned in the previous section. \citet{Androutsopoulos+2000Experimental} compared spam filters based on manually curated ``black-listed'' terms to that of machine-learned classifiers, and they found that the Naive Bayesian classifier they considered far outperformed the keyword-based approach. For a more comprehensive view of the state of ML-based spam classifiers, the interested reader is referred to a comprehensive survey by~\citet{Cormack2008EmailSpamFiltering} and a newer survey by~\citet{Dada+2019MLSpamFiltering}.
\citet{Spertus1997Smokey} considered the problem of hostile communications---messages that are inflammatory, hostile,or abusive in nature---and describes Smokey, a system for automatically filtering such messages from a user's email inbox. Spertus identified a number of signature features of hostile messages: sentences starting with ``You,'' use of imperatives, condescending language, profanities, insults, and epithets. These signatures are captured by a 47-element feature vector and supplied to a C4.5 decision tree-based binary classifier.
A third class of undesirable emails are phishing messages---messages that count on the recipient trusting them, and abusing that trust for monetary or other gain. There are many flavors of phishing, ranging from generic messages that are sent to many millions of users to messages that are targeted at one particular user (so-called ``spear-phishing''). \citet{Fette+2007Phishing} describe PILFER, a binary classifier that predicts whether a given email is a phishing email or not. PILFER uses content features of the email, including discrepancies between links and anchor text, whether any link contains a raw IP address, and whether some links go to ``non modal'' domains, as well as a number of features commonly used by spam classifiers. It also uses characteristics of any website linked from the email, such as the age of the website's domain registration. \citet{Nimeh+2007Comparison} compared the effectiveness of different classification techniques for phishing detection. They found that logistic regression provided the highest precision, while random forests performed best in terms of both recall and $F_1$ (the harmonic mean of performance and recall). The interested reader is referred to \citet{Hong2012StateOfPhishing} for a more in-depth survey of work on phishing email detection.
\section{Email Threading}
\label{sec:email-threading}
The previous two sections described tools and techniques to automatically process email messages---to file them, label them, or filter them. All these techniques help reduce information overload. Filtering out spam, phishing, and abuse also reduces annoyance, while filing and labeling make it easier to subsequently recover a message should the need arise.
Another way to impose structure on emails is by grouping the individual messages of a multiparty conversation into an {\em email thread}, and to surface conversations not at the level of individual messages but at the thread level. As we will discuss in Section~\ref{sec:refinding-using-search}, presenting messages in a threaded format is one of the most effective techniques for aiding refinding, and it has become the standard in all modern email experiences.
Almost since the inception of email, RFC 680 \citep{MyerHenderson1975MessageTransmissionProtocol} has provided a dedicated header (IN-REPLY-TO) for specifying the tree-structure of a conversation; however, the standard did not specify what the legal values of this header are, and therefore its implementation was fragmented. This situation was widely understood and well-described for instance by \citet{Lewis+1997Threading}, who then proposed to use information retrieval techniques (namely, lexical similarity of messages in the same thread due to quoted texts) to provide a work-around. Parallel attempts up to and including RFC 2822 \citep{Resnick2001InternetMessageFormat} did not manage to resolve this ambiguity. As a result, implementors of email clients had to reverse-engineer the various competing implementations of the IN-REPLY-TO header and resolve ambiguities in the client. \citet{Zawinski2002MessageThreading} provides an excellent account ``from the trenches'' of what was involved in achieving this, and a specific heuristic for performing this task. Zawinski's heuristic for reconstructing threads has been adopted by RFC 5256~\citep{CrispinMurchison2008SortAndThread}. In spite of this, recovering thread structure continued to be a subject of research; for example, \citet{YehHarnly2006Reassembly} describe how to use undocumented Microsoft Exchange header information to recover threads, and in the absence of such information, how to exploit similarity between quoting and quoted portions of a message thread in assembling the thread.
All of the above is concerned with grouping messages into a forest according to their in-reply-to relationship, such that a message is a child of another message if it was a (direct) response to the parent message. \citet{Ailon+2013Threading} refer to this as {\em syntactic threading}, and they propose the higher-level notion of {\em causal threads}: chains of email conversations that are causally related but not part of the same syntactic thread. For example, consider the scenario of purchasing an item from Amazon. Amazon may send you an email confirmation upon order, a second email once the item has shipped, and a third email upon delivery. Syntactic threading will not aggregate these three emails into the same thread. However, this {\em ordered--shipped--delivered} chain of emails is a common pattern and can be learned. Ailon et al. do so by performing template induction over the Yahoo email corpus, annotating machine-generated emails with their template IDs, mining the annotated corpus for temporal patterns (e.g., ``users are likely to receive a shipment confirmation shortly after an order confirmation''), and using these discovered relationships to group conversations into causal threads.
\section{Known-item Retrieval}
\label{sec:known-item-retrieval}
Early information retrieval research was heavily informed by publication search, and it was the Library Science community that coined the term {\em known-item search}. \citet{Swanson1972Requirements} defined known-item search as the attempt to locate a specific document that is known to exist. He distinguishes known-item search from {\em subject search}, the attempt to locate all available information on a given subject or topic, and {\em reference search}, the attempt to find the answer to a specific (often factoid) question. Swanson's article draws on earlier ethnographic studies, for example, an M.A. thesis by~\citet{Vaughan1968Effectiveness} that found that gaps in human memory recall (e.g. the precise title of a book) make known-item search a challenging problem.
\citet{Allen1989Recall} focused on the role of recall cues in known-item retrieval by examining the ability of 44 subjects to recall bibliographic, structural, and content cues of a short research paper they read three weeks earlier. Perhaps not surprisingly, the subjects performed worst on recalling bibliographic cues of the research paper. This finding is the ``raison d'etre'' for known-item retrieval systems: users are much more likely to recall the contents of an asset than its formal descriptors---its name, filing location, or the like.
\citet{Elsweiler+2007Towards} conducted a diary study to support the hypothesis that the root cause of unsuccessful known-item searches is a lapse in memory. Twenty-five participants recorded memory lapses over the course of one week, capturing 261 lapses. The majority of these memory lapses were retrospective---the failure to recall a previously memorized bit of information. Subjects employed a variety of preventative and recovery strategies to cope with such lapses. Elsweiler et al. translated these findings into a set of principles to improve the design of Personal Information Management tools: multimodal search capabilities, support for ``mental retrieval journeys,'' feedback cues about past interactions with surfaced results, and annotation facilities that allow users to encode newly remembered information in line with retrieval. Some of these ideas are further expanded in Section~\ref{sec:episodic-memory}.
\section{Refinding Using Search}
\label{sec:refinding-using-search}
Full-text search is one of the most powerful tools for locating items in a collection. As mentioned in Section~\ref{sec:known-item-retrieval}, the dominant search task for personal collections is known-item search: refinding an item that is known to be a part of the collection, be it a file, an email, or a visit to a website. Consequently, researchers have studied how to support known-item search in the context of personal search.
To gain a better understanding of such refinding tasks, \citet{ElsweilerRuthven2007Towards} asked 36 participants to record details of all email and web page refinding tasks they embarked on over a period of three weeks. They then labeled each search task according to whether the participant is looking for (1) a single nugget of information contained in a document that may or may not be known (lookup task), (2) a single document that is known to the user (item task), or (3) multiple documents that collectively answer the information need (multi-item task). For email searches, they found that the majority are lookup tasks, and lookup and item tasks combined account for about 95\% of all tasks. For web search, about 53\% were observed to be item tasks, and lookup and item tasks accounted for about 86\% of all web search tasks.
\citet{Dumais+2003Stuff} describe ``Stuff I've Seen'' (SIS), a system that records and indexes the information that a user is exposed to, including personal assets such as files and emails as well as personal exposure to public assets (e.g. web page visitations). SIS was evaluated through a user study that involved 234 participants who used the system for 47 days. To obtain qualitative insights, participants completed a questionnaire prior to using the system and again toward the end of the study; to obtain quantitative insights, interaction logs were collected throughout the study. One of the most surprising findings of the study was that participants preferred search results to be ranked by date rather than by relevance (SIS used lexical features and a BM25-based ranker). The authors speculate that users have a rough idea when they received, modified, or interacted with the item they are looking for, and they can therefore leverage dates effectively when skimming through the result list. This observation will be further explored in Section~\ref{sec:episodic-memory} below.
\citet{Whittaker+al2011Wasting} studied users' email refinding strategies in BlueMail~\citep{Tang+CHI2008TagIt}, an email client that supports filing, tagging, threading, and full-text search. Filing and tagging emails incur an up-front cost as email is received, while using full-text search and leveraging message threading do not. The authors found that opportunistic search behaviors that do not rely on foldering and tagging dominated refinding activities. They also found that refinding sessions that involved search and those that leveraged conversation threading were more successful than sessions that did not.
More recently, \citet{Ai+2017Characterizing} performed a query-log analysis of searches issued to the Microsoft Outlook cloud-based email service. The log contained about 2 million queries issued by 711,000 users over a span of one week. Most of the activities came from desktop clients of enterprise users. The authors also surveyed 324 of these users to obtain qualitative insights. The study confirmed that users' search tasks were overwhelmingly refinding of known items. Moreover, the study found that compared to web search, queries were shorter and fewer results were clicked. Finally, repeated queries were common, but different results were clicked in different sessions.
Large-scale studies along the lines of \citet{Ai+2017Characterizing} are difficult to conduct due to user privacy concerns, which make it all but impossible to analyze consumer email search behavior. \citet{Mackenzie+2019Exploring} proposed a novel approach to overcome this problem: rather than analyzing real consumer search behavior, they generate simulated searches by assigning email refinding tasks to workers on a crowd-sourcing platform, leveraging the workers' own email corpora. The worker is first presented with a target email drawn from their own email collection and then asked to refind it using a search interface. The authors found that workers employed a number of different query reformulation strategies to locate the target item, including typo correction, partial and full rewrites of the query, specialization and generalization, and occasional reversion to an earlier formulation. Query reformulations were common (with an average of about two reformulations per refinding task), suggesting that personal search indeed suffers from recall issues. The authors also examined the impact of different result orderings: time-based, relevance-based, and hybrid. Not surprisingly, they found that time-based email ordering led to more unsuccessful searches as the age of the target email increases.
Motivated by the recall issues in personal search, there were several studies on how to augment user queries to improve recall \citep{Kuzi+al:2017,Li+2019MultiView}. For instance, \citet{Li+2019MultiView} developed a method to find domain-specific synonyms for query terms that leverages user activities (query reformulations and query--result interactions) as well as word embeddings to find semantically related query terms. They validated their query expansion approach on Gmail search traffic and found that expanded queries experienced higher success rates, as measured by result click-through rates and click positions.
\section{Leveraging Episodic Memory}
\label{sec:episodic-memory}
As discussed in Section~\ref{sec:known-item-retrieval} above, the inability of users to locate an asset known to be in their collection can be viewed as a memory failure. It is not uncommon for users to forget the {\em location} of an asset (or key terms in a document that may be used to search for it) but instead to recall ancillary information that is not useful for retrieval, such as events that co-occurred when the user last interacted with the asset. The ability to recall events in time is referred to as {\em episodic memory}. Naturally, researchers have tried to leverage episodic memory for retrieval tasks.
\citet{LansdaleEdmonds1992Using} describe MEMOIRS (``Managing Episodic Memory for Office Retrieval Systems''), a prototype retrieval system designed to exploit event memory. MEMOIRS logs all interactions between users and their digital assets, thereby creating a timeline (a diary) of ``digital life events.'' Recording is automatic, and the timeline is immutable---events cannot be edited or deleted by the user. Users can browse the timeline and see all their asset interactions during a specific time window, and they can restrict the types of assets and interactions that are shown.
The idea of using timelines as a way to organize and retrieve digital assets was further explored by the Lifestreams system~\citep{FreemanFertig1995Lifestreams,FreemanGelernter1996Lifestreams}. While a MEMOIRS diary is a time-ordered sequence of interaction events with assets, a Lifestream is a time-ordered sequence of the assets themselves. Assets (files, emails, calendar entries, etc.) are added to a user's Lifestream when they are received. Lifestream supports user-definable agents that are triggered by events (e.g. by an interaction with an asset), and the invoked agent may place a clone of the document into the Lifestream (thereby emulating the functionality of MEMOIRS). Lifestreams can be browsed, filtered, and searched through an index.
\citet{Ringel+2003Landmarks} explore how episodic memory can be leveraged in a search system. Specifically, they studied how the search results of an existing personal information retrieval system (SIS, described in Section~\ref{sec:refinding-using-search} above) can be augmented with ``landmark events'' that anchor each result in time, and to what extent such landmarks support retrieval tasks. Like MEMOIRS, their system captures a user's interactions with their digital assets (i.e. personal events); the system also maintains a timeline of public events (holidays, news stories). Results to a search query are displayed on a vertical timeline, and adjacent to them the system displays landmark events, both public (e.g. a news event) and personal (e.g. a photo that was taken around that time). The authors found that adding landmarks to the chronological display of search results improved retrieval; both quantitatively (measured by task completion time) and qualitatively (measured by surveying ten participants of a user study).
Below, in Section~\ref{sec:digital-memory} on ``Human Digital Memory,'' we will further explore the idea of capturing the entirety of a user's interactions with the world and providing tools to retrieve these interactions.
\section{Test Collections}
\label{sec:test-collections}
The Information Retrieval research community has long relied on shared test collections as a way to perform experiments that can be repeated and be reproduced by other research groups, and that allows researchers to compete on a common research challenge and measure progress over time. Unfortunately, while there are a number of test collections based on public documents, there are but a few collections for personal retrieval tasks.
One such collection is the Enron corpus \citep{KlimtYangCEAS2004Introducing,KlimtYangECML2004Enron}, a set of 619,446 messages belonging to 158 individuals, all of them managers and executives of Enron. The email messages became available as part of the legal discovery process during Enron's bankruptcy proceedings. However, while many research papers have used this corpus, it is surrounded by some ethical concerns, chiefly the fact that the individuals whose messages are included did not provide their informed consent. Also, note that the Enron data set contains emails, but neither queries nor relevance judgments.
A second test set for email consists of a collection of 198,394 messages sent to a number of mailing lists of the World Wide Web Consortium (W3C), and these messages were recovered by the organizers of the Text Retrieval Conference (TREC) to provide a corpus for the email search task of the TREC-2005 Enterprise track \citep{Craswell+2005Overview}. Along with the collection comes a set of 150 queries (framed by the TREC organizers, not issued in a natural context) along with the one item in the collection that is deemed the correct result. Unfortunately, at the time of this writing, this collection is no longer publicly available.
The Avocado research email collection \citep{Oard+2015Avocado} is the newest and largest such corpus. To quote the researcher who provided the collection, it ``consists of emails and attachments taken from 279 accounts of a defunct information technology company referred to as {\em Avocado}.'' The corpus contains about 2 million items.
Separately, members of the same research group constructed a test set of 65 queries, each with about 100 results obtained by pooling the top 25 results of 18 retrieval ``systems'' (actually all implemented in Terrier), and each result labeled with its relevance and sensitivity judgments \citep{Spertus1997Smokey,Sayed+SIGIR2020TestCollection}.
\section{Crafted Rankers}
\label{sec:crafted-rankers}
As mentioned in Section~\ref{sec:refinding-using-search}, ~\citet{Dumais+2003Stuff} found that users of the SIS system preferred chronological ordering of search results over relevance-based ordering, lending support to the argument that episodic memory aids in retrieval tasks. On the other hand, \citet{Mackenzie+2019Exploring} found that when presenting results in chronological order, the rate of unsuccessful searches was correlated to the age of the target item, arguing that there is still a need for relevance-based ranking.
Scoring functions such as the cosine similarity of TF-IDF vectors, BM25F and PL2F are examples of {\em crafted} rankers, constructed by Information Retrieval researchers based on underlying models of relevance (e.g. the term vector-space model or the probabilistic relevance model). The BM25 ranking model proposed by~\citet{RobertsonWalker1994SomeSimple} has been the workhorse of lexical ranking models for a long time. \citet{Robertson+CIKM2004Simple} extended BM25 to ``fielded'' documents---structured documents where some portions (such as their title or section headers) are more significant than others. \citet{MacdonaldOunis2006Combining} propose to use a similar fielded model PL2F for email search. They evaluate this collection on the known-item search task of the TREC 2005 Enterprise track, which is based on a corpus of (public) messages sent to W3C mailing lists.
File retrieval systems (whether desktop-based or cloud-based) operate over a multitude of different file types, such as PDF documents, Word and Excel files, or email messages stored on a user's machine. These file types differ in their structure and their associated metadata. Fielded relevance models such as BM25F and PL2F are tuned to assign specific weights to each field, and these models may not perform optimally when used on a heterogeneous collection. \citet{KimCroft2010Ranking} investigated this issue. They propose to use type-specific probabilistic retrieval models for each file type, resulting in one result per type, and to merge these result lists into a final list using the CORI collection ranking metric~\citep{Callan+1995CORI}.
\citet{AbdelRahman2010NewApproach} describes an email retrieval system that leverages the threaded nature of email conversations. Prior to indexing, the system groups messages into threads using the heuristic by \citet{YehHarnly2006Reassembly} described in Section~\ref{sec:email-threading}. Conversations are then indexed using an algorithm due to~\citet{Broder+2006Indexing} (more fully described in Section~\ref{sec:infrastructure}), which maps conversations to ``document trees'' to capture the repeating nature of quoted previous messages in a conversation. The ranking function combines content, subject, and sender scores. The content score is the TF-IDF cosine similarity between query and message, the subject score is a binary match between query and subject line, and the sender score is the TF-IDF cosine similarity between the query and all messages ever received from that sender.
\section{Associative Browsing and Searching}
\label{sec:associative-browsing}
Modern search and recommender systems rely on a multitude of signals when retrieving and ranking search results or recommending items. These signals include the lexical similarity between query (or context) and item, linkage between items (e.g. hyperlinks between web pages), and user--item interactions. But while such signals are abundantly available to (say) a major web search engine, they are less so for a personal retrieval system. In particular, many types of personal collections lack explicit linkage between items, and interaction signals will be sparse due to the personal nature of the collection, where an item can only be accessed by its owner. This section looks at attempts to overcome this situation.
\citet{SoulesGanger2005Connections} describe Connections, a file system search tool that infers linkage between documents based on temporal co-access patterns and that uses link-based in addition to lexical features when retrieving and ranking search results. The system maintains a ``relationship graph'' where each node represents a file and each weighted edge represents the number of times two files have been co-accessed within (say) a half-minute window. The relationship graph is leveraged both during retrieval and ranking. During retrieval, an initial result set is obtained using standard lexical similarity between query and items, and then expanded by adding for each result all items that are nearby in the relationship graph, using edge weights and path lengths to determine proximity. The ranking phase uses the Indri ranker to perform an initial ranking, and then uses link-based signals (including PageRank and HITS) computed over the relationship graph to re-rank the results. The system was evaluated on a small test set (the results to 25 queries, assessed by six judges), and it outperformed Indri as a baseline system.
\citet{DongHalevy2005Platform} describe SemEx (``Semantic Explorer''), another personal retrieval system that creates associations between items and leverages these associations during retrieval. Unlike Connections, which inferred associations based on file co-access patterns and has but a single type of association, SemEx has a rich scheme of association types and uses handcrafted rules to extract them. Nodes in the SemEx graph can be items (emails, documents, etc.) as well as entities contained in the items (e.g. a person's name or email address); and edges are labeled with predicates (e.g. {\sl authorOf}). In effect, SemEx attempts to extract a Personal Knowledge Graph from the items in a user's collections. This rich graph allows for ``browsing by association,'' a powerful new paradigm for exploring one's personal assets, and also allows for more powerful queries. The downside is that SemEx assumes the existence of powerful association extraction tools---crafting such tools is a laborious and brittle process (e.g. metadata formats of specific file types may change over time).
\citet{Chirita+2006Beagle++} describe Beagle++, a desktop search tool that combines ideas from Connections and SemEx. It builds on top of Beagle, the desktop search engine that is part of the Gnome family of desktop tools for Linux. Beagle++ aims to improve Beagle's Lucene ranker by incorporating semantic information about the files in the corpus. Like Connections, it monitors interactions between users and items, and like SemEx, it extracts metadata such as author name from documents. This information is stored as RDF triplets, forming an RDF graph. Beagle++ computes PageRank-inspired ``ObjectRank'' scores for each item in the graph. At query time, ObjectRank scores are used to reweigh the TF-IDF scores returned by the Lucene ranker.
\citet{Chen+2009SearchYourMemory} describe XSearcher, later renamed to iMecho \citep{Chen+2009iMecho}, another desktop search engine that attempts to infer associations between items in the file system. iMecho uses multiple signals to infer associations: (1) lexical similarity between items, (2) items being co-located in the same folder, (3) explicit user activities (e.g. a user copying a file), and (4) implicit user activities such as co-access patterns. iMecho computes ObjectRank scores over the resulting graph and uses these scores at retrieval time to re-rank results. Unlike Beagle++, users can adjust the strength of each association type at query time, resulting in customizable rankings.
\citet{Kim+2010Building} describe LiFiDeA, another desktop search system that automatically creates associations between items and allows the user to leverage them for browsing and searching. LiFiDeA has two classes of items: actual documents (e.g. email, files and web sites) and {\em concepts} (e.g. authors, places or key terms). Some concepts such as the sender of an email are automatically extracted from documents while others are provided by the user. For example, a user can promote a query (which retrieves a set of documents), a tag attached to a document, or a document itself (e.g. a Wikipedia page) to be a concept. Concepts are linked to the documents they are extracted from or associated with. The LiFiDeA user interface allows browsing of both document and concept space, and to pivot from one to the other. In follow-on user studies, \citet{Kim+2011Evaluating} found that this associate browsing model indeed promotes known-item finding tasks.
\section{Learning to Rank for Personal Collections}
\label{sec:learning-to-rank}
An alternative to crafting rankers based on theories of document relevance is to automatically {\em learn} rankers from examples. In this context, an example consists of a query and the corresponding results set, each result having a binary or graded relevance label. Learning-to-Rank, first proposed by \citet{Freund+ICML1998Combining} and brought to the fore by \citet{Burges+ICML2005LTRusingGD}, is a vibrant area of research within the Information Retrieval community; the interested reader is referred to \citet{Liu2011LTRforIR} for a comprehensive overview of the early years.
\citet{Aberdeen+2010Learning} describe the learning algorithm behind Gmail Priority Inbox, a feature that displays the most {\em important} messages broken out at the top of the message list. It is worth emphasizing that this is not a relevance problem since priority messages are shown absent of any query or any explicitly stated information need. Priority Inbox uses past user interactions as a weak supervision signal for model training. The model is based on simple logistic regression, and each message is represented by content features (i.e. textual features of subject and body), thread features (did the user interact with this thread before?), and label features (was the message auto-labeled using a user-specified rule?). Since importance is a highly subjective notion, Priority Inbox trains a global model over the entire user population and a personalized additive model for each individual user. It also automatically tunes the per-user threshold above which a message is shown, based on the user's past interactions.
\citet{Carmel+2015Rank} revisited the question of whether email users preferred chronological or relevance-ordered search results. The authors present ``Relevance eXtended'' (REX), a learned ranking function for the email domain. REX uses BM25F to estimate the relevance of an email to a query, and furthermore it uses a number of query-independent importance signals: message (message freshness, user interactions with the message, presence of attachments, etc.), sender features (e.g. do this user and others actively correspond with the sender?), and recipient features (e.g. was this message sent to the user alone or to a group?). These features are combined using SVMRank~\citep{Cao+2006Adapting}, trained over a set of queries sampled from search logs. Each training example consists of two result messages returned for the same search, one clicked and the other not. In other words, REX uses clicks as a weak supervision signal. An editorial evaluation showed that users preferred REX over chronological ordering.
The approach taken by Carmel et al. of using result clicks as weak supervision signals ignores the fact that users exhibit a strong bias toward results near the top of the result list, and are far less likely to click results further down, even if these results are relevant. It is crucial to correct this bias when using interaction data as relevance labels. \citet{Wang+SIGIR2016LTRwithBias} describe a technique for debiasing such log-based training examples. The basic idea is to quantify the bias of each position (whether globally or query-dependent) by exposing a sample of users for a short period of time to randomized results. With users' bias for each position in the result list now known, the training data can be debiased using inverse propensity scoring. \citet{Joachims+WSDM2017Unbiased} explored this idea further, and in the process devised a less disruptive procedure for quantifying users' bias for each rank: rather than fully randomizing the result lists for some sample of users and some period of time (and thereby severely degrading their experience), just swap the top document with another randomly selected document in the list (a much less disruptive intervention). Subsequent works devised multiple techniques for estimating the position bias directly from biased click logs, without the need for exposing a sample of users for some period of time to degraded ranking experiences \citep{Wang+WSDM2018PBEstimation,Ai+SIGIR2018UnbiasedLTR,Agarwal+WSDM2019EstimatingPB}.
% acm-book.cls sets the natbib sort&compress option, which causes multiple-citation \citep like the one above to be rendered alphabetically, not chronologically. I would love to override this here, but don't know how.
All of the ranking models described in the previous paragraphs were based on LambdaMART~\citep{Wu+2010BoostingforIR} (a tree-based model) or some flavor of Support Vector Machines (a linear model). In recent years, there has been a great amount of interest in using deep neural networks for ranking tasks, but until recently they have been unable to replace LambdaMART, and merely served as second-stage additive scorers. \citet{Li+2019Combining} provide an example of such stacked rankers, evaluated on email and file search tasks.
%% \cite{Chen+2020Tuning} (to be completed)
\section{Desktop Search}
\label{sec:desktop-search}
Full-text search is one of the most powerful means for locating information, but it was applied to file systems relatively recently. The earliest work we are aware of is Glimpse \citep{ManberWu1994Glimpse}, a system for indexing and searching Unix file systems. Glimpse indexes individual words, but instead of a standard inverted index it uses a two-level indexing scheme that leads to smaller indices at the expense of slower queries.
Today's mainstream PC operating systems provide integrated full-text search over the local file system. Microsoft provided Indexing Service as early as 1996 and Windows Desktop Search in 2003. Apple introduced the OS X Spotlight search engine in April 2005. All of these search systems provide efficient retrieval of files containing the query terms; however, they do not provide any mechanisms for increasing recall (e.g. by expanding queries with synonymous terms) nor do they provide and form of relevance-based ranking.
\citet{Aharony2007Ranking} and \citet{Cohen+2008Ranking} studied three approaches to ranking desktop search results: a baseline crafted ranker (see Section~\ref{sec:crafted-rankers}) that uses content (via TF-IDF) and metadata features (access file, file path, file size, etc.); three learned ranking functions (see Section~\ref{sec:learning-to-rank}), the best being SVM-based; and another crafted ranker leveraging insights from the learned models. Interestingly, the learning rankers were trained on individual users’ logs.
There has been some research on creating reusable test collections for personal search tasks. \citet{Chernov+2007Building,Chernov+2008Evaluating} proposed a pooling-based technique for assembling a test collection for desktop search, and the privacy considerations that go along with such an endeavor. The authors did indeed create such a collection and used it for subsequent research, but as far as we know did not make it available to the research community. \citet{KimCroft2009Retrieval} proposed a different technique for creating such a reusable test collection for personal search tasks. Starting from the assumption that personal files are people-centric, the authors reuse a set of names contained in publicly accessible W3C discussion groups, then used a commercial search engine to collect documents for each named individual, and finally synthesized known-item queries targeting this collection.
\section{File Recommendation}
\label{sec:file-recommendation}
Search and recommendation are closely related. Both types of systems present a list of items to a user. In search settings these items are relevant with respect to an explicit query; in recommendation settings they are relevant given the user's context (e.g. previous consumption patterns). It is therefore natural to consider file recommendations.
\citet{Tata+2017QuickAccess} describe Quick Access, a service in Google Drive (a major cloud-based file system) that recommends to users the items they may need to access next. The recommender system is based on a simple feed-forward deep neural network. Each item is represented by a fine-grained time-series of past interaction events (e.g. open, edit, and comment) on that item, as well as other contextual information such as upcoming calendar events, number of items in a user's collection, and whether the account is a private or work account. The authors report an 18.5\% increase in recommendation accuracy over baseline heuristic (recommending the $k$ most-recently-used items) for highly active users, and they further report that on average users spend 50\% less time locating the file they wish to open. In a follow-up paper, \citet{Tata+2019ItemSuggest} describe the architecture of ItemSuggest, the recommender engine underlying the Quick Access service. The authors pay particular attention to how ItemSuggest manages data integration from many disparate sources (e.g. Drive activity logs and Calendar state) and how it supports the rapid exploration of new features and new model architectures. In a third paper, \citet{Chen+2020Improving} describe model quality improvement work (better candidate generation, improved DNN architecture, and latency improvements) they performed over a period of multiple years, leading to another 10.7\% increase in accuracy.
In order to better understand how users interact with file recommender systems, \citet{Xu+WWW2020Understanding} studied large-scale logs of user interactions with the Microsoft Office.com Recommended Document Pane (RDP). RDP is similar to Quick Access and moreover provides a short explanation of why an item was recommended. The 14 predefined explanations can be broken down into collaborative explanations (e.g. the item was shared by a collaborator) and individual explanations (e.g. you recently opened the item). The study measures user interactions in terms of click-through rate (CTR) and recognize rate (RR). While CTR is a raw interaction measure, RR measures whether the user recognized a useful item as it was recommended. The study found that there is a strong correlation between a recommendation's explanation and its recognition rate. Among collaborative explanations, {\sl CommentByOthers} had the highest RR; among individual explanations it was {\sl EditByYou}.
Another interesting finding of the study was that since recommendations are arranged in a linear list, they suffer from position bias similar to what has been observed for search results (see Section~\ref{sec:learning-to-rank}). Furthermore, once users have engaged with the system, their engagement remains high for the duration of the session. \citet{Jahanbaksh+CHIIR2020Effects} followed up on this large-scale quantitative study by performing a smaller-scale qualitative study, surveying 108 participants. The study provided deeper insights into the factors that drive successful interactions with file recommendations. Most noteworthy, the study found that presentation only influenced recognition time (a finding that is at odds with \citet{Tata+2017QuickAccess} who found that it drove engagement) and that users are more likely to recognize items they interacted with either recently or intensely. However, users professed that they were more ``interested'' in documents that were older and that they had interacted with less.
\section{Infrastructure for Cloud-based Personal Collections}
\label{sec:infrastructure}
Initially, personal collections tended to physically reside on a computer controlled by the owner of the collection. With the rise of cloud-based email and file storage services, this is no longer the case. Migrating collections to cloud-based services has many advantages---greater data stability (users do not have to worry about backups), accessibility from multiple devices, economies of scale, and ``wisdom of crowds'' effects resulting from ML models trained in a privacy-preserving fashion on the data and activities of large user populations. But these added capabilities also entail extra complexities: the dependence on centralized services, the reliance on network connections, the need for proper access control, and the dangers of privacy leaks. Unfortunately, not much has been published on systems infrastructure for cloud-based personal collections.
When indexing the collections of many users, a central service could maintain a separate inverted index per user (so access control can be handled at the collection level), or it could pool the documents of all users into a single inverted index (so access control has to be performed at the document level). The interested reader is referred to \citet{Hawking2011EnterpriseSearch} for a discussion of collection-level and document-level access control models. Each approach has its advantages and disadvantages---the former approach is simpler while the latter approach makes better use of hardware resources, due to better index compression and higher memory locality.
As it turns out, there are situations where multiple users own documents with overlapping content. For example, if two users engage in an email conversation and their mail clients quote the previous message when responding to it, then both users will have quoted passages in their corpus---in fact, the fraction of shared passages will increase as a conversation unfolds. \citet{Broder+2006Indexing} investigated how this phenomenon could be leveraged to reduce the size of a service-wide unified inverted index. They propose a new data structure called ``document trees'' that captures such sharing of passages of a document and stores but a single instance of the shared passage.
Respecting users' privacy and preventing any leakage of personal information---either directly or indirectly, say by machine-learned models memorizing individual training examples---is both unethical and may be (depending on the jurisdiction) illegal. There are industry-wide best practices to avoid such leakage. Here is a non-exhaustive list of such best practices: (1) all personal data must be encrypted both while at rest (i.e., persistently stored) and in flight (i.e., travelling over a network link); (2) each user has a separate encryption key, and user keys are escrowed by a highly secure key management service; (3) no individual has access to the key management services, to prevent insider attacks; (4) only cryptographically signed code can access the key management services; (5) all code is audited before being signed, again to prevent insider attacks; (6) privacy-preserving techniques such as $k$-anonymity (with high $k$) or differential privacy are used to prevent ML models from memorizing individual training examples; and (7) no personal data is retained any longer than required to provide a service to the user.
\section{Beyond Files and Email: Human Digital Memory}
\label{sec:digital-memory}
Taking a step back, the assets contained in a personal collection can be viewed as an externalized extension of that person's memory. The idea of using technology to support such externalized knowledge and to extend a person's memory can be traced back to Vannevar Bush's seminal essay ``As we may think'' \citep{Bush1945AsWeMayThink}, in which he proposed a device termed the {\em Memex} (for ``Memory Extension'') that stores and surfaces all of the information relevant to its user. Bush wrote this essay in 1945, at the very dawn of the computer age, and so his suggestions for how a Memex could be implemented involved microfilm and mechanical devices, not digital computers. However, the core idea of the essay, that technology can be used to amplify human cognitive ability, is as seductive today as it was in 1945.
Bush's vision provided a source of inspiration to generations of computer scientists, including J. C. R. Licklider and Doug Engelbart. Licklider wrote the 1960 article ``Man-computer symbiosis'' \citep{Licklider1960Symbiosis} which argued that computers are tools that can amplify human intellect; and as a government administrator he went on to promote research into much of the core technologies of today's information age: computer networking, accessible user interfaces, and the like. His prot{\'e}g{\'e} Doug Engelbart authored a 1962 essay on ``Augmenting human intellect'' \citep{Engelbart1962Augmenting}, founded the SRI Augmentation Research Center (ARC), and created the ``oN-Line System'' (NLS), the blueprint of the modern personal computer that pioneered graphical user interface, the mouse, hypertext, and video conferencing.
Almost 50 years after Bush articulated the Memex vision, with computers having entered the mainstream of society and psychologists having developed theories of memory, \citet{Lamming+1994Prosthesis} revisited the idea of the Memex. Their paper ``The design of a human memory prosthesis'' synthesizes insights from these two disciplines and argues that the time has come when memory extensions are feasible. The paper describes the PARC Tab, an early example of a pocket-sized computer that could be used for note-taking and that connected wirelessly to a larger computer. The authors argue that future memory prostheses should be aware of their environment, capture data automatically, allow for active note-taking, and support retrieval with context awareness.
\citet{Bell2001PersonalDigitalStore} laid out a vision for CyberAll, a system intended to digitize and store all of a user's personal information---music, photos, videos, writings, business transactions, and so on. The paper sketches the Windows-based prototype, describes the author's efforts to digitize content (e.g. paper books, vinyl records, and 8mm film), ruminates on the lifetime of physical media and file formats, and argues that continuous advances in storage technology make it economically feasible to store and archive all of a person's information. CyberAll evolved into MyLifeBits \citep{Gemmell+2006MyLifeBits}, and the prototype system was indeed populated with much of Bell's personal archive.
LifeLogs are an offshoot of human memory prostheses. The basic idea is to record all of a user's activities, including computer-mediated activities such as surfing the Web, but also continuous capture of physical reality, such as recording a person's location and the sounds and views that they experience. \citet{ChenJones2010Augmenting} investigate how such lifelogs advance the original vision of human memory augmentation.
\section{Future Directions for Personal Information Retrieval}
\label{sec:future-directions}
To conclude, we would like to identify several directions for future research into personal collections. The first line of research revolves around seamless retrieval from public and personal collections, the second around moving from asset retrieval to task assistance, and the third focused on how personal collections can be integrated into virtual assistants.
With the emergence of cloud-based computing services, personal collections are evermore likely to be stored in the cloud and retrieved through a cloud-based service (with optional caching on users' devices to cope with periods of disconnected operations). In such a setting, a query to a search service can easily surface both public and personal items. The research challenge is to aggregate the results of such a federated search service in a way that is most useful to its users.
People usually perform searches in the pursuit of an overarching goal. Within the Information Retrieval community, there is renewed interest in how retrieval services can go beyond delivering ``ten blue links'' in helping people to achieve these goals. For example, imagine being able to seamlessly retrieve all of your tax-related documents or to find your friend's address by issuing a single query. Recent advances in natural language processing have demonstrated that it is possible to build such advanced information seeking capabilities on top of general-purpose large language models. However, integrating personal information into such models in a way that is economically feasible is a wide-open problem.
As virtual assistants such as Google Assistant and Amazon Alexa become more pervasive, it is natural to believe that such assistants will be able to interact more and more with personal collections in the future. Some assistants already have limited access to personal information, for example, to provide access to contacts, calendar information, and so on. However, there is significant opportunity to develop advanced assistive search and intelligence capabilities over personal data that go well beyond what is possible today. This presents many interesting research opportunities at the intersection of personal collections and conversational search.
\chapter*{Authors’ Biographies}
\textbf{Michael Bendersky} is an engineering director at Google DeepMind. He is currently managing a team whose mission is improving algorithms, models, and metrics for information discovery across Google products. Michael holds a Ph.D. from the University of Massachusetts Amherst and is a distinguished member of the ACM. Michael has coauthored over a hundred publications in the areas of information retrieval, natural language processing, and machine learning. He coauthored two books in the \textit{Foundations and Trends in Information Retrieval} series: \textit{Information Retrieval with Verbose Queries} and \textit{Search and Discovery in Personal Email Collections}.
\medskip
\noindent
\textbf{Donald Metzler} is a senior staff research scientist at Google DeepMind. Prior to that, he was a research assistant professor at the University of Southern California and a senior research scientist at Yahoo! He is a distinguished member of the ACM and the recipient of multiple Test of Time and Best Paper awards. He has served as the program chair of the WSDM, ICTIR, and OAIR conferences and sat on the editorial boards of all the major journals in his field. He has published over a hundred research papers, has been awarded 10 patents, and is a coauthor of \textit{Search Engines: Information Retrieval in Practice}. He currently leads a research group focused on a variety of problems related to large language models, machine learning, and information retrieval.
\medskip
\noindent
\textbf{Marc Najork} is a distinguished research scientist at Google DeepMind. Previously, he was a senior director of research engineering at Google Research. Before joining Google in 2014, Marc was a principal researcher at Microsoft Research (2001–2014) and prior to that a researcher at the DEC/Compaq Systems Research Center (1993–2001). Marc earned a Ph.D. in computer science from the University of Illinois. He is an AAAS Fellow, an ACM Fellow, an IEEE Fellow, and a member of the SIGIR Academy. He has coauthored over a hundred research papers and holds 36 US patents. His service activities include editor-in-chief of the \textit{ACM Transactions on the Web} (2011–2014), news board cochair of the \textit{Communications of the ACM} (2008–2014), member of the ACM Publications Board (since 2020), conference chair of WSDM 2008, and program cochair of WWW 2004, WWW 2021, and WSDM 2025.
\medskip
\noindent
\textbf{Xuanhui Wang} is an engineering manager at Google DeepMind. He holds a Ph.D. and an M.S. from the University of Illinois at Urbana-Champaign and a B.E. from the University of Science and Technology of China. His research interests are in information retrieval, machine learning, and natural language processing. For learning-to-rank, he has open sourced two popular deep learning libraries: TFRanking on TensorFlow and Rax on Jax. He has published over seventy research papers in top IR/ML/NLP conferences and won several Best Paper awards: Best Short Paper Nomination of ICTIR (2020), Best Short Paper Award of ICTIR (2019), Best Paper Honorable Mention of WSDM (2017), and Best Paper Award in WSDM (2011). During his career, he has co-invented over 10 US patents.
% Bibliography file created with the ACM BibTeX style 'acm-book.bst' v1.0.
\begin{thebibliography}{111}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{#1}
\providecommand{\doi}[1]{DOI: #1}
\bibitem[Abadi et~al.(2003)Abadi, Birrell, Burrows, and
Wobber]{Abadi+2003Bankable}
M.~Abadi, A.~Birrell, M.~Burrows, and T.~Wobber.
\newblock 2003.
\newblock Bankable postage for network services.
\newblock In \emph{Proceedings of the 8th Asian Computing Science Conference},
volume 2896 of \emph{Lecture Notes in Computer Science}, pp. 72--90.
Springer.
\newblock \doi{10.1007/978-3-540-40965-6\_6}.
\newblock
\href{https://drive.google.com/file/d/1Tr20uNo2OQhhRx_-a0SxJs6Cvd3i45M8/view?usp=sharing}{[PDF]}.
\bibitem[AbdelRahman et~al.(2010)AbdelRahman, Hassan, and
Bahgat]{AbdelRahman2010NewApproach}
S.~AbdelRahman, B.~Hassan, and R.~Bahgat.
\newblock October 2010.
\newblock A new email retrieval ranking approach.
\newblock \emph{International Journal of Computer Science \& Information
Technology}, 2(5): 44--63.
\newblock \doi{10.5121/ijcsit.2010.2504}.
\newblock
\href{https://drive.google.com/file/d/1l8SjQXf6KH0pg3-IAalDEmbwVubY25Fo/view?usp=sharing}{[PDF]}.
\bibitem[Aberdeen et~al.(2010)Aberdeen, Pacovsky, and
Slater]{Aberdeen+2010Learning}
D.~Aberdeen, O.~Pacovsky, and A.~Slater.
\newblock 2010.
\newblock The learning behind gmail priority inbox.
\newblock In \emph{LCCC: NIPS 2010 Workshop on Learning on Cores, Clusters and
Clouds}.
\newblock
\href{https://drive.google.com/file/d/13vUjeNBUNskSWCtTUqwDdk1pFAvqfgvJ/view?usp=sharing}{[PDF]}.
\bibitem[Abu-Nimeh et~al.(2007)Abu-Nimeh, Nappa, Wang, and
Nair]{Nimeh+2007Comparison}
S.~Abu-Nimeh, D.~Nappa, X.~Wang, and S.~Nair.
\newblock 2007.
\newblock A comparison of machine learning techniques for phishing detection.
\newblock In \emph{Proceedings of the Anti-Phishing Working Groups 2nd Annual
eCrime Researchers Summit}, pp. 60--69.
\newblock \doi{10.1145/1299015.1299021}.
\newblock
\href{https://drive.google.com/file/d/1dlnLDt9Pd7Dn-vn-sz72vlM58GStDbgY/view?usp=sharing}{[PDF]}.
\bibitem[Agarwal et~al.(2019)Agarwal, Zaitsev, Wang, Li, Najork, and
Joachims]{Agarwal+WSDM2019EstimatingPB}
A.~Agarwal, I.~Zaitsev, X.~Wang, C.~Li, M.~Najork, and T.~Joachims.
\newblock 2019.
\newblock Estimating position bias without intrusive interventions.
\newblock In \emph{Estimating Position Bias without Intrusive Interventions},
pp. 474--482.
\newblock \doi{10.1145/3289600.3291017}.
\newblock
\href{https://drive.google.com/file/d/17eXOeA2K64zwyOm5axtyDjp8se4RhDwM/view?usp=sharing}{[PDF]}.
\bibitem[Aharony(2007)]{Aharony2007Ranking}
N.~Aharony.
\newblock March 2007.
\newblock \emph{On Ranking Techniques for Desktop Search}.
\newblock Master's thesis, Technion--Institute of Technology, Haifa, Israel.
\newblock
\href{https://drive.google.com/file/d/1ukQtl2co_jOCm0R93i6-OjMDbknUovnu/view?usp=sharing}{[PDF]}.
\bibitem[Ai et~al.(2017)Ai, Dumais, Craswell, and
Liebling]{Ai+2017Characterizing}
Q.~Ai, S.~T. Dumais, N.~Craswell, and D.~Liebling.
\newblock 2017.
\newblock Characterizing email search using large-scale behavioral logs and
surveys.
\newblock In \emph{Proceedings of the 26th International Conference on World
Wide Web}, pp. 1511--1520.
\newblock \doi{10.1145/3038912.3052615}.
\newblock
\href{https://drive.google.com/file/d/1xtwYuMzquqDWC6ufElhawOumZsBaSs_U/view?usp=sharing}{[PDF]}.
\bibitem[Ai et~al.(2018)Ai, Bi, Luo, Guo, and Croft]{Ai+SIGIR2018UnbiasedLTR}
Q.~Ai, K.~Bi, C.~Luo, J.~Guo, and W.~B. Croft.
\newblock 2018.
\newblock Unbiased learning to rank with unbiased propensity estimation.
\newblock In \emph{Proceedings of the 41st International ACM SIGIR Conference
on Research and Development in Information Retrieval}, pp. 385--394.
\newblock \doi{10.1145/3209978.3209986}.
\newblock
\href{https://drive.google.com/file/d/1xAO_4Jz5HE7RSBg7S5tZd6UFsqPrUOre/view?usp=sharing}{[PDF]}.
\bibitem[Ailon et~al.(2013)Ailon, Karnin, Liberty, and
Maarek]{Ailon+2013Threading}
N.~Ailon, Z.~S. Karnin, E.~Liberty, and Y.~Maarek.
\newblock 2013.
\newblock Threading machine generated email.
\newblock In \emph{Proceedings of the 6th ACM International Conference on Web
Search and Data Mining}, pp. 405--414.
\newblock \doi{10.1145/2433396.2433447}.
\newblock
\href{https://drive.google.com/file/d/1g_ki8-0KxSJ6jSXuKF6X7Dt6kRE3e0-3/view?usp=sharing}{[PDF]}.
\bibitem[Allen(1989)]{Allen1989Recall}
B.~Allen.
\newblock July 1989.
\newblock Recall cues in known-item retrieval.
\newblock \emph{Journal of the American Society for Information Science},
40(4): 246--252.
\newblock
\doi{10.1002/(SICI)1097-4571(198907)40:4{\textless}246::AID-ASI4{\textgreater}3.0.CO;2-Z}.
\newblock
\href{https://drive.google.com/file/d/13oDYZHifmyLMO8Rvj-qW5NWAQxPXdwFs/view?usp=sharing}{[PDF]}.
\bibitem[Androutsopoulos et~al.(2000)Androutsopoulos, Koutsias, Cbandrinos, and
Spyropoulos]{Androutsopoulos+2000Experimental}
I.~Androutsopoulos, J.~Koutsias, K.~V. Cbandrinos, and C.~D. Spyropoulos.
\newblock 2000.
\newblock An experimental comparison of naive bayesian and keyword-based
anti-spam filtering with personal e-mail messages.
\newblock In \emph{Proceedings of the 23rd Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp.
160--167.
\newblock \doi{10.1145/345508.345569}.
\newblock
\href{https://drive.google.com/file/d/1MLQWX7jl82RBUJnQfbf5o90YTH6UmQp8/view?usp=sharing}{[PDF]}.
\bibitem[{Apple Computer}(2013)]{Apple2013WhatsNew}
{Apple Computer}, 2013.
\newblock {OS X Mavericks}: What's new from {Mountain Lion}.
\newblock
\url{https://help.apple.com/osx-mavericks/whats-new-from-mountain-lion}.
\newblock (Slide 5).
\bibitem[Barreau and Nardi(1995)]{BarreauNardi1995Finding}
D.~Barreau and B.~A. Nardi.
\newblock 1995.
\newblock Finding and reminding: File organization from the desktop.
\newblock \emph{ACM SIGCHI Bulletin}, 27(3): 39--43.
\newblock \doi{10.1145/221296.221307}.
\newblock
\href{https://drive.google.com/file/d/1vZkRBGKc0H-sQCPupoH_WJnsEmaea2zo/view?usp=sharing}{[PDF]}.
\bibitem[Barreau(1995)]{Barreau1995Context}
D.~K. Barreau.
\newblock 1995.
\newblock Context as a factor in personal information management systems.
\newblock \emph{Journal of the American Society for Information Science},
46(5): 327--339.
\newblock
\href{https://drive.google.com/file/d/1eGpTvtxWXB25jCi5eVF2CH9hpP4NyFSn/view?usp=sharing}{[PDF]}.
\bibitem[Bekkerman(2004)]{Bekkerman:2004}
R.~Bekkerman.
\newblock 2004.
\newblock Automatic categorization of email into folders: Benchmark experiments
on enron and sri corpora.
\newblock Technical Report~218, Computer Science Department Faculty Publication
Series, University of Massachusetts Amherst.
\bibitem[Bell(2001)]{Bell2001PersonalDigitalStore}
G.~Bell.
\newblock January 2001.
\newblock A personal digital store.
\newblock \emph{Communications of the ACM}, 44(1): 86--91.
\newblock \doi{10.1145/357489.357513}.
\newblock
\href{https://drive.google.com/file/d/1sbEeBVr7exNUGUUhWqo5XAnRKsdcFDuo/view?usp=sharing}{[PDF]}.
\bibitem[Birrell et~al.(1997)Birrell, Perl, Schroeder, and
Wobber]{Birrell+al1997Pachyderm}
A.~Birrell, S.~Perl, M.~Schroeder, and T.~Wobber, 1997.
\newblock Pachyderm: A web-based application for email and news.
\newblock \url{http://birrell.org/andrew/talks/pachyderm.pdf}.
\bibitem[Bloehdorn et~al.(2006)Bloehdorn, G{\"o}rlitz, Schenk, and
V{\"o}lkel]{Bloehdorn+2006TagFS}
S.~Bloehdorn, O.~G{\"o}rlitz, S.~Schenk, and M.~V{\"o}lkel.
\newblock 2006.
\newblock {TagFS} --- tag semantics for hierarchical file systems.
\newblock In \emph{Proceedings of the 6th International Conference on Knowledge
Management}.
\newblock
\href{https://drive.google.com/file/d/1QiMzG-XD-lFWj8N2n9_GfXxAE8mU1hc6/view?usp=sharing}{[PDF]}.
\bibitem[Boardman and Sasse(2004)]{BoardmanSasse2004Stuff}
R.~Boardman and M.~A. Sasse.
\newblock 2004.
\newblock ``{Stuff} goes into the computer and doesn’t come out'': A
cross-tool study of personal information management.
\newblock In \emph{Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems}, pp. 583--590.
\newblock \doi{10.1145/985692.985766}.
\newblock
\href{https://drive.google.com/file/d/1LTe4uqfVrOgsigmOHyiKm3uqAYboB4oM/view?usp=sharing}{[PDF]}.
\bibitem[Bowman et~al.(1994)Bowman, Dharap, Baruah, Camargo, and
Potti]{Bowman+1994FileSystem}
M.~Bowman, C.~Dharap, M.~Baruah, B.~Camargo, and S.~Potti.
\newblock 1994.
\newblock A file system for information management.
\newblock In \emph{Proceedings of the ISMM International Conference on
Intelligent Information Management Systems}, pp. 66--71.
\newblock
\href{https://drive.google.com/file/d/1g9fMfHIQlBwdXIyeIRufjNH88SJfTiqr/view?usp=sharing}{[PDF]}.
\bibitem[Broder et~al.(2006)Broder, Eiron, Fontoura, Herscovici, Lempel,
McPherson, Qi, and Shekita]{Broder+2006Indexing}
A.~Z. Broder, N.~Eiron, M.~Fontoura, M.~Herscovici, R.~Lempel, J.~McPherson,
R.~Qi, and E.~Shekita.
\newblock 2006.
\newblock Indexing shared content in information retrieval systems.
\newblock In \emph{Proceedings of the 10th International Conference on
Extending Database Technology}, volume 3896 of \emph{Lecture Notes in
Computer Science}, pp. 313--330. Springer.
\newblock \doi{10.1007/11687238\_21}.
\newblock
\href{https://drive.google.com/file/d/119MPyMw0hgN2MD4UcWIPoxUdqEFATIiF/view?usp=sharing}{[PDF]}.
\bibitem[Burges et~al.(2005)Burges, Shaked, Renshaw, Lazier, Deeds, Hamilton,
and Hullender]{Burges+ICML2005LTRusingGD}
C.~Burges, T.~Shaked, E.~Renshaw, A.~Lazier, M.~Deeds, N.~Hamilton, and
G.~Hullender.
\newblock 2005.
\newblock Learning to rank using gradient descent.
\newblock In \emph{Proceedings of the 22nd International Conference on Machine
Learning}.
\newblock
\href{https://drive.google.com/file/d/1m6JwcNwXE2hvc8yzjY5S0N-wCDg7Y27X/view?usp=sharing}{[PDF]}.
\bibitem[Bush(1945)]{Bush1945AsWeMayThink}
V.~Bush.
\newblock July 1945.
\newblock As we may think.
\newblock \emph{The Atlantic Monthly}, 176(1): 101--108.
\newblock
\href{https://drive.google.com/file/d/14lMnO7G5t-wbRQlVflEb579XZqclU9qw/view?usp=sharing}{[PDF]}.
\bibitem[Callan et~al.(1995)Callan, Lu, and Croft]{Callan+1995CORI}
J.~P. Callan, Z.~Lu, and W.~B. Croft.
\newblock 1995.
\newblock Searching distributed collections with inference networks.
\newblock In \emph{Proceedings of the 18th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp. 21--28.
\newblock \doi{10.1145/215206.215328}.
\newblock
\href{https://drive.google.com/file/d/1Dz0WJw-iF_xmheisgkHnu40nrw1j-hEU/view?usp=sharing}{[PDF]}.
\bibitem[Cao et~al.(2006)Cao, Xu, Liu, Li, Huang, and Hon]{Cao+2006Adapting}
Y.~Cao, J.~Xu, T.-Y. Liu, H.~Li, Y.~Huang, and H.-W. Hon.
\newblock 2006.
\newblock Adapting ranking svm to document retrieval.
\newblock In \emph{Proceedings of the 29th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp.
186--193.
\newblock \doi{10.1145/1148170.1148205}.
\newblock
\href{https://drive.google.com/file/d/1IaL0KyofbTD535q7hQLXke1yQyo6Jt-5/view?usp=sharing}{[PDF]}.
\bibitem[Carmel et~al.(2015)Carmel, Halawi, Lewin-Eytan, Maarek, and
Raviv]{Carmel+2015Rank}
D.~Carmel, G.~Halawi, L.~Lewin-Eytan, Y.~Maarek, and A.~Raviv.
\newblock 2015.
\newblock Rank by time or by relevance? revisiting email search.
\newblock In \emph{Proceedings of the 24th ACM International Conference on
Information and Knowledge Management}, pp. 283--292.
\newblock \doi{10.1145/2806416.2806471}.
\newblock
\href{https://drive.google.com/file/d/1pLbO-tywDGWrHCkUL5o6RTTHxs1n1r8N/view?usp=sharing}{[PDF]}.
\bibitem[Chen et~al.(2009{\natexlab{a}})Chen, Guo, Wu, and
Wang]{Chen+2009iMecho}
J.~Chen, H.~Guo, W.~Wu, and W.~Wang.
\newblock 2009{\natexlab{a}}.
\newblock imecho: An associative memory based desktop search system.
\newblock In \emph{Proceedings of the 18th ACM Conference on Information and
Knowledge Management}, pp. 731--740.
\newblock \doi{10.1145/1645953.1646046}.
\newblock
\href{https://drive.google.com/file/d/1pwhdt9aIkLxnfSgcNIKTNhFgrgDwqGkF/view?usp=sharing}{[PDF]}.
\bibitem[Chen et~al.(2009{\natexlab{b}})Chen, Guo, Wu, and
Xie]{Chen+2009SearchYourMemory}
J.~Chen, H.~Guo, W.~Wu, and C.~Xie.
\newblock 2009{\natexlab{b}}.
\newblock Search your memory! - an associative memory based desktop search
system.
\newblock In \emph{Proceedings of the 2009 ACM SIGMOD International Conference
on Management of Data}, pp. 1099--1102.
\newblock \doi{10.1145/1559845.1559992}.
\newblock
\href{https://drive.google.com/file/d/1hYdO0MA7A14IeGbke4427qXMsY2nWsnG/view?usp=sharing}{[PDF]}.
\bibitem[Chen et~al.(2020)Chen, Qin, Wilson, Calaci, Rose, Evans, Abraham,
Metzler, Tata, and Colagrosso]{Chen+2020Improving}
S.~J. Chen, Z.~Qin, Z.~Wilson, B.~Calaci, M.~Rose, R.~Evans, S.~Abraham,
D.~Metzler, S.~Tata, and M.~Colagrosso.
\newblock 2020.
\newblock Improving recommendation quality in {Google Drive}.
\newblock In \emph{Proceedings of the 26th ACM SIGKDD International Conference
on Knowledge Discovery \& Data Mining}, pp. 2900--2908.
\newblock \doi{10.1145/3394486.3403341}.
\newblock
\href{https://drive.google.com/file/d/19CGh45baCcl8sQqaSKXIrwEnT9-Typ29/view?usp=sharing}{[PDF]}.
\bibitem[Chen and Jones(2010)]{ChenJones2010Augmenting}
Y.~Chen and G.~J.~F. Jones.
\newblock 2010.
\newblock Augmenting human memory using personal lifelogs.
\newblock In \emph{Proceedings of the 1st Augmented Human International
Conference}.
\newblock \doi{https://doi.org/10.1145/1785455.1785479}.
\newblock
\href{https://drive.google.com/file/d/1sqE-Msbpq4VpnUfIEpOCMlhxZh7zjC15/view?usp=sharing}{[PDF]}.
\bibitem[Chernov et~al.(2007)Chernov, Serdyukov, Chirita, Demartini, and
Nejdl]{Chernov+2007Building}
S.~Chernov, P.~Serdyukov, P.-A. Chirita, G.~Demartini, and W.~Nejdl.
\newblock 2007.
\newblock Building a desktop search test-bed.
\newblock In \emph{Proceedings of the 29th European Conference on {IR}
Research}, volume 4425 of \emph{Lecture Notes in Computer Science}, pp.
686--690. Springer.
\newblock \doi{10.1007/978-3-540-71496-5\_69}.
\newblock
\href{https://drive.google.com/file/d/1p3nV2vl80C63NYi7iBJCjouXmeZb2os5/view?usp=sharing}{[PDF]}.
\bibitem[Chernov et~al.(2008)Chernov, Demartini, Herder, Kopycki, and
Nejdl]{Chernov+2008Evaluating}
S.~Chernov, G.~Demartini, E.~Herder, M.~Kopycki, and W.~Nejdl.
\newblock 2008.
\newblock Evaluating personal information management using an activity logs
enriched desktop dataset.
\newblock In \emph{Personal Information Management Workshop at CHI 2008}.
\newblock
\href{https://drive.google.com/file/d/18j4qAadnesqiEH3DyuoUgMIHzndHjUnr/view?usp=sharing}{[PDF]}.
\bibitem[Chirita et~al.(2006)Chirita, Costache, Nejdl, and
Paiu]{Chirita+2006Beagle++}
P.-A. Chirita, S.~Costache, W.~Nejdl, and R.~Paiu.
\newblock 2006.
\newblock Beagle++: Semantically enhanced searching and ranking on the desktop.
\newblock In \emph{Proceedings of the 3rd European Semantic Web Conference},
volume 4011 of \emph{Lecture Notes in Computer Science}, pp. 348--362.
Springer.
\newblock \doi{10.1007/11762256\_27}.
\newblock
\href{https://drive.google.com/file/d/1C83W53x61DS5uYHj0BTSed16o2f4k4a8/view?usp=sharing}{[PDF]}.
\bibitem[Cohen et~al.(2008)Cohen, Domshlak, and Zwerdling]{Cohen+2008Ranking}
S.~Cohen, C.~Domshlak, and N.~Zwerdling.
\newblock March 2008.
\newblock On ranking techniques for desktop search.
\newblock \emph{ACM Transactions on Information Systems}, 26(2).
\newblock \doi{10.1145/1344411.1344417}.
\newblock
\href{https://drive.google.com/file/d/1e3fZjKeGAUYiNJUrnMXLUvtNR2fLBJBI/view?usp=sharing}{[PDF]}.
\bibitem[Cohen(1996)]{Cohen1996Learning}
W.~W. Cohen.
\newblock 1996.
\newblock Learning rules that classify e-mail.
\newblock In M.~A. Hearst and H.~Hirsh, eds., \emph{Papers from the AAAI Spring
Symposium: Machine Learning in Information Access}, AAAI Technical Report
SS-96-05. Association for the Advancement of Artificial Intelligence.
\newblock
\href{https://drive.google.com/file/d/1xrLtwHP3JciIJ4B_v0hxgKJtjmb57cq_/view?usp=sharing}{[PDF]}.
\bibitem[Cormack(2008)]{Cormack2008EmailSpamFiltering}
G.~V. Cormack.
\newblock 2008.
\newblock Email spam filtering: A systematic review.
\newblock \emph{Foundations and Trends in Information Retrieval}, 1(4):
335--455.
\newblock \doi{10.1561/1500000006}.
\newblock
\href{https://drive.google.com/file/d/1c3rTA38Tg23s2PR8VhvW6G-h81C4Sudm/view?usp=sharing}{[PDF]}.
\bibitem[Cranor and LaMacchia(1998)]{CranorLaMacchia1998Spam}
L.~F. Cranor and B.~A. LaMacchia.
\newblock August 1998.
\newblock Spam!
\newblock \emph{Communications of the ACM}, 41(8): 74--83.
\newblock \doi{10.1145/280324.280336}.
\newblock
\href{https://drive.google.com/file/d/1D3A24BUraWRc9PXalAml4Fx4wiDVIn80/view?usp=sharing}{[PDF]}.
\bibitem[Craswell et~al.(2005)Craswell, de~Vried, and
Soboroff]{Craswell+2005Overview}
N.~Craswell, A.~P. de~Vried, and I.~Soboroff.
\newblock 2005.
\newblock Overview of the {TREC-2005} enterprise track.
\newblock In \emph{Proceedings of the 14th Text REtrieval Conference,}.
\newblock
\href{https://drive.google.com/file/d/1fKAJb_ExNawZkwitHhD4Xuy78J1B7Lp0/view?usp=sharing}{[PDF]}.
\bibitem[Crispin and Murchison(2008)]{CrispinMurchison2008SortAndThread}
M.~Crispin and K.~Murchison, June 2008.
\newblock Internet message access protocol - sort and thread extensions.
\newblock Internet Engineering Task Force, Network Working Group, Request for
Comment 5256.
\newblock
\href{https://drive.google.com/file/d/1drmimqzWGjWb_Imq0qzXeHU8ePObenkS/view?usp=sharing}{[PDF]}.
\bibitem[Cutrell et~al.(2006)Cutrell, Robbins, Dumais, and
Sarin]{Cutrell+2006Flexible}
E.~Cutrell, D.~C. Robbins, S.~T. Dumais, and R.~Sarin.
\newblock 2006.
\newblock Fast, flexible filtering with {Phlat} -- personal search and
organization made easy.
\newblock In \emph{Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems}, pp. 261--270.
\newblock \doi{10.1145/1124772.1124812}.
\newblock
\href{https://drive.google.com/file/d/103mrjiwRTsUrDDrqIE-gOQGaS0ApPZJ5/view?usp=sharing}{[PDF]}.
\bibitem[Dada et~al.(2019)Dada, Bassi, Chiroma, Abdulhamid, Adetunmbi, and
Ajibuwa]{Dada+2019MLSpamFiltering}
E.~G. Dada, J.~S. Bassi, H.~Chiroma, S.~M. Abdulhamid, A.~O. Adetunmbi, and
O.~E. Ajibuwa.
\newblock June 2019.
\newblock Machine learning for email spam filtering: review, approaches and
open research problems.
\newblock \emph{Heliyon}, 5(6).
\newblock \doi{10.1016/j.heliyon.2019.e01802}.
\newblock
\href{https://drive.google.com/file/d/1SdnI0luOIJTpF4qqpV2Uf_wv2xCScdax/view?usp=sharing}{[PDF]}.
\bibitem[Denning(1982)]{Denning1982ElectronicJunk}
P.~J. Denning.
\newblock March 1982.
\newblock Electronic junk.
\newblock \emph{Communications of the ACM}, 25(3): 163--165.
\newblock \doi{10.1145/358453.358454}.
\newblock
\href{https://drive.google.com/file/d/1_kr7onLKSikxsssaIHIu8Xxwwu6TG6c2/view?usp=sharing}{[PDF]}.
\bibitem[Dong and Halevy(2005)]{DongHalevy2005Platform}
X.~L. Dong and A.~Halevy.
\newblock 2005.
\newblock A platform for personal information management and integration.
\newblock In \emph{Proceedings of the 2nd Biennial Conference on Innovative
Data Systems Research}.
\newblock
\href{https://drive.google.com/file/d/1CiXAkzP-lFsybQ8RPnSBaNU2cGS0asdV/view?usp=sharing}{[PDF]}.
\bibitem[Drucker et~al.(1999)Drucker, Wang, and Vapnik]{Drucker+1999SVMforSpam}
H.~Drucker, D.~Wang, and V.~N. Vapnik.
\newblock September 1999.
\newblock Support vector machines for spam categorization.
\newblock \emph{IEEE Transactions on Neural Networks}, 10(5): 1048--1054.
\newblock \doi{10.1109/72.788645}.
\newblock
\href{https://drive.google.com/file/d/1x9jyzb-abwIxxCjc59TnqP7e1ii3G331/view?usp=sharing}{[PDF]}.
\bibitem[Dumais et~al.(2003)Dumais, Cutrell, Cadiz, Jancke, Sarin, and
Robbins]{Dumais+2003Stuff}
S.~Dumais, E.~Cutrell, J.~Cadiz, G.~Jancke, R.~Sarin, and D.~C. Robbins.
\newblock 2003.
\newblock Stuff i've seen: A system for personal information retrieval and
re-use.
\newblock In \emph{Proceedings of the 26th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp. 72--79.
\newblock \doi{10.1145/860435.860451}.
\newblock
\href{https://drive.google.com/file/d/1zwPj0sngBKto92UARBbv4FtG1O1B1_0P/view?usp=sharing}{[PDF]}.
\bibitem[Dwork and Naor(1993)]{DworkNaor1993Combating}
C.~Dwork and M.~Naor.
\newblock 1993.
\newblock Pricing via processing or combatting junk mail.
\newblock In \emph{Proceedings of the 12th Annual International Cryptology
Conference}, volume~740 of \emph{Lecture Notes in Computer Science}, pp.
139--147. Springer.
\newblock \doi{10.1007/3-540-48071-4\_10}.
\newblock
\href{https://drive.google.com/file/d/1bwUCZhkMRrGc2vQ_nZmlA4-aV-uo27l6/view?usp=sharing}{[PDF]}.
\bibitem[Elsweiler and Ruthven(2007)]{ElsweilerRuthven2007Towards}
D.~Elsweiler and I.~Ruthven.
\newblock 2007.
\newblock Towards task-based personal information management evaluations.
\newblock In \emph{Proceedings of the 30th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp. 23--30.
\newblock \doi{10.1145/1277741.1277748}.
\newblock
\href{https://drive.google.com/file/d/1E-lvieBo4Na5lNMCZ6baGkhBi-wqTqev/view?usp=sharing}{[PDF]}.
\bibitem[Elsweiler et~al.(2007)Elsweiler, Ruthven, and
Jones]{Elsweiler+2007Towards}
D.~Elsweiler, I.~Ruthven, and C.~Jones.
\newblock May 2007.
\newblock Towards memory supporting personal information management tools.
\newblock \emph{Journal of the American Society for Information Science and
Technology}, 58(7): 924--946.
\newblock \doi{10.1002/asi.20570}.
\newblock
\href{https://drive.google.com/file/d/19gxH5nmqEg66NfL8asJPV6HS8dWc7hMm/view?usp=sharing}{[PDF]}.
\bibitem[Elsweiler et~al.(2008)Elsweiler, Baillie, and
Ruthven]{Elsweiler+2008Memory}
D.~Elsweiler, M.~Baillie, and I.~Ruthven.
\newblock 2008.
\newblock Exploring memory in email refinding.
\newblock \emph{{ACM} Transactions on Information Systems}, 26(4): 21:1--21:36.
\newblock \doi{10.1145/1402256.1402260}.
\newblock
\href{https://drive.google.com/file/d/19QuZQd5V6z7bzXYBAgBYLgczTPq488sP/view?usp=sharing}{[PDF]}.
\bibitem[Elsweiler et~al.(2011)Elsweiler, Baillie, and
Ruthven]{Elsweiler+2011Difficult}
D.~Elsweiler, M.~Baillie, and I.~Ruthven.
\newblock 2011.
\newblock What makes re-finding information difficult? a study of email
re-finding.
\newblock In \emph{Proceedings of the 33rd European Conference on {IR}
Research}, volume 6611 of \emph{Lecture Notes in Computer Science}, pp.
568--579. Springer.
\newblock \doi{10.1007/978-3-642-20161-5\_57}.
\newblock
\href{https://drive.google.com/file/d/1TD4GfZiY5I9WnHEWrNBdHawPn1ZcUJig/view?usp=sharing}{[PDF]}.
\bibitem[Engelbart(1962)]{Engelbart1962Augmenting}
D.~C. Engelbart.
\newblock October 1962.
\newblock Augmenting human intellect: A conceptual framework.
\newblock Summary Report AFOSR-3223, Stanford Research Institute.
\newblock
\href{https://drive.google.com/file/d/1OUq9JYP8YW57x_Gm4Yg9MyIKHhMKhHUh/view?usp=sharing}{[PDF]}.
\bibitem[Fette et~al.(2007)Fette, Sadeh, and Tomasic]{Fette+2007Phishing}
I.~Fette, N.~Sadeh, and A.~Tomasic.
\newblock 2007.
\newblock Learning to detect phishing emails.
\newblock In \emph{Proceedings of the 16th International Conference on World
Wide Web}, pp. 649--656.
\newblock \doi{10.1145/1242572.1242660}.
\newblock
\href{https://drive.google.com/file/d/1OOFkxILx7gmFjCqRruMRRKFt_elTVRWr/view?usp=sharing}{[PDF]}.
\bibitem[Freeman and Fertig(1995)]{FreemanFertig1995Lifestreams}
E.~Freeman and S.~Fertig.
\newblock 1995.
\newblock Lifestreams: Organizing your electronic life.
\newblock In R.~Burke, ed., \emph{Papers from the AAAI Fall Symposium on AI
Applications in Knowledge Navigation and Retrieval}, AAAI Technical Report
FS-95-03, pp. 38--44. Association for the Advancement of Artificial
Intelligence.
\newblock ISBN 978-0-929280-95-0.
\newblock
\href{https://drive.google.com/file/d/11HkV-sOCdyQj-qCfYpBcd1HgNGAHtJ9T/view?usp=sharing}{[PDF]}.
\bibitem[Freeman and Gelernter(1996)]{FreemanGelernter1996Lifestreams}
E.~Freeman and D.~Gelernter.
\newblock March 1996.
\newblock Lifestreams: A storage model for personal data.
\newblock \emph{ACM SIGMOD Record}, 25: 80--86.
\newblock \doi{10.1145/381854.381893}.
\newblock
\href{https://drive.google.com/file/d/1Ff_sOovCuQVaLKYZ6GgP-hqb-LciBKpj/view?usp=sharing}{[PDF]}.
\bibitem[Freund et~al.(1998)Freund, Iyer, Shapire, and
Singer]{Freund+ICML1998Combining}
Y.~Freund, R.~Iyer, R.~E. Shapire, and Y.~Singer.
\newblock 1998.
\newblock An efficient boosting algorithm for combining preferences.
\newblock In \emph{Proceedings of the 15th International Conference on Machine
Learning}, pp. 170--178.
\newblock
\href{https://drive.google.com/file/d/1XG06L53QkbCRO5oXuKBFvKXRcXz0ZWeW/view?usp=sharing}{[PDF]}.
\bibitem[Gabber et~al.(1998)Gabber, Jakobsson, Matias, and
Mayer]{Gabber+1998Curbing}
E.~Gabber, M.~Jakobsson, Y.~Matias, and A.~Mayer.
\newblock 1998.
\newblock Curbing junk e-mail via secure classification.
\newblock In \emph{Proceedings of the 2nd International Conference on Financial
Cryptography}, volume 1465 of \emph{Lecture Notes in Computer Science}, pp.
198--213. Springer.
\newblock \doi{10.1007/BFb0055484}.
\newblock
\href{https://drive.google.com/file/d/1mman20P3txxeoKRyIqV0Ph3KQRTumrDI/view?usp=sharing}{[PDF]}.
\bibitem[Gemmell et~al.(2006)Gemmell, Bell, and Lueder]{Gemmell+2006MyLifeBits}
J.~Gemmell, G.~Bell, and R.~Lueder.
\newblock January 2006.
\newblock Mylifebits: A personal database for everything.
\newblock \emph{Communications of the ACM}, 49(1): 88--95.
\newblock \doi{10.1145/1107458.1107460}.
\newblock
\href{https://drive.google.com/file/d/1uMtPyIwNNV30YtOJE8l1Yt3GeqGECk43/view?usp=sharing}{[PDF]}.
\bibitem[Gifford et~al.(1991)Gifford, Jouvelot, Sheldon, and
O'Toole]{Gifford+1991Semantic}
D.~K. Gifford, P.~Jouvelot, M.~A. Sheldon, and J.~W. O'Toole.
\newblock 1991.
\newblock Semantic file systems.
\newblock In \emph{Proceedings of the 13th ACM Symposium on Operating Systems
Principles}, pp. 16--25.
\newblock \doi{10.1145/121132.121138}.
\newblock
\href{https://drive.google.com/file/d/1YnXcW3JCmu0UV1s9G4HGa_OGCnqN_5MF/view?usp=sharing}{[PDF]}.
\bibitem[Hawking(2011)]{Hawking2011EnterpriseSearch}
D.~Hawking.
\newblock 2011.
\newblock Enterprise search.
\newblock In R.~Baeza-Yates and B.~Ribeiro-Neto, eds., \emph{Modern Information
Retrieval: The Concepts and Technology Behind Search (2nd Edition)}, pp.
645--686. Addison-Wesley.
\newblock ISBN 978-0321416919.
\newblock
\href{https://drive.google.com/file/d/15w1E1Xn5GF2b2x2yNFs5JSoD1MgD3cDw/view?usp=sharing}{[PDF]}.
\bibitem[Hong(2012)]{Hong2012StateOfPhishing}
J.~Hong.
\newblock January 2012.
\newblock The state of phishing attacks.
\newblock \emph{Communications of the ACM}, 55(1): 74--81.
\newblock \doi{10.1145/2063176.2063197}.
\newblock
\href{https://drive.google.com/file/d/1Csc4a0oCP_sUM0AtMHaOQ5zgCpyhklnp/view?usp=sharing}{[PDF]}.
\bibitem[Jahanbakhsh et~al.(2020)Jahanbakhsh, Awadallah, Dumais, and
Xu]{Jahanbaksh+CHIIR2020Effects}
F.~Jahanbakhsh, A.~H. Awadallah, S.~T. Dumais, and X.~Xu.
\newblock 2020.
\newblock Effects of past interactions on user experience with recommended
documents.
\newblock In \emph{Proceedings of the 2020 Conference on Human Information
Interaction and Retrieval}, pp. 153--162.
\newblock \doi{10.1145/3343413.3377977}.
\newblock
\href{https://drive.google.com/file/d/1WIC81qdj9rtWZY_uLNEwvX-QLZD6hNQQ/view?usp=sharing}{[PDF]}.
\bibitem[Joachims et~al.(2017)Joachims, Swaminathan, and
Schnabel]{Joachims+WSDM2017Unbiased}
T.~Joachims, A.~Swaminathan, and T.~Schnabel.
\newblock 2017.
\newblock Unbiased learning-to-rank with biased feedback.
\newblock In \emph{Proceedings of the 10th ACM International Conference on Web
Search and Data Mining}, pp. 781--789.
\newblock \doi{10.1145/3018661.3018699}.
\newblock
\href{https://drive.google.com/file/d/1KzzKBPiDmTSGUyLgHhDEA5ndqR5gRnRT/view?usp=sharing}{[PDF]}.
\bibitem[Kim and Croft(2009)]{KimCroft2009Retrieval}
J.~Kim and W.~B. Croft.
\newblock 2009.
\newblock Retrieval experiments using pseudo-desktop collections.
\newblock In \emph{Proceedings of the 18th ACM Conference on Information and
Knowledge Management}, pp. 1297--1306.
\newblock \doi{10.1145/1645953.1646117}.
\newblock
\href{https://drive.google.com/file/d/1E3_uZSibx1RY2pUe60UuUqLFoU7jJy3j/view?usp=sharing}{[PDF]}.
\bibitem[Kim and Croft(2010)]{KimCroft2010Ranking}
J.~Kim and W.~B. Croft.
\newblock 2010.
\newblock Ranking using multiple document types in desktop search.
\newblock In \emph{Proceedings of the 33rd International ACM SIGIR Conference
on Research and Development in Information Retrieval}, pp. 50--57.
\newblock \doi{10.1145/1835449.1835461}.
\newblock
\href{https://drive.google.com/file/d/1i6obuVR-b0aA59VmSPgqP4CTw3lAyvfq/view?usp=sharing}{[PDF]}.
\bibitem[Kim et~al.(2010)Kim, Bakalov, Smith, and Croft]{Kim+2010Building}
J.~Kim, A.~Bakalov, D.~A. Smith, and W.~B. Croft.
\newblock 2010.
\newblock Building a semantic representation for personal information.
\newblock In \emph{Proceedings of the 19th ACM International Conference on
Information and Knowledge Management}, pp. 1741--1744.
\newblock \doi{10.1145/1871437.1871718}.
\newblock
\href{https://drive.google.com/file/d/160tRYuhCZj9mtRAgcrAIGxaoEJNbrUkh/view?usp=sharing}{[PDF]}.
\bibitem[Kim et~al.(2011)Kim, Croft, Smith, and Bakalov]{Kim+2011Evaluating}
J.~Kim, W.~B. Croft, D.~A. Smith, and A.~Bakalov.
\newblock 2011.
\newblock Evaluating an associative browsing model for personal information.
\newblock In \emph{Proceedings of the 20th ACM International Conference on
Information and Knowledge Management}, pp. 647--652.
\newblock \doi{10.1145/2063576.2063672}.
\newblock
\href{https://drive.google.com/file/d/1xE8-BhIUOq9PrVdhEZexeoYqkI8dWp8h/view?usp=sharing}{[PDF]}.
\bibitem[Klimt and Yang(2004{\natexlab{a}})]{KlimtYangCEAS2004Introducing}
B.~Klimt and Y.~Yang.
\newblock 2004{\natexlab{a}}.
\newblock Introducing the {Enron} corpus.
\newblock In \emph{Proceedings of the 1st Conference on Email and Anti-Spam}.
\newblock
\href{https://drive.google.com/file/d/1SGxsqpjsj0NuGChp2MKiQX8rrLKPEVac/view?usp=sharing}{[PDF]}.
\bibitem[Klimt and Yang(2004{\natexlab{b}})]{KlimtYangECML2004Enron}
B.~Klimt and Y.~Yang.
\newblock 2004{\natexlab{b}}.
\newblock The {Enron} corpus: A new dataset for email classification research.
\newblock In \emph{Proceedings of the 15th European Conference on Machine
Learning}, volume 3201 of \emph{Lecture Notes in Computer Science}, pp.
217--226,. Springer.
\newblock \doi{10.1007/978-3-540-30115-8\_22}.
\newblock
\href{https://drive.google.com/file/d/1fAMN99li1ddLcZPlGhvwYaoO7dtn96nw/view?usp=sharing}{[PDF]}.
\bibitem[Kong et~al.(2020)Kong, Bendersky, Najork, Vargo, and
Colagrosso]{Kong+2020Learning}
W.~Kong, M.~Bendersky, M.~Najork, B.~Vargo, and M.~Colagrosso.
\newblock 2020.
\newblock Learning to cluster documents into workspaces using large scale
activity logs.
\newblock In \emph{Proceedings of the 26th ACM SIGKDD International Conference
on Knowledge Discovery \& Data Mining}, pp. 2416--2424.
\newblock \doi{10.1145/3394486.3403291}.
\newblock
\href{https://drive.google.com/file/d/1kk9ABp_NLwr627urSuE9TQGmGPqxPgCC/view?usp=sharing}{[PDF]}.
\bibitem[Kuzi et~al.(2017)Kuzi, Carmel, Libov, and Raviv]{Kuzi+al:2017}
S.~Kuzi, D.~Carmel, A.~Libov, and A.~Raviv.
\newblock 2017.
\newblock Query expansion for email search.
\newblock In \emph{Proceedings of the 40th International ACM SIGIR Conference
on Research and Development in Information Retrieval (SIGIR 2017)}, pp.
849--852. ACM.
\newblock ISBN 978-1-4503-5022-8.
\newblock \url{http://dx.doi.org/10.1145/3077136.3080660}.
\newblock \doi{10.1145/3077136.3080660}.
\bibitem[Lamming et~al.(1994)Lamming, Brown, Carter, Eldridge, Flynn, Louie,
Robinson, and Sellen]{Lamming+1994Prosthesis}
M.~Lamming, P.~Brown, K.~Carter, M.~Eldridge, M.~Flynn, G.~Louie, P.~Robinson,
and A.~Sellen.
\newblock 1994.
\newblock The design of a human memory prosthesis.
\newblock \emph{The Computer Journal}, 37(3): 153--163.
\newblock \doi{10.1093/comjnl/37.3.153}.
\newblock
\href{https://drive.google.com/file/d/1ccl1_SCR9EnnT0_aVJygnmt4yX6WKb5S/view?usp=sharing}{[PDF]}.
\bibitem[Lansdale and Edmonds(1992)]{LansdaleEdmonds1992Using}
M.~Lansdale and E.~Edmonds.
\newblock January 1992.
\newblock Using memory for events in the design of personal filing systems.
\newblock \emph{International Journal of Man-Machine Studies}, 36(1): 97--126.
\newblock \doi{10.1016/0020-7373(92)90054-O}.
\newblock
\href{https://drive.google.com/file/d/1hbabRfPhrVKBIoVVhgEPg6u0AXqaF0BB/view?usp=sharing}{[PDF]}.
\bibitem[Lansdale(1988)]{Lansdale1988Psychology}
M.~W. Lansdale.
\newblock March 1988.
\newblock The psychology of personal information management.
\newblock \emph{Applied Ergonomics}, 19(1): 55--66.
\newblock
\href{https://drive.google.com/file/d/1AYJUmvro7k-LryjY6yXicUO58wfB0u2i/view?usp=sharing}{[PDF]}.
\bibitem[Lewis and Knowles(1997)]{Lewis+1997Threading}
D.~D. Lewis and K.~A. Knowles.
\newblock March 1997.
\newblock Threading electronic mail: A preliminary case study.
\newblock \emph{Information Processing \& Management}, 33(2): 209--217.
\newblock \doi{10.1016/S0306-4573(96)00063-5}.
\newblock
\href{https://drive.google.com/file/d/1WqzRc2PX25Doh3QI2C0jZ8pnMNckGhfO/view?usp=sharing}{[PDF]}.
\bibitem[Li et~al.(2019{\natexlab{a}})Li, Zhang, Bendersky, Deng, Metzler, and
Najork]{Li+2019MultiView}
C.~Li, M.~Zhang, M.~Bendersky, H.~Deng, D.~Metzler, and M.~Najork.
\newblock 2019{\natexlab{a}}.
\newblock Multi-view embedding-based synonyms for email search.
\newblock In \emph{Proceedings of the 42nd International ACM SIGIR Conference
on Research and Development in Information Retrieval}, pp. 575--584.
\newblock \doi{10.1145/3331184.3331250}.
\newblock
\href{https://drive.google.com/file/d/1latAxZ1R47qNfWLXhfUbymPpOxuVdITf/view?usp=sharing}{[PDF]}.
\bibitem[Li et~al.(2019{\natexlab{b}})Li, Qin, Wang, and
Metzler]{Li+2019Combining}
P.~Li, Z.~Qin, X.~Wang, and D.~Metzler.
\newblock 2019{\natexlab{b}}.
\newblock Combining decision trees and neural networks for learning-to-rank in
personal search.
\newblock In \emph{Proceedings of the 25th ACM SIGKDD International Conference
on Knowledge Discovery \& Data Mining}, pp. 2032--2040.
\newblock \doi{10.1145/3292500.3330676}.
\newblock
\href{https://drive.google.com/file/d/1Ew1F7salRJRYB7W0LgLYzigrj4UaAEnT/view?usp=sharing}{[PDF]}.
\bibitem[Licklider(1960)]{Licklider1960Symbiosis}
J.~C.~R. Licklider.
\newblock March 1960.
\newblock Man-computer symbiosis.
\newblock \emph{IEEE Transactions on Human Factors in Electronics}, 1(1):
4--11.
\newblock \doi{10.1109/THFE2.1960.4503259}.
\newblock
\href{https://drive.google.com/file/d/14LKHkyRIQr7BhWxB5IcD8rLBy0qFGwSb/view?usp=sharing}{[PDF]}.
\bibitem[Liu(2011)]{Liu2011LTRforIR}
T.-Y. Liu.
\newblock 2011.
\newblock \emph{Learning to Rank for Information Retrieval}.
\newblock Springer.
\newblock ISBN 978-3-642-14266-6.
\newblock \doi{10.1007/978-3-642-14267-3}.
\newblock
\href{https://drive.google.com/file/d/1xxYDx8MadjvC5gaVJ2XTKcfRnTM6fVLQ/view?usp=sharing}{[PDF]}.
\bibitem[Macdonald and Ounis(2006)]{MacdonaldOunis2006Combining}
C.~Macdonald and I.~Ounis.
\newblock 2006.
\newblock Combining fields in known-item email search.
\newblock In \emph{Proceedings of the 29th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp.
675--676.
\newblock \doi{10.1145/1148170.1148312}.
\newblock
\href{https://drive.google.com/file/d/1aWkIq8RmXAEorDuKfQaeR_zQkDON_M9S/view?usp=sharing}{[PDF]}.
\bibitem[Mackenzie et~al.(2019)Mackenzie, Gupta, Qiao, Awadallah, and
Shokouhi]{Mackenzie+2019Exploring}
J.~Mackenzie, K.~Gupta, F.~Qiao, A.~H. Awadallah, and M.~Shokouhi.
\newblock 2019.
\newblock Exploring user behavior in email re-finding tasks.
\newblock In \emph{Proceedings of the 2019 World Wide Web Conference}, pp.
1245--1255.
\newblock \doi{10.1145/3308558.3313450}.
\newblock
\href{https://drive.google.com/file/d/1ElgXqcS4c-rDQmWgtRDf_ESxyapZw2nT/view?usp=sharing}{[PDF]}.
\bibitem[Malone(1983)]{Malone1983How}
T.~W. Malone.
\newblock January 1983.
\newblock How do people organize their desks? implications for the design of
office information systems.
\newblock \emph{ACM Transactions on Office Information Systems}, 1(1): 99--112.
\newblock \doi{10.1145/357423.357430}.
\newblock
\href{https://drive.google.com/file/d/15vqy2AYUatXOlVgEFTUbg3r78pJt4xl3/view?usp=sharing}{[PDF]}.
\bibitem[Manber and Wu(1994)]{ManberWu1994Glimpse}
U.~Manber and S.~Wu.
\newblock 1994.
\newblock Glimpse: A tool to search through entire file systems.
\newblock In \emph{Proceedings of the {USENIX} Winter 1994 Technical
Conference}, pp. 23--32.
\newblock
\href{https://drive.google.com/file/d/1igkh4DzgDj7_nXwsIqpRYUqRWpb4pT08/view?usp=sharing}{[PDF]}.
\bibitem[Mogul(1984)]{Mogul1984Representing}
J.~C. Mogul.
\newblock 1984.
\newblock Representing information about files.
\newblock In \emph{Proceedings of the 4th International Conference on
Distributed Computing Systems}, pp. 432--439.
\newblock
\href{https://drive.google.com/file/d/11OaDZEwOpqroYBx9nVJP6UUckp29y7tG/view?usp=sharing}{[PDF]}.
\bibitem[Myer and
Henderson(1975)]{MyerHenderson1975MessageTransmissionProtocol}
T.~H. Myer and D.~A. Henderson, April 1975.
\newblock Message transmission protocol.
\newblock Internet Engineering Task Force, Network Working Group, Request for
Comment 680.
\newblock
\href{https://drive.google.com/file/d/1he8815KVjG64xikstVL6L393uF7wj41-/view?usp=sharing}{[PDF]}.
\bibitem[Nardi et~al.(1994)Nardi, Anderson, and Erickson]{Nardi+1994Filing}
B.~Nardi, K.~Anderson, and T.~Erickson.
\newblock 1994.
\newblock Filing and finding computer files.
\newblock Technical Report \# 118, Apple Computer Inc.
\newblock
\href{https://drive.google.com/file/d/1C8AqtDR0vOFM0843IlnpeMHI0Qss45-k/view?usp=sharing}{[PDF]}.
\bibitem[Oard et~al.(2015)Oard, Webber, Kirsch, and
Golitsynskiy]{Oard+2015Avocado}
D.~Oard, W.~Webber, D.~A. Kirsch, and S.~Golitsynskiy.
\newblock 2015.
\newblock Avocado research email collection.
\newblock Philadelphia: Linguistic Data Consortium.
\newblock \doi{10.35111/wqt6-jg60}.
\bibitem[Postel(1975)]{Postel1975JunkMail}
J.~Postel, November 1975.
\newblock On the junk mail problem.
\newblock Internet Engineering Task Force, Network Working Group, Request for
Comments 706.
\newblock
\href{https://drive.google.com/file/d/17CtENLS2wrdSz2t-9ejNgiai33MRxp3b/view?usp=sharing}{[PDF]}.
\bibitem[Resnick(2001)]{Resnick2001InternetMessageFormat}
P.~Resnick, April 2001.
\newblock Internet message format.
\newblock Internet Engineering Task Force, Network Working Group, Request for
Comment 2822.
\newblock
\href{https://drive.google.com/file/d/1L-MSjd_QlDpc0SDce1oEd-DipXjpjXvd/view?usp=sharing}{[PDF]}.
\bibitem[Ringel et~al.(2003)Ringel, Cutrell, Dumais, and
Horvitz]{Ringel+2003Landmarks}
M.~Ringel, E.~Cutrell, S.~Dumais, and E.~Horvitz.
\newblock September 2003.
\newblock Milestones in time: The value of landmarks in retrieving information
from personal stores.
\newblock In \emph{Proceedings of the 9th IFIP TC13 International Conference on
Human-Computer Interaction}, pp. 184--191. IOS Press.
\newblock ISBN 978-1-58603-363-7.
\newblock
\href{https://drive.google.com/file/d/1Mo4O9XQ_aS1gYeQgr0ivW78nKDlPan3l/view?usp=sharing}{[PDF]}.
\bibitem[Robertson et~al.(2004)Robertson, Zaragoza, and
Taylor]{Robertson+CIKM2004Simple}
S.~Robertson, H.~Zaragoza, and M.~Taylor.
\newblock 2004.
\newblock Simple bm25 extension to multiple weighted fields.
\newblock In \emph{Proceedings of the 13th ACM International Conference on
Information and Knowledge Management}, pp. 42--49.
\newblock \doi{10.1145/1031171.1031181}.
\newblock
\href{https://drive.google.com/file/d/1jUvjAxzf0mHVINkBukbav-_tWp5Z9S-J/view?usp=sharing}{[PDF]}.
\bibitem[Robertson and Walker(1994)]{RobertsonWalker1994SomeSimple}
S.~E. Robertson and S.~Walker.
\newblock 1994.
\newblock Some simple effective approximations to the 2-poisson model for
probabilistic weighted retrieval.
\newblock In \emph{Proceedings of the 17th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval}, pp.
232--241.
\newblock
\href{https://drive.google.com/file/d/1XafBZVFzOGFwmufZUgqEnxdxSutbvGlt/view?usp=sharing}{[PDF]}.
\bibitem[Sahami et~al.(1998)Sahami, Dumais, Heckerman, and
Horvitz]{Sahami+1998Bayesian}
M.~Sahami, S.~Dumais, D.~Heckerman, and E.~Horvitz.
\newblock 1998.
\newblock A {Bayesian} approach to filtering junk e-mail.
\newblock In M.~Sahami, ed., \emph{Papers from the 1998 AAAI Workshop on
Learning for Text Categorization}, AAAI Technical Report WS-98-05, pp.
55--62. Association for the Advancement of Artificial Intelligence.
\newblock ISBN 978-1-57735-058-3.
\newblock
\href{https://drive.google.com/file/d/1_JJWWhvUKSNO1jK1WG5zeINWdqlH8zou/view?usp=sharing}{[PDF]}.
\bibitem[Sayed et~al.(2020)Sayed, Cox, Rivera, Christian-Lamb, Iqbal, Oard, and
Shilton]{Sayed+SIGIR2020TestCollection}
M.~F. Sayed, W.~Cox, J.~L. Rivera, C.~Christian-Lamb, M.~Iqbal, D.~W. Oard, and
K.~Shilton.
\newblock 2020.
\newblock A test collection for relevance and sensitivity.
\newblock In \emph{Proceedings of the 43rd International ACM SIGIR Conference
on Research and Development in Information Retrieval}, pp. 1605--1608.
\newblock \doi{10.1145/3397271.3401284}.
\newblock
\href{https://drive.google.com/file/d/10xSDbETZyTQT69YBxBepkq05LAouVpdV/view?usp=sharing}{[PDF]}.
\bibitem[Schenk et~al.(2006)Schenk, G{\"o}rlitz, and Staab]{Schenk+2006TagFS}
S.~Schenk, O.~G{\"o}rlitz, and S.~Staab.
\newblock 2006.
\newblock Tagfs: Bringing semantic metadata to the filesystem.
\newblock In \emph{Demos and Posters of the 3rd European Semantic Web
Conference}.
\newblock
\href{https://drive.google.com/file/d/1zn_xMEfCLXuozbMqb11UIMdWJZfGuzQC/view?usp=sharing}{[PDF]}.
\bibitem[Seltzer and Murphy(2009)]{SeltzerMurphy2009hFAD}
M.~Seltzer and N.~Murphy.
\newblock 2009.
\newblock Hierarchical file systems are dead.
\newblock In \emph{Proceedings of HotOS'09: 12th Workshop on Hot Topics in
Operating Systems}. {USENIX} Association.
\newblock
\href{https://drive.google.com/file/d/1Q8AtTHAX3eR5SjzuKwZuxQpbAi1tnIfE/view?usp=sharing}{[PDF]}.
\bibitem[Soules and Ganger(2005)]{SoulesGanger2005Connections}
C.~A.~N. Soules and G.~R. Ganger.
\newblock 2005.
\newblock Connections: Using context to enhance file search.
\newblock In \emph{Proceedings of the 20th ACM Symposium on Operating Systems
Principles}, pp. 119--132.
\newblock \doi{10.1145/1095810.1095822}.
\newblock
\href{https://drive.google.com/file/d/1N9IufEOucaojI11kMKw1FbC0zZvP07_O/view?usp=sharing}{[PDF]}.
\bibitem[Spertus(1997)]{Spertus1997Smokey}
E.~Spertus.
\newblock 1997.
\newblock Smokey: Automatic recognition of hostile messages.
\newblock In \emph{Proceedings of the 9th Conference on Innovative Applications
of Artificial Intelligence}, pp. 1058--1065.
\newblock
\href{https://drive.google.com/file/d/1b1LmKfHDetXDYXM0SZV-cG1kcBSst3Eo/view?usp=sharing}{[PDF]}.
\bibitem[Swanson(1972)]{Swanson1972Requirements}
D.~R. Swanson.
\newblock July 1972.
\newblock Requirements study for future catalogs.
\newblock \emph{Library Quarterly}, 42(3): 302--315.
\newblock \doi{10.1086/620045}.
\newblock
\href{https://drive.google.com/file/d/1howCD138BZvpQvcZBpMB173uqnYuPxaT/view?usp=sharing}{[PDF]}.
\bibitem[Tang et~al.(2008)Tang, Wilcox, Cerruti, Badenes, Nusser, and
Schoudt]{Tang+CHI2008TagIt}
J.~C. Tang, E.~Wilcox, J.~A. Cerruti, H.~Badenes, S.~Nusser, and J.~Schoudt.
\newblock 2008.
\newblock Tag-it, snag-it, or bag-it: Combining tags, threads, and folders in
e-mail.
\newblock In \emph{CHI '08 Extended Abstracts on Human Factors in Computing
Systems}, pp. 2179--2194.
\newblock \doi{10.1145/1358628.1358651}.
\newblock
\href{https://drive.google.com/file/d/1w_GUJSWXeY_hP6QmIq8o9M2iics37K3h/view?usp=sharing}{[PDF]}.
\bibitem[Tata et~al.(2017)Tata, Popescul, Najork, Colagrosso, Gibbons, Green,
Mah, Smith, Garg, Meyer, and Kan]{Tata+2017QuickAccess}
S.~Tata, A.~Popescul, M.~Najork, M.~Colagrosso, J.~Gibbons, A.~Green, A.~Mah,
M.~Smith, D.~Garg, C.~Meyer, and R.~Kan.
\newblock 2017.
\newblock Quick access: Building a smart experience for {Google Drive}.
\newblock In \emph{Proceedings of the 23rd ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining}, pp. 1643--1651.
\newblock \doi{10.1145/3097983.3098048}.
\newblock
\href{https://drive.google.com/file/d/1ZH0fMlGE7kvWQCdDCzoDXlrqsYesBzau/view?usp=sharing}{[PDF]}.
\bibitem[Tata et~al.(2019)Tata, Panait, Chen, and
Colagrosso]{Tata+2019ItemSuggest}
S.~Tata, V.~Panait, S.~J. Chen, and M.~Colagrosso.
\newblock 2019.
\newblock Itemsuggest: A data management platform for machine learned ranking
services.
\newblock In \emph{Proceedings of the 9th Biennial Conference on Innovative
Data Systems Research}.
\newblock
\href{https://drive.google.com/file/d/1pnQ3b06UW01_5cjUvlKaM3Ykz-3AFXdf/view?usp=sharing}{[PDF]}.
\bibitem[Vaughan(1968)]{Vaughan1968Effectiveness}
D.~K. Vaughan.
\newblock March 1968.
\newblock Effectiveness of book-memory data for conventional catalog retrieval.
\newblock In \emph{Requirements Study for Future Catalogs; Progress Report No.
2}, p.~53. Graduate Library School, University of Chicago.
\newblock
\href{https://drive.google.com/file/d/1q9SWQrRHC0ggLmmB_m-NY3D6jXiPmGj7/view?usp=sharing}{[PDF]}.
\bibitem[Wang et~al.(2016)Wang, Bendersky, Metzler, and
Najork]{Wang+SIGIR2016LTRwithBias}
X.~Wang, M.~Bendersky, D.~Metzler, and M.~Najork.
\newblock 2016.
\newblock Learning to rank with selection bias in personal search.
\newblock In \emph{Proceedings of the 39th International ACM SIGIR Conference
on Research and Development in Information Retrieval}, pp. 115--124.
\newblock \doi{10.1145/2911451.2911537}.
\newblock
\href{https://drive.google.com/file/d/1TUfq76jeLVoWGaiuEt2BgHyhvGB-BGfK/view?usp=sharing}{[PDF]}.
\bibitem[Wang et~al.(2018)Wang, Golbandi, Bendersky, Metzler, and
Najork]{Wang+WSDM2018PBEstimation}
X.~Wang, N.~Golbandi, M.~Bendersky, D.~Metzler, and M.~Najork.
\newblock 2018.
\newblock Position bias estimation for unbiased learning to rank in personal
search.
\newblock In \emph{Proceedings of the 11th ACM International Conference on Web
Search and Data Mining}, pp. 610--618.
\newblock \doi{10.1145/3159652.3159732}.
\newblock
\href{https://drive.google.com/file/d/1wGNN44m8rUOE71HWVbflxG3CVHsuJvYD/view?usp=sharing}{[PDF]}.
\bibitem[Wendt et~al.(2016)Wendt, Bendersky, Garcia-Pueyo, Josifovski, Miklos,
Krka, Saikia, Yang, Cartright, and Ravi]{Wendt+2016LabelProp}
J.~B. Wendt, M.~Bendersky, L.~Garcia-Pueyo, V.~Josifovski, B.~Miklos, I.~Krka,
A.~Saikia, J.~Yang, M.-A. Cartright, and S.~Ravi.
\newblock 2016.
\newblock Hierarchical label propagation and discovery for machine generated
email.
\newblock In \emph{Proceedings of the 9th ACM International Conference on Web
Search and Data Mining}, pp. 317--326.
\newblock \doi{10.1145/2835776.2835780}.
\newblock
\href{https://drive.google.com/file/d/1BYakltyuqDYWvQtrg7yqBzvLh2kyKoiM/view?usp=sharing}{[PDF]}.
\bibitem[Whittaker and Sidner(1996)]{WhittakerSidner1996EmailOverload}
S.~Whittaker and C.~Sidner.
\newblock 1996.
\newblock Email overload: exploring personal information management of email.
\newblock In \emph{Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems}, pp. 276--283.
\newblock \doi{10.1145/238386.238530}.
\newblock
\href{https://drive.google.com/file/d/18zmRRqSLLdP-IjU4dDr9-m11p5YBreg_/view?usp=sharing}{[PDF]}.
\bibitem[Whittaker et~al.(2011)Whittaker, Matthews, Cerruti, Badenes, and
Tang]{Whittaker+al2011Wasting}
S.~Whittaker, T.~Matthews, J.~Cerruti, H.~Badenes, and J.~Tang.
\newblock 2011.
\newblock Am {I} wasting my time organizing email? {A} study of email
refinding.
\newblock In \emph{Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems}, pp. 3449--3458.
\newblock \doi{10.1145/1978942.1979457}.
\newblock
\href{https://drive.google.com/file/d/1mAKKzkQ5frhCf6HI6lLKTlHwI2HxFRWI/view?usp=sharing}{[PDF]}.
\bibitem[Wu et~al.(2010)Wu, Burges, Svore, and Gao]{Wu+2010BoostingforIR}
Q.~Wu, C.~J.~C. Burges, K.~M. Svore, and J.~Gao.
\newblock June 2010.
\newblock Adapting boosting for information retrieval measures.
\newblock \emph{Information Retrieval}, 13: 254--270.
\newblock \doi{10.1007/s10791-009-9112-1}.
\newblock
\href{https://drive.google.com/file/d/1qYPoBjIJxb9dTLTPH4RoyA15_N-BH6gU/view?usp=sharing}{[PDF]}.
\bibitem[Xu et~al.(2020)Xu, Awadallah, Dumais, Omar, Popp, Rounthwaite, and
Jahanbakhsh]{Xu+WWW2020Understanding}
X.~Xu, A.~H. Awadallah, S.~T. Dumais, F.~Omar, B.~Popp, R.~Rounthwaite, and
F.~Jahanbakhsh.
\newblock 2020.
\newblock Understanding user behavior for document recommendation.
\newblock In \emph{Proceedings of The Web Conference 2020}, pp. 3012--3018.
\newblock \doi{10.1145/3366423.3380071}.
\newblock
\href{https://drive.google.com/file/d/15uC8lMXOQAeUMZcYLCF6ljO-P3dZJFBI/view?usp=sharing}{[PDF]}.
\bibitem[Yeh and Harnly(2006)]{YehHarnly2006Reassembly}
J.-Y. Yeh and A.~Harnly.
\newblock 2006.
\newblock Email thread reassembly using similarity matching.
\newblock In \emph{Proceedings of the 3rd Conference on Email and Anti-Spam}.
\newblock
\href{https://drive.google.com/file/d/18b2q-joVVkY3QtIdrQWJ0l9fSYhkaIEW/view?usp=sharing}{[PDF]}.
\bibitem[Zawinski(2002)]{Zawinski2002MessageThreading}
J.~Zawinski, 2002.
\newblock Message threading.
\newblock \url{https://www.jwz.org/doc/threading.html}.
\end{thebibliography}