AIDE: An Automated Sample-based Approach for Interactive Data Exploration
Extracted source text ยท arXiv 1510.08897
%\documentclass{sig-alternate-2013}
%\documentclass[9pt,technote]{IEEEtran}
\documentclass[10pt,journal,compsoc]{IEEEtran}
\newfont{\mycrnotice}{ptmr8t at 7pt}
\newfont{\myconfname}{ptmri8t at 7pt}
\let\crnotice\mycrnotice%
\let\confname\myconfname%
\clubpenalty=10000
\widowpenalty=10000
\newcommand{\Comment}[1]{\relax}
\newcommand{\cut}[1]{}
\newcommand{\remark}[1]{#1}
\newcommand{\todo}[1]{{\color{blue}[\textit{#1}]}}
\newcommand{\kiki}[1]{{\color{red}[\textit{#1}]}}
\usepackage{color}
\def\compactify{\itemsep=0pt \topsep=0pt \partopsep=0pt \parsep=0pt}
\let\latexusecounter=\usecounter
\newenvironment{CompactEnumerate}
{\def\usecounter{\compactify\latexusecounter}
\begin{enumerate}}
{\end{enumerate}\let\usecounter=\latexusecounter}
\newenvironment{CompactItemize}
{\def\usecounter{\compactify\latexusecounter}
\begin{itemize}}
{\end{itemize}\let\usecounter=\latexusecounter}
\usepackage{latexsym}
\usepackage{times}
\usepackage{graphics}
\usepackage{multirow}
\usepackage{color}
\usepackage{amssymb,amsmath,amsfonts}
\usepackage{wrapfig,epsfig}
\usepackage[center]{caption}
\usepackage{subfigure}
\begin{document}
\begin{sloppypar}
\title{AIDE: An Automated Sample-based Approach for Interactive Data Exploration }
\author{
%\alignauthor
Kyriaki Dimitriadou$^*$, Olga Papaemmanouil$^*$ and Yanlei Diao$^\dagger$\\
%\affaddr
{$^*$ Brandeis University, Waltham, MA, USA, $^{\dagger}$ University of Massachusetts, Amherst, MA, USA}\\
%\email
{$^*$\{kiki,olga\}@cs.brandeis.edu, $^{\dagger}$yanlei@cs.umass.edu}
}
\IEEEtitleabstractindextext{%
\begin{abstract}
In this paper, we argue that database systems be augmented with an automated data exploration service that methodically steers users through the data in a meaningful way. Such an automated system is crucial for deriving insights from complex datasets found in many big data applications such as scientific and healthcare applications as well as for reducing the human effort of data exploration. Towards this end, we present AIDE, an Automatic Interactive Data Exploration framework that assists users in discovering new interesting data patterns and eliminate expensive ad-hoc exploratory queries.
AIDE relies on a seamless integration of classification algorithms and data management optimization techniques that collectively strive to accurately learn the user interests based on his relevance feedback on strategically collected samples. We present a number of exploration techniques as well as optimizations that minimize the number of samples presented to the user while offering interactive performance. {AIDE can deliver highly accurate query predictions for very common conjunctive queries with small user effort while, given a reasonable number of samples, it can predict with high accuracy complex disjunctive queries. It provides interactive performance as it limits the user wait time per iteration of exploration to less than a few seconds. \cut{Our user study also shows that AIDE improves the current state-of-the-art of manual exploration by significantly reducing the user effort and total exploration time.}}
\end{abstract}
\begin{IEEEkeywords}
data exploration; data sampling; % NOT required for Proceedings
\end{IEEEkeywords}}
\maketitle
% A category with the (minimum) three required fields
%\category{H.2.4}{Systems}{Relational Databases}
\input{intro}
\input{system}
\input{exploration}
\input{misclassified}
\input{boundaries}
\input{optimizations}
\input{experiments}
\input{related}
\section{conclusions}\label{s:conclusions}
{Interactive Data Exploration (IDE) is a key ingredient of a diverse set of discovery-oriented applications, including ones from scientific computing and evidence-based medicine. In these applications, data discovery is a highly ad hoc interactive process where users execute numerous exploration queries using varying predicates aiming to balance the trade-off between collecting all relevant information and reducing the size of returned data. Therefore, there is a strong need to support these human-in-the-loop applications by assisting their navigation in the data space.}
In this paper, we introduce AIDE, an \emph{Automatic Interactive Data Exploration} system, that iteratively steers the user towards interesting data areas and ``predicts'' a query that retrieves her objects of interest. Our approach leverages relevance feedback on database samples to model user interests and strategically collects more samples to refine the model while minimizing the user effort. AIDE integrates machine learning and data management techniques to provide effective data exploration results (matching the user's interests with high accuracy) as well as high interactive performance. It delivers highly accurate query predictions for very common conjunctive queries with very small user effort while, given a reasonable number of samples, it can predict with high accuracy complex conjunctive queries. Furthermore, it provides interactive performance by limiting the user wait time per iteration to less than a few seconds in average. Our user study indicates that AIDE is a practical exploration framework as it significantly reduces the user effort and the total exploration time compared with the current state-of-the-art approach of manual exploration.
%In future work, we plan to improve our system by extending the set of target queries beyond conjunctive queries and linear predicates, for instance, to include non-linear predicates, as well as further optimizing the database backend for exploration query workloads using materialized views and multi-query optimization.
%\section{Acknowledgments}
%
%This work was funded by in part by NSF under Grant IIS-1049974, IIS-1253196 and IIS-1218524.
%
%
%\scriptsize
\bibliographystyle{IEEEtran}
\bibliography{qsteering}
\vspace {-14mm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{kikiPhoto}}]{Kyriaki Dimitriadou}
Kyriaki Dimitriadou is a PhD student in Computer Science at Brandeis University. She holds an MA in Computer Science from Brandeis and a BA in Applied Informatics from the University of Macedonia, Greece. Her research interests are in database systems with a focus on data exploration.
\end{IEEEbiography}
\vspace {-20mm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{olgaPhoto}}]{Olga Papaemmanouil}
Olga Papaemmanouil is an Assistant Professor of Computer Science at Brandeis University since 2009. She received her undergraduate degree in Computer Engineering and Informatics from the University of Patras, Greece,, a M.Sc. in Information Systems from the Athens University of Economics and Business and completed her PhD at Brown University in 2008. Her research interests are in data management and distributed systems with focus on data streams, query performance, cloud databases and data exploration. She is the recipient of an NSF CAREER Award (2013) and a Paris Kanellakis Fellow (2002).
\end{IEEEbiography}
\vspace {-14mm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{yanleiPhoto}}]{Yanlei Diao}
received the Bachelors degree in computer science from Fudan University in China in 1998, the M.Phil degree from the Hong Kong University of Science and Technology in 2000, and the Ph.D. degree in computer science from the University of California, Berkeley in 2005. She is an Assistant Professor in the Department of Computer Science at the University of Massachusetts. Her
research interests are in information architectures and data management systems, with a focus on data
streams, sensor data management, uncertain data management, large-scale data analysis, and flash memory databases. Dr. Diao
has been the recipient of the NSF CAREER Award, the
finalist for the Microsoft Research New Faculty Fellowship, and the recipient
of the IBM Scalable Innovation Faculty Award.
%%She spoke at the Distinguished
%Faculty Lecture Series at the University of Texas at Austin in December
%2005.
Her PhD dissertation won the 2006 ACM-SIGMOD Dissertation Award
Honorable Mention.
%Yanlei Diao is an Associate Professor of Computer Science at the University of Massachusetts Amherst. Her research interests are in information architectures and data management systems, with a focus on big data analytics, scientific analytics, data streams, uncertain data management, and RFID and sensor data management. She received her PhD in Computer Science from the University of California, Berkeley in 2005, her M.S. in Computer Science from the Hong Kong University of Science and Technology in 2000, and her B.S. in Computer Science from Fudan University in 1998.
\end{IEEEbiography}
\end{sloppypar}
\end{document}
\vfill\eject
\section{Introduction}
Traditional data management systems assume that when users pose a query they a) have good knowledge of the schema, meaning and contents of the database and b) they are certain that this particular query is the one they wanted to pose. In short, traditional DBMSs are designed for applications in which the users know what they are looking for. \cut{The above assumption is valid for numerous past and present applications.} However, as data are being collected and stored at an unprecedented rate, we are building more dynamic data-driven applications where this assumption is not always true. \cut{Indeed, managing an employee or an inventory database is a drastically different setting than looking for interesting patterns over a scientific database}
\emph{Interactive data exploration} (IDE) is one such example. In these applications, users are trying to make sense of the underlying data space by experimenting with queries, backtracking on the basis of query results and rewriting their queries aiming to discover interesting data objects. IDE often incorporates ``human-in-the-loop'' and it is fundamentally a long-running, multi-step process with the user's interests specified in imprecise terms.
%\cut{Traditional DBMSs are designed for applications in which the user queries to be asked are already well understood. }
%
%There is, however, a class of \emph{interactive data exploration} (IDE) applications, in
%which users are trying to make sense of the underlying data space by experimenting with queries, backtracking on the basis of query results and rewriting their queries aiming to discover interesting data objects. IDE often incorporates ``human-in-the-loop'' and it is fundamentally a long-running, multi-step process with the user data interests often specified in imprecise terms.
One application of IDE can be found in the domain of evidence-based medicine (EBM). Such applications often involve the generation of systematic reviews, a comprehensive assessment of the totality of evidence that addresses a well-defined question, such as the effect on mortality of giving versus not giving drug A within three hours of a symptom B. While a content expert can judge whether a given clinical trial is of interest or not (e.g., by reviewing parameter values such as disease, patient age, etc.), he often does not have a priori knowledge of the exact attributes that should be used to formulate a query to collect all relevant clinical trials. Therefore the user relies on an ad hoc process that includes three steps: 1) processing numerous selection queries with iteratively varying selection predicates, 2) reviewing returned objects (i.e., trials) and classifying them to relevant and irrelevant, and 3) adjusting accordingly the selection query for the next iteration. The goal here is to discover the selection predicates that balances the trade-off between collecting all relevant objects and reducing the size of returned results. These ``manual'' explorations are typically labor-intensive: they may take days to weeks to complete since users need to examine thousands of objects.
Scientific applications, such as ones analysing astrophysical surveys (e.g.,~\cite{LSST, SloanSky}), also suffer from similar situations. Consider an astronomer looking for interesting patterns over a scientific database: they do not know what they are looking for, they only wish to find interesting patterns; they will know that something is interesting only after they find it. In this setting, there are no clear indications about how the astronomers should formulate their queries. Instead, they may want to navigate through a subspace of the data set (e.g., a region of the sky) to find objects of interest, or may want to see a few samples, provide yes/no feedback, and expect the system to find more similar objects.
\cut{Other IDE applications include but are not limited to financial analysis and genomics.}
To address the needs of IDE applications, we propose an \emph{Automatic Interactive Data Exploration (AIDE)} framework that automatically discovers data relevant to her interest. Our approach unifies the three IDE steps---query formulation, query processing and result reviewing---into a single automatic process, significantly reducing the user's exploration effort and the overall exploration time. In particular, an AIDE user engages in a ``conversation'' with the system indicating her interests, while in the background the system builds a user model that predicts data matching these interests.
%\todo{formulates and processes queries that collect data matching the user interest}.
%To address the needs of IDE applications, we propose an \emph{automatic interactive data exploration (AIDE)} framework that automatically ``steers'' the user towards data areas relevant to his interest. Our approach integrates the three IDE steps---query formulation, query processing, and result reviewing---into a single automatic process, significantly reducing the user effort and the overall exploration time. In AIDE, the user engages in a ``conversation'' with the system indicating his interests, while in the background the system automatically \todo{formulates and processes queries that collect data matching the user interest}.
AIDE offers an iterative exploration model: in each iteration the user is prompted to provide her feedback on a {set} of sample objects as relevant or irrelevant to her exploration task. Based on her feedback, AIDE generates the user's exploration profile, i.e., a user model that classifies database objects as relevant or irrelevant. AIDE leverages this model to explore further the data space, identify strategic sampling areas and collect new samples for the next iteration. These samples are presented to the user and her new feedback is incorporated into the user model. This iterative process aims to generate a user model that identifies all relevant objects while eliminating the misclassification of irrelevant ones.
%This user model is then used to identify strategic sampling areas of the exploration data space for collecting a new set of sample objects for the next iteration. In the background, AIDE leverages the user profile to \emph{automatically} generate data extraction queries that retrieve more objects relevant to the user's IDE task while minimizing the retrieval of irrelevant ones. %This process iterates until the user explicitly terminates the exploration (e.g., when he is satisfied with the relevant objects returned).
%
AIDE's model raises new challenges. First, AIDE operates on the unlabeled space of the whole data space that the user aims to explore. To offer effective exploration results (i.e., accurately predict the user's interests) it has to decide and retrieve in an \emph{online} fashion the example objects to be extracted and labeled by the user. Second, to achieve desirable interactive experience for the user, AIDE needs not only to provide accurate results, but also to minimize the number of samples presented to the user (which determines the amount of user effort) as well as to reduce the sampling and space exploration overhead (which determines the user's wait time in each iteration).
These challenges cannot be addressed by existing machine learning techniques. Classification algorithms (e.g.,~\cite{breinman84}) can be leveraged to build the user model and the information retrieval community offers solutions on incrementally incorporating relevance feedback in these models (e.g.,~\cite{zhou03}). However, these approaches operate under the assumption that the sample set shown to the user is either known a priori or, in the case of online classification, it is provided incrementally by a different party. In other words, classification algorithms do not deal with \emph{which} data samples to show to the user, which is one of the main research challenges for AIDE.
Active learning systems~\cite{Settles10activelearning} also extract unlabeled samples to be labeled by a user and the goal is to achieve high accuracy using as few labeled samples as possible, therefore minimizing the user's labeling effort. In particular, pool-based sampling techniques selectively draw samples from a large pool of unlabeled data. However, these solutions exhaustively examine \emph{all} unlabeled objects in the pool in order to identify the best samples to show to the user based on some informativeness measure~\cite{roy2001}. Therefore, they implicitly assume negligible sample acquisition costs and hence cannot offer interactive performance on big data sets as expected by IDE applications. In either case, model learning and sample acquisition are decoupled, with the active learning algorithms not addressing the challenge of \emph{how} to minimize the cost of sample acquisition.
%While AIDE follows an active learning approach, existing approaches are not directly applicable on big data sets due to two reasons. First these techniques are domain specific (e.g., document ranking~\cite{ruthven03}, image retrieval~\cite{Panda:2006:ALV}, etc.) and secondly they exhaustively examine \emph{all} unlabeled objects in the data set in order to identify the best samples to show to the user~\cite{roy2001}. Therefore, they implicitly assume negligible sample acquisition costs and hence cannot offer interactive performance on big data sets as expected by IDE applications. In either case, model learning and sample acquisition are decoupled, with the active learning algorithms not addressing the challenge of \emph{how} to minimize the cost of sample acquisition.
%
%Active learning and particularly pool-based sampling techniques~\cite{Lewis1994} also assume an iterative learning model and in each iteration a user is asked to label of a single object out of a pool of unlabeled objects and a new classifier is built. These solutions identify in each iteration the ``best'' sample to show to the user such as the classification accuracy is build using as few samples as possible. While AIDE follows an active learning approach, existing approaches are not directly applicable on big data sets due to two reasons. First these techniques are domain specific (e.g., document ranking~\cite{ruthven03}, image retrieval~\cite{Panda:2006:ALV}, etc.) and secondly they exhaustively examine \emph{all} unlabeled objects in the data set in order to identify the best samples to show to the user~\cite{roy2001}. Therefore, they implicitly assume negligible sample acquisition costs and hence cannot offer interactive performance on big data sets as expected by IDE applications. In either case, model learning and sample acquisition are decoupled, with the active learning algorithms not addressing the challenge of \emph{how} to minimize the cost of sample acquisition.
%\cut{Furthermore, the active learning community has proposed solutions that identify the best sample to maximize the accuracy of the model while minimizing the number of samples shown to the user. However, these techniques are domain specific (e.g., document ranking~\cite{ruthven03}, image retrieval~\cite{Panda:2006:ALV}, etc.) and they exhaustively examine \emph{all} objects in the data set in order to identify the best samples to show to the user~\cite{roy2001}. Therefore, they implicitly assume negligible sample acquisition costs and hence cannot offer interactive performance on big data sets as expected by IDE applications. In either case, model learning and sample acquisition are decoupled, with the active learning algorithms not addressing the challenge of \emph{how} to minimize the cost of sample acquisition.}
%\todo{Active learning and particularly pool-based sampling has proposed various sampling strategies for selecting the best samples for labeling out of a pool of unlabeled data. These techniques identify the best next sample to show to the user that maximizes the accuracy of the model aiming the number of samples shown to the user. However, these techniques are domain specific (e.g., document ranking~\cite{ruthven03}, image retrieval~\cite{Panda:2006:ALV}, etc.) and they exhaustively examine \emph{all} objects in the data set in order to identify the best samples to show to the user~\cite{roy2001}. Therefore, they implicitly assume negligible sample acquisition costs and hence cannot offer interactive performance on big data sets as expected by IDE applications. In either case, model learning and sample acquisition are decoupled, with the active learning algorithms not addressing the challenge of \emph{how} to minimize the cost of sample acquisition.}
To address the above challenges, AIDE closely \emph{integrates} classification model learning (from existing labeled samples) and effective data exploration and sample acquisition (deciding best data areas to sample). Specifically, our techniques leverage the classification properties of decision tree learning to identify promising data exploration areas from which new samples are extracted, as well as to minimize the number of samples shown to the user. These techniques aim to predict linear patterns of user interests, i.e., we assume relevant objects are clustered in multi-dimensional hyper-rectangles. These interests can be expressed as range queries with disjunctive and/or conjunctive predicates.
This paper extends our previous our previous work on automatic data exploration~\cite{aide_sigmod14,aide_vldb15}. Specifically, we extended AIDE with a number of performance optimizations that are designed to reduce the total exploration overhead. \cut{This overhead includes the number the number of samples labeled by the user, the convergence rate to an accurate user model as well as the user's wait time.} Specifically, we introduce: (a) a skew-aware exploration technique that deals with both uniform and skewed data spaces as well as user interests that lie on either the sparse or dense parts of the distribution, (b) a probabilistic sampling strategy for selecting the most informative sample to present to the user; the strategy is designed to reduce the user's exploration effort and (c) an extended relevance feedback model that allows users to annotate ``similar'' (rather than only relevant/irrelevant) samples, allowing us to further reduce the total exploration time. We also include a new set of experimental results that demonstrate the effectiveness and efficiency of our new exploration techniques.
%
%\cut{
%This includes the number the number of samples labeled by the user, the convergence rate to an accurate user model as well as the user's wait time. Our optimizations include an extended feedback model for indicating objects which although irrelevant to the final exploration task they include certain interesting attribute values. This feedback allows AIDE to focus its exploration on possibly relevant dimensions of the exploration space. AIDE also improves its convergence rate to an accurate model based on a probabilistic technique that identifies the most informative samples to show to the user as well as based on adaptive sampling techniques that adjust the samples extracted from a particular data areas on the skewness of the underlying data space.
%}
The specific contributions of this work are the following:
\begin{CompactEnumerate}
\item We introduce AIDE, a novel, automatic data exploration framework, that navigates the user throughout the data space he wishes to explore. AIDE relies on the user's feedback on example objects to generate a user model that predicts data relevant to the user. It employs a unique combination of machine learning, data exploration, and sample acquisition techniques to deliver highly accurate predictions of linear patterns of user interests with interactive performance. Our data exploration techniques leverage the properties of classification models to identify \emph{single} objects of interest, expand them to more accurate \emph{areas of interests}, and progressively refine the prediction of these areas. Our techniques address the trade-off between quality of results (i.e., accuracy) and efficiency (i.e., the total {\em exploration time} which includes the total sample reviewing time and wait time by the user).
%Each technique strategically identifies the most effective sampling area and the number of samples for the following iteration of the exploration.
\item We introduce new optimizations that address the presence of skew in the underlying exploration space as well as a novel probabilistic approach for identifying the most informative sample set to show to the user. We also include an extended feedback model based on which the user can also indicate similar but not necessarily relevant objects. This new model allows AIDE to focus its exploration on on certain promising domain ranges reducing significantly the user's labeling effort.
\item We evaluated our implementation of AIDE using the SDSS database~\cite{SloanSky} and a user study. { Our results indicate that AIDE and its novel optimizations are highly effective and efficient. AIDE can predict common conjunctive queries with a small number of samples, while given an acceptable number of labeled samples it predicts highly complex disjunctive queries with high accuracy. AIDE also offers interactive performance as the user wait time per iteration is less than a few seconds in average. Our user study revealed that AIDE can reduce the user's labeling effort by up 87\%, with an average of 66\% reduction. When including the sample reviewing time, it reduced the total exploration time by 47\% in average.% compared with the state-of-the-art in IDE where users examine 1000s of objects to formulate complex selection queries.
}
\end{CompactEnumerate}
The rest of the paper is organized as follows. Section~\ref{s:system} outlines the AIDE framework and Section~\ref{s:exploration_overview} describes the phases of our data exploration approach. Section~\ref{s:optimizations} discusses the new performance optimizations we introduce in AIDE. Section~\ref{s:experiments} presents our experimental results. Section~\ref{s:related} discusses the related work and we conclude in Section~\ref{s:conclusions}.
\section{AIDE Framework Overview}\label{s:system}
In this section we introduce our system model, describe the classification algorithms we use and provide a definition of our exploration problem.
\begin{figure}
\centering
\includegraphics[totalheight=3.1in, angle=90] {autoframework}
\caption{\small{Automated Interactive Data Exploration Framework.}} \label{f:autoframework}
\vspace{-4mm}
\end{figure}
\subsection{System Model}\label{s:systemmodel}
% revision
The workflow of our exploration framework is depicted in Figure~\ref{f:autoframework}. AIDE presents to the user sample database objects and requests her feedback on their relevance to her exploration task, i.e., characterize them as relevant or not. For example, in the domain of evidence-based medicine, users are shown sample clinical trials and they are asked to review their abstract and their attributes (e.g., year, outcome, patience age, medication dosage, etc) and label each sample trial as interesting or not. AIDE allows also the user to annotate samples that are similar (in some attribute) but not match exactly her interest, by marking them as ``similar'' samples. Finally, the user can modify her feedback on previously seen samples, however this could potentially prolong the exploration process.
The iterative steering process starts when the user provides her feedback by labeling samples are relevant or not. The relevant and irrelevant samples are used to train a binary classification model that characterizes the user's interest, e.g., it predicts which clinical trials are relevant to the user based on the feedback collected so far ({\em Data Classification}) \footnote{"Similar" samples are not included in the training of the user model. In Section~\ref{s:maybe} we discuss in detail how we leverage these samples.}. This model may use any subset of the object's attributes to characterize user interests. However, domain experts could leverage their domain knowledge to restrict the attribute set on which the exploration is performed. For instance, one could request an exploration only on the attributes \textit{dosage} and \emph{age}. In this case, relevant trials will be characterized on a subset of these attributes (e.g., relevant trials have dosage $>$45mg).
In each iteration, more samples (e.g., records of clinical trials) are extracted and presented to the user for feedback. AIDE leverages the current user model as well as the user's feedback so far to identify promising sampling areas ({\em Space Exploration}) and retrieve the next sample set from the database (\emph{Sample Extraction}). New labeled objects are incorporated with the already labeled sample set and a new classification model is built. The steering process is completed when the user terminates the process explicitly, e.g., when reaching a satisfactory set of relevant objects or when she does not wish to label more samples. Optionally, AIDE ``translates'' the classification model into a query expression (\emph{Query Formulation}). This query will retrieve objects characterized as relevant by the user model ({\em Data Extraction Query}).
AIDE strives to converge to a model that captures the user interest, i.e., eliminating irrelevant objects while identifying a large fraction of relevant ones. Each round refines the user model by exploring further the data space. The user decides on the effort he is willing to invest (i.e., number of samples he labels) while AIDE leverages his feedback to strategically sample the exploration space, i.e., collect samples that improve the accuracy of the classification model. The more effort invested in this iterative process, the more effective the user model will be. \cut{However, higher numbers of samples increase the sample extraction and processing time, i.e., the user {\em wait time} between iterations.}
\begin{figure}[t]
\centering
\includegraphics[totalheight=2.3in, angle=-90] {dtree2}
\caption{{\small {An example decision tree.}}}
\label{f:dtree}
\vspace{-4mm}
\end{figure}
\subsection{Data Classification \& Query Formulation}\label{s:system_classifier}
AIDE relies on decision tree classifiers to identify linear patterns of user interests, i.e., relevant objects clustered in multi-dimensional hyper-rectangles. Decision tree learning~\cite{breinman84} produces classification models that predict the class of an unclassified object based on labeled training data. The major advantage of decision trees is that they provide easy to interpret models that clearly describe the features characterizing each data class. Furthermore, they perform well with large data and the decision conditions of the model can be easily translated to simple boolean expressions. This feature is important since it allows us to map decision trees to queries that retrieve the relevant data objects.
Finally, decision trees can handle both numerical and categorical data. This allows AIDE to operate on both data types assuming a distance function is provided to calculate the similarity between two data objects. Measuring the similarity between two objects is a requirement of the space exploration step. AIDE treats the similarity computation as an orthogonal step and can make use of any distance measure. For continuous data sets (e.g., numerical), the Euclidean distance can be used. Computing similarity between categorical data is more challenging due to the fact that there is no specific ordering between categorical values. However, a number of similarity measures have been proposed in the literature for categorical data, and AIDE can be extended in a straightforward way to incorporate them.
{\bf Query Formulation} Let us assume a decision tree classifier that predicts relevant and irrelevant clinical trials objects based on the attributes {\tt age} and {\tt dosage} (Figure~\ref{f:dtree}). This tree provides predicates that characterize the relevant class and predicates that describe the irrelevant class. In Figure~\ref{f:dtree}, the relevant class is described by the predicates $(age \le 20 \wedge 10 < dosage \le 15)$ and $(20< age \le 40 \wedge 0 \le dosage \le 10)$, while the irrelevant class is characterized by the predicates $(age \le 20 \wedge dosage \le 10)$ and $(20<age \le 40 \wedge dosage > 10)$ (here we ignore the predicates that refer to values outside attribute domains, such as $age>40$, $age<0$, $dosage<0$ and $dosage>15$).
Given the decision tree in Figure~\ref{f:dtree} it is straightforward to formulate the extraction query for the relevant objects ({\it select * from table where (age $\le$ 20 and dosage $>$10 and dosage $\le$ 15) or (age $>$ 20 and age $\le$ 40 and dosage $\ge$ 0 and dosage $\ge$ 10)}).
\subsection{Problem Definition}\label{s:problemdef}
%revision
Given a database schema $\mathcal{D}$,
let us assume the user has decided to focus his exploration on $d$ attributes, where these $d$ attributes may include both attributes relevant and those irrelevant to the final query that represents the true user interest.
Each exploration task is then performed in a $d$-dimensional space of $T$ tuples where each tuple represents an object characterized by $d$ attributes. For a given user, our exploration space is divided to the relevant object set $T^r$ and irrelevant set $T^{nr}$. Since the user's interests are unknown to AIDE, the sets $T^r$ and $T^{nr}$ are also unknown in advance.
AIDE aims to generate a model that predicts these two sets, i.e., classifies a tuple in $T$ as relevant or irrelevant. To achieve that, it iteratively trains a decision tree classifier. Specifically, in each iteration $i$, a sample tuple set $S_i \subseteq T$ is shown to the user and his relevance feedback assigns these samples to two data classes, the relevant object class $D^r \subseteq T^r$, and the irrelevant one, $D^{nr} \subseteq T^{nr}$. Based on the samples assigned to these classes up to the $i$-th iteration, a new decision tree classifier $C_i$ is generated. This classifier corresponds to a predicate set ${P_i^r \bigcup P_i^{nr} }$, where the predicates ${P_i^r}$ characterize the relevant class and predicates ${P_i^{nr}}$ describe the irrelevant one.
We measure AIDE's effectiveness (aka accuracy of a classification model) by evaluating the $F$-measure, the harmonic mean between precision and recall.\footnote{Here, if $tp$ are the true positives results of the classifier (i.e., correct classifications as relevant), $fp$ are the false positives (i.e., irrelevant data classified as relevant) and $fn$ are the false negatives (i.e., relevant data classified as irrelevant), we define the precision of our classifier as $precision=\frac{tp}{ tp + fp}$ and the recall as $recall=\frac{tp}{tp+fn}$.}
Our goal is to maximize the $F$-measure of the final decision tree $C$ on the total data space $T$, defined as:
$F(T)=\frac{2 \times precision(T) \times recall(T)}{precision(T)+recall(T)}
$.
%We define the total number of samples shown to the user up to the $i$-the iteration is $T_i= T_{i-1} \bigcup S_i$
The perfect precision value of 1.0 means that every object characterized as relevant by the decision tree is indeed relevant, while a good recall ensures that our final query can retrieve a good percentage of the relevant to the user objects.
%Achieving a good precision will eliminate irrelevant objects from , whereas the higher the recall value the higher the percentage of the actual relevant data the decision tree can capture.
\section{Space Exploration Techniques}\label{s:exploration_overview}
Our main research focus is on optimizing the effectiveness of the exploration (i.e., the accuracy of the final user model) while offering interactive experience to the user. To address that AIDE strives to improve on a number of efficiency factors, including the number of samples presented to the user and the number of sample extraction queries processed in the backend. In this section, we introduce our main exploration techniques that tackle these goals.
AIDE assumes that user interests are captured by \emph{range queries}, i.e., relevant objects are clustered in one or more areas in the data space. Therefore, our goal is to generate a user model that predicts \emph{relevant areas}. The user model can then be translated to a range query that selects either a single multi-dimensional relevant area (conjunctive query) or multiple ones (disjunctive query).
AIDE incorporates three exploration phases. First, we focus on collecting samples from yet unexplored areas and identifying single relevant objects (\emph{Relevant Object Discovery}). Next, we strive to leverage single relevant objects to generate a user model that identifies relevant \emph{areas} (\emph{Misclassified Exploitation}). Finally, given a set of discovered relevant areas, we gradually refine their boundaries (\emph{Boundary Exploitation}). In each iteration $i$, these three phases define the new sample set we will present to the user. Specifically, if $T^i_{d}$, $T^i_{m}$ and $T^i_{b}$ samples will be selected by the object discovery, the misclassified and the boundary exploitation phase, then the user is presented with $S_i = T^i_{d} + T^i_{m} + T^i_{b}$ samples. %Next, we discuss in detail each phase.
Our three exploration phases are designed to collectively increase the accuracy of the exploration results. Given a set of relevant objects from the object discovery step, the misclassified exploitation increases the number of relevant samples in our training set while reducing the misclassified objects (specifically false negatives). Hence, this step improves both the recall and the precision parameters of the $F$-measure metric. The boundary exploitation further refines the characterization of the already discovered relevant areas. Therefore, it discovers more relevant objects and eliminates misclassified ones, leading also to higher recall and precision. Next, we discuss in detail each phase. More details about these exploration techniques can be found in~\cite{aide_sigmod14}.
%
%\cut{
%One interesting artifact of using the $F$-measure %as our effectiveness measure,
%is that failing to discover a relevant area reduces our accuracy dramatically more than failing to precisely characterize the boundaries of that relevant area. Therefore, if we assume that the user's relevant objects are clustered in $M>1$ areas, discovering (with slightly imprecise boundaries) all $M$ relevant areas will lead to higher accuracy that predicting less than $M$ areas but with highly accurate boundaries. Therefore, our exploration techniques place a limit on the samples collected by the boundary exploitation phase, aiming to utilize the user effort more on the other two more effective phases.
%}
\subsection{Relevant Object Discovery}\label{s:exploration}
Our first exploration phase aims to discover relevant objects by showing to the user samples from diverse data areas.
To maximize the coverage of the exploration space we follow a well-structured approach that allows us to
(1) ensure that the exploration space is explored widely, (2) keep track of the already explored sub-areas, and (3) explore different data areas in different granularity.
Our approach operates on a set of {\em hierarchical exploration grids}. Given an exploration task on $d$ attributes, we define the \emph{exploration space} to be the $d$-dimensional data area defined by the \emph{domain} of these attributes. AIDE creates off-line a set of grids and each grid divides the exploration space into $d$-dimensional cells with equal width in each dimension. We refer to each grid as an \emph{exploration level} and each level has a different granularity, i.e., cells of different width. The lower the exploration level the more fine-grained the grid cells (i.e., smaller cells) it includes and therefore moving between levels allows us to ``zoom in/out'' into specific areas as needed.
{\bf Exploration Level Construction} To generate an exploration level on a $d$-dimensional exploration space we divide each normalized attribute domain\footnote{We normalize each domain to be between [0,100]. This allow us to reason about the distance between values uniformly across domains. Operating on actual domains will not affect the design of our framework or our results.} into $\beta$ equal width ranges, effectively creating $\beta^d$ grid cells. The $\beta$ parameter defines the granularity of the specific exploration level. A higher number leads to more grid cells of smaller width per dimension and the use of more samples to explore \emph{all} grid cells for fine-grained search for relevant objects.
%Higher $\beta$ values allow for a finer-grained search for objects of interest. However, these more detailed exploration levels require more samples in order to explore \emph{all} grid cells uniformly.
Each cell in our grid covers a certain range of attribute values for each of the $d$ exploration attributes. Therefore, each cell includes a set of unique attribute value combinations. %($v_1, v_2, .., v_d$).
Each combination can be mapped to a set of data objects that match these attribute values. Figure~\ref{f:grid} shows a two-level 2-dimensional hierarchical grid (we show the second level only for the top right grid cell). \cut{Next we present the general algorithm for this phase which explores uniformly all cells, collects one object from each cell and shows it to the user for feedback.}
\begin{figure}[t]
\centering
\includegraphics[totalheight=1.7in, angle=-90] {grid}
\caption{{\small Grid-based object discovery example in 2-D space.}}
\label{f:grid}
\vspace{-5mm}
\end{figure}
{\bf Discovery Phase} Our exploration retrieves one data object from each non-empty cell. The goal is to uniformly spread the samples we collect across the grid cells to ensure the highest coverage of the exploration space. We achieve that by retrieving objects that are on or close to the center of each cell. Since the exploration levels are defined on the normalized domains of $d$ attributes and we split each domain to $\beta$ equal width ranges, each cell covers a range of $\delta=100/\beta$ of the domain for each attribute. For each cell, we identify the ``virtual'' center and we retrieve a single random object within distance $\gamma<\delta/2$ along each dimension from this center (see Figure~\ref{f:grid}). This approach guarantees that at each exploration level the retrieved samples are within $\delta\pm (2\times \gamma)$ normalized distance from each other.
The sampling distance around the center of each cell, i.e., the $\gamma$ parameter, is adjusted based on the density of the cell. Sparse cells use a higher $\gamma$ value than dense ones to increase their sampling areas and hence improve the probability of retrieving a sample from the grid cell. This also reduces ineffective zoom in operations into the next level for that cell which may happen if no object is retrieved in the current exploration level. %and reduce the effort of the user %(since zoom in operations require the user to label objects from more cells).
%
In the each iteration we focus on a specific exploration level (starting frm the highest one) and we show to the user one object for each cell. If no relevant object is retrieved from one cell, we can safely infer that the \emph{whole} grid cell is not included in any relevant area. However, sub-areas of the grid could partially overlap with some relevant areas. Therefore, in the following iteration we ``zoom-in'' by repeating the above process in a lower exploration level. Figure~\ref{f:grid} shows a scenario where AIDE discovered one relevant object in all but the top right cell as well as the zoom in operation for this cell. Here, our sampling areas are the smaller four sub-cells inside the higher level cell.
%
%Another hint is based on specific attribute ranges on which the user desires to focus (e.g., clinical trials in years [2000, 2010]). These range-based hints allow our framework to operate on specific grid cells and avoid necessary sampling and computation on irrelevant data areas. Obviously the more specific these hints, the more accurate the final query and the less the user labeling effort.
%\cut{
%By using this grid-based approach, our framework is able to keep track of the already explored subareas in the data space, spread its exploration across the whole domain of all the exploration attributes, avoid sampling of overlapping areas. At each exploration level the system requires $\beta^d$ samples (equal to the number of grid cells for that level) to fully explore that level. Assuming the user is willing to label more samples the system can continue showing labels from the next, finer-grained level. %If AIDE explores $x$ grid cells in the $i$-th iteration then this phase generates $T^i_{discovery}=x$ new samples.
%}
%
%
%\cut{
%
%\subsection{Optimizations}\label{s:explore_optimizations}
%%Next we discuss two optimizations for improving the relevant object discovery phase.
%{\bf Hint-based Object Discovery} Our general grid-based exploration assumes that our system does not have any information about the areas that the user is looking for.\cut{Therefore, as long as the user is willing to provide feedback on new samples, it will continue exploring lower exploration levels for each cell with no discovered relevant objects assuming that smaller areas of interest might be discovered.} However, if the user provides some {\em hints} about his interests, this process can be optimized.
%One hint may be that the selection ranges in a given attribute $d$ will have a width of at least $x_d$. For instance, the user may be interested in clinical trials at least one year apart or sky objects within a certain distance from each other. Given the parameter $x_d$, AIDE can initiate its exploration on the exploration level with cell width $\delta \le x_d/100$. This will ensure that we will retrieve exactly one relevant object from each area of interest, therefore we will not miss any relevant areas. Assuming that the user is willing to provide feedback on at least $\beta^d$ samples (plus more samples for the two other phases) this hint can prevent unnecessary sampling on {higher exploration levels}, therefore improving the convergence to an accurate result while reducing the user's labeling effort.
%
%Another hint is based on specific attribute ranges on which the user desires to focus (e.g., clinical trials in years [2000, 2010]). These range-based hints allow our framework to operate on specific grid cells and avoid necessary sampling and computation on irrelevant data areas. Obviously the more specific these hints, the more accurate the final query and the less the user labeling effort.
%}
\subsection{Misclassified Samples Exploitation}\label{s:misclassified}
While the object discovery phase bootstraps the discovery of relevant objects, it extracts at \emph{most one} object of interest in each sampling area explored. In order to offer acceptable accuracy, decision tree classifiers require a higher number of samples from the relevant class. AIDE employs the \emph{misclassified samples exploitation} phase which improves the accuracy our predictions by increasing the number of relevant objects in our training set.
Misclassified objects can be categorized to: (i) \emph{false positives}, i.e., objects that are categorized as relevant by the classifier but labeled as irrelevant by the user and (ii) \emph{false negatives}, i.e., objects labeled as relevant but categorized as irrelevant by the classifier. False positives are less common because the classifications rules of decision trees aim to maximize the homogeneity of their predicted relevant and irrelevant areas~\cite{breinman84}. Practically, this implies that the classifier defines the relevant areas such as the relevant samples they include are maximized while minimizing the irrelevant ones. In fact, most false positives are due to wrongly predicted boundaries of these areas. Figure~\ref{f:misclassified_combo} shows examples of false positives around a predicted relevant area. Elimination of these misclassified samples will be addressed by the boundary exploitation phase (Section~\ref{s:boundaries}).
False negatives on the other hand are objects of interest that belong in an \emph{undiscovered} relevant area. Examples of false negative are also shown in Figure~\ref{f:misclassified_combo}. Relevant areas are undiscovered by the decision tree due to insufficient samples from within that area. Hence, AIDE increases the set of relevant samples by collecting more objects around false negatives.
%\cut{
%due to the metrics used for deciding the best classification rules for each class. In our case (CART~\cite{breinman84}), this metric measures the homogeneity of the discovered relevant and irrelevant areas and the algorithm opts for rules that maximize this homogeneity. Practically, this implies that the predicted relevant areas are chosen to include as many of the relevant samples while minimizing the irrelevant ones. Therefore, most false positives are due to wrongly predicted boundaries of \emph{discovered} relevant areas.
%\cut{Figure~\ref{f:misclassified} shows examples of false positives around a not precisely predicted relevant area. }
%This problem will be addressed by the boundary exploitation phase (Section~\ref{s:boundaries}) which refines the relevant areas.
%}
%
%\cut{
% Our goal though is to discover \emph{relevant areas} as opposed to single objects. To map relevant points to relevant areas, a significant number of relevant objects from within each relevant area need to be fed to the classification algorithm. To address this, AIDE employs the \emph{misclassified samples exploitation} phase which aims to increase the relevant objects in our training set while reducing the misclassified ones. increases the accuracy our predictions while striving to limiting the sampling overhead of this phase
%}
%However, the object discovery phase cannot achieve this solely (except when the relevant areas are much larger than individual grid cells). Therefore, AIDE employs the \emph{misclassified samples exploitation} phase to increase the relevant objects in our training set such that the predicted queries will select relevant \emph{areas}. This phase is designed to increase both the precision and recall of the final query as it increases the relevant samples while reducing the misclassified ones.
%Hence, it significantly increases the quality of our predictions.
%\cut{
%In this section we first discuss how misclassified samples are generated and then we propose a clustering-based exploration technique that increases the accuracy our predictions while striving to limiting the sampling overhead of this phase.
%}
%By interpreting the characteristics of these samples we propose a clustering-based exploration technique that increases the accuracy our predictions. Our technique strives to limit the number of extraction queries and hence the time overhead of this phase.
%{\bf Generation of Misclassified Samples}
\cut{
In each iteration $i$ AIDE generates a new decision tree classifier $C_i$ based on the sample set $T_i$ that predicts the relevant and irrelevant objects in our exploration space. In the first few iterations, these areas typically cannot classify the training data accurately, leading to: (i) \emph{false positives}, i.e., objects that are categorized as relevant by the classifier but labeled as irrelevant by the user and (ii) \emph{false negatives}, i.e., objects labeled as relevant but categorized as irrelevant by the classifier.
AIDE leverages the characteristics of the misclassified samples to identify promising sampling areas. %Since each type of misclassified samples has unique characteristics, we handle them differently.
}
%Note that the object discovery phase provides a single object from each sampling area (i.e., grid cell or cluster). If this object is labeled as interesting then the training set of the following iteration will include only one relevant object from that sampling area. Depending on the sampling area size and the number of relevant areas, even neighboring grid cells or clusters may not provide enough relevant samples to predict the relevant area.
%\cut{
%\begin{figure}[t]
%\centering
% \includegraphics[totalheight=2.5in, angle=-90] {figs/misclassified-combo}
% \vspace{-0.1in}
%\caption{{\small Misclassified objects and sampling around a single object.}}
%\label{f:misclassified}
%\vspace{-0.2in}
%\end{figure}
%}
%revision
%\subsection{Misclassified Samples Exploitation}
{\bf Clustering-based Exploitation}
Our misclassified exploitation phase operates under the assumption that relevant tuples will be clustered close to each other, i.e., they typically form relevant areas. This implies that sampling around false negatives will increase the number of relevant samples. Furthermore, false negatives that belong in the same relevant area will be located close to each other. Hence, AIDE generates \emph{clusters of misclassified objects} and defines a new sampling area around each cluster. Specifically, it creates clusters using the \emph{$k$-means} algorithm~\cite{breinman84} and defines one sampling area per cluster. An example of a cluster of false negatives is shown in Figure~\ref{f:misclassified_combo}.
\cut{
One possible technique to leverage this knowledge is to handle each misclassified sample independently and collect samples around \emph{each} false negative to obtain more relevant samples. Our experimental results show that this technique is very successful in identifying relevant areas. However, it incurs high time cost, mainly because (i) we execute one retrieval query per misclassified object and (ii) we often redundantly sample highly overlapping areas, spending resources (i.e., user labeling effort) without increasing AIDE's accuracy. The last problem appears when many misclassified samples are close to each other.
We also noticed that sampling around each misclassified object often requires multiple iterations of the misclassified exploitation phase before the decision tree has enough samples to identify a relevant area. Let us assume that in the $i$-th iteration the training set $T_i$ has $m_i$ false negatives based on the classifier $C_i$. Then, in the next iteration $i+1$ we add $T^{i+1}_{missclass}= m_i*f$ samples, where $f$ is the number of samples we collect around each false negative.\cut{Typically the classifier finds one new relevant sample which is not enough to characterize the relevant area it belongs to. Therefore, AIDE collects $f$ samples around this sample.} If the relevant area is still not discovered, these samples are also misclassified and then $f$ samples are collected around each one of these samples. If AIDE needs $k$ iterations to identify a relevant area, the user might have labeled $f^k$ labeled samples without improving the $F$-measure (i.e., discovering a relevant area).
To address this limitation, AIDE employs a clustering-based approach. The intuition is that misclassified samples that belong in the same relevant area will be located close to each other. Therefore, instead of sampling around each misclassified sample independently, we generate \emph{clusters of misclassified objects} and we sample around each cluster. An example of a cluster is shown in Figure~\ref{f:kmeans}. We create clusters using the \emph{$k$-means} algorithm~\cite{breinman84} and have one sampling area per cluster.
}
The main challenge in this approach is identifying the number of clusters we need to create. Ideally, we would like this number to match the number of relevant areas we have ``hit'' so far, i.e., the number of relevant areas from within which we have collected at least one object. \cut{More cluster will lead us to oversample areas we have already ``hit'' while less clusters will lead us to under-sample again these areas.} We argue that the number of relevant objects created by the object discovery phase is a strong indicator of the number of relevant areas we have already ``hit''. The object discovery phase identifies objects of interest that belong to different areas or the same relevant area. In the former case, our indicator offers correct information. In the latter case, our indicator will lead us to create more clusters than the already ``hit'' relevant areas. However, since these clusters belong in the same relevant area they are typically close to each other and therefore the decision tree classifier eventually ``merges'' them and converges to an accurate number of relevant areas.
\begin{figure}[t]
\centering
\includegraphics[totalheight=2.7in, angle=90] {misclassified-combo}
\vspace{-2mm}
\caption{{\small Misclassified objects and cluster-based sampling.}}
\label{f:misclassified_combo}
\vspace{-5mm}
\end{figure}
In each iteration $i$, the algorithm sets $k$ to be the overall number of relevant objects discovered in the object discovery phase. Since our goal is to reduce the number of sampling areas (and therefore the number of sample extraction queries), we run the clustering-based exploitation only if $k$ is less than the number of false negatives. Otherwise we collect $f$ random samples around \emph{each} false negative. We collect samples within a distance $y$ from the farthest cluster member in each dimension. For each cluster we issue a query that retrieves $f \times c$ random samples within this sampling area, where $c$ is the size of the cluster (number of cluster members). Our experimental results showed that $f$ should be set to a small number (10-25 samples) since higher values will increase the user effort without improving the exploration outcome. The closer the value $y$ is to the width of the relevant area we aim to predict, the higher the probability to collect relevant objects than irrelevant ones. \cut{An interesting optimization would be to dynamically adapt this value based on the current prediction of the relevant areas. We plan to explore this direction in our future work.
}
\subsection{Boundary Exploitation}\label{s:boundaries}
Given a set of relevant areas identified by the decision tree classifier, our next phase aims to refine these areas by incrementally adjusting their boundaries. This leads to better characterization of the user's interests, i.e., higher accuracy of our final results. In this section, we describe our general approach. \cut{We also discuss a series of optimizations that
allow us to reduce the number of samples and the sample extraction time required to refine the boundaries of already discovered areas.}
AIDE represents the decision tree classifier $C_i$ generated at the $i$-th iteration as a set of hyper-rectangles in a $d$-dimensional space defined by the predicates in $P_i^r \bigcup P_i^{nr}$, where the predicates ${P_i^r}$ characterize the relevant areas and predicates ${P_i^{nr}}$ describe the irrelevant areas. We iteratively refine these predicates by \emph{shrinking} and/or \emph{expanding} the boundaries of the hyper-rectangles.
Figure~\ref{f:boundaries_general} shows the rectangles for the classifier in Figure~\ref{f:dtree}. If our classification is based on $d$ attributes ($d=2$ in our example) then a $d$-dimensional area defined by $p \in {P_i^r}$ will include objects classified as relevant (e.g., areas A and D in Figure~\ref{f:boundaries_general}). Similarly, objects in an area defined by $p \in {P_i^{nr}}$ are classified as irrelevant (e.g., areas B and C in Figure~\ref{f:boundaries_general}).
AIDE eliminates irrelevant attributes from the decision tree classifier by \emph{domain sampling} around the boundaries. Specifically, while we shrink/expand one dimension of a relevant area we collect random samples over the \emph{whole} domain of the remaining dimensions. Figure~\ref{f:boundaries_general} demonstrates our technique: while the samples we collect are within the range $11 \le dosage \le 9$ they are randomly distributed on the domain of the $age$ dimension. \cut{Our experimental results are based on this approach and we observed that the quality of our final classifier was noticeably improved compared with an approach that selects samples bounded in all dimensions of the relevant areas.% (e.g., samples in the range $11 \le dosage \le 9 \wedge 20 \le age \le 40$).
}
Our evaluation showed that this phase has the smallest impact on the effectiveness of our model: not discovering a relevant area can reduce our accuracy more than a partially discovered relevant area with imprecise boundaries. Hence, we constrain the number of samples used during this phase to $\alpha_{max}$. This allows us to better utilize the user effort as he will provide feedback mostly on samples generated from the previous two, more effective phases.
\cut{Our approach aims to distribute an equal amount of user effort to refine each boundary. } Let us assume the decision tree has revealed $k$ $d$-dimensional relevant areas. Each area has $2^d$ boundaries. Hence we collect $\alpha_{max}/(k \times 2^d)$ random samples within a distance $\pm x$ from each boundary. This approach is applied across all the boundaries of the relevant hyper-rectangles, allowing us to shrink/expand each dimension of the relevant areas. The new collected samples, once labeled by the user, will increase the recall metric: they will discover more relevant tuples (if they exist) and eventually refine the boundaries of the relevant areas.
The $x$ parameter can affect how fast we converge to the real relevant boundary. If the difference between the predicted and real boundaries is less than $x$, this phase will retrieve both relevant and irrelevant samples around the boundary and allow the decision tree to more accurately predict the real boundary of the relevant area. Otherwise, we will mostly collect relevant samples. This will still improve our prediction of the boundary by bringing it closer to the actual one, but it will slow down the convergence to the actual relevant area. We follow a conservative approach and set $x$ to 1 (we search for objects with normalized distance $\pm 1$ from the current predicted boundary). This gradually improves our recall. %We plan to explore dynamic techniques for setting this parameter in our future work.
This phase includes a number of further optimizations, such as detecting and avoiding sampling overlapping areas as well as adjusting the number of samples to the convergence rate of the user model. These optimizations improve AIDE's effectiveness and efficiency and they are described in detail in~\cite{aide_sigmod14}.
\cut{
Figure~\ref{f:boundaries_general} shows with dotted lines the sampling areas when we shrink/expand the boundaries of the relevant area D ($20 <age \le 40 \wedge 0 \le dosage \le 10$ of our example tree in Figure~\ref{f:dtree}). Here, we collect samples that have distance $x=1$ from the $dosage=10$ boundary, i.e., random samples within the $9 \le dosage \le 11$ area. % (the orange colored area in our figure).
}
%{\bf Non-overlapping Sampling Areas} Although the boundary exploitation can be effective, it is often the case that new samples lead to only a slightly (or not at all) modified decision tree. In this case, the exploration areas did not evolve significantly between iterations, resulting in redundant sampling and increased exploration cost (e.g., user effort) without improvements on classification accuracy. Figure~\ref{f:increxploration} shows an example of this case, where one iteration indicates that relevant tuples are within area A whereas the following iteration reveals area B as the relevant one. Given the high overlap of A and B, the sampling areas around boundaries, e.g., $S_A$ and $S_B$, will also highly overlap. %This will result in exploring the same areas and therefore increases the exploration cost with no positive impact on its effectiveness.
%
%
%To address this challenge we rely on the decision tree structure. In each iteration, we identify the decision tree nodes that are modified and translate them to changes on the boundaries of relevant areas. Our sampling will then be limited to only areas with small or no overlap. For example, in Figure~\ref{f:increxploration} in the second iteration we will sample area $S_B'$ (but not $S_B$), since it does not overlap with the previous sampling areas $S_A'$ and $S_A$. This approach allows us to more efficiently steer the user towards interesting areas by reducing the number of iterations and the cost of redundant sampling.
%Finally, to avoid sampling overlapping areas, we identify the decision tree nodes that are modified and translate them to changes on the boundaries of relevant areas. Our sampling will then be limited to only areas with small or no overlap. \cut{For example, in Figure~\ref{f:increxploration} in the second iteration we will sample area $S_B'$ (but not $S_B$), since it does not overlap with the previous sampling areas $S_A'$ and $S_A$. } This approach allows us to more efficiently steer the user towards interesting areas by reducing the number of iterations and the cost of redundant sampling.
\begin{figure}[t]
\centering
\includegraphics[totalheight=2.25in, angle=90] {boundaries-general.pdf}
\vspace{-2mm}
\caption{{\small Boundary exploration for the relevant areas A and D.}} \label{f:boundaries_general}
\vspace{-5mm}
\end{figure}
%\begin{figure}[t]
%\centering
% \includegraphics[totalheight=2.3in, angle=-90] {increxploration}
% %\vspace{-2mm}
%\caption{\small{Overlap of sampling areas for similar decision trees. }} \label{f:increxploration}
%%\vspace{-5mm}
%\end{figure}
\cut{
\subsection{Optimizations}\label{s:boundaryoptimizations}
Next we propose optimizations for boundary exploitation. %These optimizations strive to improve the efficiency of the boundary exploration phase. % while maintaining the efficiency of this phase.
{\bf Adaptive Sample Size} The first optimization dynamically adapts the number of samples collected. The goal is to reduce the sample size around boundaries that are already predicted with acceptable accuracy. The challenge here is that the accuracy of the projected areas cannot be determined in real-time as the actual interest of the user is unknown to our system. To address this, we leverage information from the already built decision tree.
Our decision tree consists of a set of decision nodes (also named split rules). Each rule corresponds to a boundary of a relevant or irrelevant area. AIDE identifies the split rules that characterize a relevant area (e.g., $dosage >10$, $dosage \le 15$ in Figure~\ref{f:dtree}) and in each iteration quantifies the change of this rule (i.e., the change of its boundary value). Bigger changes of a split rule will lead to more samples extracted around the corresponding boundary. The intuition is that significant changes in the rule indicate that the corresponding boundary is not yet very accurate, and hence new samples were able to affect its split rule significantly. In the contrary, small changes in the split rule between two iterations indicate that the decision tree already has a good approximation of the boundary and the new samples did not affect the accuracy of the specific rule. In this case we restrict the samples provided to the user to a lower limit. We keep collecting this lower limit from all, even unmodified boundaries, to compensate the cases where lack of change in a boundary was due to randomness of the sample set and not to precise predictions of its coordinates.
Given a set of new training samples $T_i$ at the $i$-th iteration, our algorithm will identify the decision tree split rules that are modified and translate them to changes on the boundaries of relevant areas. %Note that if the new samples $T_i$ do not affect a node in the decision tree, then its subtree will also not be affected by the new sample set \kiki{this is not true we should erase it, I examine every boundary for a change since the subtrees can change even if the higher node does not.}. This limits the set of decision tree nodes we need to examine.
In each iteration $i$ the number of samples collected in the boundary exploitation phase is calculated as: \[T^i_{boundary}= \sum^{j=2^d}_{j=0}(pc^{j}_{i-1}*\frac{a_{max}}{k\times 2^d })+er*(k\times 2^d)\]
where $d$ is the dimensionality of the exploration space, $pc^j_{i-1}$ is the percentage of change of the boundary $j$ between the $(i-1)$-th and $i$-th iterations, and $er$ is an error variable to cover cases where the boundary is not modified but also not accurately predicted. The percentage of change $pc_{i-1}$ is calculated as the difference of the boundary's normalized values of the specific dimension.
{\bf Non-overlapping Sampling Areas} Although the boundary exploitation can be effective, it is often the case that new samples lead to only a slightly (or not at all) modified decision tree. In this case, the exploration areas did not evolve significantly between iterations, resulting in redundant sampling and increased exploration cost (e.g., user effort) without improvements on classification accuracy. Figure~\ref{f:increxploration} shows an example of this case, where one iteration indicates that relevant tuples are within area A whereas the following iteration reveals area B as the relevant one. Given the high overlap of A and B, the sampling areas around boundaries, e.g., $S_A$ and $S_B$, will also highly overlap. %This will result in exploring the same areas and therefore increases the exploration cost with no positive impact on its effectiveness.
\begin{figure}[t]
\centering
\includegraphics[totalheight=2.3in, angle=-90] {figs/increxploration}
\vspace{-2mm}
\caption{\small{Overlap of sampling areas for similar decision trees. }} \label{f:increxploration}
\vspace{-5mm}
\end{figure}
To address this challenge we rely on the decision tree structure. In each iteration, we identify the decision tree nodes that are modified and translate them to changes on the boundaries of relevant areas. Our sampling will then be limited to only areas with small or no overlap. For example, in Figure~\ref{f:increxploration} in the second iteration we will sample area $S_B'$ (but not $S_B$), since it does not overlap with the previous sampling areas $S_A'$ and $S_A$. This approach allowes us to more efficiently steer the user towards interesting areas by reducing the number of iterations and the cost of redundant sampling.
{\bf Identifying Irrelevant Attributes} Our general boundary exploitation is applied only on the attributes appearing in the decision tree and on the range of each attribute domain selected by the decision tree. Inevitably, this approach introduces skewness on the exploration attributes which in some cases may prevent the convergence to a good classification model. The problem is particularly obvious when the decision tree includes split rules with attributes irrelevant to the user interests, often due to lack of sufficient training samples. In our current experimentation we have often seen such cases which resulted into proposing a final query with different selection attributes than the user's intended query.
To handle these cases we rely on \emph{domain sampling} around the boundaries. While we shrink/expand one dimension of a relevant area we collect random samples over the \emph{whole} domain of the remaining dimensions. Figure~\ref{f:boundaries_general} demonstrates our technique: while the samples we collect are within the range $11 \le dosage \le 9$ they are randomly distributed on the domain of the $age$ dimension. Our experimental results are based on this approach and we observed that the quality of our final classifier was noticeably improved compared with an approach that selects samples bounded in all dimensions of the relevant areas (e.g., samples in the range $11 \le dosage \le 9 \wedge 20 \le age \le 40$).
{\bf Exploration on Sampled Datasets}
The previous optimization forces our sample extraction queries of this phase to execute random sampling across the whole domain of each attribute. Such queries are particularly expensive, since they need to fully scan the whole domain of all attributes. Even when covering indexes are used to prevent access to disk, the whole index needs to be read for every query, increasing the sampling extraction overhead. We observed that although sampling on the whole domain improves our accuracy by an average of 42\%, it also has a time overhead of 95\% more than the other two phases. This overhead becomes even higher as we increase the size of our database.
To improve our sample extraction time and improve the scalability of AIDE, we apply our techniques on sampled data sets. Specifically, we generate a random sampled database and extract our samples from the smaller sampled dataset. We note that this optimization can be used for both the misclassified and the boundary exploitation phases. This allows our sample extraction time to improve, with the most positive impact coming from the boundary exploitation phase. We observed that operating on a sampled dataset with size equal to 10\% of that of the original data set can improve our boundary exploitation time by up to 83.4\% and the time for the misclassified exploitation phase by up to 74.5\%.
Operating on sampled datasets could potentially reduce the accuracy of our exploration results. However, an interesting artifact of our exploration techniques is that their accuracy does not depend on the frequency of each attribute value, or on the presence of all available tuples of our database. This is because each phase executes \emph{random} selections within data hyper-rectangles and hence these selections do not need to be deterministic. In the contrary, as long as the domain value distribution within these hyper-rectangles is roughly preserved, our techniques are still equally effective on the sample dataset as in the actual one (i.e., their accuracy will not be reduced significantly). Therefore, we generate our sampled data sets using a simple random sampling approach that picks each tuple with the same probability~\cite{Olken94randomsampling}. This preserves the value distribution of the underlying attribute domains and allows us to offer a similar level of accuracy but with significantly less time overhead.
}
\section{Performance Optimizations}\label{s:optimizations}
In this section we describe a set of novel optimizations we introduced in AIDE. These include techniques that: (a) handle exploration on skewed data distributions, (b) leverage the informativeness of samples to improve AIDE's effectiveness, (c) extend the expressiveness of the user feedback model to accelerate the convergence to an accurate model and (d) reduce the size of our exploration space to offer highly interactive times. We note that the first three techniques are new optimizations that we added to the original version of AIDE introduced in \cite{aide_sigmod14,aide_vldb15}.
\subsection{Skew-aware Exploration}\label{s:explore_skewed}
Skewed data distributions are prevalent in virtually every domain of science. For example, in astronomy stars
and other objects are not uniformly distributed in the sky. Hence, telescope measurements have corresponding areas of density and sparsity. In our framework, skewed data distributions could hinder the discovery of relevant objects. This is due to the fact that our initial exploration step (Section~\ref{s:exploration}) is designed to distribute the number of collected samples evenly across this space. While for uniform data distributions this approach will be effective, in the presence of skew, it could slow convergence to an accurate user model, since dense areas will be under-sampled compared with the sparse ones.
To address this challenge we introduce a new sampling technique designed to operate effectively on both uniform and skewed data distributions, i.e., predict with high accuracy and less effort relevant areas that appear in either dense or sparse sub-areas of the exploration space.
Specifically, we modified our first exploration phase to combine the grid-based exploration approach with a clustering technique that allows us to identify dense areas and increase our sampling effort within them. Our technique uses the $k$-means algorithm~\cite{breinman84} to partition the data space into $k$ clusters and each cluster is characterized by its centroid while database objects are assigned to the cluster with the closest centroid. Thus, each cluster includes similar objects (where similarity is defined by a distance function) and each centroid serves as a good representative of the cluster's objects. Similarly to our grid-based exploration, we create multiple exploration levels, where higher levels include fewer clusters than lower ones.
In parallel, AIDE maintains its grid-based exploration levels, i.e., it divides the exploration space to $k$ sampling areas (i.e., grid cells) of the same size independently of the distribution of the exploration space (as described in Section~\ref{s:exploration}). For uniform distributions, the cluster-based and the grid-based sampling areas overlap. In this case, using any of the two types of sampling areas is sufficient to discover relevant areas. However, in the presence of skewed exploration domains most of the clusters will be concentrated to dense areas leaving sparse areas under-sampled. Maintaining our grid-based sampling areas allows us to sample also sparse sub-areas and discover relevant areas of low density.
We now describe the details of our exploration technique. AIDE starts its exploration using the cluster-based exploration levels and collects samples around the centroid of each cluster. Specifically, we select one object per cluster within distance $\gamma < \delta$ along each dimension from the cluster's centroid, where $\delta$ is the radius of the cluster. We initialize our exploration on the highest exploration level (i.e., with the few clusters). If no interesting objects are discovered, we sample the next level where finer-grained clusters are available.
Next, AIDE uses the grid-based approach to sample sparse sub-areas. Here, we calculate the density value $s$ for every grid cell, where skew is defined as $s = {u}/{p}$, $u$ is the number of unique tuples mapped n that cell and $p$ is the number of all possible unique tuples that would exist if the cell was ``full'', i.e., there were to be a tuple for each (attribute,value) combination of cell. The higher the $s$ value the denser a cell is. In this step AIDE only samples non empty sparse grid cells for which $s\leq t$. To set the $t$ value we analyse offline the distribution characteristics of our exploration space to identify sparse sub-areas and we set $t$ to the average density of the grid cells covering those areas. \cut{To set the parameter $t$ we apply the $k$-means clustering algorithm offline on different exploration spaces (i.e., attribute combinations). For each exploration space we calculate the average skew value of the cells that would only get explored in lower exploration levels if we were using the cluster-based exploration levels.} AIDE samples these sparse cells by extracting one random sample close to the center of the grid cell.
The user is presented by the samples collected by both the grid-based and the cluster-based sampling areas. This hybrid approach allows us to adjust our sample size to the skewness of our exploration space (i.e., we collect more samples from dense sub-areas) while it ensure that any sparse relevant areas will not be missed (i.e., sparse sub-areas are sufficiently explored).
\cut{After we have identified the right skew threshold for the attribute combination that is of interest to the user we compare the skew value of each grid cell to the skew threshold and collect samples from all the sparse regions and together with the samples from the clustering technique we present them to the user. In Figure~\ref{f:hybridExploration} we can observe the two cluster centers that have been created in the highly skewed areas of the data space. However, the upper right corner of the data space is sparsely populated (low $S$ value) and no cluster center falls there. Therefore we are going to utilize the grid-cell structure for that part of the data space and select samples to present to the user located close to the grid-cell centers. Our hybrid technique ensures that we will discover the user's relevant area in a skewed data space even if it is located in a sparse region, as long as samples exist within the relevant area. }
% and it collects the same number of samples within each area (i.e., one random sample close to the center of the cell).
%
%Under this approach, AIDE collects samples around the centroid of each cluster. Specifically, we select one object per cluster within distance $\gamma < \delta$ along each dimension from the cluster's centroid, where $\delta$ is the radius of the cluster. We create multiple exploration levels, where higher levels include fewer clusters than lower ones, and we initialize our exploration on the highest level. If no interesting objects are discovered, we sample the lower level where finer-grained clusters are available.
%
%In a non-skewed data space, all underlying data areas have approximately the same data distribution and naturally both the grid based and clustering approach work similarly no matter where the relevant area is located. However, both techniques have their drawbacks. As discussed in the previous section, the grid construction approach is skew-agnostic: it zooms into grid-cells located in sparse data areas that only contain a few samples thus increasing the user wait time. Furthermore, the clustering approach has to create multiple clusters (essensially jump to lower exploration levels) to "hit" a sparsely populated relevant area thus increasing the user effort. Ideally, we would like to have a technique that minimizes the user effort and maximizes the system's efficiency under all possible combinations of relevant area and data space distributions.
%
\cut{
In the presence of skewed exploration domains this approach will construct cells with highly diverse density. For those s with low density ({\em sparse areas}), AIDE has a slim chance to discover samples close to its center. The current approach will then zoom into a lower exploration level to collect samples from the smaller sub-cells. This extra cost, however, often does not result in improved accuracy because the number of samples returned from a sparse area may be too low to improve the $F$-measure substantially.
}
\cut{
To address this issue, we propose a skew-aware clustering-based approach for identifying sampling areas. More specifically, AIDE uses the $k$-means algorithm~\cite{breinman84} to partition the data space into $k$ clusters. Each cluster is characterized by its centroid and database objects are assigned to the cluster with the closest centroid. Thus, each cluster includes similar objects (where similarity is defined by a distance function) and each centroid serves as a good representative of the cluster's objects. Under this approach, AIDE collects samples around the centroid of each cluster. Specifically, we select one object per cluster within distance $\gamma < \delta$ along each dimension from the cluster's centroid, where $\delta$ is the radius of the cluster. We create multiple exploration levels, where higher levels include fewer clusters than lower ones, and we initialize our exploration on the highest level. If no interesting objects are discovered, we sample the lower level where finer-grained clusters are available.
In the presence of skewed exploration domains, this approach will be more effective since $k$-means will create most of the clusters in dense areas. %In general, the sparser the areas the less clusters are created there. that's wrong.
This will allow AIDE to focus its sampling in areas with high density. Assuming that the user interests lie mostly in dense areas, our technique will avoid redundant sampling in sparse areas and the unnecessary subsequent zooming into the next level. Therefore, AIDE will converge to an accurate result with fewer labeled samples for those skewed exploration attributes.
}
%{\bf Hybrid Exploration}
%The clustering-based technique is skew-aware: it will create clusters in highly skewed areas of the data space. If the relevant area is conveniently located in one of those highly skewed areas the user will discover the area at a high exploration level without having to label samples from lower exploration levels. However, in the opposite case where our relevant area is located in a sparsely populated space we expect cluster centers to be located in the sparse areas only when the number of clusters is high; we would only hit the area in a low exploration grid. Therefore, the user would have to spend more effort on the system by labeling each sample from every cluster at lower exploration levels and her wait time would increase since the system would have to send more queries to the database to retrieve one sample from each cluster.
%
%It is obvious that in skewed data spaces, another parameter that affects the user effort and the system's efficiency is the data distribution within the relevant area. In a non-skewed data space, all underlying data areas have approximately the same data distribution and naturally both the grid based and clustering approach work similarly no matter where the relevant area is located. However, both techniques have their drawbacks. As discussed in the previous section, the grid construction approach is skew-agnostic: it zooms into grid-cells located in sparse data areas that only contain a few samples thus increasing the user wait time. Furthermore, the clustering approach has to create multiple clusters (essensially jump to lower exploration levels) to "hit" a sparsely populated relevant area thus increasing the user effort. Ideally, we would like to have a technique that minimizes the user effort and maximizes the system's efficiency under all possible combinations of relevant area and data space distributions.
%and ghas a high impact on the effort the user will spend on the system. In such a case, a skew agnostic technique such as our grid construction approach might work more efficiently.
%Thus, we see that depending on the
%Both techniques presented above have advantages as well as shortcomings depending on the density of the data space. Our clustering-based technique outperforms the grid-based exploration when data are skewed. However, our grid-based technique outperforms the clustering-based technique when the data are skewed but our query is in a sparse area since the clustering-based technique might not create a cluster in that area. Finally, both techniques perform similarly when the data space is dense and there is no skew.
%To address the above challenge we designed a hybrid technique that combines the best of both the grid based and the clustering technique. The clustering technique will not select samples from sparse areas in high exploration levels but will only focus on dense areas. Thus, we need to identify the sparse areas and present samples from them to the user in case her relevant area is located in a sparse region. To achieve that initially we gather all the samples collected from the clustering technique since those explore dense spaces well. But we do not stop there: we try to identify sparse areas in the data space and collect samples from them to present to the user. In order to do this we keep the grid structure of the grid-based technique and we calculate a skew value $S$ for every grid cell to represent the distribution of data in that cell. We define $S$ to be $S = \frac{u}{p}$ where $u$ is the number of unique tuples that exist in the cell and $p$ is the number of all possible unique tuples that would exist if the cell was fully occupied. Intuitively, if $S$ has a low value for a grid cell that means that it is sparsely populated.
%%and we would have to move to lower exploration levels to collect a sample from that area
%
%
%\begin{figure}[t]
%\centering
% \includegraphics[totalheight=1.7in, angle=-360]{newfigs/hybridExploration}
%\caption{{\small Hybrid exploration in a skewed 2-D space.}}
%\label{f:hybridExploration}
%\vspace{-5mm}
%\end{figure}
%After we calculate the skew value $S$ for each grid cell in the current exploration level, we set a skew threshold $t$ and we consider grid cells with value $S\leq t$ to be sparsely populated. For those grid cells we select samples using our grid-based exploration where we choose a random sample close to the center of each grid cell and present it to the user. To set the value of the skew threshold $t$ we perform our clustering technique offline for different attribute combinations and we find the average skew value of the sparse grids that would only get explored in lower exploration levels by the clustering technique. After we have identified the right skew threshold for the attribute combination that is of interest to the user we compare the skew value of each grid cell to the skew threshold and collect samples from all the sparse regions and together with the samples from the clustering technique we present them to the user. In Figure~\ref{f:hybridExploration} we can observe the two cluster centers that have been created in the highly skewed areas of the data space. However, the upper right corner of the data space is sparsely populated (low $S$ value) and no cluster center falls there. Therefore we are going to utilize the grid-cell structure for that part of the data space and select samples to present to the user located close to the grid-cell centers.
%Our hybrid technique ensures that we will discover the user's relevant area in a skewed data space even if it is located in a sparse region, as long as samples exist within the relevant area. This technique outperforms both previous techniques in terms of user effort and efficiency since it helps the user discover his relevant area without zooming into lower exploration levels.
%\subsection{Optimization: Hint-based Object Discovery}\label{s:explore_optimizations}
%Our general grid-based exploration assumes that our system does not have any information about the areas that the user is looking for.\cut{Therefore, as long as the user is willing to provide feedback on new samples, it will continue exploring lower exploration levels for each cell with no discovered relevant objects assuming that smaller areas of interest might be discovered.} However, if the user provides some {\em hints} about his interests, this process can be optimized.
%One hint may be that the selection ranges in a given attribute $d$ will have a width of at least $x_d$. For instance, the user may be interested in clinical trials at least one year apart or sky objects within a certain distance from each other. Given the parameter $x_d$, AIDE can initiate its exploration on the exploration level with cell width $\delta \le x_d/100$. This will ensure that we will retrieve exactly one relevant object from each area of interest, therefore we will not miss any relevant areas. Assuming that the user is willing to provide feedback on at least $\beta^d$ samples (plus more samples for the two other phases) this hint can prevent unnecessary sampling on {higher exploration levels}, therefore improving the convergence to an accurate result while reducing the user's labeling effort.
%Another hint is based on specific attribute ranges on which the user desires to focus (e.g., clinical trials in years [2000, 2010]). These range-based hints allow our framework to operate on specific grid cells and avoid necessary sampling and computation on irrelevant data areas. Obviously the more specific these hints, the more accurate the final query and the less the user labeling effort.
%Our hybrid technique outperforms all other solution
%After the skew value has been calculated for all of the grids in the cell we locate the grids where the skew value is smaller than a threshold for those grids. We consider those grids to be sparsly populated. For those grids we select a tuple to present to the user using our grid-based exploration since if our clustering technique would be used there wouldn't be a cluster formed for this grid since it is sparsly populated. For the rest of the grids where their skew value is higher than the threshold we have set we consider them to be densly populated and it is safe to show one sample close to the centroid of the cluster.
%\todo {which case does this help} A drawback of the hybrid solution is that it might lead us to collect more samples than before in each level because we have samples from the clustering solution plus the samples from the grid solution for the grids that have a skew value lower than the threshold we have set. This helps us to face
\input{uncertainty}
\input{maybe}
\subsection{Exploration Space Reduction}\label{s:sparereduction}
\cut{
The previous optimization forces our sample extraction queries of this phase to execute random sampling across the whole domain of each attribute. Such queries are particularly expensive, since they need to fully scan the whole domain of all attributes. Even when covering indexes are used to prevent access to disk, the whole index needs to be read for every query, increasing the sampling extraction overhead. We observed that although sampling on the whole domain improves our accuracy by an average of 42\%, it also has a time overhead of 95\% more than the other two phases. This overhead becomes even higher as we increase the size of our database.
}
Our exploration techniques rely on sending a sampling query to the back end database system for each defined sampling area.
Such queries can particularly expensive. This especially true for the sampling queries generated by the boundary exploitation phase since they need to fully scan the whole domain of all attributes. Even when covering indexes are used to prevent access to disk, the whole index needs to be read for every query, increasing the sampling extraction overhead.
An interesting artifact of our exploration techniques is that their effectiveness does not depend on the frequency of each attribute value, or on the presence of all available tuples of our database. This is because each phase executes \emph{random} selections within data hyper-rectangles and hence these selections do not need to be deterministic. Hence, as long as the domain value distribution within these hyper-rectangles is roughly preserved, our techniques are still equally effective. This observation allows to apply our exploration on a sampled exploration space. Specifically, we generate our sampled data sets using a simple random sampling approach that picks each tuple with the same probability~\cite{Olken94randomsampling}. We then execute our exploration on this smaller sampled space. Since this data space maintains the same value distribution of the underlying attribute domains, our approach offers a similar level of accuracy but with significantly less time overhead.
\cut{We note that this optimization can be used for both the misclassified and the boundary exploitation phases. This allows our sample extraction time to improve, with the most positive impact coming from the boundary exploitation phase. We observed that operating on a sampled dataset with size equal to 10\% of that of the original data set can improve our boundary exploitation time by up to 83.4\% and the time for the misclassified exploitation phase by up to 74.5\%.
}
%\cut{Operating on sampled datasets could potentially reduce the accuracy of our exploration results. However, an interesting artifact of our exploration techniques is that their accuracy does not depend on the frequency of each attribute value, or on the presence of all available tuples of our database. This is because each phase executes \emph{random} selections within data hyper-rectangles and hence these selections do not need to be deterministic. In the contrary, as long as the domain value distribution within these hyper-rectangles is roughly preserved, our techniques are still equally effective on the sample dataset as in the actual one (i.e., their accuracy will not be reduced significantly). Therefore, we generate our sampled data sets using a simple random sampling approach that picks each tuple with the same probability~\cite{Olken94randomsampling}. This preserves the value distribution of the underlying attribute domains and allows us to offer a similar level of accuracy but with significantly less time overhead.
%}
\subsection{Probabilistic Sampling}\label{s:uncertainty}
AIDE relies on an pool-based active learning paradigm for discovering user interests, i.e., samples are picked from a pool of unlabeled data objects and presented to the user for labeling. Existing pool-based sampling strategies~\cite{Settles10activelearning} exhaustively examining \emph{all} unlabeled objects available, searching for the best sample to show to the user. Clearly such an approach cannot scale on big dataset. AIDE addresses this challenge by identifying a small number of sub-areas of the total exploration space to sample and within each area it collects \emph{random} samples, therefore eliminating greedy sampling techniques.
%
%
% The techniques presented in Sections~\ref{s:exploration}-\ref{s:boundaries} offer high accuracy and interactive experience by reducing the size of the available sampling pool. Specifically, our techniques identify strategic sampling areas that potentially lead to to discovering more relevant objects. Once the sampling space is reduced, our strategy for collecting samples within each ``promising'' sub-area is random sampling.
While random sampling is highly effective especially in the boundary exploitation step (e.g., it distributes the samples across the whole domain of our exploration attributes which eliminates irrelevant attributes from the classifier, see Section~\ref{s:boundaries}), it suffers from certain limitations. In particular, in the misclassified exploitation phase, random sampling treats each samples uniformly and it does not leverage the informativeness of the samples, which could potentially lead faster to an accurate user model. In other words, random sampling does not answer the question ``which candidate samples to show to the user in order to reduce the total number of labeled samples needed for learning''. To address this question, AIDE includes a new \emph{probabilistic sampling} strategy for the misclassified exploitation phase.
\cut{Both in the misclassified exploitation phase and the boundary exploitation phase the samples that we select to present to the user are randomly chosen from certain strategic areas. In the misclassified exploitation phase we collect $f$ random samples from within a distance $y$ from the farthest cluster member in each dimension for each of the clusters that are generated during the phase. In the boundary exploitation phase we collect $\alpha_{max}$ random samples within a distance $\pm x$ from each boundary of each hyper-rectangle that we have predicted.}
%Random sampling within these strategically selected data areas can lead to improvements in user wait time compared with sampling over the entire data space.
%Although random sampling within the boundary sampling areas allows us to eliminate irrelevant attributes from the predicted user model ,
Active learning has proposed a number of sample selection approaches that evaluate the informativeness of unlabeled samples~\cite{Settles10activelearning}. In all these strategies the informativeness of a sample (e.g., the probability of being relevant or not) is either generated from scratch or sampled from a given distribution. In our framework, we do not assume a known distribution for our relevant or irrelevant objects. Instead we leverage the user's relevance feedback to calculate for each unlabeled object its informativeness, i.e., its probability of being labeled as relevant or irrelevant (aka posterior probability). Given this probability, we use the \emph{uncertainty sampling} strategy to identify the next set of samples to show to the user.
We now discuss how evaluate the posterior probability of unlabeled samples, given a set of relevant samples $S^+$ and irrelevant samples $S^-$. AIDE considers each labeled sample as basis for a nearest neighbour classifier with only one training sample and consider each unlabeled object to be a test example that has to be classified into the relevant or non-relevant class. We then combining these classifiers in order to ``blend'' information from all the user's collected feedback~\cite{deselaers_08, combineclassifier_98}.
Formally, we assume that, given a sample labeled as relevant by the user $s_{+}$, the probability that a unlabeled sample $x$ is relevant ($r$) is:
\[p_{x}(r|s_{+}) \propto \exp(-similarity(x,s_{+}))\]
where $similarity(x,s_{+})$ return the similarity value between $x$ to $s_{+}$. Intuitively, this formula indicates that the probability of a sample $x$ being relevant increases exponentially with its similarity to the relevant sample $s_{+}$. This is in accordance to our former argument that relevant samples will be clustered together in the exploration space and will be forming relevant areas.
Analogously, we assume that the probability for a
sample $x$ being non-relevant ($n$) increases exponentially when the sample is similar to a sample $s_{-}$ labeled as non-relevant by the user: \[p_{x}(n|s_{-}) \propto \exp(-similarity(x,s_{-})).\]
To calculate the posterior probability of a sample $x$ being relevant, we combine the individual classifiers from the set of relevant samples $S^{+}$ and the sets of irrelevant samples $S_{-}$ by using the sum rule~\cite{combineclassifier_98}. Specifically, given that $p_{x}(r|s_{+}) = 1-p_{x}(n|s_{-})$,
\cut{Thus to calculate the probability of sample $x$ being relevant we can combine into one function the posterior probability of this sample being relevant given a set of relevant samples $S^{+}$ and the posterior probability of the same sample being non-relevant given a set of non-relevant samples $S^{-}$ :}
\begin{eqnarray*}
p_{x}(r|(S^{+}, S^{-})) &=
\frac{\alpha}{|S^{+}|} \sum_{s_{+}\in S^{+}} p_{x}(r|s_{+}) + \\& \frac{1- \alpha}{|S^{-}|} \sum_{s_{-}\in S^{-}} 1 - p_{x}(n|s_{-})
\end{eqnarray*}
where $\alpha$ is a weighting factor we added to allow us to change the impact of the relevant and non-relevant samples. In the above formula if $\alpha=1$ we only take into account its distance from the set of relevance samples to calculate its posterior probability. In the opposite case if $\alpha=0$ we only consider its distance from the set of samples that are labeled as non-relevant.
Given the posterior probability of a sample, we use the \emph{uncertainty sampling} strategy to select which samples to show to the user~\cite{Settles10activelearning}. In uncertainty sampling the user is presented with samples for which the classifier is the most uncertain about. When using a binary classification model, like in our case, uncertainty sampling selects the sample whose posterior probability of being positive is nearest to 0.5~\cite{Settles10activelearning}. These are the samples that we are the least certain about their relevance.
We apply this approach in our misclassified exploitation phase as follows. Our sampling areas are defined around the clusters of misclassified we have identified. Specifically, given a cluster of size $c$ we retrieve all samples within a distance $y$ from the farthest cluster member in each dimension.
Next, we calculate the posterior probability for each of these samples and we present to the user $f \times c$ samples whose probability is closest to 0.5, where $f$ is our estimation of the number of relevant areas not identified by the classifier (see Section~\ref{s:misclassified} on how this number is calculated). Employing this technique allows us to discover the user's relevant area with less labeled samples proving the hypothesis that some samples are more informative than others. %We discuss these findings in detail in Section~\ref{s:experiments}.
% Initially, we calculate the posterior probabilities for all the samples located within the sampling area. The probabilist model we used to calculate the posterior probabilities of the samples is based on \cite{deselaers2008}.
%In the misclassified phase initially we retrieve from the database all the samples that exist within a distance $y$ from the farthest cluster member in each dimension for each of the clusters.
%Similarly, in the boundary exploitation phase we retrieve all the samples within a distance $\pm x$ from each boundary of each hyper-rectangle that we have predicted, we calculate the posterior probability of each of these samples and we present to the user the $a_{max}$ samples whose probability is nearest 0.5.
\subsection{Similarity Feedback Model}\label{s:maybe}
In our previous paragraphs we introduced exploration techniques that rely on binary relevance feedback, i.e., the user indicates whether the sample is relevant or not to her exploration task. However, there exist numerous scenarios where although the user cannot decidedly classify the relevance of an object, she can indicate whether this object is ``close'' to her interests.
This label can be used when the user finds relevant some characteristics of the object but not necessarily all of them or if she is still uncertain about the relevance of the object, which is often the case when the user is unfamiliar with the underlying data set.
Let us consider the case of a scientist exploring an astronomical dataset searching for clusters of sky objects with unusually high brightness. Initially, the user will be able to label star objects with high brightness values as potentially interesting. However, her understanding of which brightness values are in fact unusual crystallizes only after she has examined numerous sky objects of various brightness values. After that point she can identify unusually bright sky objects and label them as relevant. In another example, medical professionals searching for clinical trials for diabetes type A on 2 year old children can indicate that studies on diabetes type B on 3 year old children are also of possible interest to her (e.g., since the symptoms, medication and side effects for 2 and 3 year old children can be quite similar). However, she will label as relevant only trials on 2 year olds. %In order to effectively address these cases AIDE needs to improve the expressivity of its feedback model.
In the technical level, using a binary feedback model imposes a number of limitations to AIDE. In the previous example let's assume the user labels trials on 3 year old children as relevant (since it is close to the age of the actual patient). This will lead to less accurate classification model (e.g., AIDE will steer the exploration to studies on 3 year olds). While the use can modify this label in subsequent iterations, this will slow the convergence of the exploration to an effective classification model. On the other hand labeling these trials as irrelevant does not capture the similarity of these trials to the actual relevant objects (e.g., studies on 3 years are closer in the exploration space to the relevant trials than studies on 10 year olds). This similarity information, if expressed, could lead AIDE to focus its exploration on small ages and converge to an accurate model with less user effort.
Furthermore, the similarity feedback could help improve AIDE efficiency when predicting small areas of interest. In our current approach, the smaller the relevant area we aim to predict, the higher user effort (i.e., number of labeled samples) is required. This is a practical challenge especially when the relevant objects are clustered within very small areas in the exploration space. The smaller the relevant area the more zoom-in operations AIDE will execute in order to discover a relevant sample from within that area (Section~\ref{s:exploration}). These operations result in sampling more areas (i.e., grid cells), which increases the user effort as well as the number of sampling queries processed. A more expressive feedback model that allows users to indicate that a sample is ``close'' to a relevant object could help us direct our zoom-in operations to only promising sub-areas of the exploration space. This will lead to an accurate user model with less user effort and exploration overhead.
%would have to label samples from fine-grained grids (i.e., small grid cells) selected by the object discovery phase (Section~\ref{s:exploration}). Recall that at each exploration level the system requires all of the grid cells in that level to be explored by presenting one sample from each grid cell to the user. Therefore, a fine-grained search with many small grid cells results in an increased user effort thus making it hard for the user to discover small relevant areas.
%could assist AIDE to identify the relevant areas with less user effort and iterations.
To address the above challenges, we extended our user feedback model as follows. Users can indicate that an object is ``close'' to her interests by annotating it as a ``similar'' sample. This label should be used for samples with at least one attribute value that appears interesting or similar (``close'') to an relevant value. The user has the option to indicate these attributes, i.e., the dimension on which she found the sample to be interesting (e.g., age range in the above medical example, brightness in the scientific example). The system can then utilize this extra information to expedite the exploration process. We note that our ``similarity'' annotations do not constitute a new label for our classification model, i.e., our decision tree classifier will continue to generate classification rules that predict only the relevant and irrelevant classes. Next, we describe our technique.
\cut{Ideally, we would like the user to express interest in objects similar (located close) to her interesting ones so that we can discover her relevant area without moving to lower exploration levels. To allow the user to express interest in samples that are close to the relevant area we extended our relevance feedback model to include one more label besides relevant and non-relevant: a \emph{weak-relevant} label. The main idea is that when we present the user with a sample that is very similar to the samples in the relevant object set $T^r$ the user can now label it as weak-relevant. The system can then utilize this extra information to expedite the exploration process of discovering the relevant area that is hidden nearby.
}
%I could mention in the motivation that when we move to a finer grained exploration level the user has to label one sample from each grid cell. If we have the information that we are located close to a relevant area we don't have to "zoom in" in the entire data space but can just focus around that weak sample to discover the area.
%We want to reduce number of samples that the user will label with the help of this hint from him.
% into $\beta$ equal width ranges, effectively creating $\beta^d$ grid cells. At each exploration level the system requires $\beta^d$ samples (equal to the number of grid cells for that level) to fully explore that level.
\begin{figure}[t]
\centering
\includegraphics[totalheight=1.5in, angle=0] {extendedFeedbackModel}
\vspace{-2mm}
\caption{{\small {Similarity feedback exploration}.}} \label{f:weakRelevantFigure}
\vspace{-5mm}
\end{figure}
\cut{To handle the case of encountering weak-relevant samples in our labeled sample set we deploy a new phase called \emph{weak-relevant sample exploitation}.}
{\bf Extended Feedback Exploration} We introduce one more exploration phase that defines sampling areas around each ``similar'' sample. Based on the definition of this label, each such sample $x$ is potentially ``close'' to a relevant object in at least one of the exploration dimensions. AIDE by default assumes that this similarity may be present in all dimensions unless the user explicitly indicates for which dimensions she discovered similar values. We refer to these as the \emph{interesting dimensions}.
Let us assume a sample $x$ annotated as ``similar' across a set of interesting dimensions $d$ (which are a subset of the set of exploration dimensions). AIDE explores all possible interesting dimensions around $x$ on the $d$ dimensional space aiming to identify relevant samples. Specifically, there are $2^d$ possible exploration directions around the sample, i.e., for each dimension we explore both higher and lower values of the $x$'s value on this dimension. Hence, we define $2^d$ sampling areas and we select one random sample close to the center of each areas to present to the user. In Figure~\ref{f:weakRelevantFigure} we show a scenario of a 2-dimensional exploration space (age and dosage from our medical example), where the user has indicated as interesting only a single dimension (age). Hence, we have created 2 sampling areas around the sample $x$ and we have selected one sample within each of these areas.
% \cut{assume that when a user labels a sample $M$ as weak-relevant a relevant area will be located close to the sample in the data space. Since we are not aware of where that area might be, we need to search towards all possible directions around the weak-relevant sample to find it. In a $d$ dimensional space there are $2^d$ possible directions around the sample; 2 directions at each dimension: left and right of the sample. In order to cover all possible cases we create $s = 2^d$ sampling areas and select one random sample close to the center of each of those sampling areas to present to the user. In figure~\ref{f:weakRelevantFigure} we see that in a 2-dimensional space (age and dosage from our medical example) we have created 4 sampling areas around the maybe sample $m$ and we have selected one sample from each of these sampling areas.}
We define the sampling areas to be located in a distance $\pm \gamma$ from the ``similar'' sample $x$ in each interesting dimension. If one of the new samples we present to the user is now closer to the relevant area we can expect that the user will annotate it as a ``similar'' sample too. In the opposite case, we assume that the user will naturally be dissatisfied with these samples and will label them as non-relevant. In Figure~\ref{f:weakRelevantFigure}, let's assume that $x$ is a study on 5 years olds. If the patient's age is 3 then samples with lower age groups (e.g., sample $x'$) will be also annotated as ``similar'' while samples with higher age groups (e.g., sample $n$) will be irrelevant.
In each iteration, AIDE will collect samples around each ``similar'' sample steering our exploration closer to the relevant area at each step. Eventually one of the sampling areas will overlap with the relevant area and the user will label the sample we extract from that area as relevant. Hence, sampling in a distance $\gamma$ from $x$ bring us closer or inside the relevant area.
The effectiveness of our $\gamma$ value correlates with the range size of the relevant area $s$ in each dimension (see Figure~\ref{f:weakRelevantFigure}). Let us assume $\gamma \leq s$ for some dimension. Then in the next iteration we will either we will either: a) sample inside the relevant area or b) our sampling areas will keep getting closer to the relevant area. The first case leads directly to the relevant area. In the second case we guarantee that we will ``hit'' the relevant range in that dimension in $d/\gamma$ iterations and hence after $d/\gamma -1$ ``similar'' sample annotations, where $d$ is the distance of the sample $x$ from the relevant range. In the opposite case where $\gamma>s$ we might move towards the relevant area but miss the area altogether; intuitively, our "step" is so large that we ``jump'' over the relevant range and never sample within it. In this case we expect the user to label the new samples we will present to her as non-relevant samples since our sampling areas are fending away from the relevant area instead of approaching it. AIDE detects this scenario and restarts this exploration phase from the original $x$ sample but a lower $\gamma$ value. Using this pattern, we keep adapting our $\gamma$ value until we ``hit'' a relevant sample. Finally if the user is willing to give us a \emph{hint} about the minimum size of the relevant areas $s$ we can set the value $\gamma$ of our ``step'' to be equal to $\delta$, which guarantees to sample within the relevant range in exactly $d/\gamma$ iterations. %Including this new sample annotation allows us to discover very small areas in the data space with less user effort since the user does not have to label samples from lower exploration levels. In the experimental section we demonstrate the reduction in user labeling effort when we include this new class of feedback in the system.
%We discuss how we select this distance $\gamma$ further below. After we present the samples to the user
%The minimum number of tuples you need to search all directions in a 2 dimensional space is 4(up, down, left, right from the tuple) and you select these tuples in a distance x+-ฮณ from the initial tuple (the exact tuple at distance x might not exist thatโs why I use +-ฮณ). This is random sampling with 4 sampling areas.
%Multiple cases here:
%You got a yes, you are done.
%You got all noโs, need to get more samples around the weak-relevant in different distance x (you can decrease x. Why not increase? I have an answer: if you show him a tuple worst than the original โfurther from the relevant- he will say no and not maybe). Donโt think this case is very possible to happen though. You got a maybe or many maybes. Cool!
%Letโs say you got one more maybe, so now you have in total two maybes including the initial one. In this case, you search around the second maybe in a smaller distance x this time since the relevant area might be in between the two maybes or could be towards another direction.
%Note: an idea, not sure if we can use it: not all maybes are of the same significance. For example: a user sees a tuple says maybe is relevant, then we show him another one and he says this is better but still not good so maybe again. Of course we do not know which one is closer to what the user wants since both of them are just maybeโs to us.
%If we want to discover which one is closer to the relevant area we can look around this maybe (like a box around the maybe) and calculate a weight for the maybe based on the answer of the other tuples around and how close to this tuple they are. So for example, a tuple that has many maybes very close to it is more significant than another tuple that has maybes in a box but in further distance than it. In this way we can find out if we are going towards the right direction and examining tuples close to the most important maybe. This is like verification. It is an idea but I am not sure how we can use it.
%How do we save tuples?
%Two cases:
%1. Assume that if a tuple is at least distance d from the target then the user will label it a maybe.
%If grid size = d if you show tuples from all the grids you will get at least one maybe.
%In the general case where your grid size != d, whenever you get a no you know that within a distance d from the no there is no relevant area (if there was the no would be a maybe) so you donโt need to show samples from lower levels that are in a distance d from the no. Also, around the maybes in a distance d obviously there is no need to re-grid (you are already showing tuples close to the maybe).
%2. You discover as you go what is the distance d that satisfies the user and he labels tuples as maybe within that distance. After you discover one area, you can calculate how far was your furthest maybe from the relevant area. This means that now you know that if a tuple is at least distance d from the target then the user will label it as a maybe โif this is something that the user keeps stable, can we assume that? -so you can use it with the same logic as in the number %1.
%3. Ordering in which dimension is better. Can it be a dimension? How can we discover that? Re-gridding. Not sure about this.
%4. In the grids that we have a maybe there is no need to re-grid. Since you are exploring towards all the dimensions in that grid and you are already covering it.
%Conclusions:
%To detect very small areas, since with maybes we will not need re-grading.
\section{Experimental Evaluation}\label{s:experiments}
Next, we present experimental results from a micro-benchmark on the SDSS dataset~\cite{SloanSky} and from a user study.
\subsection{Experimental Setup: SDSS Dataset}
\begin{figure*}[t]
\centering
\subfigure[\scriptsize{Accuracy for increasing area size (1 area)}.]{
\includegraphics[totalheight=1.03in, angle=0]{1}
\label{f:fmeasure_size}}
\subfigure[\scriptsize{Accuracy for increasing number of areas (large areas)}.]{
\includegraphics[totalheight=1.03in, angle=0]{2}
\label{f:fmeasure_areas}}
\subfigure[\scriptsize{Time for increasing area size (1 area)}.]{
\includegraphics[totalheight=1.03in, angle=0]{3}
\label{f:time_size}}
\subfigure[\scriptsize{Comparison to random exploration for increasing area size ($>$70\% accuracy, 1 area).}]{
\includegraphics[totalheight=1.03in, angle=0]{4}
\label{f:random_size}}
\subfigure[\scriptsize{Comparison to random exploration for increasing number of areas ($>$70\% accuracy, large areas).}]{
\includegraphics[totalheight=1.03in, angle=0]{5}
\label{f:random_areas}}
\subfigure[\scriptsize{Impact of exploration phases (1 large area).}]{
\includegraphics[totalheight=1.03in, angle=0]{6}
\label{f:phase_impact}}
\vspace{-4mm}
\caption{\small{ Figures (a), (b) show AIDE's effectiveness, i.e., prediction accuracy. Figure (c) shows efficiency results, i.e., time overhead. Figures (d) and (e) compare AIDE with random exploration techniques. Figure (f) demonstrates the effectiveness of AIDE's exploration phases.}}
\vspace{-1mm}
\label{f:effectiveness}
\end{figure*}
We implemented our framework on JVM 1.7. In our experiments we used various Sloan Digital Sky Survey datasets (SDSS)~\cite{SloanSky} with a size of 10GB-100GB ($3\times 10^6 - 30 \times 10^6$ tuples). Our exploration was performed on combinations of five numerical attributes ({\tt rowc}, {\tt colc}, {\tt ra}, {\tt field}, {\tt fieldID}, {\tt dec}) of the {\tt PhotoObjAll} table. These are attributes with different value distributions, allowing us to experiment with both skewed and roughly uniform exploration spaces. A covering index on these attributes was always used. We used by default a 10GB dataset and a dense exploration space on {\tt rowc} and {\tt colc}, unless otherwise noted. All our experiments were run on an Intel PowerEdge R320 server with 32GB RAM using MySQL. We used Weka~\cite{Weka_url} for executing the CART~\cite{breinman84} decision tree algorithm and the $k$-means clustering algorithm. All experiments report averages of ten exploration sessions.
{\bf Target Queries} AIDE characterizes user interests and eventually ``predicts'' the selection predicates that retrieve her relevant objects. We focus on predicting range queries (we call them \emph{target queries}) and we vary their complexity based on: a) the number of disjunctive predicates they include (\emph{number of relevant areas}) and b) the data space coverage of the relevant areas, i.e., the width of the range for each attribute (\emph{relevant area size}). Specifically, we categorize relevant areas to \emph{small}, \emph{medium} and \emph{large}. Small areas have attribute ranges with average width of 1-3\% of their normalized domain, while medium areas have width 4-6\% and large ones have 7-9\%. We also experimented with queries with a single relevant area (conjunctive queries) as well as complex disjunctive queries that select 3, 5 and 7 relevant areas. The higher the number of relevant areas and the smaller these areas, the more challenging is to predict them. % (for comparison reason we converted attribute domains to normalized domains that have a range of [1,100]).
The diversity of our target query set is driven by the query characteristics we observed in the SDSS sample query set~\cite{SloanSky-Queries}. Specifically, 90\% of their queries select a single area, while 10\% select only 4 areas. Our experiments cover even more complex cases of 5 and 7 areas. Furthermore, 20\% of the predicates used in SDSS queries cover 1-3.5\% of their domain, 3\% of them have coverage around 13\%, and 50\% of the predicates have coverage 50\% or higher while the median coverage is 3.4\%. Our target queries have domain coverage (i.e., the relevant area size) between 1-9\% and our results demonstrate that we perform better as the size of the areas increases. Hence, we believe that our query set has a good coverage of queries used in real-world applications while they also cover significantly more complex cases.
{\bf User Simulation} Given a target query, we simulate the user by executing the query to collect the exact \emph{target set }of relevant tuples. We rely on this set to label the new sample set we extract in each iteration as relevant or irrelevant depending on whether they are included in the target set. We also use this set to evaluate the accuracy ($F$-measure) of our final predicted extraction queries.
{\bf Evaluation Metrics} We measure the accuracy of our approach using the $F$-measure (Section~\ref{s:problemdef}) of our final data extraction predictions and report the number of labeled samples required to reach different accuracy levels.
{Our efficiency metric is the \emph{system execution time} (equivalent to {\em user wait time}), which include the time for the space exploration, data classification, and sample extraction.
We may also report the total {\em exploration time}, which includes both the system execution time and the sample reviewing time by the user. }
%{\bf Results Overview} {Our study showed that AIDE can deliver highly accurate query predictions for very common conjunctive queries with small user effort. By requesting feedback on only 215 out of $ 3 \times 10^6$ objects in a 10GB database, AIDE predicts large relevant areas with accuracy higher than 60\% (with 500 samples we have an accuracy higher than 90\%). In this case, the user needs to label only 0.4\% of the total relevant objects and 0.01\% of the irrelevant objects in the database (less in the 100GB database). Given a reasonable number of labeled samples, AIDE can also achieve high accuracy for more complex exploration tasks (i.e., disjunctive queries and small relevant areas).}
%
%{Furthermore, AIDE offers interactive performance. By operating on sample datasets it limits the user wait time to less than 3 seconds per iteration in average, while the total system execution time is only 13 seconds for the 10GB, 1.3 minutes for the 50GB, and 3.2 minutes for the 100GB dataset. Finally, our user study shows the practicality of AIDE: it offered a significant improvement over manual exploration, the current state-of-the-art, reducing the user labeling effort by up to 87\%, with an average of 66\% reduction, as well as the total exploration time by 47\% in average.
%%reducing the user labeling effort by 66\% in average and the total exploration time by 44\% in average.
%}
\subsection{Effectiveness \& Efficiency of AIDE}
Figure~\ref{f:fmeasure_size} shows AIDE's effectiveness when we increase the query complexity by varying the size of relevant areas from large (\emph{AIDE-Large}) to medium (\emph{AIDE-Medium}) and small (\emph{AIDE-Small}). Our queries have one relevant area which is the most common range query in SSDS. Naturally, labeling more samples improves in all cases the accuracy. As the query complexity increases the user needs to provide more samples to get the same level of accuracy. By requesting feedback on only 215 out of $ 3 \times 10^6$ objects AIDE predicts large relevant areas with accuracy higher than 60\% (with 350 samples we have an accuracy higher than 80\%). In this case, the user needs to label only 0.4\% of the total relevant objects and 0.01\% of the irrelevant objects in the database. Furthermore, AIDE needs 345 samples to predict medium areas and 600 samples for small areas to get an accuracy of at least 60\%. %Queries with a large area converge to an accuracy higher than 80\% with only 350 samples, while even 120 samples offer accuracy higher than 50\%.
\emph{Hence, AIDE decreases the user effort (i.e., reviewing objects) to a few 100's samples compared with the state-of-the-art ``manual'' exploration which involves examining 1000's of objects (e.g., target queries return 26,817-99,671 relevant objects depending on the size of the relevant areas)}.
We also increased the query complexity by varying the number of areas from one (1) to seven (7). Figure~\ref{f:fmeasure_areas} shows our results for the case of large relevant areas. While AIDE can perform very well for common conjunctive queries (i.e., with one (1) relevant area), to accurately predict highly complex disjunctive queries more samples are needed. However, even for highly complex queries of seven (7) areas we get an accuracy of 60\% or higher with reasonable number of samples (at least 450 samples).
Figure~\ref{f:time_size} shows the execution time overhead (seconds in average per iteration). In all cases, high accuracy requires the extraction of more samples which increases the exploration time. The complexity of the query (size of relevant areas) also affects the time overhead. Searching for larger relevant areas leads to more sample extraction queries around the boundaries of these relevant areas. However, our time overhead is acceptable: to get an accuracy of 60\% the user wait time per iteration is less than one second for small and medium areas, and 1.02 second for large areas, while to get highly accurate predictions (90\%-100\%) the user experiences 4.79 second wait time in average. To reach the highest accuracy ($>$ 90\%) AIDE executed 23.7 iterations in average for the large areas, 37 iterations for the medium and 33.4 iterations for the small areas.
{\bf Comparison with Random Exploration} Next we compared AIDE with two alternative exploration techniques. \emph{Random} randomly selects 20 samples per iteration, presents them to the user for feedback and then builds a classification model. \emph{Random-Grid} is similar to Random but the sample selection is done on our exploration grid, i.e., it selects one random sample around the center of each grid cell. This allows our samples to be evenly distributed across the exploration space. This approach also collects 20 samples per iteration. AIDE also limits the number of new samples it extracts per iteration: we calculated the number of samples needed for the boundary and the misclassified exploitation and we used the remaining out of 20 samples to sample grid cells. %This slightly increases our number of iterations. %Both AIDE and Random-Grid start from a pre-build exploration level with grid cell size of 10 (i.e., $\beta=10$, see Section~\ref{s:exploration}) and continue to exploration levels with grid cell sizes of 5 and 2.5\% for as long as there are still samples to be labeled.
\begin{figure*}[t]
\subfigure[\scriptsize{{Accuracy for skewed data sets} ( $>$70\% accuracy, 1 large area).}]{
\includegraphics[totalheight=1.15in, angle=0]{skewAware}
\label{f:hybrid_fmeasure}}
%\subfigure[\scriptsize{Accuracy for skewed data sets ($>$70\% accuracy, 1 large area).}]{
%\includegraphics[totalheight=1.15in, angle=0]{time-dataDistributions}
%\label{f:hybrid_time}}
\subfigure[\scriptsize{{Accuracy of probabilistic sampling} ($>$80\% accuracy, 1 area).}]{
\includegraphics[totalheight=1.15in, angle=0]{probabilisticFmeasure}
\label{f:uncertainty_fmeasure}}
\subfigure[\scriptsize{{Time for probabilistic sampling ($>$80\% accuracy, 1 area).}}]{
\includegraphics[totalheight=1.15in, angle=0]{probabilisticTime}
\label{f:uncertainty_time}}
\subfigure[\scriptsize{Accuracy for extended feedback model ($>$70\% accuracy, 1 area). }]{
\includegraphics[totalheight=1.15in, angle=0]{extendedFeedback_fmeasure}
\label{f:maybeFmeasure}}
\subfigure[\scriptsize{Accuracy for multi-dimensional exploration spaces ($>$70\% accuracy, large areas).}]{
\includegraphics[totalheight=1.15in, angle=0]{10}
\label{f:5D_fmeasure}}
\subfigure[\scriptsize{Time for multi-dimensional exploration spaces ($>$70\% accuracy, large areas).}]{
\includegraphics[totalheight=1.15in, angle=0]{11}
\label{f:5D_time}}
%\subfigure[\scriptsize{Time for extended feedback model ($>$70\% accuracy, 1 area).}]{
%\includegraphics[totalheight=1.15in, angle=0]{extendedFeedback_time}
%\label{f:maybeTime}}
\vspace{-4mm}
\caption{\small{Impact of performance optimizations: (a) Evaluation of the skew-aware exploration (Section~\ref{s:explore_skewed}),(b)-(c) Evaluation of the probabilistic sampling (Section~\ref{s:uncertainty}), (d) Evaluation the similarity feedback model (Section~\ref{s:maybe}). Figures (e)-(f) study multi-dimensional exploration spaces.}}
\label{f:optimizations}
\vspace{-4mm}
\end{figure*}
%For each pre-defined grid, both approaches use already cached random samples from each grid cell, therefore eliminating the need for extraction queries during the object discovering phase. In these experiments we set the number of samples to belong to two classes $N=800$ and $N=1600$.
Figure~\ref{f:random_size} shows the number of samples needed to achieve an accuracy of at least 70\% when our target queries have one (1) relevant area and varying size. AIDE is consistently highly effective: it requires only 308 samples for large areas and 365 and 623 samples in average for medium and small samples, respectively. Both random exploration approaches cannot discover small and medium areas with that few samples. Random fails to discover small areas of interest even when we increase the labeled set to 6,400 samples, while Random-Grid needs 5,457 samples in average for these complex queries. Random can identify medium and large relevant areas with 70\% accuracy when given at least 2,690 and 1,180 samples respectively. Random-Grid is also highly ineffective, since it needs 1,380 and 1,275 samples in average for medium and large areas. Figure~\ref{f:random_areas} shows the number of samples to achieve at least 70\% accuracy when varying the number of target relevant areas. AIDE consistently requires less samples (less than 500 samples for all cases) than Random and Random Grid (more than 1,000 samples in almost all cases). \emph{Hence, AIDE outperforms random sampling over all unlabeled objects since it samples only promising exploration sub-areas, leading to highly accurate results with less sampled data. }
{\bf Impact of Exploration Phases} We also studied the impact of each exploration phase independently. Figure~\ref{f:phase_impact} compares the number of samples we need to reach different accuracy levels for queries with one large relevant area. We compare AIDE with two variants: one that uses only the object discovery phase (\emph{Random-Grid}) and one that adds only the misclassified exploitation phase (\emph{Random-Grid+Misclassified}). The results show that combining all three phases gives the best results. Specifically, using only the object discovery phase requires consistently more than 1,000 samples to get an accuracy greater than 40\%. Adding the misclassified exploitation phase reduces the sample requirements by 60\% in average while adding the boundary exploitation phase allows us to achieve higher accuracy with 42\% less samples in average. \emph {Hence, combining all three phases is highly effective in predicting relevant areas while reducing the amount of user effort. }
\subsection{Skewed Exploration Spaces}
We also studied AIDE in the presence of skewed exploration spaces. We experimented with three types of 2-dimensional exploration spaces: (a) \emph{Uniform} where we use two roughly uniform domains ({\tt rowc}, {\tt colc}), (b) \emph{Hybrid} that includes one skewed ({\tt dec}) and one uniform domain ({\tt rowc}) and (c) \emph{Skewed} that uses two skewed domains ({\tt dec}, {\tt ra}). We also experimented with the density of the target queries: (a) \emph{Dense}queries involve dense relevant areas and (b) \emph{MixQ} queries cover both sparse and dense ranges of the relevant domains. Figure~\ref{f:hybrid_fmeasure} shows the number of samples needed to achieve accuracy greater than 70\% for queries with one large relevant area. We compare three variants of our system: (a) AIDE-Grid that uses the grid-based technique for the relevant object discovery phase, (b) \emph{AIDE-Clustering} that uses only clustering-based sampled for skewed distributions but not sampling within grid cells and (c) \emph{AIDE-SkewAware} that is a hybrid of the two previous techniques as described in Section~\ref{s:explore_skewed}. %(Section~\ref{s:explore_hybrid}).
%We measure the skewness of an attribute domain as the non-parametric skew $(\mu-\nu)/\sigma$, where $\mu$ is the mean, $\nu$ the median and $\sigma$ the standard deviation of the domain. The non-parametric skew lies between -1 and +1 and the range shows where the mean lies within one standard deviation of any median. We used two attributes {\tt rowc} and {\tt colc} with lower skew values of 0.0003 and -0.011 respectively, as well as the attributes {\tt dec} and {\tt ra} with higher values for skew: 0.31 and -0.47.
%{The results show that the clustering optimization requires 87\% less samples than the grid-based approach when predicting dense relevant areas within highly skewed data spaces.} This is because both the cluster and our relevant areas are concentrated in dense sub-areas while grid cells are created uniformly across the data space. This allows AIDE-Clustering to sample smaller, finer-grained areas than the grid-based approach, eliminating the need to zoom into the next exploration level. When the distribution is uniform (NoSkew) clusters and grid cells are highly aligned providing roughly the same results. In the hybrid case (HalfSkew) our relevant areas and our grid-cells cover both dense and sparse areas. The clustering technique though creates most of its clusters on the dense areas and hence fails to discover relevant objects in the sparse ones. It therefore has to zoom in into finer exploration levels and it required 73\% more samples to converge to the same accuracy as the grid-based technique. This result indicates that a hybrid approach could be the best strategy for the relevant object discovery phase. AIDE would be initialized with the clustered approach to explore first dense areas. When the users interests are partially revealed the system could switch to the grid-based approach if these interests appear to lie on sparse areas. We plan to explore this in a future extended version of the paper.
%\begin{figure}[t]
%\centering
%\subfigure[\scriptsize{Accuracy for skewed data sets ( $>$70\% accuracy, 1 large area).}]{
%\includegraphics[totalheight=1.55in, angle=0]{newfigs/hybrid}
%\label{f:hybrid_fmeasure}}
%\subfigure[\scriptsize{Accuracy for skewed data sets ($>$70\% accuracy, 1 large area).}]{
%\includegraphics[totalheight=1.55in, angle=0]{newfigs/time-dataDistributions}
%\label{f:hybrid_time}}
%\vspace{-4mm}
%\caption{\small{AIDE's performance on multi-dimensional exploration spaces (Figures (a-d)) and on skewed exploration spaces (Figures (e-f)). }}
%\label{f:optimizations}
%%\vspace{-4mm}
%\end{figure}
%Figures (d-e-f) demonstrate the effectiveness of our optimizations.
The results show that AIDE-SkewAware works best under any combination of query density and exploration space distribution. When the distribution is uniform (Uniform) clusters and grid cells are highly aligned providing roughly the same results for all three techniques. Note that in this case all our relevant areas will be dense. In the highly skewed data space (Skewed) we also used only dense relevant areas as the sparse areas were practically non populated. Here, both the clustering-based technique and the skew-aware technique outperform the grid-based approach requiring 87\% less samples. This is because clusters are formulated in the dense sub-space while grid cells are created uniformly across the data space covering non populated exploration areas. This allows AIDE-Clustering and AIDE-SkewAware to sample smaller, finer-grained areas than the grid-based approach, eliminating the need to zoom into the next exploration level.
Finally, for the case of hybrid distributions (Hybrid) we picked our relevant area to cover both dense ranges (for the uniform domain) and sparse ranges in the skewed domain, resulting to our mixed query case (MixQ). Here, the clustering technique creates most of its clusters on the dense areas and hence fails to discover relevant objects in the sparse ones. It therefore has to zoom into finer exploration levels and it requires 73\% more samples to converge to the same accuracy as the grid-based technique. However, AIDE-SkewAware samples both the dense areas where the clusters are located and the sparse areas which are covered by the grid cell and it discovers the relevant area. \cut{Furthermore, Figure~\ref{f:hybrid_time} compares the execution time overhead (seconds in average per iteration) for the three techniques. We can observe that all three techniques perform similarly in terms of efficiency. In all cases, the user wait time is fairly low; less than 2.1 seconds per iteration.}
\emph{We conclude that combining sampling within clusters and grid cells is the best strategy for exploring both skewed and non skewed domains}.
%\subsection{Effectiveness of Optimizations}
%%\subsection{Clustered-based Misclassified Exploitation}
%
%Next we study the impact of our proposed optimizations. % used in each space exploration phase. %Our results indicate the these techniques achieved an improvement on the exploration effectiveness (accuracy) as well as its efficiency (exploration time).
%Figures (c-d) show the effectiveness and efficiency of uncertainty sampling in a 3 dimensional space.
%\subfigure[\scriptsize{\todo{new graph} Uncertainty sampling in higher exploration space ($>$80\% accuracy, 1 relevant area, 3D exploration space).}]{
%\includegraphics[totalheight=1.05in, angle=0]{newfigs/uncertainty_attr}
%\label{f:uncertainty_attr}}
%\subfigure[\scriptsize{\todo{new graph} Time for uncertainty sampling in higher exploration space ($>$70\% accuracy, 1 relevant area, 3D exploration space). }]{
%\includegraphics[totalheight=1.15in, angle=0]{newfigs/uncertainty_attr_time}
%\label{f:uncertainty_attr_time}}
%This result indicates that a hybrid approach that combines both exploration techniques could be the best strategy for the relevant object discovery phase. In such a solution, AIDE would be initialized with the clustered based approach (since it performance well in both skewed and no skewed distributions) and when the users interests are partially revealed (some relevant areas are already discovered) the system could switch to the grid-based approach if these interests seem to lie on sparse areas. We plan to explore such techniques in a future extended version of the paper.
%\emph{Our experiments showed that even in the presence of skew in the exploration space AIDE needs a small number of samples to accurately predict complex queries. } As expected, the higher the skew of the underlying data set the more labeled samples AIDE needs to reach a certain accuracy level. Specifically, in average, the exploration of skewed data sets required only 194 more samples than the half skewed ones and 278 more than the non skewed ones. This is due to the fact that in skewed distributions the probability of discovering relevant objects is lower, leading to more iterations and hence to more labeling effort from the user. However, in all cases AIDE was able to deliver a high accuracy with a reasonable number of samples. \emph{Our experiments also revealed that exploration on sampled datasets does not affect the accuracy of our results even in the presence of skewed attributes} (\emph{Skew-Sample, HalfSkew-Sample, NoSkew-Sample}). This is because our sampled dataset preserves the distribution of the exploration domains. This result is consistent for queries of different complexity. % (i.e., with more relevant areas and different size of relevant areas).
%For the skewed distributions the relevant area was picked to be in a sub-space with skewness of -0.02 in the {\tt decl} attribute and -0.09 in the {\tt ra} attribute \kiki{maybe we should take this out? since our subspace of the query is not that skewed, maybe this is not a good thing}.
\cut{
{\bf Distance-based Hints} The user can optionally specify a lower bound for the sizes of elevant areas (see Section~\ref{s:boundaryoptimizations}). In Figure~\ref{f:distanceHint} we show the results when the user has specified that the area width along each dimension will be at least 4\% on the normalized domains (\emph{AIDE+DistanceHint}). We compare it with the regular AIDE with no hints. These results are on medium relevant areas and we vary the number of areas. AIDE+DistanceHint performs better in all cases: to reach an accuracy higher than 70\% we need in average 656 samples which is 14\% less samples than AIDE. The hint allows us to know the exact exploitation level to use in order to guarantee that from the first iteration the object discovery phase will ``hit'' all relevant areas. Hence, it eliminates the need to spend samples exploring more fine-grained exploration levels.
{\bf Clustered-based Misclassified Exploitation} Next we compare the time overhead when the clustering-based misclassified exploitation is used (\emph{SamplePerCluster}) with the approach that defines one sampling area for each misclassified object (\emph{SamplePerMisclassified}). Here, we used queries with a different number of large relevant areas and we show the exploration time for reaching an accuracy of at least 70\%. Figure~\ref{f:cluster} demonstrates that the clustering approach can improve the time overhead by 45.6\% in average, since it creates one sampling area (i.e., issues one sample extraction query) per cluster. We note that the accuracy was not affected by incorporating this optimization (we needed in average 2\% more tuples (15 tuples) to reach the same $F$-measure).
{\bf Adaptive sample size} In Figure~\ref{f:sampleSize} we compare the accuracy when keeping the sample size in the boundary exploitation phase fixed (\emph{SampleSize-Fixed}) with adapting the size based on the changes of the decision tree between iterations (\emph{SampleSize-Adaptive}). Our queries select an increasing number of disjoint large areas and we report the accuracy we achieve when the user labels 500 samples. We can observe that our accuracy improved by an average of 12\%. This is due to the fact that our strategy reduces the number of samples we collect through boundary exploitation, therefore requesting feedback on more samples collected by the other two phases. These two phases (relevant object discovery and misclassified exploitation) have a higher impact on the $F$-measure, therefore our accuracy is increased. }
\subsection{Probabilistic Sampling}
Next, we examine the effectiveness and efficiency of the probabilistic sampling technique (Section~\ref{s:uncertainty}). In Figure~\ref{f:uncertainty_fmeasure} we measure the number of samples needed to reach an $F$-measure greater than 80\% when the probabilistic sampling technique in the misclassified exploitation phase (\emph{AIDE+Probabilistic}). The experiments shows the results when we increase size of the relevant area from small areas to medium and large areas. \emph{AIDE requires less labeled samples to reach an accuracy when using the uncertainty sampling technique.} In average this new approach can reduce the user effort by 21\%. This confirms our hypothesis that some samples in the misclassified sampling area are more informative than others and they can be leveraged to improve the user's experience.
We also studied the overhead of this approach. In Figure~\ref{f:uncertainty_time} we can observe that the uncertainty sampling technique slightly increases our user wait time per iteration in all cases. This is because in each iteration we have to extract all samples within the sampling area, calculate its posterior probability and decide whether to present it to the user or not. The user wait time per iteration was increased by 25\% in average. However, in all cases the time overhead was less than 1.8 seconds which should not affect the user's interactive experience. This is because our technique searches for the most informative samples only within a small sub-set of the overall exploration space.
%This is because for large area sizes the uncertainty sampling technique selected samples from the misclassified sampling area that were all clustered very close to each other occuping a small portion of the relevant area. During the boundary exploitation phase the collected samples were all clustered together far from the relevant area. %In total, the labeled samples that were collected did not give enough information to the data classification algorithm to build an accurate model using less samples. However, for small and medium areas the misclassified sampling area highly overlapped witht the relevant area and the samples collected occupied a good portion of the relevant area thus the data classification algorithm was able to build an accurate model with less samples. Uncertainty sampling helped reduce the user effort by 28\% for small and medium areas while it increased the effort by 22\% for large areas.
%Figure~\ref{f:uncertainty_attr} shows the effectiveness of the uncertainty sampling technique for a varying relevant area size in a 3 dimensional space, where one dimension is irrelevant to the user interests. The challenge is for the uncertainty sampling technique to find and eliminate the extra dimension from the predicted relevant area. The graph shows that to reach an $F$-measure $>$ 70\% the uncertainty sampling technique required in average 18\% more samples than AIDE due to the extra irrelevant attribute. While previously the random samples were well distributed in the domain, the boundary exploitation phase now focuses the sampling in a specific area of the boundaries. This negatively affects AIDE's ability to find and eliminate the irrelevant attribute. Moreover, in Figure~\ref{f:uncertainty_attr_time} the results show that uncertainty sampling increased the average user wait time per iteration by an average of 95\%. This is due to the large number of samples fetched from the database (around 98,400 in each iteration) and the overhead of examining each one of them before deciding which one to present to the user.
%
%\begin{figure}[t]
%\centering
%\subfigure[\scriptsize{Effectiveness of probabilistic sampling ($>$80\% accuracy, 1 area).}]{
%\includegraphics[totalheight=1.55in, angle=0]{newfigs/uncertainty_fmeasure}
%\label{f:uncertainty_fmeasure}}
%\subfigure[\scriptsize{Time for probabilistic sampling ($>$80\% accuracy, 1 area).}]{
%\includegraphics[totalheight=1.55in, angle=0]{newfigs/uncertainty_time}
%\label{f:uncertainty_time}}
%%\vspace{-4mm}
%\caption{\small{Effectiveness and time overhead of probabilistic sampling. }}
%\label{f:optimizations}
%%\vspace{-4mm}
%\end{figure}
\subsection{Similarity Feedback Model}
We also studied the effect of extending our relevance feedback model to include labels for similar but not necessarily relevant attributes. Here, we label as ``similar'' samples that are within distance less than 10\% from an actual relevant object (this distance is measured in any of the exploration dimensions). Otherwise we label it as irrelevant.
Figure~\ref{f:maybeFmeasure} compares AIDE's effectiveness when using the binary feedback approach and the extended feedback model. Here, we vary the size of the target relevant area from small up to large and we measure the number of samples AIDE needs to reach an $F$-measure higher than 70\%. \emph{The results indicate that annotating the similarity of objects can significantly reduce the labeling effort of the user}. This improvement is 42\% in average across all area sizes. This feedback is particularly useful in the case of the small relevant areas where the user effort can be significant. Here, the user's ``similar'' annotations steer the exploration towards the direction of the relevant samples and the labeling effort is significantly reduced. We also measured the impact of our model on the user wait time and in all cases was under 0.1 seconds which should be unnoticeable by the user in our interactive system. We omit the graph due to space limitations. %In this way, the user discovers his relevant area without labeling samples from lower exploration levels and his labeling effort is significantly reduced.
\cut{
Figure~\ref{f:maybeTime} compares the average user wait time per iteration when using the binary and the extended relevance feedback model. As we can see, the improvement in user effort with the relevance feedback model comes with a small increase in the average user wait time per iteration. This overhead is due to the extra sampling queries that we need to execute during the extended feedback exploitation phase to sample the areas around our ``maybe'' samples. Specifically, the user wait time increased in average by 16\% but in all cases it was under 0.1 seconds which should be unnoticeable by the user in our interactive system.
%These results are on a 10\% random sample database of the original 10GB database.}
}
\subsection{Scalability}
{\bf Database Size} Figure~\ref{f:fmeasure_dbsize} shows AIDE's accuracy with a given number of labeled samples for dataset sizes of 10GB, 50GB and 100GB. Our target queries have one large relevant area and the average number of relevant objects increases as we increase the size of the dataset (our target query returns in average 26,817 relevant objects in the 10GB, 120,136 objects in the 50GB and 238,898 objects in the 100GB database). AIDE predicts these objects in all datasets with high accuracy without increasing the user's effort. \emph{We conclude that the size of the database does not affect our effectiveness.} AIDE consistently achieves high accuracy of more than 80\% on big data sets with only a few hundred samples (e.g., 400 samples). These results were consistent even for more complex queries with multiple relevant areas. % and therefore we omit these graphs due to space limitations.
\begin{figure*}[t]
\centering
\subfigure[\scriptsize{Accuracy for increasing data set size (1 large area).}]{
\includegraphics[totalheight=1.03in, angle=0]{7}
\label{f:fmeasure_dbsize}}
\subfigure[\scriptsize{Impact of space reduction on accuracy and time overhead (1 large area).}]{
\includegraphics[totalheight=1.03in, angle=0]{8}
\label{f:fmeasure_sampledb}}
\subfigure[\scriptsize{Time improvement of sampled datasets and increasing number of areas ($>$70\% accuracy, large areas).}]{
\includegraphics[totalheight=1.03in, angle=0]{9}
\label{f:timeareas_sampledb}}
\vspace{-3mm}
\caption{\small{Figure (a) shows AIDE's effectiveness on big data sets and Figures (b-c) show the impact of our exploration space reduction (b-c).}} %Figure (a) shows AIDE's accuracy for increasing data sets. Figure (b)-(c) show the impact on accuracy and time overhead of running AIDE on sampled data sets.}}
\label{f:scalability}
\vspace{-4mm}
\end{figure*}
{\bf Exploration Space Reduction} Applying our techniques to larger datasets increases the time overhead since our sampling queries have higher response times. One optimization is to execute our exploration on a sampled database (Section~\ref{s:sparereduction}). In this experiment,\cut{we applied our exploration on sampled databases with size 10\% of the size of the original ones and we evaluated the impact on our effectiveness and efficiency. Specifically, } we sampled datasets of 10GB, 50GB, 100GB and generated the 10\% sampled datasets of 1GB, 5GB and 10GB, respectively. Figure~\ref{f:fmeasure_sampledb} shows the absolute difference of the final accuracy (\emph{10GB-Accuracy, 50GB-Accuracy, 100GB-Accuracy}) when AIDE is applied on the sampled and on the total datasets. The average difference is no more than 7.15\% for the 10GB, 2.72\% for the 50GB and 5.85\% for the 100GB data set. In the same figure we also show the improvement of the system execution time (\emph{{10GB-Time, 50GB-Time, 100GB-Time}}). For 10GB (and a sampled dataset of 1GB) this time is reduced by 88\% in average, while for the larger datasets of 50GB and 100GB it is reduced by 96\%-97\%.
Figure~\ref{f:timeareas_sampledb} shows the improvement of the system execution time when AIDE runs over the sampled data sets and we increase the number of relevant areas. Here, we measure the improvement of the system execution time when we reach an accuracy higher than 70\% . The average time per iteration is 2.8 seconds for the 10GB, 37.7 for the 50GB and 111 for the 100GB database. By operating on the sampled datasets we improved our time by more than 84\% while our average improvement for each query type was more than 91\%. Our improved iteration time is 0.37 second for the 10GB, 2.14 seconds for the 50GB and 5.3 seconds for the 100GB dataset, in average. {The average number of iterations is 37 and hence AIDE offers a total execution time of 13secs for the 10GB, 1.3mins for the 50GB and 3.2mins for the 50GB dataset while the user wait time is less than 3secs per iteration in average. \emph{Hence, AIDE can scale to big datasets by applying its techniques on sampled datasets. This incurs very low impact on the accuracy while it significantly improves the system execution time. }}
%\begin{figure*}[t]
%\centering
%\subfigure[\scriptsize{Accuracy for multi-dimensional exploration spaces ($>$70\% accuracy, large areas).}]{
%\includegraphics[totalheight=1.03in, angle=0]{newfigs/10}
%\label{f:5D_fmeasure}}
%\subfigure[\scriptsize{Time for multi-dimensional exploration spaces ($>$70\% accuracy, large areas).}]{
%\includegraphics[totalheight=1.03in, angle=0]{newfigs/11}
%\label{f:5D_time}}
%\subfigure[\scriptsize{Accuracy for skewed data sets ( $>$70\% accuracy, 1 large area).}]{
%\includegraphics[totalheight=1.03in, angle=0]{newfigs/hybrid}
%\label{f:hybrid_fmeasure}}
%\subfigure[\scriptsize{Accuracy for skewed data sets ($>$70\% accuracy, 1 large area).}]{
%\includegraphics[totalheight=1.15in, angle=0]{newfigs/time-dataDistributions}
%\label{f:hybrid_time}}
%\subfigure[\scriptsize{Effectiveness of probabilistic sampling ($>$80\% accuracy, 1 area).}]{
%\includegraphics[totalheight=1.15in, angle=0]{newfigs/uncertainty_fmeasure}
%\label{f:uncertainty_fmeasure}}
%\subfigure[\scriptsize{Time for probabilistic sampling ($>$80\% accuracy, 1 area).}]{
%\includegraphics[totalheight=1.15in, angle=0]{newfigs/uncertainty_time}
%\label{f:uncertainty_time}}
%%\subfigure[\scriptsize{Accuracy for multi-dimensional relevant areas ($>$70\% accuracy, large areas).}]{
%%\includegraphics[totalheight=1.03in, angle=0]{newfigs/multidimensionalFmeasure}
%%\label{f:5D_query}}
%
%%\vspace{-4mm}
%\caption{\small{AIDE's performance on multi-dimensional exploration spaces (Figures (a-d)) and on skewed exploration spaces (Figures (e-f)). }}
%\label{f:optimizations}
%%\vspace{-4mm}
%\end{figure*}
{\bf Exploration Space Dimensionality} Figure~\ref{f:5D_fmeasure} shows the number of samples to reach an accuracy greater than 70\% as we increase the complexity of our queries (the number of relevant areas) and the size of our exploration space from 2-dimensional to 5-dimensional. These results are on large size areas and on the sampled datasets. Our target queries have conjunctions on two attributes and the main challenge for AIDE is to identify in the 3D, 4D and 5D spaces only the two relevant attributes.\emph{AIDE correctly identifies the irrelevant attributes and eliminates them from the decision tree classifier and hence from the final output query. Furthermore, although the exploration over more dimensions requires naturally more samples to reach an acceptable accuracy, the number of samples only increases by a small percentage} (the 3D space and 4D space require in average 13\% more tuples than the 2D space and the 5D space requires 32\% more tuples than the 2D space) and they remain within the range of 100's even for the complex cases of 7 areas and 5-dimensional exploration space. Figure~\ref{f:5D_time} shows that even for the very complex case of seven (7) relevant areas the time overhead is always less than 4.5 seconds, while for the less complex queries of 1 area the time drops below 1 second. {These results reveal a small increase in the user's wait time as we add more dimensions (each new dimension adds in average 0.7 seconds overhead to the previous one) but always within acceptable bounds. }
%
%{\bf Relevant Area Dimensionality} We also experimented with increasing the dimensionality of the relevant areas. For the experiment in Figure~\ref{f:5D_query} we use an exploration space of 5 dimensions and we vary the number of interesting attributes to the user from 2 to 5. These interests can be expressed as range queries with 2 to 5 disjunctions within each conjunction. Furthermore, we increase the complexity of our queries (the number of relevant areas) from 1 to 7. Our goal was to test AIDE effectiveness and efficiency in predicting highly complex target queries. In this experiments, we ensure that queries with the same number of disjunctions (i.e., number relevant areas) have the same selectivity.
%
%\cut{
%Here we ensured that all queries with the same number of relevant areas select approximately the same number of relevant samples. All queries that select 1 relevant area contain around 26,817 samples, queries with 3 areas select 81,539 samples, queries with 5 areas 136,445 and queries with 7 areas 191,522.}
%
%The challenge for AIDE is to eliminate the irrelevant attributes when the user is interested in 2 to 4 attributes and in the case were the user is interested in all of the 5 attributes of the exploration space the challenge is to include in the predicted query all the relevant attributes. The experiments showed that AIDE identifies the relevant attributes even in the presence of high dimensionality in both the exploration space and the relevant areas. \cut{Even for the most complex case where AIDE needs to formulate boundaries in 5 different dimensions for 7 distinctive relevant areas, it does so by requiring acceptable effort by the user: 980 sample labels - only \cut{0.0003\% of the entire samples in the database and} 0.04\% of the total set of relevant objects.}
%\cut{Furthermore, Figure~\ref{f:attributesTime} shows that as we increase the complexity of the target query %by adding more relevant attributes and relevant areas to it
%the wait time overhead of the user increases. However, even for the very complex case where the user is interested 7 5-dimensional areas the average user wait time per iteration is less than 11 seconds which should be unnoticable by the user in our interactive system.}
\subsection{User Study Evaluation}\label{s:userstudy}
%We chose this ``intuitive" dataset in the user study, as opposed to the previous SDSS dataset, for the following reason:
%SDSS comes with a large set of data as well as search queries written by scientists, which facilitate our simulation of user true interests and their labeling behaviors in the microbenchmarks. For the user study, however, we look for a domain for which it is easy to identify a group of users with sufficient understanding of the domain.
%(where the available query set~\cite{SloanSky-Queries} facilitated our simulation of true user interests and their labeling behaviors in our previous experiments).
Our user study used the AuctionMark dataset~\cite{AuctionMark} that includes information on auction items and their bids.
We chose this ``intuitive" dataset, as opposed to the SDSS dataset, because the user study requires identifying a significant number of users with sufficient understanding of the domain.
Thus, AuctionMark meets the requirement: we were able to identify a group of computer science graduate students with SQL experiences and designed their exploration task to be ``identifying auction items that are good deals''.
Note that the exploration task should not be trivial, i.e., users should not have an upfront understanding of the exact selection predicates that would collect all relevant objects.
%We departed from our SDSS dataset as it was difficult to identify users with sufficient understanding of the astronomical domain. Our participants were computer science graduate students with SQL experience and their exploration task was to ``identify auction items that were good deals''. We note that exploration tasks should not be trivial, i.e., users should not have an upfront understanding of the exact selection predicates that would collect all relevant objects.
The exploration data set had a size of 1.77GB and it was derived from the ITEM table of AuctionMark benchmark. It included seven attributes: initial price, current price, number of bids, number of comments, number of dates an item is in an auction, the difference between the initial and current item price, and the days until the auction is closed for that item. Each user explored the data set ``manually", i.e., iteratively formulating exploratory queries and reviewing their results until they obtained a query, $Q$, that satisfied their interests.
We then took $Q$ as the true interest of a user and used it to simulate the user labeling results in AIDE. We measured how well AIDE can predict $Q$.
%We then used AIDE for the same exploration task and we measured how well it can predict the user's final query.
%We then used AIDE for the same exploration task and we measured how well it can predict $Q$.
The results demonstrated that AIDE was able to reduce the user's reviewing effort by 66\% in average (\textit{Reviewing savings} column in Table~\ref{tab:userstudytable}). Furthermore, with the manual exploration users were shown 100s of thousands objects in total (\textit{Manual returned objects}) while AIDE shows them only a few hundred strategically selected samples. Furthermore, with the manual exploration our users needed about an hour to complete their task (\emph{Manual time}). Assuming that the most of this time was spent on tuple reviewing, we calculated the average tuple reviewing for each user. This varied significantly across users (3secs - 26secs). Using this time we estimated the total exploration time needed by AIDE including the reviewing effort (\emph{AIDE time}). AIDE was able to reduce the exploration time 47\% in average. We believe these time savings will be even more pronounced for more complex exploration tasks (e.g., in astronomical or medical domains) where examining the relevance of an object requires significant time.
Our user study revealed that five out of the seven users used only two attributes to characterize their interests \cut{while the rest needed three, four and five attributes}. Similarly to our SDSS workload, the most common type of query was conjunctive queries that selected a single relevant area. {Our exploration domain was highly skewed and all our relevant areas were on dense regions.} These characteristics indicate that our micro-benchmark on the SDSS dataset was representative of common exploration tasks while it also covered highly more complex cases, i.e., small relevant areas and disjunctive queries selecting multiple areas. %More details on our user study can be found in~\cite{kiki_report13_ss}.
%Our experiment also revealed AIDE and AIDE-Sample have similar performance regardless of the underlying data distribution. This is because our sampled dataset preserves the distribution of the exploration domains. This result is consistent for queries of different complexity. We conclude that {sampled datasets do not affect our accuracy even in the presence of skewed data spaces}. % (i.e., with more relevant areas and different size of relevant areas).
\begin{table}[t]
\centering
\scriptsize
%\begin{tabular}{ |c|c|p{1.2cm}|p{2.05cm}|p{1cm}| p{0.9cm}| p{1.5cm}| p{2cm}| p{1.5cm}| }
\begin{tabular}{ |c|r|r|r|r|r|r|}
\hline
%\multicolumn{9}{ |c| }{User Study} \\
%\hline
%User & \# attributes & \# tuples (manual) & \# total tuples returned (manual) & \# tuples (AIDE) & S(\%) & time (min) (manual) & reviewing time per tuple (min) & time (min) (AIDE)\\ \hline
\textbf{User} & \textbf{Manual:} & \textbf{Manual:} & \textbf{AIDE:} & \textbf{Reviewing} & \textbf{Manual:} &\textbf{AIDE:} \\
& \textbf{returned} & \textbf{reviewed} & \textbf{reviewed} & \textbf{savings} & \textbf{time} & \textbf{time} \\
& \textbf{objects} & \textbf{objects} & \textbf{objects} & (\%) & (min) & (min) \\
\hline
1 & 253,461 &312 & 204.9 & \textbf{34.3\%} & 60 & 39.7\\
2 & 656,880 & 160 & 82.4 & \textbf{48.5\%} & 70 & 36.3\\
3 & 933,500 & 1240 & 157 & \textbf{87.3\%} & 60 & 7.9\\
4 & 180,907 & 600 & 319 & \textbf{46.8\%} & 50 & 28.2\\ \hline
5 & 2,446,180 & 650 & 288.5 & \textbf{55.6\%} & 60 & 27.5\\ \hline
6 & 1,467,708 & 750 & 334.5 & \textbf{55.3\%} & 75 & 33.8\\ \hline
7 & 567,894 & 1064 & 288.4 & \textbf{72.8\%} & 90 & 24.8\\
\hline
\end{tabular}
\vspace {-3mm}
\caption{User study results.}
\vspace {-4mm}
\label{tab:userstudytable}
\end{table}
\section{Related Work}\label{s:related}
{\bf Query by Example} Related work on ``Query-By-Example'' (QBE) we originally proposed in~\cite{ zloof75}. Most recent work includes querying knowledge graphs by example tuples~\cite{mottin14}, formulating join queries based on example output tuples~\cite{yanyan14} and inferring user queries by asking for feedback on database tuples~\cite{haoli15, bonifati_edbt14}. Finally, in~\cite{abouzied13} they learn user queries based on given value assignments used in the intended query.
These systems provide alternative front-end query interfaces that assist the user formulate her query and do not attempt to understand user interests nor retrieve ``similar'' data objects which is AIDE's focus.
%
%In ~\cite{abouzied13} they learn user queries based on given value assignments used in the intended query, which are assumptions that we do not make. In~\cite{yanyan14} they obtain the minimal project join query from example tuples that belong in the query's output. Unlike this work, we focus on predicting conjuctive or disjunctive range queries. In~\cite{mottin14} they propose a system for querying knowledge graphs by example tuples. Our focus is on relational databases instead of graphs. %{It then learns conjunctions of quantified Horn expressions (with if-then semantics) over nested relations, which are different from the SQL queries that we want to learn.}%. and thus do not focus on the optimization techniques for sample acquisition we propose.
%
%
%
%
%Related work on ``Query-By-Example'' (QBE) (e.g.,~\cite{ zloof75}) focused on minimizing the burden to remember the finer details of SQL by translating user actions, such as assigning a value to an attribute, to query statements. In ~\cite{abouzied13} they learn user queries based on given value assignments used in the intended query, which are assumptions that we do not make. In~\cite{yanyan14} they obtain the minimal project join query from example tuples that belong in the query's output. Unlike this work, we focus on predicting conjuctive or disjunctive range queries. In~\cite{mottin14} they propose a system for querying knowledge graphs by example tuples. Our focus is on relational databases instead of graphs. %{It then learns conjunctions of quantified Horn expressions (with if-then semantics) over nested relations, which are different from the SQL queries that we want to learn.}%. and thus do not focus on the optimization techniques for sample acquisition we propose.
{\bf Data Exploration} Numerous recent research efforts focus on data exploration. The vision for automatic, interactive navigation in databases was first discussed in~\cite{ugur13} and later on in~\cite{wasay15}. \cut{AstroShelf~\cite{astroshelf2012_ss} allows users to collaboratively annotate and explore sky objects while }YMALDB~\cite{ymaldb2013} supports data exploration by recommending to the user data similar to her query results. DICE~\cite{dice2014_ss} supports exploration of data cubes using faceted search and in~\cite{manas15} they propose a new ``drill-down'' operator for exploring and summarizing groups of tuples. SciBORQ~\cite{sciborq11} relies on hierarchical database samples to support scientific exploration queries within strict query execution times. \cut{Blink~\cite{sameer12} relies on run-time sample selection to provide real-time answers with statistical error guarantees.} Idreos et al.~\cite{kersten11} envision a system for interactive data processing tasks aiming to reduce the time spent on data analysis. In~\cite{sellam13} interactively explores the space based on statistical properties of the data and provides query suggestions for further exploration while in~\cite{lilong15} they propose a technique for providing feedback during the query specification and eventually guiding the user towards her intended query. In~\cite{alex14} users rely on prefetching and incremental online processing to offer interactive exploration times for window-based queries. %Searchlight~\cite{alex15_ss} uses constraint programming techniques to support data exploration. Also, adaptive indexing has been studied for exploring time-series~\cite{zoumpatianos15}. In~\cite{haoli15} they assist users unfamiliar with SQL to formulate queries by iteratively asking for feedback on pairs of sample database and query result tables. In~\cite{} they assist users in exploring the data by snapping to the user's likely intended query. Finally,~\cite{sellam13} interactively explores the space based on statistical properties of the data and provides suggestions for further exploration.
SearchLight~\cite{alex15} offers fast searching, mining and exploration of multidimensional data based on constraint programming. All the above systems are different than AIDE: we rely on the user's feedback on data samples to predict the user's data interests and we focus on identifying strategic sampling areas that allow for accurate predictions.
%Moreover, systems allow the users to interactively explore the data by performing structured search via shape and content constraints over a multi-dimensional data space~\cite{}, based on statistical properties of the data~\cite{sellam13}
%Moreover, various approaches for assisting the user to interactively explore the data space have been recently introduced. These approaches include constraint programming techniques~\cite{alex15_ss}, adaptively indexing the data space~\cite{zoumpatianos15}, asking for feedback on pairs of sample database and query result tables~\cite{haoli15}, using statistical properties of the data~\cite{sellam13} and snapping to the user's likely intended query~\cite{}.
{\bf Query Relaxation} Query relaxation techniques have also been proposed for supporting exploration in databases~\cite{chaudhuri90}. In ~\cite{mishra09, koudas06_ss} they refine SQL queries to satisfy cardinality constraints on the query result. In~\cite{kadlag04} they rely on multi-dimensional histograms and distance metrics for range queries for accurate query size estimation. These solutions are orthogonal to our problem; they focus on adjusting the query parameters to reach a cardinality goal and therefore cannot characterize user interests.
%~\cite{Olken94randomsampling} proposed random sampling for database systems to support approximate queries and query optimization.
{\bf Active Learning} The active learning community has proposed solutions that maximize the learning outcome while minimizing the number of samples labeled by the user~\cite{roy2001, sarawagi02}. However, these techniques assume either small datasets or negligible sample extraction costs which is not a valid assumption when datasets span 100s of GBs and interactive performance is expected. Relevance feedback have been studied for image retrieval~\cite{Panda:2006:ALV}, document ranking~\cite{ruthven03}, information extraction and segmentation~\cite{settles2008} and word disambiguation~\cite{zhu2007}. All these solutions are designed for specific data types (images or text) and do not optimize for efficient {sample acquisition} and {data space exploration}.
{\bf Collaborative and Interactive Systems}
In~\cite{nodira} a collaborative system is proposed to facilitate formulation of SQL queries based on past queries and in~\cite{gloria09} they use collaborative filtering to provide query recommendations. However, both these systems do not predict ``similar'' data object. In~\cite{sadikov10} they cluster related queries as a means of understanding the intents of a given user query. The focus is on web searches and not structured databases.
% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{LSST}
\BIBentryALTinterwordspacing
``{Large Synoptic Survey Telescope}.'' [Online]. Available:
\url{{http://http:/www.lsst.org/}}
\BIBentrySTDinterwordspacing
\bibitem{SloanSky}
\BIBentryALTinterwordspacing
``{Sloan Digital Sky Survey}.'' [Online]. Available:
\url{{http://www.sdss.org/}}
\BIBentrySTDinterwordspacing
\bibitem{breinman84}
L.~Breiman, J.~H. Friedman, R.~A. Olshen, and C.~J. Stone,
\emph{{Classification and Regression Trees}}.\hskip 1em plus 0.5em minus
0.4em\relax Chapman and Hall/CRC, 1984.
\bibitem{zhou03}
X.~S. Zhou and T.~Huang, ``{Relevance Feedback in Image retrieval: A
comprehensive review},'' \emph{Multimedia System}, vol.~8, no.~2, pp.
95--145, 2003.
\bibitem{Settles10activelearning}
B.~Settles, ``Active learning literature survey,'' Tech. Rep., 2010.
\bibitem{roy2001}
N.~Roy and A.~McCallum, ``Toward optimal active learning through sampling
estimation of error reduction,'' in \emph{ICML}, 2001.
\bibitem{aide_sigmod14}
K.~Dimitriadou, O.~Papaemmanoui, and Y.~Diao, ``{Explore-by-Example: An
Automatic Query Steering Framework for Interactive Data Exploration},'' in
\emph{{SIGMOD}}, 2014.
\bibitem{aide_vldb15}
{Diao et al}, ``{AIDE: An Automatic User Navigation Service for Interactive
Data Exploration (Demo)},'' in \emph{{VLDB}}, 2015.
\bibitem{deselaers_08}
T.~Deselaers, R.~Paredes, E.~Vidal, and H.~Ney, ``{Learning weighted distances
for relevance feedback in image retrieval},'' in \emph{{ICPR}}, 2008.
\bibitem{combineclassifier_98}
J.~Kittler, M.~Hatef, R.~P.~W. Duin, and J.~Matas, ``On combining
classifiers,'' \emph{IEEE Trans. Pattern Anal. Mach. Intell.}, vol.~20,
no.~3, pp. 226--239, 1998.
\bibitem{Olken94randomsampling}
F.~Olken and D.~Rotem, ``Random sampling from databases - a survey,''
\emph{Statistics and Computing}, vol.~5, no.~1, pp. 25--42, 1994.
\bibitem{Weka_url}
``Weka: Data mining software in java, http://www.cs.waikato.ac.nz/ml/weka/.''
\bibitem{SloanSky-Queries}
\BIBentryALTinterwordspacing
``{SDSS Sample Queries}.'' [Online]. Available: \url{{,
http://cas.sdss.org/dr4/en/help/docs/realquery.asp}}
\BIBentrySTDinterwordspacing
\bibitem{AuctionMark}
\BIBentryALTinterwordspacing
``{AuctionMark Benchmark}.'' [Online]. Available:
\url{{http://hstore.cs.brown.edu/projects/auctionmark/}}
\BIBentrySTDinterwordspacing
\bibitem{zloof75}
M.~M. Zloof, ``{Query-by-example: the invocation and definition of tables and
forms},'' in \emph{VDLB}, 1975.
\bibitem{mottin14}
D.~Mottin, M.~Lissandrini, Y.~Velegrakis, and T.~Palpanas, ``Exemplar queries:
Give me an example of what you need,'' \emph{{VLDB 2014}}.
\bibitem{yanyan14}
Y.~Shen, K.~Chakrabarti, S.~Chaudhuri, B.~Ding, and L.~Novik, ``Discovering
queries based on example tuples,'' in \emph{{SIGMOD}}, 2014.
\bibitem{haoli15}
H.~Li, C.~Chan, and D.~Maier, ``{Query From Examples: An Iterative, Data-Driven
Approach to Query Construction},'' \emph{{VLDB 2015}}.
\bibitem{bonifati_edbt14}
A.~Bonifati, R.~Ciucanu, and S.~Staworko, ``{Interactive inference of join
queries},'' in \emph{{EDBT}}, 2014.
\bibitem{abouzied13}
{Abouzied et al}, ``{Learning and verifying quantified boolean queries by
example},'' in \emph{PODS}, 2013.
\bibitem{ugur13}
{U. \c{C}etintemel et all}, ``Query steering for interactive data
exploration,'' in \emph{CIDR}, 2013.
\bibitem{wasay15}
A.~Wasay, M.~Athanassoulis, and S.~Idreos, ``{Queriosity: Automated Data
Exploration},'' in \emph{IEEE International Congress on Big Data}, 2015.
\bibitem{ymaldb2013}
M.~Drosou and E.~Pitoura, ``{YMALDB: exploring relational databases via
result-driven recommendations},'' \emph{VLDB Journal}, vol.~22, pp. 849--874,
2013.
\bibitem{dice2014_ss}
{Kamat et al}, ``{Distributed and Interactive Cube Exploration},'' in
\emph{{ICDE}}, 2014.
\bibitem{manas15}
M.~Joglekar, H.~Garcia{-}Molina, and A.~G. Parameswaran, ``Smart drill-down:
{A} new data exploration operator,'' \emph{{VLDB 2015}}.
\bibitem{sciborq11}
L.~Sidirourgos, M.~Kersten, and P.~Boncz, ``{SciBORQ: Scientific data
management with Bounds On Runtime and Quality},'' in \emph{CIDR}, 2011.
\bibitem{kersten11}
{Kersten et al}, ``{The Researcher's Guide to the Data Deluge: Querying a
Scientific Database in Just a Few Seconds},'' \emph{{VLDB 2011}}.
\bibitem{sellam13}
{Sellam et al}, ``{Meet Charles, big data query advisor},'' in \emph{CIDR},
2013.
\bibitem{lilong15}
L.~Jiang and A.~Nandi, ``{SnapToQuery: Providing Interactive Feedback During
Exploratory Query Specification},'' \emph{{VLDB 2015}}.
\bibitem{alex14}
{Kalinin et al}, ``Interactive data exploration using semantic windows,'' in
\emph{{SIGMOD}}, 2014.
\bibitem{alex15}
A.~Kalinin, U.~{\c{C}}etintemel, and S.~B. Zdonik, ``Searchlight: Enabling
integrated search and exploration over large multidimensional data,''
\emph{{VLDB 2015}}.
\bibitem{chaudhuri90}
S.~Chaudhuri, ``{Generalization and a framework for query modification},'' in
\emph{ICDE}, 1990.
\bibitem{mishra09}
C.~Mishra and N.~Koudas, ``{Interactive query refinement},'' in \emph{EDBT},
2009.
\bibitem{koudas06_ss}
{Koudas et al}, ``{Relaxing join and selection queries},'' \emph{{VLDB 2006}}.
\bibitem{kadlag04}
A.~Kadlag, A.~V. Wanjari, J.~Freire, and J.~R. Haritsa, ``{Supporting
Exploratory Queries in Databases},'' in \emph{DASFAA}, 2004.
\bibitem{sarawagi02}
S.~Sarawagi and A.~Bhamidipaty, ``{Interactive Deduplication Using Active
Learning},'' in \emph{KDD}, 2002.
\bibitem{Panda:2006:ALV}
N.~Panda, K.-S. Goh, and E.~Y. Chang, ``Active learning in very large
databases,'' \emph{Multimedia Tools Appl.}, vol.~31, no.~3, pp. 249--267,
2006.
\bibitem{ruthven03}
{Ruthven et al}, ``{A survey on the use of relevance feedback for information
access systems},'' \emph{The Knowledge Engineering Review}, vol.~18, no.~2,
pp. 95--145, 2003.
\bibitem{settles2008}
B.~Settles and M.~Craven, ``An analysis of active learning strategies for
sequence labeling tasks,'' in \emph{{EMNLP}}, 2008.
\bibitem{zhu2007}
J.~Zhu, ``{Active Learning for Word Sense Disambiguation with Methods for
Addressing the Class Imbalance Problem},'' in \emph{ACL}, 2007.
\bibitem{nodira}
{Khoussainova et al}, ``{A Case for A Collaborative Query Management System},''
in \emph{CIDR}, 2009.
\bibitem{gloria09}
G.~Chatzopoulou, M.~Eirinaki, and N.~Polyzotis, ``{Query Recommendations for
Interactive Database Exploration},'' in \emph{SSDBM}, 2009.
\bibitem{sadikov10}
E.~Sadikov, J.~Madhavan, L.~Wang, and A.~Halevy, ``{Clustering query
refinements by user intent},'' in \emph{WWW}, 2010.
\end{thebibliography}