On Text-based Personality Computing: Challenges and Future Directions
Extracted source text · arXiv 2212.06711
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% additional packages
\usepackage{authblk}
% Remove the "review" option to generate the final version.
\usepackage[]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{On Text-based Personality Computing: Challenges and Future Directions}
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
% \author{Qixiang Fang, Anastasia Giachanou, Ayoub Bagheri, Laura Boeschoten, \\ {\bf Erik-Jan van Kesteren}, {\bf Mahdi Shafiee Kamalabad}, {\bf Daniel L. Oberski} \\
% Utrecht University\\Utrecht, the Netherlands\\
% \texttt{\{q.fang, a.giachanou, a.bagheri, l.boeschoten\}@uu.nl}\\
% \texttt{\{e.vankesteren1, m.shafieekamalabad, d.l.oberski\}@uu.nl}}
\author[1]{Qixiang Fang}
\author[1]{Anastasia Giachanou}
\author[1]{Ayoub Bagheri}
\author[1]{Laura Boeschoten}
\author[1]{\\Erik-Jan van Kesteren}
\author[1]{Mahdi Shafiee Kamalabad}
\author[1, 2]{Daniel L. Oberski}
\affil[1]{Utrecht University, Utrecht, the Netherlands}
\affil[2]{University Medical Center Utrecht, Utrecht, the Netherlands}
\affil[ ]{\texttt{\{q.fang, a.giachanou, a.bagheri, l.boeschoten\}@uu.nl}}
\affil[ ]{\texttt{\{e.vankesteren1, m.shafieekamalabad, d.l.oberski\}@uu.nl}}
\begin{document}
\maketitle
\begin{abstract}
Text-based personality computing (TPC) has gained many research interests in NLP. In this paper, we describe 15 challenges that we consider deserving the attention of the NLP research community. These challenges are organized by the following topics: personality taxonomies, measurement quality, datasets, performance evaluation, modelling choices, as well as ethics and fairness. When addressing each challenge, not only do we combine perspectives from both NLP and social sciences, but also offer concrete suggestions. We hope to inspire more valid and reliable TPC research.
%60 TPC papers
%Many of these issues have not been discussed by TPC researchers before.
\end{abstract}
\input{writing/1Introduction}
% \input{writing/2Literature}
\input{writing/3Background}
\input{writing/4Taxonomies}
\input{writing/5Measurement}
\input{writing/6Datasets}
\input{writing/7Evaluation}
\input{writing/8Models}
\input{writing/9Theory}
\input{writing/10Ethics}
\input{writing/11Conclusion}
% Entries for the entire Anthology, followed by custom entries
\bibliography{bibliography}
\bibliographystyle{acl_natbib}
\appendix
\begin{table*}[htb]
\centering
\small
\begin{tabular}{llllll}
\hline
\textbf{Source} & \textbf{Name} & \textbf{Size} & \textbf{Taxonomy} & \textbf{Text Type} & \textbf{Language}\\
\hline
\citet{barriere_wassa_2022} & WASSA 2022 & 2655 & Big-5 & News and essays & en\\
\citet{bassignana2020personal} & Personal-ITY & 1048 & MBTI & YouTube comments & it\\
\citet{biel2012youtube} & Youtube & 442 & Big-5 & YouTube vlogs & en\\ & Personality & & & transcripts\\
\citet{gjurkovic_pandora_2021} & PANDORA$^*$ & 10288 & Big-5 \& & Reddit posts & en, es, fr, it, de, pt\\ & & & MBTI& & nl, eo, sv, pl\\
\citet{gjurkovic_reddit_2018} & Reddit9k$^*$ & 9111 & MBTI & Reddit posts & en \\
\citet{jansen_introducing_2020} & MULAI$^*$ & 26 & Big-5 & Conversation & en\\ & & & &transcripts\\
\citet{koutsombogera_modeling_2018} & MULTISIMO$^*$ & 49 & Big-5 & Conversation & en\\ & & & &transcripts\\
\citet{kaggleData} & Kaggle & 8600 & MBTI & Forum posts & en\\
\citet{pennebaker1999linguistic} & Essays & 2479 & Big-5 & Essays & en\\
\citet{plank-hovy-2015-personality} & Twitter & 1500 & MBTI & Twitter posts & en\\
\citet{ponce2016chalearn} & ChaLearn V1 & 10000 & Big-5 & YouTube vlogs & en\\ & & & &transcripts\\
\citet{ramos_building_2018} & b5$^*$ & 1082 & Big-5 & Facebook status, & pt\\ & & & & referring expressions \\ & & & & and scene descriptions\\
\citet{rangel2015overview} & PAN 2015$^*$ & 300 & Big-5 & Twitter posts & en, es, it, nl\\
\citet{tolins_multimodal_2016} & Personality & 6 & Big-5 & Conversation & en\\ & Dyads$^*$ & & & transcriptions\\
\citet{tolins_verbal_2016} & Storytron & 44 & Big-5 & Conversation & en\\ & Retellings$^*$ & & & transcriptions\\
\citet{verhoeven_clips_2014} & CSI & 697 & Big-5 \& & Essays and reviews & nl\\ & & & MBTI \\
\citet{verhoeven_twisty_2016} & TWISTY & 18168 & MBTI & Twitter posts & de, it, nl, fr, pt, es\\
\citet{wen_automatically_2021} & PELD & 711 & Big-5 & TV show dialogs & en\\
\hline
\end{tabular}
\caption{\label{table:datasets}
Overview of Shareable Datasets. *Data available upon simple request (e.g., via a form on the dataset hosting website).
Abbreviations of languages are based on ISO 639-1 Code.
}
\end{table*}
\section{Literature Search and Overview}
\label{sec:appendixA}
To search for relevant TPC studies in the ACL Anthology, we use the following search terms: “personality”, “personality prediction”, “personality computing” and “personality recognition”. The search date is August 29, 2022. We also sort the results by “relevance” and “year of publication”, separately, as these two options return different search results. For all the returned papers, we check their titles, abstracts and (when necessary) main texts, and retain those that fall under one of the following categories:
\paragraph{Dataset Type-A}
Papers focusing on introducing new datasets for TPC. Some also benchmark TPC models against new datasets. There are 7: \citet{luyckx_personae_2008, verhoeven_clips_2014, plank-hovy-2015-personality, verhoeven_twisty_2016, bassignana_matching_2020, gjurkovic_pandora_2021, barriere_wassa_2022}.
\paragraph{Dataset Type-B}
Papers that introduce datasets for non-TPC goals (e.g., multimodal PC, natural language generation, modelling laughter in conversations) but happen to include personality and text data that can be used for TPC. There are 5: \citet{tolins_verbal_2016, tolins_multimodal_2016, koutsombogera_modeling_2018, ramos_building_2018, jansen_introducing_2020}.
\paragraph{TPC as End}
Empirical papers where TPC is the main goal. There are 31: \citet{mairesse_automatic_2006, resnik_using_2013, sinha_mining_2015, fung_zara_2016, kamijo-etal-2016-personality, levitan_identifying_2016, liu_recurrent_2016, das_developing_2017, liu_language-independent_2017, siddique_zara_2017, gjurkovic_reddit_2018, kampman_investigating_2018, tighe_modeling_2018, vu_lexical-semantic_2018, zamani_predicting_2018, pizzolli_personality_2019, yamada_incorporating_2019, cornelisse_inferring_2020, iwai_development_2020, lynn_hierarchical_2020, suman_multi-modal_2020, culnan_me_2021, hull_personality_2021, kishima_construction_2021, v_ganesan_empirical_2021, yang_learning_2021, yang_psycholinguistic_2021, ghosh_team_2022, kerz_pushing_2022, kreuter_items_2022, li_prompt-based_2022}.
\paragraph{TPC as Means}
Empirical papers where TPC is not the main goal but used for further purposes like emotion detection, natural language generation and sociolinguistic analysis. There are 17: \citet{mairesse_personage_2007, mairesse_trainable_2008, makatchev_perception_2011, gill_perceptions_2012, roshchina_evaluating_2012, preotiuc-pietro_role_2015, yang_using_2015, guntuku_current_2018, lan_definite_2018, al_khatib_exploiting_2020, iwai_acquiring_2020, he_personality_2021, hosseinia_usefulness_2021, uban_understanding_2021, vettigli_empna_2021, wen_automatically_2021, saha_stylistic_2022}.
\vspace{2mm}
Note that we exclude papers that mean PC as the computing of personality disorders or personal profiles. In total, we find and review 60 TPC papers. Also, see Appendix~\ref{sec:appendixB} for 18 shareable TPC datasets that we find via these 60 papers.
% \onecolumn
\section{Overview of Shareable TPC Datasets}
\label{sec:appendixB}
See Table \ref{table:datasets}.
\end{document}
\section{Introduction}
According to the APA Dictionary of Psychology~\citep{apaDictionary23}, personality refers to personality traits, which are “relatively stable, consistent, and enduring internal characteristics inferred from a pattern of behaviours, attitudes, feelings and habits in individuals”. Knowledge about personality can be useful in many societal and scientific applications. For instance, it can help individuals choose learning styles~\citep{komarraju2011big} and occupations~\citep{kern2019social} suited for their personality; it can help clinical psychologists to better understand psychological disorders~\citep{khan2005personality} and to deliver personalised treatment plans for mental health patients~\citep{bagby2016role}; changes in personality can even help with early diagnosis of Alzheimer’s~\citep{robins2011personality} and Parkinson’s disease~\citep{santangelo2017personality}.
Traditionally, personality assessment is based on self- and other-report questionnaires, which is labour- and time-intensive. Recently, however, automatic personality assessment based on user-generated data (e.g., texts, images, videos) and machine learning algorithms has become a popular alternative. This is known as \textbf{personality computing (PC)}~\citep{phan2021personality}, among many other names\footnote{Personality recognition~\citep{liu_language-independent_2017}, personality identification~\citep{das_developing_2017}, personality detection~\citep{yang_learning_2021} and personality prediction~\citep{yamada_incorporating_2019}}.
% which may reduce and even eliminate questionnaire response burden for humans.
In this paper, we focus on evaluating PC research in NLP, where personality is primarily inferred from text, such as tweets and Reddit posts~\citep{hosseinia_usefulness_2021}, conversations~\citep{mairesse_automatic_2006} and speech transcriptions~\citep{das_developing_2017}. We refer to such research as \textbf{text-based personality computing (TPC)}.
% Some examples are: personality prediction based on tweets and Reddit posts~\citep{hosseinia_usefulness_2021}; the study of lexical signals indicative of personality~\citep{he_personality_2021}; generation of personality-dependent natural language~\citep{saha_stylistic_2022},
In TPC, on the one hand, we see an increasing number of datasets curated, complex deep-learning algorithms adopted, and (sometimes) high prediction scores achieved. On the other hand, we see relatively little discussion about open challenges and future research directions.
For example, the presence of measurement error in questionnaire-based personality scores remains an un(der)addressed issue. Given that such scores are often used as the gold standard for training and validating TPC algorithms, we find it important to discuss related implications and remedies. Another relevant issue concerns how to reduce the risks of TPC research.
% Many other relevant issues remain un(der)addressed. An example is the presence of measurement error in questionnaire-based personality scores. Given that such scores are often used as the gold standard for training and validating TPC algorithms, we find it important to discuss: \emph{Would measurement error affect the quality of personality predictions? If yes, how can we quantify and deal with measurement error? Also, how can we collect higher-quality personality data?}
Therefore, in this paper, we reflect on current TPC research practices, identify open challenges, and suggest better ways forward. To spot such challenges, we conduct a literature search in the scope of the ACL Anthology\footnote{https://aclanthology.org/}. While there are TPC papers published in other venues, we consider our selection from ACL Anthology a good representation of TPC research in NLP. Appendix~\ref{sec:appendixA} describes our search strategy and results in detail. In total, we find and review 60 empirical TPC papers, based on which we identify 15 challenges. They are organized by the following topics: personality taxonomies (\S\ref{sec_taxonomies}), measurement quality (\S\ref{sec_measurements}), datasets (\S\ref{sec_datasets}), performance evaluation (\S\ref{sec_evaluation}), modelling choices (\S\ref{sec_modelling}), as well as ethics and fairness (\S\ref{sec_ethics}). We discuss each challenge and give concrete suggestions where we also draw on broader NLP and social science literature.
Furthermore, during our literature review, we identify 18 TPC datasets that can be (re)used by other researchers. We summarize them in Appendix~\ref{sec:appendixB}.
Note that our paper focuses on identifying and discussing TPC-related research challenges instead of providing a comprehensive overview of past TPC studies. For the latter, we refer you to the survey papers by \citet{stajner_survey_2020, FeiziDerakhshi2021TextbasedAP, mushtaq2022text}.
\section{Background}
\subsection{Current TPC in a Nutshell}
TPC concerns computing personality information from texts. This can be either a regression or a classification task, depending on whether the personality measurements are continuous or discrete. Supervised learning has been the predominant approach, relying on text datasets labelled with personality traits via either self-report or crowdsourced annotations. Among many others, log-linear models~\citep{Volkova2015InferringLU}, random forests~\citep{levitan_identifying_2016}, GloVe embeddings with Gaussian processes~\citep{Arnoux201725TT}, recurrent neural networks~\citep{liu_language-independent_2017}, convolutional neural networks~\citep{Majumder2017DeepLD}, support vector machines~\citep{lan_definite_2018}, ridge regression~\citep{he_personality_2021}, graphical networks~\citep{yang_psycholinguistic_2021} and transformers~\citep{kreuter_items_2022} have been used. Another popular, psycho-linguistically motivated approach are dictionaries/lexicons~\citep[e.g.,][]{oberlander2006whose, sinha_mining_2015, das_developing_2017}. They are typically lists of curated terms that have pre-assigned weights associated with different personality traits. This allows researchers to compute personality scores from texts by simply matching a dictionary to the texts and aggregating the weights of matched words and phrases.
\subsection{Related Work}
\label{sec:relatedWork}
We find three relevant TPC survey papers. The first one, \citet{stajner_survey_2020}, not only summarizes previous research in TPC but also discusses three interesting issues: the difference between MBTI and the Big-5 (two most popular personality taxonomies, see \S\ref{sec_taxonomies}), the difficulty in predicting MBTI from Twitter data, and ethical concerns about TPC. We find all three discussions necessary and helpful. Our paper not only provides our own (differing) perspectives on these issues, but also raises and discusses many others. The other two survey papers, \citet{mushtaq2022text} and \citet{FeiziDerakhshi2021TextbasedAP}, after reviewing prior TPC research, very briefly suggest some open challenges in TPC research. While the open challenges mentioned in these two papers (e.g., better data quality, data sharing, and ethics) partially overlap with our list of challenges, we adopt a more evidence-based approach to listing challenges, engage in much more thorough discussion, and offer concrete solutions. Furthermore, our list of challenges goes beyond those in these two papers (e.g., measurement error reduction, performance expectation, joint personality modelling).
% \citet{stajner_why_2021}, is an empirical study about the lack of linguistic cues in Twitter data for the prediction of MBTI. This paper provides the basis for some of our discussion in \S\ref{sec_taxonomies} (C1) and \S\ref{sec_evaluation} (C11).
We also find three additional papers~\citep{bleidorn2019using, stachl2020personality, phan2021personality} concerning issues in general PC research. Our paper, in contrast, focuses on challenges specific to TPC research.
Lastly, TPC is closely related to other fields, such as automatic emotion recognition~\citep{barriere_wassa_2022}, opinion mining~\citep{hosseinia_usefulness_2021} and mental health prediction~\citep{guntuku_current_2018}.
These fields are all concerned with the computation of social science constructs (see \S\ref{sec_measurements}).
\section{Personality Taxonomies}
\label{sec_taxonomies}
A TPC research project typically starts by choosing a personality taxonomy, which is a descriptive framework for personality traits~\citep{john1999big}. Among the 60 papers we review, we find two prominent taxonomies: the Myers-Briggs Type Indicator (MBTI; 14 papers) and the Big-5 (45 papers).\footnote{Also, Enneagram and HEXACO each appeared once.} Fifty of these papers adopt \emph{either} the MBTI \emph{or} the Big-5 but not both. This invites the first challenge:
\begin{quote}
\label{C1}
\textbf{Challenge 1 (C1): MBTI vs. Big-5}
\end{quote}
The MBTI originated from the theoretical work of~\citet{jung1971psychological} and was further developed by~\citet{briggs1995gifts}. It proposes four personality dimensions that characterize people’s differences in perception and judgement processes: Extraversion/Introversion (E/I), Sensing/iNtuition (S/N), Thinking/Feeling (T/F), and Judgement/Perception (J/P). Individuals are classified into one of the two possible categories across each dimension (e.g., INFP and ESTJ).
% To date, MBTI is likely the most used personality taxonomy in non-research settings (e.g., job interviews).
In contrast, the Big-5 was developed based on the lexical hypothesis: \emph{all important personality traits must have been encoded in natural language and therefore, analysis of personality-related terms should reveal the true personality taxonomy}~\citep{goldberg1990alternative}. Independent research groups~\citep{cattell1946description,goldberg1982ace,costa1992professional,tupes1992recurrent} investigated this hypothesis. They identified numerous English terms that might describe inter-individual differences (e.g., warm, curious), asked participants to rate how well these terms describe them on numerical scales, and factor-analyzed the responses, which revealed five consistent dimensions of personality: Openness (O), Conscientiousness (C), Extraversion (E), Agreeableness (A), and Neuroticism (N).\footnote{The Big-5 is also called OCEAN.} Furthermore, each dimension includes six finer-grained sub-traits called \textit{facets} (e.g., agreeableness includes trust, straightforwardness, altruism, compliance, modesty and tender-mindedness). Big-5 is the most widely accepted and researched taxonomy of personality traits in psychology (as opposed to the popularity of MBTI in non-academic settings like job interviews)~\citep{phan2021personality}.
We recommend the Big-5 over MBTI for the following reasons:
First, the Big-5 is a more realistic and accurate personality taxonomy. It scores individuals along a continuous spectrum, which describes inter-individual differences more accurately and preserves more information (as opposed to MBTI’s dichotomous approach). Also, the Big-5 includes facets, which allows for finer-grained analysis of personality. Facets are also more predictive of life outcomes (compared to dimensions; \citealt{mershon1988number, paunonen2001big}). A potential new TPC research direction can be to predict facets in addition to dimensions.
Second, Big-5 has a much stronger empirical basis than MBTI. Namely, it is grounded in large-scale quantitative analysis of natural language and survey data. Also, Big-5 questionnaires have undergone much more extensive development and validation processes than MBTI's. Consequently, many validated Big-5 questionnaires exist, which vary in length (15-240 items), the inclusion of facet measures, as well as the target populations (e.g., nationalities, professions, languages, and age groups).\footnote{E.g., the NEO Personality Inventory~\citep{mccrae1989neo}, the Revised NEO-PI and the NEO Five Factor Inventory~\citep{costa1992professional}, the Big-5 Inventory (BFI)~\citep{benet1998cinco}, BFI-2~\citep{soto2017next}, the Short 15-item Big-5 Inventory~\citep{lang2011short}.} This enables researchers to choose a questionnaire most appropriate given a population of interest, available resources (e.g., can you afford a longer questionnaire?) and research interests (e.g., are you interested in facets?). In comparison, MBTI is purely theory-driven, lacks empirical support and has officially only four questionnaires that have not been thoroughly tested~\citep{pittenger1993measuring, nowack1996myers,grant2013goodbye}.\footnote{Form M with 93 items, Form M self-scorable with 93 items, Form Q with 93 items and Step III Form with 222 items. See https://www.myersbriggs.org/.} Therefore, compared to MBTI, Big-5 is a much more credible and flexible choice for research purposes.
% Third, unlike MBTI, numerous empirical studies have investigated the relationships between Big-5 and other social science constructs (e.g. emotions, styles, mental diseases) that can be relevant to TPC research. See \S\ref{sec_measurements} (C6) and \S\ref{sec_modelling} (C16, C17) for how knowledge about such relationships can be helpful.
Third, the Big-5 is rooted in natural language (i.e., the lexical hypothesis), suggesting that Big-5-related cues may be more present than do MBTI-related cues in text data. This conjecture is supported by \citet{stajner_why_2021}, who find either insufficient or mixed signals for MBTI dimensions in tweets and short essays.
% Thus, Big-5 is a better choice when it comes to text data.
Nevertheless, despite the many advantages of Big-5 over MBTI, we acknowledge that studying MBTI can still be useful given its popularity in non-academic settings~\citep{lloyd2012myers}. Furthermore, it is also important to mention that Big-5 is not without criticisms and other personality taxonomies may be preferred, which invites the next challenge.
% When using MBTI for TPC research, it is perhaps wise to keep expectations low about prediction performance, as text data may offer few cues about MBTI. Other types of data (e.g., behaviour, image, audio) may be more useful for predicting MBTI.
\begin{quote}
\label{C2}
\textbf{C2: Beyond the Big-5}
\end{quote}
While earlier lexical studies of the English language revealed five core dimensions of personality, more recent analysis of both English and non-English languages (e.g., Italian, Dutch, German, Korean), based on larger sets of adjectives, has suggested the existence of a sixth core dimension: Honesty/Humility, giving rise to a new “Big-6” taxonomy (a.k.a. HEXACO)~\citep{ashton2007empirical}. Therefore, we encourage TPC researchers to explore alternatives to the Big-5, in timely accordance with developments in personality psychology. For a comprehensive overview of (other) personality taxonomies, see \citet{cervone2022personality}.
\section{Measurement Quality}
\label{sec_measurements}
% Variables like age and length can be measured directly and hence, relatively accurately and reliably. They are referred to as \emph{observed} variables in the measurement literature~\citep{oberski2016mixture}. In contrast,
Personality traits are latent, theoretical variables (a.k.a. social science constructs), which can not be directly or objectively observed. Examples are emotions, prejudice and political orientation.
Thus, personality traits are inherently difficult to measure. We can only approximate the true underlying personality trait scores from often noisy observations collected using personality instruments such as self- and other-report questionnaires. Due to the uncertainty in this approximation process, measurements for personality traits likely contain non-negligible error. This error is called \textbf{measurement error}, defined as the difference between an observed measurement and its true value~\citep{dodge2003oxford}.
The presence of error in personality measurements can have negative consequences for TPC research. For instance, consider the case of having substantial measurement error in questionnaire-based personality scores. When a TPC model treats these measurements as the gold standard for training and validation, the measurement error will likely propagate to the predictions, rendering the model less helpful (or even harmful) especially for diagnostic or clinical purposes. The study by \citet{Akrami2019AutomaticEO} lends support to this hypothesis, where the authors find TPC models to perform better on small datasets with low measurement error than on large datasets with high measurement error. Therefore, it is important that TPC researchers are aware of the presence and influence of measurement error and can deal with it. Unfortunately, none of the 60 TPC studies that we survey touch upon this issue, suggesting that this is an under-explored issue in TPC research. This observation inspires the next four challenges (3-6).
% Also, measurement error can set an upper limit on the model's performance.
% since a model is only as good as its data. Without knowing the extent of measurement error in the personality measurements, researchers may have unrealistic expectations about their model’s performance and, hence, may end up overfitting their model on noisy data.
% choosing high-quality personality instruments (C3), further reducing measurement error by design (C4), assessing the degree of measurement error (C5), and correcting for measurement error (C6).
% Note that in the rest of the paper, by personality instruments, we mean not only questionnaires but also prediction models. This is because personality predictions can be considered alternative personality measurements, especially when they are used for downstream goals like diagnostics. Instead of relying on questionnaire responses, prediction models infer personality traits from other data sources such as text. We also consider dictionaries/lexicons a form of prediction models because the vocabulary weights are often model-based. For the following four challenges, if applicable, we devote the discussion to both questionnaire- and model-based personality instruments.
\begin{quote}
\label{C3}
\textbf{C3: Choose high-quality instruments}
\end{quote}
Collecting high-quality personality measurements begins with using high-quality instruments, be they questionnaires or models.\footnote{We consider prediction models and dictionaries/lexicons as personality instruments, just like questionnaires.} By high quality, we specifically mean high measurement quality. To determine the measurement quality of an instrument, it is important to understand the two components of measurement error: random error and systematic error, and how they relate to two quality criteria: reliability and validity.
\textbf{Random error} refers to random variations in measurements across comparable conditions, due to factors that cannot be controlled~\citep{Trochim2015ResearchMT}. For instance, when someone completes a personality questionnaire twice, the responses may differ between the two attempts because the person misread a question in the second attempt. Random error is always present and unpredictable; it can be reduced but not eliminated. In NLP systems, random error can be due to random data splitting~\citep{gorman2019we}, stochastic algorithms~\citep{zhou2020TheCO}, and certain random processes in data annotations, such as sampling of annotators and random annotation mistakes~\citep{uma2021learning}.
In contrast, \textbf{systematic error} occurs due to factors inherent to an instrument~\citep{Trochim2015ResearchMT}. For instance, a poorly constructed Big-5 questionnaire may contain an item that is used to measure neuroticism while in fact it does not. Consequently, anyone taking this questionnaire will get a biased estimate of their neuroticism. Thus, systematic error is foreseeable and often constant or proportional to the true value. As long as its cause is identified, systematic error can be removed. In NLP systems, systematic error can occur when spurious correlations (or “short-cuts”, instead of causal relationships) are learned~\citep{Wang2022IdentifyingAM}.
\textbf{Reliability} and \textbf{validity} are the two criteria to the measurement quality of an instrument. The former concerns the extent to which an instrument can obtain the same measurement under comparable conditions, while the latter concerns the extent to which an instrument captures what it is supposed to~\citep{Trochim2015ResearchMT}. Random error reduces an instrument's reliability, while systematic error undermines its validity. Therefore, a high-quality instrument is a reliable and valid one. We describe below how we can find out about the validity and reliability of a personality instrument.
For personality questionnaires, especially of Big-5, it is relatively easy to determine their measurement quality because many corresponding validity and reliability studies exist (see \citealt{VANDERLINDEN2010315} for an overview). For model-based personality instruments, however, they rarely undergo comprehensive analysis of measurement quality. Typically, studies report the predictive performance of a personality model on some test data (using metrics like accuracy, recall, precision, F1, mean squared error, correlation). Such performance numbers can offer insight into the model's validity, assuming that the gold-standard personality measurements are low in error. However, the model’s validity in the presence of substantial measurement error in the data, as well as the model's reliability, remains unclear. Therefore, we urge future researchers to also examine and report both the validity and reliability of a model-based personality instrument. In Challenge 5, we discuss how this can be done.
% When previous research on the measurement quality of an instrument is absent, we encourage researchers to conduct their own quality analysis on the instrument.
However, it is important to exercise caution when selecting a personality instrument based on validity and reliability information obtained from earlier studies, as these studies had limitations in terms of the populations they examined (especially concerning demographic and linguistic characteristics), the time frames and contexts in which they were conducted. In a new study, the population of interest, the time and context may differ from those of previous studies. Consequently, researchers must carefully evaluate the validity and reliability evidence of an existing instrument in light of the specifics of the new study.
% For this, earlier studies need to have described their data samples in sufficient details, so that later researchers can make use of that information and compare it with their own data. Unfortunately, even just reporting simple descriptives of the used dataset is not universal practice in TPC research. Among the 60 TPC studies we review, only 35 provide some description of their samples. We encourage future researchers to report sample characteristics as thoroughly as possible. \citet{bender2018data} offer useful guidelines.
Once a personality instrument is selected, it is also important to cite the source of the instrument and report its validity and reliability information. Among the 60 reviewed papers, 5 use self-identified outcomes (like one’s MBTI type or Big-5 scores mentioned in a tweet or user profile) where tracing down the instruments is impossible; 14 make use of proprietary instruments whose reliability and validity information is inaccessible to the public; among the 41 that use an existing personality instrument, 9 do not mention the specific instrument and only 4 report validity or reliability information based on previous studies.
\begin{quote}
\label{C4}
\textbf{C4: Further reduce measurement error by study design}
\end{quote}
Even when the best possible instrument is used, there can still be substantial measurement error that results from other design factors of a study, especially when questionnaires are used. Factors like questionnaire characteristics (e.g., the number of questions, visual layout, topics, wording) and data collection modes (e.g., online, in person) can affect the measurement quality of the responses~\cite{biemer2013measurement}. Furthermore, factors related to respondents (e.g., inattention) can also affect measurement quality ~\citep{Fleischer2015InattentiveRI}.
% For instance, \citet{Peytchev2017ReductionOM} find longer questionnaires and items positioned later in a questionnaire to suffer from more measurement error. This suggests that when using questionnaires to measure personality, we should perhaps avoid using the longer ones (unless there is a good reason to do so, such as the need for studying facets which are typically measured only by longer questionnaires), and position the personality items earlier than other (less important) questions (e.g., demographics).
% For instance, in the following situations, the measurement quality of questionnaire responses will be negatively affected: the respondents are inattentive~\citep{Fleischer2015InattentiveRI,Alvarez2019PayingAT}; they take shortcuts in the question-answer process (e.g., always choosing the same response, a.k.a. straightlining; \citealt{Hamby2016SurveySI,Bais2020InvestigatingRP}); they respond too fast~\citep{Revilla2015WhatAT} or too slowly~\citep{Couper2013UsingPT}. To detect undesirable response behaviour like these, researchers can, for instance, include attention check items~\citep{Abbey2017AttentionBD} and analyse the response patterns~\citep{Bais2020InvestigatingRP}. Leaving out (part of) the measurements from respondents found to engage in undesirable response behaviour may help to improve the overall quality of the data.
Therefore, when planning personality data collection using questionnaires, it can be beneficial to take into account different possible sources of measurement error. This helps to further reduce measurement error in questionnaire responses, in addition to using a valid, reliable questionnaire. For a comprehensive overview of factors that can influence measurement quality in questionnaires and the possible ways to control for them, we refer you to \citet{callegaro2015web} and \citet{biemer2013measurement}. We also encourage collaboration with survey methodology experts.
%\citet[][Chapter 2.3, 2.5, 4]{callegaro2015web} and \citet[][Part A and B]{biemer2013measurement}.
% In addition to measurement error, other types of error can also occur in questionnaire data~\citep{biemer2013measurement}. Among them, non-response error is closely linked to measurement error. It refers to non-response on the item or questionnaire level. While non-response is, strictly speaking, not measurement error, it can still affect measurement quality. For instance, missing responses on some items can lead to less reliable personality estimates. See \citet{Leeuw2003PreventionAT} for recommendations on how to deal with item non-responses and \citet{fang_role_2021} for an overview of factors contributing to questionnaire-level non-responses.
\begin{quote}
\label{C5}
\textbf{C5: Quantify measurement error}
\end{quote}
Now, assume that personality measurements have been collected.
The next step is to quantify both random and systematic error.
% We discuss how this can be done for questionnaire-based and model-based measurements, separately.
%, as different methods apply.
For questionnaire-based measurements, we recommend using factor analysis, which is a type of latent variable model that relates a set of observed variables (e.g., personality questionnaire items) to some latent variables (e.g., one's true, underlying personality traits)~\citep{oberski2016mixture}. Depending on the model specification and data characteristics, factor analysis can decompose the total variation in the observed variables into different sources: variation due to the latent personality traits, variation due to systematic factors like questionnaire characteristics and the time of data collection, and finally, the unexplained variation (at the item level and at the questionnaire level). Larger variation due to the underlying personality traits and lower variation due to systematic factors are desirable, because they indicate higher measurement validity (i.e., less systematic error). In contrast, more unexplained variation indicates more random error (i.e., lack of reliability). Based on this variance decomposition, estimators of reliability and validity can be derived. We refer you to \citet[][Chapter 9-12]{saris2014design} for an overview of various factor analysis strategies and estimators of reliability and validity.
For model-based measurements, the number of measurements per personality trait and person is typically limited to one. Factor analysis cannot be applied to such data because the factor model is mathematically non-identifiable (i.e., there is no unique solution). Therefore, different methods for quantifying random and systematic error (or equivalently, reliability and validity) are needed.
Random error is due to factors that cannot be controlled. Therefore, by varying the instrument or measurement condition along such factors, we can quantify the associated random error. In TPC models, one such factor is small variation in text (e.g., the use of singular vs. plural noun, which has not been linked to personality traits by prior research) that should not affect predictions. By introducing simple perturbations to the data, and comparing the new predictions with the ones based on the original data, we can gauge the degree of random error associated with this factor. This approach is analogous to \citet{Ribeiro2020BeyondAB}'s invariance tests. \citet{du2021assessing} also showcase such reliability analyses for word embedding-based gender bias scores.
To quantify validity, apart from the usual performance metrics, we can check whether the predicted scores of different personality traits correlate with one another in expected ways. For instance, while the overall correlations should be low, as different personality traits are distinct constructs, some correlations should be more positive (e.g., between conscientiousness and agreeableness) than others (e.g., between openness and neuroticism).
\citet{VANDERLINDEN2010315} provide an overview of empirical correlations between personality traits across demographic groups that can be expected.
To the best of our knowledge, no TPC study has assessed the correlations among predicted personality scores.
Furthermore, it can be helpful to check whether the predicted personality scores relate to other constructs like emotions (if data is available) in expected ways. Such tests are conceptually similar to convergent and discriminant validity analyses in the social sciences~\citep{stachl2020personality}. Even in the presence of large measurement error in the data, they can be useful. For a more in-depth discussion of validity testing in machine learning and NLP, see \citet{jacobs2021measurement} and \citet{Fang2022EvaluatingTC}.
\begin{quote}
\label{C6}
\textbf{C6: Correct for measurement error}
\end{quote}
Choosing high-quality personality instruments and reducing measurement error by design are likely the most important and effective ways to ensure high quality measurements.
Once personality measurements have been collected, however, much less can be done about measurement error.
For questionnaire-based measurements, it might still be helpful to take a closer look at the results from factor analysis. For instance, do the measurements fit the assumed personality model (e.g., the Big-5)? If not, is it due to one or more questionnaire items that show unexpected relationships with the personality traits (e.g., the relationship is zero; the item correlates strongly with a different personality trait than expected)? If so, removing those problematic items can improve the validity of the personality measurements. Are there items with large unexplained variances? If so, removing them may increase reliability.
As for model-based measurements, if the personality models are proprietary or cannot be modified and retrained (e.g., due to lack of data or model details), then no correction for measurement error is possible. If retraining the model is possible, several techniques may help (see C12, C13 in \S\ref{sec_modelling}).
\section{Datasets}
\label{sec_datasets}
Across the 60 TPC studies, we find 41 unique datasets, which vary in terms of the personality taxonomy, instrument, type of text data, sample size, sample characteristics etc. Among them, however, only 18 are potentially accessible to other researchers (see Appendix~\ref{sec:appendixB}).
Shareable datasets are key to advancing TPC research, as it leads to accumulation of data and allows for replication studies. This invites the next challenge:
% We identify three challenges (7, 8, 9) concerning the availability and quality of datasets:
\begin{quote}
\label{C7}
\textbf{C7: Construct shareable datasets}
\end{quote}
% Only 18 out of the 41 datasets are potentially usable by other researchers (e.g., open-access datasets; datasets available upon request).
% Therefore, we recommend future TPC researchers to not only curate more TPC datasets but also share them.
% The actual number might be even smaller, considering situations where the dataset authors do not (have the time to) respond to dataset requests or the dataset cannot be shared due to new regulations like privacy laws.
One obstacle to sharing TPC datasets is privacy preservation, as TPC datasets often contain identifiable information (e.g., names, locations, events) about data subjects.
For instance, with social media posts, their authors can be easily found by using the content of the posts as search terms \citep{norman2022scraping}. We suggest two ways to make data sharing more privacy-preserving.
% In addition, TPC datasets may contain sensitive demographic information about data subjects.
First, data pseudonymization and anonymization techniques can be used. With pseudonymization, the data subjects can still be identified if additional information is provided. With anonymization, however, re-identification is impossible. Whether to pseudonymize or anonymize depends on many factors, such as the difficulty in data anonymization and the severity of re-identification. Nevertheless, for TPC datasets containing social media posts, anonymization is likely impossible. We refer you to \citet{Lison2021AnonymisationMF} for more information.
% For tabular data (e.g., demographics), the statistical disclosure control literature~\citep[e.g.,][]{willenborg2012elements} provides useful guidelines and techniques to preserve the privacy of data subjects. For text data, see \citet{Lison2021AnonymisationMF}.
Second, we can replace texts with paraphrases or synthetic data. The latter aims to “preserve the overall properties and characteristics of the original data without revealing information about actual individual data samples”~\citep{hittmeir2019utility}. However, whether these strategies are effective enough remain an open research question in NLP.
% Furthermore, data collection itself can also be privacy-preserving. For example, \citet{Boeschoten2022AFF} describe a framework for collecting digital trace data (e.g., Facebook posts, Instagram stories) through voluntary donation of data packages that users have rights to. In this framework, all data processing is done locally, and before the data is donated, the user is presented with a clear overview of the specific data being donated. This may not only help with the collection of more TPC datasets but also address privacy concerns on the user end. A potential limitation, however, is that data donation can be time- and labour-intensive for the researchers and may result in small datasets (considering the need for human participants).
\begin{quote}
\label{C8}
\textbf{C8: Finer-grained measurements}
\end{quote}
All the 18 shareable datasets we find include only aggregated measurements of personality traits. Namely, for MBTI, only the classification types (e.g., INFP and ESTJ; instead of scores on each questionnaire item) are available; for Big-5, only the aggregated scores (e.g., means across items) for the five dimensions. This makes it impossible for other researchers to investigate measurement quality or train TPC models on the facet or item level. Even worse, some datasets provide no information about the personality instrument used. This especially concerns datasets that obtain personality labels from Twitter or Reddit based on the mention of MBTI or Big-5 information in a post or user profile (e.g., “INTJ”; “As an extravert…”).
Other problematic treatments of aggregated personality measurements include further discretization, within-sample standardization or normalization to the target population. All leads to loss of information and limits the reusability of the measurements.
Therefore, we suggest providing raw personality measurements, ideally on the item level.
% \begin{quote}
% \textbf{Challenge 9: Bigger sample sizes, less selective samples, more languages.}
% \end{quote}
% The datasets used in the 60 TPC studies we surveyed vary substantially in sample sizes, from as low as 4 to as high as 155,337. Seventeen of them have sample sizes smaller than 1000. A small sample size can severely limit the generalizability of TPC results. It is also incompatible with many increasingly popular deep learning techniques (e.g. transformers), where learning on small sample sizes can be inefficient and suffer from high variance. To deal with the problem of small sample sizes, we recommend combining multiple datasets and using dimensionality reduction techniques especially when large language models are used ~\citep{v_ganesan_empirical_2021}.
% Furthermore, we observed that none of the datasets was collected using a probability sampling procedure (e.g., taking a random sample from a predefined population of interest). This generally limits the generalisability of the personality measurements, as a non-random sample may represent only a very selective group in the population. Making inferences based on non-probabilistic samples, however, is still possible by using methods like weighting and pseudo-randomisation. See \citet{FerriGarca2021WeightSF} and \citet{Elliott2017InferenceFN} for an introduction.
% In addition, we found only 14 datasets that include non-English texts, while only 3 of them are multilingual. The lack of multilingual datasets hinders the development and evaluation of multilingual TPC approaches that can potentially lead to generalizable results.
\begin{quote}
\label{C9}
\textbf{C9: Include demographic information}
\end{quote}
We argue that the inclusion of demographic information (e.g., age, gender, education) can be important. Not only can this help researchers decide the appropriate personality instrument to use (in relation to the population of interest; see earlier discussion in \S\ref{sec_measurements}: C3), it can also provide additional useful features for TPC models. Furthermore, researchers can make use of the demographic information to diagnose the model (e.g., whether the measurement quality or the model’s prediction performance differs across demographic groups) (i.e., fairness). However, it is important to weigh the gain from including extra personal information against potential harm (see \S\ref{sec_ethics}: C14).
% The inclusion of other personality-related constructs (e.g., emotions, opinions and mental health) can also be helpful, not only for the assessment of measurement quality (see \S\ref{sec_measurements}, C5) but also for personality prediction (see \S\ref{sec_modelling}, C13). However, it is important to weigh the gain from extra personal information against potential harm (see \S\ref{sec_ethics}, C14).
\section{Performance Evaluation}
\label{sec_evaluation}
Across the 60 surveyed TPC papers, we identify two challenges related to performance evaluation:
\begin{quote}
\label{C10}
\textbf{C10: Use more appropriate and consistent performance metrics}
\end{quote}
11 out of the 60 studies model TPC as a regression task. Among them, 9 use Pearson's correlation between predicted personality trait scores and the true scores as the performance metric. However, correlation-based metrics can be misleading, as they register only ranks and do not reflect how accurate the predictions are on the original scale of the personality scores~\citep{stachl2020personality}.
Some studies report mean squared error (MSE), which is arguably better than correlations because it quantifies the absolute difference between the predictions and the true values. However, MSE scores depend on the scale of the personality measurements and are not bounded, making interpretation and comparison (between studies) difficult.
% Thus, correlation metrics should only be used in applications where ranks in personality scoresthe true, absolute personality traits scores matter no more than the ranks.
% rank-based correlations (Spearman’s correlation or Kendall’s tau) can still be appropriate metrics.
% it mainly quantifies whether data points with higher ground truth values also tend to have higher predicted personality scores, without taking into account how much the predictions are off in absolute terms. That is, Pearson’s correlation
\citet{stachl2020personality} propose a better performance metric: $R^2=1-RSS/TSS$, where $RSS$ refers to the sum of squares of residuals and $TSS$ the total sum of squares. $R^2$ has several benefits. First, it can be considered a normalised version of MSE, which has an upper limit of 1 (perfect agreement). Second, $R^2$ has a natural zero point, which occurs when the mean is used as the prediction. Third, when the model makes worse predictions than a simple mean baseline, $R^2$ becomes negative.
While two studies report $R^2$, their calculation of $R^2$ is unclear. The researchers may have calculated $R^2$ not based on the formula shown earlier, but by squaring Pearson's correlations. This would lead to always positive $R^2$, which can be misleading.
In the 33 studies that model TPC as a classification task, a more diverse set of metrics are used (i.e., accuracy, recall, precision, F1, and AUC, in macro, micro, weighted or unweighted forms). One problem, however, is that different studies report different metrics (sometimes, only one). This makes comparison across studies difficult. We encourage future researchers to report all common metrics for classification studies (like those mentioned above).
% In addition, we encourage researchers to report validity and reliability assessment results (see \S\ref{sec_measurements}, C5).
% In \S\ref{sec_measurements} (C5), we discussed various ways to assess the validity and reliability of the predictions. Such information helps to provide a more holistic, psychometrically grounded assessment of model performance.
% \begin{quote}
% \textbf{Challenge 12: Estimating performance uncertainty}
% \end{quote}
% We observed that studies often report performance scores as point estimates, without any uncertainty estimation. From both statistical and machine learning literature, we know that model prediction performances are subject to uncertainty rooted in both the model (i.e. epistemic/systematic uncertainty) as well as the data (i.e. aleatoric/statistical uncertainty). Many methods have been proposed to quantify such uncertainty, such as calculating a confidence interval around a performance score through the use of bootstrapping or cross-validation procedures, as well as Bayesian estimation. When the goal is to compare two performance scores, statistical significance tests should be used (e.g. the Student’s t-test and McNemar’s test). For a more thorough discussion, we refer you to \citet{Dror2018TheHG} and \citet{Hllermeier2021AleatoricAE}.
\begin{quote}
\label{C11}
\textbf{C11: Report performance expectation}
\end{quote}
While it is normal to optimize the prediction performance of TPC models, it is also important to set correct expectations: What kind of performance can we realistically expect? How accurate can personality predictions be when only (short) text is used? How good does the performance need to be for a particular system? \citet{stajner_why_2021} make the first question even more relevant, as they find either few or mixed MBTI-related signals in typical text data used for MBTI prediction.
% In \S\ref{sec_measurements}, we discuss the presence and influence of measurement error in personality data that is often used as the gold standard for model training.
Quantifying measurement error in the personality scores used for modelling can help researchers to set clearer expectations about model performance, because high systematic error will limit the model's generalizability to new data, while high random error will result in unstable predictions.
% Namely, measurement error sets an upper limit on the model's prediction performance.
Thus, setting expectations "forces" researchers to learn more about their data and to avoid unrealistic expectations that may lead to problematic research practices like cherry-picking results. Unfortunately, none of the 60 reviewed papers discusses performance expectation.
% This remains an open challenge. Joint expertise from psychology and NLP may help to further address it.
% To quantify, factor analysis (see C3)
% sets an upper limit on the performance of the prediction model. Thus, (see C3).
% Where multiple annotations are available, annotator disagreement can be assessed~\citep{artstein2008inter}.
% Setting correct expectations about prediction performance not only forces
% \begin{quote}
% \textbf{Challenge 14: Conducting robustness checks}
% \end{quote}
% TPC studies, like many other empirical studies, involve many decisions that researchers need to make. For instance, which cut-off point should I use to exclude participants who have less text data? What kind of preprocessing should I apply to the text data? Should I keep the URLs in the data? How about hashtags? Which random seed should I use? Which dimension size should I specify for this transformer model? The inherent flexibility in research decisions as such is called “researcher degree of freedom”, which has been shown to easily result in false-positive findings~\citep{simmons2016false}.
% To make sure that the study’s results are robust (i.e. invariant to such degree of freedom), we recommend conducting sensitivity analysis with regard to the different subjective decisions that can be made. This means, for instance, re-running the experiments by using a different but equally acceptable preprocessing method or model, and checking whether the results vary. Naturally, exhausting all combinations of decisions is impractical, given limited research resources. Nevertheless, we still argue that it is important to check the robustness of a model against some variable conditions that can conceivably affect the result. We also acknowledge that this is not an issue specific to TPC research but applicable to most empirical NLP, CL and machine learning research. However, we still consider it worth mentioning, as the field hopefully moves forward towards more robust findings.
% \begin{quote}
% \textbf{Challenge 15: Validating on other languages and datasets}
% \end{quote}
% Among the 60 surveyed papers, we also noticed a general lack of validation across languages and datasets, which is inherently linked to the aforementioned lack of shareable and multilingual datasets. In our literature search, only 3 studies attempted at such validation. Cross-lingual and cross-dataset studies can not only reveal limitations of a model but also facilitate the development of more generalizable models~\citep[see][]{Fortuna2018MergingDF,ArmengolEstape2021AreMM}.
\section{Modelling Choices}
\label{sec_modelling}
Most TPC studies model different personality traits (i.e., dimensions) separately. This strategy is undesirable, because it ignores the correlations among personality traits that models can learn from. Modelling personality traits jointly may also help to prevent overfitting to a specific trait and thus learn more universal personality representations~\citep{Liu2019MultiTaskDN}. Hence, the next challenge:
\begin{quote}
\label{C12}
\textbf{C12: Joint personality modelling}
\end{quote}
Out of the 60 studies, only 5 attempt at (some form of) joint modelling of personality traits. \citet{yang_learning_2021} implement a transformer-based model to predict MBTI types, where the use of questionnaire texts allows the model to infer automatically the relevant MBTI dimension, and hence removes the need for independent modelling of different MBTI dimensions. \citet{gjurkovic_reddit_2018}, \citet{bassignana_matching_2020} and \citet{hosseinia_usefulness_2021} frame the prediction of MBTI types as a 16-class classification task (as there are in total 16 MBTI types), thereby using only one single model. \citet{hull_personality_2021} apply “stacked single target chains”~\citep{Xioufis2016MultitargetRV}, which feeds the predictions of one personality trait back in as features for the prediction of the next trait(s).
Multitask learning may also be useful, which trains a model on multiple tasks simultaneously and thus might help to improve the generalizability of the model~\citep{caruana1997multitask}. In addition, \citet{stachl2020personality} suggest modifying a model’s loss function such that the correlations between theoretically distinct constructs are minimised. Building on this idea, we can also specify the loss function in a way that it not only focuses on general prediction performance but also minimises the difference between the predicted covariance matrix and the observed covariance matrix of personality traits.
\begin{quote}
\label{C13}
\textbf{C13: Build on best modelling choices}
\end{quote}
As the field progresses, it is important to not only investigate new modelling ideas, but also accumulate knowledge about best modelling practices. We list below several empirically supported ideas that should not slip past the community’s attention.
First, researchers should leverage the texts in personality questionnaires. \citet{kreuter_items_2022}, \citet{vu_predicting_2020} and \citet{yang_learning_2021} find that incorporating personality questionnaire texts into model learning can lead to better personality predictions.
Second, when sample sizes are small, data augmentation and dimensionality reduction techniques are beneficial. \citet{kreuter_items_2022} show that using data augmentation to increase the training size of personality questionnaire items leads to better predictions. \citet{v_ganesan_empirical_2021} show that PCA helps to overcome the problem of fine-tuning large language models with a small TPC dataset.
Third, incorporating personality-related variables (e.g., psycholinguistic features, emotions, user interests, demographics, opinions and brand preference) as features, if available, can improve personality predictions \citep{kerz_pushing_2022, cornelisse_inferring_2020, kishima_construction_2021, li_prompt-based_2022, hosseinia_usefulness_2021, yang_using_2015}.
% Lastly, try using multiple diverse prediction models and aggregating their predictions (i.e., ensemble modelling). Several studies \citep[e.g.,][]{lynn_hierarchical_2020,kerz_pushing_2022} showed that this strategy could easily outperform state-of-the-art results.
% \section{Personality Theory Development}
% The field of TPC focuses on the prediction of personality traits. While this is unarguably the main goal, we believe it is also beneficial for the field to pay some attention to questions like: Which textual features explain the prediction of a particular personality trait? In which way (e.g. a positive or negative correlation)? Are these relationships causal or correlational? Will these observations last? Accumulation of insights from answering such questions can be helpful for the development of personality theory, which in turn can also benefit TPC.
% Among the 60 papers we reviewed, only a small number offered some insights into the first two questions (and none for the others). These are studies (e.g., \citealt{he_personality_2021}) that made use of interpretable linguistic features in combination with interpretable models like linear regression. Most studies, however, used mostly blackbox models (e.g., support vector machines, random forests and transformers) without any manual feature engineering or any attempt at explaining model predictions. The only exception is \citet{lynn_hierarchical_2020}, where the authors implemented a model with both word- and message-level attention mechanisms and illustrated how different chucks of texts at both word- and message-level contributed to the prediction of different levels of MBTI dimensions via attention weights.
% This led to the following challenge:
% \begin{quote}
% \textbf{Challenge 18: Advancing theory of personality and language}
% \end{quote}
% In most social science disciplines (including psychology), an ideal theory should be parsimonious, persistent (through time), causal (i.e. can explain things) and generalisable (across regions and populations). This means that, for TPC to provide insights into personality theory development, many requirements need to be fulfilled. To begin with, one needs to be able to explain a model’s predictions. Techniques from interpretable machine learning and NLP can offer great help (see Belinkov et al., 2020 for a comprehensive overview). Also, the search for persistent features is necessary. This means that one should watch out for potential concept drift in a TPC model. See \citet{Chalkidis2022ImprovedMC} for techniques on how to address the issue of concept drift. Furthermore, identifying causal rather than correlational features may help (see e.g., \citealt{AlvarezMelis2017ACF}; \citealt{Feder2021CausaLMCM}; \citealt{Pryzant2021CausalEO}).
\section{Ethics and Fairness}
\label{sec_ethics}
Out of the 60 reviewed papers and the 2 additional survey papers, only 7 provide some reflection about ethics and none about fairness. This can be because ethics and fairness only became central in NLP recently.
Nevertheless, the last two challenges:
\begin{quote}
\label{C14}
\textbf{C14: More ethical and useful TPC}
\end{quote}
As useful as TPC can be, it is important to ask whether gathering personal information like personality or computing them is necessary. This is especially relevant for research where TPC is only an intermediate step to another end such as opinion mining~\citep{hosseinia_usefulness_2021}, dialogue generation~\citep{mairesse_trainable_2008} and brand preference prediction~\citep{yang_using_2015}. Such studies typically argue that the computed personality traits can be used as features for another task and that it leads to better task performance; however, they do not consider alternatives (e.g., replacing the prediction of personality traits with using lexical cues that are non-personal but still indicative of personality). Thus, we encourage researchers to justify PC and to find alternatives when PC is only a means.
Even when TPC can be justified, it is important to reduce potential harm. For instance, many TPC studies and datasets make use of public social media profiles for predicting personality traits. While this is often legal, no explicit consent for PC is obtained from the social media users, which makes using public social media data an ethically ambiguous issue~\citep{norman2022scraping}. \citet{Boeschoten2022AFF} proposes a privacy-preserving data donation framework that may help to alleviate this problem, where data subjects can voluntarily donate their data download packages (e.g., from social media accounts) for research and give explicit consent.
To further increase the benefit of TPC, we can consider applying it to clinical, professional or educational settings, where (traditional) personality assessment has proven useful and relevant (e.g., personalised treatments; career recommendation; individualised learning). None of the 60 TPC studies in our review investigates these applications.
\begin{quote}
\label{C15}
\textbf{C15: Research on Fair TPC}
\end{quote}
Fairness research in machine learning concerns identifying and mitigating biases that may be present within a system, particularly towards specific groups~\citep{mehrabi_survey_2021}. For instance, a fair TPC model should exhibit equal predictive performance across different demographic groups.
Among many others, biased training data is a significant factor contributing to algorithmic bias. From the perspective of measurement quality, the choice of a personality instrument that lacks equal validity and reliability across all demographic groups of interest can introduce variations in the quality of personality measurements among different groups. Consequently, these discrepancies can perpetuate algorithmic bias within the system.
Remarkably, none of the 60 TPC papers we survey address the topic of fairness. Therefore, there is a clear need for future research on fairness in the context of TPC.
\section{Conclusion}
In this paper, we review 60 TPC papers from the ACL Anthology and identify 15 challenges that we consider deserving the attention of the research community. We focus on the following 6 topics: personality taxonomies, measurement quality, datasets, performance evaluation, modelling choices, as well as ethics and fairness. While some of these topics (e.g., personality taxonomies and ethics) have been discussed elsewhere, we provide new perspectives. Furthermore, in light of these challenges, we offer concrete recommendations for future TPC research, which we summarise below:
\begin{itemize}
\item Personality taxonomies: Choose Big-5 over MBTI; Try modelling facets and using other taxonomies like HEXACO where appropriate.
\item Measurement quality: Pay attention to measurement error in personality measurements, be they based on questionnaires or models; Try to reduce measurement error by design (e.g., choose higher-quality instruments; use better data collection practices); Provide quality evaluation (i.e., validity and reliability) for any new (and also existing) approaches.
\item Datasets: Make TPC datasets shareable, which should also contain fine-grained personality measurements and descriptions of the target population;
\item Performance evaluation: Report a diverse set of performance metrics; Report $R^2$ for a regression task.
\item Modelling choices: Make use of their psychometric properties when modelling personality traits (e.g., use joint modelling; modify the loss function to preserve the covariance information); For even better predictions, try incorporating personality questionnaire texts, applying data augmentation and dimensionality reduction techniques, as well as incorporating more personality-related variables.
% \item Personality theory development: Contribute to personality theory development by using interpretable methods and searching for persistent and causal features.
\item Ethics and fairness: Avoid unnecessary TPC; Apply TPC to clinical, professional and educational settings; Investigate fairness.
\item Lastly, engage in (interdisciplinary) research work with survey methodologists, psychologists, and psychometricians.
\end{itemize}
We hope that our paper will inspire better TPC research and new research directions.
\section*{Limitations}
Our paper has some limitations. First, we do not give detailed instructions about techniques that we recommended (e.g., factor analysis, synthetic data generation). We rely on our readers’ autonomy to acquire the necessary information (that is specific to their research projects) by further reading our recommended references. Second, we only survey TPC papers included in the ACL Anthology, despite other TPC papers existing outside this venue. While this means that the challenges we identified might be specific to these papers, we believe they are still a good representation of the TPC research done in NLP. Lastly, we limit our discussion to text data. It would be beneficial for future research to also discuss challenges facing PCs based on other types of data (e.g., images, behaviours, videos), which may offer additional insights to TPC research.
\section*{Statement of Ethics and Impact}
Our work provides a critical evaluation of past TPC research, where we present 15 open challenges that we consider deserving the attention of the research community. For each of these challenges, we offer concrete suggestions, thereby hoping to inspire higher-quality TPC research (e.g., more valid and reliability personality measurements, better datasets, better modelling practices). We also discuss issues related to ethics and fairness (see \S\ref{sec_ethics}). We hope to see more ethical and fair TPC research.
\section*{Acknowledgements}
We thank Dr. Dong Nguyen, Anna Wegmann, Christian Fang and Yupei Du from Utrecht University, as well as our anonymous reviewers for their valuable feedback on this paper.
This work is partially supported by the Dutch Research Council (NWO) (grant number: VI.Vidi.195.152 to D. L. Oberski).
\begin{thebibliography}{130}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Akrami et~al.(2019)Akrami, Fernquist, Isbister, Kaati, and
Pelzer}]{Akrami2019AutomaticEO}
Nazar Akrami, Johan Fernquist, Tim Isbister, Lisa Kaati, and Bj{\"o}rn Pelzer.
2019.
\newblock Automatic extraction of personality from text: Challenges and
opportunities.
\newblock \emph{2019 IEEE International Conference on Big Data (Big Data)},
pages 3156--3164.
\bibitem[{Al~Khatib et~al.(2020)Al~Khatib, Völske, Syed, Kolyada, and
Stein}]{al_khatib_exploiting_2020}
Khalid Al~Khatib, Michael Völske, Shahbaz Syed, Nikolay Kolyada, and Benno
Stein. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.632} {Exploiting
personal characteristics of debaters for predicting persuasiveness}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7067--7072. Association for
Computational Linguistics.
\bibitem[{APA(2022)}]{apaDictionary23}
American Psychological~Association APA. 2022.
\newblock Personality.
\newblock In \emph{APA Dictionary of Psychology}. Retrieved November 09, 2022
from https://dictionary.apa.org/personality.
\bibitem[{Arnoux et~al.(2017)Arnoux, Xu, Boyette, Mahmud, Akkiraju, and
Sinha}]{Arnoux201725TT}
Pierre-Hadrien Arnoux, Anbang Xu, Neil Boyette, Jalal Mahmud, Rama Akkiraju,
and Vibha Sinha. 2017.
\newblock 25 tweets to know you: A new model to predict personality with social
media.
\newblock In \emph{International Conference on Web and Social Media}.
\bibitem[{Ashton and Lee(2007)}]{ashton2007empirical}
Michael~C Ashton and Kibeom Lee. 2007.
\newblock Empirical, theoretical, and practical advantages of the hexaco model
of personality structure.
\newblock \emph{Personality and social psychology review}, 11(2):150--166.
\bibitem[{Bagby et~al.(2016)Bagby, Gralnick, Al-Dajani, and
Uliaszek}]{bagby2016role}
R~Michael Bagby, Tara~M Gralnick, Nadia Al-Dajani, and Amanda~A Uliaszek. 2016.
\newblock The role of the five-factor model in personality assessment and
treatment planning.
\newblock \emph{Clinical Psychology: Science and Practice}, 23(4):365.
\bibitem[{Barriere et~al.(2022)Barriere, Tafreshi, Sedoc, and
Alqahtani}]{barriere_wassa_2022}
Valentin Barriere, Shabnam Tafreshi, João Sedoc, and Sawsan Alqahtani. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.wassa-1.20} {{WASSA} 2022
shared task: Predicting empathy, emotion and personality in reaction to news
stories}.
\newblock In \emph{Proceedings of the 12th Workshop on Computational Approaches
to Subjectivity, Sentiment \& Social Media Analysis}, pages 214--227.
Association for Computational Linguistics.
\bibitem[{Bassignana et~al.(2020{\natexlab{a}})Bassignana, Nissim, Patti
et~al.}]{bassignana2020personal}
E~Bassignana, M~Nissim, V~Patti, et~al. 2020{\natexlab{a}}.
\newblock Personal-ity: A novel youtube-based corpus for personality prediction
in italian.
\newblock In \emph{7th Italian Conference on Computational Linguistics, CLiC-it
2020}, volume 2769, pages 1--7. CEUR-WS.
\bibitem[{Bassignana et~al.(2020{\natexlab{b}})Bassignana, Nissim, and
Patti}]{bassignana_matching_2020}
Elisa Bassignana, Malvina Nissim, and Viviana Patti. 2020{\natexlab{b}}.
\newblock \href {https://aclanthology.org/2020.peoples-1.2} {Matching theory
and data with personal-{ITY}: What a corpus of italian {YouTube} comments
reveals about personality}.
\newblock In \emph{Proceedings of the Third Workshop on Computational Modeling
of People's Opinions, Personality, and Emotion's in Social Media}, pages
11--22. Association for Computational Linguistics.
\bibitem[{Benet-Martinez and John(1998)}]{benet1998cinco}
Veronica Benet-Martinez and Oliver~P John. 1998.
\newblock Los cinco grandes across cultures and ethnic groups:
Multitrait-multimethod analyses of the big five in spanish and english.
\newblock \emph{Journal of personality and social psychology}, 75(3):729.
\bibitem[{Biel and Gatica-Perez(2012)}]{biel2012youtube}
Joan-Isaac Biel and Daniel Gatica-Perez. 2012.
\newblock The youtube lens: Crowdsourced personality impressions and
audiovisual analysis of vlogs.
\newblock \emph{IEEE Transactions on Multimedia}, 15(1):41--55.
\bibitem[{Biemer et~al.(2013)Biemer, Groves, Lyberg, Mathiowetz, and
Sudman}]{biemer2013measurement}
Paul~P Biemer, Robert~M Groves, Lars~E Lyberg, Nancy~A Mathiowetz, and Seymour
Sudman. 2013.
\newblock \emph{Measurement errors in surveys}, volume 548.
\newblock John Wiley \& Sons.
\bibitem[{Bleidorn and Hopwood(2019)}]{bleidorn2019using}
Wiebke Bleidorn and Christopher~James Hopwood. 2019.
\newblock Using machine learning to advance personality assessment and theory.
\newblock \emph{Personality and Social Psychology Review}, 23(2):190--203.
\bibitem[{Boeschoten et~al.(2022)Boeschoten, Ausloos, M{\"o}ller, Araujo, and
Oberski}]{Boeschoten2022AFF}
Laura~E. Boeschoten, Jef Ausloos, Judith~E. M{\"o}ller, Theo~B. Araujo, and
Daniel~L. Oberski. 2022.
\newblock A framework for privacy preserving digital trace data collection
through data donation.
\newblock \emph{Computational Communication Research}.
\bibitem[{Briggs~Myers and Myers(1995)}]{briggs1995gifts}
Isabel Briggs~Myers and Peter~B Myers. 1995.
\newblock Gifts differing: Understanding personality type.
\newblock \emph{Davis-Black Publishing}.
\bibitem[{Callegaro et~al.(2015)Callegaro, Manfreda, and
Vehovar}]{callegaro2015web}
Mario Callegaro, Katja~Lozar Manfreda, and Vasja Vehovar. 2015.
\newblock \emph{Web survey methodology}.
\newblock Sage.
\bibitem[{Caruana(1997)}]{caruana1997multitask}
Rich Caruana. 1997.
\newblock Multitask learning.
\newblock \emph{Machine learning}, 28(1):41--75.
\bibitem[{Cattell(1946)}]{cattell1946description}
Raymond~Bernard Cattell. 1946.
\newblock \emph{Description and measurement of personality.}
\newblock World Book Company.
\bibitem[{Cervone and Pervin(2022)}]{cervone2022personality}
Daniel Cervone and Lawrence~A Pervin. 2022.
\newblock \emph{Personality: Theory and research}.
\newblock John Wiley \& Sons.
\bibitem[{Cornelisse(2020)}]{cornelisse_inferring_2020}
Joran Cornelisse. 2020.
\newblock \href {https://aclanthology.org/2020.peoples-1.1} {Inferring
neuroticism of twitter users by utilizing their following interests}.
\newblock In \emph{Proceedings of the Third Workshop on Computational Modeling
of People's Opinions, Personality, and Emotion's in Social Media}, pages
1--10. Association for Computational Linguistics.
\bibitem[{Costa and McCrae(1992)}]{costa1992professional}
Paul~T Costa and Robert~R McCrae. 1992.
\newblock Professional manual: revised neo personality inventory (neo-pi-r) and
neo five-factor inventory (neo-ffi).
\newblock \emph{Odessa, FL: Psychological Assessment Resources}, 61.
\bibitem[{Culnan et~al.(2021)Culnan, Park, Krishnaswamy, and
Sharp}]{culnan_me_2021}
John Culnan, Seongjin Park, Meghavarshini Krishnaswamy, and Rebecca Sharp.
2021.
\newblock \href {https://aclanthology.org/2021.wassa-1.26} {Me, myself, and
ire: Effects of automatic transcription quality on emotion, sarcasm, and
personality detection}.
\newblock In \emph{Proceedings of the Eleventh Workshop on Computational
Approaches to Subjectivity, Sentiment and Social Media Analysis}, pages
250--256. Association for Computational Linguistics.
\bibitem[{Das and Das(2017)}]{das_developing_2017}
Kumar~Gourav Das and Dipankar Das. 2017.
\newblock \href {https://aclanthology.org/W17-7545} {Developing lexicon and
classifier for personality identification in texts}.
\newblock In \emph{Proceedings of the 14th International Conference on Natural
Language Processing ({ICON}-2017)}, pages 362--372. {NLP} Association of
India.
\bibitem[{Derakhshi et~al.(2021)Derakhshi, Feizi-Derakhshi, Ramezani,
Nikzad-Khasmakhi, Asgari-Chenaghlu, Akan, Ranjbar-Khadivi, Zafarni-Moattar,
and Jahanbakhsh-Naghadeh}]{FeiziDerakhshi2021TextbasedAP}
Ali Reza~Feizi Derakhshi, Mohammad-Reza Feizi-Derakhshi, Majid Ramezani, Narjes
Nikzad-Khasmakhi, Meysam Asgari-Chenaghlu, Taymaz Akan, Mehrdad
Ranjbar-Khadivi, Elnaz Zafarni-Moattar, and Zoleikha Jahanbakhsh-Naghadeh.
2021.
\newblock Text-based automatic personality prediction: a bibliographic review.
\newblock \emph{Journal of Computational Social Science}, 5:1555 -- 1593.
\bibitem[{Dodge et~al.(2003)Dodge, Cox, and Commenges}]{dodge2003oxford}
Yadolah Dodge, David Cox, and Daniel Commenges. 2003.
\newblock \emph{The Oxford dictionary of statistical terms}.
\newblock Oxford University Press on Demand.
\bibitem[{Du et~al.(2021)Du, Fang, and Nguyen}]{du2021assessing}
Yupei Du, Qixiang Fang, and Dong Nguyen. 2021.
\newblock Assessing the reliability of word embedding gender bias measures.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 10012--10034.
\bibitem[{Fang et~al.(2022)Fang, Nguyen, and Oberski}]{Fang2022EvaluatingTC}
Qixiang Fang, Dong Nguyen, and Daniel~L. Oberski. 2022.
\newblock Evaluating the construct validity of text embeddings with application
to survey questions.
\newblock \emph{EPJ Data Science}, 11:1--31.
\bibitem[{Fleischer et~al.(2015)Fleischer, Mead, and
Huang}]{Fleischer2015InattentiveRI}
Avi Fleischer, Alan Mead, and Jialin Huang. 2015.
\newblock Inattentive responding in mturk and other online samples.
\newblock \emph{Industrial and Organizational Psychology}, 8:196 -- 202.
\bibitem[{Fung et~al.(2016)Fung, Dey, Siddique, Lin, Yang, Wan, and
Chan}]{fung_zara_2016}
Pascale Fung, Anik Dey, Farhad~Bin Siddique, Ruixi Lin, Yang Yang, Yan Wan, and
Ho~Yin~Ricky Chan. 2016.
\newblock \href {https://doi.org/10.18653/v1/N16-3018} {Zara the supergirl: An
empathetic personality recognition system}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Demonstrations},
pages 87--91. Association for Computational Linguistics.
\bibitem[{Ghosh et~al.(2022)Ghosh, Maurya, Ekbal, and
Bhattacharyya}]{ghosh_team_2022}
Soumitra Ghosh, Dhirendra Maurya, Asif Ekbal, and Pushpak Bhattacharyya. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.wassa-1.26} {Team
{IITP}-{AINLPML} at {WASSA} 2022: Empathy detection, emotion classification
and personality detection}.
\newblock In \emph{Proceedings of the 12th Workshop on Computational Approaches
to Subjectivity, Sentiment \& Social Media Analysis}, pages 255--260.
Association for Computational Linguistics.
\bibitem[{Gill et~al.(2012)Gill, Brockmann, and
Oberlander}]{gill_perceptions_2012}
Alastair Gill, Carsten Brockmann, and Jon Oberlander. 2012.
\newblock \href {https://aclanthology.org/W12-1508} {Perceptions of alignment
and personality in generated dialogue}.
\newblock In \emph{{INLG} 2012 Proceedings of the Seventh International Natural
Language Generation Conference}, pages 40--48. Association for Computational
Linguistics.
\bibitem[{Gjurković et~al.(2021)Gjurković, Karan, Vukojević, Bošnjak, and
Snajder}]{gjurkovic_pandora_2021}
Matej Gjurković, Mladen Karan, Iva Vukojević, Mihaela Bošnjak, and Jan
Snajder. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.socialnlp-1.12} {{PANDORA}
talks: Personality and demographics on reddit}.
\newblock In \emph{Proceedings of the Ninth International Workshop on Natural
Language Processing for Social Media}, pages 138--152. Association for
Computational Linguistics.
\bibitem[{Gjurković and Šnajder(2018)}]{gjurkovic_reddit_2018}
Matej Gjurković and Jan Šnajder. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-1112} {Reddit: A gold mine for
personality prediction}.
\newblock In \emph{Proceedings of the Second Workshop on Computational Modeling
of People's Opinions, Personality, and Emotions in Social Media}, pages
87--97. Association for Computational Linguistics.
\bibitem[{Goldberg(1982)}]{goldberg1982ace}
Lewis~R Goldberg. 1982.
\newblock From ace to zombie: Some explorations in the language of personality.
\newblock \emph{Advances in personality assessment}, 1:203--234.
\bibitem[{Goldberg(1990)}]{goldberg1990alternative}
Lewis~R Goldberg. 1990.
\newblock An alternative" description of personality": the big-five factor
structure.
\newblock \emph{Journal of personality and social psychology}, 59(6):1216.
\bibitem[{Gorman and Bedrick(2019)}]{gorman2019we}
Kyle Gorman and Steven Bedrick. 2019.
\newblock We need to talk about standard splits.
\newblock In \emph{Proceedings of the 57th annual meeting of the association
for computational linguistics}, pages 2786--2791.
\bibitem[{Grant(2013)}]{grant2013goodbye}
Adam Grant. 2013.
\newblock Goodbye to mbti, the fad that won’t die.
\newblock \emph{Psychology Today}, 9.
\bibitem[{Guntuku et~al.(2018)Guntuku, Giorgi, and
Ungar}]{guntuku_current_2018}
Sharath~Chandra Guntuku, Salvatore Giorgi, and Lyle Ungar. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-0610} {Current and future
psychological health prediction using language and socio-demographics of
children for the {CLPysch} 2018 shared task}.
\newblock In \emph{Proceedings of the Fifth Workshop on Computational
Linguistics and Clinical Psychology: From Keyboard to Clinic}, pages 98--106.
Association for Computational Linguistics.
\bibitem[{He and de~Melo(2021)}]{he_personality_2021}
Xiaoli He and Gerard de~Melo. 2021.
\newblock \href {https://aclanthology.org/2021.ranlp-1.58} {Personality
predictive lexical cues and their correlations}.
\newblock In \emph{Proceedings of the International Conference on Recent
Advances in Natural Language Processing ({RANLP} 2021)}, pages 514--523.
{INCOMA} Ltd.
\bibitem[{Hittmeir et~al.(2019)Hittmeir, Ekelhart, and
Mayer}]{hittmeir2019utility}
Markus Hittmeir, Andreas Ekelhart, and Rudolf Mayer. 2019.
\newblock On the utility of synthetic data: An empirical evaluation on machine
learning tasks.
\newblock In \emph{Proceedings of the 14th International Conference on
Availability, Reliability and Security}, pages 1--6.
\bibitem[{Hosseinia et~al.(2021)Hosseinia, Dragut, Boumber, and
Mukherjee}]{hosseinia_usefulness_2021}
Marjan Hosseinia, Eduard Dragut, Dainis Boumber, and Arjun Mukherjee. 2021.
\newblock \href {https://aclanthology.org/2021.ranlp-1.62} {On the usefulness
of personality traits in opinion-oriented tasks}.
\newblock In \emph{Proceedings of the International Conference on Recent
Advances in Natural Language Processing ({RANLP} 2021)}, pages 547--556.
{INCOMA} Ltd.
\bibitem[{Hull et~al.(2021)Hull, Novak, Rytting, Rodrigues, Frank, and
Swahn}]{hull_personality_2021}
James~R. Hull, Valerie Novak, C.~Anton Rytting, Paul Rodrigues, Victor~M.
Frank, and Matthew Swahn. 2021.
\newblock \href {https://aclanthology.org/2021.ranlp-1.66} {Personality trait
identification using the russian feature extraction toolkit}.
\newblock In \emph{Proceedings of the International Conference on Recent
Advances in Natural Language Processing ({RANLP} 2021)}, pages 583--592.
{INCOMA} Ltd.
\bibitem[{Iwai et~al.(2020{\natexlab{a}})Iwai, Kawahara, Kumada, and
Kurohashi}]{iwai_acquiring_2020}
Ritsuko Iwai, Daisuke Kawahara, Takatsune Kumada, and Sadao Kurohashi.
2020{\natexlab{a}}.
\newblock Acquiring social knowledge about personality and driving-related
behavior.
\newblock In \emph{Proceedings of the 12th Language Resources and Evaluation
Conference}, pages 2306--2315. European Language Resources Association.
\bibitem[{Iwai et~al.(2020{\natexlab{b}})Iwai, Kawahara, Kumada, and
Kurohashi}]{iwai_development_2020}
Ritsuko Iwai, Daisuke Kawahara, Takatsune Kumada, and Sadao Kurohashi.
2020{\natexlab{b}}.
\newblock Development of a japanese personality dictionary based on
psychological methods.
\newblock In \emph{Proceedings of the 12th Language Resources and Evaluation
Conference}, pages 3103--3108. European Language Resources Association.
\bibitem[{Jacobs and Wallach(2021)}]{jacobs2021measurement}
Abigail~Z Jacobs and Hanna Wallach. 2021.
\newblock Measurement and fairness.
\newblock In \emph{Proceedings of the 2021 ACM conference on fairness,
accountability, and transparency}, pages 375--385.
\bibitem[{Jansen et~al.(2020)Jansen, Truong, Heylen, and
Nazareth}]{jansen_introducing_2020}
Michel-Pierre Jansen, Khiet~P. Truong, Dirk~K.J. Heylen, and Deniece~S.
Nazareth. 2020.
\newblock \href {https://aclanthology.org/2020.lrec-1.534} {Introducing
{MULAI}: A multimodal database of laughter during dyadic interactions}.
\newblock In \emph{Proceedings of the 12th Language Resources and Evaluation
Conference}, pages 4333--4342. European Language Resources Association.
\bibitem[{John and Srivastava(1999)}]{john1999big}
Oliver~P John and Sanjay Srivastava. 1999.
\newblock The big-five trait taxonomy: History, measurement, and theoretical
perspectives.
\newblock In L.~A. Pervin and O.~P. John, editors, \emph{Handbook of
personality: Theory and research}, page 102–138. Guilford Press.
\bibitem[{Jung(1971)}]{jung1971psychological}
Carl~G Jung. 1971.
\newblock \emph{Psychological Types}.
\newblock Princeton University Press, Princeton, New Jersey.
\bibitem[{Kamijo et~al.(2016)Kamijo, Nasukawa, and
Kitamura}]{kamijo-etal-2016-personality}
Koichi Kamijo, Tetsuya Nasukawa, and Hideya Kitamura. 2016.
\newblock \href {https://aclanthology.org/W16-4311} {Personality estimation
from {J}apanese text}.
\newblock In \emph{Proceedings of the Workshop on Computational Modeling of
People{'}s Opinions, Personality, and Emotions in Social Media ({PEOPLES})},
pages 101--109, Osaka, Japan. The COLING 2016 Organizing Committee.
\bibitem[{Kampman et~al.(2018)Kampman, J.~Barezi, Bertero, and
Fung}]{kampman_investigating_2018}
Onno Kampman, Elham J.~Barezi, Dario Bertero, and Pascale Fung. 2018.
\newblock \href {https://doi.org/10.18653/v1/P18-2096} {Investigating audio,
video, and text fusion methods for end-to-end automatic personality
prediction}.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Papers)}, pages 606--611.
Association for Computational Linguistics.
\bibitem[{Kern et~al.(2019)Kern, McCarthy, Chakrabarty, and
Rizoiu}]{kern2019social}
Margaret~L Kern, Paul~X McCarthy, Deepanjan Chakrabarty, and Marian-Andrei
Rizoiu. 2019.
\newblock Social media-predicted personality traits and values can help match
people to their ideal jobs.
\newblock \emph{Proceedings of the National Academy of Sciences},
116(52):26459--26464.
\bibitem[{Kerz et~al.(2022)Kerz, Qiao, Zanwar, and
Wiechmann}]{kerz_pushing_2022}
Elma Kerz, Yu~Qiao, Sourabh Zanwar, and Daniel Wiechmann. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.wassa-1.17} {Pushing on
personality detection from verbal behavior: A transformer meets text contours
of psycholinguistic features}.
\newblock In \emph{Proceedings of the 12th Workshop on Computational Approaches
to Subjectivity, Sentiment \& Social Media Analysis}, pages 182--194.
Association for Computational Linguistics.
\bibitem[{Khan et~al.(2005)Khan, Jacobson, Gardner, Prescott, and
Kendler}]{khan2005personality}
Amir~A Khan, Kristen~C Jacobson, Charles~O Gardner, Carol~A Prescott, and
Kenneth~S Kendler. 2005.
\newblock Personality and comorbidity of common psychiatric disorders.
\newblock \emph{The British Journal of Psychiatry}, 186(3):190--196.
\bibitem[{Kishima et~al.(2021)Kishima, Matsumoto, Yoshida, and
Kita}]{kishima_construction_2021}
Ryota Kishima, Kazuyuki Matsumoto, Minoru Yoshida, and Kenji Kita. 2021.
\newblock \href {https://aclanthology.org/2021.paclic-1.28} {Construction of
{MBTI} personality estimation model considering emotional information}.
\newblock In \emph{Proceedings of the 35th Pacific Asia Conference on Language,
Information and Computation}, pages 262--269. Association for Computational
Lingustics.
\bibitem[{Komarraju et~al.(2011)Komarraju, Karau, Schmeck, and
Avdic}]{komarraju2011big}
Meera Komarraju, Steven~J Karau, Ronald~R Schmeck, and Alen Avdic. 2011.
\newblock The big five personality traits, learning styles, and academic
achievement.
\newblock \emph{Personality and individual differences}, 51(4):472--477.
\bibitem[{Koutsombogera and Vogel(2018)}]{koutsombogera_modeling_2018}
Maria Koutsombogera and Carl Vogel. 2018.
\newblock \href {https://aclanthology.org/L18-1466} {Modeling collaborative
multimodal behavior in group dialogues: The {MULTISIMO} corpus}.
\newblock In \emph{Proceedings of the Eleventh International Conference on
Language Resources and Evaluation ({LREC} 2018)}. European Language Resources
Association ({ELRA}).
\bibitem[{Kreuter et~al.(2022)Kreuter, Sassenberg, and
Klinger}]{kreuter_items_2022}
Anne Kreuter, Kai Sassenberg, and Roman Klinger. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.wassa-1.35} {Items from
psychometric tests as training data for personality profiling models of
twitter users}.
\newblock In \emph{Proceedings of the 12th Workshop on Computational Approaches
to Subjectivity, Sentiment \& Social Media Analysis}, pages 315--323.
Association for Computational Linguistics.
\bibitem[{Lan and Paraboni(2018)}]{lan_definite_2018}
Alex Lan and Ivandré Paraboni. 2018.
\newblock \href {https://aclanthology.org/L18-1475} {Definite description
lexical choice: taking speaker's personality into account}.
\newblock In \emph{Proceedings of the Eleventh International Conference on
Language Resources and Evaluation ({LREC} 2018)}. European Language Resources
Association ({ELRA}).
\bibitem[{Lang et~al.(2011)Lang, John, L{\"u}dtke, Schupp, and
Wagner}]{lang2011short}
Frieder~R Lang, Dennis John, Oliver L{\"u}dtke, J{\"u}rgen Schupp, and Gert~G
Wagner. 2011.
\newblock Short assessment of the big five: Robust across survey methods except
telephone interviewing.
\newblock \emph{Behavior research methods}, 43(2):548--567.
\bibitem[{Levitan et~al.(2016)Levitan, Levitan, An, Levine, Levitan, Rosenberg,
and Hirschberg}]{levitan_identifying_2016}
Sarah~Ita Levitan, Yocheved Levitan, Guozhen An, Michelle Levine, Rivka
Levitan, Andrew Rosenberg, and Julia Hirschberg. 2016.
\newblock \href {https://doi.org/10.18653/v1/W16-0806} {Identifying individual
differences in gender, ethnicity, and personality from dialogue for deception
detection}.
\newblock In \emph{Proceedings of the Second Workshop on Computational
Approaches to Deception Detection}, pages 40--44. Association for
Computational Linguistics.
\bibitem[{Li et~al.(2022)Li, Weng, Song, Ma, Sun, and
Li}]{li_prompt-based_2022}
Bin Li, Yixuan Weng, Qiya Song, Fuyan Ma, Bin Sun, and Shutao Li. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.wassa-1.28} {Prompt-based
pre-trained model for personality and interpersonal reactivity prediction}.
\newblock In \emph{Proceedings of the 12th Workshop on Computational Approaches
to Subjectivity, Sentiment \& Social Media Analysis}, pages 265--270.
Association for Computational Linguistics.
\bibitem[{Lison et~al.(2021)Lison, Pil{\'a}n, S{\'a}nchez, Batet, and
{\O}vrelid}]{Lison2021AnonymisationMF}
Pierre Lison, I.~Pil{\'a}n, David S{\'a}nchez, Montserrat Batet, and Lilja
{\O}vrelid. 2021.
\newblock Anonymisation models for text data: State of the art, challenges and
future directions.
\newblock In \emph{ACL}.
\bibitem[{Liu et~al.(2016)Liu, Perez, and Nowson}]{liu_recurrent_2016}
Fei Liu, Julien Perez, and Scott Nowson. 2016.
\newblock \href {https://aclanthology.org/W16-4303} {A recurrent and
compositional model for personality trait recognition from short texts}.
\newblock In \emph{Proceedings of the Workshop on Computational Modeling of
People's Opinions, Personality, and Emotions in Social Media ({PEOPLES})},
pages 20--29. The {COLING} 2016 Organizing Committee.
\bibitem[{Liu et~al.(2017)Liu, Perez, and
Nowson}]{liu_language-independent_2017}
Fei Liu, Julien Perez, and Scott Nowson. 2017.
\newblock \href {https://aclanthology.org/E17-1071} {A language-independent and
compositional model for personality trait recognition from short texts}.
\newblock In \emph{Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 1, Long Papers},
pages 754--764. Association for Computational Linguistics.
\bibitem[{Liu et~al.(2019)Liu, He, Chen, and Gao}]{Liu2019MultiTaskDN}
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019.
\newblock Multi-task deep neural networks for natural language understanding.
\newblock In \emph{ACL}.
\bibitem[{Lloyd(2012)}]{lloyd2012myers}
John~B Lloyd. 2012.
\newblock The myers-briggs type indicator{\textregistered} and mainstream
psychology: analysis and evaluation of an unresolved hostility.
\newblock \emph{Journal of Beliefs \& Values}, 33(1):23--34.
\bibitem[{Luyckx and Daelemans(2008)}]{luyckx_personae_2008}
Kim Luyckx and Walter Daelemans. 2008.
\newblock \href
{http://www.lrec-conf.org/proceedings/lrec2008/pdf/759_paper.pdf} {Personae:
a corpus for author and personality prediction from text}.
\newblock In \emph{Proceedings of the Sixth International Conference on
Language Resources and Evaluation ({LREC}'08)}. European Language Resources
Association ({ELRA}).
\bibitem[{Lynn et~al.(2020)Lynn, Balasubramanian, and
Schwartz}]{lynn_hierarchical_2020}
Veronica Lynn, Niranjan Balasubramanian, and H.~Andrew Schwartz. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.472} {Hierarchical
modeling for user personality prediction: The role of message-level
attention}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 5306--5316. Association for
Computational Linguistics.
\bibitem[{Mairesse and Walker(2006)}]{mairesse_automatic_2006}
François Mairesse and Marilyn Walker. 2006.
\newblock \href {https://aclanthology.org/N06-2022} {Automatic recognition of
personality in conversation}.
\newblock In \emph{Proceedings of the Human Language Technology Conference of
the {NAACL}, Companion Volume: Short Papers}, pages 85--88. Association for
Computational Linguistics.
\bibitem[{Mairesse and Walker(2007)}]{mairesse_personage_2007}
François Mairesse and Marilyn Walker. 2007.
\newblock \href {https://aclanthology.org/P07-1063} {{PERSONAGE}: Personality
generation for dialogue}.
\newblock In \emph{Proceedings of the 45th Annual Meeting of the Association of
Computational Linguistics}, pages 496--503. Association for Computational
Linguistics.
\bibitem[{Mairesse and Walker(2008)}]{mairesse_trainable_2008}
François Mairesse and Marilyn Walker. 2008.
\newblock \href {https://aclanthology.org/P08-1020} {Trainable generation of
big-five personality styles through data-driven parameter estimation}.
\newblock In \emph{Proceedings of {ACL}-08: {HLT}}, pages 165--173. Association
for Computational Linguistics.
\bibitem[{Majumder et~al.(2017)Majumder, Poria, Gelbukh, and
Cambria}]{Majumder2017DeepLD}
Navonil Majumder, Soujanya Poria, Alexander Gelbukh, and E.~Cambria. 2017.
\newblock Deep learning-based document modeling for personality detection from
text.
\newblock \emph{IEEE Intelligent Systems}, 32:74--79.
\bibitem[{Makatchev and Simmons(2011)}]{makatchev_perception_2011}
Maxim Makatchev and Reid Simmons. 2011.
\newblock \href {https://aclanthology.org/W11-2032} {Perception of personality
and naturalness through dialogues by native speakers of american english and
arabic}.
\newblock In \emph{Proceedings of the {SIGDIAL} 2011 Conference}, pages
286--293. Association for Computational Linguistics.
\bibitem[{McCrae and Costa(1989)}]{mccrae1989neo}
RR~McCrae and PT~Costa. 1989.
\newblock The neo-pi/neo-ffi manual supplement.
\newblock \emph{Odessa, FL: Psychological Assessment Resources.}
\bibitem[{Mehrabi et~al.(2021)Mehrabi, Morstatter, Saxena, Lerman, and
Galstyan}]{mehrabi_survey_2021}
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram
Galstyan. 2021.
\newblock \href {https://doi.org/10.1145/3457607} {A survey on bias and
fairness in machine learning}.
\newblock \emph{{ACM} Computing Surveys}, 54(6):1--35.
\bibitem[{Mershon and Gorsuch(1988)}]{mershon1988number}
Bryan Mershon and Richard~L Gorsuch. 1988.
\newblock Number of factors in the personality sphere: Does increase in factors
increase predictability of real-life criteria?
\newblock \emph{Journal of Personality and Social Psychology}, 55(4):675.
\bibitem[{Mitchell(2017)}]{kaggleData}
J.~Mitchell. 2017.
\newblock (mbti) myers-briggs personality type dataset.
\newblock Data retrieved from Kaggle,
\url{https://www.kaggle.com/datasets/datasnaek/mbti-type}.
\bibitem[{Mushtaq and Kumar(2023)}]{mushtaq2022text}
Sumiya Mushtaq and Neerendra Kumar. 2023.
\newblock Text-based automatic personality recognition: Recent developments.
\newblock In \emph{Proceedings of Third International Conference on Computing,
Communications, and Cyber-Security: IC4S 2021}, pages 537--549. Springer.
\bibitem[{Norman~Adams(2022)}]{norman2022scraping}
Nicholas Norman~Adams. 2022.
\newblock ‘scraping’reddit posts for academic research? addressing some
blurred lines of consent in growing internet-based research trend during the
time of covid-19.
\newblock \emph{International journal of social research methodology}, pages
1--16.
\bibitem[{Nowack(1996)}]{nowack1996myers}
K~Nowack. 1996.
\newblock Is the myers briggs type indicator the right tool to use? performance
in practice.
\newblock \emph{American Society of Training and Development, Fall}, 6.
\bibitem[{Oberlander and Nowson(2006)}]{oberlander2006whose}
Jon Oberlander and Scott Nowson. 2006.
\newblock Whose thumb is it anyway? classifying author personality from weblog
text.
\newblock In \emph{Proceedings of the COLING/ACL 2006 Main Conference Poster
Sessions}, pages 627--634.
\bibitem[{Oberski(2016)}]{oberski2016mixture}
Daniel Oberski. 2016.
\newblock Mixture models: Latent profile and latent class analysis.
\newblock In \emph{Modern statistical methods for HCI}, pages 275--287.
Springer.
\bibitem[{Paunonen and Ashton(2001)}]{paunonen2001big}
Sampo~V Paunonen and Michael~C Ashton. 2001.
\newblock Big five factors and facets and the prediction of behavior.
\newblock \emph{Journal of personality and social psychology}, 81(3):524.
\bibitem[{Pennebaker and King(1999)}]{pennebaker1999linguistic}
James~W Pennebaker and Laura~A King. 1999.
\newblock Linguistic styles: language use as an individual difference.
\newblock \emph{Journal of personality and social psychology}, 77(6):1296.
\bibitem[{Phan and Rauthmann(2021)}]{phan2021personality}
Le~Vy Phan and John~F Rauthmann. 2021.
\newblock Personality computing: New frontiers in personality assessment.
\newblock \emph{Social and Personality Psychology Compass}, 15(7):e12624.
\bibitem[{Pittenger(1993)}]{pittenger1993measuring}
David~J Pittenger. 1993.
\newblock Measuring the mbti… and coming up short.
\newblock \emph{Journal of Career Planning and Employment}, 54(1):48--52.
\bibitem[{Pizzolli and Strapparava(2019)}]{pizzolli_personality_2019}
Daniele Pizzolli and Carlo Strapparava. 2019.
\newblock \href {https://doi.org/10.18653/v1/W19-3411} {Personality traits
recognition in literary texts}.
\newblock In \emph{Proceedings of the Second Workshop on Storytelling}, pages
107--111. Association for Computational Linguistics.
\bibitem[{Plank and Hovy(2015)}]{plank-hovy-2015-personality}
Barbara Plank and Dirk Hovy. 2015.
\newblock \href {https://doi.org/10.18653/v1/W15-2913} {Personality traits on
{T}witter{---}or{---}{H}ow to get 1,500 personality tests in a week}.
\newblock In \emph{Proceedings of the 6th Workshop on Computational Approaches
to Subjectivity, Sentiment and Social Media Analysis}, pages 92--98, Lisboa,
Portugal. Association for Computational Linguistics.
\bibitem[{Ponce-L{\'o}pez et~al.(2016)Ponce-L{\'o}pez, Chen, Oliu, Corneanu,
Clap{\'e}s, Guyon, Bar{\'o}, Escalante, and Escalera}]{ponce2016chalearn}
V{\'\i}ctor Ponce-L{\'o}pez, Baiyu Chen, Marc Oliu, Ciprian Corneanu, Albert
Clap{\'e}s, Isabelle Guyon, Xavier Bar{\'o}, Hugo~Jair Escalante, and Sergio
Escalera. 2016.
\newblock Chalearn lap 2016: First round challenge on first impressions-dataset
and results.
\newblock In \emph{European conference on computer vision}, pages 400--418.
Springer.
\bibitem[{Preoţiuc-Pietro et~al.(2015)Preoţiuc-Pietro, Eichstaedt, Park, Sap,
Smith, Tobolsky, Schwartz, and Ungar}]{preotiuc-pietro_role_2015}
Daniel Preoţiuc-Pietro, Johannes Eichstaedt, Gregory Park, Maarten Sap, Laura
Smith, Victoria Tobolsky, H.~Andrew Schwartz, and Lyle Ungar. 2015.
\newblock \href {https://doi.org/10.3115/v1/W15-1203} {The role of personality,
age, and gender in tweeting about mental illness}.
\newblock In \emph{Proceedings of the 2nd Workshop on Computational Linguistics
and Clinical Psychology: From Linguistic Signal to Clinical Reality}, pages
21--30. Association for Computational Linguistics.
\bibitem[{Ramos et~al.(2018)Ramos, Neto, Silva, Monteiro, Paraboni, and
Dias}]{ramos_building_2018}
Ricelli Ramos, Georges Neto, Barbara Silva, Danielle Monteiro, Ivandré
Paraboni, and Rafael Dias. 2018.
\newblock \href {https://aclanthology.org/L18-1183} {Building a corpus for
personality-dependent natural language understanding and generation}.
\newblock In \emph{Proceedings of the Eleventh International Conference on
Language Resources and Evaluation ({LREC} 2018)}. European Language Resources
Association ({ELRA}).
\bibitem[{Rangel et~al.(2015)Rangel, Rosso, Potthast, Stein, and
Daelemans}]{rangel2015overview}
Francisco Rangel, Paolo Rosso, Martin Potthast, Benno Stein, and Walter
Daelemans. 2015.
\newblock Overview of the 3rd author profiling task at pan 2015.
\newblock In \emph{CLEF}, page 2015. sn.
\bibitem[{Resnik et~al.(2013)Resnik, Garron, and Resnik}]{resnik_using_2013}
Philip Resnik, Anderson Garron, and Rebecca Resnik. 2013.
\newblock \href {https://aclanthology.org/D13-1133} {Using topic modeling to
improve prediction of neuroticism and depression in college students}.
\newblock In \emph{Proceedings of the 2013 Conference on Empirical Methods in
Natural Language Processing}, pages 1348--1353. Association for Computational
Linguistics.
\bibitem[{Ribeiro et~al.(2020)Ribeiro, Wu, Guestrin, and
Singh}]{Ribeiro2020BeyondAB}
Marco~Tulio Ribeiro, Tongshuang~Sherry Wu, Carlos Guestrin, and Sameer Singh.
2020.
\newblock Beyond accuracy: Behavioral testing of nlp models with checklist.
\newblock In \emph{ACL}.
\bibitem[{Robins~Wahlin and Byrne(2011)}]{robins2011personality}
Tarja-Brita Robins~Wahlin and Gerard~J Byrne. 2011.
\newblock Personality changes in alzheimer's disease: a systematic review.
\newblock \emph{International journal of geriatric psychiatry},
26(10):1019--1029.
\bibitem[{Roshchina et~al.(2012)Roshchina, Cardiff, and
Rosso}]{roshchina_evaluating_2012}
Alexandra Roshchina, John Cardiff, and Paolo Rosso. 2012.
\newblock \href
{http://www.lrec-conf.org/proceedings/lrec2012/pdf/1080_Paper.pdf}
{Evaluating the similarity estimator component of the {TWIN}
personality-based recommender system}.
\newblock In \emph{Proceedings of the Eighth International Conference on
Language Resources and Evaluation ({LREC}'12)}, pages 4098--4102. European
Language Resources Association ({ELRA}).
\bibitem[{Saha et~al.(2022)Saha, Das, and Srihari}]{saha_stylistic_2022}
Sougata Saha, Souvik Das, and Rohini Srihari. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.nlp4convai-1.16} {Stylistic
response generation by controlling personality traits and intent}.
\newblock In \emph{Proceedings of the 4th Workshop on {NLP} for Conversational
{AI}}, pages 197--211. Association for Computational Linguistics.
\bibitem[{Santangelo et~al.(2017)Santangelo, Piscopo, Barone, and
Vitale}]{santangelo2017personality}
Gabriella Santangelo, Fausta Piscopo, Paolo Barone, and Carmine Vitale. 2017.
\newblock Personality in parkinson's disease: Clinical, behavioural and
cognitive correlates.
\newblock \emph{Journal of the Neurological Sciences}, 374:17--25.
\bibitem[{Saris and Gallhofer(2014)}]{saris2014design}
Willem~E Saris and Irmtraud~N Gallhofer. 2014.
\newblock \emph{Design, evaluation, and analysis of questionnaires for survey
research}.
\newblock John Wiley \& Sons.
\bibitem[{Siddique et~al.(2017)Siddique, Kampman, Yang, Dey, and
Fung}]{siddique_zara_2017}
Farhad~Bin Siddique, Onno Kampman, Yang Yang, Anik Dey, and Pascale Fung. 2017.
\newblock \href {https://aclanthology.org/P17-4021} {Zara returns: Improved
personality induction and adaptation by an empathetic virtual agent}.
\newblock In \emph{Proceedings of {ACL} 2017, System Demonstrations}, pages
121--126. Association for Computational Linguistics.
\bibitem[{Sinha et~al.(2015)Sinha, Dey, Mitra, and Basu}]{sinha_mining_2015}
Priyanka Sinha, Lipika Dey, Pabitra Mitra, and Anupam Basu. 2015.
\newblock \href {https://doi.org/10.18653/v1/W15-2920} {Mining {HEXACO}
personality traits from enterprise social media}.
\newblock In \emph{Proceedings of the 6th Workshop on Computational Approaches
to Subjectivity, Sentiment and Social Media Analysis}, pages 140--147.
Association for Computational Linguistics.
\bibitem[{Soto and John(2017)}]{soto2017next}
Christopher~J Soto and Oliver~P John. 2017.
\newblock The next big five inventory (bfi-2): Developing and assessing a
hierarchical model with 15 facets to enhance bandwidth, fidelity, and
predictive power.
\newblock \emph{Journal of personality and social psychology}, 113(1):117.
\bibitem[{Stachl et~al.(2020)Stachl, Pargent, Hilbert, Harari, Schoedel, Vaid,
Gosling, and B{\"u}hner}]{stachl2020personality}
Clemens Stachl, Florian Pargent, Sven Hilbert, Gabriella~M Harari, Ramona
Schoedel, Sumer Vaid, Samuel~D Gosling, and Markus B{\"u}hner. 2020.
\newblock Personality research and assessment in the era of machine learning.
\newblock \emph{European Journal of Personality}, 34(5):613--631.
\bibitem[{Stajner and Yenikent(2020)}]{stajner_survey_2020}
Sanja Stajner and Seren Yenikent. 2020.
\newblock A survey of automatic personality detection from texts.
\newblock In \emph{Proceedings of the 28th International Conference on
Computational Linguistics}, pages 6284--6295. International Committee on
Computational Linguistics.
\bibitem[{Stajner and Yenikent(2021)}]{stajner_why_2021}
Sanja Stajner and Seren Yenikent. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.eacl-main.312} {Why is {MBTI}
personality detection from texts a difficult task?}
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pages
3580--3589. Association for Computational Linguistics.
\bibitem[{Suman et~al.(2020)Suman, Gupta, Saha, and
Bhattacharyya}]{suman_multi-modal_2020}
Chanchal Suman, Aditya Gupta, Sriparna Saha, and Pushpak Bhattacharyya. 2020.
\newblock \href {https://aclanthology.org/2020.icon-main.42} {A multi-modal
personality prediction system}.
\newblock In \emph{Proceedings of the 17th International Conference on Natural
Language Processing ({ICON})}, pages 317--322. {NLP} Association of India
({NLPAI}).
\bibitem[{Tighe and Cheng(2018)}]{tighe_modeling_2018}
Edward Tighe and Charibeth Cheng. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-1115} {Modeling personality
traits of filipino twitter users}.
\newblock In \emph{Proceedings of the Second Workshop on Computational Modeling
of People's Opinions, Personality, and Emotions in Social Media}, pages
112--122. Association for Computational Linguistics.
\bibitem[{Tolins et~al.(2016{\natexlab{a}})Tolins, Liu, Neff, Walker, and
Fox~Tree}]{tolins_verbal_2016}
Jackson Tolins, Kris Liu, Michael Neff, Marilyn Walker, and Jean Fox~Tree.
2016{\natexlab{a}}.
\newblock \href {https://aclanthology.org/L16-1552} {A verbal and gestural
corpus of story retellings to an expressive embodied virtual character}.
\newblock In \emph{Proceedings of the Tenth International Conference on
Language Resources and Evaluation ({LREC}'16)}, pages 3461--3468. European
Language Resources Association ({ELRA}).
\bibitem[{Tolins et~al.(2016{\natexlab{b}})Tolins, Liu, Wang, Fox~Tree, Walker,
and Neff}]{tolins_multimodal_2016}
Jackson Tolins, Kris Liu, Yingying Wang, Jean~E. Fox~Tree, Marilyn Walker, and
Michael Neff. 2016{\natexlab{b}}.
\newblock \href {https://aclanthology.org/L16-1553} {A multimodal
motion-captured corpus of matched and mismatched extravert-introvert
conversational pairs}.
\newblock In \emph{Proceedings of the Tenth International Conference on
Language Resources and Evaluation ({LREC}'16)}, pages 3469--3476. European
Language Resources Association ({ELRA}).
\bibitem[{Trochim et~al.(2015)Trochim, Donnelly, and
Arora}]{Trochim2015ResearchMT}
William M.~K. Trochim, James~P. Donnelly, and Kanika Arora. 2015.
\newblock \emph{Research Methods: The Essential Knowledge Base}.
\newblock Cengage Learning, Boston.
\bibitem[{Tupes and Christal(1992)}]{tupes1992recurrent}
Ernest~C Tupes and Raymond~E Christal. 1992.
\newblock Recurrent personality factors based on trait ratings.
\newblock \emph{Journal of personality}, 60(2):225--251.
\bibitem[{Uban et~al.(2021)Uban, Chulvi, and Rosso}]{uban_understanding_2021}
Ana~Sabina Uban, Berta Chulvi, and Paolo Rosso. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.clpsych-1.24} {Understanding
patterns of anorexia manifestations in social media data with deep learning}.
\newblock In \emph{Proceedings of the Seventh Workshop on Computational
Linguistics and Clinical Psychology: Improving Access}, pages 224--236.
Association for Computational Linguistics.
\bibitem[{Uma et~al.(2021)Uma, Fornaciari, Hovy, Paun, Plank, and
Poesio}]{uma2021learning}
Alexandra~N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and
Massimo Poesio. 2021.
\newblock Learning from disagreement: A survey.
\newblock \emph{Journal of Artificial Intelligence Research}, 72:1385--1470.
\bibitem[{V~Ganesan et~al.(2021)V~Ganesan, Matero, Ravula, Vu, and
Schwartz}]{v_ganesan_empirical_2021}
Adithya V~Ganesan, Matthew Matero, Aravind~Reddy Ravula, Huy Vu, and H.~Andrew
Schwartz. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.357} {Empirical
evaluation of pre-trained transformers for human-level {NLP}: The role of
sample size and dimensionality}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 4515--4532. Association for Computational Linguistics.
\bibitem[{{van der Linden} et~al.(2010){van der Linden}, {te Nijenhuis}, and
Bakker}]{VANDERLINDEN2010315}
Dimitri {van der Linden}, Jan {te Nijenhuis}, and Arnold~B. Bakker. 2010.
\newblock The general factor of personality: A meta-analysis of big five
intercorrelations and a criterion-related validity study.
\newblock \emph{Journal of Research in Personality}, 44(3):315--327.
\bibitem[{Verhoeven and Daelemans(2014)}]{verhoeven_clips_2014}
Ben Verhoeven and Walter Daelemans. 2014.
\newblock \href {http://www.lrec-conf.org/proceedings/lrec2014/pdf/1_Paper.pdf}
{{CLiPS} stylometry investigation ({CSI}) corpus: A dutch corpus for the
detection of age, gender, personality, sentiment and deception in text}.
\newblock In \emph{Proceedings of the Ninth International Conference on
Language Resources and Evaluation ({LREC}'14)}, pages 3081--3085. European
Language Resources Association ({ELRA}).
\bibitem[{Verhoeven et~al.(2016)Verhoeven, Daelemans, and
Plank}]{verhoeven_twisty_2016}
Ben Verhoeven, Walter Daelemans, and Barbara Plank. 2016.
\newblock \href {https://aclanthology.org/L16-1258} {{TwiSty}: A multilingual
twitter stylometry corpus for gender and personality profiling}.
\newblock In \emph{Proceedings of the Tenth International Conference on
Language Resources and Evaluation ({LREC}'16)}, pages 1632--1637. European
Language Resources Association ({ELRA}).
\bibitem[{Vettigli and Sorgente(2021)}]{vettigli_empna_2021}
Giuseppe Vettigli and Antonio Sorgente. 2021.
\newblock \href {https://aclanthology.org/2021.wassa-1.28} {{EmpNa} at {WASSA}
2021: A lightweight model for the prediction of empathy, distress and
emotions from reactions to news stories}.
\newblock In \emph{Proceedings of the Eleventh Workshop on Computational
Approaches to Subjectivity, Sentiment and Social Media Analysis}, pages
264--268. Association for Computational Linguistics.
\bibitem[{Volkova et~al.(2015)Volkova, Bachrach, Armstrong, and
Sharma}]{Volkova2015InferringLU}
Svitlana Volkova, Yoram Bachrach, Michael Armstrong, and Vijay Sharma. 2015.
\newblock Inferring latent user properties from texts published in social
media.
\newblock In \emph{AAAI Conference on Artificial Intelligence}.
\bibitem[{Vu et~al.(2020)Vu, Abdurahman, Bhatia, and
Ungar}]{vu_predicting_2020}
Huy Vu, Suhaib Abdurahman, Sudeep Bhatia, and Lyle Ungar. 2020.
\newblock Predicting {Responses} to {Psychological} {Questionnaires} from
{Participants}' {Social} {Media} {Posts} and {Question} {Text} {Embeddings}.
\newblock In \emph{Findings of the {Association} for {Computational}
{Linguistics}: {EMNLP} 2020}, pages 1512--1524, Online. Association for
Computational Linguistics.
\bibitem[{Vu et~al.(2018)Vu, Flekova, Jiang, and
Gurevych}]{vu_lexical-semantic_2018}
Xuan-Son Vu, Lucie Flekova, Lili Jiang, and Iryna Gurevych. 2018.
\newblock \href {https://aclanthology.org/2018.gwc-1.20} {Lexical-semantic
resources: yet powerful resources for automatic personality classification}.
\newblock In \emph{Proceedings of the 9th Global Wordnet Conference}, pages
172--181. Global Wordnet Association.
\bibitem[{Wang et~al.(2022)Wang, Yang, and Wang}]{Wang2022IdentifyingAM}
Tianlu Wang, Diyi Yang, and Xuezhi Wang. 2022.
\newblock Identifying and mitigating spurious correlations for improving
robustness in nlp models.
\newblock In \emph{NAACL-HLT}.
\bibitem[{Wen et~al.(2021)Wen, Cao, Yang, Liu, and
Shen}]{wen_automatically_2021}
Zhiyuan Wen, Jiannong Cao, Ruosong Yang, Shuaiqi Liu, and Jiaxing Shen. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.findings-acl.444}
{Automatically select emotion for response via personality-affected emotion
transition}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
{ACL}-{IJCNLP} 2021}, pages 5010--5020. Association for Computational
Linguistics.
\bibitem[{Xioufis et~al.(2016)Xioufis, Tsoumakas, Groves, and
Vlahavas}]{Xioufis2016MultitargetRV}
Eleftherios~Spyromitros Xioufis, Grigorios Tsoumakas, William Groves, and
Ioannis~P. Vlahavas. 2016.
\newblock Multi-target regression via input space expansion: treating targets
as inputs.
\newblock \emph{Machine Learning}, 104:55--98.
\bibitem[{Yamada et~al.(2019)Yamada, Sasano, and
Takeda}]{yamada_incorporating_2019}
Kosuke Yamada, Ryohei Sasano, and Koichi Takeda. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-2024} {Incorporating textual
information on user behavior for personality prediction}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics: Student Research Workshop}, pages 177--182.
Association for Computational Linguistics.
\bibitem[{Yang et~al.(2015)Yang, Pan, Mahmud, Yang, and
Srinivasan}]{yang_using_2015}
Chao Yang, Shimei Pan, Jalal Mahmud, Huahai Yang, and Padmini Srinivasan. 2015.
\newblock \href {https://doi.org/10.18653/v1/D15-1009} {Using personal traits
for brand preference prediction}.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing}, pages 86--96. Association for Computational
Linguistics.
\bibitem[{Yang et~al.(2021{\natexlab{a}})Yang, Yang, Quan, and
Su}]{yang_learning_2021}
Feifan Yang, Tao Yang, Xiaojun Quan, and Qinliang Su. 2021{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/2021.findings-emnlp.98} {Learning
to answer psychological questionnaire for personality detection}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
{EMNLP} 2021}, pages 1131--1142. Association for Computational Linguistics.
\bibitem[{Yang et~al.(2021{\natexlab{b}})Yang, Yang, Ouyang, and
Quan}]{yang_psycholinguistic_2021}
Tao Yang, Feifan Yang, Haolan Ouyang, and Xiaojun Quan. 2021{\natexlab{b}}.
\newblock \href {https://doi.org/10.18653/v1/2021.acl-long.326}
{Psycholinguistic tripartite graph network for personality detection}.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 4229--4239.
Association for Computational Linguistics.
\bibitem[{Zamani et~al.(2018)Zamani, Buffone, and
Schwartz}]{zamani_predicting_2018}
Mohammadzaman Zamani, Anneke Buffone, and H.~Andrew Schwartz. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-0619} {Predicting human
trustfulness from facebook language}.
\newblock In \emph{Proceedings of the Fifth Workshop on Computational
Linguistics and Clinical Psychology: From Keyboard to Clinic}, pages
174--181. Association for Computational Linguistics.
\bibitem[{Zhou et~al.(2020)Zhou, Nie, Tan, and Bansal}]{zhou2020TheCO}
Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. 2020.
\newblock The curse of performance instability in analysis datasets:
Consequences, source, and suggestions.
\newblock In \emph{EMNLP}.
\end{thebibliography}