Decay No More: A Persistent Twitter Dataset for Learning Social Meaning
Extracted source text ยท arXiv 2204.04611
\def\year{2021}\relax
%File: formatting-instructions-latex-2021.tex
%release 2021.2
\documentclass[letterpaper]{article} % DO NOT CHANGE THIS
\usepackage{aaai21} % DO NOT CHANGE THIS
\usepackage{times} % DO NOT CHANGE THIS
\usepackage{helvet} % DO NOT CHANGE THIS
\usepackage{courier} % DO NOT CHANGE THIS
\usepackage[hyphens]{url} % DO NOT CHANGE THIS
\usepackage{graphicx} % DO NOT CHANGE THIS
\urlstyle{rm} % DO NOT CHANGE THIS
\def\UrlFont{\rm} % DO NOT CHANGE THIS
\usepackage{natbib} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
\usepackage{caption} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
\frenchspacing % DO NOT CHANGE THIS
\setlength{\pdfpagewidth}{8.5in} % DO NOT CHANGE THIS
\setlength{\pdfpageheight}{11in} % DO NOT CHANGE THIS
%\nocopyright
%PDF Info Is REQUIRED.
% For /Author, add all authors within the parentheses, separated by commas. No accents or commands.
% For /Title, add Title in Mixed Case. No accents or commands. Retain the parentheses.
\pdfinfo{
/Title (AAAI Press Formatting Instructions for Authors Using LaTeX -- A Guide)
/Author (AAAI Press Staff, Pater Patel Schneider, Sunil Issar, J. Scott Penberthy, George Ferguson, Hans Guesgen, Francisco Cruz, Marc Pujol-Gonzalez)
/TemplateVersion (2021.2)
} %Leave this
% /Title ()
% Put your actual complete title (no codes, scripts, shortcuts, or LaTeX commands) within the parentheses in mixed case
% Leave the space between \Title and the beginning parenthesis alone
% /Author ()
% Put your actual complete list of authors (no codes, scripts, shortcuts, or LaTeX commands) within the parentheses in mixed case.
% Each author should be only by a comma. If the name contains accents, remove them. If there are any LaTeX commands,
% remove them.
\newcommand\BibTeX{B\textsc{ib}\TeX}
% #++++++++++++++++++++++++++++++++++++++
\usepackage{adjustbox}
\usepackage{multirow}
\renewcommand{\UrlFont}{\ttfamily\small}
\usepackage{microtype}
\usepackage{xspace}
\usepackage{mathtools}
\usepackage{amssymb}
\usepackage{rotating}
% \usepackage{arabtex}
\usepackage{footnote}
\usepackage{color, colortbl}
\usepackage{xstring}
\usepackage{comment}
\usepackage{float}
\restylefloat{table}
\usepackage{array,booktabs,makecell}
\usepackage{graphicx}
\usepackage{appendix}
\usepackage[normalem]{ulem}
% \useunder{\uline}{\ul}{}
% \usepackage[show]{boxnotes}
\usepackage{tcolorbox}
\usepackage{enumitem}
\usepackage{scalerel,xparse}
\usepackage{siunitx}
\renewcommand{\UrlFont}{\ttfamily\small}
\newcolumntype{H}{>{\setbox0=\hbox\bgroup}c<{\egroup}@{}}
% ##################
\usepackage{arydshln}
\PassOptionsToPackage{hyphens}{url}\usepackage{hyperref}
\newcommand{\moa}[1]{\noindent{\textcolor{blue}{\{{$_{mtz}$} \em #1\}}}}
\newcommand{\chiyu}[1]{\noindent{\textcolor{orange}{\{{$_{chiyu}$} \em #1\}}}}
\newcommand{\mam}[1]{\noindent{\textcolor{red}{\{{$_{mam}$} \em #1\}}}}
\setcounter{secnumdepth}{3}
\begin{document}
\title{Decay No More:\\ A Persistent Twitter Dataset for Learning Social Meaning}
\author{Chiyu Zhang ~~~~Muhammad Abdul-Mageed ~~~ El Moatez Billah Nagoudi \\
Deep Learning \& Natural Language Processing Group \\The University of British Columbia \\
\tt chiyuzh@mail.ubc.ca, \tt \{muhammad.mageed,moatez.nagoudi\}@ubc.ca}
\maketitle
\begin{abstract}
With the proliferation of social media, many studies resort to social media to construct datasets for developing social meaning understanding systems. For the popular case of Twitter, most researchers distribute tweet IDs without the actual text contents due to the data distribution policy of the platform. One issue is that the posts become increasingly inaccessible over time, which leads to unfair comparisons and a temporal bias in social media research. To alleviate this challenge of data decay, we leverage a paraphrase model to propose a new \textit{persistent} English Twitter dataset for social meaning (PTSM). PTSM consists of $17$ social meaning datasets in $10$ categories of tasks. We experiment with two SOTA pre-trained language models and show that our PTSM can substitute the actual tweets with paraphrases with \textit{marginal} performance loss.
\end{abstract}
\input{intro}
\input{related_work}
\input{experiment}
\input{results}
\input{conclusion}
\newpage
% \bibliographystyle{aaai}
\bibliography{literature}
\end{document}
\section{Introduction}
\textit{Social meaning} is substantiated in socio-pragmatics, and refers to intended meaning in real-word communication~\cite{thomas2014meaning} and how utterances should be interpreted within the social context in which they are produced. Aspects of social meaning include emotion recognition~\cite{mohammad-2018-semeval}, irony detection~\cite{van-hee2018semeval}, sarcasm detection~\cite{riloff2013sarcasm, bamman2015contextualized}, hate speech identification~\cite{waseem-2016-hateful}, and stance identification~\cite{mohammad-2016-semeval}. A successful social meaning comprehension system can ameliorate a wide range of NLP applications. For example, a dialogue system with knowledge of social meaning can provide more engaging reactions when it realizes human emotions.
%~~~~~~~~~~~~~~~~~~~~~~~~
\begin{figure}[ht]
\begin{centering}
\includegraphics[width=0.85\linewidth]{tables/sample.png}
\caption{Paraphrase example from Emo\textsubscript{Moham}.}
\label{fig:para}
\end{centering}
\end{figure}
%~~~~~~~~~~~~~~~~~~~~~~~~
With the proliferation of social media, billions of users share content in various forms. Social media platforms, such as Twitter, allow users to express their opinions, discuss topics, and connect with friends, among other practices~\cite{farzindar2015natural}. As a result, social media offers abundant resources for social meaning understanding. Over the years, researchers have developed a number of labelled datasets to train (semi-)supervised machine learning models. According to the data redistribution policy of a social media platform such as Twitter, a post's actual content cannot be shared with third parties. Hence, many studies only distribute the IDs of posts (e.g., tweets). The challenge with this set up, however, substantial social media posts become inaccessible over time due to deletion, protection, etc.~\citet{zubiaga2018longitudinal}. We empirically characterize this issue of data inaccessibility by attempting to re-collect the tweet contents of six social meaning datasets. As Table~\ref{tab:crawl} shows, we could only acquire, on average, $73\%$ of the tweets. This data decay leads to a serious issue concerning the lack of replicability in social media studies. The issue of data inaccessibility also introduce a temporal bias (as older data sets become significantly unavailable) and leads to unfair comparisons. In other words, it is difficult to compare models trained on differently sized datasets. In this work, we propose a \textit{persistent} Twitter dataset for social meaning (PTSM) to alleviate these issues. We use a state-of-the-art (SOTA) Transformer-based pre-trained model, i.e., T5~\cite{raffel2020exploring}, as backbone to train a paraphrasing model with diverse parallel data. We then use the trained model to produce sentential paraphrases of training samples. We then benchmark our SMPM with two SOTA pre-trained language models (PLM), RoBERTa~\cite{liu2019roberta} and BERTweet~\cite{nguyen-etal-2020-bertweet}. Our experiment results show that we can replace the actual tweets with their paraphrases without sacrificing performance: On average, our paraphrase-based models are only $\sim 1.70$ $F_1$ below models trained with original data across the $17$ datasets.
To summarize, we make the following \textbf{contributions}:
\textbf{(1)} We introduce PTSM, a persistent Twitter dataset for social meaning comprising $17$ different datasets, whose accessibility is enhanced via paraphrasing. To the best of our knowledge, this is the first work to employ SOTA paraphrase methods to tackle occasional data decay for social media research. \textbf{(2)} We develop a Transformer-based model for paraphrasing social media data. \textbf{(3)} We benchmark our PTSM with two SOTA PLMs and demonstrate the promise of training tweet classifiers with paraphrases only.
\input{tables/data_inaccess}
% Furthermore, social meaning tasks lack a standard evaluation benchmark to compare their model performances. It is difficult and time-consuming for researchers to track the state-of-the-art results of social meaning tasks. Hence, I advise that more social media research take replicability into account.
%Since we face challenges in acquiring the different datsets, with only $73\%$ of the data being accessible, we paraphrase the benchmark and make it available to the community to facilitate future research. We refer to our new benchmark as the \textit{Social Meaning Paraphrase Benchmark (SMPB)} (Section~\ref{sec:smpb}).
\begin{table}[h]
\centering
\tiny
\begin{tabular}{@{}lcrrc@{}}
\toprule
\multicolumn{1}{c}{\textbf{}} & \textbf{Prod. year} & \multicolumn{1}{c}{\textbf{Orig.}} & \multicolumn{1}{c}{\textbf{Retrieved}} & \multicolumn{1}{c}{\textbf{Decay Rate}} \\ \midrule
\citet{riloff2013sarcasm} & 2013 & $3.0$K & $1.8$K & 0.41 \\
\citet{ptavcek2014sarcasm} & 2014 & $100.0$K & $89.3$K & 0.11 \\
\citet{rajadesingan2015sarcasm} & 2015 & $91.0$K & $51.6$K & 0.43 \\
\citet{bamman2015contextualized} & 2015 & $19.5$K & $14.8$K & 0.34 \\
\citet{waseem-2016-hateful} & 2016 & $16.9$K & $10.9$K & 0.36 \\
\citet{rosenthal-2017-semeval} & 2017 & $50.3$K & $48.2$K & 0.04 \\ \bottomrule
\end{tabular}
\caption{Issue of data inaccessibility. These six datasets were distributed by their authors via tweet IDs. \textbf{Orig.:} original size of each dataset. \textbf{Retrieved:} data we successfully collected via Twitter API in November, 2020. \textbf{Decay Rate:} percentage of inaccessible tweets. }\label{tab:crawl}
\end{table}
\vspace{-15pt}
\section{Related Work}
Paraphrasing aims at rewriting on rephrase a text while maintaining its original semantics. Most previous works introduce paraphrasing as a way to augment training data and hence alleviate data sparsity in machine learning models. To reduce the high degree of lexical variation,~\citet{petrovic2012using, li2016using} produce paraphrases of tweets by replacing words of original text with their synonyms based on WordNet and word embedding vector closeness.~\citet{beddiar2021data} introduce a back-translation method to augment training data for hate speech detection. Different to the previous works, we utilize paraphrases to develop a tweet classifier without any subsequent use of the original tweets while training our downstream models. Our main objective is to tackle the data decay issue in machine learning of social media. For that, we offer a unified paraphrase dataset for training social meaning models that is directly comparable to original training data for a host of tasks. Our work has affinity to work aiming at facilitating meaningful model comparisons such as the general language understanding evaluation (GLUE) benchmark~\cite{wang2019glue} and SuperGLUE~\cite{sarlin2020superglue}, but we focus on availing training data that enable model building in the first place.~\citet{barbieri-2020-tweeteval} introduce TweetEval, a benchmark for tweet classification evaluation, but they are not able to share more than $50$K tweets per dataset due to Twitter distribution policies. Different to them, we are able to provide unlimited numbers of paraphrases for future research.
\section{Persistent Dataset for Social Meaning}
\input{gold_dataset}
% \subsection{Paraphrasing Model and SMPB}
\input{t5_para}
\input{tables/para_data}
\subsection{Generating PTSM}
We apply the trained paraphrasing model on the Train split of each of our $17$ social meaning datasets, using top-$p$ sampling~\cite{holtzman2019curious} with $p=0.95$ to generate $10$ paraphrases for each gold sample. To remove any `paraphrases' that are just copies of the original tweet or near-duplicates, we use a simple tri-gram similarity method where we remove any sequences whose similarity with the original tweet is $> 0.95$. We also remove paraphrases of the same tweet that are $> 0.50$ similar to one another. To ensure that paraphrases are not very different from the original tweets, we also remove any sequences whose similarity with original tweets $=0$. This process results in a paraphrasing dataset \texttt{Para-Clean}. We present example paraphrases of one sample in Figure~\ref{fig:para}. We observe that the paraphrase model fails to generate emojis since emojis are out-of-vocabulary for the original T5 model, but the model can preserve the overall semantics of the original tweet.%\footnote{We provide more example paraphrases on \url{https://anonymous.4op}.}
To explore effect of the size of paraphrase data on the downstream tasks, we extract $1$, $2$, $4$, and $5$ paraphrases from \texttt{Para-Clean} for each Train gold sample in each of our $15$ tasks. We refer to the resulting datasets as \texttt{Para$1$}, \texttt{Para$2$}, \texttt{Para$4$}, and \texttt{Para$5$}. Table~\ref{tab:para_n_data} shows the distribution of the resulting paraphrase datasets.\footnote{Note that some samples may not have enough paraphrases to construct Para$2$, Para$4$, and Para$5$.}
\subsection{Social Meaning Datasets}\label{sec:smpb}
%\subsection{Social Meaning Datasets}\label{subsec:sm_datasets}
We collect $17$ datasets representing $10$ different social meaning tasks, as follows:~\footnote{To facilitate reference, we give each dataset a name.}
\input{tables/dataset}
% \begin{enumerate}[leftmargin=15pt]
\noindent\textbf{Crisis awareness.}~We use \texttt{Crisis\textsubscript{Oltea}} ~\cite{olteanu2014crisislex}, a corpus for identifying whether a tweet is related to a given disaster or not. %The dataset collects generated during six disasters, including hurricane, tornado, etc.
\noindent \textbf{Emotion.} We utilize \texttt{Emo\textsubscript{Moham}}, introduced by~\citet{mohammad-2018-semeval}, for emotion recognition. We use the version adapted in \citet{barbieri-2020-tweeteval}.
\noindent\textbf{Hateful and offensive language.} We use \texttt{Hate\textsubscript{Bas}}~\cite{basile-2019-semeval}, \texttt{Hate\textsubscript{Waseem}}~\cite{waseem-2016-hateful}, \texttt{Hate\textsubscript{David}}~\cite{davidson-2017-hateoffensive}, and \texttt{Offense\textsubscript{Zamp}}~\cite{zampieri-2019-predicting}.
\noindent \textbf{Humor.} We use the humor detection datasets \texttt{Humor\textsubscript{Potash}}~\cite{potash-2017-semeval} and \texttt{Humor\textsubscript{Meaney}}~\cite{meaney2021hahackathon}.
\noindent\textbf{Irony.} We utilize \texttt{Irony\textsubscript{Hee-A}} (irony detection) and \texttt{Irony\textsubscript{Hee-B}} (irony type identification) from \citet{van-hee2018semeval}.
% \begin{itemize}[leftmargin=10pt]
% \item \textbf{Irony\textsubscript{Hee-A}}~~ \citet{van-hee2018semeval} initially retrieve irony tweets using related hashtags (i.e., `\#irony', `\#sarcasm', and `not'). To enhance the quality of labeled data, they reassign the label based on human annotations.
% \item \textbf{Irony\textsubscript{Hee-B}}~~ \citet{van-hee2018semeval} provide a more fine-grained annotation on the samples of Irony\textsubscript{Hee-A}. Each sample is annotated with a type of irony: (1) \textit{verbal irony by means of a polarity contrast}, (2) \textit{other types of verbal irony}, (3) \textit{situational irony}, or (4) \textit{non-ironic}.
% \end{itemize}
\noindent \textbf{Sarcasm.} %The goal in sarcasm detection is to predict whether a social media post is \textit{sarcastic} or \textit{not sarcastic}.
We use four sarcasm datasets from \texttt{Sarc\textsubscript{Riloff}}~\cite{riloff2013sarcasm}, \texttt{Sarc\textsubscript{Ptacek}}~\cite{ptavcek2014sarcasm}, \texttt{Sarc\textsubscript{Rajad}}~\cite{rajadesingan2015sarcasm}, and \texttt{Sarc\textsubscript{Bam}}~\cite{bamman2015contextualized}.
% \begin{itemize}[leftmargin=10pt]
% \item \textbf{Sarcasm\textsubscript{Riloff}}~~ \citet{riloff2013sarcasm} create a manually annotated dataset for sarcasm detection. The sarcastic tweet are originally collected by sarcasm hashtags (`\#sarcasm' or `\#sarcastic').
% \item \textbf{Sarcasm\textsubscript{ptavcek}}~~ \citet{ptavcek2014sarcasm} introduce a distance supervised dataset for sarcasm detection. The `\#sarcasm' hashtag is used as the proxy to annotate a tweet is sarcastic or not.
% \item \textbf{Sarcasm\textsubscript{Rajad}}~~ \citet{rajadesingan2015sarcasm} provide a English sarcasm detection corpus. They retrieve the sarcastic by using hashtag `\#sarcasm' and `\#not' as the distant indicator. The non-sarcastic tweets are random tweets without any seed hashtags.
% \item \textbf{Sarcasm\textsubscript{Bam}}~~ \citet{bamman2015contextualized} also utilize distant supervision to collect English sarcastic tweets. The sarcastic tweet uses the hashtag `\#sarcasm' or `\#sarcastic' as its last term. To investigate the sarcasm expression in the social context, they only extract tweets that are responses to another tweet.
% \end{itemize}
\noindent\textbf{Sentiment.} We employ the three-way sentiment analysis dataset from \texttt{Senti\textsubscript{Rosen}}~\cite{rosenthal-2017-semeval} and a binary sentiment analysis dataset from \texttt{Senti\textsubscript{Thel}}~\cite{thelwall2012sentiment}.
\noindent\textbf{Stance.} We use \texttt{Stance\textsubscript{Moham}}, a stance detection dataset from \citet{mohammad-2016-semeval}. The task is to identify the position of a given tweet towards a target of interest.
% \end{enumerate}
% (1) crisis awareness, (2) emotion recognition, (3) hate speech and offensive language detection, (4) humor detection, (5) irony detection, (6) sarcasm detection, (7) sentiment analysis, and (8) stance detection. We will briefly introduce datasets for each task, then
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\subsection{Data Crawling}\label{subsec:crawling} \vspace{-3pt}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\textbf{Data Crawling and Preparation.} We use the Twitter API~\footnote{\url{https://developer.twitter.com/}} to crawl datasets which are available only in tweet ID form. %We note that we could not download all tweets since some tweets get deleted by users or become inaccessible for some other reason. Since some datasets are old (dating back to 2013), we are only able to retrieve $73\%$ of the tweets on average (i.e., across the different datasets). This inaccessibility of the data motivates us to paraphrase the datasets.
Before we paraphrase the data or use it in our various experiments, we normalize each tweet by replacing the user names and hyperlinks to the special tokens `USER' and `URL', respectively. This ensures our paraphrased dataset will never have any actual usernames or hyperlinks, thereby protecting user identity. For datasets collected based on hashtags by original authors, we also remove the seed hashtags from the original tweets. For datasets originally used in cross-validation, we acquire $80\%$ Train, $10\%$ Dev, and $10\%$ Test via random splits. For datasets that had training and test splits but not development data, we split off $10\%$ from training data into Dev. The splits of each dataset are presented in Table~\ref{tab:gold_data}.
\begin{table}[h]
\centering
\tiny
% {%
\begin{tabular}{@{}llrrrr@{}}
\toprule
\multicolumn{1}{c}{\textbf{Task}} & \multicolumn{1}{l}{\textbf{Classes}} & \multicolumn{1}{c}{\textbf{Train}} & \multicolumn{1}{c}{\textbf{Dev}} & \multicolumn{1}{c}{\textbf{Test}} & \multicolumn{1}{c}{\textbf{Total}} \\ \midrule
Crisis\textsubscript{Oltea}%~\cite{olteanu2014crisislex}
& \{on-topic, off-topic,\} & $48.0$K & $6.0$K & $6.0$K & $60.0$K \\
Emo\textsubscript{Moham}%~\cite{mohammad-2018-semeval}
& \{anger, joy, opt., sad.\} & $3.3$K & $374$ & $1.4$K & $5.0$K \\
Hate\textsubscript{Bas}%~\cite{waseem-2016-hateful}
& \{hateful, none\} & $9.0$K & $1.0$ & $3.0$ & $13.0$K \\
Hate\textsubscript{Waseem}%~\cite{waseem-2016-hateful}
& \{racism, sexism, none\} & $8.7$K & $1.1$K & $1.1$K & $10.9$K \\
Hate\textsubscript{David}%~\cite{davidson-2017-hateoffensive}
& \{hate, offensive, neither\} & $19.8$K & $2.5$K & $2.5$K & $24.8$K \\
Humor\textsubscript{Potash}%~\cite{potash-2017-semeval}
& \{humor, not humor\} & $11.3$K & $660$ & $749$ & $12.7$K \\
Humor\textsubscript{Meaney}%~\cite{meaney2021hahackathon}
& \{humor, not humor\} & $8.0$K & $1.0$K & $1.0$K & $10.0$K \\
Irony\textsubscript{Hee-A}%~\cite{van-hee2018semeval}
& \{ironic, not ironic\} & $3.5$K & $384$ & $784$ & $4.6$K \\
Irony\textsubscript{Hee-B}%~\cite{van-hee2018semeval}
& \{IC, SI, OI, NI\} & $3.5$K & $384$ & $784$ & $4.6$K \\
Offense\textsubscript{Zamp}%~\cite{zampieri-2019-semeval}
& \{offensive, not offensive\} & $11.9$K & $1.3$K & $860$ & $14.1$K \\
Sarc\textsubscript{Riloff}%~\cite{riloff2013sarcasm}
& \{sarcastic, non-sarcastic\} & $1.4$K & $177$ & $177$ &$1.8$K \\
Sarc\textsubscript{Ptacek}%~\cite{ptavcek2014sarcasm}
& \{sarcastic, non-sarcastic\} & $71.4$K & $8.9$K & $8.9$K & $89.3$K \\
Sarc\textsubscript{Rajad}%~\cite{rajadesingan2015sarcasm}
& \{sarcastic, non-sarcastic\} & $41.3$K & $5.2$K & $5.2$K & $51.6$K \\
Sarc\textsubscript{Bam}%~\cite{bamman2015contextualized}
& \{sarcastic, non-sarcastic\} & $11.9$K & $1.5$K & $1.5$K & $14.8$K \\
Senti\textsubscript{Rosen}%~\cite{rosenthal-2017-semeval}
& \{neg., neu., pos.\} & $42.8$K & $4.8$K & $12.3$K & $59.8$K \\
Senti\textsubscript{Thel}%~\cite{rosenthal-2017-semeval}
& \{neg., pos.\} & $900$ & $100$ & $1.1$K & $2.1$K \\
Stance\textsubscript{Moham}%~\cite{mohammad-2016-semeval}
& \{against, favor, none\} & $2.6$K & $292$ & $1.3$K & $4.2$K \\ \bottomrule
\end{tabular}%
% }
\caption{Gold social meaning datasets. \textbf{opt.:}: Optimism, \textbf{sad.:} Sadness, \textbf{IC:} Ironic by clash, \textbf{SI:} Situational irony, \textbf{OI:} Other irony, \textbf{NI:} Non-ironic, \textbf{neg.:} Negative, \textbf{Neu.:} Neutral, \textbf{pos.:} Positive.}\label{tab:gold_data}% \textbf{IC}: ironic by clash, \textbf{SI}: situational irony, \textbf{OI}: other irony, \textbf{NI}: non-ironic. \vspace{-7pt}}\label{tab:gold_data} \vspace{-5pt}
\end{table}
% \begin{table*}[t]
% \centering
% \footnotesize
% \begin{tabular}{@{}llrrrr@{}}
% \toprule
% \multicolumn{1}{c}{\textbf{Task}} & \multicolumn{1}{c}{\textbf{Classes}} & \multicolumn{1}{c}{\textbf{Train}} & \multicolumn{1}{c}{\textbf{Dev}} & \multicolumn{1}{c}{\textbf{Test}} & \multicolumn{1}{c}{\textbf{Total}} \\ \midrule
% Crisis\textsubscript{Oltea}~\cite{olteanu2014crisislex} & \{on-topic, off-topic,\} & 48,065 & 6,008 & 6,009 & 60,082 \\
% Emo\textsubscript{Moham}~\cite{mohammad-2018-semeval} & \{anger, joy, optimism, sadness\} & 3,257 & 374 & 1,422 & 5,053 \\
% Hate\textsubscript{Waseem}~\cite{waseem-2016-hateful} & \{racism, sexism, none\} & 8,683 & 1,086 & 1,086 & 10,855 \\
% Hate\textsubscript{David}~\cite{davidson-2017-hateoffensive} & \{hate, offensive, neither\} & 19,826 & 2,478 & 2,479 & 24,783 \\
% Humor\textsubscript{Potash}~\cite{potash-2017-semeval} & \multirow{2}{*}{\{humor, not humor\}} & 11,325 & 660 & 749 & 12,734 \\
% Humor\textsubscript{Meaney}~\cite{meaney2021hahackathon} & & 7,200 & 800 & 1,000 & 9,000 \\
% Irony\textsubscript{Hee-A}~\cite{van-hee2018semeval} & \{ironic, not ironic\} & 3,450 & 384 & 784 & 4,618 \\
% Irony\textsubscript{Hee-B}~\cite{van-hee2018semeval} & \{IC, SI, OI, NI\} & 3,450 & 384 & 784 & 4,618 \\
% Offense\textsubscript{Zamp}~\cite{zampieri-2019-semeval} & \{offensive, not offensive,\} & 11,916 & 1,324 & 860 & 14,100 \\
% Sarc\textsubscript{Riloff}~\cite{riloff2013sarcasm} & \multirow{4}{*}{\{sarcastic, non-sarcastic\}} & 1,413 & 177 & 177 & 1,767 \\
% Sarc\textsubscript{Ptacek}~\cite{ptavcek2014sarcasm} & & 71,433 & 8,929 & 8,930 & 89,292 \\
% Sarc\textsubscript{Rajad}~\cite{rajadesingan2015sarcasm} & & 41,261 & 5,158 & 5,158 & 51,577 \\
% Sarc\textsubscript{Bam}~\cite{bamman2015contextualized} & & 11,864 & 1,483 & 1,484 & 14,831 \\
% Senti\textsubscript{Rosen}~\cite{rosenthal-2017-semeval} & \{negative, neutral, positive\} & 42,756 & 4,752 & 12,284 & 59,792 \\
% Stance\textsubscript{Moham}~\cite{mohammad-2016-semeval} & \{against, favor, none\} & 2,622 & 292 & 1,249 & 4,163 \\ \bottomrule
% \end{tabular}
% \caption{Distribution of social meaning tasks. \textbf{IC}: ironic by clash, \textbf{SI}: situational irony, \textbf{OI}: other irony, NI: non-ironic. }\label{tab:gold_data}
% \end{table*}
\input{tables/para_sample_app}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Praphrase Model}\label{sec:para}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In order to train our paraphrasing model, we collect four paraphrase datasets from PIT-2015~\cite{xu2015semeval}, LanguageNet~\cite{lan2017continuously}, Opusparcus~\cite{creutz2018open}, and Quora Question Pairs (QQP)~\cite{iyer2017first}. We only keep sentence pairs with a high semantic similarity as follows: (1) For PIT-2015, we extract sentence pairs with semantic similarity labels of $5$ and $4$. (2) For LanguageNet, we keep sentence pairs obtained with similarity labels of $4$, $5$, and $6$. (3) For Opusparcus, we take pairs whose similarity labels are $4$. (4) For QQP, we extract sentence pairs with the `duplicate' label. Table~\ref{tab:para_train_data} presents the data size of each corpus after filtering. We then merge all extracted samples (a total of $625,097$ pairs) and split them into Train, Dev, and Test ($80\%$, $10\%$, and $10\%$).
\input{tables/para_train_data}
For the modelling, we fine-tune a pre-trained T5\textsubscript{Base}~\cite{raffel2020exploring} on the Train split for $20$ epochs with a constant learning rate of $3e-4$ and a maximal sequence length of $512$. We evaluated the model on the Dev set at the end of each epoch and identified the best model ($28.18$ BLEU score).
% \chiyu{Needs more details, sequence length, \# of words, model performance}
% % Please add the following required packages to your document preamble:
% % \usepackage{multirow}
% % \usepackage{graphicx}
% % \usepackage[table,xcdraw]{xcolor}
% % If you use beamer only pass "xcolor=table" option, i.e. \documentclass[xcolor=table]{beamer}
% % \usepackage[normalem]{ulem}
% % \useunder{\uline}{\ul}{}
% \begin{table*}[h]
% \renewcommand\thetable{4}
% \tiny
% \centering
% % \resizebox{\textwidth}{!}{%
% \begin{tabular}{llc}
% \toprule
% \textbf{Original Tweet} & \textbf{Paraphrase} & \textbf{Label} \\ \midrule
% & You guys are horrible, avoid MMT & { } \\
% & what I am doing is in my control, \#AvoidMMT, you guys are terrifying & { } \\
% & You guys are \#terrorist. I have used everything I have to do. & { } \\
% \multirow{-4}{*}{\begin{tabular}[c]{@{}l@{}}USER but what I am doing is in my control, \\ \#AvoidMMT , you guys are \#terrible\end{tabular}} & You guys are awful, but I am going to stop doing it. & \multirow{-4}{*}{{ anger}} \\\hline
% % & I hate when people say 'I need to talk to you or we need to talk to you'. I guess that's the problem. & \\
% % & I hate when people tell me 'I need to talk to you or we need to speak' my anxiety immediately goes up. & \\
% % \multirow{-3}{*}{\begin{tabular}[c]{@{}l@{}}I hate when people say 'I need to talk to you or we need to talk.' \\ My anxiety immediately goes up...\end{tabular}} & Why am I afraid when people say 'I need to talk to you or we need to talk?' & \multirow{-3}{*}{anger} \\\hline
% & The 46th wedding I've ruined. When I hit 50 I can retire. It's nice to see yo & \\
% & Here's the 47th wedding I've ruined. If I'm old enough to go on the 40s I can get married. & \\
% \multirow{-3}{*}{This is the 47th wedding I've ruined. When I hit 50 I can retire.} & After a single wedding, I drew 47 weddings, and before I hit 50 I can retire" & \multirow{-3}{*}{humor} \\\hline
% & Sorry to disturb you. I have absolutely no idea what time I'll be on cam tomorrow. & \\
% \multirow{-2}{*}{\begin{tabular}[c]{@{}l@{}}Sorry guys I have absolutely no idea \\ what time i'll be on cam tomorrow but will keep you posted.\end{tabular}} & I have absolutely no idea what time I'll be on camera tomorrow but I'll keep you posted & \multirow{-2}{*}{sadness} \\\hline
% & I'll buy you Dunkin' Donuts for \$5. & \\
% & Who wants to go with me for my tattoo tomorrow? I'll buy you a Dunkin' Donuts. & \\
% \multirow{-3}{*}{\begin{tabular}[c]{@{}l@{}}Who wants to go with me to get my tattoo tomorrow? \\ I'll buy you Dunkin doughnuts\end{tabular}} & Who wants to go with me to get my tattoo tomorrow? & \multirow{-3}{*}{neutral} \\\hline
% & The day before class please eat beans, onions and garlic. Also see the videos & \\
% & The Day Before Class. You should make that meal, (do you think). & \\
% \multirow{-3}{*}{\begin{tabular}[c]{@{}l@{}}USER May I suggest, that you have a meal that is made with \\ beans, onions \& garlic, the day before class.\end{tabular}} & If you can eat just the day before class, make a wonderful meal with garlic, onions and beans. & \multirow{-3}{*}{joy} \\
% \toprule
% \end{tabular}%
% % }
% \caption{Examples of paraphrases in SMPB}
% \label{tab:para_exam_app}
% \end{table*}
\begin{table}[h!]
\renewcommand\thetable{3}
\centering
\scriptsize
\begin{tabular}{@{}lcrH@{}}
\toprule
\multicolumn{1}{c}{\textbf{Dataset}} & \textbf{Domain} & \multicolumn{1}{c}{\textbf{\# of samples}} & \textbf{Sequence l.}\\ \midrule
PIT-2015 & Tweet & 3,789 & \\
LanguageNet & Tweet & 12,988 & \\
Opusparcus & Video subtitle & 462,846 & \\
QQP & Quora & 149,263 & \\ \hdashline
Total & \multicolumn{1}{r}{} & 625,097 \\ \bottomrule
\end{tabular}
\caption{Paraphrasing datasets.}\label{tab:para_train_data}
\end{table}
\begin{table}[h!]
\centering
\tiny
% \resizebox{0.85\columnwidth}{!}{%
\begin{tabular}{@{}lrrrr@{}}
\toprule
\multicolumn{1}{c}{\textbf{Task}} & \multicolumn{1}{c}{\textbf{Para1}} & \multicolumn{1}{c}{\textbf{Para2}} & \multicolumn{1}{c}{\textbf{Para4}} & \multicolumn{1}{c}{\textbf{Para5}} \\ \midrule
Crisis\textsubscript{Oltea} & $48.0$K & $86.8$K & $120.7$K & $123.9$K \\
Emo\textsubscript{Moham} & $3.3$K & $6.3$K & $10.9$K & $12.2$K \\
Hate\textsubscript{Bas} & $9.0$K & $17.6$K & $31.1$K & $35.1$K \\
Hate\textsubscript{Waseem} & $8.7$K & $16.6$K & $28.3$K & $31.7$K \\
Hate\textsubscript{David} & $19.8$K & $38.2$K & $65.5$K & $73.4$K \\
Humor\textsubscript{Potash} & $11.3$K & $21.8$K & $38.3$K & $44.0$K \\
Humor\textsubscript{Meaney} & $8.0$K & $15.7$K & $28.7$K & $33.0$K \\
Irony\textsubscript{Hee-A} & $3.5$K & $6.6$K & $11.4$K & $12.8$K \\
Irony\textsubscript{Hee-B} & $3.5$K & $6.6$K & $11.5$K & $12.9$K \\
Offense\textsubscript{Zamp} & $11.9$K & $23.0$K & $39.6$K & $44.3$K \\
Sarc\textsubscript{Riloff} & $1.4$K & $2.7$K & $4.6$K & $5.2$K \\
Sarc\textsubscript{Ptacek} & $71.4$K & $138.9$K & $242.1$K & $272.4$K \\
Sarc\textsubscript{Rajad} & $41.3$K & $78.3$K & $131.5$K & $146.6$K \\
Sarc\textsubscript{Bam} & $11.9$K & $22.4$K & $37.5$K & $41.6$K \\
Senti\textsubscript{Rosen} & $42.8$K & $84.3$K & $154.8$K & $178.1$K \\
Senti\textsubscript{Thel} & $900$ & $1.7$K & $3.1$K & $3.5$K \\
Stance\textsubscript{Moham} & $2.6$K & $4.7$K & $6.4$K & $6.6$K \\ \bottomrule
\end{tabular}%
% }
\caption{Distribution of PTSM Train set with varying paraphrase size.}\label{tab:para_n_data}
\end{table}
% \begin{table}[h]
% \centering
% \footnotesize
% \begin{tabular}{@{}lrrrr@{}}
% \toprule
% \multicolumn{1}{c}{\textbf{Task}} & \multicolumn{1}{c}{\textbf{Para1}} & \multicolumn{1}{c}{\textbf{Para2}} & \multicolumn{1}{c}{\textbf{Para4}} & \multicolumn{1}{c}{\textbf{Para5}} \\ \midrule
% Crisis\textsubscript{Oltea} & 48,065 & 86,765 & 120,731 & 123,879 \\
% Emo\textsubscript{Moham} & 3,257 & 6,288 & 10,886 & 12,227 \\
% Hate\textsubscript{Waseem} & 8,683 & 16,570 & 28,323 & 31,698 \\
% Hate\textsubscript{David} & 19,826 & 38,177 & 65,481 & 73,367 \\
% Humor\textsubscript{Potash} & 11,325 & 21,820 & 38,318 & 43,952 \\
% Humor\textsubscript{Meaney} & 7,200 & 14,152 & 25,741 & 29,601 \\
% Irony\textsubscript{Hee-A} & 3,450 & 6,646 & 11,413 & 12,777 \\
% Irony\textsubscript{Hee-B} & 3,450 & 6,639 & 11,469 & 12,850 \\
% Offense\textsubscript{Zamp} & 11,916 & 22,951 & 39,583 & 44,348 \\
% Sarc\textsubscript{Riloff} & 1,413 & 2,713 & 4,628 & 5,167 \\
% Sarc\textsubscript{Ptacek} & 71,433 & 138,870 & 242,065 & 272,362 \\
% Sarc\textsubscript{Rajad} & 41,261 & 78,328 & 131,533 & 146,605 \\
% Sarc\textsubscript{Bam} & 11,864 & 22,434 & 37,475 & 41,570 \\
% Senti\textsubscript{Rosen} & 42,756 & 84,299 & 154,378 & 178,147 \\
% Stance\textsubscript{Moham} & 2,622 & 4,717 & 6,425 & 6,609 \\ \bottomrule
% \end{tabular}
% \caption{Distribution of SMPB Train set.}\label{tab:paraphrase}
% \end{table}
\input{tables/para_res}
\section{Experiment and Result}\label{sec:result}
\subsection{Implementation}
We evaluate the quality of the $17$ paraphrased Train datasets in Table~\ref{tab:para_n_data} via fine-tuning two Transformer-based pre-trained language models, i.e, RoBERTa\textsubscript{Base}~\cite{liu2019roberta} and BERTweet\textsubscript{Base}~\cite{nguyen-etal-2020-bertweet}. We utilize the checkpoints released by Huggingface\footnote{\url{https://huggingface.co/models}}. %and implement the models by Transformers Python library.
For Crisis\textsubscript{Oltea} and Stance\textsubscript{Moham}, we append the topic term behind the post content, separate them by an `[SEP]' token, and set maximal sequence length to $72$. For the rest of tasks, we set maximal sequence length to $64$. For all the tasks, we pass the hidden state of `[CLS]' token from the last Transformer encoder layer through two feed-forward layers (a linear layer with a $tanh$ activation function followed by another linear layer with a $softmax$ activation function) to predict the classification label. Cross-Entropy is used to calculate the training loss. Following~\citet{nguyen-etal-2020-bertweet}, we use Adam with a peak learning rate of $1e-5$ and a weight decay of $0.01$ to optimize the model and fine-tune each task for $20$ epochs with early stopping ($patience = 5$ epochs). The training batch size is $32$. We compare the model trained on PTSM to ones fine-tuned on the original gold Train set with the same hyper-parameters. We run three times with random seeds for all downstream fine-tuning, \textit{reporting the average of these three runs}. All downstream task models are fine-tuned on $1$ Nvidia V$100$ GPUs ($32$G each). We fine-tune PLMs on different Train sets (paraphrased and original) but always evaluate on the original Dev and Test sets for all of the individual tasks. We typically identify the best model on each respective Dev set and evaluate its performance on blind Test. We present the average Test macro-averaged $F_1$ over the three runs as mentioned, and introduce a global metric averaging the macro-$F_1$ scores over the $17$ datasets.
\subsection{Result}
We use our PTSM to investigate the viability of using paraphrased training data instead of gold training sets. We fine-tune the PLMs on the PTSM Train sets with varying paraphrase sizes (referred to as P$n$ in Table~\ref{tab:para_res}). As Table~\ref{tab:para_res} shows, although none of our paraphrase-based models exceed the corresponding baseline model that is fine-tuned on gold (i.e., original) Train sets in term of average $F_1$, the paraphrase-based models either slightly exceeds or approaches performance of the gold models on individual datasets. Regarding the effect of paraphrase data size, we find P2 Train to perform best both for RoBERTa and BERTweet models as compared to other amounts of paraphrase data. This shows that while doubling paraphrase data size is useful, more paraphrases do not help the models. RoBERTa-P2 (the best setting of paraphrase-based RoBERTa fine-tuning) obtains an average $F_1$ of $73.36$, which is $1.66$ less than RoBERTa-Gold (while outperforming the latter on Humor\textsubscript{Potash-17} and Senti\textsubscript{Rosen-17}). BERTweet-P2 (the best setting of paraphrase-based BERTweet fine-tuning) underperforms BERTweet-Gold with $1.87$ average $F_1$ (i.e., $74.93$). We also observe our BERTweet-P1 model achieves a sizable improvement of $3.36$ $F_1$ on Sarc\textsubscript{Riloff} over the gold model. These findings demonstrate that \textbf{(i)} we can replace social gold data (which can become increasingly \textit{inaccessible} over time) with paraphrase datasets (which are \textit{persistent}) \textbf{(ii)} without sacrificing much performance. In addition, we also fine-tune PLMs on the combination of gold Train and our paraphrase Train sets (referred to as P$n$+G in Table~\ref{tab:para_res}), but find average $F_1$ scores of our models to still remain below the models fine-tuned on gold data only. We hypothesize that one limitation of our paraphrased Train sets is the lack of emojis. However, analyzing presence of emoji contents for each of the datasets, we find that our paraphrase-based models are able to acquire comparable performance to models trained with original datasets that do not employ any emojis (e.g., none of the training samples of Senti\textsubscript{Rosen-17}, Senti\textsubscript{Thel}, and Stance\textsubscript{Moham} uses emojis). To further investigate the issue, we fine-tune PLMs on a version of the gold Train set after removing emojis. Here, we observe a slight degradation of performance: RoBERTa and BERTweet each obtains an average $F_1$ of $74.67$ and $76.67$, respectively.%\footnote{With more space, we will provide the detailed results in the camera ready version if the paper is accepted.}
These findings reflect the effect of emoji symbols on social meaning detection. This suggests we can enhance our paraphrase data by inserting the same emojis as original data. We cast this as future work. %We now discuss the limitation and conclusion.
%In addition, an ensemble of our Para-Models is slightly better than the gold RoBERTa baseline ($F_1=75.83$ vs. $75.78$). %We also explore intermediate fine-tuning where we try to further fine-tune the already fine-tuned paraphrase-based model on gold downstream data, but results we acquire are still below the performance of the model fine-tuned on gold without intermediate fine-tuning on paraphrase data (average $F_1$ $76.91$ vs. $78.12$).
\begin{table*}[h!]
\tiny
\centering
\begin{tabular}{lr:rrrr:rrrr|r:rrrr:rrrr}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{\textbf{Task}}} & \multicolumn{9}{c}{\textbf{RoBERTa}} & \multicolumn{9}{c}{\textbf{BERTweet}} \\ \cmidrule(l){2-10} \cmidrule(l){11-19}
\multicolumn{1}{c}{} & \multicolumn{1}{c:}{\textbf{Gold}} & \multicolumn{1}{c}{\textbf{P1}} & \multicolumn{1}{c}{\textbf{P2}} & \multicolumn{1}{c}{\textbf{P4}} & \multicolumn{1}{c:}{\textbf{P5}} & \multicolumn{1}{c}{\textbf{P1+G}} & \multicolumn{1}{c}{\textbf{P2+G}} & \multicolumn{1}{c}{\textbf{P4+G}} & \multicolumn{1}{c|}{\textbf{P5+G}} & \multicolumn{1}{c:}{\textbf{Gold}} & \multicolumn{1}{c}{\textbf{P1}} & \multicolumn{1}{c}{\textbf{P2}} & \multicolumn{1}{c}{\textbf{P4}} & \multicolumn{1}{c:}{\textbf{P5}} & \multicolumn{1}{c}{\textbf{P1+G}} & \multicolumn{1}{l}{\textbf{P2+G}} & \multicolumn{1}{c}{\textbf{P4+G}} & \multicolumn{1}{c}{\textbf{P5+G}} \\ \midrule
Crisis\textsubscript{O.} & 95.96 & \textbf{95.38} & 95.09 & 95.18 & 95.00 & \textbf{95.88} & 95.85 & 95.55 & 95.17 & 95.58 & 94.90 & 95.11 & \textbf{95.26} & 95.17 & \underline{\textbf{95.75}} & \underline{95.60} & 95.47 & \underline{95.63} \\
Emo\textsubscript{M.} & 77.61 & 76.29 & \textbf{77.16} & 77.02 & 76.59 & \underline{78.54} & \underline{77.65} & \underline{\textbf{78.82}} & \underline{78.54} & 80.37 & 77.61 & 78.06 & 77.66 & \textbf{78.62} & \textbf{80.71} & 80.29 & 78.91 & 79.56 \\
Hate\textsubscript{B.} & 49.74 & 44.70 & 46.48 & 45.81 & \textbf{47.89} & \textbf{48.78} & 47.99 & 48.26 & 48.21 & 56.51 & \textbf{54.01} & 52.41 & 51.71 & 52.04 & \textbf{54.88} & 54.53 & 53.01 & 51.89 \\
Hate\textsubscript{W.} & 56.84 & 54.22 & \textbf{55.00} & 54.89 & 54.92 & 56.64 & \textbf{56.67} & 56.17 & 55.46 & 57.07 & 55.33 & 55.41 & \underline{\textbf{59.33}} & 55.90 & 56.42 & \textbf{56.87} & 56.10 & 56.14 \\
Hate\textsubscript{D.} & 77.03 & 75.14 & 73.79 & 74.43 & \textbf{75.20} & \textbf{75.81} & 75.57 & 74.66 & 74.66 & 77.57 & 75.11 & \textbf{75.75} & 74.35 & 75.08 & \textbf{77.21} & 74.70 & 76.00 & 75.97 \\
Humor\textsubscript{P.} & 54.66 & 52.92 & \underline{\textbf{55.81}} & 54.18 & 52.59 & \underline{55.11} & \underline{\textbf{56.10}} & 53.25 & 50.64 & 57.56 & \textbf{52.77} & 52.65 & 52.11 & 50.92 & \textbf{56.13} & 53.42 & 53.77 & 54.34 \\
Humor\textsubscript{M.} & 92.61 & 90.96 & \textbf{92.25} & 91.20 & 91.96 & 91.59 & 92.08 & \textbf{92.45} & 91.57 & 94.21 & 93.31 & 93.30 & \textbf{93.76} & 92.67 & 93.95 & 94.00 & 93.57 & \textbf{94.16} \\
Irony\textsubscript{H-A} & 73.13 & 69.96 & 70.46 & 70.52 & \textbf{71.14} & 72.51 & \textbf{72.72} & 72.22 & 72.06 & 76.82 & \textbf{76.26} & 75.42 & 75.08 & 75.37 & \underline{\textbf{77.90}} & \underline{77.43} & \underline{77.04} & 75.60 \\
Irony\textsubscript{H-B} & 51.56 & 46.23 & \textbf{48.52} & 46.92 & 45.31 & \textbf{50.54} & 49.20 & 49.79 & 47.17 & 56.84 & 47.89 & 51.18 & 49.98 & \textbf{51.78} & 56.24 & \textbf{56.45} & 50.72 & 55.63 \\
Offense\textsubscript{Z} & 80.67 & 77.49 & \textbf{80.22} & 80.18 & 79.41 & 80.39 & 80.20 & \textbf{80.77} & 79.94 & 79.74 & 77.63 & 78.89 & 79.29 & \underline{\textbf{80.15}} & 78.39 & 78.56 & \textbf{79.47} & 79.32 \\
Sarc\textsubscript{Ri.} & 75.28 & 71.04 & 70.50 & 71.22 & \textbf{72.64} & 71.44 & 71.69 & 73.39 & \textbf{74.52} & 76.61 & \underline{\textbf{79.97}} & \underline{78.61} & \underline{78.81} & 75.97 & \textbf{\underline{80.19}} & \underline{77.01} & \underline{77.60} & \underline{77.21} \\
Sarc\textsubscript{P.} & 95.59 & 91.67 & 93.48 & 94.03 & \textbf{94.35} & \textbf{95.34} & \textbf{95.34} & 95.11 & 95.16 & 96.74 & 92.96 & 94.34 & 94.62 & \textbf{94.70} & \textbf{96.34} & 96.04 & 95.85 & 95.71 \\
Sarc\textsubscript{Ra.} & 85.64 & 80.82 & \textbf{82.34} & 82.20 & 82.29 & \textbf{84.90} & 84.71 & 84.00 & 84.29 & 86.97 & 84.08 & 84.99 & 84.81 & \textbf{85.31} & \textbf{86.95} & 86.55 & 86.14 & 86.19 \\
Sarc\textsubscript{B.} & 80.01 & 77.33 & \textbf{78.18} & 77.69 & 77.09 & \textbf{79.88} & 79.49 & 78.91 & 78.56 & 82.59 & 80.45 & 80.60 & \textbf{80.93} & 80.32 & \underline{\textbf{82.92}} & 82.11 & 81.99 & 81.98 \\
Senti\textsubscript{R.} & 70.78 & 70.59 & \underline{\textbf{71.44}} & 70.26 & 69.99 & \underline{\textbf{71.19}} & \underline{71.15} & 70.30 & 70.76 & 72.05 & \textbf{71.23} & 70.21 & 69.75 & 69.98 & 71.67 & 71.19 & 71.17 & \textbf{71.78} \\
Senti\textsubscript{T.} & 88.99 & 87.52 & \textbf{88.12} & 87.83 & 87.79 & \underline{\textbf{89.04}} & 88.46 & 88.48 & 87.62 & 89.27 & 88.18 & 89.00 & 88.49 & \underline{\textbf{89.28}} & 89.11 & \underline{89.35} & 89.16 & \underline{\textbf{89.64}} \\
Stance\textsubscript{M.} & 69.28 & 67.81 & 68.27 & \underline{\textbf{69.76}} & 69.07 & 68.56 & \textbf{68.64} & 68.44 & 67.54 & 69.11 & 67.30 & \textbf{67.92} & 66.96 & 66.76 & \underline{69.12} & \underline{\textbf{69.83}} & 68.11 & 66.74 \\
\cdashline{1-19} \multicolumn{1}{c}{\textbf{Average}} & 75.02 & 72.36 & \textbf{73.36} & 73.14 & 73.13 & \textbf{74.48} & 74.33 & 74.15 & 73.64 & 76.80 & 74.65 & \textbf{74.93} & 74.88 & 74.71 & \textbf{76.70} & 76.11 & 75.53 & 75.73 \\ \bottomrule
\end{tabular}
\caption{Benchmarking PTSM. \textbf{Gold} denotes a model fine-tuned with downstream, original Train data. \textbf{P$n$} indicates that the model is trained on Para$n$ training set. \textbf{P$n$} indicates that the model is trained on Para$n$ training set. \textbf{P$n$+G} indicates that the model is trained on the combination of Para$n$ and original gold training set. \textbf{Bold} denotes the best result for each task under each group of settings. \textbf{\underline{Underscore}} indicates that the model outperforms the corresponding baseline that is fine-tuned on gold Train set. }~\label{tab:para_res}
\end{table*}
% \begin{table*}[t]
% \scriptsize
% \centering
% \begin{tabular}{@{}lcccc:cccccc:cc@{}}
% \toprule
% \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Task}}} & \multicolumn{6}{c}{\textbf{Baseline (RoBERTa\textsubscript{BASE})}} & \multicolumn{6}{c}{\textbf{Surrogate Fine-tuning (SFT)}} \\ \cmidrule(l){2-13}
% \multicolumn{1}{c}{} & \textbf{P1} & \textbf{P2} & \textbf{P4} & \textbf{P5} & \textbf{P-Best} & \multicolumn{1}{c|}{\textbf{Gold}} & \textbf{P1} & \textbf{P2} & \textbf{P4} & \textbf{P5} & \textbf{P-Best} & \textbf{Gold} \\ \midrule
% Crisis\textsubscript{Oltea} & \textbf{95.54} & 95.20 & 95.23 & 95.39 & 95.54 & \multicolumn{1}{c|}{95.95} & 95.07 & 94.99 & \textbf{95.12} & 95.06 & 95.12 & 95.87 \\
% Emo\textsubscript{Moham} & 77.06 & \textbf{77.43} & 77.31 & 76.99 & 77.43 & \multicolumn{1}{c|}{77.99} & 75.85 & \textbf{76.94} & 76.45 & 76.63 & 76.94 & 78.69 \\
% Hate\textsubscript{Waseem} & 54.58 & \textbf{54.92} & 54.83 & 54.63 & 54.92 & \multicolumn{1}{c|}{57.34} & 54.01 & \textbf{54.89} & 54.65 & 54.52 & 54.89 & 63.97 \\
% Hate\textsubscript{David} & 74.04 & 75.26 & \textbf{75.80} & 74.97 & 75.80 & \multicolumn{1}{c|}{77.71} & 72.54 & 74.06 & 73.71 & \textbf{75.27} & 75.27 & 77.29 \\
% Humor\textsubscript{Potash} & 49.26 & 53.16 & \textbf{53.54} & 53.34 & 53.54 & \multicolumn{1}{c|}{54.40} & 50.38 & 51.99 & 51.62 & \textbf{53.22} & 53.22 & 55.51 \\
% Humor\textsubscript{Meaney} & 88.67 & 88.85 & \textbf{89.11} & 88.16 & 89.11 & \multicolumn{1}{c|}{90.33} & 90.33 & 89.73 & \textbf{90.46} & 89.21 & 90.46 & 92.02 \\
% Irony\textsubscript{Hee-A} & 71.00 & 71.53 & 71.06 & \textbf{71.53} & 71.53 & \multicolumn{1}{c|}{73.93} & 72.89 & 73.43 & \textbf{75.28} & 74.29 & 75.28 & 76.22 \\
% Irony\textsubscript{Hee-B} & 43.24 & 45.69 & 48.02 & \textbf{48.33} & 48.33 & \multicolumn{1}{c|}{52.30} & 49.01 & 49.08 & \textbf{51.82} & 51.14 & 51.82 & 60.14 \\
% Offense\textsubscript{Zamp} & 79.90 & 79.95 & \textbf{80.15} & 79.57 & 80.15 & \multicolumn{1}{c|}{80.13} & 77.98 & \textbf{79.41} & 78.96 & 79.10 & 79.41 & 79.82 \\
% Sarc\textsubscript{Riloff} & 71.40 & \textbf{72.74} & 72.45 & 72.26 & 72.74 & \multicolumn{1}{c|}{73.85} & 74.27 & 76.86 & 77.26 & \textbf{78.36} & 78.36 & 80.50 \\
% Sarc\textsubscript{Ptacek} & 90.54 & 92.58 & \textbf{93.70} & 93.62 & 93.70 & \multicolumn{1}{c|}{95.09} & 90.85 & 92.69 & 93.57 & \textbf{93.61} & 93.61 & 96.01 \\
% Sarc\textsubscript{Rajad} & 81.00 & 81.94 & 82.53 & \textbf{82.64} & 82.64 & \multicolumn{1}{c|}{85.07} & 85.71 & 86.13 & 86.19 & \textbf{86.32} & 86.32 & 87.56 \\
% Sarc\textsubscript{Bam} & 77.60 & 76.93 & \textbf{77.92} & 77.76 & 77.92 & \multicolumn{1}{c|}{79.08} & 80.20 & 80.05 & \textbf{80.90} & 80.22 & 80.90 & 81.19 \\
% Senti\textsubscript{Rosen} & 70.50 & 70.95 & 71.51 & \textbf{71.52} & 71.52 & \multicolumn{1}{c|}{71.08} & 67.35 & 68.20 & \textbf{69.18} & 68.59 & 69.18 & 71.83 \\
% Stance\textsubscript{Moham} & 67.75 & 69.07 & \textbf{69.30} & 67.34 & 69.30 & \multicolumn{1}{c|}{70.41} & 67.61 & \textbf{69.61} & 68.20 & 69.47 & 69.61 & 71.27 \\ \cdashline{1-13}
% \textbf{Average} & 72.81 & 73.75 & 74.17 & 73.87 & 74.28 & \multicolumn{1}{c|}{75.64} & 73.60 & 74.54 & 74.89 & 75.00 & 75.36 & 77.86 \\ \bottomrule
% \end{tabular} \vspace{-5pt}
% \caption{Results of paraphrasing-based model. \textbf{Gold} denotes the model train with original Train dataset. \textbf{P$n$} indicates that the model is trained on Para$n$ training set. \textbf{Bold font} denotes the best paraphrase result for each task within each of two settings (Baseline vs. SFT). \textbf{P-Best} is the performance of the best paraphrasing setting. \vspace{-10pt}}~\label{tab:para_res}
% \end{table*}
\section{Conclusion and Limitation}
Motivated by the issue of decay of social media data, we proposed simple paraphrasing as employed in our PTSM data as a way to avail training datasets for learning social meaning. We fine-tune a T5 model on diverse paraphrasing dataset and utilize the trained model to generate paraphrases of original social meaning Train sets. Through experimental results, we show that we can substitute the actual tweets with their paraphrases while incurring a marginal performance loss. Due to the closed vocabulary of T5, our paraphrasing model cannot generate emojis (even though these can be useful for social meaning detection). We can rectify this by simple post-processing in future work. Another possible limitation of our work is that we use a simple tri-gram similarity method to measure similarity between an actual tweet and its paraphrases. A more sophisticated method may enhance our resulting paraphrase data and hence possibly improve downstream model performance; we will explore this in the future.
\section*{Ethical Consideration}~\label{sec:ethic}
PTSM is collected from publicly available sources and aims at availing resources for training NLP models without need for original user data, which could be a step forward toward protecting privacy. Following Twitter policy, all the data we used for model training are anonymous. We also notice the annotation bias existing in the original datasets (e.g., Hate\textsubscript{Waseem}). We will accompany our data and model release with model cards. We will also provide more detailed ethical considerations on a dedicated GitHub repository.
\begin{thebibliography}{33}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi:\discretionary{}{}{}#1}\else
\providecommand{\doi}{doi:\discretionary{}{}{}\begingroup
\urlstyle{rm}\Url}\fi
\bibitem[{Bamman and Smith(2015)}]{bamman2015contextualized}
Bamman, D.; and Smith, N.~A. 2015.
\newblock Contextualized sarcasm detection on twitter.
\newblock In \emph{Proc. of AAAI}.
\bibitem[{Barbieri et~al.(2020)Barbieri, Camacho-Collados, Espinosa~Anke, and
Neves}]{barbieri-2020-tweeteval}
Barbieri, F.; Camacho-Collados, J.; Espinosa~Anke, L.; and Neves, L. 2020.
\newblock {T}weet{E}val: Unified Benchmark and Comparative Evaluation for Tweet
Classification.
\newblock In \emph{Findings of the ACL: EMNLP 2020}.
\bibitem[{Basile et~al.(2019)Basile, Bosco, Fersini, Nozza, Patti,
Rangel~Pardo, Rosso, and Sanguinetti}]{basile-2019-semeval}
Basile, V.; Bosco, C.; Fersini, E.; Nozza, D.; Patti, V.; Rangel~Pardo, F.~M.;
Rosso, P.; and Sanguinetti, M. 2019.
\newblock {S}em{E}val-2019 Task 5: Multilingual Detection of Hate Speech
Against Immigrants and Women in {T}witter.
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Beddiar, Jahan, and Oussalah(2021)}]{beddiar2021data}
Beddiar, D.~R.; Jahan, M.~S.; and Oussalah, M. 2021.
\newblock Data expansion using back translation and paraphrasing for hate
speech detection.
\newblock \emph{Online Social Networks and Media} .
\bibitem[{Creutz(2018)}]{creutz2018open}
Creutz, M. 2018.
\newblock Open Subtitles Paraphrase Corpus for Six Languages.
\newblock In \emph{Proc. of LREC}.
\bibitem[{Davidson et~al.(2017)Davidson, Warmsley, Macy, and
Weber}]{davidson-2017-hateoffensive}
Davidson, T.; Warmsley, D.; Macy, M.; and Weber, I. 2017.
\newblock {Automated Hate Speech Detection and the Problem of Offensive
Language}.
\newblock In \emph{Proc. of AAAI}.
\bibitem[{Farzindar and Inkpen(2015)}]{farzindar2015natural}
Farzindar, A.; and Inkpen, D. 2015.
\newblock Natural language processing for social media.
\newblock \emph{Synthesis Lectures on Human Language Technologies} .
\bibitem[{Holtzman et~al.(2020)Holtzman, Buys, Du, Forbes, and
Choi}]{holtzman2019curious}
Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2020.
\newblock The Curious Case of Neural Text Degeneration.
\newblock In \emph{Proc. of ICLR}.
\bibitem[{Iyer et~al.(2017)Iyer, Dandekar, Csernai et~al.}]{iyer2017first}
Iyer, S.; Dandekar, N.; Csernai, K.; et~al. 2017.
\newblock First quora dataset release: Question pairs.
\newblock \emph{data. quora. com} .
\bibitem[{Lan et~al.(2017)Lan, Qiu, He, and Xu}]{lan2017continuously}
Lan, W.; Qiu, S.; He, H.; and Xu, W. 2017.
\newblock A Continuously Growing Dataset of Sentential Paraphrases.
\newblock In \emph{Proceedings of the 2017 Conference on EMNLP}.
\bibitem[{Li et~al.(2016)Li, Shah, Ghassemi, Fang, Nourbakhsh, and
Liu}]{li2016using}
Li, Q.; Shah, S.; Ghassemi, M.; Fang, R.; Nourbakhsh, A.; and Liu, X. 2016.
\newblock Using paraphrases to improve tweet classification: Comparing wordnet
and word embedding approaches.
\newblock In \emph{2016 IEEE International Conference on Big Data}.
\bibitem[{Liu et~al.(2019)Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis,
Zettlemoyer, and Stoyanov}]{liu2019roberta}
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.;
Zettlemoyer, L.; and Stoyanov, V. 2019.
\newblock {RoBERTa}: A robustly optimized bert pretraining approach.
\newblock \emph{arXiv preprint arXiv:1907.11692} .
\bibitem[{Meaney et~al.(2021)Meaney, Wilson, Chiruzzo, and
Magdy}]{meaney2021hahackathon}
Meaney, J.; Wilson, S.~R.; Chiruzzo, L.; and Magdy, W. 2021.
\newblock {HaHackathon:} Detecting and Rating Humor and Offense.
\newblock In \emph{Proc. of the 59th Annual Meeting of the ACL and the 11th
IJCNLP}.
\bibitem[{Mohammad et~al.(2018)Mohammad, Bravo-Marquez, Salameh, and
Kiritchenko}]{mohammad-2018-semeval}
Mohammad, S.; Bravo-Marquez, F.; Salameh, M.; and Kiritchenko, S. 2018.
\newblock {S}em{E}val-2018 Task 1: Affect in Tweets.
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Mohammad et~al.(2016)Mohammad, Kiritchenko, Sobhani, Zhu, and
Cherry}]{mohammad-2016-semeval}
Mohammad, S.; Kiritchenko, S.; Sobhani, P.; Zhu, X.; and Cherry, C. 2016.
\newblock {S}em{E}val-2016 Task 6: Detecting Stance in Tweets.
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Nguyen, Vu, and Tuan~Nguyen(2020)}]{nguyen-etal-2020-bertweet}
Nguyen, D.~Q.; Vu, T.; and Tuan~Nguyen, A. 2020.
\newblock {BERT}weet: A pre-trained language model for {E}nglish Tweets.
\newblock In \emph{Proc. of the 2020 Conference on EMNLP: System
Demonstrations}.
\bibitem[{Olteanu et~al.(2014)Olteanu, Castillo, Diaz, and
Vieweg}]{olteanu2014crisislex}
Olteanu, A.; Castillo, C.; Diaz, F.; and Vieweg, S. 2014.
\newblock {Crisislex: A lexicon for collecting and filtering microblogged
communications in crises}.
\newblock In \emph{Proc. of AAAI}.
\bibitem[{Petrovi{\'c}, Osborne, and Lavrenko(2012)}]{petrovic2012using}
Petrovi{\'c}, S.; Osborne, M.; and Lavrenko, V. 2012.
\newblock Using paraphrases for improving first story detection in news and
Twitter.
\newblock In \emph{Proc. of NAACL}.
\bibitem[{Potash, Romanov, and Rumshisky(2017)}]{potash-2017-semeval}
Potash, P.; Romanov, A.; and Rumshisky, A. 2017.
\newblock {S}em{E}val-2017 Task 6: {\#}{H}ashtag{W}ars: Learning a Sense of
Humor.
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Pt{\'a}{\v{c}}ek, Habernal, and Hong(2014)}]{ptavcek2014sarcasm}
Pt{\'a}{\v{c}}ek, T.; Habernal, I.; and Hong, J. 2014.
\newblock Sarcasm Detection on {C}zech and {E}nglish {T}witter.
\newblock In \emph{Proc. of COLING}.
\bibitem[{Raffel et~al.(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena,
Zhou, Li, and Liu}]{raffel2020exploring}
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou,
Y.; Li, W.; and Liu, P.~J. 2020.
\newblock Exploring the Limits of Transfer Learning with a Unified Text-to-Text
Transformer.
\newblock \emph{Journal of Machine Learning Research} .
\bibitem[{Rajadesingan, Zafarani, and Liu(2015)}]{rajadesingan2015sarcasm}
Rajadesingan, A.; Zafarani, R.; and Liu, H. 2015.
\newblock Sarcasm Detection on Twitter: {A} Behavioral Modeling Approach.
\newblock In \emph{Proc. of WSDM}.
\bibitem[{Riloff et~al.(2013)Riloff, Qadir, Surve, De~Silva, Gilbert, and
Huang}]{riloff2013sarcasm}
Riloff, E.; Qadir, A.; Surve, P.; De~Silva, L.; Gilbert, N.; and Huang, R.
2013.
\newblock Sarcasm as Contrast between a Positive Sentiment and Negative
Situation.
\newblock In \emph{Proceedings of the 2013 Conference on EMNLP}.
\bibitem[{Rosenthal, Farra, and Nakov(2017)}]{rosenthal-2017-semeval}
Rosenthal, S.; Farra, N.; and Nakov, P. 2017.
\newblock {S}em{E}val-2017 Task 4: Sentiment Analysis in {T}witter.
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Sarlin et~al.(2020)Sarlin, DeTone, Malisiewicz, and
Rabinovich}]{sarlin2020superglue}
Sarlin, P.-E.; DeTone, D.; Malisiewicz, T.; and Rabinovich, A. 2020.
\newblock Superglue: Learning feature matching with graph neural networks.
\newblock In \emph{Proc. of CVPR}.
\bibitem[{Thelwall, Buckley, and Paltoglou(2012)}]{thelwall2012sentiment}
Thelwall, M.; Buckley, K.; and Paltoglou, G. 2012.
\newblock Sentiment strength detection for the social web.
\newblock \emph{J. Assoc. Inf. Sci. Technol.} .
\bibitem[{Thomas(2014)}]{thomas2014meaning}
Thomas, J.~A. 2014.
\newblock \emph{Meaning in interaction: An introduction to pragmatics}.
\newblock Routledge.
\bibitem[{Van~Hee, Lefever, and Hoste(2018)}]{van-hee2018semeval}
Van~Hee, C.; Lefever, E.; and Hoste, V. 2018.
\newblock {S}em{E}val-2018 Task 3: Irony Detection in {E}nglish Tweets.
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Wang et~al.(2019)Wang, Singh, Michael, Hill, Levy, and
Bowman}]{wang2019glue}
Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S.~R. 2019.
\newblock Glue: A multi-task benchmark and analysis platform for natural
language understanding.
\newblock In \emph{Proc. of ICLR}.
\bibitem[{Waseem and Hovy(2016)}]{waseem-2016-hateful}
Waseem, Z.; and Hovy, D. 2016.
\newblock Hateful Symbols or Hateful People? Predictive Features for Hate
Speech Detection on {T}witter.
\newblock In \emph{Proc. of NAACL}.
\bibitem[{Xu, Callison-Burch, and Dolan(2015)}]{xu2015semeval}
Xu, W.; Callison-Burch, C.; and Dolan, B. 2015.
\newblock {S}em{E}val-2015 Task 1: Paraphrase and Semantic Similarity in
{T}witter ({PIT}).
\newblock In \emph{Proc. of SemEval}.
\bibitem[{Zampieri et~al.(2019)Zampieri, Malmasi, Nakov, Rosenthal, Farra, and
Kumar}]{zampieri-2019-predicting}
Zampieri, M.; Malmasi, S.; Nakov, P.; Rosenthal, S.; Farra, N.; and Kumar, R.
2019.
\newblock Predicting the Type and Target of Offensive Posts in Social Media.
\newblock In \emph{Proc. of NAACL}.
\bibitem[{Zubiaga(2018)}]{zubiaga2018longitudinal}
Zubiaga, A. 2018.
\newblock A longitudinal assessment of the persistence of twitter datasets.
\newblock \emph{J. Assoc. Inf. Sci. Technol.} 69(8).
\end{thebibliography}