Echoes of Persuasion: The Effect of Euphony in Persuasive Communication
Extracted source text · arXiv 1508.05817
%
% File acl2014.tex
%
% Contact: koller@ling.uni-potsdam.de, yusuke@nii.ac.jp
%%
%% Based on the style files for ACL-2013, which were, in turn,
%% Based on the style files for ACL-2012, which were, in turn,
%% based on the style files for ACL-2011, which were, in turn,
%% based on the style files for ACL-2010, which were, in turn,
%% based on the style files for ACL-IJCNLP-2009, which were, in turn,
%% based on the style files for EACL-2009 and IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
\documentclass[11pt,letterpaper]{article}
\usepackage{naaclhlt2015}
\usepackage{times}
\usepackage{latexsym}
\usepackage[hyphens]{url}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{booktabs}
\usepackage{color}
%--marco added---
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\renewcommand{\baselinestretch}{0.995}
\usepackage{multirow}
\setlength\titlebox{6.5cm} % Expanding the titlebox
\interfootnotelinepenalty=10000
%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.
\title{Echoes of Persuasion: \\ The Effect of Euphony in Persuasive Communication}
%\title{Echoes of Persuasion \\ The MySteriouS SoundS of PerSuaSiveneSS}
\author{Marco Guerini \\
Trento RISE\\
Povo, I-38100 Trento\\
{\tt \footnotesize{marco.guerini@trentorise.eu}}\\\And
G\"ozde \"Ozbal \\
FBK-Irst\\
%Via Sommarive 18\\
Povo, I-38100 Trento\\
{\tt \footnotesize{gozbalde@gmail.com}} \\\And
Carlo Strapparava \\
FBK-Irst\\
%Via Sommarive 18\\
Povo, I-38100 Trento\\
{\tt \footnotesize{strappa@fbk.eu}} \\}
% \author{cc \\
%University of cc\\
%{\tt cc@dcc} \\\And
%dd\\
%University of cc\\
%{\tt dd@dcc}
%}
\date{}
\begin{document}
\maketitle
\begin{abstract}
While the effect of various lexical, syntactic, semantic and stylistic features have been addressed in persuasive language from a computational point of view, the persuasive effect of phonetics has received little attention. By modeling a notion of euphony %(pleasantness of the sounds of words, phrases and sentences)
and analyzing four datasets comprising persuasive and non-persuasive sentences in different domains (political speeches, movie quotes, slogans and tweets), we explore the impact of sounds on different forms of persuasiveness. We conduct a series of analyses and prediction experiments within and across datasets. Our results highlight the positive role of phonetic devices on persuasion.% To the best of our knowledge, this is the first systematic analysis of the impact of phonetic features on several types of persuasiveness.
\end{abstract}
\section{Hocus Pocus}
%According to Perelman and Olbrechts-Tyteca \cite{perelman:olbrechts:69}, persuasion is a skill that human beings use - in communication - in order to make their partners perform certain actions or collaborate in various activities; a similar definition can be found in \cite{moul02}.
Historically, in human sciences, several definitions of persuasion have been proposed -- see for example \cite{Toulmin:58,walton:96,Chaiken1980,cial93,Petty1986}. Most of them have a common core addressing:
%\begin{quotation}
%\noindent
\emph{methodologies aiming to change the mental state of the receiver by means of communication in view of a possible action to be performed by her/him.} \cite{perelman:olbrechts:69,moul02}.
%\end{quotation}
These methodologies might take into account the overall structure of a text such as the ordering of the arguments or simply single word choices. For a successful text both of them are often required.
%\textcolor{red}{These methodologies can concern from the deep structure of a text, such as the ordering of the arguments in a long text, down to single word choice that pertains the surface realization of the same text. While we acknowledge that for a successful text we need both strategic planning and surface realization, depending on the context and on text constranints, some methodologies may become more or less relevant.}
The focus of persuasion may vary according to the goal of the communication and it can take different forms according to the domain: from memorability (e.g., making people remember a statement or a product) to diffusion (e.g., making people pass on a content in social networks by sharing it), from behavioral change (e.g., political communication) to influencing purchasing decisions (e.g., slogans to convince people to try or buy a product) -- see for example \cite{heath2007made}.
%\textcolor{red}{Various lexical, syntactic, semantic and stylistic features have been addressed in persuasive language from a computational point of view. However, the persuasive effect of phonetics, in particular euphony, has received little attention so far.}
\textcolor{black}{While many techniques such as resorting to expert opinion, utilizing the framing effect, emotive language or exaggeration can be used to obtain such persuasive effects, we devote this study to explore particular techniques pertaining to euphony.}
%\textcolor{red}{The first sentence breaks the fluency, I don't see the relation with the next sentence} The idea that computers can handle such forms of language by changing how we think or behave, has been put forward as well \cite{fogg02}.
%\textcolor{red}{Still, there has been little or no study on the effects of euphonic language in persuasion.}
Euphony refers to the inherent pleasantness of the sounds of words, phrases and sentences, and it is utilized to achieve pleasant, rhythmical and harmonious effects. The idea that the pleasantness of the sounds in a sentence can foster its effectiveness is rooted in our culture, and is connected to the concepts of rhythm and music.
The fact that language and music interact in our brain has been shown by localizing low-level syntactic processes of music and language in the temporal lobe \cite{sammler2013co}. It has also been shown that changes in the cardiovascular and respiratory systems can be induced by music -- specifically tempo, rhythm, melodic structure \cite{bernardi2006cardiovascular}.
The importance of euphony has its roots also in ancient human psychology.
As Julian Jaynes suggests \cite{jaynes2000origin}, poetry used to be divine knowledge. %In ancient times of human psychological development, poetry
It was the sound and tenor of authorization and it commanded where plain prose could only ask. A paradigmatic example of this conception is the act of casting a spell. %is a linguistic act that can change reality.
Spells (incantations) are special linguistic objects that are meant not only to change how people think or behave but they are also so powerful that they can -- allegedly -- change reality. Spells are often very euphonic (and meaningless) sentences, e.g. ``Hocus Pocus".
%, leading some organizations to consider music as doping in sports \cite{bernardi2006cardiovascular}.}
%\textcolor{red}{DECIDE IF TO KEEP: The ``coercitive" power of music was also acknowledged in the literature: according to Tolstoy, music ``should be a State affair" given that it can be used to ``hypnotize another, or many others, and do what he likes with them" \cite{tolstoy2004kreutzer}. The fact that language and music interact in our brain has been shown as well, by localizing low-level syntactic processes of music and language in the temporal lobe \cite{sammler2013co}. It has also been shown that changes in the cardiovascular and respiratory systems can be induced by music (specifically tempo, rhythm, melodic structure), leading some organizations to consider music as doping in sports \cite{bernardi2006cardiovascular}.}
Various psycholinguistic studies addressed the effects of phonetics on the audience in different aspects such as memorability \cite{wales,doi:10.1080/10926488.2013.797728} or more specifically advertisement \cite{leech,soundAdvice}. There are also computational studies that address the problem of recognizing persuasive sentences according to various syntactic, lexical and semantic features \cite{danescu2012you,tan2014effect}. However, to the best of our knowledge, the direct impact of phonetic elements on persuasiveness has not been explored in computational settings yet.
In this paper, we fill in this gap by conducting a series of analyses and prediction experiments on four datasets representing different %ingredients
aspects of persuasive language to evaluate the importance of a set of phonetic devices (i.e. rhyme, alliteration, homogeneity and plosives) on various forms of persuasiveness. %In particular, we will focus on %by considering three different phenomena, alliteration and rhyme that are position dependent while homogeneity covers various sound repetition phenomena independently from the position. On top of this, we also consider an additional feature, plosives, which focuses on the pleasantness of sounds per se.
Our experiments show that phonetic features play an important role in the detection of persuasiveness and encode a notion of ``melodious language" that operates both within and across datasets.
%\textcolor{red}{I still think we should remove this sentence} In particular, we use methods that can assess our hypotheses and test whether there exists
%Our analysis of persuasive sentences suggests a framework by which the impact of text in a range of different domains could be investigated.}
\section{Related Work}\label{relatedWork}
%While many semantic and stylistic features have been addressed in persuasive language from a computational point of view, the persuasive effect of sounds has not been systematically explored to the best of our knowledge.
In the following, we first revise some %\textcolor{red}{corpus based}
NLP studies addressing linguistic features of successful communication. Then, we summarize a selection of studies devoted to the effects of phonetics on persuasion.
\subsection{NLP studies on persuasion}
\newcite{virality} focus on a particular form of persuasion by using New York Times articles to examine the relationship between
virality (i.e., the tendency of a content to be circulated on the Web) and emotions evoked by the content. They conduct semi-automated sentiment analysis to quantify the affectivity
and emotionality of each article. Results suggest a strong relationship between affect and virality, in this case
measured as the count of how many people emailed each article. As suggested by the authors, this metric represents a form of ``narrowcasting", as opposed to other ``broadcasting" actions such as sharing on Twitter.
Another line of research investigates the impact of various textual features on audience reactions. The work by \newcite{marco:carlo:gozde:ICWSM-11} correlates several viral
phenomena with the wording of a post, while \newcite{guerini2012linguistic} show that features such as the readability level of an abstract influence the number of downloads,
bookmarking and citations.
A particular approach to content virality is presented by \newcite{simmons2011memes},
%and takes the perspective of understanding
who explore the impact of different types of modification on memes spreading from one person to another.
\newcite{danescu2012you} measure a different ingredient of persuasion by analyzing the features of a movie quote that make it ``memorable". They compile a corpus consisting of memorable and non-memorable movie quote pairs and conduct a detailed analysis to investigate the lexical and syntactic differences between these pairs.
\newcite{louis2013makes} focus on influential science articles in newspapers by considering characteristics such as readability, description vividness, use of unusual words %(\textbf{phonemes perplexity??})
and affective content. High quality articles (NYT articles appearing in ``The Best American Science Writing" anthology) are compared against typical NYT articles.
\newcite{borghol2012untold} investigate how differences in textual description affect the spread of content-controlled videos.
\newcite{lakkaraju2013s} focus on the act of resubmissions (i.e., content that is submitted multiple times with multiple titles to multiple different communities) to understand the extent to which each factor influences the success of a content. \newcite{tan2014effect} consider how content spreads in an on-line community by pinpointing the effect of wording in terms of content informativeness, generality and affect.
\newcite{althoff2014ask} develop a model that can predict the success of requests for a free pizza gifted from the Reddit community. %(\textcolor{red}{it needs more explanation, are they giving it based on the content of the requests??}).
The authors consider high-level textual features such as politeness, reciprocity, narrative and gratitude.
%\textbf{The idea of using carefully controlled experiments
%to study effective communication strategies dates
%back at least to Hovland et al. (1953). Recent
%studies range from examining what characteristics
%of New York Times articles correlate with high
%re-sharing rates (Milkman and Berger, 2012)
%Predicting the “success”
%of various texts such as novels and movie quotes
%has been the aim of additional prior work not already
%mentioned in x1 (Ashok et al., 2013; Louis
%and Nenkova, 2013; Danescu-Niculescu-Mizil et
%al., 2012; Pitler and Nenkova, 2008; McIntyre and
%Lapata, 2009). To our knowledge, there have been
%no large-scale studies exploring wording effects in
%a both topic- and author-controlled setting. Employing
%such controls, we find that predicting the
%more effective alternative wording is much harder
%than the previously well-studied problem of pre-
%}
\subsection{Studies on the effects of phonetics} %on persuasion
%\textcolor{red}{One line of introduction could be added}\\
\newcite{doi:10.1080/10926488.2013.797728} states that alliteration and rhyme can be considered as attention-seeking devices as they enhance emphasis. The author also suggests that they are useful for acceptability and long-term retention of original expressions, decrypting their meanings, indicating informality, and breaking the ice between an audience and a speaker. Therefore, these devices are commonly used in original metaphorical and metonymical compounds.
According to \newcite{leech}, phonetic devices such as rhyme and alliteration are systematically exploited by advertisers to achieve
memorability. Similarly, \newcite{wales} underlines the effectiveness of alliteration and rhyme on emphasis and memorability of an expression.
The relation between the usage of plosives (i.e., consonants in which the vocal tract is blocked so that all airflow ceases, such as ``p", ``t" or ``k") and memorability has also been investigated. According to the study carried out by \newcite{soundAdvice} brand names starting with plosive sounds are recalled and recognized more than the ones starting with other sounds. \newcite{ozbal:brand} carry out an analysis of brand names and discover that plosives are very commonly used.
\newcite{danescu2012you}, whom we previously mentioned, carry out an auxiliary analysis and observe the differences in letter and sound distribution (e.g. usage of labials or front vowels, back sounds, coordinating conjunctions) of memorable and non-memorable quotes.
\newcite{BrainSup} propose a phonetic scorer for creative sentence generation such that generated sentences can contain various phonetic features including alliteration, rhyme and plosive sounds. The authors evaluate the proposed model on automatic slogan generation. In a more recent work \cite{P14-2058}, they enforce the existence of these features in the sentences that are automatically generated for second language learning to introduce hooks to echoic memory.
\section{Phonetic Scorer}
\label{PhoneticScorer}
For the design of the phonetic features, we were mostly inspired by the work of \newcite{BrainSup}, who built and used three phonetic scorers for creative sentence generation. Similarly to this work, all the phonetic features that we used are based on the phonetic representation of English words of the Carnegie Mellon University pronouncing dictionary\footnote{The CMU pronunciation dictionary is freely available at \url{http://www.speech.cs.cmu.edu/cgi-bin/cmudict}. We have used version 0.7a in our implementation.}. We selected four classes of phonetic devices, namely plosives, alliteration, rhyme and homogeneity, which can easily be modeled by observing the distribution of specific classes of phonemes within the sentence.
The \emph{plosive} score is calculated as the ratio of the number of plosive sounds in a sentence to the overall number of phonemes. For both \emph{alliteration} and \emph{rhyme} scorers, we provide a na\"{\i}ve implementation that does not consider stresses or syllables, but only counts the number of repeated sounds at the beginning or end of words in the sentence. The alliteration score is calculated as the number of repeated phonetic prefixes in a sentence normalized by the total number of phonemes. Similarly, the \emph{rhyme} score is calculated as the ratio of the number of repeated phonetic endings in a sentence to the total number of phonemes. Lastly, the homogeneity scorer simply calculates the degree of homogeneity in terms of phonemes used in a sentence independently from their positions. If we let $d_{\textrm{ph}}$ be the count of distinct phonemes and $t_{\textrm{ph}}$ be the total count of phonemes in a sentence, then the homogeneity score is calculated as $1 - (d_{\textrm{ph}} / t_{\textrm{ph}})$.
\section{Dataset}
\label{DS}
In this section, we describe the four datasets we used to conduct our analyses and experiments.
As we mentioned previously, the definition of persuasion is a debated topic and it can comprise distinct strategies or facets. For this reason, we experimented with datasets where at least one ingredient is clearly in the equation.
To explore the effects of wording and euphonics on persuasion, the datasets were built in a controlled setting (topic, author, sentence length) to avoid confounding factors such as author or topic popularity, by following the procedure described in \cite{danescu2012you,tan2014effect}. In addition, these datasets comprise short texts (mostly single sentences) to focus on surface realization of persuasion, where strategic planning -- which might act as a confounding factor -- plays a minor role.
The idea of using controlled experiments (usually in an A/B test setting) to study persuasive communication can be traced back at least to Hovland et al. (1953). %\textcolor{red}{Since our Phonetic Scorer is also dependent on sentence length we tried to control this variable as well.}
While two of these datasets (Twitter and Movies) were already available, the other two (CORPS and Slogans) were collected by following the methodology proposed in the first two as closely as possible\footnote{CORPS and Slogans datasets can be downloaded at the following link: \url{https://github.com/marcoguerini/paired_datasets_for_persuasion/}}.
All datasets are built around the core idea of collecting pairs consisting of a persuasive sentence ($P$) and a non-persuasive counterpart ($\neg P$), where $P$ and $\neg P$ are structurally very similar and controlled for the above mentioned confounding factors.
\textbf{Twitter}. A set of 11,404 %topic- and author-controlled
tweet pairs, where each pair comes from the same user (author control) and contains the same URL (topic control).
%(written by the same author and having the same URL in the text), where
$P$ and $\neg P$ are determined based on their retweet counts \cite{tan2014effect}.
It is worth noting that in our experiments we were able to collect only 11,019 of such tweet pairs since some of them were deleted in the meanwhile. %\textcolor{red}{, and we kept only those pairs for which both tweets were available.}
\textbf{Movie}. A set of 2,198 single-sentence memorable movie quotes ($P$) paired with non-memorable quotes ($\neg P$). For each $P$, the dataset contains a contrasting %(single-sentence)
quote $\neg P$ from the same movie such that %(either before or after it),
(i) $P$ and $\neg P$ are uttered by the same speaker, (ii) $P$ and $\neg P$ have the same number of words, (iii) $\neg P$ does not occur in the IMDb list of memorable quotes and (iv) $P$ and $\neg P$ are as close as possible to each other in the script %for the movie -- either as a single line or as part of a larger block --
\cite{danescu2012you}.
\textbf{\textsc{CORPS}}. A set of 2,600 sentence pairs uttered by various politicians. We collected these pairs from CORPS, a freely available corpus of political speeches tagged with audience reactions \cite{guerini2013new}. The methodology that we used to build the pairs is very similar to \newcite{danescu2012you}: for each $P$, where $P$ is the sentence preceding an audience reaction (e.g. \texttt{APPLAUSE}, \texttt{LAUGHTER}), we selected a contrasting single-sentence $\neg P$ from the same speech. We required $\neg P$ to be close to $P$ in the speech transcription, subject to the conditions that (i) $P$ and $\neg P$ are uttered by the same speaker - which is trivial since these are monologues, where a single speaker is addressing the audience - (ii) $P$ and $\neg P$ have the same number of words, and (iii) $\neg P$ is 5 to 15 sentences away from $P$. This last condition had to be imposed since, differently from movie quotes, we do not have the evidence of which fragment of the speech exactly provoked the audience reaction (i.e. it could be the combination of more than one sentence).
\textbf{Slogan}. A set of 1,533 slogans taken from online resources %Slogan \textcolor{red}{(CIT.)}
paired with non-slogans that are similar in content.
We collected the non-slogans from the subset of the New York Times articles in English GigaWord -- 5th Edition -- released by Linguistic Data Consortium
(LDC)\footnote{\url{http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2011T07}}. For each slogan, we picked the
most similar sentence in the New York Times articles having the same length and the highest LSA similarity \cite{Deerwester90indexingby} with
the slogan. The LSA similarity approach that we used to collect the non-slogans is very similar to the approach used by
\newcite{louis2013makes} to collect the non-persuasive counterparts of successful news articles.
In Table \ref{tab:dataset_charact}, we sum up the criteria used in the construction of each dataset. As can be observed from the table, each dataset satisfies at least two of the three criteria described above.
\begin{table} [!htb]
% \resizebox{\linewidth}{!}{
\centering
\footnotesize
\begin{tabular}{lcccrr}
\toprule
& \multicolumn{3}{c}{Criterion} & \multicolumn{2}{c}{Length}\\
DATASET & Author & Length & Topic & \multicolumn{1}{c}{$P$} & \multicolumn{1}{c}{$\neg P$} \\
\cmidrule(r){1-1} \cmidrule(r){2-4} \cmidrule(r){5-6}
\textsc{CORPS} &\cmark & \cmark & \xmark & 14.0 & 14.0 \\
Movie &\cmark & \cmark & \xmark & 9.7 & 9.7\\
Slogan & \xmark & \cmark & \cmark & 5.0 & 5.0 \\
Twitter &\cmark & \xmark & \cmark & 16.2 & 15.4 \\
\bottomrule
\end{tabular}
% }
\caption{Criteria used in the construction of each dataset and average token length of persuasive and non-persuasive pairs}
\label{tab:dataset_charact}
\end{table}
%In order to define a gold reference of how a euphonic sentence should be, we also collected a set of 534 \textbf{Tongue Twisters}. This will be used in the quantitative analysis to define an empirical threshold above which we can say that an example is ``melodious".
In the last two columns of the table, we also provide the average token length of the persuasive and non-persuasive sentences in each dataset. Finally, in Table~\ref{tab:dataset_examples} we provide examples of euphonic and persuasive sentences for each dataset together with their phonetic scores.
\begin{table*} [!htb]
% \resizebox{\linewidth}{!}{
\centering
\scriptsize
\begin{tabular}{llrrrr}
\toprule
%Dataset & Example & Rhyme & Alliteration & Plosive & Homogeneity\\
Dataset & Example & Rhyme & Alliteration & Plosive & Homogeneity\\
\midrule
%Slogan\\
%\midrule
%- Rhyme
%Movies\\
%\midrule
%-rhyme
%-Rhyme
%\parbox[t]{3.0cm}{I think we can do better and I think we must do better.}&0.789 & 0.737 & 0.342 & 0.631\\
\multirow{2}{*}{CORPS} & I think we can do better and I think we must do better. & 0.789 & 0.737 & 0.342 & 0.631\\
%-Alliteration
%We will stay on offens and stay in the fight until the fight is won.\\
& It will be waged with determination and it will be waged until we win. & 0.566 & 0.679 & 0.189 & 0.736\\
%It will be waged with determination and it will be waged until we win. & 0.566 & 0.679 & 0.189 & 0.736\\
\midrule
\multirow{3}{*}{Movie}& The night time is the right time. & 0.818 & 0.545 & 0.181 & 0.636\\
%-Alliteration
& Beautiful.... beautiful butterfly... & 0.667 & 0.708 & 0.250 & 0.583\\
%& It wasn't logic -- it was love. & 0.555 & 0.778 & 0.166 & 0.500\\
%- Plosives
& Dog eat dog, brother. & 0.533 & 0.533 & 0.400 & 0.400\\
%\midrule
%CORPS\\
\midrule
\multirow{3}{*}{Slogan} &Different Stores, Different Stories. & 0.621 & 0.896 & 0.207 & 0.690\\
%& I like Ike & 0.750 & 0.000 & 0.250 & 0.375\\
%Go well. Go Shell\\
& Why ask why? Try Bud Dry & 0.625 & 0.625 & 0.312 & 0.437\\
%- Alliteration
&Live, Love, Life. & 0.818 & 0.909 & 0.0 & 0.636\\
%&Different Stores, Different Stories. & 0.621 & 0.896 & 0.207 & 0.690\\
%-Plosives
%& Got Guv & 0.0 & 0.0 & 0.667 & 0.0\\
%\midrule
%Twitter\\
\midrule
%- Rhyme
\multirow{2}{*}{Twitter} & A Nerd in Need is a Nerd indeed. & 0.636 & 0.727 & 0.227 & 0.681\\
%- Alliteration
%It is not what we know that will make a difference. It is what we do with what we know\\
%- Plosives
& Easter cupcake baking!! & 0.0 & 0.0 & 0.412 & 0.470\\
\bottomrule
\end{tabular}
% }
\caption{Euphonic examples of persuasive sentences from each dataset, along with their phonetic scores.}
\label{tab:dataset_examples}
\end{table*}
% \begin{table} [!htb]
% \begin{center}
% \footnotesize
% \begin{tabular}{lc}
% \toprule
% DATASET & Average Tokens \\
% \midrule
% %Tongue Twisters (cleaned) & 16.5 \\
% CORPS & 14.0 \\
% Movie$_{p}$ & 12.4 \\
% Movie (memorable) & 9.7 \\
% Movie (non-memorable) & 9.7 \\
% Slogan & 5.0 \\
% Twitter (non-persuasive, raw) & 15.4 \\
% Twitter (persuasive, raw) & 16.2 \\
% \bottomrule
% \end{tabular}
% \end{center}
% \caption{Average length of samples}
% \label{tab:dataset_length}
% \end{table}
\section{Data Analysis} \label{sec:quantitative}
%\textcolor{red}{it's very long, what about using subsections?}
\begin{table*}[t]
\centering
\scriptsize
%\resizebox{\linewidth}{!}{
\begin{tabular}{lllllllll}
\toprule
& \multicolumn{2}{c}{Rhyme}&\multicolumn{2}{c}{Alliteration} & \multicolumn{2}{c}{Plosive} & \multicolumn{2}{c}{Homogeneity}\\
Dataset & \multicolumn{1}{c}{\emph{$\mu$}} & \multicolumn{1}{c}{\emph{$\sigma$}}
& \multicolumn{1}{c}{\emph{$\mu$}} & \multicolumn{1}{c}{\emph{$\sigma$}}
& \multicolumn{1}{c}{\emph{$\mu$}} & \multicolumn{1}{c}{\emph{$\sigma$}}
& \multicolumn{1}{c}{\emph{$\mu$}} & \multicolumn{1}{c}{\emph{$\sigma$}}\\
\cmidrule(r){1-1} \cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7} \cmidrule(l){8-9}
%Tongue Twister & 0.547 & 0.310 & 0.584 & 0.287 & 0.203 & 0.129 & 0.683 & 0.159 \\
%\cmidrule(r){1-1} \cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7} \cmidrule(l){8-9}
CORPS$_{\neg P}$& 0.233 & 0.143 & 0.208 & 0.142 & 0.187 & 0.058 & 0.603 & 0.173\\
CORPS$_P$ & 0.245$\dagger$ & 0.152 & 0.223** & 0.154 & 0.194*** & 0.060 & 0.588** & 0.179\\
\cmidrule(r){1-1} \cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7} \cmidrule(l){8-9}
Movie$_{\neg P}$ & 0.196 & 0.143 & 0.167 & 0.142 & 0.191 & 0.073 & 0.485 & 0.155 \\
Movie$_P$ & 0.214* & 0.165 & 0.196*** & 0.164 & 0.185$\dagger$ & 0.067 & 0.526*** & 0.164\\
%Movie (memorable) & 0.192 & 0.153 & 0.171 & 0.150 & 0.182 & 0.069 & 0.496 & 0.142\\
\cmidrule(r){1-1} \cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7} \cmidrule(l){8-9}
Slogan$_{\neg P}$& 0.071 & 0.111 & 0.047 & 0.092 & 0.204 & 0.098 & 0.343 & 0.163\\
Slogan$_P$ & 0.140*** & 0.194 & 0.123*** & 0.185 & 0.189*** & 0.098 & 0.366*** & 0.156\\
\cmidrule(r){1-1} \cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7} \cmidrule(l){8-9}
Twitter$_{\neg P}$& 0.204 & 0.116 & 0.180 & 0.114 & 0.188 & 0.058 & 0.617 & 0.134 \\
Twitter$_P$ & 0.216*** & 0.121 & 0.193*** & 0.120 & 0.185** & 0.055 & 0.636*** & 0.128\\
\bottomrule
\end{tabular}
%}
\caption{Average phonetic scores for our datasets - ***, $p<.001$; **, $p<.01$; *, $p<.05$; $\dagger$, not significant}
\label{tab:dataset_scores}
\end{table*}
To provide a first insight on the data, in Table \ref{tab:dataset_scores} we report the average phonetic scores for each data set (Mann-Whitney U Test is used for statistical significance between $P$ and $\neg P$ samples, with Bonferroni correction to ameliorate issues with multiple comparisons). The results are partially in line with our expectations of the euphony phenomena being more relevant in the persuasive sentences across the datasets.
As can be observed from the table, the average rhyme scores are higher in persuasive sentences and the difference is highly significant for Slogan and Twitter ($p<.001$), slightly significant for Movie quotes ($p<.05$), but not significant for CORPS. The average alliteration scores are again higher in persuasive sentences and all the differences are highly significant in all datasets (apart from CORPS with $p<.01$). Plosives seem not to correlate well with our intuition of persuasiveness and euphony: either there is no significance (movie quotes) or the averages of euphonic scores are higher in the non-persuasive sentences (the difference is highly significant in slogans, and significant in Twitter). The only dataset that meets our expectation is CORPS with a highly significant difference in favor of persuasive sentences. Finally, the average homogeneity scores are significantly ($p<.001$) higher in persuasive sentences in all datasets except CORPS, where the scores of non-persuasive sentences are significantly higher ($p<.01$) than persuasive ones.
Without going into details of cross-dataset comparisons we would like to note that CORPS seems a very peculiar dataset in terms of average scores, as compared to the others. In terms of rhyme and alliteration, the average scores of non-persuasive sentences ($\neg P$) in CORPS are always higher than the persuasive sentences ($P$) in the other datasets ($p<.001$ in all cases), while for homogeneity the same holds apart from Twitter. %For plosives instead, the differences are not significant.
These results may derive from the fact that a political speech is a carefully crafted text -- aimed at influencing the audience -- in its entirety, so also ``non-persuasive" sentences in CORPS are on average more persuasive than in other datasets.
%\textcolor{red}{CORREGGI VALORE Turning to the `discriminative power' of phonetic features we see that Alliteration is the index that better discriminate between $P$ and $\neg P$ sentence pairs across all datasets (1.34 times the score on average) followed by rhyme (1.21), while plosive and homogeneity have much less discriminative power (0.98 and 1.01 respectively).}
%On the contrary, we speculate that in the movie domain a sentence is more likely to be memorable only if it is especially crafted to stand out from its context.
%the memorable sentences in a movie are often especially crafted to stand out from their context.
%punchlines in movies are crafted by the authors to stand out and
%\textcolor{red}{we speculate that only some sentences in a movie are crafted by the authors to stand out, while the vast majority of them is not.}
%\subsection{CCDF Analysis} \label{sec:ccdf}
%We also wanted to understand what is
As a next step, we conducted another analysis on the distribution of ``extreme cases", i.e. sentences that have a very high phonetic score at least in one feature. This analysis derives from the intuition that a euphonic sentence might be recognized as such by humans only if its phonetic scores are above a certain threshold. In fact, sound repetition in a sentence may occur by chance, as in ``I saw \textbf{the} knife in \textbf{the} drawer", and the longer the sentence is, the higher the probability that phonetic scores will be non-zero even in absence of a euphonic effect. Therefore, the average scores for each phonetic device, as reported in Table \ref{tab:dataset_scores}, are only partially informative. % to our analysis. %/ is not informative per se. %\textcolor{red}{I really think we should remove this sentence :) \textbf{Furthermore we believe that phonetics are a device that is rarely used in persuasion $\ldots$}}
Given this premise, % and since phonetic metrics in our datasets \textcolor{red}{follow a power-law-like distribution} thickening toward low phonetic scores,
to evaluate the ``persuasive power" of the phonetic devices taken into account, we compare them in terms of empirical Complementary Cumulative Distribution Functions (CCDFs) of the persuasive/non-persuasive pairs in various datasets.
These functions are commonly used to analyze online social networks in terms of growth in size and activity (see for example \cite{ahn2007,jiang2010,leskovec2008dynamics}) and also for measuring content diffusion, e.g. the number of retweets of a given content \cite{kwak2010}. Here, we use CCDFs to account for the probability $P$ that the score of a phonetic device \emph{d} will be greater than $n$ indicating it with $\hat{F}_{d}(n)$.
%\begingroup
%\small
%\begin{equation}
%\hat{F}(n) = \frac{\text{number of examples with phonetic index} > n}{\text{total number of examples}}
%\end{equation}
%\endgroup
% $$ \hat{F}(n) = \frac{\text{number of posts with virality index} > n}{\text{total number of posts}}$$
For example, the
probability of having a text with more than .75 rhyme score is indicated with $\hat{F}_{rh}(.75) = \mathrm{P}
(\mathrm{\#rhyme}>.75)$.
%In other words, the slower the function decreases, the higher is the probability that an with that feature
%will have a high virality index.
%In the following we use CCDFs to understand the relation between text characteristics and persuasiveness.
To assess whether the CCDFs of the several types of texts we take into account show significant differences, %between the $P$ and $\neg P$ variants,
we use the Kolmogorov-Smirnov goodness-of-fit test, which specifically targets cumulative distribution functions.
In particular, for each phonetic device and dataset, we use a two-tailed Kolmogorov-Smirnov test (again with Bonferroni correction) to test whether the number of examples above the threshold is higher in the persuasive sentences than in their non-persuasive counterparts for that device.
%that feature contains more examples above the threshold in the persuasive sentences than in their non-persuasive counterparts.
\begin{table}[t]
\centering
\scriptsize
% \resizebox{\linewidth}{!}{
\begin{tabular}{lllll}
\toprule
\emph{Dataset} & $\hat{F}_{rh}(t)$ & $\hat{F}_{al}(t)$ & $\hat{F}_{pl}(t)$ & $\hat{F}_{ho}(t)$\\
\midrule
CORPS$_{\neg P}$ & 0.025 & 0.012 & 0.362 & 0.394\\
CORPS$_{P}$ & 0.033\scriptsize{**} & 0.023\scriptsize{**} & 0.415\scriptsize{***} & 0.363\scriptsize{$\dagger$}\\
\midrule
Movie$_{\neg P}$ & 0.018 & 0.011 & 0.397 & 0.092\\
Movie$_{P}$ & 0.041\scriptsize{***} & 0.025\scriptsize{***} & 0.363\scriptsize{$\dagger$} & 0.173\scriptsize{***}\\
\midrule
Slogan$_{\neg P}$ & 0.005 & 0.003 & 0.460 & 0.011\\
Slogan$_{P}$ & 0.055\scriptsize{***} & 0.043\scriptsize{***} & 0.410\scriptsize{***} & 0.018\scriptsize{***}\\
\midrule
Twitter$_{\neg P}$ & 0.006 & 0.003 & 0.385 & 0.377\\
Twitter$_{P}$ & 0.012\scriptsize{***} & 0.008\scriptsize{***} & 0.364\scriptsize{**} & 0.449\scriptsize{***}\\
\bottomrule
\end{tabular}
% }
\caption{Probability of examples above threshold, - ***, $p<.001$; **, $p<.01$; *, $p<.05$; $\dagger$, not significant}
\label{tab:ccdf}
\end{table}
Since we do not have a theoretical way to define such thresholds, we resort to empirically define them by using a specific dataset of euphonic sentences. Even if it might seem reasonable to consider poems as paradigmatic examples of ``euphonic" writing, we discard them as the phonetic devices used in poems %\textcolor{red}{as they rely mainly on rhyme and not on other phonetic devices},
may span across sentences. Instead, we resort to tongue twisters as a gold reference of how a euphonic sentence should be. Accordingly, we collected a set of 534 tongue twisters from various online resources. Then, for each phonetic index we defined our thresholds as the average of the phonetic scores in this data, in particular: $t_{rh} = 0.55$ for rhyme, $t_{al} = 0.58$ for alliteration, $t_{pl} = 0.20$ for plosives and $t_{ho} = 0.68$ for homogeneity.
In Table~\ref{tab:ccdf}, we report the results of our CCDF analysis. %, where the percentage of examples above each index threshold is signaled as $\hat{F}_{index}(t)$.
After analyzing the ``extreme cases", where euphony is granted, we see that the trends found in Table \ref{tab:dataset_scores} on the correlation between persuasiveness and euphony are confirmed and strengthened. The number of persuasive sentences with a rhyme score above threshold is 30\% more than the non-persuasive ones in CORPS, while the difference is 90\% in Twitter\footnote{In the following the ratios are computed on the real values while Table~\ref{tab:ccdf} presents the rounded values.}. The ratio of persuasive sentences above threshold to non-persuasive ones is very high in movies and slogans (more than 2 and 10 respectively). All results are either highly significant or significant. For comparison, in Table \ref{tab:dataset_scores} these differences are not significant for CORPS and only slightly significant ($p<.05$) for movies.
%With regard to rhyme scores, we see that we have 30\% more sentences above threshold in CORPS, twice the examples for movie quotes, ten times more for Slogans and 90\% more cases for tweets.
Concerning alliteration, there are 85\% more cases above threshold in the persuasive sentences of CORPS than the non-persuasive ones. For movie quotes and Twitter, the persuasive sentences above threshold are more than two times as many as the non-persuasive ones, while the ratio is more than 13 for slogans. All results are either highly significant or significant in line with the results of Table \ref{tab:dataset_scores}. Instead, for plosive scores we observe a negative or no correlation with persuasiveness, the only exception being CORPS.
%the results are not so strong and the probability of finding examples with high plosive scores is around 5 to 10\% less for persuasive sentences as compared to non-persuasive ones. The difference is not significant for movies and it is slightly significant for Twitter. These findings suggest that there is either a negative or no correlation between persuasiveness and the use of plosives ().
Regarding homogeneity, for CORPS the difference between persuasive and non-persuasive sentences is not significant (in Table \ref{tab:dataset_scores} it was significantly in favor of non-persuasive sentences), while for the other datasets there is a highly significant difference in favor of persuasive sentences (between 20\% and 80\%). As a whole, these results confirm our intuition that phonetic features play a significant role with respect to persuasiveness. In the next section we will validate this claim by means of prediction experiments.
%\textcolor{red}{Finally, turning to the `discriminative power' of phonetic features we see that also in this case Alliteration is the index that better discriminate between $P$ and $\neg P$ sentence pairs across all datasets (2.32 times the number of examples above the threshold, on average) followed by rhyme (2.22) and homogennity (1.06) while plosive has much less discriminative power (0.97).}
% \begin{figure*}[!htb]
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/corps_rhyme_ccdf.png}
% %\caption{A really Awesome Image}\label{fig:awesome_image1}
% \endminipage\hfill
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/movies_rhyme_ccdf.png}
% \endminipage\hfill
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/Slogan_rhyme_ccdf.png}
% \endminipage
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/tweet_rhyme_ccdf.png}
% \endminipage
% \caption{Rhyme CCDFs for the various datasets}\label{fig:awesome_image3}
% \end{figure*}
% \begin{figure*}[!htb]
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/corps_alliteration_ccdf.png}
% %\caption{A really Awesome Image}\label{fig:awesome_image1}
% \endminipage\hfill
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/movies_alliteration_ccdf.png}
% \endminipage\hfill
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/Slogan_alliteration_ccdf.png}
% \endminipage
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/tweet_alliteration_ccdf.png}
% \endminipage
% \caption{Alliteration CCDFs for the various datasets}\label{fig:awesome_image3}
% \end{figure*}
% \begin{figure*}[!htb]
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/corps_homogeneity_ccdf.png}
% %\caption{A really Awesome Image}\label{fig:awesome_image1}
% \endminipage\hfill
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/movies_homogeneity_ccdf.png}
% \endminipage\hfill
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/Slogan_homogeneity_ccdf.png}
% \endminipage
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/tweet_homogeneity_ccdf.png}
% \endminipage
% \caption{Homogeneity CCDFs for the various datasets}\label{fig:awesome_image3}
% \end{figure*}
% \begin{figure*}[!htb]
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/corps_plosive_ccdf.png}
% %\caption{A really Awesome Image}\label{fig:awesome_image1}
% \endminipage\hfill
% \minipage{0.24\textwidth}
% \includegraphics[width=\linewidth]{./figures/movies_plosive_ccdf.png}
% \endminipage\hfill
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/Slogan_plosive_ccdf.png}
% \endminipage
% \minipage{0.24\textwidth}%
% \includegraphics[width=\linewidth]{./figures/tweet_plosive_ccdf.png}
% \endminipage
% \caption{Plosive CCDFs for the various datasets}\label{fig:awesome_image3}
% \end{figure*}
\section{Prediction Experiments}
In this section, we describe the prediction tasks (both within and across datasets) that we carried out to investigate the impact of the phonetic features on the detection of various forms of persuasiveness. We compare three different sets of features, namely phonetic, n-grams and their combination to understand whether phonetic information can improve the performance of standard lexical approaches.
Similarly to \newcite{danescu2012you} and \newcite{tan2014effect}, we formulate a pairwise classification problem such that given a pair $(s_{1},s_{2})$ consisting of sentences $s_{1}$ and $s_{2}$, the goal is to determine the more persuasive one (i.e., the one on the \emph{left} or \emph{right}). We can consider this as a binary classification task where for each instance (i.e., pair) the possible labels are \emph{left} or \emph{right}.
\subsection{Dataset and preprocessing}
For the prediction experiments, we used the four datasets described in Section~\ref{DS} (i.e., CORPS, Twitter, Slogan and Movie), all of which consist of a persuasive sentence $P$ and its non-persuasive counterpart ($\neg P$) labeled as either \emph{left} or \emph{right}. To make the positions of the sentences in a pair irrelevant (i.e. to provide symmetry), for each instance occurring in the original datasets (e.g., $(s_{1},s_{2})$ with label \emph{left}), we added another instance including the same sentence pair in reverse order (i.e., $(s_{2},s_{1})$ with label \emph{right}). %Therefore, the number of pairs that we use for the prediction experiments is twice as big as the original.
As a preprocessing step, all the sentences were tokenized by using %tokenization module of
Stanford CoreNLP~\cite{manning-EtAl:2014:P14-5}.
\subsection{Classifier and features}
We performed a 10-fold cross-validation on each dataset and experimented with three feature sets by using a Support Vector Machine (SVM) classifier~\cite{svm}. %in the WEKA framework \cite{weka}.
We preferred SVM as a classifier due to its characteristic property to especially perform well on high-dimensional data \cite{Weichselbraun:2011:UGP:2063576.2063729}.
The first feature set consists of the phonetic features (i.e. plosive, alliteration, rhyme and homogeneity scores as detailed in Section~\ref{PhoneticScorer}). The second feature set is a standard bag of word n-grams including unigrams, bigrams and trigrams. All the non-ascii characters, punctuations and numbers were ignored. The URLs and mentions in Twitter data were replaced with tags (i.e. \emph{\_URL\_} and \emph{\_MENTION\_} respectively). In addition, for the unigram features, stop words were filtered out. %with the help of an online resource\footnote{\url{http://www.lextek.com/manuals/onix/stopwords1.html}}.
We did not apply this filtering for bigrams and trigrams to capture longer-range usage patterns such as propositional phrases. The third feature set is simply the union of both phonetic and n-gram features. %find out whether these two feature sets can benefit from each other and increase the performance of the prediction task.
To find the best configuration for each dataset and feature set, we conducted a grid search over the degree of the polynomial kernel (1 or 2) and the number of features to be used (in the range between 1,000 and 20,000). Due to the low dimensionality of the phonetic feature set, feature selection was performed only for the feature sets including n-grams. The selection was performed based on the information gain of each feature.
\begin{table}[!htb]
\centering
%\caption{Evaluation results.}
\resizebox{\linewidth}{!}{
\begin{tabular}{lccc}
\toprule
%& \multicolumn{3}{c}{Accuracy} \\
\emph{Dataset} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} \\
\cmidrule(r){1-1} \cmidrule(l){2-4}
CORPS & 0.589 (-, 1) & $0.733^{***}$ (4k, 1) & $0.736^\dagger\hspace{0.82em}$ (2k, 1) \\
Movie & 0.600 (-, 2) & $0.694^{***}$ (1k, 1) & $0.722^{***}$ (1k, 1) \\
Slogan & 0.700 (-, 2) & $0.826^{***}$ (3k, 1) & $0.883^{***}$ (5k, 1)\\
Twitter & 0.563 (-, 2) & $0.732^{***}$ (5k, 1) & $0.745^{***}$ (4k, 1) \\
\bottomrule
\end{tabular}
}
\caption{Results of the within-dataset experiments.}
\label{tab:withinDomain}
\end{table}
\begin{table*}[!htb]
\centering
\footnotesize
%\caption{Evaluation results.}
%\resizebox{\linewidth}{!}{
\begin{tabular}{lccccc}
\toprule
%& \multicolumn{3}{c}{Accuracy} \\
\emph{Dataset} & \emph{N-Gram} & \emph{N-Gram+Rhyme} & \emph{N-Gram+Plosive} & \emph{N-Gram+Homogeneity} & \emph{N-Gram+Alliteration}\\
\cmidrule(r){1-1} \cmidrule(l){2-6}
CORPS & 0.733 & $0.738^\dagger\hspace{0.82em}$ (3k, 1) & $0.740^\dagger\hspace{0.82em}$ (2k, 1) & $0.738^\dagger\hspace{0.82em}$ (3k, 1) & $0.738^\dagger\hspace{0.82em}$ (2k, 1) \\
Movie & 0.694 & $0.694^\dagger\hspace{0.82em}$ (1k, 1) & $0.692^\dagger\hspace{0.82em}$ (1k, 1) & $0.721^{***}$ (1k, 1) & $0.709^{**}\hspace{0.44em}$ (1k, 1) \\
Slogan & 0.826 & $0.864^{***}$ (2k, 1) & $0.824^\dagger\hspace{0.82em}$ (2k, 1) & $0.867^{***}$ (3k, 1) & $0.859^{***}$ (3k, 1) \\
Twitter & 0.732 & $0.740^{**}\hspace{0.44em}$ (4k, 1) & $0.733^\dagger\hspace{0.82em}$ (4k, 1) & $0.746^{***}$ (4k, 1) & $0.742^{***}$ (4k, 1) \\
\bottomrule
\end{tabular}
% }
\caption{Contribution of the phonetic features.}
\label{tab:phonetic_exp}
\end{table*}
\begin{table*}[!htb]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccccccccc}
\toprule
& \multicolumn{12}{c}{\emph{Test}} \\
& \multicolumn{3}{c}{CORPS}&\multicolumn{3}{c}{Twitter} & \multicolumn{3}{c}{Slogan} & \multicolumn{3}{c}{Movie}\\
\emph{Training} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} & \emph{Phonetic} & \emph{N-Gram} & \emph{All}\\
\cmidrule(r){1-1} \cmidrule(l){2-4} \cmidrule(l){5-7} \cmidrule(l){8-10} \cmidrule(l){11-13}
CORPS & - & - & - & 0.463 & 0.508 & 0.523 & 0.508 & 0.517 & 0.539 & 0.411 & 0.506 & 0.516\\
Twitter & 0.439 & 0.494 & 0.462 & - & - & - & 0.564 & 0.531 & 0.637 & 0.596 & 0.544 & 0.589 \\
Slogan & 0.535 & 0.512 & 0.514 & 0.535 & 0.510 & 0.539 & - & - & - & 0.532 & 0.545 & 0.588 \\
Movie & 0.431 & 0.513 & 0.498 & 0.562 & 0.533 & 0.560 & 0.581 & 0.537 & 0.589 & - & - & - \\
\bottomrule
\end{tabular}
}
\caption{Results of the cross-dataset prediction experiments optimized on the training set.}
\label{tab:performance_crossDom_optimizedOnTr}
\end{table*}
\subsection{Within-dataset experiments}
For this set of experiments, we conducted a 10-fold cross validation on each dataset separately.
In Table~\ref{tab:withinDomain}, for each dataset listed in the first column, in the subsequent columns we report the performance of the best model obtained with 10-fold cross validation using i) only phonetic features (\emph{Phonetic}), ii) only n-grams (\emph{N-Gram}), iii) both phonetic and n-gram features (\emph{All}). As mentioned previously, for each pair $(s_{1},s_{2})$ consisting of sentences $s_{1}$ and $s_{2}$, our dataset contains another pair including the same sentences in reverse order (i.e., $(s_{2},s_{1})$), resulting in a symmetric and balanced dataset. Therefore, classification performance is measured in terms of accuracy (i.e., the percentage of pairs of which labels were correctly predicted). For each accuracy value, we also report in parenthesis the number of features selected and the kernel degree of the corresponding model. While the kernel degree did not make a big difference in the performance, the number of selected features had an important effect on the accuracy of the models. As can be observed from these values, the best performance on all the datasets is achieved with a relatively small number of features. %most models achieve While the second degree kernel performed best for the models using only phonetic feature sets with one exception, the first degree kernel yielded better accuracy for the rest of the models. Concerning feature selection, the number of selected features varied between 1k and 5k to obtain the best models.
Among the values reported in the table, the ones followed by $^{***}$ are significantly different ($p<.001$) from the ones to their left, while $\dagger$ represents no significance, as calculated according to McNemar's test~\cite{McNemar47samplingIndependence}. %To illustrate, for the CORPS dataset, the best model exploiting only n-gram features performs significantly better than the one based on only phonetic features, while it is significantly outperformed by the best model using both features.
For each dataset, the weakest models (i.e. the ones using only the phonetic features in all cases) are still significantly ($p<.001$) more accurate than a random baseline (accuracy = 50\%). %also compared to a baseline that always predicts the sentence on the left as the more persuasive one resulting in an accuracy of 50\% and they all significantly outperform the baseline.
As can be observed from the table, the models using only n-grams significantly outperform the ones only based on phonetic features in all datasets. However, while the phonetic features are not very strong by themselves, their combination with n-grams results in models outperforming the n-gram based models in all cases. The difference is highly significant for all datasets except CORPS, where n-grams alone are sufficient to achieve a good performance. We speculate that the kind of persuasiveness used in political speeches is more dependent on the lexical choices of the speaker and on the use of a specific set of semantically loaded words such as \emph{bless}, \emph{victory}, \emph{God} and \emph{justice} or \emph{military}. This is in line with the work of \newcite{guerini2008corps}, who built a domain specific lexicon to study the persuasive impact of words in political speeches.
We also conducted an additional set of experiments to investigate if some phonetic features stand out among the others, and to find out the contribution and importance of each phonetic feature in isolation. % In this way, we tried to investigate not only if phonetic features are connected to persuasion, but also if there are features that stand out among the others.
To achieve that, for each dataset we conducted a 10-fold cross validation to obtain the best four models containing a single phonetic feature on top of n-gram features (i.e. \emph{N-Gram+Rhyme}, \emph{N-Gram+Plosive}, \emph{N-Gram+Homogeneity} and \emph{N-Gram+Alliteration}). In Table~\ref{tab:phonetic_exp}, we report the accuracy of the n-gram model and these four models for each dataset. Similarly to Table~\ref{tab:withinDomain}, for each accuracy value, we also report in parenthesis the number of features selected and the kernel degree of the corresponding model obtained with grid search. The results demonstrate that homegeneity is the most effective feature when added on top of n-grams, resulting in highly significant improvement against the basic n-gram models in three out of four datasets. Alliteration and rhyme closely follow homogeneity by yielding models that significantly outperform the n-gram models in three and two datasets respectively. Finally, the models containing plosives do not improve over the n-gram models in any of the four datasets. It is worth noting that in CORPS none of the n-gram models enriched with phonetic features improves over the basic n-gram models as in line with the results of the within-dataset experiments reported in Table \ref{tab:withinDomain}.
\subsection{Cross-dataset experiments}
After observing that the combination of phonetic and n-gram features can be effective in the within-dataset prediction experiments, we took a further step and investigated the interaction of the three feature sets across datasets. More specifically, we classified each dataset with the best models (one for each feature set) trained on the other datasets. With these experiments, we investigated the ability of phonetic features to generalize across the different lexicons of the datasets. As we discussed previously, the four datasets represent different forms of persuasiveness. In this respect, the results of the cross-dataset experiments can also be interpreted as a measure of the degree of compatibility among these kinds of persuasiveness.
In Table~\ref{tab:performance_crossDom_optimizedOnTr}, we present the results of the cross-dataset prediction experiments. For each training and test set pair, %the table
we report the accuracy of the best models, one for each feature set, based on cross-validation on the training set. As can be observed from the table, the figures are generally low and various domain adaptation techniques could be employed to improve the results. However, the objective of this evaluation is not to train an optimized cross-domain classifier, but to assess the potential of the feature sets to model different kinds of persuasiveness.
As expected, n-gram features show poor performance due to the lexical and stylistic differences among the datasets. In many cases, the phonetic models outperform the n-gram models, and in several cases the combination of the two feature sets deteriorates the performance of the phonetic features alone. These findings support our hypothesis that phonetic features, due to their generality, have better correlation with different forms of persuasiveness than lexical features. The experiments involving the CORPS dataset, both for training and testing, do not share this behavior. Indeed, when CORPS is used as a training or test dataset, the performance of the models is quite low (very close to or worse than the baseline in many cases) independently from the feature sets. These results suggest that the notion of persuasiveness encoded in this dataset is remarkably different from the others, as previously discussed in the data analysis in Section~\ref{sec:quantitative}. As seen in the within dataset experiments (see Table~\ref{tab:withinDomain}), CORPS is the only dataset in which the combination of lexical and phonetic features do not improve the classification accuracy. This explains the inability of the phonetic features to improve the accuracy in cross-dataset experiments when this dataset is employed.
% \begin{table*}[h]
% \centering
% \resizebox{\linewidth}{!}{
% \begin{tabular}{lcccccccccccc}
% \toprule
% & \multicolumn{12}{c}{\emph{Test}} \\
% & \multicolumn{3}{c}{CORPS}&\multicolumn{3}{c}{Twitter} & \multicolumn{3}{c}{Slogan} & \multicolumn{3}{c}{Movie}\\
% \emph{Training} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} & \emph{Phonetic} & \emph{N-Gram} & \emph{All} & \emph{Phonetic} & \emph{N-Gram} & \emph{All}\\
% \cmidrule(r){1-1} \cmidrule(l){2-4} \cmidrule(l){5-7} \cmidrule(l){8-10} \cmidrule(l){11-13}
% CORPS & - & - & - & 0.463 & 0.511 & 0.533 & 0.508 & 0.526 & 0.641 & 0.411 & 0.509 & 0.520\\
% Twitter & 0.459 & 0.499 & 0.480 & - & - & - & 0.654 & 0.539 & 0.640 & 0.596 & 0.548 & 0.598 \\
% Slogan & 0.535 & 0.519 & 0.526 & 0.544 & 0.517 & 0.548 & - & - & - & 0.544 & 0.551 & 0.591 \\
% Movie & 0.431 & 0.522 & 0.500 & 0.562 & 0.533 & 0.560 & 0.607 & 0.560 & 0.602 & - & - & - \\
% \bottomrule
% \end{tabular}
% }
% \caption{Results of the cross-dataset prediction experiments optimized on the test set.}
% \label{tab:performance_crossDom_optimizedOnTe}
% \end{table*}
\section{Conclusion}
%\emph{Another possible title?} \textcolor{red}{Happy Endings are Beginnings}}
In this paper, we focused on the impact of a set of phonetic features -- namely rhyme, alliteration, homogeneity and plosives -- on various forms of persuasiveness including memorability of slogans and movie quotes, re-tweet counts of tweets, and effectiveness of political speeches. We conducted our analysis and experiments on four datasets comprising pairs of a persuasive sentence and a non-persuasive counterpart. %While the pairs for two of these datasets (movies and Twitter) were already available, we built the pairs for the other two (Slogans and political speeches) in a similar methodology to the others.
Our data analysis shows that persuasive sentences are generally euphonic. %confirmed the consistent positive role of phonetic features on persuasion (apart from plosives) both in terms of average scores and presence of highly euphonic sentences.
%We carried out both within and across dataset prediction experiments to measure the importance of phonetic features to model persuasiveness. In within-dataset experiments,
This finding is confirmed by the prediction experiments, in which we observed that phonetic features consistently help in the detection of persuasiveness. When combined with lexical features, they help improving classification performance on three of the four datasets that we considered. The key role played by phonetic features is further underlined by the cross-dataset experiments, in which we observed that phonetic features alone generally outperform the lexical features. %The results obtained with phonetic features in both sets of experiments suggest that the notion of persuasiveness encoded by the CORPS dataset is more dependent on lexical choices than euphonic attributes.
To the best of our knowledge, this is the first systematic analysis of the impact of phonetic features on several types of persuasiveness. Our results should encourage researchers dealing with different aspects of persuasiveness to consider the inclusion of phonetic attributes in their models.
%We believe that our results should encourage researchers dealing with aspects of persuasiveness to consider the inclusion of phonetic attributes in their models.
%we observed that phonetic features consistently improve the detection of persuasiveness when combined with lexical features on three of the four datasets. The key role played by phonetic features is further confirmed by the cross-dataset experiments, in which we observed that phonetic features alone generally outperform the lexical features, and in some cases the combination of the two feature sets. The results obtained with phonetic features in both sets of experiments suggest that the notion of persuasiveness encoded by the CORPS dataset is more dependent on lexical choices than euphonic attributes. To the best of our knowledge, this is the first systematic analysis of the impact of phonetic features on several types of persuasiveness. We believe that our results should encourage researchers dealing with aspects of persuasiveness to consider the inclusion of phonetic attributes in their models.
As future work, we will investigate the impact of other phonetic devices such as assonance, consonance and rhythm on persuasiveness. It would also be interesting to focus on the connection between sound symbolism and persuasiveness, and investigate how the context or domain of persuasive statements interacts with the sounds in those statements.
\vspace{0.50em}
%\noindent We would like to conclude this paper with the most favorited and retweeted tweet of @NAACL2015, which is a good example of the positive effect of euphony in persuasiveness:
\noindent We would like to conclude this paper with the most favorite and retweeted tweet of @NAACL2015 (the Twitter account of the conference whose proceedings comprise this paper), which is a good example of the positive effect of euphony in persuasiveness:
\begin{quote}
\center
\small
\emph{The deadline for @NAACL2015 paper submissions is approaching: \\ Remember, remember, the 4th of December!}
\end{quote}
%\section*{Acknowledgments}
%The acknowledgments should go immediately before the references. Do not number the acknowledgments section. Do not include this section when submitting your paper for review.
% include your own bib file like this:
\section*{Acknowledgments}
This work has been partially supported by the Trento RISE PerTe project.
\bibliographystyle{naaclhlt2015}
\bibliography{Persuasive}
\end{document}
\begin{thebibliography}{}
\bibitem[\protect\citename{Ahn \bgroup et al.\egroup }2007]{ahn2007}
Yong-Yeol Ahn, Seungyeop Han, Haewoon Kwak, Sue Moon, and Hawoong Jeong.
\newblock 2007.
\newblock Analysis of topological characteristics of huge online social
networking services.
\newblock In {\em Proceedings of the 16th international conference on World
Wide Web}, pages 835--844. ACM.
\bibitem[\protect\citename{Althoff \bgroup et al.\egroup }2014]{althoff2014ask}
Tim Althoff, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky.
\newblock 2014.
\newblock How to ask for a favor: A case study on the success of altruistic
requests.
\newblock {\em Proocedings of ICWSM}.
\bibitem[\protect\citename{Benczes}2013]{doi:10.1080/10926488.2013.797728}
Réka Benczes.
\newblock 2013.
\newblock The role of alliteration and rhyme in novel metaphorical and
metonymical compounds.
\newblock {\em Metaphor and Symbol}, 28(3):167--184.
\bibitem[\protect\citename{Berger and Milkman}2009]{virality}
Jonah~A. Berger and Katherine~L. Milkman.
\newblock 2009.
\newblock {Social Transmission, Emotion, and the Virality of Online Content}.
\newblock {\em Social Science Research Network Working Paper Series}, December.
\bibitem[\protect\citename{Bergh \bgroup et al.\egroup }1984]{soundAdvice}
Bruce G.~Vanden Bergh, Janay Collins, Myrna Schultz, and Keith Adler.
\newblock 1984.
\newblock Sound advice on brand names.
\newblock {\em Journalism Quarterly}, 61(4):835, dec.
\bibitem[\protect\citename{Bernardi \bgroup et al.\egroup
}2006]{bernardi2006cardiovascular}
Luciano Bernardi, Cesare Porta, and Peter Sleight.
\newblock 2006.
\newblock Cardiovascular, cerebrovascular, and respiratory changes induced by
different types of music in musicians and non-musicians: the importance of
silence.
\newblock {\em Heart}, 92(4):445--452.
\bibitem[\protect\citename{Borghol \bgroup et al.\egroup
}2012]{borghol2012untold}
Youmna Borghol, Sebastien Ardon, Niklas Carlsson, Derek Eager, and Anirban
Mahanti.
\newblock 2012.
\newblock The untold story of the clones: Content-agnostic factors that impact
youtube video popularity.
\newblock In {\em Proceedings of the 18th ACM SIGKDD international conference
on Knowledge discovery and data mining}, pages 1186--1194. ACM.
\bibitem[\protect\citename{Chaiken}1980]{Chaiken1980}
Shelly Chaiken.
\newblock 1980.
\newblock Heuristic vs. sistematic information processing and the use of source
vs message cues in persuasion.
\newblock {\em Journal of Personality and Social Psychology}, 39:752--766.
\bibitem[\protect\citename{Cialdini}1993]{cial93}
Robert~B. Cialdini.
\newblock 1993.
\newblock {\em Influence. The psychology of persuasion}.
\newblock William Morrow \& Company, Inc., New York.
\bibitem[\protect\citename{Cortes and Vapnik}1995]{svm}
Corinna Cortes and Vladimir Vapnik.
\newblock 1995.
\newblock Support-vector networks.
\newblock {\em Mach. Learn.}, 20(3):273--297, September.
\bibitem[\protect\citename{Danescu-Niculescu-Mizil \bgroup et al.\egroup
}2012]{danescu2012you}
Cristian Danescu-Niculescu-Mizil, Justin Cheng, Jon Kleinberg, and Lillian Lee.
\newblock 2012.
\newblock You had me at hello: How phrasing affects memorability.
\newblock In {\em Proceedings of the ACL}.
\bibitem[\protect\citename{Deerwester \bgroup et al.\egroup
}1990]{Deerwester90indexingby}
Scott Deerwester, Susan~T. Dumais, George~W. Furnas, Thomas~K. Landauer, and
Richard Harshman.
\newblock 1990.
\newblock Indexing by latent semantic analysis.
\newblock {\em Journal of the American Society for Information Science},
41(6):391--407.
\bibitem[\protect\citename{Guerini \bgroup et al.\egroup
}2008]{guerini2008corps}
Marco Guerini, Carlo Strapparava, and Oliviero Stock.
\newblock 2008.
\newblock Corps: A corpus of tagged political speeches for persuasive
communication processing.
\newblock {\em Journal of Information Technology \& Politics}, 5(1):19--32.
\bibitem[\protect\citename{Guerini \bgroup et al.\egroup
}2011]{marco:carlo:gozde:ICWSM-11}
Marco Guerini, Carlo Strapparava, and G\"ozde \"Ozbal.
\newblock 2011.
\newblock Exploring text virality in social networks.
\newblock In {\em Proceedings of ICWSM-11}, Barcelona, Spain, July.
\bibitem[\protect\citename{Guerini \bgroup et al.\egroup
}2012]{guerini2012linguistic}
Marco Guerini, Alberto Pepe, and Bruno Lepri.
\newblock 2012.
\newblock Do linguistic style and readability of scientific abstracts affect
their virality.
\newblock {\em Proceedings of ICWSM-12}.
\bibitem[\protect\citename{Guerini \bgroup et al.\egroup }2013]{guerini2013new}
Marco Guerini, Danilo Giampiccolo, Giovanni Moretti, Rachele Sprugnoli, and
Carlo Strapparava.
\newblock 2013.
\newblock The new release of corps: A corpus of political speeches annotated
with audience reactions.
\newblock In {\em Multimodal Communication in Political Speech. Shaping Minds
and Social Action}, pages 86--98. Springer.
\bibitem[\protect\citename{Heath and Heath}2007]{heath2007made}
Chip Heath and Dan Heath.
\newblock 2007.
\newblock {\em Made to stick: Why some ideas survive and others die}.
\newblock Random House.
\bibitem[\protect\citename{Jaynes}2000]{jaynes2000origin}
Julian Jaynes.
\newblock 2000.
\newblock {\em The origin of consciousness in the breakdown of the bicameral
mind}.
\newblock Houghton Mifflin Harcourt.
\bibitem[\protect\citename{Jiang \bgroup et al.\egroup }2010]{jiang2010}
Jing Jiang, Christo Wilson, Xiao Wang, Peng Huang, Wenpeng Sha, Yafei Dai, and
Ben~Y Zhao.
\newblock 2010.
\newblock Understanding latent interactions in online social networks.
\newblock In {\em Proceedings of the 10th ACM SIGCOMM conference on Internet
measurement}, pages 369--382. ACM.
\bibitem[\protect\citename{Kwak \bgroup et al.\egroup }2010]{kwak2010}
Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon.
\newblock 2010.
\newblock What is twitter, a social network or a news media?
\newblock In {\em Proceedings of the 19th international conference on World
wide web}, pages 591--600. ACM.
\bibitem[\protect\citename{Lakkaraju \bgroup et al.\egroup
}2013]{lakkaraju2013s}
Himabindu Lakkaraju, Julian~J McAuley, and Jure Leskovec.
\newblock 2013.
\newblock What's in a name? understanding the interplay between titles,
content, and communities in social media.
\newblock In {\em ICWSM}.
\bibitem[\protect\citename{Leech}1966]{leech}
Geoffrey~N. Leech.
\newblock 1966.
\newblock {\em English in advertising : a linguistic study of advertising in
Great Britain / [by] Geoffrey N. Leech}.
\newblock Longmans London.
\bibitem[\protect\citename{Leskovec}2008]{leskovec2008dynamics}
Jurij Leskovec.
\newblock 2008.
\newblock {\em Dynamics of large networks}.
\newblock ProQuest.
\bibitem[\protect\citename{Louis and Nenkova}2013]{louis2013makes}
Annie Louis and Ani Nenkova.
\newblock 2013.
\newblock What makes writing great? first experiments on article quality
prediction in the science journalism domain.
\newblock {\em TACL}, 1:341--352.
\bibitem[\protect\citename{Manning \bgroup et al.\egroup
}2014]{manning-EtAl:2014:P14-5}
Christopher~D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven~J.
Bethard, and David McClosky.
\newblock 2014.
\newblock The {Stanford} {CoreNLP} natural language processing toolkit.
\newblock In {\em Proceedings of 52nd Annual Meeting of the Association for
Computational Linguistics: System Demonstrations}, pages 55--60.
\bibitem[\protect\citename{McNemar}1947]{McNemar47samplingIndependence}
Quinn McNemar.
\newblock 1947.
\newblock Note on the sampling error of the difference between correlated
proportions or percentages.
\newblock {\em Psychometrika}, 12(2):153--157.
\bibitem[\protect\citename{Moulin \bgroup et al.\egroup }2002]{moul02}
Bernard Moulin, Hengameh Irandoust, Micheline Belanger, and Ga{\"e}lle
Desordes.
\newblock 2002.
\newblock Explanation and argumentation capabilities: Towards the creation of
more persuasive agents.
\newblock {\em Artificial Intelligence Review}, 17:169--222.
\bibitem[\protect\citename{{\"O}zbal \bgroup et al.\egroup }2012]{ozbal:brand}
G{\"o}zde {\"O}zbal, Carlo Strapparava, and Marco Guerini.
\newblock 2012.
\newblock Brand pitt: A corpus to explore the art of naming.
\newblock In {\em Proceedings of the Eighth International Conference on
Language Resources and Evaluation (LREC-2012)}. European Language Resources
Association (ELRA).
\bibitem[\protect\citename{\"{O}zbal \bgroup et al.\egroup }2013]{BrainSup}
G\"{o}zde \"{O}zbal, Daniele Pighin, and Carlo Strapparava.
\newblock 2013.
\newblock {BRAINSUP: Brainstorming Support for Creative Sentence Generation}.
\newblock In {\em {Proceedings of the 51st Annual Meeting of the Association
for Computational Linguistics (ACL 2013)}}, pages 1446--1455, Sofia,
Bulgaria, August. Association for Computational Linguistics.
\bibitem[\protect\citename{{\"O}zbal \bgroup et al.\egroup }2014]{P14-2058}
G{\"o}zde {\"O}zbal, Daniele Pighin, and Carlo Strapparava.
\newblock 2014.
\newblock Automation and evaluation of the keyword method for second language
learning.
\newblock In {\em Proceedings of the 52nd Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers)}, pages 352--357.
Association for Computational Linguistics.
\bibitem[\protect\citename{Perelman and
Olbrechts-Tyteca}1969]{perelman:olbrechts:69}
Chaim Perelman and Lucie Olbrechts-Tyteca.
\newblock 1969.
\newblock {\em The new Rhetoric: a treatise on Argumentation}.
\newblock Notre Dame Press.
\bibitem[\protect\citename{Petty and Cacioppo}1986]{Petty1986}
Richard~E. Petty and John~T. Cacioppo.
\newblock 1986.
\newblock The elaboration likelihood model of persuasion.
\newblock {\em Advances in Experimental Social Psychology}, 19:123--205.
\bibitem[\protect\citename{Sammler \bgroup et al.\egroup }2013]{sammler2013co}
Daniela Sammler, Stefan Koelsch, Tonio Ball, Armin Brandt, Maren Grigutsch,
Hans-J{\"u}rgen Huppertz, Thomas~R Kn{\"o}sche, J{\"o}rg Wellmer, Guido
Widman, Christian~E Elger, et~al.
\newblock 2013.
\newblock Co-localizing linguistic and musical syntax with intracranial eeg.
\newblock {\em NeuroImage}, 64:134--146.
\bibitem[\protect\citename{Simmons \bgroup et al.\egroup
}2011]{simmons2011memes}
Matthew Simmons, Lada~A Adamic, and Eytan Adar.
\newblock 2011.
\newblock Memes online: Extracted, subtracted, injected, and recollected.
\newblock {\em Proceedings of ICWSM-11}.
\bibitem[\protect\citename{Tan \bgroup et al.\egroup }2014]{tan2014effect}
Chenhao Tan, Lillian Lee, and Bo~Pang.
\newblock 2014.
\newblock The effect of wording on message propagation: Topic-and
author-controlled natural experiments on twitter.
\newblock {\em ACL}.
\bibitem[\protect\citename{Toulmin}1958]{Toulmin:58}
Stephen Toulmin.
\newblock 1958.
\newblock {\em The Use of Arguments}.
\newblock Cambridge University Press, Cambridge MA.
\bibitem[\protect\citename{Wales}2001]{wales}
Katie Wales.
\newblock 2001.
\newblock {\em A dictionary of stylistics}.
\newblock Harlow, Eng. ; New York : Longman, 2nd ed edition.
\newblock Includes bibliographical references (p. 413-429).
\bibitem[\protect\citename{Walton}1996]{walton:96}
Douglas~N. Walton.
\newblock 1996.
\newblock {\em Argumentation Schemes for Presumptive Reasoning}.
\newblock Lawrence Erlbaum Associates, Mahwah, New Jersey.
\bibitem[\protect\citename{Weichselbraun \bgroup et al.\egroup
}2011]{Weichselbraun:2011:UGP:2063576.2063729}
Albert Weichselbraun, Stefan Gindl, and Arno Scharl.
\newblock 2011.
\newblock Using games with a purpose and bootstrapping to create
domain-specific sentiment lexicons.
\newblock In {\em Proceedings of the 20th ACM International Conference on
Information and Knowledge Management}, CIKM '11, pages 1053--1060, New York,
NY, USA. ACM.
\end{thebibliography}