Meme Similarity and Emotion Detection using Multimodal Analysis
Extracted source text · arXiv 2503.17493
\documentclass[conference]{IEEEtran}
\ifCLASSINFOpdf
\else
\fi
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage[margin=1in]{geometry}
\usepackage[dvipsnames]{xcolor}
\usepackage{inputenc}
\bibliographystyle{IEEEtran}
\usepackage{amsmath}
\usepackage{xcolor}
\usepackage{booktabs}
\usepackage{comment}
\usepackage{algorithm}
\usepackage{algpseudocode}
\begin{document}
\title{Meme Similarity and Emotion Detection using Multimodal Analysis}
% \author{\IEEEauthorblockN{Konyspay Aidos}
% \IEEEauthorblockA{Faculty of Information\\ Technology\\
% Almaty, Kazakhstan\\
% Email: a\_konyspay@kbtu.kz}
% }
\author{\IEEEauthorblockN{Aidos Konyspay,
Pakizar Shamoi \IEEEauthorrefmark{1}, Malika Ziyada, Zhusup Smambayev}
\IEEEauthorblockA{School of Information Technology and Engineering \\
Kazakh-British Technical University\\
Almaty, Kazakhstan\\
Email:
\IEEEauthorrefmark{1}p.shamoi@kbtu.kz
% \IEEEauthorrefmark{2}ar\_karatayev@kbtu.kz,
% \IEEEauthorrefmark{3}an\_ogorodova@kbtu.kz,
}
}
\maketitle
\IEEEpeerreviewmaketitle
\begin{abstract}
%\colorbox{Dandelion}{1. Proofread}
%\colorbox{Dandelion}{2. Methods sentence, result sentence}
%\colorbox{Dandelion}{21. Title ? }
%\colorbox{LimeGreen}{22. Choose Journal} \\
%\colorbox{Aquamarine}{21. The title could be:}
%\textcolor{Aquamarine}{1) Clustering Internet Memes Using the CLIP Model: A Study of Text-Image Similarity. 2) Exploring Meme Clustering Through Multimodal Similarity Using CLIP.} \\
\begin{comment}{
Internet memes are more than just funny pictures. They're a way people express their feelings and opinions online. But understanding what memes mean isn't easy because they mix pictures and words in complex ways. While scientists have made some progress in analyzing memes, there's still a lot to figure out. Since memes include text and visual representation, it can be difficult for recommendation systems to predict and recommend suitable content based on the user's desires. For example, modern technologies find it difficult to automatically capture all the nuances of memes or understand them by combining both visual and textual content.
%This way, we hope to improve the performance of recommendation systems for memes, including visual and textual representation using multimodal categorization of memes. \\
\textcolor{Aquamarine}{Our study addresses this issue by exploring the use of the CLIP (Contrastive Language-Image Pre-training) model, a state-of-the-art tool designed to understand both visual and textual content simultaneously. We introduce a new method to group similar memes based on their visual and textual content, using a technique that doesn't rely on predefined categories, using unsupervised learning techniques. By doing this, we aim to discover patterns that can help recommendation systems better suggest memes that users will enjoy. Our research not only improves how memes are categorized but also provides a better understanding of how visual and textual information work together in digital media. Our research aims to enhance the accuracy of meme categorization. This way, we hope to improve the performance of recommendation systems for memes, including visual and textual representation using multimodal categorization of memes.}
\colorbox{Lavender}{We need 170-250 words}
\end{comment}
Internet memes are a central element of online culture, blending images and text. While substantial research has focused on either the visual or textual components of memes, little attention has been given to their interplay. This gap raises a key question: What methodology can effectively compare memes and the emotions they elicit? Our study employs a multimodal methodological approach, analyzing both the visual and textual elements of memes. Specifically, we perform a multimodal CLIP (Contrastive Language-Image Pre-training) model for grouping similar memes based on text and visual content embeddings, enabling robust similarity assessments across modalities. Using the Reddit Meme Dataset and Memotion Dataset, we extract low-level visual features and high-level semantic features to identify similar meme pairs. To validate these automated similarity assessments, we conducted a user study with 50 participants, asking them to provide yes/no responses regarding meme similarity and their emotional reactions. The comparison of experimental results with human judgments showed a 67.23\% agreement, suggesting that the computational approach aligns well with human perception. Additionally, we implemented a text-based classifier using the DistilBERT model to categorize memes into one of six basic emotions. The results indicate that anger and joy are the dominant emotions in memes, with motivational memes eliciting stronger emotional responses. This research contributes to the study of multimodal memes, enhancing both language-based and visual approaches to analyzing and improving online visual communication and user experiences. Furthermore, it provides insights for better content moderation strategies in online platforms.
% Our findings advance the understanding of how communication modes influence meme perception.
% Internet memes can be considered as one of the main items of Internet culture, on the other
% hand, they contain both image and text, and sometimes the share of references and subtexts.
% The majority of the work done so far has been done on either the visualization component or the
% textual component while there is little done related to how both of them interact. This leads to a
% key question: What is the procedure that would allow comparing memes and the emotions they
% elicit? Our study can be considered methodologically multi-modal because we analyze both
% the visual and textual elements of memes. Applying a given set of memes from the Reddit forum, we
% extract low-level visual features and high-level semantic features which help in the selection of
% similar meme pairs. To verify the results of these automated similarity assessments we subsequently perform a user study that includes 50 participants. It gives them
% a yes/no answer to whether two memes are similar and they say how they feel. Comparing the
% results obtained in our experiment with the sample of human judgments revealed that the
% agreement between the two findings is about 67.23\% with the difference to mean that the
% computational approach is somewhat close to human perception. We also have a text-based
% classifier that tells which of the six basic emotions we associate with each meme, and
% compares the results to their labels. These results extend
% the literature to understand how communication modes
% influence memes’ perception. The findings contribute not
% only to the study of memes that involve multiple modes of
% communication but also to the language-based and visual
% approaches for analyzing and improving online visual
% communication and user experiences, as well as for making
% better decisions about moderating this content.
% Internet memes are a key element of online culture, combining images and text. They often reference events, shared experiences, or trending topics with hidden meanings. Most of the work so far has focused on either visualization or text, with little exploration of how they interact. This leads to a key question: how can we compare memes and the emotions they evoke? Our study can be considered as methodologically multi-modal because we analyzed both the visual and textual elements of memes. Using Reddit memes, we extracted visual and semantic features to identify similar meme pairs. We validated the similarity results with a user study of 51 participants. Participants provided a yes/no answer on meme similarity and share their feelings. Comparison of our results with human judgments presented 67\% agreement, showing the computational approach aligns somewhat with human perception. We used a text-based classifier to identify which of the six basic emotions is associated with the meme conveys and compared its results to the labels. These results extended the literature in understanding how communication modes influence memes’ perception. The findings contribute to the study of memes that combine multiple modes of communication. Moreover, we could apply language-based and visual methods to analyze and moderate online content, while also improving user experiences.
\begin{comment}
\textcolor{Aquamarine}{
Internet memes can be considered as one of the main items of Internet culture, on the other hand, they contain both image and text, and sometimes the share of references and subtexts.
\end{comment}
\end{abstract}
\section{Introduction}
%\colorbox{Dandelion}{3. Cite initial premises}
%\colorbox{Dandelion}{4. Proofread intro}
%One of the most popular ways to share emotions, ideas, and thoughts are through internet memes.
In today's digital age, Internet memes are one of the most popular ways to share emotions, ideas, and thoughts. For example, people shared more than one million memes daily on the Instagram platform in 2020 \cite{meme}. The authors of memes use text, images, and videos in creative ways to share ideas or jokes. Thus, memes serve as entertainment and stress relief sources while providing social meaning and highlighting important issues.
%{They mix text, images, and videos in clever ways to express ideas or jokes. They often give us a quick laugh and a break from our busy day. But memes aren't just for laughs.}
%{Memes are a unique way of communicating that can cross cultural boundaries and help people feel connected. They're not only fun to share, but they can also be a way for people to express themselves, influence others, and even make political statements. With the rise of the internet, memes have become more visible and widespread, changing and evolving as they spread. They reflect everyday life, just like many forms of art do. For people sharing memes online has become a key way to express themselves. }
Memes have a unique ability to cross cultural boundaries, creating a shared language that many people understand and enjoy \cite{olena2020memes, Grishaeva2021}. As memes spread rapidly across social media, understanding their content and impact has become increasingly important. The meaning of memes can differ between countries and cultures. Analyzing these differences can help improve recommendation algorithms on social media platforms.
%{This cultural impact and widespread use underline the critical need for automated systems that can accurately interpreting and categorizing these memes.}
However, current sentiment analysis tools predominantly focus on textual or visual data independently, often overlooking the complex interplay between text and image that defines memes. This oversight hinders computers' ability to interpret memes in a manner similar to humans, thereby reducing the effectiveness of analyses in recommendation systems.
% This oversight makes it harder for computers to understand memes the same way humans do, limiting the effectiveness of analysis in recommendation systems.
% This oversight hinders the understanding of the nuances of memes, limiting the effectiveness of analysis in recommendation systems.
%{The main idea of this research is to develop a model that will allow us to analyze memes based on both visual and textual representations simultaneously. The main result of this work is using a neural-network-based CLIP model for better meme categorization by their visual and textual content simultaneously.}
This research develops a model to group similar memes based on their visual and textual content and the emotions they evoke. By using CLIP and DistilBERT models, we aim to answer the following questions:
\begin{itemize}
\item How are the visual and textual features of memes related?
How can memes be effectively categorized into groups?
\item What emotions are commonly associated with particular groups of memes?
\end{itemize}
The contributions of this paper are as follows:
\begin{itemize}
\item \textit{Simultaneous Grouping by Content.} The study develops a method to group memes by analyzing both their visual (what you see) and textual (what you read) content at the same time. This helps to better group memes based on their overall message.
%{\item \textit{Understanding Meme Emotions.} The research explores the emotions memes evoke. By identifying emotions, we can better understand the feelings that memes induce. \textcolor{red}{memes are funny but include the range of emotions. For example, the motivational memes can be based on the emotion of angriness or happiness. We need to form this meaning and add to intro, conclusion, contribution. Few (almost zero) works did it before.
\item \textit{Understanding Meme Emotions.} The research explores the emotions memes evoke. By identifying emotions, we can better understand the feelings that memes induce. Memes are funny. However, they contain humor in various forms of emotions. For instance, motivational memes could provoke feelings of joy, anger, or even sadness. This emotional diversity in memes remains underexplored in the existing literature.
% Memes are often perceived as humorous, yet they contain a broad spectrum of emotions. For instance, motivational memes may draw from emotions such as anger or happiness, using them to inspire/motivate in different ways. Since there is neither a clear definition of memes nor a list of feelings they cause, this paper seeks to make a proposal for a natural list of basic emotions that memes produce or provoke. Such a perspective not only enriches the understanding of meme culture but also contributes to creating further research directions on the emotional impact of internet memes.
\item \textit{Expanding the Memotion Dataset. } Our study extends the existing Memotion dataset by incorporating the results of our six-class emotion analysis. Specifically, we classify memes into one of six emotions: sadness, joy, love, anger, fear, or surprise. This new addition is useful in enhancing the dataset with basic sentiments (positive, positive, negative, negative, neutral), humor, sarcasm, offense, and motivation. For example, while the dataset pointed out that something is ‘motivational’, our updated annotations indicate the emotion used; it may be joy, anger, etc. This expansion makes the dataset richer and more useful for future studies.
\end{itemize}
% The inclusion of emotions enables proper analysis and understanding of what actually most memes represent to the public rather than segregating them into several general categories.
% memes are funny but include the range of emotions. For example, the motivational memes can be based on the emotion of angriness or happiness. We need to form this meaning and add to intro, conclusion, contribution. Few (almost zero) works did it before. Memes are often perceived as humorous, yet they contain a broad spectrum of emotions. For instance, motivational memes may draw from emotions such as anger or happiness, using them to inspire/motivate in different ways. This emotional diversity in memes remains underexplored in the existing literature
%\colorbox{LimeGreen}{5. Paper is structured...}
%\colorbox{LimeGreen}{6. The main contributions are and items (multimodal,emotions may be}
The paper is structured as follows. Section I is the Introduction. Section II reviews the Related Work. Section III details the methods used in this study, including data, grouping similar memes, emotion detection in memes, CLIP, and cosine similarity. The results are presented in Section IV. Section V discusses the limitations and future work, followed by the conclusion in Section VI.
\section{Related Work}
%\colorbox{Dandelion}{7. Proofread RW}
%\colorbox{LimeGreen}{8. Add appx 20 papers}
%\colorbox{LimeGreen}{9. 1-2 paragraphs challenges, limitations}
%\colorbox{LimeGreen}{10. Structure of RW-by methods, or text/image/multi}
In our digital age, memes have become a powerful way to share laughs, opinions, and even political stances. But no matter how carefree they may seem, some memes have a negative side. For example, they could contain offensive or harmful content. Researchers use a variety of methods to analyze the appeal of memes and identify those that are not ethically appropriate. %A group of researchers has been diving deep into the world of memes, using advanced computer science to figure out what makes a meme tick, and more importantly, how to spot the ones that cross the line.%
\subsection{
Image-based Analysis of Memes
}
The absence of textual headers in memes makes image-based grouping a challenging task for researchers. For example, a recent study focused on image-based meme classification using convolutional and transformer architectures. The study showed that visual data alone can achieve high accuracy, but meme analysis often depends on cultural context \cite{Gao2024}.
Meme studies mainly focus on images because most memes include visual attributes. A recent study used Deep Convolutional Neural Networks (CNNs) to explore object and visual features in memes. It was the first to show their role in creating memes and recognizing humor \cite{Choi2023}. Similarly, some researchers used YOLOv5 to detect objects in meme images, improving meme categorization based on content \cite{Wang2024}. This is a new direction, which introduced the use of GANs for generating memes and classifying meme emotions \cite{Sharma2022}. This paper reviewed studies on image detection techniques that enhance meme categorization and sentiment analysis.
\subsection{
Text-based Analysis of Memes
}
Recent meme analysis focused on text-based approaches, as key features helped reveal the meme's meaning and cultural context. One study explored language features in contextual memes and used deep learning to examine how slang and humor influence meme valency. This model improved meme text analysis by 33\% using a structured linguistic model \cite{Xu2022}.
Similarly, another study introduced a transformer for sentiment analysis of meme texts, incorporating irony and sarcasm.
They applied their approach to improving sentiment classification in memes, outperforming traditional models \cite{Kim2023}. The study \cite{Bennett2021} used the SS method to detect offensive language in meme captions, contributing to automatic hate speech detection.
Furthermore, the study \cite{Pang2023} extended the analysis of cultural references to meme texts. They discovered that text-based meme analysis uncovers the hidden cultural semiotics of memes. This approach examines memes from a cultural perspective using only textual features. Next, \cite{Rajendran2022} presented the latest meme text summarization system. Their method extracted memes' semantic meaning using natural language processing (NLP) tools, focusing on textual content without considering contextual pictures.
New trends in meme analysis have been developed in connection with the creation of large datasets designed for the training and evaluation of AI systems. The study \cite{Guillermo2020Infotec} presented an analysis of deep learning and text categorization for meme classification via the \textit{SemEval-2020} shared activity.
\subsection{
Multimodal Analysis of Memes
}
A recent trend in meme study is multimodal analysis, which examines both the verbal and visual elements of memes. This enables a richer analysis of memes and the variety of emotions that they can convey. Recent techniques such as CLIP performed well in understanding the combination of text and photographs within memes \cite{Zhang2024Long, Kumar2022Hate}. These methods extended contrastive learning to classify memes using both visual and textual features, increasing the number of correctly classified memes.
Several studies explored how self-supervision and multimodal analysis can handle large volumes of unlabeled memes. For instance, \cite{Tung2023} introduced \textit{SemiMemes}, a semi-supervised model that improved meme classification using deep learning on unlabeled data. This approach adopts a bimodal approach, considering both the image and text as distinct yet related pieces of information. \cite{Alluri2021} analyzed multimodal sentiment-capturing transformers to categorize memes as ironic, humorous, or motivational. Subsequently, advanced large language models (LLMs) improved unsupervised multimodal topic modeling for meme classification \cite{Prakash2023}. Next, \cite{Cui2023} contributed a new contrastive learning approach to meme category detection. This approach used "Be-or-Not" prompts to generate hard negative samples, enhancing the ability to distinguish subtle meme categories.
% this approach improves the performance of multimodal models.
In \cite{Lin2023}, the author tackles the challenge of using LLMs for multimodal reasoning in the detection of dangerous memes. This work proposes new methods to improve harm detection in memes and related datasets. Additionally, \cite{Koutlis2023} introduced \textit{Meme-Tector}, which improves meme detection by combining visual elements and attention mechanisms for better reliability.
Numerous papers examined the topic of meme retrieval and search systems. For example, the \cite{simmeme} proposed a \textit{SimMeme}, a graph-based search engine that uses a graph similarity measure for enhanced meme analysis. Unlike earlier image search methods, it accounts for image similarity, user tags, and meme relationships.
\begin{comment}
One big leap forward has been using both the images and the words in memes to understand them better. This approach, known as multi-modal analysis, helps in not just catching the bad apples but also understanding the vast range of emotions memes can express \cite{Lin2023, Koutlis2023}. Some of the brightest minds have been working on teaching computers to see memes not just as pictures or texts but as the complex combination of both that they truly are \cite{Afridi2020}.
\end{comment}
% \subsection{
% Meme Retrieval and Search Systems
% }
\begin{comment}
Despite these advances, analyzing memes remains challenging. Memes often contain inside jokes and references that only specific groups understand, and their trends change rapidly. Something popular today might be outdated tomorrow. This has led to a focus not only on analyzing memes but also on developing smarter ways to search for them and even predict the next big trend. \cite{simmeme}.
\end{comment}
\subsection{
Meme Sentiment, Emotion, and Toxicity Detection
}
There is an increasing interest in identifying potentially toxic elements of memes because humor, irony, and sarcasm often carry offensive meanings. Extensive research has been conducted using deep learning and multimodal reasoning to address this topic. \cite{Ma2022} have formulated a multi-task learning algorithm combining textual and visual classification to better detect hateful memes. Similarly, \cite{Pimpalkar2022} highlighted the importance of using machine learning to recognize sentiment in visual-textual memes and analyze hidden meanings.
Focus on contextualization of the meme within social media is also crucial. For example, \cite{Joshi2023} used knowledge graphs to track and analyze memes in both Reddit and Discord. \cite{Joshi2023} analyzed the sentiment of memes and their allocation between subreddits or Discord groups.
The problem of toxic memes was addressed in \cite{Qu2022}, where the authors proposed a multimodal contrastive learning approach to analyze hate speech memes. Their work explained how memes can transform and become a source of provocation and sexual harassment in a certain period. Offensive meme detection was explored in works like \cite{Giri2021}, which employed NLP techniques to classify offensive content and assess toxicity levels.
Next, \cite{Gupta2021} applied multimodal sentiment analysis for memes, using self-supervised image in-painting to improve understanding of missing pixels. This work aims to address the growing complexity of interpreting visual-textual relations in memes, particularly negative ones.
Thus, several studies employed an opinion-mining approach to analyze the meme. While \cite{hatzivassiloglou1997predicting}, \cite{hu2004mining}, and \cite{taboada2006methods} analyzed only adjectives for sentiment polarity calculation, the authors in \cite{Amalia2018Meme} considered all parts of speech for the meme sentiment analysis. They applied Optical Character Recognition (OCR) to extract captions from memes and used the Naïve Bayes algorithm to classify the memes as positive or negative.
Recent studies have also expanded multimodal learning by integrating methods like CLIP to enhance image-text classification. For instance, \cite{Burbi2023} used a CLIP vision-language model to improve hateful meme classification by mapping meme semantics to words. Similarly, \cite{Jannat2022} focused on sentiment classification in multimodal Bengali memes, highlighting the challenges of sentiment analysis across languages.
\begin{comment}
\textcolor{red}{Studies in emotion perception show that naturalistic human characteristics in visual representations determine how readers decode emotions potentially affecting how sentiment analysis functions in memes \cite{10652090, Sakai2020, RAMACHANDRA202411, Tazin2024, Michele2024}.}
\end{comment}
The analysis of sentiments in memes depends on how people perceive emotions through visual elements, as reported in \cite{10652090}. Detecting sentiment in memes requires advanced systems because behavioral research demonstrates that multiple cues help users recognize emotions, according to \cite{Sakai2020}. The sentiment analysis of social contexts benefits from fine-tuned models according to research in stress detection and behavioral computing studies \cite{RAMACHANDRA202411, Tazin2024, Michele2024, 10412045}.
% This highlights the ongoing relevance of multimodal analysis in better understanding memes.
\begin{comment}
Figuring out how to automatically extract useful
information from memes is tough, and sometimes the tech just can't keep up with the meme's context or subtleties \cite{Gupta2021}. Despite this, the work goes on, blending different techniques like deep learning and machine learning to get a better grip on meme sentiment analysis and categorization \cite{Pimpalkar2022, Ma2022}.
\end{comment}
\begin{comment}
Some of the most promising work involves learning from a mix of supervised and unsupervised data, helping computers get smarter at understanding memes by exposing them to as many examples as possible \cite{Tung2023}. This approach is kind of like learning from both the textbook and the real world—it gives computers a more well-rounded understanding of memes.
\end{comment}
\begin{comment}
Recent work introduces the meme retrieval system with good generalization based on interaction-based text-matching, meme emotion classification is provided as well \cite{memechat}
\end{comment}
Fine-grained meme understanding seeks to interpret memes across multiple tasks, such as sentiment and offensiveness detection, by capturing interactions between text and image modalities. The proposed Metaphor-aware Multi-modal Multi-task Framework (MF) addresses existing limitations by modeling inter- and intra-modality interactions, as well as task correlations, demonstrating better performance on the MET-Meme dataset compared to current baselines\cite{WANG2024111778}.
The increasing use of memes on social media to express opinions and measure public sentiment highlights the need for models that capture their affective dimensions. Current approaches struggle with generalization due to limited visual-linguistic grounding. Recently proposed ALFRED framework, utilizing the MOOD dataset, demonstrates improved emotion detection in memes through effective cross-modal fusion and emotion-aware visual cue modeling \cite{meme_emotion_new}. Additionally, \cite{memechat} presented a model that integrates memes into general open-domain dialogue systems, specifically for meme selection. As shown in \cite{memechat}, integrating memes into dialogue systems enhances text matching, supporting meme search, and better user engagement.
\subsection{
Cultural and Sociological Perspectives on Memes
}
Memes reflect cultural elements, including morals, politics, and debates. Many researchers have investigated memes' role as a medium for intercultural communication.
\cite{Weng2014Predicting} developed the model to predict meme popularity based on the communities, showing how memes are disseminated and transformed across virtual groups. Moreover, this study examines the dynamics between traditional news topics and social media content, highlighting how topical relevance influences social media popularity. By analyzing New York Times topics alongside image-text memes from Reddit, it finds that topic alignment with news trends boosts meme virality, with early-topic memes showing notable predictive power for popularity using topicality features in a CatBoost classifier \cite{Barnes2024}.
\cite{Ferrara2013Clustering} and \cite{Jesus2020Semantic} explored how content and metadata can enable memetic clustering to find memes by meaning. These approaches help explain the shared subjectivity of memes and their potential to shape culture. \cite{zannettou2018origins} studied memes as cultural phenomena, examining how subcultures create memes to influence broader culture, showing both the positive and negative aspects of memes' role in shaping collective identity.
It is common for political memes to include a great deal of text content; however, most of them are very visually powerful. \cite{Singh2023} studied how political memes could be identified using image-only approaches. They concluded that symbols and color can help classify political meme intent.
\begin{comment}
\textcolor{BrickRed}{
Recent research has focused on analyzing and classifying memes in social media. Studies have explored various approaches to improve our understanding and analysis of memes in social media contexts, including predicting meme popularity using network features \cite{Weng2014Predicting}, clustering memes based on content and metadata \cite{Ferrara2013Clustering}, and semantic search of memes on Twitter \cite{Jesus2020Semantic}.
Researchers have developed methods for meme classification using textual and visual features \cite{Smitha2018Meme}, as well as deep learning and text categorization techniques \cite{Guillermo2020Infotec}. A dual classification system for visual-textual memes has been proposed to capture their structure and multimodal qualities \cite{cochrane}. Some studies have employed optical character recognition (OCR) and machine learning algorithms for meme opinion categorization \cite{Amalia2018Meme}. Additionally, researchers have explored multimodal approaches combining image and text analysis for tasks such as hateful meme classification \cite{Aggarwal2021Two}.
\end{comment}
\subsection{
Current Challenges and Limitations
}
% \textcolor{red}{Limitations? Current. Separate section. Pakita}
As can be seen, numerous theoretical and empirical advancements have been made since meme analysis first began. Multimodal approaches and self-supervised learning were helpful in meme classification and sentiment analysis. By using various approaches, researchers are increasingly highlighting memes as both humorous and problematic objects. Subsequent studies on feature extraction and multimodal analysis are crucial for advancing meme classification. \cite{Smitha2018Meme} used both text and image features, while \cite{Cochrane2022} introduced a two-part classification system for VT memes. These methods, along with some others, such as CLIP, have improved the ability to classify all of the different types of memes.
Nevertheless, several important limitations are evident in meme analysis. Image processing methods like CNNs and YOLOv5 are sensitive to datasets, reducing efficiency when applied to different meme styles or cultures. Moreover, text-based approaches are ineffective for meme text due to its informal and context-sensitive nature. Even modern systems that detect sarcasm and sentiment face challenges with slang, regional languages, and cultural references, making it difficult to identify meme content. These models often fail to analyze the crucial interaction between text and image, which is essential for understanding a meme's full tone and meaning. There is a need for future research that develops more efficient resource models that are context-sensitive and can capture the richness of meme analysis.
%Moreover, it is still easy to miss categories like spam or slander as well as recognize offensive or hateful content with high false positives because of the context, humor, and irony typical for memes.%}
\begin{comment}
\textcolor{Aquamarine}{
The major limitation of the presented multimodal analysis is the interaction between the textual stimuli and the images. Despite advancements done by CLIP and self-supervised methods in meme classification, the models fail to identify memes containing obscure cultural implications or jokes, are hard to model using typical modeling methods. Furthermore, the current multimodal models are also highly complex, thus fading in small scale studies or where the available resources are scarce. There is a need for future research that develop more efficient resource models that are context sensitive so as to capture the richness of meme analysis.}
\end{comment}
\begin{comment}
So, what's the big picture? It's all about getting to know memes on a deeper level. By understanding both the funny and the not-so-funny sides, researchers are making the internet a safer, more inclusive space.
Existing sentiment analysis models fail to fully capture the sentiment of memes due to their reliance on either purely textual or purely visual analysis. This uni-modal approach overlooks the synergistic effect of combining textual and visual cues present in memes. While BERT and other NLP models have greatly improved text analysis, using them for memes is still new. We need more research to find effective ways to combine visual and text analysis for memes.
\end{comment}
% \subsection{Emotion Detection in memes}
% \textcolor{BrickRed}{
% }
\section{Methods}
\subsection{Proposed Approach Overview}
\begin{figure}[tb]
\centering
\includegraphics[width=0.5\textwidth]{pictures/paper/methodology.drawio.png}
\caption{Proposed Approach for Grouping Similar Internet Memes
Using CLIP}
\label{main}
\end{figure}
In this paper, we aim to group memes and identify the emotions that they evoke. The general approach is illustrated in Fig \ref{main}. It includes several steps. Firstly, we need to get the embeddings for memes using the CLIP model. Secondly, we will calculate cosine similarity and compare CLIP embeddings to similar memes in groups. Finally, to retrieve meme emotion, we use DistilBERT.
Memes are a unique combination of texts and images, so consequently, their analysis requires a multimodal approach. To that end, we use the CLIP model to get meme embeddings. Such embeddings are beneficial as they encode different modes of memes as text and images in a single latent space.
\subsection{Data}
For this research, we used a Memotion dataset \cite{chhavi2020memotion}, containing a total of 6,992 images.
\begin{comment}
\textcolor{BrickRed}{The dataset is used to train and validate models that aim to understand and group memes based on their visual and textual content.}
\end{comment}
Each image is associated with humor, sarcasm, motivation, and overall sentiment values. Humor in each image is categorized as either 'funny' or 'not funny.' Similarly, sarcasm is labeled as 'sarcastic' or 'non-sarcastic', and motivation is marked as 'motivational' or 'non-motivational'. As for the overall sentiment, it's classified into three categories: 'positive', 'negative', and 'neutral'. The distribution of memes across these categories is shown in Fig. \ref{fig:distributions}. The word cloud of the texts in memes is presented in Fig. \ref{wordcloud}.
\begin{figure*}[tb]
\centering
% First row
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/motivational.png}
\caption{Distribution of Motivational Memes}
\label{fig:humor}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/sarcasm.png}
\caption{Distribution of Sarcasm}
\label{fig:sarcasm}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/offensive.png}
\caption{Distribution of Offensive Content}
\label{fig:offensive}
\end{subfigure}
% Second row
\vfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=0.7\textwidth]{pictures/humour.png}
\caption{Distribution of Humour Content}
\label{fig:motivational}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=0.7\textwidth]{pictures/overall_sentiment.png}
\caption{Distribution of Overall Sentiment}
\label{fig:overall_sentiment}
\end{subfigure}
\hfill
% \begin{subfigure}[b]{0.32\textwidth}
% \centering
% % Placeholder for the sixth subfigure
% % \includegraphics[width=\textwidth]{placeholder.png}
% % \caption{Placeholder}
% % \label{fig:placeholder}
% % \end{subfigure}
\caption{Bar charts illustrating the distribution of various attributes in the Memotion dataset.}
\label{fig:distributions}
\end{figure*}
\begin{figure}[tb]
\centering
\includegraphics[width=0.5\textwidth]{pictures/wordcloud.png}
\caption{Word cloud of the text in Memotion dataset.}
\label{wordcloud}
\end{figure}
This dataset also contains columns such as the image name and the corrected text of the meme, which we will use in our experiment to get image and text embeddings.
We conducted an experiment to group similar memes by image and text using CLIP on the Memotion dataset. To ensure our results' validity and real-world relevance, we repeated the experiments on the Reddit Memes Dataset \cite{kaggleRedditMemes}, which includes 3226 images along with meme titles, images, and publication times.
%\colorbox{Dandelion}{22. Proofread Methods}
\subsection{Grouping Similar Memes}
To group memes based on their similarity, we used the CLIP embeddings and performed pairwise comparisons between individual meme embeddings according to their cosine similarities (see Fig. \ref{main}). For two memes to be classified as similar, the cosine distance between their embeddings was used, and a threshold was used at 0.8. Memes with cosine similarity above this threshold were considered original, and those with higher similarity were grouped together.
This threshold was chosen to achieve a balance between grouping closely related memes and preventing overly broad categorizations. The steps for grouping memes were as follows:
\begin{itemize}
\item To get embeddings for each meme, use the CLIP model.
The following step is to compute the cosine similarity of two embeddings of all meme pairs.
\item Use a tolerance level of 0.8 when looking for similar meme pairs.
All pairs are grouped iteratively to create clusters of similar, loudly related memes from the initial list of memes.
\end{itemize}
As a result of this process, we identified meme groups with similar textual and visual content.
% providing a basis for further analysis.
\subsection{Contrastive Language-Image Pre-training}
We used CLIP \cite{Radford2021LearningTV} because it has revolutionized multi-modal learning by bridging the gap between text and images. This approach is advantageous for complex tasks like meme analysis, where both visual and textual understanding are key. The model combines image and text embeddings, trained on 400 million image-text pairs with self-supervised learning, mapping them to a shared space. For example, a cat image and the text "a picture of a cat" have similar coordinates in this space.
CLIP demonstrated impressive zero-shot capabilities and transferability across various tasks \cite{Zhou2023, Luo2022CLIP4Clip}. While initially developed for image-text pairs, CLIP has been adapted for video retrieval \cite{Luo2022CLIP4Clip}, region-based object detection \cite{Zhong2022RegionCLIP}, and even 3D point cloud understanding \cite{Zhang2022PointCLIP}. The model has shown promise in zero-shot vision-and-language navigation \cite{Dorbala2022CLIP} and enhanced image retrieval \cite{Lahajal2024Enhancing}. Recent advancements include Long-CLIP, which extends CLIP's capability to handle longer text inputs \cite{Zhang2024Long}, and CLIP4Clip, which transfers CLIP's image-text knowledge to video-text retrieval \cite{Luo2022CLIP4Clip}. These developments highlight CLIP's potential for further applications in multimodal understanding tasks.
\begin{figure}[tb]
\centering
\includegraphics[width=0.5\textwidth]{pictures/paper/CLIP3.drawio.png}
\caption{A visualization of the CLIP model minimizes the distance function between similar memes and maximizes it between different ones}
\label{clip}
\end{figure}
As you can see in Fig. \ref{clip}, CLIP trains both an image encoder (like a CNN, e.g., ResNet) and a text encoder (transformer model) together to match images with their correct text descriptions. Both the image and text encoders project their inputs into a shared embedding space. The aim is to bring related images and texts close together in this space.
CLIP aims to use contrastive loss to minimize the cosine distance for matching image-text embeddings and maximize it for non-matching pairs.
\begin{comment}
\textcolor{BrickRed}{
The main objective of CLIP is to minimize the cosine distance between embeddings of matching image-text pairs while maximizing the distance between non-matching pairs. This is achieved using a contrastive loss function.
}
\end{comment}
Given a batch of \( N \) image-text pairs \( \{(I_i, T_i)\}_{i=1}^{N} \), the contrastive loss for each pair \( (I_i, T_i) \) is computed as:
\begin{equation}
\mathcal{L}_i = -\log \frac{\exp(\text{sim}(I_i, T_i) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(I_i, T_j) / \tau)}
\end{equation}
where \( \tau \) is a temperature parameter.
The total loss for the batch is the sum of the individual losses:
\begin{equation}
\mathcal{L} = \sum_{i=1}^{N} \mathcal{L}_i
\end{equation}
When used as a classifier on a new dataset, the pre-trained text encoder automatically sets up a linear classifier. It does this by creating embeddings of the new classes from their text descriptions.
\subsection{Cosine Similarity}
Cosine similarity is a useful tool to measure how similar the image and text features are to each other when using their numerical embeddings in a high-dimensional space.
We calculated the Cosine similarity coefficient using Eq. 3. Our model uses it to assess the degree of similarity between different meme embeddings.
\begin{equation}
\small
\text{cosine similarity} = \frac{A \cdot B}{\|A\| \|B\|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}}
\end{equation}
\begin{figure}[tb]
\centering
\includegraphics[width=0.5\textwidth]{pictures/paper/similarity_memes3.drawio.png}
\caption{Visualization of calculation of the cosine distance between two memes}
\label{vis}
\end{figure}
\begin{itemize}
\item If two vectors are oriented in the same direction, their cosine similarity is 1 with an angle of 0 degrees between them.
\item If the vectors are perpendicular, their cosine similarity is 0.
\item When two vectors point in opposite directions (180-degree angle), their cosine similarity is -1.
\end{itemize}
\begin{comment}
When two vectors point in the same direction, the angle between them is 0 degrees, and their cosine similarity is 1.
\item When two vectors are perpendicular (at a 90-degree angle), their cosine similarity is 0.
\end{comment}
In summary, smaller angles between vectors mean higher cosine similarity, indicating they are more alike. This metric is particularly useful for understanding semantic similarity in document analysis and has been integrated into our comparison of embedding techniques.
As shown in Fig. \ref{vis}, we compared memes by analyzing both the images and the text. We compared the image of the first meme with the image of the second meme and the text of the first meme with the text of the second meme. Additionally, we calculated the cosine similarity between the image of the first meme and the text of the second meme and vice versa for the second meme.
\subsection{Emotion Detection in Memes}
\begin{comment}
\textcolor{BrickRed}{
Emotion Detection wes performed using DistilBERT, .....
Subfigure: 6 word clouds od meme texts based on emotions, e.g., word cloud for joy, sad memes, etc.
DistilBERT \cite{distilbert} is a more compact, faster, and efficient variant of BERT (Bidirectional Encoder Representations from Transformers) that retains much of BERT's accuracy. It builds on the same transformer architecture and training objectives as BERT, with optimizations for efficiency (see Fig. \ref{fig
}). DistilBERT’s self-attention mechanism calculates attention scores from input vectors as shown in Eq. \ref{eq
}: }
\end{comment}
Emotion detection in memes was performed using DistilBERT, a compact, fast, and efficient version of BERT. DistilBERT retains 97\% accuracy of BERT while significantly reducing the model size and output time \cite{Sanh2019DistilBERT}. It uses the same transformer architecture and learning objectives as BERT \cite{devlin2019bertpretrainingdeepbidirectional} but includes optimizations to improve efficiency (see Fig.~\ref{fig:bert_distilbert}).
\begin{figure}[tb]
\centering
\includegraphics[width=0.5\textwidth]{pictures/bert_distilbert.png}
\caption{The DistilBERT model architecture \cite{Adel2022}}
\label{fig:bert_distilbert}
\end{figure}
%{A step-by-step process to identify emotions in the textual content of memes is presented in Algorithm~\ref{alg:emotion_detection}.}
% \begin{comment}
% \begin{algorithm}
% \caption{Emotion Detection with Emoji Intensity}\label{alg:emotion_detection}
% \begin{algorithmic}[1]
% \Require CSV file with columns \texttt{Folder}, \texttt{Filename}, \texttt{Extracted Text}
% \Ensure File with \texttt{Predicted Emotion} and \texttt{Emotion Score}
% \State Load data from CSV file
% \State Verify required columns are present
% \State Load pre-trained \texttt{DistilBERT} model and tokenizer
% \State Define emoji sentiments and regex for emoji extraction
% \ForAll{rows in data}
% \State Extract text from \texttt{Extracted Text}
% \State Compute emoji intensity factor and predominant emotion
% \State Predict text emotion and score using model
% \State Combine text emotion and emoji intensity to compute final results
% \State Store results in \texttt{Predicted Emotion} and \texttt{Emotion Score}
% \EndFor
% \State Save updated dataset to \texttt{data\_test\_with\_emotions.csv}
% \end{algorithmic}
% \end{algorithm}
% \end{comment}
% \begin{algorithm}
% \caption{Emotion Detection Algorithm}\label{alg:emotion_detection}
% \begin{algorithmic}[1]
% \Require CSV file with \texttt{Extracted Text}
% \Ensure \texttt{Predicted Emotion} and \texttt{Emotion Score}
% \State Load data and verify columns
% \State Load \texttt{DistilBERT} model and tokenizer
% \ForAll{text in \texttt{Extracted Text}}
% \State Predict emotion using model
% %\State Calculate emoji intensity factor
% %\State Combine text and emoji emotions
% \EndFor
% \State Save results to CSV
% \end{algorithmic}
% \end{algorithm}
% \begin{figure}[h!]
% \centering
% \includegraphics[width=0.5\textwidth]{pictures/paper/DifferenceDistilBERT.png}
% \caption{Comparison of BERT and DistilBERT in terms of accuracy and inference speed.}
% \label{fig:bert_distilbert}
% \end{figure}
\begin{figure}[htb]
\centering
{
\begin{minipage}{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/paper/categorized_3.png}
\end{minipage}
\vspace{0.5cm} % Add vertical space
%\hspace{0.5cm} % Add horizontal space
\begin{minipage}{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/paper/categorized_4.png}
\end{minipage}
}
\vspace{0.5cm} % Add vertical space
\begin{minipage}{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/paper/categorized_5.png}
\end{minipage}
\vspace{0.5cm} % Add vertical space
\begin{minipage}{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/paper/categorized_6.png}
\end{minipage}
\vspace{0.5cm} % Add vertical space
\begin{minipage}{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/paper/categorized_7.png}
\end{minipage}
\caption{Example of obtained results - groups of memes}
\label{fig:memes}
\end{figure}
\begin{figure*}[htb]
\centering
% First row
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/wordcloud/joy.png}
\caption{Joy}
\label{fig:joy}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/wordcloud/Sadness.png}
\caption{Sadness}
\label{fig:sadness}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/wordcloud/Anger.png}
\caption{Anger}
\label{fig:anger}
\end{subfigure}
% Second row
\vfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/wordcloud/Fear.png}
\caption{Fear}
\label{fig:fear}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/wordcloud/love.png}
\caption{Love}
\label{fig:love}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{pictures/wordcloud/Surprise.png}
\caption{Surprise}
\label{fig:surprise}
\end{subfigure}
\hfill
\caption{Word clouds for different emotions detected in similar meme groups from Reddit Meme Dataset.}
\label{fig:wordCloudEmotions}
\end{figure*}
\begin{comment}
\begin{figure}[h!]
\centering
\subfloat[Joy]{\includegraphics[width=0.5\textwidth]{pictures/wordcloud/joy.png}}
\hfill
\subfloat[Sadness]{\includegraphics[width=0.5\textwidth]{pictures/wordcloud/Sadness.png}}
\hfill
\subfloat[Anger]{\includegraphics[width=0.5\textwidth]{pictures/wordcloud/Anger.png}}
\caption{Word clouds for different emotions detected in memes.}
\label{fig:wordclouds}
\end{figure}
\end{comment}
\begin{equation} \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V \label{eq
} \end{equation}
where $Q$ is query, $K$ is key, and $V$ represents value matrices \cite{igali2024trackingemotionaldynamicschat}. In addition, $d_k$ refers to the dimensionality of the key vectors. The model can focus on information from different representation subspaces at distinct positions through multi-head attention \cite{igali2024trackingemotionaldynamicschat}:
\begin{equation} \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O \label{eq
} \end{equation}
\begin{equation} \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) \label{eq
} \end{equation}
with $W_i^Q$, $W_i^K$, $W_i^V$, and $W^O$ being parameter matrices \cite{igali2024trackingemotionaldynamicschat}. Each transformer block in DistilBERT also contains a feed-forward network that applies transformations:
\begin{equation} \text{FFN}(x) = \text{max}(0, xW_1 + b_1)W_2 + b_2 \label{eq
} \end{equation}
To enhance training stability, DistilBERT employs layer normalization and residual connections:
\begin{equation} \text{LayerNorm}(x + \text{Sublayer}(x)) \label{eq
} \end{equation}
% where $\text{Sublayer}(x)$ may relate to an attention mechanism. In addition, it could refer to a feed-forward network.
% DistilBERT utilizes a distillation loss, typically implemented through a temperature-scaled cross-entropy loss. This helps the smaller model to learn from the larger BERT model \cite{igali2024trackingemotionaldynamicschat}.
DistilBERT was trained on a 20 000 rows dataset derived from a collection of English Twitter messages to identify six basic emotions: \textit{joy, fear, anger, love, sadness} and \textit{surprise} \cite{igali2024trackingemotionaldynamicschat}, \cite{saravia-etal-2018-carer}. The model DistilBERT achieved an accuracy of 0.93 and was used to classify meme texts into six main emotions.
\begin{comment}
\subsection{Recognizing memes’ Emotions}
\textcolor{Aquamarine}{
In the second experiment, we wanted to find out what emotions are contained in memes and identify the feelings connected with sets of similar memes. For this purpose, we fine-tuned DistilBERT which is a lighter version of BERT model which is designed for emotion recognition. DistilBERT was applied to extract emotions from the textual content of memes, categorizing each meme into one of six basic emotions: There are six basic human emotions to them: \textit{sadness}, \textit{joy}, \textit{love}, \textit{anger}, \textit{fear}, and \textit{surprise}.
}
To analyze emotions within meme groups:
\begin{itemize}
\item For each meme, DistilBERT determined the overall emotion of the textual content of the meme.
\item Memes within each group that was defined by similarity threshold were analyzed to identify common emotional patterns.
\item To learn about the typical emotions of similar meme clusters, the distribution of emotions among meme groups was analyzed.
\end{itemize}
\textcolor{Aquamarine}{
In this experiment, the emotional nature of memes and the emotional contingency within meme subsets were introduced.
With the application of multimodal embedding analysis and emotion recognition, the research framework presented here presents a comprehensive method of assessing and comparing memes not only in terms of their content but also their emotionality.
}
\end{comment}
\section{Results}
%\colorbox{Dandelion}{21. Proofread results}
%\begin{table}[h]
% \centering
% \begin{tabular}{|c|c|c|}
% \hline
% \textbf{Dataset} & \textbf{ResNet101} & \textbf{CLIP} \\
% \hline
% ImageNet & 76.2 & 76.2 \\
% ImageNetV2 & 64.3 & 70.1 \\
% ImageNet-R & 37.7 & 88.9 \\
% ObjectNet & 32.6 & 72.3 \\
% ImageNet-A & 2.7 & 77.1 \\
% \hline
% \end{tabular}
% \caption{\colorbox{Lavender}{Remove??}CLIP is more resilient to changes in data distributions compared to a model trained on ImageNet.}
%\end{table}
%\textcolor{Red}{
%As we can see in Table 1 the stability-ideal model (represented by the dotted line) performs well both in the ImageNet distribution and in other similar image distributions across shared classes. However, CLIP models, can be more accurate than traditional models trained on ImageNet. This holds true even for models enhanced with well-known stability techniques.
%}
\subsection{Example of grouping memes}
\begin{comment}
\begin{table*}[tb]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|}
%\hline
%\multicolumn{4}{|c|}{\textbf{Embeddings Table}} & \multicolumn{3}{c|}%{\textbf{Cosine Similarity Matrix}} \\
\hline
\textbf{image\_name} & \textbf{text\_corrected} & \textbf{img\_emb} & \textbf{text\_emb} & \textbf{image\_1} & \textbf{...} & \textbf{image\_6992} \\
\hline
image\_1.jpg & LOOK ... & [-5.74e-02 ... ] & [-5.74e-02 ... ] & tensor([[0.3461]]...) & ... & ... \\
... & ... & ... & ... & ... & ... & ... \\
image\_6992.png & The ... & [-1.07e-02 ...] & [1.24e-02 ...] & ... & ... & tensor([[0.8138]]...) & \hspace{1cm} \\
\hline
\end{tabular}
\caption{Combined table with embeddings and cosine similarity matrix for image and text modalities in the Memotion dataset}
\end{table*}
\end{comment}
%\begin{comment}
%%% EMBEBBINGS TABLE MAIN
\begin{table*}[h]
\centering
% \caption{Embeddings and Cosine Similarity Matrix for Image and Text Modalities in the Memotion Dataset}
\caption{Extracted image and text embeddings and cosine similarity scores for the Memotion dataset. Higher similarity values indicate a closer semantic relationship between image-text pairs.
}
\resizebox{\textwidth}{!}{%
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
\textbf{image\_name} & \textbf{text\_corrected} & \textbf{img\_emb} & \textbf{text\_emb} & \textbf{image\_1} & \textbf{...} & \textbf{image\_6992} \\
\midrule
\textbf{image\_1.jpg} & LOOK ... & [-5.74e-02 ... ] & [-5.74e-02 ... ] & tensor([[0.3461]]...) & ... & ... \\
\textbf{...} & ... & ... & ... & ... & ... & ... \\
\textbf{image\_6992.png} & The ... & [-1.07e-02 ...] & [1.24e-02 ...] & ... & ... & tensor([[0.8138]]...) \\
\bottomrule
\end{tabular}%
}
\label{table1}
\end{table*}
%\end{comment}
\begin{comment}
\begin{table}[tb]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{image\_name} & \textbf{text\_corrected} & \textbf{img\_emb} & \textbf{text\_emb} \\
\hline
image\_1.jpg & LOOK ... & [-5.74e-02 ... ] & [-5.74e-02 ... ] \\
... & ... & ... & ... \\
image\_6992.png & The ... & [-1.07-02 ...] & [ 1.24e-02 ... ] \\
\hline
\end{tabular}
\caption{The stored embeddings for image, text modalities for Memotion dataset}
\end{table}
\end{comment}
Fig. \ref{fig:memes} illustrates some of the grouping examples. From this, we can conclude that CLIP is not only a breakthrough in classification through unsupervised learning but also sets new standards for stability. This is because CLIP learns from image-text pairs that provide a richer context rather than just raw image classes. This highlights the importance of context not only in NLP but also in computer vision.
Table~\ref{table1} presents an overview of the embeddings extracted for both image and text modalities in the Memotion dataset, along with their respective cosine similarity scores. Higher similarity values indicate a closer semantic relationship between image-text pairs.
% It is important for our analysis, as it enables the identification of visually and semantically related memes.
\begin{comment}
\textcolor{Aquamarine}{As shown in Table~\ref{table1}, the research presents intermediate findings consisting of meme image and text embeddings as well as cosine similarity scores between them. The first two columns of the table present the meme filenames alongside their text data normalization. The table contains img\_emb image embeddings together with text\_temp text embeddings which were extracted from multimodal models. Each column in the remaining portion contains cosine similarity scores that measure the compatibility between specific meme images and all other images in the collection.
}
\end{comment}
\begin{comment}
\begin{table}[tb]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\hspace{1cm} & \textbf{image\_1} & \textbf{...} & \textbf{image\_6992} \\
\hline
image\_1.jpg & \hspace{1cm} & ... & tensor([[0.3461]]... \\
... & ... & ... & ... \\
image\_6969.png & tensor([[0.8138]]... & ... & \hspace{1cm} \\
\hline
\end{tabular}
\caption{The stored cosine similarity matrix for image, text modalities for Memotion dataset}
\end{table}
\end{comment}
\subsection{Performance Evaluation}
%\colorbox{LimeGreen}{12.Choose method - Percent Agree}
%\colorbox{LimeGreen}{13.Describe experiment with humans, formulas}
%\colorbox{LimeGreen}{14.Results: figs, tables}
% \color{Aquamarine}
\begin{comment}
The study involved 51 participants, aged between 18 and 35 years (M = 19.44, SD = 5.17). Participants were recruited through online platforms and were volunteers who consented to participate in the study. The sample included 25 females and 25 males.
The experiment utilized 21 meme groups, each consisting of a set of memes that were hypothesized to share similar themes or emotional content. These meme groups were carefully selected based on predefined criteria to ensure diversity in content and emotional expression.
Participants were presented with a survey containing 21 meme groups. For each meme group, participants were asked to answer two questions. The first question asked, "Are the memes in Figure similar?" (Yes/No). The second question inquired, "Specify the emotion in Figure?" Participants were provided with a list of emotions to choose from, such as sadness, joy, love, fear, surprise or anger.
Participants were instructed to carefully observe each meme group and respond based on their personal interpretation. The survey was administered online, and participants completed it at their own pace.
Responses were collected via an online survey platform Google Forms, ensuring that all participant data remained anonymous. Participants were informed about the purpose of the study, and their consent was obtained prior to participation. The study was conducted in accordance with ethical guidelines for research involving human participants.
The data were analyzed to assess the agreement between the CLIP model's grouping of similar memes and the participants' responses. Statistical measures such as precision, recall, and accuracy were calculated to evaluate the performance of the CLIP model. Additionally, the emotions identified by participants were compared to the emotions predicted by the DistilBERT to determine the model's accuracy in emotion detection.
\begin{itemize}
\item True Positives (TP) = 20
\item False Positives (FP) = 1
\item False Negatives (FN) = 0
\item True Negatives (TN) = 4
\end{itemize}
\subsection*{1. Precision}
Precision is the proportion of correctly identified positive instances out of all instances identified as positive.
\[
\text{Precision} = \frac{TP}{TP + FP} = \frac{20}{20 + 1} = \frac{20}{21} \approx 0.9524
\]
\subsection*{2. Recall (Sensitivity, True Positive Rate)}
Recall is the proportion of correctly identified positive instances out of all actual positive instances.
\[
\text{Recall} = \frac{TP}{TP + FN} = \frac{20}{20 + 0} = \frac{20}{20} = 1.0
\]
\subsection*{3. Accuracy}
Accuracy is the proportion of correctly classified instances (both positive and negative) out of all instances.
\[
\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{24}{25} = 0.96
\]
\subsection*{4. False Positive Rate (FPR)}
False Positive Rate is the proportion of negative instances incorrectly classified as positive out of all actual negative instances.
\[
\text{FPR} = \frac{FP}{FP + TN} = \frac{1}{1 + 4} = \frac{1}{5} = 0.2
\]
\color{red}
TODO
Describe one group of images that most users consider as not similar
Provide percent agree metric %(71)
%
\subsection*{5. Summary of Results}
\begin{itemize}
\item Precision: \( 0.9524 \) (95.24\%)
\item Recall (True Positive Rate): \( 1.0 \) (100\%)
\item Accuracy: \( 0.96 \) (96\%)
\item False Positive Rate (FPR): \( 0.2 \) (20\%)
\end{itemize}
\color{black}
The survey results indicated that participants agreed with the AI model's grouping of similar memes in 20 out of 21 cases, yielding a precision of 95.24%.
\end{comment}
We conducted a user study with 51 participants aged 18-35 to evaluate our CLIP-based meme clustering model. Each participant viewed multiple meme clusters and indicated whether the memes seemed similar. The Arbitration and Agreement Rate (A:AR) metric was used to ensure consensus. The \textit{Agreement Rate} metric is calculated as the percentage of participants agreeing with the model’s groupings for each cluster.
% We assess the model’s clustering by comparing it to participants' judgments using the \textit{Agreement Rate} metric, calculated as the percentage of participants agreeing with the model’s groupings for each cluster.
\begin{comment}
In order to assess the performance of our presented CLIP-based meme clustering model, we have organized a user study with 51 participants from the age group of 18-35. In total, the multiple clusters of memes were shown to every participant, and for each cluster, the participants responded whether or not they saw the memes as similar.
By applying the Arbitration and incenting Agreement Rate (A:AR) metric, all participants had to find agreement within the group or we would not encourage the submission.
For the purpose of evaluating the model and comparing its clustering decisions with people’s judgments, we employ the \textit{Agreement Rate} metric. In assessment of each question (meme cluster), Agreement Rate is computed as the percentage of participants who share the model’s way of grouping. This value is calculated as follows:
\end{comment}
\begin{equation}
\text{Agreement Rate} = \frac{\text{Users Agree}}{\text{Total Users}} \times 100
\end{equation}
The overall Agreement Rate is then obtained by averaging the Agreement Rates across all clusters:
\begin{equation}
\text{Average Agreement Rate} = \frac{\sum \text{Agreement Rate}}{n}
\end{equation}
denoted by \( n \) — the number of meme clusters considered in the survey.
The Agreement Rate determination for each of the clusters varied between 45.01\% and 82.35\%, with an average of 67.23\%. This means that, on average, participants concurred with the model at a moderate level of 67.23\% in the manner in which the model grouped similar memes. The findings show that some clusters received more agreement, and this insight implies a good correlation between the model and the actual human perception corresponding to the clusters.
% \textcolor{olive}{As shown in Fig. \ref{lowAR1_2}, it was hard for the model to group memes in clusters correctly. This may be due to difficulties in understanding the underlying meaning of the memes. In contrast, Fig. \ref{highAR1_2} highlights clusters with high agreement rates, where memes shared visually similar images and captions with clearly aligned meanings. This alignment helped the model to categorize memes, increasing agreement scores among participants.
% }
\begin{comment}
\begin{table}[tb]
\centering
\caption{Agreement Rates for Meme Clusters in User Survey}
\begin{tabular}{c|c}
\toprule
\textbf{Cluster} & \textbf{Agreement Rate (\%)} \\
\midrule
Figure 1 & 64.71 \\
Figure 2 & 45.10 \\
Figure 3 & 70.59 \\
Figure 4 & 52.94 \\
Figure 5 & 60.78 \\
Figure 6 & 70.59 \\
Figure 7 & 66.67 \\
Figure 8 & 82.35 \\
Figure 9 & 78.43 \\
Figure 10 & 60.78 \\
Figure 11 & 80.39 \\
Figure 12 & 70.59 \\
Figure 13 & 56.86 \\
Figure 14 & 58.82 \\
Figure 15 & 66.67 \\
Figure 16 & 68.63 \\
Figure 17 & 76.47 \\
Figure 18 & 74.51 \\
Figure 19 & 60.78 \\
Figure 20 & 74.51 \\
Figure 21 & 70.59 \\
\bottomrule
\end{tabular}
\label{tab:agreement_rates}
\end{table}
\end{comment}
\begin{table*}[h]
\centering
\caption{Agreement Rates for Meme Groups in User Survey}
\resizebox{\textwidth}{!}{%
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
\toprule
\textbf{Meme group} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6} & \textbf{7} & \textbf{8} & \textbf{9} & \textbf{10} & \textbf{11} & \textbf{12} & \textbf{13} & \textbf{14} & \textbf{15} & \textbf{16} & \textbf{17} & \textbf{18} & \textbf{19} & \textbf{20} & \textbf{21} \\
\midrule
\textbf{AR (\%)} & 64.71 & 45.10 & 70.59 & 52.94 & 60.78 & 70.59 & 66.67 & 82.35 & 78.43 & 60.78 & 80.39 & 70.59 & 56.86 & 58.82 & 66.67 & 68.63 & 76.47 & 74.51 & 60.78 & 74.51 & 70.59 \\
\bottomrule
\end{tabular}%
}
\label{tab:agreement_rates}
\end{table*}
% \begin{figure*}[htb]
% \centering
% % First row
% \begin{subfigure}[b]{0.49\textwidth}
% \centering
% \includegraphics[width=\textwidth]{pictures/low AR1.png}
% \caption{Agreement rate for meme in Figure 2}
% \label{fig:low_ar1}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.48\textwidth}
% \centering
% \includegraphics[width=\textwidth]{pictures/low AR2.png}
% \caption{Agreement rate for meme in Figure 4}
% \label{fig:low_ar2}
% \end{subfigure}
% \hfill
% \caption{Example of cluster memes with low agreement rates}
% \label{lowAR1_2}
% \end{figure*}
% \begin{figure*}[htb]
% \centering
% % First row
% \begin{subfigure}[b]{0.49\textwidth}
% \centering
% \includegraphics[width=\textwidth]{pictures/high AR1.png}
% \caption{Agreement rate for meme in Figure 8}
% \label{fig:high_ar1}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.49\textwidth}
% \centering
% \includegraphics[width=\textwidth]{pictures/high AR2.png}
% \caption{Agreement rate for meme in Figure 11}
% \label{fig:high_ar2}
% \end{subfigure}
% \hfill
% \caption{Example of cluster memes with high agreement rates}
% \label{highAR1_2}
% \end{figure*}
\begin{comment}
begin{table}[h!]
\centering
\begin{tabular}{|c|c|}
\hline
\textbf{Cluster} & \textbf{Agreement Rate (\%)} \\
\hline
Figure 1 & 64.71 \\
Figure 2 & 45.10 \\
Figure 3 & 70.59 \\
Figure 4 & 52.94 \\
Figure 5 & 60.78 \\
Figure 6 & 70.59 \\
Figure 7 & 66.67 \\
Figure 8 & 82.35 \\
Figure 9 & 78.43 \\
Figure 10 & 60.78 \\
Figure 11 & 80.39 \\
Figure 12 & 70.59 \\
Figure 13 & 56.86 \\
Figure 14 & 58.82 \\
Figure 15 & 66.67 \\
Figure 16 & 68.63 \\
Figure 17 & 76.47 \\
Figure 18 & 74.51 \\
Figure 19 & 60.78 \\
Figure 20 & 74.51 \\
Figure 21 & 70.59 \\
\hline
\end{tabular}
\caption{Agreement Rate for each meme cluster in the user survey.}
%\end{table}
\end{comment}
\color{Black}
%{Now we can compare the predictive power of our method by looking at the similarities between memes using multimodal approach. We used aggregated results and considered a similarity coefficient above 0.8.}
\subsection{Emotion Analysis Results}
Each meme was analyzed for its textual content using DistilBert, and the detected emotions were visualized using a cloud of words, where each cloud represents the most common words associated with a particular emotion (see Fig.~\ref{fig:wordCloudEmotions}). For example:
\begin{itemize}
\item \textbf{Joy}. The word cloud highlights positive and uplifting terms, reflecting happiness in memes (Fig.~\ref{fig:wordCloudEmotions}a).
\item \textbf{Sadness}: The cloud is dominated by words associated with a sad or melancholic context (Fig.~\ref{fig:wordCloudEmotions}b).
\item \textbf{Anger}: Terms expressing frustration or conflict are highlighted (Fig.~\ref{fig:wordCloudEmotions}c).
\end{itemize}
\begin{comment}
\begin{table}[h!]
\centering
\caption{Emotion Distribution in the Memotion Dataset}
\begin{tabular}{|l|r|r|}
\hline
\textbf{Emotion} & \textbf{Count} & \textbf{Percentage (\%)} \\
\hline
Joy & 3120 & 44.65 \\
Anger & 2374 & 33.98 \\
Fear & 629 & 9.00 \\
Sadness & 627 & 8.97 \\
Surprise & 143 & 2.05 \\
Love & 94 & 1.35 \\
\hline
\end{tabular}
\label{tab:emotion_distribution}
\end{table}
\end{comment}
%\begin{comment}
\begin{table}[tb]
\centering
\caption{Emotion Distribution in the Dataset}
%\resizebox{\textwidth}{!}{%
\scriptsize
\begin{tabular}{l|r|r|r|r|r|r}
\toprule
\textbf{Emotion} & \textbf{Joy} & \textbf{Anger} & \textbf{Fear} & \textbf{Sadness} & \textbf{Surprise} & \textbf{Love} \\
\midrule
\textbf{Count} & 3120 & 2374 & 629 & 627 & 143 & 94 \\
%\hline
\textbf{Percent (\%)} & 44.65 & 33.98 & 9.00 & 8.97 & 2.05 & 1.35 \\
\bottomrule
\end{tabular}%
%}
\label{tab:emotion_distribution}
\end{table}
%\end{comment}
Table \ref{tab:emotion_distribution} illustrates the distribution of emotions within the Memotion dataset, revealing that joy (44.65\%) is the most prevalent emotion, followed by anger (33.98\%), while love (1.35\%) and surprise (2.05\%) are the least represented. These findings highlight the dominance of positive and intense emotions in meme content.
In addition, we performed analysis to identify the relationship between emotions and meme content (motivational, sarcastic, offensive, humor, sentiment). As a result, we identified a statistically significant relationship between emotion and motivational content (p-value 0.045). This suggests that motivational content tends to evoke stronger emotional responses.
\begin{comment}
\begin{table}[h]
\centering
\caption{P-values for Chi-Square Tests Between Emotion and Other Variables}
\begin{tabular}{c|c|c|c|c}
\toprule
\textbf{Variable} & \textbf{Humor} & \textbf{Sarcasm} & \textbf{Offensive} & \textbf{Motivational} \\
\midrule
\textbf{P-value} & 0.376 & 0.174 & 0.345 & 0.045 \\
\bottomrule
\end{tabular}
\label{tab:chi_square_pvalues}
\end{table}
\end{comment}
\section{Limitations and Future Work}
%\colorbox{Dandelion}{17. Proofread}
%\colorbox{Dandelion}{15. Limitations 1 paragraph}
%\colorbox{Dandelion}{16. Future Works 1 paragraph}
% The study has several potential limitations.
While our study demonstrates promising results, several limitations remain, like limited user study scope, multimodal ambiguities, evolving nature of memes, and dataset biases.
Due to rapidly changing trends, memes can quickly become outdated. A model trained on static datasets may not perform well on newly emerging memes. In our analysis, we found that the model occasionally performed poorly due to memes' subtle and often ambiguous meanings. The Memotion and Reddit Meme datasets used in this study follow specific meme formats in the English language and may not fully represent the diversity of meme styles found on various platforms.
The findings revealed that the agreement rates ranged between 45.10\% and 82.35\% through clusters. This implies that the model performs poorly in handling some types of memes, probably because of variations in the positioning of texts and images. Additionally, some memes in the dataset lack text, which our emotion analysis requires.
\begin{comment}
\textcolor{Aquamarine}{
Also when emotion detection is done using only meme texts, there are bound to be loopholes since most memes are incomplete and dependent on the images and the texts.
}
\end{comment}
Only 51 participants aged 18-35 were included in the survey, so it is uncertain how various user demographics perceive memes. Variations in meme styles, languages, or cultural aspects that are not captured in the presented dataset may decrease the generalization of the findings.
Further research can be dedicated to improving memes' text and image features.
It can explore advanced cross-modal fusion techniques, such as attention-based fusion mechanisms, to better integrate textual and visual features. We also plan to incorporate external knowledge sources like social media trends. In addition, more diverse participants’ ages and cultural backgrounds must be involved. This diversity will improve the assessment of model performance across the different segments, helping us better represent the complex nature of memes.
% , it is quite possible that more thorough integration of these modalities would facilitate more consistent and homogeneous clustering.
% Therefore, true sentiment models should consider both text and graphical parts of memes as components.
% The idea of using both lexicon and sentiment analysis may help to present a richer description of the emotions that people share through memes.
% Moving forward, we plan to investigate new techniques and
\section{Conclusion}
% \colorbox{Dandelion}{18. Proofread}
% \colorbox{Dandelion}{19. Extend main message and results}
%\colorbox{LimeGreen}{20. Discussion - comparison with recent similar studies}
In this research, we aimed to develop a model for multi-modal meme analysis. Our study results show that the model effectively groups similar memes based on their visual and textual content.
Memes are often perceived as humorous, yet they contain a broad spectrum of emotions. For instance, motivational memes may draw from emotions such as anger or happiness. Additionally, we aimed to make a proposal for a list of basic emotions that memes produce or provoke. Results show that emotions of \textit{anger} and \textit{joy} are dominant in memes. Our findings also show that motivational memes tend to evoke stronger emotional responses. Such a perspective contributes to further research on the emotional impact of internet memes.
%In our analysis, we noticed that the model sometimes struggled to perform exceptionally well, mainly because memes often have subtle meanings that are not always obvious. %
A number of recent studies highlighted the effectiveness of multimodal pre-training and cross-modal interaction modeling in detecting and understanding hateful memes. Some of them have focused on the analysis of memes using CLIP and consideration of both visual and textual information. \cite{Gokul2022Hate} proposed the Hate-CLIPper architecture, which uses CLIP encoders and a feature interaction matrix to model cross-modal interactions. Similarly, \cite{Y2022On} used CLIP for multimodal contrastive learning to examine how hateful memes developed.
% Semantic patterns were found in the CLIP-generated embeddings. They showed connections both within and across modalities. Their framework identified numerous variants of antisemitic memes, demonstrating its potential for analyzing hateful content evolution.
% This approach achieved state-of-the-art performance on the Hateful Memes Challenge dataset, surpassing human performance.
Potential applications include Content Moderation in social media (e.g., Detecting Offensive Memes), Meme Search and Retrieval, Personalized Meme Recommendations, and Meme Trend Detection, among others.
\section*{Acknowledgement}
This research has been funded by the Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan (Grant No. AP22786412)
%\bibliographystyle{unsrt}
\bibliography{library.bib}
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{meme}
``Instagram year in review: How memes were the mood of 2020,'' \url{https://about.instagram.com/blog/announcements/instagram-year-in-review-how-memes-were-the-mood-of-2020}, 2020, [Accessed 16-12-2024].
\bibitem{olena2020memes}
P.~Olena, V.~Iryna, K.~Nataliia, and F.~Volodymyr, ``Memes as the phenomenon of modern digital culture,'' \emph{Wisdom}, no. 2 (15), pp. 45--55, 2020.
\bibitem{Grishaeva2021}
S.~Grishaeva, ``The potential of the internet meme as a means of cross-cultural communication,'' \emph{Bulletin of BSU}, 2021.
\bibitem{Gao2024}
X.~Gao, Z.~Liu, and T.~Chen, ``Exploring image-only methods for meme categorization,'' \emph{JMMA}, vol.~38, pp. 112--127, 2024.
\bibitem{Choi2023}
H.-S. Choi, M.-J. Lee, and D.-H. Kim, ``Humor recognition in memes using deep convolutional neural networks,'' \emph{International Journal of Computer Vision}, vol. 130, pp. 987--1002, 2023.
\bibitem{Wang2024}
J.~Wang, X.~Liu, and Y.~Zhang, ``Object detection for meme categorization: Image-based analysis using yolov5,'' \emph{Pattern Recognition Letters}, vol. 163, pp. 50--60, 2024.
\bibitem{Sharma2022}
A.~Sharma, R.~Desai, and P.~Shah, ``Emotional meme analysis through generative adversarial networks,'' in \emph{NeurIPS}.\hskip 1em plus 0.5em minus 0.4em\relax Curran Associates Inc., 2022, pp. 1450--1462.
\bibitem{Xu2022}
Y.~Xu, J.~Li, and H.~Zhang, ``Linguistic structures in text-based memes: A deep learning perspective,'' \emph{Natural Language Processing and Text Mining}, vol.~21, pp. 23--34, 2022.
\bibitem{Kim2023}
S.~jin Kim and H.~Lee, ``Text-based sentiment analysis in internet memes: A transformer approach,'' \emph{Journal of AI and Sentiment Analysis}, vol.~12, pp. 112--125, 2023.
\bibitem{Bennett2021}
L.~Bennett, J.~Edwards, and S.~Mason, ``Detecting offensive language in text-based memes using supervised learning,'' \emph{Journal of Computational Linguistics}, vol.~29, pp. 89--105, 2021.
\bibitem{Pang2023}
J.~Pang, Z.~Li, and M.~Zhao, ``Cultural references in text-based memes: A linguistic approach to contextual humor,'' \emph{Journal of Social Media and Digital Culture}, vol.~18, pp. 199--212, 2023.
\bibitem{Rajendran2022}
V.~Rajendran, M.~Goyal, and T.~Sharma, ``Text summarization in memes: A purely linguistic approach,'' \emph{International Journal of AI and NLP Research}, vol.~17, pp. 63--77, 2022.
\bibitem{Guillermo2020Infotec}
{Guillermo Ruiz}, {E. Tellez}, {Daniela Moctezuma}, {Sabino Miranda-Jiménez}, {Tania A. Ramirez-delreal}, and {Mario Graff}, ``Infotec + {CentroGEO} at {SemEval}-2020: Deep {Learning} and {Text} {Categorization} approach for {Memes} classification,'' \emph{SemEval}, 2020.
\bibitem{Zhang2024Long}
B.~Zhang, P.~Zhang, X.~Dong, Y.~Zang, and J.~Wang, ``Long-{CLIP}: Unlocking the {Long}-{Text} {Capability} of {CLIP},'' 2024.
\bibitem{Kumar2022Hate}
{Gokul Karthik Kumar} and {K. Nandakumar}, ``Hate-clipper: Multimodal hateful meme classification based on cross-modal interaction of clip features,'' \emph{NLP4PI}, 2022.
\bibitem{Tung2023}
P.~T.~H. Tung, N.~T. Viet, N.~T. Anh, and P.~D. Hung, ``Semimemes: A semi-supervised learning approach for multimodal memes analysis,'' 3 2023.
\bibitem{Alluri2021}
N.~V. Alluri and N.~D. Krishna, ``Multi-modal analysis of memes for sentiment extraction,'' in \emph{2021 ICIIP}, 2021.
\bibitem{Prakash2023}
N.~Prakash, H.~Wang, N.~K. Hoang, M.~S. Hee, and R.~K.-W. Lee, ``Promptmtopic: Unsupervised multimodal topic modeling of memes using large language models,'' in \emph{CIKM}, 2023.
\bibitem{Cui2023}
J.~Cui, L.~Li, and X.~Tao, ``Be-or-not prompt enhanced hard negatives generating for memes category detection,'' vol. 2023-July.\hskip 1em plus 0.5em minus 0.4em\relax IEEE Computer Society, 2023, pp. 174--179.
\bibitem{Lin2023}
H.~Lin, Z.~Luo, J.~Ma, and L.~Chen, ``Beneath the surface: Unveiling harmful memes with multimodal reasoning distilled from large language models,'' 12 2023.
\bibitem{Koutlis2023}
C.~Koutlis, M.~Schinas, and S.~Papadopoulos, ``Memetector: enforcing deep focus for meme detection,'' \emph{International Journal of Multimedia Information Retrieval}, vol.~12, 6 2023.
\bibitem{simmeme}
T.~Milo, A.~Somech, and B.~Youngmann, ``Simmeme: A search engine for internet memes.''
\bibitem{Ma2022}
Z.~Ma, S.~Yao, L.~Wu, S.~Gao, and Y.~Zhang, ``Hateful memes detection based on multi-task learning,'' \emph{Math.}, vol.~10, 12 2022.
\bibitem{Pimpalkar2022}
A.~Pimpalkar, A.~Chaudhari, A.~Lilhare, N.~Dighorikar, S.~Dakhole, and S.~Asawa, ``Sentiment identification from image-based memes using machine learning,'' \emph{IJIES}, vol.~7, pp. 89--96, 8 2022.
\bibitem{Joshi2023}
S.~Joshi, F.~Ilievski, and L.~Luceri, ``Contextualizing internet memes across social media platforms,'' 11 2023.
\bibitem{Qu2022}
Y.~Qu, X.~He, S.~Pierson, M.~Backes, Y.~Zhang, and S.~Zannettou, ``On the evolution of (hateful) memes by means of multimodal contrastive learning,'' \emph{IEEE Symp. on Security and Privacy}, 2022.
\bibitem{Giri2021}
R.~K. Giri, S.~C. Gupta, and U.~K. Gupta, ``An approach to detect offence in memes using natural language processing(nlp) and deep learning.''\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 1 2021.
\bibitem{Gupta2021}
K.~Gupta and C.~Kwatra, ``Multimodal meme sentiment analysis with image inpainting,'' 2021.
\bibitem{hatzivassiloglou1997predicting}
V.~Hatzivassiloglou, ``Predicting the semantic orientation of adjectives,'' in \emph{8th ACL}, 1997.
\bibitem{hu2004mining}
M.~Hu and B.~Liu, ``Mining and summarizing customer reviews,'' in \emph{ACM SIGKDD}, 2004, pp. 168--177.
\bibitem{taboada2006methods}
M.~Taboada, C.~Anthony, and K.~D. Voll, ``Methods for creating semantic orientation dictionaries.'' in \emph{LREC}, 2006, pp. 427--432.
\bibitem{Amalia2018Meme}
A.~Amalia, A.~Sharif, F.~Haisar, D.~Gunawan, and B.~B. Nasution, ``{Meme Opinion} {Categorization} by {Using} ({OCR}) and {Na}{\" i}ve {Bayes} {Algorithm},'' in \emph{ICIC}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 10 2018.
\bibitem{Burbi2023}
G.~Burbi, A.~Baldrati, L.~Agnolucci, M.~Bertini, and A.~D. Bimbo, ``Mapping memes to words for multimodal hateful meme classification.''\hskip 1em plus 0.5em minus 0.4em\relax (IEEE), 12 2023, pp. 2824--2828.
\bibitem{Jannat2022}
N.~Jannat, A.~Das, O.~Sharif, and M.~M. Hoque, ``An empirical framework for identifying sentiment from multimodal memes using fusion approach,'' \emph{ICCIT}, 2022.
\bibitem{10652090}
S.~Zhang, O.~Faruk, R.~Porzel, D.~Küster, T.~Schultz, and H.~Liu, ``Examining the effects of human-likeness of avatars on emotion perception and emotion elicitation,'' in \emph{2024ABC}, 2024, pp. 1--12.
\bibitem{Sakai2020}
M.~Sakai, M.~Shuzo, M.~Yuasa, K.~Matsui, and E.~Maeda, \emph{Biological and Behavioral Information-Based Method of Predicting Listener Emotions Toward Speaker Utterances During Group Discussion}.\hskip 1em plus 0.5em minus 0.4em\relax Springer Singapore, Dec. 2020, p. 189–207.
\bibitem{RAMACHANDRA202411}
R.~A. Ramachandra, J.~Santhosh, A.~Dengel, and S.~Ishimaru, ``Enhancing stress detection for students: Exploring the impact of fine-tuning and user-specific data calibration in deep learning,'' \emph{Int. J. of Act.\&Beh. Comp.}, vol. 2024, no.~1, pp. 1--21, 2024.
\bibitem{Tazin2024}
T.~Tazin, T.~Hossain, S.~Hossain, and S.~Inoue, \emph{Understanding Mental Health Using Ubiquitous Sensors and Machine Learning: Challenges Ahead}.\hskip 1em plus 0.5em minus 0.4em\relax CRC Press, Mar. 2024, p. 222–257.
\bibitem{Michele2024}
A.~Michele, C.~Luca, C.~Giacomo, and V.~Martina, \emph{Detection of Self-Reported Stress Level from Wearable Sensor Data Using Machine Learning and Deep Learning-Based Classifiers: Is It Feasible?}\hskip 1em plus 0.5em minus 0.4em\relax CRC Press, Mar. 2024, p. 163–171.
\bibitem{10412045}
N.~Oryngozha, P.~Shamoi, and A.~Igali, ``Detection and analysis of stress-related posts in reddit’s acamedic communities,'' \emph{IEEE Access}, vol.~12, pp. 14\,932--14\,948, 2024.
\bibitem{WANG2024111778}
B.~Wang, S.~Huang, B.~Liang, G.~Tu, M.~Yang, and R.~Xu, ``What do they “meme”? a metaphor-aware multi-modal multi-task framework for fine-grained meme understanding,'' \emph{Knowledge-Based Systems}, vol. 294, p. 111778, 2024.
\bibitem{meme_emotion_new}
S.~Sharma, R.~S, M.~S. Akhtar, and T.~Chakraborty, ``Emotion-aware multimodal fusion for meme emotion detection,'' \emph{IEEE Trans. Affect. Comput.}, vol.~15, no.~3, p. 1800–1811, 2024.
\bibitem{memechat}
H.~Lu, Z.~Guo, C.~Li, Y.~Yang, H.~He, and S.~Bao, ``Towards building an open-domain dialogue system incorporated with internet memes,'' \emph{IEEE/ACM Transactions on Audio, Speech, and Language Processing}, vol.~32, pp. 721--726, 2024.
\bibitem{Weng2014Predicting}
L.~Weng, F.~Menczer, and Y.-Y. Ahn, ``Predicting {Successful} {Memes} {Using} {Network} and {Community} {Structure},'' \emph{AAAI Conf. on Web and Soc.Med.}, vol.~8, no.~1, pp. 535--544, may 16 2014.
\bibitem{Barnes2024}
K.~Barnes, P.~Juhász, M.~Nagy, and R.~Molontay, ``Topicality boosts popularity: a comparative analysis of nyt articles and reddit memes,'' \emph{SNAM}, vol.~14, no.~1, Jun. 2024.
\bibitem{Ferrara2013Clustering}
E.~Ferrara, M.~JafariAsbagh, O.~Varol, V.~Qazvinian, F.~Menczer, and A.~Flammini, ``Clustering memes in social media,'' in \emph{Proc. of the 2013 {IEEE}/{ACM} ASNAM}.\hskip 1em plus 0.5em minus 0.4em\relax ACM, aug 25 2013.
\bibitem{Jesus2020Semantic}
{Jesus Perez-Martin}, {B. Bustos}, and {M. Saldaña}, ``Semantic {Search} of {Memes} on {Twitter},'' \emph{arXiv.org}, 2020.
\bibitem{zannettou2018origins}
S.~Zannettou, T.~Caulfield, J.~Blackburn, E.~De~Cristofaro, M.~Sirivianos, G.~Stringhini, and G.~Suarez-Tangil, ``On the origins of memes by means of fringe web communities,'' in \emph{Proc. of the internet measurement conf. 2018}, 2018, pp. 188--202.
\bibitem{Singh2023}
R.~Singh, D.~Kumar, and M.~Gupta, ``Visual representation in political memes: A deep learning approach,'' \emph{Int. J. of Computer Vision and Social Media Analysis}, vol.~29, pp. 89--105, 2023.
\bibitem{Smitha2018Meme}
E.~S. Smitha, S.~Sendhilkumar, and G.~S. Mahalaksmi, \emph{Meme {Classification} {Using} {Textual} and {Visual} {Features}}.\hskip 1em plus 0.5em minus 0.4em\relax Springer International Publishing, 2018, pp. 1015--1031.
\bibitem{Cochrane2022}
L.~Cochrane, A.~Johnson, A.~Lay, and G.~Helmandollar, ``"one does not simply categorize a meme": A dual classification system for visual-textual internet memes,'' in \emph{Proceedings of the Linguistic Society of America}, 2022.
\bibitem{chhavi2020memotion}
C.~Sharma, W.~Paka, Scott, D.~Bhageria, A.~Das, S.~Poria, T.~Chakraborty, and B.~Gamb{\"a}ck, ``{Task Report: Memotion Analysis 1.0 @SemEval 2020: The Visuo-Lingual Metaphor!}'' in \emph{SemEval}.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2020.
\bibitem{kaggleRedditMemes}
``{R}eddit {M}emes {D}ataset --- kaggle.com,'' \url{https://www.kaggle.com/datasets/sayangoswami/reddit-memes-dataset/data}, [Accessed 31-12-2024].
\bibitem{Radford2021LearningTV}
A.~Radford, J.~W. Kim, C.~Hallacy, A.~Ramesh, G.~Goh, S.~Agarwal, G.~Sastry, A.~Askell, P.~Mishkin, J.~Clark, G.~Krueger, and I.~Sutskever, ``Learning transferable visual models from natural language supervision,'' in \emph{Int. Conf. on Machine Learning}, 2021.
\bibitem{Zhou2023}
\BIBentryALTinterwordspacing
N.~Zhou, D.~Jurgens, and D.~Bamman, ``Social meme-ing: Measuring linguistic variation in memes,'' 11 2023. [Online]. Available: \url{http://arxiv.org/abs/2311.09130}
\BIBentrySTDinterwordspacing
\bibitem{Luo2022CLIP4Clip}
H.~Luo, L.~Ji, M.~Zhong, Y.~Chen, W.~Lei, N.~Duan, and T.~Li, ``Clip4clip: An empirical study of {CLIP} for end to end video clip retrieval and captioning,'' \emph{Neurocomputing}, vol. 508, pp. 293--304, 10 2022.
\bibitem{Zhong2022RegionCLIP}
Y.~Zhong, J.~Yang, P.~Zhang, C.~Li, N.~Codella, L.~H. Li, L.~Zhou, X.~Dai, L.~Yuan, Y.~Li, and J.~Gao, ``Regionclip: Region-based {Language}-{Image} {Pretraining},'' in \emph{{IEEE}/{CVF}}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 6 2022.
\bibitem{Zhang2022PointCLIP}
R.~Zhang, Z.~Guo, W.~Zhang, K.~Li, X.~Miao, B.~Cui, Y.~Qiao, P.~Gao, and H.~Li, ``Pointclip: Point {Cloud} {Understanding} by {CLIP},'' in \emph{2022 {IEEE}/{CVF} ({CVPR})}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 6 2022.
\bibitem{Dorbala2022CLIP}
V.~S. Dorbala, G.~Sigurdsson, R.~Piramuthu, J.~Thomason, and G.~S. Sukhatme, ``Clip-{Nav}: Using {CLIP} for {Zero}-{Shot} {Vision}-and-{Language} {Navigation},'' 2022.
\bibitem{Lahajal2024Enhancing}
N.~K. Lahajal and H.~S, ``Enhancing {Image} {Retrieval} : A {Comprehensive} {Study} on {Photo} {Search} using the {CLIP} {Mode},'' 2024.
\bibitem{Sanh2019DistilBERT}
V.~Sanh, L.~Debut, J.~Chaumond, and T.~Wolf, ``Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,'' \emph{ArXiv}, vol. abs/1910.01108, 2019.
\bibitem{devlin2019bertpretrainingdeepbidirectional}
J.~Devlin, M.-W. Chang, K.~Lee, and K.~Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' 2019.
\bibitem{Adel2022}
H.~Adel, A.~Dahou, A.~Mabrouk, M.~Abd~Elaziz, M.~Kayed, I.~M. El-Henawy, S.~Alshathri, and A.~Amin~Ali, ``Improving crisis events detection using distilbert with hunger games search algorithm,'' \emph{Mathematics}, vol.~10, no.~3, p. 447, Jan. 2022.
\bibitem{igali2024trackingemotionaldynamicschat}
\BIBentryALTinterwordspacing
A.~Igali, A.~Abdrakhman, Y.~Torekhan, and P.~Shamoi, ``Tracking emotional dynamics in chat conversations: A hybrid approach using distilbert and emoji sentiment analysis,'' 2024. [Online]. Available: \url{https://arxiv.org/abs/2408.01838}
\BIBentrySTDinterwordspacing
\bibitem{saravia-etal-2018-carer}
E.~Saravia, H.-C.~T. Liu, Y.-H. Huang, J.~Wu, and Y.-S. Chen, ``{CARER}: Contextualized affect representations for emotion recognition,'' in \emph{Proc. of the 2018 Conf. on Emp. Met. in NLP}.\hskip 1em plus 0.5em minus 0.4em\relax Brussels, Belgium: Ass. for Comp. Ling., 2018, pp. 3687--3697.
\bibitem{Gokul2022Hate}
{Gokul Karthik Kumar} and {K. Nandakumar}, ``Hate-{CLIPper}: Multimodal {Hateful} {Meme} {Classification} based on {Cross}-modal {Interaction} of {CLIP} {Features},'' \emph{NLP4PI}, 2022.
\bibitem{Y2022On}
{Y. Qu}, {Xinlei He}, {S. Pierson}, {M. Backes}, {Yang Zhang}, and {Savvas Zannettou}, ``On the {Evolution} of ({Hateful}) {Memes} by {Means} of {Multimodal} {Contrastive} {Learning},'' \emph{IEEE Symp.Sec\&Pr}, 2022.
\end{thebibliography}