Exploring ChatGPT for Toxicity Detection in GitHub
Extracted source text · arXiv 2312.13105
\documentclass[sigconf]{acmart}
\AtBeginDocument{%
\providecommand\BibTeX{{%
\normalfont B\kern-0.5em{\scshape i\kern-0.25em b}\kern-0.8em\TeX}}}
\setcopyright{acmcopyright}
\copyrightyear{2023}
\acmYear{2023}
\acmDOI{XXXXXXX.XXXXXXX}
\acmConference[ICSE 2024]{46th International Conference on Software Engineering}{April 2024}{Lisbon, Portugal}
\usepackage{enumitem}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{colortbl}
\usepackage{geometry}
\usepackage{lipsum}
\usepackage{tikz}
\usetikzlibrary{shapes.multipart}
\usepackage{titlesec}
\usepackage{tabularx}
\usepackage{graphics}
\definecolor{aliceblue}{rgb}{0.94, 0.97, 1.0}
%\title{Exploring a Zero-shot LLM to Detect Toxic Texts in the Open Source Community}
\title{Exploring ChatGPT for Toxicity Detection in GitHub}
%\begin{comment}
\author{Shyamal Mishra}
\affiliation{%
\institution{Drexel university}
\streetaddress{Philadelphia}
\city{Philadelphia}
\state{Pennsylvania}
\country{USA}}
\email{sm4825@drexel.edu}
\author{Preetha Chatterjee}
\affiliation{%
\institution{Drexel university}
\streetaddress{Philadelphia}
\city{Philadelphia}
\state{Pennsylvania}
\country{USA}}
\email{preetha.chatterjee@drexel.edu}
%\end{comment}
\begin{document}
\begin{abstract}
Fostering a collaborative and inclusive environment is crucial for the sustained progress of open source development. However, the prevalence of negative discourse, often manifested as toxic comments, poses significant challenges to developer well-being and productivity.
To identify such negativity in project communications, especially within large projects, automated toxicity detection models are necessary.
To train these models effectively, we need large software engineering-specific toxicity datasets. However, such datasets are limited in availability and often exhibit imbalance (e.g., only 6 in 1000 GitHub issues are toxic)~\cite{Raman2020}, posing challenges for training effective toxicity detection models. To address this problem, we explore a zero-shot LLM (ChatGPT) that is pre-trained on massive datasets but without being fine-tuned specifically for the task of detecting toxicity in software-related text. Our preliminary evaluation indicates that ChatGPT shows promise in detecting toxicity in GitHub, and warrants further investigation. We experimented with various prompts, including those designed for justifying model outputs, thereby enhancing model interpretability and paving the way for potential integration of ChatGPT-enabled toxicity detection into developer communication channels.
%Lastly, we integrate ChatGPT with the Slack chat platform, creating a bot that holds the potential to proactively mitigate toxicity in developer communication channels. %to show potential integration of ChatGPT-enabled toxicity detection
%This paper delves into the identification and understanding of such toxicity within the open source developer community. Through rigorous examination, we elucidate the rationale behind these adverse interactions. Furthermore, we propose alternative strategies to effectively mitigate the impact of harmful commentary, thereby promoting a healthier and more constructive collaborative landscape. Our findings provide valuable insights for both practitioners and stakeholders seeking to uphold the principles of open source development while curtailing the detriments posed by toxic behavior.
\end{abstract}
\maketitle
\input{intro}
%\input{soa}
\input{methodology}
\input{results}
%\input{threats}
\input{conclusion}
\bibliographystyle{IEEEtran}
\bibliography{icse_merged_bib}
\end{document}
\section{Introduction}
The open source software (OSS) development community has emerged as a powerful engine driving innovation and collaboration across various domains. The fundamental principle of collective problem solving has led to the creation of exceptional software projects. However, amidst this collaborative landscape, the prevalence of negative discourse, often manifested as toxic comments, %poses a significant challenge to the harmony and productivity of the community
causes substantial harm on developers, diminishing their well-being, motivation, job satisfaction, and productivity~\cite{Raman2020, blog_exp1, blog_exp5, Miller2022, ferreiraSTFU2021, ferreiraHeat2022, gunawardena_destructive_2022, gachechiladze_anger_2017, Cheriyan21, Ehsani2023_FSE}. In a 2017 GitHub survey, it was found that 50\% of OSS developers experienced negative interactions. Among those, 21\% mentioned that such behavior led them to stop contributing~\cite{oss_survey}.
Mitigating negativity is essential to cultivate healthier and more productive software environments and retain talent~\cite{egelmanPredicting2020, qiuDetecting2022, ortu2015bullies, Destefanis2016SoftwareDD}. Collaboration platforms such as GitHub has implemented preliminary measures to tackle negative behavior, such as issue locking~\cite{locking}, but such manual inspection remains labor-intensive due to the sheer volume of daily content. More recently, machine learning-based techniques for detecting toxicity in software-related text have started emerging~\cite{Raman2020, ferreiraIncivility2022, sarkerAutomated2022, egelmanPredicting2020}. However, these models often show a high false positive rate and limited generalizability across different communication types (e.g., issue comments , code reviews)~\cite{sarkerBenchmark2020}.
Creating effective toxicity detection models is challenging due to limited open-source toxicity datasets~\cite{Miller2022, ferreiraIncivility2022, imran2022data}. Additionally, these datasets are often imbalanced, with just 6 out of 1000 issues being toxic~\cite{Raman2020}, making training effective models a hurdle.
Therefore, there is a significant gap in the development and adoption of effective automated techniques for identifying toxic conversations in software engineering platforms.
%While advanced techniques for detecting harassment in blogs and social media are available~\cite{gunasekara-nejadgholi-2018-review, bhat-etal-2021-say-yes}, studies have shown that directly applying these tools to software-related text is ineffective~\cite{Raman2020, sarkerBenchmark2020} due to the presence of technical terms and nuanced negative behavior (e.g., prevalence of sarcasm, arrogance on GitHub).
Large Language Models (LLMs) have recently gained prominence as a powerful category of deep learning technique, demonstrating their effectiveness in related tasks such as emotion and sentiment analysis~\cite{batra2021bert, eeshita-sentiment, kamath2022emoroberta, liu2019dens, kabir2023answers, hou2023large, zhang2023sentiment, fan2023large, 10132255, Huang_2023, li2023hot, he2023prompt, ziems2023large}.
Some of the most substantial and influential LLMs, such as ChatGPT~\cite{chatgpt}, are readily available that can be deployed as a ``zero-shot" model, without requiring specific fine-tuning for a particular task. Given that creating an extensive training dataset for toxicity detection in software engineering communication is costly and resource-intensive, utilizing a zero-shot approach offers an attractive alternative.
In this paper, we explore ChatGPT~\cite{chatgpt} in detecting toxicity in software-related text. We use a benchmark toxicity dataset of 1597 GitHub issue comments~\cite{Raman2020} to evaluate the model, and conduct a qualitative analysis to understand the common errors. Specifically, we investigate the following research questions:
\vspace{-0.4cm}
\begin{itemize}[leftmargin=*]
\item RQ1. How effective is OpenAI's ChatGPT %compared to existing state-of-the-art toxicity detection tools (e.g., Perspective API, Raman et al., Sarkar et al.)
in detecting toxic text on GitHub?
\item RQ2. What types of toxic comments are misclassified or difficult to detect using ChatGPT?
\end{itemize}
\vspace{-0.4cm}
Our preliminary findings suggest that ChatGPT shows promise in detecting toxicity
in GitHub, achieving a precision of 0.49 and recall of 0.94. This result is comparable to the state-of-the-art domain-specific toxicity detectors (e.g., Raman et al.~\cite{Raman2020} showed a precision of 0.91 and a recall of 0.42 on the same dataset).
We explore different prompts, including those intended for explaining model results, thus improving model interpretability, and thus opening up possibilities for integrating ChatGPT-supported toxicity detection into developer communication platforms. We also show an example implementation of possible proactive moderation of toxic content by integrating ChatGPT into Slack, a popular developer chat platform.
%To demonstrate the effectiveness of ChatGPT in SE applications, we conduct a short case study to assess if it can be embedded in developer communication platforms to proactively mitigate toxic conversations. Specifically, we integrate ChatGPT as a bot in developer chat platforms (i.e., Slack). First, it tags toxic content. Second, it suggests alternate suggestions. Case Study: Can we use Open AI to rephrase toxic content and come up with alternate suggestions as a bot?
%In the pursuit of a comprehensive analysis, we delve into the rationale behind toxic comments, aiming to uncover the motivations and triggers that drive people to engage in harmful discourse. Furthermore, we endeavor to present alternative strategies and approaches that can effectively mitigate the impact of toxic comments while fostering a more respectful and constructive environment.
\section{Methodology}
\subsection{Dataset}
We analyze a benchmark dataset of 1597 GitHub issue comments (102 toxic, 1495 non-toxic) manually curated by Raman et al. \cite{Raman2020}. First, they used the GitHub API to identify issue threads that had been locked as ``too heated" among 118k GitHub projects. Of the 294k locked issues, 654 were explicitly marked as ``too heated". Several of these locked discussions contained toxic behavior. Next, they manually reviewed these discussions, labeling comments as toxic or non-toxic.
\begin{comment}
Our approach to choosing data for our dataset was thorough, aiming to create a complete and balanced collection of both toxic and non-toxic comments related to software issues. In our study, we utilized datasets sourced from previous research contributions by Raman et al. These datasets collectively encompassed 1597 comments that were meticulously labeled by the authors as either "toxic" or "non-toxic" following independent manual assessment.
Table 1 offers an overview of the dataset we use in this paper.
\begin{table}[htbp]
\centering
\caption{Overview of Datasets}
\begin{tabular}{|c|c|c|c|c|}
\hline
\textbf{Dataset} & \textbf{Source} & \textbf{Toxic} & \textbf{Non-Toxic} & \textbf{Total} \\
\hline
Raman et al. & github & 102 & 1495 & 1597 \\
\hline
\end{tabular}
\end{table}
\end{comment}
%During our analysis, we encountered instances that were initially categorized as non-toxic, such as phrases like "Maybe when I state that the values in the extractor are correct, people should listen!!!" However, upon closer examination and manual assessment, we recognized that these comments could indeed be perceived as toxic in certain contexts. This realization underscored the intricate challenge of accurately assessing the presence of toxicity.
%Conversely, there were instances initially labeled as toxic, but upon closer examination, we found them to be more benign. For instance, statements like "PROMISE ALSO WORKS WHEN USED WITHOUT AWAIT..Please Check" were categorized as toxic initially. However, as we reevaluated them, we interpreted them more as expressions of frustration or humor rather than genuine toxicity.
%Our careful selection of examples aimed to demonstrate the diverse ways in which individuals can convey unkindness or negativity within software-related discussions. This approach was methodical, relying on datasets from other researchers and meticulously selecting comments that aligned with our research goals to
%\subsubsection{Annotation Process}
%Our initial rubric was founded upon Sarkar et al.'s toxicity category guidelines \cite{sarkerBenchmark2020}), providing a strong framework that directed our annotation procedure. This framework includes nine rules, each with a clear rationale and examples.
%The first seven rules identify and flag toxic content, including profanity, insults, identity attacks, aggression, sexual references, flirtations, and threats. These rules are crucial for maintaining a collaborative and respectful SE discourse. Rule 8 adds nuance, recognizing that certain negative terms, when applied to oneself or one's work in the SE context, may not imply toxicity. This acknowledges the SE community's unique language dynamics, where such terms can be used to admit errors without harmful intent. Lastly, Rule 9 categorizes text as non-toxic if it doesn't fit the previous rules, accommodating general comments and discussions without harmful elements.
%It's important to note that our criteria for selecting data were not solely based on the categories outlined by Sarkar et al \cite{sarkerBenchmark2020}). We aimed for a broader representation of toxicity within the software engineering community. Our deliberate strategy involved actively seeking instances that spanned a diverse range of toxic categories. This approach allowed us to include various types of negative interactions commonly seen in software discussions, providing a well-rounded representation of toxic content.
%Including these diverse examples sheds light on the complexity of distinguishing mean comments from neutral ones. Analyzing these examples closely helps us to better understand the various ways people communicate and how context can influence the interpretation of their comments. Ultimately, this approach enhances the accuracy and usefulness of our dataset for studying negative behavior in software discussions.
%\subsubsection{Data Prepossessing}
%Our aim in this prepossessing was to create a clean and well-structured dataset, free from irrelevant or empty data points. By systematically removing blanks, toxic labels without corresponding text, and unlabeled text entries, we refined the dataset to enhance its suitability for subsequent analysis
\subsection{ OpenAI Chat Completion API}
We employ OpenAI's GPT-3.5 Turbo through the Chat Completion API to investigate toxicity in software developer community conversations. The Chat Completion API allows us to generate responses to a fixed prompt, simulating a conversation with the model. We use this API to create a conversational setting where we can evaluate the content for potential toxicity.
\subsection{Prompt Design}
Effective prompt design is a critical aspect of utilizing OpenAI APIs to achieve desired outcomes in natural language processing tasks. The choice of prompt significantly influences the quality, relevance, and accuracy of the model's responses. Our approach involves designing specific prompts that guide the model's responses in evaluating the level of toxicity in the provided conversation. The prompts were carefully designed by using the existing literature on this topic~\cite{stowe2022impli, zhou2021pie, haagsma2020magpie, white2023prompt}.
We conducted a series of experiments employing various prompt structures and formulations. The goal of these experiments was to identify the most effective prompt design that aligns with our research objective of detecting toxicity in software-related text. We tested 15 different prompts, each with unique characteristics, instructions, and response options.
\subsubsection{Prompt Variations.}
We explored few strategies to design the prompt to detect toxicity. Our prompt variations are guided by Google's Conversation AI research in toxicity detection~\cite{CAT}. Some of the prompt characteristics we investigated include:
\begin{itemize}[leftmargin=*]
\item The category of toxicity labels; i.e., binary (yes/no), multiclass (e.g., "Very Toxic," "Toxic," "Slightly Toxic")
%\item The inclusion of explicit toxicity descriptors (e.g., "Very Toxic," "Toxic," "Slightly Toxic").
\item The framing of the task; i.e., question-based (e.g., ``Does the following comment exhibit toxicity?"), statement-based (e.g., ``Assess the following statement for any indication of toxicity.")
%\item The length and details of the task
\end{itemize}
%\subsubsection{Observations and Findings}
Through these experiments, we made the following observations:
\begin{itemize}[leftmargin=*]
\item Clarity of task instruction: Prompts that provided clear and specific instructions led to more consistent and accurate assessments of toxicity. Ambiguity in prompts often resulted in less reliable outcomes.
\item Length of task instruction: While detailed prompts were helpful in conveying the desired task, overly lengthy/complex instructions sometimes led to model confusion, and verbosity in the outputs. Striking the right balance between clarity and length was essential for better performance.
\item Classification Category: The use of multi-level toxicity categories, such as "Very Toxic," "Toxic," and "Slightly Toxic," allowed for better understanding of subtle or nuanced forms of toxicity (e.g., passive aggressiveness).
\end{itemize}
\subsubsection{Selected Prompts.}
Based on the observations of our experiments, we selected three prompts that demonstrated effectiveness in detecting toxicity (in terms of our evaluation metrics). %These prompts were employed in our research to evaluate the model's performance in detecting toxicity in software developer community discussions.
Due to the lack of space, we present only the 3 selected prompts as follows:
\begin{table}[htbp]
\setlength{\tabcolsep}{2pt}
\renewcommand{\arraystretch}{1.5}
\centering
\caption{Prompt Descriptions for Toxicity Detection}
\vspace{-0.2cm}
\begin{tabular}{|p{0.15\linewidth}|p{0.75\linewidth}|}
\hline
%\textbf{Prompt No} & \textbf{ Prompt Descriptions} \\
%\hline
Prompt 1 & Given the following software developer conversation, do you think this contains any toxicity? Please provide your answer in \textit{yes/no}. \\
\hline
Prompt 2 & Examine whether the software engineering community discussion includes toxicity. Please provide your answer in \textit{Very Toxic/Toxic/Slightly Toxic/Not Toxic}. \\
\hline
Prompt 3 & Examine whether the open-source developer community discussion includes toxicity.
\newline \quad \textit{Very Toxic} - a very hateful, aggressive, disrespectful comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective.
\newline \quad \textit{Toxic} - a rude, disrespectful, unreasonable comment or otherwise somewhat likely to make a user leave a discussion or give up on sharing their perspective.
\newline \quad \textit{Slightly Toxic} or hard to say.
\newline \quad \textit{Not toxic.} \\
\hline
\end{tabular}
\end{table}
\vspace{-0.2cm}
\subsection{Model Parameters} LLM generated outputs can be non-deterministic~\cite{ouyang2023llm}. We conducted extensive experimentation with different temperature settings to observe their impact on toxicity detection. Temperature influences how this model generates text~\cite{openai_temp}. When set to a lower value, such as 0.2, the generated text becomes more deterministic, honing in on specific patterns and producing relatively conservative, "safe" responses. In contrast, increasing the temperature to higher values, such as 1.0 or above, introduces an element of randomness, bestowing the text with greater diversity and creativity. We used three temperature values; 0.2, 0.7, and 1.2 for our experiments. For the rest of the parameters, we used the default settings.
\subsection{Evaluation Metrics} We evaluate the model's performance using a set of metrics that are frequently used to evaluate classification tasks: {\em F1-score}, {\em Precision}, and {\em Recall}. These metrics provide insights into the model's ability to identify true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).
Precision is the ratio of TP instances to the total predicted positive instances (i.e., $\text{\em Precision} =\frac{TP}{TP +FP}$), and Recall is the ratio of TP instances to all instances in the positive class (i.e., $\text{\em Recall} =\frac{TP}{TP +FN}$). {\em F1-score} is the harmonic mean of {\em Precision} and {\em Recall} (i.e., $\text{\em F1-score} = 2 * \frac{Precision * Recall}{Precision + Recall}$).
For calculating the evaluation metrics, we need to convert the toxicity labels of the model outputs to binary (toxic/non-toxic). Prompt 1 outputs are already in this format. For Prompts 2 and 3, we consider \textit{Very Toxic/Toxic/Slightly Toxic} as `toxic', and not toxic as `non-toxic'.
\definecolor{graybackground}{RGB}{220, 220, 220}
\section{Preliminary Observations} In this section, we discuss our observations in using ChatGPT to detect toxic instances of GitHub issue comments.
\subsection{Evaluating Effectiveness}
\colorbox{graybackground}{%
\parbox{\dimexpr\linewidth-2\fboxsep}{%
\textcolor{black}{%
\textit{RQ1. How effective is OpenAI’s ChatGPT in detecting toxic text on GitHub?}%
}%
}%
}
\noindent
Table \ref{evaluation} shows the results of the three selected prompts with varying temperature values (0.2, 0.7, 1.2) in detecting toxicity in the Raman et al. GitHub issue comments dataset.
We observe that the choice of prompts has a notable impact on the performance of the model in toxicity detection. Among all the three prompts, Prompt 1 produced most effective results, achieving the highest F-score of 0.64 with temperature=0.2, 0.55 with temperature=0.7, and 0.54 with temperature=1.2. There could be several reasons for this outcome such as:
(a) Simplicity: Prompt 1's straightforward question simplifies the task for the model. It only requires determining whether toxicity is present or not (binary classification);
%(b) Binary Classification: Binary classification often leads to a better balance between precision and recall, making it easier for the model to achieve a high F-measure.
(b) Lack of Ambiguity: Prompt 1 avoids fine-grained distinctions between toxicity levels, reducing ambiguity in decision.
\begin{table}[htbp]
\centering
\caption{Results for Toxicity Detection}
\vspace{-0.2cm}
\label{evaluation}
\begin{tabular}{|c|c|c|c|c|}
\hline
Temperature & Prompt & Precision & Recall & F-measure \\
\hline
\multirow{0.2}{*}{0.2} & Prompt 1 & 0.49 & 0.94 & \textbf{0.64} \\
\cline{2-5}
& Prompt 2 & 0.36 & 0.88 & 0.51 \\
\cline{2-5}
& Prompt 3 & 0.33 & 0.78 & 0.48 \\
\hline
\multirow{0.7}{*}{0.7} & Prompt 1 & 0.43 & 0.49 & \textbf{0.55} \\
\cline{2-5}
& Prompt 2 & 0.35 & 0.89 & 0.50 \\
\cline{2-5}
& Prompt 3 & 0.39 & 0.71 & 0.51 \\
\hline
\multirow{1.2}{*}{1.2} & Prompt 1 & 0.41 & 0.80 & \textbf{0.54} \\
\cline{2-5}
& Prompt 2 & 0.59 & 0.42 & 0.49 \\
\cline{2-5}
& Prompt 3 & 0.29 & 0.86 & 0.43 \\
\hline
\end{tabular}
\end{table}
We also notice that the choice of temperature can significantly influence the trade-off between precision and recall in toxicity detection. For our dataset, a temperature setting of 0.2 (lower temperature values produce deterministic outputs) provided the best results. More experiments are required to explore ChatGPT's model determinism.
%This choice was motivated by the fact that lower temperature settings tend to yield outcomes that are consistently uniform, whereas higher settings foster greater diversity and creativity in the generated content, as documented in \cite{openai_temp}.
\subsection{Identifying Challenges in Toxic Comment Detection}
\colorbox{graybackground}{%
\parbox{\dimexpr\linewidth-2\fboxsep}{%
\textcolor{black}{%
\textit{RQ2. What types of toxic comments are misclassified or difficult to detect using ChatGPT?}%
}%
}%
}
\noindent
In order to gain insight into the models’ mistakes, we conduct an error analysis on the FPs and FNs for the overall best performing configuration (Temperature=0.2, Prompt=1). \textit{False Positives (FP)} occur when the model incorrectly identifies non-toxic comments as toxic; %In other words, it flags content as toxic when it isn't.
\textit{False Negatives (FN)} occur when the model fails to identify toxic comments. %It fails to flag content as toxic when it should.
%Due to the remarkably high consensus among the OpenAI API on instances of false positives (FP), our error analysis starts with analyzing these 106 FP instances.\\
\subsubsection{False Positives (FP):} We observed a total of 100 FP instances, reflecting a low precision of 0.49. Our manual examination reveals that the FP errors can be classified into a few primary categories, which are elaborated below:
%"Getting defensive? You know what you did." may carry a negative tone but lack explicit offensive language
\textbf{Absence of Explicit Offensive Language:} Instances that do not contain explicit offensive words or hate speech, typical indicators of toxicity, are misinterpreted. One such example in our dataset is, "\textit{Such a plugin already exists. And you chose to use a bleeding-edge build with it removed. You have nobody to blame but yourself.}". %and "@XX I actually unlocked this issue again to give you a chance to say sorry..."
\textbf{Sarcasm and Irony:} OpenAI API often encountered challenges in accurately detecting figurative language, such as sarcasm and irony. For example, statements such as, ``\textit{Really, thanks to you, I got ruined the world of survival and now have to do the cleaning map}", %and "So basically you have an extremely easy to exploit security vulnerability and decided to only fix this in the paid Enterprise Edition?"
containing sarcasm was misinterpreted.
%\textbf{Complex Linguistic Structures:} The presence of complex language structures or intricate arguments in these examples can obscure toxicity from automated detection systems. For instance, "Maybe you and I have different definitions of 'seriously.'..." and "Good job, then what do you want from us? Stop mentioning me."
\textbf{Context-Dependent Toxicity:} Toxicity often depends on the context, and statements that seem non-toxic in isolation may be perceived as toxic when considered within a broader context. Some examples are, ``I didn't file a bug for Fedora, people are aware that Fedora is an unstable system and it is a test bed for RHEL." and ``I hava a 512m vps, I want to build some website on it..."
\subsubsection{False Negatives (FN):} We observed only 6 FN instances, reflecting a high recall of 0.94. Our manual examination reveals that most of the FN errors can be classified into one broad category as follows:
\textbf{Nuanced Toxicity:} ChatGPT faces challenges in detecting nuances forms of toxicity, such as arrogance and passive-aggressiveness. For example, a comment like, ``As I said above... Issue 87 was a harmless use-after-free on shutdown, and nothing to do with this. I've said my piece, and I can see it was a mistake trying to engage with you, so I'm locking this thread," was misclassified, despite exhibiting arrogance.
\subsubsection{Common errors in FP and FN:} Across FPs and FNs, we observed some common error patterns.
\textbf{Lengthy Phrasing: }One factor influencing both FP and FN instances is the tokenization of text by OpenAI language models. Language models read and write text in chunks called tokens, where a token can range from a single character to an entire word. For example, the sentence "ChatGPT is great!" is encoded into six tokens: ["Chat", "G", "PT", " is", " great", "!"] \cite{openai_tokens}. This tokenization process can occasionally lead to misunderstandings, particularly when dealing with lengthy phrases or complex sentences.
\textbf{Non-Responsive Answers:} Another challenge arises when the model fails to provide answers that align with our query. For instance like, "@friend Done. Cached for , as spotted in Doctrine. Is there any relevant difference?" the model responded with, "I'm sorry, but I cannot provide a yes or no answer to this question as it requires subjective analysis of the software engineering community discussion," which indicating a failure in understanding the context or intent of the question.
\begin{table}[htbp]
\caption{Error Categories with Their Frequency (Temperature 0.2, Prompt=1)}
\vspace{-0.2cm}
\centering
\begin{tabular}{|p{6cm}|c|}
\hline
\textbf{Category} & \textbf{Count} \\
\hline
Labeling error & 46\\
Absence of Explicit Offensive Language & 23 \\
Sarcasm and Irony & 16 \\
%Complex Linguistic Structures & 7 \\
Nuanced Toxicity & 6 \\
Non-Responsive Answer & 6\\
Context-Dependent Toxicity & 5 \\
Lengthy Phrasing & 4\\
\hline
\end{tabular}
\end{table}
\textbf{Labeling error:} We observed 46 instances of human error in labelling the dataset. These errors could be attributed to various factors, including the subjective nature of human annotation and possible misinterpretation of contextual cues. Additionally, some comments can be interpreted as a retaliation to a previous toxic comment (e.g., \textit{``Alright.... Nobody accused you 'falsely' you clearly were not respectful}"), or more subtle forms of behavior such as entitlement (e.g., ``Maybe when I state that the values in the extractor are correct, people should listen!!!"). Both examples were labeled non-toxic by the annotators, but detected as toxic by ChatGPT. %the sheer volume of data to assess and the possibility of moderator fatigue or cognitive biases during the classification process.
\section{Discussion and Implications}
%Implications for OSS communities to curb toxicity
To determine the viability of ChatGPT for use by OSS communities in combating toxicity, we explored two key factors: (a) model interpretability, essential to provide meaningful insights and actionable information; and (b) potential avenues for integrating ChatGPT effectively into the software engineering workflow.
\subsection{Model Interpretability}
Model interpretability revolves around our ability to understand and explain the rationale behind the decisions made by ChatGPT. In the context of combating toxicity within OSS communities, interpretability ensures that the insights and information provided by the model are not only accurate but also comprehensible. This clarity empowers OSS practitioners to take informed actions and address toxicity effectively.
We used the same dataset to explore a few prompts that could help us understand the classification rationale. %We observed that asking for "a short justification" often is sufficient to gain some insights into the model outputs.
Table \ref{prompts} shows sample outputs of such justifications.
\begin{table}[htbp]
\centering
\caption{Sample Prompt Outputs for Toxicity Justification}
\vspace{-0.2cm}
\begin{tabular}{|p{8cm}|}
\hline
\rowcolor{aliceblue}\textbf{Prompt:} Examine whether the software engineering community discussion includes toxicity. Please provide your answer in yes/no with a short justification. \\ \hline \hline
\textbf{Statement} - \textit{you're useless, please leave}\\
\textbf{ChatGPT Output:}
Yes. It includes personal attacks and a disrespectful tone.
\\\hline
%\textbf{Prompt:} Examine whether the software engineering community discussion includes toxicity. Please provide your answer in yes/no with a short justification.
\textbf{Statement} - \textit{Pardon my English @friend, what you wrote is bullshit.}\\
\textbf{ChatGPT Output:}
Yes. The use of the word "bullshit" is offensive, which can contribute to a toxic environment.
\\\hline
%\textbf{Prompt:} Examine whether the software engineering community discussion includes toxicity. Please provide your answer in yes/no with a short justification.
\textbf{Statement} - \textit{I'll report you too @friend, for all the internet to see how disgusting and abusive you are}\\
\textbf{ChatGPT Output:}
Yes. The statement provided includes a threat to publicly shame and report someone, which can be seen as a form of toxicity. This kind of behavior is not constructive and can create a hostile environment within the community.'
\\\hline
\end{tabular}
\label{prompts}
\end{table}
\subsection{Integration into the Workflow}
We explored the potential of developer chats as platforms for integrating preemptive interventions, as they facilitate spontaneous expression of emotions often not captured in other communication types~\cite{Chatterjee21, Kuutila2020ChatAI, DARE_23}. Specfically, we integrated ChatGPT into Slack, a popular chat platform, that would be able to automatically detect toxic content, and when necessary, generate an alternative, non-toxic version. This interface will detect harmful text, provide users with real-time feedback on the tone of their message, and suggestions to reframe their messages to convey positive and constructive intentions, as shown in Figure \ref{fig:image_label}. Additional details on the bot implementation is included in our replication package.
%Additionally, when toxicity is detected in the input content, we employ prompts to guide OpenAI's GPT-3.5 Turbo in generating non-toxic paraphrased versions.The prompt structure for toxicity examination is as follows:
\vspace{-0.2cm}
\begin{figure}
\centering
\includegraphics[width=0.48\textwidth]{figure-1.png}
\vspace{-0.6cm}
\caption{Bot Implementation for Paraphrasing Toxic Text}
\label{fig:image_label}
\end{figure}
%\subsection{Fine-Tuning for Software Engineering Communities.} Explore the possibility of fine-tuning language models like GPT-3.5 Turbo on data specific to software developer communities~\footnote{\url{https://platform.openai.com/docs/guides/fine-tuning}}. Tailoring the model to the language and context commonly used within these communities could enhance the accuracy of toxicity detection.
\section{Conclusion and Future Work}
In this paper, we presented an approach for automated toxicity detection in software developer communication using a zero-shot LLM, namely ChatGPT, through a prompting approach. We evaluated GPT-3.5 Turbo on a previously curated dataset of GitHub issue comments, and observed promising preliminary results. We explored several prompts for toxicity detection, including those that outputs justification of model outputs. This holds particular significance in building trust among software engineers, encouraging their adoption and daily use of such systems in their workflows. We publish the source code and annotated dataset to facilitate the replication of our study at: \url{https://anonymous.4open.science/r/open-source-toxicity-0236/README.md}.
There are several avenues for future work. First, our study focused only on one type of developer communication, i.e., issue comments on GitHub. Future studies should evaluate our approach on larger and more diverse datasets, such as code reviews or emails. Second, further work on prompt engineering could potentially help improve ChatGPT's performance on toxicity detection in software-related text. Third, there is a possibility of fine-tuning language models like GPT-3.5 Turbo on data specific to software developer communities~\footnote{\url{https://platform.openai.com/docs/guides/fine-tuning}}. Tailoring the model to the language and context commonly used within these communities could potentially enhance the accuracy of toxicity detection. Overall, our study provides a starting point for future research to explore the potential of ChatGPT in detecting toxicity or incivil language in software engineering communication.
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{Raman2020}
\BIBentryALTinterwordspacing
N.~Raman, M.~Cao, Y.~Tsvetkov, C.~K\"{a}stner, and B.~Vasilescu, ``Stress and burnout in open source: Toward finding, understanding, and mitigating unhealthy interactions,'' in \emph{Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results}, ser. ICSE-NIER '20.\hskip 1em plus 0.5em minus 0.4em\relax New York, NY, USA: Association for Computing Machinery, 2020, p. 57–60. [Online]. Available: \url{https://doi.org/10.1145/3377816.3381732}
\BIBentrySTDinterwordspacing
\bibitem{blog_exp1}
``{What it feels like to be an open-source maintainer},'' \url{https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-an-open-source-maintainer/}, 2017, [Online; accessed 2-Jun-2023].
\bibitem{blog_exp5}
J.~Lehnardt, ``{Sustainable Open Source: The Maintainers Perspective or: How I Learned to Stop Caring and Love Open Source},'' \href{https://writing.jan.io/2017/03/06/sustainable-open-source-the-maintainers-perspective-or-how-i-learned-to-stop-caring-and-love-open-source.html}{https://writing.jan.io/2017/03/06/sustainable-open-source-the-maintainers-perspective-or-how-i-learned-to-stop-caring-and-love-open-source.html}, 2017, [Online; accessed 4-Jun-2023].
\bibitem{Miller2022}
C.~Miller, S.~Cohen, D.~Klug, B.~Vasilescu, and C.~Kästner, ``“did you miss my comment or what?” understanding toxicity in open source discussions,'' in \emph{2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)}, 2022, pp. 710--722.
\bibitem{ferreiraSTFU2021}
\BIBentryALTinterwordspacing
I.~Ferreira, J.~Cheng, and B.~Adams, ``The ``shut the f**k up" phenomenon: Characterizing incivility in open source code review discussions,'' vol.~5, pp. 1--35, 2021-10-13. [Online]. Available: \url{http://arxiv.org/abs/2108.09905}
\BIBentrySTDinterwordspacing
\bibitem{ferreiraHeat2022}
\BIBentryALTinterwordspacing
I.~Ferreira, B.~Adams, and J.~Cheng, ``How heated is it? understanding {GitHub} locked issues,'' in \emph{Proceedings of the 19th International Conference on Mining Software Repositories}, 2022-05-23, pp. 309--320. [Online]. Available: \url{http://arxiv.org/abs/2204.00155}
\BIBentrySTDinterwordspacing
\bibitem{gunawardena_destructive_2022}
\BIBentryALTinterwordspacing
S.~D. Gunawardena, P.~Devine, I.~Beaumont, L.~P. Garden, E.~Murphy-Hill, and K.~Blincoe, ``Destructive {Criticism} in {Software} {Code} {Review} {Impacts} {Inclusion},'' \emph{Proceedings of the ACM on Human-Computer Interaction}, vol.~6, no. CSCW2, pp. 292:1--292:29, Nov. 2022. [Online]. Available: \url{https://dl.acm.org/doi/10.1145/3555183}
\BIBentrySTDinterwordspacing
\bibitem{gachechiladze_anger_2017}
D.~Gachechiladze, F.~Lanubile, N.~Novielli, and A.~Serebrenik, ``Anger and {Its} {Direction} in {Collaborative} {Software} {Development},'' in \emph{2017 {IEEE}/{ACM} 39th {International} {Conference} on {Software} {Engineering}: {New} {Ideas} and {Emerging} {Technologies} {Results} {Track} ({ICSE}-{NIER})}, May 2017, pp. 11--14.
\bibitem{Cheriyan21}
\BIBentryALTinterwordspacing
J.~Cheriyan, B.~T.~R. Savarimuthu, and S.~Cranefield, ``Towards offensive language detection and reduction in four software engineering communities,'' in \emph{Evaluation and Assessment in Software Engineering}, ser. EASE 2021.\hskip 1em plus 0.5em minus 0.4em\relax New York, NY, USA: Association for Computing Machinery, 2021, p. 254–259. [Online]. Available: \url{https://doi.org/10.1145/3463274.3463805}
\BIBentrySTDinterwordspacing
\bibitem{Ehsani2023_FSE}
R.~Ehsani, R.~Rezapour, and P.~Chatterjee, ``{Exploring Moral Principles Exhibited in Software‐related Text: A Case Study on GitHub Locked Issues},'' in \emph{Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering: Ideas, Visions and Reflections Track}, ser. FSE '23, 2023.
\bibitem{oss_survey}
GitHub, ``{Open Source Survey},'' \url{https://opensourcesurvey.org/2017/}, 2017, [Online; accessed 22-May-2023].
\bibitem{egelmanPredicting2020}
\BIBentryALTinterwordspacing
C.~D. Egelman, E.~Murphy-Hill, E.~Kammer, M.~M. Hodges, C.~Green, C.~Jaspan, and J.~Lin, ``Predicting developers' negative feelings about code review,'' in \emph{Proceedings of the {ACM}/{IEEE} 42nd International Conference on Software Engineering}, ser. {ICSE} '20.\hskip 1em plus 0.5em minus 0.4em\relax Association for Computing Machinery, 2020-10-01, pp. 174--185. [Online]. Available: \url{https://dl.acm.org/doi/10.1145/3377811.3380414}
\BIBentrySTDinterwordspacing
\bibitem{qiuDetecting2022}
\BIBentryALTinterwordspacing
H.~S. Qiu, B.~Vasilescu, C.~Kästner, C.~Egelman, C.~Jaspan, and E.~Murphy-Hill, ``Detecting interpersonal conflict in issues and code review: cross pollinating open- and closed-source approaches,'' in \emph{Proceedings of the 2022 {ACM}/{IEEE} 44th International Conference on Software Engineering: Software Engineering in Society}, ser. {ICSE}-{SEIS} '22.\hskip 1em plus 0.5em minus 0.4em\relax Association for Computing Machinery, 2023-05-08, pp. 41--55. [Online]. Available: \url{https://dl.acm.org/doi/10.1145/3510458.3513019}
\BIBentrySTDinterwordspacing
\bibitem{ortu2015bullies}
M.~Ortu, B.~Adams, G.~Destefanis, P.~Tourani, M.~Marchesi, and R.~Tonelli, ``Are bullies more productive? empirical study of affectiveness vs. issue fixing time,'' in \emph{Proceedings of the 12th Working Conference on Mining Software Repositories}, ser. MSR '15.\hskip 1em plus 0.5em minus 0.4em\relax IEEE Press, 2015, pp. 303--313.
\bibitem{Destefanis2016SoftwareDD}
G.~Destefanis, M.~Ortu, S.~Counsell, S.~Swift, M.~Marchesi, and R.~Tonelli, ``Software development: do good manners matter?'' \emph{PeerJ Comput. Sci.}, vol.~2, p. e73, 2016.
\bibitem{locking}
``{GiHub Conversations Locking},'' \url{https://github.blog/2014-06-09-locking-conversations/}, 2014, [Online; accessed 7-Jun-2023].
\bibitem{ferreiraIncivility2022}
\BIBentryALTinterwordspacing
I.~Ferreira, A.~Rafiq, and J.~Cheng, ``Incivility detection in open source code review and issue discussions,'' 2022-07-07. [Online]. Available: \url{https://papers.ssrn.com/abstract=4156317}
\BIBentrySTDinterwordspacing
\bibitem{sarkerAutomated2022}
\BIBentryALTinterwordspacing
J.~Sarker, A.~K. Turzo, M.~Dong, and A.~Bosu, ``Automated identification of toxic code reviews using toxicr,'' \emph{ACM Trans. Softw. Eng. Methodol.}, feb 2023, just Accepted. [Online]. Available: \url{https://doi.org/10.1145/3583562}
\BIBentrySTDinterwordspacing
\bibitem{sarkerBenchmark2020}
\BIBentryALTinterwordspacing
J.~Sarker, A.~K. Turzo, and A.~Bosu, ``A benchmark study of the contemporary toxicity detectors on software engineering interactions,'' no. {arXiv}:2009.09331, 2020-09-19. [Online]. Available: \url{http://arxiv.org/abs/2009.09331}
\BIBentrySTDinterwordspacing
\bibitem{imran2022data}
M.~M. Imran, Y.~Jain, P.~Chatterjee, and K.~Damevski, ``Data augmentation for improving emotion recognition in software engineering communication,'' in \emph{37th IEEE/ACM International Conference on Automated Software Engineering}, 2022.
\bibitem{batra2021bert}
H.~Batra, N.~S. Punn, S.~K. Sonbhadra, and S.~Agarwal, ``Bert-based sentiment analysis: A software engineering perspective,'' in \emph{International Conference on DEXA}, 2021.
\bibitem{eeshita-sentiment}
E.~Biswas, M.~E. Karabulut, L.~Pollock, and K.~Vijay-Shanker, ``Achieving reliable sentiment analysis in the software engineering domain using bert,'' in \emph{2020 IEEE ICSME}, 2020.
\bibitem{kamath2022emoroberta}
R.~Kamath, A.~Ghoshal, S.~Eswaran, and P.~B. Honnavalli, ``Emoroberta: An enhanced emotion detection model using roberta,'' in \emph{IEEE International Conference on Electronics, Computing and Communication Technologies}, 2022.
\bibitem{liu2019dens}
C.~Liu, M.~Osama, and A.~De~Andrade, ``Dens: A dataset for multi-class emotion analysis,'' in \emph{Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, 2019, pp. 6293--6298.
\bibitem{kabir2023answers}
S.~Kabir, D.~N. Udo-Imeh, B.~Kou, and T.~Zhang, ``Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions,'' 2023.
\bibitem{hou2023large}
X.~Hou, Y.~Zhao, Y.~Liu, Z.~Yang, K.~Wang, L.~Li, X.~Luo, D.~Lo, J.~Grundy, and H.~Wang, ``Large language models for software engineering: A systematic literature review,'' 2023.
\bibitem{zhang2023sentiment}
W.~Zhang, Y.~Deng, B.~Liu, S.~J. Pan, and L.~Bing, ``Sentiment analysis in the era of large language models: A reality check,'' 2023.
\bibitem{fan2023large}
A.~Fan, B.~Gokkaya, M.~Harman, M.~Lyubarskiy, S.~Sengupta, S.~Yoo, and J.~M. Zhang, ``Large language models for software engineering: Survey and open problems,'' 2023.
\bibitem{10132255}
S.~Jalil, S.~Rafi, T.~D. LaToza, K.~Moran, and W.~Lam, ``Chatgpt and software testing education: Promises \& perils,'' in \emph{2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)}, 2023, pp. 4130--4137.
\bibitem{Huang_2023}
\BIBentryALTinterwordspacing
F.~Huang, H.~Kwak, and J.~An, ``Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech,'' in \emph{Companion Proceedings of the ACM Web Conference 2023}, ser. WWW ’23.\hskip 1em plus 0.5em minus 0.4em\relax ACM, Apr. 2023. [Online]. Available: \url{http://dx.doi.org/10.1145/3543873.3587368}
\BIBentrySTDinterwordspacing
\bibitem{li2023hot}
L.~Li, L.~Fan, S.~Atreja, and L.~Hemphill, ``"hot" chatgpt: The promise of chatgpt in detecting and discriminating hateful, offensive, and toxic comments on social media,'' 2023.
\bibitem{he2023prompt}
X.~He, S.~Zannettou, Y.~Shen, and Y.~Zhang, ``You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content,'' 2023.
\bibitem{ziems2023large}
C.~Ziems, W.~Held, O.~Shaikh, J.~Chen, Z.~Zhang, and D.~Yang, ``Can large language models transform computational social science?'' 2023.
\bibitem{chatgpt}
OpenAI, ``Chatgpt,'' \url{https://openai.com/blog/chatgpt}, 2023.
\bibitem{stowe2022impli}
K.~Stowe, P.~Utama, and I.~Gurevych, ``Impli: Investigating nli models’ performance on figurative language,'' in \emph{Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics}, 2022.
\bibitem{zhou2021pie}
J.~Zhou, H.~Gong, and S.~Bhat, ``Pie: A parallel idiomatic expression corpus for idiomatic sentence generation and paraphrasing,'' in \emph{Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021)}, 2021.
\bibitem{haagsma2020magpie}
H.~Haagsma, J.~Bos, and M.~Nissim, ``Magpie: A large corpus of potentially idiomatic expressions,'' in \emph{Proceedings of The 12th Language Resources and Evaluation Conference}, 2020.
\bibitem{white2023prompt}
J.~White, Q.~Fu, S.~Hays, M.~Sandborn, C.~Olea, H.~Gilbert, A.~Elnashar, J.~Spencer-Smith, and D.~C. Schmidt, ``A prompt pattern catalog to enhance prompt engineering with chatgpt,'' 2023.
\bibitem{CAT}
\BIBentryALTinterwordspacing
Conversationai. (n.d.) GitHub. [Online]. Available: \url{https://github.com/conversationai/conversationai.github.io/blob/main/crowdsourcing_annotation_schemes/toxicity_with_subattributes.md}
\BIBentrySTDinterwordspacing
\bibitem{ouyang2023llm}
S.~Ouyang, J.~M. Zhang, M.~Harman, and M.~Wang, ``Llm is like a box of chocolates: the non-determinism of chatgpt in code generation,'' 2023.
\bibitem{openai_temp}
\BIBentryALTinterwordspacing
O.~Platform. (n.d.). [Online]. Available: \url{https://platform.openai.com/docs/guides/gpt/managing-tokens}
\BIBentrySTDinterwordspacing
\bibitem{openai_tokens}
\BIBentryALTinterwordspacing
OpenAI. (n.d.) Managing tokens. OpenAI Documentation. [Online]. Available: \url{https://platform.openai.com/docs/guides/gpt/managing-tokens}
\BIBentrySTDinterwordspacing
\bibitem{Chatterjee21}
P.~Chatterjee, K.~Damevski, and L.~Pollock, ``Automatic extraction of opinion-based {Q\&A} from online developer chats,'' in \emph{Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)}, 2021, pp. 1260--1272.
\bibitem{Kuutila2020ChatAI}
M.~Kuutila, M.~M{\"a}ntyl{\"a}, and M.~Claes, ``Chat activity is a better predictor than chat sentiment on software developers productivity,'' \emph{Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops}, 2020.
\bibitem{DARE_23}
\BIBentryALTinterwordspacing
B.~T.~R. Savarimuthu, Z.~Zareen, J.~Cheriyan, M.~Yasir, and M.~Galster, ``Barriers for social inclusion in online software engineering communities - a study of offensive language use in gitter projects,'' in \emph{Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering}, ser. EASE '23.\hskip 1em plus 0.5em minus 0.4em\relax New York, NY, USA: Association for Computing Machinery, 2023, p. 217–222. [Online]. Available: \url{https://doi.org/10.1145/3593434.3593463}
\BIBentrySTDinterwordspacing
\end{thebibliography}