ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
Extracted source text · arXiv 2411.19668
\documentclass{article}
\usepackage{amsmath}
\usepackage{arxiv}
% \usepackage{xeCJK} % 加载 xeCJK 包以支持中文 包
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{lipsum}
\usepackage{fancyhdr} % header
\usepackage{graphicx} % graphics
\graphicspath{{media/}} % organize your images and other figures under media/ folder
\usepackage{tabularx}
\usepackage{longtable}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{array}
\usepackage{xcolor}
% \usepackage{fontspec} % 使用 fontspec 来支持不同字体
% \usepackage{polyglossia} % 处理多语言
\usepackage{wrapfig}
\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{utfsym}
% \setdefaultlanguage{chinese} % 设置默认语言为中文
\usepackage{threeparttable}
\usepackage{marvosym}
\usepackage{tcolorbox}
\usepackage{mathptmx}
\definecolor{wkyellow}{RGB}{255,241,177}
\definecolor{lightblue}{HTML}{CCE5FF}
\definecolor{lightgray}{gray}{0.9}
\definecolor{goodblue}{HTML}{0071bc}
%Header
\pagestyle{fancy}
\thispagestyle{empty}
\rhead{ \textit{ }}
\pagestyle{fancy}
% \fancyhf{} % 清空所有的页眉和页脚
\fancyhead[C]{\textbf{ChineseWebText 2.0}: Large-Scale High-quality Chinese Web Text with Multi-dimensional and Fine-grained Information } % 居中页眉
\renewcommand{\headrulewidth}{0.4pt} % 设置页眉横线的宽度
\DeclareUnicodeCharacter{0002}{\relax}
%% Title
\title{\textbf{ChineseWebText 2.0}: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information}
\author{
Wanyue Zhang$^{1,2}$\thanks{Equal contribution, sorted by rolling dice.} \quad
Ziyong Li$^{1,2}$\footnotemark[1] \quad
Wen Yang$^{1,2}$\footnotemark[1] \quad
Chunlin Leng$^{1,2}$\footnotemark[1] \quad \\
\bf{Yinan Bai}$^{1,2}$\footnotemark[1] \quad
\bf{Qianlong Du}$^{1 *}$\footnotemark[2] \quad
\bf{Chengqing Zong}$^{1,2}$ \quad
\bf{Jiajun Zhang}$^{1,2,3}$\thanks{Corresponding author} \quad
\\
\small{$^1$ Institute of Automation, Chinese Academy of Sciences }\quad \\
\small{$^2$ School of Artificial Intelligence, University of Chinese Academy of Sciences }\\
\small{$^3$ Wuhan AI Research }\\
\texttt{\{zhangwanyue2023, liziyong2023, yangwen2023, lengchunlin2023, baiyinan2023\}@ia.ac.cn}\\
\texttt{\{qianlong.du,cqzong,jjzhang\}@nlpr.ia.ac.cn}\\
}
\begin{document}
\date{}
\maketitle
\begin{abstract}
%In the development of large language models (LLMs), the scale and quality of pre-training data play a critical role throughout the training process. Currently, publicly available large-scale Chinese datasets include WuDao, WanJuan, Yuan 1.0, SkyPile, BAAI-MTP, MAP-CC, and IndustryCorpus 1.0. However, for fine-grained datasets, most of the available data is still predominantly in English, with a significant lack of large-scale fine-grained Chinese datasets. Moreover, the fine-grained information in existing Chinese datasets is insufficiently comprehensive.
%To address this issue, we employ a novel end-to-end toolchain, \textit{EvalText}, to process the data through: 1) \textbf{coarse-grained filtering}, 2) \textbf{quality assessment}, 3) \textbf{domain classification}, and 4) \textbf{toxicity analysis}. First, we filter the collected data, removing low-quality or duplicate texts. Subsequently, we apply various tools and methods to evaluate the toxicity and quality of the text and perform categorization.
%This approach ensures the creation of a more refined and comprehensive dataset, better suited for the training of large-scale Chinese LLMs.
During the development of large language models (LLMs), pre-training data play a critical role in shaping LLMs' capabilities. In recent years several large-scale and high-quality pre-training datasets have been released to accelerate the research of LLMs, including ChineseWebText1.0~\cite{chinesewebtext}, C4~\cite{2020T5C4}, Pile~\cite{2020_pile}, WanJuan~\cite{wanjuan}, MAPCC~\cite{mapcc} and others. However, as LLMs continue to evolve, focus has increasingly shifted to domain-specific capabilities and
safety concerns, making those previous coarse-grained texts insufficient for meeting training requirements. Furthermore, fine-grained information, such as quality, domain and toxicity, is becoming increasingly important in building powerful and reliable LLMs for various scenarios. To address these challenges, in this paper we propose a new tool-chain called \textbf{MDFG-tool} for constructing large-scale and high-quality Chinese datasets with multi-dimensional and fine-grained information. First, we employ manually crafted rules to discard explicit noisy texts from raw contents. Second, the quality evaluation model, domain classifier, and toxicity evaluation model are well-designed to assess the remaining cleaned data respectively. Finally, we integrate these three types of fine-grained information for each text. With this approach, we release the largest, high-quality and fine-grained Chinese text \textbf{ChineseWebText2.0}, which consists of 3.8TB and each text is associated with a quality score, domain labels, a toxicity label and a toxicity score, facilitating the LLM researchers to select data based on various types of fine-grained information. The data, codes and the tool-chain are available on this website
\footnote{\url{https://github.com/CASIA-LM/ChineseWebText-2.0}}.
\end{abstract}
\section{Introduction}
In recent years, large language models(LLMs) have undergone rapid development. LLMs like BLOOM~\cite{scao2022bloom}, LLaMA~\cite{llama}, Falcon~\cite{2023refinedweb}, T5~\cite{t5}, PaLM~\cite{chowdhery2022palm}, Qwen~\cite{bai2023qwen}, GPT-4~\cite{GPT4} and O1~\cite{o1} have showcased powerful capabilities on diverse scenarios, such as question answering, mathematical problems solving, theorem proving, scientific exams, commonsense reasoning, translation, and more. During the development, a cornerstone of these LLMs' success lies in the utilization of a large-scale and high-quality dataset, which play a pivotal role in their extraordinary performance.
%Recent years have witnessed the rapid progress of large language models (LLMs). The models, such as GPT-3~\cite{brown2020language}, BLOOM~\cite{scao2022bloom}, LLaMA~\cite{llama}, Falcon~\cite{2023refinedweb}, PaLM~\cite{chowdhery2022palm} and GPT-4~\cite{GPT4}, become more and more powerful, even performing better than humans in some natural language understanding and generation tasks.
%During the development, it is evident that the scale and quality of pre-training data play a crucial role on LLM's capability. A large-scale and high-quality dataset is the foundation of LLMs and is the source of all the LLM's amazing capabilities.
%With the development of artificial intelligence, large language models (LLMs) garner significant attention and innovation in recent years. Models like GPT-3~\cite{brown2020language}, GPT-4~\cite{GPT4}, LLAMA~\cite{llama}, OPT~\cite{zhang2022opt} and some others~\cite{chowdhery2022palm,nakano2022webgpt, touvron2023llama}, have provided substantial improvements in various applications, such as text generation, machine translation, question answering and dialogue.
%The remarkable success of large language models can be primarily attributed to the extensive pool of pre-training data at their disposal. Pre-training data consists of vast text corpora compiled from diverse sources, and encompass billions of words and sentences across a wide range of domains and topics. Through training on these corpora, large language models develop a sophisticated understanding of language structures, grammatical rules, contextual relevance, and a substantial reservoir of knowledge. With pre-training data, LLM could learn a good capability of comprehending and generating natural language texts. As a result, the quality and quantity of pretraining data play a pivotal role in determining the knowledge and capability of large language models.
%The breakthrough development of artificial intelligence has benefited from the advancement of high-quality data. The latest progress of large language models relies on higher quality and richer training datasets. In recent years, models such as GPT-3~\cite{gpt3}, GPT-4~\cite{gpt4}, LLaMA~\cite{llama}, XLNet~\cite{xlnet}, and T5~\cite{t5} have become increasingly powerful, often surpassing human performance in generation tasks and natural language understanding. A refined, high-quality dataset is often the foundation for pre-training a large language model (LLM) and is crucial for an LLM to demonstrate its powerful capabilities.
%In order to accelerate the research of LLMs, several large-scale datasets have been released in recent years, including ChineseWebText1.0~\cite{chinesewebtext}, C4~\cite{2020T5C4}, Pile~\cite{2020_pile}, WuDao~\cite{wudao}, WanJuan~\cite{wanjuan}, Yuan 1.0~\cite{yuan1}, SkyPile~\cite{skypile}, MAPCC~\cite{mapcc} etc. These large-scale datasets provide
To facilitate research progress in LLMs, a variety of large-scale datasets have been publicly released in the last few years, such as ChineseWebText1.0~\cite{chinesewebtext},C4~\cite{2020T5C4}, Pile~\cite{2020_pile}, RefinedWeb~\cite{2023refinedweb}, WuDao~\cite{wudao}, WanJuan~\cite{wanjuan}, Yuan 1.0~\cite{yuan1}, SkyPile~\cite{skypile}, MAPCC~\cite{mapcc} and others. These datasets are typically constructed by collecting raw content from diverse sources, then applying manually designed rules or advanced classifiers to filter and refine the data, thereby producing large-scale and high-quality text corpora for LLM pre-training. However, most of these work only provide coarse-grained texts, with ChineseWebText1.0 being a notable exception by offering quality scores for each text. It should be noted that with the further development of LLMs, the focus has gradually shifted from general capabilities to domain-specific capabilities and securities. Consequently, these coarse-grained datasets are no longer sufficient to meet the growing demands of LLM training. And fine-grained annotations, such as quality, domain, toxicity and so on, are becoming increasingly crucial for boosting the domain-specific capabilities and safety of LLMs.
In order to address these issues, in this paper we propose a comprehensive tool-chain \textbf{MDFG-tool}, which can generate high-quality texts with multi-dimensional and fine-grained information from raw contents. The whole process of this approach can be divided into two stages. In the first stage, manually crafted rules, such as length filtering, sensitive word filtering, and deduplication, are applied to remove explicit noisy text from the raw data. This step produces a large-scale, high-quality dataset.
Unlike previous studies that typically end after this initial filtering, our approach incorporates a second stage comprising three advanced sub-modules: a quality evaluation model, a domain classifier, and a toxicity evaluation model. Despite the initial filtering in stage one, the diverse nature of web text often leaves a considerable amount of low-quality content. To address this, our second stage utilizes a BERT-based quality evaluation model to assign a quality score to each text, enabling researchers to curate high-quality subsets based on desired thresholds. The domain classifier, in turn, assigns domain-specific labels (e.g., technology, law, education) to each text, allowing for the efficient extraction of content tailored to specific fields, thereby supporting the development of domain-specific LLMs. Additionally, the toxicity evaluation model assigns both toxicity labels and scores, assisting researchers in enhancing the safety and ethical alignment of LLMs. Finally, these three types of fine-grained annotations are integrated for each text. Based on this complete tool-chain MDFG-tool, we release the largest Chinese dataset, ChineseWebText2.0, which features multi-dimensional and fine-grained annotations. This dataset comprises 3.8 TB of data, with each entry annotated with a quality score, domain labels, a toxicity label, and a toxicity score, empowering researchers to select data according to their specific requirements. Notably, this dataset includes a 3.16 GB subset of toxicity-labeled texts, currently the largest publicly available Chinese toxicity dataset, which can significantly enhance the toxicity evaluation capability and satefy of Chinese LLMs.
Our contributions can be summarized as follows:
(1) In this paper, we propose a new complete tool-chain MDFG-tool, which could generate high-quality Chinese pre-training data with multi-dimensional and fine-grained information.
(2) In this paper, we release the largest Chinese dataset consisting of 3.8 TB, and each text in this dataset is assigned with four types of fine-grained annotations, including quality score, domain label, toxicity label and toxicity score.
(3) In this paper, our released dataset includes a subset of 3.16 GB toxicity texts, which are currently the largest publicly available toxicity text dataset.
%The initial stage follows a similar approach to previous studies, primarily using manually designed rules to filter out explicit noisy data, thereby generating the initial set of clean Chinese text. This process involves two key modules: the preparation module and the processing module. The preparation module starts by using a language identification model to extract Chinese text, followed by a hash-based deduplication algorithm to eliminate duplicate entries. Subsequently, the processing module applies a set of well-designed rules, including length filtering, sensitive word filtering, and filtering based on the ratio of Chinese characters.
%To address this issue, this paper introduces a new tool called \textbf{EvalText}, which can collect data from different datasets and perform related processing tasks. We primarily completed four tasks. The first module is the coarse-grained filtering module, which filters out low-quality or duplicate texts. The second module is the quality assessment module. Due to the diversity of different datasets, a large amount of low-quality text data is included, making it impractical to manually remove them. Therefore, we designed an evaluation model to score the quality of the texts, filtering out data with higher scores. The third module is the topic classification module. We designed a topic classification module that can map text data from different datasets to predefined categories or topics for better understanding and retrieval of text data.The fourth module is the toxicity assessment module. Given the wide range of text data sources, there may be some texts containing offensive, discriminatory, hateful, or other inappropriate or harmful content. We designed a toxicity assessment module to score the toxicity of the text data, thus removing toxic data.
%In order to expedite the research on LLMs, several large-scale datasets have been made publicly available in recent years, such as C4~\cite{2020T5C4}, Pile~\cite{2020_pile}, RefinedWeb~\cite{2023refinedweb} and WanJuan~\cite{2023wanjuan}. Previous studies usually collect the raw texts at first from various sources, such as Wikipedia, GitHub, ArXiv, Stack Exchange, and CommonCrawl, in which CommonCrawl data often accounts for the vast majority. Then, handcrafted rules are designed to filter out the raw data in three steps: extracting the data in the language of interested, filtering out the noisy texts with language-specific rules and data deduplication. It should be noted that, most of the previous studies mainly focus on the collection of English-centered texts, and there is lack of a complete tool-chain for extracting clean data centered in other languages, e.g. Chinese. Furthermore, previous work usually directly release the final data, without giving the fine-grained information of the text, such as the quality of each text, limiting the potential that assists LLM researchers to re-filter the data according to their desired quality threshold.
%In recent years, several large-scale datasets have been made publicly available, such as WuDao~\cite{wudao}, WanJuan~\cite{wanjuan}, Yuan 1.0~\cite{yuan1}, SkyPile~\cite{skypile}, BAAI-MTP~\cite{baaimtp}, MAP-CC~\cite{mapcc}, and ChineseWebText1.0~\cite{chinesewebtext}. It is important to note that previous datasets have only undergone general classification and filtering within Chinese datasets. For example, the WanJuan dataset is composed of cleaned pre-training corpora from various sources such as web pages, encyclopedias, books, patents, textbooks, and exam questions. However, these datasets lack fine-grained classification of texts, such as topic classification and toxicity assessment. Additionally, except for ChineseWebText1.0, which provides quality assessments of Chinese texts, the other datasets generally do not include further evaluations of Chinese texts. This limitation restricts LLM researchers from achieving high-quality pre-trained models.
%To address this issue, this paper introduces a new tool called \textbf{EvalText}, which can collect data from different datasets and perform related processing tasks. We primarily completed four tasks. The first module is the coarse-grained filtering module, which filters out low-quality or duplicate texts. The second module is the quality assessment module. Due to the diversity of different datasets, a large amount of low-quality text data is included, making it impractical to manually remove them. Therefore, we designed an evaluation model to score the quality of the texts, filtering out data with higher scores. The third module is the topic classification module. We designed a topic classification module that can map text data from different datasets to predefined categories or topics for better understanding and retrieval of text data.The fourth module is the toxicity assessment module. Given the wide range of text data sources, there may be some texts containing offensive, discriminatory, hateful, or other inappropriate or harmful content. We designed a toxicity assessment module to score the toxicity of the text data, thus removing toxic data.
\section{Related Work}
\label{gen_inst}
\paragraph{Text Filtering.} During the construction of pre-training datasets, most raw data from the web contains various forms of noise, such as violence, pornography, advertisements, corrupted characters, and poorly formatted text. To extract high-quality data from these raw contents, numerous text filtering methods have been proposed to automatically remove undesirable elements. Among these methods, handcrafted rules~\cite{t5,luccioni2021whatsboxpreliminaryanalysis} are commonly used to filter out explicit noise, followed by deduplication techniques~\cite{lee2022deduplicatingtrainingdatamakes} to eliminate duplicate text from different sources. Building on this prior work, this paper introduces a rule-based preprocessing module designed to extract high-quality text from raw web content.
%Rule-based text filtering methods utilize manually crafted rules to identify content-rich and semantically coherent paradigms from the collected datasets. In the large volume of text data we have gathered, there is significant noise, including instances of violence, pornography, advertisements, corrupted characters, and poorly formatted text. To extract high-quality data, numerous tools and methods have been proposed for cleaning and automatically filtering out undesirable content. These approaches include deduplication techniques~\cite{lee2022deduplicatingtrainingdatamakes}, which are used to remove a large amount of redundant text, as well as handcrafted rules~\cite{t5,luccioni2021whatsboxpreliminaryanalysis} designed to filter out violence, pornography, advertisements, unnecessary keywords, and other disruptive content. By combining multiple approaches, the quality of the data is substantially improved.This rule-based approach plays a crucial role in ensuring that the resulting dataset is both semantically rich and free from distracting or harmful content, thus making it more suitable for training large language models.
\paragraph{Quality Evaluation.} In addition to rule-based methods, quality evaluation approaches are also employed to identify high-quality texts using well-designed classifiers. Unlike rule-based methods, which primarily filter out explicit noise from raw content, quality evaluation approaches offer greater robustness and flexibility. These approaches leverage models such as logistic regression~\cite{brown2020language}, BERT~\cite{bert}, FastText~\cite{fasttext}, and others to calculate probability scores for each text. Based on these scores, texts are classified as positive or negative according to a predefined threshold. Among these models, BERT has emerged as one of the most widely used architectures for quality evaluation due to its exceptional performance in text classification and understanding tasks. BERT's effectiveness stems from its pre-training objectives, including masked language modeling and next-sentence prediction, which enable it to learn powerful text representation and comprehension capabilities. In this paper, we adopt the BERT architecture to build a robust quality evaluation model for text.
%The most basic text classification model is logistic regression~\cite{logistic_regression}, which uses the logistic function to compute the probability value for each text and classifies them as positive or negative based on a predetermined threshold. Currently, BERT~\cite{bert} and FastText~\cite{fasttext} are commonly used text classification models. BERT is a pre-trained language model based on the Transformer architecture~\cite{vaswani2017attention}, which has shown remarkable performance in tasks involving text classification and understanding. Through pre-training on large datasets by masking tokens and predicting the next sentence, BERT can comprehensively learn and understand the content of the text, providing significant advantages in text classification. FastText, on the other hand, is a neural network-based method similar to CBOW~\cite{cbow}. The key difference is that FastText predicts labels, while CBOW predicts intermediate words. Additionally, compared to basic neural networks, FastText accelerates both training and testing speeds while maintaining accuracy, and it does not require pre-trained word embeddings. In this work, FastText is employed to evaluate the quality of Chinese text.
\paragraph{Domain Classification.} The Pre-training of LLMs on vast amounts of diverse text data often results in a generalized understanding of language. However, to enhance their effectiveness for specific applications, it is advantageous to classify and curate data from various domains, such as medicine, law, and finance. This process, known as domain classification, helps refine the model to better capture domain-specific language patterns, terminology, and contextual nuances. Similar to quality evaluation methods, domain classification approaches commonly utilize models such as logistic regression, BERT, and FastText to assign domain labels to each text. Among these, FastText is a neural network-based approach similar to CBOW~\cite{cbow}. Compared to other methods, FastText offers efficient and rapid training on large-scale datasets while maintaining comparable classification performance. As a result, this paper employs the FastText model to classify domain labels for each text.
\paragraph{Toxicity Evaluation.}
Toxic texts are generally defined as those containing offensive or adult content~\cite{pradhan2020review, xiao2024chinese}. Offensive content can be classified into three levels of severity: offensive text, abusive text, and hate text~\cite{hatebert}. Adult content typically includes themes related to pornography or violence. Due to differences in cultural values and political perspectives, the analysis of toxic texts varies across countries. During the development of LLMs, the presence of toxic content in pre-training datasets would compromise the safety of LLMs. Consequently, toxicity evaluation has become increasingly important for the construction of pre-training datasets. In recent years, Caselli et al.~\cite{hatebert} construct a toxicity dataset, RAL-E, using data from banned Reddit communities. They then train a BERT-based toxicity evaluation model with this dataset. Hartvigsen et al.~\cite{hartvigsen2022toxigen} leverage GPT-3 to generate a dataset containing both subtle toxic texts and benign examples through adversarial techniques. In contrast, Deng et al.~\cite{deng2022cold} construct a Chinese toxicity dataset with data from Zhihu and Weibo, and annotate each text with the help of human experts and LLMs. However, previous work often suffers from small data scales and insufficient coverage, limiting its ability to effectively evaluate the toxicity of LLMs. To address the above issues, this paper aims to construct a more robust toxicity evaluation model and release a larger-scale, higher-coverage toxicity dataset, thereby enhancing toxicity evaluation capabilities and improving the safety of LLMs.
%Due to differences in cultural values and political perspectives, the analysis of toxic language varies across countries. The presence of toxic content in pretraining datasets can compromise model safety, making toxicity detection a critical component of model training. Recent studies have addressed this challenge in various ways. For example, Caselli et al.~\cite{hatebert} constructed the RAL-E dataset using data from banned Reddit communities and retrained the BERT model on this dataset. Hartvigsen et al.~\cite{hartvigsen2022toxigen} employed the GPT-3 model to generate a dataset comprising both subtle toxic language and benign text through adversarial techniques. In the Chinese context, Deng et al.~\cite{deng2022cold} collated text from Zhihu and Weibo and subsequently annotated the data with the assistance of human experts and models. Other researchers have also collected data from Zhihu, Weibo, and Tieba and constructed toxicity datasets targeting specific groups~\cite{toxicn}~\cite{jiang2022swsr}~\cite{zhou2022towards}. However, these Chinese datasets often suffer from small scale and limited diversity, and they are unable to adapt to the long-term evolution of toxic language on the internet. In order to address these shortcomings, this study filters toxic data from a large-scale, long-duration dataset, thereby advancing research in Chinese toxic language detection and improving the safety of language models.
%Toxicity is defined as the tendency of language models to generate toxic, rude, hostile, or subtly hateful content. Furthermore, toxicity analysis varies across nations due to differences in values, political reasons, and other factors. Thus, toxicity detection plays a critical role in training models. In recent work, significant contributions have been made in toxicity detection on various datasets by Thomas et al. (2022)~\cite{hartvigsen2022toxigen}, Deng, Jiawen et al. (2022)~\cite{deng2022cold}, and Lu, Junyu et al. (2023)~\cite{toxicn}. Additionally, Caselli, Tommaso et al.~\cite{hatebert} designed a toxicity detection model based on BERT and introduced HateBERT, a pre-trained BERT specifically aimed at detecting abusive language. They also released a large-scale English dataset of social media posts, sourced from communities banned for offensive, abusive, or hateful content. However, compared to English, Chinese toxicity datasets are still relatively scarce.
\paragraph{Datasets for Pre-training.} As the foundation of LLMs, large-scale pre-training datasets play a crucial role in enhancing the capabilities of these models. These datasets are often constructed from diverse and extensive sources, such as web crawls, books, scientific papers, and repositories like arXiv, to ensure broad coverage of knowledge. To facilitate LLM research, several large-scale datasets have been publicly released. For example, He et al.~\cite{wanjuan} introduce a 1.0 TB comprehensive dataset that includes both Chinese and English texts gathered from a wide array of online resources. Similarly, Sha Yuan et al.~\cite{wudao} develop WuDao, an ultra-large-scale Chinese corpus containing approximately 3 TB of cleaned training data extracted from 100 TB of raw web pages. This dataset spans over 50 categories, including education and technology, and utilizes over 20 cleaning rules. Chen et al.~\cite{chinesewebtext} release ChineseWebText1.0, a 1.4 TB Chinese dataset, along with a comprehensive toolchain for extracting clean web data. Notably, they introduce a quality scoring system for texts, allowing researchers to re-filter the data based on desired quality thresholds. Furthermore, IndustryCorpus~\cite{industrycorpus} presents a bilingual dataset labeled across multiple industries, combining over 100 TB of multilingual open-source datasets, such as WuDaoCorpora, RedPajama~\cite{redpajama}, and SkyPile-150B. After applying 22 domain-specific processing techniques, the resulting dataset comprises 3.4 TB of high-quality data, with 1 TB in Chinese and 2.4 TB in English. These datasets primarily focus on providing cleaned text or fine-grained information along a single dimension (e.g., quality or domain of each text). However, as LLMs continue to evolve, the focus has shifted from general capabilities to more specialized requirements, including domain-specific expertise and enhanced security considerations. This shift highlights the limitations of existing datasets, which are increasingly insufficient to meet the growing demands of training LLMs tailored for specific applications and higher ethical and security standards.
%As the foundation of LLMs, large-scale pre-training datasets play a important role on LLM's capabilities. These datasets are often constructed from diverse and extensive sources, such as web crawls, books, scientific papers, arxiv, etc., to ensure broad coverage of knowledge. In order to facilitate LLM research, some large-scale datasets have been publicly released. For example, He et al.~\cite{wanjuan} introduce a 1.0 TB comprehensive dataset that includes both Chinese and English texts gathered from a wide array of online resources. Similarly, Sha Yuan et al.~\cite{wudao} develop WuDao, an ultra-large-scale Chinese corpus with approximately 3 TB of cleaned training data extracted from 100 TB of raw web pages. This dataset spans over 50 categories, including education and technology, utilizing over 20 cleaning rules. Chen et al.~\cite{chinesewebtext} release ChineseWebText1.0, a 1.4 TB Chinese dataset, alongside a comprehensive toolchain for extracting clean web data. Notably, they introduce a quality scoring system for texts, enabling researchers to re-filter the data according to desired quality thresholds. Furthermore, IndustryCorpus~\cite{industrycorpus} presents a bilingual dataset labeled across multiple industries, combining over 100 TB of multilingual open-source datasets, such as WuDaoCorpora, RedPajama~\cite{redpajama}, and SkyPile-150B. After applying 22 domain-specific processing techniques, the resulting dataset comprises 3.4 TB of high-quality data, with 1 TB in Chinese and 2.4 TB in English. These datasets primarily focus on providing cleaned text or fine-grained information along a single dimension (e.g. quality or domain of each text). However, as LLMs continue to evolve, the focus has shifted from general capabilities to more specialized requirements, including domain-specific expertise and enhanced security considerations. This shift highlights the limitations of existing datasets, which are increasingly insufficient to meet the growing demands of training LLMs tailored for specific applications and higher ethical and security standards.
%These datasets mainly focus on collecting cleaned texts, or only include fine-grained information from a single dimension (e.g. quality or domain of each text). However, with the further development of LLMs, the focus has shifted from general capabilities to specific domain abilities and securities, making these datasets inadequate for meeting the training demand.
%and 1.08 trillion Chinese characters. This dataset was created by cleaning 100 TB of raw web pages using over 20 rules, covering more than 50 categories including education and technology. In the last year, Chen et al.~\cite{chinesewebtext} release a 1.4 TB Chinese dataset ChineseWebText1.0. In this work, they propose a complete tool-chain for extracting Chinese clean data from web sources. Notable, they first propose to generate quality score for each text which could help LLM researchers to re-filter texts with new desired quality thresholds. Furthermore, IndustryCorpus~\cite{industrycorpus} is a bilingual dataset labeled across multiple industries. It employs 22 industry-specific processing operators to filter high-quality, multilingual pre-training data from over 100 TB of open-source datasets, including WuDaoCorpora, BAAI-CCI, RedPajama~\cite{redpajama}, and SkyPile-150B, resulting in a dataset of 3.4 TB, with 1 TB in Chinese and 2.4 TB in English. These datasets mainly focus on collecting cleaned texts, or only include fine-grained information from a single dimension (e.g. quality or domain of each text). However, with the further development of LLMs, the focus has shifted from general capabilities to specific domain abilities and securities, making these datasets inadequate for meeting the training demand.
%In recent years, the expansion of multimodal and language models has made large-scale web scraping of training datasets a common practice. Due to the low cost and rapid turnaround of web crawling, it has become a crucial source for dataset acquisition. OpenAI trained the GPT-3 model with 175 billion parameters on a 570 GB English training dataset. Gunasekar et al. (2023)~\cite{textbooksneed} constructed a code dataset with 7 billion tokens. Additionally, He et al. (2023)~\cite{wanjuan} released a comprehensive multimodal dataset that includes both Chinese and English texts, collected from a wide array of online resources. However, compared to English datasets, Chinese datasets are still lacking.
%Sha Yuan et al.(2021)~\cite{wudao}established the ultra-large-scale Chinese corpus WuDao, which contains approximately 3 TB of training data and 1.08 trillion Chinese characters. This dataset was created by cleaning 100 TB of raw web pages using over 20 rules, covering more than 50 categories including education and technology. WuDao is currently the largest pure text dataset, the largest multimodal image-text dataset, and the largest Chinese dialogue dataset globally. The IndustryCorpus~\cite{industrycorpus}, developed and open-sourced by the Zhiyuan Institute, is a bilingual dataset labeled across multiple industries. It employs 22 industry-specific processing operators to filter high-quality, multilingual pre-training data from over 100 TB of open-source datasets, including WuDaoCorpora, BAAI-CCI, RedPajama~\cite{redpajama}, and SkyPile-150B, resulting in a dataset of 3.4 TB, with 1 TB in Chinese and 2.4 TB in English. However, Chinese datasets still lack many fine-grained details, such as toxicity assessments and quality evaluations, which could assist LLM researchers in selecting their desired data.
\section{Data Construction}
The diversity and complexity of textual data present significant challenges in constructing high-quality pre-training datasets and annotating each text with multi-dimensional, fine-grained information. To address these challenges, this paper proposes a pipeline system, MDFG-tool, which integrates a text filtering module with multi-dimensional, fine-grained annotation modules. This approach allows us to extract high-quality data from raw content and provide quality scores, domain labels, toxicity scores, and toxicity labels for each text. As shown in Figure~\ref{fig:structure}, the overview of our proposed approach is illustrated. Starting with raw content collection, we employ a preparation module to evaluate and select data from suitable sources. Next, a preprocessing module applies handcrafted rules—such as text length, character proportions, and sensitive words—to filter out noisy or irrelevant texts, yielding a high-quality dataset. Following this, three fine-grained annotation modules are applied to the refined texts: a quality evaluator assigns a quality score, a domain classifier determines the domain labels, and a toxicity evaluator calculates a toxicity score and assigns a toxicity label. Finally, these three types of annotations are integrated for each text, resulting in a high-quality dataset enriched with multi-dimensional, fine-grained metadata.
%The diversity and complexity of textual data present significant challenges in constructing high-quality pre-training datasets and annotating each text with multi-dimensional fine-grained information. To tackle these challenges, in this paper we propose a pipeline system \textbf{MDFG-tool}, which integrates a text filtering module and multi-dimensional fine-grained information annotation modules. With this approach, we can extract high-quality from raw contents, and then provide quality score, domain label, toxicity score and toxicity label for each text. As in Figure~\ref{fig:structure}, it illustrates the overview of our proposed approach.Starting with raw content collection, we employ a preparation module to evaluate and select data from suitable source. Next, a preprocessing module applies handcrafted rules—such as text length, character proportions, and sensitive words—to filter out noisy or irrelevant texts. With the preprocessing module, we can get a high-quality dataset. Following this, three fine-grained information annotation modules are applied to the refined texts: a quality evaluator assigns a quality score, a domain classifier determines the domain label, and a toxicity evaluator calculates a toxicity score and assigns a toxicity label. Finally, these three types of annotation will be integrated for each text, producing a high-quality dataset with multi-dimensional, fine-grained metadata.
%As in Figure~\ref{fig:structure}, it illustrates the overview of our proposed approach. For the collected raw contents, we first use a preparation module to process them, deciding whether to use the data from that source. After that, a preprocessing module will be used to filter out noise texts with handcrafted rules, including text length, character proportion, sensitive words etc. With this preprocessing module, we can get a high-quality dataset from the collected texts. Then, three fine-grained information annotation modules will be used to process the high-quality texts respectively, including quality evaluator, domain classifier and toxicity evaluator. By this way, we can generate a quality score, domain label, toxicity score and toxicity label for each text. Finally, these three types of annotation will be integrated for each text, building a high-quality dataset with multi-dimensional and fine-grained information.
%Due to the diversity and complexity of textual data, there are serious challenges in constructing high-quality pre-training datasets and annotating each text with multi-dimensional fine-grained information. To address these challenges, in this paper we propose a pipeline system \textbf{MDFG-tool}, which integrates text filtering module and multi-dimensional fine-grained information annotation modules. With this approach, we can extract high-quality from raw contents, and then generate quality score, domain label, toxicity score and toxicity label for each text.
%As illustrated in Figure~\ref{fig:structure}, our workflow outlines the proposed approach. We begin with a coarse-grained filtering module that applies rule-based methods to clean the data, focusing on criteria such as text length and sensitive words to ensure data quality. Once cleaned, the data is evaluated using a BERT-based model that assesses text quality. This process generates a quality score, and by selecting an appropriate threshold, we can extract the high-quality text data that meets our needs. FastText is then employed to classify the labels for the cleaned data.
%Currently, a vast amount of textual data is sourced from the internet. However, the prevalence of noise and irrelevant information online makes it a complex and challenging task to clean and extract valuable content. To address this, we propose a pipeline system called \textbf{EvalText} for extracting high-quality Chinese text from web-crawled data. This system integrates manually designed rules, evaluation models, and classification models. Additionally, we perform toxicity assessments on the collected web text.
%We also collected a large amount of text from the internet and conducted toxicity evaluation. The FastText model was used to filter out toxic content and assign toxicity scores. This scoring system allows researchers to set thresholds for identifying and selecting harmful text for further training.
\begin{figure}[htp]
\centering
\includegraphics[width=10cm]{picture/structure.pdf}
\caption{The pipeline of MDFG-tool.}
\label{fig:structure}
\end{figure}
\label{headings}
\subsection{Data Collection and Preparation}
In recent years, many pre-training datasets have been publicly released, containing a variety of texts from different sources. As a result, we consider these datasets as the sources of our raw content. In this paper, we collect some publicly available datasets from the internet, including MAP-CC~\cite{mapcc}, SkyPile~\cite{skypile}, WanJuan~\cite{wanjuan}, WuDao~\cite{wudao}, and ChineseWebText1.0~\cite{chinesewebtext}, among others. All the datasets we obtained are composed of Chinese texts.
%In recent years, a lot of pre-training datasets have been publicly released, which include a variety of texts from different sources. Consequently, we regard these data as the sources of our raw contents. In this paper, we collect 12 published datasets from the internet, including "MAP-CC", "SkyPile", "WanJuan"，ChineseWebText1.0 and so on. All of the obtained datasets are composed of Chinese texts.
Due to the presence of noisy data and irrelevant characters in some publicly available datasets, the acquisition of high-quality data is significantly affected. To address this issue, we implement a preparation module to process data from various sources. In this module, we first sample hundreds of texts from each dataset for manual analysis. If a dataset is found to contain excessive noise or irrelevant characters, it is entirely excluded. The empirical threshold for exclusion is when the proportion of irrelevant text exceeds 30\%. Through this process, we are able to obtain the Chinese text data required for our study.
%Due to the fact that some publicly available datasets still contain a large amount of noisy data and irrelevant characters, it seriously affects the subsequent acquisition of high-quality data. To address this issue, we implement a preparation module to process data from various sources. In this module, we first sample hundreds of texts from each dataset for manual analysis. If a dataset is found to contain excessive noise or irrelevant characters, it is excluded entirely. The empirical threshold is that the proportion of irrelevant text exceeds 30\%. Through this process, we can obtain the Chinese text data we required.
%, and then manually analyze these collected data. If there is a large amount of noise information or irrelevant characters, we will directly remove the data from that source. By this way, we can obtain the Chinese text data we required.
%Since the original CommonCrawl file is typically quite diverse and contains texts in many languages, to efficiently extract Chinese text data from it, we first employ a deduplication and language identification (LID) module to perform preliminary cleaning on the collected datasets. Following the work of CCNet~\cite{wenzek_ccnet_2020}, in this module a Hash-based inter-string deduplication method is employed to remove duplicate text from different snapshots. Additionally, a well-trained language identification model~\cite{grave2018learning}, which could support 157 languages, is applied to select Chinese data. By this way, we can obtain all the monolingual Chinese text data we required.
\subsection{Preprocessing}
After collecting Chinese texts from various sources, we introduce a preprocessing module designed to process all the collected texts and ensure the extraction of high-quality Chinese text data. Building upon the work of ChineseWebText1.0, this paper employs four types of handcrafted rules in the preprocessing module. The specifics of these rules are detailed below.
\textbf{Data Length.} Since most of the collected texts are from web content, a significant portion consists of short text lines that lack relevance to one another. These texts make the data unsuitable for training language models. To address this issue, following the approach of ChineseWebText1.0, we calculate the average line length for each document and discard those with an average line length of fewer than 10 characters. Additionally, short texts often provide limited information and fail to convey sufficient knowledge. Therefore, we also remove texts with fewer than 200 characters to ensure the dataset is more informative and effective for training.
%As most of the collected texts are from web contents, there is a significant portion of data including short text lines, which lack relevance to one another. These characters makes such data unsuitable for training language models. To address this, following the work of ChineseWebText1.0 we also calculate the average text line length for each document and discard those with an average line length of fewer than 10 characters. Additionally, short texts typically provide limited information and fail to offer sufficient knowledge. Therefore, we also remove texts with fewer than 200 characters to ensure the dataset is more informative and effective for training.
%Our collected datasets is primarily sourced from web texts, which are often characterized by short sentence structures and weak semantic connections between sentences, making this data unsuitable as pre-training corpus for language models. To reduce the presence of documents with excessive short texts, we calculate the average line length for each document and remove those with an average line length of fewer than 10 characters. Additionally, short text data contains limited information, which is inadequate for language model training; thus, we will remove texts shorter than 200 characters.
\textbf{Proportion of Characters.} During the human analysis of texts in the preparation stage, we found that some Chinese datasets contain characters from other languages, non-essential characters, special symbols, and so on. These irrelevant characters are not useful for training large language models. As a result, we exclude texts that contain fewer than 30\% Chinese characters.
%During the human analysis of texts in preparation stage, we find that some Chinese datasets include some other language characters, non-essential characters, special symbols and so on. These irrelevant characters are useless for the training of large language models. As a result, we will exclude texts that include fewer than 30\% Chinese characters.
%Web texts often contain a large amount of symbols and numbers, with little to no semantic relevance between them, which does not contribute to language model training. Additionally, although we have collected Chinese datasets, some English and even lesser-known language words may still be present. To address this, we use regular expressions to identify non-Chinese characters in each line of data and remove lines where the proportion of non-Chinese characters exceeds 0.3.
\textbf{Sensitive Words.} Texts from various sources often contain significant amounts of harmful content, such as references to drugs, violence, racial discrimination, and more. Such data can cause models to generate toxic content, leading to serious negative impacts on national security and social ethics. To mitigate these risks, we process all texts using a sensitive word list. Specifically, if the proportion of sensitive characters in any line of text exceeds 50\%, we classify the text as toxic and remove it from the dataset.
%Texts from various sources often contain significant amounts of harmful content, such as references to drugs, violence, racial discrimination, and more. Such data can cause the model to generate toxic content, leading to serious negative impacts on national security and social ethics. To mitigate these risks, we process all texts using a sensitive word list. In this approach, if the proportion of sensitive characters in each line of a text exceeds 50\%, we classify the text as toxic and remove it from the dataset.
%The texts from various sources often contains a significant a amount of harmful content, such as drugs, violence, racial discrimination and so on. Such data would cause the model to generate large amounts of toxic content, leading to serious negative impacts on national security and social ethics. To mitigate these risks, we use a sensitive word list to process all texts. In this approach, if the proportion of sensitive characters in each line of a text exceeds 50\%, we classify the data as toxic and remove it from our dataset.
%Web texts contain a substantial amount of negative content, including themes of violence, hate, crime, discrimination, pornography, and gambling. Such toxic data can lead large models to generate harmful information, posing potential risks to society. To address this, we compiled a list of sensitive words related to these negative topics. Using this list, we filter the data by counting sensitive words in each line: if a line contains more than 0.3 sensitive words on average, it is classified as toxic and subsequently removed.
\textbf{Duplication.} During the data collection process, some datasets may source data from the same websites, resulting in duplicate texts across different sources. These duplicates can significantly impact the efficiency and performance of model training. To address this issue, we adopt a deduplication method to process the collected texts. Similar to ChineseWebText1.0, our approach employs a 13-gram granularity for deduplication. If the proportion of repeated characters exceeds 50\%, the duplicate text is removed from the dataset.
%During the data collection process, some datasets may use data from the same websites, resulting in duplicate texts across different sources. These duplicates can significantly impact the efficiency and performance of model training. To address this issue, we adopt a deduplication method to process the collected texts. Similar to ChineseWebText1.0, this approach also employs a granularity of 13-gram to conduct deduplication. And if the proportion of repeated characters exceeds 50\%, the target text will be removed from our collected dataset.
%Use n-gram search to detect internally duplicated fields within each data entry. The internal duplication rate is defined as the ratio of the length of the duplicated fields to the total length of the data. If the internal duplication rate of a data entry exceeds a certain threshold (set to 0.2), the entry will be removed.
%With these four rule-based processing methods above, we can obtain a high-quality Chinese dataset. Most other methods would stop data processing at this point. However, unlike these approaches, this paper introduces three evaluation models that will further process the acquired high-quality data to extract multi-dimensional and fine-grained information, namely the quality evaluation model, domain classification model, and toxicity evaluation model.
With the four rule-based processing methods described above, we can obtain a high-quality Chinese dataset. Most other methods would stop data processing at this stage. However, unlike these approaches, this paper introduces three evaluation models that further process the acquired high-quality data to extract multi-dimensional, fine-grained information: the quality evaluation model, domain classification model, and toxicity evaluation model.
\subsection{Quality Evaluation}
In the preprocessing phase, we applied a set of handcrafted rules to filter out explicit noisy text from our dataset. However, a significant portion of low-quality data remained, which could not be effectively eliminated by manual rules alone. To perform a more fine-grained quality assessment on the remaining data, we propose a fully automated quality evaluation method. Building on the approach used in ChineseWebText1.0~\cite{chinesewebtext}, we develop a BERT-based classification model to assign a quality score to each text. With this quality score, LLM researchers can select the desired subsets based on a specified threshold. The details of the classification model are presented below.
\begin{figure}[!h]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\textwidth]{picture/data_stat_in_quanlity_evaluation.pdf}
\caption{The Composition of BERTEval Training Data.}
\label{fig:data_stat}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=0.6\textwidth]{picture/bert.pdf}
\caption{The Structure of BERTEval Model.}
\label{fig:bert_structure}
\end{subfigure}
\caption{The Overview of BERTEval Training Data Composition and Model Architecture.}
\label{fig:combined_figure_1}
\vspace{-4mm}
\end{figure}
\paragraph{Training Data}
%To enhance the diversity of the training data, we incorporate various text types in the positive training samples, including Wikipedia entries, e-books, poetry, news articles, and question-answer data. This strategy enhances the model's ability to evaluate quality across a wide range of data types. Due to the relatively high noise levels in Common Crawl data, we directly sample from Common Crawl to construct negative samples. Figure~\ref{fig:data_stat} presents a detailed breakdown of the composition and quantity of the training data, showing that the positive and negative samples are nearly in a 1:1 ratio.
To enhance the diversity of the training data, we incorporate a variety of text types into the positive training samples, including Wikipedia entries, e-books, poetry, news articles, and question-answer pairs. This strategy improves the model's ability to evaluate quality across a broad range of data types. Given the relatively high noise levels in Common Crawl data, we directly sample from Common Crawl to construct the negative samples. Figure~\ref{fig:data_stat} provides a detailed breakdown of the composition and quantity of the training data, demonstrating that the positive and negative samples are nearly in a 1:1 ratio.
\paragraph{Model Architecture}
Following the work of ChineseWebText1.0, we also employ the efficient AES model Tran-BERT-MS-ML-R~\cite{wang2022use} to evaluate text quality. As shown in Figure~\ref{fig:bert_structure}, the model is based on the BERT~\cite{devlin2018bert} architecture. To reduce computational complexity, we focus solely on the text-level representation, utilizing the $[CLS]$ embedding to extract relevant information and structural features from a comprehensive perspective of the text. Simultaneously, token-level representations are derived from the sequence output of BERT, with $x$ representing the text input. After applying max pooling, the token-level representation is concatenated with the text-level representation and passed through a dense layer with Sigmoid activation, producing a text quality score $f(x|W)$ within the range (0, 1).
% \begin{wrapfigure}{r}{0.35\textwidth}
% \centering
% \includegraphics[width=0.23\textwidth]{picture/bert.pdf}
% \caption{The structure of BERT model.}
% \label{fig:reward_acc_with_iteration}
% \vspace{-4mm}
% \end{wrapfigure}
\paragraph{Training Stage}
The BERT model in the ChineseWebText1.0 achieves excellent classification performance, so we utilize it as our base model. To further enhance the model's learning capability and robustness, we extend beyond the standard MSE loss~\cite{mesgar2018neural}. Specifically, we incorporate two additional loss functions: Margin Ranking (MR) loss~\cite{liu2021temp} and Cosine Similarity (CS) loss~\cite{wang2022use}. The training process consists of a pre-training phase and a self-training phase. During the pre-training phase, we train the model using positive and negative samples in a 1:1 ratio. This phase enables BERTEval to develop an initial ability to distinguish text quality.
In the Common Crawl corpus $D_n$, a significant portion of the texts meets quality standards, which may introduce inaccurate supervision and hinder BERTEval's ability to make precise quality estimations. To address this, we apply a self-training approach~\cite{scudder1965probability}. Let $S_n$ be a randomly sampled subset of $D_n$. In each iteration, BERTEval’s parameters $W^t$ from the previous step are used to generate pseudo-labels for $S_n$. BERTEval is then retrained on the sampled data in $S_n$ using the pseudo-labels to update its parameters to $W^{t + 1}$~\cite{mukherjee2020uncertainty}. Notably, as the positive samples are drawn from a large, highly reliable curated corpus that does not require pseudo-labeling, we sample only instances with pseudo-label $y_n$.
\paragraph{Evaluation}
% \begin{wraptable}{r}{0.5\linewidth} % "r" for right alignment, and 0.5\linewidth for half-page width
% \centering
% \vspace{-4mm}
% \renewcommand\arraystretch{1.0}
% % \setlength{\tabcolsep}{1mm}
% \caption{The Classification Results of Different Evaluation Models.}
% \label{bert_result}
% \begin{tabular}{lccc}
% \toprule
% Model & Precision (\%) & TP+FP & TP \\
% \midrule
% Regression & 50.20 & 247 & 124 \\
% Perplexity & 62.60 & 254& 159 \\
% BERTEval & \textbf{76.36} & 110& 84 \\
% \bottomrule
% \end{tabular}
% \vspace{-4mm}
% \end{wraptable}
\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-2mm}
\centering
\includegraphics[width=0.4\textwidth]{picture/human_evaluation_in_quality.pdf}
\vspace{-4mm}
\caption{The Quality Evaluation.}
\label{fig:human_evaluation_in_quality}
\vspace{-6mm}
\end{wrapfigure}
To evaluate the performance of BertEval in quality assessment, we conduct a human evaluation based on BertEval scores. First, we sample a validation set consisting of 50 samples from each BertEval score interval. Since the scores in the interval \([0, 0.1)\) are zero, we exclude this range from the evaluation. Then, we perform human evaluation of the quality acceptability of samples in different score intervals. The criteria for this evaluation are provided in the Appendix~\ref{appendix:human_evaluation}. Through this process, we can assess the difference between BertEval's scores and human quality acceptability.
Figure~\ref{fig:human_evaluation_in_quality} presents the results of the quality evaluation. Two key observations can be made: First, due to the high quality of our original data, human acceptability remains consistently high across the score intervals. Second, as the BertEval score increases, human acceptability shows a positive correlation, indicating strong consistency between BertEval scores and human quality assessments. This demonstrates that our quality evaluation model performs effectively in assessing content quality.
%%==================Zhang Wanyue PART BEGIN============================
\subsection{Domain Classification}
In the previous section, we construct a quality evaluation model for Chinese texts to facilitate the selection of high-quality training data. In contrast, this section focuses on classifying the domain of each text. In this work, we define 11 distinct domain types based on the most common application areas of LLMs: Mathematics, Books, Law, Finance, Education, Dialogue, Encyclopedia, News, Medicine, Technology, and General. We then classify each text into one of these domains using a hybrid approach that combines rule-based methods with model-driven techniques. The details of this approach are outlined below.
%In the above section, we construct a quality evaluation model for Chinese texts to facilitate the selection of high-quality training data. Different with that, this section will focus on the classification of domain for each text. In this work, we define 12 distinct domain types based on the most common application areas of LLMs, including Military, Mathematics, Books, Law, Finance, Education, Dialogue, Encyclopedia, News, Medicine, Technology, and General domain. Then we classify each text into these domains using a hybrid approach that combines rule-based methods with model-driven techniques. The details of this work are presented in the following.
%quality labels and toxicity labels for the dataset to facilitate the selection of high-quality training data by users. However, given that our dataset integrates web texts from multiple diverse domains, the existing labels fall short in meeting the requirements for domain-specific model training. Most publicly available large-scale pre-training datasets are annotated with coarse-grained categories, which limits their applicability to more specialized applications. To bridge this gap, we have developed an additional model aimed at achieving fine-grained domain classification.
%Specifically, our goal is to build a rule-based and model-based interactive guidance classification system that aims to provide accurate domain specific single label and multi label classification for each data item. The model encompasses 12 fine-grained domain-specific classification labels: Military, Mathematics, Books, Law, Finance, Education, Dialogue, Encyclopedia, News, Medicine, Technology, and General Topics. This fine-grained classification system enables users to more precisely select domain-specific data required for large-scale model training, thereby enhancing the relevance and effectiveness of the training process.
\subsubsection{Classification Algorithm}
\label{sec:domain classication algorithm}
In this section, we develop a domain classification system that combines rule-based methods with a FastText-based model. FastText is a library for efficient learning of word representations and text classification. Compared to other approaches, FastText significantly reduces training and inference time while maintaining classification performance. As shown in Figure~\ref{fig:domain_structure}, the architecture of our approach is presented. In this system, the rule-based method is first used to perform initial annotation on the texts. Then, the FastText-based model is trained on the annotated data and iteratively optimized to improve performance. Notably, we annotate each text with both single and multi-labels. The details of this architecture are presented below.
%In this section, we develop a domain classification system that combines rule-based methods with a FastText-based model. FastText is a libiary for efficient learning of word representations and text classificaiton. Compared to other approaches, FastText could significantly reduce training and inference time while maintaining classification performance. As shown in Figure~\ref{fig:domain}, it presents the architecture of our approach. In this approach, the rule-based approach is first employed to perform initial annotation on some texts. Subsequently, the FastText-based model is trained on the annotated data and then iteratively optimized to improve performance. Notably, we simultaneously annotate both single label and multi-label for each text. In the following, we will present the details of this architecture.
%In this method, the rule-based approach is first used to perform initial annotation on the data. Then, based on the annotated data, we train the fastTex model and use iterative methods to optimize the model.
%We developed an interactive rule- and model-guided classification system to provide accurate, domain-specific single-label and multi-label classifications for each data item. The architecture of domains classify
%process is illustrated in Figure~\ref{fig:domain}.In terms of model selection, FastText was chosen due to its high efficiency in training and testing while maintaining accuracy, as well as its independence from pre-trained word embeddings. The FastText algorithm, therefore, serves as the foundation for constructing a fine-grained, multi-label classification model. The methodology consists of three main steps: rule-based classification, model-based classification, and interactive refinement.
% \begin{figure}[htpb!]
% \centering
% \includegraphics[width=8cm]{picture/domains.pdf}
% \caption{The Architecture of Domains Classification Algorithm}
% \label{fig:domain}
% \end{figure}
% \begin{figure}[!h]
% \centering
% \includegraphics[width=8cm]{picture/domain_traning_data.pdf}
% \caption{Distribution of Training Data across different Domains}
% \label{fig:domain_training_data}
% \end{figure}
\begin{figure}[!h]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{picture/domain_traning_data.pdf}
\caption{Distribution of Training Data across different Domains}
\label{fig:domain_training_data}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=0.7\textwidth]{picture/domains.pdf}
\caption{The Architecture of Domains Classification Algorithm}
\label{fig:domain_structure}
\end{subfigure}
\caption{The Overview of BERTEval Training Data Composition and Model Architecture.}
\label{fig:combined_figure_2}
\vspace{-4mm}
\end{figure}
\paragraph{Rule-Based Classification}
Initially, a rule-based classification approach is used to assign preliminary labels based on expert-curated keywords. For each category, 20 to 50 keywords are designated, and their occurrences in a given text are monitored, with a frequency threshold set between 3 to 5 non-repeating instances. Texts can be assigned either a single label or multiple labels based on keyword matches. If no category-specific keywords are identified, the text is labeled as "general."
%Initially, a rule-based classification approach is employed to assign preliminary labels using expert-curated keywords. For each category, 20 to 50 keywords are designated, and their occurrence in a given text is monitored, with a frequency threshold set between 3 to 5 non-repeating instances. Texts can be assigned either single or multiple labels based on keyword matches. If no category-specific keywords are identified, the text is labeled as "general."
%Initially, a rule-based classification approach is adopted to provide preliminary labels based on expert-curated keywords. For each category, 20 to 50 keywords are assigned, and the presence of these keywords in a given text is tracked with a frequency threshold between 3 to 5 occurrences (non-repeating). Each text could receive single or multiple labels based on the keyword match. If no category-specific keywords are detected, the text will be assigned a "general" label.
\paragraph{FastText-Based Classification}
To enhance the robustness of domain classification, we introduce a FastText-based approach in addition to the rule-based methods. In this approach, the texts annotated by the rule-based method are used to guide the training of our FastText model. Specifically, two prediction strategies are implemented to support both single-label and multi-label classification. For single-label predictions, the parameter \( k=1 \) is set to return the label with the highest probability. For multi-label classification, the parameter \( k=-1 \) (indicating no limit on the number of returned labels) and a probability threshold of 0.3 are specified, allowing the model to return all labels with probabilities exceeding 0.3.
%In order to enhance the robustness of domain classification, we further introduce a FastText-based approach on the basis of rule-based methods. In this approach, we use the annotated texts by rules to guide the training of our FastText model. Specifically, in this approach two prediction strategies are implemented to facilitate both single-label and multi-label classification. For single-label predictions, the parameter \( k=1 \) is set to return the label with the highest probability. For multi-label classification, the parameter \( k=-1 \) (no limit on the number of returned labels) and a probability threshold of 0.3 is specified, allowing the model to return all labels with probabilities exceeding 0.3.
\paragraph{Iterative Optimization}
To further improve classification performance, we introduce an iterative optimization approach in this section. In this approach, we use a confidence threshold to filter the classification results of the FastText model and update the keyword set of the rule-based method based on these filtered results. Using the updated keyword set, we classify additional texts and integrate both the rule-based classifications and the data filtered by the confidence threshold into the training dataset. Finally, the FastText model is retrained using the updated data. Through these steps, we iteratively optimize the FastText model and construct a more powerful domain classification system.
%To further improve the classification performance, in this section we introduce an iterative optimization approach. In this approach, we use a confidence threshold to filter the classification results of FastText model, and then update the keyword set of rule-based method based on these filtered results. Using the updated keyword set, we classify some more texts and then integrate both the rule-based classifications and the data filtered by confidence thresholds into the training dataset. Finally, the FastText model is retrained using the updated training data. Through the above steps, we can optimize the FastText model iteratively, and construct a powerful domain classification system.
%To enhance classification accuracy and generalizability, an iterative process is employed in section. High-confidence predictions (i.e., those with a confidence threshold exceeding 0.9) from the model-based classification were used to iteratively refine the rule-based keyword set, generating progressively improved training data for the model. This interactive process allowed for the fine-tuning of labels, especially for certain data categories with characteristic structures (e.g., "instruction" data). Labels in these categories could be directly enriched based on rule-based criteria, ensuring a more comprehensive label coverage for structured text formats.
%Through the above methods, we can construct a high-performance domain classification system to classify all texts from high-quality dataset.
%This interactive, hybrid approach capitalizes on both the precision of rule-based methods and the adaptability of model-based classification, yielding a classification system that is robust across diverse data domains and responsive to iterative refinements in the labeling process.
\subsubsection{Training and Testing Data}
To enhance the diversity of the data, we collect training and testing data from various sources, including news articles, books, forums, product descriptions, and more. After several rounds of iterative optimization during the training stage, we select nearly 300,000 texts as training data and 300 texts from different domains as testing data. As shown in Table~\ref{domain_train_data}, this table presents the size of the training and testing data from different domains. Additionally, Figure~\ref{fig:domain_training_data} illustrates the detailed breakdown of the composition of the training data.
%To enhance the diversity of the data, we collect training and testing data from various sources, such as news articles, books, forums, product descriptions and so on. After several rounds of iterative optimization in the training stage, we finally select nearly 300,000 texts as training data, and 300 texts from different domains as testing data. As shown in Table~\ref{domain_train_data}, it presents the size of training and testing data from different domains. Additionally, Figure~\ref{fig:domain} illustrates the detailed breakdown of the composition of the training data.
%Our data was sourced from a variety of domains, including news articles, books, forums, product descriptions, and instructional texts. Given that a diversified corpus can enhance the generalization and robustness of classification models, we randomly selected 300,000 entries from these multiple data sources. These entries were subsequently divided into training and testing sets at a ratio of 99:1. The specific distribution of the data is illustrated in Table~\ref{domain_train_data}, and figure~\ref{fig:domain} presents a detailed breakdown of the composition of the training data.
%book dialogue education encyclopedia finance law math medicine military news technology general
\begin{table}[!h]
\caption{Composition of Training and Testing Data.}
\label{domain_train_data}
\centering
\begin{tabular}{lcc}
\toprule
Domain & Training Size ($\times 10^4 $) & Test Size \\
\midrule
book & 18581 & 20 \\
dialogue & 56385 & 59 \\
education & 39573 & 44 \\
encyclopedia & 101393 & 107 \\
finance & 21673 & 23 \\
law & 18315 & 13 \\
math & 3447 & 2 \\
medicine & 23872 & 28 \\
news & 80111 & 85 \\
technology & 55237 & 56 \\
general & 91353 & 89 \\
\midrule
Total number of data & 299700 & 300 \\
Total number of label & 509940 & 526 \\
\bottomrule
\end{tabular}
\end{table}
%The dataset used for training the classification model contained a total of 299,700 entries, while the test dataset consisted of 300 entries. It is important to note that the distribution of categories across both the training and test sets was imbalanced. Some categories contained significantly more examples than others, potentially impacting the model's ability to generalize across all categories. The imbalance in the data was visualized and analyzed using a combination of tables and graphical plots, which are provided in the results section.
\subsubsection{Evaluation}
\paragraph{Evaluation Metric}
To rigorously evaluate the performance of the classification model, it is essential to use standard evaluation metrics that assess both its accuracy and comprehensiveness. For single-label classification task, Precision and Recall will be used, as they provide a balanced understanding of the model's performance. Precision represents the ratio of correctly predicted labels to the total number of labels predicted, while Recall is the ratio of correctly predicted labels to the total number of actual labels. The formulas for these two metrics are as follows:
%To rigorously evaluate the performance of the classification model, it's essential to use standard evaluation metrics that assess both its accuracy and comprehensiveness. For this task, Precision and Recall will be used because they can provide a balanced understanding of the model's performance. Precision represents the ratio of correctly predicted labels to the total number of labels predicted, while Recall recall is the ratio of correctly predicted labels to the total number of actual labels. The formulas of these two metrics are given as follows:
%To rigorously evaluate the performance of the classification model, we employ standard evaluation metrics, including precision and recall. Precision, in this context, is defined as the ratio of correctly predicted labels to the total number of labels predicted by FastText, while recall is the ratio of correctly predicted labels to the total number of actual labels. These metrics are commonly used in multi-label classification tasks to capture both the model’s accuracy and its comprehensiveness in identifying relevant labels. The formulas for precision and recall are given as follows:
\begin{equation}
\text{Precision} = \frac{\text{Number of Correctly Predicted labels}}{\text{Total Number of Predicted labels}}
\end{equation}
\begin{equation}
\text{Recall} = \frac{\text{Number of Correctly Predicted labels}}{\text{Total Number of True labels}}
\end{equation}
Different with the metric above, multi-label classification adopts micro-averaged precision and recall to evaluate the performance of models. To calculate micro-averaged precision, we sum up all true positive predictions across all labels and divide it by the sum of true positive and false positive predictions across all labels. Similarly, micro-averaged recall is computed as the total true positive predictions divided by the sum of true positive and false negative predictions. These metrics reflect the overall performance of the model across all samples and labels, treating each instance of a label equally regardless of its class.
% 需要加入一个计算准确率和召回率的实例吗？感觉可以加一个。
\paragraph{Evaluation Results}
To evaluate the classification model's performance, we manually annotated all texts in the test dataset with both single-label and multi-label classifications. As shown in Table~\ref{tab:domain_eval_manual}, the results indicate that single-label classification achieves a precision of 88.33\% and a recall of 64.15\%. For multi-label classification, we calculate the micro-averaged precision and recall, which achieve 74.48\% and 79.35\% respectively. These findings highlight the model's strong capability in domain classification, even when dealing with complex data scenarios.
%To evaluate the classification model's performance, we manually annotated all texts in the test dataset with both single-label and multi-label classifications. As shown in Table~\ref{tab:domain_eval_manual}, the results indicate that single-label classification achieves a precision of 88.33\% and a recall of 64.15\%. For multi-label classification, the precision and recall are 74.48\% and 79.35\%, respectively. These findings highlight the model's strong capability in domain classification, even when dealing with complex data scenarios.
%To assess the performance of the classification model, we manually annotate all texts in the test dataset with both single-label and multi-label classifications. As shown in Table~\ref{tab:domain_eval_manual}, it presents the evaluation results of domain classification. From this table, we can see that the precision of single-label classification achieves 88.33\%, while its recall is 64.15\%. And the precision and recall of multi-label classification are 74.48\% and 79.35\% respectively. This result indicates that our model performs well in domain classification of complex data.
%To further validate the model's performance, we conducted a manual evaluation. Table~\ref{tab:domain_eval_manual} provides the results of the manual evaluation. For single-label data, the precision was 88.33\% and the recall was 64.15\%. For multi-label data, the precision was 74.48\% and the recall was 79.35\%. Compared with automatic evaluation, manual evaluation has slightly lower accuracy and recall rate, which may be due to the requirement for human experts to label all applicable domain labels during the manual evaluation process. Therefore, manual labeling considers more contextual information and more detailed judgment criteria. However, the results of manual evaluation still support the conclusion of automatic evaluation, with single labels achieving high accuracy and multiple labels achieving a good balance between accuracy and recall.This proves that our domain labels have achieved good alignment with human judgment.
%\paragraph{Manual Evaluation}
%To further validate the model’s performance beyond quantitative metrics, a qualitative assessment was conducted on a subset of the dataset. Specifically, a random sample of 300 data instances was manually annotated by human experts to serve as a gold standard for comparison. We then compared the model's predictions on this subset against the expert-annotated labels to evaluate real-world performance.
%\paragraph{Evaluation Results}
%In this study, we evaluated the model's performance on different label types using automatic evaluation methods. Table~\ref{tab:domain_eval_metric} shows the precision and recall for single-label and multi-label data. For single-label data, the model achieved a precision of 90.67\% and a recall of 65.03\%. This is because the reference label is set to multiple labels, and as long as a single label appears in the reference label, the prediction is considered correct. Therefore, the model can accurately identify most positive cases, but a single label is difficult to fully cover all positive cases. For multi-label data, the model's precision was 82.66\% and recall was 84.55\%, demonstrating a good balance in handling multi-label classification tasks. The model maintains high accuracy while effectively identifying the majority of positive cases. Overall, the setting of single domain labels and multi domain labels enables users to better customize their training datasets according to specific performance requirements.
%\begin{table}[htbp!]
% \caption{Domain Data Evaluation.}
% \label{tab:domain_eval_metric}
% \centering
% \begin{tabular}{lcc}
% \toprule
% Label Type & Precision(\%) & Recall(\%) \\
% \midrule
% Single-Label & 90.67 & 65.03 \\
% Multi-Label & 82.66 & 84.55 \\
% \bottomrule
% \end{tabular}
%\end{table}
%To further validate the model's performance, we conducted a manual evaluation. Table~\ref{tab:domain_eval_manual} provides the results of the manual evaluation. For single-label data, the precision was 88.33\% and the recall was 64.15\%. For multi-label data, the precision was 74.48\% and the recall was 79.35\%. Compared with automatic evaluation, manual evaluation has slightly lower accuracy and recall rate, which may be due to the requirement for human experts to label all applicable domain labels during the manual evaluation process. Therefore, manual labeling considers more contextual information and more detailed judgment criteria. However, the results of manual evaluation still support the conclusion of automatic evaluation, with single labels achieving high accuracy and multiple labels achieving a good balance between accuracy and recall.This proves that our domain labels have achieved good alignment with human judgment.
\begin{table}[htbp!]
\caption{Domain Data Evaluation.}
\label{tab:domain_eval_manual}
\centering
\begin{tabular}{lcc}
\toprule
Label Type & Precision(\%) & Recall(\%) \\
\midrule
Single-Label & 88.33 & 64.15 \\
\midrule
& Micro-Precision(\%) & Micro-Recall(\%) \\
\midrule
Multi-Label & 74.48 & 79.35 \\
\bottomrule
\end{tabular}
\end{table}
%%==================Zhang Wanyue PART END============================
\subsection{Toxicity Evaluation}
Text data from the internet often contains various types of toxic content, which can significantly compromise the security of LLMs. To address this issue, we propose using a toxicity evaluation model to assess the toxicity of all texts and generate corresponding toxicity labels and scores. Due to its efficiency in reducing training and inference time while maintaining strong classification performance, FastText is once again employed to construct the toxicity evaluation model. The details of this approach are presented below.
%Text data from the internet often contains various toxic content, which seriously affects the security of LLMs. In order to mitigate this problem, we propose to use a toxicity evaluation model to assess the toxicity of all texts, and then generate toxicity labels and toxicity scores for each text. Due to its ability to effectively reduce training and inference time while maintaining good classification performance, in this section FastText is once again used to construct a toxicity evaluation model. The details of this approach are presented in the following.
%implement a toxicity label within our dataset. Specifically, to evaluate and score the toxicity levels within the dataset, we trained a fastText model. Compared to the BERT model, the fastText model reaches an optimal balance of processing performance and computational efficiency, maintaining high accuracy while significantly reducing both training and inference time. As presented in Figure~\ref{fig:Toxicity_system}, this approach facilitates rapid model iteration.
%Textual data from the internet frequently includes toxic content, which compromises data safety. In order to mitigate this problem, we implemented a toxicity label within our dataset. Specifically, to evaluate and score the toxicity levels within the dataset, we trained a fastText model. Compared to the BERT model, the fastText model reaches an optimal balance of processing performance and computational efficiency, maintaining high accuracy while significantly reducing both training and inference time. As presented in Figure~\ref{fig:Toxicity_system}, this approach facilitates rapid model iteration.
\begin{figure}[htp]
\centering
\includegraphics[width=10cm]{picture/Toxicity.pdf}
\caption{The Architecture of Toxicity Evaluator}
\label{fig:Toxicity_system}
\vspace{-4mm}
\end{figure}
\subsubsection{Methodology}
In the training process of toxicity evaluation models, the scale and quality of training data have a significant impact on the performance of the models. However, an analysis of existing datasets reveals that toxicity data is often limited in size and lacks diversity, which substantially restricts the effectiveness of these models. At the same time, testing publicly available large language models (LLMs) has shown that some of these models possess strong toxicity evaluation capabilities, enabling them to annotate toxicity data both quickly and accurately. Based on this observation, this paper proposes an LLM-in-the-loop approach to improve the training of toxicity evaluation models, resulting in significant performance enhancements. The details of this method are outlined below.
%However, an analysis of existing datasets reveals that toxicity data is often limited in size and lacks diversity, which significantly constrains the effectiveness of toxicity evaluation models. Meanwhile, testing publicly available large language models shows that some of these models possess strong toxicity evaluation capabilities, enabling them to annotate toxicity data quickly and accurately. Based on this observation, this paper proposes a LLM-in-the-loop method to enhance the training of toxicity evaluation models, ultimately achieving significant performance improvements. The details of this method are outlined below.
%The quality of the training data has a significant impact on the performance of the model. However, datasets for the detection of toxic language remains scarce. In preliminary research, we examined existing Chinese toxicity datasets and found that they suffer from limited scale and a lack of data diversity, which constrains the model's capacity to generalize to large-scale datasets.
%Leveraging in-context learning (ICL)~\cite{dong2022survey} and instruction following capabilities~\cite{brown2020language}, large language models (LLMs) can reason based on prompts and context and exhibit robust performance in toxicity detection tasks. To address the shortage of training data, we utilized an LLM-in-the-loop methodology, applying an iterative process with the fastText model and LLMs to expand and optimize the training set, thereby enhancing the model's classification capability.
\paragraph{Initail Training}
In the training process of evaluation models, we first combine several high-quality Chinese toxicity datasets—COLD~\cite{deng2022cold}, ToxiCN~\cite{toxicn}, SWSR~\cite{jiang2022swsr}, and CDial-Bias~\cite{zhou2022towards}—to create an initial training set. To increase the diversity of the dataset and better align its distribution with that of our specific dataset, we sample a subset of texts from our high-quality dataset to serve as benign samples. Additionally, to ensure a balanced distribution between toxic and benign samples, we doubled the number of toxic samples in the initial training set. With this augmented dataset, we train the initial FastText model, referred to as Toxic Classifier R0. As shown in Table~\ref{toxicity_data}, it presents a detailed breakdown of the source and scale of each portion of the initial training dataset.
%In the training process of evaluation models, we first integrate existing high-quality Chinese toxicity datasets, COLD~\cite{deng2022cold}, ToxiCN~\cite{toxicn}, SWSR~\cite{jiang2022swsr}, and CDial-Bias~\cite{zhou2022towards}, into an initial training set. Furthermore, in order to augment the diversity of the dataset and ensure that the distribution of the training set more closely aligns with that of our dataset, we sample a subset of texts from our high-quality dataset to be utilized as benign samples. Additionally, to maintain a balanced distribution between toxic and benign samples, the number of toxic samples in the training set was doubled. Using this data, we trained an initial fastText model, named Toxic Classifier R0.Table~\ref{toxicity_data} shows a detailed account of the source and scale of each portion of the initial training dataset.
%To enhance the efficiency of data collection, we integrated existing high-quality Chinese toxicity datasets, COLD~\cite{deng2022cold}, ToxiCN~\cite{toxicn}, SWSR~\cite{jiang2022swsr}, and CDial-Bias~\cite{zhou2022towards}, into an initial training set. Furthermore, in order to augment the diversity of the dataset and ensure that the distribution of the training set more closely aligns with that of our dataset, we directly sampled a subset of data from our large-scale dataset to be utilized as benign samples. Additionally, to maintain a balanced distribution between toxic and benign samples, the number of toxic samples in the training set was doubled. Using this data, we trained an initial fastText model, named Toxic Classifier R0.Table~\ref{toxicity_data} provides a detailed account of the source and scale of each portion of the initial training dataset.
\begin{table}[htbp]
\caption{Composition of Toxicity Initial Training Data.}
\label{toxicity_data}
\centering
\begin{tabular}{llc}
\toprule
Source & Toxic Size & Benign Size \\
\midrule
COLD & 36162 & 19439 \\
ToxiCN & 12922 & 5550 \\
SWSR & 6186 & 5876 \\
CDial-Bias & 12986 & 19013 \\
Samples & 0 & 80000 \\
\midrule
Sum & 68256 & 129878 \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{LLM-in-the-loop Training}
In this section, LLMs will be utilized to further enhance the performance of toxicity evaluation models. In this approach, we first select a subset from our large-scale dataset and then Toxic Classifier R0 is employed to score each sample. Next, texts with toxicity scores greater than 0.5 will be screened out to form a candidate set. After that the candidate subset will be scored by a LLM, which categorize them into toxic and benign classes, thereby forming a new training set. Here, the chosen LLM is Qwen2.5-32B-Instruct~\cite{qwen2.5}. Finally, we integrate the new training set with the initial training set, and then retrain the FastText model with this refined training dataset. In this paper, we conduct two iterations on the above method, significantly improving the performance of the evaluation model.
%We extracted a subset from our large-scale dataset and Toxic Classifier R0 was applied to score each sample. Those with toxicity scores above 0.5 were selected as candidates for further analysis. Subsequently, the candidate subset was scored by the Qwen2.5-32B model, which categorized them into toxic and benign classes, thereby forming a new training set. The fastText model was then retrained using this refined dataset, thus enhancing its performance. This iterative data collection process enabled the continual augmentation of the training set, improving the model's generalization capability and classification accuracy. Using this approach, we conducted two rounds of iteration. During the first round of iteration, the resulting model was named Toxic Classifier R1, while in the subsequent round, it was named Toxic Classifier R2.
%\paragraph{Data-Specific Toxicity Classification}
%Notwithstanding the robust generalisation capabilities exhibited by models trained using the LLM-in-the-loop approach across a variety of datasets, their performance remains suboptimal when applied to specialised datasets, such as those containing Chinese poetry. Given the low incidence of toxic samples in these datasets, we directly extract some samples and integrate them into the training set with the aim of enhancing the model's performance across different types of data. Furthermore, we propose a hybrid approach that integrates rule-based and model-based methods to handle mathematical formulas. Specifically, we calculate the ratio of numerical and symbolic elements within each sentence and classify any sentence in which this ratio exceeds $50\%$ as an instruction containing a mathematical formula, thus categorising it as a non-toxic sample.
%This multi-method, collaborative and data-specific classification system exploits the distinctive characteristics of heterogeneous datasets, thereby ensuring both robust and efficient classification performance.
\subsubsection{Evaluation}
To conduct a comprehensive evaluation across data from various sources, a test set is randomly selected from the large-scale dataset and manually annotated with toxic or benign labels. Due to the imbalance between toxic and benign samples in our dataset, toxic and benign samples are collected separately during the test data sampling. The final test set consists of 300 toxic samples and 300 benign samples.
%In order to conduct a comprehensive evaluation across the data from various sources, a test set is randomly selected from the large-scale dataset, and then annotated manually with toxic or benign label. Due to the imbalance between toxic and benign samples in our dataset, we collect toxic and benign samples separately when sampling the test data. This test set includes 300 toxic and 300 benign samples.
Given the sampling methodology of the test set, Precision and Specificity were selected as metrics to evaluate the performance of the model on toxic and benign data, respectively. In this study, toxicity is defined as the positive class, whereas benignity is defined as the negative class. Precision is defined as the ratio of true positive predictions to total positive predictions, while Specificity measures the ratio of true negative predictions to total negative predictions. The formulas of these two metrics are shown below.
\begin{equation}
\text{Precision} = \frac{\text{True Positive}}{\text{True Positive + False Positive}}
\end{equation}
\begin{equation}
\text{Specificity} = \frac{\text{True Negative}}{\text{True Negative + False Negative}}
\end{equation}
As shown in Table~\ref{toxicity_evaluation}, the evaluation results of our model are presented. From this table, we can observe that the precision of our toxicity evaluation model for toxic texts reaches 83.67\%, while the specificity for benign texts is 97.67\%. These results indicate that our model demonstrates excellent performance in toxicity evaluation.
%As shown in Table~\ref{toxicity_evaluation}, it presents the evaluation results of our model. From this table, we can see that the precision of our toxicity evaluation model for toxic texts achieve 83.67\%, while the specificity for benign texts is 97.67\%. This result indicates that our model has excellent toxicity evaluation capability.
%Due to the imbalance between toxicity and non toxicity data in the dataset, we will sample the toxicity and non toxicity data separately when sampling the test data to ensure that there are enough toxicity samples in the test data
%Due to the imbalance between toxicity data and non-toxic data in the dataset, we will sample the toxicity data and non-toxic data separately when sampling the test data
%Specifically, o ensure adequate toxic sample representation in the test set, we select toxic and benign samples independently. Manual annotation was then conducted on all selected samples, resulting in a final test set comprising 300 toxic and 300 benign samples.
%Besides, precision is employed as the principal evaluation metric to assess the efficacy of the toxicity classification model. The imbalance between toxic and benign samples in our dataset makes it challenging to achieve a balanced test set through direct sampling, as this method often results in test sets lacking toxic samples. To ensure adequate toxic sample representation in the test set, we select toxic and benign samples independently. Manual annotation was then conducted on all selected samples, resulting in a final test set comprising 300 toxic and 300 benign samples. Table~\ref{toxicity_evaluation} provides a summary of the evaluation results.
%In order to conduct a comprehensive evaluation of the model's performance across the entire dataset, a test set was randomly selected. Precision was employed as the principal evaluation metric to assess the efficacy of the toxicity classification model. The imbalance between toxic and benign samples in our dataset makes it challenging to achieve a balanced test set through direct sampling, as this method often results in test sets lacking toxic samples. To ensure adequate toxic sample representation in the test set, we select toxic and benign samples independently. Manual annotation was then conducted on all selected samples, resulting in a final test set comprising 300 toxic and 300 benign samples. Table~\ref{toxicity_evaluation} provides a summary of the evaluation results.
\begin{table}[htbp]
\caption{The Evaluation of Toxicity Classifier.}\label{toxicity_evaluation}
\centering
\begin{tabular}{lccc}
\toprule
Category & Precision ($\%$) & TP+FP & TP \\
\midrule
Toxic & 83.67 & 300 & 251 \\
\midrule
& Specificity ($\%$) & TN+FN & TN \\
\midrule
Benign & 97.67 & 300 & 293 \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Dataset Statistics and Comparison}
After processing the collected data with the modules outlined above, this paper constructs a clean Chinese text dataset, ChineseWebText2.0, which consists of 3.8 TB of data. Each text in this dataset is assigned a quality score, domain labels, a toxicity score, and a toxicity label. As shown in Table~\ref{Comparison}, we compare our dataset with several other publicly available pre-training corpora. Most of these datasets primarily focus on collecting clean texts from various sources, while neglecting fine-grained text annotations. In this comparison, C4~\cite{2020T5C4}, The Pile~\cite{2020_pile}, and REFINEDWEB~\cite{2023refinedweb} are three public English datasets, while WuDaoCorpora~\cite{wudao}, ROOTS-zh~\cite{roots}, WanJuan1.0-zh~\cite{wanjuan}, MAPCC~\cite{mapcc}, IndustryCorpus~\cite{industrycorpus}, and ChineseWebText1.0~\cite{chinesewebtext} are Chinese corpora. Specifically, the texts in IndustryCorpus are tagged with domain labels, while each text in ChineseWebText1.0 is annotated with a quality score. Compared to these prior works, our dataset is both the largest and most recent Chinese dataset, and it features four types of fine-grained annotations for each text. This extensive annotation can aid LLM researchers in enhancing model performance in specific domains while improving safety.
%After processing the collected data with the modules above, this paper constructs a clean Chinese text dataset ChinesewebText2.0, which consists of 3.8 TB data. And each text in this dataset is assigned a quality score, domain labels, a toxicity score and a toxicity label. As shown in Table~\ref{Comparison}, we compare our dataset with some other public pre-training corpora. Most of these work mainly focus on the collecting of clean texts from various sources, ignoring the fine-grained information of texts. In these work, C4~\cite{2020T5C4}, The Pile~\cite{2020_pile}, and REFINEDWEB~\cite{2023refinedweb} are three public English datasets, while WuDaoCorpora~\cite{wudao}, ROOTS-zh~\cite{roots}, WanJuan1.0-zh~\cite{wanjuan}, MAPCC~\cite{mapcc}, IndustryCorpus~\cite{industrycorpus} and ChineseWebText1.0~\cite{chinesewebtext} are Chinese corpora. Specifically, the texts in IndustryCorpus are tagged with domain labels, while each text of ChineseWebText1.0 is annotated with quality score. Compared to these previous work, our dataset is the latest and largest Chinese dataset, and each text in our dataset is annotated with four types of fine-grained information, which could help LLM researchers to enhance the model's capabilities in specific domains and improving its safety.
%After processing the collected data using our chain-of-tools approach, we constructed a clean dataset, ChineseWebText, comprising 3.8TB of data. In the Table~\ref{Comparison}, we compare our model with several other publicly available pre-trained corpora. Specifically, C4~\cite{2020T5C4}, The Pile~\cite{2020_pile}, and REFINEDWEB~\cite{2023refinedweb} are three public English datasets, while WuDaoCorpora~\cite{wudao}, ROOTS-zh~\cite{roots}, WanJuan1.0-zh~\cite{wanjuan}, and IndustryCorpus~\cite{industrycorpus} represent three Chinese corpora. From the table, it is evident that our dataset is designed to construct large-scale, high-quality Chinese datasets with multi-dimensional and fine-grained information.
%Unlike these existing datasets, our dataset not only assigns a quality score to each text, but also includes single-domain and multi-domain labels. Additionally, it provides toxicity scoring, enabling LLM researchers to select data based on specific needs and quality thresholds.
\begin{table}[htbp!]
\begin{center} % 使用 center 环境
\begin{threeparttable}
\caption{Overview of output datasets. Note that QS$=$Quality Scoring, DC$=$Domain Classification, TC$=$Toxicity Classification, TS$=$Toxicity Scoring.}
\label{Comparison}
\begin{tabular}{*{8}{c}}
\toprule
%Dataset & Lang. & Availability & Public Size & QS & DC & TC & TS \\
\multirow{2}{*}{Dataset} & \multirow{2}{*}{Lang.} & \multirow{2}{*}{Availability} & \multirow{2}{*}{Public Size} & \multicolumn{4}{c}{Fine-grained Information}\\
%\multirow{2}{*}{Dataset & Lang. & Availability & Public Size} & \multicolumn{4}{c}{Fine-grained Information}\\
\cmidrule(lr){5-8} & & & & QS & DC & TC & TS\\
\midrule
C4~\cite{2020T5C4} & EN & Public & 807GB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
The Pile~\cite{2020_pile} & EN & Public & 825GB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
REFINEDWEB~\cite{2023refinedweb} & EN & Public & 2.8TB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
WuDaoCorporal~\cite{wudao} & ZH & Pratly Public & 200GB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
ROOTS-zh~\cite{roots} & ZH & Public & 265GB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
WanJuan1.0-zh~\cite{wanjuan} & ZH & Public & 550GB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
MAPCC~\cite{mapcc} & ZH & Public & 3TB & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
IndustryCorpus-zh~\cite{industrycorpus} & ZH & Public & 1TB & \textcolor{red}{\usym{1F5F4}} & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} &\textcolor{red}{\usym{1F5F4}}\\
ChineseWebText1.0~\cite{chinesewebtext} & ZH & Public & 1.4TB & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}}\\
\textbf{ChineseWebText2.0} & ZH & Public & 3.8TB & \textcolor{green}{\usym{2714}} & \textcolor{green}{\usym{2714}} & \textcolor{green}{\usym{2714}} & \textcolor{green}{\usym{2714}}\\
\bottomrule
\end{tabular}
\end{threeparttable}
\end{center}
\end{table}
\section{Data Analysis}
\subsection{Removal Rate for Different Stages}
To provide a high-level overview of the preparation and preprocessing stages, Figure~\ref{fig:fig4removal-rate} illustrates the processing workflow along with the removal rate at each step. This figure shows the proportion of data removed at each stage and the percentage of remaining data relative to the original collected dataset. This helps readers track the progression of data through the various processing stages, from raw data to the final high-quality dataset.
%In order to provide a high-level overview of the preparation and preprocessing stages, Figure~\ref{fig:fig4removal-rate} shows the processing workflow and the removal rate of each step. This figure details the removal ratio of data from the previous step and the absolute percentage of the remaining data from the original collected dataset, thereby facilitating readers in tracking the various processing stages from the raw data to the high-quality dataset.
After collecting raw data from various sources, we initially obtain an original Chinese dataset totaling 6.6 TB. However, due to a significant amount of irrelevant and noisy content in some sources, a manual sampling analysis is performed during the preparation stage. If irrelevant text accounted for more than 30\% of a source, that data is discarded entirely. As a result, a substantial portion of the data is removed, leaving 67.68\% of the original dataset. In the preprocessing stage, four rule-based steps are implemented to filter the remaining data. First, the Data Length step remove overly short texts to ensure each text contained sufficient informational content. Next, the Character Proportion step eliminate texts with a high percentage of noisy characters, such as English, Traditional Chinese characters, or other irrelevant symbols. Finally, the Sensitive Words and Deduplication steps are applied to remove toxic content and duplicate texts from the dataset. After the preprocessing stage, a high-quality Chinese text dataset totaling 3.8 TB is produced. In the next stage, each text in this high-quality dataset will be enriched with fine-grained annotations, including a quality score, domain labels, a toxicity score, and a toxicity label.
%After collecting raw data from various sources, we initially obtain a original Chinese dataset totaling 6.6 TB. However, due to a significant amount of irrelevant and noisy content in some sources, a manual sampling analysis is performed in preparation stage. If irrelevant text accounted for more than 50\% of a source, the data from that source will be discarded entirely. As a result, a substantial portion of the data is removed during the preparation stage, retaining only 67.68\% of the original dataset. In preprocessing stage, four rule-based steps are implemented to filter the remained data. First, the Data Length step remove overly short texts to ensure that each text contains sufficient informational content. Next, the Character Proportion step eliminate texts with a high percentage of noisy characters, such as English, Traditional Chinese characters, or other irrelevant symbols. Finally, the Sensitive Words step and the Deduplication step are employed to remove toxic content and duplicate texts from the dataset. After the preprocessing stage, we produce a high-quality Chinese text dataset totaling 3.8 TB. In the next stage, each text in this high-quality dataset will be enriched with fine-grained annotations, including a quality score, domain lablels, a toxicity score and a toxicity label.
%If irrelevant text accounted for more than 50\% of a source, the data from that source will be discarded entirely. As a result, a substantial portion of the data was removed during the preparation stage, retaining only 67.68% of the original dataset. toxic content and removed duplicate texts from the dataset.
%Figure 5 provides an intuitive overview of the entire coarse-grained data filtering process, detailing the remaining data size and corresponding cleaning rate at each processing step. Additionally, Figure 3 further illustrates the removal rate of each dataset at each processing step, allowing readers to trace the process from raw data to final data.
%Specifically, through manual inspection, we removed three datasets—BAAI-MTP, COIG, and Yuan1.0—that were unsuitable for use as pre-training corpora, leaving five datasets—MAP-CC, SkyPile, WuDao, ChineseWebText, and WanJuan—for further processing. The manually removed data amounted to approximately 2.2 TB, with about 4.5 TB remaining, making the proportion of removed data approximately 67.68\%. In the subsequent rule-based processing steps, no data was removed from the WuDao and ChineseWebText1.0 datasets. For the MAP-CC, SkyPile, and WanJuan datasets, the proportion of removed data relative to the original data sizes was 25.41\%, 4.62\%, and 14.92\%, respectively. Considering all cleaning steps, the proportion of removed data across all datasets amounted to 43.82\% of the original size, with manual inspection accounting for the largest removal at 32.32\%, and the rule-based steps removing an additional 11.50\% of the original dataset size.
\begin{figure}[htpb!]
\centering
\includegraphics[width=0.8\textwidth]{picture/data_statistics.pdf}
\caption{The proportion of data removed from the originally collected data in each processing step. The gray bars represent the proportion of data removed in each step relative to the data remaining before that step, while the other colored bars represent the retained data and its proportion relative to the originally collected data.}
\label{fig:fig4removal-rate}
\end{figure}
%\begin{table}[htbp]
% \small
% \renewcommand\arraystretch{0.8}
% \setlength{\tabcolsep}{1mm}
% \caption{The proportion of data removed in each cleaning step and the size of the data retained.}
% \label{remain_size}
% \centering
% \begin{tabular}{cp{1.7cm}p{1.7cm}p{1.7cm}p{1.9cm}p{1.7cm}p{1.7cm}}
% \toprule
% \multirow{2.5}{*}{Data Source} & \multicolumn{6}{c}{Size After filtering operation(GB)} \\
% \cmidrule{2-7}
% & \centering Original Size & \centering Manual Selection & \centering Data Length & \centering Proportion of Characters & \centering Sensitive Words & \centering Internal Duplication \arraybackslash \\
% \midrule
% \textbf{MAP-CC} & \centering 2702.23 & \centering 2702.23 & \centering 2659.56 & \centering 2459.14 & \centering 2108.38 & \centering 2015.63 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering - & \centering \textcolor{gray}{\textit{-1.58\%}} & \centering \textcolor{gray}{\textit{-7.42\%}} & \centering \textcolor{gray}{\textit{-12.98\%}} & \centering \textcolor{gray}{\textit{-3.43\%}} & \centering \arraybackslash \\
% \textbf{SkyPile} & \centering 591.15 & \centering 591.15 & \centering 591.15 & \centering 588.35 & \centering 575.87 & \centering 563.83 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering - & \centering \textcolor{gray}{\textit{0}} & \centering \textcolor{gray}{\textit{-0.04\%}} & \centering \textcolor{gray}{\textit{-2.11\%}} & \centering \textcolor{gray}{\textit{-2.04\%}} & \centering \arraybackslash \\
% \textbf{WanJuan} & \centering 494.72 & \centering 494.72 & \centering 481.65 & \centering 431.30 & \centering 430.86 & \centering 420.91 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering - & \centering \textcolor{gray}{\textit{-2.64\%}} & \centering \textcolor{gray}{\textit{-10.18\%}} & \centering \textcolor{gray}{\textit{-0.09\%}} & \centering \textcolor{gray}{\textit{-0.19\%}} & \centering \arraybackslash \\
% \textbf{WuDao} & \centering 174.53 & \centering 174.53 & \centering - & \centering - & \centering - & \centering 174.53 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering - & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} \arraybackslash \\
% \textbf{ChineseWebText1.0} & \centering 625.26 & \centering 625.26 & \centering - & \centering - & \centering - & \centering 625.26 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} \arraybackslash \\
% \textbf{BAAI-MTP} & \centering 542.34 & \centering - & \centering - & \centering - & \centering - & \centering - \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} \arraybackslash \\
% \textbf{COIG} & \centering 563.27 & \centering - & \centering - & \centering - & \centering - & \centering - \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} \arraybackslash \\
% \textbf{Yuan1.0} & \centering 1086.62 & \centering - & \centering - & \centering - & \centering - & \centering - \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} & \centering \textcolor{gray}{\textit{-}} \arraybackslash \\
% \textbf{2022-49} & \centering 445.29 & \centering 367.73 & \centering 352.59 & \centering 173.86 & \centering 164.34 & \centering 160.87 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-17.42\%}} & \centering \textcolor{gray}{\textit{-4.12\%}} & \centering \textcolor{gray}{\textit{-50.69\%}} & \centering \textcolor{gray}{\textit{-5.48\%}} & \centering \textcolor{gray}{\textit{-2.11\%}} \arraybackslash \\
% \textbf{2023-06} & \centering 396.40 & \centering 275.04 & \centering 263.59 & \centering 211.10 & \centering 177.44 & \centering 173.47 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-30.62\%}} & \centering \textcolor{gray}{\textit{-4.16\%}} & \centering \textcolor{gray}{\textit{-19.91\%}} & \centering \textcolor{gray}{\textit{-15.95\%}} & \centering \textcolor{gray}{\textit{-2.24\%}} \arraybackslash \\
% \textbf{2023-14} & \centering 441.46 & \centering 368.40 & \centering 354.18 & \centering 161.54 & \centering 153.27 & \centering 150.04 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-16.55\%}} & \centering \textcolor{gray}{\textit{-3.86\%}} & \centering \textcolor{gray}{\textit{-54.39\%}} & \centering \textcolor{gray}{\textit{-5.12\%}} & \centering \textcolor{gray}{\textit{-2.11\%}} \arraybackslash \\
% \textbf{2023-23} & \centering 371.96 & \centering 305.10 & \centering 292.58 & \centering 152.20 & \centering 146.90 & \centering 143.93 \arraybackslash \\
% \textit{removal rate} & \centering - & \centering \textcolor{gray}{\textit{-17.98\%}} & \centering \textcolor{gray}{\textit{-4.10\%}} & \centering \textcolor{gray}{\textit{-47.98\%}} & \centering \textcolor{gray}{\textit{-3.48\%}} & \centering \textcolor{gray}{\textit{-2.02\%}} \arraybackslash \\
% \midrule
% Total & 3835.32 & 1514.33(-2320.99) & 1507.54(-6.79) & 1466.25(-41.29) & 1450.90(-15.35) & 1420.70(-30.20) \\
% \textbf{Total} & \centering 6780.12 & \centering 4587.89 & \centering 4532.15 & \centering 4278.58 & \centering 3914.90 & \centering 3809.16 & \arraybackslash \\
% \textit{removal rate} & \centering - & \centering
% \textcolor{gray}{\textit{-32.32\%}} & \centering
% \textcolor{gray}{\textit{-1.21\%}} & \centering \textcolor{gray}{\textit{-5.53\%}} & \centering \textcolor{gray}{\textit{-7.93\%}} & \centering \textcolor{gray}{\textit{-2.30\%}} & \centering \arraybackslash \\
% \bottomrule
% \end{tabular}
%\end{table}
\subsection{Data Quality Distribution}
\begin{figure}[!h]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\textwidth]{picture/quality_distribution.pdf}
\caption{Quality Distribution}
\label{fig:quality_distribution}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\textwidth]{picture/human_acceptance.pdf}
\caption{Human Acceptance Evaluation}
\label{fig:human_acceptance}
\end{subfigure}
\caption{The Data Analysis on Quality Evaluation.}
\label{fig:combined_figure_3}
\vspace{-6mm}
\end{figure}
\paragraph{Quality Distribution} To investigate the quality distribution, we calculate the data proportions across different quality score ranges from our ChineseWebText 2.0 dataset. Figure~\ref{fig:quality_distribution} shows the proportion of data across different quality score intervals. The data is primarily concentrated in the mid-range score intervals \([0.2, 0.4)\), each contributing approximately 18\%. Additionally, a significant proportion lies within the high-quality interval \([0.9, 1.0)\), reflecting the presence of high-quality content in the dataset. In contrast, the lowest interval \([0.1, 0.2)\) contains only a minimal fraction, indicating a limited amount of poor-quality data. Note that the quantity of quality scores in the range [0, 0.1) is zero, so this interval has been omitted. This quality distribution provides a valuable reference for LLM researchers, enabling them to select data based on desired quality thresholds.
\paragraph{Human Acceptance Evaluation}
To validate the consistency between quality evaluation and human judgments, we randomly sample 100 samples from each score interval. The details and criteria for the human evaluation are provided in the Appendix~\ref{appendix:human_evaluation}. Figure~\ref{fig:human_acceptance} displays human acceptance rates across different score intervals, showing a clear positive trend: higher scores correlate with higher acceptance rates.
Specifically, the highest score interval \([0.5, 1.0)\) achieves an acceptance rate exceeding 90\%, while the lowest interval \([0.1, 0.2)\) still maintains an acceptance rate of 80\%. This trend highlights the overall high quality of the data.
In summary, the dataset is primarily concentrated in the mid-quality range, with higher scores strongly correlating to greater human acceptance. This alignment highlights the dataset's potential for high-quality applications, where consistency in human-like quality is crucial.
\subsection{Domain Distribution}
%\subsubsection{Data Evaluation}
%\paragraph{Metric Evaluation}
%In this study, we evaluated the model's performance on different label types using automatic evaluation methods. Table~\ref{tab:domain_eval_metric} shows the precision and recall for single-label and multi-label data. For single-label data, the model achieved a precision of 90.67\% and a recall of 65.03\%. This is because the reference label is set to multiple labels, and as long as a single label appears in the reference label, the prediction is considered correct. Therefore, the model can accurately identify most positive cases, but a single label is difficult to fully cover all positive cases. For multi-label data, the model's precision was 82.66\% and recall was 84.55\%, demonstrating a good balance in handling multi-label classification tasks. The model maintains high accuracy while effectively identifying the majority of positive cases. Overall, the setting of single domain labels and multi domain labels enables users to better customize their training datasets according to specific performance requirements.
%\begin{table}[htbp!]
% \caption{Domain Data Evaluation.}
% \label{tab:domain_eval_metric}
% \centering
% \begin{tabular}{lcc}
% \toprule
% Label Type & Precision(\%) & Recall(\%) \\
% \midrule
% Single-Label & 90.67 & 65.03 \\
% Multi-Label & 82.66 & 84.55 \\
% \bottomrule
% \end{tabular}
%\end{table}
%\paragraph{Manual Evaluation}
%To further validate the model's performance, we conducted a manual evaluation. Table~\ref{tab:domain_eval_manual} provides the results of the manual evaluation. For single-label data, the precision was 88.33\% and the recall was 64.15\%. For multi-label data, the precision was 74.48\% and the recall was 79.35\%. Compared with automatic evaluation, manual evaluation has slightly lower accuracy and recall rate, which may be due to the requirement for human experts to label all applicable domain labels during the manual evaluation process. Therefore, manual labeling considers more contextual information and more detailed judgment criteria. However, the results of manual evaluation still support the conclusion of automatic evaluation, with single labels achieving high accuracy and multiple labels achieving a good balance between accuracy and recall.This proves that our domain labels have achieved good alignment with human judgment.
%\begin{table}[htbp!]
% \caption{Domain Data Evaluation.}
% \label{tab:domain_eval_manual}
% \centering
% \begin{tabular}{lcc}
% \toprule
% Label Type & Precision(\%) & Recall(\%) \\
% \midrule
% Single-Label & 88.33 & 64.15 \\
% Multi-Label & 74.48 & 79.35 \\
% \bottomrule
% \end{tabular}
%\end{table}
% \paragraph{Domain Classification Case Study}
%\subsubsection{Domain Proportion}
%一是独立的领域占比，说明这个领域数据量很大，可以帮助训练该领域模型训练；二是不同质量下领域占比。
To investigate the distribution of our dataset across different domains, in this section, we conduct an detailed analysis of the data distribution across eleven distinct domains: \textit{book}, \textit{dialogue}, \textit{education}, \textit{encyclopedia}, \textit{finance}, \textit{law}, \textit{math}, \textit{medicine}, \textit{news}, \textit{technology}, and \textit{general}. This analysis considers two perspectives: the overall domain distribution and the quality-related domain distribution, providing comprehensive insights into the dataset's composition across different domains.
%This section presents an in-depth analysis of the data distribution across twelve distinct domains: \textit{book}, \textit{dialogue}, \textit{education}, \textit{encyclopedia}, \textit{finance}, \textit{law}, \textit{math}, \textit{medicine}, \textit{military}, \textit{news}, \textit{technology}, and \textit{general}. The analysis is conducted from two perspectives to provide insights into the domain distribution in different quality intervals and the domain distribution in the overall data.
\begin{figure}[htpb!]
\centering
\begin{minipage}{0.45\textwidth}
\centering
\begin{tabular}{lcc}
\toprule
Domain & Sample Count & Proportion (\%) \\
\midrule
Book & 45,192,607 & 2.97 \\
Dialogue & 312,324,923 & 20.54 \\
Education & 219,530,073 & 14.44 \\
Encyclopedia & 508,376,352 & 33.43 \\
Finance & 110,326,424 & 7.26 \\
Law & 40,477,319 & 2.66 \\
Math & 8,331,242 & 0.55 \\
Medicine & 104,809,588 & 6.89 \\
News & 425,848,698 & 28.01 \\
Technology & 295,978,428 & 19.47 \\
General & 508,392,319 & 33.44 \\
\bottomrule
\\
\\
\end{tabular}
\caption{Data Distribution Across Domains}
\label{tab:domain_distribution}
\end{minipage}
\hfill
\begin{minipage}{0.50\textwidth}
\centering
\includegraphics[width=\linewidth]{picture/Independent-Domain-Proportion-noBottom.pdf}
\caption{Data Distribution Across Domains.}
\label{fig:domain_distribution}
\end{minipage}
\vspace{-6mm}
\end{figure}
\paragraph{Overall Domain Distribution}
As shown in Figure~\ref{tab:domain_distribution}, the sample counts and corresponding proportions across various domains are presented. The Encyclopedia, General, and News domains dominate the dataset, comprising 33.43\%, 33.44\%, and 28.01\% of the total data, respectively. In contrast, the Math domain has the smallest share at 0.55\%, yet it still includes over 8 million samples. Figure~\ref{fig:domain_distribution} provides a bar chart that offers a more intuitive visualization of this distribution. This comprehensive domain analysis allows LLM researchers to select datasets that enhance the model's knowledge and capabilities in specific areas.
%As illustrated in Figure 8, the sample counts and corresponding proportions across various domains are presented. The Encyclopedia, General, and News domains dominate the dataset, comprising 33.43\%, 32.63\%, and 28.01\% of the data, respectively. In contrast, the Math domain has the smallest share at 0.55\%, yet it still includes over 8 million samples. Figure 9 complements this with a bar chart that provides a more intuitive visualization of the data distribution. This comprehensive domain distribution enables LLM researchers to select suitable datasets, facilitating the enhancement of the model’s knowledge and capabilities in specific domains.
%As shown in Figure 8, we present the sample counts and corresponding proportions across different domains. From the figure, we can see that the Encyclopedia, General, and News domains have the largest data shares, accounting for 33.43\%, 32.63\%, and 28.01\%, respectively. The Math domain, while having the smallest proportion at 0.55\%, still contains over 8 million samples. Figure 9 further presents a bar chart for a more intuitive visualization of the data distribution across domains. With this overall domain distribution, LLM researchers could select appropriate datasets, enabling them to enhance the model's knowledge and capabilities in specific domains.
%To understand the overall data distribution across each domain, we analyzed the count and proportion of data within each domain. Because the statistical analysis of all data domains is based on the multi domain labels annotated in section 3.2, a single piece of data may be counted multiple times. Figure~\ref{tab:domain_distribution} and Figure~\ref{fig:domain_distribution} illustrate the total number of data samples per domain and their corresponding percentages within the dataset. The data distribution indicates that each domain has a substantial amount of data, which can effectively support domain-specific model training. The high data volume in individual categories suggests that training models on domain-specific data may yield robust and specialized models, with each model benefiting from sufficient representation of domain-specific characteristics.
\paragraph{Quality-Related Domain Distribution}
In order to explore the domain distribution across different quality intervals, we perform an analysis focused on the quality-related domain distribution. Specifically, we calculate the proportions of various domains within each quality interval. As shown in Table~\ref{tab:quality_domain_distribution}, this table provides a detailed breakdown of domain proportions across different quality intervals. From the results, we observe that the distribution of domain data within each quality interval aligns closely with their overall distribution in the dataset. Based on the proportions in Table~\ref{tab:quality_domain_distribution}, researchers can select domain-specific data within targeted quality intervals, enabling the extraction of higher-quality domain-specific data subsets.
%Further analysis was conducted to explore the distribution of domain data across varying quality levels. Quality scores, ranging from 0 to 1, were divided into ten intervals with intervals of 0.1. For each quality interval, we calculated the number of data per domain based on multi domain labels, which allowed us to assess differences in domain distribution across quality levels. Considering that there is no data in the 0-0.1 interval, the table~\ref{tab:quality_domain_distribution} shows the proportion of data in 12 fields in the remaining 9 intervals.
%==================zwy: 这个表格里面的 Total 列是不是可以删去,而且 Yang 的表格里面没有数据量表格=====
\begin{table}[htpb!]
\centering
\caption{Domain Distribution Across Quality Levels.}
\label{tab:quality_domain_distribution}
\begin{tabular}{lcccccccccc}
% \hline
\toprule
\multirow{2.5}{*}{Domain} & \multicolumn{9}{c}{Proportion in Each Quality Interval (\%)} \\
\cmidrule{2-11}
& 0.1-0.2 & 0.2-0.3 & 0.3-0.4 & 0.4-0.5 & 0.5-0.6 & 0.6-0.7 & 0.7-0.8 & 0.8-0.9 & 0.9-1.0 & \textbf{Total} \\
\midrule
% Domain & 0.1-0.2 & 0.2-0.3 & 0.3-0.4 & 0.4-0.5 & 0.5-0.6 & 0.6-0.7 & 0.7-0.8 & 0.8-0.9 & 0.9-1.0 & \textbf{Total}\\
% \hline
Book & 2.37 & 1.98 & 2.61 & 2.72 & 2.57 & 2.83 & 3.27 & 3.83 & 5.47 & 2.97 \\
Dialogue & 7.99 & 12.57 & 17.48 & 21.57 & 24.99 & 26.32 & 27.79 & 27.94 & 21.26 & 20.54 \\
Education & 8.43 & 11.70 & 15.00 & 15.33 & 15.96 & 16.11 & 15.44 & 14.82 & 13.93 & 14.44 \\
Encyclopedia & 17.98 & 27.67 & 31.51 & 34.92 & 38.66 & 40.03 & 39.52 & 37.29 & 29.48 & 33.43 \\
Finance & 1.57 & 3.16 & 4.82 & 6.67 & 8.81 & 10.48 & 11.86 & 12.23 & 9.65 & 7.26 \\
Law & 0.34 & 0.80 & 1.43 & 1.97 & 2.63 & 3.54 & 4.71 & 5.46 & 5.40 & 2.66 \\
Math & 0.42 & 0.41 & 0.48 & 0.58 & 0.65 & 0.68 & 0.71 & 0.67 & 0.49 & 0.55 \\
Medicine & 3.21 & 6.72 & 7.71 & 7.02 & 7.25 & 7.28 & 7.00 & 6.61 & 5.41 & 6.89 \\
News & 18.78 & 19.60 & 24.33 & 28.55 & 32.62 & 34.78 & 34.72 & 33.61 & 31.19 & 28.01 \\
Technology & 16.75 & 18.52 & 18.44 & 20.42 & 22.26 & 22.42 & 21.45 & 19.66 & 15.32 & 19.47 \\
General & 53.56 & 44.24 & 35.12 & 31.18 & 27.47 & 25.62 & 25.36 & 26.69 & 32.61 & 33.44 \\
\bottomrule
% \hline
\end{tabular}
\end{table}
%This distribution analysis provides critical insights into the data's utility for fine-tuning models with different quality constraints. Specifically, the high-quality data in certain domains can be prioritized for tasks requiring higher accuracy, while the more evenly distributed data can be used for robust model generalization. The variability in data quality across domains further underscores the importance of targeted data selection based on the intended application of the model, enabling users to better tailor their training datasets to specific performance requirements.
\subsection{Data Toxicity Analysis}
%During the training procedure of LLMs, longer texts can provide more abundant knowledge and information, making it easier for the models to understand complex relationships in the text, and learn more knowledge. In this section, we will analyze the length distribution of the texts in our cleaner dataset. As shown in Figure~\ref{fig4length}, it illustrates the distribution of text lengths within our cleaner Chinese subset. From this figure, we can observe that the majority of text lengths are mainly distributed within 1000 characters or less. Among them, the most significant proportion is observed within the length interval of 300 to 500 characters. Texts exceeding 1000 characters account for a relatively small portion, and there is a long tail of very long texts. After analysis, we found that the maximum text length in this dataset can reach 300,000 characters. However, they are considered to be outliers and excluded from this figure. The text length distribution in this figure could provide valuable insights into the structure and characteristics of our cleaner subset, thereby help researchers in understanding the composition of the processed dataset and facilitating the utilization of the dataset.
During the training procedure of LLMs, toxic data introduces harmful knowledge, which can lead the model to generate toxic outputs. In this section, we analyze the toxicity distribution within our dataset. As shown in Figure~\ref{fig:Toxicity_distribution}, the figure illustrates the toxicity distribution of the dataset, where a higher toxicity score indicates greater toxicity. It is evident that the majority of the data in our dataset has a toxicity score of 0.0, signifying non-toxic, high-quality data. These non-toxic texts make up 97.41\% of the dataset.
\begin{figure}[htp]
\centering
\includegraphics[width=15cm]{picture/toxicity_distribution.pdf}
\caption{The Distribution of Toxicity: A threshold of 0.99 was established, and samples with scores exceeding 0.99 were categorised as toxic.}
\label{fig:Toxicity_distribution}
\end{figure}
%During the training procedure of LLMs, toxic data introduces harmful knowledge and information, which may lead the model to generate toxic outputs. In this section, we analyze the toxicity distribution within our dataset. As shown in Figure~\ref{fig:Toxicity_distribution}, it depicts the toxicity distribution of the dataset. In this figure, a higher toxicity score indicates greater toxicity. It is evident that the majority of the data in our dataset has a toxicity score of 0.0, signifying non-toxic, high-quality data. These non-toxic texts comprise 97.41\% of the dataset.
Additionally, through manual analysis of the toxicity scores, we identify that data with scores above 0.99 are classified as toxic. By applying this empirical threshold, we filter our dataset to obtain a 3.16GB subset of toxic texts, comprising 1,632,620 samples. In Table~\ref{tab:comparison_toxic}, we compare this subset with other publicly available toxic datasets. The table includes OffensEval 2019~\cite{offenseval}, AbusEval~\cite{abuseval}, HatEval~\cite{hateval}, RAL-E~\cite{hatebert}, and ToxiGen~\cite{hartvigsen2022toxigen}, which are English toxicity datasets, and COLD~\cite{deng2022cold}, ToxiCN~\cite{toxicn}, SWSR~\cite{jiang2022swsr}, and CDial-Bias~\cite{zhou2022towards}, which are Chinese toxicity datasets. The OffensEval 2019, AbusEval, and HatEval datasets are derived from Twitter and focus on offensive language, abusive language, and hate speech, respectively. The RAL-E dataset, sourced from a banned Reddit community, is a large-scale, unannotated English dataset. In contrast, ToxiGen is a toxicity dataset generated using GPT-3, targeting multiple groups. The COLD, SWSR, CDial-Bias, and ToxiCN datasets are collected from Chinese social media platforms, including Zhihu, Weibo, and Tieba, each focusing on different groups. Compared to these datasets, ours features the largest collection of toxicity data, with each text annotated with a toxicity score, providing researchers with a valuable resource to optimize and evaluate the safety of LLMs.
%Additionally, through manual analysis of the toxicity scores, we identify that data with scores above 0.99 are classified as toxic. By applying this empirical threshold, we filter our dataset and obtain a 3.16GB toxic text subset comprising 1,632,620 samples. In Table~\ref{tab:comparison_toxic}, we conduct a comparison between this subset with other publicly available toxic datasets. In this table, OffensEval 2019~\cite{offenseval}, AbusEval~\cite{abuseval}, HatEval~\cite{hateval}, RAL-E~\cite{hatebert} and ToxiGen~\cite{hartvigsen2022toxigen} are English toxicity datasets, while COLD~\cite{deng2022cold}, ToxiCN~\cite{toxicn}, SWSR~\cite{jiang2022swsr} and CDial-Bias~\cite{zhou2022towards} are Chinese toxicity datasets. The OffensEval 2019, AbusEval, and HatEval datasets are derived from Twitter and focus on the analysis of offensive language, abusive language, and hate speech, respectively. The RAL-E dataset, sourced from a banned Reddit community, is a large-scale, unannotated English dataset. In contrast, ToxiGen is a toxicity dataset generated using GPT-3, targeting multiple groups. The COLD, SWSR, CDial-Bias, and ToxiCN datasets are collected from Chinese social media platforms including Zhihu, Weibo, and Tieba, with each dataset focusing on different groups. Compared to these datasets, ours features the largest collection of toxicity data and each text contains a toxicity score, providing researchers with a valuable resource to better optimize and evaluate LLMs' safety.
\begin{table}[htbp!]
\begin{center} % 使用 center 环境
\begin{threeparttable}
\caption{Comparison of Different Toxicity Datasets.}
\label{tab:comparison_toxic}
\begin{tabular}{*{8}{c}}
\toprule
Dataset & Lang. & Label & Toxicity Score & Data Size & Toxic & Benign \\
\midrule
OffensEval 2019~\cite{offenseval} & EN & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 14,100 & 4,640 & 9,460 & \\
AbusEval~\cite{abuseval} & EN & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 14,100 & 2,927 & 11,173 & \\
HatEval~\cite{hateval} & EN & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 13,000 & 5,470 & 7,530 & \\
RAL-E~\cite{hatebert} & EN & \textcolor{red}{\usym{1F5F4}} & \textcolor{red}{\usym{1F5F4}} & 1,492,740 & - & - & \\
ToxiGen~\cite{hartvigsen2022toxigen} & EN & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 274,186 & 137,395 & 136,791 & \\
COLD~\cite{deng2022cold} & ZH & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 37,480 & 18,041 & 19,439 & \\
ToxiCN~\cite{toxicn} & ZH & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 12,011 & 6,461 & 5,550 & \\
SWSR~\cite{jiang2022swsr} & ZH & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 8,969 & 3,093 & 5,876 & \\
CDial-Bias~\cite{zhou2022towards} & ZH & \textcolor{green}{\usym{2714}} & \textcolor{red}{\usym{1F5F4}} & 28,343 & 7,233 & 21,110 & \\
\textbf{ChineseWebText2.0} & ZH & \textcolor{green}{\usym{2714}} & \textcolor{green}{\usym{2714}} & 1,803,836,772 & 1,632,620 & 1,802,204,152 & \\
\bottomrule
\end{tabular}
\end{threeparttable}
\end{center}
\end{table}
%Additionally, through manual analysis of the toxicity scores, we identified that data with scores exceeding 0.99 are classified as toxic. By applying this empirical threshold, we filtered our dataset, resulting in a subset of toxic text data amounting to 3.16GB. Table 4 compares this subset with other publicly available toxic datasets.
%Figure~\ref{fig:Toxicity_distribution} illustrates the distribution of toxicity scores within the dataset. The results demonstrate that the majority of the data are of high quality and non-toxic, with $97.41\%$ of toxicity scores clustered at 0.0.
%To guarantee the quality of the dataset, 300 toxic samples and 300 benign samples were selected independently based on the scores from the Toxicity Classifier R2. Following human expert annotation of these samples, an empirical threshold of 0.99 was established for the toxicity score by comparing the model's predictions with human annotations. Using this threshold, the model achieved an annotation precision of $83.33\%$ for toxic samples and a specificity of $97.67\%$ for benign samples. Increasing the threshold may improve the accuracy of toxicity predictions. However, this adjustment may also reduce the amount of data available.
%By setting a toxicity threshold of 0.99, we have obtained an internet-based Chinese toxic language dataset comprising 1,632,620 samples. To the best of our knowledge, this is the largest dataset containing toxicity labels and scores currently available. Despite the pervasive implementation of large language models, not all users operate with benevolent intentions. The generation of toxic content represents a substantial threat to the security and integrity of large language models. Consequently, the prevention of such content, coupled with the establishment of robust toxicity analysis and detection mechanisms, is of paramount importance for the mitigation of security risks within the field of artificial intelligence. Our dataset is applicable for training, evaluation, and detoxification efforts related to toxicity detection models, thus advancing research in Chinese toxicity detection.
\section{Conclusions and Future Work}
\label{others}
%By extracting large-scale, high-quality, multi-domain Chinese pretraining data, we propose a novel approach that first extracts Chinese text data and applies manual rules to filter out toxic content, advertisements, and other noise, which we refer to as coarse-grained filtering. The resulting dataset then undergoes quality evaluation, multi-domain classification, and toxicity assessment modules. The quality evaluation is based on a BERT-based scoring model that assigns a quality score to each text. Multi-domain classification is performed using a combination of rule-based methods and a FastText model. The toxicity assessment module initializes the FastText model using a set of toxicity data, and then iteratively refines the toxicity score in conjunction with the QWEN model. Using the aforementioned models, we have released a new and largest Chinese dataset of 3.8TB, where each dataset is accompanied by corresponding quality scores, domain classifications, and toxicity ratings, allowing researchers to filter data according to their desired quality thresholds.
In order to extract large-scale, high-quality Chinese pre-training data with multi-dimensional and fine-grained information, we have developed a novel pipeline that processes raw collected data using both handcrafted rules and advanced evaluation models. The handcrafted rules are applied to clean the raw texts, producing a high-quality dataset. Subsequently, we design three powerful models—a quality evaluation, a domain classifier, and a toxicity evaluation model—to assign each text with four types of fine-grained information. With this approach, we release the latest and largest Chinese dataset of 3.8 TB, each of which is annotated with a quality score, domain labels, a toxicity score and a toxicity label. This dataset enables LLM researchers to re-filter the data according to their desired thresholds. Additionally, We also release the complete tool-chain that transforms the raw data into a high-quality dataset with multi-dimensional and fine-grained annotations.
In the future, we will collect a wider variety of domain-specific texts to enable the dataset to cover a broader range of scenarios. Additionally, we are going to enrich the dataset by incorporating more fine-grained information into the texts. For example, we could construct a evaluation model to assess the knowledge density within each text.
\bibliographystyle{unsrt}
\bibliography{references}
\section*{Appendix}
\subsection{Human Evaluation in Quality Assessment}
\label{appendix:human_evaluation}
Following the methodology used in ChineseWebText 1.0, we employ human evaluators to assess the quality of the dataset. In this process, we randomly sample 1,000 examples, and five independent human evaluators are hired to assess the quality of each sample. The evaluation is based on four key criteria. Each text is assigned a label of either "True" or "False" during the evaluation. "True" indicates that the data meets the quality requirements for pre-training across all four criteria, while "False" indicates that the text contains some level of noise. After the evaluations are completed, we calculate the average accuracy across the five evaluators.
\begin{tcolorbox}[colback=lightblue!50!white,colframe=lightblue,title=\textcolor{black}{The Criteria for Human Evaluation in Quality Assessment.}, width=\textwidth]
\begin{itemize}
\item Informativeness: Whether the text contains enough knowledge and information, or is just meaningless crap.
\item Fluency: Whether the text has formatting issues, capitalization mistakes, or evident grammatical errors that impair readability.
\item Coherence: Whether the text progressively forms a coherent body of information on a topic through its successive sentences.
\item Toxicity: Texts used for pre-training should endeavor to exclude offensive remarks, sexually explicit content, and politically sensitive statements to mitigate potential generative risks.
\end{itemize}
\end{tcolorbox}
\end{document}
\begin{thebibliography}{10}
\bibitem{chinesewebtext}
Jianghao Chen, Pu~Jian, Tengxiao Xi, Dongyi Yi, Qianlong Du, Chenglin Ding, Guibo Zhu, Chengqing Zong, Jinqiao Wang, and Jiajun Zhang.
\newblock Chinesewebtext: Large-scale high-quality chinese web text extracted with effective evaluation model, 2023.
\bibitem{2020T5C4}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter~J. Liu.
\newblock Exploring the limits of transfer learning with a unified text-to-text transformer.
\newblock {\em Journal of Machine Learning Research}, 21(140):1--67, 2020.
\bibitem{2020_pile}
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.
\newblock The {P}ile: An 800gb dataset of diverse text for language modeling.
\newblock {\em arXiv preprint arXiv:2101.00027}, 2020.
\bibitem{wanjuan}
Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin.
\newblock Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023.
\bibitem{mapcc}
Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Wenhu Chen, and Ge~Zhang.
\newblock Chinese tiny llm: Pretraining a chinese-centric large language model, 2024.
\bibitem{scao2022bloom}
Teven~Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili{\'c}, Daniel Hesslow, Roman Castagn{\'e}, Alexandra~Sasha Luccioni, Fran{\c{c}}ois Yvon, Matthias Gall{\'e}, et~al.
\newblock Bloom: A 176b-parameter open-access multilingual language model.
\newblock {\em arXiv preprint arXiv:2211.05100}, 2022.
\bibitem{llama}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
\newblock Llama: Open and efficient foundation language models, 2023.
\bibitem{2023refinedweb}
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.
\newblock The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023.
\bibitem{t5}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter~J. Liu.
\newblock Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
\bibitem{chowdhery2022palm}
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung~Won Chung, Charles Sutton, Sebastian Gehrmann, et~al.
\newblock Palm: Scaling language modeling with pathways.
\newblock {\em arXiv preprint arXiv:2204.02311}, 2022.
\bibitem{bai2023qwen}
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu~Han, Fei Huang, et~al.
\newblock Qwen technical report.
\newblock {\em arXiv preprint arXiv:2309.16609}, 2023.
\bibitem{GPT4}
GPT-4~Technical Report.
\newblock Corr, abs/2303.08774, 2023.
\bibitem{o1}
OpenAI.
\newblock Learning to reason with llms.
\newblock \url{https://openai.com/index/learning-to-reason-with-llms/}, 2024.
\bibitem{wudao}
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, and Jie Tang.
\newblock Wudaocorpora: A super large-scale chinese corpora for pre-training language models.
\newblock {\em AI Open}, 2021.
\bibitem{yuan1}
Shaohua Wu, Xudong Zhao, Tong Yu, Rongguo Zhang, Chong Shen, Hongli Liu, Feng Li, Hong Zhu, Jiangang Luo, Liang Xu, and Xuanwei Zhang.
\newblock Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning, 2021.
\bibitem{skypile}
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo~Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou.
\newblock Skywork: A more open bilingual foundation model, 2023.
\bibitem{luccioni2021whatsboxpreliminaryanalysis}
Alexandra~Sasha Luccioni and Joseph~D. Viviano.
\newblock What's in the box? a preliminary analysis of undesirable content in the common crawl corpus, 2021.
\bibitem{lee2022deduplicatingtrainingdatamakes}
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.
\newblock Deduplicating training data makes language models better, 2022.
\bibitem{brown2020language}
Tom~B Brown.
\newblock Language models are few-shot learners.
\newblock {\em arXiv preprint arXiv:2005.14165}, 2020.
\bibitem{bert}
M.~V. Koroteev.
\newblock Bert: A review of applications in natural language processing and understanding, 2021.
\bibitem{fasttext}
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.
\newblock Bag of tricks for efficient text classification, 2016.
\bibitem{cbow}
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.
\newblock Efficient estimation of word representations in vector space, 2013.
\bibitem{pradhan2020review}
Rahul Pradhan, Ankur Chaturvedi, Aprna Tripathi, and Dilip~Kumar Sharma.
\newblock A review on offensive language detection.
\newblock {\em Advances in Data and Information Sciences: Proceedings of ICDIS 2019}, pages 433--439, 2020.
\bibitem{xiao2024chinese}
Yunze Xiao, Houda Bouamor, and Wajdi Zaghouani.
\newblock Chinese offensive language detection: Current status and future directions.
\newblock {\em arXiv preprint arXiv:2403.18314}, 2024.
\bibitem{hatebert}
Tommaso Caselli, Valerio Basile, Jelena Mitrovi{\'c}, and Michael Granitzer.
\newblock Hatebert: Retraining bert for abusive language detection in english.
\newblock {\em arXiv preprint arXiv:2010.12472}, 2020.
\bibitem{hartvigsen2022toxigen}
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar.
\newblock Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.
\newblock {\em arXiv preprint arXiv:2203.09509}, 2022.
\bibitem{deng2022cold}
Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang.
\newblock Cold: A benchmark for chinese offensive language detection.
\newblock {\em arXiv preprint arXiv:2201.06025}, 2022.
\bibitem{industrycorpus}
Baai. 2024. industrycorpus.
\newblock {\em https://huggingface.co/datasets/BAAI/IndustryCorpus/tree/main}.
\bibitem{redpajama}
Together computer. 2023c. redpajama-incite-base-3bv1.
\bibitem{wang2022use}
Yongjie Wang, Chuan Wang, Ruobing Li, and Hui Lin.
\newblock On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation.
\newblock {\em arXiv preprint arXiv:2205.03835}, 2022.
\bibitem{devlin2018bert}
Jacob Devlin.
\newblock Bert: Pre-training of deep bidirectional transformers for language understanding.
\newblock {\em arXiv preprint arXiv:1810.04805}, 2018.
\bibitem{mesgar2018neural}
Mohsen Mesgar and Michael Strube.
\newblock A neural local coherence model for text quality assessment.
\newblock In {\em Proceedings of the 2018 conference on empirical methods in natural language processing}, pages 4328--4339, 2018.
\bibitem{liu2021temp}
Zichen Liu, Hongyuan Xu, Yanlong Wen, Ning Jiang, Haiying Wu, and Xiaojie Yuan.
\newblock Temp: taxonomy expansion with dynamic margin loss through taxonomy-paths.
\newblock In {\em Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pages 3854--3863, 2021.
\bibitem{scudder1965probability}
Henry Scudder.
\newblock Probability of error of some adaptive pattern-recognition machines.
\newblock {\em IEEE Transactions on Information Theory}, 11(3):363--371, 1965.
\bibitem{mukherjee2020uncertainty}
Subhabrata Mukherjee and Ahmed Awadallah.
\newblock Uncertainty-aware self-training for few-shot text classification.
\newblock {\em Advances in Neural Information Processing Systems}, 33:21199--21212, 2020.
\bibitem{toxicn}
Junyu Lu, Bo~Xu, Xiaokun Zhang, Changrong Min, Liang Yang, and Hongfei Lin.
\newblock Facilitating fine-grained detection of chinese toxic language: Hierarchical taxonomy, resources, and benchmarks.
\newblock {\em arXiv preprint arXiv:2305.04446}, 2023.
\bibitem{jiang2022swsr}
Aiqi Jiang, Xiaohan Yang, Yang Liu, and Arkaitz Zubiaga.
\newblock Swsr: A chinese dataset and lexicon for online sexism detection.
\newblock {\em Online Social Networks and Media}, 27:100182, 2022.
\bibitem{zhou2022towards}
Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng.
\newblock Towards identifying social bias in dialog systems: Framework, dataset, and benchmark.
\newblock In {\em Findings of the Association for Computational Linguistics: EMNLP 2022}, pages 3576--3591, 2022.
\bibitem{qwen2.5}
{Qwen Team}.
\newblock Qwen2.5: A party of foundation models, September 2024.
\bibitem{roots}
Hugo Lauren{\c{c}}on, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova~del Moral, Teven Le~Scao, Leandro Von~Werra, Chenghao Mou, Eduardo Gonz{\'a}lez~Ponferrada, Huu Nguyen, et~al.
\newblock The bigscience roots corpus: A 1.6 tb composite multilingual dataset.
\newblock {\em Advances in Neural Information Processing Systems}, 35:31809--31826, 2022.
\bibitem{offenseval}
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.
\newblock Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval).
\newblock {\em arXiv preprint arXiv:1903.08983}, 2019.
\bibitem{abuseval}
Tommaso Caselli, Valerio Basile, Jelena Mitrovi{\'c}, Inga Kartoziya, and Michael Granitzer.
\newblock I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language.
\newblock In {\em Proceedings of the twelfth language resources and evaluation conference}, pages 6193--6202, 2020.
\bibitem{hateval}
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel~Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti.
\newblock Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.
\newblock In {\em Proceedings of the 13th international workshop on semantic evaluation}, pages 54--63, 2019.
\end{thebibliography}