About Project
About the Dashboard
The CSIS Hate Speech Dashboard tracks online hate speech trends in Indonesia. It uses a bespoke machine learning algorithm to collect Indonesian tweets on Twitter and identify whether they contain hate speech targeting one of five vulnerable minorities: Ahmadiyyas, Shi’as, Chinese Indonesian, Christians, and ethnic Papuans.
The dashboard is run by a team of researchers in the Centre for Strategic and International Studies (CSIS) Jakarta with support from the Asia Pacific Partnership for Atrocity Prevention (APPAP) and the Asia-Pacific Centre for the Responsibility to Protect (APR2P) Centre in the University of Queensland.
About APPAP
The Asia Pacific Partnership for Atrocity Prevention (APPAP) is an alliance of organizations in the Asia Pacific region that works towards the prevention of atrocity crimes and fostering commitment to the Responsibility to Protect (R2P) principle.
The alliance is composed of civil society, not-for-profit organizations, think tanks, and University schools and centers. The alliance meets regularly and works to prevent atrocity through groups that include the Gender and Atrocities Prevention Group and the Working Group on the Prevention of Hate Speech and Incitement. The APR2P Centre is the current secretariat of APPAP, and the alliance funds grassroots projects and education activities across Asia and the Pacific, including youth projects and the development of software applications.
Suggested Citation
Lina Alexandra, Alif Satria, Edbert Gani Suryahudaya, and Beltsazar Krisetya, “The National Hate Speech Dashboard,” CSIS Indonesia, (2021). https://hatespeech.csis.or.id
F.A.Q
How do we define hate speech?
The CSIS Hate Speech Dashboard defines hate speech as “any tweet that uses phrases which legitimize hostile actions or ascribe negative qualities towards the identity of a vulnerable community.” For a tweet to be considered hate speech it must:
- Target an individual/group’s identity,
- Contain insulting phrases that ascribe negative qualities to the identity, or
- Contain inciting phrases that call for hostility against individuals/groups of that identity.
How do we collect our data?
The CSIS Hate Speech Dashboard collects data on Twitter using the publicly available SNScraper
How do we identify hate speech?
The CSIS Hate Speech Dashboard identifies which tweets are hate speech by using an independently developed hate speech identification algorithm. This algorithm is developed in a four-step process:
- First, the team created a dataset of tweets targeting vulnerable minorities in Indonesia using open-source data-scrapping tools. A tweet is only collected if they contain more than four words, had phrases (of various spelling) that refer to aforementioned vulnerable minorities, and used the Indonesian language (regardless of their geographic location). A total of 12,000 tweets were collected.
- Second, the team trained six research assistants (RAs) to annotate whether each tweet in the dataset contained hate speech as defined above. The RAs were instructed to determine this based on a tweet’s overall meaning, not an exhaustive list of hateful words or phrases. This is done because most hate speech is context-specific: a word (e.g., “communist”) might be benign independently, but harmful if used in a sentence (e.g., “let’s bring back 1965 to those communists”). The annotations are then verified by four researchers to ensure they are accurate and standardized.
- Third, the team used a portion of this annotated dataset to develop an initial hate speech identification algorithm. This process is done using a Bag-of-Words method. In this method, an algorithm is tasked to sift through all of the tweets in the annotated dataset and identify what common multiset of words within a tweet makes it more likely to be labeled as hate speech. This method was chosen as it better enables the algorithm to classify hate speech based on a tweet’s “context” –– “communist” alone might not be hate speech, but “communist” with “1965” and “back” might be.
- Fourth, this initial hate speech identification algorithm is then tested to identify hate speech in a second, separate portion of the annotated dataset. Doing so allows the team to assess and measure the algorithm’s accuracy (i.e., whether the tweet that the algorithm identified as hate speech was hate speech and vice versa) and adjust the final model fit. Through this process, the team was able to develop a final algorithm that can identify tweets containing hate speech with 84.7% accuracy.
How do we define hate speech?
The CSIS Hate Speech Dashboard defines hate speech as “any tweet that uses phrases which legitimize hostile actions or ascribe negative qualities towards the identity of a vulnerable community.” For a tweet to be considered hate speech it must:
- Target an individual/group’s identity,
- Contain insulting phrases that ascribe negative qualities to the identity, or
- Contain inciting phrases that call for hostility against individuals/groups of that identity.
How do we collect our data?
The CSIS Hate Speech Dashboard collects data on Twitter using the publicly available SNScraper
How do we identify hate speech?
The CSIS Hate Speech Dashboard identifies which tweets are hate speech by using an independently developed hate speech identification algorithm. This algorithm is developed in a four-step process:
- First, the team created a dataset of tweets targeting vulnerable minorities in Indonesia using open-source data-scrapping tools. A tweet is only collected if they contain more than four words, had phrases (of various spelling) that refer to aforementioned vulnerable minorities, and used the Indonesian language (regardless of their geographic location). A total of 12,000 tweets were collected.
- Second, the team trained six research assistants (RAs) to annotate whether each tweet in the dataset contained hate speech as defined above. The RAs were instructed to determine this based on a tweet’s overall meaning, not an exhaustive list of hateful words or phrases. This is done because most hate speech is context-specific: a word (e.g., “communist”) might be benign independently, but harmful if used in a sentence (e.g., “let’s bring back 1965 to those communists”). The annotations are then verified by four researchers to ensure they are accurate and standardized.
- Third, the team used a portion of this annotated dataset to develop an initial hate speech identification algorithm. This process is done using a Bag-of-Words method. In this method, an algorithm is tasked to sift through all of the tweets in the annotated dataset and identify what common multiset of words within a tweet makes it more likely to be labeled as hate speech. This method was chosen as it better enables the algorithm to classify hate speech based on a tweet’s “context” –– “communist” alone might not be hate speech, but “communist” with “1965” and “back” might be.
- Fourth, this initial hate speech identification algorithm is then tested to identify hate speech in a second, separate portion of the annotated dataset. Doing so allows the team to assess and measure the algorithm’s accuracy (i.e., whether the tweet that the algorithm identified as hate speech was hate speech and vice versa) and adjust the final model fit. Through this process, the team was able to develop a final algorithm that can identify tweets containing hate speech with 84.7% accuracy.