Predictive model detects potential extremist propaganda on social media

UNIVERSITY PARK, Pa. — The militant Islamic State group, or ISIS, lost its physical territory in 2019, but it remains an active force on social media, according to researchers from the Penn State College of Information Sciences and Technology, who set out to better understand the group’s online strategies. 

In their work, which appeared in the journal Social Media Analysis and Mining, the researchers analyzed a large dataset of activity on X — formerly known as Twitter — to develop a predictive model to detect users and content related to Islamic State extremists. They identified potential propaganda messages and their characteristics and developed an image classifier to find the most frequent categories of images attached to tweets about ISIS. They further collected a dataset of tweets from potential ISIS supporters to investigate their recent activities.   

“The Islamic State group and its affiliates, sympathizers and followers continue to manipulate online communities to spread extremist propaganda,” said Younes Karimi, a graduate student pursuing a doctorate in informatics and the first author of the paper. “By studying their behavioral patterns and strategies and monitoring their online presence, we can help social media companies identify and eventually restrict such accounts in a timelier manner and abate their impact on online communities.”  

According to Karimi, the Islamic State group is increasingly relying on social media to spread propaganda, undermine its rivals and recruit sympathizers, despite countermeasures by websites like X to restrict its online activities. ISIS watch — an online channel that publishes daily updates on terrorist content banned on the cross-platform instant messaging service Telegram —reported the removal of nearly 5,000 terrorist bots and channels in the first 11 days of 2024. 

The researchers’ dataset included millions of tweets, spanning from 2009 to 2021, that were linked to the Islamic State group and its propaganda.  

“The longitudinal perspective of the dataset is important because it includes data from before and after 2015, when a major crackdown by Twitter removed user accounts and content involving the Islamic State group,” Karimi said. “In response, the extremists had to change their online strategy and move to other platforms, and little is known about their online whereabouts since that crackdown.” 

To identify potential ISIS supporters, the researchers began by building a user classifier using the old dataset. ISIS accounts identified before 2015 served as the labeled data for the study’s ISIS users. The researchers used machine learning and natural language processing techniques to differentiate the types of users sharing the extremist group’s content. 

“The users in our dataset ranged from known members of the Islamic State group to retweeters and quoters to mentioners of ISIS,” Karimi said. “We believe that users who retweet or quote Islamic State group content are more likely to be affiliates or sympathizers, while those who just mention the content are less likely to be supporters. However, tweets posted by mentioners are still very likely related to ISIS and contain topics similar to ISIS tweets, which make mentioners suitable to be considered as our non-ISIS users and non-trivial counterparts to ISIS users.”  

The researchers then analyzed the tweets to identify what they referred to as “candidate propaganda.” They compared topics used by known Islamic State group accounts prior to 2015 in the old dataset to the content posted after 2015 by potential affiliates and supporters in their recent dataset.  

They examined these tweets from three angles. The first, abnormal engagement, identified content that is pervasive and continuous in the way it’s shared. 

“We formulated and employed a method to automatically detect potential propaganda messages that are broadly disseminated at a large scale,” Karimi said. “Our method identified users in the dataset who had few followers but whose content was widely spread via retweets and likes.” 

The second angle examined ideology-based words and images, which the researchers said are often designed to elicit an emotional response and influence a large audience.  

The researcher’s third angle for examining content involved hashtags.  

“Supporters and affiliates of the Islamic State group recruited people to retweet hashtags to create trending ideas, such as strong religious references, and curate group messaging to improve the group’s branding and ensure message longevity,” Karimi said. 

Among the most-used hashtags in tweets from ISIS were “The Islamic State,” “Caliphate News,” “Urgent,” “The State of the Caliphate” and “ISIS.”  

Because this approach focuses on users and user content, the researchers said it could be applied to other social media platforms beyond X. 

“Our study can help social media safety teams track potential extremist accounts, identify their supporters and amplifiers and prevent the spread of the propaganda they need to grow their community,” Karimi said. “Doing so in a timely manner may assist law enforcement and government agencies in their intervention efforts against extremism.” 

This work was funded by a three-year, $250,000 grant from the U.S. National Science Foundation and was initially led by Anna Squicciarini, Frymoyer Chair in Information Sciences and Technology in the college, and Peter Forster, professor emeritus in the college and affiliate faculty in the Penn State School of International Affairs. Karimi joined the research team in March 2021.