Hillary Clinton takes a selfie during a campaign rally. At some points during her campaign, Clinton used language specifically popular among Russian government trolls on Twitter. Photo: Nathania Johnson/Wikimedia Commons

The Russian government’s campaign to influence the 2016 presidential election in favor of Donald Trump is well documented. There is little doubt that thousands of hackers, trolls, and foreign agents worked at the direction of Vladimir Putin’s government. Their objective was clear: to destabilize American online discourse.

Over the past several months, we conducted an inquiry into the language patterns of Russian trolls. Using official data released by Twitter, we built an interactive machine learning model capable of determining whether any text uses language specifically popular among Russian trolls on Twitter—or if it more closely resembles organic content.

Now, we are applying our model and other text analysis techniques to better understand the way that Russian trolls behaved online, especially with respect to major public figures. To that end, we will look at how the language used on Twitter by Donald Trump and Hillary Clinton—the major candidates in the 2016 election—compares to that of Russian trolls.

Sentiment Analysis—

Russian trolls are often associated with highly divisive, negative, and angering content. According to Wikipedia, a troll is “a person who starts quarrels or upsets people on the Internet to distract and sow discord by posting inflammatory and digressive [...] messages in an online community.” From this definition, one may naturally assume that the messaging of Russian government trolls is uniformly negative.

Unfortunately, this assumption turns out to be far from the truth. Using the state-of-practice sentiment analysis algorithm VADER, we calculated the sentiment of approximately four million English troll Tweets. For reference, we then also calculated the sentiment of just as many random ‘organic’ English Tweets.

The average sentiment for both the troll and organic Tweets was approximately zero. VADER sentiment scores are distributed from -1 to +1, where -1 is the most negative, +1 is the most positive, and 0 is neutral. For the average sentiment to be zero, both troll and organic Tweets must either be themselves largely neutral, or contain a relatively equal number of positive and negative tweets.

For a more complete explanation of VADER sentiment analysis, please see this detailed primer.

The answer, it turns out, is both. A significant portion of both troll and organic Tweets were near zero. Still, the sentiment distribution was largely bimodal: there are approximately as many negative Tweets in each group as positive Tweets, challenging the conventional wisdom that troll content is uniformly negative—instead, it seems to closely resemble the sentiment distribution of organic content, with approximately equal portions of negative and positive tweets, and a substantial number of neutral Tweets.

While Figure 1 clearly conveys the sheer volume of neutral tweets and the trimodal nature of the sentiment distribution, its scale muffles the contours of the non-neutral Tweets. Figure 2 shows the same distribution as Figure 1, except with neutral Tweets omitted.

As Figure 2 demonstrates, neither troll nor organic Tweets are uniformly positive or negative. Still, patterns do emerge. For example, organic Tweets tend to be slightly more positive than troll Tweets, and troll Tweets tend to be slightly more negative than organic Tweets. Troll messaging, then, is no monolith: it not only pushes both conservative and liberal content, but also uses both negative and positive language.

Clinton and Trump—

Now that we have a more nuanced understanding of the language used by Russian trolls, we can compare Donald Trump and Hillary Clinton’s rhetoric on Twitter. Importantly, we don’t claim that either Trump or Clinton engaged in activity that is itself ‘troll-like.’ Instead, we only compare the language used on their Twitter accounts—written either by them or aides who post on their behalf—to the language used by Russian trolls on Twitter. Moreover, the machine learning model we use to classify Trump and Clinton’s Tweets is primarily trained upon Tweets from 2018—therefore, it’s possible the model cannot accurately assess Tweets before then.

For more information about the machine learning model used to perform this analysis, please see our extended explanation.

To perform this analysis, we used an open-source project to collect every Tweet made by each figure that was not a ‘reply’ between June 2013 and October 2018. (Our analysis begins in June 2013, when Hillary Clinton sent her first tweet.) In total, our analysis includes 34,438 tweets (8,072 from Hillary Clinton, and 26,366 from Donald Trump).

Interestingly, we find that from June 2013 to November 2015, Hillary Clinton’s Twitter account used more language specifically popular among Russian trolls than Donald Trump’s. This trend quickly reversed in 2016, when the language of Donald Trump’s Twitter account more closely resembled that of Russian trolls than that of organic content.

What’s next—

Ultimately, only time will tell whether Russia’s online disinformation campaign will cease. For now, it rages on. Nonetheless, we hope that our analysis—along with that of other researchers—will provide a better understanding of the Russian campaign. Sunlight, after all, is the best disinfectant.