Big Data and Elections: We shine a light on Trump and Clinton

Twitter is widely used as a tool to understand and predict phenomena in the real world. Today on our blog, we have been using Twitter to understand the US Presidential Elections of November 8th 2016. There are no conclusive research results on whether it is possible to predict the outcomes of elections using tweets but we decided to investigate.

Figure 1: Shine a light on Trump or Clinton

In 2010, an article was published as part of the Fourth International AAAI Conference on Weblogs and Social Media entitled “Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment- concludes that it indeed is possible“. This research used tweets from the German Federal elections in Sept 2009 and concluded that the mere number of messages mentioning a party can reflect the election result. Furthermore, joint mentions of two parties are in line with real world political ties and coalitions. The research also went on to show that an analysis of the tweets’ political sentiment demonstrates close correspondence to the parties’ and politicians’ political positions indicating that the content of Twitter messages “plausibly reflects the offline political landscape.”

However, another article published one year later in the same Artificial Intelligence conference -“Limits of Electoral Predictions Using Twitter” stated that “unfortunately, they found no correlation between the analysis results and the electoral outcomes, contradicting previous reports,”, basing their investigation on tweets of the US 2010 Congressional elections.

A search in Google will show much more research on the feasibility of using Twitter and other social media networks for election predictions.  Additionally, a Google Trends query to check which candidate generates more search activity provides some insight on the matter:

Figure 2: Google Trends candidates’ search activity

Whatever the conclusion is, there is no doubt that Twitter reflects to some extent what is going on in a country just before important elections. During the two days running up to the elections on the 8th of November 2016, we established a real-time feed from Twitter filtering relevant hashtags, handles names and keywords. As technology, we have used Sinfonier, an ElevenPaths cybersecurity product to detect cyberthreats based on real-time information processing. In this case, Sinfonier automates the capture of tweets in real-time (see Figure 3). Sinfonier encapsulates real-time capabilities of Apache Storm in a very elegant way, so we can build a “digest Twitter-to-MongoDB” within a few minutes and with almost zero code writing.

Figure 3:  Sinfonier Topology (data extraction pipeline) designed to capture real-time data from Twitter.

We then also used real-time capabilities using Elastic Search and Kibana. Apart from visualizing the tweets in real-time in a dashboard (see Figure 4), we wanted to try out something more fun so we also got some Philips Hue lamps involved.

Figure 4: Real-time tweets on Clinton (blue) vs. Trump (red). Trump tweets almost double those of Clinton. Trump is slightly more active in replying and both re-tweet equally all tweets they are mentioned in.

Actually, in the Big Data era, where taking data-driven decisions is the ultimate goal, in a variety of situations it will be unfeasible to use traditional dashboards to convey the status of real-time KPIs. Imagine the case of factories, call center open floors and large retail centers. It could be very cool to have “realtime transparent dashboards”, so that lights (visual perception) adapt to the (big) data produced in the real world in a fast, intelligent and pervasive way. The applications are limitless!

In such cases, using dynamic lights could be a good alternative to convey the main insights of dashboards. For instance, in a call center, the light intensity and color could change according to the number and type of calls (complaints, information, products, etc.) from customers.  By connecting lamps to the Internet, we enter the world of the Internet of Things. Hue lamps can be instructed to react to phenomena on the Internet using the “If This Then That” framework. Using IFTTT, you can use the lamps to turn the light on when it is raining in Amsterdam, or when your plane lands safely in another city around the world.

So how have we connected the lamps to the tweets related to the US elections?  On this occasion, we haven’t performed a complex analysis nor a prediction of the winner as for this purpose, we would need to ask ourselves first if Twitter is the best data stream to use.

Our first step was to design two query sets, in the first query, we count all the
tweets related to common words used to refer to Hillary Clinton
(e.g. Hillary, Clinton, HillaryClinton) or Donald Trump (e.g. Donald, Trump,
DonaldTrump). The results of which are displayed in Figure 5:

Figure 6. The second query reveals a much more balanced pattern with no clear winner.

Additionally, we set rules for the lights as follows:

• Lamp 1 is connected to a twitter feed on query 1 (Figure 5), blinking the color of the candidate who had more tweets in the last 2 seconds and showing a stable color of the candidate who had more tweets in the last hour.

• Lamp 2 has the same behaviour of Lamp 1 but this time it is connected to the second query stream (figure 6).