Thursday, April 18, 2013...3:38 pm

Thatcher funeral – why Twitter sentiment analysis is nonsense

Jump to Comments

UPDATE: See the detailed methodological comment from Francesco at Pulsar TRAC for evidence that the Thatcher Funeral Twitter sentiment analysis referred to here really is super accurate.

ThatcherNatPressinfo_Twitter

We still haven’t quite managed collectively to leave Margaret Thatcher’s death behind (don’t worry – I’m sure it will pass eventually). At the moment we’re in the weird meta analysis-of-the-analysis stage that will hopefully soon disappear up itself.

But of course I’m going to add to it while it’s still here – I wouldn’t want to disappoint you.

So – Twitter sentiment. Today’s post on Campaign’s Wall Blog tells us that “social media remained negative during Thatcher’s funeral” and comes complete with some Twitter sentiment analysis infographics from digital meejah agency Face.

Twitter sentiment analysis is the kind of data analysis that journalism loves. It looks terribly serious and authoritative, but is actually just a record of random wittering by anyone at all. And because it’s all on Twitter you don’t even have to go out and vox pop strangers in the high street.

But here’s the problem. How do you actually tell what the sentiment of the tweets is?

The Thatcher funeral analysis is all a PR effort by Face to push its shiny new Pulsar TRAC social meejah analytics tool (oh, all right – I wasn’t going to put in the link, as I know it’s what they’re angling for, but here it is to save you the search).

It seemed to find a lot of negative sentiment – even, if the graphic above is to be believed, among Sun readers. I find this dubious.

As an experiment, to see how well such things worked, I used a low-rent alternative – the free Tweetfeel sentiment analysis site to analyse the sentiment of Tweets matching the search term “Thatcher funeral”. How did that go?

Screen Shot 2013-04-18 at 15.50.48Like this: “30” positive; “34” negative, making up (I’m guessing – the site is a bit vague) 53% of all Tweets meeting the search criteria.

But hang on – what are those Tweets actually saying?

Here’s one that gets a “negative” from Tweetfeel:

Screen Shot 2013-04-18 at 15.54.29

 

“Disappointing: Obama will not send a representative to thatcher funeral”

Uh – that sounds like a negative to lefty Obama, not the Great Leaderene. Sounds like that should have been a big thumbs up for the funeral.

Or this one that gets a “positive”:

Screen Shot 2013-04-18 at 15.56.49

“@frankieboyle commentary on #thatcher funeral is funny as fuck!”

You know, I’m imagining that’s not going to be as respectful as Tweetfeel seems to indicate.

Twitter_Sentiment_Thatcher_FuneralIn all, of the 63 Twitter sample, 9 Tweets of the 34 marked as negative by Tweetfeel were actually positive for the funeral, while 8 Tweets of the 30 marked as positive were really negative. (Here’s a link to a grab of the actual results so you can check yourself. If you have no life.)

I know what some of you may be thinking: “that kind of evens it out – the analysis is really accurate!”. But data shouldn’t work that way. If more than a quarter of the Tweets were analysed incorrectly, that should put the kibosh on Twitter analysis being anything other than a waste of time.

Now, as you will have spotted, I wasn’t using the shiny Pulsar TRAC tool – maybe it has fantastic algorithms to interpret what those 140 characters really mean, so it’s as accurate as all get out. But maybe it isn’t. Bear that in mind when you see journalists going squee over the next Twitter sentiment infographic…

4 Comments

  • Hi Simon,

    thanks for taking the time to review our PR stunt (why else on Earth would I spent time researching what The Sun readers think of Thatcher?)

    I share all of your doubts on the way sentiment analysis is calculated on most platforms and that’s why we have designed the platform in a different way.

    All sentiment analysis is targeted towards specific entities and we use three systems to measure it:

    – short-form content (below 140 characters, poor grammar, no context)

    – long-form content (above 140 character, structured, context-rich)

    – crowdsourced human coding: you can stream your posts to communities where people receive micropayments in return for microtasks and get the message analysed in real-time and by users who can been screened (e.g. only english speaking from the UK). Now because humans agree even less then machines, we normally tend to send every single post to three different people and then calculate the rating.

    As you know sentiment accuracy varies according to the subject of the conversation, and when it’s particularly tricky we simply up the share of content for which we want the sentiment analysis to be crowdsourced.

    In this case, we also had a research team reviewing the sentiment coding.

    So yes, the data is accurate and the followers of The Sun were prevalently negative about Thatcher.

    POSITIVE
    “RT @SamGyimah: Deeply saddened by the passing away of Baroness Thatcher. A remarkable woman, inspirational leader and great Prime Minister of our country.”

    “Wish Lady Thatcher was still here to sort out this economic crisis. She’d drag Britain off its knees and run circles around @Ed_Miliband”

    NEGATIVE
    “Whether you liked #thatcher or were happy to see her dead, 10 million for her funeral paid for by the taxpayers is absolutely disgraceful!”

    “RT @sickipediabot: RIP Margaret Thatcher. A woman who did more damage to Scotland than alcohol and the deep-frying process combined.”

    “RT @OwenJones84: Outrageous. Not a funeral. A state-funded political broadcast RT @IanDunt BREAKING: Big Ben to be silenced during Margaret Thatcher funeral”

    Hope this help clarifying why we think sentiment analysis is still interesting. If you have any questions shout, I’m @abc3d on Twitter

  • Freelance UnboundNo Gravatar
    April 21st, 2013 at 3:47 pm

    Hey Francesco – thanks for your input. I’ll look out for more of your sentimental claptrap! I’d be interested to see how it works in more detail…

  • What he failed to mention, of course, is that Twitter is hopeless for measuring ANYTHING that matters – simply because it’s a self-selecting medium.

    People who tweet are not representative of the population at large, no matter how much Face (or anyone else) might wish them to be. When you measure anything on Twitter, you’re just measuring the views of a self-appointed section of the population that has no representative value whatsoever.

    They also tweet unevenly – on topics close to their heart, not on others. The end result is so absurdly biassed – in every possible respect – as to be statistically utterly useless.
    Soilman recently posted..The sun is out. The matrix has reloaded.

  • Michael WalkerNo Gravatar
    May 14th, 2013 at 3:00 pm

    The old computing maxim ‘garbage in, garbage out’ applies. At least 95 per cent of what’s posted on social media is complete bollocks, so attempting to analyse it is meaningless.

Leave a Reply


− four = 1

CommentLuv badge

Login
WordPress SEO