Thursday, October 25, 2012

My role in adding Sentiment Analysis capability to IBM's Smarter Cities offerings

Earlier this year I announced that I was changing jobs again to incubation team leader. In this role I am supposed to lead innovative projects to introduce experimental new features in our products. The nature of these innovations means that there is bound to be some risk of failure, but I am delighted to be able to say that the first project to develop a city sentiment dashboard was an even bigger success than expected. In order to ensure were meeting real customer needs, we developed it in conjunction with a large European City who gave us very specific requirements. The end result was so well received that this week the software was formally upgraded from a customised solution into a fully supported IBM product.

Our solution uses Cognos Consumer Insight (CCI) to analyse sentiment being expressed about the themes that the city is interested in. CCI provides really excellent analysis capabilities, but the user interface can be daunting for non-experts to use.  Hence, the software we developed pulls the key information into a single easy to understand dashboard that is part of the Intelligent Operations Centre (IOC). Our dashboard was released in the form of an update to the IOC. Customers will typically deploy this software as part of a Smarter Cities project. If you are interested you should get in touch with your IBM sales representative or if you don't have an existing relationship with IBM leave a comment here and I will arrange for someone to get in touch.

Since IOC has many features, the information about the new sentiment analysis feature is buried deep on the product page. The screenshot below shows the dashboard we developed. I deliberately blurred the text in the screenshot because I don't think that I am at liberty to reveal what city we were working with or what topics they wanted to track. However, I think you can get a feel for how we managed to squeeze all of the sentiment data onto a single easy to understand screen:
  • On the left there is a hierarchical list of the themes that the customer specified that they are interested in tracking sentiment about. For each theme there is a thumbs up or thumbs up icon to give a quick indication of whether we found more positive or negative expressions of sentiment about the theme in question.
  • If you click on a theme, we will display details about the sentiment we discovered for this theme. We have pie charts showing the overall distribution of sentiment observed as well as a more detailed breakdown of the sentiment for each type of site (e.g. we might have observed radically different sentiment in newspapers, blogs and twitter).
  • On the bottom we have a trend chart showing how the sentiment statistics varied over time and finally on the right hand side we show you a few samples of positive and negative snippets that we found. Clicking on these snippets will bring you directly to the original source of the sentiment so you can read the snippet in context.

It is amazing what can be achieved by a small agile team in 6 months. Hopefully we will be equally successful with our next projects (which have not yet been decided).


  1. Very interesting, very promising. I wonder whether this can be expanded to also reflect location (if the source data can even be gathered) . In very mega-cities like Sao Paulo and New York, the sentiments can be so different across regions. The data analysis possibilities are unending, such as distribution over days of the week, or hours of the day, correlation to budget allocation, and many others.

  2. Good question Denilson. It is possible to break down sentiment by area within the city and we did this in our initial use case. However, we are concerned with the location being discussed rather than the current location of the writer e.g. a person in Ireland could be talking about their good/bad experiences during a recent trip to New York. On some topics it is not possible to specify where within the city they are talking about e.g. if they were expressing an opinion on a newly introduced tax.

  3. Brian, do you know : ?
    I wonder if you think it could be useful for a test providing direct personal sentiment upon context. We are based in Dublin:
    Thanks for insights.

  4. No - I never heard of MySmark - thanks for the pointer I will check it out