Next week I will be adressing the Intelligent Systems Summit hosted by University of Ulster in Derry/Londonderry

Here is the content I will present.

I hope to travel up and back by train, because a colleague told me that Michael Palin declared that the trip between Belfast and Londonderry is the most scenic in the British Isles. It should also be more relaxing than driving.

How big does your city have to be in order to make Sentiment Analysis worthwhile

I wrote earlier about a solution I helped develop which allows city leaders monitor the sentiment being expressed online about their city. As we present this solution to the leaders of various cities, one of the questions that is always asked is whether their city is well known enough to generate enough mentions so that the sentiment charts will be statistically significant.

The general rule of thumb we have been using is that a city must have a population of at least .25 million in order to make the tool feasible. The thing that matters is the number of mentions of the city online (we would hope for at least 5k per week) and many times (but not always) the population of a city can be a rough guide to how many mentions that are likely to be made. Therefore I decided to run a quick test to see how many mentions I would find for a pseudo-random selection of cities with both large and small populations (some of the smaller places were not technically cities) in the first week of April this year.

This table summarises the results:

City	Population	Mentions/week
Loughrea	5,057	102
Birr	5,818	291
Nazareth	14,123	3,873
Clemmons	18,627	224
Bethlehem	25,266	3,661
Navan	28,158	775
Dundalk	31,149	898
Lorient	58,135	1,971
Galway	75,529	4,758
Cergy-Pontoise	183,430	235
Bordeaux	235,891	9,555
Montpellier	255,080	6,539
Toulouse	449,328	11,933
Dublin	527,612	29,859
Boston	625,087	111,035
Jerusalem	801,000	20,767
Paris	2,234,105	178,406
Sydney	4,627,345	52,791
London	8,173,194	257,094
Bangalore	8,474,970	16,455
Moscow	11,503,501	49,534
Tokyo	13,185,502	216,606
New York	19,570,261	440,535
Beijing	20,693,000	274,062

This can be visualised by the following chart:

I think you can see that there is a correlation between city size and the number of mentions (correlation coefficient = 0.83). You can also see that Galway is getting roughly enough mentions to make sentiment analysis useful despite only having a population of 75k, while Cergy-Panoise has more than double the population but is not getting enough internet mentions to make sentiment monitoring useful.

A few examples of where the city gets a number of mentions very different from what would be predicted for their population:

• Both Bethlehem and Nazareth get many more mentions than would be predicted by their population (e.g. they are both mentions significantly more than Navan and Dundalk which have larger populations). This is probably due to the biblical significance of the towns - in fact this is why I chose them for inclusion in the test and I don't know the names of any other towns in the middle-east with such small population.
• Where the name of the city in the local language was different from the name in English I searched on both versions of the name. In general the local language version received more hits (e.g. there were 6.5 times as many mentions of 北京 as there were for Beijing. However, for the Israeli cities it was the other way around. For example the word "Jerusalem" got 20,711 mentions while the Arabic and Hebrew translations of the city name only had 17 and 39 mentions respectively. Perhaps this is an indication that people in other parts of the world are talking about the city much more than the locals.
• Cergy-Ponoise only gets 235 mentions,while Lorient gets 1,971 mentions despite having a smaller population. I am not sure why this should be the case, but perhaps it is due to the fact that Cergy-Ponoise is so close to Paris that local residents consider themselves to be Parisians. Lorient has no similar large city nearby to overshadow it.
• The statistics will vary over time.For example,if I has run my test for the 3rd week in April rather than the first, the number of mentions for Boston would have been 1,893,159 rather than 111,035 - probably due to coverage of the marathon bombing.

Notes:

• In the case of some of the cities I chose, there are multiple cities with same name - for example, the wikipedia disambiguation page for Boston lists several cities with this name, but I only counted the population of the capital of Massachusetts (the population of the other cities would probably not be very large). 
• Wikipedia sometimes has several different estimates of population because of ambiguity of how large an area to include. I only considered the first number listed(which is typically the smallest). In some cases the difference might be minor, but in others it would be very significant. For example the estimates of the population of Boston vary by an order of magnitude from 625 thousand to 7.6 million.

My role in adding Sentiment Analysis capability to IBM's Smarter Cities offerings

Earlier this year I announced that I was changing jobs again to incubation team leader. In this role I am supposed to lead innovative projects to introduce experimental new features in our products. The nature of these innovations means that there is bound to be some risk of failure, but I am delighted to be able to say that the first project to develop a city sentiment dashboard was an even bigger success than expected. In order to ensure were meeting real customer needs, we developed it in conjunction with a large European City who gave us very specific requirements. The end result was so well received that this week the software was formally upgraded from a customised solution into a fully supported IBM product.

Our solution uses Cognos Consumer Insight (CCI) to analyse sentiment being expressed about the themes that the city is interested in. CCI provides really excellent analysis capabilities, but the user interface can be daunting for non-experts to use.  Hence, the software we developed pulls the key information into a single easy to understand dashboard that is part of the Intelligent Operations Centre (IOC). Our dashboard was released in the form of an update to the IOC. Customers will typically deploy this software as part of a Smarter Cities project. If you are interested you should get in touch with your IBM sales representative or if you don't have an existing relationship with IBM leave a comment here and I will arrange for someone to get in touch.

Since IOC has many features, the information about the new sentiment analysis feature is buried deep on the product page. The screenshot below shows the dashboard we developed. I deliberately blurred the text in the screenshot because I don't think that I am at liberty to reveal what city we were working with or what topics they wanted to track. However, I think you can get a feel for how we managed to squeeze all of the sentiment data onto a single easy to understand screen:

• On the left there is a hierarchical list of the themes that the customer specified that they are interested in tracking sentiment about. For each theme there is a thumbs up or thumbs up icon to give a quick indication of whether we found more positive or negative expressions of sentiment about the theme in question.
• If you click on a theme, we will display details about the sentiment we discovered for this theme. We have pie charts showing the overall distribution of sentiment observed as well as a more detailed breakdown of the sentiment for each type of site (e.g. we might have observed radically different sentiment in newspapers, blogs and twitter).
• On the bottom we have a trend chart showing how the sentiment statistics varied over time and finally on the right hand side we show you a few samples of positive and negative snippets that we found. Clicking on these snippets will bring you directly to the original source of the sentiment so you can read the snippet in context.

It is amazing what can be achieved by a small agile team in 6 months. Hopefully we will be equally successful with our next projects (which have not yet been decided).