At the heart of this ongoing debate (and confusion) surrounding automated sentiment analysis is the issue of  “accuracy”– the degree to which software can correctly extract positive, negative, or neutral tone from text. Using “accuracy” as a criterion for useful sentiment analysis demonstrates a fundamental misunderstanding of what sentiment really is and what “accuracy” really means. Unfortunately, this misunderstanding has led media researchers and software programmers to search for ”100% sentiment analysis accuracy”, and distracted our industry from what its real focus should be– understanding how the media influences human behavior. 

Automated sentiment analysis will never be accurate. Not 1% accurate, 50% accurate, or 100% accurate. To say that an algorithm or statistical model has “accurately” identified a piece of text as positive, negative or neutral requires that sentiment is a real thing in the text that can be correctly identified, like a person’s name or a product. The problem is that positive and negative don’t really exist on paper or on a computer monitor. The scientists and philosophers who study sentiment all agree that it only exists as property of the animal nervous system. “Positive” and “negative” are neurological states that evolved to helps organisms avoid stuff that can harm them or to promote behavior that’s likely to nourish and help them propagate. Sentiment is absolutely not something that exists “out there” in the world; it only exists in our perceptions of the world.

Because positivity and negativity doesn’t really exist in blog posts, Tweets, Facebook updates, or New York Times editorials, neither human analysts or software will ever be able to “accurately” extract sentiment from them. What analysts and software can do instead is approximate or guess what a reader’s reaction might be to the text. Accurate identification could only be done by measuring actual reader’s emotional reactions to the text—which would be too costly and time-consuming to do.

You might be thinking that  the distinction between sentiment existing in the external world (e.g., text) vs. the internal world (our brains) is purely academic. But it has serious implications for how marketers, communications professionals, media professionals and software programmers tackle the issue of measuring sentiment in large volumes of text.

One relatively minor implication is that when people talk about measuring “accuracy” in automated sentiment analysis, they’re really referring to “reliability”, or agreement between an analysts’ guess about a readers’ reactions to text post and the sentiment decision made by the automated tool. This is more of a pet peeve of mine than anything else (it’s one thing when marketers or software programmers make this mistake, but researchers should know better; a good guide to distinguishing between the reliability and accuracy in measurement can be found here).

A much more serious implication is that PR and marketing pros need to stop focusing on “accuracy” (i.e., reliability) and start caring about how humans actually evaluate positivity and negativity in the external world. The latter will be a much better predictor of how the media influences behavior and, ultimately, will be most useful to companies who analyze large quantities of mainstream and social media coverage. Given that the process through which the human brain evaluates external things as positive or negative is only beginning to be understood by philosophers and scientists, I’m not optimistic that software programmers and artificial intelligence folks will be cracking that anytime soon.

Since automated sentiment analysis relies on set rules, nearly always tweaked by human analysts, I’m sure that reliability rates between a tool and a single analyst can reach 90% and beyond (at least within a single set of text on a specific topic). Still, when it comes to approximating what actual human readers are likely to think of an organization or a product in the media, there’s good reason to believe that human analysts have machines beat. Here are a few things that we do know about how humans evaluate things as good or bad. In each of these cases, human analysts will be better at simulating what an actual reader would do than an automated sentiment analysis tool:

 1) Different people are going to have different emotional reactions to text. This point might seem obvious, but it is almost universally overlooked in media measurement conversations. Depending on who you are, you’re probably going to have a different reaction . The human brain is very good at this perspective switching. Starting at a fairly young age, people can simulate the experiences of other people and make good inferences about their emotions, behaviors, etc.. If you’re at a baseball game, for example, and the hitter for the visiting team makes a winning home run, you can effortlessly recognize that that guy must feel pretty good even though it may have ruined your afternoon.

The practical implication for media researchers is that a single piece of text is likely going to have very different affective or emotional meaning depending on who’s perspective you decide to take.  A person could read the  Tweet, “Legit…apply to this contest. Almost no one has applied, so chances are…you’ll win. www.dell.com/w3” and quickly infer that the writer has a positive attitude towards the contest but that the marketing folks at Dell will probably have a negative reaction. Similarly, the post, “Did you see the next generation iPhone? It was left on a counter by mistake. Hum” might be read negatively by Apple’s PR team but that iPhone owners will probably have a completely neutral reaction to it. A good analyst is able to quickly and effortlessly take on the perspective of the article or post author, a naïve reader, company representative, potential customers, legislators, competing companies, or investors when reading an article or social media post. I have yet to see a piece of software that can approximate this. 

2) Mood impacts evaluative judgments. Because goodness and badness aren’t properties of the external world that can be detected, humans have to rely on a variety of  information to make sentiment-based judgments. One key piece of information that people tend to use is their own mood. A host of research has consistently shown that people tend to make mood-congruent judgments about objects in their world. If someone is in a good mood, they tend to rate a range of things, from their own life satisfaction to the taste of food, as being better than if they are in a bad mood. In one research study, Alice Isen and colleagues experimentally induced positive moods in some people, and then asked them to rate the service records of their household appliances (e.g., washers and dryers, coffee makers, etc.). They found that participants in good moods reported much greater satisfaction with the appliances than everyone else (you’re more likely to really like your coffee maker when you’re having a good day). The implication of this for media researchers is that other news, world events, and even  bad weather will likely to affect whether or not a reader interprets a blog post or news story as negative or positive.

3) Context matters. One of the most important thing that media analysts can learn from existing knowledge on how humans evaluate sentiment is that context often determines whether or not people perceive otherwise ambiguous things as being either good or bad. In yet another interesting psychological experiment by James Russell and colleagues, participants were shown pictures of people displaying prototypical emotions, such as happiness, surprise, anger, etc. and were told a story about what the person in the picture had just experienced. The researchers found that the story played a huge role in what emotion the participant rated the face as showing. When told that a woman making a prototypically fearful face had just been made to wait for a table at a restaurant for over an hour despite having a reservation, participants tended to rate the face as showing anger rather than fear. Findings like this suggests that situational cues are yet another piece of information that people use to make evaluative judgments about the outside world.

The practical implication to be taken from this is that other stories, blog posts, and Tweets that have been recently read will impact whether or not someone perceives a piece of media as being positive or negative. Other stories in the same magazine, blog posts preceding the one being analyzed, nearby Tweets in a Twitter feed, etc., should all be considered when determining whether or not readers are going to have a positive or negative reaction to a specific piece of text.

I don’t mean to suggest that, given the complexities of human evaluation, there’s no point in trying to improve automated sentiment analysis. But,  I don’t think that the task will be as easy as many media monitoring software providers would like you to beleive. The human brain is incredibly complex, and getting the output of automated sentiment engines to approximate the emotional reactions of real human readers (e.g., customers, voters, investors, etc.) will be a challenging task. Once these challenges are recognized, however, I’m sure that automated sentiment analysis will eventually come of age as a useful business tool.

Since Context’s founding in 1992,  digital has reshaped the media landscape, and with this shift we have seen new opportunities to derive deeper insights into how influencers and stakeholders perceive and interact with brands. This has allowed us to better measure brand reputations and ultimately measure the business impact of these perceptions.

We believe that there is much additional opportunity to leverage analytics and data-driven consulting in the planning and measurement of earned media campaigns, and that joining forces with Project Metal will help us get there faster. Project Metal is developing a series of services that combine analytics and measurement; search optimization; and digital design and build capabilities. Its services will be entirely complementary to those of the existing NextFifteen brands.

For existing Context Analytics clients it will be business as usual – we are the same team in the same locations, continuing to build on the work we deliver every day. But stay tuned for new insights and solutions to help make more informed marketing decisions.

For more information, see this PR Week story.

There are many reasons I say this:

• 99% of all the content created by PR professionals now appears online
• There is an entire industry growing up around creating tools to mine the huge volumes of data this generates
• The data that can be derived from earned media is potentially more valuable than the data from paid media campaigns because of the inherent rise in the value of earned media.

The limitations to date have been that the focus of data gathering in the PR industry has been to listen and monitor. This is hugely valuable in its own right but it doesn’t directly translate into business value. As Context Analytics becomes a part of Project Metal (article here) the most exciting thing for me is looking at how we move beyond listening and monitoring to measuring the impact of earned media on a business. This is now possible and it is an area where Context Analytics has been doing some pioneering work.

By combining media measurement, web analytics, audience data and search analytics we can now demonstrate which type of articles in which digital media destinations generate the lowest cost per customer acquisition. As mentioned in this piece of research from Alterian, earned media outreach can genuinely compete for marketing spend against direct marketing dollars or Google adwords dollars. Now that truly is a crossroads for the PR industry.

First, I should provide some background into what will seem like an esoteric debate for non-research audiences. The debate centers around whether or not PR measurement should adopt or prohibit using media cost estimates from advertising rate cards to measure earned media. For those unfamiliar with the “weighted media costs”/advertising equivalency value debate, here’s a very oversimplified summary:

AVE Advocates: “AVEs are great because they can be shown in units of dollars which CEOs and CMOs understand.”

AVE Opponents: “PR is not the equivalent of advertising. There is no evidence that a half page of earned media has equivalent impact on business outcomes as a half page of advertising.”

AVE Advocates: “But AVEs have stronger correlations with business outcomes than clip counts or impressions.”

AVE Advo-ponents (the middle grounders): “Okay, there’s something to this whole AVE thing, but the name is a little misleading since it suggests that advertising and earned media have equivalent value. Since advertising rates aren’t really indicators of value but are rather about cost, let’s call the metric something like ‘media cost equivalency’.”

Jeffrey and colleagues’ paper is written in support of this AVE advo-ponent view. It essentially offers a new name for AVEs- “media cost” , provides a method for weighing the costs by a few factors including sentiment, and finally provides some studies that are intended to support the predictive validity of weighted media costs. The case studies assess the correlational strength of sentiment-weighted clip counts, impressions, and weighted media costs/AVEs with business outcomes. In most of the case studies, Jeffrey and colleagues find that weighted media costs have the strongest correlation with business outcomes (e.g., revenue), followed by impressions and then sentiment-weighted clip counts. In the end proclaim that weighted media costs are the “best” PR metric and announce a new “paradigm shift” in earned media measurement.

I generally agree that composite metrics, such as “weighted media costs” are going to be better predictors of business outcomes than sentiment, clip count,  impressions or media costs alone. But I’m concerned that some audiences will interpret this paper as showing that media costs are a better business outcome predictor than clip counts or impressions. If you look at the data really closely (read: “skip this paragraph unless you’re prepared for some serious statistics jargon”), there is no evidence that media costs are better at predicting business outcomes than any other metric (to be completely fair, the authors never explicitly state that it is). One issue with the results is that the authors use Pearson correlations to assess the relationship between variables over time. Using Pearson correlation coefficients on time-series data can be a huge statistical “no-no” (proper time-series correlations have to account for a phenomenon called autocorrelation, which even time-lagged Pearson correlation coefficients cannot do). This means that the r and R² values presented in the paper are inflated and that the actual correlations for clip count, impressions, and weighted media costs are probably much closer to each other than the Pearson correlations reported in the paper. Secondly, the authors compare 3 correlation coefficients for three tightly interrelated variables. Clearly, clip count, impression and media costs are going to have strong positive correlations with each other (as clip count goes up, so will impressions and media costs), and sentiment is being weighted in each of the three media metrics. Because the authors used Pearson correlation coefficients (which compare each of the metrics in a silo), we don’t know the unique contributions of sentiment, clip count, impressions, and media costs to business outcomes. A more rigorous analysis where each of these variables were included in a single regression model could reveal a completely different result. It’s possible, for example, that clip count is the strongest predictor of business outcomes, followed by sentiment, impressions, and media cost. But given the way the statistics were conducted, it’s not possible to make these sorts of comparative judgments about the predictive strength of media metrics.

Regardless of what can and cannot be gleaned from the case studies, the authors make a good point that “weighted media costs” are likely to be a very good predictor of business outcomes because they combine so many different metrics, including clip count, sentiment, audience size (impressions), as well as the “prominence” of the brand mention (e.g., how often it appeared in the story) and the credibility of the source. It’s a bit of a no-brainer that the more metrics you use to predict something, the better the prediction will be. A classic example is college GPA. College admission departments know that high school GPA, SAT scores, and the number of extracurricular activities listed on the application will be a better predictor of college success than SAT scores alone. The same logic applies here: clip counts combined with coverage sentiment, audience size, brand prominence, and publication credibility is going to be a better predictor than clip counts alone.

I believe that this last point is being overlooked in the AVE debate. A lot of people are hung up on the phrase “advertising equivalency” and how this metric can be misused and misinterpreted when presented in currency values. One of the key themes in all of these case studies is that measurement that combines many different metrics is likely to be better than just using one metric, let that be sentiment, impressions, media prominence, clip count, or a media cost-like metric. Each of these metrics say something unique about a brands reputation in the media and each metric probably deserves some attention in PR measurement reports.

I want to make one final remark on this paper and the AVE debate in general. There’s been concern that using weighted media costs as a measurement standard will be problematic since access to media cost databases are expensive and only a handful of vendors offer weighted media cost-like metrics (see Katie Paine’s comment here). Fortunately, as the authors admit, the media cost is really just comprised of the prominence of the news coverage and the credibility of the publication. This probably means you don’t need to have access to proprietary media cost databases to get the predictive strength of “weighted media costs”. It’s easy to calculate the prominence of a brand within an article (was the brand mentioned in the headline, lead paragraph, etc.), and credibility can be estimated using a variety of free metrics (Google PageRank is a great one for estimating news site credibility, for example). There plenty of creative ways that communications professionals can create weighted metrics that predict business outcomes just as well as “weighted media costs” without relying on VMS or similar vendors.

The full article can be found here.

www.flickr.com/photos/44153365@N05/4058390947