At the heart of this ongoing debate (and confusion) surrounding automated sentiment analysis is the issue of  “accuracy”– the degree to which software can correctly extract positive, negative, or neutral tone from text. Using “accuracy” as a criterion for useful sentiment analysis demonstrates a fundamental misunderstanding of what sentiment really is and what “accuracy” really means. Unfortunately, this misunderstanding has led media researchers and software programmers to search for ”100% sentiment analysis accuracy”, and distracted our industry from what its real focus should be– understanding how the media influences human behavior. 

Automated sentiment analysis will never be accurate. Not 1% accurate, 50% accurate, or 100% accurate. To say that an algorithm or statistical model has “accurately” identified a piece of text as positive, negative or neutral requires that sentiment is a real thing in the text that can be correctly identified, like a person’s name or a product. The problem is that positive and negative don’t really exist on paper or on a computer monitor. The scientists and philosophers who study sentiment all agree that it only exists as property of the animal nervous system. “Positive” and “negative” are neurological states that evolved to helps organisms avoid stuff that can harm them or to promote behavior that’s likely to nourish and help them propagate. Sentiment is absolutely not something that exists “out there” in the world; it only exists in our perceptions of the world.

Because positivity and negativity doesn’t really exist in blog posts, Tweets, Facebook updates, or New York Times editorials, neither human analysts or software will ever be able to “accurately” extract sentiment from them. What analysts and software can do instead is approximate or guess what a reader’s reaction might be to the text. Accurate identification could only be done by measuring actual reader’s emotional reactions to the text—which would be too costly and time-consuming to do.

You might be thinking that  the distinction between sentiment existing in the external world (e.g., text) vs. the internal world (our brains) is purely academic. But it has serious implications for how marketers, communications professionals, media professionals and software programmers tackle the issue of measuring sentiment in large volumes of text.

One relatively minor implication is that when people talk about measuring “accuracy” in automated sentiment analysis, they’re really referring to “reliability”, or agreement between an analysts’ guess about a readers’ reactions to text post and the sentiment decision made by the automated tool. This is more of a pet peeve of mine than anything else (it’s one thing when marketers or software programmers make this mistake, but researchers should know better; a good guide to distinguishing between the reliability and accuracy in measurement can be found here).

A much more serious implication is that PR and marketing pros need to stop focusing on “accuracy” (i.e., reliability) and start caring about how humans actually evaluate positivity and negativity in the external world. The latter will be a much better predictor of how the media influences behavior and, ultimately, will be most useful to companies who analyze large quantities of mainstream and social media coverage. Given that the process through which the human brain evaluates external things as positive or negative is only beginning to be understood by philosophers and scientists, I’m not optimistic that software programmers and artificial intelligence folks will be cracking that anytime soon.

Since automated sentiment analysis relies on set rules, nearly always tweaked by human analysts, I’m sure that reliability rates between a tool and a single analyst can reach 90% and beyond (at least within a single set of text on a specific topic). Still, when it comes to approximating what actual human readers are likely to think of an organization or a product in the media, there’s good reason to believe that human analysts have machines beat. Here are a few things that we do know about how humans evaluate things as good or bad. In each of these cases, human analysts will be better at simulating what an actual reader would do than an automated sentiment analysis tool:

 1) Different people are going to have different emotional reactions to text. This point might seem obvious, but it is almost universally overlooked in media measurement conversations. Depending on who you are, you’re probably going to have a different reaction . The human brain is very good at this perspective switching. Starting at a fairly young age, people can simulate the experiences of other people and make good inferences about their emotions, behaviors, etc.. If you’re at a baseball game, for example, and the hitter for the visiting team makes a winning home run, you can effortlessly recognize that that guy must feel pretty good even though it may have ruined your afternoon.

The practical implication for media researchers is that a single piece of text is likely going to have very different affective or emotional meaning depending on who’s perspective you decide to take.  A person could read the  Tweet, “Legit…apply to this contest. Almost no one has applied, so chances are…you’ll win. www.dell.com/w3” and quickly infer that the writer has a positive attitude towards the contest but that the marketing folks at Dell will probably have a negative reaction. Similarly, the post, “Did you see the next generation iPhone? It was left on a counter by mistake. Hum” might be read negatively by Apple’s PR team but that iPhone owners will probably have a completely neutral reaction to it. A good analyst is able to quickly and effortlessly take on the perspective of the article or post author, a naïve reader, company representative, potential customers, legislators, competing companies, or investors when reading an article or social media post. I have yet to see a piece of software that can approximate this. 

2) Mood impacts evaluative judgments. Because goodness and badness aren’t properties of the external world that can be detected, humans have to rely on a variety of  information to make sentiment-based judgments. One key piece of information that people tend to use is their own mood. A host of research has consistently shown that people tend to make mood-congruent judgments about objects in their world. If someone is in a good mood, they tend to rate a range of things, from their own life satisfaction to the taste of food, as being better than if they are in a bad mood. In one research study, Alice Isen and colleagues experimentally induced positive moods in some people, and then asked them to rate the service records of their household appliances (e.g., washers and dryers, coffee makers, etc.). They found that participants in good moods reported much greater satisfaction with the appliances than everyone else (you’re more likely to really like your coffee maker when you’re having a good day). The implication of this for media researchers is that other news, world events, and even  bad weather will likely to affect whether or not a reader interprets a blog post or news story as negative or positive.

3) Context matters. One of the most important thing that media analysts can learn from existing knowledge on how humans evaluate sentiment is that context often determines whether or not people perceive otherwise ambiguous things as being either good or bad. In yet another interesting psychological experiment by James Russell and colleagues, participants were shown pictures of people displaying prototypical emotions, such as happiness, surprise, anger, etc. and were told a story about what the person in the picture had just experienced. The researchers found that the story played a huge role in what emotion the participant rated the face as showing. When told that a woman making a prototypically fearful face had just been made to wait for a table at a restaurant for over an hour despite having a reservation, participants tended to rate the face as showing anger rather than fear. Findings like this suggests that situational cues are yet another piece of information that people use to make evaluative judgments about the outside world.

The practical implication to be taken from this is that other stories, blog posts, and Tweets that have been recently read will impact whether or not someone perceives a piece of media as being positive or negative. Other stories in the same magazine, blog posts preceding the one being analyzed, nearby Tweets in a Twitter feed, etc., should all be considered when determining whether or not readers are going to have a positive or negative reaction to a specific piece of text.

I don’t mean to suggest that, given the complexities of human evaluation, there’s no point in trying to improve automated sentiment analysis. But,  I don’t think that the task will be as easy as many media monitoring software providers would like you to beleive. The human brain is incredibly complex, and getting the output of automated sentiment engines to approximate the emotional reactions of real human readers (e.g., customers, voters, investors, etc.) will be a challenging task. Once these challenges are recognized, however, I’m sure that automated sentiment analysis will eventually come of age as a useful business tool.

I recently came across this post from Nokia’s Arto  Joensuu titled Conversations are the New Conversion. In the accompanying SlideShare presentation, he makes the case that the traditional sales funnel is no longer linear and controllable. Consumers are now are in control and make their own journey through the “inverted funnel.” This puts new demands on marketers, as the traditional one-way forms of communication increasingly struggle to attract consumer attention. Arto’s presentation says that they have found that ~30% of engagements are generated from paid media, while the rest is generated through owned and earned media. This is why he argues that Social Media Optimization combined with SEO is critical. I couldn’t agree more. Whether you call it Social Media Optimization or Earned Media Optimization (which is the phrase we prefer), the basic message is the same: if you think that the media you own and the one you pay for is all you need to leverage in your marketing campaigns, then you’re missing a massive opportunity.

So what exactly is earned media? Earned media happens any time a brand or a product is mentioned or discussed in a place outside of a brand’s direct control. It can be anything from a positive review in the New York Times, to your best friend sending you a note via Facebook to check out this cool new product. Essentially, earned media is any media generated that you didn’t pay for directly, and if it is an endorsement or a recommendation by someone trusted, it can make all the difference. Conversely, one single bad review can be the ultimate deterrent, and ruin all well-laid marketing plans.

Now, it is important to note that while earned media occurs outside of a brand’s direct control, it does not mean that a brand cannot influence the process, or be part of the conversation. For one thing, PR has been – and still is – a proven tool for influencing influencers. And influence still matters today, even if the field of influence has fragmented and mutated into something many communicators are grappling with understanding. But crucially, it puts the onus on marketers and communicators to really understand not only what their target customers and their spheres of influence really care about, but how and where they talk about it. Because if you cannot communicate your message in a way that resonates with your intended target, they can skip it in an easy click.

And that is why the word “earned” is very apt. In an attention-deficit economy, it is harder and harder to earn the interest, attention, engagement, and ultimately, the trust of your customer. Therefore we think that it is critical for marketers to understand and optimize the impact earned media has on their brands. As Peter Drucker famously said, “if you cannot measure it, you cannot control it.” But understanding and optimizing earned media goes far beyond just measurement. As SEO and SEM pros will tell you, optimization means integrating analytics deeply into your planning process (and that planning process has to be actively managed and revisited). And it means going beyond “out-of-the-box” data. Data only becomes truly valuable when you apply the business context to it that makes it actionable to decision-makers. We’ll be posting more on Earned Media Optimization over the next few months, so stay tuned.

Republished From Institute For Public Relations Conversations Digest

“What do web analytics have to do with public relations?” It’s a good question, given that web analytics are most often used by SEO professionals and online marketers to track visitors and sales from search results and content advertisements.

The digitization of communications has enabled marketers to better understand the impact of their campaigns by directly measuring audience behavior. This is critical to companies that spend large sums on buying media placements or to optimize their website, as it has enabled them to understand what works and what doesn’t in dollar terms. There is no reason why the same methodologies cannot be applied to the media that a company “earns,” which is the media attention a company can generate through effective public relations and communications, or the “buzz” a product can generate online.

In fact, we would argue that earned media is actually a very powerful marketing channel that can be measured, understood and optimized on the same terms as paid media and search marketing. The number of unique visitors referred to an organization’s website by earned media, the pages that visitors access, and whether or not they completed some goal (e.g., downloaded a white paper, made a purchase, made a donation, etc.) can be directly tracked in a way that has not been possible before—at least not without extensive primary research.

In the new paper published by the Institute’s Commission on Public Relations Measurement and Evaluation, we outline practical steps for public relations practitioners who want to adopt web analytics as part of their media measurement strategy. The paper focuses on what sort of data public relations professionals can obtain from web analytics, how to conduct basic quality control for the data, and how to integrate the data with other media monitoring and research.

The paper addresses how web analytics can be used to answer broad questions such as:

• How do sale conversion rates from earned media compare to online marketing channels?
• Is our corporate Twitter account driving traffic to the right Web pages?
• Are our press releases or social media releases being cited by journalists and bloggers, and if so, do they drive traffic to our corporate site?
• Is “Key Message A” more effective at driving sales than “Key Message B?”
• Should we invest more resources in social or traditional media?
• Where do we find the audiences most likely to respond to our campaigns?

At first glance, answers to these questions might appear out of reach. Fortunately, web analytics are more accessible and cost-effective than ever. This technology is not necessarily expensive (its free if you’re using Google Analytics) and most large organizations have a web analytics team that can help public relations teams get the data and reports they need to inform communication strategy.

Since web analytics technology has some technical limitations and most organizations sell products and generate sales leads through offline channels, web analytics might not be the “holy grail” ROI measurement system that the public relations industry has been waiting for. That being said, it might be the closest thing yet.

In much the same way that online advertising has revolutionized how advertisers can measure and optimize their efforts, public relations can leverage web analytics techniques to measure actual user behavior and optimize campaigns to get the best outcomes.

Go here to download the white paper or click the link below to got to the Institute for Public Relations website to read more.

Using Web Analytics to Measure the Impact of Earned Online Media on Business Outcomes: A Methodological Approach

Since Context’s founding in 1992,  digital has reshaped the media landscape, and with this shift we have seen new opportunities to derive deeper insights into how influencers and stakeholders perceive and interact with brands. This has allowed us to better measure brand reputations and ultimately measure the business impact of these perceptions.

We believe that there is much additional opportunity to leverage analytics and data-driven consulting in the planning and measurement of earned media campaigns, and that joining forces with Project Metal will help us get there faster. Project Metal is developing a series of services that combine analytics and measurement; search optimization; and digital design and build capabilities. Its services will be entirely complementary to those of the existing NextFifteen brands.

For existing Context Analytics clients it will be business as usual – we are the same team in the same locations, continuing to build on the work we deliver every day. But stay tuned for new insights and solutions to help make more informed marketing decisions.

For more information, see this PR Week story.

There are many reasons I say this:

• 99% of all the content created by PR professionals now appears online
• There is an entire industry growing up around creating tools to mine the huge volumes of data this generates
• The data that can be derived from earned media is potentially more valuable than the data from paid media campaigns because of the inherent rise in the value of earned media.

The limitations to date have been that the focus of data gathering in the PR industry has been to listen and monitor. This is hugely valuable in its own right but it doesn’t directly translate into business value. As Context Analytics becomes a part of Project Metal (article here) the most exciting thing for me is looking at how we move beyond listening and monitoring to measuring the impact of earned media on a business. This is now possible and it is an area where Context Analytics has been doing some pioneering work.

By combining media measurement, web analytics, audience data and search analytics we can now demonstrate which type of articles in which digital media destinations generate the lowest cost per customer acquisition. As mentioned in this piece of research from Alterian, earned media outreach can genuinely compete for marketing spend against direct marketing dollars or Google adwords dollars. Now that truly is a crossroads for the PR industry.

First, I should provide some background into what will seem like an esoteric debate for non-research audiences. The debate centers around whether or not PR measurement should adopt or prohibit using media cost estimates from advertising rate cards to measure earned media. For those unfamiliar with the “weighted media costs”/advertising equivalency value debate, here’s a very oversimplified summary:

AVE Advocates: “AVEs are great because they can be shown in units of dollars which CEOs and CMOs understand.”

AVE Opponents: “PR is not the equivalent of advertising. There is no evidence that a half page of earned media has equivalent impact on business outcomes as a half page of advertising.”

AVE Advocates: “But AVEs have stronger correlations with business outcomes than clip counts or impressions.”

AVE Advo-ponents (the middle grounders): “Okay, there’s something to this whole AVE thing, but the name is a little misleading since it suggests that advertising and earned media have equivalent value. Since advertising rates aren’t really indicators of value but are rather about cost, let’s call the metric something like ‘media cost equivalency’.”

Jeffrey and colleagues’ paper is written in support of this AVE advo-ponent view. It essentially offers a new name for AVEs- “media cost” , provides a method for weighing the costs by a few factors including sentiment, and finally provides some studies that are intended to support the predictive validity of weighted media costs. The case studies assess the correlational strength of sentiment-weighted clip counts, impressions, and weighted media costs/AVEs with business outcomes. In most of the case studies, Jeffrey and colleagues find that weighted media costs have the strongest correlation with business outcomes (e.g., revenue), followed by impressions and then sentiment-weighted clip counts. In the end proclaim that weighted media costs are the “best” PR metric and announce a new “paradigm shift” in earned media measurement.

I generally agree that composite metrics, such as “weighted media costs” are going to be better predictors of business outcomes than sentiment, clip count,  impressions or media costs alone. But I’m concerned that some audiences will interpret this paper as showing that media costs are a better business outcome predictor than clip counts or impressions. If you look at the data really closely (read: “skip this paragraph unless you’re prepared for some serious statistics jargon”), there is no evidence that media costs are better at predicting business outcomes than any other metric (to be completely fair, the authors never explicitly state that it is). One issue with the results is that the authors use Pearson correlations to assess the relationship between variables over time. Using Pearson correlation coefficients on time-series data can be a huge statistical “no-no” (proper time-series correlations have to account for a phenomenon called autocorrelation, which even time-lagged Pearson correlation coefficients cannot do). This means that the r and R² values presented in the paper are inflated and that the actual correlations for clip count, impressions, and weighted media costs are probably much closer to each other than the Pearson correlations reported in the paper. Secondly, the authors compare 3 correlation coefficients for three tightly interrelated variables. Clearly, clip count, impression and media costs are going to have strong positive correlations with each other (as clip count goes up, so will impressions and media costs), and sentiment is being weighted in each of the three media metrics. Because the authors used Pearson correlation coefficients (which compare each of the metrics in a silo), we don’t know the unique contributions of sentiment, clip count, impressions, and media costs to business outcomes. A more rigorous analysis where each of these variables were included in a single regression model could reveal a completely different result. It’s possible, for example, that clip count is the strongest predictor of business outcomes, followed by sentiment, impressions, and media cost. But given the way the statistics were conducted, it’s not possible to make these sorts of comparative judgments about the predictive strength of media metrics.

Regardless of what can and cannot be gleaned from the case studies, the authors make a good point that “weighted media costs” are likely to be a very good predictor of business outcomes because they combine so many different metrics, including clip count, sentiment, audience size (impressions), as well as the “prominence” of the brand mention (e.g., how often it appeared in the story) and the credibility of the source. It’s a bit of a no-brainer that the more metrics you use to predict something, the better the prediction will be. A classic example is college GPA. College admission departments know that high school GPA, SAT scores, and the number of extracurricular activities listed on the application will be a better predictor of college success than SAT scores alone. The same logic applies here: clip counts combined with coverage sentiment, audience size, brand prominence, and publication credibility is going to be a better predictor than clip counts alone.

I believe that this last point is being overlooked in the AVE debate. A lot of people are hung up on the phrase “advertising equivalency” and how this metric can be misused and misinterpreted when presented in currency values. One of the key themes in all of these case studies is that measurement that combines many different metrics is likely to be better than just using one metric, let that be sentiment, impressions, media prominence, clip count, or a media cost-like metric. Each of these metrics say something unique about a brands reputation in the media and each metric probably deserves some attention in PR measurement reports.

I want to make one final remark on this paper and the AVE debate in general. There’s been concern that using weighted media costs as a measurement standard will be problematic since access to media cost databases are expensive and only a handful of vendors offer weighted media cost-like metrics (see Katie Paine’s comment here). Fortunately, as the authors admit, the media cost is really just comprised of the prominence of the news coverage and the credibility of the publication. This probably means you don’t need to have access to proprietary media cost databases to get the predictive strength of “weighted media costs”. It’s easy to calculate the prominence of a brand within an article (was the brand mentioned in the headline, lead paragraph, etc.), and credibility can be estimated using a variety of free metrics (Google PageRank is a great one for estimating news site credibility, for example). There plenty of creative ways that communications professionals can create weighted metrics that predict business outcomes just as well as “weighted media costs” without relying on VMS or similar vendors.

If your reaction to the headline was, “what on earth does Web analytics have to do with my job?” you probably weren’t alone. Web analytics might be thought of as the realm of SEO pros and online marketing teams, but it can be an incredibly valuable tool for PR teams too. In fact, Web analytics can give you insight into the value of PR and the types of business outcomes it helps drive in a way that hasn’t been possible without expensive primary research. In much the same way, online advertising has revolutionized how advertisers can measure and optimize outcomes, PR can leverage exactly the same tools and techniques. As communications becomes increasingly more digital, it also becomes increasingly important to measure actual user behavior and optimize campaigns to get the best outcomes.

Here are some examples of questions that Web analytics can help you answer:

• Is our corporate Twitter account driving traffic to the right Web pages?
• Are our press releases or social media releases being cited by journalists and bloggers, and if so, do they drive traffic to our corporate site?
• Is Key Message A more effective at driving sales than Key Message B?
• Should we invest more resources in social or traditional media?
• Where do we find the audiences most likely to respond to our campaigns?

While some of these questions require advanced analysis and statistics, there are many straightforward questions you can ask your internal Web analytics team for data on:

• For starters, get some data on what unpaid sites drive the most traffic to your Web site. Unpaid traffic includes any Web sites that provide a link to you for which you have not paid (i.e., not ads or paid search). Many of these sites are influential publications that publish content about your brand, so you should know who is most effective at driving awareness and demand.
• Next, ask questions about what the traffic that these sites refer looks like. Do they tend to sign up for information or buy things on the Web site (or to put in Web analytics speak: “how well do they convert?”). Where are they located geographically? What keywords did they use to find the information, if any (this is great input into determining how you should write copy about your company)?
• Then you may want to do some benchmarking. How does earned media compare to paid media? How does Twitter compare to blogs?

Your internal Web analytics team should be able to provide you some of these reports out of the system or provide you or your analyst of choice access to the application. You can also talk to your agency or research vendor who can help answer your questions on how to get started. We frequently get asked by clients to do this and also help answer complex questions such as: what messaging results in more sales? Where are the untapped audiences with the most potential? Which audience segments should you target with various messages to get optimal business outcomes? There are many ways you can use the data to give you campaign insights, and if you combine it with other data sources, the possibilities are vast.

For more information on the subject of how to get started using Web analytics for PR, you can download our white paper on the subject (published by IPR), or you can also take a look at this presentation, which Context Analytics’ Seth Duncan gave at IPR’s Measurement Summit.

The full article can be found here.

The belief that traditional media has become irrelevant is well particularly well summarized by David Meerman Scott’s recommendations to PR professionals in the popular The New Rules of Marketing and PR:

“Instead of spending tens of thousands of dollars per month on a media relations program that tries to convince a handful of reporters at select magazines, newspapers, and TV stations to cover us, we should be targeting the plugged-in bloggers, online news sites, micro-publications, public speakers, analysts, and consultants that reach the targeted audiences that are looking for what we have to offer.”

This sounds like an exciting new way of doing PR, but the only evidence that is ever presented to support this perspective are case studies of wildly successful and serendipitous marketing campaigns– usually about a brand with a viral video on YouTube (think Mentos and Diet Coke).

For the vast majority of clients I work with, mainstream media is still well in control and by far the most “relevant” source of information. Influential blogs, forums, Twitter, even YouTube discussions around most types of brands (I admit, consumer tech might be an exception) all source most of their information from two very old-school forms of media: the press release and mainstream print and online publications (and I mean very old-school mainstream media, the sort that Murdoch owns). To help illustrate, I’ve shown a scrubbed version of a blog map for a month’s worth of discussions around one of our client’s brands below. The nodes indicate mainstream and social media sites. The arrows indicate on-topic inbound links, meaning that one site has sourced information about a brand from a second site. Sites that are solid gray are blogs and forums, while sites that only have an outline are mainstream media and press releases.

It’s a lot to take in but, basically, it shows that there are a handful of often-linked to and widely-read influential blogs and forums that usually source news from mainstream publications, such as NYTimes.com, WSJ.com, etc., as well as corporate press releases (in this month, the client’s press releases were the most often cited source online– something we see pretty regularly). These influential blogs are then linked to by much smaller, less well-read blogs and forums (what is usually called the “long tail”). To put it in a simplified graphic, the linking relationships usually follow this pattern:

While there are always exceptions, I estimate that 9 out of 10 times, this is what we see when we assess the relationship between mainstream and social media for our clients: mainstream media and press releases drive conversation in influential blogs, which are then linked to and re-purposed by the long tail. Mainstream media is hardly irrelevant. It still seems to instigate conversations in forums and blogs. Bloggers and forum members will certainly edit the content, and provide their own commentary on top of it. But the demise of mainstream and rise of social media isn’t nearly as black and white as many people seem to think.

Given what Context Analytics generally finds for its clients, I would caution PR professionals against throwing out their press releases and mainstream media contacts. Mainstream newspapers might be in financial trouble, and they are certainly reducing the ranks of journalists and the amount of content produced. But, this doesn’t mean that mainstream media is any less important. If you want to influence conversations that are happening in the long-tail, convincing newspapers and magazines to cover you is still going to be your best bet.