ABSTRACT
The rise of social media as a marketing channel, and the research that has supported it, has left open questions as to its impact on actual brand performance. The current authors sought to fill the gap of knowledge on the relationship between social media and real-world conversations and outcomes for brands. Building on a decade's worth of research, they used four key metrics—volume, sentiment, sharing, and influence—to study the potential for correlations between online and offline conversations about brands. There were, in fact, almost no correlations, which suggests the need for marketers to develop separate digital and offline social influence strategies.
MANAGEMENT SLANT
Social media do not provide a clear window into the entire consumer marketplace.
Social media signals produced by listening systems must be observed with caution.
Although some brands perform well both online and offline, most perform well in only one channel or the other.
Correlations between online and offline conversations are particularly weak when evaluated over time, which indicates that improved brand performance in social media infrequently corresponds to improvements offline.
The online and offline worlds behave like separate ecosystems, with their own unique features and characteristics. Marketers thus should avoid making the assumption that a social media strategy can substitute for a broader social influence strategy.
INTRODUCTION
It has been more than a decade since the launch of Facebook and Twitter in 2004 and 2006, respectively. The novelty of social media may be over. Yet, these massive platforms—along with such emerging contenders as Snapchat, Swarm, and Yik Yak—continue to fascinate the public, the news media, and marketers given their influence on interpersonal relationships, culture, and politics.
One of social media's key attractions to marketers and researchers has been the prospect of providing access to authentic conversations and opinions of consumers, as both a marketing channel and a research platform. Marketers have opportunities to amplify and measure consumer conversations on social media platforms, with an assumption that these conversations, in turn, will drive positive business outcomes.
Key questions remain top of mind:
Does the visible conversation happening in social media accurately reflect consumer conversations about brands more generally?
Is digital word of mouth a mirror onto the harder-to-measure conversation happening offline, through face-to-face conversation at home, at work, and during our social interactions with friends?
If consumer conversation is an iceberg, then social media is the visible “tip” above the ocean surface, whereas offline conversation is the largely invisible portion that lies beneath the surface. In this analogy, can the social media tip of the iceberg predict what is a much larger conversation happening in real life?
The answer to all three questions, according to a growing body of research in this area, is no. Although digital channels “deliver scale and speed that make each WOM event potentially more influential than the events ever have been…not all WOM events… are created equal, and the better a brand understands the ways in which each relevant channel can influence the consumer, the more likely the brand will harness its power” (Fulgoni and Lipsman, 2015, p. 21).
Many companies continue to rely on social media to evaluate the effectiveness of marketing campaigns, to gauge feedback to new products and services, and to measure brand performance in general. Some—Chobani, Coca-Cola, Lean Cuisine, and Oscar Mayer, to name a few—have claimed success and won recognition for their work (Advertising Research Foundation, 2016; Fulgoni, 2015).
Social media are an important and growing influence and ought to be incorporated into many brands' marketing strategies. Brand performance in social media can predict sales (WOMMA, 2014), but does social influence online correspond to offline influence, or do these forms of peer-to-peer influence operate independently of each other? If they are independent, marketers need to have separate strategies for stimulating and supporting social influence in these two channels.
The current researchers studied correlations between week-by-week social media and offline word-of-mouth conversations about more than 500 brands during a 12-month period. They found that although there was a modest correlation between online and offline conversation volume, such correlations were too infrequent to be relied on by researchers and marketers. Correlations were near zero for other metrics, such as sentiment, sharing of brand content, and engagement by influencers.
The authors believe their work adds further evidence as to why online social media conversations are not, in themselves, entirely reliable for predicting brand performance. The implication for marketers and for researchers is that if marketers rely on social media conversation alone, they are listening to only a part of what consumers are saying about a brand, and it is misguided to draw conclusions on the basis of this as to consumer conversation writ large.
In practice, marketers should assume the online and offline conversation channels work independently of each other, and they should formulate strategies and tactics optimized to the strengths of each. This is not to say that marketers who are successful in social media cannot apply lessons learned to offline conversation, and vice versa. One should not be treated as a substitute for the other, however.
LITERATURE REVIEW AND HYPOTHESIS DEVELOPMENT
Scholarship on social influence dates back more than six decades, and many key principles still apply today. The “two-step flow of communication” has been a powerful influence on product success (Katz and Lazarsfeld, 1955; Silverman, 2001), with the term “word-of-mouth advertising” dating back to a 1966 Harvard Business Review article (Dichter, 1966). The ability to generate word of mouth often is viewed as an important way to judge advertising effectiveness (Hogan, Lemon, and Libai, 2004; Keller and Fay, 2012b), and understanding and strategically managing word of mouth is vital for advertisers (Craig, Greene, and Versaci, 2015).
In 2004, the Word of Mouth Marketing Association (WOMMA) was founded in response to a sense among market participants that a true industry was forming and would need to be promoted through thought leadership and advocacy. Social media were one of the channels that fit under the WOMMA umbrella, even though the group formed before 2006, when Facebook was opened to everyone and Twitter launched.
In 2006, Keller Fay Group (cofounded by one of the current authors, and in 2015 purchased by social media analytics firm Engagement Labs) launched a commercial research initiative to measure offline consumer conversations about brands on a continuous basis over an indefinite period. The study was inspired by a growing belief among marketers that a budding word-of-mouth industry would require a standard measurement system that covered all forms of communications, online and offline. The research was based on an ongoing weekly online survey of a cross-section of U.S. consumers, for an annual sample of 36,000 people ages 13 to 69 years. (In 2011, the firm started a similar study in the United Kingdom.)
The survey, which continues in 2017, is based on samples representative of the intended populations through the use of sample quotas and weighting for age, gender, educational attainment, race, ethnicity, and geography. Among other results, the research team found that 76 percent of all brand conversations occurred face to face. Of the total, 16 percent of the conversations took place over the phone, and 7 percent were conducted via some digital means (Keller and Libai, 2009). The results were presented at the 2009 ESOMAR Worldwide Multi-Media Measurement Conference.
A few years later, a trio of academics from the United States and Israel systematically compared the Keller Fay offline word-of-mouth data to third-party social media listening data (Lovett, Peres, and Shachar, 2013). They found significant differences in the categories and brand characteristics discussed online versus offline and also in the motivations for brand conversations in the two venues. They also found that “social signaling,” or seeking to boost one's own self-image, was the top motivator for sharing online, whereas emotional sharing was the top motivator offline. Additional studies also find that communication modality has an impact on what is discussed, with “written” communication focusing on more “interesting” topics (Berger and Iyengar, 2011).
In 2014, WOMMA took another step to assess online and offline consumer-conversation data about brands, enlisting the collaborative work of multiple marketing and analytics firms. WOMMA hired New York City-based Sequent Partners and Analytic Partners to undertake a marketing mix-modeling study in partnership with six brands that shared sales and marketing expenditure data. The study also integrated Keller Fay's offline data, plus online social media data provided by the New York social media analysis firm Converseon.
The structural equation models showed that the combination of online and offline conversations explained between 5 percent and 25 percent of sales variance for the six modeled brands. Analytic Partners moreover found that offline conversations accounted for an average of two thirds of the total social influence, and social media accounted for one third. The study also suggested that word of mouth works as an amplifier to advertising, with one third of social influence being related to paid advertising, a mechanism by which some advertising works.
The authors of the current article analyzed the extent to which social performance online is predictive of performance offline, and vice versa, using data from January through December 2016. Their efforts were intended to update the work of Lovett et al. (2013), which was based on 2008–2010 data and focused on volume among the conversation metrics. The current researchers additionally increased the breadth of measures to include sentiment (or valence), because positive and negative conversations about a brand would be expected to influence sales in opposite directions. The current research also expanded Lovett et al.'s (2013) work in terms of the statistical methods used, by correlating online and offline weekly trends.
In undertaking this work, the authors suggested that the wider adoption of social media by consumers in recent years might have increased the representativeness of opinions expressed in social media. If that were the case, then one might expect to see a greater correlation between online and offline brand conversations and recommendations—beyond what prior researchers have found. The authors thus hypothesized the following:
H1a: Brands that perform well in social media conversation volume also will perform well in offline conversations, and those that perform poorly in one will do so in the other.
H1b: Changes in brand conversation volume in social media will correspond to similar changes for those brands in offline conversations.
Drawing from previous scholarship, the authors suggested that some metrics related to online and offline conversation might have stronger correlations than others. The research team tested four different conversation metrics:
volume;
sentiment (valence);
brand sharing;
influence.
Sentiment is a commonly cited metric, widely used for social listening and found to be predictive of brand performance in some studies (Schweidel and Moe, 2014). The rise of content marketing in social media is a relatively new phenomenon that raises the question of the degree to which people are talking about and sharing brand-sourced content (Jutkowitz, 2014). The current study thus included brand sharing. Finally, the influence metric incorporates a central word-of-mouth and social media strategy popularized in The Tipping Point by Malcolm Gladwell (2000) and expanded on by Ed Keller and Jon Berry (2003) in The Influentials: How One American in Ten Tells the Other Nine How to Vote, Where to Eat, and What to Buy.
H2a: Brands that perform well online in terms of sentiment, brand-content sharing, or influence also will perform well offline for those metrics, and vice versa.
H2b: Changes in brand performance for the metrics of sentiment, brand-content sharing, and influence will correspond to similar changes offline, and vice versa.
To ensure that the findings would be generalizable, the researchers tested correlations for a large number of brands across a diverse set of product and service categories.
METHODOlOGY
The current researchers in early 2016 systematically began to combine offline consumer conversation data and online listening data for 500 U.S. brands and 350 U.K. brands, with three objectives. They wished to determine
the relationship between online and offline conversation;
the degree to which both channels are predictors of brand sales;
the estimated contribution to sales of each of the key metrics measured online and offline.
The focus of this article is on the first of these objectives, the relationship between online and offline conversation.
Data Sources
The researchers analyzed online and offline conversation data on 500 U.S. brands distributed across 16 diverse product and service categories, including food, beverages, telecommunications, electronics, beauty, financial services, travel, and sports. Major brands included Apple, AT&T, Coca-Cola, ESPN, McDonald's, Pepsi, Verizon, and Walmart (See Appendix).
Offline data for the 500 brands came from the aforementioned continuing survey method (“TalkTrack®”), which asks respondents to record the categories and brands they talked about on the day prior to taking the survey. For this analysis, data for the full calendar year 2016 were utilized. Online social media data for the same time period came from a Toronto-based online-listening service, Sysomos, which used Boolean keyword queries to identify mentions of the 500 brands across Twitter, online blogs, and newsgroups. The queries themselves were developed by Engagement Labs and tested for accuracy.
Data Transformation
A key challenge when researchers directly compare online and offline conversations and sharing about brands is that different types of data are available online versus offline. The most basic metric is volume—the amount of conversation obtained by each brand. In social media, for example, the most available kind of volumetric data is the number of online “mentions” of a brand, without any reference to how many people have been exposed to it. Offline data, however, are more analogous to “impressions,” in that the surveys measure the number of persons exposed each day to a conversation about a brand. Not only are the metrics different, but so are the values themselves; offline conversation, once projected to the total population, typically is at volumes ten times as great as online mentions.
Because of the differences between online and offline data, the current researchers' scoring methodology placed every brand's monthly performance on any given metric on a scale of 0 to 100, using both the brand's rank and its distribution among 500 scored brands. This is a scoring method similar to what is available for GoogleTrends search-term data. The scoring allowed both online and offline data to be evaluated with the same 0–100 scale.
Metrics
Each brand was scored on the basis of the same four metrics, both online and offline, which yielded four pairs of metrics (eight in total). These metrics were as follows:
Volume: Each brand's share of conversation among the 500 brands, online and offline.
Net sentiment: The share of conversation that was positive about the brand, minus the negative share.
Brand sharing: The total amount of sharing of a brand's marketing content, either digitally in social media or in conversation about the brand's advertising and marketing.
Influence: The volume of conversation among key influencers, indexed against the volume of conversation among all persons. This metric indicated whether the brand was performing above expectation among those who had the largest online and offline social networks and sharing habits.
Each of the brands in this study was measured in terms of the four pairs of metrics described above—plus two composite metrics representing the combined “total online” and “total offline” performance of each brand, based on a simple averaging of the scores across the four metrics. Each brand thus was measured on ten dimensions, including the two composite scores.
RESULTS
Volume Correlation across Brands
Conversation volume is a critical metric for marketers. The more conversations and recommendations that a brand attracts, the greater is the brand's potential reach through word of mouth. Just as marketers pay for advertising on the basis of the size of audience or number of impressions they can expect, marketers should value the number of impressions they get through word of mouth.
For this reason, volume is a good place to start in comparing online and offline conversation data for brands. There are two questions researchers should ask when comparing online and offline conversation volume for brands:
Do brands that get a lot of online conversation also get a lot of conversation offline?
Are trends in online volume similar to trends in offline volume—or, put another way, does rising conversation online correspond to rising conversation offline?
For the first test, the researchers analyzed all 500 brands collectively every month for an entire year. The brands were plotted on a scatter chart with a vertical axis based on the researchers' 0–100 online volume score for the 500 brands and a horizontal axis for the offline volume score for those same brands (See Figure 1).
The research team calculated the Pearson correlation between the online and offline volume of a brand at 44 percent, which suggests there was a relationship, but not a determinative one. Many brands had similar scores online and offline, but many had very different scores, too. Among them, for example, McDonald's scored high both online and offline, whereas Spirit Airlines performed poorly both online and offline. Aveeno performed well offline but not online; Beats Electronics performed well online but not offline. For Beats Electronics and Aveeno, the tip of the iceberg looked nothing like the invisible portion beneath the surface.
One can imagine a number of reasons why brands might have similar volume scores online and offline. To the extent that conversation is related to market penetration, to purchase or consumption frequency, or to advertising expenditures, brands might be ranked similarly in terms of conversation volume online and offline. Other factors work against such correlations, however.
On the one hand, brands that are more fashionable or innovative can perform better online than offline because of the desire of social media users to send signals about being “in the know” (Lovett et al., 2013). On the other hand, everyday products that provide value but not a lot of sexiness—over-the-counter drugs, children's products, cleaning products, and so forth—have a tendency to perform better offline than online. Another factor may be the age of the brand's target market, given that social media still have a more youthful skew compared to the general population (Kirk, Chiagouris, Lala, and Thomas, 2015).
Volume Correlations across Time
A key measurement objective for marketers is observing change over time. Change can be related to underlying long-term trends, to short-term events, or in response to specific marketing campaigns. Increases or decreases in conversation level can be a key indicator of whether a brand's marketing is working or whether a competitor is gaining. Apart from whether a brand is ranked similarly online and offline, therefore, it is important to know whether brands tend to move in similar directions over time.
The second statistical test of the current study, therefore, was to estimate the correlation of time-series data for both online and offline conversations for each of the 500 brands. The researchers looked at weekly data for the entire calendar year 2016, which meant there were 52 observations for each of the 500 brands. The findings were very mixed. They followed a normal “bell curve” distribution with a mean near zero, which indicates that measured correspondence between online and offline conversations often was driven by chance (See Figure 2).
In 58 percent of cases, there was a positive Pearson correlation, but in only 31 percent were those correlations greater than 25 percent. In 41.3 percent of cases, there was a negative correlation, including 14.2 percent of cases when those correlations were greater than −25 percent. There clearly were more positive than negative correlations, and the average correlation was 8.5 percent. For all practical purposes, however, the association was not sufficient that a brand marketer reasonably could assume that one was a mirror to the other. When it comes to measuring changes in volume, the social media “tip of the iceberg” does not reveal much about offline conversation that is happening under the surface.
Online and Offline Volume Scores for 500 Brands In October 2016
Distribution of Offline to Online Correlations For Volume
To illustrate the finding further, note that among six brands studied, there were few similarities in outcomes (See Figure 3). Amazon performed similarly online and offline, with a fairly high correlation (51 percent) between online and offline conversation volume for the 12-month period. In September 2016, however, there was a big increase in offline conversation that was not picked up online.
At the opposite extreme was Colgate, a brand that performed much better offline than online. For Colgate, the trend online was the reverse of the trend offline, which produced a negative correlation (−58 percent). More typical outcomes were observed in such brands as Hilton (31 percent), Adidas (25 percent), Kellogg's (23 percent), and Microsoft (10 percent). For these brands, there was a modestly positive correlation in the trends between online and offline conversation.
Net Sentiment Correlations
If the volume–performance relationship between online and offline conversation was modest, then the correlation for sentiment was close to negligible. Net sentiment was based on the percentage of sentiment that was positive minus the percentage that was negative. For this variable, the researchers undertook the same two statistical tests as were conducted in the volume segment.
In a scatter plot for net sentiment, the Pearson correlation was 26 percent, a modest level of correlation (See Figure 4). On a trended basis, however, for the 52 weeks in 2016, there was a positive correlation of less than 1 percent—in other words, no meaningful correlation at all.
To be sure, there were some isolated cases when sentiment correlations did exist. Toys R Us had a correlation of 44 percent, based on similarly improving sentiment with the approach of the back-to-school and holiday shopping seasons last year (See Figure 5). Heading in the opposite direction, at a 46 percent level of correlation, was Samsung, for which sentiment declined both online and offline in response to a product recall over phone batteries exploding.1
It is quite possible that crisis situations can be predictive of alignment between online and offline sentiment trends. Beyond the Samsung example, consider Wells Fargo & Co., the U.S. bank involved in a scandal of meeting performance targets by setting up fake customer accounts in late 2016.2 Both online and offline conversations experienced similar declines in net sentiment. The full-year correlation was just 9 percent, however, largely because of a big jump in online sentiment in March 2016 in response to the East Regionals of the NCAA March Madness tournament being held in the Wells Fargo Center in Philadelphia.3 There was no similar response offline.
Brand Sharing and Influence
The two other metrics included in this analysis, brand sharing and influence, showed even less correlation between online and offline conversations. Online brand sharing is based on the frequency with which people are sharing branded content via a brand's social pages (e.g. Twitter, Facebook, YouTube, Instagram). Offline brand sharing is the percentage of the brand's conversations that contain a discussion of the brand's advertising and marketing, such as television and radio commercials, print advertisements, displays at retail, outdoor advertisements, and digital marketing. Both are based, to some degree, on the effort of the marketer.
This may be one reason why, in this study, there was some correlation on an overall basis (See Figure 6). For the month of October 2016, the researchers observed a 23 percent correlation between the online and offline brand-sharing performances. On a weekly-trend basis throughout 2016, however, the correlation was essentially zero: 0.4 percent. The authors thus concluded that consumer engagement with brands' marketing content works entirely independently offline and online.
Online–Offline Volume Correlations for Six Brands
Online and Offline Net Sentiment Scores for 500 Brands in October 2016
The final metric, influence, is a measure of the degree to which the brand has better conversation performance among the most influential consumers compared with average people:
For online, influencers were identified as Twitter users with a greater than average “authority” score—a metric available in the Social Media Management and Analytics Software (Sysomos) platform—on the basis of the number and engagement of their followers.
For offline, influencers were the top 10 percent of consumers who had large real-world social networks and regularly recommended products, using a “Talk-Track” segment known as Conversation Catalysts™. They have been shown to have a greater-than-average impact on peers (Keller and Fay, 2009; Keller and Libai, 2009), particularly in accelerating new-product adoption (Libai, Muller, and Peres, 2013).
It turns out there was literally zero correlation between these two types of influencers in the aggregate, and a slightly negative correlation (−2.3 percent) over time (See Figure 7). It is fair to say that influencer-marketing strategies for brands need to be considered entirely independently from each other, the current authors suggest.
WOMMA has published guidance on influencer marketing, specifically describing a wide variety of strategies and definitions for influencer marketing. In 2014, WOMMA wrote, “Not all influencers are created equal. We have identified five distinct categories of key influencers that brands can identify and engage in their influencer marketing programs. Each category of influence demands a specific program and measurement technique while the type of influence each exhibits is distinct by audience” (WOMMA, 2014). The differences between online and offline influencers in the current research effort were consistent with this guidance.
DISCUSSION
Word of mouth has been an area of academic and marketing interest for more than half a century (Keller and Fay, 2012a). It always has been the case that people often rely on each other for advice on where to get a good meal, means of travel, and how to entertain themselves, for example. It took many marketers until the advent of social media, however, to become serious about tapping into human social relationships to grow their brands.
Although social media have helped to increase advertiser interest in social influence, advertisers' experiences in online social media likely will be different from what they experience in attempting to leverage social influence offline. This article tested several hypotheses relating to the possibility that online social media have become a proxy for measuring offline conversation about brands. Each of the four stated hypotheses (H1a, H1b, H2a, and H2b) was disproven, because of the extremely low levels of correlation between the online and offline changes in conversation metrics over the course of a year.
The findings add further evidence to the study of brand performance online and offline as very different phenomena (Lovett et al., 2013). Both are important to brand success, as was proven by the 2014 WOMMA study, but brands rarely earn the same level of success both online and offline. This suggests that brands need to embrace a strategy that deliberately fosters both online and offline social sharing and recommendations.
Future research should analyze more deeply the interplay of word of mouth and social media within more complete sets of brand-owned data for inputs. These inputs could include paid, earned, and owned marketing activities; pricing; and promotion. Researchers also could examine a broader range of business outcomes, such as new customer acquisition, retention, foot traffic, and brand preference.
Examples of Sentiment Correlations for Three Brands
Online and Offline Brand Sharing Scores In October 2016
Online and Offline Influencer Scores
The current article begins with the analogy of an iceberg and the question of whether the visible social media “tip” is predictive of the offline conversation below the surface. This research demonstrates that the answer is no.
Another apt metaphor is based on the concept of ecosystems. Online and offline conversations occur in two vastly different ecosystems, as different as ocean from desert and tropical from temperate. Most marketers can find success in both ecosystems, but they will need careful preparation. Success in one ecosystem does not translate automatically into the other. Only a marketer prepared with a good map, a smart strategy, and the right gear can expect to thrive in both the online and the offline ecosystems.
ABOUT THE AUTHORS
Brad Fay is chief commercial officer at Engagement Labs, a Canadian social media data-analytics firm with offices in New Brunswick, NJ. Engagement Labs in 2015 acquired Keller Fay Group, which the author cofounded. Fay specializes in measuring peer influence and consumer conversations, both online and offline. He is coauthor of The Face-to-Face Book: Why Real Relationships Rule in a Digital Age (Free Press, 2012) and has contributed to the Journal of Advertising Research and Admap, among other publications. A recipient of the Advertising Research Foundation Grand Innovation Award, Fay is past president of the Market Research Council and former chairman of the Word of Mouth Marketing Association.
Rick Larkin is vp of analytics at Engagement Labs. Previously, he was svp of strategy at the marketing-analytics firm MarketShare, a Neustar Solution, in New York City.
APPENDIX Product Categories and Examples of Brands Tested
Auto
BMW
Jeep
Porsche
Beauty and personal care
Aveeno
Colgate
Neutrogena
Beverages
Coca-Cola
Canada Dry
Children's products
Toys R Us
Pampers
Dining
McDonald's
Starbucks
Dominos
Financial
Wells Fargo
Charles Schwab
Food
Kellogg's
Nestlé
Health
CVS
Aleve
Home
Ikea
Ashley Furniture
Household products
Clorox
Keurig
Media
Twitter
ESPN
Disney Channel
Retail and apparel
Adidas
Banana Republic
Sports
NASCAR
NFL
Technology
Beats Electronics
Microsoft
Telecommunications
Sprint
Boost
Travel services
Hilton
Spirit Airlines
Kayak
Footnotes
↵1 “Samsung to Recall Galaxy Note 7 Smartphone over Reports of Fires,” The Wall Street Journal, September 2, 2016. Retrieved March 16, 2017, from https://www.wsj.com/articles/samsung-to-recall-galaxy-note-7-smartphone-1472805076.
↵2 “Wells Fargo to Pay $185 Million Fine over Account Openings,” The Wall Street Journal, September 8, 2016. Retrieved March 7, 2017, from https://www.wsj.com/articles/wells-fargo-to-pay-185-million-fine-over-account-openings-1473352548.
↵3 “March Madness: $18.2M Economic Impact for Philadelphia Region,” March 23, 2016. Retrieved March 7, 2017, from Bizjournals.com website: http://www.bizjournals.com/philadelphia/news/2016/03/23/march-madness-villanova-economic-impact-on-philly.html.
- © Copyright 2017 The ARF. All rights reserved.
Vol 57 Issue 2
