How Important Is User-Generated Content?

Whether it is a new episode of a TV show or the latest iPhone, companies have long suspected that customer posts to microblogging sites like Twitter can help promote their products. The notion has led companies to offer perks such as early access, free samples, or upgrades for users they think are influential in online communities.

But this is mainly a hunch on the companies’ part.

“Intuitively you might expect that user-generated content has some effect, otherwise why would firms be doing it?” says Song Yao, assistant professor of marketing at the Kellogg School.

These user-created microblogs are very difficult to study. Although a firm can place its own tweets strategically, it is impossible to know—or more importantly, test—the impact of customer tweets. And since most products are promoted in multiple ways, attributing success to traditional ads versus user-generated content can be problematic.

“You can’t say the high ratings for Game of Thrones are a result of users’ tweets—it’s just an intrinsic preference that’s driving both the show’s rating and the Twitter content,” Yao says.

But then a real-world scandal created a clever natural experiment that made it possible to tease apart the impact of user-generated content on a product’s success.

Along with Stephan Seiler at Stanford University and Wenbo Wang at the Hong Kong University of Science and Technology, Yao looked at an unusual situation: What happens to a TV show when user-generated comments disappear because parts of a microblogging platform are temporarily shut down?

From Political Coup to Market Data

In February 2012, Wang Lijun, vice mayor of a city in southwest China, revealed details of a businessman’s murder and subsequent cover-up to the United States Consulate in China. The juicy scandal—filled with stories of defection, official corruption, and political fallout—fueled rumors that spread rapidly via Sina Weibo, a Chinese microblogging platform similar to Twitter. To stem gossip, the government placed a partial block on Sina Weibo. All user-generated comments—whether about the scandal or not—were shut down for three days.

So, for example, a television channel could post that a show would air at a particular time, and a user might retweet that, but users could not add that they would be watching or that they were excited about a particular episode.“TV shows were totally irrelevant to the scandal but were affected as well,” Yao says. “There was much less microblogging activity during those three days.”

This presented a perfect opportunity to study the impact of user-generated posts on TV ratings.

The researchers tracked content created by Sina Weibo users before, during, and after the block across 24 mainland cities and Hong Kong. At the same time, they observed the ratings of 166 TV shows and analyzed how the censorship affected viewership ratings.

Microblogging’s Power

The Weibo blackout had a significant effect.

Shows with a normally heavy Weibo presence took the biggest hit. Episodes of those shows had significantly lower ratings during the three-day block compared with their normal viewership levels. These same shows were largely unaffected before or after the block—only episodes that aired during the censored Sina Weibo window were impacted.

Cities with higher proportions of Weibo users also reported greater drops in episode ratings. Unlike the mainland cities, Hong Kong, which was not subject to the censorship and has more Twitter than Sina Weibo users, saw no major impacts from the block.

“What we found is that ratings only dropped in the cities with high Weibo penetration and only dropped for shows with high Weibo activity,” Yao says.

The researchers also wanted to control for another possible variable.

“What if people’s attention was actually diverted to the scandal, and they were no longer interested in watching TV at that point because the scandal was far more interesting?” Yao says.

To eliminate that explanation, the team used the Baidu search index, a metric similar to Google trends, to measure people’s interest in the political scandal. Search activity related to the scandal, for example for the term “Wang Lijun,” spiked at three key moments: when the news broke, when his superior, Bo Xilai, was removed from office, and when Bo was arrested and charged with bribery and corruption.

But the Sina Weibo block did not coincide with any of these events, so it was unlikely people were simply focused on the scandal instead of new episodes, says Yao. “The evidence is quite convincing.”

Surprising Impact

The researchers then quantified the impact of user-generated posts on ratings.Increasing the amount of user-generated content by 100 percent (such as going from none to restoring all of it, as happened during and after the censoring event) increased viewership ratings by 2 percent.

“This seems to be a fairly small number, but it’s actually a significant impact,” Yao says. “If there’s nothing else going on, the fluctuation in ratings from episode to episode is about 9 percent, so the ‘Weibo effect’ accounts for roughly 20 percent of the normal fluctuations.”

These findings are particularly exciting because it is usually very difficult to quantify the success of user-generated content. Most marketing firms normally look to classic A/B tests to try out different pitches. But that is impossible to do with user-generated tweets.

“Without the evidence from this natural experiment, I could never make this conclusion about user-generated content with such certainty,” Yao says. “I didn’t expect the effect to be this large. In that sense, our study confirms that this is a very impactful and effective marketing method.”

Leveraging User Tweets

The results extend well beyond a single episode of a TV show, even if it is Game of Thrones.

Like traditional ads, user-generated tweets seem to work in two ways: by providing information and by persuading others to share the poster’s excitement. Firms could leverage each dimension of tweets in different ways, Yao suggests. And because the team found that user-generated content before a show had a significant impact on ratings—more so, in fact, than tweets during the show—their findings can be applied to traditional consumer advertising.

For the launch of a new product, engaging users to share information, possibly by offering early access or trial versions to more influential consumers, can boost awareness.

“In the case of a stable product, let’s say Coke or Pepsi, the tweet itself might contain little information,” he says. “But firms can work on the other dimension: How can I persuade consumers in the community to consume this product?”

“We suspect tweeting will work more effectively in a category that’s fast moving. If there’s a new product, or a movie, or a daily deal like a Groupon, social media can help disseminate information quickly,” he adds.

Regardless of the product, having better monitors of user-generated content and sentiment can help marketing efforts, Yao says. “Firms should think about how to create an active community and how to encourage the creation of user-generated content.”