Marketing Dec 1, 2021
The Internet Is Awash in Positive Product Ratings. Here’s How to Decipher the Good from the Great.
Forget stars and numeric ratings: a review’s language offers better clues to a product’s quality and likely success.
Ever had a great meal at a restaurant rated four out of five stars, but then had a lackluster meal at another restaurant with the exact same rating?
to your inbox.
If so, you might have fallen victim to what Kellogg School researchers have dubbed “the positivity problem”: the vast majority of online reviews are positive, but those favorable reviews don’t always translate to real-world quality. “You can have two products with four-and-a-half stars, but they’re not equally good, nor are they equally successful in the marketplace,” explains Derek Rucker, a professor of marketing at Kellogg.
Luckily, the researchers developed another way to help parse apart products that appear similar based on ratings. In a new paper, Rucker—along with coauthors Loran Nordgren, a professor of management and organizations at the Kellogg School, and Matthew D. Rocklage of the University of Massachusetts Boston—finds that the emotionality of the written text in a review is a much better predictor of success than the numeric or star rating that goes with it.
Emotionality is different from valence—whether a feeling is positive or negative—and from extremity—how positive or negative the feeling is. Rather, it’s the extent to which a reaction is rooted in emotion. “‘Awesome’ is a very positive word that also conveys a lot emotion with it,” Rucker explains. “‘Perfect’ is also a very positive word, but it doesn’t have a lot of emotionality.” The same goes for “fantastic” as compared with “perfect,” or “terrible” as compared with “dumb.”
Through computational analysis of hundreds of thousands of online reviews, the researchers discovered that positive star and numerical ratings didn’t reliably correlate with how well products and businesses ultimately fared. But the emotionality of the accompanying written reviews did. (Though rarer, negative ratings are actually more useful; a one-star restaurant is probably both genuinely bad and doomed to failure.)
It’s an important finding for consumers, business owners, and industry experts.
“If you’re looking at ratings, you can figure out what’s a bad restaurant, but you really can’t figure out what’s the best choice for you within a set of good restaurants,” Nordgren says. “Analyzing the language that people use allows us to understand sentiment in a way that the rating system doesn’t reveal.”
Why “Awesome” Movies Outperform “Perfect” Ones
To understand how people people’s emotionality informs success, the researchers started with movie reviews.
They gathered user-generated reviews from the popular rating site Metacritic. For each movie on the site released between 2005–2018, they analyzed numeric ratings, as well as the text, of the first 30 reviews. (These early reactions, the researchers posited, would be less influenced by the eventual success of the film or the opinions of other viewers and professional critics.)
The researchers used a computational tool that coauthor Rocklage developed called the Evaluative Lexicon (EL) to measure the emotionality of the written reviews. The EL scans the text and “distinguishes words that tend to convey more emotion from those that convey less emotion,” Rucker says—essentially, separating the “awesome” from the “perfect”—and it then generates an emotionality score.
The positivity problem is very real, the researchers found: of all the movies in the analysis, a whopping 81 percent got reviews of five or above on a ten-point scale. However, these favorable reviews didn’t necessarily translate into box-office success.
“Within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”
— Derek Rucker
Instead, the emotionality of the positive reviews was a more accurate barometer of a true hit. The EL score of users’ text reviews predicted box-office earnings, a pattern that held true even when controlling for the movie’s genre, budget, length, and release date.
And it’s not just movies. The researchers repeated the analysis for book reviews on Amazon and found the same pattern: for books that garnered at least three stars out of five, more stars didn’t reliably predict higher sales. But more emotionally charged reviews did predict sales.
Was that Bruschetta “Amazing” or Just “Enjoyable”?
Next, the researchers turned their attention to restaurants, using Yelp to gather the first 30 star and text reviews of all restaurants in Chicago in 2017. They also gathered the average number of daily reservations for each establishment on OpenTable, the most popular online reservation service in the U.S.
As with books and movies, almost all Yelp reviews were favorable—92 percent of restaurants received a rating of greater than three out of five stars. But unlike movies and books, these star ratings did correlate with daily reservations. However, the emotionality of reviews as measured by the Evaluative Lexicon was an even better predictor of which restaurants’ tables would fill up.
From movies to restaurants, “at best, the star rating is an inconsistent signal. Sometimes it predicts success, and sometimes it doesn’t,” Rucker says. But “within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”
Harnessing the Power of Online Reviews
In theory, online reviews are supposed to give consumers and industry experts new and more accurate ways to identify the best products.
“The promise of these online platforms is that now we can see and learn from others, and that should have extraordinary value,” Nordgren says. But the overwhelming number of positive reviews has dampened their usefulness, resulting in a missed opportunity. Looking at emotionality is one important way to capitalize “on that unrealized promise.”
Of course, this doesn’t mean business owners should ignore numerical ratings, Rucker points out. “If a consumer just left you a one-star review, that’s important too,” he says. “We aren’t suggesting ignoring reviews, but recognizing they are one piece of information. We should take advantage of the fact the data are much richer.”
But they can’t be the only way to measure success. “If people are giving you a positive review but there’s nothing strongly emotional in their language, they’re not really showing a passion for the brand. Maybe there’s an opportunity to improve, and our tool can help reveal that,” he says. “Emotionality is another piece of data that people should be thinking about and looking at.”
Susie Allen is a freelance writer in Chicago.
Rocklage, Matthew, Derek D. Rucker, and Loran Nordgren. 2021. “Mass-Scale Emotionality Reveals Human Behavior and Marketplace Success.” Nature Human Behaviour.
Your Team Doesn’t Need You to Be the HeroToo many leaders instinctively try to fix a crisis themselves. A U.S. Army colonel explains how to curb this tendency in yourself and allow your teams to flourish.
What Triggers a Career Hot Streak?New research reveals a recipe for success.
What’s the Secret to Successful Innovation?Hint: it’s not the product itself.
Which Form of Government Is Best?Democracies may not outlast dictatorships, but they adapt better.
How Much Do Campaign Ads Matter?Tone is key, according to new research, which found that a change in TV ad strategy could have altered the results of the 2000 presidential election.
What Went Wrong with FTX—and What’s Next for Crypto?One key issue will be introducing regulation without strangling innovation, a fintech expert explains.
How Are Black–White Biracial People Perceived in Terms of Race?Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
Immigrants to the U.S. Create More Jobs than They TakeA new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
How Experts Make Complex DecisionsBy studying 200 million chess moves, researchers shed light on what gives players an advantage—and what trips them up.
Yes, Consumers Care if Your Product Is EthicalNew research shows that morality matters—but it’s in the eye of the beholder.
Why Well-Meaning NGOs Sometimes Do More Harm than GoodStudies of aid groups in Ghana and Uganda show why it’s so important to coordinate with local governments and institutions.
Product Q&A Forums Hold a Lot of Promise. Here’s How to Make Them Work.The key to these online communities, where users can ask and answer questions, is how many questions get useful answers.
What Went Wrong at AIG?Unpacking the insurance giant's collapse during the 2008 financial crisis.
When Do Open Borders Make Economic Sense?A new study provides a window into the logic behind various immigration policies.
What the New Climate Bill Means for the U.S.—and the WorldThe Inflation Reduction Act won’t reverse inflation or halt climate change, but it's still a big deal.
Post-War Reconstruction Is a Good InvestmentUkraine’s European neighbors will need to make a major financial commitment to help rebuild its economy after the war. Fortunately, as the legacy of the post–World War II Marshall Plan shows, investing in Ukraine's future will also serve Europe's own long-term interests.
How Has Marketing Changed over the Past Half-Century?Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
The Political Divide in America Goes Beyond Polarization and TribalismThese days, political identity functions a lot like religious identity.