The Internet Is Awash in Positive Product Ratings. Here’s How to Decipher the Good from the Great.
Skip to content
The Insightful Leader Live: What to Know about Today’s AI—and Tomorrow’s | Register Now
Marketing Dec 1, 2021

The Internet Is Awash in Positive Product Ratings. Here’s How to Decipher the Good from the Great.

Forget stars and numeric ratings: a review’s language offers better clues to a product’s quality and likely success.

person checking phone for online reviews

Yevgenia Nayberg

Ever had a great meal at a restaurant rated four out of five stars, but then had a lackluster meal at another restaurant with the exact same rating?

If so, you might have fallen victim to what Kellogg School researchers have dubbed “the positivity problem”: the vast majority of online reviews are positive, but those favorable reviews don’t always translate to real-world quality. “You can have two products with four-and-a-half stars, but they’re not equally good, nor are they equally successful in the marketplace,” explains Derek Rucker, a professor of marketing at Kellogg.

Luckily, the researchers developed another way to help parse apart products that appear similar based on ratings. In a new paper, Rucker—along with coauthors Loran Nordgren, a professor of management and organizations at the Kellogg School, and Matthew D. Rocklage of the University of Massachusetts Boston—finds that the emotionality of the written text in a review is a much better predictor of success than the numeric or star rating that goes with it.

Emotionality is different from valence—whether a feeling is positive or negative—and from extremity—how positive or negative the feeling is. Rather, it’s the extent to which a reaction is rooted in emotion. “‘Awesome’ is a very positive word that also conveys a lot emotion with it,” Rucker explains. “‘Perfect’ is also a very positive word, but it doesn’t have a lot of emotionality.” The same goes for “fantastic” as compared with “perfect,” or “terrible” as compared with “dumb.”

Through computational analysis of hundreds of thousands of online reviews, the researchers discovered that positive star and numerical ratings didn’t reliably correlate with how well products and businesses ultimately fared. But the emotionality of the accompanying written reviews did. (Though rarer, negative ratings are actually more useful; a one-star restaurant is probably both genuinely bad and doomed to failure.)

It’s an important finding for consumers, business owners, and industry experts.

“If you’re looking at ratings, you can figure out what’s a bad restaurant, but you really can’t figure out what’s the best choice for you within a set of good restaurants,” Nordgren says. “Analyzing the language that people use allows us to understand sentiment in a way that the rating system doesn’t reveal.”

Why “Awesome” Movies Outperform “Perfect” Ones

To understand how people people’s emotionality informs success, the researchers started with movie reviews.

They gathered user-generated reviews from the popular rating site Metacritic. For each movie on the site released between 2005–2018, they analyzed numeric ratings, as well as the text, of the first 30 reviews. (These early reactions, the researchers posited, would be less influenced by the eventual success of the film or the opinions of other viewers and professional critics.)

The researchers used a computational tool that coauthor Rocklage developed called the Evaluative Lexicon (EL) to measure the emotionality of the written reviews. The EL scans the text and “distinguishes words that tend to convey more emotion from those that convey less emotion,” Rucker says—essentially, separating the “awesome” from the “perfect”—and it then generates an emotionality score.

The positivity problem is very real, the researchers found: of all the movies in the analysis, a whopping 81 percent got reviews of five or above on a ten-point scale. However, these favorable reviews didn’t necessarily translate into box-office success.

“Within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”

— Derek Rucker

Instead, the emotionality of the positive reviews was a more accurate barometer of a true hit. The EL score of users’ text reviews predicted box-office earnings, a pattern that held true even when controlling for the movie’s genre, budget, length, and release date.

And it’s not just movies. The researchers repeated the analysis for book reviews on Amazon and found the same pattern: for books that garnered at least three stars out of five, more stars didn’t reliably predict higher sales. But more emotionally charged reviews did predict sales.

Was that Bruschetta “Amazing” or Just “Enjoyable”?

Next, the researchers turned their attention to restaurants, using Yelp to gather the first 30 star and text reviews of all restaurants in Chicago in 2017. They also gathered the average number of daily reservations for each establishment on OpenTable, the most popular online reservation service in the U.S.

As with books and movies, almost all Yelp reviews were favorable—92 percent of restaurants received a rating of greater than three out of five stars. But unlike movies and books, these star ratings did correlate with daily reservations. However, the emotionality of reviews as measured by the Evaluative Lexicon was an even better predictor of which restaurants’ tables would fill up.

From movies to restaurants, “at best, the star rating is an inconsistent signal. Sometimes it predicts success, and sometimes it doesn’t,” Rucker says. But “within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”

Harnessing the Power of Online Reviews

In theory, online reviews are supposed to give consumers and industry experts new and more accurate ways to identify the best products.

“The promise of these online platforms is that now we can see and learn from others, and that should have extraordinary value,” Nordgren says. But the overwhelming number of positive reviews has dampened their usefulness, resulting in a missed opportunity. Looking at emotionality is one important way to capitalize “on that unrealized promise.”

Of course, this doesn’t mean business owners should ignore numerical ratings, Rucker points out. “If a consumer just left you a one-star review, that’s important too,” he says. “We aren’t suggesting ignoring reviews, but recognizing they are one piece of information. We should take advantage of the fact the data are much richer.”

But they can’t be the only way to measure success. “If people are giving you a positive review but there’s nothing strongly emotional in their language, they’re not really showing a passion for the brand. Maybe there’s an opportunity to improve, and our tool can help reveal that,” he says. “Emotionality is another piece of data that people should be thinking about and looking at.”

Featured Faculty

Previously a Visiting Scholar at Kellogg

Sandy & Morton Goldman Professor of Entrepreneurial Studies in Marketing; Professor of Marketing; Co-chair of Faculty Research

Professor of Management & Organizations

About the Writer

Susie Allen is a freelance writer in Chicago.

About the Research

Rocklage, Matthew, Derek D. Rucker, and Loran Nordgren. 2021. “Mass-Scale Emotionality Reveals Human Behavior and Marketplace Success.” Nature Human Behaviour.

Read the original

Most Popular This Week
  1. What Went Wrong at Silicon Valley Bank?
    And how can it be avoided next time? A new analysis sheds light on vulnerabilities within the U.S. banking industry.
    People visit a bank
  2. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  3. What Went Wrong at AIG?
    Unpacking the insurance giant's collapse during the 2008 financial crisis.
    What went wrong during the AIG financial crisis?
  4. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  5. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  6. Podcast: "It's Hard to Regulate U.S. Banks!"
    Silicon Valley Bank spectacularly collapsed—and a new analysis suggests that its precarious situation is not as much of an outlier as we’d hope. On this episode of The Insightful Leader, we learn what went wrong and what should happen next.
  7. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
  8. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  9. Marketers, Don’t Be Too Hasty to Act on Data
    Don’t like the trends you’re seeing? It’s tempting to take immediate action. Instead, consider a hypothesis-driven approach to solving your problems.
    CEO stands before large data wall
  10. Why Well-Meaning NGOs Sometimes Do More Harm than Good
    Studies of aid groups in Ghana and Uganda show why it’s so important to coordinate with local governments and institutions.
    To succeed, foreign aid and health programs need buy-in and coordination with local partners.
  11. Understanding the Pandemic’s Lasting Impact on Real Estate
    Work-from-home has stuck around. What does this mean for residential and commercial real-estate markets?
    realtor showing converted office building to family
  12. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  13. How Much Do Campaign Ads Matter?
    Tone is key, according to new research, which found that a change in TV ad strategy could have altered the results of the 2000 presidential election.
    Political advertisements on television next to polling place
  14. How Peer Pressure Can Lead Teens to Underachieve—Even in Schools Where It’s “Cool to Be Smart”
    New research offers lessons for administrators hoping to improve student performance.
    Eager student raises hand while other student hesitates.
  15. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  16. Leaders, Don’t Be Afraid to Admit Your Flaws
    We prefer to work for people who can make themselves vulnerable, a new study finds. But there are limits.
    person removes mask to show less happy face
  17. For Students with Disabilities, Discrimination Starts Before They Even Enter School
    Public-school principals are less welcoming to prospective families with disabled children—particularly when they’re Black.
    child in wheelchair facing padlocked school doors
  18. Executive Presence Isn’t One-Size-Fits-All. Here’s How to Develop Yours.
    A professor and executive coach unpacks this seemingly elusive trait.
    woman standing confidently
  19. How Self-Reflection Can Make You a Better Leader
    Setting aside 15 minutes a day can help you prioritize, prepare, and build a stronger team
    Self-reflection improves leadership over time
More in Marketing