The Internet Is Awash in Positive Product Ratings. Here’s How to Decipher the Good from the Great.
Skip to content
Podcast | Insight Unpacked Season 1: Extraordinary Brands and How to Build Them
Marketing Dec 1, 2021

The Internet Is Awash in Positive Product Ratings. Here’s How to Decipher the Good from the Great.

Forget stars and numeric ratings: a review’s language offers better clues to a product’s quality and likely success.

person checking phone for online reviews

Yevgenia Nayberg

Ever had a great meal at a restaurant rated four out of five stars, but then had a lackluster meal at another restaurant with the exact same rating?

Add Insight
to your inbox.

If so, you might have fallen victim to what Kellogg School researchers have dubbed “the positivity problem”: the vast majority of online reviews are positive, but those favorable reviews don’t always translate to real-world quality. “You can have two products with four-and-a-half stars, but they’re not equally good, nor are they equally successful in the marketplace,” explains Derek Rucker, a professor of marketing at Kellogg.

Luckily, the researchers developed another way to help parse apart products that appear similar based on ratings. In a new paper, Rucker—along with coauthors Loran Nordgren, a professor of management and organizations at the Kellogg School, and Matthew D. Rocklage of the University of Massachusetts Boston—finds that the emotionality of the written text in a review is a much better predictor of success than the numeric or star rating that goes with it.

Emotionality is different from valence—whether a feeling is positive or negative—and from extremity—how positive or negative the feeling is. Rather, it’s the extent to which a reaction is rooted in emotion. “‘Awesome’ is a very positive word that also conveys a lot emotion with it,” Rucker explains. “‘Perfect’ is also a very positive word, but it doesn’t have a lot of emotionality.” The same goes for “fantastic” as compared with “perfect,” or “terrible” as compared with “dumb.”

Through computational analysis of hundreds of thousands of online reviews, the researchers discovered that positive star and numerical ratings didn’t reliably correlate with how well products and businesses ultimately fared. But the emotionality of the accompanying written reviews did. (Though rarer, negative ratings are actually more useful; a one-star restaurant is probably both genuinely bad and doomed to failure.)

It’s an important finding for consumers, business owners, and industry experts.

“If you’re looking at ratings, you can figure out what’s a bad restaurant, but you really can’t figure out what’s the best choice for you within a set of good restaurants,” Nordgren says. “Analyzing the language that people use allows us to understand sentiment in a way that the rating system doesn’t reveal.”

Why “Awesome” Movies Outperform “Perfect” Ones

To understand how people people’s emotionality informs success, the researchers started with movie reviews.

They gathered user-generated reviews from the popular rating site Metacritic. For each movie on the site released between 2005–2018, they analyzed numeric ratings, as well as the text, of the first 30 reviews. (These early reactions, the researchers posited, would be less influenced by the eventual success of the film or the opinions of other viewers and professional critics.)

The researchers used a computational tool that coauthor Rocklage developed called the Evaluative Lexicon (EL) to measure the emotionality of the written reviews. The EL scans the text and “distinguishes words that tend to convey more emotion from those that convey less emotion,” Rucker says—essentially, separating the “awesome” from the “perfect”—and it then generates an emotionality score.

The positivity problem is very real, the researchers found: of all the movies in the analysis, a whopping 81 percent got reviews of five or above on a ten-point scale. However, these favorable reviews didn’t necessarily translate into box-office success.

“Within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”

— Derek Rucker

Instead, the emotionality of the positive reviews was a more accurate barometer of a true hit. The EL score of users’ text reviews predicted box-office earnings, a pattern that held true even when controlling for the movie’s genre, budget, length, and release date.

And it’s not just movies. The researchers repeated the analysis for book reviews on Amazon and found the same pattern: for books that garnered at least three stars out of five, more stars didn’t reliably predict higher sales. But more emotionally charged reviews did predict sales.

Was that Bruschetta “Amazing” or Just “Enjoyable”?

Next, the researchers turned their attention to restaurants, using Yelp to gather the first 30 star and text reviews of all restaurants in Chicago in 2017. They also gathered the average number of daily reservations for each establishment on OpenTable, the most popular online reservation service in the U.S.

As with books and movies, almost all Yelp reviews were favorable—92 percent of restaurants received a rating of greater than three out of five stars. But unlike movies and books, these star ratings did correlate with daily reservations. However, the emotionality of reviews as measured by the Evaluative Lexicon was an even better predictor of which restaurants’ tables would fill up.

From movies to restaurants, “at best, the star rating is an inconsistent signal. Sometimes it predicts success, and sometimes it doesn’t,” Rucker says. But “within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”

Harnessing the Power of Online Reviews

In theory, online reviews are supposed to give consumers and industry experts new and more accurate ways to identify the best products.

“The promise of these online platforms is that now we can see and learn from others, and that should have extraordinary value,” Nordgren says. But the overwhelming number of positive reviews has dampened their usefulness, resulting in a missed opportunity. Looking at emotionality is one important way to capitalize “on that unrealized promise.”

Of course, this doesn’t mean business owners should ignore numerical ratings, Rucker points out. “If a consumer just left you a one-star review, that’s important too,” he says. “We aren’t suggesting ignoring reviews, but recognizing they are one piece of information. We should take advantage of the fact the data are much richer.”

But they can’t be the only way to measure success. “If people are giving you a positive review but there’s nothing strongly emotional in their language, they’re not really showing a passion for the brand. Maybe there’s an opportunity to improve, and our tool can help reveal that,” he says. “Emotionality is another piece of data that people should be thinking about and looking at.”

Featured Faculty

Visiting Scholar

Sandy & Morton Goldman Professor of Entrepreneurial Studies in Marketing; Professor of Marketing; Co-chair of Faculty Research

Professor of Management & Organizations

About the Writer

Susie Allen is a freelance writer in Chicago.

About the Research

Rocklage, Matthew, Derek D. Rucker, and Loran Nordgren. 2021. “Mass-Scale Emotionality Reveals Human Behavior and Marketplace Success.” Nature Human Behaviour.

Read the original

Most Popular This Week
  1. Your Team Doesn’t Need You to Be the Hero
    Too many leaders instinctively try to fix a crisis themselves. A U.S. Army colonel explains how to curb this tendency in yourself and allow your teams to flourish.
    person with red cape trying to put out fire while firefighters stand by.
  2. What Triggers a Career Hot Streak?
    New research reveals a recipe for success.
    Collage of sculptor's work culminating in Artist of the Year recognition
  3. What’s the Secret to Successful Innovation?
    Hint: it’s not the product itself.
    standing woman speaking with man seated on stool
  4. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  5. How Much Do Campaign Ads Matter?
    Tone is key, according to new research, which found that a change in TV ad strategy could have altered the results of the 2000 presidential election.
    Political advertisements on television next to polling place
  6. What Went Wrong with FTX—and What’s Next for Crypto?
    One key issue will be introducing regulation without strangling innovation, a fintech expert explains.
    stock trader surrounded by computer monitors
  7. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  8. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  9. How Experts Make Complex Decisions
    By studying 200 million chess moves, researchers shed light on what gives players an advantage—and what trips them up.
    two people playing chess
  10. Yes, Consumers Care if Your Product Is Ethical
    New research shows that morality matters—but it’s in the eye of the beholder.
    woman chooses organic lettuce in grocery
  11. Why Well-Meaning NGOs Sometimes Do More Harm than Good
    Studies of aid groups in Ghana and Uganda show why it’s so important to coordinate with local governments and institutions.
    To succeed, foreign aid and health programs need buy-in and coordination with local partners.
  12. Product Q&A Forums Hold a Lot of Promise. Here’s How to Make Them Work.
    The key to these online communities, where users can ask and answer questions, is how many questions get useful answers.
    man sits at computer reading Q&A forum
  13. What Went Wrong at AIG?
    Unpacking the insurance giant's collapse during the 2008 financial crisis.
    What went wrong during the AIG financial crisis?
  14. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  15. What the New Climate Bill Means for the U.S.—and the World
    The Inflation Reduction Act won’t reverse inflation or halt climate change, but it's still a big deal.
    energy bill with solar panels wind turbines and pipelines
  16. Post-War Reconstruction Is a Good Investment
    Ukraine’s European neighbors will need to make a major financial commitment to help rebuild its economy after the war. Fortunately, as the legacy of the post–World War II Marshall Plan shows, investing in Ukraine's future will also serve Europe's own long-term interests.
    two people look out over a city
  17. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  18. The Political Divide in America Goes Beyond Polarization and Tribalism
    These days, political identity functions a lot like religious identity.
    people engage in conflict with swords
More in Marketing