Leveling Up the Helpfulness of Online Reviews

Restaurants get five-star reviews on Yelp for quality food and service. Popular videos on YouTube and TikTok get likes. Even service providers like medical doctors, real-estate agents, and plumbers get reviewed on websites like Google and Trustpilot. 

Online reviews have an obvious benefit, says Nalin Shani, a PhD student at the Kellogg School; “they give you an opportunity to learn from other people’s experience and judge whether it will be good for you to make a purchase or not.” 

Reviews matter to companies, too. They are a critical source of product feedback and a way to monitor quality issues. 

As useful as they can be, online reviews have grown so numerous, rote, and flooded with bots that they’ve lost much of the nuance that people once found so valuable. Some companies have responded by allowing customers to upvote reviews, as with the “helpful” button below Amazon reviews.   

But these helpfulness ratings only really become useful once there’s a big enough collection of customer reviews for other customers to read and rate. So Shani collaborated with Achal Bassamboo, a Kellogg professor of operations, and Maria Ibanez, an associate professor of operations at Kellogg, to figure out if they could address this “time lag” by relying on other features to predict a review’s helpfulness. They focused on reviews on the online-gaming platform Steam, where users can recommend or critique video games they purchased.  

Analyzing millions of reviews, the researchers studied the relationship between how helpful a review was to other people and how much experience the reviewer had with the game at the time of writing the review. Interestingly, they found, this relationship had critical differences depending on whether the review was positive or negative.    

“Most people would expect that the longer someone has played a game, the more credible they are—that their review’s helpfulness should automatically go up,” says Bassamboo. “Interestingly enough, we find that it’s not that simple.”   

A very good signal   

Having played video games on Steam since his childhood, Shani realized that the platform offered an ideal setting to study online reviews.    

Not only does it have a massive store of publicly available, unsolicited customer reviews, but it also automatically tracks and reveals the amount of time a user spent playing a game at the time of the review (their experience level), the user’s final recommendation (a thumbs up versus thumbs down), and the number of people who flagged the review as helpful. 

“What makes this data very unique is that the credibility is measurable in terms of the experience the reviewer has with the product,” Bassamboo explains. “And the fact that the reviews gave a very clear, black-or-white answer.”    

In all, the researchers analyzed 26.8 million reviews for 27,170 games written by 10.6 million unique users from October 2010 through October 2019. And they found that there was a strong link between how much time a user spent playing a video game at the time of their review and how helpful other people found the review to be.

“Reviewer experience [with a product] is a very good signal,” Shani says. “But it is not a uniform signal. It depends on the verdict of the review.”   

Among positive reviews, users who had either the shortest playing time or experience with the game or the longest playing time provided the most-helpful reviews. But for negative reviews, users with a moderate amount of playing time—that is, with a moderate amount of experience—wrote the most-helpful reviews.   

Experience and expectations   

The researchers describe this relationship between a reviewer’s experience and their review’s helpfulness as “a U-shaped curve” for positive reviews and “an inverted U-shaped curve” for negative reviews.   

“What that means is that, for positive reviews, the helpfulness of a review first decreases as reviewers play more and then starts increasing,” Shani says. “And on the negative side, it first starts increasing and then dips later.”    

The team believes that this pattern is at least partly dictated by people’s expectations. When an outsider reads a positive review and sees that the reviewer has only played the game for a short amount of time, they instinctively set a low bar for the review’s quality and credibility, compared with reviews by people with slightly more experience.   

“That’s why review helpfulness takes a dip [with more experience] at first,” Bassamboo says. “Eventually, the pattern that we might expect to see—where helpfulness increases with experience—starts to show up with people who have a lot of experience.”   

In contrast, the pattern is reversed for negative reviews because of the assumption that reviewers who have little experience with a game likely don’t know enough to offer meaningful criticism. But if a user spends a lot of time playing the game before writing a negative review, it’s seen as incongruent, if not disingenuous.   

“In the case of negative reviews, the longer people play a game beyond a particular threshold, the more credibility they lose,” Bassamboo says. “Because if you don’t like the game, why are you wasting your time playing it so much?”   

Building or cutting credibility   

The researchers identified several other factors that intensified or weakened these patterns, including the length of a review, the age of a game at the time of the review, a game’s overall rating, and a user’s review history.   

Longer reviews, for example, amplified the effect and steepened the curve. In other words, the longer the review is, the more likely it is that the most-helpful positive reviews were written by either the least- or the most-experienced users—and the more likely it is that the most-helpful negative reviews were written by users with a moderate level of experience.    

“When you write a longer positive review—both for people that have played less and for people that already have a very high experience—that creates a transparency for your credibility, that you have actually thought about this product and you are writing something that’s valid,” Shani says.   

But for negative reviews, particularly those written by users with the least or the most experience, “the length does not help you seem more credible,” Bassamboo adds, even if intuition might suggest it should. “In fact, a longer review might seem like a rant if it’s negative.”   

Practical guidance   

Beyond Steam, the findings offer practical guidance for businesses and managers hoping to optimize the use of their customer reviews. For one, the researchers recommend categorizing customer reviews based on factors such as how long the review is, how old a product is, and most importantly, how positive or negative the review is.   

“When you get into any kind of analytics,” Bassamboo says, “it is worthwhile disentangling [the data] because it gives you better insight into what’s happening.”    

Companies should avoid relying on a single rule that treats more experience as inherently better, the researchers say. For positive reviews, platforms may want to surface a mix of perspectives: fresh impressions from customers with little experience and deeper assessments from customers with extensive experience.    

For negative reviews, however, the most useful critiques may come from customers with a moderate amount of experience who have used the product enough to understand it, but not so much that readers wonder why they kept using something they disliked. For mature products, companies also should be careful not to dismiss negative reviews from longtime users because those reviews may describe problems that emerge only after extended use or as the product loses value over time.    

This distinction between positive and negative reviews could change how review pages are designed. Rather than relying on a single overall list of “most helpful” reviews, review pages could rank positive and critical reviews separately and show shoppers strong examples of each.    

The goal would then be to identify which reviews are likely to be helpful while they are still new. For example, platforms can give those reviews some early visibility before they have had time to collect helpfulness votes. Those reviews could also be routed to internal teams in product, support, or quality assurance, helping companies respond to issues sooner. 

Managers could also target specific groups of people when soliciting reviews, such as those who tend to write positive reviews and have a lot of experience or very little experience with a particular product. If managers determine how much experience a customer has with a product, they can potentially “predict or learn about how helpful the review can be,” Shani says.    

That would allow companies to feature the most-helpful reviews on their website first—even without community helpfulness ratings—which, the researchers say, could help improve customers’ search experience and increase their chances of making the right purchase.