Answering Unasked Questions
Skip to content
Marketing Apr 2, 2012

Answering Unasked Questions

Identifying bias in consumer surveys and compensating for it

Based on the research of

David Gal

Derek D. Rucker

When they set out to gain insights into the general public’s attitudes, preferences, and beliefs, designers of consumer surveys must take into account several potential sources of bias.

Some of the most common include survey takers’ dishonesty about their behavior, misinterpretation of questions, and lack of effort in providing accurate answers to survey questions.

Two Kellogg School of Management researchers have discovered a new and previously unrecognized type of bias among survey respondents. This bias stems from respondents’ frustration when surveys ignore issues that they regard as critically important. However, designers of surveys can rest relatively easy in light of the new finding. The Kellogg pair has also identified an easy and cost-effective way of overcoming the bias: giving respondents the opportunity to express their concerns.

Venting Strong Opinions
Response substitution, as the newly identified bias is called, is the process whereby some respondents frame their answers to questions in a survey in such a way as to express their views on issues outside the survey’s scope—issues on which they have strong opinions.

“People don’t answer the question they’re asked. Instead, they supply an opinion they want to share,” explains Derek Rucker, an associate professor of marketing at the Kellogg School. “The idea that respondents don’t give accurate answers is intuitive,” adds David Gal, an assistant professor of marketing at the Kellogg School. “But the idea that they would volitionally choose to answer questions differently is a very novel insight—a new source of bias that we’re the first to demonstrate.”

Imagine a diner who, after suffering poor service in a restaurant, is asked to complete a survey that focuses only on the quality of the food. Even though she enjoyed the meal, the diner might give it a negative review in order to voice her general disapproval of the service.

“She is not responding simply to the question whether the food is good but perceives the question also as an opportunity to express her negative attitude toward the restaurant due to the service,” write Gal and Rucker.

“So you not only want to ask the right questions. You want to make sure the consumer is answering the right questions.”

The concept of answers to unasked questions has obvious implications for interpreters of consumer surveys and organizations that rely on those interpretations. “[R]esearchers and marketing practitioners might misconstrue the meaning of respondents’ answers to their questions if respondents attempt to express attitudes or beliefs that the researchers have not asked about,” Gal and Rucker write.

Three Experiments
Once Gal and Rucker imagined the concept of response substitution, they devised means of testing whether it reflected reality and determining ways to compensate for it in the design of surveys. The process involved three experiments.

The first experiment asked volunteers to rate the intelligence and wastefulness of two hostesses who treated their friends to dinner. Anne entertained at home, having bought a $250 fondue set that she never used again; Jane spent the same amount at a fondue restaurant. The study participants reacted as previous research had suggested they would, perceiving only Anne as wasteful. They also rated Anne as less intelligent when the survey asked them to rank her intelligence before her wastefulness. But if they were asked to rate wastefulness before intelligence, they regarded her as more intelligent because they had already given their opinions on her wastefulness.

“This pattern of data is consistent with response substitution because wastefulness led to more negative perceptions of intelligence when participants did not have an opportunity to provide their attitude toward wastefulness,” Gal and Rucker write.

The next experiment asked a different set of volunteers to evaluate the quality of a snack bar (based on pictures of the product) manufactured by two companies, one plainly moral and the other immoral. All were asked how important they regarded a company’s moral behavior when deciding whether or not to buy its products. Some subjects were also told that they could add any open-ended thoughts or comments to their answers at the end of the survey.

Volunteers who did not have the chance to express open-ended thoughts used the question about product quality to express their attitudes about the company’s morality, Gal and Rucker report. And those who regarded the company’s moral behavior as important in their purchasing decisions were more likely to engage in such response substitution.

The third experiment asked yet another group of volunteers to imagine receiving excellent service and food at a restaurant that had recently been upgraded. They were then asked to complete a survey on the new décor and lighting. Again, some participants were given the chance to add extra comments, while others had no such opportunity. In this case, the two researchers write, “participants who believed there was a high likelihood that their answer about décor and lighting might convey their attitude toward the restaurant provided a more favorable rating when they did not know they would have an opportunity to self-express than when they did.”

Occurrence, Mechanism, and Avoidance
Taken together, the three experiments not only confirm that response substitution occurs, but also provide some understanding of its mechanism. “The first experiment demonstrates the effect,” Rucker explains. “The second experiment shows the significance of a topic is important; when consumers have an important opinion, they want to convey it. And the third experiment shows that you can reduce response substitution by telling people they’ll have an option to state any other opinions they have later.”

That third result, Gal points out, “provides an easy solution for survey designers who want to avoid response substitution: Tell the consumers that, after the survey is over, they’ll have the opportunity to express any opinions they might have. They’ll probably be more satisfied with the survey and give you more valid data.”

Lessons for Survey Users
The research carries other lessons for designers and interpreters of consumer surveys. “When answers look inconsistent or curious, you might want to ask whether the survey takers are trying to express something else,” Rucker warns. “If, say, they dine frequently but aren’t too interested in the food, they might have another reason for their low rating. So if you see inconsistencies or oddities you might look at response substitution as a possibility.”

“The main message is that, when completing surveys for companies or researchers, people may answer in terms of what they want to share,” he continues. “So you not only want to ask the right questions. You want to make sure the consumer is answering the right questions.”

Gal adds an intriguing thought that can stimulate further research on the issue. “One interesting question,” he says, “is that survey takers might start to believe what they say. Saying they don’t like a restaurant’s food even if they do might convince them that they actually don’t like it.”

Related reading on Kellogg Insight

Surveying Sensitive Topics: New tools help correct for survey bias

Desire to Acquire: Powerlessness and compensatory consumption

Be Good, Get Mad: Exerting self-control makes people more inclined to anger

Featured Faculty

Member of the Marketing Department faculty until 2014

Sandy & Morton Goldman Professor of Entrepreneurial Studies in Marketing; Professor of Marketing; Co-chair of Faculty Research

About the Writer
Peter Gwynne is a freelance writer based in Sandwich, Mass.
About the Research

Gal, David and Derek D. Rucker. 2011. “Answering the Unasked Question: Response Substitution in Consumer Surveys.” Journal of Marketing Research. 48(1): 185-195.

Read the original

Most Popular This Week
  1. One Key to a Happy Marriage? A Joint Bank Account.
    Merging finances helps newlyweds align their financial goals and avoid scorekeeping.
    married couple standing at bank teller's window
  2. Take 5: Yikes! When Unintended Consequences Strike
    Good intentions don’t always mean good results. Here’s why humility, and a lot of monitoring, are so important when making big changes.
    People pass an e-cigarette billboard
  3. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  4. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  5. Entrepreneurship Through Acquisition Is Still Entrepreneurship
    ETA is one of the fastest-growing paths to entrepreneurship. Here's how to think about it.
    An entrepreneur strides toward a business for sale.
  6. Take 5: Research-Backed Tips for Scheduling Your Day
    Kellogg faculty offer ideas for working smarter and not harder.
    A to-do list with easy and hard tasks
  7. How to Manage a Disengaged Employee—and Get Them Excited about Work Again
    Don’t give up on checked-out team members. Try these strategies instead.
    CEO cheering on team with pom-poms
  8. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  9. What Went Wrong at AIG?
    Unpacking the insurance giant's collapse during the 2008 financial crisis.
    What went wrong during the AIG financial crisis?
  10. The Appeal of Handmade in an Era of Automation
    This excerpt from the book “The Power of Human" explains why we continue to equate human effort with value.
    person, robot, and elephant make still life drawing.
  11. 2 Factors Will Determine How Much AI Transforms Our Economy
    They’ll also dictate how workers stand to fare.
    robot waiter serves couple in restaurant
  12. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  13. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  14. Sitting Near a High-Performer Can Make You Better at Your Job
    “Spillover” from certain coworkers can boost our productivity—or jeopardize our employment.
    The spillover effect in offices impacts workers in close physical proximity.
  15. How the Wormhole Decade (2000–2010) Changed the World
    Five implications no one can afford to ignore.
    The rise of the internet resulted in a global culture shift that changed the world.
  16. What’s at Stake in the Debt-Ceiling Standoff?
    Defaulting would be an unmitigated disaster, quickly felt by ordinary Americans.
    two groups of politicians negotiate while dangling upside down from the ceiling of a room
  17. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
  18. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  19. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  20. 3 Traits of Successful Market-Creating Entrepreneurs
    Creating a market isn’t for the faint of heart. But a dose of humility can go a long way.
    man standing on hilltop overlooking city
Add Insight to your inbox.
More in Marketing