Answering Unasked Questions
Skip to content
Podcast | Insight Unpacked Season 1: Extraordinary Brands and How to Build Them
Marketing Apr 2, 2012

Answering Unasked Questions

Identifying bias in consumer surveys and compensating for it

Based on the research of

David Gal

Derek D. Rucker

When they set out to gain insights into the general public’s attitudes, preferences, and beliefs, designers of consumer surveys must take into account several potential sources of bias.

Add Insight
to your inbox.

Some of the most common include survey takers’ dishonesty about their behavior, misinterpretation of questions, and lack of effort in providing accurate answers to survey questions.

Two Kellogg School of Management researchers have discovered a new and previously unrecognized type of bias among survey respondents. This bias stems from respondents’ frustration when surveys ignore issues that they regard as critically important. However, designers of surveys can rest relatively easy in light of the new finding. The Kellogg pair has also identified an easy and cost-effective way of overcoming the bias: giving respondents the opportunity to express their concerns.

Venting Strong Opinions
Response substitution, as the newly identified bias is called, is the process whereby some respondents frame their answers to questions in a survey in such a way as to express their views on issues outside the survey’s scope—issues on which they have strong opinions.

“People don’t answer the question they’re asked. Instead, they supply an opinion they want to share,” explains Derek Rucker, an associate professor of marketing at the Kellogg School. “The idea that respondents don’t give accurate answers is intuitive,” adds David Gal, an assistant professor of marketing at the Kellogg School. “But the idea that they would volitionally choose to answer questions differently is a very novel insight—a new source of bias that we’re the first to demonstrate.”

Imagine a diner who, after suffering poor service in a restaurant, is asked to complete a survey that focuses only on the quality of the food. Even though she enjoyed the meal, the diner might give it a negative review in order to voice her general disapproval of the service.

“She is not responding simply to the question whether the food is good but perceives the question also as an opportunity to express her negative attitude toward the restaurant due to the service,” write Gal and Rucker.

“So you not only want to ask the right questions. You want to make sure the consumer is answering the right questions.”

The concept of answers to unasked questions has obvious implications for interpreters of consumer surveys and organizations that rely on those interpretations. “[R]esearchers and marketing practitioners might misconstrue the meaning of respondents’ answers to their questions if respondents attempt to express attitudes or beliefs that the researchers have not asked about,” Gal and Rucker write.

Three Experiments
Once Gal and Rucker imagined the concept of response substitution, they devised means of testing whether it reflected reality and determining ways to compensate for it in the design of surveys. The process involved three experiments.

The first experiment asked volunteers to rate the intelligence and wastefulness of two hostesses who treated their friends to dinner. Anne entertained at home, having bought a $250 fondue set that she never used again; Jane spent the same amount at a fondue restaurant. The study participants reacted as previous research had suggested they would, perceiving only Anne as wasteful. They also rated Anne as less intelligent when the survey asked them to rank her intelligence before her wastefulness. But if they were asked to rate wastefulness before intelligence, they regarded her as more intelligent because they had already given their opinions on her wastefulness.

“This pattern of data is consistent with response substitution because wastefulness led to more negative perceptions of intelligence when participants did not have an opportunity to provide their attitude toward wastefulness,” Gal and Rucker write.

The next experiment asked a different set of volunteers to evaluate the quality of a snack bar (based on pictures of the product) manufactured by two companies, one plainly moral and the other immoral. All were asked how important they regarded a company’s moral behavior when deciding whether or not to buy its products. Some subjects were also told that they could add any open-ended thoughts or comments to their answers at the end of the survey.

Volunteers who did not have the chance to express open-ended thoughts used the question about product quality to express their attitudes about the company’s morality, Gal and Rucker report. And those who regarded the company’s moral behavior as important in their purchasing decisions were more likely to engage in such response substitution.

The third experiment asked yet another group of volunteers to imagine receiving excellent service and food at a restaurant that had recently been upgraded. They were then asked to complete a survey on the new décor and lighting. Again, some participants were given the chance to add extra comments, while others had no such opportunity. In this case, the two researchers write, “participants who believed there was a high likelihood that their answer about décor and lighting might convey their attitude toward the restaurant provided a more favorable rating when they did not know they would have an opportunity to self-express than when they did.”

Occurrence, Mechanism, and Avoidance
Taken together, the three experiments not only confirm that response substitution occurs, but also provide some understanding of its mechanism. “The first experiment demonstrates the effect,” Rucker explains. “The second experiment shows the significance of a topic is important; when consumers have an important opinion, they want to convey it. And the third experiment shows that you can reduce response substitution by telling people they’ll have an option to state any other opinions they have later.”

That third result, Gal points out, “provides an easy solution for survey designers who want to avoid response substitution: Tell the consumers that, after the survey is over, they’ll have the opportunity to express any opinions they might have. They’ll probably be more satisfied with the survey and give you more valid data.”

Lessons for Survey Users
The research carries other lessons for designers and interpreters of consumer surveys. “When answers look inconsistent or curious, you might want to ask whether the survey takers are trying to express something else,” Rucker warns. “If, say, they dine frequently but aren’t too interested in the food, they might have another reason for their low rating. So if you see inconsistencies or oddities you might look at response substitution as a possibility.”

“The main message is that, when completing surveys for companies or researchers, people may answer in terms of what they want to share,” he continues. “So you not only want to ask the right questions. You want to make sure the consumer is answering the right questions.”

Gal adds an intriguing thought that can stimulate further research on the issue. “One interesting question,” he says, “is that survey takers might start to believe what they say. Saying they don’t like a restaurant’s food even if they do might convince them that they actually don’t like it.”

Related reading on Kellogg Insight

Surveying Sensitive Topics: New tools help correct for survey bias

Desire to Acquire: Powerlessness and compensatory consumption

Be Good, Get Mad: Exerting self-control makes people more inclined to anger

Featured Faculty

Member of the Marketing Department faculty until 2014

Sandy & Morton Goldman Professor of Entrepreneurial Studies in Marketing; Professor of Marketing; Co-chair of Faculty Research

About the Writer
Peter Gwynne is a freelance writer based in Sandwich, Mass.
About the Research

Gal, David and Derek D. Rucker. 2011. “Answering the Unasked Question: Response Substitution in Consumer Surveys.” Journal of Marketing Research. 48(1): 185-195.

Read the original

Most Popular This Week
  1. Your Team Doesn’t Need You to Be the Hero
    Too many leaders instinctively try to fix a crisis themselves. A U.S. Army colonel explains how to curb this tendency in yourself and allow your teams to flourish.
    person with red cape trying to put out fire while firefighters stand by.
  2. What Triggers a Career Hot Streak?
    New research reveals a recipe for success.
    Collage of sculptor's work culminating in Artist of the Year recognition
  3. What’s the Secret to Successful Innovation?
    Hint: it’s not the product itself.
    standing woman speaking with man seated on stool
  4. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  5. How Much Do Campaign Ads Matter?
    Tone is key, according to new research, which found that a change in TV ad strategy could have altered the results of the 2000 presidential election.
    Political advertisements on television next to polling place
  6. What Went Wrong with FTX—and What’s Next for Crypto?
    One key issue will be introducing regulation without strangling innovation, a fintech expert explains.
    stock trader surrounded by computer monitors
  7. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  8. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  9. How Experts Make Complex Decisions
    By studying 200 million chess moves, researchers shed light on what gives players an advantage—and what trips them up.
    two people playing chess
  10. Yes, Consumers Care if Your Product Is Ethical
    New research shows that morality matters—but it’s in the eye of the beholder.
    woman chooses organic lettuce in grocery
  11. Why Well-Meaning NGOs Sometimes Do More Harm than Good
    Studies of aid groups in Ghana and Uganda show why it’s so important to coordinate with local governments and institutions.
    To succeed, foreign aid and health programs need buy-in and coordination with local partners.
  12. Product Q&A Forums Hold a Lot of Promise. Here’s How to Make Them Work.
    The key to these online communities, where users can ask and answer questions, is how many questions get useful answers.
    man sits at computer reading Q&A forum
  13. What Went Wrong at AIG?
    Unpacking the insurance giant's collapse during the 2008 financial crisis.
    What went wrong during the AIG financial crisis?
  14. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  15. What the New Climate Bill Means for the U.S.—and the World
    The Inflation Reduction Act won’t reverse inflation or halt climate change, but it's still a big deal.
    energy bill with solar panels wind turbines and pipelines
  16. Post-War Reconstruction Is a Good Investment
    Ukraine’s European neighbors will need to make a major financial commitment to help rebuild its economy after the war. Fortunately, as the legacy of the post–World War II Marshall Plan shows, investing in Ukraine's future will also serve Europe's own long-term interests.
    two people look out over a city
  17. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  18. The Political Divide in America Goes Beyond Polarization and Tribalism
    These days, political identity functions a lot like religious identity.
    people engage in conflict with swords
More in Marketing