Surveying Sensitive Topics
Skip to content
Marketing Apr 5, 2010

Surveying Sensitive Topics

New tools help correct for survey bias

Based on the research of

Ulf Bockenholt

Peter van der Heijden

Getting truthful answers to survey questions about sensitive matters—such as using illegal drugs or cheating on taxes—is a challenge. Even when promised anonymity, people who break the rules can be inclined to lie about their behavior. The randomized response (RR) model was developed to address this issue, but it is an imperfect tool. The model underestimates noncompliant behavior, and because large samples are needed, it is expensive to use. Researchers need an improved approach.

Ulf Böckenholt, a professor of marketing at the Kellogg School of Management, and his colleague Peter G. M. van der Heijden, a professor at Utrecht University, in the Netherlands, have developed a model that surmounts the shortcomings of the RR. Their model, the mixture-item randomized response (IRR), provides more accurate estimates of noncompliance because it allows researchers to identify suspect answers that affect RR results.

In a survey of Dutch disability insurance claimants, the authors found that mixture-IRR produced substantially higher estimates of cheating than RR. The mixture-IRR model “does a better job than current alternatives at getting closer to the truth,” Böckenholt says.

The Need for Accuracy
Governments need accurate estimates of compliance to make appropriate public policy decisions, Böckenholt says. The results of noncompliance can be tragic, particularly in cases that involve public safety. This was evident a decade ago when a fireworks explosion destroyed part of the Netherlands city of Enshede after rules governing the storage of fireworks were ignored. Later that year, ten people died and 130 others were injured when a fire swept through a crowded Dutch café that had failed to comply with fire regulations. Even in cases that do not involve life and death, authorities need to understand and measure noncompliance, Böckenholt notes. The U.S. Internal Revenue Service would like to know the willingness of taxpayers to comply with tax laws, for example.

To test their model, Böckenholt and van der Heijden surveyed 1,760 and 840 Dutch disability recipients in 2002 and 2004 respectively. The analysis focused on six RR questions, four related to health and two related to work. For example, one of the health questions asked: Have you noticed personally any recovery from your disability complaints without reporting it to the social welfare agency? A work-related question read: In the last 12 months, have you taken a small job alone or together with your friends that you got paid for without informing the social welfare agency? The survey followed a “forced choice” design in which respondents were asked to click on two electronic dice and answer “yes” when the dice totaled 2, 3, and 4; “no” when the dice totaled 11 or 12; and honestly in all other cases.

When Böckenholt and van der Heijden analyzed responses to the health and work questions, they found more “no” responses than their statistical model predicted. Probing the data, the authors found that truthfulness was related to respondents’ education level and the clarity of the survey instructions. Some respondents did not believe their anonymity was protected and “felt threatened by the question, possibly wanting to hide what they did,” Böckenholt says.

Böckenholt and van der Heijden’s mixture IRR-model accounts for these suspect answers. In their analysis of the responses to the six questions, estimates of noncompliance ranged from 3 percent to 16 percent. The noncompliance estimates were between 6 percent and 70 percent higher than those obtained by traditional RR methods.

“These differences are substantial and demonstrate the value of the proposed approach,” the authors wrote. “By not taking into account possible response biases, the actual incidence of noncompliance is severely underestimated.”

The estimated levels of noncompliance on health and work areas were 19 percent and 18 percent respectively, using the mixture IRR model.

“There are two aspects that are perhaps worth mentioning,” Böckenholt says. “First, about 13 percent of the respondents were identified as “self-protective” in the sense that they did not answer the question as instructed. Second, in a related study where people were asked directly about their compliance behavior about 55 percent of the respondents were identified as being self-protective. This shows that the method works in motivating most of the respondents to answer truthfully.”

The surveys also included a second set of questions focused on factors thought to account for individual differences in behavior. Behind these questions was the assumption that compliance was not solely a matter of weighing the benefits of cheating against the risk of getting caught; the attitudes of family and friends about cheating also influenced behavior. About half of these additional questions focused on enforcement actions by a government agency. Most of the remaining questions probed the perceived risks and benefits of compliance, as well as social attitudes about law abidance.

Böckenholt and van der Heijden used this additional data to interpret the responses to the health and welfare questions. They found that people were more likely to obey the rules when little was to be gained from cheating and when social pressures and personal beliefs favored compliance. Respondents’ behavior was strongly influenced by whether “friends and neighbors would see that they were doing the right thing,” Böckenholt says. If their social group condemned cheating, respondents were less inclined to break the rules.

Societal Influence
Interestingly, social pressures had a greater impact on behavior than the threat of prosecution and punishment. Worries about getting caught influenced whether respondents answered survey questions truthfully, but had little effect on whether or not they cheated, the researchers found.

These findings support previous studies showing that social norms exert a powerful influence on behavior, Böckenholt says. For example, one study found that tax fraud rose after government officials stiffened the penalties for cheating (Kahan, 1997). The increased sanctions were viewed by respondents as a signal that cheating was common and therefore the norm.

The Dutch government was considering stiffer sanctions for disability insurance fraud but decided against implementing them after learning the results of his research, Böckenholt says. He hopes the mixture-IRR model would spur research about what he calls the “dark side of consumer behavior.”

“The entire domain has not received much attention because we cannot ask questions and assume the answers are accurate,” Böckenholt says. By using this new model, he says, researchers studying gambling, alcohol abuse or other sensitive behaviors “can get a better estimate of the truth.”

Further reading: Kahan, D. (1997) Social influence, social meaning and deterrence. Virginia Law Review 83: 349-395.

About the Writer
Denise Gellene is a freelance science and business writer based in Los Angeles, California.
About the Research

Böckenholt, Ulf and Peter G. M. van der Heijden. 2007. Item randomized-response models for measuring noncompliance: Risk-return perceptions, social influences, and self-protective responses. Psychometrika. 72: 245-262.

Read the original

Most Popular This Week
  1. How Much Do Boycotts Affect a Company’s Bottom Line?
    There’s often an opposing camp pushing for a “buycott” to support the company. New research shows which group has more sway.
    grocery store aisle where two groups of people protest. One group is boycotting, while the other is buycotting
  2. 5 Takeaways on the State of ESG Investing
    ESG investing is hot. But what does it actually deliver for society and for shareholders?
    watering can pouring over windmills
  3. Could Bringing Your "Whole Self" to Work Curb Unethical Behavior?
    Organizations would be wise to help employees avoid compartmentalizing their personal and professional identities.
    A star employee brings her whole self to work.
  4. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  5. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  6. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  7. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
  8. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  9. What Went Wrong at AIG?
    Unpacking the insurance giant's collapse during the 2008 financial crisis.
    What went wrong during the AIG financial crisis?
  10. Why Well-Meaning NGOs Sometimes Do More Harm than Good
    Studies of aid groups in Ghana and Uganda show why it’s so important to coordinate with local governments and institutions.
    To succeed, foreign aid and health programs need buy-in and coordination with local partners.
  11. 3 Tips for Reinventing Your Career After a Layoff
    It’s crucial to reassess what you want to be doing instead of jumping at the first opportunity.
    woman standing confidently
  12. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  13. Podcast: Does Your Life Reflect What You Value?
    On this episode of The Insightful Leader, a former CEO explains how to organize your life around what really matters—instead of trying to do it all.
  14. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  15. In a World of Widespread Video Sharing, What’s Real and What’s Not?
    A discussion with a video-authentication expert on what it takes to unearth “deepfakes.”
    A detective pulls back his computer screen to reveal code behind the video image.
  16. College Campuses Are Becoming More Diverse. But How Much Do Students from Different Backgrounds Actually Interact?
    Increasing diversity has been a key goal, “but far less attention is paid to what happens after we get people in the door.”
    College quad with students walking away from the center
More in Marketing