Surveying Sensitive Topics
Skip to content
Marketing Apr 5, 2010

Surveying Sensitive Topics

New tools help correct for survey bias

Based on the research of

Ulf Bockenholt

Peter van der Heijden

Getting truthful answers to survey questions about sensitive matters—such as using illegal drugs or cheating on taxes—is a challenge. Even when promised anonymity, people who break the rules can be inclined to lie about their behavior. The randomized response (RR) model was developed to address this issue, but it is an imperfect tool. The model underestimates noncompliant behavior, and because large samples are needed, it is expensive to use. Researchers need an improved approach.

Ulf Böckenholt, a professor of marketing at the Kellogg School of Management, and his colleague Peter G. M. van der Heijden, a professor at Utrecht University, in the Netherlands, have developed a model that surmounts the shortcomings of the RR. Their model, the mixture-item randomized response (IRR), provides more accurate estimates of noncompliance because it allows researchers to identify suspect answers that affect RR results.

In a survey of Dutch disability insurance claimants, the authors found that mixture-IRR produced substantially higher estimates of cheating than RR. The mixture-IRR model “does a better job than current alternatives at getting closer to the truth,” Böckenholt says.

The Need for Accuracy
Governments need accurate estimates of compliance to make appropriate public policy decisions, Böckenholt says. The results of noncompliance can be tragic, particularly in cases that involve public safety. This was evident a decade ago when a fireworks explosion destroyed part of the Netherlands city of Enshede after rules governing the storage of fireworks were ignored. Later that year, ten people died and 130 others were injured when a fire swept through a crowded Dutch café that had failed to comply with fire regulations. Even in cases that do not involve life and death, authorities need to understand and measure noncompliance, Böckenholt notes. The U.S. Internal Revenue Service would like to know the willingness of taxpayers to comply with tax laws, for example.

To test their model, Böckenholt and van der Heijden surveyed 1,760 and 840 Dutch disability recipients in 2002 and 2004 respectively. The analysis focused on six RR questions, four related to health and two related to work. For example, one of the health questions asked: Have you noticed personally any recovery from your disability complaints without reporting it to the social welfare agency? A work-related question read: In the last 12 months, have you taken a small job alone or together with your friends that you got paid for without informing the social welfare agency? The survey followed a “forced choice” design in which respondents were asked to click on two electronic dice and answer “yes” when the dice totaled 2, 3, and 4; “no” when the dice totaled 11 or 12; and honestly in all other cases.

When Böckenholt and van der Heijden analyzed responses to the health and work questions, they found more “no” responses than their statistical model predicted. Probing the data, the authors found that truthfulness was related to respondents’ education level and the clarity of the survey instructions. Some respondents did not believe their anonymity was protected and “felt threatened by the question, possibly wanting to hide what they did,” Böckenholt says.

Böckenholt and van der Heijden’s mixture IRR-model accounts for these suspect answers. In their analysis of the responses to the six questions, estimates of noncompliance ranged from 3 percent to 16 percent. The noncompliance estimates were between 6 percent and 70 percent higher than those obtained by traditional RR methods.

“These differences are substantial and demonstrate the value of the proposed approach,” the authors wrote. “By not taking into account possible response biases, the actual incidence of noncompliance is severely underestimated.”

The estimated levels of noncompliance on health and work areas were 19 percent and 18 percent respectively, using the mixture IRR model.

“There are two aspects that are perhaps worth mentioning,” Böckenholt says. “First, about 13 percent of the respondents were identified as “self-protective” in the sense that they did not answer the question as instructed. Second, in a related study where people were asked directly about their compliance behavior about 55 percent of the respondents were identified as being self-protective. This shows that the method works in motivating most of the respondents to answer truthfully.”

The surveys also included a second set of questions focused on factors thought to account for individual differences in behavior. Behind these questions was the assumption that compliance was not solely a matter of weighing the benefits of cheating against the risk of getting caught; the attitudes of family and friends about cheating also influenced behavior. About half of these additional questions focused on enforcement actions by a government agency. Most of the remaining questions probed the perceived risks and benefits of compliance, as well as social attitudes about law abidance.

Böckenholt and van der Heijden used this additional data to interpret the responses to the health and welfare questions. They found that people were more likely to obey the rules when little was to be gained from cheating and when social pressures and personal beliefs favored compliance. Respondents’ behavior was strongly influenced by whether “friends and neighbors would see that they were doing the right thing,” Böckenholt says. If their social group condemned cheating, respondents were less inclined to break the rules.

Societal Influence
Interestingly, social pressures had a greater impact on behavior than the threat of prosecution and punishment. Worries about getting caught influenced whether respondents answered survey questions truthfully, but had little effect on whether or not they cheated, the researchers found.

These findings support previous studies showing that social norms exert a powerful influence on behavior, Böckenholt says. For example, one study found that tax fraud rose after government officials stiffened the penalties for cheating (Kahan, 1997). The increased sanctions were viewed by respondents as a signal that cheating was common and therefore the norm.

The Dutch government was considering stiffer sanctions for disability insurance fraud but decided against implementing them after learning the results of his research, Böckenholt says. He hopes the mixture-IRR model would spur research about what he calls the “dark side of consumer behavior.”

“The entire domain has not received much attention because we cannot ask questions and assume the answers are accurate,” Böckenholt says. By using this new model, he says, researchers studying gambling, alcohol abuse or other sensitive behaviors “can get a better estimate of the truth.”

Further reading: Kahan, D. (1997) Social influence, social meaning and deterrence. Virginia Law Review 83: 349-395.

About the Writer
Denise Gellene is a freelance science and business writer based in Los Angeles, California.
About the Research

Böckenholt, Ulf and Peter G. M. van der Heijden. 2007. Item randomized-response models for measuring noncompliance: Risk-return perceptions, social influences, and self-protective responses. Psychometrika. 72: 245-262.

Read the original

Most Popular This Week
  1. One Key to a Happy Marriage? A Joint Bank Account.
    Merging finances helps newlyweds align their financial goals and avoid scorekeeping.
    married couple standing at bank teller's window
  2. Take 5: Yikes! When Unintended Consequences Strike
    Good intentions don’t always mean good results. Here’s why humility, and a lot of monitoring, are so important when making big changes.
    People pass an e-cigarette billboard
  3. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  4. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  5. Entrepreneurship Through Acquisition Is Still Entrepreneurship
    ETA is one of the fastest-growing paths to entrepreneurship. Here's how to think about it.
    An entrepreneur strides toward a business for sale.
  6. Take 5: Research-Backed Tips for Scheduling Your Day
    Kellogg faculty offer ideas for working smarter and not harder.
    A to-do list with easy and hard tasks
  7. How to Manage a Disengaged Employee—and Get Them Excited about Work Again
    Don’t give up on checked-out team members. Try these strategies instead.
    CEO cheering on team with pom-poms
  8. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  9. What Went Wrong at AIG?
    Unpacking the insurance giant's collapse during the 2008 financial crisis.
    What went wrong during the AIG financial crisis?
  10. The Appeal of Handmade in an Era of Automation
    This excerpt from the book “The Power of Human" explains why we continue to equate human effort with value.
    person, robot, and elephant make still life drawing.
  11. 2 Factors Will Determine How Much AI Transforms Our Economy
    They’ll also dictate how workers stand to fare.
    robot waiter serves couple in restaurant
  12. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  13. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  14. Sitting Near a High-Performer Can Make You Better at Your Job
    “Spillover” from certain coworkers can boost our productivity—or jeopardize our employment.
    The spillover effect in offices impacts workers in close physical proximity.
  15. How the Wormhole Decade (2000–2010) Changed the World
    Five implications no one can afford to ignore.
    The rise of the internet resulted in a global culture shift that changed the world.
  16. What’s at Stake in the Debt-Ceiling Standoff?
    Defaulting would be an unmitigated disaster, quickly felt by ordinary Americans.
    two groups of politicians negotiate while dangling upside down from the ceiling of a room
  17. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
  18. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  19. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  20. 3 Traits of Successful Market-Creating Entrepreneurs
    Creating a market isn’t for the faint of heart. But a dose of humility can go a long way.
    man standing on hilltop overlooking city
Add Insight to your inbox.
More in Marketing