Surveying Sensitive Topics

Getting truthful answers to survey questions about sensitive matters—such as using illegal drugs or cheating on taxes—is a challenge. Even when promised anonymity, people who break the rules can be inclined to lie about their behavior. The randomized response (RR) model was developed to address this issue, but it is an imperfect tool. The model underestimates noncompliant behavior, and because large samples are needed, it is expensive to use. Researchers need an improved approach.

Ulf Böckenholt, a professor of marketing at the Kellogg School of Management, and his colleague Peter G. M. van der Heijden, a professor at Utrecht University, in the Netherlands, have developed a model that surmounts the shortcomings of the RR. Their model, the mixture-item randomized response (IRR), provides more accurate estimates of noncompliance because it allows researchers to identify suspect answers that affect RR results.

In a survey of Dutch disability insurance claimants, the authors found that mixture-IRR produced substantially higher estimates of cheating than RR. The mixture-IRR model “does a better job than current alternatives at getting closer to the truth,” Böckenholt says.

The Need for Accuracy
Governments need accurate estimates of compliance to make appropriate public policy decisions, Böckenholt says. The results of noncompliance can be tragic, particularly in cases that involve public safety. This was evident a decade ago when a fireworks explosion destroyed part of the Netherlands city of Enshede after rules governing the storage of fireworks were ignored. Later that year, ten people died and 130 others were injured when a fire swept through a crowded Dutch café that had failed to comply with fire regulations. Even in cases that do not involve life and death, authorities need to understand and measure noncompliance, Böckenholt notes. The U.S. Internal Revenue Service would like to know the willingness of taxpayers to comply with tax laws, for example.

To test their model, Böckenholt and van der Heijden surveyed 1,760 and 840 Dutch disability recipients in 2002 and 2004 respectively. The analysis focused on six RR questions, four related to health and two related to work. For example, one of the health questions asked: Have you noticed personally any recovery from your disability complaints without reporting it to the social welfare agency? A work-related question read: In the last 12 months, have you taken a small job alone or together with your friends that you got paid for without informing the social welfare agency? The survey followed a “forced choice” design in which respondents were asked to click on two electronic dice and answer “yes” when the dice totaled 2, 3, and 4; “no” when the dice totaled 11 or 12; and honestly in all other cases.

When Böckenholt and van der Heijden analyzed responses to the health and work questions, they found more “no” responses than their statistical model predicted. Probing the data, the authors found that truthfulness was related to respondents’ education level and the clarity of the survey instructions. Some respondents did not believe their anonymity was protected and “felt threatened by the question, possibly wanting to hide what they did,” Böckenholt says.

Böckenholt and van der Heijden’s mixture IRR-model accounts for these suspect answers. In their analysis of the responses to the six questions, estimates of noncompliance ranged from 3 percent to 16 percent. The noncompliance estimates were between 6 percent and 70 percent higher than those obtained by traditional RR methods.

“These differences are substantial and demonstrate the value of the proposed approach,” the authors wrote. “By not taking into account possible response biases, the actual incidence of noncompliance is severely underestimated.”

The estimated levels of noncompliance on health and work areas were 19 percent and 18 percent respectively, using the mixture IRR model.

“There are two aspects that are perhaps worth mentioning,” Böckenholt says. “First, about 13 percent of the respondents were identified as “self-protective” in the sense that they did not answer the question as instructed. Second, in a related study where people were asked directly about their compliance behavior about 55 percent of the respondents were identified as being self-protective. This shows that the method works in motivating most of the respondents to answer truthfully.”

The surveys also included a second set of questions focused on factors thought to account for individual differences in behavior. Behind these questions was the assumption that compliance was not solely a matter of weighing the benefits of cheating against the risk of getting caught; the attitudes of family and friends about cheating also influenced behavior. About half of these additional questions focused on enforcement actions by a government agency. Most of the remaining questions probed the perceived risks and benefits of compliance, as well as social attitudes about law abidance.

Böckenholt and van der Heijden used this additional data to interpret the responses to the health and welfare questions. They found that people were more likely to obey the rules when little was to be gained from cheating and when social pressures and personal beliefs favored compliance. Respondents’ behavior was strongly influenced by whether “friends and neighbors would see that they were doing the right thing,” Böckenholt says. If their social group condemned cheating, respondents were less inclined to break the rules.

Societal Influence
Interestingly, social pressures had a greater impact on behavior than the threat of prosecution and punishment. Worries about getting caught influenced whether respondents answered survey questions truthfully, but had little effect on whether or not they cheated, the researchers found.

These findings support previous studies showing that social norms exert a powerful influence on behavior, Böckenholt says. For example, one study found that tax fraud rose after government officials stiffened the penalties for cheating (Kahan, 1997). The increased sanctions were viewed by respondents as a signal that cheating was common and therefore the norm.

The Dutch government was considering stiffer sanctions for disability insurance fraud but decided against implementing them after learning the results of his research, Böckenholt says. He hopes the mixture-IRR model would spur research about what he calls the “dark side of consumer behavior.”

“The entire domain has not received much attention because we cannot ask questions and assume the answers are accurate,” Böckenholt says. By using this new model, he says, researchers studying gambling, alcohol abuse or other sensitive behaviors “can get a better estimate of the truth.”

Further reading: Kahan, D. (1997) Social influence, social meaning and deterrence. Virginia Law Review 83: 349-395.