New Survey Shows Some Privacy Scholars Lack Objectivity

A survey funded by Nokia and conducted at the Berkeley Center for Law and Technology shows what has become increasingly apparent to those who follow this line of research: some of the most prominent academic researchers have ceased to retain even a veneer of objectivity in their research on privacy. The authors, Chris Hoofnagle, Jennifer Urban and Su Li, state that their survey shows that “Americans have a low level of knowledge about [Do Not Track], but prefer that it mean that websites do not collect tracking data.”

I won’t mince words here: this is shoddy research.

There are two main survey questions in their study related to Do Not Track (for more on this proposal and why it is a bad idea, see this or this). The first is a question about whether people have even heard of the Do Not Track proposal. The survey question reads, “Policymakers are considering creating a ‘do not track’ option for the internet. Have you heard of proposals for a ‘do not track’ system, or not?” Thirteen percent of respondents indicated that they had heard of the proposal; eighty-seven percent had not. This was not a new finding as a previous survey by McDonald and Peha (2011) had found similar results.

What does this result mean? It could be that this is an example of rational ignorance. Since most individuals are not privacy fundamentalists, the costs of seeking out information about the latest proposed privacy regulations do not exceed the benefits. Therefore these individuals willingly stay uninformed about proposals like Do Not Track because it is not that interesting to them. This is likely the same large majority of consumers who choose not to read privacy policies on websites.

Or it could be that the public is just ignorant about policy in general. After all, this is the same public where a 2012 survey from Xavier University found one in three Americans would fail the civics portion of the naturalization test to become a U.S. citizen (and where 85 percent of respondents could not answer the question “What is the rule of law”). Is it really surprising that a large number of Americans are also largely uninformed about proposed Internet laws and policies?

The second survey question was as follows:

If a ‘do not track’ option were available to you when browsing the internet, which of the following things would you most want it to do? Should do not track…

• Prevent websites from collecting information about you
• Block websites from showing you advertisements
• Prevent websites from tailoring advertisements based upon the websites you have previously used
• Don’t know/refused

First, the wording of the questions was biased. The surveyors omit any response to this question that couldn’t be construed as either a complaint against online privacy (responses #1 and #3) or a complaint against online advertisements (response #2). This has the effect that virtually any response to this survey question could be used to bolster claims that users do not want tracking or online advertising.

Second, the study omitted the “correct” answer from the list of potential responses. Their survey question fundamentally misconstrues the purpose of Do Not Track which is not to prevent tracking but rather to give users the ability to express their preference regarding tracking. Even the W3C Tracking Protection Working Group which is heading the effort to define a Do Not Track standard (and is being led by privacy advocates) states that the purpose is to give users “a mechanism to express their own preference regarding tracking that is both simple to configure and efficient when implemented.” So why did the authors include three different response options but did not include the actual meaning of the policy?

Third, as noted earlier, most of the respondents had not even heard about the Do Not Track proposal. The authors knew this even before conducting their survey both because of prior research in this area and the pre-testing they did with their survey instrument. Yet they still included the name of the policy in the question. So the surveyors were basically asking the respondents to explain what they thought the proposal should mean based only on its name. Not surprisingly, when Americans are asked what they think a policy called “do not track” should do, they picked the answer “prevent websites from collecting information about you.” The authors claim this reflects respondents’ preferences on privacy. The authors of the study do not even consider an alternative hypothesis, such as that perhaps this response simply reflects the respondents’ expectations about what the policy should do based on its name. (And, if this is the case, then this is evidence that policymakers who want to have an honest conversation with the public should stop referring to the policy as “Do Not Track” and instead use something more accurate such as “Tracking Preferences.”)

It would have been simple for the authors of the study to ask even a slightly less biased question. For example, instead of asking “If a ‘do not track’ option were available to you when browsing the internet, which of the following things would you most want it to do?” they could have omitted the name of the proposal and simply asked “If a new option were available to you when browsing the internet, which of the following things would you most want it to do?” After all, since the researchers knew that the respondents did not know what “Do Not Track” meant, why include it in the survey? The only plausible reasons that I see are either sloppy research or intent to slant the responses.

Fourth, the question does not ask about tradeoffs. Previous studies have shown that most individuals are not “privacy fundamentalists” but instead have balanced views on privacy and they are willing to exchange some privacy for some benefits.  However, the questions in this survey do not ask respondents to indicate their preferences in the context of trade-offs. For example, users are not asked questions like “Would you willing to have some of your browsing history tracked in exchange for access to free websites?” Questions like this would reflect the trade-offs and decisions individuals make in the real world.

Moreover, privacy tradeoffs are a well-known phenomenon. For example, travelers routinely object to TSA screenings on privacy grounds. One survey by the Ponemon Institute of travelers found that 69 percent of travelers had privacy concerns about full body scans and 79 percent had concerns about pat-down screenings. Yet when those travelers were asked about trade-offs the results changed substantially: only 21 percent of travelers said they would choose a less invasive screening if it would result in a delay of five minutes and only nine percent said they would choose that option if the delay was ten minutes. As the authors of that survey wrote, “These findings are in line with what the Ponemon Institute and other research organizations have seen in the past: with no specific context, most consumers will claim to be very protective of their personal privacy, but when given a choice of privacy or convenience, most will choose the latter.”

Or consider another study which looked at the willingness of consumers to purchase from an online store that requires more personal information. Consumers were given the option of buying a DVD from one of two online stores: one that required personal information (date of birth and monthly income) and one that did not require this additional information. The store that required the additional personal information was selling the DVD for one euro less than the other store. Yet even though participants overwhelmingly (95 percent) reported that they were interested in the protection of the private information, they also overwhelmingly (93 percent) chose to give up this personal information in exchange for a one euro discount. Once again, this study showed that questions about privacy are useless if asked in isolation: context matters.

So why did the authors of this study not include trade-offs in their survey? Again, there is no explanation for this other than poor research or intentional biases. (Since the lead author, Chris Hoofnagle, is a former privacy activist with EPIC, I tend to suspect it is the latter.)

I am not arguing that researchers can be completely objective. Of course everyone comes with their own biases, values, and core beliefs. But that does not excuse them from at least pursuing objective research. In this case, I think it is clear that their ideology shaped their survey questions to the detriment of honest research and that this is skewing their results. If they want their results to be taken seriously, we need to restore objectivity to this research.