Why People Morally Object to Experiments

Randomized controlled trials (RCTs; sometimes referred to as A/B tests) are considered the gold standard for evidence in science and medicine. If you took anything away from your Research Methods 101 class, it should have been that the best way to assess whether a variable has a causal effect on an outcome is to conduct a randomized experiment.

Yet RCTs and A/B tests often raise ethical and moral objections. People do not like feeling like they are part of an experiment. Nevertheless, the vast majority of experiments produce useful, meaningful data without posing any greater risk to participants than they would normally encounter in their daily lives. Of course there is no question that a few researchers have made serious ethical violations while running experiments in the past. New research from our team at Geisinger shows that people tend to be skeptical of experiments, even when they approve of the policies or treatments that an experiment is designed to test.

Consider an experiment designed to test two policies – let’s call them Policy A and Policy B – where A and B are not themselves objectionable. Rolling out new Policy A for everyone from now on sounds like a good idea, or at least one that is unlikely to cause any harm. Policy B produces a similar intuition. Maybe people even think A and B are both good ideas! But if we don't know for certain that A or B will work, or if A might be more effective than B, the most rational and scientific next step would be to compare their effectiveness using an experiment.

Our research finds that, despite this rational and evidence-based logic, people tend to object to RCTs and A/B tests even when these same people approve of the two policies that are supposed to be tested. This result, which we call the “A/B Effect,” occurs when people approve of two policies but object to experiments designed to compare them.

The Hospital Safety Checklist Scenario

To illustrate, imagine that there are three hospitals experiencing the same problem: dangerous infections that can result when inserting a catheter into a vein. Leaders at these three hospitals know this is a problem, but luckily, they also know that giving medical professionals a checklist of the steps to safely perform the procedure can reduce these infections by nearly 50%. How, then, should these hospitals go about getting these safety checklists in front of doctors? Each hospital settles on an idea:

Hospital 1 researchers will give all doctors a small, printed version of the safety checklist that they can attach to their badge. Then, when performing the catheterization procedure, these doctors can simply lift up and read their badge if they forget a step. (This is Policy A).

Hospital 2 researchers will hang posters in all rooms where the procedure is performed. These posters display the safety checklist so that doctors who may need a reminder can glance at the wall to remember what to do next. (This is Policy B).

Hospital 3 researchers think of both of these ideas and want to know which one is best. They decide to run an experiment that randomly assigns half of patients to have the procedure performed in a room where doctors wear the checklist attached to their badge. The other half of patients who need the procedure will have it done in a room with the checklist hanging on the wall. After one year, researchers at the hospital will look at the data and permanently adopt whichever policy results in the lowest rate of death. (This is the A/B test).

How appropriate is the decision made in each of these three scenarios? Three-hundred research participants read and considered scenarios like these before rating each decision’s appropriateness on a scale ranging from 1 (very inappropriate) to 5 (very appropriate).

Results

Percent of people who object to Policy A, Policy B, and the Experiment (A/B Test).

Source: Patrick Heck

Figure 1 displays the results: Whereas about 13% of people objected to Hospital 1’s badge policy (by rating it somewhat or very inappropriate), and only about 5% of people objected to Hospital 2’s poster policy, more than 30% of participants objected to Hospital 3’s decision to compare A with B. Even though most people found the badge and poster policies to be appropriate, a staggering number nevertheless thought it was inappropriate to conduct a randomized experiment to learn which policy would save more lives.

We found similar results in four other scenarios, including comparative drug effectiveness trials; employee retirement plan offerings; disclosure policies in direct-to-consumer genetic testing; and autonomous vehicle development (though the effect was substantially weaker in this case).

Takeaways

What did we learn from this research? There are a couple of important takeaways.

Race and Ethnicity Essential Reads

The Impact of Race and Ethnicity on Orgasm

How Parents Talked With Their Children About BLM

First, it was clear that many people have a moral aversion to experiments, even when an experiment is explicitly designed to test two policies or treatments that these same people view as appropriate. This may create friction between leaders, researchers, or policymakers, who are interested in collecting and learning from data, and consumers, patients, and other stakeholders within an organization.

Second, this aversion cannot be explained by idiosyncratic decisions about how our research was conducted or analyzed—we have replicated the A/B Effect using different research designs, measures, and sample populations (including a large sample of doctors, nurses, and other medical professionals).

Third, we observed here (and in our last paper on the topic) that there were no consistent demographic predictors of people’s attitudes toward experiments: sex, age, race/ethnicity, household income, and education could not explain or predict the A/B Effect. We even compared the results between participants who reported having a degree in a STEM field and those who do not and found no differences in ratings of the A/B test.

Finally, it is important to note that are many good reasons to object to some experiments, and not everyone objects to all experiments. In addition to rating how appropriate each decision was, we also asked our participants to rank-order the three decisions from best to worst. This measure revealed something new: in some cases, participants in fact ranked the A/B test as the best possible decision among the three. This encouraging result suggests that there are situations where people prefer to learn from an experiment. Identifying some of those situations is an important future direction for research on attitudes toward experiments, policies, and evidence.