Unfulfilled Promise

Early claims of success of “pay-for-performance” incentives have not paid off

Image: iStock

Image: iStock

Early studies have suggested that paying doctors bonuses for meeting targets for certain health care measures would lead to improved health outcomes for patients, but the findings were not repeated in later, more rigorous studies, according to new research.

In a paper published June 23 in Preventing Chronic Disease, Stephen Soumerai, Harvard Medical School professor of population medicine at Harvard Pilgrim Health Care Institute, and Huseyin Naci at the London School of Economics analyzed the latest results.

This study is the latest in a series of papers from Soumerai and colleagues exploring how uncorrected bias and unreliable study design may contribute to flip-flops in health care recommendations and policies, and offering recommendations on what researchers, educators and journalists might do to address these issues.

Get more HMS news here

Soumerai discussed the paper and the challenges of research design with Harvard Medicine News.

HMN: Why are the promising conclusions from early studies of pay-for-performance so different from the findings of later studies?

When the initial national programs in pay-for-performance were rolled out, many early studies suggested that the policies were contributing to improved health outcomes, including findings which suggested that giving physicians cash incentives to take patient blood pressure contributed to improved patient mortality rates.

The problem was that these studies didn’t correct for other changes that were taking place in medicine at the same time. Failing to account for these ongoing baseline changes is called history bias. Uncorrected, these biases led to false, optimistic conclusions because the policies were taking credit for improvements in medical care that were already occurring.

Subsequent, stronger research—including quasi-experimental studies, randomized control trials and systematic reviews that can control for these biases—consistently overturned the early, optimistic findings.

One international systematic review found that not only was there little evidence to support pay-for-performance’s effects on quality of medical care but some studies found that it sometimes had the unintended consequence of discouraging doctors from treating the sickest patients.

HMN: What happened next?

Unfortunately, that initial, weak research on the supposed success of pay-for-performance in health care was published in top medical journals and hastened the adoption of these policies around the world.

Despite its unfulfilled promise and discouraging evidence, this costly and ineffective approach to improving health care is a widespread component of current national and international health care policies. It is entrenched in many policies created by the Affordable Care Act. This has led to billions of dollars in wasted resources around the world.

Remarkably, most of these initiatives were started after the evidence of failure or very weak effects were published.

A strong interrupted time series study of the percentage of study patients who began antihypertensive drug treatment from January 2001 through July 2006 (originally published in doi: 10.1136/bmj.d108.) Dashed line indicates when the United Kingdom’s pay-for-performance (PfP) policy was implemented (April 2004). Controlling for the improvement in treatment that was happening anyway over many years, there was no additional benefit of the $2 billion PfP policy. A single average point before and after the program (week-post design) would have contributed false hopes of an improvement in quality of care.

HMN: How common is this?

This isn’t an isolated incident; it’s a pattern that repeats itself throughout health care effectiveness research. Weak research designs are the most common reason for biased health care effectiveness research.

This paper continues a line of research we’ve been pursuing that shows how stronger research designs can avoid these costly mistakes.

HMN: What’s the solution?

Science can be a useful tool for designing and implementing effective, efficient health care, but only if we use reliable research methods such as randomized, controlled trials; interrupted time-series designs; and systematic reviews—and if we conduct rigorous pilot tests of expensive policies.

Investments of private and taxpayer funds should be based on solid evidence of safety and efficacy. The alternative—the present system—relies on weak and uncontrolled research designs, misleads policy makers and the public, and will ultimately lead to unsustainable costs, unhappy clinicians and policies that may damage—rather than improve—the quality of medical care.

Everyone involved in the process that brings this information to light—the researchers who design the studies, the journal editors who accept and reject papers and the mainstream journalists who cover health care and policy—all have a responsibility to get this work right.