Posted: April 2, 2020

History Bias, Study Design, and the Unfulfilled Promise of Pay-for-Performance Policies in Health Care


In June 2015, Preventing Chronic Disease published "How Do You Know Which Health Care Effectiveness Research You Can Trust? A Guide to Study Design for the Perplexed," which used simple graphs and easy-to-understand text — in 5 case studies — to illustrate how powerful biases, combined with weak study designs that cannot control for those biases, yielded untrustworthy findings on influenza vaccination policy, health information technology, drug safety, prevention of childhood obesity, and hospital safety ("mortality reduction") programs. The target audiences for that article were policy makers, journalists, public health and medical trainees, and the general public; the primary goal was to explain how weak or strong study designs fail or succeed in controlling for biases. In the Editor's Note, we promised to add to those examples of common biases and research designs to show why people should be cautious about accepting research results — results that may have profound and long-lasting effects on health policy or clinical practice, some of which could be detrimental to health. In this sixth case study, we revisit one of the most common and dangerous threats to research validity: history bias (ie, researchers' failure to consider relevant events or changes that precede an intervention or co-occur while it is in progress). Studies that fail to control for history can mislead policy makers and clinicians. The pay-for-performance policy used to illustrate history bias in this article is sensitive to this powerful bias, because medical practice is always changing as a result of factors unrelated to policy. Without investigating changes in a study's hoped-for outcome over time both before and after the policy or intervention being studied is implemented, investigators will probably attribute those changes to effects of the policy they are studying, causing billions of dollars of waste implementing such policies worldwide.


Huseyin Naci, PhD, London School of Economics and Political Science
Stephen B. Soumerai, ScD, Harvard Medical School

Learner Audience

Undergraduate Medical Education (UME), Graduate Medical Education (GME), Continuing Medical Education/Continuing Professional Development (CME/CPD), Interprofessional


Systems Based Practice

Resource Type(s)

Distance Learning - Synchronous

Instructional Method(s)

Discussion, Small Group [≤12]
Discussion, Large Group (>12)


Population Health, Public Health Sciences, History Bias, Pay-for-Performance, Radomized Controlled Trials, Rogorous Systematic Reviews