Menu Home

The problem with observational studies

Researchers in biomedicine often use the following logic:

I. In critical illness, biomarker X is elevated compared to healthy controls.

II. A retrospective observational study shows that the higher the level of X, the greater the risk of death.

II. It follows then, that high levels of X are harmful and that efforts should be made to reduce X in critical illness.

IV. Excitement builds for a therapy that blocks X in order to improve mortality in critical illness.

with the following results:

V. A randomized controlled trial is performed, and fails to find benefit from the X-blocking therapy.


One possible answer is that biomarker X is not part of the problem, but is part of a coordinated host defense, shaped by natural selection to help the organism respond to a physiological challenge.

If it is a host defense, we would expect a positive association between biomarker X and mortality. Most host defenses have costs and benefits. Because of those costs, defenses are expressed only as much as is required, usually in proportion to the degree of danger confronting the organism.

The logical trap that ensnares many researchers of critical illness is the supposition that X causes death. Of course, observational studies cannot show causation, only association, and we can explain the failure of randomized controlled trials as due to confounding. In many cases of confounding, the confounder is a bystander, not in the causal pathway of the disease. A classic example is the now discredited idea that coffee causes lung cancer. Early observational studies suggested that coffee drinking increased the risk of cancer. Later work discovered that in fact heavy coffee drinkers were more likely that non-drinkers to smoke cigarettes. Cigarette smoking was the true culprit of lung cancer. Coffee drinking was an innocent bystander, not the cause. In those observational studies coffee drinking was a confounder.

What has been confusing to many who study critical illness, is that sometimes X can sometimes cause death. This leads to misapprehension that X is part of the causal pathway of disease. And it explains why millions of dollars continue to be devoted to finding X-blockers in sepsis and critical illness. Some can’t shake the idea that these responses are harmful.  What is missing in this logic is that the evolution of costly defenses is expected in response to injuries and infections that have a high lethality.

Using animal models, it is possible to unmask the costs of host defenses. Those costs can be considerable, sometimes leading to death of the organism. However, it does not follow that the responses should be blocked during critical illness in humans. Blocking the benefits of those responses is likely to result in further harm. Hence, we observe that randomized controlled trials of X-blockers often cause worse outcomes and do not work.

In our hypothetical example, X is an adaptation, with both costs and benefits, and is expressed flexibly, depending on danger signals from the environment. These naturally selected physiological responses introduce a special kind of confounding in observational studies. X is not a bystander in the disease (like coffee drinking in smokers), but a participant. Although not in the causal pathway, the costs of X can produce symptoms and may occasionally hasten death. On the other hand, a knee-jerk response to block X is ill-advised, because the benefits are eliminated along with the costs.

A better approach, informed by natural selection, is to consider how to manage the costs and benefits of X, and find ways to reduce morbidity associated with its costs.

More to come on this topic.





Categories: Uncategorized

Joe Alcock

Emergency Physician, Educator, Researcher, interested in the microbiome, evolution, and medicine

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: