What's in a Word? The Use and Misuse of Efficacy and Effectiveness in HTAs

It’s a fairly common thing to see often-used words mixed up or used interchangeably. Sometimes the issue is more of a grammatical error, such as “you’re” and “your” or “it’s” and “its”. In other cases, the meaning can be quite different: being “averse” to broccoli is not at all like having an “adverse” reaction to it. In a study last year we looked at an example in drug development, where the terms “efficacy” and “effectiveness” were being used interchangeably in health technology reviews.

Textbook Definitions

Efficacy describes a drug’s effect in ideal and controlled circumstances (e.g., in clinical trials).

Effectiveness describes the success of a drug in usual or “real world” practices in which not all conditions can be controlled.

For that reason, effectiveness is much more difficult to assess and is often measured by observational studies or calculated by a meta-analysis of clinical trial results.

Interesting Findings

In our study we examined 38 health technology assessments published between 2005 and 2011, by regulators in English-speaking countries. We found it very interesting that 21 of the reviews indicated they were evaluating effectiveness, but only one actually did.

For efficacy, the intent matched the numbers, with 17 of the reviews stating and measuring efficacy. Also, the agencies stated that they measured efficacy and effectiveness at similar rates (45% for efficacy and 55% for effectiveness). They actually measured effectiveness much less frequently (only 3% of the time).

What’s the Big Deal?

Reimbursement agencies based their decisions on health technology assessments that are actually measuring something different than what they state. Agencies are stating that their evaluation of the drug is based on real world circumstances/evidence (effectiveness) but in actuality they are basing their evaluation of the drug on highly controlled efficacy data (randomized clinical trials). This could mean that the data they are using to evaluate the drug can’t be generalized to the real world as it’s intended.

The broader implication here is that if there is a widespread misuse of this critical terminology, then researchers, policymakers, and others may be misunderstanding the implications of comparative effectiveness and HTA reviews.

The first step in solving any issue is first recognizing that it exists, and then trying to understand the best course of action. In the case of efficacy and effectiveness, we’ve pointed it out and will continue to follow the data and report any discrepancies.

Precision Matters

It’s important to use language precisely – it improves communication and helps make the development process more efficient and smooth. And when language is not used with precision are we even aware of it?

Do you know of any other terms that are currently being misused or interchanged in drug development?