The extent of inconsistency between agencies has many implications, but to concretely determine the extent and manner of disagreements is no easy task.
Even when agencies evaluate the same drug for the same indication, we found from our previous research the range of decision agreement is 58% to 85%. So, a level of disagreement is apparent even when agencies look at the same drug for the same indication.
The variety of factors that influence reimbursement decisions can explain differing approval rates. For instance, agencies might be looking at different clinical evidence, or the economic and political context within each country might differ.
So our mission was to control for potential confounders to tease out the true level of variability between agency decisions. We took a case-study approach to achieve the precise matching we needed, with drugs and indications, and then evidence sources.
Ultimately, we compared fingolimod and erlotinib, categorizing commentary from the world’s agencies (CADTH, HAS, HIRA, NICE, SMC, G-BA, IQWiG, and PBAC) and conclusions on both drugs by theme and type, and qualitatively comparing agencies within each theme.
We found that on many topics, there was strong, although not unanimous, agreement among agencies—three examples include the validity of clinical outcomes, the importance of adverse events, and the appropriateness of comparators.
You Call That Evidence?
Interestingly, two main categories of variability stood out: sources of “evidence” (which we defined as anything cited in a review that logically supported the conclusion), and those criteria that are stressed.
Among the items pointed up were whether trial results from off-label dosages were evidence; the suitability of trial populations; and whether clinical significance or relevance was more valued than statistical significance, or if clinical relevance was even a consideration.
For the entire original study, including a breakdown of each agency, and where their evidence emphasis lies, click here.