Predictive Modeling: Understanding the Outliers

In predictive modeling, it is essential to evaluate how good the model fits the underlying data. The closer the model is to fitting the underlying data, the stronger the model. We recently built a model to predict the reimbursement decisions of the Scottish Medicines Consortium (SMC)  for oncology health technology assessments (HTAs). Our binary model, that is the model that attempted to predict a positive versus negative SMC reimbursement decision, was able to correctly predict 75% of the oncology decisions in the data set. By statistical standards, 75% is considered a “good” level of prediction. But what happened in the other 25% of assessments? Examining where the model “got it wrong” is an important tool in understanding the data and knowing how to apply the learnings and insights.

Our binary model included 13 variables, three of which were significant in predicting SMC positive decisions:

  • Resubmission
  • Presence of a Patient Access Scheme (PAS)
  • Manufacturer’s base-case incremental cost-effectiveness ratio (ICER)

If an assessment was a resubmission or included a PAS, the drug was 1.5 and 2.7 times more likely to get a positive decision, respectively. The probability of a positive decision was inversely related to the base-case ICER. As the base-case ICER decreased, the probability of a positive decision increased. For example, if the base-case ICER was above £100,000, the probability of a positive reimbursement decision was 3%. An ICER of £0 – £10,000 had a 79% chance of gaining a positive SMC decision.

As the ICER decreases the probability of a positive decision increases.

As the ICER decreases the probability of a positive decision increases.

Based on these results, there is still a chance of gaining a positive reimbursement decision when the ICER is over £100,000 and there is still a chance that a drug will not be recommended even though the ICER is less than £10,000. Exploring instances where a drug received a negative decision despite a low ICER or a positive decision despite a high ICER can offer insights into SMC’s decision-making, inform future analysis, and provide actionable insights for HEOR and market access teams.

Negative Decisions Despite Low ICERS

Thiotepa (Tepadina), an orphan drug, was not recommended for use in combination chemotherapy conditioning for aggressive or relapsed or refractory non-Hodgkin’s lymphoma patients prior to allogeneic haematopoietic stem cell transplant. The base-case ICER was £3,083. SMC concluded that the cost-effectiveness results were highly uncertain due to issues with the clinical efficacy data. Specifically, a naive indirect comparison was used in the economic model and the indirect comparison results were highly questionable due issues with internal validity.

Histamine dihydrochloride (Ceplene®), an orphan drug, was not recommended as maintenance therapy for adult patients with acute myeloid leukemia (AML) in first remission concurrently treated with interleukin-2 despite an ICER of £12,617. The SMC noted that histamine dihydrochoride was approved by the European Medicines Agency (EMA) under exceptional circumstances. At the time of the assessment, maintenance treatment of AML was not standard practice in the UK and the SMC expressed considerable uncertainty about the effectiveness of the drug noting that the standard of care in AML has improved since enrollment of the clinical trial. In the economic modeling, the SMC noted that the outcomes for patients treated with standard of care were poorer then might be expected with current practice and that this along with the uncertainty in the clinical efficacy data lead to a negative recommendation.

These case studies highlight negative reimbursement decisions despite preferable ICERs. In both examples, the uncertainty in the clinical efficacy of the drug trumped the economic analysis. The use of a naïve indirect comparison and questions regarding the drugs place in the treatment pathway lead to negative decisions. Orphan drug status and a low ICER alone does not guarantee a positive SMC decision. The reliability of the clinical evidence matters.

Positive Decisions Despite High ICERs

Ponatinib (Iclusig®), an orphan drug, was accepted by the SMC after a resubmission for blast phase chronic myeloid leukaemia (CML) in patients who are resistant or intolerant to dasatinib or nilotinib, for whom treatment with imatinib is not clinically appropriate, or in those with a T315I mutation. The estimated ICER versus best supportive care was £115,835. The SMC noted that the clinical evidence base showed a substantial improvement in life expectancy in the targeted patient population, based on a non-comparative phase II study.

Cetuximab (ERBITUX®) was recommended for the treatment of patients with EGFR expressing, RAS wild-type metastatic colorectal cancer. The ICER without the patient access scheme was £72,846. The PAS was a simple discount and reduced the ICER. Unfortunately, the ICER with the PAS was not reported and therefore could not be used within the predictive model. Because the drug showed improvements in overall survival, had advocacy group support, and had a PAS, SMC issued a positive reimbursement decision.

In both of these cases, improvements in survival, a PAS, and advocacy—in the form of a Patient and Clinician Engagement meeting—played a role in positive reimbursement decisions.

Insights from Outliers

Understanding the results of a predictive model requires a deeper level of inquiry than just interpreting the output. Examining the “outliers” is an essential step to taking the results and turning them into meaningful conclusions. Based on our results, we knew that resubmissions, drugs with PASs, and low ICERs were likely to result in positive decisions. At a high-level, one could conclude that economics are the only variables that truly matter in SMC decision-making. By examining the outliers, we see that while economics matter, having robust clinical data is highly influential.

Stay tuned for our next blog on the implications and use cases for modeling HTA decisions.