Real World Evidence... "I Know It When I See It!"

This week we will be attending the Evidence 2013 / Market Access World USA 2013 conference in Boston. The theme of the conference is, “Providing real world outcomes for pharma and stakeholders,” and in preparation, we decided to immerse ourselves in the subject of “real world evidence” (RWE). These days you can’t blink without seeing a conference, panel, or press release, or hearing an advocate talking about the importance of RWE – so we figured it should be easy to find meaningful articles about the definition and value of RWE. But when digging deeper, specifically around an actual definition… things get a bit murky.

When it comes to RWE there is certainly no shortage of views on:

  • Why we need it
  • Why it’s important
  • Why we should be collecting it, investing in, and discussing it
  • Why EVERYONE should be using it

And along with the desire for more RWE, there’s a definite sense of purpose, including discussion about why:

  • We need to collect it now
  • We need to include it in our processes for determining value
  • We need RWE to understand what is REALLY going on
  • We need it, because everything we have done before has been missing this “secret” ingredient

But in our quest for knowledge we noticed something that was actually quite surprising. Not one organization, be it a public payer, pharma company, consultant, conference, public advocate, or even Wikipedia has a succinct, consistent, or meaningful definition of what “real world evidence” actually is.


(What do you think, should we contact Wiki and ask for a RWE page to be created?)

When looking at the definitions out there, the one thing they do have in common is that RWE is considered to be anything outside of a controlled experiment. After that… the gloves are off. The definitions then veer toward the source, which has its obvious problems because no two sources should be valued the same.

Is the RWE data from:

  • Electronic Medical Records (EMR)
  • Patient registries
  • Phase IV trials
  • Discussion between pharmaceutical companies and payers
  • Claims data
  • Public databases
  • Epidemiology data
  • Case studies
  • Press releases
  • Etc.

You get the idea…

And if all of these sources are encompassing this new paradigm (rather than the definition) of RWE, how do we determine how reliable, verifiable, or replicable the information is? And how do we determine where in the hierarchy of evidence this information should fall?

Specifically, how should a source be judged? By the raw data, by the people who collect it, by the intention of the collection, or by the context? And once that’s determined, how should it be weighed?

We are by no means taking a swipe at RWE. Quite the opposite, we are hoping to open a real discussion around what people are talking about and what they hope to accomplish by using more of it. Also, since there is so much interest, money, focus, time, and energy being spent on things like collecting, using, and advocating for RWE… shouldn't we at least make sure that we have a precise definition and clear goals around what we hope this data will accomplish?

We are strong advocates for incorporating disparate domains of information to gain a better sense of what is actually happening in a given situation. And because we spend a great deal of time normalizing data, we also recognize that context really does matter… and without precise definitions, the end goal can be sometimes lost in the process or lost to semantics.

There is an important distinction between RWE and experimental evidence. The most important advantage of experiments is NOT their control of potentially confounding influences, it’s that they measure what they are hypothesizing using methods in which the results should be replicable and verifiable. That’s the ideal, of course. But then again, in a recent article “Trouble at the Lab,”The Economist reported on a number of concerns about the fundamental reliability of published scientific studies - we'll save that topic for another week!

Since RWE is collected by a variety of agents for many different reasons, this means that the quality of the data could be low or unknown. Or, the many different reasons may mean that the information is not well suited for the particular purpose.

Simply put, when seeking RWE the two most important questions we should be asking are:

  • Is the data accurate and applicable to the question at hand?
  • Why does this data exist and why was it collected?

In fact, those are the most important questions to ask of all data in almost any context. The concern we have is that with a vague definition of RWE, the answers will often disappoint. Not all RWE is created equal…