Why preliminary results are dangerous | The Body of Evidence

Why preliminary results are dangerous

A prominent trial for a weight loss drug was scuttled because people got too excited about preliminary results.

Read the full article at http://montrealgazette.com/opinion/opinion-how-a-promising-weight-loss-d...

*********************************************************************** 

When it comes to success, there are no shortcuts. We learned that lesson the hard
way when an important trial of a new weight-loss pill imploded in spectacular
fashion. The LIGHT trial, which looked at the
cardiovascular effects of the weight-loss pill Contrave
(buproprion/naltrexone), was mothballed because of a breach of confidentiality.
The trial now has to be redone, at a price tag of $200 million.

 There are many pills that supposedly help you lose weight. Most are nothing more than
placebos sold on late-night infomercials. But some pills represent true
scientific innovation, and actually work. They just have some bad side effects.

 Contrave has been on the market since 2014, thanks to a provisional
approval process the U.S. Food and Drug Administration uses. If its
benefits end up outweighing the risks, it will fill a critical void.

 The early results were promising: It did seem to work. Participants lost somewhere
between five and 10 pounds compared with the placebo, which is nothing to
sneer at. Its problem was a slight elevation in blood pressure and heart rate.
Obviously, a weight-loss drug that led to heart disease would be as unpopular
as its predecessors.

 The LIGHT trial was supposed to determine whether an increased cardiovascular risk
really exists. Starting in 2012, it randomized roughly 9,000 patients and
planned to follow them for five years. When the trial was 25-per-cent complete,
the Food and Drug Administration in the U.S. mandated an interim analysis. If
there were an excessive risk to Contrave, the trial would be stopped. If not,
it would proceed.

 The interim results were surprising. Contrave posed no excessive harm. In fact, the
data said it reduced heart attacks by 40 per cent. The scientists were
unflustered, and prepared to continue with the trial. Then the people at
Orexigen, which makes Contrave, seem to have lost their minds.

 Against the advice of the data-monitoring committee, the interim analysis was shared
with people at Orexigen, who shared it widely within the company. The company
filed a patent and a report with the Securities and Exchange Commission for
its new "wonder drug." But the 40-per-cent benefit was an illusion.

 To understand why, take a quarter and start flipping it. Flip it 10 times, and you
should get five heads and five tails. Of course, you might get a six-four split
or even a seven-three split simply by chance. But if you flipped it 1,000 times
you would be very close to even; maybe 503-to-497. It's the same quarter, but
the more you flip it, the more those random fluctuations smooth out and get you
closer to the theoretical truth of a 50-50 result.

 In small experiments, random chance can wreak havoc on data. Small statistical
blips can be amplified into widely inaccurate results. It has happened to me
often.

 Case in point: The 40-per-cent decrease in heart attacks was smoke and mirrors. When
the trial reached the halfway mark, that benefit shrank to 12 per cent, and
two-thirds of the way in, it had disappeared to basically zero. But by then,
the damage was done. By filing patents, Orexigen had disclosed confidential
trial data. The study investigators felt that with their preliminary
data made public, they could no longer run the trial, and pulled the plug
on it.

With the trial only partially complete, it's unclear whether Contrave is safe. It
might increase heart disease by as much as 34 per cent, but that estimate is uncertain.
The trial has to start over again, and the price tag is the aforementioned $200
million. Expect results in 2022, or five Star Wars films from now.

 A number of things went wrong here. The FDA shouldn't have allowed Orexigen
access to the data, and Orexigen shouldn't have disclosed it publicly. But,
more importantly, people should have understood the statistical dangers of
interim analyses and allowed the trial to be completed as planned. In other
words, they shouldn't have counted their chickens before they hatched.