8

An economic analysis of cancer drug approvals in the USA was recently published.

Drug regulators’ acceptance of any statistically significant improvement shown in a single randomized trial and lofty drug prices has created a situation where it is now, hypothetically, profitable for a company to run a clinical trials portfolio of chemically inert compounds. While the current cancer drug pipeline is certainly superior to inert drugs, we must rethink market incentives to encourage transformational drug development.

The lead author created a twitter thread to explain the concepts - basically, if a pharmaceutical company ran clinical tests on enough useless/inert drugs, they might expect, by chance, for there to be enough false positives for one to pass the FDA's low bar for approval of oncological drugs and make enough profit from selling the useless drug to cover all the wasted testing.

The authors cautiously assert that they don't think the industry is doing this but the situation isn't much better than it would be if they did.

Does their analysis stand up? Is the profitability of new cancer drugs so high and the approval standards so lax that industry is incentivised to develop completely useless drugs?

  • 6
    I don't see a good argument here. A drug that makes a patient live longer with a smaller cancer and thus less pain shortening the time the patient lives with a bigger cancer and more pain is clinically useful even if it doesn't increase overall life-expectancy. – Christian May 17 '18 at 14:25
  • 2
    @christian True, but the issue is more complex than that. Patients who abandon aggressive cancer treatment often live longer than those who continue with it and have better quality of life because they avoid the side effects of the drugs. Besides it looks like many of the available (and expensive) treatments don't achieve significant measurable benefits of any kind for patients. – matt_black May 17 '18 at 14:59
  • 1
    @Christian And the question whether the majority of drugs work or don't work isn't the point of this question. That question is asked in the linked skeptics.SE question I referenced. This question is about the economic incentives for drug development. – matt_black May 17 '18 at 15:02
  • 3
    Please do not send us to some twitter thread with detailed references to sources. Include reference to and quotes from the sources here so that the question get clearer. – user22865 May 17 '18 at 16:00
  • 1
    I think this Q is excellent. But is there really no better source to start investigating than this awful twitter thing? Trying to "follow" the argument there is almost worse than watching video ("tweetorials" also have a bar of approval way too low). – LаngLаngС May 17 '18 at 18:48
3

Summary:

  • This is a partial answer: I found FDA programmes that are possibly meant by the claim, but I don't have access to the claim paper and cannot check: I may be barking up the wrong tree.

  • FDA can grant a "preliminary" (my term) type of approval after only one clinical trial (and using surrogate endpoints), yes.

  • FDA decides this on the basis of the preliminary results, and considering the severity of the disease in question.
    From a statistics point of view this is important, as this is intended to weed out most of the hypothetical 100 useless drugs, thus reducing the number of false positives. We're talking ≈ 20 indications approved for the breakthrough program per year, and a rate of ≈ 30 % of breakthrough applications being allowed into that program (we cannot conclude anything about the hypothetical new placebo drugs from this as we'd e.g. expect that even if they accidentally show a significantly improved surrogate endpoint in the preliminary data, this would likely be just below those magical 5 % and very unlikely to be highly significant - and the size of expected effect [which would likely be small for the hypothetical placebo drug]).

  • The "preliminary" approval is granted on the condition that a Phase IV clinical trial follows immediately, and if that 2nd clinical trial does not show sufficient evidence of improved efficacy (or shows more harmful side-effects or is too slow), the drug is then disapproved and taken off the market again.


The FDA has a number of "sped up" programs for drug approval, that on the first glance seem to fit the claim (I don't have access to the full paper).

  • Fast track and priority review seem to be burocratic speed up programs, so not what the claim is about.
  • Breakthrough therapy (of which FDA says the name is somewhat misleading) and accelerated approval on the other hand allow preliminary approval based on less/shorter clinical trials (e.g. on surrogate end points). As I understand the description, the drugs are allowed to be on the market already while more clinical data is gathered (Phase IV trial).

  • So in the end, we still have the higher limit of two studies that the claimant proposes in their twitter series.
    For the hypothetical inefficient drug this means for the economic argument that the manufacturer would need to meet the economic breakthrough sales within the first 2 years of the drug on the market (that seems to be the time frame for submitting the phase IV confirmation to the FDA). Which is obviously much harder than reaching this point within the 20 years a patent grants exclusive rights.

Here's an example, where the further studies did not find sufficient evidence to hold up approval, and in consequence the drug was taken off the market again for that cancer:

The FDA explains:

Bevacizumab was approved for metastatic breast cancer in February 2008 under the FDA’s accelerated approval program, which allows a drug to be approved based on data that are not sufficiently complete to permit full approval. The accelerated approval program provides earlier patient access to promising new drugs to treat serious or life-threatening conditions while confirmatory clinical trials are conducted. If the clinical trials do not justify the continued approval of the drug or a specific drug indication, the agency may revoke its approval. In this case, the accelerated approval was based on promising results from one study that suggested that the drug could provide a meaningful increase in the amount of time from when treatment is started until the tumor grows or the death of the patient.
After the accelerated approval of bevacizumab for breast cancer, the drug’s sponsor, Genentech, completed two additional clinical trials and submitted the data from those studies to the FDA. These data showed only a small effect on tumor growth without evidence that patients lived any longer or had a better quality of life compared to taking standard chemotherapy alone – not enough to outweigh the risk of taking the drug.
FDA's Center for Drug Evaluation and Research, which is responsible for the approval of this drug, ultimately concluded that the results of these additional studies did not justify continued approval


  • In the Twitter posts, the claimant names some drugs where they think approval was wrong. I don't know whether they were in any of the fast approval programs. I checked one (neratinib) - the critique of the claimant seems to lie in the interpretation rather than the statistics: he agrees with the FDA on the statistics (improvement from 92 to 94 % invasive disease free survival over 2 years), and on the high risk of adverse side effects (diarrhea). Cost seems to be 125 k$/year. Taking this together, the claimant judges the drug should not be approved. EMA agrees with the claimant and has not granted authorization.
    For the question here, i.e. whether the existing study requirements work as intended, I think nematinib is the wrong example: the FDA may have "messed up " [claimant] the approval - but that seems to be due to interpretation of the study, not due to uncertainty due to limited availability of study data. Claimant and EMA judge the improvement is not large enough to outweigh the adverse side-effects, FDA judges differently: this indicates differences in weighing adverse side-effects against cancer-free survival, but not insufficient study data: they all agree that the observed improvement is significant in the statistical sense, but they disagree on whether the effect size is large enough in practice.
  • The claimant seems to be oncologist, so they have two immediate ways to deal with this:

    • Not recommending/prescribing the drug - which they probably do, and
    • Telling everyone that they think it should not be prescribed - which they do (that's how we heard of this).
  • Keep in mind that in larger (or more) studies smaller differences will become statistically significant, but of course the effect size needed in practice is not affected by this.

  • At least in the tweets, the claimant wants to see more trials. However, from a statistical point of view it doesn't make a difference whether 2 trials with 500 patients each or one trial with 1000 patients is conducted as long as all those trials are well designed and conducted.


(The mechanism and dangers in the claim are discussed in one of the a popular science book about statistics and statistical fallacies by Beck-Bornholdt and Dubben (afaik available only in German), either "Der Hund der Eier legt" or "Der Schein der Weisen" or "Mit an Wahrscheinlichkeit grenzender Sicherheit" - I don't have them here, so I cannot look up the exact quote nor the papers they refer to. In any case, these books were published 2001 - 2005, so the general problem is well known since a long time. They also propose a study design for ongoing comparison of medical treatments that avoids the ethical issue of subjecting one patient group to inefficient treatment.)


The concerns of the claimant are IMHO valid in the sense that one needs to be careful that the described situation does not arise.
However, one step further from the theoretical predictions they do would be to actually test whether the FDA does weed out placebo applications. This is in principle not difficult (scientifically), just expensive (but then, proper research is often expensive): they'd just need to take a number of placebos and go through the approval process.
Another indication that the weeding out of useless drugs does work may be gathered from the difference of possible drugs described in the literature to drugs applying for approval.


Personal statement at the end: this type of problems is not restricted to pharmaceutical industry developing inefficient drugs - it starts much earlier: there's a whole body of literature showing that scientific research practices have lead to large numbers of false-positive findings in publications in (at least) the biomedical field.
For a start, see e.g. the Nature news feature Buchen: Cancer: Missing the Mark, Nature 471, 428-432 (2011), doi:10.1038/471428a or Ioannidis' seminal paper: Why Most Published Research Findings Are False, PLoS Med 2(8): e124, 2005 or Forstmeier et al,: Detecting and avoiding likely false‐positive findings – a practical guide, Biological Reviews, 92, 4, 1941-1968, 2017

Of particular interest for this question is e.g.: Begley, C. G. & Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature, 2012, 483, 531-533
This is Amgen, i.e. pharmaceutical industry, complaining about (lack of) reliability of scientific publications.

  • 1
    This I not bad for a partial answer! I'd point out, though, that one of the allegations in the claim is that recent changes to FDA standards seem to have ignored the need for thorough post-marketing surveillance of drugs (phase IV trials) which weakens their ability to spot duds. – matt_black May 22 '18 at 9:13
  • 1
    Since the paper's just a comment, it's just two pages, and apparently the author who started the Twitter thread felt comfortable posting #1 and #2. – Nat May 22 '18 at 10:00
  • @Nat: ah, thanks - I didn't realize it's just the 2 pages. – cbeleites supports Monica May 22 '18 at 11:38
-3

The idea of testing random substances until you get a statistically significant result by mere chance is valid in a simplified theoretical model. In real world, there are several practical problems with profiting from that approach:

  1. It takes more than a "single randomized trial" to get approval. There are three Phases of clinical trials, and each phase can involve more than one trial, especially the last one.
  2. Clinical trials of new drug are very very expensive. You need a hospital to house the subjects and doctors to monitor them. Even with high drug price, cost of testing hundreds of random substances will get really high.
  3. Clinical trials often compare the candidate new drug to the currently used treatment, rather than to no treatment. A random substance is a lot less likely to beat an actual treatment.
  4. FDA has its own statisticians, who can tell a result of a random search from an actual effect. Same applies to insurance companies that choose to pay for those new drugs.
  5. A cancer patient (or their family) might place substantial value on even a small increase in a chance of improvement. Plus, with millions of people affected by cancer, even a 1% increase in effectiveness means relief to tens of thousands of patients.

Mass-testing of substances does happen at earlier stage in the research: in the test tubes, or in onco-mice. B/c they are cheap, or at least a lot cheaper than testing in human subjects.

  • 2
    Logically, you make a reasonable argument, though (important on this site) not one backed by references. More importantly you have ignored specific details already mentioned in the detailed analysis behind the claim. The FDA does now approve some cancer drugs with just one trial; the comparisons are often very weak; the expense of trials is accounted for in the evidence behind the claim and the claim asserts that their conclusion is still true. In short disagreeing with the claim is not remotely close to refuting it especially if you don't quote evidence. – matt_black May 21 '18 at 14:14
  • 1
    Welcome to Skeptics! Please provide some references to support your claims. – Oddthinking May 21 '18 at 16:28
  • 1
    This answer completely ignores the fact that the FDA changes their criteria. – Christian May 22 '18 at 16:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .