From the GRACE Archives - Originally Published November 21, 2009 | By Dr. Weiss

I’ve been impressed by many things about GRACE and its members. Among these many things are the thirst for new information and better treatments. I also admire the openness of the members both to new ideas, and to questioning these ideas.

Some new ideas succeed and, after careful evaluation, truly advance care. Others fail. As a clinical trialist, I regularly evaluate ideas to consider a clinical trial. Discussion with cards7up led me to realize that it might be helpful to share this thought process with this group—on behalf of yourselves and your loved ones, you vet ideas from the popular media, friends, family and the world wide web. I hope that these thoughts can help you to do so with more confidence.  (The original thread begun by cards7up is pasted below, after the comments)

This slide is from a talk that I gave a few weeks ago to a patient group in Philadelphia. It shows the four basic sources of ideas for a trial:

• Laboratory data
• Clinical observations
• Alternative Medicine
• Successes in other cancer

Some people might be surprised to see alternative medicine as an idea source. It is a common misperception that Western doctors are not open minded to alternative medicine. Taxol started as the bark of the Pacific Yew tree, ginger is gaining acceptance as a nausea treatment, and acupuncture has found a variety of roles in mainstream medicine. What I am closed-minded to is treating patients routinely with something whose efficacy and safety are unproven. So, new ideas are good, as long as they’re tested properly. We first take a new idea to the lab, and learn everything we can about how it might work. We test it on cancer cells in a dish. If it’s the next cure for cancer in a dish, we take it to animal testing where we test for efficacy, safety and dose. If it looks like the next magic bullet for rodent cancer (again, I can’t tell you how many times we’ve cured cancer in various rodent models) then the idea can start the clinical trials vetting process.

The vetting of a trial starts with a funding source. The investigator writes a proposal to the NIH, a pharma company, or a foundation laying out all of the data leading to the proposed trial and explains the basic structure of the proposed trial. If the funding source finds the idea promising and chooses to fund it, the investigator will next write a protocol. The protocol is a detailed guide for how a trial will be run. Once completed, it is submitted to a scientific review board. The job of this committee is to evaluate if the idea has sufficient scientific merit and promise and to ensure that the clinical rationale is sound. If they sign off, it next goes to the institutional review board (IRB) who ensures that the study is ethical and protects human subjects. If they approve the study, it may open, with either a data safety monitoring board (for large trials) or a medical monitor (for small trials) monitoring the trail to ensure subjects’ protection. Finally, at completion of a study, the results, positive or negative, should be presented and published.

Okay, so that’s how an investigator thinks. What about a practicing oncologist faced with an onslaught of new ideas published in various journals, presented at various conferences, and promoted in the media? How does a doctor, working a gazillion hours a week seeing patients choose what to read and what to believe? How does he or she evaluate a paper?

First, it is key to understand the idea of levels of evidence. Different ideas about how to treat patients have different levels of proof behind them. I’ve summarized these in the trial ladder below.

My next major point also answers the question of why your oncologist has never heard of the great new cancer therapy that you read about on the internet, or even the news. Given finite time, we want to read (and you should value most highly) the best, most proven papers. How do we know which ones they are? There’s a hierarchy of prestige levels in the cancer journal world, and authors try to get their papers accepted to the best possible journals. While I’m sure that oncologists will argue about which ones are best, , here’s my opinionated, very rough guide for clinical trials in lung cancer:

• Top tier: Journal of Clinical Oncology (JCO), New England Journal of Medicine (NEJM), Lancet, International Journal of Radiation Oncology Biophysics (“Red journal”)

• Second tier: Journal of Thoracic Oncology (JTO), Chest, Lung Cancer, Journal of the National Cancer Institute (JNCI), Clinical Lung Cancer, Journal of the American Medical Association (JAMA)

Often worth of reading: Other peer-reviewed, medline-listed medical journals

There’s also a hierarchy of conference presentations, with the American Society of Clinical Oncology (ASCO), the European Society of Medical Oncology (ESMO) and the World Lung Conference at the top. Note that findings usually first come out at conferences, then are published in journals — it should raise a red flag if a study is presented but never published.

So, to summarize, it might be worth reviewing brief criteria to know if an idea is worthy of routine implementation. We read high-quality journals. We then evaluate the findings presented and mentally place it somewhere on the trial ladder shown above. Finally, if we’ve accepted the idea that the data is true, we then start the more comprehensible process of asking if the difference is big enough and if it’s worth the toxicity.

In the end, properly vetting an idea is challenging even for trained oncologists. We meet in journal clubs to discuss new findings and we discuss them with colleagues. Those more pressed for time can turn to review articles and conferences. In these media, trusted experts summarize the data and give their interpretation. Time consuming? Yes. Difficult? Yes. But the stakes are high—failure to move a promising idea forward denies better therapy to patients in need. Treating patients with untested therapy denies them more promising therapy, risks toxicity, and denies the opportunity to move properly-vetted concepts forward to advance care.

Comments closed |15 Responses to How to vet a treatment idea

Comments are closed | 15 Responses 

JimC says:

November 21, 2009 at 9:59 am

Dr. Weiss,

Thank you for an interesting and informative post. Though we are all anxious for the next breakthrough treatment, your post helps me stay grounded amid the media hype and “not ready for prime time” preliminary results of new therapies.

It also makes me appreciate just how difficult it is for oncologists to stay at the breaking edge of new treatments without adopting a new idea to soon or too late.

Jim

•  

  Catharine says:

  November 22, 2009 at 6:03 pm

  Dr. Weiss –

  Thank you for an interesting, informative post. It’s good to get an idea of how oncologists think. Your list of trustworthy journals is useful. I hope that my oncologist follows a similar strategy. I agree with Jim: It must be difficult for practicing oncologists to stay at the “breaking edge of new treatments.”

  - Catharine

•  

  Andy says:

  November 23, 2009 at 10:01 am

  Great post, very helpful. Thanks!
  Andy

•  

  Barry Haigis says:

  December 7, 2009 at 6:28 pm

  Dr Weiss,
  Very Informative and well presented.
  Barry

•  

  rachelk says:

  March 9, 2010 at 9:26 pm

  Great info!
  Can you tell me how long the vetting process takes place? For example: On average, how long does it usually take to get approval to do the clinical trial, how long do the clinical trials phases I, II, and III take to complete and publish, and how long does it take to get FDA approval?
  Overall this process sounds very time consuming before a promising “Animal tested” drug can be used by an oncologist. In incurable cancers like SCLC (after relapse and Chemo-resistance), can an oncologist use a drug that shows promise (such as a case study, or do they still have to wait for Phase II, Phase III trials, and/or FDA approval to use it?

•  

  Dr West says:

  March 9, 2010 at 10:39 pm

  It’s quite variable, but the short answer is that it’s typically a matter of many years to go from animal models to late clinical trials and FDA approval. It depends on how quickly trials can get enrolled and how dramatic the benefit of treatment proves to be, but it’s going to be 8-15 years or even more to go from lab to widespread patient use.

  Short of that, if an agent is already commercially available, an oncologist or other physician can potentially prescribe it, and physicians will have varying degrees of inclination to pursue ideas that are still unproven and not standard. There is a real risk that these treatments may be harmful. There is also the question of whether insurance companies will pay for unproven treatments, especially ones that are expensive and that will, therefore, compete with many other potential treatments to be covered that are more likely to be beneficial to patients.

•  

  rachelk says:

  March 10, 2010 at 10:23 am

  Thank you Dr. West. If a treatment doesn’t exist that is known to benefit a patient, like relapsed Chemo-resistant SCLC is it unethical or illegal to try unproven treatments or do patients have to enter clinical trials if they want to pursue that line of treatment? Another question that I have is about clinical trials. Do patients actually have to move to the place carrying out a clinical trial or can the patient’s oncologist prescibe the medication that is being tested in the clinical trial and follow the procedures etc. so that the patients can still remain at thier current treatment facility?
  Thanks, Rachel

•  

  Dr. Weiss says:

  March 10, 2010 at 2:59 pm

  These are all great questions. Some drugs take longer to go from science to popular treatment than others. In part, it depends on how good the treatment is. The recent story with EML4/ALK and PF-02341066 (covered extensively on GRACE) is a good example of a fantastic treatment going through this process much faster than typical. Drugs with small benefit (say, the cetuximab story, also covered on GRACE) tend to move slower.

  Unproven drugs that are FDA approved for some other indication are easier to use than non-approved drugs. Obviously the decision to do so requires careful consideration and patient education to decide together that this is the right thing to do. Here, the major obstacle is getting the drugs for the patient–insurance often balks.

  You ask about both legality and ethics of giving unproven therapies when there are no other options. I do believe in offering this to patients. However, I think that the above vetting process should still be done so that what is given is the most promising available thing. And it should be done in a trial (typically, this kind of conversation centers around phase I trials, which are often first in man). My reasoning for feeling this way is that we should learn if the drug is truly safe and if it truly works. If so, having done it in the phase I trial allows us to know that we need to move forward with phase II, then phase III, and ultimately using the new therapy to help everyone to whom it applies. If the drug does not work, and especially if it is harmful, then we want to learn and disseminate that too–no one else should get the drug, and we should move onto new ideas.

  You also ask about locations of trials. Some trials can only be done at large centers with the capacity to do special kinds of infusion, run special kinds of tests, and be ready to quickly address complications during infusion. Other trials can be opened at other centers. I just opened a head and neck trial simultaneously at my hospital and two others and it may soon open at two additional sites, giving patients there access to the drugs. As more of the new drugs are oral, it becomes easier and easier to open the trials at more hospitals, speeding accrual time and providing access to more patients.

  Finally, you ask about payment on trials. Many trials have some aspects that are identical to guideline standard of care (some of the drugs in the regimen, blood counts, imaging) and one or two elements that are unique. Typically insurance pays for everything that is part of standard of care (and yes, they actually do in practice) and the trial pays for everything else (the experimental drug as well as any tests needed for the trial that are not normally part of standard care).

•  

  JimC says:

  March 10, 2010 at 3:40 pm

  Dr. Weiss,

  To clarify regarding the location of trials, is it generally true that if one wishes to remain at their current treatment facility then the trial must be opened at that facility?

  Jim

•  

  Dr. Weiss says:

  March 13, 2010 at 5:24 am

  Cancer treatments cannot be shipped from a facility with a trial to one without it. So, for example, patients in the suburbs of Philly will sometimes have to drive into the city to get trial therapy at my institution and I have several patients traveling every few weeks to Boston for the PF-02341066 trial until it opens here. That said, many new therapies are oral, decreasing the interval of the needed travel in these cases. Also, going on a trial at another institution does not mean that a patient has to give up his/her oncologist–my patients on the Pfizer trial still see me at regular intervals and after trial therapy is over (for whatever reason) the patients that I treat on protocol often go back to their community oncologists for the next line of therapy, if it is not on trial.

•  

  UscholdM says:

  June 23, 2011 at 11:00 pm

  I am glad to see that you are curious and open to alternative treatments as well as “[being] closed-minded to … treating patients routinely with something whose efficacy and safety are unproven.” HOwever, according to the 2006 article in Business Week called “Medical Guesswork”: “From heart surgury to prostate care, the health industry knows little about which common treatments really work… the portion of medicine that has been proven effective is still outrageously low — in the range of 20% to 25%.” I presume that drugs are an excpetion to this norm, for the FDA requires considerable amount of evidence for approval.

•  

   

  So my question to Dr. Weiss, is: if you are going to only use evidence based treatments, doesn’t that rule out the majority of options available?

  Thanks,
  Michael

•  

  Dr. Weiss says:

  June 24, 2011 at 11:12 am

  Well, I think that there’s a certain extent to which a doctor must apply common sense in making treatment recommendations. For less toxic therapies with a long experience for safety and efficacy, a randomized controlled trial would be impractical and perhaps even unethical. New, potentially toxic, potentially ineffective, expnesive therapies should have a higher bar before being accepted. Deciding which is which requires some common sense, knowledge of science, and the art of medicine.

•  

  jstreet says:

  September 8, 2011 at 3:34 pm

  The elephant in the room is the cost of trials and the very high bar this elephant has to scale which is the three phase trial bar and sometimes the 15 year bar.

  There are several substances, such as shark liver oil and beta 1,3 glucan which keep red blood cell count and white blood cell count normal during chemotherapy but I haven’t seen any oncologists using either one.

  Presumably it is because there is no financial incentive to test them since the first is a natural substance and the second is a complex sugar joined at the first and third molecule, hardly a patentable molecule. Besides, the beta glucans are also natural substances, contained in many mushrooms and are hypothesized to be one of the cancer fighting compound in, for example, reishi mushrooms which are used in Japan to fight cancer.

  The FDA is trying to pass a law as I write, Sep 8, 2011, to make these herbs patentable by drug companies (or at least ban their sale on the open market) but that would probably make matters worse.

  Curcumin, quercetin and many other natural substances are relatively cheap, have NOT gone through Phase I,II and III trials but have shown in many animal studies to be effective against tumor growth and metastasis. Admittedly, letting the pharmaceutical industry make huge profits from these natural substances would at least allow them to be accepted and used as FDA approved medicine but this seems to be a BACKWARDS solution.

  The FORWARD solution would be for the FDA to finance studies for natural substances and allow them to be sold on the free market as they always have been.

•  

  Dr West says:

  September 9, 2011 at 8:53 pm

  jstreet,

  Your points are well taken, and I think it is very appropriate to point out that what gets tested is a product of both what deserves to be tested and what is financially incentivized to be tested. There are drugs that I DO use, like diethylstilbesterol (DES) for prostate cancer, that are not only off patent but no longer even manufactured for patient use (a compounding pharmacy needs to make the capsules for patients), but this agent is definitely effective and will never be properly tested because it isn’t financially feasible.

  It wouldn’t be within the purview of the FDA to fund research on less expensive treatments, but I agree with you that this would be a remarkably excellent way for the NIH to spend money: federal funds to test inexpensive drugs to see whether they are effective as new treatments or as an alternative to far more expensive options being forcefully developed and then marketed because they are wildly profitable. I’m afraid that the NIH has largely moved away from this work, but I think that this is an ideal place for federal research dollars to be allocated.

  Another source that I think most people wouldn’t think of, but I believe would be a very good source for research funding on really inexpensive treatments would be from insurance companies. They are an industry with an incentive to keep health care costs lower. While they escalate premiums higher and higher, they also reap huge profits; if they would fund research that had a goal of identifying whether less expensive treatments and fewer interventions can do as well or better than extremely expensive ones, they could create excellent inexpensive alternatives.

  An advantage of these approaches over making cheap treatments patentable is that the latter just artificially raises the costs for the drugs and therefore for health care, when it would really be far better to support a system that funds the testing of cheap drugs that stay cheap.

  -Dr. West

•  

• Copied from Forum Thread: Poly MVA

• cards7up

• « on: November 17, 2009, 04:50:50 PM »

  Anyone here of this or take it? Has it helped? Developed by a Dr. Merrill Garnett a biochemist and cancer specialist. Any comments Docs?
  Thanks, JC

  « Reply #1 on: November 17, 2009, 05:29:27 PM »

  The American Cancer Society has an information sheet on it: http://www.cancer.org/docroot/ETO/content/ETO_5_3X_Poly-MVA.asp?sitearea=ETO.  According to ACS, Garnett is, "a former dentist with some graduate training in biochemistry."

  Jared Weiss, MD
  GRACE Lung Cancer Lead and Head/Neck Cancer Lead
  Medical Oncologist
  Assistant Professor of Medicine at UNC, Chapel Hill NC
  
  Comments here constitute information, not medical advice, and reflect my own opinion and not those of GRACE or the University of North Carolina.

   

  « Reply #4 on: November 17, 2009, 07:15:38 PM »

  When you check him on the internet, it says just what I posted. Also listed him  in Wikipedia. Many more listings about his company selling this stuff. Scary if he's really only a dentist. JC

  Dr. Weiss 

  GRACE Faculty, Medical Oncologist, University of North Carolina

  November 17, 2009, 08:00:07 PM »

  You've gotten me curious and I just checked out his website ( http://www.polymvaforyou.com/ )--scary is the right word.  They advertise this stuff for everything!  There's very little real data (only two real publications; by coincidence, I published in the same journal in 2004), no ongoing trials, broad claims and unsubstantiated stories of great results.  Equally disturbing is the frank promotion and sales (for both you and your pet).  You give me a good idea for an article--how to vet a novel treatment idea (something that I do regularly in deciding which ideas from alternative medicine, the lab, clinical observations, and advances in other cancers are worthy of a clinical trial in lung and head and neck cancer patients).  I'll start working on it--perhaps I can run it by you before posting?

  
  Comments here constitute information, not medical advice, and reflect my own opinion and not those of GRACE or the University of North Carolina.