Imagine you are a group of scientists and want to find out whether a novel drug X works on a specific problem Y. You run the following study:
- You enroll a small sample of 14 participants who have the problem Y.
- You give these 14 participants your novel drug, but you do not enroll a second group who received placebo medication, which means that any effect you find may be due to the placebo effect.
- On top of that, all your 14 patients keep using a well-accepted medication for problem Y while taking the novel drug X, meaning that any changes you find may be due to their normal medication, not X.
- You measure improvements of problem Y 240 minutes and 3 months after the treatment.
- You find evidence of improvements for 7 of 14 people 240 minutes after treatment, and for 2 of 14 people 3 months later; it is unclear whether this is due to placebo, normal medication, or other reasons.
- You write a scientific paper with the strong conclusion: “repeated doses of X rapidly and robustly decreased Y.”
This is exactly what what Ionescu et al. (2016) did in a new study entitled “Rapid and Sustained Reductions in Current Suicidal Ideation Following Repeated Doses of Intravenous Ketamine: Secondary Analysis of an Open-Label Study”, published in the Journal of Clinical Psychiatry.
Ketamine as treatment for depression or suicidal ideation
The authors tested the viability of Ketamine as a treatment of suicidal ideation, as described in the summary above. Wikipedia sums up nicely what Ketamine is:
“Ketamine is medication mainly used for starting and maintaining anesthesia. It induces a trance-like state while providing pain relief, sedation, and memory loss. Other uses include for chronic pain and for sedation in intensive care.”
Ketamine has become a hot topic of study in the last years, both as depression treatment, and as rapid acting treatment for suicidal ideation. Three reviews so far provide very mixed evidence: Wilkinson & Sanacora (2016) concluded recently that evidence of Ketamine effects on suicidal ideation is “very preliminary”, because studies so far were very small, often not placebo controlled, and samples often do not contain patients with high levels of suicidal ideation due to ethical concerns. Iadarola et al. (2015) draw somewhat more positive conclusions, as do Reinstatler & Youssef (2015), but the actual studies they describe show that effects of Ketamine on depressive symptoms or suicidal ideation rarely last beyond 24 hours – it’s not secret that depressive symptoms and suicidal ideation do. A fourth meta-analysis by McGirr et al. (2015) of 7 studies on the efficacy of Ketamine treatment on depression shows consistent effects lasting up to 7 days; however, they find some evidence for publication bias, the average sample size per study was 26 (meaning 13 per treatment arm), and 7 days is not a sufficient time period to investigate treatment effects. To quote Prof Colleen Loo from the University of New South Wales, a Ketamine researcher herself: “Feeling better for a few days is no good with a chronic disease like depression”.
Ionescu et al. 2016
Let’s return to the study of Ionescu et al.: they showed that depressed patients with suicidal thoughts who were slightly sedated via Ketamine showed somewhat reduced levels of suicidal ideation minutes to hours after infusions, but that there were no lasting effects. It is possible that similar effects would be achieved with all sorts of psychedelic or psychotropic drugs – or even placebo – but that doesn’t make them effective treatment options for depression or suicidal ideation. Similarly, giving Ketamine to people with cancer may relief symptom burden for a few hours, but that likewise would not mean it is a reasonable cancer treatment.
While half of the patients showed improvements on suicidal ideation 240 minutes after infusions, only 2 of 14 patients showed improvements over the course of 3 months. Nonetheless, the authors conclude:
“Despite ketamine’s short half-life (approximately 2.5 hours), its antisuicidal (and antidepressant) effects are sustained well beyond the elimination of the drug, in some patients.”
There is no evidence at all for this conclusion in the paper, because there is no way to know whether this improvement was not a placebo effect. In fact, the placebo effect usually leads to improvements in 1 of 3 patients, meaning that 2 of 14 is less than would be expected in a placebo group (Khan & Brown, 2015). Moreover, there is no way to know whether this improvement was due to Ketamine because all patients continued to use their standard antidepressant medication. Finally, another very large concern is regression to the mean, which very likely contributed to the reduction of suicidal ideation, because patients were selected on suicidal ideation.
Despite these serious shortcomings, the main conclusion in the abstract reads:
“In this preliminary study, repeated doses of open-label ketamine rapidly and robustly decreased suicidal ideation in pharmacologically treated outpatients with treatment-resistant depression with stable suicidal thoughts.”
And the title reads:
“Rapid and Sustained Reductions in Current Suicidal Ideation …”
Sorry, as much respect I have for the authors – many of them are great names in the field, and deserve great respect for their brilliant scientific and clinical contributions – conclusion and title are simply incorrect, and I wish the reviewers or editor (or one of the 14 authors) would have pointed that out. There is no “sustained reduction”, and the paper provides little support for the viability of Ketamine as a “robust” or “rapid” treatment for suicidal ideation. Think about like this: if a doctor prescribed you a drug and told you it leads to “rapid and sustained reductions” of your condition, and you found out that it only shows a sustained effect in 14% of patients – for which the effect is very likely not even due to the drug – you would be a very upset patient.
The proper conclusion from the available evidence is: “In our underpowered study that is not representative of depressive patients, only half of the patients showed improvements, for only a few hours, and nearly nobody showed lasting improvements. Due to the lack of a placebo group, and because patients continued their standard medication, interpreting the results of this study as supporting the viability of Ketamine as treatment for suicidal is scientifically impossible.”
To be fair, the authors call the paper preliminary (albeit not in the title), and the sample was a group of treatment resistant patients, making this a difficult study. And obviously, helping even 1 in 10 depressed patients should make it worth to explore new treatments, and suicidal ideation is among the most relevant depression symptoms we need to address, and address quickly. The approach taken here – trying to address symptoms instead of syndromes – is consistent with a novel research framework focused on individual symptoms, and very much worth supporting. But how informative are studies without a placebo group? One may argue that they pose unacceptable ethical problems in terms of ratio of useful information to patient risk and resource waste, because literally 0 reliable information can be gained from a study in n=14 without a placebo arm. And the fact that a placebo group is ethically problematic in severely depressed patients is no counter argument – in that case, we can enroll a second group of patients into a second treatment arm, and give them, for instance, benzodiazepines.
For now, future studies will cite the study as evidence for the efficacy of Ketamine, which is inconsistent with the data of the study. Moreover, the study has been rather widely discussed by media as providing evidence for Ketamine efficacy, which, again, is inconsistent with the data. And this in part the authors’ fault – due to the strongly exaggerated conclusions throughout the paper, including the title.
A final concern is the stunning conflicts of interests section. Then again, it is driven primarily by one author, and the majority of authors report no conflicts of interests, so this should not be overinterpreted. Nonetheless, seeing that studies with conflicts of interest conclude 5 times more often than independent investigations that drugs are efficacious (Perlis et al., 2005), conflicts of interests should always be a concern and taken seriously.
General concerns about antidepressant trials
It is helpful to understand this commentary in a much larger context – the scientific and methodological quality and rigor of antidepressants trials in general. The study by Ionescu et al. is the sixth study in only a few months that appeared in a high-impact psychiatry journal where the main conclusions are at best exaggerated (I wrote about the other studies here: 1, 2, 3, 4, 5), and my concern is that these studies are not really moving the field along towards the goal we all share: finding better treatments for people suffering from mental illness. What we need are studies with much larger samples, with placebo groups where possible, and otherwise with multiple treatment arms so we can compare the efficacy of drugs. We also need to check on people for longer periods – it is crazy that the majority of depression trials do only last 6-8 weeks, which is not an adequate timeframe for the large majority of patients. Finally, we need to follow Ionescu et al. in studying individual symptoms instead of syndromes. Depressed patients differ dramatically in their symptoms, and different treatments likely have very different positive and negative effects on different symptoms.
And I understand that getting funding for such studies is a problem, and the blame here is certainly not predominantly on the scientists and clinicians involved in conducting the clinical trials: we need journals that are much more rigorous in their peer review, we need better methodological standards for clinical trials, and we need funding agencies actually spending money on larger, high-quality studies, instead of multiple smaller studies without placebo groups.
Pingback: Treating depression with psychedelics: red flags and FAQ - Eiko Fried
Pingback: When small samples are problematic – Eiko Fried
Pingback: Small samples can be inherently problematic – Eiko Fried
I think every flaw from this paper has been put on the scene (in a very eloquent way btw). What I take away from it is that the publication process is flawed. This paper should not have been published in a high impact journal but it did. And researchers know that this kind of research is flawed but still do it because it will be published (at least somewhere but maybe even in a “good” journal).
What I do find less agreeable of this post is the underlying generalising that the concept of ketamine to be a potential helpful medication is wrong because of bad papers on the subject. They have been papers who have avoided the flaws you mentioned above and show indications that ketamine could be helpful in depression/suicidal thoughts. A reference of a review/meta-analysis of those papers serving as an example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5098773/pdf/ndt-12-2859.pdf
It often find that it is easier to debunk medical research because of the high standards of it (everything that is not top-of-the-bill is totally not ok. Don’t get me wrong: It’s a good thing because of the relevance of medical research) but medical science can be hard (financial realities, practical realities, ethical realities, etc.). The scientific concept should not be judged on the quality of the papers but on the actual results of the good research that has been done upon it (so disregard the bad papers ;) ). Those papers present a waste of time, money and energy.
I’m not an expert on this kind of research (RCT, ketamine, etc.) and I like the blog, just felt the urge to respond as a researcher in psychiatry :)
Thanks Thomas. I didn’t want to disregard the possibility that ketamine may work, but looking at the current evidence (both for this particular paper here, and for the 2 meta analysis I talk about), it doesn’t look as promising as biological psychiatry makes it look like. And we can agree to disregard the papers that do not offer much evidence (e.g. papers without control group), but I wrote the blog mostly because I was upset about the current standards in psychiatric RCT research, using ketamine as an example.
Two points regarding the paper you refer to (thanks for sharing, it’s a great resource and I hadn’t been aware of it):
1) Giving people a psychotropic substance will decrease worries and concern because you numb them emotionally. But that is not akin to treating depression. The fantastic Joanna Moncrieff has written extensively about this.
2) Second, why on Earth are there even studies that test the effect of a drug for Major Depression with a follow-up period of max 7 days? It’s not an adequate time frame to investigate the impact of a treatment on patients’ well-being. 7 weeks? Maybe. 7 months? Now we’re talking. It’s a chronic and debilitating disorder. It’s like saying “we have a new promising treatment for cancer” but only following patients over 7 days.
3) The actual evidence from the paper of ketamine efficacy as a treatment is here:
The CI of nearly all studies overlaps with an OR of 1, meaning they are nearly all non-significant. The impact on improvement over a very short time period (24 hours) is remarkable, but that only means that ketamine is a drug that makes patients feel better for 24 hours. Similar to most other psychotropic substances, many of which do not require an infusion.
So I remain unconvinced. Solution 1: investigate the efficacy of ketamine on longer timeframes. Solution 2: stop calling it treatment, and call it “last resort very short time intervention for highly suicidal patients”.
I can see you are upset about the RCT standards in biological psychiatry (and I can expand that to the rest of medicine actually). As said before, I have no experiencing in doing RCT’s myself but I can see that my colleagues really have a hard time performing such studies. So many rules, so hard to get a design approved, to get funding for longitudinal follow up, etc. No wonder that the RCT-design is considered a golden standard because it is so hard to get it done right :) (but that’s no excuse for doing it bad of course ;))
Regarding the “hyped-up” promises of biological psychiatry, I can refer to this article which investigated the actual words used in titles/abstracts of pubmed over a long period (spoiler: we use more stronger emotional words, positive and negative ones. Check the pictures in the paper: it’s groundbreaking ;) https://www.ncbi.nlm.nih.gov/pubmed/26668206). It has become a research cultural phenomena where (biological) psychiatry and psychology and many other disciplines has fallen for.
And on your two (actually three ;) ) points on the previous article
1. Maybe a matter of semantics but it think you can still call it a treatment even if it’s short-lived and just elevates the burden of symptoms for a while. A painkiller is also considered a treatment while it is only short lived and does not treat any causes of pain. Maybe they should call it a treatment for severe emotional distress (depression symptoms, suicide thoughts, anxiety symptoms, etc.) or something
2. In this argumentation I can agree more than on the first point. Major Depression Disorder is a acute-to-chronic disease model so treatment research should include a more distant time point.
I’m not sold either on the idea of using ketamine on a regular basis for major depressive disorder but mainly because we have no long term efficiency research and maybe because it is not a treatment for major depressive disorder but for elevating, short time, severe emotional distress (as suicidal thought/plans are)
PS: I must say you answer very quickly :)
Hi Thomas, absolutely agreed that it’s difficult to get funding and conduct proper research in this environment.
Hence my statement in the blog above:
“And I understand that getting funding for such studies is a problem, and the blame here is certainly not predominantly on the scientists and clinicians involved in conducting the clinical trials: we need journals that are much more rigorous in their peer review, we need better methodological standards for clinical trials, and we need funding agencies actually spending money on larger, high-quality studies, instead of multiple smaller studies without placebo groups.”
I think that fairly acknowledges that same point.
Agreed on the treatment point, but I wasn’t trying to be semantic: there is a difference between alleviating symptoms and trying to cure a disorder—maybe we can agree on that nomenclature (treatment is indeed a bit unclear). Ketamine is regularly sold as the latter instead of the former, and I just don’t think there is evidence for that.
What a fantastic post! It elucidates the challenges facing policy makers, drug discovery, researchers, and clinicians in attaining meaningful evidence.