Red flags in the new 2024 depression Ketamine paper

A short blog post for a change, with the goal to preempt expected hype around what is basically a null-finding: Nature Medicine just published a new paper on using Ketamine for treatment-resistant depression. I don’t have the 4-6 hours it takes me to vet a paper in detail (requires reading the statistical analysis plan, compare a paper to the clinical trials registry etc), so I will only focus on some issues obvious on a quick read. For those who would like to look at the paper more carefully, I provide a template on how to do so in a recent blog post.

After an initial phase, 168 participants were randomized to either receive a placebo, or one of four doses of Ketamine: 30, 60, 120 or 180 mg, twice a week for 12 weeks. Authors conclude that Ketamine “tablets were effective, safe and well tolerated”. Let us only focus on effective for now. Does the study show that Ketamine is effective, compared to placebo?

The primary endpoint in the clinical trial registration is week 13. Specifically, the registration reads:

“To evaluate the efficacy of extended release (ER) R-107 tablets (30mg, 60mg, 120mg, 180mg) as measured by the change in Montgomery-Asberg Depression Rating Scale (MADRS) score from baseline (Day 1) to Day 92 [..]. The MADRS will be assessed at Days 1, 5, 8, 15, 22, 29, 36, 64 and Day 92 (primary endpoint).”

So we are interested primarily in week 13, and how the four Ketamine groups fare compared to placebo. Let’s look how the authors report these results in the paper abstract:

“The primary objective was met; the least square mean difference of MADRS score for the 180 mg tablet group and placebo was −6.1 (95% confidence interval 1.0 to 11.16, P = 0.019) at 13 weeks.”

What the reader does not see is that the other 3 Ketamine groups — 30mg, 60mg, and 120mg — did not outperform placebo. This means “the primary objective was met” is literally false. You don’t even need to read the clinical trial registration: you can just read the paper itself where the authors write “we hypothesized that an extended-release tablet formulation of ketamine could be an effective and well-tolerated treatment”. This hypothesis is rejected, based on the results: of the around 130 patients who received Ketamine, authors demonstrate only that 18 patients in the final 180mg group showed improvements that outperformed placebo (and if you want to be nitpicky, not all of these 18 improved more than participants in the placebo group either, just some).

This is the kind of cherry-picking of results we see all the time in this literature, and Nature Medicine has a track record of publishing studies with problematic claims like this.

Now let’s look at the actual results — there are 12 of relevance for us regarding the question of whether Ketamine outperformed placebo. How did the Ketamine groups fare compared to placebo in terms of (1) reducing depression symptoms; this is the primary endpoint from above. And then how did the Ketamine groups fare compared to placebo in terms of (2) leading to what is called a ‘response’ (i.e. patients show some improvement), and (3) leading to what is called a ‘remission’ (i.e. patients are much better); these are the 2 secondary endpoints. I will focus on these 12 because they all focus on the primary outcome measure, the MADRS.

1. Depression scores, 30mg: no significant reduction vs placebo
2. Depression scores, 60mg: no significant reduction vs placebo
3. Depression scores, 120mg: no significant reduction vs placebo
4. Depression scores, 180mg: significant reduction vs placebo (p=0.019)
5. Depression response, 30mg: no significant reduction vs placebo
6. Depression response, 60mg: no significant reduction vs placebo
7. Depression response, 120mg: no significant reduction vs placebo
8. Depression response, 180mg: significant reduction vs placebo (p=0.046, very barely significant)
9. Depression remission, 30mg: no significant reduction vs placebo
10. Depression remission, 60mg: no significant reduction vs placebo
11. Depression remission, 120mg: no significant reduction vs placebo
12. Depression remission, 180mg: no significant reduction vs placebo

That means overall, the authors’ own analyses do not show evidence that more people remitted from depression at week 13 compared to placebo. There is barely significant evidence that in the final 180mg group (18 people), depression scores and response was ever so slightly better. This does not justify the overall conclusion in the abstract that Ketamine “tablets were effective, safe and well tolerated”. This is the case particularly because of ~130 patients who received some dose of Ketamine, the 180mg group of people who showed significant improvements over placebo was just 18 people in the end.

Four more notes.

(1) The conflicts of interest in the study could not be more severe: the first author is named on a patent for this particular drug, and the study was sponsored by Douglas Pharmaceuticals. Again, that doesn’t make it a bad study per se, as I have explained in detail in my 45 minute overview video on issues in this literature, but it does require careful vetting of the study.

(2) One of the most prominent experts on vetting clinical trials in this area, Ioana Cristea, reviewed the paper (as identified by Nature Medicine on the paper itself). Ioana, who as a reviewer had access to more information than is publicly available, spoke out publicly today raising several serious concerns. I will provide more information on this once it becomes available in the next few days or weeks.

(3) Independent from other issues that we will see regarding point (2), Ioana did already send over a brief point regarding the concept of ‘treatment-resistant depression’ that authors aim to study in this paper.

“The definition of the study population defines treatment-resistant depression (TRD) without even requiring that the two “failed” antidepressant agents are from the same class, similarly to the esketamine approval trials. However, the esketamine approval trials were planned and conducted more than 5 years ago. In the meantime, a lot of discussion around meaningfully defining TRD has taken place, see for example our piece on the topic. The lax definition used in this trial is in contrast to a recent consensus guideline, ostensibly developed for use in clinical trials, that recommends that the two treatment failures should be with two licensed medications for major depressive disorder with different mechanisms of action. These lax inclusion criteria imply that a significant percentage of the study population is unlikely to be treatment-resistant.”

(4) And finally, while I only wanted to cover effectiveness, it is worth taking a very brief look at safety. Remember, only the 180mg Ketamine group showed any evidence of outperforming placebo. Of the 5 severe adverse events that happened in the entire study, across all 5 groups (placebo and four Ketamine groups), 3 happened in the 180mg Ketamine group (the one in which one would expect the most adverse events because the dose is the highest). Of these adverse events, one was suicidal ideation, and another completed suicide. Of course it is difficult to determine whether this was causally linked to Ketamine, but I would have hoped that authors would grapple with this a bit more than to write than “None of the SAEs was considered treatment related (the suicide was considered by the site principal investigator to be due to the disease under study)”.