I just saw that the Daws et al. 2022 paper that was heavily discussed in 2022 and 2023 is still online, and Nature Medicine neither published criticism that has been raised, nor did they publish an expression of concern or correction.

I originally came across the paper when one of the authors wrote about it on Twitter, summarizing their work with the claim that ‘psilocybin liberates the entrenched depression brain’, which to me sounded more like a homeopathy advertisement than the summary of a scientific paper in a Nature journal.

So I wanted to reiterate that I consider some of the issues in the paper quite serious. For the record, I have not met any of the authors in person and in all my online interactions with Robin (the only author I have interacted with at all), he was always exceptionally friendly, polite, and forthcoming. So this is a discussion about the content and arguments put forward in the paper and in the discussion that followed, not about the authors.

Having said that, I was very surprised to see that the editorial board has not taken any action. The problems in the paper have been laid out very clearly by others, most prominently in a letter to the editor that appears to have been rejected by Nature Medicine, written by Dr Doss, Dr Barrett, and Dr Corlett, as well as a Motherboard story by journalist Shayla Love. Note that the original authors also wrote a response to the rejected letter to the editor, which I will get to later.

1. One-tailed tests

The authors use a one-sided rather than two-sided test in the paper, which increases their chance to find significant results. When asked about this, the last author of the paper replied on Twitter in a way that reads to me like a direct admission of p-hacking: we chose one-sided tests because we had peaked the data and it worked better.

I want to note that in an exceptionally rare occurrence, a reviewer of the paper actually spoke up in public and said he had missed this and would have asked the authors to rectify this (quote from the Motherboard story):

Others more quietly reevaluated things: When Motherboard reached out to one of the paper’s reviewers, he shared that when he originally reviewed the paper he had not noticed the “one-tailed test,” a statistical hypothesis test used in the paper that the Doss comment took issue with. “I did not pick up on the fact that there were one-tailed tests in there in reviewing the paper, or I would have asked them not to do that,” said Jared van Snellenberg, an assistant professor of psychiatry at the Renaissance School of Medicine at Stony Brook University.

2. Outcome switching

The authors appear to have engaged in what is called outcome switching, which I explain in some detail in my new video. The gist is that you have to register a primary outcome for a trial, which was the Quick Inventory of Depressive Symptoms (QIDS), but the authors instead analyzed the Beck Depression Inventory (BDI) in the Daws et al. 2022 paper. In the response by the original authors to the rejected letter to the editor, they write:

Again, this seems a direct admission of outcome switching.

3. Lack of significant interaction

When you want to show that a treatment outperforms another treatment — in the case of the Daws et al. 2022 paper, psilocybin compared to an antidepressant — you statistically test whether there is an interaction effect. The authors do not do that and still conclude, without proper evidence, that psilocybin outperforms antidepressants. This contrasts with very basic statistical standards in the RCT literature, and is a serious problem.

Again, the investigation of Shayla Love showed that one of the reviewers had actually raised this in the peer-review process, but apparently, the editorial board of Nature Medicine failed to enforce this. This was also something that independent statisticians were most concerned about:

Of all the points raised in the Doss comment, the lack of interaction between groups was the one that Kevin McConway, emeritus professor of applied statistics at The Open University, and Nicole Lazar, a professor of statistics at Penn State—two outside statisticians I consulted with—said were of the most concern.

4. Supplementary materials still inaccessible

In looking over all the information again for this blog post, I also noticed that the authors write in their paper that codes are available in their GitHub repository. Especially for a contested paper like this, being able to verify the code is crucial for reproducibility purposes.

Unfortunately, the repository remains inaccessible 2 years later, and the authors were notified about this within a few days of publication.

5. Conclusion

Dear Nature Medicine editorial board: please consider publishing an expression of concern or some other way of putting into context the serious issues this paper has. As I document in my recent blog post on problems in academia and antidotes to cynicism creep, organized skepticism is one of the core values in science:

Practice and celebrate, for your own work and that of others, what Merton referred to as a crucial norm in science: organized scepticism, i.e., that “scientific claims should be exposed to critical scrutiny”.

For this reason, I absolutely love journals like BBS or Psychological Inquiry where target articles come together with 5-15 published commentaries, which together provide an in-depth perspective on a certain topic.

I thought the rejected letter to the editor provides a conclusion that hits the nail on the head, so I’ll just quote from that:

A paper in Nature Medicine is an important milestone in the development of a field. We believe that with carefully designed trials, adequate sample sizes, and appropriate statistical inferences, that milestone will be reached for psychedelic therapies. We do not think that it has been yet. The hype and misinterpretation that is occasioned by the premature publication of this type of work – including claims that psilocybin rewires the brain and opens the mind of people with depression – offer false hope to those suffering from depression. If such over-reach, beyond the modest data, continues, there will be a cooling effect that will delay the progress of these compounds to the clinic, which does us all a huge disservice – scientists, clinicians, patients, and the broader public alike.