Update, 02/2024: We have now published the paper “History repeating: guidelines to address common problems in psychedelic science” on this topic, in part based on this blog post. We introduces 10 pressing challenges that limit conclusions regarding safety and efficacy. We share a checklist that researchers, journalists, funders, policymakers, and other stakeholders can use to assess the quality of psychedelic science.

There was a lot of hype in the last days around a new study on treating major depressive disorder via psilocybin, the active ingredient of magic mushrooms. In the paper and corresponding social media posts, the researchers make odd claims, e.g. that psychedelics liberate the entrenched depressed brain—claims one would expect from a youtube ad on homeopathy, but not from prominent researchers working on serious academic topics.

This was followed by plenty of discussion on social media among researchers and folks interested in depression and such treatments. In response to my tweets, I saw many insightful comments, also by folks with lived experiences. But there were also some baffling statements by psychiatrists who really should know better (such as that there was this one study published in a good journal so the case really is closed).

I therefore created a list of potential challenges you should be aware of when navigating this topic. I hope this list of issues is useful for journalists, folks with lived experiences, and fellow academics and clinicians. For what it’s worth, I believe I am qualified to provide my views on this: I wrote my PhD on depression and have been working on depression since 2010, with around 70 published peer-reviewed papers on the topic. I also teach courses at Leiden University on clinical trials.

So lets get on with it.

Are psychedelics safe and effective treatments for major depressive disorder?

The short answer is that we really don’t know, because there is not enough research. And the research that exists has a number of big issues. I list some of the most important ones below.

Lack of control groups

Scientific studies often are “open label” studies that lack a control group, which is an issue because it leaves open the question whether psychedelics work better than placebo. “But Eiko, placebos are much better than nothing, so what’s the problem?” The problem is that we already have placebos. They are free and without side-effects. There is no need to expose people to psychedelic substances with potentially harmful effects for the placebo effect. We also already have treatments that work quite well. So to establish evidence that psychedelics are safe and effective antidepressants, we need to compare them to placebo groups and eventually also to groups who receive established treatments.

Lack of blinding

If studies on psychedelics as antidepressants have a control group, participants often know in which group they are in. There are studies using microdosing, but usually, given that the psychedelic experience is thought to be part of the active therapeutic ingredient, participants are not what we call “blind” to the condition they are in. This is a problem for two reasons.

First, because people who believe they are in the control group rate treatments as less effective. That is, the intervention group does not work better than the control group because it works in any way above placebo, but the control group is worse than the intervention group because people actively rate it worse (than if it were just placebo). People in control groups are also more likely to drop out if they know about this. Dropout rates are therefore a nice way to check whether people were blind to the group they were in. In the SMILES trial (dieting as a treatment for depression) that we re-analyzed in 2018, we found that the dropout rate in the control group was 4 times as high as in the dieting group, due to obvious issues with blinding.

Unfortunately, researchers sometimes do not ask for blinding, which is a big red flag for me. After all, considering all the efforts that go into clinical trials, it is very little work to ask participants in which group they thought they were, and can rule out a considerable threat to valid inferences. There is also evidence that if authors ask about group membership, the results are not always written up honestly or transparently. In a recent study on Ayahuasca as an antidepressant, the authors write in the paper “We believe blindness was adequately preserved in our study”. Looking at the supplementary materials, we find that 12 out of 12 (i.e. 100%) participants in Ayahuasca group thought they received Ayahuasca, but only 5 out of 14 (i.e. 42%) participants in placebo group thought they received Ayahuasca.

There is a second reason lack of blinding is a problem: because clinicians know what group participants are in. Keep in mind that the study outcomes are often based on the perspective of these clinicians, because they get to determine the treatment progress, based on their assessment of what patients tell them. When they are not blind to the group people are in, this may bias their ratings and intervention and control group are not properly comparable anymore.

Conflicts of interest

Studies on antidepressant drugs (including psychedelic substances) are often funded by the pharma industry. That is ok in principle, but you need to be aware that there is overwhelming evidence that industry-sponsored studies are much more likely than independent investigations to find outcomes that are in favor of the tested drug. In medicine, it is more common to have such conflicts of interest than in psychology, broadly speaking, and of course researchers can maintain highest ethical standards after having been payed e.g. for advising a pharmaceutical industry. But conflicts have become so common and pronounced that I sometimes wonder how trustworthy a paper is. Take, for instance, a recent “international expert opinion” article “synthesizing the evidence for ketamine and esketamine in treatment-resistant depression”, published in one of the highest ranking psychiatry journals. These are the declared conflicts of interest, which include holding patents to treat depression via Ketamine. If massive financial interests come into play, I am skeptical how balanced such an opinion piece can be.

You find such conflicts in many studies in the psychedelics literature. One prominent paper that comes to mind was published in 2016; I wrote about the conflicts of interests in this Ketamine study in another blop post.

External validity

Studies so far often have a very small number of participants. This is problematic due to something we call external validity, i.e. how well do results of the studied sample generalize to the population we want to study. The smaller the sample, the more difficult this is. I wrote a dedicated blog post on this topic with several examples from elections. You wouldn’t trust a poll of 30 people to be a reliable indicator for who wins the next presidency. Why would you then put any more trust in an antidepressant study with 30 participants? It’s the same thing.

A related problem is that people who participate in psychedelic trials for depression are often volunteers who already have experiences with psychedelics. But suggesting that psychedelics are safe and effective requires them to work for most people in most situations (external validity again), and there simply isn’t sufficient evidence for that. “But Eiko, prior studies claim that e.g. only half of the participants said they had used psychedelic drugs before.” Yes, but keep in mind that studies often take place in the US, where taking illicit hard drugs can lead to serious legal repercussions, and there is strong reason to believe that people will not always report prior drug use truthfully. It is also not uncommon to screen people out if they have prior drug experiences, which gives people another reason to lie because they would like to gain access to potentially life-saving treatment. So if people here aren’t fully honest, I fully understand: but we need to be aware of this when it comes to external validity.

Follow-up times

Studies claim that they are testing new antidepressant drugs in people who often have severe, treatment resistant depression—a chronic and recurrent illness. And then they follow them only for a few days or weeks. Even if people get a little better in this time, that does not mean that this works as a treatment, rather than a brief reduction of symptoms.

Remember the Ayahuasca trial from above, in which participants had on average suffered from depression for 11 years? The study lasted for 7 days and then stopped. And the authors claimed their study supports “the safety and therapeutic value of psychedelics”. If you had another severe, recurrent, chronic illness like cancer, would you be ok if treatment studies followed people for 7 days only?

“But Eiko, many other studies have very similar issues, including antidepressant treatments and psychotherapy”. Absolutely. But that does not make studies on psychedelics more credible, unfortunately—it just raises doubts about the particular studies that have such issues.

Other red flags

There are many other red flags. Studies often use multiple measures to see how much people improve, and then selectively report the measures for which treatment worked best. Or they have a primary outcome registered, in the study protocol, but switch to another that “works” better. For the recent psilocybin paper I mentioned at the beginning of the blog post—the one claiming that there is a liberating effect of psilocybin on cortical activity—the authors have actually admitted to switching outcomes now, away from the registered QIDS depression scale and to the BDI depression scale. The justification was to “select a scale that we felt would be most sensitive to a relationship” with the outcome of interest (i.e., to obtain a significant result). There is literature showing that studies with industry sponsorship are more likely to show such patterns. Or studies focus on the reduction of symptoms as a primary outcome, when it has been widely known for decades that even when symptoms go away, people still often face severe problems with quality of life, impairment of functioning, and so on. Measurement of recovery should focus on how people are doing in their lives, in addition to the particular symptoms they show.

It is time consuming and not very rewarding for scientists to look into problems like these. Authors are often sponsored by the industry and their independence is questionable. Journals quickly get defensive about major flaws in papers because it points to structural problems with conflicts of interests and peer review. And even if critical commentaries get published, they often don’t get the attention they deserve. It’s much sexier to write a newspaper article about revolutionary new psychedelic treatments that make the lame walk and the blind see, compared to writing a newspaper article about the fact that a study switched outcomes and had issues of multiple testing and external validity and blinding, leading to several validity threats and raising questions whether these drugs actually work.

Luckily, although this is challenging work, some folks like Dr Ioana Cristea do this work, and you can find examples of identified problems in the growing literature on using recreational drugs to treat depression here and here. We’ve also written about this on a number of occasions, with a focus on Ketamine, here and here.

Moving forward

Broadly speaking, people with mental health problems usually have reasons for these problems. For depression, folks are often stuck in life situations that are difficult to escape, such as being unemployed and having tried hard to find a job but not finding a job. There are many other pathways into depression of course, and many ways to be depressed. But it’s rarely disconnected from the particular lives people lead. If life situations are severe enough, many of us would become depressed, no matter how resilient we are. For anxiety disorders, being not well is often preceded by severe stressors of traumatic event—again, life situations.

The idea that a one-time dose (or even repeated doses) of any drug can solve such problems for good seems bizarre to me, because that is not how the world is structured. Drugs cannot resolve unemployment, disability, sexism, stigma, or an abusive partner. Proponents often say that drugs can help better relate to these experiences, and maybe that’s possible for some people and some problems, and maybe that’s also how some psychotherapies work. But there isn’t any strong scientific evidence that psychedelics do this, and therefore, we ought to remain skeptical.

To conclude, we absolutely owe it to folks with mental health problems to look into all sorts of treatment options, even if they “only” work for some. That’s fine: one-size-fits-all solutions in clinical psychology and psychiatry haven’t worked well. And we should listen to people with lived experiences if they tell us something is helpful (the way this has worked with e.g. medical marijuana that has provided relief for so many), and researchers should also absolutely listen to clinicians who report positive experiences with treatments.

But we also owe it to folks to be skeptical of hypes and fads. There are existing treatments that work. They don’t always work as well as we want them to work, but they work. And new miracle cure treatments are dangerous because they may be given to a person who then does not get an effective, potentially life-saving treatment. We’ve had hundreds of promised miracle cures in the last century. A very small number of them worked reasonably well for some people. None worked amazingly well for everyone. Psychedelics will not be a miracle cure either. And going back to that new Nature Medicine paper: no, they will not “liberate your entrenched depressed brain“¹. That’s not how depression works. That’s not how drugs work. And that’s not how brains work.

EDIT May 16th 2022, 10pm: Shayla Love over at Vice Motherboard wrote a piece on the psilocybin study that led me to write this blog post in the first place. She talked to a lot of researchers, who identified major issues in the study.

1. Note that this is an actual tweet of the actual first author of the paper—holy hype, Batman. Similar phrases appear in Nature Medicine paper itself, twice.