A new paper was published yesterday on a blood test for schizophrenia, by the same research team that in 2021 published a paper on a blood test for depression. The papers and accompanying press release contain problematic language, and the general idea of a blood test for mental illness makes very little sense to me given evidence we have about how mental health problems work.
Therefore, in this blog, I’ll briefly summarize three points of critique. Two showcase that the general idea of a blood test for a psychiatric disorder is implausible; the third is about transparency, openness, and conflicts of interest.
1. Measurement
First, mental disorders cannot be measured accurately or reliably in the way we can accurately and reliably measure some medical conditions, for which we can determine, beyond reasonable doubt, that a conditions exists: fever, lung cancer, broken leg, liver cirrhosis, and so on. That doesn’t mean doctors always agree on a diagnosis, but there is a true state of the world (Bob has liver cirrhosis, Susan has a broken leg) we can in principle determine. This is not the case for mental health problems: there is no way to determine if someone truly has depression. To determine a diagnosis, a mental health expert will talk to a person, and if they meet enough criteria, such as sad mood, appetite loss, insomnia, and concentration problems, they can be diagnosed. But most of these problems – appetite loss and insomnia and concentration problems — can be due to many other reasons than depression, such as stressful life events, thyroid dysfunction, side-effects of chemotherapy, other mental health problems like anxiety, and so on. This is why inter-rater reliability is absolutely absysmal for a diagnosis of depression. Quote from a recent paper we wrote on the largest study conducted on inter-rater reliability, with thousands of patients:
“Interviewers had a minimum of 2 years of psychiatric postgraduate training, and for each participant, independent psychiatric assessments were conducted by two interviewers within 4–48h of each other [..] Strikingly, despite using criteria designed explicitly to promote reliability, inter-rater reliability for a diagnosis of MDD was just 0.28, placing it among the least reliable diagnoses in the DSM.”
Now, if we cannot reliably or accurately measure depression, it is not possible to claim you made a depression blood test — a test with high diagnostic accuracy. This simply isn’t possible. You can only make a blood test if you have some objective, independently verifiable criterion (broken bone, liver cirrhosis) that you can use to establish a blood test for. And schizophrenia isn’t a completely different story than depression here either: the DSM-5 field trials showed an inter-rater reliability of 0.5 for schizophrenia — higher than depression, but still there is considerable disagreement (and certainly more than for e.g. bipolar disorder with 0.7 or neurocognitive disorder with 0.8).
Equipped with this knowledge, you can now also see why this statement by the professor responsible for the research is at best circular:
Press release: “Niculescu said in general, the best biomarkers were more predictive than the standard scales used to evaluate someone with hallucinations or delusions, which means the use of this biomarker test can help reduce subjectivity and uncertainty from psychiatric assessments.”
A test (blood test) that predicts a certain feature (schizophrenia), when the best guess about whether the feature exists is a diagnostic interview, cannot improve upon diagnostic interviews. That’s like building a blood test that outperforms thermometers (interviews) to measure temperature (schizophrenia) in a world in which the only way to find out someone’s temperature is through a thermometer.
2. A blood test for what
The idea that there is one common biological pathway underlying all depression, and that this pathway has nothing to do with anxiety disorders or other forms of psychopathology — which would be required for a blood test to work — is extremely implausible. Let me unpack this.
Scientists have become increasingly unhappy about psychiatric nosology, i.e. the way we classify mental health problems. Criticism of the DSM-5 is mainstream today. This is because disease categories — Bob has depression and Susan is healthy, Mark has PTSD and Claudia does not — no longer fit with the data we have gathered in the last century1. If anything, it is obvious today that patients with the same diagnosis can differ from each other in fundamental ways. We have done some calculations and shown that there are 10377 symptom profiles that all meet the criteria for major depressive disorder, for example. Depression symptoms are also extremely common in other mental health problems. And about 50% of people with depression have at least one other mental health problem.
Given all this evidence, I cannot come up with a plausible scenario why people diagnosed with depression should have a specific blood marker, but healtyh people and people diagnosed with PTSD or OCD or other conditions should not have this marker. It just doesn’t make any sense given data we have collected in the field in the last few decades.
Concrete example in which patient 1 and 2 can be diagnosed with major depressive disorder according to the DSM-5, and patient 3 with generalized anxiety disorder.
Patient 1: sad mood, insomnia, fatigue, concentration problems, appetite loss
Patient 2: anhedonia, suicidal thoughts, weight gain, guilt, psychomotor agitation
These are all depression symptoms, and patients have no single symptom in common, but both meet criteria for depression.
Patient 3: excessive worry, difficulty controlling worry, insomnia, fatigue, concentration problems
As you can see, patients 1 and 3 — despite having different diagnoses — overlap in many more symptoms than patients 1 and 2 who both have depression. It is extremely plausible to believe that somehow, the blood of patients 1 and 2 would have a very distinct marker that would reliably detect them as depressed, but reliably detect patient 3 as non-depressed, given how messy diagnoses are and how much overlap there is between them.
3. Conflicts of interest and transparency
The third point concerns transparency and financial interests. For yesterday’s schizophrenia blood test paper, the university press release, after a single paper on the topic, already started advertising a product, which raises red flags.
Press release: “The test is anticipated to be available later this year from the IU spin-out company [company]. For more information about precision psychiatry and blood testing, visit the [company] website.”
The press release reads more like an advertisement for a product than careful reporting of scientific findings. That doesn’t automatically mean the study is incorrect or problematic, but given that authors usually approve university press releases, it means at least that they aren’t very careful with communicating findings. Or look at this here, also from the university statement:
Press release: “The new test identifies biomarkers in a person’s blood that can objectively measure their current severity and future risk for schizophrenia”
We have no objective markers for mental health problems, after a century of intense, thorough work by many smart people. Far from it. Not only do we not have any, blood tests seem extremely implausible given the arguments presented above. Therefore, the claim of an objective test for future risk for schizophrenia is really quite silly, and the language is more consistent with marketing language than science communication. But perhaps that is the goal here.
A test helps to determine whether you have a feature or not. Let’s take COVID tests as an example, aimed to determine if you have COVID. A COVID test is precise if it accurately classifies people with COVID as COVID positive, and people without COVID as COVID negative; scientists call this sensitivity and specificity, and a test needs both. Accurate tests have high predictive accuracy, which is a numerical value. Whether you are coughing is not a good test for COVID, because not everybody with COVID coughs, and many people who cough do not have COVID — the test is not accurate, nor is it sensitive nor specific.
Neither the depression nor the schizophrenia blood test papers have any statistical or numerical information in the abstract — no information on sample size, sensitivity, specificity, false positives or negatives, predictive accuracy, or anything else. Showing that a test has high predictive accuracy is the first thing scientists tend to report in abstracts for these tests, and given the authors claim of an objective test, I am surprised this information is not presented clearly and transparently in the summary of the research.
To be clear, the data are extremely impressive, with tons of repeated measurements over long periods of time. But the sample is likely not large or representative enough to guarantee that results replicate in other samples, i.e., they are not large or diverse enough to guarantee generalizability — drawing conclusions from this group of people to all people the study has implications for. But this is of course what you do when you use this research to build a blood test. You wouldn’t put a lot of trust in a poll of 300 participants about who will be the next president, especially if you learned that all participants were recruited in specific places (e.g. rural areas vs cities, Trump vs Biden campaign events, universities vs military bases, etc), i.e., were not drawn randomly and in a representative fashion. The same goes for any scientific study where we study a specific sample to learn about a larger population of interest.
These issues go hand in hand with a general lack of transparency. The paper is not accompanied by a preregistration, code, or open data. I have also been unable to find any external validation of the findings from this research group, for instance, regarding their depression test. Given that so many findings in this literature do not replicate, the least I would do before talking about breakthroughs and objective tests is wait until this has been independently verified by experts who do not stand to make money from this. As I have said in other blogs and papers, there is no problem with financial interests in general — we have many medications that work well today because of investments from the pharma industry. But is it good to keep in mind that there is overwhelming evidence that studies with financial conflicts of interests are much more likely to find outcomes that are in favor of the tested drug or test than studies carried out by independent investigators.
Given that this is the 10th or so blood test this lab has developed and tries to market, I would have liked to see a preregistration — the authors have tremendous expertise with the pipeline, after all. This would alleviate concerns that authors may not have reported all their data or results (which, again, is not a crazy thing to consider when taking into account that questionable research practices are very common in research, unfortunately). Sharing data and code in addition to preregistering studies also helps to find out if people made honest mistakes, which do happen.
“Eiko, I think you’re pretty harsh, asking for this level of transparency for a paper published in psychiatry, where it is quite common not to share data and code!”
Perhaps, but please bear in mind that the authors claim to have developed a “breakthrough new blood test for schizophrenia”, an “objective test” for future schizophrenia that they plan to market later this year already, after a single scientific study. This is absolutely worth a Nobel prize if it works the way the authors describe, and would revolutionize the way we do psychiatry completely. The implications are massive and would change the mental health landscape as we know it. Clearly, studies that make such extreme claims should come with extreme levels of evidence. Preregistration, open data and code help with this, and are absent here.
4. Conclusion
The website of the company that aims to market these blood tests has testimonies. One of them reads: “The results of the [company] reports have been uncannily accurate in finding the most effective treatments”.
I couldn’t agree more. Alas, in science, we don’t want uncanny accuracy.
- That doesn’t mean we shouldn’t take mental health problems seriously, or that these categories cannot be useful – see my detailed views on this here.