A new paper published today in Lancet Psychiatry, led by Christopher Veal, reports findings from a systemic review of 450 clinical trials for unipolar and bipolar depression. Our results can be seen as trying to answer one of the oldest questions in the field of depression measurement: which of the over 200 measures is the best?

I want to thank the principal investigator Dr. Astrid Chevance for the fantastic work, and all the researchers and folks with lived experiences who helped with the study. Below, I briefly provide the rationale for the study, and then summarize the results.

1. Background

Depression affects millions globally and is one of the leading causes of disability. The efficacy of interventions like psycho- and pharmacotherapies is typically assessed through randomized controlled trials (RCTs) via so-called outcome measures: information we want to know to determine if patients improved. As we have highlighted in a recent review paper on theoretical and statistical challenges in depression measurement published in Nature Reviews Psychology (PDF), depression is measured in all sorts of different ways, which raises all sorts of questions. For example, do all these different measures for depression (at least 280!) actually measure the same construct? And do the differences between depression measures stand in the way of cumulative science? After all, it’s difficult to compare the results of studies if they used different measurements.

In today’s paper, Chris tackles a different question related to research waste, specifically: Do we measure the things we are actually interested in? For example, the famous MADRS depression scale was developed in the 1970s by following a very small group of 64 patients during a single clinical trial in which patients received an antidepressant. They gave patients a longer list of items they thought relevant for depression, but patients only improved on 10 of those, so the scale was reduced to 10 items. This leaves a lot of questions open — perhaps some of the dropped items are highly relevant to depression but the treatment was just bad? Or perhaps some of the patients did not improve on the ‘insomnia’ item because there was construction going on in their neighborhood?

2. The PROCEED study

Luckily, Astrid and the gang have previously published the PROCEED study which helps us in determining which outcomes are relevant to stakeholders; I have summarized PROCEED in a previous blog post.

In PROCEED, we asked what relevant stakeholders think we should measure in clinical trials for depression, and received over 8183 open-ended answers from 1912 patients, 464 informal caregivers, and 627 health-care professionals from 52 countries. This is different from the typical approach — existing scales like HRSD, BDI, CES-D etc were put together by (typically old, white, male) researchers or clinicians without involvement of stakeholders. May they have missed important aspects?

Here a summary of the results:

The study identified 80 relevant domains, including clinical symptoms and functioning dimensions. Going back to the numerous depression measures used in the literature, the PROCEED outcomes raise the question which of those capture these domains comprehensively. To address this gap, our new paper aims to identify all outcome measures used in recent RCTs for depression, and to evaluate the extent to which they cover the  80 PROCEED domains.

3. Study setup

The research was conducted in 4 steps.

First, we identified 450 RCTs for adult unipolar and bipolar depression conducted between 2018 and 2022.

Second, we counted a total of 388 outcome measures across these trials. For depression specifically, we identified 296 different measures. Holy cow. We focused on efficacy-related, clinical, and functional outcomes. We did not consider biological outcomes (e.g., blood samples, because they are surrogates) or harm-related outcomes (e.g., adverse events, because they are already systematically collected).

Third, the outcome measures were categorized into two parts: objective measures (e.g., duration of hospitalization or death) on the one hand, and subjective measures (e.g., clinical symptoms or satisfaction) on the other. Subjective outcomes were further divided into

• patient-reported outcome measures (PROMs)
• clinician-reported outcome measures (ClinROMs)
• observer-reported outcome measures (ObsROMs)
• and performance outcome measures (PerfOMs)

Fourth, depression measures are often scales. For example, the MADRS I described above has 10 items, and they can differ a lot from each other. So to check whether all the identified measures overlap with the 80 outcome domains identified by the PROCEED study that matter most to patients, we investigated measures at the item level. Numerous raters (absolute heroes!) compared every single item of each scale with every outcome domain. For example, a neuropsychologist compared the PerfOMs, folks with lived experiences matched the PROMs, clinicians matched the ClinROMs, etc.

4. Study results

4.1 Descriptives

A detailed description of the type of RCTs identified can be found in the paper. Of the 388 identified outcomes, there were

• 259 (67%) PROMs
• 63 (16%) PerfOMs
• 45 (12%) ClinROMs
• 1 (<1%) ObsROM
• and 8 (2%) objective outcomes.

RCTs require one primary outcome, which is the one that is usually reported in the abstract and relied upon for interpretation. In the RCTs we analyzed, 176 (39%) of the primary outcomes were PROMs, and 276 (61%) ClinROMs. The most commonly used scales were as follows:

We also investigated results in only depression vs only bipolar trials, as well as drug vs psychotherapy trials, reported in more detail in the paper.

4.2 Outcome matching

We now compared the most commonly used outcome measured identified and mapped them on the 80 PROCEED domains; I’ll show the results separately for PROMS and ClinROMs.

For patient-rated measures, the PROMs, results are as follows. Each circle (different colors) represents one specific outcome measure, and each line represents one of the 80 domains identified in PROCEED. That means that the colored light blue circle, BDI-II, covers ummmm-let-me-count I’m going to say a bit less than half of the 80 domains. The GAD-7 (red circle), on the other hand, only taps into 5 PROCEED domains.

In summary, the 10 most commonly used PROMs focused on depression, sleep quality, quality of life, and disability or impairment. Each of these PROMS overlapped with between five (6%) and 31 (39%) of the 80 domains. If you were to give a patient all of these scales at the same time, they would still fail to cover 24 (30%) of the 80 domains identified in PROCEED—one in three! Some of these relate to symptoms (e.g., helplessness, memory loss, or dissociation), others to functioning (e.g., capacity to get out of bed or communicating feelings).

For clinician-rated outcomes, the ClinROMs, six were multi-item measures assessing depression, suicidality, anxiety, and functioning, and three were Clinical Global Impression scales, assessing  severity of depression, patient improvement, and bipolar severity. In total, 18 (23%) of PROCEED domains were not covered by these nine ClinROMs; as above, some were symptoms (e.g., emotional blunting, incurability, or feeling alone), others related to functioning (e.g., social isolation or ability to cope with a life event).

If we now rank all outcome measures, the list is as follows: