For our new WARN-D research project on building a personalized early warning system for depression, I recently looked into openly available, transdiagnostic, self-report mental health screeners. The most common recommendation was to use the DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure (Adult). This measure was released by the American Psychiatric Association (APA) — the organization behind the authoratitive psychiatric nosology Diagnostic and Statistical Manual of Mental Disorders (DSM) — and can be found on APA’s website where they recommend a broad set of recommended self-report screeners.
Given that I haven’t found any critical literature on this screener, I wanted to point out a few issues in case you are considering to use the scale yourself. If you want to turn this blog into a measurement schmeasurement kind of game, feel free to take a few minutes to look at the scale that you can find here, and note down the issues that you see. Here is a screenshot of the most relevant part:
Done? Let’s go.
1. Impairment, severity, frequency
As you saw above, the core question is:
During the past TWO (2) WEEKS, how much (or how often) have you been bothered by the following problems?
The questionnaire does not measure impairment (bothered by), the severity of symptoms (how much), or their frequency (how often). It measures a mix of all of these constructs, and different people may report these differently (some focus on frequency, others on impairment). Naturally, these 3 aspects are positively correlated, but they are not the same kind of thing.
Now, for a short screener, perhaps it can be a perk to just cover this diffuse set of issues? I don’t think so, because for certain items, certain questions are inappropriate. For example, “being bothered by something” is the wrong question for e.g. item 4, which taps into mania or biploar disorder:
During the past TWO (2) WEEKS, how much (or how often) have you been bothered by … sleeping less than usual, but still have a lot of energy
The issue with mania is that folks are often not bothered by having lots of energy, lots of sex, starting new projects, and having great ideas. Of course these sort of problems are a general challenge for self-report questionnaires, but you’d be more likely to pick up on these issues if you asked about the frequency of this kind of behavior, rather than whether it bothered people.
2. Answer rubrics
Next up, here are the answer rubrics for each question.
First, it’s unclear to me how the bold and normal parts of the answers (e.g. None; Not at all) relate to the questions. Am I to infer that the question about severity asked in the core sentence relates to the bold part, and the question about frequency asked in brackets refers to the normal font? So, how much was I bothered by sadness — none, slight, mild, etc — vs how often was I bothered by sadness — not at all, rare, several days, etc? If so, this is anything but obvious, and now people need to answer two separate questions in one item. What if answers conflict, and I was bothered a lot by it, but rarely? Item 9, unexplained aches and pains, is an example that may only happen once or twice a week, but then be very severe.
Second, the frequency answer categories overlap: “less than a day” is (at least close to equivalent) to “not at all.” Overlapping answer categories is the sort of mistake that you see in buzzfeed quizzes about the 13 tips to figure out if your elephant is cheating on you, but should not have a place in an important questionnaire such as this one.
3. Personality
I’m no expert on personality disorders. But the two questions in this screener trying to tap into this facet appear, at least at face value, odd. They are:
19. Not knowing who you really are or what you want out of life?
20. Not feeling close to other people or enjoying your relationships with them?
We wanted to use the screener in young adults, who are (as adults) within scope of this screener. When asking some students, many of them said that not knowing what you want out of life, some days of the week, is a very normal challenge students face regularly. I have not been able to find how these items are distributed in the general population or student samples, and would be curious about that.
4. The rest
There are some minor issues that likely don’t have an impact on measurement, and yet I find myself wondering how such an important scale has such issues that can be fixed rather easily.
For instance, each item follows the core question. The very first item, in context with the core question, reads:
During the past TWO (2) WEEKS, how much (or how often) have you been bothered by the following problems? — Little interest or pleasure in doing things?
Why the questionmark at the end? It’s like asking “Which color do you like most?”, which should be followed by “black, red, green” and so on rather than “black?, red?, green?”. Yes, minor, but also yes, very easy to avoid.
In the end, for our project, we decided to use this screener, but only very few questions.
Something like the K10 perhaps. It attempts to assess frequency of symptoms (in this case dep/anx) as well as impairment. Brief I know but I have found at least clinically it distinguishes those from reporting symptoms versus those who have symptoms and are also distressed or impaired by them.
The level 1 cross-cutting measure would not be reliable on its own as a screener without clinician interview or probing.
There are so many questionnaires with similar problems out there. I think this is why it’s necessary to have a group of experts look into items before developing a new questionnaire, and pre-testing items in a small sample of the target population too. When I did this for my questionnaire, I received so many useful critiques and suggestions (sometimes conflicting tho), and it definitely made me think more in dept about the format of questions, response scale, timeframe, etc.
Item 8 is also puzzling to me – being bothered by avoiding situations that make you anxious? Avoiding such situations should be a relief if a person is anxious, but situations can bother you..
Agreed, avoidance is done to relieve anxiety. I could see that it bothers some upon reflection (I cannot take the metro), but in the moment, avoidance will not (but that is not necessarily what is being asked).
I feel you! I was once creating a survey and I wanted to avoid such mistakes. YET, the survey became too long and the participants complained, why are we asking the same things with different questions…
Yes, I don’t think I’d suggest 3 sets of questions with impairment, severity, and frequency. But it’s possible to e.g. ask specifically about frequency, and directly behind that impairment if you plan out the questionnaire well. So items are in rows, and then you have 2 columns for answers, one being e.g. frequency 1-5, and behind that “did this problem significantly bother you in your daily life” yes/no or so.