Schmeasurement: DSM-5 symptom measure edition

      2 Comments on Schmeasurement: DSM-5 symptom measure edition

For our new WARN-D research project on building a personalized early warning system for depression, I recently looked into openly available, transdiagnostic, self-report mental health screeners. The most common recommendation was to use the DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure (Adult). This measure was released by the American Psychiatric Association (APA) — the organization behind the authoratitive psychiatric nosology Diagnostic and Statistical Manual of Mental Disorders (DSM) — and can be found on APA’s website where they recommend a broad set of recommended self-report screeners.

Given that I haven’t found any critical literature on this screener, I wanted to point out a few issues in case you are considering to use the scale yourself. If you want to turn this blog into a measurement schmeasurement kind of game, feel free to take a few minutes to look at the scale that you can find here, and note down the issues that you see. Here is a screenshot of the most relevant part:

Done? Let’s go.

1. Impairment, severity, frequency

As you saw above, the core question is:

During the past TWO (2) WEEKS, how much (or how often) have you been bothered by the following problems?

The questionnaire does not measure impairment (bothered by), the severity of symptoms (how much), or their frequency (how often). It measures a mix of all of these constructs, and different people may report these differently (some focus on frequency, others on impairment). Naturally, these 3 aspects are positively correlated, but they are not the same kind of thing.

Now, for a short screener, perhaps it can be a perk to just cover this diffuse set of issues? I don’t think so, because for certain items, certain questions are inappropriate. For example, “being bothered by something” is the wrong question for e.g. item 4, which taps into mania or biploar disorder:

During the past TWO (2) WEEKS, how much (or how often) have you been bothered by … sleeping less than usual, but still have a lot of energy

The issue with mania is that folks are often not bothered by having lots of energy, lots of sex, starting new projects, and having great ideas. Of course these sort of problems are a general challenge for self-report questionnaires, but you’d be more likely to pick up on these issues if you asked about the frequency of this kind of behavior, rather than whether it bothered people.

2. Answer rubrics

Next up, here are the answer rubrics for each question.

First, it’s unclear to me how the bold and normal parts of the answers (e.g. None; Not at all) relate to the questions. Am I to infer that the question about severity asked in the core sentence relates to the bold part, and the question about frequency asked in brackets refers to the normal font? So, how much was I bothered by sadness — none, slight, mild, etc — vs how often was I bothered by sadness — not at all, rare, several days, etc? If so, this is anything but obvious, and now people need to answer two separate questions in one item. What if answers conflict, and I was bothered a lot by it, but rarely? Item 9, unexplained aches and pains, is an example that may only happen once or twice a week, but then be very severe.

Second, the frequency answer categories overlap: “less than a day” is (at least close to equivalent) to “not at all.” Overlapping answer categories is the sort of mistake that you see in buzzfeed quizzes about the 13 tips to figure out if your elephant is cheating on you, but should not have a place in an important questionnaire such as this one.

3. Personality

I’m no expert on personality disorders. But the two questions in this screener trying to tap into this facet appear, at least at face value, odd. They are:

19. Not knowing who you really are or what you want out of life?
20. Not feeling close to other people or enjoying your relationships with them?

We wanted to use the screener in young adults, who are (as adults) within scope of this screener. When asking some students, many of them said that not knowing what you want out of life, some days of the week, is a very normal challenge students face regularly. I have not been able to find how these items are distributed in the general population or student samples, and would be curious about that.

4. The rest

There are some minor issues that likely don’t have an impact on measurement, and yet I find myself wondering how such an important scale has such issues that can be fixed rather easily.

For instance, each item follows the core question. The very first item, in context with the core question, reads:

During the past TWO (2) WEEKS, how much (or how often) have you been bothered by the following problems? — Little interest or pleasure in doing things?

Why the questionmark at the end? It’s like asking “Which color do you like most?”, which should be followed by “black, red, green” and so on rather than “black?, red?, green?”. Yes, minor, but also yes, very easy to avoid.

In the end, for our project, we decided to use this screener, but only very few questions.

2 thoughts on “Schmeasurement: DSM-5 symptom measure edition

  1. Ekaterina P.

    I feel you! I was once creating a survey and I wanted to avoid such mistakes. YET, the survey became too long and the participants complained, why are we asking the same things with different questions…

    1. Eiko Post author

      Yes, I don’t think I’d suggest 3 sets of questions with impairment, severity, and frequency. But it’s possible to e.g. ask specifically about frequency, and directly behind that impairment if you plan out the questionnaire well. So items are in rows, and then you have 2 columns for answers, one being e.g. frequency 1-5, and behind that “did this problem significantly bother you in your daily life” yes/no or so.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.