First preregistered network replication study with power analysis & open data

I haven’t written blog posts about individual new papers in over a year, but this one is a milestone I’m really proud of: Our new network paper on social media and well-being was published today. Specifically, we looked at the relations between Facebook use, rumination, depressive, anxiety-, and stress-related symptoms, social comparison, contingent self-esteem, and global self-esteem. Overall, we find that network structures are very similar across both datasets.

The paper breaks new ground in that it is the first published preregistered network replication using Markov Random Fields; contains open data; and is also the first published network study with using this statistical model that used a power analysis to inform data collection. I hope this will move the field forward towards adopting open science practices more widely. Thanks to Lien Faelens, Kristof Hoorelbeke, Rudi De Raedt, and Ernst Koster for involving me in the study.

Here is the abstract:

Various recent studies suggest a negative association between Facebook use and mental health. Yet, empirical evidence for this association is mixed, raising the question under which conditions Facebook use is related to negative outcomes, such as decreased well-being. Our study addresses this question by investigating the relationship between Facebook use, rumination, depressive, anxiety-, and stress-related symptoms, taking into account potential key variables such as social comparison, contingent self-esteem, and global self-esteem. In a first study, we explored the unique relations between these constructs using state-of-the-art network analysis. Subsequently, we conducted a preregistered replication study. In both studies, social comparison and self-esteem held a central position in the network, connecting social media use with indicators of psychopathology. These findings highlight the prominent role of social comparison and self-esteem in the context of social media use and well-being. Longitudinal and experimental studies will be required to further investigate these relationships.

The study comprised of the following steps:

1. We estimated network models in 207 participants. This part of the study was not preregistered, and we can improve on that the next time around.
2. We used the function netSimulator in the R-package bootnet to perform a power analysis, with the question: IF the network model were the same in another data set, how much N would we need to detect it?
3. We preregistered the result, and aimed to collect data in at least 450 participants
4. We collect the second dataset (N=468)
5. We estimated the network model again, and investigated whether it replicated

Here are the two networks in both datasets. Abbreviations: rumination (RRS), Facebook use private (e.g. messenger; MSFU_Private), Facebook use public (e.g. status updates; MSFU_Public), Facebook use passive (e.g. scrolling; MSFU_- Passive), intensity of Facebook use (FBI), social comparison (COMF), contingent self-esteem (CSS), and global self-esteem (RSES).

How similar were the networks?

[…]both models show strong overlap as seen by: (a) comparison of the adjacency matrices of Study 1 and 2 (r = 0.95), (b) comparison of the centrality indices (Strength: r = 0.80; Closeness: r = 0.91; Betweenness: r = 0.81; Fig. 4), and (c) comparison of predictability of nodes (Study 1: mean R2 = 54%, Study 2: mean R2 = 50%; r = 0.98; see supplemental material). In addition, the Network Comparison Test yielded no significant differences in terms of network structure (test of invariant network structure; M = 0.12, p = .80) or overall strength of connectivity (test of invariant global strength; Study 1 = 4.94, Study 2 = 4.74, S = 0.20, p = .67).

We finished the first version of this paper over a year ago, and given the very fast developments in the network psychometrics field, I’d do a few things differently today, such as using confirmatory network modeling, and network equivalence testing (over difference testing). I hope to write up a brief tutorial on the state-of-the-art of network preregistration and replication in the coming weeks.

You can find the full paper here, the preregistration here, and open data here.