APS 2018 — the conference of the Association for Psychological Science — ended a few days ago. Although this was my fifth APS in a row, it was a very different experience than in the last years. So I wanted to write down a few thoughts, with a focus on how to make sure to keep improving scientific quality, transparency, open science practices, and inclusiveness in the next years to come.
In the last years, APS always felt like flying out to a conference where I neither knew the organization, nor most of the researchers. I payed the membership fee for purely economic reasons — the reduction in conference fee was larger than the membership fee itself. I never quite understood how talks or posters were selected, didn’t know the “who is who” of APS, and was unaware of most science-political topics that were being discussed1. This is partly because I wasn’t that active on social media, partly because I was at a different career stage, but also — let’s face it — because the A in APS kind of stands for American, and folks from outside the US can have a harder time grasping the internal workings of APS.
This time, things felt different, because I had a much closer connection to both the organization and the researchers at APS, for different reasons: I had the honor to be awarded the APS Rising Star award a few months back; I have published in several APS journals recently, such as Perspectives and Clinical Psychological Science; Jessica Flake and I published our recent piece “Measurement Matters” in the APS Observer. In addition, I was much more up to date regarding science-political developments — largely due to being more active on social media than I used to, and due to a recent blog post I wrote on self-citation practices of an APS editor. I also met numerous APS editors and reviewers with whom I was in contact in the last year, and talked to folks working for the APS observer.
All of a sudden, and contrasting prior years, APS has turned into a real organization with familiar faces. People I know and value contribute to APS, and strive to make it better every year.
Another difference this year was that I wasn’t there primarily to talk about my own work, which is what I did in the last years. Instead, I organized a symposium together with Jessica Flake in which we highlighted general and specific measurement problems in psychology (slides); put together and chaired a symposium for four early career researchers who presented their work on using ecological momentary assessment data to help improve the life of patients with mental health problems (slides); and was part of an invited panel organized by Jennifer Tackett that discussed open science issues in Clinical Psychology, together with the other panelists Bethany Teachman, Ann Kring, Aidan Wright, and the discussant Richard Lucas.
It was awesome, but it also made me think about APS as an actual organization, in which people I know play an important role. Which means it is time to share some critical thoughts with the goal to make APS better next year.
Reward structure and open science
The reward structure in science is broken2, which is responsible for many of the problems we are facing in psychology. We are focusing on sexy over rigorous science, on quantity over quality of publications, on awarding honors to specific individuals when research is always a collaborative effort. Such a system incentivizes questionable research practices such as p-hacking, it rewards cheating, and fosters a competitive instead of a collaborative environment.
Regarding the award structure, I was incredibly humbled to have received a recent award at APS, but it felt odd to receive an individual achievement award for work I would have never been able to pull off alone. Tim van der Zee summarized related issues last week:
First time that I attended a big award ceremony (just now at #APS18SF) for scientists. Mixed feelings. On the one hand it's good to reward good science. On the other hand it seems to feed right into the problematic culture of personal prestige (1/11).
— Tim van der Zee (@Research_Tim) May 25, 2018
Let’s step back for a second and look at the larger picture: Psychology is moving forward, and much more has happened in the last 5 years than I had ever anticipated. We have registered reports, the Psychological Science Accelerator, new journals such as Collabra Psychology focusing in rigorous and open science over sexy findings3, the Loss of Confidence project4, and many other recent open science developments. If we want to make further progress, we need to reward open science practices and initiatives, and change the incentive system.
APS also featured important open science topics much more prominently than in previous years. For instance, Katie Corker gave a workshop on reproducibility:
— Katie Corker (@katiecorker) May 23, 2018
And Simine Vazire, Penny Pexman, Stephen Lindsay, and Leif Nelson talked about publishing trends, journal policies, and other efforts to promote open science.
— simine vazire (@siminevazire) May 27, 2018
But there is much more to be done, and we should do much more. So, dear APS: Can we modify the reward system more than we have so far? With poster awards next year that reward open science practices, with Rising Stars of open science? Researchers who conduct more rigorous work have disadvantages in a climate where only the number of papers and sexy results count: Symposia on trustworthy null-findings, and talks of preregistered studies and large-scale collaborations can offer alternative ways forwards. In addition, we could implement visual cues in the conference program for certain open science practices such as preregistration. It would help attendees navigate the program, and select talks we want to see.
APS is a large organization, and I have been paying membership fees for several years now. In return, I would like to learn more about the organization and its processes. Specifically related to the conference, I would like to understand how posters and talks and researchers receiving awards are selected; what percentage of researchers is successful in each of these categories; and what the demographics of these successful researchers are.
To put it differently, what are the checks and balances in place to ensure that women, minorities, and other folks who had it much harder than me — and face(d) many more obstacles than I did — have a fair chance for having their symposia accepted, and for receiving awards? How can we improve these checks and balances to be more inclusive? What is being done currently?
Again, I don’t meant to say nothing is happening: I know folks in important APS committees who care about inclusion and transparency, and am sure this is being talked about and worked on. Thanks for putting in the hard work! But it would be great if we could learn more about what is happening, what has improved and what remains to be improved, and how APS members can help with this transition.
The US are a crucial country for advancing psychological science, and APS arose in the US, so it’s not surprising that there is a US-focus at APS. But this focus was very visible this time around — most likely because I started paying more attention to it — and I wonder what we can do to be more inclusive.
Take, for example, this year’s class of rising star awardees. As Hans IJzerman pointed out correctly:
The vast majority of awards at APS are given to North American researchers. Are North American psychology researchers really that superior in their research?
— Hans IJzerman (@hansijzerman) May 25, 2018
Not all awardees are North American, but most work at US universities. About 80% were awarded to researchers in the US and Canada. Other awards went to the UK (6), the Netherlands (5), Australia (3), Belgium (2), Germany (2), Hong Kong (2), Korea (1), Israel (1), and Brazil (1)5. This means about 90% of the awards went to countries in which English is the native language.
This has structural reasons: One needs to be a fellow to be nominated, know how the nomination process works, and needs a nomination through a longterm APS member. But these structural reasons are good news in a sense, because we can change the structure to make the system more inclusive — i.e. we have levers to change this and to move forward. And note that this is not just about inclusiveness: APS, for selfish reasons, should ensure to feature the best research and researchers to help improving psychological science. Many of these are North American, of course; but there are also many that aren’t.
The US-focus is also obvious in many other areas, for instance when it comes to APS paper or poster abstracts. Samples are usually described as “n=100 participants”, unless they are non-US samples, in which case a country is mentioned (“n=100 French participants”). And discussions about grants usually focus on US grants — which makes sense if we only talk about grants of one specific country; there would be much less benefit discussing solely Austrian grants at APS. But maybe we can broaden the debate across all areas.
If you add these points together, you see the pattern, and I wanted to use this blog post to raise a bit more awareness for inclusion. Researchers from outside of the US can feel quite lost at APS, and if privileged white male Eiko — who lived in the US for 2 years during grad school — feels lost, I’m sure there are others who feel a bit lost, too.
I’m sure there are some folks at APS working hard on inclusion, but it seems fair to call this out and ask for a bit more transparency about the process. Please let us know how to support you, and let us know what we can do to foster inclusion.
- i.e. science drama
- Along with scientific publishing
- COI: I am on the editorial board
- Shoutout to Tal Yarkoni, Christopher Chabris, and Julia Rohrer
- I counted once, after 10 hours on a plane, so if someone wants to double check / cross-validate, I’d be happy for a recount