The Society for the Improvement of Psychological Science (SIPS) hosted their third annual conference in Grand Rapids MI a few days ago. This blog provides a summary and some collected resources for those who couldn’t join, and a few reflections (praise & challenges) of a SIPS virgin.
SIPS: an introduction
I will not repeat all the hype and thank yous and tell you how amazing things were — see #SIPS2018 on Twitter for that. But for those who couldn’t join us: SIPS featured some invited workshops on topics integral to open science such as replications, multi-lab collaborations, sample size determination, meta-analysis, measurement, a crash course in R, and others. You can find the program here, and all workshops and other materials on the SIPS2018 OSF page. I believe some videos of the workshops will go online in a few days.
In addition to the workshops, there were many empty time slots available which filled up with sessions that people wanted to work on. Sessions had different formats (e.g. unconference, hackathon), but the main point is that SIPS created space for people to get together and work on important topics – and it totally worked.
Getting shit done
Although many of these meetings only lasted a few hours, and although most hadn’t been planned before the conference, a lot of important topics were tackled, and many led to actionable outcomes. Let me give you a few examples:
- A group got together on the topic of self-correction and, among other outcomes, worked on the idea to have commentaries and replications listed in google scholar; they thought this through, reached out to folks who know the right person at Google to contact, and prepared an email with arguments
- Another group worked on promoting open science and transparency in journals, and put together resources such as a reading list for guest editors
- A reading list was created for teaching replicable and reproducible psychological science
- Resources for creating and finding open science jobs
- Others got together to develop a preregistration template for meta analyses which serves not only as a guide for authors, but also for as a best practices sheet for consumers and readers of meta analyses to vet their quality
- Having problems to convince your colleagues that open science is important? Check out these SIPS Quibs with useful and concise arguments
- The workgroup on increasing diversity / inclusion developed a peer-buddy system for SIPS first-timers, SIPS sibs, and discussed many other crucial topics such as increasing diversity and inclusion in awards, e.g. via crowd-sourced awards
- I was personally involved: in a 3-hour workshop I gave together with Jessica Flake on questionable measurement practices in psychology (materials; teaser video; full workshop video online soon); starting an open science community at your local university; creating an online index of searchable psychological measures with links to validity evidence & open data; a data-date website that allows researchers to tell others about their data without uploading them (sometimes impossible e.g. in clinical) to facilitate collaborations and empty file drawers; and some smaller projects
These are just some of the many amazing projects. A lot of it is work in progress, and your contributions will be welcome. You can find many more projects, including all workshops, on the OSF site of SIPS2018.
SIPS lasted for three days, and I enjoyed every second. It showed me how ineffective most of the conferences I usually go to are: people talk about work that I can just read about in preprints or papers (on my own pace); real horizontal discussions or participatory contributions are the exception. This is the reason why I have spent less and less time at talks when going to conferences over the years, and more and more time meeting up with colleagues for coffee.
SIPS was just that: brainstorming with colleagues while having coffee, with some structure on top. The clear goal was not to bemoan the state of psychological science, but actively work to improve it, and to obtain actionable outcomes. And I’m not going to lie: it worked out much better than I had anticipated. Self-organization is a thing, especially among people with a shared goal, and especially if there is a supportive atmosphere of one team: collaboration instead of competition. See here for my very personal highlights.
I will definitely be there for SIPS 2019 in Rotterdam, but see a few challenges moving forward. Two we discussed already on Twitter, in a collection of potential improvements for next year:
Unpopular thought 1: Calling on people to introduce themselves is THE HORROR for people with social anxiety disorder (just going by prevalence rates, at least 5 people in the big room). Should make sure to communicate people can opt out.— Eiko Fried (@EikoFried) June 26, 2018
Unpopular thought 2: it needs a lot of social media skills to navigate SIPS successfully. Could consider teaching these next year maybe in the spirit of inclusion.— Eiko Fried (@EikoFried) June 26, 2018
The first pertains to the introduction where everybody got together in a room (over 200 people) and briefly introduced themselves to the crowd. For me personally, that worked very well, and I wouldn’t have connected with some colleagues hadn’t I heard their introduction; others found it a bit slow. The second point pertains to the fact that I saw some people being a bit lost during SIPS (e.g. due to a lack of a Twitter account). That’s ok, social media are just important tools for organization these days, but maybe we can help such researchers next year better to be involved (e.g. by teaching a ‘How to use Twitter’ workshop), and aim to be inclusive also for some more senior researchers who might not be that tech savvy1.
Another major challenge that methodologists are familiar with is scalability … quite a few of our models don’t scale well if you involved too many variables due to e.g. combinatory explosion. This will also be an issue for SIPS 2019 because I see many core features and mechanisms of SIPS getting in trouble with 500 or 1000 attendees. Take hackathons, for instance. Groups often had a size of 10 or 15 people, but because people were encouraged to come and go, new researchers regularly joined sessions an hour or more after the start. To be inclusive, we updated them on what had happened so far: “We’ve been working on X, we are struggling with Y, you can find the current state on this website. Thoughts?” But if hackathons get considerably larger, more structure will be required to make things work.
Related, I find it interesting to think about SIPS becoming mainstream. Is the goal of SIPS to abolish the need for SIPS? And how do we avoid replacing old gods with new ones? I know about 2 cases where open science badges were offered by journals in cases where researchers did not meet the criteria for them (and also rejected the badges). One explanation is that badges are new, and not everybody knows all the rules yet. Another less generous interpretation is that badges are the new impact factor, and editors are starting to badge-hack their journals.
Also, a request for next year: can we have twitter handle & twitter picture on our conference tags? Would make identification much easier.
SIPS was one of the best conferences I’ve ever been to — everybody tried hard to be part of the solution, and technology use (e.g. OSF, google docs) and other modern mechanisms and a clear code of conduct were in place to facilitate horizontal science where everybody had the same voice, no matter the ‘status’ or academic seniority. Collaboration instead of competition, team spirit and large-scale projects instead of insular science. It’ll be hard to go back to my normal conferences, and I’ll try to sow the seeds of SIPS elsewhere.
Want to support SIPS?
— Patrick S. Forscher (@psforscher) June 27, 2018
Additionally, donate to allow early career researchers to travel to Rotterdam next year:
WOW! In response to @benjaminle's challenge - we already have just over $1000 in donations from #SIPS2018 attendees to support travel funds for #SIPS2019 attendees and beyond. Every bit helps. Thanks to @benjaminle for kicking it off and ...— Katie Corker (@katiecorker) June 28, 2018