SIPS18 collected resources, and reflections of a SIPS virgin

The Society for the Improvement of Psychological Science (SIPS) hosted their third annual conference in Grand Rapids MI a few days ago. This blog provides a summary and some collected resources for those who couldn’t join, and a few reflections (praise & challenges) of a SIPS virgin.

SIPS: an introduction

I will not repeat all the hype and thank yous and tell you how amazing things were — see #SIPS2018 on Twitter for that. But for those who couldn’t join us: SIPS featured some invited workshops on topics integral to open science such as replications, multi-lab collaborations, sample size determination, meta-analysis, measurement, a crash course in R, and others. You can find the program here, and all workshops and other materials on the SIPS2018 OSF page. I believe some videos of the workshops will go online in a few days.

In addition to the workshops, there were many empty time slots available which filled up with sessions that people wanted to work on. Sessions had different formats (e.g. unconference, hackathon), but the main point is that SIPS created space for people to get together and work on important topics – and it totally worked.

Getting shit done

Although many of these meetings only lasted a few hours, and although most hadn’t been planned before the conference, a lot of important topics were tackled, and many led to actionable outcomes. Let me give you a few examples:

  • A group got together on the topic of self-correction and, among other outcomes, worked on the idea to have commentaries and replications listed in google scholar; they thought this through, reached out to folks who know the right person at Google to contact, and prepared an email with arguments
  • Another group worked on promoting open science and transparency in journals, and put together resources such as a reading list for guest editors
  • A reading list was created for teaching replicable and reproducible psychological science
  • Resources for creating and finding open science jobs
  • Others got together to develop a preregistration template for meta analyses which serves not only as a guide for authors, but also for as a best practices sheet for consumers and readers of meta analyses to vet their quality
  • Having problems to convince your colleagues that open science is important? Check out these SIPS Quibs with useful and concise arguments
  • The workgroup on increasing diversity / inclusion developed a peer-buddy system for SIPS first-timers, SIPS sibs, and discussed many other crucial topics such as increasing diversity and inclusion in awards, e.g. via crowd-sourced awards
  • I was personally involved: in a 3-hour workshop I gave together with Jessica Flake on questionable measurement practices in psychology (materials; teaser video; full workshop video online soon); starting an open science community at your local university; creating an online index of searchable psychological measures with links to validity evidence & open data; a data-date website that allows researchers to tell others about their data without uploading them (sometimes impossible e.g. in clinical) to facilitate collaborations and empty file drawers; and some smaller projects

These are just some of the many amazing projects. A lot of it is work in progress, and your contributions will be welcome. You can find many more projects, including all workshops, on the OSF site of SIPS2018.

Praise

SIPS lasted for three days, and I enjoyed every second. It showed me how ineffective most of the conferences I usually go to are: people talk about work that I can just read about in preprints or papers (on my own pace); real horizontal discussions or participatory contributions are the exception. This is the reason why I have spent less and less time at talks when going to conferences over the years, and more and more time meeting up with colleagues for coffee.

SIPS was just that: brainstorming with colleagues while having coffee, with some structure on top. The clear goal was not to bemoan the state of psychological science, but actively work to improve it, and to obtain actionable outcomes. And I’m not going to lie: it worked out much better than I had anticipated. Self-organization is a thing, especially among people with a shared goal, and especially if there is a supportive atmosphere of one team: collaboration instead of competition. See here for my very personal highlights.

Challenges

I will definitely be there for SIPS 2019 in Rotterdam, but see a few challenges moving forward. Two we discussed already on Twitter, in a collection of potential improvements for next year:

The first pertains to the introduction where everybody got together in a room (over 200 people) and briefly introduced themselves to the crowd. For me personally, that worked very well, and I wouldn’t have connected with some colleagues hadn’t I heard their introduction; others found it a bit slow. The second point pertains to the fact that I saw some people being a bit lost during SIPS (e.g. due to a lack of a Twitter account). That’s ok, social media are just important tools for organization these days, but maybe we can help such researchers next year better to be involved (e.g. by teaching a ‘How to use Twitter’ workshop), and aim to be inclusive also for some more senior researchers who might not be that tech savvy1.

Another major challenge that methodologists are familiar with is scalability … quite a few of our models don’t scale well if you involved too many variables due to e.g. combinatory explosion. This will also be an issue for SIPS 2019 because I see many core features and mechanisms of SIPS getting in trouble with 500 or 1000 attendees. Take hackathons, for instance. Groups often had a size of 10 or 15 people, but because people were encouraged to come and go, new researchers regularly joined sessions an hour or more after the start. To be inclusive, we updated them on what had happened so far: “We’ve been working on X, we are struggling with Y, you can find the current state on this website. Thoughts?” But if hackathons get considerably larger, more structure will be required to make things work.

Related, I find it interesting to think about SIPS becoming mainstream. Is the goal of SIPS to abolish the need for SIPS? And how do we avoid replacing old gods with new ones? I know about 2 cases where open science badges were offered by journals in cases where researchers did not meet the criteria for them (and also rejected the badges). One explanation is that badges are new, and not everybody knows all the rules yet. Another less generous interpretation is that badges are the new impact factor, and editors are starting to badge-hack their journals.

Also, a request for next year: can we have twitter handle & twitter picture on our conference tags? Would make identification much easier.

Get involved!

SIPS was one of the best conferences I’ve ever been to — everybody tried hard to be part of the solution, and technology use (e.g. OSF, google docs) and other modern mechanisms and a clear code of conduct were in place to facilitate horizontal science where everybody had the same voice, no matter the ‘status’ or academic seniority. Collaboration instead of competition, team spirit and large-scale projects instead of insular science. It’ll be hard to go back to my normal conferences, and I’ll try to sow the seeds of SIPS elsewhere.

Want to support SIPS?

Become a member, check out the many work-in-progress projects on the SIPS2018 OSF site, and join projects such as the open science accelerator:

Additionally, donate to allow early career researchers to travel to Rotterdam next year:

  1. Disclaimer: I mean to say that more senior people tend to be less tech savvy, especially in relation to social media, on average

9 thoughts on “SIPS18 collected resources, and reflections of a SIPS virgin

  1. Pingback: James Coyne sued me for cyberbullying - Eiko Fried

  2. Anonymous

    I really appreciate your following 2 remarks:

    1) “And how do we avoid replacing old gods with new ones?”
    2) “Another less generous interpretation is that badges are the new impact factor, and editors are starting to badge-hack their journals. ”

    I think it could be important to be aware of making real improvements, and not masking existing problems and/or creating a whole new set of problems. I think there is a nice saying that captures what i want to express: “Primum non nocere” which translates as “First, do no harm” (https://en.wikipedia.org/wiki/Primum_non_nocere).

    I am worried that having all these “hackathons” involving groups of enthousiastic folks might not be the best way to foster critical and careful thought, and come up with actual ideas that improve matters. As a possible argument for this i ask myself whether it is possible and/or responsible to think of something for a maximum of 3 days (which is how long SIPS took) before jumping to “action”. Sure, projects probably go on after SIPS but i fear some projects may already be in phases that might be way too soon for them to be in.

    As a possible 2nd piece of evidence for my point of view, i ask myself which projects from the past 2 SIPS meetings have lead to anything substantial and possibly useful from an improving psychological science- perspective? I can only come up with 2 i know of: StudySwap and a paper about ‘constraints of generality’ (which i kind of still don’t think makes much sense but that doesn’t matter).

    Both of these things can just as easily be thought or/written without any SIPS meetings of course, which is why i also appreciated a 3rd comment by you:

    3) Is the goal of SIPS to abolish the need for SIPS?

    Reply
    1. Eiko Post author

      I’m going to disagree for the sake of the argument here ;). I think a lot can be done in 3 days because people have thought about topics for a long time already. Your argument seems to rest on the notion that researchers come to SIPS as a tabula rasa and then start digging into solving issues, but I believe many of us are well aware of some issues, have thought about tackling them, and now are surrounded by peers (with somewhat different perspectives). I think this is a perfect opportunity to get things done.

      And of course, many groups “only” concluded with something like “ok next SIPS we will prepare this, and then try to tackle it”, such as a group Marlene Werner and others put together on a repository for validity evidence of measures. I’ve thought about that for months, Jessica Flake thought about this independently (and in a slightly different way), Marlene thought about it independently, and so did David Condon and many others. And now we had the chance to tell each other about our thoughts on this and ways forward, and found out that a few hours is not sufficient to move forward on the topic. So we discussed challenges, wrote down the next steps, and formed an inofficial working group. Who knows, maybe we’ll find the time over the next year, and maybe it’ll be a big topic we’ll tackle at SIPS 2019.

      “As a possible 2nd piece of evidence for my point of view, i ask myself which projects from the past 2 SIPS meetings have lead to anything substantial and possibly useful from an improving psychological science- perspective?”
      I wasn’t there, but what about the inclusion hackathon that came to many results that were implemented this year? What about the psych science accelerator? I think there are many initiatives that came out of SIPS last year. Would be interesting to ask that question on Twitter and collect some resources.

      Reply
      1. Anonymous

        “I’m going to disagree for the sake of the argument here ;)”

        Thank you :)

        You wrote: “I think a lot can be done in 3 days because people have thought about topics for a long time already. Your argument seems to rest on the notion that researchers come to SIPS as a tabula rasa and then start digging into solving issues, but I believe many of us are well aware of some issues, have thought about tackling them, and now are surrounded by peers (with somewhat different perspectives). I think this is a perfect opportunity to get things done. ”

        Yes, that’s the only “come-back” i could foresee to my point, and here is my “come-back” in 3 points:

        1) Sure, specific topics/issues will have probably been on people’s minds, but i reason most probably not as detailed as what the projects are about as they develop during the 3 days.

        2) Even if a certain project would be exactly what someone has thought about months before, then the input of the others during a “hackathon” will have to be critically thought about as well. Which then returns to my point about whether it being responsible to jump “in to action” after a maximum of 3 days thinking about this new input/information.

        3) If you keep going back to SIPS every year, and think and/or work on a project for 3 days i wonder if that’s a good way of “getting sh#t done”.

        From my understanding SIPS was sort of the “let’s actually do something about things” version of traditional conferences where there seemed to be “a lot of talk, but no action”. I am all for doing stuff, but i wonder if it’s the best idea to use a “hakathon” 3 day conference + “getting sh#t done” -atmosphere where everyone is “all fired up” and ready to “change to the face of science” and who knows what. This is why i really appreciated your comment “is the goal of SIPS to abolish the need for SIPS?”.

        Reply
        1. Eiko Post author

          Sorry, I was traveling. Are 3 day conferences ‘the best way’ to get things done? No, of course not, you want weeks or months, larger teams, more international teams, and so forth. But nobody would argue SIPS is the best way to get things done, and nobody would disagree that there are better ways to get things done than short conferences. The point was that SIPS was different than other conferences I have been to, that I enjoyed that very much, and that we got much more done than I had anticipated. For next year, I will prepare two topics that I have thought about a lot (and others too) and get some people together for hackathons (one on formalized theory in psychology, one on validity evidence for measures used in psychology).

          Reply
    2. Anonymous

      “I can only come up with 2 i know of: StudySwap and a paper about ‘constraints of generality’ (which i kind of still don’t think makes much sense but that doesn’t matter). Both of these things can just as easily be thought or/written without any SIPS meetings of course (…)”

      Here is an example of what i meant to make clear, should it be useful in some way or form. Here are some slides that state that the idea of “StudySwap” was generated at SIPS 2016: https://osf.io/afjce/

      And here is something (very) similar to (the idea behind) “StudySwap” that seems to have been thought of prior to SIPS 2016, and has been discussed several months before SIPS 2016 on the Open Science Framework Google Group:

      https://groups.google.com/forum/#!topic/openscienceframework/vg79mZF2Azs

      Reply
  3. Anonymous

    I wasn’t there, but looking at some tweets, and your above post, this must have all been very useful indeed! I think my favorite part of SIPS 2018 has got to be 2 things, or perhaps the combination of them:

    First of all, SIPS handed out awards to, at least one, individual. This surely has got to be the way forward to “change incentives” and “improve psychological science”. Well done SIPS!

    Secondly, the final words (if this tweet is correct: https://twitter.com/lakens/status/1011718047139127296) by (what i believe is) the president of SIPS keep ringing in my mind:

    “Open Science is a behavior, not a t-shirt or a badge. It is not about the individual, but about doing high quality science”.

    So true, it’s not about the individual indeed.

    Reply
    1. Eiko Post author

      Regarding the award, I’ve written about problems with individual awards in response to the APS Rising Star award just 4 weeks ago:
      http://eiko-fried.com/reflections-on-aps-18-open-science-transparency-and-inclusion/

      See also Chris’ own reflections on this:
      https://twitter.com/CRChartier/status/1012436167827116034

      As immediate response (less than half a day after the award was handed out at SIPS), there was an email about recommending ideas for how to structure the award next year, such as a crowd-funding the suggestions.

      Reply
      1. Anonymous

        “As immediate response (less than half a day after the award was handed out at SIPS), there was an email about recommending ideas for how to structure the award next year, such as a crowd-funding the suggestions. ”

        Again i worry this is not a real solution. To me, the solution is to get rid of awards altogether, not creating different ways/criteria to hand them out (which can again be manipulated, and/or handed out unfairly, etc.). To me it’s incomprehensible to hand out and/or accept (individual) awards concerning in science, because it fundamentally is not what science is about.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.