In a recent interview with Zhixian Lian from the Chinese Social Sciences Today (CSST), I was asked a number of questions about the value of exploratory research.
You can find my answers below—the interview is also available online and was translated to Chinese. After a brief brief summary of exploratory and confirmatory research, it leads with:
To explore the definitions, distinctions, and evolving roles of these two approaches — as well as the reasons behind the decline of exploratory research and potential remedies — CSST spoke with Eiko Fried, an associate professor of psychology at Leiden University in the Netherlands. Fried argues that the growing marginalization of exploratory research not only undermines the comprehensiveness and innovation of scientific inquiry but may also pose a latent threat to the robustness of scientific progress.
CSST: What’s the importance of exploratory research, especially to social sciences?
“Science entails, somewhat simplified, two steps. First, establishing phenomena, defined as robust, recurring features in the world. And second, explaining these phenomena via theories. Some phenomena can be readily observed without collecting data or performing statistical analyses, such as that humans have the capacity for language. Other phenomena are discovered in data, such as the relationship between smoking and lung cancer, or that depression rates are higher among women.
Exploratory research and its counterpart, confirmatory research, map onto these two steps to some degree. Exploratory research is more involved towards the side of establishing phenomena: it entails looking around in the world carefully and thoughtfully to see whether anything emerges that looks interesting. Confirmatory research is more involved in theory building and explanation, and commonly defined as taking some sort of theory, generating a hypothesis from it, and seeing whether this hypothesis can be confirmed in data, for example via experiments. The 2012 experiment at the Large Hadron Collider at CERN was confirmatory: the existence of the Higgs Boson was predicted by a theory (the Standard Model of particle physics) and confirmed in an experiment, ultimately leading to the 2013 Nobel Prize.”
CSST: How do you see the biggest disparity between exploratory research and confirmatory research?
“Exploratory research can generate hypotheses, make interesting discoveries worthwhile pursuing further. Think of it as dropping many researchers into some unexplored wilderness. They walk around, explore the jungle, climb over mountains, swim through lakes. Some of them come back and say: ‘there is something interesting in this direction.’
Confirmatory research is better equipped to interrograte these original findings in more detail, and test hypotheses. To follow our analogy, the research community arrives at an initial theory, puts money together, carefully plans their trek over the mountain where supposedly interesting findings are, and carries out a series of adventures to confirm that there is something there, how it looks exactly, and how it can best be explained. If things work out, this previously unexplored part of the scientific landscape behind the mountains then gets added to our scientific map of what is known.”
CCST: What is shifting researchers away from exploratory research?
“We’ve had a replication crisis in the social sciences (and elsewhere): well-known findings we thought were robust phenomena ended up not being replicable, meaning they were not robust in the first place.
My brief explanation of this crisis is that researchers who were dropped in the wilderness—which is really what most of psychology is at the moment, a lot of things are unknown—came back from over the mountain top. But instead of saying ‘there is something interesting there’, they came back saying ‘I had hypothethized based on my theory that there is something there, and I found it! Heureka.’ And then they usually gave a TED talk.
In other words, the problem is that researchers were actually during exploratory work—they collected datasets with many variables and related them all to each other—but they wrote up their papers as confirmatory. This was inappropriate, because they didn’t really have clear hypothesis (let alone a strong theory from which to derive hypotheses) from the start, which introduces false positive findings into the literature: phenomena that are not robust.”
CCST: Social sciences lack open-ended exploration to generate hypotheses, and there is a persistent bias that exploratory research is unscientific. What do you think is the root of this bias? Could it be linked to the over reliance on quantitative empiricism in current social science paradigms? How would you analyse this phenomenon?
“There is some careful and faithful exploration of new spaces in the social sciences, but it is not so common, and I agree that there is a misunderstanding that good science is confirmatory science. Daniel Nettle made this point very well in a blog post a few years ago, arguing for the immense importance of exploratory research:
‘You shouldn’t have to claim to have a fully identified theory (especially if you haven’t actually got one, which I think is true of much psychological research, including plenty that is well worth doing). [..] Biology had hundreds of years of taxonomy and natural history before it had formal phylogenetic models; geology had detailed maps of rocks before it accepted plate tectonics; people had been studying the motion of planets long before Newtonian mechanics, and so on.’
We don’t have a lot of theories I consider ‘strong’ theories in psychology. As Robert Cummins put it: ‘We are overwhelmed with things to explain, and somewhat underwhelmed by things to explain them with’. We have written about this lack of clear theories in recent works (e.g., 1, 2, 3). Others have written about it, too, and many of us agree that the over-reliance of social sciences on testing and confirmatory work comes from the fact that we do not have a lot of robust theories. Denny Borsboom called this phenomenon ‘theoretical amnesia’ in a widely read 2013 blog post: ‘It is a sad but, in my view, inescapable conclusion: we don’t have much in the way of scientific theory in psychology. [..] And that’s why psychology is so hyper-ultra-mega empirical’.”
CCST: There is a lack of training in exploratory research, with existing curricula focusing on hypothesis testing. If tasked with redesigning a social science research methods course, how would you balance teaching exploratory and confirmatory approaches?
“Social scientists indeed receive excellent training in testing things. But you know what we get basically no training in at all? Generate new ideas. Ecologist Marten Scheffer refers to this as ‘the forgotten half of scientific thinking’ and advocated in his brief paper on the topic that generating novel ideas is of crucial importance. He talks about the importance to stimulate associative thinking, and that scientists need to relearn habits that help with that. ‘Solid scientific skills are needed to weed out right from wrong. However, our current teaching and routines are focused almost exclusively on those skills, whereas the best science tends to come from a balanced mix of rationality and adventurous association.’
So we should consider teaching students to do this better, and as part of the Young Academy Leiden, we actually organized a workshop at Leiden University—together with Marten Scheffer—on this topic a few weeks ago, which was very well received.”
CCST: Journals’ preference for statistically significant results often drives researchers toward confirmatory studies. Some advocate for journals to embrace exploratory research. Such studies typically lack definitive conclusions. What innovations do you believe are needed in the peer-review standards for exploratory papers? How to enhance credibility?
“I would slightly rephrase the first sentence: journals’ preference for statistically significant results often drives researchers to write up their papers as confirmatory studies (although they may have been conducted as exploratory in many cases). I am not sure we desperately need dedicated formats for exploratory work, although I would welcome such formats. But I would perhaps rather try to change the perception in the social science about exploratory research, which is a crucial step in establishing robust phenomena. This begins by teaching students about the value of doing exploratory work.”
CCST: Exploratory research must still adhere to scientific rigor, yet traditional standards (e.g., pre-registration, variable control) might constrain openness. How can we balance avoiding ‘data dredging’ with ‘preserving exploratory freedom’?
“A student recently came up to me after class bemoaning why I had assigned such a ‘bad’ paper to read; they explained that the paper was bad because it was not preregistered. For context, the paper I had assigned was a beautiful exploratory paper that shared all collected data (not just the analyzed variables). The student had misunderstood preregistration as a thing to strive for because it is a good thing in itself, rather than striving for preregistration because it can, in specific circumstances, be a useful tool to further open science values such as transparency.
So if someone collects a very large dataset, and finds one particular relationship among variables of interest, and writes their paper up in a way that expresses this, and shares all their data, I’m not so concerned. If someone does the same but writes their paper up as confirmatory, and doesn’t share the data — today’s standard in psychology — I am much more concerned.”