What if science moved faster, not slower?

A lot of researchers think the academic system requires them to work too quickly. Maybe they have it backwards.

It’s a common sentiment among researchers that science requires them to move more quickly than they would like. There’s a sense of urgency to get research out right away—to publish as much as possible, as quickly as possible. There isn’t sufficient time to think through things deeply. In an ideal world, a scientist would be allowed the time to move at their own pace. This would allow scientists to work on deeper, more meaningful problems. According to this way of thinking, science moves too quickly. However, there’s an alternative possibility. The problem is not that scientists are rushed into publication. It is that they don’t receive outside feedback on a project until after it has been completed. In this view, science suffers from the opposite problem: it is way too slow. What would the scientific process look like if there was a more effective pipeline for getting feedback on a project before it gets set out to a journal?

---

Science, fast and slow

For most scientists, a big part of doing science is the pressure to publish. Scientists feel this pressure because the alternative to publishing is perishing, which is generally considered an undesirable outcome. Unlike other human pursuits, science isn’t supposed to be this way. It should be about revealing profound insights about the nature of the world, the kind that reverberate through the ages. Or at least something close to that. So when a scientist, in her off time, holds the pressure-cooker reality of the enterprise up to the leisurely tempo of this romantic ideal, many scientists come to the same conclusion. Science moves too fast.

At its core, I’m sympathetic to this concern. The scientific enterprise has deep structural issues. It puts pressure on researchers to churn out lots of papers, many of them lower quality than they would like. It rewards scientists almost exclusively on whether they have a lot of highly cited papers in high-impact journals. The culmination of this incentive structure is that it can make science feel like a process of flimsy, short-term optimization rather than the robust, long-term project that it ought to be.

But while I agree with the core of the concern, I think the diagnosis is wrong. The problem is not that science moves too fast. It’s that it doesn’t move fast enough.

Specifically, the problem is the timeline of feedback. The vast majority of academic research projects do not receive meaningful third-party feedback until well after an experiment has been designed and implemented, the data collected and analyzed, and the paper formatted and submitted for publication.

In any other field, it would be ludicrous to expect a product be finalized before it hits the shelves. From big corporations to small startups, there’s a pipeline for testing out a small thing before having to make the full investment in the big version of it. Feedback from an external set of eyes other than the people developing the product is a crucial part of the process.

Scientists also need a pipeline for testing their ideas before they have to turn them into finished products. As it currently stands, scientists (specifically graduate students) can spend years working on a project and not even know whether anyone will even care about what they’ve found even if an experiment turns out exactly as they hoped. It’s not a good way to develop anything, whether a scientific or consumer product.

The issue is not that publication come too soon. It is that feedback comes too late.

---

Proposal: The Scientific MVP

My proposal is that science needs a conceptual equivalent to what is known in the startup world as a Minimum Viable Product. An MVP is a stripped-down version of the ultimate product a company plans to build. It is designed to empirically test the underlying assumptions of that product and validate whether or not it will, if built in full, find a market eager to use it.

Though it predates the concept of the MVP, a classic example would be Facebook. It didn’t start out as a behemoth social media platform attempting to connect everyone on the planet; it started off trying to connect students on a single college campus, then a select network of campuses. It validated the assumption that the style of connection offered by Facebook’s nascent product was a kind of connection that people were interested in. With an MVP, the point is build the smallest version of your ultimate vision. By testing whether the light-weight version is received in the way you hope, you get useful feedback about how to improve the core of your vision before building out the rest of it.

The process of developing an MVP is often compared to the scientific method—but applied to startups rather than traditional academic disciplines. The point is to identify the untested assumptions of a product, then create an experiment that will show whether the assumption holds up. It is a process of empirical experimentation.

This scientific inspiration may seem somewhat ironic in the context of my proposal that science should take a page from the startup playbook. But the practices of science are often appropriated by Silicon Valley, where they are refined and redeployed with increased effectiveness. This refinement is possible in the resource-rich environment of startups in a way that it is not in the resource-scare one of academia. For example, in How Google Works, Eric Schmidt’s book on the company’s cultural, innovation, and management practices, Schmidt describes the company’s organizational structural was originally designed to mimic the kind of small working groups found in scientific laboratories. After all, Google’s cofounders, Larry Page and Sergey Brin, met as PhD students. But while they were inspired by academic precedent, the management practices development at Google and other modern tech companies have gone far beyond the haphazard, informal practices of academia. The same is true with the notion of empirically-informed feedback in the minimum viable product.

The process of the MVP looks like this: First, identify the riskiest, highest-impact assumption of your product. Then design the smallest experiment necessary to meaningfully test this assumption. Once those data have been analyzed, refine the assumptions and continue testing them as necessary. The objective is to minimize the amount of time it takes to complete this cycle.

Typically, in a startup the assumption being tested is whether there is a market for a given product. The founders think they’ve got an interesting idea. Initially, they’re assuming that others will find it interesting as well. They need to test that assumption. In a study of more than one hundred startups, 35% failed because of “no market need” for their product. This was the second most common cause of failure, with the most common (38%) being “ran out of cash.” Both of these problems are designed to be addressed by the iterative experimental process of the MVP. If your riskiest assumption is that people will actually want to buy the product you’ve built once you’ve built it, then that assumption needs to be tested. And if you don’t test it quickly enough, you’re going to run out of money before you can successfully refine your assumptions about market interest. Of the 111 failed startups surveyed, three quarters died because of a problem the MVP was designed to fix.

---

“No market need” science

A lot of scientific work suffers from the same problem: a lack of interest once it is released into the relevant market. A scientist puts their work out into the world. And it turns out that even though the work is solid, the finding isn’t something that other scientists find especially interesting.

How big is this problem? What percentage of science suffers from the equivalent of “no market need”? It is difficult to estimate what percentage of scientific papers go uncited. And evidently a lot of the numbers that thrown around are highly exaggerated. But according to a 2018 analysis published in Nature, the number depends a lot on which discipline and which journal one is looking at. The overall percentage in the natural sciences could be around 21%. Of course, there are a lot of scientific papers that have a non-zero but exceedingly modest number of citations. This would expand that figure of “no market need” science quite a bit. Either way, I think a lot of scientists would anecdotally support the claim that there’s a lot of fluff out there.

The scientific community needs an equivalent concept of a minimum viable product for the same reason as the startup community. You need to show that on the other side of this big investment of time, money, and energy, someone will actually care about what you’ve done. As it stands in science, it’s possible to get through an entire PhD without actually having to haul your work before someone who isn’t you or your advisor and see how your colleagues who aren’t emotionally invested in the work react to it. In science, no market need means no citation. No citations means no career.

As much as scientists may despise this fact, having a career as scientist isn’t just about doing good science; it’s about getting other people to recognize how good that work is. As a negative example, there is the recent controversy about Francesca Gino’s alleged fake research on dishonesty. That’s a case of low quality science with high quality marketing. Another more entrenched version of this would be the replicability crisis—many headline-grabbing findings (not just in psychology, but other disciplines as well) turn out to be more intriguing than true. Getting people to appreciate one’s research is not the sole goal of the scientist. But it is one of them.

Here’s a more positive example. In an interview from a couple years ago, I asked Mark Granovetter about his paper, “The Strength of Weak Ties.” He told me that he had initially titled the paper “Alienation reconsidered.” He sent that paper to a journal and the reviewers hated it. Granovetter was discouraged, but didn’t give up. He reworked the paper a bit, keeping the same central themes and ideas while adjusting how they were framed, then presenting it as “The Strength of Weak Ties.” Today, this paper has over 70,000 citations. It is one of the most cited papers in all of sociology.

Granovetter’s example is a genuinely foundational paper in its field, but one needed some reworking before his colleagues were able to make sense of it. How many other great insights languish in obscurity because they didn’t get the adjustment needed to make them understood by the widest possible audience?

Again, my point is not that there is no feedback loop in science. It is that the feedback loop is way too long. It shouldn’t be the case that a researcher conducts a whole paper’s work of experiments, then submits it to the scrutiny of peer review with their fingers crossed hoping that their colleagues don’t hate the work they’ve done. Granovetter’s example shows that even good science can benefit from tweaking its framing. Peer reviewed feedback comes late in the process and often isn’t that helpful. Instead, we need to construct a scientific pipeline where this sort of reworking comes early in the process, and sharpens an idea into its best, most useful possible version.

A lot of scientists will recoil at this proposal because it turns scientific inquiry into something vulgar: a product. While I’m as drawn to the romantic vision of science as anyone, there’s also a pragmatic side to the way science is produced. It can be more or less effective. And from this pragmatic view, science is already a product whether you like it or not.

The product is scientific papers. This is the sole currency of the scientific community. A researcher’s work is only as valuable as the papers they publish; it is the way they are judged by their peers, as well as their departmental employer. Scientific papers themselves are not judged in terms of the amount of money they make for a company. But they are judged in a similarly quantitive way: based on the number of citations they receive. The prestige of the journal may also matter, but mainly because it is perceived as a predictive measure of the paper’s long-term potential to rack up a large citation count. The product’s “users” are other members of the scientific community, who are themselves engaged in the same process. This side of science is itself a capitalistic enterprise. The capital is not monetary, but measured in units of prestige.

Another concern would be that this proposal discourages long-term innovative thinking. It incentivizes researchers only to work on ideas that will succeed in the short-term, ones that are easily digestible to a broad audience. Who knows what will turn out to be valuable in the long-run despite being initially under-appreciated? The entire point of scientific research is to take chances on finding out something that may not be fully explained by current thinking.

In general, I agree with this concern: Science should be about big, long-term impact and not easy, short-term success. However, this concerns dramatically overestimates the extent to which scientific practices are currently set up to do this. Most scientific projects are aimed at incrementally adding minor contributions to existing frameworks. They’re just not doing so as effectively as they could be.

In fact, the MVP approach would likely open scientists up to take greater risks. Under the current scheme, scientists have to be reasonably certain that a given project will pass peer review scrutiny. And because only fully completed projects make it to the peer review threshold, scientists don’t invest in a project unless they’re sure it will pass peer review scrutiny. So they are incentivized to make cautious choices about which projects to take on. If instead they could validate a small part of a larger, more ambitious vision, then they might be more willing to take a risk.

The process of getting feedback does not make the scientist beholden to a committee. It does not subject one’s work to the homogenizing forces of short-term optimization. Certainly not any more than the current feedback loop of peer review. Instead, the MVP process I’m proposing gives scientists further information to work with, sooner and in greater depth.

I mean, after all, we’re talking about scientists. If anyone should be comfortable with the idea that more information is better, it’s them.

---

But what about… peer review? lab meetings? departmental talks? conferences?

But don’t we already have this process in place? Isn’t this what peer review is all about? In my own example, I cite how Mark Granovetter successfully reframed his paper via… peer review. And for that matter isn’t the iterative process of finding a solid framework, then publishing lots and lots of papers making small improvements to that initial idea the kind of process I’m describing?

All good questions. But no, I don’t think so. Peer review is not the same thing as what I’m describing. There are two reasons:

(1) Timescale matters a lot.

The problem I’m trying to solve is that researchers, particularly PhD students, can spend a very long time working on a project before getting any sort of validation or constructive criticism from someone who doesn’t have an emotional investment in the project’s success. Imagine what a PhD would look like if instead of investing years into a project before getting critical feedback, the system was set up to provide actionable feedback in a matter of weeks or months. My sense is that it would make PhD students feel less isolated, more confident in their work, more effective in finding a niche to fill, as well as serving to make their initial good ideas even stronger, disabuse them of their bad ideas, and decrease the probability they’d spend their whole PhD on a project that doesn’t end up working. I don’t think anyone suspects peer review is an effective cure for any of these issues.

(2) Peer review isn’t that effective.

In a post on his Experimental History substack, Adam Mastroianni argued that peer review is ineffective system, and one that hasn’t adequately been tested by scientific scrutiny. In other words, we have no empirical evidence that our primary means of evaluating empirical evidence is any good. I agree with that argument overall, but specifically for the purposes of a minimal viable product the problem is that peer review doesn’t give you feedback about the fundamental assumptions of your experiment—whether any one will care that you did it.

Peer review is designed to poke holes in your experimental methodology (which in practice often means getting feedback on whether it accords with the reviewer’s personal opinions about how best to study a topic). But it is not designed to answer the question: “If I run a scientifically sound version of this study, will anyone care about it?” As suggested by the Francesca Gino example, lots of scientifically questionable work gets through the porous filter of peer review. And lots of scientifically sound work gets through peer review, only to languish in academic obscurity. So no, I don’t think peer review is a good solution to the problem I am interested in solving.

The scientific community also has other, less formal methods for providing feedback to nascent research. For example: lab meetings, departmental brown bag talks, and poster presentations at conferences. These can be useful. But ultimately their utility depends on the culture and practices of the lab, department, or conference. It isn’t a standardized part of the scientific process, certainly not to the extent that peer review is.

In The Art of Computer Programming, Donald Knuth wrote that “premature optimization is the root of all evil.” It seems to me that the incentives of scientific production require scientists to optimize their studies before validating their core assumptions. As much as the idea of treating early-stage scientific research as a minimum viable product might be unsavory to many scientists, it is an unavoidable observation that there are many commonalities in the developmental processes of science and startups. The unfortunate fact is that the startup people have lots of money and the academics have comparatively little. Based on this alone, startup people have invested a lot more in figuring out the best-practices of their process. Because science as an enterprise has invested comparatively little into figuring out the best practices of its own production process, I think this should give scientists pause to gather up enough intellectually humble to be willing to learn from the example of a field which may be less rarefied but is far better funded.