Replication Crisis: From Philosophy of Science to Infrastructure

📉 A brief history of the replication crisis: from philosophy of science to numbers #science #replication #methodology #statistics #research

Modern science is formally organized around a simple principle: hypotheses must be testable and falsifiable (Popper, 1930s). If a result does not withstand repeated verification, the theory should give way to a better one.

But in the 2000s–2010s, it became clear that the very possibility of “repeated verification” was not so straightforward.

Gradually, systematic replication projects began to emerge — and the numbers turned out to be unpleasant.

Here are some benchmarks across different disciplines:

‣ Medicine — ~44% reproducible results (Ioannidis, 2005)
‣ Psychology — 36 out of 97 (~36%) (Open Science Collaboration, 2015)
‣ Experimental economics — 11 out of 18 (~61%) (Camerer et al., 2016)
‣ Social sciences — ~62% (Camerer et al., 2018)
‣ Philosophy — ~78% (Cova et al., 2018)

The picture is similar everywhere: a significant proportion of “significant” results do not replicate, and effect sizes in replications are usually noticeably smaller than the originals.

📉 Replication crisis → replication infrastructure

#science #replication #economics #methodology #research

If in the 2010s the replication crisis looked like a set of alarming numbers (“half of the results don’t replicate”), then recent years are no longer about diagnosis, but about building infrastructure.

The focus is shifting from one-off “checks” to permanent, institutional mechanisms of reproducibility.

A good example is the line of work by Dreber–Camerer in experimental economics. One of the central questions: can we predict replication success before it happens? It turns out that yes — the scientific community generally knows which effects are “fragile,” but systematically does not use this knowledge. The actual result was 61% replications (11 out of 18), meaning experts (with 71%) slightly overestimated reproducibility, but were not far off.

What changed institutionally by the 2020s:

📦 Many journals require data + code + replication package by default
👩‍🎓 Replications are embedded in courses and PhD programs
🧪 They do not only “repeat,” but also meta-reviews

Now in academia, a whole ecosystem has grown around this:

‣ Databases and catalogs of replications (e.g., OFS)
‣ Preregistration, especially in medical research
‣ Meta-research centers (e.g., MetaResearch)
‣ Project-based “replication laboratories” (like LabSQUARE)

It would be great to institutionalize this in industrial practices as well, what do you think?