At this point it is uncontroversial to say that science needs to stop using the PDF article as the unit of knowledge and currency. The unbearable slowness of scientific publishing, the profit motive and margins. I’m not saying anything new. The PDF also sucks because it’s hard to extract structured information from, which makes it hard to do evidence synthesis. As a result, we do much less evidence synthesis than is needed. And evidence synthesis ultimately undergirds most policy and medical decision making. I can see second by second real time odds for any sporting or newsworthy event, but a school board can’t see the best evidence on whether their 8th graders should be taught algebra. As a society we don’t treat this as an important problem. Again, not controversial.

Not only is the problem well understood, but the solution has already been laid out. What we need is AI-assisted living evidence synthesis - (1) an open knowledge graph of atomic claims (2) claims linked to evidence (3) assessment and synthesis of each piece of evidence (4) continuous updating with new data.

What few realize (yet) is that the technical capacity to build this vision for a significant portion of science already exists. Not only that— scientists and startup teams are already building many of these components. I know this because I’ve been surveying the space and talking to many of the builders. There are some missing pieces: for example evaluations of how well some of the components work. But at this point most of what’s missing is a fully end to end working integration of all of these parts. In the rest of this essay, I’m going to lay out all the parts of a working living evidence layer for science and who is working on them, and propose concrete next steps for building this system.

What’s now possible

The diagram above outlines the components of a living evidence synthesis platform, including some of the teams working on each component1. Scientific PDFs are processed into claims with associated evidence. The evidence is subjected to a forensic audit, methodological evaluation and robustness and reproducibility checks. Finally it’s given a weight in a continuously updating synthesis. What follows is a description and status of each component and a few of the teams working on them.

Document understanding

The first thing you need to be able to do is turn an article into structured data. Mostly that means parsing PDFs. There are often multicolumn layouts that confuse non-specialized PDF to text processing libraries. For scientific papers, there is the added complexity of parsing formulas and tables and figures. This is a really hot area - there are startups offering APIs, and it seems like a new open source package gets posted to Github every few weeks. What follows isn’t exhaustive. A package called GROBID was the state of the art for a while, they didn’t update their package for nearly two years until very recently. In the meantime reducto.ai released an AI powered PDF extraction API, PaddleOCR became popular, IBM released a model called Docling, and both Mistral and Gemini created models and libraries. I also know of at least one other well-funded psychology research group working on a paper parser. By contrast, there are few open evals in this space, with no extensive evals for complex table comprehension in particular. Nonetheless, I'm confident this will be a solved problem soon, given the combination of LLM advances and developer interest.

Hypothesis level extraction

There has been increasing interest in comprehending the extracted text of papers and linking information to evidence for each hypothesis. A lot of work has already been done. Trialstreamer (Marshall et al. 2020) and RobotReviewer LIVE (Marshall et al. 2023) demonstrated automated extraction of trial population, intervention, and outcome at scale on clinical RCTs. PaperQA2 (Skarlinski et al. 2024) and Ai2 ScholarQA (2024) extended this to retrieval-augmented question answering with citation grounding. Elicit, Consensus, and SciSpace operationalized claim-level extraction for end users. OpenEval (Booeshaghi et al. 2026) is the most recent and most ambitious: 1.96 million atomic claims extracted from 16,087 eLife manuscripts using Claude Sonnet 4.5, grouped into ~299,000 results, with LLM evaluations showing 81% agreement with human peer review on a 2,487-paper subset. None of these solutions link claims to test statistics, as you would need to evaluate randomized control trials. This is why I built the evidence.guide API - to extract hypotheses and associated test statistics from behavioral science papers. The best public eval of this kind of extraction I’m aware of comes from the recent SCORE project - they had humans code thousands of psychology papers to extract their claims by hand. It would be extremely helpful to the world if all scientific PDFs were available as structured open data. I’ve been working to make this happen, both directly at Berkeley and through coordination with large entities I can’t yet speak of; as hard as it is to do, I think it’s possible2.

Forensic audit

A lot of work has been done on forensic audit, but some gaps remain. Of course, for biology papers that rely on images for evidence, there are a variety of tools (notably Proofig and ImageTwin) to spot anomalies. These are still well short of what sleuths like Elizabeth Bik can do on her own, but these tools are constant companions among fraud analysts. There’s someone working on auditing Excel files for anomalies, and a number of teams are automating numerical checks like GRIM and SPRITE, including Scrutiny project, the INSPECT-SR team as well as statcheck. The regcheck team is building a way to use AI to compare preregistrations to analyses in papers, to ensure there aren’t significant deviations. Nonetheless, there are many other kinds of anomalies to screen for, both public and less publicly known. And there are no formal evals for anomaly detection that I’m aware of. Still, there’s a lot to draw from in this space and I’m pretty certain we will be able to scan papers for most kinds of obvious anomalies in the near future.

Methodological review

This area has been white hot, though I fear for many of the startups in this space, because this capability may become commoditized. There are at least six different AI peer review companies, including Refine.ink, Reviewer3, ReviewerZero.ai, Q.E.D. Science, Paper Wizard, and Isitcredible. Coarse (a pun on refine) was also recently created as an open source alternative. These systems provide qualitative feedback on the content of papers, spotting methodological weaknesses and mathematical errors. They seem to work pretty well, and many academics report bitterly that they exceed the average quality of typical peer reviewers. But there are few evals here either. What evals exist so far involve using LLM-as-judge (circularity problems abound) or comparing against human reviews of questionable quality. What you’d ideally want is an eval that measures capturing known errors in papers3.

Reproducibility and robustness

Another active area has been using AI agents to automate computational reproducibility4 and robustness5 checks in papers that report numerical results. For more recent papers where data and code are available, AI agents can see whether they can re-run the analyses and produce the numbers reported in the published paper. In addition to a handful of individual academics who have been experimenting using Claude Code for this, the Institute for Replication is a leading group working on building an end to end system. The evals related to this problem are the most mature, with CORE-Bench (Siegel, Kapoor, Narayanan 2024) and PaperBench (OpenAI 2025) available to benchmark agents on this task. There is also work on getting AI agents to test alternative ways of analyzing the data to ensure the results are robust to small analytic design choices.

Synthesis

This is the most underdeveloped area where significant investment is required. Although some automated evidence synthesis systems exist — for example, otto-sr is building an AI agent to write systematic reviews — none of these incorporate the full range of paper level signals to weight evidence appropriately. Nor is there anything like an eval or a gold standard for a good systematic review. Arguably Cochrane reviews are the closest we have to gold standard human systematic reviews, though I’ve heard academics in the know complain about their uneven quality. A key question for a synthesis platform is how to weight anomalies and methodological issues in assessing the quality of a piece of evidence. This is an unsolved problem and one I’m very keen to work on.

Continuous updating

There are many pieces of basic infrastructure available for monitoring for new research and initiating updates. OpenAlex is the current open citation graph. Retraction Watch integrated into Crossref in October 2023. Scite tracks how citations support, contrast, or mention prior claims. The Living Evidence Network demonstrated continuous-update workflows in clinical guidelines. Engineering this is a relatively straightforward task.

When you look over this technical architecture and all the progress being made, it’s hard not to be optimistic that a living guide to scientific evidence will be built.

The stakeholders are ready

The social infrastructure for this is starting to coalesce — it’s not just a pie in the sky academic exercise to imagine this coming into existence. Institutions like the Center for Open Science, the Institute for Replication, the INSPECT-SR, the Living Evidence Network and more are all working on scaling work to improve research quality.

Funders are also aligned. The Sloan Foundation has funded living evidence work through COS. Coefficient Giving supports the Institute for Replication and COS. The Astera Institute and the Institute for Progress has shown interest in this space. NIH has established an Office for Replication and Reproducibility. Although there are (very unfortunately) serious headwinds in science funding generally, there is an active group of funders interested in metascience.

A brief word about what I’ve been doing at RDI. First, as a Visiting Scholar at Berkeley I’ve been actively figuring out how a non-profit and a public university can conduct and make public the results of large-scale academic article data mining. With some of the money I raised from donors, I commissioned a legal analysis of recent case law and publisher text data mining (TDM) agreements in order to understand whether a massive open data mining of academic articles is possible (with caveats, it is). I’ve also been working to bring together stakeholders in this space, and identify gaps. I’ve also been doing some software development in this space, with more to come.

A pilot proposal

The assumption undergirding all of this is that an AI, given all this information, would make the right judgment about a scientific claim with lots of conflicting evidence, weighing all the factors appropriately. That’s the hypothesis we need to test.

Randomized control trial research is the best place to focus on first. RCTs are used to make many of the important decisions in society - from medical trials to public policy changes. And they use a relatively uniform set of inferential statistics with lots of known and available diagnostics. Behavioral science experiments, within the broader realm of RCTs, should be first used as a testbed whose results can be generalized. Because behavioral science is at the vanguard of open science practices, replications abound (there are thousands of them) to serve as ground truth training data.

Key Hypothesis

Therefore the pilot would test, in behavioral RCTs that have been replicated, whether the quality of evidence for a claim can be used to accurately predict whether that claim will replicate.

Secondary Hypotheses

1. Compared to claims that replicated, non-replicated claims demonstrate a greater share of forensic anomalies in their source literatures.

2. Hypothesis level claim and statistic extraction is accurate enough to scale living evidence without onerous human review costs.

3. Replication prediction is more accurate than prediction markets6 or journal prestige.

If all the different quality signals we gather do accurately predict which studies will replicate, then we can use that model to score evidence to power the living evidence layer.

Why this is informative regardless of outcome

If the pilot succeeds, the architecture extends to medical RCTs (where Living Evidence already operates and integration is mostly about claim representation), then to slices of basic biology with stable replication structure. If it fails, the field learns which quality signals are load-bearing and which ones metascience has oversold. Either result is a contribution to knowing what the literature supports.

Conclusion

The drawbacks of the current scientific publishing system are known. Scientists agree, metascientists agrees, philanthropists agree: the published PDF plus citation graph isn’t the right substrate for maintaining a representation of the evidence base in science. The pieces needed to build the alternative either already exist or are rapidly taking shape. The community is forming around exactly this problem, with concrete partnerships and shared infrastructure. A pilot should start on behavioral science RCTs because that's the slice of empirical science most amenable to legibility, where replication ground truth is richest, and where the failure modes are best documented. What's been missing is the galvanizing mission to assemble these pieces into something that works. That's what I'm proposing to build.

I felt pressure to p-hack. To run a few more subjects and check the results, drop a condition, move failed studies into the file drawer, anything to cross that magical p < .05 threshold. My dissertation proposal was accepted on the condition that I get a positive result in an experiment. But I didn’t want to play that game. Instead I published a meta-analysis with a null effect as part of my dissertation. The findings didn’t support the hypothesis. That’s what the data said, so that’s what I reported.

Then I left academia.

The Problem Followed Me

I went into tech, working at top companies like Facebook, Google and Airbnb. I learned a lot about what it takes to build great products and run effective teams. Eventually I co-founded my own company, a Stanford nanotech spinout building a health-sensing toothbrush. The core technology was based on published research. Thousands of papers supported the sensor approach we were using.

The technology didn’t work. When we tried to replicate the foundational science, we couldn’t. Thousands of papers, and the basic claims didn’t hold up.

The problem had followed me out of academia and into industry! Meanwhile, the news kept confirming what I’d experienced firsthand.

Fraud Makes Headlines

In 2022 we learned that a landmark paper on a protein called Aβ*56, cited nearly 2,500 times and the basis for over a billion dollars in annual Alzheimer’s research funding, contained fabricated images.2 Other labs had quietly failed to find the protein for years, but those null results went unpublished. The field had spent sixteen years chasing a lead that was never real.

Marc Tessier-Lavigne, president of Stanford, resigned after investigations revealed problems in papers he’d authored, flagged by Elisabeth Bik, a microbiologist who has personally scanned over 20,000 papers for image manipulation.3

And in May 2025, Harvard revoked tenure for the first time in 80 years.4 The professor was Francesca Gino, who had built her career studying honesty and ethics, and was fired after forensic analysis revealed she had allegedly fabricated data across multiple studies. The irony of a dishonesty researcher faking her honesty research would be funny if it weren’t so devastating.

The Scale of the Problem

The numbers are hard to fully absorb. The landmark 2015 Reproducibility Project tested 100 psychology studies from top journals and found that only 36 percent successfully replicated.5 An eight-year project attempting to verify high-impact cancer biology found that less than half of experimental effects could be reproduced, and the successful replications showed effects 85 percent smaller than originally claimed.6 Over 10,000 papers were retracted in 2023 alone, a new record, double the previous year.7

One widely-cited estimate puts the cost of irreproducible preclinical research in the United States at tens of billions of dollars annually.8 The authors acknowledge significant uncertainty in that figure, but even the lower bound suggests an enormous waste of resources. And that’s just the direct costs. It doesn’t count the years lost chasing false leads, the patients enrolled in trials testing hypotheses that were never true, the graduate students whose careers collapsed when they couldn’t reproduce their mentors’ results.

Something Changed: AI Made a New Approach Possible

Methods to address these problems exist, but they are too tedious to compute manually for all of science. Existing efforts have only scaled to thousands of papers, while fieldwide efforts have been unable to discriminate between relevant and irrelevant p-values.

For example, p-curves can detect when a literature has too many p-values clustering just below .05, a telltale sign of p-hacking.9 Meta-analytic techniques can assess whether effect sizes are consistent across studies. Preregistration checks can verify whether researchers tested what they said they’d test. Statistical forensics can flag impossible numbers.10

Having moved into AI product development as an applied practitioner, I started to see a new possibility: AI can now do the tedious extraction work, pulling hypotheses, sample sizes, test statistics, and p-values from thousands of papers automatically, so these analyses can run at scale.

I started building tools to do exactly this. And I realized the impact it could have.

The Gap in Today’s Tools

There’s a whole ecosystem of scientific AI tools emerging right now. But they all stop short of what’s really needed.

Scientific search engines like Elicit and Consensus are genuinely useful. Elicit searches 138 million papers. Consensus shows you how many studies support or oppose a claim. But as Consensus acknowledges, each claim counts the same regardless if it comes from a meta-analysis of a thousand studies or the study of a single individual.11 They help you find papers. They can’t tell you which ones to trust.

Automated peer review systems are proliferating12, tools that scan papers for methodological issues and suggest improvements. They’re helpful for researchers and they surface potential problems that need to be addressed. But they stop short of actual evaluation or scoring. They won’t tell you that a finding is probably unreliable.

Specialized fraud-detection tools exist for specific problems. ImageTwin and Proofig catch duplicated or manipulated images.13 StatCheck flags calculation errors.14 The GRIM test detects impossible means. The Institute for Replication is building an AI engine to re-execute code and verify computational reproducibility.15

These are all good. But they’re siloed. Nobody is building the integrated system, one that brings all the signals together, reasons about individual papers the way a critical scientist would, looks at the full body of meta-analytic evidence, and produces an overall assessment of how much you should trust a given claim.

That’s what I’m building.

Vision: What a Critical Scientist Looks Like at Scale

Imagine you could ask: what’s the evidence that intervention X actually works? And instead of getting a list of papers, you got an assessment.

Here are 47 studies testing this claim. 12 were preregistered; of those, 8 found the predicted effect. The non-preregistered studies show a suspicious clustering of p-values just below .05. Three independent replications failed to find the effect. The original study was underpowered and has never been directly replicated. Two papers have statistical errors that, when corrected, eliminate significance. Bottom line: weak evidence, high risk of false positive.

This is how a careful, critical scientist thinks about evidence. They don’t just count papers. They weigh methodology, check for red flags, look at replication status, consider the full pattern. We can build AI systems that do this—systems that make this kind of careful evaluation possible for every claim, not just the few that get manual scrutiny.

The output should be meaningfully predictive of two things: whether a finding will replicate, and whether it is in line with the thinking of skeptical experts. Those are the real tests of credibility.

Who Needs This?

Almost everyone, it turns out.

Grantmakers and philanthropists deciding which interventions to fund. Right now they rely on manual literature reviews that can’t possibly keep up.

Policymakers basing policies on research. The growth mindset interventions that schools adopted based on Carol Dweck’s work? A large-scale UK trial found zero statistically significant effects on any academic outcome.16

Journalists trying to report on science accurately. Every week brings new studies with dramatic claims. Which ones should they cover? Which should they be skeptical of?

Government research agencies trying to improve the quality of science they fund. You can’t reform what you can’t measure, and right now there’s no way to monitor whether policy changes—like preregistration requirements—actually shift the distribution of evidence quality across a portfolio.

The general public, increasingly trying to navigate scientific papers themselves. If you’ve ever Googled a health question and tried to read the studies, you know how hard it is to evaluate what you’re reading.

AI labs and companies building on scientific literature. As AI systems increasingly use scientific papers for training and retrieval, they need to know which papers to trust. Garbage in, garbage out, at unprecedented scale.

This is a vital public good. Reliable information about what science actually knows, and doesn’t know, is infrastructure for a functioning society.

What We’ve Built So Far

The first piece already exists: an API at evidence.guide that extracts structured data from papers. Upload a PDF, get back JSON with study details, hypotheses, test statistics, and p-values. I’ve validated it against hundreds of hand-coded papers with 92%+ accuracy on p-value extraction. There are no other extraction APIs currently out there.

The next step is to build out a more comprehensive set of quality signals and a meta-analytic reasoning model that can weigh them appropriately.

How You Can Help

I need help.

Money. We’re a 501(c)(3) nonprofit. Donations fund development, compute, and eventually a small team. Even small amounts help.

Compute. Running sophisticated analyses on millions of papers requires serious compute. The Render Network Foundation has generously provided initial support. If you have access to compute resources and want to support open science infrastructure, I want to talk to you.

Engineering talent. I’m looking for a junior (potentially new grad!) full-stack engineer or data engineer, someone passionate about this problem who can work on a tightly scoped 3-month project with potential for a full-time role. If that’s you, or you know someone, reach out.

Introductions. If you know funders, researchers, or organizations who should be aware of this work, I’d be grateful for connections.

Feedback. If you’re a researcher who would use these tools, I want to hear from you. What signals matter most? What would make this useful for your work?

Why This Matters

I left academia because the incentives were broken. I watched my startup fail because the foundational science wasn’t real. I’ve seen the toll this takes, on researchers who can’t replicate their mentors’ work, on patients enrolled in trials testing fabricated hypotheses, on the public’s trust in science itself.

The replication crisis is mostly a story of broken incentives, not bad actors. Most researchers want to do good work. The system rewards cutting corners. We need infrastructure that makes reliable science visible and unreliable science obvious.

Learn more: dawes.institute | evidence.guide

Support the work: Donate

Get in touch: info@dawes.institute