The good news is that the transition is already underway! The reality, though, is that there is a lot yet that will need to happen to get from here to there. In this piece, I’m going to explain where we are on that transition, tell a detailed story of what that transition could look like, and consider some drawbacks and objections. My goal is to convince you both that this process is in some sense inexorable and that it actually portends something good for science and scientists. There’s obviously a lot of fear and perhaps a mounting backlash to AI among academics, some of which is for justified reasons. But I think we can genuinely get excited about the possibility of real reform of academic publishing. This is important because it not only affects academia, but it affects all of us downstream consumers of the research process.

The present gestures toward the future

To that end, there have been a lot of experiments on the future of publishing, from new journals, to full end to end systems that span research submission, review and dissemination. But the effects of these experiments have been modest.1 To understand why, you have to consider all of the different functions an academic journal fulfills: distribution, curation, certification and ultimately prestige. Alt-journals have largely been able to break the monopoly on distribution, curation, and to a lesser extent, certification. But journals have largely maintained a monopoly on prestige - most academics’ careers and funding come from where they publish. In a world flooded by new publications, the brand of a top journal like Science or Nature is a powerful signal of quality. And unlike citations or other metrics of impact which take time to accumulate, publication in a top journal confers instant credibility. For most of academia, grants and tenure can be determined by just a few such papers.

Of course, one alternative approach is to change how tenure and promotion are conferred to deemphasize brand name journal publications. There are experiments like this underway, notably in the University of Maryland psychology department. This is terrific, but changing the hiring and award policies in each one of hundreds of thousands of academic departments is an uphill battle that will likely take a long time.

Free journals are an obvious lever that do exist and compete with the for-profit journals. They have made some headway, but coverage by field is patchy and the for-profit journals have still maintained most of their brand advantage. It’s possible that the brand value of journals will rapidly erode as AI submissions dilute the quality of journals. This could happen, though it’s likely that journals will do something about this, like augment peer review with AI triage. Or they might try to use AI text detectors (like Pangram) to filter out AI submissions. I doubt that will even partially stem the tide of submissions, though.

Another approach is to change the model of academic publishing. To date, there have been three kinds of approaches to replacing the prestige part of the traditional journal bundle: the overlay journal approach, the conference model, or relying on post-publication metrics.

In the overlay approach (e.g. Discrete Analysis, eLife reviewed preprints) they add peer review on top of public access repositories, unbundling curation and certification from distribution. Yet Annals of Mathematics, Inventiones, JAMS remain hugely prestige-laden for tenure. Overlay journals are a small fraction of math publishing and grow slowly. It’s possible this approach will crowd out for-profit journals over time, but for now the best one can hope for is that these overlay journals can become one alternative.

In the conference model, most prominent in computer science, conference acceptance and best paper awards are used to confer prestige and tenure. This has its own pathologies - deadline-driven half finished work, and a host of tactics to game the submission system, etc. But more importantly, in experimental sciences where it can take years to gather data to produce evidence for a claim, the short timeline of a conference model wouldn’t work. It’s also worth noting that all peer reviewed approaches in general requiring free labor are at risk from AI generated submissions, reviewer burnout etc. An ideal system would find a way to pay for good reviewers!

Finally, in the post-publication metrics approach, metrics are used to sort research and researchers. Examples include altmetrics, which relies on “attention” broadly construed, or the venerable h-index, which relies on citations. By and large academics hate these. Citation metrics, in particular, have become gamed significantly, with citation collusion rings being discovered regularly. H-index, also, of course, still relies on the paper as the unit of scientific currency. Metrics have had some impact on how scientists are judged, but everyone would say they’ve been corrupted. The last thing we need is to reduce scientists to a few incomplete and corruptible metrics.

And yet, I think AI-enabled metrics computed over a structured representation of scientific knowledge will be part of the answer…

A proposed path

Here’s the idea. We use AI to parse scientific research into atomic claims arranged in a nomological network. A nomological network is a way to represent scientific theories - the entities they operate over (the “ontology”) as well as their observable manifestations (the “operationalization”) and the relationships between them. The representation would also include the scope for the claim (e.g. how far it generalizes) and a list of auxiliary assumptions. The entities of this network can be things like “self-control”, “inflation expectations” or “ribosome”. For something like self-control, for example, we might have a number of different operationalizations, which could include a neurological signature or a survey instrument. The relationship between the entities and how we measure them is itself a claim with evidence. There are different kinds of claims one can make relating entities, like “Depression causes sleep disruptions” or “These gene SNPs correlate with educational attainment.”

Each claim also carries a posterior that reflects current evidence. And the system keeps track of provenance - i.e. which studies, how they were designed, and the various checks the evidence has passed (e.g. computational reproducibility). Moreover, as knowledge grows and evolves, the graph should too. Scientists (and AIs) could propose an entity splitting in two, merging (e.g. ego depletion is just fatigue!) proposing a new construct, contesting an operationalization, etc. And ultimately it’s the community of scientists, aided by powerful AI systems, that will govern2 and decide how this system reflects scientific consensus, or the lack thereof.

Producing something like this at scale, let alone keeping it up to date, would have been an enormous and massively expensive task to do in the past. But with current AI systems and the promise of even more capable future AI systems, building something like this is possible3. Humans can and likely will (especially at first) contribute to overseeing the creation of the overall ontology, as well as contributing much needed evaluation to ensure it’s correct. In fact, reading the foundational literature of a scientific field and then carefully making sure it’s well represented in this system is just the kind of task that would be useful for a beginning grad student to do.

Then the next step is to be able to quantify the contributions to science by changes to this graph over time. Once you create such metrics, such a system could attribute incremental improvements to our understanding to the scientists who caused it. Here’s how that could work, over time, to replace the scientific journal as a record of contribution for scientists. Each stage brings in a new set of users: research scientists first, then AI systems and policy makers, then everyone downstream of better-funded and better-targeted science.

Stage one: living layers form alongside journals

Initially, the system starts as a tool which allows research scientists to assess the scientific literature. They upload a set of published papers they locate via a search engine, and then the system extracts the claims, applies forensic audit, runs computational reproducibility where data is available, and outputs a claim-level posterior. The scientist can set a search that automatically updates the analysis when new papers come in. During this time, it’s mostly used by scientists and investigators, who are able to use the evidence aggregation and forensic tools to quickly assess a scientific literature. They are willing to use it despite its flaws, and in doing so, help provide the feedback and data needed to make LLM-based claim extraction highly accurate. They use it mainly to expose poor quality evidence and publish meta-analyses of their own. Some of these become the first living evidence systems. Therefore, at first, the coverage of this living layer is patchy; limitations in AI quality, limitations in funding, and limitations in human bandwidth to evaluate model outputs mean that it may take a few years for this system to gain momentum and coverage. Meanwhile, coordinating efforts, injections of philanthropic funding, and enabling technologies (e.g. turning PubMed into structured data AIs can accurately ingest) make investments in this burgeoning technology and employ underemployed researchers to help expand coverage of the system. During this time, journals continue to certify individual papers.

The turning point is when the review tool passes a threshold where it’s almost entirely accurate. At that point, an investigator makes their first set of major discoveries. A major area of research is found by the AI system to be unreliable or fraudulent, despite dozens of publications in top journals. Everyone takes notice. All of a sudden, the journal brand has been dented - the reliability of research is evaluable outside the traditional journal system. People start to have doubts about the journals. At this point, the system starts to really take off, and interest in scaling it starts to grow. More funding comes in, from governments, philanthropies, and AI companies themselves, who finally realize this is a valuable source of training data for them.

Stage two: certification shifts toward provenance and audit

As these evidence syntheses scale and become authoritative, certification of research quality follows naturally. More studies pass checks, and platforms (like asCollected, which recently launched) provide data provenance verification to certify they were collected legitimately. Increasingly, the entire scientific workflow is instrumented and recorded, so the entire research process is logged, tracked and certified. As the system becomes more authoritative, its mistakes are increasingly rooted out quickly by the scientists themselves, who are incentivized to have their work analyzed correctly. So misunderstandings are smoothed out, and the system’s accuracy increases further.

In addition, organizations of scientists decide to aggregate their own expertise, enabling voting and a way for anyone to see scientists’ consensus on claims4. The system and its metrics continue to evolve as scientists’ opinions change. The system is administered by a non-profit and is charged with making sure the metrics aren’t gamed. Constituted by forensic data scientists and meta-scientists, this organization is charged with measuring and maintaining the way results and contributions are certified. This is meant to resist, for example, review cartels. The audits, and the evidence aggregation mechanisms themselves continue to improve as scientists propose new ways to measure how science is done.

The evidence for atomic claims is certified by surviving forensic audit, computational reproduction where applicable, and integration into the living evidence graph with appropriate weighting. Good studies sway the balance of evidence more than bad studies. A small number of scientists start getting jobs and promotions based on their contributions to the research graph rather than traditional publications. Once that starts happening, other scientists really stop and take notice. Scientists start relying upon these systems more and more as the system gets more comprehensive. The more comprehensive this is, and the more certification it provides, and the more it’s relied upon, the more incentive journals have to participate. They need to include their articles, even if including their papers erodes the articles’ value as scientific artifact. Nonetheless, journals continue to play a role for narrative synthesis, theoretical work, and complex multi-claim research. They stop being the primary certification layer for atomic empirical claims, however.

As the system grows, the evidence layer becomes a direct input into AI training systems. Understanding the most up to date scientific knowledge becomes essential for both humans and AI scientists to quickly devise the best follow-on experiments. Meanwhile, the structuring of policy analysis has a dramatic effect on how government and philanthropic funds are deployed. It becomes very easy to understand the most up to date evidence for different global health interventions, speeding the spread of life-saving changes. Education policy improves with a clearer read on the data.

Stage three: incentives realign and many journals wither

Once there is a trusted self-updating evidence layer which can track and assign credit for scientific progress, it becomes increasingly common for scientists to directly submit their work to the evidence layer, bypassing traditional journal articles. At first, the amount a paper shifts the posterior on a named claim becomes a measurable and attributable way of rewarding scientists. But soon, more methods for measuring the impact of scientists proliferate. Some scientists make conceptual clarifications, while others make methodological improvements that clarify causal inference across dozens of empirical datasets at once. Others collect quality data that comprehensively adjudicates between competing theories5. There are so many different actions that are seen as furthering human and machine understanding. Scientists are rewarded for all of it. As researcher evaluation moves toward more holistic forms, these metrics become key contributors.

At this point, textbooks start being created automatically from these living meta-analyses. Public-facing explainers emerge alongside them and the living meta-analyses now provide the context layer for search engines and for journalists writing news stories. The general public becomes better informed, and it’s easy for anyone to see what the latest research suggests people should do in order to improve their health.

Caveats and complications

There are a number of boundary conditions on this vision, of course — some types of scientific work aren’t amenable to this kind of structured decomposition. As always, there are legitimate fears that we will accidentally disincentivize some key aspect of scientific practice not legible to the machine. But I want to resist the idea that because metrics are imperfect that we shouldn’t create them, or frankly that it’s possible to envision a world without them. We need some basis for selection, whether it’s grants or jobs6 - the overthrow of metrics would only lead us to a world that prioritizes who you know — networking and nepotism7. Moreover, the more quickly scientists work via AI, the more telemetry and data artifacts will be created. Ultimately, that may make the creation of these new metrics inevitable.

Of course it’s possible that a much worse version of this plays out. First and foremost, journals could end up co-opting and bankrolling the creation of these systems, using their massive catalogue of papers. They could steamroll any attempts to release public data, and win fair use cases that allow them to maintain their monopoly on the production of knowledge. It would be unfortunate if the infrastructure of science were privately owned. It should be a public good and ultimately maintained by scientists.

Another risk is that this AI system is opaque, full of subtle flaws that make it unreliable upon close inspection. This is why human collaboration is key. Especially at first, we’ll need the system to have confidence scores, to have humans audit high-stakes nodes, and for all the system’s interpretations of papers to be transparent and auditable. It’s crucial that the system be as open as possible to foster trust. This is another reason why the institution that maintains this system should be a non-profit entity.

It’s also possible that we end up with a system that fails to incentivize researchers properly, and ends up with more fruitless gaming. Goodhart’s law is a reality, but that doesn’t mean that all metrics are bad everywhere always. There are better and worse uses of metrics. Goodhart’s law isn’t solvable, but it can be mitigated through the correct institutional structures.

Two cheers for metrics

To mitigate co-option or gaming, metrics need to be flexible and support human judgment. To that end, I want to propose three principles to guide the development of new metrics for valuing scientific contributions.

Values made explicit, and tunable

Metrics are always a reflection of what we value8. Those values are often implicit: “we value writing papers which get cited a lot.” Goodhart’s law still holds; but if we make those values explicit and open, then at least we have the possibility of creating better, more responsive metrics. There are many different scientific values we could care about: novelty, risk-taking, rigor, methodological improvements. All of these and more can be operationalized and included.

Metrics must continue to evolve and respond

The conservatism of academia is such that the few metrics they have are completely frozen. This is a major problem. Consider an alternative model for how metrics can be responsive to efforts to game them: Google search. The Google search ranking algorithm continues to change as researchers find ways to improve it, but also in response to efforts to game it. We know approximately the kinds of signals that Google uses to rank, but we aren’t 100% sure. The real but bounded transparency balances the need for people to be incentivized to create good content for Google to rank with the need for secrecy so that the algorithm can’t be readily reverse engineered. For instance, some of the features that enable sleuths to detect fraudulent papers might need to be secret for a while so that we can better use them. Eventually, those become public and (unfortunately) bad actors will adapt. As stewards of the scientific enterprise, we’ll have to adapt too. Moreover, Google runs experiments to test changes to its ranking - there is a meaningful control. Decisions which hinge on these metrics, like grant decisions, are well served by some decisions made by an alternative procedure for comparison. This is the model we should use when thinking about how to create metrics around the scientific process.

Metric-informed, not metric-driven decision-making

Finally, metrics become tyrannical once they become full substitutes for human judgment. There are still “broken leg” cases. This is a term Paul Meehl used to describe rare circumstances that a human knows are important but sit outside a predictive model. Imagine an actuarial model is designed to predict if a professor will go to the movies on any given night. The model predicts a 90% probability based on historical data. However, the expert intuitively knows the professor just broke their leg. The expert intuitively adjusts the probability to 0%. The bureaucratic nightmare is one where it’s obvious to any human with a brain and a heart that the rules do not apply, and yet the bureaucrat doesn’t make an exception. To be data-informed means to use the data as a tool and not to be a tool of the data. Of course, Meehl’s point was that humans call for exceptions far more than is warranted, so of course there’s a balance.

Conclusion

Despite reasons for concern and pessimism, there is a real prize for humanity to be won if we steward this carefully. A claim level living evidence layer will earn scientists’ and the public’s trust as it improves. As it grows in scope and quality, the clarity it brings as to the most trustworthy science will start to shift academic culture. If we build it correctly, we can solve a problem that is only growing in importance with the advent of generative AI. Namely, how to use AI to safely expand the research enterprise, ensure high quality scientific work, and credit the best research and researchers. The current journal system is roughly sixty years old in the form we know it. It survived the move from print to web, from subscription to open access. It probably won't survive the move from paper-as-unit to claim-as-unit, because that move dissolves what the journal was actually doing. The state withers, in this case, because something better is doing its job.

(FYI I moved the links to its own section so if these annoy you, you can unsubscribe to just the link posts)

1. Natalie Cargill on Olively, an AI app that rewrites your texts to your partner and decodes theirs back to you. Cargill very persuasively makes the case that AI-mediated communication is helping people avoid the real work of attachment repair. I think the reality is more nuanced -- when the AI has the right information it can do more than erode our coping abilities, and help facilitate real therapeutic insight. The full argument will require more than a blurb. Just! Stop! Communicating! — Natalie Cargill

2. Erik Hoel diagnoses 21st century cultural stagnation as overfitting — algorithmic feeds and hyper-discriminatory measurement converge culture on narrow in-distribution outputs, with the AI em-dash tic as a canonical small-scale instance and “no better marker of culture’s unoriginality than everyone talking about culture’s unoriginality.” Big Erik Hoel fan - his book on the nature of consciousness, The World Behind the World, is a must read. Our Overfitted Century — Erik Hoel

3. Ben Recht: p-values are a regulatory mechanism, not a measurement device. Recht takes this view maybe a bit further than I would, but viewing statistics as a mechanism for technocratic decision making seems largely correct. This view is one reason I’m so fixated on better evaluation of randomized control trials, the most load bearing method we have in policy decision-making. Milton Friedman’s p-values — Ben Recht

At this point it is uncontroversial to say that science needs to stop using the PDF article as the unit of knowledge and currency. The unbearable slowness of scientific publishing, the profit motive and margins. I’m not saying anything new. The PDF also sucks because it’s hard to extract structured information from, which makes it hard to do evidence synthesis. As a result, we do much less evidence synthesis than is needed. And evidence synthesis ultimately undergirds most policy and medical decision making. I can see second by second real time odds for any sporting or newsworthy event, but a school board can’t see the best evidence on whether their 8th graders should be taught algebra. As a society we don’t treat this as an important problem. Again, not controversial.

Not only is the problem well understood, but the solution has already been laid out. What we need is AI-assisted living evidence synthesis - (1) an open knowledge graph of atomic claims (2) claims linked to evidence (3) assessment and synthesis of each piece of evidence (4) continuous updating with new data.

What few realize (yet) is that the technical capacity to build this vision for a significant portion of science already exists. Not only that— scientists and startup teams are already building many of these components. I know this because I’ve been surveying the space and talking to many of the builders. There are some missing pieces: for example evaluations of how well some of the components work. But at this point most of what’s missing is a fully end to end working integration of all of these parts. In the rest of this essay, I’m going to lay out all the parts of a working living evidence layer for science and who is working on them, and propose concrete next steps for building this system.

What’s now possible

The diagram above outlines the components of a living evidence synthesis platform, including some of the teams working on each component1. Scientific PDFs are processed into claims with associated evidence. The evidence is subjected to a forensic audit, methodological evaluation and robustness and reproducibility checks. Finally it’s given a weight in a continuously updating synthesis. What follows is a description and status of each component and a few of the teams working on them.

Document understanding

The first thing you need to be able to do is turn an article into structured data. Mostly that means parsing PDFs. There are often multicolumn layouts that confuse non-specialized PDF to text processing libraries. For scientific papers, there is the added complexity of parsing formulas and tables and figures. This is a really hot area - there are startups offering APIs, and it seems like a new open source package gets posted to Github every few weeks. What follows isn’t exhaustive. A package called GROBID was the state of the art for a while, they didn’t update their package for nearly two years until very recently. In the meantime reducto.ai released an AI powered PDF extraction API, PaddleOCR became popular, IBM released a model called Docling, and both Mistral and Gemini created models and libraries. I also know of at least one other well-funded psychology research group working on a paper parser. By contrast, there are few open evals in this space, with no extensive evals for complex table comprehension in particular. Nonetheless, I'm confident this will be a solved problem soon, given the combination of LLM advances and developer interest.

Hypothesis level extraction

There has been increasing interest in comprehending the extracted text of papers and linking information to evidence for each hypothesis. A lot of work has already been done. Trialstreamer (Marshall et al. 2020) and RobotReviewer LIVE (Marshall et al. 2023) demonstrated automated extraction of trial population, intervention, and outcome at scale on clinical RCTs. PaperQA2 (Skarlinski et al. 2024) and Ai2 ScholarQA (2024) extended this to retrieval-augmented question answering with citation grounding. Elicit, Consensus, and SciSpace operationalized claim-level extraction for end users. OpenEval (Booeshaghi et al. 2026) is the most recent and most ambitious: 1.96 million atomic claims extracted from 16,087 eLife manuscripts using Claude Sonnet 4.5, grouped into ~299,000 results, with LLM evaluations showing 81% agreement with human peer review on a 2,487-paper subset. None of these solutions link claims to test statistics, as you would need to evaluate randomized control trials. This is why I built the evidence.guide API - to extract hypotheses and associated test statistics from behavioral science papers. The best public eval of this kind of extraction I’m aware of comes from the recent SCORE project - they had humans code thousands of psychology papers to extract their claims by hand. It would be extremely helpful to the world if all scientific PDFs were available as structured open data. I’ve been working to make this happen, both directly at Berkeley and through coordination with large entities I can’t yet speak of; as hard as it is to do, I think it’s possible2.

Forensic audit

A lot of work has been done on forensic audit, but some gaps remain. Of course, for biology papers that rely on images for evidence, there are a variety of tools (notably Proofig and ImageTwin) to spot anomalies. These are still well short of what sleuths like Elizabeth Bik can do on her own, but these tools are constant companions among fraud analysts. There’s someone working on auditing Excel files for anomalies, and a number of teams are automating numerical checks like GRIM and SPRITE, including Scrutiny project, the INSPECT-SR team as well as statcheck. The regcheck team is building a way to use AI to compare preregistrations to analyses in papers, to ensure there aren’t significant deviations. Nonetheless, there are many other kinds of anomalies to screen for, both public and less publicly known. And there are no formal evals for anomaly detection that I’m aware of. Still, there’s a lot to draw from in this space and I’m pretty certain we will be able to scan papers for most kinds of obvious anomalies in the near future.

Methodological review

This area has been white hot, though I fear for many of the startups in this space, because this capability may become commoditized. There are at least six different AI peer review companies, including Refine.ink, Reviewer3, ReviewerZero.ai, Q.E.D. Science, Paper Wizard, and Isitcredible. Coarse (a pun on refine) was also recently created as an open source alternative. These systems provide qualitative feedback on the content of papers, spotting methodological weaknesses and mathematical errors. They seem to work pretty well, and many academics report bitterly that they exceed the average quality of typical peer reviewers. But there are few evals here either. What evals exist so far involve using LLM-as-judge (circularity problems abound) or comparing against human reviews of questionable quality. What you’d ideally want is an eval that measures capturing known errors in papers3.

Reproducibility and robustness

Another active area has been using AI agents to automate computational reproducibility4 and robustness5 checks in papers that report numerical results. For more recent papers where data and code are available, AI agents can see whether they can re-run the analyses and produce the numbers reported in the published paper. In addition to a handful of individual academics who have been experimenting using Claude Code for this, the Institute for Replication is a leading group working on building an end to end system. The evals related to this problem are the most mature, with CORE-Bench (Siegel, Kapoor, Narayanan 2024) and PaperBench (OpenAI 2025) available to benchmark agents on this task. There is also work on getting AI agents to test alternative ways of analyzing the data to ensure the results are robust to small analytic design choices.

Synthesis

This is the most underdeveloped area where significant investment is required. Although some automated evidence synthesis systems exist — for example, otto-sr is building an AI agent to write systematic reviews — none of these incorporate the full range of paper level signals to weight evidence appropriately. Nor is there anything like an eval or a gold standard for a good systematic review. Arguably Cochrane reviews are the closest we have to gold standard human systematic reviews, though I’ve heard academics in the know complain about their uneven quality. A key question for a synthesis platform is how to weight anomalies and methodological issues in assessing the quality of a piece of evidence. This is an unsolved problem and one I’m very keen to work on.

Continuous updating

There are many pieces of basic infrastructure available for monitoring for new research and initiating updates. OpenAlex is the current open citation graph. Retraction Watch integrated into Crossref in October 2023. Scite tracks how citations support, contrast, or mention prior claims. The Living Evidence Network demonstrated continuous-update workflows in clinical guidelines. Engineering this is a relatively straightforward task.

When you look over this technical architecture and all the progress being made, it’s hard not to be optimistic that a living guide to scientific evidence will be built.

The stakeholders are ready

The social infrastructure for this is starting to coalesce — it’s not just a pie in the sky academic exercise to imagine this coming into existence. Institutions like the Center for Open Science, the Institute for Replication, the INSPECT-SR, the Living Evidence Network and more are all working on scaling work to improve research quality.

Funders are also aligned. The Sloan Foundation has funded living evidence work through COS. Coefficient Giving supports the Institute for Replication and COS. The Astera Institute and the Institute for Progress has shown interest in this space. NIH has established an Office for Replication and Reproducibility. Although there are (very unfortunately) serious headwinds in science funding generally, there is an active group of funders interested in metascience.

A brief word about what I’ve been doing at RDI. First, as a Visiting Scholar at Berkeley I’ve been actively figuring out how a non-profit and a public university can conduct and make public the results of large-scale academic article data mining. With some of the money I raised from donors, I commissioned a legal analysis of recent case law and publisher text data mining (TDM) agreements in order to understand whether a massive open data mining of academic articles is possible (with caveats, it is). I’ve also been working to bring together stakeholders in this space, and identify gaps. I’ve also been doing some software development in this space, with more to come.

A pilot proposal

The assumption undergirding all of this is that an AI, given all this information, would make the right judgment about a scientific claim with lots of conflicting evidence, weighing all the factors appropriately. That’s the hypothesis we need to test.

Randomized control trial research is the best place to focus on first. RCTs are used to make many of the important decisions in society - from medical trials to public policy changes. And they use a relatively uniform set of inferential statistics with lots of known and available diagnostics. Behavioral science experiments, within the broader realm of RCTs, should be first used as a testbed whose results can be generalized. Because behavioral science is at the vanguard of open science practices, replications abound (there are thousands of them) to serve as ground truth training data.

Key Hypothesis

Therefore the pilot would test, in behavioral RCTs that have been replicated, whether the quality of evidence for a claim can be used to accurately predict whether that claim will replicate.

Secondary Hypotheses

1. Compared to claims that replicated, non-replicated claims demonstrate a greater share of forensic anomalies in their source literatures.

2. Hypothesis level claim and statistic extraction is accurate enough to scale living evidence without onerous human review costs.

3. Replication prediction is more accurate than prediction markets6 or journal prestige.

If all the different quality signals we gather do accurately predict which studies will replicate, then we can use that model to score evidence to power the living evidence layer.

Why this is informative regardless of outcome

If the pilot succeeds, the architecture extends to medical RCTs (where Living Evidence already operates and integration is mostly about claim representation), then to slices of basic biology with stable replication structure. If it fails, the field learns which quality signals are load-bearing and which ones metascience has oversold. Either result is a contribution to knowing what the literature supports.

Implications for funders

Because this burgeoning ecosystem of builders already exists, a well-informed philanthropic or government funder could play a crucial catalyzing role in bringing this future about. They could play at least three roles: creating open structured datasets, publishing open benchmarks and incentivizing the solving of key technical challenges.

First, open archives of papers that the government maintains, like PubMed, could be turned into structured data amenable to large scale metascience and claim aggregation. I know the US government already has an interest in doing this, though some key open questions remain unanswered. How do you determine the best models and systems for accurately extracting information from papers? How can you establish a robust way to allow researchers to flag errors and correct them? And finally, how do you create a legal regime in working with publishers to maximize the scope of available papers? For the latter, a university or a private philanthropy may be better positioned to make structured data publicly available under journal subscription terms or fair use.

Philanthropists or government funders could also coordinate to create or commission benchmarks that evaluate whether important problems have been solved. For example, an open benchmark for claim extraction from a range of different scientific article types would be extremely helpful. Ensuring the underlying data are accurate is vital for creating these evaluations. I’ve discussed opportunistically using various human-created datasets for this purpose, but a consistent problem is that errors in human data make it difficult for them to serve as a gold standard.

Finally, with structured data and benchmarks available, the government or private philanthropy could use them to incentivize groups to develop machine learning systems that meet these benchmarks. Prizes are one potentially valuable tool for this. For example, you could establish a prize for a system that accurately updates a living meta-analysis for a small set of claims. Prizes are particularly useful as signals of problem importance, and can help create vibrant ecosystems of public and private research—see, for example the role the government played in kickstarting the current work on nuclear fusion.

Conclusion

The drawbacks of the current scientific publishing system are known. Scientists agree, metascientists agrees, philanthropists agree: the published PDF plus citation graph isn’t the right substrate for maintaining a representation of the evidence base in science. The pieces needed to build the alternative either already exist or are rapidly taking shape. The community is forming around exactly this problem, with concrete partnerships and shared infrastructure. A pilot should start on behavioral science RCTs because that's the slice of empirical science most amenable to legibility, where replication ground truth is richest, and where the failure modes are best documented. What's been missing is the galvanizing mission to assemble these pieces into something that works. That's what I'm proposing to build.