Put simply, what I’m trying to midwife into existence is this: an open evidence graph, which is a shared updating public record that connects scientific claims to the evidence that supports or challenges them. This graph is meant to preserve how a result was obtained, the checks it passed and the limits of what it demonstrates. This graph would make scientific evidence transparent - it can aid humans or AIs in assessing what we think is true and why and ultimately guide how we should act in light of that evidence4.

Developing this kind of structured representation at the scale I’m proposing would be prohibitively labor-intensive without recent advances in AI. And still, the most challenging technical problem in developing this evidence graph by far is verification — how do you know that the right theory and evidence are being extracted from papers, that the knowledge is accurately represented and aggregated, and ultimately that the evaluation correctly weights the evidence? At every level we need datasets and evaluation - my recent attempt to demonstrate an eval for error finding in papers is just a tiny first step in that direction.

In what follows I want to tell you a little bit more about why I’m doing this — what futures I’m trying to forestall, the looming question of capabilities enhancement in the age of unaligned AI, and why I think this graph needs building despite the potential risks.

Avoiding feudal science

I don’t want to live in a world in which access to the most useful scientific understanding depends on powerful patrons who control the data, models, compute, and interfaces needed to produce and use it. That is the default world we are headed to. That is a world of feudal science - to gain the best scientific understanding of the world, we will have to pay or (as a scientist) work for a large model provider. A scientist will be like a serf, pledging fealty to a model overlord to get a patch of science to work on.

Some of this isn’t new. Pharmaceutical companies often fail to disclose negative trial results and keep much of their drug-development knowledge proprietary. What’s changed is the scale. AI companies (both foundation model and specialized AI-scientist companies) are already building up their own internal understanding of scientific disciplines. There is a real danger of this spreading across science, particularly as public science funding continues to be withdrawn and the best academics are increasingly lured away to industry. I’m not blaming individual academics who’ve left for greener pastures, but it’s undeniable that academia is already experiencing a brain drain.

Not only is there a danger of the sharp increase in hidden and paywalled knowledge, there is the risk of bias — as funding for research shifts toward private actors, independent research that questions or undermines those actors’ narratives will become harder to sustain. This is Henry Farrell’s concern about social science research on the impact of AI increasingly being conducted by AI companies themselves. Creating and maintaining a true public shared knowledge commons is thus essential to fight the risks of feudal science. It won’t replace independent funding or access to compute and proprietary experiments, but it would give us a shared basis for assessing claims without relying entirely on a model provider.

Preserving cognitive capacity

Another risk I’m increasingly worried about is the loss of human cognitive capacities in the face of both the undermining of training and the offloading of judgment onto AI. As an expert, I find AI incredibly useful in extending my span, doing by myself what previously would have taken an entire team of knowledge workers to accomplish. As a result of this, however, I’m passing less of my knowledge on to more junior employees. And I’m hardly the only one to both notice and contribute to this problem. I have had countless conversations with engineers about the problem of how future senior engineers are going to get made. I haven’t heard a convincing answer to this. This risk is also accelerated by the brain drain from academia. The more our best minds migrate away from the universities, the fewer researchers are available to mentor the next generation. We are eating our seed corn.

Not only are we facing a training crisis, we are facing a crisis of dependence amongst particularly younger users of these tools. Faced with the pressures of advancement, younger students and workers are increasingly using AI as a crutch without having developed the skills from doing things by hand. This, when combined with the lack of opportunities for mentorship, risks eroding human cognitive capability. This would be bad in a world of unbounded growth in AI intelligence - risking an intellectual monoculture, for example. But in a world where there are clear blind spots in what AI can do, this risk is even greater.

I’ve written about thinkers who question whether current AI has the situated understanding needed for reliable judgment. On this view, AI’s lack of embodiment may leave limitations that the existing paradigm can’t resolve. I have some faith that eventually these difficulties will be surmounted. But how long will that take? Imagine a world where we outsource our judgment to an AI that lacks the situated understanding needed to truly replace that judgment (i.e. our current world). If that limitation persists and our cognitive capacities erode, then we will not experience a successful handover to AI. Growing output and apparent competence would conceal a decline in our ability to recognize and correct mistakes5. A related risk arises in institutional decision-making. In another wonderful essay by Henry Farrell about “robot solutionism”, automated complaints and appeals prompt institutions to automate their responses, creating an escalating bureaucratic arms race. Even individually effective tools can produce a collectively harmful system. Cognitive outsourcing and model inscrutability could make those dynamics harder to recognize and correct. I’m very concerned about this.

Doing research currently requires the researcher to learn something, if not from the result, then from the process of doing it. Doing things the old-fashioned way produced all the expertise in the world. We need to preserve this process for the next generation to ensure we aren’t at the peak of human cognitive skills. I believe that creating a public knowledge graph could be a crucial tool in avoiding this future. Junior researchers could trace claims back to the experiments behind them, check the graph’s assessments, and defend proposed corrections to experienced researchers. Maintaining the graph could become part of how we train scientists, provided institutions make time for that work and reward it. The graph itself won’t solve the problem of situated understanding, but it would give people a shared record against which to exercise and develop their judgment.

AI capability risk

Finally, the risk everyone is currently writing and panicking about is the risk of misaligned super-powerful AI. The above two risks arguably would be dwarfed by this one, with all of us turned into grey goo or zoo animals. If AI truly advances science far beyond the ability of any humans to understand or reason about scientific findings, then not only will science cease to be legible to us, we’ll be putting ourselves at risk from a misaligned AI. In this world, the creation of an open evidence graph could be actually harmful if it significantly aids AI in the process of scientific discovery and understanding. Therefore this might be a reason we shouldn’t be building what I’m working on.

Imagine the open evidence graph in the hands of a superintelligent AI. Obviously it (or it plus a malicious human actor) would be even more able to wreak havoc when considering biosafety alone. But it could also potentially harm us through better social science (don’t laugh!). Supercharged social science also could be used to manipulate the structure of society or persuade us in harmful ways. A polished, apparently comprehensive evidence system might make people more willing to defer even when its conclusions are wrong and meant to mislead.

Now I’ll admit that although this is something I’ve thought about over the last year, I have mostly pushed these concerns aside. This is partly because I’ve thought the AI companies and the nonprofit safety ecosystem are full of incredibly thoughtful and intelligent people who are more than able to resolve AI risk. Recent events like the Hugging Face incident have called that into question for me. I’ve also sidelined these concerns in my mind because I’m still unsure what to think about current AI systems6 — are they permanently hamstrung by lack of being-in-the-world? Are they simply a normal technology? Or are we birthing superintelligence, as many employees and leaders of these AI companies would have us believe? The answer to this question matters a lot for which risk I’ve outlined is most severe and I genuinely don’t know the answer.

We must understand, we will understand

If we don’t build an open graph, I fear feudal science and the erosion of our cognitive commons will only accelerate. But if we do build it, we might risk accelerating dangerous AI capabilities. Somehow we have to balance these risks in deciding what to do. As you might guess, I still land on the side of building an open evidence graph. Private actors have strong incentives to build their own internal representations of scientific knowledge. That doesn’t make the additional danger from public release negligible. I’m making a judgment under uncertainty: I think the benefits of giving scientists and the public an independent record they can inspect and challenge justify building it, while evaluating what is safe to release. The other dangers I’ve outlined would persist under many possible AI futures.

There are a number of factors that could change my mind. One is evidence that releasing particular parts of the graph would materially increase dangerous capabilities.7 Another is whether a partly AI-generated open evidence graph increases unwarranted trust in AI and erodes human intellectual independence or the willingness of the scientific community to correct errors. Incorporating human expertise into the process of building and validating these systems is crucial. It’s not all-or-nothing. For example, I’d postpone releasing a part of the graph if testing showed that it substantially helped an AI carry out a dangerous biological task. My goal in designing these systems transparently is to preserve our capacity for validation. A big focus of the conference is precisely on how we can scale our validation.

Finally, in convening an amazing cross-disciplinary group in Montreal, I’m hoping that we can address these problems in a unified manner. I’m hopeful that we can create a better version of science in the age of AI, one in which the scope of human understanding of the universe, not just machine understanding, is significantly improved. We must understand, we will understand.

Claude and I planted 100 known errors into 10 open-access psychology papers and then ran them through frontier models and two commercial AI review tools. In brief:

• The best single system caught 71 of 100 errors, while the worst caught 30.

• Pooling every system’s output caught 93 of 100. Models are only partly correlated in the errors they find, making ensembling a big lever for finding issues in papers. Check your papers against multiple models!

• Seven errors could not be caught by any system. All were omissions — information deleted from a paper rather than mistakes inserted into it.

• Refine.ink contributes more unique catches than any other single system, though it’s expensive.

• I didn’t measure false positives and I don’t know how this error distribution compares to the distribution of errors in real papers.

• I’ve made the papers, errors, model outputs, and the full experiment log public. I hope people can build on this work to create a comprehensive eval benchmark across disciplines.

Introduction

There has been a proliferation of AI tools that purport to do peer review. This is coming at the exact moment when the number of papers is also starting to grow due to AI assistance. All this new knowledge is about to be created, but humans lack the collective attentional capacity to review it all! We are on the precipice of a revolution in collective knowledge infrastructure, where the next step is going to surpass the span of the human mind itself. But we aren’t yet able to fully turn ourselves over to this new technological infrastructure — after all, how can we trust it? How can we know that what it produces is right, across a given field of knowledge, as the frontier of AI continues to move (and is extremely jagged)? This problem is most urgent in the sciences. I think most scientists following these tools closely would agree that AI peer review, the ability of AI to evaluate a given piece of scientific research, is beginning to get quite good1. But how good? And how do we measure when it’s time to hand over the reins?

I was recently talking about this topic with a friend who is a medical school professor — and he made the point that the new science isn’t going to be legible. The best human minds in medicine have an increasingly narrow scope of in-depth knowledge of a specific subject area. Science will soon outpace the ability of any individual human mind, and the time it has available, to comprehend it. We will likely need a shared, continuously updated record of what is actually known in a field — a living context layer for science — if we as humans have any hope of keeping up. As an example, the concept of diabetes we have right now is merely a shorthand2 for a much more complex phenomenon — it might be better represented by a 90-dimensional vector spanning dozens of biomarkers instead of a single threshold like A1C. Correspondingly, the standard of care would stop being a threshold rule a clinician can hold in their head and become a function computed over that whole representation. To this point, a lot of disagreement on science funding has hinged on the degree to which we allow science to be legible to democratic review or not.

Creating a benchmark

To that end, I was brainstorming with Claude on developing good evaluations of peer review software. Most systems work by ensembling, having the LLMs judge each other’s output. That, as we’ll see, is a good strategy, since different LLMs are surprisingly uncorrelated as far as the errors they can catch. Nonetheless, this isn’t satisfying since we don’t know whether even pooled models miss some categories of errors. Using human peer review as the gold standard, as any professor will tell you, is also unsatisfying, since human peer review varies widely in quality. Of Claude’s suggestions, one stood out — inserting errors into papers and seeing whether a given AI system can spot them. Although this doesn’t measure false positives (identifying errors that aren’t really errors), it at least provides a grounded baseline for measuring true positives and false negatives.

Convinced by the viability of this approach, Claude and I set out3 to modify 10 existing open-access psychology papers4, inserting errors into them both by deleting key clauses and altering others. We created a taxonomy of errors in experimental psychology papers, drawn from existing taxonomies like the INSPECT-SR checklist. We ended up with a 62-category taxonomy of types of errors, ranging from statistical errors, construct validity issues, causal inference, generalizability, etc. Then we created 100 errors, 10 per paper, and inserted them. The first iteration was a bust — the errors were too basic and the benchmark was immediately saturated when we ran the papers through foundation models. So it took a few iterations to get this right. To get high-quality errors, we needed to delete text from papers, not just insert mistakes. For example, in one paper we removed a key statistical transformation that meta-analysis requires. In another paper, we deleted a sentence that explained how the key dependent measure was calculated, making it difficult to evaluate the validity of the study’s claims. So the errors in some cases became quite subtle. A careful (human or AI) reviewer should notice that there’s some important missing information.

Evaluation results

Having generated the error-filled papers, we then ran the papers through various versions of Claude, ChatGPT, and Gemini5 as well as Reviewer3 and Refine.ink, two of the more well-known paid AI paper evaluation systems. We used an LLM-as-judge to examine each of the errors each system spotted and checked to see whether they corresponded precisely to any of the planted errors6. For the foundation models, we also reran the papers three times to see the test-retest reliability, and whether running the same model multiple times could spot additional errors (they didn’t, although we could have fiddled with temperature settings and looked at this more deeply). Here are the key conclusions:

• The best individual model (GPT-5.5) only caught 71 of the 100 errors, while the worst, Reviewer3, caught 30. Since there have been a number of model upgrades (and who knows how much Refine and Reviewer3 have upgraded in the 2 months since we ran them)7, I would focus less on the specific eval hierarchy and instead consider the overall pattern of results. The sample size of errors is also small enough that a gap of a few errors isn’t meaningful, so I wouldn’t read too much into the ordering of systems that scored close together. Also note that the “findings” column in the above table indicates the number of review comments overall. Since the baseline papers themselves have issues that we haven’t assessed, we can’t say whether the comments are correct or not. It’s worth noting the trend, however, that more successful systems surfaced more issues in general.

• One of the most notable findings was alluded to at the beginning of the post: ensembling improves error catching dramatically — i.e. if you pool errors across the foundation models you can catch 91 of the 100 errors! You can also see that Refine.ink adds the largest contribution to the ensemble over and above the best single model.

• Refine.ink (and Reviewer3 to a lesser extent) do add unique error checking capability above and beyond what the foundation models provide. Refine is very expensive on a per-error basis ($8.77 for each of the errors caught, against about four cents for the frontier models), but does definitely seem to add unique value. It’s a leader in detecting issues around methodological design, statistical errors, and internal consistency, but among the weakest on causal inference, reporting completeness, and generalizability. Again, this may have improved in recent months. It does suggest that with some better evals and added compute to address these gaps, Refine.ink likely would be better than the foundation models.

• All the systems struggle with the omission-based errors. There was a small set of 7 errors that no system was able to detect8. For example, in one of the papers we dropped the open science disclosures around pre-registration and that all measures, manipulations, and exclusions were included in the paper, making it possible that the paper engaged in p-hacking. No system proactively called that out. A careful reviewer would certainly ask whether there are any unreported measures or ask the authors to disclose it.

Caveats, Limitations, Next Steps

One caveat I have to admit is that my own verification of the results could have been more thorough. I spot-checked the errors, asked Claude to explain issues I didn’t understand, iterated several times based on the results, and rechecked them. I did not, however, read all 10 papers in painstaking detail and check all 100 errors myself. My goal for this was to demonstrate the method in sufficient detail that a larger team could build upon it. Verification continues to be the bottleneck both in general and for me specifically! Even if a few of the inserted errors are wrong, directionally the results are likely to be correct. But I can’t help but feel a little bit uneasy. To that end, I am making the repo public for anyone to download and examine. The repo includes a detailed experiment log that explains every iteration on the benchmark and the evaluation runs, the original and modified papers, the AI-generated review comments, and a writeup that I iterated with Claude on until it made sense to me. Note that by publishing this benchmark I’m killing its utility — it should be straightforward to find 10 new papers and do the same exercise. I think there’s more value in demonstrating how we did it than in keeping this specific benchmark secret.

It’s important to note that this was only for empirical psychology papers — so it’s far from obvious this would generalize to other academic subfields. Interestingly, Chenhao Tan’s research group at UChicago recently used the same methodology of inserting errors into CS papers and found very similar results (roughly 70% error recovery by the best foundation model). My hope is to find collaborators interested in expanding this eval for more areas of science, create high-quality, fully vetted errors to insert, and then create a public benchmark. If we are going to rely on these systems to screen papers, we absolutely need public evaluation of them.

Another limitation is that we don’t know anything about false positives — errors that the systems flag that aren’t real errors. We know false positives are an issue. I’ve run many papers through foundation models and pushed back on illogical or fake errors that they flag. It’s important that we can now measure recall, but we don’t know precision. To measure precision requires something closer to a human gold standard where we have experts manually verify the validity of each flagged issue. This information would be very valuable to collect.

Finally, the most important limitation we have is that we don’t know how well these errors generalize to the kinds of errors found naturally in empirical papers. This mismatch could come in two flavors. First, it’s possible the errors we’ve created aren’t like errors really found in papers. Second, we don’t know what the error distribution looks like in actual empirical papers. We uniformly sampled errors across category, but it may be that certain types of errors are more common than others. It’s possible that the kinds of errors current AI systems are bad at catching are the most important and common errors, while the obvious ones are less common or less important. Ideally we would weight the results of the benchmark based on the frequency of errors found in a random sample of empirical papers. This is challenging and requires some reliable expert human coding to get. One obvious approach is to use our best peer review system (or the union of all of them) to generate errors, categorize them, and use the inverse of the false negative rate per error category as a multiplier. For example, if we only catch 10% of some error type on average, assume that there are 10x more errors of that type and use that to estimate the empirical distribution. This breaks down when you have error types no system can catch, so unfortunately this probably won’t work just yet.

Conclusion

AI peer review is here whether we like it or not. AI-generated science is also here and will increasingly be a larger and larger portion of new scientific knowledge. That knowledge will quickly outstrip the ability of humans to evaluate it. Amidst that new knowledge (especially at first), there are going to be a considerable number of mistakes. We need to start thinking very carefully about how to surmount the evaluation / human expert attention bottleneck. Training the next generation of scientists with the requisite domain expertise, critical thinking, and AI-native thinking is essential if we have any hope of avoiding being drowned in all the new information or preventing it from being riddled with errors and mistakes.

Finally, it’s important to consider that all of these AI peer review systems are missing at present a real living context layer that allows them to evaluate a given paper in light of everything else that’s known about the topic9. To some degree that information is encoded in the weights of foundation models, but these models are trained on all papers and don’t yet have the ability to weight the balance of evidence the way an expert would. Even if current systems become essentially perfect at catching errors, we still require a public record of the current best understanding of every scientific hypothesis that can be used to evaluate and contextualize new findings. To really close the gap with expert human reviewers, we need to build that too.

The good news is that the transition is already underway! The reality, though, is that there is a lot yet that will need to happen to get from here to there. In this piece, I’m going to explain where we are on that transition, tell a detailed story of what that transition could look like, and consider some drawbacks and objections. My goal is to convince you both that this process is in some sense inexorable and that it actually portends something good for science and scientists. There’s obviously a lot of fear and perhaps a mounting backlash to AI among academics, some of which is for justified reasons. But I think we can genuinely get excited about the possibility of real reform of academic publishing. This is important because it not only affects academia, but it affects all of us downstream consumers of the research process.

The present gestures toward the future

To that end, there have been a lot of experiments on the future of publishing, from new journals, to full end to end systems that span research submission, review and dissemination. But the effects of these experiments have been modest.1 To understand why, you have to consider all of the different functions an academic journal fulfills: distribution, curation, certification and ultimately prestige. Alt-journals have largely been able to break the monopoly on distribution, curation, and to a lesser extent, certification. But journals have largely maintained a monopoly on prestige - most academics’ careers and funding come from where they publish. In a world flooded by new publications, the brand of a top journal like Science or Nature is a powerful signal of quality. And unlike citations or other metrics of impact which take time to accumulate, publication in a top journal confers instant credibility. For most of academia, grants and tenure can be determined by just a few such papers.

Of course, one alternative approach is to change how tenure and promotion are conferred to deemphasize brand name journal publications. There are experiments like this underway, notably in the University of Maryland psychology department. This is terrific, but changing the hiring and award policies in each one of hundreds of thousands of academic departments is an uphill battle that will likely take a long time.

Free journals are an obvious lever that do exist and compete with the for-profit journals. They have made some headway, but coverage by field is patchy and the for-profit journals have still maintained most of their brand advantage. It’s possible that the brand value of journals will rapidly erode as AI submissions dilute the quality of journals. This could happen, though it’s likely that journals will do something about this, like augment peer review with AI triage. Or they might try to use AI text detectors (like Pangram) to filter out AI submissions. I doubt that will even partially stem the tide of submissions, though.

Another approach is to change the model of academic publishing. To date, there have been three kinds of approaches to replacing the prestige part of the traditional journal bundle: the overlay journal approach, the conference model, or relying on post-publication metrics.

In the overlay approach (e.g. Discrete Analysis, eLife reviewed preprints) they add peer review on top of public access repositories, unbundling curation and certification from distribution. Yet Annals of Mathematics, Inventiones, JAMS remain hugely prestige-laden for tenure. Overlay journals are a small fraction of math publishing and grow slowly. It’s possible this approach will crowd out for-profit journals over time, but for now the best one can hope for is that these overlay journals can become one alternative.

In the conference model, most prominent in computer science, conference acceptance and best paper awards are used to confer prestige and tenure. This has its own pathologies - deadline-driven half finished work, and a host of tactics to game the submission system, etc. But more importantly, in experimental sciences where it can take years to gather data to produce evidence for a claim, the short timeline of a conference model wouldn’t work. It’s also worth noting that all peer reviewed approaches in general requiring free labor are at risk from AI generated submissions, reviewer burnout etc. An ideal system would find a way to pay for good reviewers!

Finally, in the post-publication metrics approach, metrics are used to sort research and researchers. Examples include altmetrics, which relies on “attention” broadly construed, or the venerable h-index, which relies on citations. By and large academics hate these. Citation metrics, in particular, have become gamed significantly, with citation collusion rings being discovered regularly. H-index, also, of course, still relies on the paper as the unit of scientific currency. Metrics have had some impact on how scientists are judged, but everyone would say they’ve been corrupted. The last thing we need is to reduce scientists to a few incomplete and corruptible metrics.

And yet, I think AI-enabled metrics computed over a structured representation of scientific knowledge will be part of the answer…

A proposed path

Here’s the idea. We use AI to parse scientific research into atomic claims arranged in a nomological network. A nomological network is a way to represent scientific theories - the entities they operate over (the “ontology”) as well as their observable manifestations (the “operationalization”) and the relationships between them. The representation would also include the scope for the claim (e.g. how far it generalizes) and a list of auxiliary assumptions. The entities of this network can be things like “self-control”, “inflation expectations” or “ribosome”. For something like self-control, for example, we might have a number of different operationalizations, which could include a neurological signature or a survey instrument. The relationship between the entities and how we measure them is itself a claim with evidence. There are different kinds of claims one can make relating entities, like “Depression causes sleep disruptions” or “These gene SNPs correlate with educational attainment.”

Each claim also carries a posterior that reflects current evidence. And the system keeps track of provenance - i.e. which studies, how they were designed, and the various checks the evidence has passed (e.g. computational reproducibility). Moreover, as knowledge grows and evolves, the graph should too. Scientists (and AIs) could propose an entity splitting in two, merging (e.g. ego depletion is just fatigue!) proposing a new construct, contesting an operationalization, etc. And ultimately it’s the community of scientists, aided by powerful AI systems, that will govern2 and decide how this system reflects scientific consensus, or the lack thereof.

Producing something like this at scale, let alone keeping it up to date, would have been an enormous and massively expensive task to do in the past. But with current AI systems and the promise of even more capable future AI systems, building something like this is possible3. Humans can and likely will (especially at first) contribute to overseeing the creation of the overall ontology, as well as contributing much needed evaluation to ensure it’s correct. In fact, reading the foundational literature of a scientific field and then carefully making sure it’s well represented in this system is just the kind of task that would be useful for a beginning grad student to do.

Then the next step is to be able to quantify the contributions to science by changes to this graph over time. Once you create such metrics, such a system could attribute incremental improvements to our understanding to the scientists who caused it. Here’s how that could work, over time, to replace the scientific journal as a record of contribution for scientists. Each stage brings in a new set of users: research scientists first, then AI systems and policy makers, then everyone downstream of better-funded and better-targeted science.

Stage one: living layers form alongside journals

Initially, the system starts as a tool which allows research scientists to assess the scientific literature. They upload a set of published papers they locate via a search engine, and then the system extracts the claims, applies forensic audit, runs computational reproducibility where data is available, and outputs a claim-level posterior. The scientist can set a search that automatically updates the analysis when new papers come in. During this time, it’s mostly used by scientists and investigators, who are able to use the evidence aggregation and forensic tools to quickly assess a scientific literature. They are willing to use it despite its flaws, and in doing so, help provide the feedback and data needed to make LLM-based claim extraction highly accurate. They use it mainly to expose poor quality evidence and publish meta-analyses of their own. Some of these become the first living evidence systems. Therefore, at first, the coverage of this living layer is patchy; limitations in AI quality, limitations in funding, and limitations in human bandwidth to evaluate model outputs mean that it may take a few years for this system to gain momentum and coverage. Meanwhile, coordinating efforts, injections of philanthropic funding, and enabling technologies (e.g. turning PubMed into structured data AIs can accurately ingest) make investments in this burgeoning technology and employ underemployed researchers to help expand coverage of the system. During this time, journals continue to certify individual papers.

The turning point is when the review tool passes a threshold where it’s almost entirely accurate. At that point, an investigator makes their first set of major discoveries. A major area of research is found by the AI system to be unreliable or fraudulent, despite dozens of publications in top journals. Everyone takes notice. All of a sudden, the journal brand has been dented - the reliability of research is evaluable outside the traditional journal system. People start to have doubts about the journals. At this point, the system starts to really take off, and interest in scaling it starts to grow. More funding comes in, from governments, philanthropies, and AI companies themselves, who finally realize this is a valuable source of training data for them.

Stage two: certification shifts toward provenance and audit

As these evidence syntheses scale and become authoritative, certification of research quality follows naturally. More studies pass checks, and platforms (like asCollected, which recently launched) provide data provenance verification to certify they were collected legitimately. Increasingly, the entire scientific workflow is instrumented and recorded, so the entire research process is logged, tracked and certified. As the system becomes more authoritative, its mistakes are increasingly rooted out quickly by the scientists themselves, who are incentivized to have their work analyzed correctly. So misunderstandings are smoothed out, and the system’s accuracy increases further.

In addition, organizations of scientists decide to aggregate their own expertise, enabling voting and a way for anyone to see scientists’ consensus on claims4. The system and its metrics continue to evolve as scientists’ opinions change. The system is administered by a non-profit and is charged with making sure the metrics aren’t gamed. Constituted by forensic data scientists and meta-scientists, this organization is charged with measuring and maintaining the way results and contributions are certified. This is meant to resist, for example, review cartels. The audits, and the evidence aggregation mechanisms themselves continue to improve as scientists propose new ways to measure how science is done.

The evidence for atomic claims is certified by surviving forensic audit, computational reproduction where applicable, and integration into the living evidence graph with appropriate weighting. Good studies sway the balance of evidence more than bad studies. A small number of scientists start getting jobs and promotions based on their contributions to the research graph rather than traditional publications. Once that starts happening, other scientists really stop and take notice. Scientists start relying upon these systems more and more as the system gets more comprehensive. The more comprehensive this is, and the more certification it provides, and the more it’s relied upon, the more incentive journals have to participate. They need to include their articles, even if including their papers erodes the articles’ value as scientific artifact. Nonetheless, journals continue to play a role for narrative synthesis, theoretical work, and complex multi-claim research. They stop being the primary certification layer for atomic empirical claims, however.

As the system grows, the evidence layer becomes a direct input into AI training systems. Understanding the most up to date scientific knowledge becomes essential for both humans and AI scientists to quickly devise the best follow-on experiments. Meanwhile, the structuring of policy analysis has a dramatic effect on how government and philanthropic funds are deployed. It becomes very easy to understand the most up to date evidence for different global health interventions, speeding the spread of life-saving changes. Education policy improves with a clearer read on the data.

Stage three: incentives realign and many journals wither

Once there is a trusted self-updating evidence layer which can track and assign credit for scientific progress, it becomes increasingly common for scientists to directly submit their work to the evidence layer, bypassing traditional journal articles. At first, the amount a paper shifts the posterior on a named claim becomes a measurable and attributable way of rewarding scientists. But soon, more methods for measuring the impact of scientists proliferate. Some scientists make conceptual clarifications, while others make methodological improvements that clarify causal inference across dozens of empirical datasets at once. Others collect quality data that comprehensively adjudicates between competing theories5. There are so many different actions that are seen as furthering human and machine understanding. Scientists are rewarded for all of it. As researcher evaluation moves toward more holistic forms, these metrics become key contributors.

At this point, textbooks start being created automatically from these living meta-analyses. Public-facing explainers emerge alongside them and the living meta-analyses now provide the context layer for search engines and for journalists writing news stories. The general public becomes better informed, and it’s easy for anyone to see what the latest research suggests people should do in order to improve their health.

Caveats and complications

There are a number of boundary conditions on this vision, of course — some types of scientific work aren’t amenable to this kind of structured decomposition. As always, there are legitimate fears that we will accidentally disincentivize some key aspect of scientific practice not legible to the machine. But I want to resist the idea that because metrics are imperfect that we shouldn’t create them, or frankly that it’s possible to envision a world without them. We need some basis for selection, whether it’s grants or jobs6 - the overthrow of metrics would only lead us to a world that prioritizes who you know — networking and nepotism7. Moreover, the more quickly scientists work via AI, the more telemetry and data artifacts will be created. Ultimately, that may make the creation of these new metrics inevitable.

Of course it’s possible that a much worse version of this plays out. First and foremost, journals could end up co-opting and bankrolling the creation of these systems, using their massive catalogue of papers. They could steamroll any attempts to release public data, and win fair use cases that allow them to maintain their monopoly on the production of knowledge. It would be unfortunate if the infrastructure of science were privately owned. It should be a public good and ultimately maintained by scientists.

Another risk is that this AI system is opaque, full of subtle flaws that make it unreliable upon close inspection. This is why human collaboration is key. Especially at first, we’ll need the system to have confidence scores, to have humans audit high-stakes nodes, and for all the system’s interpretations of papers to be transparent and auditable. It’s crucial that the system be as open as possible to foster trust. This is another reason why the institution that maintains this system should be a non-profit entity.

It’s also possible that we end up with a system that fails to incentivize researchers properly, and ends up with more fruitless gaming. Goodhart’s law is a reality, but that doesn’t mean that all metrics are bad everywhere always. There are better and worse uses of metrics. Goodhart’s law isn’t solvable, but it can be mitigated through the correct institutional structures.

Two cheers for metrics

To mitigate co-option or gaming, metrics need to be flexible and support human judgment. To that end, I want to propose three principles to guide the development of new metrics for valuing scientific contributions.

Values made explicit, and tunable

Metrics are always a reflection of what we value8. Those values are often implicit: “we value writing papers which get cited a lot.” Goodhart’s law still holds; but if we make those values explicit and open, then at least we have the possibility of creating better, more responsive metrics. There are many different scientific values we could care about: novelty, risk-taking, rigor, methodological improvements. All of these and more can be operationalized and included.

Metrics must continue to evolve and respond

The conservatism of academia is such that the few metrics they have are completely frozen. This is a major problem. Consider an alternative model for how metrics can be responsive to efforts to game them: Google search. The Google search ranking algorithm continues to change as researchers find ways to improve it, but also in response to efforts to game it. We know approximately the kinds of signals that Google uses to rank, but we aren’t 100% sure. The real but bounded transparency balances the need for people to be incentivized to create good content for Google to rank with the need for secrecy so that the algorithm can’t be readily reverse engineered. For instance, some of the features that enable sleuths to detect fraudulent papers might need to be secret for a while so that we can better use them. Eventually, those become public and (unfortunately) bad actors will adapt. As stewards of the scientific enterprise, we’ll have to adapt too. Moreover, Google runs experiments to test changes to its ranking - there is a meaningful control. Decisions which hinge on these metrics, like grant decisions, are well served by some decisions made by an alternative procedure for comparison. This is the model we should use when thinking about how to create metrics around the scientific process.

Metric-informed, not metric-driven decision-making

Finally, metrics become tyrannical once they become full substitutes for human judgment. There are still “broken leg” cases. This is a term Paul Meehl used to describe rare circumstances that a human knows are important but sit outside a predictive model. Imagine an actuarial model is designed to predict if a professor will go to the movies on any given night. The model predicts a 90% probability based on historical data. However, the expert intuitively knows the professor just broke their leg. The expert intuitively adjusts the probability to 0%. The bureaucratic nightmare is one where it’s obvious to any human with a brain and a heart that the rules do not apply, and yet the bureaucrat doesn’t make an exception. To be data-informed means to use the data as a tool and not to be a tool of the data. Of course, Meehl’s point was that humans call for exceptions far more than is warranted, so of course there’s a balance.

Conclusion

Despite reasons for concern and pessimism, there is a real prize for humanity to be won if we steward this carefully. A claim level living evidence layer will earn scientists’ and the public’s trust as it improves. As it grows in scope and quality, the clarity it brings as to the most trustworthy science will start to shift academic culture. If we build it correctly, we can solve a problem that is only growing in importance with the advent of generative AI. Namely, how to use AI to safely expand the research enterprise, ensure high quality scientific work, and credit the best research and researchers. The current journal system is roughly sixty years old in the form we know it. It survived the move from print to web, from subscription to open access. It probably won't survive the move from paper-as-unit to claim-as-unit, because that move dissolves what the journal was actually doing. The state withers, in this case, because something better is doing its job.

(FYI I moved the links to its own section so if these annoy you, you can unsubscribe to just the link posts)

1. Natalie Cargill on Olively, an AI app that rewrites your texts to your partner and decodes theirs back to you. Cargill very persuasively makes the case that AI-mediated communication is helping people avoid the real work of attachment repair. I think the reality is more nuanced -- when the AI has the right information it can do more than erode our coping abilities, and help facilitate real therapeutic insight. The full argument will require more than a blurb. Just! Stop! Communicating! — Natalie Cargill

2. Erik Hoel diagnoses 21st century cultural stagnation as overfitting — algorithmic feeds and hyper-discriminatory measurement converge culture on narrow in-distribution outputs, with the AI em-dash tic as a canonical small-scale instance and “no better marker of culture’s unoriginality than everyone talking about culture’s unoriginality.” Big Erik Hoel fan - his book on the nature of consciousness, The World Behind the World, is a must read. Our Overfitted Century — Erik Hoel

3. Ben Recht: p-values are a regulatory mechanism, not a measurement device. Recht takes this view maybe a bit further than I would, but viewing statistics as a mechanism for technocratic decision making seems largely correct. This view is one reason I’m so fixated on better evaluation of randomized control trials, the most load bearing method we have in policy decision-making. Milton Friedman’s p-values — Ben Recht