I’m excited to announce the launch of Evidence Guide, the place to store and find computational audits of papers. I built this to fill a major gap in research quality evaluation.
Every day I wake up to see another researcher or sleuth announce some set of re-analyses of existing scientific literature1. There is a tremendous enthusiasm for cleaning up the scientific record, one that I obviously share. But I’ve been noticing a growing set of problems with these quality assessments.
First, a lot of these assessments lack transparency. Take the recent debate about Saez and Zucman’s research on inequality and the anonymous audit of their work. Important research should be audited. But to audit the audit, we’d need to know what model produced it and how it was conducted, and we didn’t. Eventually someone did audit the audit, but by then the discourse around their work had already shifted, rightly or wrongly.
Second, it’s difficult to evaluate the quality of different tools and models used to analyze research articles. Large-scale datasets that compare the results of different auditing tools don’t exist. And while there are one-off projects (like my own) that try to build evaluations to test these different tools, having data at scale would surely speed up these efforts.
An even more important problem is discovery. There is nowhere for these audits to go; people are just spinning up their own websites. This limits the ability of others to find audits and see what has been checked on a given paper. You could put the results on PubPeer, but its terms of service prohibit machine-generated comments, and bulk access to its data is limited.
In addition to limiting the impact of these audits, this causes duplicated work. Multiple people are likely burning tokens on auditing the same papers, when their time and compute are better spent expanding the universe of papers being audited. The goal of these projects ultimately should be full coverage of the scientific literature.
That’s why I built Evidence Guide. Evidence Guide provides an API for both writing new audits into the database and reading what audits have been conducted. The schema is broad enough to support a wide range of different audits: from putting a paper into a review tool or a foundation model, to conducting statistical recalculations, image manipulation checks and full computational reproductions. It also records where the audit came from (i.e. what tool, model, software package, or person), which is crucial for evaluation and so that auditors get credit for their work.
Of course, an audit being in the database doesn’t mean it’s right. Some audits could be done badly, or be in error. We’ll eventually build an audit layer on top of this in order to determine which critiques and tools are truly valid. So that anyone can do that kind of evaluation, the database will be fully open. I plan to make it available to anyone who wants it, however they choose to use it. This is a public good for the new scientific communication infrastructure that is starting to emerge and may one day replace the journal system.
As of now, there isn’t much on there. In the coming days my AI agents will be scouring the open web for existing paper checks and ingesting them, and I’m talking to many of the sleuths and auditors about ingesting their results. I’m also planning on creating hooks to make it really easy for anyone to upload an audit from their favorite agent harness or review tool. Over the next few months I’ll add plugins so you can check what has been run on a paper anywhere you search for new research, and crowdsourcing tools so everyone can contribute checks using their own compute and the papers they can lawfully access.
In the meantime, if you are someone currently producing checks on papers at scale, or you are a builder of tools for finding and surfacing research, please reach out to me. I’d love to talk to you about partnership and integration. Of course, if you believe in the vision of this project, tax-deductible donations to my 501(c)(3), the Robyn Dawes Institute, are extremely welcome. I have funding for development and some infrastructure, but keeping this running and scaling it is going to require more money than I currently have.
I hope we can collectively create an accurate and comprehensive assessment of scientific evidence.
A few recent examples: a MathArena check of 656 arXiv math papers, an LLM workflow run on 4,452 economics replication packages, the Hugging Face and alphaXiv hackathon to reproduce ICML papers, Xu and Yang’s replication workflow for political science and economics, an LLM error check of ~2,500 published AI papers, and Michael Frank’s AI audit of his own papers. Earlier efforts include the Black Spatula Project and YesNoError.


Proud of your work here and excited to see what comes next!
Fun fact it was I your humble SaneCite.com founder that audited the audit. :) I'll be reading through your Substack as I can this week thanks for the heads up!- Stephan