This is really interesting! I recently talked to a professor that uses refine.ink , and he said he pays $30 a paper for it (he must have bought the 10 reviews for $299.99 deal, because a single review is $50!).
Our system is very systematic, at least when it comes to checking statistics and arithmetic, so we are hoping it will perform well... we really don't know though!! We don't really have a general prompt in our system for methodological issues or simple factual errors, it's more about stress testing data and stats at the moment.
I'm super curious what you find! I'm also really interested in extending this benchmark for other fields - if you're interested in collaborating on a larger effort to do so, let me know :)
Great question - I made a PDF of this post and thought about putting it somewhere. I don't have any plans to publish this anywhere right now, though I'm open to suggestions :)
"We ended up with a 62-category taxonomy of types of errors, ranging from statistical errors, construct validity issues, causal inference, generalizability, etc. "
Can you share the full 62-category taxonomy in a table/spreadsheet? That alone is an interesting and valuable contribution.
If these were already published open access papers shouldn't AI catch all errors by simply comparing to the published original? Wouldn't the results be significantly worse if the correct text was not available as in real peer review situations?
This is a great question! I was worried about the possibility of memorization - that the original papers were in the training data and therefore that the models should find it trivially easy to catch errors. If that were the case, I'd expect the benchmark to be saturated, and furthermore, I'd also expect that model having an easy time with omission based errors. And yet that's not what we see -- we see far from saturation, we see difficulty with omission based errors. So whatever the models are doing, they aren't simply comparing to the published originals.
That being said, it's a really interesting question regarding published vs unpublished papers. I think it would be a very good next step to gather unpublished manuscripts from a group of researchers and then construct the benchmark out of those instead.
Do the advocates purport that it can perform all of the other functions of peer review, such as reference checking, evaluating the impact and novelty of the work, comparing the chosen approach to alternative methods, etc.? If so, then it would certainly be good to evaluate those claims. But this article doesn’t do that, it just focuses on the one aspect, checking for errors.
Yes, reference checking is a feature coming to many of these systems, and I've seen refine for example add an "overall assessment" that is starting to look like a more holistic evaluation. So these capabilities are on their way. I agree with you we should evaluate these systems on those functions as well! Great next project. Thank you so much for your thoughtful pushback.
Was just discussing this and some other papers with Claude and it flagged the following:
You write:
> For the foundation models, we also reran the papers three times to see the test-retest reliability, and whether running the same model multiple times could spot additional errors (they didn’t, although we could have fiddled with temperature settings and looked at this more deeply).
But in a recent paper, authors found:
> mean within-paper score SD across repeated, context-isolated runs of the same model on the same paper was 1.154 for Claude Sonnet 4.5, 1.488 for GPT-5.2 Thinking, and 1.257 for Gemini 3 Pro Preview, on the ICLR scale of 1/3/5/6/8/10.
Those are very large standard deviations! The model judgements seem to vary wildly from one run to the next, but error catching doesn't. So what's going on? Do their summary judgements vary wildly given the same error catching, or are there other inputs that make these judgements so variable?
Great question - could be that earlier models were more variable? It could be temperature settings maybe? I'm sorry I don't have a huge amount of spare time to dig into this right now :)
The omission result is the part I find most interesting. If all seven universally missed errors were omissions, it seems like the harder problem may not be error detection at all, but knowing what should have been present. That pushes the evaluation toward an explicit completeness model or external context layer (oracle problem + completeness problem). Otherwise absence has no observable referent.
I’m also curious how you think about the 93% ensemble result once precision is added. High pooled recall is valuable, but without knowing the false-positive surface, it’s hard to know what operational burden the composition creates.
Thanks for the comment -- yes I've been thinking a lot about an external context layer, both in terms of evidentiary standards for particular subfields of research and in terms of a structured claim graph. I hope to have more to say about epistemic standards soon!
Re: the false positive issue and pooling -- I agree that a measurement of false positives is a crucial next step. I've been thinking about this a lot, how we might be able to determine whether or not a given issue flagged is a real one or not. I think maybe looking at inter-LLM disagreement is interesting, or something else I've been increasingly playing with is bouncing comments back and forth between LLMs until they reach a reflective equilibrium. There's also the possibility of collecting expert human data. I'm very interesting in assembling a golden dataset of reviews, though it's going to likely be very expensive to do also and may also require many humans to verify.
I’ve written a few papers on this topic if you’re interested in collaborating. Both in claim standards and an observation to determination utilizing methodologies like DIT, metamorphic testing and behavioral testing.
These are all reliant on access to the decision system and the primitives that make up provenance.
Of course, I agree. Publication worthiness is partly dependent on whether methods are sound (which is what these systems currently purport to test), and also on the significance of the finding in light of the larger research literature and the field as a whole (which I address as a limitation of these systems toward the end of the piece). These systems are only beginning to do the latter and need much more context to be able to function like an expert in that field. But certainly error catching is an important aspect and it’s still helpful to know how well it does at that aspect. You’ll have to pardon my clickbaity title I suppose :)
It's not just the headline. I don't like the sentence, "There has been a proliferation of AI tools that purport to do peer review." This misrepresents what peer review is. If you said they are tools that assist the peer reviewer, that would be better.
On reflection, I want to push back a tiny bit. People take these systems as potential substitutes for peer review, so we should evaluate them as such. These systems do purport to be able to do the job of peer review (which is what I wrote), which is their claim (or a claim of their proponents) -- not mine.
This is really interesting! I recently talked to a professor that uses refine.ink , and he said he pays $30 a paper for it (he must have bought the 10 reviews for $299.99 deal, because a single review is $50!).
We will be testing our Forensic Metascience Agent on your dataset very soon! (https://forensicmetascience.org )
Our system is very systematic, at least when it comes to checking statistics and arithmetic, so we are hoping it will perform well... we really don't know though!! We don't really have a general prompt in our system for methodological issues or simple factual errors, it's more about stress testing data and stats at the moment.
I'm super curious what you find! I'm also really interested in extending this benchmark for other fields - if you're interested in collaborating on a larger effort to do so, let me know :)
I'm working on a systematic review in this area and would like to know whether you have a preprint or other publication available or planned.
A private benchmark with different errors would be neat as well. Glad to see the caveat in the README file.
Hi Jay!
Great question - I made a PDF of this post and thought about putting it somewhere. I don't have any plans to publish this anywhere right now, though I'm open to suggestions :)
"We ended up with a 62-category taxonomy of types of errors, ranging from statistical errors, construct validity issues, causal inference, generalizability, etc. "
Can you share the full 62-category taxonomy in a table/spreadsheet? That alone is an interesting and valuable contribution.
Yes! That taxonomy is in the public repo, here: https://github.com/Dawes-Institute/ai-peer-review-benchmark/blob/main/refine_taxonomy_v4.md
A few ideas: metaXiv (https://osf.io/preprints/metaarxiv), Zenodo (https://zenodo.org/), or Figshare (more polished UI: https://figshare.com/). The closer the report is to a preprint/working paper, the easier it will be to inspire others.
A few moments of work can go a long way toward building a culture of AI evaluation.
If these were already published open access papers shouldn't AI catch all errors by simply comparing to the published original? Wouldn't the results be significantly worse if the correct text was not available as in real peer review situations?
This is a great question! I was worried about the possibility of memorization - that the original papers were in the training data and therefore that the models should find it trivially easy to catch errors. If that were the case, I'd expect the benchmark to be saturated, and furthermore, I'd also expect that model having an easy time with omission based errors. And yet that's not what we see -- we see far from saturation, we see difficulty with omission based errors. So whatever the models are doing, they aren't simply comparing to the published originals.
That being said, it's a really interesting question regarding published vs unpublished papers. I think it would be a very good next step to gather unpublished manuscripts from a group of researchers and then construct the benchmark out of those instead.
Do the advocates purport that it can perform all of the other functions of peer review, such as reference checking, evaluating the impact and novelty of the work, comparing the chosen approach to alternative methods, etc.? If so, then it would certainly be good to evaluate those claims. But this article doesn’t do that, it just focuses on the one aspect, checking for errors.
Yes, reference checking is a feature coming to many of these systems, and I've seen refine for example add an "overall assessment" that is starting to look like a more holistic evaluation. So these capabilities are on their way. I agree with you we should evaluate these systems on those functions as well! Great next project. Thank you so much for your thoughtful pushback.
Surprising that the commercial systems caught substantially fewer errors than the top foundation models!
Yes, I was surprised to learn that too!
Was just discussing this and some other papers with Claude and it flagged the following:
You write:
> For the foundation models, we also reran the papers three times to see the test-retest reliability, and whether running the same model multiple times could spot additional errors (they didn’t, although we could have fiddled with temperature settings and looked at this more deeply).
But in a recent paper, authors found:
> mean within-paper score SD across repeated, context-isolated runs of the same model on the same paper was 1.154 for Claude Sonnet 4.5, 1.488 for GPT-5.2 Thinking, and 1.257 for Gemini 3 Pro Preview, on the ICLR scale of 1/3/5/6/8/10.
https://www.mdpi.com/2304-6775/14/3/55
Those are very large standard deviations! The model judgements seem to vary wildly from one run to the next, but error catching doesn't. So what's going on? Do their summary judgements vary wildly given the same error catching, or are there other inputs that make these judgements so variable?
Great question - could be that earlier models were more variable? It could be temperature settings maybe? I'm sorry I don't have a huge amount of spare time to dig into this right now :)
The omission result is the part I find most interesting. If all seven universally missed errors were omissions, it seems like the harder problem may not be error detection at all, but knowing what should have been present. That pushes the evaluation toward an explicit completeness model or external context layer (oracle problem + completeness problem). Otherwise absence has no observable referent.
I’m also curious how you think about the 93% ensemble result once precision is added. High pooled recall is valuable, but without knowing the false-positive surface, it’s hard to know what operational burden the composition creates.
Thanks for the comment -- yes I've been thinking a lot about an external context layer, both in terms of evidentiary standards for particular subfields of research and in terms of a structured claim graph. I hope to have more to say about epistemic standards soon!
Re: the false positive issue and pooling -- I agree that a measurement of false positives is a crucial next step. I've been thinking about this a lot, how we might be able to determine whether or not a given issue flagged is a real one or not. I think maybe looking at inter-LLM disagreement is interesting, or something else I've been increasingly playing with is bouncing comments back and forth between LLMs until they reach a reflective equilibrium. There's also the possibility of collecting expert human data. I'm very interesting in assembling a golden dataset of reviews, though it's going to likely be very expensive to do also and may also require many humans to verify.
I’ve written a few papers on this topic if you’re interested in collaborating. Both in claim standards and an observation to determination utilizing methodologies like DIT, metamorphic testing and behavioral testing.
These are all reliant on access to the decision system and the primitives that make up provenance.
Interstesting -- i've not heard about DIT. I'd love to hear more about how these techniques would help measure false positives!
I don't think the purpose of peer review is primarily to catch errors. This is useful but it's somewhat different from what peer review does.
Of course, I agree. Publication worthiness is partly dependent on whether methods are sound (which is what these systems currently purport to test), and also on the significance of the finding in light of the larger research literature and the field as a whole (which I address as a limitation of these systems toward the end of the piece). These systems are only beginning to do the latter and need much more context to be able to function like an expert in that field. But certainly error catching is an important aspect and it’s still helpful to know how well it does at that aspect. You’ll have to pardon my clickbaity title I suppose :)
It's not just the headline. I don't like the sentence, "There has been a proliferation of AI tools that purport to do peer review." This misrepresents what peer review is. If you said they are tools that assist the peer reviewer, that would be better.
On reflection, I want to push back a tiny bit. People take these systems as potential substitutes for peer review, so we should evaluate them as such. These systems do purport to be able to do the job of peer review (which is what I wrote), which is their claim (or a claim of their proponents) -- not mine.
Fair enough :-) Nobody's perfect!