Do the advocates purport that it can perform all of the other functions of peer review, such as reference checking, evaluating the impact and novelty of the work, comparing the chosen approach to alternative methods, etc.? If so, then it would certainly be good to evaluate those claims. But this article doesn’t do that, it just focuses on the one aspect, checking for errors.
Yes, reference checking is a feature coming to many of these systems, and I've seen refine for example add an "overall assessment" that is starting to look like a more holistic evaluation. So these capabilities are on their way. I agree with you we should evaluate these systems on those functions as well! Great next project. Thank you so much for your thoughtful pushback.
Of course, I agree. Publication worthiness is partly dependent on whether methods are sound (which is what these systems currently purport to test), and also on the significance of the finding in light of the larger research literature and the field as a whole (which I address as a limitation of these systems toward the end of the piece). These systems are only beginning to do the latter and need much more context to be able to function like an expert in that field. But certainly error catching is an important aspect and it’s still helpful to know how well it does at that aspect. You’ll have to pardon my clickbaity title I suppose :)
It's not just the headline. I don't like the sentence, "There has been a proliferation of AI tools that purport to do peer review." This misrepresents what peer review is. If you said they are tools that assist the peer reviewer, that would be better.
On reflection, I want to push back a tiny bit. People take these systems as potential substitutes for peer review, so we should evaluate them as such. These systems do purport to be able to do the job of peer review (which is what I wrote), which is their claim (or a claim of their proponents) -- not mine.
Do the advocates purport that it can perform all of the other functions of peer review, such as reference checking, evaluating the impact and novelty of the work, comparing the chosen approach to alternative methods, etc.? If so, then it would certainly be good to evaluate those claims. But this article doesn’t do that, it just focuses on the one aspect, checking for errors.
Yes, reference checking is a feature coming to many of these systems, and I've seen refine for example add an "overall assessment" that is starting to look like a more holistic evaluation. So these capabilities are on their way. I agree with you we should evaluate these systems on those functions as well! Great next project. Thank you so much for your thoughtful pushback.
Surprising that the commercial systems caught substantially fewer errors than the top foundation models!
Yes, I was surprised to learn that too!
I don't think the purpose of peer review is primarily to catch errors. This is useful but it's somewhat different from what peer review does.
Of course, I agree. Publication worthiness is partly dependent on whether methods are sound (which is what these systems currently purport to test), and also on the significance of the finding in light of the larger research literature and the field as a whole (which I address as a limitation of these systems toward the end of the piece). These systems are only beginning to do the latter and need much more context to be able to function like an expert in that field. But certainly error catching is an important aspect and it’s still helpful to know how well it does at that aspect. You’ll have to pardon my clickbaity title I suppose :)
It's not just the headline. I don't like the sentence, "There has been a proliferation of AI tools that purport to do peer review." This misrepresents what peer review is. If you said they are tools that assist the peer reviewer, that would be better.
On reflection, I want to push back a tiny bit. People take these systems as potential substitutes for peer review, so we should evaluate them as such. These systems do purport to be able to do the job of peer review (which is what I wrote), which is their claim (or a claim of their proponents) -- not mine.
Fair enough :-) Nobody's perfect!