Peer Review for AI Researchers (as Handled by Chatbots)
Peer Review for AI Researchers (as Handled by Chatbots)
August 2024
Of Interest to the Information Community
NEW: I wrote about a problem that's plaguing the artificial intelligence field: AI research is clearly being peer-reviewed by AI chatbots, no one really knows what to do about it ...
The experiences from a variety of conferences are discussed in the Chronicle of Higher Education piece. As a sample, we offer the following:
In 2021, 3,014 papers were submitted to the International Conference on Learning Representations (ICLR), a top venue for machine- learning research, and vetted by 4,072 reviewers. This year, the conference reported receiving more than twice as many entries — 7,262 — and accepting about one-third of them. There were also nearly 9,000 reviewers, each of whom evaluated three submissions on average.
Studies have detected AI content in roughly 11 percent to 16 percent of reviews for the most recent ICLR, and 9 percent of those for the most recent NeurIPS. Representatives for the two conferences did not respond to questions about the potential prevalence or influence of AI-written reviews.
