Detection of AI-Generated Reviewer Critiques
Description
This tool detects if a given written critique was generated by AI tools, such as large language models (LLMs), as the reviewers are not allowed to write their critiques using these AI assistants. Flagged critiques are examined by NIH staff and discussed with the reviewers.
Detailed example
This tool outputs classifications of whether the reviewer critique was likely produced by Generative AI or not.
AI / analytics pattern
Natural Language Processing: AI that processes, interprets, and shares information in human language.
Automation level / stage
b) Pilot – The use case has been deployed in a limited test or pilot capacity.
Expected benefit
NIH policy mandates that reviewers produce independent analyses of grant applications based on their expertise and knowledge. The use of genAI to produce a written critique of a grant application is in violation of NIH policy and fails to provide an independent assessment of the application.
Controls / human review
ATO: Yes; PIA: Not published
Data needed
critiques written by the reviewers, AI-generated critiques, Amazon book reviews, and AI-generated book reviews