Study finds difficulty distinguishing between human and AI-written abstracts

A new study published in JAMA Pediatrics suggests healthcare professionals struggle to identify research abstracts written by artificial intelligence (AI) compared to those written by humans.

The study, led by Dennis Ren, M.D., emergency medicine provider at Children’s National Hospital, highlights the lack of established standards for the use of AI in scientific writing and publishing.

The big picture

The researchers presented 102 healthcare professionals with four research abstracts: two written by human researchers from the Pediatric Academic Societies Meeting in 2020, and two generated by ChatGPT 3.5 (OpenAI). The participants were asked to identify the abstracts’ origin and state how they made their determination.

The participants were able to identify the abstracts’ origins correctly 43% of the time, but accuracy ranged from 20% to 57%. This suggests that healthcare professionals cannot reliably distinguish between research abstracts written by humans and those written by AI. Interestingly, 72.5% of participants believed using AI for research abstracts was ethical.

Why it matters

AI tools are becoming more widely used in scientific research, including writing and editing scientific content. However, there are no set standards for what constitutes the appropriate use of AI in scientific writing and publishing. This study asks the critical question: can healthcare professionals even tell the difference between AI and human generated content?

In conclusion, the authors state that they have no reservations about using AI to generate abstracts or even full articles as long as the final product can be reviewed and edited.

“AI may help with knowledge dissemination, but it can also be a source of misinformation and disinformation,” says Dr. Ren. “We need to teach skills of critical thinking and critical appraisal to everyone.”