Verified by 
Neuroscience
AI Predicts Neuroscience Study Results Better Than Experts
LLMs beat human neuroscience experts in predicting study outcomes.
Updated December 2, 2024 Reviewed by Kaja Perina
At the crossroads of psychology and biology is the inherently complex life sciences field of neuroscience, the study of the brain and nervous system that includes cellular and molecular biology, anatomy, biochemistry, and physiology. Neuroscience is an area that the predictive capabilities of artificial intelligence (AI) could benefit. A new peer-reviewed study published in Nature Human Behaviour demonstrates how AI large language models (LLMs) outperform human neuroscientists in predicting neuroscience study outcomes.
“We foresee a future in which LLMs serve as forward-looking generative models of the scientific literature,” wrote University College London (UCL) Psychology and Language Sciences postdoctoral research fellow Xiaoliang (Ken) Luo, PhD, and cognitive and decision sciences professor Bradley Love, PhD, and their wide consortium of research colleagues affiliated with multiple institutions from around the world.
LLMs are immense AI deep learning models that are pre-trained on massive amounts of data that are able to generate and process human language. The scientists for this study set out to evaluate whether or not LLMs could tackle complex tasks that are difficult for human neuroscientists to perform.
"LLMs’ predictions are informed by a vast scientific literature that no human could read in their lifetime," wrote the scientists.
Predicting the outcome of neuroscience studies is a daunting task for neuroscientists. Factors such as the diversity of neuroscience research methods. Neuroscience research methods include brain imaging such as functional magnetic resonance imaging (fMRI), electroencephalography (EEG), magnetoencephalography (MEG), and positron emission tomography (PET), organoids derived from pluripotent stem cells, and pharmacological interventions, to name just a few.
Other contributing factors that make predicting neuroscience research results challenging for experts include the breadth of multiple levels that range from molecular biology to behavior, the massive volume of pertinent science publications that could be in the thousands, the intricacy and variety of analytical techniques, and the study’s reproducibility, complexity and reliability make it difficult for neuroscientists to predict research outcomes according to the study authors.
The researchers developed a forward-looking benchmark for neuroscience called BrainBench to quantify and compare the ability of various general-purpose LLMs versus 171 human neuroscience experts who passed a screening test to predict neuroscience research outcomes. Several different versions of Llama, Galactica, Falcon, and Mistral comprised a total of 15 LLMs that were evaluated in this study. The test cases included the five neuroscience areas of behavioral/cognitive, systems/circuits, neurobiology of disease, development/plasticity/repair, and cellular/molecular.
The results were clear—each LLM beat human neuroscience experts by a wide margin. The LLMs average accuracy of 81.4 percent far exceeded the 63.4 percent average of human experts.
Next, the scientists created a new LLM called BrainGPT by fine-tuning an existing version of Mistral and training data from twenty years of neuroscience publications from a hundred journals published during 2002-2022. BrainGPT had an 86 percent accuracy in predicting neuroscience study results, which was a three percent gain from the general-purpose version of Mistral.
"LLMs can be part of larger systems that assist researchers in determining the best experiment to conduct next," the researchers wrote.
The ability to predict results of neuroscience research in advance can help guide neuroscientists to optimize limited resources such as time and money, enable timely adjustments based on probable outcomes, and augment our understanding of the brain and central nervous system that may lead to better treatments and health interventions.
This proof of concept is not limited to neuroscience. According to the scientists, none of their methods used was specific just to neuroscience and can be applied more broadly to other knowledge-intensive domains in the future.
"LLM’s impressive forward-looking capabilities suggest a future in which LLMs help scientists make discoveries," the researchers concluded.
Copyright © 2024 Cami Rosso
