Researchers in Sweden and the U.K. have
shown that common statistical methods to analyse brain activity through images
taken with MRI scanners had a twelve times higher rate of false results than
Anders Eklund and Hans Knutsson of Linköping University, and Thomas Nichols of the University of Warwick, published their findings based on data collected from nearly 500 cases in the esteemed American scientific journal Proceedings of the National Academy of Sciences (PNAS).
Their work suggested that common statistical methods using functional magnetic resonance imaging (fMRI) to measure brain activity by detecting changes associated with blood flow, cannot be trusted.
Eklund who works at the Swedish university’s Department of Computer and Information Science and the Department of Biomedical Engineering, respectively, tested the analysis methods by using them on known, reliable data. The methods showed false activity in the brain in 60 percent of the cases, while a reasonable figure is five percent.
The analysis methods were previously only validated against simulated data, but Dr Eklund pointed out back in 2012, in his doctoral thesis, that the results are not always reliable, according to the PNAS report.
The statistical methods used are built on a number of assumptions; if one or more of the assumptions are incorrect, the results will also be incorrect, he suggested. At that time, critics maintained that errors could certainly arise if data from a single person was analysed, but that the errors would even out in group analyses.
In his thesis, Dr Eklund proposed another method in which few assumptions are made and significantly more calculations – a thousand times more – are done, which yields a significantly more certain result. With the help of faster graphics cards, the processing time can be reduced so that the method is usable in practice.
Returning from a post-doctotral position at Virginia Tech in 2014, he realised that research was still resting on old, unreliable analysis methods.
“I bought three computers and started testing the statistical methods for group analyses. Whatever I found, it would be interesting, I thought.”
An MRI examination in Sweden can cost SEK 5,000 for an hour (530 euros, 588 dollars), and examining 500 people would carry a great financial burden. But over the last few years, more and more researchers have started sharing their collected data, so that more researchers can make use of it
“Today we can download actual measurement data from healthy control subjects,” Eklund said.
When examinations are done using fMRI, for example, images from 20 healthy people are compared with images from 20 people who have a brain injury or some impaired function.
Dr Eklund then used the current analysis methods and compared 20 healthy people with 20 other healthy people. Where the results should not have been any different, or at least within the five percent error rate, in total he made 3 million comparisons of randomly selected groups with data from 499 healthy persons and “the differences were considerably greater than five percent, up to 60 percent in the worst case.”
Eklund’s results concluded that the analyses could have shown positive results where there shouldn’t have been any, thereby indicating brain activity where there was none.
He also analysed the same data set with his more calculation-heavy method and obtained a considerably better correspondence, with differences in the expected five percent of cases.
“Thanks to modern graphics cards, large calculations can be run. It would take 1,000 times longer to run the calculations using a normal computer, but thanks to the graphics cards I reduced the processing time from ten years to 20 days,” Dr Eklund said in the PNAS report.
An American research group has also repeated his calculations. Dr Eklund, Dr Knutsson (professor at the Department of Biomedical Engineering), and their colleague at the University of Warwick also chose to put out what is known as a ‘pre-print’ on arXiv so that other researchers could present their objections.
“I hope we can get a discussion going in research circles regarding how we validate models. Today, there is both data available to validate and enough processing power to perform the calculations.”
As regards redoing past studies using fMRI, Dr Eklund said that many of them were done ten to 15 years ago and it’s not certain that the basic data still exists. “The important thing is that researchers think about what method they use in future.”
In one of the software programmes used until now, the functions have also been changed so that the results are more correct, Eklund concluded.