Interesting Article

[Charles Parker]

An interesting article?

September 15, 2007
High Error Rates Found In Published Scientific Research
Epidemiologist John Ioannidis says most claimed research findings are wrong due to mistakes made by researchers in design and in analysis of results.
Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Should government agencies that hand out research grants hand out parallel grants to independent groups to do parallel analyzes of data produced by funded researchers to find errors?
I suspect peer reviewers lack the time, data, and incentives needed to catch errors in research papers. Given the relative cheapness of data storage and data transmission technologies at least for some types of research independent parties should be given a crack at analyzing the same data to see if the results drawn by original researchers are warranted.
Another thought: Could some types of studies have agreed standard data formats with standard analyzes written ahead of time so that different studies could be compared automatically and so that bias would not enter as much into the analysis phase?
Ioannidis published a paper in Plos Medicine in August 2005 entitled Why Most Published Research Findings Are False.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Step up a level and use the kind of thinking a business executive steeped in the wisdom of W. Edwards Deming would bring to a manufacturing quality problem: Huge amounts of labor goes into conducting scientific studies and therefore a high error rate represents a huge amount of waste of labor and supplies. Science needs more automation and other process improvements to raise quality control and reduce waste.