When the details matter..Posted: September 27, 2007
Statistics can be interesting and in some way, worrying. At the moment I am analysing some data from an experiment. Part of this involves determining if a statistically significant result has been observed and involves the use of off-the-shelf statistics packages.
With some interest it appeared that when I viewed my data in a graph, there was something possibly significant. So I perform the analysis but nothing is really there, A really isn’t better than B although a graph conveys a convincing story. A few hours later as I was packing up for the day my thesis supervisor walks past and gives me a printout showing there is significance, A is better than B, and a mischievous smile. Upon looking closer I notice what has been left out, some of the high level interactions between independent variables. Of course all the interactions should have been left in, leading to my original non-significant result. The software had been tweaked to give a valid result and this sort of tweaking is in the results that you don’t normally leave in a thesis or published article. A few jokes about making sure you’re upfront with what you are trying to prove and not using post-hoc decisions and we go on our way.
The moral of the story here isn’t about statistics it’s about people and their motivation, especially in science. When using the process of science, the statistics, the experiments are all tools. How they are used is what people do and reflects the desires and motivations of those people … and the barrow that they might well be pushing. If my research funding had depended on those results it would have been easy to tweak things, creating a phantasm. However it doesn’t and my negative findings, which are in some ways good because we have increased our knowledge in the area will stand without any manipulation.
However there are many places in research where future funding is dependant on results. In the same way that pseudo-scientific claims depend on anecdotal results not based on real efficacy , real science can also be corrupted by funding worries or career development. There have been many cases of fraud in science but at least most are soon caught and practically all eventually caught when the results can’t be duplicated using the published methods. This self correcting mechanism is one of the real dividers between truth and well, everything else. The ability to see how conclusions are derived, to repeat an experiment and to be open with correcting mistakes and oversights is a real benefit to the growth and maturity of ideas. Unlike many of the dogma that seem to have survived into the 21st century, science at least throws out the bad and builds on the good.
With the amount of information available today most people who want to be scientifically literate or who just care about the truth, should be able to find out quite a lot about claims made in the media or elsewhere. Google is your friend and so can Wikipedia. If you need information on a complex subject many Universities have their course notes on-line. Plus you will also find that many claims are already challenged in various on-line forums and podcasts. Even scientific claims should be checked and motivation determined. That latest medical breakthrough might sound great but who funded it and who will benefit from the conclusions, everyone or some people more than others? And does it really make sense?
The details need to be there, they might not be easy to fathom and it might take study and deep understanding. However, in science, where you want a result to be repeatable and actually work, the details do matter. And you need well founded confidence in the motivation of people who publish those results.