In defence of the null hypothesis

In response to a fascinating article about Charles Babbage and the plantation management techniques that informed his calculating engines, Kevin Marks pointed to an earlier article about how eugenics shaped statistics that was every bit as interesting. I was familiar with some of the background to Galton, Pearson and Fisher but had not taken on board the extent to which “statistical significance” started life as a way of examining the homogeneity of human populations. Does that history negate its usefulness entirely?

I really don’t know, and my doubts are crystallised in a couple of quotes from the article.

For example, Nathaniel Joselson, a data scientist in healthcare technology says: “Just because you haven’t measured something doesn’t mean that it’s not there. Often, you can see it with your eyes, and that’s good enough.” That resurfaces the jealousy some us at the softer end of biology felt for our colleagues in, for example, physiology, who would smugly refer to a statistic they called the interocular traumatisation test, because the effect hit you between the eyes.

More worrying, Deirdre McCloskey and Stephen Ziliak, two economists, claim that:

“Existence, the question of whether, is interesting,” they said, “but it is not scientific.”
Despite the objections, the practice is still very much the norm. When we hear that meeting online is associated with greater happiness than meeting in person, that some food is associated with a decreased risk of cancer, or that such-and-such educational policy led to a statistically significant increase in test scores, and so on, we’re only hearing answers to the “question of whether.” What we should be asking is what causal mechanism explains the difference, whether it can be applied elsewhere, and how much benefit could be obtained from doing so.

But, but, but ...

The problem with that approach, in my own experience, is that people will waste an awful lot of time postulating mechanisms that explain the difference when, in truth, there really is no difference to explain. This is particularly true for what I am going to come out and call pseudoscience. Astrologers like to pontificate about how the gravitational pull of the moon, for example, can affect the tides, so of course its position at the moment of birth could affect the future development of a human being. And each time, I pointlessly point out that the mass of the midwife or anyone else in the room exerted a great gravitational pull than the moon. So, how do you persuade a true believer that there is nothing there? There is no need for a mechanism to explain the difference if there is no difference, but how do you demonstrate that there is no difference?

The same goes for all pseudoscientific claims. First, demonstrate that there actually is something that needs explanation, whether it be the ability to sex an unborn baby with a pendulum, to dowse for water, to diagnose disease from a hand (or breast) print, etc., etc., etc.

In some less extreme cases there may be a case for a Bayesian approach, but when it comes to most pseudoscientific claims, and I realise I am prejudging the issue by labelling them thus, my prior is very close to zero. It is not enough for one person to make extraordinary claims under conditions that only they control. Extraneous sources of information, and of bias, need to be excluded in order first to determine that there is something (other than human frailty) that requires explanation.

How do you do that, rigorously?