Andrew Gelman's blog had a post today which linked back to an
old post where he discussed the different kinds of errors. In our
forthcoming book, we discuss the fact that all null hypotheses are false. If that's the case (and it is) you can never make a type I error. Because you make a type I error when you reject a null hypothesis, which is true. And if you know that all null hypotheses are false, you can't make a type II error, when you say that the null hypothesis isn't false, when it is. Because you know it isn't false.
So what do we do? We can't say that we never make an error, so instead, we need knew kinds of errors, which Gelman calls Type M and Type S.
A type M error occurs when you get the magnitude of an effect wrong. If we test for a correlation between two measures, and we find that the correlation is not significant, we (should) say, 'Well, whatever the effect is, it's small (and we don't know what direction it's in).' If the correlation is actually large, we've made a type M error.
A type S correlation occurs when we get the sign of an effect wrong. Let's say we find a significant positive correlation, and conclude that the population correlation is positive. If the population correlation is actually negative, we've made a type S error.
Type M and Type S errors make a lot more sense than Type I and II errors (which, as we've seen here, don't make sense). And they're a lot easier to remember. Gelman then goes into a lot of Bayesian elaboration, which I don't want to go into. I can be a Bayesian when I need to, but I've really got to need to.
