Historical Regression
There's been a story on the news recently that a report suggests that childhood obesity is increasing because more mothers are working, and not staying at home with their children. It's been reported in a few places, one of them is here. It says"We saw that start to happen. We could track childhood obesity. There's a direct correlation," said Terry Mason, Chicago's public health commissioner."This might well be true, or it might not (that's the sort of thing that other blogs can worry about), they mentioned correlation, so we're going to talk about it on this blog.
So, a direct correlation, eh? And what's the sample size here? Let's see, well, we've got before, and we've got after. So we'll give them N=2. That's not a lot, is it? In fact, any two measures (which change) measured before and after some time have to have a correlation of r=+1, or r=-1. They don't even give a specific date to it - it's 'the 1980s'. As far as that evidence goes, you might as well say that it was Ronald Reagan getting elected, the release of Windows version 1.0 (anyone remember that? It had a clock that was pretty cool, but nothing else), or the fact that W Germany made it to the world cup finals 3 times (I'm including 1990, 'cos we're allowed to be a bit fuzzy here.)
In other words, if there WASN'T a direct correlation, we should be surprised.
This is an example of a more general problem (well, two more general problems).
The first is inspecting the data, and then making up a theory about the causal relationships that exist. But you've cheated, you looked at the data. Another example of this sort of thing is the clustering of leukaemia cases around nuclear power stations. When people first theorised this, they first looked at the data, and they said let's choose leukaemia (not any other sort of cancer) amongst children (we think of leukaemia as a children's disease, because that's what makes the news, but it's not necessarily), and let's make the children aged, errmm, less than 3, and make it, errrmmm, 5 miles from a nuclear power station. No! 10 miles! Ah look, we've got a cluster!" Obesity is a problem now, but they identify the root of this in the 1980s - 20 years ago, so any event between now and then would appear to do the job.
The second (related) problem is that of the sample size. Any time you say "Y happened, because X happened before" where X and Y are one off national events, you have an N of 2, and you can't say anything. Donohue and Levitt famously argued in 1999 that legalising abortion had decreased murder rates 20 (or so) years later (it's famous because they discuss it in Freakonomics). But they've got an N of 2. Murder rates are unstable, they weren't going to stay the same, so there was always going to be a correlation. This is discussed in a paper by Ted Goertzel called Myths of Murder and Multiple Regression (and if I ever write a paper with a title that good, I'll die happy). If you want to make this sort of statement, you need to have a bigger sample - you need to have abortion criminalised again, a few years later, because the murder rate should then rise, then ban it, and it should drop again. Alternatively, you need to have abortion legalised in different countries at different times, and then see what happens to the murder rate in those countries, the same period afterwards.
Social scientists do have a habit of being able to make up a plausible sounding theoretical explanation of any result that they happened to find. What if Donohue and Levitt had found that murder rates went up, as a result of abortion being legalized? Do you think they could have made up a plausible sounding theory to account for that too? (Go on, you try. I bet you can.) Well, it happens that other researchers have found that, by controlling for a couple of other variables, legalizing abortion really did make the murder rate go up.
It's unlikely that this will ever replicate, at least enough times to get a sensible sample, and it's unlikely that there will be changes in the workforce that mean that most mothers no longer work, so we'll never know who (if anyone) is correct. And a hallmark of science is that it has to be able to be wrong - for a theory or a finding or a fact to be considered to be scientific, we must be able to state what evidence would make us say "Oh, that was wrong then". For statistical analysis of this sort of result, there isn't anything that would enable us to reject the theory, and if we can't reject it, we might as well rely on psychoanalysis. ("You hate your father? That's because of repressed sexual urges. You love your father? That's because of unrepressed sexual urges.")
