Learning Statistics

Retiring

noreply@blogger.com (J) — Mon, 04 Aug 2008 05:03:00 +0000

I've decided to retire this blog. Well, I've decided to accept that I've almost run out of things to say on it. I've decided to merge it with my other blog, into a more general set of ramblings, and start a new blog which you can find here (if you so care).

Measurement vs Statistics

noreply@blogger.com (J) — Mon, 07 Apr 2008 17:05:00 +0000

Bad Science has an interesting article about issues of measurement, before you even get to issues of analysis. Here's the first paragraph:

"There's this vague idea - which has been going around for the past few centuries - that statistics is quite difficult. But in reality the maths is often the least of your problems: the tricky bit comes way before the number crunching, when you are deciding what to measure, how to measure it, and what those measurements mean."

Measurement vs Statistics

noreply@blogger.com (J) — Mon, 07 Apr 2008 17:05:00 +0000

Zehn Mark I

noreply@blogger.com (J) — Sat, 29 Mar 2008 10:05:00 +0000

Zehn Mark I
Originally uploaded by Arenamontanus

Here's an old German 10 mark note, with a picture of Carl Gauss on it (as in Gaussian, as in normal distribution). To the left of the picture of Gauss is a normal distribution, with the formula.

You can read more about the distribution (and see a larger version of the formula) at Wikipedia http://en.wikipedia.org/wiki/Gaussian_distribution.

Another LOLScience Picture

noreply@blogger.com (J) — Mon, 21 Jan 2008 13:02:00 +0000

icanhaspaic128452639380076250
Originally uploaded by increpare

It's Florence Nightingale who invented the pie chart (or something very similar she actually called them polar axis charts, but they led to pie charts) and was the first female member of the Royal Statistical Society.

Swanz

noreply@blogger.com (J) — Fri, 21 Dec 2007 16:36:00 +0000

swanz
Originally uploaded by mr lynch

Fickr has a group called LOL Science, here's one of the pictures. It's a combination of a hideous number of in jokes. If you get it, you can feel smug now.

If you don't get it, here's some explanation. We'll take them in order of relevance to this blog.

It's about Popper, and his philosophy of science. Popper is important, because his philosophy defined the way that we think about science, and statistics. Popper's important idea was that you couldn't prove a theory, you can only disprove it. If you believe that all swans are white, then you can never prove this by finding white swans. You have to go out and try to disprove your theory - you have to try to find a black swan. If you try really hard to find more white swans, that doesn't tell us anything. If you try really hard to find a black swan, and fail, that supports your theory (but doesn't prove it). If you succeed, you've disproved your theory. That's why we focus on the null hypothesis in statistic. The old approach of collecting data to try to prove your theory, is the inductive method, the new approach, of trying hard to disprove your theory, is called the hypothetico-deductive method. Now we know that black swans live in Australia (or the antipodes).
Note for the enthusiastic - more recently it's been suggested that an abductive approach should be employed, rather than a hypothetico-deductive approach.
I'm in ur (X), (Y)ing ur Zs, is a snowclone. The term snowclone originated on the Language Log blog, a snowclone (originally) takes the form "If eskimos have 94 words for snow, then X must have Y words for Z" (eskimos, of course, don't have 94 words for snow, or 16 or 138). The original phrase was "I'm in ur base, killing ur d00dz".
The third link is about LOLcats - this one's not about psychology or statistics, but it is about pictures of cats (or other animals) with amusing captions, usually with grammatical and spelling errors (think how your cat would write if it was aged 14 and texting its friends). The I'm in ur X... meme is a very popular one to use for LOLcats pictures. Here's an article that explains more. (That last one didn't have anything to do with statistics or psychology).

New email subscribe thing

noreply@blogger.com (J) — Sat, 04 Aug 2007 05:40:00 +0000

If you want to get an email when a new post is added to this page, enter your email address in the box on the right. (It's organized by Feedburner.)

e

noreply@blogger.com (J) — Wed, 01 Aug 2007 04:07:00 +0000

New Scientist recently had an article about e [subscription of some sort required for full article], everybody's second favourite mathematical constant. (pi is everyone's favourite mathematical constant, obviously). It talked, in a relatively non-technical manner, about the reasons why e was exciting. It didn't talk about its role in statistics, in, for example, the normal distribution, but it's probably too difficult to do that without assuming some prior knowledge. [Did someone say too boring? Out of the room! NOW!]

Language Log on Odds Ratios

noreply@blogger.com (J) — Wed, 01 Aug 2007 03:33:00 +0000

There was an interesting post on the presentation of odds ratios on the Language Log Blog, the other day. They give some examples of odds ratios being deceptive, confusing and misunderstood. It's been said plenty of times, by plenty of people (including me) but it's interesting that linguists are saying it too.

You're a Bayesian!

noreply@blogger.com (J) — Fri, 29 Jun 2007 23:21:00 +0000

I've written a bit before about Bayesian statistics, here, here and here (that last one where I stole a line from Brad Efron, who said "We can all be Bayesians when we need to be," and also in a recently published book. I'm kind of sympathetic towards Bayesian analysis, but I very rarely do it. The basis for Bayesian analysis is that we incorporate the prior probability of a result into our analysis. Some people are positively antagonistic towards Bayesian thinking - denying that there is ever a use for it - the selection of the prior probability being something of a sticking point. (Actually, Bayesian analysis is lots more complex than that, and doesn't always require what are called 'informative priors', but we won't worry about that for a minute.

However, the most recent issue of Significance had a very interesting article by Stephen Senn, in which he wrote about the TeGenero tgn1412 drug trial catastrophe which occurred in March 2006, when 6 volunteers received the drug, and two received a placebo. The 6 volunteers almost immediately had massive immune system reactions - specifically a cytokine storm, and were hospitalised for at least a month.

What we have here, is the potential of a statistical analysis We've got a 2x2 table, so let's do the stats.

                       Placebo    Drug
                Yes   0          6
Cytokine Storm
                No    2          0

A 2x2 table. We obviously can't do a chi-square test, as the sample is too small. But we can do a Fisher's exact test. If we do that we get a one-tailed p of 0.036. It's a one-tailed test, so our p-value cut off is 0.025, so we don't have evidence that the drug caused the cytokine storm, and all the subsequent ills.

But that's got to be a silly thing to say. It's obvious that the drug did cause the cytokine storm. It's not just barely significant; it's really, really obvious. Why is it so obvious? It's obvious because people don't have cytokine storms every day. In fact, if you haven't got the Spanish Flu we're pretty safe saying that you will never have a cytokine storm. In other words, it's not just the data that we have obtained here that we need to take into account. We need to take into account the probability of having a cytokine storm ever is very low. In other words, we need to take into account the prior probability. And so we have just done a Bayesian analysis.

noreply@blogger.com (J) — Tue, 26 Jun 2007 03:34:00 +0000

Tricia sent me an email:

Greetings and a quick query re power calculations. I've read the very clear text on your website about this [that would be here - JM], but have one question that it doesn't answer. We are planning a study of stress, coping etc in staff in paediatric oncology. The subjects will be all staff in all the UK paediatric oncology units - so the total population rather than a sample. I think I vaguely remember that if you do a total population study then you don't do a power calculation, as it's not a sample, and anyway you can't increase
numbers. Is that correct?

Interesting question.

Two points to answer it.

First, if you have the whole population, you don't need to do inferential statistics at all, so power is irrelevant.

If you have a cohort of sociology students at the University of Uttoxeter, their average age might be 20 years and 3 months. There's no standard error or confidence interval on that. Because you *know* their average age. A cohort of psychology students might be 20 years, 3 months, and 1 day. The psychology students are older. There's no significance test to be done, because it's a population difference.

However, that's not what you are normally interested in. Because we aren't interested in just those cohorts of students. We don't want to know if psychology students in 2007 in York are older than sociology students, we want to know about all psychology and sociology students in general.

Similarly, we aren't really interested in all staff in paediatric oncology units. We want to know about all possible staff in those units, and so the current population is just a sample from the population of possible staff. We can make a statement about those staff today, but tomorrow, they might have changed. You can assess it today but if you don't do a significance test, you can't say anything about it tomorrow.

So you need to do a significance test, and if you need to do a significance test, you need to do a power calculation. The purpose of the power calculation is to determine how large an effect you are likely to find. If you don't get a significant result, you can say "Well, the effect would have had to be as large as X, so whatever the
effect is, it's probably smaller than X."

Two further issues, that you might not have considered.

1) How large is the population? You might look into compromise power analysis if it's small, and there is nothing you can do about it.

2) The sample is going to be clustered, and that's going to make life tricky, because you need to take it into account in both the analysis, and the power calculation. That is, you need to distinguish between within unit effects, and between unit effects. (Is it that within a unit, the people who have coping style X are more stressed, than the
people who have coping style Y, or is it that in there are differences between units, in that some units have more coping style X, and less stress, with within a unit, there is no relation between style and stress. It's possible for those effects to be in opposite directions.)

You might hope ...

noreply@blogger.com (J) — Thu, 03 May 2007 04:48:00 +0000

[Update: Read the link associated with the first comment, it does reveal that Furedi was somewhat misreported - or at least reported out of context.]

Ann Furedi is chief executive of the British Pregnancy Advisory Service. You'd hope that someone who was providing advice would base it on evidence of some sort, and that that person would understand evidence. There was an article in today's Guardian, about whether GPs would sign abortion refusal forms - reporting on a study that survey 309 GPs which found that nearly one in five refused to sign abortion refusal forms.

Furedi was quoted in the Guardian as saying "she did not believe the survey accurately reflected GPs' opinions as it polled less than 1% of the UK's 40,000 GPs". But it's not the proportion of people that you sample that counts, it's the number, and it's their representativeness.

Opinion polls do a pretty good job of predicting elections, but they tend to survey around 1000 people. No one says "There are [approx, I'm guessing] 40,000,000 voters in the UK, and they sampled only 1 in 40,000 (or 0.0025%) of them, so I don't believe the results." If we move to a bigger country, say the US, then 1000 people is taken from a (hypothetical voting) population of 200,000,000 - they sample only 1 in 200,000 people, or 0.0005%, but shockingly this is no worse.

Experiments in psychology are doing pretty well if they have a sample of a couple of hundred people, and we're not trying to generalise to a country there, we're trying to generalise to everyone there is - and everyone there might be - that's an infinite sample, so we have, according to this logic, sampled no one at all.

The crucial thing is how representative the sample is. I have tried to have a look for the article, which seems to have been published in Pulse-i so I could check on it, but I couldn't find it. (You seem to need to be a doctor to register as well). If it was like most magazine surveys, it was a volunteer sample, in which case it probably is nonsense, but not for the reasons Furedi said.

Two further points:

First, it's possible, indeed likely, that Furedi was horribly misquoted. There's a good chance that the journalist who was writing the story didn't understand the issue, so took one part of what she said, and quoted it. (Imagine she might have given a 10 minute explanation of sampling, and at the end said ..."and the sample was on the small side", and that's the bit they decide to quote.

Second, the article also mentions Robbie Foy, who's a pal of mine, so I feel sort of famous by association. (Although not as famous as the fact that I used to have a girlfriend whose mother's hairdresser's sister wrote the words to the theme song to Neighbours).

Wiki on research methods

noreply@blogger.com (J) — Sun, 22 Apr 2007 06:08:00 +0000

A while ago, I wrote a book called 'Research methods and statistics in psychology', which was aimed at first year students studying psychology at university in the UK.

I've now made it into a Wiki (well, the text, anyway). Anyone is free to read it, and edit it. You can find it at www.researchmethodsinpsychology.com.

By the nature of Wikis, it's free as in beer, and free as in speech. I wonder if it's an interesting idea to let students write or change a textbook. Maybe they won't. I'm wondering how much of a role I should / will play. Those 50,000 (or so) words were my baby. Will I get upset if people mash them around? As it says on every editing page "If you don't want your writing to be edited mercilessly, then don't submit it here." Will I have urges to correct things I believe to be wrong? Will it get too big for me to keep track of whether things are wrong? (I doubt it, but you never know).

It's got Google ads on it, which means that it might make me back the $6/month that I spend on hosting. But I suspect it won't quite.

Book page updated

noreply@blogger.com (J) — Mon, 02 Apr 2007 05:20:00 +0000

I've updated the page of supplemental information for the Understanding and Using Statistics in Psychology page. You can see it here, but it's not much use if you haven't got the book.

What's the Denominator

noreply@blogger.com (J) — Sun, 01 Apr 2007 22:54:00 +0000

When you want to try to fiddle people with statistics, to make yourself look better, one thing you can do is to try to change the denominator in your equation. (The denominator is the underneath number of a fraction, so if we say 3/4, the numerator is 3, the denominator is 4).

There was a story in today's Guardian that gives a nice example of this. The story was about ticket touting, and whether Ebay is used by touts. Here's the quote:

Ebay disputes that large-scale ticket touting takes place on its site, citing research by ICM last year that showed nine out of 10 eBay sellers who had listed tickets in the previous 12 months had listed less than five.

(We'll ignore the fact that they should have said 'fewer than', not less than.)

They are using sellers as the denominator, but that's not the right one. The appropriate denominator is tickets. Let's imagine that we had 10 sellers. 9 of those 10 sellers sold 4 tickets (making 36 tickets). The tenth seller sold 964 tickets. This matches what Ebay said, but it's not relevant. What is relevant is that 96.4% of tickets are sold by people who sell more than 500 per year. That looks like a problem.

It isn't necessarily the case that this is true - but it's not ruled out by what they say. Lies, damn lies and statistics may be true - but the lies aren't very hard to see through.

Genetics

noreply@blogger.com (J) — Sun, 01 Apr 2007 04:54:00 +0000

Genetics
Originally uploaded by Beautiful Freaks.

These are my twin sons, Alex (on the right) and Daniel (left), aged 4, reading a book called Polytomous Item Response Theory Models (click on the picture to make it larger).

If I saw this picture, I'd think it was posed, but I really did find them like this.

Standard Deviations (Sample and Population)

noreply@blogger.com (J) — Wed, 28 Mar 2007 03:56:00 +0000

There are two different kinds of standard deviations, and they are a bit prone to getting confused. Here's the text of an email that Darren Van Laar sent to me:

There's a bit on p.42 that i'm unsure of though - as the population and sample standard deviation terms seem to be a bit conflated unless i'm going crazy (possible!).
Sample Sd is the unbiased estimator (is divided by n-1) and is denoted by s. Pop Sd is the other one.
Also, this makes Excel correct, but probably just me...

As I haven't got the book yet, I'm not exactly sure what's on page 42, but I can guess what it's about. And this is where there are slightly different terminologies for the standard deviation.

If you have a sample, taken from the population (which is what you almost always have), then you use the SD which divides by N - 1. This is the unbiased estimate of the standard deviation. It is the best estimate of the population standard deviation.
Because it's the estimate of the population standard deviation, it's sometimes called the population standard deviation (this is what we call it). But it's the estimate that
you should use when you have a sample, so you could call it the sample standard deviation as well.

Similarly, if you have measured the entire population, then your standard deviation is not divided by N-1, it's divided by N. Sometimes (and Excel is included), this is called the population standard deviation, because it's the standard deviation that's used when you have measured the population.

In fact, it's mostly a lot of worrying about not very much, because we never have an entire population, so we never divide by N, we always divide by N - 1 .

However, it's possible that it's wrong in the book - I haven't seen it to check, and we had such a horrible time with the typesetter that all kinds of things changed - between what we wrote and the proofs that we saw.

Book is Out

noreply@blogger.com (J) — Wed, 28 Mar 2007 03:37:00 +0000

I heard that the book I have written with Phil Banyard "Understanding and Using Statistics in Psychology" is available for purchase in your favourite bookshop.

You'd imagine that someone like the publisher would tell me stuff like that - after all. But you'd be wrong. I discovered it was out because Darren Van Laar, from Portsmouth University, emailed to ask me a question about something on Page 42.

Unfortunately, as I haven't actually got a copy of the book yet, I was only able to help partially. Also unfortunately, this means that all the things I said I was going to put on the book's website, because they wouldn't fit in the book, will actually have to be put on the book's website.

Scoring questionnaires in SPSS and Stata.

noreply@blogger.com (J) — Wed, 28 Mar 2007 03:17:00 +0000

I sent an email to the psych-postgrads list, about scoring questionnaires in SPSS. I thought I'd reproduce it here, and elaborate a bit to include Stata.

First, use syntax. You can check syntax for errors, you can rerun it when you find another late questionnaire, you can save it and re-use it years later. You can give it to your friends and make yourself a more popular person.

Here are some tips for syntax:

First, SPSS doesn't mind "whitespace", so you can make your syntax
more legible, and less prone to mistakes:

Instead of:

COMPUTE AQPHYSIC = q1aq + q5aq + q7aq + q12aq + q13aq + q20aq + q24aq + q27aq + q29aq.

Write:

COMPUTE AQPHYSIC =
q1aq +
q5aq +
q7aq +
q12aq +
q13aq +
q20aq +
q24aq +
q27aq +
q29aq
.

Secondly, items often need to be reversed, so you need to create a new variable for that, which you might call q29aq.rev. But if you do that, you've got to remember to put it in your syntax to score the questionnaire. Instead of doing that, create a new variable for every item in your scale (or subscale).

So instead, create new variables with the new scales.

For example, I wrote some syntax (in 1994) for scoring the EPQ-R (short form). Here's the bit that creates the lie scale:

COMPUTE l1.jm = epq3 .
COMPUTE l2.jm = epq8 .
COMPUTE l3.jm = epq12 .
COMPUTE l4.jm = epq16.
COMPUTE l5.jm = epq20 .
COMPUTE l6.jm = epq24 .
COMPUTE l7.jm = epq29 .
COMPUTE l8.jm = epq33 .
COMPUTE l9.jm = epq37 .
COMPUTE l10.jm = epq40 .
COMPUTE l11.jm = epq45 .
COMPUTE l12.jm = epq47 .

I create items l1.jm to l12.jm. (I call them .jm to make it obvious what they were, and because I might be using this on anyone's dataset, and I don't know what their
variables will be called.)

Next you need to reverse the items that need to be reversed:

RECODE
l1.jm l4.jm l11.jm (1=0) (0=1) .

When you've done that, you can create the sums very easily, using compute statements.

There's an additional advantage to using new variables, and that is that of writing all the variables out, you can use "to".

So:

compute l.total = sum(l1.jm to l12.jm).

And SPSS knows that means all the between l1.jm and l12.jm.

BUT WAIT, there's another problem, which is missing data. Software handles this in two ways: Excel gives it a zero. SPSS either gives it a zero, or makes the sum variable missing. Neither of these are right.

The solution is to use the mean score, and then multiple the mean by the number of items. If all variables are completed, then the mean score multiplied by the number of items will equal the total. If a score is missing, then that item will be given the average of all the items that the person did complete.

However, if a person only completed one item out of 50, we probably don't want to give them a score for the total. The solution is mean.x, where x is the number of items that must have been completed for a score to be given.

So, if you want to only give people a score on the lie scale if they have completed 8 items, you use:

compute l.total = mean.8(l1.jm to l12.jm) * 12.

However, it's often better not to use the total score anyway, the mean
score for an item is more useful, as it's more interpretable, so just
miss off the *12, and use:

compute l.total = mean.8(l1.jm to l12.jm) .

You can also do this in Stata, and it's far, far easier. In Stata, you use the -alpha- command. The command works out which items need to be reversed (very, very occasionally it gets it wrong though), and calculates a mean score.

Stata does care about carriage returns (unless you tell it not to) so you can't use the same trick to clarify your code.

In Stata, you would write (all on one line):
alpha epq8 epq12 epq16epq20 epq24 epq29 epq33 epq37 epq40 epq45 epq47 , generate (l.total) min(8) item

The generate() option tells Stata to generate a new variable. min(8) tells Stata to only include cases that have answered at least 8 items, and item tells Stata to do an item analysis, so that you can see which items Stata thought should be reversed.

Conservapedia

noreply@blogger.com (J) — Wed, 07 Mar 2007 22:12:00 +0000

If you're the kind of person who reads about stuff on the internet (which obviously you are, 'cos you're reading this), you'll have encountered Conservapedia, and various websites that have mocked some of the entries.

I had a look for some articles on statistical sorts of things, so that I could join in the mocking, that being my idea of fun. But I failed. The only article I found was one on statistics, which is brief but I couldn't really knock for accuracy. I couldn't find anything else, mean, median, mode, regression, correlation, Fisher, Pearson, all have no entry.

I tried more words more distantly related to statistics - causality takes you to a page on non-sequiturs , which does a half-hearted job of explaining the relation between correlation and causality. 'Evidence' takes you to a page about Jesus.

There are lots of pages to mock (read the Jesus page, but also note that the discussion shows that there are some more sensible people there), but really, it's too easy. However, I'll point out that on the Macroevolution page they make the error of describing macroevolution as "the unproven theory". Of course it's unproven. Theories can be lots of things - supported, useful, good, elaborate, but what they can't be is proven.

GPower 3

noreply@blogger.com (J) — Wed, 14 Feb 2007 18:32:00 +0000

G*Power version 3, a free program for power analysis, has just been released - it's a big step up from version 2, which looked a bit ancient, but still worked well. You can download it from here: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/ . You can read more about power here: http://www.jeremymiles.co.uk/misc/power

What is a facilitated system

noreply@blogger.com (J) — Sat, 10 Feb 2007 18:44:00 +0000

A little while ago on this blog, I wrote about a post on the facilitated systems blog. I said I didn't know what a facilitated system was. Well, Bill Harris has answered my (implied) question.

Types of Errors

noreply@blogger.com (J) — Thu, 08 Feb 2007 05:54:00 +0000

Andrew Gelman's blog had a post today which linked back to an old post where he discussed the different kinds of errors. In our forthcoming book, we discuss the fact that all null hypotheses are false. If that's the case (and it is) you can never make a type I error. Because you make a type I error when you reject a null hypothesis, which is true. And if you know that all null hypotheses are false, you can't make a type II error, when you say that the null hypothesis isn't false, when it is. Because you know it isn't false.

So what do we do? We can't say that we never make an error, so instead, we need knew kinds of errors, which Gelman calls Type M and Type S.

A type M error occurs when you get the magnitude of an effect wrong. If we test for a correlation between two measures, and we find that the correlation is not significant, we (should) say, 'Well, whatever the effect is, it's small (and we don't know what direction it's in).' If the correlation is actually large, we've made a type M error.

A type S correlation occurs when we get the sign of an effect wrong. Let's say we find a significant positive correlation, and conclude that the population correlation is positive. If the population correlation is actually negative, we've made a type S error.

Type M and Type S errors make a lot more sense than Type I and II errors (which, as we've seen here, don't make sense). And they're a lot easier to remember. Gelman then goes into a lot of Bayesian elaboration, which I don't want to go into. I can be a Bayesian when I need to, but I've really got to need to.

Statistics versus Qualitative Research

noreply@blogger.com (J) — Tue, 02 Jan 2007 18:01:00 +0000

I wrote an article that was published in The Psychologist called "Why I study statistics". (It was, if you're interested, volume 19, part 11, 672-673, and part of a series on Why I study ...). Anyway, in the next issue (or maybe the one after) the following letter was published, written by David Mingay:

I HAVE just read ‘Qualitative research… Emerging from the cocoon of science’ (November 2006) and didn’t understand a word of it. I turned the page and read ‘Why I study statistics’ and understood it completely. Maybe that simple observation answers the question posed by the author of the first article:‘Why does it [qualitative research] have such a low profile within academic psychology?’

I'm not sure that anyone has ever previously said they understood anything I wrote.

Update: Article 'Why I study ... statistics' available as PDF. (Nice picture of me as well).

Horse racing deaths

noreply@blogger.com (J) — Wed, 20 Dec 2006 06:09:00 +0000

One reason that we do statistical analysis and don't rely on our brains is that our brains are not to be trusted, a great deal of the time. Brains were designed by evolution to keep us alive in the risky environment of something like the African Savannah. One thing that might get you into a lot of trouble is missing out on some important pattern (there's always a tiger here at dusk, 3 days after rain, there is water over there, 3 weeks after rain, it's over here, etc). So our brains make a lot of type I errors, when it comes to spotting patterns - a type I error (seeing a pattern when there wasn't one) is a lot more serious than a type II error (not seeing a pattern, when there was one). A type I error means you avoid a certain area at dusk, which is a slight inconvenience. A type II error means you get eaten.

The Guardian has a story today about horse racing, and specifically about a run of deaths of horses at Wolverhampton racetrack. This obviously isn't a good thing, but one needs to ask the question of whether this means that there is something wrong with the track which is causing this problem, or whether it's just a statistical fluke - an artefact, which our brains spot as a pattern.

Doing the analysis to find out is tricky, for two reasons - our friends the type I and type II errors. There are quite a lot of racetracks (in England? Britain? The UK? British Isles? Europe? The world?) - what type I error rate do we want to accept, and over what area? And, if we control the type I error rate, we lose power, and therefore don't detect an effect.

This isn't just a problem for horses - the same problem arose in the Harold Shipman case - a type I error means you wrongly accuse a doctor of killing their patients. A type II error means that you miss a murderous GP.

(If you're interested, you can read about appropriate tests here, or watch a video here.)