The Sampling Lab

Chapter 6 Testing

How surprising is your result if nothing is going on?

In a world where the effect is zero, how often would chance alone produce data at least this extreme? Build that world and count.

10 min read · 7 interactive figures

A p-value calculator that explains its answer. Enter a z score, a t score with its degrees of freedom, or a sample’s mean, SD and size, and choose one or two tails. The shaded tail is the p-value; drag the blue statistic to move it.#
On this page
  1. Build the world where nothing is going on
  2. Power: will the study find the effect if it is there?
  3. Many tests, many false positives
  4. What a significant result is worth
  5. What a p-value is not
  6. After significance testing

A p-value is the probability of getting a result at least as extreme as the one you observed, calculated in a world where the null hypothesis, usually “no effect”, is true. A small p-value says your data would be surprising in that world. It is not the probability that the null hypothesis is true, it is not the probability that your result is due to chance, and it says nothing about how large or important an effect is.

The calculator at the top of the page turns a z or t statistic into a p-value and says in words what the number means (there is a longer guide to using it). The definition is short and the misreadings are many, so the rest of this chapter does not start from the definition. It starts from a world you can build: one in which the treatment does nothing at all. Run your experiment in that world thousands of times, and count how often it produces a result as striking as yours. That count, as a fraction, is the p-value.

Build the world where nothing is going on

Here is the experiment. Two groups of n people each get a treatment or a placebo, and you measure an outcome, say a score that varies from person to person with a standard deviation of 10 points. The treatment group averages 5 points higher. Is that a real effect or the luck of who ended up in which group? The null hypothesis says the treatment does nothing, so any difference between the groups is luck. Before running it, a prediction about that world.

Question 1 Prediction before the simulation.#
Figure 1 First panel: thousands of experiments run in a world with no effect. Each column counts experiments whose two groups differed by that much; the vermilion columns are at least as extreme as your result. Their share is the p-value. Second panel (beside it on wide screens, below it on phones): the p-values themselves, in a world whose true difference you set.#

With 20 people per group, a 5-point difference is not rare in the null world: about 11 % of the simulated experiments produce one at least that large in either direction, so p ≈ 0.11. Slide your observed difference up to 7 and the vermilion tails shrink to about 3 %. Notice that the simulated fraction and the exact value agree closely; the exact value comes from the normal curve the null experiments trace out, whose standard deviation, the standard error of the difference, is 10 × √(2/20) = 3.16 points. A p-value is a statement about where your result falls in that curve:

p = P( |D| ≥ |dobs|  |  H0 ) = 2 (1 − Φ(|z|)),   z = dobs ⁄ SE(6.1)

Now look at the second panel with δ = 0. The p-values are flat, as the prediction said. This is true for any test with a continuous statistic when the null hypothesis and the model’s assumptions hold, and it is the whole basis of significance testing: if you call a result “significant” whenever p < 0.05, then in a world with no effect you will be wrong 5 % of the time, no more and no less[1]. Raise δ and the p-values pile up toward 0. The share below 0.05 is then the test’s power, the subject of the next section. established

Why 0.05?

The threshold is a convention, and its origin is a single sentence. In Statistical Methods for Research Workers (1925), R. A. Fisher wrote that the value for which P = .05, or 1 in 20, is 1.96 or nearly 2, and “it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not”[2]. Fisher treated it as a rough guide for deciding what deserved a second look. It became a rule. In 2018 a group of 72 researchers proposed lowering the threshold for claims of new discoveries to 0.005[3]; others argued for abandoning fixed thresholds altogether. The American Statistical Association’s 2016 statement put the essential points plainly: a p-value does not measure the probability that the hypothesis is true, scientific conclusions should not rest only on whether it passes a threshold, and it does not measure the size or importance of an effect[4]. documented

Power: will the study find the effect if it is there?

A study can fail in two directions. It can declare an effect that is not there (a false positive, at rate α, the significance level), or miss one that is (a false negative, at rate β). Power is 1 − β, the probability of a significant result when a real effect of a given size exists. It depends on three things: the size of the effect relative to the noise, the sample size, and α.

Figure 2 Top: the distribution of the observed difference if there is no effect (grey) and if the true effect is δ (blue), with the rejection region beyond the dashed lines. Vermilion is α, the false-positive rate; blue beyond the lines is the power. Bottom: power against the true effect. Find the sample size that gives 80 % power for δ = 0.5.#

With 20 people per group and a true effect of half a standard deviation (a “medium” effect in Jacob Cohen’s much-quoted scale[5]), power is only about 35 %: the study misses a real effect almost two times in three. To reach the conventional 80 % you need 63 people per group with a z-test, 64 with a t-test. For a small effect of 0.2 standard deviations it takes about 400 per group. Button and colleagues estimated that the median power of neuroscience studies was between about 8 % and 31 %[6]. documented

Low power does more damage than wasted effort. When an underpowered study does reach significance, it is usually because the noise happened to inflate the effect, so the published estimate is too large. Look at the top panel with n = 10 and δ = 0.3: the only differences big enough to be significant are far larger than the true one. This is sometimes called the winner’s curse.

Many tests, many false positives

A 5 % false-positive rate is a property of one test. Run twenty independent tests of hypotheses that are all false, and the chance that at least one comes out significant is 1 − 0.9520 = 64 %.

Figure 3 Each dot is the p-value of one test in a study where none of the effects is real; dots in the shaded region pass the threshold. Run several studies, raise the number of tests, and switch on the Bonferroni correction.#

The simplest defence is the Bonferroni correction: with m tests, use α/m as the threshold for each, which keeps the chance of any false positive at or below α. It is conservative, and it costs power, which is why fields that run thousands of tests at once, such as genomics, use methods that control the false discovery rate instead[7]. The harder problem is tests that are never reported. A researcher who tries several outcome measures, several subgroups or several ways of handling outliers, and reports the one that worked, has run many tests while reporting one. Simmons, Nelson and Simonsohn showed by simulation that combining a few such choices can push the real false-positive rate above 50 %[8]. established

Checking the data as it arrives

One kind of hidden multiplicity is especially tempting: testing after every few participants and stopping as soon as p < 0.05. Each look is another chance for noise to cross the line, and the process never looks again after it does.

Figure 4 Thirty experiments in a world with no effect. Each line follows one experiment’s p-value as participants are added; the vermilion lines dip below 0.05 at some look and would have been stopped and reported. Look less often, or only once, and watch the false-positive rate fall back to 5 %.#

With a look after every 10 people per group up to 200, the false-positive rate is about 25 %, five times the nominal level, and it keeps rising with more frequent looks. Armitage, McPherson and Rowe worked out these rates in 1969[9]. Clinical trials still look at accumulating data, for good ethical reasons, but they do it with pre-planned interim analyses and stricter thresholds at each look, so that the overall false-positive rate stays at the stated level. established

What a significant result is worth

Here is the question people usually want a p-value to answer: given that my result is significant, how likely is it to be real? The p-value cannot answer it alone, because the answer depends on how many of the hypotheses being tested were true in the first place. John Ioannidis made the argument with arithmetic in 2005[10].

Figure 5 One thousand hypotheses, each a square. Set the share that are true and the power of the studies. Green squares are real effects that were found; vermilion squares are false positives. The share of green among all significant findings is the positive predictive value.#

If one tested hypothesis in ten is true and studies have 80 % power, then out of 1,000 hypotheses the 100 true ones yield 80 significant results and the 900 false ones yield 45. Only 64 % of the significant findings are real. Drop the power to 20 %, typical of the neuroscience estimates above, and it falls to 31 %. This positive predictive value, PPV = (1 − β)π / ((1 − β)π + α(1 − π)), is why a single significant result in a field that tests many long-shot hypotheses is weak evidence, and why replication matters. When the Open Science Collaboration repeated 100 published psychology experiments, 97 % of the originals had reported significant results and 36 % of the replications did[11]. documented

What a p-value is not

Greenland and colleagues catalogued two dozen common misinterpretations[12]. Four of them cause most of the damage, and every figure on this page is a way of seeing why each is wrong.

  • It is not the probability that the null hypothesis is true. It is computed by assuming the null is true. The probability that the null is true given the data needs the prior share of true hypotheses, as the last figure shows.
  • It is not the probability that the result is due to chance. In the null world the result is entirely due to chance by construction; the p-value measures how unusual it is there.
  • p > 0.05 is not evidence of no effect. With low power, a real effect usually produces a non-significant result. Report the confidence interval: a wide interval that includes zero says “not enough data”, not “no effect”.
  • A small p is not a large effect. With a large enough sample, a trivially small difference gives an arbitrarily small p-value. The size of the effect, with its interval, is the thing to report.

After significance testing

The replication results above set off a reform movement that is still under way. Its main proposals are mostly about practice rather than mathematics: register the hypothesis, the outcome and the analysis before collecting data, so that the forking paths are closed in advance; report effect sizes with their confidence intervals, as Chapter 5 recommends; and treat a single significant result as a reason to replicate, not as a finding. Some researchers go further and replace p-values with Bayes factors or full Bayesian analyses, which state how strongly the data favour one hypothesis over another, at the cost of specifying what each hypothesis predicts. The 72 authors who proposed the 0.005 threshold argued for it as one simple change that raises the bar for claims of new discoveries, alongside these reforms rather than in place of them[3]. active

Used for what it measures, the p-value is a useful alarm: it tells you when the data are hard to explain by chance alone under a stated model. Everything on this site, from the standard deviation to the null world, has been about the same underlying fact: you only ever see a sample, and the honest answer to any question about the population comes with a statement of how much a different sample could have changed it.