What averaging does to a distribution

The central limit theorem says that if you take many independent samples of size n from almost any population and compute each sample’s mean, those means are approximately normally distributed, centred on the population mean μ, with a standard deviation of σ/√n. The population itself can be skewed, flat or lumpy. The approximation improves as n grows, quickly for symmetric populations and slowly for skewed ones.

This is the reason the normal curve matters so much. Most of the statistics people use (confidence intervals, t-tests, margins of error in polls) are statements about sample means or sums, and the central limit theorem is what lets those statements use the normal distribution even when the data are not normal. It is also the hardest idea in an introductory course, because it is about a distribution you never observe: the distribution of a statistic over all the samples you could have drawn. The figures in this chapter let you draw them.

The distribution you never see§

Three distributions are in play, and keeping them apart is most of the battle. The population distribution is the shape of all the values out there: every household’s income, every bolt’s length. The sample is the handful you actually measured. The sampling distribution of the mean is the shape you would get if you could repeat the whole study thousands of times and plot the mean from each repetition. In practice you do the study once and get one mean. The sampling distribution describes how far that one mean is likely to be from the truth^[1].

The lab at the top of this page does what is impossible in practice: it shows you the population, and repeats the study thousands of times. It draws all three distributions, one panel each, on the same horizontal scale and each with its own count. Go back up to it as you read.

Watch the middle panel first. It is one sample, the only thing you would see in real life. At n = 5 it is five scattered dots. At n = 100 it fills in the dashed outline of the population, long right tail included: a bigger sample is a better picture of the population, so it looks more like it, never more normal. Its standard deviation hovers around σ = 14.6 whatever n is. Now compare the bottom panel. The means of 4,000 such samples narrow as n grows and lose the population’s shape. The 4,000 repetitions only make that histogram smooth enough to read; it is n that narrows it. established

At n = 1 each “mean” is a single draw, so the bottom panel copies the population. At n = 5 from the skewed population the means already form a single hump, still leaning right. By n = 30 the hump is narrow and nearly symmetric, and the normal curve lies on top of it. Now sculpt something perverse: two tall spikes at the ends and nothing in the middle. At n = 2 the means form three humps, because two draws can both be low, both high, or one of each. By n = 10 they form a bell.

The width: σ/√n§

Two facts about the sampling distribution of the mean hold exactly, for any sample size and any population with a finite variance:

The first says the sample mean is aimed at the right target. The second, the square-root law, follows from the fact that variances of independent quantities add: the sum of n draws has variance nσ², and dividing the sum by n divides the variance by n², leaving σ²/n^[2]. The standard deviation of a statistic has its own name, the standard error, so equation (4.1) says the standard error of the mean is σ/√n. Neither fact needs the central limit theorem. What the theorem adds is the shape. established

The square root is the most expensive fact in statistics. Quadrupling a sample halves its standard error; a hundredfold larger sample is needed to cut it tenfold. That is why national polls of about a thousand people are common and polls of a hundred thousand are rare: a thousand respondents give a margin of error of about three percentage points for a proportion near one half, and the hundred thousand would only bring it down to about 0.3 points, while the errors that sampling theory does not cover, such as who agrees to answer, stay as large as they were^[1].

Which hospital sees more lopsided days?§

Here is a question that Amos Tversky and Daniel Kahneman posed in 1974. Predict before you look.

Most of the people Tversky and Kahneman asked said the two hospitals would record about the same number of such days^[3]. The square-root law says otherwise: small samples vary more. The exact answer comes from the binomial distribution of Chapter 3: more than 60 % of 15 means at least 10 boys, which happens on 15.1 % of days, about 55 days a year. More than 60 % of 45 means at least 28 boys, which happens on 6.8 % of days, about 25 a year. documented

Tversky and Kahneman called the mistake “insensitivity to sample size”. It shows up wherever small groups are ranked: the counties with the highest and the lowest rates of a rare cancer tend to be small rural counties, the schools with the best and the worst average scores tend to be small schools. Small groups are more variable, so they crowd both ends of any league table; Howard Wainer called the square-root law “the most dangerous equation” for how often it is forgotten^[4]. The square-root law predicts exactly how much more variable: the proportion of boys on one day has a standard error of √(0.25/15) = 12.9 % in the small hospital and √(0.25/45) = 7.5 % in the large one.

What the theorem says, and how fast it works§

Stated carefully, the classical central limit theorem says: if X₁, …, X_n are independent draws from a distribution with mean μ and finite standard deviation σ, then the standardized mean

has a distribution that approaches the standard normal as n grows: for every z, P(Z_n ≤ z) → Φ(z)^[5]. Laplace proved a version of it around 1810^[6]; Lyapunov gave a rigorous general proof in 1901, and Lindeberg and Lévy completed the classical form in the 1920s^[7]. documented

The theorem says nothing about how large n must be, and the popular rule “30 is enough” has no theoretical basis. What controls the speed is mostly the population’s skewness γ₁. The skewness of the sample mean is exactly γ₁/√n^[8], so it fades like the standard error itself. A symmetric population such as the flat one gives near-normal means by n = 5. An exponential population, with skewness 2, needs n = 100 just to bring the means’ skewness down to 0.2. Hesterberg uses this to argue that the rule of thumb is too lax for skewed data, which is common in practice: incomes, waiting times, hospital stays^[8]. The lab reports both the simulated skewness of the means and the exact value, so you can see it fall. established

When averaging does not help§

The finite variance condition is not a technicality. The Cauchy distribution is symmetric and bell-shaped at a glance, but its tails are so heavy that it has no mean and no variance: the integral that would define the mean does not converge. Its average misbehaves completely, as the next figure shows.

For the Cauchy distribution the mean of n draws has exactly the same distribution as one draw, for every n^[2]. Averaging a thousand values tells you no more about the centre than looking at one. The median of the draws does converge, which is one reason robust statistics prefer medians when the tails may be heavy. Real data with infinite variance are unusual, but data whose variance is finite and enormous, such as insurance losses or file sizes, behave like the Cauchy for any sample you can afford: the central limit theorem holds, and needs a sample size nobody has. established

The next chapter turns the sampling distribution around. If you know how far a sample mean typically falls from the truth, you can say how far the truth is likely to be from the sample mean you got. That is a confidence interval.