The Sampling Lab

Chapter 1 Spread

Standard deviation, drawn as squares

The typical distance of a value from its mean, drawn as squares you can drag.

10 min read · 5 interactive figures

Each dot is one value, and its square has the value’s distance from the mean as its side. Drag a dot (the 9 out to 15, say), or select it and use the slider. The large square beside the axis, below it on phones, is the average square: its area is the variance and its side is the standard deviation.#
On this page
  1. Deviations, drawn as squares
  2. Calculate a standard deviation, step by step
  3. Why divide by n − 1?
  4. Reading a standard deviation
  5. Where the name comes from

A standard deviation tells you how far, typically, the values in a set of data sit from their mean. For the eight numbers 2, 4, 4, 4, 5, 5, 7 and 9 the mean is 5, and the standard deviation is 2 if those eight are the whole population, or 2.14 if they are a sample from something larger. This page shows where both numbers come from, why the recipe squares things, and why the sample version divides by n − 1 instead of n.

Two classes can have the same average test score and very different spreads of marks. In one, everyone scored between 70 and 80. In the other, half the class scored 50 and half scored 100. The mean, 75, describes both. What separates them is spread, and the standard deviation is the number statisticians have used to measure spread for more than a century[1]. It is also the yardstick for almost everything else on this site: z-scores count standard deviations, the normal curve is drawn in them, and the precision of an average is a standard deviation divided by √n.

Deviations, drawn as squares

Start with the most direct idea. Every value has a deviation: its distance from the mean, xi − x, positive above the mean and negative below. A natural measure of spread would be the average deviation, but that average is always exactly zero. The mean is the balance point of the data, so the deviations above it cancel the deviations below it.

There are two ways out. You can ignore the signs and average the absolute distances, or you can square each deviation, which makes it positive, and average the squares. The standard deviation takes the second route. The figure at the top of the page draws the squares literally: each value’s deviation is the bottom edge of a square whose other edge is the same length, on the same scale.

Try three things in that figure. First, drag the 9 out to 15. One square grows enormously, and the sample standard deviation jumps from 2.14 to almost 4.0, even though seven of the eight values did not move. Squaring gives large deviations a large say, which is the price of this measure: one wild value can dominate it. Second, drag all the dots close together and watch every square shrink to nothing. Third, switch the divisor between n and n − 1 and notice that the average square gets a little larger when it is shared among one fewer.

Written as a formula, the figure computes

s = √Σ(xi − x)² ⁄ (n − 1)sample
σ = √Σ(xi − μ)² ⁄ Npopulation(1.1)

The left form, with n − 1, is the sample standard deviation s, used when your data are a sample and you want to estimate the spread of the population they came from. The right form is the population standard deviation σ, used when the data are the entire population, every member of it, and the mean is the true mean μ[2]. The quantity inside the square root is the variance: s² or σ². The square root brings the answer back to the units of the data. If the data are heights in centimeters, the variance is in square centimeters, an awkward unit for describing people, and the standard deviation is in centimeters again.

Squaring is a choice, and the alternative is a perfectly good statistic. The mean absolute deviation of 2, 4, 4, 4, 5, 5, 7, 9 is 1.5, smaller than the standard deviation of 2 because it gives the big deviations no extra weight. Squares won for mathematical reasons: they make the algebra of sums work out cleanly, as you will see in the section on n − 1, and they are the natural scale of the normal distribution. Absolute deviations are still the better choice when the data contain outliers you do not trust[3]. established

Calculate a standard deviation, step by step

The calculator below does the arithmetic of equation (1.1) on any numbers you paste in, and shows every intermediate value, so you can check a homework answer or see where your own calculation went wrong. It treats commas, spaces and new lines as separators.

Figure 1 Paste or type numbers. The table lists each deviation and its square; the steps below it show the sum, the divisor and the square root. Choose whether the numbers are a sample or a whole population. Try the two buttons: adding a constant leaves the standard deviation unchanged, doubling every value doubles it.#

The two buttons demonstrate the two rules that make standard deviations easy to reason about. Shifting every value by the same amount moves the mean but leaves every deviation, and therefore the standard deviation, unchanged: converting temperatures by adding 273.15 does not change their spread. Scaling every value by a factor a multiplies the standard deviation by |a| and the variance by a²: converting meters to centimeters multiplies the standard deviation by 100 and the variance by 10,000.

Why divide by n − 1?

Here is the part of the recipe that most people are told to memorize. If the variance is the average squared deviation, why does the sample version divide by n − 1 instead of by n? Before reading the answer, make a prediction.

Question 1 Prediction before the simulation.#

The clue is in the word own. The deviations in a sample are measured from the sample mean x, because the true mean μ is unknown. But x is a special point: of all possible centres, it is the one that makes the sum of squared deviations as small as it can be. Drag the purple centre in the next figure and watch the total area of the squares.

Figure 2 The same squares, measured from a centre c you choose. Drag the purple centre (or use its slider). The curve underneath plots the total area against c: a parabola whose lowest point sits exactly at the sample mean.#

Wherever you put c, the total area is the area about the mean plus a penalty that grows with the square of the distance between them:

Σ(xi − c)² = Σ(xi − x)² + n(x − c)²(1.2)

Now put the true mean μ in place of c. The squares about μ are what we would like to average, because on average each one is exactly σ²: their sum has expectation nσ². The squares about x fall short of that by n(x − μ)², and that shortfall is not zero on average. The sample mean misses μ by a typical amount whose square averages σ²/n (a fact that the chapter on sampling distributions is built around), so the shortfall averages n · σ²/n = σ². Taking expectations of equation (1.2),

E[ Σ(xi − x)² ] = nσ² − σ² = (n − 1)σ²(1.3)

So the sum of squares about the sample mean carries only n − 1 variances’ worth of information, and dividing by n − 1 makes the estimate right on average. This is Bessel’s correction, and the estimator s² is called unbiased[3]. Nothing in the argument assumed a bell curve. It needs only that the values are drawn independently from a population with a finite variance. established

Algebra is convincing, and an experiment is more so. The figure below draws thousands of samples, computes both estimates from each one, and keeps a running average of each.

Figure 3 Press Run. Each sample of n values (top) gives one variance estimate by each rule; the curves are their running averages. The divide-by-n average settles on the dotted line (n − 1)σ²/n; the divide-by-(n − 1) average settles on the true variance . Try n = 2 and n = 20, and the skewed population.#

With samples of 3, dividing by n underestimates the variance by a third. With samples of 20 the shortfall is 5 %, and with 1,000 it is a tenth of a percent, which is why the distinction matters most for small samples and hardly at all for large ones. The checkbox shows a third estimator, the average squared deviation from the true mean μ, divided by n. It is also right on average, because it does not use the data to choose its own centre. The trouble is that it needs μ, and if you knew μ you would not be estimating anything.

A common way to say all this is that the sample has n − 1 degrees of freedom. The n deviations from the sample mean must add to zero, so once you know any n − 1 of them, the last is determined. Only n − 1 of them carry independent information about spread.

Going further: even s is not quite right

Bessel’s correction makes the variance unbiased. The standard deviation is its square root, and the square root of an unbiased estimate is not unbiased: on average s still comes out a little smaller than σ. For samples from a normal population the average of s is c4(n) · σ, with c4 = 0.798 for n = 2, 0.940 for n = 5 and 0.990 for n = 25. Quality-control charts correct for it; most other work ignores it, because the bias is small next to the sampling noise in s itself. Unbiasedness is also only one virtue. For normal data, dividing the sum of squares by n + 1 gives the variance estimate with the smallest average squared error, at the cost of a small bias[3]. The n − 1 convention is the standard because it is unbiased, simple, and what the t distribution of Chapter 5 is built on.

Reading a standard deviation

A standard deviation on its own is a length, and the question is what fraction of the data lies within a few of those lengths of the mean. For data shaped like a bell, about 68 % of values lie within one standard deviation of the mean and about 95 % within two. That rule belongs to the normal distribution, and the next chapter shows how badly it can fail for other shapes. For any data at all there is a weaker guarantee, Chebyshev’s inequality: at least 1 − 1/k² of the values lie within k standard deviations of the mean. That is at least 75 % within two standard deviations, and at least 89 % within three, whatever the shape[3]. established

One distinction causes more confusion than any other in elementary statistics, so it is worth setting down now. The standard deviation describes how spread out the individual values are. The standard error describes how much a statistic, usually the sample mean, would vary if you repeated the whole study. For the mean of n independent values the standard error is σ/√n, estimated by s/√n. A larger sample does not make people less variable, so it does not shrink the standard deviation; it does shrink the standard error. The calculator reports both, and Chapter 4 is about the second.

Where the name comes from

Karl Pearson introduced the term “standard deviation” in a paper on the mathematical theory of evolution published by the Royal Society in 1894[1]. Before that, astronomers and geodesists measured the scatter of repeated observations with the “mean error” and, especially, the “probable error”: the half-width of the range that contains half of the errors[6]. For normally distributed errors the probable error is 0.6745 standard deviations, a figure you can confirm in the next chapter by asking for the middle 50 % of a normal curve. Pearson’s name, and his preference for the square-root-of-mean-square form, won out, and by the time Fisher wrote his Statistical Methods for Research Workers in 1925 the standard deviation was the unit in which tables of the normal distribution were printed[7]. documented