The Sampling Lab

Chapter 5 Estimation

What a 95 % confidence interval promises

The 95 % belongs to the method, not to any one interval: run the method a thousand times and count.

8 min read · 5 interactive figures

A confidence interval calculator that shows its work. Paste your data, or enter a sample mean, standard deviation and size. The steps give the standard error, the degrees of freedom, the t critical value, the margin of error and the interval.#
On this page
  1. The method, and what 95 % refers to
  2. Why t and not z?
  3. Calculate a confidence interval
  4. When the promised coverage is not delivered
  5. Where the argument goes next

A 95 % confidence interval is a range computed from a sample by a method that, used over and over on new samples, captures the true population value 95 % of the time. For a mean, the usual method is x ± t* · s/√n. The 95 % describes the method’s long-run success rate. It is not the probability that any one computed interval contains the truth, a distinction most people, including many researchers, get wrong.

Everything in the previous chapter was about the forward problem: given the population, how do sample means behave? A confidence interval runs the argument backwards. If sample means usually land within about two standard errors of μ, then μ is usually within about two standard errors of the sample mean. The word “usually” is doing all the work, and this chapter is about what it means.

The method, and what 95 % refers to

Jerzy Neyman set out the theory in 1937. His idea was to judge an interval-making procedure by how often it succeeds across all the samples it could be applied to, and to choose procedures that succeed with a stated frequency whatever the true value is[1]. For the mean of a normal population the procedure is

x ± t*n−1 · s ⁄ √n(5.1)

where s/√n is the estimated standard error and t*n−1 is a critical value from Student’s t distribution with n − 1 degrees of freedom[2]. Many courses teach a simpler recipe first, x ± 1.96 · s/√n, with the normal curve’s 1.96 in place of t*. Before watching either recipe at work, a prediction.

Question 1 Prediction before the interval rain.#
Figure 1 Each bar is a confidence interval from a fresh sample of n values; the black tick is that sample’s mean. Bars that miss the dashed true mean turn vermilion. The plot below tracks the share of intervals that caught the truth. Change the method to “z with s” at n = 5, then try the skewed population.#

Watch the coverage line. After 20 intervals it might read 85 % or 100 %; after 1,000 it sits close to 95 %. That long-run frequency is the only thing the confidence level promises. Any single bar either crosses the dashed line or it does not, and in real life, where the dashed line is invisible, you cannot tell which kind of bar you are holding.

Hoekstra and colleagues gave researchers and students six statements about one reported interval, all of them false, including the statement that there is a 95 % probability that the true mean lies inside the reported interval. Large majorities of both groups endorsed at least one, and researchers did not do much better than first-year students[3]. Greenland and coauthors list the same mistake among the most common misinterpretations of statistical results[4]. documented

Why t and not z?

If you knew the population standard deviation σ, the interval x ± 1.96 σ/√n would catch the mean of a normal population exactly 95 % of the time. You almost never know σ, and replacing it with the sample’s s adds a second source of randomness: in some samples s comes out too small, the interval comes out too narrow, and it misses. That is what the prediction above was about: with “z with s” and n = 5 the coverage settles near 88 %, not 95 %.

William Sealy Gosset worked out the fix while employed as a brewer at the Guinness brewery in Dublin, where experiments on barley and yeast came in small samples. Guinness let him publish on condition that he use a pseudonym, and his 1908 paper “The probable error of a mean”, signed “Student”, derived the distribution of (x − μ)/(s/√n) for normal data[5][6]. Its tails are heavier than the normal’s, so its critical values are larger, by just enough to restore exact coverage: 2.262 for a 95 % interval from 10 values, falling toward the normal’s 1.960 as n grows[2]. documented

Figure 2 The t distribution for n − 1 degrees of freedom against the dashed normal curve. The shaded tails are outside ±t*. Lower n to 2 and the tails swell; raise it to 60 and the two curves almost coincide.#

At n = 2 the 95 % critical value is 12.71, so an interval from two observations is enormous, as it should be: two values tell you very little about spread. By n = 30, t* is 2.045, within 5 % of 1.96, which is where the habit of switching to z for “large samples” comes from. With software there is no reason to switch; the t interval is correct at every sample size for normal data, and the calculator below uses it.

Calculate a confidence interval

The calculator at the top of this page does the arithmetic of equation (5.1) on your own numbers and shows every step: the standard error, the degrees of freedom, the t critical value, the margin of error and the interval. It accepts raw data or a summary (mean, standard deviation and size).

The margin of error, t* · s/√n, also tells you how to plan a study. To estimate a mean to within ±E with 95 % confidence you need roughly n = (1.96 σ/E)² observations, using a guess for σ from earlier work. Halving the margin quadruples the sample, the square-root law again.

When the promised coverage is not delivered

The t interval is exact only for normal populations. The central limit theorem makes it approximately right for others, but “approximately” depends on the skewness, as Chapter 4 showed. Set the rain to the skewed population with n = 10. Coverage settles visibly below 95 %, and the misses are lopsided: most of the failed intervals lie entirely below the true mean. In a right-skewed population, samples that happen to miss the long tail have both a low mean and a small standard deviation, so their intervals are low and narrow at once. Hesterberg recommends bootstrap or transformation methods for skewed data rather than trusting n = 30[7].

Intervals for a proportion

The most common interval in the news is for a proportion: 52 % of voters, 3 % of parts defective. The textbook recipe, called the Wald interval, is p ± 1.96 √(p(1 − p)/n). It looks like equation (5.1), but its coverage is surprisingly bad. Because a count of successes is a whole number, the coverage at a given true p is a sum over the possible counts, and it can be computed exactly.

Figure 3 Exact coverage of two 95 % intervals for a proportion, for every true p, with n trials. The Wald interval saws far below the dashed nominal line, worst near 0 and 1; Wilson’s interval stays close. Increase n and watch how slowly Wald improves.#

The jagged curve is not a simulation artifact; it is exact, and its sawtooth comes from the discreteness of the counts. Brown, Cai and DasGupta showed in 2001 that the Wald interval’s coverage is erratic even for large samples and recommended the score interval Edwin Wilson had published in 1927, or for larger n the simple adjustment of Agresti and Coull, which adds two successes and two failures before applying the Wald formula[8][9][10]. Near p = 0 the Wald interval fails completely: if no successes are observed it collapses to the single point 0, which can never contain a true p above zero. established

Where the argument goes next

Two developments are worth knowing about. The first is the push to report intervals instead of significance verdicts. An interval shows both the size of an effect and how precisely it was measured, while “significant” or “not significant” throws the size away, and the American Statistical Association’s 2016 statement on p-values warned against resting conclusions on whether a p-value passes a threshold and pointed to methods that emphasize estimation, such as confidence intervals, instead[11]. The second is the Bayesian credible interval described in the box above, which answers the question people usually mean to ask (where is the mean, probably?) at the price of stating a prior. For simple problems with little prior information the two intervals nearly coincide, and the disagreement is about what the numbers mean rather than what they are. active

Confidence intervals and hypothesis tests are two views of the same calculation. An interval is the set of parameter values a test would not reject; a test asks whether one particular value, usually zero, is outside the interval. The last chapter takes the test’s point of view.