Aug 30, 2026
If you have had a course in statistics you have learned about turning a bunch of number x_1, \ldots, x_n into a couple of numbers that the pound and a half of grey matter on top of your chimpanzee brain can use to make up a plausible story to tell
The first thing you want to know is most
A random variable X is a variable, a symbol taking the place of a number in an equation or inequation, with additional information about the probability of the values it can have.
A discrete random variable can take on only values X = x_j with probability P(X = x_j) = p_j where p_j > 0 and \sum_j p_j = 1.
A continuously distributed random variable has a density function f(x)\ge0 where the probablity X is less than or equal to x is {P(X\le x) = \int_{-\infty}^x f(t)\,dt} and {\int_{-\infty}^\infty f(t)\,dt = 1}.
Every random variable is completely determined by its cumulative distribution function F(x) = P(X\le x) where F(x)\le F(y) if x < y, 0\le F(x) \le 1, \lim_{x\to-\infty}F(x) = 0, and \lim_{x\to\infty} F(x) = 1
Exercise. Show if X is discrete then F(x) = \sum_{x_j\le x} p_j is a cdf.
Exercise. Show if X is continuously distributed then {F(x) = \int_{-\infty}^x f(t)\,dt} is a cdf.
Exercise. Show every cdf is right continuous.
Hint: \cap_{y>x} [y,\infty) = [x,\infty).
Exercise. Show every cdf has left limits.
Hint: \cup_{w<x} (-\infty,w] = (-\infty,x).
Statistics is the study of finding estimators for a statistic.
A statistic is a function s from random variables to the real numbers \bm{R}. Estimators are collection of functions s_n\colon\bm{R}^n\to\bm{R}, where n\in\bm{N} is a natural number. Given independent (X_1,\ldots, X_n) having the same law as X how do we find estimators s_n such that the random variable s_n(X_1,\ldots,X_n) approximates the number s(X) as n gets large?
For example, if s(X) = E[X] is the expected value of X, then the arithmetic means m_n(x_1,\ldots,x_n) = (x_1 + \cdots + x_n)/n are estimators. If M_n = m_n(X_1,\ldots,X_n) then E[M_n] = E[X] and \operatorname{Var}(M_n) = \operatorname{Var}(X)/n\to 0 as n\to\infty. We say the random variables S_n converge to the number s in mean squared when \lim_{n\to\infty}E[(S_n - s)^2] = 0.
If s_n are estimators let S_n = s_n(X_1,\ldots,X_n). Estimators with the property E[S_n] = s(X), n\in\bm{N}, are unbiased.
We could also use the geometric means g_n(x_1,\ldots,x_n) = \sqrt[n]{x_1\cdots x_n} as estimators for E[X] if X is positive. By Jensen’s Inequality E[\sqrt[n]{X_1\cdots X_n}] \le E[X] so the geometric means are not unbiased.
Clearly E[M_n\mid X_1 = x_1, \ldots, X_n = x_n] = E[M_n\mid M_n = m_n(x_1, \ldots, x_n)] since both are equal to m_n(x_1, \ldots, x_n). If s_n and t_n are estimators with the property E[S_n\mid X_1 = x_1, \ldots, X_n = x_n] = E[S_n\mid S_n = t_n(x_1,\ldots,x_n)] then t_n are sufficient for s_n.
Some statistics are better than other statistics.
If E[s_n(X_1,\ldots,X_n)] = \sigma we say s_n is an unbiased estimator of \sigma. The arithmetic mean is an unbiased estimator of the mean. Since E[(X_1\cdots X_n)^{1/n}] \le E[X_1\cdots X_n]^{1/n} = E[X] the geometric mean is biased.
An unbiased statistic s_n is efficient if it has the smallest variance among all unbiased statistics. This leaves open the possibility of biased statistics that have lower variance than efficient statistics.
A statistic s\colon\bm{R}^n\to\bm{R} is complete if E_\theta[g(s(X_1,\ldots,X_n))] = 0 for all \theta implies g(s(X_1,\ldots,X_n)) = 0 a.s.
A statistic t\colon\bm{R}^n\to\bm{R} is sufficient if the n conditions X_j = x_j, 1\le j\le n can be replaced by one condition t(X_1,\ldots,X_n) = t(x_1,\ldots,x_n), i.e., E_\theta[g(X)\mid X_j = x_j] = E_\theta[g(X)\mid s(X_1,\ldots,X_n) = s(x_1,\ldots,x_n)].
Given two statistics s and t for \sigma where t is sufficient let \delta(X) = E[s(X)\mid t(X)] be the improved estimator. The following theorem justifies this name.
Theorem. (Rao–Blackwell–Kolmogorov) E[(\delta(X) - \sigma)^2] \le E[(s(X) - \sigma)^2].
Proof. We have $
Theorem. (Lehmann–Scheffé) If ….
Sampling from a population is not special. One always does this.
Given random variables X_\theta, \theta\in\Theta, and a partion \{\Theta_0,\Theta_1\} of \Theta how can we decide if \theta\in\Theta_0 (null hypothesis) or \theta\in\Theta_1 (alternate hypothesis)?
This is done by designing tests and collecting data samples. A test is a subset \delta_0\subseteq \bm{R}^n and is called the critical region. A data sample is a collection of numbers x = (x_1,\ldots,x_n)\in\bm{R}^n. We accept the null hypothesis if the sample belongs to this set.
Let X = (X_1,\ldots,X_n) be iid random variables with the same law as X_\theta. The power function of a test \delta_0 is \pi(\theta) = P(X\in\delta\mid\theta). If \pi = 1_{\Theta_1} then the test determines whether or not \theta\in\Theta_0 with probability 1.
Exercise. Show if \pi = 1_{\Theta_1} then x\in\delta_0 implies P(\theta\in\Theta_0) = 1.
Hint: If x\in\delta_0 then $
This is usually not possible so we look for tests that approximate the indicator function of \Theta_1. The size of a test \delta is \alpha = \sup_{\theta\in\Theta_0} \pi(\theta).