Probability

The Analysis of Data, volume 1

Random Variables: The Moment Generating Function

2.4. Moments and the Moment Generating Function

Definition 2.4.1. The $k$-moment of an RV $X$ is $\E(X^k)$.
Definition 2.4.2. The moment generating function (mgf) of the RV $X$ is the function \[m:\mathbb{R}\to\mathbb{R},\qquad m(t) = \E(\exp(tX)).\]

The name mgf stems from the fact that its derivatives, evaluated at 0 produce the moments.

Proposition 2.4.1. \[ m^{(k)}(0) = \E(X^k)\]
Proof. Using the Taylor series expansion (Section D.2) of $e^{z}$ around 0, we have \begin{align*} m'(0) &= \frac{d}{dt}\E\left(1+tX+\frac{t^2X^2}{2!}+\cdots\right)\Big|_{t=0} = \E\left(\frac{d}{dt}\left(1+tX+\frac{t^2X^2}{2!}+\cdots\right)\right)\Big|_{t=0} \\&= \E(X).\end{align*} Similarly, the second derivative produces the second moment \begin{align*} m''(0) &= \frac{d^2}{dt^2}\E\left(1+tX+\frac{t^2X^2}{2!}+\cdots\right)\Big|_{t=0} \\ &= \E\left(\frac{d^2}{dt^2}\left(1+tX+\frac{t^2X^2}{2!}+\cdots\right)\right)\Big|_{t=0} \\ &= \E(X^2), \end{align*} and so on for the higher-order derivatives.

In addition to producing the moments of $X$, the mgf is useful in identifying the distribution of $X$.

Proposition 2.4.2. Let $X_1,X_2$ be two RVs with mgfs $m_1,m_2$. If $m_1(t) = m_2(t)$ for all $t\in(-\epsilon,+\epsilon)$, for some $\epsilon>0$, then the two RVs have identical cdfs (and therefore identical pdfs or pmfs).
Proof*. According to a result from complex analysis, an analytic function (a function determined locally by its Taylor series) has a unique extension, called the analytics continuation, from $(-\epsilon,\epsilon)$ to the right half of the complex plane. Since the moment generating function $m(t)$ agrees with the characteristic function $\phi(-it)$ for $t\in\R$, and the characteristic function uniquely determines the distribution (see Section 8.7 for a description of the characteristic function and a proof that it uniquely determines the distribution), the moment generating function uniquely determines the distribution as well.

The moments of an RV $X$, provided they exist, completely characterize the Taylor series of the mgf and, therefore, characterize the mgf. Together with the above proposition, the moments $\E(X^k), k\in\mathbb{N}$ (assuming the corresponding integrals or sums converge) completely characterize the distribution of $X$ and the corresponding cdf, pdf, and pmf.