Probability

The Analysis of Data, volume 1

The Characteristic Function

8.7. The Characteristic Function

We use in this section the following standard results associated with complex numbers. The notation $i$ denotes the complex number $i=\sqrt{-1}$ and $z$ refers to the complex number $z=a+ib, a,b\in\R$.

The notation $\mathbb{C}$ corresponds to the set of all complex numbers.

Definition 8.7.1. The characteristic function associated with the random vector $\bb X$ is the following function \[ \phi:\R^d\to \mathbb{C}, \qquad \phi_{\bb X}(\bb t) = \E(\exp(i{\bb t}^{\top}\bb X))\] where the expectation is taken with respect to the distribution of $\bb X$. We sometimes omit the index $\bb X$ and simply denote the characteristic function by $\phi(\bb t)$.
Proposition 8.7.1. Let $\phi$ be the characteristic function of a random vector $\bb X$. Then,
  1. $\phi(\bb 0)=1$.
  2. $|\phi(\bb t)|\leq 1$.
  3. $\phi(\bb t)$ is a continuous function.
  4. $\phi_{-\bb X}(\bb t)=\overline{\phi_{\bb X}(\bb t)}$.
  5. $\phi_{a\bb X+\bb b}(\bb t)= e^{i{\bb t}^{\top}\bb b} \phi_{\bb X}(a\bb t)$.
  6. If ${\bb X}^{(n)}, n=1,\ldots,N$ are independent RVs, then $\phi_{\sum_{n=1}^N {\bb X}^{(n)}}(\bb t) = \prod_{n=1}^N \phi_{{\bb X}^{(n)}}(\bb t)$.
  7. If $\E\|\bb X\|<\infty$ then $\nabla \phi=i \E(\bb X)$.
  8. If $\E\|\bb X\|^2<\infty$ then $\nabla^2 \phi=-\E({\bb X}{\bb X}^{\top})$.
Proof. Statement 1 follows from $\phi(\bb 0)=\E(1)=1$. Statement 2 follows from \begin{align*} |\exp(ix)|&=|\cos x+i\sin x|=\cos^2x + i^2\sin^2(x)=\cos^2x -\sin^2(x)\\ &\leq \cos^2 x+\sin^2 x=1 \end{align*} and $|\E(X)|\leq \sup |X| \E(1)=\sup|X|$. To prove statement 3, note that if $\bb t\to \bb r$ then $\exp(i{\bb t}^{\top}\bb x)\to \exp(i{\bb r}^{\top}\bb x)$, which implies by Proposition 8.3.1 that \begin{align*} \phi(\bb t)-\phi(\bb r) &=\E(\exp(i{\bb t}^{\top}X) - \exp(i{\bb r}^{\top}\bb X)) \\ &\leq \E(\|\exp(i{\bb t}^{\top}X) - \exp(i{\bb r}^{\top}\bb X)\|) \to 0. \end{align*} Statement 4 follows from the following change of integration measure (see Proposition F.3.12): \begin{align*} \phi_{-\bb X}(\bb t) &= \int \exp(i{\bb t}^{\top}\bb x)\, dF_{-\bb X}(\bb x) = \int \exp(-i{\bb t}^{\top}\bb x)\, dF_{\bb X}(\bb x). \end{align*} Statement 5 follows from the change of integration measure (see Proposition F.3.12): \begin{align*} \phi_{a\bb X+\bb b}(\bb t) &= \int \exp(i{\bb t}^{\top}\bb x)\, dF_{a\bb X+\bb b}(\bb x) %= \int \exp(i{\bb t}^{\top}(a\bb X+\bb b))\, dF_{a\bb X+\bb b}(a\bb X+\bb b)\\ = \int \exp(i{\bb t}^{\top}(a\bb X+\bb b))\, dF_{\bb X}(\bb X)\\ &=\exp(i{\bb t}^{\top}\bb b) \int \exp(i{a\bb t}^{\top}\bb X)\, dF_{\bb X}(\bb X). \end{align*} The proof of statement 5 is similar to the proof of Proposition 4.8.1. The last two statement can be proven by expanding the exponential in $\phi$ using a Taylor series expansion, differentiating the Taylor series term by term at $\bb t=\bb 0$, and noting that only the leading term remains.

Note that part 2 of the proposition above implies that the characteristic function always exists.

Proposition 8.7.2. The characteristic function of a $N(\bb \mu,\Sigma)$ random vector is \[\phi(\bb t)=\exp(i{\bb t}^{\top}\bb\mu - {\bb t}^{\top}\Sigma\bb t/2).\]
Proof. We start by showing that in the univariate case $\phi(t)=\exp(-t^2/2)$. \begin{align*} \phi(t) &= \E \exp(itx) = \E(\cos tx + i \sin tx) = \E(\cos tx) + 0\\ &= \int \cos(tx)\exp(-x^2)/\sqrt{2\pi}\,dx. \end{align*} where the third equality above holds since the $\sin(tx)\exp(-tx^2)$ is an even function ($f(x)=-f(-x)$) whose integral is zero. Differentiating $\phi$ with respect to $t$, we have \begin{align*} \phi'(t) &= -\int x \sin(tx)\exp(-x^2)/\sqrt{2\pi}\,dx \\ &= \int \sin(tx)\frac{d\exp(-x^2)/\sqrt{2\pi}}{dx} \,dx \\ &= \sin(tx)\exp(-x^2)/\sqrt{2\pi} \Big|_{-\infty}^{\infty} - \int t\cos(tx)\exp(-x^2)/\sqrt{2\pi} \, dx \\ &= 0- \int t\cos(tx)\exp(-x^2)/\sqrt{2\pi} \, dx \\ &= -t\phi(t). \end{align*} We have thus obtained a differential equation $\phi'(t)=-t\phi(t)$ (subject to the initial condition $\phi(0)=\E(1)=1$) whose only solution is $\phi(t)=\exp(-t^2/2)$. An alternative proof uses the completion of the square method as in Proposition 3.9.2.

Part 6 of the proposition above shows that \[\text{if}\quad \bb X\sim N(\bb 0,I) \quad \text{then}\quad \phi_{\bb X}(\bb t)= \exp(-{\bb t}^{\top}\bb t/2).\]

Given an arbitrary $\bb X\sim N(\bb \mu,\Sigma)$ we can use the transformation $\bb X=\Sigma^{1/2}\bb Y+\bb \mu$ where $\bb Y\sim N(\bb 0,I)$ (as in Proposition 5.2.3). This yields \begin{align*} \E(\exp(i{\bb t}^{\top} \bb X)) &= \E\left(\exp\left(i{\bb t}^{\top}(\Sigma^{1/2}\bb Y+\bb \mu) \right)\right)\\ &= \exp(i{\bb t}^{\top}\bb \mu) \E \left(\exp(i(\Sigma^{1/2}\bb t)^{\top}\bb Y)\right) \\ &= \exp(i{\bb t}^{\top}\bb \mu) \E \left(\exp(i\bb u^{\top}\bb Y)\right) \\ &= \exp(i{\bb t}^{\top}\bb \mu) \exp\left(-{\bb u}^{\top}\bb u/2\right) \\ &= \exp(i{\bb t}^{\top}\bb \mu) \exp\left(-(\Sigma^{1/2}\bb t)^{\top} (\Sigma^{1/2}\bb t) /2\right) \\ &= \exp (i{\bb t}^{\top}\bb \mu- \bb t^{\top} \Sigma \bb t\, /2). \end{align*}

Lemma 8.7.1. Assuming that $a > 0$, \[ \int \exp(-a x^2 + bx)\,dx= \sqrt{\frac{\pi}{a}} \exp({b^2/(4a)}).\]
Proof. Writing $-ax^2+bx=-a(x-b/(2a))^2+b^2/(4a)$, we have \begin{align*} \int e^{-a x^2 + bx}\,dx &= e^{b^2/(4a)}\int e^{-a(x-b/(2a))^2}\, dx = e^{b^2/(4a)} \sqrt{\pi/a}, \end{align*} where the last equality follows from the fact that the Gaussian pdf integrates to one.