Probability
The Analysis of Data, volume 1
The Characteristic Function
$
\def\P{\mathsf{\sf P}}
\def\E{\mathsf{\sf E}}
\def\Var{\mathsf{\sf Var}}
\def\Cov{\mathsf{\sf Cov}}
\def\std{\mathsf{\sf std}}
\def\Cor{\mathsf{\sf Cor}}
\def\R{\mathbb{R}}
\def\c{\,|\,}
\def\bb{\boldsymbol}
\def\diag{\mathsf{\sf diag}}
\def\defeq{\stackrel{\tiny\text{def}}{=}}
\newcommand{\toop}{\xrightarrow{\scriptsize{\text{p}}}}
\newcommand{\tooas}{\xrightarrow{\scriptsize{\text{as}}}}
\newcommand{\tooas}{\xrightarrow{\scriptsize{\text{as}}}}
\newcommand{\tooas}{\xrightarrow{\scriptsize{\text{as}}}}
\newcommand{\tooas}{\xrightarrow{\scriptsize{\text{as}}}}
\newcommand{\tooas}{\xrightarrow{\scriptsize{\text{as}}}}
\newcommand{\tood}{\rightsquigarrow}
\newcommand{\iid}{\mbox{$\;\stackrel{\mbox{\tiny iid}}{\sim}\;$}}$
8.7. The Characteristic Function
We use in this section the following standard results associated with complex numbers. The notation $i$ denotes the complex number $i=\sqrt{-1}$ and $z$ refers to the complex number $z=a+ib, a,b\in\R$.
- $\exp(ix)=\cos x + i \sin x$
- $|a+bi|=\sqrt{a^2+b^2}$
- $\overline{a+bi}=a-bi$
The notation $\mathbb{C}$ corresponds to the set of all complex numbers.
Definition 8.7.1.
The characteristic function associated with the random vector $\bb X$ is the following function
\[ \phi:\R^d\to \mathbb{C}, \qquad \phi_{\bb X}(\bb t) = \E(\exp(i{\bb t}^{\top}\bb X))\]
where the expectation is taken with respect to the distribution of $\bb X$. We sometimes omit the index $\bb X$ and simply denote the characteristic function by $\phi(\bb t)$.
Proposition 8.7.1.
Let $\phi$ be the characteristic function of a random vector $\bb X$. Then,
- $\phi(\bb 0)=1$.
- $|\phi(\bb t)|\leq 1$.
- $\phi(\bb t)$ is a continuous function.
- $\phi_{-\bb X}(\bb t)=\overline{\phi_{\bb X}(\bb t)}$.
- $\phi_{a\bb X+\bb b}(\bb t)= e^{i{\bb t}^{\top}\bb b} \phi_{\bb X}(a\bb t)$.
- If ${\bb X}^{(n)}, n=1,\ldots,N$ are independent RVs, then
$\phi_{\sum_{n=1}^N {\bb X}^{(n)}}(\bb t) = \prod_{n=1}^N \phi_{{\bb X}^{(n)}}(\bb t)$.
- If $\E\|\bb X\|<\infty$ then $\nabla \phi=i \E(\bb X)$.
- If $\E\|\bb X\|^2<\infty$ then $\nabla^2 \phi=-\E({\bb X}{\bb X}^{\top})$.
Proof.
Statement 1 follows from $\phi(\bb 0)=\E(1)=1$. Statement 2 follows from
\begin{align*}
|\exp(ix)|&=|\cos x+i\sin x|=\cos^2x + i^2\sin^2(x)=\cos^2x -\sin^2(x)\\ &\leq \cos^2 x+\sin^2 x=1
\end{align*}
and $|\E(X)|\leq \sup |X| \E(1)=\sup|X|$. To prove statement 3, note that if $\bb t\to \bb r$ then
$\exp(i{\bb t}^{\top}\bb x)\to \exp(i{\bb r}^{\top}\bb x)$, which implies by Proposition 8.3.1 that
\begin{align*}
\phi(\bb t)-\phi(\bb r) &=\E(\exp(i{\bb t}^{\top}X) - \exp(i{\bb r}^{\top}\bb X)) \\ &\leq
\E(\|\exp(i{\bb t}^{\top}X) - \exp(i{\bb r}^{\top}\bb X)\|) \to 0.
\end{align*}
Statement 4 follows from the following change of integration measure (see Proposition F.3.12):
\begin{align*}
\phi_{-\bb X}(\bb t)
&= \int \exp(i{\bb t}^{\top}\bb x)\, dF_{-\bb X}(\bb x)
= \int \exp(-i{\bb t}^{\top}\bb x)\, dF_{\bb X}(\bb x).
\end{align*}
Statement 5 follows from the change of integration measure (see Proposition F.3.12):
\begin{align*}
\phi_{a\bb X+\bb b}(\bb t)
&= \int \exp(i{\bb t}^{\top}\bb x)\, dF_{a\bb X+\bb b}(\bb x)
%= \int \exp(i{\bb t}^{\top}(a\bb X+\bb b))\, dF_{a\bb X+\bb b}(a\bb X+\bb b)\\
= \int \exp(i{\bb t}^{\top}(a\bb X+\bb b))\, dF_{\bb X}(\bb X)\\
&=\exp(i{\bb t}^{\top}\bb b) \int \exp(i{a\bb t}^{\top}\bb X)\, dF_{\bb X}(\bb X).
\end{align*}
The proof of statement 5 is similar to the proof of Proposition 4.8.1. The last two statement can be proven by expanding the exponential in $\phi$ using a Taylor series expansion, differentiating the Taylor series term by term at $\bb t=\bb 0$, and noting that only the leading term remains.
Note that part 2 of the proposition above implies that the characteristic function always exists.
Proposition 8.7.2.
The characteristic function of a $N(\bb \mu,\Sigma)$ random vector is
\[\phi(\bb t)=\exp(i{\bb t}^{\top}\bb\mu - {\bb t}^{\top}\Sigma\bb t/2).\]
Proof.
We start by showing that in the univariate case $\phi(t)=\exp(-t^2/2)$.
\begin{align*}
\phi(t)
&= \E \exp(itx) = \E(\cos tx + i \sin tx) = \E(\cos tx) + 0\\
&= \int \cos(tx)\exp(-x^2)/\sqrt{2\pi}\,dx.
\end{align*}
where the third equality above holds since the $\sin(tx)\exp(-tx^2)$ is an even function ($f(x)=-f(-x)$) whose integral is zero. Differentiating $\phi$ with respect to $t$, we have
\begin{align*}
\phi'(t) &= -\int x \sin(tx)\exp(-x^2)/\sqrt{2\pi}\,dx
\\ &= \int \sin(tx)\frac{d\exp(-x^2)/\sqrt{2\pi}}{dx} \,dx \\
&= \sin(tx)\exp(-x^2)/\sqrt{2\pi} \Big|_{-\infty}^{\infty}
- \int t\cos(tx)\exp(-x^2)/\sqrt{2\pi} \, dx \\
&= 0- \int t\cos(tx)\exp(-x^2)/\sqrt{2\pi} \, dx \\ &= -t\phi(t).
\end{align*}
We have thus obtained a differential equation $\phi'(t)=-t\phi(t)$ (subject to the initial condition $\phi(0)=\E(1)=1$) whose only solution is $\phi(t)=\exp(-t^2/2)$. An alternative proof uses the completion of the square method as in Proposition 3.9.2.
Part 6 of the proposition above shows that
\[\text{if}\quad
\bb X\sim N(\bb 0,I) \quad \text{then}\quad \phi_{\bb X}(\bb t)= \exp(-{\bb t}^{\top}\bb t/2).\]
Given an arbitrary $\bb X\sim N(\bb \mu,\Sigma)$ we can use the transformation $\bb X=\Sigma^{1/2}\bb Y+\bb \mu$ where $\bb Y\sim N(\bb 0,I)$ (as in Proposition 5.2.3). This yields
\begin{align*}
\E(\exp(i{\bb t}^{\top} \bb X))
&= \E\left(\exp\left(i{\bb t}^{\top}(\Sigma^{1/2}\bb Y+\bb \mu) \right)\right)\\
&= \exp(i{\bb t}^{\top}\bb \mu) \E \left(\exp(i(\Sigma^{1/2}\bb t)^{\top}\bb Y)\right) \\
&= \exp(i{\bb t}^{\top}\bb \mu) \E \left(\exp(i\bb u^{\top}\bb Y)\right) \\
&= \exp(i{\bb t}^{\top}\bb \mu) \exp\left(-{\bb u}^{\top}\bb u/2\right) \\
&= \exp(i{\bb t}^{\top}\bb \mu) \exp\left(-(\Sigma^{1/2}\bb t)^{\top} (\Sigma^{1/2}\bb t) /2\right) \\
&= \exp (i{\bb t}^{\top}\bb \mu- \bb t^{\top} \Sigma \bb t\, /2).
\end{align*}
Lemma 8.7.1.
Assuming that $a > 0$,
\[ \int \exp(-a x^2 + bx)\,dx= \sqrt{\frac{\pi}{a}} \exp({b^2/(4a)}).\]
Proof.
Writing $-ax^2+bx=-a(x-b/(2a))^2+b^2/(4a)$, we have
\begin{align*}
\int e^{-a x^2 + bx}\,dx
&= e^{b^2/(4a)}\int e^{-a(x-b/(2a))^2}\, dx
= e^{b^2/(4a)} \sqrt{\pi/a},
\end{align*}
where the last equality follows from the fact that the Gaussian pdf integrates to one.