Probability

The Analysis of Data, volume 1

Probability and Measure Theory

1.7. Probability and Measure Theory*

Definition 1.2.1 appears to be formal, and yet is not completely rigorous. It states that a probability function $\P$ assigns real values to events $E\subset \Omega$ in a manner consistent with the three axioms. The problem is that the domain of the probability function $\P$ is not clearly specified. In other words, if $\P$ is a function $\P:\mathcal{F}\to\R$ from a set $\mathcal{F}$ of subsets of $\Omega$ to $\R$, the set $\mathcal{F}$ is not specified. The importance of this issue stems from the fact that the three axioms need to hold for all sets in $\mathcal{F}$.

At first glance this appears to be a minor issue that can be solved by choosing $\mathcal{F}$ to be the power set of $\Omega$: $2^{\Omega}$. This works nicely whenever $\Omega$ is finite or countably infinite. But selecting $\mathcal{F}=2^{\Omega}$ does not work well for uncountably infinite $\Omega$ such as continuous spaces. It is hard to come up with useful functions $\P:2^{\Omega}\to\R$ that satisfy the three axioms for all subsets of $\Omega$.

A satisfactory solution that works for uncountably infinite $\Omega$ is to define $\mathcal{F}$ to be a $\sigma$-algebra of subsets of $\Omega$ that is smaller than $2^{\Omega}$. In particular, when $\Omega\subset\R^d$, the Borel $\sigma$-algebra is sufficiently large to include the "interesting" subsets of $\Omega$ and yet is small enough to not restrict $\P$ too much (see Section E.1 for a definition of a $\sigma$-algebra and the Borel $\sigma$-algebra).

We also note that a probability function $\P$ is nothing but a measure $\mu$ on a measurable space $(\Omega,\mathcal{F})$ satisfying $\mu(\Omega)=1$. In other words, the triplet $(\Omega,\mathcal{F},\P)$ is a measure space where $\mathcal{F}$ is the $\sigma$-algebra of measurable sets and $\P$ is a measure satisfying $\P(\Omega)=1$. Thus, the wide array of mathematical results from measure theory (Chapter E) and Lebesgue integration (Chapter F) are directly applicable to probability theory.