Chapter 9

Essentials of General Probability Theory

In this chapter we present some measure-theoretic foundations of probability theory and random variables. This then leads us into the main tools and formulas for computing expectations and conditional expectations of random variables (more importantly, continuous random variables) under different probability measures. The main formulas that are provided here are used in further chapters for the understanding and quantitative modelling of continuous-time stochastic financial models.

9.1 Random Variables and Lebesgue Integration

Within the foundation of general probability theory, the mathematical expectation of any real-valued random variable X defined on a probability space (Ω, ℱ, ℙ) is a so-called Lebesgue integral w.r.t. a given probability measure ℙ over the sample space Ω. In order to define an integral of a random variable (i.e., a measurable set function) w.r.t. some given measure, we need a measurable space and a measure. The measurable space here is the pair (Ω, ℱ), where ℱ is a σ-algebra of events in Ω, and the measure is the probability measure function ℙ: Ω → [0, 1]. Before providing a precise definition of such an integral, we make a couple of remarks. Namely, Ω is any abstract set having either a finite, infinitely countable, or uncountable number of elements. So here we are dealing with a general probability space that includes the finite probability spaces studied in previous chapters as special cases. As usual, we denote every element in Ω by ω, where {ω} is a singleton set in Ω. Any random variable X has a positive part X+ ≔ max{X, 0} ≥ 0 and a negative part X− ≔ max{−X, 0} ≥ 0, where X = X+ − X− and |X| = X+ + X−. Note that both random variables X+ and X− are nonnegative. The expectation of X w.r.t. the measure P is defined as the difference of two nonnegative expectations:

$\text{E}\left[X\right]=\text{E}\left[X_{\text{+}}\right]-\text{E}\left[X_{-}\right].$

We write this, term by term, in the notation of a Lebesgue integral w.r.t. the measure ℙ as follows:

$\underset{\text{E}\left[X\right]}{\underbrace{{\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)}}}=\underset{\text{E}\left[X_{+}\right]}{\underbrace{{\displaystyle {\int }_{\text{Ω}}{X}_{+}\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)}}}-\underset{\text{E}\left[X_{-}\right]}{\underbrace{{\displaystyle {\int }_{\text{Ω}}{X}_{-}\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)}}}.\left(9.1\right)$

The absolute value of X has expectation E[|X|] = E[X+]+E[X−]. X is said to be integrable w.r.t. measure ℙ iff E[|X|] < ∞, i.e., E[X+] < ∞ and E[X−] < ∞, hence E[X] < ∞. A common notation used to state this is to write X ∊ L1(Ω, ℱ, ℙ). If E[X+] = ∞ and E[X−] < ∞, then we set E[X] = ∞; if E[X+] < ∞ and E[X−] = ∞, then we set E[X] = −∞; if E[X+] = E[X−] = ∞, then E[X] is not defined. Note that for strictly positive or nonnegative X, X+ ≡ X (X− ≡ 0). For strictly negative or nonpositive X we have X− ≡ −X (X+ ≡ 0). The Lebesgue integral of a random variable X over any subset A ∊ ℱ is defined as the Lebesgue integral (over Ω) of the random variable ?A X:

${\displaystyle {\int }_{A}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)\equiv }{\displaystyle {\int }_{\text{Ω}}{\mathbb{I}}_A\left(\omega \right)X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)\equiv \text{E}\left[{\mathbb{I}}_{A}X\right]\left(9.2\right)}$

where ?A (ω) = 1 if ω ∊ A and ?A(ω) = 0 if ω ∉ A. Moreover, putting X ≡ 1 gives the probability under measure ℙ of any event A ∊ ℱ as a Lebesgue integral w.r.t. ℙ over A:

$\text{E}\left[{\mathbb{I}}_A\right]=\mathbb{P}\left(A\right)={\displaystyle {\int }_A\text{d}\mathbb{P}\left(\omega \right)={\displaystyle {\int }_{\text{Ω}}{\mathbb{I}}_A\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right).(9.3)}}$

Since ℙ is an assumed probability measure, we must have ℙ(Ω) = E[?Ω] = 1.

We now develop the Lebesgue integral of X w.r.t. measure ℙ by first considering its definition for any (ℙ-a.s.) nonnegative real-valued1 random variable X: Ω → [0, ∞). The symbol ℙ-a.s. stands for almost surely w.r.t. to probability measure ℙ, i.e., a relation holds ℙ-a.s. means that it holds with probability one. So X ≥ 0 (ℙ-a.s.) means that the set for which X ≥ 0 has probability one: ℙ({ω ∊ Ω : X(ω) ≥ 0}) = 1. We form a partition of the positive real line: 0 = y0 < y1 < y2 < ... < yk < yk+1 < ... < yn, where yn → ∞ as n → ∞. A partition is defined such that the maximum sub-interval spacing approaches zero, i.e., $\underset{n\to \infty }{\lim }\underset{0\le k\le n-1}{\max }\left(yk+1-yk\right)=0$ is implied. Then, the sets in ℱ defined by Ak ≔ X−1([yk,yk+1)) ≡ {ω ∊ Ω : yk ≤ X(ω) < yk+1},k = 0,1,..., n − 1, and An ≔ X−1([yn, ∞)) = {ω ∊ Ω : X(ω) ≥ yn} are mutually exclusive and form a partition of the sample space Ω for every n ≥ 0. The Lebesgue integral is defined as the limit of a partial sum:

${\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right):=}\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{n}yk\mathbb{P}\left(Ak\right)}\equiv {\displaystyle \sum _{k=0}^{\infty }yk\mathbb{P}\left(A_k\right).}\left(9.4\right)$

Assuming it exists, this sum gives a unique value (including cases where it may be infinite). This definition uses the lower value yk of X on every set Ak. An equivalent definition replaces the value yk with the upper value yk+1 for every k ≥ 1. For any real X = X+ −X−, we simply use (9.4) for each respective nonnegative Lebesgue integral of X+ and X− and subtract to obtain the Lebesgue integral of X according to (9.1).

Assume {Ai} is a countable collection of disjoint sets in ℱ. Then, applying the identity ${\mathbb{I}}_{{\cup }_i{A}_i}={\displaystyle {\sum }_i{\mathbb{I}}_{A_i}}$ to (9.2) and using the linearity property of the Lebesgue integral gives

${\displaystyle {\int }_{{\cup }_i{A}_i}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle \sum _i{\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)}}}{\mathbb{I}}_{A_i}\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle \sum _i{\displaystyle {\int }_{A_i}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right),}}\left(9.5\right)$

i.e., $\text{E}\left[X{\mathbb{I}}_{{\cup }_i{A}_i}\right]={\displaystyle \sum }\text{E}\left[X{\mathbb{I}}_{A_i}\right].$ For X ≡ 1 we have $\text{E}\left[{\mathbb{I}}_{{\cup }_i{A}_i}\right]={\displaystyle {\sum }_i\text{E}\left[{\mathbb{I}}_{A_i}\right]:}$

$\mathbb{P}\left({\displaystyle \underset{i}{\cup }{A}_i}\right)={\displaystyle \sum _i\mathbb{P}\left(A_i\right)}.$

This recovers the countable additivity property of ℙ, which must hold for ℙ to be a measure.

To see how the Lebesgue integral in (9.4) works, consider the class of random variables having the form of a finite sum of indicator random variables $X={\displaystyle {\sum }_{j=0}^N{x}_j{\mathbb{I}}_{C_j}}$ with each xj ∊ [0, ∞), Cj = {ω ∊ Ω : X(ω) = xj} ⊂ ℱ, and where ${\left\{C_j\right\}}_{j=0}^N$ forms a partition of Ω. This is called a simple function or simple random variable. Any random variable that can take only finitely many different values, x0,..., xN, (including possibly a zero value) has this form. Note also that any simple random variable can be represented as the difference of two nonnegative simple random variables. For each sufficiently small interval [yk,yk+1) we have Ak = X−1([yk,yk+1)) = X−1(xj) = Cj if there is a value xj ∊ [yk,yk+1) for some 0 ≤ j ≤ N, or otherwise Ak = X−1([yk,yk+1)) = ø. For sufficiently large n, each xj value will be contained in exactly one sub-interval and the nth partial sum in (9.4) does not change value as n is increased indefinitely since only N intervals will contain an xj value (with probability measure ℙ(Cj)) and the rest of the intervals yield Ak = ø (i.e., ℙ(Ak) = 0). Hence, the Lebesgue integral in (9.4) is given as the finite sum

${\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle \sum _{j=0}^N{x}_j\mathbb{P}}\left(C_j\right)={\displaystyle \sum _{j=0}^N{x}_j\mathbb{P}\left(X=x_j\right).}}\left(9.6\right)$

This corresponds to the formula in (6.17) for a discrete random variable on a finite state space Ω. However, here the simple random variable is discrete and defined more generally for uncountably infinite Ω. For a discrete random variable X taking on a countably infinite number of values {x0, x1,...}, with infinitely countable or uncountable Ω, it readily follows that the Lebesgue integral recovers the familiar expectation formula:

$\text{E}\left[X\right]={\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle \sum _{j=0}^{\infty }{x}_j\mathbb{P}\left(X=x_j\right)}}$

where the probabilities pj = ℙ(X = xj) define the probability mass function (PMF) of X in the measure ℙ.

Simple random variables are particularly useful and provide an alternative equivalent way of defining the Lebesgue integral. In standard measure and integration theory, one begins by taking (9.6) as the definition for the Lebesgue integral of any simple random variable. Then, for any X ≥ 0 (a.s.) the Lebesgue integral in (9.4) is equivalently defined as the supremum of Lebesgue integral values over the set of all nonnegative simple random variables having value not greater than X:

$\text{E}\left[X\right]={\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right):=\sup \left\{{\displaystyle {\int }_{\text{Ω}}{X}^{*}\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right):X^{*}\text{is simple 0}\le X^{*}\le X}\right\}.\left(9.7\right)}$

This last definition is rather abstract but it gives rise to other more explicit representations for the Lebesgue integral upon using the fact that a nonnegative X can be expressed (a.s.) as a limiting sequence of nonnegative simple random variables. Then, the Lebesgue integral of any nonnegative X can be computed as the limit of the Lebesgue integrals corresponding to the sequence of nonnegative simple random variables that converges to X. To see how this arises, we need to first recall the concept of pointwise convergence of a sequence of random variables. We recall that a sequence of random variables, Xn,n = 1,2,..., is said to converge pointwise almost surely to some random variable X when Xn → X (a.s.) as n → ∞, i.e., $\mathbb{P}\left(\left\{\omega \in \text{Ω:}\underset{n\to \infty }{\lim }{X}_n\left(\omega \right)=X\left(\omega \right)\right\}\right)=1$. A sequence of nonnegative random variables Xn, n = 1,2,..., is said to converge pointwise monotonically to X (a.s.) if Xn → X and 0 ≤ X1 ≤ X2 ≤ ... ≤ X (a.s.), i.e., ℙ({ω ∊ Ω : Xn(ω) ↗ X(ω)}) = 1. If we have such a sequence, then each successive random variable in the sequence will approximate X better than the previous one and the approximation becomes exact in the limit n → ∞. It stands to reason that the corresponding sequence of expected values (Lebesgue integrals for each Xn) will be increasingly better approximations and will give the expected value (Lebesgue integral) of X in the limit n → ∞. This is summarized (without proof) in the following well-known theorem, appropriately called the Monotone Convergence Theorem (MCT) for random variables.

Theorem 9.1

(Monotone Convergence for random variables). If Xn,n = 1,2,..., is a sequence of nonnegative random variables converging pointwise monotonically to X (a.s.), then

$\underset{n\to \infty }{\lim }\text{E}\left[X_n\right]=\text{E}\left[X\right],i.e.,\underset{n\to \infty }{\lim }{\displaystyle {\int }_{\text{Ω}}{X}_n\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)}.}$

In fact, we have monotonic convergence, i.e., E[Xn] ↗ E[X] as n → ∞.

The MCT has many applications. For instance, we recover the known continuity properties of a probability measure ℙ. That is, let A1, A2,..., An,... be subsets (events) in Ω.

If A1 ⊂ A2 ⊂ ..., then $\underset{n\to \infty }{\lim }{A}_n={\displaystyle \underset{n=1}{\overset{\infty }{\cup }}{A}_n}$ and we have (monotone continuity from below):

$\mathbb{P}\left({\displaystyle \underset{n\text{=1}}{\overset{\infty }{\cup }}{A}_n}\right)=\underset{n\to \infty }{\lim }\mathbb{P}\left(A_n\right).$

If A1 ⊃ A2 ⊃ ..., then $\underset{n\to \infty }{\lim }{A}_n={\displaystyle \underset{n=1}{\overset{\infty }{\cap }}{A}_n}$ and we have (monotone continuity from above):

$\mathbb{P}\left({\displaystyle \underset{n=1}{\overset{\infty }{\cap }}{A}_n}\right)=\underset{n\to \infty }{\lim }\mathbb{P}\left(A_n\right).$

These follow as a corollary of Theorem 9.1. For example, monotone continuity from above is obtained by defining the sequence of monotonically increasing random variables $X_n=1-{\mathbb{I}}_{A_n},$ where An+1 ⊂ An for all n ≥ 1. Hence, Xn ↗ X ≡ 1 − ?A, $A={\displaystyle \underset{n=1}{\overset{\infty }{\cap }}{A}_n}$. By MCT we have $\underset{n\to \infty }{\lim }\text{E}\left[X_n\right]=\text{E}\left[X\right]$, i.e.,

$\underset{n\to \infty }{\lim }\text{E}\left[\text{1}-{\mathbb{I}}_{A_n}\right]=\text{E}\left[\text{1}-{\mathbb{I}}_A\right]\Rightarrow \underset{n\to \infty }{\lim }\mathbb{P}\left(A_n\right)=\mathbb{P}\left(A\right)=\mathbb{P}\left({\displaystyle \underset{n=1}{\overset{\infty }{\cap }}{A}_n}\right).$

We now apply MCT to obtain a more explicit formula for E[X] when X ≥ 0 (a.s.) by producing a monotonically increasing sequence of nonnegative simple random variables. There are many ways to do so. An explicit example is the sequence defined by

$X_n\left(\omega \right):=\{\begin{array}{cc}k/2^n& \text{if }\omega \in A_n^{\left(k\right)}\\ 0& \text{otherwise}\end{array}\left(9.8\right)$

with sets $A_n^{\left(k\right)}:=\left\{\omega \in \text{Ω}\text{:}k/2^n\le X\left(\omega \right)<\left(k+1\right)/2^n\right\},k=0,\mathrm{...},2^{2n}$. For every n ≥ 1, Xn is a simple random variable:

$X_n\left(\omega \right)={\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}{\mathbb{I}}_{A_n^{\left(k\right)}}\left(\omega \right)={\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}{\mathbb{I}}_{\left\{X\in \left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right\}}\left(\omega \right)}.\left(9.9\right)}$

We leave it as an exercise for the reader to verify that Xn(ω) ↗ X(ω) for any X ≥ 0. Therefore, by using (9.6) as the expected value for every simple Xn and passing to the limit with the use of MCT, the Lebesgue integral (expectation) of any nonnegative random variable X is given by

$\text{E}\left[X\right]=\underset{n\to \infty }{\lim }\text{E}\left[X_n\right]=\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}\mathbb{P}\left(\frac{k}{2^n}\le X<\frac{k+1}{2^n}\right).\left(9.10\right)}$

This is equivalent to (9.4) but here the partitions are chosen such that the convergence is monotonic, whereas the series in (9.4) is generally not monotonic. For arbitrary X = X+ − X− we use the above construction for the two separate nonnegative random variables X+ and X− and subtract the two expectations, giving E[X], assuming we don't have ∞ − ∞.

The Lebesgue integral above was presented in the general context of an abstract state space Ω with generally uncountable numbers of abstract elements ω and random variables X (i.e., measurable set functions) on a probability space (Ω, ℱ, ℙ). It is important to consider the case where Ω ⊂ ℝ, i.e., every ω is a real number. Recall our discussion of the Borel σ-algebra ℬ(ℝ). The pair (ℝ, ℬ(ℝ)), i.e., Ω = ℝ, ℱ = ℬ(ℝ), is a measurable space. The Lebesgue measure2 on ℝ, which we denote by m, is a measure that assigns a nonnegative real value or infinity to each Borel set B ∊ ℬ(ℝ), i.e., m : B → [0, ∞) ∪ ∞, such that all intervals [a, b], a ≤ b, have measure equal to their length: m([a, b]) = b − a. All semi-infinite intervals (− ∞, a], (− ∞, a), [b, ∞), or (b, ∞) have infinite Lebesgue measure. A single point has Lebesgue measure zero, m({x}) = 0 for any point x ∊ ℝ. The empty set also has zero measure. Any set B that is a countable union of points also has zero measure since m is countably additive, i.e.,

$m\left({\displaystyle \underset{n=1}{\overset{\infty }{\cup }}{B}_n}\right)={\displaystyle \sum _{n=1}^{\infty }m\left(B_n\right)}$

for all disjoint Borel sets Bn,n ≥ 1. This also implies finite additivity holds where $m\left({\displaystyle {\cup }_{n=1}^N{B}_n}\right)={\displaystyle {\sum }_{n=1}^{N}m\left(B_n\right)}$ for any N ≥ 1. The set of rational numbers ℚ is count able with Lebesgue measure zero. The irrationals ℝ ∩ ℚ∁ = ℝℚ are uncountable. Since (ℝℚ) ∪ ℚ = ℝ, the Lebesgue measure of any Borel set B is unchanged if we remove all the rational numbers from it, i.e., as a disjoint union, B = (Bℚ) ∪ (B ∩ ℚ), hence m(B) = m (Bℚ) + m(B ∩ ℚ) = m(Bℚ). Note that B ∩ ℚ ⊃ ℚ so m(B ∩ ℚ) ≤ m(ℚ) = 0 ⇒ m(B ∩ ℚ) = 0.

There are also other more peculiar Borel sets that are uncountable but yet have Lebesgue measure zero. The well-known Cantor ternary set on [0,1], which is discussed in detail in many textbooks on real analysis, is such an example. In the interest of space, we don't discuss this set in any detail here. It suffices to note that the points in this set are too sparsely distributed over the unit interval [0,1] and hence do not accumulate any length measure. On the other hand, the points in the Cantor set cannot be counted (i.e., listed in a sequence in one-to-one with the integers). The Cantor set is constructed by starting with [0, 1] and removing the middle third interval $\left(\frac{1}{3},\frac{2}{3}\right)$ and then repeating this process of removing the middle third for all remaining intervals in succession. One can then make a simple argument to show that the set is not countable, in essentially the same manner that the total number of branches in a binomial tree having an infinite number of time steps cannot be counted either.

By the above discussion, we see that the triplet (Ω, ℱ, ℙ) ≔ ([0, 1], ℬ([0, 1]), m) serves as an example of a probability space where m acts as a uniform probability measure on the set Ω = [0,1] ≡ {x ∊ ℝ : 0 ≤ x ≤ 1}. By our previous notation, we may also write this as Ω = {ω ∊ ℝ : 0 ≤ ω ≤ 1}. Any real-valued random variable X : [0, 1] → ℝ is a Borel (measurable) function where X-1(B) ∊ ℬ([0, 1]) for every B ∊ ℬ(ℝ). Its expected value is then the Lebesgue integral w.r.t. the Lebesgue (uniform probability) measure m:

$\text{E}\left[X\right]={\displaystyle {\int }_{\left[0,1\right]}X\left(\omega \right)\text{d}m}\left(\omega \right).\left(9.11\right)$

For example, the outcome of the experiment of picking a real number in [0,1] uniformly at random is captured by the value of the random variable X(ω) = ω. The probability that a number is chosen within some arbitrary subinterval [a, b] ⊂ [0,1] must therefore be the length of the interval. In this case the event is represented as the set {X ∊ [a, b]} = {a ≤ ω ≤ b}. Its probability is the Lebesgue integral of the indicator random variable ${\mathbb{I}}_{\left\{X\in \left[a,b\right]\right\}}\left(\omega \right)={\mathbb{I}}_{\left\{a\le \omega \le b\right\}}$:

$\mathbb{P}\left(X\in \left[a,b\right]\right)=\text{E}\left[{\mathbb{I}}_{\left\{X\in \left[a,b\right]\right\}}\right]={\displaystyle {\int }_{\left[0,1\right]}{\mathbb{I}}_{\left[a,b\right]}\left(\omega \right)\text{d}m\left(\omega \right)\equiv }{\displaystyle {\int }_{\left[a,b\right]}\text{d}m\left(\omega \right)}=m\left(\left[a,b\right]\right)=b-a.$

Note that ℙ(Ω) = ℙ(X ∊ [0,1]) = m([0,1]) = 1. Combining this with the countable additivity property of m and the fact that m : B → [0,1] for any B ∊ ℬ([0,1]) = ℬ(ℝ) ∩ [0,1] shows that the Lebesgue measure m restricted to the unit interval [0,1] is a proper probability measure. Note that we also get the probability of picking any finite or countable set of numbers in [0,1] is zero since the Lebesgue measure of such a set is zero. In fact, the probability of picking a rational number is zero:

$\mathbb{P}\left(X\in \left[0,1\right]\cap \mathbb{Q}\right)={\displaystyle {\int }_{\left[0,1\right]\cap \mathbb{Q}}\text{d}m\left(\omega \right)=m\left(\left[0,1\right]\cap \mathbb{Q}\right)=0.}$

The probability of picking an irrational number is 1, ℙ(X ∊ [0,1]ℚ) = m([0,1]ℚ) = 1. As well, the probability that we pick a number to be in the Cantor set C is zero since ℙ(X ∊ C) = m(C) = 0.

If we have a Lebesgue-measurable set Ω ⊂ ℝ with finite nonzero measure, 0 < m(Ω) < ∞, then the Lebesgue measure restricted to Ω, denoted by mΩ, gives mΩ(B) = m(B) for any set B ∊ ℬ(Ω). [Note: We assume that Ω is a Borel set since any Lebesgue-measurable set is either a Borel set or close to it in the sense that all sets that are Lebesgue-measurable and not Borel are null sets of Lebesgue measure zero.] The set Ω has "nonzero finite length." Typically, Ω is an interval or a combination of intervals, but need not be. In the above example, Ω = [0,1] (i.e., the unit interval) and m[0,1](B) = m(B) for any B ∊ ℬ([0,1]). Then, mΩ : B → [0, c], c = m(Ω), for any B ∊ ℬ(Ω). So mΩ is a uniform measure on the space (Ω, ℬ(Ω)). We simply normalize this measure to obtain a uniform probability measure defined by $\mathbb{P}\left(B\right):=\frac{1}{c}.m_{\text{Ω}}\left(B\right),$ for all B ∊ ℬ(Ω), with $\mathbb{P}\left(\text{Ω}\right)=\frac{1}{c}.m_{\text{Ω}}\left(\text{Ω}\right)=\frac{c}{c}=1.$ Hence, (Ω, ℬ(Ω), ℙ) is a probability space where ℬ(Ω) contains all the events.

At this point we recall (from real analysis) the Lebesgue integral over ℝ w.r.t. the Lebesgue measure m, which is defined regardless of any association to a probability space. The Lebesgue integral, w.r.t. m over ℝ, is defined for any Lebesgue-measurable function f, i.e., if f−1 (I) is a Lebesgue-measurable set for any interval I ∊ ℝ. For our purposes, it suffices to assume that f is any Borel-measurable real-valued function, i.e., the set f−1(B) ≡ {x ∊ ℝ : f(x) ∊ B} ∊ ℬ(ℝ) for any B ∊ ℬ(ℝ). The Lebesgue integral (w.r.t. m) over ℝ is first defined for a nonnegative Borel function f. By this we mean f is nonnegative almost everywhere (abbreviated a.e. or m-a.e.), i.e., the set of points where f < 0 has Lebesgue measure zero: m({x ∊ ℝ : f(x) < 0}) = 0. Essentially, by putting Ω = ℝ, F = ℬ(ℝ), and replacing ℙ(ω) by m(x) and X(ω) by f(x), equivalent definitions follow as those displayed above for the probability (expectation) Lebesgue integrals on the generally abstract measurable sets (Ω, F) with measure ℙ. In the Lebesgue integral the measurable space is (ℝ, ℬ(ℝ)) and the measure is the Lebesgue measure m. The Lebesgue integral of a nonnegative Borel function f (w.r.t. m over ℝ) can be defined by

${\displaystyle {\int }_{ℝ}\left(x\right)\text{d}m\left(x\right):=\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{n}ykm\left(B_k\right),}}\left(9.12\right)$

assuming the sum converges in ℝ or equals ∞. This is the analogue of the definition in (9.4). Here, the partitions yk,k ≥ 0 are defined above (9.4) and Bk ≔ f−1([yk, yk+1)) ≡ {x ∊ ℝ : yk ≤ f(x) < yk+1}, k = 0,1,..., n − 1, Bn ≔ f−1([yn, ∞)) = {x ∊ ℝ: f(x) ≥ yn}.

An equivalent (and more standard definition) is to first define the integral for any simple function $\phi \left(x\right)={\displaystyle {\sum }_{k=1}^n{a}_k{\mathbb{I}}_{A_k}\left(x\right)},$ with ak' s as real numbers and Ak' s as Lebesgue-measurable sets (for our purposes these are Borel sets in ℝ):

${\displaystyle {\int }_{ℝ}\phi \left(x\right)\text{d}m\left(x\right):={\displaystyle \sum _{k=1}^n{a}_{k}m\left(A_k\right).}}\left(9.13\right)$

[Note that the usual convention 0.∞ = 0 is used throughout.] Then, we define the Lebesgue integral for any nonnegative f as the supremum over the set of Lebesgue integral values of all nonnegative simple functions that are less than or equal to f:

${\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}m}\left(x\right):=\sup \left\{{\displaystyle {\int }_{ℝ}\phi \left(x\right)\text{d}m\left(x\right):\phi \text{is a simple function, 0 }\le \phi \le f}\right\}.\left(9.14\right)$

The Lebesgue integral, w.r.t. m over ℝ, of any Lebesgue-measurable (or Borel function) f = f+ − f−, with f± ≥ 0, is then

${\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}m\left(x\right)={\displaystyle {\int }_{ℝ}{f}_{+}\left(x\right)\text{d}m\left(x\right)-{\displaystyle {\int }_{ℝ}{f}_{-}\left(x\right)\text{d}m\left(x\right).}}}\left(9.15\right)$

As in the above case of the expectation of a random variable, f is integrable iff ${\displaystyle {\int }_{ℝ}{f}_{+}\text{d}m<\infty }\text{and }{\displaystyle {\int }_{ℝ}{f}_{-}\text{d}m}<\infty ,\text{i}\text{.e}\text{.,}|f|=f_{+}+f_{-}$ has a finite integral. If ${\displaystyle {\int }_{ℝ}{f}_{+}\text{d}m=\infty \text{and}}{\displaystyle {\int }_{ℝ}{f}_{-}\text{d}m<\infty ,\text{then}}{\displaystyle {\int }_{ℝ}f\text{d}m}=\infty ;\text{if}{\displaystyle {\int }_{ℝ}{f}_{\text{+}}\text{d}m}<\infty \text{and}{\displaystyle {\int }_{ℝ}{f}_{-}\text{d}m}=\infty ,\text{then}{\displaystyle {\int }_{ℝ}f\text{d}m}=-\infty ;\text{if}{\displaystyle {\int }_{ℝ}{f}_{+}\text{d}m}={\displaystyle {\int }_{ℝ}{f}_{-}}\text{d}m=\infty ,\text{then}{\displaystyle {\int }_{ℝ}f\text{d}m}=\infty -\infty$ is not defined. Note that the Lebesgue integral of f over any measurable (Borel) set A ∊ ℬ(ℝ) is the integral of the (Borel) measurable function?A f over ℝ:

${\displaystyle {\int }_{A}f\left(x\right)\text{d}m\left(x\right)={\displaystyle {\int }_{ℝ}{\mathbb{I}}_A\left(x\right)f\left(x\right)\text{d}m\left(x\right).}}\left(9.16\right)$

[Note: f is assumed (Borel) measurable so that the function ?A f is also measurable for any measurable set A.] The positive and negative parts of ?A f are (?A f)± = ?A f±, giving

$\begin{array}{ll}{\displaystyle {\int }_{A}f\left(x\right)\text{d}m}\left(x\right)\hfill & ={\displaystyle {\int }_{ℝ}{\mathbb{I}}_A\left(x\right)f_{+}\left(x\right)\text{d}m\left(x\right)-{\displaystyle {\int }_{ℝ}{\mathbb{I}}_A\left(x\right)f-\left(x\right)\text{d}m\left(x\right)}}\hfill \\ \hfill & ={\displaystyle {\int }_A{f}_{+}\left(x\right)\text{d}m\left(x\right)-{\displaystyle {\int }_{A}f-\left(x\right)\text{d}m\left(x\right).}}\hfill \end{array}\text{ (9.17)}$

The MCT for random variables (Theorem 9.1) has an obvious analogue for sequences of Lebesgue-measurable functions, which we now state for Borel-measurable functions. Recall that a sequence of functions, fn, n = 1,2,..., converges pointwise almost everywhere (a.e.) to a function f when fn(x) → f(x) (a.e.) as n → ∞. That is, the set for which convergence does not hold is a null set with Lebesgue measure zero: $m\left(\left\{x\in ℝ:\underset{n\to \infty }{\lim }{f}_n\left(x\right)\ne f\left(x\right)\right\}\right)=0$. A sequence of nonnegative functions fn, n = 1,2,..., is said to converge pointwise monotonically to f (a.e.) if fn(x) → f(x) and 0 ≤ f1(x) ≤ f2(x) ≤ ... ≤ f(x) (a.e.), i.e., the set of values of x ∊ ℝ for which these relations do not hold has Lebesgue measure zero. If we have such a sequence, then each successive function will better approximate f and the approximation becomes exact in the limit n → ∞. Moreover, if the sequence of functions are Borel functions, then their corresponding Lebesgue integrals converge to the Lebesgue integral of the limiting function f. This is summarized in the MCT (Monotone Convergence Theorem) for functions. Its proof is given in any standard textbook on real analysis.

Theorem 9.2

(Monotone Convergence for functions). If fn(x), n = 1, 2,..., x ∊ ℝ, is a sequence of nonnegative Borel functions converging pointwise monotonically (a.e.) to some function f, then

$\underset{n\to \infty }{\lim }{\displaystyle {\int }_{ℝ}{f}_n\left(x\right)\text{d}m\left(x\right)={\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}m\left(x\right)}}.$

In fact, we have monotonic convergence, $i.e.,{\displaystyle {\int }_{ℝ}{f}_n\left(x\right)\text{d}m\left(x\right)={\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}m\left(x\right),asn\to \infty .}}$

For every nonnegative measurable (or Borel) function f there exists a sequence of non-negative simple functions, fn, n ≥ 1, converging monotonically to f, i.e., fn (x)↗ f(x) (a.e.) as n → ∞. In fact, the analogue of the sequence in (9.9) is the sequence of simple functions:

$f_n\left(x\right)={\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}}{\mathbb{I}}_{\left\{f-1\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right)\right\}}\left(x\right)\equiv {\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}}{\mathbb{I}}_{\left\{f\left(x\right)\in \left[\frac{k}{2^n},\frac{k+1}{2^n}\right]\right\}}.\left(9.18\right)$

Using MCT (Theorem 9.2) and the simple formula in (9.13) gives us the following repre-sentation for the Lebesgue integral, w.r.t. m over ℝ, of any nonnegative function f:

${\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}m\left(x\right)=\underset{n\to \infty }{\lim }{\displaystyle {\int }_{ℝ}{f}_n\left(x\right)\text{d}m\left(x\right)=\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}m\left(A_k^n\right),}}}\text{(9}\text{.19)}$

where $A_k^n:=f^{-1}\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right)\equiv \left\{x\in ℝ:\frac{k}{2^n}\le f\left(x\right)<\frac{k+1}{2^n}\right\}.$. Such a series can be obtained for both f+ and f− and then the Lebesgue integral of f = f+ − f− w.r.t. m over ℝ is given by the difference, provided the result is defined (not ∞ − ∞). Note also that the Lebesgue integral of a nonnegative f over any measurable (Borel) set B, using (9.16), has the representation

${\displaystyle {\int }_{B}f\left(x\right)\text{d}m\left(x\right)=\underset{n\to \infty }{\lim }{\displaystyle {\int }_{ℝ}{\mathbb{I}}_B\left(x\right)f_n\left(x\right)\text{d}m\left(x\right)=\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}m\left(A_k^n\cap B\right).}}}\text{(9}\text{.20)}$

The theory of Lebesgue integration w.r.t. m provides a framework for integrating a general class of real-valued functions. Lebesgue integration also forms the foundation for probability theory. The actual computation of many specific integrals is, in practice, a difficult task, as there are no known techniques for integrating particular functions. However, most Lebesgue integrals that we encounter are equivalent to a corresponding Riemann integral. This then allows us to use all the known powerful techniques of elementary calculus to compute Riemann integrals. Consider a continuous function f : [a, b] → ℝ. Then, f is integrable and the function $F\left(x\right):={\displaystyle {\int }_{\left[a,x\right]}f\left(y\right)\text{d}m\left(y\right)\equiv {\displaystyle {\int }_a^{x}f\left(y\right)\text{d}m\left(y\right)}}$ is differentiable for x ∊ (a, b), i.e., F'(x) = f(x). This is the fundamental theorem of calculus. The theorem just below relates the Lebesgue integral w.r.t. m and the Riemann integral of a bounded function over a finite interval. This theorem is proven in most standard textbooks on real analysis. Note that the statement "f is continuous (a.e.)" means that the set of points for which f is not continuous has Lebesgue measure zero.

Theorem 9.3

(Lebesgue versus Riemann Integration). Let f : [a, b] → ℝ be bounded. Then:

1. (i) f is Riemann-integrable, i.e., ${\displaystyle {\int }_a^{b}f\left(x\right)\text{d}x}$ is denned, if and only if f is continuous (a.e.).
2. (ii) If f is Riemann-integrable, then the Lebesgue integral over [a, b] is also defined and the two integrals are the same, i.e., ${\displaystyle {\int }_a^{b}f\left(x\right)\text{d}x={\displaystyle {\int }_{\left[a,b\right]}f\left(x\right)\text{d}m\left(x\right).}}$

Hence, when computing the Lebesgue integral of a function over an interval with a well-defined Riemann integral, we can simply equate the Lebesgue integral with the corresponding Riemann integral. For example, we write ${\displaystyle {\int }_a^{b}f\left(x\right)\text{d}x\equiv {\displaystyle {\int }_{\left[a,b\right]}f\left(x\right)\text{d}m\left(x\right).}}$ As well, assuming the existence of Riemann integrals for semi-infinite or infinite intervals, we write ${\displaystyle {\int }_a^{\infty }f\left(x\right)\text{d}x\equiv {\displaystyle {\int }_{\left[a,\infty \right]}f\left(x\right)\text{d}m\left(x\right),{\displaystyle {\int }_{-\infty }^{b}f\left(x\right)\text{d}x\equiv {\displaystyle {\int }_{\left(-\infty ,b\right]}f\left(x\right)\text{d}m\left(x\right),{\displaystyle {\int }_{-\infty }^{\infty }f\left(x\right)\text{d}x\equiv {\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}m\left(x\right).}}}}}}$ The same goes for other improper well-defined Riemann integrals. More generally, if the integral is over some arbitrary Borel set B, it is also customary to use shorthand notation for the Lebesgue integral w.r.t. m over B as ${\displaystyle {\int }_{B}f\left(x\right)\text{d}x\equiv {\displaystyle {\int }_{B}f\left(x\right)\text{d}m\left(x\right).}}$ If the integrand function f (or f · ?B) is not continuous (a.e.), then the Riemann integral is not defined and the integral is understood to be the corresponding Lebesgue integral, assuming it exists.

The expected value of a general real-valued random variable X defined on a probability space (Ω, F, ℙ) is a Lebesgue integral w.r.t. ℙ over Ω. This is a construction that is general, yet not always practical when dealing with a generally abstract sample space Ω. For discrete random variables, the expectation reduces to a sum involving the probability mass function. We also saw that, for a continuous uniform random variable, its expectation (Lebesgue integral w.r.t. ℙ) reduces to a Lebesgue integral w.r.t. m and hence the latter can also be expressed as a Riemann integral. The transformation from a Lebesgue integral w.r.t. measure ℙ over Ω into a Lebesgue integral (or Riemann) over ℝ makes the theory more practical. This allows E[X] to be expressed in terms of integrals over ℝ, rather than over Ω, as follows.

We begin by recalling what a distribution measure is for a random variable X. Let B be any Borel set in ℝ. The distribution measure of X w.r.t. a probability measure ℙ is defined by the set function

${\mu }_X\left(B\right):=\mathbb{P}({\text{X}}^{-1}\left(B\right))\equiv \mathbb{P}\left(X\in B\right).\left(9.21\right)$

It is important to note that μX measures subsets in ℝ, whereas ℙ measures subsets in Ω. Namely, the probability of the event {X ∊ B} is computed as a μX-measure of B. The cumulative distribution function (CDF), FX, of the random variable X, w.r.t. ℙ, is then given in terms of this measure by

$F_X\left(x\right):={\mu }_X\left(\left(-\infty ,x\right]\right)=\mathbb{P}\left(X\in \left(-\infty ,x\right]\right)\equiv \mathbb{P}\left(X\le x\right),x\in ℝ.\left(9.22\right)$

Recall that any CDF is generally a right-continuous monotone nondecreasing function with limiting values limx→−∞ FX (x) ≡ FX (−∞) = 0 and limx→∞ FX(x) ≡ FX(∞) = 1.

Since the measure ℙ is countably additive, μX is countably additive. Indeed, for any countable collection of pairwise disjoint Borel sets {Bi} the corresponding pre-images {X−1(Bi)} are pairwise disjoint sets in Ω. Hence,

${\mu }_X\left({\displaystyle \underset{i}{\cup }{B}_i}\right)=\mathbb{P}\left(X^{-1}\left({\displaystyle \underset{i}{\cup }{B}_i}\right)\right)=\mathbb{P}\left({\displaystyle \underset{\text{i}}{\cup }{X}^{-1}\left(B_i\right)}\right)={\displaystyle \sum _i\mathbb{P}\left(X^{-1}\left(B_i\right)\right)={\displaystyle \sum _i{\mu }_X\left(B_i\right).}}$

Moreover, the measure is normalized, μX (ℝ) = ℙ(X ∊ ℝ) = 1, so (ℝ, ℬ, μX) is in fact a probability space.

Since (ℝ, ℬ, μX) is a measure space, we can define a Lebesgue integral of a Borel function f(x) w.r.t. μX(x) over ℝ in a similar manner as the Lebesgue integral w.r.t. the measure m. For a simple function $\phi \left(x\right)={\displaystyle {\sum }_{k=1}^n{a}_k{\mathbb{I}}_{A_k}\left(x\right)},$ with Ak's as Borel sets in ℝ:

${\displaystyle {\int }_{ℝ}\phi \left(x\right)\text{d}{\mu }_X\left(x\right):\text{=}{\displaystyle \sum _{k=1}^n{a}_k{\mu }_X\left(A_k\right)}.\text{(9}\text{.23)}}$

This is the analogue of (9.13). Then, the Lebesgue integral w.r.t. μX(x) over ℝ for any nonnegative f is defined as the supremum over Lebesgue integral values of all nonnegative simple functions that are less than or equal to f:

${\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}{\mu }_X\left(x\right):\text{=}\text{sup}\left\{{\displaystyle {\int }_{ℝ}\phi \left(x\right)\text{d}{\mu }_X\left(x\right):\phi \text{is a simple function,}\text{0}\le \phi \le f}\right\}.}\text{(9}\text{.24)}$

The Lebesgue integral, w.r.t. μX over ℝ, of any Borel function f = f+ − f−, is then given by the difference of the two nonnegative Lebesgue integrals:

${\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}{\displaystyle {\int }_{ℝ}{f}_{\text{+}}\left(x\right)\text{d}{\mu }_X\left(x\right)-{\displaystyle {\int }_{ℝ}{f}_{-}\left(x\right)\text{d}{\mu }_X\left(x\right),}}}\text{(9}\text{.25)}$

and for any Borel set B in ℝ we have

${\displaystyle {\int }_{B}f\left(x\right)\text{d}{\mu }_X\left(x\right)}\text{=}{\displaystyle {\int }_{ℝ}{\mathbb{I}}_B}\left(x\right)f\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}{\displaystyle {\int }_{ℝ}{\mathbb{I}}_B\left(x\right)f_{+}\left(x\right)\text{d}{\mu }_X\left(x\right)-{\displaystyle {\int }_{ℝ}{\mathbb{I}}_{\mathbb{B}}\left(x\right)f_{-}\left(x\right)\text{d}{\mu }_X\left(x\right).}}\text{(9}\text{.26)}$

Using MCT and the sequence of simple functions in (9.18), the Lebesgue integral of any nonnegative function f, w.r.t. μX over ℝ, is given by

${\displaystyle {\int }_{ℝ}f\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}\underset{n\to \infty }{\lim }{\displaystyle {\int }_{ℝ}{f}_n\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}{\mu }_X\left(A_k^n\right),}}}\text{(9}\text{.27)}$

where $A_k^n:\text{=}{f}^{-1}\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right).$

Based on the above construction, we have the following result, which gives the expected value of a random variable g(X) as a Lebesgue integral of the (ordinary) function g(x) w.r.t. the distribution measure of X over ℝ. Here we assume that g(X) is integrable, i.e., E[|g(X)|] < ∞.

Theorem 9.4.

Given a random variable X on (Ω, F, ℙ) and a Borel function g : ℝ → ℝ,

$\text{E}\left[g\left(X\right)\right]\equiv {\displaystyle {\int }_{\text{Ω}}g\left(X\left(\omega \right)\right)\text{d}\mathbb{P}}\left(\omega \right)\text{=}{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_X\left(x\right)}.\text{(9}\text{.28)}$

Proof. In many analysis textbooks we find a standard way to prove this by first showing that (9.28) follows trivially for the simplest case of a Boolean indicator function g(x) = ?A(x) and then by the linearity property of integrals the result is shown to hold for any simple function $g\left(x\right)={\displaystyle {\sum }_{k=1}^n{a}_k{\mathbb{I}}_{A_k}\left(x\right).}$ Equation (9.28) is then shown to hold for any nonnegative function by using MCT and finally it follows for any g(x) = g+(x) − g−(x). As an alternate proof, it is now instructive to see how (9.28) follows directly using (9.27) for nonnegative g with the sequence gn defined as in (9.18) with f replaced by g:

$\begin{array}{l}{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}\underset{n\to \infty }{\lim }}{\displaystyle {\int }_{ℝ}{g}_n\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}\underset{n\to \infty }{\lim }{\displaystyle \sum _{k\text{=0}}^{2^{2n}}\frac{k}{2^n}{\mu }_X\left(A_k^n\right)}}\hfill \\ \text{=}\underset{n\to \infty }{\lim }{\displaystyle \sum _{k\text{=0}}^{2^{2n}}\frac{k}{2^n}\mathbb{P}\left(X^{-1}\left(A_k^n\right)\right)}\hfill \\ \text{=}\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}\mathbb{P}\left(g\left(X\right)\in \left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right)}\hfill \\ \equiv {\displaystyle {\int }_{\text{Ω}}g\left(X\left(\omega \right)\right)\text{d}\mathbb{P}}\left(\omega \right).\hfill \end{array}$

Here we used the definition in (9.21) for each set $A_k^n=g^{-1}\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right)$ and manipulated the set $X^{-1}\left(A_k^n\right)\equiv \left\{\omega \in \text{Ω}\text{:}X\left(\omega \right)\in A_k^n\right\}=\left\{\omega \in \text{Ω}\text{:}g\left(X\left(\omega \right)\right)\in g\left(A_k^n\right)\right\}=\left\{\omega \in \text{Ω}\text{:}g\left(X\left(\omega \right)\right)\in \left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right\}.$ Hence, (9.28) holds for both nonnegative parts g+ and g− of g, i.e., it must hold for any Borel function g.

The above expectation formula is useful if the integral on the right-hand side of (9.28) can be computed more explicitly. This is still in the form of a Lebesgue integral w.r.t. the measure μX. However, we can reduce this to more familiar forms depending on the type of random variable X.

In the simplest case of a constant random variable X ≡ a, the distribution measure is the Dirac measure, μX(·) = δa(·), i.e., for any Borel set B,

${\delta }_a\left(B\right):\text{=}\{\begin{array}{cc}\text{1}& \text{if}a\in B\\ \text{0}& \text{if}a\notin B.\end{array}\text{(9}\text{.29)}$

In particular, δa({a}) = 1 and δa({x}) = 0 for x ≠ a. By (9.22), the CDF is simply FX(x) ≔ μX((−∞, x]) = δa((−∞, x]) = ?{x≥a}. The expected value as a Lebesgue integral w.r.t. ℙ is trivially given since E[g(X)] = g(a)ℙ(Ω) = g(a). According to (9.28), this value must equal

$\text{E}\left[g\left(X\right)\right]\text{=}{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\delta }_a\left(x\right)\text{=}g\left(a\right).}\text{(9}\text{.30)}$

This gives us the formula for computing an integral w.r.t. the Dirac measure and is known as the sifting property since the Dirac measure picks out only the integrand value for x = a. Extending this to any purely discrete random variable that can take on distinct values ai with probability pi > 0, i = 1,2,..., ${\displaystyle {\sum }_i{p}_i}=1,$ the distribution measure is a linear combination of Dirac measures at each point in the range of $X:{\mu }_X\left(B\right)={\displaystyle {\sum }_i{p}_i{\delta }_{a_i}}\left(B\right).$. In particular, μX ({ai}) = pi. Then, using (9.30) and the fact that the integral w.r.t. a linear combination of measures is the linear combination of integrals w.r.t. each measure,

$\text{E}\left[g\left(X\right)\right]\text{=}{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}{\displaystyle \sum _i{p}_i{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\delta }_{a_i}\left(x\right)\text{=}{\displaystyle \sum _i{p}_{i}g\left(a_i\right).}}}}\text{(9}\text{.31)}$

Using (9.22), the CDF is given as the piecewise constant (staircase function),

$F_X\left(x\right)\text{=}{\displaystyle \sum _i{p}_i{\delta }_{a_i}\left(\left(-\infty ,x\right]\right)\text{=}{\displaystyle \sum _i{p}_i{\mathbb{I}}_{\left\{x\le a_i\right\}},}}\text{(9}\text{.32)}$

with jump discontinuities only at points x = ai, i.e., FX(ai) − FX(ai−) = pi and FX(x) − FX(x−) = 0 for all x values not equal to any ai. This is the familiar form for the CDF of a purely discrete random variable. Observe that, if g is continuous at the points ai, the expectation of g(X) is equal to the Riemann-Stieltjes integral of g with FX as integrator:

$\text{E}\left[g\left(X\right)\right]\text{=}{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{F}_X\left(x\right)\text{=}{\displaystyle \sum _{i}g\left(a_i\right)\left(F_X\left(a_i-\right)\right)\text{=}{\displaystyle \sum _{i}g\left(a_i\right)p_i.}}}\text{(9}\text{.33)}$

We can also define a distribution measure for a random variable X given as a so-called mixture of random variables with each having its own distribution measure, i.e., ${\mu }_X\left(B\right)={\displaystyle {\sum }_i{p}_i{\mu }_i}\left(B\right)$ where $p_i\ge 0,i=1,2,\mathrm{...},{\displaystyle {\sum }_i{p}_i}=1,$ and each μi is a distribution measure on (ℝ, ℬ). Then, the expected value of g(X) is given as a linear combination of Lebesgue integrals w.r.t. each measure μi:

$\text{E}\left[g\left(X\right)\right]\text{=}{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_X\left(x\right)\text{=}{\displaystyle \sum _i{p}_i{\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_i\left(x\right)}}}.\text{(9}\text{.34)}$

Let us now consider the application of Theorem 9.4 to the most common important case of a continuous random variable. This is the case where the random variable has a probability density function (PDF) fX(x). In this case there is a nonnegative integrable Borel function fX such that

${\mu }_X\left(B\right)={\displaystyle {\int }_B{f}_X\left(x\right)\text{d}m\left(x\right),\text{for all Borel sets }B,}\text{(9}\text{.35)}$

with CDF

$F_X\left(b\right):\text{=}{\displaystyle {\int }_{\left(-\infty ,b\right]}{f}_X\left(x\right)\text{d}m\left(x\right)}\text{(9}\text{.36)}$

for all b ∊ ℝ. In this case, the CDF FX is continuous and hence has no jumps, i.e., FX(x) = FX(x−) = FX(x+) for all x. Moreover, FX is not just continuous but in fact an absolutely continuous function. The reader may wish to consult a textbook on real analysis to learn more about this technical detail. It suffices here to point out that FX is differentiable and its derivative is the PDF: $F_X^{\text{'}}\equiv f_X.$ The distribution measure is said to be absolutely continuous w.r.t. the Lebesgue measure m. In this case, X is an absolutely continuous random variable but we simply say that it is a continuous random variable.3 Based on (9.35) it is easy to prove, using similar steps as in the above proof of (9.28) combined with the linearity property of the Lebesgue integral w.r.t. m and where ${\mu }_X\left(A_k^n\right)={\displaystyle {\int }_{A_k^n}{f}_X\left(x\right)\text{d}m\left(x\right)},$ that the expectation is a Lebesgue integral of gfX w.r.t. m:

$\text{E}\left[g\left(X\right)\right]\text{=}{\displaystyle {\int }_{-\infty }^{\infty }g\left(x\right)f_X\left(x\right)\text{d}m}\left(x\right).\text{(9}\text{.37)}$

In most (and for our purposes essentially all) applications, the density fX (when it exists) is bounded and continuous (a.e.) on ℝ, i.e., the CDF is the Riemann integral of the PDF,

$F_X\left(b\right):={\displaystyle {\int }_{-\infty }^b{f}_X\left(x\right)\text{d}x,b\in }ℝ.\left(9.38\right)$

Moreover, if g is continuous (a.e.) on ℝ, then (9.37) reduces to the familiar well-known formula for the expected value of g(X) as a Riemann integral:

$\text{E}\left[g\left(X\right)\right]={\displaystyle {\int }_{-\infty }^{\infty }g\left(x\right)f_X\left(x\right)\text{d}x,\left(9.39\right)}$

assuming both g± are Riemann-integrable, $\text{E}\left[|g\left(X\right)|\right]\equiv {\displaystyle {\int }_{-\infty }^{\infty }|g\left(x\right)|f_X\left(x\right)\text{d}x<\infty ,}$ i.e., E[g+] < ∞ and E[g−] < ∞.

The standard normal X ~ Norm(0,1) is an important example of a continuous random variable having positive Gaussian density $f_X\left(x\right)=n\left(x\right):=\frac{1}{\sqrt{2\pi }}{e}^{-x^2/2},$ for all real x. The distribution μX is absolutely continuous w.r.t. the Lebesgue measure. In fact, fX is bounded and continuous on ℝ with the CDF given by the Riemann integral:

$F_X\left(b\right):\text{=}\mathbb{P}\left(X\le b\right)\text{=}{\displaystyle {\int }_{-\infty }^{b}n\left(x\right)\text{d}x\equiv \mathcal{N}\left(b\right),b\in ℝ.}$

Clearly, ${\mathcal{N}}^{\prime }\left(x\right)=n\left(x\right)$ where $F_X\left(x\right)=\mathcal{N}\left(x\right)$ is a proper CDF since it is monotonically increasing from $\mathcal{N}\left(-\infty \right)=0$ to $\mathcal{N}\left(\infty \right)=1$.

We now wish to make one last connection of the expectation in (9.28) to the so-called Lebesgue—Stieltjes integral encountered in real analysis. For any type of random variable X, we then realize that E[g(X)], when g is continuous (a.e.), is simply a Riemann—Stieltjes integral of g with CDF FX as integrator. Beginning with (9.22), observe that for any semi-open interval (a,b] the probability ℙ(X ∊ (a,b]) is given equivalently by the distribution measure μX((a,b]) or the difference of the CDF values at the interval endpoints:

${\mu }_X\left(\left(a,b\right]\right)={\mu }_X\left(\left(-\infty ,b\right]\right)-{\mu }_X\left(\left(-\infty ,a\right]\right)=F_X\left(b\right)-F_X\left(a\right).\left(9.40\right)$

For any semi-open interval, ${\ell }_{F_X}\left(\left(a,b\right]\right):=F_X\left(b\right)-F_X\left(a\right)$ defines its "length relative to FX." Since ${\ell }_{F_X}\left(\left(a,c\right]\right)={\ell }_{F_X}\left(\left(a,b\right]\right)+{\ell }_{F_X}\left(\left(b,c\right]\right)$ for a < b < c, this length is additive (cumulative) for adjoining intervals. In the special case that FX(x) = x we recover the usual length b − a. In contrast to the usual length, an infinitesimal interval does not necessarily have length zero relative to FX since FX is a CDF that may have jump discontinuities. In fact, a singleton set has length equal to the size of the jump discontinuity of the CDF:

${\ell }_{F_X}\left(\left\{x\right\}\right)\text{=}\underset{\in \to \infty }{\lim }{\ell }_{F_X}\left(\left(x-\in ,x\right]\right)\text{=}{F}_X\left(x\right)-\underset{\in \to \infty }{\lim }{F}_X\left(x-\in \right)\text{=}{F}_X\left(x\right)-F_X\left(x-\right).$

For a purely continuous random variable X there are no jumps so all points have zero such length, but for a random variable having a discrete part there is a nonzero length given by the PMF values pi = FX(xi) − FX(xi−) at the points corresponding to the countable set of discrete values {xi} of X. For the other types of intervals we have ${\ell }_{F_X}\left(\left(a,b\right]\right)=F_X\left(b\right)-F_X\left(a-\right),{\ell }_{F_X}\left(\left(a,b\right)\right)=F_X\left(b-\right)-F_X\left(a\right),{\ell }_{F_X}\left(\left(a,b\right]\right)=F_X\left(b-\right)-F_X\left(a-\right).$.

Based on the above definition of the length ℓF, for a given CDF F, the definition of the Lebesgue measure m is generalized to the Lebesgue—Stieltjes measure generated by F. This measure is denoted by mF. The measure mF : ℝ, → [0, 1], defined for any Borel set B ∊ ℬ(ℝ), is the smallest total length relative to F of all countable unions of semi-open intervals in ℝ that contain B:

$m_F\left(B\right):=\inf \left\{{\displaystyle \sum _{n=1}^{\infty }{\ell }_F\left(I_n\right):I_n=\left(a_n,b_n\right],a_n\le b_{n,}B\subset {\displaystyle \underset{n=1}{\overset{\infty }{\cup }}{I}_n}}\right\}.\left(9.41\right)$

Hence, mF is a measure that assigns a value in [0, 1] for each Borel set B, such that all semi-open intervals In = (an, bn] have measure mF((an, bn]) = ℓF(an, bn]) = F(bn) − F(an). All intervals, including semi-infinite intervals, have finite measure; in particular, mF(ℝ) = F(∞) − F(−∞) = 1− 0 = 1. This measure is countably additive on ℬ(ℝ):

$m_F\left({\displaystyle \underset{n=1}{\overset{\infty }{\cup }}{B}_n}\right)={\displaystyle \sum _{n=1}^{\infty }{m}_F}\left(B_n\right)$

for all disjoint Borel sets Bn, n ≥ 1. Because of the equivalence relation in (9.40) and the fact that the σ-algebra generated by all semi-open intervals in ℝ, is ℬ(ℝ), the distribution measure is the same as the Lebesgue—Stieltjes measure generated by the CDF, i.e., ${\mu }_X\left(B\right)=m_{F_X}\left(B\right)$ for every Borel set B. For example, X ≡ a has CDF FX(x) = ?{x ≥ a}. So, the measure $m_{F_X}={\delta }_a$ is the Dirac measure concentrated at a. For any purely discrete random variable, with CDF in (9.32), the measure $m_{F_X}={\displaystyle {\sum }_i{p}_i{\delta }_{a_i}}$ is a weighted sum of the Dirac measures for each point with its corresponding PMF value, i.e., $m_{F_X}\left(B\right)={\displaystyle {\sum }_i{p}_i{\delta }_{a_i}\left(B\right).}$

The Lebesgue—Stieltjes integral of a function w.r.t. $m_{F_X}$ is defined in the same manner as in (9.23), (9.24), (9.25), and (9.26) with the notation that μX is replaced by $m_{F_X}$. In particular, the expectation in (9.28) is now recognized as the Lebesgue-Stieltjes integral of g w.r.t. $m_{F_X}$ over ℝ, denoted by ${\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{m}_{F_X}\left(x\right),}$ where we write equivalently

$\text{E}\left[g\left(X\right)\right]={\displaystyle {\int }_{ℝ}g}\left(x\right)\text{d}{\mu }_X\left(x\right)\equiv {\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{m}_{F_X}\left(x\right).}\left(9.42\right)$

If g is continuous (a.e.) then it can be shown that the Lebesgue-Stieltjes integral is the same as the corresponding Riemann-Stieltjes integral of g with CDF FX as integrator function:

$\text{E}\left[g\left(X\right)\right]={\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{m}_{F_X}\left(x\right)={\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{F}_X\left(x\right).}\left(9.43\right)}$

Since any CDF is a monotone (nondecreasing) bounded function, then it is of bounded variation and hence can always be used as integrator. If X is absolutely continuous, then dFX(x) = F'X(x) dx = ƒX(x) dx and the Riemann-Stieltjes integral is the same as the usual Riemann integral in (9.39). For a purely discrete random variable with CDF in (9.32), the Riemann-Stieltjes integral in (9.43) is given by (9.33). The Riemann-Stieltjes integral in (9.43) also gives E[g(X)] for all other types of mixture random variables, as shown in Section 11.2.1, for example. Such random variables can have a discrete and continuous part (the continuous part being either absolutely continuous and/or singularly continuous).

Based on the above expectation formulas, one can also proceed to compute various quantities such as the moments E[Xn], n ≥ 1, of a real-valued random variable X (assuming E[|X|n] < ∞); the moment generating function MX(t) ≔ E[etX], which is either infinite or a function of the real parameter t on some interval of convergence about t = 0; the characteristic function ${\phi }_X\left(t\right):=\text{E}\left[{\text{e}}^{itX}\right],i\equiv \sqrt{-1},$ which is bounded for all t ∊ ℝ, i.e., |φX(t)| ≤ E[|eitX|] = 1. The characteristic function (or the moment generating function) is useful for computing the mean and variance as well as various moments of a random variable. The relevant formulas and theorems related to these functions are part of standard material that is covered in most textbooks on probability theory and are hence (in the interest of space) simply omitted here.

In closing this section we mention one other general result, which is the change of variable formula for an expectation given in (9.44) just below. This allows us to compute the expectation E[g(X)] as an integral w.r.t. the distribution measure μY of the random variable defined by Y ≔ g(X). Note that, since X is a random variable on (Ω, F, ℙ), then Y is also a random variable on (Ω, F, ℙ). That is, g is a Borel function, g−1(B) ∊ ℬ, giving Y−1(B) = X−1(g−1(B)) ∊ F for every B ∊ ℬ. Assuming Y is integrable, i.e., E[|Y|] < ∞, then

$\text{E}\left[Y\right]:={\displaystyle {\int }_{\text{Ω}}Y\left(\omega \right)\text{d}\mathbb{P}}\left(\omega \right)={\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_X\left(x\right)={\displaystyle {\int }_{ℝ}y\text{d}{\mu }_Y\left(y\right),\left(9.44\right)}}$

where μY(B) ≔ ℙ(Y—1(B)) ≡ ℙ(Y ∊ B), for every B ∊ ℬ. The proof of this formula follows readily from the relation between the two distribution measures: μY(B) ≔ ℙ(Y−1(B)) = ℙ(X−1 (g−1 (B))) = μX(g−1(B)). Hence, in the first equation line in the proof of Theorem 9.4. we have ${\mu }_X\left(A_k^n\right)={\mu }_X\left(g^{-1}\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right)\right)={\mu }_Y\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right),$, i.e.,

${\displaystyle {\int }_{ℝ}g\left(x\right)\text{d}{\mu }_X\left(x\right)=\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}{\mu }_X\left(A_k^n\right)=\underset{n\to \infty }{\lim }{\displaystyle \sum _{k=0}^{2^{2n}}\frac{k}{2^n}{\mu }_Y\left(\left[\frac{k}{2^n},\frac{k+1}{2^n}\right)\right)={\displaystyle {\int }_{ℝ}y\text{d}{\mu }_Y\left(y\right)}}}}$

which proves the formula. In summary, we see that E[g(X)] can be evaluated in three different ways: (i) by integrating the random variable Y ≡ g(X) w.r.t. ℙ over Ω, (ii) by integrating the function g(x) w.r.t. distribution measure μX(x) over ℝ, or (iii) by integrating the function ƒ(y) = y w.r.t. distribution measure μY(y) over ℝ.

9.2 Multidimensional Lebesgue Integration

The above integration theory for Borel functions of a single variable (and random variables defined as functions of a single random variable) extends into the general multidimensional case. The construction of Lebesgue integrals mirrors the above single-variable case. We recall the Borel sets in ℝn, ℬn ≡ ℬ(ℝn) and Borel functions defined over ℝn in Section 6.2.2. In ℝ2, we denote the Lebesgue measure by m2 : ℬ2 → [0, ∞]. It can be defined formally as an extension of the definition given above for the Lebesgue measure m. Given any two intervals I1, I2, then m2 measures the area of the rectangle I1 × I2: m2(I1 × I2) = ℓ(I1)ℓ(I2). In terms of the Lebesgue measure in one dimension we have the product measure m2(I1 × I2) = m(I1)m(I2). Note that the null sets (having zero measure w.r.t. m2) include any countable union of points in ℝ2 as well as some uncountable sets of the form A × {b}, A ⊂ ℝ, b ∊ ℝ or {a} × B, a ∊ ℝ, B ⊂ ℝ. Also, any graph or curve in ℝ2 has zero m2 measure. The Lebesgue integral of a Borel function over ℝ2 w.r.t. measure m2 is defined in similar fashion to what we have above for the single variable case. A simple Borel function $\phi \left(x,y\right)={\displaystyle {\sum }_{k=1}^n{a}_k{\mathbb{I}}_{A_k}\left(x,y\right),a_k\in ℝ,}$ with all Ak ∊ ℬ2, has Lebesgue integral

${\displaystyle {\int }_{{ℝ}^2}\phi \left(x,y\right)\text{d}{m}_2\left(x,y\right)}:={\displaystyle \sum _{k=1}^n{a}_k{m}_2\left(A_k\right).\left(9.45\right)}$

The Lebesgue integral of any nonnegative Borel function ƒ is defined by

${\displaystyle {\int }_{{ℝ}^2}f\left(x,y\right)\text{d}{m}_2\left(x,y\right):=\sup \left\{{\displaystyle {\int }_{{ℝ}^2}\phi \left(x,y\right)\text{d}{m}_2\left(x,y\right):\phi \text{is}\text{a}\text{simple}\text{function,}0\le \phi \le f}\right\}.\left(9.46\right)}$

The Lebesgue integral, w.r.t. m2 over ℝ2, of any Borel function ƒ = ƒ+ − ƒ−, with ƒ± ≥ 0, is then

${\displaystyle {\int }_{{ℝ}^2}f\text{d}{m}_2\equiv {\displaystyle {\int }_{{ℝ}^2}f\left(x,y\right)\text{d}{m}_2\left(x,y\right)}={\displaystyle {\int }_{{ℝ}^2}{f}_{+}\left(x,y\right)\text{d}{m}_2\left(x,y\right)}-{\displaystyle {\int }_{{ℝ}^2}f-\left(x,y\right)\text{d}{m}_2\left(x,y\right)}.\left(9.47\right)}$

ƒ is integrable iff ∫ℝ2 |ƒ| dm2 < ∞. We denote this by writing ƒ ∊ L1(ℝ2, ℬ2, m2). The Lebesgue integral of ƒ over any Borel set B ⊂ ℝ2 is the Lebesgue integral of ?B ƒ over ℝ2:

${\displaystyle {\int }_{B}f\text{d}{m}_2={\displaystyle {\int }_{{ℝ}^2}{\mathbb{I}}_B\left(x,y\right)f\left(x,y\right)\text{d}{m}_2\left(x,y\right).\left(9.48\right)}}$

Assuming an integrable function, ƒ ∊ L1(ℝ2, ℬ2, m2), then Fubini's Theorem can be applied for interchanging the order of integration:

${\displaystyle {\int }_{{ℝ}^2}f\text{d}{m}_2={\displaystyle {\int }_{ℝ}\left({\displaystyle {\int }_{ℝ}f\left(x,y\right)\text{d}m\left(x\right)}\right)\text{d}m\left(y\right)={\displaystyle {\int }_{ℝ}\left({\displaystyle {\int }_{ℝ}f\left(x,y\right)\text{d}m\left(y\right)}\right)\text{d}m\left(x\right).\left(9.49\right)}}}$

What is important for us is when ƒ is continuous (m2—a.e.) in (9.47) or ?B ƒ is continuous in (9.48). The Lebesgue integral in (9.47) is then equal to the Riemann (double) integral over ℝ2:

${\displaystyle {\int }_{{ℝ}^2}f\text{d}{m}_2={\displaystyle {\iint }_{{ℝ}^2}f\left(x,y\right)\text{d}x\text{d}y={\displaystyle {\int }_{ℝ}\left({\displaystyle {\int }_{ℝ}f\left(x,y\right)\text{d}x}\right)\text{d}y={\displaystyle {\int }_{ℝ}\left({\displaystyle {\int }_{ℝ}f\left(x,y\right)\text{d}y}\right)\text{d}x\left(9.50\right)}}}}$

where we assume ƒ ∊ L1(ℝ2, ℬ2, m2), i.e., the function is integrable. In all our applications this will be the case where, for fixed x ∊ ℝ, ƒ is continuous in y and for fixed y ∊ ℝ, ƒ is continuous in x. The set B in (9.48) is usually a rectangular region [a, b] × [c, d], or of type a ≤ x ≤ b,h1(x) ≤ y ≤ h2(x), or g1(y) ≤ x ≤ g2(y),c ≤ y ≤ d, etc. Given a set B = B1 × B2 = {(x, y) ∊ ℝ2 : x ∊ B1, y ∊ B2}, and assuming ?B ƒ is continuous and integrable on ℝ2, the Lebesgue integral in (9.48) is then

${\displaystyle {\int }_{B_1\times B_2}f\text{d}{m}_2={\displaystyle {\int }_{B_2}\left({\displaystyle {\int }_{B_1}f\left(x,y\right)\text{d}x}\right)\text{d}y={\displaystyle {\int }_{B_1}\left({\displaystyle {\int }_{B_2}f\left(x,y\right)\text{d}y}\right)\text{d}x.\left(9.51\right)}}}$

We note that when the integrals only have meaning as Lebesgue integrals, we interpret the Riemann integrals as convenient shorthand notation for the corresponding Lebesgue integrals.

In ℝ3, the Lebesgue measure, m3,: ℬ3 → [0, ∞], measures the volume of I1 × I2 × I3, where m3(I1 × I2 × I3) = ℓ(I1)ℓ(I2)ℓ(I3). For any B = B1 × B2 × B3 ∊ ℬ3, we have the product measure m3(B) = m(B1)m(B2)m(B3). The null sets of m3 include any countable union of points in ℝ3 as well as uncountable sets of the form A × B ×{c}, A, B ⊂ ℝ,c ∊ ℝ, or A × {b} × C, A, C ⊂ ℝ, b ∊ ℝ, or {a} × B × C, B, C ⊂ ℝ, a ∊ ℝ, all surfaces and lines, etc.. The Lebesgue integral of a Borel function ƒ: ℝ3 → ℝ w.r.t. measure m3 over ℝ3 is defined in analogy with the above construction in ℝ2. If ƒ(x1, x2, x3) is integrable w.r.t. m3 over ℝ3 (denoted as ƒ ∊ L1(ℝ3, ℬ3, m3)) and is furthermore a continuous function (m3—a.e.) of the three variables, then its Lebesgue integral is a Riemann (triple) integral over ℝ3:

${\displaystyle {\int }_{{ℝ}^3}f\text{d}{m}_3={\displaystyle {\int }_{ℝ}{\displaystyle {\int }_{ℝ}{\displaystyle {\int }_{ℝ}f\left(x_1,x_2,x_3\right)\text{d}{x}_1\text{d}{x}_2\text{d}{x}_3,\left(9.52\right)}}}}$

where we can also change the order of integration by successive application of Fubini's Theorem. For a Borel set B = B1 × B2 × B3 = {(x1, x2, x3) ∊ ℝ3 : x1 ∊ B1, x2 ∊ B2, x3 ∊ B3} we have

$\begin{array}{ll}{\displaystyle {\int }_{B_1\times B_2\times B_3}f\text{d}{m}_3}\hfill & ={\displaystyle {\int }_{B_3}{\displaystyle {\int }_{B_2}{\displaystyle {\int }_{B_1}f\left(x_1,x_2,x_3\right)\text{d}m\left(x_1\right)\text{d}m\left(x_2\right)\text{d}m\left(x_3\right)}}}\hfill \\ \hfill & ={\displaystyle {\int }_{B_3}{\displaystyle {\int }_{B_2}{\displaystyle {\int }_{B_1}f\left(x_1,x_2,x_3\right)\text{d}{x}_1\text{d}{x}_2\text{d}{x}_3}}}\hfill \\ \hfill & ={\displaystyle {\int }_{{ℝ}^3}f\left(x_1,x_2,x_3\right){\mathbb{I}}_B\left(x_1,x_2,x_3\right)\text{d}{x}_{1}d{x}_2\text{d}{x}_3,}\hfill \end{array}\text{ (9.53)}$

where the Riemann integral is used as shorthand for the Lebesgue integral and is equivalent to it when ƒ(x1,x2,x3) ?B(x1,x2,x3) is a continuous function of x1,x2,x3.

More generally, in ℝn the Lebesgue measure, mn : Bn → [0, ∞], gives the n-dimensional volume of any n-dimensional cube: mn(I1 × ... × In) = ℓ(I1) × ... × ℓ(In). For every cartesian n-tuple B = B1 × ... × Bn ∊ ℬn, mn(B) = m(B1)m(B2) ··· m(Bn) is a product measure. Null sets of mn are sets having zero n-dimensional volume and these include any countable union of points in ℝn, hyperplanes, lines, etc. The Lebesgue integral of a Borel function ƒ: ℝn → ℝ w.r.t. mn over ℝn for all n ≥ 2 is constructed as in the above case of n = 2. If ƒ ∊ L1(ℝn, ℬn, mn), i.e., ƒ(x) ≡ ƒ(x1,...,xn) is integrable w.r.t. mn over ℝn, and is furthermore a continuous function (mn—a.e.) of the n variables x, then its Lebesgue integral is equal to its Riemann integral over ℝn:

${{\displaystyle {\int }_{{ℝ}^n}f\text{d}m}}_n={\displaystyle {\int }_{ℝ}\mathrm{...}}{\displaystyle {\int }_{ℝ}f\left(x_1,\mathrm{...},x_n\right)\text{d}{x}_1\mathrm{...}\text{d}{x}_n.\left(9.54\right)}$

For a Borel set B = B1 × ... × Bn = {x ∊ ℝn : x1 ∊ B1,...,xn ∊ Bn} we have

$\begin{array}{ll}{\displaystyle {\int }_{B_1\times ...\times B_n}f\text{d}{m}_n}\hfill & ={\displaystyle {\int }_{B_n}\cdots {\displaystyle {\int }_{B_1}f\left(x_1,\cdots ,x_n\right)\text{d}m\left(x_1\right)...\text{d}m\left(x_n\right)}}\hfill \\ \hfill & ={\displaystyle {\int }_{B_n}\cdots {\displaystyle {\int }_{B_1}f\left(x_1,...,x_n\right)\text{d}{x}_1...\text{d}{x}_n}}\hfill \\ \hfill & ={\displaystyle {\int }_{{ℝ}^n}f\left(\text{x}\right){\mathbb{I}}_B\left(x\right){\text{d}}^n\text{x},}\hfill \end{array}\text{ (9.55)}$

where the Riemann integral is used as shorthand for the Lebesgue integral and is equivalent to it when ƒ(x)?B(x) is continuous on ℝn.

9.3 Multiple Random Variables and Joint Distributions

Let us now see how distributions and expectations are formulated for multiple random variables (i.e., random vectors) by first considering a pair of random variables (X, Y) : Ω → ℝ2 defined on the same probability space (Ω, ℱ, ℙ). The joint distribution measure

${\mu }_{X,Y}:{ℬ}_2\to \left[0,1\right]$

is the measure induced by the pair (X, Y) and defined by

${\mu }_{X,Y}\left(ℬ\right):=\mathbb{P}((X,Y)\in B),B\in {ℬ}_2.\left(9.56\right)$

This measure is countably additive and assigns a number in [0, 1] to a Borel set B in ℝ2 which corresponds to the probability of the event {(X, Y) ∊ B} ≡ {ω ∊ Ω: (X(ω), Y (ω)) ∊ B}. This measure is normalized so that μX,Y(ℝ2) = ℙ((X, Y) ∊ ℝ2) = 1 and so the measure space (ℝ2, ℬ2, μX,Y) is also a probability space. Writing B = B1 × B2, B1, B2 ∊ ℬ, then we see that

${\mu }_{X,Y}\left(B_1\times B_2\right)=\mathbb{P}(X^{-1}(B_1)\cap Y^{-1}(B_2))=\mathbb{P}(X\in B_1,Y\in B_2)\left(9.57\right)$

gives the probability of the joint event {X ∊ B1} ∩ {Y ∊ B2} ≡ {X ∊ B1, Y ∊ B2}. This joint measure determines the univariate (marginal distribution) measures of X and Y by letting B1 = ℝ or B2 = ℝ:

${\mu }_{X,Y}\left(B\times ℝ\right)=\mathbb{P}(X\in B,Y\in ℝ)=\mathbb{P}(X\in B)={\mu }_x(B),\left(9.58\right)$

${\mu }_{X,Y}\left(ℝ\times B\right)=\mathbb{P}(X\in ℝ,Y\in B)=\mathbb{P}(X\in B)={\mu }_Y(B),\left(9.59\right)$

for all B ∊ ℬ.

Letting B1 = (−∞, x],B2 = (−∞, y] in (9.57) gives the joint CDF of (X,Y):

$F_{X,Y}(x,y):={\mu }_{X,Y}((-\infty ,x]\times (-\infty ,y])=\mathbb{P}(X\le x,Y\le y),x,y\in ℝ.\left(9.60\right)$

This CDF is right-continuous on ℝ2, monotone in both x and y, and recovers the univariate (marginal) CDF of X or Y in the respective limits:

$\underset{y\to \infty }{\lim }{F}_{X,Y}(x,y)\equiv F_{X,Y}(x,\infty )=\mathbb{P}(X\le x)={\mu }_X((-\infty ,x])=F_X(x),x\in ℝ,\left(9.61\right)$

$\underset{x\to \infty }{\lim }{F}_{X,Y}(x,y)\equiv F_{X,Y}(\infty ,y)=\mathbb{P}(Y\le y)={\mu }_Y((-\infty ,y])=F_Y(y),y\in ℝ.\left(9.62\right)$

Taking the limit of infinite argument in the marginal CDF (in either case) gives

$\underset{x\to \infty ,y\to \infty }{\lim }{F}_{X,Y}(x,y)\equiv F_{X,Y}(\infty ,\infty )=F_X(\infty )=F_Y(\infty )=1.$

Taking a decreasing sequence of numbers xn ↘ −∞ gives a decreasing sequence of sets approaching the empty set: (−∞, xn] × (−∞, y] ↘ ø. By monotone continuity (from above) of the measure, μX,Y ((−∞, xn] × (−∞, y]) ↘ μX,Y(ø) = 0, i.e., for any y ∊ ℝ,

$F_{X,Y}(-\infty ,y)\equiv \underset{x\to -\infty }{\lim }{F}_{X,Y}(x,y)=\underset{x_n\searrow -\infty }{\lim }{\mu }_{X,Y}((-\infty ,x_n]\times (-\infty ,y])={\mu }_{X,Y}(\cancel{0})=0.$

Similarly, ${}_{y\to -\infty }^{\lim }{F}_{X,Y}\left(x,y\right)\equiv F_{X,Y}\left(x,-\infty \right)=0,x\in ℝ$. These two relations must clearly hold since X and Y are in ℝ and so ℙ(X < −∞, Y ≤ y) = 0 and ℙ(X ≤ x, Y < −∞) = 0.

Let x1, x2, y1, y2 be real numbers such that x1 < x2 and y1 < y2; then the joint measure of the semi-open rectangle (x1, x2] × (y1, y2] is given by

$\begin{array}{l}{\mu }_{X,Y}((x_1,x_2]\times (y_1,y_2])=\mathbb{P}(x_1<X\le x_2,y_1<Y\le y_2)\\ =F_{X,Y}(x_2,y_2)-F_{X,Y}(x_1,y_2)-F_{X,Y}(x_2,y_1)+F_{X,Y}(x_1,y_1).\left(9.63\right)\end{array}$

Based on this relation, the definition of a Lebesgue—Stieltjes measure for a single random variable, defined in (9.41), can be extended to a Lebesgue—Stieltjes measure generated by the joint CDF FX,Y of (X, Y). The quantity in (9.63) can be viewed as a measure of an "area relative to the joint CDF" FX,Y for any semi-open rectangle in ℝ2. The Lebesgue—Stieltjes measure generated by FX,Y, which we denote by $m_F^{X,Y},$ is the measure function $m_F^{X,Y}:{ℝ}^2\to \left[0,1\right]$, defined for any Borel set B = B1 × B2 ∊ ℬ2, that assigns the smallest total area relative to FX,Y (using (9.63)) of all countable unions of semi-open rectangles Ik × Jl ≡ (ak,bk] × (cl,dl], ak ≤ bk, cl ≤ dl in ℝ2 that contain B:

$m_F^{X,Y}(B):=\inf \left\{{\displaystyle \sum _{k=1}^{\infty }{\displaystyle \sum _{l=1}^{\infty }{\mu }_{X,Y}(I_k\times J_l):B_1\subset {\displaystyle \underset{k=1}{\overset{\infty }{\cup }}{I}_k,B_2\subset {\displaystyle \underset{l=1}{\overset{\infty }{\cup }}{J}_l}}}}\right\}.\left(9.64\right)$

This measure is equivalent to the joint distribution measure, i.e., $m_F^{X,Y}\left(B\right)={\mu }_{X,Y}\left(B\right)$.

Since (ℝ2, B2, μX,Y) is a measure, we can define the Lebesgue integral w.r.t. μX,Y over ℝ2 (i.e., the Lebesgue—Stieltjes integral w.r.t. μX,Y or equivalently w.r.t. $m_F^{X,Y}$) in very similar manner as was done above for the Lebesgue—Stieltjes integral w.r.t. μX in (9.23) - (9.68). For any simple function

$\phi (x,y)={\displaystyle \sum _{k=1}^K{\displaystyle \sum _{l=1}^L{a}_{k,l}{?}_{B_1^k\times B_2^l}(x)}}$

with $B_1^k\times B_2^l\in {ℬ}_2,a_{k,l}\in ℝ,$, its Lebesgue integral w.r.t. μX,Y over ℝ2 is defined by

${\displaystyle {\int }_{{ℝ}^2}\phi (x,y)\text{d}{\mu }_{X,Y}(x,y):={\displaystyle \sum _{k=1}^K{\displaystyle \sum _{l=1}^L{a}_{k,l}{\mu }_{X,Y}(B_1^k\times }{B}_1^l).\left(9.65\right)}}$

Based on this definition, the Lebesgue—Stieltjes integral w.r.t. μX,Y over ℝ2 for any non-negative Borel function ƒ : ℝ2 → ℝ is defined as the supremum over integral values of all nonnegative simple functions φ ≤ ƒ:

${\displaystyle {\int }_{{ℝ}^2}f(x,y)\text{d}{\mu }_{X,Y}(x,y)}:=\sup \left\{{\displaystyle {\int }_{{ℝ}^2}\phi (x,y)\text{d}{\mu }_{X,Y}(x,y)}:\phi \text{ is simple, }0\le \phi \le f\right\}.\left(9.66\right)$

For any Borel function ƒ = ƒ+ − ƒ−, the Lebesgue—Stieltjes integral is given by the difference of the two nonnegative integrals:

${\displaystyle {\int }_{{ℝ}^2}f\left(x,y\right)\text{d}{\mu }_{X,Y}\left(x,y\right)}={\displaystyle {\int }_{{ℝ}^2}{f}_{+}\left(x,y\right)\text{d}{\mu }_{X,Y}\left(x,y\right)}-{\displaystyle {\int }_{{ℝ}^2}{f}_{-}\left(x,y\right)\text{d}{\mu }_{X,Y}\left(x,y\right)},\left(9.67\right)$

and for any Borel set B in ℝ2 we have

${\displaystyle {\int }_{B}f(x,y)\text{d}{\mu }_{X,Y}(x,y)}={\displaystyle {\int }_{{ℝ}^2}{?}_B(x,y)f(x,y)}\text{d}{\mu }_{X,Y}(x,y).\left(9.68\right)$

We note that MCT is a general property that also applies to all Lebesgue—Stieltjes integrals.

Based on the above construction, we have the following result for the expected value of h(X, Y)(ω) ≡ h(X(ω),Y(ω)), defined as a Borel function of two random variables (X, Y). Here we assume that h(X, Y) is integrable, i.e., E[|h(X, Y)|] < ∞.

Theorem 9.5.

Given a pair of random variables (X, Y) on (Ω, ℱ, ℙ) and a Borel function h : ℝ2 → ℝ,

$\text{E}[h(X,Y)]\equiv {\displaystyle {\int }_{\Omega }h(X(\omega ),Y(\omega ))\text{d}\mathbb{P}(\omega )={\displaystyle {\int }_{{ℝ}^2}h(x,y)\text{d}{\mu }_{X,Y}(x,y).\left(9.69\right)}}$

The proof of (9.69) is very similar to the proof of (9.28). In the special case h(x, y) = g(x), (9.69) recovers (9.28). The Lebesgue—Stieltjes integral in (9.69) is a very general representation for E[h(X, Y)] where h is a, Borel function. That is, X or Y can be any type of random variable, i.e., any combination of discrete, absolutely continuous, or singularly continuous. The expectation in (9.69) reduces to various useful and familiar formulas for E[h(X, Y)] that a student learns in a standard course in probability theory. For our purpose, there are two main cases: discrete or continuous (we simply say continuous to mean absolutely continuous).

Assume that h is a, continuous function (m2—a.e.). This is virtually always the case in practice and certainly the case for all our applications in this text. The Lebesgue—Stieltjes is then a Riemann—Stieltjes integral over ℝ2. Let us consider the simple case where both X and Y are discrete random variables and h(x, y) is continuous at all values (x, y) = (xi, yj) in the range of (X, Y); then the Riemann—Stieltjes integral simply recovers the summation formula in the joint PMF pX,Y (x, y) ≡ ℙ(X = x, Y = y) of (X, Y) at the support values:

$\text{E}[h(X,Y)]={\displaystyle \sum _{\text{all }{x}_i}{\displaystyle \sum _{\text{all }{y}_j}{p}_{X,Y}(x_i,y_j)h(x_i,y_j)}}$

assuming E[|h(X, Y)|] < ∞ (i.e., summation converging for both negative and positive parts of h). Letting h(X, Y) = ?{X ≤ x,Y ≤ y}, for any fixed real values (x, y), recovers the joint CDF as a (two-dimensional piecewise constant) staircase function in (x, y):

$F_{X,Y}(x,y)=\mathbb{P}(X\le x,Y\le y)=\text{E}[{?}_{\left\{X\le x,Y\le y\right\}}]={\displaystyle \sum _{x_i\le x}{\displaystyle \sum _{y_j\le y}{p}_{X,Y}(x_i,y_j)}}$

with jump discontinuities at only the support values (x, y) = (xi, yj) of the PMF. This recovers the formula in (9.32) when y → ∞.

Let us now consider the case where (X, Y) are continuous with joint density denoted by ƒX,Y. In this case, every Borel set B ∊ ℬ2 has joint measure given by the Lebesgue integral of a nonnegative integrable Borel function ƒX,Y :ℝ2→ ℝ (namely, the joint PDF) over B:

${\mu }_{X,Y}(B)={\displaystyle {\int }_B{f}_{X,Y}(x,y)\text{d}{m}_2(x,y)\equiv {\displaystyle {\int }_{-\infty }^{\infty }{\displaystyle {\int }_{-\infty }^{\infty }{?}_B(x,y)f_{X,Y}(x,y)\text{d}x\text{d}y.\left(9.70\right)}}}$

Recall that we sometimes simply write the Lebesgue integral as a Riemann integral using the convention we adopted in the previous section. Of course, the two are equal if ?B ƒX,Y is a continuous function in (x, y). Since (9.70) holds for all B ∊ ℬ2, then it holds for all sets of the form B = (−∞, a] × (−∞, b]. Using ?B(x, y) = ?{x ≤ a,y≤b} and the definition (9.60) we have the joint CDF

$F_{X,Y}(a,b)={\displaystyle {\int }_{-\infty }^b{\displaystyle {\int }_{-\infty }^a{f}_{X,Y}(x,y)\text{d}x\text{d}y,a,b\in ℝ.\left(9.71\right)}}$

Since FX,Y (∞, ∞) = 1, then ƒX,Y integrates to unity on all of ℝ2. In fact, (9.71) holds iff (9.70) holds for all Borel sets B ∊ ℬ2. The joint CDF is continuous on ℝ2 and related to the joint PDF by differentiating (9.71),

$f_{X,Y}(x,y)=\frac{{\partial }^2}{{\partial }_x{\partial }_y}{F}_{X,Y}(x,y),x,y\in ℝ.$

The marginal CDF of X and Y are given by (9.61)—(9.62) and taking either limit a → ∞ or b → ∞ in (9.71) gives

$F_X(a)={\displaystyle {\int }_{-\infty }^a\left({\displaystyle {\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)\text{d}y}\right)dx\text{ and }{F}_Y(b)={\displaystyle {\int }_{-\infty }^b\left({\displaystyle {\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)\text{d}x}\right)}\text{ d}y,}$

for all a, b ∊ ℝ. Hence, the existence of the joint PDF ƒX,Y implies the existence of the respective marginal densities of X and Y:

$f_X(x)={\displaystyle {\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)\text{d}y\text{ and }{f}_Y(y)={\displaystyle {\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)\text{d}x.\left(9.72\right)}}$

We note that the converse is generally not true. Recall from our discussion of a single random variable, the marginal densities are nonnegative Borel functions that exist whenever (see (9.35) and (9.36))

${\mu }_X(B)={\displaystyle {\int }_B^{}{f}_X(x)\text{d}x\text{ and }{\mu }_Y(B)={\displaystyle {\int }_B^{}{f}_Y(y)\text{d}y}}$

for all Borel sets B ⊂ ℝ, or equivalently whenever

$F_X(a)={\displaystyle {\int }_{-\infty }^a{f}_X(x)\text{d}x\text{ and }{F}_Y(b)={\displaystyle {\int }_{-\infty }^b{f}_Y(y)\text{d}y}}$

for all a, b ∊ ℝ. The expectations E[h1(X)] and E[h2(Y)], for single-variable Borel functions h1 and h2, are therefore given by the respective Riemann (Lebesgue) integrals over ℝ:

$\text{E}[h_1(X)]={\displaystyle {\int }_{ℝ}{h}_1(x)f_X(x)\text{d}x\text{ and E}[h_2(Y)]={\displaystyle {\int }_{ℝ}{h}_2(y)f_Y(y)\text{d}y.\left(9.73\right)}}$

For jointly continuous (X, Y), (9.70) holds, and it is readily proven (in the same manner that (9.37) or (9.39) is proven) that the expected value in (9.69) is given by the integral over ℝ2 of the joint PDF multiplied by h, i.e.,

$\text{E}[h(X,Y)]={\displaystyle {\int }_{-\infty }^{\infty }{\displaystyle {\int }_{-\infty }^{\infty }h(x,y)f_{X,Y}(x,y)\text{d}x\text{d}y.\left(9.74\right)}}$

Recall from Definition 6.13 that X and Y are mutually independent if

$\mathbb{P}(X\in B_1,Y\in B_2)=\mathbb{P}(X\in B_1)\mathbb{P}(Y\in B_2)\left(9.75\right)$

for all B1, B2 ∊ ℬ(ℝ). That is, for a Borel rectangle B = B1 × B2 the joint distribution measure given by (9.57) is now a product of the marginal distribution measures:

${\mu }_{X,Y}(B_1\times B_2)=\mathbb{P}\text{(}X\in B_1)\mathbb{P}\text{(}Y\in B_2)={\mu }_X(B_1){\mu }_Y(B_2):={\mu }_{X\times Y}(B_1\times B_2).\left(9.76\right)$

Hence, (9.75) and (9.76) are equivalent. From (9.60) we also have that independence is equivalent to

$F_{X,Y}(x,y)=F_X(x)F_Y(y),x,y\in ℝ.\left(9.77\right)$

Moreover, two continuous random variables (X, Y) are independent if and only if their joint PDF is the product of the marginal PDFs,

$f_{X,Y}(x,y)=f_X(x)f_Y(y),x,y\in ℝ.\left(9.78\right)$

This is easily proven. In particular, assuming (X, Y) are independent, then (9.71) gives

$\begin{array}{l}{\displaystyle {\int }_{-\infty }^b{\displaystyle {\int }_{-\infty }^a{f}_{X,Y}}\left(x,y\right)}\text{d}x\text{d}y=F_{X,Y}\left(a,b\right)=F_X\left(a\right)F_Y\left(b\right)\\ ={\displaystyle {\int }_{-\infty }^a{f}_X\left(x\right)\text{d}x{\displaystyle {\int }_{\infty }^b{f}_Y\left(y\right)\text{d}y={\displaystyle {\int }_{-\infty }^b{\displaystyle {\int }_{-\infty }^a{f}_X}\left(x\right)}{f}_Y}\left(y\right)}\text{d}x\text{d}y\end{array}$

for all a, b ∊ ℝ. This implies ƒX,Y (x, y) = ƒx(x)ƒY(y). We leave the proof of the converse as an exercise for the reader. When (X, Y) are independent, the general expectation formula for all types of random variables, as given by (9.69), is now a Lebesgue—Stieltjes integral w.r.t. the above product measure μX × Y:

$\text{E}[h(X,Y)]={\displaystyle {\int }_{{ℝ}^2}h(x,y)\text{d}{\mu }_{X\times Y}(x,y)}={\displaystyle {\int }_{ℝ}{\displaystyle {\int }_{ℝ}h(x,y)\text{d}{\mu }_X}(x)\text{d}{\mu }_Y(y),\left(9.79\right)}$

where the order of integration in μX and μY is interchangeable according to Fubini's Theorem. In the case that h(X,Y) = h1(X)h2(Y),

$\text{E}[h(X,Y)]=\left({\displaystyle {\int }_{ℝ}{h}_1(x)\text{d}{\mu }_X(x)}\right)\left({\displaystyle {\int }_{ℝ}{h}_2(y)\text{d}{\mu }_Y(y)}\right)=\text{E}[h_1\left(X\right)]\text{E}[h_2(Y)].\left(9.80\right)$

Of course, for continuous (X, Y) this product of expectations is given by (9.73). Taking h1(x) = x, h2(y) = y, h(x, y) = xy shows that two mutually independent random variables have zero covariance

$\text{Cov}(X,Y)\equiv \text{E}[XY]-\text{E}[X]\text{E}[Y]=0.\left(9.81\right)$

The converse is generally not true.

An important example of a jointly continuous random vector (X, Y) is the standard bivariate normal distribution where E[X] = E[Y] = 0, Cov(X, Y) = ρ, |ρ| < 1. The well-known joint PDF is

$f_{X,Y}(x,y)=n_2(x,y;\rho ):=\frac{1}{2\pi \sqrt{1-{\rho }^2}}\exp \left(-\frac{x^2+y^2-2\rho xy}{2(1-{\rho }^2)}\right),x,y\in ℝ.\left(9.82\right)$

The joint distribution measure μX,Y is absolutely continuous w.r.t. Lebesgue measure m2; i.e., for all B ∊ ℬ2 we have

${\mu }_{X,Y}(B)={\displaystyle {\int }_B{n}_2(x,y;\rho )\text{d}{m}_2(x,y)\equiv {\displaystyle {\int }_{-\infty }^{\infty }{\displaystyle {\int }_{-\infty }^{\infty }{?}_B(x,y)n_2(x,y;\rho )\text{d}x\text{d}y.\left(9.83\right)}}}$

The joint CDF is

$F_{X,Y}(a,b)={\mathcal{N}}_2(x,y;\rho ):={\displaystyle {\int }_{-\infty }^b{\displaystyle {\int }_{-\infty }^a{n}_2(x,y;\rho )\text{d}x\text{d}y,a,b\in ℝ.\left(9.84\right)}}$

The functions n2 and ${\mathcal{N}}_2$ denote the standard bivariate normal PDF and CDF, respectively, where $n_2\left(x,y;\rho \right)=\frac{{\partial }^2}{\partial x\partial y}{\mathcal{N}}_2\left(x,y;\rho \right).$ We note also the symmetry: $n_2\left(x,y,\rho \right)=n_2\left(y,x,\rho \right)$ and ${\mathcal{N}}_2\left(x,y,\rho \right)={\mathcal{N}}_2\left(y,x,\rho \right)$. The marginal CDFs of X and Y are the standard normal CDF and follow simply from the limiting values of the joint CDF (see (9.61)—(9.62)) :

$F_X(x)=F_{X,Y}(x,\infty )={\mathcal{N}}_2(x,\infty ;\rho )=\mathcal{N}(x),F_Y(y)=F_{X,Y}(\infty ,y)={\mathcal{N}}_2(\infty ,y;\rho )=\mathcal{N}(y).$

Here we used the integral definition of ${\mathcal{N}}_2$ in (9.84). Hence, X and Y are identically distributed Norm(0, 1) random variables with standard normal (marginal) PDF

$f_X(x)={\mathcal{N}}^{\prime }(x)=n(x),f_Y(y)={\mathcal{N}}^{\prime }(y)=n(y),$

$n\left(z\right):=\frac{1}{\sqrt{2\pi }}{e}^{-z^2⁄2}$, z ∊ ℝ. [Note that these marginal PDFs also follow by integrating the joint PDF according to (9.72).] We observe that the pair (X, Y) is mutually independent if and only if the correlation coefficient ρ = 0, i.e.,

$f_{X,Y}(x,y)=n_2(x,y;0)=\frac{1}{2\pi }{e}^{-(x^2+y^2)/2}=n(x)n(y)=f_X(x)f_Y(y),x,y\in ℝ,$

and the integral in (9.84) factors into

$F_{X,Y}(a,b)={\mathcal{N}}_2(a,b;0)=\mathcal{N}(a)\mathcal{N}(b)=F_X(a)F_Y(b),a,b\in ℝ.$

Substituting the above joint PDF into (9.74) gives the expectation of a Borel function of the normal pair (X, Y) as

$\text{E}[h(X,Y)]={\displaystyle {\int }_{-\infty }^{\infty }{\displaystyle {\int }_{-\infty }^{\infty }h(x,y)n_2(x,y;\rho )\text{d}x\text{d}y.}}\left(9.85\right)$

For ρ = 0, n2{x, y; 0) = n(x)n(y), and for h(x, y) = h1(x)h2(y) this expectation reduces to (9.80), where

$\begin{array}{l}\text{E}[h_1(X)]={\displaystyle {\int }_{ℝ}^{}{h}_1(x)\text{d}{\mu }_X(x)}={\displaystyle {\int }_{ℝ}^{}{h}_1(x)n(x)\text{d}x},\\ \text{E}[h_2(Y)]={\displaystyle {\int }_{ℝ}^{}{h}_2(x)\text{d}{\mu }_Y(y)}={\displaystyle {\int }_{ℝ}^{}{h}_2(y)n(y)\text{dy}}.\end{array}$

The above formulation extends to the more general case of an n-dimensional real-valued random vector X = (X1,..., Xn) ∊ ℝn, for all integers n ≥ 1. Each Xi is a random variable on (Ω, ℱ, ℙ) where $X_i^{-1}\left(B_i\right)\in ℱ$ for every Bi ∊ ℬ(ℝ), i = 1,..., n. As a random vector X : Ω → ℝn, for every Borel set B = B1 × ... × Bn ∊ ℬn,

${\text{X}}^{-1}(B)\equiv \{X\in B\}\equiv \{X_1\in B_1,\mathrm{...},X_n\in B_n\}\in ℱ.$

The joint distribution measure of X = (X1,..., Xn), which generalizes (9.56), is defined by

${\mu }_{\text{X}}(B)\equiv {\mu }_{X_1,\mathrm{...},X_n}(B):=\mathbb{P}(\text{X}\in B)\equiv \mathbb{P}(X_1\in B_1,\mathrm{...},X_n\in B_n).\left(9.86\right)$

This measure assigns a probability to a Borel set B in ℝn which corresponds to the probability of the joint event {ω ∊ Ω: X1(ω) ∊ B1,... ,Xn(ω) ∊ Bn} = {X1 ∊ B1}∩...∩{Xn ∊ Bn}. It is normalized, μX(ℝn) = ℙ(X ∊ ℝn) = 1, so (ℝn, Bn, μX) is a probability space.

The joint (n-dimensional) measure μX determines all the univariate, bivariate, trivariate, etc., distribution measures for all single random variables Xi, pairs (Xi, Xj), triples (Xi, Xj, Xk), etc. This follows by setting some of the appropriate sets among B1,..., Bn equal to ℝ. For example, setting all sets Bj = ℝ, for all j ≠ i, and Bi = A ∊ ℬ, gives the univariate marginal distribution measures

${\mu }_{\text{X}}(B_1\times \mathrm{...}\times B_n)=\mathbb{P}(X_i\in A)={\mu }_{X_i}(A),i=1,\mathrm{...},n.$

The bivariate (marginal) distribution measure of a pair (Xi, Xj), i < j, is obtained by setting all sets Bk = ℝ, for all k ≠ i, k ≠ j, and Bi = A ∊ ℬ, Bj = B ∊ ℬ:

${\mu }_{\text{X}}(B_1\times \mathrm{...}\times B_n)=\mathbb{P}(X_i\in A,X_j\in B)={\mu }_{X_i,X_j}(A\times B),i<j=1,\mathrm{...},n.$

Letting B1 = (−∞,x1], B2 = (−∞, x2], . . ., Bn = (−∞, xn] in (9.86) gives the multivariate joint CDF of (X1,..., Xn):

$F_{\text{X}}(\text{x})\equiv F_{(X_1,\mathrm{...},X_n)}(x_1,\mathrm{...},x_n)=\mathbb{P}(X_1\le x_1,\mathrm{...},X_n\le x_n),\left(9.87\right)$

x = (x1... ,xn) ∊ ℝn. This CDF is right-continuous on ℝn and is a nondecreasing monotonic function in all variables x1,..., xn. The marginal CDFs of each Xi are recovered in the limit that xj → ∞, for all j ≠ i in (9.87):

$F_{X_i}(x_i)=\mathbb{P}(X_i\le x_i)=F_{(x_1,\mathrm{...},x_n)}(\infty ,\mathrm{...},\infty ,x_i,\infty ,\mathrm{...},\infty ),x_i\in ℝ.\left(9.88\right)$

Similarly, the (marginal) joint CDF for each random vector pair (Xi, Xj), i < j, is obtained by letting xk → ∞ for all k ≠ i, k ≠ j:

$F_{X_i,X_j}(x_i,x_j)\equiv ℝ(X_i\le x_i,X_j\le x_j)=\underset{all{x}_k\to \infty ;k\ne i,k\ne j}{\lim }{F}_{(X_1,\mathrm{...},X_n)}(x_1,\mathrm{...},x_n).$

All other (marginal) joint CDFs are obtained in the appropriate limits. For example, we can consider any k-dimensional random vector such as (X1,...,Xk), for any 1 ≤ k ≤ n, having joint CDF

$F_{X_1,\mathrm{...},X{}_k}(x_1,\mathrm{...},x_k)=F_{(X_1,\mathrm{...},X_n)}(x_1,\mathrm{...},x_k,\infty ,\mathrm{...},\infty ).$

More generally, the joint CDF $F_{X_{i_1},\mathrm{...},X_{i_k}}\left(y_1,\mathrm{...},y_k\right)$, where (y1,..., yk) ∊ ℝk, of any k-dimensional random vector $\left(X_{i_1},\mathrm{...},X_{i_k}\right)$, 1 ≤ i1 < ... < ik ≤ n, 1 ≤ k ≤ n, taken from (X1,..., Xn) is obtained by setting xj = ∞ for all $j\notin \left\{i_1,i_2,\mathrm{...},i_k\right\}$ in the n-dimensional joint CDF $F_{\left(X_1,\mathrm{...},X_n\right)}\left(x_1,\mathrm{...},x_n\right)$. This corresponds to the probability

$F_{X_{i_1},\mathrm{...},X_{i_k}}(y_1,\mathrm{...},y_k)=\mathbb{P}(X_{i_1}\le y_1,\mathrm{...},X_{i_k}\le y_k).\left(9.89\right)$

As in the two-dimensional case, the joint CDF evaluates to zero when setting any one of its arguments to −∞. Setting all xi = ∞ gives unity: FX(∞,..., ∞) = ℙ(X ∊ ℝn) = 1.

The random vector $\text{Y=}\left(Y_1,...,Y_k\right):=\left(X_{i_1},...,X_{i_k}\right),$ for any 1 ≤ k ≥ n, has a joint distribution measure defined by

${\mu }_{\text{Y}}(B_1\times \mathrm{...}\times B_k)=\mathbb{P}(X_{i_1}\in B_1,\mathrm{...},X_{i_k}\in B_k)$

for any Borel set B1 × ... × Bk ∊ ℬk ≡ ℬ(ℝk). Hence, (ℝk, ℬk, μY) is a probability space for each k = 1,..., n.

The relation in (9.63) can be extended to any n-dimensional semi-open rectangle with the use of the n-dimensional joint CDF FX(x). Moreover, the Lebesgue—Stieltjes measure defined in (9.64) can be extended into n dimensions accordingly, as generated by FX. In fact, the n-dimensional joint distribution measure μX in (9.86) is the same Lebesgue—Stieltjes measure on ℝn. The above construction of the Lebesgue—Stieltjes integral w.r.t. μX,Y (for dimension n = 2), provided by (9.65),(9.66), (9.67), and (9.68), extends in the obvious manner into dimension n ≥ 2. We write the Lebesgue—Stieltjes integral of a Borel function ƒ: ℝn → ℝ w.r.t. the joint distribution measure μX as the difference of two nonnegative integrals

${\displaystyle {\int }_{{ℝ}^n}f(\text{X})\text{d}{\mu }_{\text{X}}(\text{x})}={\displaystyle {\int }_{{ℝ}^n}{f}_{+}(\text{x})\text{d}{\mu }_{\text{X}}(\text{x})-}{\displaystyle {\int }_{{ℝ}^n}{f}_{-}(\text{x})\text{d}{\mu }_{\text{X}}(\text{x}),\left(9.90\right)}$

and for any Borel set B in ℝn we have

${\displaystyle {\int }_{B}f(\text{x})\text{d}{\mu }_{\text{X}}(\text{x})={\displaystyle {\int }_{{ℝ}^n}{\mathbb{I}}_B(\text{x})f(\text{x})\text{d}{\mu }_{\text{X}}(\text{x}).\left(9.91\right)}}$

Here, $f\left(\text{x}\right)\equiv f\left(x_1,\mathrm{...},x_n\right),\text{d}{\mu }_{\text{X}}\left(\text{x}\right)\equiv \text{d}{\mu }_{{\text{X}}_1,\mathrm{...},{\text{X}}_n}\left(x_1,\mathrm{...},x_n\right)$ is shorthand vector notation.

Given a Borel function, h : ℝn → ℝ, of a random vector X = (X1,...,Xn) on a probability space (Ω, ℱ, ℙ), Theorem 9.5 is generalized to give the expected value of the random variable h(X) ≡ h(X1,..., Xn) as an integral of h(x) = h(x1,..., xn) w.r.t. the joint distribution measure μX over ℝn:

$\text{E}[h(\text{X})]\equiv {\displaystyle {\int }_{\Omega }h}(\text{X}(\omega ))\text{d}\mathbb{P}\text{(}\omega \text{)}={\displaystyle {\int }_{{ℝ}^n}h(\text{x})\text{d}{\mu }_{\text{X}}(\text{x}).\left(9.92\right)}$

This formula can be proven in the same manner as the proof of (9.69). This Lebesgue—Stieltjes integral is a general representation for the expected value E[h(X)] where h is a, Borel function on ℝn. So, each component of the vector (X1,...,Xn) can be any type of random variable, i.e., any combination of discrete, absolutely continuous, or singularly continuous random variables. The two main types of random variables of interest to us are either discrete or continuous (i.e., absolutely continuous).

The case where all Xi are discrete (as in the binomial and multinomial financial models considered in previous chapters) simply generalizes the above double summation formulas in the case of two variables to multiple (n-fold) summation formulas involving the joint PMF $p_{X_1,\mathrm{...},X_n}\left(x_1,\mathrm{...},x_n\right)\equiv \mathbb{P}\left(X_1=x_1,\mathrm{...},X_n=x_n\right)$ at the support values:

$\text{E}[h(X_1,\mathrm{...},X_n)]={\displaystyle \sum _{\text{all }{x}_1}\mathrm{...}}{\displaystyle \sum _{\text{all }{x}_n}{p}_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_n)}h(x_1,\mathrm{...},x_n).$

Here we assume E[|h(X1,...,Xn)|] < ∞ (i.e., we assume the sums converge for both negative and positive parts of h). Choosing the indicator function $h\left(\text{X}\right)={\mathbb{I}}_{\left\{X_1\le a_1,\mathrm{...},X_n\le a_n\right\}}$ recovers the joint CDF:

$F_{(X_1,\mathrm{...},X_n)}(a_1,\mathrm{...},a_n)=\text{E}[{\mathbb{I}}_{\{X_1\le a_1,\mathrm{...},X_n\le a_n\}}]={\displaystyle \sum _{x_1\le a_1}\mathrm{...}}{\displaystyle \sum _{x_n\le a_n}{p}_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_n).}$

This is a (n-dimensional piecewise constant) staircase function in the variables (a1,..., an) with jump discontinuities occurring at only the support values of the PMF.

The most important case for continuous random variables is when (X1,... ,Xn) are continuous with joint density ƒx1,...,xn(x1...,xn) ≡ fX(x) as a nonnegative integrable Borel function fX : ℝn → ℝ, i.e., when every Borel set B ∊ ℬn has joint measure given by the Lebesgue integral of the joint density over B:

${\mu }_{\text{X}}(B)={\displaystyle {\int }_B{f}_{\text{X}}\text{d}{m}_n}\equiv {\displaystyle {\int }_{{ℝ}^n}{\mathbb{I}}_B(\text{x)}{f}_{\text{X}}(\text{x})}{\text{d}}^n\text{x}\text{.}\left(\text{9}\text{.93}\right)$

This is the generalization of (9.70), where the Lebesgue integral is written as a Riemann integral on ℝn. The Lebesgue and Riemann integrals are equal if ?BƒX is (mn—a.e.) continuous on ℝn. The joint CDF is obtained by setting B = (—∞, x1] × ... × (−∞, xn], where ${\mathbb{I}}_B\left(y_1,\mathrm{...},y_n\right)={\mathbb{I}}_{\left\{y_1\le x_1,y_n\le x_n\right\}}$:

$F_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_n)={\displaystyle {\int }_{-\infty }^{x_n}\mathrm{...}}{\displaystyle {\int }_{-\infty }^{x_1}{f}_{X_1,\mathrm{...},X_n}(y_1,\mathrm{...},y_n){\text{d}}_{y_1}\mathrm{...}{\text{d}}_{y_n}},\left(9.94\right)$

for all x1,...,xn ∊ ℝ. Note that the joint PDF ƒx integrates to unity since μx(ℝn) = ℙ(X ∊ ℝn) = 1. As we proved for n = 2, the relation in (9.94) is equivalent to (9.93). The joint CDF is continuous on ℝn and related to the joint PDF by differentiation,

$f_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_n)=\frac{{\partial }^n}{{\partial }_{x_1}\mathrm{...}{\partial }_{x_n}}{F}_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_n),x_1,\mathrm{...},x_n\in ℝ.\left(9.95\right)$

Note that (9.95) implies the existence of all marginal PDFs (densities) for all univariate Xi, bivariate (Xi, Xj), etc. In particular, all k-dimensional random vectors $\left(X_{i_1,\mathrm{...},}{X}_{i_k}\right)$, 1 ≤ i1 < ... < ik ≤ n, 1 ≤ k ≤ n, have CDF as in (9.89). Using this relation in (9.94) gives us the joint (marginal) PDF of $\left(X_{i_1,\mathrm{...},}{X}_{i_k}\right)$ as an (n — k)-dimensional integral of the joint PDF over ℝn—k in the integration variables yj, for all $j\notin \left\{i_1,i_2,\mathrm{...},i_k\right\}$. For example, the CDF of the random vector consisting of the first k variables, (X1,..., Xk), is

$F_{X_1,\mathrm{...},X_k}(x_1,\mathrm{...},x_k)=F_{X_1,\mathrm{...},X_k,X_{k+1},\mathrm{...},X_n}(x_1,\mathrm{...},x_k,\infty ,\mathrm{...},\infty )\left(9.96\right)$

$={\displaystyle {\int }_{-\infty }^{x_k}\mathrm{...}}{\displaystyle {\int }_{-\infty }^{x_1}\left({\displaystyle {\int }_{-\infty }^{\infty }\mathrm{...}}{\displaystyle {\int }_{-\infty }^{\infty }{f}_{X_1,\mathrm{...},X_n}(y_1,\mathrm{...},y_n)\text{d}{y}_{k+1}\mathrm{...}}\text{d}{y}_n\right)\text{d}{y}_1}\mathrm{...}\text{d}{y}_k.\left(9.97\right)$

The (n — k)-dimensional (inner) integral is the joint PDF of (X1,... ,Xk). This is an integrable Borel function ƒx1,...,xk: ℝk → R, given by

$f_{X_1,\mathrm{...},X_k}(x_1,\mathrm{...},x_k)={\displaystyle {\int }_{-\infty }^{\infty }\mathrm{...}}{\displaystyle {\int }_{-\infty }^{\infty }{f}_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_k,x_{k+1},\mathrm{...},x_n)\text{d}{x}_{k+1}\mathrm{...}\text{d}{x}_n.}$

Hence, for every k = 1,..., n, we have the marginal CDF and PDF relations:

$F_{X_1,\mathrm{...},X_k}(x_1,\mathrm{...},x_k)={\displaystyle {\int }_{-\infty }^{x_k}\mathrm{...}}{\displaystyle {\int }_{-\infty }^{x_1}{f}_{X_1,\mathrm{...},X_k}(y_1\mathrm{...}{y}_k)\text{d}{y}_1}\mathrm{...}\text{d}{y}_k,\left(9.98\right)$

and

$f_{X_1,\mathrm{...},X_k}(x_1,\mathrm{...},x_k)=\frac{{\partial }^k}{{\partial }_{k_1}\mathrm{...}{\partial }_{x_k}}{F}_{X_1,\mathrm{...}{X}_k}(x_1,\mathrm{...},x_k),x_1,\mathrm{...},x_k\in ℝ.\left(9.99\right)$

Based on (9.93), it can be proven that the expectation formula in (9.92) takes the form of an integral over ℝn involving the joint PDF:

$\begin{array}{ll}\text{E}[h(\text{X)}]& ={\displaystyle {\int }_{{ℝ}^n}h(\text{x})f_{\text{X}}(\text{x}){\text{d}}^n\text{x}}\hfill \\ & \equiv {\displaystyle {\int }_{-\infty }^{\infty }\mathrm{...}}{\displaystyle {\int }_{-\infty }^{\infty }h\left(x_1,\mathrm{...},x_n\right)f_{X_1,\mathrm{...},X_n}(x_1,\mathrm{...},x_n)\text{d}{x}_1\mathrm{...}\text{d}{x}_n.}\left(9.100\right)\hfill \end{array}$

Note that (9.74) is a special case of this formula for n = 2 dimensions. All marginal CDFs in (9.89) are also conveniently expressed as expectations of indicator functions where

$F_{X_{i_1},\mathrm{...},X_{i_k}}(y_1,\mathrm{...},y_k)=\mathbb{P}(X_{i_1}\le y_1,\mathrm{...},X_{i_k}\le y_k)=\text{E}[{\mathbb{I}}_{X_{i_1}\le y_1,\mathrm{...},X_{i_k}\le y_k}].$

It is convenient to define $\text{Y}=\left(Y_1,\mathrm{...},Y_k\right):=\left(X_{i_1,\mathrm{...},}{X}_{i_k}\right)$. Now, differentiating all k arguments of the (marginal) joint CDF gives the (marginal) joint PDF for a continuous random vector Y,

$f_{Y_1,\mathrm{...},Y_k}(y_1,\mathrm{...},y_k)=\frac{{\partial }^k}{{\partial }_{y_1}\mathrm{...}{\partial }_{y_k}}{F}_{Y_1,\mathrm{...},Y_k}(y_1,\mathrm{...},y_k).$

Hence, if h: ℝk → ℝ is a Borel function of only $\left(X_{i_1,\mathrm{...},}{X}_{i_k}\right)\equiv \left(Y_1,\mathrm{...},Y_k\right)$, with 1 ≤ k ≤ n components from X, (9.100) reduces to a k-dimensional integral involving the (marginal) joint PDF of Y:

$\text{E}[h(Y_1,\mathrm{...},Y_k)]={\displaystyle {\int }_{-\infty }^{\infty }\mathrm{...}}{\displaystyle {\int }_{-\infty }^{\infty }h(y_1,\mathrm{...},y_k)}{f}_{Y_1,\mathrm{...},Y_k}(y_1,\mathrm{...},y_k)\text{d}{y}_1\mathrm{...}\text{d}{y}_k.\left(9.101\right)$

Based on (9.101), and choosing appropriate functions for h, we can in principle compute several quantities of interest, such as moments, product moments, joint moment generating functions, joint characteristic functions, etc., as long as the integrals exist. In particular, the covariance between any two continuous random variables in X, say Xi and Xj, is computed by making use of the joint PDF of the pair (Xi, Xj) and the marginal densities of Xi and Xj:

$\begin{array}{rr}\hfill \text{Cov}(X_i,X_j)& :=\text{E}[X_i{X}_j]-\text{E}[X_i]\text{E}[X_j]\hfill \\ & ={\displaystyle {\int }_{ℝ}{\displaystyle {\int }_{ℝ}xy{f}_{X_i,X_j}(x,y)\text{d}x\text{d}y}-}\left({\displaystyle {\int }_{ℝ}x{f}_{X_i}(x)\text{d}x}\right)\left({\displaystyle {\int }_{ℝ}y{f}_{X_j}(y)\text{d}y}\right).\left(9.102\right)\hfill \end{array}$

Let us now consider the case where X1,...,Xn are independent. By property 1 of Definition 6.14, it follows that the joint distribution measure in (9.86) is now a product measure on ℝn:

${\mu }_{X_1,\mathrm{...},X_n}(B)={\displaystyle \prod _{i=1}^n\mathbb{P}}(X_i\in B_i)={\displaystyle \prod _{i=1}^n{\mu }_{X_i}(B_i):={\mu }_{X_1\times \mathrm{...}\times X_n}(B)\left(9.103\right)}$

for all Borel sets B = B1 × ... × Bn in ℝn. The joint CDF is then the product of marginal CDFs,

$F_{X_1,\mathrm{...},X_n}\left(x_1,\mathrm{...},x_n\right)={\displaystyle \prod _{i=1}^n\mathbb{P}\left(X_i\le x_i\right)={\displaystyle \prod _{i=1}^n{F}_{X_i}\left(x_i\right)}}.\left(9.104\right)$

For continuous random variables then, by differentiating (9.104) according to (9.95), the joint PDF is the product of marginal densities

$f_{X_1,\mathrm{...},X_n}\left(x_1,\mathrm{...},x_n\right)={\displaystyle \prod _{i=1}^n{f}_{X_i}\left(x_i\right).}\left(9.105\right)$

In fact, it can be shown that (9.105) and (9.104) are equivalent in the case of continuous random variables.

The expectation formula in (9.79) extends to n dimensions,

$\text{E}\left[h\left(\text{X}\right)\right]={\displaystyle {\int }_{ℝ}\mathrm{...}}{\displaystyle {\int }_{ℝ}h\left(\text{x}\right)\text{d}{\mu }_{X_1}\left(x_1\right)\mathrm{...}\text{d}{\mu }_{X_n}\left(x_n\right)},\left(9.106\right)$

where the order of integration is interchangeable according to Fubini's Theorem. Similarly, in the case where h(X) = h1(X1)h2(X2) . . . h(Xn), (9.80) extends to a product of n expectations:

$\text{E}\left[h_1\left(X_1\right)h{}_2\left(X_2\right)\mathrm{...}h\left(X_n\right)\right]={\displaystyle \prod _{i=1}^n{\displaystyle {\int }_{ℝ}{h}_i\left(x_i\right)\text{d}{\mu }_{X_i}\left(x_i\right)={\displaystyle \prod _{i=1}^n\text{E}\left[h_i\left(X_i\right)\right]},\left(9.107\right)}}$

where we assume that all product functions are integrable, E[|hi(Xi)|] ≤ ∞, i = 1,...,n. For continuous random variables we have the usual formula for the expectation involving the marginal densities,

$\text{E}\left[h_{\text{1}}\left(X_1\right)h_2\left(X_2\right)\mathrm{...}h\left(X_n\right)\right]={\displaystyle \prod _{i=1}^n\text{E}\left[h_i\left(X_i\right)\right]}={\displaystyle \prod _{i=1}^n{\displaystyle {\int }_{-\infty }^{\infty }{h}_i\left(x\right)f_{X_i}\left(x\right)\text{d}x.}\left(9.108\right)}$

The above formulas in the case of independence have analogues for any sub-collection of random variables, i.e., for any random vector $\text{Y}=\left(Y_1,\mathrm{...},Y_k\right):=\left(X_{i_1,\mathrm{...},}{X}_{i_k}\right),1\le i_1<i_2<\mathrm{...}<i_k\le n,1\le k\le n$, as discussed above. If all components are independent, then the joint distribution measure of Y is simply the product measure on ℝk:

${\mu }_{Y_1,\mathrm{...},Y_k}\left(B\right)={\displaystyle \prod _{i=1}^k{\mu }_{Y_i}\left(B_i\right):={\mu }_{Y_1\times \mathrm{...}\times Y_k}\left(B\right)}$

for all Borel sets B = B1 × ... × Bk in ℝk. The joint CDF of Y is the product of the marginal CDFs

$F_{Y_1,\mathrm{...},Y_k}\left(y_1,\mathrm{...},y_n\right)={\displaystyle \prod _{i=1}^k{F}_{Y_i}\left(y_i\right)},$

with joint PDF (for the case of a continuous random vector) as a product of the marginal densities

$f_{Y_1,\mathrm{...},Y_k}\left(y_1,\mathrm{...},y_n\right)={\displaystyle \prod _{i=1}^k{f}_{Y_i}\left(y_i\right).}$

Note that in the case that Xi and Xj are independent, ƒXi,Xj(x,y) = ƒXi(x)ƒXj(y), E[XiXj] = E[Xi]E[Xj], so (9.102) gives zero covariance, as required.

9.4 Conditioning

Now that we are equipped with general probability theory, we revisit the subject of conditioning and conditional expectations of random variables. We basically already covered the main topics in Section 6.3.2 of Chapter 6. We recall Definition 6.12 for the expectation of a random variable conditional on a σ-algebra. The definition was stated very generally using expectations. Since any expectation is in fact a Lebesgue integral w.r.t. a given probability measure ℙ, we can also state Definition 6.12 in the equivalent manner using Lebesgue integral notation. In particular, property (ii) in Definition 6.12 reads

${\displaystyle {\int }_{B}X\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle {\int }_B\text{E}\left[X\text{|}\mathcal{G}\right]\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)}}$

for every B ∊ G.

It is instructive to see how Proposition 6.7 on independence for two random variables follows in the case of continuous random variables. Let X and Y be jointly (absolutely) continuous and independent random variables possessing a joint PDF ƒX,Y(x,y) = ƒX(x)ƒY(y) with marginal PDFs ƒX(x) and ƒY(y). The conditional PDF of Y given X = x is hence the marginal PDF of Y: ƒY|X(y|x) = ƒX,Y(x,y)/ƒX(x) = ƒY(y). Evaluating the conditional expectation in the usual manner gives

$\begin{array}{ll}\hfill \text{E}\left[h\left(X,Y\right)|X=x\right]& =\text{E}\left[h\left(x\text{,}Y\right)|X=x\right]\\ & ={\displaystyle {\int }_{ℝ}h\left(x,y\right)f_{Y|X}\left(y|x\right)\text{d}y={\displaystyle {\int }_{ℝ}h\left(x,y\right)f_Y\left(y\right)\text{d}y=\text{E}\left[h\left(x,Y\right)\right]}.}\hfill \end{array}$

Hence, as a random variable we have E[h(X,Y) | X] = ∫ℝh(X, y)ƒY(y)dy ≔ g(X), i.e., E[h(X, Y) | X](ω) = ∫ℝ h(X(ω), y)ƒY(y)dy ≔ g(X(ω)), for each ω ∊ Ω.

Now, to formally show that the above expectation formula is in fact the correct one we need to verify properties (i) and (ii) of Definition 6.12 with G = σ(X). Property (i) holds since g defined by the above integral is a Borel function and hence σ(g(X)) ⊂ σ(X), i.e., g(X) is σ(X)-measurable. For property (ii), we need to show that

$\text{E}\left[{\mathbb{I}}_{X\in B}.\text{E}\left[h\left(X,Y\right)|X\right]\right]\equiv \text{E}\left[{\mathbb{I}}_{X\in B}.g\left(X\right)\right]=\text{E}\left[{\mathbb{I}}_{X\in B}.h\left(X,Y\right)\right]$

for every Borel set B in σ(X), i.e., B is any Borel set in the range of X and so we take any B ∊ ℬ(ℝ). Expressing these expectations using the joint PDF gives

$\text{E}\left[{\mathbb{I}}_{X\in B}.g\left(X\right)\right]={\displaystyle {\int }_{B}g\left(x\right)\left({\displaystyle {\int }_{ℝ}{f}_{X,Y}\left(x,y\right)\text{d}y}\right)\text{d}x={\displaystyle {\int }_{B}g\left(x\right)f_X\left(x\right)\text{d}x}}$

and

$\text{E}\left[{\mathbb{I}}_{X\in B}.h\left(X,Y\right)\right]={\displaystyle {\int }_B\left({\displaystyle {\int }_{ℝ}h\left(x,y\right)f_{X,Y}\left(x,y\right)\text{d}y}\right)\text{d}x.}$

For these two quantities to be equal, for every Borel set B, we necessarily must have the equivalence of the x-integrands, i.e.,

$g\left(x\right)={\displaystyle {\int }_{ℝ}h\left(x,y\right)\frac{f_{X,Y}\left(x,y\right)}{f_X\left(x\right)}\text{d}y=}{\displaystyle {\int }_{ℝ}h\left(x,y\right)f_{Y|X}\left(y|x\right)\text{d}y={\displaystyle {\int }_{ℝ}h\left(x,y\right)f_Y\left(y\right)\text{d}y.}}$

This proves the above expression for g(x) and hence for E[h(X,Y) | X].

If X and Y are jointly continuous and independent random vectors, then their joint PDF is ƒX,Y(x, y) = ƒX(x)ƒY(y) with marginal PDFs ƒX(x) and ƒY(y). The conditional PDF of Y given X = x is the marginal PDF of Y: ƒY|X(Y|X) = ƒY(Y). By basic probability theory, the conditional expectation is given by the n-dimensional integral:

$\text{E}\left[h\left(\text{X,}\text{Y}\right)|\text{X}\text{=}\text{x}\right]={\displaystyle {\int }_{{ℝ}^n}h\left(\text{x,}\text{y}\right)f_{\text{Y}\text{|}\text{X}}\left(\text{y}\text{|}\text{x}\right){\text{d}}^n\text{y}\text{=}{\displaystyle {\int }_{{ℝ}^n}h\left(\text{x,}\text{y}\right)f_{\text{Y}}\left(\text{y}\right){\text{d}}^n\text{y}}=\text{E}\left[h\left(\text{x,}\text{Y}\right)\right].}$

A similar analysis as given above (for the case of two random variables) formally shows that this is the correct formula satisfying properties (i) and (ii) of Definition 6.12 with G = σ(X). In this case the Borel sets B ⊂ ℝn.

Although we have already covered many of the important properties of conditioning in Section 6.3.3 of Chapter 6, it is still useful to summarize them in the following theorem since they pertain to the more general theory of random variables. Many of the proofs are rather straightforward and are standard in real analysis so we don’t repeat them here. We note that we have also proven some of the properties in the discrete setting in Chapter 6 and that we have already used some of the properties stated below.

Theorem 9.6.

Let X and Y be random variables on (Ω, ℱ, ℙ). Then, the following hold.

1. 1. (Linearity) For any real constants a, b:

   $\text{E}\left[aX+bY|\mathcal{G}\right]=a\text{E}\left[X|\mathcal{G}\right]+b\text{E}\left[Y|\mathcal{G}\right].$

2. 2. (Nested Expectation)

   $\text{E}\left[\text{E}\left[X|\mathcal{G}\right]\right]=\text{E}\left[X\right].$

3. 3. (Tower Property) For any $\mathcal{H}\subset \mathcal{G}\subset ℱ$,

   $\text{E}\left[\text{E}\left[X|\mathcal{G}\right]|\mathscr{H}\right]=\text{E}\left[X|\mathscr{H}\right].$

4. 4. (Independence) If X is independent of $\mathcal{G}$ then

   $\text{E}\left[X|\mathcal{G}\right]=\text{E}\left[X\right].$

5. 5. (Measurability) If X is $\mathcal{G}$-measurable then

   $\text{E}\left[X|\mathcal{G}\right]=X.$

6. 6. (Positivity) If X ≥0 (a.s.) then $\text{E}\left[X|\mathcal{G}\right]\ge 0$ (a.s.).
7. 7. (Monotone Convergence) If Xn, n ≥1, isa nonnegative sequence of random variables on (Ω, ℱ, ℙ) and increases (a.s.) to X, then the sequence $\text{E}\left[X_n|\mathcal{G}\right],n\ge 1$, increases (a.s.) to $\text{E}\left[X|\mathcal{G}\right]$.
8. 8. (Pulling out what is known) If Y is $\mathcal{G}$-measurable and XY is integrable then

   $\text{E}\left[XY|\mathcal{G}\right]=Y\text{E}\left[X|\mathcal{G}\right].$

9. 9. (Conditional Jensen's Inequality) If φ : ℝ → ℝ is a convex function and X is integrable then

   $\text{E}\left[\phi \left(X\right)|\mathcal{G}\right]\ge \phi \left(\text{E}\left[X|\mathcal{G}\right]\right).$

9.5 Changing Probability Measures

Here we shall keep the discussion very succinct. Let us begin by defining a new probability measure $\widehat{\mathbb{P}}\equiv {\widehat{\mathbb{P}}}^{\left(\varrho \right)}$ by

$\widehat{\mathbb{P}}\left(A\right)\equiv {\displaystyle {\int }_A\text{d}\widehat{\mathbb{P}}\left(\omega \right):={\displaystyle {\int }_A\varrho \left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)}},\left(9.109\right)$

for all $A\in ℱ,\text{i}\text{.e}\text{.,}\widehat{\mathbb{P}}\left(A\right)\equiv \widehat{\text{E}}\left[{\mathbb{I}}_A\right]:=\text{E}\left[\varrho {\mathbb{I}}_A\right]$, such that ϱ is chosen to be a nonnegative (a.s.) random variable on (Ω, ℱ) having unit expectation under measure ℙ:

$\text{E}\left[\varrho \right]\equiv {\displaystyle {\int }_{\text{Ω}}\varrho \left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)=1.}$

Note that in order for $\widehat{\mathbb{P}}$ to be a probability measure we necessarily have $1=\widehat{\mathbb{P}}\left(\text{Ω}\right)=\text{E}\left[\varrho {\mathbb{I}}_{\text{Ω}}\right]=\text{E}\left[\varrho \right]$. The measure $\widehat{\mathbb{P}}$ is also countably additive from the countable additivity property of the Lebesgue integral w.r.t. ℙ, i.e., for any countable collection of pairwise disjoint sets {Ai} ∊ ℱ we have, setting A ≡ ∪iAi in (9.109),

$\widehat{\mathbb{P}}\left({\cup }_i{A}_i\right)=\text{E}\left[\varrho {\mathbb{I}}_{{\cup }_{\text{i}}{A}_i}\right]=\text{E}\left[\varrho {\displaystyle \sum _i\mathbb{I}}{A}_i\right]={\displaystyle \sum _i\text{E}\left[\varrho {\mathbb{I}}_{Ai}\right]={\displaystyle \sum _i\widehat{\mathbb{P}}\left(A_i\right).}}$

Hence, $\widehat{\mathbb{P}}$ is a probability measure and (Ω, ℱ, ℙ) is a probability space.

The random variable ϱ corresponds to the so-called Radon—Nikodym derivative of $\widehat{\mathbb{P}}$ w.r.t. ℙ. Note that $\text{d}\widehat{\mathbb{P}}\left(\omega \right)=\varrho \left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)$. It is customary notation to denote ϱ by $\frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}$, i.e., $\text{d}\widehat{\mathbb{P}}\left(\omega \right)=\frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)$. The notation arises naturally when we compute the expectation of a random variable in the two different measures. The expectation of a random variable X in the original ℙ-measure is denoted by E[X] and we let $\widehat{\text{E}}\left[X\right]$ denote the expectation in the new $\widehat{\mathbb{P}}$-measure. Using the definition in (9.109), the expectation of X under measure $\widehat{\mathbb{P}}$ equals the expectation of $X\varrho \equiv X\frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}$ under measure ℙ:

$\widehat{\text{E}}\left[\text{X}\right]={\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\text{d}\widehat{\mathbb{P}}\left(\omega \right)={\displaystyle {\int }_{\text{Ω}}X\left(\omega \right)\varrho \left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)=\text{E}\left[X\varrho \right]\equiv \text{E}\left[X\frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}\right]}.\left(9.110\right)}$

Moreover, if ϱ is strictly positive (a.s.) and Y is integrable under measure ℙ, then its expectation under measure ℙ equals the expectation of $\frac{Y}{\varrho }\equiv Y\frac{\text{d}\mathbb{P}}{\text{d}\widehat{\mathbb{P}}}$ under measure $\widehat{\mathbb{P}}$:

$\text{E}\left[Y\right]={\displaystyle {\int }_{\text{Ω}}Y\left(\omega \right)\text{d}\mathbb{P}\left(\omega \right)={\displaystyle {\int }_{\text{Ω}}Y\left(\omega \right)\frac{1}{\varrho \left(\omega \right)}\text{d}\widehat{\mathbb{P}}\left(\omega \right)=\widehat{\text{E}}\left[\frac{Y}{\varrho }\right]=\widehat{\text{E}}\left[Y\frac{\text{d}\mathbb{P}}{\text{d}\widehat{\mathbb{P}}}\right]}}.\left(9.111\right)$

Note that $\frac{\text{d}\mathbb{P}}{\text{d}\widehat{\mathbb{P}}}={\left(\frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}\right)}^{-1}=\frac{1}{\varrho }$ for any strictly positive ϱ.

Remark: Here we don't state the formal Radon-Nikodym theorem. There are different versions of it in measure theory where measures can be more general than probability measures. One version of the Radon-Nikodym theorem goes as follows. Given two finite measures μ and v on a space (Ω, ℱ) where v is absolutely continuous w.r.t. μ (i.e., all sets of μ-measure zero are v-measure zero), then there is a nonnegative ℱ-measurable function $h\equiv \frac{dv}{d\mu }$ such that the v-measure of any set A ∊ ℱ is given by a Lebesgue integral w.r.t. μ: ν(A) = ∫Ah dμ. Moreover, this function is unique w.r.t. measure μ. For our purposes, the Radon—Nikodym theorem guarantees that, given two equivalent probability measures $\widehat{\mathbb{P}}$ and ℙ, there is nonnegative random variable ϱ satisfying the above relations. Moreover, ϱ is unique (a.s.). The Radon—Nikodym theorem also applies to distribution measures of random variables and joint random variables.

We have already seen how measure changes are applied for discrete random variables. Let us briefly see how measure changes can be applied in the case of absolutely continuous random variables. Assume X ∊ℝ has a PDF ƒ(x) under measure ℙ and a PDF $\widehat{f}\left(x\right)$ under measure $\widehat{\mathbb{P}}$. Although we can further generalize, we shall also assume that these densities are (a.e.) positive on ℝ. Then, for any b ∊ ℝ, we have the CDF of X under measure $\widehat{\mathbb{P}},{\widehat{F}}_X\left(b\right)\equiv {\widehat{\mu }}_X\left(\left(-\infty ,b\right]\right)$ as

${\widehat{F}}_X\left(b\right)=\widehat{\text{E}}\left[{\mathbb{I}}_{\left\{X\le b\right\}}\right]\equiv {\displaystyle {\int }_{-\infty }^b\widehat{f}\left(x\right)\text{d}x={\displaystyle {\int }_{-\infty }^b\frac{\widehat{f}\left(x\right)}{f\left(x\right)}f\left(x\right)\text{d}x\equiv \text{E}\left[\frac{\widehat{f}\left(X\right)}{f\left(X\right)}\mathbb{I}\left\{X\le b\right\}\right].\left(9.112\right)}}$

By the Radon—Nikodym derivative, we see that the ratio of densities gives the Radon—Nikodym derivative for changing distribution measures of X. In this case, $\frac{\text{d}{\widehat{\mu }}_X}{\text{d}{\mu }_{\text{X}}}\left(x\right)=\frac{\widehat{f}\left(x\right)}{f\left(x\right)}$. This is known as a likelihood ratio. As a random variable we have the Radon—Nikodym derivative $\varrho =\varrho \left(X\right)=\frac{\widehat{f}\left(x\right)}{f\left(x\right)}$. This also generalizes to the multidimensional case in the obvious manner as the ratio of the joint PDFs.

An example of the likelihood ratio in measure changes is to consider a normal random variable. Say that X ~ Norm(a, σ2), i.e., that it has mean a and variance σ2 under measure ℙ. Then, defining the change of measure $\mathbb{P}\to \widehat{\mathbb{P}}$, i.e., change of distribution measure ${\mu }_X\to {\widehat{\mu }}_X$, by the Radon—Nikodym random variable

$\varrho =\frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}=\frac{\sigma }{\widehat{\sigma }}\exp \left[\frac{{\left(X-a\right)}^2}{2{\sigma }^2}-\frac{{\left(X-\widehat{a}\right)}^2}{2{\widehat{\sigma }}^2}\right]\equiv h\left(X\right)$

we have that X ~ Norm$\left(\widehat{a},{\widehat{\sigma }}^2\right)$ under measure $\widehat{\mathbb{P}}$. This follows from the above likelihood ratio of densities. The PDF of X under measure $\widehat{\mathbb{P}}$ equals its PDF under measure ℙ times the likelihood ratio $\frac{\text{d}{\widehat{\mu }}_X}{d{\mu }_X}\left(x\right)=h\left(x\right)=\frac{\widehat{f}\left(x\right)}{f\left(x\right)}$. Hence,

$h\left(X\right)=\frac{\widehat{f}\left(X\right)}{f\left(X\right)}=\frac{\frac{1}{\widehat{\sigma }\sqrt{2\pi }}{\text{e}}^{\frac{-{\left(X-\widehat{a}\right)}^2}{2{\widehat{\sigma }}^2}}}{\frac{1}{\sigma \sqrt{2\pi }}{\text{e}}^{\frac{-{\left(X-a\right)}^2}{2{\sigma }^2}}}$

which gives the above result. We note that this change of measure changes both the mean and variance of a normal random variable.

The next (and last) result of this chapter gives us a formula for computing the expectation (under a given measure ℙ) of a random variable conditional on any sub-σ-algebra G ⊂ F via the corresponding conditional expectation of ϱX under another equivalent measure $\widehat{\mathbb{P}}$. The new measure $\widehat{\mathbb{P}}$ is defined in (9.109) with (Radon—Nikodym derivative) random variable ϱ. The theorem is useful when considering measure changes while calculating conditional expectations involving stochastic processes such as those driven by Brownian motions. The sub-σ-algebras are part of a filtration for Brownian motion. The conditioning on the filtration simplifies even further when we are dealing with practical applications involving Markov processes.

Theorem 9.7

(General Bayes Formula). Let (Ω, ℱ, ℙ) and $\left(\Omega ,ℱ,\widehat{\mathbb{P}}\right)$ be two probability spaces with $\mathbb{P}~\widehat{\mathbb{P}}$ (i.e., ℙ and $\widehat{\mathbb{P}}$ are equivalent probability measures). Let the random variable X be integrable w.r.t. $\widehat{\mathbb{P}}$ and set $\varrho \equiv \frac{\text{d}\widehat{\mathbb{P}}}{\text{d}\mathbb{P}}$. Then, the random variable ϱX is integrable w.r.t. ℙ and its expectation under measure $\widehat{\mathbb{P}}$ conditional on a sub-σ-algebra G ⊂ ℱ is given by (a.s.)

$\widehat{\text{E}}\left[X|\mathcal{G}\right]=\frac{\text{E}\left[\varrho X|\mathcal{G}\right]}{\text{E}\left[\varrho |\mathcal{G}\right]}.\left(9.113\right)$

Proof. All we need to verify is that the right-hand side of (9.113), $Y:=\frac{\text{E}\left[\varrho X|\mathcal{G}\right]}{\text{E}\left[\varrho |\mathcal{G}\right]}$, is (almost surely) the random variable $\widehat{\text{E}}\left[X|\mathcal{G}\right]$, i.e., the expectation of X under $\widehat{\mathbb{P}}$ conditional on $\mathcal{G}$. Note that Y must satisfy the two properties in Definition 6.12 with $\widehat{\mathbb{P}}$ as measure. Hence, we need to show: (i) Y is $\mathcal{G}$-measurable; (ii) $\widehat{\text{E}}\left[{\mathbb{I}}_{A}Y\right]=\widehat{\text{E}}\left[{\mathbb{I}}_{A}X\right]$, for every event $A\in \mathcal{G}$. Property (i) is obviously satisfied since Y is a ratio of two $\mathcal{G}$-measurable random variables and hence is $\mathcal{G}$-measurable. Property (ii) is shown by first applying the change of measure $\widehat{\mathbb{P}}\to \mathbb{P}$ for an unconditional expectation via (9.110), then making use of the tower property $\text{E}\left[\text{E}\left[\cdot |\mathcal{G}\right]\right]=\text{E}[\cdot ]$ in reverse order, pulling out the $\mathcal{G}$-measurable random variable ?A Y in the inner conditional expectation, cancelling out the E[ϱ | $\mathcal{G}$] term, re-applying the tower property and changing back measures $\mathbb{P}\to \widehat{\mathbb{P}}$ in the final expectation:

$\begin{array}{ll}\hfill \widehat{\text{E}}\left[{\mathbb{I}}_{A}Y\right]=\text{E}\left[\varrho {\mathbb{I}}_{A}Y\right]=\text{E}\left[\text{E}\left[\varrho {\mathbb{I}}_{A}Y|\mathcal{G}\right]\right]=\text{E}\left[{\mathbb{I}}_{A}Y\text{E}\left[\varrho |\mathcal{G}\right]\right]& =\text{E}\left[{\mathbb{I}}_A\text{E}\left[\varrho X|\mathcal{G}\right]\right]\hfill \\ & =E\left[E\left[\varrho {\mathbb{I}}_{A}X|\mathcal{G}\right]\right]\hfill \\ & =\text{E}\left[\varrho {\mathbb{I}}_{A}X\right]=\widehat{\text{E}}\left[{\mathbb{I}}_{A}X\right].\hfill \end{array}$

Finally, the assumption that X is integrable w.r.t. $\widehat{\mathbb{P}}$, i.e., $\widehat{\text{E}}\left[|X|\right]<\infty$, implies that ϱX is integrable w.r.t. ℙ by changing measures:

$\widehat{\text{E}}\left[|X|\right]=\text{E}\left[\varrho |X|\right]=\text{E}\left[|\varrho X|\right]<\infty .$

1The definition also extends to the case X: Ω → [0, ∞] where X can equal ∞, i.e., X = X · ?{0≤X < ∞} + ∞ · ?{X=∞} where the usual convention 0 · ∞ = 0 is adopted. If ℙ(X = ∞) = 0, then we simply take X = X · ?{0≤X < ∞}. If ℙ(X = ∞) > 0, i.e., the set {ω ∊ Ω: X(ω) = ∞} has positive ℙ-measure, then E[X] = ∞.

2The Lebesgue measure m of an interval In = [an, bn], or (an, bn], or [an, bn), or (an, bn), is its length, m(In) = ℓ(In) ≔ bn – an, an ≤ bn. The measure m of a Lebesgue-measurable set (for our purposes a Borel set) B is defined precisely as the smallest total length among all countable unions of intervals in ℝ that (cover) contain B, i.e., for any $B\in ℬ\left(ℝ\right),m\left(B\right):=\inf \left\{{\displaystyle {\sum }_{n=1}^{\infty }\ell }\left(I_n\right):B\subset {\cup }_{n=1}^{\infty }{I}_n\right\}$.

3There also exist very special types of random variables X where FX is not an absolutely continuous function, yet it has zero derivative $F_X^{\prime }\left(x\right)\equiv 0$ on a set of Lebesgue measure zero (a.e.). In this case X is said to be singularly continuous where the CDF is a nondecreasing monotone continuous function with zero derivative (a.e.) and hence there does not exist a PDF fX, i.e., (9.36) (and (9.38)) does not hold. The expectation of g(X) is still defined as a Lebesgue integral in (9.28). The so-called Cantor function on [0, 1] is a well-known textbook example of a CDF of a singularly continuous random variable that is uniformly distributed on the Cantor set C. The Cantor CDF is constant on the complement of the Cantor set, i.e., $F_X^{\text{'}}\left(x\right)\equiv 0\text{for}x\in \left[0,1\right]C.$ There are other known interesting properties of such random variables. However, throughout this text we will have no need for such singular cases so that all continuous random variables are also absolutely continuous.