Appendix A

Projections in Hilbert spaces

A.1 Introduction

In many situations we want information about variables that are not directly measured, assuming that we have information about some variables which are correlated with the unmeasured variable. If this (cross)correlation is known or estimated, then it can be used for estimating the value of the unmeasured variables.

Consider for instance the interest rates. Short term interest rates are quoted on a daily basis in the money markets for maturities up to, say, one year; but longer term interest rates are traded only indirectly through the bond markets. Theoretically, options dependent on interest rates are priced according to a stochastic process describing the evolution in continuous time of the short term interest rate even though this process is not directly observable. The observed (or measured) variables, when modelling interest rate processes, are the bond prices.

The Wiener filter (see Madsen [2007]) is an example where the known cross-correlation is used together with the projection theorem to estimate an unmeasured time series based on a measured time series. The Kalman filter, which will be introduced in this appendix, is some sort of online version of the Wiener filter.

The main goal of this appendix is to present the projection theorem, and to illustrate the wide range of applications of this theorem. Finally the theorem is used to formulate the (ordinary) Kalman filter. The contents of this appendix is basically based on Madsen [1992] and Brockwell and Davis [1991], and more information about the theory and applications of the projection theorem can be found in those references.

One of the advantages by considering the projection theorem as it is formulated in this appendix is that many of the well-known concepts from two- and three-dimensional Euclidean geometry, such as orthogonality, carry over to the more general Hilbert spaces considered in the following.

By using the projection theorem it can be realized that a unified set of equations can be used in different contexts. Hence it can be shown that many of the methods used in time series analysis, such as prediction, filtering and estimation, are seen in a unified context.

A.2 Hilbert spaces

A Hilbert space is simply an inner-product space, i.e. a vector space supplied with an inner product, with an additional property of completeness. The inner product is a natural generalization of the inner (or scalar) product of two vectors in n-dimensional Euclidean space. Since many of the properties of Euclidean space carry over to the inner-product spaces, it will be helpful to keep Euclidean space in mind in all that follows.

Let us first consider a well-known inner-product space, namely the Euclidean space.

Example A.1 (Euclidean space).

The set of all column vectors

$x={(x_1,...,x_k)}^T∊{ℝ}^k(\text{A}.1)$

is a real inner-product space if we define

$\langle x,y\rangle ={\displaystyle \sum _{i=1}^k{x}_i{y}_i.}(\text{A}.2)$

It is a simple matter to check that the conditions above are all satisfied.

Definition A.1 (Norm).

Let ||x|| > 0, if x ≠ 0, then the norm of an element x of an inner-product space is defined to be

$||x||=\sqrt{\langle x,x\rangle }.(\text{A}.3)$

In the Euclidean space ℝk the norm of the vector is simply its length.

Definition A.2 (The angle between elements).

The angle θ between two nonzero elements x and y belonging to any real inner-product space is defined as

$\theta =arccos[\langle x,y\rangle /(||x||y||)].(\text{A}.4)$

In particular x and y are said to be orthogonal if and only if 〈x, y〉 = 0.

Now let us define the Hilbert space:

Definition A.3 (Hilbert space).

A Hilbert space ℋ is vector space, equipped with an inner product, in which every Cauchy sequence xn converges in norm to some element in x ∊ ℋ. The inner-product space is then said to be complete.

Example A.2 (Euclidean space).

The completeness of the inner-product space ℝk can be verified. Thus ℝk is a Hilbert space.

Example A.3 (The space L2(Ω, ℱ, ℙ)).

Consider a probability space (Ω, ℱ, ℙ) and the collection C of all random variables X defined on Ω and satisfying the condition E[X2] ≤ ∞. It is rather easy to show that C is a vector space.

For any two elements X,Y ∊ C we now define the inner product

$\langle x,y\rangle =E[XY].(\text{A}.5)$

Norm convergence of a sequence Xn of elements of L2 to the limit X means

$||X_n-X{||}^2=E[|X_n-X{|}^2]\to 0asn\to \infty .(\text{A}.6)$

Norm convergence of Xn to X in an L2 space is called mean-square convergence and is written as $X_n\stackrel{m.s.}{\to }X$.

To complete the proof that L2 is a Hilbert space we need to establish completeness, i.e. that if ||Xn − X ||2 → 0 as m, n → ∞, then there exists X ∊ L2 such that Xn → X (see Brockwell and Davis [1991]).

A.3 The projection theorem

Let us start by considering two simple applications which illustrate the projection theorem in the two types of Hilbert spaces.

Example A.4 (Linear approximation in ℝ3).

Suppose three vectors are given in ℝ3.

$\begin{array}{llll}y\hfill & =\hfill & {(1/4,1/4,1)}^T,\hfill & (\text{A}.7)\hfill \\ x_1\hfill & =\hfill & {(1,0,1/4)}^T,\hfill & (\text{A}.8)\hfill \\ x_2\hfill & =\hfill & {(0,1,1/4)}^T.\hfill & (\text{A}.9)\hfill \end{array}$

Our problem is to find the linear combination $\widehat{y}={\alpha }_1{x}_1+{\alpha }_2{x}_2$ which is closest to y in the sense that S = ||y − α1x2 − α2x2||2 is minimized.

One approach to this problem is to write S in the form S = (1/4 − α1)2 + (1/4 − α2)2 + (1 − 1/4α1 − 1/4α2)2 and then to use calculus to minimize with respect to α1 and α2. In the alternative geometric approach we observe that the required vector $\widehat{y}={\alpha }_1{x}_1+{\alpha }_2{x}_2$ is the vector in the plane determined by x1 and x2 such that y − α1x1 − α2x2 is orthogonal to the plane of x1 and x2 (see the figure). The orthogonality condition may be stated as

$\langle y-{\alpha }_1{x}_1-{\alpha }_2{x}_2,x_i\rangle =0i=1,2(\text{A}.10)$

or equivalently

$\begin{array}{llll}{\alpha }_1\langle x_1,x_1\rangle +{\alpha }_2\langle x_2,x_1\rangle \hfill & =\hfill & \langle y,x\rangle \hfill & (\text{A}.11)\hfill \\ {\alpha }_1\langle x_1,x_2\rangle +{\alpha }_2\langle x_2,x_2\rangle \hfill & =\hfill & \langle y,x_2\rangle .\hfill & (\text{A}.12)\hfill \end{array}$

By solving these two equations for the particular values of x1, x2 and y specified, it is seen that α1 = α2 = 4/9, and $\widehat{y}=(4/9,4/9,2/9{)}^{\prime }$.

Example A.5 (Linear approximation in L2(Ω, ℱ,P)).

Now suppose that X1, X2 and Y are random variables in L2(Ω, ℱ, P). Only X1 and X2 are observed and we wish to estimate the value of Y by using the linear combination $\widehat{Y}={\alpha }_1{X}_1+{\alpha }_2{X}_2$ which minimizes the mean square error,

$S=E|Y-{\alpha }_1{X}_1-{\alpha }_2{X}_2{|}^2=||Y-{\alpha }_1{\text{X}}_1-{\alpha }_2{X}_2{||}^2.(\text{A}.13)$

As in the previous example there are at least two possible approaches to the problem. The first is to write

$S=E[Y^2]+{\alpha }_1^{2}E[X_1^2]+{\alpha }_2^{2}E[X_2^2]-2{\alpha }_{1}E[Y{X}_1]-2{\alpha }_{2}E[Y{X}_2]+{\alpha }_1{\alpha }_{2}E[X_1{X}_2](A.14)$

and then to minimize with respect to α1 and α2.

The second method is to use the same geometric approach as in the previous example. Our aim is to find an element in $\widehat{Y}$ in the set

$ℳ=\{X∊L^2(\Omega ,ℱ,P):X={\alpha }_1{X}_1+{\alpha }_2{X}_2({\alpha }_1,{\alpha }_2∊ℝ)\}(\text{A}.15)$

which implies that the mean square error $||Y-\widehat{Y}|{|}^2$ is as small as possible. By analogy with the previous example we might expect $\widehat{Y}$ to have the property that $Y-\widehat{Y}$ is orthogonal to all elements of ℳ. Applying it to our present problem, we can write

$\langle Y-{\alpha }_1{X}_1-{\alpha }_2{X}_2,X\rangle =0forallX∊ℳ(\text{A}.16)$

or, equivalently, by the linearity of the inner product,

$\langle Y-{\alpha }_1{X}_1-{\alpha }_2{X}_2,X_i\rangle =0,i=1,2.(\text{A}.17)$

These are the same equations for α1 and α2 as in the previous example, although the inner product is of course defined differently in (A.17). In terms of expectations we can rewrite (A.17) in the form

$\begin{array}{llll}{\alpha }_{1}E[X_1^2]+{\alpha }_{2}E[X_2{X}_1]\hfill & =\hfill & E[Y{X}_1]\hfill & (\text{A}.18)\hfill \\ {\alpha }_{1}E[X_1{X}_2]+{\alpha }_{2}E[X_2^2]\hfill & =\hfill & E[Y{X}_2]\hfill & (\text{A}.19)\hfill \end{array}$

from which α1 and α2 are easily found.

Before establishing the projection theorem for a general Hilbert space we need to introduce some new terminology.

Definition A.4 (Closed subspace).

A linear subspace ℳ of a Hilbert space ℋ is said to be a closed subspace of ℋ if ℳ contains all of its limit points (i.e. if xn ∊ M and ||xn − x|| → 0 imply that x ∊ ℳ).

Definition A.5 (Orthogonal complement).

The orthogonal complement of a subset ℳ of ℋ is defined to be the set ℳ⊥ of all elements of ℋ which are orthogonal to every element of ℳ. Thus

$x∊M^{\perp }ifandonlyif\langle x,y\rangle =0(writtenx\perp y)(\text{A}.20)$

for all y ∊ M.

Theorem A.1.

If ℳ is any subset of a Hilbert space ℋ then ℳ⊥ is a closed subspace of ℋ.

Proof. Omitted.

Theorem A.2 (The projection theorem).

If ℳ is a closed subspace of the Hilbert space ℋ and x ∊ ℋ, then

1. there is a unique element $\widehat{x}∊ℳ$ such that

$||x-\widehat{x}||=\underset{y∊ℳ}{inf}||x-y||(\text{A}.21)$

and

2. $\widehat{x}∊ℳ$ and $||x-\widehat{x}||={inf}_{y∊M}||x-y||$ if and only if $\widehat{x}∊ℳ$ and $(x-\widehat{x})∊{ℳ}^{\perp }$. The element $\widehat{x}$ is called the (orthogonal) projection of x onto ℳ.

Proof. Omitted – see Brockwell and Davis [1991].

Theorem A.3 (The projection mapping of ℋ onto ℳ).

If ℳ is a closed subspace of the Hilbert space ℋ and I is the identity mapping on ℋ, then there is a unique mapping Pℳ of ℋ onto M such that I − Pℳ maps ℋ onto ℳ⊥. Pℳ is called the projection mapping of ℋ onto ℳ.

Proof. By the projection theorem, for each x ∊ ℋ there is a unique $\widehat{x}∊ℳ$ such that $x-\widehat{x}∊{ℳ}^{\perp }$. The required mapping is therefore

$P_{ℳ}x=\widehat{x}x∊\mathscr{H}.(\text{A}.22)$

Theorem A.4 (Properties of projection mappings).

Let ℋ be a Hilbert space and let Pℳ denote the projection mapping onto a closed subspace ℳ. Then

1. Pℳ (αx + βy) = α Pℳx + β Pℳy.
2. ||x||2 = ||Pℳx||2 + ||(I − Pℳ)x||2.

3. each x ∊ ℋ has a unique representation as a sum of an element of ℳ and an element of ℳ⊥ i.e.

   $x=P_{ℳ}x+(I-P_{ℳ})x.(\text{A}.23)$

4. Pℳxn → Pℳx if ||xn − x|| → 0.
5. x ∊ ℳ if and only if Pℳx = x.
6. x ∊ ℳ⊥ if and only if Pℳx = 0.
7. ℳ1 ⊆ ℳ2 if and only if Pℳ1 Pℳ2 x = Pℳ1 x for all x ∊ ℋ.

Proof. Omitted – but rather obvious from a geometrical point of view.

A.3.1 Prediction equations

In the following a set of equations, the so-called prediction equations will be derived. The equations describe how to find the projection that gives the minimum mean square error (Minimum MSE).

Given a Hilbert space ℋ, a closed subspace ℳ and an element x ∊ ℋ, then the projection theorem shows that the element of ℳ closest to x is the unique element $\widehat{x}∊ℳ$ such that

$\langle x-\widehat{x},y\rangle =0\text{for}\text{all}y∊ℳ.(\text{A}.24)$

Compare the general equation above with the special cases in the examples prior to projection theorem.

The quantity, $\widehat{x}=P_{ℳ}x$, is frequently called the best predictor of x in the subspace ℳ.

Remark A.1.

It is helpful to visualize the projection theorem in terms of Figure A.1, which depicts the special case in which ℋ = ℝ3, and ℳ is the plane containing x1 and x2, and $\widehat{y}=P_{ℳ}y$. The prediction equation (A.24) is simply the statement that $y-\widehat{y}$ must be orthogonal to ℳ. The projection theorem tells us that $\widehat{y}=P_{ℳ}y$ is uniquely determined by this condition for any Hilbert space ℋ and closed subspace ℳ.

The projection theorem and the prediction equations play fundamental roles in time series analysis, especially for estimation, approximation, filtering and prediction. Examples will be given.

Example A.6 (Minimum MSE linear prediction).

Let {Xt, t = 0, ±1,...} be a stationary process on (Ω, ℱ, P) with mean zero and autocovariance function γ(·), and consider the problem of finding the best linear combination

${\widehat{X}}_{n+1}={\displaystyle \sum _{j=1}^n{\varphi }_{nj}{X}_{n+1-j}}(\text{A}\text{.25})$

which best approximates Xn+1 in the sense that $E[{(X_{n+1}-{\text{Σ}}_{j=1}^n{\varphi }_{nj}{X}_{n+1-j})}^2]$ is minimum. This problem is easily solved with the aid of the projection theorem by taking ℋ = L2(Ω, ℱ, ℙ) and $ℳ=\{{\text{Σ}}_{j=1}^n{\alpha }_j{X}_{n+1-j}:{\alpha }_1,...,{\alpha }_n∊ℝ\}$. Since minimization of $E[|X_{n+1}-{\widehat{X}}_{n+1}{|}^2]$ is identical to minimization of the squared norm $||X_{n+1}-{\widehat{X}}_{n+1}|{|}^2$, we see at once that ${\widehat{X}}_{n+1}=P_{ℳ}{X}_{n+1}$. The prediction equations are

$\langle X_{n+1}-{\displaystyle \sum _{j=1}^n{\varphi }_{nj}{X}_{n+1-j},Y}\rangle =0forallY\in ℳ(\text{A}\text{.26})$

which, by the linearity of the inner product, are equal to the n equations

$\langle X_{n+1}-{\displaystyle \sum _{j=1}^n{\varphi }_{nj}{X}_{n+1-j},X_k}\rangle =0k=n,n-1,\mathrm{...},1.(\text{A}\text{.27})$

Recalling the definition 〈X, Y〉 = E[XY] of the inner product in L2(Ω, ℱ, ℙ), we see that the prediction equations can be written in the form

${\text{Γ}}_n{\varphi }_n={\gamma }_n(\text{A}.28)$

where ϕn = (ϕn1,..., ϕnn)′, γn = (γ(1),..., γ(n))′ and ${\text{Γ}}_n={[\gamma (i-j)]}_{i,j=1}^n$. The projection theorem guarantees that there is at least one solution ϕn to the problem. If Γn is singular then there are infinitely many solutions, but the projection theorem guarantees that every solution will give the same (uniquely defined) predictor.

A.4 Conditional expectation and linear projections

It is well known that the conditional expectation plays a central role in time series analysis, as the optimal prediction (under some mild assumptions) is found using the conditional expectation.

Consider the random variables Y and X from L2.

Definition A.6 (The conditional expectation).

The conditional expectation of X given Y = y is

$E[X|Y=y]={\int }_{-\infty }^{\infty }x{f}_{X|Y=y}(x)\text{d}x(A.29)$

where fX|Y = y(x) is the conditional density function for X given Y = y.

Remember that E[X|Y = y] is a number, whereas E[X|Y] is a stochastic variable.

It can be shown that the operator E[X|Y] on L2 has all the properties of a projection operator, in particular

$\begin{array}{lll}\hfill E[cX+dZ|Y]& =\hfill & cE[X|Y]+dE[Z|Y],\hfill \\ \hfill E[1|Y]& =\hfill & 1.\hfill \end{array}$

Theorem A.5 (Best mean square predictor).

The conditional expectation E[X|Y] is the best mean square predictor of X in ℳY, i.e. the best function of Y for predicting X.

Proof. Follows from the projection theorem.

However, the determination of projections on ℳY is usually very difficult. On the other hand it is relatively easy instead to calculate the projection of X on span{1, Y} ⊆ ℳY, i.e. the linear projection

$E[X|Y]=a+bY(\text{A}.30)$

which gives a subset of the best function of Y (in the mean square sense) for predicting X.

The linear projection (A.30) is a projection of X onto a subspace of ℳY. Therefore it can never have a smaller mean square error than E[X|Y]. However it is of great importance for the following reasons:

• The linear projection (A.30) is easier to calculate.
• It depends only on the first and second order moments, E[Y], E[X], E[Y2], E[X2] and E[XY], of the joint distribution of (Y,X).

• If (Y,X)′ has a multivariate normal distribution then the conditional expectation is linear, i.e.

  $\text{span}\{1,Y\}={ℳ}_Y.(\text{A}.31)$

Let us now consider two multivariate stochastic variables X and Y and the corresponding second order representation (first and second order moments for (X,Y)′)

${\mu }_Y,{\mu }_x,{\text{Σ}}_{xx},{\text{Σ}}_{XY},{\text{Σ}}_{yy}.(\text{A}.32)$

Theorem A.6 (Linear projection in L2).

Given the second order representation for (X,Y)′ the linear projection is given by

$E[X|Y]={\mu }_x+{\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1}(Y-{\mu }_y)(\text{A}.33)$

and the variance is

$\mathrm{Var}[X-E[X|Y]]={\text{Σ}}_{XX}-{\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1}{\text{Σ}}_{YX}.(\text{A}.34)$

Furthermore

$\text{Cov}[X-E[X|Y],Y]=0,(\text{A}.35)$

i.e. the error X − E[X|Y] is uncorrelated with Y.

Proof. From the prediction equations:

$\begin{array}{lll}\langle X-E[X|Y],Y\rangle \hfill & =\hfill & 0,\hfill \\ \langle X-E[X|Y],1\rangle \hfill & =\hfill & 0,\hfill \end{array}$

or

$\begin{array}{lll}\langle a+bY,Y\rangle \hfill & =\hfill & 0,\hfill \\ \langle a-bY,1\rangle \hfill & =\hfill & 0.\hfill \end{array}$

Using the fact that in the multivariate case the inner product in L2 is 〈X,Y〉 = E[XYT] we get

$\begin{array}{rrr}\hfill aE{[Y]}^T+bE[Y{Y}^T]& \hfill =& \hfill E[X{Y}^T],\\ \hfill a+bE[Y]& \hfill =& \hfill E[X].\end{array}$

By solving these equations and using the fact that ΣXY = E[XY′] − E[X]E[Y]′ we obtain

$\begin{array}{llll}b\hfill & =\hfill & {\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1},\hfill & (\text{A}.36)\hfill \\ a\hfill & =\hfill & {\mu }_X-{\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1}{\mu }_y.\hfill & (\text{A}.37)\hfill \end{array}$

Hence the linear projection is

$E[X|Y]={\mu }_X-{\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1}(Y-{\mu }_Y).(\text{A}.38)$

The variance follows immediately

$\begin{array}{lll}\mathrm{Var}[X-E[X|Y]]\hfill & =\hfill & \mathrm{Var}[X-a-bY]\hfill \\ \hfill & =\hfill & {\text{Σ}}_{XX}+b{\text{Σ}}_{YY}{b}^T-b{\text{Σ}}_{YX}-{\text{Σ}}_{XY}{b}^T\hfill \\ \hfill & =\hfill & {\text{Σ}}_{XX}-{\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1}{\text{Σ}}_{YX}.\hfill \end{array}$

The orthogonality between the error X − E[X|Y] and Y follows directly from the projection theorem.

Theorem A.7.

If (X, Y)T has a normal distribution then X|Y is normal distributed with mean

$E[X|Y]={\mu }_X-{\text{Σ}}_{XY}{\text{Σ}}_{YY}^{-1}(Y-{\mu }_y)(\text{A}.39)$

and variance

$\mathrm{Var}[X|Y]=\mathrm{Var}[X-E[X|Y]]={\text{Σ}}_{XX}-{\text{Σ}}_{XY}{\text{Σ}}_{\text{YY}}^{-1}{\text{Σ}}_{YX}.(\text{A}.40)$

The error X − E[X|Y] and Y are stochastic independent.

Proof. Omitted.

Let us illustrate the importance of the equations above by a couple of examples.

Example A.7 (Regression).

Let us consider the regression in L2 of Y on X

$E[Y|X]=X\theta (\text{A}.41)$

and assume that E[X] = E[Y] = 0.

Note that — compared to the discussion above — we have interchanged X and Y. And in order to compare the results directly with the ordinary LS estimator for the general linear model in ℝn we have also interchanged X and θ.

The best estimator is found by the prediction equations

$\langle Y-E[Y|X],X\rangle =0(A.42)$

or

$\langle Y-X\theta ,X\rangle =0.(\text{A}.43)$

Then we get

${\text{Σ}}_{YX}-{\theta }^T{\text{Σ}}_{XX}=0(\text{A}.44)$

or

$\widehat{\theta }={\text{Σ}}_{XX}^{-1}{\text{Σ}}_{XY}.(\text{A}.45)$

Compare this result with the well-known LS estimator in ℛn.

Next an example where the formulation of the linear projection above is used directly. As this example is very important it is embedded in a section.

A.5 Kalman filter

As mentioned in the introduction, the Kalman filter can be used for estimating some variables, which are not directly measured, by using some measured variables, which are correlated with the unmeasured variables. In the case of the Kalman filter the correlation between the unmeasured variables X and the measured variables Y is described by a linear state space model.

Consider the linear stochastic state space model

$\begin{array}{cc}\hfill X_t=A_t{X}_{t-1}+B_t{u}_{t-1}+e_{1,t},\hfill & \hfill (\text{A}.46)\hfill \\ \hfill Y_t=C_t{X}_t+e_{2,t},\hfill & \hfill (\text{A}.47)\hfill \end{array}$

where Xt is a m-dimensional state vector, ut is the input vector and Yt is the measured output vector. The matrices At, Bt and Ct are known and have appropriate dimensions.

The two white noise sequences {e1,t} and {e2,t} are mutually uncorrelated with variance Σ1,t and Σ2,t, respectively.

The matrices At, Bt, Ct, Σ1,t and Σ2,t might be time varying, as indicated by the notation. However, in the rest of this example we skip the index t although all the given results are valid in the time varying case.

Let us consider the problem of estimating Xt+k given the observations {Ys; s = t, t − 1,...} and input {us, s = t − 1,...}. In the case k = 0 the problem is called reconstruction or filtering. The solution to this problem is given by the linear projection theorem.

It is clear that the linear projection theorem also is valid for the conditioned stochastic variable (YX)′|Z. If the stochastic variables have a normal distribution we get

$\begin{array}{llll}E[X|Y,Z]\hfill & =\hfill & E[X|Z]+\text{Cov}[X,Y|Z]Va{r}^{-1}[Y|Z](Y-E[Y|Z]),\hfill & (\text{A}.48)\hfill \\ \mathrm{Var}[X|Y,Z]\hfill & =\hfill & \mathrm{Var}[X|Z]-\text{Cov}[X,Y|Z]Va{r}^{-1}[Y|Z]C^T[X,Y|Z].\hfill & (\text{A}.49)\hfill \end{array}$

Let us now introduce

${?}_t=(Y_1,...,Y_t),(\text{A}.50)$

which is a vector of all observations until time t. The input is assumed to be known.

Further introduce

$\begin{array}{llll}{\widehat{X}}_{t+k|t}\hfill & =\hfill & E[X_{t+k}|{?}_t],\hfill & (\text{A}.51)\hfill \\ {\widehat{Y}}_{t+k|t}\hfill & =\hfill & E[Y_{t+k}|{?}_t],\hfill & (\text{A}.52)\hfill \end{array}$

and the variances

$\begin{array}{llll}{\text{Σ}}_{t+k|t}^{xx}\hfill & =\hfill & \mathrm{Var}[X_{t+k}|{?}_t],\hfill & (\text{A}.53)\hfill \\ {\text{Σ}}_{t+k|t}^{yy}\hfill & =\hfill & \mathrm{Var}[Y_{t+k}|{?}_t],\hfill & (\text{A}.54)\hfill \\ {\text{Σ}}_{t+k|t}^{xy}\hfill & =\hfill & \text{Cov}[X_{t+k},Y_{t+k}|{?}_t],\hfill & (\text{A}.55)\hfill \end{array}$

then we have the Kalman filter

Theorem A.8 (Kalman filter — Optimal reconstruction).

The reconstruction ${\widehat{X}}_{t|t}$ which has the smallest mean square error is given by

${\widehat{X}}_{t|t}={\widehat{X}}_{t|t-1}+{\text{Σ}}_{t|t-1}^{xy}{\left({\text{Σ}}_{t|t-1}^{yy}\right)}^{-1}(Y_t-{\widehat{Y}}_{t|t-1})(\text{A}.56)$

and the variance of the reconstruction error is

${\text{Σ}}_{t|t}^{xx}={\text{Σ}}_{t|t-1}^{xx}-{\text{Σ}}_{t|t-1}^{xy}{\left({\text{Σ}}_{t|t-1}^{yy}\right)}^{-1}{\left({\text{Σ}}_{t|t-1}^{xy}\right)}^T.(\text{A}.57)$

Further the construction error and the observations are orthogonal, i.e.

$\text{Cov}[X_{x+k}-E[X_{t+k}|{?}_t],{?}_t]=0.(\text{A}.58)$

Proof. Let X = Xt, Y = Yt and Z = ?t−1 and use the linear projection theorem. See e.g., Madsen [2007] for details.

Together with equations for making one-step predictions in the state space model the above equations give the Kalman filter. It is readily seen that the prediction equations are

$\begin{array}{llll}{\widehat{X}}_{t+1|t}\hfill & =\hfill & A{\widehat{X}}_{t|t}+B{u}_t,\hfill & (\text{A}.59)\hfill \\ {\widehat{\text{Σ}}}_{t+1|t}^{xx}\hfill & =\hfill & A{\text{Σ}}_{t|t}^{xx}{A}^T+{\text{Σ}}_1,\hfill & (\text{A}.60)\hfill \\ {\widehat{\text{Σ}}}_{t+1|t}^{yy}\hfill & =\hfill & C{\text{Σ}}_{t+1|t}^{xx}{C}^T+{\text{Σ}}_2\hfill & (\text{A}.61)\hfill \end{array}$

with initial values

$\begin{array}{llll}{\widehat{X}}_{1|0}\hfill & =\hfill & E[X_1]={\mu }_0,\hfill & (\text{A}.62)\hfill \\ {\text{Σ}}_{1|0}^{xx}\hfill & =\hfill & \mathrm{Var}[X_1]=V_0.\hfill & (\text{A}.63)\hfill \end{array}$

We now leave the projections in L2 and continue by considering projections in ℝn.

A.6 Projections in ℝn

Previously we showed that ℝn is a Hilbert space with the inner product

$\langle x,y\rangle =x^{T}y.(\text{A}.64)$

In many statistical applications it is convenient to consider the weighted inner product

${\langle x,y\rangle }_{{\text{Σ}}^{-1}}=x^T{\text{Σ}}^{-1}y(\text{A}.65)$

where Σ is a positive definite symmetric matrix.

For both definitions of the inner product we have the norm

$||x||=\sqrt{\langle x,x\rangle }.(\text{A}.66)$

Consider a closed subspace ℳ of the Hilbert space ℝn. The following theorem enables us to compute Pℳx directly from any specified set of vectors {x1,...,xm} (m < n) spanning ℳ.

Theorem A.9.

If xi ∊ ℝn, i = 1,..., m, and ℳ = span{x1,..., xm} then

$p_{ℳ}x=X\beta (\text{A}.67)$

where X is the n × m matrix whose jth column is xj and

$X^{T}X\beta =X^{T}x.(\text{A}.68)$

Equation (A.68) has at least one solution for β but the prediction X β is the same for all solutions. There is exactly one solution of (A.68) if and only if X′X is non-singular and in this case

$P_{ℳ}x=X{(X^{T}X)}^{-1}{X}^{T}x.(\text{A}.69)$

Proof. Since Pℳx ∊ ℳ, we can write

$P_{ℳ}x={\displaystyle \sum _{i=1}^m{\beta }_i{X}_i=X\beta }.(\text{A}.70)$

The prediction equations (A.24) are equivalent in this case to

$\langle X\beta ,x_j\rangle =\langle x,x_j\rangle ,j=1,...,m(\text{A}.71)$

or in matrix form

$X^{T}X\beta =X^{T}x.(\text{A}.72)$

The existence of at least one solution for β is guaranteed by the existence of the projection Pℳx. The fact that Xβ is the same for all solutions is guaranteed by the uniqueness of Pℳx — see the projection theorem.

Remark A.2.

If {x1,..., xm} is a linearly independent set then there must be a unique vector β such that Pℳx = Xβ. This means that (A.68) must have a unique solution, which in turn implies that X′X is non-singular and

$P_{ℳ}x=X{(X^{T}X)}^{-1}{X}^{T}xforallx∊{ℝ}^n.(\text{A}.73)$

The matrix X(XTX)−1XT must be the same for all linearly independent sets {x1,..., xm} spanning ℳ since Pℳ is uniquely defined.

Remark A.3.

Given a real n × n matrix M, how can we tell whether or not there is a subspace ℳ of ℝn such that Mx = Pℳx for all x ∊ ℝn? If there is such a subspace we say that M is a projection matrix. Such matrices are characterized by the next theorem.

Theorem A.10.

The n × n matrix M is a projection matrix if and only if

• (a) MT = M, and
• (b) M2 = M, i.e. the matrix M is idempotent.

Proof. Omitted — but it is easily verified that (a) and (b) are satisfied for the matrix X(XTX)−1XT.