1. Basic Probability

Lecture 1

Prop 1.2.1 (De Morgan's)

$(\Uinf A_i)^C = \Ninf A_i^C$

Proof

Strategy: prove they are subsets of each other

$\omega \in (\Uinf A_i)^C$ $\omega \notin \Uinf A_i \implies \omega \notin A_i \fa i \implies \omega \in A_i^C \fa i \implies \omega \in \Ninf A_i^C$ $(\Uinf A_i)^C \subseteq \Ninf A_i^C$

$\omega \in \Ninf A_i^C$ $\omega \in A_i^C \fa i \implies \omega \notin A_i \fa i \implies \omega \notin \Uinf A_i \implies \omega \in (\Uinf A_i)^C$ $\Ninf A_i^C \subseteq (\Uinf A_i)^C$

$(\Ninf A_i)^C = \Uinf A_i^C$

Proof

$\omega \in (\Ninf A_i)^C$ $\omega \notin \Ninf A_i \implies \omega \notin A_i$ $\implies \omega \in A_i^C$ $\implies \omega \in \Uinf A_i^C$ $\subseteq$ RHS

$\omega \in \Uinf A_i^C$ $\omega \in A_i^C$ $\implies \omega \notin A_i$ $\implies \omega \notin \Ninf A_i, \implies \omega \in (\Ninf A_i)^C$ $\subseteq$ LHS

Def. Power Set

$2^\Omega$ $\Omega$ .

$\#(2^\Omega) = 2^{\#\Omega}$

Def. Sigma Algebra/Field

$\sa \subseteq 2^\Omega$ $\Omega$ $\Omega$ ) that:
contains the null set
$\null \in \sa$
closed under unions
$A_1, A_2, ... \in \sa \implies \Uinf A_i\in \sa$
closed under complementation
$A\in\sa \implies A^C\in \sa$

$\underbrace{\set{\null, \Omega}}_{coarsest\ \sigma\ algebra}\subseteq\t{ all other sigma algebras } \subseteq \underbrace{2^\Omega}_{finest\ \sigma\ algebra}$

Def. Probability Measure

$\Omega$ $\sigma$ $\sa$ $P: \sa \to [0, 1]$ with the following properties:
normed
$P(\Omega) = 1$
countably additive
$P(\Uinf A_i) = \suminf P(A_i)$ $A_i$

To turn finite additivity into countable additivity, add infinitely many null sets.

$P$ $\sa \subseteq 2^\Omega$ .

Prop 1.2.2 (Some Event Must Occur)

$(\Omega, \sa, P)$ $P(\null)$ = 0

Proof

$A_i = \null$ $i = 1, 2, ...$ $A_i$ $\Uinf A_i = \null$

$P(\null) > 0$ $P(\null) = \suminf P(\null) = \infty\.P(\null) = \infty$ $\contra P(\null)\in[0,1]$ $P(\null) = 0$

Lecture 2

Hierarchy $\omega$ $A$ $\sa$ $\sb^k$ )

Prop 1.3.1 (Intersection of Sigma Algebras)

${\sa_\lambda: \lambda \in \Lambda}$ $\sigs$ $\Omega$ $\N_{\lambda\in\Lambda}A_\lambda$ $\sigma$ $\Omega$

Proof

$\N_{\lambda\in\Lambda}A_\lambda$ $\sig$ :

$\null \in \sa_\lambda \fa \lambda \implies \null \in \N_{\lambda\in\Lambda}\sa_\lambda$
$A_1, A_2, ... \in \sa_\lambda \fa \lambda \implies \Uinf A_i \in \sa_\lambda \fa \lambda \implies \Uinf A_i \in \N_{\lambda \in \Lambda A_\lambda}$
$A\in\sa_\lambda \implies A^C \in \sa_\lambda \fa \lambda \implies A^C \in \N_{\lambda\in\Lambda}\sa_\lambda$

$\sig$ .

Def. Sigma Algebra Generated by C

$\sa(\sc)$ $\sigs$ $\sc \subseteq 2^\Omega$
$\sigma$ $\Omega$ $\sc$ .

Def. Borel Set

$\sb^k$ $\sigma$ algebra generated by open sets. Formally:

$\sigma$ $\Omega = \R^k$ $a = \[a1\\ \vdots\\ a_k\], b = \[b_1\\ \vdots\\ b_k\] \in \R^k$
$\sb^k = \d\bigtimes_{i=1}^k(a_i, b_i)$
$= (a_1, b_1] \times ... \times (a_k, b_k]$
$=\left\{(x_1, ..., x_k):a_k < x_k < b_k\right\}$

$\sb^k \ne \null$ $2^{\R^k}$ $\sb^k≠2^{\R^k}$ $A⊆\R^k$ that is not a borel set.

Loosely speaking, any set that can be defined explicitly is a borel set.
(Nice) transformations of borel sets are also borel sets.

Def. Ellipsoidal Region

$r$ $x_0$ $B_r(x_0) = \{x:(x-x_0)^T(x-x_0) = \norm{x_i - x_{0i}}^2 = \sumto k (x_i - x_{0i})^2 \le r^2\}\in\sb^k$
$S_r(x_0)$ $\le$ with =.

$y = Ax+b$ $B_r(x_0)$ $A$ $A = \[a_{11}&...&a_{1k}\\ \vdots &\ddots &\vdots\\ a_{k1} &... &a_{kk}\] \in \R^{k\times k}, b\in\R^k$

$\ba AB_r(x_0)+b &= \set{y: y=Ax+b \text{ for some x} \in B_r(x_0)}\\ &=\{y: $A^{-1}(y-b)-x_0$^T$A^{-1}(y-b)-x_0$\le r^2\} \quad\gr{\leftarrow(x = A^{-1}(y-b))}\\ &=\{y: (y\underbrace{-b-Ax_0}_{-\mu})^T \underbrace{(A^{-1})^T(A^{-1})}_{\Sigma^{-1}} (y\underbrace{-b-Ax_0}_{-\mu}) \le r^2\} \quad\gr{\leftarrow(\t{pull out } A^{-1})}\\ &=\{y:(y-\mu)^T \Sigma^{-1} (y-\mu) \le r^2\}\\ &= E_r(\mu, \Sigma) \in \sb^k \ea$
$\mu = Ax_0 + b$ $\Sigma = $(A\inv)^TA\inv$\inv= $(A^T)\inv A\inv$\inv = A^TA$ $r$

Recall:
$A^T = A$
$A\inv A = I$
$v^T A v \ge 0 \fa v \in \R^k$

$\Sigma$ is…

$\Sigma^T = (A^TA)^T = A^T(A^T)^T = A^TA = \Sigma$
$\Sigma\inv\.\Sigma = (AA^T)\inv\.(AA^T) = (A^T)\inv A\inv AA^T = (A^T)\inv A^T = I$
$w \in \R^k, w^T\Sigma w = w^T AA^T w = (A^Tw)^T(A^Tw) = \norm{A^Tw}^2 \ge 0$
- $A^T$ is invertible (transpose of an invertible matrix is invertible)

Note $\mu$ $\Sigma$ is the variance matrix.

Lecture 3

Def. Limit inferior/superior of a Sequence

$A_n \subseteq \Omega$ :
$\liminf A_n = \U_{n=1}^\infty\N_{i=n}^\infty A_i = \{\omega: \omega \t{ is in all but finitely many }A_i \}$
$\omega$ is a member of at least one of the intersections
$\limsup A_n = \N_{n=1}^\infty\U_{i=n}^\infty A_i = \{\omega: \omega \t{ is in infinitely many }A_i \}$
$\omega$ is a member of all the unions

Properties:

$A = \liminf A_n = \limsup A_n$ $A_n \to A$
$\liminf A_n \subseteq \limsup A_n$

Monotone Increasing/Decreasing Sequences

$\N_{i=n}^\infty A_i$ is an increasing sequence of sets (as i increases, fewer sets are intersected, the resulting intersection gets bigger)

$\U_{i=n}^\infty A_i$ is an decreasing sequence of sets (as i increases, fewer sets are unioned, the resulting union gets smaller)

Prop 1.4.1 (Monotone Sequences Converge)

A monotone decreasing sequence of sets converges to their intersection.
$A_n \in \sa \fa n$ $A_1 \supseteq A_2 \supseteq ...$ $A_n \to A = \Ninf A_i$

Proof

Need to prove that lim inf = lim sup:

$A_n \supseteq A_{n-1} \supseteq ...$ $\U_{i=n}^\infty A_i = A_n$ $\limsup A_n = \N_{n=1}^\infty\U_{i=n}^\infty A_i = \N_{n=1}^\infty A_n$

$\N_{i=n}^\infty A_i = \Ninf A_i$ $\liminf A_n = \U_{n=1}^\infty\N_{i=n}^\infty A_i = \Ninf A_i$ (if we union the same set over and over again, we get that set)

$\N_{i=n}^\infty A_i \supseteq \Ninf A_i \fa n$ $\subseteq$
$\omega \in \N_{i=n}^\infty A_i$ $\omega \in A_n \subseteq ... \subseteq A_1$ $\omega \in \Ninf A_i \implies$ $\N_{i=n}^\infty A_i \subseteq \Ninf A_i$
$\N_{i=n}^\infty A_i = \Ninf A_i$

$\liminf A_n = \N_{i=n}^\infty A_i = \Ninf A_i = \limsup A_n$ $A_n \to A = \Ninf A_i$

A monotone increasing sequence of sets converges to their union.
$A_n \in \sa \fa n$ $A_1 \subseteq A_2 \subseteq ...$ $A_n \to A = \Uinf A_i$

Proof

$A_n \subseteq A_{n+1} \subseteq ...$ $\N_{i=n}^\infty A_i = A_n$ $\liminf A_n = \U_{n=1}^\infty\N_{i=n}^\infty A_i = \U_{n=1}^\infty A_n$

$\U_{i=n}^\infty A_i = \Uinf A_i$ $\limsup A_n = \N_{n=1}^\infty\U_{i=n}^\infty A_i = \Uinf A_i$ (intersecting the same set over and over again gives that set)

$\liminf A_n = \limsup A_n$ $A_n \to \Uinf A_i$ .

Prop 1.4.2 (Continuity of P)

$A_n \in \sa \fa n$ $A_n \to A$ $P(A_n) \to P(A)$ $n\to\infty$
Note The converse is true

Proof

By the previous proposition, we know (1) & (2)

$\U_{i=n}^\infty A_i$ is a monotone decreasing sequence, it converges to the intersection
$\U_{i=n}^\infty A_i \to \Ninf \U_{i=n}^\infty A_i = \limsup A_n$

$\N_{i=n}^\infty A_i$ is a monotone increasing sequence, it converges to the union
$\N_{i=n}^\infty A_i \to \Uinf \N_{i=n}^\infty A_i = \liminf A_n$

$P(\Uinf A_i) \to P(\limsup A_n)$ $P(\Ninf A_i) \to P(\liminf A_n)$

$P(\Ninf A_i) \le P(A_n) \le P(\Uinf A_i) \implies P(\liminf A_n) \le P(A_n) \le P(\limsup A_n) \implies P(A_n) \to P(A)$

$A_n$ $A_n \to A = \Uinf A_i$

$B_i \in \sa$ $\bc\ba &B_1 = A_1,\\ &B_2 = A_2\n A_1^C\\ &B_3 = A_3 \n A_2^C\\&...\ea\ec$ $A_n = \Uto n B_i \implies P(A_n) = \sumto n P(B_i)$

$\d\lim_{n\to\infty} P(A_n) = \lim_{n\to\infty}\sumto n P(B_i) = \suminf P(B_i) = P(\Uinf B_i) = P(\Uinf A_i) = P(\lim_{n\to\infty} A_n)$

$A_n$ $A_n^C$ is monotone increasing.

$\d\lim_{n\to\infty}P(A_n^C) = P(\lim_{n\to\infty}A_n^C) = P(\Uinf A_i^C) = P((\Ninf A_i)^C) = 1 - P(\Ninf A_i) = P(\lim_{n\to\infty}A_n)$

Prop 1.4.3 (Prob Measure on a Sigma Algebra)

$P$ $\sa$ $P:\sa \to [0, 1]$ satisfies
$P(\Omega) = 1$
$P$ is additive
$P(A_n) \to P(A)$ $n\to\infty$ $A_n \in \sa \fa n$ $A_n\to A$

Proof

(1) and (2) are contained in the def of probability measure (normed and countably additive)

Combining additivity (2) with continuity (3), we have that P is countably additive:

$A_n \to A \implies \d\lim_{n\to\infty}P(A_n) = P(A)$

$B_n = \Uto n A_i$ $A_1, A_2, ... \in A$ are mutually disjoint.

$B_n$ $\lim B_n = \U_{n=1}^\infty B_n = \U_{n=1}^\infty\Uto n A_i = \Uinf A_i$

$P(\Uinf A_i) = P(\lim B_n) = \lim P(B_n) = \lim P(\Uto n A_i) = \lim \sumto n P(A_i) = \suminf P(A_i)$ ,

$\iff$ finite additivity

Important Note $\equiv$ continuity of P. By ensuring countable additivity, we ensure continuity of P, which is needed when we have an infinite sample space.

Def. Conditional Probability Model

$(\Omega, \sa, P)$ $\sc \in \sa$ $P(C) > 0$ $C$ $(\Omega, \sa, P(\.|C))$ $P(\.|C):\sa\to[0, 1]$ $P(A|C) = \f{P(A\n C)}{{P(C)}}$

Proof

$P(\Omega | C) = \f{P(\Omega \n C)}{P(C)} = \f{P(C)}{P(C)} = 1$

$A_n, A_2, ... \in A$ are mutually disjoint,

$P(\Uinf A_i|C) = \f{P(\Uinf A_i)\n C}{P(C)} = \f{P(\Uinf(A_i \n C))}{P(C)} = \f{\suminf P(A_i \n C)}{P(C)} = \suminf P(A_i|C)$

$P$ $(\Omega, \sa, P(\.|C))$ is a probability model.

Note $\sc, \sa\n C, P(\.|C)$ )

Prop 1.5.1 (LOTP / Thm of Total Prob.)

$C_1, C_2, ...\in \sa$ $P(C_i) > 0 \fa i$ $\Omega = \Uinf C_i$ $C_i \n C_j = \null, \fa i, j$ $A \in \sa, P(A) = \suminf P(C_i)P(A|C_i)$

Proof

$A = \Uinf A \n C_i$ $C_i \n A$
$\suminf P(A\n C_i) = \suminf \f{P(A\n C_i)}{P(C_i)}P(C_i) = \suminf P(A|C_i)P(C_i)$

Fact $C_i$ $\Omega$ $A = \Uinf(A\n C_i)$ $A \n C_i$ are mutually disjoint

Proof

$C_i \n C_j = \null$ $i \ne j$ $(A\n C_i) \n (A \n C_j) = \null$ $A = \Uinf (A \n C_i) = A \n \Uinf C_i$ $\Uinf C_i = \Omega$ )

Lecture 4

Def. Statistically Independent

$(\Omega, \sa, P)$ $A, C \in \sa$ $P(A\n C) = P(A)P(C)$
$P(C) > 0$ $P(A|C) = \f{P(A\n C)}{P(C)} = {P(A)P(C)\o P(C)} = P(A)$

Statistically Independent Sigma Algebras

$A$ $B$ $\sig$ $A: \set{\null, A, A^C, \Omega}$ $\sig$ $\set {\null, B, B^C, \Omega}$

Proof

$C$ $\null$ $C \n \null = \null \fa C$ $P(C\n\null) = P(\null)P(C) = P(\null) = 0$

$C$ $\Omega$ $C\n \Omega = C \fa C$ $P(C\n\Omega) = P(C)P(\Omega) = P(C)$

$A$ $B^C$ $A \n B^C = A \n (A \n B)^C = A \setminus (A \n B)$ $P(A \n B^C) = P(A) - P(A \n B) = P(A) - P(A)P(B) = P(A)(1-P(B)) = P(A)P(B^C)$

$A^C$ $B$ are statistically independent in the same vein.

$A^C$ $B^C$ $\ba P(A^C \n B^C) &= P((A\u B)^C) = 1 - P(A \u B) = 1 - P(A) - P(B) + P(A \n B) = 1 - P(A) - P(B) + P(A)P(B) \\&= (1 - P(A))(1 - P(B)) = P(A^C) P(B^C)\ea$

Def. Mutually Statistically Independent

$(\Omega, \sa, P)$ $\set{\sa_\lambda:\lambda\in\Lambda}$ $\sigs$ $\sa$
$\sa_\lambda$ $P(A_1 \n ... \n A_n) = \d\prod_{i=1}^n P(A_i)$ $\fa n$
$\lambda_1, ..., \lambda_n \in \Lambda$ $A_1 \in \sa_{\lambda_1}, ..., A_n \in \sa_{\lambda_n}$ .

Notes

$\nimplies$ Mutual Independence
$P(A \n B) = P(A \n C) = P(B \n C) \nimplies P(A\n B\n C) = P(A)P(B)P(C)$
$P(A\n B\n C) = P(A)P(B)P(C) \nimplies$ mutual independence

Union of 3 events (Inclusion-Exclusion Principles)

$P(A \u B \u C) = P(A) + P(B) + P(C) - P(A \n B) - P(A \n C) - P(B \n C) + P(A \n B \n C)$

Proof

$\ba P(A\u B\u C) &= P((A\u B) \u C) = P(A\u B) + P(C) - P((A\u B)\n C))\\ &= P(A) + P(B) - P(A\n B) + P(C) - P((A\n C)\u(B \n C)) \\&= P(A) + P(B) + P(C) - P(A\n B) - P(A\n C) - P(B\n C) + P((A\n C) \n (B\n C)) \\ &= P(A) + P(B) + P(C) - P(A\n B) - P(A\n C) - P(B\n C) + P(A\n B \n C)\ea$

Generalized to n events

$P(A_1 \u ... \u A_n) = \sumto n P(A_i) - \sum_{i<j} P(A_i \n A_j) + \sum_{i < j < k} P(A_i \n A_j \n A_k) - ... + (-1)^{n+1}P(A_1\n...\n A_n)$

Proof

$P(A\u B) = P(A) + P(B) - P(A\n B)$

I.H. Assume it's true for n

Consider

$\ba P(A_1 \u ... \u A_n \u A_{n+1}) &= P((A_1 \u ...\u A_n) \u A_{n+1})\\ &= P(A_1\u...\u A_n) + P(A_{n+1}) - P((A_1\u...\u A_n)\n A_{n+1})\\ &= \underbrace{P(A_1 \u...\u A_n)}_{(1)} + P(A_{n+1}) - \underbrace{P((A_1 \n A_{n+1}) \u...\u (A_n \n A_{n+1}))}_{(2)}\ea$

$\ba (1)\quad &P(A_1 \u ... \u A_n)\\ &= \sumto n P(A_i) - \sum_{i<j\le n}P(A_i \n A_j) + ... + (-1)^{n+1}P(A_1 \n ... \n A_n)\ea$

$\ba (2)\quad &P((A_1 \n A_{n+1}) \u ... \u (A_n \n A_{n+1}))\\ &= \sumto n P(A_i \n A_{n+1}) - \sum_{i<j\le n}P(A_i \n A_j \n A_{n+1}) + ... + (-1)^{n+1} P(A_1 \n ... \n A_n \n A_{n+1}) \ea$

Combining the above, we have

$P(A_1 \u ... \u A_{n+1}) = \sumto {n+1} P(A_i) - \sum_{i<j} P(A_i \n A_j) + ... + (-1)^{n+2} P(A_1 \n ... \n A_{n+1})$

Intersection of 3 events

$P(A\n B\n C) = P(A) + P(B) + P(C) - P(A\u B) - P(A\u C) - P(B\u C) + P(A\u B\u C)$

Proof

$\ba &LHS\\ &= 1 - P((A\n B\n C)^C) = 1 - P(A^C\u B^C \u C^C)\\ &= 1 - [P(A^C) + P(B^C) + P(C^C) - P(A^C\n B^C) - P(A^C \n C^C) - P(B^C \n C^C) + P(A^C \n B^C \n C^C)]\\ &= 1 - [3 - P(A) - P(B) - P(C) - (1-P(A\u B)) - (1 - P(A\u C)) - (1 - P(B\u C)) + (1 - P(A\u B\u C))]\\ &= RHS \ea$

Generalized to n events

$P(A_1 \n … \n A_n) = \sumto n P(A_i) - \sum_{i<j} P(A_i \u A_j) + … + (-1)^{n+1}P(A_1 \u … \u A_n)$

2. Random Variables and Stochastic Processes

Lecture 5

Motivation $\Omega$ $X(\omega)$ $a\le X(\omega \le b)$ $X(\omega) \in [a, b]$ $\Omega$ $\R^1$ — this is difficult. To navigate this, we use inverse images.

Def. Inverse Image

$X:\Omega\to\R^1$ $B \subset \R$ $X\inv B = \{\omega\in\Omega:X(\omega)\in B\}$
$\omega$ that get mapped into B.

Note to self $X(\omega) = b \iff X\inv \{b\} = \omega$

E.g. $\Omega = \set{1, 2, 3, 4, 5}$ $X(\omega) = \bc 0 &\omega = 1\\ 0.20 &\omega = 2\\ 0.30 &\omega = 3\\ 0.01 &\omega = 4\\ 0.20 &\omega = 5\ec$

Note that X is not 1-1

$X\inv B$

$\implies X\inv B = \Omega$

$\implies X\inv B = \set{1, 2, 4, 5}$

$\implies X\inv B = {1}$

$-\infty$ $\implies X\inv B = \null$

Property Inverse images preserve Boolean operations.

Proof for Unions

$\omega \in X\inv (B_1 \u B_2)$ $X(\omega) \in B_1 \u B_2$

$\implies \omega \in X\inv B_1$ $\omega \in X\inv B_2$

$\implies \omega \in X\inv B_1 \u X\inv B_2$

$X\inv(B_1 \u B_2) \subseteq X\inv B_1 \u X\inv B_2 \ \boxed1$

$\omega \in X\inv B_1 \u X\inv B_2$ ,

$\implies \omega \in X\inv B_1$ $\omega \in X\inv B_2$

$\implies X(\omega) \in B_1$ $X(\omega) \in B_2$

$\implies X(\omega) \in B_1 \u B_2$

$\implies \omega \in X\inv (B_1 \u B_2)$

$X\inv B_1 \u X\inv B_2 \subseteq X\inv (B_1 \u B_2) \ \boxed2$

$\boxed1$ $\boxed2$ $X\inv (B_1 \u B_2) = X\inv B_1 \u X\inv B_2$ since they are subsets of each other

Proof for Complements

$\omega \in X\inv B^C$ $X(\omega) \in B^C$

$\implies X(\omega) \in B$

$\implies w \notin X\inv B$

$\implies \omega \in (X\inv B)^C$

$X\inv B^C \subseteq (X\inv B)^C \ \boxed1$

$\omega \in (X\inv B)^C$

$\implies \omega \notin X\inv B$

$\implies X(\omega) \notin B$

$\implies X(\omega) \in B^C$

$\implies \omega \in X\inv B^C$

$(X\inv B)^C \subseteq X\inv B^C \ \boxed2$

$\boxed1$ $\boxed2$ $X\inv B^C = (X\inv B)^C$

Property $B_1\n B_2 = \null$ $X\inv B_1$ $X\inv B_2$ are also disjoint.

Proof

$A\n B = \null$ $X\inv A \n X\inv B = X\inv (A\n B) = X\inv \null = \null$

Def. Random Variable

$X:\Omega \to \R^1$ $B \in \sb^1$ $\R^1$ $X\inv B\in\sa$ .
$P(X(\omega) \in B) = P(X\inv B)$

Prop 2.1.1 (Marginal Probability Measure)

$P_X$ $\sb^1$ $P_X(B) = P(X\inv B)$

Proof

$P_X: \sb^1 \to [0,1]$

Normed $P_X(\R^1) = P(X\inv \R^1) = P(\Omega) = 1$
Countably additive $B_1, B_2, ...$ $\sb^1$
$P_X(\Uinf B_i) = P(X\inv \Uinf B_i) = P(\Uinf X\inv B_i) = \suminf P(X\inv B_i) = \suminf P_X(B_i)$

Note $(\R^1, \sb^1, P_X)$

Prop 2.1.2 (Determine whether X is a random variable)

$X\inv(a, b] \in \sa$ $a, b \in \R^1$ $X$ is a random variable.

Proof

$\sb_*^1 = \set{B\in \sb^1: X\inv B \in \sa}$

$\null \in \sb^1$ $X\inv \null \in \sa$ $\null \in \sb_*^1$
$B\in \sb_*^1$ $X\inv B \in \sa \implies (X\inv B)^C = X\inv B^C \in \sa$
$B^C \in \sb^1$ $X\inv B^C \in \sa$ $B^C \in \sb_*^1$
$B_1, B_2, ... \in \sb_*^1$ $X\inv B_1, X\inv B_2, ... \in \sa \implies \Uinf X\in B_i = X\inv \Uinf B_i \in \sa$
$\Uinf B_i \in \sb^1$ $X\inv \Uinf B_i \in \sa$ $\Uinf B_i \in \sb^1_*$

$\sb_*^1$ $\sig$ $\sb^1$ $\quad \boxed{1}$

$(a, b] \in \sb^1_* \fa a,b \in \R^1 \implies \sb^1 \subseteq \sb^1_*$ $\sb^1$ $\sig$ $(a, b]$ $\quad \boxed{2}$

$\boxed{1}$ $\boxed{2}$ $\sb_*^1 = \sb^1 \implies X\inv B \in \sa ,\fa B\in \sb^1 \implies$ X is a random variable.

Examples

$X(\omega) = c$ $(-\infty, b]$ $X\inv (-\infty, b] = \bc\Omega & c\le b \\ \null & c > b\ec \ \in \sa$
$X(\omega) = \omega$ $(-\infty, b]$ $X\inv (-\infty, b] = (-\infty, b] \in \sa$
$X(\omega) = \omega^2$ $(-\infty, b]$ $X\inv (-\infty, b] = \bc[-\sqrt b, \sqrt b] &b\ge0\\ \null &b<0\ec \ \in \sa$
$X(\omega) = \omega^n$ is a r.v. since for any b,
$X\inv (-\infty, b] = \bc \null &b<0\\ [-b^{1\o n}, b^{1 \o n}] &b>0 \ec \ \in \sb^1$ $n\in\Z$ is even, and
$X\inv (-\infty, b] = \bc (-\infty, -|b|^{1\o n}] &b<0\\ (-\infty, b^{1 \o n}] &b>0 \ec \ \in \sb^1$ $n\in\Z$ is odd
$X(\omega_1, ..., \omega_k) = \omega_i$ (projection on the ith coordinate) is a r.v. since for any b,
$X\inv (-\infty, b] = \set{(\omega_1, ..., \omega_k)\in\R^k:\omega_i \le b} = \R^1\times...\times(-\infty,b]\times...\times\R^1\in\sb^n$
- $\Omega = \R^k, \sa = \sb^k$ , so it must be a r.v.

Note $\sa = 2^\Omega$ $X:\Omega\to\R^1$ is a random variable.

Prop 2.1.3 (Sum & Prod of R.V.s are R.V.s)

$\Omega$ , then (1) W = X+Y and (2) W = XY are both random variables.

Proof of (1) W = X + Y

$\omega \in W\inv(-\infty, b] = \set{\omega:X(\omega) + Y(\omega) \le b}$

$c_n \in \Q$ $c_n \downarrow b$ $\ex q\in\Q$ $X(\omega) \le q$ $Y(\omega)\le c_n-q$ ,

$\omega\in $X\inv(-\infty,q]\N Y\inv(-\infty, c_n-q]$\in\sa$

$c_n$ $C_n = \U_{q\in\Q}$\set{\omega: X(\omega)\le q}\N\set{\omega: Y(\omega)\le c_n-q}$$ $\underbrace{W\inv(-\infty, b]}_{\text{set of }\omega} \subset C_n, \fa n$

$\Q$ $C_n$ $\sa$ $C_n\in\sa$

$C_n$ $\d\lim_{n\to\infty}C_n = \N_{n=1}^\infty C_n = W\inv(-\infty, b]\in\sa \implies W = X + Y$ is a r.v.

Proof of (2) W = XY

Suppose b = 0, then

$\ba W\inv (-\infty, 0] &= \set{\omega:X(\omega)\le 0, Y(\omega) \ge 0} \U \set{\omega:X(\omega) \ge 0, Y(\omega) \le 0} \\&= \Big(X\inv(-\infty, 0]\n Y\inv[0, \infty)\Big) \U \Big(X\inv[0,\infty)\n Y\inv(-\infty, 0]\Big) \in \sa \ea$

Suppose b > 0, then

$W\inv(-\infty,b] = W\inv(-\infty, 0] \U W\inv(0, b]$

$W\inv(-\infty, 0] \in \sa$ $W\inv(0, b]\in \sa$ .

$\ba W\inv(0, b] &= \set{\omega:X(\omega) > 0, Y(\omega) > 0, X(\omega)Y(\omega) \le b} \U \set{\omega: X(\omega) < 0, Y(\omega) < 0, X(\omega)Y(\omega) \le b}\\ &= \set{\omega \in \boxed1} \U \set{\omega \in \boxed4}\ea$

Since xy=b is symmetrical over the line y=-x, proving the argument for one of 1 & 4 will suffice.

$\omega \in$ $\boxed1$ $c_n \downarrow b$ $\ex q \in \Q \n (0, \infty)$ $\omega \in X\inv(0, q] \n Y\inv(0, c_n/q] \in \sa$

$C_n = \U_{q\ \in\ \Q\n(0, \infty)} X\inv(0, q] \n Y\inv (0, c_n/q] \in \sa$ $\Q \n (0, \infty)$ is countable.

$C_n \downarrow \fbox1 \implies \fbox1 \in \sa \implies W\inv(0, b]\in \sa$

$W\inv(-\infty,b] \in \sa$ , so W=XY is a r.v.

E.g. $p(X) = \sum_{i=0}^n a_i X^i$ is a r.v. if X is a r.v.

$Y(\omega) = c$ $a_i$ are r.v.'s.

$a_iX^i$ are r.v.'s

$\sum_{i=0}^n a_i X^i$ is a r.v.

Prop 2.1.4 (Sigma Algebra generated by X)

$\sa_X = X\inv\sb^1 = \set{X\inv B:B\in\sb^1}$ $\sig$ $\sa$ $\sig$ $\Omega$ generated by X.
$\sa_X = \sa(\set{X\inv(a, b]: a, b \in \R^1})$

Proof

$\null = X\inv \null \in \sa_X$
$A_1, A_2, ... \in \sa_X$ $\ex B_1, B_2, ... \in \sb^1$ $A_i = X\inv B_i$ .
$\Uinf A_i = \Uinf X\inv B_i = X\inv \Uinf B_i \in \sa_X$ $\Uinf B_i \in \sb^1$ )
$A\in\sa_X$ $\ex B \in \sb^1$ $A = X\inv V$ .
$A^C = (X\inv B)^C = X\inv B^C \in \sa_X$ $B^C \in \sb^1$ )

$\sa_X$ $\sig$ $\sa$

Def. Random Vector

Recall
random variable $X:\Omega \to \R^1$ $B \in \sb^1$ $X\inv B\in\sa$ .
$P(X(\omega) \in B) = P(X\inv B)$ $X\inv B = \set{\omega:X(\omega)\in B}$
random vector $\x: \Omega \to \R^k$ $B\in\sb^k$ $\x\inv B \in\sa$ .
$P(\x(\omega) \in B) = P(\x\inv B)$ $\x\inv B = \set{\omega:\x(\omega)\in B}$

Properties

$\mathbf{aX+bY}$ is a random vector
$\x$ $P_\x:\sb^k \to [0, 1]$ $P_\x(B) = P(\x\inv B)$
$\sig$ $\x$ $\sa_\x = \x\inv \sb^k = \set{\x\inv B: B\in \sb^k} = \sa(\set{\x\inv(\mathbf{a, b]:a, b}\in \R^k})$

Example (Pt. 1)

$\Omega = \set{1, 2, 3}, \sa = 2^\Omega$ $P$

$\x = \[X_1\\X_2\]:\Omega \to \R^2$ $\x(\omega) = \[X_1(\omega)\\X_2(\omega)\]$ $X_1, X_2$ $\ba X_1(1) = 0\\X_1(2) = 0\\X_1(3) = 1 \ea$ $\ba X_2(1) = 1\\X_2(2) = 0\\X_2(3) = 0 \ea$

$\ba X\inv\set{(0, 1)} = \set{1}\\ X\inv\set{(0, 0)} = \set2\\ X\inv\set{(1, 0)} = \set3 \ea \implies X\inv B = \bc\null & (0,1), (0,0), (1,0) \notin B\\ \set1 \text{ or } \set2 \text{ or } \set3 &\text { if only one of }(0,0),(0,1),(1,0) \in B\\ \set{1, 2} \text{ or } \set{1, 3} \text{ or } \set{2, 3} & \text { if only two of }(0,0),(0,1),(1,0) \in B \\ \Omega &(0,1), (0,0), (1,0) \in B\ec$

$P_{\x}(B)=\bc 0 & (0,1),(0,0),(1,0) \in B \\ 1 / 3 & \text { if only one of }(0,1),(0,0),(1,0) \in B \\ 2 / 3 & \text { if only two of }(0,1),(0,0),(1,0) \in B \\ 1 & (0,1),(0,0),(1,0) \notin B \\ \ec$

Example (Pt. 2)

$X_2$ $X_1, X_2$ $X_1(1) = 0\\X_1(2) = 0\\X_1(3) = 1$ $X_2(1) = 1\\X_2(2) = 1\\X_2(3) = 0$ $P_\x$ ?

Only 2 possible outputs now: (0,1) and (1, 0)

$\ba &X\inv\set{(0, 1)} = \set{1, 2}\\ &X\inv\set{(1, 0)} = \set3 \ea \implies X\inv B = \bc\null & (0,1), (1,0) \notin B\\ \set{1, 2} &(0, 1)\in B, (1, 0)\notin B\\ \set3 &(0, 1)\notin B,(1, 0)\in B\\ \Omega &(0, 1), (1, 0) \in B\ec$

$B \in \sb^2, P_\x (B) = \bc0 & (0,1), (1,0) \notin B\\ 2/3 &(0, 1)\in B, (1, 0)\notin B\\ 1/3 &(0, 1)\notin B,(1, 0)\in B\\ 1 &(0, 1), (1, 0) \in B \ec$

Example (Pt. 3)

$P(\set{1}) = \f12, P(\set{2}) = \f13, P(\set{3}) = \f16$ $P_\x$ ?

$P_\x (B) = \bc0 & (0,1), (1,0) \notin B\\ 5/6 &(0, 1)\in B, (1, 0)\notin B\\ 1/6 &(0, 1)\notin B,(1, 0)\in B\\ 1 &o/w\ec$

Prop 2.1.5 (Cartesian Prod of Borel Sets is a Borel Set)

$B_1, ..., B_k \in \sb ^1$ $B_1 \times ...\times B_k = \set{(x_1, ..., x_k)^T | x_i \in B_i, i = 1, ..., k} \in \sb^k$ $\sb^k$ $\sig$ $\R^k$ containing all such sets

Proof

$\R^1\times...\times B_i \times...\times\R^1$ that only restrict the ith coord.

$\set{\R^1\times...\times B_i \times...\times\R^1 | B_i \in \sb^1}$ $\sig$ $\sb^k$

Sub-proof $\sb_y = \set{B\times\R^1\times...\times\R^1 : B\in\sb^1}$

$(-\infty, b]\times\R^1\times...\times\R^1 \in \sb^k \fa b \in \R^1 \implies B\times\R^1\times...\times\R^1 \in \sb^k$

$\null \times \R^1 \times ... \times \R^1 = \null \in \sb^k$
$B_i \times \R^1 \times...\times\R^1\in \sb_y$ $\Uinf B_i\times\R^1\times...\times\R^1 = (\Uinf B_i)\times\R^1\times...\R^1\in \sb_y$ $\Uinf B_i \in \sb^1$
$B \times \R^1 \times ... \times \R^1 \in \sb_y$ $(B\times\R^1\times...\times\R^1)^C = B^C\times\R^1\times...\times\R^1\in\sb_y$ $B^C\in\sb^1$ .

$B_1\times ... \times B_k = \N_{i=1}^k (\R^1\times...\times B_i \times...\times\R^1)\in\sb^k$

$(\mathbf a,\mathbf b] = (a_1, b_1]\times…\times (a_k, b_k]$ $\nexists$ $\sig$ $\R^k$ $\sb^k$ .

Prop 2.1.6 (A Vector of R.V.s is a Random Vector)

$X_i:\Omega \to \R^1$ $i = 1, ..., k$ $\x = (X_1, ..., X_k)^T:\Omega \to \R^k$ is a random vector.

Proof

$B_1, ..., B_k \in \sb^1$ $B_1 \times ... \times B_k \in \sb^k$ . Then we have

$\ba\x\inv(B_1\times...\times B_k) &= \set{\omega: \x(\omega)\in B_1\times...\times B_k}\\ &=\set{\omega : X_i(\omega)\in B_i\ \text{for i = 1, ..., k}}\\ &=\N_{i=1}^k X_i^{-1} B_i \in \sa\ea$

$\x\inv(\mathbf a, \mathbf b] \in \sa \fa \mathbf a, \mathbf b \in \R^k \implies \x\inv B\in \sa \fa B\in\sb^k \implies \x$ is a random vector.

Lecture 6

Def. K-cells

$(\mathbf{a, b}] = \d\bigtimes_{i=1}^k(a_i, b_i]$ $\d\bigtimes_{i=1}^k(-\infty, b_i]$

K-cells are the basic sets we want to assign probabilities to (using random vectors)

For k = 2, (a, b] =

Def. Cumulative Distribution Function (CDF)

$F_\x: \R^k\to[0,1]$ $\x\in\R^k$ $F_\x(x_1, ..., x_k) = P_\x\big((-\infty, x_1]\times...\times(-\infty, x_k]\big) = P_\x\big((-\infty, \x]\big)$

Def. Difference Operator

$g:\R^k\to\R^1$ $\Delta_{a, b}^{(i)}\ g:\R^{k-1}\to\R^1$ $(\Delta_{a,b}^{(i)}\ g)(x_1, ..., x_{i-1}, x_{i+1}, ..., x_k) = g(x_1, ..., x_{i-1}, b, x_{i+1}, ..., x_k) - g(x_1, ..., x_{i-1}, a, x_{i+1}, ..., x_k)$

Prop 2.2.1 (Properties of Distribution Functions)

$F_\x:\R^k\to[0, 1]$ satisfies
$a_i \le b_i$ $i = 1, ..., k$ $P_\x((\mathbf{a, b}]) = \Delta_{a_1, b_1}^{(1)}\Delta_{a_2, b_2}^{(2)}...\Delta_{a_k, b_k}^{(k)}F_\x$
$As\ x_i \downarrow -\infty, F_\x(x_1, ..., x_k)\downarrow 0$
$As\ x_i \uparrow \infty, F_\x(x_1, ..., x_k)\uparrow 1$
$F_\x$ is right continuous
$\delta_i \downarrow 0 \fa i$ $F_\x(x_1 + \delta_1, ..., x_k + \delta_k) \to F_\x(x_1, ..., x_k)$

Proof for (1)

Proof for (2)

Proof for (3)

Thm 2.2.1 (Extension Theorem)

$F:\R^k\to[0,1]$ $\ex$ $P$ $\sb^k$ $F$ $P$

Note $F$ $(\R^k, \sb^k, P)$ $\Omega = \R^k$ $\x(\omega) = \omega$

$P_\x$ $F_\x$

Def. Marginal Distributions

Def. Discrete Probability Models

Prop 2.3.1 (Countably Many Points with Positive Prob)

Prop 2.3.2 (Prob Measure Defined by p)

Def. Multinomial Distribution

Def. Multivariate Hypergeometric Distribution

Lecture 7

Def. Continuous Probability Models

Def. Absolutely Continuous Probability Models

Def. Probability Density Functions (PDF)

Prop 2.4.1 (Properties of A.C. Models)

$f(\b x) \ge 0$ with probability 1
$\d\int_{\R^k} f(\b x) dx = 1$
$\d F(\b x) = F(x_1, ..., x_k) = \int_{-\infty}^{x_k} ... \int_{-\infty}^{x_1} f(z_1, ..., z_k)dz_1...dz_k$
$\d f(\b x) = f(x_1, ..., x_k) = {\del^k F(x_1, ..., x_k)\o\del x_1...\del x_k}$

Prop 2.4.2 (Properties of PDFs)

$f:(\R^k, \sb^k)\to(\R^1, \sb^1)$ $(\R^k, \sb^k, P)$ if

$f(\b x) \ge 0 \fa x$
$\d\int_{\R^k} f(\b x) dx = 1$

Def. Multivariate Normal Distribution

Lecture 8 & 9

$\x \in \R^k$ $\mathbf{Y} = T(\x) \in \R^1$

Discrete case

$\x$ $p_\x$ $p_\mathbf Y (\mathbf y) = P_\mathbf Y$\set{\mathbf y}$ = P_\x(T\inv\{\mathbf y\}) = \d\sum_{\b x\in T\inv\{\mathbf{y}\}}p_\x(\b x)$

Def. Projections (& their Prob Functions)

$k \geq 2$ $\left(y_{1}, y_{2}\right)=T\left(x_{1}, \ldots, x_{k}\right)=\left(x_{1}, x_{2}\right)$

Prob Function Derivation:

To find the probability functions of projections, take the joint probability function, and sum out unwanted variables.

$T^{-1}\{\mathbf{y}\}=T^{-1}\left\{\left(y_{1}, y_{2}\right)'\right\}=\left\{\left(x_{1}, \ldots, x_{k}\right): x_{1}=y_{1}, x_{2}=y_{2}\right\}$

$\ba p_{\mathbf{y}}(\mathbf Y)= p_{\mathbf{Y}}\left(y_{1}, y_{2}\right) =P_{\x}\left(T^{-1}\{\mathbf{y}\}\right)&=\sum_{\b x \in T^{-1}\{\mathbf{y}\}} p_{\x}(\b x) \\ &=\sum_{\left(x_{1}, \ldots, x_{k}\right):x_{1}=y_{1}, x_{2}=y_{2}} p_{\x}\left(x_{1}, \ldots, x_{k}\right) \\ &=\sum_{\left(x_{3}, \ldots, x_{k}\right) \in R^{k-2}} p_{\x}\left(y_{1}, y_{2}, x_{3}, \ldots, x_{k}\right) \expl{fix } x_1, x_2 \text{ to } y_1, y_2 \ea$

$y=T\left(x_{1}, \ldots, x_{k}\right)=x_{2}$

Prob Function Derivation:

$T^{-1}\{\mathbf y\} =T^{-1}\{y\}=\left\{\left(x_{1}, \ldots, x_{k}\right): x_{2}=y\right\}$

$\begin{aligned} p_{\mathbf{y}}(\mathbf Y) = p_{Y}(y) &=\sum_{\left(x_{1}, \ldots, x_{k}\right): x_{2}=y} p_{\x}\left(x_{1}, \ldots, x_{k}\right) \\ &=\sum_{\left(x_{1}, x_{3}, \ldots, x_{k}\right) \in R^{k-1}} p_{\x}\left(x_{1}, y, x_{3}, \ldots, x_{k}\right) \end{aligned}$

Marginal of a Multinomial Random Vector

$\x=\left(X_{1}, \ldots, X_{k}\right)' \sim$ $\left(n, p_{1}, \ldots, p_{k}\right)$ $p_{\x}(\mathbf{a})=\binom n {a_{1}\ \ldots\ a_{k}} p_{1}^{a_{1}} \cdots p_{k}^{a_{k}}$
$\mathbf{a} \in R^{k}$ $a_{i} \in\{0, \ldots, n\}, \text { and } a_{1}+\cdots+a_{k}=n$
$k \geq 2$ $\left(y_{1}, y_{2}\right)=T\left(x_{1}, \ldots, x_{k}\right)=\left(x_{1}, x_{2}\right)$ $\mathbf{Y}=\left(X_{1}, X_{2}\right)'$

$y_{1}, y_{2}, a_{3}, \ldots, a_{k} \in\{0, \ldots, n\}$ $y_{1}+y_{2}+a_{3}+\cdots+a_{k}=n \iff$

$a_{3}, \ldots, a_{k} \in\left\{0, \ldots, n-y_{1}-y_{2}\right\}$ $a_{3}+\cdots+a_{k}=n-y_{1}-y_{2} \quad(* )$

$\ba \text{so}\ \ &p_{Y}\left(y_{1}, y_{2}\right)\\ &=\sum_{\left(a_3, \ldots,a_{k}\right) \text { sat. }( *)}\binom n {y_1\ y_{2}\ a_{3} \ldots\ a_{k}} p_{1}^{y_{1}} p_{2}^{y_{2}} p_{3}^{a_{3}} \cdots p_{k}^{a_{k}}\\ &=\frac{n !}{y_1 ! y_{2} !} p_{1}^{y_{1}} p_{2}^{y_{2}} \sum_{\left(a_{3}, \ldots, a_{k}\right) \text { sat. }(*)} \frac1{a_{3} ! \cdots a_{k} !} p_{3}^{a_{3}} \cdots p_{k}^{a_{k}} \expl{took out terms where } i = 1,2 \\ &=\frac{n !}{y_1 ! y_{2} !\left(n-y_{1}-y_{2}\right) !} p_{1}^{y_{1}} p_{2}^{y_{2}} \sum_{\left(a_{3}, \ldots, a_{k}\right) \text { sat. }(*)} \frac{\left(n-y_{1}-y_{2}\right) !}{a_{3} ! \cdots a_{k} !} p_{3}^{a_{3}} \cdots p_{k}^{a_{k}}\expl {multiplied prev by } \f{(n-y_{1}-y_{2})!}{(n-y_{1}-y_{2})!}\\ &=\binom n {y_{1}\ \ y_{2}\ \ n-y_{1}-y_{2}} p_{1}^{y_{1}} p_{2}^{y_{2}}\left(1-p_{1}-p_{2}\right)^{n-y_{1}-y_{2}} \expl{multiply by } \left(1-p_{1}-p_{2}\right)^{n-y_{1}-y_{2}}\\ &\underbrace{\sum_{\left(a_{3}, \ldots, a_{k}\right) \text{ sat. } \left({ }^{*}\right)}\left(\begin{array}{c}n-y_{1}-y_{2} \\ a_{3} \ldots a_{k}\end{array}\right)\left(\frac{p_{3}}{1-p_{1}-p_{2}}\right)^{a_{3}} \cdots\left(\frac{p_{k}}{1-p_{1}-p_{2}}\right)^{a_{k}}}_{\text{sum of all multinomial}$n-y_1-y_2, {p_3\o1-p_1-p_2}, ..., {p_k\o1-p_1-p_2}$ \text{ probabilities, so } = 1} \expl{divide by } (1-p_{1}-p_{2})^{a_3 + ... + a_k}\\ &=\binom n {y_{1}\ \ y_{2}\ \ n-y_{1}-y_{2}} p_{1}^{y_{1}} p_{2}^{y_{2}}\left(1-p_{1}-p_{2}\right)^{n-y_{1}-y_{2}}\\ \ea$

$\left(X_{1}, X_{2}\right) \sim$ $\left(n, p_{1}, p_{2}, 1-p_{1}-p_{2}\right)$

Binomial(n, p) = Multinomial(n, p, 1-p)

$\x=\left(X_{1}, \ldots, X_{k}\right)' \sim$ $\left(n, p_{1}, \ldots, p_{k}\right)$ $X_{i} \sim$ $\left(n, p_{i}\right)=$ $\left(n, p_{i}, 1-p_{i}\right)$ Note $n$ $k$ $l$ $l+1$ mutually disjoint categories

$\ba P_{X_1}(x_1) &= \d\sum_{x_2 = 0}^{n - x_1}p_{(x_1, x_2, n - x_1 - x_2)}(x_1, x_2, n - x_1 - x_2)\\ &= \d\sum_{x_2 = 0}^{n - x_1}\binom n{x_1\ x_2\ n-x_2-x_2}p_1^{x_1}p_x^{x_2}(1-p_1-p_2)^{n-x_2-x_2}\\ &= \d\sum_{x_2 = 0}^{n - x_1}{n!\o x_1!x_2!(n-x_1-x_2)!}p_1^{x_1}p_x^{x_2}(1-p_1-p_2)^{n-x_2-x_2} \expl {now multiply by }\f{(n-x_1)!}{(n-x_1)!}\\ &= {n!\o x_1!(n-x_1)!}p_1^{x_1}(1-p_1)^{n-x_1}\underbrace{\d\sum_{x_2 = 0}^{n - x_1}\f{(n-x_1)!}{x_2!(x-x_1-x_2)!}({p_2\o1-p_1})^{x_2}(1-{p_2\o1-p_1})^{n-x_1-x_2}}_{\text{sum of all binomial}$n-x_1, {p_2\o1-p_1}$\text{ probabilities, so = 1}}\\ &= {n!\o x_1!(n-x_1)!}p_1^{x_1}(1-p_1)^{n-x_1} \ea$

$X_1 \sim \text{binomial}(n, p_1)$

Sum of sub-Multinomial Random Vector ~ Binomial

$Y=X_{1}+\cdots+X_l$ $l \leq k$ $\left(X_{1}, \ldots, X_{k}\right)' \sim \text { multinomial}\left(n, p_{1}, \ldots, p_{k}\right)$
Note $T$ $T^{-1}\{\mathbf{y}\} \neq \phi$ $p_{\mathbf{Y}}(\mathbf{y})=P_{\x}\left(T^{-1}\{\mathbf{y}\}\right)=p_{\x}\left(T^{-1}\{\mathbf{y}\}\right)$

$Y = X_1 + ... + X_l$ $l$ categories.

$p_1 + ... + p_l$ .

$Y\sim\text{binomial}(n, p_1 + ... + p_l)$

Def. Indicator Function

$A \subset \Omega$ $I_{A}: \Omega \rightarrow R^{1}$ $I_{A}(\omega)= \begin{cases}1 & \text { if } \omega \in A \\ 0 & \text { if } \omega \in A^{c}\end{cases}$

Indicator Variable ~ Bernoulli(P(A))

$(\Omega, \mathcal{A}, P)$ $A \in \mathcal{A}$ $Y=I_{A}$ $Y \sim \operatorname{Bernoulli}(P(A))$

$A \in \sa \implies I_A:\Omega \to \R^1$ $\fa B \in \sb^1, I\inv_A B = \{\omega:I_A(\omega)\in B\} = \bc\null &0, 1 \notin B\\ A &1 \in B, 0 \notin B\\ A^C &0\in B, 1 \notin B\\ \Omega &0, 1\in B\ec$ $\in \sa$

$B \in \sb^1$ $I_A\inv B\in\sa$ $Y = I_A$ is a r.v.

$P_Y(1) = P(I\inv_A\{1\}) = P(\{\omega: I_A(\omega) = 1\}) = P(A) \implies Y\sim\text{Bern}(P(A))$

Transformation Determines Distribution Type

$\mathbf{Y}=T(\x)$ $\x$ is distributed.

E.g. $T(\b x)=\mathbf{c} \in \R^{l}$ $\b x$ $p_{\mathbf{Y}}(\mathbf{y})=P_{\x}\left(T^{-1}\{\mathbf{y}\}\right)= \begin{cases}P_{\x}(\R^k)=1 & \text { if } \mathbf{y}=\mathbf{c} \\ P_{\x}(\null)=0 & \text { if } \mathbf{y} \neq \mathbf{c}\end{cases}$

$\mathbf{Y}$ $\mathbf{c}$

E.g. $X \sim N(0,1)$ $P(X \leq 0)=P(X>0)=1 / 2$

$Y=T(X)=I_{(-\infty, 0]}(X)= \begin{cases}1 & \text { if } X \leq 0 \\ 0 & \text { if } X>0\end{cases}\implies$ $p_\mathbf Y(1)=P(X \leq 0)=1 / 2\\ p_\mathbf Y(0)=P(X>0)=1 / 2$ $\implies$ $Y \sim$ $(1 / 2)$

Absolutely continuous case

$\x \in \R^k$ $f_\x$ $\b Y = T(\x) \in \R^l$ $l \le k$ .

$\b Y$ $f_\b Y$ which we want to determine.

Cdf Method

$T$ $F_\x$ :

$\d f_\b y(y_1) = {\del^k F_\b Y(y_1, ..., y_l)\o \del y_1...\del y_l} = {\del^k P_\x(T\inv\{(-\infty, y_1]\times...\times(-\infty, y_l\})\o \del y_1...\del y_l}$

E.g. $F:\R^2 \to [0, 1]$ $F\left(x_{1}, x_{2}\right)= \begin{cases}0 & x_{1}<0 \text { or } x_{2}<0 \\ 1-e^{-x_{1}}-e^{-x_{2}}+e^{-x_{1}-x_{2}} & x_{1} \geq 0 \text { and } x_{2} \geq 0\end{cases}$

It was proved (in a lec 6 exercise) that this is a cdf (using thm 2.2.1),

$f\left(x_{1}, x_{2}\right)=\frac{\partial^{2} F\left(x_{1}, x_{2}\right)}{\partial x_{1} \partial x_{2}}= \begin{cases}0 & x_{1}<0 \text { or } x_{2}<0 \\ e^{-x_{1}-x_{2}} & x_{1} \geq 0 \text { and } x_{2} \geq 0\end{cases}$

$f$ $\mathrm{pdf}$ :

$f\left(x_{1}, x_{2}\right) \geq 0$ $\left(x_{1}, x_{2}\right)$

(ii) f is normed:

$\begin{aligned}&\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f\left(x_{1}, x_{2}\right) d x_{1} d x_{2}=\int_{0}^{\infty} \int_{0}^{\infty} e^{-x_{1}-x_{2}} d x_{1} d x_2 \\ =& \int_{0}^{\infty} e^{-x_{1}} d x_{1} \int_{0}^{\infty} e^{-x_{2}} d x_{2}=\left(-\left.e^{-x_{1}}\right|_{0} ^{\infty}\right)\left(-\left.e^{-x_{2}}\right|_{0} ^{\infty}\right)=1 \end{aligned}$

$\d F\left(x_{1}, x_{2}\right)=\int_{-\infty}^{x_{1}} \int_{-\infty}^{x_{2}} f\left(z_{1}, z_{2}\right) d z_{1} d z_{2}$

$Y=T\left(X_{1}, X_{2}\right)=X_{1}$ $F_{X_{1}}\left(x_{1}\right)=F\left(x_{1}, \infty\right)= \begin{cases}0 & x_{1}<0 \\ 1-e^{-x_{1}} & x_{1} \geq 0\end{cases}$

$f_{X_{1}}\left(x_{1}\right)=\frac{\partial F_{X_{1}}\left(x_{1}\right)}{\partial x_{1}}= \begin{cases}0 & x_{1}<0 \\ e^{-x_{1}} & x_{1} \geq 0\end{cases}$ $f_{X_{2}}\left(x_{2}\right)=\frac{\partial F_{X_{2}}\left(x_{2}\right)}{\partial x_{2}}= \begin{cases}0 & x_{2}<0 \\ e^{-x_{2}} & x_{2} \geq 0\end{cases}$

$X_{1}$ $X_{2}$ have exponential(1) distributions

E.g. $y = T(x_1, x_2) = x_1 + x_2$ $X_1, X_2$ $f(x_1, x_2) = \bc2 &0<x_1<x_2<1\\0&o/w\ec$

$\begin{aligned}F_y(y) &= P_Y((-\infty, y]) \\&= P_{(X_1, X_2)}(\{(x_1, x_2):x_1 + x_2 \le y\})\\ &= \begin{cases}0 & y<0 \\ \int_{0}^{y / 2} \int_{x_{1}}^{y-x_{1}} 2 d x_{2} d x_{1}=y^{2} / 2 & 0 \leq y \leq 1 \\ 1-\int_{y / 2}^{1} \int_{y-x_{2}}^{x_{2}} 2 d x_{1} d x_{2}=2 y-y^{2} / 2-1 & 1 \leq y \leq 2 \\ 1 & 2<y\end{cases} \\ f_{Y}(y) &= \begin{cases}0 & y \leq 0 \text { or } y \geq 2 \\ y & 0<y<1 \\ 2-y & 1 \leq y<2\end{cases} \end{aligned}$

Change of Variable Method

$T: R^{k} \rightarrow R^{k}$ is 1-1 and smooth (i.e. all 1st order partial derivatives exist and are continuous),

$T(\x)=\left(\begin{array}{c} T_{1}(\b x) \\ \vdots \\ T_{k}(\b x) \end{array}\right)$ Jacobian $J_{T}(\b x)=\left|\det\left(\begin{array}{ccc} \frac{\partial T_{1}(\x)}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\x)}{\partial x_{k}} \\ \vdots & & \vdots \\ \frac{\partial T_{k}(\x)}{\partial x_{1}} & \ldots & \frac{\partial T_{k}(\x)}{\partial x_{k}} \end{array}\right)\right|^{-1}$

$J_{T}(\b x)=\d \lim _{\delta \downarrow 0} \frac{\operatorname{vol}\left(B_{\delta}(\x)\right)}{\operatorname{vol}\left(T B_{\delta}(\x)\right)}$ $J_{T}^{-1}(\b x)$ $T$ $\mathrm{x}$ ,

$J_{T}(\b x)<1$ $T$ $\b x$ $J_{T}(\b x)>1$ $T$ $\b x=T^{-1}(\mathbf{y})$

$\b Y = T(\b X)$ $\delta$ ,

$\begin{aligned} f_{\mathbf{Y}}(\mathbf{y}) & \approx \frac{P_{\mathbf{Y}}\left(T B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}{\operatorname{vol}\left(T B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}=\frac{P_{\x}\left(B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}{\operatorname{vol}\left(B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right.} \frac{\operatorname{vol}\left(B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}{\operatorname{vol}\left(T B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)} \approx f_{\x}\left(T^{-1}(\mathbf{y})\right) J_{T}\left(T^{-1}(\mathbf{y})\right) \end{aligned}$

This intuitive argument can be made rigorous to prove the following.

Proposition II.5.1 (Change of Variable)

$T: \R^{k} \rightarrow \R^{k}$ $\mathbf{Y}=T(\x)$ $\x$ $f_{\mathrm{X}}$
$\mathrm{Y}$ $f_{\mathbf{Y}}(\mathbf{y})=f_{\x}\left(T^{-1}(\mathbf{y})\right) J_{T}\left(T^{-1}(\mathbf{y})\right)$

E.g. $f(x) = \f12$ $0<x<2$ $y = T(x) = x^2$ .

$T\inv (y) = y^{1/2}$ $J_T(x) = |\det(2x)|\inv = \f1{2x}$ $x \in (0, 2)$

$J_T(x) = 1$ , we see that T contracts lengths on (0, 1/2) and expands lengths on (1/2, 2)

$\begin{aligned}f_{Y}(y) &=f\left(T^{-1}(y)\right) J_{T}\left(T^{-1}(y)\right) \\ &=f\left(y^{1 / 2}\right) \frac{1}{2 y^{1 / 2}} \\ &= \begin{cases}0 & y \leq 0 \text { or } y \geq 4 \\ 1 / 4 y^{1 / 2} & 0<y<4\end{cases} \end{aligned}$

E.g. $\int_{-\infty}^{\infty} \varphi(x) d x=1$ $\varphi$ $N(0,1)$ pdf.

$\begin{aligned}\left(\int_{-\infty}^{\infty} \varphi(x) d x\right)^{2} &=\int_{-\infty}^{\infty} \varphi(x) d x \int_{-\infty}^{\infty} \varphi(y) d y =\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \frac{1}{2 \pi} \exp \left(-\frac{x^{2}+y^{2}}{2}\right) d x d y \end{aligned}$

$T(x, y)=(r, \theta)$ $r \in(0, \infty), \theta \in[0,2 \pi)$

$(x, y)=T^{-1}(r, \theta)=(r \cos \theta, r \sin \theta)$

$\begin{aligned}J_{T\inv}(r, \theta) &=\left|\det\left(\begin{array}{cc} \frac{\partial r \cos \theta}{\partial r} & \frac{\partial r \cos \theta}{\partial \theta} \\ \frac{\partial r \sin \theta}{\partial r} & \frac{\partial r \sin \theta}{\partial \theta} \end{array}\right)\right|^{-1} \\ &=\left|\det\left(\begin{array}{cc} \cos \theta & -r \sin \theta \\ \sin \theta & r \cos \theta \end{array}\right)\right|^{-1} \\ &=\left|r\left(\cos ^{2} \theta+\sin ^{2} \theta\right)\right|^{-1}=1 / r \end{aligned}$

$J_{T}(\b x) = J_{T^{-1}}^{-1}(T(\b x)) = \d{1 \o J_{T^{-1}}(T(\b x))} = r$ $r^{2}=x^{2}+y^{2}$ ,

$\begin{aligned}&\left(\int_{-\infty}^{\infty} \varphi(x) d x\right)^{2}=\int_{0}^{\infty} \int_{0}^{2 \pi} \frac{r}{2 \pi} \exp \left(-r^{2} / 2\right) d \theta d r = \int_{0}^{\infty} r \exp \left(-r^{2} / 2\right) d r=-\left.\exp \left(-r^{2} / 2\right)\right|_{0} ^{\infty}=1 \end{aligned}$

$\int_{-\infty}^{\infty} \varphi(x) d x=1$

Def. Affine Transformation

(Affine transformation are linear transformations plus a constant.)

$T: R^{k} \rightarrow R^{k}$
$T(\b x)=A \b x+\mathbf{b}=\left(\begin{array}{c} a_{11} x_{1}+\cdots+a_{1 k} x_{k}+b_{1} \\ a_{21} x_{1}+\cdots+a_{2 k} x_{k}+b_{2} \\ \vdots \\ a_{k 1} x_{1}+\cdots+a_{k k} x_{k}+b_{k} \end{array}\right)$ $\mathbf{b} \in R^{k}, A \in R^{k \times k}$

$J_{T}(\b x)=\left|\det\left(\begin{array}{ccc} \frac{\partial T_{1}(\b x)}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\b x)}{\partial x_{k}} \\ \vdots & & \vdots \\ \frac{\partial T_{k}(\b x)}{\partial x_{1}} & \ldots & \frac{\partial T_{k}(\b x)}{\partial x_{k}} \end{array}\right)\right|^{-1}=|\det A\ |^{-1}$

$T\left(\mathrm{x}_{1}\right)=T\left(\mathrm{x}_{2}\right)$ $A\left(\mathrm{x}_{1}-\mathrm{x}_{2}\right)=\mathbf{0}$ $T$ $A$ $T^{-1}(\mathbf{y})=A^{-1}(\mathbf{y}-\mathbf{b})=\b x$

$\b Y = A\b X + \b b$ $f_\b Y(\b y) = f_\b X(T\inv (\b y))J_T(T\inv (\b y)) = f_\b X(A\inv(\b y-\b b))|\det A\ |\inv$

Multivariate Normal

$\mathbf{Z} \sim N_{k}(\mathbf{0}, I)$ $f_{\mathbf{Z}}(\mathbf{z})=(2 \pi)^{-k / 2} \exp \left(-\mathbf{z}' \mathbf{z} / 2\right)$ $\mathbf{z} \in R^{k}$

$\x=A \mathbf{Z}+\mu \iff \b Z = A\inv(\x - \mu)$ $A \in R^{k \times k}$ $\mu \in R^{k}$ $\x$ is an affine transformation, we know it has an a.c. distribution with density:

$\begin{aligned}f_{\x}(\b x) &=f_\b Z\left(A^{-1}(\b x-\mu)\right)|\det A|^{-1} \\ &=(2 \pi)^{-k / 2} \exp \left(-\left(A^{-1}(\b x-\mu)\right)' A^{-1}(\b x-\mu) / 2\right)|\det A|^{-1} \expl {plug in Z} = A\inv(X-\mu)\\ &=(2 \pi)^{-k / 2}|\det A|^{-1} \exp \left(-(\b x-\mu)'\left(A^{-1}\right)' A^{-1}(\b x-\mu) / 2\right)\expl {reorder} \\ &=(2 \pi)^{-k / 2}\left|\det A \det A'\right|^{-1 / 2} \exp \left(-(\b x-\mu)'\left(A A'\right)^{-1}(\b x-\mu) / 2\right)\expl{det}(A)= \det(A^T) \\ &=(2 \pi)^{-k / 2}\left|\det A A'\right|^{-1 / 2} \exp \left(-(\b x-\mu)'\left(A A'\right)^{-1}(\b x-\mu) / 2\right)\expl{det}(AB) = \det(A)\det(B) \\ &=(2 \pi)^{-k / 2}(\det \Sigma)^{-1 / 2} \exp \left(-(\b x-\mu)' \Sigma^{-1}(\b x-\mu) / 2\right) \end{aligned}$

$\Sigma=A A'\in R^{k \times k}$

$\x$ $\x \sim N_k(\mu, \Sigma)$

Note $\Sigma$ is symmetric, invertible, and positive definite (see note from lecture 2)

Ex. $\x \sim N_k(\mu, \Sigma)$ $\b Y = A\x + \b b$ $A\in\R^{k\times k}$ $\mu \in \R^{k}$ .
$\mathbf{Y} \sim N_{k}\left(A \mu+\mathbf{b}, A \Sigma A'\right)$ .

$\ba f_\b Y(y) &= f_\b X(A\inv(\b y-\b b))|\det A\ |\inv\\ &=(2 \pi)^{-k / 2}(\det \Sigma)^{-1 / 2} \exp \left(-(A\inv(\b y-\b b)-\mu)' \Sigma^{-1}(A\inv(\b y-\b b)-\mu) / 2\right)(\det AA')^{-1/2}\\ &\ \gr\downarrow\text{merge determinants, take out }A \inv\ \Leftarrow(AB)' = B'A', \\ &=(2 \pi)^{-k / 2}(\det A \Sigma A')^{-1 / 2} \exp \left(-(A\inv[(\b y-\b b)-A\mu])' \Sigma^{-1}(A\inv[(\b y-\b b)-A\mu)] / 2\right)\\ &=(2 \pi)^{-k / 2}(\det A \Sigma A')^{-1 / 2} \exp \left(-[\b y-\b b-A\mu]'\underbrace{(A\inv)' \Sigma^{-1}A\inv}[\b y-\b b-A\mu)] / 2\right)\\ &\ \gr\downarrow\text{}(AB)\inv = B\inv A\inv\\ &=(2 \pi)^{-k / 2}(\det A \Sigma A')^{-1 / 2} \exp \left(-[\b y- (A\mu+ \b b)]'(A\Sigma A')\inv[\b y- (A\mu+ \b b)] / 2\right) \ea$

Ex. $\mathrm{X} \sim N_{k}(\mu, \Sigma)$ $\Sigma=C C'$ $C\in\R^{k\times k}$ $\mathbf{Z}=C^{-1}(\x-\mu) \sim N_{k}(\mathbf{0}, I)$ .

$Z = C\inv X + (-C\inv \mu) \iff X = CZ + \mu$

$\ba f_Z(z) &= f_X(Cz + \mu)|\det C\inv|\inv\\ &=(2 \pi)^{-k / 2}(\det \Sigma)^{-1 / 2} \exp \left(-(CZ+\mu-\mu)' \Sigma^{-1}(CZ+\mu-\mu) / 2\right)(\det C\inv(C\inv)')^{-1/2}\\ &=(2 \pi)^{-k / 2}(\det CC')^{-1 / 2}(\det (C'C)\inv)^{-1/2} \exp \left(-(CZ)' (CC')^{-1}(CZ) / 2\right)\\ &=(2 \pi)^{-k / 2}(\det I)^{-1/2} \exp \left(-Z'C'C'\inv C\inv CZ/ 2\right)\\ &=(2 \pi)^{-k / 2}\exp \left(-Z'Z/ 2\right) \ea$

Ex. $\mu=\left(\mu_{1}, \mu_{2}\right)^{\prime}$ $\Sigma=\left(\begin{array}{ll} \sigma_{11} & \sigma_{12} \\ \sigma_{12} & \sigma_{22} \end{array}\right) \expl {by symmetry of }\Sigma, \sigma_{21}=\sigma_{12}$
$f_{\x}(\x)=(2 \pi)^{-k / 2}(\det \Sigma)^{-1 / 2} \exp \left(-(\x-\mu)' \Sigma^{-1}(\x-\mu) / 2\right)$ $x_1$ $x_2$

$\Sigma\inv = \f1{\sigma_{11}\sigma_{22}-\sigma_{12}^2}\[\sigma_{22} &-\sigma_{12}\\ -\sigma_{12} &\sigma_{11}\] \implies$

$\ba& (x-\mu)'\Sigma\inv(x-\mu) \\ =& \f1{\sigma_{11}\sigma_{22}-\sigma_{12}^2} \[x_1-\mu_1 &x_2-\mu_2\] \[\sigma_{22} &-\sigma_{12}\\ -\sigma_{12} &\sigma_{11}\] \[x_1-\mu_1\\x_2-\mu_2\]\\ =& \f1{\sigma_{11}\sigma_{22}-\sigma_{12}^2} \[(\sigma_{22}(x_1 - \mu_1) - \sigma_{12}(x_2 - \mu_2)) &-\sigma_{12}(x_1 - \mu_1)+\sigma_{11}(x_2-\mu_2)\] \[x_1-\mu_1\\x_2-\mu_2\]\\ =&\frac{1}{\sigma_{11} \sigma_{22}-\sigma_{12}^2}\left[\sigma_{22}\left(x_{1}-\mu_{1}\right)^{2}-2 \sigma_{12}\left(x_{1}-\mu_{1}\right)\left(x_{2}-\mu_{2}\right)+\sigma_{11}\left(x_{2}-\mu_{2}\right)^{2}\right]\\ &\gr\downarrow\text{multiply and divide by }\sigma_{11}\sigma_{22}\\ =&\frac{\sigma_{11} \sigma_{22}}{\sigma_{11} \sigma_{22}-\sigma_{12}^{2}}\left[\frac{\left(x_{1}-\mu_{1}\right)^{2}}{\sigma_{11}}-\frac{2 \sigma_{12}}{\sigma_{11} \sigma_{22}}\left(x_{1}-\mu_{1}\right)\left(x_{2}-\mu_{2}\right)+\frac{\left(x_{2}-\mu_{2}\right)^{2}}{\sigma_{22}}\right]\\ &\gr\downarrow\text{sub in } \sigma_1 = \sqrt{\sigma_{11}}, \sigma_2 = \sqrt{\sigma_{22}}, \rho = \f{\sigma_{12}}{\sigma_1\sigma_2}\\ =&\left(1-\rho^{2}\right)^{-1}\left[\left(\frac{x_{1}-\mu_{1}}{\sigma_{1}}\right)^{2}- 2\rho\left(\frac{x_{1}-\mu_{1}}{\sigma_{1}}\right)\left(\frac{x_{2}-\mu_{2}}{\sigma_{2}}\right)+\left(\frac{x_{2}-\mu_{2}}{\sigma_{2}}\right)^{2}\right]\implies\\ \ea$

$\ba f_\x(x) &= (2\pi)^{-2/2}\underbrace{(\sigma_{11}\sigma_{22}-\sigma_{12}^2)^{-1/2}}_*\exp(-res/2) \\ &\gr(*) = [\sigma_{11}\sigma_{22}(1-\rho^2)]^{-1/2} = [\sigma_1\sigma_2(1-\rho^2)]^{-1/2} \\ &=\f1{2\pi\sigma_1\sigma_2(1-\rho^2)^{1/2}}\exp\{-res/2\} \ea$

Def. Spectral Decomposition

$\Sigma \in \R^{k \times k}$ $\Sigma =Q \d\Lambda Q^{\prime}=\sum_{i=1}^{k} \lambda_{j} \mathbf{q}_{i} \mathbf{q}_{i}^{\prime}$ where

$\begin{aligned}Q &=\left(\begin{array}{lll} \mathbf{q}_{1} & \cdots & \mathbf{q}_{k} \end{array}\right) \in \R^{k \times k} \expl { orthogonal, i.e. }Q^TQ = QQ^T = I \\ \Lambda &=\operatorname{diag}\left(\lambda_{1}, \ldots, \lambda_{k}\right) \text { with } \lambda_{1} \geq \cdots \geq \lambda_{k}>0 \end{aligned}$

Recall $D = P\inv A P$
$\lambda_i$ , and the column vectors of P are the eigen vectors

$\Sigma = Q\Lambda Q' \implies$ $\Lambda$ are the eigen values, and column vectors of Q are the eigen vectors

Properties of the Multivariate Normal

$\mu \in R^{k}$ $\Sigma \in \R^{k \times k}$ $f_{\mathbf{X}}(\mathbf{x})=(2 \pi)^{-k / 2}(\det \Sigma)^{-1 / 2} \exp \left(-(\mathbf{x}-\boldsymbol{\mu})^{\prime} \Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}) / 2\right)$ $\mathbf{X} \sim N_{k}(\boldsymbol{\mu}, \Sigma) ?$

$\Sigma$ , we have
$\Sigma \mathbf{q}_{j}=(\sum_{i=1}^{k} \lambda_{i} \mathbf{q}_{i} \underbrace{\mathbf{q}_{i}^{\prime}) \mathbf{q}_{j}}_{\bc0&i\ne j\\1&i=j\ec} =\lambda_{j} \mathbf{q}_{j}$
$\lambda_j$ $\Sigma$ $\b q_j$
$\Sigma$ $\Sigma^{1 / 2}=Q \Lambda^{1 / 2} Q^{\prime}$ $\Lambda^{1 / 2}=\operatorname{diag}\left(\lambda_{1}^{1 / 2}, \ldots, \lambda_{k}^{1 / 2}\right)$ :
$\begin{aligned} \left(\Sigma^{1 / 2}\right)^{\prime} &=Q \Lambda^{1 / 2} Q^{\prime} = \Sigma^{1 / 2} \expl{hence symmetric} \\ \Sigma^{1 / 2} \Sigma^{1 / 2} &=Q \Lambda^{1 / 2} Q^{\prime} Q \Lambda^{1 / 2} Q^{\prime}\\ &=Q \Lambda^{1 / 2} \Lambda^{1 / 2} Q^{\prime}=Q \Lambda Q^{\prime}=\Sigma \expl{hence a square root} \end{aligned}$
$\mathbf{Z} \sim N_{k}(\mathbf{0}, I)$ $A=\Sigma^{1 / 2}$ $X=A \b Z+\mu \sim N_{k}\left(\mu, A A^{\prime}\right)$ $A A^{\prime}=\Sigma^{1 / 2} \Sigma^{1 / 2}=\Sigma$
$\Sigma$ $f_\x$ $R^{k}$
$f_{\mathrm{X}}$ are given by
$\ba \partial E_{r}(\boldsymbol{\mu}, \Sigma) =\left\{\mathbf{x}:(\mathbf{x}-\boldsymbol{\mu})^{\prime} \Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})=r^{2}\right\} &\expl{ the boundary of the ellipsoidal region with center at }\mu\\ &\gr\quad\quad\ \ \text{and principal axes determined by } \Sigma, r \ea$
$\del$ here indicates boundary of a set, not partial derivative
$\b w = Q'(\b x-\mu) \iff \b x = \mu + Q\b w \expl{Q rotates the axes, }\mu \text{ shifts the ellipsoid}$ :
$\begin{aligned} \partial E_{r}(\boldsymbol{\mu}, \Sigma) &=\left\{\mathbf{x}:\underbrace{(\mathbf{x}-\boldsymbol{\mu})^{\prime} (Q} \Lambda^{-1} \underbrace{Q^{\prime}) (\mathbf{x}-\boldsymbol{\mu})}=r^{2}\right\} \\ &=\boldsymbol{\mu}+Q\left\{\mathbf{w}: \mathbf{w}^{\prime} \Lambda^{-1} \mathbf{w}=r^{2}\right\} \expl{plug in w}\\ &=\boldsymbol{\mu}+Q\left(\partial E_{r}(\mathbf{0}, \Lambda)\right)\expl{w = w-0 so }\mu = 0 \end{aligned}$
$\partial E_{r}\d(\mathbf{0}, \Lambda)=\left\{\mathbf{w}: \sum \frac{w_{i}^{2}}{r^{2} \lambda_{i}}=1\right\}$ $\R^k$ $\b e_i$ $r\lambda_i^{1/2}$
$\del E_r(\mu, \Sigma)$ $\{\mu + c\b q_i: c\in\R^1\}$ $r\lambda_i^{1/2}$
$Q_1$ $\lambda_1$ is the bigger eigen value, then there is more variability along the 1st principal axis.

Ex. $\Sigma$ $Q\Lambda Q'$ $\Sigma\inv = Q\Lambda\inv Q'$

$\Sigma\Sigma\inv = Q\Lambda Q'(Q\Lambda\inv Q') = Q\Lambda\Lambda\inv Q' = QQ' = I$

Lecture 10

Def. Stochastic Process

$\set{(t, X_t):t \in T}$ $X_t$ $(\Omega, \sa, P)$ $T$ is called the index set of the process.
Note In many applications, we need to consider stochastic processes since their dependence on index (t) is important. T can be infinite, negative, and multi-dimensional. It can be a very general set (like the nodes of a graph). Stochastic processes where t is time are referred to as time series.

E.g. $\x = (X_1, ..., X_k)$ $\set{(t, X_t):t \in T}$ $T = \{1, ..., k\}$

E.g. $X_1$ $\{(t, X_t):t\in T\}$ $T = \{1\}$

E.g. Suppose a coin is tossed (tosses are ind.) until the 1st head is observed and we record that number.

$\Omega =\left\{\left(\omega_{1}, \omega_{2}, \ldots\right): \omega_{i} \in\set{0,1}\right\}=\d\bigtimes_{i=1}^{\infty}\{0,1\}$ ,
$\Omega$ here.

$X_{i}(\omega)=\omega_{i}$ $\left\{\left(t, X_{t}\right): t \in \mathbb{N}\right\}$ is a stochastic process, called a Bernoulli(p) process.

$Y(\omega)=i$ $j<i$ $\omega_{j}=0$ $\omega_{i}=1$ $Y$ a well defined r.v.?

$p=$ probability of a head on a single toss.
$p>0$ $Y$ $\d\lim_{n\to\infty}P(\text{first n are tails}) = \lim_{n\to\infty}(1-p)^n = 0$
$Y$ is undefined.

$Y\sim$ geometric(p)

$\begin{aligned}& p_{Y}(i)=P_{Y}(\{i\})=P\left(A_{i}\right) \expl{} A_{i}=\left\{\omega: \omega_{1}=0, \ldots, \omega_{i-1}=0, \omega_{i}=1\right\}\\ =&(1-p)^{i-1} p P\left(\left\{\omega: \omega_{i+1} \in\{0,1\}, \omega_{i+2} \in\{0,1\}, \ldots\right\}\right) \\ =&(1-p)^{i-1} p \end{aligned}$

Ex. $p_{Y}$ defines a probability distribution.

$p_Y(i) \ge 0 \fa i \in \{1, 2, ...\}$
$\suminf p_Y(i) = p\suminf (1-p)^{i-1} = p\ub{\suminf(1-p)^i}_\text{geometric series} = p\f1{1-(1-p)} = 1$

Def. Sample Function

sample function $X(\omega):T\to \R^1$ $X_t(\omega)$ $t$ $T\to\R^1$ .

$X$ $(\Omega, \sa, P)$ $P_X$ as follows:

$X:\Omega\to\R^T$ $\{g\ |\ g:T\to\R^1\}$ $T$ $\R^1$
$\sb^T$ $\sig$ $\R^T$ $\{g:g(t_1)\in(a_1, b_1],...,g(t_n)\in(a_n, b_n]\}$ $\{t_1, ..., t_n\} \subset T$ $(a_1, b_1],...,(a_n, b_n]$
$X\inv B\in \sa, \fa B\in \sb^T$ $P_X(B) = P(X\inv B)$

E.g. $T = [0, \infty), \omega \sim N(0, 1), X_t(\omega) = \omega t$ $X$ $N(0, 1)$ distributed slope.

E.g. $T = [0,1], \omega\sim$ $X_t(\omega) = \cos(\omega t)$ $X$ gives a cosinusoid with random frequency.

Prop 2.6.1 (Kolmogorov's Consistency Theorem)

Background $\{s_1, ..., s_m\} \subset \{t_1, ..., t_n\}$ $(X_{s_1}, ..., X_{s_m})$ $(X_{t_1}, ..., X_{t_n})$ , then we can say that the finite dimensional distributions are consistent.
$T\subset \R$ $(\R^n, B^n, P_{(t_1, ..., t_n)})$ $\{t_1, ..., t_n\}\subset T$
$\exists$ $(\Omega, \sa, P)$ $X_t$ $\{(t, X_t):t\in T\}$ $P_{(X_{t_1}, ..., X_{t_n})} = P_{(t_1, ..., t_n)}$

E.g. $T = \{1, 2, ...\}, P_{(t_1, ..., t_n)}$ $\{0, 1\}^n$ given by the prob function

$p_{(t_1,..., t_n)}(x_1, ..., x_n) = \bc\d\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} &(x_1, ..., x_n) \in \{0, 1\}^n\\0& o/w\ec$

$T = \set{1, 2}$

$\d\sum_{x_2 = 0}^1 p_{t_1}(x_1)p_{t_2}(x_2) = \sum_{x_2 = 0}^1 p^{x_1}(1-p)^{1-x_1}p^{x_2}(1-p)^{1-x_2} = p^{x_1}(1-p)^{1-x_1}\ub{\sum_{x_2 = 0}^1 p^{x_2}(1-p)^{1-x_2}}_{1-p+p = 0} = p_{t_1}(x_1)$

$\{(t, X_t):t\in T\}$

Def. Gaussian Process

$\left\{\left(t, X_{t}\right): t \in T\right\}$ $\left(X_{t_{1}}, \ldots, X_{t_{n}}\right) \sim N_{n}\left(\mu\left(t_{1}, \ldots, t_{n}\right), \Sigma\left(t_{1}, \ldots, t_{n}\right)\right)$

$\mu\left(t_{1}, \ldots, t_{n}\right) \in R^{n}$ $\Sigma\left(t_{1}, \ldots, t_{n}\right) \in R^{n \times n}$ $\left\{t_{1}, \ldots, t_{n}\right\} \subset T$

Def. Gaussian White Noise Process

$\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ $\mu\left(t_{1}, \ldots, t_{n}\right) =(0, \ldots, 0)^{\prime},\ \Sigma\left(t_{1}, \ldots, t_{n}\right) =\operatorname{diag}\left(\sigma^{2}\left(t_{1}\right), \ldots, \sigma^{2}\left(t_{n}\right)\right)$ $\sigma^{2}: T \rightarrow(0, \infty)$

$\left(X_{t_{1}}, \ldots, X_{t_{n}}\right) \sim N_{n}\left(\mathbf{0}, \operatorname{diag}\left(\sigma^{2}\left(t_{1}\right), \ldots, \sigma^{2}\left(t_{n}\right)\right)\right)$

Lecture 11

Recall Def. Mutual Stat Ind:
$(\Omega, \sa, P)$ $\set{\sa_\lambda:\lambda\in\Lambda}$ $\sigs$ $\sa$ $\sa_\lambda$ $P(A_1 \n ... \n A_n) = \d\prod_{i=1}^n P(A_i)$ $\fa n$ $\lambda_1, ..., \lambda_n \in \Lambda$ $A_1 \in \sa_{\lambda_1}, ..., A_n \in \sa_{\lambda_n}$ .

Recall $C$ $\sa(\sc)$ $\sigs$ $\sc \subseteq 2^\Omega$ .

$\sigma$ $X$ $\sigma$ $\mathcal{A}$ $\mathcal{A}_{X}=X^{-1} \mathcal{B}^{1}=\left\{X^{-1} B: B \in \mathcal{B}^{1}\right\}$

Ex. $\sa_X$ $\sig$ $\sa$

$X\inv\null = \null \implies \null\in\sa_X$
$A_1, A_2, ... \in \sa_X \implies \ex B_1, B_2, ... \in \sb^1$ $A_i = X\inv B_i \implies \Uinf A_i = \Uinf X\inv B_i = X\inv\Uinf B_i$
$\Uinf B_i \in \sb^1, \Uinf A_i \in \sa_X$
$A \in \sa_X \implies \ex B\in\sb^1$ $A = X\inv B \implies A^C = (X\inv B)^C = X\inv B^C$
$B^C \in \sb^1$ $A^C \in \sa_X$

Def. Statistically Independent RVs

$\{X_\lambda:\lambda\in\Lambda\}$ $X_\lambda$ $\sa_{X_\lambda}$ $\sigs \{\sa_{X_\lambda}:\lambda\in\Lambda\}$ .

Prop 2.7.1 (Mut. Stat. Ind iff Joint = Prod of Marginals)

$\left\{X_{\lambda}: \lambda \in \Lambda\right\}$ $X_{\lambda}$ $\operatorname{cdf}$ $\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)$ $\left\{\lambda_{1}, \ldots, \lambda_{n}\right\} \subset \Lambda$ .
$F_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\d\prod_{i=1}^{n} F_{X_{\lambda_{i}}}\left(x_{i}\right)$ $\left(x_{1}, \ldots, x_{n}\right)$ ,

Proof

$(\Longrightarrow)$ Suppose mut. stat. ind., show the factorization holds

$\begin{aligned} & F_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right) \\ =& P_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(\left(-\infty, x_{1}\right] \times \cdots \times\left(-\infty, x_{n}\right]\right) \\ =& P\left(\left\{X_{\lambda_{1}} \in\left(-\infty, x_{1}\right]\right\} \cap \cdots \cap\left\{X_{\lambda_{n}} \in\left(-\infty, x_{n}\right]\right\}\right) \\ =& \prod_{i=1}^{n} P\left(\left\{X_{\lambda_{i}} \in\left(-\infty, x_{i}\right]\right\}\right)=\prod_{i=1}^{n} F_{X_{\lambda_{i}}}\left(x_{i}\right) \end{aligned}$

$\Longleftarrow$ ) Suppose the factorization holds, show mut. stat. ind.

$F_{\left(x_{\lambda_{1}}, \ldots, x_{\lambda_{n}}\right)}$ $P_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}$ $\prod_{i=1}^{n} F_{X_{\lambda_{j}}}$ $P_{X_{\lambda_{i}}}$ $X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}$ are mutually statistically independent.

$\left\{\prod_{i=1}^{n} F_{X_{\lambda_{i}}}:\left\{\lambda_{1}, \ldots, \lambda_{n}\right\} \subset \Lambda \text { for some } n\right\}$ $P_{X}$ $\left\{X_{\lambda}: \lambda \in \Lambda\right\}$ are mutually statistically independent.

Prop 2.7.2 (Mut. Stat. Ind iff Joint = Prod of Marginals)

$\left\{X_{\lambda}: \lambda \in \Lambda\right\}$ $\left\{\lambda_{1}, \ldots, \lambda_{n}\right\} \subset \Lambda:$
$\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)$ discrete $X_{\lambda}$ $p_{\left(x_{\lambda_{1}}, \ldots, x_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\prod_{i=1}^{n} p_{X_{\lambda_{i}}}\left(x_{i}\right)$ $\left(x_{1} , \ldots, x_{n}\right)$
$\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)$ a.c $X_{\lambda}$ $f_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\prod_{i=1}^{n} f_{X_{\lambda_{i}}}\left(x_{i}\right)$ $\left(x_{1}, \ldots, x_{n}\right)$

Proof Discrete case

$\{X_\lambda:\lambda \in \Lambda\}$ $\left(F_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\d\prod_{i=1}^{n} F_{X_{\lambda_{i}}}\left(x_{i}\right)\right)$ $X_{\lambda_i}$ $F_{X_{\lambda_i}}$ .

$X_{\lambda_i}$ $P_{X_{\lambda_i}} = F_{X_{\lambda_i}} - \d\lim_{\delta_i\downarrow0}F_{X_{\lambda_i}}(x_i - \delta_i)$
$X_{\lambda_1}, ..., X_{\lambda_n}$ has the probability function

$\ba P_{(X_{\lambda_1}, ..., X_{\lambda_n})} (x_{\lambda_1}, ..., x_{\lambda_n}) &= \d\lim_{\delta_1 \downarrow 0}...\lim_{\delta_n \downarrow 0}P_{(X_{\lambda_1}, ..., X_{\lambda_n})}(\bigtimes_{i=1}^n(x_i-\delta_i, x_i])\\ &= \lim_{\delta_1 \downarrow 0}...\lim_{\delta_n \downarrow 0}\prod_{i=1}^n(F_{X_{\lambda_i}} - F_{X_{\lambda_i}}(x_i - \delta_i))\\ &= \prod_{i=1}^n\lim_{\delta_i \downarrow 0} (F_{X_{\lambda_i}} - F_{X_{\lambda_i}}(x_i - \delta_i))\\ &= \prod_{i=1}^n p_{X_i}(x_i)\\ &= \sum_{z_1 \le x_1, ..., z_n \le x_n}p_{x_{\lambda_i}}(z_1)...p_{x_{\lambda_n}}(z_n)\\ &= \prod_{i=1}^n\sum_{z_i\le x_i}p_{x_{\lambda_i}}(z_i)\\ &= \prod_{i=1}^n F_{x_{\lambda_i}}(x_i) \ea$

Proof Absolutely Continuous Case

$F_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\d\prod_{i=1}^{n} F_{X_{\lambda_{i}}}\left(x_{i}\right)$ $f_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\d\prod_{i=1}^{n}{\del F_{X_{\lambda_{i}}}\left(x_{i}\right)\o\del x_i} = \prod_{i=1}^nf_{x_{\lambda_i}}(x_i)$ $F_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right)=\d\prod_{i=1}^{n} \int_{-\infty}^{x_i}f_{x_{\lambda_i}}(x_i)dz_i = \d\prod_{i=1}^{n} F_{X_{\lambda_{i}}}\left(x_{i}\right)$

$\{X_\lambda:\lambda\in\Lambda\}$ are mutually statistically independent by Prop 2.

E.g. $(p)$ process

$\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ $\left(x_{1}, \ldots, x_{n}\right) \in\{0,1\}^{n}$

$\begin{aligned}& p_{(x_{t_1}, \ldots, x_{t_{n}})}(x_{1}, \ldots, x_{n})\\ =& p^{\sum_{i=1}^{n} x_{i}}(1-p)^{n-\sum_{i=1}^{n} x_{i}} \\ =& \prod_{i=1}^{n} p^{x_{i}}(1-p)^{1-x_{i}} \\ =& \prod_{i=1}^{n} p_{x_{t_{i}}}\left(x_{i}\right) \end{aligned}$

$X_{t_{i}} \sim \operatorname{Bernoulli}(p)$ $X_{\lambda}$ are mut. stat. ind.

E.g. Gaussian white noise process

$\left(X_{t_{1}}, \ldots, X_{t_{n}}\right) \sim N_{n}\left(\mathbf{0}, \operatorname{diag}\left(\sigma^{2}\left(t_{1}\right), \ldots, \sigma^{2}\left(t_{n}\right)\right)\right)$ $\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ , and

$\begin{aligned} & f_{\left(X_{\lambda_{1}}, \ldots, X_{\lambda_{n}}\right)}\left(x_{1}, \ldots, x_{n}\right) \\ =&(2 \pi)^{-n / 2}\left(\sigma^{2}\left(t_{1}\right) \cdots \sigma^{2}\left(t_{n}\right)\right)^{-1 / 2} \exp \left(-\frac{1}{2} \sum_{i=1}^{n} \frac{x_{i}^{2}}{\sigma^{2}\left(t_{i}\right)}\right) \\ =& \prod_{i=1}^{n}(2 \pi)^{-1 / 2} \sigma^{-1}\left(t_{i}\right) \exp \left(-\frac{1}{2} \frac{x_{i}^{2}}{\sigma^{2}\left(t_{i}\right)}\right) \\ =& \prod_{i=1}^{n} f_{X_{t_{i}}}\left(x_{i}\right) \end{aligned}$ $\left(x_{1}, \ldots, x_{n}\right) \in R^{n}$ $X_{t_{i}} \sim N\left(0, \sigma^{2}\left(t_{i}\right)\right)$

$X_{\lambda}$ are mut. stat. ind. by Prop. 2.7.2

Def. Principal Components

$\mathbf{X} \sim N_{k}(\mu, \Sigma)$ $\Sigma=Q \Lambda Q^{\prime}$ $\mathbf{Y}=Q^{\prime} \mathbf{X} \sim N_{k}\left(Q^{\prime} \mu, Q^{\prime} \Sigma Q\right)=N_{k}\left(Q^{\prime} \mu, \Lambda\right)$ , so

$\begin{aligned}f_{\mathbf{Y}}(\mathbf{y}) &=\prod_{i=1}^{n}(2 \pi)^{-1 / 2} \lambda_{i}^{-1 / 2} \exp \left(-\frac{1}{2} \frac{\left(y_{i}-\mathbf{q}_{i}^{\prime} \boldsymbol{\mu}\right)^{2}}{\lambda_{i}}\right) \\ &=\prod_{i=1}^{n} f_{Y_{i}}\left(y_{i}\right) \end{aligned}$

$Y_{i}=\mathbf{q}_{i}^{\prime} \mathbf{X}=\sum_{j=1}^{k} q_{j i} X_{j} \sim N\left(\mathbf{q}_{i}^{\prime} \mu, \lambda_{i}\right)=N\left(\sum_{j=1}^{k} q_{j i} \mu_{j}, \lambda_{i}\right)$

$Y_{1} \ldots ., Y_{l}$ are mut. stat. ind

Lecture 12

$\mathbf{X}$ $P_{\mathbf{X}}$ $\mathbf{Y}=T(\mathbf{X}) = y$ is observed

$\mathbf{X}$ $T(\mathbf{X})=\mathbf{y}$

Conditional Dist - Discrete

$\mathbf{X}$ $p_{\mathbf{X}}$

$T(\mathbf{x}) \neq \mathbf{y}$ $\mathbf{X}$ $T(\mathbf{x})=\mathbf{y}$ $p_{\mathbf{X} \mid \mathbf{Y}}(\mathbf{x} \mid \mathbf{y}) = 0$
$T(\mathbf{x})=\mathbf{y}$ $\d p_{\mathbf{X} \mid \mathbf{Y}}(\mathbf{x} \mid \mathbf{y}) =P_{\mathbf{X} \mid \mathbf{Y}}\left(\{\mathbf{x}\} \mid T^{-1}\{\mathbf{y}\}\right) =\frac{P_{\mathbf{X}}\left(\{\mathbf{x}\} \cap T^{-1}\{\mathbf{y}\}\right)}{P_{\mathbf{X}}\left(T^{-1}\{\mathbf{y}\}\right)} =\frac{p_{\mathbf{X}}(\mathbf{x})}{\sum_{\mathbf{z} \in T^{-1}\{\mathbf{y}\}} p_{\mathbf{X}}(\mathbf{z})}=\frac{p_{\mathbf{X}}(\mathbf{x})}{p_{\mathbf{Y}}(\mathbf{y})}$

E.g. $\left(n, p_{1}, \ldots, p_{k}\right)$

$Y=T\left(X_{1}, \ldots, X_{k}\right)=X_{1} \sim \operatorname{binomial}\left(n, p_{1}\right)$ $\left(X_{1}, \ldots, X_{k}\right) \mid X_{1}=x_{1}$ $\left(X_{2}, \ldots, X_{k}\right) \mid X_{1}=x_{1}$

$x_{2}, \ldots, x_{k} \in\left\{0, \ldots, n-x_{1}\right\}, x_{2}+\cdots+x_{k}=n-x_{1}$

$\begin{aligned}\implies p_{\left(X_{2}, \ldots, X_{k}\right) \mid X_{1}}\left(x_{2}, \ldots, x_{k} \mid x_{1}\right) &=\frac{\left(\begin{array}{c} n \\ x_{1} x_{2} \ldots x_{k} \end{array}\right) p_{1}^{x_{1}} p_{2}^{x_{2}} \cdots p_{k}^{x_{k}}}{\left(\begin{array}{c} n \\ x_{1} \end{array}\right) p_{1}^{x_{1}}\left(1-p_{1}\right)^{n-x_{1}}} \\ &=\frac{\left(n-x_{1}\right) !}{x_{2} ! \cdots x_{k} !}\left(\frac{p_{2}}{1-p_{1}}\right)^{x_{2}} \cdots\left(\frac{p_{k}}{1-p_{1}}\right)^{x_{k}} \end{aligned}$

$\left(X_{2}, \ldots, X_{k}\right) \mid X_{1}=x_{1} \sim$ $\left(n-x_{1}, \frac{p_{2}}{1-p_{1}}, \ldots, \frac{p_{k}}{1-p_{1}}\right)$

Ex. $\mathbf{X} \sim \operatorname{multinomial}\left(n, p_{1}, \ldots, p_{k}\right)$ $Y=X_{1}+\cdots+X_{l}$ $l \leq k$ $\mathbf{X}$ $Y = y$

$Y=X_{1}+\ldots+X_{l} \sim$ $\left(n, p_{1}+\ldots+p_{l}\right)$ .

$P_{\left(X_{1}, \ldots, X_{k}\right) \mid Y}\left(x_{1}, \ldots, x_{k} \mid y\right)$

$=\frac{\left(\begin{array}{c}n \\ x_{1} \cdots x_{k}\end{array}\right) p_{1}^{x_{1}} \ldots p_{k}^{x_{k}}}{\left(\begin{array}{l}n \\ y\end{array}\right)\left(\d\sum_{i=1}^{l} p_{i}\right)^{y}\left(1-\d\sum_{i=1}^{l} p_{i}\right)^{n-y}}$

$=\d\underbrace{\frac{y !}{x_{1} ! \cdots x_{l} !}\left(\frac{p_{1}}{\sum_{i=1}^{l} p_{i}}\right)^{x_{1}} ... \left(\frac{p_{l}}{\sum_{i=1}^{l} p_{i}}\right)^{x_{l}}} \cdot \underbrace{\frac{(n-y) !}{x_{l+1} ! \cdots x_{k} !}\left(\frac{p_{l+1}}{\sum_{i=l+1}^{k} p_{i}}\right)^{x_{l+1}} \cdots\left(\frac{p_{k}}{\sum_{i=l+1}^{k} p_{i}}\right)^{x_{k}}}$

$\d\left(y, \frac{p_{1}}{\sum_{i=1}^{l} p_{i}}, \cdots \frac{p_{l}}{\sum_{i=1}^{l} p_{i}}\right) \quad$ $\d\left(n-y, \frac{p_{\ell+1}}{\sum_{i=l+1}^{k} p_{i}}, \cdots, \frac{p_{k}}{\sum_{i=l+1}^{k} p_{i}}\right)$

Conditional Dist - A.C.

$\mathbf{X}$ $f_{X}$ $T: R^{k} \rightarrow R^{l}$ $\mathbf{x} \in T^{-1}\{\mathbf{y}\}$ $\mathbf{X}$ $T(\mathbf{X})=\mathbf{y}$ is

$\d f_{\mathbf{X} \mid \mathbf{Y}}(\mathbf{x} \mid \mathbf{y}) = \lim _{\delta_{1} \downarrow 0, \delta_{2} \downarrow 0}\left\{\frac{P_{\mathbf{X}}\left(B_{\delta_{1}}(\mathbf{x}) \cap T^{-1} B_{\delta_{2}}(\mathbf{y})\right)}{\operatorname{Vol}\left(B_{\delta_{1}}(\mathbf{x}) \cap T^{-1} B_{\delta_{2}}(\mathbf{y})\right)} / \frac{P_{\mathbf{Y}}\left(B_{\delta_{2}}(\mathbf{y})\right)}{\operatorname{Vol}\left(B_{\delta_{2}}(\mathbf{y})\right)}\right\} \stackrel{\text { fact }}{=} \frac{f_{\mathbf{X}}(\mathbf{x}) J_{T}(x)}{f_{\mathbf{Y}}(\mathbf{y})}$

$T$ to be many to one)

$J_{T}(\mathbf{x})=\left|\det\left(\begin{array}{ccc} \frac{\partial T_{1}(\mathbf{x})}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\mathbf{x})}{\partial x_{k}} \\ \vdots & & \vdots \\ \frac{\partial T_{l}(\mathbf{x})}{\partial x_{1}} & \ldots & \frac{\partial T_{l}(\mathbf{x})}{\partial x_{k}} \end{array}\right)\left(\begin{array}{ccc} \frac{\partial T_{1}(\mathbf{x})}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\mathbf{x})}{\partial x_{k}} \\ \vdots & & \vdots \\ \frac{\partial T_{l}(\mathbf{x})}{\partial x_{1}} & \ldots & \frac{\partial T_{l}(\mathbf{x})}{\partial x_{k}} \end{array}\right)'\right|^{-1 / 2}$

E.g. Projections

$T\left(x_{1}, \ldots, x_{k}\right)=\left(x_{1}, x_{2}\right)$ $l=2$

$\ba J_{T}(\mathbf{x}) &=\left|\det\left(\begin{array}{ccc} \frac{\partial T_{1}(\mathbf{x})}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\mathbf{x})}{\partial x_{k}} \\ \frac{\partial T_{2}(\mathbf{x})}{\partial x_{1}} & \ldots & \frac{\partial T_2(\mathbf{x})}{\partial x_{k}} \end{array}\right) \left(\begin{array}{cc} \frac{\partial T_{1}(\mathbf{x})}{\partial x_{1}} & \frac{\partial T_{2}(\mathbf{x})}{\partial x_{1}} \\ \vdots & \vdots \\ \frac{\partial T_{1}(\mathbf{x})}{\partial x_{k}} & \frac{\partial T_{2}(\mathbf{x})}{\partial x_{k}} \end{array}\right)\right|^{-1 / 2}\\ &=\left|\begin{array}{cccc} \det\left(\begin{array}{ccc} 1 & 0 & \ldots & 0 \\ 0 & 1 & \ldots & 0 \end{array}\right) & \left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \\ \vdots & \vdots \\ 0 & 0 \end{array}\right) \end{array}\right|^{-1 / 2} =\left|\det\left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right)\right|^{-1 / 2} =1 \ea$

$f_{\left(X_{1}, X_{2}\right)}\left(x_{1}, x_{2}\right)=\int_{-\infty}^{\infty} \ldots \int_{-\infty}^{\infty} f_{\left(X_{1}, \ldots, X_{k}\right)}\left(x_{1}, \ldots, x_{k}\right) d x_{3} \ldots d x_{k} \\ \implies f_{\left(X_{3}, \ldots, X_{k}\right) \mid\left(X_{1}, X_{2}\right)}\left(x_{3}, \ldots, x_{k} \mid x_{1}, x_{2}\right)=\frac{f_{\left(X_{1}, \ldots, X_{k}\right)}\left(x_{1}, \ldots, x_{k}\right)}{f_{\left(X_{1}, X_{2}\right)}\left(x_{1}, x_{2}\right)}$

Ex. $T\left(x_{1_{ }}, \ldots, x_{k}\right)=x_{1}$ .

$T\left(x_{1}, \ldots, x_{2}\right)=x_{1}$ $J_{T}\left(x_{1}, \ldots, x_{n}\right)=\left|\operatorname{det}\left(\frac{\partial T}{\partial x_{1}}, \ldots . . \frac{\partial T}{\partial x_{k}}\right)\left(\begin{array}{c}\frac{\partial T}{\partial x_{1}} \\ \vdots \\ \frac{\partial T}{\partial x_{k}}\end{array}\right)\right|^{-\frac{1}{2}}$ $=\left|\operatorname{det}\left(\begin{array}{llll}1 & 0 & \cdots & 0\end{array}\right)\left(\begin{array}{l}1 \\ 0 \\ \vdots \\ 0\end{array}\right)\right|^{-\frac{1}{2}}$ $=|\operatorname{det}(1)|^{-\frac{1}{2}}=1$ $\d f_{X_{1}}\left(x_{1}\right)=\int_{-\infty}^{\infty} \ldots \int_{-\infty}^{\infty} f_{\left(X_{1}, \cdots, X_{k}\right)}\left(x_{1}, z_{2}, \cdots, z_{k}\right) d z_{1}, \ldots d z_{k}$ $\d f_{\left(X_{2}, \ldots, X_{k}\right) \mid X_{1}}\left(x_{2}, \ldots, x_{k} \mid x_{1}\right)=\frac{f_{\left(X_{1}, \ldots, X_{k}\right)}\left(x_{1}, \ldots, x_{k}\right) J_{T}\left(x_{1}, \ldots, x_{k}\right)}{f_{X_{1}}\left(x_{1}\right)}=\frac{f_{\left(X_{1}, \cdots, X_{k}\right)}\left(x_{2}, \cdots, x_{k}\right)}{f_{X_{1}}\left(x_{1}\right)}$

E.g. $N_{k}(\mu, \Sigma)$

$\mathbf{X} \sim N_{k}(\mu, \Sigma)$ $\mathbf{X}_{1}=T(\mathbf{X})=\left(X_{1}, \ldots, X_{l}\right)^{\prime}$ $l \leq k$

$\mu$ $\Sigma$ as

$\begin{aligned} \mu &=\left(\begin{array}{l} \mu_{1} \\ \mu_{2} \end{array}\right) \text { where }\quad \mu_{1} \in R^{l}, \quad \mu_{2} \in R^{k-l} \\ \Sigma &=\left(\begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{12}^{\prime} & \Sigma_{22} \end{array}\right) \text { where } \quad \Sigma_{11} \in R^{l \times l},\quad \Sigma_{12} \in R^{l \times(k-l)}, \quad \Sigma_{22} \in R^{(k-l) \times(k-l)} \end{aligned}$

Ex. $\Sigma_{11}$ $\Sigma_{22}$ $\Sigma$ is p.d.

$\ba \x' \Sigma\x &=\[\x_1\\ \x_2\]' \[\Sigma_{11} &\Sigma_{12}\\ \Sigma_{12}' &\Sigma_{22}\]\[\x_1\\ \x_2\] =\[\x_1'\Sigma_{11} + \x_2'\Sigma_{12} &\x_1'\Sigma_{12}+\x_2'\Sigma_{22}\]\[\x_1\\ \x_2\] \\ &=[\x_1'\Sigma_{11}\x_1 + \x_2'\Sigma_{12}\x_1 + \x_1'\Sigma_{12}\x_2 + \x_1'\Sigma_{22}\x_2]\\ &=[\x_1'\Sigma_{11}\x_1 + 2\x_1'\Sigma_{12}\x_2 + \x_1'\Sigma_{22}\x_2] \\ &\ge 0\ (>0 \t{ if }X \ne 0) \ea$

$\x_2 = 0, \x_1 \ne 0 \implies \x_1'\Sigma_{11}\x_1 = \x'\Sigma\x > 0 \implies \Sigma_{11}$ is p.d.

$\x_1 = 0, \x_2 \ne 0 \implies \x_2'\Sigma_{22}\x_2 = \x'\Sigma\x > 0 \implies \Sigma_{22}$ is p.d.

$\qed$

$\mathbf{Y}$ , we need another matrix decomposition:

Def. Gram-Schmidt (QR) decomposition

$A=\left(\mathbf{a}_{1} \cdots \mathbf{a}_{k}\right) \in \R^{k \times k}$ $k$ $\R^{k}$ $\mathbf{a}_{1}, \ldots, \mathbf{a}_{k}$ $\left(c_{1} \mathbf{a}_{1}+\cdots+c_{k} \mathbf{a}_{k}=\mathbf{0}\right.$ $\left.c_{1}=\ldots=c_{k}=0\right)$ $\R^k$ $L\left\{\mathbf{a}_{1}, \ldots, \mathbf{a}_{k}\right\}=\left\{c_{1} \mathbf{a}_{1}+\cdots+c_{k} \mathbf{a}_{k}: c_{1}, \ldots, c_{k} \in \R^{1}\right\}=\R^{k}$

$\left\{\mathbf{a}_{1}, \ldots, \mathbf{a}_{k}\right\}$ $\left\{\mathbf{q}_{1}, \ldots, \mathbf{q}_{k}\right\}$ $\R^{k}$

$\mathbf{q}_{1}=\f{\mathbf{a}_{1}} {\left\|\mathbf{a}_{1}\right\|}, r_{11}=\left\|\mathbf{a}_{1}\right\| >0$

$\d\mathbf{q}_{2}=\frac{\mathbf{a}_{2}-\left(\mathbf{q}_{1}^{\prime} \mathbf{a}_{2}\right) \mathbf{q}_{1}}{\left\|\mathbf{a}_{2}-\mathbf{q}_{1}^{\prime} \mathbf{a}_{2}\right\|}, r_{12}=\mathbf{q}_{1}^{\prime} \mathbf{a}_{2}, r_{22}=\left\|\mathbf{a}_{2}-\left(\mathbf{q}_{1}^{\prime} \mathbf{a}_{2}\right) \mathbf{q}_{1}\right\| >0$

$Q=\left(\mathbf{q}_{1} \cdots \mathbf{q}_{k}\right) \in \R^{k \times k}$ $R$ is a unique upper triangular matrix with positive diagonals

$A$ $Q$ $R$ :

$QR=\left(\mathbf{q}_{1} \cdots \mathbf{q}_{k}\right)\left(\begin{array}{cccc} r_{11} & r_{12} & \cdots & r_{1 k} \\ 0 & r_{22} & \cdots & r_{2 k} \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & r_{k k} \end{array}\right) = A$

Def. Orthogonal Matrix

$Q$ orthogonal $Q^TQ = QQ^T = I$

Properties:

$Q^T = Q\inv$ and they are both orthogonal
det(Q) = 1 or -1
the product of orthogonal matrices is orthogonal

Ex. $A=QR$ $R$ $Q$

$\exists$ $R_1, R_2$ $A = QR_1 = QR_2$

$Q\inv = Q'$ $Q'A = R_1 = R_2$ , which contradicts the hypothesis.

Def. Cholesky Decomposition

$\Sigma$ $Q R$ $\Sigma^{1 / 2}=Q R \implies$ $\Sigma=\Sigma^{1 / 2} \Sigma^{1 / 2}\stackrel{\text{by symmetry}}=\left(\Sigma^{1 / 2}\right)^{\prime} \Sigma^{1 / 2}= (QR)'(QR) = R'Q'QR = R^{\prime} R$

Ex. $R$ ), for 2x2 matrices.

The product of 2 upper triangular matrices with positive diagonals is upper triangular with positive diagonals.
$a_1 &b_1\\ 0 &c_1\]\[a_2 &b_2\\ 0 &c_2\] = \[a_1a_2 &a_1b_2+b_1c_2\\0&c_2c_2$ is upper triangular with positive diagonals
An upper triangular matrix with positive diagonals is nonsingular, and its inverse is upper triangular with positive diagonals that are equal to the inverse of the diagonal elements of the original matrix.
$a&b\\0&c$ is nonsingular/invertible since its determinant > 0
$a&b\\0&c\]\inv = \f1{ac}\[c&-b\\0&a\] = \[1/a &-b/{ac}\\ 0 &1/c$
The matrix R in the Cholesky decomposition is unique.
$\Sigma = R_1'R_1 = R_2'R_2$ $R_1, R_2$ are upper triangular matrices with positive diagonals.
$I = \ub{(R_1')\inv R_2'}_{A'}\ub{R_2R_1\inv}_A$ (A is just an arbitrary letter)
$A' = A\inv$ must hold:
$A' = (R_1')\inv R_2' = (R_2R_1\inv)'$
$A\inv = (R_2, R_1\inv)\inv$
$A'$ $A\inv$ is upper triangular.
This means they are both diagonal matrices, so A is diagonal as well.
$A'A = \[a &0\\ b&c\]\[a&b\\0&c\] = \[a^2&ab\\ab&b^2+c^2\] \implies \bc a^2 = 1\implies a = 1\\ab=0\implies b = 0\\b^2 + c^2 = 1 \implies c = 1\ec$
$A = I \implies R_2R_1\inv = I \implies R_2 = R_1$

$k \times k$ upper triangular matrices.

$X_1$ )

$\mathbf{X}=\left(\begin{array}{l} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{array}\right) \sim N_{k}\left(\left(\begin{array}{l} \mu_{1} \\ \mu_{2} \end{array}\right),\left(\begin{array}{ll} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{12}^{\prime} & \Sigma_{22} \end{array}\right)\right)$ $\mathbf{X}_{1} \in R^{\prime}$ $\mathbf{X}_{1} \sim N_{l}\left(\mu_{1}, \Sigma_{11}\right)$ .

Proof $\Sigma$ , we get

$\Sigma =\left(\begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{12}^{\prime} & \Sigma_{22} \end{array}\right) =\left(\begin{array}{cc} R_{11}^{\prime} & 0 \\ R_{12}^{\prime} & R_{22} \end{array}\right)\left(\begin{array}{cc} R_{11} & R_{12} \\ 0 & R_{22} \end{array}\right) =\left(\begin{array}{ll} R'_{11}R_{11}\expl{Cholesky} & R'_{11}R_{12} \\ R'_{12}R_{11} & R'_{12}R_{12}+R_{22}R'_{22} \end{array}\right)$

$\mathbf{Z} \sim N_{k}(\mathbf{0}, I)$ $Z_{1}, \ldots, Z_{k}$ $\b Z$ s.t.

$\mathbf{Z}=\left(\begin{array}{l} \mathbf{Z}_{1} \\ \mathbf{Z}_{2} \end{array}\right) \text { where } \mathbf{Z}_{1} \sim N_{l}(\mathbf{0}, I) \text { stat. ind. of } \mathbf{Z}_{2} \sim N_{k-l}(\mathbf{0}, I)$

from lecture 9 $\mathbf{a}+A \mathbf{Z} \sim N_{k}\left(\mathbf{a}, A A^{\prime}\right)$

$\ba \mathbf{X} &=\left(\begin{array}{l} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{array}\right)\\ &=\left(\begin{array}{l} \mu_{1} \\ \mu_{2} \end{array}\right)+\left(\begin{array}{cc} R_{11} & R_{12} \\ 0 & R_{22} \end{array}\right)^{\prime}\left(\begin{array}{l} \mathbf{Z}_{1} \\ \mathbf{Z}_{2} \end{array}\right)\expl{express X as } \mu + R'Z\\ &=\left(\begin{array}{l} \mu_{1} \\ \mu_{2} \end{array}\right)+\left(\begin{array}{cc} R_{11}^{\prime} & 0 \\ R_{12}^{\prime} & R_{22}^{\prime} \end{array}\right)\left(\begin{array}{l} \mathbf{Z}_{1} \\ \mathbf{Z}_{2} \end{array}\right) \\ &=\left(\begin{array}{l} \boldsymbol{\mu}_{1}+R_{11}^{\prime} \mathbf{Z}_{1} \\ \boldsymbol{\mu}_2 + R_{12}^{\prime} \mathbf{Z}_{1}+R_{22}^{\prime} \mathbf{Z}_{2} \end{array}\right) \ea$

$\mathbf{X}_{1}=\mu_{1}+R_{11}^{\prime} \mathbf{Z}_{1} \sim N_{l}\left(\mu_{1}, R_{11}^{\prime} R_{11}\right)=N_{l}\left(\mu_{1}, \Sigma_{11}\right) \expl{marginal dist of first }l \t{ coordinates}$

Ex. $I_{r}$ $r \times r$ $C=\left(\begin{array}{cc} 0 & I_{k-l} \\ I_{l} & 0 \end{array}\right)$ $\mathbf{X}_{2}$ in Proposition 2.8.1.

$C\[\x_1\\ \x_2\] = \[\x_2\\ \x_1\] \sim N_k$\[\mu_2\\ \mu_1\], \[\Sigma_{22} &\Sigma_{12}'\\ \Sigma_{12} &\Sigma_{11}\]$$

$A$ $\mathbf{X}$ .

$X_1$ )

$\mathbf{X}=\left(\begin{array}{l} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{array}\right) \sim N_{k}\left(\left(\begin{array}{l} \mu_{1} \\ \mu_{2} \end{array}\right),\left(\begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{12}^{\prime} & \Sigma_{22} \end{array}\right)\right)$ $\mathbf{X}_{1} \in R^{l}$ $\mathbf{Y}=\mathbf{X}_{2}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \mathbf{X}_{1}$ and

$\b Y\sim N_{k-l}\left(\boldsymbol{\mu}_{2}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \boldsymbol{\mu}_{1}, \Sigma_{22}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \Sigma_{12}\right)$ $\mathbf{X}_{1} \sim N_{l}\left(\boldsymbol{\mu}_{1}, \Sigma_{11}\right)$

Proof

$\begin{aligned} \left(\begin{array}{l} \mathbf{X}_{1} \\ \mathbf{Y} \end{array}\right) &=\left(\begin{array}{cc} I & 0 \\ -\Sigma_{12}^{\prime} \Sigma_{11}^{-1} & I \end{array}\right)\left(\begin{array}{l} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{array}\right)=A \mathbf{X} \end{aligned}$

$A\x\sim N_{k}\left(A \mu, A \Sigma A^{\prime}\right)$ $A \mu =\left(\begin{array}{c} \mu_{1} \\ \mu_{2}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \mu_{1} \end{array}\right), A \Sigma A^{\prime} =\left(\begin{array}{cc} \Sigma_{11} & 0 \\ 0 & \Sigma_{22}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \Sigma_{12} \end{array}\right)$

$\mathbf{W}=\left(\begin{array}{l} \mathbf{W}_{1} \\ \mathbf{W}_{2} \end{array}\right) \sim N_{k}\left(\left(\begin{array}{l} v_{1} \\ v_{2} \end{array}\right),\left(\begin{array}{cc} \Sigma_{11} & 0 \\ 0 & \Sigma_{22} \end{array}\right)\right)$ , then

$\begin{aligned} & f_{\mathbf{W}}\left(\mathbf{w}_{1}, \mathbf{w}_{2}\right)\\ =&(2 \pi)^{-k / 2}\left(\operatorname{det}\left(\begin{array}{cc} \Sigma_{11} & 0 \\ 0 & \Sigma_{22} \end{array}\right)\right)^{-1 / 2} \exp \left(-\frac{1}{2}\left(\begin{array}{c} \mathbf{w}_{1}-v_{1} \\ \mathbf{w}_{2}-v_{2} \end{array}\right)^{\prime}\left(\begin{array}{cc} \Sigma_{11}^{-1} & 0 \\ 0 & \Sigma_{22}^{-1} \end{array}\right)\left(\begin{array}{c} \mathbf{w}_{1}-v_{1} \\ \mathbf{w}_{2}-v_{2} \end{array}\right)\right) \\ =&(2 \pi)^{-1 / 2}\left(\operatorname{det} \Sigma_{11}\right)^{-1 / 2} \exp \left(-\left(\mathbf{w}_{1}-v_{1}\right)^{\prime} \Sigma_{11}^{-1}\left(\mathbf{w}_{1}-v_{1}\right) / 2\right) \times \\ &(2 \pi)^{-(k-1) / 2}\left(\operatorname{det} \Sigma_{22}\right)^{-1 / 2} \exp \left(-\left(\mathbf{w}_{2}-v_{2}\right)^{\prime} \Sigma_{22}^{-1}\left(\mathbf{w}_{2}-v_{2}\right) / 2\right) \end{aligned}$

$\mathbf{W}_{1}$ $\mathbf{W}_{2}$ are statistically independent and this proves the second part.

Ex. $A\x\sim N_{k}\left(A \mu, A \Sigma A^{\prime}\right)$ $A \mu =\left(\begin{array}{c} \mu_{1} \\ \mu_{2}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \mu_{1} \end{array}\right), A \Sigma A^{\prime} =\left(\begin{array}{cc} \Sigma_{11} & 0 \\ 0 & \Sigma_{22}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \Sigma_{12} \end{array}\right)$

$(x_{i_1}, …, x_{i_l})$ $X\sim N_k(\mu, \Sigma)$ .
$A \in \R^{k\times k}$ $e_{i_n}$ $l$ $1\le n\le l$ $k-l$ rows contain the remaining basis vectors in any order.
$A\x = \[A_1 \ \ \ \expl{l rows}\\ A_2 \expl{k-l rows}\]\x = \[A_1\x\\ A_2\x\]$ $A_1\x = \[X_{i_1}\\ \vdots\\ X_{i_l}\]$
$(A\x) \sim N_k(A\mu, A\Sigma A')$ $A\mu = \[A_1\mu\\A_2\mu\], A\Sigma A' = \[A_1\Sigma A_1' &A_1\Sigma A_2'\\ A_2\Sigma A_1' & A_2\Sigma A_2'\]$
$A_1\x = \[X_{i_1}\\ \vdots\\ X_{i_l}\]\sim N(A_1\mu, A_1\Sigma A_1')$ $A_1\mu = \[\mu_{i_1}\\ \vdots\\ \mu_{i_l}\], A\Sigma A_1' = \[\sigma_{i_1 i_1} &... &\sigma_{i_1 i_l}\\\vdots &\ddots &\vdots\\ \sigma_{i_l i_i} &... & \sigma_{i_l i_l}\]$

$X_2|X_1$ )

$\mathbf{X}_{2} \mid \mathbf{X}_{1}=\mathbf{x}_{1} \sim N_{k-l}\left(\mu_{2}+\Sigma_{12}^{\prime} \Sigma_{11}^{-1}\left(\mathbf{x}_{1}-\mu_{1}\right), \Sigma_{22}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \Sigma_{12}\right)$

Proof:

$\left(\begin{array}{l} \mathbf{x}_{1} \\ \mathbf{x}_{2} \end{array}\right)=T\left(\begin{array}{l} \mathbf{x}_{1} \\ \mathbf{y} \end{array}\right)=\left(\begin{array}{c} \mathbf{x}_{1} \\ \mathbf{y}+\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \mathbf{x}_{1} \end{array}\right)$ $J_{T}\left(\mathbf{x}_{1}, \mathbf{y}\right)=\left|\operatorname{det}\left(\begin{array}{ll} I_{1} & 0 \\ \text { stuff } & I_{k-1} \end{array}\right)\right|^{-1}=1$

$f_{\mathbf{X}}\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right)=f_{\mathbf{X}_{1}}\left(\mathbf{x}_{1}\right) f_{\mathbf{Y}}\left(\mathbf{x}_{2}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \mathbf{x}_{1}\right)$

$\d f_{\mathbf{X}_{2} \mid \mathbf{X}_{1}}\left(\mathbf{x}_{2} \mid \mathbf{x}_{1}\right)=\frac{f_{\mathbf{X}}\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right)}{f_{\mathbf{X}_{1}}\left(\mathbf{x}_{1}\right)}=f_{\mathbf{Y}}\left(\mathbf{x}_{2}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \mathbf{x}_{1}\right)$

$\d=(2 \pi)^{-(k-1) / 2}\left(\operatorname{det}\left(\Sigma_{22}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \Sigma_{12}\right)\right)^{-1 / 2} \exp \left(-\frac{\left(*\right)^{\prime}\left(\Sigma_{22}-\Sigma_{12}^{\prime} \Sigma_{11}^{-1} \Sigma_{12}\right)^{-1}(*)}{2}\right)$

$(*) = \mathbf{x}_{2}-\ub{\left(\mu_{2}+\Sigma_{12}^{\prime} \Sigma_{11}^{-1}\left(\mathbf{x}_{1}-\mu_{1}\right)\right)}_{\t{called the regression of }X_2 \t{ on } X_1}$

Def. Monte Carlo Estimation

$P_{\mathbf{X}}(A)$ $P_{\mathbf{X}}(A)$ .

$\mathbf{X} \sim P_{\mathbf{X}}$ $I_{A}(\mathbf{X}) \sim \operatorname{Bernoulli}\left(P_{\mathbf{X}}(A)\right)$ $\mathbf{X}_{1}, \ldots, \mathbf{X}_{n} \sim P_{\mathbf{X}}$ $P_{\mathbf{X}}(A)$ $\d\hat{P}_{\mathbf{X}}(A)=\frac{1}{n} \sum_{i=1}^{n} I_{A}\left(\mathbf{X}_{i}\right)$ $\sqrt{\f1n\hat{P}_{\mathbf{X}}(A)\left(1-\hat{P}_{\mathbf{X}}(A)\right)}$

$\left[\hat{P}_{\mathbf{X}}(A)\pm 3SE\right]$ $P_{\mathbf{X}}(A)$ $n$ is large enough

R practice

$\Sigma=\left(\begin{array}{lll} 21 & 26 & 24 \\ 26 & 34 & 30 \\ 24 & 30 & 36 \end{array}\right)$

$\Sigma^{1 / 2}$ $\Sigma=\Sigma^{1 / 2} \Sigma^{1 / 2}$ numerically (up to small rounding errors).


1
Sigma = array(c(21, 26, 24, 26, 34, 30, 24, 30, 36), dim = c(3,3))
2
e = eigen(Sigma) # returns eigen values & eigen vectors of Sigma
3
Q = e$vectors
4
Lambda = diag(e$values)
5
# matrix mult operator: %*% 
6
Sigmasqrt = Q %*% sqrt(Lambda) %*% t(Q) # t(Q) is Q-transpose
7
Sigmasqrt %*% Sigmasqrt # should return Sigma

$R$ $\Sigma=R^{\prime} R$ numerically (up to small rounding errors).


x
1
R = chol(Sigma)
2
t(R) %*% R # should return Sigma
3

4
Q = Sigmasqrt %*% solve(R) # solve(R) is R-inverse
5
t(R) %*% t(Q) %*% Q %*% R # should also return Sigma

$\boldsymbol{\mu}=(0,1,2)^{\prime}$ .

$\mathrm{R}$ $\mathbf{X}=\mu+\Sigma^{1 / 2} \mathbf{Z}$ $\mathbf{Z} \sim N_{3}(\mathbf{0}, I)$ $n=10^{3}$ $N_{3}(\mu, \Sigma)$ $P(\|\mathbf{X}\| \leq 10)$ and provide the interval containing the exact value with virtual certainty.


xxxxxxxxxx
20
1
mu = array(c(0, 1, 2), dim = c(1, 3))
2
one = array(1+0*(1:1000), dim = c(1000, 1)) # create a column vec of 1000 1's
3
Mu = one %*% mu # create a 1000x3 matrix (each row = mu)
4
sample = rnorm(3000, 0, 1) # generate 3000 N(0, 1) values
5

6
# Use sqrt(Sigma)
7
samplevec = Mu + array(sample, dim=c(1000,3)) %*% Sigmasqrt # create 1000 rows of 3-dim sample vectors from N_3(mu, Sigma) distribution
8

9
one = array(c(1, 1, 1), dim = c(3, 1)) # create a column vec of 3 1's
10
length = sqrt((samplevec*samplevec) %*% one) # square each element in samplevec, sum the squares, and take square root
11
count = 0 # count how many lengths are <= 10
12
for (i in 1:1000) {
13
    if(length[i] <= 10){
14
        count = count + 1
15
    }
16
}
17
prop = count/1000 # get estimated prob
18
error = sqrt(prop * (1-prop) / 1000)
19
low = prop - 3*error
20
high = prop + 3*error

$\mathrm{R}$ $\mathbf{X}=\boldsymbol{\mu}+R^{\prime} \mathbf{Z}$ $\mathbf{Z} \sim N_{3}(\mathbf{0}, I)$ $n=10^{3}$ $N_{3}(\boldsymbol{\mu}, \Sigma)$ $P(\|\mathbf{X}\| \leq 10)$ and provide the interval containing the exact value with virtual certainty.


xxxxxxxxxx
2
1
# Using R', samplevec is now the following (everything else same)
2
samplevec = Mu + array(sample, dim=c(1000,3)) %*% R # create 1000 rows of 3-dim sample vectors from N_3(mu, Sigma) distribution

Results are very similar. Part a has error = 0.0146253205093085, part b has error = 0.0149256490646136.

Ex. $\mathbf{X} \sim N_{2}\left(\left(\begin{array}{l} 1 \\ 2 \end{array}\right),\left(\begin{array}{ll} 5 / 2 & 1 / 2 \\ 1 / 2 & 5 / 2 \end{array}\right)\right)$

$X_{2} \mid X_{1}=2$ .

$X_2|X_1 = 2 \sim N$\mu_2+\f{\sigma_{12}}{\sigma_{11}}(2-\mu_1), \sigma_{22}-\f{\sigma_{12}^2}{\sigma_{11}}$$

$A=\left\{\left(x_{1}, x_{2}\right): x_{1}^{2}+x_{2}^{2} \leq 5\right\}$ .

$x_2^2 \le 5 - 2^2 = 1P_{X_{2} \mid X_{1}}\left(A \mid x_{1}\right)=P_{X_{2} \mid X_{1}}\left(\{x_{2}: x_{2}^{2} \leq 1\} \st 2\right) =P_{X_{2} \mid X_1}\left(-1 \leq x_{2} \leq 1 \mid 2\right)$ $=P_{X_{2} \mid X_{1}}\left(\frac{-1-2.2}{\sqrt{2.4}} \leq \frac{x_{2}-2.2}{\sqrt{2.4}} \leq \frac{1-2.2}{\sqrt{2.4}}\right)=P (-2.066 \leq z \leq -0.775)$ $Z\sim N(0,1)$ $=\Phi(-0.775)-\Phi(-2.066)=0.120$

$A$ .

$P_{(X_1, X_2)}(A)$ directly so we proceed via Monte Carlo:


xxxxxxxxxx
6
1
mu = c(1, 2)
2
Sigma = array(c(5/2, 1/2, 1/2, 5/2), dim = c(2, 2))
3
library(MASS) # installs library containing mvrnorm 
4
X = mvrnorm(1000, mu, Sigma) # generates a sample of n=1000 from N_2(mu, Sigma)
5
y = X[, 1]**2 + X[, 2]**2
6
mean(y\le 5) # returns the estimate, can increase n for higher accuracy

3. Expectation

Lecture 13

Properties of Indicator Functions

$A\in \sa$ $I_{A}(\omega)= \begin{cases}1 & \text { if } \omega \in A \\ 0 & \text { if } \omega \in A^{c}\end{cases}$

$\begin{aligned} I_{A^{c}}(\omega) &=1-I_{A}(\omega) \\ I_{\cap_{i=1}^{n} A_{i}}&=\prod_{i=1}^{n} I_{A_{i}} \\ I_{\cup_{i=1}^{n} A_{i}} &=1-\prod_{i=1}^{n} I_{A_{i}^{c}} \\&=1-\prod_{i=1}^{n}\left(1-I_{A_{i}}\right) \\ &=\sum_{i=1}^{n} I_{A_{i}}-\sum_{i<j} I_{A_{i}} I_{A_{j}}+\cdots+(-1)^{n+1} \prod_{i=1}^{n} I_{A_{i}} \\ &=\sum_{i=1}^{n} I_{A_{i}}-\sum_{i<j} I_{A_{i} \cap A_{j}}+\cdots+(-1)^{n+1} I_{\cap_{i=1}^{n} A_{i}} \end{aligned}$

Def. Simple Function

$A_{1}, \ldots, A_{l} \in \mathcal{A}$ $a_{1}, \ldots, a_{l} \in \R^{1}$ $X: \Omega \rightarrow R^{1}$ $X(\omega)=\d\sum_{i=1}^{l} a_{i} I_{A_{i}}(\omega)$ is a simple function.

A simple function must be a r.v. that takes only finitely many values.

Note $X_{1}, X_{2}$ $X_{1} X_{2}$ $a_{0}+a_{1} X_{1}+a_{2} X_{2}$ $a_{0}, a_{1}, a_{2}$ , since it is also a r.v. that takes finitely many values

Ex. Prove that any r.v. that takes only finitely many values is a simple function.

$X$ $c_1, ..., c_m$ .

$\{c_i\} \in \sb^1$ $X$ $C_i = X\inv\{c_i\} \in \sa$ .

$\d X = \sum_{i=1}^m c_i I_{C_i}$ which takes the form of a simple function.

Def. Canonical Form

$X(\omega)=\d\sum_{i=1}^{m} c_{i} I_{C_{i}}(\omega)$ $C_{i}=X^{-1}\left\{c_{i}\right\} \in \mathcal{A}$ $c_{1}, \ldots, c_{m} \in R^{1}$ distinct $X$ $\d\bigcup_{i=1}^{m} C_{i}=\Omega$ $i \neq j$ $C_{i} \cap C_{j}=\phi$ i.e. are mutually disjoint.

Note $\d\sum_{j=1}^{l} a_{j} P\left(A_{j}\right)=\sum_{j=1}^{m} c_{j} P\left(C_{j}\right)$

Proof $X$ $p_{X}(x)=P_{X}(\{x\})=P\left(X^{-1}\{x\}\right)= \begin{cases}0 & x \notin\left\{c_{1}, \ldots, c_{m}\right\} \\ P\left(C_{i}\right) & x=c_{i}\end{cases}$

$\omega_{1}, \ldots, \omega_{n}$ $P$ $n\to\infty$ ,

$\ba\frac{1}{n} \sum_{i=1}^{n} X\left(\omega_{i}\right) =&\frac{1}{n} \sum_{i=1}^{n} \sum_{j=1}^l a_{j} I_{A_{j}}\left(\omega_{i}\right)=\sum_{j=1}^l a_{j}\left(\frac{1}{n} \sum_{i=1}^{n} I_{A_{j}}\left(\omega_{i}\right)\right) \rightarrow \sum_{j=1}^l a_{j} P\left(A_{j}\right) \\ =&\frac{1}{n} \sum_{i=1}^{n} \sum_{j=1}^{m} c_{j} I_{C_{j}}\left(\omega_{i}\right)=\sum_{j=1}^{m} c_{j}\left(\frac{1}{n} \sum_{i=1}^{n} I_{C_{j}}\left(\omega_{i}\right)\right) \rightarrow \sum_{j=1}^{m} c_{j} P\left(C_{j}\right)\ea$

Def. Expectation of a Simple Function

$\d X=\sum_{i=1}^{l} a_{i} I_{A_{i}}$ $X$ $E(X)=\d\sum_{i=1}^{l} a_{i} P\left(A_{i}\right)$

Prop 3.1.1 (Expectation Properties)

$X_{1}, X_{2}$ are simple functions, then

$E\left(a_{0}+a_{1} X_{1}+a_{2} X_{2}\right)=a_{0}+a_{1} E\left(X_{1}\right)+a_{2} E\left(X_{2}\right)$

Proof $X_1 = \d\sum_{i=1}^m b_i I_{B_i}$ $X_2 = \sumto n c_iI_{C_i}$
$a+0 + a_1X_1 + a_2X_2 = a_0I_\Omega + \d\sum_{i=1}^m a_1b_i I_{B_i} + \sumto n a_2c_iI_{C_i}$
$a_{0}+a_{1} X_{1}+a_{2} X_{2}$ is a simple function, and by definition we have
$\ba E\left(a_{0}+a_{1} X_{1}+a_{2} X_{2}\right)&=a_{0} P(\Omega)+\d\sum_{i=1}^{m} a_{1} b_{i} P\left(B_{i}\right) + \sumto n a_2c_iP(C_i)\\&=a_{0}+a_{1} \sum_{i=1}^{m} b_{i} P\left(B_{i}\right)+a_{2} \sum_{i=1}^{n}c_{i} P\left(C_{i}\right)\\ &=a_{0}+a_{1} E\left(X_{1}\right)+a_{2} E\left(X_{1}\right)\ea$

$X_{1} \leq X_{2}$ $E\left(X_{1}\right) \leq E\left(X_{2}\right)$

Proof $X_{2}-X_{1}$ is a nonnegative simple function, distinct values taken are nonnegative.
$E\left(X_{2}-X_{1}\right)=E\left(X_{2}\right)-E\left(X_{1}\right)\ge0$

$P\left(\left\{\omega: X_{1}(\omega) \neq X_{2}(\omega)\right\}\right)=0$ $E\left(X_{1}\right)=E\left(X_{2}\right)$

Proof $X_{1}=\d\sum_{i=1}^{l} a_{i} I_{A_{i}}, X_{2}=\sum_{i=1}^{m} b_{i} I_{B_{i}}$ are in canonical form.
$P\left(A_{j}\right)=0$ $E\d\left(X_{1}\right)=\sum_{i=1}^{l} a_{i} P\left(A_{i}\right)=\sum_{i \neq j} a_{i} P\left(A_{i}\right)$
$P\left(B_{j}\right)= 0$ $E\d\left(X_{2}\right)=\sum_{i=1}^{l} b_{i} P\left(B_{i}\right)=\sum_{i \neq j} b_{i} P\left(B_{i}\right)$ , i.e. sets with probability 0 do not change the sum.
$P\left(A_{i}\right)>0, P\left(B_{j}\right)>0$ $i, j$ .
$a_{i}$ $b_{j}$ $a_{i}=b_{j}$ $A_{i}$ $B_{j}$ $P\left(A_{i} \cap B_{j}^{c}\right)=P\left(A_{i}^{c} \cap B_{j}\right)=0$ $P\left(A_{i}\right)=P\left(B_{j}\right)$ .

Motivation (for definition of expectation of a general r.v. X)

$X$ $i \in\{1, \ldots, n\}, j \in\left\{1, \ldots, 2^{n}\right\}$ , define a nonnegative simple function

$X_{n} =\d\sum_{i=1}^{n} \sum_{j=1}^{2^{n}}\left((i-1)+{(j-1) \o 2^{n}}\right) I_{A_{i, j, n}}$ $A_{i, j, n} =\d\left\{\omega:(i-1)+{(j-1)\o 2^{n}} \leq X(\omega)<(i-1)+{j\o 2^{n}}\right\} \in \mathcal{A}$

$X_n$ $X(\omega)$ $X_{n}(\omega) \leq X(\omega)$

$n \leq n^{\prime}$ $X_{n}(\omega) \leq X_{n^{\prime}}(\omega)$ $X_n$ $\d\lim _{n \rightarrow \infty} X_{n}(\omega)=X(\omega)$ $\omega \in \Omega$

$E\left(X_{n}\right)$ $\d\lim _{n \rightarrow \infty} E\left(X_{n}\right) = E(X)$ .

$X$ is a r.v. and define

$\begin{aligned} &X_{+}(\omega)=\max \{0, X(\omega)\} \text { the positive part of } X \\ &X_{-}(\omega)=\max \{0,-X(\omega)\} \text { the negative part of } X \end{aligned}$ $X=X_{+}-X_{-}$

$B$ $X_+$ $X_-$ are non-negative r.v.'s, since

$X_{+}^{-1} B=\left\{\begin{array}{ll} X^{-1}(B \cap(0, \infty)) & \text { if } 0 \notin B \\ X^{-1}(B \cap(0, \infty)) \cup \ub{X^{-1}(-\infty, 0]}_\t{mapped to 0 by $X_+$} & \text { if } 0 \in B \end{array}\right.$

$X_{-}^{-1} B=\left\{\begin{array}{ll} X^{-1}(B \cap(-\infty, 0)) & \text { if } 0 \notin B \\ X^{-1}(B \cap(-\infty, 0)) \cup \ub{X^{-1}(0, \infty]}_\t{mapped to 0 by $X_-$} & \text { if } 0 \in B \end{array}\right.$

Def. Expectation (as a Sum of Positive r.v.'s)

$X$ $X$ $E(X)=E\left(X_{+}\right)-E\left(X_{-}\right)$ $E\left(X_{+}\right), E\left(X_{-}\right)$ $E(X)$ $E\left(X_{+}\right), E\left(X_{-}\right)$ $E(X)$ $E(X_+)$ $-E(X_-)$ .

Lecture 14

Lemma 3.1.2

$Y, Z$ are nonnegative r.v.'s,

$a, b \geq 0$ $E(a Y+b Z)=a E(Y)+b E(Z)$

$Y \leq Z$ $0 \leq E(Y) \leq E(Z)$

Proof $Y_{n} \uparrow Y, Z_{n} \uparrow Z$ .

$a Y_{n}+b Z_{n}$ $a Y_{n}+b Z_{n} \uparrow a Y+b Z$ $\begin{aligned} E(a Y+b Z) &=\lim _{n} E\left(a Y_{n}+b Z_{n}\right)=a \lim _{n} E\left(Y_{n}\right)+b \lim _{n} E\left(Z_{n}\right) =a E(Y)+b E(Z) \end{aligned}$

$Y \leq Z$ $0 \leq Y_{n} \leq \max \left\{Y_{n}, Z_{n}\right\}$ $\max \left\{Y_{n}, Z_{n}\right\}$ $\max \left\{Y_{n}, Z_{n}\right\} \uparrow Z$ $0 \leq E\left(Y_{n}\right) \leq E\left(\max \left\{Y_{n}, Z_{n}\right\}\right)$ $E\left(Y_{n}\right) \rightarrow E(Y), E\left(\max \left\{Y_{n}, Z_{n}\right\}\right) \rightarrow E(Z)$

Lemma 3.1.3

$Y, Z$ $E(Y), E(Z)$ $E(Y-Z)=E(Y)-E(Z)$ .

Proof $Y-Z = X =X_{+}-X_{-}$

\begin{aligned} Y - Z & = X_{+} - X_{-} \\ Y + X_{-} & = X_{+} + Z \leftarrow LHS and RHS both non-negative \\ E (Y + X_{-}) & = E (Y) + E (X_{-}) \leftarrow E(LHS) \\ E (X_{+} + Z) & = E (X_{+}) + E (Z) \leftarrow E(RHS) \end{aligned}

$X_{+}(\omega)>0$ $X_-(\omega) = 0 \implies X_{+}(\omega)=Y(\omega)-Z(\omega)$ $Z(\omega)$ $X_+(\omega) \leq Y(\omega)+Z(\omega) \fa \omega$ $0 \leq E\left(X_{+}\right) \leq E(Y)+E(Z)<\infty$

$X_-(\omega) > 0$ $X_+(\omega) = 0 \implies X_-(\omega) = Z(\omega) - Y(\omega)$ $Y(\omega)$ $X_-(\omega) \le Z(\omega) + Y(\omega) \fa \omega$ $0 \le E(X_-) \le E(Z) + E(Y)<\infty$ .

$0 \le E(X) < \infty$ and

\begin{aligned} E (Y - Z) & = E (X) = E (X_{+}) - E (X_{-}) \\ = \underset{⏟}{E (X_{+} + Z) - E (Z)} - \underset{⏟}{E (Y + X_{-}) + E (Y)} \\ = E (Y) - E (Z) \end{aligned}

Prop 3.1.4 (Linearity of Expectations)

$Y, Z$ $E(Y), E(Z)$ $E(a Y+b Z)=a E(Y)+b E(Z)$

Proof We can decompose Y and Z and express them as a difference of 2 non-negative r.v.'s as in Lemma 3.1.3

\begin{aligned} a Y + b Z & = a (Y_{+} - Y_{-}) + b (Z_{+} - Z_{-}) \\ = {\begin{cases} (a Y_{+} + b Z_{+}) - (a Y_{-} + b Z_{-}) & if a, b \geq 0 or \\ (- a Y_{-} - b Z_{-}) - (- a Y_{+} - b Z_{+}) & if a < 0, b < 0 \\ (a Y_{+} - b Z_{-}) - (a Y_{-} - b Z_{+}) & if a \geq 0, b < 0 \\ (- a Y_{-} + b Z_{+}) - (- a Y_{+ -} + b Z_{-}) & if a < 0, b \geq 0 \end{cases} \end{aligned}

Def. St Petersburg Paradox

$\$2^i$

$\Omega=(0, \infty), \mathcal{A}=\mathcal{B}^{1} \cap(0, \infty)$
$X: \Omega \rightarrow \R^{1}$ $X(\omega)=2^{\lceil\omega\rceil}$ $X^{-1}\left\{2^{i}\right\}=(i-1, i]$ $X^{-1}\{x\}=\phi$ $x$ is not a positive integer power of 2
$P$ $P(\{i\})=(\f12)^{i}$ $(\f12)$
$X_{n}=X I_{(0, n]} \implies X_{n} \uparrow X$ $\d E\left(X_{n}\right)=\sum_{i=1}^{n} 2^{i} (\f12)^i=n \implies E(X) = \infty$
$\$ \infty$
$Y=Z=X$ $E(Y-Z)=0$ $0 \ne E(Y)-E(Z) = \infty - \infty =$ undefined

Prop 3.1.5 (Expectation of |X|)

$E(|X|)=E\left(X_{+}\right)+E\left(X_{-}\right)$

Proof $|X|=X_{+}+X_{-}$ .

$X \leq Y$ $E(X) \leq E(Y)$ .

Proof

$E(Y)=\infty$ $E(X)=-\infty$ $E\left(X_{-}\right)<\infty$ $E\left(Y_{+}\right)<\infty$ .
$X\le Y \implies X=\ub{X_{+}-X_{-}} \leq Y=Y_{+}-Y_{-} \leq \ub{Y_{+}}$ $X_{+} \leq Y_{+}+X_{-} < \infty$
$E(X_+) < \infty$
$-Y\le -X \implies -Y=\ub{Y_{-}-Y_{+}} \leq-X=X_{-}-X_{+} \leq \ub{X_{-}}$ $Y_{-} \leq X_{-}+Y_{+}$
$E(Y_+) < \infty$
$E(X), E(Y)$ $0 \leq Y-X$ $0 \leq E(Y-X)=E(Y)-E(X)$ which proves the result.

$P(X=0)=1$ $E(X)=0$ $P(X=Y)=P(\{\omega: X(\omega)=Y(\omega)\})=1$ $E(X)=E(Y)$ .

Proof

1st line $X \geq 0$ $X_{n} \uparrow X$ $0 \leq X_{n} \leq X$ $P(X = 0) \implies P\left(X_{n}=0\right)=1 \implies E\left(X_{n}\right)=0 \implies E(X)=0$ .

$X=X_{+}-X_{-}$ $\{\omega: X(\omega)=0\} \subset\left\{\omega: X_{+}(\omega)=0\right\}$ $P\left(X_{+}=0\right)=1 \implies E\left(X_{+}\right)=0$ $\{\omega: X(\omega)=0\} \subset\left\{\omega: X_{-}(\omega)=0\right\}$ $P\left(X_{-}=0\right)=1 \implies E\left(X_{-}\right)=0$ $E(X) = 0$

2nd line $P(X=Y)=1$ , then

$P(\{\omega: X(\omega)>0\})=P(\{\omega: Y(\omega)>0\}) \implies P\left(X_{+}=Y_{+}\right)=1 \implies E\left(X_{+}\right)=E\left(Y_{+}\right)$ $P(\{\omega: X(\omega)<0\})=P(\{\omega: Y(\omega)<0\}) \implies P\left(X_{-}=Y_{-}\right)=1 \implies E\left(X_{-}\right)=E\left(Y_{-}\right) \implies E(X) = E(Y)$

Lecture 15

Def. Converge with Probability 1

$\boxed{X_{n} \stackrel{w p 1}{\rightarrow}X}$ $\left\{X_{n}\right\}$ with probability 1 $X$ $P\left(\left\{\d\omega: \lim _{n \rightarrow \infty} X_{n}(\omega)=X(\omega)\right\}\right)=1$

Note We can assign a probability measure to the set since it is a sigma algebra:

$\ba \{\omega: \d\lim_{n\to\infty} X_n(\omega) = X(\omega)\} &= \N_{m=1}^\infty \liminf_{n\to\infty} \{\omega: |X_n(\omega) - X(\omega)| < \f1m\} \\ &= \N_{m=1}^\infty \U_{n=1}^\infty \N_{i=n}^\infty \{\omega: |X_n(\omega) - X(\omega)| < \f1m\} \in \sa \ea$

E.g. $(\Omega, \mathcal{A}, P)=\left(\R^{1}, \mathcal{B}^{1}, P\right)$ $P$ $[0,1]$ $\d P(B)=\int_{B \cap[0,1]} d x$ .
$X_{n}(\omega)=\frac{n}{n+1} \omega^{2}$ $X(\omega)=\omega^{2}$ $\d\left\{\omega: \lim _{n \rightarrow \infty} X_{n}(\omega)=X(\omega)\right\}=\R^{1}$
$P\left(\R^{1}\right)=\int_{[0,1]} d x=1$ $X_{n} \stackrel{w p 1}{\rightarrow} X$

E.g. $X_{*}(\omega)= \begin{cases}\omega^{2} & \text { if } \omega \neq 1 / 2 \\ 1 & \text { if } \omega=1 / 2\end{cases}$ $\left\{\omega: \d\lim _{n \rightarrow \infty} X_{n}(\omega)=X_{*}(\omega)\right\}=\R^{1} \setminus\{1 / 2\}$
$P\left(\R^{1} \backslash\{\frac12\}\right)=\int_{[0,\frac12)} d x+\int_{(\frac12,1]} d x=\frac12+\frac12=1$ $X_{n} \stackrel{w p 1}{\rightarrow} X_{*}$
$X$ $q \in Q$ $X_{* *}$ $P(\mathbf{Q})=0$ $X_{n} \stackrel{w p 1}{\rightarrow} X_{* *}$

Def. Converge Almost Surely

$v$ $(\Omega, \mathcal{A})$ $v: \mathcal{A} \rightarrow[0, \infty]$ that satisfies the following:

$v(\phi)=0$
$v\d\left(\U_{i=1}^{\infty} A_{i}\right)=\sum_{i=1}^{\infty} v\left(A_{i}\right)$ $A_{1}, A_{2}, \ldots \in \mathcal{A}$ are mutually disjoint

$h:(\Omega, \mathcal{A}) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $h: \Omega \rightarrow R^{1},\ h^{-1} B \in \mathcal{A}$ $B \in \mathcal{B}^{1}$

$h$ $v$ $\d\int_{\Omega} h(\omega) v(d \omega)$ $h$ $v$

$X$ $E(X)=\d\int_{\Omega} X(\omega) P(d \omega)$ $X$ $P$
$v$ $(\Omega, \mathcal{A})$ $\Omega$ $\d\int_{\Omega} h(\omega) v(d \omega)=\sum_{\omega \in \Omega} h(\omega)$
$v$ $\left(R^{k}, \mathcal{B}^{k}\right)$ $\d\int_{\Omega} h(\omega) v(d \omega)=\int_{R^{k}} h(\mathbf{x}) d \mathbf{x}$

$\left\{h_{n}\right\}$ $v\d\left(\left\{\omega: \lim _{n \rightarrow \infty} h_{n}(\omega) \neq h(\omega)\right\}\right)=0$ $v$ $h$ $h_{n} \stackrel{a . s .}{\rightarrow} h$

Note $P$ $h$ $\iff$ convergence with probability 1

Prop 3.2.1 (MCT & DCT)

$h_{n} \stackrel{\text { a.s. } v}{\rightarrow} h$
(i) Monotone Convergence (MCT)
$0 \leq h_{1} \leq h_{2} \leq \cdots$ $\d\int_{\Omega} h_{n}(\omega) v(d \omega) \uparrow \int_{\Omega} h(\omega) v(d \omega)$
(ii) Dominated Convergence (DCT)
$g:(\Omega, \mathcal{A}) \rightarrow\left(\R^{1}, \mathcal{B}^{1}\right)$ $\int_{\Omega}|g(\omega)| v(d \omega)<\infty$ $\left|h_{n}\right| \leq|g|$ $n$ ,
$\int_{\Omega} h_{n}(\omega) \nu(d \omega) \rightarrow \int_{\Omega} h(\omega) \nu(d \omega)$

Corollary (Applied to Expectations)

$X_{n} \stackrel{w p 1}{\rightarrow} X$

$0 \leq X_{1} \leq X_{2} \leq \cdots$ $E\left(X_{n}\right) \uparrow E(X)$ .

$Y$ $E(|Y|)<\infty$ $\left|X_{n}\right| \leq|Y|$ $n$ $E\left(X_{n}\right) \rightarrow E(X)$ .

E.g. $X$ $E(X) < \infty$ $X_{n}=X I_{\{|X| \leq n\}}$ $X_{n} \stackrel{w p 1}{\rightarrow} X$ $\left|X_{n}\right| \leq|X|$ $E\left(X_{n}\right) \rightarrow E(X)$

Prop 3.3.2 (Expectation of Compositions)

$X$ $(\Omega, \mathcal{A}, P)$ $h:\left(R^{1}, \mathcal{B}^{1}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $Y = h\circ X = h(X)$ , then

$Y=h(X)$ $(\Omega, \mathcal{A}, P)$

Proof $B \in \mathcal{B}^{1}$ $Y^{-1} B =\{\omega: Y(\omega) \in B\} =\{\omega: h(X(\omega)) \in B\} =\left\{\omega: X(\omega) \in h^{-1} B\right\} =X^{-1} h^{-1} B \in \mathcal{A}$ $h^{-1} B \in \mathcal{B}^{1}$ $X$ is a r.v.

$E(Y)=E_{P_{X}}(h)$ , if it exists.

Proof Steps: simple h -> non-negative h -> general h

$h=\d\sum_{i=1}^{k} b_{i} I_{B_{i}}$ $Y(\omega)=h(X(\omega))=\d\sum_{i=1}^{k} b_{i} I_{B_{i}}(X(\omega))=\sum_{i=1}^{k} b_{i} I_{X^{-1} B_{i}}(\omega)$ $E_P(Y)=\d\sum_{i=1}^{k} b_{i} \ub{P\left(X^{-1} B_{i}\right)}_{P \t{ on } \Omega}=\sum_{i=1}^{k} b_{i} \ub{P_{X}\left(B_{i}\right)}_{P \t{ on } \R}=E_{P_{X}}(h) \expl{expectation defined using simple functions}$

$h \geq 0$ $Y=h(X) \geq 0$ $W_{n} \uparrow h \implies W_{n}(X) \uparrow h(X)=Y$ .

$E_{P_{X}}(h)=\d\lim _{n \rightarrow \infty} E_{P_{X}}\left(W_{n}\right)=\lim _{n \rightarrow \infty} E\left(W_{n}(X)\right)=E(Y) \expl{expectation defined as a sum of positive r.v.'s}$

$\ge 0$ $h(X)=h_{+}(X)-h_{-}(X)$ . Applying the above to both parts gives the result.

$E_{P_X}(h_+) = E(Y_+), E_{P_X}(h_-) = E(Y_-) \implies E_{P_X}(h_+) - E_{P_x}(h_-) = E(Y_+) - E(Y_-) = E(Y)$

Prop 3.3.3 (Expectation Formulas)

$X$ $(\Omega, \mathcal{A}, P)$ $h:\left(R^{1}, \mathcal{B}^{1}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $E_{P_{X}}(h)$ exists.
$P_{X}$ $p_{X}$ $\d E_{P_{X}}(h)=\sum_{x \in R^{1}} h(x) p_{X}(x)$ .
$P_{X}$ $f_{X}$ $\d E_{P_{X}}(h)=\int_{-\infty}^{\infty} h(x) f_{X}(x) d x$ .

Proof $h(x)=\sum_{i=1}^{k} b_{i} I_{B_{i}}(x)$ is a simple function in canonical form. Then

\begin{aligned} E_{P_{X}} (h) = \sum_{i = 1}^{k} b_{i} P_{X} (B_{i}) & = {\begin{cases} \sum_{i = 1}^{k} b_{i} \sum_{x \in B_{i}} p_{X} (x), & if X discrete \\ \sum_{i = 1}^{k} b_{i} \int_{B_{i}} f_{X} (x) d x, & if X a.c. \end{cases} \\ = {\begin{cases} \sum_{x \in R^{1}} h (x) p_{X} (x), & if X discrete \\ \int_{- \infty}^{\infty} h (x) f_{X} (x) d x, & if X a.c. \end{cases} \\ = {\begin{cases} \int_{- \infty}^{\infty} h (x) p_{X} (x) v (d x), & v = counting measure \\ \int_{- \infty}^{\infty} h (x) f_{X} (x) v (d x), & v = volume measure \end{cases} \end{aligned}

$h$ $h \geq 0$ $h_{n} \uparrow h$ $h_{n} p_{X} \uparrow h p_{X}$ $h_{n} f_{X} \uparrow h f_{X}$ $h$ $h=h_{+}-h_{-}$ .

E.g. $X \sim N\left(\mu, \sigma^{2}\right)$ $h(x)=x$ we have

\begin{matrix} \begin{aligned} E (X) & = \int_{0}^{\infty} x \frac{1}{\sqrt{2 π} σ} \exp (- \frac{1}{2} {(\frac{x - μ}{σ})}^{2}) d x - \int_{- \infty}^{0} (- x) \frac{1}{\sqrt{2 π} σ} \exp (- \frac{1}{2} {(\frac{x - μ}{σ})}^{2}) d x \\ make the change of variable t = T (x) = \frac{(x - μ)}{σ} so x = T^{- 1} (t) = μ + σ t and J_{T} (x) = σ \\ = \int_{0}^{\infty} (μ + σ t) \frac{1}{\sqrt{2 π}} \exp (- \frac{t^{2}}{2}) d x - \int_{- \infty}^{0} - (μ + σ t) \frac{1}{\sqrt{2 π}} \exp (- \frac{t^{2}}{2}) d x \\ sub in φ (t) = \frac{1}{\sqrt{2 π}} \exp (\frac{- t^{2}}{2}) \\ = \int_{0}^{\infty} (μ + σ t) φ (t) d t + \int_{- \infty}^{0} (μ + σ t) φ (t) d t \leftarrow can recombine and simplify \\ = μ \underset{1}{\underset{⏟}{\int_{- \infty}^{\infty} φ (t) d t}} + σ \underset{0 since for an odd function, f(-x) = -f(x)}{\underset{⏟}{(\int_{0}^{\infty} t φ (t) d t + \int_{- \infty}^{0} t φ (t) d t)}} \leftarrow since \int_{- \infty}^{0} t φ (t) d t = - \int_{0}^{\infty} t φ (t) d t \\ = μ \end{aligned} \end{matrix}

$h(x)=(x-\mu)^{2}$ , we have

\begin{aligned} E ((X - μ)^{2}) & = \int_{- \infty}^{\infty} (x - μ)^{2} \frac{1}{\sqrt{2 π} σ} \exp (- \frac{1}{2} {(\frac{x - μ}{σ})}^{2}) d x \\ = σ^{2} \int_{- \infty}^{\infty} t^{2} φ (t) d t \leftarrow using same change of var as before \\ apply integration by parts with {\begin{cases} u = t ⟹ d u = d t \\ d v = t φ (t) ⟹ v = \int t \frac{1}{\sqrt{2 π}} \exp (\frac{- t^{2}}{2}) d t = - φ (t) \end{cases} \\ \int_{- \infty}^{\infty} t^{2} φ (t) d t = u v - v d u = \underset{contains \exp (- \infty)}{\underset{⏟}{[- t φ (t)]_{- \infty}^{\infty}}} + \int_{- \infty}^{\infty} φ (t) d t = 0 + 1 = 1 \\ = σ^{2} \end{aligned}

Def. Moments

$k$ $X$ $\mu_{k}=E\left(X^{k}\right)$ $k$ $X$ $\bar{\mu}_{k}=E\left(\left(X-\mu_{1}\right)^{k}\right)$ .

$X$ $\mu_{X}=E(X)$ $X$ $\sigma_{X}^{2}=\operatorname{Var}(X)=E\left(\left(X-\mu_{X}\right)^{2}\right)$ $\mu_{X}$ exists. The 3rd moment is the skewness, and the 4th moment is the kurtosis.

Prop 3.3.4 (Finite Moment Property)

$\mu_{k}$ $\mu_{l}$ $l \le k$ , i.e. the previous moments must all be finite.

Proof $\mu_{k}$ $\iff E\left(|X|^{k}\right)$ $h(x)=|x|^l$ , then

\begin{aligned} 0 \leq E (| X |^{l}) = E_{P_{X}} (h) = & \int_{- \infty}^{\infty} | x |^{l} P_{X} (d x) \\ = & \int_{- \infty}^{- 1} | x |^{l} P_{X} (d x) + \underset{(*)}{\underset{⏟}{\int_{- 1}^{1} | x |^{l} P_{X} (d x)}} + \int_{1}^{\infty} | x |^{l} P_{X} (d x) \\ to get the following line, use the fact that l \leq k for the 1st and 3rd term above \\ for the 2nd term (*): powers of a number in [-1, 1] must be smaller than 1 \\ \leq & \underset{(1)}{\underset{⏟}{\int_{- \infty}^{- 1} | x |^{k} P_{X} (d x)}} + \int_{- 1}^{1} 1 P_{X} (d x) + \underset{(2)}{\underset{⏟}{\int_{1}^{\infty} | x |^{k} P_{X} (d x)}} \\ add in the original 2nd term (*) \\ \leq & \underset{(1) + (*) + (2)}{\underset{⏟}{\int_{- \infty}^{\infty} | x |^{k} P_{X} (d x)}} + P_{X} ([- 1, 1]) < \infty \end{aligned}

Ex. $X \sim N\left(\mu, \sigma^{2}\right)$ $E\left(X^{3}\right)$ $E\left(X^{4}\right)$

$X \sim N(\mu, \sigma^2)$ $Z = \f{X - \mu}{\sigma} \sim N(0, 1)$

\begin{matrix} \begin{aligned} Z^{3} & = \frac{(X - μ)^{3}}{σ^{3}} = \frac{X^{3} - (\binom{3}{1}) μ X^{2} + (\binom{3}{2}) μ^{2} X - μ^{3}}{σ^{3}} = \frac{X^{3} - 3 μ X^{2} + 3 μ^{2} X - μ^{3}}{σ^{3}} \\ E (Z^{3}) & = \frac{E (X^{3}) - 3 μ E (X^{2}) + 3 μ^{2} E (X) - μ^{3}}{σ^{3}} \\ = \frac{E (X^{3}) - 3 μ (σ^{2} + μ^{2}) + 3 μ^{3} - μ^{3}}{σ^{3}} \\ = \frac{E (X^{3}) - 3 μ σ^{2} - μ^{3}}{σ^{3}} \\ E (Z^{3}) & = \int_{- \infty}^{\infty} z^{3} ϕ (z) d z = 0 \leftarrow odd function \\ 0 & = E (X^{3}) - 3 μ σ^{2} - μ^{3} \\ E (X^{3}) & = 3 μ σ^{2} + μ^{3} \end{aligned} \end{matrix}

\begin{matrix} \begin{aligned} Z^{4} & = \frac{(X - μ)^{4}}{σ^{4}} = \frac{X^{4} - (\binom{4}{3}) μ X^{3} + (\binom{4}{2}) μ^{2} X^{2} - (\binom{4}{1}) μ^{3} X + μ^{4}}{σ^{4}} = \frac{X^{4} - 4 μ X^{3} + 6 μ^{2} X^{2} - 4 μ^{3} X + μ^{4}}{σ^{4}} \\ E (Z^{4}) & = \frac{E (X^{4}) - 4 μ E (X^{3}) + 6 μ^{2} E (X^{2}) - 4 μ^{4} + μ^{4}}{σ^{4}} \\ = \frac{E (X^{4}) - 4 μ (μ^{3} + 3 μ σ^{2}) + 6 μ^{2} (μ^{2} + σ^{2}) - 3 μ^{4}}{σ^{4}} \\ = \frac{E (X^{4}) - 6 μ^{2} σ^{2} - μ^{4}}{σ^{4}} \\ E (Z^{4}) & = \int_{- \infty}^{\infty} z^{4} ϕ (z) d z u = z^{3}, d v = z ϕ (z) ⟹ d u = 3 z^{2}, v = - ϕ (z) \\ = [z^{3} (- ϕ (z))]_{- \infty}^{\infty} - \int_{- \infty}^{\infty} (- ϕ (z)) 3 z^{2} d z \\ = 0 + 3 = 3 \\ 3 σ^{4} & = E (X^{4}) - 6 μ^{2} σ^{2} - μ^{4} \\ E (X^{4}) & = 3 σ^{4} + 6 μ^{2} σ^{2} + μ^{4} \end{aligned} \end{matrix}

Ex. $X \sim$ $X$ $f_{X}(x)=1 / \pi\left(1+x^{2}\right)$ $-\infty<x<\infty$ $\mu$ doesn't exist.

Cauchy dist has longer tails.

Ex. Let X ∼ Geometric(θ), and let Y = min(X, 100).

(a) Compute E(Y).

(b) Compute E(Y − X).

Ex. Geometric & Negative Binomial

E& $X_1,..., X_r$ $X = X_1 +···+ X_r$ ∼ Negative-Binomial(r, θ).)

$(1 − θ) /θ^2$
$((1-\theta)^x)^n = x(x-1)(1-\theta)^{x-2}$

E& $r(1 − θ) /θ^2$ .

Ex. Gamma

E&R 3.2.16 Let α > 0 and λ > 0, and let X ∼ Gamma(α, λ). Prove that E(X) = α/λ.

E& $α/λ^2$ .

Ex. Beta

E&R 3.2.22 Suppose that X follows the Beta(a, b) distribution. Prove that E(X) = a/(a + b).

E& $Var(X) = ab/((a + b)^2(a + b + 1))$

E.g. (Monte Carlo Approximations)

$Y=h(X)$ $h:\left(R^{1}, \mathcal{B}^{1}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $E(Y)$

$P_{Y}$ is easy to work with
$X_{1}, X_{2}, \ldots \stackrel{i . i . d}{\sim} P_{X}$ $Y_{1}, Y_{2}, \ldots \stackrel{i . i . d .}{\sim} P_{Y}$ $E(Y)$ $\d\bar{Y}=\frac{1}{n} \sum_{i=1}^{n} Y_{i}=\frac{1}{n} \sum_{i=1}^{n} h\left(X_{i}\right)$ $E(Y)$ $n \rightarrow \infty$

$n$ ?

$n$ $\d\frac{\bar{Y}-E(Y)}{\sqrt{\operatorname{Var}(Y) / n}} \sim N(0,1)$ $\operatorname{Var}(Y)<\infty$
$\operatorname{Var}(Y)=E\left((Y-E(Y))^{2}\right)=E\left(Y^{2}\right)-(E(Y))^{2}$ $s^{2}=\d\frac{1}{n} \sum_{i=1}^{n} Y_{i}^{2}-\bar{Y}^{2}=\frac{1}{n} \sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}$
$\d\frac{\bar{Y}-E(Y)}{\sqrt{s^{2} / n}} \sim N(0,1)$
$Z \sim N(0,1)$ $P(-3<Z<3)=0.9973002 \approx 1$
$E(Y)$ $[\bar{Y}-3 s / \sqrt{n}, \bar{Y}+3 s / \sqrt{n}]$ with "virtual certainty". And if the interval turns out to be short, then the estimate is accurate.

Note $Y=I_{A}$ $\bar{Y}=$ $A$ $X_{1}, X_{2}, \ldots, X_{n}$ $Y_{i}^{2}=Y_{i}$ $s^{2}=\bar{Y}(1-\bar{Y})$ $P_{X}(A) = E_{P_X}(I_A)$

Lecture 16

Def. Mean Vector

$\mathbf{X} \in R^{k}$ mean vector $\mathbf{X}$ $\mu_{\mathrm{X}} =E(\mathbf{X})=\left(E\left(X_{1}\right), E\left(X_{2}\right), \ldots, E\left(X_{k}\right)\right)^{\prime} =\left(\begin{array}{llll} \mu_{1}, & \mu_{2}, \ldots, & \mu_{k} \end{array}\right)^{\prime}$ $E\left(X_{i}\right)=\mu_{i}$ exists.

Note random matrix $X=\left(X_{i i}\right) \in R^{k \times l}$ $E(X)=\left(E\left(X_{i j}\right)\right)$ $E\left(X_{i j}\right)$ $E(X) \in R^{k \times l}$ $E\left(X_{i j}\right)$ is finite.

Def. Variance Matrix

$E\left(X_{i}\right)$ $\left.E(\mathbf{X}) \in R^{k}\right)$ variance matrix $\mathbf{X}$ $\Sigma_{\mathbf{X}}=\operatorname{Var}(\mathbf{X})$

\begin{matrix} = (\begin{array}{ccc} E ({(X_{1} - μ_{1})}^{2}) & \dots & E ((X_{1} - μ_{1}) (X_{k} - μ_{k})) \\ E ((X_{2} - μ_{2}) (X_{1} - μ_{1})) & \dots & E ((X_{2} - μ_{2}) (X_{k} - μ_{k})) \\ ⋮ & ⋮ & ⋮ \\ E ((X_{k} - μ_{k}) (X_{1} - μ_{1})) & \dots & E ({(X_{k} - μ_{k})}^{2}) \end{array}) \end{matrix}

$E\left(\left(X_{i}-\mu_{i}\right)\left(X_{j}-\mu_{j}\right)\right)$ $i \neq j$ vector form $\Sigma_{\mathbf{X}} =\operatorname{Var}(\mathbf{X}) =E\left(\left(\mathbf{X}-\mu_{\mathbf{X}}\right)\left(\mathbf{X}-\mu_{\mathbf{X}}\right)^{\prime}\right)$

$= \operatorname{Cov}\left(X_{i}, X_{j}\right)=E\left(\left(X_{i}-\mu_{i}\right)\left(X_{j}-\mu_{j}\right)\right)$ $= \operatorname{Cov}\left(X_{i}, X_{i}\right)=\operatorname{Var}\left(X_{i}\right)$ $\Sigma_{\mathbf{X}}=\left(\operatorname{Cov}\left(X_{i}, X_{j}\right)\right)$

$\operatorname{Cov}\left(X_{i}, X_{j}\right)$ $i$ $j$ $\Sigma_{\mathbf{x}} \in R^{k \times k}$ symmetric $\operatorname{Cov}\left(X_{i}, X_{j}\right)=\operatorname{Cov}\left(X_{j}, X_{i}\right)$

Ex. $X$ $E\left(X^{2}\right)<\infty$ $E(X) < \infty$ .

$X$ $Y$ $E\left(X^{2}\right)<\infty, E\left(Y^{2}\right)<\infty$ $E(X Y)$ is finite.

$E\left(X_{i}^{2}\right)<\infty$ $i=1, \ldots, k$ $\Sigma_{\mathrm{x}} \in R^{k \times k}$ .

Ex. $X$ $Y$ $E\left(X^{2}\right)<\infty, E\left(Y^{2}\right)<\infty$ $\operatorname{Cov}(X, Y)=E(X Y)-E(X) E(Y)$ $\mathbf{X}$ $\Sigma_{\mathbf{X}}=\operatorname{Var}(\mathbf{X})=E\left(\mathbf{X} \mathbf{X}^{\prime}\right)-\mu_{\mathbf{X}} \mu_{\mathbf{X}}^{\prime}$

Prop 3.4.1 (Affine Transformations' Mean & Variance)

$\mathbf{X} \in R^{k}$ $\mathbf{Y}=\mathbf{a}+C \mathbf{X}$ $\mathbf{a} \in R^l, C \in R^{l \times k}$ are constant.

$\mu_{\b{X}} \in R^{k}$ $\mu_{\b{Y}}=\b{a}+C \mu_{\b{X}} \in R^l$

Proof $\mu_{\mathbf{Y}}=E(\mathbf{Y})=E(\mathbf{a}+C \mathbf{X})=\mathbf{a}+C E(\mathbf{X})$ $E\left(a_{i}+\d\sum_{j=1}^{k} c_{i j} X_{j}\right)=a_{i}+\d\sum_{j=1}^{k} c_{i j} E\left(X_{j}\right)$ $E$

$\Sigma_{\mathbf{X}} \in R^{k \times k}$ $\Sigma_{\mathbf{Y}}=C \Sigma_{\mathbf{X}} C^{\prime} \in R^{l \times l}$ since

\begin{aligned} Σ_{Y} = Var (Y) & = E ((Y - μ_{Y}) {(Y - μ_{Y})}^{'}) \\ = E ((a + C X - (a + C μ_{X})) {(a + C X - (a + C μ_{X}))}^{'}) \\ = E (C (X - μ_{X}) {(X - μ_{X})}^{'} C^{'}) \\ = C E {((X - μ_{X}) (X - μ_{X}))}^{'} C^{'} \\ = C Σ_{X} C^{'} \end{aligned}

Prop 3.4.2 (Degenerate Dist, p.s.d. Variance Matrix, Affine Plane)

$X$ $\operatorname{Var}(X)=0$ $P\left(X=\mu_{X}\right)=1$ $X$ $\mu_\b X$ ).

Proof (Repeat the below for simple functions -> non-negative functions -> general functions)
$\operatorname{Var}(X)=E\left(\left(X-\mu_{X}\right)^{2}\right)=0$ $1=P\left(\left(X-\mu_{X}\right)^{2}=0\right)=P\left(X-\mu_{X}=0\right)=P\left(X=\mu_{X}\right)$ .

$\mathbf{X} \in R^{k}$ $\Sigma_{\mathrm{X}} \in R^{k \times k}$ $\mathbf{c} \in R^{k}$ $\mathbf{c}^{\prime} \Sigma_{\mathbf{X}} \mathbf{c} \geq 0$ . Thus, any variance matrix is positive semidefinite (p.s.d.)

Proof $Y=\mathbf{c}^{\prime} \mathbf{X}$ .
$\operatorname{Var}(Y)=\mathbf{c}^{\prime} \Sigma_{\mathbf{X}} \mathbf{c} \geq 0$ since a variance is always nonnegative.

$\mathbf{c}^{\prime} \Sigma_{\mathbf{X}} \mathbf{c}=0$ $\mathbf{c} \neq \mathbf{0}$ $\mathbf{X}$ $\mu_{\mathrm{X}}+L^{\perp}\{\mathbf{c}\}$ $L^\perp(\b c) = \{a:a\perp c\} = \{a: a' c = 0\}$

Proof $Y=\mathbf{c}^{\prime} \mathbf{X}$ $\mathbf{c}^{\prime} \Sigma_{\mathbf{X}} \mathbf{c}=0$ , then Var(X) = 0, so by (i) and (ii),
$\begin{aligned} 1 =P\left(Y=\mu_{Y}\right)=P\left(\mathbf{c}^{\prime} \mathbf{X}=\mathbf{c}^{\prime} \mu_{\mathbf{X}}\right)=P\left(\mathbf{c}^{\prime}\left(\mathbf{X}-\mu_{\mathbf{X}}\right)=0\right) =P\left(\mathbf{X}-\mu_{\mathbf{X}} \in L^{\perp}\{\mathbf{c}\}\right)=P_{\mathbf{X}}\left(\mu_{\mathbf{X}}+L^{\perp}\{\mathbf{c}\}\right) \end{aligned}$

Notes

$\Sigma_{\mathrm{x}} \in R^{k \times k}$ $\Sigma_{\mathbf{X}}=Q \Lambda Q^{\prime}$ $Q=\left(\begin{array}{lll}\mathbf{q}_{1} & \cdots & \mathbf{q}_{k}\end{array}\right) \in R^{k \times k}$ $\Lambda=\operatorname{diag}\left(\lambda_{1}, \ldots, \lambda_{k}\right)$ $\lambda_{1} \geq \lambda_{2} \geq \cdots \geq \lambda_{k} \geq 0$
$\mathbf{c} \in R^{k}$ $\mathbf{c}=Q \mathbf{a}=\d\sum_{i=1}^{k} a_{i} \mathbf{q}_{i}$ $\b c = Q\b e_i = q_i$ gives the ith principal component)
$0 \leq Var(Y) = \d\mathbf{c}^{\prime} \Sigma_{\mathbf{x}} \mathbf{c}=\b a'Q'Q\Lambda Q'Q\b a = \b a'\Lambda \b a = \sum_{i=1}^{k} \lambda_{i} a_{i}^{2}$ $\lambda_{i}>0$ $\mathbf{c}^{\prime} \Sigma_{\mathbf{X}} \mathbf{c}=0$ $a_{i}=0$
$\lambda_{1} \geq \lambda_{2} \geq \cdots \geq \lambda_{l}>0$ $\lambda_{l+1}=\cdots=\lambda_{k}=0$ $\mathbf{c}^{\prime} \Sigma \mathbf{c}=0$ $\mathbf{c} \in L\left\{\mathbf{q}_{l+1}, \ldots, \mathbf{q}_{k}\right\}=L^{\perp}\left\{\mathbf{q}_{1}, \ldots, \mathbf{q}_{l}\right\}$
$\mu_\b X = 0$ $P_{\mathbf{X}}\left(L\left\{\mathbf{q}_{1}, \ldots, \mathbf{q}_{l}\right\}\right)=1$

Ex. $X \in R^{k \times l}$ $E\left(X_{i j}\right)$ $A \in R^{p \times q}, B \in R^{p \times k}, C \in R^{l \times q}$ $E(A+B X C)=A+B E(X) C$

E.g. $\mathrm{X} \sim N_{k}(\boldsymbol{\mu}, \Sigma)$

$\mathbf{Z} \sim N_{k}(\mathbf{0}, I)$ $(2 \pi)^{-k / 2} \exp \left(-\mathbf{z}^{\prime} \mathbf{z} / 2\right)=\d\prod_{i=1}^{k}(2 \pi)^{-1 / 2} \exp \left(-z_{i}^{2} / 2\right)$
$Z_{1}, \ldots, Z_{k} \stackrel{i . i . d .}{\sim} N(0,1)$ $E\left(Z_{i}\right)=0, \operatorname{Var}\left(Z_{i}\right)=1$
$i \neq j$ ,
$\begin{aligned} \operatorname{Cov}\left(Z_{i}, Z_{j}\right)=E\left(Z_{i} Z_{j}\right)=&\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \frac{z_{i} z_{j}}{2 \pi} \exp \left(-\left(z_{i}^{2}+z_{j}^{2}\right) / 2\right) d z_{i} d z_{j} \\ =& \int_{-\infty}^{\infty} z_{i}(2 \pi)^{-1 / 2} \exp \left(-z_{i}^{2} / 2\right) d z_{i} \int_{-\infty}^{\infty} z_{j}(2 \pi)^{-1 / 2} \exp \left(-z_{j}^{2} / 2\right) d z_{j} \\ =& E\left(Z_{i}\right) E\left(Z_{j}\right)=0\end{aligned}$
$E(\mathbf{Z})=\mathbf{0}, \operatorname{Var}(\mathbf{Z})=I$
$\Sigma=\Sigma^{1 / 2} \Sigma^{1 / 2}$ $\b X\implies \mathbf{Z}=\Sigma^{-1 / 2}(\mathbf{X}-\mu) \sim N_{k}(\mathbf{0}, I)$
$\mathbf{X}=\boldsymbol{\mu}+\Sigma^{1 / 2} \mathbf{Z}$ $E(\mathbf{X})=\mu+\Sigma^{1 / 2} E(\mathbf{Z})=\mu, \quad \operatorname{Var}(\mathbf{X})=\Sigma^{1 / 2} \operatorname{Var}(\mathbf{Z}) \Sigma^{1 / 2}=\Sigma^{1 / 2} \Sigma^{1 / 2}=\Sigma$

Ex. $\mathbf{X} \sim N_{k}(\boldsymbol{\mu}, \Sigma) .$ $E\left(\mathbf{X}^{\prime} \mathbf{X}\right)$

Ex. $\mathbf{X} \sim$ $\left(n, p_{1}, \ldots, p_{k}\right) .$ $\mu_{\mathrm{X}}$ $\Sigma_{\mathbf{X}}$ .

Ex. $X$ $Y$ $\rho_{X Y}=\operatorname{Corr}(X, Y)=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Sd}(X) \operatorname{Sd}(Y)}$ $S d(X)=\sqrt{\operatorname{Var}(X)}$ $X$ .

$\rho_{X Y}$ to exist and provide sufficient conditions?

$a, b, c, d$ $\operatorname{Corr}(a+b X, c+d Y)=\operatorname{Corr}(X, Y)$ $b>0, d>0$ $b=0$ $b<0, d>0$ $b<0, d<0$ ?

$Y \stackrel{\text { wp1 }}{=} a+b X$ $\operatorname{Corr}(X, Y)$ ?

$X \sim U(0,1)$ $Y=X^{2}$ $\operatorname{Corr}(X, Y)$ .

$X \sim U(-1,1)$ $Y=X^{2}$ $\operatorname{Corr}(X, Y)$ $X$ $Y$ independent?

Recall $\left\{X_{s}: s \in S\right\},\left\{Y_{t}: t \in T\right\}$ $x_{1}, \ldots, x_{m}, y_{1}, \ldots, y_{n} \in R^{1}$ $\left\{s_{1}, \ldots, s_{m}\right\} \subset S,\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ , the joint cdf satisfies

\begin{aligned} F_{(X_{s_{1}}, \dots, X_{s m}, Y_{t_{1}}, \dots, Y_{t_{n}})} (x_{1}, \dots, x_{m}, y_{1}, \dots, y_{n}) \\ = & F_{(X_{s_{1}}, \dots, X_{s_{m}})} (x_{1}, \dots, x_{m}) F_{(Y_{t_{1}}, \dots, Y_{t_{n}})} (y_{1}, \dots, y_{n}) \end{aligned}

$P_{\left(X_{s_{1}}, \ldots, X_{s_{m}}, Y_{t_1}, \ldots, Y_{t_{n}}\right)}\left(B_{1} \times B_{2}\right)=P_{\left(X_{s_{1}}, \ldots, X_{\left.s_{m}\right)}\right)}\left(B_{1}\right) P_{\left(Y_{t_{1}}, \ldots, Y_{t_{n}}\right)}\left(B_{2}\right)$ $B_{1} \in \mathcal{B}^{m}, B_{2} \in \mathcal{B}^{n}$

Prop 3.5.1 ( E(g h) = E(g)E(h) )

$\mathbf{X}$ $\mathbf{Y}$ $h_{1}, h_{2}:\left(R^{1}, \mathcal{B}^{1}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $h_{1}(\mathbf{X})$ $h_{2}(\mathbf{Y})$ $E\left(h_{1}^{2}(\mathbf{X})\right)<\infty$ $E\left(h_{2}^{2}(\mathbf{Y})\right)<\infty$ $E\left(h_{1}(\mathbf{X}) h_{2}(\mathbf{Y})\right)=E\left(h_{1}(\mathbf{X})\right) E\left(h_{2}(\mathbf{Y})\right)$

Proof $h_{1}(\mathbf{X})$ $h_{2}(\mathbf{Y})$ $x$ $y$ .

\begin{aligned} F_{(h_{1} (X), h_{2} (Y))} (x, y) = P (h_{1} (X) \leq x, h_{2} (Y) \leq y) \\ = & P (X \in h_{1}^{- 1} (- \infty, x], Y \in h_{2}^{- 1} (- \infty, y]) \\ = & P (X, Y) (h_{1}^{- 1} (- \infty, x] \times h_{2}^{- 1} (- \infty, y]) \\ = & P_{X} (h_{1}^{- 1} (- \infty, x]) P_{Y} (h_{2}^{- 1} (- \infty, y]) = F_{h_{1} (X)} (x) F_{h_{2} (Y)} (y) \end{aligned}

$h_{1}=\d\sum_{i} a_{i} I_{A_{i}}, h_{2}=\sum_{j} b_{j} I_{b_{j}}$ $h_{1}(\mathbf{x}) h_{2}(\mathbf{y})=\d\sum_{i, j} a_{i} b_{j} I_{A_{i}}(\mathbf{x}) I_{B_{j}}(\mathbf{y})=\sum_{i, j} a_{i} b_{j} I_{A_{i} \times B_{j}}(\mathbf{x}, \mathbf{y})$ $\d E\left(h_{1}(\mathbf{X}) h_{2}(\mathbf{Y})\right) =\sum_{i, j} a_{i} b_{j} P_{(\mathbf{X}, \mathbf{Y})}\left(A_{i} \times B_{j}\right) =\sum_{i, j} a_{i} b_{j} P_{\mathbf{X}}\left(A_{i}\right) P_{\mathbf{Y}}\left(B_{j}\right)=E\left(h_{1}(\mathbf{X})\right) E\left(h_{2}(\mathbf{Y})\right)$ as required.

$h_{1}, h_{2}$ $h_{1}=h_{1+}-h_{1-}, h_{2}=h_{2+}-h_{2-}$ .

Corollary 3.5.2 (Covariance of Ind. Functions = 0)

$\operatorname{Cov}\left(h_{1}(\mathbf{X}), h_{2}(\mathbf{Y})\right)=0$

Ex. $\mathbf{X} \in R^{k}$ $\mathbf{Y} \in R^{l}$ $\operatorname{Cov}(\mathbf{X}, \mathbf{Y})=E\left(\left(\mathbf{X}-\mu_{\mathbf{X}}\right)\left(\mathbf{Y}-\mu_{\mathbf{Y}}\right)^{\prime}\right)$ , provided all the relevant expectations exist.

$\operatorname{Cov}(\mathbf{X}, \mathbf{Y}) \in R^{k \times l}$ .

$\operatorname{Cov}(\mathbf{X}, \mathbf{Y}) \in R^{k \times l}$ $\mathbf{a} \in R^{p}, \mathbf{b} \in R^{q}, A \in R^{p \times k}, B \in R^{q \times l}$ $\operatorname{Cov}(\mathbf{a}+A \mathbf{X}, \mathbf{b}+B \mathbf{Y})$

$\operatorname{Cov}(\mathbf{X}, \mathbf{Y}) \in R^{k \times l}$ $\mathbf{X}$ $\mathbf{Y}$ $\operatorname{Cov}(\mathbf{X}, \mathbf{Y})$ .

Ex. $\mathbf{X} \in R^{k}$ $\Sigma_{\mathrm{X}} \in R^{k \times k}$ $\operatorname{Corr}(\mathbf{X})=R_{\mathbf{X}}=D_{\mathbf{X}}^{-1} \Sigma_{\mathbf{X}} D_{\mathbf{X}}^{-1}$ $D_{\mathrm{X}}=\operatorname{diag}\left(S d\left(X_{1}\right), \ldots, S d\left(X_{1}\right)\right)=\operatorname{diag}\left(\sqrt{\sigma_{11}}, \ldots, \sqrt{\sigma_{k k}}\right)$

$(i, j)$ $R_{\mathbf{X}}$ $\operatorname{Corr}\left(X_{i}, X_{j}\right)$ .

$\mathbf{Y}=D \mathbf{X}$ $D=\operatorname{diag}\left(d_{1}, \ldots, d_{k}\right)$ $d_{i}>0$ $i=1, \ldots, k .$ $\operatorname{Corr}(\mathbf{Y})=\operatorname{Corr}(\mathbf{X})$

$D$ $\operatorname{Corr}(\mathbf{Y})=\operatorname{Corr}(\mathbf{X})$ ?

Lecture 17

Def. Functions of a Stochastic Process

$\left\{\left(t, X_{t}\right): t \in T\right\}$ $E\left(X_{t}^{2}\right)<\infty$ $t \in T$ .

mean function $\mu: T \rightarrow R^{1}$ $\mu(t)=E\left(X_{t}\right)$

autocovariance function $\sigma: T \times T \rightarrow R^{1}$ $\sigma(s, t)=\operatorname{Cov}\left(X_{s}, X_{t}\right)$ , provided these expectations exist.

autocorrelation function $\rho: T \times T \rightarrow R^{1}$ $\rho(s, t)=\d{\sigma(s, t) \o \sqrt{\sigma(s, s)} \sqrt{\sigma(t, t)}}$ $\sigma(t, t)>0 \fa t \in T$

E.g. (iid process)

$\left\{X_{t}: t \in T\right\}$ $E\left(X_{t}\right)=m$ $\operatorname{Var}\left(X_{t}\right)=v$
$\mu(t)=m, \quad \sigma(s, t)=\bc v & s=t \\ 0 & s \neq t\ec\quad \rho(s, t)= \begin{cases}1 & s=t \\ 0 & s \neq t\end{cases}$

Def. Gaussian process

$\left\{X_{t}: t \in T\right\}$ $\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ ,
$\begin{aligned} \left(X_{t_{1}}, \ldots, X_{t_{n}}\right) & \sim N_{n}\left(\left(\begin{array}{ccc} \mu\left(t_{1}\right) \\ \vdots \\ \mu\left(t_{n}\right) \end{array}\right),\left(\begin{array}{cc} \sigma\left(t_{1}, t_{1}\right) & \cdots & \sigma\left(t_{1}, t_{n}\right) \\ \vdots & & \vdots\\ \sigma\left(t_{n}, t_{1}\right) & \cdots & \sigma\left(t_{n}, t_{n}\right) \end{array}\right)\right) =N_{n}\Big([\mu\left(t_{i}\right)],[\sigma\left(t_{i}, t_{j}\right)]\Big) \end{aligned}$
A Gaussian process is completely specified by the mean and autocovariance functions
$\mu: T \rightarrow R^{1}$ $\sigma: T \times T \rightarrow R^{1}$ $\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ $[\sigma\left(t_{i}, t_{j}\right)]$ is symmetric and positive semidefinite:
$\sigma: T \times T \rightarrow R^{1}$ $\sigma(s, t)=\sigma(t, s)$ $\d\sum_{i=1}^{n} \sum_{i=1}^{n} x_{i} x_{j} \sigma\left(t_{i}, t_{j}\right)=\mathrm{x}^{\prime}\left(\sigma\left(t_{i}, t_{j}\right)\right) \mathrm{x} \geq 0$ $\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ $x=\left(x_{1}, \ldots, x_{n}\right)^{\prime} \in R^{n}$

Def. Weakly Stationary Process

$T \subset R^{k}$ $\mu$ $\sigma$ weakly stationary $\mu(t)$ $t$ $\sigma(s, t)=\kappa(s-t)$ $\kappa: R^{k} \rightarrow R^{1}$ .

Note $\kappa$ $\kappa$ $\kappa(0) \geq 0, \kappa(t)=\kappa(-t)$ $\d\sum_{i=1}^{n} \sum_{i=1}^{n} x_{i} x_{j} \kappa\left(t_{i}-t_{j}\right) \geq 0$ $\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ $x=\left(x_{1}, \ldots, x_{n}\right)^{\prime} \in R^{n}$

$\kappa$ $\kappa(t)=\exp \left(-\tau^{2}\|t\|^{2}\right)$ $\tau^{2}>0$ is positive definite.

Def. Random Walk

$\left\{Z_{t}: t \in \mathbb{N}\right\}$ $\left\{\left(t, X_{t}\right): t \in \mathbb{N}\right\}$ $X_{t}=\d\sum_{i=1}^{t} Z_{i}$ is called a random walk
$Z_{t} \sim-1+2 \operatorname{Bernoulli}(p)$ $P\left(Z_{t}=-1\right)=1-p, P\left(Z_{t}=1\right)=p$
$E(X_t) = p, Var(X_t) = p(1-p)$ , so the random walk has the following functions:

\begin{aligned} μ (t) & = E (X_{t}) = \sum_{i = 1}^{t} E (Z_{t}) = t E (Z_{1}) = t (- (1 - p) + p) = (2 p - 1) t \\ σ (s, t) & = Cov (X_{s}, X_{t}) = Cov (\sum_{i = 1}^{s} Z_{i}, \sum_{j = 1}^{t} Z_{j}) = \sum_{i = 1}^{s} \sum_{j = 1}^{t} Cov (Z_{i}, Z_{j}) \leftarrow only non-zero when i = j \\ = \sum_{i = 1}^{min {s, t}} Var (Z_{i}) = min {s, t} Var (Z_{1}) = 4 p (1 - p) min {s, t} \\ ρ (s, t) & = \frac{4 p (1 - p) min {s, t}}{\sqrt{4 p (1 - p) s} \sqrt{4 p (1 - p) t}} = \frac{min {s, t}}{\sqrt{s t}} \end{aligned}

$\mu$ $\sigma$ satisfy the definition for stationarity)
$\left\{Z_{t}: t \in \mathbb{N}\right\}$ $N\left(m, \tau^{2}\right)$ , then it is a Gaussian random walk:

\begin{aligned} μ (t) & = E (X_{t}) = \sum_{i = 1}^{t} E (Z_{t}) = t E (Z_{1}) = m t \\ σ (s, t) & = Cov (X_{s}, X_{t}) \overset{as above}{=} min {s, t} Var (Z_{1}) = τ^{2} min {s, t} \\ ρ (s, t) & = \frac{τ^{2} min {s, t}}{\sqrt{τ^{2} s} \sqrt{τ^{2} t}} = \frac{min {s, t}}{\sqrt{s t}} \end{aligned}

$X_{t}=\d\sum_{i=1}^{t} Z_{i}$ , we have:

\begin{matrix} (\begin{matrix} X_{1} \\ X_{2} \\ ⋮ \\ X_{t} \end{matrix}) = (\begin{array}{cccc} 1 & 0 & \dots & 0 \\ 1 & 1 & ⋱ & ⋮ \\ ⋮ & ⋱ & 0 \\ 1 & 1 & \dots & 1 \end{array}) (\begin{matrix} Z_{1} \\ Z_{2} \\ ⋮ \\ Z_{t} \end{matrix}) = A Z_{t} \end{matrix}

$\b Z \sim N_t(m\v 1, \tau^2I) \implies A\b Z \sim N_t(mA\v 1, A\tau^2IA')$ $\left\{X_{t}: t \in \mathbb{N}\right\}$ are defined consistently. By KCT, this defines a s.p. and it is a Gaussian process

E.g. Weakly Stationary Gaussian Process

$\left\{Z_{t}: t \in \mathbb{Z}\right\}$ $N\left(0, \tau^{2}\right)$ $\left\{X_{t}: t \in \mathbb{Z}\right\}$ $X_{t}=Z_{t}+\theta Z_{t-1}$ $\theta \in R^{1}$

\begin{aligned} μ (t) = & E (X_{t}) = E (Z_{t}) + θ E (Z_{t - 1}) = 0 \\ σ (s, t) = & Cov (X_{s}, X_{t}) \\ = & E (X_{s} X_{t}) - E (X_{s}) E (X_{t}) \\ = & E (X_{s} X_{t}) = E ((Z_{s} + θ Z_{s - 1}) (Z_{t} + θ Z_{t - 1})) \\ = & E (Z_{s} Z_{t}) + θ [E (Z_{s} Z_{t - 1}) + E (Z_{s - 1} Z_{t})] + θ^{2} E (Z_{s - 1} Z_{t - 1}) \\ = & {\begin{cases} 0 & s < t - 1 \leftarrow when indices don’t match, covariance is 0 \\ τ^{2} θ & s = t - 1 \\ τ^{2} + τ^{2} θ^{2} & s = t \\ τ^{2} θ & s = t + 1 \\ 0 & s > t + 1 \end{cases} \end{aligned}

So the autocovariance matrix will only have 3 diagonal bands that are non-zero — variance on the diagonal, covariance on the off diagonals (when s = t-1, s = t, s = t+1).

\begin{matrix} (\begin{matrix} X_{t} \\ X_{t + 1} \\ ⋮ \\ X_{t + n} \end{matrix}) = (\begin{array}{cccc} θ & 1 & 0 & \dots & 0 \\ 0 & θ & 1 & ⋱ & ⋮ \\ ⋮ & ⋱ & ⋱ & ⋱ & 0 \\ 0 & \dots & 0 & θ & 1 \end{array}) (\begin{matrix} Z_{t - 1} \\ Z_{t} \\ ⋮ \\ Z_{t + n} \end{matrix}) = A Z_{t - 1, t + n} \end{matrix}

$AZ$ $\left\{\left(t, X_{t}\right): t \in \mathbb{Z}\right\}$ is a Gaussian process

$\sigma$ $\sigma(s, t)=\kappa(s-t)$ where

\begin{matrix} κ (t) = {\begin{array}{cc} 0 & t < - 1 \\ τ^{2} θ & t = - 1 \\ τ^{2} + τ^{2} θ^{2} & t = 0 \\ τ^{2} θ & t = 1 \\ 0 & t > 1 \end{array} \end{matrix}

it is a weakly stationary Gaussian process

Ex. $X_{1}, \ldots, X_{m}, Y_{1}, \ldots, Y_{n}$ $a_{0}, a_{1}, \ldots, a_{m}, b_{0}, b_{1}, \ldots, b_{n}$ $\d\operatorname{Cov}\left(a_{0}+\sum_{i=1}^{m} a_{i} X_{i}, b_{0}+\sum_{j=1}^{n} b_{j} Y_{j}\right)=\sum_{i=1}^{m} \sum_{j=1}^{n} a_{i} b_{j} \operatorname{Cov}\left(X_{i}, Y_{j}\right)$

Ex. $X_{1}, \ldots, X_{m}$ $a_{0}, a_{1}, \ldots, a_{m}$ $\d\operatorname{Var}\left(a_{0}+\sum_{i=1}^{m} a_{i} X_{i}\right)=\sum_{i=1}^{m} a_{i}^{2} \operatorname{Var}\left(X_{i}\right)+2 \sum_{i<j} a_{i} a_{j} \operatorname{Cov}\left(X_{i}, X_{j}\right)$

$X_{1}, \ldots, X_{m}$ are mutually statistically independent.

Ex. $\left(X_{1}, \ldots, X_{t}\right)^{\prime}$ in the Gaussian case.

Lecture 18

Def. Markov's Inequality

$X$ $x>0$ $\d P(X \geq x) \leq \frac{E(X)}{x}$
$\d P(X \geq x) = \frac{E(X)}{x}$ $P(X=x)=1-P(X=0)$ .

Proof (inequality)

$\d P(X \geq x)=E\left(I_{\{X \geq x\}}\right) \leq E\left(\frac{X}{x} I_{\{X \geq x\}}\right)=\frac{E\left(X I_{\{X \geq x\}}\right)}{x} \leq \frac{E(X)}{x}$

Proof (equality)

$\Leftarrow$ $P(X=x)=1-P(X=0)$ ,
$P_{X}$ $\{0, x\}$ $E(X)=0 P(X=0) + x P(X=x)=x P(X \geq x)$ .

$\Rightarrow$ $E(X)=x P(X \geq x)$ $x>0$ ,
$0=E(X)-E\left(xI_{\{X \geq x\}}\right)=\big( E(XI_{X < x}) + E(XI_{X \ge x}) \big) - E(xI_\{X \ge x\}) =E\left(X I_{\{X<x\}}\right)+E\left((X-x) I_{\{X \geq x\}}\right)$

$X I_{\{X<x\}}$ $(X-x) I_{\{X \geq x\}}$ $E\left(X I_{\{X<x\}}\right)=E\left((X-x) I_{\{X \geq x\}}\right)=0$

$\bc 1=P\left(X I_{\{X<x\}}=0\right) \implies I_\{X<x\} = 0 \implies X \ge x \implies P(0 < X < x) = 0 \\ 1=P\left((X-x) I_{\{X \geq x\}}=0\right) \stackrel{(*)}{\implies} I_\{X > x\} = 0 \implies X \le x \implies P(X > x)=0 \ec$

$(*)$ $X-x = 0$ $X = x$ $I$ evaluates to.
$X = x$ $I_\{X\ge x\}$ .

$P(X=x)=1-P(X=0)$

Ex. $X$ $P(\exp (t X) \geq k)$ $k>0$ .

Might need to add t > 0

Ex. $X$ $k>0$ $P(|X| \geq k) \leq E(|X|) / k$ $P(|X| \geq k) \leq E\left(X^{2}\right) / k^{2} .$ $X \sim$ $(1)$ $P(X \geq 2)$ $X \sim$ $(1)$ and compare this with the bounds.

Def. Chebyshev's Inequality

$X$ $\mu$ $\sigma^{2}$ $k>0$ $P(|X-\mu| \geq k \sigma) \leq \f1 {k^{2}}$
$P(|X-\mu| \geq k \sigma) = \f1 {k^{2}}$ $P(X \in\{\mu-k \sigma, \mu+k \sigma\})=1-P(X=\mu)$

Proof $|X-\mu|$ is non-negative we can apply Markov and obtain

$\d P(|X-\mu| \geq k \sigma)=P\left((X-\mu)^{2} \geq k^{2} \sigma^{2}\right) \leq \frac{E\left((X-\mu)^{2}\right)}{k^{2} \sigma^{2}}=\frac{\sigma^{2}}{k^{2} \sigma^{2}}=\frac{1}{k^{2}}$

and the equality result follows as with Markov.

E.g. $-P(|X-\mu| \geq 5 \sigma) \leq \f1{25}=0.04$ $X \sim N\left(\mu, \sigma^{2}\right)$ $P(|X-\mu| \geq 5 \sigma)=5.733031 e-07$

Def. Cauchy-Schwartz Inequality

Recall $|\v x^T \v y| \le \|\v x\| \|\v y\|$

Think of a set of r.v.'s as a vector space. Restrict it to a set that has second moments and it's a linear space.

$E\left(X^{2}\right)<\infty, E\left(Y^{2}\right)<\infty$ $|E(X Y)| \leq \sqrt{E\left(X^{2}\right)} \sqrt{E\left(Y^{2}\right)}$
$|E(X Y)| = \sqrt{E\left(X^{2}\right)} \sqrt{E\left(Y^{2}\right)}$ $Y=c X$ wp1
$c=\bc 0 &P(Y=0)=P(X=0)=1\\ {E(X Y) \o E\left(X^{2} \right)} &o/w\ec$

Proof $E\left(X^{2}\right)=0$ $P(X=0)=1 \implies P(X Y=0)=1 \implies E(X Y)=0$ $E(X^2) = E(XY) \implies X = 0Y$

$E\left(X^{2}\right)>0, E\left(Y^{2}\right)>0$ . $c \in R^{1}$ $0 \leq(Y-c X)^{2}=Y^{2}-2 c X Y+c^{2} X^{2} \implies 0 \leq E\left(Y^{2}\right)-2 c E(X Y)+c^{2} E\left(X^{2}\right)$ , $c$ $c=\d {E(X Y) \o E\left(X^{2}\right)}$

$\d 0 \leq E\left(Y^{2}\right)-2 \frac{(E(X Y))^{2}}{E\left(X^{2}\right)}+\frac{(E(X Y))^{2}}{E\left(X^{2}\right)}=E\left(Y^{2}\right)-\frac{(E(X Y))^{2}}{E\left(X^{2}\right)} \implies |E(X Y)| \leq \sqrt{E\left(X^{2}\right)} \sqrt{E\left(Y^{2}\right)}$

$0 = E\left((Y-c X)^{2}\right)$ $c=\d {E(X Y) \o E\left(X^{2}\right)}$ (which minimizes the parabola)

$1=P\left((Y-c X)^{2}=0\right)=P(Y-c X=0)=P(Y=c X)$

Def. Correlation Inequality

$0<\sigma_{X}^{2}<\infty, 0<\sigma_{Y}^{2}<\infty$ $-1 \leq \rho_{X Y}=\operatorname{Corr}(X, Y) \leq 1$
$-1 \leq \rho_{X Y}=\operatorname{Corr}(X, Y) = 1$ $Y \stackrel{\text { wp } 1}{=} \bc\mu_{Y}+{\sigma_{Y}\left(X-\mu_{X}\right) \o \sigma_{X}} & \rho_{XY} = 1\\ \mu_{Y}-{\sigma_{Y}\left(X-\mu_{X}\right) \o \sigma_{X}} &\rho_{X Y}=-1\ec$

Note
$X$ $Y = a+ bX+cX^2$ can be 0 even though they are not independent.

Proof $X, Y$ $X$ $\d{\left(X-\mu_{X}\right) \o \sigma_{X}}$ $Y$ $\d{\left(Y-\mu_{Y}\right) \o \sigma_{Y}}$

$\d E\left({\left(X-\mu_{X}\right)^{2} \o \sigma_{X}^{2}}\right)=E\left({(\hat{Y}-\mu_{Y})^{2} \o \sigma_{Y}^{2}}\right)=1$

$\d\left|\rho_{X Y}\right|=\left|E$\left(\frac{X-\mu_{X}}{\sigma_{X}}\right)\left(\frac{Y-\mu_{Y}}{\sigma_{Y}}\right)$\right| \leq 1 \implies -1 \leq \rho_{X Y} \leq 1$

$\d\left(\frac{Y-\mu_{Y}}{\sigma_{Y}}\right) \stackrel{\text { wp } 1}{=} c\left(\frac{X-\mu_{X}}{\sigma_{X}}\right)$ $c=\d\frac{E\left(\left(\frac{X-\mu_{X}}{\sigma_{X}}\right)\left(\frac{Y-\mu_{Y}}{\sigma_{Y}}\right)\right)}{E\left(\left(\frac{X-\mu_{X}}{\sigma_{X}}\right)^{2}\right)}=\rho_{X Y}$

$\d Y \stackrel{w p 1}{=} \mu_{Y}+\sigma_{Y} \rho_{X Y}\left(\frac{X-\mu_{X}}{\sigma_{X}}\right)$ $\rho_{X Y}=\pm 1$ , so the result follows.

$Y = a + bX$ $a = \mu_Y + \sigma_Y\rho_{XY}(\f{-\mu_X}{\sigma_X})$ $b = \d{\sigma_Y \rho_{XY}\o \sigma_X}$

Note $Y$ $\operatorname{Var}(Y)=E\left(\left(Y-\mu_{Y}\right)^{2}\right)$

Def. Best Affine Predictor

$Y$ $a+b X$ $a$ $b$ $Y$ $a+b X$ $E\left((Y-a-b X)^{2}\right)$

$Y$ $X$ $a+b X$ $a, b$ $E\left((Y-a-b X)^{2}\right)$

Ex. $0<\sigma_{X}^{2}<\infty, 0<\sigma_{Y}^{2}<\infty$ $a, b$ $E\left((Y-a-b X)^{2}\right)$ $a_{*}, b_{*}$ $a_{*}=a-\mu_{Y}+b \mu_{X}, b_{*}=b$ $E\left(\left(\left(Y-\mu_{Y}\right)-a_{*}-b_{*}\left(X-\mu_{X}\right)\right)^{2}\right)$ $a_{*}, b_{*}$ .

Ex.

$\mu_{X}=\mu_{Y}=0$ $0<\sigma_{X}^{2}<\infty, 0<\sigma_{Y}^{2}<\infty$ $a, b$ $c_{X Y}=\d{\sigma_{Y} \rho_{X Y} \o \sigma_{X}}$ $E\left(Y-c_{X Y} X\right)=0, \operatorname{Cov}\left(Y-c_{X Y} X, a+b X\right)=0$ $E\left((Y-a-b X)^{2}=\operatorname{Var}\left(Y-c_{X Y} X\right)+a^{2}+\left(b-c_{X Y}\right)^{2} \operatorname{Var}(X) .\right.$

$c_{X Y} X$ $Y$ $X$ .

$Y$ $X$ when the assumption of 0 means is not made.

$Y$ $X$ $\rho_{X Y}^{2}$ .

$\left(\begin{array}{l} X \\ Y \end{array}\right) \sim N_{2}\left(\left(\begin{array}{l} \mu_{X} \\ \mu_{Y} \end{array}\right),\left(\begin{array}{cc} \sigma_{X}^{2} & \sigma_{X} \sigma_{Y} \rho_{X Y} \\ \sigma_{X} \sigma_{Y} \rho_{X Y} & \sigma_{Y}^{2} \end{array}\right)\right)$ $E_{Y \mid X}(Y \mid x)$ $Y$ $X$ .

Lecture 19

Def. Convexity

$C \subset R^{k}$ convex set $\mathbf{x}_{1}, \mathbf{x}_{2} \in C$ $\alpha \in[0,1]$ $\alpha \mathbf{x}_{1}+(1-\alpha) \mathbf{x}_{2} \in C$ .

$\mathbf{x}_{1}$ $\b x_{2}$ $L\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right)=\left\{\alpha \mathbf{x}_{1}+(1-\alpha) \mathbf{x}_{2}: \alpha \in[0,1]\right\}$

$C$ $f: C \rightarrow R^{1}$ convex function $f\left(\alpha \mathbf{x}_{1}+(1-\alpha) \mathbf{x}_{2}\right) \leq \alpha f\left(\mathbf{x}_{1}\right)+(1-\alpha) f\left(\mathbf{x}_{2}\right)$ $\alpha \in[0,1]$ $\ge$ concave function $f: C \rightarrow R^{1}$ $-f$ is concave

$f: C \rightarrow R^{1}$ $C \subset R^{k}$
$f$ $\d \left(\frac{\partial^{2} f\left(x_{1}, \ldots, x_{k}\right)}{\partial x_{i} \partial x_{j}}\right) \in R^{k \times k}$ $\mathbf{x}\in C$

Ex. Convexity proofs

$L\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right)$ $[\mathbf{a}, \mathbf{b}] \subset R^{k}$ $(\mathbf{a}, \mathbf{b}],(\mathbf{a}, \mathbf{b}),[\mathbf{a}, \mathbf{b}) ?$ $B_{r}(\mu) \subset R^{k}$ $E_{r}(\boldsymbol{\mu}, \Sigma)$ $E_{r}(\boldsymbol{\mu}, \Sigma)=\boldsymbol{\mu}+\Sigma^{1 / 2} B_{r}(\mathbf{0})$ $f: R^{k} \rightarrow R^{1}$ $f(\mathbf{x})=a+\mathbf{c}^{\prime} \mathbf{x}$ $a \in R^{1}, \mathbf{c} \in R^{k}$ $R^{k}$ $f(x)=-\log x$ $C=(0, \infty)$ $\Sigma \in R^{k \times k}$ $f(x)=\mathbf{x}^{\prime} \Sigma \mathbf{x}$ $R^{k}$ .

Prop 3.7.5 (Supporting Hyperplane Thm)

$C \subset R^{k}$ $\mathrm{x}_{0} \in R^{k}$ $C$ $B_{r}\left(\mathbf{x}_{0}\right) \subset C$ $\left.r>0\right)$ $\mathbf{c} \in R^{k} \backslash\{\mathbf{0}\}$ $\mathbf{x} \in C$ $\mathbf{c}^{\prime} \mathbf{x} \geq \mathbf{c}^{\prime} \mathbf{x}_{0}$ .

$A \subset R^{k}$ $S=\left\{\mathbf{x} \in R^{k}: \mathbf{a}+B \mathbf{x}=\mathbf{0}\right\}$ $\mathbf{a} \in R^{l}, B \in R^{l \times k}$ $l \leq k$ $A \subset S$

E.g. $\mathbf{a}=\mathbf{0} \in R^{k}, B=0 \in R^{1 \times k}$ $A \subset R^{k}$ $\{\mathbf{x}: \mathbf{a}+B \mathbf{x}=\mathbf{0}\}=R^{k}$

E.g. $R^{k+1}$ $\mathbf{a}, \mathbf{b} \in R^{k} \backslash\{\mathbf{0}\}$ $y=\mathbf{a}+\mathbf{b}^{\prime} \mathbf{x}$

\begin{matrix} {(\begin{array}{l} x \\ y \end{array}) \in R^{k + 1} : a + (\begin{array}{ll} b^{'} & - 1 \end{array}) (\begin{array}{l} x \\ y \end{array}) = 0} \end{matrix}

$\left\{\mathbf{x} \in R^{k}: \mathbf{a}+B \mathbf{x}=\mathbf{0}\right\}$ affine subset $R^{k}$ $1, \ldots$ $k-1, R^{k}$ $k$ )

Ex $C_{1}, C_{2} \subset R^{k}$ $C_{1} \cap C_{2}$ is convex.

Ex $C \subset R^{k}$ $C_{*}=\mathbf{a}+B C=\{\mathbf{y}=\mathbf{a}+B \mathbf{x}: \mathbf{x} \in C\}$ $C_{*}$ is convex.

Ex $C$ $R^{k}$ $C$ is convex.

Def. Affine Dimension

$A \subset R^{k}$ $A$ $A$ . For example, a squiggly line has affine dimension = 2.

Prop 3.7.7 (Expectation is in Convex Set)

$C \subset R^{k}$ $P_{\mathbf{X}}(C)=P(\b X \in C)=1$ $E(\mathbf{X}) \in R^{k}$ $E(\mathbf{X}) \in C$ .

Proof $C$ )

$\operatorname{dim}$ $C$ $\b x$ $C=\{\mathbf{x}\}$ $E(\mathbf{X})=\mathbf{x} \in C$ and the result holds.

$E(\mathbf{X})=\mathbf{0}$ $\mathbf{Y}=\mathbf{X}-E(\mathbf{X})$ $C_{*}=C-E(\mathbf{X})$ $C_*$ $P_{\mathbf{Y}}\left(C_{*}\right)=P\left(\mathbf{Y} \in C_{*}\right)=P(\mathbf{X} \in C)=P_{\mathbf{X}}(C)=1$ $E(\mathbf{X}) \in C$ $E(\mathbf{Y})=\mathbf{0} \in C_{*}$ .

$\operatorname{dim} C<k$ $\mathbf{0} \notin C$ $\mathbf{c} \in R^{k} \backslash\{\mathbf{0}\}$ $\mathbf{c}^{\prime} \mathbf{x} \geq \mathbf{c}^{\prime} \mathbf{0}=0$ $\mathbf{x} \in C$ $P\left(\mathbf{c}^{\prime} \mathbf{X} \geq 0\right)=1$ $\mathbf{c}^{\prime} \mathbf{X}$ is a nonnegative r.v.

$E(\x) = 0 \implies E\left(\mathbf{c}^{\prime} \mathbf{X}\right)=\mathbf{c}^{\prime} E(\mathbf{X})=0 \implies P\left(\mathbf{c}^{\prime} \mathbf{X}=0\right)=1$ $P\left(\mathbf{X} \in\left\{\mathbf{x}: \mathbf{c}^{\prime} \mathbf{x}=0\right\} \cap C\right)=1$ $\left\{\mathbf{x}: \mathbf{c}^{\prime} \mathbf{x}=0\right\} \cap C$ $\le k-1$ $E(\x) = 0 \in\left\{\mathbf{x}: \mathbf{c}^{\prime} \mathbf{x}=0\right\} \cap C$ $\mathbf{0} \in C$ , and we have a contradiction.

Def. Jensen's Inequality

$C \subset R^{k}$ $P_{\mathbf{X}}(C)=1, E(\mathbf{X}) \in R^{k}$ $f: C \rightarrow R^{1}$ $E(f(\mathbf{X})) \geq f(E(\mathbf{X}))$
$f(\mathbf{x}) \stackrel{w p 1}{=} a+\mathbf{b}^{\prime} \mathbf{x}$ $a, \mathbf{b} .$

E.g. Jensen's Inequality

$P_{\mathbf{X}}\left(\left\{\mathbf{x}_{1}, \mathbf{x}_{2}\right\}\right)=1$ $P_{\mathbf{X}}\left(\left\{\mathbf{x}_{1}\right\}\right)=\alpha_{1}$ $P_{\mathbf{X}}\left(\left\{\mathbf{x}_{2}\right\}\right)=1-\alpha_{1}$ $L\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right) \in \mathcal{B}^{k}$ $P_{\mathbf{x}}\left(L\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right)\right)=1$

$f: L\left(\mathbf{x}_{1}, \mathbf{x}_{2}\right) \rightarrow R^{1}$ is convex, then for this simple context Jensen's inequality is immediate:

$E(f(\b X)) = \alpha_1 f(\b x_1) + (1 - \alpha_1)f(\b x_2) \ge f(\alpha_1\b x_1 + (1-\alpha_1) \b x_2) = f(E(\b X))$

$\left\{\alpha\left(\mathbf{x}_{1}, f\left(\mathbf{x}_{1}\right)\right)+(1-\alpha)\left(\mathbf{x}_{2}, f\left(\mathbf{x}_{2}\right)\right): \alpha \in[0,1]\right\}$ $R^{k+1}$

$f$ on the line segment implies the line segment lies above the graph

{(α x_{1} + (1 - α) x_{2}, f (α x_{1} + (1 - α) x_{2})) : α \in [0, 1]}

$E(\mathbf{X})=\alpha_{1} \mathbf{x}_{1}+\left(1-\alpha_{1}\right) \mathbf{x}_{2}$ $E(f(\mathbf{X})) \geq f(E(\mathbf{X}))$

Proof $C$ $\operatorname{dim} C$ $C=\{\mathbf{x}\}$ $E(f(\mathbf{X}))=f(\mathbf{x})=f(E(\mathbf{X}))$ $f(\mathbf{x}) \stackrel{w p 1}{=} f(\mathbf{x})+\mathbf{0}^{\prime} \mathbf{x}$ so the result holds.

$\operatorname{dim} C<k$ $S=\left\{\left(\mathbf{x}^{\prime}, y\right)^{\prime}: \mathbf{x} \in C, y \geq f(\mathbf{x})\right\}$ $S \subset R^{k+1}$ $\Big(E(\mathbf{X}), f\big(E(\mathbf{X})\big)\Big)$ $S$ $\mathbf{c} \in R^{k+1} \backslash\{\mathbf{0}\}$ $\mathbf{z} \in S$

\begin{matrix} c^{'} z = \sum_{i = 1}^{k} c_{i} z_{i} + c_{k + 1} z_{k + 1} \geq c^{'} (\begin{matrix} E (X) \\ f (E (X)) \end{matrix}) = \sum_{i = 1}^{k} c_{i} E (X_{i}) + c_{k + 1} f (E (X)) \end{matrix}

$c_{k+1}<0$ $z_{k+1}$ $c_{k+1} \geq 0$ must hold, and we have 2 cases:

Case 1 $c_{k+1}>0$

$Y=\d\sum_{i=1}^{k} c_{i}\left(X_{i}-E\left(X_{i}\right)\right)+c_{k+1}(f(\mathbf{X})-f(E(\mathbf{X})) \expl{(LHS - RHS) of above, where LHS$\ge$RHS, so Y$\ge$0}$

$P(Y \geq 0)=1$ $0 \leq E(Y)=c_i(0) + c_{k+1}(E(f(\mathbf{X}))-f(E(\mathbf{X})) \implies E(f(\mathbf{X})) \geq f(E(\mathbf{X}))$

$E(f(\mathbf{X}))=f(E(\mathbf{X})$ $E(Y)=0$ $P(Y=0)=1$ $Y = 0$ , so rearrange the above:

\begin{aligned} \sum_{i = 1}^{k} c_{i} (X_{i} - E (X_{i})) & = c_{k + 1} (f (X) - f (E (X)) \\ f (X) & = f (E (X)) - \sum_{i = 1}^{k} \frac{c_{i}}{c_{k + 1}} (X_{i} - E (X_{i})) \\ = (f (E (X)) + \sum_{i = 1}^{k} \frac{c_{i}}{c_{k + 1}} E (X_{i})) + \sum_{i = 1}^{k} (- \frac{c_{i}}{c_{k + 1}}) X_{i} \end{aligned}

$a + b' \b x$

Case 2 $c_{k+1}=0$

$Y=\d\sum_{i=1}^{k} c_{i}\left(X_{i}-E\left(X_{i}\right)\right)\implies E(Y)=0 \implies P(Y=0)=1\implies P\left(\mathbf{X} \in\left\{\mathbf{x}: \mathbf{c}^{\prime} \mathbf{x}=\mathbf{c}^{\prime} E(\mathbf{X})\right\} \cap C\right)=1$

$\left\{\mathbf{x}: \mathbf{c}^{\prime} \mathbf{x}=\mathbf{c}^{\prime} E(\mathbf{X})\right\} \cap C$ $\operatorname{dim}<k$ , so by the inductive hypothesis the result holds.

Note $f: C \rightarrow R^{1}$ $P_{\mathbf{X}}(C)=1, E(\mathbf{X}) \in R^{k}$ $E(f(\mathbf{X})) \leq f(E(\mathbf{X}))$

Def. Kullback-Liebler Distance

serves as a measure of distance between probability measures
$P, Q$ $(\Omega, \mathcal{A})$ $p$ $q$ respectively
$P$ $Q$ is then defined to be

\begin{aligned} K L (P ∥ Q) & = E_{P} (- \log \frac{q}{p}) \\ = - \int_{Ω} p (ω) \log \frac{q (ω)}{p (ω)} v (d ω) \\ = \int_{Ω} p (ω) \log \frac{p (ω)}{q (ω)} v (d ω) \end{aligned}

$v$ is the counting (discrete case) or volume measure (a.c. case)

Prop 3.7.9 (KL Distance >= 0)

$P, Q$ $(\Omega, \mathcal{A})$ $p$ $q$
$K L(P \| Q) \geq 0$ $P=Q$ .

Proof $-\log x$ $(0, \infty)$ , applying Jensen gives

\begin{aligned} K L (P ∥ Q) & \geq - \log (E_{P} (\frac{q}{p})) = - \log (\int_{Ω} p (ω) \frac{q (ω)}{p (ω)} v (d ω)) \\ = - \log (\int_{Ω} q (ω) v (d ω)) = - \log 1 = 0 \end{aligned}

$a, b$ such that

- \log \frac{q (ω)}{p (ω)} \overset{w p 1 w.r.t. P}{=} a + b \frac{q (ω)}{p (ω)}

$p \stackrel{\text { wp1 w.r.t. P}}{=} q$ $a=b=0$ .

$-\log x$ $a(1-x)$ $x=1$ $p \stackrel{w p 1\ P}{=} q$

Sub-proof $p \neq q$ $\omega$ $A=\{\omega: q(\omega)=p(\omega)\}$ $A^{c}=\{\omega: q(\omega)=r p(\omega)\}$ $r$ $Q(A)=P(A)$ $Q\left(A^{c}\right)=r P\left(A^{c}\right)=r(1-P(A))=r(1-Q(A))=r Q\left(A^{c}\right)$

$r=1$ $0=Q\left(A^{c}\right)=P\left(A^{c}\right)$ $p \neq q$ $P$ probability.

Ex. $P$ $N\left(\mu_{1}, \sigma_{1}^{2}\right)$ $Q$ $N\left(\mu_{2}, \sigma_{2}^{2}\right)$ $K L(P \| Q)$ .

Ex. $K L(P \| Q)=K L(Q \| P) ?$

Lecture 20

Conditional Expectation - Discrete Case

$X$ $Y$ $E(|Y|)<\infty$
$(\mathbf{X}, Y) \in R^{k+1}$ is

p_{(X, Y)} (x, y) = P_{(X, Y)} ({(x, y)}) = P (X = x, Y = y)

$Y \mid \mathbf{X}=\mathbf{x}$ $P_{Y \mid X}$ ) is

p_{Y ∣ X} (y | x) = \frac{p_{(X, Y)} (x, y)}{p_{X} (x)}

$\d p_{\mathbf{X}}(\mathbf{x})=P_{\mathbf{X}}(\{\mathbf{x}\})=P(\mathbf{X}=\mathbf{x})=\sum_{y} p_{(\mathbf{X}, Y)}(\mathbf{x}, y)>0$ (otherwise cond. dist. not defined)

$Y$ $X=x$ is given by

E_{P_{Y ∣ X}} (Y ∣ X = x) = E_{p_{Y ∣ X}} (Y ∣ X) (x) = \sum_{y} y p_{Y ∣ X} (y ∣ x)

$E(|Y|)<\infty$ , the conditional expectation is also finite, since

\begin{aligned} \sum_{y} | y | p_{Y ∣ X} (y ∣ x) = \sum_{y} | y | \frac{p_{(X, Y)} (x, y)}{p_{X} (x)} = & \frac{1}{p_{X} (x)} \sum_{y : p_{(X, Y)} (x, y) > 0} | y | p_{(X, Y)} (x, y) \\ \leq & \frac{1}{p_{X} (x)} \sum_{(z, y)} | y | p_{(X, Y)} (z, y) = \frac{1}{p_{X} (x)} E (| Y |) < \infty \end{aligned}

$E_{P_{Y \mid X}}(Y \mid \mathbf{X}):\left(R^{k}, \mathcal{B}^{k}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $E(Y \mid \mathbf{X}):(\Omega, A) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ by

E (Y ∣ X) (ω) = E_{P_{Y ∣ X}} (Y ∣ X) (X (ω))

Prop 3.8.1 E[h(X)Y] = E[h(x)E(Y|X)]

$h:\left(R^{k}, \mathcal{B}^{k}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $E(|Y h(\mathbf{X})|)<\infty$ $E(Y h(\mathbf{X}))=E(h(\mathbf{X}) E(Y \mid \mathbf{X})) .$

Proof

\begin{aligned} E (Y h (X)) = & \sum_{(x, y)} y h (x) p_{(X, Y)} (x, y) = \sum_{(x, y)} y h (x) p_{X} (x) \frac{p_{(X, Y)} (x, y)}{p_{X} (x)} = \sum_{(x, y)} y h (x) p_{X} (x) p_{Y ∣ x} (y ∣ x) \\ = & \sum_{x} h (x) (\sum_{y} y p_{Y ∣ X} (y ∣ x)) p_{X} (x) = \sum_{x} h (x) E_{p_{Y ∣ X}} (Y ∣ X) (x) p_{X} (x) = E (h (X) E (Y ∣ X)) . \end{aligned}

Corol 3.8.1 E[h(X)Y|X] = h(x)E(Y|X)

$E(Y h(\mathbf{X}) \mid \mathbf{X})=h(\mathbf{X}) E(Y \mid \mathbf{X})$

Corol 3.8.2 (Theorem of Total Expectation)

$E(Y)=E(E(Y \mid \mathbf{X}))$ $(\mathbf{X}, Y)$ $E(|Y|)<\infty$

Proof $h(\mathbf{x}) \equiv 1$ $Y=I_{A}$ $A \in \mathcal{A}$ .

$E(Y \mid \mathbf{X})(\mathbf{x})=\sum y p_{Y \mid \mathbf{x}}(y \mid \mathbf{x})=0 p_{Y \mid \mathbf{x}}(0 \mid \mathbf{x})+1 p_{Y \mid \mathbf{x}}(1 \mid \mathbf{x})=P(A \mid \mathbf{X})(\mathbf{x})$

Corol 3.8.3 (Theorem of Total Probability)

$P(A)=E(P(A \mid \mathbf{X}))$ $A \in \mathcal{A}$

Corol 3.8.4 V(Y) = E[V(Y|X)] + V[E(Y|X)]

$\operatorname{Var}(Y)=E(\operatorname{Var}(Y \mid \mathbf{X}))+\operatorname{Var}(E(Y \mid \mathbf{X}))$ $E(Y), E\left(Y^{2}\right)<\infty$

Proof:

\begin{aligned} Var (Y) & = E ((Y - E (Y))^{2}) \overset{TTE}{=} E (E ((Y - E (Y))^{2} ∣ X)) \\ = E (E ([\underset{a}{\underset{⏟}{Y - E (Y ∣ X)}} + \underset{b}{\underset{⏟}{E (Y ∣ X) - E (Y)}}]^{2} ∣ X)) \leftarrow add and subtract E (Y | X) \end{aligned}

$a^2 + 2ab + b^2$

\begin{aligned} \underset{V a r (Y | X)}{\underset{⏟}{E ((Y - E (Y ∣ X))^{2} ∣ X)}} + 2 \underset{E (Y) - E (Y | X)}{\underset{⏟}{E ((Y - E (Y ∣ X))}} \underset{constant}{\underset{⏟}{(E (Y ∣ X) - E (Y))}} ∣ X) + \underset{constant, so = (E (Y ∣ X) - E (Y))^{2}}{\underset{⏟}{E ((E (Y ∣ X) - E (Y))^{2} ∣ X)}} \\ = Var (Y ∣ X) + 2 \underset{0}{\underset{⏟}{(E (Y ∣ X) - E (Y ∣ X))}} (E (Y ∣ X) - E (Y)) + (E (Y ∣ X) - E (Y))^{2} \\ = Var (Y ∣ X) + (E (Y ∣ X) - E (Y))^{2} \end{aligned}

$E$ to both sides gives the result:

E (V a r (Y)) = E (V a r (Y | X)) + V a r (E (Y | X))

Corol 3.8.5 (Best Predictor & Residual Error)

$E(Y \mid \mathbf{X})$ best predictor $Y$ $\mathbf{X}$ $E\left((Y-h(\mathbf{X}))^{2}\right)$ $h:\left(R^{k}, \mathcal{B}^{k}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ residual error $E(\operatorname{Var}(Y \mid \mathbf{X}))$ .

Proof

\begin{aligned} E ((Y - h (X))^{2}) = E ((Y - E (Y ∣ X) + E (Y ∣ X) - h (X))^{2}) \leftarrow add and subtract E (Y | X) \\ = & E ((Y - E (Y ∣ X))^{2}) + \underset{0}{\underset{⏟}{2 E (Y - E (Y ∣ X)) (E (Y ∣ X) - h (X))}} + E ((E (Y ∣ X) - h (X))^{2}) \end{aligned}

and so

\begin{aligned} E ((Y - h (X))^{2}) & = E ((Y - E (Y ∣ X))^{2}) + E ((E (Y ∣ X) - h (X))^{2}) \\ \geq E ((Y - E (Y ∣ X))^{2}) = E (Var (Y ∣ X)) \end{aligned}

$h(\mathbf{X})=E(Y \mid \mathbf{X})$ .

Notes

$Y$ $E(|Y|)<\infty$ $E(Y \mid \mathbf{X})$ $E(Y \mid \mathbf{X}):(\Omega, A) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $E(Y h(\mathbf{X}))=E(h(\mathbf{X}) E(Y \mid \mathbf{X}))$ $h:\left(R^{k}, \mathcal{B}^{k}\right) \rightarrow\left(R^{1}, \mathcal{B}^{1}\right)$ $E(|Y h(\mathbf{X})|)<\infty$
$E\left(Y \mid\left\{\left(t, X_{t}\right): t \in T\right\}\right)$ $Y$ $\left\{\left(t, X_{t}\right): t \in T\right\}$

Conditional Expectation - Continuous Case

$(\mathbf{X}, Y)$ $f_{(X, Y)}$ $E(|Y|)<\infty$ , then

\begin{aligned} E (Y ∣ X) (x) & = \int_{- \infty}^{\infty} y f_{Y ∣ X} (y ∣ x) d y where \\ f_{Y ∣ X} (y ∣ x) & = \frac{f_{(X, Y)} (x, y)}{f_{X} (x)} and f_{X} (x) = \int_{- \infty}^{\infty} f_{(X, Y)} (x, y) d y . \end{aligned}

E.g. $N_{k}(\mu, \Sigma)$

$\left(\begin{array}{l} \mathrm{Y} \\ \mathrm{X} \end{array}\right) \sim N_{k}(\mu, \Sigma) \text { with } \mathrm{Y} \in R^l$ $\quad\begin{aligned} \mu=&\left(\begin{array}{l} \mu_{\mathrm{Y}} \\ \mu_{\mathrm{X}} \end{array}\right), \quad \Sigma=\left(\begin{array}{cc} \Sigma_{\mathrm{Y}} & \Sigma_{\mathrm{YX}} \\ \Sigma_{\mathrm{YX}}^{\prime} & \Sigma_{\mathrm{X}} \end{array}\right) \end{aligned}$ is p.d.

$\mathbf{Y} \mid \mathbf{X}=\mathbf{x} \sim N_{k}\left(\mu_{\mathbf{Y}}+\Sigma_{\mathbf{Y X}} \Sigma_{\mathbf{X}}^{-1}\left(\mathbf{x}-\mu_{\mathbf{X}}\right), \Sigma_{\mathbf{Y}}-\Sigma_{\mathbf{Y X}} \Sigma_{\mathbf{X}}^{-1} \Sigma_{\mathbf{Y X}}^{\prime}\right)$

$E(\mathbf{Y} \mid \mathbf{X})(\mathbf{x})=\mu_{\mathrm{Y}}+\Sigma_{\mathrm{YX}} \Sigma_{\mathrm{X}}^{-1}\left(\mathbf{x}-\mu_{\mathrm{X}}\right)$ $\d\sum_{i=1}^{l} E\left(\left(Y_{i}-h_{i}(\mathbf{X})\right)^{2}\right)=E\left(\|\mathbf{Y}-\mathbf{h}(\mathbf{X})\|^{2}\right)$ $\mathbf{h}:\left(R^{k-1}, \mathcal{B}^{k-1}\right) \rightarrow\left(R^{\prime}, \mathcal{B}^{\prime}\right)$

Def. Martingales

$\mathrm{H}$ $1 / 2$ $\$ x$ $\$ 2 x$ $0.5(2 x-x)-0.5 x=0$
$\$ 1$ $\$ 2$ $\$ 4$ $n$ $\$ 2^{n}$ on the next bet. They stop as soon as they win which happens with probability 1
$H$ $n$ $2^{n}-\left(1+2+\cdots+2^{n-1}\right)=2^{n}-2^{n}+1=1$ so this guarantees a profit
$\d\sum_{n=1}^{\infty}\left(\frac{1}{2}\right)^{n}\left(2^{n}-1\right)=\infty$ , so you need a big bank account if you want to use this strategy
$X_{n}$ $n$ $X_{n+1}= \begin{cases}X_{n} & \text { if stopped by toss } n \\ X_{n}+2^{n} & \text { if } H \text { at toss } n \\ X_{n}-2^{n} & \text { if } T \text { at toss } n\end{cases}$

⟹ E (X_{n + 1} ∣ X_{1}, \dots, X_{n}) (x_{1}, \dots, x_{n}) = x_{n} ⟹ E (X_{n + 1} ∣ X_{1}, \dots, X_{n}) = X_{n}

$\left\{\left(n, X_{n}\right): n \in \mathbb{N}\right\}$ with this property is called a martingale

Lecture 21

Def. Generating Functions

$\left\{a_{n}: n \in \mathbb{N}_{0}\right\}$ $\d G(t)=\sum_{i=0}^{\infty} a_{i} t^{i}$ $t \in\left(-h_{G}, h_{G}\right)$ $h_{G}>0$

$\d\left.\frac{d^{k} G(t)}{d t^{k}}\right|_{t=0}=a_{k} k!$ $a_k$
$a_{n}=n$ )
$G(t)=\d\sum_{i=0}^{\infty} a_{i} t^{i}, H(t)=\sum_{i=0}^{\infty} b_{i} t^{i}$ $K(t)=G(t) H(t)=\d\sum_{i=0}^{\infty} c_{i} t^{i}$ $c_{i}=a_{0} b_{i}+a_{1} b_{i-1}+\cdots+a_{i} b_{0}$ $\left\{c_{n}: n \in \mathbb{N}_{0}\right\}$ $h_{K}=\min \left\{h_{G}, h_{H}\right\}$

Def. Abel's Theorem

$G(t)=\d\sum_{i=0}^{\infty} a_{i} t^{i}$ $(-1,1)$ $\d\sum_{i=0}^{\infty} a_{i}$ $\infty$ $\d\lim _{t \uparrow 1} G(t)=\sum_{i=0}^{\infty} a_{i}$ .

Def. Probability Generating Functions

$X$ $P_{X}\left(\mathbb{N}_{0}\right)=1$ $X$ $\d G_{X}(t)=E\left(t^{X}\right)=\sum_{i=0}^{\infty} P(X=i) t^{i}$ $|t| \leq 1$ .

Prop 3.9.1 (Same PGF <=> same prob dist)

$G_{X}(t)=G_{Y}(t)$ $t \in(-h, h)$ $h>0$ $X$ $Y$ have the same probability distribution.

Proof $|t| \leq 1$ $G_{X}(t)=\d\sum_{i=0}^{\infty} P(X=i) t^{i}$ $\d\left.\frac{1}{k !} \frac{d^{k} G_{X}(t)}{d t^{k}}\right|_{t=0}=P(X=k)=\left.\frac{1}{k !} \frac{d^{k} G_{Y}(t)}{d t^{k}}\right|_{t=0}=P(Y=k)$

$G_{X}$ $X$

Prop 3.9.2 (PGF Properties, K-th Factorial Moment)

$X, Y$ $G_{X}, G_{Y}$ $G_{X+Y}(t)=G_{X}(t) G_{Y}(t)$ .

Proof $G_{X+Y}(t)=E\left(t^{X+Y}\right)=E\left(t^{X} t^{Y}\right) \stackrel{\text { ind }}{=} E\left(t^{X}\right) E\left(t^{Y}\right)=G_{X}(t) G_{Y}(t)$

$X$ $\operatorname{pgf} G_{X}$ $\mathbf k$ -th factorial moment $X$ :
$\mu_{[k]}=E(X(X-1) \cdots(X-k+1))=\d\sum_{i=k}^{\infty} i(i-1) \cdots(i-k+1) P(X=i)$ exists, $\d\lim _{t \uparrow 1} \frac{d^{k} G_{X}(t)}{d t^{k}}=\mu_{[k]}$ .

Proof $|t|<1$ $\d\frac{d^{k} G_{X}(t)}{d t^{k}}=\frac{d^{k}}{d t^{k}} \sum_{i=0}^{\infty} P(X=i) t^{i}=\sum_{i=k}^{\infty} i(i-1) \cdots(i-k+1) P(X=i) t^{i-k}$ is finite, and by Abel's Thm

$\d\lim _{t \uparrow 1} \sum_{i=k}^{\infty} i(i-1) \cdots(i-k+1) P(X=i) t^{i-k} =\sum_{i=k}^{\infty} i(i-1) \cdots(i-k+1) P(X=i)=\mu_{[k]}$

(iii) (Compound distributions)
$\left\{X_{i}: i=1,2, \ldots\right\}$ $G_{X}$ $N$ $G_{N}$ ,
$Y=\d\sum_{i=1}^{N} X_{i}$ $G_{Y}(t)=G_{N}\left(G_{X}(t)\right)$ .

Proof

\begin{aligned} G_{Y} (t) & = E (t^{Y}) = E (t^{\sum_{i = 1}^{N} x_{i}}) = E (\prod_{i = 1}^{N} t^{x_{i}}) \overset{T T E}{=} E (E (\prod_{i = 1}^{N} t^{x_{i}} ∣ N)) = \sum_{n = 1}^{\infty} P (N = n) E (\prod_{i = 1}^{n} t^{x_{i}}) \\ \overset{(i)}{=} \sum_{n = 1}^{\infty} P (N = n) (G_{X} (t))^{n} = G_{N} (G_{X} (t)) \end{aligned}

E.g. $X \sim$ $(\lambda)$ $\lambda>0$ $p_{X}(i)= \d\frac{\lambda^{i}}{i !} e^{-\lambda} \text { for } i=0,1,2, \ldots$

\begin{aligned} G_{X} (t) = & E (t^{X}) = \sum_{i = 0}^{\infty} t^{i} \frac{λ^{i}}{i!} e^{- λ} = e^{- λ} \sum_{i = 0}^{\infty} \frac{(t λ)^{i}}{i!} = e^{- λ} e^{t λ} = e^{λ (t - 1)} \leftarrow uses Maclaurin expansion \end{aligned}

$X \sim \operatorname{Poisson}\left(\lambda_{1}\right)$ $Y \sim \operatorname{Poisson}\left(\lambda_{2}\right)$ $X+Y \sim$ $\left(\lambda_{1}+\lambda_{2}\right)$ , since

G_{X + Y} (t) = G_{X} (t) G_{Y} (t) = e^{λ_{1} (t - 1)} e^{λ_{2} (t - 1)} = e^{(λ_{1} + λ_{2}) (t - 1)}

$X \sim$ $(\lambda)$ $\mu_{[k]}$ $k$ $\d\sum_{i=0}^{\infty} \frac{(t \lambda)^{i}}{i !}$ $t \in R^{1}$

\begin{aligned} μ_{[1]} & = lim_{t ↑ 1} \frac{d G_{X} (t)}{d t} = lim_{t ↑ 1} λ e^{λ (t - 1)} = λ \\ μ_{[2]} & = lim_{t ↑ 1} \frac{d^{2} G_{X} (t)}{d t^{2}} = lim_{t ↑ 1} λ^{2} e^{λ (t - 1)} = λ^{2} \end{aligned}

$Var(X) = E(X^2) - (E(X))^2 = \ub{E(X^2) - E(X)} - \ub{(E(X))^2 + E(X)} = E(X(X-1))-E(X)(E(X) - 1)$
$Var (X) = μ_{[2]} - μ_{[1]} (μ_{[1]} - 1) = λ^{2} - λ (λ - 1) = λ$

$X \sim \operatorname{Bernoulli}(p)$ $G_{X}(t)$ $(n, p)$ distribution.

$X \sim \operatorname{Geometric}(p)$ $G_{X}(t)$ $X$ .

$N \sim$ $(\lambda)$ $X_{1}, X_{2}, \ldots \sim$ $-1+2 \operatorname{Bernoulli}(p)$ $Y=\sum_{i=1}^{N} X_{i}$ $E(Y)$ .

Def. Moment Generating Function

$\mathbf{X} \in R^{k}$ $\mathbf{X}$ $m_{\mathbf{X}}(\mathbf{t})=E\left(\exp \left(\mathbf{t}^{\prime} \mathbf{X}\right)\right)$ $\mathbf{t} \in B_{h}(\mathbf{0})$ . The MGF does not always exist (e.g. Cauchy).

Def. Characteristic Function

$\mathbf{X}$ $c_\mathbf{X}(\mathbf{t})=E\left(\exp \left(i \mathbf{t}^{\prime} \mathbf{X}\right)\right)$ $\mathbf{t} \in R^{k}$ $e^{i \mathbf{x}}=\cos x+i \sin x$ $|\cos x|$ $|\sin x| \leq 1$ $e^{ix}$ is bounded:

E (| \exp (i t^{'} X) |) = E (| \cos (t^{'} X) + i \sin (t^{'} X) |) \leq E (| \cos (t^{'} X) |) + E (| \sin (t^{'} X) |) \leq 2

$c_\mathbf{X}(\mathbf{t})=E\left(\cos \left(\mathbf{t}^{\prime} \mathbf{X}\right)\right)+i E\left(\sin \left(\mathbf{t}^{\prime} \mathbf{X}\right)\right)$ always exists (may be complex valued)

$P_{\mathbf{X}}(B)=P_{\mathbf{X}}(-B)$ $P_{\mathbf{X}}\left(\mathbf{t}^{\prime} \mathbf{X} \leq x\right)=P_{\mathbf{X}}\left(\mathbf{t}^{\prime} \mathbf{X} \geq-x\right)$ $\mathbf{t}^{\prime} \mathbf{X}$
$\sin (-x)=-\sin (x)$ $E\left(\sin \left(\mathbf{t}^{\prime} \mathbf{X}\right)\right)=0$ $c_\mathbf{X}$ is real-valued

Prop 3.9.3 (Uniqueness of MGF & CF)

$m_{\mathbf{X}}, m_{\mathbf{Y}}$ $m_{\mathbf{X}}(\mathbf{t})=m_{\mathbf{Y}}(\mathbf{t})$ $\mathbf{t} \in B_{h}(\mathbf{0})$ $h>0$ $P_{\mathbf{X}}=P_{\mathbf{Y}}$ .

$c_{\mathbf{X}}(\mathbf{t})=c_{\mathbf{Y}}(\mathbf{t})$ $\mathbf{t} \in R^{k}$ $P_{\mathbf{X}}=P_{\mathbf{Y}}$ .

$m_{\mathrm{X}}$ $c_{\mathrm{X}}$ $\mathbf{X}$ .

$\mathbf{X}$ $m_{\mathbf{X}}$ $c_{\mathbf{X}}$ .

Note Same distribution does not mean same r.v.

Def. Mixed Moment of Random Vector

$i_{1}, \ldots, i_{k} \in \mathbb{N}_{0}$ $\left(i_{1}, \ldots, i_{k}\right)$ -th mixed momentrandom vector $\mathbf{X} \in R^{k}$ $\mu_{i_{1}, \ldots, i_{k}}=E\left(X_{1}^{i_{1}} \cdots X_{k}^{i_{k}}\right)$ whenever this expectation exists.

Prop 3.9.4 (Prev. Mixed Moments are Finite)

$i_{1} \leq j_{1}, \ldots, i_{k} \leq j_{k}$ $E\left(\left|X_{1}^{j_{1}} \cdots X_{k}^{j_{k}}\right|\right)<\infty$ $\left(j_{1}, \ldots, j_{k}\right)$ $j_{1}+\cdots+j_{k}=j$ $\mu_{i_{1}, \ldots, i_{k}}$ is finite.

Proof (for k = 2)

Exercise III.9.4

Prop 3.9.5 (i-th Mixed Moment)

$m_{\mathrm{X}}$ $X$ $\left(i_{1}, \ldots, i_{k}\right)$ -th mixed moment is given by

μ_{i_{1}, \dots, i_{k}} = {\frac{\partial^{l} m_{X} (t)}{\partial^{i_{1}} t_{1} \dots \partial^{i_{k}} t_{k}} |}_{t = 0}

$l=i_{1}+\cdots+i_{k}$ .

Proof $k=1$ $t \in B_{h}(0)$ ,

\begin{aligned} m_{X} (t) & = E (\exp (t X)) = E (I_{{X \geq 0}} \exp (t X)) + E (I_{{X < 0}} \exp (t X)) \\ = E (I_{{X \geq 0}} \exp (t X_{+})) + E (I_{{X < 0}} \exp (- t X_{-})) \\ = m_{X_{+}} (t) - P (X < 0) + m_{X_{-}} (- t) - P (X \geq 0) < \infty \\ m_{| X |} (t) & = E (\exp (t X_{+} + t X_{-})) \\ = m_{X_{+}} (t) - P (X < 0) + m_{X_{-}} (t) - P (X \geq 0) \end{aligned}

$Y_{n}=\d\sum_{j=0}^{n} \frac{t^{j} X^{j}}{j !} \rightarrow \sum_{j=0}^{\infty} \frac{t^{j} X^{j}}{j !}=\exp (t X)$ $\d\left|Y_{n}\right| \leq \sum_{j=0}^{n} \frac{|t|^{j}|X|^{j}}{j !} \uparrow \sum_{k=0}^{\infty} \frac{|t|^{j}|X|^{j}}{j !}=\exp (|t||X|) .$

$m_{|X|}$ $\d E\left(|X|^{k}\right) \leq \frac{k !}{|t|^{k}} m_{|X|}(|t|)<\infty$ $X$ are finite.

$\d\lim _{n \rightarrow \infty} E\left(Y_{n}\right) \rightarrow \sum_{j=0}^{\infty} \frac{t^{j} \mu_{j}}{j !}=m_{X}(t) \implies \mu_{j}=\left.\frac{d^{j} m_{X}(t)}{d t^{j}}\right|_{t=0}$

$\mathbf{Z}=\left(\left|X_{1}\right|, \ldots,\left|X_{k}\right|\right)$ $m_{Z}$ exists. Let

\begin{aligned} Y_{n} & = \sum_{j = 0}^{n} \frac{{(t_{1} X_{1} + \dots + t_{k} X_{k})}^{j}}{j!} \\ = \sum_{j = 0}^{n} \frac{1}{j!} \sum_{\binom{i_{1} \geq 0 \dots i_{k} \geq 0}{i_{1} + \dots + i_{k} = j}} (\begin{array}{c} j \\ i_{1} \dots i_{k} \end{array}) t_{1}^{i_{1}} \dots t_{k}^{i_{k}} X_{1}^{i_{1}} \dots X_{k}^{i_{k}} \\ | Y_{n} | & \leq \exp (| t_{1} | | X_{1} | + \dots + t_{k} | X_{k} |) \end{aligned}

$\mu_{i_{1}, \ldots, i_{k}}$ $\d E\left(Y_{n}\right) \rightarrow \sum_{j=0}^{\infty} \sum_{i_{1} \geq 0 \ldots i_{k} \geq 0 \atop i_{1}+\cdots+i_{k}=j} \frac{t_{1}^{i_{1}} \cdots t_{k}^{i_{k}}}{i_{1} ! \cdots i_{k} !} \mu_{i_{1}, \ldots, i_{k}}=m_{\mathbf{X}}(\mathbf{t})$

Prop 3.9.6 c(t) = m(it)

$m_{\mathbf{X}}$ $c_{\mathbf{X}}(\mathbf{t})=m_{\mathbf{X}}(i\b t)$

Prop 3.9.7 (MGF & CF of X+Y)

$\mathbf{X}, \mathbf{Y} \in R^{k}$ $m_{\mathbf{X}}, m_{\mathbf{Y}}$ $\left.c_{\mathbf{X}}, c_{\mathbf{Y}}\right)$ $\mathbf{X}+\mathbf{Y}$ $m_{\mathbf{X}+\mathbf{Y}}(\mathbf{t})=m_{\mathbf{X}}(\mathbf{t}) m_{\mathbf{Y}}(\mathbf{t})$ $m_{\mathbf{X}}(\mathbf{t})$ $m_{\mathbf{Y}}(\mathbf{t})$ $c f_{\mathbf{X}+\mathbf{Y}}(\mathbf{t})=c_{\mathbf{X}}(\mathbf{t}) c_{\mathbf{Y}}(\mathbf{t})$ .

Proof

\begin{aligned} c X + Y (t) & = E (\exp (i t^{'} (X + Y)) = E (\exp (i t^{'} X) \exp (i t^{'} Y)) \\ = E (\exp (i t^{'} X)) E (\exp (i t^{'} Y)) = c X (t) c Y (t) \end{aligned}

E.g. $\mathbf{X} \sim N_{k}(\boldsymbol{\mu}, \Sigma)$

$\mathbf{X}=\boldsymbol{\mu}+\Sigma^{1 / 2} \mathbf{Z}$ $\mathbf{Z} \sim N_{k}(\mathbf{0}, I)$ $Z_{1}, \ldots, Z_{k} \stackrel{i, i, d}{\sim} N(0,1)$ and

\begin{aligned} m_{Z} (t) & = E (\exp (t^{'} Z)) = E (\exp (t_{1} Z_{1} + \dots + t_{k} Z_{k})) = E (\prod_{i = 1}^{k} \exp (t_{i} Z_{i})) \overset{i . i . d}{=} \prod_{i = 1}^{k} E (\exp (t_{i} Z_{i})) \\ = \prod_{i = 1}^{k} m_{Z} (t_{i}) where \\ m_{Z} (t) & = \int_{- \infty}^{\infty} \exp (t z) \frac{1}{\sqrt{2 π}} \exp (- z^{2} / 2) d z \\ = \exp (t^{2} / 2) \int_{- \infty}^{\infty} \frac{1}{\sqrt{2 π}} \exp (- (z - t)^{2} / 2) d z \leftarrow complete the square for - z^{2} / 2 + t z \\ = \exp (t^{2} / 2) \end{aligned}

$m_{\mathbf{Z}}(\mathbf{t})=\exp{(\f12 \sumto k t_i^2)} =\exp \left(\mathbf{t}^{\prime} \mathbf{t} / 2\right)$

$\mathbf{X}=\boldsymbol{\mu}+\Sigma^{1/2} \mathbf{Z}$ $m_{\mathbf{X}}(\mathbf{t})=E\left(\exp \left(\mathbf{t}^{\prime} \mathbf{X}\right)\right)$ , we get

\begin{aligned} m_{X} (t) & = E (\exp (t^{'} (μ + Σ^{1 / 2} Z)) = \exp (t^{'} μ) E (\exp (t^{'} Σ^{1 / 2} Z)) \\ = \exp (t^{'} μ) E (\exp ({(Σ^{1 / 2} t)}^{'} Z)) \\ = \exp (t^{'} μ) \exp (t^{'} Σ t / 2) \\ = \exp (t^{'} μ + t^{'} Σ t / 2) \\ c_{X} (t) & = \exp (i t^{'} μ - t^{'} Σ t / 2) using Prop. III.9.6 \end{aligned}

$\mathbf{X}_{1}, \ldots, \mathbf{X}_{n}$ $N_{k}(\boldsymbol{\mu}, \Sigma)$ $\d\mathbf{Y}=\frac{1}{n} \sum_{i=1}^{n} \mathbf{X}_{i}$ has the following MGF:

\begin{aligned} m_{Y} (t) & = E (\exp (t^{'} \frac{1}{n} \sum_{i = 1}^{n} X_{i})) = E (\prod_{i = 1}^{n} \exp ({(\frac{t}{n})}^{'} X_{i})) \overset{i, i, d}{=} \prod_{i = 1}^{n} m_{X} (t / n) = \exp (t^{'} μ + t^{'} Σ t / 2 n) \end{aligned}

$\mathbf{Y} \sim N_{k}(\boldsymbol{\mu}, \Sigma / n)$

Prop 3.9.8 (Normal r'X -> Normal X)

$\x \in R^{k}$ $\mathbf{r}^{\prime} \mathbf{X}$ $\mathbf{r} \in R^{k}$ $\mathbf{X} \sim N_{k}(\boldsymbol{\mu}, \Sigma)$ $(\boldsymbol{\mu}, \Sigma)$ .

Proof $E\left(\mathbf{r}^{\prime} \mathbf{X}\right)=\mathbf{r}^{\prime} E(\mathbf{X})$ $\operatorname{Var}\left(\mathbf{r}^{\prime} \mathbf{X}\right)=\mathbf{r}^{\prime} \operatorname{Var}(\mathbf{X}) \mathbf{r}$ $(\boldsymbol{\mu}, \Sigma)=(E(\mathbf{X}), \operatorname{Var}(\mathbf{X}))$ . Now

m_{r^{'} X} (t) = \exp (t r^{'} μ + t^{2} r^{'} Σ r / 2) = m_{X} (t r)

which implies the result.

E.g. Cauchy

$X \sim$ $E(X)$ $m_{X}$ does not exist
$c_{X}(t)=\exp (-|t|)$
$X_{1}, \ldots, X_{n}$ $\d Y=\frac{1}{n} \sum_{i=1}^{n} X_{i}$
$\d c_{Y}(t)=\prod_{i=1}^{n} \exp (-|t| / n)=\exp (-|t|)$ $Y \sim$ Cauchy
note that sampling does not change the distribution, unlike distributions with shorter tails
$c_{X}$ $c_{X}(0)=1$ $c_{X}$ is continuous at 0 since the limit exists:

lim_{t \to 0} c_{X} (t) = lim_{t \to 0} E (\cos (t X)) + i lim_{t \to 0} E (\sin (t X)) = 1

$c_{X}$ $c_{X}(-t)=E(\cos (-t X))=E(\cos (t X))=c_{X}(t)$ $c_{X}$ $n$ $x_{1}, \ldots, x_{n}, t_{1}, \ldots, t_{n}$ $|z| = \sqrt{z \bar z} = \sqrt{z_1^2 + z_2^2}$ )

\sum_{j = 1}^{n} \sum_{k = 1}^{n} x_{j} x_{k} c_{x} (t_{j} - t_{k}) = E ({| \sum_{j = 1}^{n} x_{j} \exp (i t_{j} x) |}^{2}) \geq 0

$c_{X}$ can serve as the autocorrelation function of a weakly stationary process
$a$ $c_{X}(t)=\exp (-a|t|)$ $c_{X}(t)=\exp \left(-a^{2}|t|\right)$

$\mathrm{X}_{1}, \ldots, \mathrm{X}_{n}$ $\mathrm{X}_{i} \sim N_{k_{i}}\left(\boldsymbol{\mu}_{i}, \Sigma_{i}\right)$ $\mathbf{a} \in R^{m}, C_{i} \in R^{m \times k_{i}}$ $Y=\mathbf{a}+\sum C_{i} \mathbf{X}_{i}$ . Exercise III.9.5 E&R 3.4.13 Exercise III.9.6 E&R 3.4.16 $3.4 .20$ Exercise III.9.8 E&R 3.4.29

4. Convergence

Lecture 22

Motivation

applications of probability theory are often concerned with approximations
the underlying idea of "approximation" is the notion of a limit
$\left\{x_{n}: n \in \mathbb{N}\right\}$ $\ex x, N_\epsilon \in R^{1}$ $\varepsilon>0$ $\left|x_{n}-x\right|<\varepsilon$ $n \geq N_{\varepsilon}$ $\d\lim _{n \rightarrow \infty} x_{n}=x$ $x$ $x_{n}$ $n$ $\left|x_{n}-x\right|$ in this approximation
$\left\{X_{n}: n \in \mathbb{N}\right\}$ $X_{n}$ $X$ $\d\lim _{n \rightarrow \infty} X_{n}(\omega)=X(\omega)$ $\omega \in \Omega$
$X_{n} \stackrel{w p 1}{\rightarrow} X$ $P\left(\d\left\{\omega: \lim _{n \rightarrow \infty} X_{n}(\omega)=X(\omega)\right\}\right)=1$
note - this is concerned with the convergence of a sequence of functions

Def. Convergence in Distribution

$X_{n}$ $X$ $\d\lim _{n \rightarrow \infty} F_{X_{n}}(x)=F_{X}(x)$ $x$ $F_{X}$ $X$

$X_{n} \stackrel{d}{\rightarrow} X$ $P_{X_{n}}((a, b])=F_{X_{n}}(b)-F_{X_{n}}(a) \approx F_{X}(b)-F_{X}(a)$ $n$ $a, b$ $F_{X}$

Note convergence in distribution is about approximating the dist of a r.v. and not about approximating the value of the r.v.

E.g. $F_{X}$ ?

$P_{X_{n}}(\{-1 / n\})=P_{X_{n}}(\{1 / n\})=1 / 2$ so

\begin{matrix} F_{X_{n}} (x) = {\begin{cases} 0 & if x < - 1 / n \\ 1 / 2 & if - 1 / n \leq x < 1 / n \\ 1 & if 1 / n \leq x \end{cases} \end{matrix}

$n$ $X$ be degenerate at 0 so

\begin{aligned} F_{X} (x) & = {\begin{cases} 0 & if x < 0 \\ 1 & if 0 \leq x \end{cases} \\ lim_{n \to \infty} F_{X_{n}} (x) & = {\begin{cases} 0 & if x < 0 \\ 1 / 2 & if x = 0 \\ 1 & if 0 < x \end{cases} \end{aligned}

$\d\lim _{n \rightarrow \infty} F_{X_{n}}(x)=F_{X}(x)$ $\d\lim _{n \rightarrow \infty} F_{X_{n}}(0) \neq F_{X}(0)$ $F_{X}$
$X_{n} \stackrel{d}{\rightarrow} X$

Prop 4.1.1 (Series Expansion of CF)

$E\left(|X|^{k}\right)<\infty$ $\d c_{X}(t)=\sum_{j=0}^{k} \frac{(i t)^{j}}{j !} \mu_{j}+o\left(t^{k}\right)$ $o\left(t^{k}\right)$ $t$ $\d\lim _{t \rightarrow 0} o\left(t^{k}\right) / t^{k}=0$ .

Proof $u=e^{i s}, d v=-(x-s)^{n}$ $du = ie^{is}, v = -\d{(x-s)^{n+1}\o n+1}$

\begin{matrix} \begin{aligned} \int u d v & = u v - \int v d u \\ \int_{0}^{x} (x - s)^{n} e^{i s} d s & = \frac{x^{n + 1}}{n + 1} + \frac{i}{n + 1} \int_{0}^{x} (x - s)^{n + 1} e^{i s} d s \end{aligned} \end{matrix}

so with n = 0, we have

\begin{aligned} \int_{0}^{x} (x - s)^{0} e^{i s} d s = & x + i \int_{0}^{x} (x - s)^{1} e^{i s} d s \leftarrow by above \\ {[\begin{array}{c} \frac{e^{i s}}{i} \end{array}]}_{0}^{x} = & x + i (\frac{x^{2}}{2} + \frac{i}{2} \int (x - 2)^{2} e^{i s} d s) \\ \frac{1}{i} (e^{i x} - 1) = & x + \frac{i x^{2}}{2} + \dots + \frac{i^{n - 1} x^{n}}{n!} + \frac{i^{n}}{n!} \int_{0}^{x} (x - s)^{n} e^{i s} d s \\ (e^{i x} - 1) = & \underset{⏟}{i x + \frac{i^{2} x^{2}}{2} + \dots + \frac{i^{n} x^{n}}{n!}} + \frac{i^{n + 1}}{n!} \int_{0}^{x} (x - s)^{n} e^{i s} d s \\ e^{i x} = & \sum_{j = 0}^{n} \frac{(i x)^{j}}{j!} + \frac{i^{n + 1}}{n!} \int_{0}^{x} (x - s)^{n} e^{i s} d s . \end{aligned}

now with n-1, we have

\begin{aligned} \int_{0}^{x} (x - s)^{n - 1} e^{i s} d s & = \frac{x^{n}}{n} + \frac{i}{n} \int_{0}^{x} (x - s)^{n} e^{i s} d s \\ \int_{0}^{x} (x - s)^{n} e^{i s} d s & = \frac{n}{i} (\int_{0}^{x} (x - s)^{n - 1} e^{i s} d s - \frac{x^{n}}{n}) \leftarrow plug in n=0 like above and simplify \\ e^{i x} & = \sum_{j = 0}^{n} \frac{(i x)^{j}}{j!} + \frac{i^{n}}{(n - 1)!} \int_{0}^{x} (x - s)^{n - 1} (e^{i s} - 1) d s \end{aligned}

$\d \left|\int f(x)\right| \le \int \left|f(x)\right|$ , we have

\begin{aligned} | e^{i x} - \sum_{j = 0}^{n} \frac{(i x)^{j}}{j!} | & \leq min {\frac{| x |^{n + 1}}{(n + 1)!}, \frac{2 | x |^{n}}{n!}} \leftarrow | e^{i s} - 1 | \leq | e^{i s} | + | - 1 | = 2 \\ \frac{1}{| t |^{k}} | E (e^{i t X} - \sum_{j = 0}^{k} \frac{(i t X)^{j}}{j!}) | & \leq \frac{1}{| t |^{k}} E (min {\frac{| t X |^{k + 1}}{(k + 1)!}, \frac{2 | t X |^{k}}{k!}}) \leftarrow | E (X) | \leq E (| X |) \\ \frac{1}{| t |^{k}} | c_{X} (t) - \sum_{j = 0}^{k} \frac{(i t)^{j}}{j!} μ_{j} | & \leq E (min {\frac{| t | | X |^{k + 1}}{(k + 1)!}, \frac{2 | X |^{k}}{k!}}) \end{aligned}

$E\left(|X|^{k}\right)<\infty$ $t \rightarrow 0$ , which proves the result by DCT.

Prop 4.1.2 (Continuity Theorem)

$X_{n}$ is a sequence of r.v.'s.

$X_{n} \stackrel{d}{\rightarrow} X$ $c_{X_n}(t) \rightarrow c_{X}(t)$ $t$ .

$c_{X_{n}}(t) \rightarrow c(t)$ $t$ $c$ $c$ $X$ $X_{n} \stackrel{d}{\rightarrow} X$ .

Prop 4.1.3 (Weak Law of Large Numbers)

$X_{n}$ $E\left(X_{i}\right)=\mu \in R^{1}$ $\d\frac{1}{n} S_{n}=\frac{1}{n} \sum_{i=1}^{n} X_{i} \stackrel{d}{\rightarrow} \mu$ $\mu$

Proof $X$ $\mu$ $c_{X}(t)=E(e^{itx}) = e^{i t \mu} = \exp(it\mu)$ which is continuous at 0 . Also,

\begin{aligned} c_{\frac{1}{n} S_{n}} (t) & = E (\exp (\frac{i t}{n} \sum_{i = 1}^{n} X_{i})) \overset{i . i . d}{=} c_{X_{1}}^{n} (\frac{t}{n}) = {(1 + i μ \frac{t}{n} + o (\frac{t}{n}))}^{n} \leftarrow by Prop 4.1.1 \\ = {(1 + i μ \frac{t}{n})}^{n} {(1 + \frac{o (\frac{t}{n})}{1 + i μ \frac{t}{n}})}^{n} \to \exp (i t μ) \end{aligned}

$x_{n} \rightarrow 0$ $n x_{n}$ $\log \left(1+x_{n}\right)^{n}=n \log \left(1+x_{n}\right)=n\left(x_{n}-x_{n}^{2} / 2+x_{n}^{3} / 3-\cdots\right) \rightarrow \lim n x_{n} .$ $\exp(it\mu)$ $o(\frac tn)$ converges to 0. The result follows by (ii) of the Continuity Theorem.

Note $\f{1}{n} S_{n}=\frac{1}{n} \sum_{i=1}^{n} X_{i} \stackrel{w p 1}{\rightarrow} \mu$

$X_{n} \stackrel{w p 1}{\rightarrow} X$ $X_{n} \stackrel{d}{\rightarrow} X$ and so the the SLLN implies the WLLN.

Prop 4.1.4 (Central Limit Theorem)

$X_{n}$ $E\left(X_{i}\right)=\mu \in R^{1}, \operatorname{Var}\left(X_{i}\right)=\sigma^{2}$ $\d Z_{n}=\frac{\frac{1}{n} S_{n}-\mu}{\sigma / \sqrt{n}} \stackrel{d}{\rightarrow} Z \sim N(0,1)$

Proof $\d E\left(\frac{1}{n} S_{n}\right)=\mu,\operatorname{Var}\left(\frac{1}{n} S_{n}\right)=\frac{\sigma^{2}}{n}$ $Z_{n}$ has mean 0 and variance 1.

$Y_{i}=\d{\left(X_{i}-\mu\right) \o \sigma}$ $\d Z_{n}=\frac{1}{\sqrt{n}} \sum_{i=1}^{n} Y_{i}$ .

$Y_{1}, \ldots, Y_{n}$ are i.i.d,

\begin{aligned} c_{Z_{n}} (t) & = c_{Y_{1}}^{n} (\frac{t}{\sqrt{n}}) \\ = {(1 + \frac{i t}{\sqrt{n}} E (Y_{1}) - \frac{t^{2}}{2 n} E (Y_{1}^{2}) + o (\frac{t^{2}}{n}))}^{n} (by Prop IV.1.1) \\ = {(1 - \frac{t^{2}}{2 n} + o (\frac{t^{2}}{n}))}^{n} \to e^{- t^{2} / 2} \end{aligned}

$Z \sim N(0,1)$ and the result follows by the Continuity Theorem.

E.g. Normal approximation to the binomial

$X_{1}, X_{2}, \ldots$ $(p)$ $E\left(X_{i}\right)=p, \operatorname{Var}\left(X_{i}\right)=p(1-p)$ $S_{n}=\sumto n X_i \sim$ $\operatorname{binomial}(n, p)$ $\frac{1}{n} S_{n}=$ $X_{1}, X_{2}, \ldots, X_{n}$ $\d\frac{\frac{1}{n} S_{n}-p}{\sqrt{p(1-p) / n}} \rightarrow N(0,1)$

$n$ $Z \sim N(0,1)$

\begin{aligned} Φ (b) - Φ (a) & = P (a < Z \leq b) \approx P (a < \frac{\frac{1}{n} S_{n} - p}{\sqrt{p (1 - p) / n}} \leq b) \\ = P (n p + a \sqrt{n p (1 - p)} < S_{n} \leq n p + b \sqrt{n p (1 - p)}) \end{aligned}

Note $a, b$ reflect how long the interval about the mean is in terms of standard deviations

E.g. Poisson approximation to the binomial (rare events)

$\left(p_{n}\right)$ $X_{1}, X_{2}, \ldots, X_{n}$ $n\ o(1 / n) \rightarrow 0$ $o(1 / n) \rightarrow 0$ $p_{n}=\lambda / n+o(1 / n) \rightarrow 0$

$S_{n} \sim \operatorname{binomial}(n, \lambda / n+o(1 / n))$ ,

\begin{matrix} \begin{aligned} P (S_{n} = k) \\ = (\begin{matrix} n \\ k \end{matrix}) {(\frac{λ}{n} + o (1 / n))}^{k} {(1 - \frac{λ}{n} - o (1 / n))}^{n - k} \\ = \frac{n (n - 1) \dots (n - k + 1)}{n^{k}} \frac{λ^{k}}{k!} {(1 + \frac{n o (1 / n)}{λ})}^{k} \cdot {(1 - \frac{λ}{n})}^{n} {(1 - \frac{o (1 / n)}{1 - \frac{λ}{n}})}^{n} {(1 - \frac{λ}{n} - o (1 / n))}^{- k} \\ = [1 (1 - \frac{1}{n}) \dots (1 - \frac{k}{n} + \frac{1}{n})] {(1 + \frac{n o (1 / n)}{λ})}^{k} {(1 - \frac{o (1 / n)}{1 - \frac{λ}{n}})}^{n} {(1 - \frac{λ}{n} - o (1 / n))}^{- k} \frac{λ^{k}}{k!} {(1 - \frac{λ}{n})}^{n} \\ \to 1 \cdot 1 \cdot 1 \cdot 1 \cdot \frac{λ^{k}}{k!} e^{- λ} = \frac{λ^{k}}{k!} e^{- λ} \end{aligned} \end{matrix}

$(\lambda)$ $y \in(k, k+1)$ $k \in \mathbb{N}$ $P\left(S_{n} \leq y\right) \rightarrow \d\sum_{i=0}^{k} \frac{\lambda^{i}}{i !} e^{-\lambda}$

$S_{n} \stackrel{d}{\rightarrow}$ $(\lambda)$

Lecture 23

Def. Convergence in Probability

$X_{n}$ $X$ $\d\lim _{n \rightarrow \infty} P\left(\left|X_{n}-X\right| >\delta\right)=0$ $\delta>0$ $X_{n} \stackrel{P}{\rightarrow} X$

Note $X_{n} \stackrel{w p 1}{\rightarrow} X$ $\d P\left(\left\{\omega: \lim _{n \rightarrow \infty} X_{n}(\omega) \neq X(\omega)\right\}\right)=0$

$X_{n} \stackrel{P}{\rightarrow} X$ $\delta>0, \varepsilon>0$ $N_{\delta, \varepsilon}$ $P\left(\left\{\omega:\left|X_{n}(\omega)-X(\omega)\right| >\delta\right\}\right)<\varepsilon$ $n>N_{\delta, \varepsilon}$

Prop 4.2.1 (Convergence Hierarchy)

$X_{n} \stackrel{w p 1}{\rightarrow} X \implies X_{n} \stackrel{P}{\rightarrow} X \implies X_{n} \stackrel{d}{\rightarrow} X$

Note The converse is false.

Proof (convergence wp1 implies convergence in P)

$A_{m, n}=\left\{\omega:\left|X_{n}(\omega)-X(\omega)\right| >\f1m\right\}$ $\d\limsup _{n} A_{m, n}=\left\{\omega:\left|X_{n}(\omega)-X(\omega)\right|>\frac1m \t{ for infinitely many n}\right\}$

$\d 0=P\left(\lim \sup _{n} A_{m, n}\right)=P\left(\N_{k=1}^{\infty} \U_{n=k}^{\infty} A_{m, n}\right) =\lim _{k \rightarrow \infty} P\left(\U_{n=k}^{\infty} A_{m, n}\right) \geq \lim _{k \rightarrow \infty} P\left(A_{m, k}\right)$

$\lim _{k \rightarrow \infty} P\left(A_{m, k}\right)=0$ $X_{n} \stackrel{P}{\rightarrow} X$ .

Proof (convergence in P implies convergence in dist)

\begin{aligned} F_{X_{n}} (x) = & P (X_{n} \leq x, X \leq x + δ) + P (X_{n} \leq x, X > x + δ) \\ \leq & F_{X} (x + δ) + P (| X_{n} - X | > δ) \\ F_{X} (x - δ) = & P (X_{n} \leq x, X \leq x - δ) + P (X_{n} > x, X \leq x - δ) \\ \leq & F_{X_{n}} (x) + P (| X_{n} - X | > δ) \\ F_{X} (x) \leq & F_{X_{n}} (x + δ) + P (| X_{n} - X | > δ) \end{aligned}

$F_{X_n}(x) \to F_X(x)$ at every continuity point. Subtract term from LHS and a smaller term from RHS:

\begin{aligned} F_{X_{n}} (x) - F_{X} (x) \leq F_{X} (x + δ) - F_{X} (x - δ) + P (| X_{n} - X | > δ) \\ F_{X} (x) - F_{X_{n}} (x) \leq F_{X} (x + δ) - F_{X} (x - δ) + P (| X_{n} - X | > δ) \end{aligned}

$\varepsilon>0$ $N_{\delta, \varepsilon}$ $P\left(\left|X_{n}-X\right|>\delta\right)<\varepsilon / 2$ $n>N_{\delta, \varepsilon}$ , and so

| F_{X} (x) - F_{X_{n}} (x) | \leq F_{X} (x + δ) - F_{X} (x - δ) + ε / 2

$x$ $F_{x}$ $\delta$ $\left|F_{X}(x+\delta)-F_{X}(x-\delta)\right| \leq \varepsilon / 2$ $\left|F_{X}(x)-F_{X_{n}}(x)\right| \leq \varepsilon/2 + \varepsilon/2 = \varepsilon$ $\varepsilon$ is arbitrary this implies the result.

E.g. $X_{n} \stackrel{d}{\rightarrow} X$ $X_{n} \stackrel{P}{\rightarrow} X$

$X_{n}=Z \sim N(0,1), X=-Z \sim N(0,1)$ $X_{n} \stackrel{d}{\rightarrow} X$
${P\left(\left|X_{n}-X\right|>\delta\right)=P(2|Z|>\delta)=P(|Z|>\delta/2) = 2(1-\Phi(\delta / 2))}$ $X_{n} \stackrel{P}{\not\to} X$

Prop 4.2.2 (Convergence to a Constant)

$X_{n}\stackrel{d}{\rightarrow}\mu$ $X_{n} \stackrel{P}{\rightarrow} \mu$ .

Proof $X_{n} \stackrel{P}{\rightarrow} \mu$ $X_{n} \stackrel{d}{\rightarrow} \mu$ . For the other direction,

\begin{aligned} P (| X_{n} - μ | \leq δ) & = P (μ - δ \leq X_{n} \leq μ + δ) \\ = (F_{X_{n}} (μ + δ) - F_{X_{n}} (μ - δ)) + \underset{\leq F_{X_{n}} (μ - δ) \to 0}{\underset{⏟}{P (X_{n} = μ - δ)}} \\ P (| X_{n} - μ | \leq δ) & \to 1 - 0 + 0 = 1 \end{aligned}

$X_{n} \stackrel{P}{\rightarrow} \mu$ .

Prop 4.2.3 (Slutsky's Theorem)

$X_{n} \stackrel{d}{\rightarrow} X$ $Y_{n} \stackrel{d}{\rightarrow} c$ $X_{n}+Y_{n} \stackrel{d}{\rightarrow} X+c$ $X_{n} Y_{n} \stackrel{d}{\rightarrow} c X$ $c \neq 0, X_{n} / Y_{n} \stackrel{d}{\rightarrow} X / c$

Prop 4.2.4 (Cont. Function Convergence)

$X_{n} \stackrel{d}{\rightarrow} c$ $h$ $c$ $h\left(X_{n}\right) \stackrel{d}{\rightarrow} h(c)$ .

Proof $\varepsilon>0$ $\delta>0$ $|h(x)-h(c)| \leq \varepsilon$ $|x-c| \leq \delta$ . Therefore

P (| h (X_{n}) - h (c) | > ε) \leq P (| X_{n} - c | > δ) \to 0

E.g. $X_{1}, X_{2}, \ldots$ $\mu$ $\sigma^{2}$ . By CLT,

\frac{\frac{1}{n} \sum_{i = 1}^{n} X_{i} - μ}{σ / \sqrt{n}} = \frac{\sqrt{n} (\bar{X} - μ)}{σ} \overset{d}{\to} N (0, 1)

\begin{aligned} S^{2} = \frac{\sum_{i = 1}^{n} {(X_{i} - \bar{X})}^{2}}{n - 1} = \frac{n}{n - 1} (\frac{1}{n} \sum_{i = 1}^{n} X_{i}^{2} - {\bar{X}}^{2}) \\ ∵ \frac{n}{n - 1} \overset{w p 1}{\to} 1, \frac{1}{n} \sum_{i = 1}^{n} X_{i}^{2} \overset{d}{\to} σ^{2} + μ^{2}, {\bar{X}}^{2} \overset{d}{\to} μ^{2} \\ ∴ S^{2} \overset{d}{\to} σ^{2} ⟹ S \overset{d}{\to} σ \end{aligned}

$\d \frac{\sqrt{n}(\bar{X}-\mu)}{S}=\frac{\sigma}{S} \frac{\sqrt{n}(\bar{X}-\mu)}{\sigma} \stackrel{d}{\rightarrow} N(0,1)$ by Slutsky

Note $X_{1}, X_{2}, \ldots$ $N\left(\mu, \sigma^{2}\right)$ $(n) \stackrel{d}{\rightarrow} N(0,1)$

Def. Convergence in Expectation of Order r

$X_{n}$ $r\ (\geq 1)$ $X$ $E\left(\left|X_{n}\right|^{r}\right)<\infty$ $n$ $\d\lim _{n \rightarrow \infty} E\left(\left|X_{n}-X\right|^{r}\right)=0$ $X_{n} \stackrel{r}{\rightarrow} X$

Prop 4.3.1 (Order r implies order s; order 1 implies P)

$X_{n} \stackrel{r}{\rightarrow} X$ $X_{n} \stackrel{s}{\rightarrow} X$ $1 \leq s \leq r$ .

Proof $\d{d^{2} x^{p} \o d x^{2}}=p(p-1) x^{p-2} \geq 0$ $x \geq 0, p \geq 1$ $x^{r / s}$ $[0, \infty)$ . Therefore,

E ({| X_{n} - X |}^{r}) = E ({({| X_{n} - X |}^{s})}^{\frac{r}{s}}) \overset{Jensen}{\geq} {(E ({| X_{n} - X |}^{s}))}^{\frac{r}{s}}

Since LHS goes to 0, RHS must also go to 0.

$X_{n} \stackrel{1}{\rightarrow} X$ $X_{n} \stackrel{P}{\rightarrow} X$ .

Proof $\delta>0$

P (| X_{n} - X | > δ) \overset{Markov}{\leq} \frac{E (| X_{n} - X |)}{δ} \to 0 .

Note the converse to this proposition is false

Prop 4.3.2 (Order 2)

$r=2$ $L^{2}(P)=\left\{X: X \text { is a r.v. and } E\left(X^{2}\right)<\infty\right\}$

$\<\cdot, \cdot\>: L^{2}(P) \times L^{2}(P) \rightarrow R^{1}$ $\<X, Y\> = E(X Y)$ $(E(X Y))^{2} \stackrel{\text {Cauchy-Schwartz}} \leq E\left(X^{2}\right) E\left(Y^{2}\right)<\infty$

$\|X\| = \<X, X\>^\frac12$

$X, Y \in L^{2}(P)$ $+b X+c Y \in L^{2}(P)$ $b, c .$

$<\cdot \cdot>$ $L^{2}(P)$

$\|\cdot\|$ $L^{2}(P)$ .

Proof

Exercise IV.3.1.

$\theta$ $X-E(X), Y-E(Y) \in L^{2}(P)$ satisfies

\begin{array}{l} cos θ = \frac{⟨ X - E (X), Y - E (Y) ⟩}{∥ X - E (X) ∥ ∥ Y - E (Y) ∥} = \frac{E ((X - E (X)) (Y - E (X)))}{E ((X - E (X))^{2})^{\frac{1}{2}} E ((Y - E (Y))^{2})^{\frac{1}{2}}} = \frac{Cov (X, Y)}{S d (X) S d (Y)} = Corr (X, Y) \end{array}

$L^2$ Law of large Numbers)

$X_{n}$ $L^{2}(P)$ $\d\frac{1}{n} \sum_{i=1}^{n} X_{i} \stackrel{2}{\rightarrow} E\left(X_{1}\right)$ .

Proof

E ({(\frac{1}{n} \sum_{i = 1}^{n} X_{i} - E (X_{1}))}^{2}) = Var (\frac{1}{n} \sum_{i = 1}^{n} X_{i}) = \frac{Var (X_{1})}{n} \to 0

$X_{n} \stackrel{2}{\rightarrow} X$ $X_{n} \stackrel{1}{\rightarrow} X$ $X_{n} \stackrel{P}{\rightarrow} X$ $X_{n} \stackrel{d}{\rightarrow} X$

Note $L^{2}$

Summary (wp1 => p => d)

Strong convergence (wp1, or almost sure convergence)

$X_n$ $X$ are close

Weak convergence (convergence in distribution)

their probability distributions are close

Convergence in probability (in between strong and weak convergence)

$X_n$ $X$ are close with high probability, so their probability distributions are close

5. Gaussian Process

Lecture 24 (Discrete Time)

Recall Def. Stationary Process:

$\left\{t_{1}, \ldots, t_{n}\right\} \subset T$ $\left\{\left(t, X_{t}\right): t \in T\right\}$ is a Gaussian process if

\begin{matrix} (\begin{matrix} X_{t_{1}} \\ ⋮ \\ X_{t_{n}} \end{matrix}) \sim N_{n} ((\begin{matrix} μ (t_{1}) \\ ⋮ \\ μ (t_{n}) \end{matrix}), (\begin{array}{ccc} σ (t_{1}, t_{1}) & \dots & σ (t_{1}, t_{n}) \\ ⋮ & ⋮ \\ σ (t_{n}, t_{1}) & \dots & σ (t_{n}, t_{n}) \end{array})) \end{matrix}

$\mu: T \rightarrow R^{1}$ $\sigma: T \times T \rightarrow R^{1}$

$T \subset R^{1}$ $\mu$ $\sigma\left(t_{i}, t_{j}\right)=\kappa\left(t_{i}-t_{j}\right)$ $\kappa: T \rightarrow R^{1}$

Def. Strictly Stationary Process

$\left\{t_{1}, \ldots, t_{n}\right\} \subset T\subset R^{1}$ $\left(X_{t_{1}+h}, \ldots, X_{t_{n}+h}\right) \sim\left(X_{t_{1}}, \ldots, X_{t_{n}}\right)$ $\left\{t_{1}+h, \ldots, t_{n}+h\right\} \subset T$

Note $\sigma\left(t_{i}+h, t_{j}+h\right)=\kappa\left(t_{i}-t_{j}\right)=\sigma\left(t_{i}, t_{j}\right)$ (similar to how covariance = 0 implies statistical independence with joint normality)

Def. Autoregressive process of order 1

$N\left(0, \tau^{2}\right)$ $\left\{Z_{n}: n \in \mathbb{Z}\right\}$ $X_{n}=\alpha X_{n-1}+Z_{n}$ $X_{n-1}$ $Z_{n}$

Proof Does there exist a stationary Gaussian process satisfying the definition above?

Assume there is, then

\begin{matrix} \begin{aligned} X_{n} = & α X_{n - 1} + Z_{n} = α^{2} X_{n - 2} + α Z_{n - 1} + Z_{n} \\ = & α^{k} X_{n - k} + α^{k - 1} Z_{n - k + 1} + \dots + Z_{n} \leftarrow after k steps \\ = & α^{k} X_{n - k} + \sum_{j = 0}^{k - 1} α^{j} Z_{n - j} (*) \end{aligned} \end{matrix}

$|\alpha|<1$ $k \rightarrow \infty$ , we have

\begin{aligned} E (X_{n}) & = α^{k} E (X_{n - k}) + 0 + . . . + 0 \leftarrow {X_{n} : n \in Z} is stationary so constant mean and var \\ = α^{k} E (X_{0}) \to 0 \\ Var (α^{k} X_{n - k}) & = α^{2 k} E (X_{n - k}^{2}) = α^{2 k} E (X_{0}^{2}) \to 0 \end{aligned}

$\d E\left(\left(X_{n}-\sum_{j=0}^{k-1} \alpha^{j} Z_{n-j}\right)^{2}\right)=\alpha^{2 k} E\left(X_{n-k}^{2}\right) \rightarrow 0$ $k\to\infty$
$\d X_{n}-\sum_{j=0}^{k-1} \alpha^{j} Z_{n-j} \stackrel{2}{\rightarrow} 0$ $\d X_{n}=\sum_{i=0}^{\infty} \alpha^{i} Z_{n-i}$
$X_{n}=\alpha X_{n-1}+Z_{n}$ $X_{n}$ $\d\sum_{i=0}^{\infty} \alpha^{i} Z_{n-i}(\omega)$ $\omega \in \Omega$ $X_{n}^{-1} B \in \mathcal{A}$ $B \in \mathcal{B}^{1}$ ?
$\d\sum_{i=0}^{\infty}\left|\alpha^{i} Z_{n-i}\right|$ and let

\begin{aligned} A_{b} & = {ω : \sum_{i = 0}^{\infty} | α^{i} Z_{n - i} (ω) | \leq b} = \cap_{m = 0}^{\infty} {ω : \sum_{i = 0}^{m} | α^{i} Z_{n - i} (ω) | \leq b} \\ A & = {ω : \sum_{i = 0}^{\infty} | α^{i} Z_{n - i} (ω) | = \infty} = \cap_{b = 1}^{\infty} {ω : \sum_{i = 0}^{\infty} | α^{i} Z_{n - i} (ω) | > b} \end{aligned}

$A_{b} \in \mathcal{A}$ $A = (A_b)^C \in \sa$ $\d\sum_{i=0}^{\infty}\left|\alpha^{i} Z_{n-i}\right|$ $A$ has probability 0.

\begin{aligned} E (\sum_{i = 0}^{\infty} | α |^{i} | Z_{n - i} |) & = E (lim_{m \to \infty} \sum_{i = 0}^{m} | α^{i} Z_{n - i} |) \overset{MCT}{=} lim_{m \to \infty} E (\sum_{i = 0}^{m} | α^{i} Z_{n - i} |) \\ = E (| Z_{0} |) lim_{m \to \infty} \sum_{i = 0}^{m} | α |^{i} = E (| Z_{0} |) (1 - | α |)^{- 1} < \infty \end{aligned}

$P(A)=0$ $A$ $\Omega$ $\d\sum_{i=0}^{\infty}\left|\alpha^{i} Z_{n-i}\right|$ $\d\sum_{i=0}^{\infty}\left(\alpha^{i} Z_{n-i}\right)_{+}, \sum_{i=0}^{\infty}\left(\alpha^{i} Z_{n-i}\right)_{-}$ $X_n$ (decomposed below) is also a r.v.

X_{n} = \sum_{i = 0}^{\infty} α^{i} Z_{n - i} = \sum_{i = 0}^{\infty} {(α^{i} Z_{n - i})}_{+} - \sum_{i = 0}^{\infty} {(α^{i} Z_{n - i})}_{-}

$\d\sum_{i=0}^{\infty} \alpha^{i} Z_{n-i}$ converges wp1, we can apply DCT:

E (X_{n}) = E (lim_{m \to \infty} \sum_{i = 0}^{m} α^{i} Z_{n - i}) \overset{DCT}{=} lim_{m \to \infty} E (\sum_{i = 0}^{m} α^{i} Z_{n - i}) = 0

$\operatorname{Cov}(X, Y)$ $\operatorname{Var}(X)<\infty$ $\operatorname{Var}(Y)<\infty$

\begin{aligned} Var (X_{n}) = E (X_{n}^{2}) = E ({(\sum_{i = 0}^{\infty} α^{i} Z_{s - i})}^{2}) \leq E ({(\sum_{i = 0}^{\infty} | α^{i} Z_{s - i} |)}^{2}) = E (lim_{m \to \infty} {(\sum_{i = 0}^{m} | α^{i} Z_{s - i} |)}^{2}) \\ \overset{M C T}{=} & lim_{m \to \infty} E ({(\sum_{i = 0}^{m} | α^{i} Z_{s - i} |)}^{2}) = lim_{m \to \infty} E (\sum_{i = 0}^{m} {| α^{i} Z_{s - i} |}^{2} + 2 \sum_{0 \leq i < j \leq m} \underset{X Y}{\underset{⏟}{| α^{i} Z_{s - i} | | α^{j} Z_{s - j} |}}) \\ ↓ since | X Y | \leq X^{2} + Y^{2} \\ \leq & lim_{m \to \infty} E (5 \sum_{i = 0}^{m} {| α^{i} Z_{s - i} |}^{2}) = 5 E ({| Z_{0} |}^{2}) {(1 - | α |^{2})}^{- 1} < \infty \end{aligned}

$\operatorname{Cov}(X, Y)$ $\sigma(s, t)=\operatorname{Cov}\left(X_{s}, X_{t}\right)$ $s$ $t$
$\left\{X_{n}: n \in \mathbb{Z}\right\}$ $s \geq t$ $E\left(Z_{i} Z_{j}\right)=0$ $i \neq j, E\left(Z_{i}^{2}\right)=\tau^{2}$ $i = j$

\begin{aligned} σ (s, t) & = Cov (X_{s}, X_{t}) = E (\sum_{i = 0}^{\infty} α^{i} Z_{s - i} \sum_{j = 0}^{\infty} α^{j} Z_{t - j}) = \sum_{i = 0}^{\infty} \sum_{j = 0}^{\infty} α^{i + j} E (Z_{s - i} Z_{t - j}) \leftarrow 0 unless s-i = t-j \\ = \underset{{(i, j) : s - i = t - j}}{\sum^{\infty} \sum^{\infty}} α^{i + j} E (Z_{s - i} Z_{t - j}) = \sum_{i = s - t}^{\infty} α^{2 i + t - s} E (Z_{s - i}^{2}) = τ^{2} α^{s - t} \sum_{i = 0}^{\infty} α^{2 i} = \frac{τ^{2} α^{| s - t |}}{1 - α^{2}} \end{aligned}

$\left\{X_{n}: n \in \mathbb{Z}\right\}$ is weakly stationary. But is it a Gaussian process?
$n_{1}<\cdots<n_{k} \in \mathbb{Z}$ $\mathbf{a}=\left(a_{1}, \ldots, a_{k}\right)^{\prime} \in R^{k}$ $\d Y=\sum_{j=1}^{k} a_{j} X_{n_{j}}$ $\exp \{i x\}$ ,
$\begin{matrix} \begin{aligned} c_{Y} (t) & = E (\exp {i t Y}) \\ = E (lim_{m \to \infty} \exp {i t \sum_{j = 1}^{k} a_{j} \sum_{l = 0}^{m} α^{'} Z_{n_{j} - 1}}) \overset{DCT}{=} lim_{m \to \infty} E (\exp {i t \sum_{j = 1}^{k} a_{j} \sum_{l = 0}^{m} α^{'} Z_{n_{j} - 1}}) \end{aligned} \end{matrix}$
$\left(Z_{n_{j}}, Z_{n_{j}-1}, \ldots, Z_{n_{j}-m}\right)^{\prime} \sim N_{m+1}(\mathbf{0}, \tau^2I)$ $\d\lim _{m \rightarrow \infty} \operatorname{Cov}\left(\sum_{l=0}^{m} \alpha^{\prime} Z_{n_{i}-l}, \sum_{l=0}^{m} \alpha^{\prime} Z_{n_{j}-l}\right)=\operatorname{Cov}\left(X_{n_{i}}, X_{n_{j}}\right)$
$c_{Y} (t) = \exp (- a^{'} (σ (n_{i}, n_{j})) a / 2)$
$Y \sim N\Big(0, \mathbf{a}^{\prime}\left(\sigma\left(n_{i}, n_{j}\right) \mathbf{a}\right)\Big)$
$\left(X_{n_{j}}, \ldots, X_{n_{j}}\right)^{\prime} \sim N_{k}\left(\mathbf{0},\sigma\left(n_{i}, n_{j}\right)\right)$ $Y$ $\left\{X_{n}: n \in \mathbb{Z}\right\}$ is a stationary Gaussian process
$n_{0} \in \mathbb{Z}$ $n_{0}=0$ $k$ s.t.

\sum_{j = 0}^{k} α^{j} Z_{n_{0} - j} \sim N (0, τ^{2} \sum_{j = 0}^{k} α^{2 j}) = N (0, τ^{2} \frac{1 - α^{2 (k + 1)}}{1 - α^{2}}) \approx N (0, \frac{τ^{2}}{1 - α^{2}})

$\d X_{n_{0}}=\sum_{j=0}^{k} \alpha^{j} Z_{n_{0}-j}$ $Z_{n_{0}-k}, Z_{n_{0}-k+1}, \ldots, Z_{n_{0}+n} \stackrel{i. i . d}{\sim} N\left(0, \tau^{2}\right)$

$X_{n}=\alpha X_{n-1}+Z_{n}$ $X_{n_{0}}, X_{n_{0}+1}, \ldots, X_{n_{0}+n}$

Lecture 25 (Continuous Time)

Def. Brownian Motion

$\left\{\left(t, W_{t}\right): t \geq 0\right\}$ is a standard Wiener process (another name for Brownian Motion) if
$P\left(W_{0}=0\right)=1$
$0<t_{1}<\cdots<t_{k}$ $W_{t_{1}}, W_{t_{2}}-W_{t_{1}}, \ldots, W_{t_{k}}-W_{t_{k-1}}$ are mutually stat. ind.
$W_{t}-W_{s} \sim N(0, t-s)$ $0 \leq s \leq t$
$\left\{\left(t, X_{t}\right): t \geq 0\right\}$ $X_{t}=\tau W_{t} \sim N\left(0, \tau^{2}(t-s)\right)$ is a general Wiener process.

$\sigma(s, t)=\tau^{2}$ $(s, t)$ .

Proof $0<t_{1}<\cdots<t_{n}$ $c_{1}, \ldots, c_{n} \in R^{1}$

\begin{matrix} \begin{aligned} \sum_{i = 1}^{n} c_{i} X_{t_{i}} = τ \sum_{i = 1}^{n} c_{i} W_{t_{i}} & = τ [c_{n} (W_{t_{n}} - W_{t_{n - 1}}) + (c_{n - 1} + c_{n}) (W_{t_{n - 1}} - W_{t_{n - 2}}) + \dots + (c_{1} + \dots + c_{n}) W_{t_{1}}] \\ \sim N (0, τ^{2} \sum_{i = 1}^{n} {(\sum_{j = 1}^{n - i + 1} c_{j})}^{2} (t_{i} - t_{i - 1})) \end{aligned} \end{matrix}

$\left(X_{t_{1}}, \ldots, X_{t_{n}}\right)^{\prime}$ is multivariate normal since every linear combination is normal (Prop 3.9.8). Also,

\begin{aligned} σ (s, t) = E (X_{s} X_{t}) = τ^{2} E (W_{s} W_{t}) \overset{s \leq t}{=} τ^{2} E (W_{s} (W_{s} + W_{t} - W_{s})) \\ = & τ^{2} E (W_{s}^{2}) + τ^{2} E (W_{s} (W_{t} - W_{s})) = τ^{2} s + τ^{2} 0 = τ^{2} s = τ^{2} min (s, t) \end{aligned}

$\left(X_{t_{1}}, \ldots, X_{t_{n}}\right)^{\prime} \sim N_{n}\left(\mathbf{0}, \tau^{2}\left(\min \left(t_{i}, t_{j}\right)\right)\right)$ , so by KCT this is a Gaussian process.

Prop 5.2.2 (Alt. Brownian Motion)

$\left\{\left(t, W_{t}\right): t \geq 0\right\}$ also satisfying

$P\left(W_{t}\right.$ $\left.t\right)=1$

$P\left(W_{t}\right.$ $t)=1$

E.g. How does Brownian motion arise? It arises as a limiting process.

$Z_{1}, Z_{2}, \ldots$ $S_{0}=0$ $\d S_{n}=\sum_{i=1}^{n} Z_{i}$ $\left.Z_{i} \sim-1+2 \operatorname{Bernoulli}(1 / 2)\right)$

Prop 5.2.3 (Donsker's Thm / Invariance Principle)

{(t, n^{- 1 / 2} S_{⌊ n t ⌋}) : t \in [0, 1]} \overset{d}{\to} {(t, W_{t}) : t \in [0, 1]}

$n^{-1/2}$ $n$
$T=\left[0, T_{0}\right]$ $\Delta T_{0}=T_{0} / n$ , then

\begin{aligned} {(t, {(Δ T_{0})}^{1 / 2} S_{⌊ t / Δ T_{0} ⌋}) : t \in [0, T_{0}]} \\ = & {(t, T_{0}^{1 / 2} n^{- 1 / 2} S_{⌊ n t / T_{0} ⌋}) : t / T_{0} \in [0, 1]} \overset{d}{\to} {(t, W_{t}) : t \in [0, T_{0}]} \end{aligned}

$t \rightarrow n^{-1 / 2} S_{[n t]}$ are not continuous but using linear interpolation,

t \to n^{- 1 / 2} [(1 - n t + ⌊ n t ⌋) S_{⌊ n t ⌋} + (n t - ⌊ n t ⌋) S_{⌊ n t ⌋ + 1}]

which has continuous sample paths and the same convergence result applies

$\left\{\left(t, W_{t}\right): t \geq 0\right\}$

Def. Diffusion Process

$\left\{\left(t, X_{t}\right): t \geq 0\right\}$ $X_{t}=\alpha+\delta t+\sigma W_{t}$ $\alpha=$ $\delta=\mathrm{drift}$ $\sigma=$ volatility

$11.5 .7,11.5 .8,11.5 .12,11.5 .13 .$

Lecture 26

Recall $\left\{\left(n, X_{n}\right): n \in T\right\}$ $T$ $E\left(X_{n}\right) < \infty$ $E\left(X_{n} \mid X_{1}, \ldots, X_{n-1}\right)=X_{n-1}$ $n \in T$

E.g. Random walks are martingales

$X_{1}, X_{2}, \ldots$ $E\left(X_{1}\right)=0$ $\left\{\left(n, S_{n}\right): n \in \mathbb{N}_{0}\right\}$ $S_{0}=a$ $S_{n}=a+\sum_{i=1}^{n} X_{i}$ , then

\begin{aligned} E (S_{n} ∣ S_{0}, \dots, S_{n - 1}) = & E (S_{n - 1} + X_{n} ∣ S_{1}, \dots, S_{n - 1}) \\ = & E (S_{n - 1} ∣ S_{0}, \dots, S_{n - 1}) + E (X_{n} ∣ S_{1}, \dots, S_{n - 1}) \\ = & S_{n - 1} + E {(X_{n} ∣ X_{1}, \dots, X_{n - 1})}^{X_{i} i_{i}, i . d .} S_{n - 1} + E (X_{n}) = S_{n - 1} \end{aligned}

Prop 6.1 (Martingale Convergence Theorem)

$\left\{\left(n, X_{n}\right): n \in \mathbb{N}\right\}$ $_{n} E\left(\left|X_{n}\right|\right)<\infty$ $X$ $X_{n} \stackrel{w p 1}{\rightarrow} X$ .

Def. Stopping Time

$\left\{\left(n, X_{n}\right): n \in \mathbb{N}_{0}\right\}$ $T: \Omega \rightarrow \mathbb{N}_{0} \cup\{\infty\}$ $\{T=n\}=\{\omega: T(\omega)=n\} \in \mathcal{A}$ $n \in \mathbb{N}_{0}$ .

$\{T=\infty\}=\left(\cup_{n \in \mathbb{N}_{0}}\{T=n\}\right)^{c} \in \mathcal{A}$ $\{T \leq n\}=\cup_{k=0}^{n}\{T=n\} \in \mathcal{A}$

E.g. Hitting Time

$X_{1}, X_{2}, \ldots$ $B \in \mathcal{B}^{1}$
$T_{B}=\min \left\{n: X_{n} \in B\right\}$ $T_{B}=\infty$ $X_{n} \notin B$ $n$
$T_{B}$ $B$
$\left\{T_{B}=n\right\}=\left\{X_{1} \in B^{c}\right\} \cap \cdots \cap\left\{X_{n-1} \in B^{c}\right\} \cap\left\{X_{n} \in B\right\} \in \mathcal{A}$ $T$ is a stopping time

E.g. Clinical trials (simplified)

imagine a a sequence of patients that are suffering from a disease and a drug is given to each patient with the intention of curing the disease
$X_{1}, X_{2}, \ldots$ $(p)$ $p$ $X_{i}=1$ $S_{n} / n=$ $n$ patients have been given the drug
$\left\{\left(n, S_{n} / n\right): n \in \mathbb{N}\right\}$ $B=\left\{(n, k / n): n \geq n_{0}\right.$ $k / n \leq p_{1}$ $\left.k / n \geq p_{2}\right\} \in \mathcal{B}^{2}$ since countable
$T_{B}$ $n_{0}$ , there are too few cures or sufficiently many that it can be declared that the drug is working
$n \geq n_{0}$

{T_{B} = n} = (\cup_{k = n_{0}}^{n - 1} {p_{1} < S_{k} / k < p_{2}}) \cup {S_{k} / k \leq p_{1}} \cup {S_{k} / k \geq p_{2}} \in A

$T_{B}$ is a stopping time

Notes

$\left\{\left(n, X_{n}\right): n \in \mathbb{N}_{0}\right\}$ $T$ for the process that is finite wp1 (a finite stopping time)
$X_{T}=$ the value of the process at the time it is stopped
$X_{T}$ is a r.v.

Proof $X_{T}^{-1}(-\infty, b]=\left\{\omega: X_{T(\omega)}(\omega) \leq b\right\}=\cup_{n=0}^{\infty}\left\{T(\omega)=n, X_{n}(\omega) \leq b\right\}$ which implies the result.

$X_{T}$

Prop 6.3 (Optional Stopping Theorem)

$\left\{\left(n, X_{n}\right): n \in \mathbb{N}_{0}\right\}$ $X_{0}=a, T$ $M>0$ $E\left(X_{T}\right)=a$ .

$P\left(\max \left\{\left|X_{0}\right|, \ldots,\left|X_{n}\right|\right\} \leq M \mid T \geq n\right)=1$ $n$

$P(T \leq M)=1$

E.g. Random walks and hitting times

$\left\{\left(n, S_{n}\right): n \in \mathbb{N}_{0}\right\}$ $a$
$s<a<r$ $T=\min \left(T_{\{r\}}, T_{\{s\}}\right)$
$\{T=n\}=\left(\left\{T_{\{r\}}=n\right\} \cap\left\{T_{\{s\}}>n\right\}\right) \cup\left(\left\{T_{\{r\}} \geq n\right\} \cap\left\{T_{\{s\}}=n\right\}\right)$ $T$ is a hitting time
$E\left(S_{T}\right)=a$