1. Basic Probability

Lecture 1

Prop 1.2.1 (De Morgan's)

$(\Uinf A_i)^C = \Ninf A_i^C$

Proof

Strategy: prove they are subsets of each other

$\omega \in (\Uinf A_i)^C$ $\omega \notin \Uinf A_i \implies \omega \notin A_i \fa i \implies \omega \in A_i^C \fa i \implies \omega \in \Ninf A_i^C$ $(\Uinf A_i)^C \subseteq \Ninf A_i^C$

$\omega \in \Ninf A_i^C$ $\omega \in A_i^C \fa i \implies \omega \notin A_i \fa i \implies \omega \notin \Uinf A_i \implies \omega \in (\Uinf A_i)^C$ $\Ninf A_i^C \subseteq (\Uinf A_i)^C$

$(\Ninf A_i)^C = \Uinf A_i^C$

Proof

$\omega \in (\Ninf A_i)^C$ $\omega \notin \Ninf A_i \implies \omega \notin A_i$ $\implies \omega \in A_i^C$ $\implies \omega \in \Uinf A_i^C$ $\subseteq$ RHS

$\omega \in \Uinf A_i^C$ $\omega \in A_i^C$ $\implies \omega \notin A_i$ $\implies \omega \notin \Ninf A_i, \implies \omega \in (\Ninf A_i)^C$ $\subseteq$ LHS

Def. Power Set

$2^\Omega$ $\Omega$ .

$\#(2^\Omega) = 2^{\#\Omega}$

Def. Sigma Algebra/Field

$\sa \subseteq 2^\Omega$ $\Omega$ $\Omega$ ) that:
contains the null set
$\null \in \sa$
closed under unions
$A_1, A_2, ... \in \sa \implies \Uinf A_i\in \sa$
closed under complementation
$A\in\sa \implies A^C\in \sa$

$\underbrace{\set{\null, \Omega}}_{coarsest\ \sigma\ algebra}\subseteq\t{ all other sigma algebras } \subseteq \underbrace{2^\Omega}_{finest\ \sigma\ algebra}$

Def. Probability Measure

$\Omega$ $\sigma$ $\sa$ $P: \sa \to [0, 1]$ with the following properties:
normed
$P(\Omega) = 1$
countably additive
$P(\Uinf A_i) = \suminf P(A_i)$ $A_i$

To turn finite additivity into countable additivity, add infinitely many null sets.

$P$ $\sa \subseteq 2^\Omega$ .

Prop 1.2.2 (Some Event Must Occur)

$(\Omega, \sa, P)$ $P(\null)$ = 0

Proof

$A_i = \null$ $i = 1, 2, ...$ $A_i$ $\Uinf A_i = \null$

$P(\null) > 0$ $P(\null) = \suminf P(\null) = \infty\.P(\null) = \infty$ $\contra P(\null)\in[0,1]$ $P(\null) = 0$

Lecture 2

Hierarchy $\omega$ $A$ $\sa$ $\sb^k$ )

Prop 1.3.1 (Intersection of Sigma Algebras)

${\sa_\lambda: \lambda \in \Lambda}$ $\sigs$ $\Omega$ $\N_{\lambda\in\Lambda}A_\lambda$ $\sigma$ $\Omega$

Proof

$\N_{\lambda\in\Lambda}A_\lambda$ $\sig$ :

$\null \in \sa_\lambda \fa \lambda \implies \null \in \N_{\lambda\in\Lambda}\sa_\lambda$
$A_1, A_2, ... \in \sa_\lambda \fa \lambda \implies \Uinf A_i \in \sa_\lambda \fa \lambda \implies \Uinf A_i \in \N_{\lambda \in \Lambda A_\lambda}$
$A\in\sa_\lambda \implies A^C \in \sa_\lambda \fa \lambda \implies A^C \in \N_{\lambda\in\Lambda}\sa_\lambda$

$\sig$ .

Def. Sigma Algebra Generated by C

$\sa(\sc)$ $\sigs$ $\sc \subseteq 2^\Omega$
$\sigma$ $\Omega$ $\sc$ .

Def. Borel Set

$\sb^k$ $\sigma$ algebra generated by open sets. Formally:

$\sigma$ $\Omega = \R^k$ $a = \[a1\\ \vdots\\ a_k\], b = \[b_1\\ \vdots\\ b_k\] \in \R^k$
$\sb^k = \d\bigtimes_{i=1}^k(a_i, b_i)$
$= (a_1, b_1] \times ... \times (a_k, b_k]$
$=\left\{(x_1, ..., x_k):a_k < x_k < b_k\right\}$

$\sb^k \ne \null$ $2^{\R^k}$ $\sb^k≠2^{\R^k}$ $A⊆\R^k$ that is not a borel set.

Loosely speaking, any set that can be defined explicitly is a borel set.
(Nice) transformations of borel sets are also borel sets.

Def. Ellipsoidal Region

$r$ $x_0$ $B_r(x_0) = \{x:(x-x_0)^T(x-x_0) = \norm{x_i - x_{0i}}^2 = \sumto k (x_i - x_{0i})^2 \le r^2\}\in\sb^k$
$S_r(x_0)$ $\le$ with =.

$y = Ax+b$ $B_r(x_0)$ $A$ $A = \[a_{11}&...&a_{1k}\\ \vdots &\ddots &\vdots\\ a_{k1} &... &a_{kk}\] \in \R^{k\times k}, b\in\R^k$

$\ba AB_r(x_0)+b &= \set{y: y=Ax+b \text{ for some x} \in B_r(x_0)}\\ &=\{y: $A^{-1}(y-b)-x_0$^T$A^{-1}(y-b)-x_0$\le r^2\} \quad\gr{\leftarrow(x = A^{-1}(y-b))}\\ &=\{y: (y\underbrace{-b-Ax_0}_{-\mu})^T \underbrace{(A^{-1})^T(A^{-1})}_{\Sigma^{-1}} (y\underbrace{-b-Ax_0}_{-\mu}) \le r^2\} \quad\gr{\leftarrow(\t{pull out } A^{-1})}\\ &=\{y:(y-\mu)^T \Sigma^{-1} (y-\mu) \le r^2\}\\ &= E_r(\mu, \Sigma) \in \sb^k \ea$
$\mu = Ax_0 + b$ $\Sigma = $(A\inv)^TA\inv$\inv= $(A^T)\inv A\inv$\inv = A^TA$ $r$

Recall:
$A^T = A$
$A\inv A = I$
$v^T A v \ge 0 \fa v \in \R^k$

$\Sigma$ is…

$\Sigma^T = (A^TA)^T = A^T(A^T)^T = A^TA = \Sigma$
$\Sigma\inv\.\Sigma = (AA^T)\inv\.(AA^T) = (A^T)\inv A\inv AA^T = (A^T)\inv A^T = I$
$w \in \R^k, w^T\Sigma w = w^T AA^T w = (A^Tw)^T(A^Tw) = \norm{A^Tw}^2 \ge 0$
- $A^T$ is invertible (transpose of an invertible matrix is invertible)

Note $\mu$ $\Sigma$ is the variance matrix.

Lecture 3

Def. Limit inferior/superior of a Sequence

$A_n \subseteq \Omega$ :
$\liminf A_n = \U_{n=1}^\infty\N_{i=n}^\infty A_i = \{\omega: \omega \t{ is in all but finitely many }A_i \}$
$\omega$ is a member of at least one of the intersections
$\limsup A_n = \N_{n=1}^\infty\U_{i=n}^\infty A_i = \{\omega: \omega \t{ is in infinitely many }A_i \}$
$\omega$ is a member of all the unions

Properties:

$A = \liminf A_n = \limsup A_n$ $A_n \to A$
$\liminf A_n \subseteq \limsup A_n$

Monotone Increasing/Decreasing Sequences

$\N_{i=n}^\infty A_i$ is an increasing sequence of sets (as i increases, fewer sets are intersected, the resulting intersection gets bigger)

$\U_{i=n}^\infty A_i$ is an decreasing sequence of sets (as i increases, fewer sets are unioned, the resulting union gets smaller)

Prop 1.4.1 (Monotone Sequences Converge)

A monotone decreasing sequence of sets converges to their intersection.
$A_n \in \sa \fa n$ $A_1 \supseteq A_2 \supseteq ...$ $A_n \to A = \Ninf A_i$

Proof

Need to prove that lim inf = lim sup:

$A_n \supseteq A_{n-1} \supseteq ...$ $\U_{i=n}^\infty A_i = A_n$ $\limsup A_n = \N_{n=1}^\infty\U_{i=n}^\infty A_i = \N_{n=1}^\infty A_n$

$\N_{i=n}^\infty A_i = \Ninf A_i$ $\liminf A_n = \U_{n=1}^\infty\N_{i=n}^\infty A_i = \Ninf A_i$ (if we union the same set over and over again, we get that set)

$\N_{i=n}^\infty A_i \supseteq \Ninf A_i \fa n$ $\subseteq$
$\omega \in \N_{i=n}^\infty A_i$ $\omega \in A_n \subseteq ... \subseteq A_1$ $\omega \in \Ninf A_i \implies$ $\N_{i=n}^\infty A_i \subseteq \Ninf A_i$
$\N_{i=n}^\infty A_i = \Ninf A_i$

$\liminf A_n = \N_{i=n}^\infty A_i = \Ninf A_i = \limsup A_n$ $A_n \to A = \Ninf A_i$

A monotone increasing sequence of sets converges to their union.
$A_n \in \sa \fa n$ $A_1 \subseteq A_2 \subseteq ...$ $A_n \to A = \Uinf A_i$

Proof

$A_n \subseteq A_{n+1} \subseteq ...$ $\N_{i=n}^\infty A_i = A_n$ $\liminf A_n = \U_{n=1}^\infty\N_{i=n}^\infty A_i = \U_{n=1}^\infty A_n$

$\U_{i=n}^\infty A_i = \Uinf A_i$ $\limsup A_n = \N_{n=1}^\infty\U_{i=n}^\infty A_i = \Uinf A_i$ (intersecting the same set over and over again gives that set)

$\liminf A_n = \limsup A_n$ $A_n \to \Uinf A_i$ .

Prop 1.4.2 (Continuity of P)

$A_n \in \sa \fa n$ $A_n \to A$ $P(A_n) \to P(A)$ $n\to\infty$
Note The converse is true

Proof

By the previous proposition, we know (1) & (2)

$\U_{i=n}^\infty A_i$ is a monotone decreasing sequence, it converges to the intersection
$\U_{i=n}^\infty A_i \to \Ninf \U_{i=n}^\infty A_i = \limsup A_n$

$\N_{i=n}^\infty A_i$ is a monotone increasing sequence, it converges to the union
$\N_{i=n}^\infty A_i \to \Uinf \N_{i=n}^\infty A_i = \liminf A_n$

$P(\Uinf A_i) \to P(\limsup A_n)$ $P(\Ninf A_i) \to P(\liminf A_n)$

$P(\Ninf A_i) \le P(A_n) \le P(\Uinf A_i) \implies P(\liminf A_n) \le P(A_n) \le P(\limsup A_n) \implies P(A_n) \to P(A)$

$A_n$ $A_n \to A = \Uinf A_i$

$B_i \in \sa$ $\bc\ba &B_1 = A_1,\\ &B_2 = A_2\n A_1^C\\ &B_3 = A_3 \n A_2^C\\&...\ea\ec$ $A_n = \Uto n B_i \implies P(A_n) = \sumto n P(B_i)$

$\d\lim_{n\to\infty} P(A_n) = \lim_{n\to\infty}\sumto n P(B_i) = \suminf P(B_i) = P(\Uinf B_i) = P(\Uinf A_i) = P(\lim_{n\to\infty} A_n)$

$A_n$ $A_n^C$ is monotone increasing.

$\d\lim_{n\to\infty}P(A_n^C) = P(\lim_{n\to\infty}A_n^C) = P(\Uinf A_i^C) = P((\Ninf A_i)^C) = 1 - P(\Ninf A_i) = P(\lim_{n\to\infty}A_n)$

Prop 1.4.3 (Prob Measure on a Sigma Algebra)

$P$ $\sa$ $P:\sa \to [0, 1]$ satisfies
$P(\Omega) = 1$
$P$ is additive
$P(A_n) \to P(A)$ $n\to\infty$ $A_n \in \sa \fa n$ $A_n\to A$

Proof

(1) and (2) are contained in the def of probability measure (normed and countably additive)

Combining additivity (2) with continuity (3), we have that P is countably additive:

$A_n \to A \implies \d\lim_{n\to\infty}P(A_n) = P(A)$

$B_n = \Uto n A_i$ $A_1, A_2, ... \in A$ are mutually disjoint.

$B_n$ $\lim B_n = \U_{n=1}^\infty B_n = \U_{n=1}^\infty\Uto n A_i = \Uinf A_i$

$P(\Uinf A_i) = P(\lim B_n) = \lim P(B_n) = \lim P(\Uto n A_i) = \lim \sumto n P(A_i) = \suminf P(A_i)$ ,

$\iff$ finite additivity

Important Note $\equiv$ continuity of P. By ensuring countable additivity, we ensure continuity of P, which is needed when we have an infinite sample space.

Def. Conditional Probability Model

$(\Omega, \sa, P)$ $\sc \in \sa$ $P(C) > 0$ $C$ $(\Omega, \sa, P(\.|C))$ $P(\.|C):\sa\to[0, 1]$ $P(A|C) = \f{P(A\n C)}{{P(C)}}$

Proof

$P(\Omega | C) = \f{P(\Omega \n C)}{P(C)} = \f{P(C)}{P(C)} = 1$

$A_n, A_2, ... \in A$ are mutually disjoint,

$P(\Uinf A_i|C) = \f{P(\Uinf A_i)\n C}{P(C)} = \f{P(\Uinf(A_i \n C))}{P(C)} = \f{\suminf P(A_i \n C)}{P(C)} = \suminf P(A_i|C)$

$P$ $(\Omega, \sa, P(\.|C))$ is a probability model.

Note $\sc, \sa\n C, P(\.|C)$ )

Prop 1.5.1 (LOTP / Thm of Total Prob.)

$C_1, C_2, ...\in \sa$ $P(C_i) > 0 \fa i$ $\Omega = \Uinf C_i$ $C_i \n C_j = \null, \fa i, j$ $A \in \sa, P(A) = \suminf P(C_i)P(A|C_i)$

Proof

$A = \Uinf A \n C_i$ $C_i \n A$
$\suminf P(A\n C_i) = \suminf \f{P(A\n C_i)}{P(C_i)}P(C_i) = \suminf P(A|C_i)P(C_i)$

Fact $C_i$ $\Omega$ $A = \Uinf(A\n C_i)$ $A \n C_i$ are mutually disjoint

Proof

$C_i \n C_j = \null$ $i \ne j$ $(A\n C_i) \n (A \n C_j) = \null$ $A = \Uinf (A \n C_i) = A \n \Uinf C_i$ $\Uinf C_i = \Omega$ )

Lecture 4

Def. Statistically Independent

$(\Omega, \sa, P)$ $A, C \in \sa$ $P(A\n C) = P(A)P(C)$
$P(C) > 0$ $P(A|C) = \f{P(A\n C)}{P(C)} = {P(A)P(C)\o P(C)} = P(A)$

Statistically Independent Sigma Algebras

$A$ $B$ $\sig$ $A: \set{\null, A, A^C, \Omega}$ $\sig$ $\set {\null, B, B^C, \Omega}$

Proof

$C$ $\null$ $C \n \null = \null \fa C$ $P(C\n\null) = P(\null)P(C) = P(\null) = 0$

$C$ $\Omega$ $C\n \Omega = C \fa C$ $P(C\n\Omega) = P(C)P(\Omega) = P(C)$

$A$ $B^C$ $A \n B^C = A \n (A \n B)^C = A \setminus (A \n B)$ $P(A \n B^C) = P(A) - P(A \n B) = P(A) - P(A)P(B) = P(A)(1-P(B)) = P(A)P(B^C)$

$A^C$ $B$ are statistically independent in the same vein.

$A^C$ $B^C$ $\ba P(A^C \n B^C) &= P((A\u B)^C) = 1 - P(A \u B) = 1 - P(A) - P(B) + P(A \n B) = 1 - P(A) - P(B) + P(A)P(B) \\&= (1 - P(A))(1 - P(B)) = P(A^C) P(B^C)\ea$

Def. Mutually Statistically Independent

$(\Omega, \sa, P)$ $\set{\sa_\lambda:\lambda\in\Lambda}$ $\sigs$ $\sa$
$\sa_\lambda$ $P(A_1 \n ... \n A_n) = \d\prod_{i=1}^n P(A_i)$ $\fa n$
$\lambda_1, ..., \lambda_n \in \Lambda$ $A_1 \in \sa_{\lambda_1}, ..., A_n \in \sa_{\lambda_n}$ .

Notes

$\nimplies$ Mutual Independence
$P(A \n B) = P(A \n C) = P(B \n C) \nimplies P(A\n B\n C) = P(A)P(B)P(C)$
$P(A\n B\n C) = P(A)P(B)P(C) \nimplies$ mutual independence

Union of 3 events (Inclusion-Exclusion Principles)

$P(A \u B \u C) = P(A) + P(B) + P(C) - P(A \n B) - P(A \n C) - P(B \n C) + P(A \n B \n C)$

Proof

$\ba P(A\u B\u C) &= P((A\u B) \u C) = P(A\u B) + P(C) - P((A\u B)\n C))\\ &= P(A) + P(B) - P(A\n B) + P(C) - P((A\n C)\u(B \n C)) \\&= P(A) + P(B) + P(C) - P(A\n B) - P(A\n C) - P(B\n C) + P((A\n C) \n (B\n C)) \\ &= P(A) + P(B) + P(C) - P(A\n B) - P(A\n C) - P(B\n C) + P(A\n B \n C)\ea$

Generalized to n events

$P(A_1 \u ... \u A_n) = \sumto n P(A_i) - \sum_{i<j} P(A_i \n A_j) + \sum_{i < j < k} P(A_i \n A_j \n A_k) - ... + (-1)^{n+1}P(A_1\n...\n A_n)$

Proof

$P(A\u B) = P(A) + P(B) - P(A\n B)$

I.H. Assume it's true for n

Consider

$\ba P(A_1 \u ... \u A_n \u A_{n+1}) &= P((A_1 \u ...\u A_n) \u A_{n+1})\\ &= P(A_1\u...\u A_n) + P(A_{n+1}) - P((A_1\u...\u A_n)\n A_{n+1})\\ &= \underbrace{P(A_1 \u...\u A_n)}_{(1)} + P(A_{n+1}) - \underbrace{P((A_1 \n A_{n+1}) \u...\u (A_n \n A_{n+1}))}_{(2)}\ea$

$\ba (1)\quad &P(A_1 \u ... \u A_n)\\ &= \sumto n P(A_i) - \sum_{i<j\le n}P(A_i \n A_j) + ... + (-1)^{n+1}P(A_1 \n ... \n A_n)\ea$

$\ba (2)\quad &P((A_1 \n A_{n+1}) \u ... \u (A_n \n A_{n+1}))\\ &= \sumto n P(A_i \n A_{n+1}) - \sum_{i<j\le n}P(A_i \n A_j \n A_{n+1}) + ... + (-1)^{n+1} P(A_1 \n ... \n A_n \n A_{n+1}) \ea$

Combining the above, we have

$P(A_1 \u ... \u A_{n+1}) = \sumto {n+1} P(A_i) - \sum_{i<j} P(A_i \n A_j) + ... + (-1)^{n+2} P(A_1 \n ... \n A_{n+1})$

Intersection of 3 events

$P(A\n B\n C) = P(A) + P(B) + P(C) - P(A\u B) - P(A\u C) - P(B\u C) + P(A\u B\u C)$

Proof

$\ba &LHS\\ &= 1 - P((A\n B\n C)^C) = 1 - P(A^C\u B^C \u C^C)\\ &= 1 - [P(A^C) + P(B^C) + P(C^C) - P(A^C\n B^C) - P(A^C \n C^C) - P(B^C \n C^C) + P(A^C \n B^C \n C^C)]\\ &= 1 - [3 - P(A) - P(B) - P(C) - (1-P(A\u B)) - (1 - P(A\u C)) - (1 - P(B\u C)) + (1 - P(A\u B\u C))]\\ &= RHS \ea$

Generalized to n events

$P(A_1 \n … \n A_n) = \sumto n P(A_i) - \sum_{i<j} P(A_i \u A_j) + … + (-1)^{n+1}P(A_1 \u … \u A_n)$

2. Random Variables and Stochastic Processes

Lecture 5

Motivation $\Omega$ $X(\omega)$ $a\le X(\omega \le b)$ $X(\omega) \in [a, b]$ $\Omega$ $\R^1$ — this is difficult. To navigate this, we use inverse images.

Def. Inverse Image

$X:\Omega\to\R^1$ $B \subset \R$ $X\inv B = \{\omega\in\Omega:X(\omega)\in B\}$
$\omega$ that get mapped into B.

Note to self $X(\omega) = b \iff X\inv \{b\} = \omega$

E.g. $\Omega = \set{1, 2, 3, 4, 5}$ $X(\omega) = \bc 0 &\omega = 1\\ 0.20 &\omega = 2\\ 0.30 &\omega = 3\\ 0.01 &\omega = 4\\ 0.20 &\omega = 5\ec$

Note that X is not 1-1

$X\inv B$

$\implies X\inv B = \Omega$

$\implies X\inv B = \set{1, 2, 4, 5}$

$\implies X\inv B = {1}$

$-\infty$ $\implies X\inv B = \null$

Property Inverse images preserve Boolean operations.

Proof for Unions

$\omega \in X\inv (B_1 \u B_2)$ $X(\omega) \in B_1 \u B_2$

$\implies \omega \in X\inv B_1$ $\omega \in X\inv B_2$

$\implies \omega \in X\inv B_1 \u X\inv B_2$

$X\inv(B_1 \u B_2) \subseteq X\inv B_1 \u X\inv B_2 \ \boxed1$

$\omega \in X\inv B_1 \u X\inv B_2$ ,

$\implies \omega \in X\inv B_1$ $\omega \in X\inv B_2$

$\implies X(\omega) \in B_1$ $X(\omega) \in B_2$

$\implies X(\omega) \in B_1 \u B_2$

$\implies \omega \in X\inv (B_1 \u B_2)$

$X\inv B_1 \u X\inv B_2 \subseteq X\inv (B_1 \u B_2) \ \boxed2$

$\boxed1$ $\boxed2$ $X\inv (B_1 \u B_2) = X\inv B_1 \u X\inv B_2$ since they are subsets of each other

Proof for Complements

$\omega \in X\inv B^C$ $X(\omega) \in B^C$

$\implies X(\omega) \in B$

$\implies w \notin X\inv B$

$\implies \omega \in (X\inv B)^C$

$X\inv B^C \subseteq (X\inv B)^C \ \boxed1$

$\omega \in (X\inv B)^C$

$\implies \omega \notin X\inv B$

$\implies X(\omega) \notin B$

$\implies X(\omega) \in B^C$

$\implies \omega \in X\inv B^C$

$(X\inv B)^C \subseteq X\inv B^C \ \boxed2$

$\boxed1$ $\boxed2$ $X\inv B^C = (X\inv B)^C$

Property $B_1\n B_2 = \null$ $X\inv B_1$ $X\inv B_2$ are also disjoint.

Proof

$A\n B = \null$ $X\inv A \n X\inv B = X\inv (A\n B) = X\inv \null = \null$

Def. Random Variable

$X:\Omega \to \R^1$ $B \in \sb^1$ $\R^1$ $X\inv B\in\sa$ .
$P(X(\omega) \in B) = P(X\inv B)$

Prop 2.1.1 (Marginal Probability Measure)

$P_X$ $\sb^1$ $P_X(B) = P(X\inv B)$

Proof

$P_X: \sb^1 \to [0,1]$

Normed $P_X(\R^1) = P(X\inv \R^1) = P(\Omega) = 1$
Countably additive $B_1, B_2, ...$ $\sb^1$
$P_X(\Uinf B_i) = P(X\inv \Uinf B_i) = P(\Uinf X\inv B_i) = \suminf P(X\inv B_i) = \suminf P_X(B_i)$

Note $(\R^1, \sb^1, P_X)$

Prop 2.1.2 (Determine whether X is a random variable)

$X\inv(a, b] \in \sa$ $a, b \in \R^1$ $X$ is a random variable.

Proof

$\sb_*^1 = \set{B\in \sb^1: X\inv B \in \sa}$

$\null \in \sb^1$ $X\inv \null \in \sa$ $\null \in \sb_*^1$
$B\in \sb_*^1$ $X\inv B \in \sa \implies (X\inv B)^C = X\inv B^C \in \sa$
$B^C \in \sb^1$ $X\inv B^C \in \sa$ $B^C \in \sb_*^1$
$B_1, B_2, ... \in \sb_*^1$ $X\inv B_1, X\inv B_2, ... \in \sa \implies \Uinf X\in B_i = X\inv \Uinf B_i \in \sa$
$\Uinf B_i \in \sb^1$ $X\inv \Uinf B_i \in \sa$ $\Uinf B_i \in \sb^1_*$

$\sb_*^1$ $\sig$ $\sb^1$ $\quad \boxed{1}$

$(a, b] \in \sb^1_* \fa a,b \in \R^1 \implies \sb^1 \subseteq \sb^1_*$ $\sb^1$ $\sig$ $(a, b]$ $\quad \boxed{2}$

$\boxed{1}$ $\boxed{2}$ $\sb_*^1 = \sb^1 \implies X\inv B \in \sa ,\fa B\in \sb^1 \implies$ X is a random variable.

Examples

$X(\omega) = c$ $(-\infty, b]$ $X\inv (-\infty, b] = \bc\Omega & c\le b \\ \null & c > b\ec \ \in \sa$
$X(\omega) = \omega$ $(-\infty, b]$ $X\inv (-\infty, b] = (-\infty, b] \in \sa$
$X(\omega) = \omega^2$ $(-\infty, b]$ $X\inv (-\infty, b] = \bc[-\sqrt b, \sqrt b] &b\ge0\\ \null &b<0\ec \ \in \sa$
$X(\omega) = \omega^n$ is a r.v. since for any b,
$X\inv (-\infty, b] = \bc \null &b<0\\ [-b^{1\o n}, b^{1 \o n}] &b>0 \ec \ \in \sb^1$ $n\in\Z$ is even, and
$X\inv (-\infty, b] = \bc (-\infty, -|b|^{1\o n}] &b<0\\ (-\infty, b^{1 \o n}] &b>0 \ec \ \in \sb^1$ $n\in\Z$ is odd
$X(\omega_1, ..., \omega_k) = \omega_i$ (projection on the ith coordinate) is a r.v. since for any b,
$X\inv (-\infty, b] = \set{(\omega_1, ..., \omega_k)\in\R^k:\omega_i \le b} = \R^1\times...\times(-\infty,b]\times...\times\R^1\in\sb^n$
- $\Omega = \R^k, \sa = \sb^k$ , so it must be a r.v.

Note $\sa = 2^\Omega$ $X:\Omega\to\R^1$ is a random variable.

Prop 2.1.3 (Sum & Prod of R.V.s are R.V.s)

$\Omega$ , then (1) W = X+Y and (2) W = XY are both random variables.

Proof of (1) W = X + Y

$\omega \in W\inv(-\infty, b] = \set{\omega:X(\omega) + Y(\omega) \le b}$

$c_n \in \Q$ $c_n \downarrow b$ $\ex q\in\Q$ $X(\omega) \le q$ $Y(\omega)\le c_n-q$ ,

$\omega\in $X\inv(-\infty,q]\N Y\inv(-\infty, c_n-q]$\in\sa$

$c_n$ $C_n = \U_{q\in\Q}$\set{\omega: X(\omega)\le q}\N\set{\omega: Y(\omega)\le c_n-q}$$ $\underbrace{W\inv(-\infty, b]}_{\text{set of }\omega} \subset C_n, \fa n$

$\Q$ $C_n$ $\sa$ $C_n\in\sa$

$C_n$ $\d\lim_{n\to\infty}C_n = \N_{n=1}^\infty C_n = W\inv(-\infty, b]\in\sa \implies W = X + Y$ is a r.v.

Proof of (2) W = XY

Suppose b = 0, then

$\ba W\inv (-\infty, 0] &= \set{\omega:X(\omega)\le 0, Y(\omega) \ge 0} \U \set{\omega:X(\omega) \ge 0, Y(\omega) \le 0} \\&= \Big(X\inv(-\infty, 0]\n Y\inv[0, \infty)\Big) \U \Big(X\inv[0,\infty)\n Y\inv(-\infty, 0]\Big) \in \sa \ea$

Suppose b > 0, then

$W\inv(-\infty,b] = W\inv(-\infty, 0] \U W\inv(0, b]$

$W\inv(-\infty, 0] \in \sa$ $W\inv(0, b]\in \sa$ .

$\ba W\inv(0, b] &= \set{\omega:X(\omega) > 0, Y(\omega) > 0, X(\omega)Y(\omega) \le b} \U \set{\omega: X(\omega) < 0, Y(\omega) < 0, X(\omega)Y(\omega) \le b}\\ &= \set{\omega \in \boxed1} \U \set{\omega \in \boxed4}\ea$

Since xy=b is symmetrical over the line y=-x, proving the argument for one of 1 & 4 will suffice.

$\omega \in$ $\boxed1$ $c_n \downarrow b$ $\ex q \in \Q \n (0, \infty)$ $\omega \in X\inv(0, q] \n Y\inv(0, c_n/q] \in \sa$

$C_n = \U_{q\ \in\ \Q\n(0, \infty)} X\inv(0, q] \n Y\inv (0, c_n/q] \in \sa$ $\Q \n (0, \infty)$ is countable.

$C_n \downarrow \fbox1 \implies \fbox1 \in \sa \implies W\inv(0, b]\in \sa$

$W\inv(-\infty,b] \in \sa$ , so W=XY is a r.v.

E.g. $p(X) = \sum_{i=0}^n a_i X^i$ is a r.v. if X is a r.v.

$Y(\omega) = c$ $a_i$ are r.v.'s.

$a_iX^i$ are r.v.'s

$\sum_{i=0}^n a_i X^i$ is a r.v.

Prop 2.1.4 (Sigma Algebra generated by X)

$\sa_X = X\inv\sb^1 = \set{X\inv B:B\in\sb^1}$ $\sig$ $\sa$ $\sig$ $\Omega$ generated by X.
$\sa_X = \sa(\set{X\inv(a, b]: a, b \in \R^1})$

Proof

$\null = X\inv \null \in \sa_X$
$A_1, A_2, ... \in \sa_X$ $\ex B_1, B_2, ... \in \sb^1$ $A_i = X\inv B_i$ .
$\Uinf A_i = \Uinf X\inv B_i = X\inv \Uinf B_i \in \sa_X$ $\Uinf B_i \in \sb^1$ )
$A\in\sa_X$ $\ex B \in \sb^1$ $A = X\inv V$ .
$A^C = (X\inv B)^C = X\inv B^C \in \sa_X$ $B^C \in \sb^1$ )

$\sa_X$ $\sig$ $\sa$

Def. Random Vector

Recall
random variable $X:\Omega \to \R^1$ $B \in \sb^1$ $X\inv B\in\sa$ .
$P(X(\omega) \in B) = P(X\inv B)$ $X\inv B = \set{\omega:X(\omega)\in B}$
random vector $\x: \Omega \to \R^k$ $B\in\sb^k$ $\x\inv B \in\sa$ .
$P(\x(\omega) \in B) = P(\x\inv B)$ $\x\inv B = \set{\omega:\x(\omega)\in B}$

Properties

$\mathbf{aX+bY}$ is a random vector
$\x$ $P_\x:\sb^k \to [0, 1]$ $P_\x(B) = P(\x\inv B)$
$\sig$ $\x$ $\sa_\x = \x\inv \sb^k = \set{\x\inv B: B\in \sb^k} = \sa(\set{\x\inv(\mathbf{a, b]:a, b}\in \R^k})$

Example (Pt. 1)

$\Omega = \set{1, 2, 3}, \sa = 2^\Omega$ $P$

$\x = \[X_1\\X_2\]:\Omega \to \R^2$ $\x(\omega) = \[X_1(\omega)\\X_2(\omega)\]$ $X_1, X_2$ $\ba X_1(1) = 0\\X_1(2) = 0\\X_1(3) = 1 \ea$ $\ba X_2(1) = 1\\X_2(2) = 0\\X_2(3) = 0 \ea$

$\ba X\inv\set{(0, 1)} = \set{1}\\ X\inv\set{(0, 0)} = \set2\\ X\inv\set{(1, 0)} = \set3 \ea \implies X\inv B = \bc\null & (0,1), (0,0), (1,0) \notin B\\ \set1 \text{ or } \set2 \text{ or } \set3 &\text { if only one of }(0,0),(0,1),(1,0) \in B\\ \set{1, 2} \text{ or } \set{1, 3} \text{ or } \set{2, 3} & \text { if only two of }(0,0),(0,1),(1,0) \in B \\ \Omega &(0,1), (0,0), (1,0) \in B\ec$

$P_{\x}(B)=\bc 0 & (0,1),(0,0),(1,0) \in B \\ 1 / 3 & \text { if only one of }(0,1),(0,0),(1,0) \in B \\ 2 / 3 & \text { if only two of }(0,1),(0,0),(1,0) \in B \\ 1 & (0,1),(0,0),(1,0) \notin B \\ \ec$

Example (Pt. 2)

$X_2$ $X_1, X_2$ $X_1(1) = 0\\X_1(2) = 0\\X_1(3) = 1$ $X_2(1) = 1\\X_2(2) = 1\\X_2(3) = 0$ $P_\x$ ?

Only 2 possible outputs now: (0,1) and (1, 0)

$\ba &X\inv\set{(0, 1)} = \set{1, 2}\\ &X\inv\set{(1, 0)} = \set3 \ea \implies X\inv B = \bc\null & (0,1), (1,0) \notin B\\ \set{1, 2} &(0, 1)\in B, (1, 0)\notin B\\ \set3 &(0, 1)\notin B,(1, 0)\in B\\ \Omega &(0, 1), (1, 0) \in B\ec$

$B \in \sb^2, P_\x (B) = \bc0 & (0,1), (1,0) \notin B\\ 2/3 &(0, 1)\in B, (1, 0)\notin B\\ 1/3 &(0, 1)\notin B,(1, 0)\in B\\ 1 &(0, 1), (1, 0) \in B \ec$

Example (Pt. 3)

$P(\set{1}) = \f12, P(\set{2}) = \f13, P(\set{3}) = \f16$ $P_\x$ ?

$P_\x (B) = \bc0 & (0,1), (1,0) \notin B\\ 5/6 &(0, 1)\in B, (1, 0)\notin B\\ 1/6 &(0, 1)\notin B,(1, 0)\in B\\ 1 &o/w\ec$

Prop 2.1.5 (Cartesian Prod of Borel Sets is a Borel Set)

$B_1, ..., B_k \in \sb ^1$ $B_1 \times ...\times B_k = \set{(x_1, ..., x_k)^T | x_i \in B_i, i = 1, ..., k} \in \sb^k$ $\sb^k$ $\sig$ $\R^k$ containing all such sets

Proof

$\R^1\times...\times B_i \times...\times\R^1$ that only restrict the ith coord.

$\set{\R^1\times...\times B_i \times...\times\R^1 | B_i \in \sb^1}$ $\sig$ $\sb^k$

Sub-proof $\sb_y = \set{B\times\R^1\times...\times\R^1 : B\in\sb^1}$

$(-\infty, b]\times\R^1\times...\times\R^1 \in \sb^k \fa b \in \R^1 \implies B\times\R^1\times...\times\R^1 \in \sb^k$

$\null \times \R^1 \times ... \times \R^1 = \null \in \sb^k$
$B_i \times \R^1 \times...\times\R^1\in \sb_y$ $\Uinf B_i\times\R^1\times...\times\R^1 = (\Uinf B_i)\times\R^1\times...\R^1\in \sb_y$ $\Uinf B_i \in \sb^1$
$B \times \R^1 \times ... \times \R^1 \in \sb_y$ $(B\times\R^1\times...\times\R^1)^C = B^C\times\R^1\times...\times\R^1\in\sb_y$ $B^C\in\sb^1$ .

$B_1\times ... \times B_k = \N_{i=1}^k (\R^1\times...\times B_i \times...\times\R^1)\in\sb^k$

$(\mathbf a,\mathbf b] = (a_1, b_1]\times…\times (a_k, b_k]$ $\nexists$ $\sig$ $\R^k$ $\sb^k$ .

Prop 2.1.6 (A Vector of R.V.s is a Random Vector)

$X_i:\Omega \to \R^1$ $i = 1, ..., k$ $\x = (X_1, ..., X_k)^T:\Omega \to \R^k$ is a random vector.

Proof

$B_1, ..., B_k \in \sb^1$ $B_1 \times ... \times B_k \in \sb^k$ . Then we have

$\ba\x\inv(B_1\times...\times B_k) &= \set{\omega: \x(\omega)\in B_1\times...\times B_k}\\ &=\set{\omega : X_i(\omega)\in B_i\ \text{for i = 1, ..., k}}\\ &=\N_{i=1}^k X_i^{-1} B_i \in \sa\ea$

$\x\inv(\mathbf a, \mathbf b] \in \sa \fa \mathbf a, \mathbf b \in \R^k \implies \x\inv B\in \sa \fa B\in\sb^k \implies \x$ is a random vector.

Lecture 6

Def. K-cells

$(\mathbf{a, b}] = \d\bigtimes_{i=1}^k(a_i, b_i]$ $\d\bigtimes_{i=1}^k(-\infty, b_i]$

K-cells are the basic sets we want to assign probabilities to (using random vectors)

For k = 2, (a, b] =

Def. Cumulative Distribution Function (CDF)

$F_\x: \R^k\to[0,1]$ $\x\in\R^k$ $F_\x(x_1, ..., x_k) = P_\x\big((-\infty, x_1]\times...\times(-\infty, x_k]\big) = P_\x\big((-\infty, \x]\big)$

Def. Difference Operator

$g:\R^k\to\R^1$ $\Delta_{a, b}^{(i)}\ g:\R^{k-1}\to\R^1$ $(\Delta_{a,b}^{(i)}\ g)(x_1, ..., x_{i-1}, x_{i+1}, ..., x_k) = g(x_1, ..., x_{i-1}, b, x_{i+1}, ..., x_k) - g(x_1, ..., x_{i-1}, a, x_{i+1}, ..., x_k)$

Prop 2.2.1 (Properties of Distribution Functions)

$F_\x:\R^k\to[0, 1]$ satisfies
$a_i \le b_i$ $i = 1, ..., k$ $P_\x((\mathbf{a, b}]) = \Delta_{a_1, b_1}^{(1)}\Delta_{a_2, b_2}^{(2)}...\Delta_{a_k, b_k}^{(k)}F_\x$
$As\ x_i \downarrow -\infty, F_\x(x_1, ..., x_k)\downarrow 0$
$As\ x_i \uparrow \infty, F_\x(x_1, ..., x_k)\uparrow 1$
$F_\x$ is right continuous
$\delta_i \downarrow 0 \fa i$ $F_\x(x_1 + \delta_1, ..., x_k + \delta_k) \to F_\x(x_1, ..., x_k)$

Proof for (1)

Proof for (2)

Proof for (3)

Thm 2.2.1 (Extension Theorem)

$F:\R^k\to[0,1]$ $\ex$ $P$ $\sb^k$ $F$ $P$

Note $F$ $(\R^k, \sb^k, P)$ $\Omega = \R^k$ $\x(\omega) = \omega$

$P_\x$ $F_\x$

Def. Marginal Distributions

Def. Discrete Probability Models

Prop 2.3.1 (Countably Many Points with Positive Prob)

Prop 2.3.2 (Prob Measure Defined by p)

Def. Multinomial Distribution

Def. Multivariate Hypergeometric Distribution

Lecture 7

Def. Continuous Probability Models

Def. Absolutely Continuous Probability Models

Def. Probability Density Functions (PDF)

Prop 2.4.1 (Properties of A.C. Models)

$f(\b x) \ge 0$ with probability 1
$\d\int_{\R^k} f(\b x) dx = 1$
$\d F(\b x) = F(x_1, ..., x_k) = \int_{-\infty}^{x_k} ... \int_{-\infty}^{x_1} f(z_1, ..., z_k)dz_1...dz_k$
$\d f(\b x) = f(x_1, ..., x_k) = {\del^k F(x_1, ..., x_k)\o\del x_1...\del x_k}$

Prop 2.4.2 (Properties of PDFs)

$f:(\R^k, \sb^k)\to(\R^1, \sb^1)$ $(\R^k, \sb^k, P)$ if

$f(\b x) \ge 0 \fa x$
$\d\int_{\R^k} f(\b x) dx = 1$

Def. Multivariate Normal Distribution

Lecture 8 & 9

$\x \in \R^k$ $\mathbf{Y} = T(\x) \in \R^1$

Discrete case

$\x$ $p_\x$ $p_\mathbf Y (\mathbf y) = P_\mathbf Y$\set{\mathbf y}$ = P_\x(T\inv\{\mathbf y\}) = \d\sum_{\b x\in T\inv\{\mathbf{y}\}}p_\x(\b x)$

Def. Projections (& their Prob Functions)

$k \geq 2$ $\left(y_{1}, y_{2}\right)=T\left(x_{1}, \ldots, x_{k}\right)=\left(x_{1}, x_{2}\right)$

Prob Function Derivation:

To find the probability functions of projections, take the joint probability function, and sum out unwanted variables.

$T^{-1}\{\mathbf{y}\}=T^{-1}\left\{\left(y_{1}, y_{2}\right)'\right\}=\left\{\left(x_{1}, \ldots, x_{k}\right): x_{1}=y_{1}, x_{2}=y_{2}\right\}$

$\ba p_{\mathbf{y}}(\mathbf Y)= p_{\mathbf{Y}}\left(y_{1}, y_{2}\right) =P_{\x}\left(T^{-1}\{\mathbf{y}\}\right)&=\sum_{\b x \in T^{-1}\{\mathbf{y}\}} p_{\x}(\b x) \\ &=\sum_{\left(x_{1}, \ldots, x_{k}\right):x_{1}=y_{1}, x_{2}=y_{2}} p_{\x}\left(x_{1}, \ldots, x_{k}\right) \\ &=\sum_{\left(x_{3}, \ldots, x_{k}\right) \in R^{k-2}} p_{\x}\left(y_{1}, y_{2}, x_{3}, \ldots, x_{k}\right) \expl{fix } x_1, x_2 \text{ to } y_1, y_2 \ea$

$y=T\left(x_{1}, \ldots, x_{k}\right)=x_{2}$

Prob Function Derivation:

$T^{-1}\{\mathbf y\} =T^{-1}\{y\}=\left\{\left(x_{1}, \ldots, x_{k}\right): x_{2}=y\right\}$

$\begin{aligned} p_{\mathbf{y}}(\mathbf Y) = p_{Y}(y) &=\sum_{\left(x_{1}, \ldots, x_{k}\right): x_{2}=y} p_{\x}\left(x_{1}, \ldots, x_{k}\right) \\ &=\sum_{\left(x_{1}, x_{3}, \ldots, x_{k}\right) \in R^{k-1}} p_{\x}\left(x_{1}, y, x_{3}, \ldots, x_{k}\right) \end{aligned}$

Marginal of a Multinomial Random Vector

$\x=\left(X_{1}, \ldots, X_{k}\right)' \sim$ $\left(n, p_{1}, \ldots, p_{k}\right)$ $p_{\x}(\mathbf{a})=\binom n {a_{1}\ \ldots\ a_{k}} p_{1}^{a_{1}} \cdots p_{k}^{a_{k}}$
$\mathbf{a} \in R^{k}$ $a_{i} \in\{0, \ldots, n\}, \text { and } a_{1}+\cdots+a_{k}=n$
$k \geq 2$ $\left(y_{1}, y_{2}\right)=T\left(x_{1}, \ldots, x_{k}\right)=\left(x_{1}, x_{2}\right)$ $\mathbf{Y}=\left(X_{1}, X_{2}\right)'$

$y_{1}, y_{2}, a_{3}, \ldots, a_{k} \in\{0, \ldots, n\}$ $y_{1}+y_{2}+a_{3}+\cdots+a_{k}=n \iff$

$a_{3}, \ldots, a_{k} \in\left\{0, \ldots, n-y_{1}-y_{2}\right\}$ $a_{3}+\cdots+a_{k}=n-y_{1}-y_{2} \quad(* )$

$\ba \text{so}\ \ &p_{Y}\left(y_{1}, y_{2}\right)\\ &=\sum_{\left(a_3, \ldots,a_{k}\right) \text { sat. }( *)}\binom n {y_1\ y_{2}\ a_{3} \ldots\ a_{k}} p_{1}^{y_{1}} p_{2}^{y_{2}} p_{3}^{a_{3}} \cdots p_{k}^{a_{k}}\\ &=\frac{n !}{y_1 ! y_{2} !} p_{1}^{y_{1}} p_{2}^{y_{2}} \sum_{\left(a_{3}, \ldots, a_{k}\right) \text { sat. }(*)} \frac1{a_{3} ! \cdots a_{k} !} p_{3}^{a_{3}} \cdots p_{k}^{a_{k}} \expl{took out terms where } i = 1,2 \\ &=\frac{n !}{y_1 ! y_{2} !\left(n-y_{1}-y_{2}\right) !} p_{1}^{y_{1}} p_{2}^{y_{2}} \sum_{\left(a_{3}, \ldots, a_{k}\right) \text { sat. }(*)} \frac{\left(n-y_{1}-y_{2}\right) !}{a_{3} ! \cdots a_{k} !} p_{3}^{a_{3}} \cdots p_{k}^{a_{k}}\expl {multiplied prev by } \f{(n-y_{1}-y_{2})!}{(n-y_{1}-y_{2})!}\\ &=\binom n {y_{1}\ \ y_{2}\ \ n-y_{1}-y_{2}} p_{1}^{y_{1}} p_{2}^{y_{2}}\left(1-p_{1}-p_{2}\right)^{n-y_{1}-y_{2}} \expl{multiply by } \left(1-p_{1}-p_{2}\right)^{n-y_{1}-y_{2}}\\ &\underbrace{\sum_{\left(a_{3}, \ldots, a_{k}\right) \text{ sat. } \left({ }^{*}\right)}\left(\begin{array}{c}n-y_{1}-y_{2} \\ a_{3} \ldots a_{k}\end{array}\right)\left(\frac{p_{3}}{1-p_{1}-p_{2}}\right)^{a_{3}} \cdots\left(\frac{p_{k}}{1-p_{1}-p_{2}}\right)^{a_{k}}}_{\text{sum of all multinomial}$n-y_1-y_2, {p_3\o1-p_1-p_2}, ..., {p_k\o1-p_1-p_2}$ \text{ probabilities, so } = 1} \expl{divide by } (1-p_{1}-p_{2})^{a_3 + ... + a_k}\\ &=\binom n {y_{1}\ \ y_{2}\ \ n-y_{1}-y_{2}} p_{1}^{y_{1}} p_{2}^{y_{2}}\left(1-p_{1}-p_{2}\right)^{n-y_{1}-y_{2}}\\ \ea$

$\left(X_{1}, X_{2}\right) \sim$ $\left(n, p_{1}, p_{2}, 1-p_{1}-p_{2}\right)$

Binomial(n, p) = Multinomial(n, p, 1-p)

$\x=\left(X_{1}, \ldots, X_{k}\right)' \sim$ $\left(n, p_{1}, \ldots, p_{k}\right)$ $X_{i} \sim$ $\left(n, p_{i}\right)=$ $\left(n, p_{i}, 1-p_{i}\right)$ Note $n$ $k$ $l$ $l+1$ mutually disjoint categories

$\ba P_{X_1}(x_1) &= \d\sum_{x_2 = 0}^{n - x_1}p_{(x_1, x_2, n - x_1 - x_2)}(x_1, x_2, n - x_1 - x_2)\\ &= \d\sum_{x_2 = 0}^{n - x_1}\binom n{x_1\ x_2\ n-x_2-x_2}p_1^{x_1}p_x^{x_2}(1-p_1-p_2)^{n-x_2-x_2}\\ &= \d\sum_{x_2 = 0}^{n - x_1}{n!\o x_1!x_2!(n-x_1-x_2)!}p_1^{x_1}p_x^{x_2}(1-p_1-p_2)^{n-x_2-x_2} \expl {now multiply by }\f{(n-x_1)!}{(n-x_1)!}\\ &= {n!\o x_1!(n-x_1)!}p_1^{x_1}(1-p_1)^{n-x_1}\underbrace{\d\sum_{x_2 = 0}^{n - x_1}\f{(n-x_1)!}{x_2!(x-x_1-x_2)!}({p_2\o1-p_1})^{x_2}(1-{p_2\o1-p_1})^{n-x_1-x_2}}_{\text{sum of all binomial}$n-x_1, {p_2\o1-p_1}$\text{ probabilities, so = 1}}\\ &= {n!\o x_1!(n-x_1)!}p_1^{x_1}(1-p_1)^{n-x_1} \ea$

$X_1 \sim \text{binomial}(n, p_1)$

Sum of sub-Multinomial Random Vector ~ Binomial

$Y=X_{1}+\cdots+X_l$ $l \leq k$ $\left(X_{1}, \ldots, X_{k}\right)' \sim \text { multinomial}\left(n, p_{1}, \ldots, p_{k}\right)$
Note $T$ $T^{-1}\{\mathbf{y}\} \neq \phi$ $p_{\mathbf{Y}}(\mathbf{y})=P_{\x}\left(T^{-1}\{\mathbf{y}\}\right)=p_{\x}\left(T^{-1}\{\mathbf{y}\}\right)$

$Y = X_1 + ... + X_l$ $l$ categories.

$p_1 + ... + p_l$ .

$Y\sim\text{binomial}(n, p_1 + ... + p_l)$

Def. Indicator Function

$A \subset \Omega$ $I_{A}: \Omega \rightarrow R^{1}$ $I_{A}(\omega)= \begin{cases}1 & \text { if } \omega \in A \\ 0 & \text { if } \omega \in A^{c}\end{cases}$

Indicator Variable ~ Bernoulli(P(A))

$(\Omega, \mathcal{A}, P)$ $A \in \mathcal{A}$ $Y=I_{A}$ $Y \sim \operatorname{Bernoulli}(P(A))$

$A \in \sa \implies I_A:\Omega \to \R^1$ $\fa B \in \sb^1, I\inv_A B = \{\omega:I_A(\omega)\in B\} = \bc\null &0, 1 \notin B\\ A &1 \in B, 0 \notin B\\ A^C &0\in B, 1 \notin B\\ \Omega &0, 1\in B\ec$ $\in \sa$

$B \in \sb^1$ $I_A\inv B\in\sa$ $Y = I_A$ is a r.v.

$P_Y(1) = P(I\inv_A\{1\}) = P(\{\omega: I_A(\omega) = 1\}) = P(A) \implies Y\sim\text{Bern}(P(A))$

Transformation Determines Distribution Type

$\mathbf{Y}=T(\x)$ $\x$ is distributed.

E.g. $T(\b x)=\mathbf{c} \in \R^{l}$ $\b x$ $p_{\mathbf{Y}}(\mathbf{y})=P_{\x}\left(T^{-1}\{\mathbf{y}\}\right)= \begin{cases}P_{\x}(\R^k)=1 & \text { if } \mathbf{y}=\mathbf{c} \\ P_{\x}(\null)=0 & \text { if } \mathbf{y} \neq \mathbf{c}\end{cases}$

$\mathbf{Y}$ $\mathbf{c}$

E.g. $X \sim N(0,1)$ $P(X \leq 0)=P(X>0)=1 / 2$

$Y=T(X)=I_{(-\infty, 0]}(X)= \begin{cases}1 & \text { if } X \leq 0 \\ 0 & \text { if } X>0\end{cases}\implies$ $p_\mathbf Y(1)=P(X \leq 0)=1 / 2\\ p_\mathbf Y(0)=P(X>0)=1 / 2$ $\implies$ $Y \sim$ $(1 / 2)$

Absolutely continuous case

$\x \in \R^k$ $f_\x$ $\b Y = T(\x) \in \R^l$ $l \le k$ .

$\b Y$ $f_\b Y$ which we want to determine.

Cdf Method

$T$ $F_\x$ :

$\d f_\b y(y_1) = {\del^k F_\b Y(y_1, ..., y_l)\o \del y_1...\del y_l} = {\del^k P_\x(T\inv\{(-\infty, y_1]\times...\times(-\infty, y_l\})\o \del y_1...\del y_l}$

E.g. $F:\R^2 \to [0, 1]$ $F\left(x_{1}, x_{2}\right)= \begin{cases}0 & x_{1}<0 \text { or } x_{2}<0 \\ 1-e^{-x_{1}}-e^{-x_{2}}+e^{-x_{1}-x_{2}} & x_{1} \geq 0 \text { and } x_{2} \geq 0\end{cases}$

It was proved (in a lec 6 exercise) that this is a cdf (using thm 2.2.1),

$f\left(x_{1}, x_{2}\right)=\frac{\partial^{2} F\left(x_{1}, x_{2}\right)}{\partial x_{1} \partial x_{2}}= \begin{cases}0 & x_{1}<0 \text { or } x_{2}<0 \\ e^{-x_{1}-x_{2}} & x_{1} \geq 0 \text { and } x_{2} \geq 0\end{cases}$

$f$ $\mathrm{pdf}$ :

$f\left(x_{1}, x_{2}\right) \geq 0$ $\left(x_{1}, x_{2}\right)$

(ii) f is normed:

$\begin{aligned}&\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f\left(x_{1}, x_{2}\right) d x_{1} d x_{2}=\int_{0}^{\infty} \int_{0}^{\infty} e^{-x_{1}-x_{2}} d x_{1} d x_2 \\ =& \int_{0}^{\infty} e^{-x_{1}} d x_{1} \int_{0}^{\infty} e^{-x_{2}} d x_{2}=\left(-\left.e^{-x_{1}}\right|_{0} ^{\infty}\right)\left(-\left.e^{-x_{2}}\right|_{0} ^{\infty}\right)=1 \end{aligned}$

$\d F\left(x_{1}, x_{2}\right)=\int_{-\infty}^{x_{1}} \int_{-\infty}^{x_{2}} f\left(z_{1}, z_{2}\right) d z_{1} d z_{2}$

$Y=T\left(X_{1}, X_{2}\right)=X_{1}$ $F_{X_{1}}\left(x_{1}\right)=F\left(x_{1}, \infty\right)= \begin{cases}0 & x_{1}<0 \\ 1-e^{-x_{1}} & x_{1} \geq 0\end{cases}$

$f_{X_{1}}\left(x_{1}\right)=\frac{\partial F_{X_{1}}\left(x_{1}\right)}{\partial x_{1}}= \begin{cases}0 & x_{1}<0 \\ e^{-x_{1}} & x_{1} \geq 0\end{cases}$ $f_{X_{2}}\left(x_{2}\right)=\frac{\partial F_{X_{2}}\left(x_{2}\right)}{\partial x_{2}}= \begin{cases}0 & x_{2}<0 \\ e^{-x_{2}} & x_{2} \geq 0\end{cases}$

$X_{1}$ $X_{2}$ have exponential(1) distributions

E.g. $y = T(x_1, x_2) = x_1 + x_2$ $X_1, X_2$ $f(x_1, x_2) = \bc2 &0<x_1<x_2<1\\0&o/w\ec$

$\begin{aligned}F_y(y) &= P_Y((-\infty, y]) \\&= P_{(X_1, X_2)}(\{(x_1, x_2):x_1 + x_2 \le y\})\\ &= \begin{cases}0 & y<0 \\ \int_{0}^{y / 2} \int_{x_{1}}^{y-x_{1}} 2 d x_{2} d x_{1}=y^{2} / 2 & 0 \leq y \leq 1 \\ 1-\int_{y / 2}^{1} \int_{y-x_{2}}^{x_{2}} 2 d x_{1} d x_{2}=2 y-y^{2} / 2-1 & 1 \leq y \leq 2 \\ 1 & 2<y\end{cases} \\ f_{Y}(y) &= \begin{cases}0 & y \leq 0 \text { or } y \geq 2 \\ y & 0<y<1 \\ 2-y & 1 \leq y<2\end{cases} \end{aligned}$

Change of Variable Method

$T: R^{k} \rightarrow R^{k}$ is 1-1 and smooth (i.e. all 1st order partial derivatives exist and are continuous),

$T(\x)=\left(\begin{array}{c} T_{1}(\b x) \\ \vdots \\ T_{k}(\b x) \end{array}\right)$ Jacobian $J_{T}(\b x)=\left|\det\left(\begin{array}{ccc} \frac{\partial T_{1}(\x)}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\x)}{\partial x_{k}} \\ \vdots & & \vdots \\ \frac{\partial T_{k}(\x)}{\partial x_{1}} & \ldots & \frac{\partial T_{k}(\x)}{\partial x_{k}} \end{array}\right)\right|^{-1}$

$J_{T}(\b x)=\d \lim _{\delta \downarrow 0} \frac{\operatorname{vol}\left(B_{\delta}(\x)\right)}{\operatorname{vol}\left(T B_{\delta}(\x)\right)}$ $J_{T}^{-1}(\b x)$ $T$ $\mathrm{x}$ ,

$J_{T}(\b x)<1$ $T$ $\b x$ $J_{T}(\b x)>1$ $T$ $\b x=T^{-1}(\mathbf{y})$

$\b Y = T(\b X)$ $\delta$ ,

$\begin{aligned} f_{\mathbf{Y}}(\mathbf{y}) & \approx \frac{P_{\mathbf{Y}}\left(T B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}{\operatorname{vol}\left(T B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}=\frac{P_{\x}\left(B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}{\operatorname{vol}\left(B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right.} \frac{\operatorname{vol}\left(B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)}{\operatorname{vol}\left(T B_{\delta}\left(T^{-1}(\mathbf{y})\right)\right)} \approx f_{\x}\left(T^{-1}(\mathbf{y})\right) J_{T}\left(T^{-1}(\mathbf{y})\right) \end{aligned}$

This intuitive argument can be made rigorous to prove the following.

Proposition II.5.1 (Change of Variable)

$T: \R^{k} \rightarrow \R^{k}$ $\mathbf{Y}=T(\x)$ $\x$ $f_{\mathrm{X}}$
$\mathrm{Y}$ $f_{\mathbf{Y}}(\mathbf{y})=f_{\x}\left(T^{-1}(\mathbf{y})\right) J_{T}\left(T^{-1}(\mathbf{y})\right)$

E.g. $f(x) = \f12$ $0<x<2$ $y = T(x) = x^2$ .

$T\inv (y) = y^{1/2}$ $J_T(x) = |\det(2x)|\inv = \f1{2x}$ $x \in (0, 2)$

$J_T(x) = 1$ , we see that T contracts lengths on (0, 1/2) and expands lengths on (1/2, 2)

$\begin{aligned}f_{Y}(y) &=f\left(T^{-1}(y)\right) J_{T}\left(T^{-1}(y)\right) \\ &=f\left(y^{1 / 2}\right) \frac{1}{2 y^{1 / 2}} \\ &= \begin{cases}0 & y \leq 0 \text { or } y \geq 4 \\ 1 / 4 y^{1 / 2} & 0<y<4\end{cases} \end{aligned}$

E.g. $\int_{-\infty}^{\infty} \varphi(x) d x=1$ $\varphi$ $N(0,1)$ pdf.

$\begin{aligned}\left(\int_{-\infty}^{\infty} \varphi(x) d x\right)^{2} &=\int_{-\infty}^{\infty} \varphi(x) d x \int_{-\infty}^{\infty} \varphi(y) d y =\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \frac{1}{2 \pi} \exp \left(-\frac{x^{2}+y^{2}}{2}\right) d x d y \end{aligned}$

$T(x, y)=(r, \theta)$ $r \in(0, \infty), \theta \in[0,2 \pi)$

$(x, y)=T^{-1}(r, \theta)=(r \cos \theta, r \sin \theta)$

$\begin{aligned}J_{T\inv}(r, \theta) &=\left|\det\left(\begin{array}{cc} \frac{\partial r \cos \theta}{\partial r} & \frac{\partial r \cos \theta}{\partial \theta} \\ \frac{\partial r \sin \theta}{\partial r} & \frac{\partial r \sin \theta}{\partial \theta} \end{array}\right)\right|^{-1} \\ &=\left|\det\left(\begin{array}{cc} \cos \theta & -r \sin \theta \\ \sin \theta & r \cos \theta \end{array}\right)\right|^{-1} \\ &=\left|r\left(\cos ^{2} \theta+\sin ^{2} \theta\right)\right|^{-1}=1 / r \end{aligned}$

$J_{T}(\b x) = J_{T^{-1}}^{-1}(T(\b x)) = \d{1 \o J_{T^{-1}}(T(\b x))} = r$ $r^{2}=x^{2}+y^{2}$ ,

$\begin{aligned}&\left(\int_{-\infty}^{\infty} \varphi(x) d x\right)^{2}=\int_{0}^{\infty} \int_{0}^{2 \pi} \frac{r}{2 \pi} \exp \left(-r^{2} / 2\right) d \theta d r = \int_{0}^{\infty} r \exp \left(-r^{2} / 2\right) d r=-\left.\exp \left(-r^{2} / 2\right)\right|_{0} ^{\infty}=1 \end{aligned}$

$\int_{-\infty}^{\infty} \varphi(x) d x=1$

Def. Affine Transformation

(Affine transformation are linear transformations plus a constant.)

$T: R^{k} \rightarrow R^{k}$
$T(\b x)=A \b x+\mathbf{b}=\left(\begin{array}{c} a_{11} x_{1}+\cdots+a_{1 k} x_{k}+b_{1} \\ a_{21} x_{1}+\cdots+a_{2 k} x_{k}+b_{2} \\ \vdots \\ a_{k 1} x_{1}+\cdots+a_{k k} x_{k}+b_{k} \end{array}\right)$ $\mathbf{b} \in R^{k}, A \in R^{k \times k}$

$J_{T}(\b x)=\left|\det\left(\begin{array}{ccc} \frac{\partial T_{1}(\b x)}{\partial x_{1}} & \ldots & \frac{\partial T_{1}(\b x)}{\partial x_{k}} \\ \vdots & & \vdots \\ \frac{\partial T_{k}(\b x)}{\partial x_{1}} & \ldots & \frac{\partial T_{k}(\b x)}{\partial x_{k}} \end{array}\right)\right|^{-1}=|\det A\ |^{-1}$

$T\left(\mathrm{x}_{1}\right)=T\left(\mathrm{x}_{2}\right)$ $A\left(\mathrm{x}_{1}-\mathrm{x}_{2}\right)=\mathbf{0}$ $T$ $A$ $T^{-1}(\mathbf{y})=A^{-1}(\mathbf{y}-\mathbf{b})=\b x$

$\b Y = A\b X + \b b$ $f_\b Y(\b y) = f_\b X(T\inv (\b y))J_T(T\inv (\b y)) = f_\b X(A\inv(\b y-\b b))|\det A\ |\inv$

Multivariate Normal

$\mathbf{Z} \sim N_{k}(\mathbf{0}, I)$ $f_{\mathbf{Z}}(\mathbf{z})=(2 \pi)^{-k / 2} \exp \left(-\mathbf{z}' \mathbf{z} / 2\right)$ $\mathbf{z} \in R^{k}$

$\x=A \mathbf{Z}+\mu \iff \b Z = A\inv(\x - \mu)$ $A \in R^{k \times k}$ $\mu \in R^{k}$ $\x$ is an affine transformation, we know it has an a.c. distribution with density:

$\begin{aligned}f_{\x}(\b x) &=f_\b Z\left(A^{-1}(\b x-\mu)\right)|\det A|^{-1} \\ &=(2 \pi)^{-k / 2} \exp \left(-\left(A^{-1}(\b x-\mu)\right)' A^{-1}(\b x-\mu) / 2\right)|\det A|^{-1} \expl {plug in Z} = A\inv(X-\mu)\\ &=(2 \pi)^{-k / 2}|\det A|^{-1} \exp \left(-(\b x-\mu)'\left(A^{-1}\right)' A^{-1}(\b x-\mu) / 2\right)\expl {reorder} \\ &=(2 \pi)^{-k / 2}\left|\det A \det A'\right|^{-1 / 2} \exp \left(-(\b x-\mu)'\left(A A'\right)^{-1}(\b x-\mu) / 2\right)\expl{det}(A)= \det(A^T) \\ &=(2 \pi)^{-k / 2}\left|\det A A'\right|^{-1 / 2} \exp \left(-(\b x-\mu)'\left(A A'\right)^{-1}(\b x-\mu) / 2\right)\expl{det}(AB) = \det(A)\det(B) \\ &=(2 \pi)^{-k / 2}(\det \Sigma)^{-1 / 2} \exp \left(-(\b x-\mu)' \Sigma^{-1}(\b x-\mu) / 2\right) \end{aligned}$

$\Sigma=A A'\in R^{k \times k}$

$\x$ $\x \sim N_k(\mu, \Sigma)$

Note $\Sigma$ is symmetric, invertible, and positive definite (see note from lecture 2)

Ex. Suppose