on 16-Sep-2019 (Mon)

\(p(y ; \eta)=b(y) \exp \left(\eta^{T} T(y)-a(\eta)\right)\)

Here, \(\eta\) is called the natural parameter (also called the canonical parameter) of the distribution; T(y) is the sufficient statistic (for the distributions we consider, it will often be the case that T(y) = y); and \(a(\eta)\) is the log partition function. The quantity \(e^{-a(\eta)}\) essentially plays the role of a normalization constant, that makes sure the distribution \(p(y;\eta)\) sums/integrates over y to 1.

To work our way up to GLMs, we will begin by defining exponential family distributions. We say that a class of distributions is in the exponential family if it can be written in the form

\(p(y ; \eta)=b(y) \exp \left(\eta^{T} T(y)-a(\eta)\right)\)

Here, \(\eta\) is called the [...] (also called the [...] ) of the distribution; T(y) is the sufficient statistic (for the distributions we consider, it will often be the case that T(y) = y); and \(a(\eta)\) is the log partition function. The quantity \(e^{-a(\eta)}\) essentially plays the role of a normalization constant, that makes sure the distribution \(p(y;\eta)\) sums/integrates over y to 1.

natural parameter

canonical parameter

To work our way up to GLMs, we will begin by defining exponential family distributions. We say that a class of distributions is in the exponential family if it can be written in the form

\(p(y ; \eta)=b(y) \exp \left(\eta^{T} T(y)-a(\eta)\right)\)

Here, \(\eta\) is called the natural parameter (also called the canonical parameter) of the distribution; T(y) is the [...] (for the distributions we consider, it will often be the case that T(y) = y); and \(a(\eta)\) is the log partition function. The quantity \(e^{-a(\eta)}\) essentially plays the role of a normalization constant, that makes sure the distribution \(p(y;\eta)\) sums/integrates over y to 1.

sufficient statistic

To work our way up to GLMs, we will begin by defining exponential family distributions. We say that a class of distributions is in the exponential family if it can be written in the form

\(p(y ; \eta)=b(y) \exp \left(\eta^{T} T(y)-a(\eta)\right)\)

Here, \(\eta\) is called the natural parameter (also called the canonical parameter) of the distribution; T(y) is the sufficient statistic (for the distributions we consider, it will often be the case that T(y) = y); and \(a(\eta)\) is the [...] . The quantity \(e^{-a(\eta)}\) essentially plays the role of a normalization constant, that makes sure the distribution \(p(y;\eta)\) sums/integrates over y to 1.

1. \(y | x ; \theta \sim \text { Exponential Family }(\eta)\)

2. Goal is to predict \(h(x)=\mathrm{E}[T(y) | x]\)

3. The natural parameter \(\eta\) and the inputs x are related linearly: \(\eta = \theta^Tx\)

canonical response

canonical link

E.g. \(\phi=g(\eta)=1 /\left(1+e^{-\eta}\right)\)is the canonical response function for the Bernoulli distribution, \(\eta= g^{-1}(\phi)=\log (\phi /(1-\phi))\) is the canonical link function.