on 01-Mar-2017 (Wed)

The likelihood function, although it specifies a probability at each value of θ,isnot a probability distribution. In particular, it does not integrate to 1

Bernoulli likelihood function really refers to a single flip

there are two desiderata for mathematical tractability. First, it would be convenient if the product of p(y|θ) and p(θ), which is in the numerator of Bayes’ rule, results in a function of the same form as p(θ ). When this is the case, the prior and posterior beliefs are described using the same form of function. This quality allows us to include subsequent additional data and derive another posterior distribution, again of the same form as the prior.

Second, we desire the denominator of Bayes’ rule (Equation 5.9, p. 107), namely dθ p(y|θ)p(θ), to be solvable analytically. This quality also depends on how the form of the function p(θ ) relates to the form of the function p(y|θ). When the forms of p(y|θ) and p(θ) combine so that the posterior distribution has the same form as the prior distribution, then p(θ) is called a conjugate prior for p(y|θ)

A probability density of that form is called a beta distribution.Formally,abeta distribution has two parameters, called a and b, and the density itself is defined as p(θ|a, b) = beta(θ|a, b) = θ (a−1) (1 − θ) (b−1) /B(a, b)

Inferring a Binomial Probability via Exact Mathematical Analysis 127 A probability density of that form is called a beta distribution.Formally,abeta distribution has two parameters, called a and b, and the density itself is defined as p(θ|a, b) = beta(θ|a, b) = θ (a−1) (1 − θ) (b−1) /B(a, b) (6.3) where B (a, b) is simply a normalizing constant that ensures that the area under the beta density integrates to 1.0, as all probability density functions must

In other words, the normalizer for the beta distribution is the beta function \(B(a,b) = \int d\theta \space \theta^{a-1}(1-\theta)^{b-1}\)

In R, beta(θ|a, b) is dbeta(θ,a,b),and B(a, b) is beta(a,b)

Notice that as a gets bigger (left to right across columns of Figure 6.1), the bulk of the distribution moves rightward over higher values of θ,butasb gets bigger (top to bottomacrossrowsofFigure 6.1), the bulk of the distribution moves leftward over lower values of θ.

Notice that as a and b get bigger together, the beta distribution gets narrower.

The variables a and b are called the shape parameters of the beta distribution because they determine its shape

Often we think of our prior beliefs in terms of a central tendency and certainty about that central tendency. For example, in thinking about the probability of left handedness in the general population of people, we might think from everyday experience that it’s around 10%. But if we are not very certain about that value, we might consider the equivalent previous sample size to be small, say, n = 10

Our goal is to convert a prior belief expressed in terms of central tendency and sample size into equivalent values of a and b in the beta distribution

It turns out that the mean of the beta(θ|a, b) distribution is μ = a/(a + b) and the mode is ω = (a − 1)/(a + b − 2) for a > 1and b > 1(μ is Greek letter mu and ω is Greek letter omega)

The spread of the beta distribution is related to the “concentration” κ = a +b (κ is Greek letter kappa)

Solving those equations for a and b yields the following formulas for a and b in terms of the mean μ,themodeω, and the concentration κ: a = μκ and b = (1 − μ)κ (6.5) a = ω(κ − 2) + 1andb = (1 − ω)(κ − 2) + 1forκ>2

The value we choose for the prior κ can be thought of this way: It is the number of new flips of the coin that we would need to make us teeter between the new data and the prior belief about μ. If we would only need a few new flips to sway our beliefs, then our prior beliefs should be represented by a small κ. If we would need a large number of new flips to sway us away from our prior beliefs about μ, then our prior beliefs are worth a very large κ.

Because the beta distribution is usually skewed, it can be more intuitive to think in terms of its mode instead of its mean. When κ is smaller, as in the left column, the beta distribution is wider than when κ is larger, as in the right column

For a beta density with mean μ and standard deviation σ , the shape parameters are a = μ μ(1 − μ) σ 2 − 1 and b = (1 − μ) μ(1 − μ) σ 2 − 1

the standard deviation must make sense in the context of a beta density. In particular, the standard deviation should typically be less than 0.28867

a beta(θ|12, 12) distribution has a standard deviation of 0.1

In most applications, we will deal with beta distributions for which a ≥ 1and b ≥ 1, that is, κ>2. This reflects prior knowledge that the coin has a head side and a

The standard deviation of the beta distribution is \(\sqrt{μ(1 − μ)/(a + b +1)}\). Notice that the standard deviation gets smaller when the concentration κ = a + b gets larger.

There are some situations, however, in which it may be convenient to use beta distributions in which a < 1and/orb < 1, or for which we cannot be confident that κ>2. For example, we might believe that the coin is a trick coin that nearly always comes up heads or nearly always comes up tails, but we don’t know which. In these situations, we cannot use the parameterization in terms of the mode, which requires κ>2, and instead we can use the parameterization of the beta distribution in terms of the mean

If the prior distribution is beta(θ|a, b), and the data have z heads in N flips, then the posterior distribution is beta(θ|z + a, N − z + b)

If the initial prior is a beta distribution, then the posterior distribution is always a beta distribution

It turns out that the posterior mean can be algebraically re-arranged into a weighted average of the prior mean, a/(a + b), and the data proportion, z/N ,as follows: z + a N + a + b posterior = z N data N N + a + b weight + a a + b prior a + b N + a + b

Any region of a protein’s surface that can interact with another molecule through sets of noncovalent bonds is called a binding site

If a bind- ing site recognizes the surface of a second protein, the tight binding of two folded polypeptide chains at this site creates a larger protein molecule with a precisely defined geometry. Each polypeptide chain in such a protein is called a protein subunit

In the simplest case, two identical folded polypeptide chains bind to each other in a “head-to-head” arrangement, forming a symmetric complex of two protein subunits (a dimer) held together by interactions between two identical binding sites.

Why is a helix such a common structure in biology? As we have seen, biological structures are often formed by linking similar subunits into long, repetitive chains. If all the subunits are identical, the neighboring subunits in the chain can often fit together in only one way, adjusting their relative positions to minimize the free energy of the con- tact between them. As a result, each subunit is positioned in exactly the same way in relation to the next, so that subunit 3 fits onto subunit 2 in the same way that subunit 2 fits onto subunit 1, and so on. Because it is very rare for subunits to join up in a straight line, this arrangement generally results in a helix

Handedness is not affected by turning the helix upside down, but it is reversed if the helix is reflected in the mirror

there are also functions that require each individual protein molecule to span a large distance. These proteins generally have a relatively simple, elongated three-di- mensional structure and are commonly referred to as fibrous proteins

The coiled-coil regions are capped at each end by globular domains containing bind- ing sites. This enables this class of protein to assemble into ropelike intermediate filaments—an important component of the cytoskeleton that creates the cell’s internal structural framework

Fibrous proteins are especially abundant outside the cell, where they are a main component of the gel-like extracellular matrix that helps to bind collections of cells together to form tissues

Cells secrete extracellular matrix proteins into their surroundings, where they often assemble into sheets or long fibrils. Colla- gen is the most abundant of these proteins in animal tissues

Collagen is a triple helix formed by three extended protein chains that wrap around one another (bottom). Many rodlike collagen molecules are cross-linked together in the extracellular space to form unextendable collagen fibrils (top) that have the tensile strength of steel. The striping on the collagen fibril is caused by the regular repeating arrangement of the collagen molecules within the fibril

Elastin polypeptide chains are cross-linked together in the extracellular space to form rubberlike, elastic fibers. Each elastin molecule uncoils into a more extended conformation when the fiber is stretched and recoils spontaneously as soon as the stretching force is relaxed. The cross-linking in the extracellular space mentioned creates covalent linkages between lysine side chains, but the chemistry is different for collagen and elastin.

As a reference, it is useful to remember that standard metal screws, which insert when turned clockwise, are right-handed.

Many proteins were also known to have intrinsically disordered tails at one or the other end of a structured domain (see, for example, the histones in Figure 4–24). But the extent of such disordered structure only became clear when genomes were sequenced. This allowed bio- informatic methods to be used to analyze the amino acid sequences that genes encode, searching for disordered regions based on their unusually low hydropho- bicity and relatively high net charge.

it is now thought that perhaps a quarter of all eukaryotic proteins can adopt structures that are mostly disordered, fluctuating rapidly between many different conforma- tions.

What do these disordered regions do? Some known functions are illustrated in Figure 3–24. One predominant func- tion is to form specific binding sites for other protein molecules that are of high specificity, but readily altered by protein phosphorylation, protein dephosphor- ylation, or any of the other covalent modifications that are triggered by cell sig- naling events

an unstructured region can also serve as a “tether” to hold two protein domains in close proximity to facilitate their inter- action.

this tethering function that allows substrates to move between active sites in large multienzyme complexes

A simi- lar tethering function allows large scaffold proteins with multiple protein-binding sites to concentrate sets of interacting proteins, both increasing reaction rates and confining their reaction to a particular site in a cell

large numbers of disordered protein chains in close proximity can create micro-regions of gel-like consistency inside the cell that restrict diffusion

the abundant nucleoporins that coat the inner surface of the nuclear pore complex form a random coil meshwork (Figure 3–24) that is critical for selective nuclear transport

lysozyme—an enzyme in tears that dissolves bacterial cell walls—retains its antibacterial activity for a long time because it is stabilized by such cross-linkages.

Disulfide bonds generally fail to form in the cytosol, where a high concentra- tion of reducing agents converts S–S bonds back to cysteine –SH groups. Appar- ently, proteins do not require this type of reinforcement in the relatively mild envi- ronment inside the cell

The use of smaller subunits to build larger structures has several advantages: 1. A large structure built from one or a few repeating smaller subunits requires only a small amount of genetic information. 2. Both assembly and disassembly can be readily controlled reversible pro- cesses, because the subunits associate through multiple bonds of relatively low energy. 3. Errors in the synthesis of the structure can be more easily avoided, since correction mechanisms can operate during the course of assembly to exclude malformed subunits.

These principles are dramatically illustrated in the protein coat or capsid of many simple viruses, which takes the form of a hollow sphere based on an icosahedron

The first large macromolecular aggregate shown to be capable of self-as- sembly from its component parts was tobacco mosaic virus (TMV ).

the simplest case, a long core protein or other macromolecule provides a scaffold that determines the extent of the final assembly. This is the mechanism that deter- mines the length of the TMV particle, where the RNA chain provides the core. Similarly, a core protein interacting with actin is thought to determine the length of the thin filaments in muscle.

In these cases, part of the assembly information is provided by special enzymes and other proteins that perform the function of templates, serving as assembly factors that guide construction but take no part in the final assembled structure.

These are self-propagat- ing, stable β-sheet aggregates called amyloid fibrils. These fibrils are built from a series of identical polypeptide chains that become layered one over the other to create a continuous stack of β sheets, with the β strands oriented perpendicular to the fibril axis to form a cross-beta filament

Typically, hundreds of monomers will aggregate to form an unbranched fibrous structure that is several micrometers long and 5 to 15 nm in width

A surprisingly large fraction of pro- teins have the potential to form such structures, because the short segment of the polypeptide chain that forms the spine of the fibril can have a variety of different sequences and follow one of several different paths (Figure 3–32). However, very few proteins will actually form this structure inside cells

In normal humans, the quality control mechanisms governing proteins grad- ually decline with age, occasionally permitting normal proteins to form patho- logical aggregates. The protein aggregates may be released from dead cells and accumulate as amyloid in the extracellular matrix.

the abnormal forma- tion of highly stable amyloid fibrils is thought to play a central causative role in both Alzheimer’s and Parkinson’s diseases

A set of closely related diseases—scra- pie in sheep, Creutzfeldt–Jakob disease (CJD) in humans, Kuru in humans, and bovine spongiform encephalopathy (BSE) in cattle—are caused by a misfolded, aggregated form of a particular protein called PrP

PrP is nor- mally located on the outer surface of the plasma membrane, most prominently in neurons, and it has the unfortunate property of forming amyloid fibrils that are “infectious” because they convert normally folded molecules of PrP to the same pathological form

another remarkable feature of prions. These protein molecules can form several distinctively different types of amyloid fibrils from the same polypeptide chain. Moreover, each type of aggregate can be infectious, forcing normal protein molecules to adopt the same type of abnormal structure. Thus, several different “strains” of infectious particles can arise from the same polypeptide chain.

Eukaryotic cells, for example, store many different peptide and protein hormones that they will secrete in specialized “secretory granules,” which package a high concentra- tion of their cargo in dense cores with a regular structure (see Figure 13–65). We now know that these structured cores consist of amyloid fibrils, which in this case have a structure that causes them to dissolve to release soluble cargo after being secreted by exocytosis to the cell exterior

Many bacteria use the amyloid structure in a very different way, secreting proteins that form long amy- loid fibrils projecting from the cell exterior that help to bind bacterial neighbors into biofilms

these biofilms help bacteria to survive in adverse environments (including in humans treated with antibiotics), new drugs that specifically disrupt the fibrous networks formed by bacterial amyloids have promise for treating human infections

new experiments reveal that a large set of low com- plexity domains can form amyloid fibers that have functional roles in both the cell nucleus and the cell cytoplasm

these newly discovered structures are held together by weaker noncovalent bonds and readily dissociate in response to signals—hence their name reversible amyloids.

hormones of the endocrine system, such as glucagon and calcitonin, are efficiently stored as short amyloid fibrils, which dissociate when they reach the cell exterior.

Dep ending on the structure of the problem, it ma y not b e p ossible to design a unique mapping from to . A B If A is taller than it is wide, then it is p ossible for this equation to hav e no solution. If A is wider than it is tall, then there could b e multiple p ossible solutions. The Mo ore-P enrose pseudoin verse allo ws us to mak e some headwa y in these cases. The pseudoinv erse of is defined as a matrix A A + = lim α 0 ( A A I + α ) − 1 A

Practical algorithms for computing the pseudoinv erse are not based on this defini- tion, but rather the formula A + = V D + U , (2.47) where U , D and V are the singular v alue decomp osition of A , and the pseudoin verse D + of a diagonal matrix D is obtained by taking the recipro cal of its non-zero elemen ts then taking the transp ose of the resulting matrix

When A has more columns than rows, then solving a linear equation using the pseudoin v erse provides one of the man y p ossible solutions. Specifically , it pro vides the solution x = A + y with minimal Euclidean norm ||x||₂ among all p ossible solutions

When A has more rows than columns, it is p ossible for there to b e no solution. In this case, using the pseudoinv erse gives us the x for which Ax is as close as p ossible to in terms of Euclidean norm y || − || Ax y 2

The trace operator gives the sum of all of the diagonal en tries of a matrix: T r( ) = A i A i,i

the trace op erator provides an alternativ e w a y of writing the F rob enius norm of a matrix: ||A||_F = Tr( AA^T )

the trace op erator is in v arian t to the transp ose op erator: T r(A) = T r(A^T )

The trace of a square matrix comp osed of many factors is also in v arian t to mo ving the last factor into the first p osition, if the shap es of the corresp onding matrices allo w the resulting pro duct to b e defined: T r( ) = T r( ) = T r( ) AB C C AB B C A

This inv ariance to cyclic p erm utation holds even if the resulting pro duct has a differen t shap e. F or example, for A ∈ R m n × and B ∈ R n m × , w e ha v e T r(AB ) = T r( BA)

a scalar is its own trace: a = T r ( a

The determinant of a square matrix, denoted det ( A ) , is a function mapping matrices to real scalars. The determinant is equal to the pro duct of all the eigen v alues of the matrix.

The absolute v alue of the determinant can b e thought of as a measure of how m uc h m ultiplication by the matrix expands or con tracts space.

If the determinant is 0, then space is contracted completely along at least one dimension, causing it to lose all of its v olume. If the determinant is 1, then the transformation preserves volume

One simple mac hine learning algorithm, principal components analysis or PCA can b e deriv ed using only knowledge of basic linear algebra

Lossy compression means storing the p oints in a wa y that requires less memory but ma y lose some precision

PCA is defined b y our c hoice of the deco ding function. Sp ecifically , to mak e the deco der very simple, we choose to use matrix m ultiplication to map the co de back in to R n . Let , where g ( ) = c D c D ∈ R n l × is the matrix defining the deco ding

T o k eep the enco ding problem easy , PCA constrains the colum ns of D to b e orthogonal to eac h other.

how to generate the optimal co de p oint c ∗ for eac h input p oint x . One w a y to do this is to minimize the distance b etw een the input p oint x and its reconstruction, g ( c ∗ )

In addition, many items of expense require the company to make estimates that can significantly affect net income. Analysis of a company’s financial statements, and particularly comparison of one company’s financial statements with those of another, requires an understanding of differences in these estimates and their potential impact.

If, for example, a company shows a significant year-to-year change in its estimates of uncollectible accounts as a percentage of sales, warranty expenses as a percentage of sales, or estimated useful lives of assets, the analyst should seek to understand the underlying reasons.

Do the changes reflect a change in business operations (e.g., lower estimated warranty expenses reflecting recent experience of fewer warranty claims because of improved product quality)? Or are the changes seemingly unrelated to changes in business operations and thus possibly a signal that a company is manipulating estimates in order to achieve a particular effect on its reported net income?

There are three p ossible sources of uncertain t y: 1. Inheren t stochasticit y in the system b eing mo deled. F or example, most in terpretations of quantum mechanics describ e the dynamics of subatomic particles as b eing probabilistic. W e can also create theoretical scenarios that w e p ostulate to ha v e random dynamics, such as a hypothetical card game where w e assume that the cards are truly sh uffled in to a random order. 2. Incomplete observ ability . Ev en deterministic systems can app ear sto chastic when w e cannot observ e all of the v ariables that drive the b ehavior of the system. F or example, in the Mont y Hall problem, a game sho w con testan t is ask ed to choose b etw een three do ors and wins a prize held b ehind the c hosen do or. T w o do ors lead to a goat while a third leads to a car. The outcome giv en the contestan t’s c hoice is deterministic, but from the con testan t’s p oin t of view, the outcome is uncertain. 3. Incomplete mo deling. When we use a mo del that must discard some of the information we hav e observ ed, the discarded information results in uncertain t y in the mo del’s predictions.

Whe n we sa y that an outcome has a probability p of o ccurring, it means that if we rep eated the exp erimen t (e.g., dra w a hand of cards) infinitely many times, then prop ortion p of the rep etitions would result in that outcome.

In the case of the do ctor diagnosing the patient, we use probability to represent a degree of b elief , with 1 indicating absolute certain t y that the patien t has the flu and 0 indicating absolute certain t y that the patient do es not hav e the flu.

The former kind of probability , related directly to the rates at which even ts o ccur, is kno wn as frequen tist probability , while the latter, related to qualitative levels of certain t y , is known as Bay esian probabilit y

A random v ariable is a v ariable that can take on different v alues randomly

W e t ypically denote the random v ariable itself with a low er case letter in plain typeface, and the v alues it can take on with low er case script letters

F or vector-v alued v ariables, we would write the random v ariable as x and one of its v alues as x

On its o wn, a random v ariable is just a description of the states that are p ossible; it m ust b e coupled with a probability distribution that sp ecifies how likely each of these states are.

A probabilit y distribution is a description of how lik ely a random v ariable or set of random v ariables is to take on each of its p ossible states.

A probability distribution ov er discrete v ariables may b e describ ed using a proba- bilit y mass function (PMF)

The probabilit y mass function maps from a state of a random v ariable to the probabilit y of that random v ariable taking on that state

Probabilit y mass functions can act on many v ariables at the same time. Suc h a probability distribution ov er man y v ariables is known as a join t probabilit y distribution

T o b e a probability mass function on a random v ariable x , a function P m ust satisfy the follo wing prop erties:
• The domain of P must b e the set of all p ossible states of x.
• ∀x ∈ x , 0 ≤ P ( x ) ≤ 1 . An imp ossible ev en t has probabilit y and no state can 0 b e less probable than that. Likewise, an ev en t that is guaran teed to happ en has probabilit y , and no state can ha v e a greater c hance of o ccurring. 1
• \(\sum_{x \in x}P(x) = 1\) . W e refer to this prop erty as b eing normalized . Without this prop ert y , we could obtain probabilities greater than one by computing the probabilit y of one of man y ev en ts o ccurring.

W e can place a uniform distribution on x —that is, make each of its states equally lik ely—b y setting its probabilit y mass function to P (x = x _i ) = 1/k

T o b e a probabilit y densit y function, a function p m ust satisfy the follo wing prop erties: • The domain of m ust b e the set of all p ossible states of x. p • ∀ ∈ ≥ ≤ x x , p x ( ) 0 ( ) . p Note that we do not require x 1 . • p x dx ( ) = 1

The “;” notation means “parametrized b y”; w e consider x to b e the argument of the function, while a and b are parameters that define the function

W e often denote that x follo ws the uniform distribution on [ a, b ] b y writing x . ∼ U a, b (

The probability distribution o v er the subset is kno wn as the marginal probability distribution.

F or example, supp ose w e ha v e discrete random v ariables x and y , and we know P , ( x y . W e can find x with the : ) P ( ) sum rule ∀ ∈ x x x , P ( = ) = x y P x, y . ( = x y =

The name “marginal probabilit y” comes from the pro cess of computing marginal probabilities on pap er. When the v alues of P ( x y , ) are written in a grid with differen t v alues of x in rows and different v alues of y in columns, it is natural to sum across a row of the grid, then write P ( x ) in the margin of the pap er just to the righ t of the ro w

In many cases, we are inte rested in the probabilit y of some even t, given that some other ev en t has happ ened. This is called a conditional probability

Computing the consequences of an action is called making an in terv en tion query . In terv en tion queries are the domain of causal mo deling

An y join t probabilit y distribution o v er man y random v ariables ma y be decomp osed in to conditional distributions o v er only one v ariable: P ( x (1) , . . . , x ( ) n ) = ( P x (1) )Π n i =2 P ( x ( ) i | x (1) , . . . , x ( 1) i − ) . (3.6) This observ ation is known as the c hain rule or pro duct rule of probabilit y

T wo random v ariables x and y are indep enden t if their probabilit y distribution can b e expressed as a pro duct of tw o factors, one in v olving only x and one in v olving only y: ∀ ∈ ∈ x x , y y x y x y (3.7) , p ( = x, = ) = ( y p = ) ( x p = ) y

T wo random v ariables x and y are conditionally indep enden t giv en a random v ariable z if the conditional probability distribution o v er x and y factorizes in this w a y for ev ery v alue of z: ∀ ∈ ∈ ∈ | | | x x , y y , z z x y , p ( = x, = y z x = ) = ( z p = x z y = ) ( z p = y z = ) z

W e can denote indep endence and conditional indep endence with compact notation: x y ⊥ means that x and y are indep enden t, while x y z ⊥ | means that x and y are conditionally indep enden t giv en z

The exp ectation or exp ected v alue of some function f ( x ) with resp ect to a probabilit y distribution P ( x ) is the a v erage or mean v alue that f tak es on when x is dra wn from . F or discrete v ariables this can be computed with a summation: E_x~P[f(x)]=\(\sum_{x} P(x)f(x)\)

When the iden tit y of the distribution is clear from the context, w e ma y simply write the name of the random v ariable that the exp ectation is ov er, as in E x [ f ( x )]

Exp ectations are linear, for example, E x [ ( ) + ( )] = αf x β g x α E x [ ( )] + f x β E x [ ( )] g x

The v ariance giv es a measure of how muc h the v alues of a function of a random v ariable x v ary as we sample differen t v alues of x from its probability distribution: V ar( ( )) = f x E ( ( ) [ ( )]) f x − E f x 2

The co v ariance giv es some sense of how muc h t w o v alues are linearly related to eac h other, as w ell as the scale of these v ariables: Co v( ( ) ( )) = [( ( ) [ ( )]) ( ( ) [ ( )])] f x , g y E f x − E f x g y − E g y

High absolute v alues of the cov ariance mean that the v alues change very muc h and are b oth far from their resp ectiv e means at the same time

If the sign of the co v ariance is p ositiv e, then b oth v ariables tend to tak e on relatively high v alues sim ultaneously . If the sign of the co v ariance is negative, then one v ariable tends to tak e on a relativ ely high v alue at the times that the other takes on a relatively lo w v alue and vice v ersa.

Other measures such as correlation normalize the con tribution of each v ariable in order to measure only how m uc h the v ariables are related, rather than also b eing affected b y the scale of the separate v ariables

They are related b ecause t w o v ariables that are indep endent hav e zero co v ariance, and tw o v ariables that hav e non-zero cov ariance are dep endent

Indep endence is a stronger requirement than zero co v ariance, b ecause indep endence also excludes nonlinear relationships.

The co v ariance matrix of a random vector x ∈ R n is an n n × matrix, suc h that Cov(x)_i,j = Cov( x _i , x _j ) . (3.14) The diagonal elements of the covariance give the variance: Cov( x _i , x _i ) = Var( x _i )

The Bernoulli distribution is a distribution ov er a single binary random v ariable. It is controlled by a single parameter φ ∈ [0 , 1] , whic h gives the probability of the random v ariable b eing equal to 1. It has the following prop erties:
P (x = 1) = φ
P (x = 0) = 1-φ
P (x = x ) = φ ^x (1 − φ) ^{1 − x}
E _x [x] = φ
V ar (x) = φ (1− φ)

The m ultinoulli or categorical distribution is a distribution o v er a single discrete v ariable with k differen t states, where k is finite.

The multinoulli distribution is a sp ecial case of the m ultinomial distribution. A m ultinomial distribution is the distribution ov er vectors in { 0 , . . . , n } k represen ting ho w many times each of the k categories is visited when n samples are drawn from a multinoulli distribution. Man y texts use the term “multinomial” to refer to multinoulli distributions without clarifying that they refer only to the case. n = 1

The most commonly used distribution o v er real n um b ers is the normal distribu- tion , also kno wn as the : Gaussian distribution N ( ; x µ, σ 2 ) = 1 2 π σ 2 exp − 1 2 σ 2 ( ) x µ − 2

When w e need to frequen tly ev aluate the PDF with differen t parameter v alues, a more efficient w a y of parametrizing the distribution is to use a parameter β ∈ (0 , ∞ ) to control the precision or in v erse v ariance of the distribution: N ( ; x µ, β − 1 ) = β 2 π exp − 1 2 β x µ ( − ) 2

The cen tral limit theorem sho ws that the sum of many indep en- den t random v ariables is appro ximately normally distributed

out of all p ossible probability distributions with the same v ariance, the normal distribution enco des the maxim um amount of uncertaint y ov er the real num b ers

W e can th us think of the normal distribution as b eing the one that inserts the least amoun t of prior kno wledge into a mo del

The normal distribution generalizes to R n , in whic h case it is known as the m ultiv ariate normal distribution . It may b e parametrized with a p ositive definite symmetric matrix : Σ N ( ; ) = x µ , Σ 1 (2 ) π n det( ) Σ exp − 1 2 ( ) x µ − Σ − 1 ( ) x µ −

An even simpler v ersion is the isotropic Gaussian distribution, whose cov ariance matrix is a scalar times the iden tit y matrix

In the context of deep learning, w e often wan t to hav e a probability distribution with a sharp point at x = 0 . T o accomplish this, w e can use the exp onen tial distribution : p x λ λ ( ; ) = 1 x ≥ 0 exp ( ) − λx . (3.25) The exp onen tial distribution uses the indicator function 1 x ≥ 0 to assign probabilit y zero to all negativ e v alues of . x

A closely related probabilit y distribution that allo ws us to place a sharp p eak of probabilit y mass at an arbitrary p oin t is the µ Laplace distribution Laplace( ; ) = x µ, γ 1 2 γ exp − | − | x µ γ

In some cases, we wish to sp ecify that all of the mass in a probabilit y distribution clusters around a single p oin t. This can b e accomplished b y defining a PDF using the Dirac delta function, : p(x) = δ(x-µ) The Dirac delta function is defined such that it is zero-v alued everywhere except 0, y et integrates to 1.

mathematical ob ject called a generalized function that is defined in terms of its prop erties when integrated

W e can think of the Dirac delta function as b eing the limit p oin t of a series of functions that put less and less mass on all p oints other than zero.

By defining p ( x ) to be δ shifted b y − µ w e obtain an infinitely narrow and infinitely high p eak of probabilit y mass where . x µ =

A common use of the Dirac delta distribution is as a comp onent of an empirical distribution , ˆ p ( ) = x 1 m m i =1 δ ( x x − ( ) i ) (3.28) whic h puts probability mass 1 m on eac h of the m p oin ts x (1) , . . . , x ( ) m forming a giv en dataset or collection of samples

Another imp ortant p ersp ective on the empirical distribution is that it is the probabilit y density that maximizes the likelihoo d of the training data

One common w a y of com bining distributions is to construct a mixture distribution . A mixture distribution is made up of sev eral comp onen t distributions. On eac h trial, the choice of whic h comp onent distribution generates the sample is determined by sampling a comp onent iden tit y from a m ultinoulli distribution: P ( ) = x i P i P i ( = c ) ( = x c | ) (3.29) where c is the multinoulli distribution ov er comp onent identities

A latent v ariable is a random v ariable that w e cannot observe directly .

A very p ow erful and common type of mixture model is the Gaussian mixture mo del, in whic h the comp onen ts p ( x | c = i ) are Gaussians. Each comp onent has a separately parametrized mean µ ( ) i and cov ariance Σ ( ) i

A Gaussian mixture mo del is a univ ersal approximator of densities, in the sense that an y smo oth densit y can b e appro ximated with an y sp ecific, non-zero amoun t of error by a Gaussian mixture model with enough comp onen ts

Certain functions arise often while working with probability distributions, especially the probabilit y distributions used in deep learning models. One of these functions is the : logistic sigmoid σ(x) = 1/(1 + exp(− x))

The logistic sigmoid is commonly used to pro duce the φ parameter of a Bernoulli

an isotropic co v ariance matrix, meaning it has the same amount of v ariance in each direction

diagonal co v ariance matrix, meaning it can control the v ariance separately along each axis-aligned direction.

The third comp onen t has a full-rank cov ariance matrix, allowing it to control the v ariance separately along an arbitrary basis of directions

The 3.3 sigmoid function saturates when its argument is v ery p ositiv e or very negative, meaning that the function b ecomes v ery flat and insensitive to small changes in its input

Another commonly encountered function is the softplus function ( , Dugas et al. 2001 ): ζ(x) = log (1 + exp(x)) The softplus function can be useful for pro ducing the β or σ parameter of a normal distribution b ecause its range is (0 , ∞ )

The name of the softplus function comes from the fact that it is a smo othed or “softened” version of x + = max(0 ) , x

σ x ( ) = exp( ) x exp( ) + exp(0) x (3.33) d dx σ x σ x σ x ( ) = ( )(1 − ( )) (3.34) 1 ( ) = ( ) − σ x σ − x (3.35) log ( ) = ( ) σ x − ζ − x (3.36) d dx ζ x σ x ( ) = ( ) (3.37) ∀ ∈ x (0 1) , , σ − 1 ( ) = log x x 1 − x (3.38) ∀ x > , ζ 0 − 1 ( ) = log (exp( ) 1) x x − (3.39) ζ x ( ) = x −∞ σ y dy ( ) (3.40) ζ x ζ x x ( ) − ( − ) =

The function σ − 1 ( x ) is called the logit in statistics, but this term is more rarely used in mac hine learning

The softplus 3.41 function is intended as a smo othed v ersion of the p ositiv e part function, x + = max { 0 , x }

Just as x can b e recov ered from its p ositive part and negativ e part via the iden tit y x ⁺ − x ⁻ = x , it is also p ossible to reco v er x using the same relationship b et w een and ζ (x), ζ ( −x)

Measure theory pro vides a rigorous wa y of describing that a set of p oin ts is negligibly small. Such a set is said to hav e measure zero .

F or our purp oses, it is sufficient to understand the intuition that a set of measure zero o ccupies no v olume in the space w e are measuring. F or example, within R 2 , a line has measure zero, while a filled p olygon has positive measure