on 06-Jan-2018 (Sat)

In probability theory and statistics, the negative binomial distribution is a discrete probability distribution of the number of successes in a sequence of independent and identically distributed Bernoulli trials before a specified (non-random) number of failures (denoted r) occurs.

The negative binomial distribution also arises as a continuous mixture of Poisson distributions (i.e. a compound probability distribution) where the mixing distribution of the Poisson rate is a gamma distribution. That is, we can view the negative binomial as a Poisson(λ) distribution, where λ is itself a random variable, distributed as a gamma distribution with shape = r and scale θ = p/(1 − p) or correspondingly rate β = (1 − p)/p .

One definition is that a random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution.

The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value

The mutual information of a distribution is a special case of the Kullback–Leibler divergence in which is the full multivariate distribution and is the product of the 1-dimensional marginal distributions

In the bivariate case the expression for the mutual information is:

In mathematics, and in particular, algebra, a generalized inverse of an element x is an element y that has some properties of an inverse element but not necessarily all of them. Generalized inverses can be defined in any mathematical structure that involves associative multiplication, that is, in a semigroup.

Formally, given a matrix and a matrix , A is a generalized inverse of if it satisfies the condition : .

A common use of the pseudoinverse is to compute a 'best fit' (least squares) solution to a system of linear equations that lacks a unique solution

The pseudoinverse is defined and unique for all matrices whose entries are real or complex numbers. It can be computed using the singular value decomposition.

For , a pseudoinverse of is defined as a matrix satisfying all of the following four criteria:

1. ( AA⁺ need not be the general identity matrix, but it maps all column vectors of A to themselves);
2. ( A⁺ is a weak inverse for the multiplicative semigroup);
3. ( AA⁺ is Hermitian); and
4. ( A⁺A is also Hermitian).

Moore-Penrose Pseudo-inverse

exists for any matrix , but when the latter has full rank, can be expressed as a simple algebraic formula.

In particular, when has linearly independent columns (and thus matrix is invertible), can be computed as:

...

Conditional distributions

If N-dimensional x is partitioned as follows

and accordingly μ and Σ are partitioned as follows

then the distribution of x₁ conditional on x₂ = a is multivariate normal (x₁ | x₂ = a) ~ N( μ , Σ ) where

and covariance matrix

This matrix is the Schur complement of Σ₂₂ in Σ. This means that to calculate the conditional covariance matrix, one inverts the overall covariance matrix, drops the rows and columns corresponding to the variables being conditioned upon, and then inverts back to get the conditional covariance matrix. Here is the generalized inverse of .

Note that knowing that x₂ = a alters the variance, though the new variance does not depend on the specific value of a; perhaps more surprisingly, the mean is shifted by ; compare this with the situation of not knowing the value of a, in which case x₁ would have distribution .

An interesting fact derived in order to prove this result, is that the random vectors and are independent.

The matrix Σ₁₂Σ₂₂⁻¹ is known as the matrix of regression

...

To obtain the marginal distribution over a subset of multivariate normal random variables, one only needs to drop the irrelevant variables (the variables that one wants to marginalize out) from the mean vector and the covariance matrix. The proof for this follows from the definitions of multivariate normal distributions and linear algebra.^[16]

If Y = c + BX is an affine transformation of where c is an vector of constants and B is a constant matrix, then Y has a multivariate normal distribution with expected value c + Bμ and variance BΣB^T .

Corollaries: sums of Gaussian are Gaussian, marginals of Gaussian are Gaussian.

The equidensity contours of a non-singular multivariate normal distribution are ellipsoids (i.e. linear transformations of hyperspheres) centered at the mean.

The directions of the principal axes of the ellipsoids are given by the eigenvectors of the covariance matrix Σ. The squared relative lengths of the principal axes are given by the corresponding eigenvalues.

The distribution N(μ, Σ) is in effect N(0, I) scaled by Λ^1/2, rotated by U and translated by μ.

An ellipsoid is a surface that may be obtained from a sphere by deforming it by means of directional scalings, or more generally, of an affine transformation.

In geometry, an affine transformation, affine map^[1] or an affinity (from the Latin, affinis, "connected with") is a function between affine spaces which preserves points, straight lines and planes. Also, sets of parallel lines remain parallel after an affine transformation. An affine transformation does not necessarily preserve angles between lines or distances between points, though it does preserve ratios of distances between points lying on a straight line.

In linear algebra, eigendecomposition or sometimes spectral decomposition is the factorization of a matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors. Only diagonalizable matrices can be factorized in this way.

The eigendecomposition can be derived from the fundamental property of eigenvectors: and thus which yields .

Viewed as a machine-learning algorithm, a Gaussian process uses lazy learning and a measure of the similarity between points (the kernel function) to predict the value for an unseen point from training data.

n probability theory and statistics, the characteristic function of any real-valued random variable completely defines its probability distribution.

If a random variable admits a probability density function, then the characteristic function is the Fourier transform of the probability density function.

The Fourier transform (FT) decomposes a function of time (a signal) into the frequencies that make it up

The Fourier transform (FT) decomposes a function of time (a signal) into the frequencies that make it up

A key fact of Gaussian processes is that they can be completely defined by their second-order statistics.

if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour. Importantly the non-negative definiteness of this function enables its spectral decomposition using the Karhunen–Loeve expansion.

Stationarity refers to the process' behaviour regarding the separation of any two points x and x' . If the process is stationary, it depends on their separation, x − x', while if non-stationary it depends on the actual position of the points x and x'.

If the process depends only on |x − x'|, the Euclidean distance (not the direction) between x and x', then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be homogeneous;^[7] in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

If we expect that for "near-by" input points x and x' their corresponding output points y and y' to be "near-by" also, then the assumption of continuity is present.

Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input x to a two dimensional vector u(x) = (cos(x), sin(x)).

Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the data-set at hand.

In the bivariate case where x is partitioned into X₁ and X₂, the conditional distribution of X₁ given X₂ is

where is the correlation coefficient between X₁ and X₂.

In the Indian buffet process, the rows of correspond to customers and the columns correspond to dishes in an infinitely long buffet.

In mathematics, the n-th harmonic number is the sum of the reciprocals of the first n natural numbers:

A trend stationary process is not strictly stationary, but can easily be transformed into a stationary process by removing the underlying trend, which is solely a function of time.

processes with one or more unit roots can be made stationary through differencing.

In finance, mean reversion is the assumption that a stock's price will tend to move to the average price over time.

If the process depends only on |x − x'|, the Euclidean distance (not the direction) between x and x', then the process is considered isotropic.

A process that is concurrently stationary and isotropic is considered to be homogeneous;^[7] in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour.

In geometry, an affine transformation, affine map^[1] or an affinity (from the Latin, affinis, "connected with") is a function between affine spaces which preserves points, straight lines and planes.

An affine transformation does not necessarily preserve angles between lines or distances between points, though it does preserve ratios of distances between points lying on a straight line.