Gaussian processes (GP) and Bayesian Neural Network
Gaussian processes (GP) and Bayesian Neural Network (BNN)
GP can be used for both regression and classification
Give a reliable estimate of their own uncertainty
A Gaussian process is a probability distribution over possible functions.
Bayesian NN
Type of GP weights are considered a probability distribution and Infinite weights
Integrate over all evidence of infinite weights is analytically intractable
Simulation or numerical based alternative approaches such as Monte Carlo Markov Chain (MCMC), variational inference (VI) are considered
We can use VI that minimizes the divergence of two distributions through optimization
KL-divergence between Q(W|θ) and P(W|X) is defined as
After training, we need to sample from distribution of weight to compute the point estimate. Hence we cn get an information about the uncertainty.
1) If some domain/expert knowledge as prior knowledge exist. The best way to incorporate knowledge is Bayesian.
2) In case of limited number of samples. Over-fitting phenomenon wont happen. In addition, as we know the uncertainty of the model, we know where we need additional data points (in case we have access to the DAQ).
3) If we may have test samples outside the train set (e.g. outliers). Consider a model that distinguish cat and dog given an image. If we give it an elephant image, it's better to choose one with a low probability.
4) In case of Large variation in underling model (e.g. hybrid modes).
5) When the flexibility and the performance of the modelling is really important. Since in non-parametric approaches, the model is not confined to a set of model structure.
