Intro

  • Mutual Information: I(X;Y)=H(X)+H(Y)−H(X,Y)I(X; Y) = H(X) + H(Y) - H(X, Y)
  • KL Divergence: D(p∣∣q)=∑p(x)log⁡p(x)q(x)D(p \vert\vert q) = \sum p(x) \log\frac{p(x)}{q(x)}
    • Measures similarity between distributions
    • 0 if equal
    • Asymmetric
  • Copula Models: Generates a multivariate model with unpopular marginal probabilities by using a change of variables to convert the original model to a Gaussian model
    • Used in economics; normalizing flows is more popular in machine learning