The Maximum Likelihood Estimator uses gathered sample data to predict the population parameter or the parameter of the true distribution.

The likelihood function, L(θ) is the product of the pdf evaluated at the data points. Multiply 1 by the pdf at each data point\text{The likelihood function, } L(\theta) \text{ is the product of the pdf evaluated at the data points. Multiply 1 by the pdf at each data point}

Steps to find the MLE

  1. Find L(θ) (also known as the likelihood function)
  2. Find ln L(θ)
  3. Calculate the Score Function
∂lnL(θ)∂θ\frac{\partial ln L(\theta)}{\partial \theta}
  1. Find the Score equation by setting the Score Function = 0
  2. Solve for the parameter
  3. Check that the second derivative of ln L(θ) is negative, that is
∂2lnL(θ)∂θ2\frac{\partial^2 ln L(\theta)}{\partial \theta^2}
  1. Check the support

Example

f(xiσ)=12σe−∣x∣σL(θ)=∏i=1nf(x∣θ=σ)=∏i=1n12σe−∣xi∣σL(θ)=[12σe−∣x1∣σ][12σe−∣x2∣σ]…[12σe−∣xi∣σ]L(θ)=(12σ)ne−∑i=1n∣xi∣σf(x_i \sigma) = \frac{1}{2\sigma}e^{-\frac{|x|}{\sigma}} \\ L(\theta) = \prod_{i = 1}^{n} f(x | \theta = \sigma) = \prod_{i = 1}^{n} \frac{1}{2\sigma}e^{-\frac{|x_i|}{\sigma}} \\ L(\theta) = [\frac{1}{2\sigma}e^{-\frac{|x_1|}{\sigma}}] [\frac{1}{2\sigma}e^{-\frac{|x_2|}{\sigma}}] \dots [\frac{1}{2\sigma}e^{-\frac{|x_i|}{\sigma}}] \\ L(\theta) = (\frac{1}{2\sigma})^n e^{-\frac{\sum_{i = 1}^{n} |x_i|}{\sigma}} \\

<hr >

lnL(θ)=ln[(12σ)ne−∑i=1n∣xi∣σ]lnL(θ)=ln[(12σ)n]+ln[e−∑i=1n∣xi∣σ]lnL(θ)=n∗ln(1)−n∗ln(2)−n∗ln(σ)+ln[e−∑i=1n∣xi∣σ]lnL(θ)=0−n∗ln(2)−n∗ln(σ)+e−∑i=1n∣xi∣σln L(\theta) = ln[(\frac{1}{2\sigma})^n e^{-\frac{\sum_{i = 1}^{n} |x_i|}{\sigma}}] \\ ln L(\theta) = ln[(\frac{1}{2\sigma})^n ] + ln[e^{-\frac{\sum_{i = 1}^{n} |x_i|}{\sigma}}] \\ ln L(\theta) = n * ln(1) - n * ln(2) - n * ln(\sigma) + ln[e^{-\frac{\sum_{i = 1}^{n} |x_i|}{\sigma}}] \\ ln L(\theta) = 0 - n * ln(2) - n * ln(\sigma) + e^{-\frac{\sum_{i = 1}^{n} |x_i|}{\sigma}}

<hr >

The last term is technically sigma to the power of -1, so we can use the power rule on it

Score=ddθℓ(θ)=ddσ[−n∗ln(2)−n∗ln(σ)+e−∑i=1n∣xi∣σ]Score=−nσ+∑i=1n∣xi∣σ2Score = \frac{d}{d\theta} \ell(\theta) = \frac{d}{d\sigma} [- n * ln(2) - n * ln(\sigma) + e^{-\frac{\sum_{i = 1}^{n} |x_i|}{\sigma}}] \\ Score = -\frac{n}{\sigma} + \frac{\sum_{i = 1}^{n} |x_i|}{\sigma^2}

Maximums and minimums occur when the score is equal to 0, so we must set the score equal to 0

−nσ+∑i=1n∣xi∣σ2=0∑i=1n∣xi∣σ2=nσnσ=∑i=1n∣xi∣σ=∑i=1n∣xi∣n-\frac{n}{\sigma} + \frac{\sum_{i = 1}^{n} |x_i|}{\sigma^2} = 0 \\ \frac{\sum_{i = 1}^{n} |x_i|}{\sigma^2} = \frac{n}{\sigma} \\ n \sigma = \sum_{i = 1}^{n} |x_i| \\ \sigma = \frac{\sum_{i = 1}^{n} |x_i|}{n}

This is our Maximum Likelihood Estimator; we can estimate sigma by adding all of our data points and dividing by n. However, before doing this, we need to check the concavity to ensure that it really is the maximum likelihood estimator (must be negative).

<hr >

d2d2θ<0→ddθScore<0ddσ[−nσ+∑i=1n∣xi∣σ2]=nσ2−2∑i=1n∣xi∣σ3<0We must combine the two fractions together in order to determine whether the term is less than 0nσ2−2∑i=1n∣xi∣σ3=nσσ3−2∑i=1n∣xi∣σ3=nσ−2∑i=1n∣xi∣σ3Substituting the first derivative into the second derivative:n∑i=1n∣xi∣n−2∑i=1n∣xi∣σ3=∑i=1n∣xi∣−2∑i=1n∣xi∣σ3=−∑i=1n∣xi∣σ3<0Because the numerator must always be positive (sum of absolute values), the second derivative is therefore always less than 0.\frac{d^2}{d^2 \theta} < 0 \rightarrow \frac{d}{d \theta} Score < 0 \\ \frac{d}{d\sigma} [-\frac{n}{\sigma} + \frac{\sum_{i = 1}^{n} |x_i|}{\sigma^2}] = \frac{n}{\sigma^2} - 2\frac{\sum_{i = 1}^{n} |x_i|}{\sigma^3} < 0 \\ \text{We must combine the two fractions together in order to determine whether the term is less than 0} \\ \frac{n}{\sigma^2} - 2\frac{\sum_{i = 1}^{n} |x_i|}{\sigma^3} = \frac{n \sigma}{\sigma^3} - 2\frac{\sum_{i = 1}^{n} |x_i|}{\sigma^3} = \frac{n \sigma - 2\sum_{i = 1}^{n} |x_i|}{\sigma^3} \\ \text{Substituting the first derivative into the second derivative:} \\ \frac{n \frac{\sum_{i = 1}^{n} |x_i|}{n} - 2\sum_{i = 1}^{n} |x_i|}{\sigma^3 } = \frac{\sum_{i = 1}^{n} |x_i| - 2\sum_{i = 1}^{n} |x_i|}{\sigma^3 } = -\frac{\sum_{i = 1}^{n} |x_i|}{\sigma^3 } < 0\\ \text{Because the numerator must always be positive (sum of absolute values), the second derivative is therefore always less than 0.}

<hr >

Final answer: σ^=∑i=1n∣xi∣n is the MLE of σ\text{Final answer: } \hat{\sigma} = \frac{\sum_{i = 1}^{n} |x_i|}{n} \text{ is the MLE of } \sigma