How to Calculate Negative Log Likelihood Python

What is Negative Log Likelihood?

The negative log likelihood (NLL) is a measure of how well a statistical model fits observed data. It's derived from the likelihood function, which represents the probability of observing the given data under the model. The negative log likelihood is simply the negative natural logarithm of the likelihood function.

In machine learning, minimizing the negative log likelihood is equivalent to maximizing the likelihood function. This is often used in training models like logistic regression, where the goal is to find parameters that maximize the likelihood of the observed data.

Negative Log Likelihood Formula

The negative log likelihood is calculated using the following formula:

For a single observation, the negative log likelihood for that observation is simply -log(P(y|x;θ)). For multiple observations, we sum these values across all observations.

Calculating Negative Log Likelihood in Python

Python provides several libraries that make calculating negative log likelihood straightforward. The most common approach is to use the scipy.stats module, which contains many probability distributions with built-in methods for calculating log probabilities.

Using scipy.stats

Here's an example of how to calculate negative log likelihood for a normal distribution using scipy.stats:

import numpy as np
from scipy.stats import norm

# Observed data
data = np.array([1.2, 1.5, 1.8, 2.1, 2.4])

# Model parameters (mean and standard deviation)
mu, sigma = 1.5, 0.5

# Calculate log likelihood for each observation
log_likelihoods = norm.logpdf(data, mu, sigma)

# Calculate negative log likelihood
nll = -np.sum(log_likelihoods)

print(f"Negative Log Likelihood: {nll:.4f}")

Custom Implementation

If you need to implement the calculation yourself, you can use the natural logarithm function from the math module:

import math

def negative_log_likelihood(data, mu, sigma):
    nll = 0
    for x in data:
        # Probability density function for normal distribution
        pdf = (1 / (sigma * math.sqrt(2 * math.pi))) * math.exp(-0.5 * ((x - mu) / sigma) ** 2)
        # Add to negative log likelihood
        nll -= math.log(pdf)
    return nll

# Example usage
data = [1.2, 1.5, 1.8, 2.1, 2.4]
mu, sigma = 1.5, 0.5
print(f"Negative Log Likelihood: {negative_log_likelihood(data, mu, sigma):.4f}")

Example Calculation

Let's walk through a concrete example to illustrate how negative log likelihood works. Suppose we have the following observed data points: [1.2, 1.5, 1.8, 2.1, 2.4]. We'll assume a normal distribution with mean (μ) = 1.5 and standard deviation (σ) = 0.5.

For each data point, we calculate the probability density using the normal distribution formula:

Then we take the natural logarithm of each probability and sum them up. Finally, we take the negative of this sum to get the negative log likelihood.

The result will be a single number representing how well the normal distribution with μ=1.5 and σ=0.5 fits the observed data. A lower value indicates a better fit.

Practical Applications

Negative log likelihood has several important applications in statistics and machine learning:

• Model Selection: Comparing different models to determine which one fits the data better
• Parameter Estimation: Finding the optimal parameters for a statistical model
• Machine Learning: Used as a loss function in algorithms like logistic regression
• Hypothesis Testing: Assessing the goodness-of-fit of a model

In machine learning, minimizing the negative log likelihood is equivalent to maximizing the likelihood function, which is the foundation of many probabilistic models.