# Maximum Likelihood Estimation of Custom Models in Python with StatsModels

Maximum likelihood estimation is a common method for fitting statistical models. In Python, it is quite possible to fit maximum likelihood models using just `scipy.optimize`

. Over time, however, I have come to prefer the convenience provided by `statsmodels`

’ `GenericLikelihoodModel`

. In this post, I will show how easy it is to subclass `GenericLikelihoodModel`

and take advantage of much of `statsmodels`

’ well-developed machinery for maximum likelihood estimation of custom models.

#### Zero-inflated Poisson models

The model we use for this demonstration is a zero-inflated Poisson model. This is a model for count data that generalizes the Poisson model by allowing for an overabundance of zero observations.

The model has two parameters, \(\pi\), the proportion of excess zero observations, and \(\lambda\), the mean of the Poisson distribution. We assume that observations from this model are generated as follows. First, a weighted coin with probability \(\pi\) of landing on heads is flipped. If the result is heads, the observation is zero. If the result is tails, the observation is generated from a Poisson distribution with mean \(\lambda\). Note that there are two ways for an observation to be zero under this model:

- the coin is heads, and
- the coin is tails, and the sample from the Poisson distribution is zero.

If \(X\) has a zero-inflated Poisson distribution with parameters \(\pi\) and \(\lambda\), its probability mass function is given by

\[\begin{align*} P(X = 0) & = \pi + (1 - \pi)\ e^{-\lambda} \\ P(X = x) & = (1 - \pi)\ e^{-\lambda}\ \frac{\lambda^x}{x!} \textrm{ for } x > 0. \end{align*}\]

In this post, we will use the parameter values \(\pi = 0.3\) and \(\lambda = 2\). The probability mass function of the zero-inflated Poisson distribution is shown below, next to a normal Poisson distribution, for comparison.

```
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from scipy import stats
import seaborn as sns
from statsmodels.base.model import GenericLikelihoodModel
123456789) np.random.seed(
```

```
= 0.3
pi = 2. lambda_
```

```
def zip_pmf(x, pi=pi, lambda_=lambda_):
if pi < 0 or pi > 1 or lambda_ <= 0:
return np.zeros_like(x)
else:
return (x == 0) * pi + (1 - pi) * stats.poisson.pmf(x, lambda_)
```

#### Maximum likelihood estimation

First we generate 1,000 observations from the zero-inflated model.

```
= 1000
N
= stats.bernoulli.rvs(pi, size=N)
inflated_zero = (1 - inflated_zero) * stats.poisson.rvs(lambda_, size=N) x
```

We are now ready to estimate \(\pi\) and \(\lambda\) by maximum likelihood. To do so, we define a class that inherits from `statsmodels`

’ `GenericLikelihoodModel`

as follows.

```
class ZeroInflatedPoisson(GenericLikelihoodModel):
def __init__(self, endog, exog=None, **kwds):
if exog is None:
= np.zeros_like(endog)
exog
super(ZeroInflatedPoisson, self).__init__(endog, exog, **kwds)
def nloglikeobs(self, params):
= params[0]
pi = params[1]
lambda_
return -np.log(zip_pmf(self.endog, pi=pi, lambda_=lambda_))
def fit(self, start_params=None, maxiter=10000, maxfun=5000, **kwds):
if start_params is None:
= self.endog.mean()
lambda_start = (self.endog == 0).mean() - stats.poisson.pmf(0, lambda_start)
excess_zeros
= np.array([excess_zeros, lambda_start])
start_params
return super(ZeroInflatedPoisson, self).fit(start_params=start_params,
=maxiter, maxfun=maxfun, **kwds) maxiter
```

The key component of this class is the method `nloglikeobs`

, which returns the negative log likelihood of each observed value in `endog`

. Secondarily, we must also supply reasonable initial guesses of the parameters in `fit`

. Obtaining the maximum likelihood estimate is now simple.

```
= ZeroInflatedPoisson(x)
model = model.fit() results
```

```
Optimization terminated successfully.
Current function value: 1.586641
Iterations: 37
Function evaluations: 70
```

We see that we have estimated the parameters fairly well.

```
= results.params
pi_mle, lambda_mle
pi_mle, lambda_mle
```

`(0.31542487710071976, 2.0451304204850853)`

There are many advantages to buying into the `statsmodels`

ecosystem and subclassing `GenericLikelihoodModel`

. The already-written `statsmodels`

code handles storing the observations and the interaction with `scipy.optimize`

for us. (It is possible to control the use of `scipy.optimize`

through keyword arguments to `fit`

.)

We also gain access to many of `statsmodels`

’ built in model analysis tools. For example, we can use bootstrap resampling to estimate the variation in our parameter estimates.

```
= results.bootstrap(nrep=500, store=True)
boot_mean, boot_std, boot_samples = boot_samples[:, 0]
boot_pis = boot_samples[:, 1] boot_lambdas
```

The next time you are fitting a model using maximum likelihood, try integrating with `statsmodels`

to take advantage of the significant amount of work that has gone into its ecosystem.